From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id BE50D6EC58; Tue, 25 May 2021 23:43:02 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BE50D6EC58 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1621975382; bh=ttktbGijw0+v2uRjHG+UTzGH2mDedDbD68H7RWZ6Q5M=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=D+2teXepWNQv8v/nsH3/xpa58X5Nsl2LVIK+H7sm5pQvHu1VXo1idTrx+P+tBIROP kZK7+6NuTfoTAFxS/JHyDzojlCnx64UMMvpsZKbKkC6CIh0byCmyB7GLnnJlONsFT9 Uknpv4oBFf+e647JWVbkXbqFHYcjfG2Lly83SK8E= Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id E448B6EC58 for ; Tue, 25 May 2021 23:43:01 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E448B6EC58 Received: by smtp41.i.mail.ru with esmtpa (envelope-from ) id 1lldtE-0006g3-Sk; Tue, 25 May 2021 23:43:01 +0300 To: tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org, olegrok@tarantool.org Date: Tue, 25 May 2021 22:42:59 +0200 Message-Id: <8d7e89a4884559963c02719fca0dc0720632fc73.1621975324.git.v.shpilevoy@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9F4D89F53055293B7D3D5E71D6EBA968B6047C881F6B697D5182A05F538085040AAA20D0DD207B7F553B9BA8A49543315FA0C5225EBD1F4398C4BD8243C120F08 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE75AD53DF1D86BACA3EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637486F32EF78EC606F8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D878792B6284A298F19A33C9F8C81B888A117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC5F0C88D684269EDEA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F4460429728776938767073520B1593CA6EC85F86D618001F51B5FD3F9D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EEC24E1E72F37C03A020DA3B6C150F7642D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE364E7220B7C5505929735652A29929C6CC4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F79006373BC478629CBEC79DEFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24209795067102C07E8F7B195E1C97831FC8F9967B3302D179BFD3320250DC888 X-C1DE0DAB: 8BD88D57C5CADBC8B2710865C386751094C72BDDC9A8ED5CA3B1A56EE2B804F6B226C914C9968946695E9D90444CEC264DCC8C77FBA9901322D2CEDE4E95CF1BDBE8DEE28BC9005C095FFBCAB1CFE8AAD6CF32B5F8F9D4040F07D487CA38BAF7DFA7C2F88AFB778F589120F7DAE46353205367B2BCC23E5B02D4363B72091FE710A8D9BFF942BBA9DBE8DEE28BC9005C1AB9A781E4F81AC9 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3474B2583E5131598401380C4ACD1FA79B9BEC141E5BD5AF3E793CAB24572C0604C2D8B609EE8F67561D7E09C32AA3244CCA2DDDBB42AB52F78AB117C2381D8074259227199D06760AFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojV3CWresp27fKxcNek9nwRw== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA3110E0599A0826B3035F97CFAB81317680E87BA6DE242C35196B07784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH vshard 1/1] rebalancer: give more info at bucket_recv() fail X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" vshard.storage.bucket_recv() used to raise the natural space:insert(...) error without any additional info when it failed. For instance, due to incorrect format, or a duplicate key. When such an error happens, it is very useful to know what was the problematic space and what was the failed tuple. This patch enriches the space insertion error with that information. The new detailed error object and its message should help to fix the rebalancing issues, which quite often are about schema mismatch in different replicasets. Especially hard to debug when number of spaces is tens of even hundreds. The old way was fine for errors like "duplicate key" because on the newest version of Tarantool it contains the space name, the old and the new tuples. But errors like tuple format mismatch still are not very informative. VShard now tries to enrich all the possible errors. Closes #275 --- Branch: http://github.com/tarantool/vshard/tree/gerold103/gh-275-bucket_recv-detailed-error Issue: https://github.com/tarantool/vshard/issues/275 test/storage/storage.result | 47 +++++++++++++++++++++++++++++++++++ test/storage/storage.test.lua | 17 +++++++++++++ vshard/error.lua | 5 ++++ vshard/storage/init.lua | 8 +++++- 4 files changed, 76 insertions(+), 1 deletion(-) diff --git a/test/storage/storage.result b/test/storage/storage.result index 570d9c6..5372059 100644 --- a/test/storage/storage.result +++ b/test/storage/storage.result @@ -521,6 +521,53 @@ while box.space._bucket:get{4} do vshard.storage.recovery_wakeup() fiber.sleep(0 --- ... -- +-- gh-275: detailed info when couldn't insert into a space. +-- +res, err = vshard.storage.bucket_recv( \ + 4, util.replicasets[2], {{box.space.test.id, {{9, 4}, {10, 4}, {1, 4}}}}) +--- +... +assert(not res) +--- +- true +... +assert(err.space == 'test') +--- +- true +... +assert(err.bucket_id == 4) +--- +- true +... +assert(tostring(err.tuple) == '[1, 4]') +--- +- true +... +assert(err.reason:match('Duplicate key exists') ~= nil) +--- +- true +... +err = err.message +--- +... +assert(err:match('bucket 4 data in space "test" at tuple %[1, 4%]') ~= nil) +--- +- true +... +assert(err:match('Duplicate key exists') ~= nil) +--- +- true +... +while box.space._bucket:get{4} do \ + vshard.storage.recovery_wakeup() fiber.sleep(0.01) \ +end +--- +... +assert(box.space.test:get{9} == nil and box.space.test:get{10} == nil) +--- +- true +... +-- -- Bucket transfer -- -- Transfer to unknown replicaset. diff --git a/test/storage/storage.test.lua b/test/storage/storage.test.lua index 494e2e8..d1f3f50 100644 --- a/test/storage/storage.test.lua +++ b/test/storage/storage.test.lua @@ -125,6 +125,23 @@ vshard.storage.bucket_recv(100, 'from_uuid', {{1000, {{1}}}}) res, err = vshard.storage.bucket_recv(4, util.replicasets[2], {{1000, {{1}}}}) util.portable_error(err) while box.space._bucket:get{4} do vshard.storage.recovery_wakeup() fiber.sleep(0.01) end +-- +-- gh-275: detailed info when couldn't insert into a space. +-- +res, err = vshard.storage.bucket_recv( \ + 4, util.replicasets[2], {{box.space.test.id, {{9, 4}, {10, 4}, {1, 4}}}}) +assert(not res) +assert(err.space == 'test') +assert(err.bucket_id == 4) +assert(tostring(err.tuple) == '[1, 4]') +assert(err.reason:match('Duplicate key exists') ~= nil) +err = err.message +assert(err:match('bucket 4 data in space "test" at tuple %[1, 4%]') ~= nil) +assert(err:match('Duplicate key exists') ~= nil) +while box.space._bucket:get{4} do \ + vshard.storage.recovery_wakeup() fiber.sleep(0.01) \ +end +assert(box.space.test:get{9} == nil and box.space.test:get{10} == nil) -- -- Bucket transfer diff --git a/vshard/error.lua b/vshard/error.lua index b02bfe9..bcbcd71 100644 --- a/vshard/error.lua +++ b/vshard/error.lua @@ -149,6 +149,11 @@ local error_message_template = { msg = 'Can not delete a storage ref: %s', args = {'reason'}, }, + [30] = { + name = 'BUCKET_RECV_DATA_ERROR', + msg = 'Can not receive the bucket %s data in space "%s" at tuple %s: %s', + args = {'bucket_id', 'space', 'tuple', 'reason'}, + } } -- diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index 63e0398..7045d91 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -1254,7 +1254,13 @@ local function bucket_recv_xc(bucket_id, from, data, opts) end box.begin() for _, tuple in ipairs(space_data) do - space:insert(tuple) + local ok, err = pcall(space.insert, space, tuple) + if not ok then + box.rollback() + return nil, lerror.vshard(lerror.code.BUCKET_RECV_DATA_ERROR, + bucket_id, space.name, + box.tuple.new(tuple), err) + end limit = limit - 1 if limit == 0 then box.commit() -- 2.24.3 (Apple Git-128)