From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id ABB6E6EC55; Wed, 28 Jul 2021 00:53:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ABB6E6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1627422827; bh=fLj9zTgd1K9AdcXgnHSTJc1pKJTQD3SuWuXDc+8eLOw=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=rKm0kiMxUYY5VVjCd5MYEPAAFf6qkWINUPaE9PiUaaWT1ZbBHAit5d8V1k17MrPSY 3gs2t2KPdMILVBepiVQ/Bzv5uGUx8me/grIIgtcM9PR8c6fnE8RE4ZB7Dc0S0fQ8o4 yUEtyXDN+s5uptWx4BNRyMdGfJMCGoP1IZgwRDns= Received: from smtpng3.i.mail.ru (smtpng3.i.mail.ru [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A44626EC55 for ; Wed, 28 Jul 2021 00:53:46 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A44626EC55 Received: by smtpng3.m.smailru.net with esmtpa (envelope-from ) id 1m8V1F-00045j-Ia; Wed, 28 Jul 2021 00:53:46 +0300 To: Sergey Ostanevich References: Message-ID: <959e1a8f-f5f8-85db-e0ed-22d249ed496c@tarantool.org> Date: Tue, 27 Jul 2021 23:53:44 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C30288BCF456A452EC429C5145D5B58EFD182A05F538085040E8F40C127FE714CEC046C60187264CA74112F0F7B4BAF60AC32CE3EC1C96BE24 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7466896EF24E80F12EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063790C8302959DAAF098638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8A310FB365A2EE127E57E73A95FEF1410117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC2EE5AD8F952D28FBA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FF04B652EEC242312D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B613439FA09F3DCB32089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458834459D11680B505F0C1BD99E1C80DBAA508ED8D426B4C24 X-C1DE0DAB: 0D63561A33F958A5D31C6CB6E1624EF1BC941771014D7FDCEE263289BB5DC15ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75FA7FF33AA1A4D21C410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A533D6DF7731C6649AB9D7AECEE60CC26B4A4B7C6199238AB2DA92D41D31B8BF4570DB1DD31EDBFC1D7E09C32AA3244C23003B78083F283377AF1730D9A09548E3D93501275E802F927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojiF1u9eOpfTTLRZ9QFtCsYA== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D134DC65CB20275C3B2DC069A180ED9CB3841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 1/1] replication: set replica ID before _cluster commit X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the review! >> On 25 Jul 2021, at 19:53, Vladislav Shpilevoy via Tarantool-patches wrote: >> >> Replica registration works via looking for the smallest not >> occupied ID in _cluster and inserting it into the space. >> >> It works not so good when mvcc is enabled. In particular, if more >> than 1 replica try to register at the same time, they might get >> the same replica_id because don't see changes of each other until >> the registration in _cluster is complete. >> >> This in the end leads to all replicas failing the registration >> except one with the 'duplicate key' error (primary index in >> _cluster is replica ID). >> >> The patch makes the replicas occupy their ID before they commit it >> into _cluster. And new replica ID search now uses the replica ID >> map instead of _cluster iterator. >> >> This way the registration works like before - like MVCC does not >> exist which is fine. > > Did you discuss the MVCC capabilities - if we can address it there, by > setting a dedicated flag for this particular - and perhaps for some > other internal - space(s) to suppress MVCC ‘coverage’ for them. The > solution will be way more common with supposedly less hassle with > local structures, triggers, and so on? I did think of that in the beginning, but then I decided it is a bad idea. Firstly, from the side of unnecessary complication of MVCC which is complex enough already. Secondly, _cluster DQL is going to see the uncommitted changes despite MVCC turned on. Does not look well. In my patch I made so that you do not see the tuples in :get() and :select() in _cluster and you won't stumble into the "dirty" errors because you are not supposed to make any :insert() nor :replace() into this space manually. Automatic registration bypasses the busy IDs before trying to insert them. >> diff --git a/src/box/alter.cc b/src/box/alter.cc >> index 89bb5946c..64ba09021 100644 >> --- a/src/box/alter.cc >> +++ b/src/box/alter.cc >> @@ -4280,14 +4244,34 @@ on_replace_dd_cluster(struct trigger *trigger, void *event) >> "updates of instance uuid"); >> return -1; >> } >> - } else { >> - struct trigger *on_commit; >> - on_commit = txn_alter_trigger_new(register_replica, >> - new_tuple); >> - if (on_commit == NULL) >> - return -1; >> - txn_stmt_on_commit(stmt, on_commit); >> + return 0; >> + } >> + /* >> + * With read-views enabled there might be already a replica >> + * whose registration is in progress in another transaction. >> + * With the same replica ID. >> + */ >> + if (replica_by_id(replica_id) != NULL) { >> + diag_set(ClientError, ER_UNSUPPORTED, "Tarantool", >> + "more than 1 replica with the same ID"); > > There should be more details on replica ID/UUID/name here, same as in unregister > panic below. The info would be visible in the logs anyway because before something is added to _cluster, there are always more logs about 'joining ...', 'subscribed replica ...' and so on. But I don't mind to extend the logs. ==================== diff --git a/src/box/alter.cc b/src/box/alter.cc index 64ba09021..390199298 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -4251,9 +4251,14 @@ on_replace_dd_cluster(struct trigger *trigger, void *event) * whose registration is in progress in another transaction. * With the same replica ID. */ - if (replica_by_id(replica_id) != NULL) { - diag_set(ClientError, ER_UNSUPPORTED, "Tarantool", - "more than 1 replica with the same ID"); + struct replica *replica = replica_by_id(replica_id); + if (replica != NULL) { + const char *msg = tt_sprintf( + "more than 1 replica with the same ID %u: new " + "uuid - %s, old uuid - %s", replica_id, + tt_uuid_str(&replica_uuid), + tt_uuid_str(&replica->uuid)); + diag_set(ClientError, ER_UNSUPPORTED, "Tarantool", msg); return -1; } struct trigger *on_rollback = txn_alter_trigger_new( @@ -4265,7 +4270,7 @@ on_replace_dd_cluster(struct trigger *trigger, void *event) * replica ID now. While WAL write is in progress, new replicas * might come, they should see the ID is already in use. */ - struct replica *replica = replica_by_uuid(&replica_uuid); + replica = replica_by_uuid(&replica_uuid); if (replica != NULL) replica_set_id(replica, replica_id); else @@ -4283,6 +4288,10 @@ on_replace_dd_cluster(struct trigger *trigger, void *event) return -1; if (replica_check_id(replica_id) != 0) return -1; + tt_uuid replica_uuid; + if (tuple_field_uuid(old_tuple, BOX_CLUSTER_FIELD_UUID, + &replica_uuid) != 0) + return -1; struct replica *replica = replica_by_id(replica_id); if (replica == NULL) { @@ -4292,7 +4301,15 @@ on_replace_dd_cluster(struct trigger *trigger, void *event) * subsystem is affected. */ panic("Tried to unregister a replica not stored in " - "replica_by_id map, id is %u", replica_id); + "replica_by_id map, id is %u, uuid is %s", + replica_id, tt_uuid_str(&replica_uuid)); + } + if (!tt_uuid_is_equal(&replica->uuid, &replica_uuid)) { + panic("Tried to unregister a replica with id %u, but " + "its uuid is different from stored internally: " + "in space - %s, internally - %s", replica_id, + tt_uuid_str(&replica_uuid), + tt_uuid_str(&replica->uuid)); } /* * Unregister only after commit. Otherwise if the transaction ==================== >> diff --git a/src/box/box.cc b/src/box/box.cc >> index 8c10a99dd..5c10aceff 100644 >> --- a/src/box/box.cc >> +++ b/src/box/box.cc >> @@ -2407,22 +2407,9 @@ box_on_join(const tt_uuid *instance_uuid) >> return; /* nothing to do - already registered */ >> >> box_check_writable_xc(); >> - >> - /** Find the largest existing replica id. */ >> - struct space *space = space_cache_find_xc(BOX_CLUSTER_ID); >> - struct index *index = index_find_system_xc(space, 0); >> - struct iterator *it = index_create_iterator_xc(index, ITER_ALL, >> - NULL, 0); >> - IteratorGuard iter_guard(it); >> - struct tuple *tuple; >> - /** Assign a new replica id. */ >> - uint32_t replica_id = 1; >> - while ((tuple = iterator_next_xc(it)) != NULL) { >> - if (tuple_field_u32_xc(tuple, >> - BOX_CLUSTER_FIELD_ID) != replica_id) >> - break; >> - replica_id++; >> - } >> + uint32_t replica_id; >> + if (replica_find_new_id(&replica_id) != 0) >> + diag_raise(); > > Any info on why register fails? replica_find_new_id() sets diag with a proper error. This is why I do diag_raise() here.