From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 1424B6EC55; Thu, 15 Jul 2021 14:28:48 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1424B6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626348528; bh=u+FVkN5Osdr1ew8k4XlQOkmsv6LNdJt69HryahkYIUA=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=CNuC2CMGuusWwdrT4riLKMqFjeecguPJdIx9YguGsl5Y+MBUGjdNFrAr+2J2majTK bL1ob6xSGjbBBKvadSnfYUNTSKKeOn59rc9VWa/LFZqzN3E484tpzp8tZ07vJgeMCU 7Pi4k2vKaUwp2atGQt03xKUh8W1vdW898ZzCNnd4= Received: from smtp40.i.mail.ru (smtp40.i.mail.ru [94.100.177.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 839976EC55 for ; Thu, 15 Jul 2021 14:28:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 839976EC55 Received: by smtp40.i.mail.ru with esmtpa (envelope-from ) id 1m3zXq-0006Sa-J2; Thu, 15 Jul 2021 14:28:47 +0300 To: Cyrill Gorcunov , tml Cc: Vladislav Shpilevoy References: <20210714212328.701280-1-gorcunov@gmail.com> <20210714212328.701280-4-gorcunov@gmail.com> Message-ID: <76124bb2-95fa-5312-e9ae-8c25c527bca2@tarantool.org> Date: Thu, 15 Jul 2021 14:28:46 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210714212328.701280-4-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-4EC0790: 10 X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD941C43E597735A9C3FDAB68B812060C779F2375474EF21E8A182A05F5380850408E743A4EFB110890B0DD1E850F13F72C517EA09000C8860AB39D7B7C144FD634 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7CE03055CE01F421CEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637D19071B5A26B4BDC8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D81F0B9367E6C9163AECC95A6A886A2DE9117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC2EE5AD8F952D28FBA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FCB629EEF1311BF91D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6BF3059D42242344A089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: 0D63561A33F958A5EA32F82B8CFAD76D2EA8C794382F0312D0E63FDAF3138011D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7501A9DF589746230F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34C1D376EF32BB08960827B4C5C00F1754494E37FADCFE94A67F5365EA01DE6AEF1F343EFC66F69C561D7E09C32AA3244C38A8AF2CFC3BCEE9445BB235F92AC78E97FE24653F78E668FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojSyb42jm8PHIUQBRptiy2fQ== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A44636D1F4A36DD198A9FAC9CF0E239DAF53DADC3B85B7AB78C8424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [RFC v5 3/5] limbo: gather promote tracking into a separate structure X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 15.07.2021 00:23, Cyrill Gorcunov пишет: > It is needed to introduce ordered promote related data > modifications in next patch. > > Part-of #6036 > > Signed-off-by: Cyrill Gorcunov > --- Thanks for the patch! Generally looks good with a couple of comments. > src/box/txn_limbo.c | 24 ++++++++++++++-------- > src/box/txn_limbo.h | 49 ++++++++++++++++++++++++++++----------------- > 2 files changed, 47 insertions(+), 26 deletions(-) > > diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c > index 570f77c46..957fe0d1e 100644 > --- a/src/box/txn_limbo.c > +++ b/src/box/txn_limbo.c > @@ -37,6 +37,13 @@ > > struct txn_limbo txn_limbo; > > +static void > +txn_limbo_promote_create(struct txn_limbo_promote *pmt) > +{ > + vclock_create(&pmt->terms_map); > + pmt->terms_max = 0; > +} > + I don't like the name (limbo->promote, struct txn_limbo_promote), but can't come up with a better one. The structure doesn't hold a specific promote. It's more like "promote history", or "terms seen". Maybe something like "term history" ? "term tracker" ? "remote term set", "term set" ? Just a suggestion, my names are not too good. > static inline void > txn_limbo_create(struct txn_limbo *limbo) > { > @@ -45,8 +52,7 @@ txn_limbo_create(struct txn_limbo *limbo) > limbo->owner_id = REPLICA_ID_NIL; > fiber_cond_create(&limbo->wait_cond); > vclock_create(&limbo->vclock); > - vclock_create(&limbo->promote_term_map); > - limbo->promote_greatest_term = 0; > + txn_limbo_promote_create(&limbo->promote); > limbo->confirmed_lsn = 0; > limbo->rollback_count = 0; > limbo->is_in_rollback = false; > @@ -305,10 +311,11 @@ void > txn_limbo_checkpoint(const struct txn_limbo *limbo, > struct synchro_request *req) > { > + const struct txn_limbo_promote *pmt = &limbo->promote; > req->type = IPROTO_PROMOTE; > req->replica_id = limbo->owner_id; > req->lsn = limbo->confirmed_lsn; > - req->term = limbo->promote_greatest_term; > + req->term = pmt->terms_max; > } > > static void > @@ -726,20 +733,21 @@ txn_limbo_wait_empty(struct txn_limbo *limbo, double timeout) > void > txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) > { > + struct txn_limbo_promote *pmt = &limbo->promote; > uint64_t term = req->term; > uint32_t origin = req->origin_id; > if (txn_limbo_replica_term(limbo, origin) < term) { > - vclock_follow(&limbo->promote_term_map, origin, term); > - if (term > limbo->promote_greatest_term) > - limbo->promote_greatest_term = term; > + vclock_follow(&pmt->terms_map, origin, term); > + if (term > pmt->terms_max) > + pmt->terms_max = term; > } else if (iproto_type_is_promote_request(req->type) && > - limbo->promote_greatest_term > 1) { > + pmt->terms_max > 1) { > /* PROMOTE for outdated term. Ignore. */ > say_info("RAFT: ignoring %s request from instance " > "id %u for term %llu. Greatest term seen " > "before (%llu) is bigger.", > iproto_type_name(req->type), origin, (long long)term, > - (long long)limbo->promote_greatest_term); > + (long long)pmt->terms_max); > return; > } > > diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h > index 53e52f676..70a5fbfd5 100644 > --- a/src/box/txn_limbo.h > +++ b/src/box/txn_limbo.h > @@ -75,6 +75,31 @@ txn_limbo_entry_is_complete(const struct txn_limbo_entry *e) > return e->is_commit || e->is_rollback; > } > > +/** > + * Keep state of promote requests to handle split-brain > + * situation and other errors. > + */ > +struct txn_limbo_promote { > + /** > + * Latest terms received with PROMOTE entries from remote instances. > + * Limbo uses them to filter out the transactions coming not from the > + * limbo owner, but so outdated that they are rolled back everywhere > + * except outdated nodes. > + */ > + struct vclock terms_map; > + /** > + * The biggest PROMOTE term seen by the instance and persisted in WAL. > + * It is related to raft term, but not the same. Synchronous replication > + * represented by the limbo is interested only in the won elections > + * ended with PROMOTE request. > + * It means the limbo's term might be smaller than the raft term, while > + * there are ongoing elections, or the leader is already known and this > + * instance hasn't read its PROMOTE request yet. During other times the > + * limbo and raft are in sync and the terms are the same. > + */ > + uint64_t terms_max; > +}; > + > /** > * Limbo is a place where transactions are stored, which are > * finished, but not committed nor rolled back. These are > @@ -130,23 +155,9 @@ struct txn_limbo { > */ > struct vclock vclock; > /** > - * Latest terms received with PROMOTE entries from remote instances. > - * Limbo uses them to filter out the transactions coming not from the > - * limbo owner, but so outdated that they are rolled back everywhere > - * except outdated nodes. > - */ > - struct vclock promote_term_map; > - /** > - * The biggest PROMOTE term seen by the instance and persisted in WAL. > - * It is related to raft term, but not the same. Synchronous replication > - * represented by the limbo is interested only in the won elections > - * ended with PROMOTE request. > - * It means the limbo's term might be smaller than the raft term, while > - * there are ongoing elections, or the leader is already known and this > - * instance hasn't read its PROMOTE request yet. During other times the > - * limbo and raft are in sync and the terms are the same. > + * Track promote requests. > */ > - uint64_t promote_greatest_term; > + struct txn_limbo_promote promote; > /** > * Maximal LSN gathered quorum and either already confirmed in WAL, or > * whose confirmation is in progress right now. Any attempt to confirm > @@ -218,7 +229,8 @@ txn_limbo_last_entry(struct txn_limbo *limbo) > static inline uint64_t > txn_limbo_replica_term(const struct txn_limbo *limbo, uint32_t replica_id) > { > - return vclock_get(&limbo->promote_term_map, replica_id); > + const struct txn_limbo_promote *pmt = &limbo->promote; > + return vclock_get(&pmt->terms_map, replica_id); > } > > /** > @@ -229,8 +241,9 @@ static inline bool > txn_limbo_is_replica_outdated(const struct txn_limbo *limbo, > uint32_t replica_id) > { > + const struct txn_limbo_promote *pmt = &limbo->promote; > return txn_limbo_replica_term(limbo, replica_id) < > - limbo->promote_greatest_term; > + pmt->terms_max; > } > > /** -- Serge Petrenko