From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id D25C96E200; Fri, 18 Jun 2021 00:02:11 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D25C96E200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1623963731; bh=7CtJvhJTw5H9JRkGlxwdPQp35nE0ccmeNRHYuk9Lqeg=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=vufYthDJ+0qUCpMm1KUQZiKppJi5RD6urU5RP+sf7wBijDLIZW5Wk7i7/JUUvuDUW SJ6dKci9h3VQv/82Fde/fpGyvfCGF61Bpzio3a7W3Idr3XQt2kyWMbHcRYFkdhU/c+ U/r1XU2p1iLx+CRWwLz4/VWJwDThTEq/ZARiRBtE= Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 47F596E201 for ; Fri, 18 Jun 2021 00:00:19 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 47F596E201 Received: by smtp49.i.mail.ru with esmtpa (envelope-from ) id 1ltz7a-0003gC-HR; Fri, 18 Jun 2021 00:00:18 +0300 To: Vladislav Shpilevoy , gorcunov@gmail.com Cc: tarantool-patches@dev.tarantool.org References: <93a4765cb6918512946bf9366fe4ad478345bce1.1623331925.git.sergepetrenko@tarantool.org> <76d19171-2257-35e3-8f07-b1f0323c728f@tarantool.org> Message-ID: <3a61c787-8a92-0191-1565-118915adbfcc@tarantool.org> Date: Fri, 18 Jun 2021 00:00:18 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <76d19171-2257-35e3-8f07-b1f0323c728f@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91C2C07775F13263AD4A2A3BFBC817E640615893838B9A94F00894C459B0CD1B920AB1A9F31198C216A2D283687A7E535AE11324F96DF83CB657EB3FB719050AD X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7AB524098FB2F2222EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637FDB3827A455F08028638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D840ADA9719DBC79ADAE8DF0B6687CEABC117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCF80095D1E57F4578A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FF04B652EEC242312D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE437C869540D2AB0F3DBBCB839D0549ACD8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE3724336BCC0EE1BA82D242C3BD2E3F4C6C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637BC468E7E89D8C5D6EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824A2A04A2ABAA09D25379311020FFC8D4AD7067FEF890167A5414A2D6D9D10D1000 X-C1DE0DAB: 0D63561A33F958A52C4D3F7FBEA5BF13851D797F1171BFD089F5EE617BF0EC7ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D345DE7771146E56B0801BA8D8D6CCB2F0DB3C1D119BB2CD1AEB22E88AB372110459832A9267FF40A3A1D7E09C32AA3244C5B4FBC395EFE36AC4668E621F3677A74A95CA90A1D8AC565FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojo2TEchKDd+eytyf/AZ9xNw== X-Mailru-Sender: 583F1D7ACE8F49BD9DF7A8DAE6E2B08A7335C388A1FBB6A38FC0682AED313CFD5C136FEF95D2D9A1424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 5/7] replication: send latest effective promote in initial join X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 16.06.2021 00:00, Vladislav Shpilevoy пишет: > Thanks for working on this! > > Hm. The patch makes me think why don't we send the Raft > checkpoint on join too? > > Otherwise it might happen that a replica joined, didn't > get the most actual Raft term yet, then the leader died, > and the replica would start elections with term 1. Even if > the latest term was 1000. > > Nothing critical, Raft will probably work, but it could > be an "optimization"? Also it would be consistent with the > libmo state - send all the snapshot data on join. I tried to implement such a patch, but faced the following problem: Unfortunately, we don't have information about replica's version during join, so I can only send raft state based on term > 1. Also, while writing a commit message I understood that this patch doesn't help much. Even if a node joins, but doesn't subscribe to the leader, it will still subscribe to someone else and receive the latest Raft state. After all, our doc states full-mesh is required for Raft to work, so we'll have someone else to subscribe to and receive Raft state from for sure. The patch's small and simple but I think it's not needed. I've made a tiny amendment to this commit though, please find the diff below. > Btw, the limbo state contains a term. And it means > that after join, but before subscribe, the limbo's term > is bigger than raft's term. Even though in the comments > of the limbo we say: > > * It means the limbo's term might be smaller than the raft term, while > * there are ongoing elections, or the leader is already known and this > * instance hasn't read its PROMOTE request yet. During other times the > * limbo and raft are in sync and the terms are the same. > > which means the limbo term should be always <= raft term. > Can this break something? Is it possible to make a test confirming > that we can't send the limbo state before the raft state? I don't think this could break anything. Limbo and Raft terms are not that interconnected now. ================================ diff --git a/src/box/relay.cc b/src/box/relay.cc index 289dea0f3..e05b53d5d 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -408,12 +408,17 @@ relay_initial_join(int fd, uint64_t sync, struct vclock *vclock)         row.sync = sync;         coio_write_xrow(&relay->io, &row); -       /* Send out the latest limbo state. */ -       char body[XROW_SYNCHRO_BODY_LEN_MAX]; -       xrow_encode_synchro(&row, body, &req); -       row.replica_id = req.replica_id; -       row.sync = sync; -       coio_write_xrow(&relay->io, &row); +       /* +        * Send out the latest limbo state. Don't do that when limbo is unused, +        * let the old instances join without trouble. +        */ +       if (req.replica_id != REPLICA_ID_NIL) { +               char body[XROW_SYNCHRO_BODY_LEN_MAX]; +               xrow_encode_synchro(&row, body, &req); +               row.replica_id = req.replica_id; +               row.sync = sync; +               coio_write_xrow(&relay->io, &row); +       }         /* Send read view to the replica. */         engine_join_xc(&ctx, &relay->stream); ================================ -- Serge Petrenko