From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 4AC836E201; Fri, 18 Jun 2021 00:00:11 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 4AC836E201 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1623963611; bh=APP0sFW7HN5qFqRI3dU8WapBs0rm/QurGPOiJ94QZ1Y=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=OP4mc/KplKTtd1VVvqzx9NjiMaSl6/slLHLtdMs/8tiNDxrX2/hHMw3rJ3mpVR+HD MjnynnsyQnswBC/44m19u2B3zmQ8a5MB5xBTTs8tZhKKFeAqhkJkB7rJGbZuKUcg3M 32Jsg8KiVGchwXx8WIrkhbcZcahUAs/vt/eEb9B4= Received: from smtp45.i.mail.ru (smtp45.i.mail.ru [94.100.177.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id AEE806E200 for ; Fri, 18 Jun 2021 00:00:04 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org AEE806E200 Received: by smtp45.i.mail.ru with esmtpa (envelope-from ) id 1ltz7L-0001CI-RE; Fri, 18 Jun 2021 00:00:04 +0300 To: Vladislav Shpilevoy , gorcunov@gmail.com Cc: tarantool-patches@dev.tarantool.org References: <3dc87744-6880-78a3-b77e-3a8811763c39@tarantool.org> Message-ID: <40a4914d-9e87-8bf2-93d6-dc5d9bb2801f@tarantool.org> Date: Fri, 18 Jun 2021 00:00:03 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <3dc87744-6880-78a3-b77e-3a8811763c39@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91C2C07775F13263AD4A2A3BFBC817E640615893838B9A94F00894C459B0CD1B9A0D6D9A1332C93374310342B5C4D695DAE11324F96DF83CB4044E5D25B716709 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7CFDA40FA2326E319EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AAEFEF2B38A4D0058638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D841F31ED58755B8A264AAEF8330E6D731117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC2EE5AD8F952D28FBA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FCB629EEF1311BF91D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE0AC5B80A05675ACD86750DBF4DC650F2D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE3724336BCC0EE1BA8040F9FF01DFDA4A8C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637BC468E7E89D8C5D6EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824A2A04A2ABAA09D25379311020FFC8D4AD7067FEF890167A54D9E43083324F8EF9 X-C1DE0DAB: 0D63561A33F958A56287BCC3E8FF21A2336FCFFE26CC784DBB699AEA36290763D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3429538671E6527D322BA962E91426259EDCABF6E9221D0D3F386CB13551B9AE62E6EC556CFA280EB11D7E09C32AA3244CFA0BB47248558E6DF2E76B8D2C38442DF522A1CF68F4BE05FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojo2TEchKDd+d73XKg2u5xsQ== X-Mailru-Sender: 583F1D7ACE8F49BD9DF7A8DAE6E2B08A7335C388A1FBB6A33BA9374EA7B77F7B77B8DE0C416023C4424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 1/7] replication: always send raft state to subscribers X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 15.06.2021 23:53, Vladislav Shpilevoy пишет: > Thanks for working on this! On 10.06.2021 15:32, Serge Petrenko via > Tarantool-patches wrote: >> Tarantool used to send out raft state on subscribe only when raft was >> enabled. This was a safeguard against partially-upgraded clusters, >> where some nodes had no clue about Raft messages and couldn't handle >> them properly. Actually, Raft state should be sent out always. For >> example, promote bumps Raft term, even when raft is disabled, and >> it's important that everyone in cluster has the same term, for the >> sake of promote at least. So, send out Raft state to every subscriber >> with version >= 2.6.0 (that's when Raft was introduced). Closes #5438 >> --- src/box/box.cc | 11 +-- >> test/replication/gh-5438-raft-state.result | 73 ++++++++++++++++++++ >> test/replication/gh-5438-raft-state.test.lua | 30 ++++++++ > The test still works when I revert box.cc changes. Hi! Thanks for pointing this out! Yep, this happened because of raft broadcasts. When relay is off, it still saves the lates broadcast message from raft, and delivers it as soon as replica reconnects. Thanks to this, I've also fixed raft broadcasts possibly being sent to old instances. This could only happen after the patch which persists the latest raft broadcast and only if raft was turned on for some time. So this wasn't such a big deal for upgrading installations, because they wouldn't turn raft on. Anyway, this issue is fixed as well and I made the test fail without the patch. Here's the diff: ================================================ diff --git a/CMakeLists.txt b/CMakeLists.txt index 1196b65b9..5d72cab13 100644 diff --git a/src/box/relay.cc b/src/box/relay.cc index b1571b361..b47767769 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -760,7 +760,7 @@ relay_subscribe_f(va_list ap)            &relay->relay_pipe, NULL, NULL, cbus_process);      struct relay_is_raft_enabled_msg raft_enabler; -    if (!relay->replica->anon) +    if (!relay->replica->anon && relay->version_id >= version_id(2, 6, 0))          relay_send_is_raft_enabled(relay, &raft_enabler, true);      /* @@ -842,7 +842,7 @@ relay_subscribe_f(va_list ap)          cpipe_push(&relay->tx_pipe, &relay->status_msg.msg);      } -    if (!relay->replica->anon) +    if (!relay->replica->anon && relay->version_id >= version_id(2, 6, 0))          relay_send_is_raft_enabled(relay, &raft_enabler, false);      /* diff --git a/test/replication/gh-5438-raft-state.result b/test/replication/gh-5438-raft-state.result index 7982796a8..6985f026a 100644 --- a/test/replication/gh-5438-raft-state.result +++ b/test/replication/gh-5438-raft-state.result @@ -6,26 +6,6 @@ test_run = require('test_run').new()  --  -- gh-5428 send out Raft state to subscribers, even when Raft is disabled.  -- -box.schema.user.grant('guest', 'replication') - | --- - | ... -test_run:cmd('create server replica with rpl_master=default,\ - script="replication/replica.lua"') - | --- - | - true - | ... -test_run:cmd('start server replica') - | --- - | - true - | ... -test_run:wait_lsn('replica', 'default') - | --- - | ... -test_run:cmd('stop server replica') - | --- - | - true - | ... -  -- Bump Raft term while the replica's offline.  term = box.info.election.term   | --- @@ -41,10 +21,19 @@ test_run:wait_cond(function() return box.info.election.term > term end)   | - true   | ... --- Make sure the replica receives new term on resubscribe. +-- Make sure the replica receives new term on subscribe.  box.cfg{election_mode = 'off'}   | ---   | ... + +box.schema.user.grant('guest', 'replication') + | --- + | ... +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ...  test_run:cmd('start server replica')   | ---   | - true @@ -56,6 +45,7 @@ end)   | ---   | - true   | ... +  -- Cleanup.  box.cfg{election_mode = old_election_mode}   | --- diff --git a/test/replication/gh-5438-raft-state.test.lua b/test/replication/gh-5438-raft-state.test.lua index 179f4b1c9..60c3366c1 100644 --- a/test/replication/gh-5438-raft-state.test.lua +++ b/test/replication/gh-5438-raft-state.test.lua @@ -3,26 +3,24 @@ test_run = require('test_run').new()  --  -- gh-5428 send out Raft state to subscribers, even when Raft is disabled.  -- -box.schema.user.grant('guest', 'replication') -test_run:cmd('create server replica with rpl_master=default,\ - script="replication/replica.lua"') -test_run:cmd('start server replica') -test_run:wait_lsn('replica', 'default') -test_run:cmd('stop server replica') -  -- Bump Raft term while the replica's offline.  term = box.info.election.term  old_election_mode = box.cfg.election_mode  box.cfg{election_mode = 'candidate'}  test_run:wait_cond(function() return box.info.election.term > term end) --- Make sure the replica receives new term on resubscribe. +-- Make sure the replica receives new term on subscribe.  box.cfg{election_mode = 'off'} + +box.schema.user.grant('guest', 'replication') +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"')  test_run:cmd('start server replica')  test_run:wait_cond(function()\      return test_run:eval('replica', 'return box.info.election.term')[1] ==\             box.info.election.term\  end) +  -- Cleanup.  box.cfg{election_mode = old_election_mode}  test_run:cmd('stop server replica') ================================================ -- Serge Petrenko