From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 74ACF6E201; Fri, 18 Jun 2021 00:09:12 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 74ACF6E201 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1623964152; bh=LZ2pz5LERk8j009NZ6XySt7Eh8bO/1Vd5lAWWqZgedo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=oA36X/KnJPX2wnvTnUPiWcgZwF2/JIuy4peGjfreaNbtBF47smqaxhQU477Vx0iHf weKUyGGslnQspkc3HRmhOrQhxYJ+8vCr1tzxYt+/VLn0x9JrKSZcyyaTse/RFV17q9 6IlXET/rZqGkjIr8m6OVMKPU9LEV9F5bIwjAFpfs= Received: from smtp46.i.mail.ru (smtp46.i.mail.ru [94.100.177.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id ABF2A6E201 for ; Fri, 18 Jun 2021 00:08:41 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ABF2A6E201 Received: by smtp46.i.mail.ru with esmtpa (envelope-from ) id 1ltzFg-00046o-In; Fri, 18 Jun 2021 00:08:41 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Fri, 18 Jun 2021 00:07:35 +0300 Message-Id: X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD91C2C07775F13263A70584CC5DB91BD99BA7C94740AA1B03F00894C459B0CD1B9981386CB42387DCC9E370651D403E6B97FF79ED123BE076CB8B780B664846F86 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7CC7DB65F82284D26EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006370CE92FB8C11ED3D88638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D84C1F5293B9A9F72B4F1B5820D790FE69117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC3A703B70628EAD7BA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FBDFBBEFFF4125B51D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EEC8105B04EFE076280A5971FBB7557E96D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE3893991AD1F2BFC6A6E0066C2D8992A16C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637A7EFCB0EB5ACB161EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824A2A04A2ABAA09D25379311020FFC8D4AD7067FEF890167A54B45A391BFA2B1234 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C4C7A0BC55FA0FE5FC1E11FAF78CF6E2FB5C747E1821FD0EE10CC5F440826E7FF3B1881A6453793CE9C32612AADDFBE061C61BE10805914D3804EBA3D8E7E5B87ABF8C51168CD8EBDBAB5495298CADBD05DC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34AE88D5ADEBE7C983FDE4CA04619BAE999180D7BDD8FD268A40884AA3AFE5FBDFBBB13A0BADE9E64E1D7E09C32AA3244C70E652C905C79A25CA43168618CC279C4DBEAD0ED6C55A80927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXjymZAW17GbhIOLCK6VnamV X-Mailru-Sender: 583F1D7ACE8F49BD9DF7A8DAE6E2B08A18ACA3F901677141FE617994980E49E5106C612100B9E47E424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v2 1/8] replication: always send raft state to subscribers X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Tarantool used to send out raft state on subscribe only when raft was enabled. This was a safeguard against partially-upgraded clusters, where some nodes had no clue about Raft messages and couldn't handle them properly. Actually, Raft state should be sent out always. For example, promote will be changed to bump Raft term even when Raft is disabled, and it's important that everyone in cluster has the same term for the sake of promote at least. So, send out Raft state to every subscriber with version >= 2.6.0 (that's when Raft was introduced). Do the same for Raft broadcasts. They should be sent only to replicas with version >= 2.6.0 Closes #5438 --- src/box/box.cc | 11 ++-- src/box/relay.cc | 4 +- test/replication/gh-5438-raft-state.result | 63 ++++++++++++++++++++ test/replication/gh-5438-raft-state.test.lua | 28 +++++++++ test/replication/suite.cfg | 1 + 5 files changed, 100 insertions(+), 7 deletions(-) create mode 100644 test/replication/gh-5438-raft-state.result create mode 100644 test/replication/gh-5438-raft-state.test.lua diff --git a/src/box/box.cc b/src/box/box.cc index ab7d983c9..d6a3b4d97 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -82,6 +82,7 @@ #include "msgpack.h" #include "raft.h" #include "trivia/util.h" +#include "version.h" enum { IPROTO_THREADS_MAX = 1000, @@ -2836,13 +2837,13 @@ box_process_subscribe(struct ev_io *io, struct xrow_header *header) tt_uuid_str(&replica_uuid), sio_socketname(io->fd)); say_info("remote vclock %s local vclock %s", vclock_to_string(&replica_clock), vclock_to_string(&vclock)); - if (raft_is_enabled(box_raft())) { + if (replica_version_id >= version_id(2, 6, 0) && !anon) { /* * Send out the current raft state of the instance. Don't do - * that if Raft is disabled. It can be that a part of the - * cluster still contains old versions, which can't handle Raft - * messages. So when it is disabled, its network footprint - * should be 0. + * that if the remote instance is old. It can be that a part of + * the cluster still contains old versions, which can't handle + * Raft messages. Raft's network footprint should be 0 as seen + * by such instances. */ struct raft_request req; box_raft_checkpoint_remote(&req); diff --git a/src/box/relay.cc b/src/box/relay.cc index b1571b361..b47767769 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -760,7 +760,7 @@ relay_subscribe_f(va_list ap) &relay->relay_pipe, NULL, NULL, cbus_process); struct relay_is_raft_enabled_msg raft_enabler; - if (!relay->replica->anon) + if (!relay->replica->anon && relay->version_id >= version_id(2, 6, 0)) relay_send_is_raft_enabled(relay, &raft_enabler, true); /* @@ -842,7 +842,7 @@ relay_subscribe_f(va_list ap) cpipe_push(&relay->tx_pipe, &relay->status_msg.msg); } - if (!relay->replica->anon) + if (!relay->replica->anon && relay->version_id >= version_id(2, 6, 0)) relay_send_is_raft_enabled(relay, &raft_enabler, false); /* diff --git a/test/replication/gh-5438-raft-state.result b/test/replication/gh-5438-raft-state.result new file mode 100644 index 000000000..6985f026a --- /dev/null +++ b/test/replication/gh-5438-raft-state.result @@ -0,0 +1,63 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +-- +-- gh-5428 send out Raft state to subscribers, even when Raft is disabled. +-- +-- Bump Raft term while the replica's offline. +term = box.info.election.term + | --- + | ... +old_election_mode = box.cfg.election_mode + | --- + | ... +box.cfg{election_mode = 'candidate'} + | --- + | ... +test_run:wait_cond(function() return box.info.election.term > term end) + | --- + | - true + | ... + +-- Make sure the replica receives new term on subscribe. +box.cfg{election_mode = 'off'} + | --- + | ... + +box.schema.user.grant('guest', 'replication') + | --- + | ... +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica') + | --- + | - true + | ... +test_run:wait_cond(function()\ + return test_run:eval('replica', 'return box.info.election.term')[1] ==\ + box.info.election.term\ +end) + | --- + | - true + | ... + +-- Cleanup. +box.cfg{election_mode = old_election_mode} + | --- + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +box.schema.user.revoke('guest', 'replication') + | --- + | ... diff --git a/test/replication/gh-5438-raft-state.test.lua b/test/replication/gh-5438-raft-state.test.lua new file mode 100644 index 000000000..60c3366c1 --- /dev/null +++ b/test/replication/gh-5438-raft-state.test.lua @@ -0,0 +1,28 @@ +test_run = require('test_run').new() + +-- +-- gh-5428 send out Raft state to subscribers, even when Raft is disabled. +-- +-- Bump Raft term while the replica's offline. +term = box.info.election.term +old_election_mode = box.cfg.election_mode +box.cfg{election_mode = 'candidate'} +test_run:wait_cond(function() return box.info.election.term > term end) + +-- Make sure the replica receives new term on subscribe. +box.cfg{election_mode = 'off'} + +box.schema.user.grant('guest', 'replication') +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica') +test_run:wait_cond(function()\ + return test_run:eval('replica', 'return box.info.election.term')[1] ==\ + box.info.election.term\ +end) + +-- Cleanup. +box.cfg{election_mode = old_election_mode} +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +box.schema.user.revoke('guest', 'replication') diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg index 3a0a8649f..b0f275526 100644 --- a/test/replication/suite.cfg +++ b/test/replication/suite.cfg @@ -19,6 +19,7 @@ "gh-5213-qsync-applier-order-3.test.lua": {}, "gh-5426-election-on-off.test.lua": {}, "gh-5433-election-restart-recovery.test.lua": {}, + "gh-5438-raft-state.test.lua": {}, "gh-5445-leader-inconsistency.test.lua": {}, "gh-5506-election-on-off.test.lua": {}, "once.test.lua": {}, -- 2.30.1 (Apple Git-130)