From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 1849E6ECE3; Thu, 20 Jan 2022 03:46:21 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1849E6ECE3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1642639581; bh=gIXV27ZSXE8rnZCA8BlYe19w3VICGNc3019DKDcyn4w=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=iCfCZmlkw27FsUtEjIYQ/PoaNmjgavjv7qGf+HYcmtxMtjlrWnLt9i9ilfcC1R4UC mN+BYeZlv0egh9khEpu6xyszstClPb9ZgvjSOwYyty3gRTW8DxpClKDxVqc6ck0Xvn MJEht9wEbFEJZKNjnTGUmoRAkL5ZDPaMHNCQCYbM= Received: from smtpng3.i.mail.ru (smtpng3.i.mail.ru [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id C55F56EC40 for ; Thu, 20 Jan 2022 03:43:53 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org C55F56EC40 Received: by smtpng3.m.smailru.net with esmtpa (envelope-from ) id 1nALYP-0006wT-2h; Thu, 20 Jan 2022 03:43:53 +0300 To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org Date: Thu, 20 Jan 2022 01:43:47 +0100 Message-Id: <8bee095c452053e33df2cb285e8dbcf2088d1a25.1642639079.git.v.shpilevoy@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD98A33503A0B8627DB9805B11801DDEF907693B8FA34D62700182A05F5380850403D17D5E299C71B3FE0957D5D21FAAFE01613BCA1F6120495768106DA2FD4D037 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7586684DD3F99AA20EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006372338AE33E473C9B88638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8FFC445479813D4713D105CB99107A7E4117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6B1CFA6D474D4A6A4089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C0410BD8E874E3D5CE2B0064414D3FEF973CE5A55BA66D47C9C2B6934AE262D3EE7EAB7254005DCED114C52B35DBB74F4E7EAB7254005DCEDA5DF9383870C0FED1E0A4E2319210D9B64D260DF9561598F01A9E91200F654B08D63A13FB2B5C60D8E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D349F4E6BC39AAD02BB9307A7A21887ADAB29CB9DD9C0AA00A4E3750A3D588A0DDA23D9859ABBDB49B11D7E09C32AA3244C49C56C4B1E1382E3173D8131A2517805C86C126E7119A0FE729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojPeoZbWa28mxBFq+L8EG9Gg== X-Mailru-Sender: 689FA8AB762F739339CABD9B3CA9A7D690C30E7EF6B45AF225F05F32B58E51353841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E25FEEDEB644C299C0ED14614B50AE0675 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v2 5/5] election: activate raft split vote handling X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Raft needs to know cluster size in order to detect and handle split vote. The patch uses registered server count as cluster size. It is not documented nor has a changelog file because this is an optimization. Can't be observed except in logs or with a watch. Closes #5285 --- src/box/raft.c | 4 +- .../election_split_vote_test.lua | 92 +++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 test/replication-luatest/election_split_vote_test.lua diff --git a/src/box/raft.c b/src/box/raft.c index 1e360dc88..be6009cc1 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -229,7 +229,9 @@ box_raft_update_election_quorum(void) * be lost. */ int quorum = MIN(replication_synchro_quorum, max); - raft_cfg_election_quorum(box_raft(), quorum); + struct raft *raft = box_raft(); + raft_cfg_election_quorum(raft, quorum); + raft_cfg_cluster_size(raft, max); } void diff --git a/test/replication-luatest/election_split_vote_test.lua b/test/replication-luatest/election_split_vote_test.lua new file mode 100644 index 000000000..f31bfd7f3 --- /dev/null +++ b/test/replication-luatest/election_split_vote_test.lua @@ -0,0 +1,92 @@ +local t = require('luatest') +local cluster = require('test.luatest_helpers.cluster') +local helpers = require('test.luatest_helpers') +local wait_timeout = 120 + +-- +-- gh-5285: split vote is when in the current term there can't be winner of the +-- leader role. Number of unused votes is not enough for anyone to get the +-- quorum. It can be detected to speed up the term bump. +-- +local g = t.group('split-vote') + +g.before_each(function() + g.cluster = cluster:new({}) + local node1_uri = helpers.instance_uri('node1') + local node2_uri = helpers.instance_uri('node2') + local replication = {node1_uri, node2_uri} + local box_cfg = { + listen = node1_uri, + replication = replication, + -- To speed up new term when try to elect a first leader. + replication_timeout = 0.1, + replication_synchro_quorum = 2, + election_timeout = 1000000, + } + g.node1 = g.cluster:build_server({alias = 'node1', box_cfg = box_cfg}) + + box_cfg.listen = node2_uri + g.node2 = g.cluster:build_server({alias = 'node2', box_cfg = box_cfg}) + + g.cluster:add_server(g.node1) + g.cluster:add_server(g.node2) + g.cluster:start() +end) + +g.after_each(function() + g.cluster:drop() +end) + +g.test_split_vote = function(g) + -- Stop the replication so as the nodes can't request votes from each other. + local node1_repl = g.node1:exec(function() + local repl = box.cfg.replication + box.cfg{replication = {}} + return repl + end) + local node2_repl = g.node2:exec(function() + local repl = box.cfg.replication + box.cfg{replication = {}} + return repl + end) + + -- Both vote for self but don't see the split-vote yet. + g.node1:exec(function() + box.cfg{election_mode = 'candidate'} + end) + g.node2:exec(function() + box.cfg{election_mode = 'candidate'} + end) + + -- Wait for the votes to actually happen. + t.helpers.retrying({timeout = wait_timeout}, function() + local func = function() + return box.info.election.vote == box.info.id + end + assert(g.node1:exec(func)) + assert(g.node2:exec(func)) + end) + + -- Now let the nodes notice the split vote. + g.node1:exec(function(repl) + box.cfg{replication = repl} + end, {node1_repl}) + g.node2:exec(function(repl) + box.cfg{replication = repl} + end, {node2_repl}) + + t.helpers.retrying({timeout = wait_timeout}, function() + local msg = 'split vote is discovered' + assert(g.node1:grep_log(msg) or g.node2:grep_log(msg)) + end) + + -- Ensure a leader is eventually elected. Nothing is broken for good. + g.node1:exec(function() + box.cfg{election_timeout = 1} + end) + g.node2:exec(function() + box.cfg{election_timeout = 1} + end) + g.node1:wait_election_leader_found() + g.node2:wait_election_leader_found() +end -- 2.24.3 (Apple Git-128)