From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org> To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org Subject: [Tarantool-patches] [PATCH v2 5/5] election: activate raft split vote handling Date: Thu, 20 Jan 2022 01:43:47 +0100 [thread overview] Message-ID: <8bee095c452053e33df2cb285e8dbcf2088d1a25.1642639079.git.v.shpilevoy@tarantool.org> (raw) In-Reply-To: <cover.1642639079.git.v.shpilevoy@tarantool.org> Raft needs to know cluster size in order to detect and handle split vote. The patch uses registered server count as cluster size. It is not documented nor has a changelog file because this is an optimization. Can't be observed except in logs or with a watch. Closes #5285 --- src/box/raft.c | 4 +- .../election_split_vote_test.lua | 92 +++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 test/replication-luatest/election_split_vote_test.lua diff --git a/src/box/raft.c b/src/box/raft.c index 1e360dc88..be6009cc1 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -229,7 +229,9 @@ box_raft_update_election_quorum(void) * be lost. */ int quorum = MIN(replication_synchro_quorum, max); - raft_cfg_election_quorum(box_raft(), quorum); + struct raft *raft = box_raft(); + raft_cfg_election_quorum(raft, quorum); + raft_cfg_cluster_size(raft, max); } void diff --git a/test/replication-luatest/election_split_vote_test.lua b/test/replication-luatest/election_split_vote_test.lua new file mode 100644 index 000000000..f31bfd7f3 --- /dev/null +++ b/test/replication-luatest/election_split_vote_test.lua @@ -0,0 +1,92 @@ +local t = require('luatest') +local cluster = require('test.luatest_helpers.cluster') +local helpers = require('test.luatest_helpers') +local wait_timeout = 120 + +-- +-- gh-5285: split vote is when in the current term there can't be winner of the +-- leader role. Number of unused votes is not enough for anyone to get the +-- quorum. It can be detected to speed up the term bump. +-- +local g = t.group('split-vote') + +g.before_each(function() + g.cluster = cluster:new({}) + local node1_uri = helpers.instance_uri('node1') + local node2_uri = helpers.instance_uri('node2') + local replication = {node1_uri, node2_uri} + local box_cfg = { + listen = node1_uri, + replication = replication, + -- To speed up new term when try to elect a first leader. + replication_timeout = 0.1, + replication_synchro_quorum = 2, + election_timeout = 1000000, + } + g.node1 = g.cluster:build_server({alias = 'node1', box_cfg = box_cfg}) + + box_cfg.listen = node2_uri + g.node2 = g.cluster:build_server({alias = 'node2', box_cfg = box_cfg}) + + g.cluster:add_server(g.node1) + g.cluster:add_server(g.node2) + g.cluster:start() +end) + +g.after_each(function() + g.cluster:drop() +end) + +g.test_split_vote = function(g) + -- Stop the replication so as the nodes can't request votes from each other. + local node1_repl = g.node1:exec(function() + local repl = box.cfg.replication + box.cfg{replication = {}} + return repl + end) + local node2_repl = g.node2:exec(function() + local repl = box.cfg.replication + box.cfg{replication = {}} + return repl + end) + + -- Both vote for self but don't see the split-vote yet. + g.node1:exec(function() + box.cfg{election_mode = 'candidate'} + end) + g.node2:exec(function() + box.cfg{election_mode = 'candidate'} + end) + + -- Wait for the votes to actually happen. + t.helpers.retrying({timeout = wait_timeout}, function() + local func = function() + return box.info.election.vote == box.info.id + end + assert(g.node1:exec(func)) + assert(g.node2:exec(func)) + end) + + -- Now let the nodes notice the split vote. + g.node1:exec(function(repl) + box.cfg{replication = repl} + end, {node1_repl}) + g.node2:exec(function(repl) + box.cfg{replication = repl} + end, {node2_repl}) + + t.helpers.retrying({timeout = wait_timeout}, function() + local msg = 'split vote is discovered' + assert(g.node1:grep_log(msg) or g.node2:grep_log(msg)) + end) + + -- Ensure a leader is eventually elected. Nothing is broken for good. + g.node1:exec(function() + box.cfg{election_timeout = 1} + end) + g.node2:exec(function() + box.cfg{election_timeout = 1} + end) + g.node1:wait_election_leader_found() + g.node2:wait_election_leader_found() +end -- 2.24.3 (Apple Git-128)
next prev parent reply other threads:[~2022-01-20 0:46 UTC|newest] Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-20 0:43 [Tarantool-patches] [PATCH v2 0/5] Split vote and bugs Vladislav Shpilevoy via Tarantool-patches 2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 1/5] raft: fix crash on election_timeout reconfig Vladislav Shpilevoy via Tarantool-patches 2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 2/5] raft: fix ev_timer.at incorrect usage Vladislav Shpilevoy via Tarantool-patches 2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 3/5] raft: track all votes, even not own Vladislav Shpilevoy via Tarantool-patches 2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 4/5] raft: introduce split vote detection Vladislav Shpilevoy via Tarantool-patches 2022-01-20 13:22 ` Serge Petrenko via Tarantool-patches 2022-01-20 23:02 ` Vladislav Shpilevoy via Tarantool-patches 2022-01-25 10:17 ` Serge Petrenko via Tarantool-patches 2022-01-20 0:43 ` Vladislav Shpilevoy via Tarantool-patches [this message] 2022-01-25 10:18 ` [Tarantool-patches] [PATCH v2 0/5] Split vote and bugs Serge Petrenko via Tarantool-patches 2022-01-25 22:51 ` Vladislav Shpilevoy via Tarantool-patches
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=8bee095c452053e33df2cb285e8dbcf2088d1a25.1642639079.git.v.shpilevoy@tarantool.org \ --to=tarantool-patches@dev.tarantool.org \ --cc=sergepetrenko@tarantool.org \ --cc=v.shpilevoy@tarantool.org \ --subject='Re: [Tarantool-patches] [PATCH v2 5/5] election: activate raft split vote handling' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox