From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org
Subject: [Tarantool-patches] [PATCH v2 5/5] election: activate raft split vote handling
Date: Thu, 20 Jan 2022 01:43:47 +0100 [thread overview]
Message-ID: <8bee095c452053e33df2cb285e8dbcf2088d1a25.1642639079.git.v.shpilevoy@tarantool.org> (raw)
In-Reply-To: <cover.1642639079.git.v.shpilevoy@tarantool.org>
Raft needs to know cluster size in order to detect and handle
split vote. The patch uses registered server count as cluster
size.
It is not documented nor has a changelog file because this is an
optimization. Can't be observed except in logs or with a watch.
Closes #5285
---
src/box/raft.c | 4 +-
.../election_split_vote_test.lua | 92 +++++++++++++++++++
2 files changed, 95 insertions(+), 1 deletion(-)
create mode 100644 test/replication-luatest/election_split_vote_test.lua
diff --git a/src/box/raft.c b/src/box/raft.c
index 1e360dc88..be6009cc1 100644
--- a/src/box/raft.c
+++ b/src/box/raft.c
@@ -229,7 +229,9 @@ box_raft_update_election_quorum(void)
* be lost.
*/
int quorum = MIN(replication_synchro_quorum, max);
- raft_cfg_election_quorum(box_raft(), quorum);
+ struct raft *raft = box_raft();
+ raft_cfg_election_quorum(raft, quorum);
+ raft_cfg_cluster_size(raft, max);
}
void
diff --git a/test/replication-luatest/election_split_vote_test.lua b/test/replication-luatest/election_split_vote_test.lua
new file mode 100644
index 000000000..f31bfd7f3
--- /dev/null
+++ b/test/replication-luatest/election_split_vote_test.lua
@@ -0,0 +1,92 @@
+local t = require('luatest')
+local cluster = require('test.luatest_helpers.cluster')
+local helpers = require('test.luatest_helpers')
+local wait_timeout = 120
+
+--
+-- gh-5285: split vote is when in the current term there can't be winner of the
+-- leader role. Number of unused votes is not enough for anyone to get the
+-- quorum. It can be detected to speed up the term bump.
+--
+local g = t.group('split-vote')
+
+g.before_each(function()
+ g.cluster = cluster:new({})
+ local node1_uri = helpers.instance_uri('node1')
+ local node2_uri = helpers.instance_uri('node2')
+ local replication = {node1_uri, node2_uri}
+ local box_cfg = {
+ listen = node1_uri,
+ replication = replication,
+ -- To speed up new term when try to elect a first leader.
+ replication_timeout = 0.1,
+ replication_synchro_quorum = 2,
+ election_timeout = 1000000,
+ }
+ g.node1 = g.cluster:build_server({alias = 'node1', box_cfg = box_cfg})
+
+ box_cfg.listen = node2_uri
+ g.node2 = g.cluster:build_server({alias = 'node2', box_cfg = box_cfg})
+
+ g.cluster:add_server(g.node1)
+ g.cluster:add_server(g.node2)
+ g.cluster:start()
+end)
+
+g.after_each(function()
+ g.cluster:drop()
+end)
+
+g.test_split_vote = function(g)
+ -- Stop the replication so as the nodes can't request votes from each other.
+ local node1_repl = g.node1:exec(function()
+ local repl = box.cfg.replication
+ box.cfg{replication = {}}
+ return repl
+ end)
+ local node2_repl = g.node2:exec(function()
+ local repl = box.cfg.replication
+ box.cfg{replication = {}}
+ return repl
+ end)
+
+ -- Both vote for self but don't see the split-vote yet.
+ g.node1:exec(function()
+ box.cfg{election_mode = 'candidate'}
+ end)
+ g.node2:exec(function()
+ box.cfg{election_mode = 'candidate'}
+ end)
+
+ -- Wait for the votes to actually happen.
+ t.helpers.retrying({timeout = wait_timeout}, function()
+ local func = function()
+ return box.info.election.vote == box.info.id
+ end
+ assert(g.node1:exec(func))
+ assert(g.node2:exec(func))
+ end)
+
+ -- Now let the nodes notice the split vote.
+ g.node1:exec(function(repl)
+ box.cfg{replication = repl}
+ end, {node1_repl})
+ g.node2:exec(function(repl)
+ box.cfg{replication = repl}
+ end, {node2_repl})
+
+ t.helpers.retrying({timeout = wait_timeout}, function()
+ local msg = 'split vote is discovered'
+ assert(g.node1:grep_log(msg) or g.node2:grep_log(msg))
+ end)
+
+ -- Ensure a leader is eventually elected. Nothing is broken for good.
+ g.node1:exec(function()
+ box.cfg{election_timeout = 1}
+ end)
+ g.node2:exec(function()
+ box.cfg{election_timeout = 1}
+ end)
+ g.node1:wait_election_leader_found()
+ g.node2:wait_election_leader_found()
+end
--
2.24.3 (Apple Git-128)
next prev parent reply other threads:[~2022-01-20 0:46 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-20 0:43 [Tarantool-patches] [PATCH v2 0/5] Split vote and bugs Vladislav Shpilevoy via Tarantool-patches
2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 1/5] raft: fix crash on election_timeout reconfig Vladislav Shpilevoy via Tarantool-patches
2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 2/5] raft: fix ev_timer.at incorrect usage Vladislav Shpilevoy via Tarantool-patches
2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 3/5] raft: track all votes, even not own Vladislav Shpilevoy via Tarantool-patches
2022-01-20 0:43 ` [Tarantool-patches] [PATCH v2 4/5] raft: introduce split vote detection Vladislav Shpilevoy via Tarantool-patches
2022-01-20 13:22 ` Serge Petrenko via Tarantool-patches
2022-01-20 23:02 ` Vladislav Shpilevoy via Tarantool-patches
2022-01-25 10:17 ` Serge Petrenko via Tarantool-patches
2022-01-20 0:43 ` Vladislav Shpilevoy via Tarantool-patches [this message]
2022-01-25 10:18 ` [Tarantool-patches] [PATCH v2 0/5] Split vote and bugs Serge Petrenko via Tarantool-patches
2022-01-25 22:51 ` Vladislav Shpilevoy via Tarantool-patches
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8bee095c452053e33df2cb285e8dbcf2088d1a25.1642639079.git.v.shpilevoy@tarantool.org \
--to=tarantool-patches@dev.tarantool.org \
--cc=sergepetrenko@tarantool.org \
--cc=v.shpilevoy@tarantool.org \
--subject='Re: [Tarantool-patches] [PATCH v2 5/5] election: activate raft split vote handling' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox