Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org
Subject: [Tarantool-patches] [PATCH 7/8] raft: fix crash on election timeout decrease
Date: Sun, 13 Dec 2020 18:15:29 +0100	[thread overview]
Message-ID: <4d367a7c1a4fe8efc92de40583b5f4843b11b295.1607879643.git.v.shpilevoy@tarantool.org> (raw)
In-Reply-To: <cover.1607879643.git.v.shpilevoy@tarantool.org>

If election timeout was decreased during election to a new value
making the current election expired immediately, it could crash in
libev.

Because it would mean the remaining time until election end became
negative. The negative timeout was passed to libev without any
checks, and there is an assertion, that a timeout should always
be >= 0.

Part of #5303
---
 src/lib/raft/raft.c   |  2 ++
 test/unit/raft.c      | 20 +++++++++++++++++++-
 test/unit/raft.result |  8 +++++---
 3 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/src/lib/raft/raft.c b/src/lib/raft/raft.c
index ab007a462..4f6a5ee5e 100644
--- a/src/lib/raft/raft.c
+++ b/src/lib/raft/raft.c
@@ -895,6 +895,8 @@ raft_cfg_election_timeout(struct raft *raft, double timeout)
 		struct ev_loop *loop = raft_loop();
 		double timeout = raft_ev_timer_remaining(loop, &raft->timer) -
 				 raft->timer.at + raft->election_timeout;
+		if (timeout < 0)
+			timeout = 0;
 		raft_ev_timer_stop(loop, &raft->timer);
 		raft_ev_timer_set(&raft->timer, timeout, timeout);
 		raft_ev_timer_start(loop, &raft->timer);
diff --git a/test/unit/raft.c b/test/unit/raft.c
index b97d9d0aa..2c3935cbf 100644
--- a/test/unit/raft.c
+++ b/test/unit/raft.c
@@ -793,7 +793,7 @@ raft_test_heartbeat(void)
 static void
 raft_test_election_timeout(void)
 {
-	raft_start_test(11);
+	raft_start_test(13);
 	struct raft_node node;
 	raft_node_create(&node);
 
@@ -865,6 +865,24 @@ raft_test_election_timeout(void)
 		"{0: 3}" /* Vclock. */
 	), "re-enter candidate state");
 
+	/* Decrease election timeout to earlier than now. */
+
+	raft_run_for(election_timeout / 2);
+	raft_node_cfg_election_timeout(&node, election_timeout / 4);
+	ts = raft_time();
+	raft_run_next_event();
+
+	ok(raft_time() == ts, "the new timeout acts immediately");
+	ok(raft_node_check_full_state(&node,
+		RAFT_STATE_CANDIDATE /* State. */,
+		0 /* Leader. */,
+		5 /* Term. */,
+		1 /* Vote. */,
+		5 /* Volatile term. */,
+		1 /* Volatile vote. */,
+		"{0: 4}" /* Vclock. */
+	), "re-enter candidate state");
+
 	/*
 	 * Timeout smaller than a millisecond. Election random shift has
 	 * millisecond precision. When timeout is smaller, maximal shift is
diff --git a/test/unit/raft.result b/test/unit/raft.result
index 3fa2682c8..fcd180cfc 100644
--- a/test/unit/raft.result
+++ b/test/unit/raft.result
@@ -148,7 +148,7 @@ ok 7 - subtests
 ok 8 - subtests
 	*** raft_test_heartbeat: done ***
 	*** raft_test_election_timeout ***
-    1..11
+    1..13
     ok 1 - election is started
     ok 2 - enter candidate state
     ok 3 - new election is started
@@ -158,8 +158,10 @@ ok 8 - subtests
     ok 7 - new election timeout is respected
     ok 8 - but not too late
     ok 9 - re-enter candidate state
-    ok 10 - term is bumped, timeout was truly random
-    ok 11 - still candidate
+    ok 10 - the new timeout acts immediately
+    ok 11 - re-enter candidate state
+    ok 12 - term is bumped, timeout was truly random
+    ok 13 - still candidate
 ok 9 - subtests
 	*** raft_test_election_timeout: done ***
 	*** raft_test_election_quorum ***
-- 
2.24.3 (Apple Git-128)

  parent reply	other threads:[~2020-12-13 17:15 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-13 17:15 [Tarantool-patches] [PATCH 0/8] Raft module, part 4 - unit tests Vladislav Shpilevoy
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 1/8] fakesys: fix ev_is_active not working on fake timers Vladislav Shpilevoy
2020-12-15  9:42   ` Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 2/8] fakesys: introduce fakeev_timer_remaining() Vladislav Shpilevoy
2020-12-15  9:43   ` Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 3/8] raft: introduce raft_ev Vladislav Shpilevoy
2020-12-15 10:02   ` Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 4/8] test: introduce raft unit tests Vladislav Shpilevoy
2020-12-13 18:10   ` Vladislav Shpilevoy
2020-12-16 13:03   ` Serge Petrenko
2020-12-17 22:44     ` Vladislav Shpilevoy
2020-12-18  8:17       ` Serge Petrenko
2020-12-20 17:28         ` Vladislav Shpilevoy
2020-12-21  7:36           ` Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 5/8] raft: fix crash when received 0 term message Vladislav Shpilevoy
2020-12-16 13:05   ` Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 6/8] raft: fix ignorance of bad state receipt Vladislav Shpilevoy
2020-12-16 13:06   ` Serge Petrenko
2020-12-13 17:15 ` Vladislav Shpilevoy [this message]
2020-12-16 13:08   ` [Tarantool-patches] [PATCH 7/8] raft: fix crash on election timeout decrease Serge Petrenko
2020-12-13 17:15 ` [Tarantool-patches] [PATCH 8/8] raft: fix crash on death " Vladislav Shpilevoy
2020-12-16 13:10   ` Serge Petrenko
2020-12-21 16:50 ` [Tarantool-patches] [PATCH 0/8] Raft module, part 4 - unit tests Vladislav Shpilevoy
2020-12-21 17:29 ` Vladislav Shpilevoy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4d367a7c1a4fe8efc92de40583b5f4843b11b295.1607879643.git.v.shpilevoy@tarantool.org \
    --to=v.shpilevoy@tarantool.org \
    --cc=sergepetrenko@tarantool.org \
    --cc=tarantool-patches@dev.tarantool.org \
    --subject='Re: [Tarantool-patches] [PATCH 7/8] raft: fix crash on election timeout decrease' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox