From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 068236ECE3; Fri, 21 Jan 2022 03:42:36 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 068236ECE3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1642725756; bh=zqTq/XYlxPLiUmJNnmmS/fff6CH2zkBnLiscO3YViPU=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=rBFrl+L7lcmQ+ISnMqX471igDlM/G/XJJSQVntUb6VKWMrmPn1CqXmXcOYBy+cGoe 9c/BqNcfGito8udcn76UwYHoxJXRmCDd94Osg99ja2AocGKKEbzm/zDXd5AoM7KYcg pXinDcQwaCWP8ufPNUEJz/GngPyFXtX2TXuEQE3M= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 35ABF6ECE3 for ; Fri, 21 Jan 2022 03:42:34 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 35ABF6ECE3 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1nAi0f-00007R-4W; Fri, 21 Jan 2022 03:42:33 +0300 Message-ID: <8c1f46b0-89af-f273-8ec6-19323f50ab18@tarantool.org> Date: Fri, 21 Jan 2022 01:42:32 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9AA78FDF62ECAE61F7D831E2382AA7A6C0481E357CE3D3DF1182A05F53808504041ACC2C8ED11F4E8BF1B12F5A28249C0023BCA8DC465B8CCEB928ED0498B1E1D X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7631CC06FC9F66FEAEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063748D05F5E01EE6C998638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D81A8A5DED572330F23647641DBC5C7C0A117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCF1175FABE1C0F9B6A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD182CC0D3CB04F14752D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6B1CFA6D474D4A6A4089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: 0D63561A33F958A50D85205FF3EA4B820E66A7E7F36B0F5702C0587ADE2CA7F8D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75913C2247C57F08EB410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3438BC6CF312DA7315A29329274C908B8299555E75A7B32114CFE1A4272B583EF5BAB5DFB3A1D2E29D1D7E09C32AA3244C6D032E173DC08CB5EEB945FA02742D38E646F07CC2D4F3D8729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojeO2NaHb0kQsZF2Z9g2BkKQ== X-Mailru-Sender: 689FA8AB762F739339CABD9B3CA9A7D6A402C7AD15606D139B75A04FFD84B9043841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E25FEEDEB644C299C0ED14614B50AE0675 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 2/4] raft: track all votes, even not own X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" I see there is a crash in CI on this commit. I found and fixed the reason. See diff and full new commit below. ==================== diff --git a/src/lib/raft/raft.c b/src/lib/raft/raft.c index 8182e73e3..01b7d469f 100644 --- a/src/lib/raft/raft.c +++ b/src/lib/raft/raft.c @@ -700,8 +700,8 @@ raft_sm_schedule_new_vote(struct raft *raft, uint32_t new_vote) assert(raft->volatile_vote == 0); assert(raft->leader == 0); assert(raft->state == RAFT_STATE_FOLLOWER); - raft->volatile_vote = new_vote; assert(!raft->votes[raft->self].did_vote); + raft->volatile_vote = new_vote; raft_add_vote(raft, raft->self, raft->self); raft_sm_pause_and_dump(raft); /* Nothing visible is changed - no broadcast. */ @@ -1012,8 +1012,8 @@ raft_cfg_instance_id(struct raft *raft, uint32_t instance_id) * more than once during join. Here instance ID is configured when it is * known forever and is safe to use. */ - if (raft->vote != 0) - raft_add_vote(raft, instance_id, raft->vote); + if (raft->volatile_vote != 0) + raft_add_vote(raft, instance_id, raft->volatile_vote); } void diff --git a/test/unit/raft.c b/test/unit/raft.c index 9f18f4d8e..1a3c81c64 100644 --- a/test/unit/raft.c +++ b/test/unit/raft.c @@ -1497,10 +1497,75 @@ raft_test_promote_restore(void) raft_finish_test(); } +static void +raft_test_bump_term_before_cfg() +{ + raft_start_test(6); + struct raft_node node; + raft_node_create(&node); + /* + * Term bump is started between recovery and instance ID configuration + * but WAL write is not finished yet. + */ + raft_run_next_event(); + ok(raft_node_check_full_state(&node, + RAFT_STATE_CANDIDATE /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 2 /* Volatile term. */, + 1 /* Volatile vote. */, + "{0: 1}" /* Vclock. */ + ), "new term is started with vote for self"); + + raft_node_stop(&node); + raft_node_recover(&node); + ok(raft_node_check_full_state(&node, + RAFT_STATE_FOLLOWER /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 2 /* Volatile term. */, + 1 /* Volatile vote. */, + NULL /* Vclock. */ + ), "recovered"); + is(node.raft.self, 0, "instance id is unknown"); + + raft_node_block(&node); + is(raft_node_send_follower(&node, + 3 /* Term. */, + 2 /* Source. */ + ), 0, "bump term externally"); + ok(raft_node_check_full_state(&node, + RAFT_STATE_FOLLOWER /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 3 /* Volatile term. */, + 0 /* Volatile vote. */, + NULL /* Vclock. */ + ), "term write is in progress"); + + raft_node_cfg(&node); + raft_node_unblock(&node); + ok(raft_node_check_full_state(&node, + RAFT_STATE_CANDIDATE /* State. */, + 0 /* Leader. */, + 3 /* Term. */, + 1 /* Vote. */, + 3 /* Volatile term. */, + 1 /* Volatile vote. */, + "{0: 3}" /* Vclock. */ + ), "started new term"); + + raft_node_destroy(&node); + raft_finish_test(); +} + static int main_f(va_list ap) { - raft_start_test(14); + raft_start_test(15); (void) ap; fakeev_init(); @@ -1519,6 +1584,7 @@ main_f(va_list ap) raft_test_enable_disable(); raft_test_too_long_wal_write(); raft_test_promote_restore(); + raft_test_bump_term_before_cfg(); fakeev_free(); diff --git a/test/unit/raft_test_utils.c b/test/unit/raft_test_utils.c index b5754cf78..daf43797f 100644 --- a/test/unit/raft_test_utils.c +++ b/test/unit/raft_test_utils.c @@ -224,13 +224,18 @@ raft_node_check_full_state(const struct raft_node *node, enum raft_state state, { assert(raft_node_is_started(node)); const struct raft *raft = &node->raft; - struct vclock v; - raft_vclock_from_string(&v, vclock); + if (vclock != NULL) { + struct vclock v; + raft_vclock_from_string(&v, vclock); + if (vclock_compare(&v, raft->vclock) != 0) + return false; + } else if (raft->vclock != NULL) { + return false; + } return raft->state == state && raft->leader == leader && raft->term == term && raft->vote == vote && raft->volatile_term == volatile_term && - raft->volatile_vote == volatile_vote && - vclock_compare(&v, raft->vclock) == 0; + raft->volatile_vote == volatile_vote; } bool @@ -342,6 +347,13 @@ raft_node_stop(struct raft_node *node) void raft_node_start(struct raft_node *node) +{ + raft_node_recover(node); + raft_node_cfg(node); +} + +void +raft_node_recover(struct raft_node *node) { assert(!raft_node_is_started(node)); @@ -358,6 +370,13 @@ raft_node_start(struct raft_node *node) for (int i = 0; i < node->journal.size; ++i) raft_process_recovery(&node->raft, &node->journal.rows[i]); + raft_run_async_work(); +} + +void +raft_node_cfg(struct raft_node *node) +{ + assert(raft_node_is_started(node)); raft_cfg_is_enabled(&node->raft, node->cfg_is_enabled); raft_cfg_is_candidate(&node->raft, node->cfg_is_candidate); diff --git a/test/unit/raft_test_utils.h b/test/unit/raft_test_utils.h index c68dc3b22..8d400923e 100644 --- a/test/unit/raft_test_utils.h +++ b/test/unit/raft_test_utils.h @@ -200,6 +200,14 @@ raft_node_stop(struct raft_node *node); void raft_node_start(struct raft_node *node); +/** Start the node but not configure it yet. Only recover. */ +void +raft_node_recover(struct raft_node *node); + +/** Apply the entire Raft config to a started. */ +void +raft_node_cfg(struct raft_node *node); + /** Block async work execution. */ void raft_node_block(struct raft_node *node); ==================== Full new commit: ================================================== raft: track all votes, even not own To detect split vote a node needs to see that number of free votes is not enough for anyone to win even if it gets them all. Hence every node needs to count votes for all other nodes. The patch makes raft store votes in a bit more complicated manner than a simple bitmap for just the current instance. Part of #5285 diff --git a/src/lib/raft/raft.c b/src/lib/raft/raft.c index d6a849503..01b7d469f 100644 --- a/src/lib/raft/raft.c +++ b/src/lib/raft/raft.c @@ -152,6 +152,16 @@ raft_can_vote_for(const struct raft *raft, const struct vclock *v) return cmp == 0 || cmp == 1; } +static inline void +raft_add_vote(struct raft *raft, int src, int dst) +{ + struct raft_vote *v = &raft->votes[src]; + if (v->did_vote) + return; + v->did_vote = true; + ++raft->votes[dst].count; +} + /** Schedule broadcast of the complete Raft state to all the followers. */ static void raft_schedule_broadcast(struct raft *raft); @@ -279,6 +289,12 @@ void raft_process_recovery(struct raft *raft, const struct raft_msg *req) { say_verbose("RAFT: recover %s", raft_msg_to_string(req)); + /* + * Instance ID is not unknown until recovery ends. Because apparently it + * can change during join. In Raft it is set only one time when recovery + * ends for good. + */ + assert(raft->self == 0); if (req->term != 0) { raft->term = req->term; raft->volatile_term = req->term; @@ -335,6 +351,8 @@ raft_process_msg(struct raft *raft, const struct raft_msg *req, uint32_t source) * persisted long time ago and still broadcasted. Or a vote response. */ if (req->vote != 0) { + raft_add_vote(raft, source, req->vote); + switch (raft->state) { case RAFT_STATE_FOLLOWER: case RAFT_STATE_LEADER: @@ -395,11 +413,10 @@ raft_process_msg(struct raft *raft, const struct raft_msg *req, uint32_t source) * and now was answered by some other instance. */ assert(raft->volatile_vote == raft->self); - bool was_set = bit_set(&raft->vote_mask, source); - raft->vote_count += !was_set; - if (raft->vote_count < raft->election_quorum) { + int vote_count = raft_vote_count(raft); + if (vote_count < raft->election_quorum) { say_info("RAFT: accepted vote for self, vote " - "count is %d/%d", raft->vote_count, + "count is %d/%d", vote_count, raft->election_quorum); break; } @@ -640,13 +657,11 @@ raft_sm_become_candidate(struct raft *raft) assert(raft->state == RAFT_STATE_FOLLOWER); assert(raft->leader == 0); assert(raft->vote == raft->self); + assert(raft_vote_count(raft) >= 1); assert(raft->is_candidate); assert(!raft->is_write_in_progress); assert(raft->election_quorum > 1); raft->state = RAFT_STATE_CANDIDATE; - raft->vote_count = 1; - raft->vote_mask = 0; - bit_set(&raft->vote_mask, raft->self); raft_sm_wait_election_end(raft); /* State is visible and it is changed - broadcast. */ raft_schedule_broadcast(raft); @@ -663,6 +678,7 @@ raft_sm_schedule_new_term(struct raft *raft, uint64_t new_term) raft->volatile_vote = 0; raft->leader = 0; raft->state = RAFT_STATE_FOLLOWER; + memset(raft->votes, 0, sizeof(raft->votes)); /* * The instance could be promoted for the previous term. But promotion * has no effect on following terms. @@ -684,7 +700,9 @@ raft_sm_schedule_new_vote(struct raft *raft, uint32_t new_vote) assert(raft->volatile_vote == 0); assert(raft->leader == 0); assert(raft->state == RAFT_STATE_FOLLOWER); + assert(!raft->votes[raft->self].did_vote); raft->volatile_vote = new_vote; + raft_add_vote(raft, raft->self, raft->self); raft_sm_pause_and_dump(raft); /* Nothing visible is changed - no broadcast. */ } @@ -957,7 +975,7 @@ raft_cfg_election_quorum(struct raft *raft, int election_quorum) assert(election_quorum > 0); raft->election_quorum = election_quorum; if (raft->state == RAFT_STATE_CANDIDATE && - raft->vote_count >= raft->election_quorum) + raft_vote_count(raft) >= raft->election_quorum) raft_sm_become_leader(raft); } @@ -989,6 +1007,13 @@ raft_cfg_instance_id(struct raft *raft, uint32_t instance_id) assert(raft->self == 0); assert(instance_id != 0); raft->self = instance_id; + /* + * Couldn't do that reliably during recovery. Instance ID can change + * more than once during join. Here instance ID is configured when it is + * known forever and is safe to use. + */ + if (raft->volatile_vote != 0) + raft_add_vote(raft, instance_id, raft->volatile_vote); } void diff --git a/src/lib/raft/raft.h b/src/lib/raft/raft.h index e844c2d07..1d064bef0 100644 --- a/src/lib/raft/raft.h +++ b/src/lib/raft/raft.h @@ -138,6 +138,14 @@ struct raft_vtab { raft_schedule_async_f schedule_async; }; +/** Vote descriptor of a single node. */ +struct raft_vote { + /** Whether the node voted for anybody. */ + bool did_vote; + /** How many votes the node got in the current term. */ + int count; +}; + struct raft { /** Instance ID of this node. */ uint32_t self; @@ -189,13 +197,8 @@ struct raft { */ uint64_t term; uint32_t vote; - /** - * Bit 1 on position N means that a vote from instance with ID = N was - * obtained. - */ - vclock_map_t vote_mask; - /** Number of votes for this instance. Valid only in candidate state. */ - int vote_count; + /** Statistics which node voted for who. */ + struct raft_vote votes[VCLOCK_MAX]; /** Number of votes necessary for successful election. */ int election_quorum; /** @@ -243,6 +246,13 @@ raft_is_enabled(const struct raft *raft) return raft->is_enabled; } +/** Number of votes for self. */ +static inline int +raft_vote_count(const struct raft *raft) +{ + return raft->votes[raft->self].count; +} + /** Process a raft entry stored in WAL/snapshot. */ void raft_process_recovery(struct raft *raft, const struct raft_msg *req); diff --git a/test/unit/raft.c b/test/unit/raft.c index 520b94466..1a3c81c64 100644 --- a/test/unit/raft.c +++ b/test/unit/raft.c @@ -70,7 +70,7 @@ raft_test_leader_election(void) 1 /* Volatile vote. */, "{0: 1}" /* Vclock. */ ), "elections with a new term"); - is(node.raft.vote_count, 1, "single vote for self"); + is(raft_vote_count(&node.raft), 1, "single vote for self"); ok(node.update_count > 0, "trigger worked"); node.update_count = 0; @@ -100,7 +100,7 @@ raft_test_leader_election(void) 1 /* Vote. */, 2 /* Source. */ ), 0, "vote response from 2"); - is(node.raft.vote_count, 2, "2 votes - 1 self and 1 foreign"); + is(raft_vote_count(&node.raft), 2, "2 votes - 1 self and 1 foreign"); ok(!node.has_work, "no work to do - not enough votes yet"); raft_run_for(node.cfg_election_timeout / 2); @@ -115,7 +115,7 @@ raft_test_leader_election(void) 1 /* Vote. */, 3 /* Source. */ ), 0, "vote response from 3"); - is(node.raft.vote_count, 3, "2 votes - 1 self and 2 foreign"); + is(raft_vote_count(&node.raft), 3, "2 votes - 1 self and 2 foreign"); is(node.raft.state, RAFT_STATE_LEADER, "became leader"); ok(node.update_count > 0, "trigger worked"); node.update_count = 0; @@ -142,7 +142,7 @@ raft_test_leader_election(void) static void raft_test_recovery(void) { - raft_start_test(12); + raft_start_test(13); struct raft_msg msg; struct raft_node node; raft_node_create(&node); @@ -224,6 +224,8 @@ raft_test_recovery(void) "{0: 1}" /* Vclock. */ ), "restart always as a follower"); + is(raft_vote_count(&node.raft), 1, "vote count is restored correctly"); + raft_checkpoint_remote(&node.raft, &msg); ok(raft_msg_check(&msg, RAFT_STATE_FOLLOWER /* State. */, @@ -383,7 +385,7 @@ raft_test_vote_skip(void) 1 /* Vote. */, 2 /* Source. */ ), 0, "message is accepted"); - is(node.raft.vote_count, 1, "but ignored - too old term"); + is(raft_vote_count(&node.raft), 1, "but ignored - too old term"); /* Competing vote requests are skipped. */ @@ -392,7 +394,8 @@ raft_test_vote_skip(void) 3 /* Vote. */, 2 /* Source. */ ), 0, "message is accepted"); - is(node.raft.vote_count, 1, "but ignored - vote not for this node"); + is(raft_vote_count(&node.raft), 1, + "but ignored - vote not for this node"); is(node.raft.state, RAFT_STATE_CANDIDATE, "this node does not give up"); /* Vote requests are ignored when node is disabled. */ @@ -1071,7 +1074,7 @@ raft_test_election_quorum(void) 1 /* Vote. */, 2 /* Source. */ ), 0, "send vote response from second node"); - is(node.raft.vote_count, 2, "vote is accepted"); + is(raft_vote_count(&node.raft), 2, "vote is accepted"); is(node.raft.state, RAFT_STATE_CANDIDATE, "but still candidate"); raft_node_cfg_election_quorum(&node, 2); @@ -1494,10 +1497,75 @@ raft_test_promote_restore(void) raft_finish_test(); } +static void +raft_test_bump_term_before_cfg() +{ + raft_start_test(6); + struct raft_node node; + raft_node_create(&node); + /* + * Term bump is started between recovery and instance ID configuration + * but WAL write is not finished yet. + */ + raft_run_next_event(); + ok(raft_node_check_full_state(&node, + RAFT_STATE_CANDIDATE /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 2 /* Volatile term. */, + 1 /* Volatile vote. */, + "{0: 1}" /* Vclock. */ + ), "new term is started with vote for self"); + + raft_node_stop(&node); + raft_node_recover(&node); + ok(raft_node_check_full_state(&node, + RAFT_STATE_FOLLOWER /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 2 /* Volatile term. */, + 1 /* Volatile vote. */, + NULL /* Vclock. */ + ), "recovered"); + is(node.raft.self, 0, "instance id is unknown"); + + raft_node_block(&node); + is(raft_node_send_follower(&node, + 3 /* Term. */, + 2 /* Source. */ + ), 0, "bump term externally"); + ok(raft_node_check_full_state(&node, + RAFT_STATE_FOLLOWER /* State. */, + 0 /* Leader. */, + 2 /* Term. */, + 1 /* Vote. */, + 3 /* Volatile term. */, + 0 /* Volatile vote. */, + NULL /* Vclock. */ + ), "term write is in progress"); + + raft_node_cfg(&node); + raft_node_unblock(&node); + ok(raft_node_check_full_state(&node, + RAFT_STATE_CANDIDATE /* State. */, + 0 /* Leader. */, + 3 /* Term. */, + 1 /* Vote. */, + 3 /* Volatile term. */, + 1 /* Volatile vote. */, + "{0: 3}" /* Vclock. */ + ), "started new term"); + + raft_node_destroy(&node); + raft_finish_test(); +} + static int main_f(va_list ap) { - raft_start_test(14); + raft_start_test(15); (void) ap; fakeev_init(); @@ -1516,6 +1584,7 @@ main_f(va_list ap) raft_test_enable_disable(); raft_test_too_long_wal_write(); raft_test_promote_restore(); + raft_test_bump_term_before_cfg(); fakeev_free(); diff --git a/test/unit/raft.result b/test/unit/raft.result index 764d82969..5e63e523d 100644 --- a/test/unit/raft.result +++ b/test/unit/raft.result @@ -1,5 +1,5 @@ *** main_f *** -1..14 +1..15 *** raft_test_leader_election *** 1..24 ok 1 - 1 pending message at start @@ -29,7 +29,7 @@ ok 1 - subtests *** raft_test_leader_election: done *** *** raft_test_recovery *** - 1..12 + 1..13 ok 1 - became candidate ok 2 - remote checkpoint of a candidate ok 3 - local checkpoint of a candidate @@ -40,8 +40,9 @@ ok 1 - subtests ok 8 - remote checkpoint of a leader ok 9 - local checkpoint of a leader ok 10 - restart always as a follower - ok 11 - remote checkpoint of a leader - ok 12 - local checkpoint of a leader + ok 11 - vote count is restored correctly + ok 12 - remote checkpoint of a leader + ok 13 - local checkpoint of a leader ok 2 - subtests *** raft_test_recovery: done *** *** raft_test_bad_msg *** @@ -261,4 +262,14 @@ ok 13 - subtests ok 12 - not a candidate ok 14 - subtests *** raft_test_promote_restore: done *** + *** raft_test_bump_term_before_cfg *** + 1..6 + ok 1 - new term is started with vote for self + ok 2 - recovered + ok 3 - instance id is unknown + ok 4 - bump term externally + ok 5 - term write is in progress + ok 6 - started new term +ok 15 - subtests + *** raft_test_bump_term_before_cfg: done *** *** main_f: done *** diff --git a/test/unit/raft_test_utils.c b/test/unit/raft_test_utils.c index b5754cf78..daf43797f 100644 --- a/test/unit/raft_test_utils.c +++ b/test/unit/raft_test_utils.c @@ -224,13 +224,18 @@ raft_node_check_full_state(const struct raft_node *node, enum raft_state state, { assert(raft_node_is_started(node)); const struct raft *raft = &node->raft; - struct vclock v; - raft_vclock_from_string(&v, vclock); + if (vclock != NULL) { + struct vclock v; + raft_vclock_from_string(&v, vclock); + if (vclock_compare(&v, raft->vclock) != 0) + return false; + } else if (raft->vclock != NULL) { + return false; + } return raft->state == state && raft->leader == leader && raft->term == term && raft->vote == vote && raft->volatile_term == volatile_term && - raft->volatile_vote == volatile_vote && - vclock_compare(&v, raft->vclock) == 0; + raft->volatile_vote == volatile_vote; } bool @@ -342,6 +347,13 @@ raft_node_stop(struct raft_node *node) void raft_node_start(struct raft_node *node) +{ + raft_node_recover(node); + raft_node_cfg(node); +} + +void +raft_node_recover(struct raft_node *node) { assert(!raft_node_is_started(node)); @@ -358,6 +370,13 @@ raft_node_start(struct raft_node *node) for (int i = 0; i < node->journal.size; ++i) raft_process_recovery(&node->raft, &node->journal.rows[i]); + raft_run_async_work(); +} + +void +raft_node_cfg(struct raft_node *node) +{ + assert(raft_node_is_started(node)); raft_cfg_is_enabled(&node->raft, node->cfg_is_enabled); raft_cfg_is_candidate(&node->raft, node->cfg_is_candidate); diff --git a/test/unit/raft_test_utils.h b/test/unit/raft_test_utils.h index c68dc3b22..8d400923e 100644 --- a/test/unit/raft_test_utils.h +++ b/test/unit/raft_test_utils.h @@ -200,6 +200,14 @@ raft_node_stop(struct raft_node *node); void raft_node_start(struct raft_node *node); +/** Start the node but not configure it yet. Only recover. */ +void +raft_node_recover(struct raft_node *node); + +/** Apply the entire Raft config to a started. */ +void +raft_node_cfg(struct raft_node *node); + /** Block async work execution. */ void raft_node_block(struct raft_node *node);