From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 02A266EC5E; Mon, 12 Apr 2021 22:44:29 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 02A266EC5E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1618256669; bh=YOGfUOWxy2dX2U7KtcYAHo4WaY/hThTtZwkNCZGzVoo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=wf3jGojiYjSA0b+1Bw4HB3GltLTMLqlemBje+VU6otC4I4w1LTs5vrkvPAbC0nxK8 kE+cGbe/Y4dhmHc/kyBDjYe9pKcK0ZQAcVTITjwgCaFXh9Xu4ABoYdOvGywU5xf56i W8iZaumUBi/pQSwhnkFPA7/1346mjbvabuEP/nvU= Received: from smtp53.i.mail.ru (smtp53.i.mail.ru [94.100.177.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 215986BD23 for ; Mon, 12 Apr 2021 22:40:32 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 215986BD23 Received: by smtp53.i.mail.ru with esmtpa (envelope-from ) id 1lW2QB-0008Az-QE; Mon, 12 Apr 2021 22:40:32 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Mon, 12 Apr 2021 22:40:21 +0300 Message-Id: <77f6f4be4b6c2c2b81f4fbe1e292062eca63361b.1618256019.git.sergepetrenko@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92FFCB8E6708E748094FADAEB10E66ADA4C48BE3C291E66DA182A05F538085040303927C3E0D7D9EEDBB9714E127AF3E8BB068D18B2193177DD4F14FF4F5035D1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE70FEF15FEF2CD350AEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637968EC5F77C2942FE8638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2B6E3B7F54587F966E38E2C7D90F9CBC6EABCB1FA743BAAB4D2E47CDBA5A96583C09775C1D3CA48CF4964A708C60C975A117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE7D38CA8945C5827839FA2833FD35BB23DF004C90652538430302FCEF25BFAB3454AD6D5ED66289B5278DA827A17800CE7B08C29F7500CCD18D32BA5DBAC0009BE395957E7521B51C20BC6067A898B09E4090A508E0FED6299176DF2183F8FC7C0259A6D7820E4F02ECD04E86FAF290E2D7E9C4E3C761E06A71DD303D21008E298D5E8D9A59859A8B6B372FE9A2E580EFC725E5C173C3A84C30584FF81F342DA0735872C767BF85DA2F004C90652538430E4A6367B16DE6309 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824A2A04A2ABAA09D25379311020FFC8D4ADF048BB164F7B76AFD6C08A96310D0B9D X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8183A4AFAF3EA6BDC44C234C8B12C006B7AE4F16467A2949FE9DAF9FC8C8AE39111CDDCB9E7E9BB5F36B1881A6453793CE9C32612AADDFBE061C61BE10805914D3804EBA3D8E7E5B87ABF8C51168CD8EBDB63AF70AF8205D7DCDC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34FACF503ACC83041BAF4711F157DF1D7497E128961E9DB1C6F7A38FDCCC3584C71E1C4E05AD7E7CA81D7E09C32AA3244C4DFC82513044CE4B43EC162D0CAAFE7FF522A1CF68F4BE05927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojq8JA+pXcDunbHGYg4xYGuw== X-Mailru-Sender: 583F1D7ACE8F49BDD2846D59FC20E9F8AFCB052CDF231E7B602344A80C1E63841CF5646E804CED55424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v2 8/9] Support manual elections in `box.ctl.clear_synchro_queue()` X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This patch adds support for manual elections from `box.ctl.clear_synchro_queue()`. When an instance is in `election_mode='manual'`, calling `clear_synchro_queue()` will make it start a new election round. Follow-up #5445 Part of #3055 @TarantoolBot document Title: describe election_mode='manual' Manual election mode is introduced. It may be used when the user wants to control which instance is the leader explicitly instead of relying on Raft election algorithm. When an instance is configured with `election_mode='manual'`, it behaves as follows: 1) By default, the instance acts like a voter: it is read-only and may vote for other instances that are candidates. 2) Once `box.ctl.clear_synchro_queue()` is called, the instance becomes a candidate and starts a new election round. If the instance wins the elections, it remains leader, but won't participate in any new elections. --- src/box/box.cc | 74 +++++++++++++++++++++++++++++++++++-- src/box/errcode.h | 3 ++ src/box/raft.c | 25 ++++++++++++- src/box/raft.h | 3 ++ src/lib/raft/raft.c | 12 +++++- src/lib/raft/raft.h | 2 +- test/box/error.result | 3 ++ test/unit/raft_test_utils.c | 4 +- 8 files changed, 116 insertions(+), 10 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index b77b0a08d..dc7f434e4 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -1161,7 +1161,8 @@ box_set_election_mode(void) if (mode == ELECTION_MODE_INVALID) return -1; box_election_mode = mode; - raft_cfg_is_candidate(box_raft(), mode == ELECTION_MODE_CANDIDATE); + raft_cfg_is_candidate(box_raft(), mode == ELECTION_MODE_CANDIDATE, + true); raft_cfg_is_enabled(box_raft(), mode != ELECTION_MODE_OFF); return 0; } @@ -1525,12 +1526,77 @@ box_clear_synchro_queue(bool try_wait) if (!is_box_configured || raft_source_term(box_raft(), instance_id) == box_raft()->term) return 0; + + bool run_elections = false; + + switch (box_election_mode) { + case ELECTION_MODE_OFF: + break; + case ELECTION_MODE_VOTER: + diag_set(ClientError, ER_UNSUPPORTED, "election_mode='voter'", + "manual elections"); + return -1; + case ELECTION_MODE_MANUAL: + /* + * Even if this instance is already a leader, allow to bump term + * manually once again. + */ + assert(box_raft()->state != RAFT_STATE_CANDIDATE); + if (box_raft()->state == RAFT_STATE_LEADER) { + diag_set(ClientError, ER_ALREADY_LEADER); + return -1; + } + run_elections = true; + try_wait = false; + break; + case ELECTION_MODE_CANDIDATE: + /* + * Leader elections are enabled, and this instance is allowed to + * promote only if it's already an elected leader. No manual + * elections. + */ + if (box_raft()->state != RAFT_STATE_LEADER) { + diag_set(ClientError, ER_UNSUPPORTED, "election_mode=" + "'candidate'", "manual elections"); + return -1; + } + break; + default: + unreachable(); + } + uint32_t former_leader_id = txn_limbo.owner_id; int64_t wait_lsn = txn_limbo.confirmed_lsn; int rc = 0; int quorum = replication_synchro_quorum; in_clear_synchro_queue = true; + if (run_elections) { + /* + * Make this instance a candidate and run until some leader, not + * necessarily this instance, emerges. + */ + raft_cfg_is_candidate(box_raft(), true, false); + /* + * Trigger new elections without waiting for an old leader to + * disappear. + */ + raft_new_term(box_raft()); + box_raft_wait_leader_found(); + raft_cfg_is_candidate(box_raft(), false, false); + if (!box_raft()->is_enabled) { + diag_set(ClientError, ER_RAFT_DISABLED); + in_clear_synchro_queue = false; + return -1; + } + if (box_raft()->state != RAFT_STATE_LEADER) { + diag_set(ClientError, ER_INTERFERING_PROMOTE, + box_raft()->leader); + in_clear_synchro_queue = false; + return -1; + } + } + if (txn_limbo_is_empty(&txn_limbo)) goto promote; @@ -1548,10 +1614,10 @@ box_clear_synchro_queue(bool try_wait) * transactions. Exit in case someone did that for us. */ if (former_leader_id != txn_limbo.owner_id) { - //TODO: error once we see someone else became the leader - // already. + diag_set(ClientError, ER_INTERFERING_PROMOTE, + txn_limbo.owner_id); in_clear_synchro_queue = false; - return 0; + return -1; } } diff --git a/src/box/errcode.h b/src/box/errcode.h index 56573688e..e5c9f3b09 100644 --- a/src/box/errcode.h +++ b/src/box/errcode.h @@ -275,6 +275,9 @@ struct errcode_record { /*220 */_(ER_TOO_EARLY_SUBSCRIBE, "Can't subscribe non-anonymous replica %s until join is done") \ /*221 */_(ER_SQL_CANT_ADD_AUTOINC, "Can't add AUTOINCREMENT: space %s can't feature more than one AUTOINCREMENT field") \ /*222 */_(ER_QUORUM_WAIT, "Couldn't wait for quorum %d: %s") \ + /*223 */_(ER_INTERFERING_PROMOTE, "Instance with replica id %u was promoted first") \ + /*224 */_(ER_RAFT_DISABLED, "Elections were turned off while running box.ctl.promote()")\ + /*225 */_(ER_ALREADY_LEADER, "Can't promote an existing leader")\ /* * !IMPORTANT! Please follow instructions at start of the file diff --git a/src/box/raft.c b/src/box/raft.c index 285dbe4fd..47d4fd56d 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -95,7 +95,8 @@ box_raft_update_synchro_queue(struct raft *raft) * manually. In this case the call below will exit immediately and we'll * simply log a warning. */ - if (raft->state == RAFT_STATE_LEADER) { + if (raft->state == RAFT_STATE_LEADER && + box_election_mode != ELECTION_MODE_MANUAL) { int rc = 0; uint32_t errcode = 0; do { @@ -336,6 +337,28 @@ fail: panic("Could not write a raft request to WAL\n"); } +static int +box_raft_wait_leader_found_trig(struct trigger *trig, void *event) +{ + struct raft *raft = (struct raft *)event; + assert(raft == box_raft()); + struct fiber *waiter = (struct fiber *)trig->data; + if (raft->leader != REPLICA_ID_NIL || !raft->is_enabled) + fiber_wakeup(waiter); + return 0; +} + +void +box_raft_wait_leader_found(void) +{ + struct trigger trig; + trigger_create(&trig, box_raft_wait_leader_found_trig, fiber(), NULL); + raft_on_update(box_raft(), &trig); + fiber_yield(); + assert(box_raft()->leader != REPLICA_ID_NIL || !box_raft()->is_enabled); + trigger_clear(&trig); +} + void box_raft_init(void) { diff --git a/src/box/raft.h b/src/box/raft.h index 15f4e80d9..8fce423e1 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -97,6 +97,9 @@ box_raft_checkpoint_remote(struct raft_request *req); int box_raft_process(struct raft_request *req, uint32_t source); +void +box_raft_wait_leader_found(); + void box_raft_init(void); diff --git a/src/lib/raft/raft.c b/src/lib/raft/raft.c index e9ce8cade..7b77e05ea 100644 --- a/src/lib/raft/raft.c +++ b/src/lib/raft/raft.c @@ -846,7 +846,7 @@ raft_cfg_is_enabled(struct raft *raft, bool is_enabled) } void -raft_cfg_is_candidate(struct raft *raft, bool is_candidate) +raft_cfg_is_candidate(struct raft *raft, bool is_candidate, bool demote) { bool old_is_candidate = raft->is_candidate; raft->is_cfg_candidate = is_candidate; @@ -874,8 +874,16 @@ raft_cfg_is_candidate(struct raft *raft, bool is_candidate) raft_ev_timer_stop(raft_loop(), &raft->timer); } if (raft->state != RAFT_STATE_FOLLOWER) { - if (raft->state == RAFT_STATE_LEADER) + if (raft->state == RAFT_STATE_LEADER) { + if (!demote) { + /* + * Remain leader until someone + * triggers new elections. + */ + return; + } raft->leader = 0; + } raft->state = RAFT_STATE_FOLLOWER; /* State is visible and changed - broadcast. */ raft_schedule_broadcast(raft); diff --git a/src/lib/raft/raft.h b/src/lib/raft/raft.h index 40c8630e9..3526460af 100644 --- a/src/lib/raft/raft.h +++ b/src/lib/raft/raft.h @@ -325,7 +325,7 @@ raft_cfg_is_enabled(struct raft *raft, bool is_enabled); * the node still can vote, when Raft is enabled. */ void -raft_cfg_is_candidate(struct raft *raft, bool is_candidate); +raft_cfg_is_candidate(struct raft *raft, bool is_candidate, bool demote); /** Configure Raft leader election timeout. */ void diff --git a/test/box/error.result b/test/box/error.result index 7761c6949..dad6a21d3 100644 --- a/test/box/error.result +++ b/test/box/error.result @@ -441,6 +441,9 @@ t; | 220: box.error.TOO_EARLY_SUBSCRIBE | 221: box.error.SQL_CANT_ADD_AUTOINC | 222: box.error.QUORUM_WAIT + | 223: box.error.INTERFERING_PROMOTE + | 224: box.error.RAFT_DISABLED + | 225: box.error.ALREADY_LEADER | ... test_run:cmd("setopt delimiter ''"); diff --git a/test/unit/raft_test_utils.c b/test/unit/raft_test_utils.c index b8735f373..a10ccae6a 100644 --- a/test/unit/raft_test_utils.c +++ b/test/unit/raft_test_utils.c @@ -360,7 +360,7 @@ raft_node_start(struct raft_node *node) raft_process_recovery(&node->raft, &node->journal.rows[i]); raft_cfg_is_enabled(&node->raft, node->cfg_is_enabled); - raft_cfg_is_candidate(&node->raft, node->cfg_is_candidate); + raft_cfg_is_candidate(&node->raft, node->cfg_is_candidate, true); raft_cfg_election_timeout(&node->raft, node->cfg_election_timeout); raft_cfg_election_quorum(&node->raft, node->cfg_election_quorum); raft_cfg_death_timeout(&node->raft, node->cfg_death_timeout); @@ -402,7 +402,7 @@ raft_node_cfg_is_candidate(struct raft_node *node, bool value) { node->cfg_is_candidate = value; if (raft_node_is_started(node)) { - raft_cfg_is_candidate(&node->raft, value); + raft_cfg_is_candidate(&node->raft, value, true); raft_run_async_work(); } } -- 2.24.3 (Apple Git-128)