From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 69E4B6EC55; Wed, 14 Jul 2021 21:34:02 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 69E4B6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626287642; bh=zYDpwikpcEIiWNXsGJUlqTG12bMFx4Xw51/7MAe2Yb8=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=gOC3v0ieLt4CLp6iH1zjAWFVrISgQteZ11lf2o+DTxDckxSCruNAmEEfwgENDh34Q KcexBSNSghhmYvOwYqjPJG2GcZtX7U8Brg7LPK9KCfoAgAxEFqGNgh3rgI82ePboUI tdr4Q9OMTB7TvU7fEVv1uaXX8Hrn7vk36JqkQT6s= Received: from smtp58.i.mail.ru (smtp58.i.mail.ru [217.69.128.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 933136EC55 for ; Wed, 14 Jul 2021 21:26:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 933136EC55 Received: by smtp58.i.mail.ru with esmtpa (envelope-from ) id 1m3jaL-0007Q7-BP; Wed, 14 Jul 2021 21:26:17 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Wed, 14 Jul 2021 21:25:44 +0300 Message-Id: <65333f9e19bc2ab095ec53810f40b7b779aad032.1626287002.git.sergepetrenko@tarantool.org> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD97BB0EF39AD2B33D52D9CC5C87942E9F174CE6222ED5B65F6182A05F538085040898B8FC768C8E7E01341C3CD7BD339F5280A0F571EF2C2A1828EBD9D84BD865A X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7B23E8716B669353DEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637082C84722D1D67EF8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D813C05A64C766876DDBC4989C57980879117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B60A9A04DE7321024275ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8183A4AFAF3EA6BDC44E1F4276B80994196E68746B1F2AB10C6939FA996DB5836E455FDE82F0AD11AC29C2B6934AE262D3EE7EAB7254005DCED114C52B35DBB74F4E7EAB7254005DCED0E7AE1DD52AB28FD1E0A4E2319210D9B64D260DF9561598FCFFBF5018520E39817A45118377F5F9E8E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D343D1F112031EF3D628C4B7CEC6AEABEB56AC8FC8738F8D09EFFBF1766B509FB6E9DD3F8C5E4F640521D7E09C32AA3244CF8F12BC3D2750AFAFCBC98A4688C0F6851E887DA02A9F7BF927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojDdSFIg49M1RCJflUKIpkLg== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A446DCF67A6ACA5D040DB95D57C2D6016AAA7A8891EDA16A430B424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v4 16/16] box: introduce `box.ctl.demote` X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Introduce a new journal entry, DEMOTE. The entry has the same meaning as PROMOTE, with the only difference that it clears limbo ownership instead of transferring it to the issuer. Introduce `box.ctl.demote`, a counterpart to `box.ctl.promote`, which clears the limbo ownership (when elections are off) via writing the DEMOTE, or simply makes this instance step down from the leader's role (when elections are enabled in any mode). A new request was necessary instead of simply writing PROMOTE(origin_id = 0), because origin_id is deduced from row.replica_id, which cannot be 0 for replicated rows (it's always equal to instance_id of the row originator). Closes #6034 @TarantoolBot document Title: box.ctl.demote `box.ctl.demote()` is a counterpart to `box.ctl.promote()` which works as follows: - when `box.cfg.election_mode` is not 'off': make the instance give up leadership. - when `box.cfg.election_mode` is 'off': write a DEMOTE entry to WAL. `box.ctl.demote()` may only be issued on the synchronous transaction queue owner (i.e. leader when elections are enabled). A DEMOTE request (DEMOTE = 32) copies PROMOTE behaviour (it clears the limbo as well), but clears the synchronous transaction queue ownership instead of assigning it to a new instance. --- src/box/box.cc | 99 ++++++++-- src/box/box.h | 3 + src/box/iproto_constants.h | 10 +- src/box/lua/ctl.c | 9 + src/box/txn_limbo.c | 37 +++- src/box/txn_limbo.h | 7 + test/replication/election_basic.result | 3 + test/replication/election_basic.test.lua | 1 + test/replication/election_qsync.result | 3 + test/replication/election_qsync.test.lua | 1 + .../gh-5140-qsync-casc-rollback.result | 6 + .../gh-5140-qsync-casc-rollback.test.lua | 2 + .../gh-5144-qsync-dup-confirm.result | 6 + .../gh-5144-qsync-dup-confirm.test.lua | 2 + .../gh-5163-qsync-restart-crash.result | 6 + .../gh-5163-qsync-restart-crash.test.lua | 2 + .../gh-5167-qsync-rollback-snap.result | 6 + .../gh-5167-qsync-rollback-snap.test.lua | 2 + .../gh-5195-qsync-replica-write.result | 10 +- .../gh-5195-qsync-replica-write.test.lua | 6 +- .../gh-5213-qsync-applier-order-3.result | 9 + .../gh-5213-qsync-applier-order-3.test.lua | 3 + .../gh-5213-qsync-applier-order.result | 6 + .../gh-5213-qsync-applier-order.test.lua | 2 + .../replication/gh-5288-qsync-recovery.result | 6 + .../gh-5288-qsync-recovery.test.lua | 2 + .../gh-5298-qsync-recovery-snap.result | 6 + .../gh-5298-qsync-recovery-snap.test.lua | 2 + .../gh-5426-election-on-off.result | 3 + .../gh-5426-election-on-off.test.lua | 1 + .../gh-5433-election-restart-recovery.result | 3 + ...gh-5433-election-restart-recovery.test.lua | 1 + ...sync-clear-synchro-queue-commit-all.result | 3 + ...nc-clear-synchro-queue-commit-all.test.lua | 1 + .../replication/gh-5438-election-state.result | 3 + .../gh-5438-election-state.test.lua | 1 + .../gh-5446-qsync-eval-quorum.result | 6 + .../gh-5446-qsync-eval-quorum.test.lua | 2 + .../gh-5506-election-on-off.result | 3 + .../gh-5506-election-on-off.test.lua | 1 + .../gh-5566-final-join-synchro.result | 6 + .../gh-5566-final-join-synchro.test.lua | 2 + .../gh-5874-qsync-txn-recovery.result | 6 + .../gh-5874-qsync-txn-recovery.test.lua | 2 + .../gh-6032-promote-wal-write.result | 3 + .../gh-6032-promote-wal-write.test.lua | 1 + .../gh-6034-election-promote-bump-term.result | 5 + ...h-6034-election-promote-bump-term.test.lua | 3 + .../gh-6034-qsync-limbo-ownership.result | 186 ++++++++++++++++++ .../gh-6034-qsync-limbo-ownership.test.lua | 68 +++++++ .../gh-6057-qsync-confirm-async-no-wal.result | 7 + ...h-6057-qsync-confirm-async-no-wal.test.lua | 3 + test/replication/hang_on_synchro_fail.result | 6 + .../replication/hang_on_synchro_fail.test.lua | 2 + test/replication/qsync_advanced.result | 12 ++ test/replication/qsync_advanced.test.lua | 4 + test/replication/qsync_basic.result | 33 ++-- test/replication/qsync_basic.test.lua | 16 +- test/replication/qsync_errinj.result | 6 + test/replication/qsync_errinj.test.lua | 2 + test/replication/qsync_snapshots.result | 6 + test/replication/qsync_snapshots.test.lua | 2 + test/replication/qsync_with_anon.result | 6 + test/replication/qsync_with_anon.test.lua | 2 + test/replication/suite.cfg | 1 + 65 files changed, 623 insertions(+), 52 deletions(-) create mode 100644 test/replication/gh-6034-qsync-limbo-ownership.result create mode 100644 test/replication/gh-6034-qsync-limbo-ownership.test.lua diff --git a/src/box/box.cc b/src/box/box.cc index dcfaab29e..f68fffcab 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -1679,6 +1679,25 @@ box_issue_promote(uint32_t prev_leader_id, int64_t promote_lsn) assert(txn_limbo_is_empty(&txn_limbo)); } +/** Write and process a DEMOTE request. */ +static void +box_issue_demote(uint32_t prev_leader_id, int64_t promote_lsn) +{ + assert(box_raft()->volatile_term == box_raft()->term); + assert(promote_lsn >= 0); + txn_limbo_write_demote(&txn_limbo, promote_lsn, + box_raft()->term); + struct synchro_request req = { + .type = IPROTO_DEMOTE, + .replica_id = prev_leader_id, + .origin_id = instance_id, + .lsn = promote_lsn, + .term = box_raft()->term, + }; + txn_limbo_process(&txn_limbo, &req); + assert(txn_limbo_is_empty(&txn_limbo)); +} + /** * Check whether this instance may run a promote() and set promote parameters * according to its election mode. @@ -1707,31 +1726,37 @@ box_check_promote_election_mode(bool *try_wait, bool *run_elections) return 0; } +/* A guard to block multiple simultaneous promote()/demote() invocations. */ +static bool box_in_promote = false; + int box_promote(void) { - /* A guard to block multiple simultaneous function invocations. */ - static bool in_promote = false; - if (in_promote) { + if (box_in_promote) { diag_set(ClientError, ER_UNSUPPORTED, "box.ctl.promote", "simultaneous invocations"); return -1; } - in_promote = true; + box_in_promote = true; auto promote_guard = make_scoped_guard([&] { - in_promote = false; + box_in_promote = false; }); - /* - * Do nothing when box isn't configured and when PROMOTE was already - * written for this term (synchronous replication and leader election - * are in sync, and both chose this node as a leader). - */ if (!is_box_configured) return 0; - if (txn_limbo_replica_term(&txn_limbo, instance_id) == - box_raft()->term) + + /* + * Currently active leader (the instance that is seen as leader by both + * raft and txn_limbo) can't issue another PROMOTE. + */ + bool is_leader = txn_limbo_replica_term(&txn_limbo, instance_id) == + box_raft()->term && txn_limbo.owner_id == instance_id; + if (box_election_mode != ELECTION_MODE_OFF) + is_leader = is_leader && box_raft()->state == RAFT_STATE_LEADER; + + if (is_leader) return 0; + bool run_elections = false; bool try_wait = false; @@ -1752,6 +1777,56 @@ box_promote(void) return -1; box_issue_promote(txn_limbo.owner_id, wait_lsn); + + return 0; +} + +int +box_demote(void) +{ + if (box_in_promote) { + diag_set(ClientError, ER_UNSUPPORTED, "box.ctl.demote", + "simultaneous invocations"); + return -1; + } + box_in_promote = true; + auto promote_guard = make_scoped_guard([&] { + box_in_promote = false; + }); + + if (!is_box_configured) + return 0; + + /* Currently active leader is the only one who can issue a DEMOTE. */ + bool is_leader = txn_limbo_replica_term(&txn_limbo, instance_id) == + box_raft()->term && txn_limbo.owner_id == instance_id; + if (box_election_mode != ELECTION_MODE_OFF) + is_leader = is_leader && box_raft()->state == RAFT_STATE_LEADER; + if (!is_leader) + return 0; + + bool try_wait = false; + + if (box_check_promote_election_mode(&try_wait, NULL) < 0) + return -1; + + int64_t wait_lsn = -1; + + if (box_trigger_elections() < 0) + return -1; + + if (box_election_mode != ELECTION_MODE_OFF) + return 0; + + if (try_wait && + box_try_wait_confirm(2 * replication_synchro_timeout) < 0) + return -1; + + if ((wait_lsn = box_wait_limbo_acked()) < 0) + return -1; + + box_issue_demote(txn_limbo.owner_id, wait_lsn); + return 0; } diff --git a/src/box/box.h b/src/box/box.h index ecf32240d..aaf20d9dd 100644 --- a/src/box/box.h +++ b/src/box/box.h @@ -276,6 +276,9 @@ typedef struct tuple box_tuple_t; int box_promote(void); +int +box_demote(void); + /* box_select is private and used only by FFI */ API_EXPORT int box_select(uint32_t space_id, uint32_t index_id, diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h index e913801a8..247ca6f37 100644 --- a/src/box/iproto_constants.h +++ b/src/box/iproto_constants.h @@ -241,6 +241,8 @@ enum iproto_type { IPROTO_RAFT = 30, /** PROMOTE request. */ IPROTO_PROMOTE = 31, + /** DEMOTE request. */ + IPROTO_DEMOTE = 32, /** A confirmation message for synchronous transactions. */ IPROTO_CONFIRM = 40, @@ -312,6 +314,8 @@ iproto_type_name(uint16_t type) return "RAFT"; case IPROTO_PROMOTE: return "PROMOTE"; + case IPROTO_DEMOTE: + return "DEMOTE"; case IPROTO_CONFIRM: return "CONFIRM"; case IPROTO_ROLLBACK: @@ -366,14 +370,14 @@ static inline bool iproto_type_is_synchro_request(uint16_t type) { return type == IPROTO_CONFIRM || type == IPROTO_ROLLBACK || - type == IPROTO_PROMOTE; + type == IPROTO_PROMOTE || type == IPROTO_DEMOTE; } -/** PROMOTE entry (synchronous replication and leader elections). */ +/** PROMOTE/DEMOTE entry (synchronous replication and leader elections). */ static inline bool iproto_type_is_promote_request(uint32_t type) { - return type == IPROTO_PROMOTE; + return type == IPROTO_PROMOTE || type == IPROTO_DEMOTE; } static inline bool diff --git a/src/box/lua/ctl.c b/src/box/lua/ctl.c index 368b9ab60..a613c4111 100644 --- a/src/box/lua/ctl.c +++ b/src/box/lua/ctl.c @@ -89,6 +89,14 @@ lbox_ctl_promote(struct lua_State *L) return 0; } +static int +lbox_ctl_demote(struct lua_State *L) +{ + if (box_demote() != 0) + return luaT_error(L); + return 0; +} + static int lbox_ctl_is_recovery_finished(struct lua_State *L) { @@ -127,6 +135,7 @@ static const struct luaL_Reg lbox_ctl_lib[] = { {"promote", lbox_ctl_promote}, /* An old alias. */ {"clear_synchro_queue", lbox_ctl_promote}, + {"demote", lbox_ctl_demote}, {"is_recovery_finished", lbox_ctl_is_recovery_finished}, {"set_on_shutdown_timeout", lbox_ctl_set_on_shutdown_timeout}, {NULL, NULL} diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index c8c4f587c..570f77c46 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -518,6 +518,29 @@ txn_limbo_read_promote(struct txn_limbo *limbo, uint32_t replica_id, limbo->confirmed_lsn = 0; } +void +txn_limbo_write_demote(struct txn_limbo *limbo, int64_t lsn, uint64_t term) +{ + limbo->confirmed_lsn = lsn; + limbo->is_in_rollback = true; + struct txn_limbo_entry *e = txn_limbo_last_synchro_entry(limbo); + assert(e == NULL || e->lsn <= lsn); + (void)e; + txn_limbo_write_synchro(limbo, IPROTO_DEMOTE, lsn, term); + limbo->is_in_rollback = false; +} + +/** + * Process a DEMOTE request, which's like PROMOTE, but clears the limbo + * ownership. + * @sa txn_limbo_read_promote. + */ +static void +txn_limbo_read_demote(struct txn_limbo *limbo, int64_t lsn) +{ + return txn_limbo_read_promote(limbo, REPLICA_ID_NIL, lsn); +} + void txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn) { @@ -709,12 +732,13 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) vclock_follow(&limbo->promote_term_map, origin, term); if (term > limbo->promote_greatest_term) limbo->promote_greatest_term = term; - } else if (req->type == IPROTO_PROMOTE && + } else if (iproto_type_is_promote_request(req->type) && limbo->promote_greatest_term > 1) { /* PROMOTE for outdated term. Ignore. */ - say_info("RAFT: ignoring PROMOTE request from instance " + say_info("RAFT: ignoring %s request from instance " "id %u for term %llu. Greatest term seen " - "before (%llu) is bigger.", origin, (long long)term, + "before (%llu) is bigger.", + iproto_type_name(req->type), origin, (long long)term, (long long)limbo->promote_greatest_term); return; } @@ -725,7 +749,7 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) * The limbo was empty on the instance issuing the request. * This means this instance must empty its limbo as well. */ - assert(lsn == 0 && req->type == IPROTO_PROMOTE); + assert(lsn == 0 && iproto_type_is_promote_request(req->type)); } else if (req->replica_id != limbo->owner_id) { /* * Ignore CONFIRM/ROLLBACK messages for a foreign master. @@ -733,7 +757,7 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) * data from an old leader, who has just started and written * confirm right on synchronous transaction recovery. */ - if (req->type != IPROTO_PROMOTE) + if (!iproto_type_is_promote_request(req->type)) return; /* * Promote has a bigger term, and tries to steal the limbo. It @@ -753,6 +777,9 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) case IPROTO_PROMOTE: txn_limbo_read_promote(limbo, req->origin_id, lsn); break; + case IPROTO_DEMOTE: + txn_limbo_read_demote(limbo, lsn); + break; default: unreachable(); } diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h index 7151843f4..53e52f676 100644 --- a/src/box/txn_limbo.h +++ b/src/box/txn_limbo.h @@ -329,6 +329,13 @@ txn_limbo_checkpoint(const struct txn_limbo *limbo, void txn_limbo_write_promote(struct txn_limbo *limbo, int64_t lsn, uint64_t term); +/** + * Write a DEMOTE request. + * It has the same effect as PROMOTE and additionally clears limbo ownership. + */ +void +txn_limbo_write_demote(struct txn_limbo *limbo, int64_t lsn, uint64_t term); + /** * Update qsync parameters dynamically. */ diff --git a/test/replication/election_basic.result b/test/replication/election_basic.result index b64028c60..c0323a042 100644 --- a/test/replication/election_basic.result +++ b/test/replication/election_basic.result @@ -114,6 +114,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... -- -- See if bootstrap with election enabled works. diff --git a/test/replication/election_basic.test.lua b/test/replication/election_basic.test.lua index 77fdf6340..085a499d5 100644 --- a/test/replication/election_basic.test.lua +++ b/test/replication/election_basic.test.lua @@ -43,6 +43,7 @@ box.cfg{ election_mode = 'off', \ election_timeout = old_election_timeout \ } +box.ctl.demote() -- -- See if bootstrap with election enabled works. diff --git a/test/replication/election_qsync.result b/test/replication/election_qsync.result index c06400b38..2402c8578 100644 --- a/test/replication/election_qsync.result +++ b/test/replication/election_qsync.result @@ -165,6 +165,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/election_qsync.test.lua b/test/replication/election_qsync.test.lua index ea6fc4a61..e1aca8351 100644 --- a/test/replication/election_qsync.test.lua +++ b/test/replication/election_qsync.test.lua @@ -84,4 +84,5 @@ box.cfg{ replication = old_replication, \ replication_synchro_timeout = old_replication_synchro_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5140-qsync-casc-rollback.result b/test/replication/gh-5140-qsync-casc-rollback.result index da77631dd..d3208e1a4 100644 --- a/test/replication/gh-5140-qsync-casc-rollback.result +++ b/test/replication/gh-5140-qsync-casc-rollback.result @@ -73,6 +73,9 @@ _ = box.schema.space.create('async', {is_sync=false, engine = engine}) _ = _:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Write something to flush the master state to replica. box.space.sync:replace{1} | --- @@ -222,3 +225,6 @@ test_run:cmd('delete server replica') box.schema.user.revoke('guest', 'super') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5140-qsync-casc-rollback.test.lua b/test/replication/gh-5140-qsync-casc-rollback.test.lua index 69fc9ad02..96ddfd260 100644 --- a/test/replication/gh-5140-qsync-casc-rollback.test.lua +++ b/test/replication/gh-5140-qsync-casc-rollback.test.lua @@ -48,6 +48,7 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') _ = box.schema.space.create('async', {is_sync=false, engine = engine}) _ = _:create_index('pk') +box.ctl.promote() -- Write something to flush the master state to replica. box.space.sync:replace{1} @@ -103,3 +104,4 @@ test_run:cmd('stop server replica') test_run:cmd('delete server replica') box.schema.user.revoke('guest', 'super') +box.ctl.demote() diff --git a/test/replication/gh-5144-qsync-dup-confirm.result b/test/replication/gh-5144-qsync-dup-confirm.result index 9d265d9ff..217e44412 100644 --- a/test/replication/gh-5144-qsync-dup-confirm.result +++ b/test/replication/gh-5144-qsync-dup-confirm.result @@ -46,6 +46,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Remember the current LSN. In the end, when the following synchronous -- transaction is committed, result LSN should be this value +2: for the @@ -148,6 +151,9 @@ test_run:cmd('delete server replica2') | - true | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5144-qsync-dup-confirm.test.lua b/test/replication/gh-5144-qsync-dup-confirm.test.lua index 01a8351e0..1d6af2c62 100644 --- a/test/replication/gh-5144-qsync-dup-confirm.test.lua +++ b/test/replication/gh-5144-qsync-dup-confirm.test.lua @@ -19,6 +19,7 @@ box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') +box.ctl.promote() -- Remember the current LSN. In the end, when the following synchronous -- transaction is committed, result LSN should be this value +2: for the @@ -69,4 +70,5 @@ test_run:cmd('delete server replica1') test_run:cmd('stop server replica2') test_run:cmd('delete server replica2') +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5163-qsync-restart-crash.result b/test/replication/gh-5163-qsync-restart-crash.result index e57bc76d1..1b4d3d9b5 100644 --- a/test/replication/gh-5163-qsync-restart-crash.result +++ b/test/replication/gh-5163-qsync-restart-crash.result @@ -16,6 +16,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.space.sync:replace{1} | --- @@ -30,3 +33,6 @@ box.space.sync:select{} box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5163-qsync-restart-crash.test.lua b/test/replication/gh-5163-qsync-restart-crash.test.lua index d5aca4749..c8d54aad2 100644 --- a/test/replication/gh-5163-qsync-restart-crash.test.lua +++ b/test/replication/gh-5163-qsync-restart-crash.test.lua @@ -7,8 +7,10 @@ engine = test_run:get_cfg('engine') -- _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.space.sync:replace{1} test_run:cmd('restart server default') box.space.sync:select{} box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-5167-qsync-rollback-snap.result b/test/replication/gh-5167-qsync-rollback-snap.result index 06f58526c..13166720f 100644 --- a/test/replication/gh-5167-qsync-rollback-snap.result +++ b/test/replication/gh-5167-qsync-rollback-snap.result @@ -41,6 +41,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Write something to flush the current master's state to replica. _ = box.space.sync:insert{1} | --- @@ -163,3 +166,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5167-qsync-rollback-snap.test.lua b/test/replication/gh-5167-qsync-rollback-snap.test.lua index 475727e61..1a2a31b7c 100644 --- a/test/replication/gh-5167-qsync-rollback-snap.test.lua +++ b/test/replication/gh-5167-qsync-rollback-snap.test.lua @@ -16,6 +16,7 @@ fiber = require('fiber') box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Write something to flush the current master's state to replica. _ = box.space.sync:insert{1} _ = box.space.sync:delete{1} @@ -65,3 +66,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5195-qsync-replica-write.result b/test/replication/gh-5195-qsync-replica-write.result index 85e00e6ed..bc73bb599 100644 --- a/test/replication/gh-5195-qsync-replica-write.result +++ b/test/replication/gh-5195-qsync-replica-write.result @@ -40,6 +40,9 @@ _ = box.schema.space.create('sync', {engine = engine, is_sync = true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.cfg{replication_synchro_timeout = 1000, replication_synchro_quorum = 3} | --- @@ -71,12 +74,12 @@ test_run:wait_lsn('replica', 'default') | --- | ... -- Normal DML is blocked - the limbo is not empty and does not belong to the --- replica. But synchro queue cleanup also does a WAL write, and propagates LSN +-- replica. But promote also does a WAL write, and propagates LSN -- of the instance. box.cfg{replication_synchro_timeout = 0.001} | --- | ... -box.ctl.clear_synchro_queue() +box.ctl.promote() | --- | ... @@ -144,6 +147,9 @@ test_run:cmd('delete server replica') | - true | ... +box.ctl.demote() + | --- + | ... box.space.sync:drop() | --- | ... diff --git a/test/replication/gh-5195-qsync-replica-write.test.lua b/test/replication/gh-5195-qsync-replica-write.test.lua index 64c48be99..a59ec154e 100644 --- a/test/replication/gh-5195-qsync-replica-write.test.lua +++ b/test/replication/gh-5195-qsync-replica-write.test.lua @@ -17,6 +17,7 @@ test_run:cmd('start server replica with wait=True, wait_load=True') -- _ = box.schema.space.create('sync', {engine = engine, is_sync = true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.cfg{replication_synchro_timeout = 1000, replication_synchro_quorum = 3} lsn = box.info.lsn @@ -30,10 +31,10 @@ test_run:wait_cond(function() return box.info.lsn == lsn end) test_run:switch('replica') test_run:wait_lsn('replica', 'default') -- Normal DML is blocked - the limbo is not empty and does not belong to the --- replica. But synchro queue cleanup also does a WAL write, and propagates LSN +-- replica. But promote also does a WAL write, and propagates LSN -- of the instance. box.cfg{replication_synchro_timeout = 0.001} -box.ctl.clear_synchro_queue() +box.ctl.promote() test_run:switch('default') -- Wait second ACK receipt. @@ -59,6 +60,7 @@ test_run:switch('default') test_run:cmd('stop server replica') test_run:cmd('delete server replica') +box.ctl.demote() box.space.sync:drop() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5213-qsync-applier-order-3.result b/test/replication/gh-5213-qsync-applier-order-3.result index bcb18b5c0..e788eec77 100644 --- a/test/replication/gh-5213-qsync-applier-order-3.result +++ b/test/replication/gh-5213-qsync-applier-order-3.result @@ -45,6 +45,9 @@ s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... test_run:cmd('create server replica1 with rpl_master=default,\ script="replication/replica1.lua"') @@ -179,6 +182,9 @@ box.cfg{ -- Replica2 takes the limbo ownership and sends the transaction to the replica1. -- Along with the CONFIRM from the default node, which is still not applied -- on the replica1. +box.ctl.promote() + | --- + | ... fiber = require('fiber') | --- | ... @@ -261,3 +267,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5213-qsync-applier-order-3.test.lua b/test/replication/gh-5213-qsync-applier-order-3.test.lua index 37b569da7..304656de0 100644 --- a/test/replication/gh-5213-qsync-applier-order-3.test.lua +++ b/test/replication/gh-5213-qsync-applier-order-3.test.lua @@ -30,6 +30,7 @@ box.schema.user.grant('guest', 'super') s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() test_run:cmd('create server replica1 with rpl_master=default,\ script="replication/replica1.lua"') @@ -90,6 +91,7 @@ box.cfg{ -- Replica2 takes the limbo ownership and sends the transaction to the replica1. -- Along with the CONFIRM from the default node, which is still not applied -- on the replica1. +box.ctl.promote() fiber = require('fiber') f = fiber.new(function() box.space.test:replace{2} end) @@ -123,3 +125,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5213-qsync-applier-order.result b/test/replication/gh-5213-qsync-applier-order.result index a8c24c289..ba6cdab06 100644 --- a/test/replication/gh-5213-qsync-applier-order.result +++ b/test/replication/gh-5213-qsync-applier-order.result @@ -29,6 +29,9 @@ s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... test_run:cmd('create server replica with rpl_master=default,\ script="replication/gh-5213-replica.lua"') @@ -300,3 +303,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5213-qsync-applier-order.test.lua b/test/replication/gh-5213-qsync-applier-order.test.lua index f1eccfa84..39b1912e8 100644 --- a/test/replication/gh-5213-qsync-applier-order.test.lua +++ b/test/replication/gh-5213-qsync-applier-order.test.lua @@ -14,6 +14,7 @@ box.schema.user.grant('guest', 'super') s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() test_run:cmd('create server replica with rpl_master=default,\ script="replication/gh-5213-replica.lua"') @@ -120,3 +121,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5288-qsync-recovery.result b/test/replication/gh-5288-qsync-recovery.result index dc0babef6..704b71d93 100644 --- a/test/replication/gh-5288-qsync-recovery.result +++ b/test/replication/gh-5288-qsync-recovery.result @@ -12,6 +12,9 @@ s = box.schema.space.create('sync', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... s:insert{1} | --- | - [1] @@ -25,3 +28,6 @@ test_run:cmd('restart server default') box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5288-qsync-recovery.test.lua b/test/replication/gh-5288-qsync-recovery.test.lua index 00bff7b87..2455f7278 100644 --- a/test/replication/gh-5288-qsync-recovery.test.lua +++ b/test/replication/gh-5288-qsync-recovery.test.lua @@ -5,7 +5,9 @@ test_run = require('test_run').new() -- s = box.schema.space.create('sync', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() s:insert{1} box.snapshot() test_run:cmd('restart server default') box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-5298-qsync-recovery-snap.result b/test/replication/gh-5298-qsync-recovery-snap.result index 922831552..0883fe5f5 100644 --- a/test/replication/gh-5298-qsync-recovery-snap.result +++ b/test/replication/gh-5298-qsync-recovery-snap.result @@ -17,6 +17,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... for i = 1, 10 do box.space.sync:replace{i} end | --- | ... @@ -98,3 +101,6 @@ box.space.sync:drop() box.space.loc:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5298-qsync-recovery-snap.test.lua b/test/replication/gh-5298-qsync-recovery-snap.test.lua index 187f60d75..084cde963 100644 --- a/test/replication/gh-5298-qsync-recovery-snap.test.lua +++ b/test/replication/gh-5298-qsync-recovery-snap.test.lua @@ -8,6 +8,7 @@ engine = test_run:get_cfg('engine') -- _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() for i = 1, 10 do box.space.sync:replace{i} end -- Local rows could affect this by increasing the signature. @@ -46,3 +47,4 @@ box.cfg{ } box.space.sync:drop() box.space.loc:drop() +box.ctl.demote() diff --git a/test/replication/gh-5426-election-on-off.result b/test/replication/gh-5426-election-on-off.result index 7444ef7f2..2bdc17ec6 100644 --- a/test/replication/gh-5426-election-on-off.result +++ b/test/replication/gh-5426-election-on-off.result @@ -168,6 +168,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5426-election-on-off.test.lua b/test/replication/gh-5426-election-on-off.test.lua index bdf06903b..6277e9ef2 100644 --- a/test/replication/gh-5426-election-on-off.test.lua +++ b/test/replication/gh-5426-election-on-off.test.lua @@ -69,4 +69,5 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5433-election-restart-recovery.result b/test/replication/gh-5433-election-restart-recovery.result index f8f32416e..ed63ff409 100644 --- a/test/replication/gh-5433-election-restart-recovery.result +++ b/test/replication/gh-5433-election-restart-recovery.result @@ -169,6 +169,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5433-election-restart-recovery.test.lua b/test/replication/gh-5433-election-restart-recovery.test.lua index 4aff000bf..ae1f42c4d 100644 --- a/test/replication/gh-5433-election-restart-recovery.test.lua +++ b/test/replication/gh-5433-election-restart-recovery.test.lua @@ -84,4 +84,5 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result index 2699231e5..20fab4072 100644 --- a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result +++ b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result @@ -49,6 +49,9 @@ _ = box.schema.space.create('test', {is_sync=true}) _ = box.space.test:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Fill the limbo with pending entries. 3 mustn't receive them yet. test_run:cmd('stop server election_replica3') diff --git a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua index 03705d96c..ec0f1d77e 100644 --- a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua +++ b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua @@ -21,6 +21,7 @@ box.ctl.wait_rw() _ = box.schema.space.create('test', {is_sync=true}) _ = box.space.test:create_index('pk') +box.ctl.promote() -- Fill the limbo with pending entries. 3 mustn't receive them yet. test_run:cmd('stop server election_replica3') diff --git a/test/replication/gh-5438-election-state.result b/test/replication/gh-5438-election-state.result index 6985f026a..68b6bfad8 100644 --- a/test/replication/gh-5438-election-state.result +++ b/test/replication/gh-5438-election-state.result @@ -47,6 +47,9 @@ end) | ... -- Cleanup. +box.ctl.demote() + | --- + | ... box.cfg{election_mode = old_election_mode} | --- | ... diff --git a/test/replication/gh-5438-election-state.test.lua b/test/replication/gh-5438-election-state.test.lua index 60c3366c1..cf0f4ca23 100644 --- a/test/replication/gh-5438-election-state.test.lua +++ b/test/replication/gh-5438-election-state.test.lua @@ -22,6 +22,7 @@ test_run:wait_cond(function()\ end) -- Cleanup. +box.ctl.demote() box.cfg{election_mode = old_election_mode} test_run:cmd('stop server replica') test_run:cmd('delete server replica') diff --git a/test/replication/gh-5446-qsync-eval-quorum.result b/test/replication/gh-5446-qsync-eval-quorum.result index 5f83b248c..b3c217913 100644 --- a/test/replication/gh-5446-qsync-eval-quorum.result +++ b/test/replication/gh-5446-qsync-eval-quorum.result @@ -88,6 +88,9 @@ s = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Only one master node -> 1/2 + 1 = 1 s:insert{1} -- should pass @@ -343,3 +346,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5446-qsync-eval-quorum.test.lua b/test/replication/gh-5446-qsync-eval-quorum.test.lua index 6b9e324ed..c2901b845 100644 --- a/test/replication/gh-5446-qsync-eval-quorum.test.lua +++ b/test/replication/gh-5446-qsync-eval-quorum.test.lua @@ -37,6 +37,7 @@ end -- Create a sync space we will operate on s = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = s:create_index('pk') +box.ctl.promote() -- Only one master node -> 1/2 + 1 = 1 s:insert{1} -- should pass @@ -135,3 +136,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5506-election-on-off.result b/test/replication/gh-5506-election-on-off.result index b8abd7ecd..a7f2b6a9c 100644 --- a/test/replication/gh-5506-election-on-off.result +++ b/test/replication/gh-5506-election-on-off.result @@ -138,3 +138,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5506-election-on-off.test.lua b/test/replication/gh-5506-election-on-off.test.lua index 476b00ec0..f8915c333 100644 --- a/test/replication/gh-5506-election-on-off.test.lua +++ b/test/replication/gh-5506-election-on-off.test.lua @@ -66,3 +66,4 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5566-final-join-synchro.result b/test/replication/gh-5566-final-join-synchro.result index a09882ba6..c5ae2f283 100644 --- a/test/replication/gh-5566-final-join-synchro.result +++ b/test/replication/gh-5566-final-join-synchro.result @@ -12,6 +12,9 @@ _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.schema.user.grant('guest', 'replication') | --- @@ -137,3 +140,6 @@ test_run:cleanup_cluster() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5566-final-join-synchro.test.lua b/test/replication/gh-5566-final-join-synchro.test.lua index 2db2c742f..25f411407 100644 --- a/test/replication/gh-5566-final-join-synchro.test.lua +++ b/test/replication/gh-5566-final-join-synchro.test.lua @@ -5,6 +5,7 @@ test_run = require('test_run').new() -- _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.schema.user.grant('guest', 'replication') box.schema.user.grant('guest', 'write', 'space', 'sync') @@ -59,3 +60,4 @@ box.cfg{\ box.space.sync:drop() test_run:cleanup_cluster() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/gh-5874-qsync-txn-recovery.result b/test/replication/gh-5874-qsync-txn-recovery.result index 73f903ca7..01328a9e3 100644 --- a/test/replication/gh-5874-qsync-txn-recovery.result +++ b/test/replication/gh-5874-qsync-txn-recovery.result @@ -31,6 +31,9 @@ sync = box.schema.create_space('sync', {is_sync = true, engine = engine}) _ = sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- The transaction fails, but is written to the log anyway. box.begin() async:insert{1} sync:insert{1} box.commit() @@ -160,3 +163,6 @@ sync:drop() loc:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5874-qsync-txn-recovery.test.lua b/test/replication/gh-5874-qsync-txn-recovery.test.lua index f35eb68de..6ddf164ac 100644 --- a/test/replication/gh-5874-qsync-txn-recovery.test.lua +++ b/test/replication/gh-5874-qsync-txn-recovery.test.lua @@ -12,6 +12,7 @@ async = box.schema.create_space('async', {engine = engine}) _ = async:create_index('pk') sync = box.schema.create_space('sync', {is_sync = true, engine = engine}) _ = sync:create_index('pk') +box.ctl.promote() -- The transaction fails, but is written to the log anyway. box.begin() async:insert{1} sync:insert{1} box.commit() @@ -82,3 +83,4 @@ loc:select() async:drop() sync:drop() loc:drop() +box.ctl.demote() diff --git a/test/replication/gh-6032-promote-wal-write.result b/test/replication/gh-6032-promote-wal-write.result index 246c7974f..03112fb8d 100644 --- a/test/replication/gh-6032-promote-wal-write.result +++ b/test/replication/gh-6032-promote-wal-write.result @@ -67,3 +67,6 @@ box.cfg{\ box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-6032-promote-wal-write.test.lua b/test/replication/gh-6032-promote-wal-write.test.lua index 8c1859083..9a036a8b4 100644 --- a/test/replication/gh-6032-promote-wal-write.test.lua +++ b/test/replication/gh-6032-promote-wal-write.test.lua @@ -26,3 +26,4 @@ box.cfg{\ replication_synchro_timeout = replication_synchro_timeout,\ } box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-6034-election-promote-bump-term.result b/test/replication/gh-6034-election-promote-bump-term.result index 8be4e8243..c64da0d5d 100644 --- a/test/replication/gh-6034-election-promote-bump-term.result +++ b/test/replication/gh-6034-election-promote-bump-term.result @@ -19,3 +19,8 @@ assert(box.info.election.term == term + 1) | --- | - true | ... + +-- Cleanup. +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-6034-election-promote-bump-term.test.lua b/test/replication/gh-6034-election-promote-bump-term.test.lua index 1e814bf5d..dfb78f906 100644 --- a/test/replication/gh-6034-election-promote-bump-term.test.lua +++ b/test/replication/gh-6034-election-promote-bump-term.test.lua @@ -7,3 +7,6 @@ box.cfg{election_mode='off'} term = box.info.election.term box.ctl.promote() assert(box.info.election.term == term + 1) + +-- Cleanup. +box.ctl.demote() diff --git a/test/replication/gh-6034-qsync-limbo-ownership.result b/test/replication/gh-6034-qsync-limbo-ownership.result new file mode 100644 index 000000000..0da3e5c2d --- /dev/null +++ b/test/replication/gh-6034-qsync-limbo-ownership.result @@ -0,0 +1,186 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +-- +-- gh-6034: test that transactional limbo isn't accessible without a promotion. +-- +synchro_quorum = box.cfg.replication_synchro_quorum + | --- + | ... +election_mode = box.cfg.election_mode + | --- + | ... +box.cfg{replication_synchro_quorum = 1, election_mode='off'} + | --- + | ... + +_ = box.schema.space.create('async'):create_index('pk') + | --- + | ... +_ = box.schema.space.create('sync', {is_sync=true}):create_index('pk') + | --- + | ... + +-- Limbo is initially unclaimed, everyone is writeable. +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == 0) + | --- + | - true + | ... +box.space.async:insert{1} -- success. + | --- + | - [1] + | ... +-- Synchro spaces aren't writeable +box.space.sync:insert{1} -- error. + | --- + | - error: The synchronous transaction queue doesn't belong to any instance + | ... + +box.ctl.promote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == box.info.id) + | --- + | - true + | ... +box.space.sync:insert{1} -- success. + | --- + | - [1] + | ... + +-- Everyone but the limbo owner is read-only. +box.schema.user.grant('guest', 'replication') + | --- + | ... +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... +test_run:cmd('set variable rpl_listen to "replica.listen"') + | --- + | - true + | ... +orig_replication = box.cfg.replication + | --- + | ... +box.cfg{replication=rpl_listen} + | --- + | ... + +test_run:switch('replica') + | --- + | - true + | ... +assert(box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == test_run:get_server_id('default')) + | --- + | - true + | ... +box.space.async:insert{2} -- failure. + | --- + | - error: Can't modify data because this instance is in read-only mode. + | ... + +-- Promotion on the other node. Default should become ro. +box.ctl.promote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == box.info.id) + | --- + | - true + | ... +box.space.sync:insert{2} -- success. + | --- + | - [2] + | ... + +test_run:switch('default') + | --- + | - true + | ... +assert(box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == test_run:get_server_id('replica')) + | --- + | - true + | ... +box.space.sync:insert{3} -- failure. + | --- + | - error: Can't modify data because this instance is in read-only mode. + | ... + +box.ctl.promote() + | --- + | ... +box.ctl.demote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +box.space.sync:insert{3} -- still fails. + | --- + | - error: The synchronous transaction queue doesn't belong to any instance + | ... +assert(box.info.synchro.queue.owner == 0) + | --- + | - true + | ... +box.space.async:insert{3} -- success. + | --- + | - [3] + | ... + +-- Cleanup. +box.ctl.demote() + | --- + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +box.schema.user.revoke('guest', 'replication') + | --- + | ... +box.space.sync:drop() + | --- + | ... +box.space.async:drop() + | --- + | ... +box.cfg{\ + replication_synchro_quorum = synchro_quorum,\ + election_mode = election_mode,\ + replication = orig_replication,\ +} + | --- + | ... diff --git a/test/replication/gh-6034-qsync-limbo-ownership.test.lua b/test/replication/gh-6034-qsync-limbo-ownership.test.lua new file mode 100644 index 000000000..a0c49575a --- /dev/null +++ b/test/replication/gh-6034-qsync-limbo-ownership.test.lua @@ -0,0 +1,68 @@ +test_run = require('test_run').new() + +-- +-- gh-6034: test that transactional limbo isn't accessible without a promotion. +-- +synchro_quorum = box.cfg.replication_synchro_quorum +election_mode = box.cfg.election_mode +box.cfg{replication_synchro_quorum = 1, election_mode='off'} + +_ = box.schema.space.create('async'):create_index('pk') +_ = box.schema.space.create('sync', {is_sync=true}):create_index('pk') + +-- Limbo is initially unclaimed, everyone is writeable. +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == 0) +box.space.async:insert{1} -- success. +-- Synchro spaces aren't writeable +box.space.sync:insert{1} -- error. + +box.ctl.promote() +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == box.info.id) +box.space.sync:insert{1} -- success. + +-- Everyone but the limbo owner is read-only. +box.schema.user.grant('guest', 'replication') +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') +test_run:cmd('set variable rpl_listen to "replica.listen"') +orig_replication = box.cfg.replication +box.cfg{replication=rpl_listen} + +test_run:switch('replica') +assert(box.info.ro) +assert(box.info.synchro.queue.owner == test_run:get_server_id('default')) +box.space.async:insert{2} -- failure. + +-- Promotion on the other node. Default should become ro. +box.ctl.promote() +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == box.info.id) +box.space.sync:insert{2} -- success. + +test_run:switch('default') +assert(box.info.ro) +assert(box.info.synchro.queue.owner == test_run:get_server_id('replica')) +box.space.sync:insert{3} -- failure. + +box.ctl.promote() +box.ctl.demote() +assert(not box.info.ro) +box.space.sync:insert{3} -- still fails. +assert(box.info.synchro.queue.owner == 0) +box.space.async:insert{3} -- success. + +-- Cleanup. +box.ctl.demote() +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +box.schema.user.revoke('guest', 'replication') +box.space.sync:drop() +box.space.async:drop() +box.cfg{\ + replication_synchro_quorum = synchro_quorum,\ + election_mode = election_mode,\ + replication = orig_replication,\ +} diff --git a/test/replication/gh-6057-qsync-confirm-async-no-wal.result b/test/replication/gh-6057-qsync-confirm-async-no-wal.result index 23c77729b..e7beefb2a 100644 --- a/test/replication/gh-6057-qsync-confirm-async-no-wal.result +++ b/test/replication/gh-6057-qsync-confirm-async-no-wal.result @@ -40,6 +40,10 @@ _ = s2:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... + errinj = box.error.injection | --- | ... @@ -161,3 +165,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua b/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua index a11ddc042..bb459ea02 100644 --- a/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua +++ b/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua @@ -21,6 +21,8 @@ _ = s:create_index('pk') s2 = box.schema.create_space('test2') _ = s2:create_index('pk') +box.ctl.promote() + errinj = box.error.injection function create_hanging_async_after_confirm(sync_key, async_key1, async_key2) \ @@ -86,3 +88,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/hang_on_synchro_fail.result b/test/replication/hang_on_synchro_fail.result index 9f6fac00b..dda15af20 100644 --- a/test/replication/hang_on_synchro_fail.result +++ b/test/replication/hang_on_synchro_fail.result @@ -19,6 +19,9 @@ _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... old_synchro_quorum = box.cfg.replication_synchro_quorum | --- @@ -127,4 +130,7 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/hang_on_synchro_fail.test.lua b/test/replication/hang_on_synchro_fail.test.lua index 6c3b09fab..f0d494eae 100644 --- a/test/replication/hang_on_synchro_fail.test.lua +++ b/test/replication/hang_on_synchro_fail.test.lua @@ -8,6 +8,7 @@ box.schema.user.grant('guest', 'replication') _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() old_synchro_quorum = box.cfg.replication_synchro_quorum box.cfg{replication_synchro_quorum=3} @@ -54,4 +55,5 @@ box.cfg{replication_synchro_quorum=old_synchro_quorum,\ replication_synchro_timeout=old_synchro_timeout} box.space.sync:drop() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/qsync_advanced.result b/test/replication/qsync_advanced.result index 94b19b1f2..72ac0c326 100644 --- a/test/replication/qsync_advanced.result +++ b/test/replication/qsync_advanced.result @@ -72,6 +72,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. box.space.sync:insert{1} -- success | --- @@ -468,6 +471,9 @@ box.space.sync:select{} -- 1 box.cfg{read_only=false} -- promote replica to master | --- | ... +box.ctl.promote() + | --- + | ... test_run:switch('default') | --- | - true @@ -508,6 +514,9 @@ test_run:switch('default') box.cfg{read_only=false} | --- | ... +box.ctl.promote() + | --- + | ... test_run:switch('replica') | --- | - true @@ -781,3 +790,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_advanced.test.lua b/test/replication/qsync_advanced.test.lua index 058ece602..37c285b8d 100644 --- a/test/replication/qsync_advanced.test.lua +++ b/test/replication/qsync_advanced.test.lua @@ -30,6 +30,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. box.space.sync:insert{1} -- success test_run:cmd('switch replica') @@ -170,6 +171,7 @@ box.space.sync:select{} -- 1 test_run:switch('replica') box.space.sync:select{} -- 1 box.cfg{read_only=false} -- promote replica to master +box.ctl.promote() test_run:switch('default') box.cfg{read_only=true} -- demote master to replica test_run:switch('replica') @@ -181,6 +183,7 @@ box.space.sync:select{} -- 1, 2 -- Revert cluster configuration. test_run:switch('default') box.cfg{read_only=false} +box.ctl.promote() test_run:switch('replica') box.cfg{read_only=true} -- Testcase cleanup. @@ -279,3 +282,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/qsync_basic.result b/test/replication/qsync_basic.result index 7e711ba13..bbdfc42fe 100644 --- a/test/replication/qsync_basic.result +++ b/test/replication/qsync_basic.result @@ -14,6 +14,9 @@ s1.is_sync pk = s1:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.begin() s1:insert({1}) s1:insert({2}) box.commit() | --- | ... @@ -645,19 +648,12 @@ test_run:switch('default') | --- | - true | ... -box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 1000} - | --- - | ... -f = fiber.create(function() box.space.sync:replace{1} end) +box.ctl.demote() | --- | ... -test_run:wait_lsn('replica', 'default') +box.space.sync:replace{1} | --- - | ... - -test_run:switch('replica') - | --- - | - true + | - error: The synchronous transaction queue doesn't belong to any instance | ... function skip_row() return nil end | --- @@ -674,26 +670,22 @@ box.space.sync:replace{2} box.space.sync:before_replace(nil, skip_row) | --- | ... -assert(box.space.sync:get{2} == nil) +assert(box.space.sync:get{1} == nil) | --- | - true | ... -assert(box.space.sync:get{1} ~= nil) +assert(box.space.sync:get{2} == nil) | --- | - true | ... - -test_run:switch('default') +assert(box.info.lsn == old_lsn + 1) | --- | - true | ... -box.cfg{replication_synchro_quorum = 2} +box.ctl.promote() | --- | ... -test_run:wait_cond(function() return f:status() == 'dead' end) - | --- - | - true - | ... + box.space.sync:truncate() | --- | ... @@ -758,3 +750,6 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_basic.test.lua b/test/replication/qsync_basic.test.lua index 75c9b222b..eac465e25 100644 --- a/test/replication/qsync_basic.test.lua +++ b/test/replication/qsync_basic.test.lua @@ -6,6 +6,7 @@ s1 = box.schema.create_space('test1', {is_sync = true}) s1.is_sync pk = s1:create_index('pk') +box.ctl.promote() box.begin() s1:insert({1}) s1:insert({2}) box.commit() s1:select{} @@ -253,22 +254,18 @@ box.space.sync:count() -- instances, but also works for local rows. -- test_run:switch('default') -box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 1000} -f = fiber.create(function() box.space.sync:replace{1} end) -test_run:wait_lsn('replica', 'default') - -test_run:switch('replica') +box.ctl.demote() +box.space.sync:replace{1} function skip_row() return nil end old_lsn = box.info.lsn _ = box.space.sync:before_replace(skip_row) box.space.sync:replace{2} box.space.sync:before_replace(nil, skip_row) +assert(box.space.sync:get{1} == nil) assert(box.space.sync:get{2} == nil) -assert(box.space.sync:get{1} ~= nil) +assert(box.info.lsn == old_lsn + 1) +box.ctl.promote() -test_run:switch('default') -box.cfg{replication_synchro_quorum = 2} -test_run:wait_cond(function() return f:status() == 'dead' end) box.space.sync:truncate() -- @@ -301,3 +298,4 @@ test_run:cmd('delete server replica') box.space.test:drop() box.space.sync:drop() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/qsync_errinj.result b/test/replication/qsync_errinj.result index 635bcf939..cf1e30a90 100644 --- a/test/replication/qsync_errinj.result +++ b/test/replication/qsync_errinj.result @@ -35,6 +35,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- -- gh-5100: slow ACK sending shouldn't stun replica for the @@ -542,3 +545,6 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'super') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_errinj.test.lua b/test/replication/qsync_errinj.test.lua index 6a9fd3e1a..e7c85c58c 100644 --- a/test/replication/qsync_errinj.test.lua +++ b/test/replication/qsync_errinj.test.lua @@ -12,6 +12,7 @@ test_run:cmd('start server replica with wait=True, wait_load=True') _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- -- gh-5100: slow ACK sending shouldn't stun replica for the @@ -222,3 +223,4 @@ test_run:cmd('delete server replica') box.space.sync:drop() box.schema.user.revoke('guest', 'super') +box.ctl.demote() diff --git a/test/replication/qsync_snapshots.result b/test/replication/qsync_snapshots.result index cafdd63c8..ca418b168 100644 --- a/test/replication/qsync_snapshots.result +++ b/test/replication/qsync_snapshots.result @@ -57,6 +57,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. box.space.sync:insert{1} | --- @@ -299,3 +302,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_snapshots.test.lua b/test/replication/qsync_snapshots.test.lua index 590610974..82c2e3f7c 100644 --- a/test/replication/qsync_snapshots.test.lua +++ b/test/replication/qsync_snapshots.test.lua @@ -23,6 +23,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. box.space.sync:insert{1} box.space.sync:select{} -- 1 @@ -130,3 +131,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/qsync_with_anon.result b/test/replication/qsync_with_anon.result index 6a2952a32..99c6fb902 100644 --- a/test/replication/qsync_with_anon.result +++ b/test/replication/qsync_with_anon.result @@ -57,6 +57,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. test_run:switch('default') | --- @@ -220,6 +223,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... test_run:cleanup_cluster() | --- | ... diff --git a/test/replication/qsync_with_anon.test.lua b/test/replication/qsync_with_anon.test.lua index d7ecaa107..e73880ec7 100644 --- a/test/replication/qsync_with_anon.test.lua +++ b/test/replication/qsync_with_anon.test.lua @@ -22,6 +22,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. test_run:switch('default') box.space.sync:insert{1} -- success @@ -81,4 +82,5 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() test_run:cleanup_cluster() diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg index 7a6fd7052..9e284b3f2 100644 --- a/test/replication/suite.cfg +++ b/test/replication/suite.cfg @@ -48,6 +48,7 @@ "gh-5613-bootstrap-prefer-booted.test.lua": {}, "gh-6027-applier-error-show.test.lua": {}, "gh-6032-promote-wal-write.test.lua": {}, + "gh-6034-qsync-limbo-ownership.test.lua": {}, "gh-6034-election-promote-bump-term.test.lua": {}, "gh-6057-qsync-confirm-async-no-wal.test.lua": {}, "gh-6094-rs-uuid-mismatch.test.lua": {}, -- 2.30.1 (Apple Git-130)