From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id EA9B96EC40; Tue, 29 Jun 2021 01:16:55 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org EA9B96EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1624918616; bh=bTNGJrQpR5i0L7ipkZPHIQ6r/hBiU8uhFWi9NBdgorY=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=CmU+X/Uiu5fd2csnrLQngSoaheqiknrRy2OTs1IdHOvhgvLng3bMLplomzLzC5dSX auvbQLvjdQDeVWddXEK4MA3wr+e0CRIUq0aGeK5y+gD4GTjWZ912f5Ptsd0kLW541a mcqz/5tg/ybARrj2hm8U+1TWclxhWve47IhpiupQ= Received: from smtp61.i.mail.ru (smtp61.i.mail.ru [217.69.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id ED8506EC44 for ; Tue, 29 Jun 2021 01:13:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ED8506EC44 Received: by smtp61.i.mail.ru with esmtpa (envelope-from ) id 1lxzVF-0007oC-H7; Tue, 29 Jun 2021 01:13:18 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Tue, 29 Jun 2021 01:12:53 +0300 Message-Id: X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD954DFF1DC42D673FB517E8D1A7E34673E94B235AA752823A6182A05F5380850408E94EEE4EACE23DF5E8D5121CEE54859E5E22D19F4F4F70DC904CD47DEE0865D X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7547428DA4700DDEFC2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE7EF5E6976C8478FE4EA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BD6CF32B5F8F9D4045A6D224EE06F90D1FBAC045989C4A486CC7F00164DA146DAFE8445B8C89999728AA50765F7900637F6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F7900637F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA7E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8B569F1129A2C6445075ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2AD77751E876CB595E8F7B195E1C97831A91E565E6410D4C4654CE1D82DC659E2 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8183A4AFAF3EA6BDC44E1F4276B80994196D87436A60A035F407E0AE297D90647F64850D7DDC69555429C2B6934AE262D3EE7EAB7254005DCED114C52B35DBB74F4E7EAB7254005DCED0E7AE1DD52AB28FD1E0A4E2319210D9B64D260DF9561598F01A9E91200F654B08F3D2DDDCA87B9828E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34E229BE567979C940BF4F6152B7099BE674F4D452159D5102D56DD28A05BEF4A0EC7F43391C1ADB471D7E09C32AA3244C66CF25580EDA17F052008AE393EB17B8F94338140B71B8EE927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojNjLyMoNI2JZoA5+V0WExFA== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A446C3FEE19542AA4C0E024A86ED97D3A7A18D50A612EA6A48DA424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v3 07/12] box: introduce `box.ctl.demote` X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Introduce a new journal entry, DEMOTE. The entry has the same meaning as PROMOTE, with the only difference that it clears limbo ownership instead of transferring it to the issuer. Introduce `box.ctl.demote`, which works exactly like `box.ctl.promote`, but results in writing DEMOTE instead of PROMOTE. A new request was necessary instead of simply writing PROMOTE(origin_id = 0), because origin_id is deduced from row.replica_id, which cannot be 0 for replicated rows (it's always equal to instance_id of the row originator). Closes #6034 @TarantoolBot document Title: box.ctl.demote `box.ctl.demote()` is a new function, which works exactly like `box.ctl.promote()`, with one exception that it results in the instance writing DEMOTE request to WAL instead of a PROMOTE request. A DEMOTE request (DEMOTE = 32) copies PROMOTE behaviour (it clears the limbo as well), but clears the synchronous transaction queue ownership instead of assigning it to a new instance. --- src/box/box.cc | 48 ++++- src/box/box.h | 3 + src/box/iproto_constants.h | 10 +- src/box/lua/ctl.c | 9 + src/box/txn_limbo.c | 37 +++- src/box/txn_limbo.h | 7 + test/box/error.result | 1 + test/replication/election_basic.result | 3 + test/replication/election_basic.test.lua | 1 + test/replication/election_qsync.result | 3 + test/replication/election_qsync.test.lua | 1 + .../gh-5140-qsync-casc-rollback.result | 6 + .../gh-5140-qsync-casc-rollback.test.lua | 2 + .../gh-5144-qsync-dup-confirm.result | 6 + .../gh-5144-qsync-dup-confirm.test.lua | 2 + .../gh-5163-qsync-restart-crash.result | 6 + .../gh-5163-qsync-restart-crash.test.lua | 2 + .../gh-5167-qsync-rollback-snap.result | 6 + .../gh-5167-qsync-rollback-snap.test.lua | 2 + .../gh-5195-qsync-replica-write.result | 10 +- .../gh-5195-qsync-replica-write.test.lua | 6 +- .../gh-5213-qsync-applier-order-3.result | 9 + .../gh-5213-qsync-applier-order-3.test.lua | 3 + .../gh-5213-qsync-applier-order.result | 6 + .../gh-5213-qsync-applier-order.test.lua | 2 + .../replication/gh-5288-qsync-recovery.result | 6 + .../gh-5288-qsync-recovery.test.lua | 2 + .../gh-5298-qsync-recovery-snap.result | 6 + .../gh-5298-qsync-recovery-snap.test.lua | 2 + .../gh-5426-election-on-off.result | 3 + .../gh-5426-election-on-off.test.lua | 1 + .../gh-5433-election-restart-recovery.result | 3 + ...gh-5433-election-restart-recovery.test.lua | 1 + ...sync-clear-synchro-queue-commit-all.result | 3 + ...nc-clear-synchro-queue-commit-all.test.lua | 1 + test/replication/gh-5438-raft-state.result | 3 + test/replication/gh-5438-raft-state.test.lua | 1 + .../gh-5446-qsync-eval-quorum.result | 7 + .../gh-5446-qsync-eval-quorum.test.lua | 3 + .../gh-5506-election-on-off.result | 3 + .../gh-5506-election-on-off.test.lua | 1 + .../gh-5566-final-join-synchro.result | 6 + .../gh-5566-final-join-synchro.test.lua | 2 + .../gh-5874-qsync-txn-recovery.result | 6 + .../gh-5874-qsync-txn-recovery.test.lua | 2 + .../gh-6032-promote-wal-write.result | 3 + .../gh-6032-promote-wal-write.test.lua | 1 + .../gh-6034-limbo-ownership.result | 189 ++++++++++++++++++ .../gh-6034-limbo-ownership.test.lua | 69 +++++++ .../gh-6034-promote-bump-term.result | 3 + .../gh-6034-promote-bump-term.test.lua | 1 + .../gh-6057-qsync-confirm-async-no-wal.result | 7 + ...h-6057-qsync-confirm-async-no-wal.test.lua | 3 + test/replication/hang_on_synchro_fail.result | 6 + .../replication/hang_on_synchro_fail.test.lua | 2 + test/replication/qsync_advanced.result | 12 ++ test/replication/qsync_advanced.test.lua | 4 + test/replication/qsync_basic.result | 33 ++- test/replication/qsync_basic.test.lua | 16 +- test/replication/qsync_errinj.result | 6 + test/replication/qsync_errinj.test.lua | 2 + test/replication/qsync_snapshots.result | 6 + test/replication/qsync_snapshots.test.lua | 2 + test/replication/qsync_with_anon.result | 6 + test/replication/qsync_with_anon.test.lua | 2 + test/replication/suite.cfg | 1 + 66 files changed, 579 insertions(+), 48 deletions(-) create mode 100644 test/replication/gh-6034-limbo-ownership.result create mode 100644 test/replication/gh-6034-limbo-ownership.test.lua diff --git a/src/box/box.cc b/src/box/box.cc index 1d894be97..86c5967b9 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -1527,13 +1527,14 @@ box_wait_quorum(uint32_t lead_id, int64_t target_lsn, int quorum, return 0; } -int -box_promote(void) +static int +box_clear_synchro_queue(bool demote) { /* A guard to block multiple simultaneous function invocations. */ static bool in_promote = false; if (in_promote) { - diag_set(ClientError, ER_UNSUPPORTED, "box.ctl.promote", + diag_set(ClientError, ER_UNSUPPORTED, + demote ? "box.ctl.demote" : "box.ctl.promote", "simultaneous invocations"); return -1; } @@ -1554,13 +1555,18 @@ box_promote(void) return -1; case ELECTION_MODE_MANUAL: case ELECTION_MODE_CANDIDATE: - run_elections = box_raft()->state != RAFT_STATE_LEADER; + /* + * No elections involved in a DEMOTE or when the instance is + * already the leader. + */ + run_elections = box_raft()->state != RAFT_STATE_LEADER && + !demote; /* * Do nothing when PROMOTE was already written for this term * (synchronous replication and leader election are in sync, and * both chose this node as a leader). */ - if (txn_limbo_replica_term(&txn_limbo, instance_id) == + if (!demote && txn_limbo_replica_term(&txn_limbo, instance_id) == box_raft()->term) return 0; @@ -1674,15 +1680,22 @@ box_promote(void) rc = -1; } else { promote: - if (try_wait) { + if (try_wait || demote) { raft_new_term(box_raft()); if (box_raft_wait_persisted() < 0) return -1; } uint64_t term = box_raft()->term; - txn_limbo_write_promote(&txn_limbo, wait_lsn, term); + if (demote) { + txn_limbo_write_demote(&txn_limbo, wait_lsn, + term); + } else { + txn_limbo_write_promote(&txn_limbo, wait_lsn, + term); + } + uint16_t type = demote ? IPROTO_DEMOTE : IPROTO_PROMOTE; struct synchro_request req = { - .type = IPROTO_PROMOTE, + .type = type, .replica_id = former_leader_id, .origin_id = instance_id, .lsn = wait_lsn, @@ -1695,6 +1708,25 @@ promote: return rc; } +int +box_promote(void) +{ + return box_clear_synchro_queue(false); +} + +int +box_demote(void) +{ + if (txn_limbo.owner_id == REPLICA_ID_NIL) + return 0; + if (txn_limbo.owner_id != instance_id) { + diag_set(ClientError, ER_SYNC_QUEUE_FOREIGN, + txn_limbo.owner_id); + return -1; + } + return box_clear_synchro_queue(true); +} + int box_listen(void) { diff --git a/src/box/box.h b/src/box/box.h index ecf32240d..aaf20d9dd 100644 --- a/src/box/box.h +++ b/src/box/box.h @@ -276,6 +276,9 @@ typedef struct tuple box_tuple_t; int box_promote(void); +int +box_demote(void); + /* box_select is private and used only by FFI */ API_EXPORT int box_select(uint32_t space_id, uint32_t index_id, diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h index 137bee9da..3c9edb7d2 100644 --- a/src/box/iproto_constants.h +++ b/src/box/iproto_constants.h @@ -241,6 +241,8 @@ enum iproto_type { IPROTO_RAFT = 30, /** PROMOTE request. */ IPROTO_PROMOTE = 31, + /** DEMOTE request. */ + IPROTO_DEMOTE = 32, /** A confirmation message for synchronous transactions. */ IPROTO_CONFIRM = 40, @@ -310,6 +312,8 @@ iproto_type_name(uint16_t type) return "RAFT"; case IPROTO_PROMOTE: return "PROMOTE"; + case IPROTO_DEMOTE: + return "DEMOTE"; case IPROTO_CONFIRM: return "CONFIRM"; case IPROTO_ROLLBACK: @@ -364,14 +368,14 @@ static inline bool iproto_type_is_synchro_request(uint16_t type) { return type == IPROTO_CONFIRM || type == IPROTO_ROLLBACK || - type == IPROTO_PROMOTE; + type == IPROTO_PROMOTE || type == IPROTO_DEMOTE; } -/** PROMOTE entry (synchronous replication and leader elections). */ +/** PROMOTE/DEMOTE entry (synchronous replication and leader elections). */ static inline bool iproto_type_is_promote_request(uint32_t type) { - return type == IPROTO_PROMOTE; + return type == IPROTO_PROMOTE || type == IPROTO_DEMOTE; } static inline bool diff --git a/src/box/lua/ctl.c b/src/box/lua/ctl.c index 368b9ab60..a613c4111 100644 --- a/src/box/lua/ctl.c +++ b/src/box/lua/ctl.c @@ -89,6 +89,14 @@ lbox_ctl_promote(struct lua_State *L) return 0; } +static int +lbox_ctl_demote(struct lua_State *L) +{ + if (box_demote() != 0) + return luaT_error(L); + return 0; +} + static int lbox_ctl_is_recovery_finished(struct lua_State *L) { @@ -127,6 +135,7 @@ static const struct luaL_Reg lbox_ctl_lib[] = { {"promote", lbox_ctl_promote}, /* An old alias. */ {"clear_synchro_queue", lbox_ctl_promote}, + {"demote", lbox_ctl_demote}, {"is_recovery_finished", lbox_ctl_is_recovery_finished}, {"set_on_shutdown_timeout", lbox_ctl_set_on_shutdown_timeout}, {NULL, NULL} diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index 996f1a3fc..d21f05557 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -508,6 +508,29 @@ txn_limbo_read_promote(struct txn_limbo *limbo, uint32_t replica_id, limbo->confirmed_lsn = 0; } +void +txn_limbo_write_demote(struct txn_limbo *limbo, int64_t lsn, uint64_t term) +{ + limbo->confirmed_lsn = lsn; + limbo->is_in_rollback = true; + struct txn_limbo_entry *e = txn_limbo_last_synchro_entry(limbo); + assert(e == NULL || e->lsn <= lsn); + (void)e; + txn_limbo_write_synchro(limbo, IPROTO_DEMOTE, lsn, term); + limbo->is_in_rollback = false; +} + +/** + * Process a DEMOTE request, which's like PROMOTE, but clears the limbo + * ownership. + * @sa txn_limbo_read_promote. + */ +static void +txn_limbo_read_demote(struct txn_limbo *limbo, int64_t lsn) +{ + return txn_limbo_read_promote(limbo, REPLICA_ID_NIL, lsn); +} + void txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn) { @@ -659,12 +682,13 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) vclock_follow(&limbo->promote_term_map, origin, term); if (term > limbo->promote_greatest_term) limbo->promote_greatest_term = term; - } else if (req->type == IPROTO_PROMOTE && + } else if (iproto_type_is_promote_request(req->type) && limbo->promote_greatest_term > 1) { /* PROMOTE for outdated term. Ignore. */ - say_info("RAFT: ignoring PROMOTE request from instance " + say_info("RAFT: ignoring %s request from instance " "id %u for term %llu. Greatest term seen " - "before (%llu) is bigger.", origin, (long long)term, + "before (%llu) is bigger.", + iproto_type_name(req->type), origin, (long long)term, (long long)limbo->promote_greatest_term); return; } @@ -675,7 +699,7 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) * The limbo was empty on the instance issuing the request. * This means this instance must empty its limbo as well. */ - assert(lsn == 0 && req->type == IPROTO_PROMOTE); + assert(lsn == 0 && iproto_type_is_promote_request(req->type)); } else if (req->replica_id != limbo->owner_id) { /* * Ignore CONFIRM/ROLLBACK messages for a foreign master. @@ -683,7 +707,7 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) * data from an old leader, who has just started and written * confirm right on synchronous transaction recovery. */ - if (req->type != IPROTO_PROMOTE) + if (!iproto_type_is_promote_request(req->type)) return; /* * Promote has a bigger term, and tries to steal the limbo. It @@ -703,6 +727,9 @@ txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) case IPROTO_PROMOTE: txn_limbo_read_promote(limbo, req->origin_id, lsn); break; + case IPROTO_DEMOTE: + txn_limbo_read_demote(limbo, lsn); + break; default: unreachable(); } diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h index e409ac657..801a1a0ee 100644 --- a/src/box/txn_limbo.h +++ b/src/box/txn_limbo.h @@ -318,6 +318,13 @@ txn_limbo_wait_confirm(struct txn_limbo *limbo); void txn_limbo_write_promote(struct txn_limbo *limbo, int64_t lsn, uint64_t term); +/** + * Write a DEMOTE request. + * It has the same effect as PROMOTE and additionally clears limbo ownership. + */ +void +txn_limbo_write_demote(struct txn_limbo *limbo, int64_t lsn, uint64_t term); + /** * Update qsync parameters dynamically. */ diff --git a/test/box/error.result b/test/box/error.result index dfe593dc2..8f783f927 100644 --- a/test/box/error.result +++ b/test/box/error.result @@ -445,6 +445,7 @@ t; | 224: box.error.RAFT_DISABLED | 225: box.error.TXN_ROLLBACK | 226: box.error.SYNC_QUEUE_UNCLAIMED + | 227: box.error.SYNC_QUEUE_FOREIGN | ... test_run:cmd("setopt delimiter ''"); diff --git a/test/replication/election_basic.result b/test/replication/election_basic.result index b64028c60..c0323a042 100644 --- a/test/replication/election_basic.result +++ b/test/replication/election_basic.result @@ -114,6 +114,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... -- -- See if bootstrap with election enabled works. diff --git a/test/replication/election_basic.test.lua b/test/replication/election_basic.test.lua index 77fdf6340..085a499d5 100644 --- a/test/replication/election_basic.test.lua +++ b/test/replication/election_basic.test.lua @@ -43,6 +43,7 @@ box.cfg{ election_mode = 'off', \ election_timeout = old_election_timeout \ } +box.ctl.demote() -- -- See if bootstrap with election enabled works. diff --git a/test/replication/election_qsync.result b/test/replication/election_qsync.result index c06400b38..2402c8578 100644 --- a/test/replication/election_qsync.result +++ b/test/replication/election_qsync.result @@ -165,6 +165,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/election_qsync.test.lua b/test/replication/election_qsync.test.lua index ea6fc4a61..e1aca8351 100644 --- a/test/replication/election_qsync.test.lua +++ b/test/replication/election_qsync.test.lua @@ -84,4 +84,5 @@ box.cfg{ replication = old_replication, \ replication_synchro_timeout = old_replication_synchro_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5140-qsync-casc-rollback.result b/test/replication/gh-5140-qsync-casc-rollback.result index da77631dd..d3208e1a4 100644 --- a/test/replication/gh-5140-qsync-casc-rollback.result +++ b/test/replication/gh-5140-qsync-casc-rollback.result @@ -73,6 +73,9 @@ _ = box.schema.space.create('async', {is_sync=false, engine = engine}) _ = _:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Write something to flush the master state to replica. box.space.sync:replace{1} | --- @@ -222,3 +225,6 @@ test_run:cmd('delete server replica') box.schema.user.revoke('guest', 'super') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5140-qsync-casc-rollback.test.lua b/test/replication/gh-5140-qsync-casc-rollback.test.lua index 69fc9ad02..96ddfd260 100644 --- a/test/replication/gh-5140-qsync-casc-rollback.test.lua +++ b/test/replication/gh-5140-qsync-casc-rollback.test.lua @@ -48,6 +48,7 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') _ = box.schema.space.create('async', {is_sync=false, engine = engine}) _ = _:create_index('pk') +box.ctl.promote() -- Write something to flush the master state to replica. box.space.sync:replace{1} @@ -103,3 +104,4 @@ test_run:cmd('stop server replica') test_run:cmd('delete server replica') box.schema.user.revoke('guest', 'super') +box.ctl.demote() diff --git a/test/replication/gh-5144-qsync-dup-confirm.result b/test/replication/gh-5144-qsync-dup-confirm.result index 9d265d9ff..217e44412 100644 --- a/test/replication/gh-5144-qsync-dup-confirm.result +++ b/test/replication/gh-5144-qsync-dup-confirm.result @@ -46,6 +46,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Remember the current LSN. In the end, when the following synchronous -- transaction is committed, result LSN should be this value +2: for the @@ -148,6 +151,9 @@ test_run:cmd('delete server replica2') | - true | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5144-qsync-dup-confirm.test.lua b/test/replication/gh-5144-qsync-dup-confirm.test.lua index 01a8351e0..1d6af2c62 100644 --- a/test/replication/gh-5144-qsync-dup-confirm.test.lua +++ b/test/replication/gh-5144-qsync-dup-confirm.test.lua @@ -19,6 +19,7 @@ box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = _:create_index('pk') +box.ctl.promote() -- Remember the current LSN. In the end, when the following synchronous -- transaction is committed, result LSN should be this value +2: for the @@ -69,4 +70,5 @@ test_run:cmd('delete server replica1') test_run:cmd('stop server replica2') test_run:cmd('delete server replica2') +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5163-qsync-restart-crash.result b/test/replication/gh-5163-qsync-restart-crash.result index e57bc76d1..1b4d3d9b5 100644 --- a/test/replication/gh-5163-qsync-restart-crash.result +++ b/test/replication/gh-5163-qsync-restart-crash.result @@ -16,6 +16,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.space.sync:replace{1} | --- @@ -30,3 +33,6 @@ box.space.sync:select{} box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5163-qsync-restart-crash.test.lua b/test/replication/gh-5163-qsync-restart-crash.test.lua index d5aca4749..c8d54aad2 100644 --- a/test/replication/gh-5163-qsync-restart-crash.test.lua +++ b/test/replication/gh-5163-qsync-restart-crash.test.lua @@ -7,8 +7,10 @@ engine = test_run:get_cfg('engine') -- _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.space.sync:replace{1} test_run:cmd('restart server default') box.space.sync:select{} box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-5167-qsync-rollback-snap.result b/test/replication/gh-5167-qsync-rollback-snap.result index 06f58526c..13166720f 100644 --- a/test/replication/gh-5167-qsync-rollback-snap.result +++ b/test/replication/gh-5167-qsync-rollback-snap.result @@ -41,6 +41,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Write something to flush the current master's state to replica. _ = box.space.sync:insert{1} | --- @@ -163,3 +166,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5167-qsync-rollback-snap.test.lua b/test/replication/gh-5167-qsync-rollback-snap.test.lua index 475727e61..1a2a31b7c 100644 --- a/test/replication/gh-5167-qsync-rollback-snap.test.lua +++ b/test/replication/gh-5167-qsync-rollback-snap.test.lua @@ -16,6 +16,7 @@ fiber = require('fiber') box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Write something to flush the current master's state to replica. _ = box.space.sync:insert{1} _ = box.space.sync:delete{1} @@ -65,3 +66,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5195-qsync-replica-write.result b/test/replication/gh-5195-qsync-replica-write.result index 85e00e6ed..bc73bb599 100644 --- a/test/replication/gh-5195-qsync-replica-write.result +++ b/test/replication/gh-5195-qsync-replica-write.result @@ -40,6 +40,9 @@ _ = box.schema.space.create('sync', {engine = engine, is_sync = true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.cfg{replication_synchro_timeout = 1000, replication_synchro_quorum = 3} | --- @@ -71,12 +74,12 @@ test_run:wait_lsn('replica', 'default') | --- | ... -- Normal DML is blocked - the limbo is not empty and does not belong to the --- replica. But synchro queue cleanup also does a WAL write, and propagates LSN +-- replica. But promote also does a WAL write, and propagates LSN -- of the instance. box.cfg{replication_synchro_timeout = 0.001} | --- | ... -box.ctl.clear_synchro_queue() +box.ctl.promote() | --- | ... @@ -144,6 +147,9 @@ test_run:cmd('delete server replica') | - true | ... +box.ctl.demote() + | --- + | ... box.space.sync:drop() | --- | ... diff --git a/test/replication/gh-5195-qsync-replica-write.test.lua b/test/replication/gh-5195-qsync-replica-write.test.lua index 64c48be99..a59ec154e 100644 --- a/test/replication/gh-5195-qsync-replica-write.test.lua +++ b/test/replication/gh-5195-qsync-replica-write.test.lua @@ -17,6 +17,7 @@ test_run:cmd('start server replica with wait=True, wait_load=True') -- _ = box.schema.space.create('sync', {engine = engine, is_sync = true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.cfg{replication_synchro_timeout = 1000, replication_synchro_quorum = 3} lsn = box.info.lsn @@ -30,10 +31,10 @@ test_run:wait_cond(function() return box.info.lsn == lsn end) test_run:switch('replica') test_run:wait_lsn('replica', 'default') -- Normal DML is blocked - the limbo is not empty and does not belong to the --- replica. But synchro queue cleanup also does a WAL write, and propagates LSN +-- replica. But promote also does a WAL write, and propagates LSN -- of the instance. box.cfg{replication_synchro_timeout = 0.001} -box.ctl.clear_synchro_queue() +box.ctl.promote() test_run:switch('default') -- Wait second ACK receipt. @@ -59,6 +60,7 @@ test_run:switch('default') test_run:cmd('stop server replica') test_run:cmd('delete server replica') +box.ctl.demote() box.space.sync:drop() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5213-qsync-applier-order-3.result b/test/replication/gh-5213-qsync-applier-order-3.result index bcb18b5c0..e788eec77 100644 --- a/test/replication/gh-5213-qsync-applier-order-3.result +++ b/test/replication/gh-5213-qsync-applier-order-3.result @@ -45,6 +45,9 @@ s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... test_run:cmd('create server replica1 with rpl_master=default,\ script="replication/replica1.lua"') @@ -179,6 +182,9 @@ box.cfg{ -- Replica2 takes the limbo ownership and sends the transaction to the replica1. -- Along with the CONFIRM from the default node, which is still not applied -- on the replica1. +box.ctl.promote() + | --- + | ... fiber = require('fiber') | --- | ... @@ -261,3 +267,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5213-qsync-applier-order-3.test.lua b/test/replication/gh-5213-qsync-applier-order-3.test.lua index 37b569da7..304656de0 100644 --- a/test/replication/gh-5213-qsync-applier-order-3.test.lua +++ b/test/replication/gh-5213-qsync-applier-order-3.test.lua @@ -30,6 +30,7 @@ box.schema.user.grant('guest', 'super') s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() test_run:cmd('create server replica1 with rpl_master=default,\ script="replication/replica1.lua"') @@ -90,6 +91,7 @@ box.cfg{ -- Replica2 takes the limbo ownership and sends the transaction to the replica1. -- Along with the CONFIRM from the default node, which is still not applied -- on the replica1. +box.ctl.promote() fiber = require('fiber') f = fiber.new(function() box.space.test:replace{2} end) @@ -123,3 +125,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5213-qsync-applier-order.result b/test/replication/gh-5213-qsync-applier-order.result index a8c24c289..ba6cdab06 100644 --- a/test/replication/gh-5213-qsync-applier-order.result +++ b/test/replication/gh-5213-qsync-applier-order.result @@ -29,6 +29,9 @@ s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... test_run:cmd('create server replica with rpl_master=default,\ script="replication/gh-5213-replica.lua"') @@ -300,3 +303,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5213-qsync-applier-order.test.lua b/test/replication/gh-5213-qsync-applier-order.test.lua index f1eccfa84..39b1912e8 100644 --- a/test/replication/gh-5213-qsync-applier-order.test.lua +++ b/test/replication/gh-5213-qsync-applier-order.test.lua @@ -14,6 +14,7 @@ box.schema.user.grant('guest', 'super') s = box.schema.space.create('test', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() test_run:cmd('create server replica with rpl_master=default,\ script="replication/gh-5213-replica.lua"') @@ -120,3 +121,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5288-qsync-recovery.result b/test/replication/gh-5288-qsync-recovery.result index dc0babef6..704b71d93 100644 --- a/test/replication/gh-5288-qsync-recovery.result +++ b/test/replication/gh-5288-qsync-recovery.result @@ -12,6 +12,9 @@ s = box.schema.space.create('sync', {is_sync = true}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... s:insert{1} | --- | - [1] @@ -25,3 +28,6 @@ test_run:cmd('restart server default') box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5288-qsync-recovery.test.lua b/test/replication/gh-5288-qsync-recovery.test.lua index 00bff7b87..2455f7278 100644 --- a/test/replication/gh-5288-qsync-recovery.test.lua +++ b/test/replication/gh-5288-qsync-recovery.test.lua @@ -5,7 +5,9 @@ test_run = require('test_run').new() -- s = box.schema.space.create('sync', {is_sync = true}) _ = s:create_index('pk') +box.ctl.promote() s:insert{1} box.snapshot() test_run:cmd('restart server default') box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-5298-qsync-recovery-snap.result b/test/replication/gh-5298-qsync-recovery-snap.result index 922831552..0883fe5f5 100644 --- a/test/replication/gh-5298-qsync-recovery-snap.result +++ b/test/replication/gh-5298-qsync-recovery-snap.result @@ -17,6 +17,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... for i = 1, 10 do box.space.sync:replace{i} end | --- | ... @@ -98,3 +101,6 @@ box.space.sync:drop() box.space.loc:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5298-qsync-recovery-snap.test.lua b/test/replication/gh-5298-qsync-recovery-snap.test.lua index 187f60d75..084cde963 100644 --- a/test/replication/gh-5298-qsync-recovery-snap.test.lua +++ b/test/replication/gh-5298-qsync-recovery-snap.test.lua @@ -8,6 +8,7 @@ engine = test_run:get_cfg('engine') -- _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() for i = 1, 10 do box.space.sync:replace{i} end -- Local rows could affect this by increasing the signature. @@ -46,3 +47,4 @@ box.cfg{ } box.space.sync:drop() box.space.loc:drop() +box.ctl.demote() diff --git a/test/replication/gh-5426-election-on-off.result b/test/replication/gh-5426-election-on-off.result index 7444ef7f2..2bdc17ec6 100644 --- a/test/replication/gh-5426-election-on-off.result +++ b/test/replication/gh-5426-election-on-off.result @@ -168,6 +168,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5426-election-on-off.test.lua b/test/replication/gh-5426-election-on-off.test.lua index bdf06903b..6277e9ef2 100644 --- a/test/replication/gh-5426-election-on-off.test.lua +++ b/test/replication/gh-5426-election-on-off.test.lua @@ -69,4 +69,5 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5433-election-restart-recovery.result b/test/replication/gh-5433-election-restart-recovery.result index f8f32416e..ed63ff409 100644 --- a/test/replication/gh-5433-election-restart-recovery.result +++ b/test/replication/gh-5433-election-restart-recovery.result @@ -169,6 +169,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... box.schema.user.revoke('guest', 'super') | --- | ... diff --git a/test/replication/gh-5433-election-restart-recovery.test.lua b/test/replication/gh-5433-election-restart-recovery.test.lua index 4aff000bf..ae1f42c4d 100644 --- a/test/replication/gh-5433-election-restart-recovery.test.lua +++ b/test/replication/gh-5433-election-restart-recovery.test.lua @@ -84,4 +84,5 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() box.schema.user.revoke('guest', 'super') diff --git a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result index 2699231e5..20fab4072 100644 --- a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result +++ b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.result @@ -49,6 +49,9 @@ _ = box.schema.space.create('test', {is_sync=true}) _ = box.space.test:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Fill the limbo with pending entries. 3 mustn't receive them yet. test_run:cmd('stop server election_replica3') diff --git a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua index 03705d96c..ec0f1d77e 100644 --- a/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua +++ b/test/replication/gh-5435-qsync-clear-synchro-queue-commit-all.test.lua @@ -21,6 +21,7 @@ box.ctl.wait_rw() _ = box.schema.space.create('test', {is_sync=true}) _ = box.space.test:create_index('pk') +box.ctl.promote() -- Fill the limbo with pending entries. 3 mustn't receive them yet. test_run:cmd('stop server election_replica3') diff --git a/test/replication/gh-5438-raft-state.result b/test/replication/gh-5438-raft-state.result index 6985f026a..68b6bfad8 100644 --- a/test/replication/gh-5438-raft-state.result +++ b/test/replication/gh-5438-raft-state.result @@ -47,6 +47,9 @@ end) | ... -- Cleanup. +box.ctl.demote() + | --- + | ... box.cfg{election_mode = old_election_mode} | --- | ... diff --git a/test/replication/gh-5438-raft-state.test.lua b/test/replication/gh-5438-raft-state.test.lua index 60c3366c1..cf0f4ca23 100644 --- a/test/replication/gh-5438-raft-state.test.lua +++ b/test/replication/gh-5438-raft-state.test.lua @@ -22,6 +22,7 @@ test_run:wait_cond(function()\ end) -- Cleanup. +box.ctl.demote() box.cfg{election_mode = old_election_mode} test_run:cmd('stop server replica') test_run:cmd('delete server replica') diff --git a/test/replication/gh-5446-qsync-eval-quorum.result b/test/replication/gh-5446-qsync-eval-quorum.result index 5f83b248c..1173128a7 100644 --- a/test/replication/gh-5446-qsync-eval-quorum.result +++ b/test/replication/gh-5446-qsync-eval-quorum.result @@ -88,6 +88,9 @@ s = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = s:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Only one master node -> 1/2 + 1 = 1 s:insert{1} -- should pass @@ -343,3 +346,7 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... + diff --git a/test/replication/gh-5446-qsync-eval-quorum.test.lua b/test/replication/gh-5446-qsync-eval-quorum.test.lua index 6b9e324ed..b969df836 100644 --- a/test/replication/gh-5446-qsync-eval-quorum.test.lua +++ b/test/replication/gh-5446-qsync-eval-quorum.test.lua @@ -37,6 +37,7 @@ end -- Create a sync space we will operate on s = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = s:create_index('pk') +box.ctl.promote() -- Only one master node -> 1/2 + 1 = 1 s:insert{1} -- should pass @@ -135,3 +136,5 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() + diff --git a/test/replication/gh-5506-election-on-off.result b/test/replication/gh-5506-election-on-off.result index b8abd7ecd..a7f2b6a9c 100644 --- a/test/replication/gh-5506-election-on-off.result +++ b/test/replication/gh-5506-election-on-off.result @@ -138,3 +138,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5506-election-on-off.test.lua b/test/replication/gh-5506-election-on-off.test.lua index 476b00ec0..f8915c333 100644 --- a/test/replication/gh-5506-election-on-off.test.lua +++ b/test/replication/gh-5506-election-on-off.test.lua @@ -66,3 +66,4 @@ box.cfg{ election_mode = old_election_mode, \ replication_timeout = old_replication_timeout, \ } +box.ctl.demote() diff --git a/test/replication/gh-5566-final-join-synchro.result b/test/replication/gh-5566-final-join-synchro.result index a09882ba6..c5ae2f283 100644 --- a/test/replication/gh-5566-final-join-synchro.result +++ b/test/replication/gh-5566-final-join-synchro.result @@ -12,6 +12,9 @@ _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.schema.user.grant('guest', 'replication') | --- @@ -137,3 +140,6 @@ test_run:cleanup_cluster() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5566-final-join-synchro.test.lua b/test/replication/gh-5566-final-join-synchro.test.lua index 2db2c742f..25f411407 100644 --- a/test/replication/gh-5566-final-join-synchro.test.lua +++ b/test/replication/gh-5566-final-join-synchro.test.lua @@ -5,6 +5,7 @@ test_run = require('test_run').new() -- _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() box.schema.user.grant('guest', 'replication') box.schema.user.grant('guest', 'write', 'space', 'sync') @@ -59,3 +60,4 @@ box.cfg{\ box.space.sync:drop() test_run:cleanup_cluster() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/gh-5874-qsync-txn-recovery.result b/test/replication/gh-5874-qsync-txn-recovery.result index 73f903ca7..01328a9e3 100644 --- a/test/replication/gh-5874-qsync-txn-recovery.result +++ b/test/replication/gh-5874-qsync-txn-recovery.result @@ -31,6 +31,9 @@ sync = box.schema.create_space('sync', {is_sync = true, engine = engine}) _ = sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- The transaction fails, but is written to the log anyway. box.begin() async:insert{1} sync:insert{1} box.commit() @@ -160,3 +163,6 @@ sync:drop() loc:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-5874-qsync-txn-recovery.test.lua b/test/replication/gh-5874-qsync-txn-recovery.test.lua index f35eb68de..6ddf164ac 100644 --- a/test/replication/gh-5874-qsync-txn-recovery.test.lua +++ b/test/replication/gh-5874-qsync-txn-recovery.test.lua @@ -12,6 +12,7 @@ async = box.schema.create_space('async', {engine = engine}) _ = async:create_index('pk') sync = box.schema.create_space('sync', {is_sync = true, engine = engine}) _ = sync:create_index('pk') +box.ctl.promote() -- The transaction fails, but is written to the log anyway. box.begin() async:insert{1} sync:insert{1} box.commit() @@ -82,3 +83,4 @@ loc:select() async:drop() sync:drop() loc:drop() +box.ctl.demote() diff --git a/test/replication/gh-6032-promote-wal-write.result b/test/replication/gh-6032-promote-wal-write.result index 246c7974f..03112fb8d 100644 --- a/test/replication/gh-6032-promote-wal-write.result +++ b/test/replication/gh-6032-promote-wal-write.result @@ -67,3 +67,6 @@ box.cfg{\ box.space.sync:drop() | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-6032-promote-wal-write.test.lua b/test/replication/gh-6032-promote-wal-write.test.lua index 8c1859083..9a036a8b4 100644 --- a/test/replication/gh-6032-promote-wal-write.test.lua +++ b/test/replication/gh-6032-promote-wal-write.test.lua @@ -26,3 +26,4 @@ box.cfg{\ replication_synchro_timeout = replication_synchro_timeout,\ } box.space.sync:drop() +box.ctl.demote() diff --git a/test/replication/gh-6034-limbo-ownership.result b/test/replication/gh-6034-limbo-ownership.result new file mode 100644 index 000000000..e412b8d53 --- /dev/null +++ b/test/replication/gh-6034-limbo-ownership.result @@ -0,0 +1,189 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +fiber = require('fiber') + | --- + | ... + +-- +-- gh-6034: test that transactional limbo isn't accessible without a promotion. +-- +synchro_quorum = box.cfg.replication_synchro_quorum + | --- + | ... +election_mode = box.cfg.election_mode + | --- + | ... +box.cfg{replication_synchro_quorum = 1, election_mode='off'} + | --- + | ... + +_ = box.schema.space.create('async'):create_index('pk') + | --- + | ... +_ = box.schema.space.create('sync', {is_sync=true}):create_index('pk') + | --- + | ... + +-- Limbo is initially unclaimed, everyone is writeable. +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == 0) + | --- + | - true + | ... +box.space.async:insert{1} -- success. + | --- + | - [1] + | ... +-- Synchro spaces aren't writeable +box.space.sync:insert{1} -- error. + | --- + | - error: The synchronous transaction queue doesn't belong to any instance + | ... + +box.ctl.promote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == box.info.id) + | --- + | - true + | ... +box.space.sync:insert{1} -- success. + | --- + | - [1] + | ... + +-- Everyone but the limbo owner is read-only. +box.schema.user.grant('guest', 'replication') + | --- + | ... +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... +test_run:cmd('set variable rpl_listen to "replica.listen"') + | --- + | - true + | ... +orig_replication = box.cfg.replication + | --- + | ... +box.cfg{replication=rpl_listen} + | --- + | ... + +test_run:switch('replica') + | --- + | - true + | ... +assert(box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == test_run:eval('default', 'return box.info.id')[1]) + | --- + | - true + | ... +box.space.async:insert{2} -- failure. + | --- + | - error: Can't modify data because this instance is in read-only mode. + | ... + +-- Promotion on the other node. Default should become ro. +box.ctl.promote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == box.info.id) + | --- + | - true + | ... +box.space.sync:insert{2} -- success. + | --- + | - [2] + | ... + +test_run:switch('default') + | --- + | - true + | ... +assert(box.info.ro) + | --- + | - true + | ... +assert(box.info.synchro.queue.owner == test_run:eval('replica', 'return box.info.id')[1]) + | --- + | - true + | ... +box.space.sync:insert{3} -- failure. + | --- + | - error: Can't modify data because this instance is in read-only mode. + | ... + +box.ctl.promote() + | --- + | ... +box.ctl.demote() + | --- + | ... +assert(not box.info.ro) + | --- + | - true + | ... +box.space.sync:insert{3} -- still fails. + | --- + | - error: The synchronous transaction queue doesn't belong to any instance + | ... +assert(box.info.synchro.queue.owner == 0) + | --- + | - true + | ... +box.space.async:insert{3} -- success. + | --- + | - [3] + | ... + +-- Cleanup. +box.ctl.demote() + | --- + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +box.schema.user.revoke('guest', 'replication') + | --- + | ... +box.space.sync:drop() + | --- + | ... +box.space.async:drop() + | --- + | ... +box.cfg{\ + replication_synchro_quorum = synchro_quorum,\ + election_mode = election_mode,\ + replication = orig_replication,\ +} + | --- + | ... diff --git a/test/replication/gh-6034-limbo-ownership.test.lua b/test/replication/gh-6034-limbo-ownership.test.lua new file mode 100644 index 000000000..c17e12fa4 --- /dev/null +++ b/test/replication/gh-6034-limbo-ownership.test.lua @@ -0,0 +1,69 @@ +test_run = require('test_run').new() +fiber = require('fiber') + +-- +-- gh-6034: test that transactional limbo isn't accessible without a promotion. +-- +synchro_quorum = box.cfg.replication_synchro_quorum +election_mode = box.cfg.election_mode +box.cfg{replication_synchro_quorum = 1, election_mode='off'} + +_ = box.schema.space.create('async'):create_index('pk') +_ = box.schema.space.create('sync', {is_sync=true}):create_index('pk') + +-- Limbo is initially unclaimed, everyone is writeable. +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == 0) +box.space.async:insert{1} -- success. +-- Synchro spaces aren't writeable +box.space.sync:insert{1} -- error. + +box.ctl.promote() +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == box.info.id) +box.space.sync:insert{1} -- success. + +-- Everyone but the limbo owner is read-only. +box.schema.user.grant('guest', 'replication') +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') +test_run:cmd('set variable rpl_listen to "replica.listen"') +orig_replication = box.cfg.replication +box.cfg{replication=rpl_listen} + +test_run:switch('replica') +assert(box.info.ro) +assert(box.info.synchro.queue.owner == test_run:eval('default', 'return box.info.id')[1]) +box.space.async:insert{2} -- failure. + +-- Promotion on the other node. Default should become ro. +box.ctl.promote() +assert(not box.info.ro) +assert(box.info.synchro.queue.owner == box.info.id) +box.space.sync:insert{2} -- success. + +test_run:switch('default') +assert(box.info.ro) +assert(box.info.synchro.queue.owner == test_run:eval('replica', 'return box.info.id')[1]) +box.space.sync:insert{3} -- failure. + +box.ctl.promote() +box.ctl.demote() +assert(not box.info.ro) +box.space.sync:insert{3} -- still fails. +assert(box.info.synchro.queue.owner == 0) +box.space.async:insert{3} -- success. + +-- Cleanup. +box.ctl.demote() +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +box.schema.user.revoke('guest', 'replication') +box.space.sync:drop() +box.space.async:drop() +box.cfg{\ + replication_synchro_quorum = synchro_quorum,\ + election_mode = election_mode,\ + replication = orig_replication,\ +} diff --git a/test/replication/gh-6034-promote-bump-term.result b/test/replication/gh-6034-promote-bump-term.result index 20e352922..e5087507c 100644 --- a/test/replication/gh-6034-promote-bump-term.result +++ b/test/replication/gh-6034-promote-bump-term.result @@ -32,6 +32,9 @@ assert(box.info.election.term == term + 2) | ... -- Cleanup. +box.ctl.demote() + | --- + | ... box.cfg{election_mode=election_mode} | --- | ... diff --git a/test/replication/gh-6034-promote-bump-term.test.lua b/test/replication/gh-6034-promote-bump-term.test.lua index 5847dbb8f..af18853f3 100644 --- a/test/replication/gh-6034-promote-bump-term.test.lua +++ b/test/replication/gh-6034-promote-bump-term.test.lua @@ -13,4 +13,5 @@ box.ctl.promote() assert(box.info.election.term == term + 2) -- Cleanup. +box.ctl.demote() box.cfg{election_mode=election_mode} diff --git a/test/replication/gh-6057-qsync-confirm-async-no-wal.result b/test/replication/gh-6057-qsync-confirm-async-no-wal.result index 23c77729b..e7beefb2a 100644 --- a/test/replication/gh-6057-qsync-confirm-async-no-wal.result +++ b/test/replication/gh-6057-qsync-confirm-async-no-wal.result @@ -40,6 +40,10 @@ _ = s2:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... + errinj = box.error.injection | --- | ... @@ -161,3 +165,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua b/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua index a11ddc042..bb459ea02 100644 --- a/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua +++ b/test/replication/gh-6057-qsync-confirm-async-no-wal.test.lua @@ -21,6 +21,8 @@ _ = s:create_index('pk') s2 = box.schema.create_space('test2') _ = s2:create_index('pk') +box.ctl.promote() + errinj = box.error.injection function create_hanging_async_after_confirm(sync_key, async_key1, async_key2) \ @@ -86,3 +88,4 @@ box.cfg{ replication_synchro_quorum = old_synchro_quorum, \ replication_synchro_timeout = old_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/hang_on_synchro_fail.result b/test/replication/hang_on_synchro_fail.result index 9f6fac00b..dda15af20 100644 --- a/test/replication/hang_on_synchro_fail.result +++ b/test/replication/hang_on_synchro_fail.result @@ -19,6 +19,9 @@ _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... old_synchro_quorum = box.cfg.replication_synchro_quorum | --- @@ -127,4 +130,7 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/hang_on_synchro_fail.test.lua b/test/replication/hang_on_synchro_fail.test.lua index 6c3b09fab..f0d494eae 100644 --- a/test/replication/hang_on_synchro_fail.test.lua +++ b/test/replication/hang_on_synchro_fail.test.lua @@ -8,6 +8,7 @@ box.schema.user.grant('guest', 'replication') _ = box.schema.space.create('sync', {is_sync=true}) _ = box.space.sync:create_index('pk') +box.ctl.promote() old_synchro_quorum = box.cfg.replication_synchro_quorum box.cfg{replication_synchro_quorum=3} @@ -54,4 +55,5 @@ box.cfg{replication_synchro_quorum=old_synchro_quorum,\ replication_synchro_timeout=old_synchro_timeout} box.space.sync:drop() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/qsync_advanced.result b/test/replication/qsync_advanced.result index 94b19b1f2..72ac0c326 100644 --- a/test/replication/qsync_advanced.result +++ b/test/replication/qsync_advanced.result @@ -72,6 +72,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. box.space.sync:insert{1} -- success | --- @@ -468,6 +471,9 @@ box.space.sync:select{} -- 1 box.cfg{read_only=false} -- promote replica to master | --- | ... +box.ctl.promote() + | --- + | ... test_run:switch('default') | --- | - true @@ -508,6 +514,9 @@ test_run:switch('default') box.cfg{read_only=false} | --- | ... +box.ctl.promote() + | --- + | ... test_run:switch('replica') | --- | - true @@ -781,3 +790,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_advanced.test.lua b/test/replication/qsync_advanced.test.lua index 058ece602..37c285b8d 100644 --- a/test/replication/qsync_advanced.test.lua +++ b/test/replication/qsync_advanced.test.lua @@ -30,6 +30,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. box.space.sync:insert{1} -- success test_run:cmd('switch replica') @@ -170,6 +171,7 @@ box.space.sync:select{} -- 1 test_run:switch('replica') box.space.sync:select{} -- 1 box.cfg{read_only=false} -- promote replica to master +box.ctl.promote() test_run:switch('default') box.cfg{read_only=true} -- demote master to replica test_run:switch('replica') @@ -181,6 +183,7 @@ box.space.sync:select{} -- 1, 2 -- Revert cluster configuration. test_run:switch('default') box.cfg{read_only=false} +box.ctl.promote() test_run:switch('replica') box.cfg{read_only=true} -- Testcase cleanup. @@ -279,3 +282,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/qsync_basic.result b/test/replication/qsync_basic.result index 7e711ba13..bbdfc42fe 100644 --- a/test/replication/qsync_basic.result +++ b/test/replication/qsync_basic.result @@ -14,6 +14,9 @@ s1.is_sync pk = s1:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... box.begin() s1:insert({1}) s1:insert({2}) box.commit() | --- | ... @@ -645,19 +648,12 @@ test_run:switch('default') | --- | - true | ... -box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 1000} - | --- - | ... -f = fiber.create(function() box.space.sync:replace{1} end) +box.ctl.demote() | --- | ... -test_run:wait_lsn('replica', 'default') +box.space.sync:replace{1} | --- - | ... - -test_run:switch('replica') - | --- - | - true + | - error: The synchronous transaction queue doesn't belong to any instance | ... function skip_row() return nil end | --- @@ -674,26 +670,22 @@ box.space.sync:replace{2} box.space.sync:before_replace(nil, skip_row) | --- | ... -assert(box.space.sync:get{2} == nil) +assert(box.space.sync:get{1} == nil) | --- | - true | ... -assert(box.space.sync:get{1} ~= nil) +assert(box.space.sync:get{2} == nil) | --- | - true | ... - -test_run:switch('default') +assert(box.info.lsn == old_lsn + 1) | --- | - true | ... -box.cfg{replication_synchro_quorum = 2} +box.ctl.promote() | --- | ... -test_run:wait_cond(function() return f:status() == 'dead' end) - | --- - | - true - | ... + box.space.sync:truncate() | --- | ... @@ -758,3 +750,6 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'replication') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_basic.test.lua b/test/replication/qsync_basic.test.lua index 75c9b222b..eac465e25 100644 --- a/test/replication/qsync_basic.test.lua +++ b/test/replication/qsync_basic.test.lua @@ -6,6 +6,7 @@ s1 = box.schema.create_space('test1', {is_sync = true}) s1.is_sync pk = s1:create_index('pk') +box.ctl.promote() box.begin() s1:insert({1}) s1:insert({2}) box.commit() s1:select{} @@ -253,22 +254,18 @@ box.space.sync:count() -- instances, but also works for local rows. -- test_run:switch('default') -box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 1000} -f = fiber.create(function() box.space.sync:replace{1} end) -test_run:wait_lsn('replica', 'default') - -test_run:switch('replica') +box.ctl.demote() +box.space.sync:replace{1} function skip_row() return nil end old_lsn = box.info.lsn _ = box.space.sync:before_replace(skip_row) box.space.sync:replace{2} box.space.sync:before_replace(nil, skip_row) +assert(box.space.sync:get{1} == nil) assert(box.space.sync:get{2} == nil) -assert(box.space.sync:get{1} ~= nil) +assert(box.info.lsn == old_lsn + 1) +box.ctl.promote() -test_run:switch('default') -box.cfg{replication_synchro_quorum = 2} -test_run:wait_cond(function() return f:status() == 'dead' end) box.space.sync:truncate() -- @@ -301,3 +298,4 @@ test_run:cmd('delete server replica') box.space.test:drop() box.space.sync:drop() box.schema.user.revoke('guest', 'replication') +box.ctl.demote() diff --git a/test/replication/qsync_errinj.result b/test/replication/qsync_errinj.result index 635bcf939..cf1e30a90 100644 --- a/test/replication/qsync_errinj.result +++ b/test/replication/qsync_errinj.result @@ -35,6 +35,9 @@ _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- -- gh-5100: slow ACK sending shouldn't stun replica for the @@ -542,3 +545,6 @@ box.space.sync:drop() box.schema.user.revoke('guest', 'super') | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_errinj.test.lua b/test/replication/qsync_errinj.test.lua index 6a9fd3e1a..e7c85c58c 100644 --- a/test/replication/qsync_errinj.test.lua +++ b/test/replication/qsync_errinj.test.lua @@ -12,6 +12,7 @@ test_run:cmd('start server replica with wait=True, wait_load=True') _ = box.schema.space.create('sync', {is_sync = true, engine = engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- -- gh-5100: slow ACK sending shouldn't stun replica for the @@ -222,3 +223,4 @@ test_run:cmd('delete server replica') box.space.sync:drop() box.schema.user.revoke('guest', 'super') +box.ctl.demote() diff --git a/test/replication/qsync_snapshots.result b/test/replication/qsync_snapshots.result index cafdd63c8..ca418b168 100644 --- a/test/replication/qsync_snapshots.result +++ b/test/replication/qsync_snapshots.result @@ -57,6 +57,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. box.space.sync:insert{1} | --- @@ -299,3 +302,6 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... diff --git a/test/replication/qsync_snapshots.test.lua b/test/replication/qsync_snapshots.test.lua index 590610974..82c2e3f7c 100644 --- a/test/replication/qsync_snapshots.test.lua +++ b/test/replication/qsync_snapshots.test.lua @@ -23,6 +23,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. box.space.sync:insert{1} box.space.sync:select{} -- 1 @@ -130,3 +131,4 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() diff --git a/test/replication/qsync_with_anon.result b/test/replication/qsync_with_anon.result index 6a2952a32..99c6fb902 100644 --- a/test/replication/qsync_with_anon.result +++ b/test/replication/qsync_with_anon.result @@ -57,6 +57,9 @@ _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') | --- | ... +box.ctl.promote() + | --- + | ... -- Testcase body. test_run:switch('default') | --- @@ -220,6 +223,9 @@ box.cfg{ } | --- | ... +box.ctl.demote() + | --- + | ... test_run:cleanup_cluster() | --- | ... diff --git a/test/replication/qsync_with_anon.test.lua b/test/replication/qsync_with_anon.test.lua index d7ecaa107..e73880ec7 100644 --- a/test/replication/qsync_with_anon.test.lua +++ b/test/replication/qsync_with_anon.test.lua @@ -22,6 +22,7 @@ test_run:switch('default') box.cfg{replication_synchro_quorum=NUM_INSTANCES, replication_synchro_timeout=1000} _ = box.schema.space.create('sync', {is_sync=true, engine=engine}) _ = box.space.sync:create_index('pk') +box.ctl.promote() -- Testcase body. test_run:switch('default') box.space.sync:insert{1} -- success @@ -81,4 +82,5 @@ box.cfg{ replication_synchro_quorum = orig_synchro_quorum, \ replication_synchro_timeout = orig_synchro_timeout, \ } +box.ctl.demote() test_run:cleanup_cluster() diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg index eb88b9420..977deeb40 100644 --- a/test/replication/suite.cfg +++ b/test/replication/suite.cfg @@ -47,6 +47,7 @@ "gh-5613-bootstrap-prefer-booted.test.lua": {}, "gh-6027-applier-error-show.test.lua": {}, "gh-6032-promote-wal-write.test.lua": {}, + "gh-6034-limbo-ownership.test.lua": {}, "gh-6034-promote-bump-term.test.lua": {}, "gh-6057-qsync-confirm-async-no-wal.test.lua": {}, "gh-6094-rs-uuid-mismatch.test.lua": {}, -- 2.30.1 (Apple Git-130)