From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 1400B6EC55; Thu, 22 Jul 2021 10:20:12 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1400B6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626938412; bh=47s5ar0uaJkLEh/G768zKHRxQk1TFkOhSMoYO28lUe4=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=vA0tkvpSou3bIZv6MvI+t/DgbVImAK3y6EOqlRWFOaH+Xzs3QYT79Q5pXSSwB1GMn 31HfcGYWZyJqTTcjQ6ZBxzRQadogxUth3spfm2tmDKulAtLH9pQ0sAKmRas02wKcgn 0Hs/qKy4PkzsRaHadUtKuYPUxUyqw8VZ7UGOVhvY= Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 6E4E86EC55 for ; Thu, 22 Jul 2021 10:20:10 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6E4E86EC55 Received: by smtp41.i.mail.ru with esmtpa (envelope-from ) id 1m6T05-0000UE-29; Thu, 22 Jul 2021 10:20:09 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Thu, 22 Jul 2021 09:20:02 +0200 Message-Id: <20210722072002.13145-1-sergepetrenko@tarantool.org> X-Mailer: git-send-email 2.30.1 (Apple Git-130) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C320FB367CC9CBEE800EEBD3117ABA9A5B182A05F5380850400C94B3520E9DE6697BBBBEDAC7D82987D31A5B8906A8469C8F5A9E6E38E3ABDB X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78BEFA404F4C39CF2EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637225475490A77DC518638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8E08E26D18EC9C92840ACBB154736F2CB117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE902A1BE408319B293D5BA627BF9F2FCFD8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE397AD43380FEE24CA2D242C3BD2E3F4C6C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F790063788B3B24285A3CD0EEFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C16CF367659D795C079DF2E1256B1F83CBDC67A6C8D72D2799C2B6934AE262D3EE7EAB7254005DCED3138078569B97AEB1E0A4E2319210D9B64D260DF9561598F01A9E91200F654B00191F08AA555D0B08E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D348F1757F8DFBC5C33428CBE4682D8E6B27329065D4ACB659886CB3E794E0C27DBF3450CBC1C51F58F1D7E09C32AA3244CA926D85B21028AF28CAC8799313A9D555595C85A795C7BAE927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXgkk+wQSK0KogRBn/8xZCJ+ X-Mailru-Sender: 9482C2B233321BD2BB92BEF992491AAEDB9D0164DB15ACFC8AD269484E3D6D03EC09A834B12B1ADCF8074692E4F50D75C77752E0C033A69E6F2E6AD49324154B9149A12A345FCA4B6C18EFA0BB12DBB0 X-Mras: Ok Subject: [Tarantool-patches] [PATCH] box: introduce on_election triggers X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" On_election triggers are fired asynchronously after any Raft event with a broadcast, they are run in a worker fiber, so it's allowed to yield inside them, unlike Raft's on_update triggers we already had. Closes #5819 @TarantoolBot document Title: document triggers on election state change A new function to register triggers is added, `box.ctl.on_election()`. Triggers registered via this function are run asynchronously every time a visible change in `box.info.election` table appears. No parameters are passed to the trigger, it may check what's changed by looking at `box.info.election` and `box.info.synchro`. --- I'm not sure whether every change in `box.info.election.vote` actually runs the triggers. That's because I run triggers inside of box_raft_broadcast(), which isn't triggered on vote change. I thought of adding raft_schedule_trigger as a counterpart to raft_schedule_broadcast. Then raft_sm_schedule_new_vote() would have raft_schedule_trigger() instead of raft_schedule_broadcast(), and the triggers would fire on a vote change as well. https://github.com/tarantool/tarantool/issues/5819 https://github.com/tarantool/tarantool/tree/sp/gh-5819-raft-state-triggers .../unreleased/gh-5819-election-triggers.md | 5 + src/box/lua/ctl.c | 8 + src/box/raft.c | 3 + src/box/raft.h | 7 + .../gh-5819-election-triggers.result | 147 ++++++++++++++++++ .../gh-5819-election-triggers.test.lua | 53 +++++++ test/replication/suite.cfg | 1 + 7 files changed, 224 insertions(+) create mode 100644 changelogs/unreleased/gh-5819-election-triggers.md create mode 100644 test/replication/gh-5819-election-triggers.result create mode 100644 test/replication/gh-5819-election-triggers.test.lua diff --git a/changelogs/unreleased/gh-5819-election-triggers.md b/changelogs/unreleased/gh-5819-election-triggers.md new file mode 100644 index 000000000..edb7be8ac --- /dev/null +++ b/changelogs/unreleased/gh-5819-election-triggers.md @@ -0,0 +1,5 @@ +## feature/replication + +* Introduce on_election triggers. The triggers may be registered via +`box.ctl.on_election()` interface and are run asynchronously each time +`box.info.election` changes (gh-5819). diff --git a/src/box/lua/ctl.c b/src/box/lua/ctl.c index 368b9ab60..4b2a467d8 100644 --- a/src/box/lua/ctl.c +++ b/src/box/lua/ctl.c @@ -43,6 +43,7 @@ #include "box/schema.h" #include "box/engine.h" #include "box/memtx_engine.h" +#include "box/raft.h" static int lbox_ctl_wait_ro(struct lua_State *L) @@ -81,6 +82,12 @@ lbox_ctl_on_schema_init(struct lua_State *L) return lbox_trigger_reset(L, 2, &on_schema_init, NULL, NULL); } +static int +lbox_ctl_on_election(struct lua_State *L) +{ + return lbox_trigger_reset(L, 2, &box_raft_on_state, NULL, NULL); +} + static int lbox_ctl_promote(struct lua_State *L) { @@ -124,6 +131,7 @@ static const struct luaL_Reg lbox_ctl_lib[] = { {"wait_rw", lbox_ctl_wait_rw}, {"on_shutdown", lbox_ctl_on_shutdown}, {"on_schema_init", lbox_ctl_on_schema_init}, + {"on_election", lbox_ctl_on_election}, {"promote", lbox_ctl_promote}, /* An old alias. */ {"clear_synchro_queue", lbox_ctl_promote}, diff --git a/src/box/raft.c b/src/box/raft.c index 7f787c0c5..c9c3ba818 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -52,6 +52,8 @@ enum election_mode box_election_mode = ELECTION_MODE_INVALID; */ static struct trigger box_raft_on_update; +struct rlist box_raft_on_state = RLIST_HEAD_INITIALIZER(box_raft_on_state); + /** * Worker fiber does all the asynchronous work, which may need yields and can be * long. These are WAL writes, network broadcasts. That allows not to block the @@ -274,6 +276,7 @@ box_raft_broadcast(struct raft *raft, const struct raft_msg *msg) assert(raft == box_raft()); struct raft_request req; box_raft_msg_to_request(msg, &req); + trigger_run(&box_raft_on_state, NULL); replicaset_foreach(replica) relay_push_raft(replica->relay, &req); } diff --git a/src/box/raft.h b/src/box/raft.h index 6b6136510..dbe16eebf 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -30,11 +30,18 @@ * SUCH DAMAGE. */ #include "raft/raft.h" +#include "small/rlist.h" #if defined(__cplusplus) extern "C" { #endif +/** + * A public trigger fired on Raft state change. It's allowed to yield inside + * it, and it's run asynchronously. + */ +extern struct rlist box_raft_on_state; + enum election_mode { ELECTION_MODE_INVALID = -1, ELECTION_MODE_OFF = 0, diff --git a/test/replication/gh-5819-election-triggers.result b/test/replication/gh-5819-election-triggers.result new file mode 100644 index 000000000..546aac8a2 --- /dev/null +++ b/test/replication/gh-5819-election-triggers.result @@ -0,0 +1,147 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +box.schema.user.grant('guest', 'replication') + | --- + | ... +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica') + | --- + | - true + | ... + +repl = test_run:eval('replica', 'return box.cfg.listen')[1] + | --- + | ... +box.cfg{replication = repl} + | --- + | ... +test_run:switch('replica') + | --- + | - true + | ... +box.cfg{election_mode='voter'} + | --- + | ... +test_run:switch('default') + | --- + | - true + | ... + +election_tbl = {} + | --- + | ... + +function trig()\ + election_tbl[#election_tbl+1] = box.info.election\ +end + | --- + | ... + +_ = box.ctl.on_election(trig) + | --- + | ... + +box.cfg{replication_synchro_quorum=2} + | --- + | ... +box.cfg{election_mode='candidate'} + | --- + | ... + +test_run:wait_cond(function() return box.info.election.state == 'leader' end) + | --- + | - true + | ... +assert(#election_tbl == 3) + | --- + | - true + | ... +assert(election_tbl[1].state == 'follower') + | --- + | - true + | ... +assert(election_tbl[2].state == 'candidate') + | --- + | - true + | ... +assert(election_tbl[2].vote == 1) + | --- + | - true + | ... +assert(election_tbl[3].state == 'leader') + | --- + | - true + | ... +box.cfg{election_mode='voter'} + | --- + | ... + +assert(#election_tbl == 4) + | --- + | - true + | ... +assert(election_tbl[4].state == 'follower') + | --- + | - true + | ... + +box.cfg{election_mode='off'} + | --- + | ... +assert(#election_tbl == 5) + | --- + | - true + | ... + +box.cfg{election_mode='manual'} + | --- + | ... + +assert(#election_tbl == 6) + | --- + | - true + | ... +assert(election_tbl[6].state == 'follower') + | --- + | - true + | ... + +box.ctl.promote() + | --- + | ... + +assert(#election_tbl == 9) + | --- + | - true + | ... +assert(election_tbl[7].state == 'follower') + | --- + | - true + | ... +assert(election_tbl[8].state == 'candidate') + | --- + | - true + | ... +assert(election_tbl[9].state == 'leader') + | --- + | - true + | ... + +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +box.schema.user.revoke('guest', 'replication') + | --- + | ... diff --git a/test/replication/gh-5819-election-triggers.test.lua b/test/replication/gh-5819-election-triggers.test.lua new file mode 100644 index 000000000..5f61a59f9 --- /dev/null +++ b/test/replication/gh-5819-election-triggers.test.lua @@ -0,0 +1,53 @@ +test_run = require('test_run').new() + +box.schema.user.grant('guest', 'replication') +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica') + +repl = test_run:eval('replica', 'return box.cfg.listen')[1] +box.cfg{replication = repl} +test_run:switch('replica') +box.cfg{election_mode='voter'} +test_run:switch('default') + +election_tbl = {} + +function trig()\ + election_tbl[#election_tbl+1] = box.info.election\ +end + +_ = box.ctl.on_election(trig) + +box.cfg{replication_synchro_quorum=2} +box.cfg{election_mode='candidate'} + +test_run:wait_cond(function() return box.info.election.state == 'leader' end) +assert(#election_tbl == 3) +assert(election_tbl[1].state == 'follower') +assert(election_tbl[2].state == 'candidate') +assert(election_tbl[2].vote == 1) +assert(election_tbl[3].state == 'leader') +box.cfg{election_mode='voter'} + +assert(#election_tbl == 4) +assert(election_tbl[4].state == 'follower') + +box.cfg{election_mode='off'} +assert(#election_tbl == 5) + +box.cfg{election_mode='manual'} + +assert(#election_tbl == 6) +assert(election_tbl[6].state == 'follower') + +box.ctl.promote() + +assert(#election_tbl == 9) +assert(election_tbl[7].state == 'follower') +assert(election_tbl[8].state == 'candidate') +assert(election_tbl[9].state == 'leader') + +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +box.schema.user.revoke('guest', 'replication') diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg index a51a2d51a..f2af00038 100644 --- a/test/replication/suite.cfg +++ b/test/replication/suite.cfg @@ -46,6 +46,7 @@ "gh-5536-wal-limit.test.lua": {}, "gh-5566-final-join-synchro.test.lua": {}, "gh-5613-bootstrap-prefer-booted.test.lua": {}, + "gh-5819-election-triggers.test.lua": {}, "gh-6027-applier-error-show.test.lua": {}, "gh-6032-promote-wal-write.test.lua": {}, "gh-6057-qsync-confirm-async-no-wal.test.lua": {}, -- 2.30.1 (Apple Git-130)