* [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed
@ 2021-03-26 12:06 Cyrill Gorcunov via Tarantool-patches
2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 UTC (permalink / raw)
To: tml; +Cc: Vladislav Shpilevoy
Take a look please.
v2:
- rebase code to the fresh master branch
- keep wal_cleanup_delay option name
- pass wal_cleanup_delay as an option to gc_init, so it
won't be dependent on cfg engine
- add comment about gc_delay_unref in plain bootstrap mode
- allow to setup wal_cleanup_delay dynamically
- update comment in gc_wait_cleanup and call it conditionally
- declare wal_cleanup_delay as a double
- rename gc.cleanup_is_paused to gc.is_paused and update output
- do not show ref counter in box.info.gc() output
- update documentation
- move gc_delay_unref inside relay_subscribe call which runs
in tx context (instead of relay's context)
- update tests:
- add a comment why we need a temp space on replica node
- use explicit insert/snapshot operations
- shrkink the number of insert/snapshot to speedup testing
- use "restart" instead of stop/start pair
- use wait_log helper instead of own function
- add is_paused test
v3:
- fix changelog
- rework box_check_wal_cleanup_delay, the replication_anon
setting is considered only in box_set_wal_cleanup_delay,
ie when config is checked and parsed, moreover the order
of setup is set to be behind "replication_anon" option
processing
- delay cycle now considers deadline instead of per cycle
calculation
- use `double` type for timestamp
- test update
- verify `.is_paused` value
- minimize number of inserts
- no need to use temporary space, regular space works as well
- add comments on why we should restart the master node
v4:
- drop argument from gc_init(), since we're configuring delay
value from load_cfg.lua script there is no need to read the
delay early, simply start gc paused and unpause it on demand
- move unpause message to main wait cycle
- test update:
- verify tests and fix replication/replica_rejoin since it waits
for xlogs to be cleaned up too early
- use 10 seconds for XlogGapError instead of 0.1 second, this is
a common deadline value
v5:
- define limits for `wal_cleanup_delay`: it should be either 0,
or in range [0.001; TIMEOUT_INFINITY]. This is done to not consider
fp epsilon as a meaningul value
- fix comment about why anon replica is not using delay
- rework cleanup delay'ed cycle
- test update:
- update vinyl/replica_rejoin -- we need to disable cleanup
delay explicitly
- update replication/replica_rejoin for same reason
- drop unneded test_run:switch() calls
- add a testcase where timeout is decreased and cleanup
fiber is kicked to run even with stuck replica
issue https://github.com/tarantool/tarantool/issues/5806
branch gorcunov/gh-5806-xlog-gc-5
Cyrill Gorcunov (3):
gc/xlog: delay xlog cleanup until relays are subscribed
test: add a test for wal_cleanup_delay option
test: box-tap/gc -- add test for is_paused field
.../unreleased/add-wal_cleanup_delay.md | 5 +
src/box/box.cc | 41 ++
src/box/box.h | 1 +
src/box/gc.c | 95 +++-
src/box/gc.h | 36 ++
src/box/lua/cfg.cc | 9 +
src/box/lua/info.c | 4 +
src/box/lua/load_cfg.lua | 5 +
src/box/relay.cc | 1 +
src/box/replication.cc | 2 +
test/app-tap/init_script.result | 1 +
test/box-tap/gc.test.lua | 3 +-
test/box/admin.result | 2 +
test/box/cfg.result | 4 +
test/replication/gh-5806-master.lua | 8 +
test/replication/gh-5806-slave.lua | 8 +
test/replication/gh-5806-xlog-cleanup.result | 435 ++++++++++++++++++
.../replication/gh-5806-xlog-cleanup.test.lua | 188 ++++++++
test/replication/replica_rejoin.lua | 22 +
test/replication/replica_rejoin.result | 18 +-
test/replication/replica_rejoin.test.lua | 11 +-
test/vinyl/replica_rejoin.lua | 5 +-
test/vinyl/replica_rejoin.result | 13 +
test/vinyl/replica_rejoin.test.lua | 8 +
24 files changed, 916 insertions(+), 9 deletions(-)
create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md
create mode 100644 test/replication/gh-5806-master.lua
create mode 100644 test/replication/gh-5806-slave.lua
create mode 100644 test/replication/gh-5806-xlog-cleanup.result
create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua
create mode 100644 test/replication/replica_rejoin.lua
base-commit: f4e248c0c13a46beee238fbebc38ef687ef09d02
--
2.30.2
^ permalink raw reply [flat|nested] 12+ messages in thread* [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 12:06 [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:42 ` Serge Petrenko via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option Cyrill Gorcunov via Tarantool-patches ` (2 subsequent siblings) 3 siblings, 2 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 UTC (permalink / raw) To: tml; +Cc: Vladislav Shpilevoy In case if replica managed to be far behind the master node (so there are a number of xlog files present after the last master's snapshot) then once master node get restarted it may clean up the xlogs needed by the replica to subscribe in a fast way and instead the replica will have to rejoin reading a number of data back. Lets try to address this by delaying xlog files cleanup until replicas are got subscribed and relays are up and running. For this sake we start with cleanup fiber spinning in nop cycle ("paused" mode) and use a delay counter to wait until relays decrement them. This implies that if `_cluster` system space is not empty upon restart and the registered replica somehow vanished completely and won't ever come back, then the node administrator has to drop this replica from `_cluster` manually. Note that this delayed cleanup start doesn't prevent WAL engine from removing old files if there is no space left on a storage device. The WAL will simply drop old data without a question. We need to take into account that some administrators might not need this functionality at all, for this sake we introduce "wal_cleanup_delay" configuration option which allows to enable or disable the delay. Closes #5806 Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com> @TarantoolBot document Title: Add wal_cleanup_delay configuration parameter The `wal_cleanup_delay` option defines a delay in seconds before write ahead log files (`*.xlog`) are getting started to prune upon a node restart. This option is ignored in case if a node is running as an anonymous replica (`replication_anon = true`). Similarly if replication is unused or there is no plans to use replication at all then this option should not be considered. An initial problem to solve is the case where a node is operating so fast that its replicas do not manage to reach the node state and in case if the node is restarted at this moment (for various reasons, for example due to power outage) then `*.xlog` files might be pruned during restart. In result replicas will not find these files on the main node and have to reread all data back which is a very expensive procedure. Since replicas are tracked via `_cluster` system space this we use its content to count subscribed replicas and when all of them are up and running the cleanup procedure is automatically enabled even if `wal_cleanup_delay` is not expired. The `wal_cleanup_delay` should be set to: - `0` to disable the cleanup delay; - `>= 0` to wait for specified number of seconds. By default it is set to `14400` seconds (ie `4` hours). In case if registered replica is lost forever and timeout is set to infinity then a preferred way to enable cleanup procedure is not setting up a small timeout value but rather to delete this replica from `_cluster` space manually. Note that the option does *not* prevent WAL engine from removing old `*.xlog` files if there is no space left on a storage device, WAL engine can remove them in a force way. Current state of `*.xlog` garbage collector can be found in `box.info.gc()` output. For example ``` Lua tarantool> box.info.gc() --- ... is_paused: false ``` The `is_paused` shows if cleanup fiber is paused or not. --- .../unreleased/add-wal_cleanup_delay.md | 5 + src/box/box.cc | 41 ++++++++ src/box/box.h | 1 + src/box/gc.c | 95 ++++++++++++++++++- src/box/gc.h | 36 +++++++ src/box/lua/cfg.cc | 9 ++ src/box/lua/info.c | 4 + src/box/lua/load_cfg.lua | 5 + src/box/relay.cc | 1 + src/box/replication.cc | 2 + test/app-tap/init_script.result | 1 + test/box/admin.result | 2 + test/box/cfg.result | 4 + test/replication/replica_rejoin.lua | 22 +++++ test/replication/replica_rejoin.result | 18 +++- test/replication/replica_rejoin.test.lua | 11 ++- test/vinyl/replica_rejoin.lua | 5 +- test/vinyl/replica_rejoin.result | 13 +++ test/vinyl/replica_rejoin.test.lua | 8 ++ 19 files changed, 275 insertions(+), 8 deletions(-) create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md create mode 100644 test/replication/replica_rejoin.lua diff --git a/changelogs/unreleased/add-wal_cleanup_delay.md b/changelogs/unreleased/add-wal_cleanup_delay.md new file mode 100644 index 000000000..2e67f0a4e --- /dev/null +++ b/changelogs/unreleased/add-wal_cleanup_delay.md @@ -0,0 +1,5 @@ +## bugfix/core + +* Introduce `wal_cleanup_delay` option to prevent early cleanup + of `*.xlog` files which are needed by replicas and lead to + `XlogGapError` (gh-5806). diff --git a/src/box/box.cc b/src/box/box.cc index cc59564e1..e69b7b2ff 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -771,6 +771,19 @@ box_check_wal_queue_max_size(void) return size; } +static double +box_check_wal_cleanup_delay(void) +{ + double value = cfg_getd("wal_cleanup_delay"); + if (value < 0) { + diag_set(ClientError, ER_CFG, "wal_cleanup_delay", + "value must be >= 0"); + return -1; + } + + return value; +} + static void box_check_readahead(int readahead) { @@ -918,6 +931,8 @@ box_check_config(void) box_check_wal_mode(cfg_gets("wal_mode")); if (box_check_wal_queue_max_size() < 0) diag_raise(); + if (box_check_wal_cleanup_delay() < 0) + diag_raise(); if (box_check_memory_quota("memtx_memory") < 0) diag_raise(); box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size")); @@ -1465,6 +1480,23 @@ box_set_wal_queue_max_size(void) return 0; } +int +box_set_wal_cleanup_delay(void) +{ + double delay = box_check_wal_cleanup_delay(); + if (delay < 0) + return -1; + /* + * Anonymous replicas do not require + * delay since they can't be a source + * of replication. + */ + if (replication_anon) + delay = 0; + gc_set_wal_cleanup_delay(delay); + return 0; +} + void box_set_vinyl_memory(void) { @@ -3076,6 +3108,15 @@ box_cfg_xc(void) } fiber_gc(); + /* + * Exclude self from GC delay because we care + * about remote replicas only, still for ref/unref + * balance we do reference self node initially and + * downgrade it to zero when there is no replication + * set at all. + */ + gc_delay_unref(); + bootstrap_journal_guard.is_active = false; assert(current_journal != &bootstrap_journal); diff --git a/src/box/box.h b/src/box/box.h index 65215b087..e2321b9b0 100644 --- a/src/box/box.h +++ b/src/box/box.h @@ -243,6 +243,7 @@ void box_set_checkpoint_count(void); void box_set_checkpoint_interval(void); void box_set_checkpoint_wal_threshold(void); int box_set_wal_queue_max_size(void); +int box_set_wal_cleanup_delay(void); void box_set_memtx_memory(void); void box_set_memtx_max_tuple_size(void); void box_set_vinyl_memory(void); diff --git a/src/box/gc.c b/src/box/gc.c index 9af4ef958..10f899923 100644 --- a/src/box/gc.c +++ b/src/box/gc.c @@ -107,6 +107,11 @@ gc_init(void) /* Don't delete any files until recovery is complete. */ gc.min_checkpoint_count = INT_MAX; + gc.wal_cleanup_delay = TIMEOUT_INFINITY; + gc.delay_ref = 0; + gc.is_paused = true; + say_info("wal/engine cleanup is paused"); + vclock_create(&gc.vclock); rlist_create(&gc.checkpoints); gc_tree_new(&gc.consumers); @@ -238,6 +243,51 @@ static int gc_cleanup_fiber_f(va_list ap) { (void)ap; + + /* + * Stage 1 (optional): in case if we're booting + * up with cleanup disabled lets do wait in a + * separate cycle to minimize branching on stage 2. + */ + if (gc.is_paused) { + double start_time = fiber_clock(); + double timeout = gc.wal_cleanup_delay; + while (!fiber_is_cancelled()) { + if (fiber_yield_timeout(timeout)) { + say_info("wal/engine cleanup is resumed " + "due to timeout expiration"); + gc.is_paused = false; + gc.delay_ref = 0; + break; + } + + /* + * If a last reference is dropped + * we can exit out early. + */ + if (!gc.is_paused) { + say_info("wal/engine cleanup is resumed"); + break; + } + + /* + * Woken up to update the timeout. + */ + double elapsed = fiber_clock() - start_time; + if (elapsed >= gc.wal_cleanup_delay) { + say_info("wal/engine cleanup is resumed " + "due to timeout manual update"); + gc.is_paused = false; + gc.delay_ref = 0; + break; + } + timeout = gc.wal_cleanup_delay - elapsed; + } + } + + /* + * Stage 2: a regular cleanup cycle. + */ while (!fiber_is_cancelled()) { int64_t delta = gc.cleanup_scheduled - gc.cleanup_completed; if (delta == 0) { @@ -253,6 +303,42 @@ gc_cleanup_fiber_f(va_list ap) return 0; } +void +gc_set_wal_cleanup_delay(double wal_cleanup_delay) +{ + gc.wal_cleanup_delay = wal_cleanup_delay; + /* + * This routine may be called at arbitrary + * moment thus we must be sure the cleanup + * fiber is paused to not wake up it when + * it is already in a regular cleanup stage. + */ + if (gc.is_paused) + fiber_wakeup(gc.cleanup_fiber); +} + +void +gc_delay_ref(void) +{ + if (gc.is_paused) { + assert(gc.delay_ref >= 0); + gc.delay_ref++; + } +} + +void +gc_delay_unref(void) +{ + if (gc.is_paused) { + assert(gc.delay_ref > 0); + gc.delay_ref--; + if (gc.delay_ref == 0) { + gc.is_paused = false; + fiber_wakeup(gc.cleanup_fiber); + } + } +} + /** * Trigger asynchronous garbage collection. */ @@ -462,11 +548,12 @@ gc_checkpoint(void) * Wait for background garbage collection that might * have been triggered by this checkpoint to complete. * Strictly speaking, it isn't necessary, but it - * simplifies testing as it guarantees that by the - * time box.snapshot() returns, all outdated checkpoint - * files have been removed. + * simplifies testing. Same time if GC is paused and + * waiting for old XLOGs to be read by replicas the + * cleanup won't happen immediately after the checkpoint. */ - gc_wait_cleanup(); + if (!gc.is_paused) + gc_wait_cleanup(); return 0; } diff --git a/src/box/gc.h b/src/box/gc.h index 2a568c5f9..f5bc26e87 100644 --- a/src/box/gc.h +++ b/src/box/gc.h @@ -147,6 +147,24 @@ struct gc_state { * taken at that moment of time. */ int64_t cleanup_completed, cleanup_scheduled; + /** + * A counter to wait until all replicas are managed to + * subscribe so that we can enable cleanup fiber to + * remove old XLOGs. Otherwise some replicas might be + * far behind the master node and after the master + * node been restarted they will have to reread all + * data back due to XlogGapError, ie too early deleted + * XLOGs. + */ + int64_t delay_ref; + /** + * Delay timeout in seconds. + */ + double wal_cleanup_delay; + /** + * When set the cleanup fiber is paused. + */ + bool is_paused; /** * Set if there's a fiber making a checkpoint right now. */ @@ -206,6 +224,24 @@ gc_init(void); void gc_free(void); +/** + * Set a new delay value. + */ +void +gc_set_wal_cleanup_delay(double wal_cleanup_delay); + +/** + * Increment a reference to delay counter. + */ +void +gc_delay_ref(void); + +/** + * Decrement a reference from the delay counter. + */ +void +gc_delay_unref(void); + /** * Advance the garbage collector vclock to the given position. * Deactivate WAL consumers that need older data. diff --git a/src/box/lua/cfg.cc b/src/box/lua/cfg.cc index b37a93ed8..1142e2726 100644 --- a/src/box/lua/cfg.cc +++ b/src/box/lua/cfg.cc @@ -172,6 +172,14 @@ lbox_cfg_set_wal_queue_max_size(struct lua_State *L) return 0; } +static int +lbox_cfg_set_wal_cleanup_delay(struct lua_State *L) +{ + if (box_set_wal_cleanup_delay() < 0) + luaT_error(L); + return 0; +} + static int lbox_cfg_set_read_only(struct lua_State *L) { @@ -408,6 +416,7 @@ box_lua_cfg_init(struct lua_State *L) {"cfg_set_checkpoint_interval", lbox_cfg_set_checkpoint_interval}, {"cfg_set_checkpoint_wal_threshold", lbox_cfg_set_checkpoint_wal_threshold}, {"cfg_set_wal_queue_max_size", lbox_cfg_set_wal_queue_max_size}, + {"cfg_set_wal_cleanup_delay", lbox_cfg_set_wal_cleanup_delay}, {"cfg_set_read_only", lbox_cfg_set_read_only}, {"cfg_set_memtx_memory", lbox_cfg_set_memtx_memory}, {"cfg_set_memtx_max_tuple_size", lbox_cfg_set_memtx_max_tuple_size}, diff --git a/src/box/lua/info.c b/src/box/lua/info.c index c4c9fa0a0..fb39799a1 100644 --- a/src/box/lua/info.c +++ b/src/box/lua/info.c @@ -445,6 +445,10 @@ lbox_info_gc_call(struct lua_State *L) lua_pushboolean(L, gc.checkpoint_is_in_progress); lua_settable(L, -3); + lua_pushstring(L, "is_paused"); + lua_pushboolean(L, gc.is_paused); + lua_settable(L, -3); + lua_pushstring(L, "checkpoints"); lua_newtable(L); diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua index 9967f992e..44bb95ed1 100644 --- a/src/box/lua/load_cfg.lua +++ b/src/box/lua/load_cfg.lua @@ -73,6 +73,7 @@ local default_cfg = { wal_max_size = 256 * 1024 * 1024, wal_dir_rescan_delay= 2, wal_queue_max_size = 16 * 1024 * 1024, + wal_cleanup_delay = 4 * 3600, force_recovery = false, replication = nil, instance_uuid = nil, @@ -155,6 +156,7 @@ local template_cfg = { wal_mode = 'string', wal_max_size = 'number', wal_dir_rescan_delay= 'number', + wal_cleanup_delay = 'number', force_recovery = 'boolean', replication = 'string, number, table', instance_uuid = 'string', @@ -289,6 +291,7 @@ local dynamic_cfg = { feedback_interval = ifdef_feedback_set_params, -- do nothing, affects new replicas, which query this value on start wal_dir_rescan_delay = function() end, + wal_cleanup_delay = private.cfg_set_wal_cleanup_delay, custom_proc_title = function() require('title').update(box.cfg.custom_proc_title) end, @@ -349,6 +352,8 @@ local dynamic_cfg_order = { -- the new one. This should be fixed when box.cfg is able to -- apply some parameters together and atomically. replication_anon = 250, + -- Cleanup delay should be ignored if replication_anon is set. + wal_cleanup_delay = 260, election_mode = 300, election_timeout = 320, } diff --git a/src/box/relay.cc b/src/box/relay.cc index 41f949e8e..6edee86bf 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -824,6 +824,7 @@ relay_subscribe(struct replica *replica, int fd, uint64_t sync, tt_uuid_str(&replica->uuid)); if (replica->gc == NULL) diag_raise(); + gc_delay_unref(); } relay_start(relay, fd, sync, relay_send_row); diff --git a/src/box/replication.cc b/src/box/replication.cc index 1fa8843e7..aefb812b3 100644 --- a/src/box/replication.cc +++ b/src/box/replication.cc @@ -250,6 +250,7 @@ replica_set_id(struct replica *replica, uint32_t replica_id) tt_uuid_str(&replica->uuid)); } replicaset.replica_by_id[replica_id] = replica; + gc_delay_ref(); ++replicaset.registered_count; say_info("assigned id %d to replica %s", replica->id, tt_uuid_str(&replica->uuid)); @@ -273,6 +274,7 @@ replica_clear_id(struct replica *replica) replicaset.replica_by_id[replica->id] = NULL; assert(replicaset.registered_count > 0); --replicaset.registered_count; + gc_delay_unref(); if (replica->id == instance_id) { /* See replica_check_id(). */ assert(replicaset.is_joining); diff --git a/test/app-tap/init_script.result b/test/app-tap/init_script.result index 74cdb3efb..1fdd9a227 100644 --- a/test/app-tap/init_script.result +++ b/test/app-tap/init_script.result @@ -53,6 +53,7 @@ vinyl_run_count_per_level:2 vinyl_run_size_ratio:3.5 vinyl_timeout:60 vinyl_write_threads:4 +wal_cleanup_delay:14400 wal_dir:. wal_dir_rescan_delay:2 wal_max_size:268435456 diff --git a/test/box/admin.result b/test/box/admin.result index b619ae6cb..f8e8808e3 100644 --- a/test/box/admin.result +++ b/test/box/admin.result @@ -127,6 +127,8 @@ cfg_filter(box.cfg) - 60 - - vinyl_write_threads - 4 + - - wal_cleanup_delay + - 14400 - - wal_dir - <hidden> - - wal_dir_rescan_delay diff --git a/test/box/cfg.result b/test/box/cfg.result index 2c3d5a981..693c1b521 100644 --- a/test/box/cfg.result +++ b/test/box/cfg.result @@ -115,6 +115,8 @@ cfg_filter(box.cfg) | - 60 | - - vinyl_write_threads | - 4 + | - - wal_cleanup_delay + | - 14400 | - - wal_dir | - <hidden> | - - wal_dir_rescan_delay @@ -234,6 +236,8 @@ cfg_filter(box.cfg) | - 60 | - - vinyl_write_threads | - 4 + | - - wal_cleanup_delay + | - 14400 | - - wal_dir | - <hidden> | - - wal_dir_rescan_delay diff --git a/test/replication/replica_rejoin.lua b/test/replication/replica_rejoin.lua new file mode 100644 index 000000000..76f6e5b75 --- /dev/null +++ b/test/replication/replica_rejoin.lua @@ -0,0 +1,22 @@ +#!/usr/bin/env tarantool + +local repl_include_self = arg[1] and arg[1] == 'true' or false +local repl_list + +if repl_include_self then + repl_list = {os.getenv("MASTER"), os.getenv("LISTEN")} +else + repl_list = os.getenv("MASTER") +end + +-- Start the console first to allow test-run to attach even before +-- box.cfg is finished. +require('console').listen(os.getenv('ADMIN')) + +box.cfg({ + listen = os.getenv("LISTEN"), + replication = repl_list, + memtx_memory = 107374182, + replication_timeout = 0.1, + wal_cleanup_delay = 0, +}) diff --git a/test/replication/replica_rejoin.result b/test/replication/replica_rejoin.result index 4bc3df2ca..074cc3e67 100644 --- a/test/replication/replica_rejoin.result +++ b/test/replication/replica_rejoin.result @@ -14,6 +14,13 @@ test_run:cleanup_cluster() --- ... -- +-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber +-- been disabled thus lets turn it off explicitly every time we restart +-- the main node. +box.cfg{wal_cleanup_delay = 0} +--- +... +-- -- gh-461: check that a replica refetches the last checkpoint -- in case it fell behind the master. -- @@ -36,7 +43,7 @@ _ = box.space.test:insert{3} --- ... -- Join a replica, then stop it. -test_run:cmd("create server replica with rpl_master=default, script='replication/replica.lua'") +test_run:cmd("create server replica with rpl_master=default, script='replication/replica_rejoin.lua'") --- - true ... @@ -69,6 +76,9 @@ test_run:cmd("stop server replica") -- Restart the server to purge the replica from -- the garbage collection state. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} +--- +... -- Make some checkpoints to remove old xlogs. checkpoint_count = box.cfg.checkpoint_count --- @@ -190,6 +200,9 @@ test_run:cmd("stop server replica") - true ... test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} +--- +... checkpoint_count = box.cfg.checkpoint_count --- ... @@ -313,6 +326,9 @@ test_run:cmd("switch default") - true ... test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} +--- +... replica_listen = test_run:cmd("eval replica 'return box.cfg.listen'") --- ... diff --git a/test/replication/replica_rejoin.test.lua b/test/replication/replica_rejoin.test.lua index 494cbbcb7..223316d86 100644 --- a/test/replication/replica_rejoin.test.lua +++ b/test/replication/replica_rejoin.test.lua @@ -5,6 +5,12 @@ engine = test_run:get_cfg('engine') test_run:cleanup_cluster() +-- +-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber +-- been disabled thus lets turn it off explicitly every time we restart +-- the main node. +box.cfg{wal_cleanup_delay = 0} + -- -- gh-461: check that a replica refetches the last checkpoint -- in case it fell behind the master. @@ -17,7 +23,7 @@ _ = box.space.test:insert{2} _ = box.space.test:insert{3} -- Join a replica, then stop it. -test_run:cmd("create server replica with rpl_master=default, script='replication/replica.lua'") +test_run:cmd("create server replica with rpl_master=default, script='replication/replica_rejoin.lua'") test_run:cmd("start server replica with args='true'") test_run:cmd("switch replica") box.info.replication[1].upstream.status == 'follow' or log.error(box.info) @@ -28,6 +34,7 @@ test_run:cmd("stop server replica") -- Restart the server to purge the replica from -- the garbage collection state. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} -- Make some checkpoints to remove old xlogs. checkpoint_count = box.cfg.checkpoint_count @@ -72,6 +79,7 @@ box.space.test:replace{1, 2, 3} -- bumps LSN on the replica test_run:cmd("switch default") test_run:cmd("stop server replica") test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} checkpoint_count = box.cfg.checkpoint_count box.cfg{checkpoint_count = 1} for i = 1, 3 do box.space.test:delete{i * 10} end @@ -116,6 +124,7 @@ _ = test_run:wait_vclock('default', vclock) -- Restart the master and force garbage collection. test_run:cmd("switch default") test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} replica_listen = test_run:cmd("eval replica 'return box.cfg.listen'") replica_listen ~= nil box.cfg{replication = replica_listen} diff --git a/test/vinyl/replica_rejoin.lua b/test/vinyl/replica_rejoin.lua index 7cb7e09a4..f97209c44 100644 --- a/test/vinyl/replica_rejoin.lua +++ b/test/vinyl/replica_rejoin.lua @@ -6,8 +6,9 @@ if arg[1] == 'disable_replication' then end box.cfg({ - replication = replication, - vinyl_memory = 1024 * 1024, + replication = replication, + vinyl_memory = 1024 * 1024, + wal_cleanup_delay = 0, }) require('console').listen(os.getenv('ADMIN')) diff --git a/test/vinyl/replica_rejoin.result b/test/vinyl/replica_rejoin.result index 6e8156fee..cb38eaee7 100644 --- a/test/vinyl/replica_rejoin.result +++ b/test/vinyl/replica_rejoin.result @@ -5,6 +5,13 @@ test_run = env.new() --- ... -- +-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber +-- been disabled thus lets turn it off explicitly every time we restart +-- the main node. +box.cfg{wal_cleanup_delay = 0} +--- +... +-- -- gh-461: check that garbage collection works as expected -- after rebootstrap. -- @@ -54,6 +61,9 @@ test_run:cmd("stop server replica") ... -- Invoke garbage collector on the master. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} +--- +... checkpoint_count = box.cfg.checkpoint_count --- ... @@ -123,6 +133,9 @@ test_run:cmd("stop server replica") ... -- Invoke garbage collector on the master. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} +--- +... checkpoint_count = box.cfg.checkpoint_count --- ... diff --git a/test/vinyl/replica_rejoin.test.lua b/test/vinyl/replica_rejoin.test.lua index cbb17ef21..f52859121 100644 --- a/test/vinyl/replica_rejoin.test.lua +++ b/test/vinyl/replica_rejoin.test.lua @@ -1,6 +1,12 @@ env = require('test_run') test_run = env.new() +-- +-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber +-- been disabled thus lets turn it off explicitly every time we restart +-- the main node. +box.cfg{wal_cleanup_delay = 0} + -- -- gh-461: check that garbage collection works as expected -- after rebootstrap. @@ -23,6 +29,7 @@ test_run:cmd("stop server replica") -- Invoke garbage collector on the master. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} checkpoint_count = box.cfg.checkpoint_count box.cfg{checkpoint_count = 1} box.space.test:delete(1) @@ -48,6 +55,7 @@ test_run:cmd("stop server replica") -- Invoke garbage collector on the master. test_run:cmd("restart server default") +box.cfg{wal_cleanup_delay = 0} checkpoint_count = box.cfg.checkpoint_count box.cfg{checkpoint_count = 1} box.space.test:delete(2) -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 13:42 ` Serge Petrenko via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 1 sibling, 0 replies; 12+ messages in thread From: Serge Petrenko via Tarantool-patches @ 2021-03-26 13:42 UTC (permalink / raw) To: Cyrill Gorcunov, tml; +Cc: Vladislav Shpilevoy 26.03.2021 15:06, Cyrill Gorcunov пишет: > In case if replica managed to be far behind the master node > (so there are a number of xlog files present after the last > master's snapshot) then once master node get restarted it > may clean up the xlogs needed by the replica to subscribe > in a fast way and instead the replica will have to rejoin > reading a number of data back. > > Lets try to address this by delaying xlog files cleanup > until replicas are got subscribed and relays are up > and running. For this sake we start with cleanup fiber > spinning in nop cycle ("paused" mode) and use a delay > counter to wait until relays decrement them. > > This implies that if `_cluster` system space is not empty > upon restart and the registered replica somehow vanished > completely and won't ever come back, then the node > administrator has to drop this replica from `_cluster` > manually. > > Note that this delayed cleanup start doesn't prevent > WAL engine from removing old files if there is no > space left on a storage device. The WAL will simply > drop old data without a question. > > We need to take into account that some administrators > might not need this functionality at all, for this > sake we introduce "wal_cleanup_delay" configuration > option which allows to enable or disable the delay. > > Closes #5806 > > Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com> > > @TarantoolBot document > Title: Add wal_cleanup_delay configuration parameter > > The `wal_cleanup_delay` option defines a delay in seconds > before write ahead log files (`*.xlog`) are getting started > to prune upon a node restart. > > This option is ignored in case if a node is running as > an anonymous replica (`replication_anon = true`). Similarly > if replication is unused or there is no plans to use > replication at all then this option should not be considered. > > An initial problem to solve is the case where a node is operating > so fast that its replicas do not manage to reach the node state > and in case if the node is restarted at this moment (for various > reasons, for example due to power outage) then `*.xlog` files might > be pruned during restart. In result replicas will not find these > files on the main node and have to reread all data back which > is a very expensive procedure. > > Since replicas are tracked via `_cluster` system space this we use > its content to count subscribed replicas and when all of them are > up and running the cleanup procedure is automatically enabled even > if `wal_cleanup_delay` is not expired. > > The `wal_cleanup_delay` should be set to: > > - `0` to disable the cleanup delay; > - `>= 0` to wait for specified number of seconds. > > By default it is set to `14400` seconds (ie `4` hours). > > In case if registered replica is lost forever and timeout is set to > infinity then a preferred way to enable cleanup procedure is not setting > up a small timeout value but rather to delete this replica from `_cluster` > space manually. > > Note that the option does *not* prevent WAL engine from removing > old `*.xlog` files if there is no space left on a storage device, > WAL engine can remove them in a force way. > > Current state of `*.xlog` garbage collector can be found in > `box.info.gc()` output. For example > > ``` Lua > tarantool> box.info.gc() > --- > ... > is_paused: false > ``` > > The `is_paused` shows if cleanup fiber is paused or not. > --- > .../unreleased/add-wal_cleanup_delay.md | 5 + > src/box/box.cc | 41 ++++++++ > src/box/box.h | 1 + > src/box/gc.c | 95 ++++++++++++++++++- > src/box/gc.h | 36 +++++++ > src/box/lua/cfg.cc | 9 ++ > src/box/lua/info.c | 4 + > src/box/lua/load_cfg.lua | 5 + > src/box/relay.cc | 1 + > src/box/replication.cc | 2 + > test/app-tap/init_script.result | 1 + > test/box/admin.result | 2 + > test/box/cfg.result | 4 + > test/replication/replica_rejoin.lua | 22 +++++ > test/replication/replica_rejoin.result | 18 +++- > test/replication/replica_rejoin.test.lua | 11 ++- > test/vinyl/replica_rejoin.lua | 5 +- > test/vinyl/replica_rejoin.result | 13 +++ > test/vinyl/replica_rejoin.test.lua | 8 ++ > 19 files changed, 275 insertions(+), 8 deletions(-) > create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md > create mode 100644 test/replication/replica_rejoin.lua Thanks for the patch! LGTM. -- Serge Petrenko ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:42 ` Serge Petrenko via Tarantool-patches @ 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 20:57 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 21:59 ` Cyrill Gorcunov via Tarantool-patches 1 sibling, 2 replies; 12+ messages in thread From: Vladislav Shpilevoy via Tarantool-patches @ 2021-03-26 19:45 UTC (permalink / raw) To: Cyrill Gorcunov, tml Hi! Danke schön für der Patch! See 3 comments below. > diff --git a/test/replication/replica_rejoin.lua b/test/replication/replica_rejoin.lua > new file mode 100644 > index 000000000..76f6e5b75 > --- /dev/null > +++ b/test/replication/replica_rejoin.lua > @@ -0,0 +1,22 @@ > +#!/usr/bin/env tarantool > + > +local repl_include_self = arg[1] and arg[1] == 'true' or false 1. Why do you need that parameter? It is never used in the patch. > +local repl_list > + > +if repl_include_self then > + repl_list = {os.getenv("MASTER"), os.getenv("LISTEN")} > +else > + repl_list = os.getenv("MASTER") > +end > + > +-- Start the console first to allow test-run to attach even before > +-- box.cfg is finished. > +require('console').listen(os.getenv('ADMIN')) > + > +box.cfg({ > + listen = os.getenv("LISTEN"), > + replication = repl_list, > + memtx_memory = 107374182, 2. Why do you need a specific memtx memory setting? > + replication_timeout = 0.1, > + wal_cleanup_delay = 0, 3. I would better propose you to patch replica.lua so as we could pass wal_cleanup_delay parameter to it somehow. But I don't mind if you would keep your way + remove the dead code above. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches @ 2021-03-26 20:57 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 21:59 ` Cyrill Gorcunov via Tarantool-patches 1 sibling, 0 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 20:57 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tml On Fri, Mar 26, 2021 at 08:45:29PM +0100, Vladislav Shpilevoy wrote: > Hi! Danke schön für der Patch! > > See 3 comments below. > > > > + replication_timeout = 0.1, > > + wal_cleanup_delay = 0, > > 3. I would better propose you to patch replica.lua so as we could > pass wal_cleanup_delay parameter to it somehow. But I don't mind > if you would keep your way + remove the dead code above. > ありがとう！ Gimme some time, I'll try to patch replica.lua, if I won't manage I'll drop these redundant lines. Cyrill ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 20:57 ` Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 21:59 ` Cyrill Gorcunov via Tarantool-patches 1 sibling, 0 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 21:59 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tml On Fri, Mar 26, 2021 at 08:45:29PM +0100, Vladislav Shpilevoy wrote: > > +++ b/test/replication/replica_rejoin.lua > > @@ -0,0 +1,22 @@ > > +#!/usr/bin/env tarantool > > + > > +local repl_include_self = arg[1] and arg[1] == 'true' or false > > 1. Why do you need that parameter? It is never used in the patch. It is used on replica > > > +local repl_list > > + > > +if repl_include_self then > > + repl_list = {os.getenv("MASTER"), os.getenv("LISTEN")} > > +else > > + repl_list = os.getenv("MASTER") > > +end > > + > > +-- Start the console first to allow test-run to attach even before > > +-- box.cfg is finished. > > +require('console').listen(os.getenv('ADMIN')) > > + > > +box.cfg({ > > + listen = os.getenv("LISTEN"), > > + replication = repl_list, ^^^ here And we run replica with test_run:cmd("restart server replica with args='true'") ... Since I use a separate replica_join.lua we could drop this `if repl_include_self` sentense since we never call replica with args='false'. I'll re-check. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option 2021-03-26 12:06 [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:37 ` Serge Petrenko via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 3/3] test: box-tap/gc -- add test for is_paused field Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:08 ` [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches 3 siblings, 2 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 UTC (permalink / raw) To: tml; +Cc: Vladislav Shpilevoy Part-of #5806 Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com> --- test/replication/gh-5806-master.lua | 8 + test/replication/gh-5806-slave.lua | 8 + test/replication/gh-5806-xlog-cleanup.result | 435 ++++++++++++++++++ .../replication/gh-5806-xlog-cleanup.test.lua | 188 ++++++++ 4 files changed, 639 insertions(+) create mode 100644 test/replication/gh-5806-master.lua create mode 100644 test/replication/gh-5806-slave.lua create mode 100644 test/replication/gh-5806-xlog-cleanup.result create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua diff --git a/test/replication/gh-5806-master.lua b/test/replication/gh-5806-master.lua new file mode 100644 index 000000000..bc15dab67 --- /dev/null +++ b/test/replication/gh-5806-master.lua @@ -0,0 +1,8 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) + +box.cfg({ + listen = os.getenv("LISTEN"), + wal_cleanup_delay = tonumber(arg[1]) or 0, +}) diff --git a/test/replication/gh-5806-slave.lua b/test/replication/gh-5806-slave.lua new file mode 100644 index 000000000..3abb3e035 --- /dev/null +++ b/test/replication/gh-5806-slave.lua @@ -0,0 +1,8 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) + +box.cfg({ + listen = os.getenv("LISTEN"), + replication = os.getenv("MASTER"), +}) diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result new file mode 100644 index 000000000..e20784bcc --- /dev/null +++ b/test/replication/gh-5806-xlog-cleanup.result @@ -0,0 +1,435 @@ +-- test-run result file version 2 +-- +-- gh-5806: defer xlog cleanup to keep xlogs until +-- replicas present in "_cluster" are connected. +-- Otherwise we are getting XlogGapError since +-- master might go far forward from replica and +-- replica won't be able to connect without full +-- rebootstrap. +-- + +fiber = require('fiber') + | --- + | ... +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +-- +-- Case 1. +-- +-- First lets make sure we're getting XlogGapError in +-- case if wal_cleanup_delay is not used. +-- + +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') + | --- + | - true + | ... +test_run:cmd('start server master with wait=True, wait_load=True') + | --- + | - true + | ... + +test_run:switch('master') + | --- + | - true + | ... +box.schema.user.grant('guest', 'replication') + | --- + | ... + +-- +-- Keep small number of snaps to force cleanup +-- procedure be more intensive. +box.cfg{checkpoint_count = 1} + | --- + | ... + +engine = test_run:get_cfg('engine') + | --- + | ... +s = box.schema.space.create('test', {engine = engine}) + | --- + | ... +_ = s:create_index('pk') + | --- + | ... + +test_run:switch('default') + | --- + | - true + | ... +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +-- +-- On replica we create an own space which allows us to +-- use more complex scenario and disables replica from +-- automatic rejoin (since replica can't do auto-rejoin if +-- there gonna be an own data loss). This allows us to +-- trigger XlogGapError in the log. +test_run:switch('replica') + | --- + | - true + | ... +box.cfg{checkpoint_count = 1} + | --- + | ... +s = box.schema.space.create('testreplica') + | --- + | ... +_ = s:create_index('pk') + | --- + | ... +box.space.testreplica:insert({1}) + | --- + | - [1] + | ... +box.snapshot() + | --- + | - ok + | ... + +-- +-- Stop the replica node and generate +-- xlogs on the master. +test_run:switch('master') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... + +box.space.test:insert({1}) + | --- + | - [1] + | ... +box.snapshot() + | --- + | - ok + | ... + +-- +-- We need to restart the master node since otherwise +-- the replica will be preventing us from removing old +-- xlog because it will be tracked by gc consumer which +-- kept in memory while master node is running. +-- +-- Once restarted we write a new record into master's +-- space and run snapshot which removes old xlog required +-- by replica to subscribe leading to XlogGapError which +-- we need to test. +test_run:cmd('restart server master with wait_load=True') + | +box.space.test:insert({2}) + | --- + | - [2] + | ... +box.snapshot() + | --- + | - ok + | ... +assert(box.info.gc().is_paused == false) + | --- + | - true + | ... + +-- +-- Start replica and wait for error. +test_run:cmd('start server replica with wait=False, wait_load=False') + | --- + | - true + | ... + +-- +-- Wait error to appear, 60 seconds should be more than enough, +-- usually it happens in a couple of seconds. +test_run:switch('default') + | --- + | - true + | ... +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil + | --- + | - true + | ... + +-- +-- Cleanup. +test_run:cmd('stop server master') + | --- + | - true + | ... +test_run:cmd('cleanup server master') + | --- + | - true + | ... +test_run:cmd('delete server master') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('cleanup server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... + +-- +-- Case 2. +-- +-- Lets make sure we're not getting XlogGapError in +-- case if wal_cleanup_delay is used the code is almost +-- the same as for Case 1 except we don't disable cleanup +-- fiber but delay it up to a hour until replica is up +-- and running. +-- + +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') + | --- + | - true + | ... +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') + | --- + | - true + | ... + +test_run:switch('master') + | --- + | - true + | ... +box.schema.user.grant('guest', 'replication') + | --- + | ... + +box.cfg{checkpoint_count = 1} + | --- + | ... + +engine = test_run:get_cfg('engine') + | --- + | ... +s = box.schema.space.create('test', {engine = engine}) + | --- + | ... +_ = s:create_index('pk') + | --- + | ... + +test_run:switch('default') + | --- + | - true + | ... +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +test_run:switch('replica') + | --- + | - true + | ... +box.cfg{checkpoint_count = 1} + | --- + | ... +s = box.schema.space.create('testreplica') + | --- + | ... +_ = s:create_index('pk') + | --- + | ... +box.space.testreplica:insert({1}) + | --- + | - [1] + | ... +box.snapshot() + | --- + | - ok + | ... + +test_run:switch('master') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... + +box.space.test:insert({1}) + | --- + | - [1] + | ... +box.snapshot() + | --- + | - ok + | ... + +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') + | +box.space.test:insert({2}) + | --- + | - [2] + | ... +box.snapshot() + | --- + | - ok + | ... +assert(box.info.gc().is_paused == true) + | --- + | - true + | ... + +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +-- +-- Make sure no error happened. +test_run:switch('default') + | --- + | - true + | ... +assert(test_run:grep_log("master", "XlogGapError") == nil) + | --- + | - true + | ... + +test_run:cmd('stop server master') + | --- + | - true + | ... +test_run:cmd('cleanup server master') + | --- + | - true + | ... +test_run:cmd('delete server master') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('cleanup server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +-- +-- +-- Case 3: Fill _cluster with replica but then delete +-- the replica so that master's cleanup leave in "paused" +-- state, and then simply decrease the timeout to make +-- cleanup fiber work again. +-- +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') + | --- + | - true + | ... +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') + | --- + | - true + | ... + +test_run:switch('master') + | --- + | - true + | ... +box.schema.user.grant('guest', 'replication') + | --- + | ... + +test_run:switch('default') + | --- + | - true + | ... +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +test_run:switch('master') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('cleanup server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... + +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') + | +assert(box.info.gc().is_paused == true) + | --- + | - true + | ... + +test_run:switch('master') + | --- + | - true + | ... +box.cfg{wal_cleanup_delay = 0.01} + | --- + | ... +test_run:wait_cond(function() return box.info.gc().is_paused == false end) + | --- + | - true + | ... + +test_run:switch('default') + | --- + | - true + | ... +test_run:cmd('stop server master') + | --- + | - true + | ... +test_run:cmd('cleanup server master') + | --- + | - true + | ... +test_run:cmd('delete server master') + | --- + | - true + | ... diff --git a/test/replication/gh-5806-xlog-cleanup.test.lua b/test/replication/gh-5806-xlog-cleanup.test.lua new file mode 100644 index 000000000..ea3a35294 --- /dev/null +++ b/test/replication/gh-5806-xlog-cleanup.test.lua @@ -0,0 +1,188 @@ +-- +-- gh-5806: defer xlog cleanup to keep xlogs until +-- replicas present in "_cluster" are connected. +-- Otherwise we are getting XlogGapError since +-- master might go far forward from replica and +-- replica won't be able to connect without full +-- rebootstrap. +-- + +fiber = require('fiber') +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +-- +-- Case 1. +-- +-- First lets make sure we're getting XlogGapError in +-- case if wal_cleanup_delay is not used. +-- + +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') +test_run:cmd('start server master with wait=True, wait_load=True') + +test_run:switch('master') +box.schema.user.grant('guest', 'replication') + +-- +-- Keep small number of snaps to force cleanup +-- procedure be more intensive. +box.cfg{checkpoint_count = 1} + +engine = test_run:get_cfg('engine') +s = box.schema.space.create('test', {engine = engine}) +_ = s:create_index('pk') + +test_run:switch('default') +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') + +-- +-- On replica we create an own space which allows us to +-- use more complex scenario and disables replica from +-- automatic rejoin (since replica can't do auto-rejoin if +-- there gonna be an own data loss). This allows us to +-- trigger XlogGapError in the log. +test_run:switch('replica') +box.cfg{checkpoint_count = 1} +s = box.schema.space.create('testreplica') +_ = s:create_index('pk') +box.space.testreplica:insert({1}) +box.snapshot() + +-- +-- Stop the replica node and generate +-- xlogs on the master. +test_run:switch('master') +test_run:cmd('stop server replica') + +box.space.test:insert({1}) +box.snapshot() + +-- +-- We need to restart the master node since otherwise +-- the replica will be preventing us from removing old +-- xlog because it will be tracked by gc consumer which +-- kept in memory while master node is running. +-- +-- Once restarted we write a new record into master's +-- space and run snapshot which removes old xlog required +-- by replica to subscribe leading to XlogGapError which +-- we need to test. +test_run:cmd('restart server master with wait_load=True') +box.space.test:insert({2}) +box.snapshot() +assert(box.info.gc().is_paused == false) + +-- +-- Start replica and wait for error. +test_run:cmd('start server replica with wait=False, wait_load=False') + +-- +-- Wait error to appear, 60 seconds should be more than enough, +-- usually it happens in a couple of seconds. +test_run:switch('default') +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil + +-- +-- Cleanup. +test_run:cmd('stop server master') +test_run:cmd('cleanup server master') +test_run:cmd('delete server master') +test_run:cmd('stop server replica') +test_run:cmd('cleanup server replica') +test_run:cmd('delete server replica') + +-- +-- Case 2. +-- +-- Lets make sure we're not getting XlogGapError in +-- case if wal_cleanup_delay is used the code is almost +-- the same as for Case 1 except we don't disable cleanup +-- fiber but delay it up to a hour until replica is up +-- and running. +-- + +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') + +test_run:switch('master') +box.schema.user.grant('guest', 'replication') + +box.cfg{checkpoint_count = 1} + +engine = test_run:get_cfg('engine') +s = box.schema.space.create('test', {engine = engine}) +_ = s:create_index('pk') + +test_run:switch('default') +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') + +test_run:switch('replica') +box.cfg{checkpoint_count = 1} +s = box.schema.space.create('testreplica') +_ = s:create_index('pk') +box.space.testreplica:insert({1}) +box.snapshot() + +test_run:switch('master') +test_run:cmd('stop server replica') + +box.space.test:insert({1}) +box.snapshot() + +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') +box.space.test:insert({2}) +box.snapshot() +assert(box.info.gc().is_paused == true) + +test_run:cmd('start server replica with wait=True, wait_load=True') + +-- +-- Make sure no error happened. +test_run:switch('default') +assert(test_run:grep_log("master", "XlogGapError") == nil) + +test_run:cmd('stop server master') +test_run:cmd('cleanup server master') +test_run:cmd('delete server master') +test_run:cmd('stop server replica') +test_run:cmd('cleanup server replica') +test_run:cmd('delete server replica') +-- +-- +-- Case 3: Fill _cluster with replica but then delete +-- the replica so that master's cleanup leave in "paused" +-- state, and then simply decrease the timeout to make +-- cleanup fiber work again. +-- +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') + +test_run:switch('master') +box.schema.user.grant('guest', 'replication') + +test_run:switch('default') +test_run:cmd('create server replica with rpl_master=master,\ + script="replication/gh-5806-slave.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') + +test_run:switch('master') +test_run:cmd('stop server replica') +test_run:cmd('cleanup server replica') +test_run:cmd('delete server replica') + +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') +assert(box.info.gc().is_paused == true) + +test_run:switch('master') +box.cfg{wal_cleanup_delay = 0.01} +test_run:wait_cond(function() return box.info.gc().is_paused == false end) + +test_run:switch('default') +test_run:cmd('stop server master') +test_run:cmd('cleanup server master') +test_run:cmd('delete server master') -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 13:37 ` Serge Petrenko via Tarantool-patches 2021-03-26 13:57 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 1 sibling, 1 reply; 12+ messages in thread From: Serge Petrenko via Tarantool-patches @ 2021-03-26 13:37 UTC (permalink / raw) To: Cyrill Gorcunov, tml; +Cc: Vladislav Shpilevoy 26.03.2021 15:06, Cyrill Gorcunov пишет: > Part-of #5806 > > Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com> > --- > test/replication/gh-5806-master.lua | 8 + > test/replication/gh-5806-slave.lua | 8 + > test/replication/gh-5806-xlog-cleanup.result | 435 ++++++++++++++++++ > .../replication/gh-5806-xlog-cleanup.test.lua | 188 ++++++++ > 4 files changed, 639 insertions(+) > create mode 100644 test/replication/gh-5806-master.lua > create mode 100644 test/replication/gh-5806-slave.lua > create mode 100644 test/replication/gh-5806-xlog-cleanup.result > create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua > > diff --git a/test/replication/gh-5806-master.lua b/test/replication/gh-5806-master.lua > new file mode 100644 > index 000000000..bc15dab67 > --- /dev/null > +++ b/test/replication/gh-5806-master.lua > @@ -0,0 +1,8 @@ > +#!/usr/bin/env tarantool > + > +require('console').listen(os.getenv('ADMIN')) > + > +box.cfg({ > + listen = os.getenv("LISTEN"), > + wal_cleanup_delay = tonumber(arg[1]) or 0, > +}) > diff --git a/test/replication/gh-5806-slave.lua b/test/replication/gh-5806-slave.lua > new file mode 100644 > index 000000000..3abb3e035 > --- /dev/null > +++ b/test/replication/gh-5806-slave.lua > @@ -0,0 +1,8 @@ > +#!/usr/bin/env tarantool > + > +require('console').listen(os.getenv('ADMIN')) > + > +box.cfg({ > + listen = os.getenv("LISTEN"), > + replication = os.getenv("MASTER"), > +}) Hi! Thanks for the fixes! You may use `replica.lua` here freely. It takes the same parameters, and has the same box.cfg() call, with exception that it has set `replication_timeout` and `memtx_memory`. These two options are not a problem. In fact, `replica.lua` is used in almost all replication tests. So, there's no reason to spawn a new file for your needs. > diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result > new file mode 100644 > index 000000000..e20784bcc > --- /dev/null > +++ b/test/replication/gh-5806-xlog-cleanup.result > @@ -0,0 +1,435 @@ > +-- test-run result file version 2 > +-- > +-- gh-5806: defer xlog cleanup to keep xlogs until > +-- replicas present in "_cluster" are connected. > +-- Otherwise we are getting XlogGapError since > +-- master might go far forward from replica and > +-- replica won't be able to connect without full > +-- rebootstrap. > +-- > + > +fiber = require('fiber') > + | --- > + | ... > +test_run = require('test_run').new() > + | --- > + | ... > +engine = test_run:get_cfg('engine') > + | --- > + | ... > + > +-- > +-- Case 1. > +-- > +-- First lets make sure we're getting XlogGapError in > +-- case if wal_cleanup_delay is not used. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +-- > +-- Keep small number of snaps to force cleanup > +-- procedure be more intensive. > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > + > +engine = test_run:get_cfg('engine') > + | --- > + | ... > +s = box.schema.space.create('test', {engine = engine}) > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +-- > +-- On replica we create an own space which allows us to > +-- use more complex scenario and disables replica from > +-- automatic rejoin (since replica can't do auto-rejoin if > +-- there gonna be an own data loss). This allows us to > +-- trigger XlogGapError in the log. > +test_run:switch('replica') > + | --- > + | - true > + | ... > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > +s = box.schema.space.create('testreplica') > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > +box.space.testreplica:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +-- > +-- Stop the replica node and generate > +-- xlogs on the master. > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > + > +box.space.test:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +-- > +-- We need to restart the master node since otherwise > +-- the replica will be preventing us from removing old > +-- xlog because it will be tracked by gc consumer which > +-- kept in memory while master node is running. > +-- > +-- Once restarted we write a new record into master's > +-- space and run snapshot which removes old xlog required > +-- by replica to subscribe leading to XlogGapError which > +-- we need to test. > +test_run:cmd('restart server master with wait_load=True') > + | > +box.space.test:insert({2}) > + | --- > + | - [2] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > +assert(box.info.gc().is_paused == false) > + | --- > + | - true > + | ... > + > +-- > +-- Start replica and wait for error. > +test_run:cmd('start server replica with wait=False, wait_load=False') > + | --- > + | - true > + | ... > + > +-- > +-- Wait error to appear, 60 seconds should be more than enough, > +-- usually it happens in a couple of seconds. > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil > + | --- > + | - true > + | ... > + > +-- > +-- Cleanup. > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > + > +-- > +-- Case 2. > +-- > +-- Lets make sure we're not getting XlogGapError in > +-- case if wal_cleanup_delay is used the code is almost > +-- the same as for Case 1 except we don't disable cleanup > +-- fiber but delay it up to a hour until replica is up > +-- and running. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > + > +engine = test_run:get_cfg('engine') > + | --- > + | ... > +s = box.schema.space.create('test', {engine = engine}) > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('replica') > + | --- > + | - true > + | ... > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > +s = box.schema.space.create('testreplica') > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > +box.space.testreplica:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > + > +box.space.test:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > + | > +box.space.test:insert({2}) > + | --- > + | - [2] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > +assert(box.info.gc().is_paused == true) > + | --- > + | - true > + | ... > + > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +-- > +-- Make sure no error happened. > +test_run:switch('default') > + | --- > + | - true > + | ... > +assert(test_run:grep_log("master", "XlogGapError") == nil) > + | --- > + | - true > + | ... > + > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > +-- > +-- > +-- Case 3: Fill _cluster with replica but then delete > +-- the replica so that master's cleanup leave in "paused" > +-- state, and then simply decrease the timeout to make > +-- cleanup fiber work again. > +-- > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > + | > +assert(box.info.gc().is_paused == true) > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.cfg{wal_cleanup_delay = 0.01} > + | --- > + | ... > +test_run:wait_cond(function() return box.info.gc().is_paused == false end) > + | --- > + | - true > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > diff --git a/test/replication/gh-5806-xlog-cleanup.test.lua b/test/replication/gh-5806-xlog-cleanup.test.lua > new file mode 100644 > index 000000000..ea3a35294 > --- /dev/null > +++ b/test/replication/gh-5806-xlog-cleanup.test.lua > @@ -0,0 +1,188 @@ > +-- > +-- gh-5806: defer xlog cleanup to keep xlogs until > +-- replicas present in "_cluster" are connected. > +-- Otherwise we are getting XlogGapError since > +-- master might go far forward from replica and > +-- replica won't be able to connect without full > +-- rebootstrap. > +-- > + > +fiber = require('fiber') > +test_run = require('test_run').new() > +engine = test_run:get_cfg('engine') > + > +-- > +-- Case 1. > +-- > +-- First lets make sure we're getting XlogGapError in > +-- case if wal_cleanup_delay is not used. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +-- > +-- Keep small number of snaps to force cleanup > +-- procedure be more intensive. > +box.cfg{checkpoint_count = 1} > + > +engine = test_run:get_cfg('engine') > +s = box.schema.space.create('test', {engine = engine}) > +_ = s:create_index('pk') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +-- > +-- On replica we create an own space which allows us to > +-- use more complex scenario and disables replica from > +-- automatic rejoin (since replica can't do auto-rejoin if > +-- there gonna be an own data loss). This allows us to > +-- trigger XlogGapError in the log. > +test_run:switch('replica') > +box.cfg{checkpoint_count = 1} > +s = box.schema.space.create('testreplica') > +_ = s:create_index('pk') > +box.space.testreplica:insert({1}) > +box.snapshot() > + > +-- > +-- Stop the replica node and generate > +-- xlogs on the master. > +test_run:switch('master') > +test_run:cmd('stop server replica') > + > +box.space.test:insert({1}) > +box.snapshot() > + > +-- > +-- We need to restart the master node since otherwise > +-- the replica will be preventing us from removing old > +-- xlog because it will be tracked by gc consumer which > +-- kept in memory while master node is running. > +-- > +-- Once restarted we write a new record into master's > +-- space and run snapshot which removes old xlog required > +-- by replica to subscribe leading to XlogGapError which > +-- we need to test. > +test_run:cmd('restart server master with wait_load=True') > +box.space.test:insert({2}) > +box.snapshot() > +assert(box.info.gc().is_paused == false) > + > +-- > +-- Start replica and wait for error. > +test_run:cmd('start server replica with wait=False, wait_load=False') > + > +-- > +-- Wait error to appear, 60 seconds should be more than enough, > +-- usually it happens in a couple of seconds. > +test_run:switch('default') > +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil > + > +-- > +-- Cleanup. > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > + > +-- > +-- Case 2. > +-- > +-- Lets make sure we're not getting XlogGapError in > +-- case if wal_cleanup_delay is used the code is almost > +-- the same as for Case 1 except we don't disable cleanup > +-- fiber but delay it up to a hour until replica is up > +-- and running. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +box.cfg{checkpoint_count = 1} > + > +engine = test_run:get_cfg('engine') > +s = box.schema.space.create('test', {engine = engine}) > +_ = s:create_index('pk') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +test_run:switch('replica') > +box.cfg{checkpoint_count = 1} > +s = box.schema.space.create('testreplica') > +_ = s:create_index('pk') > +box.space.testreplica:insert({1}) > +box.snapshot() > + > +test_run:switch('master') > +test_run:cmd('stop server replica') > + > +box.space.test:insert({1}) > +box.snapshot() > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > +box.space.test:insert({2}) > +box.snapshot() > +assert(box.info.gc().is_paused == true) > + > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +-- > +-- Make sure no error happened. > +test_run:switch('default') > +assert(test_run:grep_log("master", "XlogGapError") == nil) > + > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > +-- > +-- > +-- Case 3: Fill _cluster with replica but then delete > +-- the replica so that master's cleanup leave in "paused" > +-- state, and then simply decrease the timeout to make > +-- cleanup fiber work again. > +-- > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +test_run:switch('master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > +assert(box.info.gc().is_paused == true) > + > +test_run:switch('master') > +box.cfg{wal_cleanup_delay = 0.01} > +test_run:wait_cond(function() return box.info.gc().is_paused == false end) > + > +test_run:switch('default') > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') -- Serge Petrenko ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option 2021-03-26 13:37 ` Serge Petrenko via Tarantool-patches @ 2021-03-26 13:57 ` Cyrill Gorcunov via Tarantool-patches 0 siblings, 0 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 13:57 UTC (permalink / raw) To: Serge Petrenko; +Cc: tml, Vladislav Shpilevoy On Fri, Mar 26, 2021 at 04:37:26PM +0300, Serge Petrenko wrote: > > +box.cfg({ > > + listen = os.getenv("LISTEN"), > > + replication = os.getenv("MASTER"), > > +}) > > Hi! Thanks for the fixes! > > You may use `replica.lua` here freely. It takes the same parameters, and > has the same > box.cfg() call, with exception that it has set `replication_timeout` and > `memtx_memory`. > These two options are not a problem. In fact, `replica.lua` is used in > almost all > replication tests. > > So, there's no reason to spawn a new file for your needs. Indeed, pushed an update. --- diff --git a/test/replication/gh-5806-slave.lua b/test/replication/gh-5806-slave.lua deleted file mode 100644 index 3abb3e035..000000000 --- a/test/replication/gh-5806-slave.lua +++ /dev/null @@ -1,8 +0,0 @@ -#!/usr/bin/env tarantool - -require('console').listen(os.getenv('ADMIN')) - -box.cfg({ - listen = os.getenv("LISTEN"), - replication = os.getenv("MASTER"), -}) diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result index e20784bcc..523d400a7 100644 --- a/test/replication/gh-5806-xlog-cleanup.result +++ b/test/replication/gh-5806-xlog-cleanup.result @@ -64,7 +64,7 @@ test_run:switch('default') | - true | ... test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') | --- | - true | ... @@ -239,7 +239,7 @@ test_run:switch('default') | - true | ... test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') | --- | - true | ... @@ -372,7 +372,7 @@ test_run:switch('default') | - true | ... test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') | --- | - true | ... diff --git a/test/replication/gh-5806-xlog-cleanup.test.lua b/test/replication/gh-5806-xlog-cleanup.test.lua index ea3a35294..f16be758a 100644 --- a/test/replication/gh-5806-xlog-cleanup.test.lua +++ b/test/replication/gh-5806-xlog-cleanup.test.lua @@ -35,7 +35,7 @@ _ = s:create_index('pk') test_run:switch('default') test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') test_run:cmd('start server replica with wait=True, wait_load=True') -- @@ -118,7 +118,7 @@ _ = s:create_index('pk') test_run:switch('default') test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') test_run:cmd('start server replica with wait=True, wait_load=True') test_run:switch('replica') @@ -167,7 +167,7 @@ box.schema.user.grant('guest', 'replication') test_run:switch('default') test_run:cmd('create server replica with rpl_master=master,\ - script="replication/gh-5806-slave.lua"') + script="replication/replica.lua"') test_run:cmd('start server replica with wait=True, wait_load=True') test_run:switch('master') ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:37 ` Serge Petrenko via Tarantool-patches @ 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 1 sibling, 0 replies; 12+ messages in thread From: Vladislav Shpilevoy via Tarantool-patches @ 2021-03-26 19:45 UTC (permalink / raw) To: Cyrill Gorcunov, tml Thanks for the patch! See 2 comments below. > --- /dev/null > +++ b/test/replication/gh-5806-xlog-cleanup.result > @@ -0,0 +1,435 @@ > +-- test-run result file version 2 > +-- > +-- gh-5806: defer xlog cleanup to keep xlogs until > +-- replicas present in "_cluster" are connected. > +-- Otherwise we are getting XlogGapError since > +-- master might go far forward from replica and > +-- replica won't be able to connect without full > +-- rebootstrap. > +-- > + > +fiber = require('fiber') > + | --- > + | ... > +test_run = require('test_run').new() > + | --- > + | ... > +engine = test_run:get_cfg('engine') > + | --- > + | ... > + > +-- > +-- Case 1. > +-- > +-- First lets make sure we're getting XlogGapError in > +-- case if wal_cleanup_delay is not used. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with wait=True, wait_load=True') 1. Both wait and wait_load are True by default. You don't need to specify them. Here and in other places where you made them True. > + | --- > + | - true > + | ... > +-- > +-- > +-- Case 3: Fill _cluster with replica but then delete > +-- the replica so that master's cleanup leave in "paused" > +-- state, and then simply decrease the timeout to make > +-- cleanup fiber work again. 2. Would also be good to have a test where GC is paused, but then the replica is deleted from _cluster (so gc unref is called) and gc continues. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Tarantool-patches] [PATCH v5 3/3] test: box-tap/gc -- add test for is_paused field 2021-03-26 12:06 [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:08 ` [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches 3 siblings, 0 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:06 UTC (permalink / raw) To: tml; +Cc: Vladislav Shpilevoy Once simple bootstrap is complete and there is no replicas used we should run with gc unpaused. Part-of #5806 Acked-by: Serge Petrenko <sergepetrenko@tarantool.org> Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com> --- test/box-tap/gc.test.lua | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/test/box-tap/gc.test.lua b/test/box-tap/gc.test.lua index f0155779c..ced87547e 100755 --- a/test/box-tap/gc.test.lua +++ b/test/box-tap/gc.test.lua @@ -8,11 +8,12 @@ local debug = type(box.error.injection) == "table" -- check box.info.gc() is false if snapshot is not in progress local test = tap.test('box.info.gc') -test:plan(1 + (debug and 1 or 0)) +test:plan(2 + (debug and 1 or 0)) local gc = box.info.gc() test:is(gc.checkpoint_is_in_progress, false, "checkpoint is not in progress") +test:is(gc.is_paused, false, "GC is not paused") -- check box.info.gc() is true if snapshot is in progress -- -- 2.30.2 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed 2021-03-26 12:06 [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches ` (2 preceding siblings ...) 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 3/3] test: box-tap/gc -- add test for is_paused field Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:08 ` Cyrill Gorcunov via Tarantool-patches 3 siblings, 0 replies; 12+ messages in thread From: Cyrill Gorcunov via Tarantool-patches @ 2021-03-26 12:08 UTC (permalink / raw) To: tml; +Cc: Vladislav Shpilevoy On Fri, Mar 26, 2021 at 03:06:02PM +0300, Cyrill Gorcunov wrote: > v5: > - define limits for `wal_cleanup_delay`: it should be either 0, > or in range [0.001; TIMEOUT_INFINITY]. This is done to not consider > fp epsilon as a meaningul value Limits were removed, only >= 0 is used. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-03-26 21:59 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-03-26 12:06 [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 1/3] " Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:42 ` Serge Petrenko via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 20:57 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 21:59 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option Cyrill Gorcunov via Tarantool-patches 2021-03-26 13:37 ` Serge Petrenko via Tarantool-patches 2021-03-26 13:57 ` Cyrill Gorcunov via Tarantool-patches 2021-03-26 19:45 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-26 12:06 ` [Tarantool-patches] [PATCH v5 3/3] test: box-tap/gc -- add test for is_paused field Cyrill Gorcunov via Tarantool-patches 2021-03-26 12:08 ` [Tarantool-patches] [PATCH v5 0/3] gc/xlog: delay xlog cleanup until relays are subscribed Cyrill Gorcunov via Tarantool-patches
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox