[Tarantool-patches] [PATCH v4 0/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Tarantool-patches] [PATCH v4 0/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

Take a look please.

v2:
 - rebase code to the fresh master branch
 - keep wal_cleanup_delay option name
 - pass wal_cleanup_delay as an option to gc_init, so it
   won't be dependent on cfg engine
 - add comment about gc_delay_unref in plain bootstrap mode
 - allow to setup wal_cleanup_delay dynamically
 - update comment in gc_wait_cleanup and call it conditionally
 - declare wal_cleanup_delay as a double
 - rename gc.cleanup_is_paused to gc.is_paused and update output
 - do not show ref counter in box.info.gc() output
 - update documentation
 - move gc_delay_unref inside relay_subscribe call which runs
   in tx context (instead of relay's context)
 - update tests:
   - add a comment why we need a temp space on replica node
   - use explicit insert/snapshot operations
   - shrkink the number of insert/snapshot to speedup testing
   - use "restart" instead of stop/start pair
   - use wait_log helper instead of own function
   - add is_paused test
v3:
 - fix changelog
 - rework box_check_wal_cleanup_delay, the replication_anon
   setting is considered only in box_set_wal_cleanup_delay,
   ie when config is checked and parsed, moreover the order
   of setup is set to be behind "replication_anon" option
   processing
 - delay cycle now considers deadline instead of per cycle
   calculation
 - use `double` type for timestamp
 - test update
   - verify `.is_paused` value
   - minimize number of inserts
   - no need to use temporary space, regular space works as well
   - add comments on why we should restart the master node
v4:
 - drop argument from gc_init(), since we're configuring delay
   value from load_cfg.lua script there is no need to read the
   delay early, simply start gc paused and unpause it on demand
 - move unpause message to main wait cycle
 - test update:
   - verify tests and fix replication/replica_rejoin since it waits
     for xlogs to be cleaned up too early
   - use 10 seconds for XlogGapError instead of 0.1 second, this is
     a common deadline value

issue https://github.com/tarantool/tarantool/issues/5806
branch gorcunov/gh-5806-xlog-gc-4

Cyrill Gorcunov (3):
  gc/xlog: delay xlog cleanup until relays are subscribed
  test: add a test for wal_cleanup_delay option
  test: box-tap/gc -- add test for is_paused field

 .../unreleased/add-wal_cleanup_delay.md       |   5 +
 src/box/box.cc                                |  41 ++
 src/box/box.h                                 |   1 +
 src/box/gc.c                                  |  85 ++++-
 src/box/gc.h                                  |  36 ++
 src/box/lua/cfg.cc                            |   9 +
 src/box/lua/info.c                            |   4 +
 src/box/lua/load_cfg.lua                      |   5 +
 src/box/relay.cc                              |   1 +
 src/box/replication.cc                        |   2 +
 test/app-tap/init_script.result               |   1 +
 test/box-tap/gc.test.lua                      |   3 +-
 test/box/admin.result                         |   2 +
 test/box/cfg.result                           |   4 +
 test/replication/gh-5806-master.lua           |   8 +
 test/replication/gh-5806-slave.lua            |   8 +
 test/replication/gh-5806-xlog-cleanup.result  | 356 ++++++++++++++++++
 .../replication/gh-5806-xlog-cleanup.test.lua | 155 ++++++++
 test/replication/replica_rejoin.result        |  16 +
 test/replication/replica_rejoin.test.lua      |   9 +
 20 files changed, 746 insertions(+), 5 deletions(-)
 create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md
 create mode 100644 test/replication/gh-5806-master.lua
 create mode 100644 test/replication/gh-5806-slave.lua
 create mode 100644 test/replication/gh-5806-xlog-cleanup.result
 create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua


base-commit: f4e248c0c13a46beee238fbebc38ef687ef09d02
-- 
2.30.2

[Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

In case if replica managed to be far behind the master node
(so there are a number of xlog files present after the last
master's snapshot) then once master node get restarted it
may clean up the xlogs needed by the replica to subscribe
in a fast way and instead the replica will have to rejoin
reading a number of data back.

Lets try to address this by delaying xlog files cleanup
until replicas are got subscribed and relays are up
and running. For this sake we start with cleanup fiber
spinning in nop cycle ("paused" mode) and use a delay
counter to wait until relays decrement them.

This implies that if `_cluster` system space is not empty
upon restart and the registered replica somehow vanished
completely and won't ever come back, then the node
administrator has to drop this replica from `_cluster`
manually.

Note that this delayed cleanup start doesn't prevent
WAL engine from removing old files if there is no
space left on a storage device. The WAL will simply
drop old data without a question.

We need to take into account that some administrators
might not need this functionality at all, for this
sake we introduce "wal_cleanup_delay" configuration
option which allows to enable or disable the delay.

Closes #5806

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

@TarantoolBot document
Title: Add wal_cleanup_delay configuration parameter

The `wal_cleanup_delay` option defines a delay in seconds
before write ahead log files (`*.xlog`) are getting started
to prune upon a node restart.

This option is ignored in case if a node is running as
an anonymous replica (`replication_anon = true`). Similarly
if replication is unused or there is no plans to use
replication at all then this option should not be considered.

An initial problem to solve is the case where a node is operating
so fast that its replicas do not manage to reach the node state
and in case if the node is restarted at this moment (for various
reasons, for example due to power outage) then `*.xlog` files might
be pruned during restart. In result replicas will not find these
files on the main node and have to reread all data back which
is a very expensive procedure.

Since replicas are tracked via `_cluster` system space this we use
its content to count subscribed replicas and when all of them are
up and running the cleanup procedure is automatically enabled even
if `wal_cleanup_delay` is not expired.

The `wal_cleanup_delay` should be set to:

 - `0` to disable the cleanup delay;
 - `>0` to wait for specified number of seconds.

By default it is set to `14400` seconds (ie `4` hours).

In case if registered replica is lost forever and timeout is set to
infinity then a preferred way to enable cleanup procedure is not setting
up a small timeout value but rather to delete this replica from `_cluster`
space manually.

Note that the option does *not* prevent WAL engine from removing
old `*.xlog` files if there is no space left on a storage device,
WAL engine can remove them in a force way.

Current state of `*.xlog` garbage collector can be found in
`box.info.gc()` output. For example

``` Lua
 tarantool> box.info.gc()
 ---
   ...
   is_paused: false
```

The `is_paused` shows if cleanup fiber is paused or not.
---
 .../unreleased/add-wal_cleanup_delay.md       |  5 ++
 src/box/box.cc                                | 41 +++++++++
 src/box/box.h                                 |  1 +
 src/box/gc.c                                  | 85 ++++++++++++++++++-
 src/box/gc.h                                  | 36 ++++++++
 src/box/lua/cfg.cc                            |  9 ++
 src/box/lua/info.c                            |  4 +
 src/box/lua/load_cfg.lua                      |  5 ++
 src/box/relay.cc                              |  1 +
 src/box/replication.cc                        |  2 +
 test/app-tap/init_script.result               |  1 +
 test/box/admin.result                         |  2 +
 test/box/cfg.result                           |  4 +
 test/replication/replica_rejoin.result        | 16 ++++
 test/replication/replica_rejoin.test.lua      |  9 ++
 15 files changed, 217 insertions(+), 4 deletions(-)
 create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md

diff --git a/changelogs/unreleased/add-wal_cleanup_delay.md b/changelogs/unreleased/add-wal_cleanup_delay.md
new file mode 100644
index 000000000..2e67f0a4e
--- /dev/null
+++ b/changelogs/unreleased/add-wal_cleanup_delay.md
@@ -0,0 +1,5 @@
+## bugfix/core
+
+* Introduce `wal_cleanup_delay` option to prevent early cleanup
+  of `*.xlog` files which are needed by replicas and lead to
+  `XlogGapError` (gh-5806).
diff --git a/src/box/box.cc b/src/box/box.cc
index cc59564e1..a269f7357 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -771,6 +771,19 @@ box_check_wal_queue_max_size(void)
 	return size;
 }
 
+static double
+box_check_wal_cleanup_delay(void)
+{
+	double value = cfg_getd("wal_cleanup_delay");
+	if (value < 0) {
+		diag_set(ClientError, ER_CFG, "wal_cleanup_delay",
+			 "the value must be >= 0");
+		return -1;
+	}
+
+	return value;
+}
+
 static void
 box_check_readahead(int readahead)
 {
@@ -918,6 +931,8 @@ box_check_config(void)
 	box_check_wal_mode(cfg_gets("wal_mode"));
 	if (box_check_wal_queue_max_size() < 0)
 		diag_raise();
+	if (box_check_wal_cleanup_delay() < 0)
+		diag_raise();
 	if (box_check_memory_quota("memtx_memory") < 0)
 		diag_raise();
 	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
@@ -1465,6 +1480,23 @@ box_set_wal_queue_max_size(void)
 	return 0;
 }
 
+int
+box_set_wal_cleanup_delay(void)
+{
+	double delay = box_check_wal_cleanup_delay();
+	if (delay < 0)
+		return -1;
+	/*
+	 * Anonymous replicas do not require
+	 * delay the cleanup procedure since they
+	 * are read only.
+	 */
+	if (replication_anon)
+		delay = 0;
+	gc_set_wal_cleanup_delay(delay);
+	return 0;
+}
+
 void
 box_set_vinyl_memory(void)
 {
@@ -3076,6 +3108,15 @@ box_cfg_xc(void)
 	}
 	fiber_gc();
 
+	/*
+	 * Exclude self from GC delay because we care
+	 * about remote replicas only, still for ref/unref
+	 * balance we do reference self node initially and
+	 * downgrade it to zero when there is no replication
+	 * set at all.
+	 */
+	gc_delay_unref();
+
 	bootstrap_journal_guard.is_active = false;
 	assert(current_journal != &bootstrap_journal);
 
diff --git a/src/box/box.h b/src/box/box.h
index 65215b087..e2321b9b0 100644
--- a/src/box/box.h
+++ b/src/box/box.h
@@ -243,6 +243,7 @@ void box_set_checkpoint_count(void);
 void box_set_checkpoint_interval(void);
 void box_set_checkpoint_wal_threshold(void);
 int box_set_wal_queue_max_size(void);
+int box_set_wal_cleanup_delay(void);
 void box_set_memtx_memory(void);
 void box_set_memtx_max_tuple_size(void);
 void box_set_vinyl_memory(void);
diff --git a/src/box/gc.c b/src/box/gc.c
index 9af4ef958..8dbcbcede 100644
--- a/src/box/gc.c
+++ b/src/box/gc.c
@@ -107,6 +107,11 @@ gc_init(void)
 	/* Don't delete any files until recovery is complete. */
 	gc.min_checkpoint_count = INT_MAX;
 
+	gc.wal_cleanup_delay = TIMEOUT_INFINITY;
+	gc.delay_ref = 0;
+	gc.is_paused = true;
+	say_info("wal/engine cleanup is paused");
+
 	vclock_create(&gc.vclock);
 	rlist_create(&gc.checkpoints);
 	gc_tree_new(&gc.consumers);
@@ -238,6 +243,41 @@ static int
 gc_cleanup_fiber_f(va_list ap)
 {
 	(void)ap;
+
+	/*
+	 * Stage 1 (optional): in case if we're booting
+	 * up with cleanup disabled lets do wait in a
+	 * separate cycle to minimize branching on stage 2.
+	 */
+	if (gc.is_paused) {
+		double start_time = fiber_clock();
+		while (!fiber_is_cancelled()) {
+			double deadline = start_time + gc.wal_cleanup_delay;
+			double timeout = gc.wal_cleanup_delay;
+
+			if (fiber_clock() >= deadline ||
+			    fiber_yield_timeout(timeout)) {
+				say_info("wal/engine cleanup is resumed "
+					 "due to timeout expiration");
+				gc.is_paused = false;
+				gc.delay_ref = 0;
+				break;
+			}
+
+			/*
+			 * If a last reference is dropped
+			 * we can exit out early.
+			 */
+			if (!gc.is_paused) {
+				say_info("wal/engine cleanup is resumed");
+				break;
+			}
+		}
+	}
+
+	/*
+	 * Stage 2: a regular cleanup cycle.
+	 */
 	while (!fiber_is_cancelled()) {
 		int64_t delta = gc.cleanup_scheduled - gc.cleanup_completed;
 		if (delta == 0) {
@@ -253,6 +293,42 @@ gc_cleanup_fiber_f(va_list ap)
 	return 0;
 }
 
+void
+gc_set_wal_cleanup_delay(double wal_cleanup_delay)
+{
+	gc.wal_cleanup_delay = wal_cleanup_delay;
+	/*
+	 * This routine may be called at arbitrary
+	 * moment thus we must be sure the cleanup
+	 * fiber is paused to not wake up it when
+	 * it is already in a regular cleanup stage.
+	 */
+	if (gc.is_paused)
+		fiber_wakeup(gc.cleanup_fiber);
+}
+
+void
+gc_delay_ref(void)
+{
+	if (gc.is_paused) {
+		assert(gc.delay_ref >= 0);
+		gc.delay_ref++;
+	}
+}
+
+void
+gc_delay_unref(void)
+{
+	if (gc.is_paused) {
+		assert(gc.delay_ref > 0);
+		gc.delay_ref--;
+		if (gc.delay_ref == 0) {
+			gc.is_paused = false;
+			fiber_wakeup(gc.cleanup_fiber);
+		}
+	}
+}
+
 /**
  * Trigger asynchronous garbage collection.
  */
@@ -462,11 +538,12 @@ gc_checkpoint(void)
 	 * Wait for background garbage collection that might
 	 * have been triggered by this checkpoint to complete.
 	 * Strictly speaking, it isn't necessary, but it
-	 * simplifies testing as it guarantees that by the
-	 * time box.snapshot() returns, all outdated checkpoint
-	 * files have been removed.
+	 * simplifies testing. Same time if GC is paused and
+	 * waiting for old XLOGs to be read by replicas the
+	 * cleanup won't happen immediately after the checkpoint.
 	 */
-	gc_wait_cleanup();
+	if (!gc.is_paused)
+		gc_wait_cleanup();
 	return 0;
 }
 
diff --git a/src/box/gc.h b/src/box/gc.h
index 2a568c5f9..f5bc26e87 100644
--- a/src/box/gc.h
+++ b/src/box/gc.h
@@ -147,6 +147,24 @@ struct gc_state {
 	 * taken at that moment of time.
 	 */
 	int64_t cleanup_completed, cleanup_scheduled;
+	/**
+	 * A counter to wait until all replicas are managed to
+	 * subscribe so that we can enable cleanup fiber to
+	 * remove old XLOGs. Otherwise some replicas might be
+	 * far behind the master node and after the master
+	 * node been restarted they will have to reread all
+	 * data back due to XlogGapError, ie too early deleted
+	 * XLOGs.
+	 */
+	int64_t delay_ref;
+	/**
+	 * Delay timeout in seconds.
+	 */
+	double wal_cleanup_delay;
+	/**
+	 * When set the cleanup fiber is paused.
+	 */
+	bool is_paused;
 	/**
 	 * Set if there's a fiber making a checkpoint right now.
 	 */
@@ -206,6 +224,24 @@ gc_init(void);
 void
 gc_free(void);
 
+/**
+ * Set a new delay value.
+ */
+void
+gc_set_wal_cleanup_delay(double wal_cleanup_delay);
+
+/**
+ * Increment a reference to delay counter.
+ */
+void
+gc_delay_ref(void);
+
+/**
+ * Decrement a reference from the delay counter.
+ */
+void
+gc_delay_unref(void);
+
 /**
  * Advance the garbage collector vclock to the given position.
  * Deactivate WAL consumers that need older data.
diff --git a/src/box/lua/cfg.cc b/src/box/lua/cfg.cc
index b37a93ed8..1142e2726 100644
--- a/src/box/lua/cfg.cc
+++ b/src/box/lua/cfg.cc
@@ -172,6 +172,14 @@ lbox_cfg_set_wal_queue_max_size(struct lua_State *L)
 	return 0;
 }
 
+static int
+lbox_cfg_set_wal_cleanup_delay(struct lua_State *L)
+{
+	if (box_set_wal_cleanup_delay() < 0)
+		luaT_error(L);
+	return 0;
+}
+
 static int
 lbox_cfg_set_read_only(struct lua_State *L)
 {
@@ -408,6 +416,7 @@ box_lua_cfg_init(struct lua_State *L)
 		{"cfg_set_checkpoint_interval", lbox_cfg_set_checkpoint_interval},
 		{"cfg_set_checkpoint_wal_threshold", lbox_cfg_set_checkpoint_wal_threshold},
 		{"cfg_set_wal_queue_max_size", lbox_cfg_set_wal_queue_max_size},
+		{"cfg_set_wal_cleanup_delay", lbox_cfg_set_wal_cleanup_delay},
 		{"cfg_set_read_only", lbox_cfg_set_read_only},
 		{"cfg_set_memtx_memory", lbox_cfg_set_memtx_memory},
 		{"cfg_set_memtx_max_tuple_size", lbox_cfg_set_memtx_max_tuple_size},
diff --git a/src/box/lua/info.c b/src/box/lua/info.c
index c4c9fa0a0..fb39799a1 100644
--- a/src/box/lua/info.c
+++ b/src/box/lua/info.c
@@ -445,6 +445,10 @@ lbox_info_gc_call(struct lua_State *L)
 	lua_pushboolean(L, gc.checkpoint_is_in_progress);
 	lua_settable(L, -3);
 
+	lua_pushstring(L, "is_paused");
+	lua_pushboolean(L, gc.is_paused);
+	lua_settable(L, -3);
+
 	lua_pushstring(L, "checkpoints");
 	lua_newtable(L);
 
diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua
index 9967f992e..44bb95ed1 100644
--- a/src/box/lua/load_cfg.lua
+++ b/src/box/lua/load_cfg.lua
@@ -73,6 +73,7 @@ local default_cfg = {
     wal_max_size        = 256 * 1024 * 1024,
     wal_dir_rescan_delay= 2,
     wal_queue_max_size  = 16 * 1024 * 1024,
+    wal_cleanup_delay   = 4 * 3600,
     force_recovery      = false,
     replication         = nil,
     instance_uuid       = nil,
@@ -155,6 +156,7 @@ local template_cfg = {
     wal_mode            = 'string',
     wal_max_size        = 'number',
     wal_dir_rescan_delay= 'number',
+    wal_cleanup_delay   = 'number',
     force_recovery      = 'boolean',
     replication         = 'string, number, table',
     instance_uuid       = 'string',
@@ -289,6 +291,7 @@ local dynamic_cfg = {
     feedback_interval       = ifdef_feedback_set_params,
     -- do nothing, affects new replicas, which query this value on start
     wal_dir_rescan_delay    = function() end,
+    wal_cleanup_delay       = private.cfg_set_wal_cleanup_delay,
     custom_proc_title       = function()
         require('title').update(box.cfg.custom_proc_title)
     end,
@@ -349,6 +352,8 @@ local dynamic_cfg_order = {
     -- the new one. This should be fixed when box.cfg is able to
     -- apply some parameters together and atomically.
     replication_anon        = 250,
+    -- Cleanup delay should be ignored if replication_anon is set.
+    wal_cleanup_delay       = 260,
     election_mode           = 300,
     election_timeout        = 320,
 }
diff --git a/src/box/relay.cc b/src/box/relay.cc
index 41f949e8e..6edee86bf 100644
--- a/src/box/relay.cc
+++ b/src/box/relay.cc
@@ -824,6 +824,7 @@ relay_subscribe(struct replica *replica, int fd, uint64_t sync,
 						   tt_uuid_str(&replica->uuid));
 		if (replica->gc == NULL)
 			diag_raise();
+		gc_delay_unref();
 	}
 
 	relay_start(relay, fd, sync, relay_send_row);
diff --git a/src/box/replication.cc b/src/box/replication.cc
index 1fa8843e7..aefb812b3 100644
--- a/src/box/replication.cc
+++ b/src/box/replication.cc
@@ -250,6 +250,7 @@ replica_set_id(struct replica *replica, uint32_t replica_id)
 						   tt_uuid_str(&replica->uuid));
 	}
 	replicaset.replica_by_id[replica_id] = replica;
+	gc_delay_ref();
 	++replicaset.registered_count;
 	say_info("assigned id %d to replica %s",
 		 replica->id, tt_uuid_str(&replica->uuid));
@@ -273,6 +274,7 @@ replica_clear_id(struct replica *replica)
 	replicaset.replica_by_id[replica->id] = NULL;
 	assert(replicaset.registered_count > 0);
 	--replicaset.registered_count;
+	gc_delay_unref();
 	if (replica->id == instance_id) {
 		/* See replica_check_id(). */
 		assert(replicaset.is_joining);
diff --git a/test/app-tap/init_script.result b/test/app-tap/init_script.result
index 74cdb3efb..1fdd9a227 100644
--- a/test/app-tap/init_script.result
+++ b/test/app-tap/init_script.result
@@ -53,6 +53,7 @@ vinyl_run_count_per_level:2
 vinyl_run_size_ratio:3.5
 vinyl_timeout:60
 vinyl_write_threads:4
+wal_cleanup_delay:14400
 wal_dir:.
 wal_dir_rescan_delay:2
 wal_max_size:268435456
diff --git a/test/box/admin.result b/test/box/admin.result
index b619ae6cb..f8e8808e3 100644
--- a/test/box/admin.result
+++ b/test/box/admin.result
@@ -127,6 +127,8 @@ cfg_filter(box.cfg)
     - 60
   - - vinyl_write_threads
     - 4
+  - - wal_cleanup_delay
+    - 14400
   - - wal_dir
     - <hidden>
   - - wal_dir_rescan_delay
diff --git a/test/box/cfg.result b/test/box/cfg.result
index 2c3d5a981..693c1b521 100644
--- a/test/box/cfg.result
+++ b/test/box/cfg.result
@@ -115,6 +115,8 @@ cfg_filter(box.cfg)
  |     - 60
  |   - - vinyl_write_threads
  |     - 4
+ |   - - wal_cleanup_delay
+ |     - 14400
  |   - - wal_dir
  |     - <hidden>
  |   - - wal_dir_rescan_delay
@@ -234,6 +236,8 @@ cfg_filter(box.cfg)
  |     - 60
  |   - - vinyl_write_threads
  |     - 4
+ |   - - wal_cleanup_delay
+ |     - 14400
  |   - - wal_dir
  |     - <hidden>
  |   - - wal_dir_rescan_delay
diff --git a/test/replication/replica_rejoin.result b/test/replication/replica_rejoin.result
index 4bc3df2ca..8ec803af8 100644
--- a/test/replication/replica_rejoin.result
+++ b/test/replication/replica_rejoin.result
@@ -14,6 +14,13 @@ test_run:cleanup_cluster()
 ---
 ...
 --
+-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber
+-- been disabled thus lets turn it off explicitly every time we restart
+-- the main node.
+box.cfg{wal_cleanup_delay = 0}
+---
+...
+--
 -- gh-461: check that a replica refetches the last checkpoint
 -- in case it fell behind the master.
 --
@@ -69,6 +76,9 @@ test_run:cmd("stop server replica")
 -- Restart the server to purge the replica from
 -- the garbage collection state.
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
+---
+...
 -- Make some checkpoints to remove old xlogs.
 checkpoint_count = box.cfg.checkpoint_count
 ---
@@ -190,6 +200,9 @@ test_run:cmd("stop server replica")
 - true
 ...
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
+---
+...
 checkpoint_count = box.cfg.checkpoint_count
 ---
 ...
@@ -313,6 +326,9 @@ test_run:cmd("switch default")
 - true
 ...
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
+---
+...
 replica_listen = test_run:cmd("eval replica 'return box.cfg.listen'")
 ---
 ...
diff --git a/test/replication/replica_rejoin.test.lua b/test/replication/replica_rejoin.test.lua
index 494cbbcb7..3d407d1c8 100644
--- a/test/replication/replica_rejoin.test.lua
+++ b/test/replication/replica_rejoin.test.lua
@@ -5,6 +5,12 @@ engine = test_run:get_cfg('engine')
 
 test_run:cleanup_cluster()
 
+--
+-- gh-5806: this replica_rejoin test relies on the wal cleanup fiber
+-- been disabled thus lets turn it off explicitly every time we restart
+-- the main node.
+box.cfg{wal_cleanup_delay = 0}
+
 --
 -- gh-461: check that a replica refetches the last checkpoint
 -- in case it fell behind the master.
@@ -28,6 +34,7 @@ test_run:cmd("stop server replica")
 -- Restart the server to purge the replica from
 -- the garbage collection state.
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
 
 -- Make some checkpoints to remove old xlogs.
 checkpoint_count = box.cfg.checkpoint_count
@@ -72,6 +79,7 @@ box.space.test:replace{1, 2, 3} -- bumps LSN on the replica
 test_run:cmd("switch default")
 test_run:cmd("stop server replica")
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
 checkpoint_count = box.cfg.checkpoint_count
 box.cfg{checkpoint_count = 1}
 for i = 1, 3 do box.space.test:delete{i * 10} end
@@ -116,6 +124,7 @@ _ = test_run:wait_vclock('default', vclock)
 -- Restart the master and force garbage collection.
 test_run:cmd("switch default")
 test_run:cmd("restart server default")
+box.cfg{wal_cleanup_delay = 0}
 replica_listen = test_run:cmd("eval replica 'return box.cfg.listen'")
 replica_listen ~= nil
 box.cfg{replication = replica_listen}
-- 
2.30.2

[Tarantool-patches] [PATCH v4 2/3] test: add a test for wal_cleanup_delay option

[Cyrill Gorcunov via Tarantool-patches]

Part-of #5806

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
---
 test/replication/gh-5806-master.lua           |   8 +
 test/replication/gh-5806-slave.lua            |   8 +
 test/replication/gh-5806-xlog-cleanup.result  | 356 ++++++++++++++++++
 .../replication/gh-5806-xlog-cleanup.test.lua | 155 ++++++++
 4 files changed, 527 insertions(+)
 create mode 100644 test/replication/gh-5806-master.lua
 create mode 100644 test/replication/gh-5806-slave.lua
 create mode 100644 test/replication/gh-5806-xlog-cleanup.result
 create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua

diff --git a/test/replication/gh-5806-master.lua b/test/replication/gh-5806-master.lua
new file mode 100644
index 000000000..bc15dab67
--- /dev/null
+++ b/test/replication/gh-5806-master.lua
@@ -0,0 +1,8 @@
+#!/usr/bin/env tarantool
+
+require('console').listen(os.getenv('ADMIN'))
+
+box.cfg({
+    listen              = os.getenv("LISTEN"),
+    wal_cleanup_delay   = tonumber(arg[1]) or 0,
+})
diff --git a/test/replication/gh-5806-slave.lua b/test/replication/gh-5806-slave.lua
new file mode 100644
index 000000000..3abb3e035
--- /dev/null
+++ b/test/replication/gh-5806-slave.lua
@@ -0,0 +1,8 @@
+#!/usr/bin/env tarantool
+
+require('console').listen(os.getenv('ADMIN'))
+
+box.cfg({
+    listen              = os.getenv("LISTEN"),
+    replication         = os.getenv("MASTER"),
+})
diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result
new file mode 100644
index 000000000..88b272e62
--- /dev/null
+++ b/test/replication/gh-5806-xlog-cleanup.result
@@ -0,0 +1,356 @@
+-- test-run result file version 2
+--
+-- gh-5806: defer xlog cleanup to keep xlogs until
+-- replicas present in "_cluster" are connected.
+-- Otherwise we are getting XlogGapError since
+-- master might go far forward from replica and
+-- replica won't be able to connect without full
+-- rebootstrap.
+--
+
+fiber = require('fiber')
+ | ---
+ | ...
+test_run = require('test_run').new()
+ | ---
+ | ...
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+
+--
+-- Case 1.
+--
+-- First lets make sure we're getting XlogGapError in
+-- case if wal_cleanup_delay is not used.
+--
+
+test_run:cmd('create server master with script="replication/gh-5806-master.lua"')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server master with wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.schema.user.grant('guest', 'replication')
+ | ---
+ | ...
+
+--
+-- Keep small number of snaps to force cleanup
+-- procedure be more intensive.
+box.cfg{checkpoint_count = 1}
+ | ---
+ | ...
+
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+s = box.schema.space.create('test', {engine = engine})
+ | ---
+ | ...
+_ = s:create_index('pk')
+ | ---
+ | ...
+
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('create server replica with rpl_master=master,\
+              script="replication/gh-5806-slave.lua"')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server replica with wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+
+--
+-- On replica we create an own space which allows us to
+-- use more complex scenario and disables replica from
+-- automatic rejoin (since replica can't do auto-rejoin if
+-- there gonna be an own data loss). This allows us to
+-- trigger XlogGapError in the log.
+test_run:switch('replica')
+ | ---
+ | - true
+ | ...
+box.cfg{checkpoint_count = 1}
+ | ---
+ | ...
+s = box.schema.space.create('testreplica')
+ | ---
+ | ...
+_ = s:create_index('pk')
+ | ---
+ | ...
+box.space.testreplica:insert({1})
+ | ---
+ | - [1]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+
+--
+-- Stop the replica node and generate
+-- xlogs on the master.
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('stop server replica')
+ | ---
+ | - true
+ | ...
+
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.space.test:insert({1})
+ | ---
+ | - [1]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+
+--
+-- We need to restart the master node since otherwise
+-- the replica will be preventing us from removing old
+-- xlog because it will be tracked by gc consumer which
+-- kept in memory while master node is running.
+--
+-- Once restarted we write a new record into master's
+-- space and run snapshot which removes old xlog required
+-- by replica to subscribe leading to XlogGapError which
+-- we need to test.
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('restart server master with wait_load=True')
+ | ---
+ | - true
+ | ...
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.space.test:insert({2})
+ | ---
+ | - [2]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+assert(box.info.gc().is_paused == false)
+ | ---
+ | - true
+ | ...
+
+--
+-- Start replica and wait for error.
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server replica with wait=False, wait_load=False')
+ | ---
+ | - true
+ | ...
+
+--
+-- Wait error to appear.
+test_run:wait_log('master', 'XlogGapError', 1024, 10) ~= nil
+ | ---
+ | - true
+ | ...
+
+--
+-- Cleanup.
+test_run:cmd('stop server master')
+ | ---
+ | - true
+ | ...
+test_run:cmd('stop server replica')
+ | ---
+ | - true
+ | ...
+test_run:cmd('delete server master')
+ | ---
+ | - true
+ | ...
+test_run:cmd('delete server replica')
+ | ---
+ | - true
+ | ...
+
+--
+-- Case 2.
+--
+-- Lets make sure we're not getting XlogGapError in
+-- case if wal_cleanup_delay is used the code is almost
+-- the same as for Case 1 except we don't disable cleanup
+-- fiber but delay it up to a hour until replica is up
+-- and running.
+--
+
+test_run:cmd('create server master with script="replication/gh-5806-master.lua"')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server master with args="3600", wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.schema.user.grant('guest', 'replication')
+ | ---
+ | ...
+
+box.cfg{checkpoint_count = 1}
+ | ---
+ | ...
+
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+s = box.schema.space.create('test', {engine = engine})
+ | ---
+ | ...
+_ = s:create_index('pk')
+ | ---
+ | ...
+
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('create server replica with rpl_master=master,\
+              script="replication/gh-5806-slave.lua"')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server replica with wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+
+test_run:switch('replica')
+ | ---
+ | - true
+ | ...
+box.cfg{checkpoint_count = 1}
+ | ---
+ | ...
+s = box.schema.space.create('testreplica')
+ | ---
+ | ...
+_ = s:create_index('pk')
+ | ---
+ | ...
+box.space.testreplica:insert({1})
+ | ---
+ | - [1]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('stop server replica')
+ | ---
+ | - true
+ | ...
+
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.space.test:insert({1})
+ | ---
+ | - [1]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('restart server master with args="3600", wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+test_run:switch('master')
+ | ---
+ | - true
+ | ...
+box.space.test:insert({2})
+ | ---
+ | - [2]
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+assert(box.info.gc().is_paused == true)
+ | ---
+ | - true
+ | ...
+
+test_run:switch('default')
+ | ---
+ | - true
+ | ...
+test_run:cmd('start server replica with wait=True, wait_load=True')
+ | ---
+ | - true
+ | ...
+
+--
+-- Make sure no error happened.
+assert(test_run:grep_log("master", "XlogGapError") == nil)
+ | ---
+ | - true
+ | ...
+
+test_run:cmd('stop server master')
+ | ---
+ | - true
+ | ...
+test_run:cmd('stop server replica')
+ | ---
+ | - true
+ | ...
+test_run:cmd('delete server master')
+ | ---
+ | - true
+ | ...
+test_run:cmd('delete server replica')
+ | ---
+ | - true
+ | ...
diff --git a/test/replication/gh-5806-xlog-cleanup.test.lua b/test/replication/gh-5806-xlog-cleanup.test.lua
new file mode 100644
index 000000000..8eb1b163b
--- /dev/null
+++ b/test/replication/gh-5806-xlog-cleanup.test.lua
@@ -0,0 +1,155 @@
+--
+-- gh-5806: defer xlog cleanup to keep xlogs until
+-- replicas present in "_cluster" are connected.
+-- Otherwise we are getting XlogGapError since
+-- master might go far forward from replica and
+-- replica won't be able to connect without full
+-- rebootstrap.
+--
+
+fiber = require('fiber')
+test_run = require('test_run').new()
+engine = test_run:get_cfg('engine')
+
+--
+-- Case 1.
+--
+-- First lets make sure we're getting XlogGapError in
+-- case if wal_cleanup_delay is not used.
+--
+
+test_run:cmd('create server master with script="replication/gh-5806-master.lua"')
+test_run:cmd('start server master with wait=True, wait_load=True')
+
+test_run:switch('master')
+box.schema.user.grant('guest', 'replication')
+
+--
+-- Keep small number of snaps to force cleanup
+-- procedure be more intensive.
+box.cfg{checkpoint_count = 1}
+
+engine = test_run:get_cfg('engine')
+s = box.schema.space.create('test', {engine = engine})
+_ = s:create_index('pk')
+
+test_run:switch('default')
+test_run:cmd('create server replica with rpl_master=master,\
+              script="replication/gh-5806-slave.lua"')
+test_run:cmd('start server replica with wait=True, wait_load=True')
+
+--
+-- On replica we create an own space which allows us to
+-- use more complex scenario and disables replica from
+-- automatic rejoin (since replica can't do auto-rejoin if
+-- there gonna be an own data loss). This allows us to
+-- trigger XlogGapError in the log.
+test_run:switch('replica')
+box.cfg{checkpoint_count = 1}
+s = box.schema.space.create('testreplica')
+_ = s:create_index('pk')
+box.space.testreplica:insert({1})
+box.snapshot()
+
+--
+-- Stop the replica node and generate
+-- xlogs on the master.
+test_run:switch('default')
+test_run:cmd('stop server replica')
+
+test_run:switch('master')
+box.space.test:insert({1})
+box.snapshot()
+
+--
+-- We need to restart the master node since otherwise
+-- the replica will be preventing us from removing old
+-- xlog because it will be tracked by gc consumer which
+-- kept in memory while master node is running.
+--
+-- Once restarted we write a new record into master's
+-- space and run snapshot which removes old xlog required
+-- by replica to subscribe leading to XlogGapError which
+-- we need to test.
+test_run:switch('default')
+test_run:cmd('restart server master with wait_load=True')
+test_run:switch('master')
+box.space.test:insert({2})
+box.snapshot()
+assert(box.info.gc().is_paused == false)
+
+--
+-- Start replica and wait for error.
+test_run:switch('default')
+test_run:cmd('start server replica with wait=False, wait_load=False')
+
+--
+-- Wait error to appear.
+test_run:wait_log('master', 'XlogGapError', 1024, 10) ~= nil
+
+--
+-- Cleanup.
+test_run:cmd('stop server master')
+test_run:cmd('stop server replica')
+test_run:cmd('delete server master')
+test_run:cmd('delete server replica')
+
+--
+-- Case 2.
+--
+-- Lets make sure we're not getting XlogGapError in
+-- case if wal_cleanup_delay is used the code is almost
+-- the same as for Case 1 except we don't disable cleanup
+-- fiber but delay it up to a hour until replica is up
+-- and running.
+--
+
+test_run:cmd('create server master with script="replication/gh-5806-master.lua"')
+test_run:cmd('start server master with args="3600", wait=True, wait_load=True')
+
+test_run:switch('master')
+box.schema.user.grant('guest', 'replication')
+
+box.cfg{checkpoint_count = 1}
+
+engine = test_run:get_cfg('engine')
+s = box.schema.space.create('test', {engine = engine})
+_ = s:create_index('pk')
+
+test_run:switch('default')
+test_run:cmd('create server replica with rpl_master=master,\
+              script="replication/gh-5806-slave.lua"')
+test_run:cmd('start server replica with wait=True, wait_load=True')
+
+test_run:switch('replica')
+box.cfg{checkpoint_count = 1}
+s = box.schema.space.create('testreplica')
+_ = s:create_index('pk')
+box.space.testreplica:insert({1})
+box.snapshot()
+
+test_run:switch('default')
+test_run:cmd('stop server replica')
+
+test_run:switch('master')
+box.space.test:insert({1})
+box.snapshot()
+
+test_run:switch('default')
+test_run:cmd('restart server master with args="3600", wait=True, wait_load=True')
+test_run:switch('master')
+box.space.test:insert({2})
+box.snapshot()
+assert(box.info.gc().is_paused == true)
+
+test_run:switch('default')
+test_run:cmd('start server replica with wait=True, wait_load=True')
+
+--
+-- Make sure no error happened.
+assert(test_run:grep_log("master", "XlogGapError") == nil)
+
+test_run:cmd('stop server master')
+test_run:cmd('stop server replica')
+test_run:cmd('delete server master')
+test_run:cmd('delete server replica')
-- 
2.30.2

[Tarantool-patches] [PATCH v4 3/3] test: box-tap/gc -- add test for is_paused field

[Cyrill Gorcunov via Tarantool-patches]

Once simple bootstrap is complete and there is no
replicas used we should run with gc unpaused.

Part-of #5806

Acked-by: Serge Petrenko <sergepetrenko@tarantool.org>
Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>
---
 test/box-tap/gc.test.lua | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/test/box-tap/gc.test.lua b/test/box-tap/gc.test.lua
index f0155779c..ced87547e 100755
--- a/test/box-tap/gc.test.lua
+++ b/test/box-tap/gc.test.lua
@@ -8,11 +8,12 @@ local debug = type(box.error.injection) == "table"
 
 -- check box.info.gc() is false if snapshot is not in progress
 local test = tap.test('box.info.gc')
-test:plan(1 + (debug and 1 or 0))
+test:plan(2 + (debug and 1 or 0))
 
 
 local gc = box.info.gc()
 test:is(gc.checkpoint_is_in_progress, false, "checkpoint is not in progress")
+test:is(gc.is_paused, false, "GC is not paused")
 
 -- check box.info.gc() is true if snapshot is in progress
 --
-- 
2.30.2

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Vladislav Shpilevoy via Tarantool-patches]

Hi! Thanks for the patch!

All the CI jobs are red. Please, try to fix it.

See 2 comments below.

> diff --git a/src/box/box.cc b/src/box/box.cc
> index cc59564e1..a269f7357 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -1465,6 +1480,23 @@ box_set_wal_queue_max_size(void)
>  	return 0;
>  }
>  
> +int
> +box_set_wal_cleanup_delay(void)
> +{
> +	double delay = box_check_wal_cleanup_delay();
> +	if (delay < 0)
> +		return -1;
> +	/*
> +	 * Anonymous replicas do not require
> +	 * delay the cleanup procedure since they
> +	 * are read only.

1. It is not related to being read-only. Besides, they can write
to local spaces at least.

I think it is because anyway they can't be a source of replication.
There is no anybody to keep the files for.

> +	 */
> +	if (replication_anon)
> +		delay = 0;
> +	gc_set_wal_cleanup_delay(delay);
> +	return 0;
> +}
> diff --git a/src/box/gc.c b/src/box/gc.c
> index 9af4ef958..8dbcbcede 100644
> --- a/src/box/gc.c
> +++ b/src/box/gc.c> @@ -238,6 +243,41 @@ static int
>  gc_cleanup_fiber_f(va_list ap)
>  {
>  	(void)ap;
> +
> +	/*
> +	 * Stage 1 (optional): in case if we're booting
> +	 * up with cleanup disabled lets do wait in a
> +	 * separate cycle to minimize branching on stage 2.
> +	 */
> +	if (gc.is_paused) {
> +		double start_time = fiber_clock();
> +		while (!fiber_is_cancelled()) {
> +			double deadline = start_time + gc.wal_cleanup_delay;
> +			double timeout = gc.wal_cleanup_delay;

2. You didn't fix it really. Just that now the timeout is taken
fresh from the config still making it possible to go beyond the
deadline.

The fact that fiber_clock() < deadline does not mean that
fiber_clock() + timeout is also < deadline. You need to calculate
the timeout properly and try to cover it with a test.

> +
> +			if (fiber_clock() >= deadline ||
> +			    fiber_yield_timeout(timeout)) {
> +				say_info("wal/engine cleanup is resumed "
> +					 "due to timeout expiration");
> +				gc.is_paused = false;
> +				gc.delay_ref = 0;
> +				break;
> +			}
> +
> +			/*
> +			 * If a last reference is dropped
> +			 * we can exit out early.
> +			 */
> +			if (!gc.is_paused) {
> +				say_info("wal/engine cleanup is resumed");
> +				break;
> +			}
> +		}
> +	}

Re: [Tarantool-patches] [PATCH v4 2/3] test: add a test for wal_cleanup_delay option

[Vladislav Shpilevoy via Tarantool-patches]

Thanks for working on this!

See 4 comments below.

> diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result
> new file mode 100644
> index 000000000..88b272e62
> --- /dev/null
> +++ b/test/replication/gh-5806-xlog-cleanup.result
> @@ -0,0 +1,356 @@
> +--
> +-- On replica we create an own space which allows us to
> +-- use more complex scenario and disables replica from
> +-- automatic rejoin (since replica can't do auto-rejoin if
> +-- there gonna be an own data loss). This allows us to
> +-- trigger XlogGapError in the log.
> +test_run:switch('replica')
> + | ---
> + | - true
> + | ...
> +box.cfg{checkpoint_count = 1}
> + | ---
> + | ...
> +s = box.schema.space.create('testreplica')
> + | ---
> + | ...
> +_ = s:create_index('pk')
> + | ---
> + | ...
> +box.space.testreplica:insert({1})
> + | ---
> + | - [1]
> + | ...
> +box.snapshot()
> + | ---
> + | - ok
> + | ...
> +
> +--
> +-- Stop the replica node and generate
> +-- xlogs on the master.
> +test_run:switch('default')

1. You could switch to the master right away. No
need to switch through the default node.

> + | ---
> + | - true
> + | ...
> +test_run:cmd('stop server replica')
> + | ---
> + | - true
> + | ...
> +
> +test_run:switch('master')
> + | ---
> + | - true
> + | ...
> +box.space.test:insert({1})
> + | ---
> + | - [1]
> + | ...
> +box.snapshot()
> + | ---
> + | - ok
> + | ...
> +
> +--
> +-- We need to restart the master node since otherwise
> +-- the replica will be preventing us from removing old
> +-- xlog because it will be tracked by gc consumer which
> +-- kept in memory while master node is running.
> +--
> +-- Once restarted we write a new record into master's
> +-- space and run snapshot which removes old xlog required
> +-- by replica to subscribe leading to XlogGapError which
> +-- we need to test.
> +test_run:switch('default')

2. You can restart the master without switching to default.

> + | ---
> + | - true
> + | ...
> +test_run:cmd('restart server master with wait_load=True')
> + | ---
> + | - true
> + | ...
> +test_run:switch('master')
> + | ---
> + | - true
> + | ...
> +box.space.test:insert({2})
> + | ---
> + | - [2]
> + | ...
> +box.snapshot()
> + | ---
> + | - ok
> + | ...
> +assert(box.info.gc().is_paused == false)
> + | ---
> + | - true
> + | ...
> +
> +--
> +-- Start replica and wait for error.
> +test_run:switch('default')

3. You don't need to switch to default to start the replica. All the
same switch-comments for 'Case 2' test.

> + | ---
> + | - true
> + | ...
> +test_run:cmd('start server replica with wait=False, wait_load=False')
> + | ---
> + | - true
> + | ...
> +
> +--
> +-- Wait error to appear.
> +test_run:wait_log('master', 'XlogGapError', 1024, 10) ~= nil

4. 10 seconds is a tiny timeout. Seconds is not enough as the
synchronous replication tests have shown. But why do you even need
non-default parameters?

> + | ---
> + | - true
> + | ...
> +

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

On Wed, Mar 24, 2021 at 11:10:11PM +0100, Vladislav Shpilevoy wrote:
> > +	if (gc.is_paused) {
> > +		double start_time = fiber_clock();
> > +		while (!fiber_is_cancelled()) {
> > +			double deadline = start_time + gc.wal_cleanup_delay;
> > +			double timeout = gc.wal_cleanup_delay;
> 
> 2. You didn't fix it really. Just that now the timeout is taken
> fresh from the config still making it possible to go beyond the
> deadline.
> 
> The fact that fiber_clock() < deadline does not mean that
> fiber_clock() + timeout is also < deadline. You need to calculate
> the timeout properly and try to cover it with a test.

Here is an update on top. I see a few red tests on our github actions
but still trying to figure out what is happening, since I've been
passing the tests on my local instance
---
diff --git a/src/box/box.cc b/src/box/box.cc
index a269f7357..ab298d223 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -774,10 +774,18 @@ box_check_wal_queue_max_size(void)
 static double
 box_check_wal_cleanup_delay(void)
 {
+	const double MAX_TIMEOUT = TIMEOUT_INFINITY;
+	const double MIN_TIMEOUT = 0.001;
+
 	double value = cfg_getd("wal_cleanup_delay");
-	if (value < 0) {
+	if (value < 0 || (value != 0 && value < MIN_TIMEOUT) ||
+	    value > MAX_TIMEOUT) {
+		char message[64];
+		snprintf(message, sizeof(message),
+			 "the value must be 0 or in range [%g; %g]",
+			 MIN_TIMEOUT, TIMEOUT_INFINITY);
 		diag_set(ClientError, ER_CFG, "wal_cleanup_delay",
-			 "the value must be >= 0");
+			 message);
 		return -1;
 	}
 
@@ -1488,8 +1496,8 @@ box_set_wal_cleanup_delay(void)
 		return -1;
 	/*
 	 * Anonymous replicas do not require
-	 * delay the cleanup procedure since they
-	 * are read only.
+	 * delay since they can't be a source
+	 * of replication.
 	 */
 	if (replication_anon)
 		delay = 0;
diff --git a/src/box/gc.c b/src/box/gc.c
index 8dbcbcede..10f899923 100644
--- a/src/box/gc.c
+++ b/src/box/gc.c
@@ -251,12 +251,9 @@ gc_cleanup_fiber_f(va_list ap)
 	 */
 	if (gc.is_paused) {
 		double start_time = fiber_clock();
+		double timeout = gc.wal_cleanup_delay;
 		while (!fiber_is_cancelled()) {
-			double deadline = start_time + gc.wal_cleanup_delay;
-			double timeout = gc.wal_cleanup_delay;
-
-			if (fiber_clock() >= deadline ||
-			    fiber_yield_timeout(timeout)) {
+			if (fiber_yield_timeout(timeout)) {
 				say_info("wal/engine cleanup is resumed "
 					 "due to timeout expiration");
 				gc.is_paused = false;
@@ -272,6 +269,19 @@ gc_cleanup_fiber_f(va_list ap)
 				say_info("wal/engine cleanup is resumed");
 				break;
 			}
+
+			/*
+			 * Woken up to update the timeout.
+			 */
+			double elapsed = fiber_clock() - start_time;
+			if (elapsed >= gc.wal_cleanup_delay) {
+				say_info("wal/engine cleanup is resumed "
+					 "due to timeout manual update");
+				gc.is_paused = false;
+				gc.delay_ref = 0;
+				break;
+			}
+			timeout = gc.wal_cleanup_delay - elapsed;
 		}
 	}
 

Re: [Tarantool-patches] [PATCH v4 2/3] test: add a test for wal_cleanup_delay option

[Cyrill Gorcunov via Tarantool-patches]

On Wed, Mar 24, 2021 at 11:10:23PM +0100, Vladislav Shpilevoy wrote:
> Thanks for working on this!
> 
> See 4 comments below.
> 
> > +
> > +--
> > +-- Stop the replica node and generate
> > +-- xlogs on the master.
> > +test_run:switch('default')
> 
> 1. You could switch to the master right away. No
> need to switch through the default node.

No. I'm *on* the replica at this moment and then
I need to stop it. I can't stop server being on
it, otherwise we hang. Already tried.

> > +test_run:switch('default')
> 
> 2. You can restart the master without switching to default.

Yes, thanks. And same for rest of comments, will update.

Re: [Tarantool-patches] [PATCH v4 2/3] test: add a test for wal_cleanup_delay option

[Vladislav Shpilevoy via Tarantool-patches]

On 25.03.2021 13:07, Cyrill Gorcunov wrote:
> On Wed, Mar 24, 2021 at 11:10:23PM +0100, Vladislav Shpilevoy wrote:
>> Thanks for working on this!
>>
>> See 4 comments below.
>>
>>> +
>>> +--
>>> +-- Stop the replica node and generate
>>> +-- xlogs on the master.
>>> +test_run:switch('default')
>>
>> 1. You could switch to the master right away. No
>> need to switch through the default node.
> 
> No. I'm *on* the replica at this moment and then
> I need to stop it. I can't stop server being on
> it, otherwise we hang. Already tried.

I know you are on the replica. And you need to switch to the
master. Then switch to the master. I don't see why do you need to
do it ->default->master instead of just ->master way.

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Vladislav Shpilevoy via Tarantool-patches]

> Here is an update on top. I see a few red tests on our github actions
> but still trying to figure out what is happening, since I've been
> passing the tests on my local instance
> ---
> diff --git a/src/box/box.cc b/src/box/box.cc
> index a269f7357..ab298d223 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -774,10 +774,18 @@ box_check_wal_queue_max_size(void)
>  static double
>  box_check_wal_cleanup_delay(void)
>  {
> +	const double MAX_TIMEOUT = TIMEOUT_INFINITY;
> +	const double MIN_TIMEOUT = 0.001;

I am going to repeat it here from what I said verbally and
in the chat - don't restrict the timeout. We never restrict
any timeouts. TIMEOUT_INFINITY is not literally double inf
value. It is just some huge but valid double value. User in
his code can have a bigger definition of what is infinity.

The same with the min. Why do you limit it from below? I don't
see a single reason for doing so. Only reasons against that -
it is inconsistent with the other timeouts we have, and might
conflict with how each particular user understand the "minimal"
timeout.

Instead, I asked you to check what if I pass Lua's math.huge
value. AFAIK, it is not some finite double value, and it might
break something.

> +
>  	double value = cfg_getd("wal_cleanup_delay");
> -	if (value < 0) {
> +	if (value < 0 || (value != 0 && value < MIN_TIMEOUT) ||
> +	    value > MAX_TIMEOUT) {
> +		char message[64];
> +		snprintf(message, sizeof(message),
> +			 "the value must be 0 or in range [%g; %g]",
> +			 MIN_TIMEOUT, TIMEOUT_INFINITY);
>  		diag_set(ClientError, ER_CFG, "wal_cleanup_delay",
> -			 "the value must be >= 0");
> +			 message);
>  		return -1;
>  	}
>  

Re: [Tarantool-patches] [PATCH v4 2/3] test: add a test for wal_cleanup_delay option

[Cyrill Gorcunov via Tarantool-patches]

On Thu, Mar 25, 2021 at 08:56:15PM +0100, Vladislav Shpilevoy wrote:
> >>
> >> 1. You could switch to the master right away. No
> >> need to switch through the default node.
> > 
> > No. I'm *on* the replica at this moment and then
> > I need to stop it. I can't stop server being on
> > it, otherwise we hang. Already tried.
> 
> I know you are on the replica. And you need to switch to the
> master. Then switch to the master. I don't see why do you need to
> do it ->default->master instead of just ->master way.

Yeah, thanks! I realised this a bit later. Done like

--
-- Stop the replica node and generate
-- xlogs on the master.
test_run:switch('master')
test_run:cmd('stop server replica')

box.space.test:insert({1})
box.snapshot()


	Cyrill

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

On Thu, Mar 25, 2021 at 08:59:53PM +0100, Vladislav Shpilevoy wrote:
> >  static double
> >  box_check_wal_cleanup_delay(void)
> >  {
> > +	const double MAX_TIMEOUT = TIMEOUT_INFINITY;
> > +	const double MIN_TIMEOUT = 0.001;
> 
> I am going to repeat it here from what I said verbally and
> in the chat - don't restrict the timeout. We never restrict
> any timeouts. TIMEOUT_INFINITY is not literally double inf
> value. It is just some huge but valid double value. User in
> his code can have a bigger definition of what is infinity.
> 
> The same with the min. Why do you limit it from below? I don't
> see a single reason for doing so. Only reasons against that -
> it is inconsistent with the other timeouts we have, and might
> conflict with how each particular user understand the "minimal"
> timeout.

Most important reason is an epsilon value on hw level, how many
users remember what exactly range the "machine zero" covers?
Do we really want to allow precision close to nanoseconds?
Surely it is not a problem to remove this limits from incoming
values, will do but actually this a bit bothers me.

> Instead, I asked you to check what if I pass Lua's math.huge
> value. AFAIK, it is not some finite double value, and it might
> break something.

LJLIB_PUSH(1e310) LJLIB_SET(huge)

If I remove the limits in arguments and call

	box.cfg{wal_cleanup_delay = math.huge}

then it passed into C level as 0x7fffffffffffffff
which is NaN in ieee-754.

2021-03-25 23:53:45.588 [2068763] main/103/interactive I> set 'wal_cleanup_delay' configuration option to inf

and cleanup fiber is sleeping forever because of comparition
specifics with NaN value.

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

On Fri, Mar 26, 2021 at 12:02:56AM +0300, Cyrill Gorcunov wrote:
> 
> then it passed into C level as 0x7fffffffffffffff
> which is NaN in ieee-754.
> 
> 2021-03-25 23:53:45.588 [2068763] main/103/interactive I> set 'wal_cleanup_delay' configuration option to inf

I changed the code to

static double
box_check_wal_cleanup_delay(void)
{
	double value = cfg_getd("wal_cleanup_delay");
	if (isfinite(value) == 0 && value >= 0) {
		diag_set(ClientError, ER_CFG, "wal_cleanup_delay",
			 "value must be finite and >= 0");
		return -1;
	}

	return value;
}

 | tarantool> box.cfg{wal_cleanup_delay = math.huge}
 | ---
 | - error: 'Incorrect value for option ''wal_cleanup_delay'': value must be finite and
 |     >= 0'
 | ...

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Vladislav Shpilevoy via Tarantool-patches]

On 25.03.2021 22:29, Cyrill Gorcunov wrote:
> On Fri, Mar 26, 2021 at 12:02:56AM +0300, Cyrill Gorcunov wrote:
>>
>> then it passed into C level as 0x7fffffffffffffff
>> which is NaN in ieee-754.
>>
>> 2021-03-25 23:53:45.588 [2068763] main/103/interactive I> set 'wal_cleanup_delay' configuration option to inf
> 
> I changed the code to
> 
> static double
> box_check_wal_cleanup_delay(void)
> {
> 	double value = cfg_getd("wal_cleanup_delay");
> 	if (isfinite(value) == 0 && value >= 0) {
> 		diag_set(ClientError, ER_CFG, "wal_cleanup_delay",
> 			 "value must be finite and >= 0");
> 		return -1;
> 	}
> 
> 	return value;
> }
> 
>  | tarantool> box.cfg{wal_cleanup_delay = math.huge}
>  | ---
>  | - error: 'Incorrect value for option ''wal_cleanup_delay'': value must be finite and
>  |     >= 0'
>  | ...
> 

You don't need to change it to finite if it works fine being infinite.
I only wanted to have a test that it does not crash somewhere on that.

If it would crash, it would mean we would need to fix the other timeouts
too. Otherwise your timeout still is inconsistent with them if you ban
the literal infinity.

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Vladislav Shpilevoy via Tarantool-patches]

On 25.03.2021 22:02, Cyrill Gorcunov via Tarantool-patches wrote:
> On Thu, Mar 25, 2021 at 08:59:53PM +0100, Vladislav Shpilevoy wrote:
>>>  static double
>>>  box_check_wal_cleanup_delay(void)
>>>  {
>>> +	const double MAX_TIMEOUT = TIMEOUT_INFINITY;
>>> +	const double MIN_TIMEOUT = 0.001;
>>
>> I am going to repeat it here from what I said verbally and
>> in the chat - don't restrict the timeout. We never restrict
>> any timeouts. TIMEOUT_INFINITY is not literally double inf
>> value. It is just some huge but valid double value. User in
>> his code can have a bigger definition of what is infinity.
>>
>> The same with the min. Why do you limit it from below? I don't
>> see a single reason for doing so. Only reasons against that -
>> it is inconsistent with the other timeouts we have, and might
>> conflict with how each particular user understand the "minimal"
>> timeout.
> 
> Most important reason is an epsilon value on hw level, how many
> users remember what exactly range the "machine zero" covers?

I don't remember, and I don't see why should I really. What is the
reason you ban 0.0009 but allow 0.001? This looks random and does
not protect from anything.

> Do we really want to allow precision close to nanoseconds?

We allow double. Whatever precision it gives.

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

On Fri, Mar 26, 2021 at 12:50:40AM +0100, Vladislav Shpilevoy wrote:
> > 
> >  | tarantool> box.cfg{wal_cleanup_delay = math.huge}
> >  | ---
> >  | - error: 'Incorrect value for option ''wal_cleanup_delay'': value must be finite and
> >  |     >= 0'
> >  | ...
> > 
> 
> You don't need to change it to finite if it works fine being infinite.
> I only wanted to have a test that it does not crash somewhere on that.
> 
> If it would crash, it would mean we would need to fix the other timeouts
> too. Otherwise your timeout still is inconsistent with them if you ban
> the literal infinity.

I personally think this is wrong to allow enter arbitrary values here
but won't insist, I'll drop this test.

Re: [Tarantool-patches] [PATCH v4 1/3] gc/xlog: delay xlog cleanup until relays are subscribed

[Cyrill Gorcunov via Tarantool-patches]

On Fri, Mar 26, 2021 at 12:51:57AM +0100, Vladislav Shpilevoy wrote:
> > 
> > Most important reason is an epsilon value on hw level, how many
> > users remember what exactly range the "machine zero" covers?
> 
> I don't remember, and I don't see why should I really. What is the
> reason you ban 0.0009 but allow 0.001? This looks random and does
> not protect from anything.
> 
> > Do we really want to allow precision close to nanoseconds?
> 
> We allow double. Whatever precision it gives.

OK, thanks for explanation.