From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp58.i.mail.ru (smtp58.i.mail.ru [217.69.128.38]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 4A80043E89B for ; Fri, 27 Mar 2020 13:40:11 +0300 (MSK) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) From: Serge Petrenko In-Reply-To: <20200323165140.GE5491@atlas> Date: Fri, 27 Mar 2020 13:40:09 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <1FC237DE-B191-43B2-BF40-AEDF6E190EF2@tarantool.org> References: <20200323165140.GE5491@atlas> Subject: Re: [Tarantool-patches] [PATCH v3 4/4] box: start counting local space requests separately List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Konstantin Osipov Cc: tarantool-patches@dev.tarantool.org, Vladislav Shpilevoy Hi! Thank you for the review! > 23 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 2020 =D0=B3., =D0=B2 19:51, = Konstantin Osipov =D0=BD=D0=B0=D0=BF=D0=B8=D1=81= =D0=B0=D0=BB(=D0=B0): >=20 > * Serge Petrenko [20/03/23 19:23]: >> Sign local space requests with a zero instance id. This allows to = split >> local changes aside from the changes, which should be visible to the = whole >> cluster, and stop sending NOPs to replicas to follow local vclock. >=20 > First of all, this is an unrelated fix and should be sent as a > separate series. >=20 > Second, I'm not sure how a separate fix is necessary, it seems > lexicographical vclock compare should close 4114. Well, no. I described why this fix is needed in the commit message, as you requested. I can send a separate patchset which just reworks gc to work with clocks instead of signatures, if you want me to. It=E2=80=99s a nice feature to have on its own, but it was done in scope = of fixing this bug (#4114), and it is a prerequisite for this fix, so I don=E2=80=99t think it=E2=80=99s worth splitting them apart. >=20 > An explanation and rationale for the fix belongs to the changeset > comment. Similar to how you explained GC bug. Done. >=20 > Third, if server id 0 is special, it should be special on vclock api > level, not introduced with a bunch of hacks on replication level. >=20 > We should have vclock_signature() and vclock_local_signature() > (the latter includes server id 0), vclock_cmp and > vclock_local_cmp(), vclock_lexicographical_cmp() and > vclock_local_lexicographical_cmp() and so on. >=20 > This would make sure that we don't forget to "clear" server id 0 > in some place which expects it. If I understood correctly, you mean that we have to rewrite recovery so that it accepts a flag, whether a local recovery is needed, or a relay one, when rows with id 0 are ignored. I=E2=80=99m not sure it=E2=80=99= ll work correctly or look better than hacking replica vclock in one place. If you insist on rewriting recovery, I=E2=80=99ll do that, but please = see the patchset first. Speaking of other places where server id 0 is cleared (commit #3), it=E2=80=99s still needed for backward compatibility. Older replicas do = not ignore server id 0. I reworked gc though, so now it doesn=E2=80=99t depend on replica id 0 = at all, and =C2=ABclearing=C2=BB server id 0 is now needed only for recovery. (I added vclock_min_ignore0, to be consistent with = vclock_compare_ignore0 Feel free to propose a better name, if you want to) >=20 > Finally, I find using local vclock id for local changes very neat > and justifying introduction of local vclock id. I really thought > that such an ugly hack is not worth having for anonymous replicas > support. But together with skipping nops for local changes, it > actually begins to make sense. Fine, let's try to have this > feature, but let's implement it properly then. >=20 >> --- >> src/box/box.cc | 20 +++ >> src/box/relay.cc | 15 +-- >> src/box/wal.c | 16 ++- >> test/replication/autobootstrap.result | 6 + >> test/replication/autobootstrap.test.lua | 2 + >> test/replication/before_replace.result | 8 +- >> test/replication/before_replace.test.lua | 4 +- >> .../gh-4114-local-space-replication.result | 125 = ++++++++++++++++++ >> .../gh-4114-local-space-replication.test.lua | 48 +++++++ >> test/replication/local_spaces.result | 4 + >> test/replication/local_spaces.test.lua | 3 + >> test/replication/misc.result | 6 + >> test/replication/misc.test.lua | 2 + >> test/replication/quorum.result | 6 + >> test/replication/quorum.test.lua | 2 + >> test/replication/replica_rejoin.result | 9 ++ >> test/replication/replica_rejoin.test.lua | 3 + >> test/replication/skip_conflict_row.result | 3 + >> test/replication/skip_conflict_row.test.lua | 1 + >> test/replication/suite.cfg | 1 + >> test/vinyl/errinj.result | 5 + >> test/vinyl/errinj.test.lua | 4 + >> 22 files changed, 278 insertions(+), 15 deletions(-) >> create mode 100644 = test/replication/gh-4114-local-space-replication.result >> create mode 100644 = test/replication/gh-4114-local-space-replication.test.lua >>=20 >> diff --git a/src/box/box.cc b/src/box/box.cc >> index 6de6b677c..b7ea0de2d 100644 >> --- a/src/box/box.cc >> +++ b/src/box/box.cc >> @@ -1882,6 +1882,26 @@ box_process_subscribe(struct ev_io *io, struct = xrow_header *header) >> say_info("remote vclock %s local vclock %s", >> vclock_to_string(&replica_clock), = vclock_to_string(&vclock)); >>=20 >> + /* >> + * Replica clock is used in gc state and recovery >> + * initialization, so we need to replace the remote 0-th >> + * component with our own one. This makes recovery work >> + * correctly: we're trying to recover from an xlog whose >> + * vclock is smaller than remote replica clock in each >> + * component, exluding the 0-th component. This leads to >> + * finding the correct WAL, if it exists, since we do not >> + * need to recover local rows (the ones, that contribute >> + * to the 0-th vclock component). It would be bad to set >> + * 0-th vclock component to a smaller value, since it >> + * would unnecessarily require additional WALs, which may >> + * have already been deleted. >> + * Same stands for gc. Remote instances do not need this >> + * instance's local rows, and after gc was reworked to >> + * track individual vclock components instead of >> + * signatures it's safe to set the local component to the >> + * most recent value. >> + */ >> + vclock_set(&replica_clock, 0, vclock_get(&replicaset.vclock, = 0)); >> /* >> * Process SUBSCRIBE request via replication relay >> * Send current recovery vector clock as a marker >> diff --git a/src/box/relay.cc b/src/box/relay.cc >> index 21b7c0a6f..a91e4b771 100644 >> --- a/src/box/relay.cc >> +++ b/src/box/relay.cc >> @@ -760,16 +760,15 @@ relay_send_row(struct xstream *stream, struct = xrow_header *packet) >> { >> struct relay *relay =3D container_of(stream, struct relay, = stream); >> assert(iproto_type_is_dml(packet->type)); >> - /* >> - * Transform replica local requests to IPROTO_NOP so as to >> - * promote vclock on the replica without actually modifying >> - * any data. >> - */ >> if (packet->group_id =3D=3D GROUP_LOCAL) { >> /* >> - * Replica-local requests generated while replica >> - * was anonymous have a zero instance id. Just >> - * skip all these rows. >> + * We do not relay replica-local rows to other >> + * instances, since we started signing them with >> + * a zero instance id. However, if replica-local >> + * rows, signed with a non-zero id are present in >> + * our WAL, we still need to relay them as NOPs in >> + * order to correctly promote the vclock on the >> + * replica. >> */ >> if (packet->replica_id =3D=3D REPLICA_ID_NIL) >> return; >> diff --git a/src/box/wal.c b/src/box/wal.c >> index 3b094b0e8..a74bdecd9 100644 >> --- a/src/box/wal.c >> +++ b/src/box/wal.c >> @@ -953,13 +953,19 @@ wal_assign_lsn(struct vclock *vclock_diff, = struct vclock *base, >> /** Assign LSN to all local rows. */ >> for ( ; row < end; row++) { >> if ((*row)->replica_id =3D=3D 0) { >> - (*row)->lsn =3D vclock_inc(vclock_diff, = instance_id) + >> - vclock_get(base, instance_id); >> /* >> - * Note, an anonymous replica signs local >> - * rows whith a zero instance id. >> + * All rows representing local space data >> + * manipulations are signed wth a zero >> + * instance id. This is also true for >> + * anonymous replicas, since they are >> + * only capable of writing to local and >> + * temporary spaces. >> */ >> - (*row)->replica_id =3D instance_id; >> + if ((*row)->group_id !=3D GROUP_LOCAL) >> + (*row)->replica_id =3D instance_id; >> + >> + (*row)->lsn =3D vclock_inc(vclock_diff, = (*row)->replica_id) + >> + vclock_get(base, = (*row)->replica_id); >> /* Use lsn of the first local row as transaction = id. */ >> tsn =3D tsn =3D=3D 0 ? (*row)->lsn : tsn; >> (*row)->tsn =3D tsn; >> diff --git a/test/replication/autobootstrap.result = b/test/replication/autobootstrap.result >> index 743982d47..6918e23ea 100644 >> --- a/test/replication/autobootstrap.result >> +++ b/test/replication/autobootstrap.result >> @@ -162,6 +162,9 @@ box.schema.user.revoke('test_u', 'create', = 'space') >> vclock =3D test_run:get_vclock('autobootstrap1') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> --- >> ... >> @@ -206,6 +209,9 @@ test_run:wait_fullmesh(SERVERS) >> vclock =3D test_run:get_vclock("autobootstrap1") >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> --- >> ... >> diff --git a/test/replication/autobootstrap.test.lua = b/test/replication/autobootstrap.test.lua >> index 055ea4277..f8bb1c74a 100644 >> --- a/test/replication/autobootstrap.test.lua >> +++ b/test/replication/autobootstrap.test.lua >> @@ -74,6 +74,7 @@ box.schema.user.revoke('test_u', 'create', 'space') >>=20 >> -- Synchronize >> vclock =3D test_run:get_vclock('autobootstrap1') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> _ =3D test_run:wait_vclock("autobootstrap3", vclock) >>=20 >> @@ -95,6 +96,7 @@ _ =3D test_run:cmd("switch default") >> test_run:wait_fullmesh(SERVERS) >>=20 >> vclock =3D test_run:get_vclock("autobootstrap1") >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> _ =3D test_run:wait_vclock("autobootstrap3", vclock) >>=20 >> diff --git a/test/replication/before_replace.result = b/test/replication/before_replace.result >> index ced40547e..61a552e84 100644 >> --- a/test/replication/before_replace.result >> +++ b/test/replication/before_replace.result >> @@ -266,7 +266,13 @@ box.space.test:replace{1, 1} >> --- >> - [1, 1] >> ... >> -_ =3D test_run:wait_vclock('replica', = test_run:get_vclock('default')) >> +vclock =3D test_run:get_vclock('default') >> +--- >> +... >> +vclock[0] =3D nil >> +--- >> +... >> +_ =3D test_run:wait_vclock('replica', vclock) >> --- >> ... >> -- Check that replace{1, 2} coming from the master was suppressed >> diff --git a/test/replication/before_replace.test.lua = b/test/replication/before_replace.test.lua >> index bcc6dc00d..ecd8ff044 100644 >> --- a/test/replication/before_replace.test.lua >> +++ b/test/replication/before_replace.test.lua >> @@ -101,7 +101,9 @@ _ =3D box.space.test:before_replace(function(old, = new) return new:update{{'+', 2, >> test_run:cmd("switch default") >> box.space.test:replace{1, 1} >>=20 >> -_ =3D test_run:wait_vclock('replica', = test_run:get_vclock('default')) >> +vclock =3D test_run:get_vclock('default') >> +vclock[0] =3D nil >> +_ =3D test_run:wait_vclock('replica', vclock) >>=20 >> -- Check that replace{1, 2} coming from the master was suppressed >> -- by the before_replace trigger on the replica. >> diff --git a/test/replication/gh-4114-local-space-replication.result = b/test/replication/gh-4114-local-space-replication.result >> new file mode 100644 >> index 000000000..e524c9a1b >> --- /dev/null >> +++ b/test/replication/gh-4114-local-space-replication.result >> @@ -0,0 +1,125 @@ >> +-- test-run result file version 2 >> +env =3D require('test_run') >> + | --- >> + | ... >> +test_run =3D env.new() >> + | --- >> + | ... >> + >> +-- >> +-- gh-4114. Account local space changes in a separate vclock >> +-- component. Do not replicate local space changes, even as NOPs. >> +-- >> + >> +box.schema.user.grant('guest', 'replication') >> + | --- >> + | ... >> +_ =3D box.schema.space.create('test', {is_local=3Dtrue}) >> + | --- >> + | ... >> +_ =3D box.space.test:create_index("pk") >> + | --- >> + | ... >> + >> +test_run:cmd('create server replica with rpl_master=3Ddefault, = script "replication/replica.lua"') >> + | --- >> + | - true >> + | ... >> +test_run:cmd('start server replica with wait=3DTrue, = wait_load=3DTrue') >> + | --- >> + | - true >> + | ... >> + >> +a =3D box.info.vclock[0] or 0 >> + | --- >> + | ... >> +for i =3D 1,10 do box.space.test:insert{i} end >> + | --- >> + | ... >> +box.info.vclock[0] =3D=3D a + 10 or box.info.vclock[0] - a >> + | --- >> + | - true >> + | ... >> + >> +test_run:cmd('switch replica') >> + | --- >> + | - true >> + | ... >> +box.info.vclock[0] >> + | --- >> + | - null >> + | ... >> +box.cfg{checkpoint_count=3D1} >> + | --- >> + | ... >> +box.space.test:select{} >> + | --- >> + | - [] >> + | ... >> +box.space.test:insert{1} >> + | --- >> + | - [1] >> + | ... >> +box.snapshot() >> + | --- >> + | - ok >> + | ... >> +box.space.test:insert{2} >> + | --- >> + | - [2] >> + | ... >> +box.snapshot() >> + | --- >> + | - ok >> + | ... >> +box.space.test:insert{3} >> + | --- >> + | - [3] >> + | ... >> +box.snapshot() >> + | --- >> + | - ok >> + | ... >> + >> +box.info.vclock[0] >> + | --- >> + | - 3 >> + | ... >> + >> +test_run:cmd('switch default') >> + | --- >> + | - true >> + | ... >> + >> +test_run:cmd('set variable repl_source to "replica.listen"') >> + | --- >> + | - true >> + | ... >> + >> +box.cfg{replication=3Drepl_source} >> + | --- >> + | ... >> +test_run:wait_cond(function()\ >> + return box.info.replication[2].upstream and\ >> + box.info.replication[2].upstream.status =3D=3D = 'follow'\ >> + end,\ >> + 10) >> + | --- >> + | - true >> + | ... >> + >> +-- Cleanup. >> +test_run:cmd('stop server replica') >> + | --- >> + | - true >> + | ... >> +test_run:cmd('delete server replica') >> + | --- >> + | - true >> + | ... >> +box.space.test:drop() >> + | --- >> + | ... >> +box.schema.user.revoke('guest', 'replication') >> + | --- >> + | ... >> diff --git = a/test/replication/gh-4114-local-space-replication.test.lua = b/test/replication/gh-4114-local-space-replication.test.lua >> new file mode 100644 >> index 000000000..26dccee68 >> --- /dev/null >> +++ b/test/replication/gh-4114-local-space-replication.test.lua >> @@ -0,0 +1,48 @@ >> +env =3D require('test_run') >> +test_run =3D env.new() >> + >> +-- >> +-- gh-4114. Account local space changes in a separate vclock >> +-- component. Do not replicate local space changes, even as NOPs. >> +-- >> + >> +box.schema.user.grant('guest', 'replication') >> +_ =3D box.schema.space.create('test', {is_local=3Dtrue}) >> +_ =3D box.space.test:create_index("pk") >> + >> +test_run:cmd('create server replica with rpl_master=3Ddefault, = script "replication/replica.lua"') >> +test_run:cmd('start server replica with wait=3DTrue, = wait_load=3DTrue') >> + >> +a =3D box.info.vclock[0] or 0 >> +for i =3D 1,10 do box.space.test:insert{i} end >> +box.info.vclock[0] =3D=3D a + 10 or box.info.vclock[0] - a >> + >> +test_run:cmd('switch replica') >> +box.info.vclock[0] >> +box.cfg{checkpoint_count=3D1} >> +box.space.test:select{} >> +box.space.test:insert{1} >> +box.snapshot() >> +box.space.test:insert{2} >> +box.snapshot() >> +box.space.test:insert{3} >> +box.snapshot() >> + >> +box.info.vclock[0] >> + >> +test_run:cmd('switch default') >> + >> +test_run:cmd('set variable repl_source to "replica.listen"') >> + >> +box.cfg{replication=3Drepl_source} >> +test_run:wait_cond(function()\ >> + return box.info.replication[2].upstream and\ >> + box.info.replication[2].upstream.status =3D=3D = 'follow'\ >> + end,\ >> + 10) >> + >> +-- Cleanup. >> +test_run:cmd('stop server replica') >> +test_run:cmd('delete server replica') >> +box.space.test:drop() >> +box.schema.user.revoke('guest', 'replication') >> diff --git a/test/replication/local_spaces.result = b/test/replication/local_spaces.result >> index cf2c52010..4855d8a88 100644 >> --- a/test/replication/local_spaces.result >> +++ b/test/replication/local_spaces.result >> @@ -288,6 +288,10 @@ _ =3D s3:insert{3} >> vclock =3D test_run:get_vclock('default') >> --- >> ... >> +-- Ignore 0-th component when waiting. They don't match. >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock('replica', vclock) >> --- >> ... >> diff --git a/test/replication/local_spaces.test.lua = b/test/replication/local_spaces.test.lua >> index 373e2cd20..c5e224030 100644 >> --- a/test/replication/local_spaces.test.lua >> +++ b/test/replication/local_spaces.test.lua >> @@ -112,6 +112,9 @@ _ =3D s1:insert{3} >> _ =3D s2:insert{3} >> _ =3D s3:insert{3} >> vclock =3D test_run:get_vclock('default') >> + >> +-- Ignore 0-th component when waiting. They don't match. >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock('replica', vclock) >>=20 >> test_run:cmd("switch replica") >> diff --git a/test/replication/misc.result = b/test/replication/misc.result >> index b63d72846..e5d1f560e 100644 >> --- a/test/replication/misc.result >> +++ b/test/replication/misc.result >> @@ -214,6 +214,9 @@ box.space.space1:select{} >> vclock =3D test_run:get_vclock("autobootstrap1") >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> --- >> ... >> @@ -414,6 +417,9 @@ while box.info.replication[2] =3D=3D nil do = fiber.sleep(0.01) end >> vclock =3D test_run:get_vclock('default') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock('replica_auth', vclock) >> --- >> ... >> diff --git a/test/replication/misc.test.lua = b/test/replication/misc.test.lua >> index c454a0992..d285b014a 100644 >> --- a/test/replication/misc.test.lua >> +++ b/test/replication/misc.test.lua >> @@ -88,6 +88,7 @@ c.space.space1:insert{box.NULL, "data"} -- fails, = but bumps sequence value >> c.space.space1:insert{box.NULL, 1, "data"} >> box.space.space1:select{} >> vclock =3D test_run:get_vclock("autobootstrap1") >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("autobootstrap2", vclock) >> test_run:cmd("switch autobootstrap2") >> box.space.space1:select{} >> @@ -172,6 +173,7 @@ box.schema.user.grant('cluster', 'replication') >>=20 >> while box.info.replication[2] =3D=3D nil do fiber.sleep(0.01) end >> vclock =3D test_run:get_vclock('default') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock('replica_auth', vclock) >>=20 >> test_run:cmd("stop server replica_auth") >> diff --git a/test/replication/quorum.result = b/test/replication/quorum.result >> index 07abe7f2a..5ef66bf0a 100644 >> --- a/test/replication/quorum.result >> +++ b/test/replication/quorum.result >> @@ -325,6 +325,9 @@ space:insert{2} >> vclock =3D test_run:get_vclock("default") >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("replica", vclock) >> --- >> ... >> @@ -390,6 +393,9 @@ box.cfg{replication =3D repl} >> vclock =3D test_run:get_vclock("master_quorum1") >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("master_quorum2", vclock) >> --- >> ... >> diff --git a/test/replication/quorum.test.lua = b/test/replication/quorum.test.lua >> index 5f2872675..da24f34a0 100644 >> --- a/test/replication/quorum.test.lua >> +++ b/test/replication/quorum.test.lua >> @@ -125,6 +125,7 @@ test_run:cmd("switch default") >> box.cfg{listen =3D listen} >> space:insert{2} >> vclock =3D test_run:get_vclock("default") >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("replica", vclock) >> test_run:cmd("switch replica") >> box.info.status -- running >> @@ -145,6 +146,7 @@ box.cfg{replication =3D ""} >> box.space.test:insert{1} >> box.cfg{replication =3D repl} >> vclock =3D test_run:get_vclock("master_quorum1") >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("master_quorum2", vclock) >> test_run:cmd("switch master_quorum2") >> box.space.test:select() >> diff --git a/test/replication/replica_rejoin.result = b/test/replication/replica_rejoin.result >> index b8ed79f14..dd04ae297 100644 >> --- a/test/replication/replica_rejoin.result >> +++ b/test/replication/replica_rejoin.result >> @@ -144,6 +144,9 @@ for i =3D 10, 30, 10 do box.space.test:update(i, = {{'!', 1, i}}) end >> vclock =3D test_run:get_vclock('default') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock('replica', vclock) >> --- >> ... >> @@ -295,6 +298,9 @@ for i =3D 1, 10 do box.space.test:replace{2} end >> vclock =3D test_run:get_vclock('replica') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock('default', vclock) >> --- >> ... >> @@ -345,6 +351,9 @@ for i =3D 1, 10 do box.space.test:replace{2} end >> vclock =3D test_run:get_vclock('replica') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock('default', vclock) >> --- >> ... >> diff --git a/test/replication/replica_rejoin.test.lua = b/test/replication/replica_rejoin.test.lua >> index 25c0849ec..410ef44d7 100644 >> --- a/test/replication/replica_rejoin.test.lua >> +++ b/test/replication/replica_rejoin.test.lua >> @@ -55,6 +55,7 @@ test_run:cmd("switch default") >> -- Make sure the replica follows new changes. >> for i =3D 10, 30, 10 do box.space.test:update(i, {{'!', 1, i}}) end >> vclock =3D test_run:get_vclock('default') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock('replica', vclock) >> test_run:cmd("switch replica") >> box.space.test:select() >> @@ -109,6 +110,7 @@ box.space.test:replace{1} >> test_run:cmd("switch replica") >> for i =3D 1, 10 do box.space.test:replace{2} end >> vclock =3D test_run:get_vclock('replica') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock('default', vclock) >> -- Restart the master and force garbage collection. >> test_run:cmd("switch default") >> @@ -126,6 +128,7 @@ test_run:wait_cond(function() return = #fio.glob(fio.pathjoin(box.cfg.wal_dir, '*. >> test_run:cmd("switch replica") >> for i =3D 1, 10 do box.space.test:replace{2} end >> vclock =3D test_run:get_vclock('replica') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock('default', vclock) >> -- Restart the replica. It should successfully rebootstrap. >> test_run:cmd("restart server replica with args=3D'true'") >> diff --git a/test/replication/skip_conflict_row.result = b/test/replication/skip_conflict_row.result >> index 9b2777872..d70ac8e2a 100644 >> --- a/test/replication/skip_conflict_row.result >> +++ b/test/replication/skip_conflict_row.result >> @@ -54,6 +54,9 @@ box.info.status >> vclock =3D test_run:get_vclock('default') >> --- >> ... >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("replica", vclock) >> --- >> ... >> diff --git a/test/replication/skip_conflict_row.test.lua = b/test/replication/skip_conflict_row.test.lua >> index 2982c730a..04fd08136 100644 >> --- a/test/replication/skip_conflict_row.test.lua >> +++ b/test/replication/skip_conflict_row.test.lua >> @@ -19,6 +19,7 @@ space:insert{2} >> box.info.status >>=20 >> vclock =3D test_run:get_vclock('default') >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("replica", vclock) >> test_run:cmd("switch replica") >> box.info.replication[1].upstream.message >> diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg >> index 90fd53ca6..0907ac17f 100644 >> --- a/test/replication/suite.cfg >> +++ b/test/replication/suite.cfg >> @@ -12,6 +12,7 @@ >> "on_schema_init.test.lua": {}, >> "long_row_timeout.test.lua": {}, >> "join_without_snap.test.lua": {}, >> + "gh-4114-local-space-replication.test.lua": {}, >> "gh-4402-info-errno.test.lua": {}, >> "gh-4605-empty-password.test.lua": {}, >> "gh-4606-admin-creds.test.lua": {}, >> diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result >> index 2635da265..2bd701f70 100644 >> --- a/test/vinyl/errinj.result >> +++ b/test/vinyl/errinj.result >> @@ -1241,6 +1241,11 @@ test_run:cmd("switch default") >> vclock =3D test_run:get_vclock("default") >> --- >> ... >> +-- Ignore 0-th vclock component. They don't match between >> +-- replicas. >> +vclock[0] =3D nil >> +--- >> +... >> _ =3D test_run:wait_vclock("replica", vclock) >> --- >> ... >> diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua >> index 4230cfae3..750d3bfe8 100644 >> --- a/test/vinyl/errinj.test.lua >> +++ b/test/vinyl/errinj.test.lua >> @@ -455,6 +455,10 @@ box.cfg{read_only =3D true} >>=20 >> test_run:cmd("switch default") >> vclock =3D test_run:get_vclock("default") >> + >> +-- Ignore 0-th vclock component. They don't match between >> +-- replicas. >> +vclock[0] =3D nil >> _ =3D test_run:wait_vclock("replica", vclock) >>=20 >> test_run:cmd("stop server replica") >> --=20 >> 2.21.1 (Apple Git-122.3) >>=20 >=20 > --=20 > Konstantin Osipov, Moscow, Russia > https://scylladb.com -- Serge Petrenko sergepetrenko@tarantool.org