From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id CAFD66F3C7; Fri, 26 Mar 2021 16:37:28 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CAFD66F3C7 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1616765848; bh=mUaBN9HD6o0MucHbBR+dW7WO2HeOoE/5h7EwogRMavE=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Pnjg7/fRNbVQFQD65ljl9Dq8go0fFUZkIWurHeYg+Uf8VvusU94FkXOo5oxclgzHS Lf7lPek+GVXHJ0lFKxK/w2B0S5L8kuS8LcyyGXOsjAmQXfNp90luTbsANSKPgyV5cI e5DRzeyjq1ndwzLcVaqJD92ezTCwE6ICmy6uhF4s= Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 9D0646F3C7 for ; Fri, 26 Mar 2021 16:37:27 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 9D0646F3C7 Received: by smtp37.i.mail.ru with esmtpa (envelope-from ) id 1lPmeU-0002XV-Ao; Fri, 26 Mar 2021 16:37:26 +0300 To: Cyrill Gorcunov , tml References: <20210326120605.2160131-1-gorcunov@gmail.com> <20210326120605.2160131-3-gorcunov@gmail.com> Message-ID: <7d84bf90-7306-ad25-9f0f-74867c100fad@tarantool.org> Date: Fri, 26 Mar 2021 16:37:26 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <20210326120605.2160131-3-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9064ADF4728AA0EE9587C800CEAD38A69F042767B24605C2B182A05F5380850404D0DA5DD6F318C06F56E9AB33414971BCB996803C14F7A9CFB30D4F167E76894 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7257031E595304076EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637593B4F2A76947A308638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C5DD32608FC869F5D97BD653902EBB0CAEC9500E65E9DADFFA471835C12D1D9774AD6D5ED66289B5259CC434672EE6371117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658359CC434672EE6371117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CF228E1099350BD182BA3038C0950A5D36C8A9BA7A39EFB766EC990983EF5C0329BA3038C0950A5D36D5E8D9A59859A8B6D6FBC3EC642A93BB3AA81AA40904B5D9DBF02ECDB25306B2201CA6A4E26CD07C3BBE47FD9DD3FB595F5C1EE8F4F765FCA83251EDC214901ED5E8D9A59859A8B6AADDF26287449870089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D56EA283C85FEEC7D355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A5B98CE3D0BBBC015D0F39BD4F01F7725B4A788EA44BCAE90AD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A150564C13026C18CEA6FBEE345E03226055CFE775DC5C82A79204BBBB10FDADE0A0BB7B502728201D7E09C32AA3244C0E32647D842E531AC8A285946B653572C3B3ADDA61883BB5927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojapPp7P/VpAhNRk7FPnbtRA== X-Mailru-Sender: 583F1D7ACE8F49BDD2846D59FC20E9F84BC29D49F6191AA0304C48846603FA4E9580EC9F85D42BB5424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v5 2/3] test: add a test for wal_cleanup_delay option X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 26.03.2021 15:06, Cyrill Gorcunov пишет: > Part-of #5806 > > Signed-off-by: Cyrill Gorcunov > --- > test/replication/gh-5806-master.lua | 8 + > test/replication/gh-5806-slave.lua | 8 + > test/replication/gh-5806-xlog-cleanup.result | 435 ++++++++++++++++++ > .../replication/gh-5806-xlog-cleanup.test.lua | 188 ++++++++ > 4 files changed, 639 insertions(+) > create mode 100644 test/replication/gh-5806-master.lua > create mode 100644 test/replication/gh-5806-slave.lua > create mode 100644 test/replication/gh-5806-xlog-cleanup.result > create mode 100644 test/replication/gh-5806-xlog-cleanup.test.lua > > diff --git a/test/replication/gh-5806-master.lua b/test/replication/gh-5806-master.lua > new file mode 100644 > index 000000000..bc15dab67 > --- /dev/null > +++ b/test/replication/gh-5806-master.lua > @@ -0,0 +1,8 @@ > +#!/usr/bin/env tarantool > + > +require('console').listen(os.getenv('ADMIN')) > + > +box.cfg({ > + listen = os.getenv("LISTEN"), > + wal_cleanup_delay = tonumber(arg[1]) or 0, > +}) > diff --git a/test/replication/gh-5806-slave.lua b/test/replication/gh-5806-slave.lua > new file mode 100644 > index 000000000..3abb3e035 > --- /dev/null > +++ b/test/replication/gh-5806-slave.lua > @@ -0,0 +1,8 @@ > +#!/usr/bin/env tarantool > + > +require('console').listen(os.getenv('ADMIN')) > + > +box.cfg({ > + listen = os.getenv("LISTEN"), > + replication = os.getenv("MASTER"), > +}) Hi! Thanks for the fixes! You may  use `replica.lua` here freely. It takes the same parameters, and has the same box.cfg() call, with exception that it has set `replication_timeout` and `memtx_memory`. These two options are not a problem. In fact, `replica.lua` is used in almost all replication tests. So, there's no reason to spawn a new file for your needs. > diff --git a/test/replication/gh-5806-xlog-cleanup.result b/test/replication/gh-5806-xlog-cleanup.result > new file mode 100644 > index 000000000..e20784bcc > --- /dev/null > +++ b/test/replication/gh-5806-xlog-cleanup.result > @@ -0,0 +1,435 @@ > +-- test-run result file version 2 > +-- > +-- gh-5806: defer xlog cleanup to keep xlogs until > +-- replicas present in "_cluster" are connected. > +-- Otherwise we are getting XlogGapError since > +-- master might go far forward from replica and > +-- replica won't be able to connect without full > +-- rebootstrap. > +-- > + > +fiber = require('fiber') > + | --- > + | ... > +test_run = require('test_run').new() > + | --- > + | ... > +engine = test_run:get_cfg('engine') > + | --- > + | ... > + > +-- > +-- Case 1. > +-- > +-- First lets make sure we're getting XlogGapError in > +-- case if wal_cleanup_delay is not used. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +-- > +-- Keep small number of snaps to force cleanup > +-- procedure be more intensive. > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > + > +engine = test_run:get_cfg('engine') > + | --- > + | ... > +s = box.schema.space.create('test', {engine = engine}) > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +-- > +-- On replica we create an own space which allows us to > +-- use more complex scenario and disables replica from > +-- automatic rejoin (since replica can't do auto-rejoin if > +-- there gonna be an own data loss). This allows us to > +-- trigger XlogGapError in the log. > +test_run:switch('replica') > + | --- > + | - true > + | ... > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > +s = box.schema.space.create('testreplica') > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > +box.space.testreplica:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +-- > +-- Stop the replica node and generate > +-- xlogs on the master. > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > + > +box.space.test:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +-- > +-- We need to restart the master node since otherwise > +-- the replica will be preventing us from removing old > +-- xlog because it will be tracked by gc consumer which > +-- kept in memory while master node is running. > +-- > +-- Once restarted we write a new record into master's > +-- space and run snapshot which removes old xlog required > +-- by replica to subscribe leading to XlogGapError which > +-- we need to test. > +test_run:cmd('restart server master with wait_load=True') > + | > +box.space.test:insert({2}) > + | --- > + | - [2] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > +assert(box.info.gc().is_paused == false) > + | --- > + | - true > + | ... > + > +-- > +-- Start replica and wait for error. > +test_run:cmd('start server replica with wait=False, wait_load=False') > + | --- > + | - true > + | ... > + > +-- > +-- Wait error to appear, 60 seconds should be more than enough, > +-- usually it happens in a couple of seconds. > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil > + | --- > + | - true > + | ... > + > +-- > +-- Cleanup. > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > + > +-- > +-- Case 2. > +-- > +-- Lets make sure we're not getting XlogGapError in > +-- case if wal_cleanup_delay is used the code is almost > +-- the same as for Case 1 except we don't disable cleanup > +-- fiber but delay it up to a hour until replica is up > +-- and running. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > + > +engine = test_run:get_cfg('engine') > + | --- > + | ... > +s = box.schema.space.create('test', {engine = engine}) > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('replica') > + | --- > + | - true > + | ... > +box.cfg{checkpoint_count = 1} > + | --- > + | ... > +s = box.schema.space.create('testreplica') > + | --- > + | ... > +_ = s:create_index('pk') > + | --- > + | ... > +box.space.testreplica:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > + > +box.space.test:insert({1}) > + | --- > + | - [1] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > + | > +box.space.test:insert({2}) > + | --- > + | - [2] > + | ... > +box.snapshot() > + | --- > + | - ok > + | ... > +assert(box.info.gc().is_paused == true) > + | --- > + | - true > + | ... > + > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +-- > +-- Make sure no error happened. > +test_run:switch('default') > + | --- > + | - true > + | ... > +assert(test_run:grep_log("master", "XlogGapError") == nil) > + | --- > + | - true > + | ... > + > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > +-- > +-- > +-- Case 3: Fill _cluster with replica but then delete > +-- the replica so that master's cleanup leave in "paused" > +-- state, and then simply decrease the timeout to make > +-- cleanup fiber work again. > +-- > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > + | > +assert(box.info.gc().is_paused == true) > + | --- > + | - true > + | ... > + > +test_run:switch('master') > + | --- > + | - true > + | ... > +box.cfg{wal_cleanup_delay = 0.01} > + | --- > + | ... > +test_run:wait_cond(function() return box.info.gc().is_paused == false end) > + | --- > + | - true > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server master') > + | --- > + | - true > + | ... > +test_run:cmd('cleanup server master') > + | --- > + | - true > + | ... > +test_run:cmd('delete server master') > + | --- > + | - true > + | ... > diff --git a/test/replication/gh-5806-xlog-cleanup.test.lua b/test/replication/gh-5806-xlog-cleanup.test.lua > new file mode 100644 > index 000000000..ea3a35294 > --- /dev/null > +++ b/test/replication/gh-5806-xlog-cleanup.test.lua > @@ -0,0 +1,188 @@ > +-- > +-- gh-5806: defer xlog cleanup to keep xlogs until > +-- replicas present in "_cluster" are connected. > +-- Otherwise we are getting XlogGapError since > +-- master might go far forward from replica and > +-- replica won't be able to connect without full > +-- rebootstrap. > +-- > + > +fiber = require('fiber') > +test_run = require('test_run').new() > +engine = test_run:get_cfg('engine') > + > +-- > +-- Case 1. > +-- > +-- First lets make sure we're getting XlogGapError in > +-- case if wal_cleanup_delay is not used. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +-- > +-- Keep small number of snaps to force cleanup > +-- procedure be more intensive. > +box.cfg{checkpoint_count = 1} > + > +engine = test_run:get_cfg('engine') > +s = box.schema.space.create('test', {engine = engine}) > +_ = s:create_index('pk') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +-- > +-- On replica we create an own space which allows us to > +-- use more complex scenario and disables replica from > +-- automatic rejoin (since replica can't do auto-rejoin if > +-- there gonna be an own data loss). This allows us to > +-- trigger XlogGapError in the log. > +test_run:switch('replica') > +box.cfg{checkpoint_count = 1} > +s = box.schema.space.create('testreplica') > +_ = s:create_index('pk') > +box.space.testreplica:insert({1}) > +box.snapshot() > + > +-- > +-- Stop the replica node and generate > +-- xlogs on the master. > +test_run:switch('master') > +test_run:cmd('stop server replica') > + > +box.space.test:insert({1}) > +box.snapshot() > + > +-- > +-- We need to restart the master node since otherwise > +-- the replica will be preventing us from removing old > +-- xlog because it will be tracked by gc consumer which > +-- kept in memory while master node is running. > +-- > +-- Once restarted we write a new record into master's > +-- space and run snapshot which removes old xlog required > +-- by replica to subscribe leading to XlogGapError which > +-- we need to test. > +test_run:cmd('restart server master with wait_load=True') > +box.space.test:insert({2}) > +box.snapshot() > +assert(box.info.gc().is_paused == false) > + > +-- > +-- Start replica and wait for error. > +test_run:cmd('start server replica with wait=False, wait_load=False') > + > +-- > +-- Wait error to appear, 60 seconds should be more than enough, > +-- usually it happens in a couple of seconds. > +test_run:switch('default') > +test_run:wait_log('master', 'XlogGapError', nil, 60) ~= nil > + > +-- > +-- Cleanup. > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > + > +-- > +-- Case 2. > +-- > +-- Lets make sure we're not getting XlogGapError in > +-- case if wal_cleanup_delay is used the code is almost > +-- the same as for Case 1 except we don't disable cleanup > +-- fiber but delay it up to a hour until replica is up > +-- and running. > +-- > + > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +box.cfg{checkpoint_count = 1} > + > +engine = test_run:get_cfg('engine') > +s = box.schema.space.create('test', {engine = engine}) > +_ = s:create_index('pk') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +test_run:switch('replica') > +box.cfg{checkpoint_count = 1} > +s = box.schema.space.create('testreplica') > +_ = s:create_index('pk') > +box.space.testreplica:insert({1}) > +box.snapshot() > + > +test_run:switch('master') > +test_run:cmd('stop server replica') > + > +box.space.test:insert({1}) > +box.snapshot() > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > +box.space.test:insert({2}) > +box.snapshot() > +assert(box.info.gc().is_paused == true) > + > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +-- > +-- Make sure no error happened. > +test_run:switch('default') > +assert(test_run:grep_log("master", "XlogGapError") == nil) > + > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > +-- > +-- > +-- Case 3: Fill _cluster with replica but then delete > +-- the replica so that master's cleanup leave in "paused" > +-- state, and then simply decrease the timeout to make > +-- cleanup fiber work again. > +-- > +test_run:cmd('create server master with script="replication/gh-5806-master.lua"') > +test_run:cmd('start server master with args="3600", wait=True, wait_load=True') > + > +test_run:switch('master') > +box.schema.user.grant('guest', 'replication') > + > +test_run:switch('default') > +test_run:cmd('create server replica with rpl_master=master,\ > + script="replication/gh-5806-slave.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +test_run:switch('master') > +test_run:cmd('stop server replica') > +test_run:cmd('cleanup server replica') > +test_run:cmd('delete server replica') > + > +test_run:cmd('restart server master with args="3600", wait=True, wait_load=True') > +assert(box.info.gc().is_paused == true) > + > +test_run:switch('master') > +box.cfg{wal_cleanup_delay = 0.01} > +test_run:wait_cond(function() return box.info.gc().is_paused == false end) > + > +test_run:switch('default') > +test_run:cmd('stop server master') > +test_run:cmd('cleanup server master') > +test_run:cmd('delete server master') -- Serge Petrenko