From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp60.i.mail.ru (smtp60.i.mail.ru [217.69.128.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CA4D3442BAC for ; Mon, 18 Nov 2019 16:19:52 +0300 (MSK) From: Ilya Kosarev Date: Mon, 18 Nov 2019 16:19:43 +0300 Message-Id: <4f268418f5c12eb73cd5266fb4b5a7aa60f13f82.1574082067.git.i.kosarev@tarantool.org> In-Reply-To: References: In-Reply-To: References: Subject: [Tarantool-patches] [PATCH 2/2] test: stabilize quorum test conditions List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tarantool-patches@dev.tarantool.org There were some pass conditions in quorum test which could take some time to be satisfied. Now they are wrapped using test_run:wait_cond to make the test stable. Closes #4586 --- test/replication/quorum.result | 28 +++++++++++++++++++--------- test/replication/quorum.test.lua | 16 +++++++++------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/test/replication/quorum.result b/test/replication/quorum.result index ff5fa0150..73def113f 100644 --- a/test/replication/quorum.result +++ b/test/replication/quorum.result @@ -115,15 +115,15 @@ box.info.status -- running - running ... -- Check that the replica follows all masters. -box.info.id == 1 or box.info.replication[1].upstream.status == 'follow' +box.info.id == 1 or test_run:wait_cond(function() return box.info.replication[1].upstream.status == 'follow' end, 20) --- - true ... -box.info.id == 2 or box.info.replication[2].upstream.status == 'follow' +box.info.id == 2 or test_run:wait_cond(function() return box.info.replication[2].upstream.status == 'follow' end, 20) --- - true ... -box.info.id == 3 or box.info.replication[3].upstream.status == 'follow' +box.info.id == 3 or test_run:wait_cond(function() return box.info.replication[3].upstream.status == 'follow' end, 20) --- - true ... @@ -149,6 +149,10 @@ test_run:cmd('stop server quorum1') --- - true ... +test_run:wait_cond(function() return box.space.test.index.primary ~= nil end, 20) +--- +- true +... for i = 1, 100 do box.space.test:insert{i} end --- ... @@ -166,9 +170,9 @@ test_run:cmd('switch quorum1') --- - true ... -box.space.test:count() -- 100 +test_run:wait_cond(function() return box.space.test:count() == 100 end, 20) --- -- 100 +- true ... -- Rebootstrap one node of the cluster and check that others follow. -- Note, due to ERRINJ_RELAY_TIMEOUT there is a substantial delay @@ -197,9 +201,9 @@ test_run:cmd('switch quorum1') - true ... test_run:cmd('restart server quorum1 with cleanup=1, args="0.1 0.5"') -box.space.test:count() -- 100 +test_run:wait_cond(function() return box.space.test:count() == 100 end, 20) --- -- 100 +- true ... -- The rebootstrapped replica will be assigned id = 4, -- because ids 1..3 are busy. @@ -210,8 +214,9 @@ test_run:cmd('switch quorum2') fiber = require('fiber') --- ... -while box.info.replication[4].upstream.status ~= 'follow' do fiber.sleep(0.001) end +test_run:wait_cond(function() return box.info.replication[4].upstream.status == 'follow' end, 20) --- +- true ... box.info.replication[4].upstream.status --- @@ -224,8 +229,13 @@ test_run:cmd('switch quorum3') fiber = require('fiber') --- ... -while box.info.replication[4].upstream.status ~= 'follow' do fiber.sleep(0.001) end +test_run:wait_cond(function() return box.info.replication ~= nil end, 20) --- +- true +... +test_run:wait_cond(function() return box.info.replication[4].upstream.status == 'follow' end, 20) +--- +- true ... box.info.replication[4].upstream.status --- diff --git a/test/replication/quorum.test.lua b/test/replication/quorum.test.lua index 98febb367..1842cdffe 100644 --- a/test/replication/quorum.test.lua +++ b/test/replication/quorum.test.lua @@ -47,9 +47,9 @@ box.info.ro -- false box.info.status -- running -- Check that the replica follows all masters. -box.info.id == 1 or box.info.replication[1].upstream.status == 'follow' -box.info.id == 2 or box.info.replication[2].upstream.status == 'follow' -box.info.id == 3 or box.info.replication[3].upstream.status == 'follow' +box.info.id == 1 or test_run:wait_cond(function() return box.info.replication[1].upstream.status == 'follow' end, 20) +box.info.id == 2 or test_run:wait_cond(function() return box.info.replication[2].upstream.status == 'follow' end, 20) +box.info.id == 3 or test_run:wait_cond(function() return box.info.replication[3].upstream.status == 'follow' end, 20) -- Check that box.cfg() doesn't return until the instance -- catches up with all configured replicas. @@ -59,13 +59,14 @@ test_run:cmd('switch quorum2') box.error.injection.set("ERRINJ_RELAY_TIMEOUT", 0.001) test_run:cmd('stop server quorum1') +test_run:wait_cond(function() return box.space.test.index.primary ~= nil end, 20) for i = 1, 100 do box.space.test:insert{i} end fiber = require('fiber') fiber.sleep(0.1) test_run:cmd('start server quorum1 with args="0.1 0.5"') test_run:cmd('switch quorum1') -box.space.test:count() -- 100 +test_run:wait_cond(function() return box.space.test:count() == 100 end, 20) -- Rebootstrap one node of the cluster and check that others follow. -- Note, due to ERRINJ_RELAY_TIMEOUT there is a substantial delay @@ -81,17 +82,18 @@ box.snapshot() test_run:cmd('switch quorum1') test_run:cmd('restart server quorum1 with cleanup=1, args="0.1 0.5"') -box.space.test:count() -- 100 +test_run:wait_cond(function() return box.space.test:count() == 100 end, 20) -- The rebootstrapped replica will be assigned id = 4, -- because ids 1..3 are busy. test_run:cmd('switch quorum2') fiber = require('fiber') -while box.info.replication[4].upstream.status ~= 'follow' do fiber.sleep(0.001) end +test_run:wait_cond(function() return box.info.replication[4].upstream.status == 'follow' end, 20) box.info.replication[4].upstream.status test_run:cmd('switch quorum3') fiber = require('fiber') -while box.info.replication[4].upstream.status ~= 'follow' do fiber.sleep(0.001) end +test_run:wait_cond(function() return box.info.replication ~= nil end, 20) +test_run:wait_cond(function() return box.info.replication[4].upstream.status == 'follow' end, 20) box.info.replication[4].upstream.status -- Cleanup. -- 2.17.1