[Tarantool-patches] [PATCH v1] test: flaky replication/gh-4402-info-errno.test.lua

[Tarantool-patches] [PATCH v1] test: flaky replication/gh-4402-info-errno.test.lua

[Alexander V. Tikhonov]

On heavy loaded hosts found the following issue:

  [004] --- replication/gh-4402-info-errno.result	Wed Jul 22 06:13:34 2020
  [004] +++ replication/gh-4402-info-errno.reject	Wed Jul 22 06:41:14 2020
  [004] @@ -32,7 +32,39 @@
  [004]   | ...
  [004]  d ~= nil and d.status == 'follow' or i
  [004]   | ---
  [004] - | - true
  [004] + | - version: 2.6.0-10-g8df49e4
  [004] + |   id: 1
  [004] + |   ro: false
  [004] + |   uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
  [004] + |   package: Tarantool
  [004] + |   cluster:
  [004] + |     uuid: 6ec7bcce-68e7-41a4-b84b-dc9236621579
  [004] + |   listen: unix/:(socket)
  [004] + |   replication_anon:
  [004] + |     count: 0
  [004] + |   replication:
  [004] + |     1:
  [004] + |       id: 1
  [004] + |       uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
  [004] + |       lsn: 52
  [004] + |     2:
  [004] + |       id: 2
  [004] + |       uuid: 8a989231-177a-4eb8-8030-c148bc752b0e
  [004] + |       lsn: 0
  [004] + |       downstream:
  [004] + |         status: stopped
  [004] + |         message: timed out
  [004] + |         system_message: Connection timed out
  [004] + |   signature: 52
  [004] + |   status: running
  [004] + |   vclock: {1: 52}
  [004] + |   uptime: 27
  [004] + |   lsn: 52
  [004] + |   sql: []
  [004] + |   gc: []
  [004] + |   vinyl: []
  [004] + |   memory: []
  [004] + |   pid: 99
  [004]   | ...
  [004]
  [004]  test_run:cmd('stop server replica')

It happened because replication downstream status check occurred too
early, when it was only in 'stopped' state. To give the replication
status check routine ability to reach the needed 'follow' state, it
need to wait for it using test_run:wait_downstream() routine.

Closes #5235
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5235-fix-4402
Issue: https://github.com/tarantool/tarantool/issues/5235

 test/replication/gh-4402-info-errno.result   | 5 +----
 test/replication/gh-4402-info-errno.test.lua | 3 +--
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/test/replication/gh-4402-info-errno.result b/test/replication/gh-4402-info-errno.result
index 661eea38b..9c6b352a4 100644
--- a/test/replication/gh-4402-info-errno.result
+++ b/test/replication/gh-4402-info-errno.result
@@ -27,10 +27,7 @@ i = box.info
 replica_id = i.id % 2 + 1
  | ---
  | ...
-d = i.replication[replica_id].downstream
- | ---
- | ...
-d ~= nil and d.status == 'follow' or i
+test_run:wait_downstream(replica_id, {status = 'follow'}) or i
  | ---
  | - true
  | ...
diff --git a/test/replication/gh-4402-info-errno.test.lua b/test/replication/gh-4402-info-errno.test.lua
index 1b2d9d814..8f72e7f72 100644
--- a/test/replication/gh-4402-info-errno.test.lua
+++ b/test/replication/gh-4402-info-errno.test.lua
@@ -12,8 +12,7 @@ test_run:cmd('create server replica with rpl_master=default, script="replication
 test_run:cmd('start server replica')
 i = box.info
 replica_id = i.id % 2 + 1
-d = i.replication[replica_id].downstream
-d ~= nil and d.status == 'follow' or i
+test_run:wait_downstream(replica_id, {status = 'follow'}) or i
 
 test_run:cmd('stop server replica')
 test_run:cmd("cleanup server replica")
-- 
2.17.1

[Tarantool-patches] [PATCH v1] test: flaky replication/gh-4928-tx-boundaries test

[Alexander V. Tikhonov]

On heavy loaded hosts found the following issue:

  [089] --- replication/gh-4928-tx-boundaries.result	Wed Jul 29 04:08:29 2020
  [089] +++ replication/gh-4928-tx-boundaries.reject	Wed Jul 29 04:24:02 2020
  [089] @@ -94,7 +94,7 @@
  [089]   | ...
  [089]  box.info.replication[1].upstream.status
  [089]   | ---
  [089] - | - follow
  [089] + | - disconnected
  [089]   | ...
  [089]
  [089]  box.space.glob:select{}

It happened because replication upstream status check occurred too
early, when it was only in 'disconnected' state. To give the
replication status check routine ability to reach the needed 'follow'
state, it need to wait for it using test_run:wait_upstream() routine.

Closes #5234
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5234-fix-4928
Issue: https://github.com/tarantool/tarantool/issues/5234

 test/replication/gh-4928-tx-boundaries.result   | 4 ++--
 test/replication/gh-4928-tx-boundaries.test.lua | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/test/replication/gh-4928-tx-boundaries.result b/test/replication/gh-4928-tx-boundaries.result
index 969bd8438..4bd1b4b25 100644
--- a/test/replication/gh-4928-tx-boundaries.result
+++ b/test/replication/gh-4928-tx-boundaries.result
@@ -92,9 +92,9 @@ box.info.status
  | ---
  | - running
  | ...
-box.info.replication[1].upstream.status
+test_run:wait_upstream(1, {status = 'follow'})
  | ---
- | - follow
+ | - true
  | ...
 
 box.space.glob:select{}
diff --git a/test/replication/gh-4928-tx-boundaries.test.lua b/test/replication/gh-4928-tx-boundaries.test.lua
index 92526fc51..f245bff74 100644
--- a/test/replication/gh-4928-tx-boundaries.test.lua
+++ b/test/replication/gh-4928-tx-boundaries.test.lua
@@ -47,7 +47,7 @@ test_run:wait_vclock("replica", vclock)
 test_run:cmd('switch replica')
 
 box.info.status
-box.info.replication[1].upstream.status
+test_run:wait_upstream(1, {status = 'follow'})
 
 box.space.glob:select{}
 
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v1] test: flaky replication/gh-4402-info-errno.test.lua

[Serge Petrenko]

06.09.2020 18:36, Alexander V. Tikhonov пишет:
> On heavy loaded hosts found the following issue:
>
>    [004] --- replication/gh-4402-info-errno.result	Wed Jul 22 06:13:34 2020
>    [004] +++ replication/gh-4402-info-errno.reject	Wed Jul 22 06:41:14 2020
>    [004] @@ -32,7 +32,39 @@
>    [004]   | ...
>    [004]  d ~= nil and d.status == 'follow' or i
>    [004]   | ---
>    [004] - | - true
>    [004] + | - version: 2.6.0-10-g8df49e4
>    [004] + |   id: 1
>    [004] + |   ro: false
>    [004] + |   uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
>    [004] + |   package: Tarantool
>    [004] + |   cluster:
>    [004] + |     uuid: 6ec7bcce-68e7-41a4-b84b-dc9236621579
>    [004] + |   listen: unix/:(socket)
>    [004] + |   replication_anon:
>    [004] + |     count: 0
>    [004] + |   replication:
>    [004] + |     1:
>    [004] + |       id: 1
>    [004] + |       uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
>    [004] + |       lsn: 52
>    [004] + |     2:
>    [004] + |       id: 2
>    [004] + |       uuid: 8a989231-177a-4eb8-8030-c148bc752b0e
>    [004] + |       lsn: 0
>    [004] + |       downstream:
>    [004] + |         status: stopped
>    [004] + |         message: timed out
>    [004] + |         system_message: Connection timed out
>    [004] + |   signature: 52
>    [004] + |   status: running
>    [004] + |   vclock: {1: 52}
>    [004] + |   uptime: 27
>    [004] + |   lsn: 52
>    [004] + |   sql: []
>    [004] + |   gc: []
>    [004] + |   vinyl: []
>    [004] + |   memory: []
>    [004] + |   pid: 99
>    [004]   | ...
>    [004]
>    [004]  test_run:cmd('stop server replica')
>
> It happened because replication downstream status check occurred too
> early, when it was only in 'stopped' state. To give the replication
> status check routine ability to reach the needed 'follow' state, it
> need to wait for it using test_run:wait_downstream() routine.
>
> Closes #5235
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5235-fix-4402
> Issue: https://github.com/tarantool/tarantool/issues/5235
>
>   test/replication/gh-4402-info-errno.result   | 5 +----
>   test/replication/gh-4402-info-errno.test.lua | 3 +--
>   2 files changed, 2 insertions(+), 6 deletions(-)
>
> diff --git a/test/replication/gh-4402-info-errno.result b/test/replication/gh-4402-info-errno.result
> index 661eea38b..9c6b352a4 100644
> --- a/test/replication/gh-4402-info-errno.result
> +++ b/test/replication/gh-4402-info-errno.result
> @@ -27,10 +27,7 @@ i = box.info
>   replica_id = i.id % 2 + 1
>    | ---
>    | ...
> -d = i.replication[replica_id].downstream
> - | ---
> - | ...
> -d ~= nil and d.status == 'follow' or i
> +test_run:wait_downstream(replica_id, {status = 'follow'}) or i
>    | ---
>    | - true
>    | ...
> diff --git a/test/replication/gh-4402-info-errno.test.lua b/test/replication/gh-4402-info-errno.test.lua
> index 1b2d9d814..8f72e7f72 100644
> --- a/test/replication/gh-4402-info-errno.test.lua
> +++ b/test/replication/gh-4402-info-errno.test.lua
> @@ -12,8 +12,7 @@ test_run:cmd('create server replica with rpl_master=default, script="replication
>   test_run:cmd('start server replica')
>   i = box.info
>   replica_id = i.id % 2 + 1
> -d = i.replication[replica_id].downstream
> -d ~= nil and d.status == 'follow' or i
> +test_run:wait_downstream(replica_id, {status = 'follow'}) or i
>   
>   test_run:cmd('stop server replica')
>   test_run:cmd("cleanup server replica")
Thanks for the patch! LGTM.

-- 
Serge Petrenko

Re: [Tarantool-patches] [PATCH v1] test: flaky replication/gh-4402-info-errno.test.lua

[Kirill Yukhin]

Hello,

On 06 сен 18:36, Alexander V. Tikhonov wrote:
> On heavy loaded hosts found the following issue:
> 
>   [004] --- replication/gh-4402-info-errno.result	Wed Jul 22 06:13:34 2020
>   [004] +++ replication/gh-4402-info-errno.reject	Wed Jul 22 06:41:14 2020
>   [004] @@ -32,7 +32,39 @@
>   [004]   | ...
>   [004]  d ~= nil and d.status == 'follow' or i
>   [004]   | ---
>   [004] - | - true
>   [004] + | - version: 2.6.0-10-g8df49e4
>   [004] + |   id: 1
>   [004] + |   ro: false
>   [004] + |   uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
>   [004] + |   package: Tarantool
>   [004] + |   cluster:
>   [004] + |     uuid: 6ec7bcce-68e7-41a4-b84b-dc9236621579
>   [004] + |   listen: unix/:(socket)
>   [004] + |   replication_anon:
>   [004] + |     count: 0
>   [004] + |   replication:
>   [004] + |     1:
>   [004] + |       id: 1
>   [004] + |       uuid: 41c4e3bf-cc3b-443d-88c9-39a9a8fe2df9
>   [004] + |       lsn: 52
>   [004] + |     2:
>   [004] + |       id: 2
>   [004] + |       uuid: 8a989231-177a-4eb8-8030-c148bc752b0e
>   [004] + |       lsn: 0
>   [004] + |       downstream:
>   [004] + |         status: stopped
>   [004] + |         message: timed out
>   [004] + |         system_message: Connection timed out
>   [004] + |   signature: 52
>   [004] + |   status: running
>   [004] + |   vclock: {1: 52}
>   [004] + |   uptime: 27
>   [004] + |   lsn: 52
>   [004] + |   sql: []
>   [004] + |   gc: []
>   [004] + |   vinyl: []
>   [004] + |   memory: []
>   [004] + |   pid: 99
>   [004]   | ...
>   [004]
>   [004]  test_run:cmd('stop server replica')
> 
> It happened because replication downstream status check occurred too
> early, when it was only in 'stopped' state. To give the replication
> status check routine ability to reach the needed 'follow' state, it
> need to wait for it using test_run:wait_downstream() routine.
> 
> Closes #5235
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5235-fix-4402
> Issue: https://github.com/tarantool/tarantool/issues/5235

I've checked your patch into 2.4, 2.5 and master.

--
Regards, Kirill Yukhin