[Tarantool-patches] [PATCH v1] test: fragile flaky tests from parallel runs

[Tarantool-patches] [PATCH v1] test: fragile flaky tests from parallel runs

[Alexander V. Tikhonov]

Fragiled flaky tests from parallel runs to avoid
of flaky fails in regular testing:

  snapshot.test.py            ; gh-4514
  init_storage.test.py        ; gh-4949
  stat.test.lua               ; gh-4951
  xlog/checkpoint_daemon.test.lua ; gh-4952

Part of #4953
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/fragile-flaky-tests
Issue: https://github.com/tarantool/tarantool/issues/4953

 test/box-py/suite.ini         | 1 +
 test/replication-py/suite.ini | 1 +
 test/vinyl/suite.ini          | 1 +
 test/xlog/suite.ini           | 1 +
 4 files changed, 4 insertions(+)

diff --git a/test/box-py/suite.ini b/test/box-py/suite.ini
index 18737e48f..d1ff0fbfd 100644
--- a/test/box-py/suite.ini
+++ b/test/box-py/suite.ini
@@ -5,3 +5,4 @@ script = box.lua
 lua_libs = lua/fiber.lua lua/fifo.lua
 use_unix_sockets = True
 is_parallel = True
+fragile = snapshot.test.py            ; gh-4514
diff --git a/test/replication-py/suite.ini b/test/replication-py/suite.ini
index df2dd8ef0..2658f4816 100644
--- a/test/replication-py/suite.ini
+++ b/test/replication-py/suite.ini
@@ -3,3 +3,4 @@ core = tarantool
 script =  master.lua
 description = tarantool/box, replication
 is_parallel = True
+fragile = init_storage.test.py        ; gh-4949
diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
index c8bc270f3..44c53dc68 100644
--- a/test/vinyl/suite.ini
+++ b/test/vinyl/suite.ini
@@ -15,3 +15,4 @@ pretest_clean = True
 fragile = errinj.test.lua             ; gh-4346
           select_consistency.test.lua ; gh-4385
           throttle.test.lua           ; gh-4168
+          stat.test.lua               ; gh-4951
diff --git a/test/xlog/suite.ini b/test/xlog/suite.ini
index 635ec53c1..8e4c4d374 100644
--- a/test/xlog/suite.ini
+++ b/test/xlog/suite.ini
@@ -10,3 +10,4 @@ use_unix_sockets_iproto = True
 long_run = snap_io_rate.test.lua
 is_parallel = True
 pretest_clean = True
+fragile = xlog/checkpoint_daemon.test.lua ; gh-4952
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v1] test: fragile flaky tests from parallel runs

[Sergey Bronnikov]

Hi, Sasha

in general LGTM and see comment below.

On 12:49 Thu 30 Apr , Alexander V. Tikhonov wrote:
> Fragiled flaky tests from parallel runs to avoid
> of flaky fails in regular testing:

"fragiled flaky test" is a something like "butter oil"?
I propose make commit message more clear. For example:
"Mark tests as fragile in a test's configs"

>   snapshot.test.py            ; gh-4514
>   init_storage.test.py        ; gh-4949
>   stat.test.lua               ; gh-4951
>   xlog/checkpoint_daemon.test.lua ; gh-4952
> 
> Part of #4953
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/fragile-flaky-tests
> Issue: https://github.com/tarantool/tarantool/issues/4953
> 
>  test/box-py/suite.ini         | 1 +
>  test/replication-py/suite.ini | 1 +
>  test/vinyl/suite.ini          | 1 +
>  test/xlog/suite.ini           | 1 +
>  4 files changed, 4 insertions(+)
> 
> diff --git a/test/box-py/suite.ini b/test/box-py/suite.ini
> index 18737e48f..d1ff0fbfd 100644
> --- a/test/box-py/suite.ini
> +++ b/test/box-py/suite.ini
> @@ -5,3 +5,4 @@ script = box.lua
>  lua_libs = lua/fiber.lua lua/fifo.lua
>  use_unix_sockets = True
>  is_parallel = True
> +fragile = snapshot.test.py            ; gh-4514
> diff --git a/test/replication-py/suite.ini b/test/replication-py/suite.ini
> index df2dd8ef0..2658f4816 100644
> --- a/test/replication-py/suite.ini
> +++ b/test/replication-py/suite.ini
> @@ -3,3 +3,4 @@ core = tarantool
>  script =  master.lua
>  description = tarantool/box, replication
>  is_parallel = True
> +fragile = init_storage.test.py        ; gh-4949
> diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
> index c8bc270f3..44c53dc68 100644
> --- a/test/vinyl/suite.ini
> +++ b/test/vinyl/suite.ini
> @@ -15,3 +15,4 @@ pretest_clean = True
>  fragile = errinj.test.lua             ; gh-4346
>            select_consistency.test.lua ; gh-4385
>            throttle.test.lua           ; gh-4168
> +          stat.test.lua               ; gh-4951
> diff --git a/test/xlog/suite.ini b/test/xlog/suite.ini
> index 635ec53c1..8e4c4d374 100644
> --- a/test/xlog/suite.ini
> +++ b/test/xlog/suite.ini
> @@ -10,3 +10,4 @@ use_unix_sockets_iproto = True
>  long_run = snap_io_rate.test.lua
>  is_parallel = True
>  pretest_clean = True
> +fragile = xlog/checkpoint_daemon.test.lua ; gh-4952
> -- 
> 2.17.1
> 

-- 
sergeyb@

Re: [Tarantool-patches] [PATCH v1] test: fragile flaky tests from parallel runs

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 2391 bytes --]


LGTM.
  
>Четверг, 30 апреля 2020, 13:29 +03:00 от Sergey Bronnikov <sergeyb@tarantool.org>:
> 
>Hi, Sasha
>
>in general LGTM and see comment below.
>
>On 12:49 Thu 30 Apr , Alexander V. Tikhonov wrote:
>> Fragiled flaky tests from parallel runs to avoid
>> of flaky fails in regular testing:
>
>"fragiled flaky test" is a something like "butter oil"?
>I propose make commit message more clear. For example:
>"Mark tests as fragile in a test's configs"
>
>> snapshot.test.py ; gh-4514
>> init_storage.test.py ; gh-4949
>> stat.test.lua ; gh-4951
>> xlog/checkpoint_daemon.test.lua ; gh-4952
>>
>> Part of #4953
>> ---
>>
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/fragile-flaky-tests
>> Issue:  https://github.com/tarantool/tarantool/issues/4953
>>
>> test/box-py/suite.ini | 1 +
>> test/replication-py/suite.ini | 1 +
>> test/vinyl/suite.ini | 1 +
>> test/xlog/suite.ini | 1 +
>> 4 files changed, 4 insertions(+)
>>
>> diff --git a/test/box-py/suite.ini b/test/box-py/suite.ini
>> index 18737e48f..d1ff0fbfd 100644
>> --- a/test/box-py/suite.ini
>> +++ b/test/box-py/suite.ini
>> @@ -5,3 +5,4 @@ script = box.lua
>> lua_libs = lua/fiber.lua lua/fifo.lua
>> use_unix_sockets = True
>> is_parallel = True
>> +fragile = snapshot.test.py ; gh-4514
>> diff --git a/test/replication-py/suite.ini b/test/replication-py/suite.ini
>> index df2dd8ef0..2658f4816 100644
>> --- a/test/replication-py/suite.ini
>> +++ b/test/replication-py/suite.ini
>> @@ -3,3 +3,4 @@ core = tarantool
>> script = master.lua
>> description = tarantool/box, replication
>> is_parallel = True
>> +fragile = init_storage.test.py ; gh-4949
>> diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
>> index c8bc270f3..44c53dc68 100644
>> --- a/test/vinyl/suite.ini
>> +++ b/test/vinyl/suite.ini
>> @@ -15,3 +15,4 @@ pretest_clean = True
>> fragile = errinj.test.lua ; gh-4346
>> select_consistency.test.lua ; gh-4385
>> throttle.test.lua ; gh-4168
>> + stat.test.lua ; gh-4951
>> diff --git a/test/xlog/suite.ini b/test/xlog/suite.ini
>> index 635ec53c1..8e4c4d374 100644
>> --- a/test/xlog/suite.ini
>> +++ b/test/xlog/suite.ini
>> @@ -10,3 +10,4 @@ use_unix_sockets_iproto = True
>> long_run = snap_io_rate.test.lua
>> is_parallel = True
>> pretest_clean = True
>> +fragile = xlog/checkpoint_daemon.test.lua ; gh-4952
>> --
>> 2.17.1
>>
>
>--
>sergeyb@ 
 
 
--
Oleg Piskunov
 

[-- Attachment #2: Type: text/html, Size: 3318 bytes --]

Re: [Tarantool-patches] [PATCH v1] test: fragile flaky tests from parallel runs

[Kirill Yukhin]

Hello,

On 30 апр 12:49, Alexander V. Tikhonov wrote:
> Fragiled flaky tests from parallel runs to avoid
> of flaky fails in regular testing:
> 
>   snapshot.test.py            ; gh-4514
>   init_storage.test.py        ; gh-4949
>   stat.test.lua               ; gh-4951
>   xlog/checkpoint_daemon.test.lua ; gh-4952
> 
> Part of #4953
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/fragile-flaky-tests
> Issue: https://github.com/tarantool/tarantool/issues/4953

I've checked your patch into 1.10, 2.3, 2.4 and master.

--
Regards, Kirill Yukhin