[Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Alexander V. Tikhonov]

To stabilize testing with issue #5141 the following steps had to be
done:

- Prepare tests - add test-run filter on box.snapshot error message:

    'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

  to avoid of printing changing data in results file to be able to use
  its checksums in fragile list of test-run to rerun it as flaky issue.

- Create the stable reproducer for the issue #5141 as new standalone
  test vinyl/gh-5141-invalid-vylog-file.test.lua.

- Fix flaky issue #5141 in vinyl/gh-4957-too-many-upserts.test.lua.

- Fix hanging of vinyl/gh.test.lua. Revert previous fix for #5141 in
  vinyl/gh.test.lua.

To implement it created 4 patches (in reverse order):

4. test: fix hanging of vinyl/gh.test.lua

Found that the previously fixed vinyl/gh.test.lua test in commit:

  94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e ('test: gh test hangs after gh-4957-too-many-upserts')

with adding fiber.sleep(1) workaround to avoid of raise from the
previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
changed in the other way. The new change from one side will leave
the found issue untouched to be able to resolve it within opened
issue in github. And from the other side it will let the test-run
tool to be able to avoid of this issue using fragile list feature
to save the stability of testing due to found issue is flaky and
can be passed on reruns.

The current fix changes the forever waiting loop to especially
created for such situations test_run:wait_cond() routine which has
timeout in it to avoid of hanging the test till global timeout will
occure. It will let the testing to be continued even after the fail.

Needed for #5141

3. test: fix flaky vinyl/gh-4957-too-many-upserts

Added restart the current server to resolve the issue #5141 which
reproduced in test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.

Part of #5141

2. test: create reproducer for #5141

Created the stable reproducer for the issue #5141:

  box.snapshot()
  ---
 -- ok
 +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
  ...

flaky occured in vinyl/ suite tests if running after the test:

  vinyl/gh-4957-too-many-upserts.test.lua

as new standalone test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

based on test:

  vinyl/gh-4957-too-many-upserts.test.lua

Due to issue not reproduced on FreeBSD 12, then test was blocked with:

  vinyl/gh-5141-invalid-vylog-file.skipcond

Needed for #5141

1. test: add test filter for box.snapshot

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.
Also added checksums to fragile list for the following tests:

  vinyl/iterator.test.lua                       gh-5141
  vinyl/snapshot.test.lua                       gh-4984

Needed for #5141
Needed for #4984

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5141-gh_test
Issue: https://github.com/tarantool/tarantool/issues/5141

Alexander V. Tikhonov (4):
  test: add test filter for box.snapshot
  test: create reproducer for #5141
  test: fix flaky vinyl/gh-4957-too-many-upserts
  test: fix hanging of vinyl/gh.test.lua

 test/vinyl/gh-4957-too-many-upserts.result    |  17 +++
 test/vinyl/gh-4957-too-many-upserts.test.lua  |  11 ++
 test/vinyl/gh-5141-invalid-vylog-file.result  | 140 ++++++++++++++++++
 .../vinyl/gh-5141-invalid-vylog-file.skipcond |   6 +
 .../vinyl/gh-5141-invalid-vylog-file.test.lua |  72 +++++++++
 test/vinyl/gh.result                          |   6 +-
 test/vinyl/gh.test.lua                        |   4 +-
 test/vinyl/iterator.result                    |   5 +
 test/vinyl/iterator.test.lua                  |   3 +
 test/vinyl/snapshot.result                    |   5 +
 test/vinyl/snapshot.test.lua                  |   3 +
 test/vinyl/suite.ini                          |  10 +-
 12 files changed, 272 insertions(+), 10 deletions(-)
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.result
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.skipcond
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.test.lua

-- 
2.25.1

[Tarantool-patches] [PATCH v1 1/4] test: add test filter for box.snapshot

[Alexander V. Tikhonov]

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.
Also added checksums to fragile list for the following tests:

  vinyl/iterator.test.lua                       gh-5141
  vinyl/snapshot.test.lua                       gh-4984

Needed for #5141
Needed for #4984
---
 test/vinyl/iterator.result   | 5 +++++
 test/vinyl/iterator.test.lua | 3 +++
 test/vinyl/snapshot.result   | 5 +++++
 test/vinyl/snapshot.test.lua | 3 +++
 test/vinyl/suite.ini         | 4 ++--
 5 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/test/vinyl/iterator.result b/test/vinyl/iterator.result
index 0ef2a1b46..7992a386a 100644
--- a/test/vinyl/iterator.result
+++ b/test/vinyl/iterator.result
@@ -20,6 +20,11 @@ iterator_next = function(iter) return iter.next() end
 iterate_over = function(iter) return iter.iterate_over() end
 ---
 ...
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+---
+- true
+...
 --
 -- Following tests verify that combinations
 -- of various commands are worked correctly.
diff --git a/test/vinyl/iterator.test.lua b/test/vinyl/iterator.test.lua
index 8d2a85930..b5aeeda1e 100644
--- a/test/vinyl/iterator.test.lua
+++ b/test/vinyl/iterator.test.lua
@@ -12,6 +12,9 @@ create_iterator = require('utils').create_iterator
 iterator_next = function(iter) return iter.next() end
 iterate_over = function(iter) return iter.iterate_over() end
 
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+
 --
 -- Following tests verify that combinations
 -- of various commands are worked correctly.
diff --git a/test/vinyl/snapshot.result b/test/vinyl/snapshot.result
index 1c723ed74..2297fbbee 100644
--- a/test/vinyl/snapshot.result
+++ b/test/vinyl/snapshot.result
@@ -1,6 +1,11 @@
 test_run = require('test_run').new()
 ---
 ...
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+---
+- true
+...
 fiber = require 'fiber'
 ---
 ...
diff --git a/test/vinyl/snapshot.test.lua b/test/vinyl/snapshot.test.lua
index 0667dcae0..f698c1734 100644
--- a/test/vinyl/snapshot.test.lua
+++ b/test/vinyl/snapshot.test.lua
@@ -1,5 +1,8 @@
 test_run = require('test_run').new()
 
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+
 fiber = require 'fiber'
 fio = require 'fio'
 xlog = require 'xlog'
diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
index 870f25ddd..3ba29f0c6 100644
--- a/test/vinyl/suite.ini
+++ b/test/vinyl/suite.ini
@@ -37,7 +37,7 @@ fragile = {
         },
         "snapshot.test.lua": {
             "issues": [ "gh-4984" ],
-            "checksums": [ "10783b2ecef8db6735719304f3d72b88" ]
+            "checksums": [ "a221e2583ccef2ec884b945bbbba106e", "2caa0a1c7f07b57a1ad784688b8dff40" ]
         },
         "write_iterator.test.lua": {
             "issues": [ "gh-4572" ]
@@ -57,7 +57,7 @@ fragile = {
         },
         "iterator.test.lua": {
             "issues": [ "gh-5336" ],
-            "checksums": [ "138808fd31b68d1b1c53c6a146124856" ]
+            "checksums": [ "f56d6d279692e46d84a06f470af0090d", "a8a2824fb09380330a70a589d4fc545e", "e03548bcb918e824c349dc4c52ae3715", "adbd4ad9878302d570e40aef9a6b92cc", "39ee43ee7b2004166ca54402dfe02238" ]
         },
         "ddl.test.lua": {
             "issues": [ "gh-5338" ],
-- 
2.25.1

[Tarantool-patches] [PATCH v1 2/4] test: create reproducer for #5141

[Alexander V. Tikhonov]

Created the stable reproducer for the issue #5141:

  box.snapshot()
  ---
 -- ok
 +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
  ...

flaky occured in vinyl/ suite tests if running after the test:

  vinyl/gh-4957-too-many-upserts.test.lua

as new standalone test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

based on test:

  vinyl/gh-4957-too-many-upserts.test.lua

Due to issue not reproduced on FreeBSD 12, then test was blocked with:

  vinyl/gh-5141-invalid-vylog-file.skipcond

Needed for #5141
---
 test/vinyl/gh-5141-invalid-vylog-file.result  | 140 ++++++++++++++++++
 .../vinyl/gh-5141-invalid-vylog-file.skipcond |   6 +
 .../vinyl/gh-5141-invalid-vylog-file.test.lua |  72 +++++++++
 test/vinyl/suite.ini                          |   4 +
 4 files changed, 222 insertions(+)
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.result
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.skipcond
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.test.lua

diff --git a/test/vinyl/gh-5141-invalid-vylog-file.result b/test/vinyl/gh-5141-invalid-vylog-file.result
new file mode 100644
index 000000000..15caa75a9
--- /dev/null
+++ b/test/vinyl/gh-5141-invalid-vylog-file.result
@@ -0,0 +1,140 @@
+-- test-run result file version 2
+test_run = require('test_run').new()
+ | ---
+ | ...
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+ | ---
+ | - true
+ | ...
+
+-- restart the current server to avoid of issues from previous runs
+test_run:cmd("restart server default with cleanup=True")
+ | 
+
+-- Let's test number of upserts in one transaction that exceeds
+-- the limit of operations allowed in one update.
+--
+ups_cnt = 10000
+ | ---
+ | ...
+
+s = box.schema.create_space('test', {engine = 'vinyl'})
+ | ---
+ | ...
+pk = s:create_index('pk')
+ | ---
+ | ...
+
+tuple = {}
+ | ---
+ | ...
+for i = 1, ups_cnt do tuple[i] = i end
+ | ---
+ | ...
+_ = s:insert(tuple)
+ | ---
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+
+box.begin()
+ | ---
+ | ...
+for k = 1, ups_cnt do s:upsert({1}, {{'+', k, 1}}) end
+ | ---
+ | ...
+box.commit()
+ | ---
+ | ...
+-- Upserts are not able to squash, so scheduler will get stuck.
+-- So let's not waste much time here, just check that no crash
+-- takes place.
+--
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+require('fiber').sleep(0.01)
+ | ---
+ | ...
+
+s:drop()
+ | ---
+ | ...
+
+--
+-- WARNING: do not split from previous subtest.
+-- gh-5141: vinyl: after test with number of upserts in one transaction
+-- that exceeded the limit of operations allowed in one update, then
+-- box.snapshot() call may fail with error:
+--   "Invalid VYLOG file: Slice <NUM> deleted but not registered".
+-- To avoid of the issue tarantool server restart is needed.
+--
+
+-- let's check the issue reproduced
+s0 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+ | ---
+ | ...
+i0 = s0:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+ | ---
+ | ...
+cnt = 1
+ | ---
+ | ...
+s0:replace{cnt, 'tuple'}
+ | ---
+ | - [1, 'tuple']
+ | ...
+test_run:wait_cond(function()							\
+    cnt = cnt + 1								\
+    s0:replace{cnt, 'tuple ' .. cnt}						\
+    local ok, err = pcall(box.snapshot)                  			\
+    if ok == false then                                  			\
+        require('log').info(                            			\
+            "box.snapshot() returned on loop " .. cnt .. " 'err': " .. err) 	\
+    end                                                  			\
+    return err:match "Invalid VYLOG file: Slice %d+ deleted but not registered"	\
+end, 10)
+ | ---
+ | - 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
+ | ...
+s0:drop()
+ | ---
+ | ...
+
+-- let's check the issue is stable
+s1 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+ | ---
+ | ...
+i1 = s1:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+ | ---
+ | ...
+box.snapshot()
+ | ---
+ | - error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
+ | ...
+s1:drop()
+ | ---
+ | ...
+
+-- restart the current server to resolve the issue
+test_run:cmd("restart server default with cleanup=True")
+ | 
+
+-- let's check the issue resolved
+s2 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+ | ---
+ | ...
+i2 = s2:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+ | ---
+ | ...
+box.snapshot()
+ | ---
+ | - ok
+ | ...
+s2:drop()
+ | ---
+ | ...
diff --git a/test/vinyl/gh-5141-invalid-vylog-file.skipcond b/test/vinyl/gh-5141-invalid-vylog-file.skipcond
new file mode 100644
index 000000000..2d688cae1
--- /dev/null
+++ b/test/vinyl/gh-5141-invalid-vylog-file.skipcond
@@ -0,0 +1,6 @@
+# vim: set ft=python :
+import platform
+
+# Disabled on FreeBSD due to issue #5141 not reproduced.
+if platform.system() == 'FreeBSD':
+    self.skip = 1
diff --git a/test/vinyl/gh-5141-invalid-vylog-file.test.lua b/test/vinyl/gh-5141-invalid-vylog-file.test.lua
new file mode 100644
index 000000000..3df4a879e
--- /dev/null
+++ b/test/vinyl/gh-5141-invalid-vylog-file.test.lua
@@ -0,0 +1,72 @@
+test_run = require('test_run').new()
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+
+-- restart the current server to avoid of issues from previous runs
+test_run:cmd("restart server default with cleanup=True")
+
+-- Let's test number of upserts in one transaction that exceeds
+-- the limit of operations allowed in one update.
+--
+ups_cnt = 10000
+
+s = box.schema.create_space('test', {engine = 'vinyl'})
+pk = s:create_index('pk')
+
+tuple = {}
+for i = 1, ups_cnt do tuple[i] = i end
+_ = s:insert(tuple)
+box.snapshot()
+
+box.begin()
+for k = 1, ups_cnt do s:upsert({1}, {{'+', k, 1}}) end
+box.commit()
+-- Upserts are not able to squash, so scheduler will get stuck.
+-- So let's not waste much time here, just check that no crash
+-- takes place.
+--
+box.snapshot()
+require('fiber').sleep(0.01)
+
+s:drop()
+
+--
+-- WARNING: do not split from previous subtest.
+-- gh-5141: vinyl: after test with number of upserts in one transaction
+-- that exceeded the limit of operations allowed in one update, then
+-- box.snapshot() call may fail with error:
+--   "Invalid VYLOG file: Slice <NUM> deleted but not registered".
+-- To avoid of the issue tarantool server restart is needed.
+--
+
+-- let's check the issue reproduced
+s0 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+i0 = s0:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+cnt = 1
+s0:replace{cnt, 'tuple'}
+test_run:wait_cond(function()							\
+    cnt = cnt + 1								\
+    s0:replace{cnt, 'tuple ' .. cnt}						\
+    local ok, err = pcall(box.snapshot)                  			\
+    if ok == false then                                  			\
+        require('log').info(                            			\
+            "box.snapshot() returned on loop " .. cnt .. " 'err': " .. err) 	\
+    end                                                  			\
+    return err:match "Invalid VYLOG file: Slice %d+ deleted but not registered"	\
+end, 10)
+s0:drop()
+
+-- let's check the issue is stable
+s1 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+i1 = s1:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+box.snapshot()
+s1:drop()
+
+-- restart the current server to resolve the issue
+test_run:cmd("restart server default with cleanup=True")
+
+-- let's check the issue resolved
+s2 = box.schema.space.create('tweedledum', {engine = 'vinyl'})
+i2 = s2:create_index('primary', { type = 'tree', parts = {1, 'unsigned'}})
+box.snapshot()
+s2:drop()
diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
index 3ba29f0c6..2d69878af 100644
--- a/test/vinyl/suite.ini
+++ b/test/vinyl/suite.ini
@@ -55,6 +55,10 @@ fragile = {
             "issues": [ "gh-5141" ],
             "checksums": [ "f1286e9e4710062ddfbffb61b2fe2743", "96b22440ab8a881d6b8d14c5ee1672fb" ]
         },
+        "gh-5141-invalid-vylog-file.test.lua": {
+            "issues": [ "gh-5141" ],
+            "checksums": [ "1f4d4261fc63291c4020372986c62e2e" ]
+        },
         "iterator.test.lua": {
             "issues": [ "gh-5336" ],
             "checksums": [ "f56d6d279692e46d84a06f470af0090d", "a8a2824fb09380330a70a589d4fc545e", "e03548bcb918e824c349dc4c52ae3715", "adbd4ad9878302d570e40aef9a6b92cc", "39ee43ee7b2004166ca54402dfe02238" ]
-- 
2.25.1

[Tarantool-patches] [PATCH v1 3/4] test: fix flaky vinyl/gh-4957-too-many-upserts

[Alexander V. Tikhonov]

Added restart the current server to resolve the issue #5141 which
reproduced in test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.

Part of #5141
---
 test/vinyl/gh-4957-too-many-upserts.result   | 17 +++++++++++++++++
 test/vinyl/gh-4957-too-many-upserts.test.lua | 11 +++++++++++
 2 files changed, 28 insertions(+)

diff --git a/test/vinyl/gh-4957-too-many-upserts.result b/test/vinyl/gh-4957-too-many-upserts.result
index 67292f0f8..c942e62c8 100644
--- a/test/vinyl/gh-4957-too-many-upserts.result
+++ b/test/vinyl/gh-4957-too-many-upserts.result
@@ -1,4 +1,13 @@
 -- test-run result file version 2
+test_run = require('test_run').new()
+ | ---
+ | ...
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+ | ---
+ | - true
+ | ...
+
 s = box.schema.create_space('test', {engine = 'vinyl'})
  | ---
  | ...
@@ -90,3 +99,11 @@ fiber.sleep(0.01)
 s:drop()
  | ---
  | ...
+
+-- restart the current server to resolve the issue #5141
+-- which reproduced in test:
+--   vinyl/gh-5141-invalid-vylog-file.test.lua
+test_run:cmd("restart server default with cleanup=True")
+ | 
+
+
diff --git a/test/vinyl/gh-4957-too-many-upserts.test.lua b/test/vinyl/gh-4957-too-many-upserts.test.lua
index e5adfe41c..572540a4c 100644
--- a/test/vinyl/gh-4957-too-many-upserts.test.lua
+++ b/test/vinyl/gh-4957-too-many-upserts.test.lua
@@ -1,3 +1,7 @@
+test_run = require('test_run').new()
+test_run:cmd("push filter 'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'" .. \
+             "to 'Invalid VYLOG file: Slice <NUM> deleted but not registered'")
+
 s = box.schema.create_space('test', {engine = 'vinyl'})
 pk = s:create_index('pk')
 s:insert{1, 1}
@@ -36,3 +40,10 @@ box.snapshot()
 fiber.sleep(0.01)
 
 s:drop()
+
+-- restart the current server to resolve the issue #5141
+-- which reproduced in test:
+--   vinyl/gh-5141-invalid-vylog-file.test.lua
+test_run:cmd("restart server default with cleanup=True")
+
+
-- 
2.25.1

[Tarantool-patches] [PATCH v1 4/4] test: fix hanging of vinyl/gh.test.lua

[Alexander V. Tikhonov]

Found that the previously fixed vinyl/gh.test.lua test in commit:

  94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e ('test: gh test hangs after gh-4957-too-many-upserts')

with adding fiber.sleep(1) workaround to avoid of raise from the
previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
changed in the other way. The new change from one side will leave
the found issue untouched to be able to resolve it within opened
issue in github. And from the other side it will let the test-run
tool to be able to avoid of this issue using fragile list feature
to save the stability of testing due to found issue is flaky and
can be passed on reruns.

The current fix changes the forever waiting loop to especially
created for such situations test_run:wait_cond() routine which has
timeout in it to avoid of hanging the test till global timeout will
occure. It will let the testing to be continued even after the fail.

Needed for #5141
---
 test/vinyl/gh.result   | 6 ++----
 test/vinyl/gh.test.lua | 4 +---
 test/vinyl/suite.ini   | 2 +-
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/test/vinyl/gh.result b/test/vinyl/gh.result
index 0fd74af83..a0d3405de 100644
--- a/test/vinyl/gh.result
+++ b/test/vinyl/gh.result
@@ -7,9 +7,6 @@ env = require('test_run')
 test_run = env.new()
 ---
 ...
-fiber.sleep(1)
----
-...
 -- gh-283: hang after three creates and drops
 s = box.schema.space.create('space0', {engine='vinyl'})
 ---
@@ -716,8 +713,9 @@ test_run:cmd("setopt delimiter ''");
 cont = false
 ---
 ...
-while finished ~= 2 do fiber.sleep(0.01) end
+test_run:wait_cond(function() return finished ~= 2 end)
 ---
+- true
 ...
 s:drop()
 ---
diff --git a/test/vinyl/gh.test.lua b/test/vinyl/gh.test.lua
index c72444f5f..b6b9e8b07 100644
--- a/test/vinyl/gh.test.lua
+++ b/test/vinyl/gh.test.lua
@@ -2,8 +2,6 @@ fiber = require('fiber')
 env = require('test_run')
 test_run = env.new()
 
-fiber.sleep(1)
-
 -- gh-283: hang after three creates and drops
 s = box.schema.space.create('space0', {engine='vinyl'})
 i = s:create_index('space0', {type = 'tree', parts = {1, 'string'}})
@@ -311,7 +309,7 @@ end;
 test_run:cmd("setopt delimiter ''");
 
 cont = false
-while finished ~= 2 do fiber.sleep(0.01) end
+test_run:wait_cond(function() return finished ~= 2 end)
 
 s:drop()
 
diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
index 2d69878af..0f7b21793 100644
--- a/test/vinyl/suite.ini
+++ b/test/vinyl/suite.ini
@@ -53,7 +53,7 @@ fragile = {
         },
         "gh.test.lua": {
             "issues": [ "gh-5141" ],
-            "checksums": [ "f1286e9e4710062ddfbffb61b2fe2743", "96b22440ab8a881d6b8d14c5ee1672fb" ]
+            "checksums": [ "f1286e9e4710062ddfbffb61b2fe2743", "96b22440ab8a881d6b8d14c5ee1672fb", "fc77a97c3c891c57d4717087f7a65bd0" ]
         },
         "gh-5141-invalid-vylog-file.test.lua": {
             "issues": [ "gh-5141" ],
-- 
2.25.1

Re: [Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Kirill Yukhin]

Hello,

On 30 окт 09:59, Alexander V. Tikhonov wrote:
> To stabilize testing with issue #5141 the following steps had to be
> done:
> 
> - Prepare tests - add test-run filter on box.snapshot error message:
> 
>     'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
>   to avoid of printing changing data in results file to be able to use
>   its checksums in fragile list of test-run to rerun it as flaky issue.
> 
> - Create the stable reproducer for the issue #5141 as new standalone
>   test vinyl/gh-5141-invalid-vylog-file.test.lua.
> 
> - Fix flaky issue #5141 in vinyl/gh-4957-too-many-upserts.test.lua.
> 
> - Fix hanging of vinyl/gh.test.lua. Revert previous fix for #5141 in
>   vinyl/gh.test.lua.
> 
> To implement it created 4 patches (in reverse order):
> 
> 4. test: fix hanging of vinyl/gh.test.lua
> 
> Found that the previously fixed vinyl/gh.test.lua test in commit:
> 
>   94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e ('test: gh test hangs after gh-4957-too-many-upserts')
> 
> with adding fiber.sleep(1) workaround to avoid of raise from the
> previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
> changed in the other way. The new change from one side will leave
> the found issue untouched to be able to resolve it within opened
> issue in github. And from the other side it will let the test-run
> tool to be able to avoid of this issue using fragile list feature
> to save the stability of testing due to found issue is flaky and
> can be passed on reruns.
> 
> The current fix changes the forever waiting loop to especially
> created for such situations test_run:wait_cond() routine which has
> timeout in it to avoid of hanging the test till global timeout will
> occure. It will let the testing to be continued even after the fail.
> 
> Needed for #5141
> 
> 3. test: fix flaky vinyl/gh-4957-too-many-upserts
> 
> Added restart the current server to resolve the issue #5141 which
> reproduced in test:
> 
>   vinyl/gh-5141-invalid-vylog-file.test.lua
> 
> Added test-run filter on box.snapshot error message:
> 
>   'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
> to avoid of printing changing data in results file to be able to use
> its checksums in fragile list of test-run to rerun it as flaky issue.
> 
> Part of #5141
> 
> 2. test: create reproducer for #5141
> 
> Created the stable reproducer for the issue #5141:
> 
>   box.snapshot()
>   ---
>  -- ok
>  +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
>   ...
> 
> flaky occured in vinyl/ suite tests if running after the test:
> 
>   vinyl/gh-4957-too-many-upserts.test.lua
> 
> as new standalone test:
> 
>   vinyl/gh-5141-invalid-vylog-file.test.lua
> 
> based on test:
> 
>   vinyl/gh-4957-too-many-upserts.test.lua
> 
> Due to issue not reproduced on FreeBSD 12, then test was blocked with:
> 
>   vinyl/gh-5141-invalid-vylog-file.skipcond
> 
> Needed for #5141
> 
> 1. test: add test filter for box.snapshot
> 
> Added test-run filter on box.snapshot error message:
> 
>   'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
> to avoid of printing changing data in results file to be able to use
> its checksums in fragile list of test-run to rerun it as flaky issue.
> Also added checksums to fragile list for the following tests:
> 
>   vinyl/iterator.test.lua                       gh-5141
>   vinyl/snapshot.test.lua                       gh-4984
> 
> Needed for #5141
> Needed for #4984
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5141-gh_test
> Issue: https://github.com/tarantool/tarantool/issues/5141

LGTM.
I've checked your patchset into 1.10, 2.5, 2.6 and master.

--
Regards, Kirill Yukhin

Re: [Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Nikita Pettik]

On 30 Oct 09:59, Alexander V. Tikhonov wrote:
> To stabilize testing with issue #5141 the following steps had to be
> done:

Сорри что влезаю, но что тут происходит? Парни, если тест сломан,
его надо нормально починить, а не костылить слипами и тд. Там есть
одна явная бага (5436), ей надо заняться. Я просто вижу какое-то насилование
винильных тестов, это по-другому и не назовешь. Если надо срочно починить CI -
просто отрубите этот тест (с апсертами) и все. Я постараюсь им заняться в
ближайшее (и видимо свободное) время.
 
> - Prepare tests - add test-run filter on box.snapshot error message:
> 
>     'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
>   to avoid of printing changing data in results file to be able to use
>   its checksums in fragile list of test-run to rerun it as flaky issue.
> 
> - Create the stable reproducer for the issue #5141 as new standalone
>   test vinyl/gh-5141-invalid-vylog-file.test.lua.

А зачем репродюсер тут? Я же уже и так открыл 5436 и вынес минимальный
репро туда.
 

Re: [Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Alexander V. Tikhonov]

Никита, привет. Да у нас есть проблема от этого теста почти во всех
винильных тестах использующих box.snapshot. Включать и выключать тесты,
как мне кажется, это совсем крайний случай, поэтому мы попытались хоть
как-то изолировать проблему, но чтобы проблема продолжала
идентифицироваться был заведён тест полностью копирующий те части
тестов, что её обнаруживали, без каких-либо изменений. Да, то что есть
минимальный репродьюсер, это отлично, и то что тесты требуют правок
сомнений нет. Но с нашей стороны сейчас основная задача это
стабилизировать тестирование не растеряв найденые проблемы. Поэтому
предлагаю сейчас не пытаться найти новый путь решения проблемы, а
реально зафиксироваться на текущем состоянии, так как и тестирование
стабилизировано и тесты выявляют найденную проблему. В рамках правок
билета 5436 тесты явно будут исправляться и нет смысла с нашей стороны
предлагать свои решения для этого раньше, чем билет будет исправлен.

В ближайшее время мы хотим предложить правила и структуру тестирования,
которая позволит нам с одной стороны проводить полноценное тестирование,
а с другой позволит поддерживать CI в зелёной зоне, это позволит нам
реализовать возможность использования схемы со стейджтнгом, который
сможет автоматически проливать прошедшие тестирование патчи в стабильную
релизную ветку, что спасёт нас от предрелизного аврала и траты времени
многих разработчиков в этот дорогой период на поиски проблем и их
спешного исправления, когда их время нужно им самим для завершения их
задач. Основная проблема на данный момент заключается в том, что тесты
у нас воспринимаются, как не отдельные функциональные тесты, а именно
как части стресс тестирования, которые выявляют многие новые проблемы.
С одной стороны это обязательно нужно, но с другой стороны никто в
текущей ситуации не гарантирует стабильный запуск тестов в определённых
последовательностях, и это генерит флеки падения, которые сильно
стреляют. Мы не собираемся лишать всех подобного способа тестирования,
и хотим предложить именно отдельно стояший вариант запуска стресс
тестирования, в котором можно будет отслеживать и последовательности
запуска и нагрузки. В данный момент наш тест-ран сам призвольно выбирает
количество параллельных запусков и решает какие и сколько воркеров
нужно, это меняется от машины к машине так как количество параллельых
задач у нас зависит от количества процессоров на машинах. Так же у нас
по разному загружены машины в тестировании и это меняет нагрузку в
тестировании, всё это генерит флеки падения.

On Sun, Nov 01, 2020 at 09:57:35PM +0000, Nikita Pettik wrote:
> On 30 Oct 09:59, Alexander V. Tikhonov wrote:
> > To stabilize testing with issue #5141 the following steps had to be
> > done:
> 
> Сорри что влезаю, но что тут происходит? Парни, если тест сломан,
> его надо нормально починить, а не костылить слипами и тд. Там есть
> одна явная бага (5436), ей надо заняться. Я просто вижу какое-то насилование
> винильных тестов, это по-другому и не назовешь. Если надо срочно починить CI -
> просто отрубите этот тест (с апсертами) и все. Я постараюсь им заняться в
> ближайшее (и видимо свободное) время.
>  
> > - Prepare tests - add test-run filter on box.snapshot error message:
> > 
> >     'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> > 
> >   to avoid of printing changing data in results file to be able to use
> >   its checksums in fragile list of test-run to rerun it as flaky issue.
> > 
> > - Create the stable reproducer for the issue #5141 as new standalone
> >   test vinyl/gh-5141-invalid-vylog-file.test.lua.
> 
> А зачем репродюсер тут? Я же уже и так открыл 5436 и вынес минимальный
> репро туда.
>