[Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Alexander V. Tikhonov]

To stabilize testing with issue #5141 the following steps had to be
done:

- Prepare tests - add test-run filter on box.snapshot error message:

    'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

  to avoid of printing changing data in results file to be able to use
  its checksums in fragile list of test-run to rerun it as flaky issue.

- Create the stable reproducer for the issue #5141 as new standalone
  test vinyl/gh-5141-invalid-vylog-file.test.lua.

- Fix flaky issue #5141 in vinyl/gh-4957-too-many-upserts.test.lua.

- Fix hanging of vinyl/gh.test.lua. Revert previous fix for #5141 in
  vinyl/gh.test.lua.

To implement it created 4 patches (in reverse order):

4. test: fix hanging of vinyl/gh.test.lua

Found that the previously fixed vinyl/gh.test.lua test in commit:

  94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e ('test: gh test hangs after gh-4957-too-many-upserts')

with adding fiber.sleep(1) workaround to avoid of raise from the
previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
changed in the other way. The new change from one side will leave
the found issue untouched to be able to resolve it within opened
issue in github. And from the other side it will let the test-run
tool to be able to avoid of this issue using fragile list feature
to save the stability of testing due to found issue is flaky and
can be passed on reruns.

The current fix changes the forever waiting loop to especially
created for such situations test_run:wait_cond() routine which has
timeout in it to avoid of hanging the test till global timeout will
occure. It will let the testing to be continued even after the fail.

Needed for #5141

3. test: fix flaky vinyl/gh-4957-too-many-upserts

Added restart the current server to resolve the issue #5141 which
reproduced in test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.

Part of #5141

2. test: create reproducer for #5141

Created the stable reproducer for the issue #5141:

  box.snapshot()
  ---
 -- ok
 +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
  ...

flaky occured in vinyl/ suite tests if running after the test:

  vinyl/gh-4957-too-many-upserts.test.lua

as new standalone test:

  vinyl/gh-5141-invalid-vylog-file.test.lua

based on test:

  vinyl/gh-4957-too-many-upserts.test.lua

Due to issue not reproduced on FreeBSD 12, then test was blocked with:

  vinyl/gh-5141-invalid-vylog-file.skipcond

Needed for #5141

1. test: add test filter for box.snapshot

Added test-run filter on box.snapshot error message:

  'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'

to avoid of printing changing data in results file to be able to use
its checksums in fragile list of test-run to rerun it as flaky issue.
Also added checksums to fragile list for the following tests:

  vinyl/iterator.test.lua                       gh-5141
  vinyl/snapshot.test.lua                       gh-4984

Needed for #5141
Needed for #4984

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5141-gh_test
Issue: https://github.com/tarantool/tarantool/issues/5141

Alexander V. Tikhonov (4):
  test: add test filter for box.snapshot
  test: create reproducer for #5141
  test: fix flaky vinyl/gh-4957-too-many-upserts
  test: fix hanging of vinyl/gh.test.lua

 test/vinyl/gh-4957-too-many-upserts.result    |  17 +++
 test/vinyl/gh-4957-too-many-upserts.test.lua  |  11 ++
 test/vinyl/gh-5141-invalid-vylog-file.result  | 140 ++++++++++++++++++
 .../vinyl/gh-5141-invalid-vylog-file.skipcond |   6 +
 .../vinyl/gh-5141-invalid-vylog-file.test.lua |  72 +++++++++
 test/vinyl/gh.result                          |   6 +-
 test/vinyl/gh.test.lua                        |   4 +-
 test/vinyl/iterator.result                    |   5 +
 test/vinyl/iterator.test.lua                  |   3 +
 test/vinyl/snapshot.result                    |   5 +
 test/vinyl/snapshot.test.lua                  |   3 +
 test/vinyl/suite.ini                          |  10 +-
 12 files changed, 272 insertions(+), 10 deletions(-)
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.result
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.skipcond
 create mode 100644 test/vinyl/gh-5141-invalid-vylog-file.test.lua

-- 
2.25.1