Re: [Tarantool-patches] [PATCH v1 0/4] test: stabilize testing with issue #5141

[Kirill Yukhin <kyukhin@tarantool.org>]

Hello,

On 30 окт 09:59, Alexander V. Tikhonov wrote:
> To stabilize testing with issue #5141 the following steps had to be
> done:
> 
> - Prepare tests - add test-run filter on box.snapshot error message:
> 
>     'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
>   to avoid of printing changing data in results file to be able to use
>   its checksums in fragile list of test-run to rerun it as flaky issue.
> 
> - Create the stable reproducer for the issue #5141 as new standalone
>   test vinyl/gh-5141-invalid-vylog-file.test.lua.
> 
> - Fix flaky issue #5141 in vinyl/gh-4957-too-many-upserts.test.lua.
> 
> - Fix hanging of vinyl/gh.test.lua. Revert previous fix for #5141 in
>   vinyl/gh.test.lua.
> 
> To implement it created 4 patches (in reverse order):
> 
> 4. test: fix hanging of vinyl/gh.test.lua
> 
> Found that the previously fixed vinyl/gh.test.lua test in commit:
> 
>   94dc5bddc1fb0f9e6ebebfc5aa6be586e5b6759e ('test: gh test hangs after gh-4957-too-many-upserts')
> 
> with adding fiber.sleep(1) workaround to avoid of raise from the
> previously run vinyl/gh-4957-too-many-upserts.test.lua test can be
> changed in the other way. The new change from one side will leave
> the found issue untouched to be able to resolve it within opened
> issue in github. And from the other side it will let the test-run
> tool to be able to avoid of this issue using fragile list feature
> to save the stability of testing due to found issue is flaky and
> can be passed on reruns.
> 
> The current fix changes the forever waiting loop to especially
> created for such situations test_run:wait_cond() routine which has
> timeout in it to avoid of hanging the test till global timeout will
> occure. It will let the testing to be continued even after the fail.
> 
> Needed for #5141
> 
> 3. test: fix flaky vinyl/gh-4957-too-many-upserts
> 
> Added restart the current server to resolve the issue #5141 which
> reproduced in test:
> 
>   vinyl/gh-5141-invalid-vylog-file.test.lua
> 
> Added test-run filter on box.snapshot error message:
> 
>   'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
> to avoid of printing changing data in results file to be able to use
> its checksums in fragile list of test-run to rerun it as flaky issue.
> 
> Part of #5141
> 
> 2. test: create reproducer for #5141
> 
> Created the stable reproducer for the issue #5141:
> 
>   box.snapshot()
>   ---
>  -- ok
>  +- error: 'Invalid VYLOG file: Slice <NUM> deleted but not registered'
>   ...
> 
> flaky occured in vinyl/ suite tests if running after the test:
> 
>   vinyl/gh-4957-too-many-upserts.test.lua
> 
> as new standalone test:
> 
>   vinyl/gh-5141-invalid-vylog-file.test.lua
> 
> based on test:
> 
>   vinyl/gh-4957-too-many-upserts.test.lua
> 
> Due to issue not reproduced on FreeBSD 12, then test was blocked with:
> 
>   vinyl/gh-5141-invalid-vylog-file.skipcond
> 
> Needed for #5141
> 
> 1. test: add test filter for box.snapshot
> 
> Added test-run filter on box.snapshot error message:
> 
>   'Invalid VYLOG file: Slice [0-9]+ deleted but not registered'
> 
> to avoid of printing changing data in results file to be able to use
> its checksums in fragile list of test-run to rerun it as flaky issue.
> Also added checksums to fragile list for the following tests:
> 
>   vinyl/iterator.test.lua                       gh-5141
>   vinyl/snapshot.test.lua                       gh-4984
> 
> Needed for #5141
> Needed for #4984
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5141-gh_test
> Issue: https://github.com/tarantool/tarantool/issues/5141

LGTM.
I've checked your patchset into 1.10, 2.5, 2.6 and master.

--
Regards, Kirill Yukhin