[PATCH v2] test: vinyl/errinj test fails under highload

[PATCH v2] test: vinyl/errinj test fails under highload

[avtikhon]

Test "check that all dump/compaction tasks that are in progress at
the time when the server stops are aborted immediately.", but in
real the awaiting time of 1 second is not enough due to runs in
parallel and it fails, like:

[009] --- vinyl/errinj.result Tue Apr 16 16:43:36 2019
[009] +++ vinyl/errinj.reject Wed Apr 17 09:42:36 2019
[009] @@ -530,7 +530,7 @@
[009] ...
[009] t2 - t1 < 1
[009] ---
[009] -- true
[009] +- false
[009] ...
[009] test_run:cmd("cleanup server test")
[009] ---
[009]

in 100 parallel runs the failed delays were found:

[002] +- 1.4104716777802
[022] +- 1.3933029174805
[044] +- 1.4296517372131
[033] +- 1.6380662918091
[001] +- 1.9799520969391
[027] +- 1.7067711353302
[043] +- 1.3778221607208
[034] +- 1.3820221424103
[032] +- 1.3820221424103
[020] +- 1.6275615692139
[050] +- 1.6275615692139
[048] +- 1.1880359649658

Decided to avoid of use the time check at all and change the
ERRINJ_VY_RUN_WRITE_STMT_TIMEOUT to ERRINJ_VY_DUMP_DELAY injection.
In this way the time checks were completely removed.

Next issue met was the error:

vy_quota.c:298 !> SystemError Failed to allocate 2097240 bytes
    in lsregion for vinyl transaction: Cannot allocate memory

That is why the merged 2 subtests were divided into 2 standalone
subtests to be able to set the memory limit of the 2nd subtest to
2097240 value.

Close #4169
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4169-vinyl-errinj-highload
Issue: https://github.com/tarantool/tarantool/issues/4169

 test/vinyl/errinj.result   | 46 +++++++++++++++++++++++++++-----------
 test/vinyl/errinj.test.lua | 17 +++++++++-----
 2 files changed, 44 insertions(+), 19 deletions(-)

diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result
index 6fc4d9e5d..c69347da6 100644
--- a/test/vinyl/errinj.result
+++ b/test/vinyl/errinj.result
@@ -488,10 +488,40 @@ box.snapshot()
 ---
 - ok
 ...
+test_run:cmd('switch default')
+---
+- true
+...
+test_run:cmd("stop server test") -- don't stuck
+---
+- true
+...
+test_run:cmd("cleanup server test")
+---
+- true
+...
 --
 -- Check that all dump/compaction tasks that are in progress at
 -- the time when the server stops are aborted immediately.
 --
+test_run:cmd("create server double_quota with script='vinyl/low_quota.lua'")
+---
+- true
+...
+test_run:cmd("start server double_quota with args='2097240'")
+---
+- true
+...
+test_run:cmd('switch double_quota')
+---
+- true
+...
+fiber = require 'fiber'
+---
+...
+box.cfg{vinyl_timeout = 0.001}
+---
+...
 s = box.schema.space.create('test', {engine = 'vinyl'})
 ---
 ...
@@ -501,7 +531,7 @@ _ = s:create_index('i1', {parts = {1, 'unsigned'}})
 _ = s:create_index('i2', {parts = {2, 'unsigned'}})
 ---
 ...
-box.error.injection.set('ERRINJ_VY_RUN_WRITE_STMT_TIMEOUT', 0.01)
+box.error.injection.set('ERRINJ_VY_DUMP_DELAY', true)
 ---
 - ok
 ...
@@ -518,21 +548,11 @@ test_run:cmd('switch default')
 ---
 - true
 ...
-t1 = fiber.time()
----
-...
-test_run:cmd("stop server test")
----
-- true
-...
-t2 = fiber.time()
----
-...
-t2 - t1 < 1
+test_run:cmd("stop server double_quota") -- don't stuck
 ---
 - true
 ...
-test_run:cmd("cleanup server test")
+test_run:cmd("cleanup server double_quota")
 ---
 - true
 ...
diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua
index b500b33c6..c3ce9706f 100644
--- a/test/vinyl/errinj.test.lua
+++ b/test/vinyl/errinj.test.lua
@@ -175,24 +175,29 @@ s:drop()
 -- Wait for the dump task to complete.
 box.error.injection.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
 box.snapshot()
+test_run:cmd('switch default')
+test_run:cmd("stop server test") -- don't stuck
+test_run:cmd("cleanup server test")
 
 --
 -- Check that all dump/compaction tasks that are in progress at
 -- the time when the server stops are aborted immediately.
 --
+test_run:cmd("create server double_quota with script='vinyl/low_quota.lua'")
+test_run:cmd("start server double_quota with args='2097240'")
+test_run:cmd('switch double_quota')
+fiber = require 'fiber'
+box.cfg{vinyl_timeout = 0.001}
 s = box.schema.space.create('test', {engine = 'vinyl'})
 _ = s:create_index('i1', {parts = {1, 'unsigned'}})
 _ = s:create_index('i2', {parts = {2, 'unsigned'}})
-box.error.injection.set('ERRINJ_VY_RUN_WRITE_STMT_TIMEOUT', 0.01)
+box.error.injection.set('ERRINJ_VY_DUMP_DELAY', true)
 for i = 1, 1000 do s:replace{i, i} end
 _ = fiber.create(function() box.snapshot() end)
 fiber.sleep(0.01)
 test_run:cmd('switch default')
-t1 = fiber.time()
-test_run:cmd("stop server test")
-t2 = fiber.time()
-t2 - t1 < 1
-test_run:cmd("cleanup server test")
+test_run:cmd("stop server double_quota") -- don't stuck
+test_run:cmd("cleanup server double_quota")
 
 s = box.schema.space.create('test', {engine = 'vinyl'})
 _ = s:create_index('i1', {parts = {1, 'unsigned'}})
-- 
2.17.1

Re: [PATCH v2] test: vinyl/errinj test fails under highload

[Vladimir Davydov]

Pushed to master, thanks!