[Tarantool-patches] [PATCH v2 0/2] test: flaky hang vinyl/ddl.test.lua test

[Tarantool-patches] [PATCH v2 0/2] test: flaky hang vinyl/ddl.test.lua test

[Alexander V. Tikhonov]

Patchset consists of 2 commits:

1. Revert "Fix flaky test engine/ddl"
    
   Found that the previous fix of the engine/ddl.test.lua test committed
   with:
    
      5f96ee5907672bd32dd7fe8dad144ac3d328187f ('Fix flaky test engine/ddl')
    
   did not fix the issue #4353 in real and it was reverted.
 
2. test: flaky hang vinyl/ddl.test.lua test
    
   Found hanging test vinyl/ddl.test.lua on:
    
     [159]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
     [159]  ---
     [159]  - true
     [159]  ...
     [159] -box.snapshot()
     [159] ----
     [159] -- ok
     [159] -...
    
   The real issue happend before it when test failed on:
    
     [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
     [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
     [091] @@ -2558,7 +2558,7 @@
     [091]  ...
     [091]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
     [091]  ---
     [091] -- true
     [091] +- false
     [091]  ...
    
   Our tests have structure when different standalone subtests exists
   in the test files. To be able to check all of them this hang must
   be neutralized to give the next standalone subtest ability to pass.
   To avoid of this hang decided to disable box.snapshot check if the
   previous check of the current subtest failed.
    
   Needed for #4353

Alexander V. Tikhonov (2):
  Revert "Fix flaky test engine/ddl"
  test: flaky hang vinyl/ddl.test.lua test
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4353-hang-ddl
Issue: https://github.com/tarantool/tarantool/issues/4353

 test/engine/ddl.result   | 91 ++++++++++++++++++++++++++--------------
 test/engine/ddl.test.lua | 76 ++++++++++++++++++++++++++-------
 test/engine/suite.ini    |  2 +-
 3 files changed, 122 insertions(+), 47 deletions(-)

-- 
2.24.1 (Apple Git-126)

[Tarantool-patches] [PATCH v2 1/2] Revert "Fix flaky test engine/ddl"

[Alexander V. Tikhonov]

Found that the previous fix of the engine/ddl.test.lua test committed
with:

  5f96ee5907672bd32dd7fe8dad144ac3d328187f ('Fix flaky test engine/ddl')

did not fix the issue #4353 in real and it was reverted.

Needed for #4353
---
 test/engine/ddl.result   | 15 ++++++---------
 test/engine/ddl.test.lua | 14 ++++++--------
 2 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/test/engine/ddl.result b/test/engine/ddl.result
index 8b44d1ee1..b7c04aafe 100644
--- a/test/engine/ddl.result
+++ b/test/engine/ddl.result
@@ -2556,23 +2556,20 @@ ch:get()
 ---
 - true
 ...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
 ---
 - true
 ...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 ---
 - true
 ...
 inspector:cmd("restart server default")
-inspector = require('test_run').new()
----
-...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
 ---
 - true
 ...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 ---
 - true
 ...
@@ -2580,11 +2577,11 @@ box.snapshot()
 ---
 - ok
 ...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
 ---
 - true
 ...
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 ---
 - true
 ...
diff --git a/test/engine/ddl.test.lua b/test/engine/ddl.test.lua
index 1d77705dd..7d408807f 100644
--- a/test/engine/ddl.test.lua
+++ b/test/engine/ddl.test.lua
@@ -1016,17 +1016,15 @@ _ = fiber.create(function() gen_load() ch:put(true) end)
 _ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
 ch:get()
 
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 
 inspector:cmd("restart server default")
 
-inspector = require('test_run').new()
-
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 box.snapshot()
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
-inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
+box.space.test.index.pk:count() == box.space.test.index.sk:count()
+box.space.test.index.pk:count() == box.space.test.index.tk:count()
 
 box.space.test:drop()
-- 
2.24.1 (Apple Git-126)

[Tarantool-patches] [PATCH v2 2/2] test: flaky hang vinyl/ddl.test.lua test

[Alexander V. Tikhonov]

Found hanging test vinyl/ddl.test.lua on:

  [159]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
  [159]  ---
  [159]  - true
  [159]  ...
  [159] -box.snapshot()
  [159] ----
  [159] -- ok
  [159] -...

The real issue happend before it when test failed on:

  [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
  [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
  [091] @@ -2558,7 +2558,7 @@
  [091]  ...
  [091]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
  [091]  ---
  [091] -- true
  [091] +- false
  [091]  ...

Our tests have structure when different standalone subtests exists
in the test files. To be able to check all of them this hang must
be neutralized to give the next standalone subtest ability to pass.
To avoid of this hang decided to disable box.snapshot check if the
previous check of the current subtest failed.

Needed for #4353
---
 test/engine/ddl.result   | 92 +++++++++++++++++++++++++++-------------
 test/engine/ddl.test.lua | 76 +++++++++++++++++++++++++++------
 test/engine/suite.ini    |  2 +-
 3 files changed, 125 insertions(+), 45 deletions(-)

diff --git a/test/engine/ddl.result b/test/engine/ddl.result
index b7c04aafe..6603da719 100644
--- a/test/engine/ddl.result
+++ b/test/engine/ddl.result
@@ -2526,62 +2526,94 @@ function gen_load()
 end;
 ---
 ...
-inspector:cmd("setopt delimiter ''");
----
-- true
-...
-fiber = require('fiber')
----
-...
-ch = fiber.channel(1)
----
-...
-_ = fiber.create(function() gen_load() ch:put(true) end)
+function check_equal(check, pk, k)
+    if pk ~= k then
+        require('log').error("Error on fiber check: failed '" .. check .. 
+	                     "' check on equal pk " .. pk .. " and k = " .. k)
+        return false
+    end
+    return true
+end;
 ---
 ...
-_ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
+function check_fiber()
+    _ = fiber.create(function() gen_load() ch:put(true) end)
+    _ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
+
+    ch:get(10)
+
+    local index = box.space.test.index
+    if not check_equal("1st step secondary keys", index.pk:count(), index.sk:count()) then
+        return false
+    end
+
+    _ = fiber.create(function() gen_load() ch:put(true) end)
+    _ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
+
+    ch:get(10)
+
+    index = box.space.test.index
+    if not check_equal("2nd step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("2nd step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    return true
+end;
 ---
 ...
-ch:get()
+inspector:cmd("setopt delimiter ''");
 ---
 - true
 ...
-_ = fiber.create(function() gen_load() ch:put(true) end)
----
-...
-_ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
+fiber = require('fiber')
 ---
 ...
-ch:get()
+ch = fiber.channel(1)
 ---
-- true
 ...
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
+check_fiber()
 ---
 - true
 ...
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
+inspector:cmd("restart server default")
+inspector = require('test_run').new()
 ---
-- true
 ...
-inspector:cmd("restart server default")
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
+inspector:cmd("setopt delimiter ';'")
 ---
 - true
 ...
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
+function check_equal(check, pk, k)
+    if pk ~= k then
+        require('log').error("Error on server restart check: failed '" .. check ..
+                             "' check on equal pk " .. pk .. " and k = " .. k)
+        return false
+    end
+    return true
+end;
 ---
-- true
 ...
-box.snapshot()
+function check_server_restart()
+    local index = box.space.test.index
+    if not check_equal("1rd step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("1rd step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    box.snapshot()
+    index = box.space.test.index
+    if not check_equal("2th step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("2th step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    return true
+end;
 ---
-- ok
 ...
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
+inspector:cmd("setopt delimiter ''");
 ---
 - true
 ...
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
+check_server_restart()
 ---
 - true
 ...
diff --git a/test/engine/ddl.test.lua b/test/engine/ddl.test.lua
index 7d408807f..5ba80e075 100644
--- a/test/engine/ddl.test.lua
+++ b/test/engine/ddl.test.lua
@@ -1003,28 +1003,76 @@ function gen_load()
     end
 end;
 
+function check_equal(check, pk, k)
+    if pk ~= k then
+        require('log').error("Error on fiber check: failed '" .. check .. 
+	                     "' check on equal pk " .. pk .. " and k = " .. k)
+        return false
+    end
+    return true
+end;
+
+function check_fiber()
+    _ = fiber.create(function() gen_load() ch:put(true) end)
+    _ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
+
+    ch:get(10)
+
+    local index = box.space.test.index
+    if not check_equal("1st step secondary keys", index.pk:count(), index.sk:count()) then
+        return false
+    end
+
+    _ = fiber.create(function() gen_load() ch:put(true) end)
+    _ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
+
+    ch:get(10)
+
+    index = box.space.test.index
+    if not check_equal("2nd step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("2nd step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    return true
+end;
+
 inspector:cmd("setopt delimiter ''");
 
 fiber = require('fiber')
 ch = fiber.channel(1)
+check_fiber()
 
-_ = fiber.create(function() gen_load() ch:put(true) end)
-_ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
-ch:get()
+inspector:cmd("restart server default")
+inspector = require('test_run').new()
 
-_ = fiber.create(function() gen_load() ch:put(true) end)
-_ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
-ch:get()
+inspector:cmd("setopt delimiter ';'")
 
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
+function check_equal(check, pk, k)
+    if pk ~= k then
+        require('log').error("Error on server restart check: failed '" .. check ..
+                             "' check on equal pk " .. pk .. " and k = " .. k)
+        return false
+    end
+    return true
+end;
 
-inspector:cmd("restart server default")
+function check_server_restart()
+    local index = box.space.test.index
+    if not check_equal("1rd step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("1rd step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    box.snapshot()
+    index = box.space.test.index
+    if not check_equal("2th step secondary keys", index.pk:count(), index.sk:count()) or
+            not check_equal("2th step third keys", index.pk:count(), index.tk:count()) then
+        return false
+    end
+    return true
+end;
 
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
-box.snapshot()
-box.space.test.index.pk:count() == box.space.test.index.sk:count()
-box.space.test.index.pk:count() == box.space.test.index.tk:count()
+inspector:cmd("setopt delimiter ''");
+
+check_server_restart()
 
 box.space.test:drop()
diff --git a/test/engine/suite.ini b/test/engine/suite.ini
index ca68e8f83..7a8bfd9b7 100644
--- a/test/engine/suite.ini
+++ b/test/engine/suite.ini
@@ -15,7 +15,7 @@ fragile = {
     "tests": {
         "ddl.test.lua": {
             "issues": [ "gh-4353" ],
-            "checksums": [ "dd8851d80183cc75052119ba646e295d" ]
+            "checksums": [ "c04fa9a0fae1bb63a86a5304f7679245" ]
         },
         "gh-4973-concurrent-alter-fails.test.lua": {
             "issues": [ "gh-5157" ],
-- 
2.24.1 (Apple Git-126)

Re: [Tarantool-patches] [PATCH v2 2/2] test: flaky hang vinyl/ddl.test.lua test

[Aleksandr Lyapunov]

Hi! Thanks for the path!
See 4 comments below:

On 04.12.2020 11:05, Alexander V. Tikhonov wrote:
> Found hanging test vinyl/ddl.test.lua on:
>
>    [159]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
>    [159]  ---
>    [159]  - true
>    [159]  ...
>    [159] -box.snapshot()
>    [159] ----
>    [159] -- ok
>    [159] -...
>
> The real issue happend before it when test failed on:
>
>    [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
>    [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
>    [091] @@ -2558,7 +2558,7 @@
>    [091]  ...
>    [091]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
>    [091]  ---
>    [091] -- true
>    [091] +- false
>    [091]  ...
>
> Our tests have structure when different standalone subtests exists
> in the test files. To be able to check all of them this hang must
> be neutralized to give the next standalone subtest ability to pass.
> To avoid of this hang decided to disable box.snapshot check if the
> previous check of the current subtest failed.
>
> Needed for #4353
> ---
>   test/engine/ddl.result   | 92 +++++++++++++++++++++++++++-------------
>   test/engine/ddl.test.lua | 76 +++++++++++++++++++++++++++------
>   test/engine/suite.ini    |  2 +-
>   3 files changed, 125 insertions(+), 45 deletions(-)
>
> diff --git a/test/engine/ddl.result b/test/engine/ddl.result
> index b7c04aafe..6603da719 100644
> --- a/test/engine/ddl.result
> +++ b/test/engine/ddl.result
> @@ -2526,62 +2526,94 @@ function gen_load()
>   end;
>   ---
>   ...
> -inspector:cmd("setopt delimiter ''");
> ----
> -- true
> -...
> -fiber = require('fiber')
> ----
> -...
> -ch = fiber.channel(1)
> ----
> -...
> -_ = fiber.create(function() gen_load() ch:put(true) end)
> +function check_equal(check, pk, k)
> +    if pk ~= k then
> +        require('log').error("Error on fiber check: failed '" .. check ..
> +	                     "' check on equal pk " .. pk .. " and k = " .. k)
> +        return false
> +    end
> +    return true
> +end;
>   ---
>   ...
> -_ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
> +function check_fiber()
> +    _ = fiber.create(function() gen_load() ch:put(true) end)
> +    _ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
> +
> +    ch:get(10)
> +
> +    local index = box.space.test.index
> +    if not check_equal("1st step secondary keys", index.pk:count(), index.sk:count()) then
> +        return false
> +    end
> +
> +    _ = fiber.create(function() gen_load() ch:put(true) end)
> +    _ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
> +
> +    ch:get(10)
> +
> +    index = box.space.test.index
> +    if not check_equal("2nd step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("2nd step third keys", index.pk:count(), index.tk:count()) then
> +        return false
> +    end
> +    return true
> +end;
>   ---
>   ...
> -ch:get()
> +inspector:cmd("setopt delimiter ''");
>   ---
>   - true
>   ...
> -_ = fiber.create(function() gen_load() ch:put(true) end)
> ----
> -...
> -_ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
> +fiber = require('fiber')
>   ---
>   ...
> -ch:get()
> +ch = fiber.channel(1)
>   ---
> -- true
>   ...
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> +check_fiber()
>   ---
>   - true
>   ...
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> +inspector:cmd("restart server default")
> +inspector = require('test_run').new()
>   ---
> -- true
>   ...
> -inspector:cmd("restart server default")
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> +inspector:cmd("setopt delimiter ';'")
>   ---
>   - true
>   ...
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> +function check_equal(check, pk, k)
> +    if pk ~= k then
> +        require('log').error("Error on server restart check: failed '" .. check ..
> +                             "' check on equal pk " .. pk .. " and k = " .. k)
> +        return false
> +    end
> +    return true
> +end;
>   ---
> -- true
>   ...
> -box.snapshot()
> +function check_server_restart()
> +    local index = box.space.test.index
> +    if not check_equal("1rd step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("1rd step third keys", index.pk:count(), index.tk:count()) then
> +        return false
> +    end
> +    box.snapshot()
> +    index = box.space.test.index
> +    if not check_equal("2th step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("2th step third keys", index.pk:count(), index.tk:count()) then
> +        return false
> +    end
> +    return true
> +end;
>   ---
> -- ok
>   ...
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> +inspector:cmd("setopt delimiter ''");
>   ---
>   - true
>   ...
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> +check_server_restart()
>   ---
>   - true
>   ...
> diff --git a/test/engine/ddl.test.lua b/test/engine/ddl.test.lua
> index 7d408807f..5ba80e075 100644
> --- a/test/engine/ddl.test.lua
> +++ b/test/engine/ddl.test.lua
> @@ -1003,28 +1003,76 @@ function gen_load()
>       end
>   end;
>   
> +function check_equal(check, pk, k)
> +    if pk ~= k then
> +        require('log').error("Error on fiber check: failed '" .. check ..
> +	                     "' check on equal pk " .. pk .. " and k = " .. k)
> +        return false
> +    end
> +    return true
> +end;
> +
> +function check_fiber()
> +    _ = fiber.create(function() gen_load() ch:put(true) end)
> +    _ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
> +
> +    ch:get(10)
Why have you replaced ch:get() with ch:get(10) ?
> +
> +    local index = box.space.test.index
> +    if not check_equal("1st step secondary keys", index.pk:count(), index.sk:count()) then
> +        return false
> +    end
> +
> +    _ = fiber.create(function() gen_load() ch:put(true) end)
> +    _ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
> +
> +    ch:get(10)
And here?
> +
> +    index = box.space.test.index
> +    if not check_equal("2nd step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("2nd step third keys", index.pk:count(), index.tk:count()) then
> +        return false
> +    end
> +    return true
> +end;
> +
>   inspector:cmd("setopt delimiter ''");
>   
>   fiber = require('fiber')
>   ch = fiber.channel(1)
> +check_fiber()
>   
> -_ = fiber.create(function() gen_load() ch:put(true) end)
> -_ = box.space.test:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
> -ch:get()
> +inspector:cmd("restart server default")
> +inspector = require('test_run').new()
>   
> -_ = fiber.create(function() gen_load() ch:put(true) end)
> -_ = box.space.test:create_index('tk', {unique = true, parts = {3, 'unsigned'}})
> -ch:get()
> +inspector:cmd("setopt delimiter ';'")
>   
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> +function check_equal(check, pk, k)
> +    if pk ~= k then
> +        require('log').error("Error on server restart check: failed '" .. check ..
> +                             "' check on equal pk " .. pk .. " and k = " .. k)
> +        return false
> +    end
> +    return true
> +end;
>   
> -inspector:cmd("restart server default")
> +function check_server_restart()
> +    local index = box.space.test.index
> +    if not check_equal("1rd step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("1rd step third keys", index.pk:count(), index.tk:count()) then
2x '1rd' -> '1st'
> +        return false
> +    end
> +    box.snapshot()
> +    index = box.space.test.index
> +    if not check_equal("2th step secondary keys", index.pk:count(), index.sk:count()) or
> +            not check_equal("2th step third keys", index.pk:count(), index.tk:count()) then
2x '2th' -> '2nd'
> +        return false
> +    end
> +    return true
> +end;
>   
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> -box.snapshot()
> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
> +inspector:cmd("setopt delimiter ''");
> +
> +check_server_restart()
>   
>   box.space.test:drop()
> diff --git a/test/engine/suite.ini b/test/engine/suite.ini
> index ca68e8f83..7a8bfd9b7 100644
> --- a/test/engine/suite.ini
> +++ b/test/engine/suite.ini
> @@ -15,7 +15,7 @@ fragile = {
>       "tests": {
>           "ddl.test.lua": {
>               "issues": [ "gh-4353" ],
> -            "checksums": [ "dd8851d80183cc75052119ba646e295d" ]
> +            "checksums": [ "c04fa9a0fae1bb63a86a5304f7679245" ]
>           },
>           "gh-4973-concurrent-alter-fails.test.lua": {
>               "issues": [ "gh-5157" ],

Re: [Tarantool-patches] [PATCH v2 2/2] test: flaky hang vinyl/ddl.test.lua test

[Aleksandr Lyapunov]

Oh I just have been told that ch:get(10) mean timeout 10 second. Ok.

On 04.12.2020 11:19, Aleksandr Lyapunov via Tarantool-patches wrote:
> Hi! Thanks for the path!
> See 4 comments below:
>
> On 04.12.2020 11:05, Alexander V. Tikhonov wrote:
>> Found hanging test vinyl/ddl.test.lua on:
>>
>>    [159]  inspector:wait_cond(function() return 
>> box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
>>    [159]  ---
>>    [159]  - true
>>    [159]  ...
>>    [159] -box.snapshot()
>>    [159] ----
>>    [159] -- ok
>>    [159] -...
>>
>> The real issue happend before it when test failed on:
>>
>>    [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
>>    [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
>>    [091] @@ -2558,7 +2558,7 @@
>>    [091]  ...
>>    [091]  inspector:wait_cond(function() return 
>> box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
>>    [091]  ---
>>    [091] -- true
>>    [091] +- false
>>    [091]  ...
>>
>> Our tests have structure when different standalone subtests exists
>> in the test files. To be able to check all of them this hang must
>> be neutralized to give the next standalone subtest ability to pass.
>> To avoid of this hang decided to disable box.snapshot check if the
>> previous check of the current subtest failed.
>>
>> Needed for #4353
>> ---
>>   test/engine/ddl.result   | 92 +++++++++++++++++++++++++++-------------
>>   test/engine/ddl.test.lua | 76 +++++++++++++++++++++++++++------
>>   test/engine/suite.ini    |  2 +-
>>   3 files changed, 125 insertions(+), 45 deletions(-)
>>
>> diff --git a/test/engine/ddl.result b/test/engine/ddl.result
>> index b7c04aafe..6603da719 100644
>> --- a/test/engine/ddl.result
>> +++ b/test/engine/ddl.result
>> @@ -2526,62 +2526,94 @@ function gen_load()
>>   end;
>>   ---
>>   ...
>> -inspector:cmd("setopt delimiter ''");
>> ----
>> -- true
>> -...
>> -fiber = require('fiber')
>> ----
>> -...
>> -ch = fiber.channel(1)
>> ----
>> -...
>> -_ = fiber.create(function() gen_load() ch:put(true) end)
>> +function check_equal(check, pk, k)
>> +    if pk ~= k then
>> +        require('log').error("Error on fiber check: failed '" .. 
>> check ..
>> +                         "' check on equal pk " .. pk .. " and k = " 
>> .. k)
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   ---
>>   ...
>> -_ = box.space.test:create_index('sk', {unique = false, parts = {2, 
>> 'unsigned'}})
>> +function check_fiber()
>> +    _ = fiber.create(function() gen_load() ch:put(true) end)
>> +    _ = box.space.test:create_index('sk', {unique = false, parts = 
>> {2, 'unsigned'}})
>> +
>> +    ch:get(10)
>> +
>> +    local index = box.space.test.index
>> +    if not check_equal("1st step secondary keys", index.pk:count(), 
>> index.sk:count()) then
>> +        return false
>> +    end
>> +
>> +    _ = fiber.create(function() gen_load() ch:put(true) end)
>> +    _ = box.space.test:create_index('tk', {unique = true, parts = 
>> {3, 'unsigned'}})
>> +
>> +    ch:get(10)
>> +
>> +    index = box.space.test.index
>> +    if not check_equal("2nd step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("2nd step third keys", index.pk:count(), 
>> index.tk:count()) then
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   ---
>>   ...
>> -ch:get()
>> +inspector:cmd("setopt delimiter ''");
>>   ---
>>   - true
>>   ...
>> -_ = fiber.create(function() gen_load() ch:put(true) end)
>> ----
>> -...
>> -_ = box.space.test:create_index('tk', {unique = true, parts = {3, 
>> 'unsigned'}})
>> +fiber = require('fiber')
>>   ---
>>   ...
>> -ch:get()
>> +ch = fiber.channel(1)
>>   ---
>> -- true
>>   ...
>> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> +check_fiber()
>>   ---
>>   - true
>>   ...
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> +inspector:cmd("restart server default")
>> +inspector = require('test_run').new()
>>   ---
>> -- true
>>   ...
>> -inspector:cmd("restart server default")
>> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> +inspector:cmd("setopt delimiter ';'")
>>   ---
>>   - true
>>   ...
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> +function check_equal(check, pk, k)
>> +    if pk ~= k then
>> +        require('log').error("Error on server restart check: failed 
>> '" .. check ..
>> +                             "' check on equal pk " .. pk .. " and k 
>> = " .. k)
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   ---
>> -- true
>>   ...
>> -box.snapshot()
>> +function check_server_restart()
>> +    local index = box.space.test.index
>> +    if not check_equal("1rd step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("1rd step third keys", index.pk:count(), 
>> index.tk:count()) then
>> +        return false
>> +    end
>> +    box.snapshot()
>> +    index = box.space.test.index
>> +    if not check_equal("2th step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("2th step third keys", index.pk:count(), 
>> index.tk:count()) then
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   ---
>> -- ok
>>   ...
>> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> +inspector:cmd("setopt delimiter ''");
>>   ---
>>   - true
>>   ...
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> +check_server_restart()
>>   ---
>>   - true
>>   ...
>> diff --git a/test/engine/ddl.test.lua b/test/engine/ddl.test.lua
>> index 7d408807f..5ba80e075 100644
>> --- a/test/engine/ddl.test.lua
>> +++ b/test/engine/ddl.test.lua
>> @@ -1003,28 +1003,76 @@ function gen_load()
>>       end
>>   end;
>>   +function check_equal(check, pk, k)
>> +    if pk ~= k then
>> +        require('log').error("Error on fiber check: failed '" .. 
>> check ..
>> +                         "' check on equal pk " .. pk .. " and k = " 
>> .. k)
>> +        return false
>> +    end
>> +    return true
>> +end;
>> +
>> +function check_fiber()
>> +    _ = fiber.create(function() gen_load() ch:put(true) end)
>> +    _ = box.space.test:create_index('sk', {unique = false, parts = 
>> {2, 'unsigned'}})
>> +
>> +    ch:get(10)
> Why have you replaced ch:get() with ch:get(10) ?
>> +
>> +    local index = box.space.test.index
>> +    if not check_equal("1st step secondary keys", index.pk:count(), 
>> index.sk:count()) then
>> +        return false
>> +    end
>> +
>> +    _ = fiber.create(function() gen_load() ch:put(true) end)
>> +    _ = box.space.test:create_index('tk', {unique = true, parts = 
>> {3, 'unsigned'}})
>> +
>> +    ch:get(10)
> And here?
>> +
>> +    index = box.space.test.index
>> +    if not check_equal("2nd step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("2nd step third keys", index.pk:count(), 
>> index.tk:count()) then
>> +        return false
>> +    end
>> +    return true
>> +end;
>> +
>>   inspector:cmd("setopt delimiter ''");
>>     fiber = require('fiber')
>>   ch = fiber.channel(1)
>> +check_fiber()
>>   -_ = fiber.create(function() gen_load() ch:put(true) end)
>> -_ = box.space.test:create_index('sk', {unique = false, parts = {2, 
>> 'unsigned'}})
>> -ch:get()
>> +inspector:cmd("restart server default")
>> +inspector = require('test_run').new()
>>   -_ = fiber.create(function() gen_load() ch:put(true) end)
>> -_ = box.space.test:create_index('tk', {unique = true, parts = {3, 
>> 'unsigned'}})
>> -ch:get()
>> +inspector:cmd("setopt delimiter ';'")
>>   -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> +function check_equal(check, pk, k)
>> +    if pk ~= k then
>> +        require('log').error("Error on server restart check: failed 
>> '" .. check ..
>> +                             "' check on equal pk " .. pk .. " and k 
>> = " .. k)
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   -inspector:cmd("restart server default")
>> +function check_server_restart()
>> +    local index = box.space.test.index
>> +    if not check_equal("1rd step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("1rd step third keys", index.pk:count(), 
>> index.tk:count()) then
> 2x '1rd' -> '1st'
>> +        return false
>> +    end
>> +    box.snapshot()
>> +    index = box.space.test.index
>> +    if not check_equal("2th step secondary keys", index.pk:count(), 
>> index.sk:count()) or
>> +            not check_equal("2th step third keys", index.pk:count(), 
>> index.tk:count()) then
> 2x '2th' -> '2nd'
>> +        return false
>> +    end
>> +    return true
>> +end;
>>   -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> -box.snapshot()
>> -box.space.test.index.pk:count() == box.space.test.index.sk:count()
>> -box.space.test.index.pk:count() == box.space.test.index.tk:count()
>> +inspector:cmd("setopt delimiter ''");
>> +
>> +check_server_restart()
>>     box.space.test:drop()
>> diff --git a/test/engine/suite.ini b/test/engine/suite.ini
>> index ca68e8f83..7a8bfd9b7 100644
>> --- a/test/engine/suite.ini
>> +++ b/test/engine/suite.ini
>> @@ -15,7 +15,7 @@ fragile = {
>>       "tests": {
>>           "ddl.test.lua": {
>>               "issues": [ "gh-4353" ],
>> -            "checksums": [ "dd8851d80183cc75052119ba646e295d" ]
>> +            "checksums": [ "c04fa9a0fae1bb63a86a5304f7679245" ]
>>           },
>>           "gh-4973-concurrent-alter-fails.test.lua": {
>>               "issues": [ "gh-5157" ],

Re: [Tarantool-patches] [PATCH v2 0/2] test: flaky hang vinyl/ddl.test.lua test

[Aleksandr Lyapunov]

Hi! thanks for the patch!
LGTM, I understand now.
Please see my minor comments.

On 04.12.2020 11:05, Alexander V. Tikhonov wrote:
> Patchset consists of 2 commits:
>
> 1. Revert "Fix flaky test engine/ddl"
>      
>     Found that the previous fix of the engine/ddl.test.lua test committed
>     with:
>      
>        5f96ee5907672bd32dd7fe8dad144ac3d328187f ('Fix flaky test engine/ddl')
>      
>     did not fix the issue #4353 in real and it was reverted.
>   
> 2. test: flaky hang vinyl/ddl.test.lua test
>      
>     Found hanging test vinyl/ddl.test.lua on:
>      
>       [159]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
>       [159]  ---
>       [159]  - true
>       [159]  ...
>       [159] -box.snapshot()
>       [159] ----
>       [159] -- ok
>       [159] -...
>      
>     The real issue happend before it when test failed on:
>      
>       [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
>       [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
>       [091] @@ -2558,7 +2558,7 @@
>       [091]  ...
>       [091]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
>       [091]  ---
>       [091] -- true
>       [091] +- false
>       [091]  ...
>      
>     Our tests have structure when different standalone subtests exists
>     in the test files. To be able to check all of them this hang must
>     be neutralized to give the next standalone subtest ability to pass.
>     To avoid of this hang decided to disable box.snapshot check if the
>     previous check of the current subtest failed.
>      
>     Needed for #4353
>
> Alexander V. Tikhonov (2):
>    Revert "Fix flaky test engine/ddl"
>    test: flaky hang vinyl/ddl.test.lua test
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4353-hang-ddl
> Issue: https://github.com/tarantool/tarantool/issues/4353
>
>   test/engine/ddl.result   | 91 ++++++++++++++++++++++++++--------------
>   test/engine/ddl.test.lua | 76 ++++++++++++++++++++++++++-------
>   test/engine/suite.ini    |  2 +-
>   3 files changed, 122 insertions(+), 47 deletions(-)
>

Re: [Tarantool-patches] [PATCH v2 0/2] test: flaky hang vinyl/ddl.test.lua test

[Kirill Yukhin]

Hello,

On 04 Dec 11:05, Alexander V. Tikhonov wrote:
> Patchset consists of 2 commits:
> 
> 1. Revert "Fix flaky test engine/ddl"
>     
>    Found that the previous fix of the engine/ddl.test.lua test committed
>    with:
>     
>       5f96ee5907672bd32dd7fe8dad144ac3d328187f ('Fix flaky test engine/ddl')
>     
>    did not fix the issue #4353 in real and it was reverted.
>  
> 2. test: flaky hang vinyl/ddl.test.lua test
>     
>    Found hanging test vinyl/ddl.test.lua on:
>     
>      [159]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.tk:count() end)
>      [159]  ---
>      [159]  - true
>      [159]  ...
>      [159] -box.snapshot()
>      [159] ----
>      [159] -- ok
>      [159] -...
>     
>    The real issue happend before it when test failed on:
>     
>      [091] --- engine/ddl.result   Thu May 14 16:12:09 2020
>      [091] +++ engine/ddl.reject   Fri May 15 04:15:07 2020
>      [091] @@ -2558,7 +2558,7 @@
>      [091]  ...
>      [091]  inspector:wait_cond(function() return box.space.test.index.pk:count() == box.space.test.index.sk:count() end)
>      [091]  ---
>      [091] -- true
>      [091] +- false
>      [091]  ...
>     
>    Our tests have structure when different standalone subtests exists
>    in the test files. To be able to check all of them this hang must
>    be neutralized to give the next standalone subtest ability to pass.
>    To avoid of this hang decided to disable box.snapshot check if the
>    previous check of the current subtest failed.
>     
>    Needed for #4353
> 
> Alexander V. Tikhonov (2):
>   Revert "Fix flaky test engine/ddl"
>   test: flaky hang vinyl/ddl.test.lua test
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4353-hang-ddl
> Issue: https://github.com/tarantool/tarantool/issues/4353

I've checked your patch into 1.10 2.5 2.6 and master.

--
Regards, Kirill Yukhin