[tarantool-patches] [PATCH 0/1] Move luajit-related tests to luajit repo

[tarantool-patches] [PATCH 0/1] Move luajit-related tests to luajit repo

[Igor Munkin]

All luajit-tap tests moved to luajit repo within this patch

Branch: https://github.com/tarantool/luajit/tree/imun/luajit-test-suite
Issue: https://github.com/tarantool/tarantool/issues/4478

Igor Munkin (1):
  test: add tests from tarantool source tree

 test/fold_bug_LuaJIT_505.test.lua        |  21 +++
 test/gh.test.lua                         |  17 +++
 test/suite.ini                           |   5 +
 test/table_chain_bug_LuaJIT_494.test.lua | 178 +++++++++++++++++++++++
 test/unsink_64_kptr.test.lua             |  44 ++++++
 5 files changed, 265 insertions(+)
 create mode 100755 test/fold_bug_LuaJIT_505.test.lua
 create mode 100755 test/gh.test.lua
 create mode 100644 test/suite.ini
 create mode 100755 test/table_chain_bug_LuaJIT_494.test.lua
 create mode 100755 test/unsink_64_kptr.test.lua

-- 
2.22.1

[tarantool-patches] [PATCH 1/1] test: add tests from tarantool source tree

[Igor Munkin]

Introduced structure has the following benefits:
* Excess testing machinery currently is not required directly in
luajit repo considering whole CI process setup for tarantool and its
static linking with libluajit
* All new tests related to luajit fixes and enhancements can be added
within a single patch with the corresponding changeset

NB!: All added test chunks use tarantool tap module
---
 test/fold_bug_LuaJIT_505.test.lua        |  21 +++
 test/gh.test.lua                         |  17 +++
 test/suite.ini                           |   5 +
 test/table_chain_bug_LuaJIT_494.test.lua | 178 +++++++++++++++++++++++
 test/unsink_64_kptr.test.lua             |  44 ++++++
 5 files changed, 265 insertions(+)
 create mode 100755 test/fold_bug_LuaJIT_505.test.lua
 create mode 100755 test/gh.test.lua
 create mode 100644 test/suite.ini
 create mode 100755 test/table_chain_bug_LuaJIT_494.test.lua
 create mode 100755 test/unsink_64_kptr.test.lua

diff --git a/test/fold_bug_LuaJIT_505.test.lua b/test/fold_bug_LuaJIT_505.test.lua
new file mode 100755
index 0000000..71e2cba
--- /dev/null
+++ b/test/fold_bug_LuaJIT_505.test.lua
@@ -0,0 +1,21 @@
+#!/usr/bin/env tarantool
+
+tap = require('tap')
+
+test = tap.test("505")
+test:plan(1)
+
+-- Test file to demonstrate Lua fold machinery icorrect behavior, details:
+--     https://github.com/LuaJIT/LuaJIT/issues/505
+
+jit.opt.start(0, "hotloop=1")
+require('jit.dump').start("+bti", "505.trace")
+for _ = 1, 20 do
+    local value = "abc"
+    local pos_c = string.find(value, "c", 1, true)
+    local value2 = string.sub(value, 1, pos_c - 1)
+    local pos_b = string.find(value2, "b", 2, true)
+    assert(pos_b == 2, "FAIL: position of 'b' is " .. pos_b)
+end
+
+test:ok("PASS")
diff --git a/test/gh.test.lua b/test/gh.test.lua
new file mode 100755
index 0000000..00b71a6
--- /dev/null
+++ b/test/gh.test.lua
@@ -0,0 +1,17 @@
+#!/usr/bin/env tarantool
+
+-- Miscellaneous test for LuaJIT bugs
+tap = require('tap')
+
+test = tap.test("gh")
+test:plan(2)
+--
+-- gh-3196: incorrect string length if Lua hash returns 0
+--
+h = "\x1F\x93\xE2\x1C\xCA\xDE\x28\x08\x26\x01\xED\x0A\x2F\xE4\x21\x02\x97\x77\xD9\x3E"
+test:is(h:len(), 20)
+
+h = "\x0F\x93\xE2\x1C\xCA\xDE\x28\x08\x26\x01\xED\x0A\x2F\xE4\x21\x02\x97\x77\xD9\x3E"
+test:is(h:len(), 20)
+
+test:check()
diff --git a/test/suite.ini b/test/suite.ini
new file mode 100644
index 0000000..3e860c9
--- /dev/null
+++ b/test/suite.ini
@@ -0,0 +1,5 @@
+[default]
+core = app
+description = Luajit tests
+is_parallel = True
+pretest_clean = True
diff --git a/test/table_chain_bug_LuaJIT_494.test.lua b/test/table_chain_bug_LuaJIT_494.test.lua
new file mode 100755
index 0000000..06c0f0d
--- /dev/null
+++ b/test/table_chain_bug_LuaJIT_494.test.lua
@@ -0,0 +1,178 @@
+#!/usr/bin/env tarantool
+
+tap = require('tap')
+
+test = tap.test("494")
+test:plan(1)
+
+-- Test file to demonstrate Lua table hash chain bugs discussed in
+--     https://github.com/LuaJIT/LuaJIT/issues/494
+-- Credit: prepared by Peter Cawley here with minor edits:
+--     https://gist.github.com/corsix/1fc9b13a2dd5f3659417b62dd54d4500
+
+--- Plumbing
+ffi = require"ffi"
+ffi.cdef"char* strstr(const char*, const char*)"
+strstr = ffi.C.strstr
+cast = ffi.cast
+str_hash_offset = cast("uint32_t*", strstr("*", ""))[-2] == 1 and 3 or 2
+function str_hash(s)
+    return cast("uint32_t*", strstr(s, "")) - str_hash_offset
+end
+table_new = require"table.new"
+
+--- Prepare some objects
+victims = {}
+orig_hash = {}
+for c in ("abcdef"):gmatch"." do
+    v = c .. "{09add58a-13a4-44e0-a52c-d44d0f9b2b95}"
+    victims[c] = v
+    orig_hash[c] = str_hash(v)[0]
+end
+collectgarbage()
+
+do --- Basic version of the problem
+    for k, v in pairs(victims) do
+        str_hash(v)[0] = 0
+    end
+    t = table_new(0, 8)
+    -- Make chain a -> b -> c -> d, all with a as primary
+    t[victims.a] = true
+    t[victims.d] = true
+    t[victims.c] = true
+    t[victims.b] = true
+    -- Change c's primary to b, and d's primary to c
+    t[victims.d] = nil
+    t[victims.c] = nil
+    str_hash(victims.c)[0] = 5
+    str_hash(victims.d)[0] = 6
+    t[victims.c] = true
+    t[victims.d] = true
+    -- Insert something with b as primary
+    str_hash(victims.e)[0] = 5
+    t[victims.e] = true
+    -- Check for consistency
+    for c in ("abcde"):gmatch"." do
+        assert(t[victims[c]], c)
+    end
+end
+collectgarbage()
+
+do --- Just `mn != freenode` can lead to infinite loops
+    for k, v in pairs(victims) do
+        str_hash(v)[0] = 0
+    end
+    t = table_new(0, 8)
+    -- Make chain a -> b -> c -> d, all with a as primary
+    t[victims.a] = true
+    t[victims.d] = true
+    t[victims.c] = true
+    t[victims.b] = true
+    -- Change c's primary to b, and d's primary to d
+    t[victims.d] = nil
+    t[victims.c] = nil
+    str_hash(victims.c)[0] = 5
+    str_hash(victims.d)[0] = 7
+    t[victims.c] = true
+    t[victims.d] = true
+    -- Insert something with b as primary
+    str_hash(victims.e)[0] = 5
+    t[victims.e] = true
+    -- Insert something with d as primary (infinite lookup loop)
+    str_hash(victims.f)[0] = 7
+    t[victims.f] = true
+end
+collectgarbage()
+
+do --- Just `mn != nn` can lead to infinite loops
+    for k, v in pairs(victims) do
+        str_hash(v)[0] = 0
+    end
+    t = table_new(0, 8)
+    -- Make chain a -> b -> c -> d -> e, all with a as primary
+    t[victims.a] = true
+    t[victims.e] = true
+    t[victims.d] = true
+    t[victims.c] = true
+    t[victims.b] = true
+    -- Change c's primary to b, d's primary to d, and e's primary to d
+    t[victims.e] = nil
+    t[victims.d] = nil
+    t[victims.c] = nil
+    str_hash(victims.c)[0] = 4
+    str_hash(victims.d)[0] = 6
+    str_hash(victims.e)[0] = 6
+    t[victims.c] = true
+    t[victims.d] = true
+    t[victims.e] = true
+    -- Insert something with b as primary (infinite rechaining loop)
+    str_hash(victims.f)[0] = 4
+    t[victims.f] = true
+end
+
+for i = 0, 10 do --- Non-strings can need rechaining too
+    collectgarbage()
+
+    k = tonumber((("0x%xp-1074"):format(i)))
+    str_hash(victims.a)[0] = 0
+    str_hash(victims.b)[0] = 0
+    t = table_new(0, 4)
+    -- a -> b, both with a as primary
+    t[victims.a] = true
+    t[victims.b] = true
+    -- Change b's primary to b
+    t[victims.b] = nil
+    str_hash(victims.b)[0] = 3
+    t[victims.b] = true
+    -- Might get a -> b -> k, with k's primary as b
+    t[k] = true
+    -- Change b's primary to a
+    t[victims.b] = nil
+    str_hash(victims.b)[0] = 0
+    t[victims.b] = true
+    -- Insert something with b as primary
+    str_hash(victims.c)[0] = 3
+    t[victims.c] = true
+    -- Check for consistency
+    assert(t[k], i)
+end
+
+for i = 0, 10 do --- Non-strings can be moved to freenode
+    collectgarbage()
+
+    k = false
+    str_hash(victims.a)[0] = 0
+    str_hash(victims.b)[0] = 0
+    t = table_new(0, 4)
+    -- a -> k -> b, all with a as primary
+    t[victims.a] = true
+    t[victims.b] = true
+    t[k] = true
+    -- Change b's primary to k
+    t[victims.b] = nil
+    str_hash(victims.b)[0] = 2
+    t[victims.b] = true
+    -- Insert a non-string with primary of k
+    t[tonumber((("0x%xp-1074"):format(i)))] = true
+    -- Check for consistency
+    assert(t[victims.b], i)
+end
+collectgarbage()
+
+do --- Do not forget to advance freenode in the not-string case
+    t = table_new(0, 4)
+    -- Chain of colliding numbers
+    t[0x0p-1074] = true
+    t[0x4p-1074] = true
+    t[0x8p-1074] = true
+    -- Steal middle node of the chain to be a main node (infinite walking loop)
+    t[0x2p-1074] = true
+end
+collectgarbage()
+
+--- Restore interpreter invariants, just in case
+for c, v in pairs(victims) do
+    str_hash(v)[0] = orig_hash[c]
+end
+
+test:ok("PASS")
diff --git a/test/unsink_64_kptr.test.lua b/test/unsink_64_kptr.test.lua
new file mode 100755
index 0000000..8995763
--- /dev/null
+++ b/test/unsink_64_kptr.test.lua
@@ -0,0 +1,44 @@
+#!/usr/bin/env tarantool
+
+tap = require('tap')
+
+test = tap.test("232")
+test:plan(1)
+
+--- From: Thibault Charbonnier <thibaultcha@me.com>
+--- tests: ffi: added a test case unsinking a 64-bit pointer from a constant.
+---
+--- This test case reproduces the issue observed at:
+--- https://github.com/openresty/lua-resty-core/issues/232 and was
+--- contributed by @lukego and myself.
+---
+--- Co-authored-by: Luke Gorrie <lukego@gmail.com>
+---
+local ffi = require("ffi")
+
+local array = ffi.new("struct { int x; } [1]")
+
+-- This test forces the VM to unsink a pointer that was constructed
+-- from a constant. The IR will include a 'cnewi' instruction to
+-- allocate an FFI pointer object, the pointer value will be an IR
+-- constant, the allocation will be sunk, and the allocation will
+-- at some point be "unsunk" due to a reference in the snapshot for
+-- a taken exit.
+
+-- Note: JIT will recognize <array> as a "singleton" and allow its
+-- address to be inlined ("constified") instead of looking up the
+-- upvalue at runtime.
+
+local function fn(i)
+    local struct = array[0]     -- Load pointer that the JIT will constify.
+    if i == 1000 then end       -- Force trace exit when i==1000.
+    struct.x = 0                -- Ensure that 'struct' is live after exit.
+end
+
+-- Loop over the function to make it compile and take a trace exit
+-- during the final iteration.
+for i = 1, 1000 do
+    fn(i)
+end
+
+test:ok("PASS")
-- 
2.22.1

[tarantool-patches] Re: [PATCH 1/1] test: add tests from tarantool source tree

[Alexander Turenko]

I found the following difference with the current master:

 | --- a/luajit-tap/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:45:04.886639461 +0300
 | +++ b/test/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:44:22.221641210 +0300
 | @@ -8,7 +8,8 @@
 |  -- Test file to demonstrate Lua fold machinery icorrect behavior, details:
 |  --     https://github.com/LuaJIT/LuaJIT/issues/505
 |  
 | -jit.opt.start("hotloop=1")
 | +jit.opt.start(0, "hotloop=1")
 | +require('jit.dump').start("+bti", "505.trace")
 |  for _ = 1, 20 do
 |      local value = "abc"
 |      local pos_c = string.find(value, "c", 1, true)

From what it appears? If it is necessary, then move it to its own commit
or describe in the commit message.

Please, base your branch on top of a last commit in a default branch
(named 'tarantool'). Now it contains one extra commit ('luajit: fix
string.find recording').

Verified in-source build, works ok.

Tried out-of-source build: small and luajit-tap tests are not run (just
not found).

If we'll create just symlinks in 'test' repository directory and will
remove creating symlinks from test/CMakeLists.txt, then luajit-tap would
work good, but small tests would not be built (and so small tests will
be found).

It seems it is subject for a separate issue: don't sure about test-run
or tarantool. Something was changed in test-run I guess; maybe here:
https://github.com/tarantool/test-run/commit/b42093abc0e7fe895ff0ac7c0685d016c783b3a1

Don't sure whether it worth to block on this problem or file an issue
and going on. Please, ask Kirill for that.

Other then that everything look okay for me.

WBR, Alexander Turenko.

On Thu, Sep 05, 2019 at 05:20:07PM +0300, Igor Munkin wrote:
> Introduced structure has the following benefits:
> * Excess testing machinery currently is not required directly in
> luajit repo considering whole CI process setup for tarantool and its
> static linking with libluajit
> * All new tests related to luajit fixes and enhancements can be added
> within a single patch with the corresponding changeset
> 
> NB!: All added test chunks use tarantool tap module
> ---
>  test/fold_bug_LuaJIT_505.test.lua        |  21 +++
>  test/gh.test.lua                         |  17 +++
>  test/suite.ini                           |   5 +
>  test/table_chain_bug_LuaJIT_494.test.lua | 178 +++++++++++++++++++++++
>  test/unsink_64_kptr.test.lua             |  44 ++++++
>  5 files changed, 265 insertions(+)
>  create mode 100755 test/fold_bug_LuaJIT_505.test.lua
>  create mode 100755 test/gh.test.lua
>  create mode 100644 test/suite.ini
>  create mode 100755 test/table_chain_bug_LuaJIT_494.test.lua
>  create mode 100755 test/unsink_64_kptr.test.lua

[tarantool-patches] Re[2]: [tarantool-patches] [PATCH 1/1] test: add tests from tarantool source tree

[Igor Munkin]

[-- Attachment #1: Type: text/plain, Size: 3276 bytes --]


Sasha,

Thanks for you review. 
>Вторник, 10 сентября 2019, 16:58 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>I found the following difference with the current master:
>
> | --- a/luajit-tap/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:45:04.886639461 +0300
> | +++ b/test/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:44:22.221641210 +0300
> | @@ -8,7 +8,8 @@
> |  -- Test file to demonstrate Lua fold machinery icorrect behavior, details:
> |  --  https://github.com/LuaJIT/LuaJIT/issues/505
> | 
> | -jit.opt.start("hotloop=1")
> | +jit.opt.start(0, "hotloop=1")
> | +require('jit.dump').start("+bti", "505.trace")
> |  for _ = 1, 20 do
> |      local value = "abc"
> |      local pos_c = string.find(value, "c", 1, true)
>
>From what it appears? If it is necessary, then move it to its own commit
>or describe in the commit message. 
The difference you found is an artefact left after investigation of the problem related to string.find recording. These changes are not required within this patch.
>
>
>Please, base your branch on top of a last commit in a default branch
>(named 'tarantool'). Now it contains one extra commit ('luajit: fix
>string.find recording').
Another artefact.
>
>
>Verified in-source build, works ok.
>
>Tried out-of-source build: small and luajit-tap tests are not run (just
>not found).
>
>If we'll create just symlinks in 'test' repository directory and will
>remove creating symlinks from test/CMakeLists.txt, then luajit-tap would
>work good, but small tests would not be built (and so small tests will
>be found).
>
>It seems it is subject for a separate issue: don't sure about test-run
>or tarantool. Something was changed in test-run I guess; maybe here:
>https://github.com/tarantool/test-run/commit/b42093abc0e7fe895ff0ac7c0685d016c783b3a1
>
>Don't sure whether it worth to block on this problem or file an issue
>and going on. Please, ask Kirill for that.
>
>Other then that everything look okay for me. 
I've already fixed all mentioned flaws in the feature branch and forcely update the corresponding remote branch. Please inform me whether a new patch is required to be sent.
>
>
>WBR, Alexander Turenko.
>
>On Thu, Sep 05, 2019 at 05:20:07PM +0300, Igor Munkin wrote:
>> Introduced structure has the following benefits:
>> * Excess testing machinery currently is not required directly in
>> luajit repo considering whole CI process setup for tarantool and its
>> static linking with libluajit
>> * All new tests related to luajit fixes and enhancements can be added
>> within a single patch with the corresponding changeset
>> 
>> NB!: All added test chunks use tarantool tap module
>> ---
>>  test/fold_bug_LuaJIT_505.test.lua        |  21 +++
>>  test/gh.test.lua                         |  17 +++
>>  test/suite.ini                           |   5 +
>>  test/table_chain_bug_LuaJIT_494.test.lua | 178 +++++++++++++++++++++++
>>  test/unsink_64_kptr.test.lua             |  44 ++++++
>>  5 files changed, 265 insertions(+)
>>  create mode 100755 test/fold_bug_LuaJIT_505.test.lua
>>  create mode 100755 test/gh.test.lua
>>  create mode 100644 test/suite.ini
>>  create mode 100755 test/table_chain_bug_LuaJIT_494.test.lua
>>  create mode 100755 test/unsink_64_kptr.test.lua

-- 
IM

[-- Attachment #2: Type: text/html, Size: 4739 bytes --]

[tarantool-patches] Re: [PATCH 1/1] test: add tests from tarantool source tree

[Alexander Turenko]

LGTM, CCed Kirill.

WBR, Alexander Turenko.

On Tue, Sep 10, 2019 at 06:03:17PM +0300, Igor Munkin wrote:
> 
> Sasha,
> 
> Thanks for you review. 
> >Вторник, 10 сентября 2019, 16:58 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
> >
> >I found the following difference with the current master:
> >
> > | --- a/luajit-tap/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:45:04.886639461 +0300
> > | +++ b/test/fold_bug_LuaJIT_505.test.lua	2019-09-10 15:44:22.221641210 +0300
> > | @@ -8,7 +8,8 @@
> > |  -- Test file to demonstrate Lua fold machinery icorrect behavior, details:
> > |  --  https://github.com/LuaJIT/LuaJIT/issues/505
> > | 
> > | -jit.opt.start("hotloop=1")
> > | +jit.opt.start(0, "hotloop=1")
> > | +require('jit.dump').start("+bti", "505.trace")
> > |  for _ = 1, 20 do
> > |      local value = "abc"
> > |      local pos_c = string.find(value, "c", 1, true)
> >
> >From what it appears? If it is necessary, then move it to its own commit
> >or describe in the commit message. 
>
> The difference you found is an artefact left after investigation of the problem related to string.find recording. These changes are not required within this patch.

Okay.

> >
> >
> >Please, base your branch on top of a last commit in a default branch
> >(named 'tarantool'). Now it contains one extra commit ('luajit: fix
> >string.find recording').
> Another artefact.

Okay.

> >
> >
> >Verified in-source build, works ok.
> >
> >Tried out-of-source build: small and luajit-tap tests are not run (just
> >not found).
> >
> >If we'll create just symlinks in 'test' repository directory and will
> >remove creating symlinks from test/CMakeLists.txt, then luajit-tap would
> >work good, but small tests would not be built (and so small tests will
> >be found).
> >
> >It seems it is subject for a separate issue: don't sure about test-run
> >or tarantool. Something was changed in test-run I guess; maybe here:
> >https://github.com/tarantool/test-run/commit/b42093abc0e7fe895ff0ac7c0685d016c783b3a1
> >
> >Don't sure whether it worth to block on this problem or file an issue
> >and going on. Please, ask Kirill for that.

It is about the patch to tarantool more; let's skip it here.

Anyway, I filed https://github.com/tarantool/tarantool/issues/4485 to
investigate it deeper a bit later.

> >
> >Other then that everything look okay for me. 
> I've already fixed all mentioned flaws in the feature branch and forcely update the corresponding remote branch. Please inform me whether a new patch is required to be sent.

Okay.

> >On Thu, Sep 05, 2019 at 05:20:07PM +0300, Igor Munkin wrote:
> >> Introduced structure has the following benefits:
> >> * Excess testing machinery currently is not required directly in
> >> luajit repo considering whole CI process setup for tarantool and its
> >> static linking with libluajit
> >> * All new tests related to luajit fixes and enhancements can be added
> >> within a single patch with the corresponding changeset
> >> 
> >> NB!: All added test chunks use tarantool tap module
> >> ---
> >>  test/fold_bug_LuaJIT_505.test.lua        |  21 +++
> >>  test/gh.test.lua                         |  17 +++
> >>  test/suite.ini                           |   5 +
> >>  test/table_chain_bug_LuaJIT_494.test.lua | 178 +++++++++++++++++++++++
> >>  test/unsink_64_kptr.test.lua             |  44 ++++++
> >>  5 files changed, 265 insertions(+)
> >>  create mode 100755 test/fold_bug_LuaJIT_505.test.lua
> >>  create mode 100755 test/gh.test.lua
> >>  create mode 100644 test/suite.ini
> >>  create mode 100755 test/table_chain_bug_LuaJIT_494.test.lua
> >>  create mode 100755 test/unsink_64_kptr.test.lua
> 
> -- 
> IM