[Tarantool-patches] [PATCH luajit] Fix maxslots when recording BC_TSETM.

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

From: Mike Pall <mike>

Analyzed by Sergey Kaplun.

(cherry-picked from commit 0cc5fdfbc0810073485150eb184dc358dab507d9)

Recording of the `BC_TSETM` bytecode may keep too optimistic JIT
maxslot. In that case, the slot above the top of the Lua stack may be
considered used. When any VM event handler is called before the
recording of the next instruction, this leads to an assertion failure in
`rec_check_slots()`.

This patch sets the `ra` as a maxslot, as far as the `ra` - 1 contains a
table, which is always the highest slot after this bytecode. Also, it
adds an assertion that we check slots below the top of the Lua stack.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#8825
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1025-tsetm-maxslot
Tarantool PR: https://github.com/tarantool/tarantool/pull/9040
Issues:
* https://github.com/LuaJIT/LuaJIT/issues/1025
* https://github.com/tarantool/tarantool/issues/8825

 src/lj_record.c                               |  2 +
 .../lj-1025-tsetm-maxslot.test.lua            | 52 +++++++++++++++++++
 2 files changed, 54 insertions(+)
 create mode 100644 test/tarantool-tests/lj-1025-tsetm-maxslot.test.lua

diff --git a/src/lj_record.c b/src/lj_record.c
index 34d1210a..58b040ec 100644
--- a/src/lj_record.c
+++ b/src/lj_record.c
@@ -115,6 +115,7 @@ static void rec_check_slots(jit_State *J)
       cTValue *tv = &base[s];
       IRRef ref = tref_ref(tr);
       IRIns *ir = NULL;  /* Silence compiler. */
+      lj_assertJ(tv < J->L->top, "slot %d above top of Lua stack", s);
       if (!LJ_FR2 || ref || !(tr & (TREF_FRAME | TREF_CONT))) {
 	lj_assertJ(ref >= J->cur.nk && ref < J->cur.nins,
 		   "slot %d ref %04d out of range", s, ref - REF_BIAS);
@@ -2342,6 +2343,7 @@ void lj_record_ins(jit_State *J)
 
   case BC_TSETM:
     rec_tsetm(J, ra, (BCReg)(J->L->top - J->L->base), (int32_t)rcv->u32.lo);
+    J->maxslot = ra;  /* The table slot at ra-1 is the highest used slot. */
     break;
 
   case BC_TNEW:
diff --git a/test/tarantool-tests/lj-1025-tsetm-maxslot.test.lua b/test/tarantool-tests/lj-1025-tsetm-maxslot.test.lua
new file mode 100644
index 00000000..7ae0a99d
--- /dev/null
+++ b/test/tarantool-tests/lj-1025-tsetm-maxslot.test.lua
@@ -0,0 +1,52 @@
+local tap = require('tap')
+
+-- Test file to demonstrate LuaJIT incorrect recording of `TSETM`
+-- bytecode.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1025.
+
+local test = tap.test('lj-1025-tsetm-maxslot'):skipcond({
+  ['Test requires JIT enabled'] = not jit.status(),
+})
+
+test:plan(1)
+
+local jit_dump = require('jit.dump')
+
+local TEST_VALUE = '5'
+local TEST_IDX = 5
+
+local function slot5()
+  return nil, nil, nil, nil, TEST_VALUE
+end
+
+local storage
+local function test_tsetm(...)
+  -- Usage of `TSETM` bytecode.
+  storage = {slot5()}
+  -- Use this function again to trick use-def analysis and avoid
+  -- cleaning JIT slots, so the last JIT slot contains
+  -- `TEST_VALUE`.
+  return slot5(...)
+end
+
+-- Wrapper to avoid the recording of just the inner `slot5()`
+-- function.
+local function wrap()
+  test_tsetm()
+end
+
+jit.opt.start('hotloop=1')
+-- We need to call the VM event handler after each recorded bytecode
+-- instruction to pollute the Lua stack and the issue
+-- becomes observable.
+jit_dump.start('b', '/dev/null')
+
+-- Compile and execute the trace with `TSETM`.
+wrap()
+wrap()
+wrap()
+
+test:is(storage[TEST_IDX], TEST_VALUE,
+        'BC_TSETM recording with enabled jit.dump')
+
+test:done(true)
-- 
2.41.0