[Tarantool-patches] [PATCH luajit 1/2] Fix maxslots when recording BC_VARG.

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

From: Mike Pall <mike>

Analyzed by Sergey Kaplun.

(cherry-picked from commit 94ada59628dd6ce5d6d2dad1d35a68ad30127f53)

While recording BC_VARG `J->maxslot` isn't shrunk to the effective stack
top. This leads to dead value stored in the JIT slots and the following
assertion failure for these slots check in `rec_check_slots()`. Note,
that `rec_varg()` modifies `maxslot` only under the condition that
`maxslot` should be increased, but the dead values are left for the
opposite case.

This patch removes the condition inside `rec_varg()` only for the case
when varargs are not defined on trace (`framedepth` is 0), but the
similar issue still occurs for the case when vararg are defined on the
trace.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#8825
---
 src/lj_record.c                               |  3 +--
 .../lj-1024-varg-maxslot.test.lua             | 23 +++++++++++++++++++
 2 files changed, 24 insertions(+), 2 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1024-varg-maxslot.test.lua

diff --git a/src/lj_record.c b/src/lj_record.c
index a90cba77..112524d3 100644
--- a/src/lj_record.c
+++ b/src/lj_record.c
@@ -1812,8 +1812,7 @@ static void rec_varg(jit_State *J, BCReg dst, ptrdiff_t nresults)
       }
       for (i = nvararg; i < nresults; i++)
 	J->base[dst+i] = TREF_NIL;
-      if (dst + (BCReg)nresults > J->maxslot)
-	J->maxslot = dst + (BCReg)nresults;
+      J->maxslot = dst + (BCReg)nresults;
     } else if (select_detect(J)) {  /* y = select(x, ...) */
       TRef tridx = J->base[dst-1];
       TRef tr = TREF_NIL;
diff --git a/test/tarantool-tests/lj-1024-varg-maxslot.test.lua b/test/tarantool-tests/lj-1024-varg-maxslot.test.lua
new file mode 100644
index 00000000..14270595
--- /dev/null
+++ b/test/tarantool-tests/lj-1024-varg-maxslot.test.lua
@@ -0,0 +1,23 @@
+local tap = require('tap')
+local test = tap.test('lj-noticket-varg-usedef'):skipcond({
+  ['Test requires JIT enabled'] = not jit.status(),
+})
+
+test:plan(1)
+
+jit.opt.start('hotloop=1')
+
+local counter = 0
+-- luacheck: ignore
+local anchor
+while counter < 3 do
+  counter = counter + 1
+  -- BC_VARG 5 1 0. `...` is nil (argument for the script).
+  -- luacheck: ignore
+  -- XXX: some condition to use several slots on the Lua stack.
+  anchor = 1 >= 1, ...
+end
+
+test:ok(true, 'BC_VARG recording 0th frame depth')
+
+os.exit(test:check() and 0 or 1)
-- 
2.34.1