Hi!
Thanks for the patch!
LGTM, except for a few nits regarding the commit message.
 
> 
>>From: Mike Pall <mike>
>>
>>Analyzed by Sergey Kaplun.
>>
>>(cherry-picked from commit 94ada59628dd6ce5d6d2dad1d35a68ad30127f53)
>>
>>While recording BC_VARG `J->maxslot` isn't shrunk to the effective stack
>Typo: s/shrunk/shrinking
>>top. This leads to dead value stored in the JIT slots and the following
>Typo: s/value/values/
>>assertion failure for these slots check in `rec_check_slots()`. Note,
>>that `rec_varg()` modifies `maxslot` only under the condition that
>>`maxslot` should be increased, but the dead values are left for the
>>opposite case.
>>
>>This patch removes the condition inside `rec_varg()` only for the case
>>when varargs are not defined on trace (`framedepth` is 0), but the
>>similar issue still occurs for the case when vararg are defined on the
>Typo: s/vararg/varagrs/
>>trace.
>>
>>Sergey Kaplun:
>>* added the description and the test for the problem
>>
>>Part of tarantool/tarantool#8825
>>---
>> src/lj_record.c | 3 +--
>> .../lj-1024-varg-maxslot.test.lua | 23 +++++++++++++++++++
>> 2 files changed, 24 insertions(+), 2 deletions(-)
>> create mode 100644 test/tarantool-tests/lj-1024-varg-maxslot.test.lua
>>
>>diff --git a/src/lj_record.c b/src/lj_record.c
>>index a90cba77..112524d3 100644
>>--- a/src/lj_record.c
>>+++ b/src/lj_record.c
>>@@ -1812,8 +1812,7 @@ static void rec_varg(jit_State *J, BCReg dst, ptrdiff_t nresults)
>>       }
>>       for (i = nvararg; i < nresults; i++)
>>  J->base[dst+i] = TREF_NIL;
>>- if (dst + (BCReg)nresults > J->maxslot)
>>- J->maxslot = dst + (BCReg)nresults;
>>+ J->maxslot = dst + (BCReg)nresults;
>>     } else if (select_detect(J)) { /* y = select(x, ...) */
>>       TRef tridx = J->base[dst-1];
>>       TRef tr = TREF_NIL;
>>diff --git a/test/tarantool-tests/lj-1024-varg-maxslot.test.lua b/test/tarantool-tests/lj-1024-varg-maxslot.test.lua
>>new file mode 100644
>>index 00000000..14270595
>>--- /dev/null
>>+++ b/test/tarantool-tests/lj-1024-varg-maxslot.test.lua
>>@@ -0,0 +1,23 @@
>>+local tap = require('tap')
>>+local test = tap.test('lj-noticket-varg-usedef'):skipcond({
>>+ ['Test requires JIT enabled'] = not jit.status(),
>>+})
>>+
>>+test:plan(1)
>>+
>>+jit.opt.start('hotloop=1')
>>+
>>+local counter = 0
>>+-- luacheck: ignore
>>+local anchor
>>+while counter < 3 do
>>+ counter = counter + 1
>>+ -- BC_VARG 5 1 0. `...` is nil (argument for the script).
>>+ -- luacheck: ignore
>>+ -- XXX: some condition to use several slots on the Lua stack.
>>+ anchor = 1 >= 1, ...
>>+end
>>+
>>+test:ok(true, 'BC_VARG recording 0th frame depth')
>>+
>>+os.exit(test:check() and 0 or 1)
>>--
>>2.34.1
>--
>Best regards,
>Maxim Kokryashkin