Hello, Sergey,

thanks for the patch!

LGTM with minor comment below.

Sergey

From: Mike Pall <mike>

Reported by Sergey Kaplun.

(cherry picked from commit 048972dbfdb6b441fe8a9bfe4d1f048966579ba8)

In the case when LuaJIT is recording the side trace after the
up-recursion call, there is no check that the updated `maxslot` value
doesn't overflow the `LJ_MAX_JSLOTS` limit. If it records several huge
returns in a row, the overflow of the aforementioned limit may occur.
This triggers an assertion failure in `rec_check_slots()`.

This patch fixes it by adding the corresponding check in the
`lj_record_ret()`.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#11278

---
Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1358-jslot-overflow-uprecursion
Related issues:
* https://github.com/tarantool/tarantool/issues/11278
* https://github.com/LuaJIT/LuaJIT/issues/1358

Also https://github.com/tarantool/security/issues/145.

<snipped>