[Tarantool-patches] [PATCH luajit 0/3][v3] Fix stack overflow in pcall/xpcall

[Sergey Bronnikov via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

The proposed patches fixes stack overflow in pcall/xpcall.

Related issues:
- https://github.com/LuaJIT/LuaJIT/issues/1048
- https://github.com/LuaJIT/LuaJIT/issues/1402
- https://github.com/tarantool/tarantool/issues/12134

Git branch: https://github.com/tarantool/luajit/tree/ligurio/gh-xxxx-fix-stack-checks-in-vararg-calls

Changes in v2:
- Added patch for MIPS64
- Fixed issues reported on review

Changes in v3:
- Added a new file with test gh-1402-call_init-regression.test.lua
- Adjusted N_ITERATIONS in the last patch with appropriate comment
- Fixups after commentaries from Sergey Kaplun

Mike Pall (3):
  MIPS64: Fix xpcall() error case.
  LJ_FR2: Fix stack checks in vararg calls.
  Add stack check to pcall/xpcall.

 src/lj_def.h                                  |  2 +-
 src/lj_dispatch.c                             |  2 +-
 src/luajit-gdb.py                             |  2 +-
 src/vm_arm.dasc                               |  7 ++
 src/vm_arm64.dasc                             |  9 ++
 src/vm_mips.dasc                              | 10 ++-
 src/vm_mips64.dasc                            | 16 +++-
 src/vm_ppc.dasc                               |  9 ++
 src/vm_x64.dasc                               |  6 ++
 src/vm_x86.dasc                               |  6 ++
 .../gh-1402-call_init-regression.test.lua     | 36 ++++++++
 ...048-fix-stack-checks-vararg-calls.test.lua | 86 +++++++++++++++++++
 12 files changed, 184 insertions(+), 7 deletions(-)
 create mode 100644 test/tarantool-tests/gh-1402-call_init-regression.test.lua
 create mode 100644 test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

--
2.43.0