[Tarantool-patches] [PATCH luajit] Rework stack overflow handling.

[Sergey Kaplun]

Hi, Sergey!
Thanks for the review!
Please, consider my answers below.

On 06.06.25, Sergey Bronnikov wrote:
> Hello, Sergey,
> 
> thanks for the patch! See my comments below.
> 
> Sergey
> 
> On 4/25/25 16:42, Sergey Kaplun wrote:
> > From: Mike Pall <mike>
> >
> > Reported by pwnhacker0x18. Fixed by Peter Cawley.
> >
> > (cherry picked from commit defe61a56751a0db5f00ff3ab7b8f45436ba74c8)
> >
> > In case of the Lua stack overflow error, LuaJIT restores the `L->top`
> > value and pushes the error message above. It is possible that the
> > restored value is greater than `L->maxstack`, so pushing the error
> > message causes dirty write out-of-bounds.
> >
> > This patch prevents it by overwriting stack overflow handling machinery.
> > Now, in the aforementioned case, the last frame is replaced with a dummy
> > frame to avoid dirty writes.
> >
> > Sergey Kaplun:
> > * added the description and the test for the problem
> >
> > Part of tarantool/tarantool#11278
> > ---
> >
> > Related issues:
> > *https://github.com/LuaJIT/LuaJIT/issues/1152
> > *https://github.com/tarantool/tarantool/issues/11278
> > Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1152-stack-buffer-overflow-on-error
> >
> > The CI is red due to CMake minimum required version in c-dt module in
> > Tarantool.
> These problems are already fixed, please re-push the branch.

Done.

> >
> > The cherry-pick also included a part fixed in the merge commit:
> > https://github.com/LuaJIT/LuaJIT/commit/0d313b243
> >
> >   src/lj_debug.c                                |  1 +
> >   src/lj_err.c                                  | 23 +++++-
> >   src/lj_err.h                                  |  1 +
> >   src/lj_state.c                                | 58 +++++++++-----
> >   test/LuaJIT-tests/lang/stackov.lua            |  8 +-
> >   ...52-stack-buffer-overflow-on-error.test.lua | 79 +++++++++++++++++++
> >   test/tarantool-tests/utils/CMakeLists.txt     |  1 +
> >   7 files changed, 148 insertions(+), 23 deletions(-)
> >   create mode 100644 test/tarantool-tests/lj-1152-stack-buffer-overflow-on-error.test.lua
> >

<snipped>

> > diff --git a/test/LuaJIT-tests/lang/stackov.lua b/test/LuaJIT-tests/lang/stackov.lua
> > index 8afa86b4..b052ad80 100644
> > --- a/test/LuaJIT-tests/lang/stackov.lua
> > +++ b/test/LuaJIT-tests/lang/stackov.lua
> I don't get why we need this change. With reverted patch this test is 
> passed.

With reverted -- yes. But after reworking the stack overflow handling,
it is possible that there is no place on the stack to call the error
handler (in that case, `debug.traceback`). Since this thing looks like
implementation-defined behaviour, I prefer to make the test less strict.

> > @@ -31,13 +31,17 @@ end
> >   do --- Base test.
> >     local err, s = xpcall(f, debug.traceback)
> >     assert(err == false)
> > -  test_error_msg(f, s)
> > +  -- There is no place on the stack to invoke the handler.
> > +  -- Just test the error reason.
> > +  assert(string.match(s, "stack overflow"))
> >   end
> >   
> >   do --- Stack overflow with non-empty arg list.
> >     local err, s = xpcall(g, debug.traceback, 1)
> >     assert(err == false)
> > -  test_error_msg(g, s)
> > +  -- There is no place on the stack to invoke the handler.
> > +  -- Just test the error reason.
> > +  assert(string.match(s, "stack overflow"))
> >   end
> >   
> >   do --- Vararg tail call with non-empty arg list. +slow

<snipped>

-- 
Best regards,
Sergey Kaplun