[Tarantool-patches] [PATCH luajit] Rework stack overflow handling.

[Sergey Kaplun]

Sergey,

On 06.06.25, Sergey Bronnikov wrote:
> Hello,
> 
> LGTM with a comment below.
> 
> Sergey
> 
> On 6/6/25 16:40, Sergey Kaplun wrote:
> 
> <snipped>
> 
> >>> diff --git a/test/LuaJIT-tests/lang/stackov.lua b/test/LuaJIT-tests/lang/stackov.lua
> >>> index 8afa86b4..b052ad80 100644
> >>> --- a/test/LuaJIT-tests/lang/stackov.lua
> >>> +++ b/test/LuaJIT-tests/lang/stackov.lua
> >> I don't get why we need this change. With reverted patch this test is
> >> passed.
> > With reverted -- yes. But after reworking the stack overflow handling,
> > it is possible that there is no place on the stack to call the error
> > handler (in that case, `debug.traceback`). Since this thing looks like
> > implementation-defined behaviour, I prefer to make the test less strict.
> >
> Please add a message to the test or/and to the commit message.

Added the note to the commit message. Now it looks like the following:
| Rework stack overflow handling.
|
| Reported by pwnhacker0x18. Fixed by Peter Cawley.
|
| (cherry picked from commit defe61a56751a0db5f00ff3ab7b8f45436ba74c8)
|
| In case of the Lua stack overflow error, LuaJIT restores the `L->top`
| value and pushes the error message above. It is possible that the
| restored value is greater than `L->maxstack`, so pushing the error
| message causes dirty write out-of-bounds.
|
| This patch prevents it by overwriting stack overflow handling machinery.
| Now, in the aforementioned case, the last frame is replaced with a dummy
| frame to avoid dirty writes. In some cases, there may not be enough
| space on the stack to invoke the error handler. See the related changes
| in the <test/LuaJIT-tests/lang/stackov.lua>.
|
| Sergey Kaplun:
| * added the description and the test for the problem
|
| Part of tarantool/tarantool#11278


> >>> @@ -31,13 +31,17 @@ end
> >>>    do --- Base test.
> >>>      local err, s = xpcall(f, debug.traceback)
> >>>      assert(err == false)
> >>> -  test_error_msg(f, s)
> >>> +  -- There is no place on the stack to invoke the handler.
> >>> +  -- Just test the error reason.
> >>> +  assert(string.match(s, "stack overflow"))
> >>>    end
> >>>    
> >>>    do --- Stack overflow with non-empty arg list.
> >>>      local err, s = xpcall(g, debug.traceback, 1)
> >>>      assert(err == false)
> >>> -  test_error_msg(g, s)
> >>> +  -- There is no place on the stack to invoke the handler.
> >>> +  -- Just test the error reason.
> >>> +  assert(string.match(s, "stack overflow"))
> >>>    end
> >>>    
> >>>    do --- Vararg tail call with non-empty arg list. +slow
> > <snipped>
> >

-- 
Best regards,
Sergey Kaplun