From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 30C186ECCD; Fri, 24 Jul 2026 11:41:51 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 30C186ECCD DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1784882511; bh=l2kGcbN9JkNw/1W6/k+s3XibMF2XvnDJXih+3wmCLwg=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=GtFZgqf3+p05Ac/oJoXCQEoYP6MK+qzxuvSS35+0cU1bNljgCGvI9W2tx6Pc+0pMr hSyWKJZLEkCoWaNu1UcuczOGDHvUDbHiSLLIS3R4RAuc22r8xxa+TNhWodgIjGegYI 9lSTRkvdydwwx0oB0vcvjag/zGkAmsieMmInuV7Q= Received: from send60.i.mail.ru (send60.i.mail.ru [89.221.237.155]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id D59926ECCD for ; Fri, 24 Jul 2026 11:41:49 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D59926ECCD Received: by exim-smtp-6c76488b9f-45q4v with esmtpa (envelope-from ) id 1wnBTY-00000000RJL-2Ga6; Fri, 24 Jul 2026 11:41:49 +0300 Content-Type: multipart/alternative; boundary="------------mOKT6v1i3bFUix7psOAHXoIC" Message-ID: <06a88784-87f9-4797-9344-8359d3bfb112@tarantool.org> Date: Fri, 24 Jul 2026 11:41:45 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Sergey Kaplun , Evgeniy Temirgaleev Cc: tarantool-patches@dev.tarantool.org References: <20260720085254.2452841-1-skaplun@tarantool.org> Content-Language: en-US In-Reply-To: <20260720085254.2452841-1-skaplun@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 78E4E2B564C1792B X-77F55803: 4F1203BC0FB41BD93ED3C6BF6FFA701F47C7C3D94A5A406F9166DAAD7C6D9BC8182A05F5380850408ABD19596B30FC4B3DE06ABAFEAF6705CBBB8F88E7985C3F638ED93D5B242B6393B27CBAA3F87F55 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE79683A3C835791080EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AC83A81C8FD4AD23D82A6BABE6F325AC2E85FA5F3EDFCBAA7353EFBB55337566DDF253B6BCADDB79D0FEB98E92D0BB472C238374FB93BFA8F6D7B8AC2D526703389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0A3E989B1926288338941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6E5E764EB5D94DBD4CC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB86D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7D6DD5572478B0BCD731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A56F927F994E0F45E45002B1117B3ED696799143F8415762557E0012C66AE17B00823CB91A9FED034534781492E4B8EEAD27E9584FBD6BDD31BDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE1918E10F71CB4DF9F96AB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D6595A67AB1083A81D53FB2B3F39D0850B22B37E97D7E6057BE325A17D9F32F180FE96AAD83DBE5DE612B8341EE9D5BE9A0A3EF40A4374963A4A83B94E9CFB13D666A7BFD1F985F0D69B6536EB022892E5344C41F94D744909CE2512F26BEC029E55448553D2254B8D95CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVWmxowtcrDwU1k/XhmqFeaM= X-DA7885C5: 4FFBE6E3AA2CA45CF255D290C0D534F9F431B4074CF4BFFB3770E1528A7A3AF0AFBD222BCD0A02345B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393520AF17B8A65FDE2F6938E34B93E358EA9231963D555C051E12D5E082A45EB80EF86D5F70DA33880E41E8EF7A07863ECB274557F927329BE2DDF8182D28ACDB545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit] Fix stack overflow relimit handling. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------mOKT6v1i3bFUix7psOAHXoIC Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hello, Thanks for the patch! LGTM Sergey On 7/20/26 11:52, Sergey Kaplun wrote: > From: Mike Pall > > Thanks to Sergey Kaplun. > > (cherry picked from commit d2c1327f57dc96f6ed8b11474f3bc5e09a9d227a) > > If the error handler can't fit to the top of the Lua stack, the stack is > resized up to `LJ_STACK_MAXEX + 1 + 2 * LUA_MINSTACK` hoping that > `lj_state_relimitstack()` shrinks it back. In case when another error is > (gently) handled in the error handler (`pcall(error)` or failed > `load()`) the stack is shrunk to `LJ_STACK_MAX` leaving the rest of the > handler to be executed on the clipped stack, which leads to the > heap-overflow or a crash. > > This patch fixes the issue by hardening the relimiting border to match > the extra space allocated before the stack overflow. Also, it fixes some > typos in the comments. > > Sergey Kaplun: > * added the description and the test for the problem > > Part of tarantool/tarantool#12880 > --- > > The test is leading to the core dump on GC64 build. > > Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1471-fix-stackov-relimit > Related issues: > *https://github.com/LuaJIT/LuaJIT/issues/1471 > *https://github.com/tarantool/tarantool/issues/12880 > > src/lj_state.c | 17 ++++--- > .../lj-1471-fix-stackov-relimit.test.lua | 48 +++++++++++++++++++ > 2 files changed, 59 insertions(+), 6 deletions(-) > create mode 100644 test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua > > diff --git a/src/lj_state.c b/src/lj_state.c > index 2fa6a2f6..ea0f24f8 100644 > --- a/src/lj_state.c > +++ b/src/lj_state.c > @@ -44,8 +44,9 @@ > #define LJ_STACK_MAX LUAI_MAXSTACK /* Max. stack size. */ > #define LJ_STACK_START (2*LJ_STACK_MIN) /* Starting stack size. */ > #define LJ_STACK_MAXEX (LJ_STACK_MAX + 1 + LJ_STACK_EXTRA) > +#define LJ_STACK_ERREX (1 + 2*LJ_STACK_MIN) /* Extra for error handling. */ > > -/* Explanation of LJ_STACK_EXTRA: > +/* Explanation for LJ_STACK_EXTRA: > ** > ** Calls to metamethods store their arguments beyond the current top > ** without checking for the stack limit. This avoids stack resizes which > @@ -58,6 +59,11 @@ > ** slots above top, but then mobj is always a function. So we can get by > ** with 5 extra slots. > ** LJ_FR2: We need 2 more slots for the frame PC and the continuation PC. > +** > +** Explanation for LJ_STACK_ERREX: > +** > +** The 1 is space for the error message, and 2 * LJ_STACK_MIN is for > +** the lj_state_checkstack() call in lj_err_run(). > */ > > /* Resize stack slots and adjust pointers in state. */ > @@ -101,7 +107,8 @@ static void resizestack(lua_State *L, MSize n) > /* Relimit stack after error, in case the limit was overdrawn. */ > void lj_state_relimitstack(lua_State *L) > { > - if (L->stacksize > LJ_STACK_MAXEX && L->top-tvref(L->stack) < LJ_STACK_MAX-1) > + if (L->stacksize > LJ_STACK_MAXEX && > + L->top-tvref(L->stack) < LJ_STACK_MAX - 1 - LJ_STACK_ERREX) > resizestack(L, LJ_STACK_MAX); > } > > @@ -147,11 +154,9 @@ void LJ_FASTCALL lj_state_growstack(lua_State *L, MSize need) > /* An error handler might want to inspect the stack overflow error, but > ** will need some stack space to run in. We give it a stack size beyond > ** the normal limit in order to do so, then rely on lj_state_relimitstack > - ** calls during unwinding to bring us back to a convential stack size. > - ** The + 1 is space for the error message, and 2 * LUA_MINSTACK is for > - ** the lj_state_checkstack() call in lj_err_run(). > + ** calls during unwinding to bring us back to a conventional stack size. > */ > - resizestack(L, LJ_STACK_MAX + 1 + 2 * LUA_MINSTACK); > + resizestack(L, LJ_STACK_MAX + LJ_STACK_ERREX); > lj_err_stkov(L); /* May invoke an error handler. */ > } else { > /* If we're here, then the stack overflow error handler is requesting > diff --git a/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua b/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua > new file mode 100644 > index 00000000..6a10cc20 > --- /dev/null > +++ b/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua > @@ -0,0 +1,48 @@ > +local tap = require('tap') > + > +-- The test file demonstrates a heap-overflow due to Lua stack > +-- out-of-bounds access after an incorrect stack shrinking after > +-- unwinding. > +-- See alsohttps://github.com/LuaJIT/LuaJIT/issues/1471. > + > +local test = tap.test('lj-1471-fix-stackov-relimit') > + > +test:plan(1) > + > +local function recursive_add(v1, v2) > + -- Slot to eat the stack. > + -- luacheck: no unused > + local _ > + recursive_add(v1, v2) > +end > + > +local table_mt = { > + __add = recursive_add, > +} > + > +local t1 = setmetatable({}, table_mt) > +local t2 = setmetatable({}, table_mt) > + > +coroutine.wrap(function() > + xpcall(error, function() > + pcall(error) -- Shrink stack back after unwinding. > + -- XXX: Empirical amount of stack slots to observe the issue. > + -- After the stack overflow, the `xpcall()` handler is invoked > + -- again (seehttps://github.com/LuaJIT/LuaJIT/issues/1382). > + -- The stack is overallocated beyond its normal limit to > + -- handle the error. After the `pcall(error)`, stack is > + -- shrinking back to its normal size, leaving no space for > + -- metamethod invocation. It is leaving to heap-overflow and a > + -- crash. > + -- luacheck: no unused > + local _, _, _, _, _, _, _, _, _, _ > + local _, _, _, _, _, _, _, _, _, _ > + local _, _, _, _, _, _, _, _, _, _ > + local _ > + local _ = t1 + t2 > + end) > +end)() > + > +test:ok(true, 'no heap overflow after stack relimiting') > + > +test:done(true) --------------mOKT6v1i3bFUix7psOAHXoIC Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit

Hello,

Thanks for the patch! LGTM

Sergey

On 7/20/26 11:52, Sergey Kaplun wrote:
From: Mike Pall <mike>

Thanks to Sergey Kaplun.

(cherry picked from commit d2c1327f57dc96f6ed8b11474f3bc5e09a9d227a)

If the error handler can't fit to the top of the Lua stack, the stack is
resized up to `LJ_STACK_MAXEX + 1 + 2 * LUA_MINSTACK` hoping that
`lj_state_relimitstack()` shrinks it back. In case when another error is
(gently) handled in the error handler (`pcall(error)` or failed
`load()`) the stack is shrunk to `LJ_STACK_MAX` leaving the rest of the
handler to be executed on the clipped stack, which leads to the
heap-overflow or a crash.

This patch fixes the issue by hardening the relimiting border to match
the extra space allocated before the stack overflow. Also, it fixes some
typos in the comments.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#12880
---

The test is leading to the core dump on GC64 build.

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1471-fix-stackov-relimit
Related issues:
* https://github.com/LuaJIT/LuaJIT/issues/1471
* https://github.com/tarantool/tarantool/issues/12880

 src/lj_state.c                                | 17 ++++---
 .../lj-1471-fix-stackov-relimit.test.lua      | 48 +++++++++++++++++++
 2 files changed, 59 insertions(+), 6 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua

diff --git a/src/lj_state.c b/src/lj_state.c
index 2fa6a2f6..ea0f24f8 100644
--- a/src/lj_state.c
+++ b/src/lj_state.c
@@ -44,8 +44,9 @@
 #define LJ_STACK_MAX	LUAI_MAXSTACK	/* Max. stack size. */
 #define LJ_STACK_START	(2*LJ_STACK_MIN)	/* Starting stack size. */
 #define LJ_STACK_MAXEX	(LJ_STACK_MAX + 1 + LJ_STACK_EXTRA)
+#define LJ_STACK_ERREX	(1 + 2*LJ_STACK_MIN)	/* Extra for error handling. */
 
-/* Explanation of LJ_STACK_EXTRA:
+/* Explanation for LJ_STACK_EXTRA:
 **
 ** Calls to metamethods store their arguments beyond the current top
 ** without checking for the stack limit. This avoids stack resizes which
@@ -58,6 +59,11 @@
 ** slots above top, but then mobj is always a function. So we can get by
 ** with 5 extra slots.
 ** LJ_FR2: We need 2 more slots for the frame PC and the continuation PC.
+**
+** Explanation for LJ_STACK_ERREX:
+**
+** The 1 is space for the error message, and 2 * LJ_STACK_MIN is for
+** the lj_state_checkstack() call in lj_err_run().
 */
 
 /* Resize stack slots and adjust pointers in state. */
@@ -101,7 +107,8 @@ static void resizestack(lua_State *L, MSize n)
 /* Relimit stack after error, in case the limit was overdrawn. */
 void lj_state_relimitstack(lua_State *L)
 {
-  if (L->stacksize > LJ_STACK_MAXEX && L->top-tvref(L->stack) < LJ_STACK_MAX-1)
+  if (L->stacksize > LJ_STACK_MAXEX &&
+      L->top-tvref(L->stack) < LJ_STACK_MAX - 1 - LJ_STACK_ERREX)
     resizestack(L, LJ_STACK_MAX);
 }
 
@@ -147,11 +154,9 @@ void LJ_FASTCALL lj_state_growstack(lua_State *L, MSize need)
       /* An error handler might want to inspect the stack overflow error, but
       ** will need some stack space to run in. We give it a stack size beyond
       ** the normal limit in order to do so, then rely on lj_state_relimitstack
-      ** calls during unwinding to bring us back to a convential stack size.
-      ** The + 1 is space for the error message, and 2 * LUA_MINSTACK is for
-      ** the lj_state_checkstack() call in lj_err_run().
+      ** calls during unwinding to bring us back to a conventional stack size.
       */
-      resizestack(L, LJ_STACK_MAX + 1 + 2 * LUA_MINSTACK);
+      resizestack(L, LJ_STACK_MAX + LJ_STACK_ERREX);
       lj_err_stkov(L);  /* May invoke an error handler. */
     } else {
       /* If we're here, then the stack overflow error handler is requesting
diff --git a/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua b/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua
new file mode 100644
index 00000000..6a10cc20
--- /dev/null
+++ b/test/tarantool-tests/lj-1471-fix-stackov-relimit.test.lua
@@ -0,0 +1,48 @@
+local tap = require('tap')
+
+-- The test file demonstrates a heap-overflow due to Lua stack
+-- out-of-bounds access after an incorrect stack shrinking after
+-- unwinding.
+-- See also https://github.com/LuaJIT/LuaJIT/issues/1471.
+
+local test = tap.test('lj-1471-fix-stackov-relimit')
+
+test:plan(1)
+
+local function recursive_add(v1, v2)
+  -- Slot to eat the stack.
+  -- luacheck: no unused
+  local _
+  recursive_add(v1, v2)
+end
+
+local table_mt = {
+  __add = recursive_add,
+}
+
+local t1 = setmetatable({}, table_mt)
+local t2 = setmetatable({}, table_mt)
+
+coroutine.wrap(function()
+  xpcall(error, function()
+    pcall(error) -- Shrink stack back after unwinding.
+    -- XXX: Empirical amount of stack slots to observe the issue.
+    -- After the stack overflow, the `xpcall()` handler is invoked
+    -- again (see https://github.com/LuaJIT/LuaJIT/issues/1382).
+    -- The stack is overallocated beyond its normal limit to
+    -- handle the error. After the `pcall(error)`, stack is
+    -- shrinking back to its normal size, leaving no space for
+    -- metamethod invocation. It is leaving to heap-overflow and a
+    -- crash.
+    -- luacheck: no unused
+    local _, _, _, _, _, _, _, _, _, _
+    local _, _, _, _, _, _, _, _, _, _
+    local _, _, _, _, _, _, _, _, _, _
+    local _
+    local _ = t1 + t2
+  end)
+end)()
+
+test:ok(true, 'no heap overflow after stack relimiting')
+
+test:done(true)
--------------mOKT6v1i3bFUix7psOAHXoIC--