From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id CA87F1A3B247; Mon, 16 Mar 2026 13:44:53 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CA87F1A3B247 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1773657893; bh=ZZqBZHOzV4Ey7PpLVbd0ewEQ+yrj7eUPnQpoacGF5dU=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=XIukME5tI2uBEI4v/ZOoSyw2PrAUAWqsWmWHYbkCt1uWanREJOifPAHE4P8Z6PSJa DUHODdZEAQcbsP4llkBq58ioArpoFTOD0vqjnBaA2KzgOfqf6iRlU5UedlWAGIoo1A VZ/KixnN10OYSS278OGOJGSO409t70UuZPXUasKU= Received: from send172.i.mail.ru (send172.i.mail.ru [95.163.59.11]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id BFC511A3B245 for ; Mon, 16 Mar 2026 13:44:52 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BFC511A3B245 Received: by exim-smtp-64cdfc6c8d-dgtjr with esmtpa (envelope-from ) id 1w25RL-00000000I4m-0yzG; Mon, 16 Mar 2026 13:44:52 +0300 Content-Type: multipart/alternative; boundary="------------kzEm4J55gSNMzTgpY4d3kM0k" Message-ID: <43895852-2b40-4e97-a9d5-d158539ae643@tarantool.org> Date: Mon, 16 Mar 2026 13:44:50 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org References: <20260313112440.27237-1-skaplun@tarantool.org> In-Reply-To: <20260313112440.27237-1-skaplun@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91ABAE9865AC7DC884CC270DE33D3AA4DEF70278E8A0DC23E182A05F5380850405B37DA6AE856B2133DE06ABAFEAF67055E374EDFB3231161C87018ADC0F078E29910F280B933BCA3 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE75DF2B1F23425CAE5EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AC83A81C8FD4AD23D82A6BABE6F325AC2E85FA5F3EDFCBAA7353EFBB55337566BCD8AD057D84402CDD08F817D50E2EFA18A14C536B34187E5AF7819899A26F44389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0C26CFBAC0749D213D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BEA77C8EAE1CE44B0A471835C12D1D977C4224003CC8364762BB6847A3DEAEFB0F43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7CC415C329B279CF9D43847C11F186F3C59DAA53EE0834AAEE X-C1DE0DAB: 0D63561A33F958A5D00D02EC6F7DD06D5002B1117B3ED696401C9E8E0A0AC413219207EC0A953D2C823CB91A9FED034534781492E4B8EEAD21D4E6D365FE45D1BDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE191716CD42B3DD1D34CAB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D6593964BE9AFA80F8146A050E95259A5A4A986D2C263EA00FF66D6042257B6B0A512DE7AFE48A40DF62B8341EE9D5BE9A0A9CA96286FE118EF748D9083D45F3ECF6D710280CFFBA1ADA8CD93680B12512CF4C41F94D744909CE2512F26BEC029E55448553D2254B8D95CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVdbVVJCphTR/h+rKs7fT7FE= X-DA7885C5: AC400A92EDE0C09EF255D290C0D534F9C603250FB072CE3EBED08D9AA6AC7CD52050FE350260F3505B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393520AF17B8A65FDE275FAF059ABC23115484A957B55DC3E554AF5F80DA312CD79EF86D5F70DA33880E41E8EF7A07863ECB274557F927329BE2DDF8182D28ACDB545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit] Handle OOM error on stack resize in coroutine.resume and lua_checkstack. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------kzEm4J55gSNMzTgpY4d3kM0k Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit H, Sergey, thanks for the patch! LGTM Sergey On 3/13/26 14:24, Sergey Kaplun wrote: > From: Mike Pall > > Thanks to Peter Cawley. > > (cherry picked from commit a5d2f70c73e406beb617afa829a7af5b8c1d842c) > > If the OOM or stack overflow error is obtained during > `lj_state_growstack()` in the `coroutine.resume()` it may raise an error > leading to the incorrect error message and unexpected behaviour for > this corner case. Also, `lua_checkstack()` may raise an OOM error > leading to panic on an unprotected lua_State, for example. > > This patch handles these cases by protecting `lj_state_growstack()`. > > Sergey Kaplun: > * added the description and the test for the problem > > Part of tarantool/tarantool#12134 > --- > > Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1066-err-coroutine-resume > Related issues: > *https://github.com/LuaJIT/LuaJIT/issues/1066 > *https://github.com/tarantool/tarantool/issues/12134 > > src/lib_base.c | 5 +- > src/lj_api.c | 7 +- > src/lj_state.c | 12 +++ > src/lj_state.h | 1 + > .../lj-1066-err-lua-checkstack.test.c | 84 +++++++++++++++++++ > .../lj-1066-err-coroutine-resume.test.lua | 27 ++++++ > 6 files changed, 134 insertions(+), 2 deletions(-) > create mode 100644 test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c > create mode 100644 test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua > > diff --git a/src/lib_base.c b/src/lib_base.c > index 23728d6c..a5b907de 100644 > --- a/src/lib_base.c > +++ b/src/lib_base.c > @@ -606,7 +606,10 @@ static int ffh_resume(lua_State *L, lua_State *co, int wrap) > setstrV(L, L->base-LJ_FR2, lj_err_str(L, em)); > return FFH_RES(2); > } > - lj_state_growstack(co, (MSize)(L->top - L->base)); > + if (lj_state_cpgrowstack(co, (MSize)(L->top - L->base)) != LUA_OK) { > + cTValue *msg = --co->top; > + lj_err_callermsg(L, strVdata(msg)); > + } > return FFH_RETRY; > } > > diff --git a/src/lj_api.c b/src/lj_api.c > index 2e915306..16a1da0e 100644 > --- a/src/lj_api.c > +++ b/src/lj_api.c > @@ -116,7 +116,12 @@ LUA_API int lua_checkstack(lua_State *L, int size) > if (size > LUAI_MAXCSTACK || (L->top - L->base + size) > LUAI_MAXCSTACK) { > return 0; /* Stack overflow. */ > } else if (size > 0) { > - lj_state_checkstack(L, (MSize)size); > + int avail = (int)(mref(L->maxstack, TValue) - L->top); > + if (size > avail && > + lj_state_cpgrowstack(L, (MSize)(size - avail)) != LUA_OK) { > + L->top--; > + return 0; /* Out of memory. */ > + } > } > return 1; > } > diff --git a/src/lj_state.c b/src/lj_state.c > index 8a1acf87..1b698485 100644 > --- a/src/lj_state.c > +++ b/src/lj_state.c > @@ -170,6 +170,18 @@ void LJ_FASTCALL lj_state_growstack1(lua_State *L) > lj_state_growstack(L, 1); > } > > +static TValue *cpgrowstack(lua_State *co, lua_CFunction dummy, void *ud) > +{ > + UNUSED(dummy); > + lj_state_growstack(co, *(MSize *)ud); > + return NULL; > +} > + > +int LJ_FASTCALL lj_state_cpgrowstack(lua_State *L, MSize need) > +{ > + return lj_vm_cpcall(L, NULL, &need, cpgrowstack); > +} > + > /* Allocate basic stack for new state. */ > static void stack_init(lua_State *L1, lua_State *L) > { > diff --git a/src/lj_state.h b/src/lj_state.h > index 02a0eafa..cb3581d3 100644 > --- a/src/lj_state.h > +++ b/src/lj_state.h > @@ -18,6 +18,7 @@ LJ_FUNC void lj_state_relimitstack(lua_State *L); > LJ_FUNC void lj_state_shrinkstack(lua_State *L, MSize used); > LJ_FUNCA void LJ_FASTCALL lj_state_growstack(lua_State *L, MSize need); > LJ_FUNC void LJ_FASTCALL lj_state_growstack1(lua_State *L); > +LJ_FUNC int LJ_FASTCALL lj_state_cpgrowstack(lua_State *L, MSize need); > > static LJ_AINLINE void lj_state_checkstack(lua_State *L, MSize need) > { > diff --git a/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c b/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c > new file mode 100644 > index 00000000..d1cc8b06 > --- /dev/null > +++ b/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c > @@ -0,0 +1,84 @@ > +#include "lua.h" > +#include "luaconf.h" > + > +#include "test.h" > +#include "utils.h" > + > +#include > + > +/* XXX: Still need normal assert for sanity checks. */ > +#undef NDEBUG > +#include > + > +static lua_Alloc old_allocf = NULL; > +static void *old_alloc_state = NULL; > + > +/* Always OOM on reallocation (not on allocation). */ > +static void *allocf_null_realloc(void *ud, void *ptr, size_t osize, > + size_t nsize) > +{ > + assert(old_allocf != NULL); > + if (ptr != NULL && osize < nsize) > + return NULL; > + else > + return old_allocf(ud, ptr, osize, nsize); > +} > + > +static void enable_allocinject(lua_State *L) > +{ > + assert(old_allocf == NULL); > + old_allocf = lua_getallocf(L, &old_alloc_state); > + lua_setallocf(L, allocf_null_realloc, old_alloc_state); > +} > + > +/* Restore the default allocator function. */ > +static void disable_allocinject(lua_State *L) > +{ > + assert(old_allocf != NULL); > + lua_setallocf(L, old_allocf, old_alloc_state); > + old_allocf = NULL; > + old_alloc_state = NULL; > +} > + > +static int checkstack_res = -1; > + > +static int test_checkstack(lua_State *L) > +{ > + /* > + * There is no stack overflow error, but the OOM error > + * due to stack reallocation, before the patch. > + */ > + checkstack_res = lua_checkstack(L, LUAI_MAXCSTACK / 2); > + return 0; > +} > + > +static int oom_on_lua_checkstack(void *test_state) > +{ > + lua_State *L = test_state; > + /* Use fresh-new coroutine for stack manipulations. */ > + lua_State *L1 = lua_newthread(L); > + > + enable_allocinject(L); > + /* > + * `L1` should have enough space to `cpcall()` without > + * stack reallocation. > + */ > + int status = lua_cpcall(L1, test_checkstack, NULL); > + disable_allocinject(L); > + > + assert_true(status == LUA_OK); > + assert_true(checkstack_res == 0); > + > + return TEST_EXIT_SUCCESS; > +} > + > +int main(void) > +{ > + lua_State *L = utils_lua_init(); > + const struct test_unit tgroup[] = { > + test_unit_def(oom_on_lua_checkstack), > + }; > + const int test_result = test_run_group(tgroup, L); > + utils_lua_close(L); > + return test_result; > +} > diff --git a/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua b/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua > new file mode 100644 > index 00000000..3324c351 > --- /dev/null > +++ b/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua > @@ -0,0 +1,27 @@ > +local tap = require('tap') > + > +-- Test file to demonstrate LuaJIT misbehaviour in case of error > +-- on growing stack for the coroutine during `coroutine.resume()`. > +-- See also:https://github.com/LuaJIT/LuaJIT/issues/1066. > + > +local test = tap.test('lj-1066-err-coroutine-resume') > + > +test:plan(2) > + > +local function resume_wrap() > + local function recurser() > + recurser(coroutine.yield()) > + end > + local co = coroutine.create(recurser) > + -- Cause the stack overflow and throws an error with incorrect > + -- error message before the patch. Use some arguments to obtain > + -- the stack overflow faster. > + while coroutine.resume(co, 1, 2, 3, 4, 5, 6, 7, 8, 9) do end > +end > + > +local status, errmsg = pcall(resume_wrap) > + > +test:is(status, false, 'status is correct') > +test:like(errmsg, 'stack overflow', 'error message is correct') > + > +test:done(true) --------------kzEm4J55gSNMzTgpY4d3kM0k Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit

H, Sergey,

thanks for the patch! LGTM

Sergey

On 3/13/26 14:24, Sergey Kaplun wrote:
From: Mike Pall <mike>

Thanks to Peter Cawley.

(cherry picked from commit a5d2f70c73e406beb617afa829a7af5b8c1d842c)

If the OOM or stack overflow error is obtained during
`lj_state_growstack()` in the `coroutine.resume()` it may raise an error
leading to the incorrect error message and unexpected behaviour for
this corner case. Also, `lua_checkstack()` may raise an OOM error
leading to panic on an unprotected lua_State, for example.

This patch handles these cases by protecting `lj_state_growstack()`.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#12134
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1066-err-coroutine-resume
Related issues:
* https://github.com/LuaJIT/LuaJIT/issues/1066
* https://github.com/tarantool/tarantool/issues/12134

 src/lib_base.c                                |  5 +-
 src/lj_api.c                                  |  7 +-
 src/lj_state.c                                | 12 +++
 src/lj_state.h                                |  1 +
 .../lj-1066-err-lua-checkstack.test.c         | 84 +++++++++++++++++++
 .../lj-1066-err-coroutine-resume.test.lua     | 27 ++++++
 6 files changed, 134 insertions(+), 2 deletions(-)
 create mode 100644 test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c
 create mode 100644 test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua

diff --git a/src/lib_base.c b/src/lib_base.c
index 23728d6c..a5b907de 100644
--- a/src/lib_base.c
+++ b/src/lib_base.c
@@ -606,7 +606,10 @@ static int ffh_resume(lua_State *L, lua_State *co, int wrap)
     setstrV(L, L->base-LJ_FR2, lj_err_str(L, em));
     return FFH_RES(2);
   }
-  lj_state_growstack(co, (MSize)(L->top - L->base));
+  if (lj_state_cpgrowstack(co, (MSize)(L->top - L->base)) != LUA_OK) {
+    cTValue *msg = --co->top;
+    lj_err_callermsg(L, strVdata(msg));
+  }
   return FFH_RETRY;
 }
 
diff --git a/src/lj_api.c b/src/lj_api.c
index 2e915306..16a1da0e 100644
--- a/src/lj_api.c
+++ b/src/lj_api.c
@@ -116,7 +116,12 @@ LUA_API int lua_checkstack(lua_State *L, int size)
   if (size > LUAI_MAXCSTACK || (L->top - L->base + size) > LUAI_MAXCSTACK) {
     return 0;  /* Stack overflow. */
   } else if (size > 0) {
-    lj_state_checkstack(L, (MSize)size);
+    int avail = (int)(mref(L->maxstack, TValue) - L->top);
+    if (size > avail &&
+	lj_state_cpgrowstack(L, (MSize)(size - avail)) != LUA_OK) {
+      L->top--;
+      return 0;  /* Out of memory. */
+    }
   }
   return 1;
 }
diff --git a/src/lj_state.c b/src/lj_state.c
index 8a1acf87..1b698485 100644
--- a/src/lj_state.c
+++ b/src/lj_state.c
@@ -170,6 +170,18 @@ void LJ_FASTCALL lj_state_growstack1(lua_State *L)
   lj_state_growstack(L, 1);
 }
 
+static TValue *cpgrowstack(lua_State *co, lua_CFunction dummy, void *ud)
+{
+  UNUSED(dummy);
+  lj_state_growstack(co, *(MSize *)ud);
+  return NULL;
+}
+
+int LJ_FASTCALL lj_state_cpgrowstack(lua_State *L, MSize need)
+{
+  return lj_vm_cpcall(L, NULL, &need, cpgrowstack);
+}
+
 /* Allocate basic stack for new state. */
 static void stack_init(lua_State *L1, lua_State *L)
 {
diff --git a/src/lj_state.h b/src/lj_state.h
index 02a0eafa..cb3581d3 100644
--- a/src/lj_state.h
+++ b/src/lj_state.h
@@ -18,6 +18,7 @@ LJ_FUNC void lj_state_relimitstack(lua_State *L);
 LJ_FUNC void lj_state_shrinkstack(lua_State *L, MSize used);
 LJ_FUNCA void LJ_FASTCALL lj_state_growstack(lua_State *L, MSize need);
 LJ_FUNC void LJ_FASTCALL lj_state_growstack1(lua_State *L);
+LJ_FUNC int LJ_FASTCALL lj_state_cpgrowstack(lua_State *L, MSize need);
 
 static LJ_AINLINE void lj_state_checkstack(lua_State *L, MSize need)
 {
diff --git a/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c b/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c
new file mode 100644
index 00000000..d1cc8b06
--- /dev/null
+++ b/test/tarantool-c-tests/lj-1066-err-lua-checkstack.test.c
@@ -0,0 +1,84 @@
+#include "lua.h"
+#include "luaconf.h"
+
+#include "test.h"
+#include "utils.h"
+
+#include <stdlib.h>
+
+/* XXX: Still need normal assert for sanity checks. */
+#undef NDEBUG
+#include <assert.h>
+
+static lua_Alloc old_allocf = NULL;
+static void *old_alloc_state = NULL;
+
+/* Always OOM on reallocation (not on allocation). */
+static void *allocf_null_realloc(void *ud, void *ptr, size_t osize,
+				 size_t nsize)
+{
+	assert(old_allocf != NULL);
+	if (ptr != NULL && osize < nsize)
+		return NULL;
+	else
+		return old_allocf(ud, ptr, osize, nsize);
+}
+
+static void enable_allocinject(lua_State *L)
+{
+	assert(old_allocf == NULL);
+	old_allocf = lua_getallocf(L, &old_alloc_state);
+	lua_setallocf(L, allocf_null_realloc, old_alloc_state);
+}
+
+/* Restore the default allocator function. */
+static void disable_allocinject(lua_State *L)
+{
+	assert(old_allocf != NULL);
+	lua_setallocf(L, old_allocf, old_alloc_state);
+	old_allocf = NULL;
+	old_alloc_state = NULL;
+}
+
+static int checkstack_res = -1;
+
+static int test_checkstack(lua_State *L)
+{
+	/*
+	 * There is no stack overflow error, but the OOM error
+	 * due to stack reallocation, before the patch.
+	 */
+	checkstack_res = lua_checkstack(L, LUAI_MAXCSTACK / 2);
+	return 0;
+}
+
+static int oom_on_lua_checkstack(void *test_state)
+{
+	lua_State *L = test_state;
+	/* Use fresh-new coroutine for stack manipulations. */
+	lua_State *L1 = lua_newthread(L);
+
+	enable_allocinject(L);
+	/*
+	 * `L1` should have enough space to `cpcall()` without
+	 * stack reallocation.
+	 */
+	int status = lua_cpcall(L1, test_checkstack, NULL);
+	disable_allocinject(L);
+
+	assert_true(status == LUA_OK);
+	assert_true(checkstack_res == 0);
+
+	return TEST_EXIT_SUCCESS;
+}
+
+int main(void)
+{
+	lua_State *L = utils_lua_init();
+	const struct test_unit tgroup[] = {
+		test_unit_def(oom_on_lua_checkstack),
+	};
+	const int test_result = test_run_group(tgroup, L);
+	utils_lua_close(L);
+	return test_result;
+}
diff --git a/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua b/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua
new file mode 100644
index 00000000..3324c351
--- /dev/null
+++ b/test/tarantool-tests/lj-1066-err-coroutine-resume.test.lua
@@ -0,0 +1,27 @@
+local tap = require('tap')
+
+-- Test file to demonstrate LuaJIT misbehaviour in case of error
+-- on growing stack for the coroutine during `coroutine.resume()`.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1066.
+
+local test = tap.test('lj-1066-err-coroutine-resume')
+
+test:plan(2)
+
+local function resume_wrap()
+  local function recurser()
+    recurser(coroutine.yield())
+  end
+  local co = coroutine.create(recurser)
+  -- Cause the stack overflow and throws an error with incorrect
+  -- error message before the patch. Use some arguments to obtain
+  -- the stack overflow faster.
+  while coroutine.resume(co, 1, 2, 3, 4, 5, 6, 7, 8, 9) do end
+end
+
+local status, errmsg = pcall(resume_wrap)
+
+test:is(status, false, 'status is correct')
+test:like(errmsg, 'stack overflow', 'error message is correct')
+
+test:done(true)
--------------kzEm4J55gSNMzTgpY4d3kM0k--