From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id DA8B11502A03; Tue, 9 Sep 2025 11:22:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org DA8B11502A03 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1757406124; bh=6DeXEWMU73n9f4nxxZb073hGOvBAjz32q211ksCL3SI=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=A5/SkLHCR1MjIPI1vZUlQ4ctoh0gJbvlGKPnstoHuAZlANDGF2k6AgXnum7GAWfji 0C5Idr5YhGRH4pFpLyXDXVb7mDB62z/vocPpQs3QSsYD+gmKSuvJV4uhacR4BZxlUh bsMt0hpsMp4iYkr4x5oovvEo2bez+pr7hC+nCfp0= Received: from send173.i.mail.ru (send173.i.mail.ru [95.163.59.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3C0CB1502A0C for ; Tue, 9 Sep 2025 11:22:02 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3C0CB1502A0C Received: by exim-smtp-c584fb9f-x4hfl with esmtpa (envelope-from ) id 1uvtc1-000000007R1-06Yn; Tue, 09 Sep 2025 11:22:01 +0300 Content-Type: multipart/alternative; boundary="------------a2W66xOCO6HLYO2K9uiakOPk" Message-ID: Date: Tue, 9 Sep 2025 11:22:00 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org References: <20250819074020.12306-1-skaplun@tarantool.org> In-Reply-To: <20250819074020.12306-1-skaplun@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91D98D60FB68F24CC656B8FA7074E8D364B6932C2F8108C53182A05F53808504044A06639F647A5333DE06ABAFEAF6705654E6B2D96777AA5AE920CBC47FB8628B8B780B664846F86 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE77633BACAB33B9508C2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6759CC434672EE6371C2A783ECEC0211ADC4224003CC836476D5A39DEEDB180909611E41BBFE2FEB2B1EC703003B3B9C761B3A00190A3E2F6B9EF0C3942132321FE688C0EA3765F4F89FA2833FD35BB23D9E625A9149C048EE1E561CDFBCA1751F2CC0D3CB04F14752D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8B1F8789D36234D406A471835C12D1D977C4224003CC836476EB9C4185024447017B076A6E789B0E975F5C1EE8F4F765FC6A4E49BB0F3BA1413AA81AA40904B5D9CF19DD082D7633A0C84D3B47A649675F3AA81AA40904B5D98AA50765F79006371C58C39218EE08BCD81D268191BDAD3D3666184CF4C3C14F3FC91FA280E0CE3D1A620F70A64A45A98AA50765F79006372E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89FB26E97DCB74E6252CE5475246E174218B5C8C57E37DE458BEDA766A37F9254B7 X-C1DE0DAB: 0D63561A33F958A5B7FBFC9EE6630C2B5002B1117B3ED69618E520488064253E54BB1175C6E7DD94823CB91A9FED034534781492E4B8EEADA91A6E18C88C5E2F X-C8649E89: 1C3962B70DF3F0ADE00A9FD3E00BEEDF3FED46C3ACD6F73ED3581295AF09D3DF87807E0823442EA2ED31085941D9CD0AF7F820E7B07EA4CF9AAEAB26D2D5FC3F0464D5DD74B8CE4602A9E318468088C7D02A4C07EC5AB7BF45AE5861559675ED1E832756A85E66F73D2E152D579A69E2797D9CFC44F38089F831FA868EFBC931111DC66A97D0BFE2913E6812662D5F2AB9AF64DB4688768036DF5FE9C0001AF333F2C28C22F508233FCF178C6DD14203 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVdVMtzNxwZu56UQjYzvH9N4= X-Mailru-Sender: 811C44EDE0507D1FE3AB2BB2D6096E35AF714AB457039CC3AA70CB78A12CDBB58EFF4ED1997C741E382A378CFB021EDC645D15D82EE4B272BD6E4642A116CA93524AA66B5ACBE6721EF430B9A63E2A504198E0F3ECE9B5443453F38A29522196 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit] Correctly close VM state after early OOM during open. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------a2W66xOCO6HLYO2K9uiakOPk Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi, Sergey, thanks for the patch! LGTM with a minor comment. Sergey On 8/19/25 10:40, Sergey Kaplun wrote: > From: Mike Pall > > Reported by Assumeru. > > (cherry picked from commit 5ca25ee83ec1b0343556cd5783ade449676b4037) > > `lua_newstate()` sets `g->str.mask` to `~(MSize)0` before calling > `cpluaopen(). If OOM happens before the `lj_str_init()` call, > `lj_gc_freeall()` calls `gc_sweepstr()` in a loop with incorrect top > limit `g->str.mask`, which leads to the crash. > > This patch changes the order of the loop iteration with the correct > bottom limit. > > Sergey Kaplun: > * added the description and the test for the problem > > Part of tarantool/tarantool#11691 > --- > > Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1248-close-state-early-OOM > Related issues: > *https://github.com/tarantool/tarantool/issues/11691 > *https://github.com/LuaJIT/LuaJIT/issues/1248 > *https://github.com/LuaJIT/LuaJIT/issues/1311 > > Note: The test works for the GC64 build only, since we can't set a > custom allocator for the non-GC64 LJ_64 build. Also, to avoid failures > related to the lj-1311 the !LJ_NO_UNWIND builds are disabled. Just for the record: CMake configuration for reproducing the issue: CFLAGS="-DLJ_NO_UNWIND" cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -S . -B build -DCMAKE_BUILD_TYPE=Debug -DLUA_USE_ASSERT=ON -DLUA_USE_APICHECK=ON -DLUAJIT_ENABLE_GC64=ON > > src/lj_gc.c | 5 +- > .../lj-1248-close-state-early-OOM.test.c | 71 +++++++++++++++++++ > 2 files changed, 73 insertions(+), 3 deletions(-) > create mode 100644 test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c > > diff --git a/src/lj_gc.c b/src/lj_gc.c > index f455b55b..3142482f 100644 > --- a/src/lj_gc.c > +++ b/src/lj_gc.c > @@ -598,12 +598,11 @@ void lj_gc_finalize_cdata(lua_State *L) > /* Free all remaining GC objects. */ > void lj_gc_freeall(global_State *g) > { > - MSize i, strmask; > + MSize i; > /* Free everything, except super-fixed objects (the main thread). */ > g->gc.currentwhite = LJ_GC_WHITES | LJ_GC_SFIXED; > gc_fullsweep(g, &g->gc.root); > - strmask = g->strmask; > - for (i = 0; i <= strmask; i++) /* Free all string hash chains. */ > + for (i = g->strmask; i != ~(MSize)0; i--) /* Free all string hash chains. */ > gc_fullsweep(g, &g->strhash[i]); > } > > diff --git a/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c > new file mode 100644 > index 00000000..6c9cb2ca > --- /dev/null > +++ b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c > @@ -0,0 +1,71 @@ > +#include "lua.h" > +/* XXX: The "lj_arch.h" header is included for the skipcond. */ > +#include "lj_arch.h" > + > +#include "test.h" > + > +#include > + > +/* > + * LuaJIT requires at least 12000 something bytes for initial > + * allocations. The `GG_State` requires a little bit more than > + * 6000 bytes (around 3000 bytes is the `jit_State`). > + */ > + > +/* Currently allocated Lua memory and its limit. */ > +static size_t current_memory = 0; > +const size_t memory_limit = 7000; > + > +void *limited_alloc_f(void *msp, void *ptr, size_t osize, size_t nsize) > +{ > + void *ret_ptr = NULL; > + /* Overflow is OK here. */ > + const size_t requested_diff = nsize - osize; > + (void)msp; > + > + if (current_memory + requested_diff > memory_limit) > + return NULL; > + > + if (nsize == 0) { > + free(ptr); > + current_memory -= osize; > + } else if (ptr == NULL) { > + ret_ptr = malloc(nsize); > + current_memory += ret_ptr ? nsize : 0; > + } else { > + ret_ptr = realloc(ptr, nsize); > + current_memory += ret_ptr ? requested_diff : 0; > + } > + return ret_ptr; > +} > + > +static int limited_memory_on_lua_newstate(void *test_state) > +{ > + (void)test_state; > +#if LJ_64 && !LJ_GC64 > + (void)limited_alloc_f; > + return skip("Can't use custom allocator for 64-bit host without GC64"); > +#else > + /* > + * Check that there is no crash and the limit is small enough. > + */ > + lua_State *L = lua_newstate(limited_alloc_f, NULL); s/lua_State/const lua_State/ > + assert_true(L == NULL); > + return TEST_EXIT_SUCCESS; > +#endif > +} > + > +#ifndef LJ_NO_UNWIND > +# define LJ_NO_UNWIND 0 > +#endif > + > +int main(void) > +{ > + /* Seehttps://github.com/LuaJIT/LuaJIT/issues/1311. */ > + if (!LJ_NO_UNWIND) > + return skip_all("Disabled for external unwinding build due to #1311"); > + const struct test_unit tgroup[] = { > + test_unit_def(limited_memory_on_lua_newstate), > + }; > + return test_run_group(tgroup, NULL); > +} --------------a2W66xOCO6HLYO2K9uiakOPk Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit

Hi, Sergey,

thanks for the patch! LGTM with a minor comment.

Sergey

On 8/19/25 10:40, Sergey Kaplun wrote:
From: Mike Pall <mike>

Reported by Assumeru.

(cherry picked from commit 5ca25ee83ec1b0343556cd5783ade449676b4037)

`lua_newstate()` sets `g->str.mask` to `~(MSize)0` before calling
`cpluaopen(). If OOM happens before the `lj_str_init()` call,
`lj_gc_freeall()` calls `gc_sweepstr()` in a loop with incorrect top
limit `g->str.mask`, which leads to the crash.

This patch changes the order of the loop iteration with the correct
bottom limit.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#11691
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1248-close-state-early-OOM
Related issues:
* https://github.com/tarantool/tarantool/issues/11691
* https://github.com/LuaJIT/LuaJIT/issues/1248
* https://github.com/LuaJIT/LuaJIT/issues/1311

Note: The test works for the GC64 build only, since we can't set a
custom allocator for the non-GC64 LJ_64 build. Also, to avoid failures
related to the lj-1311 the !LJ_NO_UNWIND builds are disabled.

Just for the record: CMake configuration for reproducing the issue:

CFLAGS="-DLJ_NO_UNWIND" cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -S . -B build -DCMAKE_BUILD_TYPE=Debug -DLUA_USE_ASSERT=ON -DLUA_USE_APICHECK=ON -DLUAJIT_ENABLE_GC64=ON


 src/lj_gc.c                                   |  5 +-
 .../lj-1248-close-state-early-OOM.test.c      | 71 +++++++++++++++++++
 2 files changed, 73 insertions(+), 3 deletions(-)
 create mode 100644 test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c

diff --git a/src/lj_gc.c b/src/lj_gc.c
index f455b55b..3142482f 100644
--- a/src/lj_gc.c
+++ b/src/lj_gc.c
@@ -598,12 +598,11 @@ void lj_gc_finalize_cdata(lua_State *L)
 /* Free all remaining GC objects. */
 void lj_gc_freeall(global_State *g)
 {
-  MSize i, strmask;
+  MSize i;
   /* Free everything, except super-fixed objects (the main thread). */
   g->gc.currentwhite = LJ_GC_WHITES | LJ_GC_SFIXED;
   gc_fullsweep(g, &g->gc.root);
-  strmask = g->strmask;
-  for (i = 0; i <= strmask; i++)  /* Free all string hash chains. */
+  for (i = g->strmask; i != ~(MSize)0; i--)  /* Free all string hash chains. */
     gc_fullsweep(g, &g->strhash[i]);
 }
 
diff --git a/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c
new file mode 100644
index 00000000..6c9cb2ca
--- /dev/null
+++ b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c
@@ -0,0 +1,71 @@
+#include "lua.h"
+/* XXX: The "lj_arch.h" header is included for the skipcond. */
+#include "lj_arch.h"
+
+#include "test.h"
+
+#include <stdlib.h>
+
+/*
+ * LuaJIT requires at least 12000 something bytes for initial
+ * allocations. The `GG_State` requires a little bit more than
+ * 6000 bytes (around 3000 bytes is the `jit_State`).
+ */
+
+/* Currently allocated Lua memory and its limit. */
+static size_t current_memory = 0;
+const size_t memory_limit = 7000;
+
+void *limited_alloc_f(void *msp, void *ptr, size_t osize, size_t nsize)
+{
+	void *ret_ptr = NULL;
+	/* Overflow is OK here. */
+	const size_t requested_diff = nsize - osize;
+	(void)msp;
+
+	if (current_memory + requested_diff > memory_limit)
+		return NULL;
+
+	if (nsize == 0) {
+		free(ptr);
+		current_memory -= osize;
+	} else if (ptr == NULL) {
+		ret_ptr = malloc(nsize);
+		current_memory += ret_ptr ? nsize : 0;
+	} else {
+		ret_ptr = realloc(ptr, nsize);
+		current_memory += ret_ptr ? requested_diff : 0;
+	}
+	return ret_ptr;
+}
+
+static int limited_memory_on_lua_newstate(void *test_state)
+{
+	(void)test_state;
+#if LJ_64 && !LJ_GC64
+	(void)limited_alloc_f;
+	return skip("Can't use custom allocator for 64-bit host without GC64");
+#else
+	/*
+	 * Check that there is no crash and the limit is small enough.
+	 */
+	lua_State *L = lua_newstate(limited_alloc_f, NULL);
s/lua_State/const lua_State/
+	assert_true(L == NULL);
+	return TEST_EXIT_SUCCESS;
+#endif
+}
+
+#ifndef LJ_NO_UNWIND
+#  define LJ_NO_UNWIND 0
+#endif
+
+int main(void)
+{
+	/* See https://github.com/LuaJIT/LuaJIT/issues/1311. */
+	if (!LJ_NO_UNWIND)
+		return skip_all("Disabled for external unwinding build due to #1311");
+	const struct test_unit tgroup[] = {
+		test_unit_def(limited_memory_on_lua_newstate),
+	};
+	return test_run_group(tgroup, NULL);
+}
--------------a2W66xOCO6HLYO2K9uiakOPk--