<!DOCTYPE html>
<html data-lt-installed="true">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body style="padding-bottom: 1px;">
    <p>Hi, Sergey,</p>
    <p>thanks for the patch! LGTM with a minor comment.</p>
    <p>Sergey<br>
    </p>
    <div class="moz-cite-prefix">On 8/19/25 10:40, Sergey Kaplun wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:20250819074020.12306-1-skaplun@tarantool.org">
      <pre wrap="" class="moz-quote-pre">From: Mike Pall <mike>

Reported by Assumeru.

(cherry picked from commit 5ca25ee83ec1b0343556cd5783ade449676b4037)

`lua_newstate()` sets `g->str.mask` to `~(MSize)0` before calling
`cpluaopen(). If OOM happens before the `lj_str_init()` call,
`lj_gc_freeall()` calls `gc_sweepstr()` in a loop with incorrect top
limit `g->str.mask`, which leads to the crash.

This patch changes the order of the loop iteration with the correct
bottom limit.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#11691
---

Branch: <a class="moz-txt-link-freetext" href="https://github.com/tarantool/luajit/tree/skaplun/lj-1248-close-state-early-OOM">https://github.com/tarantool/luajit/tree/skaplun/lj-1248-close-state-early-OOM</a>
Related issues:
* <a class="moz-txt-link-freetext" href="https://github.com/tarantool/tarantool/issues/11691">https://github.com/tarantool/tarantool/issues/11691</a>
* <a class="moz-txt-link-freetext" href="https://github.com/LuaJIT/LuaJIT/issues/1248">https://github.com/LuaJIT/LuaJIT/issues/1248</a>
* <a class="moz-txt-link-freetext" href="https://github.com/LuaJIT/LuaJIT/issues/1311">https://github.com/LuaJIT/LuaJIT/issues/1311</a>

Note: The test works for the GC64 build only, since we can't set a
custom allocator for the non-GC64 LJ_64 build. Also, to avoid failures
related to the lj-1311 the !LJ_NO_UNWIND builds are disabled.</pre>
    </blockquote>
    <p>Just for the record: CMake configuration for reproducing the
      issue:</p>
    <p>CFLAGS="-DLJ_NO_UNWIND" cmake -DCMAKE_C_COMPILER=clang
      -DCMAKE_CXX_COMPILER=clang++ -S . -B build
      -DCMAKE_BUILD_TYPE=Debug -DLUA_USE_ASSERT=ON -DLUA_USE_APICHECK=ON
      -DLUAJIT_ENABLE_GC64=ON</p>
    <blockquote type="cite"
      cite="mid:20250819074020.12306-1-skaplun@tarantool.org">
      <pre wrap="" class="moz-quote-pre">

 src/lj_gc.c                                   |  5 +-
 .../lj-1248-close-state-early-OOM.test.c      | 71 +++++++++++++++++++
 2 files changed, 73 insertions(+), 3 deletions(-)
 create mode 100644 test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c

diff --git a/src/lj_gc.c b/src/lj_gc.c
index f455b55b..3142482f 100644
--- a/src/lj_gc.c
+++ b/src/lj_gc.c
@@ -598,12 +598,11 @@ void lj_gc_finalize_cdata(lua_State *L)
 /* Free all remaining GC objects. */
 void lj_gc_freeall(global_State *g)
 {
-  MSize i, strmask;
+  MSize i;
   /* Free everything, except super-fixed objects (the main thread). */
   g->gc.currentwhite = LJ_GC_WHITES | LJ_GC_SFIXED;
   gc_fullsweep(g, &g->gc.root);
-  strmask = g->strmask;
-  for (i = 0; i <= strmask; i++)  /* Free all string hash chains. */
+  for (i = g->strmask; i != ~(MSize)0; i--)  /* Free all string hash chains. */
     gc_fullsweep(g, &g->strhash[i]);
 }
 
diff --git a/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c
new file mode 100644
index 00000000..6c9cb2ca
--- /dev/null
+++ b/test/tarantool-c-tests/lj-1248-close-state-early-OOM.test.c
@@ -0,0 +1,71 @@
+#include "lua.h"
+/* XXX: The "lj_arch.h" header is included for the skipcond. */
+#include "lj_arch.h"
+
+#include "test.h"
+
+#include <stdlib.h>
+
+/*
+ * LuaJIT requires at least 12000 something bytes for initial
+ * allocations. The `GG_State` requires a little bit more than
+ * 6000 bytes (around 3000 bytes is the `jit_State`).
+ */
+
+/* Currently allocated Lua memory and its limit. */
+static size_t current_memory = 0;
+const size_t memory_limit = 7000;
+
+void *limited_alloc_f(void *msp, void *ptr, size_t osize, size_t nsize)
+{
+       void *ret_ptr = NULL;
+       /* Overflow is OK here. */
+       const size_t requested_diff = nsize - osize;
+       (void)msp;
+
+       if (current_memory + requested_diff > memory_limit)
+               return NULL;
+
+       if (nsize == 0) {
+               free(ptr);
+               current_memory -= osize;
+       } else if (ptr == NULL) {
+               ret_ptr = malloc(nsize);
+               current_memory += ret_ptr ? nsize : 0;
+       } else {
+               ret_ptr = realloc(ptr, nsize);
+               current_memory += ret_ptr ? requested_diff : 0;
+       }
+       return ret_ptr;
+}
+
+static int limited_memory_on_lua_newstate(void *test_state)
+{
+       (void)test_state;
+#if LJ_64 && !LJ_GC64
+       (void)limited_alloc_f;
+       return skip("Can't use custom allocator for 64-bit host without GC64");
+#else
+       /*
+        * Check that there is no crash and the limit is small enough.
+        */
+       lua_State *L = lua_newstate(limited_alloc_f, NULL);</pre>
    </blockquote>
    s/lua_State/const lua_State/<br>
    <blockquote type="cite"
      cite="mid:20250819074020.12306-1-skaplun@tarantool.org">
      <pre wrap="" class="moz-quote-pre">
+       assert_true(L == NULL);
+       return TEST_EXIT_SUCCESS;
+#endif
+}
+
+#ifndef LJ_NO_UNWIND
+#  define LJ_NO_UNWIND 0
+#endif
+
+int main(void)
+{
+       /* See <a class="moz-txt-link-freetext" href="https://github.com/LuaJIT/LuaJIT/issues/1311">https://github.com/LuaJIT/LuaJIT/issues/1311</a>. */
+       if (!LJ_NO_UNWIND)
+               return skip_all("Disabled for external unwinding build due to #1311");
+       const struct test_unit tgroup[] = {
+               test_unit_def(limited_memory_on_lua_newstate),
+       };
+       return test_run_group(tgroup, NULL);
+}
</pre>
    </blockquote>
  </body>
  <lt-container></lt-container>
</html>