[Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

Reported by Sergey Kaplun.

(cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)

The raising of the OOM error when rehashing the finalizer table (when we
can't allocate a new hash part) leads to crashes in either
`lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
raising, which either has no DWARF eh_frame or loses the context of the
JIT compiler.

This patch drops rehashing of the finalizer table to avoid these
crashes.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#10199
Resolves tarantool/tarantool#10290
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1247-fin-tab-rehashing-on-trace
Related Issues:
* https://github.com/tarantool/tarantool/issues/10290
* https://github.com/LuaJIT/LuaJIT/issues/1247
* https://github.com/tarantool/tarantool/issues/10199

 src/lj_gc.c                                   |   7 -
 src/lj_obj.h                                  |   2 +-
 test/tarantool-tests/CMakeLists.txt           |   1 +
 ...j-1247-fin-tab-rehashing-on-trace.test.lua | 127 ++++++++++++++++++
 .../CMakeLists.txt                            |   1 +
 .../lj_1247_allocinject.c                     |  49 +++++++
 6 files changed, 179 insertions(+), 8 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
 create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
 create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c

diff --git a/src/lj_gc.c b/src/lj_gc.c
index 4c222f21..a2fc93a0 100644
--- a/src/lj_gc.c
+++ b/src/lj_gc.c
@@ -548,7 +548,6 @@ static void gc_finalize(lua_State *L)
     setcdataV(L, &tmp, gco2cd(o));
     tv = lj_tab_set(L, tabref(g->gcroot[GCROOT_FFI_FIN]), &tmp);
     if (!tvisnil(tv)) {
-      g->gc.nocdatafin = 0;
       copyTV(L, &tmp, tv);
       setnilV(tv);  /* Clear entry in finalizer table. */
       gc_call_finalizer(g, L, &tmp, o);
@@ -694,9 +693,6 @@ static size_t gc_onestep(lua_State *L)
 	lj_str_resize(L, g->strmask >> 1);  /* Shrink string table. */
       if (gcref(g->gc.mmudata)) {  /* Need any finalizations? */
 	g->gc.state = GCSfinalize;
-#if LJ_HASFFI
-	g->gc.nocdatafin = 1;
-#endif
       } else {  /* Otherwise skip this phase to help the JIT. */
 	g->gc.state = GCSpause;  /* End of GC cycle. */
 	g->gc.debt = 0;
@@ -713,9 +709,6 @@ static size_t gc_onestep(lua_State *L)
 	g->gc.estimate -= GCFINALIZECOST;
       return GCFINALIZECOST;
     }
-#if LJ_HASFFI
-    if (!g->gc.nocdatafin) lj_tab_rehash(L, tabref(g->gcroot[GCROOT_FFI_FIN]));
-#endif
     g->gc.state = GCSpause;  /* End of GC cycle. */
     g->gc.debt = 0;
     return 0;
diff --git a/src/lj_obj.h b/src/lj_obj.h
index 06ea0cd0..ff22e5f8 100644
--- a/src/lj_obj.h
+++ b/src/lj_obj.h
@@ -611,7 +611,7 @@ typedef struct GCState {
   GCSize threshold;	/* Memory threshold. */
   uint8_t currentwhite;	/* Current white color. */
   uint8_t state;	/* GC state. */
-  uint8_t nocdatafin;	/* No cdata finalizer called. */
+  uint8_t unused0;
 #if LJ_64
   uint8_t lightudnum;	/* Number of lightuserdata segments - 1. */
 #else
diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt
index e3750bf3..e5d5a470 100644
--- a/test/tarantool-tests/CMakeLists.txt
+++ b/test/tarantool-tests/CMakeLists.txt
@@ -37,6 +37,7 @@ add_subdirectory(lj-flush-on-trace)
 add_subdirectory(lj-1004-oom-error-frame)
 add_subdirectory(lj-1066-fix-cur_L-after-coroutine-resume)
 add_subdirectory(lj-1166-error-stitch)
+add_subdirectory(lj-1247-fin-tab-rehashing-on-trace)
 
 # The part of the memory profiler toolchain is located in tools
 # directory, jit, profiler, and bytecode toolchains are located
diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
new file mode 100644
index 00000000..308043a2
--- /dev/null
+++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
@@ -0,0 +1,127 @@
+local tap = require('tap')
+
+-- The test file to demonstrate the incorrect JIT behaviour during
+-- OOM on the finalizer table rehashing in the context of the JIT
+-- trace.
+-- See also:
+-- * https://github.com/LuaJIT/LuaJIT/issues/1247,
+-- * https://github.com/tarantool/tarantool/issues/10290.
+
+local test = tap.test('lj-1247-fin-tab-rehashing-on-trace'):skipcond({
+  ['Broken unwiding in tarantool_panic_handler'] = _TARANTOOL and
+                                                   (jit.os == 'OSX'),
+  ['Disabled on MacOS due to #8652'] = jit.os == 'OSX',
+  ['Test requires JIT enabled'] = not jit.status(),
+})
+
+-- XXX: The original issue has 2 ways to crash:
+-- 1) in `lj_trace_unwind()`
+-- 2) in `lj_trace_exit()`
+-- But, since we have an additional GC pressure due to requiring a
+-- `tap` module, the second case needs an impossibly big
+-- `gcstepmul` value to reproduce the issue. So, since the root
+-- issue is the same and now rehashing of finalizer table is
+-- omitted, we test only the first case.
+test:plan(2)
+
+local allocinject = require('lj_1247_allocinject')
+
+local ffi = require('ffi')
+ffi.cdef[[
+  struct test {int a;};
+]]
+
+local N_GC_STEPS = 100
+local N_GC_FINALIZERS = 100
+
+local function empty() end
+
+-- Create a chunk like the following:
+--[[
+  local tostring = tostring
+  local r = ...
+  for _ = 1, 4 do
+    r[1] = tostring(1)
+    -- ...
+    r[N_GCSTEPS] = tostring(N_GC_STEPS)
+  end
+--]]
+local function create_chunk(n_steps)
+  local chunk = 'local tostring = tostring\n'
+  chunk = chunk .. ('local r = ...\n')
+  chunk = chunk .. 'for _ = 1, 4 do\n'
+  for i = 1, n_steps do
+    chunk = chunk .. ('  r[%d] = tostring(%d)\n'):format(i, i)
+  end
+  chunk = chunk .. 'end\n'
+  chunk = chunk .. 'return r\n'
+  return chunk
+end
+
+local function add_more_garbage(size)
+  return ffi.new('char[?]', size)
+end
+
+-- Helper to skip the atomic phase.
+local function skip_atomic()
+  local first_gc_called = false
+  local function mark_fin() first_gc_called = true end
+  jit.off(mark_fin)
+  debug.getmetatable(newproxy(true)).__gc = mark_fin
+
+  -- Skip the atomic phase.
+  jit.off()
+  while not first_gc_called do collectgarbage('step') end
+  jit.on()
+end
+
+local function crash_on_trace_unwind_gc_setup()
+  skip_atomic()
+  collectgarbage('setstepmul', 1000)
+  add_more_garbage(1024 * 1024)
+end
+
+local f = assert(loadstring(create_chunk(N_GC_STEPS)))
+
+-- Create a really long trace.
+jit.flush()
+jit.opt.start('hotloop=2', 'maxirconst=5000', 'maxrecord=10000', 'maxsnap=1000',
+              '-fold')
+
+-- luacheck: no unused
+local gc_anchor = {}
+local function anchor_finalizer(i)
+  gc_anchor[i] = ffi.gc(ffi.new('struct test', i), empty)
+end
+
+for i = 1, N_GC_FINALIZERS do
+  anchor_finalizer(i)
+end
+
+-- Record the trace first.
+f({})
+
+-- The table for anchoring cdata objects.
+local res_tab = {}
+
+collectgarbage()
+collectgarbage()
+collectgarbage('setpause', 0)
+collectgarbage('setstepmul', 1)
+
+gc_anchor = nil
+
+crash_on_trace_unwind_gc_setup()
+
+-- OOM on every allocation (i.e., on finalizer table rehashing
+-- too).
+allocinject.enable()
+
+local r, err = pcall(f, res_tab)
+
+allocinject.disable()
+
+test:ok(not r, 'correct status')
+test:like(err, 'not enough memory', 'correct error message')
+
+test:done(true)
diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
new file mode 100644
index 00000000..c3742e45
--- /dev/null
+++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
@@ -0,0 +1 @@
+BuildTestCLib(lj_1247_allocinject lj_1247_allocinject.c)
diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c
new file mode 100644
index 00000000..81aea60b
--- /dev/null
+++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c
@@ -0,0 +1,49 @@
+#include "lua.h"
+#include "lauxlib.h"
+
+#undef NDEBUG
+#include <assert.h>
+
+static lua_Alloc old_allocf = NULL;
+static void *old_alloc_state = NULL;
+
+/* Function to be used instead of the default allocator. */
+static void *allocf_with_injection(void *ud, void *ptr, size_t osize,
+				   size_t nsize)
+{
+	/* Always OOM on allocation (not on realloc). */
+	if (ptr == NULL)
+		return NULL;
+	else
+		return old_allocf(ud, ptr, osize, nsize);
+}
+
+static int enable(lua_State *L)
+{
+	assert(old_allocf == NULL);
+	old_allocf = lua_getallocf(L, &old_alloc_state);
+	lua_setallocf(L, allocf_with_injection, old_alloc_state);
+	return 0;
+}
+
+static int disable(lua_State *L)
+{
+	assert(old_allocf != NULL);
+	assert(old_allocf != allocf_with_injection);
+	lua_setallocf(L, old_allocf, old_alloc_state);
+	old_allocf = NULL;
+	old_alloc_state = NULL;
+	return 0;
+}
+
+static const struct luaL_Reg allocinject[] = {
+	{"enable", enable},
+	{"disable", disable},
+	{NULL, NULL}
+};
+
+LUA_API int luaopen_lj_1247_allocinject(lua_State *L)
+{
+	luaL_register(L, "lj_1247_allocinject", allocinject);
+	return 1;
+}
-- 
2.46.0

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Bronnikov via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 10346 bytes --]

Hi, Sergey!


thanks for the patch! LGTM with minor comments below

On 02.09.2024 15:54, Sergey Kaplun wrote:
> From: Mike Pall <mike>
>
> Reported by Sergey Kaplun.
>
> (cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)
>
> The raising of the OOM error when rehashing the finalizer table (when we
> can't allocate a new hash part) leads to crashes in either
> `lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
> raising, which either has no DWARF eh_frame or loses the context of the

I would add a link to a page about eh_frame, for example [1]

Feel free to ignore.

1. 
https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html

> JIT compiler.
>
> This patch drops rehashing of the finalizer table to avoid these

I would replace "finalizer" with "cdata finalizer".

And I would say about drawbacks of this. Otherwise, it looks like

rehashing was not needed from the beginning.

> crashes.
>
> Sergey Kaplun:
> * added the description and the test for the problem
>
> Part of tarantool/tarantool#10199
> Resolves tarantool/tarantool#10290
Usually "Closes" or "Fixes". Feel free to ignore.
> ---
>
> Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1247-fin-tab-rehashing-on-trace
> Related Issues:
> *https://github.com/tarantool/tarantool/issues/10290
> *https://github.com/LuaJIT/LuaJIT/issues/1247
> *https://github.com/tarantool/tarantool/issues/10199
>
>   src/lj_gc.c                                   |   7 -
>   src/lj_obj.h                                  |   2 +-
>   test/tarantool-tests/CMakeLists.txt           |   1 +
>   ...j-1247-fin-tab-rehashing-on-trace.test.lua | 127 ++++++++++++++++++
>   .../CMakeLists.txt                            |   1 +
>   .../lj_1247_allocinject.c                     |  49 +++++++
>   6 files changed, 179 insertions(+), 8 deletions(-)
>   create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
>   create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
>   create mode 100644 test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c
>
> diff --git a/src/lj_gc.c b/src/lj_gc.c
> index 4c222f21..a2fc93a0 100644
> --- a/src/lj_gc.c
> +++ b/src/lj_gc.c
> @@ -548,7 +548,6 @@ static void gc_finalize(lua_State *L)
>       setcdataV(L, &tmp, gco2cd(o));
>       tv = lj_tab_set(L, tabref(g->gcroot[GCROOT_FFI_FIN]), &tmp);
>       if (!tvisnil(tv)) {
> -      g->gc.nocdatafin = 0;
>         copyTV(L, &tmp, tv);
>         setnilV(tv);  /* Clear entry in finalizer table. */
>         gc_call_finalizer(g, L, &tmp, o);
> @@ -694,9 +693,6 @@ static size_t gc_onestep(lua_State *L)
>   	lj_str_resize(L, g->strmask >> 1);  /* Shrink string table. */
>         if (gcref(g->gc.mmudata)) {  /* Need any finalizations? */
>   	g->gc.state = GCSfinalize;
> -#if LJ_HASFFI
> -	g->gc.nocdatafin = 1;
> -#endif
>         } else {  /* Otherwise skip this phase to help the JIT. */
>   	g->gc.state = GCSpause;  /* End of GC cycle. */
>   	g->gc.debt = 0;
> @@ -713,9 +709,6 @@ static size_t gc_onestep(lua_State *L)
>   	g->gc.estimate -= GCFINALIZECOST;
>         return GCFINALIZECOST;
>       }
> -#if LJ_HASFFI
> -    if (!g->gc.nocdatafin) lj_tab_rehash(L, tabref(g->gcroot[GCROOT_FFI_FIN]));
> -#endif
>       g->gc.state = GCSpause;  /* End of GC cycle. */
>       g->gc.debt = 0;
>       return 0;
> diff --git a/src/lj_obj.h b/src/lj_obj.h
> index 06ea0cd0..ff22e5f8 100644
> --- a/src/lj_obj.h
> +++ b/src/lj_obj.h
> @@ -611,7 +611,7 @@ typedef struct GCState {
>     GCSize threshold;	/* Memory threshold. */
>     uint8_t currentwhite;	/* Current white color. */
>     uint8_t state;	/* GC state. */
> -  uint8_t nocdatafin;	/* No cdata finalizer called. */
> +  uint8_t unused0;
>   #if LJ_64
>     uint8_t lightudnum;	/* Number of lightuserdata segments - 1. */
>   #else
> diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt
> index e3750bf3..e5d5a470 100644
> --- a/test/tarantool-tests/CMakeLists.txt
> +++ b/test/tarantool-tests/CMakeLists.txt
> @@ -37,6 +37,7 @@ add_subdirectory(lj-flush-on-trace)
>   add_subdirectory(lj-1004-oom-error-frame)
>   add_subdirectory(lj-1066-fix-cur_L-after-coroutine-resume)
>   add_subdirectory(lj-1166-error-stitch)
> +add_subdirectory(lj-1247-fin-tab-rehashing-on-trace)
>   
>   # The part of the memory profiler toolchain is located in tools
>   # directory, jit, profiler, and bytecode toolchains are located
> diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
> new file mode 100644
> index 00000000..308043a2
> --- /dev/null
> +++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace.test.lua
> @@ -0,0 +1,127 @@
> +local tap = require('tap')
> +
> +-- The test file to demonstrate the incorrect JIT behaviour during
> +-- OOM on the finalizer table rehashing in the context of the JIT
> +-- trace.
> +-- See also:
> +-- *https://github.com/LuaJIT/LuaJIT/issues/1247,
> +-- *https://github.com/tarantool/tarantool/issues/10290.
> +
> +local test = tap.test('lj-1247-fin-tab-rehashing-on-trace'):skipcond({
> +  ['Broken unwiding in tarantool_panic_handler'] = _TARANTOOL and
> +                                                   (jit.os == 'OSX'),
> +  ['Disabled on MacOS due to #8652'] = jit.os == 'OSX',
> +  ['Test requires JIT enabled'] = not jit.status(),
> +})
> +
> +-- XXX: The original issue has 2 ways to crash:
> +-- 1) in `lj_trace_unwind()`
> +-- 2) in `lj_trace_exit()`
> +-- But, since we have an additional GC pressure due to requiring a
> +-- `tap` module, the second case needs an impossibly big
> +-- `gcstepmul` value to reproduce the issue. So, since the root
> +-- issue is the same and now rehashing of finalizer table is
> +-- omitted, we test only the first case.
> +test:plan(2)
> +
> +local allocinject = require('lj_1247_allocinject')
> +
> +local ffi = require('ffi')
> +ffi.cdef[[
> +  struct test {int a;};
> +]]
> +
> +local N_GC_STEPS = 100
> +local N_GC_FINALIZERS = 100
> +
> +local function empty() end
> +
> +-- Create a chunk like the following:
> +--[[
> +  local tostring = tostring
> +  local r = ...
> +  for _ = 1, 4 do
> +    r[1] = tostring(1)
> +    -- ...
> +    r[N_GCSTEPS] = tostring(N_GC_STEPS)
> +  end
> +--]]
> +local function create_chunk(n_steps)
> +  local chunk = 'local tostring = tostring\n'
> +  chunk = chunk .. ('local r = ...\n')
> +  chunk = chunk .. 'for _ = 1, 4 do\n'
> +  for i = 1, n_steps do
> +    chunk = chunk .. ('  r[%d] = tostring(%d)\n'):format(i, i)
> +  end
> +  chunk = chunk .. 'end\n'
> +  chunk = chunk .. 'return r\n'
> +  return chunk
> +end
> +
> +local function add_more_garbage(size)
> +  return ffi.new('char[?]', size)
> +end
> +
> +-- Helper to skip the atomic phase.
> +local function skip_atomic()
> +  local first_gc_called = false
> +  local function mark_fin() first_gc_called = true end
> +  jit.off(mark_fin)
> +  debug.getmetatable(newproxy(true)).__gc = mark_fin
> +
> +  -- Skip the atomic phase.
> +  jit.off()
> +  while not first_gc_called do collectgarbage('step') end
> +  jit.on()
> +end
> +
> +local function crash_on_trace_unwind_gc_setup()
> +  skip_atomic()
> +  collectgarbage('setstepmul', 1000)
> +  add_more_garbage(1024 * 1024)
> +end
> +
> +local f = assert(loadstring(create_chunk(N_GC_STEPS)))
> +
> +-- Create a really long trace.
> +jit.flush()
> +jit.opt.start('hotloop=2', 'maxirconst=5000', 'maxrecord=10000', 'maxsnap=1000',
> +              '-fold')
> +
> +-- luacheck: no unused
> +local gc_anchor = {}
> +local function anchor_finalizer(i)
> +  gc_anchor[i] = ffi.gc(ffi.new('struct test', i), empty)
> +end
> +
> +for i = 1, N_GC_FINALIZERS do
> +  anchor_finalizer(i)
> +end
> +
> +-- Record the trace first.
> +f({})
> +
> +-- The table for anchoring cdata objects.
> +local res_tab = {}
> +
> +collectgarbage()
> +collectgarbage()
> +collectgarbage('setpause', 0)
> +collectgarbage('setstepmul', 1)
> +
> +gc_anchor = nil
> +
> +crash_on_trace_unwind_gc_setup()
> +
> +-- OOM on every allocation (i.e., on finalizer table rehashing
> +-- too).
> +allocinject.enable()
> +
> +local r, err = pcall(f, res_tab)
> +
> +allocinject.disable()
> +
> +test:ok(not r, 'correct status')
> +test:like(err, 'not enough memory', 'correct error message')
> +
> +test:done(true)
> diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
> new file mode 100644
> index 00000000..c3742e45
> --- /dev/null
> +++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/CMakeLists.txt
> @@ -0,0 +1 @@
> +BuildTestCLib(lj_1247_allocinject lj_1247_allocinject.c)
> diff --git a/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c
> new file mode 100644
> index 00000000..81aea60b
> --- /dev/null
> +++ b/test/tarantool-tests/lj-1247-fin-tab-rehashing-on-trace/lj_1247_allocinject.c
> @@ -0,0 +1,49 @@
> +#include "lua.h"
> +#include "lauxlib.h"
> +
> +#undef NDEBUG
> +#include <assert.h>
> +
> +static lua_Alloc old_allocf = NULL;
> +static void *old_alloc_state = NULL;
> +
> +/* Function to be used instead of the default allocator. */
> +static void *allocf_with_injection(void *ud, void *ptr, size_t osize,
> +				   size_t nsize)
> +{
> +	/* Always OOM on allocation (not on realloc). */
> +	if (ptr == NULL)
> +		return NULL;
> +	else
> +		return old_allocf(ud, ptr, osize, nsize);
> +}
> +
> +static int enable(lua_State *L)
> +{
> +	assert(old_allocf == NULL);
> +	old_allocf = lua_getallocf(L, &old_alloc_state);
> +	lua_setallocf(L, allocf_with_injection, old_alloc_state);
> +	return 0;
> +}
> +
> +static int disable(lua_State *L)
> +{
> +	assert(old_allocf != NULL);
> +	assert(old_allocf != allocf_with_injection);
> +	lua_setallocf(L, old_allocf, old_alloc_state);
> +	old_allocf = NULL;
> +	old_alloc_state = NULL;
> +	return 0;
> +}
> +
> +static const struct luaL_Reg allocinject[] = {
> +	{"enable", enable},
> +	{"disable", disable},
> +	{NULL, NULL}
> +};
> +
> +LUA_API int luaopen_lj_1247_allocinject(lua_State *L)
> +{
> +	luaL_register(L, "lj_1247_allocinject", allocinject);
> +	return 1;
> +}

[-- Attachment #2: Type: text/html, Size: 11603 bytes --]

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Kaplun via Tarantool-patches]

Hi, Sergey!
Thanks for the review!
Fixed your comments, rebased the branch on the tarantool/master and
force-pushed it.

On 06.09.24, Sergey Bronnikov wrote:
> Hi, Sergey!
> 
> 
> thanks for the patch! LGTM with minor comments below
> 
> On 02.09.2024 15:54, Sergey Kaplun wrote:
> > From: Mike Pall <mike>
> >
> > Reported by Sergey Kaplun.
> >
> > (cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)
> >
> > The raising of the OOM error when rehashing the finalizer table (when we
> > can't allocate a new hash part) leads to crashes in either
> > `lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
> > raising, which either has no DWARF eh_frame or loses the context of the
> 
> I would add a link to a page about eh_frame, for example [1]
> 
> Feel free to ignore.
> 
> 1. 
> https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html

Added, thanks.

> 
> > JIT compiler.
> >
> > This patch drops rehashing of the finalizer table to avoid these
> 
> I would replace "finalizer" with "cdata finalizer".
> 
> And I would say about drawbacks of this. Otherwise, it looks like

Added.
See the new commit message below:

| FFI: Drop finalizer table rehash after GC cycle.
|
| Reported by Sergey Kaplun.
|
| (cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)
|
| The raising of the OOM error when rehashing the finalizer table (when we
| can't allocate a new hash part) leads to crashes in either
| `lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
| raising, which either has no DWARF eh_frame [1] or loses the context of
| the JIT compiler.
|
| This patch drops rehashing of the cdata finalizer table to avoid these
| crashes. It will prevent the cdata finalizer table from shrinking when
| the huge amount of the cdata objects is collected by the GC. OTOH, the
| finzlizer table most probably will grop anyway to the old size, so this
| is not crucial.
|
| Sergey Kaplun:
| * added the description and the test for the problem
|
| [1]: https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
|
| Part of tarantool/tarantool#10199
| Resolves tarantool/tarantool#10290

> 
> rehashing was not needed from the beginning.
> 
> > crashes.
> >
> > Sergey Kaplun:
> > * added the description and the test for the problem
> >
> > Part of tarantool/tarantool#10199
> > Resolves tarantool/tarantool#10290
> Usually "Closes" or "Fixes". Feel free to ignore.

For LuaJIT, we always use "Resolves", since it is __closed__ when the
submodule is bumped in the release branches and master.

> > ---
> >
> > Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1247-fin-tab-rehashing-on-trace
> > Related Issues:
> > *https://github.com/tarantool/tarantool/issues/10290
> > *https://github.com/LuaJIT/LuaJIT/issues/1247
> > *https://github.com/tarantool/tarantool/issues/10199
> >

<snipped>

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Bronnikov via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]

Hi, Sergey,


thanks for the fixes. Please replace "grop" to "grow" in commit

message. LGTM

On 07.09.2024 08:15, Sergey Kaplun wrote:
> Hi, Sergey!
> Thanks for the review!
> Fixed your comments, rebased the branch on the tarantool/master and
> force-pushed it.
>
> On 06.09.24, Sergey Bronnikov wrote:
>> Hi, Sergey!
>>
>>
>> thanks for the patch! LGTM with minor comments below
>>
>> On 02.09.2024 15:54, Sergey Kaplun wrote:
>>> From: Mike Pall <mike>
>>>
>>> Reported by Sergey Kaplun.
>>>
>>> (cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)
>>>
>>> The raising of the OOM error when rehashing the finalizer table (when we
>>> can't allocate a new hash part) leads to crashes in either
>>> `lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
>>> raising, which either has no DWARF eh_frame or loses the context of the
>> I would add a link to a page about eh_frame, for example [1]
>>
>> Feel free to ignore.
>>
>> 1.
>> https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
> Added, thanks.
>
>>> JIT compiler.
>>>
>>> This patch drops rehashing of the finalizer table to avoid these
>> I would replace "finalizer" with "cdata finalizer".
>>
>> And I would say about drawbacks of this. Otherwise, it looks like
> Added.
> See the new commit message below:
>
> | FFI: Drop finalizer table rehash after GC cycle.
> |
> | Reported by Sergey Kaplun.
> |
> | (cherry picked from commit fb22d0f80f291827a4004e16bc589b54bcc4a3c7)
> |
> | The raising of the OOM error when rehashing the finalizer table (when we
> | can't allocate a new hash part) leads to crashes in either
> | `lj_trace_exit()` or `lj_trace_unwind()` due to unprotected error
> | raising, which either has no DWARF eh_frame [1] or loses the context of
> | the JIT compiler.
> |
> | This patch drops rehashing of the cdata finalizer table to avoid these
> | crashes. It will prevent the cdata finalizer table from shrinking when
> | the huge amount of the cdata objects is collected by the GC. OTOH, the
> | finzlizer table most probably will grop anyway to the old size, so this
> | is not crucial.
> |
> | Sergey Kaplun:
> | * added the description and the test for the problem
> |
> | [1]:https://refspecs.linuxfoundation.org/LSB_3.0.0/LSB-Core-generic/LSB-Core-generic/ehframechpt.html
> |
> | Part of tarantool/tarantool#10199
> | Resolves tarantool/tarantool#10290
>
>> rehashing was not needed from the beginning.
>>
>>> crashes.
>>>
>>> Sergey Kaplun:
>>> * added the description and the test for the problem
>>>
>>> Part of tarantool/tarantool#10199
>>> Resolves tarantool/tarantool#10290
>> Usually "Closes" or "Fixes". Feel free to ignore.
> For LuaJIT, we always use "Resolves", since it is __closed__ when the
> submodule is bumped in the release branches and master.
>
>>> ---
>>>
>>> Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1247-fin-tab-rehashing-on-trace
>>> Related Issues:
>>> *https://github.com/tarantool/tarantool/issues/10290
>>> *https://github.com/LuaJIT/LuaJIT/issues/1247
>>> *https://github.com/tarantool/tarantool/issues/10199
>>>
> <snipped>
>

[-- Attachment #2: Type: text/html, Size: 5216 bytes --]

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Kaplun via Tarantool-patches]

Hi, Sergey!
Thanks for the review!

On 09.09.24, Sergey Bronnikov wrote:
> Hi, Sergey,
> 
> 
> thanks for the fixes. Please replace "grop" to "grow" in commit
> 
> message. LGTM

Fixed, thanks!
Branch is force-pushed.

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Maxim Kokryashkin via Tarantool-patches]

Hi, Sergey!
Thanks for the patch!
LGTM
On Mon, Sep 09, 2024 at 05:15:35PM GMT, Sergey Kaplun wrote:
> Hi, Sergey!
> Thanks for the review!
>
> On 09.09.24, Sergey Bronnikov wrote:
> > Hi, Sergey,
> >
> >
> > thanks for the fixes. Please replace "grop" to "grow" in commit
> >
> > message. LGTM
>
> Fixed, thanks!
> Branch is force-pushed.
>
> --
> Best regards,
> Sergey Kaplun

[Tarantool-patches] [PATCH luajit] Drop unused function wrapper.

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

(cherry picked from commit 87ae18af97fd4de790bb6c476b212e047689cc93)

This patch is a follow-up to the previous commit. Since the rehashing of
the finalizer table is gone, `lj_tab_rehash()` becomes unused and can be
dropped.

Sergey Kaplun:
* added the description for the patch

Part of tarantool/tarantool#10199
Follows up tarantool/tarantool#10290
---
 src/lj_tab.c | 7 -------
 src/lj_tab.h | 3 ---
 2 files changed, 10 deletions(-)

diff --git a/src/lj_tab.c b/src/lj_tab.c
index 1d6a4b7f..c99c6521 100644
--- a/src/lj_tab.c
+++ b/src/lj_tab.c
@@ -388,13 +388,6 @@ static void rehashtab(lua_State *L, GCtab *t, cTValue *ek)
   lj_tab_resize(L, t, asize, hsize2hbits(total));
 }
 
-#if LJ_HASFFI
-void lj_tab_rehash(lua_State *L, GCtab *t)
-{
-  rehashtab(L, t, niltv(L));
-}
-#endif
-
 void lj_tab_reasize(lua_State *L, GCtab *t, uint32_t nasize)
 {
   lj_tab_resize(L, t, nasize+1, t->hmask > 0 ? lj_fls(t->hmask)+1 : 0);
diff --git a/src/lj_tab.h b/src/lj_tab.h
index 71e34945..8575e7bd 100644
--- a/src/lj_tab.h
+++ b/src/lj_tab.h
@@ -41,9 +41,6 @@ LJ_FUNC GCtab * LJ_FASTCALL lj_tab_new1(lua_State *L, uint32_t ahsize);
 LJ_FUNCA GCtab * LJ_FASTCALL lj_tab_dup(lua_State *L, const GCtab *kt);
 LJ_FUNC void LJ_FASTCALL lj_tab_clear(GCtab *t);
 LJ_FUNC void LJ_FASTCALL lj_tab_free(global_State *g, GCtab *t);
-#if LJ_HASFFI
-LJ_FUNC void lj_tab_rehash(lua_State *L, GCtab *t);
-#endif
 LJ_FUNC void lj_tab_resize(lua_State *L, GCtab *t, uint32_t asize, uint32_t hbits);
 LJ_FUNCA void lj_tab_reasize(lua_State *L, GCtab *t, uint32_t nasize);
 
-- 
2.46.0

Re: [Tarantool-patches] [PATCH luajit] Drop unused function wrapper.

[Maxim Kokryashkin via Tarantool-patches]

Hi, Sergey!
Thanks for the patch!
LGTM.

Re: [Tarantool-patches] [PATCH luajit] Drop unused function wrapper.

[Sergey Bronnikov via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

Hi, Sergey

thanks for the patch! LGTM

On 19.09.2024 14:19, Sergey Kaplun wrote:
> From: Mike Pall <mike>
>
> (cherry picked from commit 87ae18af97fd4de790bb6c476b212e047689cc93)
>
> This patch is a follow-up to the previous commit. Since the rehashing of
> the finalizer table is gone, `lj_tab_rehash()` becomes unused and can be
> dropped.
>
> Sergey Kaplun:
> * added the description for the patch
>
> Part of tarantool/tarantool#10199
> Follows up tarantool/tarantool#10290
> ---
>   src/lj_tab.c | 7 -------
>   src/lj_tab.h | 3 ---
>   2 files changed, 10 deletions(-)
>
> diff --git a/src/lj_tab.c b/src/lj_tab.c
> index 1d6a4b7f..c99c6521 100644
> --- a/src/lj_tab.c
> +++ b/src/lj_tab.c
> @@ -388,13 +388,6 @@ static void rehashtab(lua_State *L, GCtab *t, cTValue *ek)
>     lj_tab_resize(L, t, asize, hsize2hbits(total));
>   }
>   
> -#if LJ_HASFFI
> -void lj_tab_rehash(lua_State *L, GCtab *t)
> -{
> -  rehashtab(L, t, niltv(L));
> -}
> -#endif
> -
>   void lj_tab_reasize(lua_State *L, GCtab *t, uint32_t nasize)
>   {
>     lj_tab_resize(L, t, nasize+1, t->hmask > 0 ? lj_fls(t->hmask)+1 : 0);
> diff --git a/src/lj_tab.h b/src/lj_tab.h
> index 71e34945..8575e7bd 100644
> --- a/src/lj_tab.h
> +++ b/src/lj_tab.h
> @@ -41,9 +41,6 @@ LJ_FUNC GCtab * LJ_FASTCALL lj_tab_new1(lua_State *L, uint32_t ahsize);
>   LJ_FUNCA GCtab * LJ_FASTCALL lj_tab_dup(lua_State *L, const GCtab *kt);
>   LJ_FUNC void LJ_FASTCALL lj_tab_clear(GCtab *t);
>   LJ_FUNC void LJ_FASTCALL lj_tab_free(global_State *g, GCtab *t);
> -#if LJ_HASFFI
> -LJ_FUNC void lj_tab_rehash(lua_State *L, GCtab *t);
> -#endif
>   LJ_FUNC void lj_tab_resize(lua_State *L, GCtab *t, uint32_t asize, uint32_t hbits);
>   LJ_FUNCA void lj_tab_reasize(lua_State *L, GCtab *t, uint32_t nasize);
>   

[-- Attachment #2: Type: text/html, Size: 2141 bytes --]

Re: [Tarantool-patches] [PATCH luajit] FFI: Drop finalizer table rehash after GC cycle.

[Sergey Kaplun via Tarantool-patches]

I've applied the patchset into all long-term branches in tarantool/luajit
and bumped a new version in master [1], release/3.2 [2] and
release/2.11 [3].

[1]: https://github.com/tarantool/tarantool/pull/10712
[2]: https://github.com/tarantool/tarantool/pull/10713
[3]: https://github.com/tarantool/tarantool/pull/10714

-- 
Best regards,
Sergey Kaplun