[Tarantool-patches] [PATCH luajit] Handle stack reallocation in debug.setmetatable() and lua_setmetatable().

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

From: Mike Pall <mike>

Thanks to Sergey Kaplun.

(cherry picked from commit 88ed9fdbbba632d174a473a0a97c914089c2916d)

When we use the aforementioned functions to set a metatable for types
with one shared metatable, we must flush all traces since they are
specialized to base metatables. If we have enabled vmevent handlers,
they invoke a callback on trace flushing. This callback may reallocate
the Lua stack. Thus invalidates the reference to the `TValue *` object
`o` by the given index in the `lua_setmetatable()` and leads to a
heap-use-after-free error.

This patch fixes the behaviour by recalculating the address by the given
index after possible stack reallocation.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#9595
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1172-debug-handling-ref
Tarantool PR: https://github.com/tarantool/tarantool/pull/9786
Related issues:
* https://github.com/tarantool/tarantool/issues/9595
* https://github.com/LuaJIT/LuaJIT/issues/1172


 src/lj_api.c                                  |  1 +
 .../lj-1172-debug-handling-ref.test.lua       | 30 +++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 test/tarantool-tests/lj-1172-debug-handling-ref.test.lua

diff --git a/src/lj_api.c b/src/lj_api.c
index 3bacad33..2e915306 100644
--- a/src/lj_api.c
+++ b/src/lj_api.c
@@ -1067,6 +1067,7 @@ LUA_API int lua_setmetatable(lua_State *L, int idx)
     /* Flush cache, since traces specialize to basemt. But not during __gc. */
     if (lj_trace_flushall(L))
       lj_err_caller(L, LJ_ERR_NOGCMM);
+    o = index2adr(L, idx);  /* Stack may have been reallocated. */
     if (tvisbool(o)) {
       /* NOBARRIER: basemt is a GC root. */
       setgcref(basemt_it(g, LJ_TTRUE), obj2gco(mt));
diff --git a/test/tarantool-tests/lj-1172-debug-handling-ref.test.lua b/test/tarantool-tests/lj-1172-debug-handling-ref.test.lua
new file mode 100644
index 00000000..cac1c223
--- /dev/null
+++ b/test/tarantool-tests/lj-1172-debug-handling-ref.test.lua
@@ -0,0 +1,30 @@
+local tap = require('tap')
+
+-- Test file to demonstrate the heap-use-after-free, error for
+-- `debug.setmetatable()` and enabled `jit.dump()`.
+-- The test fails under ASAN.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1172.
+
+local test = tap.test('lj-1172-debug-handling-ref'):skipcond({
+  ['Test requires JIT enabled'] = not jit.status(),
+})
+
+local jdump = require('jit.dump')
+
+test:plan(1)
+
+jdump.start('t', '/dev/null')
+
+-- Use `coroutine.wrap()` to create a new Lua stack with a minimum
+-- number of stack slots.
+coroutine.wrap(function()
+  -- "TRACE flush" event handler causes stack reallocation and
+  -- leads to heap-use-after-free. This event handler is called
+  -- because all traces are specialized to base metatables, so
+  -- if we update any base metatable, we must flush all traces.
+  debug.setmetatable(1, {})
+end)()
+
+test:ok(true, 'no heap-use-after-free error')
+
+test:done(true)
-- 
2.44.0