[Tarantool-patches] [PATCH luajit 0/2] Correct allocation limit without JIT

[Tarantool-patches] [PATCH luajit 0/2] Correct allocation limit without JIT

[Sergey Kaplun via Tarantool-patches]

This patch-set fixes the allocation limit for the build with disabled
JIT. The first commit helps to catch the error early in tests by the
correct check of the address returned by the allocator.

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1430-alloc-limit
Related issues:
* https://github.com/LuaJIT/LuaJIT/issues/1430
* https://github.com/tarantool/tarantool/issues/12134
Related ML thread: https://www.freelists.org/post/luajit/Provide-error-outputdetails-upon-luaL-newstate-failures

Mike Pall (2):
  Fix pointer check for non-GC64 mode.
  x64/!LJ_GC64: The allocation limit is required for a no-JIT build,
    too.

 src/lj_alloc.c                                |  4 +-
 src/lj_def.h                                  |  3 +-
 .../lj-1430-internal-alloc-limit.test.lua     | 39 +++++++++++++++++++
 3 files changed, 43 insertions(+), 3 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1430-internal-alloc-limit.test.lua

-- 
2.53.0

[Tarantool-patches] [PATCH luajit 1/2] Fix pointer check for non-GC64 mode.

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

Thanks to Stefan Hett.

(cherry picked from commit 471f8936cbd6aa80a937e375fe53ecadab93696a)

This commit fixes the check for the pointer returned by the internal
LuaJIT allocator. For non-GC64 mode, the accessible address range should
fit in 31 bits due to VM (and JIT) restrictions. This commit fixes the
check.

Since it is just an assertion check, there are no tests added for the
change. But this commit simplifies the reproducer for the next patch
(although not required).

Sergey Kaplun:
* added the description for the problem

Part of tarantool/tarantool#12134
---
 src/lj_def.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/lj_def.h b/src/lj_def.h
index a5bca6b0..b06462fe 100644
--- a/src/lj_def.h
+++ b/src/lj_def.h
@@ -105,9 +105,10 @@ typedef unsigned int uintptr_t;
 #define checku16(x)	((x) == (int32_t)(uint16_t)(x))
 #define checki32(x)	((x) == (int32_t)(x))
 #define checku32(x)	((x) == (uint32_t)(x))
+#define checkptr31(x)	(((uint64_t)(uintptr_t)(x) >> 31) == 0)
 #define checkptr32(x)	((uintptr_t)(x) == (uint32_t)(uintptr_t)(x))
 #define checkptr47(x)	(((uint64_t)(uintptr_t)(x) >> 47) == 0)
-#define checkptrGC(x)	(LJ_GC64 ? checkptr47((x)) : LJ_64 ? checkptr32((x)) :1)
+#define checkptrGC(x)	(LJ_GC64 ? checkptr47((x)) : LJ_64 ? checkptr31((x)) :1)
 
 /* Every half-decent C compiler transforms this into a rotate instruction. */
 #define lj_rol(x, n)	(((x)<<(n)) | ((x)>>(-(int)(n)&(8*sizeof(x)-1))))
-- 
2.53.0

[Tarantool-patches] [PATCH luajit 2/2] x64/!LJ_GC64: The allocation limit is required for a no-JIT build, too.

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

Thanks to Sergey Kaplun.

(cherry picked from commit eff4006837792b6105e0a1743283ddde3548fc09)

For the non-GC64 build with disabled JIT, LuaJIT's internal allocator
may return 32-bit addresses. This may lead to the assertion failure
during the reallocation of the array part of the table or to the crash
(if assertions are disabled) due to an incorrect arithmetic in the x86
VM. For example, the addition with a 32-bit wide address may overflow
in TSETV or TGETV and cause the crash.

This patch sets the allocation limit for the build without JIT.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#12134
---
 src/lj_alloc.c                                |  4 +-
 .../lj-1430-internal-alloc-limit.test.lua     | 39 +++++++++++++++++++
 2 files changed, 41 insertions(+), 2 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1430-internal-alloc-limit.test.lua

diff --git a/src/lj_alloc.c b/src/lj_alloc.c
index f82c9854..97eb94d2 100644
--- a/src/lj_alloc.c
+++ b/src/lj_alloc.c
@@ -99,8 +99,8 @@
 
 #if LJ_GC64
 #define LJ_ALLOC_MBITS		47	/* 128 TB in LJ_GC64 mode. */
-#elif LJ_TARGET_X64 && LJ_HASJIT
-/* Due to limitations in the x64 compiler backend. */
+#elif LJ_TARGET_X64
+/* Due to limitations in the x64 non-GC64 VM. */
 #define LJ_ALLOC_MBITS		31	/* 2 GB on x64 with !LJ_GC64. */
 #else
 #define LJ_ALLOC_MBITS		32	/* 4 GB on other archs with !LJ_GC64. */
diff --git a/test/tarantool-tests/lj-1430-internal-alloc-limit.test.lua b/test/tarantool-tests/lj-1430-internal-alloc-limit.test.lua
new file mode 100644
index 00000000..969d26d6
--- /dev/null
+++ b/test/tarantool-tests/lj-1430-internal-alloc-limit.test.lua
@@ -0,0 +1,39 @@
+local tap = require('tap')
+
+-- Test file to demonstrate incorrect allocation limit for the
+-- non-GC64 build with disabled JIT.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1430.
+
+local test = tap.test('lj-1430-internal-alloc-limit')
+
+test:plan(1)
+
+-- This function creates a bunch of long array-like tables.
+-- Eventually for one of the tables the address of the array
+-- element will not fit in the 31-bit range, causing the incorrect
+-- arithmetic inside the VM and a crash or assertion failure
+-- during the reallocation.
+local function test_payload()
+  local POOL_SZ = 8
+  -- luacheck: no unused
+  local pools = {}
+  for i = 1, POOL_SZ do
+    pools[i] = {}
+  end
+
+  local v = 1
+  for j = 1, POOL_SZ do
+    for i = 1, 0x2000000 do
+      pools[j][i] = v
+    end
+  end
+end
+
+-- Protect the call to avoid the OOM.
+pcall(test_payload)
+
+-- Free memory for the TAP tests.
+collectgarbage()
+
+test:ok(true, 'no crash or assertion failure')
+test:done(true)
-- 
2.53.0