Hi, Sergey,

thanks for the review. Fixes applied and force-pushed.

Sergey

Hi, Sergey!
Thanks for the patch!
LGTM, after fixing a few nits below.

On 09.06.26, Sergey Bronnikov wrote:

From: Mike Pall <mike>

Reported by Sergey Bronnikov.

(cherry picked from commit 8f421c81ec6aaae0bcd80e01f4353de200afbbc5)

The Undefined Behaviour Sanitizer [1] produce a warning because

Typo: s/produce/produces/

Fixed.

the function `lua_createtable()` takes signed integer arguments,
but the `lj_tab_new_ah()` was not properly validating or converting
these signed values before using them in unsigned arithmetic.

The fix changes the signature of `lj_tab_new_ah()` to accept
uint32_t directly, and adjusts `lua_createtable()` to cast the
incoming signed int values to uint32_t before passing them.

[1]: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

Sergey Bronnikov:
* added the description and the test for the problem

Part of tarantool/tarantool#12480
---
Branch: https://github.com/tarantool/luajit/tree/ligurio/lj-1458-ub-lj_tab_new

Related issues:
- https://github.com/tarantool/tarantool/issues/12480
- https://github.com/LuaJIT/LuaJIT/issues/1458


 src/lj_api.c                                  |  2 +-
 src/lj_tab.c                                  |  4 +--
 src/lj_tab.h                                  |  2 +-
 .../lj-1458-ub-table.new.test.lua             | 30 +++++++++++++++++++
 4 files changed, 34 insertions(+), 4 deletions(-)
 create mode 100644 test/tarantool-tests/lj-1458-ub-table.new.test.lua

<snipped>

diff --git a/test/tarantool-tests/lj-1458-ub-table.new.test.lua b/test/tarantool-tests/lj-1458-ub-table.new.test.lua
new file mode 100644
index 00000000..d0cf6ff5
--- /dev/null
+++ b/test/tarantool-tests/lj-1458-ub-table.new.test.lua

Typo: s/table.new/table-new/

@@ -0,0 +1,30 @@
+local tap = require('tap')
+
+-- The test file to demonstrate UBSan warning for `table.new()`
+-- with a minimal and maximum array and hash parts values.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1458.
+local test = tap.test('lj-1458-ub-table-new')
+
+test:plan(8)

Lets use table_sizes * 2 here.

Updated.

+
+local table_new = require('table.new')
+
+local INT_MAX =  2 ^ 31 - 1
+local INT_MIN = -2 ^ 31
+
+local table_sizes = {
+  { 0, INT_MIN },
+  { 0, INT_MAX },
+  { INT_MIN, 0 },
+  { INT_MAX, 0 },
+}
+
+for _, case in ipairs(table_sizes) do
+  local apart, hpart = unpack(case)
+  local ok, err = pcall(table_new, apart, hpart)
+  local message = ('table.new(%d, %d)'):format(apart, hpart)
+  test:is(ok, false, message .. ' is failed')

Typo? s/failed/OK/

Right, fixed:

--- a/test/tarantool-tests/lj-1458-ub-table-new.test.lua
+++ b/test/tarantool-tests/lj-1458-ub-table-new.test.lua
@@ -23,7 +23,7 @@ for _, case in ipairs(table_sizes) do
   local apart, hpart = unpack(case)
   local ok, err = pcall(table_new, apart, hpart)
   local message = ('table.new(%d, %d)'):format(apart, hpart)
-  test:is(ok, false, message .. ' is failed')
+  test:is(ok, false, message .. ' is ok')
   test:ok(err:match('table overflow'), message .. ' correct error message')
 end

+  test:ok(err:match('table overflow'), message .. ' correct error message')
+end
+
+test:done(true)
-- 
2.43.0