[Tarantool-patches] [PATCH luajit] FFI: Prevent sanitizer warning in carith_ptr().

[Sergey Bronnikov via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

From: Mike Pall <mike>

Reported by Sergey Bronnikov.

(cherry picked from commit c3b379bf50c8819c61daa3afd9f21d9ec5708bd5)

The Undefined Behaviour Sanitizer [1] produce a warning about
signed integer overflow in the function carith_ptr(), when idx
equals INT_MIN in the expression idx = -idx. In practice,
negating -2147483648 wraps back to itself instead of producing the
correct positive value, leading to incorrect pointer arithmetic in
carith_ptr() for MM_sub (pointer subtraction). The patch fixes
that by converting idx to unsigned (uintptr_t), computes two's
complement negation via bitwise negation + 1 (safe from overflow
in unsigned arithmetic), and casts back to ptrdiff_t. This avoids
signed overflow entirely and correctly negates even the INT_MIN
edge case.

[1]: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html

Sergey Bronnikov:
* added the description and the test for the problem

Part of tarantool/tarantool#12480
---

Branch: https://github.com/tarantool/luajit/tree/ligurio/lj-1459-ub-carith_ptr
Related issues:
- https://github.com/LuaJIT/LuaJIT/issues/1459
- https://github.com/tarantool/tarantool/issues/12880

 src/lj_carith.c                                  |  2 +-
 .../lj-1459-subtraction-carith.test.lua          | 16 ++++++++++++++++
 2 files changed, 17 insertions(+), 1 deletion(-)
 create mode 100644 test/tarantool-tests/lj-1459-subtraction-carith.test.lua

diff --git a/src/lj_carith.c b/src/lj_carith.c
index eb56d552..e971bfc2 100644
--- a/src/lj_carith.c
+++ b/src/lj_carith.c
@@ -135,7 +135,7 @@ static int carith_ptr(lua_State *L, CTState *cts, CDArith *ca, MMS mm)
       return 0;
     lj_cconv_ct_ct(cts, ctype_get(cts, CTID_INT_PSZ), ca->ct[1],
 		   (uint8_t *)&idx, ca->p[1], 0);
-    if (mm == MM_sub) idx = -idx;
+    if (mm == MM_sub) idx = (ptrdiff_t)(~(uintptr_t)idx+1u);
   } else if (mm == MM_add && ctype_isnum(ctp->info) &&
       (ctype_isptr(ca->ct[1]->info) || ctype_isrefarray(ca->ct[1]->info))) {
     /* Swap pointer and index. */
diff --git a/test/tarantool-tests/lj-1459-subtraction-carith.test.lua b/test/tarantool-tests/lj-1459-subtraction-carith.test.lua
new file mode 100644
index 00000000..df401a56
--- /dev/null
+++ b/test/tarantool-tests/lj-1459-subtraction-carith.test.lua
@@ -0,0 +1,16 @@
+local tap = require('tap')
+
+-- The test file to demonstrate UBSan warning in carith_ptr().
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1459.
+local test = tap.test('lj-1459-subtraction-carith')
+
+test:plan(2)
+
+local func = load('_ = nil - 0LL%0')
+local res, err = pcall(func)
+
+test:is(res, false, 'correct result')
+local error_msg = "attempt to perform arithmetic on 'nil' and 'int64_t'"
+test:ok(err:match(error_msg), 'error on subtraction')
+
+test:done(true)
-- 
2.43.0