[Tarantool-patches] [PATCH luajit] Reject negative getfenv()/setfenv() levels to prevent compiler warning.

[Tarantool-patches] [PATCH luajit] Reject negative getfenv()/setfenv() levels to prevent compiler warning.

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

Thanks to Sergey Kaplun.

(cherry picked from commit 9d777346bc4e3b033dd78393980d41ee7bc34867)

When the number represented the level value is given to the
`getfenv()`/`setfenv()`, it is cast to the `int`. Assume the given value
is `2^31`, i.e. the resulting value after the cast is `INT_MIN`. After
this, it will be decremented in `lj_debug_level()` and underflowed to
the `INT_MAX`. That produces the UBSan warning about signed integer
overflow.

This patch raises the error early in the aforementioned functions, since
a negative level value is meaningless.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#11055
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1329-getfenv-setfenv-negative
Related issues:
* https://github.com/tarantool/tarantool/issues/11055
* https://github.com/LuaJIT/LuaJIT/issues/1329

 src/lib_base.c                                |  4 +++
 .../lj-1329-getfenv-setfenv-negative.test.lua | 27 +++++++++++++++++++
 2 files changed, 31 insertions(+)
 create mode 100644 test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua

diff --git a/src/lib_base.c b/src/lib_base.c
index ad151975..23728d6c 100644
--- a/src/lib_base.c
+++ b/src/lib_base.c
@@ -144,6 +144,8 @@ LJLIB_CF(getfenv)		LJLIB_REC(.)
   cTValue *o = L->base;
   if (!(o < L->top && tvisfunc(o))) {
     int level = lj_lib_optint(L, 1, 1);
+    if (level < 0)
+      lj_err_arg(L, 1, LJ_ERR_INVLVL);
     o = lj_debug_frame(L, level, &level);
     if (o == NULL)
       lj_err_arg(L, 1, LJ_ERR_INVLVL);
@@ -166,6 +168,8 @@ LJLIB_CF(setfenv)
       setgcref(L->env, obj2gco(t));
       return 0;
     }
+    if (level < 0)
+      lj_err_arg(L, 1, LJ_ERR_INVLVL);
     o = lj_debug_frame(L, level, &level);
     if (o == NULL)
       lj_err_arg(L, 1, LJ_ERR_INVLVL);
diff --git a/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua b/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua
new file mode 100644
index 00000000..bc10c16f
--- /dev/null
+++ b/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua
@@ -0,0 +1,27 @@
+local tap = require('tap')
+
+-- The test file to demonstrate UBSan warning for `setfenv()` and
+-- `getfenv()` with a huge `level` value.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1329.
+local test = tap.test('lj-1329-getfenv-setfenv-negative')
+
+test:plan(4)
+
+-- This number will be equal to `INT_MIN` when casted to `int`.
+-- After this, it will be decremented in `lj_debug_level()` and
+-- underflowed to the `INT_MAX`. That produces the UBSan warning
+-- about signed integer overflow.
+local LEVEL = 2 ^ 31
+local ERRMSG = 'invalid level'
+
+-- Tests check the UBSan runtime error. Add assertions just to be
+-- sure that we don't change the behaviour.
+local status, errmsg = pcall(getfenv, LEVEL)
+test:ok(not status, 'getfenv: correct status')
+test:like(errmsg, ERRMSG, 'getfenv: correct error message')
+
+status, errmsg = pcall(setfenv, LEVEL, {})
+test:ok(not status, 'setfenv: correct status')
+test:like(errmsg, ERRMSG, 'setfenv: correct error message')
+
+test:done(true)
-- 
2.47.1

Re: [Tarantool-patches] [PATCH luajit] Reject negative getfenv()/setfenv() levels to prevent compiler warning.

[Sergey Bronnikov via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 3412 bytes --]

Hello, Sergey!

thanks for the patch! LGTM

Sergey

On 03.02.2025 16:31, Sergey Kaplun wrote:
> From: Mike Pall <mike>
>
> Thanks to Sergey Kaplun.
>
> (cherry picked from commit 9d777346bc4e3b033dd78393980d41ee7bc34867)
>
> When the number represented the level value is given to the
> `getfenv()`/`setfenv()`, it is cast to the `int`. Assume the given value
> is `2^31`, i.e. the resulting value after the cast is `INT_MIN`. After
> this, it will be decremented in `lj_debug_level()` and underflowed to
> the `INT_MAX`. That produces the UBSan warning about signed integer
> overflow.
>
> This patch raises the error early in the aforementioned functions, since
> a negative level value is meaningless.
>
> Sergey Kaplun:
> * added the description and the test for the problem
>
> Part of tarantool/tarantool#11055
> ---
>
> Branch:https://github.com/tarantool/luajit/tree/skaplun/lj-1329-getfenv-setfenv-negative
> Related issues:
> *https://github.com/tarantool/tarantool/issues/11055
> *https://github.com/LuaJIT/LuaJIT/issues/1329
>
>   src/lib_base.c                                |  4 +++
>   .../lj-1329-getfenv-setfenv-negative.test.lua | 27 +++++++++++++++++++
>   2 files changed, 31 insertions(+)
>   create mode 100644 test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua
>
> diff --git a/src/lib_base.c b/src/lib_base.c
> index ad151975..23728d6c 100644
> --- a/src/lib_base.c
> +++ b/src/lib_base.c
> @@ -144,6 +144,8 @@ LJLIB_CF(getfenv)		LJLIB_REC(.)
>     cTValue *o = L->base;
>     if (!(o < L->top && tvisfunc(o))) {
>       int level = lj_lib_optint(L, 1, 1);
> +    if (level < 0)
> +      lj_err_arg(L, 1, LJ_ERR_INVLVL);
>       o = lj_debug_frame(L, level, &level);
>       if (o == NULL)
>         lj_err_arg(L, 1, LJ_ERR_INVLVL);
> @@ -166,6 +168,8 @@ LJLIB_CF(setfenv)
>         setgcref(L->env, obj2gco(t));
>         return 0;
>       }
> +    if (level < 0)
> +      lj_err_arg(L, 1, LJ_ERR_INVLVL);
>       o = lj_debug_frame(L, level, &level);
>       if (o == NULL)
>         lj_err_arg(L, 1, LJ_ERR_INVLVL);
> diff --git a/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua b/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua
> new file mode 100644
> index 00000000..bc10c16f
> --- /dev/null
> +++ b/test/tarantool-tests/lj-1329-getfenv-setfenv-negative.test.lua
> @@ -0,0 +1,27 @@
> +local tap = require('tap')
> +
> +-- The test file to demonstrate UBSan warning for `setfenv()` and
> +-- `getfenv()` with a huge `level` value.
> +-- See also:https://github.com/LuaJIT/LuaJIT/issues/1329.
> +local test = tap.test('lj-1329-getfenv-setfenv-negative')
> +
> +test:plan(4)
> +
> +-- This number will be equal to `INT_MIN` when casted to `int`.
> +-- After this, it will be decremented in `lj_debug_level()` and
> +-- underflowed to the `INT_MAX`. That produces the UBSan warning
> +-- about signed integer overflow.
> +local LEVEL = 2 ^ 31
> +local ERRMSG = 'invalid level'
> +
> +-- Tests check the UBSan runtime error. Add assertions just to be
> +-- sure that we don't change the behaviour.
> +local status, errmsg = pcall(getfenv, LEVEL)
> +test:ok(not status, 'getfenv: correct status')
> +test:like(errmsg, ERRMSG, 'getfenv: correct error message')
> +
> +status, errmsg = pcall(setfenv, LEVEL, {})
> +test:ok(not status, 'setfenv: correct status')
> +test:like(errmsg, ERRMSG, 'setfenv: correct error message')
> +
> +test:done(true)

[-- Attachment #2: Type: text/html, Size: 4161 bytes --]