[Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Sergey Kaplun via Tarantool-patches]

From: Mike Pall <mike>

Thanks to HybridDog.

When build with optimization compiler may throw away overflow check in
`unpack()` base library function.

This patch prevents aforementioned error by comparing the unsigned
amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#7230
---

Issue/PR:
* https://github.com/LuaJIT/LuaJIT/pull/574
* https://github.com/tarantool/tarantool/issues/7230
Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
PR: https://github.com/tarantool/tarantool/pull/7596

 src/lib_base.c                                       |  6 ++++--
 test/tarantool-tests/lj-574-overflow-unpack.test.lua | 12 ++++++++++++
 2 files changed, 16 insertions(+), 2 deletions(-)
 create mode 100644 test/tarantool-tests/lj-574-overflow-unpack.test.lua

diff --git a/src/lib_base.c b/src/lib_base.c
index 613a1859..cf57b4f2 100644
--- a/src/lib_base.c
+++ b/src/lib_base.c
@@ -224,9 +224,11 @@ LJLIB_CF(unpack)
   int32_t n, i = lj_lib_optint(L, 2, 1);
   int32_t e = (L->base+3-1 < L->top && !tvisnil(L->base+3-1)) ?
 	      lj_lib_checkint(L, 3) : (int32_t)lj_tab_len(t);
+  uint32_t nu;
   if (i > e) return 0;
-  n = e - i + 1;
-  if (n <= 0 || !lua_checkstack(L, n))
+  nu = (uint32_t)e - (uint32_t)i;
+  n = (int32_t)(nu+1);
+  if (nu >= LUAI_MAXCSTACK || !lua_checkstack(L, n))
     lj_err_caller(L, LJ_ERR_UNPACK);
   do {
     cTValue *tv = lj_tab_getint(t, i);
diff --git a/test/tarantool-tests/lj-574-overflow-unpack.test.lua b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
new file mode 100644
index 00000000..6715d947
--- /dev/null
+++ b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
@@ -0,0 +1,12 @@
+local tap = require('tap')
+
+-- Test file to demonstrate integer overflow in the `unpack()`
+-- function due to compiler optimization.
+-- See also https://github.com/LuaJIT/LuaJIT/pull/574.
+local test = tap.test('lj-574-overflow-unpack')
+test:plan(1)
+
+local r, e = pcall(unpack, {}, 0, 2^31 - 1)
+test:ok(not r and e == 'too many results to unpack', 'overflow check in unpack')
+
+os.exit(test:check() and 0 or 1)
-- 
2.34.1

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[sergos via Tarantool-patches]

Hi Sergey!

Thanks for the patch, LGTM.

Sergos.


> On 23 Aug 2022, at 11:06, Sergey Kaplun <skaplun@tarantool.org> wrote:
> 
> From: Mike Pall <mike>
> 
> Thanks to HybridDog.
> 
> When build with optimization compiler may throw away overflow check in
> `unpack()` base library function.
> 
> This patch prevents aforementioned error by comparing the unsigned
> amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
> 
> Sergey Kaplun:
> * added the description and the test for the problem
> 
> Part of tarantool/tarantool#7230
> ---
> 
> Issue/PR:
> * https://github.com/LuaJIT/LuaJIT/pull/574
> * https://github.com/tarantool/tarantool/issues/7230
> Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
> PR: https://github.com/tarantool/tarantool/pull/7596
> 
> src/lib_base.c                                       |  6 ++++--
> test/tarantool-tests/lj-574-overflow-unpack.test.lua | 12 ++++++++++++
> 2 files changed, 16 insertions(+), 2 deletions(-)
> create mode 100644 test/tarantool-tests/lj-574-overflow-unpack.test.lua
> 
> diff --git a/src/lib_base.c b/src/lib_base.c
> index 613a1859..cf57b4f2 100644
> --- a/src/lib_base.c
> +++ b/src/lib_base.c
> @@ -224,9 +224,11 @@ LJLIB_CF(unpack)
>   int32_t n, i = lj_lib_optint(L, 2, 1);
>   int32_t e = (L->base+3-1 < L->top && !tvisnil(L->base+3-1)) ?
> 	      lj_lib_checkint(L, 3) : (int32_t)lj_tab_len(t);
> +  uint32_t nu;
>   if (i > e) return 0;
> -  n = e - i + 1;
> -  if (n <= 0 || !lua_checkstack(L, n))
> +  nu = (uint32_t)e - (uint32_t)i;
> +  n = (int32_t)(nu+1);
> +  if (nu >= LUAI_MAXCSTACK || !lua_checkstack(L, n))
>     lj_err_caller(L, LJ_ERR_UNPACK);
>   do {
>     cTValue *tv = lj_tab_getint(t, i);
> diff --git a/test/tarantool-tests/lj-574-overflow-unpack.test.lua b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
> new file mode 100644
> index 00000000..6715d947
> --- /dev/null
> +++ b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
> @@ -0,0 +1,12 @@
> +local tap = require('tap')
> +
> +-- Test file to demonstrate integer overflow in the `unpack()`
> +-- function due to compiler optimization.
> +-- See also https://github.com/LuaJIT/LuaJIT/pull/574.
> +local test = tap.test('lj-574-overflow-unpack')
> +test:plan(1)
> +
> +local r, e = pcall(unpack, {}, 0, 2^31 - 1)
> +test:ok(not r and e == 'too many results to unpack', 'overflow check in unpack')
> +
> +os.exit(test:check() and 0 or 1)
> -- 
> 2.34.1
> 

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Sergey Kaplun via Tarantool-patches]

Hi, Sergos!

Thanks for the review!

On 23.08.22, sergos wrote:
> Hi Sergey!
> 
> Thanks for the patch, LGTM.
> 
> Sergos.
> 
> 
> > On 23 Aug 2022, at 11:06, Sergey Kaplun <skaplun@tarantool.org> wrote:
> > 
> > From: Mike Pall <mike>
> > 
> > Thanks to HybridDog.

Added the following forgotten title as usual:)

| (cherry picked from commit 179cf2eb84fef2b9a524469c3c8cc49363b8fb10)

Branch is force-pushed.

> > 
> > When build with optimization compiler may throw away overflow check in
> > `unpack()` base library function.
> > 
> > This patch prevents aforementioned error by comparing the unsigned
> > amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
> > 
> > Sergey Kaplun:
> > * added the description and the test for the problem
> > 
> > Part of tarantool/tarantool#7230
> > ---
> > 
> > Issue/PR:
> > * https://github.com/LuaJIT/LuaJIT/pull/574
> > * https://github.com/tarantool/tarantool/issues/7230
> > Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
> > PR: https://github.com/tarantool/tarantool/pull/7596
> > 

<snipped>

> > 2.34.1
> > 
> 

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Maxim Kokryashkin via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 2686 bytes --]


Hi, Sergey!
Thanks for the patch!
Please consider my comments below:
  
>from Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>:
> 
>From: Mike Pall <mike>
>
>Thanks to HybridDog.
>
>When build with optimization compiler may throw away overflow check in
>`unpack()` base library function.
Typo: s/build with optimization/built with optimization,
Also,  I think we should mention the specific optimization that causes the mentioned behavior
since it is not mentioned in both the LuaJIT’s issue and the original Lua issue.
>
>This patch prevents aforementioned error by comparing the unsigned
>amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
>
>Sergey Kaplun:
>* added the description and the test for the problem
>
>Part of tarantool/tarantool#7230
>---
>
>Issue/PR:
>*  https://github.com/LuaJIT/LuaJIT/pull/574
>*  https://github.com/tarantool/tarantool/issues/7230
>Branch:  https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
>PR:  https://github.com/tarantool/tarantool/pull/7596
>
> src/lib_base.c | 6 ++++--
> test/tarantool-tests/lj-574-overflow-unpack.test.lua | 12 ++++++++++++
> 2 files changed, 16 insertions(+), 2 deletions(-)
> create mode 100644 test/tarantool-tests/lj-574-overflow-unpack.test.lua
>
>diff --git a/src/lib_base.c b/src/lib_base.c
>index 613a1859..cf57b4f2 100644
>--- a/src/lib_base.c
>+++ b/src/lib_base.c
>@@ -224,9 +224,11 @@ LJLIB_CF(unpack)
>   int32_t n, i = lj_lib_optint(L, 2, 1);
>   int32_t e = (L->base+3-1 < L->top && !tvisnil(L->base+3-1)) ?
>  lj_lib_checkint(L, 3) : (int32_t)lj_tab_len(t);
>+ uint32_t nu;
>   if (i > e) return 0;
>- n = e - i + 1;
>- if (n <= 0 || !lua_checkstack(L, n))
>+ nu = (uint32_t)e - (uint32_t)i;
>+ n = (int32_t)(nu+1);
>+ if (nu >= LUAI_MAXCSTACK || !lua_checkstack(L, n))
>     lj_err_caller(L, LJ_ERR_UNPACK);
>   do {
>     cTValue *tv = lj_tab_getint(t, i);
>diff --git a/test/tarantool-tests/lj-574-overflow-unpack.test.lua b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
>new file mode 100644
>index 00000000..6715d947
>--- /dev/null
>+++ b/test/tarantool-tests/lj-574-overflow-unpack.test.lua
>@@ -0,0 +1,12 @@
>+local tap = require('tap')
>+
>+-- Test file to demonstrate integer overflow in the `unpack()`
>+-- function due to compiler optimization.
>+-- See also  https://github.com/LuaJIT/LuaJIT/pull/574 .
>+local test = tap.test('lj-574-overflow-unpack')
>+test:plan(1)
>+
>+local r, e = pcall(unpack, {}, 0, 2^31 - 1)
>+test:ok(not r and e == 'too many results to unpack', 'overflow check in unpack')
>+
>+os.exit(test:check() and 0 or 1)
>--
>2.34.1
--
Best regards,
Maxim Kokryashkin

[-- Attachment #2: Type: text/html, Size: 4364 bytes --]

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Sergey Kaplun via Tarantool-patches]

Hi, Maxim!

Thanks for the review!

On 06.09.22, Maxim Kokryashkin wrote:
> 
> Hi, Sergey!
> Thanks for the patch!
> Please consider my comments below:
>   
> >from Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>:
> > 
> >From: Mike Pall <mike>
> >
> >Thanks to HybridDog.
> >
> >When build with optimization compiler may throw away overflow check in
> >`unpack()` base library function.
> Typo: s/build with optimization/built with optimization,
> Also,  I think we should mention the specific optimization that causes the mentioned behavior
> since it is not mentioned in both the LuaJIT’s issue and the original Lua issue.

Fixed.
The new commit message is the following:
I mention the -fstrict-overflow flag as the crucial one (obviously
some more are needed).

| Fix overflow check in unpack().
|
| Thanks to HybridDog.
|
| (cherry picked from commit 179cf2eb84fef2b9a524469c3c8cc49363b8fb10)
|
| When built with -O2 optimization flag (includes -fstrict-overflow)
| compiler throws away overflow check in `unpack()` base library function.
|
| This patch prevents aforementioned error by comparing the unsigned
| amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
|
| Sergey Kaplun:
| * added the description and the test for the problem
|
| Part of tarantool/tarantool#7230

Branch is force-pushed.

> >
> >This patch prevents aforementioned error by comparing the unsigned
> >amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
> >
> >Sergey Kaplun:
> >* added the description and the test for the problem
> >
> >Part of tarantool/tarantool#7230
> >---
> >
> >Issue/PR:
> >*  https://github.com/LuaJIT/LuaJIT/pull/574
> >*  https://github.com/tarantool/tarantool/issues/7230
> >Branch:  https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
> >PR:  https://github.com/tarantool/tarantool/pull/7596
> >
> > src/lib_base.c | 6 ++++--
> > test/tarantool-tests/lj-574-overflow-unpack.test.lua | 12 ++++++++++++

<snipped>

> >--
> >2.34.1
> --
> Best regards,
> Maxim Kokryashkin

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Maxim Kokryashkin via Tarantool-patches]

[-- Attachment #1: Type: text/plain, Size: 2362 bytes --]


Hi, Sergey!
Thanks for the fixes!
LGTM
 
--
Best regards,
Maxim Kokryashkin
 
  
>Среда, 7 сентября 2022, 14:59 +03:00 от Sergey Kaplun <skaplun@tarantool.org>:
> 
>Hi, Maxim!
>
>Thanks for the review!
>
>On 06.09.22, Maxim Kokryashkin wrote:
>>
>> Hi, Sergey!
>> Thanks for the patch!
>> Please consider my comments below:
>>  
>> >from Sergey Kaplun via Tarantool-patches < tarantool-patches@dev.tarantool.org >:
>> > 
>> >From: Mike Pall <mike>
>> >
>> >Thanks to HybridDog.
>> >
>> >When build with optimization compiler may throw away overflow check in
>> >`unpack()` base library function.
>> Typo: s/build with optimization/built with optimization,
>> Also,  I think we should mention the specific optimization that causes the mentioned behavior
>> since it is not mentioned in both the LuaJIT’s issue and the original Lua issue.
>
>Fixed.
>The new commit message is the following:
>I mention the -fstrict-overflow flag as the crucial one (obviously
>some more are needed).
>
>| Fix overflow check in unpack().
>|
>| Thanks to HybridDog.
>|
>| (cherry picked from commit 179cf2eb84fef2b9a524469c3c8cc49363b8fb10)
>|
>| When built with -O2 optimization flag (includes -fstrict-overflow)
>| compiler throws away overflow check in `unpack()` base library function.
>|
>| This patch prevents aforementioned error by comparing the unsigned
>| amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
>|
>| Sergey Kaplun:
>| * added the description and the test for the problem
>|
>| Part of tarantool/tarantool#7230
>
>Branch is force-pushed.
>
>> >
>> >This patch prevents aforementioned error by comparing the unsigned
>> >amount of values to unpack with `LUAI_MAXCSTACK` instead of 0.
>> >
>> >Sergey Kaplun:
>> >* added the description and the test for the problem
>> >
>> >Part of tarantool/tarantool#7230
>> >---
>> >
>> >Issue/PR:
>> >*  https://github.com/LuaJIT/LuaJIT/pull/574
>> >*  https://github.com/tarantool/tarantool/issues/7230
>> >Branch:  https://github.com/tarantool/luajit/tree/skaplun/lj-574-overflow-unpack-full-ci
>> >PR:  https://github.com/tarantool/tarantool/pull/7596
>> >
>> > src/lib_base.c | 6 ++++--
>> > test/tarantool-tests/lj-574-overflow-unpack.test.lua | 12 ++++++++++++
>
><snipped>
>
>> >--
>> >2.34.1
>> --
>> Best regards,
>> Maxim Kokryashkin
>
>--
>Best regards,
>Sergey Kaplun
 

[-- Attachment #2: Type: text/html, Size: 3616 bytes --]

Re: [Tarantool-patches] [PATCH luajit] Fix overflow check in unpack().

[Igor Munkin via Tarantool-patches]

Sergey,

I've checked the patch into all long-term branches in tarantool/luajit
and bumped a new version in master, 2.10 and 1.10.

-- 
Best regards,
IM