[Tarantool-patches] [PATCH luajit] Avoid out-of-range number of results when compiling select(k, ...).

[Sergey Bronnikov]

Finally reproduced with the following CMake flags:

CMAKE_BUILD_TYPE="Debug"
CMAKE_C_COMPILER="clang"
CMAKE_EXPORT_COMPILE_COMMANDS:BOOL="TRUE"
LUAJIT_ENABLE_COVERAGE:BOOL="FALSE"
LUAJIT_ENABLE_GC64:BOOL="TRUE"
LUAJIT_USE_ASAN:BOOL="TRUE"
LUAJIT_USE_SYSMALLOC:BOOL="TRUE"
LUA_USE_APICHECK:BOOL="TRUE"
   LUA_USE_ASSERT:BOOL="TRUE"

On 2/8/24 12:36, Sergey Kaplun wrote:
> Hi, Sergey!
> Thanks for the review!
>
> On 08.02.24, Sergey Bronnikov wrote:
>> Hi, Sergey
>>
>> thanks for the patch
>>
>> couldn't reproduce a problem by provided test.
> As you can see in the test comment, the test failed under ASAN with a
> heap buffer overflow. Unfortunately, I didn't come up with an idea of
> how to observe the misbehaviour (too long IRs for the trace) without
> ASAN.
>
>> What compilation options I've used:
>>
>>
>> 1st attempt:
>>
>> cmake -S . -B build -DLUA_USE_ASSERT=ON -DLUA_USE_APICHECK=ON
>>
>>
>> 2nd attempt:
>>
>> CMAKE_BUILD_TYPE="Debug"
>> CMAKE_C_COMPILER="clang"
>> CMAKE_EXPORT_COMPILE_COMMANDS:BOOL="TRUE"
>> LUAJIT_ENABLE_COVERAGE:BOOL="FALSE"
>> LUAJIT_ENABLE_GC64:BOOL="TRUE"
>> LUAJIT_USE_ASAN:BOOL="FALSE"
>> LUAJIT_USE_SYSMALLOC:BOOL="FALSE"
>> LUA_USE_APICHECK:BOOL="TRUE"
>>     LUA_USE_ASSERT:BOOL="TRUE"
>>
>>
>> Sergey
>>
>> On 2/7/24 15:06, Sergey Kaplun wrote:
> <snipped>
>
>>> +local function varg_frame_wp()
>>> +  -- XXX: Need some constant negative value as the first argument
>>> +  -- of `select()` when recording the trace.
>>> +  -- Also, it should be huge enough to be greater than
>>> +  -- `J->maxslot`. The value on the first iteration is ignored.
>>> +  -- This will fail under ASAN due to a heap buffer overflow.
>>> +  varg_frame(recording and -(LJ_MAX_JSLOTS + 1) or 1)
>>> +end
> <snipped>
>
>>> +
>>> +test:done(true)