From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 79CF3198061E; Mon, 16 Mar 2026 11:31:37 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 79CF3198061E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1773649897; bh=3Iv8lqzKjBDso0zuU0lcLcGY8Or7idZlDsEZPKa92Vg=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=v4OJyv+pZE7tw5wf2Oq0EOVSC7ygwFI8/jhrqoED29Sm5YXfk0RkEuFfpXTjcEmfP 9tEPhwzXt3PW/nYmHeCKKeVz0P5sG3kMWkF2nD8fxaV8KWJLH5mhqVoVhQiv7rcImc ieSj2cPblJQ6XOuUP5kna1kuDIKzA138ABy7Nufw= Received: from send278.i.mail.ru (send278.i.mail.ru [95.163.59.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id C340C198063F for ; Mon, 16 Mar 2026 11:31:35 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org C340C198063F Received: by exim-smtp-64cdfc6c8d-dbqdq with esmtpa (envelope-from ) id 1w23MM-00000000CYo-1bhl; Mon, 16 Mar 2026 11:31:34 +0300 Content-Type: multipart/alternative; boundary="------------5vtFpNrm0hrqOBABxQwRqAqs" Message-ID: <6e2f5639-ca62-4031-ac29-b1b0851234ad@tarantool.org> Date: Mon, 16 Mar 2026 11:31:33 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Sergey Kaplun Cc: Sergey Bronnikov , tarantool-patches@dev.tarantool.org References: <1cc101f2-173e-47af-b373-cfb47868f313@tarantool.org> In-Reply-To: X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD91ABAE9865AC7DC889514E9E256EAD50689274261A8C811C4182A05F5380850407CFF2EF38D407E103DE06ABAFEAF6705DE8052AD4B04F133A2450BF1717AD9E01319EB51F02C4965 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7011EB7026DD4A9BAEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AC83A81C8FD4AD23D82A6BABE6F325AC2E85FA5F3EDFCBAA7353EFBB55337566BCD8AD057D84402C51B6B7F6F65A443E811AAA931155CD746714486E47E2E2B5389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0A29E2F051442AF778941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6E5E764EB5D94DBD4CC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB86D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7CD707F342D9BDC98731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A57C25FFCD403E776E5002B1117B3ED696CA25A73E8FBA5927D57BAD45EC4C5DE1823CB91A9FED034534781492E4B8EEAD3F23D99D50CED82FBDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE191716CD42B3DD1D34CAB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D659BE0128C2ECF7A6A3DC111035D6D18C6F553C74DE09A5E61B6DAFED71B038C4D31CF9D568DA219CCEB8341EE9D5BE9A0A93C72A309F8D9F947CBC589BAF59EBF43182C19AE57445386536EB022892E5344C41F94D744909CE2512F26BEC029E55448553D2254B8D95CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVdbVVJCphTR/THgRyBgpxKs= X-DA7885C5: A4CF1C287C447F45F255D290C0D534F9128F34BA460DA3F13449760DB544A65B3D7B8E61B11972955B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393520AF17B8A65FDE21B905A8D85B1366A484524C51C02A0BF098BF22D411779BCEF86D5F70DA33880E41E8EF7A07863ECB274557F927329BE2DDF8182D28ACDB545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3][v3] Add stack check to pcall/xpcall. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------5vtFpNrm0hrqOBABxQwRqAqs Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, Sergey, I've updated N_ITERATIONS and force-pushed the branch. Sergey On 3/12/26 20:19, Sergey Kaplun wrote: > Hi, Sergey! > Thanks for the fixes! > > LGTM, after fixing the last nit below. > > On 12.03.26, Sergey Bronnikov wrote: >> Hi, Sergey, >> >> thanks for review! See my comments below. >> >> Sergey >> >> On 3/12/26 13:16, Sergey Kaplun via Tarantool-patches wrote: >>> Hi, Sergey! >>> Thanks for the patch! >>> Please, fix my comments below. >>> >>> Don't forget to add the corresponding iterative changes. >>> >>> On 12.03.26, Sergey Bronnikov wrote: >>>> From: Mike Pall >>>> >>>> Analyzed by Peter Cawley. >>>> >>>> (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) >>>> >>>> The patch adds the stack check to fast functions `pcall()` and >>>> `xpcall()`. >>> Please add more verbose description: >>> >>> | (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) >>> | >>> | The `pcall()` and `xpcall()` calls in GC64 mode require 2 slots. This >>> | means that all arguments should be moved up during emitting of the frame >>> | link to the stack. Hence, this may cause stack overflow without the >>> | corresponding check. >>> | >>> | This patch adds the corresponding checks to the VM. Non-GC64 VMs are >>> | updated as well for the consistency. >> Updated >>>> Sergey Bronnikov: >>>> * added the description and the test for the problem >>>> >>>> Part of tarantool/tarantool#12134 >>>> --- >>>> src/vm_arm.dasc | 7 ++++ >>>> src/vm_arm64.dasc | 8 +++++ >>>> src/vm_mips.dasc | 10 +++++- >>>> src/vm_mips64.dasc | 14 ++++++-- >>>> src/vm_ppc.dasc | 9 +++++ >>>> src/vm_x64.dasc | 6 ++++ >>>> src/vm_x86.dasc | 6 ++++ >>>> ...048-fix-stack-checks-vararg-calls.test.lua | 35 ++++++++++++++++++- >>>> 8 files changed, 90 insertions(+), 5 deletions(-) >>>> > > >>>> diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc >>>> index 6c2975b4..4e60ee07 100644 >>>> --- a/src/vm_mips64.dasc >>>> +++ b/src/vm_mips64.dasc >>>> @@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx) >>>> |//-- Base library: catch errors ---------------------------------------- >>>> | >>>> |.ffunc pcall >>>> + | ld TMP1, L->maxstack >>>> + | daddu TMP2, BASE,NARGS8:RC >>>> + | sltu AT, TMP1, TMP2 >>>> + | bnez AT, ->fff_fallback >>>> + |. lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >>>> |daddiuNARGS8:RC,NARGS8:RC, -8 >>>> - | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >>>> |bltzNARGS8:RC, ->fff_fallback >>>> |. move TMP2, BASE >>>> | daddiu BASE, BASE, 16 >>>> @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx) >>>> |. nop >>>> | >>>> |.ffunc xpcall >>>> - |daddiuNARGS8:TMP0,NARGS8:RC, -16 >>> This neglets the first patch in the series. See the comment below. >>> >>>> - | ld CARG1, 0(BASE) >>>> + | ld TMP1, L->maxstack >>>> + | daddu TMP2, BASE,NARGS8:RC >>>> + | sltu AT, TMP1, TMP2 >>>> + | bnez AT, ->fff_fallback >>>> + |. ld CARG1, 0(BASE) >>>> + |daddiuNARGS8:RC,NARGS8:RC, -16 >>> This line is incorrect. This neglets the 1st patch in the series. >>> >>> It should be >>> | |daddiuNARGS8:TMP0,NARGS8:RC, -16 >> Right. However, probably we should leave this line near ".ffunc xpcall". >> What do you think? > Why? This break the `maxstack` check (since the RC is differs before the > addition with TMP2). Ok, let's leave as is, seems I took a look on intermediate version, before applying changes you requested on review. The current patch looks fine for me: @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx)    |.  nop    |    |.ffunc xpcall +  |  ld TMP1, L->maxstack +  |  daddu TMP2, BASE, NARGS8:RC +  |  sltu AT, TMP1, TMP2 +  |  bnez AT, ->fff_fallback +  |.  ld CARG1, 0(BASE)    |  daddiu NARGS8:TMP0, NARGS8:RC, -16 -  |  ld CARG1, 0(BASE)    |   ld CARG2, 8(BASE)    |    bltz NARGS8:TMP0, ->fff_fallback    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) > > See the latest LuaJIT version: > https://github.com/LuaJIT/LuaJIT/blob/659a61693aa3b87661864ad0f12eee14c865cd7f/src/vm_mips64.dasc#L1450 > >> Now updated as the following: >> >> --- a/src/vm_mips64.dasc >> +++ b/src/vm_mips64.dasc >> @@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx) >>    |  sltu AT, TMP1, TMP2 >>    |  bnez AT, ->fff_fallback >>    |.  ld CARG1, 0(BASE) >> -  |  daddiuNARGS8:RC,NARGS8:RC, -16 >> +  |  daddiuNARGS8:TMP0,NARGS8:RC, -16 >>    |   ld CARG2, 8(BASE) >>    |    bltzNARGS8:TMP0, ->fff_fallback >>    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) >> >>>> | ld CARG2, 8(BASE) >>>> |bltzNARGS8:TMP0, ->fff_fallback >>>> |. lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) > > >>>> diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >>>> index 3a8ad63d..ad8b151b 100644 >>>> --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >>>> +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >>>> @@ -5,7 +5,7 @@ local tap = require('tap') >>>> -- Seealsohttps://github.com/LuaJIT/LuaJIT/issues/1048. >>>> local test = tap.test('lj-1048-fix-stack-checks-vararg-calls') >>>> >>>> -test:plan(2) >>>> +test:plan(5) >>>> >>>> -- The test case demonstrates a segmentation fault due to stack >>>> -- overflow by recursive calling `pcall()`. The functions are >>>> @@ -50,4 +50,37 @@ pcall(coroutine.wrap(looper), prober_2, 0) >>>> >>>> test:ok(true, 'no stack overflow with metamethod') >>>> >>>> +-- The testcases demonstrates a stack overflow in >>>> +-- `pcall()`/xpcall()` triggered using metamethod `__call`. >>>> + >>>> +t = coroutine.wrap(setmetatable)({}, { __call = pcall }) >>> I've meant the following: >>> >>> | t = setmetatable({}, { __call = pcall }) >>> | coroutine.wrap(function() t() end)() >>> >> Updated >> >> @@ -53,7 +53,8 @@test:ok(true, 'no stack overflow with metamethod') >>  -- The testcases demonstrates a stack overflow in >>  -- `pcall()`/xpcall()` triggered using metamethod `__call`. >> >> -t = coroutine.wrap(setmetatable)({}, { __call = pcall }) >> +t = setmetatable({}, { __call = pcall }) >> +coroutine.wrap(function() t() end)() >> >> test:ok(true, 'no stack overflow with metamethod __call with pcall()') >> >> >>>> + >>>> +test:ok(true, 'no stack overflow with metamethod __call with pcall()') >>>> + >>>> +t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) >>> I've meant the following: >>> >>> | t = setmetatable({}, { __call = xpcall }) >>> | coroutine.wrap(function() t() end)() >>> >>> But this won't work since the second amount of xpcall must be the >>> function. So, this test case is invalid. We must to duplicate the second >>> approach with `xpcall()` >>> >>> This works fine. >>> | LUA_PATH="src/?.lua;;" gdb --args src/luajit -e ' >>> | local t = {} >>> | local function xpcall_wrapper() >>> | return xpcall(unpack(t)) >>> | end >>> | >>> | local N_ITERATIONS = 200 >>> | >>> | for i = 1, N_ITERATIONS do >>> | t[i], t[i + 1], t[i + 2] = xpcall, type, {} >>> | coroutine.wrap(xpcall_wrapper)() >>> | end >>> | ' >> Updated: >> >> diff --git >> a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> index 6395dfaa..825568f9 100644 >> --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> @@ -58,7 +58,17 @@ coroutine.wrap(function() t() end)() >> >> test:ok(true, 'no stack overflow with metamethod __call with pcall()') >> >> -t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) >> +t = {} >> +local function xpcall_wrapper() >> +  return xpcall(unpack(t)) >> +end >> + >> +local N_ITERATIONS_1 = 200 > Why do we need two variables with the same value of iterations? > Let's use N_ITERATIONS with the comment for xpcall and pcall. Updated: diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua index 825568f9..036d53e9 100644 --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -63,9 +63,13 @@ local function xpcall_wrapper()    return xpcall(unpack(t))  end -local N_ITERATIONS_1 = 200 +-- The problems are only reproduced on LuaJIT GC64 and is better +-- reproduced under Valgrind than AddressSanitizer. The chosen +-- value was found experimentally and always results in an attempt +-- to write beyond the allocated memory. +local N_ITERATIONS = 200 -for i = 1, N_ITERATIONS_1 do +for i = 1, N_ITERATIONS do    t[i], t[i + 1], t[i + 2] = xpcall, type, {}    coroutine.wrap(xpcall_wrapper)()  end @@ -81,13 +85,7 @@ local function pcall_wrapper()    return pcall(unpack(t))  end --- The problem is only reproduced on LuaJIT GC64 and is better --- reproduced under Valgrind than AddressSanitizer. The chosen --- value was found experimentally and always results in an attempt --- to write beyond the allocated memory. -local N_ITERATIONS_2 = 200 - -for i = 1, N_ITERATIONS_2 do +for i = 1, N_ITERATIONS do    t[i], t[i + 1], t[i + 2] = pcall, type, {}    coroutine.wrap(pcall_wrapper)()  end > >> + >> +for i = 1, N_ITERATIONS_1 do >> +  t[i], t[i + 1], t[i + 2] = xpcall, type, {} >> +  coroutine.wrap(xpcall_wrapper)() >> +end >> >> test:ok(true, 'no stack overflow with metamethod __call with xpcall()') >> >> @@ -67,19 +77,19 @@test:ok(true, 'no stack overflow with metamethod >> __call with xpcall()') >>  -- triggered using `unpack()`. >> >>  t = {} >> -local function f() >> +local function pcall_wrapper() >>    return pcall(unpack(t)) >>  end >> >> --- The problem is only reproduced on LuaJIT GC64 and is best >> +-- The problem is only reproduced on LuaJIT GC64 and is better >>  -- reproduced under Valgrind than AddressSanitizer. The chosen >>  -- value was found experimentally and always results in an attempt >>  -- to write beyond the allocated memory. >> -local N_ITERATIONS = 200 >> +local N_ITERATIONS_2 = 200 >> >> -for i = 1, N_ITERATIONS do >> +for i = 1, N_ITERATIONS_2 do >>    t[i], t[i + 1], t[i + 2] = pcall, type, {} >> -  coroutine.wrap(f)() >> +  coroutine.wrap(pcall_wrapper)() >>  end >> >> test:ok(true, 'no stack overflow with unpacked pcalls') >> >>>> + >>>> +test:ok(true, 'no stack overflow with metamethod __call with xpcall()') >>>> + >>>> +-- The testcase demonstrates a stack overflow in >>>> +-- `pcall()`/`xpcall()` similar to the first testcase, but it is >>>> +-- triggered using `unpack()`. >>>> + >>>> +t = {} >>>> +local function f() > > >>>> -- >>>> 2.43.0 >>>> --------------5vtFpNrm0hrqOBABxQwRqAqs Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi, Sergey,

I've updated N_ITERATIONS and force-pushed the branch.

Sergey

On 3/12/26 20:19, Sergey Kaplun wrote:
Hi, Sergey!
Thanks for the fixes!

LGTM, after fixing the last nit below.

On 12.03.26, Sergey Bronnikov wrote:

Hi, Sergey,

thanks for review! See my comments below.

Sergey

On 3/12/26 13:16, Sergey Kaplun via Tarantool-patches wrote:

Hi, Sergey!
Thanks for the patch!
Please, fix my comments below.

Don't forget to add the corresponding iterative changes.

On 12.03.26, Sergey Bronnikov wrote:

From: Mike Pall <mike>

Analyzed by Peter Cawley.

(cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)

The patch adds the stack check to fast functions `pcall()` and
`xpcall()`.

Please add more verbose description:

| (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)
|
| The `pcall()` and `xpcall()` calls in GC64 mode require 2 slots. This
| means that all arguments should be moved up during emitting of the frame
| link to the stack. Hence, this may cause stack overflow without the
| corresponding check.
|
| This patch adds the corresponding checks to the VM. Non-GC64 VMs are
| updated as well for the consistency.

Updated

Sergey Bronnikov:
* added the description and the test for the problem

Part of tarantool/tarantool#12134
---
  src/vm_arm.dasc                               |  7 ++++
  src/vm_arm64.dasc                             |  8 +++++
  src/vm_mips.dasc                              | 10 +++++-
  src/vm_mips64.dasc                            | 14 ++++++--
  src/vm_ppc.dasc                               |  9 +++++
  src/vm_x64.dasc                               |  6 ++++
  src/vm_x86.dasc                               |  6 ++++
  ...048-fix-stack-checks-vararg-calls.test.lua | 35 ++++++++++++++++++-
  8 files changed, 90 insertions(+), 5 deletions(-)


<snipped>


diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc
index 6c2975b4..4e60ee07 100644
--- a/src/vm_mips64.dasc
+++ b/src/vm_mips64.dasc
@@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx)
    |//-- Base library: catch errors ----------------------------------------
    |
    |.ffunc pcall
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE,NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
    |  daddiuNARGS8:RC,NARGS8:RC, -8
-  |  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
    |  bltzNARGS8:RC, ->fff_fallback
    |.   move TMP2, BASE
    |   daddiu BASE, BASE, 16
@@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx)
    |.  nop
    |
    |.ffunc xpcall
-  |  daddiuNARGS8:TMP0,NARGS8:RC, -16

This neglets the first patch in the series. See the comment below.


-  |  ld CARG1, 0(BASE)
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE,NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  ld CARG1, 0(BASE)
+  |  daddiuNARGS8:RC,NARGS8:RC, -16

This line is incorrect. This neglets the 1st patch in the series.

It should be
| |  daddiuNARGS8:TMP0,NARGS8:RC, -16

Right. However, probably we should leave this line near ".ffunc xpcall". 
What do you think?

Why? This break the `maxstack` check (since the RC is differs before the
addition with TMP2).

Ok, let's leave as is, seems I took a look on intermediate version, before applying changes you requested on review.

The current patch looks fine for me:

@@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx)
   |.  nop
   |
   |.ffunc xpcall
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE, NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  ld CARG1, 0(BASE)
   |  daddiu NARGS8:TMP0, NARGS8:RC, -16
-  |  ld CARG1, 0(BASE)
   |   ld CARG2, 8(BASE)
   |    bltz NARGS8:TMP0, ->fff_fallback
   |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)


See the latest LuaJIT version:
https://github.com/LuaJIT/LuaJIT/blob/659a61693aa3b87661864ad0f12eee14c865cd7f/src/vm_mips64.dasc#L1450


Now updated as the following:

--- a/src/vm_mips64.dasc
+++ b/src/vm_mips64.dasc
@@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx)
    |  sltu AT, TMP1, TMP2
    |  bnez AT, ->fff_fallback
    |.  ld CARG1, 0(BASE)
-  |  daddiu NARGS8:RC, NARGS8:RC, -16
+  |  daddiu NARGS8:TMP0, NARGS8:RC, -16
    |   ld CARG2, 8(BASE)
    |    bltz NARGS8:TMP0, ->fff_fallback
    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)



          

            
    |   ld CARG2, 8(BASE)
    |    bltzNARGS8:TMP0, ->fff_fallback
    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)


          

        

      

      
<snipped>



      

        

          

            
diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
index 3a8ad63d..ad8b151b 100644
--- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
+++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
@@ -5,7 +5,7 @@ local tap = require('tap')
  -- See alsohttps://github.com/LuaJIT/LuaJIT/issues/1048.
  local test = tap.test('lj-1048-fix-stack-checks-vararg-calls')
  
-test:plan(2)
+test:plan(5)
  
  -- The test case demonstrates a segmentation fault due to stack
  -- overflow by recursive calling `pcall()`. The functions are
@@ -50,4 +50,37 @@ pcall(coroutine.wrap(looper), prober_2, 0)
  
  test:ok(true, 'no stack overflow with metamethod')
  
+-- The testcases demonstrates a stack overflow in
+-- `pcall()`/xpcall()` triggered using metamethod `__call`.
+
+t = coroutine.wrap(setmetatable)({}, { __call = pcall })


          

          
I've meant the following:

| t = setmetatable({}, { __call = pcall })
| coroutine.wrap(function() t() end)()



        

        
Updated

@@ -53,7 +53,8 @@ test:ok(true, 'no stack overflow with metamethod')
  -- The testcases demonstrates a stack overflow in
  -- `pcall()`/xpcall()` triggered using metamethod `__call`.

-t = coroutine.wrap(setmetatable)({}, { __call = pcall })
+t = setmetatable({}, { __call = pcall })
+coroutine.wrap(function() t() end)()

test:ok(true, 'no stack overflow with metamethod __call with pcall()')




        

          

            
+
+test:ok(true, 'no stack overflow with metamethod __call with pcall()')
+
+t = coroutine.wrap(setmetatable)({}, { __call = xpcall })


          

          
I've meant the following:

| t = setmetatable({}, { __call = xpcall })
| coroutine.wrap(function() t() end)()

But this won't work since the second amount of xpcall must be the
function. So, this test case is invalid. We must to duplicate the second
approach with `xpcall()`

This works fine.
| LUA_PATH="src/?.lua;;" gdb --args src/luajit -e '
| local t = {}
| local function xpcall_wrapper()
|   return xpcall(unpack(t))
| end
|
| local N_ITERATIONS = 200
|
| for i = 1, N_ITERATIONS do
|   t[i], t[i + 1], t[i + 2] = xpcall, type, {}
|   coroutine.wrap(xpcall_wrapper)()
| end
| '


        

        
Updated:

diff --git 
a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua 
b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
index 6395dfaa..825568f9 100644
--- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
+++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
@@ -58,7 +58,17 @@ coroutine.wrap(function() t() end)()

test:ok(true, 'no stack overflow with metamethod __call with pcall()')

-t = coroutine.wrap(setmetatable)({}, { __call = xpcall })
+t = {}
+local function xpcall_wrapper()
+  return xpcall(unpack(t))
+end
+
+local N_ITERATIONS_1 = 200


      

      
Why do we need two variables with the same value of iterations?
Let's use N_ITERATIONS with the comment for xpcall and pcall.

    

    
Updated:

    
diff --git
      a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      index 825568f9..036d53e9 100644

      ---
      a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      +++
      b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      @@ -63,9 +63,13 @@ local function xpcall_wrapper()

         return xpcall(unpack(t))

       end

       

      -local N_ITERATIONS_1 = 200

      +-- The problems are only reproduced on LuaJIT GC64 and is better

      +-- reproduced under Valgrind than AddressSanitizer. The chosen

      +-- value was found experimentally and always results in an
      attempt

      +-- to write beyond the allocated memory.

      +local N_ITERATIONS = 200

       

      -for i = 1, N_ITERATIONS_1 do

      +for i = 1, N_ITERATIONS do

         t[i], t[i + 1], t[i + 2] = xpcall, type, {}

         coroutine.wrap(xpcall_wrapper)()

       end

      @@ -81,13 +85,7 @@ local function pcall_wrapper()

         return pcall(unpack(t))

       end

       

      --- The problem is only reproduced on LuaJIT GC64 and is better

      --- reproduced under Valgrind than AddressSanitizer. The chosen

      --- value was found experimentally and always results in an
      attempt

      --- to write beyond the allocated memory.

      -local N_ITERATIONS_2 = 200

      -

      -for i = 1, N_ITERATIONS_2 do

      +for i = 1, N_ITERATIONS do

         t[i], t[i + 1], t[i + 2] = pcall, type, {}

         coroutine.wrap(pcall_wrapper)()

       end

      

    

    

      



      

        
+
+for i = 1, N_ITERATIONS_1 do
+  t[i], t[i + 1], t[i + 2] = xpcall, type, {}
+  coroutine.wrap(xpcall_wrapper)()
+end

test:ok(true, 'no stack overflow with metamethod __call with xpcall()')

@@ -67,19 +77,19 @@ test:ok(true, 'no stack overflow with metamethod 
__call with xpcall()')
  -- triggered using `unpack()`.

  t = {}
-local function f()
+local function pcall_wrapper()
    return pcall(unpack(t))
  end

--- The problem is only reproduced on LuaJIT GC64 and is best
+-- The problem is only reproduced on LuaJIT GC64 and is better
  -- reproduced under Valgrind than AddressSanitizer. The chosen
  -- value was found experimentally and always results in an attempt
  -- to write beyond the allocated memory.
-local N_ITERATIONS = 200
+local N_ITERATIONS_2 = 200

-for i = 1, N_ITERATIONS do
+for i = 1, N_ITERATIONS_2 do
    t[i], t[i + 1], t[i + 2] = pcall, type, {}
-  coroutine.wrap(f)()
+  coroutine.wrap(pcall_wrapper)()
  end

test:ok(true, 'no stack overflow with unpacked pcalls')



        

          

            
+
+test:ok(true, 'no stack overflow with metamethod __call with xpcall()')
+
+-- The testcase demonstrates a stack overflow in
+-- `pcall()`/`xpcall()` similar to the first testcase, but it is
+-- triggered using `unpack()`.
+
+t = {}
+local function f()


          

        

      

      
<snipped>



      

        

          

            
-- 
2.43.0



          

        

      

      

    

  
  


--------------5vtFpNrm0hrqOBABxQwRqAqs--