From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 621D01B7C412; Thu, 12 Mar 2026 19:23:23 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 621D01B7C412 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1773332603; bh=j4JPfHD63TvDWh4YefTc0pNVWGa7Nk6Xp1E80iXPmI8=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=iTFaQKynx6qr08OEG6TrNmPXF/ADoxVaGfyEAdWBhYm2Su1/5jZ/2EDn5HmANlKf7 1Z2fiyb4gljnU6hqjXT8rBzlRM0LmbajwVS7RS0EakEpArN+QuHmhhxaS2k+hHxYWU 0Yold3ByyPICe5OrNvTNnkfYY+fSH0opgVL8k4rM= Received: from send126.i.mail.ru (send126.i.mail.ru [89.221.237.221]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 1FC3B1A39297 for ; Thu, 12 Mar 2026 19:23:22 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1FC3B1A39297 Received: by exim-smtp-64cdfc6c8d-fqxjz with esmtpa (envelope-from ) id 1w0ioj-000000008zq-0UPs; Thu, 12 Mar 2026 19:23:21 +0300 Content-Type: multipart/alternative; boundary="------------ZK4VGZBcUlvhjCvoLK3LlcnL" Message-ID: <1cc101f2-173e-47af-b373-cfb47868f313@tarantool.org> Date: Thu, 12 Mar 2026 19:23:20 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Sergey Kaplun , Sergey Bronnikov Cc: tarantool-patches@dev.tarantool.org References: In-Reply-To: X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91ABAE9865AC7DC8818B4CD52367FE98173B6651699D6E8DA182A05F5380850403DBCC760DF36B9FB3DE06ABAFEAF6705481484AE7D723D8804C6C4287C95B5602F7B8F85B30CF495 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE70B8ADF238913687CB287FD4696A6DC2FA8DF7F3B2552694A4E2F5AFA99E116B42401471946AA11AFA417C69337E82CC2BCF491FFA38154B6B5C8C57E37DE458B543C44CD4AFD0029C6CDE5D1141D2B1CB0F1D63548D0BF3DEA7D775D469A744D6581DF97CA9411EEE9D0DA57551CE7078941B15DA834481FA18204E546F3947C1D471462564A2E19F6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F7900637CAEE156C82D3D7D9389733CBF5DBD5E9B5C8C57E37DE458BD9DD9810294C998ED8FC6C240DEA76428AA50765F79006377F2651C28026B19DD81D268191BDAD3DBD4B6F7A4D31EC0BE2F48590F00D11D6D81D268191BDAD3D78DA827A17800CE77728C553F82A4578EC76A7562686271ED91E3A1F190DE8FD2E808ACE2090B5E14AD6D5ED66289B5259CC434672EE63711DD303D21008E298D5E8D9A59859A8B6B372FE9A2E580EFC725E5C173C3A84C3C9EEE74C166EF7BC35872C767BF85DA2F004C90652538430E4A6367B16DE6309 X-C1DE0DAB: 0D63561A33F958A5237EE80C98B65F1F5002B1117B3ED69690BB8B71C3F67215B91D2EB2DEE3878C823CB91A9FED034534781492E4B8EEADD3CF082B023A3D3CBDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE191716CD42B3DD1D34CAB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D6595C7B19A0CDDD20631F864D01557F4FF6C41F97DBFCD80ADE29B1DFA8979E2F103E3DD955D6BB9A34B8341EE9D5BE9A0AFA743F9E1D9C8B141E671C21D2ABE61A959443F311C8A44D6536EB022892E5344C41F94D744909CE2512F26BEC029E55448553D2254B8D95CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVdbVVJCphTR/1khTRi/Mh6U= X-Mailru-Sender: C4F68CFF4024C8867DFDF7C7F2588458CB9260F41CECE5330841F56765D88C4032064735FD40A10ACDE10F0E9DBD3FFB645D15D82EE4B272BD6E4642A116CA93524AA66B5ACBE6721EF430B9A63E2A504198E0F3ECE9B5443453F38A29522196 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3][v3] Add stack check to pcall/xpcall. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------ZK4VGZBcUlvhjCvoLK3LlcnL Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, Sergey, thanks for review! See my comments below. Sergey On 3/12/26 13:16, Sergey Kaplun via Tarantool-patches wrote: > Hi, Sergey! > Thanks for the patch! > Please, fix my comments below. > > Don't forget to add the corresponding iterative changes. > > On 12.03.26, Sergey Bronnikov wrote: >> From: Mike Pall >> >> Analyzed by Peter Cawley. >> >> (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) >> >> The patch adds the stack check to fast functions `pcall()` and >> `xpcall()`. > Please add more verbose description: > > | (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) > | > | The `pcall()` and `xpcall()` calls in GC64 mode require 2 slots. This > | means that all arguments should be moved up during emitting of the frame > | link to the stack. Hence, this may cause stack overflow without the > | corresponding check. > | > | This patch adds the corresponding checks to the VM. Non-GC64 VMs are > | updated as well for the consistency. Updated >> Sergey Bronnikov: >> * added the description and the test for the problem >> >> Part of tarantool/tarantool#12134 >> --- >> src/vm_arm.dasc | 7 ++++ >> src/vm_arm64.dasc | 8 +++++ >> src/vm_mips.dasc | 10 +++++- >> src/vm_mips64.dasc | 14 ++++++-- >> src/vm_ppc.dasc | 9 +++++ >> src/vm_x64.dasc | 6 ++++ >> src/vm_x86.dasc | 6 ++++ >> ...048-fix-stack-checks-vararg-calls.test.lua | 35 ++++++++++++++++++- >> 8 files changed, 90 insertions(+), 5 deletions(-) >> >> diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc >> index 7095e660..efe9dcb2 100644 >> --- a/src/vm_arm.dasc >> +++ b/src/vm_arm.dasc > > >> diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc >> index 5ef37243..074c1f31 100644 >> --- a/src/vm_arm64.dasc >> +++ b/src/vm_arm64.dasc > > >> diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc >> index 32caabf7..69d09d52 100644 >> --- a/src/vm_mips.dasc >> +++ b/src/vm_mips.dasc > > >> diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc >> index 6c2975b4..4e60ee07 100644 >> --- a/src/vm_mips64.dasc >> +++ b/src/vm_mips64.dasc >> @@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx) >> |//-- Base library: catch errors ---------------------------------------- >> | >> |.ffunc pcall >> + | ld TMP1, L->maxstack >> + | daddu TMP2, BASE,NARGS8:RC >> + | sltu AT, TMP1, TMP2 >> + | bnez AT, ->fff_fallback >> + |. lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >> | daddiuNARGS8:RC,NARGS8:RC, -8 >> - | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >> | bltzNARGS8:RC, ->fff_fallback >> |. move TMP2, BASE >> | daddiu BASE, BASE, 16 >> @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx) >> |. nop >> | >> |.ffunc xpcall >> - | daddiuNARGS8:TMP0,NARGS8:RC, -16 > This neglets the first patch in the series. See the comment below. > >> - | ld CARG1, 0(BASE) >> + | ld TMP1, L->maxstack >> + | daddu TMP2, BASE,NARGS8:RC >> + | sltu AT, TMP1, TMP2 >> + | bnez AT, ->fff_fallback >> + |. ld CARG1, 0(BASE) >> + | daddiuNARGS8:RC,NARGS8:RC, -16 > This line is incorrect. This neglets the 1st patch in the series. > > It should be > | | daddiuNARGS8:TMP0,NARGS8:RC, -16 Right. However, probably we should leave this line near ".ffunc xpcall". What do you think? Now updated as the following: --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx)    |  sltu AT, TMP1, TMP2    |  bnez AT, ->fff_fallback    |.  ld CARG1, 0(BASE) -  |  daddiu NARGS8:RC, NARGS8:RC, -16 +  |  daddiu NARGS8:TMP0, NARGS8:RC, -16    |   ld CARG2, 8(BASE)    |    bltz NARGS8:TMP0, ->fff_fallback    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) > >> | ld CARG2, 8(BASE) >> | bltzNARGS8:TMP0, ->fff_fallback >> |. lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) >> diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc >> index 980ad897..f2ea933b 100644 >> --- a/src/vm_ppc.dasc >> +++ b/src/vm_ppc.dasc > > >> diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc >> index 8b6781a6..c57b76b7 100644 >> --- a/src/vm_x64.dasc >> +++ b/src/vm_x64.dasc > > >> diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc >> index 7c11c78e..36804d11 100644 >> --- a/src/vm_x86.dasc >> +++ b/src/vm_x86.dasc > > >> diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> index 3a8ad63d..ad8b151b 100644 >> --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> @@ -5,7 +5,7 @@ local tap = require('tap') >> -- See alsohttps://github.com/LuaJIT/LuaJIT/issues/1048. >> local test = tap.test('lj-1048-fix-stack-checks-vararg-calls') >> >> -test:plan(2) >> +test:plan(5) >> >> -- The test case demonstrates a segmentation fault due to stack >> -- overflow by recursive calling `pcall()`. The functions are >> @@ -50,4 +50,37 @@ pcall(coroutine.wrap(looper), prober_2, 0) >> >> test:ok(true, 'no stack overflow with metamethod') >> >> +-- The testcases demonstrates a stack overflow in >> +-- `pcall()`/xpcall()` triggered using metamethod `__call`. >> + >> +t = coroutine.wrap(setmetatable)({}, { __call = pcall }) > I've meant the following: > > | t = setmetatable({}, { __call = pcall }) > | coroutine.wrap(function() t() end)() > Updated @@ -53,7 +53,8 @@ test:ok(true, 'no stack overflow with metamethod')  -- The testcases demonstrates a stack overflow in  -- `pcall()`/xpcall()` triggered using metamethod `__call`. -t = coroutine.wrap(setmetatable)({}, { __call = pcall }) +t = setmetatable({}, { __call = pcall }) +coroutine.wrap(function() t() end)() test:ok(true, 'no stack overflow with metamethod __call with pcall()') >> + >> +test:ok(true, 'no stack overflow with metamethod __call with pcall()') >> + >> +t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) > I've meant the following: > > | t = setmetatable({}, { __call = xpcall }) > | coroutine.wrap(function() t() end)() > > But this won't work since the second amount of xpcall must be the > function. So, this test case is invalid. We must to duplicate the second > approach with `xpcall()` > > This works fine. > | LUA_PATH="src/?.lua;;" gdb --args src/luajit -e ' > | local t = {} > | local function xpcall_wrapper() > | return xpcall(unpack(t)) > | end > | > | local N_ITERATIONS = 200 > | > | for i = 1, N_ITERATIONS do > | t[i], t[i + 1], t[i + 2] = xpcall, type, {} > | coroutine.wrap(xpcall_wrapper)() > | end > | ' Updated: diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua index 6395dfaa..825568f9 100644 --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -58,7 +58,17 @@ coroutine.wrap(function() t() end)() test:ok(true, 'no stack overflow with metamethod __call with pcall()') -t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) +t = {} +local function xpcall_wrapper() +  return xpcall(unpack(t)) +end + +local N_ITERATIONS_1 = 200 + +for i = 1, N_ITERATIONS_1 do +  t[i], t[i + 1], t[i + 2] = xpcall, type, {} +  coroutine.wrap(xpcall_wrapper)() +end test:ok(true, 'no stack overflow with metamethod __call with xpcall()') @@ -67,19 +77,19 @@ test:ok(true, 'no stack overflow with metamethod __call with xpcall()')  -- triggered using `unpack()`.  t = {} -local function f() +local function pcall_wrapper()    return pcall(unpack(t))  end --- The problem is only reproduced on LuaJIT GC64 and is best +-- The problem is only reproduced on LuaJIT GC64 and is better  -- reproduced under Valgrind than AddressSanitizer. The chosen  -- value was found experimentally and always results in an attempt  -- to write beyond the allocated memory. -local N_ITERATIONS = 200 +local N_ITERATIONS_2 = 200 -for i = 1, N_ITERATIONS do +for i = 1, N_ITERATIONS_2 do    t[i], t[i + 1], t[i + 2] = pcall, type, {} -  coroutine.wrap(f)() +  coroutine.wrap(pcall_wrapper)()  end test:ok(true, 'no stack overflow with unpacked pcalls') >> + >> +test:ok(true, 'no stack overflow with metamethod __call with xpcall()') >> + >> +-- The testcase demonstrates a stack overflow in >> +-- `pcall()`/`xpcall()` similar to the first testcase, but it is >> +-- triggered using `unpack()`. >> + >> +t = {} >> +local function f() > Lets name it `pcall_wrapper()` > @@ -66,7 +68,7 @@ test:ok(true, 'no stack overflow with metamethod __call with xpcall()')  -- triggered using `unpack()`.  t = {} -local function f() +local function pcall_wrapper()    return pcall(unpack(t))  end @@ -78,7 +80,7 @@ local N_ITERATIONS = 200  for i = 1, N_ITERATIONS do    t[i], t[i + 1], t[i + 2] = pcall, type, {} -  coroutine.wrap(f)() +  coroutine.wrap(pcall_wrapper)()  end test:ok(true, 'no stack overflow with unpacked pcalls') >> + return pcall(unpack(t)) >> +end >> + >> +-- The problem is only reproduced on LuaJIT GC64 and is best > Typo: s/best/better/    return pcall(unpack(t))  end --- The problem is only reproduced on LuaJIT GC64 and is best +-- The problem is only reproduced on LuaJIT GC64 and is better  -- reproduced under Valgrind than AddressSanitizer. The chosen  -- value was found experimentally and always results in an attempt  -- to write beyond the allocated memory. > >> +-- reproduced under Valgrind than AddressSanitizer. The chosen >> +-- value was found experimentally and always results in an attempt >> +-- to write beyond the allocated memory. >> +local N_ITERATIONS = 200 >> + >> +for i = 1, N_ITERATIONS do >> + t[i], t[i + 1], t[i + 2] = pcall, type, {} >> + coroutine.wrap(f)() >> +end >> + >> +test:ok(true, 'no stack overflow with unpacked pcalls') >> + >> test:done(true) >> -- >> 2.43.0 >> --------------ZK4VGZBcUlvhjCvoLK3LlcnL Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi, Sergey,

thanks for review! See my comments below.

Sergey

On 3/12/26 13:16, Sergey Kaplun via Tarantool-patches wrote:
Hi, Sergey!
Thanks for the patch!
Please, fix my comments below.

Don't forget to add the corresponding iterative changes.

On 12.03.26, Sergey Bronnikov wrote:

From: Mike Pall <mike>

Analyzed by Peter Cawley.

(cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)

The patch adds the stack check to fast functions `pcall()` and
`xpcall()`.

Please add more verbose description:

| (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)
|
| The `pcall()` and `xpcall()` calls in GC64 mode require 2 slots. This
| means that all arguments should be moved up during emitting of the frame
| link to the stack. Hence, this may cause stack overflow without the
| corresponding check.
|
| This patch adds the corresponding checks to the VM. Non-GC64 VMs are
| updated as well for the consistency.
Updated 

      

        
Sergey Bronnikov:
* added the description and the test for the problem

Part of tarantool/tarantool#12134
---
 src/vm_arm.dasc                               |  7 ++++
 src/vm_arm64.dasc                             |  8 +++++
 src/vm_mips.dasc                              | 10 +++++-
 src/vm_mips64.dasc                            | 14 ++++++--
 src/vm_ppc.dasc                               |  9 +++++
 src/vm_x64.dasc                               |  6 ++++
 src/vm_x86.dasc                               |  6 ++++
 ...048-fix-stack-checks-vararg-calls.test.lua | 35 ++++++++++++++++++-
 8 files changed, 90 insertions(+), 5 deletions(-)

diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc
index 7095e660..efe9dcb2 100644
--- a/src/vm_arm.dasc
+++ b/src/vm_arm.dasc


      

      
<snipped>



      

        
diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc
index 5ef37243..074c1f31 100644
--- a/src/vm_arm64.dasc
+++ b/src/vm_arm64.dasc


      

      
<snipped>



      

        
diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc
index 32caabf7..69d09d52 100644
--- a/src/vm_mips.dasc
+++ b/src/vm_mips.dasc


      

      
<snipped>



      

        
diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc
index 6c2975b4..4e60ee07 100644
--- a/src/vm_mips64.dasc
+++ b/src/vm_mips64.dasc
@@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx)
   |//-- Base library: catch errors ----------------------------------------
   |
   |.ffunc pcall
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE, NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
   |  daddiu NARGS8:RC, NARGS8:RC, -8
-  |  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
   |  bltz NARGS8:RC, ->fff_fallback
   |.   move TMP2, BASE
   |   daddiu BASE, BASE, 16
@@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx)
   |.  nop
   |
   |.ffunc xpcall
-  |  daddiu NARGS8:TMP0, NARGS8:RC, -16


      

      
This neglets the first patch in the series. See the comment below.



      

        
-  |  ld CARG1, 0(BASE)
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE, NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  ld CARG1, 0(BASE)


      

      

      

        
+  |  daddiu NARGS8:RC, NARGS8:RC, -16


      

      
This line is incorrect. This neglets the 1st patch in the series.

It should be
| |  daddiu NARGS8:TMP0, NARGS8:RC, -16


    

    
Right. However, probably we should leave this line near ".ffunc
      xpcall". What do you think?

    
Now updated as the following:

    
--- a/src/vm_mips64.dasc

      +++ b/src/vm_mips64.dasc

      @@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx)

         |  sltu AT, TMP1, TMP2

         |  bnez AT, ->fff_fallback

         |.  ld CARG1, 0(BASE)

      -  |  daddiu NARGS8:RC, NARGS8:RC, -16

      +  |  daddiu NARGS8:TMP0, NARGS8:RC, -16

         |   ld CARG2, 8(BASE)

         |    bltz NARGS8:TMP0, ->fff_fallback

         |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)

      

    

    

      



      

        
   |   ld CARG2, 8(BASE)
   |    bltz NARGS8:TMP0, ->fff_fallback
   |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)
diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc
index 980ad897..f2ea933b 100644
--- a/src/vm_ppc.dasc
+++ b/src/vm_ppc.dasc


      

      
<snipped>



      

        
diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc
index 8b6781a6..c57b76b7 100644
--- a/src/vm_x64.dasc
+++ b/src/vm_x64.dasc


      

      
<snipped>



      

        
diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc
index 7c11c78e..36804d11 100644
--- a/src/vm_x86.dasc
+++ b/src/vm_x86.dasc


      

      
<snipped>



      

        
diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
index 3a8ad63d..ad8b151b 100644
--- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
+++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
@@ -5,7 +5,7 @@ local tap = require('tap')
 -- See also https://github.com/LuaJIT/LuaJIT/issues/1048.
 local test = tap.test('lj-1048-fix-stack-checks-vararg-calls')
 
-test:plan(2)
+test:plan(5)
 
 -- The test case demonstrates a segmentation fault due to stack
 -- overflow by recursive calling `pcall()`. The functions are
@@ -50,4 +50,37 @@ pcall(coroutine.wrap(looper), prober_2, 0)
 
 test:ok(true, 'no stack overflow with metamethod')
 
+-- The testcases demonstrates a stack overflow in
+-- `pcall()`/xpcall()` triggered using metamethod `__call`.
+
+t = coroutine.wrap(setmetatable)({}, { __call = pcall })


      

      
I've meant the following:

| t = setmetatable({}, { __call = pcall })
| coroutine.wrap(function() t() end)()



    

    
Updated

    
@@ -53,7 +53,8 @@ test:ok(true, 'no stack overflow with
      metamethod')

       -- The testcases demonstrates a stack overflow in

       -- `pcall()`/xpcall()` triggered using metamethod `__call`.

       

      -t = coroutine.wrap(setmetatable)({}, { __call = pcall })

      +t = setmetatable({}, { __call = pcall })

      +coroutine.wrap(function() t() end)()

       

       test:ok(true, 'no stack overflow with metamethod __call with
      pcall()')

       

      

    

    

      

        
+
+test:ok(true, 'no stack overflow with metamethod __call with pcall()')
+
+t = coroutine.wrap(setmetatable)({}, { __call = xpcall })


      

      
I've meant the following:

| t = setmetatable({}, { __call = xpcall })
| coroutine.wrap(function() t() end)()

But this won't work since the second amount of xpcall must be the
function. So, this test case is invalid. We must to duplicate the second
approach with `xpcall()`

This works fine.
| LUA_PATH="src/?.lua;;" gdb --args src/luajit -e '
| local t = {}
| local function xpcall_wrapper()
|   return xpcall(unpack(t))
| end
|
| local N_ITERATIONS = 200
|
| for i = 1, N_ITERATIONS do
|   t[i], t[i + 1], t[i + 2] = xpcall, type, {}
|   coroutine.wrap(xpcall_wrapper)()
| end
| '


    

    
Updated:

    
diff --git
      a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      index 6395dfaa..825568f9 100644

      ---
      a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      +++
      b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua

      @@ -58,7 +58,17 @@ coroutine.wrap(function() t() end)()

       

       test:ok(true, 'no stack overflow with metamethod __call with
      pcall()')

       

      -t = coroutine.wrap(setmetatable)({}, { __call = xpcall })

      +t = {}

      +local function xpcall_wrapper()

      +  return xpcall(unpack(t))

      +end

      +

      +local N_ITERATIONS_1 = 200

      +

      +for i = 1, N_ITERATIONS_1 do

      +  t[i], t[i + 1], t[i + 2] = xpcall, type, {}

      +  coroutine.wrap(xpcall_wrapper)()

      +end

       

       test:ok(true, 'no stack overflow with metamethod __call with
      xpcall()')

       

      @@ -67,19 +77,19 @@ test:ok(true, 'no stack overflow with
      metamethod __call with xpcall()')

       -- triggered using `unpack()`.

       

       t = {}

      -local function f()

      +local function pcall_wrapper()

         return pcall(unpack(t))

       end

       

      --- The problem is only reproduced on LuaJIT GC64 and is best

      +-- The problem is only reproduced on LuaJIT GC64 and is better

       -- reproduced under Valgrind than AddressSanitizer. The chosen

       -- value was found experimentally and always results in an
      attempt

       -- to write beyond the allocated memory.

      -local N_ITERATIONS = 200

      +local N_ITERATIONS_2 = 200

       

      -for i = 1, N_ITERATIONS do

      +for i = 1, N_ITERATIONS_2 do

         t[i], t[i + 1], t[i + 2] = pcall, type, {}

      -  coroutine.wrap(f)()

      +  coroutine.wrap(pcall_wrapper)()

       end

       

       test:ok(true, 'no stack overflow with unpacked pcalls')

      

    

    

      

      

        
+
+test:ok(true, 'no stack overflow with metamethod __call with xpcall()')
+
+-- The testcase demonstrates a stack overflow in
+-- `pcall()`/`xpcall()` similar to the first testcase, but it is
+-- triggered using `unpack()`.
+
+t = {}
+local function f()


      

      
Lets name it `pcall_wrapper()`



    

    @@ -66,7 +68,7 @@ test:ok(true, 'no stack overflow with metamethod
    __call with xpcall()')

     -- triggered using `unpack()`.

     

     t = {}

    -local function f()

    +local function pcall_wrapper()

       return pcall(unpack(t))

     end

     

    @@ -78,7 +80,7 @@ local N_ITERATIONS = 200

     

     for i = 1, N_ITERATIONS do

       t[i], t[i + 1], t[i + 2] = pcall, type, {}

    -  coroutine.wrap(f)()

    +  coroutine.wrap(pcall_wrapper)()

     end

     

     test:ok(true, 'no stack overflow with unpacked pcalls')

    

    

      

        
+  return pcall(unpack(t))
+end
+
+-- The problem is only reproduced on LuaJIT GC64 and is best


      

      
Typo: s/best/better/

    

       return pcall(unpack(t))

     end

     

    --- The problem is only reproduced on LuaJIT GC64 and is best

    +-- The problem is only reproduced on LuaJIT GC64 and is better

     -- reproduced under Valgrind than AddressSanitizer. The chosen

     -- value was found experimentally and always results in an attempt

     -- to write beyond the allocated memory.

    

    

      



      

        
+-- reproduced under Valgrind than AddressSanitizer. The chosen
+-- value was found experimentally and always results in an attempt
+-- to write beyond the allocated memory.
+local N_ITERATIONS = 200
+
+for i = 1, N_ITERATIONS do
+  t[i], t[i + 1], t[i + 2] = pcall, type, {}
+  coroutine.wrap(f)()
+end
+
+test:ok(true, 'no stack overflow with unpacked pcalls')
+
 test:done(true)
-- 
2.43.0



      

      

    

  
  


--------------ZK4VGZBcUlvhjCvoLK3LlcnL--