From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 965471AA65B6; Wed, 11 Mar 2026 17:37:40 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 965471AA65B6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1773239860; bh=mtgnLhbjC+MxoisHlKdpU7bfnA7gWGvnmB8rtR9Z9JI=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=Ar4pdRg+tmSkVAdNudoCsj71L01fK/DzIt5/mOk4qm2CtY9Mq8cPL3P0oZuFit/iE z8tQRL1UKnohVNtqjOv2ZD+bnf9JOs/alv0XTzMOqs6oHrsfT61q0znkW/yvfpKUFU WerXZVhj5pOAAfko7S42KiRQDeOxi2P3/8W3k158= Received: from send129.i.mail.ru (send129.i.mail.ru [89.221.237.224]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 2AE151AA65B3 for ; Wed, 11 Mar 2026 17:37:39 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2AE151AA65B3 Received: by exim-smtp-695fc89d9f-kxhjm with esmtpa (envelope-from ) id 1w0Kgr-00000000GdI-3Uoi; Wed, 11 Mar 2026 17:37:38 +0300 Content-Type: multipart/alternative; boundary="------------Z7ly1tb6n70QzN0bdy205g6F" Message-ID: <5f1d22b9-2fd2-4036-8011-3f721c0abbe7@tarantool.org> Date: Wed, 11 Mar 2026 17:37:36 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Sergey Kaplun , Sergey Bronnikov Cc: tarantool-patches@dev.tarantool.org References: <4a852879bdecd2cedbe1bcb4ebecc89531fc9fe4.1765350224.git.sergeyb@tarantool.org> Content-Language: en-US In-Reply-To: X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91ABAE9865AC7DC88CE822C5E3FB8F8FA45BDD9FF6990928F182A05F538085040F8013249EEA4D7843DE06ABAFEAF6705B830E4266322F0870973A3605B4D6FFAE49391A50CB82CE1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE736C75B72B9FDC350B287FD4696A6DC2FA8DF7F3B2552694A4E2F5AFA99E116B42401471946AA11AFA417C69337E82CC2BCF491FFA38154B6B5C8C57E37DE458B543C44CD4AFD0029C6CDE5D1141D2B1CEBA4C270A9741AC0D93FD59A751653753EF5236DB7D085ADBFBD230C9BB4ADFD8941B15DA834481FA18204E546F3947C062BEEFFB5F8EA3EF6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F79006372A3B24BF85B2E607389733CBF5DBD5E9B5C8C57E37DE458B9E9CE733340B9D5F3BBE47FD9DD3FB595F5C1EE8F4F765FC72CEEB2601E22B093A03B725D353964B0B7D0EA88DDEDAC722CA9DD8327EE4930A3850AC1BE2E735B17145F0B7815491C4224003CC83647689D4C264860C145E X-C1DE0DAB: 0D63561A33F958A55A2C617DEADB15905002B1117B3ED6962E4D7109442D5047B2920F75BA9A967F823CB91A9FED034534781492E4B8EEADC0A73878EBD0941BBDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE191716CD42B3DD1D34CAB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D6598C5A654DF4A1BD23CF2A1141AE01E5C1409E46A0165A1C653C1D85E89DFA6590F8C5CB83110190A5B8341EE9D5BE9A0A589FB89F2ADB6C8B011E164711DBD73C798552ACF71E7C486536EB022892E5344C41F94D744909CE2512F26BEC029E55448553D2254B8D95CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXMZebaIdHP2ghjoIc/363UZI6Kf1ptIMVdbVVJCphTR/P2DbSP6zLPM= X-DA7885C5: 4AC29D1E5F94D2FBF255D290C0D534F9201088EFB8F6D39FCCAEDE1360C8A6963991E87418B596E55B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393520AF17B8A65FDE29BC89992F6004D053C8A885A2E5255E721EC91439EEB2A68EF86D5F70DA33880E41E8EF7A07863ECB274557F927329BE2DDF8182D28ACDB545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3][v2] Add stack check to pcall/xpcall. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------Z7ly1tb6n70QzN0bdy205g6F Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, Sergey, thanks for review! Sergey On 2/11/26 13:24, Sergey Kaplun via Tarantool-patches wrote: > Hi, Sergey! > Thanks for the patch! > Please consider my comments below. > > On 10.12.25, Sergey Bronnikov wrote: >> From: Mike Pall >> >> Analyzed by Peter Cawley. >> >> (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) >> >> In the previous commit ("LJ_FR2: Fix stack checks in vararg calls.") >> stack overflow for vararg functions and metamethod invocations >> was fixed partially and there are still cases where stack overflow >> happens, see comments in the test. The patch fixes the issue by > Please describe the issue regardless previous commit. Just mentioned > missing stack checks for `pcall()`, `xpcall()` is enough. Updated commit message:     Add stack check to pcall/xpcall.     Analyzed by Peter Cawley.     (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)     The patch adds the stack check to fast functions `pcall()` and     `xpcall()`.     Sergey Bronnikov:     * added the description and the test for the problem     Part of tarantool/tarantool#12134 >> adding the stack check to fast functions `pcall()` and `xpcall()`. >> >> Sergey Bronnikov: >> * added the description and the test for the problem >> >> Part of tarantool/tarantool#12134 >> --- >> src/vm_arm.dasc | 7 +++++ >> src/vm_arm64.dasc | 8 +++++ >> src/vm_mips.dasc | 10 +++++- >> src/vm_mips64.dasc | 14 +++++++-- >> src/vm_ppc.dasc | 9 ++++++ >> src/vm_x64.dasc | 6 ++++ >> src/vm_x86.dasc | 6 ++++ >> ...048-fix-stack-checks-vararg-calls.test.lua | 31 ++++++++++++++++++- >> 8 files changed, 86 insertions(+), 5 deletions(-) >> >> diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc >> index 7095e660..efe9dcb2 100644 >> --- a/src/vm_arm.dasc >> +++ b/src/vm_arm.dasc > > >> diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc >> index cf8e575a..53ff7162 100644 >> --- a/src/vm_arm64.dasc >> +++ b/src/vm_arm64.dasc > > >> diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc >> index 32caabf7..69d09d52 100644 >> --- a/src/vm_mips.dasc >> +++ b/src/vm_mips.dasc > > >> diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc >> index 6c2975b4..4e60ee07 100644 >> --- a/src/vm_mips64.dasc >> +++ b/src/vm_mips64.dasc >> @@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx) >> |//-- Base library: catch errors ---------------------------------------- >> | >> |.ffunc pcall >> + | ld TMP1, L->maxstack >> + | daddu TMP2, BASE,NARGS8:RC >> + | sltu AT, TMP1, TMP2 >> + | bnez AT, ->fff_fallback >> + |. lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >> | daddiuNARGS8:RC,NARGS8:RC, -8 >> - | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) >> | bltzNARGS8:RC, ->fff_fallback >> |. move TMP2, BASE >> | daddiu BASE, BASE, 16 >> @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx) >> |. nop >> | >> |.ffunc xpcall >> - | daddiuNARGS8:TMP0,NARGS8:RC, -16 > This change is incorrect. > It wipes out the first patch in the series. Sorry, my bad. --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx)    |  sltu AT, TMP1, TMP2    |  bnez AT, ->fff_fallback    |.  ld CARG1, 0(BASE) -  |  daddiu NARGS8:RC, NARGS8:RC, -16 +  |  daddiu NARGS8:TMP0, NARGS8:RC, -16    |   ld CARG2, 8(BASE)    |    bltz NARGS8:TMP0, ->fff_fallback    |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) >> - | ld CARG1, 0(BASE) >> + | ld TMP1, L->maxstack >> + | daddu TMP2, BASE,NARGS8:RC >> + | sltu AT, TMP1, TMP2 >> + | bnez AT, ->fff_fallback >> + |. ld CARG1, 0(BASE) >> + | daddiuNARGS8:RC,NARGS8:RC, -16 >> | ld CARG2, 8(BASE) >> | bltzNARGS8:TMP0, ->fff_fallback >> |. lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) >> diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc >> index 980ad897..f2ea933b 100644 >> --- a/src/vm_ppc.dasc >> +++ b/src/vm_ppc.dasc > > >> diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc >> index d5296759..141f5f82 100644 >> --- a/src/vm_x64.dasc >> +++ b/src/vm_x64.dasc > > >> diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc >> index b043b830..1ba5abce 100644 >> --- a/src/vm_x86.dasc >> +++ b/src/vm_x86.dasc > > >> diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> index d471d41e..b135042b 100644 >> --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua >> @@ -1,4 +1,5 @@ >> local tap = require('tap') >> +local ffi = require('ffi') >> >> -- A test file to demonstrate a stack overflow in `pcall()` in >> -- some cases, see below testcase descriptions. >> @@ -7,7 +8,7 @@ local test = tap.test('lj-1048-fix-stack-checks-vararg-calls'):skipcond({ >> ['Test requires JIT enabled'] = not jit.status(), >> }) >> >> -test:plan(2) >> +test:plan(4) > This patch covers only `pcall()` cases, please add the same tests for > `xpcall()` (I suppose the most simple is `xpcall()` as `__call` > metamethod). Added: diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua index 0ce3c756..8f45941c 100644 --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -6,7 +6,7 @@ local ffi = require('ffi')  -- See also https://github.com/LuaJIT/LuaJIT/issues/1048.  local test = tap.test('lj-1048-fix-stack-checks-vararg-calls') -test:plan(4) +test:plan(5)  -- The test case demonstrates a segmentation fault due to stack  -- overflow by recursive calling `pcall()`. The functions are @@ -51,12 +51,16 @@ pcall(coroutine.wrap(looper), prober_2, 0) test:ok(true, 'no stack overflow with metamethod') --- The testcase demonstrates a stack overflow in +-- The testcases demonstrates a stack overflow in  -- `pcall()`/xpcall()` triggered using metamethod `__call`.  t = coroutine.wrap(setmetatable)({}, { __call = pcall }) -test:ok(true, 'no stack overflow with metamethod __call') +test:ok(true, 'no stack overflow with metamethod __call with pcall()') + +t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) + +test:ok(true, 'no stack overflow with metamethod __call with xpcall()')  -- The testcase demonstrates a stack overflow in  -- `pcall()`/`xpcall()` similar to the first testcase, but it is >> >> -- The testcase demonstrate a segmentation fault due to stack >> -- overflow by recursive calling `pcall()`. The functions are >> @@ -50,4 +51,32 @@ pcall(coroutine.wrap(looper), prober_2, 0) >> >> test:ok(true, 'no stack overflow with metamethod') >> >> +-- The testcase demonstrate a stack overflow in > Typo: s/demonstrate/demonstrates/ Fixed. >> +-- `pcall()`/xpcall()` triggered using metamethod `__call`. >> + >> +t = setmetatable({}, { __call = pcall })() > It's better to do this at the separate Lua stack, i.e. inside > `coroutine.wrap()`. Updated.  -- `pcall()`/xpcall()` triggered using metamethod `__call`. -t = setmetatable({}, { __call = pcall })() +t = coroutine.wrap(setmetatable)({}, { __call = pcall }) >> + >> +test:ok(true, 'no stack overflow with metamethod __call') >> + >> +-- The testcase demonstrate a stack overflow in > Typo: s/demonstrate/demonstrates/ Fixed. >> +-- `pcall()`/`xpcall()` similar to the first testcase, but it is >> +-- triggered using `unpack()`. >> + >> +t = {} >> +local function f() >> + return pcall(unpack(t)) >> +end >> + > Please explain the amount of necessary iterations of calls. After a small research we found that issue cannot be reproduced in LuaJIT GC32 flavors. N_ITERATION should be equal to at least 200 for reproducing. The problem reproduced better under Valgrind than ASAN. >> +local N_ITERATIONS = 100 >> +if ffi.abi('gc64') then >> + N_ITERATIONS = 180 >> +end >> + >> +for i = 1, N_ITERATIONS do >> + t[i], t[i + 1], t[i + 2] = pcall, pairs, {} > Let's use `type` here instead of pairs. Updated. >> + coroutine.wrap(f)() >> +end >> + >> +test:ok(true, 'no stack overflow with unpacked pcalls') >> + >> test:done(true) >> -- >> 2.43.0 >> --------------Z7ly1tb6n70QzN0bdy205g6F Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi, Sergey,

thanks for review!

Sergey

On 2/11/26 13:24, Sergey Kaplun via Tarantool-patches wrote:
Hi, Sergey!
Thanks for the patch!
Please consider my comments below.

On 10.12.25, Sergey Bronnikov wrote:

From: Mike Pall <mike>

Analyzed by Peter Cawley.

(cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)

In the previous commit ("LJ_FR2: Fix stack checks in vararg calls.")
stack overflow for vararg functions and metamethod invocations
was fixed partially and there are still cases where stack overflow
happens, see comments in the test. The patch fixes the issue by

Please describe the issue regardless previous commit. Just mentioned
missing stack checks for `pcall()`, `xpcall()` is enough.

Updated commit message:


    Add stack check to pcall/xpcall.
    
    Analyzed by Peter Cawley.
    
    (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36)
    
    The patch adds the stack check to fast functions `pcall()` and
    `xpcall()`.
    
    Sergey Bronnikov:
    * added the description and the test for the problem
    
    Part of tarantool/tarantool#12134


adding the stack check to fast functions `pcall()` and `xpcall()`.

Sergey Bronnikov:
* added the description and the test for the problem

Part of tarantool/tarantool#12134
---
 src/vm_arm.dasc                               |  7 +++++
 src/vm_arm64.dasc                             |  8 +++++
 src/vm_mips.dasc                              | 10 +++++-
 src/vm_mips64.dasc                            | 14 +++++++--
 src/vm_ppc.dasc                               |  9 ++++++
 src/vm_x64.dasc                               |  6 ++++
 src/vm_x86.dasc                               |  6 ++++
 ...048-fix-stack-checks-vararg-calls.test.lua | 31 ++++++++++++++++++-
 8 files changed, 86 insertions(+), 5 deletions(-)

diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc
index 7095e660..efe9dcb2 100644
--- a/src/vm_arm.dasc
+++ b/src/vm_arm.dasc

<snipped>


diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc
index cf8e575a..53ff7162 100644
--- a/src/vm_arm64.dasc
+++ b/src/vm_arm64.dasc

<snipped>


diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc
index 32caabf7..69d09d52 100644
--- a/src/vm_mips.dasc
+++ b/src/vm_mips.dasc

<snipped>


diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc
index 6c2975b4..4e60ee07 100644
--- a/src/vm_mips64.dasc
+++ b/src/vm_mips64.dasc
@@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx)
   |//-- Base library: catch errors ----------------------------------------
   |
   |.ffunc pcall
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE, NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
   |  daddiu NARGS8:RC, NARGS8:RC, -8
-  |  lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH)
   |  bltz NARGS8:RC, ->fff_fallback
   |.   move TMP2, BASE
   |   daddiu BASE, BASE, 16
@@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx)
   |.  nop
   |
   |.ffunc xpcall
-  |  daddiu NARGS8:TMP0, NARGS8:RC, -16

This change is incorrect.
It wipes out the first patch in the series.

Sorry, my bad.

--- a/src/vm_mips64.dasc
+++ b/src/vm_mips64.dasc
@@ -1449,7 +1449,7 @@ static void build_subroutines(BuildCtx *ctx)
   |  sltu AT, TMP1, TMP2
   |  bnez AT, ->fff_fallback
   |.  ld CARG1, 0(BASE)
-  |  daddiu NARGS8:RC, NARGS8:RC, -16
+  |  daddiu NARGS8:TMP0, NARGS8:RC, -16
   |   ld CARG2, 8(BASE)
   |    bltz NARGS8:TMP0, ->fff_fallback
   |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)

-  |  ld CARG1, 0(BASE)
+  |  ld TMP1, L->maxstack
+  |  daddu TMP2, BASE, NARGS8:RC
+  |  sltu AT, TMP1, TMP2
+  |  bnez AT, ->fff_fallback
+  |.  ld CARG1, 0(BASE)
+  |  daddiu NARGS8:RC, NARGS8:RC, -16
   |   ld CARG2, 8(BASE)
   |    bltz NARGS8:TMP0, ->fff_fallback
   |.    lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH)
diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc
index 980ad897..f2ea933b 100644
--- a/src/vm_ppc.dasc
+++ b/src/vm_ppc.dasc

<snipped>


diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc
index d5296759..141f5f82 100644
--- a/src/vm_x64.dasc
+++ b/src/vm_x64.dasc

<snipped>


diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc
index b043b830..1ba5abce 100644
--- a/src/vm_x86.dasc
+++ b/src/vm_x86.dasc

<snipped>


diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
index d471d41e..b135042b 100644
--- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
+++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
@@ -1,4 +1,5 @@
 local tap = require('tap')
+local ffi = require('ffi')
 
 -- A test file to demonstrate a stack overflow in `pcall()` in
 -- some cases, see below testcase descriptions.
@@ -7,7 +8,7 @@ local test = tap.test('lj-1048-fix-stack-checks-vararg-calls'):skipcond({
   ['Test requires JIT enabled'] = not jit.status(),
 })
 
-test:plan(2)
+test:plan(4)

This patch covers only `pcall()` cases, please add the same tests for
`xpcall()` (I suppose the most simple is `xpcall()` as `__call`
metamethod).

Added:

diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
index 0ce3c756..8f45941c 100644
--- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
+++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua
@@ -6,7 +6,7 @@ local ffi = require('ffi')
 -- See also https://github.com/LuaJIT/LuaJIT/issues/1048.
 local test = tap.test('lj-1048-fix-stack-checks-vararg-calls')
 
-test:plan(4)
+test:plan(5)
 
 -- The test case demonstrates a segmentation fault due to stack
 -- overflow by recursive calling `pcall()`. The functions are
@@ -51,12 +51,16 @@ pcall(coroutine.wrap(looper), prober_2, 0)
 
 test:ok(true, 'no stack overflow with metamethod')
 
--- The testcase demonstrates a stack overflow in
+-- The testcases demonstrates a stack overflow in
 -- `pcall()`/xpcall()` triggered using metamethod `__call`.
 
 t = coroutine.wrap(setmetatable)({}, { __call = pcall })
 
-test:ok(true, 'no stack overflow with metamethod __call')
+test:ok(true, 'no stack overflow with metamethod __call with pcall()')
+
+t = coroutine.wrap(setmetatable)({}, { __call = xpcall })
+
+test:ok(true, 'no stack overflow with metamethod __call with xpcall()')
 
 -- The testcase demonstrates a stack overflow in
 -- `pcall()`/`xpcall()` similar to the first testcase, but it is

 
 -- The testcase demonstrate a segmentation fault due to stack
 -- overflow by recursive calling `pcall()`. The functions are
@@ -50,4 +51,32 @@ pcall(coroutine.wrap(looper), prober_2, 0)
 
 test:ok(true, 'no stack overflow with metamethod')
 
+-- The testcase demonstrate a stack overflow in

Typo: s/demonstrate/demonstrates/
Fixed. 
+-- `pcall()`/xpcall()` triggered using metamethod `__call`.
+
+t = setmetatable({}, { __call = pcall })()

It's better to do this at the separate Lua stack, i.e. inside
`coroutine.wrap()`.

Updated.

 -- `pcall()`/xpcall()` triggered using metamethod `__call`.
 
-t = setmetatable({}, { __call = pcall })()
+t = coroutine.wrap(setmetatable)({}, { __call = pcall })

+
+test:ok(true, 'no stack overflow with metamethod __call')
+
+-- The testcase demonstrate a stack overflow in

Typo: s/demonstrate/demonstrates/
Fixed. 
+-- `pcall()`/`xpcall()` similar to the first testcase, but it is
+-- triggered using `unpack()`.
+
+t = {}
+local function f()
+  return pcall(unpack(t))
+end
+

Please explain the amount of necessary iterations of calls.

After a small research we found that issue cannot be reproduced in LuaJIT GC32 flavors.

N_ITERATION should be equal to at least 200 for reproducing.

The problem reproduced better under Valgrind than ASAN.


+local N_ITERATIONS = 100
+if ffi.abi('gc64') then
+  N_ITERATIONS = 180
+end
+
+for i = 1, N_ITERATIONS do
+  t[i], t[i + 1], t[i + 2] = pcall, pairs, {}

Let's use `type` here instead of pairs.
Updated. 
+  coroutine.wrap(f)()
+end
+
+test:ok(true, 'no stack overflow with unpacked pcalls')
+
 test:done(true)
-- 
2.43.0
--------------Z7ly1tb6n70QzN0bdy205g6F--