From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 8F4A91965684; Thu, 12 Mar 2026 11:52:21 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8F4A91965684 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1773305541; bh=EMAWnL2JqbBcLxxOx2NQfigek8o+bMVTM4ocjdgI8KQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=wvgVe7F5JO3gCvNAmjZ57wvsgb22M3O7Ei349oHnrJf6jAyYqoiFnxcTRaf89ptlJ fJRjvnpvusq8F6SQAqEhfJN87M97uxfvQMdFRoc/f7Ax+RewcxNjQbUMZl2jOTvGxI NH2SGlp07gyMNM0TslP3P9E1bQhegKg/U5lPzH2g= Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A3B221AA801E for ; Thu, 12 Mar 2026 11:52:01 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A3B221AA801E Received: by mail-lf1-f46.google.com with SMTP id 2adb3069b0e04-5a126b79512so779551e87.3 for ; Thu, 12 Mar 2026 01:52:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773305521; x=1773910321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=400kQ3o1CEbje80v5JNgh2C0WuabLH2YXkR2/+g/Y9o=; b=IZ1wPfsfZqKUKkXUARF44+85f5cO1e56oJPvFk2QpDhLwxIOWC0ztgwNlpQb3a8YS4 DauoIEm3oOtWUGHeLb9AQXUgpf5yHwdm6QxZYsjVWCnS7txJr/L6RHX7OXVzjgwMWlGf SpL4Fn1xIV6faTTjtpswAVYv14VZ1XPj4IzwWSohFzjCeZ6bAYMT8+cmFGuK+QXD244S 4+3VPBbXOSAwgJQ8of33HwoLORev2NyrSFXxX6fbx4lbsGvZ4PpbneDriVwQZlMsnRMo gDQJNIbrKPm6FDnaR0zFitMlGN6QOTUJOmGwSOsfxFNsuD/r4gnFIyeTpvWTeUn1gH3p EWFw== X-Gm-Message-State: AOJu0YyOP3jCS2T3k7o3EmxtrqtCbaN4XHDhgIUDL7iG9NZZwUbeHjtH SDUQyhSKyae9fbOwG4fqcxnKVeaKyZOvFE/xfNqMUN7uisNeTN1yxZpnWM9MLQ== X-Gm-Gg: ATEYQzxmVuRPZy5OBG1iLD7A0ndkw4nYPbfNFSQXy1I3vjYzbIDXOmDCHIh1MuKzhII DBlUfXgL0uYF/CqmlyxOhrd/0mAsSLXwp2fPj63HvsZXN3h+voiL23kpWPkwcC58e4T72YpEuUH jFBimkAmTao0Dip3mu4guaNL0UJnsPmt5dWBOJC0dtl0lpT4END23gIN8QSyh/XK9juJxjHMpu9 HbOTnyOLrq7XjZkVNviC/RLr7nXWDGTC+zPqVE+9ydjQyl0ppIGXqoTlTCVgVP8ZsKkkCyETW2e 7c+qDK3auOP7AAXEaUBrJpe4a2dLNWIRe4t6L9uGvwnbIfxBABcvRdbJWxYilrmA/jcr32VYvCT Qz2BelT1kwSuf/PnyEYtmsBq9lFiSqnbjx32yLaw3EcwS4WQfONDJo+NqVkuiNxTG/M4eUMdaaA gBUnxfAgiNZ5rIGXSS5OWJI4WjrQ== X-Received: by 2002:a05:6512:608:10b0:5a1:3f3f:a29b with SMTP id 2adb3069b0e04-5a156ba03ccmr1495880e87.6.1773305520317; Thu, 12 Mar 2026 01:52:00 -0700 (PDT) Received: from localhost ([79.164.223.111]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5a155f33c0asm846451e87.18.2026.03.12.01.51.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 01:51:59 -0700 (PDT) X-Google-Original-From: Sergey Bronnikov To: tarantool-patches@dev.tarantool.org, Sergey Kaplun Date: Thu, 12 Mar 2026 11:49:59 +0300 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH luajit 3/3][v3] Add stack check to pcall/xpcall. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" From: Mike Pall Analyzed by Peter Cawley. (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) The patch adds the stack check to fast functions `pcall()` and `xpcall()`. Sergey Bronnikov: * added the description and the test for the problem Part of tarantool/tarantool#12134 --- src/vm_arm.dasc | 7 ++++ src/vm_arm64.dasc | 8 +++++ src/vm_mips.dasc | 10 +++++- src/vm_mips64.dasc | 14 ++++++-- src/vm_ppc.dasc | 9 +++++ src/vm_x64.dasc | 6 ++++ src/vm_x86.dasc | 6 ++++ ...048-fix-stack-checks-vararg-calls.test.lua | 35 ++++++++++++++++++- 8 files changed, 90 insertions(+), 5 deletions(-) diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc index 7095e660..efe9dcb2 100644 --- a/src/vm_arm.dasc +++ b/src/vm_arm.dasc @@ -1201,8 +1201,11 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ldr RB, L->maxstack + | add INS, BASE, NARGS8:RC | ldrb RA, [DISPATCH, #DISPATCH_GL(hookmask)] | cmp NARGS8:RC, #8 + | cmphs RB, INS | blo ->fff_fallback | tst RA, #HOOK_ACTIVE // Remember active hook before pcall. | mov RB, BASE @@ -1213,7 +1216,11 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc_2 xpcall + | ldr RB, L->maxstack + | add INS, BASE, NARGS8:RC | ldrb RA, [DISPATCH, #DISPATCH_GL(hookmask)] + | cmp RB, INS + | blo ->fff_fallback | checkfunc CARG4, ->fff_fallback // Traceback must be a function. | mov RB, BASE | strd CARG12, [BASE, #8] // Swap function and traceback. diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc index 5ef37243..074c1f31 100644 --- a/src/vm_arm64.dasc +++ b/src/vm_arm64.dasc @@ -1166,6 +1166,10 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ldr TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC + | cmp TMP1, TMP2 + | blo ->fff_fallback | cmp NARGS8:RC, #8 | ldrb TMP0w, GL->hookmask | blo ->fff_fallback @@ -1185,6 +1189,10 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc xpcall + | ldr TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC + | cmp TMP1, TMP2 + | blo ->fff_fallback | ldp CARG1, CARG2, [BASE] | ldrb TMP0w, GL->hookmask | subs NARGS8:TMP1, NARGS8:RC, #16 diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc index 32caabf7..69d09d52 100644 --- a/src/vm_mips.dasc +++ b/src/vm_mips.dasc @@ -1382,9 +1382,13 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | lw TMP1, L->maxstack + | addu TMP2, BASE, NARGS8:RC | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | beqz NARGS8:RC, ->fff_fallback - | move TMP2, BASE + |. sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. move TMP2, BASE | addiu BASE, BASE, 8 | // Remember active hook before pcall. | srl TMP3, TMP3, HOOK_ACTIVE_SHIFT @@ -1394,8 +1398,12 @@ static void build_subroutines(BuildCtx *ctx) |. addiu NARGS8:RC, NARGS8:RC, -8 | |.ffunc xpcall + | lw TMP1, L->maxstack + | addu TMP2, BASE, NARGS8:RC | sltiu AT, NARGS8:RC, 16 | lw CARG4, 8+HI(BASE) + | sltu TMP1, TMP1, TMP2 + | or AT, AT, TMP1 | bnez AT, ->fff_fallback |. lw CARG3, 8+LO(BASE) | lw CARG1, LO(BASE) diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc index 6c2975b4..4e60ee07 100644 --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ld TMP1, L->maxstack + | daddu TMP2, BASE, NARGS8:RC + | sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | daddiu NARGS8:RC, NARGS8:RC, -8 - | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | bltz NARGS8:RC, ->fff_fallback |. move TMP2, BASE | daddiu BASE, BASE, 16 @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx) |. nop | |.ffunc xpcall - | daddiu NARGS8:TMP0, NARGS8:RC, -16 - | ld CARG1, 0(BASE) + | ld TMP1, L->maxstack + | daddu TMP2, BASE, NARGS8:RC + | sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. ld CARG1, 0(BASE) + | daddiu NARGS8:RC, NARGS8:RC, -16 | ld CARG2, 8(BASE) | bltz NARGS8:TMP0, ->fff_fallback |. lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc index 980ad897..f2ea933b 100644 --- a/src/vm_ppc.dasc +++ b/src/vm_ppc.dasc @@ -1755,8 +1755,12 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | lwz TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC | cmplwi NARGS8:RC, 8 | lbz TMP3, DISPATCH_GL(hookmask)(DISPATCH) + | cmplw cr1, TMP1, TMP2 + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | blt ->fff_fallback | mr TMP2, BASE | la BASE, 8(BASE) @@ -1767,14 +1771,19 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc xpcall + | lwz TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC | cmplwi NARGS8:RC, 16 | lwz CARG3, 8(BASE) + | cmplw cr1, TMP1, TMP2 |.if FPU | lfd FARG2, 8(BASE) + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | lfd FARG1, 0(BASE) |.else | lwz CARG1, 0(BASE) | lwz CARG2, 4(BASE) + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | lwz CARG4, 12(BASE) |.endif | blt ->fff_fallback diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc index 8b6781a6..c57b76b7 100644 --- a/src/vm_x64.dasc +++ b/src/vm_x64.dasc @@ -1545,6 +1545,9 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc_1 pcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | lea RA, [BASE+16] | sub NARGS:RDd, 1 | mov PCd, 16+FRAME_PCALL @@ -1563,6 +1566,9 @@ static void build_subroutines(BuildCtx *ctx) | jmp ->vm_call_dispatch | |.ffunc_2 xpcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | mov LFUNC:RA, [BASE+8] | checktp_nc LFUNC:RA, LJ_TFUNC, ->fff_fallback | mov LFUNC:RB, [BASE] // Swap function and traceback. diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc index 7c11c78e..36804d11 100644 --- a/src/vm_x86.dasc +++ b/src/vm_x86.dasc @@ -1914,6 +1914,9 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc_1 pcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | lea RA, [BASE+8] | sub NARGS:RD, 1 | mov PC, 8+FRAME_PCALL @@ -1925,6 +1928,9 @@ static void build_subroutines(BuildCtx *ctx) | jmp ->vm_call_dispatch | |.ffunc_2 xpcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | cmp dword [BASE+12], LJ_TFUNC; jne ->fff_fallback | mov RB, [BASE+4] // Swap function and traceback. | mov [BASE+12], RB diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua index 3a8ad63d..ad8b151b 100644 --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -5,7 +5,7 @@ local tap = require('tap') -- See also https://github.com/LuaJIT/LuaJIT/issues/1048. local test = tap.test('lj-1048-fix-stack-checks-vararg-calls') -test:plan(2) +test:plan(5) -- The test case demonstrates a segmentation fault due to stack -- overflow by recursive calling `pcall()`. The functions are @@ -50,4 +50,37 @@ pcall(coroutine.wrap(looper), prober_2, 0) test:ok(true, 'no stack overflow with metamethod') +-- The testcases demonstrates a stack overflow in +-- `pcall()`/xpcall()` triggered using metamethod `__call`. + +t = coroutine.wrap(setmetatable)({}, { __call = pcall }) + +test:ok(true, 'no stack overflow with metamethod __call with pcall()') + +t = coroutine.wrap(setmetatable)({}, { __call = xpcall }) + +test:ok(true, 'no stack overflow with metamethod __call with xpcall()') + +-- The testcase demonstrates a stack overflow in +-- `pcall()`/`xpcall()` similar to the first testcase, but it is +-- triggered using `unpack()`. + +t = {} +local function f() + return pcall(unpack(t)) +end + +-- The problem is only reproduced on LuaJIT GC64 and is best +-- reproduced under Valgrind than AddressSanitizer. The chosen +-- value was found experimentally and always results in an attempt +-- to write beyond the allocated memory. +local N_ITERATIONS = 200 + +for i = 1, N_ITERATIONS do + t[i], t[i + 1], t[i + 2] = pcall, type, {} + coroutine.wrap(f)() +end + +test:ok(true, 'no stack overflow with unpacked pcalls') + test:done(true) -- 2.43.0