From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 07F54167862B; Wed, 10 Dec 2025 10:25:44 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 07F54167862B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1765351544; bh=9fLW2Ha2SbLggNdKOWWuPss3DR+Wqs/NVAAUpHVEgpo=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=nA+Zg13WK6tkf98bbJJIjUf1k2STX9QYEmNLcKcMKgZHfUXeCsiICMuUtbli0nRwx akEVML8fbrZ5cj0hfsgEFclo4q17R5PpcWlpnCzEe7wJXZeRmjuE52deGVDJc9UWFt jXboZUOp3e2RJuhd9CrecvZyscSTsUXFyUOs+JDA= Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 7C745167862B for ; Wed, 10 Dec 2025 10:25:28 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7C745167862B Received: by mail-lf1-f54.google.com with SMTP id 2adb3069b0e04-595825c8eb3so6430649e87.0 for ; Tue, 09 Dec 2025 23:25:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765351527; x=1765956327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=eeQTDaKIRaJrfkscUKTlEDxAuMehjyljZAvX10WwbCQ=; b=hk/fODrR2pPhq60zVyRlss3nvQSXa2k1gzmhOtTkJ7LaeuvknQK3ZmiqOzEDX03xYU cGOFIOWVkb61AZPIhz4NTyINGWz4TmYy6FofbsrJn1WOFKC30ZUfRAgTW6JfS1KR1GOQ 4ffoxb9RNQymGYJKNPXiBA6xZElmXJlPusvQPDfYM2YdD57uN35RN2Z5eoenRPEenR4R Lv3FZY76hdov9iaARviK7lbDkMzMkS/F7N7K5JVsogOhk4edeJGXje/uu4FBIowxcxrr Vs21JEEycieE0o66trzj91OUUBqB83H6GtaKCnsNI0BIHmYC37dRbdz2UufLLSOA0PnO KObQ== X-Gm-Message-State: AOJu0YzjCRQ7lo4HlrlxHFJUfrDygSPyFYngLAX2E+dXYNaz3sHVNrdU eKjwyV8xJrNC7hVuVROtgnkI/SWMwGpLYBsIx87OPo5cV+QIocnZ4V4RiPhqTl5th4E= X-Gm-Gg: AY/fxX6fLuZHz/KJEB2eWbRPmkwd6zxTHMtfeGZDVaeMtsbLKPdGpjb/wu8YQUoaUxQ nsfMlyi8gDSsDewJ2c8w5H3c1Qayaxs0UbXHHauexzF4NBsmF1Bl9x/0yN/pY19et9mf4AqoaSb QBf0uDrgSP5B13PnorGpIZHTbeC6/3uexriFWhWZxy3WrL3R9rp+3E248gVsPBV7vxSdhoheQ2d DfOmnwG/PgEiia7rGGVdJ2fggVurHDmkWm8PYSRi0H0ga7L8eqieUo/UKLGVg3iZmj6Wa50EFKR ZLspC7lKJipO3B1Jdue231tU4tfMuUO+s83a24K4aHWJtYmM+yb9d9IsOhMK+R+tLIfLam6JLO9 hGYYDiRJ5luYCaGQAv8f+tE/05yadQJFjXPPHDfF7nKsv0jQzhPOnKQff8dfuCMCvXygc0Gli/z SH4Q== X-Google-Smtp-Source: AGHT+IGhMUNwEBDT6tcCG/MEDmfioaR2c7m0Jw41vQyWgmOGdjAgjr0tUjcGWAvckPA9Fas+Kajrfw== X-Received: by 2002:a05:6512:131c:b0:595:7fed:aae9 with SMTP id 2adb3069b0e04-598ee4dd251mr648957e87.12.1765351527287; Tue, 09 Dec 2025 23:25:27 -0800 (PST) Received: from localhost ([5.187.32.135]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-597d7b24719sm5990615e87.24.2025.12.09.23.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Dec 2025 23:25:26 -0800 (PST) X-Google-Original-From: Sergey Bronnikov To: tarantool-patches@dev.tarantool.org, Sergey Kaplun Date: Wed, 10 Dec 2025 10:23:31 +0300 Message-ID: <4a852879bdecd2cedbe1bcb4ebecc89531fc9fe4.1765350224.git.sergeyb@tarantool.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH luajit 3/3][v2] Add stack check to pcall/xpcall. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" From: Mike Pall Analyzed by Peter Cawley. (cherry picked from commit a4c1640432a9d8a60624cdc8065b15078c228e36) In the previous commit ("LJ_FR2: Fix stack checks in vararg calls.") stack overflow for vararg functions and metamethod invocations was fixed partially and there are still cases where stack overflow happens, see comments in the test. The patch fixes the issue by adding the stack check to fast functions `pcall()` and `xpcall()`. Sergey Bronnikov: * added the description and the test for the problem Part of tarantool/tarantool#12134 --- src/vm_arm.dasc | 7 +++++ src/vm_arm64.dasc | 8 +++++ src/vm_mips.dasc | 10 +++++- src/vm_mips64.dasc | 14 +++++++-- src/vm_ppc.dasc | 9 ++++++ src/vm_x64.dasc | 6 ++++ src/vm_x86.dasc | 6 ++++ ...048-fix-stack-checks-vararg-calls.test.lua | 31 ++++++++++++++++++- 8 files changed, 86 insertions(+), 5 deletions(-) diff --git a/src/vm_arm.dasc b/src/vm_arm.dasc index 7095e660..efe9dcb2 100644 --- a/src/vm_arm.dasc +++ b/src/vm_arm.dasc @@ -1201,8 +1201,11 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ldr RB, L->maxstack + | add INS, BASE, NARGS8:RC | ldrb RA, [DISPATCH, #DISPATCH_GL(hookmask)] | cmp NARGS8:RC, #8 + | cmphs RB, INS | blo ->fff_fallback | tst RA, #HOOK_ACTIVE // Remember active hook before pcall. | mov RB, BASE @@ -1213,7 +1216,11 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc_2 xpcall + | ldr RB, L->maxstack + | add INS, BASE, NARGS8:RC | ldrb RA, [DISPATCH, #DISPATCH_GL(hookmask)] + | cmp RB, INS + | blo ->fff_fallback | checkfunc CARG4, ->fff_fallback // Traceback must be a function. | mov RB, BASE | strd CARG12, [BASE, #8] // Swap function and traceback. diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc index cf8e575a..53ff7162 100644 --- a/src/vm_arm64.dasc +++ b/src/vm_arm64.dasc @@ -1166,6 +1166,10 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ldr TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC + | cmp TMP1, TMP2 + | blo ->fff_fallback | cmp NARGS8:RC, #8 | ldrb TMP0w, GL->hookmask | blo ->fff_fallback @@ -1185,6 +1189,10 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc xpcall + | ldr TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC + | cmp TMP1, TMP2 + | blo ->fff_fallback | ldp CARG1, CARG2, [BASE] | ldrb TMP0w, GL->hookmask | subs NARGS8:TMP1, NARGS8:RC, #16 diff --git a/src/vm_mips.dasc b/src/vm_mips.dasc index 32caabf7..69d09d52 100644 --- a/src/vm_mips.dasc +++ b/src/vm_mips.dasc @@ -1382,9 +1382,13 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | lw TMP1, L->maxstack + | addu TMP2, BASE, NARGS8:RC | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | beqz NARGS8:RC, ->fff_fallback - | move TMP2, BASE + |. sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. move TMP2, BASE | addiu BASE, BASE, 8 | // Remember active hook before pcall. | srl TMP3, TMP3, HOOK_ACTIVE_SHIFT @@ -1394,8 +1398,12 @@ static void build_subroutines(BuildCtx *ctx) |. addiu NARGS8:RC, NARGS8:RC, -8 | |.ffunc xpcall + | lw TMP1, L->maxstack + | addu TMP2, BASE, NARGS8:RC | sltiu AT, NARGS8:RC, 16 | lw CARG4, 8+HI(BASE) + | sltu TMP1, TMP1, TMP2 + | or AT, AT, TMP1 | bnez AT, ->fff_fallback |. lw CARG3, 8+LO(BASE) | lw CARG1, LO(BASE) diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc index 6c2975b4..4e60ee07 100644 --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -1418,8 +1418,12 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | ld TMP1, L->maxstack + | daddu TMP2, BASE, NARGS8:RC + | sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | daddiu NARGS8:RC, NARGS8:RC, -8 - | lbu TMP3, DISPATCH_GL(hookmask)(DISPATCH) | bltz NARGS8:RC, ->fff_fallback |. move TMP2, BASE | daddiu BASE, BASE, 16 @@ -1440,8 +1444,12 @@ static void build_subroutines(BuildCtx *ctx) |. nop | |.ffunc xpcall - | daddiu NARGS8:TMP0, NARGS8:RC, -16 - | ld CARG1, 0(BASE) + | ld TMP1, L->maxstack + | daddu TMP2, BASE, NARGS8:RC + | sltu AT, TMP1, TMP2 + | bnez AT, ->fff_fallback + |. ld CARG1, 0(BASE) + | daddiu NARGS8:RC, NARGS8:RC, -16 | ld CARG2, 8(BASE) | bltz NARGS8:TMP0, ->fff_fallback |. lbu TMP1, DISPATCH_GL(hookmask)(DISPATCH) diff --git a/src/vm_ppc.dasc b/src/vm_ppc.dasc index 980ad897..f2ea933b 100644 --- a/src/vm_ppc.dasc +++ b/src/vm_ppc.dasc @@ -1755,8 +1755,12 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc pcall + | lwz TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC | cmplwi NARGS8:RC, 8 | lbz TMP3, DISPATCH_GL(hookmask)(DISPATCH) + | cmplw cr1, TMP1, TMP2 + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | blt ->fff_fallback | mr TMP2, BASE | la BASE, 8(BASE) @@ -1767,14 +1771,19 @@ static void build_subroutines(BuildCtx *ctx) | b ->vm_call_dispatch | |.ffunc xpcall + | lwz TMP1, L->maxstack + | add TMP2, BASE, NARGS8:RC | cmplwi NARGS8:RC, 16 | lwz CARG3, 8(BASE) + | cmplw cr1, TMP1, TMP2 |.if FPU | lfd FARG2, 8(BASE) + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | lfd FARG1, 0(BASE) |.else | lwz CARG1, 0(BASE) | lwz CARG2, 4(BASE) + | cror 4*cr0+lt, 4*cr0+lt, 4*cr1+lt | lwz CARG4, 12(BASE) |.endif | blt ->fff_fallback diff --git a/src/vm_x64.dasc b/src/vm_x64.dasc index d5296759..141f5f82 100644 --- a/src/vm_x64.dasc +++ b/src/vm_x64.dasc @@ -1545,6 +1545,9 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc_1 pcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | lea RA, [BASE+16] | sub NARGS:RDd, 1 | mov PCd, 16+FRAME_PCALL @@ -1563,6 +1566,9 @@ static void build_subroutines(BuildCtx *ctx) | jmp ->vm_call_dispatch | |.ffunc_2 xpcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | mov LFUNC:RA, [BASE+8] | checktp_nc LFUNC:RA, LJ_TFUNC, ->fff_fallback | mov LFUNC:RB, [BASE] // Swap function and traceback. diff --git a/src/vm_x86.dasc b/src/vm_x86.dasc index b043b830..1ba5abce 100644 --- a/src/vm_x86.dasc +++ b/src/vm_x86.dasc @@ -1914,6 +1914,9 @@ static void build_subroutines(BuildCtx *ctx) |//-- Base library: catch errors ---------------------------------------- | |.ffunc_1 pcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | lea RA, [BASE+8] | sub NARGS:RD, 1 | mov PC, 8+FRAME_PCALL @@ -1925,6 +1928,9 @@ static void build_subroutines(BuildCtx *ctx) | jmp ->vm_call_dispatch | |.ffunc_2 xpcall + | mov L:RB, SAVE_L + | lea RA, [BASE+NARGS:RD*8] + | cmp RA, L:RB->maxstack; ja ->fff_fallback | cmp dword [BASE+12], LJ_TFUNC; jne ->fff_fallback | mov RB, [BASE+4] // Swap function and traceback. | mov [BASE+12], RB diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua index d471d41e..b135042b 100644 --- a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -1,4 +1,5 @@ local tap = require('tap') +local ffi = require('ffi') -- A test file to demonstrate a stack overflow in `pcall()` in -- some cases, see below testcase descriptions. @@ -7,7 +8,7 @@ local test = tap.test('lj-1048-fix-stack-checks-vararg-calls'):skipcond({ ['Test requires JIT enabled'] = not jit.status(), }) -test:plan(2) +test:plan(4) -- The testcase demonstrate a segmentation fault due to stack -- overflow by recursive calling `pcall()`. The functions are @@ -50,4 +51,32 @@ pcall(coroutine.wrap(looper), prober_2, 0) test:ok(true, 'no stack overflow with metamethod') +-- The testcase demonstrate a stack overflow in +-- `pcall()`/xpcall()` triggered using metamethod `__call`. + +t = setmetatable({}, { __call = pcall })() + +test:ok(true, 'no stack overflow with metamethod __call') + +-- The testcase demonstrate a stack overflow in +-- `pcall()`/`xpcall()` similar to the first testcase, but it is +-- triggered using `unpack()`. + +t = {} +local function f() + return pcall(unpack(t)) +end + +local N_ITERATIONS = 100 +if ffi.abi('gc64') then + N_ITERATIONS = 180 +end + +for i = 1, N_ITERATIONS do + t[i], t[i + 1], t[i + 2] = pcall, pairs, {} + coroutine.wrap(f)() +end + +test:ok(true, 'no stack overflow with unpacked pcalls') + test:done(true) -- 2.43.0