From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 7DED71678626; Wed, 10 Dec 2025 10:25:12 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7DED71678626 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1765351512; bh=l8G5lT1RYkNmS0a3FxKOu88dB7PAZqyoEQVKaV3Yh/0=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=vo7m5fcRL49XFaRXrjuUWSxbXSPagVW6+53a1MRyUNh8FFHzxWY8t4qQL5G5YoGcM Yv18LxVx0Lb774YtiWlxp4+J9xdHTCPAmpnxNDGkt0+OxV9FquOZgKMokL60zIuRGm dLtMWzMFk81/GhKG9bDnJ2H4XstefRK/HQmtMdcQ= Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 2C7831678624 for ; Wed, 10 Dec 2025 10:25:05 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2C7831678624 Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-597d712c0a7so6920739e87.0 for ; Tue, 09 Dec 2025 23:25:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1765351504; x=1765956304; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=9/ipT0u49NPzEa5oyeHEZCT/fIxliL2vZiuCNT0q+V8=; b=qUGdNX6X7Hy5+zaMS57xMVFLiY86B5GTCtADITOrKiAaVIhX/VohZVGO7yBFobnzhO 8GayhnCfsCmrKr2tMGBcd3sAdKJ7X1yYr/c7eyyYg060IbvanxSMcu6VygqtUbUltBA6 2SOhoeXBRmLdQ4P6ewTDLFENUSg+uCBoY7Tam5arbZgP4OARraUSxW40htHJR4d0bQCx aPAgwSyJn62FFtU8P0cSuaV0h7o4Ll8N5CPVTW9jwIgXQzGfJqNKDNYnDzv1EyqXLG+F Rzu79K4Q8nU+4cqA3VBKspbB372xENgxSIh6INzXw6exnB36odTZVvSr02DzgPXaMOtC i/Dw== X-Gm-Message-State: AOJu0Yz04m4ZmfTIeS+6+ijgs8xN0UKN/Uaa9/4kVhcA0L2M679u244M uij8fFbzkCKLumW2VsXn2E/ANmZTB8QjsrLl7rSsdfUmu6XStR5Z+QGYGbwG4oN58bE= X-Gm-Gg: AY/fxX4syM1fE+VtvTpIBByPn1mJqNsN3MOBf85tL8QBw1uuFqvAS80XH6Z8mckgbdO zGeRHFvgDwmfJ2Z+dg28MBKa22W7KD6K+cjuwQUmH6h6M5HTFOK+I68nxjQGuX7AUCNtfImXtor NlyqP+J0phl06j8c/+gq7hNWABXsJMI74I9JSGwd4xwstF7yR3LD8wroSu5jryYeMQJ0tHJU7Ch GS1Vn3U6d4K41pNgEk7+aFT6ged1wk5XASTgK6Kk7Dt858iy8lL5kY9j8jN3PkhfctJouyAFtMU LNiYK51JLdit/sv7RutuuTKMDxfT3N/exawIeRpNm6iH7W9aRYJ3rcD6ZmnWbfgi3afZ5RJEQ3y 1uzRuu/zA9gJIG7VBSUuAlqnkRb5mga0FDKMlsLg7ixh++4j76hplIIvA29hfY1BFdu4RwrsMCa nbuw== X-Google-Smtp-Source: AGHT+IHw7dWwQu3NS7Ic+gxHeAvO8rkzvZqqeuDFwfOHc7pw8/xGU5N+iZNX7QIv+2OkSwRNE9dMrA== X-Received: by 2002:a05:6512:2392:b0:598:eb4e:2acd with SMTP id 2adb3069b0e04-598ee52e3c8mr543035e87.31.1765351503675; Tue, 09 Dec 2025 23:25:03 -0800 (PST) Received: from localhost ([5.187.32.135]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-597d7c27a83sm6058979e87.75.2025.12.09.23.25.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Dec 2025 23:25:03 -0800 (PST) X-Google-Original-From: Sergey Bronnikov To: tarantool-patches@dev.tarantool.org, Sergey Kaplun Date: Wed, 10 Dec 2025 10:23:29 +0300 Message-ID: <51e75e7052824de65036abd2f5807a1224f438aa.1765350224.git.sergeyb@tarantool.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH luajit 2/3][v2] LJ_FR2: Fix stack checks in vararg calls. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" From: Mike Pall Thanks to Peter Cawley. (cherry picked from commit d1a2fef8a8f53b0055ee041f7f63d83a27444ffa) Stack overflow can cause a segmentation fault in vararg function on ARM64 and MIPS64 in LJ_FR2 mode. This happen because stack check in BC_IFUNCV is off by one on these platforms without the patch. The original stack check for ARM64 and MIPS64 was incorrect: | RA == BASE + (RD=NARGS)*8 + framesize * 8 >= maxstack while stack check on x86_64 is correct and therefore is not affected by the problem: | RA == BASE + (RD=NARGS+1)*8 + framesize * 8 +8 > maxstack The patch partially fixes aforementioned issue by bumping LJ_STACK_EXTRA by 1 to give a space to write the entire frame link and fixing a number of last free slot in the stack (LJ_FR2 summand adjustment). A fixup for a number of required slots in `call_init()` was added for consistency with non-gc64 flavor. Sergey Bronnikov: * added the description and the test for the problem Part of tarantool/tarantool#12134 --- src/lj_def.h | 2 +- src/lj_dispatch.c | 2 +- src/vm_arm64.dasc | 1 + src/vm_mips64.dasc | 1 + ...048-fix-stack-checks-vararg-calls.test.lua | 53 +++++++++++++++++++ 5 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua diff --git a/src/lj_def.h b/src/lj_def.h index a5bca6b0..7e4f251e 100644 --- a/src/lj_def.h +++ b/src/lj_def.h @@ -69,7 +69,7 @@ typedef unsigned int uintptr_t; #define LJ_MAX_UPVAL 60 /* Max. # of upvalues. */ #define LJ_MAX_IDXCHAIN 100 /* __index/__newindex chain limit. */ -#define LJ_STACK_EXTRA (5+2*LJ_FR2) /* Extra stack space (metamethods). */ +#define LJ_STACK_EXTRA (5+3*LJ_FR2) /* Extra stack space (metamethods). */ #define LJ_NUM_CBPAGE 1 /* Number of FFI callback pages. */ diff --git a/src/lj_dispatch.c b/src/lj_dispatch.c index a44a5adf..431cb3c2 100644 --- a/src/lj_dispatch.c +++ b/src/lj_dispatch.c @@ -453,7 +453,7 @@ static int call_init(lua_State *L, GCfunc *fn) int numparams = pt->numparams; int gotparams = (int)(L->top - L->base); int need = pt->framesize; - if ((pt->flags & PROTO_VARARG)) need += 1+gotparams; + if ((pt->flags & PROTO_VARARG)) need += 1+LJ_FR2+gotparams; lj_state_checkstack(L, (MSize)need); numparams -= gotparams; return numparams >= 0 ? numparams : 0; diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc index c5f0a7a7..cf8e575a 100644 --- a/src/vm_arm64.dasc +++ b/src/vm_arm64.dasc @@ -3779,6 +3779,7 @@ static void build_ins(BuildCtx *ctx, BCOp op, int defop) | add TMP2, BASE, RC | add LFUNC:CARG3, CARG3, TMP0, lsl #47 | add RA, RA, RC + | sub CARG1, CARG1, #8 | add TMP0, RC, #16+FRAME_VARG | str LFUNC:CARG3, [TMP2], #8 // Store (tagged) copy of LFUNC. | ldr KBASE, [PC, #-4+PC2PROTO(k)] diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc index da187a7a..6c2975b4 100644 --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -5268,6 +5268,7 @@ static void build_ins(BuildCtx *ctx, BCOp op, int defop) | settp LFUNC:RB, TMP0 | daddu TMP0, RA, RC | sd LFUNC:RB, 0(TMP1) // Store (tagged) copy of LFUNC. + | daddiu TMP2, TMP2, -8 | daddiu TMP3, RC, 16+FRAME_VARG | sltu AT, TMP0, TMP2 | ld KBASE, -4+PC2PROTO(k)(PC) diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua new file mode 100644 index 00000000..d471d41e --- /dev/null +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -0,0 +1,53 @@ +local tap = require('tap') + +-- A test file to demonstrate a stack overflow in `pcall()` in +-- some cases, see below testcase descriptions. +-- See also https://github.com/LuaJIT/LuaJIT/issues/1048. +local test = tap.test('lj-1048-fix-stack-checks-vararg-calls'):skipcond({ + ['Test requires JIT enabled'] = not jit.status(), +}) + +test:plan(2) + +-- The testcase demonstrate a segmentation fault due to stack +-- overflow by recursive calling `pcall()`. The functions are +-- vararg because stack check in BC_IFUNCV is off by one on ARM64 +-- and MIPS64 without the patch. +local function prober_1(...) -- luacheck: no unused + -- Any fast function can be used as metamethod, but `type` is + -- convenient here because it works fast and can be used with + -- any data type. Lua function cannot be used since it + -- will check the stack on each invocation. + pcall(pcall, pcall, pcall, pcall, pcall, pcall, pcall, pcall, type, 0) +end + +local function looper(prober, n, ...) + prober(...) + return looper(prober, n + 1, n, ...) +end + +pcall(coroutine.wrap(looper), prober_1, 0) + +test:ok(true, 'no stack overflow with recursive pcall') + +-- The testcase demonstrate a segmentation fault due to stack +-- overflow when `pcall()` is used as `__newindex` metamethod. +-- The function is vararg because stack check in BC_IFUNCV is off +-- by one on ARM64 and MIPS64 without the patch. + +-- Any fast function can be used as metamethod, but `type` is +-- convenient here because it works fast and can be used with +-- any data type. Lua function cannot be used since it +-- will check the stack on each invocation. +local t = setmetatable({}, { __newindex = pcall, __call = type }) + +local function prober_2(...) -- luacheck: no unused + -- Invokes `pcall(t, t, t)`. + t[t] = t +end + +pcall(coroutine.wrap(looper), prober_2, 0) + +test:ok(true, 'no stack overflow with metamethod') + +test:done(true) -- 2.43.0