From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B4FA5151249D; Wed, 27 Aug 2025 12:45:31 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B4FA5151249D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1756287931; bh=xkBBAwtq6uAfveMEGLxFjEnpRFxCNhMqSkPgFLPjAmk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=fkbemKf+hAmDJEHc0X9B1u0iJvDODAhv2w0/wG0x63Ubo3k8nmnNfQKQgGVjn6gMt 92PraOGU/P3EjlSGUKowXOQMb9aTCkw3uhvCnDBsEqXTGL6nNpgyFAAGIvFBak5qIi YSQx3ROG5zP+UJUwPEwa+5ReIu6+p3U/h/bQRpZE= Received: from mail-lf1-f47.google.com (mail-lf1-f47.google.com [209.85.167.47]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id D1E43151249C for ; Wed, 27 Aug 2025 12:45:11 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D1E43151249C Received: by mail-lf1-f47.google.com with SMTP id 2adb3069b0e04-55f3dcb2b9fso726849e87.1 for ; Wed, 27 Aug 2025 02:45:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756287911; x=1756892711; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C7ryy+VpILJPIPaPzJes+5TgCg1FBDLz/jWswj243qQ=; b=iuR5oB73lJmx5mfnSwzZJaeeUof3vqtjhfPxAn8N8hvhSP7G0rjoPUq3k8Vs1iKWhv HRRrHTR7+4kAhuuCOYKg88cyH5JiWm9F7bzPFJB8z3u4eqy7+vxZM0GiJEOJ3401OTLf +qEmMKGvKDUZg4Q9frhup6MNfFrAKQOaXZV+erbhVKijOQrc8/pL6Lb3ewao5eqdkqqh lqpPPN8gemMkBTrFNHndH1yS/QMKYhyTXU+Vdl0Bl846w0Kj0fYhzo7AYD9jnWlUTa1R oAPcStMdf8IR5tvIP1tMuYo2gSX7AeDlXBhJLGkKww7wGFhxaJEe5jTDsaOyM9P+nc8H QJFA== X-Gm-Message-State: AOJu0YzYjEPNeF88N2RQgEXDT7qRSUU7ziBgb+7bLXfSgwYVvS8b7nM1 KL6dWLlHRbFkGLUxAKU1Tm3Qf2UymEm7GDAcx4yhU0hNwlbxUXiAAkABrc1nrtRlzd9FGg== X-Gm-Gg: ASbGnctAzqa2ymfbHv4i0RTQQ2YlK1BxE29f1opTwKC0jOInLMaakMnGlGwfvP8DE40 megNseA2DW+UWqQf488MyAsybfk0Gdk2AZtBqkUfrVrVsZlzcrfvCE820fFijevaFdiKBeDOs1U Vj9eGy0pNsT7VlyYlx1jPkt+8bbALifDfP3k7UpygzjyOaN2MOHBTSU6jIa0wlirzocMBpPvy0U LfF7YMf4xesGqk/nhYlJ5b7Nq4ssiGMbHZCGUo4Spgu4WOWQLgXaWoa00R09baNDn0VJmXvB7E0 IOXZu6o5A+bmkJNmFmKvdEWV8qqSXu6yH8yr4OXr8xCqUKwdibIWDgOhrnJQnAthJH+Oqq/brzM KBChWejp0UZ3ZUg== X-Google-Smtp-Source: AGHT+IGFk35b7lo3PznIDfmxY09cjmnrYooSDQGHZT736TTbyDeUhjJ37Jmlig6/VEroSTdblRANWQ== X-Received: by 2002:a05:6512:15a9:b0:55f:492d:f70f with SMTP id 2adb3069b0e04-55f4f4c66a0mr1723242e87.23.1756287910710; Wed, 27 Aug 2025 02:45:10 -0700 (PDT) Received: from localhost ([5.187.32.133]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-55f35ca6ddesm2690390e87.139.2025.08.27.02.45.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Aug 2025 02:45:09 -0700 (PDT) X-Google-Original-From: Sergey Bronnikov To: tarantool-patches@dev.tarantool.org, Sergey Kaplun Date: Wed, 27 Aug 2025 12:44:37 +0300 Message-ID: <43f2870a9d46587fde4b3dd31c46af0563dac455.1756287598.git.sergeyb@tarantool.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH luajit 1/2] LJ_FR2: Fix stack checks in vararg calls. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks to Peter Cawley. (cherry picked from commit d1a2fef8a8f53b0055ee041f7f63d83a27444ffa) The builtin `pcall()` has two separate ways by which it can grow the stack by one slot: 1. Resolving the `__call` metamethod of its first argument. 2. Growing the stack by one slot in LJ_FR2 mode. The first case leads to a stack smash if `pcall()` is used as `__call`. Setting a metatable with this metamethod will cause an infinite loop which fills up the stack with `pcall`-frames and then keeps going beyond the end of the stack until it segfaults. Either of these points can cause an issue if `pcall()` is used as `__newindex`. The patch partially fixes aforementioned issues. Sergey Bronnikov: * added the description and the test for the problem Part of tarantool/tarantool#11691 --- src/lj_def.h | 2 +- src/lj_dispatch.c | 2 +- src/vm_arm64.dasc | 1 + src/vm_mips64.dasc | 1 + ...048-fix-stack-checks-vararg-calls.test.lua | 56 +++++++++++++++++++ 5 files changed, 60 insertions(+), 2 deletions(-) create mode 100644 test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua diff --git a/src/lj_def.h b/src/lj_def.h index a5bca6b0..7e4f251e 100644 --- a/src/lj_def.h +++ b/src/lj_def.h @@ -69,7 +69,7 @@ typedef unsigned int uintptr_t; #define LJ_MAX_UPVAL 60 /* Max. # of upvalues. */ #define LJ_MAX_IDXCHAIN 100 /* __index/__newindex chain limit. */ -#define LJ_STACK_EXTRA (5+2*LJ_FR2) /* Extra stack space (metamethods). */ +#define LJ_STACK_EXTRA (5+3*LJ_FR2) /* Extra stack space (metamethods). */ #define LJ_NUM_CBPAGE 1 /* Number of FFI callback pages. */ diff --git a/src/lj_dispatch.c b/src/lj_dispatch.c index a44a5adf..431cb3c2 100644 --- a/src/lj_dispatch.c +++ b/src/lj_dispatch.c @@ -453,7 +453,7 @@ static int call_init(lua_State *L, GCfunc *fn) int numparams = pt->numparams; int gotparams = (int)(L->top - L->base); int need = pt->framesize; - if ((pt->flags & PROTO_VARARG)) need += 1+gotparams; + if ((pt->flags & PROTO_VARARG)) need += 1+LJ_FR2+gotparams; lj_state_checkstack(L, (MSize)need); numparams -= gotparams; return numparams >= 0 ? numparams : 0; diff --git a/src/vm_arm64.dasc b/src/vm_arm64.dasc index c5f0a7a7..cf8e575a 100644 --- a/src/vm_arm64.dasc +++ b/src/vm_arm64.dasc @@ -3779,6 +3779,7 @@ static void build_ins(BuildCtx *ctx, BCOp op, int defop) | add TMP2, BASE, RC | add LFUNC:CARG3, CARG3, TMP0, lsl #47 | add RA, RA, RC + | sub CARG1, CARG1, #8 | add TMP0, RC, #16+FRAME_VARG | str LFUNC:CARG3, [TMP2], #8 // Store (tagged) copy of LFUNC. | ldr KBASE, [PC, #-4+PC2PROTO(k)] diff --git a/src/vm_mips64.dasc b/src/vm_mips64.dasc index 44fba36c..7f49df5b 100644 --- a/src/vm_mips64.dasc +++ b/src/vm_mips64.dasc @@ -5267,6 +5267,7 @@ static void build_ins(BuildCtx *ctx, BCOp op, int defop) | settp LFUNC:RB, TMP0 | daddu TMP0, RA, RC | sd LFUNC:RB, 0(TMP1) // Store (tagged) copy of LFUNC. + | daddiu TMP2, TMP2, -8 | daddiu TMP3, RC, 16+FRAME_VARG | sltu AT, TMP0, TMP2 | ld KBASE, -4+PC2PROTO(k)(PC) diff --git a/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua new file mode 100644 index 00000000..e300d5c1 --- /dev/null +++ b/test/tarantool-tests/lj-1048-fix-stack-checks-vararg-calls.test.lua @@ -0,0 +1,56 @@ +local tap = require('tap') + +-- A test file to demonstrate a stack overflow in `pcall()` in +-- some cases, see below testcase descriptions. +-- See also https://github.com/LuaJIT/LuaJIT/issues/1048. +local test = tap.test('lj-1048-fix-stack-checks-vararg-calls'):skipcond({ + ['Test requires JIT enabled'] = not jit.status(), +}) + +test:plan(2) + +-- The first testcase demonstrate a stack overflow in `pcall()` +-- by recursive calling `pcall()`. The functions are vararg +-- because stack check in BC_IFUNCV is off by one without the +-- patch. +local function prober_1(...) -- luacheck: no unused + pcall(pcall, pcall, pcall, pcall, pcall, pcall, pcall, pcall, pairs, {}) +end + +local function looper_1(n, ...) + prober_1(...) + prober_1(nil, ...) + return looper_1(n + 1, n, ...) +end + +pcall(coroutine.wrap(looper_1), 0) + +test:ok(true, 'no stack overflow with recursive pcall') + +-- The second testcase demonstrate a stack overflow in `pcall()` +-- with using metamethods. A stack overflow is triggered when +-- `pcall()` is used as `__call` metamethod, setting metatable +-- will cause an infinite loop which fills up the stack with +-- `pcall`-frames and then keeps going beyond the end of the +-- stack until it segfaults. Also, a stack overflow can be +-- triggered when `pcall()` is used as `__newindex` metamethod. +-- The functions are vararg because stack check in BC_IFUNCV is +-- off by one without the patch. + +local mt = setmetatable({}, { __newindex = pcall, __call = pairs }) + +local function prober_2(...) -- luacheck: no unused + mt[mt] = mt +end + +local function looper_2(n, ...) + prober_2(...) + prober_2(nil, ...) + return looper_2(n + 1, n, ...) +end + +pcall(coroutine.wrap(looper_2), 0) + +test:ok(true, 'no stack overflow with using metamethod') + +test:done(true) -- 2.43.0