[Tarantool-patches] [PATCH luajit] Fix zero stripping in %g number formatting.

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

From: Mike Pall <mike>

Reported by pwnhacker0x18.

(cherry picked from commit 343ce0edaf3906a62022936175b2f5410024cbfc)

In the situation when the precision (`prec`) and amount of digits
(`hilen`) for the decimal representation are the same and `ndhi` == 0,
the `ndlo` part will become 64 (the size of the `nd` stack buffer), and
the overflow occurs.

This patch adds the corresponding mask (0x3f == 63) for the `ndlo`
incrementation result.

Sergey Kaplun:
* added the description and the test for the problem

Part of tarantool/tarantool#9595
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/lj-1149-g-number-formating
Tarantool PR: https://github.com/tarantool/tarantool/pull/9633

The test <app-tap/gh-2717-no-quit-sigint.test.lua> fails on M1 with the
timeout (see the example [1]). This fail is patch-unrelated, since I've
obscured this failure even for the branch without sources changes (tests
only).

Related Issues:
* https://github.com/LuaJIT/LuaJIT/issues/1149
* https://github.com/tarantool/tarantool/issues/9595

[1]: https://github.com/tarantool/luajit/actions/runs/7712549489/job/21020513973#step:8:5522

Duration of failed tests (seconds):
* 60.54 app-tap/gh-2717-no-quit-sigint.test.lua

 src/lj_strfmt_num.c                           |  3 ++-
 .../lj-1149-g-number-formating-bufov.test.lua | 20 +++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100644 test/tarantool-tests/lj-1149-g-number-formating-bufov.test.lua

diff --git a/src/lj_strfmt_num.c b/src/lj_strfmt_num.c
index c26204b7..c8d9febf 100644
--- a/src/lj_strfmt_num.c
+++ b/src/lj_strfmt_num.c
@@ -454,7 +454,8 @@ static char *lj_strfmt_wfnum(SBuf *sb, SFormat sf, lua_Number n, char *p)
 	    prec--;
 	    if (!i) {
 	      if (ndlo == ndhi) { prec = 0; break; }
-	      lj_strfmt_wuint9(tail, nd[++ndlo]);
+	      ndlo = (ndlo + 1) & 0x3f;
+	      lj_strfmt_wuint9(tail, nd[ndlo]);
 	      i = 9;
 	    }
 	  }
diff --git a/test/tarantool-tests/lj-1149-g-number-formating-bufov.test.lua b/test/tarantool-tests/lj-1149-g-number-formating-bufov.test.lua
new file mode 100644
index 00000000..040fd5de
--- /dev/null
+++ b/test/tarantool-tests/lj-1149-g-number-formating-bufov.test.lua
@@ -0,0 +1,20 @@
+local tap = require('tap')
+
+-- Test file to demonstrate stack-buffer-overflow in the
+-- `lj_strfmt_wfnum()` call.
+-- See also: https://github.com/LuaJIT/LuaJIT/issues/1149.
+
+local test = tap.test('lj-1149-g-number-formating-bufov')
+test:plan(1)
+
+-- XXX: The test shows stack-buffer-overflow only under ASAN.
+-- The number value for the test is with the same precision
+-- (`prec` = 5) and amount of digits (`hilen` = 5) for the decimal
+-- representation. Hence, with `ndhi` == 0, the `ndlo` part will
+-- become 64 (the size of the `nd` stack buffer), and the overflow
+-- occurs.
+-- See details in the <src/lj_strfmt_num.c>:`lj_strfmt_wfnum()`.
+test:is(string.format('%7g', 0x1.144399609d407p+401), '5.5733e+120',
+        'correct format %7g result')
+
+test:done(true)
-- 
2.43.0