Re: [Tarantool-patches] [PATCH luajit] LJ_GC64: Fix lua_concat().

[Maxim Kokryashkin via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Hi, Sergey!
Thanks for the patch!
Please consider my comments below.
On Tue, Sep 26, 2023 at 09:56:31AM +0300, Sergey Bronnikov via Tarantool-patches wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
> 
> Reported by Mathias Westerdahl.
> 
> (cherry picked from commit 633f265f67f322cbe2c5fd11d3e46d968ac220f7)
> 
> Lua 5.1 Reference Manual [1] defines a function `lua_concat`, that:
> 
> > void lua_concat (lua_State *L, int n);
> >
> > Concatenates the n values at the top of the stack, pops them, and leaves
> > the result at the top.
> 
> Without the patch `lua_concat()` behaved incorrectly with userdata with
> defined `__concat` metamethod. The problem is GC64-specific.
> 
> Assuming we have three literals and a userdata with defined "__concat"
> metamethod on top of the Lua stack:
> 
> 1 [string]
> 2 [string]
> 3 [string]
> 4 [string]
> 5 [userdata] <--- top
> 
> On attempt to concatenate *two* items on top Lua stack, `lua_concat()`
Typo: s/on top/on top of the/
> concatenates *four* items and leaves result at the top:
Typo: s/at the/on the/
> 
> 1 [string]
> 2 [string][string][string][userdata] <--- top
> 
> The problem is in incorrect calculation of `n` counter in a loop in
Typo: s/in a loop/in the loop/
> implementation of function `lua_concat`. Without a fix `n` is equal to 3
Typo: s/a fix/the fix/
> at the end of the first iteration and therefore it goes to the next
> iteration of concatenation. In a fixed implementation of `lua_concat()`
Typo: s/In a fixed/In the fixed/
> `n` is equal to 1 at the end of the first loop iteration, decremented in
> a loop postcondition and breaks a loop.
Typo: s/a loop/the loop/
> 
> The patch fixes incorrect behaviour.
> 
> 1. https://www.lua.org/manual/5.1/manual.html
> 
> Sergey Bronnikov:
> * added the description and the test for the problem
> 
> Part of tarantool/tarantool#8825
> ---
> PR: https://github.com/tarantool/tarantool/pull/9176
> Branch: https://github.com/tarantool/luajit/commits/ligurio/lj-881-fix-concat
> Issues:
> 	- https://github.com/LuaJIT/LuaJIT/issues/881
> 	- https://github.com/tarantool/tarantool/issues/8825
> 
>  src/lj_api.c                                  |   2 +-
>  .../lj-881-fix-lua-concat.test.c              | 116 ++++++++++++++++++
>  2 files changed, 117 insertions(+), 1 deletion(-)
>  create mode 100644 test/tarantool-c-tests/lj-881-fix-lua-concat.test.c
> 
> diff --git a/src/lj_api.c b/src/lj_api.c
> index 05e02029..3bacad33 100644
> --- a/src/lj_api.c
> +++ b/src/lj_api.c
> @@ -801,7 +801,7 @@ LUA_API void lua_concat(lua_State *L, int n)
>  	L->top -= n;
>  	break;
>        }
> -      n -= (int)(L->top - top);
> +      n -= (int)(L->top - (top - 2*LJ_FR2));
>        L->top = top+2;
>        jit_secure_call(L, top, 1+1);
>        L->top -= 1+LJ_FR2;
> diff --git a/test/tarantool-c-tests/lj-881-fix-lua-concat.test.c b/test/tarantool-c-tests/lj-881-fix-lua-concat.test.c
> new file mode 100644
> index 00000000..464ed27e
> --- /dev/null
> +++ b/test/tarantool-c-tests/lj-881-fix-lua-concat.test.c
> @@ -0,0 +1,116 @@
> +#include <string.h>
> +
> +#include "lua.h"
> +#include "lauxlib.h"
> +
> +#include "test.h"
> +#include "utils.h"
> +
> +/*
> + * This test demonstrates LuaJIT's incorrect behaviour, when
> + * calling `lua_concat()` with userdata with __concat metamethod.
> + * See https://github.com/LuaJIT/LuaJIT/issues/881 for details.
> + */
> +
> +#define TYPE_NAME_INFO "Info"
> +
> +typedef struct Info
> +{
> +	int value;
> +} Info;
> +
> +static void lua_pushinfo(lua_State *L, Info *info)
> +{
> +	Info *infop = (Info *)lua_newuserdata(L, sizeof(Info));
> +	*infop = *info;
> +
> +	luaL_getmetatable(L, TYPE_NAME_INFO);
> +	lua_setmetatable(L, -2);
> +}
> +
> +static int INFO_concat(lua_State *L)
> +{
> +	const char *s = luaL_checkstring(L, 1);
> +	Info *info = (Info*)luaL_checkudata(L, 2, TYPE_NAME_INFO);
> +	lua_pushfstring(L, "%s[%s.value=%d]", s, TYPE_NAME_INFO, info->value);
> +	return 1;
> +}
> +
> +static const luaL_Reg INFO_methods[] =
> +{
> +	{0, 0}
Nit: It is more common to write `NULL` as sentinel.
> +};
This table is redundant.
> +
> +static const luaL_Reg INFO_meta[] =
> +{
> +	{"__concat",    INFO_concat},
> +	{0, 0}
Nit: It is more common to write `NULL` as sentinel.
> +};
> +
> +static int lua_concat_testcase(void *test_state)
> +{
> +	lua_State *L = test_state;
> +
> +	/* Create methods table, add it to the globals. */
> +	luaL_register(L, TYPE_NAME_INFO, INFO_methods);
> +	int methods_idx = lua_gettop(L);
The methods table is redundant.
> +	/* Create metatable and add it to the Lua registry. */
> +	luaL_newmetatable(L, TYPE_NAME_INFO);
> +
> +	int metatable_idx = lua_gettop(L);
> +	/* Fill metatable. */
> +	luaL_register(L, 0, INFO_meta);
The middle argument is meant to be NULL.
> +
> +	lua_pushliteral(L, "__metatable");
> +	/* Duplicate methods table. */
> +	lua_pushvalue(L, methods_idx);
> +	lua_settable(L, metatable_idx);
> +	lua_pop(L, 2);

This whole part can be reduced to this:
| /* Create metatable and add it to the Lua registry. */
| luaL_newmetatable(L, TYPE_NAME_INFO);
|
| /* Fill metatable. */
| luaL_register(L, 0, INFO_meta);
| lua_pop(L, 1);
|

> +
> +	assert_true(lua_gettop(L) == 0);
> +
> +	Info info;
> +	info.value = 7;
> +
> +	lua_pushliteral(L, "C");
> +	lua_pushliteral(L, "B");
> +	lua_pushliteral(L, "A");
> +	lua_pushinfo(L, &info);
> +
> +	int top = 4;
> +	assert_true(lua_gettop(L) == top);
> +
> +	/**
> +	 * void lua_concat (lua_State *L, int n);
> +	 *
> +	 * Concatenates the n values at the top of the stack,
> +	 * pops them, and leaves the result at the top. If n is 1,
> +	 * the result is the single value on the stack; if n is 0,
> +	 * the result is the empty string [1].
> +	 *
> +	 * 1. https://www.lua.org/manual/5.1/manual.html
> +	 */
> +	int n = 2;
> +	lua_concat(L, n);
> +
> +	const char *str = lua_tostring(L, -1);
> +	assert_true(top - n + 1 == lua_gettop(L));
We have `assert_int_equal` in `test.h` for that.
> +	assert_true(strcmp(str, "A[Info.value=7]") == 0);
One should never use any of the unrestricted string functions, as
they are unsafe. Always use the `n`-limited versions: `strncmp`
instead of `strcmp`, `strnlen` instead of `strlen`, etc.

Side note: there is a TODO in test.h about `assert_str_equal`,
maybe it's a great time to introduce it in this patch.

> +
> +	/* Cleanup. */
> +	lua_settop(L, 0);
> +
> +	return TEST_EXIT_SUCCESS;
> +}
Some general comments about that function:
1. Please mark variables as `const` where possible.
2. C style requires you to put variable declarations at the
top of the function.
3. You have three distinct parts in this function: set up procedure,
the test itself, and the the tear down. It would be better to split them
into three separate functions to improve readability.
> +
> +int main(void)
> +{
> +	lua_State *L = utils_lua_init();
> +	const struct test_unit tgroup[] = {
> +		test_unit_def(lua_concat_testcase),
> +	};
> +	const int test_result = test_run_group(tgroup, L);
> +	utils_lua_close(L);
> +
> +	return test_result;
> +}
> -- 
> 2.34.1
> 
Best regards,
Maxim Kokryashkin