Hi!

Thanks to @igormunkin we discovered more details of this problem.
Turns out we can see lua_State stored in port and it seems that
something is totally wrong here: it points to some function address
(port_lua_dump_16()). Thus I will make some more research into it and
this patch should be discarded for now.
(gdb) p ((struct port_lua *)port)->vtab
$19 = (const port_vtab *) 0x4e759f <port_lua_dump>
(gdb) p ((struct port_lua *)port)->L
$20 = (lua_State *) 0x4e75c9 <port_lua_dump_16>
  
>Среда, 22 июля 2020, 11:59 +03:00 от Nikita Pettik <korablev@tarantool.org>:
> 
>On 22 Jul 01:32, Ilya Kosarev wrote:
>> (gdb) p port->vtab->dump_msgpack
>> count = port_dump_msgpack_16(&port, out);
>> else
>> count = port_dump_msgpack(&port, out);
>> port_destroy(&port);
>> if (count < 0) {
>> obuf_rollback_to_svp(out, &svp);
>> goto error;
>> }
>>
>> iproto_reply_select(out, &svp, msg->header.sync,
>> ::schema_version, count);
>> iproto_wpos_create(&msg->wpos, out);
>> return;
>> error:
>> tx_reply_error(msg);
>> }
>>
>> As we can see, we fail to the error path through count == -1. It comes
>> from port_lua_dump() calling port_lua_do_dump() with encode_lua_call().
>>
>> static int
>> port_lua_dump(struct port *base, struct obuf *out)
>> {
>> return port_lua_do_dump(base, out, encode_lua_call);
>> }
>>
>> encode_lua_call() actually sets port->size equal to lua_gettop(), being
>> returned by port_lua_do_dump().
>
>As a first assumption it is OK. But there might be other reasons
>for negative value in size var. For instance, out-of-bound memory
>access in luamp_encode() which overwrites value of stack var..
> 
>> src/lj_api.c | 7 ++++++-
>> 1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/lj_api.c b/src/lj_api.c
>> index c9b5d22..5ca2c57 100644
>> --- a/src/lj_api.c
>> +++ b/src/lj_api.c
>> @@ -135,7 +135,12 @@ LUA_API const lua_Number *lua_version(lua_State *L)
>>
>> LUA_API int lua_gettop(lua_State *L)
>> {
>> - return (int)(L->top - L->base);
>> + int stack_size = (int)(L->top - L->base);
>> + /*
>> + * Stack size has to be non-negative according to Lua reference manual.
>> + */
>> + assert(stack_size >= 0);
>> + return stack_size;
>> }
>>
>> LUA_API void lua_settop(lua_State *L, int idx)
>> --
>> 2.17.1
>> 
 
 
--
Ilya Kosarev