From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 22BD36EC40; Fri, 13 Aug 2021 12:29:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 22BD36EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1628846957; bh=WG0kNhnd+qSdMkrkJp/XifhJtFyHJmQyvfeC18l5Fxw=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=AzwHgPiHId20O6M0+KUevWK+12MNp32xIOPPIotSHY9ieO5OnjfcGMI3Zf+hWe/I8 lZmh3dSo3pqnyaB6GMqChpXRrJvBIEHoOrwWdqSjuQRqUCSO0bYYNsH14/74OWqQTW N1L9hQ+zBQP+naZwvMhWi+8JnyhbMimcJOpLsr1w= Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 1041D6EC40 for ; Fri, 13 Aug 2021 12:29:16 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1041D6EC40 Received: by smtp41.i.mail.ru with esmtpa (envelope-from ) id 1mETV4-0007vE-PE; Fri, 13 Aug 2021 12:29:15 +0300 Date: Fri, 13 Aug 2021 12:27:58 +0300 To: Igor Munkin , Vladislav Shpilevoy , Vitaliia Ioffe , tarantool-patches@dev.tarantool.org Message-ID: References: <20210618181416.25454-1-skaplun@tarantool.org> <20210801123417.GA27855@tarantool.org> <20210812173516.GN27855@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD9736CF3E71F18CE0C3E1D5927724F4AAA182A05F5380850408ED3D7DC26F010848397C4D5782D1389FA42C93593C2F8BD0A1C3CB38705C3E1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7C2204D4F9A221771EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637279A5203CF71F5518638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D883366E7B11465FDF209B9FDF705EA498117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6300D3B61E77C8D3B089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: 0D63561A33F958A54D415AA245A45A18F544510BA31150AEF52C5945FD2E2988D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA750E14360347543F58410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D9DC20663B80603F30515AF7BB6BD786E83831A2DEDDBE56DD73A714E89F9ED714AB5F2D1F0409741D7E09C32AA3244C92AD537300D5028169E262268FB75C28F522A1CF68F4BE05927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj0dLV0c3jbkwBprENyudPbw== X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB419E59567F1F2DD192DE3C53C01459DA03BCBA6A6DD56719AF2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A89437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] lua: refactor port_lua_do_dump and encode_lua_call X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Also, add the branch for 1.10: https://github.com/tarantool/tarantool/tree/skaplun/gh-noticket-refactor-lua-call-1.10 The difference is that the port structure itself is pushed instead ctx structure on the stack. The patch is the following: =================================================================== commit 59eafb8e6d8f41de06bdb28b930c9ff1d3fbe63e Author: Sergey Kaplun Date: Fri Jun 18 18:28:30 2021 +0300 lua: refactor port_lua_do_dump and encode_lua_call (cherry picked from commit 434d31bdb52d80384af55acd2c3a637e5154e257) The old code flow was the following: 1) `struct port_lua` given to `port_lua_do_dump()` has Lua stack with arguments to encode to MessagePack. 2) The main coroutine `tarantool_L` is used to call `encode_lua_call()` or `encode_lua_call_16`() via `lua_cpcall()`. 3) Objects on port coroutine are encoded via `luamp_encode()` or `luamp_encode_call16()`. 4) This encoding may raise an error on unprotected `port->L` coroutine. This coroutine has no protected frame on it and this call should fail in pure Lua. Calling anything on unprotected coroutine is not allowed in Lua [1]: | If an error happens outside any protected environment, Lua calls a | panic function Lua 5.1 sets protection only for specific lua_State [2] and calls a panic function if we raise an error on unprotected lua_State [3]. Nevertheless, no panic occurs now due to two facts: * The first one is LuaJIT's support of C++ exception handling [4] that allows to raise an error in Lua and catch it in C++ or vice versa. But documentation still doesn't allow raising errors on unprotected coroutines (at least we must use try-catch block). * The second one is the patch made in LuaJIT to restore currently executed coroutine, when C function or fast function raises an error [5][6] (see the related issue here [7][8]). For these reasons, when an error occurs, the unwinder searches and finds the C-protected stack frame from the `lua_cpcall()` for `tarantool_L` coroutine and unwinds until that point (without aforementioned patches LuaJIT just calls a panic function and exits). If an error is raised, and `lua_cpcall()` returns not `LUA_OK`, then the error from `port->L` coroutine is converted into a Tarantool error and a diagnostic is set. Such auxiliary usage of `tarantool_L` is not idiomatic for Lua. Internal unwinder used on M1 is not such flexible, so such misuse leads to panic call. Also the `tarantool_L` usage is redundant. So this patch drops it and uses only port coroutine instead with `lua_pcall()`. Functions to encode are saved to the `LUA_REGISTRY` table to reduce GC pressure, like it is done for other handlers [9]. [1]: https://www.lua.org/manual/5.2/manual.html#4.6 [2]: https://www.lua.org/source/5.1/lstate.h.html#lua_State [3]: https://www.lua.org/source/5.1/ldo.c.html#luaD_throw [4]: https://luajit.org/extensions.html#exceptions [5]: https://github.com/tarantool/luajit/commit/ed412cd9f55fe87fd32a69c86e1732690fc5c1b0 [6]: https://github.com/tarantool/luajit/commit/97699d9ee2467389b6aea21a098e38aff3469b5f [7]: https://github.com/tarantool/tarantool/issues/1516 [8]: https://www.freelists.org/post/luajit/Issue-with-PCALL-in-21 [9]: https://github.com/tarantool/tarantool/commit/e88c0d21ab765d4c53bed2437c49d77b3ffe4216 Closes #6248 Closes #4617 diff --git a/src/box/lua/call.c b/src/box/lua/call.c index 593317265..f173d97b7 100644 --- a/src/box/lua/call.c +++ b/src/box/lua/call.c @@ -53,6 +53,8 @@ */ enum handlers { HANDLER_CALL, + HANDLER_ENCODE_CALL, + HANDLER_ENCODE_CALL_16, HANDLER_EVAL, HANDLER_MAX, }; @@ -347,10 +349,26 @@ execute_lua_eval(lua_State *L) return lua_gettop(L); } + +/** + * Encode call results to msgpack from Lua stack. + * Lua stack has the following structure -- the last element is + * lightuserdata pointer to encode_lua_ctx, all other values are + * arguments to process. + * The function encodes all given Lua objects to msgpack stream + * from context, sets port's size and returns no value on the Lua + * stack. + * XXX: This function *MUST* be called under lua_pcall(), because + * luamp_encode() may raise an error. + */ static int encode_lua_call(lua_State *L) { + assert(lua_islightuserdata(L, -1)); struct port_lua *port = (struct port_lua *) lua_topointer(L, -1); + assert(port->L == L); + /* Delete port from the stack. */ + lua_pop(L, 1); /* * Add all elements from Lua stack to the buffer. * @@ -361,18 +379,33 @@ encode_lua_call(lua_State *L) luamp_error, port->L); struct luaL_serializer *cfg = luaL_msgpack_default; - int size = lua_gettop(port->L); + const int size = lua_gettop(L); for (int i = 1; i <= size; ++i) - luamp_encode(port->L, cfg, &stream, i); + luamp_encode(L, cfg, &stream, i); port->size = size; mpstream_flush(&stream); return 0; } +/** + * Encode call_16 results to msgpack from Lua stack. + * Lua stack has the following structure -- the last element is + * lightuserdata pointer to encode_lua_ctx, all other values are + * arguments to process. + * The function encodes all given Lua objects to msgpack stream + * from context, sets port's size and returns no value on the Lua + * stack. + * XXX: This function *MUST* be called under lua_pcall(), because + * luamp_encode() may raise an error. + */ static int encode_lua_call_16(lua_State *L) { + assert(lua_islightuserdata(L, -1)); struct port_lua *port = (struct port_lua *) lua_topointer(L, -1); + assert(port->L == L); + /* Delete port from the stack. */ + lua_pop(L, 1); /* * Add all elements from Lua stack to the buffer. * @@ -383,42 +416,45 @@ encode_lua_call_16(lua_State *L) luamp_error, port->L); struct luaL_serializer *cfg = luaL_msgpack_default; - port->size = luamp_encode_call_16(port->L, cfg, &stream); + port->size = luamp_encode_call_16(L, cfg, &stream); mpstream_flush(&stream); return 0; } static inline int -port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler) +port_lua_do_dump(struct port *base, struct obuf *out, enum handlers handler) { struct port_lua *port = (struct port_lua *)base; assert(port->vtab == &port_lua_vtab); /* Use port to pass arguments to encoder quickly. */ port->out = out; + lua_State *L = port->L; /* - * Use the same global state, assuming the encoder doesn't - * yield. + * At the moment Lua stack holds only values to encode. + * Insert corresponding encoder to the bottom and push + * encode context as lightuserdata to the top. */ - struct lua_State *L = tarantool_L; - int top = lua_gettop(L); - if (lua_cpcall(L, handler, port) != 0) { - luaT_toerror(port->L); + const int size = lua_gettop(L); + lua_rawgeti(L, LUA_REGISTRYINDEX, execute_lua_refs[handler]); + assert(lua_isfunction(L, -1) && lua_iscfunction(L, -1)); + lua_insert(L, 1); + lua_pushlightuserdata(L, port); + /* nargs -- all arguments + lightuserdata. */ + if (luaT_call(L, size + 1, 0) != 0) return -1; - } - lua_settop(L, top); return port->size; } static int port_lua_dump(struct port *base, struct obuf *out) { - return port_lua_do_dump(base, out, encode_lua_call); + return port_lua_do_dump(base, out, HANDLER_ENCODE_CALL); } static int port_lua_dump_16(struct port *base, struct obuf *out) { - return port_lua_do_dump(base, out, encode_lua_call_16); + return port_lua_do_dump(base, out, HANDLER_ENCODE_CALL_16); } static void @@ -499,6 +535,8 @@ box_lua_call_init(struct lua_State *L) lua_CFunction handles[] = { [HANDLER_CALL] = execute_lua_call, + [HANDLER_ENCODE_CALL] = encode_lua_call, + [HANDLER_ENCODE_CALL_16] = encode_lua_call_16, [HANDLER_EVAL] = execute_lua_eval, }; =================================================================== -- Best regards, Sergey Kaplun