From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id BB7CD22561 for ; Wed, 4 Sep 2019 17:41:20 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BTHn3AV8PKql for ; Wed, 4 Sep 2019 17:41:20 -0400 (EDT) Received: from smtpng3.m.smailru.net (smtpng3.m.smailru.net [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 791D721B00 for ; Wed, 4 Sep 2019 17:41:20 -0400 (EDT) From: Vladislav Shpilevoy Subject: [tarantool-patches] [PATCH 3/4] tuple: use global msgpack serializer in Lua tuple Date: Wed, 4 Sep 2019 23:44:46 +0200 Message-Id: <9af85b590f9baa6ce80a4eb506f001c655765b84.1567633062.git.v.shpilevoy@tarantool.org> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: tarantool-patches@freelists.org Cc: alexander.turenko@tarantool.org Tuple is a C library exposed to Lua. In Lua to translate Lua objects into tuples and back luaL_serializer structure is used. In Tarantool we have several global serializers, one of which is for msgpack. Tuples store data in msgpack, and in theory should have used that global msgpack serializer. But in fact the tuple module had its own private serializer because of tuples encoding specifics such as never encode sparse arrays as maps. This patch makes tuple Lua module use global msgpack serializer always. But how does tuple handle sparse arrays now? In fact, the tuple module still has its own serializer, but it is updated each time when the msgpack serializer is changed. Part of #4434 --- src/box/lua/tuple.c | 32 ++++++++++++++++++-------- src/lua/utils.c | 2 ++ src/lua/utils.h | 9 ++++++++ test/box/tuple.result | 46 ++++++++++++++++++++++++++++++++++++++ test/box/tuple.test.lua | 21 +++++++++++++++++ test/unit/luaT_tuple_new.c | 2 +- 6 files changed, 102 insertions(+), 10 deletions(-) diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c index 3902288bf..3ac32601c 100644 --- a/src/box/lua/tuple.c +++ b/src/box/lua/tuple.c @@ -58,8 +58,27 @@ static const char *tuplelib_name = "box.tuple"; static const char *tuple_iteratorlib_name = "box.tuple.iterator"; +/* + * Special serializer for box.tuple.new() to disable storage + * optimization for excessively sparse arrays as a tuple always + * must be regular MP_ARRAY. + */ static struct luaL_serializer tuple_serializer; +/** + * Take actual version of msgpack serializer slightly modified to + * correctly encode tuples. + */ +static inline struct luaL_serializer * +tuple_serializer_actualize(void) +{ + if (tuple_serializer.version == luaL_msgpack_default->version) + return &tuple_serializer; + tuple_serializer = *luaL_msgpack_default; + tuple_serializer.encode_sparse_ratio = 0; + return &tuple_serializer; +} + extern char tuple_lua[]; /* Lua source */ uint32_t CTID_STRUCT_TUPLE_REF; @@ -119,7 +138,8 @@ luaT_tuple_new(struct lua_State *L, int idx, box_tuple_format_t *format) } } else { /* Create the tuple from a Lua table. */ - luamp_encode_tuple(L, &tuple_serializer, &stream, idx); + luamp_encode_tuple(L, tuple_serializer_actualize(), &stream, + idx); } mpstream_flush(&stream); struct tuple *tuple = box_tuple_new(format, buf->buf, @@ -564,14 +584,8 @@ box_lua_tuple_init(struct lua_State *L) luamp_set_encode_extension(luamp_encode_extension_box); - /* - * Create special serializer for box.tuple.new(). - * Disable storage optimization for excessively - * sparse arrays as a tuple always must be regular - * MP_ARRAY. - */ - luaL_serializer_create(&tuple_serializer); - tuple_serializer.encode_sparse_ratio = 0; + tuple_serializer.version = -1; + tuple_serializer_actualize(); /* Get CTypeID for `struct tuple' */ int rc = luaL_cdef(L, "struct tuple;"); diff --git a/src/lua/utils.c b/src/lua/utils.c index a082a2e5b..ebbe208b4 100644 --- a/src/lua/utils.c +++ b/src/lua/utils.c @@ -251,6 +251,7 @@ static struct { void luaL_serializer_create(struct luaL_serializer *cfg) { + cfg->version = 0; for (int i = 0; OPTIONS[i].name != NULL; i++) { int *pval = (int *) ((char *) cfg + OPTIONS[i].offset); *pval = OPTIONS[i].defvalue; @@ -313,6 +314,7 @@ luaL_serializer_cfg(struct lua_State *L) /* Updated parameters. */ luaL_checktype(L, 2, LUA_TTABLE); struct luaL_serializer *cfg = luaL_checkserializer(L); + ++cfg->version; for (int i = 0; OPTIONS[i].name != NULL; ++i) { if (luaL_serializer_parse_option(L, i, cfg) == NULL) lua_pop(L, 1); diff --git a/src/lua/utils.h b/src/lua/utils.h index d42cc3992..e802b8c4c 100644 --- a/src/lua/utils.h +++ b/src/lua/utils.h @@ -242,6 +242,15 @@ struct luaL_serializer { /** Enable support for compact represenation (internal, YAML-only). */ int has_compact; + /** + * Constantly growing number increased each time the + * serializer is changed. It can be used by modules which + * need certain settings constant. Such modules can keep a + * local copy of the serializer, and copy it from the main + * one only when the version is changed. This is what Lua + * tuple implementation did at the moment of writing this. + */ + int version; }; extern int luaL_nil_ref; diff --git a/test/box/tuple.result b/test/box/tuple.result index 895462518..83f74d111 100644 --- a/test/box/tuple.result +++ b/test/box/tuple.result @@ -1395,3 +1395,49 @@ d:update{{'-', 1, dec.new('1e37')}} --- - [80000000000000000000000000000000000000] ... +-- +-- gh-4434: tuple should use global msgpack serializer. +-- +max_depth = msgpack.cfg.encode_max_depth +--- +... +t = nil +--- +... +for i = 1, max_depth + 5 do t = {t} end +--- +... +tuple = box.tuple.new(t):totable() +--- +... +level = 0 +--- +... +while tuple ~= nil do level = level + 1 tuple = tuple[1] end +--- +... +level == max_depth or {level, max_depth} +--- +- true +... +msgpack.cfg({encode_max_depth = max_depth + 5}) +--- +... +tuple = box.tuple.new(t):totable() +--- +... +level = 0 +--- +... +while tuple ~= nil do level = level + 1 tuple = tuple[1] end +--- +... +-- Level should be bigger now, because the default msgpack +-- serializer allows deeper tables. +level == max_depth + 5 or {level, max_depth} +--- +- true +... +msgpack.cfg({encode_max_depth = max_depth}) +--- +... diff --git a/test/box/tuple.test.lua b/test/box/tuple.test.lua index 9762fc8b3..c8a0c03c5 100644 --- a/test/box/tuple.test.lua +++ b/test/box/tuple.test.lua @@ -473,3 +473,24 @@ d = box.tuple.new(dec.new('9e37')) d d:update{{'+', 1, dec.new('1e37')}} d:update{{'-', 1, dec.new('1e37')}} + +-- +-- gh-4434: tuple should use global msgpack serializer. +-- +max_depth = msgpack.cfg.encode_max_depth +t = nil +for i = 1, max_depth + 5 do t = {t} end +tuple = box.tuple.new(t):totable() +level = 0 +while tuple ~= nil do level = level + 1 tuple = tuple[1] end +level == max_depth or {level, max_depth} + +msgpack.cfg({encode_max_depth = max_depth + 5}) +tuple = box.tuple.new(t):totable() +level = 0 +while tuple ~= nil do level = level + 1 tuple = tuple[1] end +-- Level should be bigger now, because the default msgpack +-- serializer allows deeper tables. +level == max_depth + 5 or {level, max_depth} + +msgpack.cfg({encode_max_depth = max_depth}) diff --git a/test/unit/luaT_tuple_new.c b/test/unit/luaT_tuple_new.c index 609d64e45..8f25c8e07 100644 --- a/test/unit/luaT_tuple_new.c +++ b/test/unit/luaT_tuple_new.c @@ -170,8 +170,8 @@ main() luaL_openlibs(L); box_init(); - box_lua_tuple_init(L); luaopen_msgpack(L); + box_lua_tuple_init(L); lua_pop(L, 1); return test_basic(L); -- 2.20.1 (Apple Git-117)