From: Kirill Shcherbatov <kshcherbatov@tarantool.org> To: tarantool-patches@freelists.org, Vladislav Shpilevoy <v.shpilevoy@tarantool.org> Subject: [tarantool-patches] box: disable sparse optimization in box.tuple.new() Date: Tue, 12 Feb 2019 14:18:52 +0300 [thread overview] Message-ID: <708eb7bb-88a5-7663-3682-bd4d4a61d2c5@tarantool.org> (raw) In-Reply-To: <38d2414b18464bc28be295281de1779fd7cfb1cc.1549881301.git.kshcherbatov@tarantool.org> The box.tuple.new() used to call luamp_encode_tuple with default LUA serializer config 'luaL_msgpack_default'. This routine may consider an array to be excessively sparse when + encode_sparse_ratio > 0 + max(table) > encode_sparse_safe + max(table) > count(table) * encode_sparse_ratio. Sparse optimization save memory via representing excessively sparse tuple as MP_MAP. But Tarantool tuple always must be MP_ARRAY so it is not relevant for box.tuple.new semantics. So it is disabled with encode_sparse_ratio = 0 in a new local serializer config. Closes #3882 --- src/box/lua/tuple.c | 9 ++++++++ test/box/tuple.result | 50 +++++++++++++++++++++++++++++++++++++++++ test/box/tuple.test.lua | 26 +++++++++++++++++++++ 3 files changed, 85 insertions(+) diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c index 756856f4e..52fb79d5b 100644 --- a/src/box/lua/tuple.c +++ b/src/box/lua/tuple.c @@ -108,8 +108,17 @@ lbox_tuple_new(lua_State *L) luamp_error, L); if (argc == 1 && (lua_istable(L, 1) || luaT_istuple(L, 1))) { + /** + * Disable storage optimization for excessively + * sparse arrays as a tuple always must be regular + * MP_ARRAY. + */ + int encode_sparse_ratio = + luaL_msgpack_default->encode_sparse_ratio; + luaL_msgpack_default->encode_sparse_ratio = 0; /* New format: box.tuple.new({1, 2, 3}) */ luamp_encode_tuple(L, luaL_msgpack_default, &stream, 1); + luaL_msgpack_default->encode_sparse_ratio = encode_sparse_ratio; } else { /* Backward-compatible format: box.tuple.new(1, 2, 3). */ mpstream_encode_array(&stream, argc); diff --git a/test/box/tuple.result b/test/box/tuple.result index b42012485..16aa66b1a 100644 --- a/test/box/tuple.result +++ b/test/box/tuple.result @@ -1164,3 +1164,53 @@ test_run:cmd("clear filter") --- - true ... +-- +-- gh-3882: Inappropriate storage optimization for sparse arrays +-- in box.tuple.new. +-- +t = {} +--- +... +t[1] = 1 +--- +... +t[2] = 2 +--- +... +t[11] = 11 +--- +... +box.tuple.new(t) +--- +- [1, 2, null, null, null, null, null, null, null, null, 11] +... +s2 = box.schema.space.create('test') +--- +... +test_run:cmd("setopt delimiter ';'") +--- +- true +... +s2:format({{name="a", type="str"}, {name="b", type="str", is_nullable=true}, + {name="c", type="str", is_nullable=true}, + {name="d", type="str", is_nullable=true}, + {name="e", type="str", is_nullable=true}, + {name="f", type="str", is_nullable=true}, + {name="g", type="str", is_nullable=true}, + {name="h", type="str", is_nullable=true}, + {name="i", type="str", is_nullable=true}, + {name="j", type="str", is_nullable=true}, + {name="k", type="str", is_nullable=true}}); +--- +... +test_run:cmd("setopt delimiter ''"); +--- +- true +... +s2:frommap({a="1", k="11"}) +--- +- ['1', null, null, null, null, null, null, null, null, null, '11'] +... +s2:drop() +--- +... diff --git a/test/box/tuple.test.lua b/test/box/tuple.test.lua index 276bb0f67..0c89feace 100644 --- a/test/box/tuple.test.lua +++ b/test/box/tuple.test.lua @@ -384,3 +384,29 @@ t2 = box.tuple.new(2) t1 = t1:update{{'+', 1, 1}} test_run:cmd("clear filter") + +-- +-- gh-3882: Inappropriate storage optimization for sparse arrays +-- in box.tuple.new. +-- +t = {} +t[1] = 1 +t[2] = 2 +t[11] = 11 +box.tuple.new(t) + +s2 = box.schema.space.create('test') +test_run:cmd("setopt delimiter ';'") +s2:format({{name="a", type="str"}, {name="b", type="str", is_nullable=true}, + {name="c", type="str", is_nullable=true}, + {name="d", type="str", is_nullable=true}, + {name="e", type="str", is_nullable=true}, + {name="f", type="str", is_nullable=true}, + {name="g", type="str", is_nullable=true}, + {name="h", type="str", is_nullable=true}, + {name="i", type="str", is_nullable=true}, + {name="j", type="str", is_nullable=true}, + {name="k", type="str", is_nullable=true}}); +test_run:cmd("setopt delimiter ''"); +s2:frommap({a="1", k="11"}) +s2:drop() -- 2.20.1
next prev parent reply other threads:[~2019-02-12 11:18 UTC|newest] Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-02-11 10:43 [tarantool-patches] [PATCH v1 1/1] " Kirill Shcherbatov 2019-02-12 11:18 ` Kirill Shcherbatov [this message] 2019-02-12 11:33 ` [tarantool-patches] " Konstantin Osipov 2019-02-12 17:50 ` Vladislav Shpilevoy 2019-02-13 12:19 ` Konstantin Osipov 2019-02-15 15:17 ` Kirill Shcherbatov 2019-02-15 21:31 ` Vladislav Shpilevoy 2019-02-18 8:45 ` [tarantool-patches] Re: [PATCH v1 1/1] " Kirill Yukhin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=708eb7bb-88a5-7663-3682-bd4d4a61d2c5@tarantool.org \ --to=kshcherbatov@tarantool.org \ --cc=tarantool-patches@freelists.org \ --cc=v.shpilevoy@tarantool.org \ --subject='Re: [tarantool-patches] box: disable sparse optimization in box.tuple.new()' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox