Tarantool development patches archive
 help / color / mirror / Atom feed
From: Kirill Shcherbatov <kshcherbatov@tarantool.org>
To: tarantool-patches@freelists.org,
	Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Subject: [tarantool-patches] box: disable sparse optimization in box.tuple.new()
Date: Tue, 12 Feb 2019 14:18:52 +0300	[thread overview]
Message-ID: <708eb7bb-88a5-7663-3682-bd4d4a61d2c5@tarantool.org> (raw)
In-Reply-To: <38d2414b18464bc28be295281de1779fd7cfb1cc.1549881301.git.kshcherbatov@tarantool.org>

The box.tuple.new() used to call luamp_encode_tuple with
default LUA serializer config 'luaL_msgpack_default'. This
routine may consider an array to be excessively sparse when
  + encode_sparse_ratio > 0
  + max(table) > encode_sparse_safe
  + max(table) > count(table) * encode_sparse_ratio.
Sparse optimization save memory via representing excessively
sparse tuple as MP_MAP. But Tarantool tuple always must be
MP_ARRAY so it is not relevant for box.tuple.new semantics.
So it is disabled with encode_sparse_ratio = 0 in a new local
serializer config.

Closes #3882
---
 src/box/lua/tuple.c     |  9 ++++++++
 test/box/tuple.result   | 50 +++++++++++++++++++++++++++++++++++++++++
 test/box/tuple.test.lua | 26 +++++++++++++++++++++
 3 files changed, 85 insertions(+)

diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c
index 756856f4e..52fb79d5b 100644
--- a/src/box/lua/tuple.c
+++ b/src/box/lua/tuple.c
@@ -108,8 +108,17 @@ lbox_tuple_new(lua_State *L)
 		      luamp_error, L);
 
 	if (argc == 1 && (lua_istable(L, 1) || luaT_istuple(L, 1))) {
+		/**
+		 * Disable storage optimization for excessively
+		 * sparse arrays as a tuple always must be regular
+		 * MP_ARRAY.
+		 */
+		int encode_sparse_ratio =
+			luaL_msgpack_default->encode_sparse_ratio;
+		luaL_msgpack_default->encode_sparse_ratio = 0;
 		/* New format: box.tuple.new({1, 2, 3}) */
 		luamp_encode_tuple(L, luaL_msgpack_default, &stream, 1);
+		luaL_msgpack_default->encode_sparse_ratio = encode_sparse_ratio;
 	} else {
 		/* Backward-compatible format: box.tuple.new(1, 2, 3). */
 		mpstream_encode_array(&stream, argc);
diff --git a/test/box/tuple.result b/test/box/tuple.result
index b42012485..16aa66b1a 100644
--- a/test/box/tuple.result
+++ b/test/box/tuple.result
@@ -1164,3 +1164,53 @@ test_run:cmd("clear filter")
 ---
 - true
 ...
+--
+-- gh-3882: Inappropriate storage optimization for sparse arrays
+--          in box.tuple.new.
+--
+t = {}
+---
+...
+t[1] = 1
+---
+...
+t[2] = 2
+---
+...
+t[11] = 11
+---
+...
+box.tuple.new(t)
+---
+- [1, 2, null, null, null, null, null, null, null, null, 11]
+...
+s2 = box.schema.space.create('test')
+---
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+s2:format({{name="a", type="str"}, {name="b", type="str", is_nullable=true},
+           {name="c", type="str", is_nullable=true},
+           {name="d", type="str", is_nullable=true},
+           {name="e", type="str", is_nullable=true},
+           {name="f", type="str", is_nullable=true},
+           {name="g", type="str", is_nullable=true},
+           {name="h", type="str", is_nullable=true},
+           {name="i", type="str", is_nullable=true},
+           {name="j", type="str", is_nullable=true},
+           {name="k", type="str", is_nullable=true}});
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+s2:frommap({a="1", k="11"})
+---
+- ['1', null, null, null, null, null, null, null, null, null, '11']
+...
+s2:drop()
+---
+...
diff --git a/test/box/tuple.test.lua b/test/box/tuple.test.lua
index 276bb0f67..0c89feace 100644
--- a/test/box/tuple.test.lua
+++ b/test/box/tuple.test.lua
@@ -384,3 +384,29 @@ t2 = box.tuple.new(2)
 t1 = t1:update{{'+', 1, 1}}
 
 test_run:cmd("clear filter")
+
+--
+-- gh-3882: Inappropriate storage optimization for sparse arrays
+--          in box.tuple.new.
+--
+t = {}
+t[1] = 1
+t[2] = 2
+t[11] = 11
+box.tuple.new(t)
+
+s2 = box.schema.space.create('test')
+test_run:cmd("setopt delimiter ';'")
+s2:format({{name="a", type="str"}, {name="b", type="str", is_nullable=true},
+           {name="c", type="str", is_nullable=true},
+           {name="d", type="str", is_nullable=true},
+           {name="e", type="str", is_nullable=true},
+           {name="f", type="str", is_nullable=true},
+           {name="g", type="str", is_nullable=true},
+           {name="h", type="str", is_nullable=true},
+           {name="i", type="str", is_nullable=true},
+           {name="j", type="str", is_nullable=true},
+           {name="k", type="str", is_nullable=true}});
+test_run:cmd("setopt delimiter ''");
+s2:frommap({a="1", k="11"})
+s2:drop()
-- 
2.20.1

  reply	other threads:[~2019-02-12 11:18 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-11 10:43 [tarantool-patches] [PATCH v1 1/1] " Kirill Shcherbatov
2019-02-12 11:18 ` Kirill Shcherbatov [this message]
2019-02-12 11:33   ` [tarantool-patches] " Konstantin Osipov
2019-02-12 17:50   ` Vladislav Shpilevoy
2019-02-13 12:19     ` Konstantin Osipov
2019-02-15 15:17     ` Kirill Shcherbatov
2019-02-15 21:31       ` Vladislav Shpilevoy
2019-02-18  8:45 ` [tarantool-patches] Re: [PATCH v1 1/1] " Kirill Yukhin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=708eb7bb-88a5-7663-3682-bd4d4a61d2c5@tarantool.org \
    --to=kshcherbatov@tarantool.org \
    --cc=tarantool-patches@freelists.org \
    --cc=v.shpilevoy@tarantool.org \
    --subject='Re: [tarantool-patches] box: disable sparse optimization in box.tuple.new()' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox