* [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
@ 2021-06-23 19:12 Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file Alexander Turenko via Tarantool-patches
` (7 more replies)
0 siblings, 8 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-06-23 19:12 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov
Cc: Alexander Turenko, tarantool-patches
Moved the serializer helpers into its own compilation unit, add some
comments and a basic test: everything is just to simplify diving into
this code.
Guys, please, look, whether it seems useful enough to include into
tarantool's mainline? Should we name it serializer.[ch] or
somehow like serializer_helpers.[ch]?
Part of https://github.com/tarantool/tarantool/issues/3228
Branch: Totktonada/gh-3228-extract-serializer-helpers
Alexander Turenko (4):
lua: move serializer helpers into its own file
lua: move luaL_newserializer() comment into header
lua: split serializer functions into sections
test: add a basic unit test for serializer helpers
src/CMakeLists.txt | 1 +
src/box/lua/call.c | 1 +
src/box/lua/console.c | 1 +
src/box/lua/execute.c | 1 +
src/box/lua/info.c | 1 +
src/box/lua/init.c | 2 +
src/box/lua/serialize_lua.c | 1 +
src/box/lua/slab.c | 1 +
src/box/lua/tuple.c | 1 +
src/box/sql/mem.c | 1 +
src/lua/decimal.c | 2 +
src/lua/fiber.c | 1 +
src/lua/init.c | 5 +
src/lua/msgpack.c | 1 +
src/lua/msgpack.h | 1 +
src/lua/pickle.c | 1 +
src/lua/serializer.c | 653 ++++++++++++++++++++++++++++++
src/lua/serializer.h | 376 +++++++++++++++++
src/lua/utils.c | 601 +--------------------------
src/lua/utils.h | 292 +------------
test/unit/CMakeLists.txt | 3 +
test/unit/serializer.c | 444 ++++++++++++++++++++
test/unit/serializer.result | 135 ++++++
third_party/lua-cjson/lua_cjson.c | 1 +
third_party/lua-yaml/lyaml.cc | 1 +
25 files changed, 1639 insertions(+), 889 deletions(-)
create mode 100644 src/lua/serializer.c
create mode 100644 src/lua/serializer.h
create mode 100644 test/unit/serializer.c
create mode 100644 test/unit/serializer.result
--
2.31.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
@ 2021-06-23 19:12 ` Alexander Turenko via Tarantool-patches
2021-07-04 13:10 ` Vladislav Shpilevoy via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 2/4] lua: move luaL_newserializer() comment into header Alexander Turenko via Tarantool-patches
` (6 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-06-23 19:12 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov
Cc: Alexander Turenko, tarantool-patches
It is easier to glance on tightly coupled structures and functions, when
they're not mixed with others.
Just move without actual changes.
Part of #3228
---
src/CMakeLists.txt | 1 +
src/box/lua/call.c | 1 +
src/box/lua/console.c | 1 +
src/box/lua/execute.c | 1 +
src/box/lua/info.c | 1 +
src/box/lua/init.c | 2 +
src/box/lua/serialize_lua.c | 1 +
src/box/lua/slab.c | 1 +
src/box/lua/tuple.c | 1 +
src/box/sql/mem.c | 1 +
src/lua/decimal.c | 2 +
src/lua/fiber.c | 1 +
src/lua/init.c | 5 +
src/lua/msgpack.c | 1 +
src/lua/msgpack.h | 1 +
src/lua/pickle.c | 1 +
src/lua/serializer.c | 651 ++++++++++++++++++++++++++++++
src/lua/serializer.h | 344 ++++++++++++++++
src/lua/utils.c | 601 +--------------------------
src/lua/utils.h | 292 +-------------
third_party/lua-cjson/lua_cjson.c | 1 +
third_party/lua-yaml/lyaml.cc | 1 +
22 files changed, 1023 insertions(+), 889 deletions(-)
create mode 100644 src/lua/serializer.c
create mode 100644 src/lua/serializer.h
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 324574fec..c0e272bd9 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -122,6 +122,7 @@ set (server_sources
lua/trigger.c
lua/msgpack.c
lua/utils.c
+ lua/serializer.c
lua/errno.c
lua/tnt_iconv.c
lua/error.c
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 0315e720c..d42b54d42 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -38,6 +38,7 @@
#include "tt_static.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lua/msgpack.h"
#include "lua/trigger.h"
diff --git a/src/box/lua/console.c b/src/box/lua/console.c
index 2e8204db6..d97c3ff30 100644
--- a/src/box/lua/console.c
+++ b/src/box/lua/console.c
@@ -34,6 +34,7 @@
#include "box/port.h"
#include "box/error.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lua/fiber.h"
#include "fiber.h"
#include "coio.h"
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index 926a0a61c..1b59b2e4a 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -1,5 +1,6 @@
#include "execute.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lua/msgpack.h"
#include "box/sql/sqlInt.h"
#include "box/port.h"
diff --git a/src/box/lua/info.c b/src/box/lua/info.c
index 0eb48b823..f2bc42eee 100644
--- a/src/box/lua/info.c
+++ b/src/box/lua/info.c
@@ -52,6 +52,7 @@
#include "box/raft.h"
#include "box/txn_limbo.h"
#include "lua/utils.h"
+#include "lua/serializer.h" /* luaL_setmaphint */
#include "fiber.h"
#include "sio.h"
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index 3a6d60864..6eb2ab651 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -34,6 +34,8 @@
#include <lauxlib.h>
#include <lualib.h>
+#include "lib/core/mp_extension_types.h"
+
#include "lua/utils.h" /* luaT_error() */
#include "lua/trigger.h"
#include "lua/msgpack.h"
diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
index caa08a60f..7144305cf 100644
--- a/src/box/lua/serialize_lua.c
+++ b/src/box/lua/serialize_lua.c
@@ -34,6 +34,7 @@
#include "trivia/util.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "say.h"
#include "lib/core/decimal.h"
diff --git a/src/box/lua/slab.c b/src/box/lua/slab.c
index 9f5e7e95c..dd89a980f 100644
--- a/src/box/lua/slab.c
+++ b/src/box/lua/slab.c
@@ -32,6 +32,7 @@
#include "box/lua/slab.h"
#include "lua/utils.h"
+#include "lua/serializer.h" /* luaL_setmaphint */
#include <lua.h>
#include <lauxlib.h>
diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c
index f7198a025..609f2eda0 100644
--- a/src/box/lua/tuple.c
+++ b/src/box/lua/tuple.c
@@ -32,6 +32,7 @@
#include "box/xrow_update.h"
#include "lua/utils.h" /* luaT_error() */
+#include "lua/serializer.h"
#include "lua/msgpack.h" /* luamp_encode_XXX() */
#include "diag.h" /* diag_set() */
#include <small/ibuf.h>
diff --git a/src/box/sql/mem.c b/src/box/sql/mem.c
index 6f3bf52e5..2595e2fd4 100644
--- a/src/box/sql/mem.c
+++ b/src/box/sql/mem.c
@@ -38,6 +38,7 @@
#include "mpstream/mpstream.h"
#include "box/port.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lua/msgpack.h"
#include "uuid/mp_uuid.h"
#include "mp_decimal.h"
diff --git a/src/lua/decimal.c b/src/lua/decimal.c
index d3400521d..003680a48 100644
--- a/src/lua/decimal.c
+++ b/src/lua/decimal.c
@@ -79,6 +79,8 @@ ldecimal_##name(struct lua_State *L) { \
return 1; \
}
+uint32_t CTID_DECIMAL;
+
/** Push a new decimal on the stack and return a pointer to it. */
decimal_t *
lua_pushdecimal(struct lua_State *L)
diff --git a/src/lua/fiber.c b/src/lua/fiber.c
index 91898c283..d236e50c4 100644
--- a/src/lua/fiber.c
+++ b/src/lua/fiber.c
@@ -32,6 +32,7 @@
#include <fiber.h>
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "backtrace.h"
#include "tt_static.h"
diff --git a/src/lua/init.c b/src/lua/init.c
index 93e93a103..ff11d202b 100644
--- a/src/lua/init.c
+++ b/src/lua/init.c
@@ -69,6 +69,10 @@
#include <readline/readline.h>
#include <readline/history.h>
+/* Don't include the entire header only for *_init(). */
+int
+tarantool_lua_serializer_init(struct lua_State *L);
+
/**
* The single Lua state of the transaction processor (tx) thread.
*/
@@ -475,6 +479,7 @@ tarantool_lua_init(const char *tarantool_bin, int argc, char **argv)
tarantool_lua_socket_init(L);
tarantool_lua_pickle_init(L);
tarantool_lua_digest_init(L);
+ tarantool_lua_serializer_init(L);
tarantool_lua_swim_init(L);
tarantool_lua_decimal_init(L);
luaopen_http_client_driver(L);
diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c
index 1e74a6a3c..b6ecf2b1e 100644
--- a/src/lua/msgpack.c
+++ b/src/lua/msgpack.c
@@ -31,6 +31,7 @@
#include "lua/msgpack.h"
#include "mpstream/mpstream.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#if defined(LUAJIT)
#include <lj_ctype.h>
diff --git a/src/lua/msgpack.h b/src/lua/msgpack.h
index 5a91e2812..cf85654a5 100644
--- a/src/lua/msgpack.h
+++ b/src/lua/msgpack.h
@@ -41,6 +41,7 @@ extern "C" {
#include <lua.h>
+struct luaL_field;
struct luaL_serializer;
struct mpstream;
struct serializer_opts;
diff --git a/src/lua/pickle.c b/src/lua/pickle.c
index 65208b5b3..b88f5ac75 100644
--- a/src/lua/pickle.c
+++ b/src/lua/pickle.c
@@ -38,6 +38,7 @@
#include <lualib.h>
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lua/msgpack.h" /* luaL_msgpack_default */
#include <fiber.h>
#include "bit/bit.h"
diff --git a/src/lua/serializer.c b/src/lua/serializer.c
new file mode 100644
index 000000000..6c3dd73af
--- /dev/null
+++ b/src/lua/serializer.c
@@ -0,0 +1,651 @@
+/*
+ * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include <assert.h>
+#include <stdbool.h>
+#include <math.h> /* modf, isfinite */
+#include <lua.h>
+#include <lauxlib.h>
+
+#include "lua/serializer.h"
+
+#include "trigger.h"
+#include "lib/core/decimal.h" /* decimal_t */
+#include "lib/core/mp_extension_types.h"
+#include "lua/error.h"
+
+#include "trivia/util.h"
+#include "diag.h"
+#include "serializer_opts.h"
+#include "lua/utils.h"
+
+int luaL_map_metatable_ref = LUA_REFNIL;
+int luaL_array_metatable_ref = LUA_REFNIL;
+extern uint32_t CTID_UUID;
+extern uint32_t CTID_DECIMAL;
+
+#define OPTION(type, name, defvalue) { #name, \
+ offsetof(struct luaL_serializer, name), type, defvalue}
+/**
+ * Configuration options for serializers
+ * @sa struct luaL_serializer
+ */
+static struct {
+ const char *name;
+ size_t offset; /* offset in structure */
+ int type;
+ int defvalue;
+} OPTIONS[] = {
+ OPTION(LUA_TBOOLEAN, encode_sparse_convert, 1),
+ OPTION(LUA_TNUMBER, encode_sparse_ratio, 2),
+ OPTION(LUA_TNUMBER, encode_sparse_safe, 10),
+ OPTION(LUA_TNUMBER, encode_max_depth, 128),
+ OPTION(LUA_TBOOLEAN, encode_deep_as_nil, 0),
+ OPTION(LUA_TBOOLEAN, encode_invalid_numbers, 1),
+ OPTION(LUA_TNUMBER, encode_number_precision, 14),
+ OPTION(LUA_TBOOLEAN, encode_load_metatables, 1),
+ OPTION(LUA_TBOOLEAN, encode_use_tostring, 0),
+ OPTION(LUA_TBOOLEAN, encode_invalid_as_nil, 0),
+ OPTION(LUA_TBOOLEAN, decode_invalid_numbers, 1),
+ OPTION(LUA_TBOOLEAN, decode_save_metatables, 1),
+ OPTION(LUA_TNUMBER, decode_max_depth, 128),
+ { NULL, 0, 0, 0},
+};
+
+void
+luaL_serializer_create(struct luaL_serializer *cfg)
+{
+ rlist_create(&cfg->on_update);
+ for (int i = 0; OPTIONS[i].name != NULL; i++) {
+ int *pval = (int *) ((char *) cfg + OPTIONS[i].offset);
+ *pval = OPTIONS[i].defvalue;
+ }
+}
+
+void
+luaL_serializer_copy_options(struct luaL_serializer *dst,
+ const struct luaL_serializer *src)
+{
+ memcpy(dst, src, offsetof(struct luaL_serializer, end_of_options));
+}
+
+/**
+ * Configure one field in @a cfg. Value of the field is kept on
+ * Lua stack after this function, and should be popped manually.
+ * @param L Lua stack.
+ * @param i Index of option in OPTIONS[].
+ * @param cfg Serializer to inherit configuration.
+ * @retval Pointer to the value of option.
+ * @retval NULL if option is not in the table.
+ */
+static int *
+luaL_serializer_parse_option(struct lua_State *L, int i,
+ struct luaL_serializer *cfg)
+{
+ lua_getfield(L, 2, OPTIONS[i].name);
+ if (lua_isnil(L, -1))
+ return NULL;
+ /*
+ * Update struct luaL_serializer using pointer to a
+ * configuration value (all values must be `int` for that).
+ */
+ int *pval = (int *) ((char *) cfg + OPTIONS[i].offset);
+ switch (OPTIONS[i].type) {
+ case LUA_TBOOLEAN:
+ *pval = lua_toboolean(L, -1);
+ break;
+ case LUA_TNUMBER:
+ *pval = lua_tointeger(L, -1);
+ break;
+ default:
+ unreachable();
+ }
+ return pval;
+}
+
+void
+luaL_serializer_parse_options(struct lua_State *L,
+ struct luaL_serializer *cfg)
+{
+ for (int i = 0; OPTIONS[i].name != NULL; ++i) {
+ luaL_serializer_parse_option(L, i, cfg);
+ lua_pop(L, 1);
+ }
+}
+
+/**
+ * @brief serializer.cfg{} Lua binding for serializers.
+ * serializer.cfg is a table that contains current configuration values from
+ * luaL_serializer structure. serializer.cfg has overriden __call() method
+ * to change configuration keys in internal userdata (like box.cfg{}).
+ * Please note that direct change in serializer.cfg.key will not affect
+ * internal state of userdata. Changes via cfg() are reflected in
+ * both Lua cfg table, and C serializer structure.
+ * @param L lua stack
+ * @return 0
+ */
+static int
+luaL_serializer_cfg(struct lua_State *L)
+{
+ /* Serializer.cfg */
+ luaL_checktype(L, 1, LUA_TTABLE);
+ /* Updated parameters. */
+ luaL_checktype(L, 2, LUA_TTABLE);
+ struct luaL_serializer *cfg = luaL_checkserializer(L);
+ for (int i = 0; OPTIONS[i].name != NULL; ++i) {
+ if (luaL_serializer_parse_option(L, i, cfg) == NULL)
+ lua_pop(L, 1);
+ else
+ lua_setfield(L, 1, OPTIONS[i].name);
+ }
+ trigger_run(&cfg->on_update, cfg);
+ return 0;
+}
+
+/**
+ * @brief serializer.new() Lua binding.
+ * @param L stack
+ * @param reg methods to register
+ * @param parent parent serializer to inherit configuration
+ * @return new serializer
+ */
+struct luaL_serializer *
+luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg)
+{
+ luaL_checkstack(L, 1, "too many upvalues");
+
+ /* Create new module */
+ lua_newtable(L);
+
+ /* Create new configuration */
+ struct luaL_serializer *serializer = (struct luaL_serializer *)
+ lua_newuserdata(L, sizeof(*serializer));
+ luaL_getmetatable(L, LUAL_SERIALIZER);
+ lua_setmetatable(L, -2);
+ luaL_serializer_create(serializer);
+
+ for (; reg->name != NULL; reg++) {
+ /* push luaL_serializer as upvalue */
+ lua_pushvalue(L, -1);
+ /* register method */
+ lua_pushcclosure(L, reg->func, 1);
+ lua_setfield(L, -3, reg->name);
+ }
+
+ /* Add cfg{} */
+ lua_newtable(L); /* cfg */
+ lua_newtable(L); /* metatable */
+ lua_pushvalue(L, -3); /* luaL_serializer */
+ lua_pushcclosure(L, luaL_serializer_cfg, 1);
+ lua_setfield(L, -2, "__call");
+ lua_setmetatable(L, -2);
+ /* Save configuration values to serializer.cfg */
+ for (int i = 0; OPTIONS[i].name != NULL; i++) {
+ int *pval = (int *) ((char *) serializer + OPTIONS[i].offset);
+ switch (OPTIONS[i].type) {
+ case LUA_TBOOLEAN:
+ lua_pushboolean(L, *pval);
+ break;
+ case LUA_TNUMBER:
+ lua_pushinteger(L, *pval);
+ break;
+ default:
+ unreachable();
+ }
+ lua_setfield(L, -2, OPTIONS[i].name);
+ }
+ lua_setfield(L, -3, "cfg");
+
+ lua_pop(L, 1); /* remove upvalues */
+
+ luaL_pushnull(L);
+ lua_setfield(L, -2, "NULL");
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_array_metatable_ref);
+ lua_setfield(L, -2, "array_mt");
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_map_metatable_ref);
+ lua_setfield(L, -2, "map_mt");
+
+ if (modname != NULL) {
+ /* Register module */
+ lua_getfield(L, LUA_REGISTRYINDEX, "_LOADED");
+ lua_pushstring(L, modname); /* add alias */
+ lua_pushvalue(L, -3);
+ lua_settable(L, -3);
+ lua_pop(L, 1); /* _LOADED */
+ }
+
+ return serializer;
+}
+
+static int
+lua_gettable_wrapper(lua_State *L)
+{
+ lua_gettable(L, -2);
+ return 1;
+}
+
+static void
+lua_field_inspect_ucdata(struct lua_State *L, struct luaL_serializer *cfg,
+ int idx, struct luaL_field *field)
+{
+ if (!cfg->encode_load_metatables)
+ return;
+
+ /*
+ * Try to call LUAL_SERIALIZE method on udata/cdata
+ * LuaJIT specific: lua_getfield/lua_gettable raises exception on
+ * cdata if field doesn't exist.
+ */
+ int top = lua_gettop(L);
+ lua_pushcfunction(L, lua_gettable_wrapper);
+ lua_pushvalue(L, idx);
+ lua_pushliteral(L, LUAL_SERIALIZE);
+ if (lua_pcall(L, 2, 1, 0) == 0 && !lua_isnil(L, -1)) {
+ if (!lua_isfunction(L, -1))
+ luaL_error(L, "invalid " LUAL_SERIALIZE " value");
+ /* copy object itself */
+ lua_pushvalue(L, idx);
+ lua_pcall(L, 1, 1, 0);
+ /* replace obj with the unpacked value */
+ lua_replace(L, idx);
+ if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
+ luaT_error(L);
+ } /* else ignore lua_gettable exceptions */
+ lua_settop(L, top); /* remove temporary objects */
+}
+
+/**
+ * Call __serialize method of a table object by index
+ * if the former exists.
+ *
+ * If __serialize does not exist then function does nothing
+ * and the function returns 1;
+ *
+ * If __serialize exists, is a function (which doesn't
+ * raise any error) then a result of serialization
+ * replaces old value by the index and the function returns 0;
+ *
+ * If the serialization is a hint string (like 'array' or 'map'),
+ * then field->type, field->size and field->compact
+ * are set if necessary and the function returns 0;
+ *
+ * Otherwise it is an error, set diag and the funciton returns -1;
+ *
+ * Return values:
+ * -1 - error occurs, diag is set, the top of guest stack
+ * is undefined.
+ * 0 - __serialize field is available in the metatable,
+ * the result value is put in the origin slot,
+ * encoding is finished.
+ * 1 - __serialize field is not available in the metatable,
+ * proceed with default table encoding.
+ */
+static int
+lua_field_try_serialize(struct lua_State *L, struct luaL_serializer *cfg,
+ int idx, struct luaL_field *field)
+{
+ if (luaL_getmetafield(L, idx, LUAL_SERIALIZE) == 0)
+ return 1;
+ if (lua_isfunction(L, -1)) {
+ /* copy object itself */
+ lua_pushvalue(L, idx);
+ if (lua_pcall(L, 1, 1, 0) != 0) {
+ diag_set(LuajitError, lua_tostring(L, -1));
+ return -1;
+ }
+ if (luaL_tofield(L, cfg, NULL, -1, field) != 0)
+ return -1;
+ lua_replace(L, idx);
+ return 0;
+ }
+ if (!lua_isstring(L, -1)) {
+ diag_set(LuajitError, "invalid " LUAL_SERIALIZE " value");
+ return -1;
+ }
+ const char *type = lua_tostring(L, -1);
+ if (strcmp(type, "array") == 0 || strcmp(type, "seq") == 0 ||
+ strcmp(type, "sequence") == 0) {
+ field->type = MP_ARRAY; /* Override type */
+ field->size = luaL_arrlen(L, idx);
+ /* YAML: use flow mode if __serialize == 'seq' */
+ if (cfg->has_compact && type[3] == '\0')
+ field->compact = true;
+ } else if (strcmp(type, "map") == 0 || strcmp(type, "mapping") == 0) {
+ field->type = MP_MAP; /* Override type */
+ field->size = luaL_maplen(L, idx);
+ /* YAML: use flow mode if __serialize == 'map' */
+ if (cfg->has_compact && type[3] == '\0')
+ field->compact = true;
+ } else {
+ diag_set(LuajitError, "invalid " LUAL_SERIALIZE " value");
+ return -1;
+ }
+ /* Remove value set by luaL_getmetafield. */
+ lua_pop(L, 1);
+ return 0;
+}
+
+static int
+lua_field_inspect_table(struct lua_State *L, struct luaL_serializer *cfg,
+ int idx, struct luaL_field *field)
+{
+ assert(lua_type(L, idx) == LUA_TTABLE);
+ uint32_t size = 0;
+ uint32_t max = 0;
+
+ if (cfg->encode_load_metatables) {
+ int top = lua_gettop(L);
+ int res = lua_field_try_serialize(L, cfg, idx, field);
+ if (res == -1)
+ return -1;
+ assert(lua_gettop(L) == top);
+ (void)top;
+ if (res == 0)
+ return 0;
+ /* Fallthrough with res == 1 */
+ }
+
+ field->type = MP_ARRAY;
+
+ /* Calculate size and check that table can represent an array */
+ lua_pushnil(L);
+ while (lua_next(L, idx)) {
+ size++;
+ lua_pop(L, 1); /* pop the value */
+ lua_Number k;
+ if (lua_type(L, -1) != LUA_TNUMBER ||
+ ((k = lua_tonumber(L, -1)) != size &&
+ (k < 1 || floor(k) != k))) {
+ /* Finish size calculation */
+ while (lua_next(L, idx)) {
+ size++;
+ lua_pop(L, 1); /* pop the value */
+ }
+ field->type = MP_MAP;
+ field->size = size;
+ return 0;
+ }
+ if (k > max)
+ max = k;
+ }
+
+ /* Encode excessively sparse arrays as objects (if enabled) */
+ if (cfg->encode_sparse_ratio > 0 &&
+ max > size * (uint32_t)cfg->encode_sparse_ratio &&
+ max > (uint32_t)cfg->encode_sparse_safe) {
+ if (!cfg->encode_sparse_convert) {
+ diag_set(LuajitError, "excessively sparse array");
+ return -1;
+ }
+ field->type = MP_MAP;
+ field->size = size;
+ return 0;
+ }
+
+ assert(field->type == MP_ARRAY);
+ field->size = max;
+ return 0;
+}
+
+static void
+lua_field_tostring(struct lua_State *L, struct luaL_serializer *cfg, int idx,
+ struct luaL_field *field)
+{
+ int top = lua_gettop(L);
+ lua_getglobal(L, "tostring");
+ lua_pushvalue(L, idx);
+ lua_call(L, 1, 1);
+ lua_replace(L, idx);
+ lua_settop(L, top);
+ if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
+ luaT_error(L);
+}
+
+int
+luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
+ const struct serializer_opts *opts, int index,
+ struct luaL_field *field)
+{
+ if (index < 0)
+ index = lua_gettop(L) + index + 1;
+
+ double num;
+ double intpart;
+ size_t size;
+
+#define CHECK_NUMBER(x) ({ \
+ if (!isfinite(x) && !cfg->encode_invalid_numbers) { \
+ if (!cfg->encode_invalid_as_nil) { \
+ diag_set(LuajitError, "number must not be NaN or Inf"); \
+ return -1; \
+ } \
+ field->type = MP_NIL; \
+ }})
+
+ switch (lua_type(L, index)) {
+ case LUA_TNUMBER:
+ num = lua_tonumber(L, index);
+ if (isfinite(num) && modf(num, &intpart) != 0.0) {
+ field->type = MP_DOUBLE;
+ field->dval = num;
+ } else if (num >= 0 && num < exp2(64)) {
+ field->type = MP_UINT;
+ field->ival = (uint64_t) num;
+ } else if (num >= -exp2(63) && num < exp2(63)) {
+ field->type = MP_INT;
+ field->ival = (int64_t) num;
+ } else {
+ field->type = MP_DOUBLE;
+ field->dval = num;
+ CHECK_NUMBER(num);
+ }
+ return 0;
+ case LUA_TCDATA:
+ {
+ GCcdata *cd = cdataV(L->base + index - 1);
+ void *cdata = (void *)cdataptr(cd);
+
+ int64_t ival;
+ switch (cd->ctypeid) {
+ case CTID_BOOL:
+ field->type = MP_BOOL;
+ field->bval = *(bool*) cdata;
+ return 0;
+ case CTID_CCHAR:
+ case CTID_INT8:
+ ival = *(int8_t *) cdata;
+ field->type = (ival >= 0) ? MP_UINT : MP_INT;
+ field->ival = ival;
+ return 0;
+ case CTID_INT16:
+ ival = *(int16_t *) cdata;
+ field->type = (ival >= 0) ? MP_UINT : MP_INT;
+ field->ival = ival;
+ return 0;
+ case CTID_INT32:
+ ival = *(int32_t *) cdata;
+ field->type = (ival >= 0) ? MP_UINT : MP_INT;
+ field->ival = ival;
+ return 0;
+ case CTID_INT64:
+ ival = *(int64_t *) cdata;
+ field->type = (ival >= 0) ? MP_UINT : MP_INT;
+ field->ival = ival;
+ return 0;
+ case CTID_UINT8:
+ field->type = MP_UINT;
+ field->ival = *(uint8_t *) cdata;
+ return 0;
+ case CTID_UINT16:
+ field->type = MP_UINT;
+ field->ival = *(uint16_t *) cdata;
+ return 0;
+ case CTID_UINT32:
+ field->type = MP_UINT;
+ field->ival = *(uint32_t *) cdata;
+ return 0;
+ case CTID_UINT64:
+ field->type = MP_UINT;
+ field->ival = *(uint64_t *) cdata;
+ return 0;
+ case CTID_FLOAT:
+ field->type = MP_FLOAT;
+ field->fval = *(float *) cdata;
+ CHECK_NUMBER(field->fval);
+ return 0;
+ case CTID_DOUBLE:
+ field->type = MP_DOUBLE;
+ field->dval = *(double *) cdata;
+ CHECK_NUMBER(field->dval);
+ return 0;
+ case CTID_P_CVOID:
+ case CTID_P_VOID:
+ if (*(void **) cdata == NULL) {
+ field->type = MP_NIL;
+ return 0;
+ }
+ /* Fall through */
+ default:
+ field->type = MP_EXT;
+ if (cd->ctypeid == CTID_DECIMAL) {
+ field->ext_type = MP_DECIMAL;
+ field->decval = (decimal_t *) cdata;
+ } else if (cd->ctypeid == CTID_UUID) {
+ field->ext_type = MP_UUID;
+ field->uuidval = (struct tt_uuid *) cdata;
+ } else if (cd->ctypeid == CTID_CONST_STRUCT_ERROR_REF &&
+ opts != NULL &&
+ opts->error_marshaling_enabled) {
+ field->ext_type = MP_ERROR;
+ } else {
+ field->ext_type = MP_UNKNOWN_EXTENSION;
+ }
+ }
+ return 0;
+ }
+ case LUA_TBOOLEAN:
+ field->type = MP_BOOL;
+ field->bval = lua_toboolean(L, index);
+ return 0;
+ case LUA_TNIL:
+ field->type = MP_NIL;
+ return 0;
+ case LUA_TSTRING:
+ field->sval.data = lua_tolstring(L, index, &size);
+ field->sval.len = (uint32_t) size;
+ field->type = MP_STR;
+ return 0;
+ case LUA_TTABLE:
+ {
+ field->compact = false;
+ return lua_field_inspect_table(L, cfg, index, field);
+ }
+ case LUA_TLIGHTUSERDATA:
+ case LUA_TUSERDATA:
+ field->sval.data = NULL;
+ field->sval.len = 0;
+ if (lua_touserdata(L, index) == NULL) {
+ field->type = MP_NIL;
+ return 0;
+ }
+ /* Fall through */
+ default:
+ field->type = MP_EXT;
+ field->ext_type = MP_UNKNOWN_EXTENSION;
+ }
+#undef CHECK_NUMBER
+ return 0;
+}
+
+void
+luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
+ struct luaL_field *field)
+{
+ if (idx < 0)
+ idx = lua_gettop(L) + idx + 1;
+ assert(field->type == MP_EXT && field->ext_type == MP_UNKNOWN_EXTENSION); /* must be called after tofield() */
+
+ if (cfg->encode_load_metatables) {
+ int type = lua_type(L, idx);
+ if (type == LUA_TCDATA) {
+ /*
+ * Don't call __serialize on primitive types
+ * https://github.com/tarantool/tarantool/issues/1226
+ */
+ GCcdata *cd = cdataV(L->base + idx - 1);
+ if (cd->ctypeid > CTID_CTYPEID)
+ lua_field_inspect_ucdata(L, cfg, idx, field);
+ } else if (type == LUA_TUSERDATA) {
+ lua_field_inspect_ucdata(L, cfg, idx, field);
+ }
+ }
+
+ if (field->type == MP_EXT && field->ext_type == MP_UNKNOWN_EXTENSION &&
+ cfg->encode_use_tostring)
+ lua_field_tostring(L, cfg, idx, field);
+
+ if (field->type != MP_EXT || field->ext_type != MP_UNKNOWN_EXTENSION)
+ return;
+
+ if (cfg->encode_invalid_as_nil) {
+ field->type = MP_NIL;
+ return;
+ }
+
+ luaL_error(L, "unsupported Lua type '%s'",
+ lua_typename(L, lua_type(L, idx)));
+}
+
+int
+tarantool_lua_serializer_init(struct lua_State *L)
+{
+ static const struct luaL_Reg serializermeta[] = {
+ {NULL, NULL},
+ };
+ luaL_register_type(L, LUAL_SERIALIZER, serializermeta);
+
+ lua_createtable(L, 0, 1);
+ lua_pushliteral(L, "map"); /* YAML will use flow mode */
+ lua_setfield(L, -2, LUAL_SERIALIZE);
+ /* automatically reset hints on table change */
+ luaL_loadstring(L, "setmetatable((...), nil); return rawset(...)");
+ lua_setfield(L, -2, "__newindex");
+ luaL_map_metatable_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+
+ lua_createtable(L, 0, 1);
+ lua_pushliteral(L, "seq"); /* YAML will use flow mode */
+ lua_setfield(L, -2, LUAL_SERIALIZE);
+ /* automatically reset hints on table change */
+ luaL_loadstring(L, "setmetatable((...), nil); return rawset(...)");
+ lua_setfield(L, -2, "__newindex");
+ luaL_array_metatable_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+
+ return 0;
+}
diff --git a/src/lua/serializer.h b/src/lua/serializer.h
new file mode 100644
index 000000000..54b0bc11a
--- /dev/null
+++ b/src/lua/serializer.h
@@ -0,0 +1,344 @@
+#ifndef TARANTOOL_LUA_SERIALIZER_H_INCLUDED
+#define TARANTOOL_LUA_SERIALIZER_H_INCLUDED
+/*
+ * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#if defined(__cplusplus)
+extern "C" {
+#endif /* defined(__cplusplus) */
+
+#include <assert.h>
+#include <stdbool.h>
+#include <math.h> /* isfinite */
+#include <lua.h>
+#include <lauxlib.h>
+
+#include "trigger.h"
+#include "lib/core/decimal.h" /* decimal_t */
+#include "lib/core/mp_extension_types.h"
+#include "lua/error.h"
+
+struct serializer_opts;
+struct lua_State;
+struct tt_uuid;
+
+#define LUAL_SERIALIZER "serializer"
+#define LUAL_SERIALIZE "__serialize"
+
+extern int luaL_map_metatable_ref;
+extern int luaL_array_metatable_ref;
+
+/**
+ * Common configuration options for Lua serializers (MsgPack, YAML, JSON)
+ */
+struct luaL_serializer {
+ /**
+ * luaL_tofield tries to classify table into one of four kinds
+ * during encoding:
+ *
+ * + map - at least one table index is not unsigned integer.
+ * + regular array - all array indexes are available.
+ * + sparse array - at least one array index is missing.
+ * + excessively sparse arrat - the number of values missing
+ * exceeds the configured ratio.
+ *
+ * An array is excessively sparse when **all** the following
+ * conditions are met:
+ *
+ * + encode_sparse_ratio > 0.
+ * + max(table) > encode_sparse_safe.
+ * + max(table) > count(table) * encode_sparse_ratio.
+ *
+ * luaL_tofield will never consider an array to be excessively sparse
+ * when encode_sparse_ratio = 0. The encode_sparse_safe limit ensures
+ * that small Lua arrays are always encoded as sparse arrays.
+ * By default, attempting to encode an excessively sparse array will
+ * generate an error. If encode_sparse_convert is set to true,
+ * excessively sparse arrays will be handled as maps.
+ *
+ * This conversion logic is modeled after Mark Pulford's CJSON module.
+ * @sa http://www.kyne.com.au/~mark/software/lua-cjson-manual.html
+ */
+ int encode_sparse_convert;
+ /** @see encode_sparse_convert */
+ int encode_sparse_ratio;
+ /** @see encode_sparse_convert */
+ int encode_sparse_safe;
+ /** Max recursion depth for encoding (MsgPack, CJSON only) */
+ int encode_max_depth;
+ /**
+ * A flag whether a table with too high nest level should
+ * be cropped. The not-encoded fields are replaced with
+ * one null. If not set, too high nesting is considered an
+ * error.
+ */
+ int encode_deep_as_nil;
+ /** Enables encoding of NaN and Inf numbers */
+ int encode_invalid_numbers;
+ /** Floating point numbers precision (YAML, CJSON only) */
+ int encode_number_precision;
+
+ /**
+ * Enables __serialize meta-value checking:
+ *
+ * + 'seq', 'sequence', 'array' - table encoded as an array
+ * + 'map', 'mappping' - table encoded as a map.
+ * 'seq' or 'map' also enable flow (compact) mode for YAML serializer
+ * (flow="[1,2,3]" vs block=" - 1\n - 2\n - 3\n").
+ * + function - the meta-method is called to unpack serializable
+ * representation of table, cdata or userdata objects.
+ */
+ int encode_load_metatables;
+ /** Enables tostring() usage for unknown types */
+ int encode_use_tostring;
+ /** Use NULL for all unrecognizable types */
+ int encode_invalid_as_nil;
+
+ /** Enables decoding NaN and Inf numbers */
+ int decode_invalid_numbers;
+ /** Save __serialize meta-value for decoded arrays and maps */
+ int decode_save_metatables;
+ /** Max recursion depts for decoding (CJSON only) */
+ int decode_max_depth;
+
+ /** Enable support for compact represenation (internal, YAML-only). */
+ int has_compact;
+ /**
+ * Border where copyable fields end. Is used to copy
+ * serializer options into an existing serializer without
+ * erasure of its non-option fields like triggers.
+ */
+ char end_of_options[0];
+ /**
+ * Trigger object to subscribe on updates of a more
+ * general serializer. For example, tuple serializer
+ * subscribes on msgpack.
+ */
+ struct trigger update_trigger;
+ /**
+ * List of triggers on update of this serializer. To push
+ * updates down to dependent serializers.
+ */
+ struct rlist on_update;
+};
+
+struct luaL_serializer *
+luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg);
+
+/**
+ * Copy all option fields of @a src into @a dst. Other fields,
+ * such as triggers, are not touched.
+ */
+void
+luaL_serializer_copy_options(struct luaL_serializer *dst,
+ const struct luaL_serializer *src);
+
+static inline struct luaL_serializer *
+luaL_checkserializer(struct lua_State *L) {
+ return (struct luaL_serializer *)
+ luaL_checkudata(L, lua_upvalueindex(1), LUAL_SERIALIZER);
+}
+
+/**
+ * Initialize serializer with default parameters.
+ * @param cfg Serializer to inherit configuration.
+ */
+void
+luaL_serializer_create(struct luaL_serializer *cfg);
+
+/**
+ * Parse configuration table into @a cfg.
+ * @param L Lua stack.
+ * @param cfg Serializer to inherit configuration.
+ */
+void
+luaL_serializer_parse_options(struct lua_State *l,
+ struct luaL_serializer *cfg);
+
+/** A single value on the Lua stack. */
+struct luaL_field {
+ union {
+ struct {
+ const char *data;
+ uint32_t len;
+ } sval;
+ int64_t ival;
+ double dval;
+ float fval;
+ bool bval;
+ /* Array or map. */
+ uint32_t size;
+ decimal_t *decval;
+ struct tt_uuid *uuidval;
+ };
+ enum mp_type type;
+ /* subtypes of MP_EXT */
+ enum mp_extension_type ext_type;
+ bool compact; /* a flag used by YAML serializer */
+};
+
+/**
+ * @brief Convert a value from the Lua stack to a lua_field structure.
+ * This function is designed for use with Lua bindings and data
+ * serialization functions (YAML, MsgPack, JSON, etc.).
+ *
+ * Conversion rules:
+ * - LUA_TNUMBER when is integer and >= 0 -> UINT
+ * - LUA_TNUMBER when is integer and < 0 -> INT
+ * - LUA_TNUMBER when is not integer -> DOUBLE
+ * - LUA_TBOOLEAN -> BOOL
+ * - LUA_TSTRING -> STRING
+ * - LUA_TNIL -> NIL
+ * - LUA_TTABLE when is array -> ARRAY
+ * - LUA_TTABLE when is not array -> MAP
+ * - LUA_TUSERDATA, LUA_TLIGHTUSERDATA, CTID_P_VOID when == NULL -> NIL
+ * - CTID_INT*, CTID_CCHAR when >= 0 -> UINT
+ * - CTID_INT*, CTID_CCHAR when < 0 -> INT
+ * - CTID_FLOAT -> FLOAT
+ * - CTID_DOUBLE -> DOUBLE
+ * - CTID_BOOL -> BOOL
+ * - otherwise -> EXT
+ *
+ * ARRAY vs MAP recognition works based on encode_sparse_convert,
+ * encode_sparse_ratio, encode_sparse_safe and encode_load_metatables config
+ * parameters (see above). Tables are not saved to lua_field structure and
+ * should be processed manually, according to returned type and size value.
+ *
+ * This function doesn't try to unpack unknown types and simple returns MP_EXT.
+ * The caller can use luaL_tofield() for basic conversion, then invoke internal
+ * hooks(if available) and then call luaL_checkfield(), which will try to
+ * unpack cdata/userdata objects or raise and error.
+ *
+ * @param L stack
+ * @param cfg configuration
+ * @param opts the Lua serializer additional options.
+ * @param index stack index
+ * @param field conversion result
+ *
+ * @retval 0 Success.
+ * @retval -1 Error.
+ */
+int
+luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
+ const struct serializer_opts *opts, int index,
+ struct luaL_field *field);
+
+/**
+ * @brief Try to convert userdata/cdata values using defined conversion logic.
+ * Must be used only after lua_tofield().
+ *
+ * @param L stack
+ * @param cfg configuration
+ * @param idx stack index
+ * @param field conversion result
+ */
+void
+luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
+ struct luaL_field *field);
+
+/**
+ * @brief A wrapper for luaL_tofield() and luaL_convertfield() that
+ * tries to convert value or raise an error.
+ * @param L stack
+ * @param cfg configuration
+ * @param idx stack index
+ * @param field conversion result
+ * @sa lua_tofield()
+ * @sa luaL_convertfield()
+ *
+ * Common conversion order for tables:
+ * size/count detection -> (sparse array checking) -> (__serialize)
+ *
+ * Common conversion order for userdata/cdata objects:
+ * (internal trigger) -> (__serialize) -> (tostring) -> (nil) -> exception
+ *
+ * Common conversion order for other types:
+ * (tostring) -> (nil) -> exception
+ */
+static inline void
+luaL_checkfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
+ struct luaL_field *field)
+{
+ if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
+ luaT_error(L);
+ if (field->type != MP_EXT || field->ext_type != MP_UNKNOWN_EXTENSION)
+ return;
+ luaL_convertfield(L, cfg, idx, field);
+}
+
+/**
+ * Push Lua Table with __serialize = 'map' hint onto the stack.
+ * Tables with __serialize hint are properly handled by all serializers.
+ * @param L stack
+ * @param idx index in the stack
+ */
+static inline void
+luaL_setmaphint(struct lua_State *L, int idx)
+{
+ if (idx < 0)
+ idx = lua_gettop(L) + idx + 1;
+ assert(lua_type(L, idx) == LUA_TTABLE);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_map_metatable_ref);
+ lua_setmetatable(L, idx);
+}
+
+/**
+ * Push Lua Table with __serialize = 'seq' hint onto the stack.
+ * Tables with __serialize hint are properly handled by all serializers.
+ * @param L stack
+ * @param idx index in the stack
+ */
+static inline void
+luaL_setarrayhint(struct lua_State *L, int idx)
+{
+ if (idx < 0)
+ idx = lua_gettop(L) + idx + 1;
+ assert(lua_type(L, idx) == LUA_TTABLE);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_array_metatable_ref);
+ lua_setmetatable(L, idx);
+}
+
+static inline void
+luaL_checkfinite(struct lua_State *L, struct luaL_serializer *cfg,
+ lua_Number number)
+{
+ if (!cfg->decode_invalid_numbers && !isfinite(number))
+ luaL_error(L, "number must not be NaN or Inf");
+}
+
+int
+tarantool_lua_serializer_init(struct lua_State *L);
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
+
+#endif /* TARANTOOL_LUA_SERIALIZER_H_INCLUDED */
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 3ce821374..34cec0eed 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -38,11 +38,7 @@
#include <diag.h>
#include <fiber.h>
-#include "serializer_opts.h"
-
int luaL_nil_ref = LUA_REFNIL;
-int luaL_map_metatable_ref = LUA_REFNIL;
-int luaL_array_metatable_ref = LUA_REFNIL;
static int luaT_newthread_ref = LUA_NOREF;
@@ -50,9 +46,7 @@ static uint32_t CTID_STRUCT_IBUF;
static uint32_t CTID_STRUCT_IBUF_PTR;
static uint32_t CTID_CHAR_PTR;
static uint32_t CTID_CONST_CHAR_PTR;
-static uint32_t CTID_UUID;
-uint32_t CTID_DECIMAL;
-
+uint32_t CTID_UUID;
void *
luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
@@ -239,578 +233,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
}
-#define OPTION(type, name, defvalue) { #name, \
- offsetof(struct luaL_serializer, name), type, defvalue}
-/**
- * Configuration options for serializers
- * @sa struct luaL_serializer
- */
-static struct {
- const char *name;
- size_t offset; /* offset in structure */
- int type;
- int defvalue;
-} OPTIONS[] = {
- OPTION(LUA_TBOOLEAN, encode_sparse_convert, 1),
- OPTION(LUA_TNUMBER, encode_sparse_ratio, 2),
- OPTION(LUA_TNUMBER, encode_sparse_safe, 10),
- OPTION(LUA_TNUMBER, encode_max_depth, 128),
- OPTION(LUA_TBOOLEAN, encode_deep_as_nil, 0),
- OPTION(LUA_TBOOLEAN, encode_invalid_numbers, 1),
- OPTION(LUA_TNUMBER, encode_number_precision, 14),
- OPTION(LUA_TBOOLEAN, encode_load_metatables, 1),
- OPTION(LUA_TBOOLEAN, encode_use_tostring, 0),
- OPTION(LUA_TBOOLEAN, encode_invalid_as_nil, 0),
- OPTION(LUA_TBOOLEAN, decode_invalid_numbers, 1),
- OPTION(LUA_TBOOLEAN, decode_save_metatables, 1),
- OPTION(LUA_TNUMBER, decode_max_depth, 128),
- { NULL, 0, 0, 0},
-};
-
-void
-luaL_serializer_create(struct luaL_serializer *cfg)
-{
- rlist_create(&cfg->on_update);
- for (int i = 0; OPTIONS[i].name != NULL; i++) {
- int *pval = (int *) ((char *) cfg + OPTIONS[i].offset);
- *pval = OPTIONS[i].defvalue;
- }
-}
-
-void
-luaL_serializer_copy_options(struct luaL_serializer *dst,
- const struct luaL_serializer *src)
-{
- memcpy(dst, src, offsetof(struct luaL_serializer, end_of_options));
-}
-
-/**
- * Configure one field in @a cfg. Value of the field is kept on
- * Lua stack after this function, and should be popped manually.
- * @param L Lua stack.
- * @param i Index of option in OPTIONS[].
- * @param cfg Serializer to inherit configuration.
- * @retval Pointer to the value of option.
- * @retval NULL if option is not in the table.
- */
-static int *
-luaL_serializer_parse_option(struct lua_State *L, int i,
- struct luaL_serializer *cfg)
-{
- lua_getfield(L, 2, OPTIONS[i].name);
- if (lua_isnil(L, -1))
- return NULL;
- /*
- * Update struct luaL_serializer using pointer to a
- * configuration value (all values must be `int` for that).
- */
- int *pval = (int *) ((char *) cfg + OPTIONS[i].offset);
- switch (OPTIONS[i].type) {
- case LUA_TBOOLEAN:
- *pval = lua_toboolean(L, -1);
- break;
- case LUA_TNUMBER:
- *pval = lua_tointeger(L, -1);
- break;
- default:
- unreachable();
- }
- return pval;
-}
-
-void
-luaL_serializer_parse_options(struct lua_State *L,
- struct luaL_serializer *cfg)
-{
- for (int i = 0; OPTIONS[i].name != NULL; ++i) {
- luaL_serializer_parse_option(L, i, cfg);
- lua_pop(L, 1);
- }
-}
-
-/**
- * @brief serializer.cfg{} Lua binding for serializers.
- * serializer.cfg is a table that contains current configuration values from
- * luaL_serializer structure. serializer.cfg has overriden __call() method
- * to change configuration keys in internal userdata (like box.cfg{}).
- * Please note that direct change in serializer.cfg.key will not affect
- * internal state of userdata. Changes via cfg() are reflected in
- * both Lua cfg table, and C serializer structure.
- * @param L lua stack
- * @return 0
- */
-static int
-luaL_serializer_cfg(struct lua_State *L)
-{
- /* Serializer.cfg */
- luaL_checktype(L, 1, LUA_TTABLE);
- /* Updated parameters. */
- luaL_checktype(L, 2, LUA_TTABLE);
- struct luaL_serializer *cfg = luaL_checkserializer(L);
- for (int i = 0; OPTIONS[i].name != NULL; ++i) {
- if (luaL_serializer_parse_option(L, i, cfg) == NULL)
- lua_pop(L, 1);
- else
- lua_setfield(L, 1, OPTIONS[i].name);
- }
- trigger_run(&cfg->on_update, cfg);
- return 0;
-}
-
-/**
- * @brief serializer.new() Lua binding.
- * @param L stack
- * @param reg methods to register
- * @param parent parent serializer to inherit configuration
- * @return new serializer
- */
-struct luaL_serializer *
-luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg)
-{
- luaL_checkstack(L, 1, "too many upvalues");
-
- /* Create new module */
- lua_newtable(L);
-
- /* Create new configuration */
- struct luaL_serializer *serializer = (struct luaL_serializer *)
- lua_newuserdata(L, sizeof(*serializer));
- luaL_getmetatable(L, LUAL_SERIALIZER);
- lua_setmetatable(L, -2);
- luaL_serializer_create(serializer);
-
- for (; reg->name != NULL; reg++) {
- /* push luaL_serializer as upvalue */
- lua_pushvalue(L, -1);
- /* register method */
- lua_pushcclosure(L, reg->func, 1);
- lua_setfield(L, -3, reg->name);
- }
-
- /* Add cfg{} */
- lua_newtable(L); /* cfg */
- lua_newtable(L); /* metatable */
- lua_pushvalue(L, -3); /* luaL_serializer */
- lua_pushcclosure(L, luaL_serializer_cfg, 1);
- lua_setfield(L, -2, "__call");
- lua_setmetatable(L, -2);
- /* Save configuration values to serializer.cfg */
- for (int i = 0; OPTIONS[i].name != NULL; i++) {
- int *pval = (int *) ((char *) serializer + OPTIONS[i].offset);
- switch (OPTIONS[i].type) {
- case LUA_TBOOLEAN:
- lua_pushboolean(L, *pval);
- break;
- case LUA_TNUMBER:
- lua_pushinteger(L, *pval);
- break;
- default:
- unreachable();
- }
- lua_setfield(L, -2, OPTIONS[i].name);
- }
- lua_setfield(L, -3, "cfg");
-
- lua_pop(L, 1); /* remove upvalues */
-
- luaL_pushnull(L);
- lua_setfield(L, -2, "NULL");
- lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_array_metatable_ref);
- lua_setfield(L, -2, "array_mt");
- lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_map_metatable_ref);
- lua_setfield(L, -2, "map_mt");
-
- if (modname != NULL) {
- /* Register module */
- lua_getfield(L, LUA_REGISTRYINDEX, "_LOADED");
- lua_pushstring(L, modname); /* add alias */
- lua_pushvalue(L, -3);
- lua_settable(L, -3);
- lua_pop(L, 1); /* _LOADED */
- }
-
- return serializer;
-}
-
-static int
-lua_gettable_wrapper(lua_State *L)
-{
- lua_gettable(L, -2);
- return 1;
-}
-
-static void
-lua_field_inspect_ucdata(struct lua_State *L, struct luaL_serializer *cfg,
- int idx, struct luaL_field *field)
-{
- if (!cfg->encode_load_metatables)
- return;
-
- /*
- * Try to call LUAL_SERIALIZE method on udata/cdata
- * LuaJIT specific: lua_getfield/lua_gettable raises exception on
- * cdata if field doesn't exist.
- */
- int top = lua_gettop(L);
- lua_pushcfunction(L, lua_gettable_wrapper);
- lua_pushvalue(L, idx);
- lua_pushliteral(L, LUAL_SERIALIZE);
- if (lua_pcall(L, 2, 1, 0) == 0 && !lua_isnil(L, -1)) {
- if (!lua_isfunction(L, -1))
- luaL_error(L, "invalid " LUAL_SERIALIZE " value");
- /* copy object itself */
- lua_pushvalue(L, idx);
- lua_pcall(L, 1, 1, 0);
- /* replace obj with the unpacked value */
- lua_replace(L, idx);
- if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
- luaT_error(L);
- } /* else ignore lua_gettable exceptions */
- lua_settop(L, top); /* remove temporary objects */
-}
-
-/**
- * Call __serialize method of a table object by index
- * if the former exists.
- *
- * If __serialize does not exist then function does nothing
- * and the function returns 1;
- *
- * If __serialize exists, is a function (which doesn't
- * raise any error) then a result of serialization
- * replaces old value by the index and the function returns 0;
- *
- * If the serialization is a hint string (like 'array' or 'map'),
- * then field->type, field->size and field->compact
- * are set if necessary and the function returns 0;
- *
- * Otherwise it is an error, set diag and the funciton returns -1;
- *
- * Return values:
- * -1 - error occurs, diag is set, the top of guest stack
- * is undefined.
- * 0 - __serialize field is available in the metatable,
- * the result value is put in the origin slot,
- * encoding is finished.
- * 1 - __serialize field is not available in the metatable,
- * proceed with default table encoding.
- */
-static int
-lua_field_try_serialize(struct lua_State *L, struct luaL_serializer *cfg,
- int idx, struct luaL_field *field)
-{
- if (luaL_getmetafield(L, idx, LUAL_SERIALIZE) == 0)
- return 1;
- if (lua_isfunction(L, -1)) {
- /* copy object itself */
- lua_pushvalue(L, idx);
- if (lua_pcall(L, 1, 1, 0) != 0) {
- diag_set(LuajitError, lua_tostring(L, -1));
- return -1;
- }
- if (luaL_tofield(L, cfg, NULL, -1, field) != 0)
- return -1;
- lua_replace(L, idx);
- return 0;
- }
- if (!lua_isstring(L, -1)) {
- diag_set(LuajitError, "invalid " LUAL_SERIALIZE " value");
- return -1;
- }
- const char *type = lua_tostring(L, -1);
- if (strcmp(type, "array") == 0 || strcmp(type, "seq") == 0 ||
- strcmp(type, "sequence") == 0) {
- field->type = MP_ARRAY; /* Override type */
- field->size = luaL_arrlen(L, idx);
- /* YAML: use flow mode if __serialize == 'seq' */
- if (cfg->has_compact && type[3] == '\0')
- field->compact = true;
- } else if (strcmp(type, "map") == 0 || strcmp(type, "mapping") == 0) {
- field->type = MP_MAP; /* Override type */
- field->size = luaL_maplen(L, idx);
- /* YAML: use flow mode if __serialize == 'map' */
- if (cfg->has_compact && type[3] == '\0')
- field->compact = true;
- } else {
- diag_set(LuajitError, "invalid " LUAL_SERIALIZE " value");
- return -1;
- }
- /* Remove value set by luaL_getmetafield. */
- lua_pop(L, 1);
- return 0;
-}
-
-static int
-lua_field_inspect_table(struct lua_State *L, struct luaL_serializer *cfg,
- int idx, struct luaL_field *field)
-{
- assert(lua_type(L, idx) == LUA_TTABLE);
- uint32_t size = 0;
- uint32_t max = 0;
-
- if (cfg->encode_load_metatables) {
- int top = lua_gettop(L);
- int res = lua_field_try_serialize(L, cfg, idx, field);
- if (res == -1)
- return -1;
- assert(lua_gettop(L) == top);
- (void)top;
- if (res == 0)
- return 0;
- /* Fallthrough with res == 1 */
- }
-
- field->type = MP_ARRAY;
-
- /* Calculate size and check that table can represent an array */
- lua_pushnil(L);
- while (lua_next(L, idx)) {
- size++;
- lua_pop(L, 1); /* pop the value */
- lua_Number k;
- if (lua_type(L, -1) != LUA_TNUMBER ||
- ((k = lua_tonumber(L, -1)) != size &&
- (k < 1 || floor(k) != k))) {
- /* Finish size calculation */
- while (lua_next(L, idx)) {
- size++;
- lua_pop(L, 1); /* pop the value */
- }
- field->type = MP_MAP;
- field->size = size;
- return 0;
- }
- if (k > max)
- max = k;
- }
-
- /* Encode excessively sparse arrays as objects (if enabled) */
- if (cfg->encode_sparse_ratio > 0 &&
- max > size * (uint32_t)cfg->encode_sparse_ratio &&
- max > (uint32_t)cfg->encode_sparse_safe) {
- if (!cfg->encode_sparse_convert) {
- diag_set(LuajitError, "excessively sparse array");
- return -1;
- }
- field->type = MP_MAP;
- field->size = size;
- return 0;
- }
-
- assert(field->type == MP_ARRAY);
- field->size = max;
- return 0;
-}
-
-static void
-lua_field_tostring(struct lua_State *L, struct luaL_serializer *cfg, int idx,
- struct luaL_field *field)
-{
- int top = lua_gettop(L);
- lua_getglobal(L, "tostring");
- lua_pushvalue(L, idx);
- lua_call(L, 1, 1);
- lua_replace(L, idx);
- lua_settop(L, top);
- if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
- luaT_error(L);
-}
-
-int
-luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
- const struct serializer_opts *opts, int index,
- struct luaL_field *field)
-{
- if (index < 0)
- index = lua_gettop(L) + index + 1;
-
- double num;
- double intpart;
- size_t size;
-
-#define CHECK_NUMBER(x) ({ \
- if (!isfinite(x) && !cfg->encode_invalid_numbers) { \
- if (!cfg->encode_invalid_as_nil) { \
- diag_set(LuajitError, "number must not be NaN or Inf"); \
- return -1; \
- } \
- field->type = MP_NIL; \
- }})
-
- switch (lua_type(L, index)) {
- case LUA_TNUMBER:
- num = lua_tonumber(L, index);
- if (isfinite(num) && modf(num, &intpart) != 0.0) {
- field->type = MP_DOUBLE;
- field->dval = num;
- } else if (num >= 0 && num < exp2(64)) {
- field->type = MP_UINT;
- field->ival = (uint64_t) num;
- } else if (num >= -exp2(63) && num < exp2(63)) {
- field->type = MP_INT;
- field->ival = (int64_t) num;
- } else {
- field->type = MP_DOUBLE;
- field->dval = num;
- CHECK_NUMBER(num);
- }
- return 0;
- case LUA_TCDATA:
- {
- GCcdata *cd = cdataV(L->base + index - 1);
- void *cdata = (void *)cdataptr(cd);
-
- int64_t ival;
- switch (cd->ctypeid) {
- case CTID_BOOL:
- field->type = MP_BOOL;
- field->bval = *(bool*) cdata;
- return 0;
- case CTID_CCHAR:
- case CTID_INT8:
- ival = *(int8_t *) cdata;
- field->type = (ival >= 0) ? MP_UINT : MP_INT;
- field->ival = ival;
- return 0;
- case CTID_INT16:
- ival = *(int16_t *) cdata;
- field->type = (ival >= 0) ? MP_UINT : MP_INT;
- field->ival = ival;
- return 0;
- case CTID_INT32:
- ival = *(int32_t *) cdata;
- field->type = (ival >= 0) ? MP_UINT : MP_INT;
- field->ival = ival;
- return 0;
- case CTID_INT64:
- ival = *(int64_t *) cdata;
- field->type = (ival >= 0) ? MP_UINT : MP_INT;
- field->ival = ival;
- return 0;
- case CTID_UINT8:
- field->type = MP_UINT;
- field->ival = *(uint8_t *) cdata;
- return 0;
- case CTID_UINT16:
- field->type = MP_UINT;
- field->ival = *(uint16_t *) cdata;
- return 0;
- case CTID_UINT32:
- field->type = MP_UINT;
- field->ival = *(uint32_t *) cdata;
- return 0;
- case CTID_UINT64:
- field->type = MP_UINT;
- field->ival = *(uint64_t *) cdata;
- return 0;
- case CTID_FLOAT:
- field->type = MP_FLOAT;
- field->fval = *(float *) cdata;
- CHECK_NUMBER(field->fval);
- return 0;
- case CTID_DOUBLE:
- field->type = MP_DOUBLE;
- field->dval = *(double *) cdata;
- CHECK_NUMBER(field->dval);
- return 0;
- case CTID_P_CVOID:
- case CTID_P_VOID:
- if (*(void **) cdata == NULL) {
- field->type = MP_NIL;
- return 0;
- }
- /* Fall through */
- default:
- field->type = MP_EXT;
- if (cd->ctypeid == CTID_DECIMAL) {
- field->ext_type = MP_DECIMAL;
- field->decval = (decimal_t *) cdata;
- } else if (cd->ctypeid == CTID_UUID) {
- field->ext_type = MP_UUID;
- field->uuidval = (struct tt_uuid *) cdata;
- } else if (cd->ctypeid == CTID_CONST_STRUCT_ERROR_REF &&
- opts != NULL &&
- opts->error_marshaling_enabled) {
- field->ext_type = MP_ERROR;
- } else {
- field->ext_type = MP_UNKNOWN_EXTENSION;
- }
- }
- return 0;
- }
- case LUA_TBOOLEAN:
- field->type = MP_BOOL;
- field->bval = lua_toboolean(L, index);
- return 0;
- case LUA_TNIL:
- field->type = MP_NIL;
- return 0;
- case LUA_TSTRING:
- field->sval.data = lua_tolstring(L, index, &size);
- field->sval.len = (uint32_t) size;
- field->type = MP_STR;
- return 0;
- case LUA_TTABLE:
- {
- field->compact = false;
- return lua_field_inspect_table(L, cfg, index, field);
- }
- case LUA_TLIGHTUSERDATA:
- case LUA_TUSERDATA:
- field->sval.data = NULL;
- field->sval.len = 0;
- if (lua_touserdata(L, index) == NULL) {
- field->type = MP_NIL;
- return 0;
- }
- /* Fall through */
- default:
- field->type = MP_EXT;
- field->ext_type = MP_UNKNOWN_EXTENSION;
- }
-#undef CHECK_NUMBER
- return 0;
-}
-
-void
-luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
- struct luaL_field *field)
-{
- if (idx < 0)
- idx = lua_gettop(L) + idx + 1;
- assert(field->type == MP_EXT && field->ext_type == MP_UNKNOWN_EXTENSION); /* must be called after tofield() */
-
- if (cfg->encode_load_metatables) {
- int type = lua_type(L, idx);
- if (type == LUA_TCDATA) {
- /*
- * Don't call __serialize on primitive types
- * https://github.com/tarantool/tarantool/issues/1226
- */
- GCcdata *cd = cdataV(L->base + idx - 1);
- if (cd->ctypeid > CTID_CTYPEID)
- lua_field_inspect_ucdata(L, cfg, idx, field);
- } else if (type == LUA_TUSERDATA) {
- lua_field_inspect_ucdata(L, cfg, idx, field);
- }
- }
-
- if (field->type == MP_EXT && field->ext_type == MP_UNKNOWN_EXTENSION &&
- cfg->encode_use_tostring)
- lua_field_tostring(L, cfg, idx, field);
-
- if (field->type != MP_EXT || field->ext_type != MP_UNKNOWN_EXTENSION)
- return;
-
- if (cfg->encode_invalid_as_nil) {
- field->type = MP_NIL;
- return;
- }
-
- luaL_error(L, "unsupported Lua type '%s'",
- lua_typename(L, lua_type(L, idx)));
-}
-
/**
* A helper to register a single type metatable.
*/
@@ -1263,31 +685,10 @@ luaT_newthread(struct lua_State *L)
int
tarantool_lua_utils_init(struct lua_State *L)
{
- static const struct luaL_Reg serializermeta[] = {
- {NULL, NULL},
- };
-
- luaL_register_type(L, LUAL_SERIALIZER, serializermeta);
/* Create NULL constant */
*(void **) luaL_pushcdata(L, CTID_P_VOID) = NULL;
luaL_nil_ref = luaL_ref(L, LUA_REGISTRYINDEX);
- lua_createtable(L, 0, 1);
- lua_pushliteral(L, "map"); /* YAML will use flow mode */
- lua_setfield(L, -2, LUAL_SERIALIZE);
- /* automatically reset hints on table change */
- luaL_loadstring(L, "setmetatable((...), nil); return rawset(...)");
- lua_setfield(L, -2, "__newindex");
- luaL_map_metatable_ref = luaL_ref(L, LUA_REGISTRYINDEX);
-
- lua_createtable(L, 0, 1);
- lua_pushliteral(L, "seq"); /* YAML will use flow mode */
- lua_setfield(L, -2, LUAL_SERIALIZE);
- /* automatically reset hints on table change */
- luaL_loadstring(L, "setmetatable((...), nil); return rawset(...)");
- lua_setfield(L, -2, "__newindex");
- luaL_array_metatable_ref = luaL_ref(L, LUA_REGISTRYINDEX);
-
int rc = luaL_cdef(L, "struct ibuf;");
assert(rc == 0);
CTID_STRUCT_IBUF = luaL_ctypeid(L, "struct ibuf");
diff --git a/src/lua/utils.h b/src/lua/utils.h
index 4a164868b..947d9240b 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -33,10 +33,9 @@
#include <stdint.h>
#include <string.h>
-#include <math.h> /* modf, isfinite */
+#include <math.h> /* floor */
#include <msgpuck.h> /* enum mp_type */
-#include "trigger.h"
#if defined(__cplusplus)
extern "C" {
@@ -56,10 +55,6 @@ extern "C" {
#include "lua/error.h"
-#include "lib/core/mp_extension_types.h"
-#include "lib/core/decimal.h" /* decimal_t */
-
-struct serializer_opts;
struct lua_State;
struct ibuf;
typedef struct ibuf box_ibuf_t;
@@ -73,6 +68,8 @@ struct tt_uuid;
*/
extern struct lua_State *tarantool_L;
+extern uint32_t CTID_UUID;
+
struct tt_uuid *
luaL_pushuuid(struct lua_State *L);
@@ -197,250 +194,7 @@ luaL_maplen(struct lua_State *L, int idx)
return size;
}
-/**
- * Common configuration options for Lua serializers (MsgPack, YAML, JSON)
- */
-struct luaL_serializer {
- /**
- * luaL_tofield tries to classify table into one of four kinds
- * during encoding:
- *
- * + map - at least one table index is not unsigned integer.
- * + regular array - all array indexes are available.
- * + sparse array - at least one array index is missing.
- * + excessively sparse arrat - the number of values missing
- * exceeds the configured ratio.
- *
- * An array is excessively sparse when **all** the following
- * conditions are met:
- *
- * + encode_sparse_ratio > 0.
- * + max(table) > encode_sparse_safe.
- * + max(table) > count(table) * encode_sparse_ratio.
- *
- * luaL_tofield will never consider an array to be excessively sparse
- * when encode_sparse_ratio = 0. The encode_sparse_safe limit ensures
- * that small Lua arrays are always encoded as sparse arrays.
- * By default, attempting to encode an excessively sparse array will
- * generate an error. If encode_sparse_convert is set to true,
- * excessively sparse arrays will be handled as maps.
- *
- * This conversion logic is modeled after Mark Pulford's CJSON module.
- * @sa http://www.kyne.com.au/~mark/software/lua-cjson-manual.html
- */
- int encode_sparse_convert;
- /** @see encode_sparse_convert */
- int encode_sparse_ratio;
- /** @see encode_sparse_convert */
- int encode_sparse_safe;
- /** Max recursion depth for encoding (MsgPack, CJSON only) */
- int encode_max_depth;
- /**
- * A flag whether a table with too high nest level should
- * be cropped. The not-encoded fields are replaced with
- * one null. If not set, too high nesting is considered an
- * error.
- */
- int encode_deep_as_nil;
- /** Enables encoding of NaN and Inf numbers */
- int encode_invalid_numbers;
- /** Floating point numbers precision (YAML, CJSON only) */
- int encode_number_precision;
-
- /**
- * Enables __serialize meta-value checking:
- *
- * + 'seq', 'sequence', 'array' - table encoded as an array
- * + 'map', 'mappping' - table encoded as a map.
- * 'seq' or 'map' also enable flow (compact) mode for YAML serializer
- * (flow="[1,2,3]" vs block=" - 1\n - 2\n - 3\n").
- * + function - the meta-method is called to unpack serializable
- * representation of table, cdata or userdata objects.
- */
- int encode_load_metatables;
- /** Enables tostring() usage for unknown types */
- int encode_use_tostring;
- /** Use NULL for all unrecognizable types */
- int encode_invalid_as_nil;
-
- /** Enables decoding NaN and Inf numbers */
- int decode_invalid_numbers;
- /** Save __serialize meta-value for decoded arrays and maps */
- int decode_save_metatables;
- /** Max recursion depts for decoding (CJSON only) */
- int decode_max_depth;
-
- /** Enable support for compact represenation (internal, YAML-only). */
- int has_compact;
- /**
- * Border where copyable fields end. Is used to copy
- * serializer options into an existing serializer without
- * erasure of its non-option fields like triggers.
- */
- char end_of_options[0];
- /**
- * Trigger object to subscribe on updates of a more
- * general serializer. For example, tuple serializer
- * subscribes on msgpack.
- */
- struct trigger update_trigger;
- /**
- * List of triggers on update of this serializer. To push
- * updates down to dependent serializers.
- */
- struct rlist on_update;
-};
-
extern int luaL_nil_ref;
-extern int luaL_map_metatable_ref;
-extern int luaL_array_metatable_ref;
-
-#define LUAL_SERIALIZER "serializer"
-#define LUAL_SERIALIZE "__serialize"
-
-struct luaL_serializer *
-luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg);
-
-/**
- * Copy all option fields of @a src into @a dst. Other fields,
- * such as triggers, are not touched.
- */
-void
-luaL_serializer_copy_options(struct luaL_serializer *dst,
- const struct luaL_serializer *src);
-
-static inline struct luaL_serializer *
-luaL_checkserializer(struct lua_State *L) {
- return (struct luaL_serializer *)
- luaL_checkudata(L, lua_upvalueindex(1), LUAL_SERIALIZER);
-}
-
-/**
- * Initialize serializer with default parameters.
- * @param cfg Serializer to inherit configuration.
- */
-void
-luaL_serializer_create(struct luaL_serializer *cfg);
-
-/**
- * Parse configuration table into @a cfg.
- * @param L Lua stack.
- * @param cfg Serializer to inherit configuration.
- */
-void
-luaL_serializer_parse_options(struct lua_State *l,
- struct luaL_serializer *cfg);
-
-/** A single value on the Lua stack. */
-struct luaL_field {
- union {
- struct {
- const char *data;
- uint32_t len;
- } sval;
- int64_t ival;
- double dval;
- float fval;
- bool bval;
- /* Array or map. */
- uint32_t size;
- decimal_t *decval;
- struct tt_uuid *uuidval;
- };
- enum mp_type type;
- /* subtypes of MP_EXT */
- enum mp_extension_type ext_type;
- bool compact; /* a flag used by YAML serializer */
-};
-
-/**
- * @brief Convert a value from the Lua stack to a lua_field structure.
- * This function is designed for use with Lua bindings and data
- * serialization functions (YAML, MsgPack, JSON, etc.).
- *
- * Conversion rules:
- * - LUA_TNUMBER when is integer and >= 0 -> UINT
- * - LUA_TNUMBER when is integer and < 0 -> INT
- * - LUA_TNUMBER when is not integer -> DOUBLE
- * - LUA_TBOOLEAN -> BOOL
- * - LUA_TSTRING -> STRING
- * - LUA_TNIL -> NIL
- * - LUA_TTABLE when is array -> ARRAY
- * - LUA_TTABLE when is not array -> MAP
- * - LUA_TUSERDATA, LUA_TLIGHTUSERDATA, CTID_P_VOID when == NULL -> NIL
- * - CTID_INT*, CTID_CCHAR when >= 0 -> UINT
- * - CTID_INT*, CTID_CCHAR when < 0 -> INT
- * - CTID_FLOAT -> FLOAT
- * - CTID_DOUBLE -> DOUBLE
- * - CTID_BOOL -> BOOL
- * - otherwise -> EXT
- *
- * ARRAY vs MAP recognition works based on encode_sparse_convert,
- * encode_sparse_ratio, encode_sparse_safe and encode_load_metatables config
- * parameters (see above). Tables are not saved to lua_field structure and
- * should be processed manually, according to returned type and size value.
- *
- * This function doesn't try to unpack unknown types and simple returns MP_EXT.
- * The caller can use luaL_tofield() for basic conversion, then invoke internal
- * hooks(if available) and then call luaL_checkfield(), which will try to
- * unpack cdata/userdata objects or raise and error.
- *
- * @param L stack
- * @param cfg configuration
- * @param opts the Lua serializer additional options.
- * @param index stack index
- * @param field conversion result
- *
- * @retval 0 Success.
- * @retval -1 Error.
- */
-int
-luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
- const struct serializer_opts *opts, int index,
- struct luaL_field *field);
-
-/**
- * @brief Try to convert userdata/cdata values using defined conversion logic.
- * Must be used only after lua_tofield().
- *
- * @param L stack
- * @param cfg configuration
- * @param idx stack index
- * @param field conversion result
- */
-void
-luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
- struct luaL_field *field);
-
-/**
- * @brief A wrapper for luaL_tofield() and luaL_convertfield() that
- * tries to convert value or raise an error.
- * @param L stack
- * @param cfg configuration
- * @param idx stack index
- * @param field conversion result
- * @sa lua_tofield()
- * @sa luaL_convertfield()
- *
- * Common conversion order for tables:
- * size/count detection -> (sparse array checking) -> (__serialize)
- *
- * Common conversion order for userdata/cdata objects:
- * (internal trigger) -> (__serialize) -> (tostring) -> (nil) -> exception
- *
- * Common conversion order for other types:
- * (tostring) -> (nil) -> exception
- */
-static inline void
-luaL_checkfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
- struct luaL_field *field)
-{
- if (luaL_tofield(L, cfg, NULL, idx, field) < 0)
- luaT_error(L);
- if (field->type != MP_EXT || field->ext_type != MP_UNKNOWN_EXTENSION)
- return;
- luaL_convertfield(L, cfg, idx, field);
-}
void
luaL_register_type(struct lua_State *L, const char *type_name,
@@ -556,38 +310,6 @@ luaT_toibuf(struct lua_State *L, int idx);
int
luaT_toerror(lua_State *L);
-/**
- * Push Lua Table with __serialize = 'map' hint onto the stack.
- * Tables with __serialize hint are properly handled by all serializers.
- * @param L stack
- * @param idx index in the stack
- */
-static inline void
-luaL_setmaphint(struct lua_State *L, int idx)
-{
- if (idx < 0)
- idx = lua_gettop(L) + idx + 1;
- assert(lua_type(L, idx) == LUA_TTABLE);
- lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_map_metatable_ref);
- lua_setmetatable(L, idx);
-}
-
-/**
- * Push Lua Table with __serialize = 'seq' hint onto the stack.
- * Tables with __serialize hint are properly handled by all serializers.
- * @param L stack
- * @param idx index in the stack
- */
-static inline void
-luaL_setarrayhint(struct lua_State *L, int idx)
-{
- if (idx < 0)
- idx = lua_gettop(L) + idx + 1;
- assert(lua_type(L, idx) == LUA_TTABLE);
- lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_array_metatable_ref);
- lua_setmetatable(L, idx);
-}
-
/**
* Push ffi's NULL (cdata<void *>: NULL) onto the stack.
* Can be used as replacement of nil in Lua tables.
@@ -616,14 +338,6 @@ luaL_isnull(struct lua_State *L, int idx)
return false;
}
-static inline void
-luaL_checkfinite(struct lua_State *L, struct luaL_serializer *cfg,
- lua_Number number)
-{
- if (!cfg->decode_invalid_numbers && !isfinite(number))
- luaL_error(L, "number must not be NaN or Inf");
-}
-
/**
* @brief Creates a new Lua coroutine in a protected frame. If
* <lua_newthread> call underneath succeeds, the created Lua state
diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c
index dbd86cbab..5123b9a74 100644
--- a/third_party/lua-cjson/lua_cjson.c
+++ b/third_party/lua-cjson/lua_cjson.c
@@ -48,6 +48,7 @@
#include "strbuf.h"
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
#include "tt_static.h"
#include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */
diff --git a/third_party/lua-yaml/lyaml.cc b/third_party/lua-yaml/lyaml.cc
index 9c3a4a646..5469e9f4f 100644
--- a/third_party/lua-yaml/lyaml.cc
+++ b/third_party/lua-yaml/lyaml.cc
@@ -49,6 +49,7 @@ extern "C" {
#include "b64.h"
} /* extern "C" */
#include "lua/utils.h"
+#include "lua/serializer.h"
#include "lib/core/decimal.h"
#include "tt_static.h"
#include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
--
2.31.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Tarantool-patches] [PATCH 2/4] lua: move luaL_newserializer() comment into header
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file Alexander Turenko via Tarantool-patches
@ 2021-06-23 19:12 ` Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 3/4] lua: split serializer functions into sections Alexander Turenko via Tarantool-patches
` (5 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-06-23 19:12 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov
Cc: Alexander Turenko, tarantool-patches
It is non-static function, so it looks logical to have the API comment
in the header.
Made a few code style fixes, while I'm here.
Part of #3228
---
src/lua/serializer.c | 10 ++--------
src/lua/serializer.h | 13 +++++++++++--
2 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/src/lua/serializer.c b/src/lua/serializer.c
index 6c3dd73af..38b7dd8b9 100644
--- a/src/lua/serializer.c
+++ b/src/lua/serializer.c
@@ -169,15 +169,9 @@ luaL_serializer_cfg(struct lua_State *L)
return 0;
}
-/**
- * @brief serializer.new() Lua binding.
- * @param L stack
- * @param reg methods to register
- * @param parent parent serializer to inherit configuration
- * @return new serializer
- */
struct luaL_serializer *
-luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg)
+luaL_newserializer(struct lua_State *L, const char *modname,
+ const luaL_Reg *reg)
{
luaL_checkstack(L, 1, "too many upvalues");
diff --git a/src/lua/serializer.h b/src/lua/serializer.h
index 54b0bc11a..e29d802bb 100644
--- a/src/lua/serializer.h
+++ b/src/lua/serializer.h
@@ -150,8 +150,16 @@ struct luaL_serializer {
struct rlist on_update;
};
+/**
+ * @brief serializer.new() Lua binding.
+ * @param L stack
+ * @param reg methods to register
+ * @param parent parent serializer to inherit configuration
+ * @return new serializer
+ */
struct luaL_serializer *
-luaL_newserializer(struct lua_State *L, const char *modname, const luaL_Reg *reg);
+luaL_newserializer(struct lua_State *L, const char *modname,
+ const luaL_Reg *reg);
/**
* Copy all option fields of @a src into @a dst. Other fields,
@@ -162,7 +170,8 @@ luaL_serializer_copy_options(struct luaL_serializer *dst,
const struct luaL_serializer *src);
static inline struct luaL_serializer *
-luaL_checkserializer(struct lua_State *L) {
+luaL_checkserializer(struct lua_State *L)
+{
return (struct luaL_serializer *)
luaL_checkudata(L, lua_upvalueindex(1), LUAL_SERIALIZER);
}
--
2.31.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Tarantool-patches] [PATCH 3/4] lua: split serializer functions into sections
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 2/4] lua: move luaL_newserializer() comment into header Alexander Turenko via Tarantool-patches
@ 2021-06-23 19:12 ` Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 4/4] test: add a basic unit test for serializer helpers Alexander Turenko via Tarantool-patches
` (4 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-06-23 19:12 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov
Cc: Alexander Turenko, tarantool-patches
And added a general comment about the compilation unit. Everything is to
simplify reading.
Part of #3228
---
src/lua/serializer.c | 8 ++++++++
src/lua/serializer.h | 23 +++++++++++++++++++++++
2 files changed, 31 insertions(+)
diff --git a/src/lua/serializer.c b/src/lua/serializer.c
index 38b7dd8b9..8db6746a3 100644
--- a/src/lua/serializer.c
+++ b/src/lua/serializer.c
@@ -51,6 +51,8 @@ int luaL_array_metatable_ref = LUA_REFNIL;
extern uint32_t CTID_UUID;
extern uint32_t CTID_DECIMAL;
+/* {{{ luaL_serializer manipulations */
+
#define OPTION(type, name, defvalue) { #name, \
offsetof(struct luaL_serializer, name), type, defvalue}
/**
@@ -238,6 +240,10 @@ luaL_newserializer(struct lua_State *L, const char *modname,
return serializer;
}
+/* }}} luaL_serializer manipulations */
+
+/* {{{ Fill luaL_field */
+
static int
lua_gettable_wrapper(lua_State *L)
{
@@ -617,6 +623,8 @@ luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
lua_typename(L, lua_type(L, idx)));
}
+/* }}} Fill luaL_field */
+
int
tarantool_lua_serializer_init(struct lua_State *L)
{
diff --git a/src/lua/serializer.h b/src/lua/serializer.h
index e29d802bb..0c473c8da 100644
--- a/src/lua/serializer.h
+++ b/src/lua/serializer.h
@@ -31,6 +31,17 @@
* SUCH DAMAGE.
*/
+/**
+ * A set of helpers for converting Lua values into another
+ * representation.
+ *
+ * <struct luaL_serializer> is the serializer object: options and
+ * options inheritance.
+ *
+ * <struct luaL_field> is a Lua value descriptor, which
+ * characterizes the value.
+ */
+
#if defined(__cplusplus)
extern "C" {
#endif /* defined(__cplusplus) */
@@ -56,6 +67,8 @@ struct tt_uuid;
extern int luaL_map_metatable_ref;
extern int luaL_array_metatable_ref;
+/* {{{ luaL_serializer manipulations */
+
/**
* Common configuration options for Lua serializers (MsgPack, YAML, JSON)
*/
@@ -192,6 +205,10 @@ void
luaL_serializer_parse_options(struct lua_State *l,
struct luaL_serializer *cfg);
+/* }}} luaL_serializer manipulations */
+
+/* {{{ Fill luaL_field */
+
/** A single value on the Lua stack. */
struct luaL_field {
union {
@@ -303,6 +320,10 @@ luaL_checkfield(struct lua_State *L, struct luaL_serializer *cfg, int idx,
luaL_convertfield(L, cfg, idx, field);
}
+/* }}} Fill luaL_field */
+
+/* {{{ Set map / array hint */
+
/**
* Push Lua Table with __serialize = 'map' hint onto the stack.
* Tables with __serialize hint are properly handled by all serializers.
@@ -335,6 +356,8 @@ luaL_setarrayhint(struct lua_State *L, int idx)
lua_setmetatable(L, idx);
}
+/* }}} Set map / array hint */
+
static inline void
luaL_checkfinite(struct lua_State *L, struct luaL_serializer *cfg,
lua_Number number)
--
2.31.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Tarantool-patches] [PATCH 4/4] test: add a basic unit test for serializer helpers
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
` (2 preceding siblings ...)
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 3/4] lua: split serializer functions into sections Alexander Turenko via Tarantool-patches
@ 2021-06-23 19:12 ` Alexander Turenko via Tarantool-patches
2021-06-24 6:17 ` [Tarantool-patches] [PATCH 0/4] RFC: Isolate " Cyrill Gorcunov via Tarantool-patches
` (3 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-06-23 19:12 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov
Cc: Alexander Turenko, tarantool-patches
The main reason of writting this test is to learn how those serializer
helpers work. It also may be a good base for future test cases.
Part of #3228
---
test/unit/CMakeLists.txt | 3 +
test/unit/serializer.c | 444 ++++++++++++++++++++++++++++++++++++
test/unit/serializer.result | 135 +++++++++++
3 files changed, 582 insertions(+)
create mode 100644 test/unit/serializer.c
create mode 100644 test/unit/serializer.result
diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
index 8df77bac3..5bb7cd6e7 100644
--- a/test/unit/CMakeLists.txt
+++ b/test/unit/CMakeLists.txt
@@ -275,3 +275,6 @@ add_executable(popen-child popen-child.c)
add_executable(popen.test popen.c core_test_utils.c)
target_link_libraries(popen.test misc unit core)
+
+add_executable(serializer.test serializer.c)
+target_link_libraries(serializer.test unit box ${LUAJIT_LIBRARIES})
diff --git a/test/unit/serializer.c b/test/unit/serializer.c
new file mode 100644
index 000000000..bca239fa8
--- /dev/null
+++ b/test/unit/serializer.c
@@ -0,0 +1,444 @@
+#include <lua.h> /* lua_*() */
+#include <lauxlib.h> /* luaL_*() */
+#include <lualib.h> /* luaL_openlibs() */
+#include "unit.h" /* plan, header, footer, is, ok */
+#include "lua/serializer.h" /* functions to test */
+#include "mp_extension_types.h" /* enum mp_extension_type */
+
+static int
+check_luaL_field(const struct luaL_field *field, const struct luaL_field *exp,
+ const char *description)
+{
+ plan(4);
+ header();
+
+ is(field->type, exp->type, "%s: type", description);
+
+ /* More types may be added on demand. */
+ switch (exp->type) {
+ case MP_STR: {
+ /*
+ * Don't compare string values for equality: check
+ * whether actual result contains the expected
+ * pattern at beginning. It is just to simplify
+ * writting of test cases.
+ */
+ int rc = strstr(field->sval.data, exp->sval.data) ==
+ field->sval.data;
+ ok(rc, "%s: sval.data", description);
+ /* Don't compare 'sval.len'. */
+ ok(true, "# skip; %s: Don't compare 'exp_type'", description);
+ ok(true, "# skip; %s: Don't compare 'compact'", description);
+ break;
+ }
+ case MP_ARRAY:
+ case MP_MAP:
+ is(field->size, exp->size, "%s: size", description);
+ ok(true, "# skip; %s: Don't compare 'exp_type'", description);
+ is(field->compact, exp->compact, "%s: compact", description);
+ break;
+ case MP_EXT:
+ ok(true, "# skip; %s: Don't check MP_EXT data", description);
+ is(field->ext_type, exp->ext_type, "%s: ext_type", description);
+ ok(true, "# skip; %s: Don't compare 'compact'", description);
+ break;
+ default:
+ assert(false);
+ }
+
+ footer();
+ return check_plan();
+}
+
+static int
+test_luaL_field_basic(struct lua_State *L)
+{
+ struct {
+ /* A string to output with a test case. */
+ const char *description;
+ /* A code to push a Lua object to inspect. */
+ const char *src;
+ /*
+ * Whether to call and verify luaL_checkfield()
+ * instead of luaL_tofield() (the default is
+ * luaL_tofield()).
+ */
+ bool invoke_checkfield;
+ /*
+ * Expected <struct luaL_field> after call to
+ * luaL_tofield() / luaL_checkfield().
+ *
+ * For MP_STR: interpret .sval.data as beginning
+ * of the actual string (for testing purposes:
+ * tostring() representation of userdata and cdata
+ * contains an address at the end). Don't compare
+ * .sval.len.
+ */
+ struct luaL_field exp_field;
+ /*
+ * A code to check the resulting value on the Lua
+ * stack after luaL_tofield() / luaL_checkfield().
+ *
+ * 'res' is the object to verify.
+ *
+ * 'src' contains the source object.
+ *
+ * Use 'cmp(a, b)' for a deep comparison.
+ */
+ const char *check_lua;
+ } cases[] = {
+ {
+ .description = "table as array",
+ .src = "return {1, 2, 3}",
+ .exp_field = {
+ .size = 3,
+ .type = MP_ARRAY,
+ .compact = false,
+ },
+ .check_lua = "return res == src",
+ },
+ {
+ .description = "table as map",
+ .src = "return {foo = 'bar'}",
+ .exp_field = {
+ .size = 1,
+ .type = MP_MAP,
+ .compact = false,
+ },
+ .check_lua = "return res == src",
+ },
+ {
+ .description = "table with __serialize = 'map'",
+ .src = "return setmetatable({1, 2, 3}, {"
+ " __serialize = 'map'"
+ "})",
+ .exp_field = {
+ .size = 3,
+ .type = MP_MAP,
+ .compact = true,
+ },
+ .check_lua = "return res == src",
+ },
+ {
+ .description = "table with __serialize function",
+ .src = "return setmetatable({foo = 'bar'}, {"
+ " __serialize = function(self)"
+ " return {1, 2, 3}"
+ " end"
+ "})",
+ .exp_field = {
+ .size = 3,
+ .type = MP_ARRAY,
+ .compact = false,
+ },
+ .check_lua = "return cmp(res, {1, 2, 3})",
+ },
+ {
+ .description = "unknown userdata",
+ .src = "return newproxy()",
+ .exp_field = {
+ .type = MP_EXT,
+ .ext_type = MP_UNKNOWN_EXTENSION,
+ },
+ .check_lua = "return type(res) == 'userdata' and"
+ " res == src",
+ },
+ {
+ .description = "unknown userdata (checkfield)",
+ .src = "return newproxy()",
+ .invoke_checkfield = true,
+ .exp_field = {
+ .sval = {
+ .data = "userdata: 0x",
+ },
+ .type = MP_STR,
+ },
+ .check_lua = "return type(res) == 'string' and"
+ " res:match('^userdata: ')",
+ },
+ {
+ .description = "userdata with __serialize function",
+ .src = "do"
+ " local ud = newproxy(true)"
+ " local mt = getmetatable(ud)"
+ " mt.__serialize = function(self)"
+ " return {1, 2, 3}"
+ " end"
+ /*
+ * See the comment in the next test
+ * case.
+ */
+ " mt.__index = mt"
+ /* Keep the trailing whitespace. */
+ " return ud "
+ "end",
+ .exp_field = {
+ .type = MP_EXT,
+ .ext_type = MP_UNKNOWN_EXTENSION,
+ },
+ .check_lua = "return type(res) == 'userdata' and"
+ " res == src",
+ },
+ {
+ .description = "userdata with __serialize function "
+ "(checkfield)",
+ .src = "do"
+ " local ud = newproxy(true)"
+ " local mt = getmetatable(ud)"
+ " mt.__serialize = function(self)"
+ " return {1, 2, 3}"
+ " end"
+ /*
+ * Surprise! __serialize should be
+ * accessible via lua_gettable(),
+ * otherwise our serializer will
+ * not recognize it. So we should
+ * set __index to the metatable
+ * itself.
+ *
+ * luaL_register_type() does it.
+ * However, personally, I don't see
+ * a strict reason for that and for
+ * me it looks hacky. It would be
+ * better to teach the serializer
+ * to inspect the metatable
+ * directly.
+ */
+ " mt.__index = mt"
+ /* Keep the trailing whitespace. */
+ " return ud "
+ "end",
+ .invoke_checkfield = true,
+ .exp_field = {
+ .size = 3,
+ .type = MP_ARRAY,
+ },
+ .check_lua = "return cmp(res, {1, 2, 3})",
+ },
+ {
+ .description = "unknown cdata",
+ .src = "do"
+ " local ffi = require('ffi')"
+ " ffi.cdef([["
+ " struct foo {"
+ " int x;"
+ " };"
+ " ]])"
+ /* Keep the trailing whitespace. */
+ " return ffi.new('struct foo', {x = 42}) "
+ "end",
+ .exp_field = {
+ .type = MP_EXT,
+ .ext_type = MP_UNKNOWN_EXTENSION,
+ },
+ .check_lua = "do"
+ " local ffi = require('ffi')"
+ " return type(res) == 'cdata' and"
+ " ffi.istype('struct foo', res) and"
+ " res == src and"
+ /* Keep the trailing whitespace. */
+ " res.x == 42 "
+ "end",
+ },
+ {
+ .description = "unknown cdata (checkfield)",
+ .src = "do"
+ " local ffi = require('ffi')"
+ /*
+ * ffi.cdef() is in the previous
+ * test case.
+ */
+ /* Keep the trailing whitespace. */
+ " return ffi.new('struct foo', {x = 42}) "
+ "end",
+ .invoke_checkfield = true,
+ .exp_field = {
+ .sval = {
+ .data = "cdata<struct foo>: 0x"
+ },
+ .type = MP_STR,
+ },
+ .check_lua = "return type(res) == 'string' and"
+ " res:match('^cdata<struct foo>: ')",
+ },
+ {
+ .description = "cdata with __serialize",
+ .src = "do"
+ " local ffi = require('ffi')"
+ " local mt = {"
+ " __serialize = function(self)"
+ " return {1, 2, 3}"
+ " end"
+ " }"
+ /*
+ * See the comment for the userdata
+ * test case above.
+ */
+ " mt.__index = mt"
+ /*
+ * ffi.cdef() is in one of previous
+ * test cases.
+ */
+ " ffi.metatype('struct foo', mt)"
+ /* Keep the trailing whitespace. */
+ " return ffi.new('struct foo', {x = 42}) "
+ "end",
+ .exp_field = {
+ .type = MP_EXT,
+ .ext_type = MP_UNKNOWN_EXTENSION,
+ },
+ .check_lua = "do"
+ " local ffi = require('ffi')"
+ " return type(res) == 'cdata' and"
+ " ffi.istype('struct foo', res) and"
+ " res == src and"
+ /* Keep the trailing whitespace. */
+ " res.x == 42 "
+ "end",
+ },
+ {
+ .description = "cdata with __serialize (checkfield)",
+ .src = "do"
+ " local ffi = require('ffi')"
+ /*
+ * ffi.cdef() and ffi.metatype()
+ * are in previous test cases.
+ */
+ /* Keep the trailing whitespace. */
+ " return ffi.new('struct foo', {x = 42}) "
+ "end",
+ .invoke_checkfield = true,
+ .exp_field = {
+ .size = 3,
+ .type = MP_ARRAY,
+ },
+ .check_lua = "return cmp(res, {1, 2, 3})",
+ },
+ };
+
+ int planned = (int) (sizeof(cases) / sizeof(cases[0]));
+ plan(4 * planned);
+ header();
+
+ /*
+ * Initialize serializer with almost default options.
+ *
+ * Set 'has_compact' to test it (otherwise
+ * <struct luaL_field>.compact will not be set).
+ *
+ * Set 'encode_use_tostring' just to don't introduce
+ * complex code with catching a Lua error from a C
+ * function.
+ */
+ struct luaL_serializer cfg;
+ luaL_serializer_create(&cfg);
+ cfg.has_compact = true;
+ cfg.encode_use_tostring = true;
+
+ /* Add cmp function for check_lua. */
+ luaL_loadstring(L,
+ "do"
+ " local cmp"
+ ""
+ " cmp = function(a, b)"
+ " if type(a) ~= type(b) then"
+ " return false"
+ " end"
+ ""
+ " if type(a) == 'table' then"
+ " for k, v in pairs(a) do"
+ " if not cmp(v, b[k]) then"
+ " return false"
+ " end"
+ " end"
+ " for k, v in pairs(b) do"
+ " if not cmp(v, a[k]) then"
+ " return false"
+ " end"
+ " end"
+ " return true"
+ " end"
+ ""
+ " return a == b"
+ " end"
+ ""
+ /* Caution: keep the trailing whitespace. */
+ " return cmp "
+ "end");
+ lua_call(L, 0, 1);
+ lua_setglobal(L, "cmp");
+
+ for (int i = 0; i < planned; ++i) {
+ int initial_top = lua_gettop(L);
+
+ const char *description = cases[i].description;
+
+ /* Push a Lua object to the Lua stack. */
+ luaL_loadstring(L, cases[i].src);
+ lua_call(L, 0, 1);
+
+ /* Set _G.src. */
+ lua_pushvalue(L, -1);
+ lua_setglobal(L, "src");
+
+ /* Call luaL_tofield() / luaL_checkfield(). */
+ int top = lua_gettop(L);
+ struct luaL_field field;
+ if (cases[i].invoke_checkfield) {
+ luaL_checkfield(L, &cfg, -1, &field);
+ ok(true, "# skip; %s: luaL_checkfield() has no retval",
+ description);
+ } else {
+ int rc = luaL_tofield(L, &cfg, NULL, -1, &field);
+ is(rc, 0, "%s: luaL_tofield retval", description);
+ }
+
+ /* Check stack size. */
+ is(lua_gettop(L) - top, 0, "%s: Lua stack size", description);
+
+ /*
+ * Set _G.res. The object is placed to the same
+ * index as the source object: the top item in our
+ * case.
+ */
+ lua_pushvalue(L, -1);
+ lua_setglobal(L, "res");
+
+ /* Check resulting Lua object. */
+ luaL_loadstring(L, cases[i].check_lua);
+ lua_call(L, 0, 1);
+ is(lua_toboolean(L, -1), 1, "%s: Lua result", description);
+ lua_pop(L, 1);
+
+ /* Check luaL_field content. */
+ check_luaL_field(&field, &cases[i].exp_field, description);
+
+ /* Unset _G.src and _G.res. */
+ lua_pushnil(L);
+ lua_setglobal(L, "src");
+ lua_pushnil(L);
+ lua_setglobal(L, "res");
+
+ /* Clean up the Lua stack. */
+ lua_pop(L, 1);
+ assert(lua_gettop(L) == initial_top);
+ }
+
+ /* Unset _G.cmp. */
+ lua_pushnil(L);
+ lua_setglobal(L, "cmp");
+
+ footer();
+ return check_plan();
+}
+
+int
+main()
+{
+ struct lua_State *L = luaL_newstate();
+ luaL_openlibs(L);
+
+ tarantool_lua_serializer_init(L);
+
+ return test_luaL_field_basic(L);
+}
diff --git a/test/unit/serializer.result b/test/unit/serializer.result
new file mode 100644
index 000000000..74479ac49
--- /dev/null
+++ b/test/unit/serializer.result
@@ -0,0 +1,135 @@
+1..48
+ *** test_luaL_field_basic ***
+ok 1 - table as array: luaL_tofield retval
+ok 2 - table as array: Lua stack size
+ok 3 - table as array: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - table as array: type
+ ok 2 - table as array: size
+ ok 3 - # skip; table as array: Don't compare 'exp_type'
+ ok 4 - table as array: compact
+ *** check_luaL_field: done ***
+ok 4 - subtests
+ok 5 - table as map: luaL_tofield retval
+ok 6 - table as map: Lua stack size
+ok 7 - table as map: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - table as map: type
+ ok 2 - table as map: size
+ ok 3 - # skip; table as map: Don't compare 'exp_type'
+ ok 4 - table as map: compact
+ *** check_luaL_field: done ***
+ok 8 - subtests
+ok 9 - table with __serialize = 'map': luaL_tofield retval
+ok 10 - table with __serialize = 'map': Lua stack size
+ok 11 - table with __serialize = 'map': Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - table with __serialize = 'map': type
+ ok 2 - table with __serialize = 'map': size
+ ok 3 - # skip; table with __serialize = 'map': Don't compare 'exp_type'
+ ok 4 - table with __serialize = 'map': compact
+ *** check_luaL_field: done ***
+ok 12 - subtests
+ok 13 - table with __serialize function: luaL_tofield retval
+ok 14 - table with __serialize function: Lua stack size
+ok 15 - table with __serialize function: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - table with __serialize function: type
+ ok 2 - table with __serialize function: size
+ ok 3 - # skip; table with __serialize function: Don't compare 'exp_type'
+ ok 4 - table with __serialize function: compact
+ *** check_luaL_field: done ***
+ok 16 - subtests
+ok 17 - unknown userdata: luaL_tofield retval
+ok 18 - unknown userdata: Lua stack size
+ok 19 - unknown userdata: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - unknown userdata: type
+ ok 2 - # skip; unknown userdata: Don't check MP_EXT data
+ ok 3 - unknown userdata: ext_type
+ ok 4 - # skip; unknown userdata: Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 20 - subtests
+ok 21 - # skip; unknown userdata (checkfield): luaL_checkfield() has no retval
+ok 22 - unknown userdata (checkfield): Lua stack size
+ok 23 - unknown userdata (checkfield): Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - unknown userdata (checkfield): type
+ ok 2 - unknown userdata (checkfield): sval.data
+ ok 3 - # skip; unknown userdata (checkfield): Don't compare 'exp_type'
+ ok 4 - # skip; unknown userdata (checkfield): Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 24 - subtests
+ok 25 - userdata with __serialize function: luaL_tofield retval
+ok 26 - userdata with __serialize function: Lua stack size
+ok 27 - userdata with __serialize function: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - userdata with __serialize function: type
+ ok 2 - # skip; userdata with __serialize function: Don't check MP_EXT data
+ ok 3 - userdata with __serialize function: ext_type
+ ok 4 - # skip; userdata with __serialize function: Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 28 - subtests
+ok 29 - # skip; userdata with __serialize function (checkfield): luaL_checkfield() has no retval
+ok 30 - userdata with __serialize function (checkfield): Lua stack size
+ok 31 - userdata with __serialize function (checkfield): Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - userdata with __serialize function (checkfield): type
+ ok 2 - userdata with __serialize function (checkfield): size
+ ok 3 - # skip; userdata with __serialize function (checkfield): Don't compare 'exp_type'
+ ok 4 - userdata with __serialize function (checkfield): compact
+ *** check_luaL_field: done ***
+ok 32 - subtests
+ok 33 - unknown cdata: luaL_tofield retval
+ok 34 - unknown cdata: Lua stack size
+ok 35 - unknown cdata: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - unknown cdata: type
+ ok 2 - # skip; unknown cdata: Don't check MP_EXT data
+ ok 3 - unknown cdata: ext_type
+ ok 4 - # skip; unknown cdata: Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 36 - subtests
+ok 37 - # skip; unknown cdata (checkfield): luaL_checkfield() has no retval
+ok 38 - unknown cdata (checkfield): Lua stack size
+ok 39 - unknown cdata (checkfield): Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - unknown cdata (checkfield): type
+ ok 2 - unknown cdata (checkfield): sval.data
+ ok 3 - # skip; unknown cdata (checkfield): Don't compare 'exp_type'
+ ok 4 - # skip; unknown cdata (checkfield): Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 40 - subtests
+ok 41 - cdata with __serialize: luaL_tofield retval
+ok 42 - cdata with __serialize: Lua stack size
+ok 43 - cdata with __serialize: Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - cdata with __serialize: type
+ ok 2 - # skip; cdata with __serialize: Don't check MP_EXT data
+ ok 3 - cdata with __serialize: ext_type
+ ok 4 - # skip; cdata with __serialize: Don't compare 'compact'
+ *** check_luaL_field: done ***
+ok 44 - subtests
+ok 45 - # skip; cdata with __serialize (checkfield): luaL_checkfield() has no retval
+ok 46 - cdata with __serialize (checkfield): Lua stack size
+ok 47 - cdata with __serialize (checkfield): Lua result
+ 1..4
+ *** check_luaL_field ***
+ ok 1 - cdata with __serialize (checkfield): type
+ ok 2 - cdata with __serialize (checkfield): size
+ ok 3 - # skip; cdata with __serialize (checkfield): Don't compare 'exp_type'
+ ok 4 - cdata with __serialize (checkfield): compact
+ *** check_luaL_field: done ***
+ok 48 - subtests
+ *** test_luaL_field_basic: done ***
--
2.31.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
` (3 preceding siblings ...)
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 4/4] test: add a basic unit test for serializer helpers Alexander Turenko via Tarantool-patches
@ 2021-06-24 6:17 ` Cyrill Gorcunov via Tarantool-patches
2021-06-28 6:31 ` Cyrill Gorcunov via Tarantool-patches
` (2 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Cyrill Gorcunov via Tarantool-patches @ 2021-06-24 6:17 UTC (permalink / raw)
To: Alexander Turenko via Tarantool-patches
Cc: Cyrill Gorcunov, Vladislav Shpilevoy, Alexander Turenko
On Wed, Jun 23, 2021 at 10:12:38PM +0300, Alexander Turenko via Tarantool-patches wrote:
> Moved the serializer helpers into its own compilation unit, add some
> comments and a basic test: everything is just to simplify diving into
> this code.
>
> Guys, please, look, whether it seems useful enough to include into
> tarantool's mainline? Should we name it serializer.[ch] or
> somehow like serializer_helpers.[ch]?
>
> Part of https://github.com/tarantool/tarantool/issues/3228
> Branch: Totktonada/gh-3228-extract-serializer-helpers
The series looks ok from a glance, I'll take more precise look
once time permit, but overall idea looks very neat!
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
` (4 preceding siblings ...)
2021-06-24 6:17 ` [Tarantool-patches] [PATCH 0/4] RFC: Isolate " Cyrill Gorcunov via Tarantool-patches
@ 2021-06-28 6:31 ` Cyrill Gorcunov via Tarantool-patches
2021-07-04 13:09 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-07 10:08 ` Alexander Turenko via Tarantool-patches
7 siblings, 0 replies; 16+ messages in thread
From: Cyrill Gorcunov via Tarantool-patches @ 2021-06-28 6:31 UTC (permalink / raw)
To: Alexander Turenko via Tarantool-patches
Cc: Cyrill Gorcunov, Vladislav Shpilevoy, Alexander Turenko
On Wed, Jun 23, 2021 at 10:12:38PM +0300, Alexander Turenko via Tarantool-patches wrote:
> Moved the serializer helpers into its own compilation unit, add some
> comments and a basic test: everything is just to simplify diving into
> this code.
>
> Guys, please, look, whether it seems useful enough to include into
> tarantool's mainline? Should we name it serializer.[ch] or
> somehow like serializer_helpers.[ch]?
>
> Part of https://github.com/tarantool/tarantool/issues/3228
> Branch: Totktonada/gh-3228-extract-serializer-helpers
The series looks ok to me, I think we should give it a spin.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
` (5 preceding siblings ...)
2021-06-28 6:31 ` Cyrill Gorcunov via Tarantool-patches
@ 2021-07-04 13:09 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
2021-07-07 10:08 ` Alexander Turenko via Tarantool-patches
7 siblings, 1 reply; 16+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-04 13:09 UTC (permalink / raw)
To: Alexander Turenko, Cyrill Gorcunov, Roman Khabibov; +Cc: tarantool-patches
Hi! Thanks for the patchset!
On 23.06.2021 21:12, Alexander Turenko via Tarantool-patches wrote:
> Moved the serializer helpers into its own compilation unit, add some
> comments and a basic test: everything is just to simplify diving into
> this code.
>
> Guys, please, look, whether it seems useful enough to include into
> tarantool's mainline? Should we name it serializer.[ch] or
> somehow like serializer_helpers.[ch]?
>
> Part of https://github.com/tarantool/tarantool/issues/3228
Are you sure you need to fix it? It looks like a regular leg shooting.
It might be simple to detect in the case described by Mons, but what if
the recursion is not so easily visible?
setmetatable({},{
__serialize = function(a)
return {{{{a}}}}
end
})
You would need to use recursion detection algorithms like the one
we used to ask on interviews. And I am not sure it is worth it if
it can't be done in a simple way.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file Alexander Turenko via Tarantool-patches
@ 2021-07-04 13:10 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
0 siblings, 1 reply; 16+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-04 13:10 UTC (permalink / raw)
To: Alexander Turenko, Cyrill Gorcunov, Roman Khabibov; +Cc: tarantool-patches
Thanks for the patch!
See 3 comments below.
> diff --git a/src/lua/init.c b/src/lua/init.c
> index 93e93a103..ff11d202b 100644
> --- a/src/lua/init.c
> +++ b/src/lua/init.c
> @@ -69,6 +69,10 @@
> #include <readline/readline.h>
> #include <readline/history.h>
>
> +/* Don't include the entire header only for *_init(). */
1. But what is the problem with that? It would be simpler and
consistent with the other inits.
> diff --git a/src/lua/serializer.c b/src/lua/serializer.c
> new file mode 100644
> index 000000000..6c3dd73af
> --- /dev/null
> +++ b/src/lua/serializer.c
> @@ -0,0 +1,651 @@
> +/*
> + * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +#include <assert.h>
> +#include <stdbool.h>
> +#include <math.h> /* modf, isfinite */
> +#include <lua.h>
> +#include <lauxlib.h>
2. These are already included in the header. And some of the others
below too.
> diff --git a/src/lua/serializer.h b/src/lua/serializer.h
> new file mode 100644
> index 000000000..54b0bc11a
> --- /dev/null
> +++ b/src/lua/serializer.h
> @@ -0,0 +1,344 @@
> +#ifndef TARANTOOL_LUA_SERIALIZER_H_INCLUDED
> +#define TARANTOOL_LUA_SERIALIZER_H_INCLUDED
3. According to the code style, new files should use `#pragma once`.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file
2021-07-04 13:10 ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
2021-07-05 20:59 ` Vladislav Shpilevoy via Tarantool-patches
0 siblings, 1 reply; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-07-05 6:30 UTC (permalink / raw)
To: Vladislav Shpilevoy; +Cc: Cyrill Gorcunov, tarantool-patches
On Sun, Jul 04, 2021 at 03:10:00PM +0200, Vladislav Shpilevoy wrote:
> Thanks for the patch!
>
> See 3 comments below.
>
> > diff --git a/src/lua/init.c b/src/lua/init.c
> > index 93e93a103..ff11d202b 100644
> > --- a/src/lua/init.c
> > +++ b/src/lua/init.c
> > @@ -69,6 +69,10 @@
> > #include <readline/readline.h>
> > #include <readline/history.h>
> >
> > +/* Don't include the entire header only for *_init(). */
>
> 1. But what is the problem with that? It would be simpler and
> consistent with the other inits.
It looks more accurate for me: acquire only what we actually use. Just
like defining an opacue structure in C or using `from foo import bar` in
Python.
It is not the main reason for me, but it also can positively impact
compilation time (less preprocessing).
However, maybe, it would be more accurate to define a macro like
LUA_MODULE_INIT_ONLY before including those headers and add #if
conditions to them.
Anyway, since it looks unusual for you, I'll stick with just #include
<lua/serializer.h>.
> > diff --git a/src/lua/serializer.c b/src/lua/serializer.c
> > new file mode 100644
> > index 000000000..6c3dd73af
> > --- /dev/null
> > +++ b/src/lua/serializer.c
> > @@ -0,0 +1,651 @@
> > <..stripped license text..>
> > +#include <assert.h>
> > +#include <stdbool.h>
> > +#include <math.h> /* modf, isfinite */
> > +#include <lua.h>
> > +#include <lauxlib.h>
>
> 2. These are already included in the header. And some of the others
> below too.
I discussed it once (a long time ago) with Vladimir D. He said that the
rule of thumb is to include a header into a foo.c file if we use
something from it: disregarding whether the header is included into
foo.h or not.
It looks meaningful for me: declare where use. I guess it'll ease future
changes of only .c or only .h a bit.
I'll leave it as is if you don't object.
To be honest, #include directives management is a pain. If there is some
good linter, I would integrate it into the project.
> > diff --git a/src/lua/serializer.h b/src/lua/serializer.h
> > new file mode 100644
> > index 000000000..54b0bc11a
> > --- /dev/null
> > +++ b/src/lua/serializer.h
> > @@ -0,0 +1,344 @@
> > +#ifndef TARANTOOL_LUA_SERIALIZER_H_INCLUDED
> > +#define TARANTOOL_LUA_SERIALIZER_H_INCLUDED
>
> 3. According to the code style, new files should use `#pragma once`.
Changed.
WBR, Alexander Turenko.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-07-04 13:09 ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
0 siblings, 0 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-07-05 6:30 UTC (permalink / raw)
To: Vladislav Shpilevoy; +Cc: Cyrill Gorcunov, tarantool-patches
On Sun, Jul 04, 2021 at 03:09:07PM +0200, Vladislav Shpilevoy wrote:
> Hi! Thanks for the patchset!
>
> On 23.06.2021 21:12, Alexander Turenko via Tarantool-patches wrote:
> > Moved the serializer helpers into its own compilation unit, add some
> > comments and a basic test: everything is just to simplify diving into
> > this code.
> >
> > Guys, please, look, whether it seems useful enough to include into
> > tarantool's mainline? Should we name it serializer.[ch] or
> > somehow like serializer_helpers.[ch]?
> >
> > Part of https://github.com/tarantool/tarantool/issues/3228
>
> Are you sure you need to fix it? It looks like a regular leg shooting.
> It might be simple to detect in the case described by Mons, but what if
> the recursion is not so easily visible?
>
> setmetatable({},{
> __serialize = function(a)
> return {{{{a}}}}
> end
> })
>
> You would need to use recursion detection algorithms like the one
> we used to ask on interviews. And I am not sure it is worth it if
> it can't be done in a simple way.
I think it worth to rearrange the code and add a test disregarding
whether we'll decide to fix or leave the problem.
I'll update the issue on the week with description of all problems found
around __serialize (see at end of the email as well). After this I'll
ask Roman to update its patch (I'll add a checklist what should be done
and how). I'll keep you in CC for those discussions. So you'll have
ability to say 'it looks to complex' at any stage.
In my opinion, it is highly undesirable to get segfault (or even a Lua
error) from a serializer, because it is often used for logging. More or
less correct result is better than fail. Even if the passed Lua object
is ill-formed in some way. (However, sure, I want to keep the code as
readable as possible and I would not accept a solution that is hard for
me to dive into. I hope we'll implement something well balanced.)
To be honest, even our usual "unsupported Lua type 'function'" error
(which is raised for a function if `encode_use_tostring` is not `true`)
is often undesirable. Raw idea: provide a helper like
`yaml.encode_noxc()`, which will never raise an error and will be
suitable for logging in the general case (it'll set
`encode_use_tostring` under the hood).
WBR, Alexander Turenko.
----
Sure, there are two problems with __serialize, which lead to segfault:
- recursion within single Lua object serialization;
- recursion over several Lua objects.
But there is one problem of another kind.
A return value of __serialize does not participate in references search.
| local x = {whoami = 'x'}
| yaml.encode({
| foo = x,
| bar = setmetatable({}, {__serialize = function(_) return x end})
| })
** now **
| ---
| foo:
| whoami: x
| bar:
| whoami: x
| ...
** should be **
| ---
| foo: &1
| whoami: x
| bar: *1
| ...
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file
2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
@ 2021-07-05 20:59 ` Vladislav Shpilevoy via Tarantool-patches
0 siblings, 0 replies; 16+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-05 20:59 UTC (permalink / raw)
To: Alexander Turenko; +Cc: Cyrill Gorcunov, tarantool-patches
Hi! Thanks for the fixes!
The patchset and the plans for 3228 LGTM.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
` (6 preceding siblings ...)
2021-07-04 13:09 ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-07 10:08 ` Alexander Turenko via Tarantool-patches
2021-07-07 19:09 ` Alexander Turenko via Tarantool-patches
7 siblings, 1 reply; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-07-07 10:08 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov; +Cc: tarantool-patches
On Wed, Jun 23, 2021 at 10:12:38PM +0300, Alexander Turenko wrote:
> Moved the serializer helpers into its own compilation unit, add some
> comments and a basic test: everything is just to simplify diving into
> this code.
>
> Guys, please, look, whether it seems useful enough to include into
> tarantool's mainline? Should we name it serializer.[ch] or
> somehow like serializer_helpers.[ch]?
>
> Part of https://github.com/tarantool/tarantool/issues/3228
> Branch: Totktonada/gh-3228-extract-serializer-helpers
>
> Alexander Turenko (4):
> lua: move serializer helpers into its own file
> lua: move luaL_newserializer() comment into header
> lua: split serializer functions into sections
> test: add a basic unit test for serializer helpers
Thanks for reviews!
Pushed to master, 2.8, 2.7.
I'll backport it to 1.10 and show you first.
WBR, Alexander Turenko.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-07-07 10:08 ` Alexander Turenko via Tarantool-patches
@ 2021-07-07 19:09 ` Alexander Turenko via Tarantool-patches
2021-07-07 22:16 ` Vladislav Shpilevoy via Tarantool-patches
0 siblings, 1 reply; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-07-07 19:09 UTC (permalink / raw)
To: Vladislav Shpilevoy, Cyrill Gorcunov, Roman Khabibov; +Cc: tarantool-patches
On Wed, Jul 07, 2021 at 01:08:10PM +0300, Alexander Turenko wrote:
> On Wed, Jun 23, 2021 at 10:12:38PM +0300, Alexander Turenko wrote:
> > Moved the serializer helpers into its own compilation unit, add some
> > comments and a basic test: everything is just to simplify diving into
> > this code.
> >
> > Guys, please, look, whether it seems useful enough to include into
> > tarantool's mainline? Should we name it serializer.[ch] or
> > somehow like serializer_helpers.[ch]?
> >
> > Part of https://github.com/tarantool/tarantool/issues/3228
> > Branch: Totktonada/gh-3228-extract-serializer-helpers
> >
> > Alexander Turenko (4):
> > lua: move serializer helpers into its own file
> > lua: move luaL_newserializer() comment into header
> > lua: split serializer functions into sections
> > test: add a basic unit test for serializer helpers
>
> Thanks for reviews!
>
> Pushed to master, 2.8, 2.7.
>
> I'll backport it to 1.10 and show you first.
>
> WBR, Alexander Turenko.
Given 4 commits are backported almost trivially to 1.10, I'll not resend
them. I added two separate oneliners for 1.10. I'll just place them at
bottom of the email.
All 6 commits could be found on the
Totktonada/gh-3228-extract-serializer-helpers-1.10 branch.
I think that it would be convenient for further development to backport
this patchset to 1.10. It looks safe enough, because it is mainly just
code rearrangement: unlikely it may lead to new subtle bugs (aside of
building, which should be verified quite good by CI).
Please, share thoughts if you have objections against pushing it to
1.10.
CCed Kirill Yu.
----
commit 4a22b90866dfda4252577aac4dd7252e518490ae
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed Jul 7 18:28:37 2021 +0300
build/curl: disable ZSTD content encoding support
I'm not against ZSTD content encoding.
The problem I want to solve is the following. I'll add a unit test in a
future commit:
| add_executable(serializer.test serializer.c)
| target_link_libraries(serializer.test unit box ${LUAJIT_LIBRARIES})
Linking of the test fails if curl's configure script detects ZSTD
presence and enables it. It is irrelevant for building of libcurl using
CMake (since 2.6.0-196-g2b0760192).
ZSTD content encoding support was added in curl-7.72.0, we pull it in
1.10.9-124-ged90a8c22 ('security: update libcurl from 7.71.1 to
7.76.0'). The key problem is that curl's configure script (unlike CMake)
performs autodetection of present libraries by default.
If we'll want to enable this content encoding, it is better to do so
deliberately: with explicit linking against our built-in ZSTD (now the
choice depends on presence of the library in the system) and correct
dependencies in our CMake scripts (correct CURL_LIBRARIES).
Part of #3228
diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake
index 313adfb97..8fe8ff25a 100644
--- a/cmake/BuildLibCURL.cmake
+++ b/cmake/BuildLibCURL.cmake
@@ -95,6 +95,7 @@ macro(curl_build)
--with-ca-fallback
--without-brotli
+ --without-zstd
--without-gnutls
--without-mbedtls
--without-cyassl
commit 378a18a0850ad63ceb0bb5ef141eaa34988b01ed
Author: Alexander Turenko <alexander.turenko@tarantool.org>
Date: Wed Jul 7 19:27:38 2021 +0300
build: add uri library as dependency for core
evio.cc from libcore.a uses uri_parse() from liburi.a. This dependency
is already there on master.
In a future commit I'll add a test:
| add_executable(serializer.test serializer.c)
| target_link_libraries(serializer.test unit box ${LUAJIT_LIBRARIES})
Linking of the test fails if the dependency is not set in CMake scripts.
Part of #3228
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index d8706b9a8..d5a7c6d22 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -120,7 +120,7 @@ endif ()
add_library(core STATIC ${core_sources})
add_dependencies(core curl)
target_link_libraries(core
- salad small
+ salad small uri
${LIBEV_LIBRARIES}
${LIBEIO_LIBRARIES}
${LIBCORO_LIBRARIES}
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-07-07 19:09 ` Alexander Turenko via Tarantool-patches
@ 2021-07-07 22:16 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-12 7:51 ` Alexander Turenko via Tarantool-patches
0 siblings, 1 reply; 16+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-07 22:16 UTC (permalink / raw)
To: Alexander Turenko, Cyrill Gorcunov, Roman Khabibov; +Cc: tarantool-patches
Hi! Thanks for the backporting!
LGTM as well.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers
2021-07-07 22:16 ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-12 7:51 ` Alexander Turenko via Tarantool-patches
0 siblings, 0 replies; 16+ messages in thread
From: Alexander Turenko via Tarantool-patches @ 2021-07-12 7:51 UTC (permalink / raw)
To: Vladislav Shpilevoy; +Cc: Cyrill Gorcunov, tarantool-patches
On Thu, Jul 08, 2021 at 12:16:17AM +0200, Vladislav Shpilevoy wrote:
> Hi! Thanks for the backporting!
>
> LGTM as well.
Pushed to 1.10.
WBR, Alexander Turenko.
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2021-07-12 7:52 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-23 19:12 [Tarantool-patches] [PATCH 0/4] RFC: Isolate serializer helpers Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 1/4] lua: move serializer helpers into its own file Alexander Turenko via Tarantool-patches
2021-07-04 13:10 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
2021-07-05 20:59 ` Vladislav Shpilevoy via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 2/4] lua: move luaL_newserializer() comment into header Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 3/4] lua: split serializer functions into sections Alexander Turenko via Tarantool-patches
2021-06-23 19:12 ` [Tarantool-patches] [PATCH 4/4] test: add a basic unit test for serializer helpers Alexander Turenko via Tarantool-patches
2021-06-24 6:17 ` [Tarantool-patches] [PATCH 0/4] RFC: Isolate " Cyrill Gorcunov via Tarantool-patches
2021-06-28 6:31 ` Cyrill Gorcunov via Tarantool-patches
2021-07-04 13:09 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-05 6:30 ` Alexander Turenko via Tarantool-patches
2021-07-07 10:08 ` Alexander Turenko via Tarantool-patches
2021-07-07 19:09 ` Alexander Turenko via Tarantool-patches
2021-07-07 22:16 ` Vladislav Shpilevoy via Tarantool-patches
2021-07-12 7:51 ` Alexander Turenko via Tarantool-patches
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox