[Tarantool-patches] [PATCH 1.10 v4 0/5] module api: extend for external merger Lua module

[Tarantool-patches] [PATCH 1.10 v4 0/5] module api: extend for external merger Lua module

[Timur Safin]

Ok we are all tired and need to sleep, but let me reiterate it once again:

The current patchset is a 1.10 backport of a 2.X series which was in turn 
continuation of patch series which Alexander Turenko has sent before.

The major difference here - is the way how we export symbols:
- in 1.10 there was no `src/exports.h` existing, and we were using
  old good `extra/exports` instead;
- also we have added 1 extra backport of `luaL_iscallable`.

Enjoy!

Changelog of v2 to v1
---------------------
Please do not be surprised to not seen v2 in your inbox - it was cancelled 
after discussion with Alexander Turenko (see v2.1 below with descriptions)

- Unpublished again `lauT_temp_state`, `luaT_release_temp_state` - they are
  performance wrappers around creation of Lua thread states. Agreed that 
  performance impact not that big, thus created similar compatibility wrappers
  on the module side;
- unpublished `luaT_newthread`, because for module usage `lua_newthread` is 
  enough;
- unpublished `luaL_register_module` & `luaL_register_type` and reworked module
  code with `luaL_register`.

Smallish changelog of v2.1 to v2
--------------------------------
Alexander Turenko has provided valuable feedback to the v2 branch I've shown,
so we have yet more reduced this patchset
- Unpublished `luaL_cdata_iscallable`, it's unnecessary for external module
  which is using now public `luaL_iscallable` module api call, instead 
  of compatibility layer in module;
- Unpublished `luaL_checkconstchar` because it's not yet needed anywhere
  externally;
- Accordingly to thsoe changes above we have reduced code in module_api
  test files.

Changelog for v3
----------------
- Reshuffled patchset to merge tests with their corresponding code;

Changelog v3.1
--------------
- Added ibuf wrapper, the simpler way (only reserve, and rpos, wpos accessors)

Changelog v4
------------
- final polishing of examples
- tests for luaT_toibuf, with ibuf consistency checks
- tests for key_def_dup
- tests for tuple_validate


Alexander Turenko (1):
  module api: add luaL_iscallable with support of cdata metatype

Timur Safin (4):
  module api: export box_tuple_validate
  module api: export box_key_def_dup
  module api: introduced luaT_toibuf instead of luaL_checkibuf
  module api: box_ibuf_* wrappers

 extra/exports                    |   7 ++
 src/CMakeLists.txt               |   1 +
 src/box/CMakeLists.txt           |   1 +
 src/box/ibuf.c                   |  65 ++++++++++++++
 src/box/ibuf.h                   |  67 +++++++++++++++
 src/box/key_def.c                |   6 ++
 src/box/key_def.h                |  10 +++
 src/box/tuple.c                  |   8 ++
 src/box/tuple.h                  |  11 +++
 src/lua/msgpack.c                |   2 +-
 src/lua/utils.c                  |  45 +++++++++-
 src/lua/utils.h                  |  32 +++++--
 test/app-tap/module_api.c        | 140 +++++++++++++++++++++++++++++++
 test/app-tap/module_api.test.lua | 131 ++++++++++++++++++++++++++++-
 14 files changed, 517 insertions(+), 9 deletions(-)
 create mode 100644 src/box/ibuf.c
 create mode 100644 src/box/ibuf.h

-- 
2.20.1

[Tarantool-patches] [PATCH 1.10 v4 1/5] module api: export box_tuple_validate

[Timur Safin]

For external merger we need means to validate tuple data,
thus exporting `box_tuple_validate` which is wrapper around
`tuple_validate` without revealing access to tuple
internals.

Part of #5384
---
 extra/exports                    |  1 +
 src/box/tuple.c                  |  8 ++++++
 src/box/tuple.h                  | 11 ++++++++
 test/app-tap/module_api.c        | 48 ++++++++++++++++++++++++++++++++
 test/app-tap/module_api.test.lua | 29 ++++++++++++++++++-
 5 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/extra/exports b/extra/exports
index 960c50635..bd05cb08e 100644
--- a/extra/exports
+++ b/extra/exports
@@ -178,6 +178,7 @@ box_tuple_upsert
 box_tuple_extract_key
 box_tuple_compare
 box_tuple_compare_with_key
+box_tuple_validate
 box_return_tuple
 box_space_id_by_name
 box_index_id_by_name
diff --git a/src/box/tuple.c b/src/box/tuple.c
index 1db69e414..fb83948d7 100644
--- a/src/box/tuple.c
+++ b/src/box/tuple.c
@@ -576,6 +576,14 @@ box_tuple_new(box_tuple_format_t *format, const char *data, const char *end)
 	return tuple_bless(ret);
 }
 
+int
+box_tuple_validate(box_tuple_t *tuple, box_tuple_format_t *format)
+{
+	return tuple_validate(format, tuple);
+}
+
+/* }}} box_tuple_* */
+
 int
 tuple_snprint(char *buf, int size, const struct tuple *tuple)
 {
diff --git a/src/box/tuple.h b/src/box/tuple.h
index 5c6dc6b8c..217c438c5 100644
--- a/src/box/tuple.h
+++ b/src/box/tuple.h
@@ -284,6 +284,17 @@ box_tuple_t *
 box_tuple_upsert(const box_tuple_t *tuple, const char *expr, const
 		 char *expr_end);
 
+/**
+ * Check tuple data correspondence to the space format.
+ * @param tuple  Tuple to validate.
+ * @param format Format to which the tuple must match.
+ *
+ * @retval  0 The tuple is valid.
+ * @retval -1 The tuple is invalid.
+ */
+int
+box_tuple_validate(box_tuple_t *tuple, box_tuple_format_t *format);
+
 /** \endcond public */
 
 /**
diff --git a/test/app-tap/module_api.c b/test/app-tap/module_api.c
index c80896c45..672272e25 100644
--- a/test/app-tap/module_api.c
+++ b/test/app-tap/module_api.c
@@ -1800,6 +1800,52 @@ test_tuple_new(struct lua_State *L)
 
 /* }}} test_tuple_new */
 
+/*
+ * Check that argument is a tuple of any format, without
+ * its verification
+ */
+static int
+test_tuple_validate_default(lua_State *L)
+{
+	int valid = 0;
+	box_tuple_t *tuple = luaT_istuple(L, -1);
+
+	if (tuple != NULL) {
+		box_tuple_format_t *format = box_tuple_format_default();
+		valid = box_tuple_validate(tuple, format) == 0;
+	}
+	lua_pushboolean(L, valid);
+
+	return 1;
+}
+
+/*
+ * Validate tuple with format of single boolean field
+ */
+static int
+test_tuple_validate_formatted(lua_State *L)
+{
+	int valid = 0;
+	box_tuple_t *tuple = luaT_istuple(L, -1);
+
+	if (tuple != NULL) {
+		uint32_t fields[] = { 0 };
+		uint32_t types[] = { FIELD_TYPE_BOOLEAN };
+		box_key_def_t *key_defs[] = {
+			box_key_def_new(fields, types, 1)
+		};
+		assert(key_defs[0] != NULL);
+		struct tuple_format *format =
+			box_tuple_format_new(key_defs, 1);
+		assert(format);
+
+		valid = box_tuple_validate(tuple, format) == 0;
+	}
+	lua_pushboolean(L, valid);
+
+	return 1;
+}
+
 LUA_API int
 luaopen_module_api(lua_State *L)
 {
@@ -1837,6 +1883,8 @@ luaopen_module_api(lua_State *L)
 		{"test_key_def_merge", test_key_def_merge},
 		{"test_key_def_extract_key", test_key_def_extract_key},
 		{"test_key_def_validate_key", test_key_def_validate_key},
+		{"tuple_validate_def", test_tuple_validate_default},
+		{"tuple_validate_fmt", test_tuple_validate_formatted},
 		{NULL, NULL}
 	};
 	luaL_register(L, "module_api", lib);
diff --git a/test/app-tap/module_api.test.lua b/test/app-tap/module_api.test.lua
index 10b719686..f45592ef9 100755
--- a/test/app-tap/module_api.test.lua
+++ b/test/app-tap/module_api.test.lua
@@ -36,6 +36,32 @@ local function test_pushcdata(test, module)
     test:is(gc_counter, 1, 'pushcdata gc')
 end
 
+local function test_tuple_validate(test, module)
+    test:plan(12)
+
+    local nottuple1 = {}
+    local nottuple2 = {true, 2}
+    local nottuple3 = {false, nil, 2}
+    local nottuple4 = {1, box.NULL, 2, 3}
+    local tuple1 = box.tuple.new(nottuple1)
+    local tuple2 = box.tuple.new(nottuple2)
+    local tuple3 = box.tuple.new(nottuple3)
+    local tuple4 = box.tuple.new(nottuple4)
+
+    test:ok(not module.tuple_validate_def(nottuple1), "not tuple 1")
+    test:ok(not module.tuple_validate_def(nottuple2), "not tuple 2")
+    test:ok(not module.tuple_validate_def(nottuple3), "not tuple 3")
+    test:ok(not module.tuple_validate_def(nottuple4), "not tuple 4")
+    test:ok(module.tuple_validate_def(tuple1), "tuple 1")
+    test:ok(module.tuple_validate_def(tuple2), "tuple 2")
+    test:ok(module.tuple_validate_def(tuple3), "tuple 3")
+    test:ok(module.tuple_validate_def(tuple4), "tuple 4")
+    test:ok(not module.tuple_validate_fmt(tuple1), "tuple 1 (fmt)")
+    test:ok(module.tuple_validate_fmt(tuple2), "tuple 2 (fmt)")
+    test:ok(module.tuple_validate_fmt(tuple3), "tuple 3 (fmt)")
+    test:ok(not module.tuple_validate_fmt(tuple4), "tuple 4 (fmt)")
+end
+
 local function test_iscdata(test, module)
     local ffi = require('ffi')
     ffi.cdef([[
@@ -97,7 +123,7 @@ local function test_iscdata(test, module)
 end
 
 local test = require('tap').test("module_api", function(test)
-    test:plan(33)
+    test:plan(34)
     local status, module = pcall(require, 'module_api')
     test:is(status, true, "module")
     test:ok(status, "module is loaded")
@@ -123,6 +149,7 @@ local test = require('tap').test("module_api", function(test)
 
     test:test("pushcdata", test_pushcdata, module)
     test:test("iscdata", test_iscdata, module)
+    test:test("tuple_validate", test_tuple_validate, module)
 
     space:drop()
 end)
-- 
2.20.1

[Tarantool-patches] [PATCH 1.10 v4 2/5] module api: export box_key_def_dup

[Timur Safin]

Exporting `box_key_def_dup` as accessor to the internal `key_def_dup`

Part of #5384
---
 extra/exports                    |  1 +
 src/box/key_def.c                |  6 ++++++
 src/box/key_def.h                | 10 ++++++++++
 test/app-tap/module_api.c        | 29 +++++++++++++++++++++++++++++
 test/app-tap/module_api.test.lua |  2 +-
 5 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/extra/exports b/extra/exports
index bd05cb08e..da3308230 100644
--- a/extra/exports
+++ b/extra/exports
@@ -154,6 +154,7 @@ box_key_def_validate_tuple
 box_key_def_extract_key
 box_key_def_validate_full_key
 box_key_def_validate_key
+box_key_def_dup
 box_key_part_def_create
 box_tuple_format_default
 box_tuple_new
diff --git a/src/box/key_def.c b/src/box/key_def.c
index 5d556c300..15947d1e2 100644
--- a/src/box/key_def.c
+++ b/src/box/key_def.c
@@ -333,6 +333,12 @@ box_key_def_new_v2(box_key_part_def_t *parts, uint32_t part_count)
 	return key_def;
 }
 
+box_key_def_t *
+box_key_def_dup(const box_key_def_t *key_def)
+{
+	return key_def_dup(key_def);
+}
+
 void
 box_key_def_delete(box_key_def_t *key_def)
 {
diff --git a/src/box/key_def.h b/src/box/key_def.h
index 6326da7b5..01556958f 100644
--- a/src/box/key_def.h
+++ b/src/box/key_def.h
@@ -309,6 +309,16 @@ box_key_part_def_create(box_key_part_def_t *part);
 API_EXPORT box_key_def_t *
 box_key_def_new_v2(box_key_part_def_t *parts, uint32_t part_count);
 
+/**
+ * Duplicate key_def.
+ * @param key_def Original key_def.
+ *
+ * @retval not NULL Duplicate of src.
+ * @retval     NULL Memory error.
+ */
+API_EXPORT box_key_def_t *
+box_key_def_dup(const box_key_def_t *key_def);
+
 /**
  * Delete key definition
  *
diff --git a/test/app-tap/module_api.c b/test/app-tap/module_api.c
index 672272e25..6d4f7a413 100644
--- a/test/app-tap/module_api.c
+++ b/test/app-tap/module_api.c
@@ -1464,6 +1464,34 @@ test_key_def_validate_key(struct lua_State *L)
 	return 1;
 }
 
+static int
+test_key_def_dup(lua_State *L)
+{
+	box_key_def_t *key_def = NULL;
+	box_key_part_def_t part;
+	box_key_part_def_t *dump = NULL;
+	uint32_t dump_part_count = 0;
+
+	key_part_def_set_nondefault(&part);
+	key_def = box_key_def_new_v2(&part, 1);
+	assert(key_def != NULL);
+	box_key_def_t *key_def_dup = box_key_def_dup(key_def);
+	assert(key_def_dup != NULL);
+
+	dump = box_key_def_dump_parts(key_def_dup, &dump_part_count);
+	assert(dump != NULL);
+	assert(dump_part_count == 1);
+
+	key_part_def_check_equal(&part, &dump[0]);
+	key_part_def_check_zeros(&dump[0]);
+
+	box_key_def_delete(key_def_dup);
+	box_key_def_delete(key_def);
+
+	lua_pushboolean(L, 1);
+	return 1;
+}
+
 /* }}} key_def api v2 */
 
 static int
@@ -1883,6 +1911,7 @@ luaopen_module_api(lua_State *L)
 		{"test_key_def_merge", test_key_def_merge},
 		{"test_key_def_extract_key", test_key_def_extract_key},
 		{"test_key_def_validate_key", test_key_def_validate_key},
+		{"test_key_def_dup", test_key_def_dup},
 		{"tuple_validate_def", test_tuple_validate_default},
 		{"tuple_validate_fmt", test_tuple_validate_formatted},
 		{NULL, NULL}
diff --git a/test/app-tap/module_api.test.lua b/test/app-tap/module_api.test.lua
index f45592ef9..0bf6c8a02 100755
--- a/test/app-tap/module_api.test.lua
+++ b/test/app-tap/module_api.test.lua
@@ -123,7 +123,7 @@ local function test_iscdata(test, module)
 end
 
 local test = require('tap').test("module_api", function(test)
-    test:plan(34)
+    test:plan(35)
     local status, module = pcall(require, 'module_api')
     test:is(status, true, "module")
     test:ok(status, "module is loaded")
-- 
2.20.1

[Tarantool-patches] [PATCH 1.10 v4 3/5] module api: introduced luaT_toibuf instead of luaL_checkibuf

[Timur Safin]

* made `luaL_checkibuf` public;
* renamed it to `luaT_toibuf` to follow naming convention
  (it's not raising error, and is casting to ibuf type).

Part of #5384
---
 extra/exports                    |  1 +
 src/lua/msgpack.c                |  2 +-
 src/lua/utils.c                  |  2 +-
 src/lua/utils.h                  | 22 +++++++++++++++++-----
 test/app-tap/module_api.c        | 10 ++++++++++
 test/app-tap/module_api.test.lua | 21 ++++++++++++++++++++-
 6 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/extra/exports b/extra/exports
index da3308230..37077120d 100644
--- a/extra/exports
+++ b/extra/exports
@@ -134,6 +134,7 @@ luaT_error
 luaT_call
 luaT_cpcall
 luaT_state
+luaT_toibuf
 luaT_tolstring
 luaT_tuple_encode
 luaT_tuple_new
diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c
index e3935d383..838f582ff 100644
--- a/src/lua/msgpack.c
+++ b/src/lua/msgpack.c
@@ -442,7 +442,7 @@ lua_msgpack_encode(lua_State *L)
 
 	struct ibuf *buf;
 	if (index > 1) {
-		buf = luaL_checkibuf(L, 2);
+		buf = luaT_toibuf(L, 2);
 		if (buf == NULL) {
 			return luaL_error(L, "msgpack.encode: argument 2 "
 					  "must be of type 'struct ibuf'");
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 318715960..90515affc 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -1030,7 +1030,7 @@ luaT_tolstring(lua_State *L, int idx, size_t *len)
 }
 
 struct ibuf *
-luaL_checkibuf(struct lua_State *L, int idx)
+luaT_toibuf(struct lua_State *L, int idx)
 {
 	if (lua_type(L, idx) != LUA_TCDATA)
 		return NULL;
diff --git a/src/lua/utils.h b/src/lua/utils.h
index d6fbd0e48..a44c419d3 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -505,6 +505,14 @@ luaT_state(void);
 LUA_API const char *
 luaT_tolstring(lua_State *L, int idx, size_t *ssize);
 
+/**
+ * Check if a value on @a L stack by index @a idx is an ibuf
+ * object. Both 'struct ibuf' and 'struct ibuf *' are accepted.
+ * Returns NULL, if can't convert - not an ibuf object.
+ */
+LUA_API struct ibuf *
+luaT_toibuf(struct lua_State *L, int idx);
+
 /** \endcond public */
 
 void
@@ -589,12 +597,16 @@ luaL_checkfinite(struct lua_State *L, struct luaL_serializer *cfg,
 }
 
 /**
- * Check if a value on @a L stack by index @a idx is an ibuf
- * object. Both 'struct ibuf' and 'struct ibuf *' are accepted.
- * Returns NULL, if can't convert - not an ibuf object.
+ * @brief Creates a new Lua coroutine in a protected frame. If
+ * <lua_newthread> call underneath succeeds, the created Lua state
+ * is on the top of the guest stack and a pointer to this state is
+ * returned. Otherwise LUA_ERRMEM error is handled and the result
+ * is NULL.
+ * @param L is a Lua state
+ * @sa <lua_newthread>
  */
-struct ibuf *
-luaL_checkibuf(struct lua_State *L, int idx);
+struct lua_State *
+luaT_newthread(struct lua_State *L);
 
 /**
  * Check if a value on @a L stack by index @a idx is pointer at
diff --git a/test/app-tap/module_api.c b/test/app-tap/module_api.c
index 6d4f7a413..e29abcb85 100644
--- a/test/app-tap/module_api.c
+++ b/test/app-tap/module_api.c
@@ -155,6 +155,15 @@ test_checkint64(lua_State *L)
 	return 1;
 }
 
+static int
+test_checkibuf(lua_State *L)
+{
+	struct ibuf *buf;
+	buf = luaT_toibuf(L, -1);
+	lua_pushboolean(L, buf != NULL);
+	return 1;
+}
+
 static int
 test_touint64(lua_State *L)
 {
@@ -1887,6 +1896,7 @@ luaopen_module_api(lua_State *L)
 		{"test_pushint64", test_pushint64 },
 		{"test_checkuint64", test_checkuint64 },
 		{"test_checkint64", test_checkint64 },
+		{"checkibuf", test_checkibuf},
 		{"test_touint64", test_touint64 },
 		{"test_toint64", test_toint64 },
 		{"test_fiber", test_fiber },
diff --git a/test/app-tap/module_api.test.lua b/test/app-tap/module_api.test.lua
index 0bf6c8a02..acda3deec 100755
--- a/test/app-tap/module_api.test.lua
+++ b/test/app-tap/module_api.test.lua
@@ -36,6 +36,24 @@ local function test_pushcdata(test, module)
     test:is(gc_counter, 1, 'pushcdata gc')
 end
 
+local function test_buffers(test, module)
+    test:plan(8)
+    local ffi = require('ffi')
+    local buffer = require('buffer')
+
+    local ibuf = buffer.ibuf()
+    local pbuf = ibuf:alloc(128)
+
+    test:ok(not module.checkibuf(nil), 'checkibuf of nil')
+    test:ok(not module.checkibuf({}), 'checkibuf of {}')
+    test:ok(not module.checkibuf(1LL), 'checkibuf of 1LL')
+    test:ok(not module.checkibuf(box.NULL), 'checkibuf of box.NULL')
+    test:ok(not module.checkibuf(buffer.reg1), 'checkibuf of reg1')
+    test:ok(module.checkibuf(buffer.IBUF_SHARED), "checkibuf of ibuf*")
+    test:ok(module.checkibuf(ibuf), 'checkibuf of ibuf')
+    test:ok(not module.checkibuf(pbuf), 'checkibuf of pointer to ibuf data')
+end
+
 local function test_tuple_validate(test, module)
     test:plan(12)
 
@@ -123,7 +141,7 @@ local function test_iscdata(test, module)
 end
 
 local test = require('tap').test("module_api", function(test)
-    test:plan(35)
+    test:plan(36)
     local status, module = pcall(require, 'module_api')
     test:is(status, true, "module")
     test:ok(status, "module is loaded")
@@ -149,6 +167,7 @@ local test = require('tap').test("module_api", function(test)
 
     test:test("pushcdata", test_pushcdata, module)
     test:test("iscdata", test_iscdata, module)
+    test:test("buffers", test_buffers, module)
     test:test("tuple_validate", test_tuple_validate, module)
 
     space:drop()
-- 
2.20.1

[Tarantool-patches] [PATCH 1.10 v4 4/5] module api: box_ibuf_* wrappers

[Timur Safin]

Introduced the bare minimum of ibuf accessors, which make
external merger possible:
- box_ibuf_reserve
- box_ibuf_read_range
- box_ibuf_write_range

Part of #5384
---
 extra/exports                    |  3 ++
 src/CMakeLists.txt               |  1 +
 src/box/CMakeLists.txt           |  1 +
 src/box/ibuf.c                   | 65 +++++++++++++++++++++++++++++++
 src/box/ibuf.h                   | 67 ++++++++++++++++++++++++++++++++
 test/app-tap/module_api.c        | 44 +++++++++++++++++++++
 test/app-tap/module_api.test.lua |  2 +-
 7 files changed, 182 insertions(+), 1 deletion(-)
 create mode 100644 src/box/ibuf.c
 create mode 100644 src/box/ibuf.h

diff --git a/extra/exports b/extra/exports
index 37077120d..d918b7e59 100644
--- a/extra/exports
+++ b/extra/exports
@@ -181,6 +181,9 @@ box_tuple_extract_key
 box_tuple_compare
 box_tuple_compare_with_key
 box_tuple_validate
+box_ibuf_reserve
+box_ibuf_read_range
+box_ibuf_write_range
 box_return_tuple
 box_space_id_by_name
 box_index_id_by_name
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 45a8f7733..87c9ccc0e 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -204,6 +204,7 @@ set(api_headers
     ${CMAKE_SOURCE_DIR}/src/box/tuple_extract_key.h
     ${CMAKE_SOURCE_DIR}/src/box/schema_def.h
     ${CMAKE_SOURCE_DIR}/src/box/box.h
+    ${CMAKE_SOURCE_DIR}/src/box/ibuf.h
     ${CMAKE_SOURCE_DIR}/src/box/index.h
     ${CMAKE_SOURCE_DIR}/src/box/iterator_type.h
     ${CMAKE_SOURCE_DIR}/src/box/error.h
diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
index 52413d3cf..4dbdb66cd 100644
--- a/src/box/CMakeLists.txt
+++ b/src/box/CMakeLists.txt
@@ -113,6 +113,7 @@ add_library(box STATIC
     journal.c
     wal.c
     call.c
+    ibuf.c
     ${lua_sources}
     lua/init.c
     lua/call.c
diff --git a/src/box/ibuf.c b/src/box/ibuf.c
new file mode 100644
index 000000000..12b9f2781
--- /dev/null
+++ b/src/box/ibuf.c
@@ -0,0 +1,65 @@
+/*
+ * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdlib.h>
+
+#include "ibuf.h"
+#include <small/ibuf.h>
+
+void *
+box_ibuf_reserve(box_ibuf_t *ibuf, size_t size)
+{
+	return ibuf_reserve(ibuf, size);
+}
+
+void
+box_ibuf_read_range(box_ibuf_t *ibuf, char ***rpos, char ***wpos)
+{
+	assert(ibuf != NULL);
+	if (ibuf == NULL)
+		return;
+	if (rpos != NULL)
+		*rpos = &ibuf->rpos;
+	if (wpos != NULL)
+		*wpos = &ibuf->wpos;
+}
+
+void
+box_ibuf_write_range(box_ibuf_t *ibuf, char ***wpos, char ***end)
+{
+	if (ibuf == NULL)
+		return;
+	if (wpos != NULL)
+		*wpos = &ibuf->wpos;
+	if (end != NULL)
+		*end = &ibuf->end;
+
+}
diff --git a/src/box/ibuf.h b/src/box/ibuf.h
new file mode 100644
index 000000000..d17f8f4b4
--- /dev/null
+++ b/src/box/ibuf.h
@@ -0,0 +1,67 @@
+#pragma once
+/*
+ * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdlib.h>
+
+#include <trivia/util.h>
+
+#if defined(__cplusplus)
+extern "C" {
+#endif /* defined(__cplusplus) */
+
+/** \cond public */
+
+typedef struct ibuf box_ibuf_t;
+
+/**
+ * Reserve requested amount of bytes in ibuf buffer
+ */
+API_EXPORT void *
+box_ibuf_reserve(box_ibuf_t *ibuf, size_t size);
+
+/**
+ * Return pointers to read range pointers used [rpos..wpos)
+ */
+API_EXPORT void
+box_ibuf_read_range(box_ibuf_t *ibuf, char ***rpos, char ***wpos);
+
+/**
+ * Return pointers to write range pointers used [wpos..end)
+ */
+API_EXPORT void
+box_ibuf_write_range(box_ibuf_t *ibuf, char ***wpos, char ***end);
+
+/** \endcond public */
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
diff --git a/test/app-tap/module_api.c b/test/app-tap/module_api.c
index e29abcb85..35df2403b 100644
--- a/test/app-tap/module_api.c
+++ b/test/app-tap/module_api.c
@@ -164,6 +164,49 @@ test_checkibuf(lua_State *L)
 	return 1;
 }
 
+static int
+test_box_ibuf(lua_State *L)
+{
+	struct slab_cache *slabc = cord_slab_cache();
+	assert(slabc != NULL);
+	box_ibuf_t ibuf;
+
+	ibuf_create(&ibuf, slabc, 16320);
+	assert(ibuf_used(&ibuf) == 0);
+	void *ptr = box_ibuf_reserve(&ibuf, 65536);
+	assert(ptr != NULL);
+	char **rpos;
+	char **wpos;
+	box_ibuf_read_range(&ibuf, &rpos, &wpos);
+
+	ptr = ibuf_alloc(&ibuf, 10);
+	assert(ptr != NULL);
+
+	assert(ibuf_used(&ibuf) == 10);
+	assert((*wpos - *rpos) == 10);
+
+	/* let be a little bit paranoid and double check */
+	box_ibuf_read_range(&ibuf, &rpos, &wpos);
+	assert((*wpos - *rpos) == 10);
+
+	ptr = ibuf_alloc(&ibuf, 10000);
+	assert(ptr);
+	assert(ibuf_used(&ibuf) == 10010);
+	assert((*wpos - *rpos) == 10010);
+
+	size_t unused = ibuf_unused(&ibuf);
+	char **end;
+	box_ibuf_write_range(&ibuf, &wpos, &end);
+	assert((*end - *wpos) == (ptrdiff_t)unused);
+
+	ibuf_reset(&ibuf);
+	assert(ibuf_used(&ibuf) == 0);
+	assert(*rpos == *wpos);
+
+	lua_pushboolean(L, 1);
+	return 1;
+}
+
 static int
 test_touint64(lua_State *L)
 {
@@ -1924,6 +1967,7 @@ luaopen_module_api(lua_State *L)
 		{"test_key_def_dup", test_key_def_dup},
 		{"tuple_validate_def", test_tuple_validate_default},
 		{"tuple_validate_fmt", test_tuple_validate_formatted},
+		{"test_box_ibuf", test_box_ibuf},
 		{NULL, NULL}
 	};
 	luaL_register(L, "module_api", lib);
diff --git a/test/app-tap/module_api.test.lua b/test/app-tap/module_api.test.lua
index acda3deec..b8c10a310 100755
--- a/test/app-tap/module_api.test.lua
+++ b/test/app-tap/module_api.test.lua
@@ -141,7 +141,7 @@ local function test_iscdata(test, module)
 end
 
 local test = require('tap').test("module_api", function(test)
-    test:plan(36)
+    test:plan(37)
     local status, module = pcall(require, 'module_api')
     test:is(status, true, "module")
     test:ok(status, "module is loaded")
-- 
2.20.1

[Tarantool-patches] [PATCH 1.10 v4 5/5] module api: add luaL_iscallable with support of cdata metatype

[Timur Safin]

From: Alexander Turenko <alexander.turenko@tarantool.org>

Part of #5384.

(cherry picked from 82dc8c7110f744752dd81a975d51ac97398d120d)

Co-authored-by: Timur Safin <tsafin@tarantool.org>
---
 extra/exports                    |  1 +
 src/lua/utils.c                  | 43 ++++++++++++++++
 src/lua/utils.h                  | 10 ++++
 test/app-tap/module_api.c        |  9 ++++
 test/app-tap/module_api.test.lua | 85 +++++++++++++++++++++++++++++++-
 5 files changed, 146 insertions(+), 2 deletions(-)

diff --git a/extra/exports b/extra/exports
index d918b7e59..94e04264e 100644
--- a/extra/exports
+++ b/extra/exports
@@ -138,6 +138,7 @@ luaT_toibuf
 luaT_tolstring
 luaT_tuple_encode
 luaT_tuple_new
+luaL_iscallable
 box_txn
 box_txn_begin
 box_txn_commit
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 90515affc..e9b783afb 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -1058,6 +1058,49 @@ luaL_checkconstchar(struct lua_State *L, int idx, const char **res,
 	return 0;
 }
 
+/* Based on ffi_meta___call() from luajit/src/lib_ffi.c. */
+static int
+luaL_cdata_iscallable(lua_State *L, int idx)
+{
+	/* Calculate absolute value in the stack. */
+	if (idx < 0)
+		idx = lua_gettop(L) + idx + 1;
+
+	/* Get cdata from the stack. */
+	assert(lua_type(L, idx) == LUA_TCDATA);
+	GCcdata *cd = cdataV(L->base + idx - 1);
+
+	CTState *cts = ctype_cts(L);
+	CTypeID id = cd->ctypeid;
+	CType *ct = ctype_raw(cts, id);
+	if (ctype_isptr(ct->info))
+		id = ctype_cid(ct->info);
+
+	/* Get ctype metamethod. */
+	cTValue *tv = lj_ctype_meta(cts, id, MM_call);
+
+	return tv != NULL;
+}
+
+int
+luaL_iscallable(lua_State *L, int idx)
+{
+	/* Whether it is function. */
+	int res = lua_isfunction(L, idx);
+	if (res == 1)
+		return 1;
+
+	/* Whether it is cdata with metatype with __call field. */
+	if (lua_type(L, idx) == LUA_TCDATA)
+		return luaL_cdata_iscallable(L, idx);
+
+	/* Whether it has metatable with __call field. */
+	res = luaL_getmetafield(L, idx, "__call");
+	if (res == 1)
+		lua_pop(L, 1); /* Pop __call value. */
+	return res;
+}
+
 lua_State *
 luaT_state(void)
 {
diff --git a/src/lua/utils.h b/src/lua/utils.h
index a44c419d3..3eaf1e417 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -505,6 +505,16 @@ luaT_state(void);
 LUA_API const char *
 luaT_tolstring(lua_State *L, int idx, size_t *ssize);
 
+/**
+ * Check whether a Lua object is a function or has
+ * metatable/metatype with a __call field.
+ *
+ * Note: It does not check type of __call metatable/metatype
+ * field.
+ */
+LUA_API int
+luaL_iscallable(lua_State *L, int idx);
+
 /**
  * Check if a value on @a L stack by index @a idx is an ibuf
  * object. Both 'struct ibuf' and 'struct ibuf *' are accepted.
diff --git a/test/app-tap/module_api.c b/test/app-tap/module_api.c
index 35df2403b..cc3475d60 100644
--- a/test/app-tap/module_api.c
+++ b/test/app-tap/module_api.c
@@ -1895,7 +1895,15 @@ test_tuple_validate_default(lua_State *L)
 		valid = box_tuple_validate(tuple, format) == 0;
 	}
 	lua_pushboolean(L, valid);
+	return 1;
+}
 
+static int
+test_iscallable(lua_State *L)
+{
+	int exp = lua_toboolean(L, 2);
+	int res = luaL_iscallable(L, 1);
+	lua_pushboolean(L, res == exp);
 	return 1;
 }
 
@@ -1954,6 +1962,7 @@ luaopen_module_api(lua_State *L)
 		{"test_cpcall", test_cpcall},
 		{"test_state", test_state},
 		{"test_tostring", test_tostring},
+		{"iscallable", test_iscallable},
 		{"iscdata", test_iscdata},
 		{"test_box_region", test_box_region},
 		{"test_tuple_encode", test_tuple_encode},
diff --git a/test/app-tap/module_api.test.lua b/test/app-tap/module_api.test.lua
index b8c10a310..e5dcea55a 100755
--- a/test/app-tap/module_api.test.lua
+++ b/test/app-tap/module_api.test.lua
@@ -3,7 +3,9 @@
 local fio = require('fio')
 
 box.cfg{log = "tarantool.log"}
-build_path = os.getenv("BUILDDIR")
+-- Use BUILDDIR passed from test-run or cwd when run w/o
+-- test-run to find test/app-tap/module_api.{so,dylib}.
+build_path = os.getenv("BUILDDIR") or '.'
 package.cpath = fio.pathjoin(build_path, 'test/app-tap/?.so'   ) .. ';' ..
                 fio.pathjoin(build_path, 'test/app-tap/?.dylib') .. ';' ..
                 package.cpath
@@ -140,8 +142,86 @@ local function test_iscdata(test, module)
     end
 end
 
+local function test_iscallable(test, module)
+    local ffi = require('ffi')
+
+    ffi.cdef([[
+        struct cdata_1 { int foo; };
+        struct cdata_2 { int foo; };
+    ]])
+
+    local cdata_1 = ffi.new('struct cdata_1')
+    local cdata_1_ref = ffi.new('struct cdata_1 &')
+    local cdata_2 = ffi.new('struct cdata_2')
+    local cdata_2_ref = ffi.new('struct cdata_2 &')
+
+    local nop = function() end
+
+    ffi.metatype('struct cdata_2', {
+        __call = nop,
+    })
+
+    local cases = {
+        {
+            obj = nop,
+            exp = true,
+            description = 'function',
+        },
+        {
+            obj = nil,
+            exp = false,
+            description = 'nil',
+        },
+        {
+            obj = 1,
+            exp = false,
+            description = 'number',
+        },
+        {
+            obj = {},
+            exp = false,
+            description = 'table without metatable',
+        },
+        {
+            obj = setmetatable({}, {}),
+            exp = false,
+            description = 'table without __call metatable field',
+        },
+        {
+            obj = setmetatable({}, {__call = nop}),
+            exp = true,
+            description = 'table with __call metatable field'
+        },
+        {
+            obj = cdata_1,
+            exp = false,
+            description = 'cdata without __call metatable field',
+        },
+        {
+            obj = cdata_1_ref,
+            exp = false,
+            description = 'cdata reference without __call metatable field',
+        },
+        {
+            obj = cdata_2,
+            exp = true,
+            description = 'cdata with __call metatable field',
+        },
+        {
+            obj = cdata_2_ref,
+            exp = true,
+            description = 'cdata reference with __call metatable field',
+        },
+    }
+
+    test:plan(#cases)
+    for _, case in ipairs(cases) do
+        test:ok(module.iscallable(case.obj, case.exp), case.description)
+    end
+end
+
 local test = require('tap').test("module_api", function(test)
-    test:plan(37)
+    test:plan(38)
     local status, module = pcall(require, 'module_api')
     test:is(status, true, "module")
     test:ok(status, "module is loaded")
@@ -167,6 +247,7 @@ local test = require('tap').test("module_api", function(test)
 
     test:test("pushcdata", test_pushcdata, module)
     test:test("iscdata", test_iscdata, module)
+    test:test("iscallable", test_iscallable, module)
     test:test("buffers", test_buffers, module)
     test:test("tuple_validate", test_tuple_validate, module)
 
-- 
2.20.1

Re: [Tarantool-patches] [PATCH 1.10 v4 0/5] module api: extend for external merger Lua module

[Timur Safin]

Forgot to mention the branch 

https://github.com/tarantool/tarantool/tree/tsafin/gh-5273-expand-module-api-1.10-v4

Timur 
: -----Original Message-----
: From: Timur Safin <tsafin@tarantool.org>
: Sent: Wednesday, October 14, 2020 3:16 AM
: To: v.shpilevoy@tarantool.org; alexander.turenko@tarantool.org
: Cc: Timur Safin <tsafin@tarantool.org>; tarantool-
: patches@dev.tarantool.org
: Subject: [PATCH 1.10 v4 0/5] module api: extend for external merger Lua
: module
: 
: Ok we are all tired and need to sleep, but let me reiterate it once again:
: 
: The current patchset is a 1.10 backport of a 2.X series which was in turn
: continuation of patch series which Alexander Turenko has sent before.
: 
: The major difference here - is the way how we export symbols:
: - in 1.10 there was no `src/exports.h` existing, and we were using
:   old good `extra/exports` instead;
: - also we have added 1 extra backport of `luaL_iscallable`.
: 
: Enjoy!
: 
: Changelog of v2 to v1
: ---------------------
: Please do not be surprised to not seen v2 in your inbox - it was cancelled
: after discussion with Alexander Turenko (see v2.1 below with descriptions)
: 
: - Unpublished again `lauT_temp_state`, `luaT_release_temp_state` - they
: are
:   performance wrappers around creation of Lua thread states. Agreed that
:   performance impact not that big, thus created similar compatibility
: wrappers
:   on the module side;
: - unpublished `luaT_newthread`, because for module usage `lua_newthread`
: is
:   enough;
: - unpublished `luaL_register_module` & `luaL_register_type` and reworked
: module
:   code with `luaL_register`.
: 
: Smallish changelog of v2.1 to v2
: --------------------------------
: Alexander Turenko has provided valuable feedback to the v2 branch I've
: shown,
: so we have yet more reduced this patchset
: - Unpublished `luaL_cdata_iscallable`, it's unnecessary for external
: module
:   which is using now public `luaL_iscallable` module api call, instead
:   of compatibility layer in module;
: - Unpublished `luaL_checkconstchar` because it's not yet needed anywhere
:   externally;
: - Accordingly to thsoe changes above we have reduced code in module_api
:   test files.
: 
: Changelog for v3
: ----------------
: - Reshuffled patchset to merge tests with their corresponding code;
: 
: Changelog v3.1
: --------------
: - Added ibuf wrapper, the simpler way (only reserve, and rpos, wpos
: accessors)
: 
: Changelog v4
: ------------
: - final polishing of examples
: - tests for luaT_toibuf, with ibuf consistency checks
: - tests for key_def_dup
: - tests for tuple_validate
: 
: 
: Alexander Turenko (1):
:   module api: add luaL_iscallable with support of cdata metatype
: 
: Timur Safin (4):
:   module api: export box_tuple_validate
:   module api: export box_key_def_dup
:   module api: introduced luaT_toibuf instead of luaL_checkibuf
:   module api: box_ibuf_* wrappers
: 
:  extra/exports                    |   7 ++
:  src/CMakeLists.txt               |   1 +
:  src/box/CMakeLists.txt           |   1 +
:  src/box/ibuf.c                   |  65 ++++++++++++++
:  src/box/ibuf.h                   |  67 +++++++++++++++
:  src/box/key_def.c                |   6 ++
:  src/box/key_def.h                |  10 +++
:  src/box/tuple.c                  |   8 ++
:  src/box/tuple.h                  |  11 +++
:  src/lua/msgpack.c                |   2 +-
:  src/lua/utils.c                  |  45 +++++++++-
:  src/lua/utils.h                  |  32 +++++--
:  test/app-tap/module_api.c        | 140 +++++++++++++++++++++++++++++++
:  test/app-tap/module_api.test.lua | 131 ++++++++++++++++++++++++++++-
:  14 files changed, 517 insertions(+), 9 deletions(-)
:  create mode 100644 src/box/ibuf.c
:  create mode 100644 src/box/ibuf.h
: 
: --
: 2.20.1

Re: [Tarantool-patches] [PATCH 1.10 v4 2/5] module api: export box_key_def_dup

[Alexander Turenko]

> +static int
> +test_key_def_dup(lua_State *L)
> +{
> +	box_key_def_t *key_def = NULL;
> +	box_key_part_def_t part;
> +	box_key_part_def_t *dump = NULL;
> +	uint32_t dump_part_count = 0;
> +
> +	key_part_def_set_nondefault(&part);
> +	key_def = box_key_def_new_v2(&part, 1);
> +	assert(key_def != NULL);
> +	box_key_def_t *key_def_dup = box_key_def_dup(key_def);
> +	assert(key_def_dup != NULL);
> +
> +	dump = box_key_def_dump_parts(key_def_dup, &dump_part_count);
> +	assert(dump != NULL);
> +	assert(dump_part_count == 1);
> +
> +	key_part_def_check_equal(&part, &dump[0]);
> +	key_part_def_check_zeros(&dump[0]);

key_part_def_check_zeros() is to verify how _dump_parts() works per se.
It has no any connection with the duplicated key_def content.

So I would left just key_part_def_check_equal() here.

> +
> +	box_key_def_delete(key_def_dup);
> +	box_key_def_delete(key_def);

The dumped parts are allocated on the box region. I would truncate the
region at the end:

 | size_t region_svp = box_region_used();
 | <...>
 | box_region_truncate(region_svp);

May be ignored in a test, but I would do so just to highlight the proper
usage of the API. If not us, who?

> +
> +	lua_pushboolean(L, 1);
> +	return 1;
> +}
> +