[Tarantool-patches] [PATCH v10 3/4] box/cbox: implement cbox Lua module

[Cyrill Gorcunov <gorcunov@gmail.com>]

Currently to run "C" function from some external module
one have to register it first in "_func" system space. This
is a problem if node is in read-only mode (replica).

Still people would like to have a way to run such functions
even in ro mode. For this sake we implement "cbox" lua module.

Fixes #4692

Signed-off-by: Cyrill Gorcunov <gorcunov@gmail.com>

@TarantoolBot document
Title: cbox module

Overview
========

`cbox` module provides a way to create, delete and execute
`C` procedures. Unlinke `box.schema.func` functionality this
the functions created with `cbox` help are not persistent and
live purely in memory. Once a node get turned off they are
vanished. An initial purpose for them is to execute them on
nodes which are running in read-only mode.

Module functions
================

`cbox.func.load([dso.]name) -> obj | nil, err`
----------------------------------------------

Loads a new function with name `name` from module `dso.`.
The module name is optional and if not provided implies
to be the same as `name`.

The `load` call must be paired with `unload` and these
calls are accounted. Until coupled `unload` is called
the instance is present in memory. Any `load` calls
followed by another `load` with same name simply
increase a reference to the existing function.

Possible errors:
 - IllegalParams: function name is either not supplied
   or not a string.
 - IllegalParams: function name is too long.
 - IllegalParams: function references limit exceeded.
 - OutOfMemory: unable to allocate a function.

On success a new callable object is returned,
otherwise `nil, error` pair.

Example:

``` Lua
f, err = require('cbox').func.load('func')
if not f then
    print(err)
end
```

`cbox.func.unload([dso.]name) -> true | nil, err`
-------------------------------------------------

Unload a function with name `[dso.]name`. Since function
instances are accounted the function is not unloaded until
number of `unload` calls matched to the number of `load`
calls.

Possible errors:
 - IllegalParams: function name is either not supplied
   or not a string.
 - IllegalParams: the function does not exist.

On success `true` is returned, otherwise `nil, error` pair.

Example:

``` Lua
ok, err = require('cbox').func.unload('func')
if not ok then
    print(err)
end
```

`cbox.module.reload(name) -> true | nil, err`
---------------------------------------------

Reloads a module with name `name` and all functions which
were associated for the module. Each module keeps a list of
functions belonging to the module and reload procedure cause
the bound function to update their addresses such that
function execution will be routed via a new library.

Modules are loaded with that named local binding which means
that reload of module symbols won't affect the functions which
are started execution already, only new calls will be rerouted.

Possible errors:
 - IllegalParams: module name is either not supplied
   or not a string.
 - ClientError: a module with the name provided does
   not exist.

On success `true` is returned, otherwise `nil,error` pair.

Example:

``` Lua
ok, err = require('cbox').module.reload('func')
if not ok then
    print(err)
end

Executing a loaded function
===========================

Once function is loaded it can be executed by ordinary Lua call.
Lets consider the following example. We have a `C` function which
takes two numbers and returns their sum.

``` C
int
cfunc_sum(box_function_ctx_t *ctx, const char *args, const char *args_end)
{
	uint32_t arg_count = mp_decode_array(&args);
	if (arg_count != 2) {
		return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s",
			"invalid argument count");
	}
	uint64_t a = mp_decode_uint(&args);
	uint64_t b = mp_decode_uint(&args);

	char res[16];
	char *end = mp_encode_uint(res, a + b);
	box_return_mp(ctx, res, end);
	return 0;
}
```

The name of the function is `cfunc_sum` and the function is built into
`cfunc.so` shared library.

First we should load it as

``` Lua
cfunc_sum, err = require('cbox').func.load('cfunc.cfunc_sum')
if not cfunc_sum then
    print(err)
end
```

Once successfully loaded we can execute it. Note that unlike regular
Lua functions the context of `C` functions is different. They never
thrown an exception but return `true|nil, res` form where first value set
to `nil` in case of error condition and `res` carries an error description.

Lets call the `cfunc_sum` with wrong number of arguments

``` Lua
local ok, res = cfunc_sum()
if not ok then
    print(res)
end
```

We will the `"invalid argument count"` message in output.
The error message has been set by the `box_error_set` in `C`
code above.

On success the first returned value set to `true` and `res` represent
function execution result.

``` Lua
local ok, res = cfunc_sum(1, 2)
assert(ok);
print(res)
```

We will see the number `3` in output.
---
 src/box/CMakeLists.txt |   1 +
 src/box/lua/cbox.c     | 443 +++++++++++++++++++++++++++++++++++++++++
 src/box/lua/cbox.h     |  24 +++
 src/box/lua/init.c     |   2 +
 4 files changed, 470 insertions(+)
 create mode 100644 src/box/lua/cbox.c
 create mode 100644 src/box/lua/cbox.h

diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
index 6b0ba1f58..586cd330f 100644
--- a/src/box/CMakeLists.txt
+++ b/src/box/CMakeLists.txt
@@ -199,6 +199,7 @@ add_library(box STATIC
     lua/init.c
     lua/call.c
     lua/cfg.cc
+    lua/cbox.c
     lua/console.c
     lua/serialize_lua.c
     lua/tuple.c
diff --git a/src/box/lua/cbox.c b/src/box/lua/cbox.c
new file mode 100644
index 000000000..708850d3e
--- /dev/null
+++ b/src/box/lua/cbox.c
@@ -0,0 +1,443 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file.
+ */
+
+#include <string.h>
+#include <lua.h>
+
+#include "assoc.h"
+#include "diag.h"
+
+#include "box/module_cache.h"
+#include "box/error.h"
+#include "box/port.h"
+
+#include "trivia/util.h"
+#include "lua/utils.h"
+
+/** A type to find a function handle from an object. */
+static const char *cbox_func_handle_uname = "cbox_func_handle";
+
+/**
+ * Function descriptor.
+ */
+struct cbox_func {
+	/**
+	 * Symbol descriptor for the function in
+	 * an associated module.
+	 */
+	struct module_sym mod_sym;
+	/**
+	 * Number of loads of the function.
+	 */
+	int64_t load_count;
+	/**
+	 * Function name.
+	 */
+	const char *name;
+	/**
+	 * Function name length.
+	 */
+	size_t name_len;
+	/**
+	 * Function name in-place keeper.
+	 */
+	char inplace[0];
+};
+
+/**
+ * Function name to cbox_func hash.
+ */
+static struct mh_strnptr_t *func_hash = NULL;
+
+/**
+ * Find function in cbox_func hash.
+ */
+struct cbox_func *
+cbox_func_find(const char *name, size_t name_len)
+{
+	mh_int_t e = mh_strnptr_find_inp(func_hash, name, name_len);
+	if (e == mh_end(func_hash))
+		return NULL;
+	return mh_strnptr_node(func_hash, e)->val;
+}
+
+/**
+ * Delete a function instance from the hash or decrease
+ * a reference if the function is still loaded.
+ */
+static void
+cbox_func_del(struct cbox_func *cf)
+{
+	assert(cf->load_count > 0);
+	if (cf->load_count-- != 1)
+		return;
+
+	mh_int_t e = mh_strnptr_find_inp(func_hash, cf->name, cf->name_len);
+	assert(e != mh_end(func_hash));
+	mh_strnptr_del(func_hash, e, NULL);
+}
+
+/**
+ * Add a function instance into the hash or increase
+ * a reference if the function is already exist.
+ */
+static bool
+cbox_func_add(struct cbox_func *cf)
+{
+	assert(cf->load_count >= 0);
+	if (cf->load_count++ != 0)
+		return true;
+
+	const struct mh_strnptr_node_t nd = {
+		.str	= cf->name,
+		.len	= cf->name_len,
+		.hash	= mh_strn_hash(cf->name, cf->name_len),
+		.val	= cf,
+	};
+
+	mh_int_t e = mh_strnptr_put(func_hash, &nd, NULL, NULL);
+	if (e == mh_end(func_hash)) {
+		diag_set(OutOfMemory, sizeof(nd),
+			 "malloc", "cbox_func node");
+		return false;
+	}
+	return true;
+}
+
+/**
+ * Allocate a new function instance.
+ */
+static struct cbox_func *
+cbox_func_new(const char *name, size_t name_len)
+{
+	const ssize_t cf_size = sizeof(struct cbox_func);
+	size_t size = cf_size + name_len + 1;
+	struct cbox_func *cf = malloc(size);
+	if (cf == NULL) {
+		diag_set(OutOfMemory, size, "malloc", "cf");
+		return NULL;
+	}
+
+	cf->mod_sym.addr	= NULL;
+	cf->mod_sym.module	= NULL;
+	cf->load_count		= 0;
+	cf->mod_sym.name	= cf->inplace;
+	cf->name		= cf->inplace;
+	cf->name_len		= name_len;
+
+	memcpy(cf->inplace, name, name_len);
+	cf->inplace[name_len] = '\0';
+
+	return cf;
+}
+
+/**
+ * Load a new function.
+ *
+ * This function takes a function name from the caller
+ * stack @a L and creates a new function object. If
+ * the function is already loaded we simply return
+ * a reference to existing one.
+ *
+ * Possible errors:
+ *
+ * - IllegalParams: function name is either not supplied
+ *   or not a string.
+ * - IllegalParams: function references limit exceeded.
+ * - OutOfMemory: unable to allocate a function.
+ *
+ * @returns function object on success or {nil,error} on error,
+ * the error is set to the diagnostics area.
+ */
+static int
+lcbox_func_load(struct lua_State *L)
+{
+	const char *method = "cbox.func.load";
+	struct cbox_func *cf = NULL;
+
+	if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) {
+		const char *fmt =
+			"Expects %s(\'name\') but no name passed";
+		diag_set(IllegalParams, fmt, method);
+		return luaT_push_nil_and_error(L);
+	}
+
+	size_t name_len;
+	const char *name = lua_tolstring(L, 1, &name_len);
+
+	cf = cbox_func_find(name, name_len);
+	if (cf == NULL) {
+		cf = cbox_func_new(name, name_len);
+		if (cf == NULL)
+			return luaT_push_nil_and_error(L);
+	}
+	if (!cbox_func_add(cf))
+		return luaT_push_nil_and_error(L);
+
+	*(struct cbox_func **)lua_newuserdata(L, sizeof(cf)) = cf;
+	luaL_getmetatable(L, cbox_func_handle_uname);
+	lua_setmetatable(L, -2);
+	return 1;
+}
+
+/**
+ * Unload a function.
+ *
+ * This function takes a function name from the caller
+ * stack @a L and unloads a function object.
+ *
+ * Possible errors:
+ *
+ * - IllegalParams: function name is either not supplied
+ *   or not a string.
+ * - IllegalParams: the function does not exist.
+ *
+ * @returns true on success or {nil,error} on error,
+ * the error is set to the diagnostics area.
+ */
+static int
+lcbox_func_unload(struct lua_State *L)
+{
+	const char *method = "cbox.func.unload";
+	const char *name = NULL;
+
+	if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) {
+		const char *fmt =
+			"Expects %s(\'name\') but no name passed";
+		diag_set(IllegalParams, fmt, method);
+		return luaT_push_nil_and_error(L);
+	}
+
+	size_t name_len;
+	name = lua_tolstring(L, 1, &name_len);
+
+	struct cbox_func *cf = cbox_func_find(name, name_len);
+	if (cf == NULL) {
+		const char *fmt = tnt_errcode_desc(ER_NO_SUCH_FUNCTION);
+		diag_set(IllegalParams, fmt, name);
+		return luaT_push_nil_and_error(L);
+	}
+
+	cbox_func_del(cf);
+	lua_pushboolean(L, true);
+	return 1;
+}
+
+/**
+ * Reload a module.
+ *
+ * This function takes a module name from the caller
+ * stack @a L and reloads all functions associated with
+ * the module.
+ *
+ * Possible errors:
+ *
+ * - IllegalParams: module name is either not supplied
+ *   or not a string.
+ * - IllegalParams: the function does not exist.
+ * - ClientError: a module with the name provided does
+ *   not exist.
+ *
+ * @returns true on success or {nil,error} on error,
+ * the error is set to the diagnostics area.
+ */
+static int
+lcbox_module_reload(struct lua_State *L)
+{
+	const char *method = "cbox.module.reload";
+	const char *fmt = "Expects %s(\'name\') but no name passed";
+
+	if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) {
+		diag_set(IllegalParams, fmt, method);
+		return luaT_push_nil_and_error(L);
+	}
+
+	size_t name_len;
+	const char *name = lua_tolstring(L, 1, &name_len);
+	if (name == NULL || name_len < 1) {
+		diag_set(IllegalParams, fmt, method);
+		return luaT_push_nil_and_error(L);
+	}
+
+	struct module *module = NULL;
+	if (module_reload(name, &name[name_len], &module) == 0) {
+		if (module != NULL) {
+			lua_pushboolean(L, true);
+			return 1;
+		}
+		diag_set(ClientError, ER_NO_SUCH_MODULE, name);
+	}
+	return luaT_push_nil_and_error(L);
+}
+
+/**
+ * Fetch cbox_func instance from an object.
+ */
+static struct cbox_func *
+cbox_fetch_func_handle(struct lua_State *L)
+{
+	struct cbox_func **cf_ptr = luaL_testudata(L, 1, cbox_func_handle_uname);
+	if (cf_ptr != NULL) {
+		assert(*cf_ptr != NULL);
+		return *cf_ptr;
+	}
+	return NULL;
+}
+
+/**
+ * Function handle representation for REPL (console).
+ */
+static int
+lcbox_handle_serialize(struct lua_State *L)
+{
+	struct cbox_func *cf = cbox_fetch_func_handle(L);
+	if (cf == NULL) {
+		diag_set(IllegalParams, "Bad params, use __serialize(obj)");
+		return luaT_error(L);
+	}
+
+	lua_createtable(L, 0, 0);
+	lua_pushstring(L, cf->name);
+	lua_setfield(L, -2, "name");
+
+	return 1;
+}
+
+/**
+ * Handle __index request for a function object.
+ */
+static int
+lcbox_handle_index(struct lua_State *L)
+{
+	/*
+	 * Instead of showing userdata pointer
+	 * lets provide a serialized value.
+	 */
+	lua_getmetatable(L, 1);
+	lua_pushvalue(L, 2);
+	lua_rawget(L, -2);
+	if (!lua_isnil(L, -1))
+		return 1;
+
+	struct cbox_func *cf = cbox_fetch_func_handle(L);
+	size_t len = 0;
+	const char *key = lua_tolstring(L, 2, &len);
+
+	if (lua_type(L, 2) != LUA_TSTRING || cf == NULL || key == NULL) {
+		diag_set(IllegalParams,
+			 "Bad params, use __index(obj, <string>)");
+		return luaT_error(L);
+	}
+
+	if (strcmp(key, "name") == 0) {
+		lua_pushstring(L, cf->name);
+		return 1;
+	}
+
+	return 0;
+}
+
+/**
+ * Free function handle if there is no active loads left.
+ */
+static int
+lcbox_handle_gc(struct lua_State *L)
+{
+	struct cbox_func *cf = cbox_fetch_func_handle(L);
+	if (cf != NULL && cf->load_count == 0) {
+		TRASH(cf);
+		free(cf);
+	}
+	return 0;
+}
+
+/**
+ * Call a function by its name from the Lua code.
+ */
+static int
+lcbox_handle_call(struct lua_State *L)
+{
+	struct cbox_func *cf = cbox_fetch_func_handle(L);
+	if (cf == NULL) {
+		diag_set(IllegalParams, "Function is corrupted");
+		return luaT_push_nil_and_error(L);
+	}
+
+	/*
+	 * FIXME: We should get rid of luaT_newthread but this
+	 * requires serious modifications. In particular
+	 * port_lua_do_dump uses tarantool_L reference and
+	 * coro_ref must be valid as well.
+	 */
+	lua_State *args_L = luaT_newthread(tarantool_L);
+	if (args_L == NULL)
+		return luaT_push_nil_and_error(L);
+
+	int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+	lua_xmove(L, args_L, lua_gettop(L) - 1);
+
+	struct port args;
+	port_lua_create(&args, args_L);
+	((struct port_lua *)&args)->ref = coro_ref;
+
+	struct port ret;
+	if (module_sym_call(&cf->mod_sym, &args, &ret) != 0) {
+		port_destroy(&args);
+		return luaT_push_nil_and_error(L);
+	}
+
+	int top = lua_gettop(L);
+	lua_pushboolean(L, true);
+	port_dump_lua(&ret, L, true);
+	int cnt = lua_gettop(L) - top;
+
+	port_destroy(&ret);
+	port_destroy(&args);
+
+	return cnt;
+}
+
+/**
+ * Initialize cbox module.
+ */
+void
+box_lua_cbox_init(struct lua_State *L)
+{
+	func_hash = mh_strnptr_new();
+	if (func_hash == NULL) {
+		panic("Can't allocate cbox hash table");
+	}
+
+	static const struct luaL_Reg cbox_methods[] = {
+		{ NULL, NULL },
+	};
+	luaL_register_module(L, "cbox", cbox_methods);
+	lua_pop(L, 1);
+
+	static const struct luaL_Reg func_methods[] = {
+		{ "load",		lcbox_func_load		},
+		{ "unload",		lcbox_func_unload	},
+		{ NULL, NULL },
+	};
+	luaL_register_module(L, "cbox.func", func_methods);
+	lua_pop(L, 1);
+
+	static const struct luaL_Reg module_methods[] = {
+		{ "reload",		lcbox_module_reload	},
+		{ NULL, NULL },
+	};
+	luaL_register_module(L, "cbox.module", module_methods);
+	lua_pop(L, 1);
+
+	static const struct luaL_Reg func_handle_methods[] = {
+		{ "__index",		lcbox_handle_index	},
+		{ "__serialize",	lcbox_handle_serialize	},
+		{ "__call",		lcbox_handle_call	},
+		{ "__gc",		lcbox_handle_gc		},
+		{ NULL, NULL },
+	};
+	luaL_register_type(L, cbox_func_handle_uname, func_handle_methods);
+}
diff --git a/src/box/lua/cbox.h b/src/box/lua/cbox.h
new file mode 100644
index 000000000..17955fb44
--- /dev/null
+++ b/src/box/lua/cbox.h
@@ -0,0 +1,24 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file.
+ */
+
+#pragma once
+
+#if defined(__cplusplus)
+extern "C" {
+#endif /* defined(__cplusplus) */
+
+struct lua_State;
+
+/**
+ * Initialize cbox Lua module.
+ *
+ * @param L Lua state where to register the cbox module.
+ */
+void
+box_lua_cbox_init(struct lua_State *L);
+#if defined(__cplusplus)
+}
+#endif /* defined(__plusplus) */
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index d0316ef86..b37aa284a 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -61,6 +61,7 @@
 #include "box/lua/cfg.h"
 #include "box/lua/xlog.h"
 #include "box/lua/console.h"
+#include "box/lua/cbox.h"
 #include "box/lua/tuple.h"
 #include "box/lua/execute.h"
 #include "box/lua/key_def.h"
@@ -466,6 +467,7 @@ box_lua_init(struct lua_State *L)
 	box_lua_tuple_init(L);
 	box_lua_call_init(L);
 	box_lua_cfg_init(L);
+	box_lua_cbox_init(L);
 	box_lua_slab_init(L);
 	box_lua_index_init(L);
 	box_lua_space_init(L);
-- 
2.26.2