From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 0554C44643C for ; Thu, 5 Nov 2020 18:18:51 +0300 (MSK) Received: by mail-lf1-f66.google.com with SMTP id h6so2838607lfj.3 for ; Thu, 05 Nov 2020 07:18:51 -0800 (PST) From: Cyrill Gorcunov Date: Thu, 5 Nov 2020 18:18:07 +0300 Message-Id: <20201105151808.456573-4-gorcunov@gmail.com> In-Reply-To: <20201105151808.456573-1-gorcunov@gmail.com> References: <20201105151808.456573-1-gorcunov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v10 3/4] box/cbox: implement cbox Lua module List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tml Cc: Vladislav Shpilevoy Currently to run "C" function from some external module one have to register it first in "_func" system space. This is a problem if node is in read-only mode (replica). Still people would like to have a way to run such functions even in ro mode. For this sake we implement "cbox" lua module. Fixes #4692 Signed-off-by: Cyrill Gorcunov @TarantoolBot document Title: cbox module Overview ======== `cbox` module provides a way to create, delete and execute `C` procedures. Unlinke `box.schema.func` functionality this the functions created with `cbox` help are not persistent and live purely in memory. Once a node get turned off they are vanished. An initial purpose for them is to execute them on nodes which are running in read-only mode. Module functions ================ `cbox.func.load([dso.]name) -> obj | nil, err` ---------------------------------------------- Loads a new function with name `name` from module `dso.`. The module name is optional and if not provided implies to be the same as `name`. The `load` call must be paired with `unload` and these calls are accounted. Until coupled `unload` is called the instance is present in memory. Any `load` calls followed by another `load` with same name simply increase a reference to the existing function. Possible errors: - IllegalParams: function name is either not supplied or not a string. - IllegalParams: function name is too long. - IllegalParams: function references limit exceeded. - OutOfMemory: unable to allocate a function. On success a new callable object is returned, otherwise `nil, error` pair. Example: ``` Lua f, err = require('cbox').func.load('func') if not f then print(err) end ``` `cbox.func.unload([dso.]name) -> true | nil, err` ------------------------------------------------- Unload a function with name `[dso.]name`. Since function instances are accounted the function is not unloaded until number of `unload` calls matched to the number of `load` calls. Possible errors: - IllegalParams: function name is either not supplied or not a string. - IllegalParams: the function does not exist. On success `true` is returned, otherwise `nil, error` pair. Example: ``` Lua ok, err = require('cbox').func.unload('func') if not ok then print(err) end ``` `cbox.module.reload(name) -> true | nil, err` --------------------------------------------- Reloads a module with name `name` and all functions which were associated for the module. Each module keeps a list of functions belonging to the module and reload procedure cause the bound function to update their addresses such that function execution will be routed via a new library. Modules are loaded with that named local binding which means that reload of module symbols won't affect the functions which are started execution already, only new calls will be rerouted. Possible errors: - IllegalParams: module name is either not supplied or not a string. - ClientError: a module with the name provided does not exist. On success `true` is returned, otherwise `nil,error` pair. Example: ``` Lua ok, err = require('cbox').module.reload('func') if not ok then print(err) end Executing a loaded function =========================== Once function is loaded it can be executed by ordinary Lua call. Lets consider the following example. We have a `C` function which takes two numbers and returns their sum. ``` C int cfunc_sum(box_function_ctx_t *ctx, const char *args, const char *args_end) { uint32_t arg_count = mp_decode_array(&args); if (arg_count != 2) { return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s", "invalid argument count"); } uint64_t a = mp_decode_uint(&args); uint64_t b = mp_decode_uint(&args); char res[16]; char *end = mp_encode_uint(res, a + b); box_return_mp(ctx, res, end); return 0; } ``` The name of the function is `cfunc_sum` and the function is built into `cfunc.so` shared library. First we should load it as ``` Lua cfunc_sum, err = require('cbox').func.load('cfunc.cfunc_sum') if not cfunc_sum then print(err) end ``` Once successfully loaded we can execute it. Note that unlike regular Lua functions the context of `C` functions is different. They never thrown an exception but return `true|nil, res` form where first value set to `nil` in case of error condition and `res` carries an error description. Lets call the `cfunc_sum` with wrong number of arguments ``` Lua local ok, res = cfunc_sum() if not ok then print(res) end ``` We will the `"invalid argument count"` message in output. The error message has been set by the `box_error_set` in `C` code above. On success the first returned value set to `true` and `res` represent function execution result. ``` Lua local ok, res = cfunc_sum(1, 2) assert(ok); print(res) ``` We will see the number `3` in output. --- src/box/CMakeLists.txt | 1 + src/box/lua/cbox.c | 443 +++++++++++++++++++++++++++++++++++++++++ src/box/lua/cbox.h | 24 +++ src/box/lua/init.c | 2 + 4 files changed, 470 insertions(+) create mode 100644 src/box/lua/cbox.c create mode 100644 src/box/lua/cbox.h diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt index 6b0ba1f58..586cd330f 100644 --- a/src/box/CMakeLists.txt +++ b/src/box/CMakeLists.txt @@ -199,6 +199,7 @@ add_library(box STATIC lua/init.c lua/call.c lua/cfg.cc + lua/cbox.c lua/console.c lua/serialize_lua.c lua/tuple.c diff --git a/src/box/lua/cbox.c b/src/box/lua/cbox.c new file mode 100644 index 000000000..708850d3e --- /dev/null +++ b/src/box/lua/cbox.c @@ -0,0 +1,443 @@ +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file. + */ + +#include +#include + +#include "assoc.h" +#include "diag.h" + +#include "box/module_cache.h" +#include "box/error.h" +#include "box/port.h" + +#include "trivia/util.h" +#include "lua/utils.h" + +/** A type to find a function handle from an object. */ +static const char *cbox_func_handle_uname = "cbox_func_handle"; + +/** + * Function descriptor. + */ +struct cbox_func { + /** + * Symbol descriptor for the function in + * an associated module. + */ + struct module_sym mod_sym; + /** + * Number of loads of the function. + */ + int64_t load_count; + /** + * Function name. + */ + const char *name; + /** + * Function name length. + */ + size_t name_len; + /** + * Function name in-place keeper. + */ + char inplace[0]; +}; + +/** + * Function name to cbox_func hash. + */ +static struct mh_strnptr_t *func_hash = NULL; + +/** + * Find function in cbox_func hash. + */ +struct cbox_func * +cbox_func_find(const char *name, size_t name_len) +{ + mh_int_t e = mh_strnptr_find_inp(func_hash, name, name_len); + if (e == mh_end(func_hash)) + return NULL; + return mh_strnptr_node(func_hash, e)->val; +} + +/** + * Delete a function instance from the hash or decrease + * a reference if the function is still loaded. + */ +static void +cbox_func_del(struct cbox_func *cf) +{ + assert(cf->load_count > 0); + if (cf->load_count-- != 1) + return; + + mh_int_t e = mh_strnptr_find_inp(func_hash, cf->name, cf->name_len); + assert(e != mh_end(func_hash)); + mh_strnptr_del(func_hash, e, NULL); +} + +/** + * Add a function instance into the hash or increase + * a reference if the function is already exist. + */ +static bool +cbox_func_add(struct cbox_func *cf) +{ + assert(cf->load_count >= 0); + if (cf->load_count++ != 0) + return true; + + const struct mh_strnptr_node_t nd = { + .str = cf->name, + .len = cf->name_len, + .hash = mh_strn_hash(cf->name, cf->name_len), + .val = cf, + }; + + mh_int_t e = mh_strnptr_put(func_hash, &nd, NULL, NULL); + if (e == mh_end(func_hash)) { + diag_set(OutOfMemory, sizeof(nd), + "malloc", "cbox_func node"); + return false; + } + return true; +} + +/** + * Allocate a new function instance. + */ +static struct cbox_func * +cbox_func_new(const char *name, size_t name_len) +{ + const ssize_t cf_size = sizeof(struct cbox_func); + size_t size = cf_size + name_len + 1; + struct cbox_func *cf = malloc(size); + if (cf == NULL) { + diag_set(OutOfMemory, size, "malloc", "cf"); + return NULL; + } + + cf->mod_sym.addr = NULL; + cf->mod_sym.module = NULL; + cf->load_count = 0; + cf->mod_sym.name = cf->inplace; + cf->name = cf->inplace; + cf->name_len = name_len; + + memcpy(cf->inplace, name, name_len); + cf->inplace[name_len] = '\0'; + + return cf; +} + +/** + * Load a new function. + * + * This function takes a function name from the caller + * stack @a L and creates a new function object. If + * the function is already loaded we simply return + * a reference to existing one. + * + * Possible errors: + * + * - IllegalParams: function name is either not supplied + * or not a string. + * - IllegalParams: function references limit exceeded. + * - OutOfMemory: unable to allocate a function. + * + * @returns function object on success or {nil,error} on error, + * the error is set to the diagnostics area. + */ +static int +lcbox_func_load(struct lua_State *L) +{ + const char *method = "cbox.func.load"; + struct cbox_func *cf = NULL; + + if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) { + const char *fmt = + "Expects %s(\'name\') but no name passed"; + diag_set(IllegalParams, fmt, method); + return luaT_push_nil_and_error(L); + } + + size_t name_len; + const char *name = lua_tolstring(L, 1, &name_len); + + cf = cbox_func_find(name, name_len); + if (cf == NULL) { + cf = cbox_func_new(name, name_len); + if (cf == NULL) + return luaT_push_nil_and_error(L); + } + if (!cbox_func_add(cf)) + return luaT_push_nil_and_error(L); + + *(struct cbox_func **)lua_newuserdata(L, sizeof(cf)) = cf; + luaL_getmetatable(L, cbox_func_handle_uname); + lua_setmetatable(L, -2); + return 1; +} + +/** + * Unload a function. + * + * This function takes a function name from the caller + * stack @a L and unloads a function object. + * + * Possible errors: + * + * - IllegalParams: function name is either not supplied + * or not a string. + * - IllegalParams: the function does not exist. + * + * @returns true on success or {nil,error} on error, + * the error is set to the diagnostics area. + */ +static int +lcbox_func_unload(struct lua_State *L) +{ + const char *method = "cbox.func.unload"; + const char *name = NULL; + + if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) { + const char *fmt = + "Expects %s(\'name\') but no name passed"; + diag_set(IllegalParams, fmt, method); + return luaT_push_nil_and_error(L); + } + + size_t name_len; + name = lua_tolstring(L, 1, &name_len); + + struct cbox_func *cf = cbox_func_find(name, name_len); + if (cf == NULL) { + const char *fmt = tnt_errcode_desc(ER_NO_SUCH_FUNCTION); + diag_set(IllegalParams, fmt, name); + return luaT_push_nil_and_error(L); + } + + cbox_func_del(cf); + lua_pushboolean(L, true); + return 1; +} + +/** + * Reload a module. + * + * This function takes a module name from the caller + * stack @a L and reloads all functions associated with + * the module. + * + * Possible errors: + * + * - IllegalParams: module name is either not supplied + * or not a string. + * - IllegalParams: the function does not exist. + * - ClientError: a module with the name provided does + * not exist. + * + * @returns true on success or {nil,error} on error, + * the error is set to the diagnostics area. + */ +static int +lcbox_module_reload(struct lua_State *L) +{ + const char *method = "cbox.module.reload"; + const char *fmt = "Expects %s(\'name\') but no name passed"; + + if (lua_gettop(L) != 1 || !lua_isstring(L, 1)) { + diag_set(IllegalParams, fmt, method); + return luaT_push_nil_and_error(L); + } + + size_t name_len; + const char *name = lua_tolstring(L, 1, &name_len); + if (name == NULL || name_len < 1) { + diag_set(IllegalParams, fmt, method); + return luaT_push_nil_and_error(L); + } + + struct module *module = NULL; + if (module_reload(name, &name[name_len], &module) == 0) { + if (module != NULL) { + lua_pushboolean(L, true); + return 1; + } + diag_set(ClientError, ER_NO_SUCH_MODULE, name); + } + return luaT_push_nil_and_error(L); +} + +/** + * Fetch cbox_func instance from an object. + */ +static struct cbox_func * +cbox_fetch_func_handle(struct lua_State *L) +{ + struct cbox_func **cf_ptr = luaL_testudata(L, 1, cbox_func_handle_uname); + if (cf_ptr != NULL) { + assert(*cf_ptr != NULL); + return *cf_ptr; + } + return NULL; +} + +/** + * Function handle representation for REPL (console). + */ +static int +lcbox_handle_serialize(struct lua_State *L) +{ + struct cbox_func *cf = cbox_fetch_func_handle(L); + if (cf == NULL) { + diag_set(IllegalParams, "Bad params, use __serialize(obj)"); + return luaT_error(L); + } + + lua_createtable(L, 0, 0); + lua_pushstring(L, cf->name); + lua_setfield(L, -2, "name"); + + return 1; +} + +/** + * Handle __index request for a function object. + */ +static int +lcbox_handle_index(struct lua_State *L) +{ + /* + * Instead of showing userdata pointer + * lets provide a serialized value. + */ + lua_getmetatable(L, 1); + lua_pushvalue(L, 2); + lua_rawget(L, -2); + if (!lua_isnil(L, -1)) + return 1; + + struct cbox_func *cf = cbox_fetch_func_handle(L); + size_t len = 0; + const char *key = lua_tolstring(L, 2, &len); + + if (lua_type(L, 2) != LUA_TSTRING || cf == NULL || key == NULL) { + diag_set(IllegalParams, + "Bad params, use __index(obj, )"); + return luaT_error(L); + } + + if (strcmp(key, "name") == 0) { + lua_pushstring(L, cf->name); + return 1; + } + + return 0; +} + +/** + * Free function handle if there is no active loads left. + */ +static int +lcbox_handle_gc(struct lua_State *L) +{ + struct cbox_func *cf = cbox_fetch_func_handle(L); + if (cf != NULL && cf->load_count == 0) { + TRASH(cf); + free(cf); + } + return 0; +} + +/** + * Call a function by its name from the Lua code. + */ +static int +lcbox_handle_call(struct lua_State *L) +{ + struct cbox_func *cf = cbox_fetch_func_handle(L); + if (cf == NULL) { + diag_set(IllegalParams, "Function is corrupted"); + return luaT_push_nil_and_error(L); + } + + /* + * FIXME: We should get rid of luaT_newthread but this + * requires serious modifications. In particular + * port_lua_do_dump uses tarantool_L reference and + * coro_ref must be valid as well. + */ + lua_State *args_L = luaT_newthread(tarantool_L); + if (args_L == NULL) + return luaT_push_nil_and_error(L); + + int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX); + lua_xmove(L, args_L, lua_gettop(L) - 1); + + struct port args; + port_lua_create(&args, args_L); + ((struct port_lua *)&args)->ref = coro_ref; + + struct port ret; + if (module_sym_call(&cf->mod_sym, &args, &ret) != 0) { + port_destroy(&args); + return luaT_push_nil_and_error(L); + } + + int top = lua_gettop(L); + lua_pushboolean(L, true); + port_dump_lua(&ret, L, true); + int cnt = lua_gettop(L) - top; + + port_destroy(&ret); + port_destroy(&args); + + return cnt; +} + +/** + * Initialize cbox module. + */ +void +box_lua_cbox_init(struct lua_State *L) +{ + func_hash = mh_strnptr_new(); + if (func_hash == NULL) { + panic("Can't allocate cbox hash table"); + } + + static const struct luaL_Reg cbox_methods[] = { + { NULL, NULL }, + }; + luaL_register_module(L, "cbox", cbox_methods); + lua_pop(L, 1); + + static const struct luaL_Reg func_methods[] = { + { "load", lcbox_func_load }, + { "unload", lcbox_func_unload }, + { NULL, NULL }, + }; + luaL_register_module(L, "cbox.func", func_methods); + lua_pop(L, 1); + + static const struct luaL_Reg module_methods[] = { + { "reload", lcbox_module_reload }, + { NULL, NULL }, + }; + luaL_register_module(L, "cbox.module", module_methods); + lua_pop(L, 1); + + static const struct luaL_Reg func_handle_methods[] = { + { "__index", lcbox_handle_index }, + { "__serialize", lcbox_handle_serialize }, + { "__call", lcbox_handle_call }, + { "__gc", lcbox_handle_gc }, + { NULL, NULL }, + }; + luaL_register_type(L, cbox_func_handle_uname, func_handle_methods); +} diff --git a/src/box/lua/cbox.h b/src/box/lua/cbox.h new file mode 100644 index 000000000..17955fb44 --- /dev/null +++ b/src/box/lua/cbox.h @@ -0,0 +1,24 @@ +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file. + */ + +#pragma once + +#if defined(__cplusplus) +extern "C" { +#endif /* defined(__cplusplus) */ + +struct lua_State; + +/** + * Initialize cbox Lua module. + * + * @param L Lua state where to register the cbox module. + */ +void +box_lua_cbox_init(struct lua_State *L); +#if defined(__cplusplus) +} +#endif /* defined(__plusplus) */ diff --git a/src/box/lua/init.c b/src/box/lua/init.c index d0316ef86..b37aa284a 100644 --- a/src/box/lua/init.c +++ b/src/box/lua/init.c @@ -61,6 +61,7 @@ #include "box/lua/cfg.h" #include "box/lua/xlog.h" #include "box/lua/console.h" +#include "box/lua/cbox.h" #include "box/lua/tuple.h" #include "box/lua/execute.h" #include "box/lua/key_def.h" @@ -466,6 +467,7 @@ box_lua_init(struct lua_State *L) box_lua_tuple_init(L); box_lua_call_init(L); box_lua_cfg_init(L); + box_lua_cbox_init(L); box_lua_slab_init(L); box_lua_index_init(L); box_lua_space_init(L); -- 2.26.2