From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 93D97686CA; Fri, 5 Feb 2021 21:55:40 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 93D97686CA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1612551340; bh=eJOfUuubY5XXKRASKVqVfIcEnvJS4YupO6j2vn6Qhgs=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=a1ds50gUpZgYiGioQYCnsLHw2mZqEwPAjmbMmwhBMPDApZsVepe0Tu9wObuWLHSCJ TtNZZ8Rz4IlCefvlEV6jWeBcuUddSS/JkaEIJkc0QP3dYZkMkd2uymJaaaddD3G95K jEQ7vnRwEDMpa1aE2fKmflAx+guaQHQmVBJdd2ow= Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CFBE3686CA for ; Fri, 5 Feb 2021 21:55:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CFBE3686CA Received: by mail-lj1-f176.google.com with SMTP id f19so8990513ljn.5 for ; Fri, 05 Feb 2021 10:55:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kI0VRT6PPpeD3Le29e82waM9LibKhlOOwXIRflEY0+M=; b=eKziiEQ5v4UePhmcGF9DlWSqs9CFTGyrfs150HrST0XmLI0ORTdsGv5Hxg7U1LhgGc dLWB24TVkNKkW1oRvnm9x/hda78zAm5JQ/nUplPVRcMPg40AtroThY/B67n8EMCb+eh0 X1tqmm8gdK9HtkBAS2lcbwb8Z37asRQYs6C5gRabEWAKes83N6M08xj0/r90i0kA4/g6 ///1dB3Ql3Bm9Wt4t0VEHNUYPimXKeFufBQsAPf++jiqCknwL/jP8U5f8R8Zvlhu38r9 IHW6kCfn8lWfIEGj/Y8dxqRQHQxSqMz4/4N7nS+R21I+Vex5+1e6Mn/QVy+lFuKMnNgp Srcw== X-Gm-Message-State: AOAM531jOJSqLz4CydhRdwhhxOoKTa+O4XqxLr7FliYKf9HaxKqrnkzD N6oMGcfNV/3ZZiHhVZocWu0= X-Google-Smtp-Source: ABdhPJwpUpYgP2m1KgRRK0eg3chEk/nuMzT1A0CmtHsCLUUjIlvfXvQ+fVmu4DgmslkUlx/V4cmjcQ== X-Received: by 2002:a2e:a201:: with SMTP id h1mr3475079ljm.130.1612551303043; Fri, 05 Feb 2021 10:55:03 -0800 (PST) Received: from grain.localdomain ([5.18.103.226]) by smtp.gmail.com with ESMTPSA id i78sm1069308lfi.263.2021.02.05.10.55.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:55:01 -0800 (PST) Received: by grain.localdomain (Postfix, from userid 1000) id 7CC70560271; Fri, 5 Feb 2021 21:54:37 +0300 (MSK) To: tml Date: Fri, 5 Feb 2021 21:54:27 +0300 Message-Id: <20210205185436.638894-3-gorcunov@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210205185436.638894-1-gorcunov@gmail.com> References: <20210205185436.638894-1-gorcunov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v15 02/11] module_cache: move module handling into own subsystem X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Cyrill Gorcunov via Tarantool-patches Reply-To: Cyrill Gorcunov Cc: Mons Anderson , Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" The module handling should not be bound to particular function implementation (we will have two users: already existing functions for "_func" space, and a new upcoming one which will be serving functions on read only instances). For this sake all module related code is moved to module_cache file where we do symbol resolving, calling and tracking of symbol usage. Part-of #4642 Signed-off-by: Cyrill Gorcunov --- src/box/CMakeLists.txt | 1 + src/box/func.c | 473 +----------------------------------- src/box/func.h | 83 +------ src/box/func_def.h | 14 -- src/box/module_cache.c | 535 +++++++++++++++++++++++++++++++++++++++++ src/box/module_cache.h | 139 +++++++++++ 6 files changed, 679 insertions(+), 566 deletions(-) create mode 100644 src/box/module_cache.c create mode 100644 src/box/module_cache.h diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt index 19203f770..339e2c8a9 100644 --- a/src/box/CMakeLists.txt +++ b/src/box/CMakeLists.txt @@ -187,6 +187,7 @@ add_library(box STATIC sql_stmt_cache.c wal.c call.c + module_cache.c merger.c ibuf.c ${sql_sources} diff --git a/src/box/func.c b/src/box/func.c index 8030aa07c..4e7025fdb 100644 --- a/src/box/func.c +++ b/src/box/func.c @@ -31,30 +31,12 @@ #include "func.h" #include "fiber.h" #include "trivia/config.h" -#include "assoc.h" #include "lua/utils.h" #include "lua/call.h" -#include "error.h" -#include "errinj.h" #include "diag.h" #include "port.h" #include "schema.h" #include "session.h" -#include "libeio/eio.h" -#include -#include - -/** - * Parsed symbol and package names. - */ -struct func_name { - /** Null-terminated symbol name, e.g. "func" for "mod.submod.func" */ - const char *sym; - /** Package name, e.g. "mod.submod" for "mod.submod.func" */ - const char *package; - /** A pointer to the last character in ->package + 1 */ - const char *package_end; -}; struct func_c { /** Function object base class. */ @@ -63,427 +45,6 @@ struct func_c { struct module_sym mod_sym; }; -/*** - * Split function name to symbol and package names. - * For example, str = foo.bar.baz => sym = baz, package = foo.bar - * @param str function name, e.g. "module.submodule.function". - * @param[out] name parsed symbol and package names. - */ -static void -func_split_name(const char *str, struct func_name *name) -{ - name->package = str; - name->package_end = strrchr(str, '.'); - if (name->package_end != NULL) { - /* module.submodule.function => module.submodule, function */ - name->sym = name->package_end + 1; /* skip '.' */ - } else { - /* package == function => function, function */ - name->sym = name->package; - name->package_end = str + strlen(str); - } -} - -/** - * Arguments for luaT_module_find used by lua_cpcall() - */ -struct module_find_ctx { - const char *package; - const char *package_end; - char *path; - size_t path_len; -}; - -/** - * A cpcall() helper for module_find() - */ -static int -luaT_module_find(lua_State *L) -{ - struct module_find_ctx *ctx = (struct module_find_ctx *) - lua_topointer(L, 1); - - /* - * Call package.searchpath(name, package.cpath) and use - * the path to the function in dlopen(). - */ - lua_getglobal(L, "package"); - - lua_getfield(L, -1, "search"); - - /* Argument of search: name */ - lua_pushlstring(L, ctx->package, ctx->package_end - ctx->package); - - lua_call(L, 1, 1); - if (lua_isnil(L, -1)) - return luaL_error(L, "module not found"); - /* Convert path to absolute */ - char resolved[PATH_MAX]; - if (realpath(lua_tostring(L, -1), resolved) == NULL) { - diag_set(SystemError, "realpath"); - return luaT_error(L); - } - - snprintf(ctx->path, ctx->path_len, "%s", resolved); - return 0; -} - -/** - * Find path to module using Lua's package.cpath - * @param package package name - * @param package_end a pointer to the last byte in @a package + 1 - * @param[out] path path to shared library - * @param path_len size of @a path buffer - * @retval 0 on success - * @retval -1 on error, diag is set - */ -static int -module_find(const char *package, const char *package_end, char *path, - size_t path_len) -{ - struct module_find_ctx ctx = { package, package_end, path, path_len }; - lua_State *L = tarantool_L; - int top = lua_gettop(L); - if (luaT_cpcall(L, luaT_module_find, &ctx) != 0) { - int package_len = (int) (package_end - package); - diag_set(ClientError, ER_LOAD_MODULE, package_len, package, - lua_tostring(L, -1)); - lua_settop(L, top); - return -1; - } - assert(top == lua_gettop(L)); /* cpcall discard results */ - return 0; -} - -static struct mh_strnptr_t *modules = NULL; - -static void -module_gc(struct module *module); - -int -module_init(void) -{ - modules = mh_strnptr_new(); - if (modules == NULL) { - diag_set(OutOfMemory, sizeof(*modules), "malloc", - "modules hash table"); - return -1; - } - return 0; -} - -void -module_free(void) -{ - while (mh_size(modules) > 0) { - mh_int_t i = mh_first(modules); - struct module *module = - (struct module *) mh_strnptr_node(modules, i)->val; - /* Can't delete modules if they have active calls */ - module_gc(module); - } - mh_strnptr_delete(modules); -} - -/** - * Look up a module in the modules cache. - */ -static struct module * -module_cache_find(const char *name, const char *name_end) -{ - mh_int_t i = mh_strnptr_find_inp(modules, name, name_end - name); - if (i == mh_end(modules)) - return NULL; - return (struct module *)mh_strnptr_node(modules, i)->val; -} - -/** - * Save module to the module cache. - */ -static inline int -module_cache_put(struct module *module) -{ - size_t package_len = strlen(module->package); - uint32_t name_hash = mh_strn_hash(module->package, package_len); - const struct mh_strnptr_node_t strnode = { - module->package, package_len, name_hash, module}; - - if (mh_strnptr_put(modules, &strnode, NULL, NULL) == mh_end(modules)) { - diag_set(OutOfMemory, sizeof(strnode), "malloc", "modules"); - return -1; - } - return 0; -} - -/** - * Delete a module from the module cache - */ -static void -module_cache_del(const char *name, const char *name_end) -{ - mh_int_t i = mh_strnptr_find_inp(modules, name, name_end - name); - if (i == mh_end(modules)) - return; - mh_strnptr_del(modules, i, NULL); -} - -/* - * Load a dso. - * Create a new symlink based on temporary directory and try to - * load via this symink to load a dso twice for cases of a function - * reload. - */ -static struct module * -module_load(const char *package, const char *package_end) -{ - char path[PATH_MAX]; - if (module_find(package, package_end, path, sizeof(path)) != 0) - return NULL; - - int package_len = package_end - package; - struct module *module = (struct module *) - malloc(sizeof(*module) + package_len + 1); - if (module == NULL) { - diag_set(OutOfMemory, sizeof(struct module) + package_len + 1, - "malloc", "struct module"); - return NULL; - } - memcpy(module->package, package, package_len); - module->package[package_len] = 0; - rlist_create(&module->funcs); - module->calls = 0; - - const char *tmpdir = getenv("TMPDIR"); - if (tmpdir == NULL) - tmpdir = "/tmp"; - char dir_name[PATH_MAX]; - int rc = snprintf(dir_name, sizeof(dir_name), "%s/tntXXXXXX", tmpdir); - if (rc < 0 || (size_t) rc >= sizeof(dir_name)) { - diag_set(SystemError, "failed to generate path to tmp dir"); - goto error; - } - - if (mkdtemp(dir_name) == NULL) { - diag_set(SystemError, "failed to create unique dir name: %s", - dir_name); - goto error; - } - char load_name[PATH_MAX]; - rc = snprintf(load_name, sizeof(load_name), "%s/%.*s." TARANTOOL_LIBEXT, - dir_name, package_len, package); - if (rc < 0 || (size_t) rc >= sizeof(dir_name)) { - diag_set(SystemError, "failed to generate path to DSO"); - goto error; - } - - struct stat st; - if (stat(path, &st) < 0) { - diag_set(SystemError, "failed to stat() module %s", path); - goto error; - } - - int source_fd = open(path, O_RDONLY); - if (source_fd < 0) { - diag_set(SystemError, "failed to open module %s file for" \ - " reading", path); - goto error; - } - int dest_fd = open(load_name, O_WRONLY|O_CREAT|O_TRUNC, - st.st_mode & 0777); - if (dest_fd < 0) { - diag_set(SystemError, "failed to open file %s for writing ", - load_name); - close(source_fd); - goto error; - } - - off_t ret = eio_sendfile_sync(dest_fd, source_fd, 0, st.st_size); - close(source_fd); - close(dest_fd); - if (ret != st.st_size) { - diag_set(SystemError, "failed to copy DSO %s to %s", - path, load_name); - goto error; - } - - module->handle = dlopen(load_name, RTLD_NOW | RTLD_LOCAL); - if (unlink(load_name) != 0) - say_warn("failed to unlink dso link %s", load_name); - if (rmdir(dir_name) != 0) - say_warn("failed to delete temporary dir %s", dir_name); - if (module->handle == NULL) { - diag_set(ClientError, ER_LOAD_MODULE, package_len, - package, dlerror()); - goto error; - } - struct errinj *e = errinj(ERRINJ_DYN_MODULE_COUNT, ERRINJ_INT); - if (e != NULL) - ++e->iparam; - return module; -error: - free(module); - return NULL; -} - -static void -module_delete(struct module *module) -{ - struct errinj *e = errinj(ERRINJ_DYN_MODULE_COUNT, ERRINJ_INT); - if (e != NULL) - --e->iparam; - dlclose(module->handle); - TRASH(module); - free(module); -} - -/* - * Check if a dso is unused and can be closed. - */ -static void -module_gc(struct module *module) -{ - if (rlist_empty(&module->funcs) && module->calls == 0) - module_delete(module); -} - -/* - * Import a function from the module. - */ -static box_function_f -module_sym(struct module *module, const char *name) -{ - box_function_f f = (box_function_f)dlsym(module->handle, name); - if (f == NULL) { - diag_set(ClientError, ER_LOAD_FUNCTION, name, dlerror()); - return NULL; - } - return f; -} - -int -module_sym_load(struct module_sym *mod_sym) -{ - assert(mod_sym->addr == NULL); - - struct func_name name; - func_split_name(mod_sym->name, &name); - - /* - * In case if module has been loaded already by - * some previous call we can eliminate redundant - * loading and take it from the cache. - */ - struct module *cached, *module; - cached = module_cache_find(name.package, name.package_end); - if (cached == NULL) { - module = module_load(name.package, name.package_end); - if (module == NULL) - return -1; - if (module_cache_put(module) != 0) { - module_delete(module); - return -1; - } - } else { - module = cached; - } - - mod_sym->addr = module_sym(module, name.sym); - if (mod_sym->addr == NULL) { - if (cached == NULL) { - /* - * In case if it was a first load we should - * clean the cache immediately otherwise - * the module continue being referenced even - * if there will be no use of it. - * - * Note the module_sym set an error thus be - * careful to not wipe it. - */ - module_cache_del(name.package, name.package_end); - module_delete(module); - } - return -1; - } - mod_sym->module = module; - rlist_add(&module->funcs, &mod_sym->item); - return 0; -} - -void -module_sym_unload(struct module_sym *mod_sym) -{ - if (mod_sym->module == NULL) - return; - - rlist_del(&mod_sym->item); - if (rlist_empty(&mod_sym->module->funcs)) { - struct func_name name; - func_split_name(mod_sym->name, &name); - module_cache_del(name.package, name.package_end); - } - module_gc(mod_sym->module); - - mod_sym->module = NULL; - mod_sym->addr = NULL; -} - -int -module_reload(const char *package, const char *package_end, struct module **module) -{ - struct module *old_module = module_cache_find(package, package_end); - if (old_module == NULL) { - /* Module wasn't loaded - do nothing. */ - *module = NULL; - return 0; - } - - struct module *new_module = module_load(package, package_end); - if (new_module == NULL) - return -1; - - struct module_sym *mod_sym, *tmp; - rlist_foreach_entry_safe(mod_sym, &old_module->funcs, item, tmp) { - struct func_name name; - func_split_name(mod_sym->name, &name); - mod_sym->addr = module_sym(new_module, name.sym); - if (mod_sym->addr == NULL) - goto restore; - mod_sym->module = new_module; - rlist_move(&new_module->funcs, &mod_sym->item); - } - module_cache_del(package, package_end); - if (module_cache_put(new_module) != 0) - goto restore; - module_gc(old_module); - *module = new_module; - return 0; -restore: - /* - * Some old-dso func can't be load from new module, restore old - * functions. - */ - do { - struct func_name name; - func_split_name(mod_sym->name, &name); - mod_sym->addr = module_sym(old_module, name.sym); - if (mod_sym->addr == NULL) { - /* - * Something strange was happen, an early loaden - * function was not found in an old dso. - */ - panic("Can't restore module function, " - "server state is inconsistent"); - } - mod_sym->module = old_module; - rlist_move(&old_module->funcs, &mod_sym->item); - } while (mod_sym != rlist_first_entry(&old_module->funcs, - struct module_sym, - item)); - assert(rlist_empty(&new_module->funcs)); - module_delete(new_module); - return -1; -} - static struct func * func_c_new(struct func_def *def); @@ -563,39 +124,9 @@ func_c_call(struct func *base, struct port *args, struct port *ret) { assert(base->vtab == &func_c_vtab); assert(base != NULL && base->def->language == FUNC_LANGUAGE_C); - struct func_c *func = (struct func_c *) base; - if (func->mod_sym.addr == NULL) { - if (module_sym_load(&func->mod_sym) != 0) - return -1; - } - - struct region *region = &fiber()->gc; - size_t region_svp = region_used(region); - uint32_t data_sz; - const char *data = port_get_msgpack(args, &data_sz); - if (data == NULL) - return -1; - - port_c_create(ret); - box_function_ctx_t ctx = { ret }; - /* Module can be changed after function reload. */ - struct module *module = func->mod_sym.module; - assert(module != NULL); - ++module->calls; - int rc = func->mod_sym.addr(&ctx, data, data + data_sz); - --module->calls; - module_gc(module); - region_truncate(region, region_svp); - if (rc != 0) { - if (diag_last_error(&fiber()->diag) == NULL) { - /* Stored procedure forget to set diag */ - diag_set(ClientError, ER_PROC_C, "unknown error"); - } - port_destroy(ret); - return -1; - } - return rc; + struct func_c *func = (struct func_c *)base; + return module_sym_call(&func->mod_sym, args, ret); } static struct func_vtab func_c_vtab = { diff --git a/src/box/func.h b/src/box/func.h index 9a7f17446..11a466b28 100644 --- a/src/box/func.h +++ b/src/box/func.h @@ -34,7 +34,8 @@ #include #include #include -#include "small/rlist.h" + +#include "module_cache.h" #include "func_def.h" #include "user_def.h" @@ -44,42 +45,6 @@ extern "C" { struct func; -/** - * Dynamic shared module. - */ -struct module { - /** Module dlhandle. */ - void *handle; - /** List of imported functions. */ - struct rlist funcs; - /** Count of active calls. */ - size_t calls; - /** Module's package name. */ - char package[0]; -}; - -/** - * Callable symbol bound to a module. - */ -struct module_sym { - /** - * Anchor for module membership. - */ - struct rlist item; - /** - * For C functions, address of the function. - */ - box_function_f addr; - /** - * A module the symbol belongs to. - */ - struct module *module; - /** - * Function name definition. - */ - char *name; -}; - /** Virtual method table for func object. */ struct func_vtab { /** Call function with given arguments. */ @@ -106,18 +71,6 @@ struct func { struct access access[BOX_USER_MAX]; }; -/** - * Initialize modules subsystem. - */ -int -module_init(void); - -/** - * Cleanup modules subsystem. - */ -void -module_free(void); - struct func * func_new(struct func_def *def); @@ -130,38 +83,6 @@ func_delete(struct func *func); int func_call(struct func *func, struct port *args, struct port *ret); -/** - * Resolve C entry (find the respective DLL and fetch the - * symbol from it). - * - * @param mod_sym module symbol pointer. - * @retval -1 on error. - * @retval 0 on success. - */ -int -module_sym_load(struct module_sym *mod_sym); - -/** - * Unload module symbol and drop it from the package - * cache if there is no users left. - * - * @param mod_sym module symbol pointer. - */ -void -module_sym_unload(struct module_sym *mod_sym); - -/** - * Reload dynamically loadable module. - * - * @param package name begin pointer. - * @param package_end package_end name end pointer. - * @param[out] module a pointer to store module object on success. - * @retval -1 on error. - * @retval 0 on success. - */ -int -module_reload(const char *package, const char *package_end, struct module **module); - #if defined(__cplusplus) } /* extern "C" */ #endif /* defined(__cplusplus) */ diff --git a/src/box/func_def.h b/src/box/func_def.h index d99d89190..75cd6a0d3 100644 --- a/src/box/func_def.h +++ b/src/box/func_def.h @@ -168,20 +168,6 @@ func_def_dup(struct func_def *def); int func_def_check(struct func_def *def); -/** - * API of C stored function. - */ - -struct port; - -struct box_function_ctx { - struct port *port; -}; - -typedef struct box_function_ctx box_function_ctx_t; -typedef int (*box_function_f)(box_function_ctx_t *ctx, - const char *args, const char *args_end); - #ifdef __cplusplus } #endif diff --git a/src/box/module_cache.c b/src/box/module_cache.c new file mode 100644 index 000000000..4d89baf9b --- /dev/null +++ b/src/box/module_cache.c @@ -0,0 +1,535 @@ +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file. + */ + +#include +#include +#include +#include + +#include "assoc.h" +#include "diag.h" +#include "error.h" +#include "errinj.h" +#include "fiber.h" +#include "port.h" + +#include "error.h" +#include "lua/utils.h" +#include "libeio/eio.h" + +#include "module_cache.h" + +/** Modules name to descriptor hash. */ +static struct mh_strnptr_t *mod_hash = NULL; + +/** + * Parsed symbol and package names. + */ +struct func_name { + /** + * Null-terminated symbol name, e.g. + * "func" for "mod.submod.func". + */ + const char *sym; + /** + * Package name, e.g. "mod.submod" for + * "mod.submod.func". + */ + const char *package; + /** + * A pointer to the last character in ->package + 1. + */ + const char *package_end; +}; + +/*** + * Split function name to symbol and package names. + * + * For example, str = foo.bar.baz => sym = baz, package = foo.bar + * + * @param str function name, e.g. "module.submodule.function". + * @param[out] name parsed symbol and a package name. + */ +static void +func_split_name(const char *str, struct func_name *name) +{ + name->package = str; + name->package_end = strrchr(str, '.'); + if (name->package_end != NULL) { + /* module.submodule.function => module.submodule, function */ + name->sym = name->package_end + 1; /* skip '.' */ + } else { + /* package == function => function, function */ + name->sym = name->package; + name->package_end = str + strlen(str); + } +} + +/** + * Look up a module in the modules cache. + */ +static struct module * +module_cache_find(const char *name, const char *name_end) +{ + mh_int_t e = mh_strnptr_find_inp(mod_hash, name, name_end - name); + if (e == mh_end(mod_hash)) + return NULL; + return mh_strnptr_node(mod_hash, e)->val; +} + +/** + * Save a module to the modules cache. + */ +static int +module_cache_add(struct module *module) +{ + size_t package_len = strlen(module->package); + const struct mh_strnptr_node_t nd = { + .str = module->package, + .len = package_len, + .hash = mh_strn_hash(module->package, package_len), + .val = module, + }; + + if (mh_strnptr_put(mod_hash, &nd, NULL, NULL) == mh_end(mod_hash)) { + diag_set(OutOfMemory, sizeof(nd), "malloc", + "module cache node"); + return -1; + } + return 0; +} + +/** + * Delete a module from the modules cache. + */ +static void +module_cache_del(const char *name, const char *name_end) +{ + mh_int_t e = mh_strnptr_find_inp(mod_hash, name, name_end - name); + if (e != mh_end(mod_hash)) + mh_strnptr_del(mod_hash, e, NULL); +} + +/** + * Arguments for luaT_module_find used by lua_cpcall(). + */ +struct module_find_ctx { + const char *package; + const char *package_end; + char *path; + size_t path_len; +}; + +/** + * A cpcall() helper for module_find(). + */ +static int +luaT_module_find(lua_State *L) +{ + struct module_find_ctx *ctx = (void *)lua_topointer(L, 1); + + /* + * Call package.searchpath(name, package.cpath) and use + * the path to the function in dlopen(). + */ + lua_getglobal(L, "package"); + lua_getfield(L, -1, "search"); + + /* Argument of search: name */ + lua_pushlstring(L, ctx->package, ctx->package_end - ctx->package); + + lua_call(L, 1, 1); + if (lua_isnil(L, -1)) + return luaL_error(L, "module not found"); + + /* Convert path to absolute */ + char resolved[PATH_MAX]; + if (realpath(lua_tostring(L, -1), resolved) == NULL) { + diag_set(SystemError, "realpath"); + return luaT_error(L); + } + + snprintf(ctx->path, ctx->path_len, "%s", resolved); + return 0; +} + +/** + * Find a path to a module using Lua's package.cpath. + * + * @param package package name + * @param package_end a pointer to the last byte in @a package + 1 + * @param[out] path path to shared library + * @param path_len size of @a path buffer + * + * @retval 0 on success + * @retval -1 on error, diag is set + */ +static int +module_find(const char *package, const char *package_end, + char *path, size_t path_len) +{ + struct module_find_ctx ctx = { + .package = package, + .package_end = package_end, + .path = path, + .path_len = path_len, + }; + lua_State *L = tarantool_L; + int top = lua_gettop(L); + if (luaT_cpcall(L, luaT_module_find, &ctx) != 0) { + diag_set(ClientError, ER_LOAD_MODULE, + (int)(package_end - package), + package, lua_tostring(L, -1)); + lua_settop(L, top); + return -1; + } + assert(top == lua_gettop(L)); /* cpcall discard results */ + return 0; +} + +/** + * Load dynamic shared object, ie module library. + * + * Create a new symlink based on temporary directory + * and try to load via this symink to load a dso twice + * for cases of a function reload. + */ +static struct module * +module_load(const char *package, const char *package_end) +{ + char path[PATH_MAX]; + if (module_find(package, package_end, path, sizeof(path)) != 0) + return NULL; + + int package_len = package_end - package; + struct module *module = malloc(sizeof(*module) + package_len + 1); + if (module == NULL) { + diag_set(OutOfMemory, sizeof(*module) + package_len + 1, + "malloc", "struct module"); + return NULL; + } + memcpy(module->package, package, package_len); + module->package[package_len] = 0; + rlist_create(&module->funcs_list); + module->calls = 0; + + const char *tmpdir = getenv("TMPDIR"); + if (tmpdir == NULL) + tmpdir = "/tmp"; + + char dir_name[PATH_MAX]; + int rc = snprintf(dir_name, sizeof(dir_name), "%s/tntXXXXXX", tmpdir); + if (rc < 0 || (size_t)rc >= sizeof(dir_name)) { + diag_set(SystemError, "failed to generate path to tmp dir"); + goto error; + } + + if (mkdtemp(dir_name) == NULL) { + diag_set(SystemError, "failed to create unique dir name: %s", + dir_name); + goto error; + } + + char load_name[PATH_MAX]; + rc = snprintf(load_name, sizeof(load_name), "%s/%.*s." TARANTOOL_LIBEXT, + dir_name, package_len, package); + if (rc < 0 || (size_t)rc >= sizeof(dir_name)) { + diag_set(SystemError, "failed to generate path to DSO"); + goto error; + } + + struct stat st; + if (stat(path, &st) < 0) { + diag_set(SystemError, "failed to stat() module %s", path); + goto error; + } + + int source_fd = open(path, O_RDONLY); + if (source_fd < 0) { + diag_set(SystemError, "failed to open module %s", path); + goto error; + } + + int dest_fd = open(load_name, O_WRONLY | O_CREAT | O_TRUNC, + st.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO)); + if (dest_fd < 0) { + diag_set(SystemError, "failed to open file %s for writing ", + load_name); + close(source_fd); + goto error; + } + + off_t ret = eio_sendfile_sync(dest_fd, source_fd, 0, st.st_size); + close(source_fd); + close(dest_fd); + if (ret != st.st_size) { + diag_set(SystemError, "failed to copy DSO %s to %s", + path, load_name); + goto error; + } + + module->handle = dlopen(load_name, RTLD_NOW | RTLD_LOCAL); + if (unlink(load_name) != 0) + say_warn("failed to unlink dso link %s", load_name); + if (rmdir(dir_name) != 0) + say_warn("failed to delete temporary dir %s", dir_name); + if (module->handle == NULL) { + diag_set(ClientError, ER_LOAD_MODULE, package_len, + package, dlerror()); + goto error; + } + + struct errinj *e = errinj(ERRINJ_DYN_MODULE_COUNT, ERRINJ_INT); + if (e != NULL) + ++e->iparam; + return module; + +error: + free(module); + return NULL; +} + +/** + * Delete a module. + */ +static void +module_delete(struct module *module) +{ + struct errinj *e = errinj(ERRINJ_DYN_MODULE_COUNT, ERRINJ_INT); + if (e != NULL) + --e->iparam; + dlclose(module->handle); + TRASH(module); + free(module); +} + +/** + * Check if a module is unused and delete it then. + */ +static void +module_gc(struct module *module) +{ + if (rlist_empty(&module->funcs_list) && module->calls == 0) + module_delete(module); +} + +/** + * Import a function from a module. + */ +static box_function_f +module_sym(struct module *module, const char *name) +{ + box_function_f f = dlsym(module->handle, name); + if (f == NULL) { + diag_set(ClientError, ER_LOAD_FUNCTION, name, dlerror()); + return NULL; + } + return f; +} + +int +module_sym_load(struct module_sym *mod_sym) +{ + assert(mod_sym->addr == NULL); + + struct func_name name; + func_split_name(mod_sym->name, &name); + + /* + * In case if module has been loaded already by + * some previous call we can eliminate redundant + * loading and take it from the cache. + */ + struct module *cached, *module; + cached = module_cache_find(name.package, name.package_end); + if (cached == NULL) { + module = module_load(name.package, name.package_end); + if (module == NULL) + return -1; + if (module_cache_add(module) != 0) { + module_delete(module); + return -1; + } + } else { + module = cached; + } + + mod_sym->addr = module_sym(module, name.sym); + if (mod_sym->addr == NULL) { + if (cached == NULL) { + /* + * In case if it was a first load we should + * clean the cache immediately otherwise + * the module continue being referenced even + * if there will be no use of it. + * + * Note the module_sym set an error thus be + * careful to not wipe it. + */ + module_cache_del(name.package, name.package_end); + module_delete(module); + } + return -1; + } + + mod_sym->module = module; + rlist_add(&module->funcs_list, &mod_sym->item); + return 0; +} + +void +module_sym_unload(struct module_sym *mod_sym) +{ + if (mod_sym->addr == NULL) + return; + + rlist_del(&mod_sym->item); + if (rlist_empty(&mod_sym->module->funcs_list)) { + struct func_name name; + func_split_name(mod_sym->name, &name); + module_cache_del(name.package, name.package_end); + } + module_gc(mod_sym->module); + + mod_sym->module = NULL; + mod_sym->addr = NULL; +} + +int +module_sym_call(struct module_sym *mod_sym, struct port *args, + struct port *ret) +{ + if (mod_sym->addr == NULL) { + if (module_sym_load(mod_sym) != 0) + return -1; + } + + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + uint32_t data_sz; + const char *data = port_get_msgpack(args, &data_sz); + if (data == NULL) + return -1; + + port_c_create(ret); + box_function_ctx_t ctx = { + .port = ret, + }; + + /* + * Module can be changed after function reload. Also + * keep in mind that stored C procedure may yield inside. + */ + struct module *module = mod_sym->module; + assert(module != NULL); + ++module->calls; + int rc = mod_sym->addr(&ctx, data, data + data_sz); + --module->calls; + module_gc(module); + region_truncate(region, region_svp); + + if (rc != 0) { + if (diag_last_error(&fiber()->diag) == NULL) { + /* Stored procedure forget to set diag */ + diag_set(ClientError, ER_PROC_C, "unknown error"); + } + port_destroy(ret); + return -1; + } + + return rc; +} + +int +module_reload(const char *package, const char *package_end, + struct module **module) +{ + struct module *old = module_cache_find(package, package_end); + if (old == NULL) { + /* Module wasn't loaded - do nothing. */ + *module = NULL; + return 0; + } + + struct module *new = module_load(package, package_end); + if (new == NULL) + return -1; + + struct module_sym *mod_sym, *tmp; + rlist_foreach_entry_safe(mod_sym, &old->funcs_list, item, tmp) { + struct func_name name; + func_split_name(mod_sym->name, &name); + + mod_sym->addr = module_sym(new, name.sym); + if (mod_sym->addr == NULL) { + say_error("module: reload %s, symbol %s not found", + package, name.sym); + goto restore; + } + + mod_sym->module = new; + rlist_move(&new->funcs_list, &mod_sym->item); + } + + module_cache_del(package, package_end); + if (module_cache_add(new) != 0) + goto restore; + + module_gc(old); + *module = new; + return 0; + +restore: + /* + * Some old-dso func can't be load from new module, + * restore old functions. + */ + do { + struct func_name name; + func_split_name(mod_sym->name, &name); + mod_sym->addr = module_sym(old, name.sym); + if (mod_sym->addr == NULL) { + /* + * Something strange was happen, an early loaden + * function was not found in an old dso. + */ + panic("Can't restore module function, " + "server state is inconsistent"); + } + mod_sym->module = old; + rlist_move(&old->funcs_list, &mod_sym->item); + } while (mod_sym != rlist_first_entry(&old->funcs_list, + struct module_sym, + item)); + assert(rlist_empty(&new->funcs_list)); + module_delete(new); + return -1; +} + +int +module_init(void) +{ + mod_hash = mh_strnptr_new(); + if (mod_hash == NULL) { + diag_set(OutOfMemory, sizeof(*mod_hash), + "malloc", "modules hash table"); + return -1; + } + return 0; +} + +void +module_free(void) +{ + while (mh_size(mod_hash) > 0) { + mh_int_t i = mh_first(mod_hash); + struct module *m = mh_strnptr_node(mod_hash, i)->val; + module_gc(m); + } + mh_strnptr_delete(mod_hash); + mod_hash = NULL; +} diff --git a/src/box/module_cache.h b/src/box/module_cache.h new file mode 100644 index 000000000..0f5d2b64a --- /dev/null +++ b/src/box/module_cache.h @@ -0,0 +1,139 @@ +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file. + */ + +#pragma once + +#include "small/rlist.h" + +#if defined(__cplusplus) +extern "C" { +#endif /* defined(__cplusplus) */ + + +/** + * API of C stored function. + */ + +struct port; + +struct box_function_ctx { + struct port *port; +}; + +typedef struct box_function_ctx box_function_ctx_t; +typedef int (*box_function_f)(box_function_ctx_t *ctx, + const char *args, + const char *args_end); + +/** + * Dynamic shared module. + */ +struct module { + /** + * Module dlhandle. + */ + void *handle; + /** + * List of associated symbols (functions). + */ + struct rlist funcs_list; + /** + * Count of active calls. + */ + size_t calls; + /** + * Module's package name. + */ + char package[0]; +}; + +/** + * Callable symbol bound to a module. + */ +struct module_sym { + /** + * Anchor for module::funcs_list. + */ + struct rlist item; + /** + * For C functions, address of the function. + */ + box_function_f addr; + /** + * A module the symbol belongs to. + */ + struct module *module; + /** + * Symbol (function) name definition. + */ + char *name; +}; + +/** + * Load a new module symbol. + * + * @param mod_sym symbol to load. + * + * @returns 0 on succse, -1 otherwise, diag is set. + */ +int +module_sym_load(struct module_sym *mod_sym); + +/** + * Unload a module's symbol. + * + * @param mod_sym symbol to unload. + */ +void +module_sym_unload(struct module_sym *mod_sym); + +/** + * Execute a module symbol (run a function). + * + * The function packs function arguments into a message pack + * and send it as a function argument. Function may return + * results via @a ret stream. + * + * @param mod_sym module symbol to run. + * @param args function arguments. + * @param[out] ret execution results. + * + * @returns 0 on success, -1 otherwise, diag is set. + */ +int +module_sym_call(struct module_sym *mod_sym, struct port *args, + struct port *ret); + +/** + * Reload a module and all associated symbols. + * + * @param package shared library path start. + * @param package_end shared library path end. + * @param[out] module pointer to the reloaded module. + * + * @return 0 on succes, -1 otherwise, diag is set. + */ +int +module_reload(const char *package, const char *package_end, + struct module **module); + +/** + * Initialize modules subsystem. + * + * @return 0 on succes, -1 otherwise, diag is set. + */ +int +module_init(void); + +/** + * Free modules subsystem. + */ +void +module_free(void); + +#if defined(__cplusplus) +} +#endif /* defined(__plusplus) */ -- 2.29.2