* [PATCH v3 0/6] box: rework functions machinery
@ 2019-06-13 14:08 Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 1/6] box: rework func cache update machinery Kirill Shcherbatov
` (5 more replies)
0 siblings, 6 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
This patchset reworks Tarantool's functions machinery to support
persistent Lua functions and provide an universal way to call any
registered function.
This patchset consists of following conceptual changes:
1. Func updates on alter were simplified: now replace _func tuple is
forbidden and func_new constructor may fail. This makes possible to
define a failable function object constructors.
2. box_lua_call and box_lua_eval were refactored to do nottake call_request
argument and to use input and output port instead. A new msgpack_port allows
to use this abstraction also for a regular memory msgpack.
3. Introduced an abstract function class and func_c and func_lua
implementations that allow to call C and Lua functions correspondingly
with uniform API defined on base func class.
4. Introduced an infrastructure to call functions from Lua. A new box.func
folder exports functions objects that have :call and :drop methods and their
definition.
4. Introduced persistent Lua functions. Such functions are a part of Tarantool
snapshoot so they are available even after restart. The new is_sandboxed
option initializes a new persistent Lua function is a separate 'sandbox':
an isolated environment with strong restrictions: only limited number of
on-board functions are available, no global state is available at all.
Persistent Lua functions is an important step to implement functional indexes
in Tarantool. Only a deterministic, sandboxed, persistent Lua function may
be used as a functional index extractor.
Changes in version 3:
Source code is re-organised to be bit clear and to provide a better function
object abstraction.
http://github.com/tarantool/tarantool/tree/kshch/gh-4182-persistent-lua-functions
https://github.com/tarantool/tarantool/issues/4182
Kirill Shcherbatov (6):
box: rework func cache update machinery
box: rework box_lua_{call, eval} to use input port
box: rework func object as a function frontend
box: export registered functions in box.func folder
box: refactor box_lua_find helper
box: introduce Lua persistent functions
src/box/alter.cc | 111 ++++---
src/box/bootstrap.snap | Bin 4475 -> 4532 bytes
src/box/call.c | 151 ++-------
src/box/call.h | 4 -
src/box/execute.c | 1 +
src/box/func.c | 199 +++++++++--
src/box/func.h | 40 +--
src/box/func_def.c | 25 ++
src/box/func_def.h | 34 +-
src/box/lua/call.c | 621 +++++++++++++++++++++++++++++++----
src/box/lua/call.h | 13 +-
src/box/lua/execute.c | 7 +-
src/box/lua/execute.h | 4 +-
src/box/lua/init.c | 1 +
src/box/lua/misc.cc | 10 +-
src/box/lua/schema.lua | 41 ++-
src/box/lua/upgrade.lua | 25 +-
src/box/port.c | 3 +-
src/box/port.h | 19 +-
src/box/schema.cc | 38 +--
src/box/schema.h | 15 +-
src/box/schema_def.h | 4 +
src/box/session.cc | 11 +-
src/box/session.h | 8 +
src/lib/core/port.h | 15 +-
test/box-py/bootstrap.result | 6 +-
test/box/access_misc.result | 6 +-
test/box/function1.c | 33 ++
test/box/function1.result | 332 +++++++++++++++++++
test/box/function1.test.lua | 115 +++++++
test/box/misc.result | 1 +
31 files changed, 1544 insertions(+), 349 deletions(-)
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 1/6] box: rework func cache update machinery
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-18 10:52 ` Vladimir Davydov
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
` (4 subsequent siblings)
5 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Tarantool used to assume that func_new call must not fail and
it used to build a new func object by given definition just on
func cache replace operation. We need to fix it to perform
user-dependent risky actions like Lua function assemble in
further patches.
The replace method is disallowed for _func space because it is
redundant and difficult to maintain in case of functions that
have pre-compiled runtime.
Needed for #4182, #1260
---
src/box/alter.cc | 57 +++++++++++++++++++------------------
src/box/func.c | 8 ------
src/box/func.h | 3 --
src/box/func_def.c | 17 +++++++++++
src/box/func_def.h | 12 ++++++++
src/box/schema.cc | 37 +++++++++---------------
src/box/schema.h | 10 +++----
test/box/function1.result | 4 +++
test/box/function1.test.lua | 1 +
9 files changed, 81 insertions(+), 68 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index a37a68ce4..18b50a5da 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2574,24 +2574,11 @@ func_def_new_from_tuple(struct tuple *tuple)
/** Remove a function from function cache */
static void
-func_cache_remove_func(struct trigger * /* trigger */, void *event)
+func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
- struct txn_stmt *stmt = txn_last_stmt((struct txn *) event);
- uint32_t fid = tuple_field_u32_xc(stmt->old_tuple ?
- stmt->old_tuple : stmt->new_tuple,
- BOX_FUNC_FIELD_ID);
- func_cache_delete(fid);
-}
-
-/** Replace a function in the function cache */
-static void
-func_cache_replace_func(struct trigger * /* trigger */, void *event)
-{
- struct txn_stmt *stmt = txn_last_stmt((struct txn*) event);
- struct func_def *def = func_def_new_from_tuple(stmt->new_tuple);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
- def_guard.is_active = false;
+ struct func *old_func = (struct func *) trigger->data;
+ func_cache_delete(old_func->def->fid);
+ func_delete(old_func);
}
/**
@@ -2612,13 +2599,21 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
struct func *old_func = func_by_id(fid);
if (new_tuple != NULL && old_func == NULL) { /* INSERT */
struct func_def *def = func_def_new_from_tuple(new_tuple);
+ auto def_guard = make_scoped_guard([=] { free(def); });
access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
PRIV_C);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
+ struct func *func = func_new(def);
+ if (func == NULL)
+ diag_raise();
+ auto func_guard = make_scoped_guard([=] {
+ func_cache_delete(def->fid);
+ func_delete(func);
+ });
def_guard.is_active = false;
+ func_cache_insert(func);
+ func_guard.is_active = false;
struct trigger *on_rollback =
- txn_alter_trigger_new(func_cache_remove_func, NULL);
+ txn_alter_trigger_new(func_cache_remove_func, func);
txn_on_rollback(txn, on_rollback);
} else if (new_tuple == NULL) { /* DELETE */
uint32_t uid;
@@ -2636,16 +2631,22 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
"function has grants");
}
struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_remove_func, NULL);
+ txn_alter_trigger_new(func_cache_remove_func, old_func);
txn_on_commit(txn, on_commit);
} else { /* UPDATE, REPLACE */
- struct func_def *def = func_def_new_from_tuple(new_tuple);
- auto def_guard = make_scoped_guard([=] { free(def); });
- access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
- PRIV_A);
- struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_replace_func, NULL);
- txn_on_commit(txn, on_commit);
+ assert(new_tuple != NULL && old_tuple != NULL);
+ /**
+ * Allow an alter that doesn't change the
+ * definition to support upgrade script.
+ */
+ struct func_def *new_def = func_def_new_from_tuple(new_tuple);
+ auto new_def_guard = make_scoped_guard([=] { free(new_def); });
+ struct func_def *old_def = func_def_new_from_tuple(old_tuple);
+ auto old_def_guard = make_scoped_guard([=] { free(old_def); });
+ if (func_def_cmp(new_def, old_def) != 0) {
+ tnt_raise(ClientError, ER_UNSUPPORTED, "function",
+ "alter");
+ }
}
}
diff --git a/src/box/func.c b/src/box/func.c
index a817851fd..d1b8879ad 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -451,14 +451,6 @@ func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
return rc;
}
-void
-func_update(struct func *func, struct func_def *def)
-{
- func_unload(func);
- free(func->def);
- func->def = def;
-}
-
void
func_delete(struct func *func)
{
diff --git a/src/box/func.h b/src/box/func.h
index 8dcd61d7b..fa4d738ab 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -100,9 +100,6 @@ module_free(void);
struct func *
func_new(struct func_def *def);
-void
-func_update(struct func *func, struct func_def *def);
-
void
func_delete(struct func *func);
diff --git a/src/box/func_def.c b/src/box/func_def.c
index 76ed77b24..2b135e2d7 100644
--- a/src/box/func_def.c
+++ b/src/box/func_def.c
@@ -1,3 +1,20 @@
#include "func_def.h"
+#include "string.h"
const char *func_language_strs[] = {"LUA", "C"};
+
+int
+func_def_cmp(struct func_def *def1, struct func_def *def2)
+{
+ if (def1->fid != def2->fid)
+ return def1->fid - def2->fid;
+ if (def1->uid != def2->uid)
+ return def1->uid - def2->uid;
+ if (def1->setuid != def2->setuid)
+ return def1->setuid - def2->setuid;
+ if (def1->language != def2->language)
+ return def1->language - def2->language;
+ if (strcmp(def1->name, def2->name) != 0)
+ return strcmp(def1->name, def2->name);
+ return 0;
+}
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 5b52ab498..4c9738aea 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -34,6 +34,10 @@
#include "trivia/util.h"
#include <stdbool.h>
+#ifdef __cplusplus
+extern "C" {
+#endif
+
/**
* The supported language of the stored function.
*/
@@ -79,6 +83,10 @@ func_def_sizeof(uint32_t name_len)
return sizeof(struct func_def) + name_len + 1;
}
+/** Compare two given function definitions. */
+int
+func_def_cmp(struct func_def *def1, struct func_def *def2);
+
/**
* API of C stored function.
*/
@@ -86,4 +94,8 @@ typedef struct box_function_ctx box_function_ctx_t;
typedef int (*box_function_f)(box_function_ctx_t *ctx,
const char *args, const char *args_end);
+#ifdef __cplusplus
+}
+#endif
+
#endif /* TARANTOOL_BOX_FUNC_DEF_H_INCLUDED */
diff --git a/src/box/schema.cc b/src/box/schema.cc
index 6d3153815..f834610a5 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -541,40 +541,32 @@ schema_free(void)
}
void
-func_cache_replace(struct func_def *def)
+func_cache_insert(struct func *func)
{
- struct func *old = func_by_id(def->fid);
- if (old) {
- func_update(old, def);
- return;
- }
if (mh_size(funcs) >= BOX_FUNCTION_MAX)
tnt_raise(ClientError, ER_FUNCTION_MAX, BOX_FUNCTION_MAX);
- struct func *func = func_new(def);
- if (func == NULL) {
-error:
- panic_syserror("Out of memory for the data "
- "dictionary cache (stored function).");
- }
- const struct mh_i32ptr_node_t node = { def->fid, func };
- mh_int_t k1 = mh_i32ptr_put(funcs, &node, NULL, NULL);
- if (k1 == mh_end(funcs)) {
- func->def = NULL;
- func_delete(func);
- goto error;
- }
+
+ mh_int_t k1, k2;
+ struct mh_i32ptr_node_t old_node, *old_node_ptr = &old_node;
+ const struct mh_i32ptr_node_t node = { func->def->fid, func };
size_t def_name_len = strlen(func->def->name);
uint32_t name_hash = mh_strn_hash(func->def->name, def_name_len);
const struct mh_strnptr_node_t strnode = {
func->def->name, def_name_len, name_hash, func };
- mh_int_t k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
+ k1 = mh_i32ptr_put(funcs, &node, &old_node_ptr, NULL);
+ if (k1 == mh_end(funcs))
+ goto error;
+ assert(old_node_ptr == NULL);
+ k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
if (k2 == mh_end(funcs_by_name)) {
mh_i32ptr_del(funcs, k1, NULL);
- func->def = NULL;
- func_delete(func);
goto error;
}
+ return;
+error:
+ panic_syserror("Out of memory for the data dictionary cache "
+ "(stored function).");
}
void
@@ -590,7 +582,6 @@ func_cache_delete(uint32_t fid)
strlen(func->def->name));
if (k != mh_end(funcs))
mh_strnptr_del(funcs_by_name, k, NULL);
- func_delete(func);
}
struct func *
diff --git a/src/box/schema.h b/src/box/schema.h
index 6f9a96117..d7b3c4c2d 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -163,14 +163,12 @@ schema_free();
struct space *schema_space(uint32_t id);
/**
- * Insert a new function or update the old one.
- *
- * @param def Function definition. In a case of success the ownership
- * of @a def is transfered to the data dictionary, thus the caller
- * must not delete it.
+ * Insert a new function object in the function cache.
+ * @param func Function object to insert.
+ * @retval Returns 0 on success, -1 otherwise.
*/
void
-func_cache_replace(struct func_def *def);
+func_cache_insert(struct func *func);
void
func_cache_delete(uint32_t fid);
diff --git a/test/box/function1.result b/test/box/function1.result
index 5b04fa97d..cadeb0467 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -16,6 +16,10 @@ c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
---
...
+box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+---
+- error: function does not support alter
+...
box.schema.user.grant('guest', 'execute', 'function', 'function1')
---
...
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index 855dd5382..e983495b6 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -7,6 +7,7 @@ net = require('net.box')
c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
+box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
box.schema.user.grant('guest', 'execute', 'function', 'function1')
_ = box.schema.space.create('test')
_ = box.space.test:create_index('primary')
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 1/6] box: rework func cache update machinery Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-17 9:35 ` [tarantool-patches] " Konstantin Osipov
` (4 more replies)
2019-06-13 14:08 ` [PATCH v3 3/6] box: rework func object as a function frontend Kirill Shcherbatov
` (3 subsequent siblings)
5 siblings, 5 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.
To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
Needed for #4182, #1260
---
src/box/call.c | 46 ++++++----------
src/box/call.h | 4 --
src/box/execute.c | 1 +
src/box/func.c | 27 ++++++++--
src/box/func.h | 11 ++--
src/box/lua/call.c | 121 +++++++++++++++++++++++++++++++-----------
src/box/lua/call.h | 8 ++-
src/box/lua/execute.c | 7 +--
src/box/lua/execute.h | 4 +-
src/box/lua/misc.cc | 5 +-
src/box/port.c | 3 +-
src/box/port.h | 15 ++++++
src/lib/core/port.h | 15 ++++--
13 files changed, 184 insertions(+), 83 deletions(-)
diff --git a/src/box/call.c b/src/box/call.c
index 56da53fb3..1ab3993da 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -105,33 +105,6 @@ access_check_func(const char *name, uint32_t name_len, struct func **funcp)
return 0;
}
-static int
-box_c_call(struct func *func, struct call_request *request, struct port *port)
-{
- assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
-
- /* Create a call context */
- port_tuple_create(port);
- box_function_ctx_t ctx = { port };
-
- /* Clear all previous errors */
- diag_clear(&fiber()->diag);
- assert(!in_txn()); /* transaction is not started */
-
- /* Call function from the shared library */
- int rc = func_call(func, &ctx, request->args, request->args_end);
- func = NULL; /* May be deleted by DDL */
- if (rc != 0) {
- if (diag_last_error(&fiber()->diag) == NULL) {
- /* Stored procedure forget to set diag */
- diag_set(ClientError, ER_PROC_C, "unknown error");
- }
- port_destroy(port);
- return -1;
- }
- return 0;
-}
-
int
box_module_reload(const char *name)
{
@@ -199,11 +172,15 @@ box_process_call(struct call_request *request, struct port *port)
}
int rc;
+ struct port in_port;
+ port_msgpack_create(&in_port, request->args,
+ request->args_end - request->args);
if (func && func->def->language == FUNC_LANGUAGE_C) {
- rc = box_c_call(func, request, port);
+ rc = func_call(func, &in_port, port);
} else {
- rc = box_lua_call(request, port);
+ rc = box_lua_call(name, name_len, &in_port, port);
}
+ port_destroy(&in_port);
/* Restore the original user */
if (orig_credentials)
fiber_set_user(fiber(), orig_credentials);
@@ -229,16 +206,23 @@ box_process_eval(struct call_request *request, struct port *port)
/* Check permissions */
if (access_check_universe(PRIV_X) != 0)
return -1;
- if (box_lua_eval(request, port) != 0) {
+ struct port in_port;
+ port_msgpack_create(&in_port, request->args,
+ request->args_end - request->args);
+ const char *expr = request->expr;
+ uint32_t expr_len = mp_decode_strl(&expr);
+ if (box_lua_eval(expr, expr_len, &in_port, port) != 0) {
+ port_destroy(&in_port);
txn_rollback();
return -1;
}
if (in_txn()) {
diag_set(ClientError, ER_FUNCTION_TX_ACTIVE);
+ port_destroy(&in_port);
txn_rollback();
return -1;
}
-
+ port_destroy(&in_port);
return 0;
}
diff --git a/src/box/call.h b/src/box/call.h
index 1b54551be..45580bc9d 100644
--- a/src/box/call.h
+++ b/src/box/call.h
@@ -38,10 +38,6 @@ extern "C" {
struct port;
struct call_request;
-struct box_function_ctx {
- struct port *port;
-};
-
/**
* Reload loadable module by name.
*
diff --git a/src/box/execute.c b/src/box/execute.c
index e81cc32aa..a91c939fc 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -108,6 +108,7 @@ const struct port_vtab port_sql_vtab = {
/* .dump_msgpack_16 = */ NULL,
/* .dump_lua = */ port_sql_dump_lua,
/* .dump_plain = */ NULL,
+ /* .get_msgpack = */ NULL,
/* .destroy = */ port_sql_destroy,
};
diff --git a/src/box/func.c b/src/box/func.c
index d1b8879ad..740fad3d7 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -34,6 +34,9 @@
#include "lua/utils.h"
#include "error.h"
#include "diag.h"
+#include "fiber.h"
+#include "port.h"
+#include "txn.h"
#include <dlfcn.h>
/**
@@ -433,21 +436,39 @@ func_load(struct func *func)
}
int
-func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
- const char *args_end)
+func_call(struct func *func, struct port *in_port, struct port *out_port)
{
+ assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
+ /* Transaction is not started. */
+ assert(!in_txn());
if (func->func == NULL) {
if (func_load(func) != 0)
return -1;
}
+ uint32_t args_sz;
+ const char *args = port_get_msgpack(in_port, &args_sz);
+ if (args == NULL)
+ return -1;
+
+ port_tuple_create(out_port);
+ box_function_ctx_t ctx = { out_port };
+
/* Module can be changed after function reload. */
struct module *module = func->module;
assert(module != NULL);
++module->calls;
- int rc = func->func(ctx, args, args_end);
+ int rc = func->func(&ctx, args, args + args_sz);
--module->calls;
module_gc(module);
+ if (rc != 0) {
+ if (diag_last_error(&fiber()->diag) == NULL) {
+ /* Stored procedure forget to set diag */
+ diag_set(ClientError, ER_PROC_C, "unknown error");
+ }
+ port_destroy(out_port);
+ return -1;
+ }
return rc;
}
diff --git a/src/box/func.h b/src/box/func.h
index fa4d738ab..d38a2b283 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -42,6 +42,12 @@
extern "C" {
#endif /* defined(__cplusplus) */
+struct port;
+
+struct box_function_ctx {
+ struct port *port;
+};
+
/**
* Dynamic shared module.
*/
@@ -104,11 +110,10 @@ void
func_delete(struct func *func);
/**
- * Call stored C function using @a args.
+ * Call function with arguments represented with given in_port.
*/
int
-func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
- const char *args_end);
+func_call(struct func *func, struct port *in_port, struct port *out_port);
/**
* Reload dynamically loadable module.
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 04020ef6f..e32845095 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -266,6 +266,55 @@ luamp_encode_call_16(lua_State *L, struct luaL_serializer *cfg,
return root.size;
}
+static const struct port_vtab port_msgpack_vtab;
+
+void
+port_msgpack_create(struct port *base, const char *data, uint32_t data_sz)
+{
+ struct port_msgpack *port_msgpack = (struct port_msgpack *) base;
+ memset(port_msgpack, 0, sizeof(*port_msgpack));
+ port_msgpack->vtab = &port_msgpack_vtab;
+ port_msgpack->data = data;
+ port_msgpack->data_sz = data_sz;
+}
+
+static void
+port_msgpack_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
+{
+ assert(is_flat == true);
+ struct port_msgpack *port = (struct port_msgpack *) base;
+ assert(port->vtab == &port_msgpack_vtab);
+
+ const char *args = port->data;
+ uint32_t arg_count = mp_decode_array(&args);
+ for (uint32_t i = 0; i < arg_count; i++)
+ luamp_decode(L, luaL_msgpack_default, &args);
+}
+
+static const char *
+port_msgpack_get_msgpack(struct port *base, uint32_t *size)
+{
+ struct port_msgpack *port = (struct port_msgpack *) base;
+ assert(port->vtab == &port_msgpack_vtab);
+ *size = port->data_sz;
+ return port->data;
+}
+
+static void
+port_msgpack_destroy(struct port *base)
+{
+ (void) base;
+}
+
+static const struct port_vtab port_msgpack_vtab = {
+ .dump_msgpack = NULL,
+ .dump_msgpack_16 = NULL,
+ .dump_lua = port_msgpack_dump_lua,
+ .dump_plain = NULL,
+ .get_msgpack = port_msgpack_get_msgpack,
+ .destroy = port_msgpack_destroy,
+};
+
static const struct port_vtab port_lua_vtab;
void
@@ -282,28 +331,31 @@ port_lua_create(struct port *port, struct lua_State *L)
port_lua->ref = -1;
}
+struct lua_process_ctx {
+ const char *name;
+ uint32_t name_len;
+};
+
static int
execute_lua_call(lua_State *L)
{
- struct call_request *request = (struct call_request *)
- lua_topointer(L, 1);
+ struct port *in_port = (struct port *) lua_topointer(L, 1);
+ struct lua_process_ctx *ctx =
+ (struct lua_process_ctx *) lua_topointer(L, 2);
lua_settop(L, 0); /* clear the stack to simplify the logic below */
- const char *name = request->name;
- uint32_t name_len = mp_decode_strl(&name);
+ const char *name = ctx->name;
+ uint32_t name_len = ctx->name_len;
int oc = 0; /* how many objects are on stack after box_lua_find */
/* Try to find a function by name in Lua */
oc = box_lua_find(L, name, name + name_len);
/* Push the rest of args (a tuple). */
- const char *args = request->args;
-
- uint32_t arg_count = mp_decode_array(&args);
- luaL_checkstack(L, arg_count, "call: out of stack");
+ int top = lua_gettop(L);
+ port_dump_lua(in_port, L, true);
+ int arg_count = lua_gettop(L) - top;
- for (uint32_t i = 0; i < arg_count; i++)
- luamp_decode(L, luaL_msgpack_default, &args);
lua_call(L, arg_count + oc - 1, LUA_MULTRET);
return lua_gettop(L);
}
@@ -311,24 +363,23 @@ execute_lua_call(lua_State *L)
static int
execute_lua_eval(lua_State *L)
{
- struct call_request *request = (struct call_request *)
- lua_topointer(L, 1);
+ struct port *in_port = (struct port *) lua_topointer(L, 1);
+ struct lua_process_ctx *ctx =
+ (struct lua_process_ctx *) lua_topointer(L, 2);
lua_settop(L, 0); /* clear the stack to simplify the logic below */
/* Compile expression */
- const char *expr = request->expr;
- uint32_t expr_len = mp_decode_strl(&expr);
+ const char *expr = ctx->name;
+ uint32_t expr_len = ctx->name_len;
if (luaL_loadbuffer(L, expr, expr_len, "=eval")) {
diag_set(LuajitError, lua_tostring(L, -1));
luaT_error(L);
}
/* Unpack arguments */
- const char *args = request->args;
- uint32_t arg_count = mp_decode_array(&args);
- luaL_checkstack(L, arg_count, "eval: out of stack");
- for (uint32_t i = 0; i < arg_count; i++)
- luamp_decode(L, luaL_msgpack_default, &args);
+ int top = lua_gettop(L);
+ port_dump_lua(in_port, L, true);
+ int arg_count = lua_gettop(L) - top;
/* Call compiled code */
lua_call(L, arg_count, LUA_MULTRET);
@@ -429,37 +480,47 @@ static const struct port_vtab port_lua_vtab = {
.dump_msgpack_16 = port_lua_dump_16,
.dump_lua = NULL,
.dump_plain = port_lua_dump_plain,
+ .get_msgpack = NULL,
.destroy = port_lua_destroy,
};
static inline int
-box_process_lua(struct call_request *request, struct port *base,
- lua_CFunction handler)
+box_process_lua(lua_CFunction handler, void *arg, struct port *in_port,
+ struct port *out_port)
{
lua_State *L = lua_newthread(tarantool_L);
int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
- port_lua_create(base, L);
- ((struct port_lua *) base)->ref = coro_ref;
+ port_lua_create(out_port, L);
+ ((struct port_lua *) out_port)->ref = coro_ref;
lua_pushcfunction(L, handler);
- lua_pushlightuserdata(L, request);
- if (luaT_call(L, 1, LUA_MULTRET) != 0) {
- port_lua_destroy(base);
+ lua_pushlightuserdata(L, in_port);
+ lua_pushlightuserdata(L, arg);
+ if (luaT_call(L, 2, LUA_MULTRET) != 0) {
+ port_lua_destroy(out_port);
return -1;
}
return 0;
}
int
-box_lua_call(struct call_request *request, struct port *port)
+box_lua_call(const char *name, uint32_t name_len,
+ struct port *in_port, struct port *out_port)
{
- return box_process_lua(request, port, execute_lua_call);
+ struct lua_process_ctx ctx;
+ ctx.name = name;
+ ctx.name_len = name_len;
+ return box_process_lua(execute_lua_call, &ctx, in_port, out_port);
}
int
-box_lua_eval(struct call_request *request, struct port *port)
+box_lua_eval(const char *expr, uint32_t expr_len,
+ struct port *in_port, struct port *out_port)
{
- return box_process_lua(request, port, execute_lua_eval);
+ struct lua_process_ctx ctx;
+ ctx.name = expr;
+ ctx.name_len = expr_len;
+ return box_process_lua(execute_lua_eval, &ctx, in_port, out_port);
}
static int
diff --git a/src/box/lua/call.h b/src/box/lua/call.h
index 0542123da..1801d5090 100644
--- a/src/box/lua/call.h
+++ b/src/box/lua/call.h
@@ -35,6 +35,8 @@
extern "C" {
#endif /* defined(__cplusplus) */
+#include <stdint.h>
+
struct lua_State;
void
@@ -48,10 +50,12 @@ struct call_request;
* (implementation of 'CALL' command code).
*/
int
-box_lua_call(struct call_request *request, struct port *port);
+box_lua_call(const char *name, uint32_t name_len,
+ struct port *in_port, struct port *out_port);
int
-box_lua_eval(struct call_request *request, struct port *port);
+box_lua_eval(const char *expr, uint32_t expr_len,
+ struct port *in_port, struct port *out_port);
#if defined(__cplusplus)
} /* extern "C" */
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index 239aba47b..187bdb73d 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -39,8 +39,9 @@ lua_sql_get_metadata(struct sql_stmt *stmt, struct lua_State *L,
}
void
-port_sql_dump_lua(struct port *port, struct lua_State *L)
+port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
{
+ assert(is_flat == false);
assert(port->vtab == &port_sql_vtab);
struct sql *db = sql_get();
struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
@@ -49,7 +50,7 @@ port_sql_dump_lua(struct port *port, struct lua_State *L)
lua_createtable(L, 0, 2);
lua_sql_get_metadata(stmt, L, column_count);
lua_setfield(L, -2, "metadata");
- port_tuple_vtab.dump_lua(port, L);
+ port_tuple_vtab.dump_lua(port, L, is_flat);
lua_setfield(L, -2, "rows");
} else {
assert(((struct port_tuple *)port)->size == 0);
@@ -262,7 +263,7 @@ lbox_execute(struct lua_State *L)
if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
&fiber()->gc) != 0)
return luaT_error(L);
- port_dump_lua(&port, L);
+ port_dump_lua(&port, L, false);
port_destroy(&port);
return 1;
}
diff --git a/src/box/lua/execute.h b/src/box/lua/execute.h
index bc4df19f5..23e193fa4 100644
--- a/src/box/lua/execute.h
+++ b/src/box/lua/execute.h
@@ -30,6 +30,8 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
+#include <stdbool.h>
+
struct port;
struct sql_bind;
struct lua_State;
@@ -42,7 +44,7 @@ struct lua_State;
* @param L Lua stack.
*/
void
-port_sql_dump_lua(struct port *port, struct lua_State *L);
+port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat);
/**
* Parse Lua table of SQL parameters.
diff --git a/src/box/lua/misc.cc b/src/box/lua/misc.cc
index 8de7401fa..15681c377 100644
--- a/src/box/lua/misc.cc
+++ b/src/box/lua/misc.cc
@@ -61,8 +61,9 @@ lbox_encode_tuple_on_gc(lua_State *L, int idx, size_t *p_len)
* but implemented here to eliminate port.c dependency on Lua.
*/
extern "C" void
-port_tuple_dump_lua(struct port *base, struct lua_State *L)
+port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
{
+ assert(is_flat == false);
struct port_tuple *port = port_tuple(base);
lua_createtable(L, port->size, 0);
struct port_tuple_entry *pe = port->first;
@@ -109,7 +110,7 @@ lbox_select(lua_State *L)
* table always crashed the first (can't be fixed with pcall).
* https://github.com/tarantool/tarantool/issues/1182
*/
- port_dump_lua(&port, L);
+ port_dump_lua(&port, L, false);
port_destroy(&port);
return 1; /* lua table with tuples */
}
diff --git a/src/box/port.c b/src/box/port.c
index 99046449a..7f552bcfe 100644
--- a/src/box/port.c
+++ b/src/box/port.c
@@ -125,7 +125,7 @@ port_tuple_dump_msgpack(struct port *base, struct obuf *out)
}
extern void
-port_tuple_dump_lua(struct port *base, struct lua_State *L);
+port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat);
void
port_init(void)
@@ -145,5 +145,6 @@ const struct port_vtab port_tuple_vtab = {
.dump_msgpack_16 = port_tuple_dump_msgpack_16,
.dump_lua = port_tuple_dump_lua,
.dump_plain = NULL,
+ .get_msgpack = NULL,
.destroy = port_tuple_destroy,
};
diff --git a/src/box/port.h b/src/box/port.h
index f18803660..4c0c3cf7a 100644
--- a/src/box/port.h
+++ b/src/box/port.h
@@ -32,6 +32,7 @@
*/
#include "trivia/util.h"
#include <port.h>
+#include <stdbool.h>
#if defined(__cplusplus)
extern "C" {
@@ -80,6 +81,20 @@ port_tuple_create(struct port *port);
int
port_tuple_add(struct port *port, struct tuple *tuple);
+/** Port implementation used for storing raw data. */
+struct port_msgpack {
+ const struct port_vtab *vtab;
+ const char *data;
+ uint32_t data_sz;
+};
+
+static_assert(sizeof(struct port_msgpack) <= sizeof(struct port),
+ "sizeof(struct port_msgpack) must be <= sizeof(struct port)");
+
+/** Initialize a port to dump row data. */
+void
+port_msgpack_create(struct port *port, const char *data, uint32_t data_sz);
+
/** Port for storing the result of a Lua CALL/EVAL. */
struct port_lua {
const struct port_vtab *vtab;
diff --git a/src/lib/core/port.h b/src/lib/core/port.h
index 8ace40fc5..2f1bc5ed1 100644
--- a/src/lib/core/port.h
+++ b/src/lib/core/port.h
@@ -30,6 +30,7 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
+#include <stdbool.h>
#include <stdint.h>
#if defined(__cplusplus)
@@ -63,7 +64,7 @@ struct port_vtab {
*/
int (*dump_msgpack_16)(struct port *port, struct obuf *out);
/** Dump the content of a port to Lua stack. */
- void (*dump_lua)(struct port *port, struct lua_State *L);
+ void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
/**
* Dump a port content as a plain text into a buffer,
* allocated inside.
@@ -74,6 +75,8 @@ struct port_vtab {
* @retval not nil Plain text.
*/
const char *(*dump_plain)(struct port *port, uint32_t *size);
+ /** Get the content of a port as a msgpack data. */
+ const char *(*get_msgpack)(struct port *port, uint32_t *size);
/** Destroy a port and release associated resources. */
void (*destroy)(struct port *port);
};
@@ -109,9 +112,9 @@ port_dump_msgpack_16(struct port *port, struct obuf *out)
}
static inline void
-port_dump_lua(struct port *port, struct lua_State *L)
+port_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
{
- port->vtab->dump_lua(port, L);
+ port->vtab->dump_lua(port, L, is_flat);
}
static inline const char *
@@ -120,6 +123,12 @@ port_dump_plain(struct port *port, uint32_t *size)
return port->vtab->dump_plain(port, size);
}
+static inline const char *
+port_get_msgpack(struct port *port, uint32_t *size)
+{
+ return port->vtab->get_msgpack(port, size);
+}
+
#if defined(__cplusplus)
} /* extern "C" */
#endif /* defined __cplusplus */
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 3/6] box: rework func object as a function frontend
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 1/6] box: rework func cache update machinery Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-18 13:23 ` Vladimir Davydov
2019-06-13 14:08 ` [PATCH v3 4/6] box: export registered functions in box.func folder Kirill Shcherbatov
` (2 subsequent siblings)
5 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
The function func object used to provide a call method only for
C functions. In scope of this patch it reworked to be a uniform
function call frontend both for C and Lua functions.
Introduced classes func_c and func_lua, that provide own
constructors which produce implementation-specific object with
call and destroy methods.
Needed for #4182, #1260
---
src/box/alter.cc | 1 +
src/box/call.c | 105 ++---------------------------
src/box/func.c | 165 ++++++++++++++++++++++++++++++++++++++++-----
src/box/func.h | 25 ++++---
src/box/func_def.h | 2 +
src/box/lua/call.c | 46 +++++++++++++
src/box/lua/call.h | 5 ++
src/box/session.cc | 11 ++-
src/box/session.h | 8 +++
9 files changed, 235 insertions(+), 133 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 18b50a5da..7b0eb3334 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2552,6 +2552,7 @@ func_def_new_from_tuple(struct tuple *tuple)
func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid);
memcpy(def->name, name, len);
def->name[len] = 0;
+ def->name_len = len;
if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
def->setuid = tuple_field_u32_xc(tuple, BOX_FUNC_FIELD_SETUID);
else
diff --git a/src/box/call.c b/src/box/call.c
index 1ab3993da..42db883e1 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -43,68 +43,6 @@
#include "small/obuf.h"
#include "tt_static.h"
-/**
- * Find a function by name and check "EXECUTE" permissions.
- *
- * @param name function name
- * @param name_len length of @a name
- * @param[out] funcp function object
- * Sic: *pfunc == NULL means that perhaps the user has a global
- * "EXECUTE" privilege, so no specific grant to a function.
- *
- * @retval -1 on access denied
- * @retval 0 on success
- */
-static inline int
-access_check_func(const char *name, uint32_t name_len, struct func **funcp)
-{
- struct func *func = func_by_name(name, name_len);
- struct credentials *credentials = effective_user();
- /*
- * If the user has universal access, don't bother with checks.
- * No special check for ADMIN user is necessary
- * since ADMIN has universal access.
- */
- if ((credentials->universal_access & (PRIV_X | PRIV_U)) ==
- (PRIV_X | PRIV_U)) {
-
- *funcp = func;
- return 0;
- }
- user_access_t access = PRIV_X | PRIV_U;
- /* Check access for all functions. */
- access &= ~entity_access_get(SC_FUNCTION)[credentials->auth_token].effective;
- user_access_t func_access = access & ~credentials->universal_access;
- if (func == NULL ||
- /* Check for missing Usage access, ignore owner rights. */
- func_access & PRIV_U ||
- /* Check for missing specific access, respect owner rights. */
- (func->def->uid != credentials->uid &&
- func_access & ~func->access[credentials->auth_token].effective)) {
-
- /* Access violation, report error. */
- struct user *user = user_find(credentials->uid);
- if (user != NULL) {
- if (!(access & credentials->universal_access)) {
- diag_set(AccessDeniedError,
- priv_name(PRIV_U),
- schema_object_name(SC_UNIVERSE), "",
- user->def->name);
- } else {
- diag_set(AccessDeniedError,
- priv_name(PRIV_X),
- schema_object_name(SC_FUNCTION),
- tt_cstr(name, name_len),
- user->def->name);
- }
- }
- return -1;
- }
-
- *funcp = func;
- return 0;
-}
-
int
box_module_reload(const char *name)
{
@@ -137,53 +75,18 @@ box_process_call(struct call_request *request, struct port *port)
assert(name != NULL);
uint32_t name_len = mp_decode_strl(&name);
- struct func *func = NULL;
- /**
- * Sic: func == NULL means that perhaps the user has a global
- * "EXECUTE" privilege, so no specific grant to a function.
- */
- if (access_check_func(name, name_len, &func) != 0)
- return -1; /* permission denied */
-
- /**
- * Change the current user id if the function is
- * a set-definer-uid one. If the function is not
- * defined, it's obviously not a setuid one.
- */
- struct credentials *orig_credentials = NULL;
- if (func && func->def->setuid) {
- orig_credentials = effective_user();
- /* Remember and change the current user id. */
- if (func->owner_credentials.auth_token >= BOX_USER_MAX) {
- /*
- * Fill the cache upon first access, since
- * when func is created, no user may
- * be around to fill it (recovery of
- * system spaces from a snapshot).
- */
- struct user *owner = user_find(func->def->uid);
- if (owner == NULL)
- return -1;
- credentials_init(&func->owner_credentials,
- owner->auth_token,
- owner->def->uid);
- }
- fiber_set_user(fiber(), &func->owner_credentials);
- }
-
int rc;
struct port in_port;
port_msgpack_create(&in_port, request->args,
request->args_end - request->args);
- if (func && func->def->language == FUNC_LANGUAGE_C) {
+ struct func *func = func_by_name(name, name_len);
+ if (func != NULL) {
rc = func_call(func, &in_port, port);
- } else {
+ } else if ((rc = access_check_universe_object(PRIV_X | PRIV_U,
+ SC_FUNCTION, tt_cstr(name, name_len))) == 0) {
rc = box_lua_call(name, name_len, &in_port, port);
}
port_destroy(&in_port);
- /* Restore the original user */
- if (orig_credentials)
- fiber_set_user(fiber(), orig_credentials);
if (rc != 0) {
txn_rollback();
diff --git a/src/box/func.c b/src/box/func.c
index 740fad3d7..20ff56668 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -32,11 +32,14 @@
#include "trivia/config.h"
#include "assoc.h"
#include "lua/utils.h"
+#include "lua/call.h"
#include "error.h"
#include "diag.h"
#include "fiber.h"
#include "port.h"
#include "txn.h"
+#include "schema.h"
+#include "session.h"
#include <dlfcn.h>
/**
@@ -51,6 +54,24 @@ struct func_name {
const char *package_end;
};
+struct func_c {
+ /** Function object base class. */
+ struct func base;
+ /**
+ * Anchor for module membership.
+ */
+ struct rlist item;
+ /**
+ * For C functions, the body of the function.
+ */
+ box_function_f func;
+ /**
+ * Each stored function keeps a handle to the
+ * dynamic library for the C callback.
+ */
+ struct module *module;
+};
+
/***
* Split function name to symbol and package names.
* For example, str = foo.bar.baz => sym = baz, package = foo.bar
@@ -315,10 +336,10 @@ module_reload(const char *package, const char *package_end, struct module **modu
if (new_module == NULL)
return -1;
- struct func *func, *tmp_func;
+ struct func_c *func, *tmp_func;
rlist_foreach_entry_safe(func, &old_module->funcs, item, tmp_func) {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
func->func = module_sym(new_module, name.sym);
if (func->func == NULL)
goto restore;
@@ -339,7 +360,7 @@ restore:
*/
do {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
func->func = module_sym(old_module, name.sym);
if (func->func == NULL) {
/*
@@ -352,20 +373,27 @@ restore:
func->module = old_module;
rlist_move(&old_module->funcs, &func->item);
} while (func != rlist_first_entry(&old_module->funcs,
- struct func, item));
+ struct func_c, item));
assert(rlist_empty(&new_module->funcs));
module_delete(new_module);
return -1;
}
+struct func *
+func_c_new(struct func_def *def);
+
struct func *
func_new(struct func_def *def)
{
- struct func *func = (struct func *) malloc(sizeof(struct func));
- if (func == NULL) {
- diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
- return NULL;
+ struct func *func;
+ if (def->language == FUNC_LANGUAGE_C) {
+ func = func_c_new(def);
+ } else {
+ assert(def->language == FUNC_LANGUAGE_LUA);
+ func = func_lua_new(def);
}
+ if (func == NULL)
+ return NULL;
func->def = def;
/** Nobody has access to the function but the owner. */
memset(func->access, 0, sizeof(func->access));
@@ -381,19 +409,39 @@ func_new(struct func_def *def)
* checks (see user_has_data()).
*/
func->owner_credentials.auth_token = BOX_USER_MAX; /* invalid value */
+ return func;
+}
+
+static struct func_vtab func_c_vtab;
+
+struct func *
+func_c_new(struct func_def *def)
+{
+ (void) def;
+ assert(def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) malloc(sizeof(struct func_c));
+ if (func == NULL) {
+ diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
+ return NULL;
+ }
+ func->base.vtab = &func_c_vtab;
func->func = NULL;
func->module = NULL;
- return func;
+ return &func->base;
}
static void
-func_unload(struct func *func)
+func_c_unload(struct func *base)
{
+ assert(base->vtab == &func_c_vtab);
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) base;
+
if (func->module) {
rlist_del(&func->item);
if (rlist_empty(&func->module->funcs)) {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
module_cache_del(name.package, name.package_end);
}
module_gc(func->module);
@@ -407,12 +455,15 @@ func_unload(struct func *func)
* symbol from it).
*/
static int
-func_load(struct func *func)
+func_c_load(struct func *base)
{
+ assert(base->vtab == &func_c_vtab);
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) base;
assert(func->func == NULL);
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
struct module *module = module_cache_find(name.package,
name.package_end);
@@ -436,13 +487,16 @@ func_load(struct func *func)
}
int
-func_call(struct func *func, struct port *in_port, struct port *out_port)
+func_c_call(struct func *base, struct port *in_port, struct port *out_port)
{
- assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
+ assert(base->vtab == &func_c_vtab);
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) base;
+
/* Transaction is not started. */
assert(!in_txn());
if (func->func == NULL) {
- if (func_load(func) != 0)
+ if (func_c_load(base) != 0)
return -1;
}
@@ -472,10 +526,87 @@ func_call(struct func *func, struct port *in_port, struct port *out_port)
return rc;
}
+static struct func_vtab func_c_vtab = {
+ .call = func_c_call,
+ .destroy = func_c_unload,
+};
+
void
func_delete(struct func *func)
{
- func_unload(func);
+ func->vtab->destroy(func);
free(func->def);
free(func);
}
+
+/** Check "EXECUTE" permissions for a given function. */
+static int
+func_access_check(struct func *base)
+{
+ struct credentials *credentials = effective_user();
+ /*
+ * If the user has universal access, don't bother with
+ * checks. No special check for ADMIN user is necessary
+ * since ADMIN has universal access.
+ */
+ if ((credentials->universal_access & (PRIV_X | PRIV_U)) ==
+ (PRIV_X | PRIV_U))
+ return 0;
+ user_access_t access = PRIV_X | PRIV_U;
+ /* Check access for all functions. */
+ access &= ~entity_access_get(SC_FUNCTION)[
+ credentials->auth_token].effective;
+ user_access_t func_access = access & ~credentials->universal_access;
+ if (/* Check for missing Usage access, ignore owner rights. */
+ func_access & PRIV_U ||
+ /* Check for missing specific access, respect owner rights. */
+ (base->def->uid != credentials->uid &&
+ func_access & ~base->access[credentials->auth_token].effective)) {
+ /* Access violation, report error. */
+ struct user *user = user_find(credentials->uid);
+ if (user != NULL) {
+ diag_set(AccessDeniedError, priv_name(PRIV_X),
+ schema_object_name(SC_FUNCTION),
+ base->def->name, user->def->name);
+ }
+ return -1;
+ }
+ return 0;
+}
+
+int
+func_call(struct func *base, struct port *in_port, struct port *out_port)
+{
+ if (func_access_check(base) != 0)
+ return -1;
+ /**
+ * Change the current user id if the function is
+ * a set-definer-uid one. If the function is not
+ * defined, it's obviously not a setuid one.
+ */
+ struct credentials *orig_credentials = NULL;
+ if (base->def->setuid) {
+ orig_credentials = effective_user();
+ /* Remember and change the current user id. */
+ if (base->owner_credentials.auth_token >= BOX_USER_MAX) {
+ /*
+ * Fill the cache upon first access, since
+ * when func is created, no user may
+ * be around to fill it (recovery of
+ * system spaces from a snapshot).
+ */
+ struct user *owner = user_find(base->def->uid);
+ if (owner == NULL)
+ return -1;
+ credentials_init(&base->owner_credentials,
+ owner->auth_token,
+ owner->def->uid);
+ }
+ fiber_set_user(fiber(), &base->owner_credentials);
+ }
+ int rc = base->vtab->call(base, in_port, out_port);
+ /* Restore the original user */
+ if (orig_credentials)
+ fiber_set_user(fiber(), orig_credentials);
+ return rc;
+}
diff --git a/src/box/func.h b/src/box/func.h
index d38a2b283..1271bde67 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -43,6 +43,7 @@ extern "C" {
#endif /* defined(__cplusplus) */
struct port;
+struct func;
struct box_function_ctx {
struct port *port;
@@ -62,24 +63,22 @@ struct module {
bool is_unloading;
};
+/** Virtual method table for func object. */
+struct func_vtab {
+ /** Call function with given arguments. */
+ int (*call)(struct func *func, struct port *in_port,
+ struct port *out_port);
+ /** Release implementation-specific function context. */
+ void (*destroy)(struct func *func);
+};
+
/**
* Stored function.
*/
struct func {
struct func_def *def;
- /**
- * Anchor for module membership.
- */
- struct rlist item;
- /**
- * For C functions, the body of the function.
- */
- box_function_f func;
- /**
- * Each stored function keeps a handle to the
- * dynamic library for the C callback.
- */
- struct module *module;
+ /** Virtual method table. */
+ struct func_vtab *vtab;
/**
* Authentication id of the owner of the function,
* used for set-user-id functions.
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 4c9738aea..7df6678d8 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -67,6 +67,8 @@ struct func_def {
* The language of the stored function.
*/
enum func_language language;
+ /** The length of the function name. */
+ uint32_t name_len;
/** Function name. */
char name[0];
};
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index e32845095..8d0328ef7 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -31,6 +31,8 @@
#include "box/lua/call.h"
#include "box/call.h"
#include "box/error.h"
+#include "box/func.h"
+#include "box/func_def.h"
#include "fiber.h"
#include "lua/utils.h"
@@ -523,6 +525,50 @@ box_lua_eval(const char *expr, uint32_t expr_len,
return box_process_lua(execute_lua_eval, &ctx, in_port, out_port);
}
+struct func_lua {
+ /** Function object base class. */
+ struct func base;
+};
+
+static struct func_vtab func_lua_vtab;
+
+struct func *
+func_lua_new(struct func_def *def)
+{
+ (void) def;
+ assert(def->language == FUNC_LANGUAGE_LUA);
+ struct func_lua *func =
+ (struct func_lua *) malloc(sizeof(struct func_lua));
+ if (func == NULL) {
+ diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
+ return NULL;
+ }
+ func->base.vtab = &func_lua_vtab;
+ return &func->base;
+}
+
+static void
+func_lua_unload(struct func *base)
+{
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA);
+ assert(base->vtab == &func_lua_vtab);
+ (void) base;
+}
+
+static inline int
+func_lua_call(struct func *func, struct port *in_port, struct port *out_port)
+{
+ assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA);
+ assert(func->vtab == &func_lua_vtab);
+ return box_lua_call(func->def->name, func->def->name_len,
+ in_port, out_port);
+}
+
+static struct func_vtab func_lua_vtab = {
+ .call = func_lua_call,
+ .destroy = func_lua_unload,
+};
+
static int
lbox_module_reload(lua_State *L)
{
diff --git a/src/box/lua/call.h b/src/box/lua/call.h
index 1801d5090..ef881a048 100644
--- a/src/box/lua/call.h
+++ b/src/box/lua/call.h
@@ -44,6 +44,7 @@ box_lua_call_init(struct lua_State *L);
struct port;
struct call_request;
+struct func_def;
/**
* Invoke a Lua stored procedure from the binary protocol
@@ -57,6 +58,10 @@ int
box_lua_eval(const char *expr, uint32_t expr_len,
struct port *in_port, struct port *out_port);
+/** Construct a Lua function object. */
+struct func *
+func_lua_new(struct func_def *def);
+
#if defined(__cplusplus)
} /* extern "C" */
#endif /* defined(__cplusplus) */
diff --git a/src/box/session.cc b/src/box/session.cc
index 4bb13d031..7ec03b4d0 100644
--- a/src/box/session.cc
+++ b/src/box/session.cc
@@ -297,7 +297,8 @@ access_check_session(struct user *user)
}
int
-access_check_universe(user_access_t access)
+access_check_universe_object(user_access_t access, enum schema_object_type type,
+ const char *object_name)
{
struct credentials *credentials = effective_user();
access |= PRIV_U;
@@ -313,7 +314,7 @@ access_check_universe(user_access_t access)
if (user != NULL) {
diag_set(AccessDeniedError,
priv_name(denied_access),
- schema_object_name(SC_UNIVERSE), "",
+ schema_object_name(type), object_name,
user->def->name);
} else {
/*
@@ -328,6 +329,12 @@ access_check_universe(user_access_t access)
return 0;
}
+int
+access_check_universe(user_access_t access)
+{
+ return access_check_universe_object(access, SC_UNIVERSE, "");
+}
+
int
generic_session_push(struct session *session, uint64_t sync, struct port *port)
{
diff --git a/src/box/session.h b/src/box/session.h
index 3a7397146..6bafb8561 100644
--- a/src/box/session.h
+++ b/src/box/session.h
@@ -312,6 +312,14 @@ access_check_session(struct user *user);
int
access_check_universe(user_access_t access);
+/**
+ * Check whether or not the current user can be granted
+ * the requested access to the universe object.
+ */
+int
+access_check_universe_object(user_access_t access, enum schema_object_type type,
+ const char *object_name);
+
static inline int
session_push(struct session *session, uint64_t sync, struct port *port)
{
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 4/6] box: export registered functions in box.func folder
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
` (2 preceding siblings ...)
2019-06-13 14:08 ` [PATCH v3 3/6] box: rework func object as a function frontend Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-18 14:06 ` Vladimir Davydov
2019-06-18 16:11 ` Vladimir Davydov
2019-06-13 14:08 ` [PATCH v3 5/6] box: refactor box_lua_find helper Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 6/6] box: introduce Lua persistent functions Kirill Shcherbatov
5 siblings, 2 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Needed for #4182, #1260
@TarantoolBot document
Title: Export registered functions to box.func folder
Now all registered with box.schema.func.create functions are
exported in box.func folder.
Each function have :call and :drop method. The :drop method
just a shortcut for box.schema.func.drop interface.
The :call method is similar to net.box connection:call method
and allows to call a registered function directly.
All access checks are performed on each function call.
Example:
function sum(a, b) return a + b end
box.schema.func.create('sum')
box.func.sum
---
- language: LUA
setuid: false
name: sum
id: 2
...
box.func.sum:call({1, 3})
---
- 4
...
box.func.sum:drop()
---
src/box/alter.cc | 2 +
src/box/func.h | 1 +
src/box/lua/call.c | 224 ++++++++++++++++++++++++++++++++----
src/box/lua/init.c | 1 +
src/box/lua/misc.cc | 7 +-
src/box/lua/schema.lua | 29 +++++
src/box/port.h | 4 +-
src/box/schema.cc | 1 +
src/box/schema.h | 5 +
test/box/function1.c | 33 ++++++
test/box/function1.result | 205 +++++++++++++++++++++++++++++++++
test/box/function1.test.lua | 66 +++++++++++
test/box/misc.result | 1 +
13 files changed, 554 insertions(+), 25 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 7b0eb3334..081786820 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2579,6 +2579,7 @@ func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
struct func *old_func = (struct func *) trigger->data;
func_cache_delete(old_func->def->fid);
+ trigger_run_xc(&on_alter_func, old_func);
func_delete(old_func);
}
@@ -2612,6 +2613,7 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
});
def_guard.is_active = false;
func_cache_insert(func);
+ trigger_run_xc(&on_alter_func, func);
func_guard.is_active = false;
struct trigger *on_rollback =
txn_alter_trigger_new(func_cache_remove_func, func);
diff --git a/src/box/func.h b/src/box/func.h
index 1271bde67..f7081eccd 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -44,6 +44,7 @@ extern "C" {
struct port;
struct func;
+struct lua_State;
struct box_function_ctx {
struct port *port;
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 8d0328ef7..23eb29eb8 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -33,10 +33,13 @@
#include "box/error.h"
#include "box/func.h"
#include "box/func_def.h"
+#include "box/schema.h"
#include "fiber.h"
+#include "tt_static.h"
#include "lua/utils.h"
#include "lua/msgpack.h"
+#include "lua/trigger.h"
#include "box/xrow.h"
#include "box/port.h"
@@ -397,16 +400,12 @@ encode_lua_call(lua_State *L)
*
* TODO: forbid explicit yield from __serialize or __index here
*/
- struct mpstream stream;
- mpstream_init(&stream, port->out, obuf_reserve_cb, obuf_alloc_cb,
- luamp_error, port->L);
-
struct luaL_serializer *cfg = luaL_msgpack_default;
int size = lua_gettop(port->L);
for (int i = 1; i <= size; ++i)
- luamp_encode(port->L, cfg, &stream, i);
+ luamp_encode(port->L, cfg, port->stream, i);
port->size = size;
- mpstream_flush(&stream);
+ mpstream_flush(port->stream);
return 0;
}
@@ -419,23 +418,20 @@ encode_lua_call_16(lua_State *L)
*
* TODO: forbid explicit yield from __serialize or __index here
*/
- struct mpstream stream;
- mpstream_init(&stream, port->out, obuf_reserve_cb, obuf_alloc_cb,
- luamp_error, port->L);
-
struct luaL_serializer *cfg = luaL_msgpack_default;
- port->size = luamp_encode_call_16(port->L, cfg, &stream);
- mpstream_flush(&stream);
+ port->size = luamp_encode_call_16(port->L, cfg, port->stream);
+ mpstream_flush(port->stream);
return 0;
}
static inline int
-port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler)
+port_lua_do_dump(struct port *base, struct mpstream *stream,
+ lua_CFunction handler)
{
- struct port_lua *port = (struct port_lua *)base;
+ struct port_lua *port = (struct port_lua *) base;
assert(port->vtab == &port_lua_vtab);
/* Use port to pass arguments to encoder quickly. */
- port->out = out;
+ port->stream = stream;
/*
* Use the same global state, assuming the encoder doesn't
* yield.
@@ -453,13 +449,56 @@ port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler)
static int
port_lua_dump(struct port *base, struct obuf *out)
{
- return port_lua_do_dump(base, out, encode_lua_call);
+ struct port_lua *port = (struct port_lua *) base;
+ struct mpstream stream;
+ mpstream_init(&stream, out, obuf_reserve_cb, obuf_alloc_cb,
+ luamp_error, port->L);
+ return port_lua_do_dump(base, &stream, encode_lua_call);
}
static int
port_lua_dump_16(struct port *base, struct obuf *out)
{
- return port_lua_do_dump(base, out, encode_lua_call_16);
+ struct port_lua *port = (struct port_lua *)base;
+ struct mpstream stream;
+ mpstream_init(&stream, out, obuf_reserve_cb, obuf_alloc_cb,
+ luamp_error, port->L);
+ return port_lua_do_dump(base, &stream, encode_lua_call_16);
+}
+
+static void
+port_lua_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
+{
+ assert(is_flat == true);
+ struct port_lua *port = (struct port_lua *) base;
+ uint32_t size = lua_gettop(port->L);
+ lua_xmove(port->L, L, size);
+ port->size = size;
+}
+
+static const char *
+port_lua_get_msgpack(struct port *base, uint32_t *size)
+{
+ struct port_lua *port = (struct port_lua *) base;
+ struct region *region = &fiber()->gc;
+ uint32_t region_svp = region_used(region);
+ struct mpstream stream;
+ mpstream_init(&stream, region, region_reserve_cb, region_alloc_cb,
+ luamp_error, port->L);
+ mpstream_encode_array(&stream, lua_gettop(port->L));
+ int rc = port_lua_do_dump(base, &stream, encode_lua_call);
+ if (rc < 0) {
+ region_truncate(region, region_svp);
+ return NULL;
+ }
+ *size = region_used(region) - region_svp;
+ const char *data = region_join(region, *size);
+ if (data == NULL) {
+ diag_set(OutOfMemory, *size, "region", "data");
+ region_truncate(region, region_svp);
+ return NULL;
+ }
+ return data;
}
static void
@@ -480,9 +519,9 @@ port_lua_dump_plain(struct port *port, uint32_t *size);
static const struct port_vtab port_lua_vtab = {
.dump_msgpack = port_lua_dump,
.dump_msgpack_16 = port_lua_dump_16,
- .dump_lua = NULL,
+ .dump_lua = port_lua_dump_lua,
.dump_plain = port_lua_dump_plain,
- .get_msgpack = NULL,
+ .get_msgpack = port_lua_get_msgpack,
.destroy = port_lua_destroy,
};
@@ -578,10 +617,150 @@ lbox_module_reload(lua_State *L)
return 0;
}
+int
+lbox_func_call(struct lua_State *L)
+{
+ if (lua_gettop(L) < 2 || !lua_isstring(L, 1))
+ return luaL_error(L, "Use func:call(...)");
+
+ size_t name_len;
+ const char *name = lua_tolstring(L, 1, &name_len);
+ struct func *func = func_by_name(name, name_len);
+ if (func == NULL) {
+ diag_set(ClientError, ER_NO_SUCH_FUNCTION,
+ tt_cstr(name, name_len));
+ return luaT_error(L);
+ }
+
+ /*
+ * Prepare a new Lua stack for input arguments
+ * before the function call to pass it into the
+ * pcall-sandboxed tarantool_L handler.
+ */
+ lua_State *args_L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_xmove(L, args_L, lua_gettop(L) - 1);
+ struct port in_port;
+ port_lua_create(&in_port, args_L);
+ ((struct port_lua *) &in_port)->ref = coro_ref;
+
+ struct port out_port;
+ if (func_call(func, &in_port, &out_port) != 0) {
+ port_destroy(&in_port);
+ return luaT_error(L);
+ }
+
+ int top = lua_gettop(L);
+ port_dump_lua(&out_port, L, true);
+ int cnt = lua_gettop(L) - top;
+
+ port_destroy(&out_port);
+ port_destroy(&in_port);
+ return cnt;
+}
+
+static void
+lbox_func_new(struct lua_State *L, struct func *func)
+{
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ if (!lua_istable(L, -1)) {
+ lua_pop(L, 1); /* pop nil */
+ lua_newtable(L);
+ lua_setfield(L, -2, "func");
+ lua_getfield(L, -1, "func");
+ }
+ lua_rawgeti(L, -1, func->def->fid);
+ if (lua_isnil(L, -1)) {
+ /*
+ * If the function already exists, modify it,
+ * rather than create a new one -- to not
+ * invalidate Lua variable references to old func
+ * outside the box.schema.func[].
+ */
+ lua_pop(L, 1);
+ lua_newtable(L);
+ lua_rawseti(L, -2, func->def->fid);
+ lua_rawgeti(L, -1, func->def->fid);
+ } else {
+ /* Clear the reference to old func by old name. */
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_settable(L, -4);
+ }
+ int top = lua_gettop(L);
+ lua_pushstring(L, "id");
+ lua_pushnumber(L, func->def->fid);
+ lua_settable(L, top);
+ lua_pushstring(L, "name");
+ lua_pushstring(L, func->def->name);
+ lua_settable(L, top);
+ lua_pushstring(L, "setuid");
+ lua_pushboolean(L, func->def->setuid);
+ lua_settable(L, top);
+ lua_pushstring(L, "language");
+ lua_pushstring(L, func_language_strs[func->def->language]);
+ lua_settable(L, top);
+
+ /* Bless func object. */
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_pushstring(L, "schema");
+ lua_gettable(L, -2);
+ lua_pushstring(L, "func");
+ lua_gettable(L, -2);
+ lua_pushstring(L, "bless");
+ lua_gettable(L, -2);
+
+ lua_pushvalue(L, top);
+ lua_call(L, 1, 0);
+ lua_pop(L, 3);
+
+ lua_setfield(L, -2, func->def->name);
+
+ lua_pop(L, 2);
+}
+
+static void
+lbox_func_delete(struct lua_State *L, struct func *func)
+{
+ uint32_t fid = func->def->fid;
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ assert(!lua_isnil(L, -1));
+ lua_rawgeti(L, -1, fid);
+ if (!lua_isnil(L, -1)) {
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_rawset(L, -4);
+ lua_pop(L, 1); /* pop func */
+ lua_pushnil(L);
+ lua_rawseti(L, -2, fid);
+ } else {
+ lua_pop(L, 1);
+ }
+ lua_pop(L, 2); /* box, func */
+}
+
+static void
+lbox_func_new_or_delete(struct trigger *trigger, void *event)
+{
+ struct lua_State *L = (struct lua_State *) trigger->data;
+ struct func *func = (struct func *)event;
+ if (func != NULL)
+ lbox_func_new(L, func);
+ else
+ lbox_func_delete(L, func);
+}
+
+static struct trigger on_alter_func_in_lua = {
+ RLIST_LINK_INITIALIZER, lbox_func_new_or_delete, NULL, NULL
+};
+
static const struct luaL_Reg boxlib_internal[] = {
{"call_loadproc", lbox_call_loadproc},
{"sql_create_function", lbox_sql_create_function},
{"module_reload", lbox_module_reload},
+ {"func_call", lbox_func_call},
{NULL, NULL}
};
@@ -590,7 +769,12 @@ box_lua_call_init(struct lua_State *L)
{
luaL_register(L, "box.internal", boxlib_internal);
lua_pop(L, 1);
-
+ /*
+ * Register the trigger that will push persistent
+ * Lua functions objects to Lua.
+ */
+ on_alter_func_in_lua.data = L;
+ trigger_add(&on_alter_func, &on_alter_func_in_lua);
#if 0
/* Get CTypeID for `struct port *' */
int rc = luaL_cdef(L, "struct port;");
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index 76b987b4b..7ffed409d 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -39,6 +39,7 @@
#include "box/box.h"
#include "box/txn.h"
+#include "box/func.h"
#include "box/vclock.h"
#include "box/lua/error.h"
diff --git a/src/box/lua/misc.cc b/src/box/lua/misc.cc
index 15681c377..3d894b09f 100644
--- a/src/box/lua/misc.cc
+++ b/src/box/lua/misc.cc
@@ -63,13 +63,14 @@ lbox_encode_tuple_on_gc(lua_State *L, int idx, size_t *p_len)
extern "C" void
port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
{
- assert(is_flat == false);
struct port_tuple *port = port_tuple(base);
- lua_createtable(L, port->size, 0);
+ if (!is_flat)
+ lua_createtable(L, port->size, 0);
struct port_tuple_entry *pe = port->first;
for (int i = 0; pe != NULL; pe = pe->next) {
luaT_pushtuple(L, pe->tuple);
- lua_rawseti(L, -2, ++i);
+ if (!is_flat)
+ lua_rawseti(L, -2, ++i);
}
}
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 3f080eced..9c3ee063c 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2190,7 +2190,36 @@ function box.schema.func.exists(name_or_id)
return tuple ~= nil
end
+-- Helper function to check func:method() usage
+local function check_func_arg(func, method)
+ if type(func) ~= 'table' or func.name == nil then
+ local fmt = 'Use func:%s(...) instead of func.%s(...)'
+ error(string.format(fmt, method, method))
+ end
+end
+
+local func_mt = {}
+
+func_mt.drop = function(func, opts)
+ check_func_arg(func, 'drop')
+ box.schema.func.drop(func.name, opts)
+end
+
+func_mt.call = function(func, args)
+ check_func_arg(func, 'call')
+ args = args or {}
+ if type(args) ~= 'table' then
+ error('Use func:call(table)')
+ end
+ return box.schema.func.call(func.name, unpack(args))
+end
+
+function box.schema.func.bless(func)
+ setmetatable(func, {__index = func_mt})
+end
+
box.schema.func.reload = internal.module_reload
+box.schema.func.call = internal.func_call
box.internal.collation = {}
box.internal.collation.create = function(name, coll_type, locale, opts)
diff --git a/src/box/port.h b/src/box/port.h
index 4c0c3cf7a..8765327c7 100644
--- a/src/box/port.h
+++ b/src/box/port.h
@@ -102,8 +102,8 @@ struct port_lua {
struct lua_State *L;
/** Reference to L in tarantool_L. */
int ref;
- /** The argument of port_dump */
- struct obuf *out;
+ /** The argument to dump a port. */
+ struct mpstream *stream;
/** Number of entries dumped to the port. */
int size;
};
diff --git a/src/box/schema.cc b/src/box/schema.cc
index f834610a5..ead6079b5 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -72,6 +72,7 @@ uint32_t space_cache_version = 0;
struct rlist on_schema_init = RLIST_HEAD_INITIALIZER(on_schema_init);
struct rlist on_alter_space = RLIST_HEAD_INITIALIZER(on_alter_space);
struct rlist on_alter_sequence = RLIST_HEAD_INITIALIZER(on_alter_sequence);
+struct rlist on_alter_func = RLIST_HEAD_INITIALIZER(on_alter_func);
/**
* Lock of scheme modification
diff --git a/src/box/schema.h b/src/box/schema.h
index d7b3c4c2d..9ff4ffa1c 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -238,6 +238,11 @@ extern struct rlist on_alter_sequence;
*/
extern struct rlist on_access_denied;
+/**
+ * Triggers fired after committing a change in _func space.
+ */
+extern struct rlist on_alter_func;
+
/**
* Context passed to on_access_denied trigger.
*/
diff --git a/test/box/function1.c b/test/box/function1.c
index 053e4fe91..ee5a422b5 100644
--- a/test/box/function1.c
+++ b/test/box/function1.c
@@ -42,6 +42,39 @@ args(box_function_ctx_t *ctx, const char *args, const char *args_end)
return box_return_tuple(ctx, tuple);
}
+int
+divide(box_function_ctx_t *ctx, const char *args, const char *args_end)
+{
+ uint32_t arg_count = mp_decode_array(&args);
+ if (arg_count < 2)
+ goto error;
+
+ if (mp_typeof(*args) != MP_UINT)
+ goto error;
+ double a = mp_decode_uint(&args);
+ if (mp_typeof(*args) != MP_UINT)
+ goto error;
+ double b = mp_decode_uint(&args);
+ if (b == 0)
+ goto error;
+
+ char tuple_buf[512];
+ char *d = tuple_buf;
+ d = mp_encode_array(d, 1);
+ d = mp_encode_double(d, a / b);
+ assert(d <= tuple_buf + sizeof(tuple_buf));
+
+ box_tuple_format_t *fmt = box_tuple_format_default();
+ box_tuple_t *tuple = box_tuple_new(fmt, tuple_buf, d);
+ if (tuple == NULL)
+ return -1;
+ return box_return_tuple(ctx, tuple);
+error:
+ return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s",
+ "invalid argument");
+}
+
+
/*
* For each UINT key in arguments create or increment counter in
* box.space.test space.
diff --git a/test/box/function1.result b/test/box/function1.result
index cadeb0467..331bd466a 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -57,6 +57,25 @@ c:call('function1.args', { 15 })
---
- [[15, 'hello']]
...
+box.func["function1.args"]
+---
+- language: C
+ setuid: false
+ name: function1.args
+ id: 2
+...
+box.func["function1.args"]:call()
+---
+- error: Use func:call(...)
+...
+box.func["function1.args"]:call({ "xx" })
+---
+- error: first tuple field must be uint
+...
+box.func["function1.args"]:call({ 15 })
+---
+- [15, 'hello']
+...
box.schema.func.drop("function1.args")
---
...
@@ -299,3 +318,189 @@ test_run:cmd("setopt delimiter ''");
c:close()
---
...
+-- Test registered functions interface.
+function divide(a, b) return a / b end
+---
+...
+box.schema.func.create("divide")
+---
+...
+func = box.func.divide
+---
+...
+func.call({4, 2})
+---
+- error: 'builtin/box/schema.lua:2197: Use func:call(...) instead of func.call(...)'
+...
+func:call(4, 2)
+---
+- error: 'builtin/box/schema.lua:2212: Use func:call(table)'
+...
+func:call()
+---
+- error: Use func:call(...)
+...
+func:call({})
+---
+- error: Use func:call(...)
+...
+func:call({4})
+---
+- error: '[string "function divide(a, b) return a / b end "]:1: attempt to perform
+ arithmetic on local ''b'' (a nil value)'
+...
+func:call({4, 2})
+---
+- 2
+...
+func:call({4, 2, 1})
+---
+- 2
+...
+func:drop()
+---
+...
+func
+---
+- language: LUA
+ setuid: false
+ name: divide
+ id: 2
+...
+func.drop()
+---
+- error: 'builtin/box/schema.lua:2197: Use func:drop(...) instead of func.drop(...)'
+...
+func:drop()
+---
+- error: Function 'divide' does not exist
+...
+func:call({4, 2})
+---
+- error: Function 'divide' does not exist
+...
+box.internal.func_call('divide', 4, 2)
+---
+- error: Function 'divide' does not exist
+...
+box.schema.func.create("function1.divide", {language = 'C'})
+---
+...
+func = box.func["function1.divide"]
+---
+...
+func:call(4, 2)
+---
+- error: 'builtin/box/schema.lua:2212: Use func:call(table)'
+...
+func:call()
+---
+- error: Use func:call(...)
+...
+func:call({})
+---
+- error: Use func:call(...)
+...
+func:call({4})
+---
+- error: invalid argument
+...
+func:call({4, 2})
+---
+- [2]
+...
+func:call({4, 2, 1})
+---
+- [2]
+...
+func:drop()
+---
+...
+func
+---
+- language: C
+ setuid: false
+ name: function1.divide
+ id: 2
+...
+func:drop()
+---
+- error: Function 'function1.divide' does not exist
+...
+func:call({4, 2})
+---
+- error: Function 'function1.divide' does not exist
+...
+box.internal.func_call('function1.divide', 4, 2)
+---
+- error: Function 'function1.divide' does not exist
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+function minmax(array)
+ local min = 999
+ local max = -1
+ for _, v in pairs(array) do
+ min = math.min(min, v)
+ max = math.max(max, v)
+ end
+ return min, max
+end
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create("minmax")
+---
+...
+func = box.func.minmax
+---
+...
+func:call({{1, 2, 99, 3, -1}})
+---
+- -1
+- 99
+...
+func:drop()
+---
+...
+-- Test access checks for registered functions.
+function secret() return 1 end
+---
+...
+box.schema.func.create("secret")
+---
+...
+box.func.secret:call({})
+---
+- error: Use func:call(...)
+...
+function secret_leak() return box.func.secret:call() end
+---
+...
+box.schema.func.create('secret_leak')
+---
+...
+box.schema.user.grant('guest', 'execute', 'function', 'secret_leak')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('secret_leak')
+---
+- error: Use func:call(...)
+...
+conn:close()
+---
+...
+box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
+---
+...
+box.schema.func.drop('secret_leak')
+---
+...
+box.schema.func.drop('secret')
+---
+...
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index e983495b6..cf4a71979 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -21,6 +21,10 @@ box.schema.user.grant('guest', 'execute', 'function', 'function1.args')
c:call('function1.args')
c:call('function1.args', { "xx" })
c:call('function1.args', { 15 })
+box.func["function1.args"]
+box.func["function1.args"]:call()
+box.func["function1.args"]:call({ "xx" })
+box.func["function1.args"]:call({ 15 })
box.schema.func.drop("function1.args")
box.schema.func.create('function1.multi_inc', {language = "C"})
@@ -112,3 +116,65 @@ identifier.run_test(
);
test_run:cmd("setopt delimiter ''");
c:close()
+
+-- Test registered functions interface.
+function divide(a, b) return a / b end
+box.schema.func.create("divide")
+func = box.func.divide
+func.call({4, 2})
+func:call(4, 2)
+func:call()
+func:call({})
+func:call({4})
+func:call({4, 2})
+func:call({4, 2, 1})
+func:drop()
+func
+func.drop()
+func:drop()
+func:call({4, 2})
+box.internal.func_call('divide', 4, 2)
+
+box.schema.func.create("function1.divide", {language = 'C'})
+func = box.func["function1.divide"]
+func:call(4, 2)
+func:call()
+func:call({})
+func:call({4})
+func:call({4, 2})
+func:call({4, 2, 1})
+func:drop()
+func
+func:drop()
+func:call({4, 2})
+box.internal.func_call('function1.divide', 4, 2)
+
+test_run:cmd("setopt delimiter ';'")
+function minmax(array)
+ local min = 999
+ local max = -1
+ for _, v in pairs(array) do
+ min = math.min(min, v)
+ max = math.max(max, v)
+ end
+ return min, max
+end
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create("minmax")
+func = box.func.minmax
+func:call({{1, 2, 99, 3, -1}})
+func:drop()
+
+-- Test access checks for registered functions.
+function secret() return 1 end
+box.schema.func.create("secret")
+box.func.secret:call({})
+function secret_leak() return box.func.secret:call() end
+box.schema.func.create('secret_leak')
+box.schema.user.grant('guest', 'execute', 'function', 'secret_leak')
+conn = net.connect(box.cfg.listen)
+conn:call('secret_leak')
+conn:close()
+box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
+box.schema.func.drop('secret_leak')
+box.schema.func.drop('secret')
diff --git a/test/box/misc.result b/test/box/misc.result
index 43b5a4a15..030c87c55 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -65,6 +65,7 @@ t
- error
- execute
- feedback
+ - func
- index
- info
- internal
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 5/6] box: refactor box_lua_find helper
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
` (3 preceding siblings ...)
2019-06-13 14:08 ` [PATCH v3 4/6] box: export registered functions in box.func folder Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-18 14:22 ` Vladimir Davydov
2019-06-13 14:08 ` [PATCH v3 6/6] box: introduce Lua persistent functions Kirill Shcherbatov
5 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
The box_lua_find routine used to work with an empty stack only.
It is unacceptable in following patches because this helper
need to be reused in following patches for environment table
construction.
Refactored function is renamed to luaT_func_find.
Needed for #4182, #1260
---
src/box/lua/call.c | 54 +++++++++++++++++++++++++---------------------
1 file changed, 30 insertions(+), 24 deletions(-)
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 23eb29eb8..90c28a160 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -54,21 +54,19 @@
* on top of the stack.
*/
static int
-box_lua_find(lua_State *L, const char *name, const char *name_end)
+luaT_func_find(lua_State *L, const char *name, const char *name_end,
+ int *count)
{
int index = LUA_GLOBALSINDEX;
- int objstack = 0;
+ int objstack = 0, top = lua_gettop(L);
const char *start = name, *end;
while ((end = (const char *) memchr(start, '.', name_end - start))) {
lua_checkstack(L, 3);
lua_pushlstring(L, start, end - start);
lua_gettable(L, index);
- if (! lua_istable(L, -1)) {
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
- }
+ if (! lua_istable(L, -1))
+ return -1;
start = end + 1; /* next piece of a.b.c */
index = lua_gettop(L); /* top of the stack */
}
@@ -79,14 +77,11 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
lua_pushlstring(L, start, end - start);
lua_gettable(L, index);
if (! (lua_istable(L, -1) ||
- lua_islightuserdata(L, -1) || lua_isuserdata(L, -1) )) {
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
- }
+ lua_islightuserdata(L, -1) || lua_isuserdata(L, -1)))
+ return -1;
start = end + 1; /* next piece of a.b.c */
index = lua_gettop(L); /* top of the stack */
- objstack = index;
+ objstack = index - top;
}
@@ -95,25 +90,27 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
if (!lua_isfunction(L, -1) && !lua_istable(L, -1)) {
/* lua_call or lua_gettable would raise a type error
* for us, but our own message is more verbose. */
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
+ return -1;
}
+
/* setting stack that it would contain only
* the function pointer. */
if (index != LUA_GLOBALSINDEX) {
if (objstack == 0) { /* no object, only a function */
- lua_replace(L, 1);
+ lua_replace(L, top + 1);
+ lua_pop(L, lua_gettop(L) - top - 1);
} else if (objstack == 1) { /* just two values, swap them */
lua_insert(L, -2);
+ lua_pop(L, lua_gettop(L) - top - 2);
} else { /* long path */
- lua_insert(L, 1);
- lua_insert(L, 2);
+ lua_insert(L, top + 1);
+ lua_insert(L, top + 2);
+ lua_pop(L, objstack - 1);
objstack = 1;
}
- lua_settop(L, 1 + objstack);
}
- return 1 + objstack;
+ *count = 1 + objstack;
+ return 0;
}
/**
@@ -129,7 +126,12 @@ lbox_call_loadproc(struct lua_State *L)
const char *name;
size_t name_len;
name = lua_tolstring(L, 1, &name_len);
- return box_lua_find(L, name, name + name_len);
+ int count;
+ if (luaT_func_find(L, name, name + name_len, &count) != 0) {
+ diag_set(ClientError, ER_NO_SUCH_PROC, name_len, name);
+ return luaT_error(L);
+ }
+ return count;
}
/*
@@ -352,9 +354,13 @@ execute_lua_call(lua_State *L)
const char *name = ctx->name;
uint32_t name_len = ctx->name_len;
- int oc = 0; /* how many objects are on stack after box_lua_find */
+ /* How many objects are on stack after luaT_func_find. */
+ int oc = 0;
/* Try to find a function by name in Lua */
- oc = box_lua_find(L, name, name + name_len);
+ if (luaT_func_find(L, name, name + name_len, &oc) != 0) {
+ diag_set(ClientError, ER_NO_SUCH_PROC, name_len, name);
+ return luaT_error(L);
+ }
/* Push the rest of args (a tuple). */
int top = lua_gettop(L);
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v3 6/6] box: introduce Lua persistent functions
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
` (4 preceding siblings ...)
2019-06-13 14:08 ` [PATCH v3 5/6] box: refactor box_lua_find helper Kirill Shcherbatov
@ 2019-06-13 14:08 ` Kirill Shcherbatov
2019-06-18 16:23 ` Vladimir Davydov
5 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-13 14:08 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
This patch changes a format of the _func system space
to support persistent Lua functions. The format of the
system space is
[<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
<language> STR, <body> STR, <is_deterministic> BOOL,
<is_sandboxed>, <opts> MAP]
Closes #4182
Needed for #1260
@TarantoolBot document
Title: Persistent Lua functions
Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshoot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"
A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
a. only limited set of Lua functions and modules are available:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8;
b. global variables are forbidden
Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.
The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
<if_not_exists = true|FALSE>, <language = LUA|c>,
<body = string ('')>, <is_deterministic = true|FALSE>,
<is_sandboxed = true|FALSE>)
Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
is_deterministic = true, is_sandboxed = true})
box.func.sum
---
- is_sandboxed: true
is_deterministic: true
id: 2
setuid: false
body: function(a, b) return a + b end
name: sum
language: LUA
...
box.func.sum:call({1, 3})
---
- 4
...
---
src/box/alter.cc | 53 +++++++---
src/box/bootstrap.snap | Bin 4475 -> 4532 bytes
src/box/func.c | 7 +-
src/box/func_def.c | 8 ++
src/box/func_def.h | 20 +++-
src/box/lua/call.c | 182 ++++++++++++++++++++++++++++++++++-
src/box/lua/schema.lua | 12 ++-
src/box/lua/upgrade.lua | 25 ++++-
src/box/schema_def.h | 4 +
test/box-py/bootstrap.result | 6 +-
test/box/access_misc.result | 6 +-
test/box/function1.result | 145 +++++++++++++++++++++++++---
test/box/function1.test.lua | 50 +++++++++-
13 files changed, 481 insertions(+), 37 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 081786820..5232a56c1 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2537,27 +2537,45 @@ func_def_get_ids_from_tuple(struct tuple *tuple, uint32_t *fid, uint32_t *uid)
static struct func_def *
func_def_new_from_tuple(struct tuple *tuple)
{
- uint32_t len;
- const char *name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME,
- &len);
- if (len > BOX_NAME_MAX)
+ uint32_t field_count = tuple_field_count(tuple);
+ uint32_t name_len, body_len;
+ const char *name, *body;
+ name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME, &name_len);
+ if (name_len > BOX_NAME_MAX) {
tnt_raise(ClientError, ER_CREATE_FUNCTION,
tt_cstr(name, BOX_INVALID_NAME_MAX),
"function name is too long");
- identifier_check_xc(name, len);
- struct func_def *def = (struct func_def *) malloc(func_def_sizeof(len));
+ }
+ identifier_check_xc(name, name_len);
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ body = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_BODY,
+ &body_len);
+ } else {
+ body = NULL;
+ body_len = 0;
+ }
+ uint32_t def_sz = func_def_sizeof(name_len, body_len);
+ struct func_def *def =
+ (struct func_def *) malloc(def_sz);
if (def == NULL)
- tnt_raise(OutOfMemory, func_def_sizeof(len), "malloc", "def");
+ tnt_raise(OutOfMemory, def_sz, "malloc", "def");
auto def_guard = make_scoped_guard([=] { free(def); });
func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid);
- memcpy(def->name, name, len);
- def->name[len] = 0;
- def->name_len = len;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
+ memcpy(def->name, name, name_len);
+ def->name[name_len] = 0;
+ def->name_len = name_len;
+ if (body_len > 0) {
+ def->body = def->name + name_len + 1;
+ memcpy(def->body, body, body_len);
+ def->body[body_len] = 0;
+ } else {
+ def->body = NULL;
+ }
+ if (field_count > BOX_FUNC_FIELD_SETUID)
def->setuid = tuple_field_u32_xc(tuple, BOX_FUNC_FIELD_SETUID);
else
def->setuid = false;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_LANGUAGE) {
+ if (field_count > BOX_FUNC_FIELD_LANGUAGE) {
const char *language =
tuple_field_cstr_xc(tuple, BOX_FUNC_FIELD_LANGUAGE);
def->language = STR2ENUM(func_language, language);
@@ -2569,6 +2587,17 @@ func_def_new_from_tuple(struct tuple *tuple)
/* Lua is the default. */
def->language = FUNC_LANGUAGE_LUA;
}
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ assert(field_count > BOX_FUNC_FIELD_OPTS);
+ def->is_deterministic =
+ tuple_field_bool_xc(tuple,
+ BOX_FUNC_FIELD_IS_DETERMINISTIC);
+ def->is_sandboxed =
+ tuple_field_bool_xc(tuple,
+ BOX_FUNC_FIELD_IS_SANDBOXED);
+ } else {
+ def->is_deterministic = false;
+ }
def_guard.is_active = false;
return def;
}
diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap
index 56943ef7e7d0fe0e3dbb5d346544e2c78c3dc154..fb313d66eaa965afb373ef7820804365309886bd 100644
GIT binary patch
literal 4532
diff --git a/src/box/func.c b/src/box/func.c
index 20ff56668..f0ec97c27 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -417,8 +417,13 @@ static struct func_vtab func_c_vtab;
struct func *
func_c_new(struct func_def *def)
{
- (void) def;
assert(def->language == FUNC_LANGUAGE_C);
+ if (def->body != NULL || def->is_sandboxed) {
+ diag_set(ClientError, ER_CREATE_FUNCTION, def->name,
+ "body and is_sandboxed options are not compatible "
+ "with C language");
+ return NULL;
+ }
struct func_c *func = (struct func_c *) malloc(sizeof(struct func_c));
if (func == NULL) {
diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
diff --git a/src/box/func_def.c b/src/box/func_def.c
index 2b135e2d7..73e493786 100644
--- a/src/box/func_def.c
+++ b/src/box/func_def.c
@@ -14,7 +14,15 @@ func_def_cmp(struct func_def *def1, struct func_def *def2)
return def1->setuid - def2->setuid;
if (def1->language != def2->language)
return def1->language - def2->language;
+ if (def1->is_deterministic != def2->is_deterministic)
+ return def1->is_deterministic - def2->is_deterministic;
+ if (def1->is_sandboxed != def2->is_sandboxed)
+ return def1->is_sandboxed - def2->is_sandboxed;
if (strcmp(def1->name, def2->name) != 0)
return strcmp(def1->name, def2->name);
+ if ((def1->body != NULL) != (def2->body != NULL))
+ return def1->body - def2->body;
+ if (def1->body != NULL && strcmp(def1->body, def2->body) != 0)
+ return strcmp(def1->body, def2->body);
return 0;
}
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 7df6678d8..4886d5440 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -58,11 +58,24 @@ struct func_def {
uint32_t fid;
/** Owner of the function. */
uint32_t uid;
+ /** Definition of the persistent function. */
+ char *body;
/**
* True if the function requires change of user id before
* invocation.
*/
bool setuid;
+ /**
+ * Whether this function is deterministic (can produce
+ * only one result for a given list of parameters).
+ */
+ bool is_deterministic;
+ /**
+ * Whether the routine mast be initialized with isolated
+ * sandbox where only a limited number if functions is
+ * available.
+ */
+ bool is_sandboxed;
/**
* The language of the stored function.
*/
@@ -79,10 +92,13 @@ struct func_def {
* for a function of length @a a name_len.
*/
static inline size_t
-func_def_sizeof(uint32_t name_len)
+func_def_sizeof(uint32_t name_len, uint32_t body_len)
{
/* +1 for '\0' name terminating. */
- return sizeof(struct func_def) + name_len + 1;
+ size_t sz = sizeof(struct func_def) + name_len + 1;
+ if (body_len > 0)
+ sz += body_len + 1;
+ return sz;
}
/** Compare two given function definitions. */
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 90c28a160..43445373f 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -573,22 +573,142 @@ box_lua_eval(const char *expr, uint32_t expr_len,
struct func_lua {
/** Function object base class. */
struct func base;
+ /**
+ * The reference index of Lua function object.
+ * Is equal to LUA_REFNIL when undefined.
+ */
+ int lua_ref;
};
static struct func_vtab func_lua_vtab;
+static struct func_vtab func_persistent_lua_vtab;
+
+static const char *default_sandbox_exports[] = {
+ "assert", "error", "ipairs", "math", "next", "pairs", "pcall", "print",
+ "select", "string", "table", "tonumber", "tostring", "type", "unpack",
+ "xpcall", "utf8",
+};
+
+/**
+ * Assemble a new sandbox with given exports table on top of the
+ * Lua stack. All modules in exports list are copying deeply
+ * to ensure the immutablility of this system object.
+ */
+static int
+luaT_prepare_sandbox(struct lua_State *L, const char *exports[],
+ uint32_t exports_count)
+{
+ int count, rc = -1;
+ const char *deepcopy = "table.deepcopy";
+ int luaL_deepcopy_func_ref = LUA_REFNIL;
+ if (luaT_func_find(L, deepcopy, deepcopy + strlen(deepcopy),
+ &count) != 0)
+ goto end;
+ luaL_deepcopy_func_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ assert(luaL_deepcopy_func_ref != LUA_REFNIL);
+ lua_createtable(L, exports_count, 0);
+ for (unsigned i = 0; i < exports_count; i++) {
+ int count;
+ uint32_t name_len = strlen(exports[i]);
+ if (luaT_func_find(L, exports[i], exports[i] + name_len,
+ &count) != 0)
+ goto end;
+ switch (lua_type(L, -1)) {
+ case LUA_TTABLE:
+ lua_rawgeti(L, LUA_REGISTRYINDEX,
+ luaL_deepcopy_func_ref);
+ lua_insert(L, -2);
+ lua_call(L, 1, LUA_MULTRET);
+ FALLTHROUGH;
+ case LUA_TFUNCTION:
+ break;
+ default:
+ unreachable();
+ }
+ lua_setfield(L, -2, exports[i]);
+ }
+ rc = 0;
+end:
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, luaL_deepcopy_func_ref);
+ return rc;
+}
+
+static int
+func_persistent_lua_load(struct lua_State *L)
+{
+ struct func_lua *func = (struct func_lua *) lua_topointer(L, 1);
+ assert(func->base.def->body != NULL);
+ lua_settop(L, 0);
+
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+ const char *load_pref = "return ";
+ uint32_t load_str_sz =
+ strlen(load_pref) + strlen(func->base.def->body) + 1;
+ char *load_str = region_alloc(region, load_str_sz);
+ if (load_str == NULL) {
+ diag_set(OutOfMemory, load_str_sz, "region", "load_str");
+ goto error;
+ }
+ sprintf(load_str, "%s%s", load_pref, func->base.def->body);
+ if (luaL_loadstring(L, load_str) != 0 ||
+ luaT_call(L, 0, 1) != 0 || !lua_isfunction(L, -1)) {
+ diag_set(ClientError, ER_LOAD_FUNCTION, func->base.def->name,
+ func->base.def->body);
+ goto error;
+ }
+ if (func->base.def->is_sandboxed) {
+ if (luaT_prepare_sandbox(L, default_sandbox_exports,
+ nelem(default_sandbox_exports)) != 0) {
+ diag_set(ClientError, ER_LOAD_FUNCTION,
+ func->base.def->name, func->base.def->body);
+ goto error;
+ }
+ lua_setfenv(L, -2);
+ }
+ func->lua_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ region_truncate(region, region_svp);
+ return lua_gettop(L);
+error:
+ region_truncate(region, region_svp);
+ return luaT_error(L);
+}
struct func *
func_lua_new(struct func_def *def)
{
- (void) def;
assert(def->language == FUNC_LANGUAGE_LUA);
+ if (def->is_sandboxed && def->body == NULL) {
+ diag_set(ClientError, ER_CREATE_FUNCTION, def->name,
+ "is_sandboxed option may be set only for persistent "
+ "Lua function (when body option is set)");
+ return NULL;
+ }
struct func_lua *func =
(struct func_lua *) malloc(sizeof(struct func_lua));
if (func == NULL) {
diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
return NULL;
}
- func->base.vtab = &func_lua_vtab;
+ if (def->body != NULL) {
+ /** Load persistent Lua function. */
+ struct lua_State *L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_pushcfunction(L, func_persistent_lua_load);
+ func->base.def = def;
+ lua_pushlightuserdata(L, func);
+ int rc = luaT_call(L, 1, LUA_MULTRET);
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, coro_ref);
+ if (rc != 0) {
+ free(func);
+ return NULL;
+ }
+ assert(func->lua_ref != LUA_REFNIL);
+ func->base.vtab = &func_persistent_lua_vtab;
+ } else {
+ func->lua_ref = LUA_REFNIL;
+ func->base.vtab = &func_lua_vtab;
+ }
return &func->base;
}
@@ -614,6 +734,53 @@ static struct func_vtab func_lua_vtab = {
.destroy = func_lua_unload,
};
+static void
+func_persistent_lua_unload(struct func *base)
+{
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
+ base->def->body != NULL);
+ assert(base->vtab == &func_persistent_lua_vtab);
+ struct func_lua *func = (struct func_lua *) base;
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, func->lua_ref);
+ func->lua_ref = LUA_REFNIL;
+}
+
+static int
+func_persistent_lua_call_cb(lua_State *L)
+{
+ struct port *in_port = (struct port *) lua_topointer(L, 1);
+ struct func_lua *func = (struct func_lua *) lua_topointer(L, 2);
+ lua_settop(L, 0); /* clear the stack to simplify the logic below */
+
+ assert(func->lua_ref != LUA_REFNIL);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, func->lua_ref);
+
+ /* Push the rest of args (a tuple). */
+ int top = lua_gettop(L);
+ port_dump_lua(in_port, L, true);
+ int arg_count = lua_gettop(L) - top;
+
+ lua_call(L, arg_count, LUA_MULTRET);
+ return lua_gettop(L);
+}
+
+static inline int
+func_persistent_lua_call(struct func *base, struct port *in_port,
+ struct port *out_port)
+{
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
+ base->def->body != NULL);
+ assert(base->vtab == &func_persistent_lua_vtab);
+ return box_process_lua(func_persistent_lua_call_cb, base,
+ in_port, out_port);
+
+}
+
+static struct func_vtab func_persistent_lua_vtab = {
+ .call = func_persistent_lua_call,
+ .destroy = func_persistent_lua_unload,
+};
+
static int
lbox_module_reload(lua_State *L)
{
@@ -707,6 +874,17 @@ lbox_func_new(struct lua_State *L, struct func *func)
lua_pushstring(L, "language");
lua_pushstring(L, func_language_strs[func->def->language]);
lua_settable(L, top);
+ lua_pushstring(L, "is_deterministic");
+ lua_pushboolean(L, func->def->is_deterministic);
+ lua_settable(L, top);
+ if (func->def->body != NULL) {
+ lua_pushstring(L, "body");
+ lua_pushstring(L, func->def->body);
+ lua_settable(L, top);
+ lua_pushstring(L, "is_sandboxed");
+ lua_pushboolean(L, func->def->is_sandboxed);
+ lua_settable(L, top);
+ }
/* Bless func object. */
lua_getfield(L, LUA_GLOBALSINDEX, "box");
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 9c3ee063c..9d8df54dc 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2138,7 +2138,9 @@ box.schema.func.create = function(name, opts)
opts = opts or {}
check_param_table(opts, { setuid = 'boolean',
if_not_exists = 'boolean',
- language = 'string'})
+ language = 'string', body = 'string',
+ is_deterministic = 'boolean',
+ is_sandboxed = 'boolean', opts = 'table'})
local _func = box.space[box.schema.FUNC_ID]
local _vfunc = box.space[box.schema.VFUNC_ID]
local func = _vfunc.index.name:get{name}
@@ -2148,10 +2150,14 @@ box.schema.func.create = function(name, opts)
end
return
end
- opts = update_param_table(opts, { setuid = false, language = 'lua'})
+ opts = update_param_table(opts, { setuid = false, language = 'lua',
+ body = '', is_deterministic = false,
+ is_sandboxed = false, opts = setmap{}})
opts.language = string.upper(opts.language)
opts.setuid = opts.setuid and 1 or 0
- _func:auto_increment{session.euid(), name, opts.setuid, opts.language}
+ _func:auto_increment{session.euid(), name, opts.setuid, opts.language,
+ opts.body, opts.is_deterministic, opts.is_sandboxed,
+ opts.opts}
end
box.schema.func.drop = function(name, opts)
diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua
index 3385b8e17..f2edf86df 100644
--- a/src/box/lua/upgrade.lua
+++ b/src/box/lua/upgrade.lua
@@ -326,7 +326,8 @@ local function initial_1_7_5()
-- create "box.schema.user.info" function
log.info('create function "box.schema.user.info" with setuid')
- _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA'}
+ _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA',
+ '', false, false, MAP}
-- grant 'public' role access to 'box.schema.user.info' function
log.info('grant execute on function "box.schema.user.info" to public')
@@ -820,10 +821,32 @@ local function create_vcollation_space()
box.space[box.schema.VCOLLATION_ID]:format(format)
end
+local function upgrade_func_to_2_2_1()
+ log.info("Update _func format")
+ local _func = box.space[box.schema.FUNC_ID]
+ local format = {}
+ format[1] = {name='id', type='unsigned'}
+ format[2] = {name='owner', type='unsigned'}
+ format[3] = {name='name', type='string'}
+ format[4] = {name='setuid', type='unsigned'}
+ format[5] = {name='language', type='string'}
+ format[6] = {name='body', type='string'}
+ format[7] = {name='is_deterministic', type='boolean'}
+ format[8] = {name='is_sandboxed', type='boolean'}
+ format[9] = {name='opts', type='map'}
+ for _, v in box.space._func:pairs() do
+ _ = box.space._func:replace({v.id, v.owner, v.name, v.setuid,
+ v[5] or 'LUA', '', false, false,
+ setmap({})})
+ end
+ _func:format(format)
+end
+
local function upgrade_to_2_2_1()
upgrade_sequence_to_2_2_1()
upgrade_ck_constraint_to_2_2_1()
create_vcollation_space()
+ upgrade_func_to_2_2_1()
end
--------------------------------------------------------------------------------
diff --git a/src/box/schema_def.h b/src/box/schema_def.h
index 88b5502b8..ac2b3bfef 100644
--- a/src/box/schema_def.h
+++ b/src/box/schema_def.h
@@ -167,6 +167,10 @@ enum {
BOX_FUNC_FIELD_NAME = 2,
BOX_FUNC_FIELD_SETUID = 3,
BOX_FUNC_FIELD_LANGUAGE = 4,
+ BOX_FUNC_FIELD_BODY = 5,
+ BOX_FUNC_FIELD_IS_DETERMINISTIC = 6,
+ BOX_FUNC_FIELD_IS_SANDBOXED = 7,
+ BOX_FUNC_FIELD_OPTS = 8,
};
/** _collation fields. */
diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result
index b20dc41e5..5ca7f3740 100644
--- a/test/box-py/bootstrap.result
+++ b/test/box-py/bootstrap.result
@@ -53,7 +53,9 @@ box.space._space:select{}
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'is_deterministic', 'type': 'boolean'}, {'name': 'is_sandboxed',
+ 'type': 'boolean'}, {'name': 'opts', 'type': 'map'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -152,7 +154,7 @@ box.space._user:select{}
...
box.space._func:select{}
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', false, false, {}]
...
box.space._priv:select{}
---
diff --git a/test/box/access_misc.result b/test/box/access_misc.result
index 53d366106..e7a6f0984 100644
--- a/test/box/access_misc.result
+++ b/test/box/access_misc.result
@@ -793,7 +793,9 @@ box.space._space:select()
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'is_deterministic', 'type': 'boolean'}, {'name': 'is_sandboxed',
+ 'type': 'boolean'}, {'name': 'opts', 'type': 'map'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -829,7 +831,7 @@ box.space._space:select()
...
box.space._func:select()
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', false, false, {}]
...
session = nil
---
diff --git a/test/box/function1.result b/test/box/function1.result
index 331bd466a..172f56082 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -16,7 +16,10 @@ c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
---
...
-box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+function setmap(tab) return setmetatable(tab, { __serialize = 'map' }) end
+---
+...
+box.space._func:replace{2, 1, 'function1', 0, 'LUA', '', false, false, setmap({})}
---
- error: function does not support alter
...
@@ -59,10 +62,11 @@ c:call('function1.args', { 15 })
...
box.func["function1.args"]
---
-- language: C
+- is_deterministic: false
+ id: 2
setuid: false
name: function1.args
- id: 2
+ language: C
...
box.func["function1.args"]:call()
---
@@ -330,11 +334,11 @@ func = box.func.divide
...
func.call({4, 2})
---
-- error: 'builtin/box/schema.lua:2197: Use func:call(...) instead of func.call(...)'
+- error: 'builtin/box/schema.lua:2203: Use func:call(...) instead of func.call(...)'
...
func:call(4, 2)
---
-- error: 'builtin/box/schema.lua:2212: Use func:call(table)'
+- error: 'builtin/box/schema.lua:2218: Use func:call(table)'
...
func:call()
---
@@ -362,14 +366,15 @@ func:drop()
...
func
---
-- language: LUA
+- is_deterministic: false
+ id: 2
setuid: false
name: divide
- id: 2
+ language: LUA
...
func.drop()
---
-- error: 'builtin/box/schema.lua:2197: Use func:drop(...) instead of func.drop(...)'
+- error: 'builtin/box/schema.lua:2203: Use func:drop(...) instead of func.drop(...)'
...
func:drop()
---
@@ -391,7 +396,7 @@ func = box.func["function1.divide"]
...
func:call(4, 2)
---
-- error: 'builtin/box/schema.lua:2212: Use func:call(table)'
+- error: 'builtin/box/schema.lua:2218: Use func:call(table)'
...
func:call()
---
@@ -418,10 +423,11 @@ func:drop()
...
func
---
-- language: C
+- is_deterministic: false
+ id: 2
setuid: false
name: function1.divide
- id: 2
+ language: C
...
func:drop()
---
@@ -504,3 +510,120 @@ box.schema.func.drop('secret_leak')
box.schema.func.drop('secret')
---
...
+--
+-- gh-4182: Introduce persistent Lua functions.
+--
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then
+ return nil, 'Invalid field type'
+ end
+ local t = tuple.address:upper():split()
+ for k,v in pairs(t) do t[k] = v end
+ return t
+ end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('addrsplit', {body = body, language = "C"})
+---
+- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
+ not compatible with C language'
+...
+box.schema.func.create('addrsplit', {is_sandboxed = true, language = "C"})
+---
+- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
+ not compatible with C language'
+...
+box.schema.func.create('addrsplit', {is_sandboxed = true})
+---
+- error: 'Failed to create function ''addrsplit'': is_sandboxed option may be set
+ only for persistent Lua function (when body option is set)'
+...
+box.schema.func.create('invalid', {body = "function(tuple) ret tuple"})
+---
+- error: 'Failed to dynamically load function ''invalid'': function(tuple) ret tuple'
+...
+box.schema.func.create('addrsplit', {body = body, is_deterministic = true})
+---
+...
+box.schema.user.grant('guest', 'execute', 'function', 'addrsplit')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+---
+- ['MOSCOW', 'DOLGOPRUDNY']
+...
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+---
+- - MOSCOW
+ - DOLGOPRUDNY
+...
+conn:close()
+---
+...
+box.snapshot()
+---
+- ok
+...
+test_run:cmd("restart server default")
+test_run = require('test_run').new()
+---
+...
+net = require('net.box')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+---
+- ['MOSCOW', 'DOLGOPRUDNY']
+...
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+---
+- - MOSCOW
+ - DOLGOPRUDNY
+...
+conn:close()
+---
+...
+box.schema.user.revoke('guest', 'execute', 'function', 'addrsplit')
+---
+...
+box.func.addrsplit:drop()
+---
+...
+-- Test snadboxed functions.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body = [[function(number)
+ math.abs = math.log
+ return math.abs(number)
+ end]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('monkey', {body = body, is_sandboxed = true})
+---
+...
+box.func.monkey:call({1})
+---
+- 0
+...
+math.abs(1)
+---
+- 1
+...
+box.func.monkey:drop()
+---
+...
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index cf4a71979..b4021a704 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -7,7 +7,8 @@ net = require('net.box')
c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
-box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+function setmap(tab) return setmetatable(tab, { __serialize = 'map' }) end
+box.space._func:replace{2, 1, 'function1', 0, 'LUA', '', false, false, setmap({})}
box.schema.user.grant('guest', 'execute', 'function', 'function1')
_ = box.schema.space.create('test')
_ = box.space.test:create_index('primary')
@@ -178,3 +179,50 @@ conn:close()
box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
box.schema.func.drop('secret_leak')
box.schema.func.drop('secret')
+
+--
+-- gh-4182: Introduce persistent Lua functions.
+--
+test_run:cmd("setopt delimiter ';'")
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then
+ return nil, 'Invalid field type'
+ end
+ local t = tuple.address:upper():split()
+ for k,v in pairs(t) do t[k] = v end
+ return t
+ end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('addrsplit', {body = body, language = "C"})
+box.schema.func.create('addrsplit', {is_sandboxed = true, language = "C"})
+box.schema.func.create('addrsplit', {is_sandboxed = true})
+box.schema.func.create('invalid', {body = "function(tuple) ret tuple"})
+box.schema.func.create('addrsplit', {body = body, is_deterministic = true})
+box.schema.user.grant('guest', 'execute', 'function', 'addrsplit')
+conn = net.connect(box.cfg.listen)
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+conn:close()
+box.snapshot()
+test_run:cmd("restart server default")
+test_run = require('test_run').new()
+net = require('net.box')
+conn = net.connect(box.cfg.listen)
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+conn:close()
+box.schema.user.revoke('guest', 'execute', 'function', 'addrsplit')
+box.func.addrsplit:drop()
+
+-- Test snadboxed functions.
+test_run:cmd("setopt delimiter ';'")
+body = [[function(number)
+ math.abs = math.log
+ return math.abs(number)
+ end]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('monkey', {body = body, is_sandboxed = true})
+box.func.monkey:call({1})
+math.abs(1)
+box.func.monkey:drop()
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
@ 2019-06-17 9:35 ` Konstantin Osipov
2019-06-17 10:27 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-18 12:12 ` Vladimir Davydov
` (3 subsequent siblings)
4 siblings, 1 reply; 29+ messages in thread
From: Konstantin Osipov @ 2019-06-17 9:35 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/13 17:18]:
> Re-factor box_lua_call and box_lua_eval so that they don't take
> call_request. This approach is more scalable: in case of a
> functional index, the user expects to see a tuple with field
> names so we should be able to pass not only raw msgpack, but
> also a tuple to a Lua call so we need an universal way to pass
> arguments to _call methods.
I think get_msgpack() is a violation of encapsulation - you call
"get_msgpack" to later push the port arguments to C stack, is that
right? So you should call port:dump_c() instead, shouldn't you?
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-17 9:35 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-17 10:27 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-17 10:27 UTC (permalink / raw)
To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev
On 17.06.2019 12:35, Konstantin Osipov wrote:
> * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/13 17:18]:
>> Re-factor box_lua_call and box_lua_eval so that they don't take
>> call_request. This approach is more scalable: in case of a
>> functional index, the user expects to see a tuple with field
>> names so we should be able to pass not only raw msgpack, but
>> also a tuple to a Lua call so we need an universal way to pass
>> arguments to _call methods.
>
> I think get_msgpack() is a violation of encapsulation - you call
> "get_msgpack" to later push the port arguments to C stack, is that
> right? So you should call port:dump_c() instead, shouldn't you?
No, this is msgpack serializer; not to obuf, but as a memory region.
In further patches this endpoint is implemented with region allocation
for port_lua e.g.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 1/6] box: rework func cache update machinery
2019-06-13 14:08 ` [PATCH v3 1/6] box: rework func cache update machinery Kirill Shcherbatov
@ 2019-06-18 10:52 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 10:52 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:21PM +0300, Kirill Shcherbatov wrote:
> Tarantool used to assume that func_new call must not fail and
> it used to build a new func object by given definition just on
> func cache replace operation. We need to fix it to perform
> user-dependent risky actions like Lua function assemble in
> further patches.
>
> The replace method is disallowed for _func space because it is
> redundant and difficult to maintain in case of functions that
> have pre-compiled runtime.
>
> Needed for #4182, #1260
> ---
> src/box/alter.cc | 57 +++++++++++++++++++------------------
> src/box/func.c | 8 ------
> src/box/func.h | 3 --
> src/box/func_def.c | 17 +++++++++++
> src/box/func_def.h | 12 ++++++++
> src/box/schema.cc | 37 +++++++++---------------
> src/box/schema.h | 10 +++----
> test/box/function1.result | 4 +++
> test/box/function1.test.lua | 1 +
> 9 files changed, 81 insertions(+), 68 deletions(-)
Generally, this patch is fine. See a few very minor comments below.
>
> diff --git a/src/box/alter.cc b/src/box/alter.cc
> index a37a68ce4..18b50a5da 100644
> --- a/src/box/alter.cc
> +++ b/src/box/alter.cc
> @@ -2574,24 +2574,11 @@ func_def_new_from_tuple(struct tuple *tuple)
>
> /** Remove a function from function cache */
> static void
> -func_cache_remove_func(struct trigger * /* trigger */, void *event)
> +func_cache_remove_func(struct trigger *trigger, void * /* event */)
> {
> - struct txn_stmt *stmt = txn_last_stmt((struct txn *) event);
> - uint32_t fid = tuple_field_u32_xc(stmt->old_tuple ?
> - stmt->old_tuple : stmt->new_tuple,
> - BOX_FUNC_FIELD_ID);
> - func_cache_delete(fid);
> -}
> -
> -/** Replace a function in the function cache */
> -static void
> -func_cache_replace_func(struct trigger * /* trigger */, void *event)
> -{
> - struct txn_stmt *stmt = txn_last_stmt((struct txn*) event);
> - struct func_def *def = func_def_new_from_tuple(stmt->new_tuple);
> - auto def_guard = make_scoped_guard([=] { free(def); });
> - func_cache_replace(def);
> - def_guard.is_active = false;
> + struct func *old_func = (struct func *) trigger->data;
> + func_cache_delete(old_func->def->fid);
> + func_delete(old_func);
> }
>
> /**
> @@ -2612,13 +2599,21 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
> struct func *old_func = func_by_id(fid);
> if (new_tuple != NULL && old_func == NULL) { /* INSERT */
> struct func_def *def = func_def_new_from_tuple(new_tuple);
> + auto def_guard = make_scoped_guard([=] { free(def); });
> access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
> PRIV_C);
> - auto def_guard = make_scoped_guard([=] { free(def); });
> - func_cache_replace(def);
> + struct func *func = func_new(def);
> + if (func == NULL)
> + diag_raise();
> + auto func_guard = make_scoped_guard([=] {
> + func_cache_delete(def->fid);
> + func_delete(func);
> + });
Nit: I would make func_cache_insert exception-free (see my comment
below), and move on_rollback trigger allocation upper. This way we
wouldn't need the func_guard at all.
> def_guard.is_active = false;
> + func_cache_insert(func);
> + func_guard.is_active = false;
> struct trigger *on_rollback =
> - txn_alter_trigger_new(func_cache_remove_func, NULL);
> + txn_alter_trigger_new(func_cache_remove_func, func);
> txn_on_rollback(txn, on_rollback);
> } else if (new_tuple == NULL) { /* DELETE */
> uint32_t uid;
> @@ -2636,16 +2631,22 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
> "function has grants");
> }
> struct trigger *on_commit =
> - txn_alter_trigger_new(func_cache_remove_func, NULL);
> + txn_alter_trigger_new(func_cache_remove_func, old_func);
> txn_on_commit(txn, on_commit);
> } else { /* UPDATE, REPLACE */
> - struct func_def *def = func_def_new_from_tuple(new_tuple);
> - auto def_guard = make_scoped_guard([=] { free(def); });
> - access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
> - PRIV_A);
> - struct trigger *on_commit =
> - txn_alter_trigger_new(func_cache_replace_func, NULL);
> - txn_on_commit(txn, on_commit);
> + assert(new_tuple != NULL && old_tuple != NULL);
> + /**
> + * Allow an alter that doesn't change the
> + * definition to support upgrade script.
> + */
> + struct func_def *new_def = func_def_new_from_tuple(new_tuple);
> + auto new_def_guard = make_scoped_guard([=] { free(new_def); });
> + struct func_def *old_def = func_def_new_from_tuple(old_tuple);
> + auto old_def_guard = make_scoped_guard([=] { free(old_def); });
Nit: one guard should be enough, I think:
struct func_def *old_def = NULL, *new_def = NULL;
auto guard = make_scoped_guard([=] {
free(old_def);
free(new_def);
});
old_def = func_def_new_from_tuple(old_tuple);
new_def = func_def_new_from_tuple(new_tuple);
> + if (func_def_cmp(new_def, old_def) != 0) {
> + tnt_raise(ClientError, ER_UNSUPPORTED, "function",
> + "alter");
> + }
> }
> }
>
> diff --git a/src/box/func.c b/src/box/func.c
> index a817851fd..d1b8879ad 100644
> --- a/src/box/func.c
> +++ b/src/box/func.c
> @@ -451,14 +451,6 @@ func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
> return rc;
> }
>
> -void
> -func_update(struct func *func, struct func_def *def)
> -{
> - func_unload(func);
> - free(func->def);
> - func->def = def;
> -}
> -
> void
> func_delete(struct func *func)
> {
> diff --git a/src/box/func.h b/src/box/func.h
> index 8dcd61d7b..fa4d738ab 100644
> --- a/src/box/func.h
> +++ b/src/box/func.h
> @@ -100,9 +100,6 @@ module_free(void);
> struct func *
> func_new(struct func_def *def);
>
> -void
> -func_update(struct func *func, struct func_def *def);
> -
> void
> func_delete(struct func *func);
>
> diff --git a/src/box/func_def.c b/src/box/func_def.c
> index 76ed77b24..2b135e2d7 100644
> --- a/src/box/func_def.c
> +++ b/src/box/func_def.c
> @@ -1,3 +1,20 @@
> #include "func_def.h"
> +#include "string.h"
>
> const char *func_language_strs[] = {"LUA", "C"};
> +
> +int
> +func_def_cmp(struct func_def *def1, struct func_def *def2)
> +{
> + if (def1->fid != def2->fid)
> + return def1->fid - def2->fid;
> + if (def1->uid != def2->uid)
> + return def1->uid - def2->uid;
> + if (def1->setuid != def2->setuid)
> + return def1->setuid - def2->setuid;
> + if (def1->language != def2->language)
> + return def1->language - def2->language;
> + if (strcmp(def1->name, def2->name) != 0)
> + return strcmp(def1->name, def2->name);
> + return 0;
> +}
> diff --git a/src/box/func_def.h b/src/box/func_def.h
> index 5b52ab498..4c9738aea 100644
> --- a/src/box/func_def.h
> +++ b/src/box/func_def.h
> @@ -34,6 +34,10 @@
> #include "trivia/util.h"
> #include <stdbool.h>
>
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> /**
> * The supported language of the stored function.
> */
> @@ -79,6 +83,10 @@ func_def_sizeof(uint32_t name_len)
> return sizeof(struct func_def) + name_len + 1;
> }
>
> +/** Compare two given function definitions. */
> +int
> +func_def_cmp(struct func_def *def1, struct func_def *def2);
> +
> /**
> * API of C stored function.
> */
> @@ -86,4 +94,8 @@ typedef struct box_function_ctx box_function_ctx_t;
> typedef int (*box_function_f)(box_function_ctx_t *ctx,
> const char *args, const char *args_end);
>
> +#ifdef __cplusplus
> +}
> +#endif
> +
> #endif /* TARANTOOL_BOX_FUNC_DEF_H_INCLUDED */
> diff --git a/src/box/schema.cc b/src/box/schema.cc
> index 6d3153815..f834610a5 100644
> --- a/src/box/schema.cc
> +++ b/src/box/schema.cc
> @@ -541,40 +541,32 @@ schema_free(void)
> }
>
> void
> -func_cache_replace(struct func_def *def)
> +func_cache_insert(struct func *func)
> {
> - struct func *old = func_by_id(def->fid);
> - if (old) {
> - func_update(old, def);
> - return;
> - }
> if (mh_size(funcs) >= BOX_FUNCTION_MAX)
> tnt_raise(ClientError, ER_FUNCTION_MAX, BOX_FUNCTION_MAX);
Nit: let's also move this check to func_def_new_from_tuple.
This would consistent with space_def_new_from_tuple and it
would make func_cache_insert protocol more straighforward -
panic on OOM, otherwise exception-free.
> - struct func *func = func_new(def);
> - if (func == NULL) {
> -error:
> - panic_syserror("Out of memory for the data "
> - "dictionary cache (stored function).");
> - }
> - const struct mh_i32ptr_node_t node = { def->fid, func };
> - mh_int_t k1 = mh_i32ptr_put(funcs, &node, NULL, NULL);
> - if (k1 == mh_end(funcs)) {
> - func->def = NULL;
> - func_delete(func);
> - goto error;
> - }
> +
> + mh_int_t k1, k2;
> + struct mh_i32ptr_node_t old_node, *old_node_ptr = &old_node;
> + const struct mh_i32ptr_node_t node = { func->def->fid, func };
Nit: I wouldn't change this code. Instead I would add to the beginning
of this function:
assert(func_by_id(func->def->id) == NULL);
assert(func_by_name(func->def->name, func->def->name_len) == NULL);
> size_t def_name_len = strlen(func->def->name);
> uint32_t name_hash = mh_strn_hash(func->def->name, def_name_len);
> const struct mh_strnptr_node_t strnode = {
> func->def->name, def_name_len, name_hash, func };
>
> - mh_int_t k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
> + k1 = mh_i32ptr_put(funcs, &node, &old_node_ptr, NULL);
> + if (k1 == mh_end(funcs))
> + goto error;
> + assert(old_node_ptr == NULL);
> + k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
> if (k2 == mh_end(funcs_by_name)) {
> mh_i32ptr_del(funcs, k1, NULL);
> - func->def = NULL;
> - func_delete(func);
> goto error;
> }
> + return;
> +error:
> + panic_syserror("Out of memory for the data dictionary cache "
> + "(stored function).");
> }
>
> void
> @@ -590,7 +582,6 @@ func_cache_delete(uint32_t fid)
> strlen(func->def->name));
> if (k != mh_end(funcs))
> mh_strnptr_del(funcs_by_name, k, NULL);
> - func_delete(func);
> }
>
> struct func *
> diff --git a/src/box/schema.h b/src/box/schema.h
> index 6f9a96117..d7b3c4c2d 100644
> --- a/src/box/schema.h
> +++ b/src/box/schema.h
> @@ -163,14 +163,12 @@ schema_free();
> struct space *schema_space(uint32_t id);
>
> /**
> - * Insert a new function or update the old one.
> - *
> - * @param def Function definition. In a case of success the ownership
> - * of @a def is transfered to the data dictionary, thus the caller
> - * must not delete it.
> + * Insert a new function object in the function cache.
> + * @param func Function object to insert.
> + * @retval Returns 0 on success, -1 otherwise.
Nit: there's no return value :-)
It would be nice to mention that there must not be a function with
the same name or id present in the cache.
> */
> void
> -func_cache_replace(struct func_def *def);
> +func_cache_insert(struct func *func);
>
> void
> func_cache_delete(uint32_t fid);
> diff --git a/test/box/function1.result b/test/box/function1.result
> index 5b04fa97d..cadeb0467 100644
> --- a/test/box/function1.result
> +++ b/test/box/function1.result
> @@ -16,6 +16,10 @@ c = net.connect(os.getenv("LISTEN"))
> box.schema.func.create('function1', {language = "C"})
> ---
> ...
> +box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
> +---
> +- error: function does not support alter
> +...
> box.schema.user.grant('guest', 'execute', 'function', 'function1')
> ---
> ...
> diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
> index 855dd5382..e983495b6 100644
> --- a/test/box/function1.test.lua
> +++ b/test/box/function1.test.lua
> @@ -7,6 +7,7 @@ net = require('net.box')
> c = net.connect(os.getenv("LISTEN"))
>
> box.schema.func.create('function1', {language = "C"})
> +box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
> box.schema.user.grant('guest', 'execute', 'function', 'function1')
> _ = box.schema.space.create('test')
> _ = box.space.test:create_index('primary')
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
2019-06-17 9:35 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-18 12:12 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-18 13:58 ` Vladimir Davydov
` (2 subsequent siblings)
4 siblings, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 12:12 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:22PM +0300, Kirill Shcherbatov wrote:
> Re-factor box_lua_call and box_lua_eval so that they don't take
> call_request. This approach is more scalable: in case of a
> functional index, the user expects to see a tuple with field
> names so we should be able to pass not only raw msgpack, but
> also a tuple to a Lua call so we need an universal way to pass
> arguments to _call methods.
>
> To pass a tuple msgpack introduced a new port_msgpack: the port
> class with dump_lua method.
>
> Needed for #4182, #1260
> ---
> src/box/call.c | 46 ++++++----------
> src/box/call.h | 4 --
> src/box/execute.c | 1 +
> src/box/func.c | 27 ++++++++--
> src/box/func.h | 11 ++--
> src/box/lua/call.c | 121 +++++++++++++++++++++++++++++++-----------
> src/box/lua/call.h | 8 ++-
> src/box/lua/execute.c | 7 +--
> src/box/lua/execute.h | 4 +-
> src/box/lua/misc.cc | 5 +-
> src/box/port.c | 3 +-
> src/box/port.h | 15 ++++++
> src/lib/core/port.h | 15 ++++--
> 13 files changed, 184 insertions(+), 83 deletions(-)
The patch looks mostly fine to me. See a few comments below.
>
> diff --git a/src/box/call.c b/src/box/call.c
> index 56da53fb3..1ab3993da 100644
> --- a/src/box/call.c
> +++ b/src/box/call.c
> @@ -105,33 +105,6 @@ access_check_func(const char *name, uint32_t name_len, struct func **funcp)
> return 0;
> }
>
> -static int
> -box_c_call(struct func *func, struct call_request *request, struct port *port)
> -{
> - assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
> -
> - /* Create a call context */
> - port_tuple_create(port);
> - box_function_ctx_t ctx = { port };
> -
> - /* Clear all previous errors */
> - diag_clear(&fiber()->diag);
> - assert(!in_txn()); /* transaction is not started */
> -
> - /* Call function from the shared library */
> - int rc = func_call(func, &ctx, request->args, request->args_end);
> - func = NULL; /* May be deleted by DDL */
> - if (rc != 0) {
> - if (diag_last_error(&fiber()->diag) == NULL) {
> - /* Stored procedure forget to set diag */
> - diag_set(ClientError, ER_PROC_C, "unknown error");
> - }
> - port_destroy(port);
> - return -1;
> - }
> - return 0;
> -}
> -
> int
> box_module_reload(const char *name)
> {
> @@ -199,11 +172,15 @@ box_process_call(struct call_request *request, struct port *port)
> }
>
> int rc;
> + struct port in_port;
Nit: I don't quite like the names in_port/out_port - what's the point in
including the type name into a variable name? I'd call them simply
'args' and 'ret' (or 'retval' or 'out').
> + port_msgpack_create(&in_port, request->args,
> + request->args_end - request->args);
> if (func && func->def->language == FUNC_LANGUAGE_C) {
> - rc = box_c_call(func, request, port);
> + rc = func_call(func, &in_port, port);
IMO box_c_call should be removed by the next patch. This patch should
simply replace msgpack args and call_request with port. I don't want to
make you re-split though. Up to you.
> } else {
> - rc = box_lua_call(request, port);
> + rc = box_lua_call(name, name_len, &in_port, port);
> }
> + port_destroy(&in_port);
> /* Restore the original user */
> if (orig_credentials)
> fiber_set_user(fiber(), orig_credentials);
> @@ -229,16 +206,23 @@ box_process_eval(struct call_request *request, struct port *port)
> /* Check permissions */
> if (access_check_universe(PRIV_X) != 0)
> return -1;
> - if (box_lua_eval(request, port) != 0) {
> + struct port in_port;
> + port_msgpack_create(&in_port, request->args,
> + request->args_end - request->args);
> + const char *expr = request->expr;
> + uint32_t expr_len = mp_decode_strl(&expr);
> + if (box_lua_eval(expr, expr_len, &in_port, port) != 0) {
> + port_destroy(&in_port);
We know that port_destroy is a no-op for port_msgpack. I think it's okay
to omit the call here.
> txn_rollback();
> return -1;
> }
>
> if (in_txn()) {
> diag_set(ClientError, ER_FUNCTION_TX_ACTIVE);
> + port_destroy(&in_port);
> txn_rollback();
> return -1;
> }
> -
> + port_destroy(&in_port);
> return 0;
> }
> diff --git a/src/box/call.h b/src/box/call.h
> index 1b54551be..45580bc9d 100644
> --- a/src/box/call.h
> +++ b/src/box/call.h
> @@ -38,10 +38,6 @@ extern "C" {
> struct port;
> struct call_request;
>
> -struct box_function_ctx {
> - struct port *port;
> -};
> -
> /**
> * Reload loadable module by name.
> *
> diff --git a/src/box/execute.c b/src/box/execute.c
> index e81cc32aa..a91c939fc 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -108,6 +108,7 @@ const struct port_vtab port_sql_vtab = {
> /* .dump_msgpack_16 = */ NULL,
> /* .dump_lua = */ port_sql_dump_lua,
> /* .dump_plain = */ NULL,
> + /* .get_msgpack = */ NULL,
> /* .destroy = */ port_sql_destroy,
> };
>
> diff --git a/src/box/func.c b/src/box/func.c
> index d1b8879ad..740fad3d7 100644
> --- a/src/box/func.c
> +++ b/src/box/func.c
> @@ -34,6 +34,9 @@
> #include "lua/utils.h"
> #include "error.h"
> #include "diag.h"
> +#include "fiber.h"
> +#include "port.h"
> +#include "txn.h"
> #include <dlfcn.h>
>
> /**
> @@ -433,21 +436,39 @@ func_load(struct func *func)
> }
>
> int
> -func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
> - const char *args_end)
> +func_call(struct func *func, struct port *in_port, struct port *out_port)
> {
> + assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
> + /* Transaction is not started. */
> + assert(!in_txn());
I don't think it's a good idea to retain this requirement for any C
function call. First, it introduces a dependency on txn. Second, we
might want to call a C function from a transaction one day. Let's leave
this check in box_process_call.
> if (func->func == NULL) {
> if (func_load(func) != 0)
> return -1;
> }
>
> + uint32_t args_sz;
> + const char *args = port_get_msgpack(in_port, &args_sz);
> + if (args == NULL)
> + return -1;
> +
> + port_tuple_create(out_port);
> + box_function_ctx_t ctx = { out_port };
> +
> /* Module can be changed after function reload. */
> struct module *module = func->module;
> assert(module != NULL);
> ++module->calls;
> - int rc = func->func(ctx, args, args_end);
> + int rc = func->func(&ctx, args, args + args_sz);
> --module->calls;
> module_gc(module);
> + if (rc != 0) {
> + if (diag_last_error(&fiber()->diag) == NULL) {
> + /* Stored procedure forget to set diag */
> + diag_set(ClientError, ER_PROC_C, "unknown error");
> + }
> + port_destroy(out_port);
> + return -1;
> + }
> return rc;
> }
>
> diff --git a/src/box/func.h b/src/box/func.h
> index fa4d738ab..d38a2b283 100644
> --- a/src/box/func.h
> +++ b/src/box/func.h
> @@ -42,6 +42,12 @@
> extern "C" {
> #endif /* defined(__cplusplus) */
>
> +struct port;
> +
> +struct box_function_ctx {
> + struct port *port;
> +};
> +
I'd rather move it to src/box/func_def.h, because we have
box_function_ctx_t and box_function_f defined there.
> /**
> * Dynamic shared module.
> */
> @@ -104,11 +110,10 @@ void
> func_delete(struct func *func);
>
> /**
> - * Call stored C function using @a args.
> + * Call function with arguments represented with given in_port.
> */
> int
> -func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
> - const char *args_end);
> +func_call(struct func *func, struct port *in_port, struct port *out_port);
>
> /**
> * Reload dynamically loadable module.
> diff --git a/src/box/lua/call.c b/src/box/lua/call.c
> index 04020ef6f..e32845095 100644
> --- a/src/box/lua/call.c
> +++ b/src/box/lua/call.c
I guess we don't need to include box/xrow.h into this file anymore.
Please remove.
> @@ -266,6 +266,55 @@ luamp_encode_call_16(lua_State *L, struct luaL_serializer *cfg,
> return root.size;
> }
>
> +static const struct port_vtab port_msgpack_vtab;
> +
> +void
> +port_msgpack_create(struct port *base, const char *data, uint32_t data_sz)
> +{
> + struct port_msgpack *port_msgpack = (struct port_msgpack *) base;
> + memset(port_msgpack, 0, sizeof(*port_msgpack));
> + port_msgpack->vtab = &port_msgpack_vtab;
> + port_msgpack->data = data;
> + port_msgpack->data_sz = data_sz;
> +}
> +
> +static void
> +port_msgpack_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
> +{
> + assert(is_flat == true);
> + struct port_msgpack *port = (struct port_msgpack *) base;
> + assert(port->vtab == &port_msgpack_vtab);
> +
> + const char *args = port->data;
> + uint32_t arg_count = mp_decode_array(&args);
> + for (uint32_t i = 0; i < arg_count; i++)
> + luamp_decode(L, luaL_msgpack_default, &args);
> +}
> +
> +static const char *
> +port_msgpack_get_msgpack(struct port *base, uint32_t *size)
> +{
> + struct port_msgpack *port = (struct port_msgpack *) base;
> + assert(port->vtab == &port_msgpack_vtab);
> + *size = port->data_sz;
> + return port->data;
> +}
> +
> +static void
> +port_msgpack_destroy(struct port *base)
> +{
> + (void) base;
> +}
> +
> +static const struct port_vtab port_msgpack_vtab = {
> + .dump_msgpack = NULL,
> + .dump_msgpack_16 = NULL,
> + .dump_lua = port_msgpack_dump_lua,
> + .dump_plain = NULL,
> + .get_msgpack = port_msgpack_get_msgpack,
> + .destroy = port_msgpack_destroy,
> +};
> +
port_msgpack is used solely from src/box/call.c so I think we should
define vtab there. The 'destroy' method could be set to NULL - as I
wrote above we should probably omit calling it altogether.
We can't define port_msgpack_dump_lua in src/box/call.c, because it
depends on Lua API. Okay, let's define it in src/box/lua/misc.cc then,
just like we did in case of port_tuple_dump_lua.
> static const struct port_vtab port_lua_vtab;
>
> void
> @@ -282,28 +331,31 @@ port_lua_create(struct port *port, struct lua_State *L)
> port_lua->ref = -1;
> }
>
> +struct lua_process_ctx {
I'd rather call it execute_lua_ctx.
> + const char *name;
> + uint32_t name_len;
> +};
> +
> static int
> execute_lua_call(lua_State *L)
> {
> - struct call_request *request = (struct call_request *)
> - lua_topointer(L, 1);
> + struct port *in_port = (struct port *) lua_topointer(L, 1);
> + struct lua_process_ctx *ctx =
> + (struct lua_process_ctx *) lua_topointer(L, 2);
Why not include the port containing call/eval arguments into the context
as well?
> @@ -429,37 +480,47 @@ static const struct port_vtab port_lua_vtab = {
> .dump_msgpack_16 = port_lua_dump_16,
> .dump_lua = NULL,
> .dump_plain = port_lua_dump_plain,
> + .get_msgpack = NULL,
> .destroy = port_lua_destroy,
> };
>
> static inline int
> -box_process_lua(struct call_request *request, struct port *base,
> - lua_CFunction handler)
> +box_process_lua(lua_CFunction handler, void *arg, struct port *in_port,
s/void *arg/struct execute_lua_ctx *ctx
?
> diff --git a/src/box/lua/call.h b/src/box/lua/call.h
> index 0542123da..1801d5090 100644
> --- a/src/box/lua/call.h
> +++ b/src/box/lua/call.h
> @@ -35,6 +35,8 @@
> extern "C" {
> #endif /* defined(__cplusplus) */
>
> +#include <stdint.h>
> +
Nit: header inclusions should go before extern "C".
> struct lua_State;
>
> void
> diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
> index 239aba47b..187bdb73d 100644
> --- a/src/box/lua/execute.c
> +++ b/src/box/lua/execute.c
> @@ -39,8 +39,9 @@ lua_sql_get_metadata(struct sql_stmt *stmt, struct lua_State *L,
> }
>
> void
> -port_sql_dump_lua(struct port *port, struct lua_State *L)
> +port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
> {
> + assert(is_flat == false);
> assert(port->vtab == &port_sql_vtab);
> struct sql *db = sql_get();
> struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
> @@ -49,7 +50,7 @@ port_sql_dump_lua(struct port *port, struct lua_State *L)
> lua_createtable(L, 0, 2);
> lua_sql_get_metadata(stmt, L, column_count);
> lua_setfield(L, -2, "metadata");
> - port_tuple_vtab.dump_lua(port, L);
> + port_tuple_vtab.dump_lua(port, L, is_flat);
Nit: better pass 'false' explicitly IMO.
> diff --git a/src/box/port.h b/src/box/port.h
> index f18803660..4c0c3cf7a 100644
> --- a/src/box/port.h
> +++ b/src/box/port.h
> @@ -32,6 +32,7 @@
> */
> #include "trivia/util.h"
> #include <port.h>
> +#include <stdbool.h>
>
> #if defined(__cplusplus)
> extern "C" {
> @@ -80,6 +81,20 @@ port_tuple_create(struct port *port);
> int
> port_tuple_add(struct port *port, struct tuple *tuple);
>
> +/** Port implementation used for storing raw data. */
> +struct port_msgpack {
> + const struct port_vtab *vtab;
> + const char *data;
> + uint32_t data_sz;
> +};
> +
> +static_assert(sizeof(struct port_msgpack) <= sizeof(struct port),
> + "sizeof(struct port_msgpack) must be <= sizeof(struct port)");
> +
> +/** Initialize a port to dump row data. */
s/row/raw
> +void
> +port_msgpack_create(struct port *port, const char *data, uint32_t data_sz);
> +
> /** Port for storing the result of a Lua CALL/EVAL. */
> struct port_lua {
> const struct port_vtab *vtab;
> diff --git a/src/lib/core/port.h b/src/lib/core/port.h
> index 8ace40fc5..2f1bc5ed1 100644
> --- a/src/lib/core/port.h
> +++ b/src/lib/core/port.h
> @@ -30,6 +30,7 @@
> * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> * SUCH DAMAGE.
> */
> +#include <stdbool.h>
> #include <stdint.h>
>
> #if defined(__cplusplus)
> @@ -63,7 +64,7 @@ struct port_vtab {
> */
> int (*dump_msgpack_16)(struct port *port, struct obuf *out);
> /** Dump the content of a port to Lua stack. */
> - void (*dump_lua)(struct port *port, struct lua_State *L);
> + void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
Please document the new is_flat option.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 3/6] box: rework func object as a function frontend
2019-06-13 14:08 ` [PATCH v3 3/6] box: rework func object as a function frontend Kirill Shcherbatov
@ 2019-06-18 13:23 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 13:23 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:23PM +0300, Kirill Shcherbatov wrote:
> The function func object used to provide a call method only for
> C functions. In scope of this patch it reworked to be a uniform
> function call frontend both for C and Lua functions.
>
> Introduced classes func_c and func_lua, that provide own
> constructors which produce implementation-specific object with
> call and destroy methods.
>
> Needed for #4182, #1260
> ---
> src/box/alter.cc | 1 +
> src/box/call.c | 105 ++---------------------------
> src/box/func.c | 165 ++++++++++++++++++++++++++++++++++++++++-----
> src/box/func.h | 25 ++++---
> src/box/func_def.h | 2 +
> src/box/lua/call.c | 46 +++++++++++++
> src/box/lua/call.h | 5 ++
> src/box/session.cc | 11 ++-
> src/box/session.h | 8 +++
> 9 files changed, 235 insertions(+), 133 deletions(-)
Generally, the patch is fine. See some minor comments inline.
> diff --git a/src/box/func.c b/src/box/func.c
> index 740fad3d7..20ff56668 100644
> --- a/src/box/func.c
> +++ b/src/box/func.c
> @@ -381,19 +409,39 @@ func_new(struct func_def *def)
> * checks (see user_has_data()).
> */
> func->owner_credentials.auth_token = BOX_USER_MAX; /* invalid value */
> + return func;
> +}
> +
> +static struct func_vtab func_c_vtab;
> +
> +struct func *
> +func_c_new(struct func_def *def)
Nit: should be static.
> +{
> + (void) def;
> + assert(def->language == FUNC_LANGUAGE_C);
> + struct func_c *func = (struct func_c *) malloc(sizeof(struct func_c));
> + if (func == NULL) {
> + diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
> + return NULL;
> + }
> + func->base.vtab = &func_c_vtab;
> func->func = NULL;
> func->module = NULL;
> - return func;
> + return &func->base;
> }
>
> static void
> -func_unload(struct func *func)
> +func_c_unload(struct func *base)
> {
> + assert(base->vtab == &func_c_vtab);
> + assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
> + struct func_c *func = (struct func_c *) base;
> +
> if (func->module) {
> rlist_del(&func->item);
> if (rlist_empty(&func->module->funcs)) {
> struct func_name name;
> - func_split_name(func->def->name, &name);
> + func_split_name(func->base.def->name, &name);
> module_cache_del(name.package, name.package_end);
> }
> module_gc(func->module);
> @@ -407,12 +455,15 @@ func_unload(struct func *func)
> * symbol from it).
> */
> static int
> -func_load(struct func *func)
> +func_c_load(struct func *base)
Why does it take struct func? Why not struct func_c?
> {
> + assert(base->vtab == &func_c_vtab);
> + assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
> + struct func_c *func = (struct func_c *) base;
> assert(func->func == NULL);
>
> struct func_name name;
> - func_split_name(func->def->name, &name);
> + func_split_name(func->base.def->name, &name);
>
> struct module *module = module_cache_find(name.package,
> name.package_end);
> @@ -472,10 +526,87 @@ func_call(struct func *func, struct port *in_port, struct port *out_port)
> return rc;
> }
>
> +static struct func_vtab func_c_vtab = {
> + .call = func_c_call,
> + .destroy = func_c_unload,
This looks confusing. Let's call the destructor 'func_c_destroy'.
Also, I think that free(func) should be called not by the generic
func_destroy(), but by the virtual destructor, because it's good
to have malloc and free in the same compilation unit.
I like func_c_load/func_c_unload symmetry. Let's make those functions
take struct func_c instead of struct func. Then func_c_destroy would
call func_c_unload and free struct func.
> +};
> +
> void
> func_delete(struct func *func)
> {
> - func_unload(func);
> + func->vtab->destroy(func);
> free(func->def);
> free(func);
> }
> +
> +/** Check "EXECUTE" permissions for a given function. */
> +static int
> +func_access_check(struct func *base)
Nit: since you don't cast it to a subclass, better name it simply
'func', not 'base' IMO.
> +{
> + struct credentials *credentials = effective_user();
> + /*
> + * If the user has universal access, don't bother with
> + * checks. No special check for ADMIN user is necessary
> + * since ADMIN has universal access.
> + */
> + if ((credentials->universal_access & (PRIV_X | PRIV_U)) ==
> + (PRIV_X | PRIV_U))
> + return 0;
> + user_access_t access = PRIV_X | PRIV_U;
> + /* Check access for all functions. */
> + access &= ~entity_access_get(SC_FUNCTION)[
> + credentials->auth_token].effective;
> + user_access_t func_access = access & ~credentials->universal_access;
> + if (/* Check for missing Usage access, ignore owner rights. */
> + func_access & PRIV_U ||
> + /* Check for missing specific access, respect owner rights. */
> + (base->def->uid != credentials->uid &&
> + func_access & ~base->access[credentials->auth_token].effective)) {
Nit: bad indentation: func_access... should be aligned by
the parenthesis.
> diff --git a/src/box/func.h b/src/box/func.h
> index d38a2b283..1271bde67 100644
> --- a/src/box/func.h
> +++ b/src/box/func.h
> @@ -62,24 +63,22 @@ struct module {
> bool is_unloading;
> };
>
> +/** Virtual method table for func object. */
> +struct func_vtab {
> + /** Call function with given arguments. */
> + int (*call)(struct func *func, struct port *in_port,
> + struct port *out_port);
> + /** Release implementation-specific function context. */
> + void (*destroy)(struct func *func);
> +};
> +
> /**
> * Stored function.
> */
> struct func {
> struct func_def *def;
> - /**
> - * Anchor for module membership.
> - */
> - struct rlist item;
> - /**
> - * For C functions, the body of the function.
> - */
> - box_function_f func;
> - /**
> - * Each stored function keeps a handle to the
> - * dynamic library for the C callback.
> - */
> - struct module *module;
> + /** Virtual method table. */
> + struct func_vtab *vtab;
Nit: should be const.
> diff --git a/src/box/lua/call.c b/src/box/lua/call.c
> index e32845095..8d0328ef7 100644
> --- a/src/box/lua/call.c
> +++ b/src/box/lua/call.c
> @@ -523,6 +525,50 @@ box_lua_eval(const char *expr, uint32_t expr_len,
> return box_process_lua(execute_lua_eval, &ctx, in_port, out_port);
> }
>
> +struct func_lua {
> + /** Function object base class. */
> + struct func base;
> +};
> +
> +static struct func_vtab func_lua_vtab;
> +
> +struct func *
> +func_lua_new(struct func_def *def)
> +{
> + (void) def;
> + assert(def->language == FUNC_LANGUAGE_LUA);
> + struct func_lua *func =
> + (struct func_lua *) malloc(sizeof(struct func_lua));
> + if (func == NULL) {
> + diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
> + return NULL;
> + }
> + func->base.vtab = &func_lua_vtab;
> + return &func->base;
> +}
> +
> +static void
> +func_lua_unload(struct func *base)
Again, I think this should be called func_lua_destroy and this function
should free func_lua struct.
> +{
> + assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA);
> + assert(base->vtab == &func_lua_vtab);
> + (void) base;
> +}
> +
> +static inline int
> +func_lua_call(struct func *func, struct port *in_port, struct port *out_port)
> +{
> + assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA);
> + assert(func->vtab == &func_lua_vtab);
> + return box_lua_call(func->def->name, func->def->name_len,
> + in_port, out_port);
> +}
> +
> +static struct func_vtab func_lua_vtab = {
> + .call = func_lua_call,
> + .destroy = func_lua_unload,
> +};
> +
> static int
> lbox_module_reload(lua_State *L)
> {
> diff --git a/src/box/session.h b/src/box/session.h
> index 3a7397146..6bafb8561 100644
> --- a/src/box/session.h
> +++ b/src/box/session.h
> @@ -312,6 +312,14 @@ access_check_session(struct user *user);
> int
> access_check_universe(user_access_t access);
>
> +/**
> + * Check whether or not the current user can be granted
> + * the requested access to the universe object.
The comment is kinda confusing. Let's just say something like
Same as access_check_universe(), but in case the current user doesn't
have universal access, set AccessDeniedError for the given object type
and name.
> + */
> +int
> +access_check_universe_object(user_access_t access, enum schema_object_type type,
Nit: I'd s/type/object_type for consistency with object_name.
> + const char *object_name);
> +
> static inline int
> session_push(struct session *session, uint64_t sync, struct port *port)
> {
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
2019-06-17 9:35 ` [tarantool-patches] " Konstantin Osipov
2019-06-18 12:12 ` Vladimir Davydov
@ 2019-06-18 13:58 ` Vladimir Davydov
2019-06-19 18:28 ` [tarantool-patches] " Konstantin Osipov
2019-06-19 18:30 ` [tarantool-patches] " Konstantin Osipov
4 siblings, 0 replies; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 13:58 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:22PM +0300, Kirill Shcherbatov wrote:
> diff --git a/src/box/func.c b/src/box/func.c
> index d1b8879ad..740fad3d7 100644
> --- a/src/box/func.c
> +++ b/src/box/func.c
> @@ -433,21 +436,39 @@ func_load(struct func *func)
> }
>
> int
> -func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
> - const char *args_end)
> +func_call(struct func *func, struct port *in_port, struct port *out_port)
> {
> + assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
> + /* Transaction is not started. */
> + assert(!in_txn());
> if (func->func == NULL) {
> if (func_load(func) != 0)
> return -1;
> }
>
> + uint32_t args_sz;
> + const char *args = port_get_msgpack(in_port, &args_sz);
> + if (args == NULL)
> + return -1;
> +
> + port_tuple_create(out_port);
> + box_function_ctx_t ctx = { out_port };
> +
> /* Module can be changed after function reload. */
> struct module *module = func->module;
> assert(module != NULL);
> ++module->calls;
> - int rc = func->func(ctx, args, args_end);
> + int rc = func->func(&ctx, args, args + args_sz);
> --module->calls;
> module_gc(module);
> + if (rc != 0) {
> + if (diag_last_error(&fiber()->diag) == NULL) {
> + /* Stored procedure forget to set diag */
> + diag_set(ClientError, ER_PROC_C, "unknown error");
> + }
> + port_destroy(out_port);
> + return -1;
> + }
Shouldn't you call region_truncate() here to clean up after the function
and possibly port_get_msgpack()?
> return rc;
> }
>
> diff --git a/src/lib/core/port.h b/src/lib/core/port.h
> index 8ace40fc5..2f1bc5ed1 100644
> --- a/src/lib/core/port.h
> +++ b/src/lib/core/port.h
> @@ -74,6 +75,8 @@ struct port_vtab {
> * @retval not nil Plain text.
> */
> const char *(*dump_plain)(struct port *port, uint32_t *size);
> + /** Get the content of a port as a msgpack data. */
> + const char *(*get_msgpack)(struct port *port, uint32_t *size);
Please mention what's the lifetime of the returned data. I guess you
should say that it's implementation dependent: it may either be returned
directly from the port, in which case the data will stay alive as long
as the port is alive, or it may be allocated on the fiber()->gc, in
which case the caller is responsible for cleaning up.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 4/6] box: export registered functions in box.func folder
2019-06-13 14:08 ` [PATCH v3 4/6] box: export registered functions in box.func folder Kirill Shcherbatov
@ 2019-06-18 14:06 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-18 16:11 ` Vladimir Davydov
1 sibling, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 14:06 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:24PM +0300, Kirill Shcherbatov wrote:
> diff --git a/src/box/func.h b/src/box/func.h
> index 1271bde67..f7081eccd 100644
> --- a/src/box/func.h
> +++ b/src/box/func.h
> @@ -419,23 +418,20 @@ encode_lua_call_16(lua_State *L)
> *
> * TODO: forbid explicit yield from __serialize or __index here
> */
> - struct mpstream stream;
> - mpstream_init(&stream, port->out, obuf_reserve_cb, obuf_alloc_cb,
> - luamp_error, port->L);
> -
> struct luaL_serializer *cfg = luaL_msgpack_default;
> - port->size = luamp_encode_call_16(port->L, cfg, &stream);
> - mpstream_flush(&stream);
> + port->size = luamp_encode_call_16(port->L, cfg, port->stream);
> + mpstream_flush(port->stream);
> return 0;
> }
>
> static inline int
> -port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler)
> +port_lua_do_dump(struct port *base, struct mpstream *stream,
> + lua_CFunction handler)
> {
> - struct port_lua *port = (struct port_lua *)base;
> + struct port_lua *port = (struct port_lua *) base;
> assert(port->vtab == &port_lua_vtab);
> /* Use port to pass arguments to encoder quickly. */
> - port->out = out;
> + port->stream = stream;
Why does mpstream (or obuf) have to be a member of port_lua? Just to
pass it to encode_lua_call? If so, let's please introduce an auxiliary
struct that would store port and mpstream while we are at it.
Other than that, the patch is good.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 5/6] box: refactor box_lua_find helper
2019-06-13 14:08 ` [PATCH v3 5/6] box: refactor box_lua_find helper Kirill Shcherbatov
@ 2019-06-18 14:22 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 14:22 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:25PM +0300, Kirill Shcherbatov wrote:
> The box_lua_find routine used to work with an empty stack only.
> It is unacceptable in following patches because this helper
> need to be reused in following patches for environment table
> construction.
>
> Refactored function is renamed to luaT_func_find.
>
> Needed for #4182, #1260
> ---
> src/box/lua/call.c | 54 +++++++++++++++++++++++++---------------------
> 1 file changed, 30 insertions(+), 24 deletions(-)
>
> diff --git a/src/box/lua/call.c b/src/box/lua/call.c
> index 23eb29eb8..90c28a160 100644
> --- a/src/box/lua/call.c
> +++ b/src/box/lua/call.c
> @@ -54,21 +54,19 @@
> * on top of the stack.
> */
> static int
> -box_lua_find(lua_State *L, const char *name, const char *name_end)
> +luaT_func_find(lua_State *L, const char *name, const char *name_end,
> + int *count)
Don't see any point in renaming it or using luaT_ prefix for this and
other helper functions in this compilation unit - other functions in the
same file have box_ prefix, e.g. box_process_lua. I may be wrong, but it
looks like luaT_ prefix is primarily used for generic helper functions
from util.c.
> {
> int index = LUA_GLOBALSINDEX;
> - int objstack = 0;
> + int objstack = 0, top = lua_gettop(L);
> const char *start = name, *end;
>
> while ((end = (const char *) memchr(start, '.', name_end - start))) {
> lua_checkstack(L, 3);
> lua_pushlstring(L, start, end - start);
> lua_gettable(L, index);
> - if (! lua_istable(L, -1)) {
> - diag_set(ClientError, ER_NO_SUCH_PROC,
> - name_end - name, name);
Why drop diag_set? Looking at the following patch, I see that you want
to set another error when you create a sandbox. Well, it's okay to
overwrite a diag. Moreover, we have plans to implement stacked diag so
that instead of overwriting the error, subsequent diag_set would append
a new error to the diag and we could print something like a stack trace
leading to the error.
> - luaT_error(L);
> - }
> + if (! lua_istable(L, -1))
> + return -1;
> start = end + 1; /* next piece of a.b.c */
> index = lua_gettop(L); /* top of the stack */
> }
> @@ -95,25 +90,27 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
> if (!lua_isfunction(L, -1) && !lua_istable(L, -1)) {
> /* lua_call or lua_gettable would raise a type error
> * for us, but our own message is more verbose. */
> - diag_set(ClientError, ER_NO_SUCH_PROC,
> - name_end - name, name);
> - luaT_error(L);
> + return -1;
> }
> +
> /* setting stack that it would contain only
> * the function pointer. */
> if (index != LUA_GLOBALSINDEX) {
> if (objstack == 0) { /* no object, only a function */
> - lua_replace(L, 1);
> + lua_replace(L, top + 1);
> + lua_pop(L, lua_gettop(L) - top - 1);
> } else if (objstack == 1) { /* just two values, swap them */
> lua_insert(L, -2);
> + lua_pop(L, lua_gettop(L) - top - 2);
> } else { /* long path */
> - lua_insert(L, 1);
> - lua_insert(L, 2);
> + lua_insert(L, top + 1);
> + lua_insert(L, top + 2);
> + lua_pop(L, objstack - 1);
> objstack = 1;
> }
> - lua_settop(L, 1 + objstack);
> }
> - return 1 + objstack;
> + *count = 1 + objstack;
> + return 0;
Why change the return value? We can return -1 on error, >= 0 on success.
IMHO better than returning the value by pointer.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 4/6] box: export registered functions in box.func folder
2019-06-13 14:08 ` [PATCH v3 4/6] box: export registered functions in box.func folder Kirill Shcherbatov
2019-06-18 14:06 ` Vladimir Davydov
@ 2019-06-18 16:11 ` Vladimir Davydov
1 sibling, 0 replies; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 16:11 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:24PM +0300, Kirill Shcherbatov wrote:
> diff --git a/test/box/function1.result b/test/box/function1.result
> index cadeb0467..331bd466a 100644
> --- a/test/box/function1.result
> +++ b/test/box/function1.result
> @@ -57,6 +57,25 @@ c:call('function1.args', { 15 })
> ---
> - [[15, 'hello']]
> ...
> +box.func["function1.args"]
> +---
> +- language: C
> + setuid: false
> + name: function1.args
> + id: 2
> +...
> +box.func["function1.args"]:call()
> +---
> +- error: Use func:call(...)
> +...
> +box.func["function1.args"]:call({ "xx" })
> +---
> +- error: first tuple field must be uint
> +...
> +box.func["function1.args"]:call({ 15 })
> +---
> +- [15, 'hello']
> +...
> box.schema.func.drop("function1.args")
> ---
> ...
> @@ -299,3 +318,189 @@ test_run:cmd("setopt delimiter ''");
> c:close()
> ---
> ...
> +-- Test registered functions interface.
> +function divide(a, b) return a / b end
> +---
> +...
> +box.schema.func.create("divide")
> +---
> +...
> +func = box.func.divide
> +---
> +...
> +func.call({4, 2})
> +---
> +- error: 'builtin/box/schema.lua:2197: Use func:call(...) instead of func.call(...)'
> +...
> +func:call(4, 2)
> +---
> +- error: 'builtin/box/schema.lua:2212: Use func:call(table)'
> +...
We'll have to patch this result file every time we update schema.lua.
Let's please add a filter. Take a look at how engine/savepoint.test.lua
handles it.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3 6/6] box: introduce Lua persistent functions
2019-06-13 14:08 ` [PATCH v3 6/6] box: introduce Lua persistent functions Kirill Shcherbatov
@ 2019-06-18 16:23 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-18 16:23 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, Jun 13, 2019 at 05:08:26PM +0300, Kirill Shcherbatov wrote:
> This patch changes a format of the _func system space
> to support persistent Lua functions. The format of the
> system space is
>
> [<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
> <language> STR, <body> STR, <is_deterministic> BOOL,
> <is_sandboxed>, <opts> MAP]
I guess this part could be dropped as it's covered by the docbot request
right below.
>
> Closes #4182
> Needed for #1260
>
> @TarantoolBot document
> Title: Persistent Lua functions
>
> Now Tarantool supports 'persistent' Lua functions.
> Such functions are stored in snapshoot and are available after
snapshot
> restart.
> To create a persistent Lua function, specify a function body
> in box.schema.func.create call:
> e.g. body = "function(a, b) return a + b end"
>
> A Lua persistent function may be 'sandboxed'. The 'sandboxed'
> function is executed in isolated environment:
> a. only limited set of Lua functions and modules are available:
> -assert -error -pairs -ipairs -next -pcall -xpcall -type
> -print -select -string -tonumber -tostring -unpack -math -utf8;
> b. global variables are forbidden
>
> Finally, the new 'is_deterministic' flag allows to mark a
> registered function as deterministic, i.e. the function that
> can produce only one result for a given list of parameters.
>
> The new box.schema.func.create interface is:
> box.schema.func.create('funcname', <setuid = true|FALSE>,
> <if_not_exists = true|FALSE>, <language = LUA|c>,
> <body = string ('')>, <is_deterministic = true|FALSE>,
> <is_sandboxed = true|FALSE>)
Why is FALSE written in CAPS? :)
> diff --git a/src/box/alter.cc b/src/box/alter.cc
> index 081786820..5232a56c1 100644
> --- a/src/box/alter.cc
> +++ b/src/box/alter.cc
> @@ -2569,6 +2587,17 @@ func_def_new_from_tuple(struct tuple *tuple)
> /* Lua is the default. */
> def->language = FUNC_LANGUAGE_LUA;
> }
> + if (field_count > BOX_FUNC_FIELD_BODY) {
> + assert(field_count > BOX_FUNC_FIELD_OPTS);
One can try to insert a tuple that has 'body', but doesn't have 'opts'
field. We shouldn't crash in this case, but rather fail gracefully. So
please remove this assertion.
> + def->is_deterministic =
> + tuple_field_bool_xc(tuple,
> + BOX_FUNC_FIELD_IS_DETERMINISTIC);
> + def->is_sandboxed =
> + tuple_field_bool_xc(tuple,
> + BOX_FUNC_FIELD_IS_SANDBOXED);
> + } else {
> + def->is_deterministic = false;
> + }
> def_guard.is_active = false;
> return def;
> }
> diff --git a/src/box/func_def.h b/src/box/func_def.h
> index 7df6678d8..4886d5440 100644
> --- a/src/box/func_def.h
> +++ b/src/box/func_def.h
> @@ -58,11 +58,24 @@ struct func_def {
> uint32_t fid;
> /** Owner of the function. */
> uint32_t uid;
> + /** Definition of the persistent function. */
> + char *body;
> /**
> * True if the function requires change of user id before
> * invocation.
> */
> bool setuid;
> + /**
> + * Whether this function is deterministic (can produce
> + * only one result for a given list of parameters).
> + */
> + bool is_deterministic;
> + /**
> + * Whether the routine mast be initialized with isolated
s/mast/must
> + * sandbox where only a limited number if functions is
> + * available.
> + */
> + bool is_sandboxed;
> /**
> * The language of the stored function.
> */
> diff --git a/src/box/lua/call.c b/src/box/lua/call.c
> index 90c28a160..43445373f 100644
> --- a/src/box/lua/call.c
> +++ b/src/box/lua/call.c
> @@ -573,22 +573,142 @@ box_lua_eval(const char *expr, uint32_t expr_len,
> struct func_lua {
> /** Function object base class. */
> struct func base;
> + /**
> + * The reference index of Lua function object.
> + * Is equal to LUA_REFNIL when undefined.
IMO better say:
For a persistent function: a reference to the function body.
Otherwise LUA_REFNIL.
> + */
> + int lua_ref;
> };
>
> static struct func_vtab func_lua_vtab;
> +static struct func_vtab func_persistent_lua_vtab;
> +
> +static const char *default_sandbox_exports[] = {
> + "assert", "error", "ipairs", "math", "next", "pairs", "pcall", "print",
> + "select", "string", "table", "tonumber", "tostring", "type", "unpack",
> + "xpcall", "utf8",
> +};
> +
> +/**
> + * Assemble a new sandbox with given exports table on top of the
> + * Lua stack. All modules in exports list are copying deeply
> + * to ensure the immutablility of this system object.
> + */
> +static int
> +luaT_prepare_sandbox(struct lua_State *L, const char *exports[],
Let's call this function simply 'prepare_lua_sandbox'.
> + uint32_t exports_count)
Should be named 'export_count'. Also, please use 'int'.
> +{
> + int count, rc = -1;
> + const char *deepcopy = "table.deepcopy";
> + int luaL_deepcopy_func_ref = LUA_REFNIL;
> + if (luaT_func_find(L, deepcopy, deepcopy + strlen(deepcopy),
> + &count) != 0)
> + goto end;
> + luaL_deepcopy_func_ref = luaL_ref(L, LUA_REGISTRYINDEX);
> + assert(luaL_deepcopy_func_ref != LUA_REFNIL);
> + lua_createtable(L, exports_count, 0);
> + for (unsigned i = 0; i < exports_count; i++) {
> + int count;
> + uint32_t name_len = strlen(exports[i]);
> + if (luaT_func_find(L, exports[i], exports[i] + name_len,
> + &count) != 0)
> + goto end;
> + switch (lua_type(L, -1)) {
> + case LUA_TTABLE:
> + lua_rawgeti(L, LUA_REGISTRYINDEX,
> + luaL_deepcopy_func_ref);
> + lua_insert(L, -2);
> + lua_call(L, 1, LUA_MULTRET);
> + FALLTHROUGH;
You don't really need to FALLTHROUGH here. Let's please use 'break'.
> + case LUA_TFUNCTION:
> + break;
> + default:
> + unreachable();
> + }
> + lua_setfield(L, -2, exports[i]);
> + }
> + rc = 0;
> +end:
> + luaL_unref(tarantool_L, LUA_REGISTRYINDEX, luaL_deepcopy_func_ref);
> + return rc;
> +}
> +
> +static int
> +func_persistent_lua_load(struct lua_State *L)
A comment would be welcome.
> +{
> + struct func_lua *func = (struct func_lua *) lua_topointer(L, 1);
> + assert(func->base.def->body != NULL);
> + lua_settop(L, 0);
> +
> + struct region *region = &fiber()->gc;
> + size_t region_svp = region_used(region);
> + const char *load_pref = "return ";
> + uint32_t load_str_sz =
> + strlen(load_pref) + strlen(func->base.def->body) + 1;
> + char *load_str = region_alloc(region, load_str_sz);
> + if (load_str == NULL) {
> + diag_set(OutOfMemory, load_str_sz, "region", "load_str");
> + goto error;
> + }
> + sprintf(load_str, "%s%s", load_pref, func->base.def->body);
> + if (luaL_loadstring(L, load_str) != 0 ||
> + luaT_call(L, 0, 1) != 0 || !lua_isfunction(L, -1)) {
> + diag_set(ClientError, ER_LOAD_FUNCTION, func->base.def->name,
> + func->base.def->body);
Should be an error message here, not a function body. I think we should
pass diag_last_error()->errmsg.
> + goto error;
> + }
> + if (func->base.def->is_sandboxed) {
> + if (luaT_prepare_sandbox(L, default_sandbox_exports,
> + nelem(default_sandbox_exports)) != 0) {
> + diag_set(ClientError, ER_LOAD_FUNCTION,
> + func->base.def->name, func->base.def->body);
Ditto.
> + goto error;
> + }
> + lua_setfenv(L, -2);
> + }
> + func->lua_ref = luaL_ref(L, LUA_REGISTRYINDEX);
> + region_truncate(region, region_svp);
> + return lua_gettop(L);
> +error:
> + region_truncate(region, region_svp);
> + return luaT_error(L);
> +}
>
> struct func *
> func_lua_new(struct func_def *def)
> {
> - (void) def;
> assert(def->language == FUNC_LANGUAGE_LUA);
> + if (def->is_sandboxed && def->body == NULL) {
> + diag_set(ClientError, ER_CREATE_FUNCTION, def->name,
> + "is_sandboxed option may be set only for persistent "
> + "Lua function (when body option is set)");
> + return NULL;
> + }
> struct func_lua *func =
> (struct func_lua *) malloc(sizeof(struct func_lua));
> if (func == NULL) {
> diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
> return NULL;
> }
> - func->base.vtab = &func_lua_vtab;
> + if (def->body != NULL) {
> + /** Load persistent Lua function. */
> + struct lua_State *L = lua_newthread(tarantool_L);
> + int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
> + lua_pushcfunction(L, func_persistent_lua_load);
> + func->base.def = def;
> + lua_pushlightuserdata(L, func);
> + int rc = luaT_call(L, 1, LUA_MULTRET);
Is this really necessary? Can't you call func_persistent_lua_load
directly? AFAICS it doesn't throw any exceptions.
> + luaL_unref(tarantool_L, LUA_REGISTRYINDEX, coro_ref);
> + if (rc != 0) {
> + free(func);
> + return NULL;
> + }
> + assert(func->lua_ref != LUA_REFNIL);
> + func->base.vtab = &func_persistent_lua_vtab;
> + } else {
> + func->lua_ref = LUA_REFNIL;
> + func->base.vtab = &func_lua_vtab;
> + }
> return &func->base;
> }
>
> @@ -614,6 +734,53 @@ static struct func_vtab func_lua_vtab = {
> .destroy = func_lua_unload,
> };
>
> +static void
> +func_persistent_lua_unload(struct func *base)
func_persistent_lua_load takes lua_State while this function takes
struct func. Looks inconsistent.
> +{
> + assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
> + base->def->body != NULL);
> + assert(base->vtab == &func_persistent_lua_vtab);
> + struct func_lua *func = (struct func_lua *) base;
> + luaL_unref(tarantool_L, LUA_REGISTRYINDEX, func->lua_ref);
> + func->lua_ref = LUA_REFNIL;
> +}
> +
> +static int
> +func_persistent_lua_call_cb(lua_State *L)
I'd try to make this function more generic: name it
execute_lua_call_by_ref and pass lua_ref to it instead of
func_lua. We could probably reuse the same context we use
for executing CALL/EVAL (lua_exec_ctx).
> +{
> + struct port *in_port = (struct port *) lua_topointer(L, 1);
> + struct func_lua *func = (struct func_lua *) lua_topointer(L, 2);
> + lua_settop(L, 0); /* clear the stack to simplify the logic below */
> +
> + assert(func->lua_ref != LUA_REFNIL);
> + lua_rawgeti(L, LUA_REGISTRYINDEX, func->lua_ref);
> +
> + /* Push the rest of args (a tuple). */
> + int top = lua_gettop(L);
> + port_dump_lua(in_port, L, true);
> + int arg_count = lua_gettop(L) - top;
> +
> + lua_call(L, arg_count, LUA_MULTRET);
> + return lua_gettop(L);
> +}
> +
> +static inline int
> +func_persistent_lua_call(struct func *base, struct port *in_port,
> + struct port *out_port)
> +{
> + assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
> + base->def->body != NULL);
> + assert(base->vtab == &func_persistent_lua_vtab);
> + return box_process_lua(func_persistent_lua_call_cb, base,
> + in_port, out_port);
> +
> +}
> +
> +static struct func_vtab func_persistent_lua_vtab = {
> + .call = func_persistent_lua_call,
> + .destroy = func_persistent_lua_unload,
> +};
> +
> static int
> lbox_module_reload(lua_State *L)
> {
> diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
> index 9c3ee063c..9d8df54dc 100644
> --- a/src/box/lua/schema.lua
> +++ b/src/box/lua/schema.lua
> @@ -820,10 +821,32 @@ local function create_vcollation_space()
> box.space[box.schema.VCOLLATION_ID]:format(format)
> end
>
> +local function upgrade_func_to_2_2_1()
> + log.info("Update _func format")
> + local _func = box.space[box.schema.FUNC_ID]
> + local format = {}
> + format[1] = {name='id', type='unsigned'}
> + format[2] = {name='owner', type='unsigned'}
> + format[3] = {name='name', type='string'}
> + format[4] = {name='setuid', type='unsigned'}
> + format[5] = {name='language', type='string'}
> + format[6] = {name='body', type='string'}
> + format[7] = {name='is_deterministic', type='boolean'}
> + format[8] = {name='is_sandboxed', type='boolean'}
> + format[9] = {name='opts', type='map'}
> + for _, v in box.space._func:pairs() do
> + _ = box.space._func:replace({v.id, v.owner, v.name, v.setuid,
> + v[5] or 'LUA', '', false, false,
Why not v.language?
> + setmap({})})
> + end
> + _func:format(format)
> +end
> +
> local function upgrade_to_2_2_1()
> upgrade_sequence_to_2_2_1()
> upgrade_ck_constraint_to_2_2_1()
> create_vcollation_space()
> + upgrade_func_to_2_2_1()
> end
>
> --------------------------------------------------------------------------------
> diff --git a/test/box/function1.result b/test/box/function1.result
> index 331bd466a..172f56082 100644
> --- a/test/box/function1.result
> +++ b/test/box/function1.result
> @@ -504,3 +510,120 @@ box.schema.func.drop('secret_leak')
> box.schema.func.drop('secret')
> ---
> ...
> +--
> +-- gh-4182: Introduce persistent Lua functions.
> +--
> +test_run:cmd("setopt delimiter ';'")
> +---
> +- true
> +...
> +body = [[function(tuple)
> + if type(tuple.address) ~= 'string' then
> + return nil, 'Invalid field type'
> + end
> + local t = tuple.address:upper():split()
> + for k,v in pairs(t) do t[k] = v end
> + return t
> + end
> +]]
> +test_run:cmd("setopt delimiter ''");
> +---
> +...
> +box.schema.func.create('addrsplit', {body = body, language = "C"})
> +---
> +- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
> + not compatible with C language'
> +...
> +box.schema.func.create('addrsplit', {is_sandboxed = true, language = "C"})
> +---
> +- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
> + not compatible with C language'
> +...
> +box.schema.func.create('addrsplit', {is_sandboxed = true})
> +---
> +- error: 'Failed to create function ''addrsplit'': is_sandboxed option may be set
> + only for persistent Lua function (when body option is set)'
> +...
> +box.schema.func.create('invalid', {body = "function(tuple) ret tuple"})
> +---
> +- error: 'Failed to dynamically load function ''invalid'': function(tuple) ret tuple'
> +...
Please also check what happens when body is loaded okay, but happens to
be not a function.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 6/6] box: introduce Lua persistent functions
2019-06-18 16:23 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
>> The new box.schema.func.create interface is:
>> box.schema.func.create('funcname', <setuid = true|FALSE>,
>> <if_not_exists = true|FALSE>, <language = LUA|c>,
>> <body = string ('')>, <is_deterministic = true|FALSE>,
>> <is_sandboxed = true|FALSE>)
>
> Why is FALSE written in CAPS? :)
It is default values.
>> + _ = box.space._func:replace({v.id, v.owner, v.name, v.setuid,
>> + v[5] or 'LUA', '', false, false,
>
> Why not v.language?
It was a bug in previous Tarantool version: _func:format() lacks of language field name.
===========================================================
Closes #4182
Needed for #1260
@TarantoolBot document
Title: Persistent Lua functions
Now Tarantool supports 'persistent' Lua functions.
Such functions are stored in snapshot and are available after
restart.
To create a persistent Lua function, specify a function body
in box.schema.func.create call:
e.g. body = "function(a, b) return a + b end"
A Lua persistent function may be 'sandboxed'. The 'sandboxed'
function is executed in isolated environment:
a. only limited set of Lua functions and modules are available:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8;
b. global variables are forbidden
Finally, the new 'is_deterministic' flag allows to mark a
registered function as deterministic, i.e. the function that
can produce only one result for a given list of parameters.
The new box.schema.func.create interface is:
box.schema.func.create('funcname', <setuid = true|FALSE>,
<if_not_exists = true|FALSE>, <language = LUA|c>,
<body = string ('')>, <is_deterministic = true|FALSE>,
<is_sandboxed = true|FALSE>)
Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
is_deterministic = true, is_sandboxed = true})
box.func.sum
---
- is_sandboxed: true
is_deterministic: true
id: 2
setuid: false
body: function(a, b) return a + b end
name: sum
language: LUA
...
box.func.sum:call({1, 3})
---
- 4
...
---
src/box/alter.cc | 54 ++++++---
src/box/bootstrap.snap | Bin 4475 -> 4532 bytes
src/box/func.c | 7 +-
src/box/func_def.c | 8 ++
src/box/func_def.h | 20 +++-
src/box/lua/call.c | 216 ++++++++++++++++++++++++++++++++++-
src/box/lua/schema.lua | 12 +-
src/box/lua/upgrade.lua | 25 +++-
src/box/schema_def.h | 4 +
test/box-py/bootstrap.result | 6 +-
test/box/access_misc.result | 6 +-
test/box/function1.result | 191 +++++++++++++++++++++++++++++--
test/box/function1.test.lua | 67 ++++++++++-
13 files changed, 582 insertions(+), 34 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 33f9b0a71..532a24cef 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2537,31 +2537,49 @@ func_def_get_ids_from_tuple(struct tuple *tuple, uint32_t *fid, uint32_t *uid)
static struct func_def *
func_def_new_from_tuple(struct tuple *tuple)
{
- uint32_t len;
- const char *name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME,
- &len);
- if (len > BOX_NAME_MAX)
+ uint32_t field_count = tuple_field_count(tuple);
+ uint32_t name_len, body_len;
+ const char *name, *body;
+ name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME, &name_len);
+ if (name_len > BOX_NAME_MAX) {
tnt_raise(ClientError, ER_CREATE_FUNCTION,
tt_cstr(name, BOX_INVALID_NAME_MAX),
"function name is too long");
- identifier_check_xc(name, len);
- struct func_def *def = (struct func_def *) malloc(func_def_sizeof(len));
+ }
+ identifier_check_xc(name, name_len);
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ body = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_BODY,
+ &body_len);
+ } else {
+ body = NULL;
+ body_len = 0;
+ }
+ uint32_t def_sz = func_def_sizeof(name_len, body_len);
+ struct func_def *def =
+ (struct func_def *) malloc(def_sz);
if (def == NULL)
- tnt_raise(OutOfMemory, func_def_sizeof(len), "malloc", "def");
+ tnt_raise(OutOfMemory, def_sz, "malloc", "def");
auto def_guard = make_scoped_guard([=] { free(def); });
func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid);
if (def->fid > BOX_FUNCTION_MAX) {
tnt_raise(ClientError, ER_CREATE_FUNCTION,
- tt_cstr(name, len), "function id is too big");
+ tt_cstr(name, name_len), "function id is too big");
}
- memcpy(def->name, name, len);
- def->name[len] = 0;
- def->name_len = len;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
+ memcpy(def->name, name, name_len);
+ def->name[name_len] = 0;
+ def->name_len = name_len;
+ if (body_len > 0) {
+ def->body = def->name + name_len + 1;
+ memcpy(def->body, body, body_len);
+ def->body[body_len] = 0;
+ } else {
+ def->body = NULL;
+ }
+ if (field_count > BOX_FUNC_FIELD_SETUID)
def->setuid = tuple_field_u32_xc(tuple, BOX_FUNC_FIELD_SETUID);
else
def->setuid = false;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_LANGUAGE) {
+ if (field_count > BOX_FUNC_FIELD_LANGUAGE) {
const char *language =
tuple_field_cstr_xc(tuple, BOX_FUNC_FIELD_LANGUAGE);
def->language = STR2ENUM(func_language, language);
@@ -2573,6 +2591,16 @@ func_def_new_from_tuple(struct tuple *tuple)
/* Lua is the default. */
def->language = FUNC_LANGUAGE_LUA;
}
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ def->is_deterministic =
+ tuple_field_bool_xc(tuple,
+ BOX_FUNC_FIELD_IS_DETERMINISTIC);
+ def->is_sandboxed =
+ tuple_field_bool_xc(tuple,
+ BOX_FUNC_FIELD_IS_SANDBOXED);
+ } else {
+ def->is_deterministic = false;
+ }
def_guard.is_active = false;
return def;
}
diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap
index 56943ef7e7d0fe0e3dbb5d346544e2c78c3dc154..fb313d66eaa965afb373ef7820804365309886bd 100644
diff --git a/src/box/func.c b/src/box/func.c
index c57027809..d7c35cf68 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -416,8 +416,13 @@ static struct func_vtab func_c_vtab;
static struct func *
func_c_new(struct func_def *def)
{
- (void) def;
assert(def->language == FUNC_LANGUAGE_C);
+ if (def->body != NULL || def->is_sandboxed) {
+ diag_set(ClientError, ER_CREATE_FUNCTION, def->name,
+ "body and is_sandboxed options are not compatible "
+ "with C language");
+ return NULL;
+ }
struct func_c *func = (struct func_c *) malloc(sizeof(struct func_c));
if (func == NULL) {
diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
diff --git a/src/box/func_def.c b/src/box/func_def.c
index 2b135e2d7..73e493786 100644
--- a/src/box/func_def.c
+++ b/src/box/func_def.c
@@ -14,7 +14,15 @@ func_def_cmp(struct func_def *def1, struct func_def *def2)
return def1->setuid - def2->setuid;
if (def1->language != def2->language)
return def1->language - def2->language;
+ if (def1->is_deterministic != def2->is_deterministic)
+ return def1->is_deterministic - def2->is_deterministic;
+ if (def1->is_sandboxed != def2->is_sandboxed)
+ return def1->is_sandboxed - def2->is_sandboxed;
if (strcmp(def1->name, def2->name) != 0)
return strcmp(def1->name, def2->name);
+ if ((def1->body != NULL) != (def2->body != NULL))
+ return def1->body - def2->body;
+ if (def1->body != NULL && strcmp(def1->body, def2->body) != 0)
+ return strcmp(def1->body, def2->body);
return 0;
}
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 866d425a1..12f807f0e 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -58,11 +58,24 @@ struct func_def {
uint32_t fid;
/** Owner of the function. */
uint32_t uid;
+ /** Definition of the persistent function. */
+ char *body;
/**
* True if the function requires change of user id before
* invocation.
*/
bool setuid;
+ /**
+ * Whether this function is deterministic (can produce
+ * only one result for a given list of parameters).
+ */
+ bool is_deterministic;
+ /**
+ * Whether the routine must be initialized with isolated
+ * sandbox where only a limited number if functions is
+ * available.
+ */
+ bool is_sandboxed;
/**
* The language of the stored function.
*/
@@ -79,10 +92,13 @@ struct func_def {
* for a function of length @a a name_len.
*/
static inline size_t
-func_def_sizeof(uint32_t name_len)
+func_def_sizeof(uint32_t name_len, uint32_t body_len)
{
/* +1 for '\0' name terminating. */
- return sizeof(struct func_def) + name_len + 1;
+ size_t sz = sizeof(struct func_def) + name_len + 1;
+ if (body_len > 0)
+ sz += body_len + 1;
+ return sz;
}
/** Compare two given function definitions. */
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index f98ab42ac..4d83f53ac 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -300,6 +300,7 @@ struct execute_lua_ctx {
uint32_t name_len;
};
struct mpstream *stream;
+ int lua_ref;
};
struct port *args;
};
@@ -328,6 +329,24 @@ execute_lua_call(lua_State *L)
return lua_gettop(L);
}
+static int
+execute_lua_call_by_ref(lua_State *L)
+{
+ struct execute_lua_ctx *ctx =
+ (struct execute_lua_ctx *) lua_topointer(L, 1);
+ lua_settop(L, 0); /* clear the stack to simplify the logic below */
+
+ lua_rawgeti(L, LUA_REGISTRYINDEX, ctx->lua_ref);
+
+ /* Push the rest of args (a tuple). */
+ int top = lua_gettop(L);
+ port_dump_lua(ctx->args, L, true);
+ int arg_count = lua_gettop(L) - top;
+
+ lua_call(L, arg_count, LUA_MULTRET);
+ return lua_gettop(L);
+}
+
static int
execute_lua_eval(lua_State *L)
{
@@ -536,22 +555,168 @@ box_lua_eval(const char *expr, uint32_t expr_len,
struct func_lua {
/** Function object base class. */
struct func base;
+ /**
+ * For a persistent function: a reference to the
+ * function body. Otherwise LUA_REFNIL.
+ */
+ int lua_ref;
};
static struct func_vtab func_lua_vtab;
+static struct func_vtab func_persistent_lua_vtab;
+
+static const char *default_sandbox_exports[] = {
+ "assert", "error", "ipairs", "math", "next", "pairs", "pcall", "print",
+ "select", "string", "table", "tonumber", "tostring", "type", "unpack",
+ "xpcall", "utf8",
+};
+
+/**
+ * Assemble a new sandbox with given exports table on the top of
+ * a given Lua stack. All modules in exports list are copying
+ * deeply to ensure the immutability of this system object.
+ */
+static int
+prepare_lua_sandbox(struct lua_State *L, const char *exports[],
+ int export_count)
+{
+ lua_createtable(L, export_count, 0);
+ if (export_count == 0)
+ return 0;
+ int rc = -1;
+ const char *deepcopy = "table.deepcopy";
+ int luaL_deepcopy_func_ref = LUA_REFNIL;
+ int ret = box_lua_find(L, deepcopy, deepcopy + strlen(deepcopy));
+ if (ret < 0)
+ goto end;
+ luaL_deepcopy_func_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ assert(luaL_deepcopy_func_ref != LUA_REFNIL);
+ for (int i = 0; i < export_count; i++) {
+ uint32_t name_len = strlen(exports[i]);
+ ret = box_lua_find(L, exports[i], exports[i] + name_len);
+ if (ret < 0)
+ goto end;
+ switch (lua_type(L, -1)) {
+ case LUA_TTABLE:
+ lua_rawgeti(L, LUA_REGISTRYINDEX,
+ luaL_deepcopy_func_ref);
+ lua_insert(L, -2);
+ lua_call(L, 1, 1);
+ break;
+ case LUA_TFUNCTION:
+ break;
+ default:
+ unreachable();
+ }
+ lua_setfield(L, -2, exports[i]);
+ }
+ rc = 0;
+end:
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, luaL_deepcopy_func_ref);
+ return rc;
+}
+
+/**
+ * Assemble a Lua function object by user-defined function body.
+ */
+static int
+func_persistent_lua_load(struct func_lua *func)
+{
+ int rc = -1;
+ int top = lua_gettop(tarantool_L);
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+ const char *load_pref = "return ";
+ uint32_t load_str_sz =
+ strlen(load_pref) + strlen(func->base.def->body) + 1;
+ char *load_str = region_alloc(region, load_str_sz);
+ if (load_str == NULL) {
+ diag_set(OutOfMemory, load_str_sz, "region", "load_str");
+ return -1;
+ }
+ sprintf(load_str, "%s%s", load_pref, func->base.def->body);
+
+ /*
+ * Perform loading of the persistent Lua function
+ * in a new sandboxed Lua thread. The sandbox is
+ * required to guarantee the safety of executing
+ * an arbitrary user-defined code
+ * (e.g. body = 'fiber.yield()').
+ */
+ struct lua_State *coro_L = lua_newthread(tarantool_L);
+ if (!func->base.def->is_sandboxed) {
+ /*
+ * Keep an original env to apply for non-sandboxed
+ * persistent function. It is required because
+ * built object inherits parent env.
+ */
+ lua_getfenv(tarantool_L, -1);
+ lua_insert(tarantool_L, -2);
+ }
+ if (prepare_lua_sandbox(tarantool_L, NULL, 0) != 0)
+ unreachable();
+ lua_setfenv(tarantool_L, -2);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ if (luaL_loadstring(coro_L, load_str) != 0 ||
+ lua_pcall(coro_L, 0, 1, 0) != 0) {
+ diag_set(ClientError, ER_LOAD_FUNCTION, func->base.def->name,
+ luaT_tolstring(coro_L, -1, NULL));
+ goto end;
+ }
+ if (!lua_isfunction(coro_L, -1)) {
+ diag_set(ClientError, ER_LOAD_FUNCTION, func->base.def->name,
+ "given body doesn't define a function");
+ goto end;
+ }
+ lua_xmove(coro_L, tarantool_L, 1);
+ if (func->base.def->is_sandboxed) {
+ if (prepare_lua_sandbox(tarantool_L, default_sandbox_exports,
+ nelem(default_sandbox_exports)) != 0) {
+ diag_set(ClientError, ER_LOAD_FUNCTION,
+ func->base.def->name,
+ diag_last_error(diag_get())->errmsg);
+ goto end;
+ }
+ } else {
+ lua_insert(tarantool_L, -2);
+ }
+ lua_setfenv(tarantool_L, -2);
+ func->lua_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ rc = 0;
+end:
+ lua_settop(tarantool_L, top);
+ region_truncate(region, region_svp);
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, coro_ref);
+ return rc;
+}
struct func *
func_lua_new(struct func_def *def)
{
- (void) def;
assert(def->language == FUNC_LANGUAGE_LUA);
+ if (def->is_sandboxed && def->body == NULL) {
+ diag_set(ClientError, ER_CREATE_FUNCTION, def->name,
+ "is_sandboxed option may be set only for persistent "
+ "Lua function (when body option is set)");
+ return NULL;
+ }
struct func_lua *func =
(struct func_lua *) malloc(sizeof(struct func_lua));
if (func == NULL) {
diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
return NULL;
}
- func->base.vtab = &func_lua_vtab;
+ if (def->body != NULL) {
+ func->base.def = def;
+ func->base.vtab = &func_persistent_lua_vtab;
+ if (func_persistent_lua_load(func) != 0) {
+ free(func);
+ return NULL;
+ }
+ } else {
+ func->lua_ref = LUA_REFNIL;
+ func->base.vtab = &func_lua_vtab;
+ }
return &func->base;
}
@@ -576,6 +741,42 @@ static struct func_vtab func_lua_vtab = {
.destroy = func_lua_destroy,
};
+static void
+func_persistent_lua_unload(struct func_lua *func)
+{
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, func->lua_ref);
+}
+
+static void
+func_persistent_lua_destroy(struct func *base)
+{
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
+ base->def->body != NULL);
+ assert(base->vtab == &func_persistent_lua_vtab);
+ struct func_lua *func = (struct func_lua *) base;
+ func_persistent_lua_unload(func);
+ free(func);
+}
+
+static inline int
+func_persistent_lua_call(struct func *base, struct port *args, struct port *ret)
+{
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA &&
+ base->def->body != NULL);
+ assert(base->vtab == &func_persistent_lua_vtab);
+ struct func_lua *func = (struct func_lua *)base;
+ struct execute_lua_ctx ctx;
+ ctx.lua_ref = func->lua_ref;
+ ctx.args = args;
+ return box_process_lua(execute_lua_call_by_ref, &ctx, ret);
+
+}
+
+static struct func_vtab func_persistent_lua_vtab = {
+ .call = func_persistent_lua_call,
+ .destroy = func_persistent_lua_destroy,
+};
+
static int
lbox_module_reload(lua_State *L)
{
@@ -669,6 +870,17 @@ lbox_func_new(struct lua_State *L, struct func *func)
lua_pushstring(L, "language");
lua_pushstring(L, func_language_strs[func->def->language]);
lua_settable(L, top);
+ lua_pushstring(L, "is_deterministic");
+ lua_pushboolean(L, func->def->is_deterministic);
+ lua_settable(L, top);
+ if (func->def->body != NULL) {
+ lua_pushstring(L, "body");
+ lua_pushstring(L, func->def->body);
+ lua_settable(L, top);
+ lua_pushstring(L, "is_sandboxed");
+ lua_pushboolean(L, func->def->is_sandboxed);
+ lua_settable(L, top);
+ }
/* Bless func object. */
lua_getfield(L, LUA_GLOBALSINDEX, "box");
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 9c3ee063c..9d8df54dc 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2138,7 +2138,9 @@ box.schema.func.create = function(name, opts)
opts = opts or {}
check_param_table(opts, { setuid = 'boolean',
if_not_exists = 'boolean',
- language = 'string'})
+ language = 'string', body = 'string',
+ is_deterministic = 'boolean',
+ is_sandboxed = 'boolean', opts = 'table'})
local _func = box.space[box.schema.FUNC_ID]
local _vfunc = box.space[box.schema.VFUNC_ID]
local func = _vfunc.index.name:get{name}
@@ -2148,10 +2150,14 @@ box.schema.func.create = function(name, opts)
end
return
end
- opts = update_param_table(opts, { setuid = false, language = 'lua'})
+ opts = update_param_table(opts, { setuid = false, language = 'lua',
+ body = '', is_deterministic = false,
+ is_sandboxed = false, opts = setmap{}})
opts.language = string.upper(opts.language)
opts.setuid = opts.setuid and 1 or 0
- _func:auto_increment{session.euid(), name, opts.setuid, opts.language}
+ _func:auto_increment{session.euid(), name, opts.setuid, opts.language,
+ opts.body, opts.is_deterministic, opts.is_sandboxed,
+ opts.opts}
end
box.schema.func.drop = function(name, opts)
diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua
index 3385b8e17..f2edf86df 100644
--- a/src/box/lua/upgrade.lua
+++ b/src/box/lua/upgrade.lua
@@ -326,7 +326,8 @@ local function initial_1_7_5()
-- create "box.schema.user.info" function
log.info('create function "box.schema.user.info" with setuid')
- _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA'}
+ _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA',
+ '', false, false, MAP}
-- grant 'public' role access to 'box.schema.user.info' function
log.info('grant execute on function "box.schema.user.info" to public')
@@ -820,10 +821,32 @@ local function create_vcollation_space()
box.space[box.schema.VCOLLATION_ID]:format(format)
end
+local function upgrade_func_to_2_2_1()
+ log.info("Update _func format")
+ local _func = box.space[box.schema.FUNC_ID]
+ local format = {}
+ format[1] = {name='id', type='unsigned'}
+ format[2] = {name='owner', type='unsigned'}
+ format[3] = {name='name', type='string'}
+ format[4] = {name='setuid', type='unsigned'}
+ format[5] = {name='language', type='string'}
+ format[6] = {name='body', type='string'}
+ format[7] = {name='is_deterministic', type='boolean'}
+ format[8] = {name='is_sandboxed', type='boolean'}
+ format[9] = {name='opts', type='map'}
+ for _, v in box.space._func:pairs() do
+ _ = box.space._func:replace({v.id, v.owner, v.name, v.setuid,
+ v[5] or 'LUA', '', false, false,
+ setmap({})})
+ end
+ _func:format(format)
+end
+
local function upgrade_to_2_2_1()
upgrade_sequence_to_2_2_1()
upgrade_ck_constraint_to_2_2_1()
create_vcollation_space()
+ upgrade_func_to_2_2_1()
end
--------------------------------------------------------------------------------
diff --git a/src/box/schema_def.h b/src/box/schema_def.h
index 88b5502b8..ac2b3bfef 100644
--- a/src/box/schema_def.h
+++ b/src/box/schema_def.h
@@ -167,6 +167,10 @@ enum {
BOX_FUNC_FIELD_NAME = 2,
BOX_FUNC_FIELD_SETUID = 3,
BOX_FUNC_FIELD_LANGUAGE = 4,
+ BOX_FUNC_FIELD_BODY = 5,
+ BOX_FUNC_FIELD_IS_DETERMINISTIC = 6,
+ BOX_FUNC_FIELD_IS_SANDBOXED = 7,
+ BOX_FUNC_FIELD_OPTS = 8,
};
/** _collation fields. */
diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result
index b20dc41e5..5ca7f3740 100644
--- a/test/box-py/bootstrap.result
+++ b/test/box-py/bootstrap.result
@@ -53,7 +53,9 @@ box.space._space:select{}
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'is_deterministic', 'type': 'boolean'}, {'name': 'is_sandboxed',
+ 'type': 'boolean'}, {'name': 'opts', 'type': 'map'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -152,7 +154,7 @@ box.space._user:select{}
...
box.space._func:select{}
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', false, false, {}]
...
box.space._priv:select{}
---
diff --git a/test/box/access_misc.result b/test/box/access_misc.result
index 53d366106..e7a6f0984 100644
--- a/test/box/access_misc.result
+++ b/test/box/access_misc.result
@@ -793,7 +793,9 @@ box.space._space:select()
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'is_deterministic', 'type': 'boolean'}, {'name': 'is_sandboxed',
+ 'type': 'boolean'}, {'name': 'opts', 'type': 'map'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -829,7 +831,7 @@ box.space._space:select()
...
box.space._func:select()
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', false, false, {}]
...
session = nil
---
diff --git a/test/box/function1.result b/test/box/function1.result
index 99006926e..3630f7ede 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -16,7 +16,10 @@ c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
---
...
-box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+function setmap(tab) return setmetatable(tab, { __serialize = 'map' }) end
+---
+...
+box.space._func:replace{2, 1, 'function1', 0, 'LUA', '', false, false, setmap({})}
---
- error: function does not support alter
...
@@ -59,10 +62,11 @@ c:call('function1.args', { 15 })
...
box.func["function1.args"]
---
-- language: C
+- is_deterministic: false
+ id: 2
setuid: false
name: function1.args
- id: 2
+ language: C
...
box.func["function1.args"]:call()
---
@@ -368,10 +372,11 @@ func:drop()
...
func
---
-- language: LUA
+- is_deterministic: false
+ id: 2
setuid: false
name: divide
- id: 2
+ language: LUA
...
func.drop()
---
@@ -424,10 +429,11 @@ func:drop()
...
func
---
-- language: C
+- is_deterministic: false
+ id: 2
setuid: false
name: function1.divide
- id: 2
+ language: C
...
func:drop()
---
@@ -510,6 +516,177 @@ box.schema.func.drop('secret_leak')
box.schema.func.drop('secret')
---
...
+--
+-- gh-4182: Introduce persistent Lua functions.
+--
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then
+ return nil, 'Invalid field type'
+ end
+ local t = tuple.address:upper():split()
+ for k,v in pairs(t) do t[k] = v end
+ return t
+ end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('addrsplit', {body = body, language = "C"})
+---
+- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
+ not compatible with C language'
+...
+box.schema.func.create('addrsplit', {is_sandboxed = true, language = "C"})
+---
+- error: 'Failed to create function ''addrsplit'': body and is_sandboxed options are
+ not compatible with C language'
+...
+box.schema.func.create('addrsplit', {is_sandboxed = true})
+---
+- error: 'Failed to create function ''addrsplit'': is_sandboxed option may be set
+ only for persistent Lua function (when body option is set)'
+...
+box.schema.func.create('invalid', {body = "function(tuple) ret tuple"})
+---
+- error: 'Failed to dynamically load function ''invalid'': [string "return function(tuple)
+ ret tuple"]:1: ''='' expected near ''tuple'''
+...
+box.schema.func.create('addrsplit', {body = body, is_deterministic = true})
+---
+...
+box.schema.user.grant('guest', 'execute', 'function', 'addrsplit')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+---
+- ['MOSCOW', 'DOLGOPRUDNY']
+...
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+---
+- - MOSCOW
+ - DOLGOPRUDNY
+...
+conn:close()
+---
+...
+box.snapshot()
+---
+- ok
+...
+test_run:cmd("restart server default")
+test_run = require('test_run').new()
+---
+...
+test_run:cmd("push filter '(.builtin/.*.lua):[0-9]+' to '\\1'")
+---
+- true
+...
+net = require('net.box')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+---
+- ['MOSCOW', 'DOLGOPRUDNY']
+...
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+---
+- - MOSCOW
+ - DOLGOPRUDNY
+...
+conn:close()
+---
+...
+box.schema.user.revoke('guest', 'execute', 'function', 'addrsplit')
+---
+...
+box.func.addrsplit:drop()
+---
+...
+-- Test sandboxed functions.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body = [[function(number)
+ math.abs = math.log
+ return math.abs(number)
+ end]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('monkey', {body = body, is_sandboxed = true})
+---
+...
+box.func.monkey:call({1})
+---
+- 0
+...
+math.abs(1)
+---
+- 1
+...
+box.func.monkey:drop()
+---
+...
+sum = 0
+---
+...
+function inc_g(val) sum = sum + val end
+---
+...
+box.schema.func.create('call_inc_g', {body = "function(val) inc_g(val) end"})
+---
+...
+box.func.call_inc_g:call({1})
+---
+...
+assert(sum == 1)
+---
+- true
+...
+box.schema.func.create('call_inc_g_safe', {body = "function(val) inc_g(val) end", is_sandboxed = true})
+---
+...
+box.func.call_inc_g_safe:call({1})
+---
+- error: '[string "return function(val) inc_g(val) end"]:1: attempt to call global
+ ''inc_g'' (a nil value)'
+...
+assert(sum == 1)
+---
+- true
+...
+box.func.call_inc_g:drop()
+---
+...
+box.func.call_inc_g_safe:drop()
+---
+...
+-- Test persistent function assemble corner cases
+box.schema.func.create('compiletime_tablef', {body = "{}"})
+---
+- error: 'Failed to dynamically load function ''compiletime_tablef'': given body doesn''t
+ define a function'
+...
+box.schema.func.create('compiletime_call_inc_g', {body = "inc_g()"})
+---
+- error: 'Failed to dynamically load function ''compiletime_call_inc_g'': [string
+ "return inc_g()"]:1: attempt to call global ''inc_g'' (a nil value)'
+...
+assert(sum == 1)
+---
+- true
+...
test_run:cmd("clear filter")
---
- true
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index 25966b915..b77ebe06b 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -7,7 +7,8 @@ net = require('net.box')
c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
-box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+function setmap(tab) return setmetatable(tab, { __serialize = 'map' }) end
+box.space._func:replace{2, 1, 'function1', 0, 'LUA', '', false, false, setmap({})}
box.schema.user.grant('guest', 'execute', 'function', 'function1')
_ = box.schema.space.create('test')
_ = box.space.test:create_index('primary')
@@ -180,4 +181,68 @@ box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
box.schema.func.drop('secret_leak')
box.schema.func.drop('secret')
+--
+-- gh-4182: Introduce persistent Lua functions.
+--
+test_run:cmd("setopt delimiter ';'")
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then
+ return nil, 'Invalid field type'
+ end
+ local t = tuple.address:upper():split()
+ for k,v in pairs(t) do t[k] = v end
+ return t
+ end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('addrsplit', {body = body, language = "C"})
+box.schema.func.create('addrsplit', {is_sandboxed = true, language = "C"})
+box.schema.func.create('addrsplit', {is_sandboxed = true})
+box.schema.func.create('invalid', {body = "function(tuple) ret tuple"})
+box.schema.func.create('addrsplit', {body = body, is_deterministic = true})
+box.schema.user.grant('guest', 'execute', 'function', 'addrsplit')
+conn = net.connect(box.cfg.listen)
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+conn:close()
+box.snapshot()
+test_run:cmd("restart server default")
+test_run = require('test_run').new()
+test_run:cmd("push filter '(.builtin/.*.lua):[0-9]+' to '\\1'")
+net = require('net.box')
+conn = net.connect(box.cfg.listen)
+conn:call('addrsplit', {{address = "Moscow Dolgoprudny"}})
+box.func.addrsplit:call({{address = "Moscow Dolgoprudny"}})
+conn:close()
+box.schema.user.revoke('guest', 'execute', 'function', 'addrsplit')
+box.func.addrsplit:drop()
+
+-- Test sandboxed functions.
+test_run:cmd("setopt delimiter ';'")
+body = [[function(number)
+ math.abs = math.log
+ return math.abs(number)
+ end]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('monkey', {body = body, is_sandboxed = true})
+box.func.monkey:call({1})
+math.abs(1)
+box.func.monkey:drop()
+
+sum = 0
+function inc_g(val) sum = sum + val end
+box.schema.func.create('call_inc_g', {body = "function(val) inc_g(val) end"})
+box.func.call_inc_g:call({1})
+assert(sum == 1)
+box.schema.func.create('call_inc_g_safe', {body = "function(val) inc_g(val) end", is_sandboxed = true})
+box.func.call_inc_g_safe:call({1})
+assert(sum == 1)
+box.func.call_inc_g:drop()
+box.func.call_inc_g_safe:drop()
+
+-- Test persistent function assemble corner cases
+box.schema.func.create('compiletime_tablef', {body = "{}"})
+box.schema.func.create('compiletime_call_inc_g', {body = "inc_g()"})
+assert(sum == 1)
+
test_run:cmd("clear filter")
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 5/6] box: refactor box_lua_find helper
2019-06-18 14:22 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
Done.
============================================================
The box_lua_find routine used to work with an empty stack only.
It is unacceptable in following patches because this helper
need to be reused in following patches for environment table
construction.
Needed for #4182, #1260
---
src/box/lua/call.c | 34 +++++++++++++++++++++-------------
1 file changed, 21 insertions(+), 13 deletions(-)
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 8c70c1088..f98ab42ac 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -56,7 +56,7 @@ static int
box_lua_find(lua_State *L, const char *name, const char *name_end)
{
int index = LUA_GLOBALSINDEX;
- int objstack = 0;
+ int objstack = 0, top = lua_gettop(L);
const char *start = name, *end;
while ((end = (const char *) memchr(start, '.', name_end - start))) {
@@ -66,7 +66,7 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
if (! lua_istable(L, -1)) {
diag_set(ClientError, ER_NO_SUCH_PROC,
name_end - name, name);
- luaT_error(L);
+ return -1;
}
start = end + 1; /* next piece of a.b.c */
index = lua_gettop(L); /* top of the stack */
@@ -81,11 +81,12 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
lua_islightuserdata(L, -1) || lua_isuserdata(L, -1) )) {
diag_set(ClientError, ER_NO_SUCH_PROC,
name_end - name, name);
- luaT_error(L);
+ return -1;
}
+
start = end + 1; /* next piece of a.b.c */
index = lua_gettop(L); /* top of the stack */
- objstack = index;
+ objstack = index - top;
}
@@ -96,21 +97,24 @@ box_lua_find(lua_State *L, const char *name, const char *name_end)
* for us, but our own message is more verbose. */
diag_set(ClientError, ER_NO_SUCH_PROC,
name_end - name, name);
- luaT_error(L);
+ return -1;
}
+
/* setting stack that it would contain only
* the function pointer. */
if (index != LUA_GLOBALSINDEX) {
if (objstack == 0) { /* no object, only a function */
- lua_replace(L, 1);
+ lua_replace(L, top + 1);
+ lua_pop(L, lua_gettop(L) - top - 1);
} else if (objstack == 1) { /* just two values, swap them */
lua_insert(L, -2);
+ lua_pop(L, lua_gettop(L) - top - 2);
} else { /* long path */
- lua_insert(L, 1);
- lua_insert(L, 2);
+ lua_insert(L, top + 1);
+ lua_insert(L, top + 2);
+ lua_pop(L, objstack - 1);
objstack = 1;
}
- lua_settop(L, 1 + objstack);
}
return 1 + objstack;
}
@@ -128,7 +132,10 @@ lbox_call_loadproc(struct lua_State *L)
const char *name;
size_t name_len;
name = lua_tolstring(L, 1, &name_len);
- return box_lua_find(L, name, name + name_len);
+ int count = box_lua_find(L, name, name + name_len);
+ if (count < 0)
+ return luaT_error(L);
+ return count;
}
/*
@@ -307,9 +314,10 @@ execute_lua_call(lua_State *L)
const char *name = ctx->name;
uint32_t name_len = ctx->name_len;
- int oc = 0; /* how many objects are on stack after box_lua_find */
- /* Try to find a function by name in Lua */
- oc = box_lua_find(L, name, name + name_len);
+ /* How many objects are on stack after box_lua_find. */
+ int oc = box_lua_find(L, name, name + name_len);
+ if (oc < 0)
+ return luaT_error(L);
/* Push the rest of args (a tuple). */
int top = lua_gettop(L);
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 4/6] box: export registered functions in box.func folder
2019-06-18 14:06 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
Done.
=========================================================
Needed for #4182, #1260
@TarantoolBot document
Title: Export registered functions to box.func folder
Now all registered with box.schema.func.create functions are
exported in box.func folder.
Each function have :call and :drop method. The :drop method
just a shortcut for box.schema.func.drop interface.
The :call method is similar to net.box connection:call method
and allows to call a registered function directly.
All access checks are performed on each function call.
Example:
function sum(a, b) return a + b end
box.schema.func.create('sum')
box.func.sum
---
- language: LUA
setuid: false
name: sum
id: 2
...
box.func.sum:call({1, 3})
---
- 4
...
box.func.sum:drop()
---
src/box/alter.cc | 2 +
src/box/lua/call.c | 247 ++++++++++++++++++++++++++++++++----
src/box/lua/init.c | 1 +
src/box/lua/misc.cc | 8 +-
src/box/lua/schema.lua | 29 +++++
src/box/port.h | 2 -
src/box/schema.cc | 1 +
src/box/schema.h | 5 +
test/box/function1.c | 33 +++++
test/box/function1.result | 215 +++++++++++++++++++++++++++++++
test/box/function1.test.lua | 69 ++++++++++
test/box/misc.result | 1 +
12 files changed, 581 insertions(+), 32 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 32c4b566a..33f9b0a71 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2583,6 +2583,7 @@ func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
struct func *old_func = (struct func *) trigger->data;
func_cache_delete(old_func->def->fid);
+ trigger_run_xc(&on_alter_func, old_func);
func_delete(old_func);
}
@@ -2615,6 +2616,7 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
def_guard.is_active = false;
func_cache_insert(func);
on_rollback->data = func;
+ trigger_run_xc(&on_alter_func, func);
txn_on_rollback(txn, on_rollback);
} else if (new_tuple == NULL) { /* DELETE */
uint32_t uid;
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 8b7223a7c..8c70c1088 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -33,10 +33,13 @@
#include "box/error.h"
#include "box/func.h"
#include "box/func_def.h"
+#include "box/schema.h"
#include "fiber.h"
+#include "tt_static.h"
#include "lua/utils.h"
#include "lua/msgpack.h"
+#include "lua/trigger.h"
#include "box/port.h"
#include "box/lua/tuple.h"
@@ -284,8 +287,13 @@ port_lua_create(struct port *port, struct lua_State *L)
}
struct execute_lua_ctx {
- const char *name;
- uint32_t name_len;
+ union {
+ struct {
+ const char *name;
+ uint32_t name_len;
+ };
+ struct mpstream *stream;
+ };
struct port *args;
};
@@ -340,58 +348,56 @@ execute_lua_eval(lua_State *L)
static int
encode_lua_call(lua_State *L)
{
- struct port_lua *port = (struct port_lua *) lua_topointer(L, -1);
+ struct execute_lua_ctx *ctx =
+ (struct execute_lua_ctx *) lua_topointer(L, 1);
/*
* Add all elements from Lua stack to the buffer.
*
* TODO: forbid explicit yield from __serialize or __index here
*/
- struct mpstream stream;
- mpstream_init(&stream, port->out, obuf_reserve_cb, obuf_alloc_cb,
- luamp_error, port->L);
-
struct luaL_serializer *cfg = luaL_msgpack_default;
+ struct port_lua *port = (struct port_lua *) ctx->args;
int size = lua_gettop(port->L);
for (int i = 1; i <= size; ++i)
- luamp_encode(port->L, cfg, &stream, i);
+ luamp_encode(port->L, cfg, ctx->stream, i);
port->size = size;
- mpstream_flush(&stream);
+ mpstream_flush(ctx->stream);
return 0;
}
static int
encode_lua_call_16(lua_State *L)
{
- struct port_lua *port = (struct port_lua *) lua_topointer(L, -1);
+ struct execute_lua_ctx *ctx =
+ (struct execute_lua_ctx *) lua_topointer(L, 1);
/*
* Add all elements from Lua stack to the buffer.
*
* TODO: forbid explicit yield from __serialize or __index here
*/
- struct mpstream stream;
- mpstream_init(&stream, port->out, obuf_reserve_cb, obuf_alloc_cb,
- luamp_error, port->L);
-
struct luaL_serializer *cfg = luaL_msgpack_default;
- port->size = luamp_encode_call_16(port->L, cfg, &stream);
- mpstream_flush(&stream);
+ struct port_lua *port = (struct port_lua *) ctx->args;
+ port->size = luamp_encode_call_16(port->L, cfg, ctx->stream);
+ mpstream_flush(ctx->stream);
return 0;
}
static inline int
-port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler)
+port_lua_do_dump(struct port *base, struct mpstream *stream,
+ lua_CFunction handler)
{
- struct port_lua *port = (struct port_lua *)base;
+ struct port_lua *port = (struct port_lua *) base;
assert(port->vtab == &port_lua_vtab);
- /* Use port to pass arguments to encoder quickly. */
- port->out = out;
/*
* Use the same global state, assuming the encoder doesn't
* yield.
*/
+ struct execute_lua_ctx ctx;
+ ctx.args = base;
+ ctx.stream = stream;
struct lua_State *L = tarantool_L;
int top = lua_gettop(L);
- if (lua_cpcall(L, handler, port) != 0) {
+ if (lua_cpcall(L, handler, &ctx) != 0) {
luaT_toerror(port->L);
return -1;
}
@@ -402,13 +408,57 @@ port_lua_do_dump(struct port *base, struct obuf *out, lua_CFunction handler)
static int
port_lua_dump(struct port *base, struct obuf *out)
{
- return port_lua_do_dump(base, out, encode_lua_call);
+ struct port_lua *port = (struct port_lua *) base;
+ struct mpstream stream;
+ mpstream_init(&stream, out, obuf_reserve_cb, obuf_alloc_cb,
+ luamp_error, port->L);
+ return port_lua_do_dump(base, &stream, encode_lua_call);
}
static int
port_lua_dump_16(struct port *base, struct obuf *out)
{
- return port_lua_do_dump(base, out, encode_lua_call_16);
+ struct port_lua *port = (struct port_lua *)base;
+ struct mpstream stream;
+ mpstream_init(&stream, out, obuf_reserve_cb, obuf_alloc_cb,
+ luamp_error, port->L);
+ return port_lua_do_dump(base, &stream, encode_lua_call_16);
+}
+
+static void
+port_lua_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
+{
+ (void) is_flat;
+ assert(is_flat == true);
+ struct port_lua *port = (struct port_lua *) base;
+ uint32_t size = lua_gettop(port->L);
+ lua_xmove(port->L, L, size);
+ port->size = size;
+}
+
+static const char *
+port_lua_get_msgpack(struct port *base, uint32_t *size)
+{
+ struct port_lua *port = (struct port_lua *) base;
+ struct region *region = &fiber()->gc;
+ uint32_t region_svp = region_used(region);
+ struct mpstream stream;
+ mpstream_init(&stream, region, region_reserve_cb, region_alloc_cb,
+ luamp_error, port->L);
+ mpstream_encode_array(&stream, lua_gettop(port->L));
+ int rc = port_lua_do_dump(base, &stream, encode_lua_call);
+ if (rc < 0) {
+ region_truncate(region, region_svp);
+ return NULL;
+ }
+ *size = region_used(region) - region_svp;
+ const char *data = region_join(region, *size);
+ if (data == NULL) {
+ diag_set(OutOfMemory, *size, "region", "data");
+ region_truncate(region, region_svp);
+ return NULL;
+ }
+ return data;
}
static void
@@ -429,9 +479,9 @@ port_lua_dump_plain(struct port *port, uint32_t *size);
static const struct port_vtab port_lua_vtab = {
.dump_msgpack = port_lua_dump,
.dump_msgpack_16 = port_lua_dump_16,
- .dump_lua = NULL,
+ .dump_lua = port_lua_dump_lua,
.dump_plain = port_lua_dump_plain,
- .get_msgpack = NULL,
+ .get_msgpack = port_lua_get_msgpack,
.destroy = port_lua_destroy,
};
@@ -527,10 +577,150 @@ lbox_module_reload(lua_State *L)
return 0;
}
+int
+lbox_func_call(struct lua_State *L)
+{
+ if (lua_gettop(L) < 1 || !lua_isstring(L, 1))
+ return luaL_error(L, "Use func:call(...)");
+
+ size_t name_len;
+ const char *name = lua_tolstring(L, 1, &name_len);
+ struct func *func = func_by_name(name, name_len);
+ if (func == NULL) {
+ diag_set(ClientError, ER_NO_SUCH_FUNCTION,
+ tt_cstr(name, name_len));
+ return luaT_error(L);
+ }
+
+ /*
+ * Prepare a new Lua stack for input arguments
+ * before the function call to pass it into the
+ * pcall-sandboxed tarantool_L handler.
+ */
+ lua_State *args_L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_xmove(L, args_L, lua_gettop(L) - 1);
+ struct port args;
+ port_lua_create(&args, args_L);
+ ((struct port_lua *) &args)->ref = coro_ref;
+
+ struct port ret;
+ if (func_call(func, &args, &ret) != 0) {
+ port_destroy(&args);
+ return luaT_error(L);
+ }
+
+ int top = lua_gettop(L);
+ port_dump_lua(&ret, L, true);
+ int cnt = lua_gettop(L) - top;
+
+ port_destroy(&ret);
+ port_destroy(&args);
+ return cnt;
+}
+
+static void
+lbox_func_new(struct lua_State *L, struct func *func)
+{
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ if (!lua_istable(L, -1)) {
+ lua_pop(L, 1); /* pop nil */
+ lua_newtable(L);
+ lua_setfield(L, -2, "func");
+ lua_getfield(L, -1, "func");
+ }
+ lua_rawgeti(L, -1, func->def->fid);
+ if (lua_isnil(L, -1)) {
+ /*
+ * If the function already exists, modify it,
+ * rather than create a new one -- to not
+ * invalidate Lua variable references to old func
+ * outside the box.schema.func[].
+ */
+ lua_pop(L, 1);
+ lua_newtable(L);
+ lua_rawseti(L, -2, func->def->fid);
+ lua_rawgeti(L, -1, func->def->fid);
+ } else {
+ /* Clear the reference to old func by old name. */
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_settable(L, -4);
+ }
+ int top = lua_gettop(L);
+ lua_pushstring(L, "id");
+ lua_pushnumber(L, func->def->fid);
+ lua_settable(L, top);
+ lua_pushstring(L, "name");
+ lua_pushstring(L, func->def->name);
+ lua_settable(L, top);
+ lua_pushstring(L, "setuid");
+ lua_pushboolean(L, func->def->setuid);
+ lua_settable(L, top);
+ lua_pushstring(L, "language");
+ lua_pushstring(L, func_language_strs[func->def->language]);
+ lua_settable(L, top);
+
+ /* Bless func object. */
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_pushstring(L, "schema");
+ lua_gettable(L, -2);
+ lua_pushstring(L, "func");
+ lua_gettable(L, -2);
+ lua_pushstring(L, "bless");
+ lua_gettable(L, -2);
+
+ lua_pushvalue(L, top);
+ lua_call(L, 1, 0);
+ lua_pop(L, 3);
+
+ lua_setfield(L, -2, func->def->name);
+
+ lua_pop(L, 2);
+}
+
+static void
+lbox_func_delete(struct lua_State *L, struct func *func)
+{
+ uint32_t fid = func->def->fid;
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ assert(!lua_isnil(L, -1));
+ lua_rawgeti(L, -1, fid);
+ if (!lua_isnil(L, -1)) {
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_rawset(L, -4);
+ lua_pop(L, 1); /* pop func */
+ lua_pushnil(L);
+ lua_rawseti(L, -2, fid);
+ } else {
+ lua_pop(L, 1);
+ }
+ lua_pop(L, 2); /* box, func */
+}
+
+static void
+lbox_func_new_or_delete(struct trigger *trigger, void *event)
+{
+ struct lua_State *L = (struct lua_State *) trigger->data;
+ struct func *func = (struct func *)event;
+ if (func != NULL)
+ lbox_func_new(L, func);
+ else
+ lbox_func_delete(L, func);
+}
+
+static struct trigger on_alter_func_in_lua = {
+ RLIST_LINK_INITIALIZER, lbox_func_new_or_delete, NULL, NULL
+};
+
static const struct luaL_Reg boxlib_internal[] = {
{"call_loadproc", lbox_call_loadproc},
{"sql_create_function", lbox_sql_create_function},
{"module_reload", lbox_module_reload},
+ {"func_call", lbox_func_call},
{NULL, NULL}
};
@@ -539,7 +729,12 @@ box_lua_call_init(struct lua_State *L)
{
luaL_register(L, "box.internal", boxlib_internal);
lua_pop(L, 1);
-
+ /*
+ * Register the trigger that will push persistent
+ * Lua functions objects to Lua.
+ */
+ on_alter_func_in_lua.data = L;
+ trigger_add(&on_alter_func, &on_alter_func_in_lua);
#if 0
/* Get CTypeID for `struct port *' */
int rc = luaL_cdef(L, "struct port;");
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index 76b987b4b..7ffed409d 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -39,6 +39,7 @@
#include "box/box.h"
#include "box/txn.h"
+#include "box/func.h"
#include "box/vclock.h"
#include "box/lua/error.h"
diff --git a/src/box/lua/misc.cc b/src/box/lua/misc.cc
index 3058d8dac..3ef47713d 100644
--- a/src/box/lua/misc.cc
+++ b/src/box/lua/misc.cc
@@ -63,14 +63,14 @@ lbox_encode_tuple_on_gc(lua_State *L, int idx, size_t *p_len)
extern "C" void
port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
{
- (void) is_flat;
- assert(is_flat == false);
struct port_tuple *port = port_tuple(base);
- lua_createtable(L, port->size, 0);
+ if (!is_flat)
+ lua_createtable(L, port->size, 0);
struct port_tuple_entry *pe = port->first;
for (int i = 0; pe != NULL; pe = pe->next) {
luaT_pushtuple(L, pe->tuple);
- lua_rawseti(L, -2, ++i);
+ if (!is_flat)
+ lua_rawseti(L, -2, ++i);
}
}
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 3f080eced..9c3ee063c 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2190,7 +2190,36 @@ function box.schema.func.exists(name_or_id)
return tuple ~= nil
end
+-- Helper function to check func:method() usage
+local function check_func_arg(func, method)
+ if type(func) ~= 'table' or func.name == nil then
+ local fmt = 'Use func:%s(...) instead of func.%s(...)'
+ error(string.format(fmt, method, method))
+ end
+end
+
+local func_mt = {}
+
+func_mt.drop = function(func, opts)
+ check_func_arg(func, 'drop')
+ box.schema.func.drop(func.name, opts)
+end
+
+func_mt.call = function(func, args)
+ check_func_arg(func, 'call')
+ args = args or {}
+ if type(args) ~= 'table' then
+ error('Use func:call(table)')
+ end
+ return box.schema.func.call(func.name, unpack(args))
+end
+
+function box.schema.func.bless(func)
+ setmetatable(func, {__index = func_mt})
+end
+
box.schema.func.reload = internal.module_reload
+box.schema.func.call = internal.func_call
box.internal.collation = {}
box.internal.collation.create = function(name, coll_type, locale, opts)
diff --git a/src/box/port.h b/src/box/port.h
index db93f8eea..a7f5d81bd 100644
--- a/src/box/port.h
+++ b/src/box/port.h
@@ -102,8 +102,6 @@ struct port_lua {
struct lua_State *L;
/** Reference to L in tarantool_L. */
int ref;
- /** The argument of port_dump */
- struct obuf *out;
/** Number of entries dumped to the port. */
int size;
};
diff --git a/src/box/schema.cc b/src/box/schema.cc
index d63add535..bf0ed33b7 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -72,6 +72,7 @@ uint32_t space_cache_version = 0;
struct rlist on_schema_init = RLIST_HEAD_INITIALIZER(on_schema_init);
struct rlist on_alter_space = RLIST_HEAD_INITIALIZER(on_alter_space);
struct rlist on_alter_sequence = RLIST_HEAD_INITIALIZER(on_alter_sequence);
+struct rlist on_alter_func = RLIST_HEAD_INITIALIZER(on_alter_func);
/**
* Lock of scheme modification
diff --git a/src/box/schema.h b/src/box/schema.h
index 7e8ac6c11..f0039b29d 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -237,6 +237,11 @@ extern struct rlist on_alter_sequence;
*/
extern struct rlist on_access_denied;
+/**
+ * Triggers fired after committing a change in _func space.
+ */
+extern struct rlist on_alter_func;
+
/**
* Context passed to on_access_denied trigger.
*/
diff --git a/test/box/function1.c b/test/box/function1.c
index 053e4fe91..ee5a422b5 100644
--- a/test/box/function1.c
+++ b/test/box/function1.c
@@ -42,6 +42,39 @@ args(box_function_ctx_t *ctx, const char *args, const char *args_end)
return box_return_tuple(ctx, tuple);
}
+int
+divide(box_function_ctx_t *ctx, const char *args, const char *args_end)
+{
+ uint32_t arg_count = mp_decode_array(&args);
+ if (arg_count < 2)
+ goto error;
+
+ if (mp_typeof(*args) != MP_UINT)
+ goto error;
+ double a = mp_decode_uint(&args);
+ if (mp_typeof(*args) != MP_UINT)
+ goto error;
+ double b = mp_decode_uint(&args);
+ if (b == 0)
+ goto error;
+
+ char tuple_buf[512];
+ char *d = tuple_buf;
+ d = mp_encode_array(d, 1);
+ d = mp_encode_double(d, a / b);
+ assert(d <= tuple_buf + sizeof(tuple_buf));
+
+ box_tuple_format_t *fmt = box_tuple_format_default();
+ box_tuple_t *tuple = box_tuple_new(fmt, tuple_buf, d);
+ if (tuple == NULL)
+ return -1;
+ return box_return_tuple(ctx, tuple);
+error:
+ return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s",
+ "invalid argument");
+}
+
+
/*
* For each UINT key in arguments create or increment counter in
* box.space.test space.
diff --git a/test/box/function1.result b/test/box/function1.result
index cadeb0467..99006926e 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -57,6 +57,25 @@ c:call('function1.args', { 15 })
---
- [[15, 'hello']]
...
+box.func["function1.args"]
+---
+- language: C
+ setuid: false
+ name: function1.args
+ id: 2
+...
+box.func["function1.args"]:call()
+---
+- error: invalid argument count
+...
+box.func["function1.args"]:call({ "xx" })
+---
+- error: first tuple field must be uint
+...
+box.func["function1.args"]:call({ 15 })
+---
+- [15, 'hello']
+...
box.schema.func.drop("function1.args")
---
...
@@ -260,6 +279,10 @@ s:drop()
test_run = require('test_run').new()
---
...
+test_run:cmd("push filter '(.builtin/.*.lua):[0-9]+' to '\\1'")
+---
+- true
+...
identifier = require("identifier")
---
...
@@ -299,3 +322,195 @@ test_run:cmd("setopt delimiter ''");
c:close()
---
...
+-- Test registered functions interface.
+function divide(a, b) return a / b end
+---
+...
+box.schema.func.create("divide")
+---
+...
+func = box.func.divide
+---
+...
+func.call({4, 2})
+---
+- error: 'builtin/box/schema.lua: Use func:call(...) instead of func.call(...)'
+...
+func:call(4, 2)
+---
+- error: 'builtin/box/schema.lua: Use func:call(table)'
+...
+func:call()
+---
+- error: '[string "function divide(a, b) return a / b end "]:1: attempt to perform
+ arithmetic on local ''a'' (a nil value)'
+...
+func:call({})
+---
+- error: '[string "function divide(a, b) return a / b end "]:1: attempt to perform
+ arithmetic on local ''a'' (a nil value)'
+...
+func:call({4})
+---
+- error: '[string "function divide(a, b) return a / b end "]:1: attempt to perform
+ arithmetic on local ''b'' (a nil value)'
+...
+func:call({4, 2})
+---
+- 2
+...
+func:call({4, 2, 1})
+---
+- 2
+...
+func:drop()
+---
+...
+func
+---
+- language: LUA
+ setuid: false
+ name: divide
+ id: 2
+...
+func.drop()
+---
+- error: 'builtin/box/schema.lua: Use func:drop(...) instead of func.drop(...)'
+...
+func:drop()
+---
+- error: Function 'divide' does not exist
+...
+func:call({4, 2})
+---
+- error: Function 'divide' does not exist
+...
+box.internal.func_call('divide', 4, 2)
+---
+- error: Function 'divide' does not exist
+...
+box.schema.func.create("function1.divide", {language = 'C'})
+---
+...
+func = box.func["function1.divide"]
+---
+...
+func:call(4, 2)
+---
+- error: 'builtin/box/schema.lua: Use func:call(table)'
+...
+func:call()
+---
+- error: invalid argument
+...
+func:call({})
+---
+- error: invalid argument
+...
+func:call({4})
+---
+- error: invalid argument
+...
+func:call({4, 2})
+---
+- [2]
+...
+func:call({4, 2, 1})
+---
+- [2]
+...
+func:drop()
+---
+...
+func
+---
+- language: C
+ setuid: false
+ name: function1.divide
+ id: 2
+...
+func:drop()
+---
+- error: Function 'function1.divide' does not exist
+...
+func:call({4, 2})
+---
+- error: Function 'function1.divide' does not exist
+...
+box.internal.func_call('function1.divide', 4, 2)
+---
+- error: Function 'function1.divide' does not exist
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+function minmax(array)
+ local min = 999
+ local max = -1
+ for _, v in pairs(array) do
+ min = math.min(min, v)
+ max = math.max(max, v)
+ end
+ return min, max
+end
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create("minmax")
+---
+...
+func = box.func.minmax
+---
+...
+func:call({{1, 2, 99, 3, -1}})
+---
+- -1
+- 99
+...
+func:drop()
+---
+...
+-- Test access checks for registered functions.
+function secret() return 1 end
+---
+...
+box.schema.func.create("secret")
+---
+...
+box.func.secret:call({})
+---
+- 1
+...
+function secret_leak() return box.func.secret:call() end
+---
+...
+box.schema.func.create('secret_leak')
+---
+...
+box.schema.user.grant('guest', 'execute', 'function', 'secret_leak')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call('secret_leak')
+---
+- error: Execute access to function 'secret' is denied for user 'guest'
+...
+conn:close()
+---
+...
+box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
+---
+...
+box.schema.func.drop('secret_leak')
+---
+...
+box.schema.func.drop('secret')
+---
+...
+test_run:cmd("clear filter")
+---
+- true
+...
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index e983495b6..25966b915 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -21,6 +21,10 @@ box.schema.user.grant('guest', 'execute', 'function', 'function1.args')
c:call('function1.args')
c:call('function1.args', { "xx" })
c:call('function1.args', { 15 })
+box.func["function1.args"]
+box.func["function1.args"]:call()
+box.func["function1.args"]:call({ "xx" })
+box.func["function1.args"]:call({ 15 })
box.schema.func.drop("function1.args")
box.schema.func.create('function1.multi_inc', {language = "C"})
@@ -86,6 +90,7 @@ s:drop()
-- gh-2914: check identifier constraints.
test_run = require('test_run').new()
+test_run:cmd("push filter '(.builtin/.*.lua):[0-9]+' to '\\1'")
identifier = require("identifier")
test_run:cmd("setopt delimiter ';'")
--
@@ -112,3 +117,67 @@ identifier.run_test(
);
test_run:cmd("setopt delimiter ''");
c:close()
+
+-- Test registered functions interface.
+function divide(a, b) return a / b end
+box.schema.func.create("divide")
+func = box.func.divide
+func.call({4, 2})
+func:call(4, 2)
+func:call()
+func:call({})
+func:call({4})
+func:call({4, 2})
+func:call({4, 2, 1})
+func:drop()
+func
+func.drop()
+func:drop()
+func:call({4, 2})
+box.internal.func_call('divide', 4, 2)
+
+box.schema.func.create("function1.divide", {language = 'C'})
+func = box.func["function1.divide"]
+func:call(4, 2)
+func:call()
+func:call({})
+func:call({4})
+func:call({4, 2})
+func:call({4, 2, 1})
+func:drop()
+func
+func:drop()
+func:call({4, 2})
+box.internal.func_call('function1.divide', 4, 2)
+
+test_run:cmd("setopt delimiter ';'")
+function minmax(array)
+ local min = 999
+ local max = -1
+ for _, v in pairs(array) do
+ min = math.min(min, v)
+ max = math.max(max, v)
+ end
+ return min, max
+end
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create("minmax")
+func = box.func.minmax
+func:call({{1, 2, 99, 3, -1}})
+func:drop()
+
+-- Test access checks for registered functions.
+function secret() return 1 end
+box.schema.func.create("secret")
+box.func.secret:call({})
+function secret_leak() return box.func.secret:call() end
+box.schema.func.create('secret_leak')
+box.schema.user.grant('guest', 'execute', 'function', 'secret_leak')
+conn = net.connect(box.cfg.listen)
+conn:call('secret_leak')
+conn:close()
+box.schema.user.revoke('guest', 'execute', 'function', 'secret_leak')
+box.schema.func.drop('secret_leak')
+box.schema.func.drop('secret')
+
+test_run:cmd("clear filter")
diff --git a/test/box/misc.result b/test/box/misc.result
index 0b3902304..c6bdcc540 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -65,6 +65,7 @@ t
- error
- execute
- feedback
+ - func
- index
- info
- internal
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 3/6] box: rework func object as a function frontend
2019-06-18 13:23 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
Done.
========================================================
The function func object used to provide a call method only for
C functions. In scope of this patch it reworked to be a uniform
function call frontend both for C and Lua functions.
Introduced classes func_c and func_lua, that provide own
constructors which produce implementation-specific object with
call and destroy methods.
Needed for #4182, #1260
---
src/box/alter.cc | 1 +
src/box/call.c | 131 ++--------------------------------
src/box/func.c | 170 ++++++++++++++++++++++++++++++++++++++++-----
src/box/func.h | 23 +++---
src/box/func_def.h | 2 +
src/box/lua/call.c | 45 ++++++++++++
src/box/lua/call.h | 5 ++
src/box/session.cc | 11 ++-
src/box/session.h | 9 +++
9 files changed, 240 insertions(+), 157 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index adad957f2..32c4b566a 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2556,6 +2556,7 @@ func_def_new_from_tuple(struct tuple *tuple)
}
memcpy(def->name, name, len);
def->name[len] = 0;
+ def->name_len = len;
if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
def->setuid = tuple_field_u32_xc(tuple, BOX_FUNC_FIELD_SETUID);
else
diff --git a/src/box/call.c b/src/box/call.c
index 573a0d698..bd03e8b96 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -76,90 +76,6 @@ static const struct port_vtab port_msgpack_vtab = {
.destroy = NULL,
};
-/**
- * Find a function by name and check "EXECUTE" permissions.
- *
- * @param name function name
- * @param name_len length of @a name
- * @param[out] funcp function object
- * Sic: *pfunc == NULL means that perhaps the user has a global
- * "EXECUTE" privilege, so no specific grant to a function.
- *
- * @retval -1 on access denied
- * @retval 0 on success
- */
-static inline int
-access_check_func(const char *name, uint32_t name_len, struct func **funcp)
-{
- struct func *func = func_by_name(name, name_len);
- struct credentials *credentials = effective_user();
- /*
- * If the user has universal access, don't bother with checks.
- * No special check for ADMIN user is necessary
- * since ADMIN has universal access.
- */
- if ((credentials->universal_access & (PRIV_X | PRIV_U)) ==
- (PRIV_X | PRIV_U)) {
-
- *funcp = func;
- return 0;
- }
- user_access_t access = PRIV_X | PRIV_U;
- /* Check access for all functions. */
- access &= ~entity_access_get(SC_FUNCTION)[credentials->auth_token].effective;
- user_access_t func_access = access & ~credentials->universal_access;
- if (func == NULL ||
- /* Check for missing Usage access, ignore owner rights. */
- func_access & PRIV_U ||
- /* Check for missing specific access, respect owner rights. */
- (func->def->uid != credentials->uid &&
- func_access & ~func->access[credentials->auth_token].effective)) {
-
- /* Access violation, report error. */
- struct user *user = user_find(credentials->uid);
- if (user != NULL) {
- if (!(access & credentials->universal_access)) {
- diag_set(AccessDeniedError,
- priv_name(PRIV_U),
- schema_object_name(SC_UNIVERSE), "",
- user->def->name);
- } else {
- diag_set(AccessDeniedError,
- priv_name(PRIV_X),
- schema_object_name(SC_FUNCTION),
- tt_cstr(name, name_len),
- user->def->name);
- }
- }
- return -1;
- }
-
- *funcp = func;
- return 0;
-}
-
-static int
-box_c_call(struct func *func, struct port *args, struct port *ret)
-{
- assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
-
- /* Clear all previous errors */
- diag_clear(&fiber()->diag);
- assert(!in_txn()); /* transaction is not started */
-
- /* Call function from the shared library */
- int rc = func_call(func, args, ret);
- func = NULL; /* May be deleted by DDL */
- if (rc != 0) {
- if (diag_last_error(&fiber()->diag) == NULL) {
- /* Stored procedure forget to set diag */
- diag_set(ClientError, ER_PROC_C, "unknown error");
- }
- return -1;
- }
- return 0;
-}
-
int
box_module_reload(const char *name)
{
@@ -191,53 +107,20 @@ box_process_call(struct call_request *request, struct port *port)
const char *name = request->name;
assert(name != NULL);
uint32_t name_len = mp_decode_strl(&name);
-
- struct func *func = NULL;
- /**
- * Sic: func == NULL means that perhaps the user has a global
- * "EXECUTE" privilege, so no specific grant to a function.
- */
- if (access_check_func(name, name_len, &func) != 0)
- return -1; /* permission denied */
-
- /**
- * Change the current user id if the function is
- * a set-definer-uid one. If the function is not
- * defined, it's obviously not a setuid one.
- */
- struct credentials *orig_credentials = NULL;
- if (func && func->def->setuid) {
- orig_credentials = effective_user();
- /* Remember and change the current user id. */
- if (func->owner_credentials.auth_token >= BOX_USER_MAX) {
- /*
- * Fill the cache upon first access, since
- * when func is created, no user may
- * be around to fill it (recovery of
- * system spaces from a snapshot).
- */
- struct user *owner = user_find(func->def->uid);
- if (owner == NULL)
- return -1;
- credentials_init(&func->owner_credentials,
- owner->auth_token,
- owner->def->uid);
- }
- fiber_set_user(fiber(), &func->owner_credentials);
- }
+ /* Transaction is not started. */
+ assert(!in_txn());
int rc;
struct port args;
port_msgpack_create(&args, request->args,
request->args_end - request->args);
- if (func && func->def->language == FUNC_LANGUAGE_C) {
- rc = box_c_call(func, &args, port);
- } else {
+ struct func *func = func_by_name(name, name_len);
+ if (func != NULL) {
+ rc = func_call(func, &args, port);
+ } else if ((rc = access_check_universe_object(PRIV_X | PRIV_U,
+ SC_FUNCTION, tt_cstr(name, name_len))) == 0) {
rc = box_lua_call(name, name_len, &args, port);
}
- /* Restore the original user */
- if (orig_credentials)
- fiber_set_user(fiber(), orig_credentials);
if (rc != 0) {
txn_rollback();
diff --git a/src/box/func.c b/src/box/func.c
index 1c37f6523..c57027809 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -33,9 +33,12 @@
#include "trivia/config.h"
#include "assoc.h"
#include "lua/utils.h"
+#include "lua/call.h"
#include "error.h"
#include "diag.h"
#include "port.h"
+#include "schema.h"
+#include "session.h"
#include <dlfcn.h>
/**
@@ -50,6 +53,24 @@ struct func_name {
const char *package_end;
};
+struct func_c {
+ /** Function object base class. */
+ struct func base;
+ /**
+ * Anchor for module membership.
+ */
+ struct rlist item;
+ /**
+ * For C functions, the body of the function.
+ */
+ box_function_f func;
+ /**
+ * Each stored function keeps a handle to the
+ * dynamic library for the C callback.
+ */
+ struct module *module;
+};
+
/***
* Split function name to symbol and package names.
* For example, str = foo.bar.baz => sym = baz, package = foo.bar
@@ -314,10 +335,10 @@ module_reload(const char *package, const char *package_end, struct module **modu
if (new_module == NULL)
return -1;
- struct func *func, *tmp_func;
+ struct func_c *func, *tmp_func;
rlist_foreach_entry_safe(func, &old_module->funcs, item, tmp_func) {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
func->func = module_sym(new_module, name.sym);
if (func->func == NULL)
goto restore;
@@ -338,7 +359,7 @@ restore:
*/
do {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
func->func = module_sym(old_module, name.sym);
if (func->func == NULL) {
/*
@@ -351,20 +372,27 @@ restore:
func->module = old_module;
rlist_move(&old_module->funcs, &func->item);
} while (func != rlist_first_entry(&old_module->funcs,
- struct func, item));
+ struct func_c, item));
assert(rlist_empty(&new_module->funcs));
module_delete(new_module);
return -1;
}
+static struct func *
+func_c_new(struct func_def *def);
+
struct func *
func_new(struct func_def *def)
{
- struct func *func = (struct func *) malloc(sizeof(struct func));
- if (func == NULL) {
- diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
- return NULL;
+ struct func *func;
+ if (def->language == FUNC_LANGUAGE_C) {
+ func = func_c_new(def);
+ } else {
+ assert(def->language == FUNC_LANGUAGE_LUA);
+ func = func_lua_new(def);
}
+ if (func == NULL)
+ return NULL;
func->def = def;
/** Nobody has access to the function but the owner. */
memset(func->access, 0, sizeof(func->access));
@@ -380,19 +408,35 @@ func_new(struct func_def *def)
* checks (see user_has_data()).
*/
func->owner_credentials.auth_token = BOX_USER_MAX; /* invalid value */
+ return func;
+}
+
+static struct func_vtab func_c_vtab;
+
+static struct func *
+func_c_new(struct func_def *def)
+{
+ (void) def;
+ assert(def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) malloc(sizeof(struct func_c));
+ if (func == NULL) {
+ diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
+ return NULL;
+ }
+ func->base.vtab = &func_c_vtab;
func->func = NULL;
func->module = NULL;
- return func;
+ return &func->base;
}
static void
-func_unload(struct func *func)
+func_c_unload(struct func_c *func)
{
if (func->module) {
rlist_del(&func->item);
if (rlist_empty(&func->module->funcs)) {
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
module_cache_del(name.package, name.package_end);
}
module_gc(func->module);
@@ -401,17 +445,27 @@ func_unload(struct func *func)
func->func = NULL;
}
+static void
+func_c_destroy(struct func *base)
+{
+ assert(base->vtab == &func_c_vtab);
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) base;
+ func_c_unload(func);
+ free(func);
+}
+
/**
* Resolve func->func (find the respective DLL and fetch the
* symbol from it).
*/
static int
-func_load(struct func *func)
+func_c_load(struct func_c *func)
{
assert(func->func == NULL);
struct func_name name;
- func_split_name(func->def->name, &name);
+ func_split_name(func->base.def->name, &name);
struct module *module = module_cache_find(name.package,
name.package_end);
@@ -435,11 +489,13 @@ func_load(struct func *func)
}
int
-func_call(struct func *func, struct port *args, struct port *ret)
+func_c_call(struct func *base, struct port *args, struct port *ret)
{
- assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
+ assert(base->vtab == &func_c_vtab);
+ assert(base != NULL && base->def->language == FUNC_LANGUAGE_C);
+ struct func_c *func = (struct func_c *) base;
if (func->func == NULL) {
- if (func_load(func) != 0)
+ if (func_c_load(func) != 0)
return -1;
}
@@ -462,16 +518,94 @@ func_call(struct func *func, struct port *args, struct port *ret)
module_gc(module);
region_truncate(region, region_svp);
if (rc != 0) {
+ if (diag_last_error(&fiber()->diag) == NULL) {
+ /* Stored procedure forget to set diag */
+ diag_set(ClientError, ER_PROC_C, "unknown error");
+ }
port_destroy(ret);
return -1;
}
return rc;
}
+static struct func_vtab func_c_vtab = {
+ .call = func_c_call,
+ .destroy = func_c_destroy,
+};
+
void
func_delete(struct func *func)
{
- func_unload(func);
free(func->def);
- free(func);
+ func->vtab->destroy(func);
+}
+
+/** Check "EXECUTE" permissions for a given function. */
+static int
+func_access_check(struct func *func)
+{
+ struct credentials *credentials = effective_user();
+ /*
+ * If the user has universal access, don't bother with
+ * checks. No special check for ADMIN user is necessary
+ * since ADMIN has universal access.
+ */
+ if ((credentials->universal_access & (PRIV_X | PRIV_U)) ==
+ (PRIV_X | PRIV_U))
+ return 0;
+ user_access_t access = PRIV_X | PRIV_U;
+ /* Check access for all functions. */
+ access &= ~entity_access_get(SC_FUNCTION)[
+ credentials->auth_token].effective;
+ user_access_t func_access = access & ~credentials->universal_access;
+ if ((func_access & PRIV_U) != 0 ||
+ (func->def->uid != credentials->uid &&
+ func_access & ~func->access[credentials->auth_token].effective)) {
+ /* Access violation, report error. */
+ struct user *user = user_find(credentials->uid);
+ if (user != NULL) {
+ diag_set(AccessDeniedError, priv_name(PRIV_X),
+ schema_object_name(SC_FUNCTION),
+ func->def->name, user->def->name);
+ }
+ return -1;
+ }
+ return 0;
+}
+
+int
+func_call(struct func *base, struct port *args, struct port *ret)
+{
+ if (func_access_check(base) != 0)
+ return -1;
+ /**
+ * Change the current user id if the function is
+ * a set-definer-uid one. If the function is not
+ * defined, it's obviously not a setuid one.
+ */
+ struct credentials *orig_credentials = NULL;
+ if (base->def->setuid) {
+ orig_credentials = effective_user();
+ /* Remember and change the current user id. */
+ if (base->owner_credentials.auth_token >= BOX_USER_MAX) {
+ /*
+ * Fill the cache upon first access, since
+ * when func is created, no user may
+ * be around to fill it (recovery of
+ * system spaces from a snapshot).
+ */
+ struct user *owner = user_find(base->def->uid);
+ if (owner == NULL)
+ return -1;
+ credentials_init(&base->owner_credentials,
+ owner->auth_token,
+ owner->def->uid);
+ }
+ fiber_set_user(fiber(), &base->owner_credentials);
+ }
+ int rc = base->vtab->call(base, args, ret);
+ /* Restore the original user */
+ if (orig_credentials)
+ fiber_set_user(fiber(), orig_credentials);
+ return rc;
}
diff --git a/src/box/func.h b/src/box/func.h
index f6494d064..e121e6bb9 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -59,24 +59,21 @@ struct module {
bool is_unloading;
};
+/** Virtual method table for func object. */
+struct func_vtab {
+ /** Call function with given arguments. */
+ int (*call)(struct func *func, struct port *args, struct port *ret);
+ /** Release implementation-specific function context. */
+ void (*destroy)(struct func *func);
+};
+
/**
* Stored function.
*/
struct func {
struct func_def *def;
- /**
- * Anchor for module membership.
- */
- struct rlist item;
- /**
- * For C functions, the body of the function.
- */
- box_function_f func;
- /**
- * Each stored function keeps a handle to the
- * dynamic library for the C callback.
- */
- struct module *module;
+ /** Virtual method table. */
+ const struct func_vtab *vtab;
/**
* Authentication id of the owner of the function,
* used for set-user-id functions.
diff --git a/src/box/func_def.h b/src/box/func_def.h
index ef33cbbaa..866d425a1 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -67,6 +67,8 @@ struct func_def {
* The language of the stored function.
*/
enum func_language language;
+ /** The length of the function name. */
+ uint32_t name_len;
/** Function name. */
char name[0];
};
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 7cac90f3b..8b7223a7c 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -31,6 +31,8 @@
#include "box/lua/call.h"
#include "box/call.h"
#include "box/error.h"
+#include "box/func.h"
+#include "box/func_def.h"
#include "fiber.h"
#include "lua/utils.h"
@@ -473,6 +475,49 @@ box_lua_eval(const char *expr, uint32_t expr_len,
return box_process_lua(execute_lua_eval, &ctx, ret);
}
+struct func_lua {
+ /** Function object base class. */
+ struct func base;
+};
+
+static struct func_vtab func_lua_vtab;
+
+struct func *
+func_lua_new(struct func_def *def)
+{
+ (void) def;
+ assert(def->language == FUNC_LANGUAGE_LUA);
+ struct func_lua *func =
+ (struct func_lua *) malloc(sizeof(struct func_lua));
+ if (func == NULL) {
+ diag_set(OutOfMemory, sizeof(*func), "malloc", "func");
+ return NULL;
+ }
+ func->base.vtab = &func_lua_vtab;
+ return &func->base;
+}
+
+static void
+func_lua_destroy(struct func *func)
+{
+ assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA);
+ assert(func->vtab == &func_lua_vtab);
+ free(func);
+}
+
+static inline int
+func_lua_call(struct func *func, struct port *args, struct port *ret)
+{
+ assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA);
+ assert(func->vtab == &func_lua_vtab);
+ return box_lua_call(func->def->name, func->def->name_len, args, ret);
+}
+
+static struct func_vtab func_lua_vtab = {
+ .call = func_lua_call,
+ .destroy = func_lua_destroy,
+};
+
static int
lbox_module_reload(lua_State *L)
{
diff --git a/src/box/lua/call.h b/src/box/lua/call.h
index d03bcd0f8..83aa43949 100644
--- a/src/box/lua/call.h
+++ b/src/box/lua/call.h
@@ -44,6 +44,7 @@ box_lua_call_init(struct lua_State *L);
struct port;
struct call_request;
+struct func_def;
/**
* Invoke a Lua stored procedure from the binary protocol
@@ -57,6 +58,10 @@ int
box_lua_eval(const char *expr, uint32_t expr_len,
struct port *args, struct port *ret);
+/** Construct a Lua function object. */
+struct func *
+func_lua_new(struct func_def *def);
+
#if defined(__cplusplus)
} /* extern "C" */
#endif /* defined(__cplusplus) */
diff --git a/src/box/session.cc b/src/box/session.cc
index 4bb13d031..3a7c02401 100644
--- a/src/box/session.cc
+++ b/src/box/session.cc
@@ -297,7 +297,8 @@ access_check_session(struct user *user)
}
int
-access_check_universe(user_access_t access)
+access_check_universe_object(user_access_t access,
+ enum schema_object_type object_type, const char *object_name)
{
struct credentials *credentials = effective_user();
access |= PRIV_U;
@@ -313,7 +314,7 @@ access_check_universe(user_access_t access)
if (user != NULL) {
diag_set(AccessDeniedError,
priv_name(denied_access),
- schema_object_name(SC_UNIVERSE), "",
+ schema_object_name(object_type), object_name,
user->def->name);
} else {
/*
@@ -328,6 +329,12 @@ access_check_universe(user_access_t access)
return 0;
}
+int
+access_check_universe(user_access_t access)
+{
+ return access_check_universe_object(access, SC_UNIVERSE, "");
+}
+
int
generic_session_push(struct session *session, uint64_t sync, struct port *port)
{
diff --git a/src/box/session.h b/src/box/session.h
index 3a7397146..54a02536e 100644
--- a/src/box/session.h
+++ b/src/box/session.h
@@ -312,6 +312,15 @@ access_check_session(struct user *user);
int
access_check_universe(user_access_t access);
+/**
+ * Same as access_check_universe(), but in case the current user
+ * doesn't have universal access, set AccessDeniedError for the
+ * given object type and name.
+ */
+int
+access_check_universe_object(user_access_t access,
+ enum schema_object_type object_type, const char *object_name);
+
static inline int
session_push(struct session *session, uint64_t sync, struct port *port)
{
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-18 12:12 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
2019-06-19 16:11 ` Vladimir Davydov
0 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
>> module_gc(module);
>> + if (rc != 0) {
>> + if (diag_last_error(&fiber()->diag) == NULL) {
>> + /* Stored procedure forget to set diag */
>> + diag_set(ClientError, ER_PROC_C, "unknown error");
>> + }
>> + port_destroy(out_port);
>> + return -1;
>> + }
>
>Shouldn't you call region_truncate() here to clean up after the function
>and possibly port_get_msgpack()?
No, the func_c_call implementation must clean region by it's own.
======================================================
Re-factor box_lua_call and box_lua_eval so that they don't take
call_request. This approach is more scalable: in case of a
functional index, the user expects to see a tuple with field
names so we should be able to pass not only raw msgpack, but
also a tuple to a Lua call so we need an universal way to pass
arguments to _call methods.
To pass a tuple msgpack introduced a new port_msgpack: the port
class with dump_lua method.
Needed for #4182, #1260
---
src/box/call.c | 57 +++++++++++++++++++++++++++-------
src/box/call.h | 4 ---
src/box/execute.c | 1 +
src/box/func.c | 23 ++++++++++++--
src/box/func.h | 8 +++--
src/box/func_def.h | 7 +++++
src/box/lua/call.c | 71 +++++++++++++++++++++++++------------------
src/box/lua/call.h | 8 +++--
src/box/lua/execute.c | 8 +++--
src/box/lua/execute.h | 4 ++-
src/box/lua/misc.cc | 19 ++++++++++--
src/box/port.c | 3 +-
src/box/port.h | 15 +++++++++
src/lib/core/port.h | 30 +++++++++++++++---
14 files changed, 194 insertions(+), 64 deletions(-)
diff --git a/src/box/call.c b/src/box/call.c
index 56da53fb3..573a0d698 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -43,6 +43,39 @@
#include "small/obuf.h"
#include "tt_static.h"
+static const struct port_vtab port_msgpack_vtab;
+
+void
+port_msgpack_create(struct port *base, const char *data, uint32_t data_sz)
+{
+ struct port_msgpack *port_msgpack = (struct port_msgpack *) base;
+ memset(port_msgpack, 0, sizeof(*port_msgpack));
+ port_msgpack->vtab = &port_msgpack_vtab;
+ port_msgpack->data = data;
+ port_msgpack->data_sz = data_sz;
+}
+
+static const char *
+port_msgpack_get_msgpack(struct port *base, uint32_t *size)
+{
+ struct port_msgpack *port = (struct port_msgpack *) base;
+ assert(port->vtab == &port_msgpack_vtab);
+ *size = port->data_sz;
+ return port->data;
+}
+
+extern void
+port_msgpack_dump_lua(struct port *base, struct lua_State *L, bool is_flat);
+
+static const struct port_vtab port_msgpack_vtab = {
+ .dump_msgpack = NULL,
+ .dump_msgpack_16 = NULL,
+ .dump_lua = port_msgpack_dump_lua,
+ .dump_plain = NULL,
+ .get_msgpack = port_msgpack_get_msgpack,
+ .destroy = NULL,
+};
+
/**
* Find a function by name and check "EXECUTE" permissions.
*
@@ -106,27 +139,22 @@ access_check_func(const char *name, uint32_t name_len, struct func **funcp)
}
static int
-box_c_call(struct func *func, struct call_request *request, struct port *port)
+box_c_call(struct func *func, struct port *args, struct port *ret)
{
assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
- /* Create a call context */
- port_tuple_create(port);
- box_function_ctx_t ctx = { port };
-
/* Clear all previous errors */
diag_clear(&fiber()->diag);
assert(!in_txn()); /* transaction is not started */
/* Call function from the shared library */
- int rc = func_call(func, &ctx, request->args, request->args_end);
+ int rc = func_call(func, args, ret);
func = NULL; /* May be deleted by DDL */
if (rc != 0) {
if (diag_last_error(&fiber()->diag) == NULL) {
/* Stored procedure forget to set diag */
diag_set(ClientError, ER_PROC_C, "unknown error");
}
- port_destroy(port);
return -1;
}
return 0;
@@ -199,10 +227,13 @@ box_process_call(struct call_request *request, struct port *port)
}
int rc;
+ struct port args;
+ port_msgpack_create(&args, request->args,
+ request->args_end - request->args);
if (func && func->def->language == FUNC_LANGUAGE_C) {
- rc = box_c_call(func, request, port);
+ rc = box_c_call(func, &args, port);
} else {
- rc = box_lua_call(request, port);
+ rc = box_lua_call(name, name_len, &args, port);
}
/* Restore the original user */
if (orig_credentials)
@@ -229,7 +260,12 @@ box_process_eval(struct call_request *request, struct port *port)
/* Check permissions */
if (access_check_universe(PRIV_X) != 0)
return -1;
- if (box_lua_eval(request, port) != 0) {
+ struct port args;
+ port_msgpack_create(&args, request->args,
+ request->args_end - request->args);
+ const char *expr = request->expr;
+ uint32_t expr_len = mp_decode_strl(&expr);
+ if (box_lua_eval(expr, expr_len, &args, port) != 0) {
txn_rollback();
return -1;
}
@@ -239,6 +275,5 @@ box_process_eval(struct call_request *request, struct port *port)
txn_rollback();
return -1;
}
-
return 0;
}
diff --git a/src/box/call.h b/src/box/call.h
index 1b54551be..45580bc9d 100644
--- a/src/box/call.h
+++ b/src/box/call.h
@@ -38,10 +38,6 @@ extern "C" {
struct port;
struct call_request;
-struct box_function_ctx {
- struct port *port;
-};
-
/**
* Reload loadable module by name.
*
diff --git a/src/box/execute.c b/src/box/execute.c
index e81cc32aa..a91c939fc 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -108,6 +108,7 @@ const struct port_vtab port_sql_vtab = {
/* .dump_msgpack_16 = */ NULL,
/* .dump_lua = */ port_sql_dump_lua,
/* .dump_plain = */ NULL,
+ /* .get_msgpack = */ NULL,
/* .destroy = */ port_sql_destroy,
};
diff --git a/src/box/func.c b/src/box/func.c
index d1b8879ad..1c37f6523 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -29,11 +29,13 @@
* SUCH DAMAGE.
*/
#include "func.h"
+#include "fiber.h"
#include "trivia/config.h"
#include "assoc.h"
#include "lua/utils.h"
#include "error.h"
#include "diag.h"
+#include "port.h"
#include <dlfcn.h>
/**
@@ -433,21 +435,36 @@ func_load(struct func *func)
}
int
-func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
- const char *args_end)
+func_call(struct func *func, struct port *args, struct port *ret)
{
+ assert(func != NULL && func->def->language == FUNC_LANGUAGE_C);
if (func->func == NULL) {
if (func_load(func) != 0)
return -1;
}
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+ uint32_t data_sz;
+ const char *data = port_get_msgpack(args, &data_sz);
+ if (data == NULL)
+ return -1;
+
+ port_tuple_create(ret);
+ box_function_ctx_t ctx = { ret };
+
/* Module can be changed after function reload. */
struct module *module = func->module;
assert(module != NULL);
++module->calls;
- int rc = func->func(ctx, args, args_end);
+ int rc = func->func(&ctx, data, data + data_sz);
--module->calls;
module_gc(module);
+ region_truncate(region, region_svp);
+ if (rc != 0) {
+ port_destroy(ret);
+ return -1;
+ }
return rc;
}
diff --git a/src/box/func.h b/src/box/func.h
index fa4d738ab..f6494d064 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -42,6 +42,9 @@
extern "C" {
#endif /* defined(__cplusplus) */
+struct func;
+struct lua_State;
+
/**
* Dynamic shared module.
*/
@@ -104,11 +107,10 @@ void
func_delete(struct func *func);
/**
- * Call stored C function using @a args.
+ * Call function with arguments represented with given args.
*/
int
-func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
- const char *args_end);
+func_call(struct func *func, struct port *args, struct port *ret);
/**
* Reload dynamically loadable module.
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 4c9738aea..ef33cbbaa 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -90,6 +90,13 @@ func_def_cmp(struct func_def *def1, struct func_def *def2);
/**
* API of C stored function.
*/
+
+struct port;
+
+struct box_function_ctx {
+ struct port *port;
+};
+
typedef struct box_function_ctx box_function_ctx_t;
typedef int (*box_function_f)(box_function_ctx_t *ctx,
const char *args, const char *args_end);
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 04020ef6f..7cac90f3b 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -36,7 +36,6 @@
#include "lua/utils.h"
#include "lua/msgpack.h"
-#include "box/xrow.h"
#include "box/port.h"
#include "box/lua/tuple.h"
#include "small/obuf.h"
@@ -282,28 +281,31 @@ port_lua_create(struct port *port, struct lua_State *L)
port_lua->ref = -1;
}
+struct execute_lua_ctx {
+ const char *name;
+ uint32_t name_len;
+ struct port *args;
+};
+
static int
execute_lua_call(lua_State *L)
{
- struct call_request *request = (struct call_request *)
- lua_topointer(L, 1);
+ struct execute_lua_ctx *ctx =
+ (struct execute_lua_ctx *) lua_topointer(L, 1);
lua_settop(L, 0); /* clear the stack to simplify the logic below */
- const char *name = request->name;
- uint32_t name_len = mp_decode_strl(&name);
+ const char *name = ctx->name;
+ uint32_t name_len = ctx->name_len;
int oc = 0; /* how many objects are on stack after box_lua_find */
/* Try to find a function by name in Lua */
oc = box_lua_find(L, name, name + name_len);
/* Push the rest of args (a tuple). */
- const char *args = request->args;
-
- uint32_t arg_count = mp_decode_array(&args);
- luaL_checkstack(L, arg_count, "call: out of stack");
+ int top = lua_gettop(L);
+ port_dump_lua(ctx->args, L, true);
+ int arg_count = lua_gettop(L) - top;
- for (uint32_t i = 0; i < arg_count; i++)
- luamp_decode(L, luaL_msgpack_default, &args);
lua_call(L, arg_count + oc - 1, LUA_MULTRET);
return lua_gettop(L);
}
@@ -311,24 +313,22 @@ execute_lua_call(lua_State *L)
static int
execute_lua_eval(lua_State *L)
{
- struct call_request *request = (struct call_request *)
- lua_topointer(L, 1);
+ struct execute_lua_ctx *ctx =
+ (struct execute_lua_ctx *) lua_topointer(L, 1);
lua_settop(L, 0); /* clear the stack to simplify the logic below */
/* Compile expression */
- const char *expr = request->expr;
- uint32_t expr_len = mp_decode_strl(&expr);
+ const char *expr = ctx->name;
+ uint32_t expr_len = ctx->name_len;
if (luaL_loadbuffer(L, expr, expr_len, "=eval")) {
diag_set(LuajitError, lua_tostring(L, -1));
luaT_error(L);
}
/* Unpack arguments */
- const char *args = request->args;
- uint32_t arg_count = mp_decode_array(&args);
- luaL_checkstack(L, arg_count, "eval: out of stack");
- for (uint32_t i = 0; i < arg_count; i++)
- luamp_decode(L, luaL_msgpack_default, &args);
+ int top = lua_gettop(L);
+ port_dump_lua(ctx->args, L, true);
+ int arg_count = lua_gettop(L) - top;
/* Call compiled code */
lua_call(L, arg_count, LUA_MULTRET);
@@ -429,37 +429,48 @@ static const struct port_vtab port_lua_vtab = {
.dump_msgpack_16 = port_lua_dump_16,
.dump_lua = NULL,
.dump_plain = port_lua_dump_plain,
+ .get_msgpack = NULL,
.destroy = port_lua_destroy,
};
static inline int
-box_process_lua(struct call_request *request, struct port *base,
- lua_CFunction handler)
+box_process_lua(lua_CFunction handler, struct execute_lua_ctx *ctx,
+ struct port *ret)
{
lua_State *L = lua_newthread(tarantool_L);
int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
- port_lua_create(base, L);
- ((struct port_lua *) base)->ref = coro_ref;
+ port_lua_create(ret, L);
+ ((struct port_lua *) ret)->ref = coro_ref;
lua_pushcfunction(L, handler);
- lua_pushlightuserdata(L, request);
+ lua_pushlightuserdata(L, ctx);
if (luaT_call(L, 1, LUA_MULTRET) != 0) {
- port_lua_destroy(base);
+ port_lua_destroy(ret);
return -1;
}
return 0;
}
int
-box_lua_call(struct call_request *request, struct port *port)
+box_lua_call(const char *name, uint32_t name_len,
+ struct port *args, struct port *ret)
{
- return box_process_lua(request, port, execute_lua_call);
+ struct execute_lua_ctx ctx;
+ ctx.name = name;
+ ctx.name_len = name_len;
+ ctx.args = args;
+ return box_process_lua(execute_lua_call, &ctx, ret);
}
int
-box_lua_eval(struct call_request *request, struct port *port)
+box_lua_eval(const char *expr, uint32_t expr_len,
+ struct port *args, struct port *ret)
{
- return box_process_lua(request, port, execute_lua_eval);
+ struct execute_lua_ctx ctx;
+ ctx.name = expr;
+ ctx.name_len = expr_len;
+ ctx.args = args;
+ return box_process_lua(execute_lua_eval, &ctx, ret);
}
static int
diff --git a/src/box/lua/call.h b/src/box/lua/call.h
index 0542123da..d03bcd0f8 100644
--- a/src/box/lua/call.h
+++ b/src/box/lua/call.h
@@ -31,6 +31,8 @@
* SUCH DAMAGE.
*/
+#include <stdint.h>
+
#if defined(__cplusplus)
extern "C" {
#endif /* defined(__cplusplus) */
@@ -48,10 +50,12 @@ struct call_request;
* (implementation of 'CALL' command code).
*/
int
-box_lua_call(struct call_request *request, struct port *port);
+box_lua_call(const char *name, uint32_t name_len,
+ struct port *args, struct port *ret);
int
-box_lua_eval(struct call_request *request, struct port *port);
+box_lua_eval(const char *expr, uint32_t expr_len,
+ struct port *args, struct port *ret);
#if defined(__cplusplus)
} /* extern "C" */
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index 239aba47b..10fa35de2 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -39,8 +39,10 @@ lua_sql_get_metadata(struct sql_stmt *stmt, struct lua_State *L,
}
void
-port_sql_dump_lua(struct port *port, struct lua_State *L)
+port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
{
+ (void) is_flat;
+ assert(is_flat == false);
assert(port->vtab == &port_sql_vtab);
struct sql *db = sql_get();
struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
@@ -49,7 +51,7 @@ port_sql_dump_lua(struct port *port, struct lua_State *L)
lua_createtable(L, 0, 2);
lua_sql_get_metadata(stmt, L, column_count);
lua_setfield(L, -2, "metadata");
- port_tuple_vtab.dump_lua(port, L);
+ port_tuple_vtab.dump_lua(port, L, false);
lua_setfield(L, -2, "rows");
} else {
assert(((struct port_tuple *)port)->size == 0);
@@ -262,7 +264,7 @@ lbox_execute(struct lua_State *L)
if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
&fiber()->gc) != 0)
return luaT_error(L);
- port_dump_lua(&port, L);
+ port_dump_lua(&port, L, false);
port_destroy(&port);
return 1;
}
diff --git a/src/box/lua/execute.h b/src/box/lua/execute.h
index bc4df19f5..23e193fa4 100644
--- a/src/box/lua/execute.h
+++ b/src/box/lua/execute.h
@@ -30,6 +30,8 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
+#include <stdbool.h>
+
struct port;
struct sql_bind;
struct lua_State;
@@ -42,7 +44,7 @@ struct lua_State;
* @param L Lua stack.
*/
void
-port_sql_dump_lua(struct port *port, struct lua_State *L);
+port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat);
/**
* Parse Lua table of SQL parameters.
diff --git a/src/box/lua/misc.cc b/src/box/lua/misc.cc
index 8de7401fa..3058d8dac 100644
--- a/src/box/lua/misc.cc
+++ b/src/box/lua/misc.cc
@@ -61,8 +61,10 @@ lbox_encode_tuple_on_gc(lua_State *L, int idx, size_t *p_len)
* but implemented here to eliminate port.c dependency on Lua.
*/
extern "C" void
-port_tuple_dump_lua(struct port *base, struct lua_State *L)
+port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
{
+ (void) is_flat;
+ assert(is_flat == false);
struct port_tuple *port = port_tuple(base);
lua_createtable(L, port->size, 0);
struct port_tuple_entry *pe = port->first;
@@ -72,6 +74,19 @@ port_tuple_dump_lua(struct port *base, struct lua_State *L)
}
}
+extern "C" void
+port_msgpack_dump_lua(struct port *base, struct lua_State *L, bool is_flat)
+{
+ (void) is_flat;
+ assert(is_flat == true);
+ struct port_msgpack *port = (struct port_msgpack *) base;
+
+ const char *args = port->data;
+ uint32_t arg_count = mp_decode_array(&args);
+ for (uint32_t i = 0; i < arg_count; i++)
+ luamp_decode(L, luaL_msgpack_default, &args);
+}
+
/* }}} */
/** {{{ Lua/C implementation of index:select(): used only by Vinyl **/
@@ -109,7 +124,7 @@ lbox_select(lua_State *L)
* table always crashed the first (can't be fixed with pcall).
* https://github.com/tarantool/tarantool/issues/1182
*/
- port_dump_lua(&port, L);
+ port_dump_lua(&port, L, false);
port_destroy(&port);
return 1; /* lua table with tuples */
}
diff --git a/src/box/port.c b/src/box/port.c
index 99046449a..7f552bcfe 100644
--- a/src/box/port.c
+++ b/src/box/port.c
@@ -125,7 +125,7 @@ port_tuple_dump_msgpack(struct port *base, struct obuf *out)
}
extern void
-port_tuple_dump_lua(struct port *base, struct lua_State *L);
+port_tuple_dump_lua(struct port *base, struct lua_State *L, bool is_flat);
void
port_init(void)
@@ -145,5 +145,6 @@ const struct port_vtab port_tuple_vtab = {
.dump_msgpack_16 = port_tuple_dump_msgpack_16,
.dump_lua = port_tuple_dump_lua,
.dump_plain = NULL,
+ .get_msgpack = NULL,
.destroy = port_tuple_destroy,
};
diff --git a/src/box/port.h b/src/box/port.h
index f18803660..db93f8eea 100644
--- a/src/box/port.h
+++ b/src/box/port.h
@@ -32,6 +32,7 @@
*/
#include "trivia/util.h"
#include <port.h>
+#include <stdbool.h>
#if defined(__cplusplus)
extern "C" {
@@ -80,6 +81,20 @@ port_tuple_create(struct port *port);
int
port_tuple_add(struct port *port, struct tuple *tuple);
+/** Port implementation used for storing raw data. */
+struct port_msgpack {
+ const struct port_vtab *vtab;
+ const char *data;
+ uint32_t data_sz;
+};
+
+static_assert(sizeof(struct port_msgpack) <= sizeof(struct port),
+ "sizeof(struct port_msgpack) must be <= sizeof(struct port)");
+
+/** Initialize a port to dump raw data. */
+void
+port_msgpack_create(struct port *port, const char *data, uint32_t data_sz);
+
/** Port for storing the result of a Lua CALL/EVAL. */
struct port_lua {
const struct port_vtab *vtab;
diff --git a/src/lib/core/port.h b/src/lib/core/port.h
index 8ace40fc5..e25e8fc5e 100644
--- a/src/lib/core/port.h
+++ b/src/lib/core/port.h
@@ -30,6 +30,7 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
+#include <stdbool.h>
#include <stdint.h>
#if defined(__cplusplus)
@@ -62,8 +63,13 @@ struct port_vtab {
* header. Used by the legacy Tarantool 1.6 format.
*/
int (*dump_msgpack_16)(struct port *port, struct obuf *out);
- /** Dump the content of a port to Lua stack. */
- void (*dump_lua)(struct port *port, struct lua_State *L);
+ /**
+ * Dump the content of a port to a given Lua stack.
+ * When is_flat == true is specified, the data is dumped
+ * directly to Lua stack, item-by-item. Otherwise, a
+ * result table is created.
+ */
+ void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
/**
* Dump a port content as a plain text into a buffer,
* allocated inside.
@@ -74,6 +80,16 @@ struct port_vtab {
* @retval not nil Plain text.
*/
const char *(*dump_plain)(struct port *port, uint32_t *size);
+ /**
+ * Get the content of a port as a msgpack data.
+ * The lifecycle of the returned value is
+ * implementation-specific: it may either be returned
+ * directly from the port, in which case the data will
+ * stay alive as long as the port is alive, or it may be
+ * allocated on the fiber()->gc, in which case the caller
+ * is responsible for cleaning up.
+ **/
+ const char *(*get_msgpack)(struct port *port, uint32_t *size);
/** Destroy a port and release associated resources. */
void (*destroy)(struct port *port);
};
@@ -109,9 +125,9 @@ port_dump_msgpack_16(struct port *port, struct obuf *out)
}
static inline void
-port_dump_lua(struct port *port, struct lua_State *L)
+port_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
{
- port->vtab->dump_lua(port, L);
+ port->vtab->dump_lua(port, L, is_flat);
}
static inline const char *
@@ -120,6 +136,12 @@ port_dump_plain(struct port *port, uint32_t *size)
return port->vtab->dump_plain(port, size);
}
+static inline const char *
+port_get_msgpack(struct port *port, uint32_t *size)
+{
+ return port->vtab->get_msgpack(port, size);
+}
+
#if defined(__cplusplus)
} /* extern "C" */
#endif /* defined __cplusplus */
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 1/6] box: rework func cache update machinery
2019-06-18 10:52 ` Vladimir Davydov
@ 2019-06-19 15:51 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-19 15:51 UTC (permalink / raw)
To: tarantool-patches, Vladimir Davydov
Done.
=========================================================
Tarantool used to assume that func_new call must not fail and
it used to build a new func object by given definition just on
func cache replace operation. We need to fix it to perform
user-dependent risky actions like Lua function assemble in
further patches.
The replace method is disallowed for _func space because it is
redundant and difficult to maintain in case of functions that
have pre-compiled runtime.
Needed for #4182, #1260
---
src/box/alter.cc | 60 +++++++++++++++++++-----------------
src/box/errcode.h | 2 +-
src/box/func.c | 8 -----
src/box/func.h | 3 --
src/box/func_def.c | 17 ++++++++++
src/box/func_def.h | 12 ++++++++
src/box/schema.cc | 30 ++++++------------
src/box/schema.h | 9 ++----
test/box/function1.result | 4 +++
test/box/function1.test.lua | 1 +
test/box/misc.result | 1 -
test/wal_off/func_max.result | 4 +--
12 files changed, 81 insertions(+), 70 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index a37a68ce4..adad957f2 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2550,6 +2550,10 @@ func_def_new_from_tuple(struct tuple *tuple)
tnt_raise(OutOfMemory, func_def_sizeof(len), "malloc", "def");
auto def_guard = make_scoped_guard([=] { free(def); });
func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid);
+ if (def->fid > BOX_FUNCTION_MAX) {
+ tnt_raise(ClientError, ER_CREATE_FUNCTION,
+ tt_cstr(name, len), "function id is too big");
+ }
memcpy(def->name, name, len);
def->name[len] = 0;
if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
@@ -2574,24 +2578,11 @@ func_def_new_from_tuple(struct tuple *tuple)
/** Remove a function from function cache */
static void
-func_cache_remove_func(struct trigger * /* trigger */, void *event)
-{
- struct txn_stmt *stmt = txn_last_stmt((struct txn *) event);
- uint32_t fid = tuple_field_u32_xc(stmt->old_tuple ?
- stmt->old_tuple : stmt->new_tuple,
- BOX_FUNC_FIELD_ID);
- func_cache_delete(fid);
-}
-
-/** Replace a function in the function cache */
-static void
-func_cache_replace_func(struct trigger * /* trigger */, void *event)
+func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
- struct txn_stmt *stmt = txn_last_stmt((struct txn*) event);
- struct func_def *def = func_def_new_from_tuple(stmt->new_tuple);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
- def_guard.is_active = false;
+ struct func *old_func = (struct func *) trigger->data;
+ func_cache_delete(old_func->def->fid);
+ func_delete(old_func);
}
/**
@@ -2612,13 +2603,17 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
struct func *old_func = func_by_id(fid);
if (new_tuple != NULL && old_func == NULL) { /* INSERT */
struct func_def *def = func_def_new_from_tuple(new_tuple);
+ auto def_guard = make_scoped_guard([=] { free(def); });
access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
PRIV_C);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
- def_guard.is_active = false;
struct trigger *on_rollback =
txn_alter_trigger_new(func_cache_remove_func, NULL);
+ struct func *func = func_new(def);
+ if (func == NULL)
+ diag_raise();
+ def_guard.is_active = false;
+ func_cache_insert(func);
+ on_rollback->data = func;
txn_on_rollback(txn, on_rollback);
} else if (new_tuple == NULL) { /* DELETE */
uint32_t uid;
@@ -2636,16 +2631,25 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
"function has grants");
}
struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_remove_func, NULL);
+ txn_alter_trigger_new(func_cache_remove_func, old_func);
txn_on_commit(txn, on_commit);
} else { /* UPDATE, REPLACE */
- struct func_def *def = func_def_new_from_tuple(new_tuple);
- auto def_guard = make_scoped_guard([=] { free(def); });
- access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
- PRIV_A);
- struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_replace_func, NULL);
- txn_on_commit(txn, on_commit);
+ assert(new_tuple != NULL && old_tuple != NULL);
+ /**
+ * Allow an alter that doesn't change the
+ * definition to support upgrade script.
+ */
+ struct func_def *old_def = NULL, *new_def = NULL;
+ auto guard = make_scoped_guard([=] {
+ free(old_def);
+ free(new_def);
+ });
+ old_def = func_def_new_from_tuple(old_tuple);
+ new_def = func_def_new_from_tuple(new_tuple);
+ if (func_def_cmp(new_def, old_def) != 0) {
+ tnt_raise(ClientError, ER_UNSUPPORTED, "function",
+ "alter");
+ }
}
}
diff --git a/src/box/errcode.h b/src/box/errcode.h
index 28d438333..55299b735 100644
--- a/src/box/errcode.h
+++ b/src/box/errcode.h
@@ -106,7 +106,7 @@ struct errcode_record {
/* 51 */_(ER_NO_SUCH_FUNCTION, "Function '%s' does not exist") \
/* 52 */_(ER_FUNCTION_EXISTS, "Function '%s' already exists") \
/* 53 */_(ER_BEFORE_REPLACE_RET, "Invalid return value of space:before_replace trigger: expected tuple or nil, got %s") \
- /* 54 */_(ER_FUNCTION_MAX, "A limit on the total number of functions has been reached: %u") \
+ /* 54 */_(ER_UNUSED2, "") \
/* 55 */_(ER_TRIGGER_EXISTS, "Trigger '%s' already exists") \
/* 56 */_(ER_USER_MAX, "A limit on the total number of users has been reached: %u") \
/* 57 */_(ER_NO_SUCH_ENGINE, "Space engine '%s' does not exist") \
diff --git a/src/box/func.c b/src/box/func.c
index a817851fd..d1b8879ad 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -451,14 +451,6 @@ func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
return rc;
}
-void
-func_update(struct func *func, struct func_def *def)
-{
- func_unload(func);
- free(func->def);
- func->def = def;
-}
-
void
func_delete(struct func *func)
{
diff --git a/src/box/func.h b/src/box/func.h
index 8dcd61d7b..fa4d738ab 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -100,9 +100,6 @@ module_free(void);
struct func *
func_new(struct func_def *def);
-void
-func_update(struct func *func, struct func_def *def);
-
void
func_delete(struct func *func);
diff --git a/src/box/func_def.c b/src/box/func_def.c
index 76ed77b24..2b135e2d7 100644
--- a/src/box/func_def.c
+++ b/src/box/func_def.c
@@ -1,3 +1,20 @@
#include "func_def.h"
+#include "string.h"
const char *func_language_strs[] = {"LUA", "C"};
+
+int
+func_def_cmp(struct func_def *def1, struct func_def *def2)
+{
+ if (def1->fid != def2->fid)
+ return def1->fid - def2->fid;
+ if (def1->uid != def2->uid)
+ return def1->uid - def2->uid;
+ if (def1->setuid != def2->setuid)
+ return def1->setuid - def2->setuid;
+ if (def1->language != def2->language)
+ return def1->language - def2->language;
+ if (strcmp(def1->name, def2->name) != 0)
+ return strcmp(def1->name, def2->name);
+ return 0;
+}
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 5b52ab498..4c9738aea 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -34,6 +34,10 @@
#include "trivia/util.h"
#include <stdbool.h>
+#ifdef __cplusplus
+extern "C" {
+#endif
+
/**
* The supported language of the stored function.
*/
@@ -79,6 +83,10 @@ func_def_sizeof(uint32_t name_len)
return sizeof(struct func_def) + name_len + 1;
}
+/** Compare two given function definitions. */
+int
+func_def_cmp(struct func_def *def1, struct func_def *def2);
+
/**
* API of C stored function.
*/
@@ -86,4 +94,8 @@ typedef struct box_function_ctx box_function_ctx_t;
typedef int (*box_function_f)(box_function_ctx_t *ctx,
const char *args, const char *args_end);
+#ifdef __cplusplus
+}
+#endif
+
#endif /* TARANTOOL_BOX_FUNC_DEF_H_INCLUDED */
diff --git a/src/box/schema.cc b/src/box/schema.cc
index 6d3153815..d63add535 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -528,6 +528,7 @@ schema_free(void)
struct func *func = ((struct func *)
mh_i32ptr_node(funcs, i)->val);
func_cache_delete(func->def->fid);
+ func_delete(func);
}
mh_i32ptr_delete(funcs);
while (mh_size(sequences) > 0) {
@@ -541,39 +542,27 @@ schema_free(void)
}
void
-func_cache_replace(struct func_def *def)
+func_cache_insert(struct func *func)
{
- struct func *old = func_by_id(def->fid);
- if (old) {
- func_update(old, def);
- return;
- }
- if (mh_size(funcs) >= BOX_FUNCTION_MAX)
- tnt_raise(ClientError, ER_FUNCTION_MAX, BOX_FUNCTION_MAX);
- struct func *func = func_new(def);
- if (func == NULL) {
-error:
- panic_syserror("Out of memory for the data "
- "dictionary cache (stored function).");
- }
- const struct mh_i32ptr_node_t node = { def->fid, func };
+ assert(func_by_id(func->def->fid) == NULL);
+ assert(func_by_name(func->def->name, strlen(func->def->name)) == NULL);
+ const struct mh_i32ptr_node_t node = { func->def->fid, func };
mh_int_t k1 = mh_i32ptr_put(funcs, &node, NULL, NULL);
if (k1 == mh_end(funcs)) {
- func->def = NULL;
func_delete(func);
- goto error;
+ panic_syserror("Out of memory for the data dictionary cache "
+ "(stored function).");
}
size_t def_name_len = strlen(func->def->name);
uint32_t name_hash = mh_strn_hash(func->def->name, def_name_len);
const struct mh_strnptr_node_t strnode = {
func->def->name, def_name_len, name_hash, func };
-
mh_int_t k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
if (k2 == mh_end(funcs_by_name)) {
mh_i32ptr_del(funcs, k1, NULL);
- func->def = NULL;
func_delete(func);
- goto error;
+ panic_syserror("Out of memory for the data dictionary cache "
+ "(stored function).");
}
}
@@ -590,7 +579,6 @@ func_cache_delete(uint32_t fid)
strlen(func->def->name));
if (k != mh_end(funcs))
mh_strnptr_del(funcs_by_name, k, NULL);
- func_delete(func);
}
struct func *
diff --git a/src/box/schema.h b/src/box/schema.h
index 6f9a96117..7e8ac6c11 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -163,14 +163,11 @@ schema_free();
struct space *schema_space(uint32_t id);
/**
- * Insert a new function or update the old one.
- *
- * @param def Function definition. In a case of success the ownership
- * of @a def is transfered to the data dictionary, thus the caller
- * must not delete it.
+ * Insert a new function object in the function cache.
+ * @param func Function object to insert.
*/
void
-func_cache_replace(struct func_def *def);
+func_cache_insert(struct func *func);
void
func_cache_delete(uint32_t fid);
diff --git a/test/box/function1.result b/test/box/function1.result
index 5b04fa97d..cadeb0467 100644
--- a/test/box/function1.result
+++ b/test/box/function1.result
@@ -16,6 +16,10 @@ c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
---
...
+box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
+---
+- error: function does not support alter
+...
box.schema.user.grant('guest', 'execute', 'function', 'function1')
---
...
diff --git a/test/box/function1.test.lua b/test/box/function1.test.lua
index 855dd5382..e983495b6 100644
--- a/test/box/function1.test.lua
+++ b/test/box/function1.test.lua
@@ -7,6 +7,7 @@ net = require('net.box')
c = net.connect(os.getenv("LISTEN"))
box.schema.func.create('function1', {language = "C"})
+box.space._func:replace{2, 1, 'function1', 0, 'LUA'}
box.schema.user.grant('guest', 'execute', 'function', 'function1')
_ = box.schema.space.create('test')
_ = box.space.test:create_index('primary')
diff --git a/test/box/misc.result b/test/box/misc.result
index 43b5a4a15..0b3902304 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -384,7 +384,6 @@ t;
51: box.error.NO_SUCH_FUNCTION
52: box.error.FUNCTION_EXISTS
53: box.error.BEFORE_REPLACE_RET
- 54: box.error.FUNCTION_MAX
55: box.error.TRIGGER_EXISTS
56: box.error.USER_MAX
57: box.error.NO_SUCH_ENGINE
diff --git a/test/wal_off/func_max.result b/test/wal_off/func_max.result
index 5a43821b2..ab4217845 100644
--- a/test/wal_off/func_max.result
+++ b/test/wal_off/func_max.result
@@ -42,7 +42,7 @@ test_run:cmd("setopt delimiter ''");
...
func_limit()
---
-- error: 'A limit on the total number of functions has been reached: 32000'
+- error: 'Failed to create function ''func32000'': function id is too big'
...
drop_limit_func()
---
@@ -62,7 +62,7 @@ session.su('testuser')
...
func_limit()
---
-- error: 'A limit on the total number of functions has been reached: 32000'
+- error: 'Failed to create function ''func32000'': function id is too big'
...
drop_limit_func()
---
--
2.21.0
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
@ 2019-06-19 16:11 ` Vladimir Davydov
0 siblings, 0 replies; 29+ messages in thread
From: Vladimir Davydov @ 2019-06-19 16:11 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Wed, Jun 19, 2019 at 06:51:15PM +0300, Kirill Shcherbatov wrote:
> >> module_gc(module);
> >> + if (rc != 0) {
> >> + if (diag_last_error(&fiber()->diag) == NULL) {
> >> + /* Stored procedure forget to set diag */
> >> + diag_set(ClientError, ER_PROC_C, "unknown error");
> >> + }
> >> + port_destroy(out_port);
> >> + return -1;
> >> + }
> >
> >Shouldn't you call region_truncate() here to clean up after the function
> >and possibly port_get_msgpack()?
> No, the func_c_call implementation must clean region by it's own.
But you did add region_truncate() to the place I pointed to.
I'm confused...
Anyway, please resend the whole series in a separate thread rather
than bloating this one.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
` (2 preceding siblings ...)
2019-06-18 13:58 ` Vladimir Davydov
@ 2019-06-19 18:28 ` Konstantin Osipov
2019-06-20 7:53 ` Kirill Shcherbatov
2019-06-19 18:30 ` [tarantool-patches] " Konstantin Osipov
4 siblings, 1 reply; 29+ messages in thread
From: Konstantin Osipov @ 2019-06-19 18:28 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/13 17:18]:
> Re-factor box_lua_call and box_lua_eval so that they don't take
> call_request. This approach is more scalable: in case of a
> functional index, the user expects to see a tuple with field
> names so we should be able to pass not only raw msgpack, but
> also a tuple to a Lua call so we need an universal way to pass
> arguments to _call methods.
What is is_flat?
Please why do you have to ask for an explanation, shouldn't you
update the declaration comment with an explanation?
You also don't mention is_flat in the CS comment.
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
` (3 preceding siblings ...)
2019-06-19 18:28 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-19 18:30 ` Konstantin Osipov
4 siblings, 0 replies; 29+ messages in thread
From: Konstantin Osipov @ 2019-06-19 18:30 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/13 17:18]:
> + /** Get the content of a port as a msgpack data. */
> + const char *(*get_msgpack)(struct port *port, uint32_t *size);
The comment is bare. In fact it's infuriatingly trivial. Please
explain why and how.
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-19 18:28 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-20 7:53 ` Kirill Shcherbatov
2019-06-20 8:09 ` Konstantin Osipov
0 siblings, 1 reply; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-20 7:53 UTC (permalink / raw)
To: Konstantin Osipov, tarantool-patches; +Cc: vdavydov.dev
> What is is_flat?
>
> Please why do you have to ask for an explanation, shouldn't you
> update the declaration comment with an explanation?
>
> You also don't mention is_flat in the CS comment.
....
> The comment is bare. In fact it's infuriatingly trivial. Please
> explain why and how.
Hi, you are right, this really was a flaw of previous version.
But Vova already pointed to it and it is already fixed.
/**
* Dump the content of a port to a given Lua stack.
* When is_flat == true is specified, the data is dumped
* directly to Lua stack, item-by-item. Otherwise, a
* result table is created.
*/
void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
/**
* Get the content of a port as a msgpack data.
* The lifecycle of the returned value is
* implementation-specific: it may either be returned
* directly from the port, in which case the data will
* stay alive as long as the port is alive, or it may be
* allocated on the fiber()->gc, in which case the caller
* is responsible for cleaning up.
**/
const char *(*get_msgpack)(struct port *port, uint32_t *size);
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-20 7:53 ` Kirill Shcherbatov
@ 2019-06-20 8:09 ` Konstantin Osipov
2019-06-20 8:44 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 29+ messages in thread
From: Konstantin Osipov @ 2019-06-20 8:09 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/20 10:54]:
> /**
> * Dump the content of a port to a given Lua stack.
> * When is_flat == true is specified, the data is dumped
> * directly to Lua stack, item-by-item. Otherwise, a
> * result table is created.
> */
Still I miss why we need this mode from this comment. Is this for
1.6 compatibility? I would ditch it in this case, we are not
obliged to support 1.6 in 2.3.
> void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
>
> /**
> * Get the content of a port as a msgpack data.
> * The lifecycle of the returned value is
> * implementation-specific: it may either be returned
> * directly from the port, in which case the data will
> * stay alive as long as the port is alive, or it may be
> * allocated on the fiber()->gc, in which case the caller
> * is responsible for cleaning up.
> **/
Once again, please also explain how it is going tobe used - by an
obsolete convention, C stored routines expect msgpack data as
input format for its arguments.
> const char *(*get_msgpack)(struct port *port, uint32_t *size);
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port
2019-06-20 8:09 ` Konstantin Osipov
@ 2019-06-20 8:44 ` Kirill Shcherbatov
0 siblings, 0 replies; 29+ messages in thread
From: Kirill Shcherbatov @ 2019-06-20 8:44 UTC (permalink / raw)
To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev
On 20.06.2019 11:09, Konstantin Osipov wrote:
> * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/06/20 10:54]:
>> /**
>> * Dump the content of a port to a given Lua stack.
>> * When is_flat == true is specified, the data is dumped
>> * directly to Lua stack, item-by-item. Otherwise, a
>> * result table is created.
>> */
>
> Still I miss why we need this mode from this comment. Is this for
> 1.6 compatibility? I would ditch it in this case, we are not
> obliged to support 1.6 in 2.3.
/**
* Dump the content of a port to a given Lua stack.
* When is_flat == true is specified, the data is dumped
* directly to Lua stack, item-by-item. Otherwise, a
* result table is created. The is_flat == true mode
* follows Lua functions convention to pass arguments
* and return a results, while is_flat == false follows
* Tarantool's :select convention when the table of
* results is returned.
*/
void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
>
>> void (*dump_lua)(struct port *port, struct lua_State *L, bool is_flat);
>>
>> /**
>> * Get the content of a port as a msgpack data.
>> * The lifecycle of the returned value is
>> * implementation-specific: it may either be returned
>> * directly from the port, in which case the data will
>> * stay alive as long as the port is alive, or it may be
>> * allocated on the fiber()->gc, in which case the caller
>> * is responsible for cleaning up.
>> **/
>
> Once again, please also explain how it is going tobe used - by an
> obsolete convention, C stored routines expect msgpack data as
> input format for its arguments.
>
>> const char *(*get_msgpack)(struct port *port, uint32_t *size);
/**
* Get the content of a port as a msgpack data.
* By an obsolete convention, C stored routines expect
* msgpack data as input format for its arguments.
* This API is also usefull to process a function
* returned value as msgpack data in memory.
* The lifecycle of the returned value is
* implementation-specific: it may either be returned
* directly from the port, in which case the data will
* stay alive as long as the port is alive, or it may be
* allocated on the fiber()->gc, in which case the caller
* is responsible for cleaning up.
**/
const char *(*get_msgpack)(struct port *port, uint32_t *size);
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2019-06-20 8:44 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-13 14:08 [PATCH v3 0/6] box: rework functions machinery Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 1/6] box: rework func cache update machinery Kirill Shcherbatov
2019-06-18 10:52 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 2/6] box: rework box_lua_{call, eval} to use input port Kirill Shcherbatov
2019-06-17 9:35 ` [tarantool-patches] " Konstantin Osipov
2019-06-17 10:27 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-18 12:12 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-19 16:11 ` Vladimir Davydov
2019-06-18 13:58 ` Vladimir Davydov
2019-06-19 18:28 ` [tarantool-patches] " Konstantin Osipov
2019-06-20 7:53 ` Kirill Shcherbatov
2019-06-20 8:09 ` Konstantin Osipov
2019-06-20 8:44 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-19 18:30 ` [tarantool-patches] " Konstantin Osipov
2019-06-13 14:08 ` [PATCH v3 3/6] box: rework func object as a function frontend Kirill Shcherbatov
2019-06-18 13:23 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 4/6] box: export registered functions in box.func folder Kirill Shcherbatov
2019-06-18 14:06 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-18 16:11 ` Vladimir Davydov
2019-06-13 14:08 ` [PATCH v3 5/6] box: refactor box_lua_find helper Kirill Shcherbatov
2019-06-18 14:22 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
2019-06-13 14:08 ` [PATCH v3 6/6] box: introduce Lua persistent functions Kirill Shcherbatov
2019-06-18 16:23 ` Vladimir Davydov
2019-06-19 15:51 ` [tarantool-patches] " Kirill Shcherbatov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox