* [PATCH v1 0/8] box: functional indexes
@ 2019-05-30 10:45 Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 1/8] box: refactor box_lua_find helper Kirill Shcherbatov
` (10 more replies)
0 siblings, 11 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
The patch set introduces persistent Lua functions and functional
indexes.
Lua functions managed by Tarantool are called 'persistent'.
They are stored in snapshoot and are available after server
restart.
Persistent Lua functions are exported in box.schema.func.persistent
folder. They may be called via .call exported object method or
via netbox:call.
Some restrictions must be accounted writing such routines:
1. only limited set of Lua functions and modules are available:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8;
2. global variables are forbidden.
Moreover all functions and their metadata are now exported in
box.func table (not only persistent). Persistent Lua functions
have box.func.FUNCNAME.call() method to execute it in interactive
console.
Persistent Lua functions are also available with net.box call.
A functional index use some registered persistent function as
key extractor.
You are not allowed to change functional index extractor function
while there are some functional indexes depends of it.
Branch: http://github.com/tarantool/tarantool/tree/kshch/gh-1260-functional-index-new
Issue: https://github.com/tarantool/tarantool/issues/1260
Kirill Shcherbatov (8):
box: refactor box_lua_find helper
box: rework func cache update machinery
schema: rework _func system space format
box: load persistent Lua functions on creation
netbox: call persistent functions in netbox
box: export _func functions with box.func folder
box: introduce memtx_slab_alloc helper
box: introduce functional indexes in memtx
src/box/CMakeLists.txt | 1 +
src/box/alter.cc | 175 +++++++--
src/box/bootstrap.snap | Bin 4393 -> 4447 bytes
src/box/call.c | 2 +-
src/box/errcode.h | 1 +
src/box/func.c | 203 +++++++++-
src/box/func.h | 38 +-
src/box/func_def.h | 28 +-
src/box/index_def.c | 22 +-
src/box/index_def.h | 16 +
src/box/key_def.c | 13 +-
src/box/key_def.h | 31 +-
src/box/lua/call.c | 121 +++---
src/box/lua/call.h | 3 +-
src/box/lua/init.c | 2 +
src/box/lua/key_def.c | 2 +-
src/box/lua/schema.lua | 13 +-
src/box/lua/space.cc | 5 +
src/box/lua/tuple_fextract_key.c | 233 +++++++++++
src/box/lua/tuple_fextract_key.h | 100 +++++
src/box/lua/upgrade.lua | 22 +-
src/box/memtx_engine.c | 44 ++-
src/box/memtx_engine.h | 19 +
src/box/memtx_space.c | 18 +
src/box/memtx_tree.c | 355 ++++++++++++++++-
src/box/schema.cc | 49 ++-
src/box/schema.h | 31 +-
src/box/schema_def.h | 3 +
src/box/sql.c | 2 +-
src/box/sql/build.c | 2 +-
src/box/sql/select.c | 2 +-
src/box/sql/where.c | 2 +-
src/box/tuple_compare.cc | 112 +++++-
src/box/tuple_extract_key.cc | 33 +-
src/box/vinyl.c | 9 +-
src/lua/utils.c | 126 ++++++
src/lua/utils.h | 19 +
test/box-py/bootstrap.result | 6 +-
test/box/access_misc.result | 6 +-
test/box/bitset.result | 24 ++
test/box/bitset.test.lua | 9 +
test/box/hash.result | 24 ++
test/box/hash.test.lua | 9 +
test/box/misc.result | 2 +
test/box/persistent_func.result | 201 ++++++++++
test/box/persistent_func.test.lua | 88 +++++
test/box/rtree_misc.result | 24 ++
test/box/rtree_misc.test.lua | 9 +
test/engine/engine.cfg | 5 +-
test/engine/functional.result | 620 ++++++++++++++++++++++++++++++
test/engine/functional.test.lua | 214 +++++++++++
test/unit/luaT_tuple_new.c | 2 +-
test/unit/merger.test.c | 4 +-
53 files changed, 2898 insertions(+), 206 deletions(-)
create mode 100644 src/box/lua/tuple_fextract_key.c
create mode 100644 src/box/lua/tuple_fextract_key.h
create mode 100644 test/box/persistent_func.result
create mode 100644 test/box/persistent_func.test.lua
create mode 100644 test/engine/functional.result
create mode 100644 test/engine/functional.test.lua
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 1/8] box: refactor box_lua_find helper
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 2/8] box: rework func cache update machinery Kirill Shcherbatov
` (9 subsequent siblings)
10 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
The box_lua_find routine used to work with an empty stack only.
It is unacceptable in following patches because this helper
need to be reused in following patches for environment table
construction.
The definition also is moved to lua/utils.h.
Needed for #4182, #1260
---
src/box/lua/call.c | 85 ++++++++--------------------------------------
src/lua/utils.c | 59 ++++++++++++++++++++++++++++++++
src/lua/utils.h | 11 ++++++
3 files changed, 85 insertions(+), 70 deletions(-)
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index 04020ef6f..c729778c4 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -44,73 +44,6 @@
#include "trivia/util.h"
#include "mpstream.h"
-/**
- * A helper to find a Lua function by name and put it
- * on top of the stack.
- */
-static int
-box_lua_find(lua_State *L, const char *name, const char *name_end)
-{
- int index = LUA_GLOBALSINDEX;
- int objstack = 0;
- const char *start = name, *end;
-
- while ((end = (const char *) memchr(start, '.', name_end - start))) {
- lua_checkstack(L, 3);
- lua_pushlstring(L, start, end - start);
- lua_gettable(L, index);
- if (! lua_istable(L, -1)) {
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
- }
- start = end + 1; /* next piece of a.b.c */
- index = lua_gettop(L); /* top of the stack */
- }
-
- /* box.something:method */
- if ((end = (const char *) memchr(start, ':', name_end - start))) {
- lua_checkstack(L, 3);
- lua_pushlstring(L, start, end - start);
- lua_gettable(L, index);
- if (! (lua_istable(L, -1) ||
- lua_islightuserdata(L, -1) || lua_isuserdata(L, -1) )) {
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
- }
- start = end + 1; /* next piece of a.b.c */
- index = lua_gettop(L); /* top of the stack */
- objstack = index;
- }
-
-
- lua_pushlstring(L, start, name_end - start);
- lua_gettable(L, index);
- if (!lua_isfunction(L, -1) && !lua_istable(L, -1)) {
- /* lua_call or lua_gettable would raise a type error
- * for us, but our own message is more verbose. */
- diag_set(ClientError, ER_NO_SUCH_PROC,
- name_end - name, name);
- luaT_error(L);
- }
- /* setting stack that it would contain only
- * the function pointer. */
- if (index != LUA_GLOBALSINDEX) {
- if (objstack == 0) { /* no object, only a function */
- lua_replace(L, 1);
- } else if (objstack == 1) { /* just two values, swap them */
- lua_insert(L, -2);
- } else { /* long path */
- lua_insert(L, 1);
- lua_insert(L, 2);
- objstack = 1;
- }
- lua_settop(L, 1 + objstack);
- }
- return 1 + objstack;
-}
-
/**
* A helper to find lua stored procedures for box.call.
* box.call iteslf is pure Lua, to avoid issues
@@ -124,7 +57,12 @@ lbox_call_loadproc(struct lua_State *L)
const char *name;
size_t name_len;
name = lua_tolstring(L, 1, &name_len);
- return box_lua_find(L, name, name + name_len);
+ int count;
+ if (luaT_func_find(L, name, name + name_len, &count) != 0) {
+ diag_set(ClientError, ER_NO_SUCH_PROC, name_len, name);
+ return luaT_error(L);
+ }
+ return count;
}
/*
@@ -292,9 +230,16 @@ execute_lua_call(lua_State *L)
const char *name = request->name;
uint32_t name_len = mp_decode_strl(&name);
- int oc = 0; /* how many objects are on stack after box_lua_find */
+ /*
+ * How many objects are on stack after
+ * luaT_func_find call.
+ */
+ int oc = 0;
/* Try to find a function by name in Lua */
- oc = box_lua_find(L, name, name + name_len);
+ if (luaT_func_find(L, name, name + name_len, &oc) != 0) {
+ diag_set(ClientError, ER_NO_SUCH_PROC, name_len, name);
+ return luaT_error(L);
+ }
/* Push the rest of args (a tuple). */
const char *args = request->args;
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 01a0cd894..27ff6b396 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -1189,6 +1189,65 @@ void luaL_iterator_delete(struct luaL_iterator *it)
/* }}} */
+int
+luaT_func_find(struct lua_State *L, const char *name, const char *name_end,
+ int *count)
+{
+ int index = LUA_GLOBALSINDEX;
+ int objstack = 0, top = lua_gettop(L);
+ const char *start = name, *end;
+
+ while ((end = (const char *) memchr(start, '.', name_end - start))) {
+ lua_checkstack(L, 3);
+ lua_pushlstring(L, start, end - start);
+ lua_gettable(L, index);
+ if (! lua_istable(L, -1))
+ return -1;
+ start = end + 1; /* next piece of a.b.c */
+ index = lua_gettop(L); /* top of the stack */
+ }
+
+ /* box.something:method */
+ if ((end = (const char *) memchr(start, ':', name_end - start))) {
+ lua_checkstack(L, 3);
+ lua_pushlstring(L, start, end - start);
+ lua_gettable(L, index);
+ if (! (lua_istable(L, -1) ||
+ lua_islightuserdata(L, -1) || lua_isuserdata(L, -1) ))
+ return -1;
+ start = end + 1; /* next piece of a.b.c */
+ index = lua_gettop(L); /* top of the stack */
+ objstack = index - top;
+ }
+
+ lua_pushlstring(L, start, name_end - start);
+ lua_gettable(L, index);
+ if (!lua_isfunction(L, -1) && !lua_istable(L, -1)) {
+ /* lua_call or lua_gettable would raise a type error
+ * for us, but our own message is more verbose. */
+ return -1;
+ }
+
+ /* setting stack that it would contain only
+ * the function pointer. */
+ if (index != LUA_GLOBALSINDEX) {
+ if (objstack == 0) { /* no object, only a function */
+ lua_replace(L, top + 1);
+ lua_pop(L, lua_gettop(L) - top - 1);
+ } else if (objstack == 1) { /* just two values, swap them */
+ lua_insert(L, -2);
+ lua_pop(L, lua_gettop(L) - top - 2);
+ } else { /* long path */
+ lua_insert(L, top + 1);
+ lua_insert(L, top + 2);
+ lua_pop(L, objstack - 1);
+ objstack = 1;
+ }
+ }
+ *count = 1 + objstack;
+ return 0;
+}
+
int
tarantool_lua_utils_init(struct lua_State *L)
{
diff --git a/src/lua/utils.h b/src/lua/utils.h
index 943840ec0..81e936bee 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -592,6 +592,17 @@ void luaL_iterator_delete(struct luaL_iterator *it);
/* }}} */
+/**
+ * A helper to find a Lua function by name and put it
+ * on top of the stack.
+ * Returns 0 in case of succsess and -1 otherwise; in case of
+ * success also uses count[out] argument to return the how many
+ * objects are pushed on to stack.
+ */
+int
+luaT_func_find(struct lua_State *L, const char *name, const char *name_end,
+ int *count);
+
int
tarantool_lua_utils_init(struct lua_State *L);
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 2/8] box: rework func cache update machinery
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 1/8] box: refactor box_lua_find helper Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 3/8] schema: rework _func system space format Kirill Shcherbatov
` (8 subsequent siblings)
10 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Tarantool used to assume that func_new call must not fail and
it used to build a new func object by given definition just on
func cache replace operation. It is not good approach itself and
moreover we a going to implement a persistent function object
that would assemble itself on creation that might fail.
Needed for #4182, #1260
---
src/box/alter.cc | 43 ++++++++++++++++++++++++++-----------------
src/box/func.c | 13 +++++++------
src/box/func.h | 12 +++++++++---
src/box/schema.cc | 46 ++++++++++++++++++++--------------------------
src/box/schema.h | 15 +++++++++------
5 files changed, 71 insertions(+), 58 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index ed9e55907..3b57a7d82 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2463,24 +2463,22 @@ func_def_new_from_tuple(struct tuple *tuple)
/** Remove a function from function cache */
static void
-func_cache_remove_func(struct trigger * /* trigger */, void *event)
+func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
- struct txn_stmt *stmt = txn_last_stmt((struct txn *) event);
- uint32_t fid = tuple_field_u32_xc(stmt->old_tuple ?
- stmt->old_tuple : stmt->new_tuple,
- BOX_FUNC_FIELD_ID);
- func_cache_delete(fid);
+ struct func *old_func = (struct func *) trigger->data;
+ func_cache_delete(old_func->def->fid);
+ func_delete(old_func);
}
/** Replace a function in the function cache */
static void
-func_cache_replace_func(struct trigger * /* trigger */, void *event)
+func_cache_replace_func(struct trigger *trigger, void * /* event */)
{
- struct txn_stmt *stmt = txn_last_stmt((struct txn*) event);
- struct func_def *def = func_def_new_from_tuple(stmt->new_tuple);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
- def_guard.is_active = false;
+ struct func *new_func = (struct func *) trigger->data;
+ struct func *old_func;
+ func_cache_replace(new_func, &old_func);
+ assert(old_func != NULL);
+ func_delete(old_func);
}
/**
@@ -2501,13 +2499,20 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
struct func *old_func = func_by_id(fid);
if (new_tuple != NULL && old_func == NULL) { /* INSERT */
struct func_def *def = func_def_new_from_tuple(new_tuple);
+ auto def_guard = make_scoped_guard([=] { free(def); });
access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
PRIV_C);
- auto def_guard = make_scoped_guard([=] { free(def); });
- func_cache_replace(def);
+ struct func *func = func_new(def);
+ if (func == NULL)
+ diag_raise();
+ auto func_guard = make_scoped_guard([=] { func_delete(func); });
def_guard.is_active = false;
+ struct func *old_func = NULL;
+ func_cache_replace(func, &old_func);
+ assert(old_func == NULL);
+ func_guard.is_active = false;
struct trigger *on_rollback =
- txn_alter_trigger_new(func_cache_remove_func, NULL);
+ txn_alter_trigger_new(func_cache_remove_func, func);
txn_on_rollback(txn, on_rollback);
} else if (new_tuple == NULL) { /* DELETE */
uint32_t uid;
@@ -2525,15 +2530,19 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
"function has grants");
}
struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_remove_func, NULL);
+ txn_alter_trigger_new(func_cache_remove_func, old_func);
txn_on_commit(txn, on_commit);
} else { /* UPDATE, REPLACE */
struct func_def *def = func_def_new_from_tuple(new_tuple);
auto def_guard = make_scoped_guard([=] { free(def); });
access_check_ddl(def->name, def->fid, def->uid, SC_FUNCTION,
PRIV_A);
+ struct func *func = func_new(def);
+ if (func == NULL)
+ diag_raise();
+ def_guard.is_active = false;
struct trigger *on_commit =
- txn_alter_trigger_new(func_cache_replace_func, NULL);
+ txn_alter_trigger_new(func_cache_replace_func, func);
txn_on_commit(txn, on_commit);
}
}
diff --git a/src/box/func.c b/src/box/func.c
index a817851fd..f7465be7e 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -452,17 +452,18 @@ func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
}
void
-func_update(struct func *func, struct func_def *def)
+func_delete(struct func *func)
{
func_unload(func);
free(func->def);
- func->def = def;
+ free(func);
}
void
-func_delete(struct func *func)
+func_capture_module(struct func *new_func, struct func *old_func)
{
- func_unload(func);
- free(func->def);
- free(func);
+ new_func->module = old_func->module;
+ new_func->func = old_func->func;
+ old_func->module = NULL;
+ old_func->func = NULL;
}
diff --git a/src/box/func.h b/src/box/func.h
index 8dcd61d7b..a4a758b58 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -100,9 +100,6 @@ module_free(void);
struct func *
func_new(struct func_def *def);
-void
-func_update(struct func *func, struct func_def *def);
-
void
func_delete(struct func *func);
@@ -113,6 +110,15 @@ int
func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
const char *args_end);
+/**
+ * Take over the modules that belonged to the old function.
+ * Reset module and function pointers in old function.
+ * This helper allows you to inherit already loaded function
+ * objects, that is useful on update the function cache.
+*/
+void
+func_capture_module(struct func *new_func, struct func *old_func);
+
/**
* Reload dynamically loadable module.
*
diff --git a/src/box/schema.cc b/src/box/schema.cc
index 9a55c2f14..b70992297 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -533,40 +533,35 @@ schema_free(void)
}
void
-func_cache_replace(struct func_def *def)
+func_cache_replace(struct func *new_func, struct func **old_func)
{
- struct func *old = func_by_id(def->fid);
- if (old) {
- func_update(old, def);
- return;
- }
if (mh_size(funcs) >= BOX_FUNCTION_MAX)
tnt_raise(ClientError, ER_FUNCTION_MAX, BOX_FUNCTION_MAX);
- struct func *func = func_new(def);
- if (func == NULL) {
-error:
- panic_syserror("Out of memory for the data "
- "dictionary cache (stored function).");
- }
- const struct mh_i32ptr_node_t node = { def->fid, func };
- mh_int_t k1 = mh_i32ptr_put(funcs, &node, NULL, NULL);
- if (k1 == mh_end(funcs)) {
- func->def = NULL;
- func_delete(func);
- goto error;
- }
- size_t def_name_len = strlen(func->def->name);
- uint32_t name_hash = mh_strn_hash(func->def->name, def_name_len);
+
+ mh_int_t k1, k2;
+ struct mh_i32ptr_node_t old_node, *old_node_ptr = &old_node;
+ const struct mh_i32ptr_node_t node = { new_func->def->fid, new_func };
+ size_t def_name_len = strlen(new_func->def->name);
+ uint32_t name_hash = mh_strn_hash(new_func->def->name, def_name_len);
const struct mh_strnptr_node_t strnode = {
- func->def->name, def_name_len, name_hash, func };
+ new_func->def->name, def_name_len, name_hash, new_func };
- mh_int_t k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
+ k1 = mh_i32ptr_put(funcs, &node, &old_node_ptr, NULL);
+ if (k1 == mh_end(funcs))
+ goto error;
+ if (old_node_ptr != NULL)
+ *old_func = (struct func *)old_node_ptr->val;
+ k2 = mh_strnptr_put(funcs_by_name, &strnode, NULL, NULL);
if (k2 == mh_end(funcs_by_name)) {
mh_i32ptr_del(funcs, k1, NULL);
- func->def = NULL;
- func_delete(func);
goto error;
}
+ if (*old_func != NULL)
+ func_capture_module(new_func, *old_func);
+ return;
+error:
+ panic_syserror("Out of memory for the data dictionary cache "
+ "(stored function).");
}
void
@@ -582,7 +577,6 @@ func_cache_delete(uint32_t fid)
strlen(func->def->name));
if (k != mh_end(funcs))
mh_strnptr_del(funcs_by_name, k, NULL);
- func_delete(func);
}
struct func *
diff --git a/src/box/schema.h b/src/box/schema.h
index 6f9a96117..ffc41f401 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -163,14 +163,17 @@ schema_free();
struct space *schema_space(uint32_t id);
/**
- * Insert a new function or update the old one.
- *
- * @param def Function definition. In a case of success the ownership
- * of @a def is transfered to the data dictionary, thus the caller
- * must not delete it.
+ * Replace an existent (if any) function or insert a new one
+ * in function cache.
+ * @param func_new Function object to insert. In case of success,
+ * when old function object is exists, all loaded
+ * modules data are inherent with
+ * func_capture_module() when possible.
+ * @param func_old[out] The replaced function object if any.
+ * @retval Returns 0 on success, -1 otherwise.
*/
void
-func_cache_replace(struct func_def *def);
+func_cache_replace(struct func *new_func, struct func **old_func);
void
func_cache_delete(uint32_t fid);
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 3/8] schema: rework _func system space format
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 1/8] box: refactor box_lua_find helper Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 2/8] box: rework func cache update machinery Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 12:06 ` [tarantool-patches] " Konstantin Osipov
2019-05-30 13:10 ` Vladislav Shpilevoy
2019-05-30 10:45 ` [PATCH v1 4/8] box: load persistent Lua functions on creation Kirill Shcherbatov
` (7 subsequent siblings)
10 siblings, 2 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
This patch updates a _func system space to prepare it for
working with persistent functions. The format of the _func
system space is
[<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
<language> STR, <body> STR, <returns> STR,
<is_deterministic> BOOL]
It is preparational step to introduce persistent Lua function
in Tarantool.
The new <body> field is a string that represents the function
body, returns is the type of value that it returns, and the
is_deterministic flag is responsible whether this routine
deterministic or not (can produce only one result for a given
list of parameters).
Updated function definition structure, decode operation and
migration script correspondingly.
Part of #4182
Needed for #1260
---
src/box/alter.cc | 63 +++++++++++++++++++++++++++++------
src/box/bootstrap.snap | Bin 4393 -> 4447 bytes
src/box/func_def.h | 22 ++++++++----
src/box/lua/schema.lua | 10 ++++--
src/box/lua/upgrade.lua | 22 +++++++++++-
src/box/schema_def.h | 3 ++
test/box-py/bootstrap.result | 6 ++--
test/box/access_misc.result | 6 ++--
8 files changed, 106 insertions(+), 26 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 3b57a7d82..c9446bfe0 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2426,26 +2426,47 @@ func_def_get_ids_from_tuple(struct tuple *tuple, uint32_t *fid, uint32_t *uid)
static struct func_def *
func_def_new_from_tuple(struct tuple *tuple)
{
- uint32_t len;
- const char *name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME,
- &len);
- if (len > BOX_NAME_MAX)
+ uint32_t field_count = tuple_field_count(tuple);
+ uint32_t name_len, body_len;
+ const char *name, *body;
+ name = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_NAME, &name_len);
+ if (name_len > BOX_NAME_MAX) {
tnt_raise(ClientError, ER_CREATE_FUNCTION,
tt_cstr(name, BOX_INVALID_NAME_MAX),
"function name is too long");
- identifier_check_xc(name, len);
- struct func_def *def = (struct func_def *) malloc(func_def_sizeof(len));
+ }
+ identifier_check_xc(name, name_len);
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ body = tuple_field_str_xc(tuple, BOX_FUNC_FIELD_BODY,
+ &body_len);
+ } else {
+ body = NULL;
+ body_len = 0;
+ }
+
+ uint32_t def_sz = func_def_sizeof(name_len, body_len);
+ struct func_def *def = (struct func_def *) malloc(def_sz);
if (def == NULL)
- tnt_raise(OutOfMemory, func_def_sizeof(len), "malloc", "def");
+ tnt_raise(OutOfMemory, def_sz, "malloc", "def");
auto def_guard = make_scoped_guard([=] { free(def); });
func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid);
- memcpy(def->name, name, len);
- def->name[len] = 0;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_SETUID)
+
+ def->name = (char *)def + sizeof(struct func_def);
+ memcpy(def->name, name, name_len);
+ def->name[name_len] = 0;
+ if (body_len > 0) {
+ def->body = def->name + name_len + 1;
+ memcpy(def->body, body, body_len);
+ def->body[body_len] = 0;
+ } else {
+ def->body = NULL;
+ }
+
+ if (field_count > BOX_FUNC_FIELD_SETUID)
def->setuid = tuple_field_u32_xc(tuple, BOX_FUNC_FIELD_SETUID);
else
def->setuid = false;
- if (tuple_field_count(tuple) > BOX_FUNC_FIELD_LANGUAGE) {
+ if (field_count > BOX_FUNC_FIELD_LANGUAGE) {
const char *language =
tuple_field_cstr_xc(tuple, BOX_FUNC_FIELD_LANGUAGE);
def->language = STR2ENUM(func_language, language);
@@ -2457,6 +2478,26 @@ func_def_new_from_tuple(struct tuple *tuple)
/* Lua is the default. */
def->language = FUNC_LANGUAGE_LUA;
}
+ if (def->language != FUNC_LANGUAGE_LUA && body_len > 0) {
+ tnt_raise(ClientError, ER_CREATE_FUNCTION, name,
+ "function body may be specified only for "
+ "Lua language");
+ }
+ if (field_count > BOX_FUNC_FIELD_BODY) {
+ assert(field_count > BOX_FUNC_FIELD_IS_DETERMINISTIC);
+ const char *type =
+ tuple_field_cstr_xc(tuple, BOX_FUNC_FIELD_RETURNS);
+ def->returns = STR2ENUM(field_type, type);
+ if (def->returns == field_type_MAX) {
+ tnt_raise(ClientError, ER_CREATE_FUNCTION, name,
+ "function return value has unknown field type");
+ }
+ def->is_deterministic =
+ tuple_field_bool_xc(tuple, BOX_FUNC_FIELD_IS_DETERMINISTIC);
+ } else {
+ def->returns = FIELD_TYPE_ANY;
+ def->is_deterministic = false;
+ }
def_guard.is_active = false;
return def;
}
diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 5b52ab498..78fef9d22 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -32,6 +32,7 @@
*/
#include "trivia/util.h"
+#include "field_def.h"
#include <stdbool.h>
/**
@@ -54,17 +55,21 @@ struct func_def {
uint32_t fid;
/** Owner of the function. */
uint32_t uid;
+ /** Function name. */
+ char *name;
+ /** Definition of the routine. */
+ char *body;
/**
* True if the function requires change of user id before
* invocation.
*/
bool setuid;
- /**
- * The language of the stored function.
- */
+ /** The language of the stored function. */
enum func_language language;
- /** Function name. */
- char name[0];
+ /** The type of the returned value. */
+ enum field_type returns;
+ /** Whether this function is deterministic. */
+ bool is_deterministic;
};
/**
@@ -73,10 +78,13 @@ struct func_def {
* for a function of length @a a name_len.
*/
static inline size_t
-func_def_sizeof(uint32_t name_len)
+func_def_sizeof(uint32_t name_len, uint32_t body_len)
{
/* +1 for '\0' name terminating. */
- return sizeof(struct func_def) + name_len + 1;
+ size_t sz = sizeof(struct func_def) + name_len + 1;
+ if (body_len > 0)
+ sz += body_len + 1;
+ return sz;
}
/**
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 39bd8da6d..960ea0fc7 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2105,7 +2105,8 @@ box.schema.func.create = function(name, opts)
opts = opts or {}
check_param_table(opts, { setuid = 'boolean',
if_not_exists = 'boolean',
- language = 'string'})
+ language = 'string', body = 'string',
+ returns = 'string', is_deterministic = 'boolean'})
local _func = box.space[box.schema.FUNC_ID]
local _vfunc = box.space[box.schema.VFUNC_ID]
local func = _vfunc.index.name:get{name}
@@ -2115,10 +2116,13 @@ box.schema.func.create = function(name, opts)
end
return
end
- opts = update_param_table(opts, { setuid = false, language = 'lua'})
+ opts = update_param_table(opts, { setuid = false, language = 'lua',
+ body = '', returns = 'any',
+ is_deterministic = false})
opts.language = string.upper(opts.language)
opts.setuid = opts.setuid and 1 or 0
- _func:auto_increment{session.euid(), name, opts.setuid, opts.language}
+ _func:auto_increment{session.euid(), name, opts.setuid, opts.language,
+ opts.body, opts.returns, opts.is_deterministic}
end
box.schema.func.drop = function(name, opts)
diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua
index 070662698..c38e5e3ba 100644
--- a/src/box/lua/upgrade.lua
+++ b/src/box/lua/upgrade.lua
@@ -324,7 +324,7 @@ local function initial_1_7_5()
-- create "box.schema.user.info" function
log.info('create function "box.schema.user.info" with setuid')
- _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA'}
+ _func:replace{1, ADMIN, 'box.schema.user.info', 1, 'LUA', '', 'any', false}
-- grant 'public' role access to 'box.schema.user.info' function
log.info('grant execute on function "box.schema.user.info" to public')
@@ -774,8 +774,28 @@ local function upgrade_sequence_to_2_2_1()
_space_sequence:format(format)
end
+local function upgrade_func_to_2_2_1()
+ log.info("Update _func format")
+ local _func = box.space[box.schema.FUNC_ID]
+ local format = {}
+ format[1] = {name='id', type='unsigned'}
+ format[2] = {name='owner', type='unsigned'}
+ format[3] = {name='name', type='string'}
+ format[4] = {name='setuid', type='unsigned'}
+ format[5] = {name='language', type='string'}
+ format[6] = {name='body', type='string'}
+ format[7] = {name='returns', type='string'}
+ format[8] = {name='is_deterministic', type='boolean'}
+ for _, v in box.space._func:pairs() do
+ _ = box.space._func:replace({v.id, v.owner, v.name, v.setuid,
+ v[5] or 'LUA', '', 'any', false})
+ end
+ _func:format(format)
+end
+
local function upgrade_to_2_2_1()
upgrade_sequence_to_2_2_1()
+ upgrade_func_to_2_2_1()
end
--------------------------------------------------------------------------------
diff --git a/src/box/schema_def.h b/src/box/schema_def.h
index b817b49f6..25f6b3d52 100644
--- a/src/box/schema_def.h
+++ b/src/box/schema_def.h
@@ -163,6 +163,9 @@ enum {
BOX_FUNC_FIELD_NAME = 2,
BOX_FUNC_FIELD_SETUID = 3,
BOX_FUNC_FIELD_LANGUAGE = 4,
+ BOX_FUNC_FIELD_BODY = 5,
+ BOX_FUNC_FIELD_RETURNS = 6,
+ BOX_FUNC_FIELD_IS_DETERMINISTIC = 7,
};
/** _collation fields. */
diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result
index 0684914c0..84c03c6fe 100644
--- a/test/box-py/bootstrap.result
+++ b/test/box-py/bootstrap.result
@@ -49,7 +49,9 @@ box.space._space:select{}
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'returns', 'type': 'string'}, {'name': 'is_deterministic',
+ 'type': 'boolean'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -142,7 +144,7 @@ box.space._user:select{}
...
box.space._func:select{}
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', 'any', false]
...
box.space._priv:select{}
---
diff --git a/test/box/access_misc.result b/test/box/access_misc.result
index 24bdd9d63..3c8318b0c 100644
--- a/test/box/access_misc.result
+++ b/test/box/access_misc.result
@@ -789,7 +789,9 @@ box.space._space:select()
'type': 'string'}, {'name': 'opts', 'type': 'map'}, {'name': 'parts', 'type': 'array'}]]
- [296, 1, '_func', 'memtx', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
- 'type': 'unsigned'}]]
+ 'type': 'unsigned'}, {'name': 'language', 'type': 'string'}, {'name': 'body',
+ 'type': 'string'}, {'name': 'returns', 'type': 'string'}, {'name': 'is_deterministic',
+ 'type': 'boolean'}]]
- [297, 1, '_vfunc', 'sysview', 0, {}, [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner',
'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'setuid',
'type': 'unsigned'}]]
@@ -822,7 +824,7 @@ box.space._space:select()
...
box.space._func:select()
---
-- - [1, 1, 'box.schema.user.info', 1, 'LUA']
+- - [1, 1, 'box.schema.user.info', 1, 'LUA', '', 'any', false]
...
session = nil
---
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 4/8] box: load persistent Lua functions on creation
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (2 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 3/8] schema: rework _func system space format Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-31 8:16 ` [tarantool-patches] " Konstantin Osipov
2019-05-30 10:45 ` [PATCH v1 5/8] netbox: call persistent functions in netbox Kirill Shcherbatov
` (6 subsequent siblings)
10 siblings, 1 reply; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
This patch proceed persistent Lua function load on function
object creation.
Each persistent function may use limited amount of Lua functions
and modules:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8
Global variables are forbidden in persistent Lua functions.
Part of #4182
Needed for #1260
---
src/box/func.c | 69 ++++++++++++++++++++++
src/box/func.h | 5 ++
src/box/func_def.h | 6 ++
src/lua/utils.c | 67 ++++++++++++++++++++++
src/lua/utils.h | 8 +++
test/box/persistent_func.result | 95 +++++++++++++++++++++++++++++++
test/box/persistent_func.test.lua | 47 +++++++++++++++
7 files changed, 297 insertions(+)
create mode 100644 test/box/persistent_func.result
create mode 100644 test/box/persistent_func.test.lua
diff --git a/src/box/func.c b/src/box/func.c
index f7465be7e..31db7b477 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -34,6 +34,7 @@
#include "lua/utils.h"
#include "error.h"
#include "diag.h"
+#include "fiber.h"
#include <dlfcn.h>
/**
@@ -355,6 +356,63 @@ restore:
return -1;
}
+/*
+ * Assemble a Lua function object using luaL_loadstring of the
+ * special 'return FUNCTION_BODY' expression and calling it.
+ * Set default sandbox to make function use only a limited number
+ * of functions and modules.
+ */
+static int
+execute_func_lua_load(struct lua_State *L)
+{
+ struct func *func = (struct func *) lua_topointer(L, 1);
+ assert(func->lua_func_ref == LUA_REFNIL);
+ lua_settop(L, 0);
+
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+ const char *load_pref = "return ";
+ uint32_t load_str_sz = strlen(load_pref) + strlen(func->def->body) + 1;
+ char *load_str = region_alloc(region, load_str_sz);
+ if (load_str == NULL) {
+ diag_set(OutOfMemory, load_str_sz, "region", "load_str");
+ goto error;
+ }
+ sprintf(load_str, "%s%s", load_pref, func->def->body);
+ if (luaL_loadstring(L, load_str) != 0 ||
+ luaT_call(L, 0, 1) != 0 || !lua_isfunction(L, -1) ||
+ luaT_get_sandbox(L) != 0) {
+ diag_set(ClientError, ER_LOAD_FUNCTION, func->def->name,
+ func->def->body);
+ goto error;
+ }
+ lua_setfenv(L, -2);
+ func->lua_func_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ region_truncate(region, region_svp);
+ return lua_gettop(L);
+error:
+ region_truncate(region, region_svp);
+ return luaT_error(L);
+}
+
+/**
+ * Perform persistent Lua function loading for given function
+ * object.
+ * Returns 0 in case of success, -1 otherwise and sets the diag
+ * message.
+ */
+static int
+func_lua_load(struct func *func)
+{
+ lua_State *L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_pushcfunction(L, execute_func_lua_load);
+ lua_pushlightuserdata(L, func);
+ int rc = luaT_call(L, 1, 1);
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, coro_ref);
+ return rc;
+}
+
struct func *
func_new(struct func_def *def)
{
@@ -380,6 +438,14 @@ func_new(struct func_def *def)
func->owner_credentials.auth_token = BOX_USER_MAX; /* invalid value */
func->func = NULL;
func->module = NULL;
+ func->lua_func_ref = LUA_REFNIL;
+ if (func_def_is_persistent(func->def)) {
+ if (func_lua_load(func) != 0) {
+ free(func);
+ return NULL;
+ }
+ assert(func->lua_func_ref != LUA_REFNIL);
+ }
return func;
}
@@ -395,8 +461,11 @@ func_unload(struct func *func)
}
module_gc(func->module);
}
+ if (func->lua_func_ref != LUA_REFNIL)
+ luaL_unref(tarantool_L, LUA_REGISTRYINDEX, func->lua_func_ref);
func->module = NULL;
func->func = NULL;
+ func->lua_func_ref = LUA_REFNIL;
}
/**
diff --git a/src/box/func.h b/src/box/func.h
index a4a758b58..7c3e81c51 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -65,6 +65,11 @@ struct func {
* Anchor for module membership.
*/
struct rlist item;
+ /**
+ * The reference index of Lua function object.
+ * Is equal to LUA_REFNIL when undefined.
+ */
+ int lua_func_ref;
/**
* For C functions, the body of the function.
*/
diff --git a/src/box/func_def.h b/src/box/func_def.h
index 78fef9d22..2cbaddd1a 100644
--- a/src/box/func_def.h
+++ b/src/box/func_def.h
@@ -87,6 +87,12 @@ func_def_sizeof(uint32_t name_len, uint32_t body_len)
return sz;
}
+static inline bool
+func_def_is_persistent(struct func_def *def)
+{
+ return def->body != NULL;
+}
+
/**
* API of C stored function.
*/
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 27ff6b396..0d1cca423 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -46,6 +46,14 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
uint32_t CTID_CHAR_PTR;
uint32_t CTID_CONST_CHAR_PTR;
+static const char *default_sandbox_exports[] =
+ {"assert", "error", "ipairs", "math", "next", "pairs", "pcall",
+ "print", "select", "string", "table", "tonumber", "tostring",
+ "type", "unpack", "xpcall", "utf8"};
+
+static int luaL_deepcopy_func_ref = LUA_REFNIL;
+static int luaL_default_sandbox_ref = LUA_REFNIL;
+
void *
luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
{
@@ -1248,6 +1256,65 @@ luaT_func_find(struct lua_State *L, const char *name, const char *name_end,
return 0;
}
+/**
+ * Assemble a new sandbox with given exports table on top of the
+ * Lua stack. All modules in exports list are copying deeply
+ * to ensure the immutablility of this system object.
+ */
+static int
+luaT_prepare_sandbox(struct lua_State *L, const char *exports[],
+ uint32_t exports_count)
+{
+ assert(luaL_deepcopy_func_ref != LUA_REFNIL);
+ lua_createtable(L, exports_count, 0);
+ for (unsigned i = 0; i < exports_count; i++) {
+ int count;
+ uint32_t name_len = strlen(exports[i]);
+ if (luaT_func_find(L, exports[i], exports[i] + name_len,
+ &count) != 0)
+ return -1;
+ switch (lua_type(L, -1)) {
+ case LUA_TTABLE:
+ lua_rawgeti(L, LUA_REGISTRYINDEX,
+ luaL_deepcopy_func_ref);
+ lua_insert(L, -2);
+ lua_call(L, 1, LUA_MULTRET);
+ FALLTHROUGH;
+ case LUA_TFUNCTION:
+ break;
+ default:
+ unreachable();
+ }
+ lua_setfield(L, -2, exports[i]);
+ }
+ return 0;
+}
+
+int
+luaT_get_sandbox(struct lua_State *L)
+{
+ if (luaL_deepcopy_func_ref == LUA_REFNIL) {
+ int count;
+ const char *deepcopy = "table.deepcopy";
+ if (luaT_func_find(L, deepcopy, deepcopy + strlen(deepcopy),
+ &count) != 0)
+ return -1;
+ luaL_deepcopy_func_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ assert(luaL_deepcopy_func_ref != LUA_REFNIL);
+ }
+ if (luaL_default_sandbox_ref == LUA_REFNIL) {
+ if (luaT_prepare_sandbox(L, default_sandbox_exports,
+ nelem(default_sandbox_exports)) != 0)
+ return -1;
+ luaL_default_sandbox_ref = luaL_ref(L, LUA_REGISTRYINDEX);
+ assert(luaL_default_sandbox_ref != LUA_REFNIL);
+ }
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_deepcopy_func_ref);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, luaL_default_sandbox_ref);
+ lua_call(L, 1, LUA_MULTRET);
+ return 0;
+}
+
int
tarantool_lua_utils_init(struct lua_State *L)
{
diff --git a/src/lua/utils.h b/src/lua/utils.h
index 81e936bee..36bb2e53b 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -603,6 +603,14 @@ int
luaT_func_find(struct lua_State *L, const char *name, const char *name_end,
int *count);
+/**
+ * Prepare a new 'default' sandbox table on the top of the Lua
+ * stack. The 'default' sandbox consists of a minimum set of
+ * functions that are sufficient to serve persistent functions.
+ */
+int
+luaT_get_sandbox(struct lua_State *L);
+
int
tarantool_lua_utils_init(struct lua_State *L);
diff --git a/test/box/persistent_func.result b/test/box/persistent_func.result
new file mode 100644
index 000000000..0644de7fe
--- /dev/null
+++ b/test/box/persistent_func.result
@@ -0,0 +1,95 @@
+env = require('test_run')
+---
+...
+test_run = env.new()
+---
+...
+--
+-- gh-4182: Add persistent LUA functions.
+--
+-- Test valid function.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then return nil, 'Invalid field type' end
+ local t = tuple.address:lower():split()
+ for k,v in pairs(t) do t[k] = {v} end
+ return t
+end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('test', {body = body, language = "C"})
+---
+- error: 'Failed to create function ''test'': function body may be specified only
+ for Lua language'
+...
+box.schema.func.create('test', {body = body})
+---
+...
+box.schema.func.exists('test')
+---
+- true
+...
+box.schema.func.create('test2', {body = body, is_deterministic = true})
+---
+...
+-- Test function with spell error - case 1.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body_bad2 = [[function(tuple)
+ ret tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('body_bad2', {body = body_bad2})
+---
+- error: "Failed to dynamically load function 'body_bad2': function(tuple) \tret tuple
+ end "
+...
+-- Test function with spell error - case 2.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body_bad3 = [[func(tuple)
+ return tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('body_bad3', {body = body_bad3})
+---
+- error: "Failed to dynamically load function 'body_bad3': func(tuple) \treturn tuple
+ end "
+...
+-- Restart server.
+test_run:cmd("restart server default")
+net = require('net.box')
+---
+...
+test_run = require('test_run').new()
+---
+...
+box.schema.func.exists('test')
+---
+- true
+...
+box.schema.func.drop('test')
+---
+...
+box.schema.func.exists('test')
+---
+- false
+...
+box.schema.func.drop('test2')
+---
+...
diff --git a/test/box/persistent_func.test.lua b/test/box/persistent_func.test.lua
new file mode 100644
index 000000000..37a761d32
--- /dev/null
+++ b/test/box/persistent_func.test.lua
@@ -0,0 +1,47 @@
+env = require('test_run')
+test_run = env.new()
+
+--
+-- gh-4182: Add persistent LUA functions.
+--
+-- Test valid function.
+test_run:cmd("setopt delimiter ';'")
+body = [[function(tuple)
+ if type(tuple.address) ~= 'string' then return nil, 'Invalid field type' end
+ local t = tuple.address:lower():split()
+ for k,v in pairs(t) do t[k] = {v} end
+ return t
+end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('test', {body = body, language = "C"})
+box.schema.func.create('test', {body = body})
+box.schema.func.exists('test')
+box.schema.func.create('test2', {body = body, is_deterministic = true})
+
+-- Test function with spell error - case 1.
+test_run:cmd("setopt delimiter ';'")
+body_bad2 = [[function(tuple)
+ ret tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('body_bad2', {body = body_bad2})
+
+-- Test function with spell error - case 2.
+test_run:cmd("setopt delimiter ';'")
+body_bad3 = [[func(tuple)
+ return tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('body_bad3', {body = body_bad3})
+
+-- Restart server.
+test_run:cmd("restart server default")
+net = require('net.box')
+test_run = require('test_run').new()
+box.schema.func.exists('test')
+box.schema.func.drop('test')
+box.schema.func.exists('test')
+box.schema.func.drop('test2')
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 5/8] netbox: call persistent functions in netbox
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (3 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 4/8] box: load persistent Lua functions on creation Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 6/8] box: export _func functions with box.func folder Kirill Shcherbatov
` (5 subsequent siblings)
10 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
This patch makes persistent Lua functions are available via
net.box.connect() :call method.
Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {body = lua_code,
opts = {is_deterministic = true}})
conn:call("sum", {1, 3})
Part of #4182
Needed for #1260
---
src/box/call.c | 2 +-
src/box/lua/call.c | 36 ++++++++++++-
src/box/lua/call.h | 3 +-
test/box/persistent_func.result | 84 +++++++++++++++++++++++++++++++
test/box/persistent_func.test.lua | 36 +++++++++++++
5 files changed, 157 insertions(+), 4 deletions(-)
diff --git a/src/box/call.c b/src/box/call.c
index 56da53fb3..773b914b1 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -202,7 +202,7 @@ box_process_call(struct call_request *request, struct port *port)
if (func && func->def->language == FUNC_LANGUAGE_C) {
rc = box_c_call(func, request, port);
} else {
- rc = box_lua_call(request, port);
+ rc = box_lua_call(func, request, port);
}
/* Restore the original user */
if (orig_credentials)
diff --git a/src/box/lua/call.c b/src/box/lua/call.c
index c729778c4..2cd982ba8 100644
--- a/src/box/lua/call.c
+++ b/src/box/lua/call.c
@@ -32,6 +32,8 @@
#include "box/call.h"
#include "box/error.h"
#include "fiber.h"
+#include "box/func.h"
+#include "box/schema.h"
#include "lua/utils.h"
#include "lua/msgpack.h"
@@ -253,6 +255,31 @@ execute_lua_call(lua_State *L)
return lua_gettop(L);
}
+static int
+execute_persistent_function(lua_State *L)
+{
+ struct call_request *request = (struct call_request *)
+ lua_topointer(L, 1);
+ lua_settop(L, 0);
+
+ const char *name = request->name;
+ uint32_t name_len = mp_decode_strl(&name);
+ struct func *func = func_by_name(name, name_len);
+ assert(func != NULL);
+ assert(func->lua_func_ref != LUA_REFNIL);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, func->lua_func_ref);
+
+ /* Push the rest of args (a tuple). */
+ const char *args = request->args;
+ uint32_t arg_count = mp_decode_array(&args);
+ luaL_checkstack(L, arg_count, "lua_func: out of stack");
+ for (uint32_t i = 0; i < arg_count; i++)
+ luamp_decode(L, luaL_msgpack_default, &args);
+
+ lua_call(L, arg_count, LUA_MULTRET);
+ return lua_gettop(L);
+}
+
static int
execute_lua_eval(lua_State *L)
{
@@ -396,9 +423,14 @@ box_process_lua(struct call_request *request, struct port *base,
}
int
-box_lua_call(struct call_request *request, struct port *port)
+box_lua_call(struct func *func, struct call_request *request, struct port *port)
{
- return box_process_lua(request, port, execute_lua_call);
+ if (func != NULL && func_def_is_persistent(func->def)) {
+ return box_process_lua(request, port,
+ execute_persistent_function);
+ } else {
+ return box_process_lua(request, port, execute_lua_call);
+ }
}
int
diff --git a/src/box/lua/call.h b/src/box/lua/call.h
index 0542123da..06bcfe77e 100644
--- a/src/box/lua/call.h
+++ b/src/box/lua/call.h
@@ -42,13 +42,14 @@ box_lua_call_init(struct lua_State *L);
struct port;
struct call_request;
+struct func;
/**
* Invoke a Lua stored procedure from the binary protocol
* (implementation of 'CALL' command code).
*/
int
-box_lua_call(struct call_request *request, struct port *port);
+box_lua_call(struct func *func, struct call_request *request, struct port *port);
int
box_lua_eval(struct call_request *request, struct port *port);
diff --git a/test/box/persistent_func.result b/test/box/persistent_func.result
index 0644de7fe..1b3eaa8b2 100644
--- a/test/box/persistent_func.result
+++ b/test/box/persistent_func.result
@@ -7,6 +7,15 @@ test_run = env.new()
--
-- gh-4182: Add persistent LUA functions.
--
+box.schema.user.grant('guest', 'execute', 'universe')
+---
+...
+net = require('net.box')
+---
+...
+conn = net.connect(box.cfg.listen)
+---
+...
-- Test valid function.
test_run:cmd("setopt delimiter ';'")
---
@@ -34,9 +43,58 @@ box.schema.func.exists('test')
---
- true
...
+conn:call("test", {{address = "Moscow Dolgoprudny"}})
+---
+- [['moscow'], ['dolgoprudny']]
+...
box.schema.func.create('test2', {body = body, is_deterministic = true})
---
...
+-- Test that monkey-patch attack is not possible.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body_monkey = [[function(tuple)
+ math.abs = math.log
+ return tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('body_monkey', {body = body_monkey})
+---
+...
+conn:call("body_monkey", {{address = "Moscow Dolgoprudny"}})
+---
+- {'address': 'Moscow Dolgoprudny'}
+...
+math.abs(-666.666)
+---
+- 666.666
+...
+-- Test taht 'require' is forbidden.
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+body_bad1 = [[function(tuple)
+ local json = require('json')
+ return json.encode(tuple)
+end
+]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('json_serializer', {body = body_bad1})
+---
+...
+conn:call("json_serializer", {{address = "Moscow Dolgoprudny"}})
+---
+- error: '[string "return function(tuple) ..."]:1: attempt to call global ''require''
+ (a nil value)'
+...
-- Test function with spell error - case 1.
test_run:cmd("setopt delimiter ';'")
---
@@ -71,6 +129,13 @@ box.schema.func.create('body_bad3', {body = body_bad3})
- error: "Failed to dynamically load function 'body_bad3': func(tuple) \treturn tuple
end "
...
+conn:call("body_bad3", {{address = "Moscow Dolgoprudny"}})
+---
+- error: Procedure 'body_bad3' is not defined
+...
+conn:close()
+---
+...
-- Restart server.
test_run:cmd("restart server default")
net = require('net.box')
@@ -79,6 +144,16 @@ net = require('net.box')
test_run = require('test_run').new()
---
...
+conn = net.connect(box.cfg.listen)
+---
+...
+conn:call("test", {{address = "Moscow Dolgoprudny"}})
+---
+- [['moscow'], ['dolgoprudny']]
+...
+conn:close()
+---
+...
box.schema.func.exists('test')
---
- true
@@ -90,6 +165,15 @@ box.schema.func.exists('test')
---
- false
...
+box.schema.func.drop('body_monkey')
+---
+...
+box.schema.func.drop('json_serializer')
+---
+...
box.schema.func.drop('test2')
---
...
+box.schema.user.revoke('guest', 'execute', 'universe')
+---
+...
diff --git a/test/box/persistent_func.test.lua b/test/box/persistent_func.test.lua
index 37a761d32..363d687ce 100644
--- a/test/box/persistent_func.test.lua
+++ b/test/box/persistent_func.test.lua
@@ -4,6 +4,10 @@ test_run = env.new()
--
-- gh-4182: Add persistent LUA functions.
--
+box.schema.user.grant('guest', 'execute', 'universe')
+net = require('net.box')
+conn = net.connect(box.cfg.listen)
+
-- Test valid function.
test_run:cmd("setopt delimiter ';'")
body = [[function(tuple)
@@ -17,8 +21,32 @@ test_run:cmd("setopt delimiter ''");
box.schema.func.create('test', {body = body, language = "C"})
box.schema.func.create('test', {body = body})
box.schema.func.exists('test')
+conn:call("test", {{address = "Moscow Dolgoprudny"}})
box.schema.func.create('test2', {body = body, is_deterministic = true})
+-- Test that monkey-patch attack is not possible.
+test_run:cmd("setopt delimiter ';'")
+body_monkey = [[function(tuple)
+ math.abs = math.log
+ return tuple
+end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('body_monkey', {body = body_monkey})
+conn:call("body_monkey", {{address = "Moscow Dolgoprudny"}})
+math.abs(-666.666)
+
+-- Test taht 'require' is forbidden.
+test_run:cmd("setopt delimiter ';'")
+body_bad1 = [[function(tuple)
+ local json = require('json')
+ return json.encode(tuple)
+end
+]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('json_serializer', {body = body_bad1})
+conn:call("json_serializer", {{address = "Moscow Dolgoprudny"}})
+
-- Test function with spell error - case 1.
test_run:cmd("setopt delimiter ';'")
body_bad2 = [[function(tuple)
@@ -36,12 +64,20 @@ end
]]
test_run:cmd("setopt delimiter ''");
box.schema.func.create('body_bad3', {body = body_bad3})
+conn:call("body_bad3", {{address = "Moscow Dolgoprudny"}})
+conn:close()
-- Restart server.
test_run:cmd("restart server default")
net = require('net.box')
test_run = require('test_run').new()
+conn = net.connect(box.cfg.listen)
+conn:call("test", {{address = "Moscow Dolgoprudny"}})
+conn:close()
box.schema.func.exists('test')
box.schema.func.drop('test')
box.schema.func.exists('test')
+box.schema.func.drop('body_monkey')
+box.schema.func.drop('json_serializer')
box.schema.func.drop('test2')
+box.schema.user.revoke('guest', 'execute', 'universe')
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 6/8] box: export _func functions with box.func folder
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (4 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 5/8] netbox: call persistent functions in netbox Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-06-03 16:22 ` Vladimir Davydov
2019-06-03 16:24 ` Vladimir Davydov
2019-05-30 10:45 ` [PATCH v1 7/8] box: introduce memtx_slab_alloc helper Kirill Shcherbatov
` (4 subsequent siblings)
10 siblings, 2 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Closes #4182
Needed for #1260
@TarantoolBot document
Title: Persistent Lua functions
Lua functions managed by Tarantool are called 'persistent'.
They are stored in snapshoot and are available after server
restart.
Persistent Lua functions are exported in box.schema.func.persistent
folder. They may be called via .call exported object method or
via netbox:call.
Some restrictions must be accounted writing such routines:
1. only limited set of Lua functions and modules are available:
-assert -error -pairs -ipairs -next -pcall -xpcall -type
-print -select -string -tonumber -tostring -unpack -math -utf8;
2. global variables are forbidden.
Moreover all functions and their metadata are now exported in
box.func table (not only persistent). Persistent Lua functions
have box.func.FUNCNAME.call() method to execute it in interactive
console.
Persistent Lua functions are also available with net.box call.
Example:
lua_code = [[function(a, b) return a + b end]]
box.schema.func.create('sum', {
body = lua_code, is_deterministic = true,
returns = 'integer', is_deterministic = true})
box.func.sum
- call: 'function: 0x4016b6a0'
returns: integer
is_deterministic: true
id: 2
language: LUA
name: sum
is_persistent: true
setuid: false
box.func.sum.call(1, 2)
- 3
-- netbox connection call
conn:call("sum", {1, 3})
- 3
---
src/box/alter.cc | 6 +-
src/box/func.c | 119 ++++++++++++++++++++++++++++++
src/box/func.h | 6 ++
src/box/lua/init.c | 2 +
src/box/lua/schema.lua | 1 +
src/box/schema.cc | 1 +
src/box/schema.h | 16 ++--
test/box/misc.result | 1 +
test/box/persistent_func.result | 22 ++++++
test/box/persistent_func.test.lua | 5 ++
10 files changed, 173 insertions(+), 6 deletions(-)
diff --git a/src/box/alter.cc b/src/box/alter.cc
index c9446bfe0..28644a667 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -2507,7 +2507,9 @@ static void
func_cache_remove_func(struct trigger *trigger, void * /* event */)
{
struct func *old_func = (struct func *) trigger->data;
- func_cache_delete(old_func->def->fid);
+ uint32_t fid = old_func->def->fid;
+ func_cache_delete(fid);
+ trigger_run_xc(&on_alter_func, (void *)(uintptr_t)fid);
func_delete(old_func);
}
@@ -2520,6 +2522,7 @@ func_cache_replace_func(struct trigger *trigger, void * /* event */)
func_cache_replace(new_func, &old_func);
assert(old_func != NULL);
func_delete(old_func);
+ trigger_run_xc(&on_alter_func, (void *)(uintptr_t)new_func->def->fid);
}
/**
@@ -2552,6 +2555,7 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
func_cache_replace(func, &old_func);
assert(old_func == NULL);
func_guard.is_active = false;
+ trigger_run_xc(&on_alter_func, (void *)(uintptr_t)fid);
struct trigger *on_rollback =
txn_alter_trigger_new(func_cache_remove_func, func);
txn_on_rollback(txn, on_rollback);
diff --git a/src/box/func.c b/src/box/func.c
index 31db7b477..b5b68daac 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -31,10 +31,12 @@
#include "func.h"
#include "trivia/config.h"
#include "assoc.h"
+#include "lua/trigger.h"
#include "lua/utils.h"
#include "error.h"
#include "diag.h"
#include "fiber.h"
+#include "schema.h"
#include <dlfcn.h>
/**
@@ -536,3 +538,120 @@ func_capture_module(struct func *new_func, struct func *old_func)
old_func->module = NULL;
old_func->func = NULL;
}
+
+static void
+box_lua_func_new(struct lua_State *L, struct func *func)
+{
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ if (!lua_istable(L, -1)) {
+ lua_pop(L, 1); /* pop nil */
+ lua_newtable(L);
+ lua_setfield(L, -2, "func");
+ lua_getfield(L, -1, "func");
+ }
+ lua_rawgeti(L, -1, func->def->fid);
+ if (lua_isnil(L, -1)) {
+ /*
+ * If the function already exists, modify it,
+ * rather than create a new one -- to not
+ * invalidate Lua variable references to old func
+ * outside the box.schema.func[].
+ */
+ lua_pop(L, 1);
+ lua_newtable(L);
+ lua_rawseti(L, -2, func->def->fid);
+ lua_rawgeti(L, -1, func->def->fid);
+ } else {
+ /* Clear the reference to old space by old name. */
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_settable(L, -4);
+ }
+
+ int top = lua_gettop(L);
+ lua_pushstring(L, "id");
+ lua_pushnumber(L, func->def->fid);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "name");
+ lua_pushstring(L, func->def->name);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "is_persistent");
+ lua_pushboolean(L, func_def_is_persistent(func->def));
+ lua_settable(L, top);
+
+ lua_pushstring(L, "setuid");
+ lua_pushboolean(L, func->def->setuid);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "is_deterministic");
+ lua_pushboolean(L, func->def->is_deterministic);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "language");
+ lua_pushstring(L, func_language_strs[func->def->language]);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "returns");
+ lua_pushstring(L, field_type_strs[func->def->returns]);
+ lua_settable(L, top);
+
+ lua_pushstring(L, "call");
+ lua_rawgeti(L, LUA_REGISTRYINDEX, func->lua_func_ref);
+ lua_settable(L, top);
+
+ lua_setfield(L, -2, func->def->name);
+
+ lua_pop(L, 2); /* box, func */
+}
+
+
+static void
+box_lua_func_delete(struct lua_State *L, uint32_t fid)
+{
+ lua_getfield(L, LUA_GLOBALSINDEX, "box");
+ lua_getfield(L, -1, "func");
+ lua_rawgeti(L, -1, fid);
+ if (!lua_isnil(L, -1)) {
+ lua_getfield(L, -1, "name");
+ lua_pushnil(L);
+ lua_rawset(L, -4);
+ lua_pop(L, 1); /* pop func */
+
+ lua_pushnil(L);
+ lua_rawseti(L, -2, fid);
+ } else {
+ lua_pop(L, 1);
+ }
+ lua_pop(L, 2); /* box, func */
+}
+
+static void
+box_lua_func_new_or_delete(struct trigger *trigger, void *event)
+{
+ struct lua_State *L = (struct lua_State *) trigger->data;
+ uint32_t fid = (uint32_t)(uintptr_t)event;
+ struct func *func = func_by_id(fid);
+ /* Export only persistent Lua functions. */
+ if (func != NULL)
+ box_lua_func_new(L, func);
+ else
+ box_lua_func_delete(L, fid);
+}
+
+static struct trigger on_alter_func_in_lua = {
+ RLIST_LINK_INITIALIZER, box_lua_func_new_or_delete, NULL, NULL
+};
+
+void
+box_lua_func_init(struct lua_State *L)
+{
+ /*
+ * Register the trigger that will push persistent
+ * Lua functions objects to Lua.
+ */
+ on_alter_func_in_lua.data = L;
+ trigger_add(&on_alter_func, &on_alter_func_in_lua);
+}
diff --git a/src/box/func.h b/src/box/func.h
index 7c3e81c51..3dfabbf40 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -136,6 +136,12 @@ func_capture_module(struct func *new_func, struct func *old_func);
int
module_reload(const char *package, const char *package_end, struct module **module);
+struct lua_State;
+
+/** Initialize Lua export trigger for _func objects. */
+void
+box_lua_func_init(struct lua_State *L);
+
#if defined(__cplusplus)
} /* extern "C" */
#endif /* defined(__cplusplus) */
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index 76b987b4b..67a822eb4 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -39,6 +39,7 @@
#include "box/box.h"
#include "box/txn.h"
+#include "box/func.h"
#include "box/vclock.h"
#include "box/lua/error.h"
@@ -315,6 +316,7 @@ box_lua_init(struct lua_State *L)
box_lua_xlog_init(L);
box_lua_execute_init(L);
luaopen_net_box(L);
+ box_lua_func_init(L);
lua_pop(L, 1);
tarantool_lua_console_init(L);
lua_pop(L, 1);
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 960ea0fc7..2599a24a3 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -2101,6 +2101,7 @@ local function object_name(object_type, object_id)
end
box.schema.func = {}
+box.schema.func.persistent = {}
box.schema.func.create = function(name, opts)
opts = opts or {}
check_param_table(opts, { setuid = 'boolean',
diff --git a/src/box/schema.cc b/src/box/schema.cc
index b70992297..7f6086210 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -72,6 +72,7 @@ uint32_t space_cache_version = 0;
struct rlist on_schema_init = RLIST_HEAD_INITIALIZER(on_schema_init);
struct rlist on_alter_space = RLIST_HEAD_INITIALIZER(on_alter_space);
struct rlist on_alter_sequence = RLIST_HEAD_INITIALIZER(on_alter_sequence);
+struct rlist on_alter_func = RLIST_HEAD_INITIALIZER(on_alter_func);
/**
* Lock of scheme modification
diff --git a/src/box/schema.h b/src/box/schema.h
index ffc41f401..78d48d3da 100644
--- a/src/box/schema.h
+++ b/src/box/schema.h
@@ -130,6 +130,11 @@ int
schema_find_id(uint32_t system_space_id, uint32_t index_id, const char *name,
uint32_t len, uint32_t *object_id);
+struct func;
+
+struct func *
+func_by_id(uint32_t fid);
+
#if defined(__cplusplus)
} /* extern "C" */
@@ -178,11 +183,6 @@ func_cache_replace(struct func *new_func, struct func **old_func);
void
func_cache_delete(uint32_t fid);
-struct func;
-
-struct func *
-func_by_id(uint32_t fid);
-
static inline struct func *
func_cache_find(uint32_t fid)
{
@@ -243,6 +243,12 @@ extern struct rlist on_alter_sequence;
*/
extern struct rlist on_access_denied;
+/**
+ * Triggers fired after committing a change in _func space.
+ * It is passed the txn statement that altered the space.
+ */
+extern struct rlist on_alter_func;
+
/**
* Context passed to on_access_denied trigger.
*/
diff --git a/test/box/misc.result b/test/box/misc.result
index 4fcd13a78..ee51283bb 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -65,6 +65,7 @@ t
- error
- execute
- feedback
+ - func
- index
- info
- internal
diff --git a/test/box/persistent_func.result b/test/box/persistent_func.result
index 1b3eaa8b2..c18be4a3b 100644
--- a/test/box/persistent_func.result
+++ b/test/box/persistent_func.result
@@ -43,6 +43,15 @@ box.schema.func.exists('test')
---
- true
...
+box.func.test ~= nil
+---
+- true
+...
+box.func.test.call({address = "Moscow Dolgoprudny"})
+---
+- - - moscow
+ - - dolgoprudny
+...
conn:call("test", {{address = "Moscow Dolgoprudny"}})
---
- [['moscow'], ['dolgoprudny']]
@@ -147,6 +156,15 @@ test_run = require('test_run').new()
conn = net.connect(box.cfg.listen)
---
...
+box.func.test ~= nil
+---
+- true
+...
+box.func.test.call({address = "Moscow Dolgoprudny"})
+---
+- - - moscow
+ - - dolgoprudny
+...
conn:call("test", {{address = "Moscow Dolgoprudny"}})
---
- [['moscow'], ['dolgoprudny']]
@@ -161,6 +179,10 @@ box.schema.func.exists('test')
box.schema.func.drop('test')
---
...
+box.func.test ~= nil
+---
+- false
+...
box.schema.func.exists('test')
---
- false
diff --git a/test/box/persistent_func.test.lua b/test/box/persistent_func.test.lua
index 363d687ce..048afa815 100644
--- a/test/box/persistent_func.test.lua
+++ b/test/box/persistent_func.test.lua
@@ -21,6 +21,8 @@ test_run:cmd("setopt delimiter ''");
box.schema.func.create('test', {body = body, language = "C"})
box.schema.func.create('test', {body = body})
box.schema.func.exists('test')
+box.func.test ~= nil
+box.func.test.call({address = "Moscow Dolgoprudny"})
conn:call("test", {{address = "Moscow Dolgoprudny"}})
box.schema.func.create('test2', {body = body, is_deterministic = true})
@@ -72,10 +74,13 @@ test_run:cmd("restart server default")
net = require('net.box')
test_run = require('test_run').new()
conn = net.connect(box.cfg.listen)
+box.func.test ~= nil
+box.func.test.call({address = "Moscow Dolgoprudny"})
conn:call("test", {{address = "Moscow Dolgoprudny"}})
conn:close()
box.schema.func.exists('test')
box.schema.func.drop('test')
+box.func.test ~= nil
box.schema.func.exists('test')
box.schema.func.drop('body_monkey')
box.schema.func.drop('json_serializer')
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 7/8] box: introduce memtx_slab_alloc helper
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (5 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 6/8] box: export _func functions with box.func folder Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 8/8] box: introduce functional indexes in memtx Kirill Shcherbatov
` (3 subsequent siblings)
10 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
The new memtx_slab_alloc routine allocates a memory using memtx
engine slab allocator. Previously this code used to be call only
to allocate a new tuple for memtx engine. In further patches we
are going to allocate the keys returned by functional index
extractor routine so we need this function in engine module API.
The pair call memtx_slab_free is also exported.
Needed for #1260
---
src/box/memtx_engine.c | 41 +++++++++++++++++++++++++++--------------
src/box/memtx_engine.h | 19 +++++++++++++++++++
2 files changed, 46 insertions(+), 14 deletions(-)
diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c
index f4312484a..8faa9fa57 100644
--- a/src/box/memtx_engine.c
+++ b/src/box/memtx_engine.c
@@ -1127,6 +1127,29 @@ memtx_engine_set_max_tuple_size(struct memtx_engine *memtx, size_t max_size)
memtx->max_tuple_size = max_size;
}
+void *
+memtx_slab_alloc(struct memtx_engine *memtx, size_t size)
+{
+ void *ptr;
+ while ((ptr = smalloc(&memtx->alloc, size)) == NULL) {
+ bool stop;
+ memtx_engine_run_gc(memtx, &stop);
+ if (stop)
+ break;
+ }
+ return ptr;
+}
+
+void
+memtx_slab_free(struct memtx_engine *memtx, void *ptr, size_t size,
+ bool is_not_delayed)
+{
+ if (memtx->alloc.free_mode != SMALL_DELAYED_FREE || is_not_delayed)
+ smfree(&memtx->alloc, ptr, size);
+ else
+ smfree_delayed(&memtx->alloc, ptr, size);
+}
+
struct tuple *
memtx_tuple_new(struct tuple_format *format, const char *data, const char *end)
{
@@ -1152,14 +1175,7 @@ memtx_tuple_new(struct tuple_format *format, const char *data, const char *end)
error_log(diag_last_error(diag_get()));
goto end;
}
-
- struct memtx_tuple *memtx_tuple;
- while ((memtx_tuple = smalloc(&memtx->alloc, total)) == NULL) {
- bool stop;
- memtx_engine_run_gc(memtx, &stop);
- if (stop)
- break;
- }
+ struct memtx_tuple *memtx_tuple = memtx_slab_alloc(memtx, total);
if (memtx_tuple == NULL) {
diag_set(OutOfMemory, total, "slab allocator", "memtx_tuple");
goto end;
@@ -1196,12 +1212,9 @@ memtx_tuple_delete(struct tuple_format *format, struct tuple *tuple)
struct memtx_tuple *memtx_tuple =
container_of(tuple, struct memtx_tuple, base);
size_t total = tuple_size(tuple) + offsetof(struct memtx_tuple, base);
- if (memtx->alloc.free_mode != SMALL_DELAYED_FREE ||
- memtx_tuple->version == memtx->snapshot_version ||
- format->is_temporary)
- smfree(&memtx->alloc, memtx_tuple, total);
- else
- smfree_delayed(&memtx->alloc, memtx_tuple, total);
+ memtx_slab_free(memtx, memtx_tuple, total,
+ memtx_tuple->version == memtx->snapshot_version ||
+ format->is_temporary);
}
struct tuple_format_vtab memtx_tuple_format_vtab = {
diff --git a/src/box/memtx_engine.h b/src/box/memtx_engine.h
index ccb51678d..60f3a11bf 100644
--- a/src/box/memtx_engine.h
+++ b/src/box/memtx_engine.h
@@ -250,6 +250,25 @@ bool
memtx_index_def_change_requires_rebuild(struct index *index,
const struct index_def *new_def);
+/**
+ * Allocate size bytes using memtx engine slab allocator and
+ * return a pointer to the allocated memory.
+ * The memory is not initialized.
+ *
+ * Returns not NULL pointer to the allocated memory is case of
+ * success and NULL otherwise.
+ */
+void *
+memtx_slab_alloc(struct memtx_engine *memtx, size_t size);
+
+/**
+ * Release ptr size bytes memory chunk allocated with
+ * memtx_slab_alloc.
+ */
+void
+memtx_slab_free(struct memtx_engine *memtx, void *ptr, size_t size,
+ bool is_not_delayed);
+
#if defined(__cplusplus)
} /* extern "C" */
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH v1 8/8] box: introduce functional indexes in memtx
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (6 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 7/8] box: introduce memtx_slab_alloc helper Kirill Shcherbatov
@ 2019-05-30 10:45 ` Kirill Shcherbatov
2019-05-30 11:18 ` [tarantool-patches] [PATCH v1 0/8] box: functional indexes Vladislav Shpilevoy
` (2 subsequent siblings)
10 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 10:45 UTC (permalink / raw)
To: tarantool-patches, vdavydov.dev; +Cc: Kirill Shcherbatov
Closes #1260
@TarantoolBot document
Title: introduce functional indexes in memtx
Now you can define a functional index using registered persistent
function as key extractor.
There is some restrictions for function and key definition for
functional index:
- referenced extractor function must be persistent, deterministic
and return scalar type or array.
- key parts describe the referenced routine's returned key
(when the routine returned value type is array in meaning of
multikey index key): all parts must be sequential and the
first part's fieldno must be 1.
- key parts can't use JSON paths.
Functional index can't be primary.
You are not allowed to change functional index extractor function
while there are some functional indexes depends of it.
Example:
s = box.schema.space.create('withdata')
s:format({{name = 'name', type = 'string'},
{name = 'address', type = 'string'}})
pk = s:create_index('name', {parts = {1, 'string'}})
lua_code = [[function(tuple)
local address = string.split(tuple.address)
local ret = {}
for _, v in pairs(address) do
table.insert(ret, utf8.upper(v))
end
return ret
end]]
box.schema.func.create('addr_extractor', {body = lua_code,
is_deterministic = true, returns = 'array'})
idx = s:create_index('addr', {unique = false, functional_fid =
box.func.addr_extractor.id, parts = {{1, 'string'}}})
s:insert({"James", "SIS Building Lambeth London UK"})
s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
idx:select('UK')
---
- - ['James', 'SIS Building Lambeth London UK']
- ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
...
idx:select('SIS')
---
- - ['James', 'SIS Building Lambeth London UK']
...
---
src/box/CMakeLists.txt | 1 +
src/box/alter.cc | 67 +++-
src/box/errcode.h | 1 +
src/box/func.c | 2 +
src/box/func.h | 15 +
src/box/index_def.c | 22 +-
src/box/index_def.h | 16 +
src/box/key_def.c | 13 +-
src/box/key_def.h | 31 +-
src/box/lua/key_def.c | 2 +-
src/box/lua/schema.lua | 2 +
src/box/lua/space.cc | 5 +
| 233 ++++++++++++
| 100 +++++
src/box/memtx_engine.c | 3 +
src/box/memtx_space.c | 18 +
src/box/memtx_tree.c | 355 +++++++++++++++++-
src/box/schema.cc | 2 +-
src/box/sql.c | 2 +-
src/box/sql/build.c | 2 +-
src/box/sql/select.c | 2 +-
src/box/sql/where.c | 2 +-
src/box/tuple_compare.cc | 112 +++++-
| 33 +-
src/box/vinyl.c | 9 +-
test/box/bitset.result | 24 ++
test/box/bitset.test.lua | 9 +
test/box/hash.result | 24 ++
test/box/hash.test.lua | 9 +
test/box/misc.result | 1 +
test/box/rtree_misc.result | 24 ++
test/box/rtree_misc.test.lua | 9 +
test/engine/engine.cfg | 5 +-
test/engine/functional.result | 620 +++++++++++++++++++++++++++++++
test/engine/functional.test.lua | 214 +++++++++++
test/unit/luaT_tuple_new.c | 2 +-
test/unit/merger.test.c | 4 +-
37 files changed, 1965 insertions(+), 30 deletions(-)
create mode 100644 src/box/lua/tuple_fextract_key.c
create mode 100644 src/box/lua/tuple_fextract_key.h
create mode 100644 test/engine/functional.result
create mode 100644 test/engine/functional.test.lua
diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
index 0864c3433..ff1bd4c6e 100644
--- a/src/box/CMakeLists.txt
+++ b/src/box/CMakeLists.txt
@@ -146,6 +146,7 @@ add_library(box STATIC
lua/execute.c
lua/key_def.c
lua/merger.c
+ lua/tuple_fextract_key.c
${bin_sources})
target_link_libraries(box box_error tuple stat xrow xlog vclock crc32 scramble
diff --git a/src/box/alter.cc b/src/box/alter.cc
index 28644a667..fef2bed12 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -29,6 +29,7 @@
* SUCH DAMAGE.
*/
#include "alter.h"
+#include "box.h"
#include "column_mask.h"
#include "schema.h"
#include "user.h"
@@ -234,6 +235,44 @@ index_opts_decode(struct index_opts *opts, const char *map,
}
}
+/**
+ * Ensure that given function definition is a valid functional
+ * index extractor routine compatible with given key definition.
+ *
+ * - The key definition must be sequential and started with
+ * fieldno = 0
+ * - The function definition must be Lua deterministic persistent
+ * function that returns scalar type or array.
+ * - When the function return scalar, the key definition must
+ * contain a single part of the same type.
+*/
+static bool
+index_is_valid_functional_definition(struct func_def *func_def,
+ struct key_def *key_def)
+{
+ if (!key_def_is_functional(key_def) ||
+ !key_def_is_sequential(key_def) || key_def->parts->fieldno != 0 ||
+ key_def->has_json_paths)
+ return false;
+
+ if (func_def->language != FUNC_LANGUAGE_LUA ||
+ func_def->body == NULL || !func_def->is_deterministic)
+ return false;
+ enum field_type returns = func_def->returns;
+ if (returns != FIELD_TYPE_ARRAY && returns != FIELD_TYPE_BOOLEAN &&
+ returns != FIELD_TYPE_INTEGER && returns != FIELD_TYPE_UNSIGNED &&
+ returns != FIELD_TYPE_NUMBER && returns != FIELD_TYPE_SCALAR &&
+ returns != FIELD_TYPE_STRING)
+ return false;
+ if (!key_def_fextract_key_is_flat(key_def) &&
+ returns != FIELD_TYPE_ARRAY)
+ return false;
+ if (key_def_fextract_key_is_flat(key_def) &&
+ returns != FIELD_TYPE_ARRAY && returns != key_def->parts->type)
+ return false;
+ return true;
+}
+
/**
* Create a index_def object from a record in _index
* system space.
@@ -245,6 +284,7 @@ index_opts_decode(struct index_opts *opts, const char *map,
* - there are parts for the specified part count
* - types of parts in the parts array are known to the system
* - fieldno of each part in the parts array is within limits
+ * - referenced functional index extractor routine is valid
*/
static struct index_def *
index_def_new_from_tuple(struct tuple *tuple, struct space *space)
@@ -288,9 +328,29 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space)
space->def->fields,
space->def->field_count, &fiber()->gc) != 0)
diag_raise();
- key_def = key_def_new(part_def, part_count);
+ bool is_functional = opts.functional_fid > 0;
+ key_def = key_def_new(part_def, part_count, is_functional);
if (key_def == NULL)
diag_raise();
+ /**
+ * Can't verify functional index extractor routine
+ * reference on load because the function object
+ * had not been registered in Tarantool yet.
+ */
+ if (strcmp(box_status(), "loading") != 0 && is_functional > 0) {
+ struct func *func = func_by_id(opts.functional_fid);
+ if (func == NULL) {
+ tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS,
+ BOX_INDEX_FIELD_OPTS,
+ "referenced function doesn't exist");
+ }
+ if (!index_is_valid_functional_definition(func->def, key_def)) {
+ tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS,
+ BOX_INDEX_FIELD_OPTS,
+ "referenced function doesn't satisfy "
+ "functional index constraints");
+ }
+ }
struct index_def *index_def =
index_def_new(id, index_id, name, name_len, type,
&opts, key_def, space_index_key_def(space, 0));
@@ -2541,6 +2601,11 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event)
uint32_t fid = tuple_field_u32_xc(old_tuple ? old_tuple : new_tuple,
BOX_FUNC_FIELD_ID);
struct func *old_func = func_by_id(fid);
+ if (old_func != NULL && old_func->refs != 0) {
+ tnt_raise(ClientError, ER_DROP_FUNCTION,
+ (unsigned) old_func->def->uid,
+ "function has references");
+ }
if (new_tuple != NULL && old_func == NULL) { /* INSERT */
struct func_def *def = func_def_new_from_tuple(new_tuple);
auto def_guard = make_scoped_guard([=] { free(def); });
diff --git a/src/box/errcode.h b/src/box/errcode.h
index 9c15f3322..0b7d5ce6c 100644
--- a/src/box/errcode.h
+++ b/src/box/errcode.h
@@ -247,6 +247,7 @@ struct errcode_record {
/*192 */_(ER_INDEX_DEF_UNSUPPORTED, "%s are prohibited in an index definition") \
/*193 */_(ER_CK_DEF_UNSUPPORTED, "%s are prohibited in a CHECK constraint definition") \
/*194 */_(ER_MULTIKEY_INDEX_MISMATCH, "Field %s is used as multikey in one index and as single key in another") \
+ /*195 */_(ER_FUNCTIONAL_EXTRACTOR_ERROR,"Functional index extractor error: %s") \
/*
* !IMPORTANT! Please follow instructions at start of the file
diff --git a/src/box/func.c b/src/box/func.c
index b5b68daac..6a3f893bf 100644
--- a/src/box/func.c
+++ b/src/box/func.c
@@ -440,6 +440,7 @@ func_new(struct func_def *def)
func->owner_credentials.auth_token = BOX_USER_MAX; /* invalid value */
func->func = NULL;
func->module = NULL;
+ func->refs = 0;
func->lua_func_ref = LUA_REFNIL;
if (func_def_is_persistent(func->def)) {
if (func_lua_load(func) != 0) {
@@ -525,6 +526,7 @@ func_call(struct func *func, box_function_ctx_t *ctx, const char *args,
void
func_delete(struct func *func)
{
+ assert(func->refs == 0);
func_unload(func);
free(func->def);
free(func);
diff --git a/src/box/func.h b/src/box/func.h
index 3dfabbf40..781dac276 100644
--- a/src/box/func.h
+++ b/src/box/func.h
@@ -88,6 +88,8 @@ struct func {
* Cached runtime access information.
*/
struct access access[BOX_USER_MAX];
+ /** Reference counter. */
+ uint16_t refs;
};
/**
@@ -108,6 +110,19 @@ func_new(struct func_def *def);
void
func_delete(struct func *func);
+static inline void
+func_ref(struct func *func)
+{
+ ++func->refs;
+}
+
+static inline void
+func_unref(struct func *func)
+{
+ assert(func->refs > 0);
+ --func->refs;
+}
+
/**
* Call stored C function using @a args.
*/
diff --git a/src/box/index_def.c b/src/box/index_def.c
index 28de89274..8595d1455 100644
--- a/src/box/index_def.c
+++ b/src/box/index_def.c
@@ -43,6 +43,7 @@ const struct index_opts index_opts_default = {
/* .unique = */ true,
/* .dimension = */ 2,
/* .distance = */ RTREE_INDEX_DISTANCE_TYPE_EUCLID,
+ /* .functional_fid = */ 0,
/* .range_size = */ 0,
/* .page_size = */ 8192,
/* .run_count_per_level = */ 2,
@@ -57,6 +58,8 @@ const struct opt_def index_opts_reg[] = {
OPT_DEF("dimension", OPT_INT64, struct index_opts, dimension),
OPT_DEF_ENUM("distance", rtree_index_distance_type, struct index_opts,
distance, NULL),
+ OPT_DEF("functional_fid", OPT_UINT32, struct index_opts,
+ functional_fid),
OPT_DEF("range_size", OPT_INT64, struct index_opts, range_size),
OPT_DEF("page_size", OPT_INT64, struct index_opts, page_size),
OPT_DEF("run_count_per_level", OPT_INT64, struct index_opts, run_count_per_level),
@@ -251,8 +254,15 @@ index_def_to_key_def(struct rlist *index_defs, int *size)
{
int key_count = 0;
struct index_def *index_def;
- rlist_foreach_entry(index_def, index_defs, link)
+ rlist_foreach_entry(index_def, index_defs, link) {
+ /*
+ * Don't use functional index key definition
+ * to build a space format.
+ */
+ if (index_def_is_functional(index_def))
+ continue;
key_count++;
+ }
size_t sz = sizeof(struct key_def *) * key_count;
struct key_def **keys = (struct key_def **) region_alloc(&fiber()->gc,
sz);
@@ -262,8 +272,11 @@ index_def_to_key_def(struct rlist *index_defs, int *size)
}
*size = key_count;
key_count = 0;
- rlist_foreach_entry(index_def, index_defs, link)
+ rlist_foreach_entry(index_def, index_defs, link) {
+ if (index_def_is_functional(index_def))
+ continue;
keys[key_count++] = index_def->key_def;
+ }
return keys;
}
@@ -296,6 +309,11 @@ index_def_is_valid(struct index_def *index_def, const char *space_name)
space_name, "primary key cannot be multikey");
return false;
}
+ if (index_def->iid == 0 && key_def_is_functional(index_def->key_def)) {
+ diag_set(ClientError, ER_MODIFY_INDEX, index_def->name,
+ space_name, "primary key cannot be functional");
+ return false;
+ }
for (uint32_t i = 0; i < index_def->key_def->part_count; i++) {
assert(index_def->key_def->parts[i].type < field_type_MAX);
if (index_def->key_def->parts[i].fieldno > BOX_INDEX_FIELD_MAX) {
diff --git a/src/box/index_def.h b/src/box/index_def.h
index 6dac28377..b5f669607 100644
--- a/src/box/index_def.h
+++ b/src/box/index_def.h
@@ -135,6 +135,11 @@ struct index_opts {
* RTREE distance type.
*/
enum rtree_index_distance_type distance;
+ /**
+ * Index of the function used for functional index.
+ * The value > 0 for functional index, and 0 otherwise.
+ */
+ uint32_t functional_fid;
/**
* Vinyl index options.
*/
@@ -207,6 +212,8 @@ index_opts_cmp(const struct index_opts *o1, const struct index_opts *o2)
return o1->run_size_ratio < o2->run_size_ratio ? -1 : 1;
if (o1->bloom_fpr != o2->bloom_fpr)
return o1->bloom_fpr < o2->bloom_fpr ? -1 : 1;
+ if (o1->functional_fid != o2->functional_fid)
+ return o1->functional_fid - o2->functional_fid;
return 0;
}
@@ -365,6 +372,15 @@ index_def_cmp(const struct index_def *key1, const struct index_def *key2);
bool
index_def_is_valid(struct index_def *index_def, const char *space_name);
+static inline bool
+index_def_is_functional(const struct index_def *index_def)
+{
+ assert(index_def->opts.functional_fid == 0 ||
+ (key_def_is_functional(index_def->key_def) &&
+ key_def_is_functional(index_def->cmp_def)));
+ return index_def->opts.functional_fid > 0;
+}
+
#if defined(__cplusplus)
} /* extern "C" */
diff --git a/src/box/key_def.c b/src/box/key_def.c
index eebfb7fe4..3245b469c 100644
--- a/src/box/key_def.c
+++ b/src/box/key_def.c
@@ -240,7 +240,8 @@ key_def_set_part(struct key_def *def, uint32_t part_no, uint32_t fieldno,
}
struct key_def *
-key_def_new(const struct key_part_def *parts, uint32_t part_count)
+key_def_new(const struct key_part_def *parts, uint32_t part_count,
+ bool is_functional)
{
size_t sz = 0;
for (uint32_t i = 0; i < part_count; i++)
@@ -254,6 +255,8 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count)
def->part_count = part_count;
def->unique_part_count = part_count;
+ if (is_functional)
+ def->functional_part_count = part_count;
/* A pointer to the JSON paths data in the new key_def. */
char *path_pool = (char *)def + key_def_sizeof(part_count, 0);
@@ -326,6 +329,7 @@ box_key_def_new(uint32_t *fields, uint32_t *types, uint32_t part_count)
key_def->part_count = part_count;
key_def->unique_part_count = part_count;
+ key_def->functional_part_count = 0;
for (uint32_t item = 0; item < part_count; ++item) {
if (key_def_set_part(key_def, item, fields[item],
@@ -707,6 +711,9 @@ static bool
key_def_can_merge(const struct key_def *key_def,
const struct key_part *to_merge)
{
+ if (key_def_is_functional(key_def))
+ return true;
+
const struct key_part *part = key_def_find(key_def, to_merge);
if (part == NULL)
return true;
@@ -721,6 +728,7 @@ key_def_can_merge(const struct key_def *key_def,
struct key_def *
key_def_merge(const struct key_def *first, const struct key_def *second)
{
+ assert(!key_def_is_functional(second));
uint32_t new_part_count = first->part_count + second->part_count;
/*
* Find and remove part duplicates, i.e. parts counted
@@ -752,6 +760,7 @@ key_def_merge(const struct key_def *first, const struct key_def *second)
new_def->is_nullable = first->is_nullable || second->is_nullable;
new_def->has_optional_parts = first->has_optional_parts ||
second->has_optional_parts;
+ new_def->functional_part_count = first->functional_part_count;
/* JSON paths data in the new key_def. */
char *path_pool = (char *)new_def + key_def_sizeof(new_part_count, 0);
@@ -824,7 +833,7 @@ key_def_find_pk_in_cmp_def(const struct key_def *cmp_def,
}
/* Finally, allocate the new key definition. */
- extracted_def = key_def_new(parts, pk_def->part_count);
+ extracted_def = key_def_new(parts, pk_def->part_count, false);
out:
region_truncate(region, region_svp);
return extracted_def;
diff --git a/src/box/key_def.h b/src/box/key_def.h
index f4a1a8fd1..c63306fca 100644
--- a/src/box/key_def.h
+++ b/src/box/key_def.h
@@ -192,6 +192,12 @@ struct key_def {
* unique_part_count == part count of a merged key_def.
*/
uint32_t unique_part_count;
+ /**
+ * Is not 0 iff this is functional index extractor
+ * routine's returned key definition, is equal to the
+ * count of parts in produced keys.
+ */
+ uint32_t functional_part_count;
/** True, if at least one part can store NULL. */
bool is_nullable;
/** True if some key part has JSON path. */
@@ -328,7 +334,8 @@ key_def_sizeof(uint32_t part_count, uint32_t path_pool_size)
* and initialize its parts.
*/
struct key_def *
-key_def_new(const struct key_part_def *parts, uint32_t part_count);
+key_def_new(const struct key_part_def *parts, uint32_t part_count,
+ bool is_functional);
/**
* Dump part definitions of the given key def.
@@ -475,6 +482,28 @@ key_def_is_multikey(const struct key_def *key_def)
return key_def->multikey_path != NULL;
}
+static inline bool
+key_def_is_functional(const struct key_def *key_def)
+{
+ return key_def->functional_part_count > 0;
+}
+
+/**
+ * Return whether given functional key definition is "flat".
+ *
+ * The key extracted with key_def:tuple_extract_key have an array
+ * header even when part_count = 1. As the keys produced with
+ * functional index extractor are typically resident for memtex
+ * memory, "flat" optimization allows don't store redundant
+ * headers.
+ */
+static inline bool
+key_def_fextract_key_is_flat(struct key_def *key_def)
+{
+ assert(key_def_is_functional(key_def));
+ return key_def->functional_part_count == 1;
+}
+
/**
* Return true if @a key_def defines has fields that requires
* special collation comparison.
diff --git a/src/box/lua/key_def.c b/src/box/lua/key_def.c
index dfcc89442..b3ab0eaca 100644
--- a/src/box/lua/key_def.c
+++ b/src/box/lua/key_def.c
@@ -443,7 +443,7 @@ lbox_key_def_new(struct lua_State *L)
lua_pop(L, 1);
}
- struct key_def *key_def = key_def_new(parts, part_count);
+ struct key_def *key_def = key_def_new(parts, part_count, false);
region_truncate(region, region_svp);
if (key_def == NULL)
return luaT_error(L);
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index 2599a24a3..844cbb245 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -964,6 +964,7 @@ local alter_index_template = {
type = 'string',
parts = 'table',
sequence = 'boolean, number, string, table',
+ functional_fid = 'number',
}
for k, v in pairs(index_options) do
alter_index_template[k] = v
@@ -1057,6 +1058,7 @@ box.schema.index.create = function(space_id, name, options)
run_count_per_level = options.run_count_per_level,
run_size_ratio = options.run_size_ratio,
bloom_fpr = options.bloom_fpr,
+ functional_fid = options.functional_fid,
}
local field_type_aliases = {
num = 'unsigned'; -- Deprecated since 1.7.2
diff --git a/src/box/lua/space.cc b/src/box/lua/space.cc
index 87adaeb16..cb29f7bde 100644
--- a/src/box/lua/space.cc
+++ b/src/box/lua/space.cc
@@ -274,6 +274,11 @@ lbox_fillspace(struct lua_State *L, struct space *space, int i)
lua_setfield(L, -2, "dimension");
}
+ if (index_def_is_functional(index_def)) {
+ lua_pushnumber(L, index_opts->functional_fid);
+ lua_setfield(L, -2, "functional_fid");
+ }
+
lua_pushstring(L, index_type_strs[index_def->type]);
lua_setfield(L, -2, "type");
--git a/src/box/lua/tuple_fextract_key.c b/src/box/lua/tuple_fextract_key.c
new file mode 100644
index 000000000..5611e2649
--- /dev/null
+++ b/src/box/lua/tuple_fextract_key.c
@@ -0,0 +1,233 @@
+/*
+ * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include "tuple_fextract_key.h"
+#include "fiber.h"
+#include "mpstream.h"
+#include "box/func.h"
+#include "box/tuple.h"
+#include "box/lua/tuple.h"
+#include "box/lua/misc.h"
+#include "lua/msgpack.h"
+#include "small/region.h"
+
+/**
+ * Service object to pass execution context to Lua machine
+ * and obtain a result.
+ */
+struct tuple_fextract_key_ctx {
+ /**
+ * The pointer to the persistent Lua functional index key
+ * extractor routine to execute.
+ */
+ struct func *func;
+ /** The key definition of returned values. */
+ struct key_def *key_def;
+ /**
+ * True to pass tuple raw data instead of tuple
+ * pointer.
+ */
+ bool extract_raw;
+ union {
+ /** Tuple to process. */
+ struct tuple *tuple;
+ /** Tuple data to process. */
+ const char *tuple_data;
+ };
+ /**
+ * Return value, the pointer to the region memory chunk
+ * with all extracted keys.
+ */
+ char *key;
+ /**
+ * Return value, the pointer to the end of the region
+ * memory chunk with all extracted keys.
+ */
+ char *key_end;
+};
+
+/**
+ * The Lua routine to execute a functional index extractor
+ * routine.
+ */
+static int
+execute_tuple_fextract_key(struct lua_State *L)
+{
+ struct tuple_fextract_key_ctx *ctx =
+ (struct tuple_fextract_key_ctx *) lua_topointer(L, 1);
+ lua_settop(L, 0);
+
+ /* Execute functional index extractor routine. */
+ struct func *func = ctx->func;
+ assert(func != NULL);
+ lua_rawgeti(L, LUA_REGISTRYINDEX, func->lua_func_ref);
+ if (ctx->extract_raw) {
+ assert(mp_typeof(*ctx->tuple_data) == MP_ARRAY);
+ uint32_t arg_count = mp_decode_array(&ctx->tuple_data);
+ luaL_checkstack(L, arg_count,
+ "execute_tuple_fextract_key: out of stack");
+ for (uint32_t i = 0; i < arg_count; i++)
+ luamp_decode(L, luaL_msgpack_default, &ctx->tuple_data);
+ } else {
+ luaT_pushtuple(L, ctx->tuple);
+ }
+ if (lua_pcall(L, 1, 1, 0) != 0) {
+ diag_set(ClientError, ER_FUNCTIONAL_EXTRACTOR_ERROR,
+ luaT_tolstring(L, -1, NULL));
+ return luaT_error(L);
+ }
+
+ /* Encode result of the region. */
+ struct region *region = &fiber()->gc;
+ size_t used = region_used(region);
+ struct mpstream stream;
+ mpstream_init(&stream, region, region_reserve_cb, region_alloc_cb,
+ luamp_error, L);
+ enum mp_type type = luamp_encode(L, luaL_msgpack_default, &stream, 1);
+ mpstream_flush(&stream);
+ uint32_t key_sz = region_used(region) - used;
+ ctx->key = (char *) region_join(region, key_sz);
+ if (ctx->key == NULL) {
+ diag_set(OutOfMemory, key_sz, "region_alloc", "data");
+ return luaT_error(L);
+ }
+ ctx->key_end = ctx->key + key_sz;
+
+ /* Test return value type. */
+ bool is_nullable = func->def->returns == FIELD_TYPE_ARRAY ? false :
+ key_part_is_nullable(ctx->key_def->parts);
+ if (!field_mp_type_is_compatible(func->def->returns, type,
+ is_nullable)) {
+ diag_set(ClientError, ER_FUNCTIONAL_EXTRACTOR_ERROR,
+ "returned type doesn't follow the definition");
+ return luaT_error(L);
+ }
+ if (func->def->returns == FIELD_TYPE_ARRAY) {
+ /* Unpack multikey extractor return. */
+ mp_tuple_assert(ctx->key, ctx->key_end);
+ (void)mp_decode_array((const char **)&ctx->key);
+ }
+ return 0;
+}
+
+char *
+tuple_fextract_key(struct tuple *tuple, struct func *func,
+ struct key_def *key_def, uint32_t *key_size)
+{
+ assert(func->def->language == FUNC_LANGUAGE_LUA &&
+ func->lua_func_ref != LUA_REFNIL);
+
+ struct tuple_fextract_key_ctx ctx;
+ ctx.func = func;
+ ctx.key_def = key_def;
+ ctx.extract_raw = false;
+ ctx.tuple = tuple;
+
+ struct lua_State *L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_pushcfunction(L, execute_tuple_fextract_key);
+ lua_pushlightuserdata(L, &ctx);
+ int rc = luaT_call(L, 1, LUA_MULTRET);
+ luaL_unref(L, LUA_REGISTRYINDEX, coro_ref);
+ if (rc != 0)
+ return NULL;
+ *key_size = ctx.key_end - ctx.key;
+ return ctx.key;
+}
+
+char *
+tuple_fextract_key_raw(const char *tuple_data, struct func *func,
+ struct key_def *key_def, uint32_t *key_size)
+{
+ assert(func->def->language == FUNC_LANGUAGE_LUA &&
+ func->lua_func_ref != LUA_REFNIL);
+
+ struct tuple_fextract_key_ctx ctx;
+ ctx.func = func;
+ ctx.key_def = key_def;
+ ctx.extract_raw = true;
+ ctx.tuple_data = tuple_data;
+
+ struct lua_State *L = lua_newthread(tarantool_L);
+ int coro_ref = luaL_ref(tarantool_L, LUA_REGISTRYINDEX);
+ lua_pushcfunction(L, execute_tuple_fextract_key);
+ lua_pushlightuserdata(L, &ctx);
+ int rc = luaT_call(L, 1, LUA_MULTRET);
+ luaL_unref(L, LUA_REGISTRYINDEX, coro_ref);
+ if (rc != 0)
+ return NULL;
+
+ *key_size = ctx.key_end - ctx.key;
+ return ctx.key;
+}
+
+int
+fextract_key_iterator_next(struct fextract_key_iterator *it, const char **key)
+{
+ assert(it->data <= it->data_end);
+ if (it->data == it->data_end) {
+ *key = NULL;
+ return 0;
+ }
+ *key = it->data;
+ if (!it->validate) {
+ mp_next(&it->data);
+ assert(it->data <= it->data_end);
+ return 0;
+ }
+
+ uint32_t part_count;
+ if (key_def_fextract_key_is_flat(it->key_def)) {
+ part_count = it->key_def->functional_part_count;
+ } else {
+ enum mp_type type = mp_typeof(*it->data);
+ if (type != MP_ARRAY)
+ goto error;
+ part_count = mp_decode_array(&it->data);
+ if (part_count != it->key_def->functional_part_count)
+ goto error;
+ }
+
+ for (uint32_t i = 0; i < part_count; i++) {
+ assert(it->data <= it->data_end);
+ const struct key_part *part = &it->key_def->parts[i];
+ bool is_nullable = key_part_is_nullable(part);
+ if (!field_mp_type_is_compatible(part->type,
+ mp_typeof(*it->data), is_nullable))
+ goto error;
+ mp_next(&it->data);
+ }
+ return 0;
+error:
+ diag_set(ClientError, ER_FUNCTIONAL_EXTRACTOR_ERROR,
+ tt_sprintf("returned key %s doesn't follow the definition",
+ mp_str(*key)));
+ return -1;
+}
--git a/src/box/lua/tuple_fextract_key.h b/src/box/lua/tuple_fextract_key.h
new file mode 100644
index 000000000..354ffc642
--- /dev/null
+++ b/src/box/lua/tuple_fextract_key.h
@@ -0,0 +1,100 @@
+#ifndef TARANTOOL_BOX_LUA_TUPLE_FEXTRACT_KEY_H_INCLUDED
+#define TARANTOOL_BOX_LUA_TUPLE_FEXTRACT_KEY_H_INCLUDED
+/*
+ * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include <stdbool.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#if defined(__cplusplus)
+extern "C" {
+#endif /* defined(__cplusplus) */
+
+struct func;
+struct tuple;
+struct key_def;
+
+/**
+ * Extract key from tuple by given extractor function object and
+ * functional key definition and return buffer allocated on
+ * region with extracted keys.
+ */
+char *
+tuple_fextract_key(struct tuple *tuple, struct func *func,
+ struct key_def *key_def, uint32_t *key_size);
+
+/**
+ * Extract key from raw msgpuck by given extractor function
+ * object and functional key definition and return buffer
+ * allocated on region with extracted keys.
+ */
+char *
+tuple_fextract_key_raw(const char *tuple_data, struct func *func,
+ struct key_def *key_def, uint32_t *key_size);
+
+/**
+ * An iterator to iterate sequence of extracted keys for given
+ * key definition and inspect key parts properties.
+ */
+struct fextract_key_iterator {
+ struct key_def *key_def;
+ const char *data;
+ const char *data_end;
+ bool validate;
+};
+
+/** Initialize a new keys iterator instance. */
+static inline void
+fextract_key_iterator_create(struct fextract_key_iterator *it,
+ struct key_def *key_def, bool validate,
+ const char *data, uint32_t data_sz)
+{
+ it->data = data;
+ it->data_end = data + data_sz;
+ it->key_def = key_def;
+ it->validate = validate;
+}
+
+/**
+ * Perform key iterator step and update iterator state.
+ * Update key pointer with an actual key.
+ *
+ * Returns 0 on success. In case of error returns -1 sets
+ * the corresponding diag message.
+ */
+int
+fextract_key_iterator_next(struct fextract_key_iterator *it, const char **key);
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
+
+#endif /* #ifndef TARANTOOL_BOX_LUA_TUPLE_FEXTRACT_KEY_H_INCLUDED */
diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c
index 8faa9fa57..400137377 100644
--- a/src/box/memtx_engine.c
+++ b/src/box/memtx_engine.c
@@ -1312,6 +1312,9 @@ memtx_index_def_change_requires_rebuild(struct index *index,
return true;
if (!old_def->opts.is_unique && new_def->opts.is_unique)
return true;
+ if (index_def_is_functional(old_def) !=
+ index_def_is_functional(new_def))
+ return true;
const struct key_def *old_cmp_def, *new_cmp_def;
if (index_depends_on_pk(index)) {
diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
index 5ddb4f7ee..f44c12ac8 100644
--- a/src/box/memtx_space.c
+++ b/src/box/memtx_space.c
@@ -644,6 +644,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
"HASH index cannot be multikey");
return -1;
}
+ if (index_def_is_functional(index_def)) {
+ diag_set(ClientError, ER_MODIFY_INDEX,
+ index_def->name, space_name(space),
+ "HASH index cannot be functional");
+ return -1;
+ }
break;
case TREE:
/* TREE index has no limitations. */
@@ -673,6 +679,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
"RTREE index cannot be multikey");
return -1;
}
+ if (index_def_is_functional(index_def)) {
+ diag_set(ClientError, ER_MODIFY_INDEX,
+ index_def->name, space_name(space),
+ "RTREE index cannot be functional");
+ return -1;
+ }
/* no furter checks of parts needed */
return 0;
case BITSET:
@@ -701,6 +713,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
"BITSET index cannot be multikey");
return -1;
}
+ if (index_def_is_functional(index_def)) {
+ diag_set(ClientError, ER_MODIFY_INDEX,
+ index_def->name, space_name(space),
+ "BITSET index cannot be functional");
+ return -1;
+ }
/* no furter checks of parts needed */
return 0;
default:
diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c
index 268ce4050..8f968fb04 100644
--- a/src/box/memtx_tree.c
+++ b/src/box/memtx_tree.c
@@ -38,6 +38,8 @@
#include "tuple.h"
#include <third_party/qsort_arg.h>
#include <small/mempool.h>
+#include "func.h"
+#include "lua/tuple_fextract_key.h"
/**
* Struct that is used as a key in BPS tree definition.
@@ -101,6 +103,37 @@ memtx_tree_data_identical(const struct memtx_tree_data *a,
#undef bps_tree_key_t
#undef bps_tree_arg_t
+
+/**
+ * Metadata object stored in index tuple_meta_hash hashtable.
+ * The object must be allocated with memtx_slab_alloc allocator.
+ */
+struct memtx_tree_index_meta {
+ /** Tuple pointer is used as a hash key. */
+ struct tuple *tuple;
+ /**
+ * The size of the rest of object. Is used to calculate
+ * the size object to release acquired memory.
+ */
+ uint32_t data_sz;
+ /** Metadata object payload. */
+ char data[0];
+};
+
+/**
+ * Map: (struct tuple *) => (struct memtx_tree_index_meta *)
+ */
+#define MH_SOURCE 1
+#define mh_name _tuple_meta
+#define mh_key_t struct tuple *
+#define mh_node_t struct memtx_tree_index_meta *
+#define mh_arg_t void *
+#define mh_hash(a, arg) ((uintptr_t)(*(a))->tuple)
+#define mh_hash_key(a, arg) ((uintptr_t) a)
+#define mh_cmp(a, b, arg) ((*(a))->tuple != (*(b))->tuple)
+#define mh_cmp_key(a, b, arg) ((a) != (*(b))->tuple)
+#include "salad/mhash.h"
+
struct memtx_tree_index {
struct index base;
struct memtx_tree tree;
@@ -108,10 +141,77 @@ struct memtx_tree_index {
size_t build_array_size, build_array_alloc_size;
struct memtx_gc_task gc_task;
struct memtx_tree_iterator gc_iterator;
+ struct func *func;
+ struct mh_tuple_meta_t *tuple_meta_hash;
};
/* {{{ Utilities. *************************************************/
+/**
+ * Allocate a new tuple metadata object and register it in the
+ * index metadata hashtable.
+ * Returns not NULL tuple metadata chunk pointer on success.
+ * In case of error returns NULL and sets diag message.
+ */
+static struct memtx_tree_index_meta *
+memtx_tree_index_meta_new(struct memtx_tree_index *index, struct tuple *tuple,
+ const char *data, uint32_t data_sz)
+{
+ struct memtx_engine *memtx = (struct memtx_engine *)index->base.engine;
+ uint32_t meta_sz = sizeof(struct memtx_tree_index_meta) + data_sz;
+ struct memtx_tree_index_meta *meta =
+ (struct memtx_tree_index_meta *)memtx_slab_alloc(memtx, meta_sz);
+ if (meta == NULL) {
+ diag_set(OutOfMemory, data_sz, "memtx_slab_alloc",
+ "memtx_tuple");
+ return NULL;
+ }
+ meta->tuple = tuple;
+ meta->data_sz = data_sz;
+ memcpy(meta->data, data, data_sz);
+ mh_int_t id = mh_tuple_meta_put(index->tuple_meta_hash,
+ (const struct memtx_tree_index_meta **)&meta,
+ NULL, NULL);
+ if (id == mh_end(index->tuple_meta_hash)) {
+ diag_set(OutOfMemory, sizeof(struct memtx_tree_index_meta *),
+ "mh_tuple_meta_put", "meta");
+ memtx_slab_free(memtx, meta, meta_sz, true);
+ return NULL;
+ }
+ return meta;
+}
+
+/**
+ * Release a given tuple metadata chunk, clean the corresponding
+ * hashtable entry.
+ */
+static void
+memtx_tree_index_meta_delete(struct memtx_tree_index *index,
+ struct memtx_tree_index_meta *meta)
+{
+ mh_int_t id = mh_tuple_meta_find(index->tuple_meta_hash,
+ meta->tuple, NULL);
+ assert(id != mh_end(index->tuple_meta_hash));
+ mh_tuple_meta_del(index->tuple_meta_hash, id, NULL);
+
+ struct memtx_engine *memtx = (struct memtx_engine *)index->base.engine;
+ uint32_t meta_sz = sizeof(struct memtx_tree_index_meta) + meta->data_sz;
+ memtx_slab_free(memtx, meta, meta_sz, true);
+}
+
+/** Get tuple metadata chunk by given tuple. */
+static struct memtx_tree_index_meta *
+memtx_tree_index_meta_get(struct memtx_tree_index *index, struct tuple *tuple)
+{
+ mh_int_t id = mh_tuple_meta_find(index->tuple_meta_hash, tuple, NULL);
+ if (id == mh_end(index->tuple_meta_hash))
+ return NULL;
+ struct memtx_tree_index_meta **entry =
+ mh_tuple_meta_node(index->tuple_meta_hash, id);
+ assert(entry != NULL && *entry != NULL);
+ return *entry;
+}
+
static inline struct key_def *
memtx_tree_cmp_def(struct memtx_tree *tree)
{
@@ -388,6 +488,21 @@ tree_iterator_start(struct iterator *iterator, struct tuple **ret)
static void
memtx_tree_index_free(struct memtx_tree_index *index)
{
+
+ if (index->func != NULL) {
+ func_unref(index->func);
+ index->func = NULL;
+ }
+ if (index->tuple_meta_hash != NULL) {
+ while (mh_size(index->tuple_meta_hash)) {
+ mh_int_t i = mh_first(index->tuple_meta_hash);
+ struct memtx_tree_index_meta **meta =
+ mh_tuple_meta_node(index->tuple_meta_hash, i);
+ assert(meta != NULL && *meta != NULL);
+ memtx_tree_index_meta_delete(index, *meta);
+ }
+ mh_tuple_meta_delete(index->tuple_meta_hash);
+ }
memtx_tree_destroy(&index->tree);
free(index->build_array);
free(index);
@@ -591,12 +706,12 @@ memtx_tree_index_replace(struct index *base, struct tuple *old_tuple,
static int
memtx_tree_index_replace_multikey_one(struct memtx_tree_index *index,
struct tuple *old_tuple, struct tuple *new_tuple,
- enum dup_replace_mode mode, int multikey_idx,
+ enum dup_replace_mode mode, hint_t hint,
struct tuple **replaced_tuple)
{
struct memtx_tree_data new_data, dup_data;
new_data.tuple = new_tuple;
- new_data.hint = multikey_idx;
+ new_data.hint = (hint_t) hint;
dup_data.tuple = NULL;
if (memtx_tree_insert(&index->tree, new_data, &dup_data) != 0) {
diag_set(OutOfMemory, MEMTX_EXTENT_SIZE, "memtx_tree_index",
@@ -684,7 +799,8 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple,
struct tuple *replaced_tuple;
err = memtx_tree_index_replace_multikey_one(index,
old_tuple, new_tuple, mode,
- multikey_idx, &replaced_tuple);
+ (hint_t)multikey_idx,
+ &replaced_tuple);
if (err != 0)
break;
if (replaced_tuple != NULL) {
@@ -716,6 +832,129 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple,
return 0;
}
+/**
+ * Rollback the sequence of memtx_tree_index_replace_multikey_one
+ * insertions with functional index by given new_meta chunk.
+ *
+ * This routine can't fail because all replaced_tuple (when
+ * specified) nodes in tree are already allocated (they might be
+ * overridden with new_tuple, but they definitely present) and
+ * delete operation is fault-tolerant.
+ */
+static void
+memtx_tree_index_replace_functional_rollback(struct memtx_tree_index *index,
+ struct memtx_tree_index_meta *new_meta,
+ struct tuple *replaced_tuple)
+{
+ assert(new_meta != NULL);
+ struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree);
+
+ const char *key;
+ struct fextract_key_iterator it;
+ struct memtx_tree_data data;
+ if (replaced_tuple != NULL) {
+ /* Restore replaced tuple index occurrences. */
+ struct memtx_tree_index_meta *old_meta =
+ memtx_tree_index_meta_get(index, replaced_tuple);
+ assert(old_meta != NULL);
+ fextract_key_iterator_create(&it, cmp_def, false,
+ old_meta->data, old_meta->data_sz);
+ data.tuple = replaced_tuple;
+ while (fextract_key_iterator_next(&it, &key) == 0 &&
+ key != NULL) {
+ data.hint = (hint_t) key;
+ memtx_tree_insert(&index->tree, data, NULL);
+ }
+ assert(key == NULL);
+ }
+ /* Rollback new_tuple insertion. */
+ data.tuple = new_meta->tuple;
+ fextract_key_iterator_create(&it, cmp_def, false, new_meta->data,
+ new_meta->data_sz);
+ while (fextract_key_iterator_next(&it, &key) == 0 && key != NULL) {
+ data.hint = (hint_t) key;
+ memtx_tree_delete_identical(&index->tree, data);
+ }
+ assert(key == NULL);
+}
+
+static int
+memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple,
+ struct tuple *new_tuple, enum dup_replace_mode mode,
+ struct tuple **result)
+{
+ struct memtx_tree_index *index = (struct memtx_tree_index *)base;
+ struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree);
+ assert(index->func != NULL && key_def_is_functional(cmp_def));
+
+ int rc = -1;
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+
+ *result = NULL;
+ const char *key;
+ struct fextract_key_iterator it;
+ if (new_tuple != NULL) {
+ uint32_t keys_sz;
+ char *keys = tuple_fextract_key(new_tuple, index->func,
+ cmp_def, &keys_sz);
+ if (keys == NULL)
+ goto end;
+ struct memtx_tree_index_meta *new_meta =
+ memtx_tree_index_meta_new(index, new_tuple, keys,
+ keys_sz);
+ if (new_meta == NULL)
+ goto end;
+ fextract_key_iterator_create(&it, cmp_def, true, new_meta->data,
+ new_meta->data_sz);
+ int err = 0;
+ while (fextract_key_iterator_next(&it, &key) == 0 &&
+ key != NULL) {
+ struct tuple *replaced_tuple;
+ err = memtx_tree_index_replace_multikey_one(index,
+ old_tuple, new_tuple, mode,
+ (hint_t) key, &replaced_tuple);
+ if (err != 0)
+ break;
+ if (replaced_tuple != NULL) {
+ assert(*result == NULL ||
+ *result == replaced_tuple);
+ *result = replaced_tuple;
+ }
+ }
+ if (key != NULL || err != 0) {
+ memtx_tree_index_replace_functional_rollback(index,
+ new_meta, *result);
+ memtx_tree_index_meta_delete(index, new_meta);
+ goto end;
+ }
+ if (*result != NULL) {
+ assert(old_tuple == NULL || old_tuple == *result);
+ old_tuple = *result;
+ }
+ }
+ if (old_tuple != NULL) {
+ struct memtx_tree_index_meta *old_meta =
+ memtx_tree_index_meta_get(index, old_tuple);
+ assert(old_meta != NULL);
+ fextract_key_iterator_create(&it, cmp_def, false,
+ old_meta->data, old_meta->data_sz);
+ struct memtx_tree_data data;
+ data.tuple = old_tuple;
+ while (fextract_key_iterator_next(&it, &key) == 0 &&
+ key != NULL) {
+ data.hint = (hint_t) key;
+ memtx_tree_delete_identical(&index->tree, data);
+ }
+ assert(key == NULL);
+ memtx_tree_index_meta_delete(index, old_meta);
+ }
+ rc = 0;
+end:
+ region_truncate(region, region_svp);
+ return rc;
+}
+
static struct iterator *
memtx_tree_index_create_iterator(struct index *base, enum iterator_type type,
const char *key, uint32_t part_count)
@@ -766,7 +1005,16 @@ memtx_tree_index_begin_build(struct index *base)
{
struct memtx_tree_index *index = (struct memtx_tree_index *)base;
assert(memtx_tree_size(&index->tree) == 0);
- (void)index;
+ if (index_def_is_functional(base->def) && index->func == NULL) {
+ /*
+ * The function reference hadn't been resolved
+ * on index creation because the function object
+ * wasn't exists on that moment.
+ */
+ index->func = func_by_id(base->def->opts.functional_fid);
+ assert(index->func != NULL);
+ func_ref(index->func);
+ }
}
static int
@@ -847,6 +1095,49 @@ memtx_tree_index_build_next_multikey(struct index *base, struct tuple *tuple)
return 0;
}
+static int
+memtx_tree_index_build_next_functional(struct index *base, struct tuple *tuple)
+{
+ struct memtx_tree_index *index = (struct memtx_tree_index *)base;
+ struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree);
+ assert(index->func != NULL && key_def_is_functional(cmp_def));
+
+ struct region *region = &fiber()->gc;
+ size_t region_svp = region_used(region);
+
+ uint32_t keys_sz;
+ char *keys = tuple_fextract_key(tuple, index->func, cmp_def, &keys_sz);
+ if (keys == NULL) {
+ region_truncate(region, region_svp);
+ return -1;
+ }
+
+ struct memtx_tree_index_meta *new_meta =
+ memtx_tree_index_meta_new(index, tuple, keys, keys_sz);
+ if (new_meta == NULL) {
+ region_truncate(region, region_svp);
+ return -1;
+ }
+
+ struct fextract_key_iterator it;
+ fextract_key_iterator_create(&it, cmp_def, false, new_meta->data,
+ new_meta->data_sz);
+
+ const char *key;
+ struct memtx_tree_data data;
+ data.tuple = tuple;
+ while (fextract_key_iterator_next(&it, &key) == 0 && key != NULL) {
+ if (memtx_tree_index_build_array_append(index, tuple,
+ (hint_t) key) != 0) {
+ region_truncate(region, region_svp);
+ return -1;
+ }
+ }
+ assert(key == NULL);
+ region_truncate(region, region_svp);
+ return 0;
+}
+
/**
* Process build_array of specified index and remove duplicates
* of equal tuples (in terms of index's cmp_def and have same
@@ -882,7 +1173,8 @@ memtx_tree_index_end_build(struct index *base)
struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree);
qsort_arg(index->build_array, index->build_array_size,
sizeof(index->build_array[0]), memtx_tree_qcompare, cmp_def);
- if (key_def_is_multikey(cmp_def)) {
+ if (key_def_is_multikey(cmp_def) ||
+ key_def_is_functional(cmp_def)) {
/*
* Multikey index may have equal(in terms of
* cmp_def) keys inserted by different multikey
@@ -1017,6 +1309,36 @@ static const struct index_vtab memtx_tree_index_multikey_vtab = {
/* .end_build = */ memtx_tree_index_end_build,
};
+static const struct index_vtab memtx_tree_index_functional_vtab = {
+ /* .destroy = */ memtx_tree_index_destroy,
+ /* .commit_create = */ generic_index_commit_create,
+ /* .abort_create = */ generic_index_abort_create,
+ /* .commit_modify = */ generic_index_commit_modify,
+ /* .commit_drop = */ generic_index_commit_drop,
+ /* .update_def = */ memtx_tree_index_update_def,
+ /* .depends_on_pk = */ memtx_tree_index_depends_on_pk,
+ /* .def_change_requires_rebuild = */
+ memtx_index_def_change_requires_rebuild,
+ /* .size = */ memtx_tree_index_size,
+ /* .bsize = */ memtx_tree_index_bsize,
+ /* .min = */ generic_index_min,
+ /* .max = */ generic_index_max,
+ /* .random = */ memtx_tree_index_random,
+ /* .count = */ memtx_tree_index_count,
+ /* .get = */ memtx_tree_index_get,
+ /* .replace = */ memtx_tree_index_replace_functional,
+ /* .create_iterator = */ memtx_tree_index_create_iterator,
+ /* .create_snapshot_iterator = */
+ memtx_tree_index_create_snapshot_iterator,
+ /* .stat = */ generic_index_stat,
+ /* .compact = */ generic_index_compact,
+ /* .reset_stat = */ generic_index_reset_stat,
+ /* .begin_build = */ memtx_tree_index_begin_build,
+ /* .reserve = */ memtx_tree_index_reserve,
+ /* .build_next = */ memtx_tree_index_build_next_functional,
+ /* .end_build = */ memtx_tree_index_end_build,
+};
+
struct index *
memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def)
{
@@ -1027,9 +1349,26 @@ memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def)
"malloc", "struct memtx_tree_index");
return NULL;
}
- const struct index_vtab *vtab = key_def_is_multikey(def->key_def) ?
- &memtx_tree_index_multikey_vtab :
- &memtx_tree_index_vtab;
+ const struct index_vtab *vtab;
+ if (index_def_is_functional(def)) {
+ index->tuple_meta_hash = mh_tuple_meta_new();
+ if (index->tuple_meta_hash == NULL) {
+ diag_set(OutOfMemory, sizeof(*index->tuple_meta_hash),
+ "malloc", "tuple_meta_hash");
+ free(index);
+ return NULL;
+ }
+ index->func = func_by_id(def->opts.functional_fid);
+ assert(index->func != NULL ||
+ memtx->state == MEMTX_INITIAL_RECOVERY);
+ if (index->func != NULL)
+ func_ref(index->func);
+ vtab = &memtx_tree_index_functional_vtab;
+ } else if (key_def_is_multikey(def->key_def)) {
+ vtab = &memtx_tree_index_multikey_vtab;
+ } else {
+ vtab = &memtx_tree_index_vtab;
+ }
if (index_create(&index->base, (struct engine *)memtx,
vtab, def) != 0) {
free(index);
diff --git a/src/box/schema.cc b/src/box/schema.cc
index 7f6086210..be8cb576c 100644
--- a/src/box/schema.cc
+++ b/src/box/schema.cc
@@ -271,7 +271,7 @@ sc_space_new(uint32_t id, const char *name,
struct trigger *replace_trigger,
struct trigger *stmt_begin_trigger)
{
- struct key_def *key_def = key_def_new(key_parts, key_part_count);
+ struct key_def *key_def = key_def_new(key_parts, key_part_count, false);
if (key_def == NULL)
diag_raise();
auto key_def_guard =
diff --git a/src/box/sql.c b/src/box/sql.c
index fbfa59992..632f229df 100644
--- a/src/box/sql.c
+++ b/src/box/sql.c
@@ -385,7 +385,7 @@ sql_ephemeral_space_create(uint32_t field_count, struct sql_key_info *key_info)
}
}
struct key_def *ephemer_key_def = key_def_new(ephemer_key_parts,
- field_count);
+ field_count, false);
if (ephemer_key_def == NULL)
return NULL;
diff --git a/src/box/sql/build.c b/src/box/sql/build.c
index e2353d8cc..ba40b6eae 100644
--- a/src/box/sql/build.c
+++ b/src/box/sql/build.c
@@ -2108,7 +2108,7 @@ index_fill_def(struct Parse *parse, struct index *index,
part->coll_id = coll_id;
part->path = NULL;
}
- key_def = key_def_new(key_parts, expr_list->nExpr);
+ key_def = key_def_new(key_parts, expr_list->nExpr, false);
if (key_def == NULL)
goto tnt_error;
/*
diff --git a/src/box/sql/select.c b/src/box/sql/select.c
index d3472a922..e484824b5 100644
--- a/src/box/sql/select.c
+++ b/src/box/sql/select.c
@@ -1445,7 +1445,7 @@ sql_key_info_to_key_def(struct sql_key_info *key_info)
{
if (key_info->key_def == NULL) {
key_info->key_def = key_def_new(key_info->parts,
- key_info->part_count);
+ key_info->part_count, false);
}
return key_info->key_def;
}
diff --git a/src/box/sql/where.c b/src/box/sql/where.c
index 19ee2d03a..2543fa487 100644
--- a/src/box/sql/where.c
+++ b/src/box/sql/where.c
@@ -2796,7 +2796,7 @@ whereLoopAddBtree(WhereLoopBuilder * pBuilder, /* WHERE clause information */
part.coll_id = COLL_NONE;
part.path = NULL;
- struct key_def *key_def = key_def_new(&part, 1);
+ struct key_def *key_def = key_def_new(&part, 1, false);
if (key_def == NULL) {
tnt_error:
pWInfo->pParse->is_aborted = true;
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index c1a70a087..966ca9f76 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -1271,6 +1271,88 @@ static const comparator_with_key_signature cmp_wk_arr[] = {
/* }}} tuple_compare_with_key */
+template<bool is_flat, bool is_nullable>
+static inline int
+functional_compare(struct tuple *tuple_a, hint_t tuple_a_hint,
+ struct tuple *tuple_b, hint_t tuple_b_hint,
+ struct key_def *key_def)
+{
+ assert(key_def_is_functional(key_def));
+ assert(is_nullable == key_def->is_nullable);
+ assert(is_flat == key_def_fextract_key_is_flat(key_def));
+ /* Extracted keys compare. */
+ const char *key_a = (const char *) tuple_a_hint;
+ const char *key_b = (const char *) tuple_b_hint;
+ uint32_t key_part_count = 1;
+ if (!is_flat) {
+ assert(mp_typeof(*key_a) == MP_ARRAY);
+ assert(mp_typeof(*key_b) == MP_ARRAY);
+ uint32_t part_count_a = mp_decode_array(&key_a);
+ uint32_t part_count_b = mp_decode_array(&key_b);
+ key_part_count = MIN(part_count_a, part_count_b);
+ }
+ uint32_t part_count =
+ MIN(key_def->functional_part_count, key_part_count);
+ int rc = key_compare_parts<is_nullable>(key_a, key_b, part_count,
+ key_def);
+ if (rc != 0)
+ return rc;
+ /*
+ * Primary index definiton key compare.
+ * It cannot contain nullable parts so the code is
+ * simplified correspondingly.
+ */
+ const char *tuple_a_raw = tuple_data(tuple_a);
+ const char *tuple_b_raw = tuple_data(tuple_b);
+ struct tuple_format *format_a = tuple_format(tuple_a);
+ struct tuple_format *format_b = tuple_format(tuple_b);
+ const uint32_t *field_map_a = tuple_field_map(tuple_a);
+ const uint32_t *field_map_b = tuple_field_map(tuple_b);
+ const char *field_a, *field_b;
+ for (uint32_t i = key_def->functional_part_count;
+ i < key_def->part_count; i++) {
+ struct key_part *part = &key_def->parts[i];
+ field_a = tuple_field_raw_by_part(format_a, tuple_a_raw,
+ field_map_a, part,
+ MULTIKEY_NONE);
+ field_b = tuple_field_raw_by_part(format_b, tuple_b_raw,
+ field_map_b, part,
+ MULTIKEY_NONE);
+ assert(field_a != NULL && field_b != NULL);
+ rc = tuple_compare_field(field_a, field_b, part->type,
+ part->coll);
+ if (rc != 0)
+ return rc;
+ else
+ continue;
+ }
+ return 0;
+}
+
+template<bool is_flat, bool is_nullable>
+static inline int
+functional_compare_with_key(struct tuple *tuple, hint_t tuple_hint,
+ const char *key, uint32_t part_count,
+ hint_t key_hint, struct key_def *key_def)
+{
+ assert(key_def->functional_part_count > 0);
+ assert(is_nullable == key_def->is_nullable);
+ assert(is_flat == key_def_fextract_key_is_flat(key_def));
+ (void)tuple;
+ (void)key_hint;
+ const char *tuple_key = (const char *)tuple_hint;
+ if (!is_flat) {
+ assert(mp_typeof(*tuple_key) == MP_ARRAY);
+ uint32_t tuple_key_count = mp_decode_array(&tuple_key);
+ part_count = MIN(part_count, tuple_key_count);
+ part_count = MIN(part_count, key_def->functional_part_count);
+ } else {
+ part_count = 1;
+ }
+ return key_compare_parts<is_nullable>(tuple_key, key, part_count,
+ key_def);
+}
+
/* {{{ tuple_hint */
/**
@@ -1593,7 +1675,8 @@ key_hint_multikey(const char *key, uint32_t part_count, struct key_def *key_def)
* do nothing on key hint calculation an it is valid
* because it is never used(unlike tuple hint).
*/
- assert(key_def_is_multikey(key_def));
+ assert(key_def_is_multikey(key_def) ||
+ key_def_is_functional(key_def));
return HINT_NONE;
}
@@ -1627,7 +1710,7 @@ key_def_set_hint_func(struct key_def *def)
static void
key_def_set_hint_func(struct key_def *def)
{
- if (key_def_is_multikey(def)) {
+ if (key_def_is_multikey(def) || key_def_is_functional(def)) {
def->key_hint = key_hint_multikey;
def->tuple_hint = tuple_hint_multikey;
return;
@@ -1752,11 +1835,32 @@ key_def_set_compare_func_json(struct key_def *def)
}
}
+template<bool is_nullable>
+static void
+key_def_set_compare_func_functional(struct key_def *def)
+{
+ assert(key_def_is_functional(def));
+ if (key_def_fextract_key_is_flat(def)) {
+ def->tuple_compare = functional_compare<true, is_nullable>;
+ def->tuple_compare_with_key =
+ functional_compare_with_key<true, is_nullable>;
+ } else {
+ def->tuple_compare = functional_compare<false, is_nullable>;
+ def->tuple_compare_with_key =
+ functional_compare_with_key<false, is_nullable>;
+ }
+}
+
void
key_def_set_compare_func(struct key_def *def)
{
- if (!key_def_has_collation(def) &&
- !def->is_nullable && !def->has_json_paths) {
+ if (key_def_is_functional(def)) {
+ if (def->is_nullable)
+ key_def_set_compare_func_functional<true>(def);
+ else
+ key_def_set_compare_func_functional<false>(def);
+ } else if (!key_def_has_collation(def) &&
+ !def->is_nullable && !def->has_json_paths) {
key_def_set_compare_func_fast(def);
} else if (!def->has_json_paths) {
if (def->is_nullable && def->has_optional_parts) {
--git a/src/box/tuple_extract_key.cc b/src/box/tuple_extract_key.cc
index 3bd3cde70..adac1cdd6 100644
--- a/src/box/tuple_extract_key.cc
+++ b/src/box/tuple_extract_key.cc
@@ -358,6 +358,27 @@ tuple_extract_key_slowpath_raw(const char *data, const char *data_end,
return key;
}
+static char *
+tuple_extract_key_functional(struct tuple *tuple, struct key_def *key_def,
+ int multikey_idx, uint32_t *key_size)
+{
+ (void)tuple;
+ (void)key_def; (void)multikey_idx; (void)key_size;
+ unreachable();
+ return NULL;
+}
+
+static char *
+tuple_extract_key_functional_raw(const char *data, const char *data_end,
+ struct key_def *key_def, int multikey_idx,
+ uint32_t *key_size)
+{
+ (void)data; (void)data_end;
+ (void)key_def; (void)multikey_idx; (void)key_size;
+ unreachable();
+ return NULL;
+}
+
/**
* Initialize tuple_extract_key() and tuple_extract_key_raw()
*/
@@ -406,7 +427,17 @@ key_def_set_extract_func(struct key_def *key_def)
bool contains_sequential_parts =
key_def_contains_sequential_parts(key_def);
bool has_optional_parts = key_def->has_optional_parts;
- if (!key_def->has_json_paths) {
+ if (key_def_is_functional(key_def)) {
+ /*
+ * In case of functional index the key
+ * definition has no enough describes the
+ * functional index extractor routine's return
+ * and cannot perform key extraction itself.
+ */
+ key_def->tuple_extract_key = tuple_extract_key_functional;
+ key_def->tuple_extract_key_raw =
+ tuple_extract_key_functional_raw;
+ } else if (!key_def->has_json_paths) {
if (!contains_sequential_parts && !has_optional_parts) {
key_def_set_extract_func_plain<false, false>(key_def);
} else if (!contains_sequential_parts && has_optional_parts) {
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 9e731d137..bf808c786 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -701,6 +701,11 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def)
return -1;
}
}
+ if (index_def_is_functional(index_def)) {
+ diag_set(ClientError, ER_UNSUPPORTED, "Vinyl",
+ "functional indexes");
+ return -1;
+ }
return 0;
}
@@ -985,6 +990,8 @@ vinyl_index_def_change_requires_rebuild(struct index *index,
if (!old_def->opts.is_unique && new_def->opts.is_unique)
return true;
+ if (index_def_is_functional(old_def) != index_def_is_functional(new_def))
+ return true;
assert(index_depends_on_pk(index));
const struct key_def *old_cmp_def = old_def->cmp_def;
@@ -3163,7 +3170,7 @@ vy_send_lsm(struct vy_join_ctx *ctx, struct vy_lsm_recovery_info *lsm_info)
/* Create key definition and tuple format. */
ctx->key_def = key_def_new(lsm_info->key_parts,
- lsm_info->key_part_count);
+ lsm_info->key_part_count, false);
if (ctx->key_def == NULL)
goto out;
ctx->format = vy_stmt_format_new(&ctx->env->stmt_env, &ctx->key_def, 1,
diff --git a/test/box/bitset.result b/test/box/bitset.result
index 78f74ec37..e761bb007 100644
--- a/test/box/bitset.result
+++ b/test/box/bitset.result
@@ -1996,3 +1996,27 @@ _ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}}
s:drop()
---
...
+-- Bitset index cannot be functional.
+s = box.schema.space.create('withdata')
+---
+...
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+---
+...
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+_ = s:create_index('idx', {type = 'bitset', functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Can''t create or modify index ''idx'' in space ''withdata'': BITSET index
+ cannot be functional'
+...
+s:drop()
+---
+...
+box.schema.func.drop('sum')
+---
+...
diff --git a/test/box/bitset.test.lua b/test/box/bitset.test.lua
index eb013a1c0..a6d8e26e8 100644
--- a/test/box/bitset.test.lua
+++ b/test/box/bitset.test.lua
@@ -153,3 +153,12 @@ s = box.schema.space.create('test')
_ = s:create_index('primary')
_ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}})
s:drop()
+
+-- Bitset index cannot be functional.
+s = box.schema.space.create('withdata')
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+_ = s:create_index('pk')
+_ = s:create_index('idx', {type = 'bitset', functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+s:drop()
+box.schema.func.drop('sum')
diff --git a/test/box/hash.result b/test/box/hash.result
index 9f08c49b8..a6b797089 100644
--- a/test/box/hash.result
+++ b/test/box/hash.result
@@ -847,3 +847,27 @@ _ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}})
s:drop()
---
...
+-- Hash index cannot be functional.
+s = box.schema.space.create('withdata')
+---
+...
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+---
+...
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+_ = s:create_index('idx', {type = 'hash', functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Can''t create or modify index ''idx'' in space ''withdata'': HASH index
+ cannot be functional'
+...
+s:drop()
+---
+...
+box.schema.func.drop('sum')
+---
+...
diff --git a/test/box/hash.test.lua b/test/box/hash.test.lua
index 9801873c4..c6a379460 100644
--- a/test/box/hash.test.lua
+++ b/test/box/hash.test.lua
@@ -353,3 +353,12 @@ s = box.schema.space.create('test')
_ = s:create_index('primary')
_ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}})
s:drop()
+
+-- Hash index cannot be functional.
+s = box.schema.space.create('withdata')
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+_ = s:create_index('pk')
+_ = s:create_index('idx', {type = 'hash', functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+s:drop()
+box.schema.func.drop('sum')
diff --git a/test/box/misc.result b/test/box/misc.result
index ee51283bb..fe7f1dbe8 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -524,6 +524,7 @@ t;
192: box.error.INDEX_DEF_UNSUPPORTED
193: box.error.CK_DEF_UNSUPPORTED
194: box.error.MULTIKEY_INDEX_MISMATCH
+ 195: box.error.FUNCTIONAL_EXTRACTOR_ERROR
...
test_run:cmd("setopt delimiter ''");
---
diff --git a/test/box/rtree_misc.result b/test/box/rtree_misc.result
index 6e48bacc7..dd62cf0c7 100644
--- a/test/box/rtree_misc.result
+++ b/test/box/rtree_misc.result
@@ -682,3 +682,27 @@ _ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}})
s:drop()
---
...
+-- Rtree index cannot be functional.
+s = box.schema.space.create('withdata')
+---
+...
+lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]]
+---
+...
+box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+_ = s:create_index('idx', {type = 'rtree', functional_fid = box.func.fextract.id, parts = {{1, 'array'}}})
+---
+- error: 'Can''t create or modify index ''idx'' in space ''withdata'': RTREE index
+ cannot be functional'
+...
+s:drop()
+---
+...
+box.schema.func.drop('fextract')
+---
+...
diff --git a/test/box/rtree_misc.test.lua b/test/box/rtree_misc.test.lua
index 000a928e8..8f2687110 100644
--- a/test/box/rtree_misc.test.lua
+++ b/test/box/rtree_misc.test.lua
@@ -243,3 +243,12 @@ s = box.schema.space.create('test')
_ = s:create_index('primary')
_ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}})
s:drop()
+
+-- Rtree index cannot be functional.
+s = box.schema.space.create('withdata')
+lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]]
+box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, returns = 'array'})
+_ = s:create_index('pk')
+_ = s:create_index('idx', {type = 'rtree', functional_fid = box.func.fextract.id, parts = {{1, 'array'}}})
+s:drop()
+box.schema.func.drop('fextract')
diff --git a/test/engine/engine.cfg b/test/engine/engine.cfg
index 9f07629b4..826e8eb1f 100644
--- a/test/engine/engine.cfg
+++ b/test/engine/engine.cfg
@@ -2,7 +2,10 @@
"*": {
"memtx": {"engine": "memtx"},
"vinyl": {"engine": "vinyl"}
- }
+ },
+ "functional.test.lua": {
+ "memtx": {"engine": "memtx"}
+ }
}
diff --git a/test/engine/functional.result b/test/engine/functional.result
new file mode 100644
index 000000000..b463fa5f5
--- /dev/null
+++ b/test/engine/functional.result
@@ -0,0 +1,620 @@
+test_run = require('test_run').new()
+---
+...
+engine = test_run:get_cfg('engine')
+---
+...
+--
+-- gh-1260: Funclional indexes.
+--
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+---
+...
+lua_code2 = [[function(tuple) return {{tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]}} end]]
+---
+...
+box.schema.func.create('sum_nonpersistent')
+---
+...
+box.schema.func.create('sum_ivaliddef1', {body = lua_code, returns = 'unsigned'})
+---
+...
+box.schema.func.create('sum_ivaliddef2', {body = lua_code, is_deterministic = true})
+---
+...
+box.schema.func.create('sum_ivaliddef3', {body = lua_code, is_deterministic = true, returns = 'map'})
+---
+...
+box.schema.func.create('sum_ivaliddef4', {body = lua_code, is_deterministic = true, returns = 'string'})
+---
+...
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+---
+...
+box.schema.func.create('sums', {body = lua_code2, is_deterministic = true, returns = 'array'})
+---
+...
+-- Functional index can't be primary.
+_ = s:create_index('idx', {functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Can''t create or modify index ''idx'' in space ''withdata'': primary key
+ cannot be functional'
+...
+pk = s:create_index('pk')
+---
+...
+-- Invalid fid.
+_ = s:create_index('idx', {functional_fid = 6666, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t exist'
+...
+-- Can't use non-persistent function in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_nonpersistent.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use non-deterministic function in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef1.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use function that may return everything in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef2.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use function that returns map in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef3.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use function that returns a scalar is incompatable with 'flat' returned key definiton.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef4.id, parts = {{1, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use functin that returns a scalar is incompatable with non-'flat' returned key definition.
+_ = s:create_index('idx', {functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use non-sequential parts in returned key definition.
+_ = s:create_index('idx', {functional_fid = box.func.sums.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use parts started not by 1 field.
+_ = s:create_index('idx', {functional_fid = box.func.sums.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't use JSON paths in returned key definiton.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef4.id, parts = {{"[1]data", 'unsigned'}}})
+---
+- error: 'Wrong index options (field 4): referenced function doesn''t satisfy functional
+ index constraints'
+...
+-- Can't drop or replace function referenced by functional index.
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+---
+...
+box.schema.func.drop('sum')
+---
+- error: 'Can''t drop function 1: function has references'
+...
+box.space._func:replace({7, 1, 'sum', 0, 'LUA', 'function(tuple) return tuple[1] + tuple[2] end', 'integer', true})
+---
+- error: 'Can''t drop function 1: function has references'
+...
+box.snapshot()
+---
+- ok
+...
+test_run:cmd("restart server default")
+box.schema.func.drop('sum')
+---
+- error: 'Can''t drop function 1: function has references'
+...
+box.space._func:replace({7, 1, 'sum', 0, 'LUA', 'function(tuple) return tuple[1] + tuple[2] end', 'integer', true})
+---
+- error: 'Can''t drop function 1: function has references'
+...
+s = box.space.withdata
+---
+...
+idx = s.index.idx
+---
+...
+idx:drop()
+---
+...
+box.schema.func.drop('sum')
+---
+...
+test_run = require('test_run').new()
+---
+...
+engine = test_run:get_cfg('engine')
+---
+...
+-- Invalid functional index extractor routine return (type).
+lua_code = [[function(tuple) return "hello" end]]
+---
+...
+box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+---
+...
+idx = s:create_index('idx', {functional_fid = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}})
+---
+...
+s:insert({1})
+---
+- error: 'Functional index extractor error: returned type doesn''t follow the definition'
+...
+idx:drop()
+---
+...
+-- Invalid functional index extractor routine return (keys).
+lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]]
+---
+...
+box.schema.func.create('invalidreturn2', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+idx = s:create_index('idx', {functional_fid = box.func.invalidreturn2.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}})
+---
+...
+s:insert({1})
+---
+- error: 'Functional index extractor error: returned key ["hello", "world"] doesn''t
+ follow the definition'
+...
+idx:drop()
+---
+...
+-- Invalid functional extractor: runtime extractor error
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+lua_code = [[function(tuple)
+ local json = require('json')
+ return json.encode(tuple)
+ end]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, returns = 'string'})
+---
+...
+idx = s:create_index('idx', {functional_fid = box.func.runtimeerror.id, parts = {{1, 'string'}}})
+---
+...
+s:insert({1})
+---
+- error: 'Functional index extractor error: [string "return function(tuple) local
+ ..."]:1: attempt to call global ''require'' (a nil value)'
+...
+idx:drop()
+---
+...
+-- Remove old persistent functions
+for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end
+---
+...
+s:drop()
+---
+...
+-- Functional test cases.
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'integer'})
+---
+...
+pk = s:create_index('pk')
+---
+...
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+---
+...
+s:insert({1, 2})
+---
+- [1, 2]
+...
+s:insert({2, 1})
+---
+- error: Duplicate key exists in unique index 'idx' in space 'withdata'
+...
+idx:get(3)
+---
+- [1, 2]
+...
+idx:delete(3)
+---
+- [1, 2]
+...
+s:select()
+---
+- []
+...
+s:insert({2, 1})
+---
+- [2, 1]
+...
+idx:get(3)
+---
+- [2, 1]
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
+-- Multikey functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] + tuple[2], tuple[1]} end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+pk = s:create_index('pk')
+---
+...
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+---
+...
+s:insert({1, 2})
+---
+- [1, 2]
+...
+s:insert({3, 5})
+---
+- error: Duplicate key exists in unique index 'idx' in space 'withdata'
+...
+s:insert({5, 3})
+---
+- [5, 3]
+...
+idx:select()
+---
+- - [1, 2]
+ - [1, 2]
+ - [5, 3]
+ - [5, 3]
+...
+idx:get(8)
+---
+- [5, 3]
+...
+idx:get(3)
+---
+- [1, 2]
+...
+idx:get(1)
+---
+- [1, 2]
+...
+idx:get(5)
+---
+- [5, 3]
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
+-- Multikey multipart functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+lua_code = [[function(tuple) return {{600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]}} end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+pk = s:create_index('pk')
+---
+...
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}, {2, 'integer'}}})
+---
+...
+s:insert({1, 2})
+---
+- [1, 2]
+...
+s:insert({2, 1})
+---
+- [2, 1]
+...
+s:insert({3, 3})
+---
+- [3, 3]
+...
+idx:select({600}, {iterator = "GE"})
+---
+- - [1, 2]
+ - [2, 1]
+ - [3, 3]
+...
+idx:get({603, 603})
+---
+- [3, 3]
+...
+idx:select({503}, {iterator = "LE"})
+---
+- - [3, 3]
+ - [2, 1]
+ - [1, 2]
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
+-- Multikey non-unique functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+lua_code = [[function(tuple) return {500 + tuple[1], 500 + tuple[2], 500 + tuple[2]} end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+pk = s:create_index('pk')
+---
+...
+idx = s:create_index('idx', {unique = false, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+---
+...
+s:insert({1, 2})
+---
+- [1, 2]
+...
+s:insert({2, 1})
+---
+- [2, 1]
+...
+idx:select({501})
+---
+- - [1, 2]
+ - [2, 1]
+...
+idx:select({502})
+---
+- - [1, 2]
+ - [2, 1]
+...
+s:replace({1, 3})
+---
+- [1, 3]
+...
+idx:select({501})
+---
+- - [1, 3]
+ - [2, 1]
+...
+idx:select({502})
+---
+- - [2, 1]
+...
+idx:select({503})
+---
+- - [1, 3]
+...
+box.snapshot()
+---
+- ok
+...
+test_run:cmd("restart server default")
+s = box.space.withdata
+---
+...
+idx = s.index.idx
+---
+...
+idx:select({501})
+---
+- - [1, 3]
+ - [2, 1]
+...
+idx:select({502})
+---
+- - [2, 1]
+...
+idx:select({503})
+---
+- - [1, 3]
+...
+s:replace({1, 2})
+---
+- [1, 2]
+...
+idx:select({501})
+---
+- - [1, 2]
+ - [2, 1]
+...
+idx:select({502})
+---
+- - [1, 2]
+ - [2, 1]
+...
+idx:select({503})
+---
+- []
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
+-- Multikey UTF-8 address extractor
+test_run = require('test_run').new()
+---
+...
+engine = test_run:get_cfg('engine')
+---
+...
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+s:format({{name = 'name', type = 'string'}, {name = 'address', type = 'string'}})
+---
+...
+pk = s:create_index('name', {parts = {1, 'string'}})
+---
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+lua_code = [[function(tuple)
+ local address = string.split(tuple.address)
+ local ret = {}
+ for _, v in pairs(address) do table.insert(ret, utf8.upper(v)) end
+ return ret
+ end]]
+test_run:cmd("setopt delimiter ''");
+---
+...
+box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+idx = s:create_index('addr', {unique = false, functional_fid = box.func.addr_extractor.id, parts = {{1, 'string'}}})
+---
+...
+s:insert({"James", "SIS Building Lambeth London UK"})
+---
+- ['James', 'SIS Building Lambeth London UK']
+...
+s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
+---
+- ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
+...
+idx:select('UK')
+---
+- - ['James', 'SIS Building Lambeth London UK']
+ - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK']
+...
+idx:select('SIS')
+---
+- - ['James', 'SIS Building Lambeth London UK']
+...
+s:drop()
+---
+...
+box.schema.func.drop('addr_extractor')
+---
+...
+-- Partial index using functional index extractor
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+pk = s:create_index('pk')
+---
+...
+lua_code = [[function(tuple) if tuple[1] % 2 == 0 then return {} else return {tuple[1]} end end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+---
+...
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+---
+...
+s:insert({1})
+---
+- [1]
+...
+s:insert({2})
+---
+- [2]
+...
+s:insert({3})
+---
+- [3]
+...
+s:insert({4})
+---
+- [4]
+...
+idx:select()
+---
+- - [1]
+ - [3]
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
+-- Return nil from functional index extractor.
+s = box.schema.space.create('withdata', {engine = engine})
+---
+...
+pk = s:create_index('pk')
+---
+...
+lua_code = [[function(tuple) return nil end]]
+---
+...
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'integer'})
+---
+...
+idx = s:create_index('idx', {unique = false, functional_fid = box.func.extract.id, parts = {{1, 'integer', is_nullable = true}}})
+---
+...
+s:insert({1})
+---
+- [1]
+...
+s:insert({2})
+---
+- [2]
+...
+s:insert({3})
+---
+- [3]
+...
+s:insert({4})
+---
+- [4]
+...
+idx:select()
+---
+- - [1]
+ - [2]
+ - [3]
+ - [4]
+...
+s:drop()
+---
+...
+box.schema.func.drop('extract')
+---
+...
diff --git a/test/engine/functional.test.lua b/test/engine/functional.test.lua
new file mode 100644
index 000000000..3421c3ce4
--- /dev/null
+++ b/test/engine/functional.test.lua
@@ -0,0 +1,214 @@
+test_run = require('test_run').new()
+engine = test_run:get_cfg('engine')
+
+--
+-- gh-1260: Funclional indexes.
+--
+s = box.schema.space.create('withdata', {engine = engine})
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+lua_code2 = [[function(tuple) return {{tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]}} end]]
+box.schema.func.create('sum_nonpersistent')
+box.schema.func.create('sum_ivaliddef1', {body = lua_code, returns = 'unsigned'})
+box.schema.func.create('sum_ivaliddef2', {body = lua_code, is_deterministic = true})
+box.schema.func.create('sum_ivaliddef3', {body = lua_code, is_deterministic = true, returns = 'map'})
+box.schema.func.create('sum_ivaliddef4', {body = lua_code, is_deterministic = true, returns = 'string'})
+
+box.schema.func.create('sum', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+box.schema.func.create('sums', {body = lua_code2, is_deterministic = true, returns = 'array'})
+
+-- Functional index can't be primary.
+_ = s:create_index('idx', {functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+pk = s:create_index('pk')
+-- Invalid fid.
+_ = s:create_index('idx', {functional_fid = 6666, parts = {{1, 'unsigned'}}})
+-- Can't use non-persistent function in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_nonpersistent.id, parts = {{1, 'unsigned'}}})
+-- Can't use non-deterministic function in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef1.id, parts = {{1, 'unsigned'}}})
+-- Can't use function that may return everything in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef2.id, parts = {{1, 'unsigned'}}})
+-- Can't use function that returns map in functional index.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef3.id, parts = {{1, 'unsigned'}}})
+-- Can't use function that returns a scalar is incompatable with 'flat' returned key definiton.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef4.id, parts = {{1, 'unsigned'}}})
+-- Can't use functin that returns a scalar is incompatable with non-'flat' returned key definition.
+_ = s:create_index('idx', {functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}})
+-- Can't use non-sequential parts in returned key definition.
+_ = s:create_index('idx', {functional_fid = box.func.sums.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}})
+-- Can't use parts started not by 1 field.
+_ = s:create_index('idx', {functional_fid = box.func.sums.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}})
+-- Can't use JSON paths in returned key definiton.
+_ = s:create_index('idx', {functional_fid = box.func.sum_ivaliddef4.id, parts = {{"[1]data", 'unsigned'}}})
+
+-- Can't drop or replace function referenced by functional index.
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.sum.id, parts = {{1, 'unsigned'}}})
+box.schema.func.drop('sum')
+box.space._func:replace({7, 1, 'sum', 0, 'LUA', 'function(tuple) return tuple[1] + tuple[2] end', 'integer', true})
+box.snapshot()
+test_run:cmd("restart server default")
+box.schema.func.drop('sum')
+box.space._func:replace({7, 1, 'sum', 0, 'LUA', 'function(tuple) return tuple[1] + tuple[2] end', 'integer', true})
+s = box.space.withdata
+idx = s.index.idx
+idx:drop()
+box.schema.func.drop('sum')
+
+test_run = require('test_run').new()
+engine = test_run:get_cfg('engine')
+
+-- Invalid functional index extractor routine return (type).
+lua_code = [[function(tuple) return "hello" end]]
+box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, returns = 'unsigned'})
+idx = s:create_index('idx', {functional_fid = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}})
+s:insert({1})
+idx:drop()
+
+-- Invalid functional index extractor routine return (keys).
+lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]]
+box.schema.func.create('invalidreturn2', {body = lua_code, is_deterministic = true, returns = 'array'})
+idx = s:create_index('idx', {functional_fid = box.func.invalidreturn2.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}})
+s:insert({1})
+idx:drop()
+
+-- Invalid functional extractor: runtime extractor error
+test_run:cmd("setopt delimiter ';'")
+lua_code = [[function(tuple)
+ local json = require('json')
+ return json.encode(tuple)
+ end]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, returns = 'string'})
+idx = s:create_index('idx', {functional_fid = box.func.runtimeerror.id, parts = {{1, 'string'}}})
+s:insert({1})
+idx:drop()
+
+-- Remove old persistent functions
+for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end
+s:drop()
+
+-- Functional test cases.
+s = box.schema.space.create('withdata', {engine = engine})
+lua_code = [[function(tuple) return tuple[1] + tuple[2] end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'integer'})
+pk = s:create_index('pk')
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+s:insert({1, 2})
+s:insert({2, 1})
+idx:get(3)
+idx:delete(3)
+s:select()
+s:insert({2, 1})
+idx:get(3)
+s:drop()
+box.schema.func.drop('extract')
+
+-- Multikey functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] + tuple[2], tuple[1]} end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+pk = s:create_index('pk')
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+s:insert({1, 2})
+s:insert({3, 5})
+s:insert({5, 3})
+idx:select()
+idx:get(8)
+idx:get(3)
+idx:get(1)
+idx:get(5)
+s:drop()
+box.schema.func.drop('extract')
+
+-- Multikey multipart functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+lua_code = [[function(tuple) return {{600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]}} end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+pk = s:create_index('pk')
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}, {2, 'integer'}}})
+s:insert({1, 2})
+s:insert({2, 1})
+s:insert({3, 3})
+idx:select({600}, {iterator = "GE"})
+idx:get({603, 603})
+idx:select({503}, {iterator = "LE"})
+s:drop()
+box.schema.func.drop('extract')
+
+-- Multikey non-unique functional index.
+s = box.schema.space.create('withdata', {engine = engine})
+lua_code = [[function(tuple) return {500 + tuple[1], 500 + tuple[2], 500 + tuple[2]} end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+pk = s:create_index('pk')
+idx = s:create_index('idx', {unique = false, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+s:insert({1, 2})
+s:insert({2, 1})
+idx:select({501})
+idx:select({502})
+s:replace({1, 3})
+idx:select({501})
+idx:select({502})
+idx:select({503})
+box.snapshot()
+test_run:cmd("restart server default")
+s = box.space.withdata
+idx = s.index.idx
+idx:select({501})
+idx:select({502})
+idx:select({503})
+s:replace({1, 2})
+idx:select({501})
+idx:select({502})
+idx:select({503})
+s:drop()
+box.schema.func.drop('extract')
+
+-- Multikey UTF-8 address extractor
+test_run = require('test_run').new()
+engine = test_run:get_cfg('engine')
+s = box.schema.space.create('withdata', {engine = engine})
+s:format({{name = 'name', type = 'string'}, {name = 'address', type = 'string'}})
+pk = s:create_index('name', {parts = {1, 'string'}})
+test_run:cmd("setopt delimiter ';'")
+lua_code = [[function(tuple)
+ local address = string.split(tuple.address)
+ local ret = {}
+ for _, v in pairs(address) do table.insert(ret, utf8.upper(v)) end
+ return ret
+ end]]
+test_run:cmd("setopt delimiter ''");
+box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, returns = 'array'})
+idx = s:create_index('addr', {unique = false, functional_fid = box.func.addr_extractor.id, parts = {{1, 'string'}}})
+s:insert({"James", "SIS Building Lambeth London UK"})
+s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"})
+idx:select('UK')
+idx:select('SIS')
+s:drop()
+box.schema.func.drop('addr_extractor')
+
+-- Partial index using functional index extractor
+s = box.schema.space.create('withdata', {engine = engine})
+pk = s:create_index('pk')
+lua_code = [[function(tuple) if tuple[1] % 2 == 0 then return {} else return {tuple[1]} end end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'array'})
+idx = s:create_index('idx', {unique = true, functional_fid = box.func.extract.id, parts = {{1, 'integer'}}})
+s:insert({1})
+s:insert({2})
+s:insert({3})
+s:insert({4})
+idx:select()
+s:drop()
+box.schema.func.drop('extract')
+
+-- Return nil from functional index extractor.
+s = box.schema.space.create('withdata', {engine = engine})
+pk = s:create_index('pk')
+lua_code = [[function(tuple) return nil end]]
+box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'integer'})
+idx = s:create_index('idx', {unique = false, functional_fid = box.func.extract.id, parts = {{1, 'integer', is_nullable = true}}})
+s:insert({1})
+s:insert({2})
+s:insert({3})
+s:insert({4})
+idx:select()
+s:drop()
+box.schema.func.drop('extract')
diff --git a/test/unit/luaT_tuple_new.c b/test/unit/luaT_tuple_new.c
index 0a16fa039..609d64e45 100644
--- a/test/unit/luaT_tuple_new.c
+++ b/test/unit/luaT_tuple_new.c
@@ -124,7 +124,7 @@ test_basic(struct lua_State *L)
part.nullable_action = ON_CONFLICT_ACTION_DEFAULT;
part.sort_order = SORT_ORDER_ASC;
part.path = NULL;
- struct key_def *key_def = key_def_new(&part, 1);
+ struct key_def *key_def = key_def_new(&part, 1, false);
box_tuple_format_t *another_format = box_tuple_format_new(&key_def, 1);
key_def_delete(key_def);
diff --git a/test/unit/merger.test.c b/test/unit/merger.test.c
index b4a989a20..345a2364e 100644
--- a/test/unit/merger.test.c
+++ b/test/unit/merger.test.c
@@ -214,7 +214,7 @@ test_merger(struct tuple_format *format)
merge_source_array_new(true),
};
- struct key_def *key_def = key_def_new(&key_part_unsigned, 1);
+ struct key_def *key_def = key_def_new(&key_part_unsigned, 1, false);
struct merge_source *merger = merger_new(key_def, sources, source_count,
false);
key_def_delete(key_def);
@@ -252,7 +252,7 @@ test_basic()
plan(4);
header();
- struct key_def *key_def = key_def_new(&key_part_integer, 1);
+ struct key_def *key_def = key_def_new(&key_part_integer, 1, false);
struct tuple_format *format = box_tuple_format_new(&key_def, 1);
assert(format != NULL);
--
2.21.0
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 0/8] box: functional indexes
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (7 preceding siblings ...)
2019-05-30 10:45 ` [PATCH v1 8/8] box: introduce functional indexes in memtx Kirill Shcherbatov
@ 2019-05-30 11:18 ` Vladislav Shpilevoy
2019-05-30 11:43 ` [tarantool-patches] " Kirill Shcherbatov
2019-05-30 11:47 ` [tarantool-patches] " Konstantin Osipov
2019-06-03 15:55 ` Vladimir Davydov
10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-30 11:18 UTC (permalink / raw)
To: tarantool-patches, Kirill Shcherbatov, vdavydov.dev
Hi! Looks promising!
I am not a reviewer, but I've found an error
on travis: https://travis-ci.org/tarantool/tarantool/jobs/539160331
On 30/05/2019 13:45, Kirill Shcherbatov wrote:
> The patch set introduces persistent Lua functions and functional
> indexes.
>
> Lua functions managed by Tarantool are called 'persistent'.
> They are stored in snapshoot and are available after server
> restart.
> Persistent Lua functions are exported in box.schema.func.persistent
> folder. They may be called via .call exported object method or
> via netbox:call.
>
> Some restrictions must be accounted writing such routines:
> 1. only limited set of Lua functions and modules are available:
> -assert -error -pairs -ipairs -next -pcall -xpcall -type
> -print -select -string -tonumber -tostring -unpack -math -utf8;
> 2. global variables are forbidden.
>
> Moreover all functions and their metadata are now exported in
> box.func table (not only persistent). Persistent Lua functions
> have box.func.FUNCNAME.call() method to execute it in interactive
> console.
> Persistent Lua functions are also available with net.box call.
>
>
> A functional index use some registered persistent function as
> key extractor.
> You are not allowed to change functional index extractor function
> while there are some functional indexes depends of it.
>
> Branch: http://github.com/tarantool/tarantool/tree/kshch/gh-1260-functional-index-new
> Issue: https://github.com/tarantool/tarantool/issues/1260
>
> Kirill Shcherbatov (8):
> box: refactor box_lua_find helper
> box: rework func cache update machinery
> schema: rework _func system space format
> box: load persistent Lua functions on creation
> netbox: call persistent functions in netbox
> box: export _func functions with box.func folder
> box: introduce memtx_slab_alloc helper
> box: introduce functional indexes in memtx
>
> src/box/CMakeLists.txt | 1 +
> src/box/alter.cc | 175 +++++++--
> src/box/bootstrap.snap | Bin 4393 -> 4447 bytes
> src/box/call.c | 2 +-
> src/box/errcode.h | 1 +
> src/box/func.c | 203 +++++++++-
> src/box/func.h | 38 +-
> src/box/func_def.h | 28 +-
> src/box/index_def.c | 22 +-
> src/box/index_def.h | 16 +
> src/box/key_def.c | 13 +-
> src/box/key_def.h | 31 +-
> src/box/lua/call.c | 121 +++---
> src/box/lua/call.h | 3 +-
> src/box/lua/init.c | 2 +
> src/box/lua/key_def.c | 2 +-
> src/box/lua/schema.lua | 13 +-
> src/box/lua/space.cc | 5 +
> src/box/lua/tuple_fextract_key.c | 233 +++++++++++
> src/box/lua/tuple_fextract_key.h | 100 +++++
> src/box/lua/upgrade.lua | 22 +-
> src/box/memtx_engine.c | 44 ++-
> src/box/memtx_engine.h | 19 +
> src/box/memtx_space.c | 18 +
> src/box/memtx_tree.c | 355 ++++++++++++++++-
> src/box/schema.cc | 49 ++-
> src/box/schema.h | 31 +-
> src/box/schema_def.h | 3 +
> src/box/sql.c | 2 +-
> src/box/sql/build.c | 2 +-
> src/box/sql/select.c | 2 +-
> src/box/sql/where.c | 2 +-
> src/box/tuple_compare.cc | 112 +++++-
> src/box/tuple_extract_key.cc | 33 +-
> src/box/vinyl.c | 9 +-
> src/lua/utils.c | 126 ++++++
> src/lua/utils.h | 19 +
> test/box-py/bootstrap.result | 6 +-
> test/box/access_misc.result | 6 +-
> test/box/bitset.result | 24 ++
> test/box/bitset.test.lua | 9 +
> test/box/hash.result | 24 ++
> test/box/hash.test.lua | 9 +
> test/box/misc.result | 2 +
> test/box/persistent_func.result | 201 ++++++++++
> test/box/persistent_func.test.lua | 88 +++++
> test/box/rtree_misc.result | 24 ++
> test/box/rtree_misc.test.lua | 9 +
> test/engine/engine.cfg | 5 +-
> test/engine/functional.result | 620 ++++++++++++++++++++++++++++++
> test/engine/functional.test.lua | 214 +++++++++++
> test/unit/luaT_tuple_new.c | 2 +-
> test/unit/merger.test.c | 4 +-
> 53 files changed, 2898 insertions(+), 206 deletions(-)
> create mode 100644 src/box/lua/tuple_fextract_key.c
> create mode 100644 src/box/lua/tuple_fextract_key.h
> create mode 100644 test/box/persistent_func.result
> create mode 100644 test/box/persistent_func.test.lua
> create mode 100644 test/engine/functional.result
> create mode 100644 test/engine/functional.test.lua
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v1 0/8] box: functional indexes
2019-05-30 11:18 ` [tarantool-patches] [PATCH v1 0/8] box: functional indexes Vladislav Shpilevoy
@ 2019-05-30 11:43 ` Kirill Shcherbatov
0 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-05-30 11:43 UTC (permalink / raw)
To: tarantool-patches, Vladislav Shpilevoy; +Cc: vdavydov.dev
On 30.05.2019 14:18, Vladislav Shpilevoy wrote:
> Hi! Looks promising!
>
> I am not a reviewer, but I've found an error
> on travis: https://travis-ci.org/tarantool/tarantool/jobs/539160331
Hi! Thank you for the tip. It was unused variable there. Fixed.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 0/8] box: functional indexes
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (8 preceding siblings ...)
2019-05-30 11:18 ` [tarantool-patches] [PATCH v1 0/8] box: functional indexes Vladislav Shpilevoy
@ 2019-05-30 11:47 ` Konstantin Osipov
2019-06-03 15:55 ` Vladimir Davydov
10 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-30 11:47 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/05/30 14:12]:
> The patch set introduces persistent Lua functions and functional
> indexes.
Thank you for the patch.
I asked before and continue to ask for an RFC for persistent
functions to be sent to the list for review.
After the RFC is reviewed some substantial changes may be
required to the patch and it's better to not waste time on it.
Thanks,
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 3/8] schema: rework _func system space format
2019-05-30 10:45 ` [PATCH v1 3/8] schema: rework _func system space format Kirill Shcherbatov
@ 2019-05-30 12:06 ` Konstantin Osipov
2019-06-03 16:14 ` Vladimir Davydov
2019-05-30 13:10 ` Vladislav Shpilevoy
1 sibling, 1 reply; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-30 12:06 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/05/30 14:12]:
> This patch updates a _func system space to prepare it for
> working with persistent functions. The format of the _func
> system space is
> [<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
Why is setuid uint?
> <language> STR, <body> STR, <returns> STR,
> <is_deterministic> BOOL]
Shouldn't you add 'options' to the format right away, to make life
easier?
What about input type and return types, parameter count?
This data is useful in SQL.
I wonder why I have to comment on the patch... Ah, I guess because
there is still no approved specification.
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 3/8] schema: rework _func system space format
2019-05-30 10:45 ` [PATCH v1 3/8] schema: rework _func system space format Kirill Shcherbatov
2019-05-30 12:06 ` [tarantool-patches] " Konstantin Osipov
@ 2019-05-30 13:10 ` Vladislav Shpilevoy
1 sibling, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-30 13:10 UTC (permalink / raw)
To: tarantool-patches, Kirill Shcherbatov, vdavydov.dev
On 30/05/2019 13:45, Kirill Shcherbatov wrote:
> This patch updates a _func system space to prepare it for
> working with persistent functions. The format of the _func
> system space is
> [<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
> <language> STR, <body> STR, <returns> STR,
> <is_deterministic> BOOL]
As I understand, a user always should set a function
return type. Even for Lua functions (the only supported
language now, but SQL is coming).
In case of a functional index it should be exactly the same
type as of the index key. For example, if a functional index
has an alone unsigned key part, the function should return
exactly unsigned.
But what about alter? What if I had unsigned function and
key part, but decided to make it 'number' or 'scalar'? I can
do that with normal indexes and space formats, even without
index rebuild or check, for free. In two steps.
Or what if I had 'number', but decided to change it to 'scalar'?
Again, I can do that with index parts and space format, despite
non-transactional DDL, in two steps. Of course, all the tuples
are checked to conform the new stricter format, but it works, and
it is muuuch cheeper than recreate an index.
In your case alter is not possible at all. A user need to drop
the functional index, change type in space format, and create
the index again. Even if type change was 'unsigned' -> 'number'.
It does not match our 'Tarantool is schema less' feature.
Does not look ok, IMO. You could reuse the 'compatible types'
mechanism we use for gradual alter of index parts and space
format.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 4/8] box: load persistent Lua functions on creation
2019-05-30 10:45 ` [PATCH v1 4/8] box: load persistent Lua functions on creation Kirill Shcherbatov
@ 2019-05-31 8:16 ` Konstantin Osipov
2019-06-03 8:26 ` [tarantool-patches] " Kirill Shcherbatov
0 siblings, 1 reply; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-31 8:16 UTC (permalink / raw)
To: tarantool-patches; +Cc: vdavydov.dev, Kirill Shcherbatov
* Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/05/30 14:12]:
Could you please benchmark how much slower is a function
invocation in a fresh sandbox compared to a re-cycled one?
How many invocations per second can we get for a simple ffunction
like return a*a?
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v1 4/8] box: load persistent Lua functions on creation
2019-05-31 8:16 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-03 8:26 ` Kirill Shcherbatov
0 siblings, 0 replies; 20+ messages in thread
From: Kirill Shcherbatov @ 2019-06-03 8:26 UTC (permalink / raw)
To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev
On 31.05.2019 11:16, Konstantin Osipov wrote:
> * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/05/30 14:12]:
>
> Could you please benchmark how much slower is a function
> invocation in a fresh sandbox compared to a re-cycled one?
>
> How many invocations per second can we get for a simple ffunction
> like return a*a?
Hi! I don't quite understand what you asked me to do.
I measured performance in the following scenarios:
Case 1. Function load
lua_code = [[function(tuple) return tuple[1]*tuple[2] end]]
box.schema.func.create('extract', {body = lua_code, is_deterministic = true, returns = 'integer'})
1) function creation doesn't assemble a new sandbox, but makes a deepcopy of existent one (as on branch)
50830 FUNCTION LOADS/SEC
2) function creation assembles a new sandbox (with luaT_prepare_sandbox) and applies it (lua_setfenv)
49817 FUNCTION LOADS/SEC
3) function creation use an existent sandbox and doesn't make it's deepcopy (no monkey-patch protection
between the extractors)
204188 FUNCTION LOADS/SEC
Case 2. (part of the last patch) Function call
"return 1": 914445 CALLS/SEC
"return tuple[1] + tuple[2]": 882009 CALLS/SEC
"sharonovAddrExtractor": 164499 CALLS/SEC
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v1 0/8] box: functional indexes
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
` (9 preceding siblings ...)
2019-05-30 11:47 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-03 15:55 ` Vladimir Davydov
10 siblings, 0 replies; 20+ messages in thread
From: Vladimir Davydov @ 2019-06-03 15:55 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
I won't be commenting on the code this time, because I have a couple of
high-level concerns that might require you to rewrite the whole patch
set from scratch, provided you agree with my suggestions, of course.
1. The code lacks a layer of abstraction that would provide an easy way
to call any persistent function, irrespective of the language it is
written in. Even though we have only two languages, C and Lua, for
now, we don't have a unified infrastructure that would encapsulate a
function call:
- box_c_call => func_call is used for calling C functions via
iproto.
- box_lua_call => execute_lua_call / execute_persistent_function
is used for calling Lua functions via iproto.
- tuple_fextract_key => execute_tuple_fextract_key is used for
calling persistent Lua functions for indexing data.
This is a mess. I doubt we will be able to add persistent SQL
procedures with such a design. We probably need to make func_call()
handle all kinds of functions, irrespective of language, both
persistent and not (make it virtual probably).
2. I don't like the way you made functional indexes so special to memtx
tree implementation, i.e.
- introduced a separate index vtab for memtx_tree
- made func a member of struct memtx_tree rather than key_def
- added tuple_fextract_key instead of using tuple_extract_key
- wrote special comparators for functional indexes
- attached (tuple => func value) hash to memtx_tree rather than
making it global
This makes it nearly impossible to support functional indexes in
Vinyl without copy-and-pasting a lot of code.
I'm inclined to think that the functional index layer should
be completely hidden behind key_def and its methods and thus
independent of engine internals, i.e.
- func should be a member of key_def
- hash (tuple+func => func value) should be global (or we should
have a separate hash per each function); nodes corresponding to
a particular tuple should be freed when the tuple is deleted
- tuple validation against func should be done when a tuple is
validated against a tuple_format, not when it is inserted into
a memtx index, i.e. tuple_format should "know" about functions
its tuples are supposed to be compatible with; we can cache
function values computed during tuple validation in the hash so
that we don't need to recompute them when we insert the tuple
into an index
- tuple_field_by_part should transparently handle functional index
parts by getting the function value from the hash or computing it
if there's no match
- tree hints should store comparison hints in case the index
function returns a single value; if the function returns an
array, tree hints should store multikey offsets, just like for
non-functional indexes; function value should be obtained from
the hash with the aid of tuple_field_by_part
By implementing this generic infrastructure, we'll get functional
index support for all kinds of indexes for free (vinyl will require
some extra work, but it'll be minimal). Although multikey functional
indexes will perform slower due to hash lookup on each comparison,
single value functional indexes will be faster thanks to comparison
hints. I think it's okay. We will also be able to disable function
value caching completely (or almost completely) to save some memory
if we want to (we might add a configuration option for this in
future).
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [tarantool-patches] [PATCH v1 3/8] schema: rework _func system space format
2019-05-30 12:06 ` [tarantool-patches] " Konstantin Osipov
@ 2019-06-03 16:14 ` Vladimir Davydov
0 siblings, 0 replies; 20+ messages in thread
From: Vladimir Davydov @ 2019-06-03 16:14 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: tarantool-patches, Kirill Shcherbatov
On Thu, May 30, 2019 at 03:06:26PM +0300, Konstantin Osipov wrote:
> * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/05/30 14:12]:
> > This patch updates a _func system space to prepare it for
> > working with persistent functions. The format of the _func
> > system space is
> > [<id> UINT, <owner> UINT, <name> STR, <setuid> UINT,
>
> Why is setuid uint?
>
> > <language> STR, <body> STR, <returns> STR,
> > <is_deterministic> BOOL]
>
> Shouldn't you add 'options' to the format right away, to make life
> easier?
+1
I'd also move is_deterministic to 'options', because it is an option
per se.
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v1 6/8] box: export _func functions with box.func folder
2019-05-30 10:45 ` [PATCH v1 6/8] box: export _func functions with box.func folder Kirill Shcherbatov
@ 2019-06-03 16:22 ` Vladimir Davydov
2019-06-03 16:24 ` Vladimir Davydov
1 sibling, 0 replies; 20+ messages in thread
From: Vladimir Davydov @ 2019-06-03 16:22 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, May 30, 2019 at 01:45:33PM +0300, Kirill Shcherbatov wrote:
> Closes #4182
> Needed for #1260
>
> @TarantoolBot document
> Title: Persistent Lua functions
>
> Lua functions managed by Tarantool are called 'persistent'.
> They are stored in snapshoot and are available after server
> restart.
> Persistent Lua functions are exported in box.schema.func.persistent
> folder. They may be called via .call exported object method or
> via netbox:call.
>
> Some restrictions must be accounted writing such routines:
> 1. only limited set of Lua functions and modules are available:
> -assert -error -pairs -ipairs -next -pcall -xpcall -type
> -print -select -string -tonumber -tostring -unpack -math -utf8;
> 2. global variables are forbidden.
>
> Moreover all functions and their metadata are now exported in
> box.func table (not only persistent). Persistent Lua functions
> have box.func.FUNCNAME.call() method to execute it in interactive
> console.
> Persistent Lua functions are also available with net.box call.
>
> Example:
> lua_code = [[function(a, b) return a + b end]]
> box.schema.func.create('sum', {
> body = lua_code, is_deterministic = true,
> returns = 'integer', is_deterministic = true})
> box.func.sum
> - call: 'function: 0x4016b6a0'
> returns: integer
> is_deterministic: true
> id: 2
> language: LUA
> name: sum
> is_persistent: true
> setuid: false
>
> box.func.sum.call(1, 2)
I think 'call' should work for all kinds of functions, not only
persistent. That is, this should probably be the first patch in
the series.
Also, I'd expect 'call' to be a method of the function class:
box.func.sum:call(1, 2)
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH v1 6/8] box: export _func functions with box.func folder
2019-05-30 10:45 ` [PATCH v1 6/8] box: export _func functions with box.func folder Kirill Shcherbatov
2019-06-03 16:22 ` Vladimir Davydov
@ 2019-06-03 16:24 ` Vladimir Davydov
1 sibling, 0 replies; 20+ messages in thread
From: Vladimir Davydov @ 2019-06-03 16:24 UTC (permalink / raw)
To: Kirill Shcherbatov; +Cc: tarantool-patches
On Thu, May 30, 2019 at 01:45:33PM +0300, Kirill Shcherbatov wrote:
> Closes #4182
> Needed for #1260
>
> @TarantoolBot document
> Title: Persistent Lua functions
>
> Lua functions managed by Tarantool are called 'persistent'.
> They are stored in snapshoot and are available after server
> restart.
> Persistent Lua functions are exported in box.schema.func.persistent
> folder. They may be called via .call exported object method or
> via netbox:call.
>
> Some restrictions must be accounted writing such routines:
> 1. only limited set of Lua functions and modules are available:
> -assert -error -pairs -ipairs -next -pcall -xpcall -type
> -print -select -string -tonumber -tostring -unpack -math -utf8;
> 2. global variables are forbidden.
>
> Moreover all functions and their metadata are now exported in
> box.func table (not only persistent). Persistent Lua functions
> have box.func.FUNCNAME.call() method to execute it in interactive
> console.
> Persistent Lua functions are also available with net.box call.
>
> Example:
> lua_code = [[function(a, b) return a + b end]]
> box.schema.func.create('sum', {
> body = lua_code, is_deterministic = true,
> returns = 'integer', is_deterministic = true})
> box.func.sum
> - call: 'function: 0x4016b6a0'
> returns: integer
> is_deterministic: true
> id: 2
> language: LUA
> name: sum
> is_persistent: true
> setuid: false
>
> box.func.sum.call(1, 2)
Also, what about access checks? I may be wrong, but it looks like anyone
can call a function this way in your implementation. I guess we should
check 'execute' permission.
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2019-06-03 16:24 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-30 10:45 [PATCH v1 0/8] box: functional indexes Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 1/8] box: refactor box_lua_find helper Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 2/8] box: rework func cache update machinery Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 3/8] schema: rework _func system space format Kirill Shcherbatov
2019-05-30 12:06 ` [tarantool-patches] " Konstantin Osipov
2019-06-03 16:14 ` Vladimir Davydov
2019-05-30 13:10 ` Vladislav Shpilevoy
2019-05-30 10:45 ` [PATCH v1 4/8] box: load persistent Lua functions on creation Kirill Shcherbatov
2019-05-31 8:16 ` [tarantool-patches] " Konstantin Osipov
2019-06-03 8:26 ` [tarantool-patches] " Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 5/8] netbox: call persistent functions in netbox Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 6/8] box: export _func functions with box.func folder Kirill Shcherbatov
2019-06-03 16:22 ` Vladimir Davydov
2019-06-03 16:24 ` Vladimir Davydov
2019-05-30 10:45 ` [PATCH v1 7/8] box: introduce memtx_slab_alloc helper Kirill Shcherbatov
2019-05-30 10:45 ` [PATCH v1 8/8] box: introduce functional indexes in memtx Kirill Shcherbatov
2019-05-30 11:18 ` [tarantool-patches] [PATCH v1 0/8] box: functional indexes Vladislav Shpilevoy
2019-05-30 11:43 ` [tarantool-patches] " Kirill Shcherbatov
2019-05-30 11:47 ` [tarantool-patches] " Konstantin Osipov
2019-06-03 15:55 ` Vladimir Davydov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox