* [PATCH v4 0/4] box: functional indexes
@ 2019-07-24 7:36 Kirill Shcherbatov
2019-07-24 7:36 ` [PATCH v4 1/4] box: introduce tuple_chunk infrastructure Kirill Shcherbatov
` (4 more replies)
0 siblings, 5 replies; 22+ messages in thread
From: Kirill Shcherbatov @ 2019-07-24 7:36 UTC (permalink / raw)
To: tarantool-patches, kostja; +Cc: vdavydov.dev, Kirill Shcherbatov
This patchset introduces functional indexes in memtx.
Functional index is an index that use user-defined function to extract
a key by tuple.
In current implementation only persistent deterministic sandboxed Lua
function previously created with box.schema.func.create may be used
in functional index.
This provides a potential ability to support new languages to
transparently extend supported extractors in future (e.g. SQL extractor,
C extractor).
Because an _index space is loaded before _func space, functional index
initially created in "disabled state". Then, on loading from a new space
_func_index, it completely rebuilds.
Comment:
- I'll send a follow-up patch to specify function by name a bit later.
Changes in version 4:
- functional indexes are implemented with memtx_tree vtab like
version 1 did
- index object doesn't use hash at all
http://github.com/tarantool/tarantool/tree/kshch/gh-1260-functional-index-new
https://github.com/tarantool/tarantool/issues/1260
Kirill Shcherbatov (4):
box: introduce tuple_chunk infrastructure
box: generalize memtx_multikey_tree methods
box: refactor memtx_tree_delete_identical
box: introduce functional indexes
src/box/alter.h | 1 +
src/box/errcode.h | 1 +
src/box/func_key.h | 117 +++++
src/box/index.h | 9 +
src/box/index_def.h | 16 +
src/box/key_def.h | 22 +-
src/box/schema_def.h | 9 +
src/box/tuple.h | 28 ++
src/box/tuple_format.h | 9 +
src/lib/salad/bps_tree.h | 9 +-
src/box/func_key.c | 135 ++++++
src/box/index_def.c | 21 +-
src/box/key_def.c | 32 +-
src/box/lua/key_def.c | 2 +-
src/box/memtx_engine.c | 29 ++
src/box/memtx_space.c | 18 +
src/box/memtx_tree.c | 334 +++++++++++++-
src/box/sql.c | 2 +-
src/box/sql/build.c | 2 +-
src/box/sql/select.c | 2 +-
src/box/sql/where.c | 2 +-
src/box/tuple.c | 8 +
src/box/vinyl.c | 9 +-
src/box/vy_stmt.c | 2 +
test/unit/luaT_tuple_new.c | 2 +-
test/unit/merger.test.c | 4 +-
src/box/CMakeLists.txt | 1 +
src/box/alter.cc | 107 ++++-
src/box/bootstrap.snap | Bin 5863 -> 5914 bytes
src/box/index.cc | 28 ++
src/box/lua/schema.lua | 20 +
src/box/lua/space.cc | 7 +
src/box/lua/upgrade.lua | 18 +
src/box/schema.cc | 12 +-
src/box/tuple_compare.cc | 104 ++++-
src/box/tuple_extract_key.cc | 29 +-
src/box/tuple_hash.cc | 1 +
test/app-tap/tarantoolctl.test.lua | 4 +-
test/box-py/bootstrap.result | 5 +
test/box/access.result | 3 +
test/box/access.test.lua | 1 +
test/box/access_misc.result | 132 +++---
test/box/access_sysview.result | 6 +-
test/box/alter.result | 7 +-
test/box/bitset.result | 24 +
test/box/bitset.test.lua | 9 +
test/box/hash.result | 24 +
test/box/hash.test.lua | 9 +
test/box/misc.result | 1 +
test/box/rtree_misc.result | 24 +
test/box/rtree_misc.test.lua | 9 +
test/engine/engine.cfg | 5 +-
test/engine/functional.result | 690 +++++++++++++++++++++++++++++
test/engine/functional.test.lua | 233 ++++++++++
test/unit/bps_tree.cc | 8 +-
test/unit/bps_tree.result | 4 +-
test/vinyl/misc.result | 23 +
test/vinyl/misc.test.lua | 9 +
test/wal_off/alter.result | 2 +-
59 files changed, 2251 insertions(+), 133 deletions(-)
create mode 100644 src/box/func_key.h
create mode 100644 src/box/func_key.c
create mode 100644 test/engine/functional.result
create mode 100644 test/engine/functional.test.lua
--
2.22.0
^ permalink raw reply [flat|nested] 22+ messages in thread* [PATCH v4 1/4] box: introduce tuple_chunk infrastructure 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov @ 2019-07-24 7:36 ` Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 2/4] box: generalize memtx_multikey_tree methods Kirill Shcherbatov ` (3 subsequent siblings) 4 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 7:36 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev, Kirill Shcherbatov Introduced a new object tuple_chunk: a memory allocation is associated with given tuple. tuple_format's vtab is extended with few new methods to manage tuple_chunks lifecycle. Implemented corresponding methid for memtx engine: a memory chunks are allocated with memtx's smalloc allocator. Needed for #1260 --- src/box/tuple.h | 28 ++++++++++++++++++++++++++++ src/box/tuple_format.h | 9 +++++++++ src/box/memtx_engine.c | 27 +++++++++++++++++++++++++++ src/box/tuple.c | 8 ++++++++ src/box/vy_stmt.c | 2 ++ 5 files changed, 74 insertions(+) diff --git a/src/box/tuple.h b/src/box/tuple.h index 99dfeb82d..60b6fb474 100644 --- a/src/box/tuple.h +++ b/src/box/tuple.h @@ -447,6 +447,34 @@ tuple_delete(struct tuple *tuple) format->vtab.tuple_delete(format, tuple); } +/** Tuple chunk memory object. */ +struct tuple_chunk { + /** The payload size. Needed to perform memory release.*/ + uint32_t data_sz; + /** Metadata object payload. */ + char data[0]; +}; + +/** Calculate the size of tuple_chunk object by given data_sz. */ +uint32_t +tuple_chunk_sz(uint32_t data_sz); + +/** Allocate a new tuple_chunk for given tuple. */ +static inline struct tuple_chunk * +tuple_chunk_new(struct tuple *tuple, uint32_t data_sz) +{ + struct tuple_format *format = tuple_format(tuple); + return format->vtab.tuple_chunk_new(format, tuple, data_sz); +} + +/** Free a tuple_chunk is allocated for given tuple. */ +static inline void +tuple_chunk_delete(struct tuple *tuple, struct tuple_chunk *tuple_chunk) +{ + struct tuple_format *format = tuple_format(tuple); + format->vtab.tuple_chunk_delete(format, tuple_chunk); +} + /** * Check tuple data correspondence to space format. * Actually, checks everything that is checked by diff --git a/src/box/tuple_format.h b/src/box/tuple_format.h index e4f5f0018..63efbde04 100644 --- a/src/box/tuple_format.h +++ b/src/box/tuple_format.h @@ -64,6 +64,7 @@ enum { TUPLE_INDEX_BASE = 1 }; enum { TUPLE_OFFSET_SLOT_NIL = INT32_MAX }; struct tuple; +struct tuple_chunk; struct tuple_format; struct coll; @@ -82,6 +83,14 @@ struct tuple_format_vtab { struct tuple* (*tuple_new)(struct tuple_format *format, const char *data, const char *end); + /** Free a tuple_chunk allocated for given tuple. */ + void + (*tuple_chunk_delete)(struct tuple_format *format, + struct tuple_chunk *tuple_chunk); + /** Allocate a new tuple_chunk for given tuple. */ + struct tuple_chunk * + (*tuple_chunk_new)(struct tuple_format *format, struct tuple *tuple, + uint32_t data_sz); }; /** Tuple field meta information for tuple_format. */ diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index 428491c2d..fbb3151c9 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -1126,9 +1126,36 @@ memtx_tuple_delete(struct tuple_format *format, struct tuple *tuple) smfree_delayed(&memtx->alloc, memtx_tuple, total); } +void +metmx_tuple_chunk_delete(struct tuple_format *format, + struct tuple_chunk *tuple_chunk) +{ + struct memtx_engine *memtx = (struct memtx_engine *)format->engine; + uint32_t sz = tuple_chunk_sz(tuple_chunk->data_sz); + smfree(&memtx->alloc, tuple_chunk, sz); +} + +struct tuple_chunk * +memtx_tuple_chunk_new(struct tuple_format *format, struct tuple *tuple, + uint32_t data_sz) +{ + struct memtx_engine *memtx = (struct memtx_engine *)format->engine; + uint32_t sz = tuple_chunk_sz(data_sz); + struct tuple_chunk *tuple_chunk = + (struct tuple_chunk *) smalloc(&memtx->alloc, sz); + if (tuple == NULL) { + diag_set(OutOfMemory, sz, "smalloc", "tuple"); + return NULL; + } + tuple_chunk->data_sz = data_sz; + return tuple_chunk; +} + struct tuple_format_vtab memtx_tuple_format_vtab = { memtx_tuple_delete, memtx_tuple_new, + metmx_tuple_chunk_delete, + memtx_tuple_chunk_new, }; /** diff --git a/src/box/tuple.c b/src/box/tuple.c index c0e94d55b..25f85f732 100644 --- a/src/box/tuple.c +++ b/src/box/tuple.c @@ -67,6 +67,8 @@ runtime_tuple_new(struct tuple_format *format, const char *data, const char *end static struct tuple_format_vtab tuple_format_runtime_vtab = { runtime_tuple_delete, runtime_tuple_new, + NULL, + NULL, }; static struct tuple * @@ -785,3 +787,9 @@ mp_str(const char *data) return "<failed to format message pack>"; return buf; } + +uint32_t +tuple_chunk_sz(uint32_t data_sz) +{ + return sizeof(struct tuple_chunk) + data_sz; +} diff --git a/src/box/vy_stmt.c b/src/box/vy_stmt.c index f936cd61f..518a24f43 100644 --- a/src/box/vy_stmt.c +++ b/src/box/vy_stmt.c @@ -118,6 +118,8 @@ vy_stmt_env_create(struct vy_stmt_env *env) { env->tuple_format_vtab.tuple_new = vy_tuple_new; env->tuple_format_vtab.tuple_delete = vy_tuple_delete; + env->tuple_format_vtab.tuple_chunk_new = NULL; + env->tuple_format_vtab.tuple_chunk_delete = NULL; env->max_tuple_size = 1024 * 1024; env->key_format = vy_stmt_format_new(env, NULL, 0, NULL, 0, 0, NULL); if (env->key_format == NULL) -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 2/4] box: generalize memtx_multikey_tree methods 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 1/4] box: introduce tuple_chunk infrastructure Kirill Shcherbatov @ 2019-07-24 7:36 ` Kirill Shcherbatov 2019-07-24 19:24 ` Konstantin Osipov 2019-07-24 7:36 ` [PATCH v4 3/4] box: refactor memtx_tree_delete_identical Kirill Shcherbatov ` (2 subsequent siblings) 4 siblings, 1 reply; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 7:36 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev, Kirill Shcherbatov Updated multikey memtx_tree methods a bit to make possible to reuse them in case of functional indexes. The BPS_TREE_IDENTICAL is only used in bps_tree_delete_identical (when BPS_TREE_NO_DEBUG macro is defined) therefore it may be reused to delete an identical entry in case of functional index in further patches. Needed for #1260 --- src/box/memtx_tree.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index 3edf94044..5f5848c69 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -73,7 +73,7 @@ static bool memtx_tree_data_identical(const struct memtx_tree_data *a, const struct memtx_tree_data *b) { - return a->tuple == b->tuple && a->hint == b->hint; + return a->tuple == b->tuple; } #define BPS_TREE_NAME memtx_tree @@ -85,6 +85,7 @@ memtx_tree_data_identical(const struct memtx_tree_data *a, tuple_compare_with_key((&a)->tuple, (&a)->hint, (b)->key,\ (b)->part_count, (b)->hint, arg) #define BPS_TREE_IDENTICAL(a, b) memtx_tree_data_identical(&a, &b) +#define BPS_TREE_NO_DEBUG 1 #define bps_tree_elem_t struct memtx_tree_data #define bps_tree_key_t struct memtx_tree_key_data * #define bps_tree_arg_t struct key_def * @@ -97,6 +98,7 @@ memtx_tree_data_identical(const struct memtx_tree_data *a, #undef BPS_TREE_COMPARE #undef BPS_TREE_COMPARE_KEY #undef BPS_TREE_IDENTICAL +#undef BPS_TREE_NO_DEBUG #undef bps_tree_elem_t #undef bps_tree_key_t #undef bps_tree_arg_t @@ -591,13 +593,15 @@ memtx_tree_index_replace(struct index *base, struct tuple *old_tuple, static int memtx_tree_index_replace_multikey_one(struct memtx_tree_index *index, struct tuple *old_tuple, struct tuple *new_tuple, - enum dup_replace_mode mode, int multikey_idx, - struct tuple **replaced_tuple) + enum dup_replace_mode mode, hint_t hint, + struct memtx_tree_data *replaced_data, + bool *is_multikey_conflict) { struct memtx_tree_data new_data, dup_data; new_data.tuple = new_tuple; - new_data.hint = multikey_idx; + new_data.hint = hint; dup_data.tuple = NULL; + *is_multikey_conflict = false; if (memtx_tree_insert(&index->tree, new_data, &dup_data) != 0) { diag_set(OutOfMemory, MEMTX_EXTENT_SIZE, "memtx_tree_index", "replace"); @@ -610,7 +614,7 @@ memtx_tree_index_replace_multikey_one(struct memtx_tree_index *index, * times, the previous key occurrence is pushed * out of the index. */ - dup_data.tuple = NULL; + *is_multikey_conflict = true; } else if ((errcode = replace_check_dup(old_tuple, dup_data.tuple, mode)) != 0) { /* Rollback replace. */ @@ -624,7 +628,7 @@ memtx_tree_index_replace_multikey_one(struct memtx_tree_index *index, } return -1; } - *replaced_tuple = dup_data.tuple; + *replaced_data = dup_data; return 0; } @@ -681,16 +685,19 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple, tuple_multikey_count(new_tuple, cmp_def); for (; (uint32_t) multikey_idx < multikey_count; multikey_idx++) { - struct tuple *replaced_tuple; + bool is_multikey_conflict; + struct memtx_tree_data replaced_data; err = memtx_tree_index_replace_multikey_one(index, old_tuple, new_tuple, mode, - multikey_idx, &replaced_tuple); + multikey_idx, &replaced_data, + &is_multikey_conflict); if (err != 0) break; - if (replaced_tuple != NULL) { + if (replaced_data.tuple != NULL && + !is_multikey_conflict) { assert(*result == NULL || - *result == replaced_tuple); - *result = replaced_tuple; + *result == replaced_data.tuple); + *result = replaced_data.tuple; } } if (err != 0) { -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 2/4] box: generalize memtx_multikey_tree methods 2019-07-24 7:36 ` [PATCH v4 2/4] box: generalize memtx_multikey_tree methods Kirill Shcherbatov @ 2019-07-24 19:24 ` Konstantin Osipov 0 siblings, 0 replies; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 19:24 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > Updated multikey memtx_tree methods a bit to make possible to > reuse them in case of functional indexes. The BPS_TREE_IDENTICAL > is only used in bps_tree_delete_identical (when BPS_TREE_NO_DEBUG > macro is defined) therefore it may be reused to delete an > identical entry in case of functional index in further patches. Pushed. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 3/4] box: refactor memtx_tree_delete_identical 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 1/4] box: introduce tuple_chunk infrastructure Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 2/4] box: generalize memtx_multikey_tree methods Kirill Shcherbatov @ 2019-07-24 7:36 ` Kirill Shcherbatov 2019-07-24 19:24 ` Konstantin Osipov 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov 2019-07-24 12:25 ` [tarantool-patches] [PATCH v4 4/5] box: fix memtx_tree_index_build_array_deduplicate Kirill Shcherbatov 4 siblings, 1 reply; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 7:36 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev, Kirill Shcherbatov Renamed memtx_tree_delete_identical to memtx_tree_delete_value because it is more representative name. Changed it's signature to return a deleted item, because it may require following destruction. This refactoring is required in scope of functional indexes. Needed for #1260 --- src/lib/salad/bps_tree.h | 9 ++++++--- src/box/memtx_tree.c | 4 ++-- test/unit/bps_tree.cc | 8 ++++---- test/unit/bps_tree.result | 4 ++-- 4 files changed, 14 insertions(+), 11 deletions(-) diff --git a/src/lib/salad/bps_tree.h b/src/lib/salad/bps_tree.h index 58d69bf66..db1343b67 100644 --- a/src/lib/salad/bps_tree.h +++ b/src/lib/salad/bps_tree.h @@ -360,7 +360,7 @@ typedef uint32_t bps_tree_block_id_t; #define bps_tree_insert _api_name(insert) #define bps_tree_insert_get_iterator _api_name(insert_get_iterator) #define bps_tree_delete _api_name(delete) -#define bps_tree_delete_identical _api_name(delete_identical) +#define bps_tree_delete_value _api_name(delete_value) #define bps_tree_size _api_name(size) #define bps_tree_mem_used _api_name(mem_used) #define bps_tree_random _api_name(random) @@ -4527,7 +4527,8 @@ bps_tree_delete(struct bps_tree *tree, bps_tree_elem_t elem) * found in tree or is not identical. */ static inline int -bps_tree_delete_identical(struct bps_tree *tree, bps_tree_elem_t elem) +bps_tree_delete_value(struct bps_tree *tree, bps_tree_elem_t elem, + bps_tree_elem_t *deleted_elem) { if (tree->root_id == (bps_tree_block_id_t)(-1)) return -1; @@ -4543,6 +4544,8 @@ bps_tree_delete_identical(struct bps_tree *tree, bps_tree_elem_t elem) if (!BPS_TREE_IDENTICAL(elem, leaf->elems[leaf_path_elem.insertion_point])) return -1; + if (deleted_elem != NULL) + *deleted_elem = leaf->elems[leaf_path_elem.insertion_point]; bps_tree_process_delete_leaf(tree, &leaf_path_elem); return 0; } @@ -6085,7 +6088,7 @@ bps_tree_debug_check_internal_functions(bool assertme) #undef bps_tree_find #undef bps_tree_insert #undef bps_tree_delete -#undef bps_tree_delete_identical +#undef bps_tree_delete_value #undef bps_tree_size #undef bps_tree_mem_used #undef bps_tree_random diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index 5f5848c69..41eb6dbec 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -667,7 +667,7 @@ memtx_tree_index_replace_multikey_rollback(struct memtx_tree_index *index, data.tuple = new_tuple; for (int i = 0; i < err_multikey_idx; i++) { data.hint = i; - memtx_tree_delete_identical(&index->tree, data); + memtx_tree_delete_value(&index->tree, data, NULL); } } @@ -717,7 +717,7 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple, tuple_multikey_count(old_tuple, cmp_def); for (int i = 0; (uint32_t) i < multikey_count; i++) { data.hint = i; - memtx_tree_delete_identical(&index->tree, data); + memtx_tree_delete_value(&index->tree, data, NULL); } } return 0; diff --git a/test/unit/bps_tree.cc b/test/unit/bps_tree.cc index a622ecc87..58b3e47c9 100644 --- a/test/unit/bps_tree.cc +++ b/test/unit/bps_tree.cc @@ -839,7 +839,7 @@ insert_get_iterator() } static void -delete_identical_check() +delete_value_check() { header(); struct_tree tree; @@ -847,11 +847,11 @@ delete_identical_check() struct elem_t e1 = {1, 1}; struct_tree_insert(&tree, e1, NULL); struct elem_t e2 = {1, 2}; - if (struct_tree_delete_identical(&tree, e2) == 0) + if (struct_tree_delete_value(&tree, e2, NULL) == 0) fail("deletion of the non-identical element must fail", "false"); if (struct_tree_find(&tree, 1) == NULL) fail("test non-identical element deletion failure", "false"); - if (struct_tree_delete_identical(&tree, e1) != 0) + if (struct_tree_delete_value(&tree, e1, NULL) != 0) fail("deletion of the identical element must not fail", "false"); if (struct_tree_find(&tree, 1) != NULL) fail("test identical element deletion completion", "false"); @@ -873,5 +873,5 @@ main(void) if (extents_count != 0) fail("memory leak!", "true"); insert_get_iterator(); - delete_identical_check(); + delete_value_check(); } diff --git a/test/unit/bps_tree.result b/test/unit/bps_tree.result index dc21bf340..130ec555c 100644 --- a/test/unit/bps_tree.result +++ b/test/unit/bps_tree.result @@ -280,5 +280,5 @@ Count: 10575 *** approximate_count: done *** *** insert_get_iterator *** *** insert_get_iterator: done *** - *** delete_identical_check *** - *** delete_identical_check: done *** + *** delete_value_check *** + *** delete_value_check: done *** -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 3/4] box: refactor memtx_tree_delete_identical 2019-07-24 7:36 ` [PATCH v4 3/4] box: refactor memtx_tree_delete_identical Kirill Shcherbatov @ 2019-07-24 19:24 ` Konstantin Osipov 0 siblings, 0 replies; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 19:24 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > Renamed memtx_tree_delete_identical to memtx_tree_delete_value > because it is more representative name. Changed it's signature > to return a deleted item, because it may require following > destruction. > This refactoring is required in scope of functional indexes. Pushed with a follow up. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov ` (2 preceding siblings ...) 2019-07-24 7:36 ` [PATCH v4 3/4] box: refactor memtx_tree_delete_identical Kirill Shcherbatov @ 2019-07-24 7:36 ` Kirill Shcherbatov 2019-07-24 12:24 ` [tarantool-patches] " Kirill Shcherbatov ` (7 more replies) 2019-07-24 12:25 ` [tarantool-patches] [PATCH v4 4/5] box: fix memtx_tree_index_build_array_deduplicate Kirill Shcherbatov 4 siblings, 8 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 7:36 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev, Kirill Shcherbatov Closes #1260 @TarantoolBot document Title: introduce functional indexes in memtx Now you can define a functional index using registered persistent function as key extractor. There is some restrictions for function and key definition for functional index: - referenced function must be persistent, deterministic and must return scalar type or array. - key parts describe the referenced routine's returned key (when the routine returned value type is array in meaning of multikey index key): all parts must be sequential and the first part's fieldno must be 1. - key parts can't use JSON paths. - you are not allowed to access tuple fields by names in function Functional index can't be primary. You are not allowed to change functional index extractor function while there are some functional indexes depends of it. To define a functional multikey index, just return multiple values from function. Each value (even when it is a single scalar must be returned as table i.e. {1}) and must follow functional index key definition. Example: s = box.schema.space.create('withdata') s:format({{name = 'name', type = 'string'}, {name = 'address', type = 'string'}}) pk = s:create_index('name', {parts = {1, 'string'}}) lua_code = [[function(tuple) local address = string.split(tuple[2]) local ret = {} for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end return unpack(ret) end]] box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) s:insert({"James", "SIS Building Lambeth London UK"}) s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) idx:select('Uk') --- - - ['James', 'SIS Building Lambeth London UK'] - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] ... --- src/box/alter.h | 1 + src/box/errcode.h | 1 + src/box/func_key.h | 117 +++++ src/box/index.h | 9 + src/box/index_def.h | 16 + src/box/key_def.h | 22 +- src/box/schema_def.h | 9 + src/box/func_key.c | 135 ++++++ src/box/index_def.c | 21 +- src/box/key_def.c | 32 +- src/box/lua/key_def.c | 2 +- src/box/memtx_engine.c | 2 + src/box/memtx_space.c | 18 + src/box/memtx_tree.c | 301 ++++++++++++- src/box/sql.c | 2 +- src/box/sql/build.c | 2 +- src/box/sql/select.c | 2 +- src/box/sql/where.c | 2 +- src/box/vinyl.c | 9 +- test/unit/luaT_tuple_new.c | 2 +- test/unit/merger.test.c | 4 +- src/box/CMakeLists.txt | 1 + src/box/alter.cc | 107 ++++- src/box/bootstrap.snap | Bin 5863 -> 5914 bytes src/box/index.cc | 28 ++ src/box/lua/schema.lua | 20 + src/box/lua/space.cc | 7 + src/box/lua/upgrade.lua | 18 + src/box/schema.cc | 12 +- src/box/tuple_compare.cc | 104 ++++- src/box/tuple_extract_key.cc | 29 +- src/box/tuple_hash.cc | 1 + test/app-tap/tarantoolctl.test.lua | 4 +- test/box-py/bootstrap.result | 5 + test/box/access.result | 3 + test/box/access.test.lua | 1 + test/box/access_misc.result | 132 +++--- test/box/access_sysview.result | 6 +- test/box/alter.result | 7 +- test/box/bitset.result | 24 + test/box/bitset.test.lua | 9 + test/box/hash.result | 24 + test/box/hash.test.lua | 9 + test/box/misc.result | 1 + test/box/rtree_misc.result | 24 + test/box/rtree_misc.test.lua | 9 + test/engine/engine.cfg | 5 +- test/engine/functional.result | 690 +++++++++++++++++++++++++++++ test/engine/functional.test.lua | 233 ++++++++++ test/vinyl/misc.result | 23 + test/vinyl/misc.test.lua | 9 + test/wal_off/alter.result | 2 +- 52 files changed, 2145 insertions(+), 111 deletions(-) create mode 100644 src/box/func_key.h create mode 100644 src/box/func_key.c create mode 100644 test/engine/functional.result create mode 100644 test/engine/functional.test.lua diff --git a/src/box/alter.h b/src/box/alter.h index c339ccea6..1bc837359 100644 --- a/src/box/alter.h +++ b/src/box/alter.h @@ -47,5 +47,6 @@ extern struct trigger on_replace_space_sequence; extern struct trigger on_replace_trigger; extern struct trigger on_replace_fk_constraint; extern struct trigger on_replace_ck_constraint; +extern struct trigger on_replace_func_index; #endif /* INCLUDES_TARANTOOL_BOX_ALTER_H */ diff --git a/src/box/errcode.h b/src/box/errcode.h index 361ad3a45..1335ea6da 100644 --- a/src/box/errcode.h +++ b/src/box/errcode.h @@ -250,6 +250,7 @@ struct errcode_record { /*195 */_(ER_CREATE_CK_CONSTRAINT, "Failed to create check constraint '%s': %s") \ /*196 */_(ER_CK_CONSTRAINT_FAILED, "Check constraint failed '%s': %s") \ /*197 */_(ER_SQL_COLUMN_COUNT, "Unequal number of entries in row expression: left side has %u, but right side - %u") \ + /*198 */_(ER_FUNCTIONAL_INDEX_FUNC_ERROR,"Functional index function '%s' error: %s") \ /* * !IMPORTANT! Please follow instructions at start of the file diff --git a/src/box/func_key.h b/src/box/func_key.h new file mode 100644 index 000000000..7930c91a2 --- /dev/null +++ b/src/box/func_key.h @@ -0,0 +1,117 @@ +#ifndef TARANTOOL_BOX_FUNC_KEY_H_INCLUDED +#define TARANTOOL_BOX_FUNC_KEY_H_INCLUDED +/* + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include <stdbool.h> +#include <inttypes.h> + +#include "tuple_compare.h" + +#ifdef __cplusplus +extern "C" { +#endif + +struct func; +struct key_def; +struct tuple; + +/** + * Execute a given function (a functional index function) and + * return an extracted key_data and key_data_sz and count of + * extracted keys. + * + * Returns not NULL key_data pointer in case of success. + * Routine allocates memory on fiber's region. + */ +const char * +func_key_extract(struct tuple *tuple, struct func *func, + const char **data_end, uint32_t *key_count); + +/** + * An iterator to iterate over the key_data returned by function + * and validate it with given key definition (when required). + */ +struct func_key_iterator { + /** The pointer to currently processed key. */ + const char *data; + /** The pointer to the end of extracted key_data. */ + const char *data_end; + /** + * The sequential functional index key definition that + * describes a format of functional index function keys. + */ + struct key_def *key_def; + /** Whether iterator must validate processed keys. */ + bool validate; +}; + +/** + * Initialize a new functional index function returned keys + * iterator. + */ +static inline void +func_key_iterator_create(struct func_key_iterator *it, const char *data, + const char *data_end, struct key_def *key_def, + bool validate) +{ + it->data = data; + it->data_end = data_end; + it->key_def = key_def; + it->validate = validate; +} + +/** + * Perform key iterator step and update iterator state. + * Update key pointer with an actual key. + * + * Returns 0 on success. In case of error returns -1 and sets + * the corresponding diag message. + */ +int +func_key_iterator_next(struct func_key_iterator *it, const char **key, + uint32_t *key_sz); + +/** + * Allocate a new func_key_hint for given tuple and return a + * pointer to the hint value memory. + */ +hint_t +func_key_hint_new(struct tuple *tuple, const char *key, uint32_t key_sz); + +/** Release a given func_key_hint memory chunk. */ +void +func_key_hint_delete(struct tuple *tuple, hint_t func_key_hint); + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif /* TARANTOOL_BOX_FUNC_KEY_H_INCLUDED */ diff --git a/src/box/index.h b/src/box/index.h index 97d600c96..2b1d0104b 100644 --- a/src/box/index.h +++ b/src/box/index.h @@ -685,8 +685,17 @@ void generic_index_compact(struct index *); void generic_index_reset_stat(struct index *); void generic_index_begin_build(struct index *); int generic_index_reserve(struct index *, uint32_t); +struct iterator * +generic_index_create_iterator(struct index *base, enum iterator_type type, + const char *key, uint32_t part_count); int generic_index_build_next(struct index *, struct tuple *); void generic_index_end_build(struct index *); +int +disabled_index_build_next(struct index *index, struct tuple *tuple); +int +disabled_index_replace(struct index *index, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result); #if defined(__cplusplus) } /* extern "C" */ diff --git a/src/box/index_def.h b/src/box/index_def.h index 6dac28377..d1e58488f 100644 --- a/src/box/index_def.h +++ b/src/box/index_def.h @@ -163,6 +163,8 @@ struct index_opts { * filled after running ANALYZE command. */ struct index_stat *stat; + /** Identifier of the functional index function. */ + uint32_t functional_fid; }; extern const struct index_opts index_opts_default; @@ -207,6 +209,8 @@ index_opts_cmp(const struct index_opts *o1, const struct index_opts *o2) return o1->run_size_ratio < o2->run_size_ratio ? -1 : 1; if (o1->bloom_fpr != o2->bloom_fpr) return o1->bloom_fpr < o2->bloom_fpr ? -1 : 1; + if (o1->functional_fid != o2->functional_fid) + return o1->functional_fid - o2->functional_fid; return 0; } @@ -298,6 +302,18 @@ index_def_update_optionality(struct index_def *def, uint32_t min_field_count) key_def_update_optionality(def->cmp_def, min_field_count); } +/** + * Update func pointer for functional index key definitions. + * @param def Index def, containing key definitions to update. + * @param func The functional index function pointer. + */ +static inline void +index_def_update_func(struct index_def *def, struct func *func) +{ + def->key_def->func_index_func = func; + def->cmp_def->func_index_func = func; +} + /** * Add an index definition to a list, preserving the * first position of the primary key. diff --git a/src/box/key_def.h b/src/box/key_def.h index df83d055c..5ac498d9e 100644 --- a/src/box/key_def.h +++ b/src/box/key_def.h @@ -192,6 +192,13 @@ struct key_def { * unique_part_count == part count of a merged key_def. */ uint32_t unique_part_count; + /** + * Count of parts in functional index defintion. + * All functional_part_count key_part(s) of an + * initialized key def instance have func != NULL pointer. + * != 0 iff it is functional index definition. + */ + uint32_t functional_part_count; /** True, if at least one part can store NULL. */ bool is_nullable; /** True if some key part has JSON path. */ @@ -205,6 +212,8 @@ struct key_def { bool has_optional_parts; /** Key fields mask. @sa column_mask.h for details. */ uint64_t column_mask; + /** A pointer to functional index function. */ + struct func *func_index_func; /** * In case of the multikey index, a pointer to the * JSON path string, the path to the root node of @@ -330,7 +339,8 @@ key_def_sizeof(uint32_t part_count, uint32_t path_pool_size) * and initialize its parts. */ struct key_def * -key_def_new(const struct key_part_def *parts, uint32_t part_count); +key_def_new(const struct key_part_def *parts, uint32_t part_count, + bool is_functional); /** * Dump part definitions of the given key def. @@ -341,6 +351,16 @@ int key_def_dump_parts(const struct key_def *def, struct key_part_def *parts, struct region *region); +/** + * Return true if a given key definition defines functional index + * key. + */ +static inline bool +key_def_is_functional(const struct key_def *key_def) +{ + return key_def->functional_part_count > 0; +} + /** * Update 'has_optional_parts' of @a key_def with correspondence * to @a min_field_count. diff --git a/src/box/schema_def.h b/src/box/schema_def.h index a97b6d531..85f652d52 100644 --- a/src/box/schema_def.h +++ b/src/box/schema_def.h @@ -112,6 +112,8 @@ enum { BOX_FK_CONSTRAINT_ID = 356, /** Space id of _ck_contraint. */ BOX_CK_CONSTRAINT_ID = 364, + /** Space id of _func_index. */ + BOX_FUNC_INDEX_ID = 372, /** End of the reserved range of system spaces. */ BOX_SYSTEM_ID_MAX = 511, BOX_ID_NIL = 2147483647 @@ -267,6 +269,13 @@ enum { BOX_CK_CONSTRAINT_FIELD_CODE = 4, }; +/** _func_index fields. */ +enum { + BOX_FUNC_INDEX_FIELD_SPACE_ID = 0, + BOX_FUNC_INDEX_FIELD_INDEX_ID = 1, + BOX_FUNC_INDEX_FUNCTION_ID = 2, +}; + /* * Different objects which can be subject to access * control. diff --git a/src/box/func_key.c b/src/box/func_key.c new file mode 100644 index 000000000..fd743213f --- /dev/null +++ b/src/box/func_key.c @@ -0,0 +1,135 @@ +/* + * Copyright 2010-2016, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include "func_key.h" + +#include "errcode.h" +#include "diag.h" +#include "func.h" +#include "key_def.h" +#include "port.h" +#include "tt_static.h" +#include "tuple.h" +#include "tuple_compare.h" + +const char * +func_key_extract(struct tuple *tuple, struct func *func, + const char **data_end, uint32_t *key_count) +{ + struct port out_port, in_port; + port_tuple_create(&in_port); + port_tuple_add(&in_port, tuple); + int rc = func_call(func, &in_port, &out_port); + port_destroy(&in_port); + if (rc != 0) + goto error; + uint32_t key_data_sz; + const char *key_data = port_get_msgpack(&out_port, &key_data_sz); + port_destroy(&out_port); + if (key_data == NULL) + goto error; + + assert(mp_typeof(*key_data) == MP_ARRAY); + *data_end = key_data + key_data_sz; + *key_count = mp_decode_array(&key_data); + return key_data; +error: + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, func->def->name, + diag_last_error(diag_get())->errmsg); + return NULL; +} + +int +func_key_iterator_next(struct func_key_iterator *it, const char **key, + uint32_t *key_sz) +{ + assert(it->data <= it->data_end); + if (it->data == it->data_end) { + *key = NULL; + *key_sz = 0; + return 0; + } + *key = it->data; + if (!it->validate) { + mp_next(&it->data); + assert(it->data <= it->data_end); + *key_sz = it->data - *key; + return 0; + } + + if (mp_typeof(*it->data) != MP_ARRAY) { + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, + "returned key type is invalid"); + return -1; + } + const char *rptr = *key; + uint32_t part_count = mp_decode_array(&rptr); + uint32_t functional_part_count = it->key_def->functional_part_count; + if (part_count != functional_part_count) { + const char *error_msg = + tt_sprintf(tnt_errcode_desc(ER_EXACT_MATCH), + functional_part_count, part_count); + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, error_msg); + return -1; + } + const char *key_end; + if (key_validate_parts(it->key_def, rptr, functional_part_count, true, + &key_end) != 0) { + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, + diag_last_error(diag_get())->errmsg); + return -1; + } + + *key_sz = key_end - *key; + it->data = key_end; + return 0; +} + +hint_t +func_key_hint_new(struct tuple *tuple, const char *key, uint32_t key_sz) +{ + struct tuple_chunk *chunk = tuple_chunk_new(tuple, key_sz); + if (chunk == NULL) + return HINT_NONE; + memcpy(chunk->data, key, key_sz); + return (hint_t)chunk->data; +} + +void +func_key_hint_delete(struct tuple *tuple, hint_t func_key_hint) +{ + struct tuple_chunk *chunk = + container_of((typeof(chunk->data) *)func_key_hint, + struct tuple_chunk, data); + tuple_chunk_delete(tuple, chunk); +} diff --git a/src/box/index_def.c b/src/box/index_def.c index eb309a30c..a17b1b16c 100644 --- a/src/box/index_def.c +++ b/src/box/index_def.c @@ -50,6 +50,7 @@ const struct index_opts index_opts_default = { /* .bloom_fpr = */ 0.05, /* .lsn = */ 0, /* .stat = */ NULL, + /* .func = */ 0, }; const struct opt_def index_opts_reg[] = { @@ -63,6 +64,7 @@ const struct opt_def index_opts_reg[] = { OPT_DEF("run_size_ratio", OPT_FLOAT, struct index_opts, run_size_ratio), OPT_DEF("bloom_fpr", OPT_FLOAT, struct index_opts, bloom_fpr), OPT_DEF("lsn", OPT_INT64, struct index_opts, lsn), + OPT_DEF("func", OPT_UINT32, struct index_opts, functional_fid), OPT_DEF_LEGACY("sql"), OPT_END, }; @@ -251,8 +253,15 @@ index_def_to_key_def(struct rlist *index_defs, int *size) { int key_count = 0; struct index_def *index_def; - rlist_foreach_entry(index_def, index_defs, link) + rlist_foreach_entry(index_def, index_defs, link) { + /* + * Don't use functional index key definition + * to build a space format. + */ + if (key_def_is_functional(index_def->key_def)) + continue; key_count++; + } size_t sz = sizeof(struct key_def *) * key_count; struct key_def **keys = (struct key_def **) region_alloc(&fiber()->gc, sz); @@ -262,8 +271,11 @@ index_def_to_key_def(struct rlist *index_defs, int *size) } *size = key_count; key_count = 0; - rlist_foreach_entry(index_def, index_defs, link) + rlist_foreach_entry(index_def, index_defs, link) { + if (key_def_is_functional(index_def->key_def)) + continue; keys[key_count++] = index_def->key_def; + } return keys; } @@ -296,6 +308,11 @@ index_def_is_valid(struct index_def *index_def, const char *space_name) space_name, "primary key cannot be multikey"); return false; } + if (index_def->iid == 0 && key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, index_def->name, + space_name, "primary key cannot be functional"); + return false; + } for (uint32_t i = 0; i < index_def->key_def->part_count; i++) { assert(index_def->key_def->parts[i].type < field_type_MAX); if (index_def->key_def->parts[i].fieldno > BOX_INDEX_FIELD_MAX) { diff --git a/src/box/key_def.c b/src/box/key_def.c index ee758eefa..109adfc57 100644 --- a/src/box/key_def.c +++ b/src/box/key_def.c @@ -241,7 +241,8 @@ key_def_set_part(struct key_def *def, uint32_t part_no, uint32_t fieldno, } struct key_def * -key_def_new(const struct key_part_def *parts, uint32_t part_count) +key_def_new(const struct key_part_def *parts, uint32_t part_count, + bool is_functional) { size_t sz = 0; for (uint32_t i = 0; i < part_count; i++) @@ -255,7 +256,6 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) def->part_count = part_count; def->unique_part_count = part_count; - /* A pointer to the JSON paths data in the new key_def. */ char *path_pool = (char *)def + key_def_sizeof(part_count, 0); for (uint32_t i = 0; i < part_count; i++) { @@ -266,8 +266,7 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) if (coll_id == NULL) { diag_set(ClientError, ER_WRONG_INDEX_OPTIONS, i + 1, "collation was not found by ID"); - key_def_delete(def); - return NULL; + goto error; } coll = coll_id->coll; } @@ -276,14 +275,24 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) part->nullable_action, coll, part->coll_id, part->sort_order, part->path, path_len, &path_pool, TUPLE_OFFSET_SLOT_NIL, - 0) != 0) { - key_def_delete(def); - return NULL; + 0) != 0) + goto error; + } + if (is_functional) { + def->functional_part_count = part_count; + if (!key_def_is_sequential(def) || parts->fieldno != 0 || + def->has_json_paths) { + diag_set(ClientError, ER_WRONG_INDEX_OPTIONS, 0, + "invalid functional key definition"); + goto error; } } assert(path_pool == (char *)def + sz); key_def_set_func(def); return def; +error: + key_def_delete(def); + return NULL; } int @@ -677,6 +686,7 @@ key_def_find_by_fieldno(const struct key_def *key_def, uint32_t fieldno) const struct key_part * key_def_find(const struct key_def *key_def, const struct key_part *to_find) { + assert(!key_def_is_functional(key_def)); const struct key_part *part = key_def->parts; const struct key_part *end = part + key_def->part_count; for (; part != end; part++) { @@ -708,6 +718,9 @@ static bool key_def_can_merge(const struct key_def *key_def, const struct key_part *to_merge) { + if (key_def_is_functional(key_def)) + return true; + const struct key_part *part = key_def_find(key_def, to_merge); if (part == NULL) return true; @@ -722,6 +735,7 @@ key_def_can_merge(const struct key_def *key_def, struct key_def * key_def_merge(const struct key_def *first, const struct key_def *second) { + assert(!key_def_is_functional(second)); uint32_t new_part_count = first->part_count + second->part_count; /* * Find and remove part duplicates, i.e. parts counted @@ -754,6 +768,8 @@ key_def_merge(const struct key_def *first, const struct key_def *second) new_def->has_optional_parts = first->has_optional_parts || second->has_optional_parts; new_def->is_multikey = first->is_multikey || second->is_multikey; + new_def->functional_part_count = first->functional_part_count; + new_def->func_index_func = first->func_index_func; /* JSON paths data in the new key_def. */ char *path_pool = (char *)new_def + key_def_sizeof(new_part_count, 0); @@ -826,7 +842,7 @@ key_def_find_pk_in_cmp_def(const struct key_def *cmp_def, } /* Finally, allocate the new key definition. */ - extracted_def = key_def_new(parts, pk_def->part_count); + extracted_def = key_def_new(parts, pk_def->part_count, false); out: region_truncate(region, region_svp); return extracted_def; diff --git a/src/box/lua/key_def.c b/src/box/lua/key_def.c index 041b5ec98..3a3a5ec0c 100644 --- a/src/box/lua/key_def.c +++ b/src/box/lua/key_def.c @@ -445,7 +445,7 @@ lbox_key_def_new(struct lua_State *L) lua_pop(L, 1); } - struct key_def *key_def = key_def_new(parts, part_count); + struct key_def *key_def = key_def_new(parts, part_count, false); region_truncate(region, region_svp); if (key_def == NULL) return luaT_error(L); diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index fbb3151c9..95a1ecb90 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -1248,6 +1248,8 @@ memtx_index_def_change_requires_rebuild(struct index *index, return true; if (!old_def->opts.is_unique && new_def->opts.is_unique) return true; + if (old_def->opts.functional_fid != new_def->opts.functional_fid) + return true; const struct key_def *old_cmp_def, *new_cmp_def; if (index_depends_on_pk(index)) { diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index a8e7b8080..216ef3912 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -659,6 +659,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "HASH index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "HASH index cannot be functional"); + return -1; + } break; case TREE: /* TREE index has no limitations. */ @@ -688,6 +694,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "RTREE index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "RTREE index cannot be functional"); + return -1; + } /* no furter checks of parts needed */ return 0; case BITSET: @@ -716,6 +728,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "BITSET index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "BITSET index cannot be functional"); + return -1; + } /* no furter checks of parts needed */ return 0; default: diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index 41eb6dbec..76544b59b 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -35,6 +35,7 @@ #include "errinj.h" #include "memory.h" #include "fiber.h" +#include "func_key.h" #include "tuple.h" #include <third_party/qsort_arg.h> #include <small/mempool.h> @@ -723,6 +724,184 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple, return 0; } +/** + * The journal for multikey functional index replace operatoin + * is required to rollback an incomplete action, restore the + * original func_key_hint(s) hints bouth as to commit a completed + * replace action and destruct useless func_key_hint(s) hints. +*/ +struct journal_entry { + /** An inserted record copy. */ + struct memtx_tree_data inserted; + /** A replaced record copy. */ + struct memtx_tree_data replaced; +}; + +/** + * Rollback a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Routine uses given journal + * to return given index object in it's original state. + */ +static void +memtx_tree_index_replace_functional_rollback(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple != NULL) { + memtx_tree_insert(&index->tree, journal[i].replaced, + NULL); + } else { + memtx_tree_delete_value(&index->tree, + journal[i].inserted, NULL); + } + func_key_hint_delete(journal[i].inserted.tuple, + journal[i].inserted.hint); + } +} + +/** + * Commit a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Rotine uses given operations + * journal to release unused memory. + */ +static void +memtx_tree_index_replace_functional_commit(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + (void) index; + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple == NULL) + continue; + func_key_hint_delete(journal[i].replaced.tuple, + journal[i].replaced.hint); + } +} + +/** + * Functional index replace method works like + * memtx_tree_index_replace_multikey exept few moments. + * It uses functional index function from key definition + * to evaluate a key and allocate it on region. Then each + * returned key is reallocated in engine's memory as + * func_key_hint object and is used as comparison hint. + * To control func_key_hint(s) lifecycle in case of functional + * index we use a tiny journal object is allocated on region. + * It allows to restore original nodes with their original + * func_key_hint(s) pointers in case of failure and release + * useless hints of replaced items in case of success. + */ +static int +memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_functional(cmp_def)); + + int rc = -1; + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + *result = NULL; + const char *key, *key_end; + uint32_t key_sz, key_cnt; + struct func_key_iterator it; + if (new_tuple != NULL) { + key = func_key_extract(new_tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + goto end; + + int journal_idx = 0; + struct journal_entry *journal = + region_alloc(region, key_cnt * sizeof(*journal)); + if (journal == NULL) { + diag_set(OutOfMemory, key_cnt * sizeof(*journal), + "region", "journal"); + goto end; + } + + int err = 0; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && + key != NULL) { + hint_t key_hint = func_key_hint_new(new_tuple, key, + key_sz); + if (key_hint == HINT_NONE) { + err = -1; + break; + } + + /* Perform insertion, log it in journal. */ + bool is_multikey_conflict; + journal[journal_idx].replaced.tuple = NULL; + journal[journal_idx].inserted.tuple = new_tuple; + journal[journal_idx].inserted.hint = key_hint; + err = memtx_tree_index_replace_multikey_one(index, + old_tuple, new_tuple, mode, + key_hint, + &journal[journal_idx].replaced, + &is_multikey_conflict); + if (err != 0) + break; + /** + * Modify a 'replace' record of journal + * because an original node shouldn't be + * restored in case of multikey conflict. + */ + if (is_multikey_conflict) + journal[journal_idx].replaced.tuple = NULL; + else if (journal[journal_idx].replaced.tuple != NULL) + *result = journal[journal_idx].replaced.tuple; + + ++journal_idx; + } + if (key != NULL || err != 0) { + memtx_tree_index_replace_functional_rollback(index, + journal, journal_idx); + goto end; + } + if (*result != NULL) { + assert(old_tuple == NULL || old_tuple == *result); + old_tuple = *result; + } + memtx_tree_index_replace_functional_commit(index, + journal, journal_idx); + } + if (old_tuple != NULL) { + key = func_key_extract(old_tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + goto end; + struct memtx_tree_data data, deleted_data; + data.tuple = old_tuple; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && + key != NULL) { + data.hint = (hint_t) key; /* Raw data. */ + deleted_data.tuple = NULL; + memtx_tree_delete_value(&index->tree, data, + &deleted_data); + if (deleted_data.tuple != NULL) { + /* + * Release related hint on + * successfull node deletion. + */ + func_key_hint_delete(deleted_data.tuple, + deleted_data.hint); + } + } + assert(key == NULL); + } + rc = 0; +end: + region_truncate(region, region_svp); + return rc; +} + static struct iterator * memtx_tree_index_create_iterator(struct index *base, enum iterator_type type, const char *key, uint32_t part_count) @@ -854,6 +1033,46 @@ memtx_tree_index_build_next_multikey(struct index *base, struct tuple *tuple) return 0; } +static int +memtx_tree_index_build_next_functional(struct index *base, struct tuple *tuple) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_functional(cmp_def)); + + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + uint32_t key_sz, key_cnt; + const char *key, *key_end; + key = func_key_extract(tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + return -1; + + uint32_t insert_idx = index->build_array_size; + struct func_key_iterator it; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && key != NULL) { + hint_t key_hint = func_key_hint_new(tuple, key, key_sz); + if (key_hint == HINT_NONE) + goto error; + if (memtx_tree_index_build_array_append(index, tuple, + key_hint) != 0) + goto error; + } + assert(key == NULL); + region_truncate(region, region_svp); + return 0; +error: + for (uint32_t i = insert_idx; i < index->build_array_size; i++) { + func_key_hint_delete(index->build_array[i].tuple, + index->build_array[i].hint); + } + region_truncate(region, region_svp); + return -1; +} + /** * Process build_array of specified index and remove duplicates * of equal tuples (in terms of index's cmp_def and have same @@ -889,7 +1108,7 @@ memtx_tree_index_end_build(struct index *base) struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); qsort_arg(index->build_array, index->build_array_size, sizeof(index->build_array[0]), memtx_tree_qcompare, cmp_def); - if (cmp_def->is_multikey) { + if (cmp_def->is_multikey || key_def_is_functional(cmp_def)) { /* * Multikey index may have equal(in terms of * cmp_def) keys inserted by different multikey @@ -1024,6 +1243,72 @@ static const struct index_vtab memtx_tree_index_multikey_vtab = { /* .end_build = */ memtx_tree_index_end_build, }; +static const struct index_vtab memtx_tree_index_functional_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ memtx_tree_index_update_def, + /* .depends_on_pk = */ memtx_tree_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + memtx_index_def_change_requires_rebuild, + /* .size = */ memtx_tree_index_size, + /* .bsize = */ memtx_tree_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ memtx_tree_index_random, + /* .count = */ memtx_tree_index_count, + /* .get = */ memtx_tree_index_get, + /* .replace = */ memtx_tree_index_replace_functional, + /* .create_iterator = */ memtx_tree_index_create_iterator, + /* .create_snapshot_iterator = */ + memtx_tree_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ memtx_tree_index_begin_build, + /* .reserve = */ memtx_tree_index_reserve, + /* .build_next = */ memtx_tree_index_build_next_functional, + /* .end_build = */ memtx_tree_index_end_build, +}; + +/** + * A disabled index vtab provides safe dummy methods for + * 'inactive' index. It is required to perform a fault-tolerant + * recovery from snapshoot in case of functional index (because + * key defintion is not completely initialized at that moment). + */ +static const struct index_vtab memtx_tree_index_disabled_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ generic_index_update_def, + /* .depends_on_pk = */ generic_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + generic_index_def_change_requires_rebuild, + /* .size = */ generic_index_size, + /* .bsize = */ generic_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ generic_index_random, + /* .count = */ generic_index_count, + /* .get = */ generic_index_get, + /* .replace = */ disabled_index_replace, + /* .create_iterator = */ generic_index_create_iterator, + /* .create_snapshot_iterator = */ + generic_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ generic_index_begin_build, + /* .reserve = */ generic_index_reserve, + /* .build_next = */ disabled_index_build_next, + /* .end_build = */ generic_index_end_build, +}; + struct index * memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) { @@ -1034,9 +1319,17 @@ memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) "malloc", "struct memtx_tree_index"); return NULL; } - const struct index_vtab *vtab = def->key_def->is_multikey ? - &memtx_tree_index_multikey_vtab : - &memtx_tree_index_vtab; + const struct index_vtab *vtab; + if (key_def_is_functional(def->key_def)) { + if (def->key_def->func_index_func == NULL) + vtab = &memtx_tree_index_disabled_vtab; + else + vtab = &memtx_tree_index_functional_vtab; + } else if (def->key_def->is_multikey) { + vtab = &memtx_tree_index_multikey_vtab; + } else { + vtab = &memtx_tree_index_vtab; + } if (index_create(&index->base, (struct engine *)memtx, vtab, def) != 0) { free(index); diff --git a/src/box/sql.c b/src/box/sql.c index 4c9a4c15b..0ab3a506f 100644 --- a/src/box/sql.c +++ b/src/box/sql.c @@ -347,7 +347,7 @@ sql_ephemeral_space_create(uint32_t field_count, struct sql_key_info *key_info) } } struct key_def *ephemer_key_def = key_def_new(ephemer_key_parts, - field_count); + field_count, false); if (ephemer_key_def == NULL) return NULL; diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 2aefa2a3f..1f62f4136 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -2334,7 +2334,7 @@ index_fill_def(struct Parse *parse, struct index *index, part->coll_id = coll_id; part->path = NULL; } - key_def = key_def_new(key_parts, expr_list->nExpr); + key_def = key_def_new(key_parts, expr_list->nExpr, false); if (key_def == NULL) goto tnt_error; /* diff --git a/src/box/sql/select.c b/src/box/sql/select.c index bd144c1bb..5d2cd42bf 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1438,7 +1438,7 @@ sql_key_info_to_key_def(struct sql_key_info *key_info) { if (key_info->key_def == NULL) { key_info->key_def = key_def_new(key_info->parts, - key_info->part_count); + key_info->part_count, false); } return key_info->key_def; } diff --git a/src/box/sql/where.c b/src/box/sql/where.c index 5458c6a75..ed507bf4d 100644 --- a/src/box/sql/where.c +++ b/src/box/sql/where.c @@ -2775,7 +2775,7 @@ whereLoopAddBtree(WhereLoopBuilder * pBuilder, /* WHERE clause information */ part.coll_id = COLL_NONE; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1); + struct key_def *key_def = key_def_new(&part, 1, false); if (key_def == NULL) { tnt_error: pWInfo->pParse->is_aborted = true; diff --git a/src/box/vinyl.c b/src/box/vinyl.c index cf7af26b7..5582da833 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -701,6 +701,11 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) return -1; } } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_UNSUPPORTED, "Vinyl", + "functional indexes"); + return -1; + } return 0; } @@ -986,6 +991,8 @@ vinyl_index_def_change_requires_rebuild(struct index *index, if (!old_def->opts.is_unique && new_def->opts.is_unique) return true; + if (old_def->opts.functional_fid != new_def->opts.functional_fid) + return true; assert(index_depends_on_pk(index)); const struct key_def *old_cmp_def = old_def->cmp_def; @@ -3165,7 +3172,7 @@ vy_send_lsm(struct vy_join_ctx *ctx, struct vy_lsm_recovery_info *lsm_info) /* Create key definition and tuple format. */ ctx->key_def = key_def_new(lsm_info->key_parts, - lsm_info->key_part_count); + lsm_info->key_part_count, false); if (ctx->key_def == NULL) goto out; ctx->format = vy_stmt_format_new(&ctx->env->stmt_env, &ctx->key_def, 1, diff --git a/test/unit/luaT_tuple_new.c b/test/unit/luaT_tuple_new.c index 0a16fa039..609d64e45 100644 --- a/test/unit/luaT_tuple_new.c +++ b/test/unit/luaT_tuple_new.c @@ -124,7 +124,7 @@ test_basic(struct lua_State *L) part.nullable_action = ON_CONFLICT_ACTION_DEFAULT; part.sort_order = SORT_ORDER_ASC; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1); + struct key_def *key_def = key_def_new(&part, 1, false); box_tuple_format_t *another_format = box_tuple_format_new(&key_def, 1); key_def_delete(key_def); diff --git a/test/unit/merger.test.c b/test/unit/merger.test.c index b4a989a20..345a2364e 100644 --- a/test/unit/merger.test.c +++ b/test/unit/merger.test.c @@ -214,7 +214,7 @@ test_merger(struct tuple_format *format) merge_source_array_new(true), }; - struct key_def *key_def = key_def_new(&key_part_unsigned, 1); + struct key_def *key_def = key_def_new(&key_part_unsigned, 1, false); struct merge_source *merger = merger_new(key_def, sources, source_count, false); key_def_delete(key_def); @@ -252,7 +252,7 @@ test_basic() plan(4); header(); - struct key_def *key_def = key_def_new(&key_part_integer, 1); + struct key_def *key_def = key_def_new(&key_part_integer, 1, false); struct tuple_format *format = box_tuple_format_new(&key_def, 1); assert(format != NULL); diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt index 481842a39..1c6fda447 100644 --- a/src/box/CMakeLists.txt +++ b/src/box/CMakeLists.txt @@ -101,6 +101,7 @@ add_library(box STATIC fk_constraint.c func.c func_def.c + func_key.c alter.cc schema.cc schema_def.c diff --git a/src/box/alter.cc b/src/box/alter.cc index f98a77a51..c36b6643e 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -45,6 +45,7 @@ #include "fiber.h" /* for gc_pool */ #include "scoped_guard.h" #include "third_party/base64.h" +#include "memtx_engine.h" #include <new> /* for placement new */ #include <stdio.h> /* snprintf() */ #include <ctype.h> @@ -195,7 +196,7 @@ err: */ static void index_opts_decode(struct index_opts *opts, const char *map, - struct region *region) + struct space *space, struct region *region) { index_opts_create(opts); if (opts_decode(opts, index_opts_reg, &map, ER_WRONG_INDEX_OPTIONS, @@ -229,6 +230,22 @@ index_opts_decode(struct index_opts *opts, const char *map, "bloom_fpr must be greater than 0 and " "less than or equal to 1"); } + /** + * Can't verify functional index function + * reference on load because the function object + * had not been registered in Tarantool yet. + */ + struct memtx_engine *engine = (struct memtx_engine *)space->engine; + if (engine->state == MEMTX_OK && opts->functional_fid > 0) { + struct func *func = func_cache_find(opts->functional_fid); + if (func->def->language != FUNC_LANGUAGE_LUA || + func->def->body == NULL || !func->def->is_deterministic || + !func->def->is_sandboxed) { + tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, + "referenced function doesn't satisfy " + "functional index constraints"); + } + } } /** @@ -260,7 +277,7 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) const char *opts_field = tuple_field_with_type_xc(tuple, BOX_INDEX_FIELD_OPTS, MP_MAP); - index_opts_decode(&opts, opts_field, &fiber()->gc); + index_opts_decode(&opts, opts_field, space, &fiber()->gc); const char *parts = tuple_field(tuple, BOX_INDEX_FIELD_PARTS); uint32_t part_count = mp_decode_array(&parts); if (name_len > BOX_NAME_MAX) { @@ -285,7 +302,7 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) space->def->fields, space->def->field_count, &fiber()->gc) != 0) diag_raise(); - key_def = key_def_new(part_def, part_count); + key_def = key_def_new(part_def, part_count, opts.functional_fid > 0); if (key_def == NULL) diag_raise(); struct index_def *index_def = @@ -1370,6 +1387,30 @@ RebuildIndex::~RebuildIndex() index_def_delete(new_index_def); } +/** + * RebuildFunctionalIndex - prepare functional index definition, + * drop the old index data and rebuild index from by reading the + * primary key. + */ +class RebuildFunctionalIndex: public RebuildIndex +{ + struct index_def * + functional_index_def_new(struct index_def *index_def, + struct func *func) + { + struct index_def *new_index_def = index_def_dup_xc(index_def); + index_def_update_func(new_index_def, func); + return new_index_def; + } +public: + RebuildFunctionalIndex(struct alter_space *alter, + struct index_def *old_index_def_arg, + struct func *func) : + RebuildIndex(alter, + functional_index_def_new(old_index_def_arg, func), + old_index_def_arg) {} +}; + /** TruncateIndex - truncate an index. */ class TruncateIndex: public AlterSpaceOp { @@ -2841,6 +2882,12 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event) (unsigned) old_func->def->uid, "function has grants"); } + if (old_func != NULL && + space_has_data(BOX_FUNC_INDEX_ID, 1, old_func->def->fid)) { + tnt_raise(ClientError, ER_DROP_FUNCTION, + (unsigned) old_func->def->uid, + "function has references"); + } struct trigger *on_commit = txn_alter_trigger_new(on_drop_func_commit, old_func); struct trigger *on_rollback = @@ -4689,6 +4736,56 @@ on_replace_dd_ck_constraint(struct trigger * /* trigger*/, void *event) trigger_run_xc(&on_alter_space, space); } +/** A trigger invoked on replace in the _func_index space. */ +static void +on_replace_dd_func_index(struct trigger *trigger, void *event) +{ + (void) trigger; + struct txn *txn = (struct txn *) event; + struct txn_stmt *stmt = txn_current_stmt(txn); + struct tuple *old_tuple = stmt->old_tuple; + struct tuple *new_tuple = stmt->new_tuple; + + struct alter_space *alter = NULL; + struct func *func = NULL; + struct index *index; + struct space *space; + if (old_tuple == NULL && new_tuple != NULL) { + uint32_t space_id = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FIELD_SPACE_ID); + uint32_t index_id = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FIELD_INDEX_ID); + uint32_t fid = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FUNCTION_ID); + space = space_cache_find_xc(space_id); + index = index_find_xc(space, index_id); + func = func_cache_find(fid); + } else if (old_tuple != NULL && new_tuple == NULL) { + uint32_t space_id = tuple_field_u32_xc(old_tuple, + BOX_FUNC_INDEX_FIELD_SPACE_ID); + uint32_t index_id = tuple_field_u32_xc(old_tuple, + BOX_FUNC_INDEX_FIELD_INDEX_ID); + space = space_cache_find_xc(space_id); + index = index_find_xc(space, index_id); + func = NULL; + } else { + assert(old_tuple != NULL && new_tuple != NULL); + tnt_raise(ClientError, ER_UNSUPPORTED, "func_index", "alter"); + } + + alter = alter_space_new(space); + auto scoped_guard = make_scoped_guard([=] {alter_space_delete(alter);}); + alter_space_move_indexes(alter, 0, index->def->iid); + (void) new RebuildFunctionalIndex(alter, index->def, func); + alter_space_move_indexes(alter, index->def->iid + 1, + space->index_id_max + 1); + (void) new MoveCkConstraints(alter); + (void) new UpdateSchemaVersion(alter); + alter_space_do(txn, alter); + + scoped_guard.is_active = false; +} + struct trigger alter_space_on_replace_space = { RLIST_LINK_INITIALIZER, on_replace_dd_space, NULL, NULL }; @@ -4749,4 +4846,8 @@ struct trigger on_replace_ck_constraint = { RLIST_LINK_INITIALIZER, on_replace_dd_ck_constraint, NULL, NULL }; +struct trigger on_replace_func_index = { + RLIST_LINK_INITIALIZER, on_replace_dd_func_index, NULL, NULL +}; + /* vim: set foldmethod=marker */ diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap diff --git a/src/box/index.cc b/src/box/index.cc index 843d0e73d..00a1b502e 100644 --- a/src/box/index.cc +++ b/src/box/index.cc @@ -679,6 +679,16 @@ generic_index_replace(struct index *index, struct tuple *old_tuple, return -1; } +struct iterator * +generic_index_create_iterator(struct index *base, enum iterator_type type, + const char *key, uint32_t part_count) +{ + (void) type; (void) key; (void) part_count; + diag_set(UnsupportedIndexFeature, base->def, "read view"); + return NULL; +} + + struct snapshot_iterator * generic_index_create_snapshot_iterator(struct index *index) { @@ -729,4 +739,22 @@ generic_index_end_build(struct index *) { } +int +disabled_index_build_next(struct index *index, struct tuple *tuple) +{ + (void) index; (void) tuple; + return 0; +} + +int +disabled_index_replace(struct index *index, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result) +{ + (void) old_tuple; (void) new_tuple; (void) mode; + (void) index; + *result = NULL; + return 0; +} + /* }}} */ diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua index aadcd3fa9..d2bbea3cf 100644 --- a/src/box/lua/schema.lua +++ b/src/box/lua/schema.lua @@ -515,6 +515,7 @@ box.schema.space.drop = function(space_id, space_name, opts) local _space_sequence = box.space[box.schema.SPACE_SEQUENCE_ID] local _fk_constraint = box.space[box.schema.FK_CONSTRAINT_ID] local _ck_constraint = box.space[box.schema.CK_CONSTRAINT_ID] + local _func_index = box.space[box.schema.FUNC_INDEX_ID] local sequence_tuple = _space_sequence:delete{space_id} if sequence_tuple ~= nil and sequence_tuple.is_generated == true then -- Delete automatically generated sequence. @@ -529,6 +530,9 @@ box.schema.space.drop = function(space_id, space_name, opts) for _, t in _ck_constraint.index.primary:pairs({space_id}) do _ck_constraint:delete({space_id, t.name}) end + for _, t in _func_index.index.primary:pairs({space_id}) do + _func_index:delete({space_id, t.index_id, t.func_id}) + end local keys = _vindex:select(space_id) for i = #keys, 1, -1 do local v = keys[i] @@ -956,6 +960,8 @@ local index_options = { range_size = 'number', page_size = 'number', bloom_fpr = 'number', + func = 'number', + is_multikey = 'boolean', } -- @@ -1061,6 +1067,8 @@ box.schema.index.create = function(space_id, name, options) run_count_per_level = options.run_count_per_level, run_size_ratio = options.run_size_ratio, bloom_fpr = options.bloom_fpr, + func = options.func, + is_multikey = options.is_multikey, } local field_type_aliases = { num = 'unsigned'; -- Deprecated since 1.7.2 @@ -1086,6 +1094,10 @@ box.schema.index.create = function(space_id, name, options) space.name, name) _index:insert{space_id, iid, name, options.type, index_opts, parts} space_sequence_alter_commit(sequence_proxy) + if (options.func ~= nil and options.func > 0) then + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + _func_index:insert{space_id, iid, options.func} + end return space.index[name] end @@ -1101,6 +1113,10 @@ box.schema.index.drop = function(space_id, index_id) end end local _index = box.space[box.schema.INDEX_ID] + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + for _, v in box.space._func_index:pairs{space_id, index_id} do + _func_index:delete(v) + end _index:delete{space_id, index_id} end @@ -1202,6 +1218,10 @@ box.schema.index.alter = function(space_id, index_id, options) space.name, options.name) _index:replace{space_id, index_id, options.name, options.type, index_opts, parts} + if (options.func ~= nil and options.func > 0) then + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + _func_index:insert{space_id, iid, options.func} + end space_sequence_alter_commit(sequence_proxy) end diff --git a/src/box/lua/space.cc b/src/box/lua/space.cc index 18039fd6a..c8ad0be36 100644 --- a/src/box/lua/space.cc +++ b/src/box/lua/space.cc @@ -335,6 +335,11 @@ lbox_fillspace(struct lua_State *L, struct space *space, int i) lua_setfield(L, -2, "dimension"); } + if (index_opts->functional_fid > 0) { + lua_pushnumber(L, index_opts->functional_fid); + lua_setfield(L, -2, "func_id"); + } + lua_pushstring(L, index_type_strs[index_def->type]); lua_setfield(L, -2, "type"); @@ -629,6 +634,8 @@ box_lua_space_init(struct lua_State *L) lua_setfield(L, -2, "VSEQUENCE_ID"); lua_pushnumber(L, BOX_SPACE_SEQUENCE_ID); lua_setfield(L, -2, "SPACE_SEQUENCE_ID"); + lua_pushnumber(L, BOX_FUNC_INDEX_ID); + lua_setfield(L, -2, "FUNC_INDEX_ID"); lua_pushnumber(L, BOX_SYSTEM_ID_MIN); lua_setfield(L, -2, "SYSTEM_ID_MIN"); lua_pushnumber(L, BOX_SYSTEM_ID_MAX); diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua index 02c1cb0ff..f570a1c08 100644 --- a/src/box/lua/upgrade.lua +++ b/src/box/lua/upgrade.lua @@ -885,11 +885,29 @@ local function upgrade_func_to_2_2_1() collation = 'unicode_ci'}}}) end +local function create_func_index() + log.info("Create _func_index space") + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + local _space = box.space._space + local _index = box.space._index + local format = {{name='space_id', type='unsigned'}, + {name='index_id', type='unsigned'}, + {name='func_id', type='unsigned'}} + _space:insert{_func_index.id, ADMIN, '_func_index', 'memtx', 0, + setmap({}), format} + _index:insert{_func_index.id, 0, 'primary', 'tree', {unique = true}, + {{0, 'unsigned'}, {1, 'unsigned'}, {2, 'unsigned'}}} + _index:insert{_func_index.id, 1, 'fid', 'tree', {unique = false}, + {{2, 'unsigned'}}} + +end + local function upgrade_to_2_2_1() upgrade_sequence_to_2_2_1() upgrade_ck_constraint_to_2_2_1() create_vcollation_space() upgrade_func_to_2_2_1() + create_func_index() end -------------------------------------------------------------------------------- diff --git a/src/box/schema.cc b/src/box/schema.cc index 20666386d..5d4a3ff00 100644 --- a/src/box/schema.cc +++ b/src/box/schema.cc @@ -266,7 +266,7 @@ sc_space_new(uint32_t id, const char *name, uint32_t key_part_count, struct trigger *replace_trigger) { - struct key_def *key_def = key_def_new(key_parts, key_part_count); + struct key_def *key_def = key_def_new(key_parts, key_part_count, false); if (key_def == NULL) diag_raise(); auto key_def_guard = @@ -462,6 +462,16 @@ schema_init() sc_space_new(BOX_CK_CONSTRAINT_ID, "_ck_constraint", key_parts, 2, &on_replace_ck_constraint); + /* _func_index - check constraints. */ + key_parts[0].fieldno = 0; /* space id */ + key_parts[0].type = FIELD_TYPE_UNSIGNED; + key_parts[1].fieldno = 1; /* index id */ + key_parts[1].type = FIELD_TYPE_UNSIGNED; + key_parts[2].fieldno = 2; /* function id */ + key_parts[2].type = FIELD_TYPE_UNSIGNED; + sc_space_new(BOX_FUNC_INDEX_ID, "_func_index", key_parts, 3, + &on_replace_func_index); + /* * _vinyl_deferred_delete - blackhole that is needed * for writing deferred DELETE statements generated by diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc index c03b584d0..2ea6f7620 100644 --- a/src/box/tuple_compare.cc +++ b/src/box/tuple_compare.cc @@ -1271,6 +1271,80 @@ static const comparator_with_key_signature cmp_wk_arr[] = { KEY_COMPARATOR(1, FIELD_TYPE_STRING , 2, FIELD_TYPE_STRING) }; +template<bool is_nullable> +static inline int +functional_compare(struct tuple *tuple_a, hint_t tuple_a_hint, + struct tuple *tuple_b, hint_t tuple_b_hint, + struct key_def *key_def) +{ + assert(key_def_is_functional(key_def)); + assert(is_nullable == key_def->is_nullable); + + const char *key_a = (const char *)tuple_a_hint; + const char *key_b = (const char *)tuple_b_hint; + assert(mp_typeof(*key_a) == MP_ARRAY); + uint32_t part_count_a = mp_decode_array(&key_a); + assert(mp_typeof(*key_b) == MP_ARRAY); + uint32_t part_count_b = mp_decode_array(&key_b); + + uint32_t key_part_count = MIN(part_count_a, part_count_b); + uint32_t part_count = MIN(key_def->functional_part_count, + key_part_count); + int rc = key_compare_parts<is_nullable>(key_a, key_b, part_count, + key_def); + if (rc != 0) + return rc; + /* + * Primary index definiton key compare. + * It cannot contain nullable parts so the code is + * simplified correspondingly. + */ + const char *tuple_a_raw = tuple_data(tuple_a); + const char *tuple_b_raw = tuple_data(tuple_b); + struct tuple_format *format_a = tuple_format(tuple_a); + struct tuple_format *format_b = tuple_format(tuple_b); + const uint32_t *field_map_a = tuple_field_map(tuple_a); + const uint32_t *field_map_b = tuple_field_map(tuple_b); + const char *field_a, *field_b; + for (uint32_t i = key_def->functional_part_count; + i < key_def->part_count; i++) { + struct key_part *part = &key_def->parts[i]; + field_a = tuple_field_raw_by_part(format_a, tuple_a_raw, + field_map_a, part, + MULTIKEY_NONE); + field_b = tuple_field_raw_by_part(format_b, tuple_b_raw, + field_map_b, part, + MULTIKEY_NONE); + assert(field_a != NULL && field_b != NULL); + rc = tuple_compare_field(field_a, field_b, part->type, + part->coll); + if (rc != 0) + return rc; + else + continue; + } + return 0; +} + +template<bool is_nullable> +static inline int +functional_compare_with_key(struct tuple *tuple, hint_t tuple_hint, + const char *key, uint32_t part_count, + hint_t key_hint, struct key_def *key_def) +{ + (void)tuple; (void)key_hint; + assert(key_def->functional_part_count > 0); + assert(is_nullable == key_def->is_nullable); + const char *tuple_key = (const char *)tuple_hint; + assert(mp_typeof(*tuple_key) == MP_ARRAY); + + uint32_t tuple_key_count = mp_decode_array(&tuple_key); + part_count = MIN(part_count, tuple_key_count); + part_count = MIN(part_count, key_def->functional_part_count); + return key_compare_parts<is_nullable>(tuple_key, key, part_count, + key_def); +} + #undef KEY_COMPARATOR /* }}} tuple_compare_with_key */ @@ -1592,7 +1666,7 @@ tuple_hint(struct tuple *tuple, struct key_def *key_def) } static hint_t -key_hint_multikey(const char *key, uint32_t part_count, struct key_def *key_def) +key_hint_stub(const char *key, uint32_t part_count, struct key_def *key_def) { (void) key; (void) part_count; @@ -1600,19 +1674,19 @@ key_hint_multikey(const char *key, uint32_t part_count, struct key_def *key_def) /* * Multikey hint for tuple is an index of the key in * array, it always must be defined. While - * tuple_hint_multikey assumes that it must be + * key_hint_stub assumes that it must be * initialized manually (so it mustn't be called), * the virtual method for a key makes sense. Overriding * this method such way, we extend existend code to * do nothing on key hint calculation an it is valid * because it is never used(unlike tuple hint). */ - assert(key_def->is_multikey); + assert(key_def->is_multikey || key_def_is_functional(key_def)); return HINT_NONE; } static hint_t -tuple_hint_multikey(struct tuple *tuple, struct key_def *key_def) +key_hint_stub(struct tuple *tuple, struct key_def *key_def) { (void) tuple; (void) key_def; @@ -1641,9 +1715,9 @@ key_def_set_hint_func(struct key_def *def) static void key_def_set_hint_func(struct key_def *def) { - if (def->is_multikey) { - def->key_hint = key_hint_multikey; - def->tuple_hint = tuple_hint_multikey; + if (def->is_multikey || key_def_is_functional(def)) { + def->key_hint = key_hint_stub; + def->tuple_hint = key_hint_stub; return; } switch (def->parts->type) { @@ -1769,10 +1843,24 @@ key_def_set_compare_func_json(struct key_def *def) } } +template<bool is_nullable> +static void +key_def_set_compare_func_functional(struct key_def *def) +{ + assert(key_def_is_functional(def)); + def->tuple_compare = functional_compare<is_nullable>; + def->tuple_compare_with_key = functional_compare_with_key<is_nullable>; +} + void key_def_set_compare_func(struct key_def *def) { - if (!key_def_has_collation(def) && + if (key_def_is_functional(def)) { + if (def->is_nullable) + key_def_set_compare_func_functional<true>(def); + else + key_def_set_compare_func_functional<false>(def); + } else if (!key_def_has_collation(def) && !def->is_nullable && !def->has_json_paths) { key_def_set_compare_func_fast(def); } else if (!def->has_json_paths) { diff --git a/src/box/tuple_extract_key.cc b/src/box/tuple_extract_key.cc index 471c7df80..c84809dae 100644 --- a/src/box/tuple_extract_key.cc +++ b/src/box/tuple_extract_key.cc @@ -120,6 +120,7 @@ tuple_extract_key_slowpath(struct tuple *tuple, struct key_def *key_def, key_def_contains_sequential_parts(key_def)); assert(is_multikey == key_def->is_multikey); assert(!key_def->is_multikey || multikey_idx != MULTIKEY_NONE); + assert(!key_def_is_functional(key_def)); assert(mp_sizeof_nil() == 1); const char *data = tuple_data(tuple); uint32_t part_count = key_def->part_count; @@ -251,6 +252,7 @@ tuple_extract_key_slowpath_raw(const char *data, const char *data_end, assert(!has_optional_parts || key_def->is_nullable); assert(has_optional_parts == key_def->has_optional_parts); assert(!key_def->is_multikey || multikey_idx != MULTIKEY_NONE); + assert(!key_def_is_functional(key_def)); assert(mp_sizeof_nil() == 1); /* allocate buffer with maximal possible size */ char *key = (char *) region_alloc(&fiber()->gc, data_end - data); @@ -367,6 +369,7 @@ key_def_set_extract_func_plain(struct key_def *def) { assert(!def->has_json_paths); assert(!def->is_multikey); + assert(!key_def_is_functional(def)); if (key_def_is_sequential(def)) { assert(contains_sequential_parts || def->part_count == 1); def->tuple_extract_key = tuple_extract_key_sequential @@ -387,6 +390,7 @@ static void key_def_set_extract_func_json(struct key_def *def) { assert(def->has_json_paths); + assert(!key_def_is_functional(def)); if (def->is_multikey) { def->tuple_extract_key = tuple_extract_key_slowpath <contains_sequential_parts, @@ -400,13 +404,36 @@ key_def_set_extract_func_json(struct key_def *def) <has_optional_parts, true>; } +static char * +tuple_extract_key_stub(struct tuple *tuple, struct key_def *key_def, + int multikey_idx, uint32_t *key_size) +{ + (void)tuple; (void)key_def; (void)multikey_idx; (void)key_size; + unreachable(); + return NULL; +} + +static char * +tuple_extract_key_raw_stub(const char *data, const char *data_end, + struct key_def *key_def, int multikey_idx, + uint32_t *key_size) +{ + (void)data; (void)data_end; + (void)key_def; (void)multikey_idx; (void)key_size; + unreachable(); + return NULL; +} + void key_def_set_extract_func(struct key_def *key_def) { bool contains_sequential_parts = key_def_contains_sequential_parts(key_def); bool has_optional_parts = key_def->has_optional_parts; - if (!key_def->has_json_paths) { + if (key_def_is_functional(key_def)) { + key_def->tuple_extract_key = tuple_extract_key_stub; + key_def->tuple_extract_key_raw = tuple_extract_key_raw_stub; + } else if (!key_def->has_json_paths) { if (!contains_sequential_parts && !has_optional_parts) { key_def_set_extract_func_plain<false, false>(key_def); } else if (!contains_sequential_parts && has_optional_parts) { diff --git a/src/box/tuple_hash.cc b/src/box/tuple_hash.cc index 780e3d053..dad87871a 100644 --- a/src/box/tuple_hash.cc +++ b/src/box/tuple_hash.cc @@ -365,6 +365,7 @@ tuple_hash_slowpath(struct tuple *tuple, struct key_def *key_def) assert(has_json_paths == key_def->has_json_paths); assert(has_optional_parts == key_def->has_optional_parts); assert(!key_def->is_multikey); + assert(!key_def_is_functional(key_def)); uint32_t h = HASH_SEED; uint32_t carry = 0; uint32_t total_size = 0; diff --git a/test/app-tap/tarantoolctl.test.lua b/test/app-tap/tarantoolctl.test.lua index 957b883f4..df2ee377f 100755 --- a/test/app-tap/tarantoolctl.test.lua +++ b/test/app-tap/tarantoolctl.test.lua @@ -405,8 +405,8 @@ do check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 1", "\n", 3) check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 1 --replica 2", "\n", 3) check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 2", "\n", 0) - check_ctlcat_snap(test_i, dir, "--space=280", "---\n", 23) - check_ctlcat_snap(test_i, dir, "--space=288", "---\n", 50) + check_ctlcat_snap(test_i, dir, "--space=280", "---\n", 24) + check_ctlcat_snap(test_i, dir, "--space=288", "---\n", 52) end) end) diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result index f2d1f46fb..a5d645df8 100644 --- a/test/box-py/bootstrap.result +++ b/test/box-py/bootstrap.result @@ -93,6 +93,8 @@ box.space._space:select{} - [364, 1, '_ck_constraint', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'is_deferred', 'type': 'boolean'}, {'name': 'language', 'type': 'str'}, {'name': 'code', 'type': 'str'}]] + - [372, 1, '_func_index', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, + {'name': 'index_id', 'type': 'unsigned'}, {'name': 'func_id', 'type': 'unsigned'}]] ... box.space._index:select{} --- @@ -148,6 +150,9 @@ box.space._index:select{} - [356, 0, 'primary', 'tree', {'unique': true}, [[0, 'string'], [1, 'unsigned']]] - [356, 1, 'child_id', 'tree', {'unique': false}, [[1, 'unsigned']]] - [364, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'string']]] + - [372, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'unsigned'], + [2, 'unsigned']]] + - [372, 1, 'fid', 'tree', {'unique': false}, [[2, 'unsigned']]] ... box.space._user:select{} --- diff --git a/test/box/access.result b/test/box/access.result index 5ee92a443..ba72b5f74 100644 --- a/test/box/access.result +++ b/test/box/access.result @@ -1496,6 +1496,9 @@ box.schema.user.grant('tester', 'read', 'space', '_fk_constraint') box.schema.user.grant('tester', 'read', 'space', '_ck_constraint') --- ... +box.schema.user.grant('tester', 'read', 'space', '_func_index') +--- +... box.session.su("tester") --- ... diff --git a/test/box/access.test.lua b/test/box/access.test.lua index 79340b0f5..219cdb04a 100644 --- a/test/box/access.test.lua +++ b/test/box/access.test.lua @@ -557,6 +557,7 @@ box.schema.user.grant('tester', 'read', 'space', '_space_sequence') box.schema.user.grant('tester', 'read', 'space', '_trigger') box.schema.user.grant('tester', 'read', 'space', '_fk_constraint') box.schema.user.grant('tester', 'read', 'space', '_ck_constraint') +box.schema.user.grant('tester', 'read', 'space', '_func_index') box.session.su("tester") -- successful create s1 = box.schema.space.create("test_space") diff --git a/test/box/access_misc.result b/test/box/access_misc.result index c69cf0283..31b935914 100644 --- a/test/box/access_misc.result +++ b/test/box/access_misc.result @@ -833,140 +833,142 @@ box.space._space:select() - [364, 1, '_ck_constraint', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'is_deferred', 'type': 'boolean'}, {'name': 'language', 'type': 'str'}, {'name': 'code', 'type': 'str'}]] + - [372, 1, '_func_index', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, + {'name': 'index_id', 'type': 'unsigned'}, {'name': 'func_id', 'type': 'unsigned'}]] ... box.space._func:select() --- - - [1, 1, 'box.schema.user.info', 1, 'LUA', '', 'function', [], 'any', 'none', 'none', - false, false, true, ['LUA'], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, ['LUA'], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [2, 1, 'TRIM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [3, 1, 'TYPEOF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [4, 1, 'PRINTF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [5, 1, 'UNICODE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [6, 1, 'CHAR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [7, 1, 'HEX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [8, 1, 'VERSION', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [9, 1, 'QUOTE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [10, 1, 'REPLACE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [11, 1, 'SUBSTR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [12, 1, 'GROUP_CONCAT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [13, 1, 'JULIANDAY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [14, 1, 'DATE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [15, 1, 'TIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [16, 1, 'DATETIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [17, 1, 'STRFTIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [18, 1, 'CURRENT_TIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [19, 1, 'CURRENT_TIMESTAMP', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [20, 1, 'CURRENT_DATE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [21, 1, 'LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [22, 1, 'POSITION', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [23, 1, 'ROUND', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [24, 1, 'UPPER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [25, 1, 'LOWER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [26, 1, 'IFNULL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [27, 1, 'RANDOM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [28, 1, 'CEIL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [29, 1, 'CEILING', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [30, 1, 'CHARACTER_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [31, 1, 'CHAR_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [32, 1, 'FLOOR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [33, 1, 'MOD', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [34, 1, 'OCTET_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [35, 1, 'ROW_COUNT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [36, 1, 'COUNT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [37, 1, 'LIKE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [38, 1, 'ABS', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [39, 1, 'EXP', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [40, 1, 'LN', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [41, 1, 'POWER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [42, 1, 'SQRT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [43, 1, 'SUM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [44, 1, 'TOTAL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [45, 1, 'AVG', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [46, 1, 'RANDOMBLOB', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [47, 1, 'NULLIF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [48, 1, 'ZEROBLOB', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [49, 1, 'MIN', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [50, 1, 'MAX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [51, 1, 'COALESCE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [52, 1, 'EVERY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [53, 1, 'EXISTS', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [54, 1, 'EXTRACT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [55, 1, 'SOME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [56, 1, 'GREATER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [57, 1, 'LESSER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [58, 1, 'SOUNDEX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [59, 1, 'LIKELIHOOD', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [60, 1, 'LIKELY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [61, 1, 'UNLIKELY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [62, 1, '_sql_stat_get', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [63, 1, '_sql_stat_push', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [64, 1, '_sql_stat_init', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [65, 1, 'LUA', 1, 'LUA', 'function(code) return assert(loadstring(code))() end', 'function', ['string'], 'any', 'none', 'none', false, false, true, ['LUA', 'SQL'], - {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] ... session = nil --- diff --git a/test/box/access_sysview.result b/test/box/access_sysview.result index d65aa37ae..a82127ebb 100644 --- a/test/box/access_sysview.result +++ b/test/box/access_sysview.result @@ -246,11 +246,11 @@ box.session.su('guest') ... #box.space._vspace:select{} --- -- 24 +- 25 ... #box.space._vindex:select{} --- -- 51 +- 53 ... #box.space._vuser:select{} --- @@ -282,7 +282,7 @@ box.session.su('guest') ... #box.space._vindex:select{} --- -- 51 +- 53 ... #box.space._vuser:select{} --- diff --git a/test/box/alter.result b/test/box/alter.result index a6db011ff..91a239bbc 100644 --- a/test/box/alter.result +++ b/test/box/alter.result @@ -92,7 +92,7 @@ space = box.space[t[1]] ... space.id --- -- 365 +- 373 ... space.field_count --- @@ -137,7 +137,7 @@ space_deleted ... space:replace{0} --- -- error: Space '365' does not exist +- error: Space '373' does not exist ... _index:insert{_space.id, 0, 'primary', 'tree', {unique=true}, {{0, 'unsigned'}}} --- @@ -218,6 +218,9 @@ _index:select{} - [356, 0, 'primary', 'tree', {'unique': true}, [[0, 'string'], [1, 'unsigned']]] - [356, 1, 'child_id', 'tree', {'unique': false}, [[1, 'unsigned']]] - [364, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'string']]] + - [372, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'unsigned'], + [2, 'unsigned']]] + - [372, 1, 'fid', 'tree', {'unique': false}, [[2, 'unsigned']]] ... -- modify indexes of a system space _index:delete{_index.id, 0} diff --git a/test/box/bitset.result b/test/box/bitset.result index 78f74ec37..fab0828a4 100644 --- a/test/box/bitset.result +++ b/test/box/bitset.result @@ -1996,3 +1996,27 @@ _ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}} s:drop() --- ... +-- Bitset index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': BITSET index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/box/bitset.test.lua b/test/box/bitset.test.lua index eb013a1c0..1fea31563 100644 --- a/test/box/bitset.test.lua +++ b/test/box/bitset.test.lua @@ -153,3 +153,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}}) s:drop() + +-- Bitset index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/box/hash.result b/test/box/hash.result index 9f08c49b8..a42f81c69 100644 --- a/test/box/hash.result +++ b/test/box/hash.result @@ -847,3 +847,27 @@ _ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}}) s:drop() --- ... +-- Hash index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'hash', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': HASH index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/box/hash.test.lua b/test/box/hash.test.lua index 9801873c4..e5ab18f63 100644 --- a/test/box/hash.test.lua +++ b/test/box/hash.test.lua @@ -353,3 +353,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}}) s:drop() + +-- Hash index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'hash', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/box/misc.result b/test/box/misc.result index 791730935..31e4e86f2 100644 --- a/test/box/misc.result +++ b/test/box/misc.result @@ -526,6 +526,7 @@ t; 195: box.error.CREATE_CK_CONSTRAINT 196: box.error.CK_CONSTRAINT_FAILED 197: box.error.SQL_COLUMN_COUNT + 198: box.error.FUNCTIONAL_INDEX_FUNC_ERROR ... test_run:cmd("setopt delimiter ''"); --- diff --git a/test/box/rtree_misc.result b/test/box/rtree_misc.result index 6e48bacc7..c256d52e7 100644 --- a/test/box/rtree_misc.result +++ b/test/box/rtree_misc.result @@ -682,3 +682,27 @@ _ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}}) s:drop() --- ... +-- Rtree index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]] +--- +... +box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'rtree', func = box.func.fextract.id, parts = {{1, 'array'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': RTREE index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('fextract') +--- +... diff --git a/test/box/rtree_misc.test.lua b/test/box/rtree_misc.test.lua index 000a928e8..2d1d490f3 100644 --- a/test/box/rtree_misc.test.lua +++ b/test/box/rtree_misc.test.lua @@ -243,3 +243,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}}) s:drop() + +-- Rtree index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]] +box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'rtree', func = box.func.fextract.id, parts = {{1, 'array'}}}) +s:drop() +box.schema.func.drop('fextract') diff --git a/test/engine/engine.cfg b/test/engine/engine.cfg index 9f07629b4..826e8eb1f 100644 --- a/test/engine/engine.cfg +++ b/test/engine/engine.cfg @@ -2,7 +2,10 @@ "*": { "memtx": {"engine": "memtx"}, "vinyl": {"engine": "vinyl"} - } + }, + "functional.test.lua": { + "memtx": {"engine": "memtx"} + } } diff --git a/test/engine/functional.result b/test/engine/functional.result new file mode 100644 index 000000000..3fd7aa64a --- /dev/null +++ b/test/engine/functional.result @@ -0,0 +1,690 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +-- +-- gh-1260: Funclional indexes. +-- +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +lua_code2 = [[function(tuple) return {tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('s_nonpersistent') + | --- + | ... +box.schema.func.create('s_ivaliddef1', {body = lua_code, is_deterministic = false, is_sandboxed = true}) + | --- + | ... +box.schema.func.create('s_ivaliddef2', {body = lua_code, is_deterministic = true, is_sandboxed = false}) + | --- + | ... + +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +box.schema.func.create('ss', {body = lua_code2, is_deterministic = true, is_sandboxed = true}) + | --- + | ... + +-- Functional index can't be primary. +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Can''t create or modify index ''idx'' in space ''withdata'': primary key + | cannot be functional' + | ... +pk = s:create_index('pk') + | --- + | ... +-- Invalid fid. +_ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) + | --- + | - error: Function '6666' does not exist + | ... +-- Can't use non-persistent function in functional index. +_ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-deterministic function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-sandboxed function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-sequential parts in returned key definition. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... +-- Can't use parts started not by 1 field. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... +-- Can't use JSON paths in returned key definiton. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{"[1]data", 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... + +-- Can't drop a function referenced by functional index. +idx = s:create_index('idx', {unique = true, func = box.func.s.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +box.schema.func.drop('s') + | --- + | - error: 'Can''t drop function 1: function has references' + | ... +box.snapshot() + | --- + | - ok + | ... +test_run:cmd("restart server default") + | +box.schema.func.drop('s') + | --- + | - error: 'Can''t drop function 1: function has references' + | ... +s = box.space.withdata + | --- + | ... +idx = s.index.idx + | --- + | ... +idx:drop() + | --- + | ... +box.schema.func.drop('s') + | --- + | ... + +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +-- Invalid functional index extractor routine return: the extractor must return keys. +lua_code = [[function(tuple) return "hello" end]] + | --- + | ... +box.schema.func.create('invalidreturn0', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn0.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn0'' error: returned key type is + | invalid' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional index extractor routine return: a stirng instead of unsigned +lua_code = [[function(tuple) return {"hello"} end]] + | --- + | ... +box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn1'' error: Supplied key type of + | part 0 does not match index part type: expected unsigned' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] + | --- + | ... +box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn3'' error: Supplied key type of + | part 0 does not match index part type: expected unsigned' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional extractor: runtime extractor error +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +lua_code = [[function(tuple) + local json = require('json') + return json.encode(tuple) + end]] +test_run:cmd("setopt delimiter ''"); + | --- + | ... +box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.runtimeerror.id, parts = {{1, 'string'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''runtimeerror'' error: [string "return function(tuple) local + | ..."]:1: attempt to call global ''require'' (a nil value)' + | ... +idx:drop() + | --- + | ... + +-- Remove old persistent functions +for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end + | --- + | ... +s:drop() + | --- + | ... + +-- Functional test cases. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({2, 1}) + | --- + | - error: Duplicate key exists in unique index 'idx' in space 'withdata' + | ... +idx:get(3) + | --- + | - [1, 2] + | ... +idx:delete(3) + | --- + | - [1, 2] + | ... +s:select() + | --- + | - [] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +idx:get(3) + | --- + | - [2, 1] + | ... +s:drop() + | --- + | ... +box.schema.func.drop('extr') + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... + +-- Multikey functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({3, 5}) + | --- + | - error: Duplicate key exists in unique index 'idx' in space 'withdata' + | ... +s:insert({5, 3}) + | --- + | - [5, 3] + | ... +idx:select() + | --- + | - - [1, 2] + | - [1, 2] + | - [5, 3] + | - [5, 3] + | ... +idx:get(8) + | --- + | - [5, 3] + | ... +idx:get(3) + | --- + | - [1, 2] + | ... +idx:get(1) + | --- + | - [1, 2] + | ... +idx:get(5) + | --- + | - [5, 3] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey multipart functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +s:insert({3, 3}) + | --- + | - [3, 3] + | ... +idx:select({600}, {iterator = "GE"}) + | --- + | - - [1, 2] + | - [2, 1] + | - [3, 3] + | ... +idx:get({603, 603}) + | --- + | - [3, 3] + | ... +idx:select({503}, {iterator = "LE"}) + | --- + | - - [3, 3] + | - [2, 1] + | - [1, 2] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey non-unique functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +idx:select({501}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +s:replace({1, 3}) + | --- + | - [1, 3] + | ... +idx:select({501}) + | --- + | - - [1, 3] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [2, 1] + | ... +idx:select({503}) + | --- + | - - [1, 3] + | ... +box.snapshot() + | --- + | - ok + | ... +test_run:cmd("restart server default") + | +s = box.space.withdata + | --- + | ... +idx = s.index.idx + | --- + | ... +idx:select({501}) + | --- + | - - [1, 3] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [2, 1] + | ... +idx:select({503}) + | --- + | - - [1, 3] + | ... +s:replace({1, 2}) + | --- + | - [1, 2] + | ... +idx:select({501}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({503}) + | --- + | - [] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey UTF-8 address extractor +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('name', {parts = {1, 'string'}}) + | --- + | ... +s:insert({"James", "SIS Building Lambeth London UK"}) + | --- + | - ['James', 'SIS Building Lambeth London UK'] + | ... +s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) + | --- + | - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] + | ... +-- Create functional index on space with data +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +lua_code = [[function(tuple) + local address = string.split(tuple[2]) + local ret = {} + for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end + return unpack(ret) + end]] +test_run:cmd("setopt delimiter ''"); + | --- + | ... +box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) + | --- + | ... +idx:select('uk') + | --- + | - - ['James', 'SIS Building Lambeth London UK'] + | - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] + | ... +idx:select('Sis') + | --- + | - - ['James', 'SIS Building Lambeth London UK'] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('addr_extractor') + | --- + | ... + +-- Partial index with functional index extractor +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1}) + | --- + | - [1] + | ... +s:insert({2}) + | --- + | - [2] + | ... +s:insert({3}) + | --- + | - [3] + | ... +s:insert({4}) + | --- + | - [4] + | ... +idx:select() + | --- + | - - [1] + | - [3] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Return nil from functional index extractor. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +lua_code = [[function(tuple) return {nil} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer', is_nullable = true}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''extr'' error: Invalid key part count in an exact + | match (expected 1, got 0)' + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multiple functional indexes +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] - tuple[2]} end]] + | --- + | ... +box.schema.func.create('sub', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx1 = s:create_index('s_idx', {unique = true, func = box.func.s.id, parts = {{1, 'integer'}}}) + | --- + | ... +idx2 = s:create_index('sub_idx', {unique = true, func = box.func.sub.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({4, 1}) + | --- + | - [4, 1] + | ... +idx1:get(5) + | --- + | - [4, 1] + | ... +idx2:get(3) + | --- + | - [4, 1] + | ... +idx1:drop() + | --- + | ... +idx2:get(3) + | --- + | - [4, 1] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('s') + | --- + | ... +box.schema.func.drop('sub') + | --- + | ... diff --git a/test/engine/functional.test.lua b/test/engine/functional.test.lua new file mode 100644 index 000000000..269152593 --- /dev/null +++ b/test/engine/functional.test.lua @@ -0,0 +1,233 @@ +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +-- +-- gh-1260: Funclional indexes. +-- +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +lua_code2 = [[function(tuple) return {tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]} end]] +box.schema.func.create('s_nonpersistent') +box.schema.func.create('s_ivaliddef1', {body = lua_code, is_deterministic = false, is_sandboxed = true}) +box.schema.func.create('s_ivaliddef2', {body = lua_code, is_deterministic = true, is_sandboxed = false}) + +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +box.schema.func.create('ss', {body = lua_code2, is_deterministic = true, is_sandboxed = true}) + +-- Functional index can't be primary. +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +pk = s:create_index('pk') +-- Invalid fid. +_ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) +-- Can't use non-persistent function in functional index. +_ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-deterministic function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-sandboxed function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-sequential parts in returned key definition. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) +-- Can't use parts started not by 1 field. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}}) +-- Can't use JSON paths in returned key definiton. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{"[1]data", 'unsigned'}}}) + +-- Can't drop a function referenced by functional index. +idx = s:create_index('idx', {unique = true, func = box.func.s.id, parts = {{1, 'unsigned'}}}) +box.schema.func.drop('s') +box.snapshot() +test_run:cmd("restart server default") +box.schema.func.drop('s') +s = box.space.withdata +idx = s.index.idx +idx:drop() +box.schema.func.drop('s') + +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +-- Invalid functional index extractor routine return: the extractor must return keys. +lua_code = [[function(tuple) return "hello" end]] +box.schema.func.create('invalidreturn0', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn0.id, parts = {{1, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: a stirng instead of unsigned +lua_code = [[function(tuple) return {"hello"} end]] +box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] +box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional extractor: runtime extractor error +test_run:cmd("setopt delimiter ';'") +lua_code = [[function(tuple) + local json = require('json') + return json.encode(tuple) + end]] +test_run:cmd("setopt delimiter ''"); +box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.runtimeerror.id, parts = {{1, 'string'}}}) +s:insert({1}) +idx:drop() + +-- Remove old persistent functions +for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end +s:drop() + +-- Functional test cases. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +s:insert({1, 2}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({2, 1}) +idx:get(3) +idx:delete(3) +s:select() +s:insert({2, 1}) +idx:get(3) +s:drop() +box.schema.func.drop('extr') +collectgarbage() + +-- Multikey functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1, 2}) +s:insert({3, 5}) +s:insert({5, 3}) +idx:select() +idx:get(8) +idx:get(3) +idx:get(1) +idx:get(5) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey multipart functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) +s:insert({1, 2}) +s:insert({2, 1}) +s:insert({3, 3}) +idx:select({600}, {iterator = "GE"}) +idx:get({603, 603}) +idx:select({503}, {iterator = "LE"}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey non-unique functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1, 2}) +s:insert({2, 1}) +idx:select({501}) +idx:select({502}) +s:replace({1, 3}) +idx:select({501}) +idx:select({502}) +idx:select({503}) +box.snapshot() +test_run:cmd("restart server default") +s = box.space.withdata +idx = s.index.idx +idx:select({501}) +idx:select({502}) +idx:select({503}) +s:replace({1, 2}) +idx:select({501}) +idx:select({502}) +idx:select({503}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey UTF-8 address extractor +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('name', {parts = {1, 'string'}}) +s:insert({"James", "SIS Building Lambeth London UK"}) +s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) +-- Create functional index on space with data +test_run:cmd("setopt delimiter ';'") +lua_code = [[function(tuple) + local address = string.split(tuple[2]) + local ret = {} + for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end + return unpack(ret) + end]] +test_run:cmd("setopt delimiter ''"); +box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) +idx:select('uk') +idx:select('Sis') +s:drop() +collectgarbage() +box.schema.func.drop('addr_extractor') + +-- Partial index with functional index extractor +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('pk') +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1}) +s:insert({2}) +s:insert({3}) +s:insert({4}) +idx:select() +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Return nil from functional index extractor. +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('pk') +lua_code = [[function(tuple) return {nil} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer', is_nullable = true}}}) +s:insert({1}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multiple functional indexes +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +lua_code = [[function(tuple) return {tuple[1] - tuple[2]} end]] +box.schema.func.create('sub', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx1 = s:create_index('s_idx', {unique = true, func = box.func.s.id, parts = {{1, 'integer'}}}) +idx2 = s:create_index('sub_idx', {unique = true, func = box.func.sub.id, parts = {{1, 'integer'}}}) +s:insert({4, 1}) +idx1:get(5) +idx2:get(3) +idx1:drop() +idx2:get(3) +s:drop() +collectgarbage() +box.schema.func.drop('s') +box.schema.func.drop('sub') diff --git a/test/vinyl/misc.result b/test/vinyl/misc.result index b2aacdc55..508a64660 100644 --- a/test/vinyl/misc.result +++ b/test/vinyl/misc.result @@ -432,3 +432,26 @@ stat.bytes_compressed < stat.bytes / 10 s:drop() --- ... +-- Vinyl doesn't support functional indexes. +s = box.schema.space.create('withdata', {engine = 'vinyl'}) +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: Vinyl does not support functional indexes +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/vinyl/misc.test.lua b/test/vinyl/misc.test.lua index f8da578d0..bd311da77 100644 --- a/test/vinyl/misc.test.lua +++ b/test/vinyl/misc.test.lua @@ -182,3 +182,12 @@ test_run:wait_cond(function() return i:stat().disk.compaction.count > 0 end) stat = i:stat().disk stat.bytes_compressed < stat.bytes / 10 s:drop() + +-- Vinyl doesn't support functional indexes. +s = box.schema.space.create('withdata', {engine = 'vinyl'}) +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/wal_off/alter.result b/test/wal_off/alter.result index bce15711d..62cb11db7 100644 --- a/test/wal_off/alter.result +++ b/test/wal_off/alter.result @@ -28,7 +28,7 @@ end; ... #spaces; --- -- 65503 +- 65502 ... -- cleanup for k, v in pairs(spaces) do -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov @ 2019-07-24 12:24 ` Kirill Shcherbatov 2019-07-24 19:41 ` Konstantin Osipov ` (6 subsequent siblings) 7 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 12:24 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev Closes #1260 @TarantoolBot document Title: introduce functional indexes in memtx Now you can define a functional index using registered persistent function as key extractor. There is some restrictions for function and key definition for functional index: - referenced function must be persistent, deterministic and must return scalar type or array. - key parts describe the referenced routine's returned key (when the routine returned value type is array in meaning of multikey index key): all parts must be sequential and the first part's fieldno must be 1. - key parts can't use JSON paths. - you are not allowed to access tuple fields by names in function Functional index can't be primary. You are not allowed to change functional index extractor function while there are some functional indexes depends of it. To define a functional multikey index, just return multiple values from function. Each value (even when it is a single scalar must be returned as table i.e. {1}) and must follow functional index key definition. Example: s = box.schema.space.create('withdata') s:format({{name = 'name', type = 'string'}, {name = 'address', type = 'string'}}) pk = s:create_index('name', {parts = {1, 'string'}}) lua_code = [[function(tuple) local address = string.split(tuple[2]) local ret = {} for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end return unpack(ret) end]] box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) idx = s:create_index('addr', {unique = false, func = 'addr_extractor', parts = {{1, 'string', collation = 'unicode_ci'}}}) s:insert({"James", "SIS Building Lambeth London UK"}) s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) idx:select('Uk') --- - - ['James', 'SIS Building Lambeth London UK'] - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] ... --- src/box/alter.h | 1 + src/box/errcode.h | 1 + src/box/func_key.h | 117 +++++ src/box/index.h | 9 + src/box/index_def.h | 16 + src/box/key_def.h | 22 +- src/box/schema_def.h | 9 + src/box/func_key.c | 135 ++++++ src/box/index_def.c | 21 +- src/box/key_def.c | 33 +- src/box/lua/key_def.c | 2 +- src/box/memtx_engine.c | 2 + src/box/memtx_space.c | 18 + src/box/memtx_tree.c | 319 ++++++++++++- src/box/sql.c | 2 +- src/box/sql/build.c | 2 +- src/box/sql/select.c | 2 +- src/box/sql/where.c | 2 +- src/box/vinyl.c | 9 +- test/unit/luaT_tuple_new.c | 2 +- test/unit/merger.test.c | 4 +- src/box/CMakeLists.txt | 1 + src/box/alter.cc | 107 ++++- src/box/bootstrap.snap | Bin 5863 -> 5914 bytes src/box/index.cc | 28 ++ src/box/lua/schema.lua | 35 ++ src/box/lua/space.cc | 7 + src/box/lua/upgrade.lua | 18 + src/box/schema.cc | 12 +- src/box/tuple_compare.cc | 104 ++++- src/box/tuple_extract_key.cc | 29 +- src/box/tuple_hash.cc | 1 + test/app-tap/tarantoolctl.test.lua | 4 +- test/box-py/bootstrap.result | 5 + test/box/access.result | 3 + test/box/access.test.lua | 1 + test/box/access_misc.result | 132 +++--- test/box/access_sysview.result | 6 +- test/box/alter.result | 7 +- test/box/bitset.result | 24 + test/box/bitset.test.lua | 9 + test/box/hash.result | 24 + test/box/hash.test.lua | 9 + test/box/misc.result | 1 + test/box/rtree_misc.result | 24 + test/box/rtree_misc.test.lua | 9 + test/engine/engine.cfg | 5 +- test/engine/functional.result | 690 +++++++++++++++++++++++++++++ test/engine/functional.test.lua | 233 ++++++++++ test/vinyl/misc.result | 23 + test/vinyl/misc.test.lua | 9 + test/wal_off/alter.result | 2 +- 52 files changed, 2177 insertions(+), 113 deletions(-) create mode 100644 src/box/func_key.h create mode 100644 src/box/func_key.c create mode 100644 test/engine/functional.result create mode 100644 test/engine/functional.test.lua diff --git a/src/box/alter.h b/src/box/alter.h index c339ccea6..1bc837359 100644 --- a/src/box/alter.h +++ b/src/box/alter.h @@ -47,5 +47,6 @@ extern struct trigger on_replace_space_sequence; extern struct trigger on_replace_trigger; extern struct trigger on_replace_fk_constraint; extern struct trigger on_replace_ck_constraint; +extern struct trigger on_replace_func_index; #endif /* INCLUDES_TARANTOOL_BOX_ALTER_H */ diff --git a/src/box/errcode.h b/src/box/errcode.h index 361ad3a45..1335ea6da 100644 --- a/src/box/errcode.h +++ b/src/box/errcode.h @@ -250,6 +250,7 @@ struct errcode_record { /*195 */_(ER_CREATE_CK_CONSTRAINT, "Failed to create check constraint '%s': %s") \ /*196 */_(ER_CK_CONSTRAINT_FAILED, "Check constraint failed '%s': %s") \ /*197 */_(ER_SQL_COLUMN_COUNT, "Unequal number of entries in row expression: left side has %u, but right side - %u") \ + /*198 */_(ER_FUNCTIONAL_INDEX_FUNC_ERROR,"Functional index function '%s' error: %s") \ /* * !IMPORTANT! Please follow instructions at start of the file diff --git a/src/box/func_key.h b/src/box/func_key.h new file mode 100644 index 000000000..7930c91a2 --- /dev/null +++ b/src/box/func_key.h @@ -0,0 +1,117 @@ +#ifndef TARANTOOL_BOX_FUNC_KEY_H_INCLUDED +#define TARANTOOL_BOX_FUNC_KEY_H_INCLUDED +/* + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include <stdbool.h> +#include <inttypes.h> + +#include "tuple_compare.h" + +#ifdef __cplusplus +extern "C" { +#endif + +struct func; +struct key_def; +struct tuple; + +/** + * Execute a given function (a functional index function) and + * return an extracted key_data and key_data_sz and count of + * extracted keys. + * + * Returns not NULL key_data pointer in case of success. + * Routine allocates memory on fiber's region. + */ +const char * +func_key_extract(struct tuple *tuple, struct func *func, + const char **data_end, uint32_t *key_count); + +/** + * An iterator to iterate over the key_data returned by function + * and validate it with given key definition (when required). + */ +struct func_key_iterator { + /** The pointer to currently processed key. */ + const char *data; + /** The pointer to the end of extracted key_data. */ + const char *data_end; + /** + * The sequential functional index key definition that + * describes a format of functional index function keys. + */ + struct key_def *key_def; + /** Whether iterator must validate processed keys. */ + bool validate; +}; + +/** + * Initialize a new functional index function returned keys + * iterator. + */ +static inline void +func_key_iterator_create(struct func_key_iterator *it, const char *data, + const char *data_end, struct key_def *key_def, + bool validate) +{ + it->data = data; + it->data_end = data_end; + it->key_def = key_def; + it->validate = validate; +} + +/** + * Perform key iterator step and update iterator state. + * Update key pointer with an actual key. + * + * Returns 0 on success. In case of error returns -1 and sets + * the corresponding diag message. + */ +int +func_key_iterator_next(struct func_key_iterator *it, const char **key, + uint32_t *key_sz); + +/** + * Allocate a new func_key_hint for given tuple and return a + * pointer to the hint value memory. + */ +hint_t +func_key_hint_new(struct tuple *tuple, const char *key, uint32_t key_sz); + +/** Release a given func_key_hint memory chunk. */ +void +func_key_hint_delete(struct tuple *tuple, hint_t func_key_hint); + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif /* TARANTOOL_BOX_FUNC_KEY_H_INCLUDED */ diff --git a/src/box/index.h b/src/box/index.h index 97d600c96..2b1d0104b 100644 --- a/src/box/index.h +++ b/src/box/index.h @@ -685,8 +685,17 @@ void generic_index_compact(struct index *); void generic_index_reset_stat(struct index *); void generic_index_begin_build(struct index *); int generic_index_reserve(struct index *, uint32_t); +struct iterator * +generic_index_create_iterator(struct index *base, enum iterator_type type, + const char *key, uint32_t part_count); int generic_index_build_next(struct index *, struct tuple *); void generic_index_end_build(struct index *); +int +disabled_index_build_next(struct index *index, struct tuple *tuple); +int +disabled_index_replace(struct index *index, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result); #if defined(__cplusplus) } /* extern "C" */ diff --git a/src/box/index_def.h b/src/box/index_def.h index 6dac28377..d1e58488f 100644 --- a/src/box/index_def.h +++ b/src/box/index_def.h @@ -163,6 +163,8 @@ struct index_opts { * filled after running ANALYZE command. */ struct index_stat *stat; + /** Identifier of the functional index function. */ + uint32_t functional_fid; }; extern const struct index_opts index_opts_default; @@ -207,6 +209,8 @@ index_opts_cmp(const struct index_opts *o1, const struct index_opts *o2) return o1->run_size_ratio < o2->run_size_ratio ? -1 : 1; if (o1->bloom_fpr != o2->bloom_fpr) return o1->bloom_fpr < o2->bloom_fpr ? -1 : 1; + if (o1->functional_fid != o2->functional_fid) + return o1->functional_fid - o2->functional_fid; return 0; } @@ -298,6 +302,18 @@ index_def_update_optionality(struct index_def *def, uint32_t min_field_count) key_def_update_optionality(def->cmp_def, min_field_count); } +/** + * Update func pointer for functional index key definitions. + * @param def Index def, containing key definitions to update. + * @param func The functional index function pointer. + */ +static inline void +index_def_update_func(struct index_def *def, struct func *func) +{ + def->key_def->func_index_func = func; + def->cmp_def->func_index_func = func; +} + /** * Add an index definition to a list, preserving the * first position of the primary key. diff --git a/src/box/key_def.h b/src/box/key_def.h index df83d055c..5ac498d9e 100644 --- a/src/box/key_def.h +++ b/src/box/key_def.h @@ -192,6 +192,13 @@ struct key_def { * unique_part_count == part count of a merged key_def. */ uint32_t unique_part_count; + /** + * Count of parts in functional index defintion. + * All functional_part_count key_part(s) of an + * initialized key def instance have func != NULL pointer. + * != 0 iff it is functional index definition. + */ + uint32_t functional_part_count; /** True, if at least one part can store NULL. */ bool is_nullable; /** True if some key part has JSON path. */ @@ -205,6 +212,8 @@ struct key_def { bool has_optional_parts; /** Key fields mask. @sa column_mask.h for details. */ uint64_t column_mask; + /** A pointer to functional index function. */ + struct func *func_index_func; /** * In case of the multikey index, a pointer to the * JSON path string, the path to the root node of @@ -330,7 +339,8 @@ key_def_sizeof(uint32_t part_count, uint32_t path_pool_size) * and initialize its parts. */ struct key_def * -key_def_new(const struct key_part_def *parts, uint32_t part_count); +key_def_new(const struct key_part_def *parts, uint32_t part_count, + bool is_functional); /** * Dump part definitions of the given key def. @@ -341,6 +351,16 @@ int key_def_dump_parts(const struct key_def *def, struct key_part_def *parts, struct region *region); +/** + * Return true if a given key definition defines functional index + * key. + */ +static inline bool +key_def_is_functional(const struct key_def *key_def) +{ + return key_def->functional_part_count > 0; +} + /** * Update 'has_optional_parts' of @a key_def with correspondence * to @a min_field_count. diff --git a/src/box/schema_def.h b/src/box/schema_def.h index a97b6d531..85f652d52 100644 --- a/src/box/schema_def.h +++ b/src/box/schema_def.h @@ -112,6 +112,8 @@ enum { BOX_FK_CONSTRAINT_ID = 356, /** Space id of _ck_contraint. */ BOX_CK_CONSTRAINT_ID = 364, + /** Space id of _func_index. */ + BOX_FUNC_INDEX_ID = 372, /** End of the reserved range of system spaces. */ BOX_SYSTEM_ID_MAX = 511, BOX_ID_NIL = 2147483647 @@ -267,6 +269,13 @@ enum { BOX_CK_CONSTRAINT_FIELD_CODE = 4, }; +/** _func_index fields. */ +enum { + BOX_FUNC_INDEX_FIELD_SPACE_ID = 0, + BOX_FUNC_INDEX_FIELD_INDEX_ID = 1, + BOX_FUNC_INDEX_FUNCTION_ID = 2, +}; + /* * Different objects which can be subject to access * control. diff --git a/src/box/func_key.c b/src/box/func_key.c new file mode 100644 index 000000000..fd743213f --- /dev/null +++ b/src/box/func_key.c @@ -0,0 +1,135 @@ +/* + * Copyright 2010-2016, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include "func_key.h" + +#include "errcode.h" +#include "diag.h" +#include "func.h" +#include "key_def.h" +#include "port.h" +#include "tt_static.h" +#include "tuple.h" +#include "tuple_compare.h" + +const char * +func_key_extract(struct tuple *tuple, struct func *func, + const char **data_end, uint32_t *key_count) +{ + struct port out_port, in_port; + port_tuple_create(&in_port); + port_tuple_add(&in_port, tuple); + int rc = func_call(func, &in_port, &out_port); + port_destroy(&in_port); + if (rc != 0) + goto error; + uint32_t key_data_sz; + const char *key_data = port_get_msgpack(&out_port, &key_data_sz); + port_destroy(&out_port); + if (key_data == NULL) + goto error; + + assert(mp_typeof(*key_data) == MP_ARRAY); + *data_end = key_data + key_data_sz; + *key_count = mp_decode_array(&key_data); + return key_data; +error: + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, func->def->name, + diag_last_error(diag_get())->errmsg); + return NULL; +} + +int +func_key_iterator_next(struct func_key_iterator *it, const char **key, + uint32_t *key_sz) +{ + assert(it->data <= it->data_end); + if (it->data == it->data_end) { + *key = NULL; + *key_sz = 0; + return 0; + } + *key = it->data; + if (!it->validate) { + mp_next(&it->data); + assert(it->data <= it->data_end); + *key_sz = it->data - *key; + return 0; + } + + if (mp_typeof(*it->data) != MP_ARRAY) { + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, + "returned key type is invalid"); + return -1; + } + const char *rptr = *key; + uint32_t part_count = mp_decode_array(&rptr); + uint32_t functional_part_count = it->key_def->functional_part_count; + if (part_count != functional_part_count) { + const char *error_msg = + tt_sprintf(tnt_errcode_desc(ER_EXACT_MATCH), + functional_part_count, part_count); + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, error_msg); + return -1; + } + const char *key_end; + if (key_validate_parts(it->key_def, rptr, functional_part_count, true, + &key_end) != 0) { + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, + it->key_def->func_index_func->def->name, + diag_last_error(diag_get())->errmsg); + return -1; + } + + *key_sz = key_end - *key; + it->data = key_end; + return 0; +} + +hint_t +func_key_hint_new(struct tuple *tuple, const char *key, uint32_t key_sz) +{ + struct tuple_chunk *chunk = tuple_chunk_new(tuple, key_sz); + if (chunk == NULL) + return HINT_NONE; + memcpy(chunk->data, key, key_sz); + return (hint_t)chunk->data; +} + +void +func_key_hint_delete(struct tuple *tuple, hint_t func_key_hint) +{ + struct tuple_chunk *chunk = + container_of((typeof(chunk->data) *)func_key_hint, + struct tuple_chunk, data); + tuple_chunk_delete(tuple, chunk); +} diff --git a/src/box/index_def.c b/src/box/index_def.c index eb309a30c..a17b1b16c 100644 --- a/src/box/index_def.c +++ b/src/box/index_def.c @@ -50,6 +50,7 @@ const struct index_opts index_opts_default = { /* .bloom_fpr = */ 0.05, /* .lsn = */ 0, /* .stat = */ NULL, + /* .func = */ 0, }; const struct opt_def index_opts_reg[] = { @@ -63,6 +64,7 @@ const struct opt_def index_opts_reg[] = { OPT_DEF("run_size_ratio", OPT_FLOAT, struct index_opts, run_size_ratio), OPT_DEF("bloom_fpr", OPT_FLOAT, struct index_opts, bloom_fpr), OPT_DEF("lsn", OPT_INT64, struct index_opts, lsn), + OPT_DEF("func", OPT_UINT32, struct index_opts, functional_fid), OPT_DEF_LEGACY("sql"), OPT_END, }; @@ -251,8 +253,15 @@ index_def_to_key_def(struct rlist *index_defs, int *size) { int key_count = 0; struct index_def *index_def; - rlist_foreach_entry(index_def, index_defs, link) + rlist_foreach_entry(index_def, index_defs, link) { + /* + * Don't use functional index key definition + * to build a space format. + */ + if (key_def_is_functional(index_def->key_def)) + continue; key_count++; + } size_t sz = sizeof(struct key_def *) * key_count; struct key_def **keys = (struct key_def **) region_alloc(&fiber()->gc, sz); @@ -262,8 +271,11 @@ index_def_to_key_def(struct rlist *index_defs, int *size) } *size = key_count; key_count = 0; - rlist_foreach_entry(index_def, index_defs, link) + rlist_foreach_entry(index_def, index_defs, link) { + if (key_def_is_functional(index_def->key_def)) + continue; keys[key_count++] = index_def->key_def; + } return keys; } @@ -296,6 +308,11 @@ index_def_is_valid(struct index_def *index_def, const char *space_name) space_name, "primary key cannot be multikey"); return false; } + if (index_def->iid == 0 && key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, index_def->name, + space_name, "primary key cannot be functional"); + return false; + } for (uint32_t i = 0; i < index_def->key_def->part_count; i++) { assert(index_def->key_def->parts[i].type < field_type_MAX); if (index_def->key_def->parts[i].fieldno > BOX_INDEX_FIELD_MAX) { diff --git a/src/box/key_def.c b/src/box/key_def.c index ee758eefa..b7c918db4 100644 --- a/src/box/key_def.c +++ b/src/box/key_def.c @@ -241,7 +241,8 @@ key_def_set_part(struct key_def *def, uint32_t part_no, uint32_t fieldno, } struct key_def * -key_def_new(const struct key_part_def *parts, uint32_t part_count) +key_def_new(const struct key_part_def *parts, uint32_t part_count, + bool is_functional) { size_t sz = 0; for (uint32_t i = 0; i < part_count; i++) @@ -255,7 +256,6 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) def->part_count = part_count; def->unique_part_count = part_count; - /* A pointer to the JSON paths data in the new key_def. */ char *path_pool = (char *)def + key_def_sizeof(part_count, 0); for (uint32_t i = 0; i < part_count; i++) { @@ -266,8 +266,7 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) if (coll_id == NULL) { diag_set(ClientError, ER_WRONG_INDEX_OPTIONS, i + 1, "collation was not found by ID"); - key_def_delete(def); - return NULL; + goto error; } coll = coll_id->coll; } @@ -275,15 +274,24 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count) if (key_def_set_part(def, i, part->fieldno, part->type, part->nullable_action, coll, part->coll_id, part->sort_order, part->path, path_len, - &path_pool, TUPLE_OFFSET_SLOT_NIL, - 0) != 0) { - key_def_delete(def); - return NULL; + &path_pool, TUPLE_OFFSET_SLOT_NIL, 0) != 0) + goto error; + } + if (is_functional) { + def->functional_part_count = part_count; + if (!key_def_is_sequential(def) || parts->fieldno != 0 || + def->has_json_paths) { + diag_set(ClientError, ER_WRONG_INDEX_OPTIONS, 0, + "invalid functional key definition"); + goto error; } } assert(path_pool == (char *)def + sz); key_def_set_func(def); return def; +error: + key_def_delete(def); + return NULL; } int @@ -677,6 +685,7 @@ key_def_find_by_fieldno(const struct key_def *key_def, uint32_t fieldno) const struct key_part * key_def_find(const struct key_def *key_def, const struct key_part *to_find) { + assert(!key_def_is_functional(key_def)); const struct key_part *part = key_def->parts; const struct key_part *end = part + key_def->part_count; for (; part != end; part++) { @@ -708,6 +717,9 @@ static bool key_def_can_merge(const struct key_def *key_def, const struct key_part *to_merge) { + if (key_def_is_functional(key_def)) + return true; + const struct key_part *part = key_def_find(key_def, to_merge); if (part == NULL) return true; @@ -722,6 +734,7 @@ key_def_can_merge(const struct key_def *key_def, struct key_def * key_def_merge(const struct key_def *first, const struct key_def *second) { + assert(!key_def_is_functional(second)); uint32_t new_part_count = first->part_count + second->part_count; /* * Find and remove part duplicates, i.e. parts counted @@ -754,6 +767,8 @@ key_def_merge(const struct key_def *first, const struct key_def *second) new_def->has_optional_parts = first->has_optional_parts || second->has_optional_parts; new_def->is_multikey = first->is_multikey || second->is_multikey; + new_def->functional_part_count = first->functional_part_count; + new_def->func_index_func = first->func_index_func; /* JSON paths data in the new key_def. */ char *path_pool = (char *)new_def + key_def_sizeof(new_part_count, 0); @@ -826,7 +841,7 @@ key_def_find_pk_in_cmp_def(const struct key_def *cmp_def, } /* Finally, allocate the new key definition. */ - extracted_def = key_def_new(parts, pk_def->part_count); + extracted_def = key_def_new(parts, pk_def->part_count, false); out: region_truncate(region, region_svp); return extracted_def; diff --git a/src/box/lua/key_def.c b/src/box/lua/key_def.c index 041b5ec98..3a3a5ec0c 100644 --- a/src/box/lua/key_def.c +++ b/src/box/lua/key_def.c @@ -445,7 +445,7 @@ lbox_key_def_new(struct lua_State *L) lua_pop(L, 1); } - struct key_def *key_def = key_def_new(parts, part_count); + struct key_def *key_def = key_def_new(parts, part_count, false); region_truncate(region, region_svp); if (key_def == NULL) return luaT_error(L); diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index fbb3151c9..95a1ecb90 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -1248,6 +1248,8 @@ memtx_index_def_change_requires_rebuild(struct index *index, return true; if (!old_def->opts.is_unique && new_def->opts.is_unique) return true; + if (old_def->opts.functional_fid != new_def->opts.functional_fid) + return true; const struct key_def *old_cmp_def, *new_cmp_def; if (index_depends_on_pk(index)) { diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index a8e7b8080..216ef3912 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -659,6 +659,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "HASH index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "HASH index cannot be functional"); + return -1; + } break; case TREE: /* TREE index has no limitations. */ @@ -688,6 +694,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "RTREE index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "RTREE index cannot be functional"); + return -1; + } /* no furter checks of parts needed */ return 0; case BITSET: @@ -716,6 +728,12 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) "BITSET index cannot be multikey"); return -1; } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_MODIFY_INDEX, + index_def->name, space_name(space), + "BITSET index cannot be functional"); + return -1; + } /* no furter checks of parts needed */ return 0; default: diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index dc167e113..678d2f8e2 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -35,6 +35,7 @@ #include "errinj.h" #include "memory.h" #include "fiber.h" +#include "func_key.h" #include "tuple.h" #include <third_party/qsort_arg.h> #include <small/mempool.h> @@ -723,6 +724,190 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple, return 0; } + +/** Release a memory is allocated for func_key_hint. */ +static void +memtx_tree_data_func_hint_destructor(struct memtx_tree_data *data) +{ + func_key_hint_delete(data->tuple, data->hint); +} + +/** + * The journal for multikey functional index replace operatoin + * is required to rollback an incomplete action, restore the + * original func_key_hint(s) hints bouth as to commit a completed + * replace action and destruct useless func_key_hint(s) hints. +*/ +struct journal_entry { + /** An inserted record copy. */ + struct memtx_tree_data inserted; + /** A replaced record copy. */ + struct memtx_tree_data replaced; +}; + +/** + * Rollback a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Routine uses given journal + * to return given index object in it's original state. + */ +static void +memtx_tree_index_replace_functional_rollback(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple != NULL) { + memtx_tree_insert(&index->tree, journal[i].replaced, + NULL); + } else { + memtx_tree_delete_value(&index->tree, + journal[i].inserted, NULL); + } + memtx_tree_data_func_hint_destructor(&journal[i].inserted); + } +} + +/** + * Commit a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Rotine uses given operations + * journal to release unused memory. + */ +static void +memtx_tree_index_replace_functional_commit(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + (void) index; + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple == NULL) + continue; + memtx_tree_data_func_hint_destructor(&journal[i].replaced); + } +} + +/** + * Functional index replace method works like + * memtx_tree_index_replace_multikey exept few moments. + * It uses functional index function from key definition + * to evaluate a key and allocate it on region. Then each + * returned key is reallocated in engine's memory as + * func_key_hint object and is used as comparison hint. + * To control func_key_hint(s) lifecycle in case of functional + * index we use a tiny journal object is allocated on region. + * It allows to restore original nodes with their original + * func_key_hint(s) pointers in case of failure and release + * useless hints of replaced items in case of success. + */ +static int +memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_functional(cmp_def)); + + int rc = -1; + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + *result = NULL; + const char *key, *key_end; + uint32_t key_sz, key_cnt; + struct func_key_iterator it; + if (new_tuple != NULL) { + key = func_key_extract(new_tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + goto end; + + int journal_idx = 0; + struct journal_entry *journal = + region_alloc(region, key_cnt * sizeof(*journal)); + if (journal == NULL) { + diag_set(OutOfMemory, key_cnt * sizeof(*journal), + "region", "journal"); + goto end; + } + + int err = 0; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && + key != NULL) { + hint_t key_hint = func_key_hint_new(new_tuple, key, + key_sz); + if (key_hint == HINT_NONE) { + err = -1; + break; + } + + /* Perform insertion, log it in journal. */ + bool is_multikey_conflict; + journal[journal_idx].replaced.tuple = NULL; + journal[journal_idx].inserted.tuple = new_tuple; + journal[journal_idx].inserted.hint = key_hint; + err = memtx_tree_index_replace_multikey_one(index, + old_tuple, new_tuple, mode, + key_hint, + &journal[journal_idx].replaced, + &is_multikey_conflict); + if (err != 0) + break; + /** + * Modify a 'replace' record of journal + * because an original node shouldn't be + * restored in case of multikey conflict. + */ + if (is_multikey_conflict) + journal[journal_idx].replaced.tuple = NULL; + else if (journal[journal_idx].replaced.tuple != NULL) + *result = journal[journal_idx].replaced.tuple; + + ++journal_idx; + } + if (key != NULL || err != 0) { + memtx_tree_index_replace_functional_rollback(index, + journal, journal_idx); + goto end; + } + if (*result != NULL) { + assert(old_tuple == NULL || old_tuple == *result); + old_tuple = *result; + } + memtx_tree_index_replace_functional_commit(index, + journal, journal_idx); + } + if (old_tuple != NULL) { + key = func_key_extract(old_tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + goto end; + struct memtx_tree_data data, deleted_data; + data.tuple = old_tuple; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && + key != NULL) { + data.hint = (hint_t) key; /* Raw data. */ + deleted_data.tuple = NULL; + memtx_tree_delete_value(&index->tree, data, + &deleted_data); + if (deleted_data.tuple != NULL) { + /* + * Release related hint on + * successfull node deletion. + */ + memtx_tree_data_func_hint_destructor( + &deleted_data); + } + } + assert(key == NULL); + } + rc = 0; +end: + region_truncate(region, region_svp); + return rc; +} + static struct iterator * memtx_tree_index_create_iterator(struct index *base, enum iterator_type type, const char *key, uint32_t part_count) @@ -854,13 +1039,52 @@ memtx_tree_index_build_next_multikey(struct index *base, struct tuple *tuple) return 0; } +static int +memtx_tree_index_build_next_functional(struct index *base, struct tuple *tuple) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_functional(cmp_def)); + + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + uint32_t key_sz, key_cnt; + const char *key, *key_end; + key = func_key_extract(tuple, cmp_def->func_index_func, + &key_end, &key_cnt); + if (key == NULL) + return -1; + + uint32_t insert_idx = index->build_array_size; + struct func_key_iterator it; + func_key_iterator_create(&it, key, key_end, cmp_def, true); + while (func_key_iterator_next(&it, &key, &key_sz) == 0 && key != NULL) { + hint_t key_hint = func_key_hint_new(tuple, key, key_sz); + if (key_hint == HINT_NONE) + goto error; + if (memtx_tree_index_build_array_append(index, tuple, + key_hint) != 0) + goto error; + } + assert(key == NULL); + region_truncate(region, region_svp); + return 0; +error: + for (uint32_t i = insert_idx; i < index->build_array_size; i++) + memtx_tree_data_func_hint_destructor(&index->build_array[i]); + region_truncate(region, region_svp); + return -1; +} + /** * Process build_array of specified index and remove duplicates * of equal tuples (in terms of index's cmp_def and have same * tuple pointer). The build_array is expected to be sorted. */ static void -memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index) +memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index, + void (*data_destructor)(struct memtx_tree_data *data)) { if (index->build_array_size == 0) return; @@ -878,10 +1102,18 @@ memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index) if (++w_idx == r_idx) continue; index->build_array[w_idx] = index->build_array[r_idx]; + if (data_destructor == NULL) + continue; + for (size_t i = w_idx + 1; i < r_idx; i++) + data_destructor(&index->build_array[i]); } r_idx++; } index->build_array_size = w_idx + 1; + if (data_destructor != NULL) { + for (size_t i = index->build_array_size + 1; i < r_idx; i++) + data_destructor(&index->build_array[i]); + } } static void @@ -899,7 +1131,10 @@ memtx_tree_index_end_build(struct index *base) * the following memtx_tree_build assumes that * all keys are unique. */ - memtx_tree_index_build_array_deduplicate(index); + memtx_tree_index_build_array_deduplicate(index, NULL); + } else if (key_def_is_functional(cmp_def)) { + memtx_tree_index_build_array_deduplicate(index, + memtx_tree_data_func_hint_destructor); } memtx_tree_build(&index->tree, index->build_array, index->build_array_size); @@ -1026,6 +1261,72 @@ static const struct index_vtab memtx_tree_index_multikey_vtab = { /* .end_build = */ memtx_tree_index_end_build, }; +static const struct index_vtab memtx_tree_index_functional_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ memtx_tree_index_update_def, + /* .depends_on_pk = */ memtx_tree_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + memtx_index_def_change_requires_rebuild, + /* .size = */ memtx_tree_index_size, + /* .bsize = */ memtx_tree_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ memtx_tree_index_random, + /* .count = */ memtx_tree_index_count, + /* .get = */ memtx_tree_index_get, + /* .replace = */ memtx_tree_index_replace_functional, + /* .create_iterator = */ memtx_tree_index_create_iterator, + /* .create_snapshot_iterator = */ + memtx_tree_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ memtx_tree_index_begin_build, + /* .reserve = */ memtx_tree_index_reserve, + /* .build_next = */ memtx_tree_index_build_next_functional, + /* .end_build = */ memtx_tree_index_end_build, +}; + +/** + * A disabled index vtab provides safe dummy methods for + * 'inactive' index. It is required to perform a fault-tolerant + * recovery from snapshoot in case of functional index (because + * key defintion is not completely initialized at that moment). + */ +static const struct index_vtab memtx_tree_index_disabled_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ generic_index_update_def, + /* .depends_on_pk = */ generic_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + generic_index_def_change_requires_rebuild, + /* .size = */ generic_index_size, + /* .bsize = */ generic_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ generic_index_random, + /* .count = */ generic_index_count, + /* .get = */ generic_index_get, + /* .replace = */ disabled_index_replace, + /* .create_iterator = */ generic_index_create_iterator, + /* .create_snapshot_iterator = */ + generic_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ generic_index_begin_build, + /* .reserve = */ generic_index_reserve, + /* .build_next = */ disabled_index_build_next, + /* .end_build = */ generic_index_end_build, +}; + struct index * memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) { @@ -1036,9 +1337,17 @@ memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) "malloc", "struct memtx_tree_index"); return NULL; } - const struct index_vtab *vtab = def->key_def->is_multikey ? - &memtx_tree_index_multikey_vtab : - &memtx_tree_index_vtab; + const struct index_vtab *vtab; + if (key_def_is_functional(def->key_def)) { + if (def->key_def->func_index_func == NULL) + vtab = &memtx_tree_index_disabled_vtab; + else + vtab = &memtx_tree_index_functional_vtab; + } else if (def->key_def->is_multikey) { + vtab = &memtx_tree_index_multikey_vtab; + } else { + vtab = &memtx_tree_index_vtab; + } if (index_create(&index->base, (struct engine *)memtx, vtab, def) != 0) { free(index); diff --git a/src/box/sql.c b/src/box/sql.c index 4c9a4c15b..0ab3a506f 100644 --- a/src/box/sql.c +++ b/src/box/sql.c @@ -347,7 +347,7 @@ sql_ephemeral_space_create(uint32_t field_count, struct sql_key_info *key_info) } } struct key_def *ephemer_key_def = key_def_new(ephemer_key_parts, - field_count); + field_count, false); if (ephemer_key_def == NULL) return NULL; diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 2aefa2a3f..1f62f4136 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -2334,7 +2334,7 @@ index_fill_def(struct Parse *parse, struct index *index, part->coll_id = coll_id; part->path = NULL; } - key_def = key_def_new(key_parts, expr_list->nExpr); + key_def = key_def_new(key_parts, expr_list->nExpr, false); if (key_def == NULL) goto tnt_error; /* diff --git a/src/box/sql/select.c b/src/box/sql/select.c index bd144c1bb..5d2cd42bf 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1438,7 +1438,7 @@ sql_key_info_to_key_def(struct sql_key_info *key_info) { if (key_info->key_def == NULL) { key_info->key_def = key_def_new(key_info->parts, - key_info->part_count); + key_info->part_count, false); } return key_info->key_def; } diff --git a/src/box/sql/where.c b/src/box/sql/where.c index 5458c6a75..ed507bf4d 100644 --- a/src/box/sql/where.c +++ b/src/box/sql/where.c @@ -2775,7 +2775,7 @@ whereLoopAddBtree(WhereLoopBuilder * pBuilder, /* WHERE clause information */ part.coll_id = COLL_NONE; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1); + struct key_def *key_def = key_def_new(&part, 1, false); if (key_def == NULL) { tnt_error: pWInfo->pParse->is_aborted = true; diff --git a/src/box/vinyl.c b/src/box/vinyl.c index cf7af26b7..5582da833 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -701,6 +701,11 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) return -1; } } + if (key_def_is_functional(index_def->key_def)) { + diag_set(ClientError, ER_UNSUPPORTED, "Vinyl", + "functional indexes"); + return -1; + } return 0; } @@ -986,6 +991,8 @@ vinyl_index_def_change_requires_rebuild(struct index *index, if (!old_def->opts.is_unique && new_def->opts.is_unique) return true; + if (old_def->opts.functional_fid != new_def->opts.functional_fid) + return true; assert(index_depends_on_pk(index)); const struct key_def *old_cmp_def = old_def->cmp_def; @@ -3165,7 +3172,7 @@ vy_send_lsm(struct vy_join_ctx *ctx, struct vy_lsm_recovery_info *lsm_info) /* Create key definition and tuple format. */ ctx->key_def = key_def_new(lsm_info->key_parts, - lsm_info->key_part_count); + lsm_info->key_part_count, false); if (ctx->key_def == NULL) goto out; ctx->format = vy_stmt_format_new(&ctx->env->stmt_env, &ctx->key_def, 1, diff --git a/test/unit/luaT_tuple_new.c b/test/unit/luaT_tuple_new.c index 0a16fa039..609d64e45 100644 --- a/test/unit/luaT_tuple_new.c +++ b/test/unit/luaT_tuple_new.c @@ -124,7 +124,7 @@ test_basic(struct lua_State *L) part.nullable_action = ON_CONFLICT_ACTION_DEFAULT; part.sort_order = SORT_ORDER_ASC; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1); + struct key_def *key_def = key_def_new(&part, 1, false); box_tuple_format_t *another_format = box_tuple_format_new(&key_def, 1); key_def_delete(key_def); diff --git a/test/unit/merger.test.c b/test/unit/merger.test.c index b4a989a20..345a2364e 100644 --- a/test/unit/merger.test.c +++ b/test/unit/merger.test.c @@ -214,7 +214,7 @@ test_merger(struct tuple_format *format) merge_source_array_new(true), }; - struct key_def *key_def = key_def_new(&key_part_unsigned, 1); + struct key_def *key_def = key_def_new(&key_part_unsigned, 1, false); struct merge_source *merger = merger_new(key_def, sources, source_count, false); key_def_delete(key_def); @@ -252,7 +252,7 @@ test_basic() plan(4); header(); - struct key_def *key_def = key_def_new(&key_part_integer, 1); + struct key_def *key_def = key_def_new(&key_part_integer, 1, false); struct tuple_format *format = box_tuple_format_new(&key_def, 1); assert(format != NULL); diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt index 481842a39..1c6fda447 100644 --- a/src/box/CMakeLists.txt +++ b/src/box/CMakeLists.txt @@ -101,6 +101,7 @@ add_library(box STATIC fk_constraint.c func.c func_def.c + func_key.c alter.cc schema.cc schema_def.c diff --git a/src/box/alter.cc b/src/box/alter.cc index f98a77a51..93ba7f137 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -45,6 +45,7 @@ #include "fiber.h" /* for gc_pool */ #include "scoped_guard.h" #include "third_party/base64.h" +#include "memtx_engine.h" #include <new> /* for placement new */ #include <stdio.h> /* snprintf() */ #include <ctype.h> @@ -195,7 +196,7 @@ err: */ static void index_opts_decode(struct index_opts *opts, const char *map, - struct region *region) + struct space *space, struct region *region) { index_opts_create(opts); if (opts_decode(opts, index_opts_reg, &map, ER_WRONG_INDEX_OPTIONS, @@ -229,6 +230,22 @@ index_opts_decode(struct index_opts *opts, const char *map, "bloom_fpr must be greater than 0 and " "less than or equal to 1"); } + /** + * Can't verify functional index function + * reference on load because the function object + * had not been registered in Tarantool yet. + */ + struct memtx_engine *engine = (struct memtx_engine *)space->engine; + if (opts->functional_fid > 0 && engine->state == MEMTX_OK) { + struct func *func = func_cache_find(opts->functional_fid); + if (func->def->language != FUNC_LANGUAGE_LUA || + func->def->body == NULL || !func->def->is_deterministic || + !func->def->is_sandboxed) { + tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, + "referenced function doesn't satisfy " + "functional index constraints"); + } + } } /** @@ -260,7 +277,7 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) const char *opts_field = tuple_field_with_type_xc(tuple, BOX_INDEX_FIELD_OPTS, MP_MAP); - index_opts_decode(&opts, opts_field, &fiber()->gc); + index_opts_decode(&opts, opts_field, space, &fiber()->gc); const char *parts = tuple_field(tuple, BOX_INDEX_FIELD_PARTS); uint32_t part_count = mp_decode_array(&parts); if (name_len > BOX_NAME_MAX) { @@ -285,7 +302,7 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) space->def->fields, space->def->field_count, &fiber()->gc) != 0) diag_raise(); - key_def = key_def_new(part_def, part_count); + key_def = key_def_new(part_def, part_count, opts.functional_fid > 0); if (key_def == NULL) diag_raise(); struct index_def *index_def = @@ -1370,6 +1387,30 @@ RebuildIndex::~RebuildIndex() index_def_delete(new_index_def); } +/** + * RebuildFunctionalIndex - prepare functional index definition, + * drop the old index data and rebuild index from by reading the + * primary key. + */ +class RebuildFunctionalIndex: public RebuildIndex +{ + struct index_def * + functional_index_def_new(struct index_def *index_def, + struct func *func) + { + struct index_def *new_index_def = index_def_dup_xc(index_def); + index_def_update_func(new_index_def, func); + return new_index_def; + } +public: + RebuildFunctionalIndex(struct alter_space *alter, + struct index_def *old_index_def_arg, + struct func *func) : + RebuildIndex(alter, + functional_index_def_new(old_index_def_arg, func), + old_index_def_arg) {} +}; + /** TruncateIndex - truncate an index. */ class TruncateIndex: public AlterSpaceOp { @@ -2841,6 +2882,12 @@ on_replace_dd_func(struct trigger * /* trigger */, void *event) (unsigned) old_func->def->uid, "function has grants"); } + if (old_func != NULL && + space_has_data(BOX_FUNC_INDEX_ID, 1, old_func->def->fid)) { + tnt_raise(ClientError, ER_DROP_FUNCTION, + (unsigned) old_func->def->uid, + "function has references"); + } struct trigger *on_commit = txn_alter_trigger_new(on_drop_func_commit, old_func); struct trigger *on_rollback = @@ -4689,6 +4736,56 @@ on_replace_dd_ck_constraint(struct trigger * /* trigger*/, void *event) trigger_run_xc(&on_alter_space, space); } +/** A trigger invoked on replace in the _func_index space. */ +static void +on_replace_dd_func_index(struct trigger *trigger, void *event) +{ + (void) trigger; + struct txn *txn = (struct txn *) event; + struct txn_stmt *stmt = txn_current_stmt(txn); + struct tuple *old_tuple = stmt->old_tuple; + struct tuple *new_tuple = stmt->new_tuple; + + struct alter_space *alter = NULL; + struct func *func = NULL; + struct index *index; + struct space *space; + if (old_tuple == NULL && new_tuple != NULL) { + uint32_t space_id = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FIELD_SPACE_ID); + uint32_t index_id = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FIELD_INDEX_ID); + uint32_t fid = tuple_field_u32_xc(new_tuple, + BOX_FUNC_INDEX_FUNCTION_ID); + space = space_cache_find_xc(space_id); + index = index_find_xc(space, index_id); + func = func_cache_find(fid); + } else if (old_tuple != NULL && new_tuple == NULL) { + uint32_t space_id = tuple_field_u32_xc(old_tuple, + BOX_FUNC_INDEX_FIELD_SPACE_ID); + uint32_t index_id = tuple_field_u32_xc(old_tuple, + BOX_FUNC_INDEX_FIELD_INDEX_ID); + space = space_cache_find_xc(space_id); + index = index_find_xc(space, index_id); + func = NULL; + } else { + assert(old_tuple != NULL && new_tuple != NULL); + tnt_raise(ClientError, ER_UNSUPPORTED, "func_index", "alter"); + } + + alter = alter_space_new(space); + auto scoped_guard = make_scoped_guard([=] {alter_space_delete(alter);}); + alter_space_move_indexes(alter, 0, index->def->iid); + (void) new RebuildFunctionalIndex(alter, index->def, func); + alter_space_move_indexes(alter, index->def->iid + 1, + space->index_id_max + 1); + (void) new MoveCkConstraints(alter); + (void) new UpdateSchemaVersion(alter); + alter_space_do(txn, alter); + + scoped_guard.is_active = false; +} + struct trigger alter_space_on_replace_space = { RLIST_LINK_INITIALIZER, on_replace_dd_space, NULL, NULL }; @@ -4749,4 +4846,8 @@ struct trigger on_replace_ck_constraint = { RLIST_LINK_INITIALIZER, on_replace_dd_ck_constraint, NULL, NULL }; +struct trigger on_replace_func_index = { + RLIST_LINK_INITIALIZER, on_replace_dd_func_index, NULL, NULL +}; + /* vim: set foldmethod=marker */ diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap index ae698731e480e0d8cc250659222032291922e549..a52256b65c113a2055f0a8f65f9b90398d3816d5 100644 GIT binary patch delta 5146 diff --git a/src/box/index.cc b/src/box/index.cc index 843d0e73d..00a1b502e 100644 --- a/src/box/index.cc +++ b/src/box/index.cc @@ -679,6 +679,16 @@ generic_index_replace(struct index *index, struct tuple *old_tuple, return -1; } +struct iterator * +generic_index_create_iterator(struct index *base, enum iterator_type type, + const char *key, uint32_t part_count) +{ + (void) type; (void) key; (void) part_count; + diag_set(UnsupportedIndexFeature, base->def, "read view"); + return NULL; +} + + struct snapshot_iterator * generic_index_create_snapshot_iterator(struct index *index) { @@ -729,4 +739,22 @@ generic_index_end_build(struct index *) { } +int +disabled_index_build_next(struct index *index, struct tuple *tuple) +{ + (void) index; (void) tuple; + return 0; +} + +int +disabled_index_replace(struct index *index, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result) +{ + (void) old_tuple; (void) new_tuple; (void) mode; + (void) index; + *result = NULL; + return 0; +} + /* }}} */ diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua index aadcd3fa9..4b2273c10 100644 --- a/src/box/lua/schema.lua +++ b/src/box/lua/schema.lua @@ -515,6 +515,7 @@ box.schema.space.drop = function(space_id, space_name, opts) local _space_sequence = box.space[box.schema.SPACE_SEQUENCE_ID] local _fk_constraint = box.space[box.schema.FK_CONSTRAINT_ID] local _ck_constraint = box.space[box.schema.CK_CONSTRAINT_ID] + local _func_index = box.space[box.schema.FUNC_INDEX_ID] local sequence_tuple = _space_sequence:delete{space_id} if sequence_tuple ~= nil and sequence_tuple.is_generated == true then -- Delete automatically generated sequence. @@ -529,6 +530,9 @@ box.schema.space.drop = function(space_id, space_name, opts) for _, t in _ck_constraint.index.primary:pairs({space_id}) do _ck_constraint:delete({space_id, t.name}) end + for _, t in _func_index.index.primary:pairs({space_id}) do + _func_index:delete({space_id, t.index_id, t.func_id}) + end local keys = _vindex:select(space_id) for i = #keys, 1, -1 do local v = keys[i] @@ -956,6 +960,8 @@ local index_options = { range_size = 'number', page_size = 'number', bloom_fpr = 'number', + func = 'number, string', + is_multikey = 'boolean', } -- @@ -980,6 +986,15 @@ end local create_index_template = table.deepcopy(alter_index_template) create_index_template.if_not_exists = "boolean" +-- Find a function id by given function name +local function func_id_by_name(func_name) + local func = box.space._func.index.name:get(func_name) + if func == nil then + box.error(box.error.NO_SUCH_FUNCTION, func_name) + end + return func.id +end + box.schema.index.create = function(space_id, name, options) check_param(space_id, 'space_id', 'number') check_param(name, 'name', 'string') @@ -1061,6 +1076,8 @@ box.schema.index.create = function(space_id, name, options) run_count_per_level = options.run_count_per_level, run_size_ratio = options.run_size_ratio, bloom_fpr = options.bloom_fpr, + func = options.func, + is_multikey = options.is_multikey, } local field_type_aliases = { num = 'unsigned'; -- Deprecated since 1.7.2 @@ -1081,11 +1098,18 @@ box.schema.index.create = function(space_id, name, options) if parts_can_be_simplified then parts = simplify_index_parts(parts) end + if index_opts.func ~= nil and type(index_opts.func) == 'string' then + index_opts.func = func_id_by_name(index_opts.func) + end local sequence_proxy = space_sequence_alter_prepare(format, parts, options, space_id, iid, space.name, name) _index:insert{space_id, iid, name, options.type, index_opts, parts} space_sequence_alter_commit(sequence_proxy) + if index_opts.func ~= nil then + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + _func_index:insert{space_id, iid, index_opts.func} + end return space.index[name] end @@ -1101,6 +1125,10 @@ box.schema.index.drop = function(space_id, index_id) end end local _index = box.space[box.schema.INDEX_ID] + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + for _, v in box.space._func_index:pairs{space_id, index_id} do + _func_index:delete(v) + end _index:delete{space_id, index_id} end @@ -1197,11 +1225,18 @@ box.schema.index.alter = function(space_id, index_id, options) parts = simplify_index_parts(parts) end end + if index_opts.func ~= nil and type(index_opts.func) == 'string' then + index_opts.func = func_id_by_name(index_opts.func) + end local sequence_proxy = space_sequence_alter_prepare(format, parts, options, space_id, index_id, space.name, options.name) _index:replace{space_id, index_id, options.name, options.type, index_opts, parts} + if index_opts.func ~= nil then + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + _func_index:insert{space_id, iid, index_opts.func} + end space_sequence_alter_commit(sequence_proxy) end diff --git a/src/box/lua/space.cc b/src/box/lua/space.cc index 18039fd6a..c8ad0be36 100644 --- a/src/box/lua/space.cc +++ b/src/box/lua/space.cc @@ -335,6 +335,11 @@ lbox_fillspace(struct lua_State *L, struct space *space, int i) lua_setfield(L, -2, "dimension"); } + if (index_opts->functional_fid > 0) { + lua_pushnumber(L, index_opts->functional_fid); + lua_setfield(L, -2, "func_id"); + } + lua_pushstring(L, index_type_strs[index_def->type]); lua_setfield(L, -2, "type"); @@ -629,6 +634,8 @@ box_lua_space_init(struct lua_State *L) lua_setfield(L, -2, "VSEQUENCE_ID"); lua_pushnumber(L, BOX_SPACE_SEQUENCE_ID); lua_setfield(L, -2, "SPACE_SEQUENCE_ID"); + lua_pushnumber(L, BOX_FUNC_INDEX_ID); + lua_setfield(L, -2, "FUNC_INDEX_ID"); lua_pushnumber(L, BOX_SYSTEM_ID_MIN); lua_setfield(L, -2, "SYSTEM_ID_MIN"); lua_pushnumber(L, BOX_SYSTEM_ID_MAX); diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua index 02c1cb0ff..f570a1c08 100644 --- a/src/box/lua/upgrade.lua +++ b/src/box/lua/upgrade.lua @@ -885,11 +885,29 @@ local function upgrade_func_to_2_2_1() collation = 'unicode_ci'}}}) end +local function create_func_index() + log.info("Create _func_index space") + local _func_index = box.space[box.schema.FUNC_INDEX_ID] + local _space = box.space._space + local _index = box.space._index + local format = {{name='space_id', type='unsigned'}, + {name='index_id', type='unsigned'}, + {name='func_id', type='unsigned'}} + _space:insert{_func_index.id, ADMIN, '_func_index', 'memtx', 0, + setmap({}), format} + _index:insert{_func_index.id, 0, 'primary', 'tree', {unique = true}, + {{0, 'unsigned'}, {1, 'unsigned'}, {2, 'unsigned'}}} + _index:insert{_func_index.id, 1, 'fid', 'tree', {unique = false}, + {{2, 'unsigned'}}} + +end + local function upgrade_to_2_2_1() upgrade_sequence_to_2_2_1() upgrade_ck_constraint_to_2_2_1() create_vcollation_space() upgrade_func_to_2_2_1() + create_func_index() end -------------------------------------------------------------------------------- diff --git a/src/box/schema.cc b/src/box/schema.cc index 20666386d..5d4a3ff00 100644 --- a/src/box/schema.cc +++ b/src/box/schema.cc @@ -266,7 +266,7 @@ sc_space_new(uint32_t id, const char *name, uint32_t key_part_count, struct trigger *replace_trigger) { - struct key_def *key_def = key_def_new(key_parts, key_part_count); + struct key_def *key_def = key_def_new(key_parts, key_part_count, false); if (key_def == NULL) diag_raise(); auto key_def_guard = @@ -462,6 +462,16 @@ schema_init() sc_space_new(BOX_CK_CONSTRAINT_ID, "_ck_constraint", key_parts, 2, &on_replace_ck_constraint); + /* _func_index - check constraints. */ + key_parts[0].fieldno = 0; /* space id */ + key_parts[0].type = FIELD_TYPE_UNSIGNED; + key_parts[1].fieldno = 1; /* index id */ + key_parts[1].type = FIELD_TYPE_UNSIGNED; + key_parts[2].fieldno = 2; /* function id */ + key_parts[2].type = FIELD_TYPE_UNSIGNED; + sc_space_new(BOX_FUNC_INDEX_ID, "_func_index", key_parts, 3, + &on_replace_func_index); + /* * _vinyl_deferred_delete - blackhole that is needed * for writing deferred DELETE statements generated by diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc index c03b584d0..2ea6f7620 100644 --- a/src/box/tuple_compare.cc +++ b/src/box/tuple_compare.cc @@ -1271,6 +1271,80 @@ static const comparator_with_key_signature cmp_wk_arr[] = { KEY_COMPARATOR(1, FIELD_TYPE_STRING , 2, FIELD_TYPE_STRING) }; +template<bool is_nullable> +static inline int +functional_compare(struct tuple *tuple_a, hint_t tuple_a_hint, + struct tuple *tuple_b, hint_t tuple_b_hint, + struct key_def *key_def) +{ + assert(key_def_is_functional(key_def)); + assert(is_nullable == key_def->is_nullable); + + const char *key_a = (const char *)tuple_a_hint; + const char *key_b = (const char *)tuple_b_hint; + assert(mp_typeof(*key_a) == MP_ARRAY); + uint32_t part_count_a = mp_decode_array(&key_a); + assert(mp_typeof(*key_b) == MP_ARRAY); + uint32_t part_count_b = mp_decode_array(&key_b); + + uint32_t key_part_count = MIN(part_count_a, part_count_b); + uint32_t part_count = MIN(key_def->functional_part_count, + key_part_count); + int rc = key_compare_parts<is_nullable>(key_a, key_b, part_count, + key_def); + if (rc != 0) + return rc; + /* + * Primary index definiton key compare. + * It cannot contain nullable parts so the code is + * simplified correspondingly. + */ + const char *tuple_a_raw = tuple_data(tuple_a); + const char *tuple_b_raw = tuple_data(tuple_b); + struct tuple_format *format_a = tuple_format(tuple_a); + struct tuple_format *format_b = tuple_format(tuple_b); + const uint32_t *field_map_a = tuple_field_map(tuple_a); + const uint32_t *field_map_b = tuple_field_map(tuple_b); + const char *field_a, *field_b; + for (uint32_t i = key_def->functional_part_count; + i < key_def->part_count; i++) { + struct key_part *part = &key_def->parts[i]; + field_a = tuple_field_raw_by_part(format_a, tuple_a_raw, + field_map_a, part, + MULTIKEY_NONE); + field_b = tuple_field_raw_by_part(format_b, tuple_b_raw, + field_map_b, part, + MULTIKEY_NONE); + assert(field_a != NULL && field_b != NULL); + rc = tuple_compare_field(field_a, field_b, part->type, + part->coll); + if (rc != 0) + return rc; + else + continue; + } + return 0; +} + +template<bool is_nullable> +static inline int +functional_compare_with_key(struct tuple *tuple, hint_t tuple_hint, + const char *key, uint32_t part_count, + hint_t key_hint, struct key_def *key_def) +{ + (void)tuple; (void)key_hint; + assert(key_def->functional_part_count > 0); + assert(is_nullable == key_def->is_nullable); + const char *tuple_key = (const char *)tuple_hint; + assert(mp_typeof(*tuple_key) == MP_ARRAY); + + uint32_t tuple_key_count = mp_decode_array(&tuple_key); + part_count = MIN(part_count, tuple_key_count); + part_count = MIN(part_count, key_def->functional_part_count); + return key_compare_parts<is_nullable>(tuple_key, key, part_count, + key_def); +} + #undef KEY_COMPARATOR /* }}} tuple_compare_with_key */ @@ -1592,7 +1666,7 @@ tuple_hint(struct tuple *tuple, struct key_def *key_def) } static hint_t -key_hint_multikey(const char *key, uint32_t part_count, struct key_def *key_def) +key_hint_stub(const char *key, uint32_t part_count, struct key_def *key_def) { (void) key; (void) part_count; @@ -1600,19 +1674,19 @@ key_hint_multikey(const char *key, uint32_t part_count, struct key_def *key_def) /* * Multikey hint for tuple is an index of the key in * array, it always must be defined. While - * tuple_hint_multikey assumes that it must be + * key_hint_stub assumes that it must be * initialized manually (so it mustn't be called), * the virtual method for a key makes sense. Overriding * this method such way, we extend existend code to * do nothing on key hint calculation an it is valid * because it is never used(unlike tuple hint). */ - assert(key_def->is_multikey); + assert(key_def->is_multikey || key_def_is_functional(key_def)); return HINT_NONE; } static hint_t -tuple_hint_multikey(struct tuple *tuple, struct key_def *key_def) +key_hint_stub(struct tuple *tuple, struct key_def *key_def) { (void) tuple; (void) key_def; @@ -1641,9 +1715,9 @@ key_def_set_hint_func(struct key_def *def) static void key_def_set_hint_func(struct key_def *def) { - if (def->is_multikey) { - def->key_hint = key_hint_multikey; - def->tuple_hint = tuple_hint_multikey; + if (def->is_multikey || key_def_is_functional(def)) { + def->key_hint = key_hint_stub; + def->tuple_hint = key_hint_stub; return; } switch (def->parts->type) { @@ -1769,10 +1843,24 @@ key_def_set_compare_func_json(struct key_def *def) } } +template<bool is_nullable> +static void +key_def_set_compare_func_functional(struct key_def *def) +{ + assert(key_def_is_functional(def)); + def->tuple_compare = functional_compare<is_nullable>; + def->tuple_compare_with_key = functional_compare_with_key<is_nullable>; +} + void key_def_set_compare_func(struct key_def *def) { - if (!key_def_has_collation(def) && + if (key_def_is_functional(def)) { + if (def->is_nullable) + key_def_set_compare_func_functional<true>(def); + else + key_def_set_compare_func_functional<false>(def); + } else if (!key_def_has_collation(def) && !def->is_nullable && !def->has_json_paths) { key_def_set_compare_func_fast(def); } else if (!def->has_json_paths) { diff --git a/src/box/tuple_extract_key.cc b/src/box/tuple_extract_key.cc index 471c7df80..c84809dae 100644 --- a/src/box/tuple_extract_key.cc +++ b/src/box/tuple_extract_key.cc @@ -120,6 +120,7 @@ tuple_extract_key_slowpath(struct tuple *tuple, struct key_def *key_def, key_def_contains_sequential_parts(key_def)); assert(is_multikey == key_def->is_multikey); assert(!key_def->is_multikey || multikey_idx != MULTIKEY_NONE); + assert(!key_def_is_functional(key_def)); assert(mp_sizeof_nil() == 1); const char *data = tuple_data(tuple); uint32_t part_count = key_def->part_count; @@ -251,6 +252,7 @@ tuple_extract_key_slowpath_raw(const char *data, const char *data_end, assert(!has_optional_parts || key_def->is_nullable); assert(has_optional_parts == key_def->has_optional_parts); assert(!key_def->is_multikey || multikey_idx != MULTIKEY_NONE); + assert(!key_def_is_functional(key_def)); assert(mp_sizeof_nil() == 1); /* allocate buffer with maximal possible size */ char *key = (char *) region_alloc(&fiber()->gc, data_end - data); @@ -367,6 +369,7 @@ key_def_set_extract_func_plain(struct key_def *def) { assert(!def->has_json_paths); assert(!def->is_multikey); + assert(!key_def_is_functional(def)); if (key_def_is_sequential(def)) { assert(contains_sequential_parts || def->part_count == 1); def->tuple_extract_key = tuple_extract_key_sequential @@ -387,6 +390,7 @@ static void key_def_set_extract_func_json(struct key_def *def) { assert(def->has_json_paths); + assert(!key_def_is_functional(def)); if (def->is_multikey) { def->tuple_extract_key = tuple_extract_key_slowpath <contains_sequential_parts, @@ -400,13 +404,36 @@ key_def_set_extract_func_json(struct key_def *def) <has_optional_parts, true>; } +static char * +tuple_extract_key_stub(struct tuple *tuple, struct key_def *key_def, + int multikey_idx, uint32_t *key_size) +{ + (void)tuple; (void)key_def; (void)multikey_idx; (void)key_size; + unreachable(); + return NULL; +} + +static char * +tuple_extract_key_raw_stub(const char *data, const char *data_end, + struct key_def *key_def, int multikey_idx, + uint32_t *key_size) +{ + (void)data; (void)data_end; + (void)key_def; (void)multikey_idx; (void)key_size; + unreachable(); + return NULL; +} + void key_def_set_extract_func(struct key_def *key_def) { bool contains_sequential_parts = key_def_contains_sequential_parts(key_def); bool has_optional_parts = key_def->has_optional_parts; - if (!key_def->has_json_paths) { + if (key_def_is_functional(key_def)) { + key_def->tuple_extract_key = tuple_extract_key_stub; + key_def->tuple_extract_key_raw = tuple_extract_key_raw_stub; + } else if (!key_def->has_json_paths) { if (!contains_sequential_parts && !has_optional_parts) { key_def_set_extract_func_plain<false, false>(key_def); } else if (!contains_sequential_parts && has_optional_parts) { diff --git a/src/box/tuple_hash.cc b/src/box/tuple_hash.cc index 780e3d053..dad87871a 100644 --- a/src/box/tuple_hash.cc +++ b/src/box/tuple_hash.cc @@ -365,6 +365,7 @@ tuple_hash_slowpath(struct tuple *tuple, struct key_def *key_def) assert(has_json_paths == key_def->has_json_paths); assert(has_optional_parts == key_def->has_optional_parts); assert(!key_def->is_multikey); + assert(!key_def_is_functional(key_def)); uint32_t h = HASH_SEED; uint32_t carry = 0; uint32_t total_size = 0; diff --git a/test/app-tap/tarantoolctl.test.lua b/test/app-tap/tarantoolctl.test.lua index 957b883f4..df2ee377f 100755 --- a/test/app-tap/tarantoolctl.test.lua +++ b/test/app-tap/tarantoolctl.test.lua @@ -405,8 +405,8 @@ do check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 1", "\n", 3) check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 1 --replica 2", "\n", 3) check_ctlcat_xlog(test_i, dir, "--from=3 --to=6 --format=json --show-system --replica 2", "\n", 0) - check_ctlcat_snap(test_i, dir, "--space=280", "---\n", 23) - check_ctlcat_snap(test_i, dir, "--space=288", "---\n", 50) + check_ctlcat_snap(test_i, dir, "--space=280", "---\n", 24) + check_ctlcat_snap(test_i, dir, "--space=288", "---\n", 52) end) end) diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result index f2d1f46fb..a5d645df8 100644 --- a/test/box-py/bootstrap.result +++ b/test/box-py/bootstrap.result @@ -93,6 +93,8 @@ box.space._space:select{} - [364, 1, '_ck_constraint', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'is_deferred', 'type': 'boolean'}, {'name': 'language', 'type': 'str'}, {'name': 'code', 'type': 'str'}]] + - [372, 1, '_func_index', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, + {'name': 'index_id', 'type': 'unsigned'}, {'name': 'func_id', 'type': 'unsigned'}]] ... box.space._index:select{} --- @@ -148,6 +150,9 @@ box.space._index:select{} - [356, 0, 'primary', 'tree', {'unique': true}, [[0, 'string'], [1, 'unsigned']]] - [356, 1, 'child_id', 'tree', {'unique': false}, [[1, 'unsigned']]] - [364, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'string']]] + - [372, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'unsigned'], + [2, 'unsigned']]] + - [372, 1, 'fid', 'tree', {'unique': false}, [[2, 'unsigned']]] ... box.space._user:select{} --- diff --git a/test/box/access.result b/test/box/access.result index 5ee92a443..ba72b5f74 100644 --- a/test/box/access.result +++ b/test/box/access.result @@ -1496,6 +1496,9 @@ box.schema.user.grant('tester', 'read', 'space', '_fk_constraint') box.schema.user.grant('tester', 'read', 'space', '_ck_constraint') --- ... +box.schema.user.grant('tester', 'read', 'space', '_func_index') +--- +... box.session.su("tester") --- ... diff --git a/test/box/access.test.lua b/test/box/access.test.lua index 79340b0f5..219cdb04a 100644 --- a/test/box/access.test.lua +++ b/test/box/access.test.lua @@ -557,6 +557,7 @@ box.schema.user.grant('tester', 'read', 'space', '_space_sequence') box.schema.user.grant('tester', 'read', 'space', '_trigger') box.schema.user.grant('tester', 'read', 'space', '_fk_constraint') box.schema.user.grant('tester', 'read', 'space', '_ck_constraint') +box.schema.user.grant('tester', 'read', 'space', '_func_index') box.session.su("tester") -- successful create s1 = box.schema.space.create("test_space") diff --git a/test/box/access_misc.result b/test/box/access_misc.result index c69cf0283..31b935914 100644 --- a/test/box/access_misc.result +++ b/test/box/access_misc.result @@ -833,140 +833,142 @@ box.space._space:select() - [364, 1, '_ck_constraint', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'is_deferred', 'type': 'boolean'}, {'name': 'language', 'type': 'str'}, {'name': 'code', 'type': 'str'}]] + - [372, 1, '_func_index', 'memtx', 0, {}, [{'name': 'space_id', 'type': 'unsigned'}, + {'name': 'index_id', 'type': 'unsigned'}, {'name': 'func_id', 'type': 'unsigned'}]] ... box.space._func:select() --- - - [1, 1, 'box.schema.user.info', 1, 'LUA', '', 'function', [], 'any', 'none', 'none', - false, false, true, ['LUA'], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, ['LUA'], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [2, 1, 'TRIM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [3, 1, 'TYPEOF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [4, 1, 'PRINTF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [5, 1, 'UNICODE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [6, 1, 'CHAR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [7, 1, 'HEX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [8, 1, 'VERSION', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [9, 1, 'QUOTE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [10, 1, 'REPLACE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [11, 1, 'SUBSTR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [12, 1, 'GROUP_CONCAT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [13, 1, 'JULIANDAY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [14, 1, 'DATE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [15, 1, 'TIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [16, 1, 'DATETIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [17, 1, 'STRFTIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [18, 1, 'CURRENT_TIME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [19, 1, 'CURRENT_TIMESTAMP', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [20, 1, 'CURRENT_DATE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [21, 1, 'LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [22, 1, 'POSITION', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [23, 1, 'ROUND', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [24, 1, 'UPPER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [25, 1, 'LOWER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [26, 1, 'IFNULL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [27, 1, 'RANDOM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [28, 1, 'CEIL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [29, 1, 'CEILING', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [30, 1, 'CHARACTER_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [31, 1, 'CHAR_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [32, 1, 'FLOOR', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [33, 1, 'MOD', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [34, 1, 'OCTET_LENGTH', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [35, 1, 'ROW_COUNT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [36, 1, 'COUNT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [37, 1, 'LIKE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [38, 1, 'ABS', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [39, 1, 'EXP', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [40, 1, 'LN', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [41, 1, 'POWER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [42, 1, 'SQRT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [43, 1, 'SUM', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [44, 1, 'TOTAL', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [45, 1, 'AVG', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [46, 1, 'RANDOMBLOB', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [47, 1, 'NULLIF', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [48, 1, 'ZEROBLOB', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [49, 1, 'MIN', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [50, 1, 'MAX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [51, 1, 'COALESCE', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [52, 1, 'EVERY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [53, 1, 'EXISTS', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [54, 1, 'EXTRACT', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [55, 1, 'SOME', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', false, - false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [56, 1, 'GREATER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [57, 1, 'LESSER', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [58, 1, 'SOUNDEX', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [59, 1, 'LIKELIHOOD', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [60, 1, 'LIKELY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [61, 1, 'UNLIKELY', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', 'none', - false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [62, 1, '_sql_stat_get', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [63, 1, '_sql_stat_push', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [64, 1, '_sql_stat_init', 1, 'SQL_BUILTIN', '', 'function', [], 'any', 'none', - 'none', false, false, true, [], {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + 'none', false, false, true, [], {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] - [65, 1, 'LUA', 1, 'LUA', 'function(code) return assert(loadstring(code))() end', 'function', ['string'], 'any', 'none', 'none', false, false, true, ['LUA', 'SQL'], - {}, '', '2019-07-11 13:57:48', '2019-07-11 13:57:48'] + {}, '', '2019-07-15 17:54:58', '2019-07-15 17:54:58'] ... session = nil --- diff --git a/test/box/access_sysview.result b/test/box/access_sysview.result index d65aa37ae..a82127ebb 100644 --- a/test/box/access_sysview.result +++ b/test/box/access_sysview.result @@ -246,11 +246,11 @@ box.session.su('guest') ... #box.space._vspace:select{} --- -- 24 +- 25 ... #box.space._vindex:select{} --- -- 51 +- 53 ... #box.space._vuser:select{} --- @@ -282,7 +282,7 @@ box.session.su('guest') ... #box.space._vindex:select{} --- -- 51 +- 53 ... #box.space._vuser:select{} --- diff --git a/test/box/alter.result b/test/box/alter.result index a6db011ff..91a239bbc 100644 --- a/test/box/alter.result +++ b/test/box/alter.result @@ -92,7 +92,7 @@ space = box.space[t[1]] ... space.id --- -- 365 +- 373 ... space.field_count --- @@ -137,7 +137,7 @@ space_deleted ... space:replace{0} --- -- error: Space '365' does not exist +- error: Space '373' does not exist ... _index:insert{_space.id, 0, 'primary', 'tree', {unique=true}, {{0, 'unsigned'}}} --- @@ -218,6 +218,9 @@ _index:select{} - [356, 0, 'primary', 'tree', {'unique': true}, [[0, 'string'], [1, 'unsigned']]] - [356, 1, 'child_id', 'tree', {'unique': false}, [[1, 'unsigned']]] - [364, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'string']]] + - [372, 0, 'primary', 'tree', {'unique': true}, [[0, 'unsigned'], [1, 'unsigned'], + [2, 'unsigned']]] + - [372, 1, 'fid', 'tree', {'unique': false}, [[2, 'unsigned']]] ... -- modify indexes of a system space _index:delete{_index.id, 0} diff --git a/test/box/bitset.result b/test/box/bitset.result index 78f74ec37..fab0828a4 100644 --- a/test/box/bitset.result +++ b/test/box/bitset.result @@ -1996,3 +1996,27 @@ _ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}} s:drop() --- ... +-- Bitset index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': BITSET index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/box/bitset.test.lua b/test/box/bitset.test.lua index eb013a1c0..1fea31563 100644 --- a/test/box/bitset.test.lua +++ b/test/box/bitset.test.lua @@ -153,3 +153,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('bitset', {type = 'bitset', parts = {{'[2][*]', 'unsigned'}}}) s:drop() + +-- Bitset index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/box/hash.result b/test/box/hash.result index 9f08c49b8..a42f81c69 100644 --- a/test/box/hash.result +++ b/test/box/hash.result @@ -847,3 +847,27 @@ _ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}}) s:drop() --- ... +-- Hash index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'hash', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': HASH index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/box/hash.test.lua b/test/box/hash.test.lua index 9801873c4..e5ab18f63 100644 --- a/test/box/hash.test.lua +++ b/test/box/hash.test.lua @@ -353,3 +353,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('hash', {type = 'hash', parts = {{'[2][*]', 'unsigned'}}}) s:drop() + +-- Hash index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'hash', func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/box/misc.result b/test/box/misc.result index 791730935..31e4e86f2 100644 --- a/test/box/misc.result +++ b/test/box/misc.result @@ -526,6 +526,7 @@ t; 195: box.error.CREATE_CK_CONSTRAINT 196: box.error.CK_CONSTRAINT_FAILED 197: box.error.SQL_COLUMN_COUNT + 198: box.error.FUNCTIONAL_INDEX_FUNC_ERROR ... test_run:cmd("setopt delimiter ''"); --- diff --git a/test/box/rtree_misc.result b/test/box/rtree_misc.result index 6e48bacc7..c256d52e7 100644 --- a/test/box/rtree_misc.result +++ b/test/box/rtree_misc.result @@ -682,3 +682,27 @@ _ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}}) s:drop() --- ... +-- Rtree index cannot be functional. +s = box.schema.space.create('withdata') +--- +... +lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]] +--- +... +box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {type = 'rtree', func = box.func.fextract.id, parts = {{1, 'array'}}}) +--- +- error: 'Can''t create or modify index ''idx'' in space ''withdata'': RTREE index + cannot be functional' +... +s:drop() +--- +... +box.schema.func.drop('fextract') +--- +... diff --git a/test/box/rtree_misc.test.lua b/test/box/rtree_misc.test.lua index 000a928e8..2d1d490f3 100644 --- a/test/box/rtree_misc.test.lua +++ b/test/box/rtree_misc.test.lua @@ -243,3 +243,12 @@ s = box.schema.space.create('test') _ = s:create_index('primary') _ = s:create_index('rtree', {type = 'rtree', parts = {{'[2][*]', 'array'}}}) s:drop() + +-- Rtree index cannot be functional. +s = box.schema.space.create('withdata') +lua_code = [[function(tuple) return {tuple[1] + tuple[2], tuple[1] - tuple[2]} end]] +box.schema.func.create('fextract', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {type = 'rtree', func = box.func.fextract.id, parts = {{1, 'array'}}}) +s:drop() +box.schema.func.drop('fextract') diff --git a/test/engine/engine.cfg b/test/engine/engine.cfg index 9f07629b4..826e8eb1f 100644 --- a/test/engine/engine.cfg +++ b/test/engine/engine.cfg @@ -2,7 +2,10 @@ "*": { "memtx": {"engine": "memtx"}, "vinyl": {"engine": "vinyl"} - } + }, + "functional.test.lua": { + "memtx": {"engine": "memtx"} + } } diff --git a/test/engine/functional.result b/test/engine/functional.result new file mode 100644 index 000000000..6af054856 --- /dev/null +++ b/test/engine/functional.result @@ -0,0 +1,690 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +-- +-- gh-1260: Funclional indexes. +-- +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +lua_code2 = [[function(tuple) return {tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('s_nonpersistent') + | --- + | ... +box.schema.func.create('s_ivaliddef1', {body = lua_code, is_deterministic = false, is_sandboxed = true}) + | --- + | ... +box.schema.func.create('s_ivaliddef2', {body = lua_code, is_deterministic = true, is_sandboxed = false}) + | --- + | ... + +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +box.schema.func.create('ss', {body = lua_code2, is_deterministic = true, is_sandboxed = true}) + | --- + | ... + +-- Functional index can't be primary. +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Can''t create or modify index ''idx'' in space ''withdata'': primary key + | cannot be functional' + | ... +pk = s:create_index('pk') + | --- + | ... +-- Invalid fid. +_ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) + | --- + | - error: Function '6666' does not exist + | ... +-- Can't use non-persistent function in functional index. +_ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-deterministic function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-sandboxed function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional + | index constraints' + | ... +-- Can't use non-sequential parts in returned key definition. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... +-- Can't use parts started not by 1 field. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... +-- Can't use JSON paths in returned key definiton. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{"[1]data", 'unsigned'}}}) + | --- + | - error: 'Wrong index options (field 0): invalid functional key definition' + | ... + +-- Can't drop a function referenced by functional index. +idx = s:create_index('idx', {unique = true, func = box.func.s.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +box.schema.func.drop('s') + | --- + | - error: 'Can''t drop function 1: function has references' + | ... +box.snapshot() + | --- + | - ok + | ... +test_run:cmd("restart server default") + | +box.schema.func.drop('s') + | --- + | - error: 'Can''t drop function 1: function has references' + | ... +s = box.space.withdata + | --- + | ... +idx = s.index.idx + | --- + | ... +idx:drop() + | --- + | ... +box.schema.func.drop('s') + | --- + | ... + +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +-- Invalid functional index extractor routine return: the extractor must return keys. +lua_code = [[function(tuple) return "hello" end]] + | --- + | ... +box.schema.func.create('invalidreturn0', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn0.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn0'' error: returned key type is + | invalid' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional index extractor routine return: a stirng instead of unsigned +lua_code = [[function(tuple) return {"hello"} end]] + | --- + | ... +box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn1'' error: Supplied key type of + | part 0 does not match index part type: expected unsigned' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] + | --- + | ... +box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''invalidreturn3'' error: Supplied key type of + | part 0 does not match index part type: expected unsigned' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional extractor: runtime extractor error +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +lua_code = [[function(tuple) + local json = require('json') + return json.encode(tuple) + end]] +test_run:cmd("setopt delimiter ''"); + | --- + | ... +box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.runtimeerror.id, parts = {{1, 'string'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''runtimeerror'' error: [string "return function(tuple) local + | ..."]:1: attempt to call global ''require'' (a nil value)' + | ... +idx:drop() + | --- + | ... + +-- Remove old persistent functions +for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end + | --- + | ... +s:drop() + | --- + | ... + +-- Functional test cases. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +idx = s:create_index('idx', {unique = true, func = 'extr', parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({2, 1}) + | --- + | - error: Duplicate key exists in unique index 'idx' in space 'withdata' + | ... +idx:get(3) + | --- + | - [1, 2] + | ... +idx:delete(3) + | --- + | - [1, 2] + | ... +s:select() + | --- + | - [] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +idx:get(3) + | --- + | - [2, 1] + | ... +s:drop() + | --- + | ... +box.schema.func.drop('extr') + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... + +-- Multikey functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({3, 5}) + | --- + | - error: Duplicate key exists in unique index 'idx' in space 'withdata' + | ... +s:insert({5, 3}) + | --- + | - [5, 3] + | ... +idx:select() + | --- + | - - [1, 2] + | - [1, 2] + | - [5, 3] + | - [5, 3] + | ... +idx:get(8) + | --- + | - [5, 3] + | ... +idx:get(3) + | --- + | - [1, 2] + | ... +idx:get(1) + | --- + | - [1, 2] + | ... +idx:get(5) + | --- + | - [5, 3] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey multipart functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +s:insert({3, 3}) + | --- + | - [3, 3] + | ... +idx:select({600}, {iterator = "GE"}) + | --- + | - - [1, 2] + | - [2, 1] + | - [3, 3] + | ... +idx:get({603, 603}) + | --- + | - [3, 3] + | ... +idx:select({503}, {iterator = "LE"}) + | --- + | - - [3, 3] + | - [2, 1] + | - [1, 2] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey non-unique functional index. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1, 2}) + | --- + | - [1, 2] + | ... +s:insert({2, 1}) + | --- + | - [2, 1] + | ... +idx:select({501}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +s:replace({1, 3}) + | --- + | - [1, 3] + | ... +idx:select({501}) + | --- + | - - [1, 3] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [2, 1] + | ... +idx:select({503}) + | --- + | - - [1, 3] + | ... +box.snapshot() + | --- + | - ok + | ... +test_run:cmd("restart server default") + | +s = box.space.withdata + | --- + | ... +idx = s.index.idx + | --- + | ... +idx:select({501}) + | --- + | - - [1, 3] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [2, 1] + | ... +idx:select({503}) + | --- + | - - [1, 3] + | ... +s:replace({1, 2}) + | --- + | - [1, 2] + | ... +idx:select({501}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({502}) + | --- + | - - [1, 2] + | - [2, 1] + | ... +idx:select({503}) + | --- + | - [] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multikey UTF-8 address extractor +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('name', {parts = {1, 'string'}}) + | --- + | ... +s:insert({"James", "SIS Building Lambeth London UK"}) + | --- + | - ['James', 'SIS Building Lambeth London UK'] + | ... +s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) + | --- + | - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] + | ... +-- Create functional index on space with data +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +lua_code = [[function(tuple) + local address = string.split(tuple[2]) + local ret = {} + for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end + return unpack(ret) + end]] +test_run:cmd("setopt delimiter ''"); + | --- + | ... +box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) + | --- + | ... +idx:select('uk') + | --- + | - - ['James', 'SIS Building Lambeth London UK'] + | - ['Sherlock', '221B Baker St Marylebone London NW1 6XE UK'] + | ... +idx:select('Sis') + | --- + | - - ['James', 'SIS Building Lambeth London UK'] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('addr_extractor') + | --- + | ... + +-- Partial index with functional index extractor +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({1}) + | --- + | - [1] + | ... +s:insert({2}) + | --- + | - [2] + | ... +s:insert({3}) + | --- + | - [3] + | ... +s:insert({4}) + | --- + | - [4] + | ... +idx:select() + | --- + | - - [1] + | - [3] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Return nil from functional index extractor. +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +lua_code = [[function(tuple) return {nil} end]] + | --- + | ... +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer', is_nullable = true}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Functional index function ''extr'' error: Invalid key part count in an exact + | match (expected 1, got 0)' + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('extr') + | --- + | ... + +-- Multiple functional indexes +s = box.schema.space.create('withdata', {engine = engine}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] + | --- + | ... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +lua_code = [[function(tuple) return {tuple[1] - tuple[2]} end]] + | --- + | ... +box.schema.func.create('sub', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +pk = s:create_index('pk') + | --- + | ... +idx1 = s:create_index('s_idx', {unique = true, func = box.func.s.id, parts = {{1, 'integer'}}}) + | --- + | ... +idx2 = s:create_index('sub_idx', {unique = true, func = box.func.sub.id, parts = {{1, 'integer'}}}) + | --- + | ... +s:insert({4, 1}) + | --- + | - [4, 1] + | ... +idx1:get(5) + | --- + | - [4, 1] + | ... +idx2:get(3) + | --- + | - [4, 1] + | ... +idx1:drop() + | --- + | ... +idx2:get(3) + | --- + | - [4, 1] + | ... +s:drop() + | --- + | ... +collectgarbage() + | --- + | - 0 + | ... +box.schema.func.drop('s') + | --- + | ... +box.schema.func.drop('sub') + | --- + | ... diff --git a/test/engine/functional.test.lua b/test/engine/functional.test.lua new file mode 100644 index 000000000..92a74aab7 --- /dev/null +++ b/test/engine/functional.test.lua @@ -0,0 +1,233 @@ +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +-- +-- gh-1260: Funclional indexes. +-- +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +lua_code2 = [[function(tuple) return {tuple[1] + tuple[2], 2 * tuple[1] + tuple[2]} end]] +box.schema.func.create('s_nonpersistent') +box.schema.func.create('s_ivaliddef1', {body = lua_code, is_deterministic = false, is_sandboxed = true}) +box.schema.func.create('s_ivaliddef2', {body = lua_code, is_deterministic = true, is_sandboxed = false}) + +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +box.schema.func.create('ss', {body = lua_code2, is_deterministic = true, is_sandboxed = true}) + +-- Functional index can't be primary. +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +pk = s:create_index('pk') +-- Invalid fid. +_ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) +-- Can't use non-persistent function in functional index. +_ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-deterministic function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-sandboxed function in functional index. +_ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) +-- Can't use non-sequential parts in returned key definition. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) +-- Can't use parts started not by 1 field. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{2, 'unsigned'}, {3, 'unsigned'}}}) +-- Can't use JSON paths in returned key definiton. +_ = s:create_index('idx', {func = box.func.ss.id, parts = {{"[1]data", 'unsigned'}}}) + +-- Can't drop a function referenced by functional index. +idx = s:create_index('idx', {unique = true, func = box.func.s.id, parts = {{1, 'unsigned'}}}) +box.schema.func.drop('s') +box.snapshot() +test_run:cmd("restart server default") +box.schema.func.drop('s') +s = box.space.withdata +idx = s.index.idx +idx:drop() +box.schema.func.drop('s') + +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +-- Invalid functional index extractor routine return: the extractor must return keys. +lua_code = [[function(tuple) return "hello" end]] +box.schema.func.create('invalidreturn0', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn0.id, parts = {{1, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: a stirng instead of unsigned +lua_code = [[function(tuple) return {"hello"} end]] +box.schema.func.create('invalidreturn1', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn1.id, parts = {{1, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] +box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional extractor: runtime extractor error +test_run:cmd("setopt delimiter ';'") +lua_code = [[function(tuple) + local json = require('json') + return json.encode(tuple) + end]] +test_run:cmd("setopt delimiter ''"); +box.schema.func.create('runtimeerror', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.runtimeerror.id, parts = {{1, 'string'}}}) +s:insert({1}) +idx:drop() + +-- Remove old persistent functions +for _, v in pairs(box.func) do if v.is_persistent then box.schema.func.drop(v.name) end end +s:drop() + +-- Functional test cases. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +s:insert({1, 2}) +idx = s:create_index('idx', {unique = true, func = 'extr', parts = {{1, 'integer'}}}) +s:insert({2, 1}) +idx:get(3) +idx:delete(3) +s:select() +s:insert({2, 1}) +idx:get(3) +s:drop() +box.schema.func.drop('extr') +collectgarbage() + +-- Multikey functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1, 2}) +s:insert({3, 5}) +s:insert({5, 3}) +idx:select() +idx:get(8) +idx:get(3) +idx:get(1) +idx:get(5) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey multipart functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) +s:insert({1, 2}) +s:insert({2, 1}) +s:insert({3, 3}) +idx:select({600}, {iterator = "GE"}) +idx:get({603, 603}) +idx:select({503}, {iterator = "LE"}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey non-unique functional index. +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1, 2}) +s:insert({2, 1}) +idx:select({501}) +idx:select({502}) +s:replace({1, 3}) +idx:select({501}) +idx:select({502}) +idx:select({503}) +box.snapshot() +test_run:cmd("restart server default") +s = box.space.withdata +idx = s.index.idx +idx:select({501}) +idx:select({502}) +idx:select({503}) +s:replace({1, 2}) +idx:select({501}) +idx:select({502}) +idx:select({503}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multikey UTF-8 address extractor +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('name', {parts = {1, 'string'}}) +s:insert({"James", "SIS Building Lambeth London UK"}) +s:insert({"Sherlock", "221B Baker St Marylebone London NW1 6XE UK"}) +-- Create functional index on space with data +test_run:cmd("setopt delimiter ';'") +lua_code = [[function(tuple) + local address = string.split(tuple[2]) + local ret = {} + for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end + return unpack(ret) + end]] +test_run:cmd("setopt delimiter ''"); +box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) +idx:select('uk') +idx:select('Sis') +s:drop() +collectgarbage() +box.schema.func.drop('addr_extractor') + +-- Partial index with functional index extractor +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('pk') +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +s:insert({1}) +s:insert({2}) +s:insert({3}) +s:insert({4}) +idx:select() +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Return nil from functional index extractor. +s = box.schema.space.create('withdata', {engine = engine}) +pk = s:create_index('pk') +lua_code = [[function(tuple) return {nil} end]] +box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer', is_nullable = true}}}) +s:insert({1}) +s:drop() +collectgarbage() +box.schema.func.drop('extr') + +-- Multiple functional indexes +s = box.schema.space.create('withdata', {engine = engine}) +lua_code = [[function(tuple) return {tuple[1] + tuple[2]} end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +lua_code = [[function(tuple) return {tuple[1] - tuple[2]} end]] +box.schema.func.create('sub', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +pk = s:create_index('pk') +idx1 = s:create_index('s_idx', {unique = true, func = box.func.s.id, parts = {{1, 'integer'}}}) +idx2 = s:create_index('sub_idx', {unique = true, func = box.func.sub.id, parts = {{1, 'integer'}}}) +s:insert({4, 1}) +idx1:get(5) +idx2:get(3) +idx1:drop() +idx2:get(3) +s:drop() +collectgarbage() +box.schema.func.drop('s') +box.schema.func.drop('sub') diff --git a/test/vinyl/misc.result b/test/vinyl/misc.result index b2aacdc55..508a64660 100644 --- a/test/vinyl/misc.result +++ b/test/vinyl/misc.result @@ -432,3 +432,26 @@ stat.bytes_compressed < stat.bytes / 10 s:drop() --- ... +-- Vinyl doesn't support functional indexes. +s = box.schema.space.create('withdata', {engine = 'vinyl'}) +--- +... +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +--- +... +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +--- +... +_ = s:create_index('pk') +--- +... +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +--- +- error: Vinyl does not support functional indexes +... +s:drop() +--- +... +box.schema.func.drop('s') +--- +... diff --git a/test/vinyl/misc.test.lua b/test/vinyl/misc.test.lua index f8da578d0..bd311da77 100644 --- a/test/vinyl/misc.test.lua +++ b/test/vinyl/misc.test.lua @@ -182,3 +182,12 @@ test_run:wait_cond(function() return i:stat().disk.compaction.count > 0 end) stat = i:stat().disk stat.bytes_compressed < stat.bytes / 10 s:drop() + +-- Vinyl doesn't support functional indexes. +s = box.schema.space.create('withdata', {engine = 'vinyl'}) +lua_code = [[function(tuple) return tuple[1] + tuple[2] end]] +box.schema.func.create('s', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +_ = s:create_index('pk') +_ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) +s:drop() +box.schema.func.drop('s') diff --git a/test/wal_off/alter.result b/test/wal_off/alter.result index bce15711d..62cb11db7 100644 --- a/test/wal_off/alter.result +++ b/test/wal_off/alter.result @@ -28,7 +28,7 @@ end; ... #spaces; --- -- 65503 +- 65502 ... -- cleanup for k, v in pairs(spaces) do -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov 2019-07-24 12:24 ` [tarantool-patches] " Kirill Shcherbatov @ 2019-07-24 19:41 ` Konstantin Osipov 2019-07-24 20:04 ` Konstantin Osipov ` (5 subsequent siblings) 7 siblings, 0 replies; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 19:41 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: I think generally you're on track, and patch needs a careful review now. Please solicit a review from Vova. For the tuple_compare part, please ask Vlad to take a look - he did a lot in this part of the code. The only issue that surprised me is that you allocate a separate tuple chunk for each key. I somehow thought that you're going to use the same chunk for all bps tree entries, which I now realize is impossible, since you'll need to store a chunk offset somewhere. The problem is that slab alloc doesn't have a slab for tiny chunks now - the smallest chunk is 24 bytes. This is obviously an overkill for this scenario. Please fix the slab arena accordingly. There are lots of style comments, I'll place them on the branch. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov 2019-07-24 12:24 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 19:41 ` Konstantin Osipov @ 2019-07-24 20:04 ` Konstantin Osipov 2019-07-24 20:22 ` Konstantin Osipov ` (4 subsequent siblings) 7 siblings, 0 replies; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 20:04 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: there is another patch on the branch , build_array_deduplicate fix. Could you add a test case for this bug? the fix itself is lgtm. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov ` (2 preceding siblings ...) 2019-07-24 20:04 ` Konstantin Osipov @ 2019-07-24 20:22 ` Konstantin Osipov 2019-07-25 11:20 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 20:44 ` Konstantin Osipov ` (3 subsequent siblings) 7 siblings, 1 reply; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 20:22 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > lua_code = [[function(tuple) > local address = string.split(tuple[2]) > local ret = {} > for _, v in pairs(address) do > table.insert(ret, {utf8.upper(v)}) > end > return unpack(ret) > end]] I think the convention should be that a multikey function must always return a table (a msgpack array for a C function). It's OK to wrap every key into a table as long as it simplifies the code (key part count is encoded automatically by struct port), but otherwise I would stick to scalars. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 20:22 ` Konstantin Osipov @ 2019-07-25 11:20 ` Kirill Shcherbatov 0 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-25 11:20 UTC (permalink / raw) To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev On 24.07.2019 23:22, Konstantin Osipov wrote: > * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: >> lua_code = [[function(tuple) >> local address = string.split(tuple[2]) >> local ret = {} >> for _, v in pairs(address) do >> table.insert(ret, {utf8.upper(v)}) >> end >> return unpack(ret) >> end]] > > I think the convention should be that a multikey function must always > return a table (a msgpack array for a C function). > > It's OK to wrap every key into a table as long as it simplifies > the code (key part count is encoded automatically by struct port), but otherwise > I would stick to scalars. Ok, now a user must implicitly specify is_multikey and return a table of keys; The corresponding change diff is below: ====================================================== --- src/box/index_def.h | 4 +++ src/box/key_def.h | 2 +- src/box/key_list.h | 2 +- src/box/index_def.c | 2 ++ src/box/key_def.c | 5 +-- src/box/key_list.c | 26 ++++++++++++-- src/box/lua/key_def.c | 2 +- src/box/memtx_engine.c | 2 ++ src/box/memtx_tree.c | 3 ++ src/box/sql.c | 3 +- src/box/sql/build.c | 2 +- src/box/sql/select.c | 3 +- src/box/sql/where.c | 2 +- src/box/vinyl.c | 4 ++- test/unit/luaT_tuple_new.c | 2 +- test/unit/merger.test.c | 6 ++-- src/box/alter.cc | 3 +- src/box/lua/space.cc | 18 +++++++++- src/box/schema.cc | 3 +- test/engine/functional.result | 62 ++++++++++++++++++++++++++------- test/engine/functional.test.lua | 38 +++++++++++++------- 21 files changed, 151 insertions(+), 43 deletions(-) diff --git a/src/box/index_def.h b/src/box/index_def.h index 71b6db32d..f7788c63e 100644 --- a/src/box/index_def.h +++ b/src/box/index_def.h @@ -165,6 +165,8 @@ struct index_opts { struct index_stat *stat; /** Identifier of the functional index function. */ uint32_t func_id; + /** Whether functional index extractor is multikey. */ + bool is_multikey; }; extern const struct index_opts index_opts_default; @@ -211,6 +213,8 @@ index_opts_cmp(const struct index_opts *o1, const struct index_opts *o2) return o1->bloom_fpr < o2->bloom_fpr ? -1 : 1; if (o1->func_id != o2->func_id) return o1->func_id - o2->func_id; + if (o1->is_multikey != o2->is_multikey) + return o1->is_multikey - o2->is_multikey; return 0; } diff --git a/src/box/key_def.h b/src/box/key_def.h index 5c898fc87..4357408b7 100644 --- a/src/box/key_def.h +++ b/src/box/key_def.h @@ -340,7 +340,7 @@ key_def_sizeof(uint32_t part_count, uint32_t path_pool_size) */ struct key_def * key_def_new(const struct key_part_def *parts, uint32_t part_count, - bool is_functional); + bool is_functional, bool is_multikey); /** * Dump part definitions of the given key def. diff --git a/src/box/key_list.h b/src/box/key_list.h index 9107dfeeb..f7ed344c8 100644 --- a/src/box/key_list.h +++ b/src/box/key_list.h @@ -52,7 +52,7 @@ struct tuple; * Uses fiber region to allocate memory. */ const char * -key_list_create(struct tuple *tuple, struct func *func, +key_list_create(struct tuple *tuple, struct func *func, bool is_multikey, const char **data_end, uint32_t *key_count); /** diff --git a/src/box/index_def.c b/src/box/index_def.c index 7d06c65f3..420ca14f4 100644 --- a/src/box/index_def.c +++ b/src/box/index_def.c @@ -51,6 +51,7 @@ const struct index_opts index_opts_default = { /* .lsn = */ 0, /* .stat = */ NULL, /* .func = */ 0, + /* .is_multikey = */ false, }; const struct opt_def index_opts_reg[] = { @@ -65,6 +66,7 @@ const struct opt_def index_opts_reg[] = { OPT_DEF("bloom_fpr", OPT_FLOAT, struct index_opts, bloom_fpr), OPT_DEF("lsn", OPT_INT64, struct index_opts, lsn), OPT_DEF("func", OPT_UINT32, struct index_opts, func_id), + OPT_DEF("is_multikey", OPT_BOOL, struct index_opts, is_multikey), OPT_DEF_LEGACY("sql"), OPT_END, }; diff --git a/src/box/key_def.c b/src/box/key_def.c index ffc34db59..ea46a1583 100644 --- a/src/box/key_def.c +++ b/src/box/key_def.c @@ -242,7 +242,7 @@ key_def_set_part(struct key_def *def, uint32_t part_no, uint32_t fieldno, struct key_def * key_def_new(const struct key_part_def *parts, uint32_t part_count, - bool is_functional) + bool is_functional, bool is_multikey) { size_t sz = 0; for (uint32_t i = 0; i < part_count; i++) @@ -279,6 +279,7 @@ key_def_new(const struct key_part_def *parts, uint32_t part_count, goto error; } if (is_functional) { + def->is_multikey = is_multikey; def->part_count_for_func_index = part_count; if (!key_def_is_sequential(def) || parts->fieldno != 0 || def->has_json_paths) { @@ -842,7 +843,7 @@ key_def_find_pk_in_cmp_def(const struct key_def *cmp_def, } /* Finally, allocate the new key definition. */ - extracted_def = key_def_new(parts, pk_def->part_count, false); + extracted_def = key_def_new(parts, pk_def->part_count, false, false); out: region_truncate(region, region_svp); return extracted_def; diff --git a/src/box/key_list.c b/src/box/key_list.c index 5b5552116..6b800d9c7 100644 --- a/src/box/key_list.c +++ b/src/box/key_list.c @@ -33,6 +33,7 @@ #include "errcode.h" #include "diag.h" #include "func.h" +#include "fiber.h" #include "key_def.h" #include "port.h" #include "tt_static.h" @@ -40,9 +41,12 @@ #include "tuple_compare.h" const char * -key_list_create(struct tuple *tuple, struct func *func, +key_list_create(struct tuple *tuple, struct func *func, bool is_multikey, const char **data_end, uint32_t *key_count) { + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + struct port out_port, in_port; port_tuple_create(&in_port); port_tuple_add(&in_port, tuple); @@ -52,15 +56,31 @@ key_list_create(struct tuple *tuple, struct func *func, goto error; uint32_t key_data_sz; const char *key_data = port_get_msgpack(&out_port, &key_data_sz); + *data_end = key_data + key_data_sz; port_destroy(&out_port); if (key_data == NULL) goto error; assert(mp_typeof(*key_data) == MP_ARRAY); - *data_end = key_data + key_data_sz; - *key_count = mp_decode_array(&key_data); + if (mp_decode_array(&key_data) != 1) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + func->def->name, "to many values were returned"); + goto error; + } + if (is_multikey) { + if (mp_typeof(*key_data) != MP_ARRAY) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + func->def->name, + "multikey function mustn't return scalar"); + goto error; + } + *key_count = mp_decode_array(&key_data); + } else { + *key_count = 1; + } return key_data; error: + region_truncate(region, region_svp); diag_set(ClientError, ER_FUNC_INDEX_FUNC, func->def->name, diag_last_error(diag_get())->errmsg); return NULL; diff --git a/src/box/lua/key_def.c b/src/box/lua/key_def.c index 3a3a5ec0c..e572fec1b 100644 --- a/src/box/lua/key_def.c +++ b/src/box/lua/key_def.c @@ -445,7 +445,7 @@ lbox_key_def_new(struct lua_State *L) lua_pop(L, 1); } - struct key_def *key_def = key_def_new(parts, part_count, false); + struct key_def *key_def = key_def_new(parts, part_count, false, false); region_truncate(region, region_svp); if (key_def == NULL) return luaT_error(L); diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index 9f8d5a6a4..8589d2f1e 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -1250,6 +1250,8 @@ memtx_index_def_change_requires_rebuild(struct index *index, return true; if (old_def->opts.func_id != new_def->opts.func_id) return true; + if (old_def->opts.is_multikey != new_def->opts.is_multikey) + return true; const struct key_def *old_cmp_def, *new_cmp_def; if (index_depends_on_pk(index)) { diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index f24fec15f..b11f09dd5 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -863,6 +863,7 @@ memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple, struct key_list_iterator it; if (new_tuple != NULL) { key_list = key_list_create(new_tuple, cmp_def->func_index_func, + cmp_def->is_multikey, &key_list_end, &key_cnt); if (key_list == NULL) goto end; @@ -926,6 +927,7 @@ memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple, } if (old_tuple != NULL) { key_list = key_list_create(old_tuple, cmp_def->func_index_func, + cmp_def->is_multikey, &key_list_end, &key_cnt); if (key_list == NULL) goto end; @@ -1101,6 +1103,7 @@ memtx_tree_index_build_next_functional(struct index *base, struct tuple *tuple) const char *key_list; const char *key_list_end; key_list = key_list_create(tuple, cmp_def->func_index_func, + cmp_def->is_multikey, &key_list_end, &key_cnt); if (key_list == NULL) return -1; diff --git a/src/box/sql.c b/src/box/sql.c index 0ab3a506f..ba5bcbf10 100644 --- a/src/box/sql.c +++ b/src/box/sql.c @@ -347,7 +347,8 @@ sql_ephemeral_space_create(uint32_t field_count, struct sql_key_info *key_info) } } struct key_def *ephemer_key_def = key_def_new(ephemer_key_parts, - field_count, false); + field_count, false, + false); if (ephemer_key_def == NULL) return NULL; diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 0a6759e41..b965c7c5c 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -2338,7 +2338,7 @@ index_fill_def(struct Parse *parse, struct index *index, part->coll_id = coll_id; part->path = NULL; } - key_def = key_def_new(key_parts, expr_list->nExpr, false); + key_def = key_def_new(key_parts, expr_list->nExpr, false, false); if (key_def == NULL) goto tnt_error; /* diff --git a/src/box/sql/select.c b/src/box/sql/select.c index c312f61f1..bb7fa783e 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1438,7 +1438,8 @@ sql_key_info_to_key_def(struct sql_key_info *key_info) { if (key_info->key_def == NULL) { key_info->key_def = key_def_new(key_info->parts, - key_info->part_count, false); + key_info->part_count, + false, false); } return key_info->key_def; } diff --git a/src/box/sql/where.c b/src/box/sql/where.c index ed507bf4d..b85243a88 100644 --- a/src/box/sql/where.c +++ b/src/box/sql/where.c @@ -2775,7 +2775,7 @@ whereLoopAddBtree(WhereLoopBuilder * pBuilder, /* WHERE clause information */ part.coll_id = COLL_NONE; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1, false); + struct key_def *key_def = key_def_new(&part, 1, false, false); if (key_def == NULL) { tnt_error: pWInfo->pParse->is_aborted = true; diff --git a/src/box/vinyl.c b/src/box/vinyl.c index c938350d9..dce32b574 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -993,6 +993,8 @@ vinyl_index_def_change_requires_rebuild(struct index *index, return true; if (old_def->opts.func_id != new_def->opts.func_id) return true; + if (old_def->opts.is_multikey != new_def->opts.is_multikey) + return true; assert(index_depends_on_pk(index)); const struct key_def *old_cmp_def = old_def->cmp_def; @@ -3172,7 +3174,7 @@ vy_send_lsm(struct vy_join_ctx *ctx, struct vy_lsm_recovery_info *lsm_info) /* Create key definition and tuple format. */ ctx->key_def = key_def_new(lsm_info->key_parts, - lsm_info->key_part_count, false); + lsm_info->key_part_count, false, false); if (ctx->key_def == NULL) goto out; ctx->format = vy_stmt_format_new(&ctx->env->stmt_env, &ctx->key_def, 1, diff --git a/test/unit/luaT_tuple_new.c b/test/unit/luaT_tuple_new.c index 609d64e45..d5c59c975 100644 --- a/test/unit/luaT_tuple_new.c +++ b/test/unit/luaT_tuple_new.c @@ -124,7 +124,7 @@ test_basic(struct lua_State *L) part.nullable_action = ON_CONFLICT_ACTION_DEFAULT; part.sort_order = SORT_ORDER_ASC; part.path = NULL; - struct key_def *key_def = key_def_new(&part, 1, false); + struct key_def *key_def = key_def_new(&part, 1, false, false); box_tuple_format_t *another_format = box_tuple_format_new(&key_def, 1); key_def_delete(key_def); diff --git a/test/unit/merger.test.c b/test/unit/merger.test.c index 345a2364e..8a4379a10 100644 --- a/test/unit/merger.test.c +++ b/test/unit/merger.test.c @@ -214,7 +214,8 @@ test_merger(struct tuple_format *format) merge_source_array_new(true), }; - struct key_def *key_def = key_def_new(&key_part_unsigned, 1, false); + struct key_def *key_def = key_def_new(&key_part_unsigned, 1, + false, false); struct merge_source *merger = merger_new(key_def, sources, source_count, false); key_def_delete(key_def); @@ -252,7 +253,8 @@ test_basic() plan(4); header(); - struct key_def *key_def = key_def_new(&key_part_integer, 1, false); + struct key_def *key_def = key_def_new(&key_part_integer, 1, + false, false); struct tuple_format *format = box_tuple_format_new(&key_def, 1); assert(format != NULL); diff --git a/src/box/alter.cc b/src/box/alter.cc index 3363e2b85..0b23dd6bb 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -302,7 +302,8 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) space->def->fields, space->def->field_count, &fiber()->gc) != 0) diag_raise(); - key_def = key_def_new(part_def, part_count, opts.func_id > 0); + key_def = key_def_new(part_def, part_count, opts.func_id > 0, + opts.is_multikey); if (key_def == NULL) diag_raise(); struct index_def *index_def = diff --git a/src/box/lua/space.cc b/src/box/lua/space.cc index 72c6bcada..6a778b2f2 100644 --- a/src/box/lua/space.cc +++ b/src/box/lua/space.cc @@ -42,6 +42,8 @@ extern "C" { #include <lualib.h> } /* extern "C" */ +#include "box/func.h" +#include "box/func_def.h" #include "box/space.h" #include "box/schema.h" #include "box/user_def.h" @@ -336,8 +338,22 @@ lbox_fillspace(struct lua_State *L, struct space *space, int i) } if (index_opts->func_id > 0) { + lua_pushstring(L, "func"); + lua_newtable(L); + lua_pushnumber(L, index_opts->func_id); - lua_setfield(L, -2, "func_id"); + lua_setfield(L, -2, "fid"); + + lua_pushboolean(L, index_opts->is_multikey); + lua_setfield(L, -2, "is_multikey"); + + struct func *func = func_by_id(index_opts->func_id); + if (func != NULL) { + lua_pushstring(L, func->def->name); + lua_setfield(L, -2, "name"); + } + + lua_settable(L, -3); } lua_pushstring(L, index_type_strs[index_def->type]); diff --git a/src/box/schema.cc b/src/box/schema.cc index 5d4a3ff00..d5ccdd1b6 100644 --- a/src/box/schema.cc +++ b/src/box/schema.cc @@ -266,7 +266,8 @@ sc_space_new(uint32_t id, const char *name, uint32_t key_part_count, struct trigger *replace_trigger) { - struct key_def *key_def = key_def_new(key_parts, key_part_count, false); + struct key_def *key_def = key_def_new(key_parts, key_part_count, + false, false); if (key_def == NULL) diag_raise(); auto key_def_guard = diff --git a/test/engine/functional.result b/test/engine/functional.result index 8a0eba614..ca8d0e366 100644 --- a/test/engine/functional.result +++ b/test/engine/functional.result @@ -159,14 +159,33 @@ idx:drop() | --- | ... --- Invalid functional index extractor routine return: the second returned key invalid. +-- Invalid functional index extractor routine return: invalid return format for multikey index. lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] | --- | ... +box.schema.func.create('invalidreturn2', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn2.id, is_multikey = true, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Failed to build a key for functional index ''invalidreturn2'': Failed to + | build a key for functional index ''invalidreturn2'': to many values were returned' + | ... +idx:drop() + | --- + | ... + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]] + | --- + | ... box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) | --- | ... -idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, is_multikey = true, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) | --- | ... s:insert({1}) @@ -178,6 +197,25 @@ idx:drop() | --- | ... +-- Invalid functional index extractor routine return: multikey return in case of regular index. +lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]] + | --- + | ... +box.schema.func.create('invalidreturn4', {body = lua_code, is_deterministic = true, is_sandboxed = true}) + | --- + | ... +idx = s:create_index('idx', {func = box.func.invalidreturn4.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) + | --- + | ... +s:insert({1}) + | --- + | - error: 'Failed to build a key for functional index ''invalidreturn4'': Supplied + | key type of part 0 does not match index part type: expected unsigned' + | ... +idx:drop() + | --- + | ... + -- Invalid functional extractor: runtime extractor error test_run:cmd("setopt delimiter ';'") | --- @@ -273,7 +311,7 @@ collectgarbage() s = box.schema.space.create('withdata', {engine = engine}) | --- | ... -lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] +lua_code = [[function(tuple) return {{tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]}} end]] | --- | ... box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) @@ -282,7 +320,7 @@ box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_san pk = s:create_index('pk') | --- | ... -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) | --- | ... s:insert({1, 2}) @@ -335,7 +373,7 @@ box.schema.func.drop('extr') s = box.schema.space.create('withdata', {engine = engine}) | --- | ... -lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] +lua_code = [[function(tuple) return {{600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]}} end]] | --- | ... box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) @@ -344,7 +382,7 @@ box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_san pk = s:create_index('pk') | --- | ... -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}, {2, 'integer'}}}) | --- | ... s:insert({1, 2}) @@ -390,7 +428,7 @@ box.schema.func.drop('extr') s = box.schema.space.create('withdata', {engine = engine}) | --- | ... -lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] +lua_code = [[function(tuple) return {{500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]}} end]] | --- | ... box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) @@ -399,7 +437,7 @@ box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_san pk = s:create_index('pk') | --- | ... -idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) | --- | ... s:insert({1, 2}) @@ -521,7 +559,7 @@ lua_code = [[function(tuple) local address = string.split(tuple[2]) local ret = {} for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end - return unpack(ret) + return ret end]] test_run:cmd("setopt delimiter ''"); | --- @@ -529,7 +567,7 @@ test_run:cmd("setopt delimiter ''"); box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) | --- | ... -idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, is_multikey = true, parts = {{1, 'string', collation = 'unicode_ci'}}}) | --- | ... idx:select('uk') @@ -559,13 +597,13 @@ s = box.schema.space.create('withdata', {engine = engine}) pk = s:create_index('pk') | --- | ... -lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {{tuple[1]}} else return {} end end]] | --- | ... box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) | --- | ... -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) | --- | ... s:insert({1}) diff --git a/test/engine/functional.test.lua b/test/engine/functional.test.lua index 92a74aab7..ae88f2900 100644 --- a/test/engine/functional.test.lua +++ b/test/engine/functional.test.lua @@ -60,10 +60,24 @@ idx = s:create_index('idx', {func = box.func.invalidreturn1.id, parts = {{1, 'un s:insert({1}) idx:drop() --- Invalid functional index extractor routine return: the second returned key invalid. +-- Invalid functional index extractor routine return: invalid return format for multikey index. lua_code = [[function(tuple) return {"hello", "world"}, {1, 2} end]] +box.schema.func.create('invalidreturn2', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn2.id, is_multikey = true, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: the second returned key invalid. +lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]] box.schema.func.create('invalidreturn3', {body = lua_code, is_deterministic = true, is_sandboxed = true}) -idx = s:create_index('idx', {func = box.func.invalidreturn3.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +idx = s:create_index('idx', {func = box.func.invalidreturn3.id, is_multikey = true, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) +s:insert({1}) +idx:drop() + +-- Invalid functional index extractor routine return: multikey return in case of regular index. +lua_code = [[function(tuple) return {{"hello", "world"}, {1, 2}} end]] +box.schema.func.create('invalidreturn4', {body = lua_code, is_deterministic = true, is_sandboxed = true}) +idx = s:create_index('idx', {func = box.func.invalidreturn4.id, parts = {{1, 'unsigned'}, {2, 'unsigned'}}}) s:insert({1}) idx:drop() @@ -102,10 +116,10 @@ collectgarbage() -- Multikey functional index. s = box.schema.space.create('withdata', {engine = engine}) -lua_code = [[function(tuple) return {tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]} end]] +lua_code = [[function(tuple) return {{tuple[1] + tuple[2]}, {tuple[1] + tuple[2]}, {tuple[1]}} end]] box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) pk = s:create_index('pk') -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) s:insert({1, 2}) s:insert({3, 5}) s:insert({5, 3}) @@ -120,10 +134,10 @@ box.schema.func.drop('extr') -- Multikey multipart functional index. s = box.schema.space.create('withdata', {engine = engine}) -lua_code = [[function(tuple) return {600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]} end]] +lua_code = [[function(tuple) return {{600 + tuple[1], 600 + tuple[2]}, {500 + tuple[1], 500 + tuple[2]}} end]] box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) pk = s:create_index('pk') -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}, {2, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}, {2, 'integer'}}}) s:insert({1, 2}) s:insert({2, 1}) s:insert({3, 3}) @@ -136,10 +150,10 @@ box.schema.func.drop('extr') -- Multikey non-unique functional index. s = box.schema.space.create('withdata', {engine = engine}) -lua_code = [[function(tuple) return {500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]} end]] +lua_code = [[function(tuple) return {{500 + tuple[1]}, {500 + tuple[2]}, {500 + tuple[2]}} end]] box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) pk = s:create_index('pk') -idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = false, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) s:insert({1, 2}) s:insert({2, 1}) idx:select({501}) @@ -176,11 +190,11 @@ lua_code = [[function(tuple) local address = string.split(tuple[2]) local ret = {} for _, v in pairs(address) do table.insert(ret, {utf8.upper(v)}) end - return unpack(ret) + return ret end]] test_run:cmd("setopt delimiter ''"); box.schema.func.create('addr_extractor', {body = lua_code, is_deterministic = true, is_sandboxed = true}) -idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, parts = {{1, 'string', collation = 'unicode_ci'}}}) +idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, is_multikey = true, parts = {{1, 'string', collation = 'unicode_ci'}}}) idx:select('uk') idx:select('Sis') s:drop() @@ -190,9 +204,9 @@ box.schema.func.drop('addr_extractor') -- Partial index with functional index extractor s = box.schema.space.create('withdata', {engine = engine}) pk = s:create_index('pk') -lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {tuple[1]} end end]] +lua_code = [[function(tuple) if tuple[1] % 2 == 1 then return {{tuple[1]}} else return {} end end]] box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_sandboxed = true}) -idx = s:create_index('idx', {unique = true, func = box.func.extr.id, parts = {{1, 'integer'}}}) +idx = s:create_index('idx', {unique = true, func = box.func.extr.id, is_multikey = true, parts = {{1, 'integer'}}}) s:insert({1}) s:insert({2}) s:insert({3}) -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov ` (3 preceding siblings ...) 2019-07-24 20:22 ` Konstantin Osipov @ 2019-07-24 20:44 ` Konstantin Osipov 2019-07-25 11:22 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 21:07 ` Konstantin Osipov ` (2 subsequent siblings) 7 siblings, 1 reply; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 20:44 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > index f98a77a51..c36b6643e 100644 > --- a/src/box/alter.cc > +++ b/src/box/alter.cc > @@ -45,6 +45,7 @@ > #include "fiber.h" /* for gc_pool */ > #include "scoped_guard.h" > #include "third_party/base64.h" > +#include "memtx_engine.h" > #include <new> /* for placement new */ > #include <stdio.h> /* snprintf() */ > #include <ctype.h> > @@ -195,7 +196,7 @@ err: > */ > static void > index_opts_decode(struct index_opts *opts, const char *map, > - struct region *region) > + struct space *space, struct region *region) > { > index_opts_create(opts); > if (opts_decode(opts, index_opts_reg, &map, ER_WRONG_INDEX_OPTIONS, > @@ -229,6 +230,22 @@ index_opts_decode(struct index_opts *opts, const char *map, > "bloom_fpr must be greater than 0 and " > "less than or equal to 1"); > } > + /** > + * Can't verify functional index function > + * reference on load because the function object > + * had not been registered in Tarantool yet. > + */ > + struct memtx_engine *engine = (struct memtx_engine *)space->engine; > + if (engine->state == MEMTX_OK && opts->functional_fid > 0) { > + struct func *func = func_cache_find(opts->functional_fid); > + if (func->def->language != FUNC_LANGUAGE_LUA || > + func->def->body == NULL || !func->def->is_deterministic || > + !func->def->is_sandboxed) { > + tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, > + "referenced function doesn't satisfy " > + "functional index constraints"); > + } > + } this part is obviously incorrect. alter should not depend on memtx_engine. And you don't need it at all. Please verify key parts inside the trigger on _func_index space, then you don't need to check for recovery state. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 20:44 ` Konstantin Osipov @ 2019-07-25 11:22 ` Kirill Shcherbatov 0 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-25 11:22 UTC (permalink / raw) To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev >> + /** >> + * Can't verify functional index function >> + * reference on load because the function object >> + * had not been registered in Tarantool yet. >> + */ >> + struct memtx_engine *engine = (struct memtx_engine *)space->engine; >> + if (engine->state == MEMTX_OK && opts->functional_fid > 0) { >> + struct func *func = func_cache_find(opts->functional_fid); >> + if (func->def->language != FUNC_LANGUAGE_LUA || >> + func->def->body == NULL || !func->def->is_deterministic || >> + !func->def->is_sandboxed) { >> + tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, >> + "referenced function doesn't satisfy " >> + "functional index constraints"); >> + } >> + } > > this part is obviously incorrect. alter should not depend on > memtx_engine. And you don't need it at all. Please verify > key parts inside the trigger on _func_index space, then you don't > need to check for recovery state. Don't mind. But we need to clean-up index object in case of failure. The change diff is below: (I've also dropped collectgarbage calls - it is a legacy) =========================================================== --- src/box/alter.cc | 27 +++++++------------- test/engine/functional.result | 44 +++++++++------------------------ test/engine/functional.test.lua | 12 +++------ 3 files changed, 25 insertions(+), 58 deletions(-) diff --git a/src/box/alter.cc b/src/box/alter.cc index 0b23dd6bb..eb2db1dc9 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -196,7 +196,7 @@ err: */ static void index_opts_decode(struct index_opts *opts, const char *map, - struct space *space, struct region *region) + struct region *region) { index_opts_create(opts); if (opts_decode(opts, index_opts_reg, &map, ER_WRONG_INDEX_OPTIONS, @@ -230,22 +230,6 @@ index_opts_decode(struct index_opts *opts, const char *map, "bloom_fpr must be greater than 0 and " "less than or equal to 1"); } - /** - * Can't verify functional index function - * reference on load because the function object - * had not been registered in Tarantool yet. - */ - struct memtx_engine *engine = (struct memtx_engine *)space->engine; - if (opts->func_id > 0 && engine->state == MEMTX_OK) { - struct func *func = func_cache_find(opts->func_id); - if (func->def->language != FUNC_LANGUAGE_LUA || - func->def->body == NULL || !func->def->is_deterministic || - !func->def->is_sandboxed) { - tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, - "referenced function doesn't satisfy " - "functional index constraints"); - } - } } /** @@ -277,7 +261,7 @@ index_def_new_from_tuple(struct tuple *tuple, struct space *space) const char *opts_field = tuple_field_with_type_xc(tuple, BOX_INDEX_FIELD_OPTS, MP_MAP); - index_opts_decode(&opts, opts_field, space, &fiber()->gc); + index_opts_decode(&opts, opts_field, &fiber()->gc); const char *parts = tuple_field(tuple, BOX_INDEX_FIELD_PARTS); uint32_t part_count = mp_decode_array(&parts); if (name_len > BOX_NAME_MAX) { @@ -4761,6 +4745,13 @@ on_replace_dd_func_index(struct trigger *trigger, void *event) space = space_cache_find_xc(space_id); index = index_find_xc(space, index_id); func = func_cache_find(fid); + if (func->def->language != FUNC_LANGUAGE_LUA || + func->def->body == NULL || !func->def->is_deterministic || + !func->def->is_sandboxed) { + tnt_raise(ClientError, ER_WRONG_INDEX_OPTIONS, 0, + "referenced function doesn't satisfy " + "functional index constraints"); + } } else if (old_tuple != NULL && new_tuple == NULL) { uint32_t space_id = tuple_field_u32_xc(old_tuple, BOX_FUNC_INDEX_FIELD_SPACE_ID); diff --git a/test/engine/functional.result b/test/engine/functional.result index ca8d0e366..c9f2d813a 100644 --- a/test/engine/functional.result +++ b/test/engine/functional.result @@ -49,24 +49,36 @@ _ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) | --- | - error: Function '6666' does not exist | ... +s.index.idx:drop() + | --- + | ... -- Can't use non-persistent function in functional index. _ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) | --- | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional | index constraints' | ... +s.index.idx:drop() + | --- + | ... -- Can't use non-deterministic function in functional index. _ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) | --- | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional | index constraints' | ... +s.index.idx:drop() + | --- + | ... -- Can't use non-sandboxed function in functional index. _ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) | --- | - error: 'Wrong index options (field 0): referenced function doesn''t satisfy functional | index constraints' | ... +s.index.idx:drop() + | --- + | ... -- Can't use non-sequential parts in returned key definition. _ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) | --- @@ -302,10 +314,6 @@ s:drop() box.schema.func.drop('extr') | --- | ... -collectgarbage() - | --- - | - 0 - | ... -- Multikey functional index. s = box.schema.space.create('withdata', {engine = engine}) @@ -361,10 +369,6 @@ idx:get(5) s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('extr') | --- | ... @@ -416,10 +420,6 @@ idx:select({503}, {iterator = "LE"}) s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('extr') | --- | ... @@ -521,10 +521,6 @@ idx:select({503}) s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('extr') | --- | ... @@ -582,10 +578,6 @@ idx:select('Sis') s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('addr_extractor') | --- | ... @@ -630,10 +622,6 @@ idx:select() s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('extr') | --- | ... @@ -662,10 +650,6 @@ s:insert({1}) s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('extr') | --- | ... @@ -717,10 +701,6 @@ idx2:get(3) s:drop() | --- | ... -collectgarbage() - | --- - | - 0 - | ... box.schema.func.drop('s') | --- | ... diff --git a/test/engine/functional.test.lua b/test/engine/functional.test.lua index ae88f2900..9718fbfb3 100644 --- a/test/engine/functional.test.lua +++ b/test/engine/functional.test.lua @@ -19,12 +19,16 @@ _ = s:create_index('idx', {func = box.func.s.id, parts = {{1, 'unsigned'}}}) pk = s:create_index('pk') -- Invalid fid. _ = s:create_index('idx', {func = 6666, parts = {{1, 'unsigned'}}}) +s.index.idx:drop() -- Can't use non-persistent function in functional index. _ = s:create_index('idx', {func = box.func.s_nonpersistent.id, parts = {{1, 'unsigned'}}}) +s.index.idx:drop() -- Can't use non-deterministic function in functional index. _ = s:create_index('idx', {func = box.func.s_ivaliddef1.id, parts = {{1, 'unsigned'}}}) +s.index.idx:drop() -- Can't use non-sandboxed function in functional index. _ = s:create_index('idx', {func = box.func.s_ivaliddef2.id, parts = {{1, 'unsigned'}}}) +s.index.idx:drop() -- Can't use non-sequential parts in returned key definition. _ = s:create_index('idx', {func = box.func.ss.id, parts = {{1, 'unsigned'}, {3, 'unsigned'}}}) -- Can't use parts started not by 1 field. @@ -112,7 +116,6 @@ s:insert({2, 1}) idx:get(3) s:drop() box.schema.func.drop('extr') -collectgarbage() -- Multikey functional index. s = box.schema.space.create('withdata', {engine = engine}) @@ -129,7 +132,6 @@ idx:get(3) idx:get(1) idx:get(5) s:drop() -collectgarbage() box.schema.func.drop('extr') -- Multikey multipart functional index. @@ -145,7 +147,6 @@ idx:select({600}, {iterator = "GE"}) idx:get({603, 603}) idx:select({503}, {iterator = "LE"}) s:drop() -collectgarbage() box.schema.func.drop('extr') -- Multikey non-unique functional index. @@ -174,7 +175,6 @@ idx:select({501}) idx:select({502}) idx:select({503}) s:drop() -collectgarbage() box.schema.func.drop('extr') -- Multikey UTF-8 address extractor @@ -198,7 +198,6 @@ idx = s:create_index('addr', {unique = false, func = box.func.addr_extractor.id, idx:select('uk') idx:select('Sis') s:drop() -collectgarbage() box.schema.func.drop('addr_extractor') -- Partial index with functional index extractor @@ -213,7 +212,6 @@ s:insert({3}) s:insert({4}) idx:select() s:drop() -collectgarbage() box.schema.func.drop('extr') -- Return nil from functional index extractor. @@ -224,7 +222,6 @@ box.schema.func.create('extr', {body = lua_code, is_deterministic = true, is_san idx = s:create_index('idx', {unique = false, func = box.func.extr.id, parts = {{1, 'integer', is_nullable = true}}}) s:insert({1}) s:drop() -collectgarbage() box.schema.func.drop('extr') -- Multiple functional indexes @@ -242,6 +239,5 @@ idx2:get(3) idx1:drop() idx2:get(3) s:drop() -collectgarbage() box.schema.func.drop('s') box.schema.func.drop('sub') -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov ` (4 preceding siblings ...) 2019-07-24 20:44 ` Konstantin Osipov @ 2019-07-24 21:07 ` Konstantin Osipov 2019-07-25 8:27 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 21:17 ` Konstantin Osipov 2019-07-24 21:56 ` Konstantin Osipov 7 siblings, 1 reply; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 21:07 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > +/** > + * Execute a given function (a functional index function) and > + * return an extracted key_data and key_data_sz and count of > + * extracted keys. > + * > + * Returns not NULL key_data pointer in case of success. > + * Routine allocates memory on fiber's region. > + */ > +const char * > +func_key_extract(struct tuple *tuple, struct func *func, > + const char **data_end, uint32_t *key_count); Please pass the allocator explicitly. I don't understand why you force the user of this module to first create a temporary buffer, then iterate over this buffer to create hint objects. Why can't this function return an iterator right away, and the iterator itself could return hint_t objects already, not individual keys? -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 21:07 ` Konstantin Osipov @ 2019-07-25 8:27 ` Kirill Shcherbatov 2019-07-25 8:40 ` Konstantin Osipov 0 siblings, 1 reply; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-25 8:27 UTC (permalink / raw) To: tarantool-patches, Konstantin Osipov; +Cc: Vladimir Davydov > I don't understand why you force the user of this module to first > create a temporary buffer, then iterate over this buffer to create > hint objects. Why can't this function return an iterator right > away, and the iterator itself could return hint_t objects already, > not individual keys? E.g. in :replace/delete part we don't need a persistent memory chunk: we need to evaluate function and iterate over the returned keys to perform a deletion. =========================================== key_list = key_list_create(old_tuple, cmp_def->func_index_func, &key_list_end, &key_cnt); if (key_list == NULL) goto end; struct memtx_tree_data data, deleted_data; data.tuple = old_tuple; key_list_iterator_create(&it, key_list, key_list_end, cmp_def, true); const char *key; uint32_t key_sz; while (key_list_iterator_next(&it, &key, &key_sz) == 0 && key != NULL) { data.hint = (hint_t) key; /* Raw data. */ deleted_data.tuple = NULL; memtx_tree_delete_value(&index->tree, data, &deleted_data); ..... ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-25 8:27 ` [tarantool-patches] " Kirill Shcherbatov @ 2019-07-25 8:40 ` Konstantin Osipov 2019-07-25 11:18 ` Kirill Shcherbatov 0 siblings, 1 reply; 22+ messages in thread From: Konstantin Osipov @ 2019-07-25 8:40 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, Vladimir Davydov * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/25 11:31]: > > I don't understand why you force the user of this module to first > > create a temporary buffer, then iterate over this buffer to create > > hint objects. Why can't this function return an iterator right > > away, and the iterator itself could return hint_t objects already, > > not individual keys? > > E.g. in :replace/delete part we don't need a persistent memory chunk: we need to > evaluate function and iterate over the returned keys to perform a deletion. > > =========================================== > > key_list = key_list_create(old_tuple, cmp_def->func_index_func, > &key_list_end, &key_cnt); > if (key_list == NULL) > goto end; > struct memtx_tree_data data, deleted_data; > data.tuple = old_tuple; > key_list_iterator_create(&it, key_list, key_list_end, cmp_def, true); > const char *key; > uint32_t key_sz; > while (key_list_iterator_next(&it, &key, &key_sz) == 0 && > key != NULL) { > data.hint = (hint_t) key; /* Raw data. */ Generally yes, you get the idea, but I don't get why key is const char *, and why you need key_sz at all. The iterator can return hint_t right away. Plus, you can merge key_list_create and key_list_iterator_create into a single function, there is no reason why one has to perform two calls instead of one. > deleted_data.tuple = NULL; > memtx_tree_delete_value(&index->tree, data, > &deleted_data); > ..... -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-25 8:40 ` Konstantin Osipov @ 2019-07-25 11:18 ` Kirill Shcherbatov 0 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-25 11:18 UTC (permalink / raw) To: tarantool-patches, Konstantin Osipov; +Cc: Vladimir Davydov Kostya, consider this API please: ================================================== diff --git a/src/box/key_list.h b/src/box/key_list.h new file mode 100644 index 000000000..3208f8d48 --- /dev/null +++ b/src/box/key_list.h @@ -0,0 +1,102 @@ +#ifndef TARANTOOL_BOX_KEY_LIST_H_INCLUDED +#define TARANTOOL_BOX_KEY_LIST_H_INCLUDED +/* + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include <stdbool.h> +#include <inttypes.h> + +#ifdef __cplusplus +extern "C" { +#endif + +struct key_def; +struct tuple; + +/** + * Function to prepare a value returned by + * key_list_iterator_next method. + */ +typedef void *(*key_allocator_t)(struct tuple *tuple, const char *key, + uint32_t key_sz); + +/** + * An iterator to iterate over the key_data returned by function + * and validate it with given key definition (when required). + */ +struct key_list_iterator { + /** The ancestor tuple. */ + struct tuple *tuple; + /** The pointer to currently processed key. */ + const char *data; + /** The pointer to the end of extracted key_data. */ + const char *data_end; + /** + * The sequential functional index key definition that + * describes a format of functional index function keys. + */ + struct key_def *key_def; + /** Whether iterator must validate processed keys. */ + bool validate; + /** The method to allocate returned keys. */ + key_allocator_t key_allocator; +}; + +/** + * Initialize a new functional index function returned keys + * iterator. + * Execute a function specified in a given functional index key + * definition (a functional index function) and initialize a new + * iterator on MsgPack array of with keys. Each key is a MsgPack + * array as well. + * + * Returns 0 in case of success, -1 otherwise. + * Uses fiber region to allocate memory. + */ +int +key_list_iterator_create(struct key_list_iterator *it, struct tuple *tuple, + struct key_def *key_def, bool validate, + key_allocator_t key_allocator, uint32_t *key_count); + +/** + * Perform key iterator step and update iterator state. + * Update key pointer with an actual key. + * + * Returns 0 on success. In case of error returns -1 and sets + * the corresponding diag message. + */ +int +key_list_iterator_next(struct key_list_iterator *it, void **value); + +#ifdef __cplusplus +} /* extern "C" */ +#endif + +#endif /* TARANTOOL_BOX_KEY_LIST_H_INCLUDED */ ================================================== diff --git a/src/box/key_list.c b/src/box/key_list.c new file mode 100644 index 000000000..004aee9fe --- /dev/null +++ b/src/box/key_list.c @@ -0,0 +1,142 @@ +/* + * Copyright 2010-2016, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include "key_list.h" + +#include "errcode.h" +#include "diag.h" +#include "func.h" +#include "fiber.h" +#include "key_def.h" +#include "port.h" +#include "tt_static.h" +#include "tuple.h" +#include "tuple_compare.h" + +int +key_list_iterator_create(struct key_list_iterator *it, struct tuple *tuple, + struct key_def *key_def, bool validate, + key_allocator_t key_allocator, uint32_t *key_count) +{ + it->key_def = key_def; + it->validate = validate; + it->tuple = tuple; + it->key_allocator = key_allocator; + + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + struct func *func = key_def->func_index_func; + + struct port out_port, in_port; + port_tuple_create(&in_port); + port_tuple_add(&in_port, tuple); + int rc = func_call(func, &in_port, &out_port); + port_destroy(&in_port); + if (rc != 0) + goto error; + uint32_t key_data_sz; + const char *key_data = port_get_msgpack(&out_port, &key_data_sz); + port_destroy(&out_port); + if (key_data == NULL) + goto error; + + it->data_end = key_data + key_data_sz; + assert(mp_typeof(*key_data) == MP_ARRAY); + if (mp_decode_array(&key_data) != 1) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + func->def->name, "to many values were returned"); + goto error; + } + if (key_def->is_multikey) { + if (mp_typeof(*key_data) != MP_ARRAY) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + func->def->name, + "multikey function mustn't return scalar"); + goto error; + } + *key_count = mp_decode_array(&key_data); + } else { + *key_count = 1; + } + it->data = key_data; + return 0; +error: + region_truncate(region, region_svp); + diag_set(ClientError, ER_FUNC_INDEX_FUNC, func->def->name, + diag_last_error(diag_get())->errmsg); + return -1; +} + +int +key_list_iterator_next(struct key_list_iterator *it, void **value) +{ + assert(it->data <= it->data_end); + if (it->data == it->data_end) { + *value = NULL; + return 0; + } + const char *key = it->data; + if (!it->validate) { + mp_next(&it->data); + assert(it->data <= it->data_end); + *value = it->key_allocator(it->tuple, key, it->data - key); + return *value != NULL ? 0 : -1; + } + + if (mp_typeof(*it->data) != MP_ARRAY) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + it->key_def->func_index_func->def->name, + "returned key type is invalid"); + return -1; + } + const char *rptr = key; + uint32_t part_count = mp_decode_array(&rptr); + if (part_count != it->key_def->part_count_for_func_index) { + const char *error_msg = + tt_sprintf(tnt_errcode_desc(ER_EXACT_MATCH), + it->key_def->part_count_for_func_index, + part_count); + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + it->key_def->func_index_func->def->name, error_msg); + return -1; + } + const char *key_end; + if (key_validate_parts(it->key_def, rptr, part_count, true, + &key_end) != 0) { + diag_set(ClientError, ER_FUNC_INDEX_FUNC, + it->key_def->func_index_func->def->name, + diag_last_error(diag_get())->errmsg); + return -1; + } + + it->data = key_end; + *value = it->key_allocator(it->tuple, key, key_end - key); + return *value != NULL ? 0 : -1; +} ================================================== diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index a9c1871db..77fcaf839 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -35,6 +35,7 @@ #include "errinj.h" #include "memory.h" #include "fiber.h" +#include "key_list.h" #include "tuple.h" #include <third_party/qsort_arg.h> #include <small/mempool.h> @@ -769,6 +770,201 @@ memtx_tree_index_replace_multikey(struct index *base, struct tuple *old_tuple, return 0; } +static void * +key_hint_plain(struct tuple *tuple, const char *key, uint32_t key_sz) +{ + (void)tuple; (void)key_sz; + return (void *)key; +} + +static void * +key_hint_new(struct tuple *tuple, const char *key, uint32_t key_sz) +{ + struct tuple_chunk *chunk = tuple_chunk_new(tuple, key_sz); + if (chunk == NULL) + return NULL; + memcpy(chunk->data, key, key_sz); + return chunk->data; +} + +static void +key_hint_delete(struct tuple *tuple, hint_t hint) +{ + struct tuple_chunk *chunk = + container_of((typeof(chunk->data) *)hint, + struct tuple_chunk, data); + tuple_chunk_delete(tuple, chunk); +} + +/** Release memory allocated for key_hint. */ +static void +key_hint_destroy(struct memtx_tree_data *data) +{ + key_hint_delete(data->tuple, data->hint); +} + +/** + * The journal for multikey functional index replace operation + * is required to rollback an incomplete action, restore the + * original key_hint(s) hints both as to commit a completed + * replace action and destruct useless key_hint(s). +*/ +struct journal_entry { + /** An inserted record copy. */ + struct memtx_tree_data inserted; + /** A replaced record copy. */ + struct memtx_tree_data replaced; +}; + +/** + * Rollback a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Routine uses given journal + * to return given index object in it's original state. + */ +static void +memtx_tree_index_replace_functional_rollback(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple != NULL) { + memtx_tree_insert(&index->tree, journal[i].replaced, + NULL); + } else { + memtx_tree_delete_value(&index->tree, + journal[i].inserted, NULL); + } + key_hint_destroy(&journal[i].inserted); + } +} + +/** + * Commit a sequence of memtx_tree_index_replace_multikey_one + * insertions for functional index. Rotine uses given operations + * journal to release unused memory. + */ +static void +memtx_tree_index_replace_functional_commit(struct memtx_tree_index *index, + struct journal_entry *journal, + int journal_sz) +{ + (void) index; + for (int i = 0; i < journal_sz; i++) { + if (journal[i].replaced.tuple == NULL) + continue; + key_hint_destroy(&journal[i].replaced); + } +} + +/** + * @sa memtx_tree_index_replace_multikey(). + * Use functional index function from the key definition + * to build a key list. Then each returned key is reallocated in + * engine's memory as key_hint object and is used as comparison + * hint. + * To control key_hint(s) life cycle in case of functional + * index we use a tiny journal object is allocated on region. + * It allows to restore original nodes with their original + * key_hint(s) pointers in case of failure and release + * useless hints of replaced items in case of success. + */ +static int +memtx_tree_index_replace_functional(struct index *base, struct tuple *old_tuple, + struct tuple *new_tuple, enum dup_replace_mode mode, + struct tuple **result) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_for_func_index(cmp_def)); + + int rc = -1; + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + *result = NULL; + uint32_t key_cnt; + struct key_list_iterator it; + if (new_tuple != NULL) { + if (key_list_iterator_create(&it, new_tuple, cmp_def, true, + key_hint_new, &key_cnt) != 0) + goto end; + + int journal_idx = 0; + struct journal_entry *journal = + region_alloc(region, key_cnt * sizeof(*journal)); + if (journal == NULL) { + diag_set(OutOfMemory, key_cnt * sizeof(*journal), + "region", "journal"); + goto end; + } + + int err = 0; + void *key; + while ((err = key_list_iterator_next(&it, &key)) == 0 && key != NULL) { + /* Perform insertion, log it in journal. */ + bool is_multikey_conflict; + journal[journal_idx].replaced.tuple = NULL; + journal[journal_idx].inserted.tuple = new_tuple; + journal[journal_idx].inserted.hint = (hint_t)key; + err = memtx_tree_index_replace_multikey_one(index, + old_tuple, new_tuple, mode, + (hint_t)key, + &journal[journal_idx].replaced, + &is_multikey_conflict); + if (err != 0) + break; + /** + * Modify a 'replace' record of journal + * because an original node shouldn't be + * restored in case of multikey conflict. + */ + if (is_multikey_conflict) + journal[journal_idx].replaced.tuple = NULL; + else if (journal[journal_idx].replaced.tuple != NULL) + *result = journal[journal_idx].replaced.tuple; + + ++journal_idx; + } + if (key != NULL || err != 0) { + memtx_tree_index_replace_functional_rollback(index, + journal, journal_idx); + goto end; + } + if (*result != NULL) { + assert(old_tuple == NULL || old_tuple == *result); + old_tuple = *result; + } + memtx_tree_index_replace_functional_commit(index, + journal, journal_idx); + } + if (old_tuple != NULL) { + if (key_list_iterator_create(&it, old_tuple, cmp_def, false, + key_hint_plain, &key_cnt) != 0) + goto end; + struct memtx_tree_data data, deleted_data; + data.tuple = old_tuple; + void *key; + while (key_list_iterator_next(&it, &key) == 0 && key != NULL) { + data.hint = (hint_t) key; + deleted_data.tuple = NULL; + memtx_tree_delete_value(&index->tree, data, + &deleted_data); + if (deleted_data.tuple != NULL) { + /* + * Release related hint on + * successfull node deletion. + */ + key_hint_destroy(&deleted_data); + } + } + assert(key == NULL); + } + rc = 0; +end: + region_truncate(region, region_svp); + return rc; +} + static struct iterator * memtx_tree_index_create_iterator(struct index *base, enum iterator_type type, const char *key, uint32_t part_count) @@ -900,13 +1096,47 @@ memtx_tree_index_build_next_multikey(struct index *base, struct tuple *tuple) return 0; } +static int +memtx_tree_index_build_next_functional(struct index *base, struct tuple *tuple) +{ + struct memtx_tree_index *index = (struct memtx_tree_index *)base; + struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); + assert(key_def_is_for_func_index(cmp_def)); + + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + + uint32_t key_cnt; + struct key_list_iterator it; + if (key_list_iterator_create(&it, tuple, cmp_def, false, key_hint_new, + &key_cnt) != 0) + return -1; + + void *key; + uint32_t insert_idx = index->build_array_size; + while (key_list_iterator_next(&it, &key) == 0 && key != NULL) { + if (memtx_tree_index_build_array_append(index, tuple, + (hint_t)key) != 0) + goto error; + } + assert(key == NULL); + region_truncate(region, region_svp); + return 0; +error: + for (uint32_t i = insert_idx; i < index->build_array_size; i++) + key_hint_destroy(&index->build_array[i]); + region_truncate(region, region_svp); + return -1; +} + /** * Process build_array of specified index and remove duplicates * of equal tuples (in terms of index's cmp_def and have same * tuple pointer). The build_array is expected to be sorted. */ static void -memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index) +memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index, + void (*data_destructor)(struct memtx_tree_data *data)) { if (index->build_array_size == 0) return; @@ -924,10 +1154,18 @@ memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index) if (++w_idx == r_idx) continue; index->build_array[w_idx] = index->build_array[r_idx]; + if (data_destructor == NULL) + continue; + for (size_t i = w_idx + 1; i < r_idx; i++) + data_destructor(&index->build_array[i]); } r_idx++; } index->build_array_size = w_idx + 1; + if (data_destructor != NULL) { + for (size_t i = index->build_array_size + 1; i < r_idx; i++) + data_destructor(&index->build_array[i]); + } } static void @@ -945,7 +1183,9 @@ memtx_tree_index_end_build(struct index *base) * the following memtx_tree_build assumes that * all keys are unique. */ - memtx_tree_index_build_array_deduplicate(index); + memtx_tree_index_build_array_deduplicate(index, NULL); + } else if (key_def_is_for_func_index(cmp_def)) { + memtx_tree_index_build_array_deduplicate(index, key_hint_destroy); } memtx_tree_build(&index->tree, index->build_array, index->build_array_size); @@ -1072,6 +1312,72 @@ static const struct index_vtab memtx_tree_index_multikey_vtab = { /* .end_build = */ memtx_tree_index_end_build, }; +static const struct index_vtab memtx_tree_index_functional_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ memtx_tree_index_update_def, + /* .depends_on_pk = */ memtx_tree_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + memtx_index_def_change_requires_rebuild, + /* .size = */ memtx_tree_index_size, + /* .bsize = */ memtx_tree_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ memtx_tree_index_random, + /* .count = */ memtx_tree_index_count, + /* .get = */ memtx_tree_index_get, + /* .replace = */ memtx_tree_index_replace_functional, + /* .create_iterator = */ memtx_tree_index_create_iterator, + /* .create_snapshot_iterator = */ + memtx_tree_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ memtx_tree_index_begin_build, + /* .reserve = */ memtx_tree_index_reserve, + /* .build_next = */ memtx_tree_index_build_next_functional, + /* .end_build = */ memtx_tree_index_end_build, +}; + +/** + * A disabled index vtab provides safe dummy methods for + * 'inactive' index. It is required to perform a fault-tolerant + * recovery from snapshoot in case of functional index (because + * key defintion is not completely initialized at that moment). + */ +static const struct index_vtab memtx_tree_index_disabled_vtab = { + /* .destroy = */ memtx_tree_index_destroy, + /* .commit_create = */ generic_index_commit_create, + /* .abort_create = */ generic_index_abort_create, + /* .commit_modify = */ generic_index_commit_modify, + /* .commit_drop = */ generic_index_commit_drop, + /* .update_def = */ generic_index_update_def, + /* .depends_on_pk = */ generic_index_depends_on_pk, + /* .def_change_requires_rebuild = */ + generic_index_def_change_requires_rebuild, + /* .size = */ generic_index_size, + /* .bsize = */ generic_index_bsize, + /* .min = */ generic_index_min, + /* .max = */ generic_index_max, + /* .random = */ generic_index_random, + /* .count = */ generic_index_count, + /* .get = */ generic_index_get, + /* .replace = */ disabled_index_replace, + /* .create_iterator = */ generic_index_create_iterator, + /* .create_snapshot_iterator = */ + generic_index_create_snapshot_iterator, + /* .stat = */ generic_index_stat, + /* .compact = */ generic_index_compact, + /* .reset_stat = */ generic_index_reset_stat, + /* .begin_build = */ generic_index_begin_build, + /* .reserve = */ generic_index_reserve, + /* .build_next = */ disabled_index_build_next, + /* .end_build = */ generic_index_end_build, +}; + struct index * memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) { @@ -1082,9 +1388,17 @@ memtx_tree_index_new(struct memtx_engine *memtx, struct index_def *def) "malloc", "struct memtx_tree_index"); return NULL; } - const struct index_vtab *vtab = def->key_def->is_multikey ? - &memtx_tree_index_multikey_vtab : - &memtx_tree_index_vtab; + const struct index_vtab *vtab; + if (key_def_is_for_func_index(def->key_def)) { + if (def->key_def->func_index_func == NULL) + vtab = &memtx_tree_index_disabled_vtab; + else + vtab = &memtx_tree_index_functional_vtab; + } else if (def->key_def->is_multikey) { + vtab = &memtx_tree_index_multikey_vtab; + } else { + vtab = &memtx_tree_index_vtab; + } if (index_create(&index->base, (struct engine *)memtx, vtab, def) != 0) { free(index); ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov ` (5 preceding siblings ...) 2019-07-24 21:07 ` Konstantin Osipov @ 2019-07-24 21:17 ` Konstantin Osipov 2019-07-24 21:56 ` Konstantin Osipov 7 siblings, 0 replies; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 21:17 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: Would it be possible split the part that adds disabled indexes into a separate patch? -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov ` (6 preceding siblings ...) 2019-07-24 21:17 ` Konstantin Osipov @ 2019-07-24 21:56 ` Konstantin Osipov 2019-07-25 8:33 ` [tarantool-patches] " Kirill Shcherbatov 7 siblings, 1 reply; 22+ messages in thread From: Konstantin Osipov @ 2019-07-24 21:56 UTC (permalink / raw) To: Kirill Shcherbatov; +Cc: tarantool-patches, vdavydov.dev * Kirill Shcherbatov <kshcherbatov@tarantool.org> [19/07/24 10:38]: > --- a/src/box/errcode.h > +++ b/src/box/errcode.h > @@ -250,6 +250,7 @@ struct errcode_record { > /*195 */_(ER_CREATE_CK_CONSTRAINT, "Failed to create check constraint '%s': %s") \ > /*196 */_(ER_CK_CONSTRAINT_FAILED, "Check constraint failed '%s': %s") \ > /*197 */_(ER_SQL_COLUMN_COUNT, "Unequal number of entries in row expression: left side has %u, but right side - %u") \ > + /*198 */_(ER_FUNCTIONAL_INDEX_FUNC_ERROR,"Functional index function '%s' error: %s") \ > There is ER_ prefix, no need to add ERROR suffix. The error code should reference index name and table name, not function name, or all of this. Imagine getting this error for a Lua procedure, it should not require too many extra steps to nail down the cause. There should be different error codes for different errors. Resetting the original error is also a very unfriendly, please add a ticket and a follow up patch to your todo list for stacked errors. I mean, it's the last drop, how many times shall we step on lack of stacked diagnostics before we add them? > +/** > + * Update func pointer for functional index key definitions. > + * @param def Index def, containing key definitions to update. > + * @param func The functional index function pointer. > + */ > +static inline void > +index_def_update_func(struct index_def *def, struct func *func) > +{ > + def->key_def->func_index_func = func; > + def->cmp_def->func_index_func = func; > +} should be set_func, not update_func. > uint32_t unique_part_count; > + /** > + * Count of parts in functional index defintion. > + * All functional_part_count key_part(s) of an > + * initialized key def instance have func != NULL pointer. > + * != 0 iff it is functional index definition. > + */ > + uint32_t functional_part_count; I don't get this comment, we don't store func pointer in key_part. Why do you need this at all? part_count should be enough. > + /** A pointer to functional index function. */ > + struct func *func_index_func; Please explain the life cycle of this member (it is not immediately initialized, when is it assigned then?, when is it safe to rely on it?) and how it is used. > + /** Space id of _func_index. */ > + BOX_FUNC_INDEX_ID = 372, Please explain how this space is used. Please document why we decided to have a separate space. > + *key = it->data; > + if (!it->validate) { Please document circumstances when validate is false. > + if (mp_typeof(*it->data) != MP_ARRAY) { > + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, > + if (part_count != functional_part_count) { > + const char *error_msg = > + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, > + if (key_validate_parts(it->key_def, rptr, functional_part_count, true, > + &key_end) != 0) { > + diag_set(ClientError, ER_FUNCTIONAL_INDEX_FUNC_ERROR, These three places should use its own error code, not ER_FUNC_INDEX_FUNC, ER_FUNC_INDEX_FUNC_FORMAT I guess. > @@ -251,8 +253,15 @@ index_def_to_key_def(struct rlist *index_defs, int *size) > { > int key_count = 0; > struct index_def *index_def; > - rlist_foreach_entry(index_def, index_defs, link) > + rlist_foreach_entry(index_def, index_defs, link) { > + /* > + * Don't use functional index key definition > + * to build a space format. > + */ > + if (key_def_is_functional(index_def->key_def)) > + continue; > key_count++; Why not skip such keys when building a space format? What is the benefit of depriving the format module of these key defs? > @@ -262,8 +271,11 @@ index_def_to_key_def(struct rlist *index_defs, int *size) > } > *size = key_count; > key_count = 0; > - rlist_foreach_entry(index_def, index_defs, link) > + rlist_foreach_entry(index_def, index_defs, link) { > + if (key_def_is_functional(index_def->key_def)) > + continue; > keys[key_count++] = index_def->key_def; > + } > return keys; Same here. I'm sure space format module can figure out that it deals with a functional key definition and skip it if necessary. > + if (index_def->iid == 0 && key_def_is_functional(index_def->key_def)) { > + diag_set(ClientError, ER_MODIFY_INDEX, index_def->name, > + space_name, "primary key cannot be functional"); primary key can not use a function > struct key_def * > -key_def_new(const struct key_part_def *parts, uint32_t part_count) > +key_def_new(const struct key_part_def *parts, uint32_t part_count, > + bool is_functional) I don't like it that key_def_new has become overly complicated with collations, multikey and functional indexes. Perhaps we need to introduce key_def_is_valid, similar to index_def_is_valid(), but I need to think. > key_def_can_merge(const struct key_def *key_def, > const struct key_part *to_merge) > { > + if (key_def_is_functional(key_def)) > + return true; Please explain what's going on here in a comment. > struct key_def * > key_def_merge(const struct key_def *first, const struct key_def *second) > { > + assert(!key_def_is_functional(second)); Same here. > - struct key_def *key_def = key_def_new(parts, part_count); > + struct key_def *key_def = key_def_new(parts, part_count, false); I don't get this, there should be two flags (is_multikey, is_functional). How did this compile? > > + if (index_opts->functional_fid > 0) { > + lua_pushnumber(L, index_opts->functional_fid); > + lua_setfield(L, -2, "func_id"); > + } Why not function name? > +template<bool is_nullable> > +static inline int > +functional_compare(struct tuple *tuple_a, hint_t tuple_a_hint, > + struct tuple *tuple_b, hint_t tuple_b_hint, > + struct key_def *key_def) > +{ Please solicit Vlad's review and explain the changes here in a changeset comment. > +static char * > +tuple_extract_key_stub(struct tuple *tuple, struct key_def *key_def, > + int multikey_idx, uint32_t *key_size) > +{ > + (void)tuple; (void)key_def; (void)multikey_idx; (void)key_size; > + unreachable(); > + return NULL; > +} > + > +static char * > +tuple_extract_key_raw_stub(const char *data, const char *data_end, > + struct key_def *key_def, int multikey_idx, > + uint32_t *key_size) > +{ > + (void)data; (void)data_end; > + (void)key_def; (void)multikey_idx; (void)key_size; > + unreachable(); > + return NULL; > +} Why do you need these? -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] Re: [PATCH v4 4/4] box: introduce functional indexes 2019-07-24 21:56 ` Konstantin Osipov @ 2019-07-25 8:33 ` Kirill Shcherbatov 0 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-25 8:33 UTC (permalink / raw) To: tarantool-patches, Konstantin Osipov; +Cc: vdavydov.dev >> + /** >> + * Count of parts in functional index defintion. >> + * All functional_part_count key_part(s) of an >> + * initialized key def instance have func != NULL pointer. >> + * != 0 iff it is functional index definition. >> + */ >> + uint32_t functional_part_count; > > I don't get this comment, we don't store func pointer in key_part. > Why do you need this at all? part_count should be enough. In case of secondary key it's definition consists of two parts (produced with key_def_merge) - functional index key parts that refer to hint_t memory and tuple parts are specified in primary key. >> + if (index_opts->functional_fid > 0) { >> + lua_pushnumber(L, index_opts->functional_fid); >> + lua_setfield(L, -2, "func_id"); >> + } > > Why not function name? Function name is not defined during recovery. I may print it in other cases, if it is ok for you if (index_opts->func_id > 0) { lua_pushstring(L, "func"); lua_newtable(L); lua_pushnumber(L, index_opts->func_id); lua_setfield(L, -2, "fid"); lua_pushboolean(L, index_opts->is_multikey); lua_setfield(L, -2, "is_multikey"); struct func *func = func_by_id(index_opts->func_id); if (func != NULL) { lua_pushstring(L, func->def->name); lua_setfield(L, -2, "name"); } lua_settable(L, -3); } >> +static char * >> +tuple_extract_key_stub(struct tuple *tuple, struct key_def *key_def, >> + int multikey_idx, uint32_t *key_size) >> +{ >> + (void)tuple; (void)key_def; (void)multikey_idx; (void)key_size; >> + unreachable(); >> + return NULL; >> +} >> + >> +static char * >> +tuple_extract_key_raw_stub(const char *data, const char *data_end, >> + struct key_def *key_def, int multikey_idx, >> + uint32_t *key_size) >> +{ >> + (void)data; (void)data_end; >> + (void)key_def; (void)multikey_idx; (void)key_size; >> + unreachable(); >> + return NULL; >> +} > > Why do you need these? > This code is really unreachable now. However this produce assertion error in case of further invalid refactoring. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [tarantool-patches] [PATCH v4 4/5] box: fix memtx_tree_index_build_array_deduplicate 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov ` (3 preceding siblings ...) 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov @ 2019-07-24 12:25 ` Kirill Shcherbatov 4 siblings, 0 replies; 22+ messages in thread From: Kirill Shcherbatov @ 2019-07-24 12:25 UTC (permalink / raw) To: tarantool-patches, kostja; +Cc: vdavydov.dev Function memtx_tree_index_build_array_deduplicate worked incorrectly: when build_array is empty it changed it size to 1. Follow up #1257 Needed for #1260 --- src/box/memtx_tree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/box/memtx_tree.c b/src/box/memtx_tree.c index 41eb6dbec..dc167e113 100644 --- a/src/box/memtx_tree.c +++ b/src/box/memtx_tree.c @@ -862,6 +862,8 @@ memtx_tree_index_build_next_multikey(struct index *base, struct tuple *tuple) static void memtx_tree_index_build_array_deduplicate(struct memtx_tree_index *index) { + if (index->build_array_size == 0) + return; struct key_def *cmp_def = memtx_tree_cmp_def(&index->tree); size_t w_idx = 0, r_idx = 1; while (r_idx < index->build_array_size) { -- 2.22.0 ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2019-07-25 11:22 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-07-24 7:36 [PATCH v4 0/4] box: functional indexes Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 1/4] box: introduce tuple_chunk infrastructure Kirill Shcherbatov 2019-07-24 7:36 ` [PATCH v4 2/4] box: generalize memtx_multikey_tree methods Kirill Shcherbatov 2019-07-24 19:24 ` Konstantin Osipov 2019-07-24 7:36 ` [PATCH v4 3/4] box: refactor memtx_tree_delete_identical Kirill Shcherbatov 2019-07-24 19:24 ` Konstantin Osipov 2019-07-24 7:36 ` [PATCH v4 4/4] box: introduce functional indexes Kirill Shcherbatov 2019-07-24 12:24 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 19:41 ` Konstantin Osipov 2019-07-24 20:04 ` Konstantin Osipov 2019-07-24 20:22 ` Konstantin Osipov 2019-07-25 11:20 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 20:44 ` Konstantin Osipov 2019-07-25 11:22 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 21:07 ` Konstantin Osipov 2019-07-25 8:27 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-25 8:40 ` Konstantin Osipov 2019-07-25 11:18 ` Kirill Shcherbatov 2019-07-24 21:17 ` Konstantin Osipov 2019-07-24 21:56 ` Konstantin Osipov 2019-07-25 8:33 ` [tarantool-patches] " Kirill Shcherbatov 2019-07-24 12:25 ` [tarantool-patches] [PATCH v4 4/5] box: fix memtx_tree_index_build_array_deduplicate Kirill Shcherbatov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox