From v.shpilevoy at tarantool.org Sat Aug 1 01:31:30 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 1 Aug 2020 00:31:30 +0200 Subject: [Tarantool-patches] [PATCH 0/3] Qsync tricky crashes In-Reply-To: <20200731135913.GI3264@grain> References: <20200731135913.GI3264@grain> Message-ID: <766b8cfc-05e9-871f-3091-f4f1679275cb@tarantool.org> Pushed to master and 2.5. On 31.07.2020 15:59, Cyrill Gorcunov wrote: > On Fri, Jul 31, 2020 at 12:37:42AM +0200, Vladislav Shpilevoy wrote: >> The patchset fixes 2 crashes in qsync. 5185 wasn't reproduced, but a similar >> test gave another crash, fixed here. >> >> Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5185-5195-qsync-crashes >> Issue: https://github.com/tarantool/tarantool/issues/5185 >> Issue: https://github.com/tarantool/tarantool/issues/5195 >> >> @ChangeLog >> * Fixed a crash when replica cleared synchronous transaction queue, while it was not empty on master (gh-5195). >> * Fixed a crash when synchronous transaction rollback and confirm could be written simultaneously for the same LSN (gh-5185). > > Ack. Lets give them a spin. > From v.shpilevoy at tarantool.org Sun Aug 2 17:44:28 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sun, 2 Aug 2020 16:44:28 +0200 Subject: [Tarantool-patches] [PATCH 2/2] vinyl: rework upsert operation In-Reply-To: <14aec86329c7820470e40e47a3d1b5655b531732.1595985135.git.korablev@tarantool.org> References: <14aec86329c7820470e40e47a3d1b5655b531732.1595985135.git.korablev@tarantool.org> Message-ID: <444ff8e3-d6c2-e6a5-8b54-18a521208ba5@tarantool.org> Thanks for the patch! ASAN tests on the branch are failing: https://gitlab.com/tarantool/tarantool/-/jobs/661977877 See 19 comments below. > src/box/vinyl.c | 2 +- > src/box/vy_stmt.c | 28 ++-- > src/box/vy_stmt.h | 5 +- > src/box/vy_upsert.c | 305 +++++++++++++++++++++++++++------------- > test/unit/vy_iterators_helper.c | 2 +- > test/vinyl/upsert.result | 289 +++++++++++++++++++++++++++++++++++++ > test/vinyl/upsert.test.lua | 121 ++++++++++++++++ > 7 files changed, 644 insertions(+), 108 deletions(-) > > diff --git a/src/box/vy_upsert.c b/src/box/vy_upsert.c > index 797492c2b..caf2482c7 100644 > --- a/src/box/vy_upsert.c > +++ b/src/box/vy_upsert.c > @@ -68,12 +68,173 @@ vy_upsert_try_to_squash(struct tuple_format *format, > operations[0].iov_len = squashed_size; > > *result_stmt = vy_stmt_new_upsert(format, key_mp, key_mp_end, > - operations, 1); > + operations, 1, false); > if (*result_stmt == NULL) > return -1; > return 0; > } > > +/** > + * Check that key hasn't been changed after applying upsert operation. > + */ > +static bool > +vy_apply_result_does_cross_pk(struct tuple *old_stmt, const char *result, > + const char *result_end, struct key_def *cmp_def, > + uint64_t col_mask) > +{ > + if (!key_update_can_be_skipped(cmp_def->column_mask, col_mask)) { > + struct tuple *tuple = > + vy_stmt_new_replace(tuple_format(old_stmt), result, > + result_end); > + int cmp_res = vy_stmt_compare(old_stmt, HINT_NONE, tuple, > + HINT_NONE, cmp_def); 1. Bad indentation. > + tuple_unref(tuple); > + return cmp_res != 0; > + } > + return false; > +} > + > +/** > + * Apply update operations stored in @new_stmt (which is assumed to 2. Please, look at the doxygen syntax on the official page or here: https://github.com/tarantool/tarantool/wiki/Code-review-procedure This is a single very simple rule I keep repeating I don't already remember how many times - use @a , not @. I don't understand why does everyone keep violating it. 2. Parameter 'new_stmt' does not exist. As well as 'old_stmt'. What did you mean? > + * be upsert statement) on tuple @old_stmt. If @old_stmt is void > + * statement (i.e. it is NULL or delete statement) then operations > + * are applied on tuple @new_stmt. All operations which can't be > + * applied are skipped; errors may be logged depending on @supress_error 3. supress_error -> supress_error. 4. What do you mean as 'all operations'? Operation groups from various upserts? Or individual operations? > + * flag. > + * > + * @upsert Upsert statement to be applied on @stmt. 5. If you want to use doxygen, use @param . > + * @stmt Statement to be used as base for upsert operations. > + * @cmp_def Key definition required to provide check of primary key > + * modification. > + * @retrun Tuple containing result of upsert application; > + * NULL in case OOM. 6. retrun -> return. 7. I guess you are among the ones who voted for 80 symbol comments - I suggest you to use it. Since this is our new code style now. > + */ > +static struct tuple * > +vy_apply_upsert_on_terminal_stmt(struct tuple *upsert, struct tuple *stmt, > + struct key_def *cmp_def, bool suppress_error) > +{ > + assert(vy_stmt_type(upsert) == IPROTO_UPSERT); > + assert(stmt == NULL || vy_stmt_type(stmt) != IPROTO_UPSERT); > + > + uint32_t mp_size; > + const char *new_ops = vy_stmt_upsert_ops(upsert, &mp_size); > + /* Msgpack containing result of upserts application. */ > + const char *result_mp; > + if (vy_stmt_is_void(stmt)) 8. This {is_void} helper is used 2 times inside one funtion on the same value. Seems like you could simply inline it, remeber result into a variable {bool is_void;} and use it instead. > + result_mp = vy_upsert_data_range(upsert, &mp_size); > + else > + result_mp = tuple_data_range(stmt, &mp_size); > + const char *result_mp_end = result_mp + mp_size; > + /* > + * xrow_upsert_execute() allocates result using region, > + * so save starting point to release it later. > + */ > + struct region *region = &fiber()->gc; > + size_t region_svp = region_used(region); > + uint64_t column_mask = COLUMN_MASK_FULL; > + struct tuple_format *format = tuple_format(upsert); > + > + uint32_t ups_cnt = mp_decode_array(&new_ops); > + const char *ups_ops = new_ops; > + /* > + * In case upsert folds into insert, we must skip first > + * update operations. > + */ > + if (vy_stmt_is_void(stmt)) { > + ups_cnt--; > + mp_next(&ups_ops); > + } > + for (uint32_t i = 0; i < ups_cnt; ++i) { > + assert(mp_typeof(*ups_ops) == MP_ARRAY); > + const char *ups_ops_end = ups_ops; > + mp_next(&ups_ops_end); > + const char *exec_res = result_mp; > + exec_res = xrow_upsert_execute(ups_ops, ups_ops_end, result_mp, > + result_mp_end, format, &mp_size, > + 0, suppress_error, &column_mask); > + if (exec_res == NULL) { > + if (! suppress_error) { 9. According to one another recent code style change, unary operators should not have a whitespace after them. > + assert(diag_last_error(diag_get()) != NULL); 10. Use {diag_is_empty}. Or better - save {diag_last_error(diag_get())} into {struct error *e} before the assertion, and use {assert(e != NULL);}. > + struct error *e = diag_last_error(diag_get()); > + /* Bail out immediately in case of OOM. */ > + if (e->type != &type_ClientError) { > + region_truncate(region, region_svp); > + return NULL; > + } > + diag_log(); > + } > + ups_ops = ups_ops_end; > + continue; > + } > + /* > + * If it turns out that resulting tuple modifies primary > + * key, than simply ignore this upsert. 11. than -> then. > + */ > + if (vy_apply_result_does_cross_pk(stmt, exec_res, > + exec_res + mp_size, cmp_def, > + column_mask)) { > + if (! suppress_error) { > + say_error("upsert operations %s are not applied"\ > + " due to primary key modification", > + mp_str(ups_ops)); > + } > + ups_ops = ups_ops_end; > + continue; > + } > + ups_ops = ups_ops_end; > + /* > + * In case statement exists its format must > + * satisfy space's format. Otherwise, upsert's > + * tuple is checked to fit format once it is > + * processed in vy_upsert(). > + */ > + if (stmt != NULL) { > + if (tuple_validate_raw(tuple_format(stmt), > + exec_res) != 0) { > + if (! suppress_error) > + diag_log(); > + continue; > + } > + } > + result_mp = exec_res; > + result_mp_end = exec_res + mp_size; > + } > + struct tuple *new_terminal_stmt = vy_stmt_new_replace(format, result_mp, > + result_mp_end); > + region_truncate(region, region_svp); > + if (new_terminal_stmt == NULL) > + return NULL; > + vy_stmt_set_lsn(new_terminal_stmt, vy_stmt_lsn(upsert)); > + return new_terminal_stmt; > +} > + > +static bool > +tuple_format_is_suitable_for_squash(struct tuple_format *format) > +{ > + struct tuple_field *field; > + json_tree_foreach_entry_preorder(field, &format->fields.root, > + struct tuple_field, token) { > + if (field->type == FIELD_TYPE_UNSIGNED) > + return false; 12. Bad indentation. Also this squash rule is not going to work because integer type also can overflow, both below INT64_MIN and above UINT64_MAX. Decimal types can overflow. Besides, decimal can fail when a non-decimal value does not fit the decimal type during conversion. For example, a huge double value. DBL_MAX is bigger than maximal value available in our decimal type. See xrow_update_arith_make() for all errors. Since squash is mostly about squashing +/-, looks like it won't work almost always, and becomes useless. P.S. In the end of the review I noticed that this prevents squashing not only of operations with unsigned fields. It will prevent squashing if the whole format has at least one unsigned. This makes current implementation of squash even more useless, because forbids to use the fastest field type, which is default when you create an index without specification of field type btw. > + } > + return true; > +} > + > +/** > + * Unpack upsert's update operations from msgpack array > + * into array of iovecs. > + */ > +static void > +upsert_ops_to_iovec(const char *ops, uint32_t ops_cnt, struct iovec *iov_arr) > +{ > + for (uint32_t i = 0; i < ops_cnt; ++i) { > + assert(mp_typeof(*ops) == MP_ARRAY); > + iov_arr[i].iov_base = (char *) ops; > + mp_next(&ops); > + iov_arr[i].iov_len = ops - (char *) iov_arr[i].iov_base; > + } > +} > + > struct tuple * > vy_apply_upsert(struct tuple *new_stmt, struct tuple *old_stmt, > struct key_def *cmp_def, bool suppress_error) > @@ -87,122 +248,74 @@ vy_apply_upsert(struct tuple *new_stmt, struct tuple *old_stmt, > assert(new_stmt != old_stmt); > assert(vy_stmt_type(new_stmt) == IPROTO_UPSERT); > > - if (old_stmt == NULL || vy_stmt_type(old_stmt) == IPROTO_DELETE) { > - /* > - * INSERT case: return new stmt. > - */ > - return vy_stmt_replace_from_upsert(new_stmt); > + struct tuple *result_stmt = NULL; > + if (old_stmt == NULL || vy_stmt_type(old_stmt) != IPROTO_UPSERT) { > + return vy_apply_upsert_on_terminal_stmt(new_stmt, old_stmt, > + cmp_def, suppress_error); > } > > - struct tuple_format *format = tuple_format(new_stmt); > - > + assert(vy_stmt_type(old_stmt) == IPROTO_UPSERT); 13. The assertion looks useless, since it is reverse of the {if} condition above, but up to you. > /* > - * Unpack UPSERT operation from the new stmt > + * Unpack UPSERT operation from the old and new stmts. > */ > + assert(old_stmt != NULL); 14. This is strage to check old_stmt->type in the previous assertion before you checked old_stmt != NULL. > uint32_t mp_size; > - const char *new_ops; > - new_ops = vy_stmt_upsert_ops(new_stmt, &mp_size); > - const char *new_ops_end = new_ops + mp_size; > + const char *old_ops = vy_stmt_upsert_ops(old_stmt, &mp_size); > + const char *old_ops_end = old_ops + mp_size; > + assert(old_ops_end > old_ops); > + const char *old_stmt_mp = vy_upsert_data_range(old_stmt, &mp_size); > + const char *old_stmt_mp_end = old_stmt_mp + mp_size; > + const char *new_ops = vy_stmt_upsert_ops(new_stmt, &mp_size); > > /* > - * Apply new operations to the old stmt > + * UPSERT + UPSERT case: squash arithmetic operations. > + * Note that we can process this only in case result > + * can't break format under no circumstances. Since > + * subtraction can lead to negative values, unsigned > + * field are considered to be inappropriate. > */ > - const char *result_mp; > - if (vy_stmt_type(old_stmt) == IPROTO_UPSERT) > - result_mp = vy_upsert_data_range(old_stmt, &mp_size); > - else > - result_mp = tuple_data_range(old_stmt, &mp_size); > - const char *result_mp_end = result_mp + mp_size; > - struct tuple *result_stmt = NULL; > + struct tuple_format *format = tuple_format(old_stmt); > struct region *region = &fiber()->gc; > size_t region_svp = region_used(region); > - uint8_t old_type = vy_stmt_type(old_stmt); > - uint64_t column_mask = COLUMN_MASK_FULL; > - result_mp = xrow_upsert_execute(new_ops, new_ops_end, result_mp, > - result_mp_end, format, &mp_size, > - 0, suppress_error, &column_mask); > - if (result_mp == NULL) { > - region_truncate(region, region_svp); > - return NULL; > + if (tuple_format_is_suitable_for_squash(format)) { > + const char *new_ops_end = new_ops + mp_size; > + if (vy_upsert_try_to_squash(format, old_stmt_mp, old_stmt_mp_end, > + old_ops, old_ops_end, new_ops, > + new_ops_end, &result_stmt) != 0) { > + /* OOM */ > + region_truncate(region, region_svp); > + return NULL; > + } 15. vy_upsert_try_to_squash() returns a result into result_stmt. But you ignore it. Basically, whatever it returns, you act like squash didn't happen and it never works now. You continue to work with 2 old operation set arrays. Also result_stmt leaks. What is also strange - I added {assert(false);} here and the tests passed. I thought we had quite a lot squash tests. Seems they are all for formats having unsigned field type. (Actualy the tests failed, but not here - on my machine vinyl tests fail in almost 100% runs somewhere with random errors, could be luajit problems on Mac maybe.) > } > - result_mp_end = result_mp + mp_size; > - if (old_type != IPROTO_UPSERT) { > - assert(old_type == IPROTO_INSERT || > - old_type == IPROTO_REPLACE); > - /* > - * UPDATE case: return the updated old stmt. > - */ > - result_stmt = vy_stmt_new_replace(format, result_mp, > - result_mp_end); > - region_truncate(region, region_svp); > - if (result_stmt == NULL) > - return NULL; /* OOM */ > - vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); > - goto check_key; > - } > - > - /* > - * Unpack UPSERT operation from the old stmt > - */ > - assert(old_stmt != NULL); > - const char *old_ops; > - old_ops = vy_stmt_upsert_ops(old_stmt, &mp_size); > - const char *old_ops_end = old_ops + mp_size; > - assert(old_ops_end > old_ops); > - > /* > - * UPSERT + UPSERT case: combine operations > + * Adding update operations. We keep order of update operations in > + * the array the same. It is vital since first set of operations > + * must be skipped in case upsert folds into insert. For instance: > + * old_ops = {{{op1}, {op2}}, {{op3}}} > + * new_ops = {{{op4}, {op5}}} > + * res_ops = {{{op1}, {op2}}, {{op3}}, {{op4}, {op5}}} > + * If upsert corresponding to old_ops becomes insert, then > + * {{op1}, {op2}} update operations are not applied. > */ > - assert(old_ops_end - old_ops > 0); > - if (vy_upsert_try_to_squash(format, result_mp, result_mp_end, > - old_ops, old_ops_end, new_ops, new_ops_end, > - &result_stmt) != 0) { > + uint32_t old_ops_cnt = mp_decode_array(&old_ops); > + uint32_t new_ops_cnt = mp_decode_array(&new_ops); > + size_t ops_size = sizeof(struct iovec) * (old_ops_cnt + new_ops_cnt); > + struct iovec *operations = region_alloc(region, ops_size); 16. region_alloc_array. 17. But you don't really need that. Nor upsert_ops_to_iovec() function. You could keep the old code almost as is, because for vy_stmt_new_with_ops() to work correctly, it is not necessary to have each operation set in a separate iovec. Anyway they are all copied as is without unpacking. You could have 1 iovec for the root MP_ARRAY, 1 iovec for the old operation sets, 1 iovec for the new operation sets. Having first iovec with root MP_ARRAY would allow to delete is_ops_encoded. > + if (operations == NULL) { > region_truncate(region, region_svp); > + diag_set(OutOfMemory, ops_size, "region_alloc", "operations"); > return NULL; > } > - if (result_stmt != NULL) { > - region_truncate(region, region_svp); > - vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); > - goto check_key; > - } > + upsert_ops_to_iovec(old_ops, old_ops_cnt, operations); > + upsert_ops_to_iovec(new_ops, new_ops_cnt, &operations[old_ops_cnt]); > > - /* Failed to squash, simply add one upsert to another */ > - int old_ops_cnt, new_ops_cnt; > - struct iovec operations[3]; > - > - old_ops_cnt = mp_decode_array(&old_ops); > - operations[1].iov_base = (void *)old_ops; > - operations[1].iov_len = old_ops_end - old_ops; > - > - new_ops_cnt = mp_decode_array(&new_ops); > - operations[2].iov_base = (void *)new_ops; > - operations[2].iov_len = new_ops_end - new_ops; > - > - char ops_buf[16]; > - char *header = mp_encode_array(ops_buf, old_ops_cnt + new_ops_cnt); > - operations[0].iov_base = (void *)ops_buf; > - operations[0].iov_len = header - ops_buf; > - > - result_stmt = vy_stmt_new_upsert(format, result_mp, result_mp_end, > - operations, 3); > + result_stmt = vy_stmt_new_upsert(format, old_stmt_mp, old_stmt_mp_end, > + operations, old_ops_cnt + new_ops_cnt, > + false); > region_truncate(region, region_svp); > if (result_stmt == NULL) > return NULL; > vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); > > -check_key: > - /* > - * Check that key hasn't been changed after applying operations. > - */ > - if (!key_update_can_be_skipped(cmp_def->column_mask, column_mask) && > - vy_stmt_compare(old_stmt, HINT_NONE, result_stmt, > - HINT_NONE, cmp_def) != 0) { > - /* > - * Key has been changed: ignore this UPSERT and > - * @retval the old stmt. > - */ > - tuple_unref(result_stmt); > - result_stmt = vy_stmt_dup(old_stmt); > - } > return result_stmt; > diff --git a/test/vinyl/upsert.result b/test/vinyl/upsert.result > index 3a7f6629d..a20db2ad2 100644 > --- a/test/vinyl/upsert.result > +++ b/test/vinyl/upsert.result > @@ -899,3 +899,292 @@ s:select() > s:drop() > --- > ... > +-- gh-5107: don't squash upsert operations into one array. > +-- > +-- Test upsert execution/squash referring to fields in reversed > +-- order (via negative indexing). > +-- > +s = box.schema.create_space('test', {engine = 'vinyl'}) > +--- > +... > +pk = s:create_index('pk') > +--- > +... > +s:insert({1, 1, 1}) > +--- > +- [1, 1, 1] > +... > +box.snapshot() > +--- > +- ok > +... > +s:upsert({1}, {{'=', 3, 100}}) > +--- > +... > +s:upsert({1}, {{'=', -1, 200}}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:select() -- {1, 1, 200} > +--- > +- - [1, 1, 200] > +... > +s:delete({1}) > +--- > +... > +s:insert({1, 1, 1}) > +--- > +- [1, 1, 1] > +... > +box.snapshot() > +--- > +- ok > +... > +s:upsert({1}, {{'=', -3, 100}}) > +--- > +... > +s:upsert({1}, {{'=', -1, 200}}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +-- Two upserts are NOT squashed into one, so only one > +-- (first one) is skipped, meanwhile second one is applied. > +-- > +s:select() -- {1, 1, 1} > +--- > +- - [1, 1, 200] > +... > +s:delete({1}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:upsert({1, 1}, {{'=', -2, 300}}) -- {1, 1} > +--- > +... > +s:upsert({1}, {{'+', -1, 100}}) -- {1, 101} > +--- > +... > +s:upsert({1}, {{'-', 2, 100}}) -- {1, 1} > +--- > +... > +s:upsert({1}, {{'+', -1, 200}}) -- {1, 201} > +--- > +... > +s:upsert({1}, {{'-', 2, 200}}) -- {1, 1} > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:select() -- {1, 1} > +--- > +- - [1, 1] > +... > +s:delete({1}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:upsert({1, 1, 1}, {{'!', -1, 300}}) -- {1, 1, 1} > +--- > +... > +s:upsert({1}, {{'+', -2, 100}}) -- {1, 101, 1} > +--- > +... > +s:upsert({1}, {{'=', -1, 100}}) -- {1, 101, 100} > +--- > +... > +s:upsert({1}, {{'+', -1, 200}}) -- {1, 101, 300} > +--- > +... > +s:upsert({1}, {{'-', -2, 100}}) -- {1, 1, 300} > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:select() > +--- > +- - [1, 1, 300] > +... > +s:drop() > +--- > +... > +-- Upsert operations which break space format are not applied. 18. You need to put references to the relevant issues in the tests, using -- -- gh-NNNN: description. -- format. > +-- > +s = box.schema.space.create('test', { engine = 'vinyl', field_count = 2 }) > +--- > +... > +pk = s:create_index('pk') > +--- > +... > +s:replace{1, 1} > +--- > +- [1, 1] > +... > +-- Error is logged, upsert is not applied. > +-- > +s:upsert({1, 1}, {{'=', 3, 5}}) > +--- > +... > +-- During read the incorrect upsert is ignored. > +-- > +s:select{} > +--- > +- - [1, 1] > +... > +-- Try to set incorrect field_count in a transaction. > +-- > +box.begin() > +--- > +... > +s:replace{2, 2} > +--- > +- [2, 2] > +... > +s:upsert({2, 2}, {{'=', 3, 2}}) > +--- > +... > +s:select{} > +--- > +- - [1, 1] > + - [2, 2] > +... > +box.commit() > +--- > +... > +s:select{} > +--- > +- - [1, 1] > + - [2, 2] > +... > +-- Read incorrect upsert from a run: it should be ignored. > +-- > +box.snapshot() > +--- > +- ok > +... > +s:select{} > +--- > +- - [1, 1] > + - [2, 2] > +... > +s:upsert({2, 2}, {{'=', 3, 20}}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:select{} > +--- > +- - [1, 1] > + - [2, 2] > +... > +-- Execute replace/delete after invalid upsert. > +-- > +box.snapshot() > +--- > +- ok > +... > +s:upsert({2, 2}, {{'=', 3, 30}}) > +--- > +... > +s:replace{2, 3} > +--- > +- [2, 3] > +... > +s:select{} > +--- > +- - [1, 1] > + - [2, 3] > +... > +s:upsert({1, 1}, {{'=', 3, 30}}) > +--- > +... > +s:delete{1} > +--- > +... > +s:select{} > +--- > +- - [2, 3] > +... > +-- Invalid upsert in a sequence of upserts is skipped meanwhile > +-- the rest are applied. > +-- > +box.snapshot() > +--- > +- ok > +... > +s:upsert({2, 2}, {{'+', 2, 5}}) > +--- > +... > +s:upsert({2, 2}, {{'=', 3, 40}}) > +--- > +... > +s:upsert({2, 2}, {{'+', 2, 5}}) > +--- > +... > +s:select{} > +--- > +- - [2, 13] > +... > +box.snapshot() > +--- > +- ok > +... > +s:select{} > +--- > +- - [2, 13] > +... > +s:drop() > +--- > +... > +-- Make sure upserts satisfy associativity rule. > +-- > +s = box.schema.space.create('test', {engine='vinyl'}) > +--- > +... > +i = s:create_index('pk', {parts={2, 'uint'}}) > +--- > +... > +s:replace{1, 2, 3, 'default'} > +--- > +- [1, 2, 3, 'default'] > +... > +box.snapshot() > +--- > +- ok > +... > +s:upsert({2, 2, 2}, {{'=', 4, 'upserted'}}) > +--- > +... > +-- Upsert will fail and thus ignored. > +-- > +s:upsert({2, 2, 2}, {{'#', 1, 1}, {'!', 3, 1}}) > +--- > +... > +box.snapshot() > +--- > +- ok > +... > +s:select{} > +--- > +- - [1, 2, 3, 'upserted'] > +... > +s:drop() > +--- > +... 19. All tests work with unsigned fields. So squashing is not tested here. From korablev at tarantool.org Mon Aug 3 11:42:35 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Mon, 3 Aug 2020 08:42:35 +0000 Subject: [Tarantool-patches] [PATCH] tuple: drop extra restrictions for multikey index In-Reply-To: <1596201198.417099977@f747.i.mail.ru> References: <20200724200633.12122-1-i.kosarev@tarantool.org> <20200730142620.GA25238@tarantool.org> <1596201198.417099977@f747.i.mail.ru> Message-ID: <20200803084235.GA8230@tarantool.org> On 31 Jul 16:13, Ilya Kosarev wrote: > > Hi! > ? > Thanks for your review. > ? > See my answers below. > ? > I will send v2 of the patch but i think 2 questions above?should be clarified. > ? > >???????, 30 ???? 2020, 17:26 +03:00 ?? Nikita Pettik : > >? > >On 24 Jul 23:06, Ilya Kosarev wrote: > > > >I *strongly dislike* that the patch is in fact should be 10 lines > >but instead we have 250 diff lines. Please elaborate on that > >(comments above are mostly about that). > Well, this patch reverts the majority of changes from > 4cf94ef8cb90b84ea71f313cff3e016f85894fd5 (tuple: make fields nullable > by default except array/map). That is where the size comes from. Then you are able simply to revert that patch and introduce new one. But I'd provide only required fixes. > >> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c > >> index 8452ab430..875592026 100644 > >> --- a/src/box/memtx_space.c > >> +++ b/src/box/memtx_space.c > >> @@ -589,9 +589,7 @@ memtx_space_ephemeral_rowid_next(struct space *space, uint64_t *rowid) > >> static int > >> memtx_space_check_index_def(struct space *space, struct index_def *index_def) > >> { > >> - struct key_def *key_def = index_def->key_def; > >> - > >> - if (key_def->is_nullable) { > >> + if (index_def->key_def->is_nullable) { > > > >Again I see this redundant diff. Please drop it, it doesn't fix > >refactor/fix anything but makes diff bigger and complicates git > >history. > The problem is that this is the reversion of that diff. Well, I guess > as far as that patch was pushed to master we can leave that change > there and revert only the meaningful part? Yes. From sergos at tarantool.org Mon Aug 3 23:53:28 2020 From: sergos at tarantool.org (sergos at tarantool.org) Date: Mon, 3 Aug 2020 23:53:28 +0300 Subject: [Tarantool-patches] [RFC] Raft over qsync Message-ID: <4E8CD3F1-803C-494A-AC81-B26968280E04@tarantool.org> * **Status**: In progress * **Start date**: 27-07-2020 * **Authors**: Sergey Ostanevich @sergos \, Vladislav Shpilevoy @Gerold103 \, Cyrill Gorcunov @cyrillos \ * **Issues**: https://github.com/tarantool/tarantool/issues/5202 ## Summary The #4842 is introduced a quorum based replication (qsync) in Tarantool environment. To augment the synchronous replication with automated leader election and failover, we need to make a choice on how to implement one of algorithms available. Our preference is Raft since it is has good comprehensibility. I would refer to the https://raft.github.io/raft.pdf further in the document. The biggest problem is to link together the master-master nature of log replication in Tarantool with the strong leader concept in Raft. ## Background and motivation Customers requested synchronous replication with automated failover as part of Tarantool development. These features also fits well with Tarantool future we envision in MRG. ## Detailed design Qsync implementation is performed in a way that allows users to run Tarantool-based solution without any changes to their code base. Literally if you have a database set up, it will continue running after upgrade the same way as it was prior to 2.5 version. You can start employing the synchronous replication by introduction of specific spaces in your schema and after that qsync with come to play. There were no changes to the protocol, so you can mix 2.5 instances with previous in both ways - replicate from new to old either vice-versa - until you introduce the first synchronous space. The machinery under the hood oblige all instances to follow a new process of transaction: if transaction touches a synchronous space then it will require a special command from the WAL - confirm. Since the obligation is an incremental requirement we can keep the backward compatibility and in both ways. We expect to elaborate similar approach to the Raft-based failover machinery. Which means one can use the qsync replication without the Raft enabled, being able to elaborate its own failover mechanism. Although, if Raft is enabled then all instances in the cluster are obliged to follow the rules implied by the Raft, such as ignore log entries from a leader with stale term number. ### Leader Election We expect the leader election algorithm can be reused from the Raft implementation [2] with additional mapping to the Tarantool replication mechanism. The replication mechanism of Tarantool with qsync provides the following features, that required by Raft algorithm: * The new entry is delivered to the majority of the replicas and only after this it is committed, the fact of commit is delivered to client * The log reflects the fact of an entry is committed by a special entry 'Commit' as part of qsync machinery * Log consistency comes from the append-only nature of the log and the vclock check-up during the log append * Log inconsistencies are handled during replication (see log replication below) Raft implementation will hold leader's term and it's vote for the current term in its local structures, while issuing WAL entries to reflect changes in the status will ensure the persistence of the Raft state. After a fail node will read the WAL entries and apply them to its runtime state. These entries should go into WAL under the ID number 0, so that they will not be propagated. To propagate the results of election there should be an entry in WAL with dedicated info: raft_term and leader_id. An elected leader should issue such an entry upon its election as a first entry. ### Log Replication The qsync RFC explains how we enforce the log replication in a way it is described in clause 5.3 of the [1]: committed entry always has a commit message in the xlog. Key difference here is that log entry index comprises of two parts: the LSN and the served ID. The follower's log consistency will be achieved during a) leader election, when follower will only vote for a candidate who has VCLOCK components greater or equal to follower's and b) during the join to a new leader, when follower will have an option to drop it's waiting queue (named limbo in qsync implementation), either perform a full rejoin. The latter is painful, still is the only way to follow the current representation of xlog that contains no replay info. There is a requirement in 5.1 of [1]: > If a server receives a request with a stale term number, it rejects the > request. that requires to introduce a machinery that will inspect every DML message from IPROTO to check if it satisfies the requirement. To introduce this machinery there should be an additional info put into the replicated DML map under IPROTO_RAFT_TERM: a term of the leader taken from the Raft runtime implementation. This info should be added to the DML - and checked - only in case the cluster is configured to use the Raft machinery. As a result an instance that is joining a cluster with the Raft protocol enabled has to enable the protocol either, otherwise it should be disconnected. ## Rationale and alternatives C implementation of Raft using binary protocol has an alternative of implementation using Lua, for example [3]. Although, the performance of the latter can have a significant impact in part of log replication enforcement. ## References * [1] https://raft.github.io/raft.pdf * [2] https://github.com/willemt/raft * [3] https://github.com/igorcoding/tarantool-raft From kostja.osipov at gmail.com Mon Aug 3 23:56:28 2020 From: kostja.osipov at gmail.com (Konstantin Osipov) Date: Mon, 3 Aug 2020 23:56:28 +0300 Subject: [Tarantool-patches] [RFC] Raft over qsync In-Reply-To: <4E8CD3F1-803C-494A-AC81-B26968280E04@tarantool.org> References: <4E8CD3F1-803C-494A-AC81-B26968280E04@tarantool.org> Message-ID: <20200803205628.GA387134@atlas> * sergos at tarantool.org [20/08/03 23:54]: willem is a very naive implementation, I'd recommend looking at etcd/raft instead. > * **Status**: In progress > * **Start date**: 27-07-2020 > * **Authors**: Sergey Ostanevich @sergos \, Vladislav Shpilevoy @Gerold103 \, Cyrill Gorcunov @cyrillos \ > * **Issues**: https://github.com/tarantool/tarantool/issues/5202 > > ## Summary > > The #4842 is introduced a quorum based replication (qsync) in Tarantool > environment. To augment the synchronous replication with automated leader > election and failover, we need to make a choice on how to implement one of > algorithms available. Our preference is Raft since it is has good > comprehensibility. I would refer to the https://raft.github.io/raft.pdf further > in the document. > > The biggest problem is to link together the master-master nature of log > replication in Tarantool with the strong leader concept in Raft. > > ## Background and motivation > > Customers requested synchronous replication with automated failover as part of > Tarantool development. These features also fits well with Tarantool future we > envision in MRG. > > ## Detailed design > > Qsync implementation is performed in a way that allows users to run > Tarantool-based solution without any changes to their code base. Literally if > you have a database set up, it will continue running after upgrade the same way > as it was prior to 2.5 version. You can start employing the synchronous > replication by introduction of specific spaces in your schema and after that > qsync with come to play. There were no changes to the protocol, so you can mix > 2.5 instances with previous in both ways - replicate from new to old either > vice-versa - until you introduce the first synchronous space. > > The machinery under the hood oblige all instances to follow a new process of > transaction: if transaction touches a synchronous space then it will require a > special command from the WAL - confirm. Since the obligation is an incremental > requirement we can keep the backward compatibility and in both ways. > > We expect to elaborate similar approach to the Raft-based failover machinery. > Which means one can use the qsync replication without the Raft enabled, being > able to elaborate its own failover mechanism. Although, if Raft is enabled then > all instances in the cluster are obliged to follow the rules implied by the > Raft, such as ignore log entries from a leader with stale term number. > > ### Leader Election > > We expect the leader election algorithm can be reused from the Raft > implementation [2] with additional mapping to the Tarantool replication > mechanism. > > The replication mechanism of Tarantool with qsync provides the following > features, that required by Raft algorithm: > * The new entry is delivered to the majority of the replicas and only after this > it is committed, the fact of commit is delivered to client > * The log reflects the fact of an entry is committed by a special entry 'Commit' > as part of qsync machinery > * Log consistency comes from the append-only nature of the log and the vclock > check-up during the log append > * Log inconsistencies are handled during replication (see log replication below) > > Raft implementation will hold leader's term and it's vote for the current term > in its local structures, while issuing WAL entries to reflect changes in the > status will ensure the persistence of the Raft state. After a fail node will > read the WAL entries and apply them to its runtime state. These entries should > go into WAL under the ID number 0, so that they will not be propagated. > > To propagate the results of election there should be an entry in WAL with > dedicated info: raft_term and leader_id. An elected leader should issue such an > entry upon its election as a first entry. > > > ### Log Replication > > The qsync RFC explains how we enforce the log replication in a way it is > described in clause 5.3 of the [1]: committed entry always has a commit message > in the xlog. Key difference here is that log entry index comprises of two parts: > the LSN and the served ID. The follower's log consistency will be achieved > during a) leader election, when follower will only vote for a candidate who has > VCLOCK components greater or equal to follower's and b) during the join to a new > leader, when follower will have an option to drop it's waiting queue (named > limbo in qsync implementation), either perform a full rejoin. > The latter is painful, still is the only way to follow the current > representation of xlog that contains no replay info. > > There is a requirement in 5.1 of [1]: > > > If a server receives a request with a stale term number, it rejects the > > request. > > that requires to introduce a machinery that will inspect every DML message from > IPROTO to check if it satisfies the requirement. To introduce this machinery > there should be an additional info put into the replicated DML map under > IPROTO_RAFT_TERM: a term of the leader taken from the Raft runtime > implementation. This info should be added to the DML - and checked - only in > case the cluster is configured to use the Raft machinery. > > As a result an instance that is joining a cluster with the Raft protocol enabled > has to enable the protocol either, otherwise it should be disconnected. > > ## Rationale and alternatives > > C implementation of Raft using binary protocol has an alternative of > implementation using Lua, for example [3]. Although, the performance of the > latter can have a significant impact in part of log replication enforcement. > > ## References > > * [1] https://raft.github.io/raft.pdf > * [2] https://github.com/willemt/raft > * [3] https://github.com/igorcoding/tarantool-raft -- Konstantin Osipov, Moscow, Russia From kyukhin at tarantool.org Tue Aug 4 10:28:41 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Tue, 4 Aug 2020 10:28:41 +0300 Subject: [Tarantool-patches] [PATCH v2 0/2] fiber: backport for stack madvise/mprotect errors handling In-Reply-To: References: Message-ID: <20200804072841.kehwy2prt2ymsjvz@tarantool.org> Hello, On 29 ??? 22:44, Ilya Kosarev wrote: > This patchset is a backport for > c67522973b32fae955835109d4f86eada8f67ae5 and > 8d53fadc0cb15a45ea0cd461d2d5243be51b37e0 patches on stack > madvise/mprotect errors handling. > > Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5211-diag-at-fiber-mprotect-and-madvise > Issue: https://github.com/tarantool/tarantool/issues/5211 LGTM. I've checked your patchset into 1.10. -- Regards, Kirill Yukhin From i.kosarev at tarantool.org Tue Aug 4 12:55:55 2020 From: i.kosarev at tarantool.org (=?UTF-8?B?SWx5YSBLb3NhcmV2?=) Date: Tue, 04 Aug 2020 12:55:55 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v2_1/3=5D_alter=3A_use_goo?= =?utf-8?q?d_c++_style?= In-Reply-To: <1594649887-15890-2-git-send-email-alyapunov@tarantool.org> References: <1594649887-15890-1-git-send-email-alyapunov@tarantool.org> <1594649887-15890-2-git-send-email-alyapunov@tarantool.org> Message-ID: <1596534955.779570951@f731.i.mail.ru> Hi! Yes, this?patch seems to be nice. ? See 1 comment below. >???????????, 13 ???? 2020, 17:18 +03:00 ?? Aleksandr Lyapunov : >? >We should use noexcept where it's appropriate. >1.It informs a developer he could not care about exception safety >2.It instructs a compiler to check throws if possible > >We should override for compile-time check of misprints in names >of methods and types of arguments. > >Part of #5153 >--- >?src/box/alter.cc | 180 +++++++++++++++++++++++++++---------------------------- >?1 file changed, 90 insertions(+), 90 deletions(-) > >diff --git a/src/box/alter.cc b/src/box/alter.cc >index 249cee2..15a51d0 100644 >--- a/src/box/alter.cc >+++ b/src/box/alter.cc >@@ -765,7 +765,7 @@ struct alter_space; >? >?class AlterSpaceOp { >?public: >- AlterSpaceOp(struct alter_space *alter); >+ explicit AlterSpaceOp(struct alter_space *alter); >? >? /** Link in alter_space::ops. */ >? struct rlist link; >@@ -774,7 +774,7 @@ public: >? * the space definition and/or key list that will be used >? * for creating the new space. Must not yield or fail. >? */ >- virtual void alter_def(struct alter_space * /* alter */) {} >+ virtual void alter_def(struct alter_space * /* alter */) noexcept {} >? /** >? * Called after creating a new space. Used for performing >? * long-lasting operations, such as index rebuild or format >@@ -788,21 +788,21 @@ public: >? * state to the new space (e.g. move unchanged indexes). >? * Must not yield or fail. >? */ >- virtual void alter(struct alter_space * /* alter */) {} >+ virtual void alter(struct alter_space * /* alter */) noexcept {} >? /** >? * Called after the change has been successfully written >? * to WAL. Must not fail. >? */ >? virtual void commit(struct alter_space * /* alter */, >- int64_t /* signature */) {} >+ int64_t /* signature */) noexcept {} >? /** >? * Called in case a WAL error occurred. It is supposed to undo >? * the effect of AlterSpaceOp::prepare and AlterSpaceOp::alter. >? * Must not fail. >? */ >- virtual void rollback(struct alter_space * /* alter */) {} >+ virtual void rollback(struct alter_space * /* alter */) noexcept {} >? >- virtual ~AlterSpaceOp() {} >+ virtual ~AlterSpaceOp() noexcept {} >? >? void *operator new(size_t size) >? { >@@ -984,7 +984,7 @@ struct mh_i32_t *AlterSpaceLock::registry; >??* Replace the old space with a new one in the space cache. >??*/ >?static int >-alter_space_commit(struct trigger *trigger, void *event) >+alter_space_commit(struct trigger *trigger, void *event) noexcept >?{ >? struct txn *txn = (struct txn *) event; >? struct alter_space *alter = (struct alter_space *) trigger->data; >@@ -1002,12 +1002,8 @@ alter_space_commit(struct trigger *trigger, void *event) >? * indexes into their new places. >? */ >? class AlterSpaceOp *op; >- try { >- rlist_foreach_entry(op, &alter->ops, link) >- op->commit(alter, signature); >- } catch (Exception *e) { >- return -1; >- } >+ rlist_foreach_entry(op, &alter->ops, link) >+ op->commit(alter, signature); I guess this change should be in ?alter: do not catch exceptions when it's not necessary? commit. >? >? alter->new_space = NULL; /* for alter_space_delete(). */ >? /* >@@ -1029,7 +1025,7 @@ alter_space_commit(struct trigger *trigger, void *event) >??* alter_space_commit() failure (unlikely) >??*/ >?static int >-alter_space_rollback(struct trigger *trigger, void * /* event */) >+alter_space_rollback(struct trigger *trigger, void * /* event */) noexcept >?{ >? struct alter_space *alter = (struct alter_space *) trigger->data; >? /* Rollback alter ops */ >@@ -1188,9 +1184,9 @@ alter_space_do(struct txn_stmt *stmt, struct alter_space *alter) >?class CheckSpaceFormat: public AlterSpaceOp >?{ >?public: >- CheckSpaceFormat(struct alter_space *alter) >+ explicit CheckSpaceFormat(struct alter_space *alter) >? :AlterSpaceOp(alter) {} >- virtual void prepare(struct alter_space *alter); >+ void prepare(struct alter_space *alter) override; >?}; >? >?void >@@ -1222,15 +1218,15 @@ public: >? * names into an old dictionary and deletes new one. >? */ >? struct tuple_dictionary *new_dict; >- virtual void alter_def(struct alter_space *alter); >- virtual void alter(struct alter_space *alter); >- virtual void rollback(struct alter_space *alter); >- virtual ~ModifySpace(); >+ void alter_def(struct alter_space *alter) noexcept override; >+ void alter(struct alter_space *alter) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >+ ~ModifySpace() noexcept override; >?}; >? >?/** Amend the definition of the new space. */ >?void >-ModifySpace::alter_def(struct alter_space *alter) >+ModifySpace::alter_def(struct alter_space *alter) noexcept >?{ >? /* >? * Use the old dictionary for the new space, because >@@ -1249,7 +1245,7 @@ ModifySpace::alter_def(struct alter_space *alter) >?} >? >?void >-ModifySpace::alter(struct alter_space *alter) >+ModifySpace::alter(struct alter_space *alter) noexcept >?{ >? /* >? * Move new names into an old dictionary, which already is >@@ -1260,12 +1256,12 @@ ModifySpace::alter(struct alter_space *alter) >?} >? >?void >-ModifySpace::rollback(struct alter_space *alter) >+ModifySpace::rollback(struct alter_space *alter) noexcept >?{ >? tuple_dictionary_swap(alter->new_space->def->dict, new_dict); >?} >? >-ModifySpace::~ModifySpace() >+ModifySpace::~ModifySpace() noexcept >?{ >? if (new_dict != NULL) >? tuple_dictionary_unref(new_dict); >@@ -1281,9 +1277,9 @@ public: >? DropIndex(struct alter_space *alter, struct index *index) >? :AlterSpaceOp(alter), old_index(index) {} >? struct index *old_index; >- virtual void alter_def(struct alter_space *alter); >- virtual void prepare(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t lsn); >+ void alter_def(struct alter_space *alter) noexcept override; >+ void prepare(struct alter_space *alter) override; >+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >?}; >? >?/* >@@ -1291,7 +1287,7 @@ public: >??* the new index from it. >??*/ >?void >-DropIndex::alter_def(struct alter_space * /* alter */) >+DropIndex::alter_def(struct alter_space * /* alter */) noexcept >?{ >? rlist_del_entry(old_index->def, link); >?} >@@ -1305,7 +1301,7 @@ DropIndex::prepare(struct alter_space *alter) >?} >? >?void >-DropIndex::commit(struct alter_space *alter, int64_t signature) >+DropIndex::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? (void)alter; >? index_commit_drop(old_index, signature); >@@ -1323,18 +1319,18 @@ public: >? :AlterSpaceOp(alter), iid(iid_arg) {} >? /** id of the index on the move. */ >? uint32_t iid; >- virtual void alter(struct alter_space *alter); >- virtual void rollback(struct alter_space *alter); >+ void alter(struct alter_space *alter) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >?}; >? >?void >-MoveIndex::alter(struct alter_space *alter) >+MoveIndex::alter(struct alter_space *alter) noexcept >?{ >? space_swap_index(alter->old_space, alter->new_space, iid, iid); >?} >? >?void >-MoveIndex::rollback(struct alter_space *alter) >+MoveIndex::rollback(struct alter_space *alter) noexcept >?{ >? space_swap_index(alter->old_space, alter->new_space, iid, iid); >?} >@@ -1365,23 +1361,23 @@ public: >? struct index *old_index; >? struct index *new_index; >? struct index_def *new_index_def; >- virtual void alter_def(struct alter_space *alter); >- virtual void alter(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t lsn); >- virtual void rollback(struct alter_space *alter); >- virtual ~ModifyIndex(); >+ void alter_def(struct alter_space *alter) noexcept override; >+ void alter(struct alter_space *alter) noexcept override; >+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >+ ~ModifyIndex() noexcept override; >?}; >? >?/** Update the definition of the new space */ >?void >-ModifyIndex::alter_def(struct alter_space *alter) >+ModifyIndex::alter_def(struct alter_space *alter) noexcept >?{ >? rlist_del_entry(old_index->def, link); >? index_def_list_add(&alter->key_list, new_index_def); >?} >? >?void >-ModifyIndex::alter(struct alter_space *alter) >+ModifyIndex::alter(struct alter_space *alter) noexcept >?{ >? new_index = space_index(alter->new_space, new_index_def->iid); >? assert(old_index->def->iid == new_index->def->iid); >@@ -1397,14 +1393,14 @@ ModifyIndex::alter(struct alter_space *alter) >?} >? >?void >-ModifyIndex::commit(struct alter_space *alter, int64_t signature) >+ModifyIndex::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? (void)alter; >? index_commit_modify(new_index, signature); >?} >? >?void >-ModifyIndex::rollback(struct alter_space *alter) >+ModifyIndex::rollback(struct alter_space *alter) noexcept >?{ >? /* >? * Restore indexes. >@@ -1416,7 +1412,7 @@ ModifyIndex::rollback(struct alter_space *alter) >? index_update_def(old_index); >?} >? >-ModifyIndex::~ModifyIndex() >+ModifyIndex::~ModifyIndex() noexcept >?{ >? index_def_delete(new_index_def); >?} >@@ -1432,15 +1428,15 @@ public: >? CreateIndex(struct alter_space *alter, struct index_def *def) >? :AlterSpaceOp(alter), new_index(NULL), new_index_def(def) >? {} >- virtual void alter_def(struct alter_space *alter); >- virtual void prepare(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t lsn); >- virtual ~CreateIndex(); >+ void alter_def(struct alter_space *alter) noexcept override; >+ void prepare(struct alter_space *alter) override; >+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >+ ~CreateIndex() noexcept override; >?}; >? >?/** Add definition of the new key to the new space def. */ >?void >-CreateIndex::alter_def(struct alter_space *alter) >+CreateIndex::alter_def(struct alter_space *alter) noexcept >?{ >? index_def_list_add(&alter->key_list, new_index_def); >?} >@@ -1480,7 +1476,7 @@ CreateIndex::prepare(struct alter_space *alter) >?} >? >?void >-CreateIndex::commit(struct alter_space *alter, int64_t signature) >+CreateIndex::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? (void) alter; >? assert(new_index != NULL); >@@ -1488,7 +1484,7 @@ CreateIndex::commit(struct alter_space *alter, int64_t signature) >? new_index = NULL; >?} >? >-CreateIndex::~CreateIndex() >+CreateIndex::~CreateIndex() noexcept >?{ >? if (new_index != NULL) >? index_abort_create(new_index); >@@ -1521,15 +1517,15 @@ public: >? struct index_def *new_index_def; >? /** Old index index_def. */ >? struct index_def *old_index_def; >- virtual void alter_def(struct alter_space *alter); >- virtual void prepare(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t signature); >- virtual ~RebuildIndex(); >+ void alter_def(struct alter_space *alter) noexcept override; >+ void prepare(struct alter_space *alter) override; >+ void commit(struct alter_space *alter, int64_t signature) noexcept override; >+ ~RebuildIndex() noexcept override; >?}; >? >?/** Add definition of the new key to the new space def. */ >?void >-RebuildIndex::alter_def(struct alter_space *alter) >+RebuildIndex::alter_def(struct alter_space *alter) noexcept >?{ >? rlist_del_entry(old_index_def, link); >? index_def_list_add(&alter->key_list, new_index_def); >@@ -1545,7 +1541,7 @@ RebuildIndex::prepare(struct alter_space *alter) >?} >? >?void >-RebuildIndex::commit(struct alter_space *alter, int64_t signature) >+RebuildIndex::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? struct index *old_index = space_index(alter->old_space, >? old_index_def->iid); >@@ -1556,7 +1552,7 @@ RebuildIndex::commit(struct alter_space *alter, int64_t signature) >? new_index = NULL; >?} >? >-RebuildIndex::~RebuildIndex() >+RebuildIndex::~RebuildIndex() noexcept >?{ >? if (new_index != NULL) >? index_abort_create(new_index); >@@ -1600,9 +1596,10 @@ public: >? TruncateIndex(struct alter_space *alter, uint32_t iid) >? : AlterSpaceOp(alter), iid(iid), >? old_index(NULL), new_index(NULL) {} >- virtual void prepare(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t signature); >- virtual ~TruncateIndex(); >+ void prepare(struct alter_space *alter) override; >+ void commit(struct alter_space *alter, >+ int64_t signature) noexcept override; >+ ~TruncateIndex() noexcept override; >?}; >? >?void >@@ -1632,7 +1629,7 @@ TruncateIndex::prepare(struct alter_space *alter) >?} >? >?void >-TruncateIndex::commit(struct alter_space *alter, int64_t signature) >+TruncateIndex::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? (void)alter; >? index_commit_drop(old_index, signature); >@@ -1640,7 +1637,7 @@ TruncateIndex::commit(struct alter_space *alter, int64_t signature) >? new_index = NULL; >?} >? >-TruncateIndex::~TruncateIndex() >+TruncateIndex::~TruncateIndex() noexcept >?{ >? if (new_index == NULL) >? return; >@@ -1657,11 +1654,11 @@ class UpdateSchemaVersion: public AlterSpaceOp >?public: >? UpdateSchemaVersion(struct alter_space * alter) >? :AlterSpaceOp(alter) {} >- virtual void alter(struct alter_space *alter); >+ void alter(struct alter_space *alter) noexcept override; >?}; >? >?void >-UpdateSchemaVersion::alter(struct alter_space *alter) >+UpdateSchemaVersion::alter(struct alter_space *alter) noexcept >?{ >?????(void)alter; >?????++schema_version; >@@ -1684,10 +1681,10 @@ public: >? RebuildCkConstraints(struct alter_space *alter) : AlterSpaceOp(alter), >? ck_constraint(RLIST_HEAD_INITIALIZER(ck_constraint)) {} >? struct rlist ck_constraint; >- virtual void prepare(struct alter_space *alter); >- virtual void alter(struct alter_space *alter); >- virtual void rollback(struct alter_space *alter); >- virtual ~RebuildCkConstraints(); >+ void prepare(struct alter_space *alter) override; >+ void alter(struct alter_space *alter) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >+ ~RebuildCkConstraints() noexcept override; >?}; >? >?void >@@ -1716,18 +1713,18 @@ RebuildCkConstraints::space_swap_ck_constraint(struct space *old_space, >?} >? >?void >-RebuildCkConstraints::alter(struct alter_space *alter) >+RebuildCkConstraints::alter(struct alter_space *alter) noexcept >?{ >? space_swap_ck_constraint(alter->old_space, alter->new_space); >?} >? >?void >-RebuildCkConstraints::rollback(struct alter_space *alter) >+RebuildCkConstraints::rollback(struct alter_space *alter) noexcept >?{ >? space_swap_ck_constraint(alter->new_space, alter->old_space); >?} >? >-RebuildCkConstraints::~RebuildCkConstraints() >+RebuildCkConstraints::~RebuildCkConstraints() noexcept >?{ >? struct ck_constraint *old_ck_constraint, *tmp; >? rlist_foreach_entry_safe(old_ck_constraint, &ck_constraint, link, tmp) { >@@ -1754,8 +1751,8 @@ class MoveCkConstraints: public AlterSpaceOp >? struct space *new_space); >?public: >? MoveCkConstraints(struct alter_space *alter) : AlterSpaceOp(alter) {} >- virtual void alter(struct alter_space *alter); >- virtual void rollback(struct alter_space *alter); >+ void alter(struct alter_space *alter) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >?}; >? >?void >@@ -1769,13 +1766,13 @@ MoveCkConstraints::space_swap_ck_constraint(struct space *old_space, >?} >? >?void >-MoveCkConstraints::alter(struct alter_space *alter) >+MoveCkConstraints::alter(struct alter_space *alter) noexcept >?{ >? space_swap_ck_constraint(alter->old_space, alter->new_space); >?} >? >?void >-MoveCkConstraints::rollback(struct alter_space *alter) >+MoveCkConstraints::rollback(struct alter_space *alter) noexcept >?{ >? space_swap_ck_constraint(alter->new_space, alter->old_space); >?} >@@ -1834,11 +1831,12 @@ public: >? if (new_id == NULL) >? diag_raise(); >? } >- virtual void prepare(struct alter_space *alter); >- virtual void alter(struct alter_space *alter); >- virtual void rollback(struct alter_space *alter); >- virtual void commit(struct alter_space *alter, int64_t signature); >- virtual ~CreateConstraintID(); >+ void prepare(struct alter_space *alter) override; >+ void alter(struct alter_space *alter) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >+ void commit(struct alter_space *alter, >+ int64_t signature) noexcept override; >+ ~CreateConstraintID() noexcept override; >?}; >? >?void >@@ -1850,7 +1848,7 @@ CreateConstraintID::prepare(struct alter_space *alter) >?} >? >?void >-CreateConstraintID::alter(struct alter_space *alter) >+CreateConstraintID::alter(struct alter_space *alter) noexcept >?{ >? /* Alter() can't fail, so can't just throw an error. */ >? if (space_add_constraint_id(alter->old_space, new_id) != 0) >@@ -1858,14 +1856,15 @@ CreateConstraintID::alter(struct alter_space *alter) >?} >? >?void >-CreateConstraintID::rollback(struct alter_space *alter) >+CreateConstraintID::rollback(struct alter_space *alter) noexcept >?{ >? space_delete_constraint_id(alter->new_space, new_id->name); >? new_id = NULL; >?} >? >?void >-CreateConstraintID::commit(struct alter_space *alter, int64_t signature) >+CreateConstraintID::commit(struct alter_space *alter, >+ int64_t signature) noexcept >?{ >? (void) alter; >? (void) signature; >@@ -1876,7 +1875,7 @@ CreateConstraintID::commit(struct alter_space *alter, int64_t signature) >? new_id = NULL; >?} >? >-CreateConstraintID::~CreateConstraintID() >+CreateConstraintID::~CreateConstraintID() noexcept >?{ >? if (new_id != NULL) >? constraint_id_delete(new_id); >@@ -1891,19 +1890,20 @@ public: >? DropConstraintID(struct alter_space *alter, const char *name) >? :AlterSpaceOp(alter), old_id(NULL), name(name) >? {} >- virtual void alter(struct alter_space *alter); >- virtual void commit(struct alter_space *alter , int64_t signature); >- virtual void rollback(struct alter_space *alter); >+ void alter(struct alter_space *alter) noexcept override; >+ void commit(struct alter_space *alter, >+ int64_t signature) noexcept override; >+ void rollback(struct alter_space *alter) noexcept override; >?}; >? >?void >-DropConstraintID::alter(struct alter_space *alter) >+DropConstraintID::alter(struct alter_space *alter) noexcept >?{ >? old_id = space_pop_constraint_id(alter->old_space, name); >?} >? >?void >-DropConstraintID::commit(struct alter_space *alter, int64_t signature) >+DropConstraintID::commit(struct alter_space *alter, int64_t signature) noexcept >?{ >? (void) alter; >? (void) signature; >@@ -1911,7 +1911,7 @@ DropConstraintID::commit(struct alter_space *alter, int64_t signature) >?} >? >?void >-DropConstraintID::rollback(struct alter_space *alter) >+DropConstraintID::rollback(struct alter_space *alter) noexcept >?{ >? if (space_add_constraint_id(alter->new_space, old_id) != 0) { >? panic("Can't recover after constraint drop rollback (out of " >-- >2.7.4 >? ? ? -- Ilya Kosarev ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From i.kosarev at tarantool.org Tue Aug 4 13:04:04 2020 From: i.kosarev at tarantool.org (=?UTF-8?B?SWx5YSBLb3NhcmV2?=) Date: Tue, 04 Aug 2020 13:04:04 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v2_1/3=5D_alter=3A_use_goo?= =?utf-8?q?d_c++_style?= In-Reply-To: <1596534955.779570951@f731.i.mail.ru> References: <1594649887-15890-1-git-send-email-alyapunov@tarantool.org> <1594649887-15890-2-git-send-email-alyapunov@tarantool.org> <1596534955.779570951@f731.i.mail.ru> Message-ID: <1596535444.410127167@f712.i.mail.ru> Also there are 4 places where tabs were replaced by spaces, marked them below. >???????, 4 ??????? 2020, 12:55 +03:00 ?? Ilya Kosarev : >? >Hi! > >Yes, this?patch seems to be nice. >? >See 1 comment below. >>???????????, 13 ???? 2020, 17:18 +03:00 ?? Aleksandr Lyapunov < alyapunov at tarantool.org >: >>? >>We should use noexcept where it's appropriate. >>1.It informs a developer he could not care about exception safety >>2.It instructs a compiler to check throws if possible >> >>We should override for compile-time check of misprints in names >>of methods and types of arguments. >> >>Part of #5153 >>--- >>?src/box/alter.cc | 180 +++++++++++++++++++++++++++---------------------------- >>?1 file changed, 90 insertions(+), 90 deletions(-) >> >>diff --git a/src/box/alter.cc b/src/box/alter.cc >>index 249cee2..15a51d0 100644 >>--- a/src/box/alter.cc >>+++ b/src/box/alter.cc >>@@ -765,7 +765,7 @@ struct alter_space; >>? >>?class AlterSpaceOp { >>?public: >>- AlterSpaceOp(struct alter_space *alter); >>+ explicit AlterSpaceOp(struct alter_space *alter); >>? >>? /** Link in alter_space::ops. */ >>? struct rlist link; >>@@ -774,7 +774,7 @@ public: >>? * the space definition and/or key list that will be used >>? * for creating the new space. Must not yield or fail. >>? */ >>- virtual void alter_def(struct alter_space * /* alter */) {} >>+ virtual void alter_def(struct alter_space * /* alter */) noexcept {} >>? /** >>? * Called after creating a new space. Used for performing >>? * long-lasting operations, such as index rebuild or format >>@@ -788,21 +788,21 @@ public: >>? * state to the new space (e.g. move unchanged indexes). >>? * Must not yield or fail. >>? */ >>- virtual void alter(struct alter_space * /* alter */) {} >>+ virtual void alter(struct alter_space * /* alter */) noexcept {} >>? /** >>? * Called after the change has been successfully written >>? * to WAL. Must not fail. >>? */ >>? virtual void commit(struct alter_space * /* alter */, >>- int64_t /* signature */) {} >>+ int64_t /* signature */) noexcept {} >>? /** >>? * Called in case a WAL error occurred. It is supposed to undo >>? * the effect of AlterSpaceOp::prepare and AlterSpaceOp::alter. >>? * Must not fail. >>? */ >>- virtual void rollback(struct alter_space * /* alter */) {} >>+ virtual void rollback(struct alter_space * /* alter */) noexcept {} >>? >>- virtual ~AlterSpaceOp() {} >>+ virtual ~AlterSpaceOp() noexcept {} >>? >>? void *operator new(size_t size) >>? { >>@@ -984,7 +984,7 @@ struct mh_i32_t *AlterSpaceLock::registry; >>??* Replace the old space with a new one in the space cache. >>??*/ >>?static int >>-alter_space_commit(struct trigger *trigger, void *event) >>+alter_space_commit(struct trigger *trigger, void *event) noexcept >>?{ >>? struct txn *txn = (struct txn *) event; >>? struct alter_space *alter = (struct alter_space *) trigger->data; >>@@ -1002,12 +1002,8 @@ alter_space_commit(struct trigger *trigger, void *event) >>? * indexes into their new places. >>? */ >>? class AlterSpaceOp *op; >>- try { >>- rlist_foreach_entry(op, &alter->ops, link) >>- op->commit(alter, signature); >>- } catch (Exception *e) { >>- return -1; >>- } >>+ rlist_foreach_entry(op, &alter->ops, link) >>+ op->commit(alter, signature); >I guess this change should be in >?alter: do not catch exceptions when it's not necessary? commit. >>? >>? alter->new_space = NULL; /* for alter_space_delete(). */ >>? /* >>@@ -1029,7 +1025,7 @@ alter_space_commit(struct trigger *trigger, void *event) >>??* alter_space_commit() failure (unlikely) >>??*/ >>?static int >>-alter_space_rollback(struct trigger *trigger, void * /* event */) >>+alter_space_rollback(struct trigger *trigger, void * /* event */) noexcept >>?{ >>? struct alter_space *alter = (struct alter_space *) trigger->data; >>? /* Rollback alter ops */ >>@@ -1188,9 +1184,9 @@ alter_space_do(struct txn_stmt *stmt, struct alter_space *alter) >>?class CheckSpaceFormat: public AlterSpaceOp >>?{ >>?public: >>- CheckSpaceFormat(struct alter_space *alter) >>+ explicit CheckSpaceFormat(struct alter_space *alter) >>? :AlterSpaceOp(alter) {} >>- virtual void prepare(struct alter_space *alter); >>+ void prepare(struct alter_space *alter) override; >>?}; >>? >>?void >>@@ -1222,15 +1218,15 @@ public: >>? * names into an old dictionary and deletes new one. >>? */ >>? struct tuple_dictionary *new_dict; >>- virtual void alter_def(struct alter_space *alter); >>- virtual void alter(struct alter_space *alter); >>- virtual void rollback(struct alter_space *alter); >>- virtual ~ModifySpace(); >>+ void alter_def(struct alter_space *alter) noexcept override; >>+ void alter(struct alter_space *alter) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>+ ~ModifySpace() noexcept override; >>?}; >>? >>?/** Amend the definition of the new space. */ >>?void >>-ModifySpace::alter_def(struct alter_space *alter) >>+ModifySpace::alter_def(struct alter_space *alter) noexcept >>?{ >>? /* >>? * Use the old dictionary for the new space, because >>@@ -1249,7 +1245,7 @@ ModifySpace::alter_def(struct alter_space *alter) >>?} >>? >>?void >>-ModifySpace::alter(struct alter_space *alter) >>+ModifySpace::alter(struct alter_space *alter) noexcept >>?{ >>? /* >>? * Move new names into an old dictionary, which already is >>@@ -1260,12 +1256,12 @@ ModifySpace::alter(struct alter_space *alter) >>?} >>? >>?void >>-ModifySpace::rollback(struct alter_space *alter) >>+ModifySpace::rollback(struct alter_space *alter) noexcept >>?{ >>? tuple_dictionary_swap(alter->new_space->def->dict, new_dict); >>?} >>? >>-ModifySpace::~ModifySpace() >>+ModifySpace::~ModifySpace() noexcept >>?{ >>? if (new_dict != NULL) >>? tuple_dictionary_unref(new_dict); >>@@ -1281,9 +1277,9 @@ public: >>? DropIndex(struct alter_space *alter, struct index *index) >>? :AlterSpaceOp(alter), old_index(index) {} >>? struct index *old_index; >>- virtual void alter_def(struct alter_space *alter); >>- virtual void prepare(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t lsn); >>+ void alter_def(struct alter_space *alter) noexcept override; >>+ void prepare(struct alter_space *alter) override; >>+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >>?}; >>? >>?/* >>@@ -1291,7 +1287,7 @@ public: >>??* the new index from it. >>??*/ >>?void >>-DropIndex::alter_def(struct alter_space * /* alter */) >>+DropIndex::alter_def(struct alter_space * /* alter */) noexcept >>?{ >>? rlist_del_entry(old_index->def, link); >>?} >>@@ -1305,7 +1301,7 @@ DropIndex::prepare(struct alter_space *alter) >>?} >>? >>?void >>-DropIndex::commit(struct alter_space *alter, int64_t signature) >>+DropIndex::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? (void)alter; >>? index_commit_drop(old_index, signature); >>@@ -1323,18 +1319,18 @@ public: >>? :AlterSpaceOp(alter), iid(iid_arg) {} >>? /** id of the index on the move. */ >>? uint32_t iid; >>- virtual void alter(struct alter_space *alter); >>- virtual void rollback(struct alter_space *alter); >>+ void alter(struct alter_space *alter) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>?}; >>? >>?void >>-MoveIndex::alter(struct alter_space *alter) >>+MoveIndex::alter(struct alter_space *alter) noexcept >>?{ >>? space_swap_index(alter->old_space, alter->new_space, iid, iid); >>?} >>? >>?void >>-MoveIndex::rollback(struct alter_space *alter) >>+MoveIndex::rollback(struct alter_space *alter) noexcept >>?{ >>? space_swap_index(alter->old_space, alter->new_space, iid, iid); >>?} >>@@ -1365,23 +1361,23 @@ public: >>? struct index *old_index; >>? struct index *new_index; >>? struct index_def *new_index_def; >>- virtual void alter_def(struct alter_space *alter); >>- virtual void alter(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t lsn); >>- virtual void rollback(struct alter_space *alter); >>- virtual ~ModifyIndex(); >>+ void alter_def(struct alter_space *alter) noexcept override; >>+ void alter(struct alter_space *alter) noexcept override; >>+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>+ ~ModifyIndex() noexcept override; >>?}; >>? >>?/** Update the definition of the new space */ >>?void >>-ModifyIndex::alter_def(struct alter_space *alter) >>+ModifyIndex::alter_def(struct alter_space *alter) noexcept >>?{ >>? rlist_del_entry(old_index->def, link); >>? index_def_list_add(&alter->key_list, new_index_def); >>?} >>? >>?void >>-ModifyIndex::alter(struct alter_space *alter) >>+ModifyIndex::alter(struct alter_space *alter) noexcept >>?{ >>? new_index = space_index(alter->new_space, new_index_def->iid); >>? assert(old_index->def->iid == new_index->def->iid); >>@@ -1397,14 +1393,14 @@ ModifyIndex::alter(struct alter_space *alter) >>?} >>? >>?void >>-ModifyIndex::commit(struct alter_space *alter, int64_t signature) >>+ModifyIndex::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? (void)alter; >>? index_commit_modify(new_index, signature); >>?} >>? >>?void >>-ModifyIndex::rollback(struct alter_space *alter) >>+ModifyIndex::rollback(struct alter_space *alter) noexcept >>?{ >>? /* >>? * Restore indexes. >>@@ -1416,7 +1412,7 @@ ModifyIndex::rollback(struct alter_space *alter) >>? index_update_def(old_index); >>?} >>? >>-ModifyIndex::~ModifyIndex() >>+ModifyIndex::~ModifyIndex() noexcept >>?{ >>? index_def_delete(new_index_def); >>?} >>@@ -1432,15 +1428,15 @@ public: >>? CreateIndex(struct alter_space *alter, struct index_def *def) >>? :AlterSpaceOp(alter), new_index(NULL), new_index_def(def) >>? {} >>- virtual void alter_def(struct alter_space *alter); >>- virtual void prepare(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t lsn); >>- virtual ~CreateIndex(); >>+ void alter_def(struct alter_space *alter) noexcept override; >>+ void prepare(struct alter_space *alter) override; >>+ void commit(struct alter_space *alter, int64_t lsn) noexcept override; >>+ ~CreateIndex() noexcept override; >>?}; >>? >>?/** Add definition of the new key to the new space def. */ >>?void >>-CreateIndex::alter_def(struct alter_space *alter) >>+CreateIndex::alter_def(struct alter_space *alter) noexcept >>?{ >>? index_def_list_add(&alter->key_list, new_index_def); >>?} >>@@ -1480,7 +1476,7 @@ CreateIndex::prepare(struct alter_space *alter) >>?} >>? >>?void >>-CreateIndex::commit(struct alter_space *alter, int64_t signature) >>+CreateIndex::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? (void) alter; >>? assert(new_index != NULL); >>@@ -1488,7 +1484,7 @@ CreateIndex::commit(struct alter_space *alter, int64_t signature) >>? new_index = NULL; >>?} >>? >>-CreateIndex::~CreateIndex() >>+CreateIndex::~CreateIndex() noexcept >>?{ >>? if (new_index != NULL) >>? index_abort_create(new_index); >>@@ -1521,15 +1517,15 @@ public: >>? struct index_def *new_index_def; >>? /** Old index index_def. */ >>? struct index_def *old_index_def; >>- virtual void alter_def(struct alter_space *alter); >>- virtual void prepare(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t signature); >>- virtual ~RebuildIndex(); >>+ void alter_def(struct alter_space *alter) noexcept override; >>+ void prepare(struct alter_space *alter) override; >>+ void commit(struct alter_space *alter, int64_t signature) noexcept override; >>+ ~RebuildIndex() noexcept override; >>?}; >>? >>?/** Add definition of the new key to the new space def. */ >>?void >>-RebuildIndex::alter_def(struct alter_space *alter) >>+RebuildIndex::alter_def(struct alter_space *alter) noexcept >>?{ >>? rlist_del_entry(old_index_def, link); >>? index_def_list_add(&alter->key_list, new_index_def); >>@@ -1545,7 +1541,7 @@ RebuildIndex::prepare(struct alter_space *alter) >>?} >>? >>?void >>-RebuildIndex::commit(struct alter_space *alter, int64_t signature) >>+RebuildIndex::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? struct index *old_index = space_index(alter->old_space, >>? old_index_def->iid); >>@@ -1556,7 +1552,7 @@ RebuildIndex::commit(struct alter_space *alter, int64_t signature) >>? new_index = NULL; >>?} >>? >>-RebuildIndex::~RebuildIndex() >>+RebuildIndex::~RebuildIndex() noexcept >>?{ >>? if (new_index != NULL) >>? index_abort_create(new_index); >>@@ -1600,9 +1596,10 @@ public: >>? TruncateIndex(struct alter_space *alter, uint32_t iid) >>? : AlterSpaceOp(alter), iid(iid), >>? old_index(NULL), new_index(NULL) {} >>- virtual void prepare(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t signature); >>- virtual ~TruncateIndex(); >>+ void prepare(struct alter_space *alter) override; >>+ void commit(struct alter_space *alter, >>+ int64_t signature) noexcept override; * Here on a new line. >>+ ~TruncateIndex() noexcept override; >>?}; >>? >>?void >>@@ -1632,7 +1629,7 @@ TruncateIndex::prepare(struct alter_space *alter) >>?} >>? >>?void >>-TruncateIndex::commit(struct alter_space *alter, int64_t signature) >>+TruncateIndex::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? (void)alter; >>? index_commit_drop(old_index, signature); >>@@ -1640,7 +1637,7 @@ TruncateIndex::commit(struct alter_space *alter, int64_t signature) >>? new_index = NULL; >>?} >>? >>-TruncateIndex::~TruncateIndex() >>+TruncateIndex::~TruncateIndex() noexcept >>?{ >>? if (new_index == NULL) >>? return; >>@@ -1657,11 +1654,11 @@ class UpdateSchemaVersion: public AlterSpaceOp >>?public: >>? UpdateSchemaVersion(struct alter_space * alter) >>? :AlterSpaceOp(alter) {} >>- virtual void alter(struct alter_space *alter); >>+ void alter(struct alter_space *alter) noexcept override; >>?}; >>? >>?void >>-UpdateSchemaVersion::alter(struct alter_space *alter) >>+UpdateSchemaVersion::alter(struct alter_space *alter) noexcept >>?{ >>?????(void)alter; >>?????++schema_version; >>@@ -1684,10 +1681,10 @@ public: >>? RebuildCkConstraints(struct alter_space *alter) : AlterSpaceOp(alter), >>? ck_constraint(RLIST_HEAD_INITIALIZER(ck_constraint)) {} >>? struct rlist ck_constraint; >>- virtual void prepare(struct alter_space *alter); >>- virtual void alter(struct alter_space *alter); >>- virtual void rollback(struct alter_space *alter); >>- virtual ~RebuildCkConstraints(); >>+ void prepare(struct alter_space *alter) override; >>+ void alter(struct alter_space *alter) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>+ ~RebuildCkConstraints() noexcept override; >>?}; >>? >>?void >>@@ -1716,18 +1713,18 @@ RebuildCkConstraints::space_swap_ck_constraint(struct space *old_space, >>?} >>? >>?void >>-RebuildCkConstraints::alter(struct alter_space *alter) >>+RebuildCkConstraints::alter(struct alter_space *alter) noexcept >>?{ >>? space_swap_ck_constraint(alter->old_space, alter->new_space); >>?} >>? >>?void >>-RebuildCkConstraints::rollback(struct alter_space *alter) >>+RebuildCkConstraints::rollback(struct alter_space *alter) noexcept >>?{ >>? space_swap_ck_constraint(alter->new_space, alter->old_space); >>?} >>? >>-RebuildCkConstraints::~RebuildCkConstraints() >>+RebuildCkConstraints::~RebuildCkConstraints() noexcept >>?{ >>? struct ck_constraint *old_ck_constraint, *tmp; >>? rlist_foreach_entry_safe(old_ck_constraint, &ck_constraint, link, tmp) { >>@@ -1754,8 +1751,8 @@ class MoveCkConstraints: public AlterSpaceOp >>? struct space *new_space); >>?public: >>? MoveCkConstraints(struct alter_space *alter) : AlterSpaceOp(alter) {} >>- virtual void alter(struct alter_space *alter); >>- virtual void rollback(struct alter_space *alter); >>+ void alter(struct alter_space *alter) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>?}; >>? >>?void >>@@ -1769,13 +1766,13 @@ MoveCkConstraints::space_swap_ck_constraint(struct space *old_space, >>?} >>? >>?void >>-MoveCkConstraints::alter(struct alter_space *alter) >>+MoveCkConstraints::alter(struct alter_space *alter) noexcept >>?{ >>? space_swap_ck_constraint(alter->old_space, alter->new_space); >>?} >>? >>?void >>-MoveCkConstraints::rollback(struct alter_space *alter) >>+MoveCkConstraints::rollback(struct alter_space *alter) noexcept >>?{ >>? space_swap_ck_constraint(alter->new_space, alter->old_space); >>?} >>@@ -1834,11 +1831,12 @@ public: >>? if (new_id == NULL) >>? diag_raise(); >>? } >>- virtual void prepare(struct alter_space *alter); >>- virtual void alter(struct alter_space *alter); >>- virtual void rollback(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter, int64_t signature); >>- virtual ~CreateConstraintID(); >>+ void prepare(struct alter_space *alter) override; >>+ void alter(struct alter_space *alter) noexcept override; >>+ void rollback(struct alter_space *alter) noexcept override; >>+ void commit(struct alter_space *alter, >>+ int64_t signature) noexcept override; * Here on a new line. >>+ ~CreateConstraintID() noexcept override; >>?}; >>? >>?void >>@@ -1850,7 +1848,7 @@ CreateConstraintID::prepare(struct alter_space *alter) >>?} >>? >>?void >>-CreateConstraintID::alter(struct alter_space *alter) >>+CreateConstraintID::alter(struct alter_space *alter) noexcept >>?{ >>? /* Alter() can't fail, so can't just throw an error. */ >>? if (space_add_constraint_id(alter->old_space, new_id) != 0) >>@@ -1858,14 +1856,15 @@ CreateConstraintID::alter(struct alter_space *alter) >>?} >>? >>?void >>-CreateConstraintID::rollback(struct alter_space *alter) >>+CreateConstraintID::rollback(struct alter_space *alter) noexcept >>?{ >>? space_delete_constraint_id(alter->new_space, new_id->name); >>? new_id = NULL; >>?} >>? >>?void >>-CreateConstraintID::commit(struct alter_space *alter, int64_t signature) >>+CreateConstraintID::commit(struct alter_space *alter, >>+ int64_t signature) noexcept * Here on a new line. >>?{ >>? (void) alter; >>? (void) signature; >>@@ -1876,7 +1875,7 @@ CreateConstraintID::commit(struct alter_space *alter, int64_t signature) >>? new_id = NULL; >>?} >>? >>-CreateConstraintID::~CreateConstraintID() >>+CreateConstraintID::~CreateConstraintID() noexcept >>?{ >>? if (new_id != NULL) >>? constraint_id_delete(new_id); >>@@ -1891,19 +1890,20 @@ public: >>? DropConstraintID(struct alter_space *alter, const char *name) >>? :AlterSpaceOp(alter), old_id(NULL), name(name) >>? {} >>- virtual void alter(struct alter_space *alter); >>- virtual void commit(struct alter_space *alter , int64_t signature); >>- virtual void rollback(struct alter_space *alter); >>+ void alter(struct alter_space *alter) noexcept override; >>+ void commit(struct alter_space *alter, >>+ int64_t signature) noexcept override; * Here on a new line. >>+ void rollback(struct alter_space *alter) noexcept override; >>?}; >>? >>?void >>-DropConstraintID::alter(struct alter_space *alter) >>+DropConstraintID::alter(struct alter_space *alter) noexcept >>?{ >>? old_id = space_pop_constraint_id(alter->old_space, name); >>?} >>? >>?void >>-DropConstraintID::commit(struct alter_space *alter, int64_t signature) >>+DropConstraintID::commit(struct alter_space *alter, int64_t signature) noexcept >>?{ >>? (void) alter; >>? (void) signature; >>@@ -1911,7 +1911,7 @@ DropConstraintID::commit(struct alter_space *alter, int64_t signature) >>?} >>? >>?void >>-DropConstraintID::rollback(struct alter_space *alter) >>+DropConstraintID::rollback(struct alter_space *alter) noexcept >>?{ >>? if (space_add_constraint_id(alter->new_space, old_id) != 0) { >>? panic("Can't recover after constraint drop rollback (out of " >>-- >>2.7.4 >>? >? >? >-- >Ilya Kosarev >? ? ? -- Ilya Kosarev ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From i.kosarev at tarantool.org Tue Aug 4 13:10:10 2020 From: i.kosarev at tarantool.org (=?UTF-8?B?SWx5YSBLb3NhcmV2?=) Date: Tue, 04 Aug 2020 13:10:10 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v2_2/3=5D_alter=3A_fix_cod?= =?utf-8?q?estyle?= In-Reply-To: <1594649887-15890-3-git-send-email-alyapunov@tarantool.org> References: <1594649887-15890-1-git-send-email-alyapunov@tarantool.org> <1594649887-15890-3-git-send-email-alyapunov@tarantool.org> Message-ID: <1596535810.142039609@f749.i.mail.ru> As far as i see, there are more places in alter.cc with wrong codestyle (spaces instead of tabs) but i guess those are the worst. ? >???????????, 13 ???? 2020, 17:18 +03:00 ?? Aleksandr Lyapunov : >? >--- >?src/box/alter.cc | 4 ++-- >?1 file changed, 2 insertions(+), 2 deletions(-) > >diff --git a/src/box/alter.cc b/src/box/alter.cc >index 15a51d0..9b2b8e8 100644 >--- a/src/box/alter.cc >+++ b/src/box/alter.cc >@@ -1660,8 +1660,8 @@ public: >?void >?UpdateSchemaVersion::alter(struct alter_space *alter) noexcept >?{ >- (void)alter; >- ++schema_version; >+ (void) alter; >+ ++schema_version; >?} >? >?/** >-- >2.7.4 >? ? ? -- Ilya Kosarev ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Wed Aug 5 00:04:11 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 4 Aug 2020 23:04:11 +0200 Subject: [Tarantool-patches] [PATCH vshard 1/1] storage: allow replica to boot before master Message-ID: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> Before the patch a replica couldn't call vshard.storage.cfg() before the master is bootstrapped. That could lead to an exception in the middle of vshard.storage.cfg(). It sounds not so bad and even looks unreachable, because replica bootstrap anyway is blocked in box.cfg() until a master node is started, otherwise the replica instance is terminated. But the box.cfg bootstrap != vshard bootstrap. It could be that both replica and master nodes are already bootstrapped and working, but vshard.storage.cfg() wasn't called yet. In that case vshard.storage.cfg() on the replica wouldn't block in box.cfg(), and would happily fail a few lines later in vshard.storage.cfg() at attempt to touch not existing box.space._bucket. That was the case for cartridge. The framework configures instances in its own way before vshard.storage.cfg(). Then it calls vshard.storage.cfg() on them in arbitrary order, and sometimes replicas were configured earlier than the master. The fail is fixed by simply skipping the _bucket's trigger installation on the replica node. Because so far it is not needed here for anything. The trigger's sole purpose is to increment bucket generation used only for DML on _bucket on the master node. Now vshard.storage.cfg() on replica is able to finish before master does the same. However the replica still won't be able to handle vshard.storage.* requests until receives vshard schema from master. Closes #237 --- Branch: http://github.com/tarantool/vshard/tree/gerold103/gh-237-boot-replica-first Issue: https://github.com/tarantool/vshard/issues/237 test/lua_libs/storage_template.lua | 55 +++++++++++- test/reload_evolution/storage.result | 10 +++ test/reload_evolution/storage.test.lua | 8 ++ test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ test/router/boot_replica_first.test.lua | 42 +++++++++ vshard/storage/init.lua | 18 +++- vshard/storage/reload_evolution.lua | 10 ++- 7 files changed, 251 insertions(+), 4 deletions(-) create mode 100644 test/router/boot_replica_first.result create mode 100644 test/router/boot_replica_first.test.lua diff --git a/test/lua_libs/storage_template.lua b/test/lua_libs/storage_template.lua index 29a753d..84e4180 100644 --- a/test/lua_libs/storage_template.lua +++ b/test/lua_libs/storage_template.lua @@ -1,5 +1,7 @@ #!/usr/bin/env tarantool +local luri = require('uri') + NAME = require('fio').basename(arg[0], '.lua') fiber = require('fiber') test_run = require('test_run').new() @@ -10,12 +12,63 @@ log = require('log') if not cfg.shard_index then cfg.shard_index = 'bucket_id' end +instance_uuid = util.name_to_uuid[NAME] + +-- +-- Bootstrap the instance exactly like vshard does. But don't +-- initialize any vshard-specific code. +-- +local function boot_like_vshard() + assert(type(box.cfg) == 'function') + for rs_uuid, rs in pairs(cfg.sharding) do + for replica_uuid, replica in pairs(rs.replicas) do + if replica_uuid == instance_uuid then + local box_cfg = {replication = {}} + box_cfg.instance_uuid = replica_uuid + box_cfg.replicaset_uuid = rs_uuid + box_cfg.listen = replica.uri + box_cfg.read_only = not replica.master + box_cfg.replication_connect_quorum = 0 + box_cfg.replication_timeout = 0.1 + for _, replica in pairs(rs.replicas) do + table.insert(box_cfg.replication, replica.uri) + end + box.cfg(box_cfg) + if not replica.master then + return + end + local uri = luri.parse(replica.uri) + box.schema.user.create(uri.login, { + password = uri.password, if_not_exists = true, + }) + box.schema.user.grant(uri.login, 'super') + return + end + end + end + assert(false) +end + +local omit_cfg = false +local i = 1 +while arg[i] ~= nil do + local key = arg[i] + i = i + 1 + if key == 'boot_before_cfg' then + boot_like_vshard() + omit_cfg = true + end +end vshard = require('vshard') echo_count = 0 cfg.replication_connect_timeout = 3 cfg.replication_timeout = 0.1 -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) + +if not omit_cfg then + vshard.storage.cfg(cfg, instance_uuid) +end + function bootstrap_storage(engine) box.once("testapp:schema:1", function() if rawget(_G, 'CHANGE_SPACE_IDS') then diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result index ebf4fdc..4652c4f 100644 --- a/test/reload_evolution/storage.result +++ b/test/reload_evolution/storage.result @@ -92,6 +92,16 @@ test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ... vshard.storage.internal.reload_version --- +- 2 +... +-- +-- gh-237: should be only one trigger. During gh-237 the trigger installation +-- became conditional and therefore required to remember the current trigger +-- somewhere. When reloaded from the old version, the trigger needed to be +-- fetched from _bucket:on_replace(). +-- +#box.space._bucket:on_replace() +--- - 1 ... -- Make sure storage operates well. diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua index 56c1693..06f7117 100644 --- a/test/reload_evolution/storage.test.lua +++ b/test/reload_evolution/storage.test.lua @@ -37,6 +37,14 @@ package.loaded['vshard.storage'] = nil vshard.storage = require("vshard.storage") test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil vshard.storage.internal.reload_version +-- +-- gh-237: should be only one trigger. During gh-237 the trigger installation +-- became conditional and therefore required to remember the current trigger +-- somewhere. When reloaded from the old version, the trigger needed to be +-- fetched from _bucket:on_replace(). +-- +#box.space._bucket:on_replace() + -- Make sure storage operates well. vshard.storage.bucket_force_drop(2000) vshard.storage.bucket_force_create(2000) diff --git a/test/router/boot_replica_first.result b/test/router/boot_replica_first.result new file mode 100644 index 0000000..1705230 --- /dev/null +++ b/test/router/boot_replica_first.result @@ -0,0 +1,112 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } + | --- + | ... +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) + | --- + | ... +util = require('util') + | --- + | ... +util.wait_master(test_run, REPLICASET_1, 'box_1_a') + | --- + | ... +_ = test_run:cmd("create server router with script='router/router_2.lua'") + | --- + | ... +_ = test_run:cmd("start server router") + | --- + | ... + +-- +-- gh-237: replica should be able to boot before master. Before the issue was +-- fixed, replica always tried to install a trigger on _bucket space even when +-- it was not created on a master yet - that lead to an exception in +-- storage.cfg. Now it should not install the trigger at all, because anyway it +-- is not needed on replica for anything. +-- + +test_run:switch('box_1_b') + | --- + | - true + | ... +vshard.storage.cfg(cfg, instance_uuid) + | --- + | ... +-- _bucket is not created yet. Will fail. +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) + | --- + | - attempt to index field '_bucket' (a nil value) + | ... + +test_run:switch('default') + | --- + | - true + | ... +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') + | --- + | ... + +test_run:switch('box_1_a') + | --- + | - true + | ... +vshard.storage.cfg(cfg, instance_uuid) + | --- + | ... + +test_run:switch('box_1_b') + | --- + | - true + | ... +test_run:wait_lsn('box_1_b', 'box_1_a') + | --- + | ... +-- Fails, but gracefully. Because the bucket is not found here. +vshard.storage.call(1, 'read', 'echo', {100}) + | --- + | - null + | - bucket_id: 1 + | reason: Not found + | code: 1 + | type: ShardingError + | message: 'Cannot perform action with bucket 1, reason: Not found' + | name: WRONG_BUCKET + | ... +-- Should not have triggers. +#box.space._bucket:on_replace() + | --- + | - 0 + | ... + +test_run:switch('router') + | --- + | - true + | ... +vshard.router.bootstrap() + | --- + | - true + | ... +vshard.router.callro(1, 'echo', {100}) + | --- + | - 100 + | ... + +test_run:switch("default") + | --- + | - true + | ... +test_run:cmd('stop server router') + | --- + | - true + | ... +test_run:cmd('delete server router') + | --- + | - true + | ... +test_run:drop_cluster(REPLICASET_1) + | --- + | ... diff --git a/test/router/boot_replica_first.test.lua b/test/router/boot_replica_first.test.lua new file mode 100644 index 0000000..7b1b3fd --- /dev/null +++ b/test/router/boot_replica_first.test.lua @@ -0,0 +1,42 @@ +test_run = require('test_run').new() +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) +util = require('util') +util.wait_master(test_run, REPLICASET_1, 'box_1_a') +_ = test_run:cmd("create server router with script='router/router_2.lua'") +_ = test_run:cmd("start server router") + +-- +-- gh-237: replica should be able to boot before master. Before the issue was +-- fixed, replica always tried to install a trigger on _bucket space even when +-- it was not created on a master yet - that lead to an exception in +-- storage.cfg. Now it should not install the trigger at all, because anyway it +-- is not needed on replica for anything. +-- + +test_run:switch('box_1_b') +vshard.storage.cfg(cfg, instance_uuid) +-- _bucket is not created yet. Will fail. +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) + +test_run:switch('default') +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') + +test_run:switch('box_1_a') +vshard.storage.cfg(cfg, instance_uuid) + +test_run:switch('box_1_b') +test_run:wait_lsn('box_1_b', 'box_1_a') +-- Fails, but gracefully. Because the bucket is not found here. +vshard.storage.call(1, 'read', 'echo', {100}) +-- Should not have triggers. +#box.space._bucket:on_replace() + +test_run:switch('router') +vshard.router.bootstrap() +vshard.router.callro(1, 'echo', {100}) + +test_run:switch("default") +test_run:cmd('stop server router') +test_run:cmd('delete server router') +test_run:drop_cluster(REPLICASET_1) diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index c6a78fe..ed577f9 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -94,6 +94,17 @@ if not M then -- detect that _bucket was not changed between yields. -- bucket_generation = 0, + -- + -- Reference to the function used as on_replace trigger on + -- _bucket space. It is used to replace the trigger with + -- a new function when reload happens. It is kept + -- explicitly because the old function is deleted on + -- reload from the global namespace. On the other hand, it + -- is still stored in _bucket:on_replace() somewhere, but + -- it is not known where. The only 100% way to be able to + -- replace the old function is to keep its reference. + -- + bucket_on_replace = nil, ------------------- Garbage collection ------------------- -- Fiber to remove garbage buckets data. @@ -2435,8 +2446,11 @@ local function storage_cfg(cfg, this_replica_uuid, is_reload) local uri = luri.parse(this_replica.uri) schema_upgrade(is_master, uri.login, uri.password) - local old_trigger = box.space._bucket:on_replace()[1] - box.space._bucket:on_replace(bucket_generation_increment, old_trigger) + if is_master then + local old_trigger = M.bucket_on_replace + box.space._bucket:on_replace(bucket_generation_increment, old_trigger) + M.bucket_on_replace = bucket_generation_increment + end lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) lreplicaset.outdate_replicasets(M.replicasets) diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua index 5d09b11..f38af74 100644 --- a/vshard/storage/reload_evolution.lua +++ b/vshard/storage/reload_evolution.lua @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) -- Code to update Lua objects. end +migrations[#migrations + 1] = function(M) + local bucket = box.space._bucket + if bucket then + assert(M.bucket_on_replace == nil) + M.bucket_on_replace = bucket:on_replace()[1] + end +end + -- -- Perform an update based on a version stored in `M` (internals). -- @param M Old module internals which should be updated. @@ -33,7 +41,7 @@ local function upgrade(M) log.error(err_msg) error(err_msg) end - for i = start_version, #migrations do + for i = start_version + 1, #migrations do local ok, err = pcall(migrations[i], M) if ok then log.info('vshard.storage.reload_evolution: upgraded to %d version', -- 2.21.1 (Apple Git-122.3) From yaroslav.dynnikov at tarantool.org Wed Aug 5 01:45:25 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Wed, 5 Aug 2020 01:45:25 +0300 Subject: [Tarantool-patches] [PATCH vshard 1/1] storage: allow replica to boot before master In-Reply-To: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> References: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> Message-ID: Hi, Vlad Thanks for the fix a lot. In general it looks good to me, but I've got few questions. Find them below. Best regards Yaroslav Dynnikov On Wed, 5 Aug 2020 at 00:04, Vladislav Shpilevoy wrote: > Before the patch a replica couldn't call vshard.storage.cfg() > before the master is bootstrapped. That could lead to an exception > in the middle of vshard.storage.cfg(). > > It sounds not so bad and even looks unreachable, because replica > bootstrap anyway is blocked in box.cfg() until a master node is > started, otherwise the replica instance is terminated. > But the box.cfg bootstrap != vshard bootstrap. > > It could be that both replica and master nodes are already > bootstrapped and working, but vshard.storage.cfg() wasn't called > yet. In that case vshard.storage.cfg() on the replica wouldn't > block in box.cfg(), and would happily fail a few lines later in > vshard.storage.cfg() at attempt to touch not existing > box.space._bucket. > > That was the case for cartridge. The framework configures > instances in its own way before vshard.storage.cfg(). Then it > calls vshard.storage.cfg() on them in arbitrary order, and > sometimes replicas were configured earlier than the master. > > The fail is fixed by simply skipping the _bucket's trigger > installation on the replica node. Because so far it is not needed > here for anything. The trigger's sole purpose is to increment > bucket generation used only for DML on _bucket on the master node. > > Now vshard.storage.cfg() on replica is able to finish before > master does the same. However the replica still won't be able to > handle vshard.storage.* requests until receives vshard schema > from master. > > Closes #237 > --- > Branch: > http://github.com/tarantool/vshard/tree/gerold103/gh-237-boot-replica-first > Issue: https://github.com/tarantool/vshard/issues/237 > > test/lua_libs/storage_template.lua | 55 +++++++++++- > test/reload_evolution/storage.result | 10 +++ > test/reload_evolution/storage.test.lua | 8 ++ > test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ > test/router/boot_replica_first.test.lua | 42 +++++++++ > vshard/storage/init.lua | 18 +++- > vshard/storage/reload_evolution.lua | 10 ++- > 7 files changed, 251 insertions(+), 4 deletions(-) > create mode 100644 test/router/boot_replica_first.result > create mode 100644 test/router/boot_replica_first.test.lua > > diff --git a/test/lua_libs/storage_template.lua > b/test/lua_libs/storage_template.lua > index 29a753d..84e4180 100644 > --- a/test/lua_libs/storage_template.lua > +++ b/test/lua_libs/storage_template.lua > @@ -1,5 +1,7 @@ > #!/usr/bin/env tarantool > > +local luri = require('uri') > + > NAME = require('fio').basename(arg[0], '.lua') > fiber = require('fiber') > test_run = require('test_run').new() > @@ -10,12 +12,63 @@ log = require('log') > if not cfg.shard_index then > cfg.shard_index = 'bucket_id' > end > +instance_uuid = util.name_to_uuid[NAME] > + > +-- > +-- Bootstrap the instance exactly like vshard does. But don't > +-- initialize any vshard-specific code. > +-- > +local function boot_like_vshard() > + assert(type(box.cfg) == 'function') > + for rs_uuid, rs in pairs(cfg.sharding) do > + for replica_uuid, replica in pairs(rs.replicas) do > + if replica_uuid == instance_uuid then > + local box_cfg = {replication = {}} > + box_cfg.instance_uuid = replica_uuid > + box_cfg.replicaset_uuid = rs_uuid > + box_cfg.listen = replica.uri > + box_cfg.read_only = not replica.master > + box_cfg.replication_connect_quorum = 0 > + box_cfg.replication_timeout = 0.1 > + for _, replica in pairs(rs.replicas) do > + table.insert(box_cfg.replication, replica.uri) > + end > + box.cfg(box_cfg) > + if not replica.master then > + return > + end > + local uri = luri.parse(replica.uri) > + box.schema.user.create(uri.login, { > + password = uri.password, if_not_exists = true, > + }) > + box.schema.user.grant(uri.login, 'super') > + return > + end > + end > + end > + assert(false) > Should there be some meaningful message? > +end > + > +local omit_cfg = false > +local i = 1 > +while arg[i] ~= nil do > + local key = arg[i] > + i = i + 1 > + if key == 'boot_before_cfg' then > + boot_like_vshard() > + omit_cfg = true > + end > +end > > vshard = require('vshard') > echo_count = 0 > cfg.replication_connect_timeout = 3 > cfg.replication_timeout = 0.1 > -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) > + > +if not omit_cfg then > + vshard.storage.cfg(cfg, instance_uuid) > +end > + > function bootstrap_storage(engine) > box.once("testapp:schema:1", function() > if rawget(_G, 'CHANGE_SPACE_IDS') then > diff --git a/test/reload_evolution/storage.result > b/test/reload_evolution/storage.result > index ebf4fdc..4652c4f 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -92,6 +92,16 @@ test_run:grep_log('storage_2_a', > 'vshard.storage.reload_evolution: upgraded to') > ... > vshard.storage.internal.reload_version > --- > +- 2 > +... > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger > installation > +-- became conditional and therefore required to remember the current > trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > +--- > - 1 > ... > -- Make sure storage operates well. > diff --git a/test/reload_evolution/storage.test.lua > b/test/reload_evolution/storage.test.lua > index 56c1693..06f7117 100644 > --- a/test/reload_evolution/storage.test.lua > +++ b/test/reload_evolution/storage.test.lua > @@ -37,6 +37,14 @@ package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') ~= nil > vshard.storage.internal.reload_version > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger > installation > +-- became conditional and therefore required to remember the current > trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > + > -- Make sure storage operates well. > vshard.storage.bucket_force_drop(2000) > vshard.storage.bucket_force_create(2000) > diff --git a/test/router/boot_replica_first.result > b/test/router/boot_replica_first.result > new file mode 100644 > index 0000000..1705230 > --- /dev/null > +++ b/test/router/boot_replica_first.result > @@ -0,0 +1,112 @@ > +-- test-run result file version 2 > +test_run = require('test_run').new() > + | --- > + | ... > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > + | --- > + | ... > +test_run:create_cluster(REPLICASET_1, 'router', {args = > 'boot_before_cfg'}) > + | --- > + | ... > +util = require('util') > + | --- > + | ... > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > + | --- > + | ... > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > + | --- > + | ... > +_ = test_run:cmd("start server router") > + | --- > + | ... > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue > was > +-- fixed, replica always tried to install a trigger on _bucket space even > when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because > anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + | --- > + | - attempt to index field '_bucket' (a nil value) > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + | --- > + | ... > + > +test_run:switch('box_1_a') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +test_run:wait_lsn('box_1_b', 'box_1_a') > + | --- > + | ... > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > + | --- > + | - null > + | - bucket_id: 1 > + | reason: Not found > + | code: 1 > + | type: ShardingError > + | message: 'Cannot perform action with bucket 1, reason: Not found' > + | name: WRONG_BUCKET > + | ... > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + | --- > + | - 0 > + | ... > + > +test_run:switch('router') > + | --- > + | - true > + | ... > +vshard.router.bootstrap() > + | --- > + | - true > + | ... > +vshard.router.callro(1, 'echo', {100}) > + | --- > + | - 100 > + | ... > + > +test_run:switch("default") > + | --- > + | - true > + | ... > +test_run:cmd('stop server router') > + | --- > + | - true > + | ... > +test_run:cmd('delete server router') > + | --- > + | - true > + | ... > +test_run:drop_cluster(REPLICASET_1) > + | --- > + | ... > diff --git a/test/router/boot_replica_first.test.lua > b/test/router/boot_replica_first.test.lua > new file mode 100644 > index 0000000..7b1b3fd > --- /dev/null > +++ b/test/router/boot_replica_first.test.lua > @@ -0,0 +1,42 @@ > +test_run = require('test_run').new() > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > +test_run:create_cluster(REPLICASET_1, 'router', {args = > 'boot_before_cfg'}) > +util = require('util') > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > +_ = test_run:cmd("start server router") > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue > was > +-- fixed, replica always tried to install a trigger on _bucket space even > when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because > anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > +vshard.storage.cfg(cfg, instance_uuid) > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + > +test_run:switch('default') > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + > +test_run:switch('box_1_a') > +vshard.storage.cfg(cfg, instance_uuid) > + > +test_run:switch('box_1_b') > +test_run:wait_lsn('box_1_b', 'box_1_a') > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + > +test_run:switch('router') > +vshard.router.bootstrap() > +vshard.router.callro(1, 'echo', {100}) > + > +test_run:switch("default") > +test_run:cmd('stop server router') > +test_run:cmd('delete server router') > +test_run:drop_cluster(REPLICASET_1) > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index c6a78fe..ed577f9 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -94,6 +94,17 @@ if not M then > -- detect that _bucket was not changed between yields. > -- > bucket_generation = 0, > + -- > + -- Reference to the function used as on_replace trigger on > + -- _bucket space. It is used to replace the trigger with > + -- a new function when reload happens. It is kept > + -- explicitly because the old function is deleted on > + -- reload from the global namespace. On the other hand, it > + -- is still stored in _bucket:on_replace() somewhere, but > + -- it is not known where. The only 100% way to be able to > + -- replace the old function is to keep its reference. > + -- > + bucket_on_replace = nil, > > ------------------- Garbage collection ------------------- > -- Fiber to remove garbage buckets data. > @@ -2435,8 +2446,11 @@ local function storage_cfg(cfg, this_replica_uuid, > is_reload) > local uri = luri.parse(this_replica.uri) > schema_upgrade(is_master, uri.login, uri.password) > > - local old_trigger = box.space._bucket:on_replace()[1] > - box.space._bucket:on_replace(bucket_generation_increment, old_trigger) > + if is_master then > + local old_trigger = M.bucket_on_replace > + box.space._bucket:on_replace(bucket_generation_increment, > old_trigger) > + M.bucket_on_replace = bucket_generation_increment > + end > > lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) > lreplicaset.outdate_replicasets(M.replicasets) > diff --git a/vshard/storage/reload_evolution.lua > b/vshard/storage/reload_evolution.lua > index 5d09b11..f38af74 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) > -- Code to update Lua objects. > end > > +migrations[#migrations + 1] = function(M) > + local bucket = box.space._bucket > + if bucket then > + assert(M.bucket_on_replace == nil) > + M.bucket_on_replace = bucket:on_replace()[1] > + end > +end > + > -- > -- Perform an update based on a version stored in `M` (internals). > -- @param M Old module internals which should be updated. > @@ -33,7 +41,7 @@ local function upgrade(M) > log.error(err_msg) > error(err_msg) > end > - for i = start_version, #migrations do > + for i = start_version + 1, #migrations do > Is it already tested somewhere else (migrations I mean)? This change looks like a fix for some other problem. > local ok, err = pcall(migrations[i], M) > if ok then > log.info('vshard.storage.reload_evolution: upgraded to %d > version', > -- > 2.21.1 (Apple Git-122.3) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Wed Aug 5 02:45:17 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 5 Aug 2020 01:45:17 +0200 Subject: [Tarantool-patches] [PATCH 0/2] JSON field multikey crash Message-ID: The patchset fixes 2 crashes related to multikey in JSON path tuple field access code. Also during working on this I found https://github.com/tarantool/tarantool/issues/5226, but couldn't find a simple solution. Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5224-tuple-field-by-path-crash Issue: https://github.com/tarantool/tarantool/issues/5224 @ChangeLog * Fixed a crash when JSON tuple field access was used to get a multikey indexed field, and when a JSON contained [*] in the beginning; Vladislav Shpilevoy (2): tuple: fix multikey field JSON access crash tuple: fix access by JSON path starting from '[*]' src/box/tuple.c | 3 +- src/box/tuple.h | 8 + test/box/gh-5224-multikey-field-access.result | 164 ++++++++++++++++++ .../gh-5224-multikey-field-access.test.lua | 72 ++++++++ 4 files changed, 246 insertions(+), 1 deletion(-) create mode 100644 test/box/gh-5224-multikey-field-access.result create mode 100644 test/box/gh-5224-multikey-field-access.test.lua -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Wed Aug 5 02:45:18 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 5 Aug 2020 01:45:18 +0200 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: References: Message-ID: When a tuple had format with multikey indexes in it, any attempt to get a multikey indexed field by a JSON path from Lua led to a crash. That was because of incorrect interpretation of offset slot value in tuple's field map. Tuple field map is an array stored before the tuple's MessagePack data. Each element is a 4 byte offset to an indexed value to be able to get it for O(1) time without MessagePack decoding of all the previous fields. At least it was so before multikeys. Now tuple field map is not just an array. It is rather a 2-level array, somehow similar to ext4 FS. Some elements of the root array are positive numbers pointing at data. Some elements point at a second 'indirect' array, so called 'extra', size of which is individual for each tuple. These second arrays are used by multikey indexes to store offsets to each multikey indexed value in a tuple. It means, that if there is an offset slot, it can't be just used as is. It is allowed only if the field is not multikey. Otherwise it is neccessary to somehow get an index in the second 'indirect' array. This is what was happening - a multikey field was found, its offset slot was valid, but it was pointing at an 'indirect' array, not at the data. JSON tuple field access tried to use it as a data offset. The patch makes JSON field access degrade to fullscan when a field is multikey, but no multikey array index is provided. Closes #5224 --- src/box/tuple.h | 8 + test/box/gh-5224-multikey-field-access.result | 155 ++++++++++++++++++ .../gh-5224-multikey-field-access.test.lua | 66 ++++++++ 3 files changed, 229 insertions(+) create mode 100644 test/box/gh-5224-multikey-field-access.result create mode 100644 test/box/gh-5224-multikey-field-access.test.lua diff --git a/src/box/tuple.h b/src/box/tuple.h index 4752323e4..2b48d6af9 100644 --- a/src/box/tuple.h +++ b/src/box/tuple.h @@ -628,6 +628,14 @@ tuple_field_raw_by_path(struct tuple_format *format, const char *tuple, goto parse; if (offset_slot_hint != NULL) *offset_slot_hint = offset_slot; + /* + * When the field is multikey, the offset slot points not at the + * data. It points at 'extra' array of offsets for this multikey + * index. That array can only be accessed if index in that array + * is known. + */ + if (field->is_multikey_part && multikey_idx == MULTIKEY_NONE) + goto parse; offset_slot_access: /* Indexed field */ offset = field_map_get_offset(field_map, offset_slot, diff --git a/test/box/gh-5224-multikey-field-access.result b/test/box/gh-5224-multikey-field-access.result new file mode 100644 index 000000000..014e810d0 --- /dev/null +++ b/test/box/gh-5224-multikey-field-access.result @@ -0,0 +1,155 @@ +-- test-run result file version 2 +-- +-- gh-5224: tuple field access by JSON path crashed when tried to get a multikey +-- indexed field. +-- +format = {} + | --- + | ... +format[1] = {name = 'f1', type = 'unsigned'} + | --- + | ... +format[2] = {name = 'f2', type = 'array'} + | --- + | ... +s = box.schema.create_space('test', {format = format}) + | --- + | ... +_ = s:create_index('pk') + | --- + | ... +_ = s:create_index('sk', { \ + parts = { \ + {field = 2, path = "[*].tags", type = "unsigned"} \ + } \ +}) + | --- + | ... + +t = s:replace{1, {{tags = 2}}} + | --- + | ... +t['[2][1].tags'] + | --- + | - 2 + | ... + +t = s:replace{1, {{tags = 2}, {tags = 3}, {tags = 4}}} + | --- + | ... +t['[2]'] + | --- + | - [{'tags': 2}, {'tags': 3}, {'tags': 4}] + | ... +t['[2][1]'] + | --- + | - {'tags': 2} + | ... +t['[2][1].tags'] + | --- + | - 2 + | ... +t['[2][2]'] + | --- + | - {'tags': 3} + | ... +t['[2][2].tags'] + | --- + | - 3 + | ... +t['[2][3]'] + | --- + | - {'tags': 4} + | ... +t['[2][3].tags'] + | --- + | - 4 + | ... + +s:truncate() + | --- + | ... +s.index.sk:drop() + | --- + | ... +_ = s:create_index('sk', { \ + parts = { \ + {field = 2, path = "[*].p1.p2", type = "unsigned"} \ + } \ +}) + | --- + | ... + +t = s:replace{1, {{p1 = {p2 = 2}}}} + | --- + | ... +t['[2][1].p1.p2'] + | --- + | - 2 + | ... + +t = s:replace{1, { \ + { \ + p1 = { \ + p2 = 2, p3 = 3 \ + }, \ + p4 = 4 \ + }, \ + { \ + p1 = {p2 = 5} \ + }, \ + { \ + p1 = {p2 = 6} \ + } \ +}} + | --- + | ... + +t['[2][1].p1.p2'] + | --- + | - 2 + | ... +t['[2][1].p1.p3'] + | --- + | - 3 + | ... +t['[2][1].p1'] + | --- + | - {'p2': 2, 'p3': 3} + | ... +t['[2][1]'] + | --- + | - {'p4': 4, 'p1': {'p2': 2, 'p3': 3}} + | ... +t['[2][1].p4'] + | --- + | - 4 + | ... +t['[2][2].p1.p2'] + | --- + | - 5 + | ... +t['[2][2].p1'] + | --- + | - {'p2': 5} + | ... +t['[2][2]'] + | --- + | - {'p1': {'p2': 5}} + | ... +t['[2][3].p1.p2'] + | --- + | - 6 + | ... +t['[2][3].p1'] + | --- + | - {'p2': 6} + | ... +t['[2][3]'] + | --- + | - {'p1': {'p2': 6}} + | ... + +s:drop() + | --- + | ... diff --git a/test/box/gh-5224-multikey-field-access.test.lua b/test/box/gh-5224-multikey-field-access.test.lua new file mode 100644 index 000000000..19765171e --- /dev/null +++ b/test/box/gh-5224-multikey-field-access.test.lua @@ -0,0 +1,66 @@ +-- +-- gh-5224: tuple field access by JSON path crashed when tried to get a multikey +-- indexed field. +-- +format = {} +format[1] = {name = 'f1', type = 'unsigned'} +format[2] = {name = 'f2', type = 'array'} +s = box.schema.create_space('test', {format = format}) +_ = s:create_index('pk') +_ = s:create_index('sk', { \ + parts = { \ + {field = 2, path = "[*].tags", type = "unsigned"} \ + } \ +}) + +t = s:replace{1, {{tags = 2}}} +t['[2][1].tags'] + +t = s:replace{1, {{tags = 2}, {tags = 3}, {tags = 4}}} +t['[2]'] +t['[2][1]'] +t['[2][1].tags'] +t['[2][2]'] +t['[2][2].tags'] +t['[2][3]'] +t['[2][3].tags'] + +s:truncate() +s.index.sk:drop() +_ = s:create_index('sk', { \ + parts = { \ + {field = 2, path = "[*].p1.p2", type = "unsigned"} \ + } \ +}) + +t = s:replace{1, {{p1 = {p2 = 2}}}} +t['[2][1].p1.p2'] + +t = s:replace{1, { \ + { \ + p1 = { \ + p2 = 2, p3 = 3 \ + }, \ + p4 = 4 \ + }, \ + { \ + p1 = {p2 = 5} \ + }, \ + { \ + p1 = {p2 = 6} \ + } \ +}} + +t['[2][1].p1.p2'] +t['[2][1].p1.p3'] +t['[2][1].p1'] +t['[2][1]'] +t['[2][1].p4'] +t['[2][2].p1.p2'] +t['[2][2].p1'] +t['[2][2]'] +t['[2][3].p1.p2'] +t['[2][3].p1'] +t['[2][3]'] + +s:drop() -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Wed Aug 5 02:45:19 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 5 Aug 2020 01:45:19 +0200 Subject: [Tarantool-patches] [PATCH 2/2] tuple: fix access by JSON path starting from '[*]' In-Reply-To: References: Message-ID: <21871bd97383aa16c15d8bdad20775f2a85548ae.1596584571.git.v.shpilevoy@tarantool.org> Tuple JSON field access crashed when '[*]' was used as a first part of the JSON path. The patch makes it treated like 'field not found'. Follow-up #5224 --- src/box/tuple.c | 3 ++- test/box/gh-5224-multikey-field-access.result | 9 +++++++++ test/box/gh-5224-multikey-field-access.test.lua | 6 ++++++ 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/src/box/tuple.c b/src/box/tuple.c index 9f0f24c64..e77753a5b 100644 --- a/src/box/tuple.c +++ b/src/box/tuple.c @@ -531,7 +531,8 @@ tuple_field_raw_by_full_path(struct tuple_format *format, const char *tuple, break; } default: - assert(token.type == JSON_TOKEN_END); + assert(token.type == JSON_TOKEN_END || + token.type == JSON_TOKEN_ANY); return NULL; } return tuple_field_raw_by_path(format, tuple, field_map, fieldno, diff --git a/test/box/gh-5224-multikey-field-access.result b/test/box/gh-5224-multikey-field-access.result index 014e810d0..20df44cc3 100644 --- a/test/box/gh-5224-multikey-field-access.result +++ b/test/box/gh-5224-multikey-field-access.result @@ -150,6 +150,15 @@ t['[2][3]'] | - {'p1': {'p2': 6}} | ... +-- +-- Multikey path part could crash when used as a first part of the path during +-- accessing a tuple field. +-- +t['[*]'] + | --- + | - null + | ... + s:drop() | --- | ... diff --git a/test/box/gh-5224-multikey-field-access.test.lua b/test/box/gh-5224-multikey-field-access.test.lua index 19765171e..6f6691d3c 100644 --- a/test/box/gh-5224-multikey-field-access.test.lua +++ b/test/box/gh-5224-multikey-field-access.test.lua @@ -63,4 +63,10 @@ t['[2][3].p1.p2'] t['[2][3].p1'] t['[2][3]'] +-- +-- Multikey path part could crash when used as a first part of the path during +-- accessing a tuple field. +-- +t['[*]'] + s:drop() -- 2.21.1 (Apple Git-122.3) From yaroslav.dynnikov at tarantool.org Wed Aug 5 12:46:57 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Wed, 5 Aug 2020 12:46:57 +0300 Subject: [Tarantool-patches] [PATCH vshard 1/1] storage: allow replica to boot before master In-Reply-To: References: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> Message-ID: And one more question. Best regards Yaroslav Dynnikov On Wed, 5 Aug 2020 at 01:45, Yaroslav Dynnikov < yaroslav.dynnikov at tarantool.org> wrote: > Hi, Vlad > > Thanks for the fix a lot. In general it looks good to me, but I've got few > questions. Find them below. > > Best regards > Yaroslav Dynnikov > > > On Wed, 5 Aug 2020 at 00:04, Vladislav Shpilevoy < > v.shpilevoy at tarantool.org> wrote: > >> Before the patch a replica couldn't call vshard.storage.cfg() >> before the master is bootstrapped. That could lead to an exception >> in the middle of vshard.storage.cfg(). >> >> It sounds not so bad and even looks unreachable, because replica >> bootstrap anyway is blocked in box.cfg() until a master node is >> started, otherwise the replica instance is terminated. >> But the box.cfg bootstrap != vshard bootstrap. >> >> It could be that both replica and master nodes are already >> bootstrapped and working, but vshard.storage.cfg() wasn't called >> yet. In that case vshard.storage.cfg() on the replica wouldn't >> block in box.cfg(), and would happily fail a few lines later in >> vshard.storage.cfg() at attempt to touch not existing >> box.space._bucket. >> >> That was the case for cartridge. The framework configures >> instances in its own way before vshard.storage.cfg(). Then it >> calls vshard.storage.cfg() on them in arbitrary order, and >> sometimes replicas were configured earlier than the master. >> >> The fail is fixed by simply skipping the _bucket's trigger >> installation on the replica node. Because so far it is not needed >> here for anything. The trigger's sole purpose is to increment >> bucket generation used only for DML on _bucket on the master node. >> >> Now vshard.storage.cfg() on replica is able to finish before >> master does the same. However the replica still won't be able to >> handle vshard.storage.* requests until receives vshard schema >> from master. >> >> Closes #237 >> --- >> Branch: >> http://github.com/tarantool/vshard/tree/gerold103/gh-237-boot-replica-first >> Issue: https://github.com/tarantool/vshard/issues/237 >> >> test/lua_libs/storage_template.lua | 55 +++++++++++- >> test/reload_evolution/storage.result | 10 +++ >> test/reload_evolution/storage.test.lua | 8 ++ >> test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ >> test/router/boot_replica_first.test.lua | 42 +++++++++ >> vshard/storage/init.lua | 18 +++- >> vshard/storage/reload_evolution.lua | 10 ++- >> 7 files changed, 251 insertions(+), 4 deletions(-) >> create mode 100644 test/router/boot_replica_first.result >> create mode 100644 test/router/boot_replica_first.test.lua >> >> diff --git a/test/lua_libs/storage_template.lua >> b/test/lua_libs/storage_template.lua >> index 29a753d..84e4180 100644 >> --- a/test/lua_libs/storage_template.lua >> +++ b/test/lua_libs/storage_template.lua >> @@ -1,5 +1,7 @@ >> #!/usr/bin/env tarantool >> >> +local luri = require('uri') >> + >> NAME = require('fio').basename(arg[0], '.lua') >> fiber = require('fiber') >> test_run = require('test_run').new() >> @@ -10,12 +12,63 @@ log = require('log') >> if not cfg.shard_index then >> cfg.shard_index = 'bucket_id' >> end >> +instance_uuid = util.name_to_uuid[NAME] >> + >> +-- >> +-- Bootstrap the instance exactly like vshard does. But don't >> +-- initialize any vshard-specific code. >> +-- >> +local function boot_like_vshard() >> + assert(type(box.cfg) == 'function') >> + for rs_uuid, rs in pairs(cfg.sharding) do >> + for replica_uuid, replica in pairs(rs.replicas) do >> + if replica_uuid == instance_uuid then >> + local box_cfg = {replication = {}} >> + box_cfg.instance_uuid = replica_uuid >> + box_cfg.replicaset_uuid = rs_uuid >> + box_cfg.listen = replica.uri >> + box_cfg.read_only = not replica.master >> + box_cfg.replication_connect_quorum = 0 >> + box_cfg.replication_timeout = 0.1 >> + for _, replica in pairs(rs.replicas) do >> + table.insert(box_cfg.replication, replica.uri) >> + end >> + box.cfg(box_cfg) >> + if not replica.master then >> + return >> + end >> + local uri = luri.parse(replica.uri) >> + box.schema.user.create(uri.login, { >> + password = uri.password, if_not_exists = true, >> + }) >> + box.schema.user.grant(uri.login, 'super') >> + return >> + end >> + end >> + end >> + assert(false) >> > > Should there be some meaningful message? > > >> +end >> + >> +local omit_cfg = false >> +local i = 1 >> +while arg[i] ~= nil do >> + local key = arg[i] >> + i = i + 1 >> + if key == 'boot_before_cfg' then >> + boot_like_vshard() >> + omit_cfg = true >> + end >> +end >> >> vshard = require('vshard') >> echo_count = 0 >> cfg.replication_connect_timeout = 3 >> cfg.replication_timeout = 0.1 >> -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) >> + >> +if not omit_cfg then >> + vshard.storage.cfg(cfg, instance_uuid) >> +end >> + >> function bootstrap_storage(engine) >> box.once("testapp:schema:1", function() >> if rawget(_G, 'CHANGE_SPACE_IDS') then >> diff --git a/test/reload_evolution/storage.result >> b/test/reload_evolution/storage.result >> index ebf4fdc..4652c4f 100644 >> --- a/test/reload_evolution/storage.result >> +++ b/test/reload_evolution/storage.result >> @@ -92,6 +92,16 @@ test_run:grep_log('storage_2_a', >> 'vshard.storage.reload_evolution: upgraded to') >> ... >> vshard.storage.internal.reload_version >> --- >> +- 2 >> +... >> +-- >> +-- gh-237: should be only one trigger. During gh-237 the trigger >> installation >> +-- became conditional and therefore required to remember the current >> trigger >> +-- somewhere. When reloaded from the old version, the trigger needed to >> be >> +-- fetched from _bucket:on_replace(). >> +-- >> +#box.space._bucket:on_replace() >> +--- >> - 1 >> ... >> -- Make sure storage operates well. >> diff --git a/test/reload_evolution/storage.test.lua >> b/test/reload_evolution/storage.test.lua >> index 56c1693..06f7117 100644 >> --- a/test/reload_evolution/storage.test.lua >> +++ b/test/reload_evolution/storage.test.lua >> @@ -37,6 +37,14 @@ package.loaded['vshard.storage'] = nil >> vshard.storage = require("vshard.storage") >> test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: >> upgraded to') ~= nil >> vshard.storage.internal.reload_version >> +-- >> +-- gh-237: should be only one trigger. During gh-237 the trigger >> installation >> +-- became conditional and therefore required to remember the current >> trigger >> +-- somewhere. When reloaded from the old version, the trigger needed to >> be >> +-- fetched from _bucket:on_replace(). >> +-- >> +#box.space._bucket:on_replace() >> + >> -- Make sure storage operates well. >> vshard.storage.bucket_force_drop(2000) >> vshard.storage.bucket_force_create(2000) >> diff --git a/test/router/boot_replica_first.result >> b/test/router/boot_replica_first.result >> new file mode 100644 >> index 0000000..1705230 >> --- /dev/null >> +++ b/test/router/boot_replica_first.result >> @@ -0,0 +1,112 @@ >> +-- test-run result file version 2 >> +test_run = require('test_run').new() >> + | --- >> + | ... >> +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } >> + | --- >> + | ... >> +test_run:create_cluster(REPLICASET_1, 'router', {args = >> 'boot_before_cfg'}) >> + | --- >> + | ... >> +util = require('util') >> + | --- >> + | ... >> +util.wait_master(test_run, REPLICASET_1, 'box_1_a') >> + | --- >> + | ... >> +_ = test_run:cmd("create server router with >> script='router/router_2.lua'") >> + | --- >> + | ... >> +_ = test_run:cmd("start server router") >> + | --- >> + | ... >> + >> +-- >> +-- gh-237: replica should be able to boot before master. Before the >> issue was >> +-- fixed, replica always tried to install a trigger on _bucket space >> even when >> +-- it was not created on a master yet - that lead to an exception in >> +-- storage.cfg. Now it should not install the trigger at all, because >> anyway it >> +-- is not needed on replica for anything. >> +-- >> + >> +test_run:switch('box_1_b') >> + | --- >> + | - true >> + | ... >> +vshard.storage.cfg(cfg, instance_uuid) >> + | --- >> + | ... >> +-- _bucket is not created yet. Will fail. >> +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) >> + | --- >> + | - attempt to index field '_bucket' (a nil value) >> + | ... >> + >> +test_run:switch('default') >> + | --- >> + | - true >> + | ... >> +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') >> + | --- >> + | ... >> + >> +test_run:switch('box_1_a') >> + | --- >> + | - true >> + | ... >> +vshard.storage.cfg(cfg, instance_uuid) >> + | --- >> + | ... >> + >> +test_run:switch('box_1_b') >> + | --- >> + | - true >> + | ... >> +test_run:wait_lsn('box_1_b', 'box_1_a') >> + | --- >> + | ... >> +-- Fails, but gracefully. Because the bucket is not found here. >> +vshard.storage.call(1, 'read', 'echo', {100}) >> + | --- >> + | - null >> + | - bucket_id: 1 >> + | reason: Not found >> + | code: 1 >> + | type: ShardingError >> + | message: 'Cannot perform action with bucket 1, reason: Not found' >> + | name: WRONG_BUCKET >> + | ... >> +-- Should not have triggers. >> +#box.space._bucket:on_replace() >> + | --- >> + | - 0 >> + | ... >> + >> +test_run:switch('router') >> + | --- >> + | - true >> + | ... >> +vshard.router.bootstrap() >> + | --- >> + | - true >> + | ... >> +vshard.router.callro(1, 'echo', {100}) >> + | --- >> + | - 100 >> + | ... >> + >> +test_run:switch("default") >> + | --- >> + | - true >> + | ... >> +test_run:cmd('stop server router') >> + | --- >> + | - true >> + | ... >> +test_run:cmd('delete server router') >> + | --- >> + | - true >> + | ... >> +test_run:drop_cluster(REPLICASET_1) >> + | --- >> + | ... >> diff --git a/test/router/boot_replica_first.test.lua >> b/test/router/boot_replica_first.test.lua >> new file mode 100644 >> index 0000000..7b1b3fd >> --- /dev/null >> +++ b/test/router/boot_replica_first.test.lua >> @@ -0,0 +1,42 @@ >> +test_run = require('test_run').new() >> +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } >> +test_run:create_cluster(REPLICASET_1, 'router', {args = >> 'boot_before_cfg'}) >> +util = require('util') >> +util.wait_master(test_run, REPLICASET_1, 'box_1_a') >> +_ = test_run:cmd("create server router with >> script='router/router_2.lua'") >> +_ = test_run:cmd("start server router") >> + >> +-- >> +-- gh-237: replica should be able to boot before master. Before the >> issue was >> +-- fixed, replica always tried to install a trigger on _bucket space >> even when >> +-- it was not created on a master yet - that lead to an exception in >> +-- storage.cfg. Now it should not install the trigger at all, because >> anyway it >> +-- is not needed on replica for anything. >> +-- >> + >> +test_run:switch('box_1_b') >> +vshard.storage.cfg(cfg, instance_uuid) >> +-- _bucket is not created yet. Will fail. >> +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) >> + >> +test_run:switch('default') >> +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') >> + >> +test_run:switch('box_1_a') >> +vshard.storage.cfg(cfg, instance_uuid) >> + >> +test_run:switch('box_1_b') >> +test_run:wait_lsn('box_1_b', 'box_1_a') >> +-- Fails, but gracefully. Because the bucket is not found here. >> +vshard.storage.call(1, 'read', 'echo', {100}) >> +-- Should not have triggers. >> +#box.space._bucket:on_replace() >> + >> +test_run:switch('router') >> +vshard.router.bootstrap() >> +vshard.router.callro(1, 'echo', {100}) >> + >> +test_run:switch("default") >> +test_run:cmd('stop server router') >> +test_run:cmd('delete server router') >> +test_run:drop_cluster(REPLICASET_1) >> diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua >> index c6a78fe..ed577f9 100644 >> --- a/vshard/storage/init.lua >> +++ b/vshard/storage/init.lua >> @@ -94,6 +94,17 @@ if not M then >> -- detect that _bucket was not changed between yields. >> -- >> bucket_generation = 0, >> + -- >> + -- Reference to the function used as on_replace trigger on >> + -- _bucket space. It is used to replace the trigger with >> + -- a new function when reload happens. It is kept >> + -- explicitly because the old function is deleted on >> + -- reload from the global namespace. On the other hand, it >> + -- is still stored in _bucket:on_replace() somewhere, but >> + -- it is not known where. The only 100% way to be able to >> + -- replace the old function is to keep its reference. >> + -- >> + bucket_on_replace = nil, >> >> ------------------- Garbage collection ------------------- >> -- Fiber to remove garbage buckets data. >> @@ -2435,8 +2446,11 @@ local function storage_cfg(cfg, this_replica_uuid, >> is_reload) >> local uri = luri.parse(this_replica.uri) >> schema_upgrade(is_master, uri.login, uri.password) >> >> - local old_trigger = box.space._bucket:on_replace()[1] >> - box.space._bucket:on_replace(bucket_generation_increment, >> old_trigger) >> + if is_master then >> + local old_trigger = M.bucket_on_replace >> + box.space._bucket:on_replace(bucket_generation_increment, >> old_trigger) >> + M.bucket_on_replace = bucket_generation_increment >> + end >> > It seems that the trigger is never removed. If a replica was a master some time ago, it still has the trigger. Is it safe? >> lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) >> lreplicaset.outdate_replicasets(M.replicasets) >> diff --git a/vshard/storage/reload_evolution.lua >> b/vshard/storage/reload_evolution.lua >> index 5d09b11..f38af74 100644 >> --- a/vshard/storage/reload_evolution.lua >> +++ b/vshard/storage/reload_evolution.lua >> @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) >> -- Code to update Lua objects. >> end >> >> +migrations[#migrations + 1] = function(M) >> + local bucket = box.space._bucket >> + if bucket then >> + assert(M.bucket_on_replace == nil) >> + M.bucket_on_replace = bucket:on_replace()[1] >> + end >> +end >> + >> -- >> -- Perform an update based on a version stored in `M` (internals). >> -- @param M Old module internals which should be updated. >> @@ -33,7 +41,7 @@ local function upgrade(M) >> log.error(err_msg) >> error(err_msg) >> end >> - for i = start_version, #migrations do >> + for i = start_version + 1, #migrations do >> > > Is it already tested somewhere else (migrations I mean)? This change looks > like a fix for some other problem. > > >> local ok, err = pcall(migrations[i], M) >> if ok then >> log.info('vshard.storage.reload_evolution: upgraded to %d >> version', >> -- >> 2.21.1 (Apple Git-122.3) >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From i.kosarev at tarantool.org Wed Aug 5 13:12:51 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Wed, 5 Aug 2020 13:12:51 +0300 Subject: [Tarantool-patches] [PATCH v2] tuple: drop extra restrictions for multikey index Message-ID: <20200805101251.30600-1-i.kosarev@tarantool.org> Multikey index did not work properly with nullable root field in tuple_raw_multikey_count(). Now it is fixed and corresponding restrictions are dropped. This also means that we can drop implicit nullability update for array/map fields and make all fields nullable by default, as it was until e1d3fe8ab8eed65394ad17409401a93b6fcdc435 (tuple format: don't allow null where array/map is expected), as far as default non-nullability itself doesn't solve any real problems while providing confusing behavior (gh-5027). Follow-up #5027 Closes #5192 --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5192-fix-multikey-index-restrictions Issue: https://github.com/tarantool/tarantool/issues/5192 @ChangeLog: * Dropped restrictions on nullable multikey index root. All fields are now nullable by default as it was before 2.2.1 (gh-5192). Changes in v2: - removed insignificant changes - fixed tests comments src/box/memtx_space.c | 8 --- src/box/tuple.c | 5 +- src/box/tuple_format.c | 20 ------ src/box/vinyl.c | 8 --- test/engine/gh-5027-fields-nullability.result | 71 +++---------------- .../gh-5027-fields-nullability.test.lua | 35 +++------ .../gh-5192-multikey-root-nullability.result | 62 ++++++++++++++++ ...gh-5192-multikey-root-nullability.test.lua | 20 ++++++ test/engine/json.result | 20 +++--- test/engine/json.test.lua | 6 +- test/engine/multikey.result | 8 +-- test/engine/multikey.test.lua | 4 +- 12 files changed, 123 insertions(+), 144 deletions(-) create mode 100644 test/engine/gh-5192-multikey-root-nullability.result create mode 100644 test/engine/gh-5192-multikey-root-nullability.test.lua diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index 8452ab430..d30ce44b8 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -604,14 +604,6 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) return -1; } } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } switch (index_def->type) { case HASH: if (! index_def->opts.is_unique) { diff --git a/src/box/tuple.c b/src/box/tuple.c index 9f0f24c64..94a3a96ba 100644 --- a/src/box/tuple.c +++ b/src/box/tuple.c @@ -554,7 +554,10 @@ tuple_raw_multikey_count(struct tuple_format *format, const char *data, NULL, MULTIKEY_NONE); if (array_raw == NULL) return 0; - assert(mp_typeof(*array_raw) == MP_ARRAY); + enum mp_type type = mp_typeof(*array_raw); + if (type == MP_NIL) + return 0; + assert(type == MP_ARRAY); return mp_decode_array(&array_raw); } diff --git a/src/box/tuple_format.c b/src/box/tuple_format.c index bae6c67cd..9b817d3cf 100644 --- a/src/box/tuple_format.c +++ b/src/box/tuple_format.c @@ -242,24 +242,6 @@ tuple_field_ensure_child_compatibility(struct tuple_field *parent, return 0; } -/** - * Tuple fields are nullable by default. However, it is not ok - * for array/map fields, as far as their implicit nullability - * might break field accessors expectations, provide confusing - * error messages and cause incorrect behaviour of - * tuple_multikey_count(). Thus array/map fields have to be - * non-nullable by default, which means we have to update default - * nullability for them. - */ -static void -tuple_field_update_nullability(struct tuple_field *field) -{ - if ((field->type == FIELD_TYPE_ARRAY || - field->type == FIELD_TYPE_MAP) && - tuple_field_is_nullable(field)) - field->nullable_action = ON_CONFLICT_ACTION_DEFAULT; -} - /** * Given a field number and a path, add the corresponding field * to the tuple format, allocating intermediate fields if @@ -335,13 +317,11 @@ tuple_format_add_field(struct tuple_format *format, uint32_t fieldno, parent->offset_slot = *current_slot; } } - tuple_field_update_nullability(parent); parent->is_key_part = true; next->is_multikey_part = is_multikey; parent = next; token_count++; } - tuple_field_update_nullability(parent); /* * The path has already been verified by the * key_def_decode_parts function. diff --git a/src/box/vinyl.c b/src/box/vinyl.c index 32301d7ba..5fa3ea3a5 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -658,14 +658,6 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) diag_set(ClientError, ER_NULLABLE_PRIMARY, space_name(space)); return -1; } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } /* Check that there are no ANY, ARRAY, MAP parts */ for (uint32_t i = 0; i < key_def->part_count; i++) { struct key_part *part = &key_def->parts[i]; diff --git a/test/engine/gh-5027-fields-nullability.result b/test/engine/gh-5027-fields-nullability.result index 1121727f6..382944d53 100644 --- a/test/engine/gh-5027-fields-nullability.result +++ b/test/engine/gh-5027-fields-nullability.result @@ -12,19 +12,19 @@ _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=true}}}) | --- | ... -s:replace{1} +s:replace{1} -- ok | --- | - [1] | ... -s:replace{1, box.NULL} +s:replace{1, box.NULL} -- ok | --- | - [1, null] | ... -s:replace{1, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL} -- ok | --- | - [1, null, null] | ... -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL, box.NULL} -- ok | --- | - [1, null, null, null] | ... @@ -41,79 +41,26 @@ _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) | --- | ... -s:replace{1} +s:replace{1} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL} +s:replace{1, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL, 5} +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok | --- | - [1, null, null, null, 5] | ... s:drop() | --- | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - [1, null] - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | - error: multikey index does not support nullable root field - | ... -s:replace{2, box.NULL} - | --- - | - [2, null] - | ... -s:drop() - | --- - | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array'}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | ... -s:replace{2, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -s:replace{3, {}} - | --- - | - [3, []] - | ... -s:drop() - | --- - | ... diff --git a/test/engine/gh-5027-fields-nullability.test.lua b/test/engine/gh-5027-fields-nullability.test.lua index 960103d6c..664883d0d 100644 --- a/test/engine/gh-5027-fields-nullability.test.lua +++ b/test/engine/gh-5027-fields-nullability.test.lua @@ -3,35 +3,18 @@ test_run = require('test_run').new() s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=true}}}) -s:replace{1} -s:replace{1, box.NULL} -s:replace{1, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1} -- ok +s:replace{1, box.NULL} -- ok +s:replace{1, box.NULL, box.NULL} -- ok +s:replace{1, box.NULL, box.NULL, box.NULL} -- ok s:drop() s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) -s:replace{1} -s:replace{1, box.NULL} -s:replace{1, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL, 5} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array'}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:replace{3, {}} +s:replace{1} -- error +s:replace{1, box.NULL} -- error +s:replace{1, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok s:drop() diff --git a/test/engine/gh-5192-multikey-root-nullability.result b/test/engine/gh-5192-multikey-root-nullability.result new file mode 100644 index 000000000..dcbb5067f --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.result @@ -0,0 +1,62 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- ok + | --- + | - [1, null] + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- ok + | --- + | - [2, null] + | ... +_ = s:delete(2) + | --- + | ... +s:truncate() + | --- + | ... +s:drop() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array'}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +s:replace{3, {}} -- ok + | --- + | - [3, []] + | ... +s:drop() + | --- + | ... diff --git a/test/engine/gh-5192-multikey-root-nullability.test.lua b/test/engine/gh-5192-multikey-root-nullability.test.lua new file mode 100644 index 000000000..5a8305bf9 --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.test.lua @@ -0,0 +1,20 @@ +test_run = require('test_run').new() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- ok +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- ok +_ = s:delete(2) +s:truncate() +s:drop() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array'}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- error +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- error +s:replace{3, {}} -- ok +s:drop() \ No newline at end of file diff --git a/test/engine/json.result b/test/engine/json.result index 2175266fc..b8fd9a1b6 100644 --- a/test/engine/json.result +++ b/test/engine/json.result @@ -738,12 +738,11 @@ _ = s:create_index('sk', {parts = {{'[2][1].a', 'unsigned'}}}) ... s:insert{1, box.NULL} -- error --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{2, {box.NULL}} -- error --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{3} -- error --- @@ -764,16 +763,15 @@ s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} --- ... -s:insert{7, box.NULL} -- error +s:insert{7, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [7, null] ... -s:insert{8, {box.NULL}} -- error +s:insert{8, {box.NULL}} -- ok --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- [8, [null]] ... --- Skipping nullable fields is okay though. +-- Skipping nullable fields is also okay. s:insert{9} -- ok --- - [9] @@ -792,7 +790,9 @@ s:insert{12, {{a = box.NULL}}} -- ok ... s.index.sk:select() --- -- - [9] +- - [7, null] + - [8, [null]] + - [9] - [10, []] - [11, [{'b': 1}]] - [12, [{'a': null}]] diff --git a/test/engine/json.test.lua b/test/engine/json.test.lua index 4771c3162..371bbad91 100644 --- a/test/engine/json.test.lua +++ b/test/engine/json.test.lua @@ -220,9 +220,9 @@ s:insert{4, {}} -- error s:insert{5, {{b = 1}}} -- error s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} -s:insert{7, box.NULL} -- error -s:insert{8, {box.NULL}} -- error --- Skipping nullable fields is okay though. +s:insert{7, box.NULL} -- ok +s:insert{8, {box.NULL}} -- ok +-- Skipping nullable fields is also okay. s:insert{9} -- ok s:insert{10, {}} -- ok s:insert{11, {{b = 1}}} -- ok diff --git a/test/engine/multikey.result b/test/engine/multikey.result index 968be4cc3..ff72983ec 100644 --- a/test/engine/multikey.result +++ b/test/engine/multikey.result @@ -794,9 +794,9 @@ _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) --- ... -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [1, null] ... s:insert{2, {box.NULL}} -- error --- @@ -814,9 +814,9 @@ s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} --- ... -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [5, null] ... s:insert{6, {box.NULL}} -- ok --- diff --git a/test/engine/multikey.test.lua b/test/engine/multikey.test.lua index ed7033494..653a9c9b1 100644 --- a/test/engine/multikey.test.lua +++ b/test/engine/multikey.test.lua @@ -214,12 +214,12 @@ s:drop() s = box.schema.space.create('test', {engine = engine}) _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok s:insert{2, {box.NULL}} -- error s:insert{3, {}} -- ok s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok s:insert{6, {box.NULL}} -- ok s:insert{7, {}} -- ok s:insert{8, {2}} -- ok -- 2.17.1 From korablev at tarantool.org Wed Aug 5 15:29:21 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Wed, 5 Aug 2020 12:29:21 +0000 Subject: [Tarantool-patches] [PATCH v2] tuple: drop extra restrictions for multikey index In-Reply-To: <20200805101251.30600-1-i.kosarev@tarantool.org> References: <20200805101251.30600-1-i.kosarev@tarantool.org> Message-ID: <20200805122921.GA15209@tarantool.org> On 05 Aug 13:12, Ilya Kosarev wrote: > @ChangeLog: > * Dropped restrictions on nullable multikey index root. All fields are > now nullable by default as it was before 2.2.1 (gh-5192). I'd underline the fact that problem was fixed in another (local code change) way. What is more, I guess we should document this behaviour: our current docs say nothing concerning nulls as multikey roots. > diff --git a/test/engine/gh-5027-fields-nullability.test.lua b/test/engine/gh-5027-fields-nullability.test.lua > index 960103d6c..664883d0d 100644 > --- a/test/engine/gh-5027-fields-nullability.test.lua > +++ b/test/engine/gh-5027-fields-nullability.test.lua > @@ -3,35 +3,18 @@ test_run = require('test_run').new() > s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) > _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) > _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=true}}}) > -s:replace{1} > -s:replace{1, box.NULL} > -s:replace{1, box.NULL, box.NULL} > -s:replace{1, box.NULL, box.NULL, box.NULL} > +s:replace{1} -- ok > +s:replace{1, box.NULL} -- ok > +s:replace{1, box.NULL, box.NULL} -- ok > +s:replace{1, box.NULL, box.NULL, box.NULL} -- ok I'd drop these -- ok comments since they only enlarge diff. > diff --git a/test/engine/gh-5192-multikey-root-nullability.test.lua b/test/engine/gh-5192-multikey-root-nullability.test.lua > new file mode 100644 > index 000000000..5a8305bf9 > --- /dev/null > +++ b/test/engine/gh-5192-multikey-root-nullability.test.lua > @@ -0,0 +1,20 @@ > +test_run = require('test_run').new() > + > +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) > +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) > +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) > +s:replace{1, box.NULL} -- ok > +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) > +s:replace{2, box.NULL} -- ok > +_ = s:delete(2) > +s:truncate() Why do you need to truncate space before drop? > +s:drop() > + > +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) > +_ = s:format({{name='id'}, {name='data', type='array'}}) > +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) > +s:replace{1, box.NULL} -- error > +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) > +s:replace{2, box.NULL} -- error > +s:replace{3, {}} -- ok > +s:drop() > \ No newline at end of file Nit: put pls newline at the end of file to avoid this warnings. > diff --git a/test/engine/json.test.lua b/test/engine/json.test.lua > index 4771c3162..371bbad91 100644 > --- a/test/engine/json.test.lua > +++ b/test/engine/json.test.lua > @@ -220,9 +220,9 @@ s:insert{4, {}} -- error > s:insert{5, {{b = 1}}} -- error > s:insert{6, {{a = 1}}} -- ok > s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} > -s:insert{7, box.NULL} -- error > -s:insert{8, {box.NULL}} -- error > --- Skipping nullable fields is okay though. > +s:insert{7, box.NULL} -- ok > +s:insert{8, {box.NULL}} -- ok > +-- Skipping nullable fields is also okay. Is this change really required? :) > s:insert{9} -- ok > s:insert{10, {}} -- ok > s:insert{11, {{b = 1}}} -- ok From huston.mavr at gmail.com Wed Aug 5 20:08:13 2020 From: huston.mavr at gmail.com (Mavr Huston) Date: Wed, 5 Aug 2020 20:08:13 +0300 Subject: [Tarantool-patches] [PATCH] build: refactor static build process In-Reply-To: <20200727223734.cnxrdzik2cyt3ey4@tkn_work_nb> References: <20200622181649.10100-1-huston.mavr@gmail.com> <20200727223734.cnxrdzik2cyt3ey4@tkn_work_nb> Message-ID: Hi, thanks for the review! libicu installs as ExternalProject_Add too, its missed in commit message; Problem with curses and ncurses was on macOS and linux, because libcurses is an entire copy of libncurses, and tarantool links with system libcurses instead of libncurses installed as tarantool dependency, but module FindCurses.cmkae provides workaround for this problem - CURSES_NEED_NCURSES flag.- to use ncurses instead of curses. (i will fix this part at commit message) About disable-shred flag, used at libcurl building - we want to link only with static libraries, so we prevent creating unused .so. I've renamed static_build_no_deps_* jobs after review to static_build_cmake_* Also about such path tarantool-prefix/* - it's a cmake ExternalProject_Add() default path (i've also added comment at .travis.mk) Useless comments "Init macOS test env" deleted. > if (BUILD_STATIC) > - set(LIBZ_LIB_NAME libz.a) > + find_library(LIBZ_LIBRARY NAMES libz.a) > else() > - set(LIBZ_LIB_NAME z) > + find_library(LIBZ_LIBRARY NAMES z) > endif() > - find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) Here we simplified code, by deleting useless variable. I've added commentaries to cmake/compiler.cmake about libunwind on macOS and about ignoring flag -static-libstdc++ on macOS I've fixed static-build for using system compiler: gcc/g++ on linux and clang/clang++ on macOS I've refactored IF (NOT APPLE) condition to IF (APPLE) at static-build/CMakeLists.txt I've mentioned macOS dependencies at static-build/README.md xcode-tools and others, also I've added example with CMAKE_TARANTOOL_ARGS. Added commentaries about _EP_INSTALL_DIR at static-build/CMakeLists.txt Also deleted unused use_unix_sockets_iproto = True Also deleted curl-features.test.lua, because after rebase on master it fails, due to missing curl_version_info symbol at tarantool binary. This symbol lost after #807c7fa584f21ee955b2a14623d70f7510a3650d (build: update curl submodule to 7.71.1 version ) After pass the review I'll squash this changes to base commit and update commit message. Here is a diff of changes: ===================================== diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 5ec5dd9b9..c9aef3dc7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -534,14 +534,14 @@ static_build: script: - ${GITLAB_MAKE} test_static_build -static_build_no_deps_linux: +static_build_cmake_linux: <<: *docker_test_definition script: - - ${GITLAB_MAKE} test_static_build_no_deps_linux + - ${GITLAB_MAKE} test_static_build_cmake_linux -static_build_no_deps_osx_15: +static_build_cmake_osx_15: stage: test tags: - osx_15 script: - - ${GITLAB_MAKE} test_static_build_no_deps_osx + - ${GITLAB_MAKE} test_static_build_cmake_osx diff --git a/.travis.mk b/.travis.mk index 64862348f..482672429 100644 --- a/.travis.mk +++ b/.travis.mk @@ -149,8 +149,8 @@ test_static_build: deps_debian_static CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk test_debian_no_deps # New static build - -test_static_build_no_deps_linux: +# builddir used in this target - is a default build path from cmake ExternalProject_Add() +test_static_build_cmake_linux: cd static-build && cmake . && make -j && ctest -V cd test && /usr/bin/python test-run.py --force \ --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build $(TEST_RUN_EXTRA_PARAMS) @@ -218,7 +218,6 @@ INIT_TEST_ENV_OSX=\ rm -rf /tmp/tnt test_osx_no_deps: build_osx - # Init macOS test env ${INIT_TEST_ENV_OSX}; \ cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) @@ -233,9 +232,9 @@ base_deps_osx: brew install --force ${STATIC_OSX_PKGS} || brew upgrade ${STATIC_OSX_PKGS} pip install --force-reinstall -r test-run/requirements.txt -test_static_build_no_deps_osx: base_deps_osx +# builddir used in this target - is a default build path from cmake ExternalProject_Add() +test_static_build_cmake_osx: base_deps_osx cd static-build && cmake . && make -j && ctest -V - # Init macOS test env ${INIT_TEST_ENV_OSX}; \ cd test && ./test-run.py --vardir /tmp/tnt \ --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake index 8422181d6..afe480679 100644 --- a/cmake/FindReadline.cmake +++ b/cmake/FindReadline.cmake @@ -14,6 +14,14 @@ if(BUILD_STATIC) if (NOT CURSES_INFO_LIBRARY) set(CURSES_INFO_LIBRARY "") endif() + + # From Modules/FindCurses.cmake: + # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the + # ``find_package(Curses)`` call if NCurses functionality is required. + # This flag is set for linking with required library (installed + # via static-build/CMakeLists.txt). If this variable won't be set + # then tarantool binary links with system library curses which is an + # entire copy of ncurses set(CURSES_NEED_NCURSES TRUE) endif() diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake index 14f1e1186..db2ae6227 100644 --- a/cmake/compiler.cmake +++ b/cmake/compiler.cmake @@ -131,6 +131,8 @@ set(CMAKE_REQUIRED_INCLUDES "") if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) set(UNWIND_LIB_NAME libunwind.a) else() + # libunwind can't be compiled on macOS. + # But there exists libunwind.dylib as a part of MacOSSDK set(UNWIND_LIB_NAME unwind) endif() find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME}) @@ -192,6 +194,9 @@ if (ENABLE_BACKTRACE) find_package_message(UNWIND_LIBRARIES "Found unwind" "${UNWIND_LIBRARIES}") endif() +# On macOS there is no '-static-libstdc++' flag and it's use will +# raise following error: +# error: argument unused during compilation: '-static-libstdc++' if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) # Static linking for c++ routines add_compile_flags("C;CXX" "-static-libstdc++") diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt index 86582af0a..53ceb609c 100644 --- a/static-build/CMakeLists.txt +++ b/static-build/CMakeLists.txt @@ -9,11 +9,18 @@ set(NCURSES_VERSION 6.2) set(READLINE_VERSION 8.0) set(UNWIND_VERSION 1.3-rc1) -find_program(C_COMPILER gcc) -find_program(CXX_COMPILER g++) +if (APPLE) + find_program(C_COMPILER clang) + find_program(CXX_COMPILER clang++) +else() + find_program(C_COMPILER gcc) + find_program(CXX_COMPILER g++) +endif() set(CMAKE_C_COMPILER ${C_COMPILER}) set(CMAKE_CXX_COMPILER ${CXX_COMPILER}) +# Install all libraries required by tarantool at current build dir + # # OpenSSL # @@ -80,7 +87,18 @@ ExternalProject_Add(readline # # ICONV # -if (NOT APPLE) +if (APPLE) + ExternalProject_Add(iconv + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + /configure + --prefix= + --disable-shared + --enable-static + --with-gnu-ld + STEP_TARGETS download + ) +else() # In linux iconv is embedded into glibc # So we find system header and copy it locally find_path(ICONV_INCLUDE_DIR iconv.h) @@ -101,20 +119,11 @@ if (NOT APPLE) add_custom_target(iconv DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h" ) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property set_target_properties(iconv PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX} ) -else() - ExternalProject_Add(iconv - URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz - CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} - /configure - --prefix= - --disable-shared - --enable-static - --with-gnu-ld - STEP_TARGETS download - ) endif() # @@ -162,6 +171,8 @@ if (APPLE) endif() add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES}) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property set_target_properties(unwind PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX} ) @@ -178,6 +189,8 @@ else() ) endif() +# Get install directories of builded libraries for building +# tarantool with custon CMAKE_PREFIX_PATH foreach(PROJ openssl icu zlib ncurses readline iconv unwind) ExternalProject_Get_Property(${PROJ} install_dir) set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir}) @@ -197,16 +210,14 @@ ExternalProject_Add(tarantool -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH} -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE -DOPENSSL_USE_STATIC_LIBS=TRUE - -DCMAKE_BUILD_TYPE=Debug -DBUILD_STATIC=TRUE -DENABLE_DIST=TRUE -DENABLE_BACKTRACE=TRUE - -DPACKAGE:STRING=${PACKAGE_NAME} -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} ${CMAKE_TARANTOOL_ARGS} - BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j STEP_TARGETS build + BUILD_COMMAND $(MAKE) ) enable_testing() diff --git a/static-build/README.md b/static-build/README.md index 29fe085c3..0019e963f 100644 --- a/static-build/README.md +++ b/static-build/README.md @@ -13,6 +13,24 @@ yum install -y \ python-msgpack python-yaml python-argparse python-six python-gevent ``` +MacOS: + +Before you start please install default Xcode Tools by Apple: + +``` +sudo xcode-select --install +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer +``` + +Install brew using command from +[Homebrew repository instructions](https://github.com/Homebrew/inst) + +After that run next script: + +```bash + brew install autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rbs + pip install --force-reinstall -r test-run/requirements.txt +``` ### Usage @@ -21,3 +39,20 @@ cmake . make -j ctest -V ``` + +## Customize your build + +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` variable + +### Usage + +There is three types of `CMAKE_BUILD_TYPE`: +* Debug - default +* Release +* RelWithDebInfo + +And you want to build tarantool with RelWithDebInfo: + +```bash +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" . +``` diff --git a/static-build/test/static-build/curl-features.test.lua b/static-build/test/static-build/curl-features.test.lua deleted file mode 100755 index 57b1c4306..000000000 --- a/static-build/test/static-build/curl-features.test.lua +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env tarantool - -local tap = require('tap') -local ffi = require('ffi') -ffi.cdef([[ - struct curl_version_info_data { - int age; /* see description below */ - const char *version; /* human readable string */ - unsigned int version_num; /* numeric representation */ - const char *host; /* human readable string */ - int features; /* bitmask, see below */ - char *ssl_version; /* human readable string */ - long ssl_version_num; /* not used, always zero */ - const char *libz_version; /* human readable string */ - const char * const *protocols; /* protocols */ - - /* when 'age' is CURLVERSION_SECOND or higher, the members below exist */ - const char *ares; /* human readable string */ - int ares_num; /* number */ - - /* when 'age' is CURLVERSION_THIRD or higher, the members below exist */ - const char *libidn; /* human readable string */ - - /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the members - below exist */ - int iconv_ver_num; /* '_libiconv_version' if iconv support enabled */ - - const char *libssh_version; /* human readable string */ - - /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the members - below exist */ - unsigned int brotli_ver_num; /* Numeric Brotli version - (MAJOR << 24) | (MINOR << 12) | PATCH */ - const char *brotli_version; /* human readable string. */ - - /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the members - below exist */ - unsigned int nghttp2_ver_num; /* Numeric nghttp2 version - (MAJOR << 16) | (MINOR << 8) | PATCH */ - const char *nghttp2_version; /* human readable string. */ - - const char *quic_version; /* human readable quic (+ HTTP/3) library + - version or NULL */ - - /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the members - below exist */ - const char *cainfo; /* the built-in default CURLOPT_CAINFO, might - be NULL */ - const char *capath; /* the built-in default CURLOPT_CAPATH, might - be NULL */ - }; - - struct curl_version_info_data *curl_version_info(int age); -]]) - -local info = ffi.C.curl_version_info(7) -local test = tap.test('curl-features') -test:plan(2) - -if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then - test:diag('ssl_version: ' .. ffi.string(info.ssl_version)) -end -if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then - test:diag('libz_version: ' .. ffi.string(info.libz_version)) -end - -os.exit(test:check() and 0 or 1) diff --git a/static-build/test/static-build/suite.ini b/static-build/test/static-build/suite.ini index 4da3d5d2f..92e349466 100644 --- a/static-build/test/static-build/suite.ini +++ b/static-build/test/static-build/suite.ini @@ -3,4 +3,3 @@ core = app description = Static build tests script = box.lua is_parallel = True -use_unix_sockets_iproto = True -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Thu Aug 6 01:15:46 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 00:15:46 +0200 Subject: [Tarantool-patches] [PATCH vshard 1/1] storage: allow replica to boot before master In-Reply-To: References: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> Message-ID: Thanks for the review! > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index c6a78fe..ed577f9 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -2435,8 +2446,11 @@ local function storage_cfg(cfg, this_replica_uuid, is_reload) > ? ? ?local uri = luri.parse(this_replica.uri) > ? ? ?schema_upgrade(is_master, uri.login, uri.password) > > -? ? local old_trigger = box.space._bucket:on_replace()[1] > -? ? box.space._bucket:on_replace(bucket_generation_increment, old_trigger) > +? ? if is_master then > +? ? ? ? local old_trigger = M.bucket_on_replace > +? ? ? ? box.space._bucket:on_replace(bucket_generation_increment, old_trigger) > +? ? ? ? M.bucket_on_replace = bucket_generation_increment > +? ? end > > > It seems that the trigger is never removed. If a replica was a master some time ago, it still has the trigger. > Is it safe? Yes, it is. The trigger only increments a counter in M. And the counter is not used for anything out of the node. It should be just locally monotonic. So it is ok to increment it even on replica. After all this is how it worked before. But again, after this comment I had second thoughts, and decided to remove the trigger when switched to replica to be consistent. See v2 in a new mail thread. From v.shpilevoy at tarantool.org Thu Aug 6 01:15:47 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 00:15:47 +0200 Subject: [Tarantool-patches] [PATCH vshard 1/1] storage: allow replica to boot before master In-Reply-To: References: <0963ccd10f91b249116916d5c1c91f5eef11e438.1596575031.git.v.shpilevoy@tarantool.org> Message-ID: <326078fd-79f7-7052-37e1-cbbd118e114c@tarantool.org> Hi! Thanks for the review! > diff --git a/test/lua_libs/storage_template.lua b/test/lua_libs/storage_template.lua > index 29a753d..84e4180 100644 > --- a/test/lua_libs/storage_template.lua > +++ b/test/lua_libs/storage_template.lua > @@ -10,12 +12,63 @@ log = require('log') > ?if not cfg.shard_index then > ? ? ?cfg.shard_index = 'bucket_id' > ?end > +instance_uuid = util.name_to_uuid[NAME] > + > +-- > +-- Bootstrap the instance exactly like vshard does. But don't > +-- initialize any vshard-specific code. > +-- > +local function boot_like_vshard() > +? ? assert(type(box.cfg) == 'function') > +? ? for rs_uuid, rs in pairs(cfg.sharding) do > +? ? ? ? for replica_uuid, replica in pairs(rs.replicas) do > +? ? ? ? ? ? if replica_uuid == instance_uuid then > +? ? ? ? ? ? ? ? local box_cfg = {replication = {}} > +? ? ? ? ? ? ? ? box_cfg.instance_uuid = replica_uuid > +? ? ? ? ? ? ? ? box_cfg.replicaset_uuid = rs_uuid > +? ? ? ? ? ? ? ? box_cfg.listen = replica.uri > +? ? ? ? ? ? ? ? box_cfg.read_only = not replica.master > +? ? ? ? ? ? ? ? box_cfg.replication_connect_quorum = 0 > +? ? ? ? ? ? ? ? box_cfg.replication_timeout = 0.1 > +? ? ? ? ? ? ? ? for _, replica in pairs(rs.replicas) do > +? ? ? ? ? ? ? ? ? ? table.insert(box_cfg.replication, replica.uri) > +? ? ? ? ? ? ? ? end > +? ? ? ? ? ? ? ? box.cfg(box_cfg) > +? ? ? ? ? ? ? ? if not replica.master then > +? ? ? ? ? ? ? ? ? ? return > +? ? ? ? ? ? ? ? end > +? ? ? ? ? ? ? ? local uri = luri.parse(replica.uri) > +? ? ? ? ? ? ? ? box.schema.user.create(uri.login, { > +? ? ? ? ? ? ? ? ? ? password = uri.password, if_not_exists = true, > +? ? ? ? ? ? ? ? }) > +? ? ? ? ? ? ? ? box.schema.user.grant(uri.login, 'super') > +? ? ? ? ? ? ? ? return > +? ? ? ? ? ? end > +? ? ? ? end > +? ? end > +? ? assert(false) > > > Should there be some meaningful message? Nope. This is just a test, so if it will fail, I will fix it. Don't want to invent anything here. The code is unreachable, and assert(false) is telling that fine. If you mean the message which would help to locate the error - not needed as well. The assertion fail message will contain file name and line number. > +end > + > +local omit_cfg = false > +local i = 1 > +while arg[i] ~= nil do > +? ? local key = arg[i] > +? ? i = i + 1 > +? ? if key == 'boot_before_cfg' then > +? ? ? ? boot_like_vshard() > +? ? ? ? omit_cfg = true > +? ? end > +end > diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua > index 5d09b11..f38af74 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) > ? ? ?-- Code to update Lua objects. > ?end > > +migrations[#migrations + 1] = function(M) > +? ? local bucket = box.space._bucket > +? ? if bucket then > +? ? ? ? assert(M.bucket_on_replace == nil) > +? ? ? ? M.bucket_on_replace = bucket:on_replace()[1] > +? ? end > +end > + > ?-- > ?-- Perform an update based on a version stored in `M` (internals). > ?-- @param M Old module internals which should be updated. > @@ -33,7 +41,7 @@ local function upgrade(M) > ? ? ? ? ?log.error(err_msg) > ? ? ? ? ?error(err_msg) > ? ? ?end > -? ? for i = start_version, #migrations? do > +? ? for i = start_version + 1, #migrations do > > > Is it already tested somewhere else (migrations I mean)? This change looks like a fix for some other problem. Good you noticed this. I thought this fix can't be separated, because reload evolution didn't have any meaningful migrations, and that couldn't be tested. But after your question I had second thoughts and it appears the bug does have a "test" already. I extracted it into a separate commit. Although the test is reverted by the second commit anyway. See v2 patchset in a new thread. From v.shpilevoy at tarantool.org Thu Aug 6 01:15:47 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 00:15:47 +0200 Subject: [Tarantool-patches] [PATCH v2 vshard 0/2] storage: allow replica to boot before master Message-ID: The patchset fixes a problem appearing in tarantool cartridge when replica is configured before master and fails with an ugly error without a serious reason for that. As a prerequisite the reload evolution subsystem is fixed, because it had a bug affecting the main commit. The bug didn't appear earlier, because the evolution was never used for anything so far. Changes in v2: - The reload evolution fix is moved into a separate commit; - Instance un-installs _bucket trigger when master is switched to replica. Branch: http://github.com/tarantool/vshard/tree/gerold103/gh-237-boot-replica-first Issue: https://github.com/tarantool/vshard/issues/237 Vladislav Shpilevoy (2): storage: fix reload applying migration twice storage: allow replica to boot before master test/lua_libs/storage_template.lua | 55 +++++++++++- test/misc/reconfigure.result | 33 +++++++ test/misc/reconfigure.test.lua | 9 ++ test/reload_evolution/storage.result | 10 +++ test/reload_evolution/storage.test.lua | 8 ++ test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ test/router/boot_replica_first.test.lua | 42 +++++++++ vshard/storage/init.lua | 21 ++++- vshard/storage/reload_evolution.lua | 10 ++- 9 files changed, 296 insertions(+), 4 deletions(-) create mode 100644 test/router/boot_replica_first.result create mode 100644 test/router/boot_replica_first.test.lua -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Thu Aug 6 01:15:48 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 00:15:48 +0200 Subject: [Tarantool-patches] [PATCH v2 vshard 1/2] storage: fix reload applying migration twice In-Reply-To: References: Message-ID: Reload evolution is a subsystem for storage hot code reload. When code is reloaded, it applies migration scripts one by one. But there was a bug - the last migration script was applied even when code version didn't really change. The bug didn't affect anything, since so far the reload evolution was not used for anything and had just one dummy migration script doing nothing. Now a first migration script is going to be added, so the bug is fixed before that. Needed for #237 --- test/reload_evolution/storage.result | 6 +++++- test/reload_evolution/storage.test.lua | 6 +++++- vshard/storage/reload_evolution.lua | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result index ebf4fdc..dde0a1f 100644 --- a/test/reload_evolution/storage.result +++ b/test/reload_evolution/storage.result @@ -86,7 +86,11 @@ package.loaded['vshard.storage'] = nil vshard.storage = require("vshard.storage") --- ... -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil +-- Should be nil. Because there was a bug that reload always reapplied the last +-- migration callback. When it was fixed, the last callback wasn't called twice. +-- But since the callback was only one, now nothing is called, and nothing is +-- logged. +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil --- - true ... diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua index 56c1693..7858112 100644 --- a/test/reload_evolution/storage.test.lua +++ b/test/reload_evolution/storage.test.lua @@ -35,7 +35,11 @@ box.space.test:insert({42, bucket_id_to_move}) package.path = original_package_path package.loaded['vshard.storage'] = nil vshard.storage = require("vshard.storage") -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil +-- Should be nil. Because there was a bug that reload always reapplied the last +-- migration callback. When it was fixed, the last callback wasn't called twice. +-- But since the callback was only one, now nothing is called, and nothing is +-- logged. +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil vshard.storage.internal.reload_version -- Make sure storage operates well. vshard.storage.bucket_force_drop(2000) diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua index 5d09b11..1abc9e2 100644 --- a/vshard/storage/reload_evolution.lua +++ b/vshard/storage/reload_evolution.lua @@ -33,7 +33,7 @@ local function upgrade(M) log.error(err_msg) error(err_msg) end - for i = start_version, #migrations do + for i = start_version + 1, #migrations do local ok, err = pcall(migrations[i], M) if ok then log.info('vshard.storage.reload_evolution: upgraded to %d version', -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Thu Aug 6 01:15:49 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 00:15:49 +0200 Subject: [Tarantool-patches] [PATCH v2 vshard 2/2] storage: allow replica to boot before master In-Reply-To: References: Message-ID: <0cd777f3990939b32d96bf964dea55e8073e65bb.1596665591.git.v.shpilevoy@tarantool.org> Before the patch a replica couldn't call vshard.storage.cfg() before the master is bootstrapped. That could lead to an exception in the middle of vshard.storage.cfg(). It sounds not so bad and even looks unreachable, because replica bootstrap anyway is blocked in box.cfg() until a master node is started, otherwise the replica instance is terminated. But the box.cfg bootstrap != vshard bootstrap. It could be that both replica and master nodes are already bootstrapped and working, but vshard.storage.cfg() wasn't called yet. In that case vshard.storage.cfg() on the replica wouldn't block in box.cfg(), and would happily fail a few lines later in vshard.storage.cfg() at attempt to touch not existing box.space._bucket. That was the case for cartridge. The framework configures instances in its own way before vshard.storage.cfg(). Then it calls vshard.storage.cfg() on them in arbitrary order, and sometimes replicas were configured earlier than the master. The fail is fixed by simply skipping the _bucket's trigger installation on the replica node. Because so far it is not needed here for anything. The trigger's sole purpose is to increment bucket generation used only for DML on _bucket on the master node. Now vshard.storage.cfg() on replica is able to finish before master does the same. However the replica still won't be able to handle vshard.storage.* requests until receives vshard schema from master. Closes #237 --- test/lua_libs/storage_template.lua | 55 +++++++++++- test/misc/reconfigure.result | 33 +++++++ test/misc/reconfigure.test.lua | 9 ++ test/reload_evolution/storage.result | 16 ++-- test/reload_evolution/storage.test.lua | 14 +-- test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ test/router/boot_replica_first.test.lua | 42 +++++++++ vshard/storage/init.lua | 21 ++++- vshard/storage/reload_evolution.lua | 8 ++ 9 files changed, 297 insertions(+), 13 deletions(-) create mode 100644 test/router/boot_replica_first.result create mode 100644 test/router/boot_replica_first.test.lua diff --git a/test/lua_libs/storage_template.lua b/test/lua_libs/storage_template.lua index 29a753d..84e4180 100644 --- a/test/lua_libs/storage_template.lua +++ b/test/lua_libs/storage_template.lua @@ -1,5 +1,7 @@ #!/usr/bin/env tarantool +local luri = require('uri') + NAME = require('fio').basename(arg[0], '.lua') fiber = require('fiber') test_run = require('test_run').new() @@ -10,12 +12,63 @@ log = require('log') if not cfg.shard_index then cfg.shard_index = 'bucket_id' end +instance_uuid = util.name_to_uuid[NAME] + +-- +-- Bootstrap the instance exactly like vshard does. But don't +-- initialize any vshard-specific code. +-- +local function boot_like_vshard() + assert(type(box.cfg) == 'function') + for rs_uuid, rs in pairs(cfg.sharding) do + for replica_uuid, replica in pairs(rs.replicas) do + if replica_uuid == instance_uuid then + local box_cfg = {replication = {}} + box_cfg.instance_uuid = replica_uuid + box_cfg.replicaset_uuid = rs_uuid + box_cfg.listen = replica.uri + box_cfg.read_only = not replica.master + box_cfg.replication_connect_quorum = 0 + box_cfg.replication_timeout = 0.1 + for _, replica in pairs(rs.replicas) do + table.insert(box_cfg.replication, replica.uri) + end + box.cfg(box_cfg) + if not replica.master then + return + end + local uri = luri.parse(replica.uri) + box.schema.user.create(uri.login, { + password = uri.password, if_not_exists = true, + }) + box.schema.user.grant(uri.login, 'super') + return + end + end + end + assert(false) +end + +local omit_cfg = false +local i = 1 +while arg[i] ~= nil do + local key = arg[i] + i = i + 1 + if key == 'boot_before_cfg' then + boot_like_vshard() + omit_cfg = true + end +end vshard = require('vshard') echo_count = 0 cfg.replication_connect_timeout = 3 cfg.replication_timeout = 0.1 -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) + +if not omit_cfg then + vshard.storage.cfg(cfg, instance_uuid) +end + function bootstrap_storage(engine) box.once("testapp:schema:1", function() if rawget(_G, 'CHANGE_SPACE_IDS') then diff --git a/test/misc/reconfigure.result b/test/misc/reconfigure.result index be58eca..168be5d 100644 --- a/test/misc/reconfigure.result +++ b/test/misc/reconfigure.result @@ -277,6 +277,14 @@ box.cfg.replication --- - - storage:storage at 127.0.0.1:3301 ... +box.cfg.read_only +--- +- false +... +#box.space._bucket:on_replace() +--- +- 1 +... _ = test_run:switch('storage_2_a') --- ... @@ -303,6 +311,15 @@ box.cfg.replication - - storage:storage at 127.0.0.1:3303 - storage:storage at 127.0.0.1:3304 ... +box.cfg.read_only +--- +- true +... +-- Should be zero on the slave node. Even though earlier the node was a master. +#box.space._bucket:on_replace() +--- +- 0 +... _ = test_run:switch('storage_2_b') --- ... @@ -329,6 +346,14 @@ box.cfg.replication - - storage:storage at 127.0.0.1:3303 - storage:storage at 127.0.0.1:3304 ... +box.cfg.read_only +--- +- false +... +#box.space._bucket:on_replace() +--- +- 1 +... _ = test_run:switch('storage_3_a') --- ... @@ -354,6 +379,14 @@ box.cfg.replication --- - - storage:storage at 127.0.0.1:3306 ... +box.cfg.read_only +--- +- false +... +#box.space._bucket:on_replace() +--- +- 1 +... _ = test_run:switch('router_1') --- ... diff --git a/test/misc/reconfigure.test.lua b/test/misc/reconfigure.test.lua index 1b894c8..e891010 100644 --- a/test/misc/reconfigure.test.lua +++ b/test/misc/reconfigure.test.lua @@ -111,6 +111,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end table.sort(uris) uris box.cfg.replication +box.cfg.read_only +#box.space._bucket:on_replace() _ = test_run:switch('storage_2_a') info = vshard.storage.info() @@ -119,6 +121,9 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end table.sort(uris) uris box.cfg.replication +box.cfg.read_only +-- Should be zero on the slave node. Even though earlier the node was a master. +#box.space._bucket:on_replace() _ = test_run:switch('storage_2_b') info = vshard.storage.info() @@ -127,6 +132,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end table.sort(uris) uris box.cfg.replication +box.cfg.read_only +#box.space._bucket:on_replace() _ = test_run:switch('storage_3_a') info = vshard.storage.info() @@ -135,6 +142,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end table.sort(uris) uris box.cfg.replication +box.cfg.read_only +#box.space._bucket:on_replace() _ = test_run:switch('router_1') info = vshard.router.info() diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result index dde0a1f..4652c4f 100644 --- a/test/reload_evolution/storage.result +++ b/test/reload_evolution/storage.result @@ -86,16 +86,22 @@ package.loaded['vshard.storage'] = nil vshard.storage = require("vshard.storage") --- ... --- Should be nil. Because there was a bug that reload always reapplied the last --- migration callback. When it was fixed, the last callback wasn't called twice. --- But since the callback was only one, now nothing is called, and nothing is --- logged. -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil --- - true ... vshard.storage.internal.reload_version --- +- 2 +... +-- +-- gh-237: should be only one trigger. During gh-237 the trigger installation +-- became conditional and therefore required to remember the current trigger +-- somewhere. When reloaded from the old version, the trigger needed to be +-- fetched from _bucket:on_replace(). +-- +#box.space._bucket:on_replace() +--- - 1 ... -- Make sure storage operates well. diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua index 7858112..06f7117 100644 --- a/test/reload_evolution/storage.test.lua +++ b/test/reload_evolution/storage.test.lua @@ -35,12 +35,16 @@ box.space.test:insert({42, bucket_id_to_move}) package.path = original_package_path package.loaded['vshard.storage'] = nil vshard.storage = require("vshard.storage") --- Should be nil. Because there was a bug that reload always reapplied the last --- migration callback. When it was fixed, the last callback wasn't called twice. --- But since the callback was only one, now nothing is called, and nothing is --- logged. -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil vshard.storage.internal.reload_version +-- +-- gh-237: should be only one trigger. During gh-237 the trigger installation +-- became conditional and therefore required to remember the current trigger +-- somewhere. When reloaded from the old version, the trigger needed to be +-- fetched from _bucket:on_replace(). +-- +#box.space._bucket:on_replace() + -- Make sure storage operates well. vshard.storage.bucket_force_drop(2000) vshard.storage.bucket_force_create(2000) diff --git a/test/router/boot_replica_first.result b/test/router/boot_replica_first.result new file mode 100644 index 0000000..1705230 --- /dev/null +++ b/test/router/boot_replica_first.result @@ -0,0 +1,112 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } + | --- + | ... +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) + | --- + | ... +util = require('util') + | --- + | ... +util.wait_master(test_run, REPLICASET_1, 'box_1_a') + | --- + | ... +_ = test_run:cmd("create server router with script='router/router_2.lua'") + | --- + | ... +_ = test_run:cmd("start server router") + | --- + | ... + +-- +-- gh-237: replica should be able to boot before master. Before the issue was +-- fixed, replica always tried to install a trigger on _bucket space even when +-- it was not created on a master yet - that lead to an exception in +-- storage.cfg. Now it should not install the trigger at all, because anyway it +-- is not needed on replica for anything. +-- + +test_run:switch('box_1_b') + | --- + | - true + | ... +vshard.storage.cfg(cfg, instance_uuid) + | --- + | ... +-- _bucket is not created yet. Will fail. +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) + | --- + | - attempt to index field '_bucket' (a nil value) + | ... + +test_run:switch('default') + | --- + | - true + | ... +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') + | --- + | ... + +test_run:switch('box_1_a') + | --- + | - true + | ... +vshard.storage.cfg(cfg, instance_uuid) + | --- + | ... + +test_run:switch('box_1_b') + | --- + | - true + | ... +test_run:wait_lsn('box_1_b', 'box_1_a') + | --- + | ... +-- Fails, but gracefully. Because the bucket is not found here. +vshard.storage.call(1, 'read', 'echo', {100}) + | --- + | - null + | - bucket_id: 1 + | reason: Not found + | code: 1 + | type: ShardingError + | message: 'Cannot perform action with bucket 1, reason: Not found' + | name: WRONG_BUCKET + | ... +-- Should not have triggers. +#box.space._bucket:on_replace() + | --- + | - 0 + | ... + +test_run:switch('router') + | --- + | - true + | ... +vshard.router.bootstrap() + | --- + | - true + | ... +vshard.router.callro(1, 'echo', {100}) + | --- + | - 100 + | ... + +test_run:switch("default") + | --- + | - true + | ... +test_run:cmd('stop server router') + | --- + | - true + | ... +test_run:cmd('delete server router') + | --- + | - true + | ... +test_run:drop_cluster(REPLICASET_1) + | --- + | ... diff --git a/test/router/boot_replica_first.test.lua b/test/router/boot_replica_first.test.lua new file mode 100644 index 0000000..7b1b3fd --- /dev/null +++ b/test/router/boot_replica_first.test.lua @@ -0,0 +1,42 @@ +test_run = require('test_run').new() +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) +util = require('util') +util.wait_master(test_run, REPLICASET_1, 'box_1_a') +_ = test_run:cmd("create server router with script='router/router_2.lua'") +_ = test_run:cmd("start server router") + +-- +-- gh-237: replica should be able to boot before master. Before the issue was +-- fixed, replica always tried to install a trigger on _bucket space even when +-- it was not created on a master yet - that lead to an exception in +-- storage.cfg. Now it should not install the trigger at all, because anyway it +-- is not needed on replica for anything. +-- + +test_run:switch('box_1_b') +vshard.storage.cfg(cfg, instance_uuid) +-- _bucket is not created yet. Will fail. +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) + +test_run:switch('default') +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') + +test_run:switch('box_1_a') +vshard.storage.cfg(cfg, instance_uuid) + +test_run:switch('box_1_b') +test_run:wait_lsn('box_1_b', 'box_1_a') +-- Fails, but gracefully. Because the bucket is not found here. +vshard.storage.call(1, 'read', 'echo', {100}) +-- Should not have triggers. +#box.space._bucket:on_replace() + +test_run:switch('router') +vshard.router.bootstrap() +vshard.router.callro(1, 'echo', {100}) + +test_run:switch("default") +test_run:cmd('stop server router') +test_run:cmd('delete server router') +test_run:drop_cluster(REPLICASET_1) diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index c6a78fe..5464824 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -94,6 +94,17 @@ if not M then -- detect that _bucket was not changed between yields. -- bucket_generation = 0, + -- + -- Reference to the function used as on_replace trigger on + -- _bucket space. It is used to replace the trigger with + -- a new function when reload happens. It is kept + -- explicitly because the old function is deleted on + -- reload from the global namespace. On the other hand, it + -- is still stored in _bucket:on_replace() somewhere, but + -- it is not known where. The only 100% way to be able to + -- replace the old function is to keep its reference. + -- + bucket_on_replace = nil, ------------------- Garbage collection ------------------- -- Fiber to remove garbage buckets data. @@ -2435,8 +2446,14 @@ local function storage_cfg(cfg, this_replica_uuid, is_reload) local uri = luri.parse(this_replica.uri) schema_upgrade(is_master, uri.login, uri.password) - local old_trigger = box.space._bucket:on_replace()[1] - box.space._bucket:on_replace(bucket_generation_increment, old_trigger) + if M.bucket_on_replace then + box.space._bucket:on_replace(nil, M.bucket_on_replace) + M.bucket_on_replace = nil + end + if is_master then + box.space._bucket:on_replace(bucket_generation_increment) + M.bucket_on_replace = bucket_generation_increment + end lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) lreplicaset.outdate_replicasets(M.replicasets) diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua index 1abc9e2..f38af74 100644 --- a/vshard/storage/reload_evolution.lua +++ b/vshard/storage/reload_evolution.lua @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) -- Code to update Lua objects. end +migrations[#migrations + 1] = function(M) + local bucket = box.space._bucket + if bucket then + assert(M.bucket_on_replace == nil) + M.bucket_on_replace = bucket:on_replace()[1] + end +end + -- -- Perform an update based on a version stored in `M` (internals). -- @param M Old module internals which should be updated. -- 2.21.1 (Apple Git-122.3) From olegrok at tarantool.org Thu Aug 6 09:40:02 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Thu, 6 Aug 2020 09:40:02 +0300 Subject: [Tarantool-patches] [PATCH v2 vshard 1/2] storage: fix reload applying migration twice In-Reply-To: References: Message-ID: <6179c32d-0970-db1b-09a4-178d8ab102b0@tarantool.org> Hi! Thanks for your patch. LGTM. On 06/08/2020 01:15, Vladislav Shpilevoy wrote: > Reload evolution is a subsystem for storage hot code reload. When > code is reloaded, it applies migration scripts one by one. But > there was a bug - the last migration script was applied even when > code version didn't really change. > > The bug didn't affect anything, since so far the reload evolution > was not used for anything and had just one dummy migration script > doing nothing. > > Now a first migration script is going to be added, so the bug is > fixed before that. > > Needed for #237 > --- > test/reload_evolution/storage.result | 6 +++++- > test/reload_evolution/storage.test.lua | 6 +++++- > vshard/storage/reload_evolution.lua | 2 +- > 3 files changed, 11 insertions(+), 3 deletions(-) > > diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result > index ebf4fdc..dde0a1f 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -86,7 +86,11 @@ package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- > ... > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil > +-- Should be nil. Because there was a bug that reload always reapplied the last > +-- migration callback. When it was fixed, the last callback wasn't called twice. > +-- But since the callback was only one, now nothing is called, and nothing is > +-- logged. > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil > --- > - true > ... > diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua > index 56c1693..7858112 100644 > --- a/test/reload_evolution/storage.test.lua > +++ b/test/reload_evolution/storage.test.lua > @@ -35,7 +35,11 @@ box.space.test:insert({42, bucket_id_to_move}) > package.path = original_package_path > package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil > +-- Should be nil. Because there was a bug that reload always reapplied the last > +-- migration callback. When it was fixed, the last callback wasn't called twice. > +-- But since the callback was only one, now nothing is called, and nothing is > +-- logged. > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil > vshard.storage.internal.reload_version > -- Make sure storage operates well. > vshard.storage.bucket_force_drop(2000) > diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua > index 5d09b11..1abc9e2 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -33,7 +33,7 @@ local function upgrade(M) > log.error(err_msg) > error(err_msg) > end > - for i = start_version, #migrations do > + for i = start_version + 1, #migrations do > local ok, err = pcall(migrations[i], M) > if ok then > log.info('vshard.storage.reload_evolution: upgraded to %d version', From olegrok at tarantool.org Thu Aug 6 09:40:12 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Thu, 6 Aug 2020 09:40:12 +0300 Subject: [Tarantool-patches] [PATCH v2 vshard 2/2] storage: allow replica to boot before master In-Reply-To: <0cd777f3990939b32d96bf964dea55e8073e65bb.1596665591.git.v.shpilevoy@tarantool.org> References: <0cd777f3990939b32d96bf964dea55e8073e65bb.1596665591.git.v.shpilevoy@tarantool.org> Message-ID: <894ab65e-64f2-e593-4f58-1c239ce3994f@tarantool.org> Thanks for your patch. LGTM. On 06/08/2020 01:15, Vladislav Shpilevoy wrote: > Before the patch a replica couldn't call vshard.storage.cfg() > before the master is bootstrapped. That could lead to an exception > in the middle of vshard.storage.cfg(). > > It sounds not so bad and even looks unreachable, because replica > bootstrap anyway is blocked in box.cfg() until a master node is > started, otherwise the replica instance is terminated. > But the box.cfg bootstrap != vshard bootstrap. > > It could be that both replica and master nodes are already > bootstrapped and working, but vshard.storage.cfg() wasn't called > yet. In that case vshard.storage.cfg() on the replica wouldn't > block in box.cfg(), and would happily fail a few lines later in > vshard.storage.cfg() at attempt to touch not existing > box.space._bucket. > > That was the case for cartridge. The framework configures > instances in its own way before vshard.storage.cfg(). Then it > calls vshard.storage.cfg() on them in arbitrary order, and > sometimes replicas were configured earlier than the master. > > The fail is fixed by simply skipping the _bucket's trigger > installation on the replica node. Because so far it is not needed > here for anything. The trigger's sole purpose is to increment > bucket generation used only for DML on _bucket on the master node. > > Now vshard.storage.cfg() on replica is able to finish before > master does the same. However the replica still won't be able to > handle vshard.storage.* requests until receives vshard schema > from master. > > Closes #237 > --- > test/lua_libs/storage_template.lua | 55 +++++++++++- > test/misc/reconfigure.result | 33 +++++++ > test/misc/reconfigure.test.lua | 9 ++ > test/reload_evolution/storage.result | 16 ++-- > test/reload_evolution/storage.test.lua | 14 +-- > test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ > test/router/boot_replica_first.test.lua | 42 +++++++++ > vshard/storage/init.lua | 21 ++++- > vshard/storage/reload_evolution.lua | 8 ++ > 9 files changed, 297 insertions(+), 13 deletions(-) > create mode 100644 test/router/boot_replica_first.result > create mode 100644 test/router/boot_replica_first.test.lua > > diff --git a/test/lua_libs/storage_template.lua b/test/lua_libs/storage_template.lua > index 29a753d..84e4180 100644 > --- a/test/lua_libs/storage_template.lua > +++ b/test/lua_libs/storage_template.lua > @@ -1,5 +1,7 @@ > #!/usr/bin/env tarantool > > +local luri = require('uri') > + > NAME = require('fio').basename(arg[0], '.lua') > fiber = require('fiber') > test_run = require('test_run').new() > @@ -10,12 +12,63 @@ log = require('log') > if not cfg.shard_index then > cfg.shard_index = 'bucket_id' > end > +instance_uuid = util.name_to_uuid[NAME] > + > +-- > +-- Bootstrap the instance exactly like vshard does. But don't > +-- initialize any vshard-specific code. > +-- > +local function boot_like_vshard() > + assert(type(box.cfg) == 'function') > + for rs_uuid, rs in pairs(cfg.sharding) do > + for replica_uuid, replica in pairs(rs.replicas) do > + if replica_uuid == instance_uuid then > + local box_cfg = {replication = {}} > + box_cfg.instance_uuid = replica_uuid > + box_cfg.replicaset_uuid = rs_uuid > + box_cfg.listen = replica.uri > + box_cfg.read_only = not replica.master > + box_cfg.replication_connect_quorum = 0 > + box_cfg.replication_timeout = 0.1 > + for _, replica in pairs(rs.replicas) do > + table.insert(box_cfg.replication, replica.uri) > + end > + box.cfg(box_cfg) > + if not replica.master then > + return > + end > + local uri = luri.parse(replica.uri) > + box.schema.user.create(uri.login, { > + password = uri.password, if_not_exists = true, > + }) > + box.schema.user.grant(uri.login, 'super') > + return > + end > + end > + end > + assert(false) > +end > + > +local omit_cfg = false > +local i = 1 > +while arg[i] ~= nil do > + local key = arg[i] > + i = i + 1 > + if key == 'boot_before_cfg' then > + boot_like_vshard() > + omit_cfg = true > + end > +end > > vshard = require('vshard') > echo_count = 0 > cfg.replication_connect_timeout = 3 > cfg.replication_timeout = 0.1 > -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) > + > +if not omit_cfg then > + vshard.storage.cfg(cfg, instance_uuid) > +end > + > function bootstrap_storage(engine) > box.once("testapp:schema:1", function() > if rawget(_G, 'CHANGE_SPACE_IDS') then > diff --git a/test/misc/reconfigure.result b/test/misc/reconfigure.result > index be58eca..168be5d 100644 > --- a/test/misc/reconfigure.result > +++ b/test/misc/reconfigure.result > @@ -277,6 +277,14 @@ box.cfg.replication > --- > - -storage:storage at 127.0.0.1:3301 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('storage_2_a') > --- > ... > @@ -303,6 +311,15 @@ box.cfg.replication > - -storage:storage at 127.0.0.1:3303 > -storage:storage at 127.0.0.1:3304 > ... > +box.cfg.read_only > +--- > +- true > +... > +-- Should be zero on the slave node. Even though earlier the node was a master. > +#box.space._bucket:on_replace() > +--- > +- 0 > +... > _ = test_run:switch('storage_2_b') > --- > ... > @@ -329,6 +346,14 @@ box.cfg.replication > - -storage:storage at 127.0.0.1:3303 > -storage:storage at 127.0.0.1:3304 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('storage_3_a') > --- > ... > @@ -354,6 +379,14 @@ box.cfg.replication > --- > - -storage:storage at 127.0.0.1:3306 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('router_1') > --- > ... > diff --git a/test/misc/reconfigure.test.lua b/test/misc/reconfigure.test.lua > index 1b894c8..e891010 100644 > --- a/test/misc/reconfigure.test.lua > +++ b/test/misc/reconfigure.test.lua > @@ -111,6 +111,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_2_a') > info = vshard.storage.info() > @@ -119,6 +121,9 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +-- Should be zero on the slave node. Even though earlier the node was a master. > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_2_b') > info = vshard.storage.info() > @@ -127,6 +132,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_3_a') > info = vshard.storage.info() > @@ -135,6 +142,8 @@ for k,v in pairs(info.replicasets) do table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('router_1') > info = vshard.router.info() > diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result > index dde0a1f..4652c4f 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -86,16 +86,22 @@ package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- > ... > --- Should be nil. Because there was a bug that reload always reapplied the last > --- migration callback. When it was fixed, the last callback wasn't called twice. > --- But since the callback was only one, now nothing is called, and nothing is > --- logged. > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil > --- > - true > ... > vshard.storage.internal.reload_version > --- > +- 2 > +... > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger installation > +-- became conditional and therefore required to remember the current trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > +--- > - 1 > ... > -- Make sure storage operates well. > diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua > index 7858112..06f7117 100644 > --- a/test/reload_evolution/storage.test.lua > +++ b/test/reload_evolution/storage.test.lua > @@ -35,12 +35,16 @@ box.space.test:insert({42, bucket_id_to_move}) > package.path = original_package_path > package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- Should be nil. Because there was a bug that reload always reapplied the last > --- migration callback. When it was fixed, the last callback wasn't called twice. > --- But since the callback was only one, now nothing is called, and nothing is > --- logged. > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') == nil > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil > vshard.storage.internal.reload_version > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger installation > +-- became conditional and therefore required to remember the current trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > + > -- Make sure storage operates well. > vshard.storage.bucket_force_drop(2000) > vshard.storage.bucket_force_create(2000) > diff --git a/test/router/boot_replica_first.result b/test/router/boot_replica_first.result > new file mode 100644 > index 0000000..1705230 > --- /dev/null > +++ b/test/router/boot_replica_first.result > @@ -0,0 +1,112 @@ > +-- test-run result file version 2 > +test_run = require('test_run').new() > + | --- > + | ... > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > + | --- > + | ... > +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) > + | --- > + | ... > +util = require('util') > + | --- > + | ... > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > + | --- > + | ... > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > + | --- > + | ... > +_ = test_run:cmd("start server router") > + | --- > + | ... > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue was > +-- fixed, replica always tried to install a trigger on _bucket space even when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + | --- > + | - attempt to index field '_bucket' (a nil value) > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + | --- > + | ... > + > +test_run:switch('box_1_a') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +test_run:wait_lsn('box_1_b', 'box_1_a') > + | --- > + | ... > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > + | --- > + | - null > + | - bucket_id: 1 > + | reason: Not found > + | code: 1 > + | type: ShardingError > + | message: 'Cannot perform action with bucket 1, reason: Not found' > + | name: WRONG_BUCKET > + | ... > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + | --- > + | - 0 > + | ... > + > +test_run:switch('router') > + | --- > + | - true > + | ... > +vshard.router.bootstrap() > + | --- > + | - true > + | ... > +vshard.router.callro(1, 'echo', {100}) > + | --- > + | - 100 > + | ... > + > +test_run:switch("default") > + | --- > + | - true > + | ... > +test_run:cmd('stop server router') > + | --- > + | - true > + | ... > +test_run:cmd('delete server router') > + | --- > + | - true > + | ... > +test_run:drop_cluster(REPLICASET_1) > + | --- > + | ... > diff --git a/test/router/boot_replica_first.test.lua b/test/router/boot_replica_first.test.lua > new file mode 100644 > index 0000000..7b1b3fd > --- /dev/null > +++ b/test/router/boot_replica_first.test.lua > @@ -0,0 +1,42 @@ > +test_run = require('test_run').new() > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > +test_run:create_cluster(REPLICASET_1, 'router', {args = 'boot_before_cfg'}) > +util = require('util') > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > +_ = test_run:cmd("start server router") > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue was > +-- fixed, replica always tried to install a trigger on _bucket space even when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > +vshard.storage.cfg(cfg, instance_uuid) > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + > +test_run:switch('default') > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + > +test_run:switch('box_1_a') > +vshard.storage.cfg(cfg, instance_uuid) > + > +test_run:switch('box_1_b') > +test_run:wait_lsn('box_1_b', 'box_1_a') > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + > +test_run:switch('router') > +vshard.router.bootstrap() > +vshard.router.callro(1, 'echo', {100}) > + > +test_run:switch("default") > +test_run:cmd('stop server router') > +test_run:cmd('delete server router') > +test_run:drop_cluster(REPLICASET_1) > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index c6a78fe..5464824 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -94,6 +94,17 @@ if not M then > -- detect that _bucket was not changed between yields. > -- > bucket_generation = 0, > + -- > + -- Reference to the function used as on_replace trigger on > + -- _bucket space. It is used to replace the trigger with > + -- a new function when reload happens. It is kept > + -- explicitly because the old function is deleted on > + -- reload from the global namespace. On the other hand, it > + -- is still stored in _bucket:on_replace() somewhere, but > + -- it is not known where. The only 100% way to be able to > + -- replace the old function is to keep its reference. > + -- > + bucket_on_replace = nil, > > ------------------- Garbage collection ------------------- > -- Fiber to remove garbage buckets data. > @@ -2435,8 +2446,14 @@ local function storage_cfg(cfg, this_replica_uuid, is_reload) > local uri = luri.parse(this_replica.uri) > schema_upgrade(is_master, uri.login, uri.password) > > - local old_trigger = box.space._bucket:on_replace()[1] > - box.space._bucket:on_replace(bucket_generation_increment, old_trigger) > + if M.bucket_on_replace then > + box.space._bucket:on_replace(nil, M.bucket_on_replace) > + M.bucket_on_replace = nil > + end > + if is_master then > + box.space._bucket:on_replace(bucket_generation_increment) > + M.bucket_on_replace = bucket_generation_increment > + end > > lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) > lreplicaset.outdate_replicasets(M.replicasets) > diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua > index 1abc9e2..f38af74 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) > -- Code to update Lua objects. > end > > +migrations[#migrations + 1] = function(M) > + local bucket = box.space._bucket > + if bucket then > + assert(M.bucket_on_replace == nil) > + M.bucket_on_replace = bucket:on_replace()[1] > + end > +end > + > -- > -- Perform an update based on a version stored in `M` (internals). > -- @param M Old module internals which should be updated. From yaroslav.dynnikov at tarantool.org Thu Aug 6 11:29:33 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Thu, 6 Aug 2020 11:29:33 +0300 Subject: [Tarantool-patches] [PATCH v2 vshard 1/2] storage: fix reload applying migration twice In-Reply-To: References: Message-ID: Hi! Nice patchset, LGTM. Best regards Yaroslav Dynnikov On Thu, 6 Aug 2020 at 01:15, Vladislav Shpilevoy wrote: > Reload evolution is a subsystem for storage hot code reload. When > code is reloaded, it applies migration scripts one by one. But > there was a bug - the last migration script was applied even when > code version didn't really change. > > The bug didn't affect anything, since so far the reload evolution > was not used for anything and had just one dummy migration script > doing nothing. > > Now a first migration script is going to be added, so the bug is > fixed before that. > > Needed for #237 > --- > test/reload_evolution/storage.result | 6 +++++- > test/reload_evolution/storage.test.lua | 6 +++++- > vshard/storage/reload_evolution.lua | 2 +- > 3 files changed, 11 insertions(+), 3 deletions(-) > > diff --git a/test/reload_evolution/storage.result > b/test/reload_evolution/storage.result > index ebf4fdc..dde0a1f 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -86,7 +86,11 @@ package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- > ... > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') ~= nil > +-- Should be nil. Because there was a bug that reload always reapplied > the last > +-- migration callback. When it was fixed, the last callback wasn't called > twice. > +-- But since the callback was only one, now nothing is called, and > nothing is > +-- logged. > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') == nil > --- > - true > ... > diff --git a/test/reload_evolution/storage.test.lua > b/test/reload_evolution/storage.test.lua > index 56c1693..7858112 100644 > --- a/test/reload_evolution/storage.test.lua > +++ b/test/reload_evolution/storage.test.lua > @@ -35,7 +35,11 @@ box.space.test:insert({42, bucket_id_to_move}) > package.path = original_package_path > package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') ~= nil > +-- Should be nil. Because there was a bug that reload always reapplied > the last > +-- migration callback. When it was fixed, the last callback wasn't called > twice. > +-- But since the callback was only one, now nothing is called, and > nothing is > +-- logged. > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') == nil > vshard.storage.internal.reload_version > -- Make sure storage operates well. > vshard.storage.bucket_force_drop(2000) > diff --git a/vshard/storage/reload_evolution.lua > b/vshard/storage/reload_evolution.lua > index 5d09b11..1abc9e2 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -33,7 +33,7 @@ local function upgrade(M) > log.error(err_msg) > error(err_msg) > end > - for i = start_version, #migrations do > + for i = start_version + 1, #migrations do > local ok, err = pcall(migrations[i], M) > if ok then > log.info('vshard.storage.reload_evolution: upgraded to %d > version', > -- > 2.21.1 (Apple Git-122.3) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaroslav.dynnikov at tarantool.org Thu Aug 6 11:30:12 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Thu, 6 Aug 2020 11:30:12 +0300 Subject: [Tarantool-patches] [PATCH v2 vshard 2/2] storage: allow replica to boot before master In-Reply-To: <0cd777f3990939b32d96bf964dea55e8073e65bb.1596665591.git.v.shpilevoy@tarantool.org> References: <0cd777f3990939b32d96bf964dea55e8073e65bb.1596665591.git.v.shpilevoy@tarantool.org> Message-ID: Thanks for the patch, LGTM. Best regards Yaroslav Dynnikov On Thu, 6 Aug 2020 at 01:15, Vladislav Shpilevoy wrote: > Before the patch a replica couldn't call vshard.storage.cfg() > before the master is bootstrapped. That could lead to an exception > in the middle of vshard.storage.cfg(). > > It sounds not so bad and even looks unreachable, because replica > bootstrap anyway is blocked in box.cfg() until a master node is > started, otherwise the replica instance is terminated. > But the box.cfg bootstrap != vshard bootstrap. > > It could be that both replica and master nodes are already > bootstrapped and working, but vshard.storage.cfg() wasn't called > yet. In that case vshard.storage.cfg() on the replica wouldn't > block in box.cfg(), and would happily fail a few lines later in > vshard.storage.cfg() at attempt to touch not existing > box.space._bucket. > > That was the case for cartridge. The framework configures > instances in its own way before vshard.storage.cfg(). Then it > calls vshard.storage.cfg() on them in arbitrary order, and > sometimes replicas were configured earlier than the master. > > The fail is fixed by simply skipping the _bucket's trigger > installation on the replica node. Because so far it is not needed > here for anything. The trigger's sole purpose is to increment > bucket generation used only for DML on _bucket on the master node. > > Now vshard.storage.cfg() on replica is able to finish before > master does the same. However the replica still won't be able to > handle vshard.storage.* requests until receives vshard schema > from master. > > Closes #237 > --- > test/lua_libs/storage_template.lua | 55 +++++++++++- > test/misc/reconfigure.result | 33 +++++++ > test/misc/reconfigure.test.lua | 9 ++ > test/reload_evolution/storage.result | 16 ++-- > test/reload_evolution/storage.test.lua | 14 +-- > test/router/boot_replica_first.result | 112 ++++++++++++++++++++++++ > test/router/boot_replica_first.test.lua | 42 +++++++++ > vshard/storage/init.lua | 21 ++++- > vshard/storage/reload_evolution.lua | 8 ++ > 9 files changed, 297 insertions(+), 13 deletions(-) > create mode 100644 test/router/boot_replica_first.result > create mode 100644 test/router/boot_replica_first.test.lua > > diff --git a/test/lua_libs/storage_template.lua > b/test/lua_libs/storage_template.lua > index 29a753d..84e4180 100644 > --- a/test/lua_libs/storage_template.lua > +++ b/test/lua_libs/storage_template.lua > @@ -1,5 +1,7 @@ > #!/usr/bin/env tarantool > > +local luri = require('uri') > + > NAME = require('fio').basename(arg[0], '.lua') > fiber = require('fiber') > test_run = require('test_run').new() > @@ -10,12 +12,63 @@ log = require('log') > if not cfg.shard_index then > cfg.shard_index = 'bucket_id' > end > +instance_uuid = util.name_to_uuid[NAME] > + > +-- > +-- Bootstrap the instance exactly like vshard does. But don't > +-- initialize any vshard-specific code. > +-- > +local function boot_like_vshard() > + assert(type(box.cfg) == 'function') > + for rs_uuid, rs in pairs(cfg.sharding) do > + for replica_uuid, replica in pairs(rs.replicas) do > + if replica_uuid == instance_uuid then > + local box_cfg = {replication = {}} > + box_cfg.instance_uuid = replica_uuid > + box_cfg.replicaset_uuid = rs_uuid > + box_cfg.listen = replica.uri > + box_cfg.read_only = not replica.master > + box_cfg.replication_connect_quorum = 0 > + box_cfg.replication_timeout = 0.1 > + for _, replica in pairs(rs.replicas) do > + table.insert(box_cfg.replication, replica.uri) > + end > + box.cfg(box_cfg) > + if not replica.master then > + return > + end > + local uri = luri.parse(replica.uri) > + box.schema.user.create(uri.login, { > + password = uri.password, if_not_exists = true, > + }) > + box.schema.user.grant(uri.login, 'super') > + return > + end > + end > + end > + assert(false) > +end > + > +local omit_cfg = false > +local i = 1 > +while arg[i] ~= nil do > + local key = arg[i] > + i = i + 1 > + if key == 'boot_before_cfg' then > + boot_like_vshard() > + omit_cfg = true > + end > +end > > vshard = require('vshard') > echo_count = 0 > cfg.replication_connect_timeout = 3 > cfg.replication_timeout = 0.1 > -vshard.storage.cfg(cfg, util.name_to_uuid[NAME]) > + > +if not omit_cfg then > + vshard.storage.cfg(cfg, instance_uuid) > +end > + > function bootstrap_storage(engine) > box.once("testapp:schema:1", function() > if rawget(_G, 'CHANGE_SPACE_IDS') then > diff --git a/test/misc/reconfigure.result b/test/misc/reconfigure.result > index be58eca..168be5d 100644 > --- a/test/misc/reconfigure.result > +++ b/test/misc/reconfigure.result > @@ -277,6 +277,14 @@ box.cfg.replication > --- > - - storage:storage at 127.0.0.1:3301 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('storage_2_a') > --- > ... > @@ -303,6 +311,15 @@ box.cfg.replication > - - storage:storage at 127.0.0.1:3303 > - storage:storage at 127.0.0.1:3304 > ... > +box.cfg.read_only > +--- > +- true > +... > +-- Should be zero on the slave node. Even though earlier the node was a > master. > +#box.space._bucket:on_replace() > +--- > +- 0 > +... > _ = test_run:switch('storage_2_b') > --- > ... > @@ -329,6 +346,14 @@ box.cfg.replication > - - storage:storage at 127.0.0.1:3303 > - storage:storage at 127.0.0.1:3304 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('storage_3_a') > --- > ... > @@ -354,6 +379,14 @@ box.cfg.replication > --- > - - storage:storage at 127.0.0.1:3306 > ... > +box.cfg.read_only > +--- > +- false > +... > +#box.space._bucket:on_replace() > +--- > +- 1 > +... > _ = test_run:switch('router_1') > --- > ... > diff --git a/test/misc/reconfigure.test.lua > b/test/misc/reconfigure.test.lua > index 1b894c8..e891010 100644 > --- a/test/misc/reconfigure.test.lua > +++ b/test/misc/reconfigure.test.lua > @@ -111,6 +111,8 @@ for k,v in pairs(info.replicasets) do > table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_2_a') > info = vshard.storage.info() > @@ -119,6 +121,9 @@ for k,v in pairs(info.replicasets) do > table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +-- Should be zero on the slave node. Even though earlier the node was a > master. > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_2_b') > info = vshard.storage.info() > @@ -127,6 +132,8 @@ for k,v in pairs(info.replicasets) do > table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('storage_3_a') > info = vshard.storage.info() > @@ -135,6 +142,8 @@ for k,v in pairs(info.replicasets) do > table.insert(uris, v.master.uri) end > table.sort(uris) > uris > box.cfg.replication > +box.cfg.read_only > +#box.space._bucket:on_replace() > > _ = test_run:switch('router_1') > info = vshard.router.info() > diff --git a/test/reload_evolution/storage.result > b/test/reload_evolution/storage.result > index dde0a1f..4652c4f 100644 > --- a/test/reload_evolution/storage.result > +++ b/test/reload_evolution/storage.result > @@ -86,16 +86,22 @@ package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- > ... > --- Should be nil. Because there was a bug that reload always reapplied > the last > --- migration callback. When it was fixed, the last callback wasn't called > twice. > --- But since the callback was only one, now nothing is called, and > nothing is > --- logged. > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') == nil > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') ~= nil > --- > - true > ... > vshard.storage.internal.reload_version > --- > +- 2 > +... > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger > installation > +-- became conditional and therefore required to remember the current > trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > +--- > - 1 > ... > -- Make sure storage operates well. > diff --git a/test/reload_evolution/storage.test.lua > b/test/reload_evolution/storage.test.lua > index 7858112..06f7117 100644 > --- a/test/reload_evolution/storage.test.lua > +++ b/test/reload_evolution/storage.test.lua > @@ -35,12 +35,16 @@ box.space.test:insert({42, bucket_id_to_move}) > package.path = original_package_path > package.loaded['vshard.storage'] = nil > vshard.storage = require("vshard.storage") > --- Should be nil. Because there was a bug that reload always reapplied > the last > --- migration callback. When it was fixed, the last callback wasn't called > twice. > --- But since the callback was only one, now nothing is called, and > nothing is > --- logged. > -test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') == nil > +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: > upgraded to') ~= nil > vshard.storage.internal.reload_version > +-- > +-- gh-237: should be only one trigger. During gh-237 the trigger > installation > +-- became conditional and therefore required to remember the current > trigger > +-- somewhere. When reloaded from the old version, the trigger needed to be > +-- fetched from _bucket:on_replace(). > +-- > +#box.space._bucket:on_replace() > + > -- Make sure storage operates well. > vshard.storage.bucket_force_drop(2000) > vshard.storage.bucket_force_create(2000) > diff --git a/test/router/boot_replica_first.result > b/test/router/boot_replica_first.result > new file mode 100644 > index 0000000..1705230 > --- /dev/null > +++ b/test/router/boot_replica_first.result > @@ -0,0 +1,112 @@ > +-- test-run result file version 2 > +test_run = require('test_run').new() > + | --- > + | ... > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > + | --- > + | ... > +test_run:create_cluster(REPLICASET_1, 'router', {args = > 'boot_before_cfg'}) > + | --- > + | ... > +util = require('util') > + | --- > + | ... > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > + | --- > + | ... > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > + | --- > + | ... > +_ = test_run:cmd("start server router") > + | --- > + | ... > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue > was > +-- fixed, replica always tried to install a trigger on _bucket space even > when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because > anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + | --- > + | - attempt to index field '_bucket' (a nil value) > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + | --- > + | ... > + > +test_run:switch('box_1_a') > + | --- > + | - true > + | ... > +vshard.storage.cfg(cfg, instance_uuid) > + | --- > + | ... > + > +test_run:switch('box_1_b') > + | --- > + | - true > + | ... > +test_run:wait_lsn('box_1_b', 'box_1_a') > + | --- > + | ... > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > + | --- > + | - null > + | - bucket_id: 1 > + | reason: Not found > + | code: 1 > + | type: ShardingError > + | message: 'Cannot perform action with bucket 1, reason: Not found' > + | name: WRONG_BUCKET > + | ... > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + | --- > + | - 0 > + | ... > + > +test_run:switch('router') > + | --- > + | - true > + | ... > +vshard.router.bootstrap() > + | --- > + | - true > + | ... > +vshard.router.callro(1, 'echo', {100}) > + | --- > + | - 100 > + | ... > + > +test_run:switch("default") > + | --- > + | - true > + | ... > +test_run:cmd('stop server router') > + | --- > + | - true > + | ... > +test_run:cmd('delete server router') > + | --- > + | - true > + | ... > +test_run:drop_cluster(REPLICASET_1) > + | --- > + | ... > diff --git a/test/router/boot_replica_first.test.lua > b/test/router/boot_replica_first.test.lua > new file mode 100644 > index 0000000..7b1b3fd > --- /dev/null > +++ b/test/router/boot_replica_first.test.lua > @@ -0,0 +1,42 @@ > +test_run = require('test_run').new() > +REPLICASET_1 = { 'box_1_a', 'box_1_b', 'box_1_c' } > +test_run:create_cluster(REPLICASET_1, 'router', {args = > 'boot_before_cfg'}) > +util = require('util') > +util.wait_master(test_run, REPLICASET_1, 'box_1_a') > +_ = test_run:cmd("create server router with script='router/router_2.lua'") > +_ = test_run:cmd("start server router") > + > +-- > +-- gh-237: replica should be able to boot before master. Before the issue > was > +-- fixed, replica always tried to install a trigger on _bucket space even > when > +-- it was not created on a master yet - that lead to an exception in > +-- storage.cfg. Now it should not install the trigger at all, because > anyway it > +-- is not needed on replica for anything. > +-- > + > +test_run:switch('box_1_b') > +vshard.storage.cfg(cfg, instance_uuid) > +-- _bucket is not created yet. Will fail. > +util.check_error(vshard.storage.call, 1, 'read', 'echo', {100}) > + > +test_run:switch('default') > +util.map_evals(test_run, {REPLICASET_1}, 'bootstrap_storage(\'memtx\')') > + > +test_run:switch('box_1_a') > +vshard.storage.cfg(cfg, instance_uuid) > + > +test_run:switch('box_1_b') > +test_run:wait_lsn('box_1_b', 'box_1_a') > +-- Fails, but gracefully. Because the bucket is not found here. > +vshard.storage.call(1, 'read', 'echo', {100}) > +-- Should not have triggers. > +#box.space._bucket:on_replace() > + > +test_run:switch('router') > +vshard.router.bootstrap() > +vshard.router.callro(1, 'echo', {100}) > + > +test_run:switch("default") > +test_run:cmd('stop server router') > +test_run:cmd('delete server router') > +test_run:drop_cluster(REPLICASET_1) > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index c6a78fe..5464824 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -94,6 +94,17 @@ if not M then > -- detect that _bucket was not changed between yields. > -- > bucket_generation = 0, > + -- > + -- Reference to the function used as on_replace trigger on > + -- _bucket space. It is used to replace the trigger with > + -- a new function when reload happens. It is kept > + -- explicitly because the old function is deleted on > + -- reload from the global namespace. On the other hand, it > + -- is still stored in _bucket:on_replace() somewhere, but > + -- it is not known where. The only 100% way to be able to > + -- replace the old function is to keep its reference. > + -- > + bucket_on_replace = nil, > > ------------------- Garbage collection ------------------- > -- Fiber to remove garbage buckets data. > @@ -2435,8 +2446,14 @@ local function storage_cfg(cfg, this_replica_uuid, > is_reload) > local uri = luri.parse(this_replica.uri) > schema_upgrade(is_master, uri.login, uri.password) > > - local old_trigger = box.space._bucket:on_replace()[1] > - box.space._bucket:on_replace(bucket_generation_increment, old_trigger) > + if M.bucket_on_replace then > + box.space._bucket:on_replace(nil, M.bucket_on_replace) > + M.bucket_on_replace = nil > + end > + if is_master then > + box.space._bucket:on_replace(bucket_generation_increment) > + M.bucket_on_replace = bucket_generation_increment > + end > > lreplicaset.rebind_replicasets(new_replicasets, M.replicasets) > lreplicaset.outdate_replicasets(M.replicasets) > diff --git a/vshard/storage/reload_evolution.lua > b/vshard/storage/reload_evolution.lua > index 1abc9e2..f38af74 100644 > --- a/vshard/storage/reload_evolution.lua > +++ b/vshard/storage/reload_evolution.lua > @@ -17,6 +17,14 @@ migrations[#migrations + 1] = function(M) > -- Code to update Lua objects. > end > > +migrations[#migrations + 1] = function(M) > + local bucket = box.space._bucket > + if bucket then > + assert(M.bucket_on_replace == nil) > + M.bucket_on_replace = bucket:on_replace()[1] > + end > +end > + > -- > -- Perform an update based on a version stored in `M` (internals). > -- @param M Old module internals which should be updated. > -- > 2.21.1 (Apple Git-122.3) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yaroslav.dynnikov at tarantool.org Thu Aug 6 16:19:55 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Thu, 6 Aug 2020 16:19:55 +0300 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported Message-ID: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> In the recent update of libcurl (2.5.0-278-g807c7fa58) its layout has changed: private function `Curl_version_init()` which used to fill-in info structure was eliminated. As a result, no symbols for `libcurl_la-version.o` remained used, so it wasn't included in tarantool binary. And `curl_version` and `curl_version_info` symbols went missing. According to libcurl naming conventions all exported symbols are named as `curl_*`. This patch lists them all explicitly in `exprots.h` and adds the test. Close #5223 Issue: https://github.com/tarantool/tarantool/issues/5223 Branch: https://github.com/tarantool/tarantool/tree/rosik/gh-5223-missing-curl-symbols --- src/exports.h | 81 ++++++++++ test/box-tap/gh-5223-curl-exports.test.lua | 177 +++++++++++++++++++++ 2 files changed, 258 insertions(+) create mode 100755 test/box-tap/gh-5223-curl-exports.test.lua diff --git a/src/exports.h b/src/exports.h index 7cf283e5b..7f0601f4f 100644 --- a/src/exports.h +++ b/src/exports.h @@ -113,6 +113,87 @@ EXPORT(csv_feed) EXPORT(csv_iterator_create) EXPORT(csv_next) EXPORT(csv_setopt) +EXPORT(curl_easy_cleanup) +EXPORT(curl_easy_duphandle) +EXPORT(curl_easy_escape) +EXPORT(curl_easy_getinfo) +EXPORT(curl_easy_init) +EXPORT(curl_easy_pause) +EXPORT(curl_easy_perform) +EXPORT(curl_easy_recv) +EXPORT(curl_easy_reset) +EXPORT(curl_easy_send) +EXPORT(curl_easy_setopt) +EXPORT(curl_easy_strerror) +EXPORT(curl_easy_unescape) +EXPORT(curl_easy_upkeep) +EXPORT(curl_escape) +EXPORT(curl_formadd) +EXPORT(curl_formfree) +EXPORT(curl_formget) +EXPORT(curl_free) +EXPORT(curl_getdate) +EXPORT(curl_getenv) +EXPORT(curl_global_cleanup) +EXPORT(curl_global_init) +EXPORT(curl_global_init_mem) +EXPORT(curl_global_sslset) +EXPORT(curl_maprintf) +EXPORT(curl_mfprintf) +EXPORT(curl_mime_addpart) +EXPORT(curl_mime_data) +EXPORT(curl_mime_data_cb) +EXPORT(curl_mime_encoder) +EXPORT(curl_mime_filedata) +EXPORT(curl_mime_filename) +EXPORT(curl_mime_free) +EXPORT(curl_mime_headers) +EXPORT(curl_mime_init) +EXPORT(curl_mime_name) +EXPORT(curl_mime_subparts) +EXPORT(curl_mime_type) +EXPORT(curl_mprintf) +EXPORT(curl_msnprintf) +EXPORT(curl_msprintf) +EXPORT(curl_multi_add_handle) +EXPORT(curl_multi_assign) +EXPORT(curl_multi_cleanup) +EXPORT(curl_multi_fdset) +EXPORT(curl_multi_info_read) +EXPORT(curl_multi_init) +EXPORT(curl_multi_perform) +EXPORT(curl_multi_poll) +EXPORT(curl_multi_remove_handle) +EXPORT(curl_multi_setopt) +EXPORT(curl_multi_socket) +EXPORT(curl_multi_socket_action) +EXPORT(curl_multi_socket_all) +EXPORT(curl_multi_strerror) +EXPORT(curl_multi_timeout) +EXPORT(curl_multi_wait) +EXPORT(curl_mvaprintf) +EXPORT(curl_mvfprintf) +EXPORT(curl_mvprintf) +EXPORT(curl_mvsnprintf) +EXPORT(curl_mvsprintf) +EXPORT(curl_pushheader_byname) +EXPORT(curl_pushheader_bynum) +EXPORT(curl_share_cleanup) +EXPORT(curl_share_init) +EXPORT(curl_share_setopt) +EXPORT(curl_share_strerror) +EXPORT(curl_slist_append) +EXPORT(curl_slist_free_all) +EXPORT(curl_strequal) +EXPORT(curl_strnequal) +EXPORT(curl_unescape) +EXPORT(curl_url) +EXPORT(curl_url_cleanup) +EXPORT(curl_url_dup) +EXPORT(curl_url_get) +EXPORT(curl_url_set) +EXPORT(curl_version) +EXPORT(curl_version_info) EXPORT(decimal_unpack) EXPORT(error_ref) EXPORT(error_set_prev) diff --git a/test/box-tap/gh-5223-curl-exports.test.lua b/test/box-tap/gh-5223-curl-exports.test.lua new file mode 100755 index 000000000..300d60b07 --- /dev/null +++ b/test/box-tap/gh-5223-curl-exports.test.lua @@ -0,0 +1,177 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local ffi = require('ffi') +ffi.cdef([[ + void *dlsym(void *handle, const char *symbol); + struct curl_version_info_data { + int age; /* see description below */ + const char *version; /* human readable string */ + unsigned int version_num; /* numeric representation */ + const char *host; /* human readable string */ + int features; /* bitmask, see below */ + char *ssl_version; /* human readable string */ + long ssl_version_num; /* not used, always zero */ + const char *libz_version; /* human readable string */ + const char * const *protocols; /* protocols */ + + /* when 'age' is CURLVERSION_SECOND or higher, the members below exist */ + const char *ares; /* human readable string */ + int ares_num; /* number */ + + /* when 'age' is CURLVERSION_THIRD or higher, the members below exist */ + const char *libidn; /* human readable string */ + + /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the members + below exist */ + int iconv_ver_num; /* '_libiconv_version' if iconv support enabled */ + + const char *libssh_version; /* human readable string */ + + /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the members + below exist */ + unsigned int brotli_ver_num; /* Numeric Brotli version + (MAJOR << 24) | (MINOR << 12) | PATCH */ + const char *brotli_version; /* human readable string. */ + + /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the members + below exist */ + unsigned int nghttp2_ver_num; /* Numeric nghttp2 version + (MAJOR << 16) | (MINOR << 8) | PATCH */ + const char *nghttp2_version; /* human readable string. */ + + const char *quic_version; /* human readable quic (+ HTTP/3) library + + version or NULL */ + + /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the members + below exist */ + const char *cainfo; /* the built-in default CURLOPT_CAINFO, might + be NULL */ + const char *capath; /* the built-in default CURLOPT_CAPATH, might + be NULL */ + }; + + struct curl_version_info_data *curl_version_info(int age); +]]) + +local info = ffi.C.curl_version_info(7) +local test = tap.test('curl-features') +test:plan(3) + +if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then + test:diag('ssl_version: ' .. ffi.string(info.ssl_version)) +end +if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then + test:diag('libz_version: ' .. ffi.string(info.libz_version)) +end + +local RTLD_DEFAULT +-- See `man 3 dlsym`: +-- RTLD_DEFAULT +-- Find the first occurrence of the desired symbol using the default +-- shared object search order. The search will include global symbols +-- in the executable and its dependencies, as well as symbols in shared +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. +if jit.os == "OSX" then + RTLD_DEFAULT = ffi.cast("void *", -2LL) +else + RTLD_DEFAULT = ffi.cast("void *", 0LL) +end + +-- The following list was obtained by parsing libcurl.a static library: +-- nm libcurl.a | grep -oP 'T \K(curl_.+)$' | sort +local curl_symbols = { + 'curl_easy_cleanup', + 'curl_easy_duphandle', + 'curl_easy_escape', + 'curl_easy_getinfo', + 'curl_easy_init', + 'curl_easy_pause', + 'curl_easy_perform', + 'curl_easy_recv', + 'curl_easy_reset', + 'curl_easy_send', + 'curl_easy_setopt', + 'curl_easy_strerror', + 'curl_easy_unescape', + 'curl_easy_upkeep', + 'curl_escape', + 'curl_formadd', + 'curl_formfree', + 'curl_formget', + 'curl_free', + 'curl_getdate', + 'curl_getenv', + 'curl_global_cleanup', + 'curl_global_init', + 'curl_global_init_mem', + 'curl_global_sslset', + 'curl_maprintf', + 'curl_mfprintf', + 'curl_mime_addpart', + 'curl_mime_data', + 'curl_mime_data_cb', + 'curl_mime_encoder', + 'curl_mime_filedata', + 'curl_mime_filename', + 'curl_mime_free', + 'curl_mime_headers', + 'curl_mime_init', + 'curl_mime_name', + 'curl_mime_subparts', + 'curl_mime_type', + 'curl_mprintf', + 'curl_msnprintf', + 'curl_msprintf', + 'curl_multi_add_handle', + 'curl_multi_assign', + 'curl_multi_cleanup', + 'curl_multi_fdset', + 'curl_multi_info_read', + 'curl_multi_init', + 'curl_multi_perform', + 'curl_multi_poll', + 'curl_multi_remove_handle', + 'curl_multi_setopt', + 'curl_multi_socket', + 'curl_multi_socket_action', + 'curl_multi_socket_all', + 'curl_multi_strerror', + 'curl_multi_timeout', + 'curl_multi_wait', + 'curl_mvaprintf', + 'curl_mvfprintf', + 'curl_mvprintf', + 'curl_mvsnprintf', + 'curl_mvsprintf', + 'curl_pushheader_byname', + 'curl_pushheader_bynum', + 'curl_share_cleanup', + 'curl_share_init', + 'curl_share_setopt', + 'curl_share_strerror', + 'curl_slist_append', + 'curl_slist_free_all', + 'curl_strequal', + 'curl_strnequal', + 'curl_unescape', + 'curl_url', + 'curl_url_cleanup', + 'curl_url_dup', + 'curl_url_get', + 'curl_url_set', + 'curl_version', + 'curl_version_info', +} + +test:test('curl_symbols', function(t) + t:plan(#curl_symbols) + for _, sym in ipairs(curl_symbols) do + t:ok( + ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, + ('Symbol %q found'):format(sym) + ) + end +end) + +os.exit(test:check() and 0 or 1) -- 2.25.1 From huston.mavr at gmail.com Thu Aug 6 16:32:32 2020 From: huston.mavr at gmail.com (Alexandr Barulev) Date: Thu, 6 Aug 2020 16:32:32 +0300 Subject: [Tarantool-patches] [PATCH] build: refactor static build process In-Reply-To: References: <20200622181649.10100-1-huston.mavr@gmail.com> <20200727223734.cnxrdzik2cyt3ey4@tkn_work_nb> Message-ID: I?ve squashed commit and changed it?s message; Also I?ve sended diff at previous answer https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build ??, 5 ???. 2020 ?. ? 20:08, Mavr Huston : > Hi, thanks for the review! > > libicu installs as ExternalProject_Add too, its missed in commit message; > > Problem with curses and ncurses was on macOS and linux, because libcurses > is an entire copy of libncurses, and tarantool links with system > libcurses instead of libncurses installed as tarantool dependency, but > module FindCurses.cmkae provides workaround for this problem - > CURSES_NEED_NCURSES flag.- to use ncurses instead of curses. (i will fix > this part at commit message) > > About disable-shred flag, used at libcurl building - we want to link only > with > static libraries, so we prevent creating unused .so. > > I've renamed static_build_no_deps_* jobs after review to > static_build_cmake_* > > Also about such path tarantool-prefix/* - it's a cmake > ExternalProject_Add() > default path (i've also added comment at .travis.mk) > > Useless comments "Init macOS test env" deleted. > > > if (BUILD_STATIC) > > - set(LIBZ_LIB_NAME libz.a) > > + find_library(LIBZ_LIBRARY NAMES libz.a) > > else() > > - set(LIBZ_LIB_NAME z) > > + find_library(LIBZ_LIBRARY NAMES z) > > endif() > > - find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) > Here we simplified code, by deleting useless variable. > > I've added commentaries to cmake/compiler.cmake about libunwind on macOS > and about ignoring flag -static-libstdc++ on macOS > > I've fixed static-build for using system compiler: gcc/g++ on linux > and clang/clang++ on macOS > > I've refactored IF (NOT APPLE) condition to IF (APPLE) at > static-build/CMakeLists.txt > > I've mentioned macOS dependencies at static-build/README.md xcode-tools and > others, also I've added example with CMAKE_TARANTOOL_ARGS. > > Added commentaries about _EP_INSTALL_DIR at static-build/CMakeLists.txt > > Also deleted unused use_unix_sockets_iproto = True > > Also deleted curl-features.test.lua, because after rebase on master it > fails, > due to missing curl_version_info symbol at tarantool binary. This symbol > lost > after #807c7fa584f21ee955b2a14623d70f7510a3650d (build: update curl > submodule > to 7.71.1 version ) > > > After pass the review I'll squash this changes to base commit and update > commit > message. > > Here is a diff of changes: > ===================================== > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 5ec5dd9b9..c9aef3dc7 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -534,14 +534,14 @@ static_build: > script: > - ${GITLAB_MAKE} test_static_build > > -static_build_no_deps_linux: > +static_build_cmake_linux: > <<: *docker_test_definition > script: > - - ${GITLAB_MAKE} test_static_build_no_deps_linux > + - ${GITLAB_MAKE} test_static_build_cmake_linux > > -static_build_no_deps_osx_15: > +static_build_cmake_osx_15: > stage: test > tags: > - osx_15 > script: > - - ${GITLAB_MAKE} test_static_build_no_deps_osx > + - ${GITLAB_MAKE} test_static_build_cmake_osx > diff --git a/.travis.mk b/.travis.mk > index 64862348f..482672429 100644 > --- a/.travis.mk > +++ b/.travis.mk > @@ -149,8 +149,8 @@ test_static_build: deps_debian_static > CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk > test_debian_no_deps > > # New static build > - > -test_static_build_no_deps_linux: > +# builddir used in this target - is a default build path from cmake > ExternalProject_Add() > +test_static_build_cmake_linux: > cd static-build && cmake . && make -j && ctest -V > cd test && /usr/bin/python test-run.py --force \ > --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build > $(TEST_RUN_EXTRA_PARAMS) > @@ -218,7 +218,6 @@ INIT_TEST_ENV_OSX=\ > rm -rf /tmp/tnt > > test_osx_no_deps: build_osx > - # Init macOS test env > ${INIT_TEST_ENV_OSX}; \ > cd test && ./test-run.py --vardir /tmp/tnt --force > $(TEST_RUN_EXTRA_PARAMS) > > @@ -233,9 +232,9 @@ base_deps_osx: > brew install --force ${STATIC_OSX_PKGS} || brew upgrade > ${STATIC_OSX_PKGS} > pip install --force-reinstall -r test-run/requirements.txt > > -test_static_build_no_deps_osx: base_deps_osx > +# builddir used in this target - is a default build path from cmake > ExternalProject_Add() > +test_static_build_cmake_osx: base_deps_osx > cd static-build && cmake . && make -j && ctest -V > - # Init macOS test env > ${INIT_TEST_ENV_OSX}; \ > cd test && ./test-run.py --vardir /tmp/tnt \ > --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ > diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake > index 8422181d6..afe480679 100644 > --- a/cmake/FindReadline.cmake > +++ b/cmake/FindReadline.cmake > @@ -14,6 +14,14 @@ if(BUILD_STATIC) > if (NOT CURSES_INFO_LIBRARY) > set(CURSES_INFO_LIBRARY "") > endif() > + > + # From Modules/FindCurses.cmake: > + # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the > + # ``find_package(Curses)`` call if NCurses functionality is required. > + # This flag is set for linking with required library (installed > + # via static-build/CMakeLists.txt). If this variable won't be set > + # then tarantool binary links with system library curses which is an > + # entire copy of ncurses > set(CURSES_NEED_NCURSES TRUE) > endif() > > diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake > index 14f1e1186..db2ae6227 100644 > --- a/cmake/compiler.cmake > +++ b/cmake/compiler.cmake > @@ -131,6 +131,8 @@ set(CMAKE_REQUIRED_INCLUDES "") > if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) > set(UNWIND_LIB_NAME libunwind.a) > else() > + # libunwind can't be compiled on macOS. > + # But there exists libunwind.dylib as a part of MacOSSDK > set(UNWIND_LIB_NAME unwind) > endif() > find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME}) > @@ -192,6 +194,9 @@ if (ENABLE_BACKTRACE) > find_package_message(UNWIND_LIBRARIES "Found unwind" > "${UNWIND_LIBRARIES}") > endif() > > +# On macOS there is no '-static-libstdc++' flag and it's use will > +# raise following error: > +# error: argument unused during compilation: '-static-libstdc++' > if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) > # Static linking for c++ routines > add_compile_flags("C;CXX" "-static-libstdc++") > diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt > index 86582af0a..53ceb609c 100644 > --- a/static-build/CMakeLists.txt > +++ b/static-build/CMakeLists.txt > @@ -9,11 +9,18 @@ set(NCURSES_VERSION 6.2) > set(READLINE_VERSION 8.0) > set(UNWIND_VERSION 1.3-rc1) > > -find_program(C_COMPILER gcc) > -find_program(CXX_COMPILER g++) > +if (APPLE) > + find_program(C_COMPILER clang) > + find_program(CXX_COMPILER clang++) > +else() > + find_program(C_COMPILER gcc) > + find_program(CXX_COMPILER g++) > +endif() > set(CMAKE_C_COMPILER ${C_COMPILER}) > set(CMAKE_CXX_COMPILER ${CXX_COMPILER}) > > +# Install all libraries required by tarantool at current build dir > + > # > # OpenSSL > # > @@ -80,7 +87,18 @@ ExternalProject_Add(readline > # > # ICONV > # > -if (NOT APPLE) > +if (APPLE) > + ExternalProject_Add(iconv > + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz > + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} > + /configure > + --prefix= > + --disable-shared > + --enable-static > + --with-gnu-ld > + STEP_TARGETS download > + ) > +else() > # In linux iconv is embedded into glibc > # So we find system header and copy it locally > find_path(ICONV_INCLUDE_DIR iconv.h) > @@ -101,20 +119,11 @@ if (NOT APPLE) > add_custom_target(iconv > DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h" > ) > + # This is a hack for further getting install directory of library > + # by ExternalProject_Get_Property > set_target_properties(iconv > PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX} > ) > -else() > - ExternalProject_Add(iconv > - URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz > - CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} > - /configure > - --prefix= > - --disable-shared > - --enable-static > - --with-gnu-ld > - STEP_TARGETS download > - ) > endif() > > # > @@ -162,6 +171,8 @@ if (APPLE) > endif() > > add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES}) > + # This is a hack for further getting install directory of library > + # by ExternalProject_Get_Property > set_target_properties(unwind > PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX} > ) > @@ -178,6 +189,8 @@ else() > ) > endif() > > +# Get install directories of builded libraries for building > +# tarantool with custon CMAKE_PREFIX_PATH > foreach(PROJ openssl icu zlib ncurses readline iconv unwind) > ExternalProject_Get_Property(${PROJ} install_dir) > set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir}) > @@ -197,16 +210,14 @@ ExternalProject_Add(tarantool > -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH} > -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE > -DOPENSSL_USE_STATIC_LIBS=TRUE > - -DCMAKE_BUILD_TYPE=Debug > -DBUILD_STATIC=TRUE > -DENABLE_DIST=TRUE > -DENABLE_BACKTRACE=TRUE > - -DPACKAGE:STRING=${PACKAGE_NAME} > -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} > -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} > ${CMAKE_TARANTOOL_ARGS} > - BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j > STEP_TARGETS build > + BUILD_COMMAND $(MAKE) > ) > > enable_testing() > diff --git a/static-build/README.md b/static-build/README.md > index 29fe085c3..0019e963f 100644 > --- a/static-build/README.md > +++ b/static-build/README.md > @@ -13,6 +13,24 @@ yum install -y \ > python-msgpack python-yaml python-argparse python-six python-gevent > ``` > > +MacOS: > + > +Before you start please install default Xcode Tools by Apple: > + > +``` > +sudo xcode-select --install > +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer > +``` > + > +Install brew using command from > +[Homebrew repository instructions](https://github.com/Homebrew/inst) > + > +After that run next script: > + > +```bash > + brew install autoconf automake libtool cmake > file://$${PWD}/tools/brew_taps/tntpython2.rbs > + pip install --force-reinstall -r test-run/requirements.txt > +``` > > ### Usage > > @@ -21,3 +39,20 @@ cmake . > make -j > ctest -V > ``` > + > +## Customize your build > + > +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` > variable > + > +### Usage > + > +There is three types of `CMAKE_BUILD_TYPE`: > +* Debug - default > +* Release > +* RelWithDebInfo > + > +And you want to build tarantool with RelWithDebInfo: > + > +```bash > +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" . > +``` > diff --git a/static-build/test/static-build/curl-features.test.lua > b/static-build/test/static-build/curl-features.test.lua > deleted file mode 100755 > index 57b1c4306..000000000 > --- a/static-build/test/static-build/curl-features.test.lua > +++ /dev/null > @@ -1,67 +0,0 @@ > -#!/usr/bin/env tarantool > - > -local tap = require('tap') > -local ffi = require('ffi') > -ffi.cdef([[ > - struct curl_version_info_data { > - int age; /* see description below */ > - const char *version; /* human readable string */ > - unsigned int version_num; /* numeric representation */ > - const char *host; /* human readable string */ > - int features; /* bitmask, see below */ > - char *ssl_version; /* human readable string */ > - long ssl_version_num; /* not used, always zero */ > - const char *libz_version; /* human readable string */ > - const char * const *protocols; /* protocols */ > - > - /* when 'age' is CURLVERSION_SECOND or higher, the members below > exist */ > - const char *ares; /* human readable string */ > - int ares_num; /* number */ > - > - /* when 'age' is CURLVERSION_THIRD or higher, the members below > exist */ > - const char *libidn; /* human readable string */ > - > - /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the > members > - below exist */ > - int iconv_ver_num; /* '_libiconv_version' if iconv support > enabled */ > - > - const char *libssh_version; /* human readable string */ > - > - /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the > members > - below exist */ > - unsigned int brotli_ver_num; /* Numeric Brotli version > - (MAJOR << 24) | (MINOR << 12) | > PATCH */ > - const char *brotli_version; /* human readable string. */ > - > - /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the > members > - below exist */ > - unsigned int nghttp2_ver_num; /* Numeric nghttp2 version > - (MAJOR << 16) | (MINOR << 8) | > PATCH */ > - const char *nghttp2_version; /* human readable string. */ > - > - const char *quic_version; /* human readable quic (+ HTTP/3) > library + > - version or NULL */ > - > - /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the > members > - below exist */ > - const char *cainfo; /* the built-in default > CURLOPT_CAINFO, might > - be NULL */ > - const char *capath; /* the built-in default > CURLOPT_CAPATH, might > - be NULL */ > - }; > - > - struct curl_version_info_data *curl_version_info(int age); > -]]) > - > -local info = ffi.C.curl_version_info(7) > -local test = tap.test('curl-features') > -test:plan(2) > - > -if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then > - test:diag('ssl_version: ' .. ffi.string(info.ssl_version)) > -end > -if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then > - test:diag('libz_version: ' .. ffi.string(info.libz_version)) > -end > - > -os.exit(test:check() and 0 or 1) > diff --git a/static-build/test/static-build/suite.ini > b/static-build/test/static-build/suite.ini > index 4da3d5d2f..92e349466 100644 > --- a/static-build/test/static-build/suite.ini > +++ b/static-build/test/static-build/suite.ini > @@ -3,4 +3,3 @@ core = app > description = Static build tests > script = box.lua > is_parallel = True > -use_unix_sockets_iproto = True > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imun at tarantool.org Thu Aug 6 17:41:53 2020 From: imun at tarantool.org (Igor Munkin) Date: Thu, 6 Aug 2020 17:41:53 +0300 Subject: [Tarantool-patches] [V2] luajit: pass properly compile options to LuaJIT for RelWithDebInfo In-Reply-To: References: Message-ID: <20200806144153.GW18920@tarantool.org> Timur, Thanks for the patch! Please, consider my comments below. On 24.07.20, Timur Safin wrote: > It was discovered the harder way (while debugging) that LuaJIT > sources do not generate line info debug information while we are Typo (still on the remote branch): s/whil/while/. Typo (still on the remote branch): s/we in/we are in/. > in any of optimized modes (e.g. RelWithDebInfo). > > Simplistically we need to append `-g -ggdb` to optimization > flags in luajit_cflags, but that would be fragile in the longer > run. We need just to pass CMAKE_C_FLAGS_ instead, Well, you claim that the simple solution is fragile, but I don't see the reason. Could you please clarify your concerns? > because it would contain -g -ggdb for RelWithDebInfo, iff Typo: s/iff/if/. > selected compiler support those options. Typo: s/support/supports/. > > Closes #4827 > --- > cmake/luajit.cmake | 17 ++++++----------- > 1 file changed, 6 insertions(+), 11 deletions(-) > > diff --git a/cmake/luajit.cmake b/cmake/luajit.cmake > index 555bc8371..76155c0c6 100644 > --- a/cmake/luajit.cmake > +++ b/cmake/luajit.cmake > @@ -139,7 +139,9 @@ macro(luajit_build) > # will most certainly wreak havok. > # > # This stuff is extremely fragile, proceed with caution. > - set (luajit_cflags ${CMAKE_C_FLAGS}) > + string (TOUPPER ${CMAKE_BUILD_TYPE} upper_build_type) > + set (c_flags_init "CMAKE_C_FLAGS_${upper_build_type}") > + set (luajit_cflags ${${c_flags_init}}) Why do you set the new flags, instead of appending them to ${CMAKE_C_FLAGS}? If there is a particular reason, please drop a few words regarding it. Besides, I checked LuaJIT build flags: * prior to your patch | /usr/bin/cc -O2 -Wall -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DNDEBUG | -DNVALGRIND -DNVALGRIND=1 -DLUAJIT_SMART_STRINGS=1 -D_FILE_OFFSET_BITS=64 | -D_LARGEFILE_SOURCE -U_FORTIFY_SOURCE -fno-stack-protector -fexceptions | -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common | -fopenmp -msse2 -Wno-parentheses-equality -Wno-tautological-compare | -Wno-misleading-indent ation -Wno-varargs -Wno-implicit-fallthrough -c | -o lj_record.o lj_record.c * and with your changes | /usr/bin/cc -Wall -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DNDEBUG | -DNVALGRIND -DNVALGRIND=1 -DLUAJIT_SMART_STRINGS=1 -D_FILE_OFFSET_BITS=64 | -D_LARGEFILE_SOURCE -U_FORTIFY_SOURCE -fno-stack-protector -O2 -g -DNDEBUG | -ggdb -O2 -Wno-parentheses-equality -Wno-tautological-compare | -Wno-misleading-indentation -Wno-varargs -Wno-implicit-fallthrough -c | -o lj_record.o lj_record.c I have the same concerns Sasha mentioned in his review regarding -fexceptions and -funwind-tables flags. There are several issues in LuaJIT queue but it seems all them are related to the "host" program build (i.e. Tarantool), not LuaJIT itself. However, -fno-omit-frame-pointer is set for internal Tarantool purposes (despite Mike's comments in LuaJIT Makefile). So, I guess you can't freely drop them. > set (luajit_ldflags ${CMAKE_EXE_LINKER_FLAGS}) > separate_arguments(luajit_cflags) > separate_arguments(luajit_ldflags) > @@ -168,19 +170,12 @@ macro(luajit_build) > "-pagezero_size 10000 -image_base 100000000") > endif() > > - # We are consciously ommiting debug info in RelWithDebInfo mode Here I also agree with Sasha here: there was a reason to omit debuginfo for RelWithDebInfo builds, but you mentioned a word neither in commit message nor in a comment nearby. > + set (upper_build_type) > + set (c_flags_init) > + > if (${CMAKE_BUILD_TYPE} STREQUAL "Debug") > - set (luajit_ccopt -O0) > - if (CC_HAS_GGDB) > - set (luajit_ccdebug -g -ggdb) > - else () > - set (luajit_ccdebug -g) > - endif () > add_definitions(-DLUA_USE_APICHECK=1) > add_definitions(-DLUA_USE_ASSERT=1) > - else () > - set (luajit_ccopt -O2) > - set (luajit_ccdbebug "") > endif() > if (${CMAKE_SYSTEM_NAME} STREQUAL Darwin) > # Pass sysroot - prepended in front of system header/lib dirs, > -- > 2.20.1 > -- Best regards, IM From tsafin at tarantool.org Thu Aug 6 18:06:35 2020 From: tsafin at tarantool.org (Timur Safin) Date: Thu, 6 Aug 2020 18:06:35 +0300 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported In-Reply-To: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> References: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> Message-ID: <07fa01d66c03$2da443d0$88eccb70$@tarantool.org> Much respect for the outstanding symbol loader test! LGTM Timur : From: Tarantool-patches On : Behalf Of Yaroslav Dynnikov : Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported : : In the recent update of libcurl (2.5.0-278-g807c7fa58) its layout has : changed: private function `Curl_version_init()` which used to fill-in : info structure was eliminated. As a result, no symbols for : `libcurl_la-version.o` remained used, so it wasn't included in tarantool : binary. And `curl_version` and `curl_version_info` symbols went missing. : : According to libcurl naming conventions all exported symbols are named : as `curl_*`. This patch lists them all explicitly in `exprots.h` and : adds the test. : : Close #5223 : : Issue: https://github.com/tarantool/tarantool/issues/5223 : Branch: https://github.com/tarantool/tarantool/tree/rosik/gh-5223-missing- : curl-symbols : : --- : src/exports.h | 81 ++++++++++ : test/box-tap/gh-5223-curl-exports.test.lua | 177 +++++++++++++++++++++ : 2 files changed, 258 insertions(+) : create mode 100755 test/box-tap/gh-5223-curl-exports.test.lua : : diff --git a/src/exports.h b/src/exports.h : index 7cf283e5b..7f0601f4f 100644 : --- a/src/exports.h : +++ b/src/exports.h : @@ -113,6 +113,87 @@ EXPORT(csv_feed) : EXPORT(csv_iterator_create) : EXPORT(csv_next) : EXPORT(csv_setopt) : +EXPORT(curl_easy_cleanup) : +EXPORT(curl_easy_duphandle) : +EXPORT(curl_easy_escape) : +EXPORT(curl_easy_getinfo) : +EXPORT(curl_easy_init) : +EXPORT(curl_easy_pause) : +EXPORT(curl_easy_perform) : +EXPORT(curl_easy_recv) : +EXPORT(curl_easy_reset) : +EXPORT(curl_easy_send) : +EXPORT(curl_easy_setopt) : +EXPORT(curl_easy_strerror) : +EXPORT(curl_easy_unescape) : +EXPORT(curl_easy_upkeep) : +EXPORT(curl_escape) : +EXPORT(curl_formadd) : +EXPORT(curl_formfree) : +EXPORT(curl_formget) : +EXPORT(curl_free) : +EXPORT(curl_getdate) : +EXPORT(curl_getenv) : +EXPORT(curl_global_cleanup) : +EXPORT(curl_global_init) : +EXPORT(curl_global_init_mem) : +EXPORT(curl_global_sslset) : +EXPORT(curl_maprintf) : +EXPORT(curl_mfprintf) : +EXPORT(curl_mime_addpart) : +EXPORT(curl_mime_data) : +EXPORT(curl_mime_data_cb) : +EXPORT(curl_mime_encoder) : +EXPORT(curl_mime_filedata) : +EXPORT(curl_mime_filename) : +EXPORT(curl_mime_free) : +EXPORT(curl_mime_headers) : +EXPORT(curl_mime_init) : +EXPORT(curl_mime_name) : +EXPORT(curl_mime_subparts) : +EXPORT(curl_mime_type) : +EXPORT(curl_mprintf) : +EXPORT(curl_msnprintf) : +EXPORT(curl_msprintf) : +EXPORT(curl_multi_add_handle) : +EXPORT(curl_multi_assign) : +EXPORT(curl_multi_cleanup) : +EXPORT(curl_multi_fdset) : +EXPORT(curl_multi_info_read) : +EXPORT(curl_multi_init) : +EXPORT(curl_multi_perform) : +EXPORT(curl_multi_poll) : +EXPORT(curl_multi_remove_handle) : +EXPORT(curl_multi_setopt) : +EXPORT(curl_multi_socket) : +EXPORT(curl_multi_socket_action) : +EXPORT(curl_multi_socket_all) : +EXPORT(curl_multi_strerror) : +EXPORT(curl_multi_timeout) : +EXPORT(curl_multi_wait) : +EXPORT(curl_mvaprintf) : +EXPORT(curl_mvfprintf) : +EXPORT(curl_mvprintf) : +EXPORT(curl_mvsnprintf) : +EXPORT(curl_mvsprintf) : +EXPORT(curl_pushheader_byname) : +EXPORT(curl_pushheader_bynum) : +EXPORT(curl_share_cleanup) : +EXPORT(curl_share_init) : +EXPORT(curl_share_setopt) : +EXPORT(curl_share_strerror) : +EXPORT(curl_slist_append) : +EXPORT(curl_slist_free_all) : +EXPORT(curl_strequal) : +EXPORT(curl_strnequal) : +EXPORT(curl_unescape) : +EXPORT(curl_url) : +EXPORT(curl_url_cleanup) : +EXPORT(curl_url_dup) : +EXPORT(curl_url_get) : +EXPORT(curl_url_set) : +EXPORT(curl_version) : +EXPORT(curl_version_info) : EXPORT(decimal_unpack) : EXPORT(error_ref) : EXPORT(error_set_prev) : diff --git a/test/box-tap/gh-5223-curl-exports.test.lua b/test/box-tap/gh- : 5223-curl-exports.test.lua : new file mode 100755 : index 000000000..300d60b07 : --- /dev/null : +++ b/test/box-tap/gh-5223-curl-exports.test.lua : @@ -0,0 +1,177 @@ : +#!/usr/bin/env tarantool : + : +local tap = require('tap') : +local ffi = require('ffi') : +ffi.cdef([[ : + void *dlsym(void *handle, const char *symbol); : + struct curl_version_info_data { : + int age; /* see description below */ : + const char *version; /* human readable string */ : + unsigned int version_num; /* numeric representation */ : + const char *host; /* human readable string */ : + int features; /* bitmask, see below */ : + char *ssl_version; /* human readable string */ : + long ssl_version_num; /* not used, always zero */ : + const char *libz_version; /* human readable string */ : + const char * const *protocols; /* protocols */ : + : + /* when 'age' is CURLVERSION_SECOND or higher, the members below : exist */ : + const char *ares; /* human readable string */ : + int ares_num; /* number */ : + : + /* when 'age' is CURLVERSION_THIRD or higher, the members below : exist */ : + const char *libidn; /* human readable string */ : + : + /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the : members : + below exist */ : + int iconv_ver_num; /* '_libiconv_version' if iconv support : enabled */ : + : + const char *libssh_version; /* human readable string */ : + : + /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the : members : + below exist */ : + unsigned int brotli_ver_num; /* Numeric Brotli version : + (MAJOR << 24) | (MINOR << 12) | : PATCH */ : + const char *brotli_version; /* human readable string. */ : + : + /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the : members : + below exist */ : + unsigned int nghttp2_ver_num; /* Numeric nghttp2 version : + (MAJOR << 16) | (MINOR << 8) | : PATCH */ : + const char *nghttp2_version; /* human readable string. */ : + : + const char *quic_version; /* human readable quic (+ HTTP/3) : library + : + version or NULL */ : + : + /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the : members : + below exist */ : + const char *cainfo; /* the built-in default : CURLOPT_CAINFO, might : + be NULL */ : + const char *capath; /* the built-in default : CURLOPT_CAPATH, might : + be NULL */ : + }; : + : + struct curl_version_info_data *curl_version_info(int age); : +]]) : + : +local info = ffi.C.curl_version_info(7) : +local test = tap.test('curl-features') : +test:plan(3) : + : +if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then : + test:diag('ssl_version: ' .. ffi.string(info.ssl_version)) : +end : +if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then : + test:diag('libz_version: ' .. ffi.string(info.libz_version)) : +end : + : +local RTLD_DEFAULT : +-- See `man 3 dlsym`: : +-- RTLD_DEFAULT : +-- Find the first occurrence of the desired symbol using the default : +-- shared object search order. The search will include global symbols : +-- in the executable and its dependencies, as well as symbols in shared : +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. : +if jit.os == "OSX" then : + RTLD_DEFAULT = ffi.cast("void *", -2LL) : +else : + RTLD_DEFAULT = ffi.cast("void *", 0LL) : +end : + : +-- The following list was obtained by parsing libcurl.a static library: : +-- nm libcurl.a | grep -oP 'T \K(curl_.+)$' | sort : +local curl_symbols = { : + 'curl_easy_cleanup', : + 'curl_easy_duphandle', : + 'curl_easy_escape', : + 'curl_easy_getinfo', : + 'curl_easy_init', : + 'curl_easy_pause', : + 'curl_easy_perform', : + 'curl_easy_recv', : + 'curl_easy_reset', : + 'curl_easy_send', : + 'curl_easy_setopt', : + 'curl_easy_strerror', : + 'curl_easy_unescape', : + 'curl_easy_upkeep', : + 'curl_escape', : + 'curl_formadd', : + 'curl_formfree', : + 'curl_formget', : + 'curl_free', : + 'curl_getdate', : + 'curl_getenv', : + 'curl_global_cleanup', : + 'curl_global_init', : + 'curl_global_init_mem', : + 'curl_global_sslset', : + 'curl_maprintf', : + 'curl_mfprintf', : + 'curl_mime_addpart', : + 'curl_mime_data', : + 'curl_mime_data_cb', : + 'curl_mime_encoder', : + 'curl_mime_filedata', : + 'curl_mime_filename', : + 'curl_mime_free', : + 'curl_mime_headers', : + 'curl_mime_init', : + 'curl_mime_name', : + 'curl_mime_subparts', : + 'curl_mime_type', : + 'curl_mprintf', : + 'curl_msnprintf', : + 'curl_msprintf', : + 'curl_multi_add_handle', : + 'curl_multi_assign', : + 'curl_multi_cleanup', : + 'curl_multi_fdset', : + 'curl_multi_info_read', : + 'curl_multi_init', : + 'curl_multi_perform', : + 'curl_multi_poll', : + 'curl_multi_remove_handle', : + 'curl_multi_setopt', : + 'curl_multi_socket', : + 'curl_multi_socket_action', : + 'curl_multi_socket_all', : + 'curl_multi_strerror', : + 'curl_multi_timeout', : + 'curl_multi_wait', : + 'curl_mvaprintf', : + 'curl_mvfprintf', : + 'curl_mvprintf', : + 'curl_mvsnprintf', : + 'curl_mvsprintf', : + 'curl_pushheader_byname', : + 'curl_pushheader_bynum', : + 'curl_share_cleanup', : + 'curl_share_init', : + 'curl_share_setopt', : + 'curl_share_strerror', : + 'curl_slist_append', : + 'curl_slist_free_all', : + 'curl_strequal', : + 'curl_strnequal', : + 'curl_unescape', : + 'curl_url', : + 'curl_url_cleanup', : + 'curl_url_dup', : + 'curl_url_get', : + 'curl_url_set', : + 'curl_version', : + 'curl_version_info', : +} : + : +test:test('curl_symbols', function(t) : + t:plan(#curl_symbols) : + for _, sym in ipairs(curl_symbols) do : + t:ok( : + ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, : + ('Symbol %q found'):format(sym) : + ) : + end : +end) : + : +os.exit(test:check() and 0 or 1) : -- : 2.25.1 From olegrok at tarantool.org Thu Aug 6 19:00:43 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Thu, 6 Aug 2020 19:00:43 +0300 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: References: Message-ID: <6a8870f3-ac79-7021-941d-11b90e687282@tarantool.org> Hi! Thanks for your patch. It's not a review but I have a question. On 05/08/2020 02:45, Vladislav Shpilevoy wrote: > When a tuple had format with multikey indexes in it, any attempt > to get a multikey indexed field by a JSON path from Lua led to a > crash. > > That was because of incorrect interpretation of offset slot value > in tuple's field map. > > Tuple field map is an array stored before the tuple's MessagePack > data. Each element is a 4 byte offset to an indexed value to be > able to get it for O(1) time without MessagePack decoding of all > the previous fields. > > At least it was so before multikeys. Now tuple field map is not > just an array. It is rather a 2-level array, somehow similar to > ext4 FS. Some elements of the root array are positive numbers > pointing at data. Some elements point at a second 'indirect' > array, so called 'extra', size of which is individual for each > tuple. These second arrays are used by multikey indexes to store > offsets to each multikey indexed value in a tuple. Do json path updates use offsets? Is such issue relevant for them? I tried to update poisoned tuple but seems it works fine. But maybe I've missed something. From v.shpilevoy at tarantool.org Thu Aug 6 22:57:41 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 21:57:41 +0200 Subject: [Tarantool-patches] [PATCH v2 vshard 0/2] storage: allow replica to boot before master In-Reply-To: References: Message-ID: <0ac0515a-5fab-66e7-c712-113f4efad26e@tarantool.org> Pushed to master. From v.shpilevoy at tarantool.org Thu Aug 6 23:04:10 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 22:04:10 +0200 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: <6a8870f3-ac79-7021-941d-11b90e687282@tarantool.org> References: <6a8870f3-ac79-7021-941d-11b90e687282@tarantool.org> Message-ID: <8892a38d-0608-314f-b32d-531db3bd6de3@tarantool.org> On 06.08.2020 18:00, Oleg Babin wrote: > Hi! Thanks for your patch. It's not a review but I have a question. > > On 05/08/2020 02:45, Vladislav Shpilevoy wrote: >> When a tuple had format with multikey indexes in it, any attempt >> to get a multikey indexed field by a JSON path from Lua led to a >> crash. >> >> That was because of incorrect interpretation of offset slot value >> in tuple's field map. >> >> Tuple field map is an array stored before the tuple's MessagePack >> data. Each element is a 4 byte offset to an indexed value to be >> able to get it for O(1) time without MessagePack decoding of all >> the previous fields. >> >> At least it was so before multikeys. Now tuple field map is not >> just an array. It is rather a 2-level array, somehow similar to >> ext4 FS. Some elements of the root array are positive numbers >> pointing at data. Some elements point at a second 'indirect' >> array, so called 'extra', size of which is individual for each >> tuple. These second arrays are used by multikey indexes to store >> offsets to each multikey indexed value in a tuple. > > > Do json path updates use offsets? Is such issue relevant for them? > > I tried to update poisoned tuple but seems it works fine. But maybe I've missed something. No, JSON updates always decode whole tuple, at least all fields <= max affected field. So offsets are not used. I was thinking about adding them, but so far there was no a request for it, nor benches how would it help exactly. From v.shpilevoy at tarantool.org Fri Aug 7 00:48:31 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 6 Aug 2020 23:48:31 +0200 Subject: [Tarantool-patches] [PATCH vshard 1/1] replicaset: check URI match when rebind connection Message-ID: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> Reconfiguration in router and storage always creates new replica objects, and outdates old objects. Since most of the time reconfiguration does not change cluster topology and URIs, old connections are relocated to the new replicaset objects so as not to reconnect on each router.cfg() and storage.cfg(). Connections were moved from old objects to new ones by UUID match not taking into account possible URI change. If URI was changed, the new replica object took the old connection with old URI, and kept trying to connect to the old address. The patch makes connection relocation work only if old and new URI match. Closes #245 --- Branch: http://github.com/tarantool/vshard/tree/gerold103/gh-245-router-reconnect Issue: https://github.com/tarantool/vshard/issues/245 test/router/reconnect_to_master.result | 55 ++++++++++++++++++++++++ test/router/reconnect_to_master.test.lua | 26 +++++++++++ vshard/replicaset.lua | 2 +- 3 files changed, 82 insertions(+), 1 deletion(-) diff --git a/test/router/reconnect_to_master.result b/test/router/reconnect_to_master.result index d640f0b..aa33f9c 100644 --- a/test/router/reconnect_to_master.result +++ b/test/router/reconnect_to_master.result @@ -169,6 +169,61 @@ is_disconnected() --- - false ... +-- +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was changed in +-- the config for any instance, it used old connection, because reconfiguration +-- compared connections by UUID instead of URI. +-- +util = require('util') +--- +... +-- Firstly, clean router from storage_1_a connection. +rs1_uuid = util.replicasets[1] +--- +... +rs1_cfg = cfg.sharding[rs1_uuid] +--- +... +cfg.sharding[rs1_uuid] = nil +--- +... +vshard.router.cfg(cfg) +--- +... +-- Now break the URI in the config. +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri +--- +... +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = 'https://bad_uri.com:123' +--- +... +-- Apply the bad config. +cfg.sharding[rs1_uuid] = rs1_cfg +--- +... +vshard.router.cfg(cfg) +--- +... +-- Should fail - master is not available because of the bad URI. +res, err = vshard.router.callrw(1, 'echo', {1}) +--- +... +res == nil and err ~= nil +--- +- true +... +-- Repair the config. +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri +--- +... +vshard.router.cfg(cfg) +--- +... +-- Should drop the old connection object and connect fine. +vshard.router.callrw(1, 'echo', {1}) +--- +- 1 +... _ = test_run:switch("default") --- ... diff --git a/test/router/reconnect_to_master.test.lua b/test/router/reconnect_to_master.test.lua index c315c9f..87270af 100644 --- a/test/router/reconnect_to_master.test.lua +++ b/test/router/reconnect_to_master.test.lua @@ -70,6 +70,32 @@ while is_disconnected() and i < max_iters do i = i + 1 fiber.sleep(0.1) end -- Master connection is active again. is_disconnected() +-- +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was changed in +-- the config for any instance, it used old connection, because reconfiguration +-- compared connections by UUID instead of URI. +-- +util = require('util') +-- Firstly, clean router from storage_1_a connection. +rs1_uuid = util.replicasets[1] +rs1_cfg = cfg.sharding[rs1_uuid] +cfg.sharding[rs1_uuid] = nil +vshard.router.cfg(cfg) +-- Now break the URI in the config. +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = 'https://bad_uri.com:123' +-- Apply the bad config. +cfg.sharding[rs1_uuid] = rs1_cfg +vshard.router.cfg(cfg) +-- Should fail - master is not available because of the bad URI. +res, err = vshard.router.callrw(1, 'echo', {1}) +res == nil and err ~= nil +-- Repair the config. +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri +vshard.router.cfg(cfg) +-- Should drop the old connection object and connect fine. +vshard.router.callrw(1, 'echo', {1}) + _ = test_run:switch("default") _ = test_run:cmd('stop server router_1') _ = test_run:cmd('cleanup server router_1') diff --git a/vshard/replicaset.lua b/vshard/replicaset.lua index 0191936..b13d05e 100644 --- a/vshard/replicaset.lua +++ b/vshard/replicaset.lua @@ -456,7 +456,7 @@ local function rebind_replicasets(replicasets, old_replicasets) for replica_uuid, replica in pairs(replicaset.replicas) do local old_replica = old_replicaset and old_replicaset.replicas[replica_uuid] - if old_replica then + if old_replica and old_replica.uri == replica.uri then local conn = old_replica.conn replica.conn = conn replica.down_ts = old_replica.down_ts -- 2.21.1 (Apple Git-122.3) From yaroslav.dynnikov at tarantool.org Fri Aug 7 01:02:18 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Fri, 7 Aug 2020 01:02:18 +0300 Subject: [Tarantool-patches] [PATCH vshard 1/1] replicaset: check URI match when rebind connection In-Reply-To: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> References: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> Message-ID: Hi! Thanks for the fix. LGTM. Best regards Yaroslav Dynnikov On Fri, 7 Aug 2020 at 00:48, Vladislav Shpilevoy wrote: > Reconfiguration in router and storage always creates new > replica objects, and outdates old objects. Since most of the > time reconfiguration does not change cluster topology and URIs, > old connections are relocated to the new replicaset objects so as > not to reconnect on each router.cfg() and storage.cfg(). > > Connections were moved from old objects to new ones by UUID match > not taking into account possible URI change. > > If URI was changed, the new replica object took the old connection > with old URI, and kept trying to connect to the old address. > > The patch makes connection relocation work only if old and new URI > match. > > Closes #245 > --- > Branch: > http://github.com/tarantool/vshard/tree/gerold103/gh-245-router-reconnect > Issue: https://github.com/tarantool/vshard/issues/245 > > test/router/reconnect_to_master.result | 55 ++++++++++++++++++++++++ > test/router/reconnect_to_master.test.lua | 26 +++++++++++ > vshard/replicaset.lua | 2 +- > 3 files changed, 82 insertions(+), 1 deletion(-) > > diff --git a/test/router/reconnect_to_master.result > b/test/router/reconnect_to_master.result > index d640f0b..aa33f9c 100644 > --- a/test/router/reconnect_to_master.result > +++ b/test/router/reconnect_to_master.result > @@ -169,6 +169,61 @@ is_disconnected() > --- > - false > ... > +-- > +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was > changed in > +-- the config for any instance, it used old connection, because > reconfiguration > +-- compared connections by UUID instead of URI. > +-- > +util = require('util') > +--- > +... > +-- Firstly, clean router from storage_1_a connection. > +rs1_uuid = util.replicasets[1] > +--- > +... > +rs1_cfg = cfg.sharding[rs1_uuid] > +--- > +... > +cfg.sharding[rs1_uuid] = nil > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Now break the URI in the config. > +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri > +--- > +... > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = ' > https://bad_uri.com:123' > +--- > +... > +-- Apply the bad config. > +cfg.sharding[rs1_uuid] = rs1_cfg > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Should fail - master is not available because of the bad URI. > +res, err = vshard.router.callrw(1, 'echo', {1}) > +--- > +... > +res == nil and err ~= nil > +--- > +- true > +... > +-- Repair the config. > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Should drop the old connection object and connect fine. > +vshard.router.callrw(1, 'echo', {1}) > +--- > +- 1 > +... > _ = test_run:switch("default") > --- > ... > diff --git a/test/router/reconnect_to_master.test.lua > b/test/router/reconnect_to_master.test.lua > index c315c9f..87270af 100644 > --- a/test/router/reconnect_to_master.test.lua > +++ b/test/router/reconnect_to_master.test.lua > @@ -70,6 +70,32 @@ while is_disconnected() and i < max_iters do i = i + 1 > fiber.sleep(0.1) end > -- Master connection is active again. > is_disconnected() > > +-- > +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was > changed in > +-- the config for any instance, it used old connection, because > reconfiguration > +-- compared connections by UUID instead of URI. > +-- > +util = require('util') > +-- Firstly, clean router from storage_1_a connection. > +rs1_uuid = util.replicasets[1] > +rs1_cfg = cfg.sharding[rs1_uuid] > +cfg.sharding[rs1_uuid] = nil > +vshard.router.cfg(cfg) > +-- Now break the URI in the config. > +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = ' > https://bad_uri.com:123' > +-- Apply the bad config. > +cfg.sharding[rs1_uuid] = rs1_cfg > +vshard.router.cfg(cfg) > +-- Should fail - master is not available because of the bad URI. > +res, err = vshard.router.callrw(1, 'echo', {1}) > +res == nil and err ~= nil > +-- Repair the config. > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri > +vshard.router.cfg(cfg) > +-- Should drop the old connection object and connect fine. > +vshard.router.callrw(1, 'echo', {1}) > + > _ = test_run:switch("default") > _ = test_run:cmd('stop server router_1') > _ = test_run:cmd('cleanup server router_1') > diff --git a/vshard/replicaset.lua b/vshard/replicaset.lua > index 0191936..b13d05e 100644 > --- a/vshard/replicaset.lua > +++ b/vshard/replicaset.lua > @@ -456,7 +456,7 @@ local function rebind_replicasets(replicasets, > old_replicasets) > for replica_uuid, replica in pairs(replicaset.replicas) do > local old_replica = old_replicaset and > old_replicaset.replicas[replica_uuid] > - if old_replica then > + if old_replica and old_replica.uri == replica.uri then > local conn = old_replica.conn > replica.conn = conn > replica.down_ts = old_replica.down_ts > -- > 2.21.1 (Apple Git-122.3) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Fri Aug 7 01:11:46 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Fri, 7 Aug 2020 00:11:46 +0200 Subject: [Tarantool-patches] [PATCH 1/1] router: update known bucket count when rs removed Message-ID: When replicaset was removed, router didn't update router.known_bucket_count value. It is used by discovery to decide when to turn on aggressive discovery. And by router.info() to provide alerts, change status. As a result, in case of replicaset removal, known_bucket_count was left bigger than it actually is, and it could be fixed only by restart. Discovery could become non-agressive too early, and router.info() could show weird alerts about bucke_count being configured differently on storages and router. Besides, router.info().bucket was showing not realistic numbers. --- Branch: http://github.com/tarantool/tarantool/tree/gerold103/router-known-buckets test/router/router2.result | 61 ++++++++++++++++++++++++++++++++++++ test/router/router2.test.lua | 24 ++++++++++++++ vshard/router/init.lua | 3 ++ 3 files changed, 88 insertions(+) diff --git a/test/router/router2.result b/test/router/router2.result index 40d6bff..492421b 100644 --- a/test/router/router2.result +++ b/test/router/router2.result @@ -250,6 +250,67 @@ vshard.router.static.discovery_fiber | - null | ... +-- +-- Known bucket count should be updated properly when replicaset +-- is removed from the config. +-- +vshard.router.info().bucket + | --- + | - unreachable: 0 + | available_ro: 0 + | unknown: 0 + | available_rw: 3000 + | ... +rs1_uuid = util.replicasets[1] + | --- + | ... +rs1 = cfg.sharding[rs1_uuid] + | --- + | ... +cfg.sharding[rs1_uuid] = nil + | --- + | ... +vshard.router.cfg(cfg) + | --- + | ... +vshard.router.info().bucket + | --- + | - unreachable: 0 + | available_ro: 0 + | unknown: 1500 + | available_rw: 1500 + | ... +cfg.sharding[rs1_uuid] = rs1 + | --- + | ... +vshard.router.cfg(cfg) + | --- + | ... +vshard.router.discovery_set('on') + | --- + | ... +function wait_all_rw() \ + local total = vshard.router.bucket_count() \ + local res = vshard.router.info().bucket.available_rw == total \ + if not res then \ + vshard.router.discovery_wakeup() \ + end \ + return res \ +end + | --- + | ... +test_run:wait_cond(wait_all_rw) + | --- + | - true + | ... +vshard.router.info().bucket + | --- + | - unreachable: 0 + | available_ro: 0 + | unknown: 0 + | available_rw: 3000 + | ... + _ = test_run:switch("default") | --- | ... diff --git a/test/router/router2.test.lua b/test/router/router2.test.lua index afdf4e1..72c99e6 100644 --- a/test/router/router2.test.lua +++ b/test/router/router2.test.lua @@ -95,6 +95,30 @@ vshard.router.static.discovery_fiber vshard.router.discovery_set('once') vshard.router.static.discovery_fiber +-- +-- Known bucket count should be updated properly when replicaset +-- is removed from the config. +-- +vshard.router.info().bucket +rs1_uuid = util.replicasets[1] +rs1 = cfg.sharding[rs1_uuid] +cfg.sharding[rs1_uuid] = nil +vshard.router.cfg(cfg) +vshard.router.info().bucket +cfg.sharding[rs1_uuid] = rs1 +vshard.router.cfg(cfg) +vshard.router.discovery_set('on') +function wait_all_rw() \ + local total = vshard.router.bucket_count() \ + local res = vshard.router.info().bucket.available_rw == total \ + if not res then \ + vshard.router.discovery_wakeup() \ + end \ + return res \ +end +test_run:wait_cond(wait_all_rw) +vshard.router.info().bucket + _ = test_run:switch("default") _ = test_run:cmd("stop server router_1") _ = test_run:cmd("cleanup server router_1") diff --git a/vshard/router/init.lua b/vshard/router/init.lua index d8c9246..0d94210 100644 --- a/vshard/router/init.lua +++ b/vshard/router/init.lua @@ -908,14 +908,17 @@ local function router_cfg(router, cfg, is_reload) router.failover_ping_timeout = vshard_cfg.failover_ping_timeout router.sync_timeout = vshard_cfg.sync_timeout local old_route_map = router.route_map + local known_bucket_count = 0 router.route_map = table_new(router.total_bucket_count, 0) for bucket, rs in pairs(old_route_map) do local new_rs = router.replicasets[rs.uuid] if new_rs then router.route_map[bucket] = new_rs new_rs.bucket_count = new_rs.bucket_count + 1 + known_bucket_count = known_bucket_count + 1 end end + router.known_bucket_count = known_bucket_count if router.failover_fiber == nil then router.failover_fiber = util.reloadable_fiber_create( 'vshard.failover.' .. router.name, M, 'failover_f', router) -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Fri Aug 7 01:14:13 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Fri, 7 Aug 2020 00:14:13 +0200 Subject: [Tarantool-patches] [PATCH 1/1] router: update known bucket count when rs removed In-Reply-To: References: Message-ID: <4f828799-394e-c6d7-c6a5-700f87a55896@tarantool.org> Forgot to add 'vshard' after 'PATCH' in the title. Also the real branch is http://github.com/tarantool/vshard/tree/gerold103/router-known-buckets (I generate patch emails with a script, and forgot to pass the option with repository name.) From olegrok at tarantool.org Fri Aug 7 11:07:16 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Fri, 7 Aug 2020 11:07:16 +0300 Subject: [Tarantool-patches] [PATCH 1/1] router: update known bucket count when rs removed In-Reply-To: References: Message-ID: Thanks for your patch! LGTM. On 07/08/2020 01:11, Vladislav Shpilevoy wrote: > When replicaset was removed, router didn't update > router.known_bucket_count value. It is used by discovery to decide > when to turn on aggressive discovery. And by router.info() to > provide alerts, change status. > > As a result, in case of replicaset removal, known_bucket_count > was left bigger than it actually is, and it could be fixed only > by restart. > > Discovery could become non-agressive too early, and router.info() > could show weird alerts about bucke_count being configured > differently on storages and router. Besides, router.info().bucket > was showing not realistic numbers. > --- > Branch:http://github.com/tarantool/tarantool/tree/gerold103/router-known-buckets > > test/router/router2.result | 61 ++++++++++++++++++++++++++++++++++++ > test/router/router2.test.lua | 24 ++++++++++++++ > vshard/router/init.lua | 3 ++ > 3 files changed, 88 insertions(+) > > diff --git a/test/router/router2.result b/test/router/router2.result > index 40d6bff..492421b 100644 > --- a/test/router/router2.result > +++ b/test/router/router2.result > @@ -250,6 +250,67 @@ vshard.router.static.discovery_fiber > | - null > | ... > > +-- > +-- Known bucket count should be updated properly when replicaset > +-- is removed from the config. > +-- > +vshard.router.info().bucket > + | --- > + | - unreachable: 0 > + | available_ro: 0 > + | unknown: 0 > + | available_rw: 3000 > + | ... > +rs1_uuid = util.replicasets[1] > + | --- > + | ... > +rs1 = cfg.sharding[rs1_uuid] > + | --- > + | ... > +cfg.sharding[rs1_uuid] = nil > + | --- > + | ... > +vshard.router.cfg(cfg) > + | --- > + | ... > +vshard.router.info().bucket > + | --- > + | - unreachable: 0 > + | available_ro: 0 > + | unknown: 1500 > + | available_rw: 1500 > + | ... > +cfg.sharding[rs1_uuid] = rs1 > + | --- > + | ... > +vshard.router.cfg(cfg) > + | --- > + | ... > +vshard.router.discovery_set('on') > + | --- > + | ... > +function wait_all_rw() \ > + local total = vshard.router.bucket_count() \ > + local res = vshard.router.info().bucket.available_rw == total \ > + if not res then \ > + vshard.router.discovery_wakeup() \ > + end \ > + return res \ > +end > + | --- > + | ... > +test_run:wait_cond(wait_all_rw) > + | --- > + | - true > + | ... > +vshard.router.info().bucket > + | --- > + | - unreachable: 0 > + | available_ro: 0 > + | unknown: 0 > + | available_rw: 3000 > + | ... > + > _ = test_run:switch("default") > | --- > | ... > diff --git a/test/router/router2.test.lua b/test/router/router2.test.lua > index afdf4e1..72c99e6 100644 > --- a/test/router/router2.test.lua > +++ b/test/router/router2.test.lua > @@ -95,6 +95,30 @@ vshard.router.static.discovery_fiber > vshard.router.discovery_set('once') > vshard.router.static.discovery_fiber > > +-- > +-- Known bucket count should be updated properly when replicaset > +-- is removed from the config. > +-- > +vshard.router.info().bucket > +rs1_uuid = util.replicasets[1] > +rs1 = cfg.sharding[rs1_uuid] > +cfg.sharding[rs1_uuid] = nil > +vshard.router.cfg(cfg) > +vshard.router.info().bucket > +cfg.sharding[rs1_uuid] = rs1 > +vshard.router.cfg(cfg) > +vshard.router.discovery_set('on') > +function wait_all_rw() \ > + local total = vshard.router.bucket_count() \ > + local res = vshard.router.info().bucket.available_rw == total \ > + if not res then \ > + vshard.router.discovery_wakeup() \ > + end \ > + return res \ > +end > +test_run:wait_cond(wait_all_rw) > +vshard.router.info().bucket > + > _ = test_run:switch("default") > _ = test_run:cmd("stop server router_1") > _ = test_run:cmd("cleanup server router_1") > diff --git a/vshard/router/init.lua b/vshard/router/init.lua > index d8c9246..0d94210 100644 > --- a/vshard/router/init.lua > +++ b/vshard/router/init.lua > @@ -908,14 +908,17 @@ local function router_cfg(router, cfg, is_reload) > router.failover_ping_timeout = vshard_cfg.failover_ping_timeout > router.sync_timeout = vshard_cfg.sync_timeout > local old_route_map = router.route_map > + local known_bucket_count = 0 > router.route_map = table_new(router.total_bucket_count, 0) > for bucket, rs in pairs(old_route_map) do > local new_rs = router.replicasets[rs.uuid] > if new_rs then > router.route_map[bucket] = new_rs > new_rs.bucket_count = new_rs.bucket_count + 1 > + known_bucket_count = known_bucket_count + 1 > end > end > + router.known_bucket_count = known_bucket_count > if router.failover_fiber == nil then > router.failover_fiber = util.reloadable_fiber_create( > 'vshard.failover.' .. router.name, M, 'failover_f', router) From olegrok at tarantool.org Fri Aug 7 11:07:22 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Fri, 7 Aug 2020 11:07:22 +0300 Subject: [Tarantool-patches] [PATCH vshard 1/1] replicaset: check URI match when rebind connection In-Reply-To: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> References: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> Message-ID: <9b0b6fad-f296-aec9-881a-1efe13e89f72@tarantool.org> Hi! Thanks for your patch. LGTM. On 07/08/2020 00:48, Vladislav Shpilevoy wrote: > Reconfiguration in router and storage always creates new > replica objects, and outdates old objects. Since most of the > time reconfiguration does not change cluster topology and URIs, > old connections are relocated to the new replicaset objects so as > not to reconnect on each router.cfg() and storage.cfg(). > > Connections were moved from old objects to new ones by UUID match > not taking into account possible URI change. > > If URI was changed, the new replica object took the old connection > with old URI, and kept trying to connect to the old address. > > The patch makes connection relocation work only if old and new URI > match. > > Closes #245 > --- > Branch:http://github.com/tarantool/vshard/tree/gerold103/gh-245-router-reconnect > Issue:https://github.com/tarantool/vshard/issues/245 > > test/router/reconnect_to_master.result | 55 ++++++++++++++++++++++++ > test/router/reconnect_to_master.test.lua | 26 +++++++++++ > vshard/replicaset.lua | 2 +- > 3 files changed, 82 insertions(+), 1 deletion(-) > > diff --git a/test/router/reconnect_to_master.result b/test/router/reconnect_to_master.result > index d640f0b..aa33f9c 100644 > --- a/test/router/reconnect_to_master.result > +++ b/test/router/reconnect_to_master.result > @@ -169,6 +169,61 @@ is_disconnected() > --- > - false > ... > +-- > +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was changed in > +-- the config for any instance, it used old connection, because reconfiguration > +-- compared connections by UUID instead of URI. > +-- > +util = require('util') > +--- > +... > +-- Firstly, clean router from storage_1_a connection. > +rs1_uuid = util.replicasets[1] > +--- > +... > +rs1_cfg = cfg.sharding[rs1_uuid] > +--- > +... > +cfg.sharding[rs1_uuid] = nil > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Now break the URI in the config. > +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri > +--- > +... > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = 'https://bad_uri.com:123' > +--- > +... > +-- Apply the bad config. > +cfg.sharding[rs1_uuid] = rs1_cfg > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Should fail - master is not available because of the bad URI. > +res, err = vshard.router.callrw(1, 'echo', {1}) > +--- > +... > +res == nil and err ~= nil > +--- > +- true > +... > +-- Repair the config. > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri > +--- > +... > +vshard.router.cfg(cfg) > +--- > +... > +-- Should drop the old connection object and connect fine. > +vshard.router.callrw(1, 'echo', {1}) > +--- > +- 1 > +... > _ = test_run:switch("default") > --- > ... > diff --git a/test/router/reconnect_to_master.test.lua b/test/router/reconnect_to_master.test.lua > index c315c9f..87270af 100644 > --- a/test/router/reconnect_to_master.test.lua > +++ b/test/router/reconnect_to_master.test.lua > @@ -70,6 +70,32 @@ while is_disconnected() and i < max_iters do i = i + 1 fiber.sleep(0.1) end > -- Master connection is active again. > is_disconnected() > > +-- > +-- gh-245: dynamic uri reconfiguration didn't work - even if URI was changed in > +-- the config for any instance, it used old connection, because reconfiguration > +-- compared connections by UUID instead of URI. > +-- > +util = require('util') > +-- Firstly, clean router from storage_1_a connection. > +rs1_uuid = util.replicasets[1] > +rs1_cfg = cfg.sharding[rs1_uuid] > +cfg.sharding[rs1_uuid] = nil > +vshard.router.cfg(cfg) > +-- Now break the URI in the config. > +old_uri = rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = 'https://bad_uri.com:123' > +-- Apply the bad config. > +cfg.sharding[rs1_uuid] = rs1_cfg > +vshard.router.cfg(cfg) > +-- Should fail - master is not available because of the bad URI. > +res, err = vshard.router.callrw(1, 'echo', {1}) > +res == nil and err ~= nil > +-- Repair the config. > +rs1_cfg.replicas[util.name_to_uuid.storage_1_a].uri = old_uri > +vshard.router.cfg(cfg) > +-- Should drop the old connection object and connect fine. > +vshard.router.callrw(1, 'echo', {1}) > + > _ = test_run:switch("default") > _ = test_run:cmd('stop server router_1') > _ = test_run:cmd('cleanup server router_1') > diff --git a/vshard/replicaset.lua b/vshard/replicaset.lua > index 0191936..b13d05e 100644 > --- a/vshard/replicaset.lua > +++ b/vshard/replicaset.lua > @@ -456,7 +456,7 @@ local function rebind_replicasets(replicasets, old_replicasets) > for replica_uuid, replica in pairs(replicaset.replicas) do > local old_replica = old_replicaset and > old_replicaset.replicas[replica_uuid] > - if old_replica then > + if old_replica and old_replica.uri == replica.uri then > local conn = old_replica.conn > replica.conn = conn > replica.down_ts = old_replica.down_ts From imun at tarantool.org Fri Aug 7 12:19:46 2020 From: imun at tarantool.org (Igor Munkin) Date: Fri, 7 Aug 2020 12:19:46 +0300 Subject: [Tarantool-patches] [PATCH 00/19] Add static analysis of Lua code in test/ with luacheck In-Reply-To: References: Message-ID: <20200807091946.GX18920@tarantool.org> Sergey, I see little changes changes from the previous version. However, I addressed the following comment to you in my last review[1]: |> > Sergey, |> > |> > I guess we definitely need another series for this. |> |> Ja-ja, will do. |> |> > Furthermore, IIRC we |> > decided to leave shadowing, so the current changes are simply not |> > relevant. Please adjust the patches related to tests considering our |> > recent discussion in cover letter and send a new patchset. |> |> I have reverted all changes related to globally supressed warnings. |> As I can see you claimed the irrelevant changes are reverted. Unfortunately, I still see excess hunks in the patch[2] (e.g. related to arguments and locals shadowing). Sorry, you *might* be misled by my wording, since I mentioned cover letter, not the Sasha's review for the previous patch[3]. OK, let's try one more time: please consider the results of the discussion we had related to source Lua files series. On 16.07.20, sergeyb at tarantool.org wrote: > From: Sergey Bronnikov > > Note: patch series for test/ was already sent to list in a thread with > static analysis for src/ and extra/. Igor M. started to review it there, see > [1], and then in private discussion we decided to split changes for > test/ and {src/,extra/} for two different patch series. As changes for src/ and > extra/ has been merged to master [2] I have rebased existed changes for > test/, applied changes suggested by Igor in the initial review and > reverted changes for warnings supressed globally in .luacheckrc. > > 1. https://lists.tarantool.org/pipermail/tarantool-patches/2020-June/017301.html > 2. https://lists.tarantool.org/pipermail/tarantool-patches/2020-July/018568.html > > Changelog v2: > > - reverted changes for diff-based tests > - reverted changes for suppressed warnings > - format .luacheckrc according to Lua style guide > - created GH issue for every ignored warning in a separated files > - make each commit atomic again - all changes in commits are related to > commit itself, not other commits > > GH issue: https://github.com/tarantool/tarantool/issues/4681 > GitHub branch: https://github.com/tarantool/tarantool/tree/ligurio/gh-4681-fix-luacheck-warnings-test > CI: https://gitlab.com/tarantool/tarantool/-/pipelines/167419608 > > Sergey Bronnikov (19): > Fix luacheck warnings in test/app-tap > Fix luacheck warnings in test/app > Fix luacheck warnings in test/box > Fix luacheck warnings in test/box-py > Fix luacheck warnings in test/box-tap > Fix luacheck warnings in test/engine > Fix luacheck warnings in test/engine_long > Fix luacheck warnings in test/long_run-py > Fix luacheck warnings in test/replication > Fix luacheck warnings in test/replication-py > Fix luacheck warnings in test/sql > Fix luacheck warnings in test/sql-tap > Fix luacheck warnings in test/swim > Fix luacheck warnings in test/vinyl > Fix luacheck warnings in test/wal_off > Fix luacheck warnings in test/xlog > Fix luacheck warnings in test/xlog-py > Add luacheck supression for luajit test > luajit: bump new version > > .luacheckrc | 195 ++++++- > test/app-tap/cfg.test.lua | 2 +- > test/app-tap/clock.test.lua | 4 +- > test/app-tap/console.test.lua | 14 +- > test/app-tap/console_lua.test.lua | 18 +- > test/app-tap/csv.test.lua | 56 +- > test/app-tap/debug.test.lua | 10 +- > test/app-tap/errno.test.lua | 24 +- > test/app-tap/fail_main.test.lua | 6 +- > .../gh-4761-json-per-call-options.test.lua | 11 +- > test/app-tap/http_client.test.lua | 297 +++++------ > test/app-tap/iconv.test.lua | 6 +- > test/app-tap/init_script.test.lua | 16 +- > test/app-tap/inspector.test.lua | 5 +- > test/app-tap/json.test.lua | 1 - > test/app-tap/logger.test.lua | 23 +- > test/app-tap/lua/serializer_test.lua | 46 +- > test/app-tap/minimal.test.lua | 4 +- > test/app-tap/module_api.test.lua | 12 +- > test/app-tap/msgpackffi.test.lua | 3 +- > test/app-tap/pcall.test.lua | 6 +- > test/app-tap/popen.test.lua | 35 +- > test/app-tap/snapshot.test.lua | 17 +- > test/app-tap/string.test.lua | 502 +++++++++--------- > test/app-tap/tap.test.lua | 26 +- > test/app-tap/tarantoolctl.test.lua | 82 ++- > test/app-tap/trigger.test.lua | 48 +- > test/app-tap/yaml.test.lua | 16 +- > test/box-py/box.lua | 2 +- > test/box-tap/auth.test.lua | 20 +- > test/box-tap/cfg.test.lua | 34 +- > test/box-tap/cfgup.test.lua | 4 +- > test/box-tap/extended_error.test.lua | 11 +- > test/box-tap/feedback_daemon.test.lua | 4 +- > test/box-tap/gc.test.lua | 4 +- > test/box-tap/on_schema_init.test.lua | 4 +- > test/box-tap/schema_mt.test.lua | 22 +- > test/box-tap/session.storage.test.lua | 10 +- > test/box-tap/session.test.lua | 56 +- > test/box-tap/trigger_atexit.test.lua | 10 +- > test/box-tap/trigger_yield.test.lua | 10 +- > test/box/box.lua | 7 +- > test/box/lua/bitset.lua | 7 +- > test/box/lua/cfg_bad_vinyl_dir.lua | 2 +- > test/box/lua/cfg_rtree.lua | 2 +- > test/box/lua/cfg_test1.lua | 2 +- > test/box/lua/cfg_test2.lua | 2 +- > test/box/lua/cfg_test3.lua | 2 +- > test/box/lua/cfg_test4.lua | 2 +- > test/box/lua/cfg_test5.lua | 2 +- > test/box/lua/cfg_test6.lua | 2 +- > test/box/lua/fifo.lua | 2 +- > test/box/lua/identifier.lua | 9 +- > test/box/lua/index_random_test.lua | 2 +- > test/box/lua/utils.lua | 10 +- > test/box/on_schema_init.lua | 8 +- > test/box/proxy.lua | 2 +- > test/box/tiny.lua | 2 +- > test/engine/box.lua | 8 +- > test/engine/conflict.lua | 6 +- > test/engine_long/suite.lua | 5 +- > test/long_run-py/lua/finalizers.lua | 7 +- > test/long_run-py/suite.lua | 7 +- > test/replication-py/master.lua | 2 +- > test/replication-py/panic.lua | 2 +- > test/replication-py/replica.lua | 4 - > test/replication/lua/fast_replica.lua | 9 +- > test/replication/lua/rlimit.lua | 2 +- > test/replication/master.lua | 2 +- > test/replication/master_quorum.lua | 4 +- > test/replication/on_replace.lua | 6 +- > test/replication/replica.lua | 4 +- > test/replication/replica_on_schema_init.lua | 4 +- > test/sql-tap/alter.test.lua | 4 +- > test/sql-tap/analyze5.test.lua | 2 +- > test/sql-tap/analyze9.test.lua | 22 +- > test/sql-tap/between.test.lua | 4 +- > test/sql-tap/date.test.lua | 3 +- > test/sql-tap/delete1.test.lua | 2 +- > test/sql-tap/e_delete.test.lua | 2 +- > test/sql-tap/e_expr.test.lua | 11 +- > test/sql-tap/func.test.lua | 2 +- > test/sql-tap/func3.test.lua | 2 +- > test/sql-tap/gh-2723-concurrency.test.lua | 8 +- > .../gh-3083-ephemeral-unref-tuples.test.lua | 2 +- > .../gh-3307-xfer-optimization-issue.test.lua | 2 +- > .../gh-3332-tuple-format-leak.test.lua | 2 +- > .../gh2127-indentifier-max-length.test.lua | 10 +- > test/sql-tap/identifier-characters.test.lua | 2 +- > test/sql-tap/index1.test.lua | 3 +- > test/sql-tap/index7.test.lua | 2 +- > test/sql-tap/join3.test.lua | 2 +- > test/sql-tap/lua-tables.test.lua | 2 +- > test/sql-tap/lua/sqltester.lua | 11 +- > test/sql-tap/misc1.test.lua | 10 +- > test/sql-tap/misc5.test.lua | 2 +- > test/sql-tap/select1.test.lua | 10 +- > test/sql-tap/select2.test.lua | 8 +- > test/sql-tap/select4.test.lua | 1 - > test/sql-tap/select5.test.lua | 1 - > test/sql-tap/select9.test.lua | 12 +- > test/sql-tap/selectA.test.lua | 8 +- > test/sql-tap/selectB.test.lua | 14 +- > test/sql-tap/selectG.test.lua | 1 - > test/sql-tap/sort.test.lua | 2 +- > test/sql-tap/sql-errors.test.lua | 2 +- > test/sql-tap/table.test.lua | 3 +- > test/sql-tap/tkt-38cb5df375.test.lua | 1 - > test/sql-tap/tkt-91e2e8ba6f.test.lua | 3 - > test/sql-tap/tkt-9a8b09f8e6.test.lua | 3 - > test/sql-tap/tkt-bd484a090c.test.lua | 3 +- > test/sql-tap/tkt2192.test.lua | 3 +- > test/sql-tap/tkt3493.test.lua | 3 - > test/sql-tap/trigger2.test.lua | 4 +- > test/sql-tap/triggerA.test.lua | 1 - > test/sql-tap/where2.test.lua | 9 +- > test/sql-tap/where3.test.lua | 2 +- > test/sql-tap/where4.test.lua | 4 +- > test/sql-tap/where5.test.lua | 2 +- > test/sql-tap/where6.test.lua | 2 +- > test/sql-tap/where7.test.lua | 16 +- > test/sql-tap/whereA.test.lua | 2 +- > test/sql-tap/whereB.test.lua | 2 +- > test/sql-tap/whereC.test.lua | 5 +- > test/sql-tap/whereD.test.lua | 4 +- > test/sql-tap/whereF.test.lua | 4 +- > test/sql-tap/whereG.test.lua | 4 +- > test/sql-tap/whereI.test.lua | 4 +- > test/sql-tap/whereK.test.lua | 4 +- > test/sql-tap/with1.test.lua | 14 +- > test/sql-tap/with2.test.lua | 18 +- > test/sql/lua/sql_tokenizer.lua | 2 +- > test/swim/box.lua | 4 +- > test/vinyl/large.lua | 3 +- > test/vinyl/stress.lua | 22 +- > test/vinyl/txn_proxy.lua | 6 +- > test/vinyl/upgrade/fill.lua | 8 +- > test/vinyl/vinyl.lua | 17 - > test/xlog-py/box.lua | 2 +- > test/xlog/panic.lua | 2 +- > .../2.1.3/gh-4771-upgrade-sequence/fill.lua | 12 +- > test/xlog/xlog.lua | 2 +- > third_party/luajit | 2 +- > 143 files changed, 1135 insertions(+), 1051 deletions(-) > > -- > 2.26.2 > [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-July/018632.html [2]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-July/018634.html [3]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-June/017304.html -- Best regards, IM From avtikhon at tarantool.org Fri Aug 7 19:10:35 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 7 Aug 2020 19:10:35 +0300 Subject: [Tarantool-patches] [PATCH v1] update_repo: correct fix for RPMs on missing metadata In-Reply-To: <20200727141840.fisd4tim7fxel55u@tkn_work_nb> References: <9d538c87b2a0106b325c1ec104206a9f38f207df.1595855406.git.avtikhon@tarantool.org> <20200727141840.fisd4tim7fxel55u@tkn_work_nb> Message-ID: <20200807160953.GA8091@hpalx> Hi Alexander, thanks a lot for the review and the message for the code. I've used it for commit comments and code message. On Mon, Jul 27, 2020 at 05:18:40PM +0300, Alexander Turenko wrote: > On Mon, Jul 27, 2020 at 04:11:06PM +0300, Alexander V. Tikhonov wrote: > > Found that removing RPMs additional call to its removement is > > needed, when no metadata was found. > > --- > > > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/correct_rpm_remove > > > > tools/update_repo.sh | 7 +++++++ > > 1 file changed, 7 insertions(+) > > > > diff --git a/tools/update_repo.sh b/tools/update_repo.sh > > index 5a68e3e05..d9b91a753 100755 > > --- a/tools/update_repo.sh > > +++ b/tools/update_repo.sh > > @@ -869,6 +869,13 @@ function remove_rpm { > > done > > done > > > > + # remove all found file by the given pattern in options > > I would describe the case a bit: the loop above already delete files, > which are present in the metadata. However it is possible that some > broken update left orphan files: they are present in the storage, but > does not mentioned in the metadata. > > > + for suffix in 'x86_64' 'noarch' 'src'; do > > + file="$bucket_path/$packpath/${remove}-1.${os}${option_dist}.${suffix}.rpm" > > + $aws ls $file || continue > > + $aws rm $file > > + done > > + > > I don't mind, however I'm unable to review the file thoroughly. From v.shpilevoy at tarantool.org Sat Aug 8 00:56:01 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Fri, 7 Aug 2020 23:56:01 +0200 Subject: [Tarantool-patches] [PATCH vshard 1/1] replicaset: check URI match when rebind connection In-Reply-To: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> References: <7f34d056c1f2b0f461740a4b04221881074106cd.1596750426.git.v.shpilevoy@tarantool.org> Message-ID: <1187d1e0-49e1-28c9-6404-45430e9594a4@tarantool.org> Pushed to master. From korablev at tarantool.org Sat Aug 8 17:21:45 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Sat, 8 Aug 2020 17:21:45 +0300 Subject: [Tarantool-patches] [PATCH V2] vinyl: rework upsert operation Message-ID: Previous upsert implementation had a few drawbacks which led to number of bugs and issues. Issue #5092 (redundant update operations execution) In a nutshell, application of upsert(s) (on top of another upsert) consists of two actions (see vy_apply_upsert()): execute and squash. Consider example: insert({1, 1}) -- terminal statement, stored on disk upsert({1}, {{'-', 2, 20}}) -- old ups1 upsert({1}, {{'+', 2, 10}}) -- new ups2 'Execute' step takes update operations from the new upsert and combines them with key of the old upsert. {1} + {'+', 2, 10} can't be evaluated since key consists of only one field. Note that in case upsert doesn't fold into insert the upsert's tuple and the tuple stored in index can be different In our particular case, tuple stored on disk has two fields ({1, 1}), so first upsert's update operation can be applied to it: {1, 1} + {'+', 2, 10} --> {1, 11}. If upsert's operation can't be executed using key of old upsert, we simply continue processing squash step. In turn 'squash' is a combination of update operations: arithmetic operations are combined so we don't have to store actions over the same field; the rest operations - are merged into single array. As a result, we get one upsert with squashed operations: upsert({1}, {{'+', 2, -10}}). Then vy_apply_upsert() is called again to apply new upsert on the top of terminal statement - insert{1, 1}. Since now tuple has second field, update operations can be executed and corresponding result is {1, -9}. It is the final result of upsert application procedure. Now imagine that we have following upserts: upsert({1, 1}, {{'-', 2, 20}}) -- old ups1 upsert({1}, {{'+', 2, 10}}) -- new ups2 In this case execution successfully finishes and modifies old upsert's tuple: {1, 1} + {'+', 2, 10} --> {1, 11} However, we still have to squash/accumulate update operations since they may be applied on tuple stored on disk later. After all, we have following upsert: upsert({2, 11}, {{'+', 2, -10}}). Then it is applied on the top of insert({1, 1}) and we get the same result as in the first case - {1, -9}. The only difference is that upsert's tuple was modified. As one can see, execution of update operations applied to upsert's tuple is redundant in the case index already contains tuple with the same key (i.e. when upserts turns into update). Instead, we are able to accumulate/squash update operations only. When the last upsert is being applied, we can either execute all update operation on tuple fetched from index (i.e. upsert is update) OR on tuple specified in the first upsert (i.e. first upsert is insert). Issue #5105 (upsert doesn't follow associative property) Secondly, current approach breaks associative property: after upsert's update operations are merged into one array, part of them (related to one upsert) can be skipped, meanwhile the rest - is applied. For instance: -- Index is over second field. i = s:create_index('pk', {parts={2, 'uint'}}) s:replace{1, 2, 3, 'default'} s:upsert({2, 2, 2}, {{'=', 4, 'upserted'}}) -- First update operation modifies primary key, so upsert must be ignored. s:upsert({2, 2, 2}, {{'#', 1, 1}, {'!', 3, 1}}) After merging two upserts we get the next one: upsert({2, 2, 2}, {{'=', 4, 'upserted'}, {'#', 1, 1}, {'!', 3, 1}} While we executing update operations, we don't distinguish operations from different upserts. Thus, if one operation fails, the rest are ignored as well. As a result, first (in general case - all preceding squashed upserts) upsert won't be applied, even despite the fact it is absolutely correct. What is more, user gets no error/warning concerning this fact. Issue #1622 (no upsert result validation) After upsert application, there's no check verifying that result satisfies space's format: number of fields, their types, overflows etc. Due to this tuples violating format may appear in the space, which in turn may lead to unpredictable consequences. To resolve these issues, let's group update operations of each upsert into separate array. So that operations related to particular upsert are stored in single array. In terms of previous example we will get: upsert({2, 2, 2}, {{{'=', 4, 'upserted'}}, {{'#', 1, 1}, {'!', 3, 1}}} Also note that we don't longer have to apply update operations on tuple in vy_apply_upsert() when it comes for two upserts: it can be done once we face terminal statement; or if there's no underlying statement (i.e. it is delete statement or no statement at all) we apply all update arrays except the first one on upsert's tuple. In case one of operations from array fail, we skip the rest operations from this array and process to the next array. After successful application of update operations of each array, we check that the resulting tuple fits into space format. If they aren't, we rollback applied operations, log error and moving to the next group of operations. Arithmetic operations still can be combined in case there's no unsigned fields in space format. Otherwise, result of subtraction can turn out to be negative and resulting tuple won't satisfy this property. Closes #1622 Closes #5105 Closes #5092 Part of #5107 --- Issues: https://github.com/tarantool/tarantool/issues/1622 https://github.com/tarantool/tarantool/issues/5105 https://github.com/tarantool/tarantool/issues/5092 https://github.com/tarantool/tarantool/issues/5107 Branch: https://github.com/tarantool/tarantool/tree/np/gh-5107-dont-squash-ops @ChangeLog: - Rework upsert operation in vinyl so that now (gh-5107): - if upsert can't be applied it is skipped and corresponding error is logged (gh-1622); - upserts now follow associative property: result of several upserts doesn't depend on the order of their application (gh-5105); - upserts referring to -1 fieldno are handled correctly now (gh-5087). src/box/vinyl.c | 2 +- src/box/vy_stmt.c | 28 ++- src/box/vy_stmt.h | 5 +- src/box/vy_upsert.c | 370 ++++++++++++++++++++---------- src/box/xrow_update.c | 61 +++++ test/unit/vy_iterators_helper.c | 2 +- test/vinyl/upsert.result | 394 ++++++++++++++++++++++++++++++++ test/vinyl/upsert.test.lua | 165 +++++++++++++ 8 files changed, 893 insertions(+), 134 deletions(-) diff --git a/src/box/vinyl.c b/src/box/vinyl.c index 32301d7ba..eab688147 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -1984,7 +1984,7 @@ vy_lsm_upsert(struct vy_tx *tx, struct vy_lsm *lsm, operations[0].iov_base = (void *)expr; operations[0].iov_len = expr_end - expr; vystmt = vy_stmt_new_upsert(lsm->mem_format, tuple, tuple_end, - operations, 1); + operations, 1, false); if (vystmt == NULL) return -1; assert(vy_stmt_type(vystmt) == IPROTO_UPSERT); diff --git a/src/box/vy_stmt.c b/src/box/vy_stmt.c index 92e0aa1c5..2ee82b3f2 100644 --- a/src/box/vy_stmt.c +++ b/src/box/vy_stmt.c @@ -313,16 +313,22 @@ vy_key_dup(const char *key) static struct tuple * vy_stmt_new_with_ops(struct tuple_format *format, const char *tuple_begin, const char *tuple_end, struct iovec *ops, - int op_count, enum iproto_type type) + int op_count, enum iproto_type type, bool is_ops_encoded) { mp_tuple_assert(tuple_begin, tuple_end); const char *tmp = tuple_begin; mp_decode_array(&tmp); + /* + * ops are grouped in one extra array. + * See vy_apply_upsert() for details. + */ size_t ops_size = 0; for (int i = 0; i < op_count; ++i) ops_size += ops[i].iov_len; + if (!is_ops_encoded) + ops_size += mp_sizeof_array(op_count); struct tuple *stmt = NULL; struct region *region = &fiber()->gc; @@ -360,6 +366,8 @@ vy_stmt_new_with_ops(struct tuple_format *format, const char *tuple_begin, field_map_build(&builder, wpos - field_map_size); memcpy(wpos, tuple_begin, mpsize); wpos += mpsize; + if (!is_ops_encoded) + wpos = mp_encode_array(wpos, op_count); for (struct iovec *op = ops, *end = ops + op_count; op != end; ++op) { memcpy(wpos, op->iov_base, op->iov_len); @@ -374,10 +382,11 @@ end: struct tuple * vy_stmt_new_upsert(struct tuple_format *format, const char *tuple_begin, const char *tuple_end, struct iovec *operations, - uint32_t ops_cnt) + uint32_t ops_cnt, bool is_ops_encoded) { return vy_stmt_new_with_ops(format, tuple_begin, tuple_end, - operations, ops_cnt, IPROTO_UPSERT); + operations, ops_cnt, IPROTO_UPSERT, + is_ops_encoded); } struct tuple * @@ -385,7 +394,7 @@ vy_stmt_new_replace(struct tuple_format *format, const char *tuple_begin, const char *tuple_end) { return vy_stmt_new_with_ops(format, tuple_begin, tuple_end, - NULL, 0, IPROTO_REPLACE); + NULL, 0, IPROTO_REPLACE, true); } struct tuple * @@ -393,7 +402,7 @@ vy_stmt_new_insert(struct tuple_format *format, const char *tuple_begin, const char *tuple_end) { return vy_stmt_new_with_ops(format, tuple_begin, tuple_end, - NULL, 0, IPROTO_INSERT); + NULL, 0, IPROTO_INSERT, true); } struct tuple * @@ -401,7 +410,7 @@ vy_stmt_new_delete(struct tuple_format *format, const char *tuple_begin, const char *tuple_end) { return vy_stmt_new_with_ops(format, tuple_begin, tuple_end, - NULL, 0, IPROTO_DELETE); + NULL, 0, IPROTO_DELETE, true); } struct tuple * @@ -735,19 +744,20 @@ vy_stmt_decode(struct xrow_header *xrow, struct tuple_format *format) /* Always use key format for DELETE statements. */ stmt = vy_stmt_new_with_ops(env->key_format, request.key, request.key_end, - NULL, 0, IPROTO_DELETE); + NULL, 0, IPROTO_DELETE, true); break; case IPROTO_INSERT: case IPROTO_REPLACE: stmt = vy_stmt_new_with_ops(format, request.tuple, request.tuple_end, - NULL, 0, request.type); + NULL, 0, request.type, true); break; case IPROTO_UPSERT: ops.iov_base = (char *)request.ops; ops.iov_len = request.ops_end - request.ops; stmt = vy_stmt_new_upsert(format, request.tuple, - request.tuple_end, &ops, 1); + request.tuple_end, &ops, + 1, true); break; default: /* TODO: report filename. */ diff --git a/src/box/vy_stmt.h b/src/box/vy_stmt.h index 25219230d..65acbecac 100644 --- a/src/box/vy_stmt.h +++ b/src/box/vy_stmt.h @@ -528,6 +528,8 @@ vy_stmt_new_delete(struct tuple_format *format, const char *tuple_begin, * @param part_count Part count from key definition. * @param operations Vector of update operations. * @param ops_cnt Length of the update operations vector. + * @param is_ops_encoded True, if update operations are already packed + * into extra msgpack array. * * @retval NULL Memory allocation error. * @retval not NULL Success. @@ -535,7 +537,8 @@ vy_stmt_new_delete(struct tuple_format *format, const char *tuple_begin, struct tuple * vy_stmt_new_upsert(struct tuple_format *format, const char *tuple_begin, const char *tuple_end, - struct iovec *operations, uint32_t ops_cnt); + struct iovec *operations, uint32_t ops_cnt, + bool is_ops_encoded); /** * Create REPLACE statement from UPSERT statement. diff --git a/src/box/vy_upsert.c b/src/box/vy_upsert.c index e697b6321..3811e91b3 100644 --- a/src/box/vy_upsert.c +++ b/src/box/vy_upsert.c @@ -39,38 +39,217 @@ #include "column_mask.h" /** - * Try to squash two upsert series (msgspacked index_base + ops) - * Try to create a tuple with squahed operations + * Check that key hasn't been changed after applying upsert operation. + */ +static bool +vy_apply_result_does_cross_pk(struct tuple *old_stmt, const char *result, + const char *result_end, struct key_def *cmp_def, + uint64_t col_mask) +{ + if (!key_update_can_be_skipped(cmp_def->column_mask, col_mask)) { + struct tuple *tuple = + vy_stmt_new_replace(tuple_format(old_stmt), result, + result_end); + int cmp_res = vy_stmt_compare(old_stmt, HINT_NONE, tuple, + HINT_NONE, cmp_def); + tuple_unref(tuple); + return cmp_res != 0; + } + return false; +} + +/** + * Apply update operations stored in @a upsert on tuple @a stmt. If @a stmt is + * void statement (i.e. it is NULL or delete statement) then operations are + * applied on tuple stored in @a upsert. Update operations of @a upsert which + * can't be applied are skipped along side with other operations from single + * group (i.e. packed in one msgpack array); errors may be logged depending on + * @a suppress_error flag. * - * @retval 0 && *result_stmt != NULL : successful squash - * @retval 0 && *result_stmt == NULL : unsquashable sources - * @retval -1 - memory error + * @param upsert Upsert statement to be applied on @a stmt. + * @param stmt Statement to be used as base for upsert operations. + * @param cmp_def Key definition required to provide check of primary key + * modification. + * @return Tuple containing result of upsert application; NULL in case OOM. + */ +static struct tuple * +vy_apply_upsert_on_terminal_stmt(struct tuple *upsert, struct tuple *stmt, + struct key_def *cmp_def, bool suppress_error) +{ + assert(vy_stmt_type(upsert) == IPROTO_UPSERT); + assert(stmt == NULL || vy_stmt_type(stmt) != IPROTO_UPSERT); + + uint32_t mp_size; + const char *new_ops = vy_stmt_upsert_ops(upsert, &mp_size); + /* Msgpack containing result of upserts application. */ + const char *result_mp; + bool stmt_is_void = stmt == NULL || vy_stmt_type(stmt) == IPROTO_DELETE; + if (stmt_is_void) + result_mp = vy_upsert_data_range(upsert, &mp_size); + else + result_mp = tuple_data_range(stmt, &mp_size); + const char *result_mp_end = result_mp + mp_size; + /* + * xrow_upsert_execute() allocates result using region, + * so save starting point to release it later. + */ + struct region *region = &fiber()->gc; + size_t region_svp = region_used(region); + uint64_t column_mask = COLUMN_MASK_FULL; + struct tuple_format *format = tuple_format(upsert); + + uint32_t ups_cnt = mp_decode_array(&new_ops); + const char *ups_ops = new_ops; + /* + * In case upsert folds into insert, we must skip first + * update operations. + */ + if (stmt_is_void) { + ups_cnt--; + mp_next(&ups_ops); + } + for (uint32_t i = 0; i < ups_cnt; ++i) { + assert(mp_typeof(*ups_ops) == MP_ARRAY); + const char *ups_ops_end = ups_ops; + mp_next(&ups_ops_end); + const char *exec_res = result_mp; + exec_res = xrow_upsert_execute(ups_ops, ups_ops_end, result_mp, + result_mp_end, format, &mp_size, + 0, suppress_error, &column_mask); + if (exec_res == NULL) { + if (! suppress_error) { + struct error *e = diag_last_error(diag_get()); + assert(e != NULL); + /* Bail out immediately in case of OOM. */ + if (e->type != &type_ClientError) { + region_truncate(region, region_svp); + return NULL; + } + diag_log(); + } + ups_ops = ups_ops_end; + continue; + } + /* + * If it turns out that resulting tuple modifies primary + * key, then simply ignore this upsert. + */ + if (vy_apply_result_does_cross_pk(stmt, exec_res, + exec_res + mp_size, cmp_def, + column_mask)) { + if (!suppress_error) { + say_error("upsert operations %s are not applied"\ + " due to primary key modification", + mp_str(ups_ops)); + } + ups_ops = ups_ops_end; + continue; + } + ups_ops = ups_ops_end; + /* + * Result statement must satisfy space's format. Since upsert's + * tuple correctness is already checked in vy_upsert(), let's + * use its format to provide result verification. + */ + struct tuple_format *format = tuple_format(upsert); + if (tuple_validate_raw(format, exec_res) != 0) { + if (! suppress_error) + diag_log(); + continue; + } + result_mp = exec_res; + result_mp_end = exec_res + mp_size; + } + struct tuple *new_terminal_stmt = vy_stmt_new_replace(format, result_mp, + result_mp_end); + region_truncate(region, region_svp); + if (new_terminal_stmt == NULL) + return NULL; + vy_stmt_set_lsn(new_terminal_stmt, vy_stmt_lsn(upsert)); + return new_terminal_stmt; +} + +/** + * Unpack upsert's update operations from msgpack array + * into array of iovecs. */ +static void +upsert_ops_to_iovec(const char *ops, uint32_t ops_cnt, struct iovec *iov_arr) +{ + for (uint32_t i = 0; i < ops_cnt; ++i) { + assert(mp_typeof(*ops) == MP_ARRAY); + iov_arr[i].iov_base = (char *) ops; + mp_next(&ops); + iov_arr[i].iov_len = ops - (char *) iov_arr[i].iov_base; + } +} + +/** + * Attempt at squashing arithmetic operations applied on the same tuple + * fields. We never squash first member (i.e. operations from the oldest + * upsert) from old operations, since it must be skipped in case such + * upsert turns into insert. + * Also note that content of @a new_ops and @a old_ops arrays can be modified + * as a result of function invocation. + * Array containing resulting operations is allocated using region. + * In case of OOM function returns -1; 0 otherwise. + * */ static int vy_upsert_try_to_squash(struct tuple_format *format, - const char *key_mp, const char *key_mp_end, - const char *old_serie, const char *old_serie_end, - const char *new_serie, const char *new_serie_end, - struct tuple **result_stmt) + struct iovec *new_ops, uint32_t new_ops_cnt, + struct iovec *old_ops, uint32_t old_ops_cnt, + struct iovec **res_ops, uint32_t *res_ops_cnt) { - *result_stmt = NULL; - - size_t squashed_size; - const char *squashed = - xrow_upsert_squash(old_serie, old_serie_end, - new_serie, new_serie_end, format, - &squashed_size); - if (squashed == NULL) + uint32_t squashed_ops = 0; + for (uint32_t i = 0; i < new_ops_cnt; ++i) { + const char *new = (const char *) new_ops[i].iov_base; + const char *new_end = new + new_ops[i].iov_len; + /* + * Don't touch first update operation since it should + * be skipped in case first upsert folds into insert. + */ + for (uint32_t j = 1; j < old_ops_cnt; ++j) { + const char *old = + (const char *) old_ops[j].iov_base; + const char *old_end = + (const char *) old + old_ops[j].iov_len; + /* Operation may already be squashed. */ + if (old == NULL) + continue; + size_t squashed_size; + const char *res = + xrow_upsert_squash(old, old_end, new, new_end, + format, &squashed_size); + if (res != NULL) { + new_ops[i].iov_base = (char *) res; + new_ops[i].iov_len = squashed_size; + old_ops[j].iov_base = NULL; + old_ops[j].iov_len = 0; + squashed_ops++; + } + } + } + *res_ops_cnt = new_ops_cnt + old_ops_cnt - squashed_ops; + if (squashed_ops == 0) { + *res_ops = old_ops; return 0; - /* Successful squash! */ - struct iovec operations[1]; - operations[0].iov_base = (void *)squashed; - operations[0].iov_len = squashed_size; - - *result_stmt = vy_stmt_new_upsert(format, key_mp, key_mp_end, - operations, 1); - if (*result_stmt == NULL) + } + size_t ops_size; + *res_ops = region_alloc_array(&fiber()->gc, typeof(*res_ops[0]), + *res_ops_cnt, &ops_size); + if (*res_ops == NULL) { + diag_set(OutOfMemory, ops_size, "region_alloc_array", + "res_ops"); return -1; + } + /* Fill gaps (i.e. old squashed operations) in the resulting array. */ + for (uint32_t i = 0; i < old_ops_cnt; ++i) { + if (old_ops[i].iov_base != NULL) + (*res_ops)[i] = old_ops[i]; + } + for (uint32_t i = old_ops_cnt - squashed_ops, j = 0; j < new_ops_cnt; + ++i, ++j) + (*res_ops)[i] = new_ops[j]; return 0; } @@ -87,122 +266,69 @@ vy_apply_upsert(struct tuple *new_stmt, struct tuple *old_stmt, assert(new_stmt != old_stmt); assert(vy_stmt_type(new_stmt) == IPROTO_UPSERT); - if (old_stmt == NULL || vy_stmt_type(old_stmt) == IPROTO_DELETE) { - /* - * INSERT case: return new stmt. - */ - return vy_stmt_replace_from_upsert(new_stmt); + struct tuple *result_stmt = NULL; + if (old_stmt == NULL || vy_stmt_type(old_stmt) != IPROTO_UPSERT) { + return vy_apply_upsert_on_terminal_stmt(new_stmt, old_stmt, + cmp_def, suppress_error); } - struct tuple_format *format = tuple_format(new_stmt); - + assert(old_stmt != NULL); + assert(vy_stmt_type(old_stmt) == IPROTO_UPSERT); /* - * Unpack UPSERT operation from the new stmt + * Unpack UPSERT operation from the old and new stmts. */ uint32_t mp_size; - const char *new_ops; - new_ops = vy_stmt_upsert_ops(new_stmt, &mp_size); - const char *new_ops_end = new_ops + mp_size; - + const char *old_ops = vy_stmt_upsert_ops(old_stmt, &mp_size); + const char *old_stmt_mp = vy_upsert_data_range(old_stmt, &mp_size); + const char *old_stmt_mp_end = old_stmt_mp + mp_size; + const char *new_ops = vy_stmt_upsert_ops(new_stmt, &mp_size); /* - * Apply new operations to the old stmt + * UPSERT + UPSERT case: try to squash arithmetic operations. + * Firstly, unpack operations to iovec array. */ - const char *result_mp; - if (vy_stmt_type(old_stmt) == IPROTO_UPSERT) - result_mp = vy_upsert_data_range(old_stmt, &mp_size); - else - result_mp = tuple_data_range(old_stmt, &mp_size); - const char *result_mp_end = result_mp + mp_size; - struct tuple *result_stmt = NULL; + struct tuple_format *format = tuple_format(old_stmt); struct region *region = &fiber()->gc; size_t region_svp = region_used(region); - uint8_t old_type = vy_stmt_type(old_stmt); - uint64_t column_mask = COLUMN_MASK_FULL; - result_mp = xrow_upsert_execute(new_ops, new_ops_end, result_mp, - result_mp_end, format, &mp_size, - 0, suppress_error, &column_mask); - if (result_mp == NULL) { + uint32_t old_ops_cnt = mp_decode_array(&old_ops); + uint32_t new_ops_cnt = mp_decode_array(&new_ops); + size_t ops_size; + struct iovec *operations = + region_alloc_array(region, typeof(operations[0]), + old_ops_cnt + new_ops_cnt, &ops_size); + if (operations == NULL) { region_truncate(region, region_svp); + diag_set(OutOfMemory, ops_size, "region_alloc_array", + "operations"); return NULL; } - result_mp_end = result_mp + mp_size; - if (old_type != IPROTO_UPSERT) { - assert(old_type == IPROTO_INSERT || - old_type == IPROTO_REPLACE); - /* - * UPDATE case: return the updated old stmt. - */ - result_stmt = vy_stmt_new_replace(format, result_mp, - result_mp_end); + upsert_ops_to_iovec(old_ops, old_ops_cnt, operations); + upsert_ops_to_iovec(new_ops, new_ops_cnt, &operations[old_ops_cnt]); + struct iovec *res_ops = NULL; + uint32_t res_ops_cnt = 0; + if (vy_upsert_try_to_squash(format, &operations[old_ops_cnt], + new_ops_cnt, operations, old_ops_cnt, + &res_ops, &res_ops_cnt) != 0) { + /* OOM */ region_truncate(region, region_svp); - if (result_stmt == NULL) - return NULL; /* OOM */ - vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); - goto check_key; + return NULL; } - + assert(res_ops_cnt > 0); + assert(res_ops != NULL); /* - * Unpack UPSERT operation from the old stmt + * Adding update operations. We keep order of update operations in + * the array the same. It is vital since first set of operations + * must be skipped in case upsert folds into insert. For instance: + * old_ops = {{{op1}, {op2}}, {{op3}}} + * new_ops = {{{op4}, {op5}}} + * res_ops = {{{op1}, {op2}}, {{op3}}, {{op4}, {op5}}} + * If upsert corresponding to old_ops becomes insert, then + * {{op1}, {op2}} update operations are not applied. */ - assert(old_stmt != NULL); - const char *old_ops; - old_ops = vy_stmt_upsert_ops(old_stmt, &mp_size); - const char *old_ops_end = old_ops + mp_size; - assert(old_ops_end > old_ops); - - /* - * UPSERT + UPSERT case: combine operations - */ - assert(old_ops_end - old_ops > 0); - if (vy_upsert_try_to_squash(format, result_mp, result_mp_end, - old_ops, old_ops_end, new_ops, new_ops_end, - &result_stmt) != 0) { - region_truncate(region, region_svp); - return NULL; - } - if (result_stmt != NULL) { - region_truncate(region, region_svp); - vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); - goto check_key; - } - - /* Failed to squash, simply add one upsert to another */ - int old_ops_cnt, new_ops_cnt; - struct iovec operations[3]; - - old_ops_cnt = mp_decode_array(&old_ops); - operations[1].iov_base = (void *)old_ops; - operations[1].iov_len = old_ops_end - old_ops; - - new_ops_cnt = mp_decode_array(&new_ops); - operations[2].iov_base = (void *)new_ops; - operations[2].iov_len = new_ops_end - new_ops; - - char ops_buf[16]; - char *header = mp_encode_array(ops_buf, old_ops_cnt + new_ops_cnt); - operations[0].iov_base = (void *)ops_buf; - operations[0].iov_len = header - ops_buf; - - result_stmt = vy_stmt_new_upsert(format, result_mp, result_mp_end, - operations, 3); + result_stmt = vy_stmt_new_upsert(format, old_stmt_mp, old_stmt_mp_end, + res_ops, res_ops_cnt, false); region_truncate(region, region_svp); if (result_stmt == NULL) return NULL; vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); - -check_key: - /* - * Check that key hasn't been changed after applying operations. - */ - if (!key_update_can_be_skipped(cmp_def->column_mask, column_mask) && - vy_stmt_compare(old_stmt, HINT_NONE, result_stmt, - HINT_NONE, cmp_def) != 0) { - /* - * Key has been changed: ignore this UPSERT and - * @retval the old stmt. - */ - tuple_unref(result_stmt); - result_stmt = vy_stmt_dup(old_stmt); - } return result_stmt; } diff --git a/src/box/xrow_update.c b/src/box/xrow_update.c index 833975f03..62893c735 100644 --- a/src/box/xrow_update.c +++ b/src/box/xrow_update.c @@ -416,6 +416,60 @@ xrow_upsert_execute(const char *expr,const char *expr_end, return xrow_update_finish(&update, format, p_tuple_len); } +/** Return true if value stored in @a arith_op satisfy format requirements. */ +static bool +update_arith_op_does_satisfy_format(struct xrow_update_op *arith_op, + struct tuple_format *format) +{ + /* + * Upsert's tuple may contain more tuples than specified in + * space's format. In this case they are assumed to be unsigned. + */ + enum field_type fld_type = FIELD_TYPE_UNSIGNED; + if (arith_op->field_no < 0 || + (uint32_t) arith_op->field_no < tuple_format_field_count(format)) { + struct tuple_field *fld = + tuple_format_field(format, arith_op->field_no); + assert(fld != NULL); + fld_type = fld->type; + } + struct xrow_update_arg_arith arith = arith_op->arg.arith; + if (fld_type == FIELD_TYPE_UNSIGNED) { + if (arith.type == XUPDATE_TYPE_INT) { + if (!int96_is_uint64(&arith.int96)) + return false; + return true; + } + /* Can't store decimals/floating points in unsigned field. */ + return false; + } + if (fld_type == FIELD_TYPE_INTEGER) { + if (arith.type == XUPDATE_TYPE_INT) + return true; + return false; + } + if (fld_type == FIELD_TYPE_DOUBLE) { + if (arith.type == XUPDATE_TYPE_FLOAT || + arith.type == XUPDATE_TYPE_DOUBLE) + return true; + return false; + } + if (fld_type == FIELD_TYPE_NUMBER || fld_type == FIELD_TYPE_SCALAR) { + if (arith.type != XUPDATE_TYPE_DECIMAL) + return true; + return false; + } + if (fld_type == FIELD_TYPE_DECIMAL) { + if (arith.type == XUPDATE_TYPE_DECIMAL) + return true; + return false; + } + if (fld_type == FIELD_TYPE_ANY) + return true; + /* All other field types are not compatible with numerics. */ + return false; +} + const char * xrow_upsert_squash(const char *expr1, const char *expr1_end, const char *expr2, const char *expr2_end, @@ -535,6 +589,13 @@ xrow_upsert_squash(const char *expr1, const char *expr1_end, if (xrow_update_arith_make(op[1], arith, &res.arg.arith) != 0) return NULL; + res.field_no = op[0]->field_no; + /* + * Do not squash operations if they don't satisfy + * format. + */ + if (!update_arith_op_does_satisfy_format(&res, format)) + return NULL; res_ops = mp_encode_array(res_ops, 3); res_ops = mp_encode_str(res_ops, (const char *)&op[0]->opcode, 1); diff --git a/test/unit/vy_iterators_helper.c b/test/unit/vy_iterators_helper.c index 0d20f19ef..15470920b 100644 --- a/test/unit/vy_iterators_helper.c +++ b/test/unit/vy_iterators_helper.c @@ -112,7 +112,7 @@ vy_new_simple_stmt(struct tuple_format *format, struct key_def *key_def, ops = mp_encode_int(ops, templ->upsert_value); operations[0].iov_base = tmp; operations[0].iov_len = ops - tmp; - ret = vy_stmt_new_upsert(format, buf, pos, operations, 1); + ret = vy_stmt_new_upsert(format, buf, pos, operations, 1, true); fail_if(ret == NULL); break; } diff --git a/test/vinyl/upsert.result b/test/vinyl/upsert.result index 3a7f6629d..343084e81 100644 --- a/test/vinyl/upsert.result +++ b/test/vinyl/upsert.result @@ -899,3 +899,397 @@ s:select() s:drop() --- ... +-- gh-5107: don't squash upsert operations into one array. +-- +-- gh-5087: test upsert execution/squash referring to fields in reversed +-- order (via negative indexing). +-- +s = box.schema.create_space('test', {engine = 'vinyl'}) +--- +... +pk = s:create_index('pk') +--- +... +s:insert({1, 1, 1}) +--- +- [1, 1, 1] +... +box.snapshot() +--- +- ok +... +s:upsert({1}, {{'=', 3, 100}}) +--- +... +s:upsert({1}, {{'=', -1, 200}}) +--- +... +box.snapshot() +--- +- ok +... +s:select() -- {1, 1, 200} +--- +- - [1, 1, 200] +... +s:delete({1}) +--- +... +s:insert({1, 1, 1}) +--- +- [1, 1, 1] +... +box.snapshot() +--- +- ok +... +s:upsert({1}, {{'=', -3, 100}}) +--- +... +s:upsert({1}, {{'=', -1, 200}}) +--- +... +box.snapshot() +--- +- ok +... +-- gh-5105: Two upserts are NOT squashed into one, so only one (first one) +-- is skipped, meanwhile second one is applied. +-- +s:select() -- {1, 1, 1} +--- +- - [1, 1, 200] +... +s:delete({1}) +--- +... +box.snapshot() +--- +- ok +... +s:upsert({1, 1}, {{'=', -2, 300}}) -- {1, 1} +--- +... +s:upsert({1}, {{'+', -1, 100}}) -- {1, 101} +--- +... +s:upsert({1}, {{'-', 2, 100}}) -- {1, 1} +--- +... +s:upsert({1}, {{'+', -1, 200}}) -- {1, 201} +--- +... +s:upsert({1}, {{'-', 2, 200}}) -- {1, 1} +--- +... +box.snapshot() +--- +- ok +... +s:select() -- {1, 1} +--- +- - [1, 1] +... +s:delete({1}) +--- +... +box.snapshot() +--- +- ok +... +s:upsert({1, 1, 1}, {{'!', -1, 300}}) -- {1, 1, 1} +--- +... +s:upsert({1}, {{'+', -2, 100}}) -- {1, 101, 1} +--- +... +s:upsert({1}, {{'=', -1, 100}}) -- {1, 101, 100} +--- +... +s:upsert({1}, {{'+', -1, 200}}) -- {1, 101, 300} +--- +... +s:upsert({1}, {{'-', -2, 100}}) -- {1, 1, 300} +--- +... +box.snapshot() +--- +- ok +... +s:select() +--- +- - [1, 1, 300] +... +s:drop() +--- +... +-- gh-1622: upsert operations which break space format are not applied. +-- +s = box.schema.space.create('test', { engine = 'vinyl', field_count = 2 }) +--- +... +pk = s:create_index('pk') +--- +... +s:replace{1, 1} +--- +- [1, 1] +... +-- Error is logged, upsert is not applied. +-- +s:upsert({1, 1}, {{'=', 3, 5}}) +--- +... +-- During read the incorrect upsert is ignored. +-- +s:select{} +--- +- - [1, 1] +... +-- Try to set incorrect field_count in a transaction. +-- +box.begin() +--- +... +s:replace{2, 2} +--- +- [2, 2] +... +s:upsert({2, 2}, {{'=', 3, 2}}) +--- +... +s:select{} +--- +- - [1, 1] + - [2, 2] +... +box.commit() +--- +... +s:select{} +--- +- - [1, 1] + - [2, 2] +... +-- Read incorrect upsert from a run: it should be ignored. +-- +box.snapshot() +--- +- ok +... +s:select{} +--- +- - [1, 1] + - [2, 2] +... +s:upsert({2, 2}, {{'=', 3, 20}}) +--- +... +box.snapshot() +--- +- ok +... +s:select{} +--- +- - [1, 1] + - [2, 2] +... +-- Execute replace/delete after invalid upsert. +-- +box.snapshot() +--- +- ok +... +s:upsert({2, 2}, {{'=', 3, 30}}) +--- +... +s:replace{2, 3} +--- +- [2, 3] +... +s:select{} +--- +- - [1, 1] + - [2, 3] +... +s:upsert({1, 1}, {{'=', 3, 30}}) +--- +... +s:delete{1} +--- +... +s:select{} +--- +- - [2, 3] +... +-- Invalid upsert in a sequence of upserts is skipped meanwhile +-- the rest are applied. +-- +box.snapshot() +--- +- ok +... +s:upsert({2, 2}, {{'+', 2, 5}}) +--- +... +s:upsert({2, 2}, {{'=', 3, 40}}) +--- +... +s:upsert({2, 2}, {{'+', 2, 5}}) +--- +... +s:select{} +--- +- - [2, 13] +... +box.snapshot() +--- +- ok +... +s:select{} +--- +- - [2, 13] +... +s:drop() +--- +... +-- Test different scenarious during which update operations squash can't +-- take place due to format violations. +-- +decimal = require('decimal') +--- +... +s = box.schema.space.create('test', { engine = 'vinyl', field_count = 5 }) +--- +... +s:format({{name='id', type='unsigned'}, {name='u', type='unsigned'},\ + {name='s', type='scalar'}, {name='f', type='double'},\ + {name='d', type='decimal'}}) +--- +... +pk = s:create_index('pk') +--- +... +s:replace{1, 1, 1, 1.1, decimal.new(1.1) } +--- +- [1, 1, 1, 1.1, 1.1] +... +s:replace{2, 1, 1, 1.1, decimal.new(1.1)} +--- +- [2, 1, 1, 1.1, 1.1] +... +box.snapshot() +--- +- ok +... +-- Can't assign integer to float field. First operation is still applied. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 4, 4}}) +--- +... +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'=', 4, 4}}) +--- +... +-- Can't add floating point to integer (result is floating point). +-- +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 5}}) +--- +... +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 5.5}}) +--- +... +box.snapshot() +--- +- ok +... +s:select() +--- +- - [1, 1, 1, 5.1, 1.1] + - [2, 6, 1, 1.1, 1.1] +... +-- Integer overflow check. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 3, 9223372036854775808}}) +--- +... +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 3, 9223372036854775808}}) +--- +... +-- Negative result of subtraction stored in unsigned field. +-- +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 2}}) +--- +... +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 2, 10}}) +--- +... +box.snapshot() +--- +- ok +... +s:select() +--- +- - [1, 1, 9223372036854775809, 5.1, 1.1] + - [2, 8, 1, 1.1, 1.1] +... +-- Decimals do not fit into numerics and vice versa. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 5, 2}}) +--- +... +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 5, 1}}) +--- +... +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, decimal.new(2.1)}}) +--- +... +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 2, decimal.new(1.2)}}) +--- +... +box.snapshot() +--- +- ok +... +s:select() +--- +- - [1, 1, 9223372036854775809, 5.1, 2.1] + - [2, 8, 1, 1.1, 1.1] +... +s:drop() +--- +... +-- Make sure upserts satisfy associativity rule. +-- +s = box.schema.space.create('test', {engine='vinyl'}) +--- +... +i = s:create_index('pk', {parts={2, 'uint'}}) +--- +... +s:replace{1, 2, 3, 'default'} +--- +- [1, 2, 3, 'default'] +... +box.snapshot() +--- +- ok +... +s:upsert({2, 2, 2}, {{'=', 4, 'upserted'}}) +--- +... +-- Upsert will fail and thus ignored. +-- +s:upsert({2, 2, 2}, {{'#', 1, 1}, {'!', 3, 1}}) +--- +... +box.snapshot() +--- +- ok +... +s:select{} +--- +- - [1, 2, 3, 'upserted'] +... +s:drop() +--- +... diff --git a/test/vinyl/upsert.test.lua b/test/vinyl/upsert.test.lua index 1d77474da..5c5391f44 100644 --- a/test/vinyl/upsert.test.lua +++ b/test/vinyl/upsert.test.lua @@ -372,3 +372,168 @@ box.snapshot() s:select() s:drop() + +-- gh-5107: don't squash upsert operations into one array. +-- +-- gh-5087: test upsert execution/squash referring to fields in reversed +-- order (via negative indexing). +-- +s = box.schema.create_space('test', {engine = 'vinyl'}) +pk = s:create_index('pk') +s:insert({1, 1, 1}) +box.snapshot() + +s:upsert({1}, {{'=', 3, 100}}) +s:upsert({1}, {{'=', -1, 200}}) +box.snapshot() +s:select() -- {1, 1, 200} + +s:delete({1}) +s:insert({1, 1, 1}) +box.snapshot() + +s:upsert({1}, {{'=', -3, 100}}) +s:upsert({1}, {{'=', -1, 200}}) +box.snapshot() +-- gh-5105: Two upserts are NOT squashed into one, so only one (first one) +-- is skipped, meanwhile second one is applied. +-- +s:select() -- {1, 1, 1} + +s:delete({1}) +box.snapshot() + +s:upsert({1, 1}, {{'=', -2, 300}}) -- {1, 1} +s:upsert({1}, {{'+', -1, 100}}) -- {1, 101} +s:upsert({1}, {{'-', 2, 100}}) -- {1, 1} +s:upsert({1}, {{'+', -1, 200}}) -- {1, 201} +s:upsert({1}, {{'-', 2, 200}}) -- {1, 1} +box.snapshot() +s:select() -- {1, 1} + +s:delete({1}) +box.snapshot() + +s:upsert({1, 1, 1}, {{'!', -1, 300}}) -- {1, 1, 1} +s:upsert({1}, {{'+', -2, 100}}) -- {1, 101, 1} +s:upsert({1}, {{'=', -1, 100}}) -- {1, 101, 100} +s:upsert({1}, {{'+', -1, 200}}) -- {1, 101, 300} +s:upsert({1}, {{'-', -2, 100}}) -- {1, 1, 300} +box.snapshot() +s:select() + +s:drop() + +-- gh-1622: upsert operations which break space format are not applied. +-- +s = box.schema.space.create('test', { engine = 'vinyl', field_count = 2 }) +pk = s:create_index('pk') +s:replace{1, 1} +-- Error is logged, upsert is not applied. +-- +s:upsert({1, 1}, {{'=', 3, 5}}) +-- During read the incorrect upsert is ignored. +-- +s:select{} + +-- Try to set incorrect field_count in a transaction. +-- +box.begin() +s:replace{2, 2} +s:upsert({2, 2}, {{'=', 3, 2}}) +s:select{} +box.commit() +s:select{} + +-- Read incorrect upsert from a run: it should be ignored. +-- +box.snapshot() +s:select{} +s:upsert({2, 2}, {{'=', 3, 20}}) +box.snapshot() +s:select{} + +-- Execute replace/delete after invalid upsert. +-- +box.snapshot() +s:upsert({2, 2}, {{'=', 3, 30}}) +s:replace{2, 3} +s:select{} + +s:upsert({1, 1}, {{'=', 3, 30}}) +s:delete{1} +s:select{} + +-- Invalid upsert in a sequence of upserts is skipped meanwhile +-- the rest are applied. +-- +box.snapshot() +s:upsert({2, 2}, {{'+', 2, 5}}) +s:upsert({2, 2}, {{'=', 3, 40}}) +s:upsert({2, 2}, {{'+', 2, 5}}) +s:select{} +box.snapshot() +s:select{} + +s:drop() + +-- Test different scenarious during which update operations squash can't +-- take place due to format violations. +-- +decimal = require('decimal') + +s = box.schema.space.create('test', { engine = 'vinyl', field_count = 5 }) +s:format({{name='id', type='unsigned'}, {name='u', type='unsigned'},\ + {name='s', type='scalar'}, {name='f', type='double'},\ + {name='d', type='decimal'}}) +pk = s:create_index('pk') +s:replace{1, 1, 1, 1.1, decimal.new(1.1) } +s:replace{2, 1, 1, 1.1, decimal.new(1.1)} +box.snapshot() +-- Can't assign integer to float field. First operation is still applied. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 4, 4}}) +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'=', 4, 4}}) +-- Can't add floating point to integer (result is floating point). +-- +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 5}}) +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 5.5}}) +box.snapshot() +s:select() +-- Integer overflow check. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 3, 9223372036854775808}}) +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 3, 9223372036854775808}}) +-- Negative result of subtraction stored in unsigned field. +-- +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, 2}}) +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 2, 10}}) +box.snapshot() +s:select() +-- Decimals do not fit into numerics and vice versa. +-- +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 5, 2}}) +s:upsert({1, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 5, 1}}) +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'+', 2, decimal.new(2.1)}}) +s:upsert({2, 1, 1, 2.5, decimal.new(1.1)}, {{'-', 2, decimal.new(1.2)}}) +box.snapshot() +s:select() + +s:drop() + +-- Make sure upserts satisfy associativity rule. +-- +s = box.schema.space.create('test', {engine='vinyl'}) +i = s:create_index('pk', {parts={2, 'uint'}}) +s:replace{1, 2, 3, 'default'} +box.snapshot() + +s:upsert({2, 2, 2}, {{'=', 4, 'upserted'}}) +-- Upsert will fail and thus ignored. +-- +s:upsert({2, 2, 2}, {{'#', 1, 1}, {'!', 3, 1}}) +box.snapshot() + +s:select{} + +s:drop() -- 2.17.1 From korablev at tarantool.org Sat Aug 8 17:23:01 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Sat, 8 Aug 2020 14:23:01 +0000 Subject: [Tarantool-patches] [PATCH 0/2] vinyl: rework upsert internals In-Reply-To: <2566473a-c279-5b2a-0864-aee5c0e930db@tarantool.org> References: <2566473a-c279-5b2a-0864-aee5c0e930db@tarantool.org> Message-ID: <20200808142301.GA28157@tarantool.org> On 31 Jul 01:32, Vladislav Shpilevoy wrote: > Hi! Thanks for the patchset! > > On 29.07.2020 03:15, Nikita Pettik wrote: > > Issues: > > https://github.com/tarantool/tarantool/issues/1622 > > https://github.com/tarantool/tarantool/issues/5105 > > https://github.com/tarantool/tarantool/issues/5092 > > https://github.com/tarantool/tarantool/issues/5107 > > Branch: > > https://github.com/tarantool/tarantool/tree/np/gh-5107-dont-squash-ops > > Need a changelog here. Added in V2: @ChangeLog: - Rework upsert operation in vinyl so that now (gh-5107): - if upsert can't be applied it is skipped and corresponding error is logged (gh-1622); - upserts now follow associative property: result of several upserts doesn't depend on the order of their application (gh-5105); - upserts referring to -1 fieldno are handled correctly now (gh-5087). From korablev at tarantool.org Sat Aug 8 17:51:15 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Sat, 8 Aug 2020 14:51:15 +0000 Subject: [Tarantool-patches] [PATCH 2/2] vinyl: rework upsert operation In-Reply-To: <444ff8e3-d6c2-e6a5-8b54-18a521208ba5@tarantool.org> References: <14aec86329c7820470e40e47a3d1b5655b531732.1595985135.git.korablev@tarantool.org> <444ff8e3-d6c2-e6a5-8b54-18a521208ba5@tarantool.org> Message-ID: <20200808145115.GB28157@tarantool.org> On 02 Aug 16:44, Vladislav Shpilevoy wrote: > Thanks for the patch! > > ASAN tests on the branch are failing: > https://gitlab.com/tarantool/tarantool/-/jobs/661977877 Hi. I've sent second iteration of patch containing fixes to all your comments below. ASAN now is OK as well: https://gitlab.com/tarantool/tarantool/-/jobs/677105339 > > +static bool > > +vy_apply_result_does_cross_pk(struct tuple *old_stmt, const char *result, > > + const char *result_end, struct key_def *cmp_def, > > + uint64_t col_mask) > > +{ > > + if (!key_update_can_be_skipped(cmp_def->column_mask, col_mask)) { > > + struct tuple *tuple = > > + vy_stmt_new_replace(tuple_format(old_stmt), result, > > + result_end); > > + int cmp_res = vy_stmt_compare(old_stmt, HINT_NONE, tuple, > > + HINT_NONE, cmp_def); > > 1. Bad indentation. Fixed. > > + tuple_unref(tuple); > > + return cmp_res != 0; > > + } > > + return false; > > +} > > + > > +/** > > + * Apply update operations stored in @new_stmt (which is assumed to > > 2. Please, look at the doxygen syntax on the official page or here: > https://github.com/tarantool/tarantool/wiki/Code-review-procedure > This is a single very simple rule I keep repeating I don't already > remember how many times - use @a , not @. > I don't understand why does everyone keep violating it. I'm really sorry for that (haven't been writing comments in doxy style for a while). Fixed all comments. > 2. Parameter 'new_stmt' does not exist. As well as 'old_stmt'. What > did you mean? > > > + * be upsert statement) on tuple @old_stmt. If @old_stmt is void > > + * statement (i.e. it is NULL or delete statement) then operations > > + * are applied on tuple @new_stmt. All operations which can't be > > + * applied are skipped; errors may be logged depending on @supress_error > > 3. supress_error -> supress_error. Fixed. > 4. What do you mean as 'all operations'? Operation groups from various > upserts? Or individual operations? > > > + * flag. > > + * > > + * @upsert Upsert statement to be applied on @stmt. > > 5. If you want to use doxygen, use @param . Fixed. > > + * @stmt Statement to be used as base for upsert operations. > > + * @cmp_def Key definition required to provide check of primary key > > + * modification. > > + * @retrun Tuple containing result of upsert application; > > + * NULL in case OOM. > > 6. retrun -> return. Fixed. > 7. I guess you are among the ones who voted for 80 symbol comments - I > suggest you to use it. Since this is our new code style now. OK! > > + */ > > +static struct tuple * > > +vy_apply_upsert_on_terminal_stmt(struct tuple *upsert, struct tuple *stmt, > > + struct key_def *cmp_def, bool suppress_error) > > +{ > > + assert(vy_stmt_type(upsert) == IPROTO_UPSERT); > > + assert(stmt == NULL || vy_stmt_type(stmt) != IPROTO_UPSERT); > > + > > + uint32_t mp_size; > > + const char *new_ops = vy_stmt_upsert_ops(upsert, &mp_size); > > + /* Msgpack containing result of upserts application. */ > > + const char *result_mp; > > + if (vy_stmt_is_void(stmt)) > > 8. This {is_void} helper is used 2 times inside one funtion on the same > value. Seems like you could simply inline it, remeber result into a variable > {bool is_void;} and use it instead. Ok, refactored and dropped commit containing introduction of vy_stmt_is_void(). > > + for (uint32_t i = 0; i < ups_cnt; ++i) { > > + assert(mp_typeof(*ups_ops) == MP_ARRAY); > > + const char *ups_ops_end = ups_ops; > > + mp_next(&ups_ops_end); > > + const char *exec_res = result_mp; > > + exec_res = xrow_upsert_execute(ups_ops, ups_ops_end, result_mp, > > + result_mp_end, format, &mp_size, > > + 0, suppress_error, &column_mask); > > + if (exec_res == NULL) { > > + if (! suppress_error) { > > 9. According to one another recent code style change, unary operators > should not have a whitespace after them. Fixed. > > + assert(diag_last_error(diag_get()) != NULL); > > 10. Use {diag_is_empty}. Or better - save {diag_last_error(diag_get())} into > {struct error *e} before the assertion, and use {assert(e != NULL);}. OK, fixed. > > + struct error *e = diag_last_error(diag_get()); > > + /* Bail out immediately in case of OOM. */ > > + if (e->type != &type_ClientError) { > > + region_truncate(region, region_svp); > > + return NULL; > > + } > > + diag_log(); > > + } > > + ups_ops = ups_ops_end; > > + continue; > > + } > > + /* > > + * If it turns out that resulting tuple modifies primary > > + * key, than simply ignore this upsert. > > 11. than -> then. Fixed. > > +static bool > > +tuple_format_is_suitable_for_squash(struct tuple_format *format) > > +{ > > + struct tuple_field *field; > > + json_tree_foreach_entry_preorder(field, &format->fields.root, > > + struct tuple_field, token) { > > + if (field->type == FIELD_TYPE_UNSIGNED) > > + return false; > > 12. Bad indentation. > Also this squash rule is not going to work because integer type also can > overflow, both below INT64_MIN and above UINT64_MAX. Decimal types > can overflow. Besides, decimal can fail when a non-decimal value does not > fit the decimal type during conversion. For example, a huge double value. > DBL_MAX is bigger than maximal value available in our decimal type. See > xrow_update_arith_make() for all errors. > > Since squash is mostly about squashing +/-, looks like it won't work > almost always, and becomes useless. It's true. In its previous implementation it was almost useless. I've reworked this part in V2 and integrated format check right in xrow_upsert_squash() so that we can operate on particular values of squash result (e.g. if format declares unsigned field and the result of squash is negative - operations are not squashed). See update_arith_op_does_satisfy_format() and vy_upsert_try_to_squash(). > P.S. > > In the end of the review I noticed that this prevents squashing not > only of operations with unsigned fields. It will prevent squashing if > the whole format has at least one unsigned. This makes current implementation > of squash even more useless, because forbids to use the fastest field type, > which is default when you create an index without specification of field type > btw. > > > @@ -87,122 +248,74 @@ vy_apply_upsert(struct tuple *new_stmt, struct tuple *old_stmt, > > assert(new_stmt != old_stmt); > > assert(vy_stmt_type(new_stmt) == IPROTO_UPSERT); > > > > - if (old_stmt == NULL || vy_stmt_type(old_stmt) == IPROTO_DELETE) { > > - /* > > - * INSERT case: return new stmt. > > - */ > > - return vy_stmt_replace_from_upsert(new_stmt); > > + struct tuple *result_stmt = NULL; > > + if (old_stmt == NULL || vy_stmt_type(old_stmt) != IPROTO_UPSERT) { > > + return vy_apply_upsert_on_terminal_stmt(new_stmt, old_stmt, > > + cmp_def, suppress_error); > > } > > > > + assert(vy_stmt_type(old_stmt) == IPROTO_UPSERT); > > 13. The assertion looks useless, since it is reverse of the {if} > condition above, but up to you. Skipped (imho it increases a bit code readability). > > /* > > - * Unpack UPSERT operation from the new stmt > > + * Unpack UPSERT operation from the old and new stmts. > > */ > > + assert(old_stmt != NULL); > > 14. This is strage to check old_stmt->type in the previous assertion before > you checked old_stmt != NULL. Agree, swapped these asserts. > > - if (result_mp == NULL) { > > - region_truncate(region, region_svp); > > - return NULL; > > + if (tuple_format_is_suitable_for_squash(format)) { > > + const char *new_ops_end = new_ops + mp_size; > > + if (vy_upsert_try_to_squash(format, old_stmt_mp, old_stmt_mp_end, > > + old_ops, old_ops_end, new_ops, > > + new_ops_end, &result_stmt) != 0) { > > + /* OOM */ > > + region_truncate(region, region_svp); > > + return NULL; > > + } > > 15. vy_upsert_try_to_squash() returns a result into result_stmt. But > you ignore it. Basically, whatever it returns, you act like squash > didn't happen and it never works now. You continue to work with 2 old > operation set arrays. Also result_stmt leaks. > > What is also strange - I added {assert(false);} here and the tests > passed. I thought we had quite a lot squash tests. Seems they are all > for formats having unsigned field type. My aplogies for this broken part of patch, somehow I've missed it.. I've reworked it in patch V2 (see comment above). > (Actualy the tests failed, but not here - on my machine vinyl tests > fail in almost 100% runs somewhere with random errors, could be > luajit problems on Mac maybe.) > > > + * If upsert corresponding to old_ops becomes insert, then > > + * {{op1}, {op2}} update operations are not applied. > > */ > > - assert(old_ops_end - old_ops > 0); > > - if (vy_upsert_try_to_squash(format, result_mp, result_mp_end, > > - old_ops, old_ops_end, new_ops, new_ops_end, > > - &result_stmt) != 0) { > > + uint32_t old_ops_cnt = mp_decode_array(&old_ops); > > + uint32_t new_ops_cnt = mp_decode_array(&new_ops); > > + size_t ops_size = sizeof(struct iovec) * (old_ops_cnt + new_ops_cnt); > > + struct iovec *operations = region_alloc(region, ops_size); > > 16. region_alloc_array. > > 17. But you don't really need that. Nor upsert_ops_to_iovec() function. iovecs really simply code and workflow with update operations. I use them more intensely in new patch version. > You could keep the old code almost as is, because for vy_stmt_new_with_ops() > to work correctly, it is not necessary to have each operation set in a > separate iovec. Anyway they are all copied as is without unpacking. You > could have 1 iovec for the root MP_ARRAY, 1 iovec for the old operation sets, > 1 iovec for the new operation sets. > > Having first iovec with root MP_ARRAY would allow to delete is_ops_encoded. Not sure if it possible in V2... > > + if (operations == NULL) { > > region_truncate(region, region_svp); > > + diag_set(OutOfMemory, ops_size, "region_alloc", "operations"); > > return NULL; > > } > > - if (result_stmt != NULL) { > > - region_truncate(region, region_svp); > > - vy_stmt_set_lsn(result_stmt, vy_stmt_lsn(new_stmt)); > > - goto check_key; > > - } > > + upsert_ops_to_iovec(old_ops, old_ops_cnt, operations); > > + upsert_ops_to_iovec(new_ops, new_ops_cnt, &operations[old_ops_cnt]); > > > > 18. You need to put references to the relevant issues in the tests, > using > > -- > -- gh-NNNN: description. > -- Fixed (added tags). > > +- ok > > +... > > +s:select{} > > +--- > > +- - [1, 2, 3, 'upserted'] > > +... > > +s:drop() > > +--- > > +... > > 19. All tests work with unsigned fields. So squashing is not tested here. In new patch squashing requirements become more tolerant, so squashing now takes place in these tests. From alyapunov at tarantool.org Mon Aug 10 13:10:06 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Mon, 10 Aug 2020 13:10:06 +0300 Subject: [Tarantool-patches] [PATCH 0/2] JSON field multikey crash In-Reply-To: References: Message-ID: <05992d38-6a4a-4e2e-db6e-041c4184d7e7@tarantool.org> Hi! thanks for the patch. Btw, why do we call them 'JSON fields'? Is it supposed that it should work somehow with Java Script Object Notation serialization format? On 8/5/20 2:45 AM, Vladislav Shpilevoy wrote: > The patchset fixes 2 crashes related to multikey in JSON path > tuple field access code. > > Also during working on this I found > https://github.com/tarantool/tarantool/issues/5226, but couldn't > find a simple solution. > > Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5224-tuple-field-by-path-crash > Issue: https://github.com/tarantool/tarantool/issues/5224 > > @ChangeLog > * Fixed a crash when JSON tuple field access was used to get a multikey indexed field, and when a JSON contained [*] in the beginning; > > Vladislav Shpilevoy (2): > tuple: fix multikey field JSON access crash > tuple: fix access by JSON path starting from '[*]' > > src/box/tuple.c | 3 +- > src/box/tuple.h | 8 + > test/box/gh-5224-multikey-field-access.result | 164 ++++++++++++++++++ > .../gh-5224-multikey-field-access.test.lua | 72 ++++++++ > 4 files changed, 246 insertions(+), 1 deletion(-) > create mode 100644 test/box/gh-5224-multikey-field-access.result > create mode 100644 test/box/gh-5224-multikey-field-access.test.lua > From korablev at tarantool.org Mon Aug 10 19:09:14 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Mon, 10 Aug 2020 16:09:14 +0000 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: References: Message-ID: <20200810160914.GA3803@tarantool.org> On 05 Aug 01:45, Vladislav Shpilevoy wrote: > When a tuple had format with multikey indexes in it, any attempt > to get a multikey indexed field by a JSON path from Lua led to a > crash. > > That was because of incorrect interpretation of offset slot value > in tuple's field map. > > Tuple field map is an array stored before the tuple's MessagePack > data. Each element is a 4 byte offset to an indexed value to be > able to get it for O(1) time without MessagePack decoding of all > the previous fields. > > At least it was so before multikeys. Now tuple field map is not > just an array. It is rather a 2-level array, somehow similar to > ext4 FS. Some elements of the root array are positive numbers > pointing at data. Some elements point at a second 'indirect' > array, so called 'extra', size of which is individual for each > tuple. These second arrays are used by multikey indexes to store > offsets to each multikey indexed value in a tuple. > > It means, that if there is an offset slot, it can't be just used > as is. It is allowed only if the field is not multikey. Otherwise > it is neccessary to somehow get an index in the second 'indirect' > array. > > This is what was happening - a multikey field was found, its > offset slot was valid, but it was pointing at an 'indirect' array, > not at the data. JSON tuple field access tried to use it as a data > offset. > > The patch makes JSON field access degrade to fullscan when a field > is multikey, but no multikey array index is provided. > > Closes #5224 LGTM From korablev at tarantool.org Mon Aug 10 20:52:40 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Mon, 10 Aug 2020 17:52:40 +0000 Subject: [Tarantool-patches] [PATCH 2/2] tuple: fix access by JSON path starting from '[*]' In-Reply-To: <21871bd97383aa16c15d8bdad20775f2a85548ae.1596584571.git.v.shpilevoy@tarantool.org> References: <21871bd97383aa16c15d8bdad20775f2a85548ae.1596584571.git.v.shpilevoy@tarantool.org> Message-ID: <20200810175240.GB3803@tarantool.org> On 05 Aug 01:45, Vladislav Shpilevoy wrote: > Tuple JSON field access crashed when '[*]' was used as a first > part of the JSON path. The patch makes it treated like 'field not > found'. > > Follow-up #5224 LGTM From v.shpilevoy at tarantool.org Tue Aug 11 00:05:09 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 10 Aug 2020 23:05:09 +0200 Subject: [Tarantool-patches] [PATCH 1/1] router: update known bucket count when rs removed In-Reply-To: References: Message-ID: <80628f8b-57f5-2483-2a71-c9e66b54843a@tarantool.org> Yaroslav approved the patch in a private discussion. Pushed to master with 2 LGTM. From v.shpilevoy at tarantool.org Tue Aug 11 01:18:05 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 11 Aug 2020 00:18:05 +0200 Subject: [Tarantool-patches] [PATCH 1/1] box: snapshot should not include rolled back data Message-ID: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> Box.snapshot() could include rolled back data in case synchronous transaction ROLLBACK arrived during WAL rotation in preparation of a checkpoint. More specifically, snapshot consists of fixating the engines' content (creation of a read-view), doing WAL rotation, and writing the snapshot itself. All data changes after content fixation won't go into the snap. So if ROLLBACK arrives during WAL rotation, the fixated content will have rolled back data, not present in the newest dataset. The patch makes it fail if during WAL rotation anything was rolled back. The bug sometimes appeared in an existing test about qsync snapshots, but with a very poor reproducibility. In a new test file it is reproduced 100% without the patch. Closes #5167 --- Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5167-flaky-qsync-snapshots Issue: https://github.com/tarantool/tarantool/issues/5167 @ChangeLog * Snapshot could contain changes from a rolled back synchronous transaction (gh-5167). src/box/gc.c | 12 +- src/box/wal.c | 4 + src/lib/core/errinj.h | 1 + test/box/errinj.result | 1 + .../gh-5167-qsync-rollback-snap.result | 165 ++++++++++++++++++ .../gh-5167-qsync-rollback-snap.test.lua | 67 +++++++ test/replication/suite.ini | 2 +- 7 files changed, 250 insertions(+), 2 deletions(-) create mode 100644 test/replication/gh-5167-qsync-rollback-snap.result create mode 100644 test/replication/gh-5167-qsync-rollback-snap.test.lua diff --git a/src/box/gc.c b/src/box/gc.c index d50a64d66..76f7c6325 100644 --- a/src/box/gc.c +++ b/src/box/gc.c @@ -382,6 +382,7 @@ gc_do_checkpoint(bool is_scheduled) { int rc; struct wal_checkpoint checkpoint; + int64_t limbo_rollback_count = txn_limbo.rollback_count; assert(!gc.checkpoint_is_in_progress); gc.checkpoint_is_in_progress = true; @@ -396,7 +397,16 @@ gc_do_checkpoint(bool is_scheduled) rc = wal_begin_checkpoint(&checkpoint); if (rc != 0) goto out; - + /* + * Check if the checkpoint contains rolled back data. That makes the + * checkpoint not self-sufficient - it needs the xlog file with + * ROLLBACK. Drop it. + */ + if (txn_limbo.rollback_count != limbo_rollback_count) { + rc = -1; + diag_set(ClientError, ER_SYNC_ROLLBACK); + goto out; + } /* * Wait the confirms on all "sync" transactions before * create a snapshot. diff --git a/src/box/wal.c b/src/box/wal.c index 220e68245..d8c92aa36 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -662,6 +662,7 @@ wal_begin_checkpoint_f(struct cbus_call_msg *data) } vclock_copy(&msg->vclock, &writer->vclock); msg->wal_size = writer->checkpoint_wal_size; + ERROR_INJECT_SLEEP(ERRINJ_WAL_DELAY); return 0; } @@ -1272,6 +1273,9 @@ wal_write_async(struct journal *journal, struct journal_entry *entry) writer->last_entry = entry; batch->approx_len += entry->approx_len; writer->wal_pipe.n_input += entry->n_rows * XROW_IOVMAX; +#ifndef NDEBUG + ++errinj(ERRINJ_WAL_WRITE_COUNT, ERRINJ_INT)->iparam; +#endif cpipe_flush_input(&writer->wal_pipe); return 0; diff --git a/src/lib/core/errinj.h b/src/lib/core/errinj.h index aace8736f..814c57c2e 100644 --- a/src/lib/core/errinj.h +++ b/src/lib/core/errinj.h @@ -82,6 +82,7 @@ struct errinj { _(ERRINJ_WAL_DELAY, ERRINJ_BOOL, {.bparam = false}) \ _(ERRINJ_WAL_DELAY_COUNTDOWN, ERRINJ_INT, {.iparam = -1}) \ _(ERRINJ_WAL_FALLOCATE, ERRINJ_INT, {.iparam = 0}) \ + _(ERRINJ_WAL_WRITE_COUNT, ERRINJ_INT, {.iparam = 0}) \ _(ERRINJ_INDEX_ALLOC, ERRINJ_BOOL, {.bparam = false}) \ _(ERRINJ_TUPLE_ALLOC, ERRINJ_BOOL, {.bparam = false}) \ _(ERRINJ_TUPLE_FIELD, ERRINJ_BOOL, {.bparam = false}) \ diff --git a/test/box/errinj.result b/test/box/errinj.result index 4bea0f46f..613d22c51 100644 --- a/test/box/errinj.result +++ b/test/box/errinj.result @@ -114,6 +114,7 @@ evals - ERRINJ_WAL_ROTATE: false - ERRINJ_WAL_SYNC: false - ERRINJ_WAL_WRITE: false + - ERRINJ_WAL_WRITE_COUNT: 3 - ERRINJ_WAL_WRITE_DISK: false - ERRINJ_WAL_WRITE_EOF: false - ERRINJ_WAL_WRITE_PARTIAL: -1 diff --git a/test/replication/gh-5167-qsync-rollback-snap.result b/test/replication/gh-5167-qsync-rollback-snap.result new file mode 100644 index 000000000..06f58526c --- /dev/null +++ b/test/replication/gh-5167-qsync-rollback-snap.result @@ -0,0 +1,165 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +orig_synchro_quorum = box.cfg.replication_synchro_quorum + | --- + | ... +orig_synchro_timeout = box.cfg.replication_synchro_timeout + | --- + | ... +box.schema.user.grant('guest', 'super') + | --- + | ... + +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +-- +-- gh-5167: +-- +fiber = require('fiber') + | --- + | ... +box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} + | --- + | ... +_ = box.schema.space.create('sync', {is_sync = true, engine = engine}) + | --- + | ... +_ = box.space.sync:create_index('pk') + | --- + | ... +-- Write something to flush the current master's state to replica. +_ = box.space.sync:insert{1} + | --- + | ... +_ = box.space.sync:delete{1} + | --- + | ... + +box.cfg{replication_synchro_quorum = 3} + | --- + | ... +ok, err = nil + | --- + | ... +f = fiber.create(function() \ + ok, err = pcall(box.space.sync.insert, box.space.sync, {1}) \ +end) + | --- + | ... + +test_run:switch('replica') + | --- + | - true + | ... +fiber = require('fiber') + | --- + | ... +test_run:wait_cond(function() return box.space.sync:count() == 1 end) + | --- + | - true + | ... +-- Snapshot will stuck in WAL thread on rotation before starting wait on the +-- limbo. +box.error.injection.set("ERRINJ_WAL_DELAY", true) + | --- + | - ok + | ... +wal_write_count = box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") + | --- + | ... +ok, err = nil + | --- + | ... +f = fiber.create(function() ok, err = pcall(box.snapshot) end) + | --- + | ... + +test_run:switch('default') + | --- + | - true + | ... +box.cfg{replication_synchro_timeout = 0.0001} + | --- + | ... +test_run:wait_cond(function() return f:status() == 'dead' end) + | --- + | - true + | ... +ok, err + | --- + | - false + | - Quorum collection for a synchronous transaction is timed out + | ... +box.space.sync:select{} + | --- + | - [] + | ... + +test_run:switch('replica') + | --- + | - true + | ... +-- Rollback was received. Note, it is not legit to look for space:count() == 0. +-- Because ideally ROLLBACK should not be applied before written to WAL. That +-- means count() will be > 0 until WAL write succeeds. +test_run:wait_cond(function() \ + return box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") > wal_write_count \ +end) + | --- + | - true + | ... +-- Now WAL rotation is done. Snapshot will fail, because will see that a +-- rollback happened during that. Meaning that the rotated WAL contains +-- not confirmed data, and it can't be used as a checkpoint. +box.error.injection.set("ERRINJ_WAL_DELAY", false) + | --- + | - ok + | ... +test_run:wait_cond(function() return f:status() == 'dead' end) + | --- + | - true + | ... +ok, err + | --- + | - false + | - A rollback for a synchronous transaction is received + | ... + +test_run:switch('default') + | --- + | - true + | ... +box.space.sync:drop() + | --- + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +box.schema.user.revoke('guest', 'super') + | --- + | ... +box.cfg{ \ + replication_synchro_quorum = orig_synchro_quorum, \ + replication_synchro_timeout = orig_synchro_timeout, \ +} + | --- + | ... diff --git a/test/replication/gh-5167-qsync-rollback-snap.test.lua b/test/replication/gh-5167-qsync-rollback-snap.test.lua new file mode 100644 index 000000000..475727e61 --- /dev/null +++ b/test/replication/gh-5167-qsync-rollback-snap.test.lua @@ -0,0 +1,67 @@ +test_run = require('test_run').new() +engine = test_run:get_cfg('engine') + +orig_synchro_quorum = box.cfg.replication_synchro_quorum +orig_synchro_timeout = box.cfg.replication_synchro_timeout +box.schema.user.grant('guest', 'super') + +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') + +-- +-- gh-5167: +-- +fiber = require('fiber') +box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} +_ = box.schema.space.create('sync', {is_sync = true, engine = engine}) +_ = box.space.sync:create_index('pk') +-- Write something to flush the current master's state to replica. +_ = box.space.sync:insert{1} +_ = box.space.sync:delete{1} + +box.cfg{replication_synchro_quorum = 3} +ok, err = nil +f = fiber.create(function() \ + ok, err = pcall(box.space.sync.insert, box.space.sync, {1}) \ +end) + +test_run:switch('replica') +fiber = require('fiber') +test_run:wait_cond(function() return box.space.sync:count() == 1 end) +-- Snapshot will stuck in WAL thread on rotation before starting wait on the +-- limbo. +box.error.injection.set("ERRINJ_WAL_DELAY", true) +wal_write_count = box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") +ok, err = nil +f = fiber.create(function() ok, err = pcall(box.snapshot) end) + +test_run:switch('default') +box.cfg{replication_synchro_timeout = 0.0001} +test_run:wait_cond(function() return f:status() == 'dead' end) +ok, err +box.space.sync:select{} + +test_run:switch('replica') +-- Rollback was received. Note, it is not legit to look for space:count() == 0. +-- Because ideally ROLLBACK should not be applied before written to WAL. That +-- means count() will be > 0 until WAL write succeeds. +test_run:wait_cond(function() \ + return box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") > wal_write_count \ +end) +-- Now WAL rotation is done. Snapshot will fail, because will see that a +-- rollback happened during that. Meaning that the rotated WAL contains +-- not confirmed data, and it can't be used as a checkpoint. +box.error.injection.set("ERRINJ_WAL_DELAY", false) +test_run:wait_cond(function() return f:status() == 'dead' end) +ok, err + +test_run:switch('default') +box.space.sync:drop() +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +box.schema.user.revoke('guest', 'super') +box.cfg{ \ + replication_synchro_quorum = orig_synchro_quorum, \ + replication_synchro_timeout = orig_synchro_timeout, \ +} diff --git a/test/replication/suite.ini b/test/replication/suite.ini index 73f73eb89..247289d58 100644 --- a/test/replication/suite.ini +++ b/test/replication/suite.ini @@ -3,7 +3,7 @@ core = tarantool script = master.lua description = tarantool/box, replication disabled = consistent.test.lua -release_disabled = catch.test.lua errinj.test.lua gc.test.lua gc_no_space.test.lua before_replace.test.lua qsync_advanced.test.lua qsync_errinj.test.lua quorum.test.lua recover_missing_xlog.test.lua sync.test.lua long_row_timeout.test.lua gh-4739-vclock-assert.test.lua gh-4730-applier-rollback.test.lua gh-5140-qsync-casc-rollback.test.lua gh-5144-qsync-dup-confirm.test.lua +release_disabled = catch.test.lua errinj.test.lua gc.test.lua gc_no_space.test.lua before_replace.test.lua qsync_advanced.test.lua qsync_errinj.test.lua quorum.test.lua recover_missing_xlog.test.lua sync.test.lua long_row_timeout.test.lua gh-4739-vclock-assert.test.lua gh-4730-applier-rollback.test.lua gh-5140-qsync-casc-rollback.test.lua gh-5144-qsync-dup-confirm.test.lua gh-5167-rollback-snap.test.lua config = suite.cfg lua_libs = lua/fast_replica.lua lua/rlimit.lua use_unix_sockets = True -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Tue Aug 11 01:22:04 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 11 Aug 2020 00:22:04 +0200 Subject: [Tarantool-patches] [PATCH 0/2] JSON field multikey crash In-Reply-To: <05992d38-6a4a-4e2e-db6e-041c4184d7e7@tarantool.org> References: <05992d38-6a4a-4e2e-db6e-041c4184d7e7@tarantool.org> Message-ID: <648d4151-3035-cadb-bf87-c3b01c40593f@tarantool.org> Hi! On 10.08.2020 12:10, Aleksandr Lyapunov wrote: > Hi! thanks for the patch. > Btw, why do we call them 'JSON fields'? Is it supposed that it should work somehow > with Java Script Object Notation serialization format? Yes. This is what JSON means. In context of this patch it is meant 'JSON field access'. Access of a tuple field by a JSON path. From v.shpilevoy at tarantool.org Tue Aug 11 01:49:50 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 11 Aug 2020 00:49:50 +0200 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported In-Reply-To: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> References: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> Message-ID: <4a43127c-6e11-8f28-27ee-62e24fa65a4b@tarantool.org> Hi! Thanks for the patch! > diff --git a/test/box-tap/gh-5223-curl-exports.test.lua b/test/box-tap/gh-5223-curl-exports.test.lua > new file mode 100755 > index 000000000..300d60b07 > --- /dev/null > +++ b/test/box-tap/gh-5223-curl-exports.test.lua > @@ -0,0 +1,177 @@ > +#!/usr/bin/env tarantool > + > +local tap = require('tap') > +local ffi = require('ffi') > +ffi.cdef([[ > + void *dlsym(void *handle, const char *symbol); > + struct curl_version_info_data { 1. I don't think you need to copy-paste the entire structure. Anyway it is returned from curl_version_info() by pointer, so you can just announce the pointer type. And in the test check that ffi.C.curl_version_info(7) returned something not nil. I don't think you need to access any fields of this struct. It is going to lead to UB in case the struct will change anyhow in the sources. > + int age; /* see description below */ > + const char *version; /* human readable string */ > + unsigned int version_num; /* numeric representation */ > + const char *host; /* human readable string */ > + int features; /* bitmask, see below */ > + char *ssl_version; /* human readable string */ > + long ssl_version_num; /* not used, always zero */ > + const char *libz_version; /* human readable string */ > + const char * const *protocols; /* protocols */ > + > + /* when 'age' is CURLVERSION_SECOND or higher, the members below exist */ > + const char *ares; /* human readable string */ > + int ares_num; /* number */ > + > + /* when 'age' is CURLVERSION_THIRD or higher, the members below exist */ > + const char *libidn; /* human readable string */ > + > + /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the members > + below exist */ > + int iconv_ver_num; /* '_libiconv_version' if iconv support enabled */ > + > + const char *libssh_version; /* human readable string */ > + > + /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the members > + below exist */ > + unsigned int brotli_ver_num; /* Numeric Brotli version > + (MAJOR << 24) | (MINOR << 12) | PATCH */ > + const char *brotli_version; /* human readable string. */ > + > + /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the members > + below exist */ > + unsigned int nghttp2_ver_num; /* Numeric nghttp2 version > + (MAJOR << 16) | (MINOR << 8) | PATCH */ > + const char *nghttp2_version; /* human readable string. */ > + > + const char *quic_version; /* human readable quic (+ HTTP/3) library + > + version or NULL */ > + > + /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the members > + below exist */ > + const char *cainfo; /* the built-in default CURLOPT_CAINFO, might > + be NULL */ > + const char *capath; /* the built-in default CURLOPT_CAPATH, might > + be NULL */ > + }; > + > + struct curl_version_info_data *curl_version_info(int age); > +]]) > + > +local info = ffi.C.curl_version_info(7) 2. Why '7'? > +local test = tap.test('curl-features') > +test:plan(3) > + From roman.habibov at tarantool.org Tue Aug 11 03:33:34 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:33:34 +0300 Subject: [Tarantool-patches] [PATCH v3 0/4] Support ALTER TABLE ADD COLUMN Message-ID: <20200811003338.45084-1-roman.habibov@tarantool.org> I split the last patch into three patches for convenience. @ChangeLog - Added statement support. Column description is the same as in statement (gh-2349, gh-3075). Branch: https://github.com/tarantool/tarantool/tree/romanhabibov/gh-3075-add-column-v2 Issue: https://github.com/tarantool/tarantool/issues/3075 https://github.com/tarantool/tarantool/issues/2349 Roman Khabibov (4): sql: rename TK_COLUMN to TK_COLUMN_NAME sql: refactor create_table_def and parse schema: add box_space_field_MAX sql: support column addition extra/addopcodes.sh | 2 +- extra/mkkeywordhash.c | 7 +- src/box/errcode.h | 2 + src/box/schema_def.h | 1 + src/box/sql/build.c | 602 ++++++++++++++++++++++----------- src/box/sql/expr.c | 45 +-- src/box/sql/fk_constraint.c | 2 +- src/box/sql/parse.y | 44 ++- src/box/sql/parse_def.h | 53 ++- src/box/sql/prepare.c | 7 +- src/box/sql/resolve.c | 10 +- src/box/sql/select.c | 10 +- src/box/sql/sqlInt.h | 61 +++- src/box/sql/treeview.c | 2 +- src/box/sql/where.c | 18 +- src/box/sql/whereexpr.c | 12 +- test/box/error.result | 2 + test/sql/add-column.result | 471 ++++++++++++++++++++++++++ test/sql/add-column.test.sql | 167 +++++++++ test/sql/checks.result | 20 ++ test/sql/checks.test.lua | 9 + test/sql/foreign-keys.result | 28 ++ test/sql/foreign-keys.test.lua | 11 + 23 files changed, 1275 insertions(+), 311 deletions(-) create mode 100644 test/sql/add-column.result create mode 100644 test/sql/add-column.test.sql -- 2.21.0 (Apple Git-122) From roman.habibov at tarantool.org Tue Aug 11 03:33:35 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:33:35 +0300 Subject: [Tarantool-patches] [PATCH v3 1/4] sql: rename TK_COLUMN to TK_COLUMN_NAME In-Reply-To: <20200811003338.45084-1-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> Message-ID: <20200811003338.45084-2-roman.habibov@tarantool.org> Rename TK_COLUMN used for tokens treated as a column name to TK_COLUMN_NAME. It is needed to allow the typing of COLUMN keyword in statement. Needed for #3075 --- extra/addopcodes.sh | 2 +- extra/mkkeywordhash.c | 5 +---- src/box/sql/build.c | 2 +- src/box/sql/expr.c | 45 +++++++++++++++++++------------------ src/box/sql/fk_constraint.c | 2 +- src/box/sql/resolve.c | 10 ++++----- src/box/sql/select.c | 10 ++++----- src/box/sql/sqlInt.h | 8 +++---- src/box/sql/treeview.c | 2 +- src/box/sql/where.c | 18 ++++++++------- src/box/sql/whereexpr.c | 12 +++++----- 11 files changed, 58 insertions(+), 58 deletions(-) diff --git a/extra/addopcodes.sh b/extra/addopcodes.sh index cb6c84725..986c62683 100755 --- a/extra/addopcodes.sh +++ b/extra/addopcodes.sh @@ -39,7 +39,7 @@ extras=" \ END_OF_FILE \ UNCLOSED_STRING \ FUNCTION \ - COLUMN \ + COLUMN_NAME \ AGG_FUNCTION \ AGG_COLUMN \ UMINUS \ diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c index dd42c8f5f..486b6b30d 100644 --- a/extra/mkkeywordhash.c +++ b/extra/mkkeywordhash.c @@ -75,10 +75,7 @@ static Keyword aKeywordTable[] = { { "CAST", "TK_CAST", false }, { "CHECK", "TK_CHECK", true }, { "COLLATE", "TK_COLLATE", true }, - /* gh-3075: Reserved until ALTER ADD COLUMN is implemeneted. - * Move it back to ALTER when done. - */ - /* { "COLUMN", "TK_COLUMNKW", true }, */ + { "COLUMN_NAME", "TK_COLUMN_NAME", true }, { "COLUMN", "TK_STANDARD", true }, { "COMMIT", "TK_COMMIT", true }, { "CONFLICT", "TK_CONFLICT", false }, diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 8f6b403b9..619bbf8e3 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -2295,7 +2295,7 @@ index_fill_def(struct Parse *parse, struct index *index, goto cleanup; struct Expr *column_expr = sqlExprSkipCollate(expr); - if (column_expr->op != TK_COLUMN) { + if (column_expr->op != TK_COLUMN_NAME) { diag_set(ClientError, ER_UNSUPPORTED, "Tarantool", "functional indexes"); goto tnt_error; diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index bc2182446..d389b3daf 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -62,7 +62,7 @@ sql_expr_type(struct Expr *pExpr) assert(!ExprHasProperty(pExpr, EP_IntValue)); return pExpr->type; case TK_AGG_COLUMN: - case TK_COLUMN: + case TK_COLUMN_NAME: case TK_TRIGGER: assert(pExpr->iColumn >= 0); return pExpr->space_def->fields[pExpr->iColumn].type; @@ -262,13 +262,13 @@ sql_expr_coll(Parse *parse, Expr *p, bool *is_explicit_coll, uint32_t *coll_id, *is_explicit_coll = true; break; } - if ((op == TK_AGG_COLUMN || op == TK_COLUMN || + if ((op == TK_AGG_COLUMN || op == TK_COLUMN_NAME || op == TK_REGISTER || op == TK_TRIGGER) && p->space_def != NULL) { /* * op==TK_REGISTER && p->space_def!=0 * happens when pExpr was originally - * a TK_COLUMN but was previously + * a TK_COLUMN_NAME but was previously * evaluated and cached in a register. */ int j = p->iColumn; @@ -2061,11 +2061,11 @@ exprNodeIsConstant(Walker * pWalker, Expr * pExpr) return WRC_Abort; } case TK_ID: - case TK_COLUMN: + case TK_COLUMN_NAME: case TK_AGG_FUNCTION: case TK_AGG_COLUMN: testcase(pExpr->op == TK_ID); - testcase(pExpr->op == TK_COLUMN); + testcase(pExpr->op == TK_COLUMN_NAME); testcase(pExpr->op == TK_AGG_FUNCTION); testcase(pExpr->op == TK_AGG_COLUMN); if (pWalker->eCode == 3 && pExpr->iTable == pWalker->u.iCur) { @@ -2236,7 +2236,7 @@ sqlExprCanBeNull(const Expr * p) case TK_FLOAT: case TK_BLOB: return 0; - case TK_COLUMN: + case TK_COLUMN_NAME: assert(p->space_def != 0); return ExprHasProperty(p, EP_CanBeNull) || (p->iColumn >= 0 @@ -2267,7 +2267,7 @@ sql_expr_needs_no_type_change(const struct Expr *p, enum field_type type) return type == FIELD_TYPE_STRING; case TK_BLOB: return type == FIELD_TYPE_VARBINARY; - case TK_COLUMN: + case TK_COLUMN_NAME: /* p cannot be part of a CHECK constraint. */ assert(p->iTable >= 0); return p->iColumn < 0 && sql_type_is_numeric(type); @@ -2324,7 +2324,7 @@ isCandidateForInOpt(Expr * pX) /* All SELECT results must be columns. */ for (i = 0; i < pEList->nExpr; i++) { Expr *pRes = pEList->a[i].pExpr; - if (pRes->op != TK_COLUMN) + if (pRes->op != TK_COLUMN_NAME) return 0; assert(pRes->iTable == pSrc->a[0].iCursor); /* Not a correlated subquery */ } @@ -3707,10 +3707,10 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) } return target; } - /* Otherwise, fall thru into the TK_COLUMN case */ + /* Otherwise, fall thru into the TK_COLUMN_NAME case */ FALLTHROUGH; } - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ int iTab = pExpr->iTable; int col = pExpr->iColumn; if (iTab < 0) { @@ -4102,7 +4102,7 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) assert(nFarg == 1); assert(pFarg->a[0].pExpr != 0); exprOp = pFarg->a[0].pExpr->op; - if (exprOp == TK_COLUMN + if (exprOp == TK_COLUMN_NAME || exprOp == TK_AGG_COLUMN) { assert(SQL_FUNC_LENGTH == OPFLAG_LENGTHARG); @@ -4319,7 +4319,7 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) endLabel = sqlVdbeMakeLabel(v); if ((pX = pExpr->pLeft) != 0) { tempX = *pX; - testcase(pX->op == TK_COLUMN); + testcase(pX->op == TK_COLUMN_NAME); exprToRegister(&tempX, exprCodeVector(pParse, &tempX, ®Free1)); @@ -4344,11 +4344,11 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) pTest = aListelem[i].pExpr; } nextCase = sqlVdbeMakeLabel(v); - testcase(pTest->op == TK_COLUMN); + testcase(pTest->op == TK_COLUMN_NAME); sqlExprIfFalse(pParse, pTest, nextCase, SQL_JUMPIFNULL); testcase(aListelem[i + 1].pExpr->op == - TK_COLUMN); + TK_COLUMN_NAME); sqlExprCode(pParse, aListelem[i + 1].pExpr, target); sqlVdbeGoto(v, endLabel); @@ -5081,7 +5081,8 @@ sqlExprCompare(Expr * pA, Expr * pB, int iTab) } return 2; } - if (pA->op != TK_COLUMN && pA->op != TK_AGG_COLUMN && pA->u.zToken) { + if (pA->op != TK_COLUMN_NAME && pA->op != TK_AGG_COLUMN && + pA->u.zToken) { if (pA->op == TK_FUNCTION) { if (sqlStrICmp(pA->u.zToken, pB->u.zToken) != 0) return 2; @@ -5161,8 +5162,8 @@ sqlExprListCompare(ExprList * pA, ExprList * pB, int iTab) * pE1: x IS NULL pE2: x IS NOT NULL Result: false * pE1: x IS ?2 pE2: x IS NOT NULL Reuslt: false * - * When comparing TK_COLUMN nodes between pE1 and pE2, if pE2 has - * Expr.iTable<0 then assume a table number given by iTab. + * When comparing TK_COLUMN_NAME nodes between pE1 and pE2, if + * pE2 has Expr.iTable<0 then assume a table number given by iTab. * * When in doubt, return false. Returning true might give a performance * improvement. Returning false might cause a performance reduction, but @@ -5209,11 +5210,11 @@ exprSrcCount(Walker * pWalker, Expr * pExpr) { /* The NEVER() on the second term is because sqlFunctionUsesThisSrc() * is always called before sqlExprAnalyzeAggregates() and so the - * TK_COLUMNs have not yet been converted into TK_AGG_COLUMN. If + * TK_COLUMN_NAMEs have not yet been converted into TK_AGG_COLUMN. If * sqlFunctionUsesThisSrc() is used differently in the future, the * NEVER() will need to be removed. */ - if (pExpr->op == TK_COLUMN || NEVER(pExpr->op == TK_AGG_COLUMN)) { + if (pExpr->op == TK_COLUMN_NAME || NEVER(pExpr->op == TK_AGG_COLUMN)) { int i; struct SrcCount *p = pWalker->u.pSrcCount; SrcList *pSrc = p->pSrc; @@ -5299,9 +5300,9 @@ analyzeAggregate(Walker * pWalker, Expr * pExpr) switch (pExpr->op) { case TK_AGG_COLUMN: - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ testcase(pExpr->op == TK_AGG_COLUMN); - testcase(pExpr->op == TK_COLUMN); + testcase(pExpr->op == TK_COLUMN_NAME); /* Check to see if the column is in one of the tables in the FROM * clause of the aggregate query */ @@ -5370,7 +5371,7 @@ analyzeAggregate(Walker * pWalker, Expr * pExpr) if (pE-> op == - TK_COLUMN + TK_COLUMN_NAME && pE-> iTable diff --git a/src/box/sql/fk_constraint.c b/src/box/sql/fk_constraint.c index 482220a95..3f3625ad5 100644 --- a/src/box/sql/fk_constraint.c +++ b/src/box/sql/fk_constraint.c @@ -338,7 +338,7 @@ static struct Expr * sql_expr_new_column_by_cursor(struct sql *db, struct space_def *def, int cursor, int column) { - struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN); + struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN_NAME); if (expr == NULL) return NULL; expr->space_def = def; diff --git a/src/box/sql/resolve.c b/src/box/sql/resolve.c index 6f625dc18..5fe8ee3f6 100644 --- a/src/box/sql/resolve.c +++ b/src/box/sql/resolve.c @@ -189,7 +189,7 @@ sqlMatchSpanName(const char *zSpan, * pExpr->space_def Points to the space_def structure of X.Y * (even if X and/or Y are implied.) * pExpr->iColumn Set to the column number within the table. - * pExpr->op Set to TK_COLUMN. + * pExpr->op Set to TK_COLUMN_NAME. * pExpr->pLeft Any expression this points to is deleted * pExpr->pRight Any expression this points to is deleted. * @@ -461,7 +461,7 @@ lookupName(Parse * pParse, /* The parsing context */ pExpr->pLeft = 0; sql_expr_delete(db, pExpr->pRight, false); pExpr->pRight = 0; - pExpr->op = (isTrigger ? TK_TRIGGER : TK_COLUMN); + pExpr->op = (isTrigger ? TK_TRIGGER : TK_COLUMN_NAME); lookupname_end: if (cnt == 1) { assert(pNC != 0); @@ -485,7 +485,7 @@ struct Expr * sql_expr_new_column(struct sql *db, struct SrcList *src_list, int src_idx, int column) { - struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN); + struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN_NAME); if (expr == NULL) return NULL; struct SrcList_item *item = &src_list->a[src_idx]; @@ -518,7 +518,7 @@ exprProbability(Expr * p) /* * This routine is callback for sqlWalkExpr(). * - * Resolve symbolic names into TK_COLUMN operators for the current + * Resolve symbolic names into TK_COLUMN_NAME operators for the current * node in the expression tree. Return 0 to continue the search down * the tree or 2 to abort the tree walk. * @@ -1451,7 +1451,7 @@ resolveSelectStep(Walker * pWalker, Select * p) * * The node at the root of the subtree is modified as follows: * - * Expr.op Changed to TK_COLUMN + * Expr.op Changed to TK_COLUMN_NAME * Expr.pTab Points to the Table object for X.Y * Expr.iColumn The column index in X.Y. -1 for the rowid. * Expr.iTable The VDBE cursor number for X.Y diff --git a/src/box/sql/select.c b/src/box/sql/select.c index b0554a172..80fbf69e0 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1817,7 +1817,7 @@ generate_column_metadata(struct Parse *pParse, struct SrcList *pTabList, vdbe_metadata_set_col_nullability(v, i, -1); const char *colname = pEList->a[i].zName; const char *span = pEList->a[i].zSpan; - if (p->op == TK_COLUMN || p->op == TK_AGG_COLUMN) { + if (p->op == TK_COLUMN_NAME || p->op == TK_AGG_COLUMN) { char *zCol; int iCol = p->iColumn; for (j = 0; ALWAYS(j < pTabList->nSrc); j++) { @@ -1939,7 +1939,7 @@ sqlColumnsFromExprList(Parse * parse, ExprList * expr_list, pColExpr = pColExpr->pRight; assert(pColExpr != 0); } - if (pColExpr->op == TK_COLUMN + if (pColExpr->op == TK_COLUMN_NAME && ALWAYS(pColExpr->space_def != NULL)) { /* For columns use the column name name */ int iCol = pColExpr->iColumn; @@ -3653,7 +3653,7 @@ substExpr(Parse * pParse, /* Report errors here */ sql *db = pParse->db; if (pExpr == 0) return 0; - if (pExpr->op == TK_COLUMN && pExpr->iTable == iTable) { + if (pExpr->op == TK_COLUMN_NAME && pExpr->iTable == iTable) { if (pExpr->iColumn < 0) { pExpr->op = TK_NULL; } else { @@ -6044,7 +6044,7 @@ sqlSelect(Parse * pParse, /* The parser context */ /* Create a label to jump to when we want to abort the query */ addrEnd = sqlVdbeMakeLabel(v); - /* Convert TK_COLUMN nodes into TK_AGG_COLUMN and make entries in + /* Convert TK_COLUMN_NAME nodes into TK_AGG_COLUMN and make entries in * sAggInfo for all TK_AGG_FUNCTION nodes in expressions of the * SELECT statement. */ @@ -6416,7 +6416,7 @@ sqlSelect(Parse * pParse, /* The parser context */ flag != WHERE_ORDERBY_MIN ? 1 : 0; pMinMax->a[0].pExpr->op = - TK_COLUMN; + TK_COLUMN_NAME; } } diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index adf90d824..beb83ce95 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -1517,7 +1517,7 @@ typedef int ynVar; * valid. * * An expression of the form ID or ID.ID refers to a column in a table. - * For such expressions, Expr.op is set to TK_COLUMN and Expr.iTable is + * For such expressions, Expr.op is set to TK_COLUMN_NAME and Expr.iTable is * the integer cursor number of a VDBE cursor pointing to that table and * Expr.iColumn is the column number for the specific column. If the * expression is used as a result in an aggregate SELECT, then the @@ -1587,20 +1587,20 @@ struct Expr { #if SQL_MAX_EXPR_DEPTH>0 int nHeight; /* Height of the tree headed by this node */ #endif - int iTable; /* TK_COLUMN: cursor number of table holding column + int iTable; /* TK_COLUMN_NAME: cursor number of table holding column * TK_REGISTER: register number * TK_TRIGGER: 1 -> new, 0 -> old * EP_Unlikely: 134217728 times likelihood * TK_SELECT: 1st register of result vector */ - ynVar iColumn; /* TK_COLUMN: column index. + ynVar iColumn; /* TK_COLUMN_NAME: column index. * TK_VARIABLE: variable number (always >= 1). * TK_SELECT_COLUMN: column of the result vector */ i16 iAgg; /* Which entry in pAggInfo->aCol[] or ->aFunc[] */ i16 iRightJoinTable; /* If EP_FromJoin, the right table of the join */ u8 op2; /* TK_REGISTER: original value of Expr.op - * TK_COLUMN: the value of p5 for OP_Column + * TK_COLUMN_NAME: the value of p5 for OP_Column * TK_AGG_FUNCTION: nesting depth */ AggInfo *pAggInfo; /* Used by TK_AGG_COLUMN and TK_AGG_FUNCTION */ diff --git a/src/box/sql/treeview.c b/src/box/sql/treeview.c index a04597979..ea26fcf6d 100644 --- a/src/box/sql/treeview.c +++ b/src/box/sql/treeview.c @@ -327,7 +327,7 @@ sqlTreeViewExpr(TreeView * pView, const Expr * pExpr, u8 moreToFollow) zFlgs); break; } - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ if (pExpr->iTable < 0) { /* This only happens when coding check constraints */ sqlTreeViewLine(pView, "COLUMN(%d)%s", diff --git a/src/box/sql/where.c b/src/box/sql/where.c index e9e936856..77f863b0e 100644 --- a/src/box/sql/where.c +++ b/src/box/sql/where.c @@ -276,7 +276,7 @@ whereScanNext(WhereScan * pScan) sqlExprSkipCollate(pTerm-> pExpr-> pRight))-> - op == TK_COLUMN) { + op == TK_COLUMN_NAME) { int j; for (j = 0; j < pScan->nEquiv; j++) { if (pScan->aiCur[j] == pX->iTable @@ -319,7 +319,7 @@ whereScanNext(WhereScan * pScan) } } if ((pTerm->eOperator & WO_EQ) != 0 - && (pX = pTerm->pExpr->pRight)->op == TK_COLUMN + && (pX = pTerm->pExpr->pRight)->op == TK_COLUMN_NAME && pX->iTable == pScan->aiCur[0] && pX->iColumn == pScan->aiColumn[0]) { continue; @@ -555,7 +555,7 @@ findIndexCol(Parse * pParse, /* Parse context */ struct key_part *part_to_match = &idx_def->key_def->parts[iCol]; for (int i = 0; i < pList->nExpr; i++) { Expr *p = sqlExprSkipCollate(pList->a[i].pExpr); - if (p->op == TK_COLUMN && p->iTable == iBase && + if (p->op == TK_COLUMN_NAME && p->iTable == iBase && p->iColumn == (int) part_to_match->fieldno) { bool is_found; uint32_t id; @@ -601,7 +601,8 @@ isDistinctRedundant(Parse * pParse, /* Parsing context */ */ for (int i = 0; i < pDistinct->nExpr; i++) { Expr *p = sqlExprSkipCollate(pDistinct->a[i].pExpr); - if (p->op == TK_COLUMN && p->iTable == iBase && p->iColumn < 0) + if (p->op == TK_COLUMN_NAME && p->iTable == iBase && + p->iColumn < 0) return 1; } if (space == NULL) @@ -2245,7 +2246,7 @@ whereRangeVectorLen(Parse * pParse, /* Parsing context */ * leftmost index column. */ struct key_part *parts = idx_def->key_def->parts; - if (pLhs->op != TK_COLUMN || pLhs->iTable != iCur || + if (pLhs->op != TK_COLUMN_NAME || pLhs->iTable != iCur || pLhs->iColumn != (int)parts[i + nEq].fieldno || parts[i + nEq].sort_order != parts[nEq].sort_order) break; @@ -2677,7 +2678,7 @@ indexMightHelpWithOrderBy(WhereLoopBuilder * pBuilder, return 0; for (ii = 0; ii < pOB->nExpr; ii++) { Expr *pExpr = sqlExprSkipCollate(pOB->a[ii].pExpr); - if (pExpr->op == TK_COLUMN && pExpr->iTable == iCursor) { + if (pExpr->op == TK_COLUMN_NAME && pExpr->iTable == iCursor) { if (pExpr->iColumn < 0) return 1; for (jj = 0; jj < part_count; jj++) { @@ -3213,7 +3214,7 @@ wherePathSatisfiesOrderBy(WhereInfo * pWInfo, /* The WHERE clause */ if (MASKBIT(i) & obSat) continue; pOBExpr = sqlExprSkipCollate(pOrderBy->a[i].pExpr); - if (pOBExpr->op != TK_COLUMN) + if (pOBExpr->op != TK_COLUMN_NAME) continue; if (pOBExpr->iTable != iCur) continue; @@ -3361,7 +3362,8 @@ wherePathSatisfiesOrderBy(WhereInfo * pWInfo, /* The WHERE clause */ if ((wctrlFlags & (WHERE_GROUPBY | WHERE_DISTINCTBY)) == 0) bOnce = 0; if (iColumn >= (-1)) { - if (pOBExpr->op != TK_COLUMN) + if (pOBExpr->op != + TK_COLUMN_NAME) continue; if (pOBExpr->iTable != iCur) continue; diff --git a/src/box/sql/whereexpr.c b/src/box/sql/whereexpr.c index d9b5c78f5..c2ce4339f 100644 --- a/src/box/sql/whereexpr.c +++ b/src/box/sql/whereexpr.c @@ -279,7 +279,7 @@ like_optimization_is_valid(Parse *pParse, Expr *pExpr, Expr **ppPrefix, pList = pExpr->x.pList; pLeft = pList->a[1].pExpr; /* Value might be numeric */ - if (pLeft->op != TK_COLUMN || + if (pLeft->op != TK_COLUMN_NAME || sql_expr_type(pLeft) != FIELD_TYPE_STRING) { /* IMP: R-02065-49465 The left-hand side of the * LIKE operator must be the name of an indexed @@ -928,7 +928,7 @@ exprSelectUsage(WhereMaskSet * pMaskSet, Select * pS) * number of the table that is indexed and *piColumn to the column number * of the column that is indexed. * - * If pExpr is a TK_COLUMN column reference, then this routine always returns + * If pExpr is a TK_COLUMN_NAME column reference, then this routine always returns * true even if that particular column is not indexed, because the column * might be added to an automatic index later. */ @@ -950,7 +950,7 @@ exprMightBeIndexed(int op, /* The specific comparison operator */ pExpr = pExpr->x.pList->a[0].pExpr; } - if (pExpr->op == TK_COLUMN) { + if (pExpr->op == TK_COLUMN_NAME) { *piCur = pExpr->iTable; *piColumn = pExpr->iColumn; return 1; @@ -1272,7 +1272,7 @@ exprAnalyze(SrcList * pSrc, /* the FROM clause */ * Note that the virtual term must be tagged with TERM_VNULL. */ if (pExpr->op == TK_NOTNULL - && pExpr->pLeft->op == TK_COLUMN + && pExpr->pLeft->op == TK_COLUMN_NAME && pExpr->pLeft->iColumn >= 0) { Expr *pNewExpr; Expr *pLeft = pExpr->pLeft; @@ -1397,7 +1397,7 @@ sqlWhereExprUsage(WhereMaskSet * pMaskSet, Expr * p) Bitmask mask; if (p == 0) return 0; - if (p->op == TK_COLUMN) { + if (p->op == TK_COLUMN_NAME) { mask = sqlWhereGetMask(pMaskSet, p->iTable); return mask; } @@ -1475,7 +1475,7 @@ sqlWhereTabFuncArgs(Parse * pParse, /* Parsing context */ * unused. */ assert(k < (int)space_def->field_count); - pColRef = sql_expr_new_anon(pParse->db, TK_COLUMN); + pColRef = sql_expr_new_anon(pParse->db, TK_COLUMN_NAME); if (pColRef == NULL) { pParse->is_aborted = true; return; -- 2.21.0 (Apple Git-122) From roman.habibov at tarantool.org Tue Aug 11 03:33:36 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:33:36 +0300 Subject: [Tarantool-patches] [PATCH v3 2/4] sql: refactor create_table_def and parse In-Reply-To: <20200811003338.45084-1-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> Message-ID: <20200811003338.45084-3-roman.habibov@tarantool.org> Move ck, fk constraint lists and autoincrement info from struct create_table_def to struct Parse to make the code more reusable when implementing . Needed for #3075 --- src/box/sql/build.c | 179 +++++++++++++++++++++------------------- src/box/sql/parse_def.h | 33 -------- src/box/sql/prepare.c | 7 +- src/box/sql/sqlInt.h | 14 ++++ 4 files changed, 116 insertions(+), 117 deletions(-) diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 619bbf8e3..9013bc86f 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -590,7 +590,7 @@ sql_create_check_contraint(struct Parse *parser) } } else { assert(! is_alter); - uint32_t ck_idx = ++parser->create_table_def.check_count; + uint32_t ck_idx = ++parser->check_count; name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, ck_idx); } size_t name_len = strlen(name); @@ -652,8 +652,7 @@ sql_create_check_contraint(struct Parse *parser) sqlVdbeCountChanges(v); sqlVdbeChangeP5(v, OPFLAG_NCHANGE); } else { - rlist_add_entry(&parser->create_table_def.new_check, ck_parse, - link); + rlist_add_entry(&parser->checks, ck_parse, link); } } @@ -930,7 +929,7 @@ emitNewSysSequenceRecord(Parse *pParse, int reg_seq_id, const char *seq_name) static int emitNewSysSpaceSequenceRecord(Parse *pParse, int reg_space_id, int reg_seq_id) { - uint32_t fieldno = pParse->create_table_def.autoinc_fieldno; + uint32_t fieldno = pParse->autoinc_fieldno; Vdbe *v = sqlGetVdbe(pParse); int first_col = pParse->nMem + 1; @@ -1147,6 +1146,88 @@ resolve_link(struct Parse *parse_context, const struct space_def *def, return -1; } +/** + * Emit code to create sequences, indexes, check and foreign key + * constraints appeared in . + */ +static void +sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) +{ + assert(reg_space_id != 0); + struct space *space = parse->create_table_def.new_space; + assert(space != NULL); + uint32_t i = 0; + for (; i < space->index_count; ++i) { + struct index *idx = space->index[i]; + vdbe_emit_create_index(parse, space->def, idx->def, + reg_space_id, idx->def->iid); + } + + /* + * Check to see if we need to create an _sequence table + * for keeping track of autoincrement keys. + */ + if (parse->has_autoinc) { + /* Do an insertion into _sequence. */ + int reg_seq_id = ++parse->nMem; + struct Vdbe *v = sqlGetVdbe(parse); + assert(v != NULL); + sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); + int reg_seq_rec = emitNewSysSequenceRecord(parse, reg_seq_id, + space->def->name); + sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_rec); + /* Do an insertion into _space_sequence. */ + int reg_space_seq_record = + emitNewSysSpaceSequenceRecord(parse, reg_space_id, + reg_seq_id); + sqlVdbeAddOp2(v, OP_SInsert, BOX_SPACE_SEQUENCE_ID, + reg_space_seq_record); + } + + /* Code creation of FK constraints, if any. */ + struct fk_constraint_parse *fk_parse; + rlist_foreach_entry(fk_parse, &parse->fkeys, link) { + struct fk_constraint_def *fk_def = fk_parse->fk_def; + if (fk_parse->selfref_cols != NULL) { + struct ExprList *cols = fk_parse->selfref_cols; + for (uint32_t i = 0; i < fk_def->field_count; ++i) { + if (resolve_link(parse, space->def, + cols->a[i].zName, + &fk_def->links[i].parent_field, + fk_def->name) != 0) + return; + } + fk_def->parent_id = reg_space_id; + } else if (fk_parse->is_self_referenced) { + struct key_def *pk_key_def = + sql_space_primary_key(space)->def->key_def; + if (pk_key_def->part_count != fk_def->field_count) { + diag_set(ClientError, ER_CREATE_FK_CONSTRAINT, + fk_def->name, "number of columns in "\ + "foreign key does not match the "\ + "number of columns in the primary "\ + "index of referenced table"); + parse->is_aborted = true; + return; + } + for (uint32_t i = 0; i < fk_def->field_count; ++i) { + fk_def->links[i].parent_field = + pk_key_def->parts[i].fieldno; + } + fk_def->parent_id = reg_space_id; + } + fk_def->child_id = reg_space_id; + vdbe_emit_fk_constraint_create(parse, fk_def, space->def->name); + } + + /* Code creation of CK constraints, if any. */ + struct ck_constraint_parse *ck_parse; + rlist_foreach_entry(ck_parse, &parse->checks, link) { + vdbe_emit_ck_constraint_create(parse, ck_parse->ck_def, + reg_space_id, space->def->name); + } +} + /* * This routine is called to report the final ")" that terminates * a CREATE TABLE statement. @@ -1213,73 +1294,7 @@ sqlEndTable(struct Parse *pParse) int reg_space_id = getNewSpaceId(pParse); vdbe_emit_space_create(pParse, reg_space_id, name_reg, new_space); - for (uint32_t i = 0; i < new_space->index_count; ++i) { - struct index *idx = new_space->index[i]; - vdbe_emit_create_index(pParse, new_space->def, idx->def, - reg_space_id, idx->def->iid); - } - - /* - * Check to see if we need to create an _sequence table - * for keeping track of autoincrement keys. - */ - if (pParse->create_table_def.has_autoinc) { - assert(reg_space_id != 0); - /* Do an insertion into _sequence. */ - int reg_seq_id = ++pParse->nMem; - sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); - int reg_seq_record = - emitNewSysSequenceRecord(pParse, reg_seq_id, - new_space->def->name); - sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_record); - /* Do an insertion into _space_sequence. */ - int reg_space_seq_record = - emitNewSysSpaceSequenceRecord(pParse, reg_space_id, - reg_seq_id); - sqlVdbeAddOp2(v, OP_SInsert, BOX_SPACE_SEQUENCE_ID, - reg_space_seq_record); - } - /* Code creation of FK constraints, if any. */ - struct fk_constraint_parse *fk_parse; - rlist_foreach_entry(fk_parse, &pParse->create_table_def.new_fkey, - link) { - struct fk_constraint_def *fk_def = fk_parse->fk_def; - if (fk_parse->selfref_cols != NULL) { - struct ExprList *cols = fk_parse->selfref_cols; - for (uint32_t i = 0; i < fk_def->field_count; ++i) { - if (resolve_link(pParse, new_space->def, - cols->a[i].zName, - &fk_def->links[i].parent_field, - fk_def->name) != 0) - return; - } - fk_def->parent_id = reg_space_id; - } else if (fk_parse->is_self_referenced) { - struct index *pk = sql_space_primary_key(new_space); - if (pk->def->key_def->part_count != fk_def->field_count) { - diag_set(ClientError, ER_CREATE_FK_CONSTRAINT, - fk_def->name, "number of columns in "\ - "foreign key does not match the "\ - "number of columns in the primary "\ - "index of referenced table"); - pParse->is_aborted = true; - return; - } - for (uint32_t i = 0; i < fk_def->field_count; ++i) { - fk_def->links[i].parent_field = - pk->def->key_def->parts[i].fieldno; - } - fk_def->parent_id = reg_space_id; - } - fk_def->child_id = reg_space_id; - vdbe_emit_fk_constraint_create(pParse, fk_def, space_name_copy); - } - struct ck_constraint_parse *ck_parse; - rlist_foreach_entry(ck_parse, &pParse->create_table_def.new_check, - link) { - vdbe_emit_ck_constraint_create(pParse, ck_parse->ck_def, - reg_space_id, space_name_copy); - } + sql_vdbe_create_constraints(pParse, reg_space_id); } void @@ -1893,7 +1908,7 @@ sql_create_foreign_key(struct Parse *parse_context) goto tnt_error; } memset(fk_parse, 0, sizeof(*fk_parse)); - rlist_add_entry(&table_def->new_fkey, fk_parse, link); + rlist_add_entry(&parse_context->fkeys, fk_parse, link); } struct Token *parent = create_fk_def->parent_name; assert(parent != NULL); @@ -1911,7 +1926,7 @@ sql_create_foreign_key(struct Parse *parse_context) if (parent_space == NULL) { if (is_self_referenced) { struct fk_constraint_parse *fk = - rlist_first_entry(&table_def->new_fkey, + rlist_first_entry(&parse_context->fkeys, struct fk_constraint_parse, link); fk->selfref_cols = parent_cols; @@ -1926,7 +1941,7 @@ sql_create_foreign_key(struct Parse *parse_context) constraint_name = sqlMPrintf(db, "fk_unnamed_%s_%d", space->def->name, - ++table_def->fkey_count); + ++parse_context->fkey_count); } else { constraint_name = sql_name_from_token(db, &create_def->name); @@ -2042,7 +2057,7 @@ sql_create_foreign_key(struct Parse *parse_context) */ if (!is_alter) { struct fk_constraint_parse *fk_parse = - rlist_first_entry(&table_def->new_fkey, + rlist_first_entry(&parse_context->fkeys, struct fk_constraint_parse, link); fk_parse->fk_def = fk_def; } else { @@ -2065,12 +2080,10 @@ tnt_error: void fk_constraint_change_defer_mode(struct Parse *parse_context, bool is_deferred) { - if (parse_context->db->init.busy || - rlist_empty(&parse_context->create_table_def.new_fkey)) + if (parse_context->db->init.busy || rlist_empty(&parse_context->fkeys)) return; - rlist_first_entry(&parse_context->create_table_def.new_fkey, - struct fk_constraint_parse, link)->fk_def->is_deferred = - is_deferred; + rlist_first_entry(&parse_context->fkeys, struct fk_constraint_parse, + link)->fk_def->is_deferred = is_deferred; } /** @@ -3306,15 +3319,15 @@ vdbe_emit_halt_with_presence_test(struct Parse *parser, int space_id, int sql_add_autoincrement(struct Parse *parse_context, uint32_t fieldno) { - if (parse_context->create_table_def.has_autoinc) { + if (parse_context->has_autoinc) { diag_set(ClientError, ER_SQL_SYNTAX_WITH_POS, parse_context->line_count, parse_context->line_pos, "table must feature at most one AUTOINCREMENT field"); parse_context->is_aborted = true; return -1; } - parse_context->create_table_def.has_autoinc = true; - parse_context->create_table_def.autoinc_fieldno = fieldno; + parse_context->has_autoinc = true; + parse_context->autoinc_fieldno = fieldno; return 0; } diff --git a/src/box/sql/parse_def.h b/src/box/sql/parse_def.h index cb0ecd2fc..1105fda6e 100644 --- a/src/box/sql/parse_def.h +++ b/src/box/sql/parse_def.h @@ -205,26 +205,6 @@ struct create_entity_def { struct create_table_def { struct create_entity_def base; struct space *new_space; - /** - * Number of FK constraints declared within - * CREATE TABLE statement. - */ - uint32_t fkey_count; - /** - * Foreign key constraint appeared in CREATE TABLE stmt. - */ - struct rlist new_fkey; - /** - * Number of CK constraints declared within - * CREATE TABLE statement. - */ - uint32_t check_count; - /** Check constraint appeared in CREATE TABLE stmt. */ - struct rlist new_check; - /** True, if table to be created has AUTOINCREMENT PK. */ - bool has_autoinc; - /** Id of field with AUTOINCREMENT. */ - uint32_t autoinc_fieldno; }; struct create_view_def { @@ -482,9 +462,6 @@ create_table_def_init(struct create_table_def *table_def, struct Token *name, { create_entity_def_init(&table_def->base, ENTITY_TYPE_TABLE, NULL, name, if_not_exists); - rlist_create(&table_def->new_fkey); - rlist_create(&table_def->new_check); - table_def->autoinc_fieldno = 0; } static inline void @@ -499,14 +476,4 @@ create_view_def_init(struct create_view_def *view_def, struct Token *name, view_def->aliases = aliases; } -static inline void -create_table_def_destroy(struct create_table_def *table_def) -{ - if (table_def->new_space == NULL) - return; - struct fk_constraint_parse *fk; - rlist_foreach_entry(fk, &table_def->new_fkey, link) - sql_expr_list_delete(sql_get(), fk->selfref_cols); -} - #endif /* TARANTOOL_BOX_SQL_PARSE_DEF_H_INCLUDED */ diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c index a5a258805..b78eb317f 100644 --- a/src/box/sql/prepare.c +++ b/src/box/sql/prepare.c @@ -200,6 +200,9 @@ sql_parser_create(struct Parse *parser, struct sql *db, uint32_t sql_flags) parser->sql_flags = sql_flags; parser->line_count = 1; parser->line_pos = 1; + rlist_create(&parser->fkeys); + rlist_create(&parser->checks); + parser->has_autoinc = false; region_create(&parser->region, &cord()->slabc); } @@ -211,7 +214,9 @@ sql_parser_destroy(Parse *parser) sql *db = parser->db; sqlDbFree(db, parser->aLabel); sql_expr_list_delete(db, parser->pConstExpr); - create_table_def_destroy(&parser->create_table_def); + struct fk_constraint_parse *fk; + rlist_foreach_entry(fk, &parser->fkeys, link) + sql_expr_list_delete(sql_get(), fk->selfref_cols); if (db != NULL) { assert(db->lookaside.bDisable >= parser->disableLookaside); diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index beb83ce95..fa87e7bd2 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -2257,6 +2257,20 @@ struct Parse { * sqlEndTable() function). */ struct create_table_def create_table_def; + /* + * FK and CK constraints appeared in a . + */ + struct rlist fkeys; + struct rlist checks; + uint32_t fkey_count; + uint32_t check_count; + /* + * True, if column within a statement to be + * created has . + */ + bool has_autoinc; + /* Id of field with . */ + uint32_t autoinc_fieldno; bool initiateTTrans; /* Initiate Tarantool transaction */ /** If set - do not emit byte code at all, just parse. */ bool parse_only; -- 2.21.0 (Apple Git-122) From roman.habibov at tarantool.org Tue Aug 11 03:33:37 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:33:37 +0300 Subject: [Tarantool-patches] [PATCH v3 3/4] schema: add box_space_field_MAX In-Reply-To: <20200811003338.45084-1-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> Message-ID: <20200811003338.45084-4-roman.habibov@tarantool.org> Just add box_space_field_MAX to the _space fields enum. Needed for #3075 --- src/box/schema_def.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/box/schema_def.h b/src/box/schema_def.h index f86cd42f1..238d9f1e4 100644 --- a/src/box/schema_def.h +++ b/src/box/schema_def.h @@ -131,6 +131,7 @@ enum { BOX_SPACE_FIELD_FIELD_COUNT = 4, BOX_SPACE_FIELD_OPTS = 5, BOX_SPACE_FIELD_FORMAT = 6, + box_space_field_MAX = 7, }; /** _index fields. */ -- 2.21.0 (Apple Git-122) From roman.habibov at tarantool.org Tue Aug 11 03:33:38 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:33:38 +0300 Subject: [Tarantool-patches] [PATCH v3 4/4] sql: support column addition In-Reply-To: <20200811003338.45084-1-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> Message-ID: <20200811003338.45084-5-roman.habibov@tarantool.org> Enable to add column to existing space with statement. Column definition can be supplemented with the four types of constraints, , clauses and <[NOT] NULL>, AUTOINCREMENT. Closes #2349, #3075 @TarantoolBot document Title: Add columns to existing tables in SQL Now, it is possible to add columns to existing empty spaces using statement. The column definition is the same as in statement. For example: ``` tarantool> box.execute("CREATE TABLE test (a INTEGER PRIMARY KEY)") --- - row_count: 1 ... tarantool> box.execute([[ALTER TABLE test ADD COLUMN b TEXT > CHECK (LENGTH(b) > 1) > NOT NULL > DEFAULT ('aa') > COLLATE "unicode_ci" > ]]) --- - row_count: 0 ... ``` --- extra/mkkeywordhash.c | 2 +- src/box/errcode.h | 2 + src/box/sql/build.c | 431 +++++++++++++++++++++--------- src/box/sql/parse.y | 44 ++- src/box/sql/parse_def.h | 20 ++ src/box/sql/sqlInt.h | 45 +++- test/box/error.result | 2 + test/sql/add-column.result | 471 +++++++++++++++++++++++++++++++++ test/sql/add-column.test.sql | 167 ++++++++++++ test/sql/checks.result | 20 ++ test/sql/checks.test.lua | 9 + test/sql/foreign-keys.result | 28 ++ test/sql/foreign-keys.test.lua | 11 + 13 files changed, 1108 insertions(+), 144 deletions(-) create mode 100644 test/sql/add-column.result create mode 100644 test/sql/add-column.test.sql diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c index 486b6b30d..dea047241 100644 --- a/extra/mkkeywordhash.c +++ b/extra/mkkeywordhash.c @@ -76,7 +76,7 @@ static Keyword aKeywordTable[] = { { "CHECK", "TK_CHECK", true }, { "COLLATE", "TK_COLLATE", true }, { "COLUMN_NAME", "TK_COLUMN_NAME", true }, - { "COLUMN", "TK_STANDARD", true }, + { "COLUMN", "TK_COLUMN", true }, { "COMMIT", "TK_COMMIT", true }, { "CONFLICT", "TK_CONFLICT", false }, { "CONSTRAINT", "TK_CONSTRAINT", true }, diff --git a/src/box/errcode.h b/src/box/errcode.h index 3c21375f5..cbcffb3a8 100644 --- a/src/box/errcode.h +++ b/src/box/errcode.h @@ -271,6 +271,8 @@ struct errcode_record { /*216 */_(ER_SYNC_QUORUM_TIMEOUT, "Quorum collection for a synchronous transaction is timed out") \ /*217 */_(ER_SYNC_ROLLBACK, "A rollback for a synchronous transaction is received") \ /*218 */_(ER_TUPLE_METADATA_IS_TOO_BIG, "Can't create tuple: metadata size %u is too big") \ + /*219 */_(ER_SQL_CANT_ADD_COLUMN_TO_VIEW, "Can't add column '%s'. '%s' is a view") \ + /*220 */_(ER_SQL_CANT_ADD_AUTOINC, "Can't add AUTOINCREMENT: the space '%s' already has one auto incremented field") \ /* * !IMPORTANT! Please follow instructions at start of the file diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 9013bc86f..6f3d2747d 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -285,48 +285,113 @@ sql_field_retrieve(Parse *parser, struct space_def *space_def, uint32_t id) return field; } -/* - * Add a new column to the table currently being constructed. +/** + * Make shallow copy of @a space on region. * - * The parser calls this routine once for each column declaration - * in a CREATE TABLE statement. sqlStartTable() gets called - * first to get things going. Then this routine is called for each - * column. + * Function is used to add a new column to an existing space with + * statement. Copy space def and index + * array to create constraints appeared in the statement. The + * index array copy will be modified by adding new elements to it. + * It is necessary, because the statement may contain several + * index definitions (constraints). */ +static struct space * +sql_shallow_space_copy(struct Parse *parse, struct space *space) +{ + assert(space->def != NULL); + struct space *ret = sql_ephemeral_space_new(parse, space->def->name); + if (ret == NULL) + return NULL; + ret->index_count = space->index_count; + ret->index_id_max = space->index_id_max; + uint32_t indexes_sz = sizeof(struct index *) * (ret->index_count); + ret->index = (struct index **) malloc(indexes_sz); + if (ret->index == NULL) { + diag_set(OutOfMemory, indexes_sz, "realloc", "ret->index"); + return NULL; + } + memcpy(ret->index, space->index, indexes_sz); + memcpy(ret->def, space->def, sizeof(struct space_def)); + ret->def->opts.is_temporary = true; + ret->def->opts.is_ephemeral = true; + if (ret->def->field_count != 0) { + uint32_t fields_size = 0; + ret->def->fields = + region_alloc_array(&parse->region, + typeof(struct field_def), + ret->def->field_count, &fields_size); + if (ret->def->fields == NULL) { + diag_set(OutOfMemory, fields_size, "region_alloc", + "ret->def->fields"); + free(ret->index); + return NULL; + } + memcpy(ret->def->fields, space->def->fields, fields_size); + } + + return ret; +} + void -sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) +sql_create_column_start(struct Parse *parse) { - assert(type_def != NULL); - char *z; - sql *db = pParse->db; - if (pParse->create_table_def.new_space == NULL) - return; - struct space_def *def = pParse->create_table_def.new_space->def; + struct create_column_def *create_column_def = &parse->create_column_def; + struct alter_entity_def *alter_entity_def = + &create_column_def->base.base; + assert(alter_entity_def->entity_type == ENTITY_TYPE_COLUMN); + assert(alter_entity_def->alter_action == ALTER_ACTION_CREATE); + struct space *space = parse->create_table_def.new_space; + bool is_alter = space == NULL; + struct sql *db = parse->db; + if (is_alter) { + const char *space_name = + alter_entity_def->entity_name->a[0].zName; + space = space_by_name(space_name); + if (space == NULL) { + diag_set(ClientError, ER_NO_SUCH_SPACE, space_name); + goto tnt_error; + } + space = sql_shallow_space_copy(parse, space); + if (space == NULL) + goto tnt_error; + } + create_column_def->space = space; + struct space_def *def = space->def; + assert(def->opts.is_ephemeral); #if SQL_MAX_COLUMN if ((int)def->field_count + 1 > db->aLimit[SQL_LIMIT_COLUMN]) { diag_set(ClientError, ER_SQL_COLUMN_COUNT_MAX, def->name, def->field_count + 1, db->aLimit[SQL_LIMIT_COLUMN]); - pParse->is_aborted = true; - return; + goto tnt_error; } #endif + + struct region *region = &parse->region; + struct Token *name = &create_column_def->base.name; + char *column_name = + sql_normalized_name_region_new(region, name->z, name->n); + if (column_name == NULL) + goto tnt_error; + + if (is_alter && def->opts.is_view) { + diag_set(ClientError, ER_SQL_CANT_ADD_COLUMN_TO_VIEW, + column_name, def->name); + goto tnt_error; + } + /* - * As sql_field_retrieve will allocate memory on region - * ensure that def is also temporal and would be dropped. + * Format can be set in Lua, then exact_field_count can be + * zero, but field_count is not. */ - assert(def->opts.is_ephemeral); - if (sql_field_retrieve(pParse, def, def->field_count) == NULL) + if (def->exact_field_count == 0) + def->exact_field_count = def->field_count; + if (sql_field_retrieve(parse, def, def->field_count) == NULL) return; - struct region *region = &pParse->region; - z = sql_normalized_name_region_new(region, pName->z, pName->n); - if (z == NULL) { - pParse->is_aborted = true; - return; - } + struct field_def *column_def = &def->fields[def->field_count]; memcpy(column_def, &field_def_default, sizeof(field_def_default)); - column_def->name = z; + column_def->name = column_name; /* * Marker ON_CONFLICT_ACTION_DEFAULT is used to detect * attempts to define NULL multiple time or to detect @@ -334,18 +399,86 @@ sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) */ column_def->nullable_action = ON_CONFLICT_ACTION_DEFAULT; column_def->is_nullable = true; - column_def->type = type_def->type; + column_def->type = create_column_def->type_def->type; def->field_count++; + + sqlSrcListDelete(db, alter_entity_def->entity_name); + return; +tnt_error: + parse->is_aborted = true; + sqlSrcListDelete(db, alter_entity_def->entity_name); +} + +static void +sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id); + +void +sql_create_column_end(struct Parse *parse) +{ + struct space *space = parse->create_column_def.space; + assert(space != NULL); + struct space_def *def = space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (field->nullable_action == ON_CONFLICT_ACTION_DEFAULT) { + field->nullable_action = ON_CONFLICT_ACTION_NONE; + field->is_nullable = true; + } + /* + * Encode the format array and emit code to update _space. + */ + uint32_t table_stmt_sz = 0; + struct region *region = &parse->region; + char *table_stmt = sql_encode_table(region, def, &table_stmt_sz); + char *raw = sqlDbMallocRaw(parse->db, table_stmt_sz); + if (table_stmt == NULL || raw == NULL) { + parse->is_aborted = true; + return; + } + memcpy(raw, table_stmt, table_stmt_sz); + + struct Vdbe *v = sqlGetVdbe(parse); + assert(v != NULL); + + struct space *system_space = space_by_id(BOX_SPACE_ID); + assert(system_space != NULL); + int cursor = parse->nTab++; + vdbe_emit_open_cursor(parse, cursor, 0, system_space); + sqlVdbeChangeP5(v, OPFLAG_SYSTEMSP); + + int key_reg = ++parse->nMem; + sqlVdbeAddOp2(v, OP_Integer, def->id, key_reg); + int addr = sqlVdbeAddOp4Int(v, OP_Found, cursor, 0, key_reg, 1); + sqlVdbeAddOp2(v, OP_Halt, -1, ON_CONFLICT_ACTION_ABORT); + sqlVdbeJumpHere(v, addr); + + int tuple_reg = sqlGetTempRange(parse, box_space_field_MAX + 1); + for (int i = 0; i < box_space_field_MAX - 1; ++i) + sqlVdbeAddOp3(v, OP_Column, cursor, i, tuple_reg + i); + sqlVdbeAddOp1(v, OP_Close, cursor); + + sqlVdbeAddOp2(v, OP_Integer, def->field_count, tuple_reg + 4); + sqlVdbeAddOp4(v, OP_Blob, table_stmt_sz, tuple_reg + 6, + SQL_SUBTYPE_MSGPACK, raw, P4_DYNAMIC); + sqlVdbeAddOp3(v, OP_MakeRecord, tuple_reg, box_space_field_MAX, + tuple_reg + box_space_field_MAX); + sqlVdbeAddOp4(v, OP_IdxReplace, tuple_reg + box_space_field_MAX, 0, 0, + (char *) system_space, P4_SPACEPTR); + sql_vdbe_create_constraints(parse, key_reg); + + /* + * Clean up array allocated in sql_shallow_space_copy(). + */ + free(space->index); } void sql_column_add_nullable_action(struct Parse *parser, enum on_conflict_action nullable_action) { - struct space *space = parser->create_table_def.new_space; - if (space == NULL || NEVER(space->def->field_count < 1)) + assert(parser->create_column_def.space != NULL); + struct space_def *def = parser->create_column_def.space->def; + if (NEVER(def->field_count < 1)) return; - struct space_def *def = space->def; struct field_def *field = &def->fields[def->field_count - 1]; if (field->nullable_action != ON_CONFLICT_ACTION_DEFAULT && nullable_action != field->nullable_action) { @@ -364,51 +497,42 @@ sql_column_add_nullable_action(struct Parse *parser, } /* - * The expression is the default value for the most recently added column - * of the table currently under construction. + * The expression is the default value for the most recently added + * column. * * Default value expressions must be constant. Raise an exception if this * is not the case. * * This routine is called by the parser while in the middle of - * parsing a CREATE TABLE statement. + * parsing a or a + * statement. */ void sqlAddDefaultValue(Parse * pParse, ExprSpan * pSpan) { sql *db = pParse->db; - struct space *p = pParse->create_table_def.new_space; - if (p != NULL) { - assert(p->def->opts.is_ephemeral); - struct space_def *def = p->def; - if (!sqlExprIsConstantOrFunction - (pSpan->pExpr, db->init.busy)) { - const char *column_name = - def->fields[def->field_count - 1].name; - diag_set(ClientError, ER_CREATE_SPACE, def->name, - tt_sprintf("default value of column '%s' is "\ - "not constant", column_name)); + assert(pParse->create_column_def.space != NULL); + struct space_def *def = pParse->create_column_def.space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (!sqlExprIsConstantOrFunction(pSpan->pExpr, db->init.busy)) { + diag_set(ClientError, ER_CREATE_SPACE, def->name, + tt_sprintf("default value of column '%s' is not " + "constant", field->name)); + pParse->is_aborted = true; + } else { + struct region *region = &pParse->region; + uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); + field->default_value = region_alloc(region, default_length + 1); + if (field->default_value == NULL) { + diag_set(OutOfMemory, default_length + 1, + "region_alloc", "field->default_value"); pParse->is_aborted = true; - } else { - assert(def != NULL); - struct field_def *field = - &def->fields[def->field_count - 1]; - struct region *region = &pParse->region; - uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); - field->default_value = region_alloc(region, - default_length + 1); - if (field->default_value == NULL) { - diag_set(OutOfMemory, default_length + 1, - "region_alloc", - "field->default_value"); - pParse->is_aborted = true; - return; - } - strncpy(field->default_value, pSpan->zStart, - default_length); - field->default_value[default_length] = '\0'; + goto add_default_value_exit; } + strncpy(field->default_value, pSpan->zStart, default_length); + field->default_value[default_length] = '\0'; } +add_default_value_exit: sql_expr_delete(db, pSpan->pExpr, false); } @@ -447,6 +571,8 @@ sqlAddPrimaryKey(struct Parse *pParse) int nTerm; struct ExprList *pList = pParse->create_index_def.cols; struct space *space = pParse->create_table_def.new_space; + if (space == NULL) + space = pParse->create_column_def.space; if (space == NULL) goto primary_key_exit; if (sql_space_primary_key(space) != NULL) { @@ -574,8 +700,10 @@ sql_create_check_contraint(struct Parse *parser) (struct alter_entity_def *) create_ck_def; assert(alter_def->entity_type == ENTITY_TYPE_CK); (void) alter_def; - struct space *space = parser->create_table_def.new_space; - bool is_alter = space == NULL; + struct space *space = parser->create_column_def.space; + if (space == NULL) + space = parser->create_table_def.new_space; + bool is_alter_add_constr = space == NULL; /* Prepare payload for ck constraint definition. */ struct region *region = &parser->region; @@ -589,9 +717,23 @@ sql_create_check_contraint(struct Parse *parser) return; } } else { - assert(! is_alter); - uint32_t ck_idx = ++parser->check_count; - name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, ck_idx); + assert(!is_alter_add_constr); + uint32_t idx = ++parser->check_count; + /* + * If it is we should + * count the existing CHECK constraints in the + * space and form a name based on this. + */ + if (parser->create_table_def.new_space == NULL) { + struct space *original_space = + space_by_name(space->def->name); + assert(original_space != NULL); + struct rlist *checks = &original_space->ck_constraint; + struct ck_constraint *ck; + rlist_foreach_entry(ck, checks, link) + idx++; + } + name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, idx); } size_t name_len = strlen(name); @@ -634,7 +776,7 @@ sql_create_check_contraint(struct Parse *parser) trim_space_snprintf(ck_def->expr_str, expr_str, expr_str_len); memcpy(ck_def->name, name, name_len); ck_def->name[name_len] = '\0'; - if (is_alter) { + if (is_alter_add_constr) { const char *space_name = alter_def->entity_name->a[0].zName; struct space *space = space_by_name(space_name); if (space == NULL) { @@ -663,9 +805,8 @@ sql_create_check_contraint(struct Parse *parser) void sqlAddCollateType(Parse * pParse, Token * pToken) { - struct space *space = pParse->create_table_def.new_space; - if (space == NULL) - return; + struct space *space = pParse->create_column_def.space; + assert(space != NULL); uint32_t i = space->def->field_count - 1; sql *db = pParse->db; char *coll_name = sql_name_from_token(db, pToken); @@ -704,8 +845,7 @@ sql_column_collation(struct space_def *def, uint32_t column, uint32_t *coll_id) * * In cases mentioned above collation is fetched by id. */ - if (space == NULL) { - assert(def->opts.is_ephemeral); + if (def->opts.is_ephemeral) { assert(column < (uint32_t)def->field_count); *coll_id = def->fields[column].coll_id; struct coll_id *collation = coll_by_id(*coll_id); @@ -794,7 +934,8 @@ vdbe_emit_create_index(struct Parse *parse, struct space_def *def, memcpy(raw, index_parts, index_parts_sz); index_parts = raw; - if (parse->create_table_def.new_space != NULL) { + if (parse->create_table_def.new_space != NULL || + parse->create_column_def.space != NULL) { sqlVdbeAddOp2(v, OP_SCopy, space_id_reg, entry_reg); sqlVdbeAddOp2(v, OP_Integer, idx_def->iid, entry_reg + 1); } else { @@ -1032,18 +1173,21 @@ vdbe_emit_fk_constraint_create(struct Parse *parse_context, P4_DYNAMIC); /* * In case we are adding FK constraints during execution - * of statement, we don't have child - * id, but we know register where it will be stored. + * of or + * statement, we don't have child id, but we know register + * where it will be stored. */ - if (parse_context->create_table_def.new_space != NULL) { + bool is_alter_add_constr = + parse_context->create_table_def.new_space == NULL && + parse_context->create_column_def.space == NULL; + if (!is_alter_add_constr) { sqlVdbeAddOp2(vdbe, OP_SCopy, fk->child_id, constr_tuple_reg + 1); } else { sqlVdbeAddOp2(vdbe, OP_Integer, fk->child_id, constr_tuple_reg + 1); } - if (parse_context->create_table_def.new_space != NULL && - fk_constraint_is_self_referenced(fk)) { + if (!is_alter_add_constr && fk_constraint_is_self_referenced(fk)) { sqlVdbeAddOp2(vdbe, OP_SCopy, fk->parent_id, constr_tuple_reg + 2); } else { @@ -1107,7 +1251,7 @@ vdbe_emit_fk_constraint_create(struct Parse *parse_context, constr_tuple_reg + 9); sqlVdbeAddOp2(vdbe, OP_SInsert, BOX_FK_CONSTRAINT_ID, constr_tuple_reg + 9); - if (parse_context->create_table_def.new_space == NULL) { + if (is_alter_add_constr) { sqlVdbeCountChanges(vdbe); sqlVdbeChangeP5(vdbe, OPFLAG_NCHANGE); } @@ -1148,15 +1292,21 @@ resolve_link(struct Parse *parse_context, const struct space_def *def, /** * Emit code to create sequences, indexes, check and foreign key - * constraints appeared in . + * constraints appeared in or + * . */ static void sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) { assert(reg_space_id != 0); struct space *space = parse->create_table_def.new_space; - assert(space != NULL); + bool is_alter = space == NULL; uint32_t i = 0; + if (is_alter) { + space = parse->create_column_def.space; + i = space_by_name(space->def->name)->index_count; + } + assert(space != NULL); for (; i < space->index_count; ++i) { struct index *idx = space->index[i]; vdbe_emit_create_index(parse, space->def, idx->def, @@ -1175,6 +1325,21 @@ sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); int reg_seq_rec = emitNewSysSequenceRecord(parse, reg_seq_id, space->def->name); + if (is_alter) { + int errcode = ER_SQL_CANT_ADD_AUTOINC; + const char *error_msg = + tt_sprintf(tnt_errcode_desc(errcode), + space->def->name); + if (vdbe_emit_halt_with_presence_test(parse, + BOX_SEQUENCE_ID, + 2, + reg_seq_rec + 3, + 1, errcode, + error_msg, false, + OP_NoConflict) + != 0) + return; + } sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_rec); /* Do an insertion into _space_sequence. */ int reg_space_seq_record = @@ -1873,24 +2038,28 @@ sql_create_foreign_key(struct Parse *parse_context) char *parent_name = NULL; char *constraint_name = NULL; bool is_self_referenced = false; + struct space *space = parse_context->create_column_def.space; struct create_table_def *table_def = &parse_context->create_table_def; - struct space *space = table_def->new_space; + if (space == NULL) + space = table_def->new_space; /* - * Space under construction during CREATE TABLE - * processing. NULL for ALTER TABLE statement handling. + * Space under construction during + * processing or shallow copy of space during . NULL for statement handling. */ - bool is_alter = space == NULL; + bool is_alter_add_constr = space == NULL; uint32_t child_cols_count; struct ExprList *child_cols = create_fk_def->child_cols; if (child_cols == NULL) { - assert(!is_alter); + assert(!is_alter_add_constr); child_cols_count = 1; } else { child_cols_count = child_cols->nExpr; } struct ExprList *parent_cols = create_fk_def->parent_cols; struct space *child_space = NULL; - if (is_alter) { + if (is_alter_add_constr) { const char *child_name = alter_def->entity_name->a[0].zName; child_space = space_by_name(child_name); if (child_space == NULL) { @@ -1908,6 +2077,8 @@ sql_create_foreign_key(struct Parse *parse_context) goto tnt_error; } memset(fk_parse, 0, sizeof(*fk_parse)); + if (parse_context->create_column_def.space != NULL) + child_space = space; rlist_add_entry(&parse_context->fkeys, fk_parse, link); } struct Token *parent = create_fk_def->parent_name; @@ -1920,28 +2091,45 @@ sql_create_foreign_key(struct Parse *parse_context) * self-referenced, but in this case parent (which is * also child) table will definitely exist. */ - is_self_referenced = !is_alter && + is_self_referenced = !is_alter_add_constr && strcmp(parent_name, space->def->name) == 0; struct space *parent_space = space_by_name(parent_name); - if (parent_space == NULL) { - if (is_self_referenced) { - struct fk_constraint_parse *fk = - rlist_first_entry(&parse_context->fkeys, - struct fk_constraint_parse, - link); - fk->selfref_cols = parent_cols; - fk->is_self_referenced = true; - } else { - diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name);; - goto tnt_error; - } + if (parent_space == NULL && !is_self_referenced) { + diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name); + goto tnt_error; + } + if (is_self_referenced) { + struct fk_constraint_parse *fk = + rlist_first_entry(&parse_context->fkeys, + struct fk_constraint_parse, + link); + fk->selfref_cols = parent_cols; + fk->is_self_referenced = true; } - if (!is_alter) { + if (!is_alter_add_constr) { if (create_def->name.n == 0) { - constraint_name = - sqlMPrintf(db, "fk_unnamed_%s_%d", - space->def->name, - ++parse_context->fkey_count); + uint32_t idx = ++parse_context->fkey_count; + /* + * If it is we + * should count the existing FK + * constraints in the space and form a + * name based on this. + */ + if (table_def->new_space == NULL) { + struct space *original_space = + space_by_name(space->def->name); + assert(original_space != NULL); + struct rlist *child_fk = + &original_space->child_fk_constraint; + if (!rlist_empty(child_fk)) { + struct fk_constraint *fk; + rlist_foreach_entry(fk, child_fk, + in_child_space) + idx++; + } + } + constraint_name = sqlMPrintf(db, "fk_unnamed_%s_%d", + space->def->name, idx); } else { constraint_name = sql_name_from_token(db, &create_def->name); @@ -2001,7 +2189,8 @@ sql_create_foreign_key(struct Parse *parse_context) } int actions = create_fk_def->actions; fk_def->field_count = child_cols_count; - fk_def->child_id = child_space != NULL ? child_space->def->id : 0; + fk_def->child_id = table_def->new_space == NULL ? + child_space->def->id : 0; fk_def->parent_id = parent_space != NULL ? parent_space->def->id : 0; fk_def->is_deferred = create_constr_def->is_deferred; fk_def->match = (enum fk_constraint_match) (create_fk_def->match); @@ -2021,7 +2210,7 @@ sql_create_foreign_key(struct Parse *parse_context) constraint_name) != 0) { goto exit_create_fk; } - if (!is_alter) { + if (!is_alter_add_constr) { if (child_cols == NULL) { assert(i == 0); /* @@ -2050,12 +2239,13 @@ sql_create_foreign_key(struct Parse *parse_context) memcpy(fk_def->name, constraint_name, name_len); fk_def->name[name_len] = '\0'; /* - * In case of CREATE TABLE processing, all foreign keys - * constraints must be created after space itself, so - * lets delay it until sqlEndTable() call and simply + * In case of or processing, all foreign keys constraints must + * be created after space itself, so lets delay it until + * sqlEndTable() or sql_add_column_end() call and simply * maintain list of all FK constraints inside parser. */ - if (!is_alter) { + if (!is_alter_add_constr) { struct fk_constraint_parse *fk_parse = rlist_first_entry(&parse_context->fkeys, struct fk_constraint_parse, link); @@ -2407,7 +2597,10 @@ sql_create_index(struct Parse *parse) { * Find the table that is to be indexed. * Return early if not found. */ - struct space *space = NULL; + struct space *space = parse->create_table_def.new_space; + if (space == NULL) + space = parse->create_column_def.space; + bool is_create_table_or_add_col = space != NULL; struct Token token = create_entity_def->name; if (tbl_name != NULL) { assert(token.n > 0 && token.z != NULL); @@ -2420,10 +2613,8 @@ sql_create_index(struct Parse *parse) { } goto exit_create_index; } - } else { - if (parse->create_table_def.new_space == NULL) - goto exit_create_index; - space = parse->create_table_def.new_space; + } else if (space == NULL) { + goto exit_create_index; } struct space_def *def = space->def; @@ -2458,7 +2649,7 @@ sql_create_index(struct Parse *parse) { * 2) UNIQUE constraint is non-named and standard * auto-index name will be generated. */ - if (parse->create_table_def.new_space == NULL) { + if (!is_create_table_or_add_col) { assert(token.z != NULL); name = sql_name_from_token(db, &token); if (name == NULL) { @@ -2624,7 +2815,7 @@ sql_create_index(struct Parse *parse) { * constraint, but has different onError (behavior on * constraint violation), then an error is raised. */ - if (parse->create_table_def.new_space != NULL) { + if (is_create_table_or_add_col) { for (uint32_t i = 0; i < space->index_count; ++i) { struct index *existing_idx = space->index[i]; uint32_t iid = existing_idx->def->iid; @@ -2712,7 +2903,7 @@ sql_create_index(struct Parse *parse) { sqlVdbeAddOp0(vdbe, OP_Expire); } - if (tbl_name != NULL) + if (!is_create_table_or_add_col) goto exit_create_index; table_add_index(space, index); index = NULL; diff --git a/src/box/sql/parse.y b/src/box/sql/parse.y index 995875566..0c9887851 100644 --- a/src/box/sql/parse.y +++ b/src/box/sql/parse.y @@ -226,19 +226,24 @@ create_table_end ::= . { sqlEndTable(pParse); } */ columnlist ::= columnlist COMMA tcons. -columnlist ::= columnlist COMMA columnname carglist autoinc(I). { - uint32_t fieldno = pParse->create_table_def.new_space->def->field_count - 1; - if (I == 1 && sql_add_autoincrement(pParse, fieldno) != 0) - return; +columnlist ::= columnlist COMMA column_def create_column_end. +columnlist ::= column_def create_column_end. + +column_def ::= column_name_and_type carglist. + +column_name_and_type ::= nm(A) typedef(Y). { + create_column_def_init(&pParse->create_column_def, NULL, &A, &Y); + sql_create_column_start(pParse); } -columnlist ::= columnname carglist autoinc(I). { - uint32_t fieldno = pParse->create_table_def.new_space->def->field_count - 1; +create_column_end ::= autoinc(I). { + uint32_t fieldno = pParse->create_column_def.space->def->field_count - 1; if (I == 1 && sql_add_autoincrement(pParse, fieldno) != 0) return; + if (pParse->create_table_def.new_space == NULL) + sql_create_column_end(pParse); } columnlist ::= tcons. -columnname(A) ::= nm(A) typedef(Y). {sqlAddColumn(pParse,&A,&Y);} // An IDENTIFIER can be a generic identifier, or one of several // keywords. Any non-standard keyword can also be an identifier. @@ -281,9 +286,11 @@ nm(A) ::= id(A). { } } -// "carglist" is a list of additional constraints that come after the -// column name and column type in a CREATE TABLE statement. -// +/* + * "carglist" is a list of additional constraints and clauses that + * come after the column name and column type in a + * or statement. + */ carglist ::= carglist ccons. carglist ::= . %type cconsname { struct Token } @@ -1735,11 +1742,28 @@ alter_table_start(A) ::= ALTER TABLE fullname(T) . { A = T; } %type alter_add_constraint {struct alter_args} alter_add_constraint(A) ::= alter_table_start(T) ADD CONSTRAINT nm(N). { + A.table_name = T; + A.name = N; + pParse->initiateTTrans = true; + } + +%type alter_add_column {struct alter_args} +alter_add_column(A) ::= alter_table_start(T) ADD column_name(N). { A.table_name = T; A.name = N; pParse->initiateTTrans = true; } +column_name(N) ::= COLUMN nm(A). { N = A; } +column_name(N) ::= nm(A). { N = A; } + +cmd ::= alter_column_def carglist create_column_end. + +alter_column_def ::= alter_add_column(N) typedef(Y). { + create_column_def_init(&pParse->create_column_def, N.table_name, &N.name, &Y); + sql_create_column_start(pParse); +} + cmd ::= alter_add_constraint(N) FOREIGN KEY LP eidlist(FA) RP REFERENCES nm(T) eidlist_opt(TA) matcharg(M) refargs(R) defer_subclause_opt(D). { create_fk_def_init(&pParse->create_fk_def, N.table_name, &N.name, FA, &T, TA, diff --git a/src/box/sql/parse_def.h b/src/box/sql/parse_def.h index 1105fda6e..336914c57 100644 --- a/src/box/sql/parse_def.h +++ b/src/box/sql/parse_def.h @@ -35,6 +35,7 @@ #include "box/fk_constraint.h" #include "box/key_def.h" #include "box/sql.h" +#include "box/constraint_id.h" /** * This file contains auxiliary structures and functions which @@ -154,6 +155,7 @@ enum sql_index_type { enum entity_type { ENTITY_TYPE_TABLE = 0, + ENTITY_TYPE_COLUMN, ENTITY_TYPE_VIEW, ENTITY_TYPE_INDEX, ENTITY_TYPE_TRIGGER, @@ -207,6 +209,14 @@ struct create_table_def { struct space *new_space; }; +struct create_column_def { + struct create_entity_def base; + /** Shallow space_def copy. */ + struct space *space; + /** Column type. */ + struct type_def *type_def; +}; + struct create_view_def { struct create_entity_def base; /** @@ -464,6 +474,16 @@ create_table_def_init(struct create_table_def *table_def, struct Token *name, if_not_exists); } +static inline void +create_column_def_init(struct create_column_def *column_def, + struct SrcList *table_name, struct Token *name, + struct type_def *type_def) +{ + create_entity_def_init(&column_def->base, ENTITY_TYPE_COLUMN, + table_name, name, false); + column_def->type_def = type_def; +} + static inline void create_view_def_init(struct create_view_def *view_def, struct Token *name, struct Token *create, struct ExprList *aliases, diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index fa87e7bd2..32142a871 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -2251,22 +2251,26 @@ struct Parse { struct enable_entity_def enable_entity_def; }; /** - * Table def is not part of union since information - * being held must survive till the end of parsing of - * whole CREATE TABLE statement (to pass it to - * sqlEndTable() function). + * Table def or column def is not part of union since + * information being held must survive till the end of + * parsing of whole or + * statement (to pass it to + * sqlEndTable() sql_create_column_end() function). */ struct create_table_def create_table_def; + struct create_column_def create_column_def; /* - * FK and CK constraints appeared in a . + * FK and CK constraints appeared in a or + * an statement. */ struct rlist fkeys; struct rlist checks; uint32_t fkey_count; uint32_t check_count; /* - * True, if column within a statement to be - * created has . + * True, if column in a or an + * statement to be created has + * . */ bool has_autoinc; /* Id of field with . */ @@ -2860,15 +2864,30 @@ struct space *sqlResultSetOfSelect(Parse *, Select *); struct space * sqlStartTable(Parse *, Token *); -void sqlAddColumn(Parse *, Token *, struct type_def *); + +/** + * Add new field to the format of ephemeral space in + * create_table_def. If it is create shallow copy of + * the existing space and add field to its format. + */ +void +sql_create_column_start(struct Parse *parse); + +/** + * Emit code to update entry in _space and code to create + * constraints (entries in _index, _ck_constraint, _fk_constraint) + * described with this column. + */ +void +sql_create_column_end(struct Parse *parse); /** * This routine is called by the parser while in the middle of - * parsing a CREATE TABLE statement. A "NOT NULL" constraint has - * been seen on a column. This routine sets the is_nullable flag - * on the column currently under construction. - * If nullable_action has been already set, this function raises - * an error. + * parsing a or a + * statement. A "NOT NULL" constraint has been seen on a column. + * This routine sets the is_nullable flag on the column currently + * under construction. If nullable_action has been already set, + * this function raises an error. * * @param parser SQL Parser object. * @param nullable_action on_conflict_action value. diff --git a/test/box/error.result b/test/box/error.result index cdecdb221..6fc3cb99f 100644 --- a/test/box/error.result +++ b/test/box/error.result @@ -437,6 +437,8 @@ t; | 216: box.error.SYNC_QUORUM_TIMEOUT | 217: box.error.SYNC_ROLLBACK | 218: box.error.TUPLE_METADATA_IS_TOO_BIG + | 219: box.error.SQL_CANT_ADD_COLUMN_TO_VIEW + | 220: box.error.SQL_CANT_ADD_AUTOINC | ... test_run:cmd("setopt delimiter ''"); diff --git a/test/sql/add-column.result b/test/sql/add-column.result new file mode 100644 index 000000000..f86259105 --- /dev/null +++ b/test/sql/add-column.result @@ -0,0 +1,471 @@ +-- test-run result file version 2 +-- +-- gh-3075: Check statement. +-- +CREATE TABLE t1 (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... + +-- +-- COLUMN keyword is optional. Check it here, but omit it below. +-- +ALTER TABLE t1 ADD COLUMN b INT; + | --- + | - row_count: 0 + | ... + +-- +-- A column with the same name already exists. +-- +ALTER TABLE t1 ADD b SCALAR; + | --- + | - null + | - Space field 'B' is duplicate + | ... + +-- +-- Can't add column to a view. +-- +CREATE VIEW v AS SELECT * FROM t1; + | --- + | - row_count: 1 + | ... +ALTER TABLE v ADD b INT; + | --- + | - null + | - Can't add column 'B'. 'V' is a view + | ... +DROP VIEW v; + | --- + | - row_count: 1 + | ... + +-- +-- Check PRIMARY KEY constraint works with an added column. +-- +CREATE TABLE pk_check (a INT CONSTRAINT pk PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE pk_check DROP CONSTRAINT pk; + | --- + | - row_count: 1 + | ... +ALTER TABLE pk_check ADD b INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO pk_check VALUES (1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO pk_check VALUES (1, 1); + | --- + | - null + | - Duplicate key exists in unique index 'pk_unnamed_PK_CHECK_1' in space 'PK_CHECK' + | ... +DROP TABLE pk_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check UNIQUE constraint works with an added column. +-- +CREATE TABLE unique_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE unique_check ADD b INT UNIQUE; + | --- + | - row_count: 0 + | ... +INSERT INTO unique_check VALUES (1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO unique_check VALUES (2, 1); + | --- + | - null + | - Duplicate key exists in unique index 'unique_unnamed_UNIQUE_CHECK_2' in space 'UNIQUE_CHECK' + | ... +DROP TABLE unique_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check CHECK constraint works with an added column. +-- +CREATE TABLE ck_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE ck_check ADD b INT CHECK (b > 0); + | --- + | - row_count: 0 + | ... +INSERT INTO ck_check VALUES (1, 0); + | --- + | - null + | - 'Check constraint failed ''ck_unnamed_CK_CHECK_1'': b > 0' + | ... +DROP TABLE ck_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check FOREIGN KEY constraint works with an added column. +-- +CREATE TABLE fk_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE fk_check ADD b INT REFERENCES t1(a); + | --- + | - row_count: 0 + | ... +INSERT INTO fk_check VALUES (0, 1); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +INSERT INTO fk_check VALUES (2, 0); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +INSERT INTO fk_check VALUES (2, 1); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +DROP TABLE fk_check; + | --- + | - row_count: 1 + | ... +DROP TABLE t1; + | --- + | - row_count: 1 + | ... +-- +-- Check FOREIGN KEY (self-referenced) constraint works with an +-- added column. +-- +CREATE TABLE self (id INT PRIMARY KEY AUTOINCREMENT, a INT UNIQUE) + | --- + | - row_count: 1 + | ... +ALTER TABLE self ADD b INT REFERENCES self(a) + | --- + | - row_count: 0 + | ... +INSERT INTO self(a,b) VALUES(1, 1); + | --- + | - autoincrement_ids: + | - 1 + | row_count: 1 + | ... +UPDATE self SET a = 2, b = 2; + | --- + | - row_count: 1 + | ... +UPDATE self SET b = 3; + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +UPDATE self SET a = 3; + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +DROP TABLE self; + | --- + | - row_count: 1 + | ... + +-- +-- Check AUTOINCREMENT works with an added column. +-- +CREATE TABLE autoinc_check (a INT CONSTRAINT pk PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE autoinc_check DROP CONSTRAINT pk; + | --- + | - row_count: 1 + | ... +ALTER TABLE autoinc_check ADD b INT PRIMARY KEY AUTOINCREMENT; + | --- + | - row_count: 0 + | ... +INSERT INTO autoinc_check(a) VALUES(1); + | --- + | - autoincrement_ids: + | - 1 + | row_count: 1 + | ... +INSERT INTO autoinc_check(a) VALUES(1); + | --- + | - autoincrement_ids: + | - 2 + | row_count: 1 + | ... +TRUNCATE TABLE autoinc_check; + | --- + | - row_count: 0 + | ... + +-- +-- Can't add second column with AUTOINCREMENT. +-- +ALTER TABLE autoinc_check ADD c INT AUTOINCREMENT; + | --- + | - null + | - 'Can''t add AUTOINCREMENT: the space ''AUTOINC_CHECK'' already has one auto incremented + | field' + | ... +DROP TABLE autoinc_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check COLLATE clause works with an added column. +-- +CREATE TABLE collate_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE collate_check ADD b TEXT COLLATE "unicode_ci"; + | --- + | - row_count: 0 + | ... +INSERT INTO collate_check VALUES (1, 'a'); + | --- + | - row_count: 1 + | ... +INSERT INTO collate_check VALUES (2, 'A'); + | --- + | - row_count: 1 + | ... +SELECT * FROM collate_check WHERE b LIKE 'a'; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: string + | rows: + | - [1, 'a'] + | - [2, 'A'] + | ... +DROP TABLE collate_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check DEFAULT clause works with an added column. +-- +CREATE TABLE default_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE default_check ADD b TEXT DEFAULT ('a'); + | --- + | - row_count: 0 + | ... +INSERT INTO default_check(a) VALUES (1); + | --- + | - row_count: 1 + | ... +SELECT * FROM default_check; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: string + | rows: + | - [1, 'a'] + | ... +DROP TABLE default_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check NULL constraint works with an added column. +-- +CREATE TABLE null_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE null_check ADD b TEXT NULL; + | --- + | - row_count: 0 + | ... +INSERT INTO null_check(a) VALUES (1); + | --- + | - row_count: 1 + | ... +DROP TABLE null_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check NOT NULL constraint works with an added column. +-- +CREATE TABLE notnull_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE notnull_check ADD b TEXT NOT NULL; + | --- + | - row_count: 0 + | ... +INSERT INTO notnull_check(a) VALUES (1); + | --- + | - null + | - 'Failed to execute SQL statement: NOT NULL constraint failed: NOTNULL_CHECK.B' + | ... +DROP TABLE notnull_check; + | --- + | - row_count: 1 + | ... + +-- +-- Can't add a column with DEAFULT or NULL to a non-empty space. +-- This ability isn't implemented yet. +-- +CREATE TABLE non_empty (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +INSERT INTO non_empty VALUES (1); + | --- + | - row_count: 1 + | ... +ALTER TABLE non_empty ADD b INT NULL; + | --- + | - null + | - Tuple field count 1 does not match space field count 2 + | ... +ALTER TABLE non_empty ADD b INT DEFAULT (1); + | --- + | - null + | - Tuple field count 1 does not match space field count 2 + | ... +DROP TABLE non_empty; + | --- + | - row_count: 1 + | ... + +-- +-- Add to a no-SQL adjusted space without format. +-- +\set language lua + | --- + | - true + | ... +_ = box.schema.space.create('WITHOUT_FORMAT') + | --- + | ... +\set language sql + | --- + | - true + | ... +ALTER TABLE without_format ADD a INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO without_format VALUES (1); + | --- + | - row_count: 1 + | ... +DROP TABLE without_format; + | --- + | - row_count: 1 + | ... + +-- +-- Add to a no-SQL adjusted space with format. +-- +\set language lua + | --- + | - true + | ... +with_format = box.schema.space.create('WITH_FORMAT') + | --- + | ... +with_format:format{{name = 'A', type = 'unsigned'}} + | --- + | ... +\set language sql + | --- + | - true + | ... +ALTER TABLE with_format ADD b INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO with_format VALUES (1, 1); + | --- + | - row_count: 1 + | ... +DROP TABLE with_format; + | --- + | - row_count: 1 + | ... + +-- +-- Add multiple columns (with a constraint) inside a transaction. +-- +CREATE TABLE t2 (a INT PRIMARY KEY) + | --- + | - row_count: 1 + | ... +\set language lua + | --- + | - true + | ... +box.begin() \ +box.execute('ALTER TABLE t2 ADD b INT') \ +box.execute('ALTER TABLE t2 ADD c INT UNIQUE') \ +box.commit() + | --- + | ... +\set language sql + | --- + | - true + | ... +INSERT INTO t2 VALUES (1, 1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO t2 VALUES (2, 1, 1); + | --- + | - null + | - Duplicate key exists in unique index 'unique_unnamed_T2_2' in space 'T2' + | ... +SELECT * FROM t2; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: integer + | - name: C + | type: integer + | rows: + | - [1, 1, 1] + | ... +DROP TABLE t2; + | --- + | - row_count: 1 + | ... diff --git a/test/sql/add-column.test.sql b/test/sql/add-column.test.sql new file mode 100644 index 000000000..f8ab3f756 --- /dev/null +++ b/test/sql/add-column.test.sql @@ -0,0 +1,167 @@ +-- +-- gh-3075: Check statement. +-- +CREATE TABLE t1 (a INT PRIMARY KEY); + +-- +-- COLUMN keyword is optional. Check it here, but omit it below. +-- +ALTER TABLE t1 ADD COLUMN b INT; + +-- +-- A column with the same name already exists. +-- +ALTER TABLE t1 ADD b SCALAR; + +-- +-- Can't add column to a view. +-- +CREATE VIEW v AS SELECT * FROM t1; +ALTER TABLE v ADD b INT; +DROP VIEW v; + +-- +-- Check PRIMARY KEY constraint works with an added column. +-- +CREATE TABLE pk_check (a INT CONSTRAINT pk PRIMARY KEY); +ALTER TABLE pk_check DROP CONSTRAINT pk; +ALTER TABLE pk_check ADD b INT PRIMARY KEY; +INSERT INTO pk_check VALUES (1, 1); +INSERT INTO pk_check VALUES (1, 1); +DROP TABLE pk_check; + +-- +-- Check UNIQUE constraint works with an added column. +-- +CREATE TABLE unique_check (a INT PRIMARY KEY); +ALTER TABLE unique_check ADD b INT UNIQUE; +INSERT INTO unique_check VALUES (1, 1); +INSERT INTO unique_check VALUES (2, 1); +DROP TABLE unique_check; + +-- +-- Check CHECK constraint works with an added column. +-- +CREATE TABLE ck_check (a INT PRIMARY KEY); +ALTER TABLE ck_check ADD b INT CHECK (b > 0); +INSERT INTO ck_check VALUES (1, 0); +DROP TABLE ck_check; + +-- +-- Check FOREIGN KEY constraint works with an added column. +-- +CREATE TABLE fk_check (a INT PRIMARY KEY); +ALTER TABLE fk_check ADD b INT REFERENCES t1(a); +INSERT INTO fk_check VALUES (0, 1); +INSERT INTO fk_check VALUES (2, 0); +INSERT INTO fk_check VALUES (2, 1); +DROP TABLE fk_check; +DROP TABLE t1; +-- +-- Check FOREIGN KEY (self-referenced) constraint works with an +-- added column. +-- +CREATE TABLE self (id INT PRIMARY KEY AUTOINCREMENT, a INT UNIQUE) +ALTER TABLE self ADD b INT REFERENCES self(a) +INSERT INTO self(a,b) VALUES(1, 1); +UPDATE self SET a = 2, b = 2; +UPDATE self SET b = 3; +UPDATE self SET a = 3; +DROP TABLE self; + +-- +-- Check AUTOINCREMENT works with an added column. +-- +CREATE TABLE autoinc_check (a INT CONSTRAINT pk PRIMARY KEY); +ALTER TABLE autoinc_check DROP CONSTRAINT pk; +ALTER TABLE autoinc_check ADD b INT PRIMARY KEY AUTOINCREMENT; +INSERT INTO autoinc_check(a) VALUES(1); +INSERT INTO autoinc_check(a) VALUES(1); +TRUNCATE TABLE autoinc_check; + +-- +-- Can't add second column with AUTOINCREMENT. +-- +ALTER TABLE autoinc_check ADD c INT AUTOINCREMENT; +DROP TABLE autoinc_check; + +-- +-- Check COLLATE clause works with an added column. +-- +CREATE TABLE collate_check (a INT PRIMARY KEY); +ALTER TABLE collate_check ADD b TEXT COLLATE "unicode_ci"; +INSERT INTO collate_check VALUES (1, 'a'); +INSERT INTO collate_check VALUES (2, 'A'); +SELECT * FROM collate_check WHERE b LIKE 'a'; +DROP TABLE collate_check; + +-- +-- Check DEFAULT clause works with an added column. +-- +CREATE TABLE default_check (a INT PRIMARY KEY); +ALTER TABLE default_check ADD b TEXT DEFAULT ('a'); +INSERT INTO default_check(a) VALUES (1); +SELECT * FROM default_check; +DROP TABLE default_check; + +-- +-- Check NULL constraint works with an added column. +-- +CREATE TABLE null_check (a INT PRIMARY KEY); +ALTER TABLE null_check ADD b TEXT NULL; +INSERT INTO null_check(a) VALUES (1); +DROP TABLE null_check; + +-- +-- Check NOT NULL constraint works with an added column. +-- +CREATE TABLE notnull_check (a INT PRIMARY KEY); +ALTER TABLE notnull_check ADD b TEXT NOT NULL; +INSERT INTO notnull_check(a) VALUES (1); +DROP TABLE notnull_check; + +-- +-- Can't add a column with DEAFULT or NULL to a non-empty space. +-- This ability isn't implemented yet. +-- +CREATE TABLE non_empty (a INT PRIMARY KEY); +INSERT INTO non_empty VALUES (1); +ALTER TABLE non_empty ADD b INT NULL; +ALTER TABLE non_empty ADD b INT DEFAULT (1); +DROP TABLE non_empty; + +-- +-- Add to a no-SQL adjusted space without format. +-- +\set language lua +_ = box.schema.space.create('WITHOUT_FORMAT') +\set language sql +ALTER TABLE without_format ADD a INT PRIMARY KEY; +INSERT INTO without_format VALUES (1); +DROP TABLE without_format; + +-- +-- Add to a no-SQL adjusted space with format. +-- +\set language lua +with_format = box.schema.space.create('WITH_FORMAT') +with_format:format{{name = 'A', type = 'unsigned'}} +\set language sql +ALTER TABLE with_format ADD b INT PRIMARY KEY; +INSERT INTO with_format VALUES (1, 1); +DROP TABLE with_format; + +-- +-- Add multiple columns (with a constraint) inside a transaction. +-- +CREATE TABLE t2 (a INT PRIMARY KEY) +\set language lua +box.begin() \ +box.execute('ALTER TABLE t2 ADD b INT') \ +box.execute('ALTER TABLE t2 ADD c INT UNIQUE') \ +box.commit() +\set language sql +INSERT INTO t2 VALUES (1, 1, 1); +INSERT INTO t2 VALUES (2, 1, 1); +SELECT * FROM t2; +DROP TABLE t2; diff --git a/test/sql/checks.result b/test/sql/checks.result index 7b18e5d6b..513ed1b62 100644 --- a/test/sql/checks.result +++ b/test/sql/checks.result @@ -856,6 +856,26 @@ box.execute("DROP TABLE t6") --- - row_count: 1 ... +-- +-- gh-3075: Check the auto naming of CHECK constraints in +-- . +-- +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY CHECK (a > 0))") +--- +- row_count: 1 +... +box.execute("ALTER TABLE check_naming ADD b INT CHECK (b > 0)") +--- +- row_count: 0 +... +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"ck_unnamed_CHECK_NAMING_2\"") +--- +- row_count: 1 +... +box.execute("DROP TABLE check_naming") +--- +- row_count: 1 +... test_run:cmd("clear filter") --- - true diff --git a/test/sql/checks.test.lua b/test/sql/checks.test.lua index 301f8ea69..a79131466 100644 --- a/test/sql/checks.test.lua +++ b/test/sql/checks.test.lua @@ -280,4 +280,13 @@ box.func.MYFUNC:drop() box.execute("INSERT INTO t6 VALUES(11);"); box.execute("DROP TABLE t6") +-- +-- gh-3075: Check the auto naming of CHECK constraints in +-- . +-- +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY CHECK (a > 0))") +box.execute("ALTER TABLE check_naming ADD b INT CHECK (b > 0)") +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"ck_unnamed_CHECK_NAMING_2\"") +box.execute("DROP TABLE check_naming") + test_run:cmd("clear filter") diff --git a/test/sql/foreign-keys.result b/test/sql/foreign-keys.result index 33689a06e..de2a0c512 100644 --- a/test/sql/foreign-keys.result +++ b/test/sql/foreign-keys.result @@ -499,5 +499,33 @@ box.space.S:drop() box.space.T:drop() --- ... +-- +-- gh-3075: Check the auto naming of FOREIGN KEY constraints in +-- . +-- +box.execute("CREATE TABLE t1 (a INT PRIMARY KEY)") +--- +- row_count: 1 +... +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY REFERENCES t1(a))") +--- +- row_count: 1 +... +box.execute("ALTER TABLE check_naming ADD b INT REFERENCES t1(a)") +--- +- row_count: 0 +... +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"fk_unnamed_CHECK_NAMING_2\"") +--- +- row_count: 1 +... +box.execute("DROP TABLE check_naming") +--- +- row_count: 1 +... +box.execute("DROP TABLE t1") +--- +- row_count: 1 +... --- Clean-up SQL DD hash. -test_run:cmd('restart server default with cleanup=1') diff --git a/test/sql/foreign-keys.test.lua b/test/sql/foreign-keys.test.lua index d2dd88d28..29918c5d4 100644 --- a/test/sql/foreign-keys.test.lua +++ b/test/sql/foreign-keys.test.lua @@ -209,5 +209,16 @@ box.space.T:select() box.space.S:drop() box.space.T:drop() +-- +-- gh-3075: Check the auto naming of FOREIGN KEY constraints in +-- . +-- +box.execute("CREATE TABLE t1 (a INT PRIMARY KEY)") +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY REFERENCES t1(a))") +box.execute("ALTER TABLE check_naming ADD b INT REFERENCES t1(a)") +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"fk_unnamed_CHECK_NAMING_2\"") +box.execute("DROP TABLE check_naming") +box.execute("DROP TABLE t1") + --- Clean-up SQL DD hash. -test_run:cmd('restart server default with cleanup=1') -- 2.21.0 (Apple Git-122) From roman.habibov at tarantool.org Tue Aug 11 03:34:00 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:34:00 +0300 Subject: [Tarantool-patches] [PATCH v2 2/2] sql: support column addition In-Reply-To: References: <20200403152752.8923-1-roman.habibov@tarantool.org> <20200403152752.8923-3-roman.habibov@tarantool.org> <5CF72787-A1F0-4C48-BA8F-08F02B6960F6@tarantool.org> Message-ID: <3FF95208-4C8E-4094-A005-7A50415A8651@tarantool.org> Hi! Thanks for the review. > On Jul 12, 2020, at 19:45, Vladislav Shpilevoy wrote: > > Hi! Thanks for the patch! > > See 33 comments below. > >> commit 82c448dc66f6233faeb40dda353652c2fd5a3d29 >> Author: Roman Khabibov >> Date: Thu Jan 2 19:06:14 2020 +0300 >> >> sql: support column addition >> >> Enable to add column to existing space with >> statement. Column definition can be >> supplemented with the four types of constraints, , >> clauses and <[NOT] NULL>, AUTOINCREMENT. >> >> Closes #2349, #3075 >> >> @TarantoolBot document >> Title: Add columns to existing tables in SQL >> Now, it is possible to add columns to existing empty spaces using >> >> statement. The column definition is the same as in >> statement. >> >> For example: >> >> tarantool> box.execute([[CREATE TABLE test ( >> a INTEGER PRIMARY KEY >> );]]) >> --- >> - row_count: 1 >> ... >> >> tarantool> box.execute([[ALTER TABLE test ADD COLUMN >> b TEXT >> NOT NULL >> DEFAULT ('a') >> COLLATE "unicode_ci" >> ;]]) >> --- >> - row_count: 0 >> ... >> --- > > 1. The commit message is different from what I see on the branch. On the > branch there is no 'Closes #2349'. What is the most actual version? > > 2. The example from the doc request does not work: > > tarantool> box.execute([[CREATE TABLE test (a INTEGER PRIMARY KEY);]]) > --- > - row_count: 1 > ... > > tarantool> box.execute([[ALTER TABLE test ADD COLUMN b TEXT NOT NULL DEFAULT ('a') COLLATE "unicode_ci";]]) > --- > - null > - 'At line 1 at or near position 22: keyword ''COLUMN'' is reserved. Please use double > quotes if ''COLUMN'' is an identifier.' > ... > >> diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c >> index 486b6b30d..dea047241 100644 >> --- a/extra/mkkeywordhash.c >> +++ b/extra/mkkeywordhash.c >> @@ -76,7 +76,7 @@ static Keyword aKeywordTable[] = { >> { "CHECK", "TK_CHECK", true }, >> { "COLLATE", "TK_COLLATE", true }, >> { "COLUMN_NAME", "TK_COLUMN_NAME", true }, >> - { "COLUMN", "TK_STANDARD", true }, >> + { "COLUMN", "TK_COLUMN", true }, > > 3. This is how the same hunk looks on the branch: > > ==================== > diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c > index 006285622..51c8cbb18 100644 > --- a/extra/mkkeywordhash.c > +++ b/extra/mkkeywordhash.c > @@ -75,11 +75,7 @@ static Keyword aKeywordTable[] = { > { "CAST", "TK_CAST", false }, > { "CHECK", "TK_CHECK", true }, > { "COLLATE", "TK_COLLATE", true }, > - /* gh-3075: Reserved until ALTER ADD COLUMN is implemeneted. > - * Move it back to ALTER when done. > - */ > - /* { "COLUMN", "TK_COLUMNKW", true }, */ > - { "COLUMN", "TK_STANDARD", true }, > + { "COLUMN", "TK_COLUMN", true }, > { "COMMIT", "TK_COMMIT", true }, > { "CONFLICT", "TK_CONFLICT", false }, > { "CONSTRAINT", "TK_CONSTRAINT", true }, > ==================== > > It is very different. Why? > >> diff --git a/src/box/errcode.h b/src/box/errcode.h >> index d1e4d02a9..3e94bee7a 100644 >> --- a/src/box/errcode.h >> +++ b/src/box/errcode.h >> @@ -266,6 +266,8 @@ struct errcode_record { >> /*211 */_(ER_WRONG_QUERY_ID, "Prepared statement with id %u does not exist") \ >> /*212 */_(ER_SEQUENCE_NOT_STARTED, "Sequence '%s' is not started") \ >> /*213 */_(ER_NO_SUCH_SESSION_SETTING, "Session setting %s doesn't exist") \ >> + /*214 */_(ER_SQL_CANT_ADD_COLUMN_TO_VIEW, "Can't add column '%s'. '%s' is a view") \ >> + /*215 */_(ER_SQL_CANT_ADD_AUTOINC, "Can't add AUTOINCREMENT: the space '%s' already has one auto-incremented field") \ > > 4. The same here: > > ==================== > --- a/src/box/errcode.h > +++ b/src/box/errcode.h > @@ -264,6 +264,7 @@ struct errcode_record { > /*209 */_(ER_SESSION_SETTING_INVALID_VALUE, "Session setting %s expected a value of type %s") \ > /*210 */_(ER_SQL_PREPARE, "Failed to prepare SQL statement: %s") \ > /*211 */_(ER_WRONG_QUERY_ID, "Prepared statement with id %u does not exist") \ > + /*212 */_(ER_SQL_CANT_ADD_COLUMN_TO_VIEW, "Can't add column '%s' to the view '%s'") \ > > /* > * !IMPORTANT! Please follow instructions at start of the file > ==================== > > From this point I will review the email, not the branch. > Generally the patch still looks very raw. I hope this is mostly > because I couldn't review it properly locally. There was two branches on GitHub: romanhabibov/gh-3075-add-column and romanhabibov/gh-3075-add-column-v2. I dropped the first so we don't get confused anymore. >> diff --git a/src/box/sql/build.c b/src/box/sql/build.c >> index ac42fe842..45fb90d38 100644 >> --- a/src/box/sql/build.c >> +++ b/src/box/sql/build.c >> @@ -285,48 +285,112 @@ sql_field_retrieve(Parse *parser, struct space_def *space_def, uint32_t id) >> return field; >> } >> >> -/* >> - * Add a new column to the table currently being constructed. >> +/** >> + * Make shallow copy of @a space on region. >> * >> - * The parser calls this routine once for each column declaration >> - * in a CREATE TABLE statement. sqlStartTable() gets called >> - * first to get things going. Then this routine is called for each >> - * column. >> + * Function is used to add a new column to the existing space with >> + * . Copy info about indexes and >> + * definition to create constraints appeared in the statement. > > 5. I don't think I understood anything from the comment. Why is it needed > to create a copy (I remember why, a bit, but I mostly forgot it)? To reuse the existing code. This is the best way, I think. Otherwise, we will have to essentially rewrite the functions for creating constraints. -/* - * Add a new column to the table currently being constructed. +/** + * Make shallow copy of @a space on region. * - * The parser calls this routine once for each column declaration - * in a CREATE TABLE statement. sqlStartTable() gets called - * first to get things going. Then this routine is called for each - * column. + * Function is used to add a new column to an existing space with + * statement. Copy space def and index + * array to create constraints appeared in the statement. The + * index array copy will be modified by adding new elements to it. + * It is necessary, because the statement may contain several + * index definitions (constraints). */ +static struct space * +sql_shallow_space_copy(struct Parse *parse, struct space *space) >> */ >> +static struct space * >> +sql_shallow_space_copy(struct Parse *parse, struct space *space) >> +{ >> + assert(space->def != NULL); >> + struct space *ret = sql_ephemeral_space_new(parse, space->def->name); >> + if (ret == NULL) >> + return NULL; >> + ret->index_count = space->index_count; >> + ret->index_id_max = space->index_id_max; >> + uint32_t indexes_sz = sizeof(struct index *) * (ret->index_count); >> + ret->index = (struct index **) malloc(indexes_sz); > > 6. When is this array freed? In sql_create_column_end(struct Parse *parse) + /* + * Clean up array allocated in sql_shallow_space_copy(). + */ + free(space->index); >> + if (ret->index == NULL) { >> + diag_set(OutOfMemory, indexes_sz, "realloc", "ret->index"); >> + return NULL; >> + } >> + for (uint32_t i = 0; i < ret->index_count; i++) >> + ret->index[i] = space->index[i]; > > 7. What is the problem to make memcpy() instead of the cycle? No problem. See below. >> + memcpy(ret->def, space->def, sizeof(struct space_def)); >> + ret->def->opts.is_temporary = true; >> + ret->def->opts.is_ephemeral = true; >> + uint32_t fields_size = sizeof(struct field_def) * ret->def->field_count; >> + ret->def->fields = region_alloc(&parse->region, fields_size); > > 8. Use region_alloc_array. Otherwise it will crash in ASAN build. It > should be visible in CI. Ok. >> + if (ret->def->fields == NULL) { >> + diag_set(OutOfMemory, fields_size, "region_alloc", >> + "ret->def->fields"); > > 9. index array leaks here. Fixed. +static struct space * +sql_shallow_space_copy(struct Parse *parse, struct space *space) +{ + assert(space->def != NULL); + struct space *ret = sql_ephemeral_space_new(parse, space->def->name); + if (ret == NULL) + return NULL; + ret->index_count = space->index_count; + ret->index_id_max = space->index_id_max; + uint32_t indexes_sz = sizeof(struct index *) * (ret->index_count); + ret->index = (struct index **) malloc(indexes_sz); + if (ret->index == NULL) { + diag_set(OutOfMemory, indexes_sz, "realloc", "ret->index"); + return NULL; + } + memcpy(ret->index, space->index, indexes_sz); + memcpy(ret->def, space->def, sizeof(struct space_def)); + ret->def->opts.is_temporary = true; + ret->def->opts.is_ephemeral = true; + if (ret->def->field_count != 0) { + uint32_t fields_size = 0; + ret->def->fields = + region_alloc_array(&parse->region, + typeof(struct field_def), + ret->def->field_count, &fields_size); + if (ret->def->fields == NULL) { + diag_set(OutOfMemory, fields_size, "region_alloc", + "ret->def->fields"); + free(ret->index); + return NULL; + } + memcpy(ret->def->fields, space->def->fields, fields_size); + } + + return ret; +} >> + return NULL; >> + } >> + memcpy(ret->def->fields, space->def->fields, fields_size); >> + >> + return ret; >> +} >> + >> void >> -sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) >> +sql_create_column_start(struct Parse *parse) >> { >> - assert(type_def != NULL); >> - char *z; >> - sql *db = pParse->db; >> - if (pParse->create_table_def.new_space == NULL) >> - return; >> - struct space_def *def = pParse->create_table_def.new_space->def; >> + struct create_column_def *create_column_def = &parse->create_column_def; >> + struct alter_entity_def *alter_entity_def = >> + &create_column_def->base.base; >> + assert(alter_entity_def->entity_type == ENTITY_TYPE_COLUMN); >> + assert(alter_entity_def->alter_action == ALTER_ACTION_CREATE); >> + struct space *space = parse->create_table_def.new_space; >> + bool is_alter = space == NULL; >> + struct sql *db = parse->db; >> + if (is_alter) { >> + const char *space_name = >> + alter_entity_def->entity_name->a[0].zName; >> + space = space_by_name(space_name); >> + if (space == NULL) { >> + diag_set(ClientError, ER_NO_SUCH_SPACE, space_name); >> + goto tnt_error; >> + } >> + space = sql_shallow_space_copy(parse, space); >> + if (space == NULL) >> + goto tnt_error; >> + } >> + create_column_def->space = space; >> + struct space_def *def = space->def; >> + assert(def->opts.is_ephemeral); >> >> #if SQL_MAX_COLUMN >> if ((int)def->field_count + 1 > db->aLimit[SQL_LIMIT_COLUMN]) { >> diag_set(ClientError, ER_SQL_COLUMN_COUNT_MAX, def->name, >> def->field_count + 1, db->aLimit[SQL_LIMIT_COLUMN]); >> - pParse->is_aborted = true; >> - return; >> + goto tnt_error; > > 10. Was there a leak before your patch? Because of not > called sqlSrcListDelete(db, alter_entity_def->entity_name); No, because alter_entity_def is a part of create_column_def. create_column_def didn?t exist before my patch. >> } >> #endif >> + >> + struct region *region = &parse->region; >> + struct Token *name = &create_column_def->base.name; >> + char *column_name = >> + sql_normalized_name_region_new(region, name->z, name->n); >> + if (column_name == NULL) >> + goto tnt_error; >> + >> + if (parse->create_table_def.new_space == NULL && def->opts.is_view) { > > 11. You have is_alter flag for that. Yes. Fixed. >> + diag_set(ClientError, ER_SQL_CANT_ADD_COLUMN_TO_VIEW, >> + column_name, def->name); >> + goto tnt_error; >> + } >> + >> /* >> - * As sql_field_retrieve will allocate memory on region >> - * ensure that def is also temporal and would be dropped. >> + * Format can be set in Lua, then exact_field_count can be >> + * zero, but field_count is not. >> */ >> - assert(def->opts.is_ephemeral); >> - if (sql_field_retrieve(pParse, def, def->field_count) == NULL) >> - return; >> - struct region *region = &pParse->region; >> - z = sql_normalized_name_region_new(region, pName->z, pName->n); >> - if (z == NULL) { >> - pParse->is_aborted = true; >> + if (def->exact_field_count == 0) >> + def->exact_field_count = def->field_count; >> + if (sql_field_retrieve(parse, def, def->field_count) == NULL) >> return; >> + >> + for (uint32_t i = 0; i < def->field_count; i++) { >> + if (strcmp(column_name, def->fields[i].name) == 0) { >> + diag_set(ClientError, ER_SPACE_FIELD_IS_DUPLICATE, >> + column_name); >> + goto tnt_error; >> + } > > 12. I remember this code was deliberately removed, because the same check is > already done in box. Why did you return it? Sorry, removed. >> } >> struct field_def *column_def = &def->fields[def->field_count]; >> - memcpy(column_def, &field_def_default, sizeof(field_def_default)); >> - column_def->name = z; >> + memcpy(column_def, &field_def_default, sizeof(struct field_def)); > > 13. Unnecessary change of memcpy(). Yes. Returned old version. +sql_create_column_start(struct Parse *parse) { - assert(type_def != NULL); - char *z; - sql *db = pParse->db; - if (pParse->create_table_def.new_space == NULL) - return; - struct space_def *def = pParse->create_table_def.new_space->def; + struct create_column_def *create_column_def = &parse->create_column_def; + struct alter_entity_def *alter_entity_def = + &create_column_def->base.base; + assert(alter_entity_def->entity_type == ENTITY_TYPE_COLUMN); + assert(alter_entity_def->alter_action == ALTER_ACTION_CREATE); + struct space *space = parse->create_table_def.new_space; + bool is_alter = space == NULL; + struct sql *db = parse->db; + if (is_alter) { + const char *space_name = + alter_entity_def->entity_name->a[0].zName; + space = space_by_name(space_name); + if (space == NULL) { + diag_set(ClientError, ER_NO_SUCH_SPACE, space_name); + goto tnt_error; + } + space = sql_shallow_space_copy(parse, space); + if (space == NULL) + goto tnt_error; + } + create_column_def->space = space; + struct space_def *def = space->def; + assert(def->opts.is_ephemeral); #if SQL_MAX_COLUMN if ((int)def->field_count + 1 > db->aLimit[SQL_LIMIT_COLUMN]) { diag_set(ClientError, ER_SQL_COLUMN_COUNT_MAX, def->name, def->field_count + 1, db->aLimit[SQL_LIMIT_COLUMN]); - pParse->is_aborted = true; - return; + goto tnt_error; } #endif + + struct region *region = &parse->region; + struct Token *name = &create_column_def->base.name; + char *column_name = + sql_normalized_name_region_new(region, name->z, name->n); + if (column_name == NULL) + goto tnt_error; + + if (is_alter && def->opts.is_view) { + diag_set(ClientError, ER_SQL_CANT_ADD_COLUMN_TO_VIEW, + column_name, def->name); + goto tnt_error; + } + /* - * As sql_field_retrieve will allocate memory on region - * ensure that def is also temporal and would be dropped. + * Format can be set in Lua, then exact_field_count can be + * zero, but field_count is not. */ - assert(def->opts.is_ephemeral); - if (sql_field_retrieve(pParse, def, def->field_count) == NULL) + if (def->exact_field_count == 0) + def->exact_field_count = def->field_count; + if (sql_field_retrieve(parse, def, def->field_count) == NULL) return; - struct region *region = &pParse->region; - z = sql_normalized_name_region_new(region, pName->z, pName->n); - if (z == NULL) { - pParse->is_aborted = true; - return; - } + struct field_def *column_def = &def->fields[def->field_count]; memcpy(column_def, &field_def_default, sizeof(field_def_default)); - column_def->name = z; + column_def->name = column_name; /* * Marker ON_CONFLICT_ACTION_DEFAULT is used to detect * attempts to define NULL multiple time or to detect @@ -334,18 +399,86 @@ sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) */ column_def->nullable_action = ON_CONFLICT_ACTION_DEFAULT; column_def->is_nullable = true; - column_def->type = type_def->type; + column_def->type = create_column_def->type_def->type; def->field_count++; + + sqlSrcListDelete(db, alter_entity_def->entity_name); + return; +tnt_error: + parse->is_aborted = true; + sqlSrcListDelete(db, alter_entity_def->entity_name); +} + >> + column_def->name = column_name; >> /* >> * Marker ON_CONFLICT_ACTION_DEFAULT is used to detect >> * attempts to define NULL multiple time or to detect >> @@ -334,19 +398,100 @@ sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) >> */ >> column_def->nullable_action = ON_CONFLICT_ACTION_DEFAULT; >> column_def->is_nullable = true; >> - column_def->type = type_def->type; >> + column_def->type = create_column_def->type_def->type; >> def->field_count++; >> + >> + sqlSrcListDelete(db, alter_entity_def->entity_name); >> + return; >> +tnt_error: >> + parse->is_aborted = true; >> + sqlSrcListDelete(db, alter_entity_def->entity_name); >> +} >> + >> +static void >> +sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id); >> + >> +void >> +sql_create_column_end(struct Parse *parse) >> +{ >> + struct create_column_def *create_column_def = &parse->create_column_def; >> + struct space *space = parse->create_table_def.new_space; >> + bool is_alter = space == NULL; >> + space = create_column_def->space; >> + struct space_def *def = space->def; >> + if (is_alter) { >> + struct field_def *field = &def->fields[def->field_count - 1]; >> + if (field->nullable_action == ON_CONFLICT_ACTION_DEFAULT) { >> + if (create_column_def->is_pk) { >> + field->nullable_action = >> + ON_CONFLICT_ACTION_ABORT; >> + field->is_nullable = false; >> + } else { >> + field->nullable_action = >> + ON_CONFLICT_ACTION_NONE; >> + field->is_nullable = true; >> + } >> + } >> + /* >> + * Encode the format array and emit code to update _space. >> + */ >> + uint32_t table_stmt_sz = 0; >> + struct region *region = &parse->region; >> + char *table_stmt = sql_encode_table(region, def, >> + &table_stmt_sz); >> + char *raw = sqlDbMallocRaw(parse->db, table_stmt_sz); >> + if (table_stmt == NULL || raw == NULL) { >> + parse->is_aborted = true; >> + return; >> + } >> + memcpy(raw, table_stmt, table_stmt_sz); >> + >> + struct Vdbe *v = sqlGetVdbe(parse); >> + assert(v != NULL); >> + >> + struct space *system_space = space_by_id(BOX_SPACE_ID); >> + assert(system_space != NULL); >> + int cursor = parse->nTab++; >> + vdbe_emit_open_cursor(parse, cursor, 0, system_space); >> + sqlVdbeChangeP5(v, OPFLAG_SYSTEMSP); >> + >> + int key_reg = ++parse->nMem; >> + sqlVdbeAddOp2(v, OP_Integer, def->id, key_reg); >> + int addr = sqlVdbeAddOp4Int(v, OP_Found, cursor, 0, key_reg, 1); >> + sqlVdbeAddOp2(v, OP_Halt, -1, ON_CONFLICT_ACTION_ABORT); >> + sqlVdbeJumpHere(v, addr); >> + >> + int tuple_reg = sqlGetTempRange(parse, box_space_field_MAX + 1); >> + for (int i = 0; i < box_space_field_MAX - 1; ++i) >> + sqlVdbeAddOp3(v, OP_Column, cursor, i, tuple_reg + i); >> + sqlVdbeAddOp1(v, OP_Close, cursor); >> + >> + sqlVdbeAddOp2(v, OP_Integer, def->field_count, tuple_reg + 4); >> + sqlVdbeAddOp4(v, OP_Blob, table_stmt_sz, tuple_reg + 6, >> + SQL_SUBTYPE_MSGPACK, raw, P4_DYNAMIC); >> + sqlVdbeAddOp3(v, OP_MakeRecord, tuple_reg, box_space_field_MAX, >> + tuple_reg + box_space_field_MAX); >> + sqlVdbeAddOp4(v, OP_IdxReplace, tuple_reg + box_space_field_MAX, >> + 0, 0, (char *) system_space, P4_SPACEPTR); >> + sql_vdbe_create_constraints(parse, key_reg); >> + } >> + >> + memset(create_column_def, 0, sizeof(struct create_column_def)); >> + create_column_def->nullable_action = ON_CONFLICT_ACTION_DEFAULT; > > 14. Why do you touch column creation def after its usage? Removed. +void +sql_create_column_end(struct Parse *parse) +{ + struct space *space = parse->create_column_def.space; + assert(space != NULL); + struct space_def *def = space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (field->nullable_action == ON_CONFLICT_ACTION_DEFAULT) { + field->nullable_action = ON_CONFLICT_ACTION_NONE; + field->is_nullable = true; + } + /* + * Encode the format array and emit code to update _space. + */ + uint32_t table_stmt_sz = 0; + struct region *region = &parse->region; + char *table_stmt = sql_encode_table(region, def, &table_stmt_sz); + char *raw = sqlDbMallocRaw(parse->db, table_stmt_sz); + if (table_stmt == NULL || raw == NULL) { + parse->is_aborted = true; + return; + } + memcpy(raw, table_stmt, table_stmt_sz); + + struct Vdbe *v = sqlGetVdbe(parse); + assert(v != NULL); + + struct space *system_space = space_by_id(BOX_SPACE_ID); + assert(system_space != NULL); + int cursor = parse->nTab++; + vdbe_emit_open_cursor(parse, cursor, 0, system_space); + sqlVdbeChangeP5(v, OPFLAG_SYSTEMSP); + + int key_reg = ++parse->nMem; + sqlVdbeAddOp2(v, OP_Integer, def->id, key_reg); + int addr = sqlVdbeAddOp4Int(v, OP_Found, cursor, 0, key_reg, 1); + sqlVdbeAddOp2(v, OP_Halt, -1, ON_CONFLICT_ACTION_ABORT); + sqlVdbeJumpHere(v, addr); + + int tuple_reg = sqlGetTempRange(parse, box_space_field_MAX + 1); + for (int i = 0; i < box_space_field_MAX - 1; ++i) + sqlVdbeAddOp3(v, OP_Column, cursor, i, tuple_reg + i); + sqlVdbeAddOp1(v, OP_Close, cursor); + + sqlVdbeAddOp2(v, OP_Integer, def->field_count, tuple_reg + 4); + sqlVdbeAddOp4(v, OP_Blob, table_stmt_sz, tuple_reg + 6, + SQL_SUBTYPE_MSGPACK, raw, P4_DYNAMIC); + sqlVdbeAddOp3(v, OP_MakeRecord, tuple_reg, box_space_field_MAX, + tuple_reg + box_space_field_MAX); + sqlVdbeAddOp4(v, OP_IdxReplace, tuple_reg + box_space_field_MAX, 0, 0, + (char *) system_space, P4_SPACEPTR); + sql_vdbe_create_constraints(parse, key_reg); + + /* + * Clean up array allocated in sql_shallow_space_copy(). + */ + free(space->index); } >> } >> >> void >> sql_column_add_nullable_action(struct Parse *parser, >> enum on_conflict_action nullable_action) >> { >> - struct space *space = parser->create_table_def.new_space; >> - if (space == NULL || NEVER(space->def->field_count < 1)) >> + struct space_def *def = NULL; >> + struct field_def *field = NULL; >> + struct space *space = parser->create_column_def.space; >> + assert (space != NULL); > > 15. Please, don't put whitespace after macro/function name and its arguments. Sorry. Fixed. >> + def = space->def; >> + if (NEVER(def->field_count < 1)) >> return; >> - struct space_def *def = space->def; >> - struct field_def *field = &def->fields[def->field_count - 1]; >> + field = &def->fields[def->field_count - 1]; > > 16. Why did you do that change about struct field_def? It seems not to > be needed. Yes. void sql_column_add_nullable_action(struct Parse *parser, enum on_conflict_action nullable_action) { - struct space *space = parser->create_table_def.new_space; - if (space == NULL || NEVER(space->def->field_count < 1)) + assert(parser->create_column_def.space != NULL); + struct space_def *def = parser->create_column_def.space->def; + if (NEVER(def->field_count < 1)) return; - struct space_def *def = space->def; struct field_def *field = &def->fields[def->field_count - 1]; if (field->nullable_action != ON_CONFLICT_ACTION_DEFAULT && nullable_action != field->nullable_action) { @@ -364,51 +497,42 @@ sql_column_add_nullable_action(struct Parse *parser, } > >> if (field->nullable_action != ON_CONFLICT_ACTION_DEFAULT && >> nullable_action != field->nullable_action) { >> /* Prevent defining nullable_action many times. */ >> @@ -364,51 +509,46 @@ sql_column_add_nullable_action(struct Parse *parser, >> } >> >> /* >> - * The expression is the default value for the most recently added column >> - * of the table currently under construction. >> + * The expression is the default value for the most recently added >> + * column. >> * >> * Default value expressions must be constant. Raise an exception if this >> * is not the case. >> * >> * This routine is called by the parser while in the middle of >> - * parsing a CREATE TABLE statement. >> + * parsing a or a >> + * statement. >> */ >> void >> sqlAddDefaultValue(Parse * pParse, ExprSpan * pSpan) >> { >> sql *db = pParse->db; >> - struct space *p = pParse->create_table_def.new_space; >> - if (p != NULL) { >> - assert(p->def->opts.is_ephemeral); >> - struct space_def *def = p->def; >> - if (!sqlExprIsConstantOrFunction >> - (pSpan->pExpr, db->init.busy)) { >> - const char *column_name = >> - def->fields[def->field_count - 1].name; >> - diag_set(ClientError, ER_CREATE_SPACE, def->name, >> - tt_sprintf("default value of column '%s' is "\ >> - "not constant", column_name)); >> + struct space_def *def = NULL; >> + struct field_def *field = NULL; >> + struct space *space = pParse->create_column_def.space; >> + assert (space != NULL); >> + def = space->def; > > 17. Why can't you declare and initialize space_def in one line? The same > for field. Fixed. void sqlAddDefaultValue(Parse * pParse, ExprSpan * pSpan) { sql *db = pParse->db; - struct space *p = pParse->create_table_def.new_space; - if (p != NULL) { - assert(p->def->opts.is_ephemeral); - struct space_def *def = p->def; - if (!sqlExprIsConstantOrFunction - (pSpan->pExpr, db->init.busy)) { - const char *column_name = - def->fields[def->field_count - 1].name; - diag_set(ClientError, ER_CREATE_SPACE, def->name, - tt_sprintf("default value of column '%s' is "\ - "not constant", column_name)); + assert(pParse->create_column_def.space != NULL); + struct space_def *def = pParse->create_column_def.space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (!sqlExprIsConstantOrFunction(pSpan->pExpr, db->init.busy)) { + diag_set(ClientError, ER_CREATE_SPACE, def->name, + tt_sprintf("default value of column '%s' is not " + "constant", field->name)); + pParse->is_aborted = true; + } else { + struct region *region = &pParse->region; + uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); + field->default_value = region_alloc(region, default_length + 1); + if (field->default_value == NULL) { + diag_set(OutOfMemory, default_length + 1, + "region_alloc", "field->default_value"); pParse->is_aborted = true; - } else { - assert(def != NULL); - struct field_def *field = - &def->fields[def->field_count - 1]; - struct region *region = &pParse->region; - uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); - field->default_value = region_alloc(region, - default_length + 1); - if (field->default_value == NULL) { - diag_set(OutOfMemory, default_length + 1, - "region_alloc", - "field->default_value"); - pParse->is_aborted = true; - return; - } - strncpy(field->default_value, pSpan->zStart, - default_length); - field->default_value[default_length] = '\0'; + goto add_default_value_exit; } + strncpy(field->default_value, pSpan->zStart, default_length); + field->default_value[default_length] = '\0'; } +add_default_value_exit: sql_expr_delete(db, pSpan->pExpr, false); } >> + field = &def->fields[def->field_count - 1]; >> + if (!sqlExprIsConstantOrFunction(pSpan->pExpr, db->init.busy)) { >> + diag_set(ClientError, ER_CREATE_SPACE, def->name, >> + tt_sprintf("default value of column '%s' is not " >> + "constant", field->name)); >> + pParse->is_aborted = true; >> + } else { >> + assert(def != NULL); >> + struct region *region = &pParse->region; >> + uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); >> + field->default_value = region_alloc(region, default_length + 1); >> + if (field->default_value == NULL) { >> + diag_set(OutOfMemory, default_length + 1, >> + "region_alloc", "field->default_value"); >> pParse->is_aborted = true; >> - } else { >> - assert(def != NULL); >> - struct field_def *field = >> - &def->fields[def->field_count - 1]; >> - struct region *region = &pParse->region; >> - uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); >> - field->default_value = region_alloc(region, >> - default_length + 1); >> - if (field->default_value == NULL) { >> - diag_set(OutOfMemory, default_length + 1, >> - "region_alloc", >> - "field->default_value"); >> - pParse->is_aborted = true; >> - return; >> - } >> - strncpy(field->default_value, pSpan->zStart, >> - default_length); >> - field->default_value[default_length] = '\0'; >> + goto add_default_value_exit; >> } >> + strncpy(field->default_value, pSpan->zStart, default_length); >> + field->default_value[default_length] = '\0'; >> } >> +add_default_value_exit: >> sql_expr_delete(db, pSpan->pExpr, false); >> } >> >> @@ -447,6 +587,8 @@ sqlAddPrimaryKey(struct Parse *pParse) >> int nTerm; >> struct ExprList *pList = pParse->create_index_def.cols; >> struct space *space = pParse->create_table_def.new_space; >> + if (space == NULL) >> + space = pParse->create_column_def.space; >> if (space == NULL) >> goto primary_key_exit; >> if (sql_space_primary_key(space) != NULL) { >> @@ -574,8 +716,10 @@ sql_create_check_contraint(struct Parse *parser) >> (struct alter_entity_def *) create_ck_def; >> assert(alter_def->entity_type == ENTITY_TYPE_CK); >> (void) alter_def; >> - struct space *space = parser->create_table_def.new_space; >> - bool is_alter = space == NULL; >> + struct space *space = parser->create_column_def.space; >> + if (space == NULL) >> + space = parser->create_table_def.new_space; >> + bool is_alter_add_constr = space == NULL; > > 18. Why is it called 'is_alter' in other functions, but 'is_alter_add_constr' > here? If I added ?is_alter_add_constr?, it means that we should distinguish current statements: or (maybe ). It?s important for branching within constraint creation functions (index, ck, fk). >> >> /* Prepare payload for ck constraint definition. */ >> struct region *region = &parser->region; >> @@ -589,9 +733,18 @@ sql_create_check_contraint(struct Parse *parser) >> return; >> } >> } else { >> - assert(! is_alter); >> - uint32_t ck_idx = ++parser->create_table_def.check_count; >> - name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, ck_idx); >> + assert(!is_alter_add_constr); >> + uint32_t idx = ++parser->check_count; >> + if (parser->create_table_def.new_space == NULL) { >> + struct space *original_space = >> + space_by_name(space->def->name); >> + assert(original_space != NULL); >> + struct rlist *checks = &original_space->ck_constraint; >> + struct ck_constraint *ck; >> + rlist_foreach_entry(ck, checks, link) >> + idx++; >> + } >> + name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, idx); >> } >> size_t name_len = strlen(name); >> >> @@ -634,9 +787,9 @@ sql_create_check_contraint(struct Parse *parser) >> trim_space_snprintf(ck_def->expr_str, expr_str, expr_str_len); >> memcpy(ck_def->name, name, name_len); >> ck_def->name[name_len] = '\0'; >> - if (is_alter) { >> + if (is_alter_add_constr) { >> const char *space_name = alter_def->entity_name->a[0].zName; >> - struct space *space = space_by_name(space_name); >> + space = space_by_name(space_name); > > 19. Why this change? Dropped. > >> if (space == NULL) { >> diag_set(ClientError, ER_NO_SUCH_SPACE, space_name); >> parser->is_aborted = true; >> @@ -652,8 +805,7 @@ sql_create_check_contraint(struct Parse *parser) >> sqlVdbeCountChanges(v); >> sqlVdbeChangeP5(v, OPFLAG_NCHANGE); >> } else { >> - rlist_add_entry(&parser->create_table_def.new_check, ck_parse, >> - link); >> + rlist_add_entry(&parser->checks, ck_parse, link); >> } >> } >> >> @@ -664,18 +816,19 @@ sql_create_check_contraint(struct Parse *parser) >> void >> sqlAddCollateType(Parse * pParse, Token * pToken) >> { >> - struct space *space = pParse->create_table_def.new_space; >> - if (space == NULL) >> - return; >> + struct space *space = pParse->create_column_def.space; >> + uint32_t *coll_id = NULL; >> + assert(space != NULL); >> uint32_t i = space->def->field_count - 1; >> + coll_id = &space->def->fields[i].coll_id; > > 20. Why not declare and initialize coll_id in one line? Done. void sqlAddCollateType(Parse * pParse, Token * pToken) { - struct space *space = pParse->create_table_def.new_space; - if (space == NULL) - return; + struct space *space = pParse->create_column_def.space; + assert(space != NULL); uint32_t i = space->def->field_count - 1; sql *db = pParse->db; char *coll_name = sql_name_from_token(db, pToken); >> sql *db = pParse->db; >> char *coll_name = sql_name_from_token(db, pToken); >> if (coll_name == NULL) { >> pParse->is_aborted = true; >> return; >> } >> - uint32_t *coll_id = &space->def->fields[i].coll_id; >> - if (sql_get_coll_seq(pParse, coll_name, coll_id) != NULL) { >> + if (sql_get_coll_seq(pParse, coll_name, coll_id) != NULL && >> + space != NULL) { > > 21. You have assert(space != NULL) above. Why do you need this check again? Removed. >> /* If the column is declared as " PRIMARY KEY COLLATE ", >> * then an index may have been created on this column before the >> * collation type was added. Correct this if it is the case. >> @@ -692,6 +845,7 @@ sqlAddCollateType(Parse * pParse, Token * pToken) >> sqlDbFree(db, coll_name); >> } >> >> + > > 22. ??? Oops. Removed. >> struct coll * >> sql_column_collation(struct space_def *def, uint32_t column, uint32_t *coll_id) >> { >> @@ -1147,6 +1304,105 @@ resolve_link(struct Parse *parse_context, const struct space_def *def, >> return -1; >> } >> >> +/** >> + * Emit code to create sequences, indexes, check and foreign >> + * constraints appeared in or >> + * statement. >> + */ >> +static void >> +sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) > > 23. Such huge code movements better do in a separate commit. I can't > tell whether you changed anything here, or just moved the code. Done. See commit in the new version of the patch set. >> +{ >> + assert(reg_space_id != 0); >> + struct space *space = parse->create_table_def.new_space; >> + bool is_alter = space == NULL; >> + uint32_t i = 0; >> + if (is_alter) { >> + space = parse->create_column_def.space; >> + i = space_by_name(space->def->name)->index_count; >> + } >> + assert(space != NULL); >> + for (; i < space->index_count; ++i) { >> + struct index *idx = space->index[i]; >> + vdbe_emit_create_index(parse, space->def, idx->def, >> + reg_space_id, idx->def->iid); >> + } >> + >> + /* >> + * Check to see if we need to create an _sequence table >> + * for keeping track of autoincrement keys. >> + */ >> + if (parse->has_autoinc) { >> + /* Do an insertion into _sequence. */ >> + int reg_seq_id = ++parse->nMem; >> + struct Vdbe *v = sqlGetVdbe(parse); >> + assert(v != NULL); >> + sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); >> + int reg_seq_record = >> + emitNewSysSequenceRecord(parse, reg_seq_id, >> + space->def->name); >> + const char *error_msg = >> + tt_sprintf(tnt_errcode_desc(ER_SQL_CANT_ADD_AUTOINC), >> + space->def->name); >> + if (vdbe_emit_halt_with_presence_test(parse, >> + BOX_SEQUENCE_ID, 2, >> + reg_seq_record + 3, 1, >> + ER_SQL_CANT_ADD_AUTOINC, >> + error_msg, false, >> + OP_NoConflict) != 0) >> + return; >> + sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_record); >> + /* Do an insertion into _space_sequence. */ >> + int reg_space_seq_record = >> + emitNewSysSpaceSequenceRecord(parse, reg_space_id, >> + reg_seq_id); >> + sqlVdbeAddOp2(v, OP_SInsert, BOX_SPACE_SEQUENCE_ID, >> + reg_space_seq_record); >> + } >> + >> + /* Code creation of FK constraints, if any. */ >> + struct fk_constraint_parse *fk_parse; >> + rlist_foreach_entry(fk_parse, &parse->fkeys, link) { >> + struct fk_constraint_def *fk_def = fk_parse->fk_def; >> + if (fk_parse->selfref_cols != NULL) { >> + struct ExprList *cols = fk_parse->selfref_cols; >> + for (uint32_t i = 0; i < fk_def->field_count; ++i) { >> + if (resolve_link(parse, space->def, >> + cols->a[i].zName, >> + &fk_def->links[i].parent_field, >> + fk_def->name) != 0) >> + return; >> + } >> + fk_def->parent_id = reg_space_id; >> + } else if (fk_parse->is_self_referenced) { >> + struct key_def *pk_key_def = >> + sql_space_primary_key(space)->def->key_def; >> + if (pk_key_def->part_count != fk_def->field_count) { >> + diag_set(ClientError, ER_CREATE_FK_CONSTRAINT, >> + fk_def->name, "number of columns in "\ >> + "foreign key does not match the "\ >> + "number of columns in the primary "\ >> + "index of referenced table"); >> + parse->is_aborted = true; >> + return; >> + } >> + for (uint32_t i = 0; i < fk_def->field_count; ++i) { >> + fk_def->links[i].parent_field = >> + pk_key_def->parts[i].fieldno; >> + } >> + fk_def->parent_id = reg_space_id; >> + } >> + fk_def->child_id = reg_space_id; >> + vdbe_emit_fk_constraint_create(parse, fk_def, space->def->name); >> + } >> + >> + /* Code creation of CK constraints, if any. */ >> + struct ck_constraint_parse *ck_parse; >> + rlist_foreach_entry(ck_parse, &parse->checks, link) { >> + vdbe_emit_ck_constraint_create(parse, ck_parse->ck_def, >> + reg_space_id, space->def->name); >> + } >> +} >> + >> /* >> * This routine is called to report the final ")" that terminates >> * a CREATE TABLE statement. >> @@ -1213,73 +1469,7 @@ sqlEndTable(struct Parse *pParse) >> >> int reg_space_id = getNewSpaceId(pParse); >> vdbe_emit_space_create(pParse, reg_space_id, name_reg, new_space); >> - for (uint32_t i = 0; i < new_space->index_count; ++i) { >> - struct index *idx = new_space->index[i]; >> - vdbe_emit_create_index(pParse, new_space->def, idx->def, >> - reg_space_id, idx->def->iid); >> - } >> - >> - /* >> - * Check to see if we need to create an _sequence table >> - * for keeping track of autoincrement keys. >> - */ >> - if (pParse->create_table_def.has_autoinc) { >> - assert(reg_space_id != 0); >> - /* Do an insertion into _sequence. */ >> - int reg_seq_id = ++pParse->nMem; >> - sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); >> - int reg_seq_record = >> - emitNewSysSequenceRecord(pParse, reg_seq_id, >> - new_space->def->name); >> - sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_record); >> - /* Do an insertion into _space_sequence. */ >> - int reg_space_seq_record = >> - emitNewSysSpaceSequenceRecord(pParse, reg_space_id, >> - reg_seq_id); >> - sqlVdbeAddOp2(v, OP_SInsert, BOX_SPACE_SEQUENCE_ID, >> - reg_space_seq_record); >> - } >> - /* Code creation of FK constraints, if any. */ >> - struct fk_constraint_parse *fk_parse; >> - rlist_foreach_entry(fk_parse, &pParse->create_table_def.new_fkey, >> - link) { >> - struct fk_constraint_def *fk_def = fk_parse->fk_def; >> - if (fk_parse->selfref_cols != NULL) { >> - struct ExprList *cols = fk_parse->selfref_cols; >> - for (uint32_t i = 0; i < fk_def->field_count; ++i) { >> - if (resolve_link(pParse, new_space->def, >> - cols->a[i].zName, >> - &fk_def->links[i].parent_field, >> - fk_def->name) != 0) >> - return; >> - } >> - fk_def->parent_id = reg_space_id; >> - } else if (fk_parse->is_self_referenced) { >> - struct index *pk = sql_space_primary_key(new_space); >> - if (pk->def->key_def->part_count != fk_def->field_count) { >> - diag_set(ClientError, ER_CREATE_FK_CONSTRAINT, >> - fk_def->name, "number of columns in "\ >> - "foreign key does not match the "\ >> - "number of columns in the primary "\ >> - "index of referenced table"); >> - pParse->is_aborted = true; >> - return; >> - } >> - for (uint32_t i = 0; i < fk_def->field_count; ++i) { >> - fk_def->links[i].parent_field = >> - pk->def->key_def->parts[i].fieldno; >> - } >> - fk_def->parent_id = reg_space_id; >> - } >> - fk_def->child_id = reg_space_id; >> - vdbe_emit_fk_constraint_create(pParse, fk_def, space_name_copy); >> - } >> - struct ck_constraint_parse *ck_parse; >> - rlist_foreach_entry(ck_parse, &pParse->create_table_def.new_check, >> - link) { >> - vdbe_emit_ck_constraint_create(pParse, ck_parse->ck_def, >> - reg_space_id, space_name_copy); >> - } >> + sql_vdbe_create_constraints(pParse, reg_space_id); >> } >> diff --git a/src/box/sql/parse.y b/src/box/sql/parse.y >> index 995875566..5904414a3 100644 >> --- a/src/box/sql/parse.y >> +++ b/src/box/sql/parse.y >> @@ -314,6 +320,7 @@ ccons ::= cconsname(N) PRIMARY KEY sortorder(Z). { >> create_index_def_init(&pParse->create_index_def, NULL, &N, NULL, >> SQL_INDEX_TYPE_CONSTRAINT_PK, Z, false); >> sqlAddPrimaryKey(pParse); >> + pParse->create_column_def.is_pk = true; > > 24. Why can't this be done in sqlAddPrimaryKey(pParse);? What if this > is CREATE TABLE, and there are many columns in PK? I removed this field from create_column_def. It turned out to be unnecessary. >> } >> ccons ::= cconsname(N) UNIQUE. { >> create_index_def_init(&pParse->create_index_def, NULL, &N, NULL, >> diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c >> index a5a258805..b31bac437 100644 >> --- a/src/box/sql/prepare.c >> +++ b/src/box/sql/prepare.c >> @@ -197,9 +197,13 @@ sql_parser_create(struct Parse *parser, struct sql *db, uint32_t sql_flags) >> { >> memset(parser, 0, sizeof(struct Parse)); >> parser->db = db; >> + parser->create_column_def.nullable_action = ON_CONFLICT_ACTION_DEFAULT; > > 25. Why isn't there a proper constructor for create_column_def structure? Dropped. >> parser->sql_flags = sql_flags; >> parser->line_count = 1; >> parser->line_pos = 1; >> + rlist_create(&parser->fkeys); >> + rlist_create(&parser->checks); > 26. Why did you merge these fields into struct Parse? They were specially > moved out of here for the purpose of some kind of isolation, into parse_def.h > structures. Because these fields are convenient to use right here, so as not to duplicate the code in build.c. These fields are used by two statements: CREATE TABLE and ALTER TABLE ADD COLUMN. >> + parser->has_autoinc = false; >> region_create(&parser->region, &cord()->slabc); >> } >> >> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h >> index aa6a470f8..3143ec521 100644 >> --- a/src/box/sql/sqlInt.h >> +++ b/src/box/sql/sqlInt.h >> @@ -2249,12 +2249,26 @@ struct Parse { >> struct enable_entity_def enable_entity_def; >> }; >> /** >> - * Table def is not part of union since information >> - * being held must survive till the end of parsing of >> - * whole CREATE TABLE statement (to pass it to >> - * sqlEndTable() function). >> + * Table def or column def is not part of union since >> + * information being held must survive till the end of >> + * parsing of whole or >> + * statement (to pass it to >> + * sqlEndTable() sql_create_column_end() function). >> */ >> struct create_table_def create_table_def; >> + struct create_column_def create_column_def; >> + /** >> + * FK and CK constraints appeared in a or >> + * a statement. >> + */ >> + struct rlist fkeys; >> + struct rlist checks; >> + uint32_t fkey_count; >> + uint32_t check_count; >> + /** True, if column to be created has . */ >> + bool has_autoinc; > > 27. What column? This is struct Parse, it is not a column. I know, but I haven't come up with anything better. diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index fa87e7bd2..32142a871 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -2251,22 +2251,26 @@ struct Parse { struct enable_entity_def enable_entity_def; }; /** - * Table def is not part of union since information - * being held must survive till the end of parsing of - * whole CREATE TABLE statement (to pass it to - * sqlEndTable() function). + * Table def or column def is not part of union since + * information being held must survive till the end of + * parsing of whole or + * statement (to pass it to + * sqlEndTable() sql_create_column_end() function). */ struct create_table_def create_table_def; + struct create_column_def create_column_def; /* - * FK and CK constraints appeared in a . + * FK and CK constraints appeared in a or + * an statement. */ struct rlist fkeys; struct rlist checks; uint32_t fkey_count; uint32_t check_count; /* - * True, if column within a statement to be - * created has . + * True, if column in a or an + * statement to be created has + * . */ bool has_autoinc; /* Id of field with . */ >> + /** Id of field with . */ >> + uint32_t autoinc_fieldno; >> bool initiateTTrans; /* Initiate Tarantool transaction */ >> /** If set - do not emit byte code at all, just parse. */ >> bool parse_only; >> @@ -2844,15 +2858,31 @@ struct space *sqlResultSetOfSelect(Parse *, Select *); >> >> struct space * >> sqlStartTable(Parse *, Token *); >> -void sqlAddColumn(Parse *, Token *, struct type_def *); >> + >> +/** >> + * Add new field to the format of ephemeral space in >> + * create_table_def. If it is create shallow copy of >> + * the existing space and add field to its format. >> + */ >> +void >> +sql_create_column_start(struct Parse *parse); >> + >> +/** >> + * Nullify create_column_def. If it is emit code to >> + * update entry in _space and code to create constraints (entries >> + * in _index, _ck_constraint, _fk_constraint) described with this >> + * column. >> + */ >> +void >> +sql_create_column_end(struct Parse *parse); >> >> /** >> * This routine is called by the parser while in the middle of >> - * parsing a CREATE TABLE statement. A "NOT NULL" constraint has >> - * been seen on a column. This routine sets the is_nullable flag >> - * on the column currently under construction. >> - * If nullable_action has been already set, this function raises >> - * an error. >> + * parsing a or a >> + * statement. A "NOT NULL" constraint has been seen on a column. >> + * This routine sets the is_nullable flag on the column currently >> + * under construction. If nullable_action has been already set, >> + * this function raises an error. >> * >> * @param parser SQL Parser object. >> * @param nullable_action on_conflict_action value. >> diff --git a/test/box/error.result b/test/box/error.result >> index 2196fa541..eb55f9cdf 100644 >> --- a/test/box/error.result >> +++ b/test/box/error.result >> @@ -432,6 +432,8 @@ t; >> | 211: box.error.WRONG_QUERY_ID >> | 212: box.error.SEQUENCE_NOT_STARTED >> | 213: box.error.NO_SUCH_SESSION_SETTING >> + | 214: box.error.SQL_CANT_ADD_COLUMN_TO_VIEW >> + | 215: box.error.SQL_CANT_ADD_AUTOINC >> | ... >> >> test_run:cmd("setopt delimiter ''"); >> diff --git a/test/sql/add-column.result b/test/sql/add-column.result >> new file mode 100644 >> index 000000000..06c95facb >> --- /dev/null >> +++ b/test/sql/add-column.result > > 28. You may want to look at the original SQLite tests. They probably > have some. I looked. I took everything that is relevant for Tarantool from there. >> @@ -0,0 +1,276 @@ >> +-- test-run result file version 2 >> +test_run = require('test_run').new() >> + | --- >> + | ... >> +engine = test_run:get_cfg('engine') >> + | --- >> + | ... >> +_ = box.space._session_settings:update('sql_default_engine', {{'=', 2, engine}}) >> + | --- >> + | ... >> + >> +-- >> +-- gh-3075: Check statement. >> +-- >> +box.execute('CREATE TABLE t1 (a INTEGER PRIMARY KEY);') >> + | --- >> + | - row_count: 1 >> + | ... >> +-- >> +-- COLUMN keyword is optional. Check it here, but omit it below. >> +-- >> +box.execute('ALTER TABLE t1 ADD COLUMN b INTEGER;') >> + | --- >> + | - row_count: 0 >> + | ... >> + >> +-- >> +-- Can't add column to a view. >> +-- >> +box.execute('CREATE VIEW v AS SELECT * FROM t1;') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('ALTER TABLE v ADD b INTEGER;') >> + | --- >> + | - null >> + | - Can't add column 'B'. 'V' is a view >> + | ... >> +box.execute('DROP VIEW v;') >> + | --- >> + | - row_count: 1 >> + | ... >> + >> +-- >> +-- Check column constraints typing and work. >> +-- >> +box.execute('CREATE TABLE t2 (a INTEGER CONSTRAINT pk_constr PRIMARY KEY);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('ALTER TABLE t2 DROP CONSTRAINT pk_constr') >> + | --- >> + | - row_count: 1 >> + | ... >> +test_run:cmd("setopt delimiter ';'"); >> + | --- >> + | - true >> + | ... >> +box.execute([[ALTER TABLE t2 ADD b INTEGER CONSTRAINT pk_constr PRIMARY KEY >> + CHECK (b > 0) >> + REFERENCES t1(a) >> + CONSTRAINT u_constr UNIQUE]]) >> +test_run:cmd("setopt delimiter ''"); >> + | --- >> + | ... >> +box.execute('INSERT INTO t1 VALUES (1, 1);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('INSERT INTO t2 VALUES (1, 1);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('INSERT INTO t2 VALUES (1, 1);') >> + | --- >> + | - null >> + | - Duplicate key exists in unique index 'PK_CONSTR' in space 'T2' >> + | ... >> + >> +box.execute('INSERT INTO t1 VALUES (0, 1);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('INSERT INTO t2 VALUES (2, 0);') >> + | --- >> + | - null >> + | - 'Check constraint failed ''ck_unnamed_T2_1'': b > 0' >> + | ... >> + >> +box.execute('INSERT INTO t2 VALUES (2, 3);') >> + | --- >> + | - null >> + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' >> + | ... >> + >> +box.execute('DROP TABLE t2;') >> + | --- >> + | - row_count: 1 >> + | ... >> + >> +-- >> +-- Check self-referenced FK creation. >> +-- >> +box.execute('CREATE TABLE t2 (a INTEGER PRIMARY KEY);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('ALTER TABLE t2 ADD b INT REFERENCES t1') >> + | --- >> + | - row_count: 0 > > 29. Worth checking if it works. Not just created and dropped right away. See the new version of tests in the diff. >> + | ... >> + >> +box.execute('DROP TABLE t2;') >> + | --- >> + | - row_count: 1 >> + | ... >> + >> +-- >> +-- Check AUTOINCREMENT work. >> +-- >> +box.execute("CREATE TABLE t2(a INTEGER CONSTRAINT pk PRIMARY KEY);") >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute("ALTER TABLE t2 DROP CONSTRAINT pk;") >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute("ALTER TABLE t2 ADD b INTEGER PRIMARY KEY AUTOINCREMENT;") >> + | --- >> + | - row_count: 0 > > 30. The same. > >> + | ... >> +box.execute("ALTER TABLE t2 ADD c INTEGER AUTOINCREMENT;") >> + | --- >> + | - null >> + | - 'Can''t add AUTOINCREMENT: the space ''T2'' already has one auto-incremented field' >> + | ... >> + >> +box.execute('DROP TABLE t2;') >> + | --- >> + | - row_count: 1 >> + | ... >> + >> +-- >> +-- Check clauses after column typing and work. >> +-- >> +box.execute('CREATE TABLE t2 (a INTEGER PRIMARY KEY, b INTEGER);') >> + | --- >> + | - row_count: 1 >> + | ... >> +test_run:cmd("setopt delimiter ';'"); >> + | --- >> + | - true >> + | ... >> +box.execute([[ALTER TABLE t2 ADD c TEXT NOT NULL DEFAULT ('a') >> + COLLATE "unicode_ci";]]); >> + | --- >> + | - row_count: 0 >> + | ... >> +test_run:cmd("setopt delimiter ''"); >> + | --- >> + | - true >> + | ... >> +box.execute('INSERT INTO t2(a, b) VALUES (1, 1);') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('SELECT * FROM t2;') >> + | --- >> + | - metadata: >> + | - name: A >> + | type: integer >> + | - name: B >> + | type: integer >> + | - name: C >> + | type: string >> + | rows: >> + | - [1, 1, 'a'] >> + | ... >> +box.execute('INSERT INTO t2 VALUES (2, 2, NULL);') >> + | --- >> + | - null >> + | - 'Failed to execute SQL statement: NOT NULL constraint failed: T2.C' >> + | ... >> +box.execute('SELECT * FROM t2 WHERE c LIKE \'A\';') >> + | --- >> + | - metadata: >> + | - name: A >> + | type: integer >> + | - name: B >> + | type: integer >> + | - name: C >> + | type: string >> + | rows: >> + | - [1, 1, 'a'] >> + | ... >> + >> +-- >> +-- Try to add to a non-empty space a [non-]nullable field. >> +-- >> +box.execute('ALTER TABLE t2 ADD d INTEGER;') >> + | --- >> + | - null >> + | - Tuple field count 3 does not match space field count 4 >> + | ... >> +box.execute('ALTER TABLE t2 ADD d TEXT NOT NULL'); >> + | --- >> + | - null >> + | - Tuple field count 3 does not match space field count 4 >> + | ... >> +box.execute('ALTER TABLE t2 ADD e TEXT NULL'); >> + | --- >> + | - null >> + | - Tuple field count 3 does not match space field count 4 >> + | ... >> + >> +-- >> +-- Add to a space with no-SQL adjusted or without format. >> +-- >> +_ = box.schema.space.create('WITHOUT') >> + | --- >> + | ... >> +box.execute("ALTER TABLE WITHOUT ADD a INTEGER;") > > 31. No need to caps table names in SQL. Fixed. > 32. Need to check if it actually worked. The same below. > >> + | --- >> + | - row_count: 0 >> + | ... >> +box.execute("DROP TABLE WITHOUT;") >> + | --- >> + | - row_count: 1 >> + | ... >> + >> +s = box.schema.space.create('NOSQL') >> + | --- >> + | ... >> +s:format{{name = 'A', type = 'unsigned'}} >> + | --- >> + | ... >> +box.execute("ALTER TABLE NOSQL ADD b INTEGER") >> + | --- >> + | - row_count: 0 >> + | ... >> + >> +box.execute('DROP TABLE t2;') >> + | --- >> + | - row_count: 1 >> + | ... >> +-- >> +-- Add multiple columns inside a transaction. >> +-- >> +box.execute('CREATE TABLE t2 (a INTEGER PRIMARY KEY)') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.begin() \ >> +box.execute('ALTER TABLE t2 ADD b INT') \ >> +box.execute('ALTER TABLE t2 ADD c INT') \ >> +box.commit() >> + | --- >> + | ... >> + >> +box.execute('INSERT INTO t2 VALUES (1, 1, 1)') >> + | --- >> + | - row_count: 1 >> + | ... >> +box.execute('SELECT * FROM t2;') >> + | --- >> + | - metadata: >> + | - name: A >> + | type: integer >> + | - name: B >> + | type: integer >> + | - name: C >> + | type: integer >> + | rows: >> + | - [1, 1, 1] > > 33. What if I add a column with UNIQUE constraint? Added. P.S. You can see that I added tests in checks.test.lua and foreign-keys.test.lua. I tested the naming of these constraints. This naming is (maybe isn?t) temporary, because it allows such a case: box.execute("CREATE TABLE t1 (a INT PRIMARY KEY);") --- - row_count: 1 ... box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY REFERENCES t1(a));") --- - row_count: 1 ... box.execute("ALTER TABLE check_naming ADD b INT REFERENCES t1(a);") --- - row_count: 0 ... box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"fk_unnamed_CHECK_NAMING_1\"") --- - row_count: 1 ... box.execute("ALTER TABLE check_naming ADD c INT REFERENCES t1(a);") --- - null - FOREIGN KEY constraint 'fk_unnamed_CHECK_NAMING_2' already exists in space 'CHECK_NAMING' ... To avoid such collisions, we need to use a hash table of names. Let?s put it aside for later. commit d0fb61f0283f64f1253307d38d1c8cfdea4c933b Author: Roman Khabibov Date: Thu Jan 2 19:06:14 2020 +0300 sql: support column addition Enable to add column to existing space with statement. Column definition can be supplemented with the four types of constraints, , clauses and <[NOT] NULL>, AUTOINCREMENT. Closes #2349, #3075 @TarantoolBot document Title: Add columns to existing tables in SQL Now, it is possible to add columns to existing empty spaces using statement. The column definition is the same as in statement. For example: ``` tarantool> box.execute("CREATE TABLE test (a INTEGER PRIMARY KEY)") --- - row_count: 1 ... tarantool> box.execute([[ALTER TABLE test ADD COLUMN b TEXT > CHECK (LENGTH(b) > 1) > NOT NULL > DEFAULT ('aa') > COLLATE "unicode_ci" > ]]) --- - row_count: 0 ... ``` diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c index 486b6b30d..dea047241 100644 --- a/extra/mkkeywordhash.c +++ b/extra/mkkeywordhash.c @@ -76,7 +76,7 @@ static Keyword aKeywordTable[] = { { "CHECK", "TK_CHECK", true }, { "COLLATE", "TK_COLLATE", true }, { "COLUMN_NAME", "TK_COLUMN_NAME", true }, - { "COLUMN", "TK_STANDARD", true }, + { "COLUMN", "TK_COLUMN", true }, { "COMMIT", "TK_COMMIT", true }, { "CONFLICT", "TK_CONFLICT", false }, { "CONSTRAINT", "TK_CONSTRAINT", true }, diff --git a/src/box/errcode.h b/src/box/errcode.h index 3c21375f5..cbcffb3a8 100644 --- a/src/box/errcode.h +++ b/src/box/errcode.h @@ -271,6 +271,8 @@ struct errcode_record { /*216 */_(ER_SYNC_QUORUM_TIMEOUT, "Quorum collection for a synchronous transaction is timed out") \ /*217 */_(ER_SYNC_ROLLBACK, "A rollback for a synchronous transaction is received") \ /*218 */_(ER_TUPLE_METADATA_IS_TOO_BIG, "Can't create tuple: metadata size %u is too big") \ + /*219 */_(ER_SQL_CANT_ADD_COLUMN_TO_VIEW, "Can't add column '%s'. '%s' is a view") \ + /*220 */_(ER_SQL_CANT_ADD_AUTOINC, "Can't add AUTOINCREMENT: the space '%s' already has one auto incremented field") \ /* * !IMPORTANT! Please follow instructions at start of the file diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 9013bc86f..6f3d2747d 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -285,48 +285,113 @@ sql_field_retrieve(Parse *parser, struct space_def *space_def, uint32_t id) return field; } -/* - * Add a new column to the table currently being constructed. +/** + * Make shallow copy of @a space on region. * - * The parser calls this routine once for each column declaration - * in a CREATE TABLE statement. sqlStartTable() gets called - * first to get things going. Then this routine is called for each - * column. + * Function is used to add a new column to an existing space with + * statement. Copy space def and index + * array to create constraints appeared in the statement. The + * index array copy will be modified by adding new elements to it. + * It is necessary, because the statement may contain several + * index definitions (constraints). */ +static struct space * +sql_shallow_space_copy(struct Parse *parse, struct space *space) +{ + assert(space->def != NULL); + struct space *ret = sql_ephemeral_space_new(parse, space->def->name); + if (ret == NULL) + return NULL; + ret->index_count = space->index_count; + ret->index_id_max = space->index_id_max; + uint32_t indexes_sz = sizeof(struct index *) * (ret->index_count); + ret->index = (struct index **) malloc(indexes_sz); + if (ret->index == NULL) { + diag_set(OutOfMemory, indexes_sz, "realloc", "ret->index"); + return NULL; + } + memcpy(ret->index, space->index, indexes_sz); + memcpy(ret->def, space->def, sizeof(struct space_def)); + ret->def->opts.is_temporary = true; + ret->def->opts.is_ephemeral = true; + if (ret->def->field_count != 0) { + uint32_t fields_size = 0; + ret->def->fields = + region_alloc_array(&parse->region, + typeof(struct field_def), + ret->def->field_count, &fields_size); + if (ret->def->fields == NULL) { + diag_set(OutOfMemory, fields_size, "region_alloc", + "ret->def->fields"); + free(ret->index); + return NULL; + } + memcpy(ret->def->fields, space->def->fields, fields_size); + } + + return ret; +} + void -sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) +sql_create_column_start(struct Parse *parse) { - assert(type_def != NULL); - char *z; - sql *db = pParse->db; - if (pParse->create_table_def.new_space == NULL) - return; - struct space_def *def = pParse->create_table_def.new_space->def; + struct create_column_def *create_column_def = &parse->create_column_def; + struct alter_entity_def *alter_entity_def = + &create_column_def->base.base; + assert(alter_entity_def->entity_type == ENTITY_TYPE_COLUMN); + assert(alter_entity_def->alter_action == ALTER_ACTION_CREATE); + struct space *space = parse->create_table_def.new_space; + bool is_alter = space == NULL; + struct sql *db = parse->db; + if (is_alter) { + const char *space_name = + alter_entity_def->entity_name->a[0].zName; + space = space_by_name(space_name); + if (space == NULL) { + diag_set(ClientError, ER_NO_SUCH_SPACE, space_name); + goto tnt_error; + } + space = sql_shallow_space_copy(parse, space); + if (space == NULL) + goto tnt_error; + } + create_column_def->space = space; + struct space_def *def = space->def; + assert(def->opts.is_ephemeral); #if SQL_MAX_COLUMN if ((int)def->field_count + 1 > db->aLimit[SQL_LIMIT_COLUMN]) { diag_set(ClientError, ER_SQL_COLUMN_COUNT_MAX, def->name, def->field_count + 1, db->aLimit[SQL_LIMIT_COLUMN]); - pParse->is_aborted = true; - return; + goto tnt_error; } #endif + + struct region *region = &parse->region; + struct Token *name = &create_column_def->base.name; + char *column_name = + sql_normalized_name_region_new(region, name->z, name->n); + if (column_name == NULL) + goto tnt_error; + + if (is_alter && def->opts.is_view) { + diag_set(ClientError, ER_SQL_CANT_ADD_COLUMN_TO_VIEW, + column_name, def->name); + goto tnt_error; + } + /* - * As sql_field_retrieve will allocate memory on region - * ensure that def is also temporal and would be dropped. + * Format can be set in Lua, then exact_field_count can be + * zero, but field_count is not. */ - assert(def->opts.is_ephemeral); - if (sql_field_retrieve(pParse, def, def->field_count) == NULL) + if (def->exact_field_count == 0) + def->exact_field_count = def->field_count; + if (sql_field_retrieve(parse, def, def->field_count) == NULL) return; - struct region *region = &pParse->region; - z = sql_normalized_name_region_new(region, pName->z, pName->n); - if (z == NULL) { - pParse->is_aborted = true; - return; - } + struct field_def *column_def = &def->fields[def->field_count]; memcpy(column_def, &field_def_default, sizeof(field_def_default)); - column_def->name = z; + column_def->name = column_name; /* * Marker ON_CONFLICT_ACTION_DEFAULT is used to detect * attempts to define NULL multiple time or to detect @@ -334,18 +399,86 @@ sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) */ column_def->nullable_action = ON_CONFLICT_ACTION_DEFAULT; column_def->is_nullable = true; - column_def->type = type_def->type; + column_def->type = create_column_def->type_def->type; def->field_count++; + + sqlSrcListDelete(db, alter_entity_def->entity_name); + return; +tnt_error: + parse->is_aborted = true; + sqlSrcListDelete(db, alter_entity_def->entity_name); +} + +static void +sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id); + +void +sql_create_column_end(struct Parse *parse) +{ + struct space *space = parse->create_column_def.space; + assert(space != NULL); + struct space_def *def = space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (field->nullable_action == ON_CONFLICT_ACTION_DEFAULT) { + field->nullable_action = ON_CONFLICT_ACTION_NONE; + field->is_nullable = true; + } + /* + * Encode the format array and emit code to update _space. + */ + uint32_t table_stmt_sz = 0; + struct region *region = &parse->region; + char *table_stmt = sql_encode_table(region, def, &table_stmt_sz); + char *raw = sqlDbMallocRaw(parse->db, table_stmt_sz); + if (table_stmt == NULL || raw == NULL) { + parse->is_aborted = true; + return; + } + memcpy(raw, table_stmt, table_stmt_sz); + + struct Vdbe *v = sqlGetVdbe(parse); + assert(v != NULL); + + struct space *system_space = space_by_id(BOX_SPACE_ID); + assert(system_space != NULL); + int cursor = parse->nTab++; + vdbe_emit_open_cursor(parse, cursor, 0, system_space); + sqlVdbeChangeP5(v, OPFLAG_SYSTEMSP); + + int key_reg = ++parse->nMem; + sqlVdbeAddOp2(v, OP_Integer, def->id, key_reg); + int addr = sqlVdbeAddOp4Int(v, OP_Found, cursor, 0, key_reg, 1); + sqlVdbeAddOp2(v, OP_Halt, -1, ON_CONFLICT_ACTION_ABORT); + sqlVdbeJumpHere(v, addr); + + int tuple_reg = sqlGetTempRange(parse, box_space_field_MAX + 1); + for (int i = 0; i < box_space_field_MAX - 1; ++i) + sqlVdbeAddOp3(v, OP_Column, cursor, i, tuple_reg + i); + sqlVdbeAddOp1(v, OP_Close, cursor); + + sqlVdbeAddOp2(v, OP_Integer, def->field_count, tuple_reg + 4); + sqlVdbeAddOp4(v, OP_Blob, table_stmt_sz, tuple_reg + 6, + SQL_SUBTYPE_MSGPACK, raw, P4_DYNAMIC); + sqlVdbeAddOp3(v, OP_MakeRecord, tuple_reg, box_space_field_MAX, + tuple_reg + box_space_field_MAX); + sqlVdbeAddOp4(v, OP_IdxReplace, tuple_reg + box_space_field_MAX, 0, 0, + (char *) system_space, P4_SPACEPTR); + sql_vdbe_create_constraints(parse, key_reg); + + /* + * Clean up array allocated in sql_shallow_space_copy(). + */ + free(space->index); } void sql_column_add_nullable_action(struct Parse *parser, enum on_conflict_action nullable_action) { - struct space *space = parser->create_table_def.new_space; - if (space == NULL || NEVER(space->def->field_count < 1)) + assert(parser->create_column_def.space != NULL); + struct space_def *def = parser->create_column_def.space->def; + if (NEVER(def->field_count < 1)) return; - struct space_def *def = space->def; struct field_def *field = &def->fields[def->field_count - 1]; if (field->nullable_action != ON_CONFLICT_ACTION_DEFAULT && nullable_action != field->nullable_action) { @@ -364,51 +497,42 @@ sql_column_add_nullable_action(struct Parse *parser, } /* - * The expression is the default value for the most recently added column - * of the table currently under construction. + * The expression is the default value for the most recently added + * column. * * Default value expressions must be constant. Raise an exception if this * is not the case. * * This routine is called by the parser while in the middle of - * parsing a CREATE TABLE statement. + * parsing a or a + * statement. */ void sqlAddDefaultValue(Parse * pParse, ExprSpan * pSpan) { sql *db = pParse->db; - struct space *p = pParse->create_table_def.new_space; - if (p != NULL) { - assert(p->def->opts.is_ephemeral); - struct space_def *def = p->def; - if (!sqlExprIsConstantOrFunction - (pSpan->pExpr, db->init.busy)) { - const char *column_name = - def->fields[def->field_count - 1].name; - diag_set(ClientError, ER_CREATE_SPACE, def->name, - tt_sprintf("default value of column '%s' is "\ - "not constant", column_name)); + assert(pParse->create_column_def.space != NULL); + struct space_def *def = pParse->create_column_def.space->def; + struct field_def *field = &def->fields[def->field_count - 1]; + if (!sqlExprIsConstantOrFunction(pSpan->pExpr, db->init.busy)) { + diag_set(ClientError, ER_CREATE_SPACE, def->name, + tt_sprintf("default value of column '%s' is not " + "constant", field->name)); + pParse->is_aborted = true; + } else { + struct region *region = &pParse->region; + uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); + field->default_value = region_alloc(region, default_length + 1); + if (field->default_value == NULL) { + diag_set(OutOfMemory, default_length + 1, + "region_alloc", "field->default_value"); pParse->is_aborted = true; - } else { - assert(def != NULL); - struct field_def *field = - &def->fields[def->field_count - 1]; - struct region *region = &pParse->region; - uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); - field->default_value = region_alloc(region, - default_length + 1); - if (field->default_value == NULL) { - diag_set(OutOfMemory, default_length + 1, - "region_alloc", - "field->default_value"); - pParse->is_aborted = true; - return; - } - strncpy(field->default_value, pSpan->zStart, - default_length); - field->default_value[default_length] = '\0'; + goto add_default_value_exit; } + strncpy(field->default_value, pSpan->zStart, default_length); + field->default_value[default_length] = '\0'; } +add_default_value_exit: sql_expr_delete(db, pSpan->pExpr, false); } @@ -447,6 +571,8 @@ sqlAddPrimaryKey(struct Parse *pParse) int nTerm; struct ExprList *pList = pParse->create_index_def.cols; struct space *space = pParse->create_table_def.new_space; + if (space == NULL) + space = pParse->create_column_def.space; if (space == NULL) goto primary_key_exit; if (sql_space_primary_key(space) != NULL) { @@ -574,8 +700,10 @@ sql_create_check_contraint(struct Parse *parser) (struct alter_entity_def *) create_ck_def; assert(alter_def->entity_type == ENTITY_TYPE_CK); (void) alter_def; - struct space *space = parser->create_table_def.new_space; - bool is_alter = space == NULL; + struct space *space = parser->create_column_def.space; + if (space == NULL) + space = parser->create_table_def.new_space; + bool is_alter_add_constr = space == NULL; /* Prepare payload for ck constraint definition. */ struct region *region = &parser->region; @@ -589,9 +717,23 @@ sql_create_check_contraint(struct Parse *parser) return; } } else { - assert(! is_alter); - uint32_t ck_idx = ++parser->check_count; - name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, ck_idx); + assert(!is_alter_add_constr); + uint32_t idx = ++parser->check_count; + /* + * If it is we should + * count the existing CHECK constraints in the + * space and form a name based on this. + */ + if (parser->create_table_def.new_space == NULL) { + struct space *original_space = + space_by_name(space->def->name); + assert(original_space != NULL); + struct rlist *checks = &original_space->ck_constraint; + struct ck_constraint *ck; + rlist_foreach_entry(ck, checks, link) + idx++; + } + name = tt_sprintf("ck_unnamed_%s_%d", space->def->name, idx); } size_t name_len = strlen(name); @@ -634,7 +776,7 @@ sql_create_check_contraint(struct Parse *parser) trim_space_snprintf(ck_def->expr_str, expr_str, expr_str_len); memcpy(ck_def->name, name, name_len); ck_def->name[name_len] = '\0'; - if (is_alter) { + if (is_alter_add_constr) { const char *space_name = alter_def->entity_name->a[0].zName; struct space *space = space_by_name(space_name); if (space == NULL) { @@ -663,9 +805,8 @@ sql_create_check_contraint(struct Parse *parser) void sqlAddCollateType(Parse * pParse, Token * pToken) { - struct space *space = pParse->create_table_def.new_space; - if (space == NULL) - return; + struct space *space = pParse->create_column_def.space; + assert(space != NULL); uint32_t i = space->def->field_count - 1; sql *db = pParse->db; char *coll_name = sql_name_from_token(db, pToken); @@ -704,8 +845,7 @@ sql_column_collation(struct space_def *def, uint32_t column, uint32_t *coll_id) * * In cases mentioned above collation is fetched by id. */ - if (space == NULL) { - assert(def->opts.is_ephemeral); + if (def->opts.is_ephemeral) { assert(column < (uint32_t)def->field_count); *coll_id = def->fields[column].coll_id; struct coll_id *collation = coll_by_id(*coll_id); @@ -794,7 +934,8 @@ vdbe_emit_create_index(struct Parse *parse, struct space_def *def, memcpy(raw, index_parts, index_parts_sz); index_parts = raw; - if (parse->create_table_def.new_space != NULL) { + if (parse->create_table_def.new_space != NULL || + parse->create_column_def.space != NULL) { sqlVdbeAddOp2(v, OP_SCopy, space_id_reg, entry_reg); sqlVdbeAddOp2(v, OP_Integer, idx_def->iid, entry_reg + 1); } else { @@ -1032,18 +1173,21 @@ vdbe_emit_fk_constraint_create(struct Parse *parse_context, P4_DYNAMIC); /* * In case we are adding FK constraints during execution - * of statement, we don't have child - * id, but we know register where it will be stored. + * of or + * statement, we don't have child id, but we know register + * where it will be stored. */ - if (parse_context->create_table_def.new_space != NULL) { + bool is_alter_add_constr = + parse_context->create_table_def.new_space == NULL && + parse_context->create_column_def.space == NULL; + if (!is_alter_add_constr) { sqlVdbeAddOp2(vdbe, OP_SCopy, fk->child_id, constr_tuple_reg + 1); } else { sqlVdbeAddOp2(vdbe, OP_Integer, fk->child_id, constr_tuple_reg + 1); } - if (parse_context->create_table_def.new_space != NULL && - fk_constraint_is_self_referenced(fk)) { + if (!is_alter_add_constr && fk_constraint_is_self_referenced(fk)) { sqlVdbeAddOp2(vdbe, OP_SCopy, fk->parent_id, constr_tuple_reg + 2); } else { @@ -1107,7 +1251,7 @@ vdbe_emit_fk_constraint_create(struct Parse *parse_context, constr_tuple_reg + 9); sqlVdbeAddOp2(vdbe, OP_SInsert, BOX_FK_CONSTRAINT_ID, constr_tuple_reg + 9); - if (parse_context->create_table_def.new_space == NULL) { + if (is_alter_add_constr) { sqlVdbeCountChanges(vdbe); sqlVdbeChangeP5(vdbe, OPFLAG_NCHANGE); } @@ -1148,15 +1292,21 @@ resolve_link(struct Parse *parse_context, const struct space_def *def, /** * Emit code to create sequences, indexes, check and foreign key - * constraints appeared in . + * constraints appeared in or + * . */ static void sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) { assert(reg_space_id != 0); struct space *space = parse->create_table_def.new_space; - assert(space != NULL); + bool is_alter = space == NULL; uint32_t i = 0; + if (is_alter) { + space = parse->create_column_def.space; + i = space_by_name(space->def->name)->index_count; + } + assert(space != NULL); for (; i < space->index_count; ++i) { struct index *idx = space->index[i]; vdbe_emit_create_index(parse, space->def, idx->def, @@ -1175,6 +1325,21 @@ sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) sqlVdbeAddOp2(v, OP_NextSequenceId, 0, reg_seq_id); int reg_seq_rec = emitNewSysSequenceRecord(parse, reg_seq_id, space->def->name); + if (is_alter) { + int errcode = ER_SQL_CANT_ADD_AUTOINC; + const char *error_msg = + tt_sprintf(tnt_errcode_desc(errcode), + space->def->name); + if (vdbe_emit_halt_with_presence_test(parse, + BOX_SEQUENCE_ID, + 2, + reg_seq_rec + 3, + 1, errcode, + error_msg, false, + OP_NoConflict) + != 0) + return; + } sqlVdbeAddOp2(v, OP_SInsert, BOX_SEQUENCE_ID, reg_seq_rec); /* Do an insertion into _space_sequence. */ int reg_space_seq_record = @@ -1873,24 +2038,28 @@ sql_create_foreign_key(struct Parse *parse_context) char *parent_name = NULL; char *constraint_name = NULL; bool is_self_referenced = false; + struct space *space = parse_context->create_column_def.space; struct create_table_def *table_def = &parse_context->create_table_def; - struct space *space = table_def->new_space; + if (space == NULL) + space = table_def->new_space; /* - * Space under construction during CREATE TABLE - * processing. NULL for ALTER TABLE statement handling. + * Space under construction during + * processing or shallow copy of space during . NULL for statement handling. */ - bool is_alter = space == NULL; + bool is_alter_add_constr = space == NULL; uint32_t child_cols_count; struct ExprList *child_cols = create_fk_def->child_cols; if (child_cols == NULL) { - assert(!is_alter); + assert(!is_alter_add_constr); child_cols_count = 1; } else { child_cols_count = child_cols->nExpr; } struct ExprList *parent_cols = create_fk_def->parent_cols; struct space *child_space = NULL; - if (is_alter) { + if (is_alter_add_constr) { const char *child_name = alter_def->entity_name->a[0].zName; child_space = space_by_name(child_name); if (child_space == NULL) { @@ -1908,6 +2077,8 @@ sql_create_foreign_key(struct Parse *parse_context) goto tnt_error; } memset(fk_parse, 0, sizeof(*fk_parse)); + if (parse_context->create_column_def.space != NULL) + child_space = space; rlist_add_entry(&parse_context->fkeys, fk_parse, link); } struct Token *parent = create_fk_def->parent_name; @@ -1920,28 +2091,45 @@ sql_create_foreign_key(struct Parse *parse_context) * self-referenced, but in this case parent (which is * also child) table will definitely exist. */ - is_self_referenced = !is_alter && + is_self_referenced = !is_alter_add_constr && strcmp(parent_name, space->def->name) == 0; struct space *parent_space = space_by_name(parent_name); - if (parent_space == NULL) { - if (is_self_referenced) { - struct fk_constraint_parse *fk = - rlist_first_entry(&parse_context->fkeys, - struct fk_constraint_parse, - link); - fk->selfref_cols = parent_cols; - fk->is_self_referenced = true; - } else { - diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name);; - goto tnt_error; - } + if (parent_space == NULL && !is_self_referenced) { + diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name); + goto tnt_error; + } + if (is_self_referenced) { + struct fk_constraint_parse *fk = + rlist_first_entry(&parse_context->fkeys, + struct fk_constraint_parse, + link); + fk->selfref_cols = parent_cols; + fk->is_self_referenced = true; } - if (!is_alter) { + if (!is_alter_add_constr) { if (create_def->name.n == 0) { - constraint_name = - sqlMPrintf(db, "fk_unnamed_%s_%d", - space->def->name, - ++parse_context->fkey_count); + uint32_t idx = ++parse_context->fkey_count; + /* + * If it is we + * should count the existing FK + * constraints in the space and form a + * name based on this. + */ + if (table_def->new_space == NULL) { + struct space *original_space = + space_by_name(space->def->name); + assert(original_space != NULL); + struct rlist *child_fk = + &original_space->child_fk_constraint; + if (!rlist_empty(child_fk)) { + struct fk_constraint *fk; + rlist_foreach_entry(fk, child_fk, + in_child_space) + idx++; + } + } + constraint_name = sqlMPrintf(db, "fk_unnamed_%s_%d", + space->def->name, idx); } else { constraint_name = sql_name_from_token(db, &create_def->name); @@ -2001,7 +2189,8 @@ sql_create_foreign_key(struct Parse *parse_context) } int actions = create_fk_def->actions; fk_def->field_count = child_cols_count; - fk_def->child_id = child_space != NULL ? child_space->def->id : 0; + fk_def->child_id = table_def->new_space == NULL ? + child_space->def->id : 0; fk_def->parent_id = parent_space != NULL ? parent_space->def->id : 0; fk_def->is_deferred = create_constr_def->is_deferred; fk_def->match = (enum fk_constraint_match) (create_fk_def->match); @@ -2021,7 +2210,7 @@ sql_create_foreign_key(struct Parse *parse_context) constraint_name) != 0) { goto exit_create_fk; } - if (!is_alter) { + if (!is_alter_add_constr) { if (child_cols == NULL) { assert(i == 0); /* @@ -2050,12 +2239,13 @@ sql_create_foreign_key(struct Parse *parse_context) memcpy(fk_def->name, constraint_name, name_len); fk_def->name[name_len] = '\0'; /* - * In case of CREATE TABLE processing, all foreign keys - * constraints must be created after space itself, so - * lets delay it until sqlEndTable() call and simply + * In case of or processing, all foreign keys constraints must + * be created after space itself, so lets delay it until + * sqlEndTable() or sql_add_column_end() call and simply * maintain list of all FK constraints inside parser. */ - if (!is_alter) { + if (!is_alter_add_constr) { struct fk_constraint_parse *fk_parse = rlist_first_entry(&parse_context->fkeys, struct fk_constraint_parse, link); @@ -2407,7 +2597,10 @@ sql_create_index(struct Parse *parse) { * Find the table that is to be indexed. * Return early if not found. */ - struct space *space = NULL; + struct space *space = parse->create_table_def.new_space; + if (space == NULL) + space = parse->create_column_def.space; + bool is_create_table_or_add_col = space != NULL; struct Token token = create_entity_def->name; if (tbl_name != NULL) { assert(token.n > 0 && token.z != NULL); @@ -2420,10 +2613,8 @@ sql_create_index(struct Parse *parse) { } goto exit_create_index; } - } else { - if (parse->create_table_def.new_space == NULL) - goto exit_create_index; - space = parse->create_table_def.new_space; + } else if (space == NULL) { + goto exit_create_index; } struct space_def *def = space->def; @@ -2458,7 +2649,7 @@ sql_create_index(struct Parse *parse) { * 2) UNIQUE constraint is non-named and standard * auto-index name will be generated. */ - if (parse->create_table_def.new_space == NULL) { + if (!is_create_table_or_add_col) { assert(token.z != NULL); name = sql_name_from_token(db, &token); if (name == NULL) { @@ -2624,7 +2815,7 @@ sql_create_index(struct Parse *parse) { * constraint, but has different onError (behavior on * constraint violation), then an error is raised. */ - if (parse->create_table_def.new_space != NULL) { + if (is_create_table_or_add_col) { for (uint32_t i = 0; i < space->index_count; ++i) { struct index *existing_idx = space->index[i]; uint32_t iid = existing_idx->def->iid; @@ -2712,7 +2903,7 @@ sql_create_index(struct Parse *parse) { sqlVdbeAddOp0(vdbe, OP_Expire); } - if (tbl_name != NULL) + if (!is_create_table_or_add_col) goto exit_create_index; table_add_index(space, index); index = NULL; diff --git a/src/box/sql/parse.y b/src/box/sql/parse.y index 995875566..0c9887851 100644 --- a/src/box/sql/parse.y +++ b/src/box/sql/parse.y @@ -226,19 +226,24 @@ create_table_end ::= . { sqlEndTable(pParse); } */ columnlist ::= columnlist COMMA tcons. -columnlist ::= columnlist COMMA columnname carglist autoinc(I). { - uint32_t fieldno = pParse->create_table_def.new_space->def->field_count - 1; - if (I == 1 && sql_add_autoincrement(pParse, fieldno) != 0) - return; +columnlist ::= columnlist COMMA column_def create_column_end. +columnlist ::= column_def create_column_end. + +column_def ::= column_name_and_type carglist. + +column_name_and_type ::= nm(A) typedef(Y). { + create_column_def_init(&pParse->create_column_def, NULL, &A, &Y); + sql_create_column_start(pParse); } -columnlist ::= columnname carglist autoinc(I). { - uint32_t fieldno = pParse->create_table_def.new_space->def->field_count - 1; +create_column_end ::= autoinc(I). { + uint32_t fieldno = pParse->create_column_def.space->def->field_count - 1; if (I == 1 && sql_add_autoincrement(pParse, fieldno) != 0) return; + if (pParse->create_table_def.new_space == NULL) + sql_create_column_end(pParse); } columnlist ::= tcons. -columnname(A) ::= nm(A) typedef(Y). {sqlAddColumn(pParse,&A,&Y);} // An IDENTIFIER can be a generic identifier, or one of several // keywords. Any non-standard keyword can also be an identifier. @@ -281,9 +286,11 @@ nm(A) ::= id(A). { } } -// "carglist" is a list of additional constraints that come after the -// column name and column type in a CREATE TABLE statement. -// +/* + * "carglist" is a list of additional constraints and clauses that + * come after the column name and column type in a + * or statement. + */ carglist ::= carglist ccons. carglist ::= . %type cconsname { struct Token } @@ -1735,11 +1742,28 @@ alter_table_start(A) ::= ALTER TABLE fullname(T) . { A = T; } %type alter_add_constraint {struct alter_args} alter_add_constraint(A) ::= alter_table_start(T) ADD CONSTRAINT nm(N). { + A.table_name = T; + A.name = N; + pParse->initiateTTrans = true; + } + +%type alter_add_column {struct alter_args} +alter_add_column(A) ::= alter_table_start(T) ADD column_name(N). { A.table_name = T; A.name = N; pParse->initiateTTrans = true; } +column_name(N) ::= COLUMN nm(A). { N = A; } +column_name(N) ::= nm(A). { N = A; } + +cmd ::= alter_column_def carglist create_column_end. + +alter_column_def ::= alter_add_column(N) typedef(Y). { + create_column_def_init(&pParse->create_column_def, N.table_name, &N.name, &Y); + sql_create_column_start(pParse); +} + cmd ::= alter_add_constraint(N) FOREIGN KEY LP eidlist(FA) RP REFERENCES nm(T) eidlist_opt(TA) matcharg(M) refargs(R) defer_subclause_opt(D). { create_fk_def_init(&pParse->create_fk_def, N.table_name, &N.name, FA, &T, TA, diff --git a/src/box/sql/parse_def.h b/src/box/sql/parse_def.h index 1105fda6e..336914c57 100644 --- a/src/box/sql/parse_def.h +++ b/src/box/sql/parse_def.h @@ -35,6 +35,7 @@ #include "box/fk_constraint.h" #include "box/key_def.h" #include "box/sql.h" +#include "box/constraint_id.h" /** * This file contains auxiliary structures and functions which @@ -154,6 +155,7 @@ enum sql_index_type { enum entity_type { ENTITY_TYPE_TABLE = 0, + ENTITY_TYPE_COLUMN, ENTITY_TYPE_VIEW, ENTITY_TYPE_INDEX, ENTITY_TYPE_TRIGGER, @@ -207,6 +209,14 @@ struct create_table_def { struct space *new_space; }; +struct create_column_def { + struct create_entity_def base; + /** Shallow space_def copy. */ + struct space *space; + /** Column type. */ + struct type_def *type_def; +}; + struct create_view_def { struct create_entity_def base; /** @@ -464,6 +474,16 @@ create_table_def_init(struct create_table_def *table_def, struct Token *name, if_not_exists); } +static inline void +create_column_def_init(struct create_column_def *column_def, + struct SrcList *table_name, struct Token *name, + struct type_def *type_def) +{ + create_entity_def_init(&column_def->base, ENTITY_TYPE_COLUMN, + table_name, name, false); + column_def->type_def = type_def; +} + static inline void create_view_def_init(struct create_view_def *view_def, struct Token *name, struct Token *create, struct ExprList *aliases, diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index fa87e7bd2..32142a871 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -2251,22 +2251,26 @@ struct Parse { struct enable_entity_def enable_entity_def; }; /** - * Table def is not part of union since information - * being held must survive till the end of parsing of - * whole CREATE TABLE statement (to pass it to - * sqlEndTable() function). + * Table def or column def is not part of union since + * information being held must survive till the end of + * parsing of whole or + * statement (to pass it to + * sqlEndTable() sql_create_column_end() function). */ struct create_table_def create_table_def; + struct create_column_def create_column_def; /* - * FK and CK constraints appeared in a . + * FK and CK constraints appeared in a or + * an statement. */ struct rlist fkeys; struct rlist checks; uint32_t fkey_count; uint32_t check_count; /* - * True, if column within a statement to be - * created has . + * True, if column in a or an + * statement to be created has + * . */ bool has_autoinc; /* Id of field with . */ @@ -2860,15 +2864,30 @@ struct space *sqlResultSetOfSelect(Parse *, Select *); struct space * sqlStartTable(Parse *, Token *); -void sqlAddColumn(Parse *, Token *, struct type_def *); + +/** + * Add new field to the format of ephemeral space in + * create_table_def. If it is create shallow copy of + * the existing space and add field to its format. + */ +void +sql_create_column_start(struct Parse *parse); + +/** + * Emit code to update entry in _space and code to create + * constraints (entries in _index, _ck_constraint, _fk_constraint) + * described with this column. + */ +void +sql_create_column_end(struct Parse *parse); /** * This routine is called by the parser while in the middle of - * parsing a CREATE TABLE statement. A "NOT NULL" constraint has - * been seen on a column. This routine sets the is_nullable flag - * on the column currently under construction. - * If nullable_action has been already set, this function raises - * an error. + * parsing a or a + * statement. A "NOT NULL" constraint has been seen on a column. + * This routine sets the is_nullable flag on the column currently + * under construction. If nullable_action has been already set, + * this function raises an error. * * @param parser SQL Parser object. * @param nullable_action on_conflict_action value. diff --git a/test/box/error.result b/test/box/error.result index cdecdb221..6fc3cb99f 100644 --- a/test/box/error.result +++ b/test/box/error.result @@ -437,6 +437,8 @@ t; | 216: box.error.SYNC_QUORUM_TIMEOUT | 217: box.error.SYNC_ROLLBACK | 218: box.error.TUPLE_METADATA_IS_TOO_BIG + | 219: box.error.SQL_CANT_ADD_COLUMN_TO_VIEW + | 220: box.error.SQL_CANT_ADD_AUTOINC | ... test_run:cmd("setopt delimiter ''"); diff --git a/test/sql/add-column.result b/test/sql/add-column.result new file mode 100644 index 000000000..f86259105 --- /dev/null +++ b/test/sql/add-column.result @@ -0,0 +1,471 @@ +-- test-run result file version 2 +-- +-- gh-3075: Check statement. +-- +CREATE TABLE t1 (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... + +-- +-- COLUMN keyword is optional. Check it here, but omit it below. +-- +ALTER TABLE t1 ADD COLUMN b INT; + | --- + | - row_count: 0 + | ... + +-- +-- A column with the same name already exists. +-- +ALTER TABLE t1 ADD b SCALAR; + | --- + | - null + | - Space field 'B' is duplicate + | ... + +-- +-- Can't add column to a view. +-- +CREATE VIEW v AS SELECT * FROM t1; + | --- + | - row_count: 1 + | ... +ALTER TABLE v ADD b INT; + | --- + | - null + | - Can't add column 'B'. 'V' is a view + | ... +DROP VIEW v; + | --- + | - row_count: 1 + | ... + +-- +-- Check PRIMARY KEY constraint works with an added column. +-- +CREATE TABLE pk_check (a INT CONSTRAINT pk PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE pk_check DROP CONSTRAINT pk; + | --- + | - row_count: 1 + | ... +ALTER TABLE pk_check ADD b INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO pk_check VALUES (1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO pk_check VALUES (1, 1); + | --- + | - null + | - Duplicate key exists in unique index 'pk_unnamed_PK_CHECK_1' in space 'PK_CHECK' + | ... +DROP TABLE pk_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check UNIQUE constraint works with an added column. +-- +CREATE TABLE unique_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE unique_check ADD b INT UNIQUE; + | --- + | - row_count: 0 + | ... +INSERT INTO unique_check VALUES (1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO unique_check VALUES (2, 1); + | --- + | - null + | - Duplicate key exists in unique index 'unique_unnamed_UNIQUE_CHECK_2' in space 'UNIQUE_CHECK' + | ... +DROP TABLE unique_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check CHECK constraint works with an added column. +-- +CREATE TABLE ck_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE ck_check ADD b INT CHECK (b > 0); + | --- + | - row_count: 0 + | ... +INSERT INTO ck_check VALUES (1, 0); + | --- + | - null + | - 'Check constraint failed ''ck_unnamed_CK_CHECK_1'': b > 0' + | ... +DROP TABLE ck_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check FOREIGN KEY constraint works with an added column. +-- +CREATE TABLE fk_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE fk_check ADD b INT REFERENCES t1(a); + | --- + | - row_count: 0 + | ... +INSERT INTO fk_check VALUES (0, 1); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +INSERT INTO fk_check VALUES (2, 0); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +INSERT INTO fk_check VALUES (2, 1); + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +DROP TABLE fk_check; + | --- + | - row_count: 1 + | ... +DROP TABLE t1; + | --- + | - row_count: 1 + | ... +-- +-- Check FOREIGN KEY (self-referenced) constraint works with an +-- added column. +-- +CREATE TABLE self (id INT PRIMARY KEY AUTOINCREMENT, a INT UNIQUE) + | --- + | - row_count: 1 + | ... +ALTER TABLE self ADD b INT REFERENCES self(a) + | --- + | - row_count: 0 + | ... +INSERT INTO self(a,b) VALUES(1, 1); + | --- + | - autoincrement_ids: + | - 1 + | row_count: 1 + | ... +UPDATE self SET a = 2, b = 2; + | --- + | - row_count: 1 + | ... +UPDATE self SET b = 3; + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +UPDATE self SET a = 3; + | --- + | - null + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' + | ... +DROP TABLE self; + | --- + | - row_count: 1 + | ... + +-- +-- Check AUTOINCREMENT works with an added column. +-- +CREATE TABLE autoinc_check (a INT CONSTRAINT pk PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE autoinc_check DROP CONSTRAINT pk; + | --- + | - row_count: 1 + | ... +ALTER TABLE autoinc_check ADD b INT PRIMARY KEY AUTOINCREMENT; + | --- + | - row_count: 0 + | ... +INSERT INTO autoinc_check(a) VALUES(1); + | --- + | - autoincrement_ids: + | - 1 + | row_count: 1 + | ... +INSERT INTO autoinc_check(a) VALUES(1); + | --- + | - autoincrement_ids: + | - 2 + | row_count: 1 + | ... +TRUNCATE TABLE autoinc_check; + | --- + | - row_count: 0 + | ... + +-- +-- Can't add second column with AUTOINCREMENT. +-- +ALTER TABLE autoinc_check ADD c INT AUTOINCREMENT; + | --- + | - null + | - 'Can''t add AUTOINCREMENT: the space ''AUTOINC_CHECK'' already has one auto incremented + | field' + | ... +DROP TABLE autoinc_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check COLLATE clause works with an added column. +-- +CREATE TABLE collate_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE collate_check ADD b TEXT COLLATE "unicode_ci"; + | --- + | - row_count: 0 + | ... +INSERT INTO collate_check VALUES (1, 'a'); + | --- + | - row_count: 1 + | ... +INSERT INTO collate_check VALUES (2, 'A'); + | --- + | - row_count: 1 + | ... +SELECT * FROM collate_check WHERE b LIKE 'a'; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: string + | rows: + | - [1, 'a'] + | - [2, 'A'] + | ... +DROP TABLE collate_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check DEFAULT clause works with an added column. +-- +CREATE TABLE default_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE default_check ADD b TEXT DEFAULT ('a'); + | --- + | - row_count: 0 + | ... +INSERT INTO default_check(a) VALUES (1); + | --- + | - row_count: 1 + | ... +SELECT * FROM default_check; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: string + | rows: + | - [1, 'a'] + | ... +DROP TABLE default_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check NULL constraint works with an added column. +-- +CREATE TABLE null_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE null_check ADD b TEXT NULL; + | --- + | - row_count: 0 + | ... +INSERT INTO null_check(a) VALUES (1); + | --- + | - row_count: 1 + | ... +DROP TABLE null_check; + | --- + | - row_count: 1 + | ... + +-- +-- Check NOT NULL constraint works with an added column. +-- +CREATE TABLE notnull_check (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +ALTER TABLE notnull_check ADD b TEXT NOT NULL; + | --- + | - row_count: 0 + | ... +INSERT INTO notnull_check(a) VALUES (1); + | --- + | - null + | - 'Failed to execute SQL statement: NOT NULL constraint failed: NOTNULL_CHECK.B' + | ... +DROP TABLE notnull_check; + | --- + | - row_count: 1 + | ... + +-- +-- Can't add a column with DEAFULT or NULL to a non-empty space. +-- This ability isn't implemented yet. +-- +CREATE TABLE non_empty (a INT PRIMARY KEY); + | --- + | - row_count: 1 + | ... +INSERT INTO non_empty VALUES (1); + | --- + | - row_count: 1 + | ... +ALTER TABLE non_empty ADD b INT NULL; + | --- + | - null + | - Tuple field count 1 does not match space field count 2 + | ... +ALTER TABLE non_empty ADD b INT DEFAULT (1); + | --- + | - null + | - Tuple field count 1 does not match space field count 2 + | ... +DROP TABLE non_empty; + | --- + | - row_count: 1 + | ... + +-- +-- Add to a no-SQL adjusted space without format. +-- +\set language lua + | --- + | - true + | ... +_ = box.schema.space.create('WITHOUT_FORMAT') + | --- + | ... +\set language sql + | --- + | - true + | ... +ALTER TABLE without_format ADD a INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO without_format VALUES (1); + | --- + | - row_count: 1 + | ... +DROP TABLE without_format; + | --- + | - row_count: 1 + | ... + +-- +-- Add to a no-SQL adjusted space with format. +-- +\set language lua + | --- + | - true + | ... +with_format = box.schema.space.create('WITH_FORMAT') + | --- + | ... +with_format:format{{name = 'A', type = 'unsigned'}} + | --- + | ... +\set language sql + | --- + | - true + | ... +ALTER TABLE with_format ADD b INT PRIMARY KEY; + | --- + | - row_count: 0 + | ... +INSERT INTO with_format VALUES (1, 1); + | --- + | - row_count: 1 + | ... +DROP TABLE with_format; + | --- + | - row_count: 1 + | ... + +-- +-- Add multiple columns (with a constraint) inside a transaction. +-- +CREATE TABLE t2 (a INT PRIMARY KEY) + | --- + | - row_count: 1 + | ... +\set language lua + | --- + | - true + | ... +box.begin() \ +box.execute('ALTER TABLE t2 ADD b INT') \ +box.execute('ALTER TABLE t2 ADD c INT UNIQUE') \ +box.commit() + | --- + | ... +\set language sql + | --- + | - true + | ... +INSERT INTO t2 VALUES (1, 1, 1); + | --- + | - row_count: 1 + | ... +INSERT INTO t2 VALUES (2, 1, 1); + | --- + | - null + | - Duplicate key exists in unique index 'unique_unnamed_T2_2' in space 'T2' + | ... +SELECT * FROM t2; + | --- + | - metadata: + | - name: A + | type: integer + | - name: B + | type: integer + | - name: C + | type: integer + | rows: + | - [1, 1, 1] + | ... +DROP TABLE t2; + | --- + | - row_count: 1 + | ... diff --git a/test/sql/add-column.test.sql b/test/sql/add-column.test.sql new file mode 100644 index 000000000..f8ab3f756 --- /dev/null +++ b/test/sql/add-column.test.sql @@ -0,0 +1,167 @@ +-- +-- gh-3075: Check statement. +-- +CREATE TABLE t1 (a INT PRIMARY KEY); + +-- +-- COLUMN keyword is optional. Check it here, but omit it below. +-- +ALTER TABLE t1 ADD COLUMN b INT; + +-- +-- A column with the same name already exists. +-- +ALTER TABLE t1 ADD b SCALAR; + +-- +-- Can't add column to a view. +-- +CREATE VIEW v AS SELECT * FROM t1; +ALTER TABLE v ADD b INT; +DROP VIEW v; + +-- +-- Check PRIMARY KEY constraint works with an added column. +-- +CREATE TABLE pk_check (a INT CONSTRAINT pk PRIMARY KEY); +ALTER TABLE pk_check DROP CONSTRAINT pk; +ALTER TABLE pk_check ADD b INT PRIMARY KEY; +INSERT INTO pk_check VALUES (1, 1); +INSERT INTO pk_check VALUES (1, 1); +DROP TABLE pk_check; + +-- +-- Check UNIQUE constraint works with an added column. +-- +CREATE TABLE unique_check (a INT PRIMARY KEY); +ALTER TABLE unique_check ADD b INT UNIQUE; +INSERT INTO unique_check VALUES (1, 1); +INSERT INTO unique_check VALUES (2, 1); +DROP TABLE unique_check; + +-- +-- Check CHECK constraint works with an added column. +-- +CREATE TABLE ck_check (a INT PRIMARY KEY); +ALTER TABLE ck_check ADD b INT CHECK (b > 0); +INSERT INTO ck_check VALUES (1, 0); +DROP TABLE ck_check; + +-- +-- Check FOREIGN KEY constraint works with an added column. +-- +CREATE TABLE fk_check (a INT PRIMARY KEY); +ALTER TABLE fk_check ADD b INT REFERENCES t1(a); +INSERT INTO fk_check VALUES (0, 1); +INSERT INTO fk_check VALUES (2, 0); +INSERT INTO fk_check VALUES (2, 1); +DROP TABLE fk_check; +DROP TABLE t1; +-- +-- Check FOREIGN KEY (self-referenced) constraint works with an +-- added column. +-- +CREATE TABLE self (id INT PRIMARY KEY AUTOINCREMENT, a INT UNIQUE) +ALTER TABLE self ADD b INT REFERENCES self(a) +INSERT INTO self(a,b) VALUES(1, 1); +UPDATE self SET a = 2, b = 2; +UPDATE self SET b = 3; +UPDATE self SET a = 3; +DROP TABLE self; + +-- +-- Check AUTOINCREMENT works with an added column. +-- +CREATE TABLE autoinc_check (a INT CONSTRAINT pk PRIMARY KEY); +ALTER TABLE autoinc_check DROP CONSTRAINT pk; +ALTER TABLE autoinc_check ADD b INT PRIMARY KEY AUTOINCREMENT; +INSERT INTO autoinc_check(a) VALUES(1); +INSERT INTO autoinc_check(a) VALUES(1); +TRUNCATE TABLE autoinc_check; + +-- +-- Can't add second column with AUTOINCREMENT. +-- +ALTER TABLE autoinc_check ADD c INT AUTOINCREMENT; +DROP TABLE autoinc_check; + +-- +-- Check COLLATE clause works with an added column. +-- +CREATE TABLE collate_check (a INT PRIMARY KEY); +ALTER TABLE collate_check ADD b TEXT COLLATE "unicode_ci"; +INSERT INTO collate_check VALUES (1, 'a'); +INSERT INTO collate_check VALUES (2, 'A'); +SELECT * FROM collate_check WHERE b LIKE 'a'; +DROP TABLE collate_check; + +-- +-- Check DEFAULT clause works with an added column. +-- +CREATE TABLE default_check (a INT PRIMARY KEY); +ALTER TABLE default_check ADD b TEXT DEFAULT ('a'); +INSERT INTO default_check(a) VALUES (1); +SELECT * FROM default_check; +DROP TABLE default_check; + +-- +-- Check NULL constraint works with an added column. +-- +CREATE TABLE null_check (a INT PRIMARY KEY); +ALTER TABLE null_check ADD b TEXT NULL; +INSERT INTO null_check(a) VALUES (1); +DROP TABLE null_check; + +-- +-- Check NOT NULL constraint works with an added column. +-- +CREATE TABLE notnull_check (a INT PRIMARY KEY); +ALTER TABLE notnull_check ADD b TEXT NOT NULL; +INSERT INTO notnull_check(a) VALUES (1); +DROP TABLE notnull_check; + +-- +-- Can't add a column with DEAFULT or NULL to a non-empty space. +-- This ability isn't implemented yet. +-- +CREATE TABLE non_empty (a INT PRIMARY KEY); +INSERT INTO non_empty VALUES (1); +ALTER TABLE non_empty ADD b INT NULL; +ALTER TABLE non_empty ADD b INT DEFAULT (1); +DROP TABLE non_empty; + +-- +-- Add to a no-SQL adjusted space without format. +-- +\set language lua +_ = box.schema.space.create('WITHOUT_FORMAT') +\set language sql +ALTER TABLE without_format ADD a INT PRIMARY KEY; +INSERT INTO without_format VALUES (1); +DROP TABLE without_format; + +-- +-- Add to a no-SQL adjusted space with format. +-- +\set language lua +with_format = box.schema.space.create('WITH_FORMAT') +with_format:format{{name = 'A', type = 'unsigned'}} +\set language sql +ALTER TABLE with_format ADD b INT PRIMARY KEY; +INSERT INTO with_format VALUES (1, 1); +DROP TABLE with_format; + +-- +-- Add multiple columns (with a constraint) inside a transaction. +-- +CREATE TABLE t2 (a INT PRIMARY KEY) +\set language lua +box.begin() \ +box.execute('ALTER TABLE t2 ADD b INT') \ +box.execute('ALTER TABLE t2 ADD c INT UNIQUE') \ +box.commit() +\set language sql +INSERT INTO t2 VALUES (1, 1, 1); +INSERT INTO t2 VALUES (2, 1, 1); +SELECT * FROM t2; +DROP TABLE t2; diff --git a/test/sql/checks.result b/test/sql/checks.result index 7b18e5d6b..513ed1b62 100644 --- a/test/sql/checks.result +++ b/test/sql/checks.result @@ -856,6 +856,26 @@ box.execute("DROP TABLE t6") --- - row_count: 1 ... +-- +-- gh-3075: Check the auto naming of CHECK constraints in +-- . +-- +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY CHECK (a > 0))") +--- +- row_count: 1 +... +box.execute("ALTER TABLE check_naming ADD b INT CHECK (b > 0)") +--- +- row_count: 0 +... +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"ck_unnamed_CHECK_NAMING_2\"") +--- +- row_count: 1 +... +box.execute("DROP TABLE check_naming") +--- +- row_count: 1 +... test_run:cmd("clear filter") --- - true diff --git a/test/sql/checks.test.lua b/test/sql/checks.test.lua index 301f8ea69..a79131466 100644 --- a/test/sql/checks.test.lua +++ b/test/sql/checks.test.lua @@ -280,4 +280,13 @@ box.func.MYFUNC:drop() box.execute("INSERT INTO t6 VALUES(11);"); box.execute("DROP TABLE t6") +-- +-- gh-3075: Check the auto naming of CHECK constraints in +-- . +-- +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY CHECK (a > 0))") +box.execute("ALTER TABLE check_naming ADD b INT CHECK (b > 0)") +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"ck_unnamed_CHECK_NAMING_2\"") +box.execute("DROP TABLE check_naming") + test_run:cmd("clear filter") diff --git a/test/sql/foreign-keys.result b/test/sql/foreign-keys.result index 33689a06e..de2a0c512 100644 --- a/test/sql/foreign-keys.result +++ b/test/sql/foreign-keys.result @@ -499,5 +499,33 @@ box.space.S:drop() box.space.T:drop() --- ... +-- +-- gh-3075: Check the auto naming of FOREIGN KEY constraints in +-- . +-- +box.execute("CREATE TABLE t1 (a INT PRIMARY KEY)") +--- +- row_count: 1 +... +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY REFERENCES t1(a))") +--- +- row_count: 1 +... +box.execute("ALTER TABLE check_naming ADD b INT REFERENCES t1(a)") +--- +- row_count: 0 +... +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"fk_unnamed_CHECK_NAMING_2\"") +--- +- row_count: 1 +... +box.execute("DROP TABLE check_naming") +--- +- row_count: 1 +... +box.execute("DROP TABLE t1") +--- +- row_count: 1 +... --- Clean-up SQL DD hash. -test_run:cmd('restart server default with cleanup=1') diff --git a/test/sql/foreign-keys.test.lua b/test/sql/foreign-keys.test.lua index d2dd88d28..29918c5d4 100644 --- a/test/sql/foreign-keys.test.lua +++ b/test/sql/foreign-keys.test.lua @@ -209,5 +209,16 @@ box.space.T:select() box.space.S:drop() box.space.T:drop() +-- +-- gh-3075: Check the auto naming of FOREIGN KEY constraints in +-- . +-- +box.execute("CREATE TABLE t1 (a INT PRIMARY KEY)") +box.execute("CREATE TABLE check_naming (a INT PRIMARY KEY REFERENCES t1(a))") +box.execute("ALTER TABLE check_naming ADD b INT REFERENCES t1(a)") +box.execute("ALTER TABLE check_naming DROP CONSTRAINT \"fk_unnamed_CHECK_NAMING_2\"") +box.execute("DROP TABLE check_naming") +box.execute("DROP TABLE t1") + --- Clean-up SQL DD hash. -test_run:cmd('restart server default with cleanup=1') From roman.habibov at tarantool.org Tue Aug 11 03:34:04 2020 From: roman.habibov at tarantool.org (Roman Khabibov) Date: Tue, 11 Aug 2020 03:34:04 +0300 Subject: [Tarantool-patches] [PATCH v2 1/2] sql: rename TK_COLUMN to TK_COLUMN_NAME In-Reply-To: <4241abff-452b-ba0c-841f-5adc8c3d910f@tarantool.org> References: <20200403152752.8923-1-roman.habibov@tarantool.org> <20200403152752.8923-2-roman.habibov@tarantool.org> <4241abff-452b-ba0c-841f-5adc8c3d910f@tarantool.org> Message-ID: <1981AD55-5001-47DB-982C-0ADD9CF64EE5@tarantool.org> Hi! Thanks for the review. > On Apr 25, 2020, at 01:55, Vladislav Shpilevoy wrote: > > Hi! Thanks for the patch! > >> @@ -5081,7 +5081,7 @@ sqlExprCompare(Expr * pA, Expr * pB, int iTab) >> } >> return 2; >> } >> - if (pA->op != TK_COLUMN && pA->op != TK_AGG_COLUMN && pA->u.zToken) { >> + if (pA->op != TK_COLUMN_NAME && pA->op != TK_AGG_COLUMN && pA->u.zToken) { > > 1. This is out of 80 symbols. Please, wrap the line. Fixed. >> if (pA->op == TK_FUNCTION) { >> if (sqlStrICmp(pA->u.zToken, pB->u.zToken) != 0) >> return 2; >> @@ -5161,7 +5161,7 @@ sqlExprListCompare(ExprList * pA, ExprList * pB, int iTab) >> * pE1: x IS NULL pE2: x IS NOT NULL Result: false >> * pE1: x IS ?2 pE2: x IS NOT NULL Reuslt: false >> * >> - * When comparing TK_COLUMN nodes between pE1 and pE2, if pE2 has >> + * When comparing TK_COLUMN_NAME nodes between pE1 and pE2, if pE2 has >> * Expr.iTable<0 then assume a table number given by iTab. >> * >> * When in doubt, return false. Returning true might give a performance >> @@ -5209,11 +5209,11 @@ exprSrcCount(Walker * pWalker, Expr * pExpr) >> { >> /* The NEVER() on the second term is because sqlFunctionUsesThisSrc() >> * is always called before sqlExprAnalyzeAggregates() and so the >> - * TK_COLUMNs have not yet been converted into TK_AGG_COLUMN. If >> + * TK_COLUMN_NAMEs have not yet been converted into TK_AGG_COLUMN. If > > 2. While you are here, please, remove the multiplied whitespace. Done. >> * sqlFunctionUsesThisSrc() is used differently in the future, the >> * NEVER() will need to be removed. >> */ >> - if (pExpr->op == TK_COLUMN || NEVER(pExpr->op == TK_AGG_COLUMN)) { >> + if (pExpr->op == TK_COLUMN_NAME || NEVER(pExpr->op == TK_AGG_COLUMN)) { >> int i; >> struct SrcCount *p = pWalker->u.pSrcCount; >> SrcList *pSrc = p->pSrc; commit ee578dbfb97b09b911f169826a68aafe37d9300b Author: Roman Khabibov Date: Fri Apr 3 16:27:34 2020 +0300 sql: rename TK_COLUMN to TK_COLUMN_NAME Rename TK_COLUMN used for tokens treated as a column name to TK_COLUMN_NAME. It is needed to allow the typing of COLUMN keyword in statement. Needed for #3075 diff --git a/extra/addopcodes.sh b/extra/addopcodes.sh index cb6c84725..986c62683 100755 --- a/extra/addopcodes.sh +++ b/extra/addopcodes.sh @@ -39,7 +39,7 @@ extras=" \ END_OF_FILE \ UNCLOSED_STRING \ FUNCTION \ - COLUMN \ + COLUMN_NAME \ AGG_FUNCTION \ AGG_COLUMN \ UMINUS \ diff --git a/extra/mkkeywordhash.c b/extra/mkkeywordhash.c index dd42c8f5f..486b6b30d 100644 --- a/extra/mkkeywordhash.c +++ b/extra/mkkeywordhash.c @@ -75,10 +75,7 @@ static Keyword aKeywordTable[] = { { "CAST", "TK_CAST", false }, { "CHECK", "TK_CHECK", true }, { "COLLATE", "TK_COLLATE", true }, - /* gh-3075: Reserved until ALTER ADD COLUMN is implemeneted. - * Move it back to ALTER when done. - */ - /* { "COLUMN", "TK_COLUMNKW", true }, */ + { "COLUMN_NAME", "TK_COLUMN_NAME", true }, { "COLUMN", "TK_STANDARD", true }, { "COMMIT", "TK_COMMIT", true }, { "CONFLICT", "TK_CONFLICT", false }, diff --git a/src/box/sql/build.c b/src/box/sql/build.c index 8f6b403b9..619bbf8e3 100644 --- a/src/box/sql/build.c +++ b/src/box/sql/build.c @@ -2295,7 +2295,7 @@ index_fill_def(struct Parse *parse, struct index *index, goto cleanup; struct Expr *column_expr = sqlExprSkipCollate(expr); - if (column_expr->op != TK_COLUMN) { + if (column_expr->op != TK_COLUMN_NAME) { diag_set(ClientError, ER_UNSUPPORTED, "Tarantool", "functional indexes"); goto tnt_error; diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index bc2182446..d389b3daf 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -62,7 +62,7 @@ sql_expr_type(struct Expr *pExpr) assert(!ExprHasProperty(pExpr, EP_IntValue)); return pExpr->type; case TK_AGG_COLUMN: - case TK_COLUMN: + case TK_COLUMN_NAME: case TK_TRIGGER: assert(pExpr->iColumn >= 0); return pExpr->space_def->fields[pExpr->iColumn].type; @@ -262,13 +262,13 @@ sql_expr_coll(Parse *parse, Expr *p, bool *is_explicit_coll, uint32_t *coll_id, *is_explicit_coll = true; break; } - if ((op == TK_AGG_COLUMN || op == TK_COLUMN || + if ((op == TK_AGG_COLUMN || op == TK_COLUMN_NAME || op == TK_REGISTER || op == TK_TRIGGER) && p->space_def != NULL) { /* * op==TK_REGISTER && p->space_def!=0 * happens when pExpr was originally - * a TK_COLUMN but was previously + * a TK_COLUMN_NAME but was previously * evaluated and cached in a register. */ int j = p->iColumn; @@ -2061,11 +2061,11 @@ exprNodeIsConstant(Walker * pWalker, Expr * pExpr) return WRC_Abort; } case TK_ID: - case TK_COLUMN: + case TK_COLUMN_NAME: case TK_AGG_FUNCTION: case TK_AGG_COLUMN: testcase(pExpr->op == TK_ID); - testcase(pExpr->op == TK_COLUMN); + testcase(pExpr->op == TK_COLUMN_NAME); testcase(pExpr->op == TK_AGG_FUNCTION); testcase(pExpr->op == TK_AGG_COLUMN); if (pWalker->eCode == 3 && pExpr->iTable == pWalker->u.iCur) { @@ -2236,7 +2236,7 @@ sqlExprCanBeNull(const Expr * p) case TK_FLOAT: case TK_BLOB: return 0; - case TK_COLUMN: + case TK_COLUMN_NAME: assert(p->space_def != 0); return ExprHasProperty(p, EP_CanBeNull) || (p->iColumn >= 0 @@ -2267,7 +2267,7 @@ sql_expr_needs_no_type_change(const struct Expr *p, enum field_type type) return type == FIELD_TYPE_STRING; case TK_BLOB: return type == FIELD_TYPE_VARBINARY; - case TK_COLUMN: + case TK_COLUMN_NAME: /* p cannot be part of a CHECK constraint. */ assert(p->iTable >= 0); return p->iColumn < 0 && sql_type_is_numeric(type); @@ -2324,7 +2324,7 @@ isCandidateForInOpt(Expr * pX) /* All SELECT results must be columns. */ for (i = 0; i < pEList->nExpr; i++) { Expr *pRes = pEList->a[i].pExpr; - if (pRes->op != TK_COLUMN) + if (pRes->op != TK_COLUMN_NAME) return 0; assert(pRes->iTable == pSrc->a[0].iCursor); /* Not a correlated subquery */ } @@ -3707,10 +3707,10 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) } return target; } - /* Otherwise, fall thru into the TK_COLUMN case */ + /* Otherwise, fall thru into the TK_COLUMN_NAME case */ FALLTHROUGH; } - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ int iTab = pExpr->iTable; int col = pExpr->iColumn; if (iTab < 0) { @@ -4102,7 +4102,7 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) assert(nFarg == 1); assert(pFarg->a[0].pExpr != 0); exprOp = pFarg->a[0].pExpr->op; - if (exprOp == TK_COLUMN + if (exprOp == TK_COLUMN_NAME || exprOp == TK_AGG_COLUMN) { assert(SQL_FUNC_LENGTH == OPFLAG_LENGTHARG); @@ -4319,7 +4319,7 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) endLabel = sqlVdbeMakeLabel(v); if ((pX = pExpr->pLeft) != 0) { tempX = *pX; - testcase(pX->op == TK_COLUMN); + testcase(pX->op == TK_COLUMN_NAME); exprToRegister(&tempX, exprCodeVector(pParse, &tempX, ®Free1)); @@ -4344,11 +4344,11 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) pTest = aListelem[i].pExpr; } nextCase = sqlVdbeMakeLabel(v); - testcase(pTest->op == TK_COLUMN); + testcase(pTest->op == TK_COLUMN_NAME); sqlExprIfFalse(pParse, pTest, nextCase, SQL_JUMPIFNULL); testcase(aListelem[i + 1].pExpr->op == - TK_COLUMN); + TK_COLUMN_NAME); sqlExprCode(pParse, aListelem[i + 1].pExpr, target); sqlVdbeGoto(v, endLabel); @@ -5081,7 +5081,8 @@ sqlExprCompare(Expr * pA, Expr * pB, int iTab) } return 2; } - if (pA->op != TK_COLUMN && pA->op != TK_AGG_COLUMN && pA->u.zToken) { + if (pA->op != TK_COLUMN_NAME && pA->op != TK_AGG_COLUMN && + pA->u.zToken) { if (pA->op == TK_FUNCTION) { if (sqlStrICmp(pA->u.zToken, pB->u.zToken) != 0) return 2; @@ -5161,8 +5162,8 @@ sqlExprListCompare(ExprList * pA, ExprList * pB, int iTab) * pE1: x IS NULL pE2: x IS NOT NULL Result: false * pE1: x IS ?2 pE2: x IS NOT NULL Reuslt: false * - * When comparing TK_COLUMN nodes between pE1 and pE2, if pE2 has - * Expr.iTable<0 then assume a table number given by iTab. + * When comparing TK_COLUMN_NAME nodes between pE1 and pE2, if + * pE2 has Expr.iTable<0 then assume a table number given by iTab. * * When in doubt, return false. Returning true might give a performance * improvement. Returning false might cause a performance reduction, but @@ -5209,11 +5210,11 @@ exprSrcCount(Walker * pWalker, Expr * pExpr) { /* The NEVER() on the second term is because sqlFunctionUsesThisSrc() * is always called before sqlExprAnalyzeAggregates() and so the - * TK_COLUMNs have not yet been converted into TK_AGG_COLUMN. If + * TK_COLUMN_NAMEs have not yet been converted into TK_AGG_COLUMN. If * sqlFunctionUsesThisSrc() is used differently in the future, the * NEVER() will need to be removed. */ - if (pExpr->op == TK_COLUMN || NEVER(pExpr->op == TK_AGG_COLUMN)) { + if (pExpr->op == TK_COLUMN_NAME || NEVER(pExpr->op == TK_AGG_COLUMN)) { int i; struct SrcCount *p = pWalker->u.pSrcCount; SrcList *pSrc = p->pSrc; @@ -5299,9 +5300,9 @@ analyzeAggregate(Walker * pWalker, Expr * pExpr) switch (pExpr->op) { case TK_AGG_COLUMN: - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ testcase(pExpr->op == TK_AGG_COLUMN); - testcase(pExpr->op == TK_COLUMN); + testcase(pExpr->op == TK_COLUMN_NAME); /* Check to see if the column is in one of the tables in the FROM * clause of the aggregate query */ @@ -5370,7 +5371,7 @@ analyzeAggregate(Walker * pWalker, Expr * pExpr) if (pE-> op == - TK_COLUMN + TK_COLUMN_NAME && pE-> iTable diff --git a/src/box/sql/fk_constraint.c b/src/box/sql/fk_constraint.c index 482220a95..3f3625ad5 100644 --- a/src/box/sql/fk_constraint.c +++ b/src/box/sql/fk_constraint.c @@ -338,7 +338,7 @@ static struct Expr * sql_expr_new_column_by_cursor(struct sql *db, struct space_def *def, int cursor, int column) { - struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN); + struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN_NAME); if (expr == NULL) return NULL; expr->space_def = def; diff --git a/src/box/sql/resolve.c b/src/box/sql/resolve.c index 6f625dc18..5fe8ee3f6 100644 --- a/src/box/sql/resolve.c +++ b/src/box/sql/resolve.c @@ -189,7 +189,7 @@ sqlMatchSpanName(const char *zSpan, * pExpr->space_def Points to the space_def structure of X.Y * (even if X and/or Y are implied.) * pExpr->iColumn Set to the column number within the table. - * pExpr->op Set to TK_COLUMN. + * pExpr->op Set to TK_COLUMN_NAME. * pExpr->pLeft Any expression this points to is deleted * pExpr->pRight Any expression this points to is deleted. * @@ -461,7 +461,7 @@ lookupName(Parse * pParse, /* The parsing context */ pExpr->pLeft = 0; sql_expr_delete(db, pExpr->pRight, false); pExpr->pRight = 0; - pExpr->op = (isTrigger ? TK_TRIGGER : TK_COLUMN); + pExpr->op = (isTrigger ? TK_TRIGGER : TK_COLUMN_NAME); lookupname_end: if (cnt == 1) { assert(pNC != 0); @@ -485,7 +485,7 @@ struct Expr * sql_expr_new_column(struct sql *db, struct SrcList *src_list, int src_idx, int column) { - struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN); + struct Expr *expr = sql_expr_new_anon(db, TK_COLUMN_NAME); if (expr == NULL) return NULL; struct SrcList_item *item = &src_list->a[src_idx]; @@ -518,7 +518,7 @@ exprProbability(Expr * p) /* * This routine is callback for sqlWalkExpr(). * - * Resolve symbolic names into TK_COLUMN operators for the current + * Resolve symbolic names into TK_COLUMN_NAME operators for the current * node in the expression tree. Return 0 to continue the search down * the tree or 2 to abort the tree walk. * @@ -1451,7 +1451,7 @@ resolveSelectStep(Walker * pWalker, Select * p) * * The node at the root of the subtree is modified as follows: * - * Expr.op Changed to TK_COLUMN + * Expr.op Changed to TK_COLUMN_NAME * Expr.pTab Points to the Table object for X.Y * Expr.iColumn The column index in X.Y. -1 for the rowid. * Expr.iTable The VDBE cursor number for X.Y diff --git a/src/box/sql/select.c b/src/box/sql/select.c index b0554a172..80fbf69e0 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1817,7 +1817,7 @@ generate_column_metadata(struct Parse *pParse, struct SrcList *pTabList, vdbe_metadata_set_col_nullability(v, i, -1); const char *colname = pEList->a[i].zName; const char *span = pEList->a[i].zSpan; - if (p->op == TK_COLUMN || p->op == TK_AGG_COLUMN) { + if (p->op == TK_COLUMN_NAME || p->op == TK_AGG_COLUMN) { char *zCol; int iCol = p->iColumn; for (j = 0; ALWAYS(j < pTabList->nSrc); j++) { @@ -1939,7 +1939,7 @@ sqlColumnsFromExprList(Parse * parse, ExprList * expr_list, pColExpr = pColExpr->pRight; assert(pColExpr != 0); } - if (pColExpr->op == TK_COLUMN + if (pColExpr->op == TK_COLUMN_NAME && ALWAYS(pColExpr->space_def != NULL)) { /* For columns use the column name name */ int iCol = pColExpr->iColumn; @@ -3653,7 +3653,7 @@ substExpr(Parse * pParse, /* Report errors here */ sql *db = pParse->db; if (pExpr == 0) return 0; - if (pExpr->op == TK_COLUMN && pExpr->iTable == iTable) { + if (pExpr->op == TK_COLUMN_NAME && pExpr->iTable == iTable) { if (pExpr->iColumn < 0) { pExpr->op = TK_NULL; } else { @@ -6044,7 +6044,7 @@ sqlSelect(Parse * pParse, /* The parser context */ /* Create a label to jump to when we want to abort the query */ addrEnd = sqlVdbeMakeLabel(v); - /* Convert TK_COLUMN nodes into TK_AGG_COLUMN and make entries in + /* Convert TK_COLUMN_NAME nodes into TK_AGG_COLUMN and make entries in * sAggInfo for all TK_AGG_FUNCTION nodes in expressions of the * SELECT statement. */ @@ -6416,7 +6416,7 @@ sqlSelect(Parse * pParse, /* The parser context */ flag != WHERE_ORDERBY_MIN ? 1 : 0; pMinMax->a[0].pExpr->op = - TK_COLUMN; + TK_COLUMN_NAME; } } diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index adf90d824..beb83ce95 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -1517,7 +1517,7 @@ typedef int ynVar; * valid. * * An expression of the form ID or ID.ID refers to a column in a table. - * For such expressions, Expr.op is set to TK_COLUMN and Expr.iTable is + * For such expressions, Expr.op is set to TK_COLUMN_NAME and Expr.iTable is * the integer cursor number of a VDBE cursor pointing to that table and * Expr.iColumn is the column number for the specific column. If the * expression is used as a result in an aggregate SELECT, then the @@ -1587,20 +1587,20 @@ struct Expr { #if SQL_MAX_EXPR_DEPTH>0 int nHeight; /* Height of the tree headed by this node */ #endif - int iTable; /* TK_COLUMN: cursor number of table holding column + int iTable; /* TK_COLUMN_NAME: cursor number of table holding column * TK_REGISTER: register number * TK_TRIGGER: 1 -> new, 0 -> old * EP_Unlikely: 134217728 times likelihood * TK_SELECT: 1st register of result vector */ - ynVar iColumn; /* TK_COLUMN: column index. + ynVar iColumn; /* TK_COLUMN_NAME: column index. * TK_VARIABLE: variable number (always >= 1). * TK_SELECT_COLUMN: column of the result vector */ i16 iAgg; /* Which entry in pAggInfo->aCol[] or ->aFunc[] */ i16 iRightJoinTable; /* If EP_FromJoin, the right table of the join */ u8 op2; /* TK_REGISTER: original value of Expr.op - * TK_COLUMN: the value of p5 for OP_Column + * TK_COLUMN_NAME: the value of p5 for OP_Column * TK_AGG_FUNCTION: nesting depth */ AggInfo *pAggInfo; /* Used by TK_AGG_COLUMN and TK_AGG_FUNCTION */ diff --git a/src/box/sql/treeview.c b/src/box/sql/treeview.c index a04597979..ea26fcf6d 100644 --- a/src/box/sql/treeview.c +++ b/src/box/sql/treeview.c @@ -327,7 +327,7 @@ sqlTreeViewExpr(TreeView * pView, const Expr * pExpr, u8 moreToFollow) zFlgs); break; } - case TK_COLUMN:{ + case TK_COLUMN_NAME:{ if (pExpr->iTable < 0) { /* This only happens when coding check constraints */ sqlTreeViewLine(pView, "COLUMN(%d)%s", diff --git a/src/box/sql/where.c b/src/box/sql/where.c index e9e936856..77f863b0e 100644 --- a/src/box/sql/where.c +++ b/src/box/sql/where.c @@ -276,7 +276,7 @@ whereScanNext(WhereScan * pScan) sqlExprSkipCollate(pTerm-> pExpr-> pRight))-> - op == TK_COLUMN) { + op == TK_COLUMN_NAME) { int j; for (j = 0; j < pScan->nEquiv; j++) { if (pScan->aiCur[j] == pX->iTable @@ -319,7 +319,7 @@ whereScanNext(WhereScan * pScan) } } if ((pTerm->eOperator & WO_EQ) != 0 - && (pX = pTerm->pExpr->pRight)->op == TK_COLUMN + && (pX = pTerm->pExpr->pRight)->op == TK_COLUMN_NAME && pX->iTable == pScan->aiCur[0] && pX->iColumn == pScan->aiColumn[0]) { continue; @@ -555,7 +555,7 @@ findIndexCol(Parse * pParse, /* Parse context */ struct key_part *part_to_match = &idx_def->key_def->parts[iCol]; for (int i = 0; i < pList->nExpr; i++) { Expr *p = sqlExprSkipCollate(pList->a[i].pExpr); - if (p->op == TK_COLUMN && p->iTable == iBase && + if (p->op == TK_COLUMN_NAME && p->iTable == iBase && p->iColumn == (int) part_to_match->fieldno) { bool is_found; uint32_t id; @@ -601,7 +601,8 @@ isDistinctRedundant(Parse * pParse, /* Parsing context */ */ for (int i = 0; i < pDistinct->nExpr; i++) { Expr *p = sqlExprSkipCollate(pDistinct->a[i].pExpr); - if (p->op == TK_COLUMN && p->iTable == iBase && p->iColumn < 0) + if (p->op == TK_COLUMN_NAME && p->iTable == iBase && + p->iColumn < 0) return 1; } if (space == NULL) @@ -2245,7 +2246,7 @@ whereRangeVectorLen(Parse * pParse, /* Parsing context */ * leftmost index column. */ struct key_part *parts = idx_def->key_def->parts; - if (pLhs->op != TK_COLUMN || pLhs->iTable != iCur || + if (pLhs->op != TK_COLUMN_NAME || pLhs->iTable != iCur || pLhs->iColumn != (int)parts[i + nEq].fieldno || parts[i + nEq].sort_order != parts[nEq].sort_order) break; @@ -2677,7 +2678,7 @@ indexMightHelpWithOrderBy(WhereLoopBuilder * pBuilder, return 0; for (ii = 0; ii < pOB->nExpr; ii++) { Expr *pExpr = sqlExprSkipCollate(pOB->a[ii].pExpr); - if (pExpr->op == TK_COLUMN && pExpr->iTable == iCursor) { + if (pExpr->op == TK_COLUMN_NAME && pExpr->iTable == iCursor) { if (pExpr->iColumn < 0) return 1; for (jj = 0; jj < part_count; jj++) { @@ -3213,7 +3214,7 @@ wherePathSatisfiesOrderBy(WhereInfo * pWInfo, /* The WHERE clause */ if (MASKBIT(i) & obSat) continue; pOBExpr = sqlExprSkipCollate(pOrderBy->a[i].pExpr); - if (pOBExpr->op != TK_COLUMN) + if (pOBExpr->op != TK_COLUMN_NAME) continue; if (pOBExpr->iTable != iCur) continue; @@ -3361,7 +3362,8 @@ wherePathSatisfiesOrderBy(WhereInfo * pWInfo, /* The WHERE clause */ if ((wctrlFlags & (WHERE_GROUPBY | WHERE_DISTINCTBY)) == 0) bOnce = 0; if (iColumn >= (-1)) { - if (pOBExpr->op != TK_COLUMN) + if (pOBExpr->op != + TK_COLUMN_NAME) continue; if (pOBExpr->iTable != iCur) continue; diff --git a/src/box/sql/whereexpr.c b/src/box/sql/whereexpr.c index d9b5c78f5..c2ce4339f 100644 --- a/src/box/sql/whereexpr.c +++ b/src/box/sql/whereexpr.c @@ -279,7 +279,7 @@ like_optimization_is_valid(Parse *pParse, Expr *pExpr, Expr **ppPrefix, pList = pExpr->x.pList; pLeft = pList->a[1].pExpr; /* Value might be numeric */ - if (pLeft->op != TK_COLUMN || + if (pLeft->op != TK_COLUMN_NAME || sql_expr_type(pLeft) != FIELD_TYPE_STRING) { /* IMP: R-02065-49465 The left-hand side of the * LIKE operator must be the name of an indexed @@ -928,7 +928,7 @@ exprSelectUsage(WhereMaskSet * pMaskSet, Select * pS) * number of the table that is indexed and *piColumn to the column number * of the column that is indexed. * - * If pExpr is a TK_COLUMN column reference, then this routine always returns + * If pExpr is a TK_COLUMN_NAME column reference, then this routine always returns * true even if that particular column is not indexed, because the column * might be added to an automatic index later. */ @@ -950,7 +950,7 @@ exprMightBeIndexed(int op, /* The specific comparison operator */ pExpr = pExpr->x.pList->a[0].pExpr; } - if (pExpr->op == TK_COLUMN) { + if (pExpr->op == TK_COLUMN_NAME) { *piCur = pExpr->iTable; *piColumn = pExpr->iColumn; return 1; @@ -1272,7 +1272,7 @@ exprAnalyze(SrcList * pSrc, /* the FROM clause */ * Note that the virtual term must be tagged with TERM_VNULL. */ if (pExpr->op == TK_NOTNULL - && pExpr->pLeft->op == TK_COLUMN + && pExpr->pLeft->op == TK_COLUMN_NAME && pExpr->pLeft->iColumn >= 0) { Expr *pNewExpr; Expr *pLeft = pExpr->pLeft; @@ -1397,7 +1397,7 @@ sqlWhereExprUsage(WhereMaskSet * pMaskSet, Expr * p) Bitmask mask; if (p == 0) return 0; - if (p->op == TK_COLUMN) { + if (p->op == TK_COLUMN_NAME) { mask = sqlWhereGetMask(pMaskSet, p->iTable); return mask; } @@ -1475,7 +1475,7 @@ sqlWhereTabFuncArgs(Parse * pParse, /* Parsing context */ * unused. */ assert(k < (int)space_def->field_count); - pColRef = sql_expr_new_anon(pParse->db, TK_COLUMN); + pColRef = sql_expr_new_anon(pParse->db, TK_COLUMN_NAME); if (pColRef == NULL) { pParse->is_aborted = true; return; From gorcunov at gmail.com Tue Aug 11 10:53:20 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 11 Aug 2020 10:53:20 +0300 Subject: [Tarantool-patches] [PATCH 1/1] box: snapshot should not include rolled back data In-Reply-To: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> References: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> Message-ID: <20200811075320.GA2074@grain> On Tue, Aug 11, 2020 at 12:18:05AM +0200, Vladislav Shpilevoy wrote: > Box.snapshot() could include rolled back data in case synchronous > transaction ROLLBACK arrived during WAL rotation in preparation of > a checkpoint. > > More specifically, snapshot consists of fixating the engines' > content (creation of a read-view), doing WAL rotation, and writing > the snapshot itself. All data changes after content fixation won't > go into the snap. So if ROLLBACK arrives during WAL rotation, the > fixated content will have rolled back data, not present in the > newest dataset. > > The patch makes it fail if during WAL rotation anything was rolled > back. The bug sometimes appeared in an existing test about qsync > snapshots, but with a very poor reproducibility. In a new test > file it is reproduced 100% without the patch. > > Closes #5167 Ack. Lets give it a shot. From alyapunov at tarantool.org Tue Aug 11 12:44:01 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Tue, 11 Aug 2020 12:44:01 +0300 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: References: Message-ID: <53095890-c0b6-6eae-b3b1-9898e7ca05e3@tarantool.org> Hi! thanks again for the patch. see one comment below. On 8/5/20 2:45 AM, Vladislav Shpilevoy wrote: > goto parse; > if (offset_slot_hint != NULL) > *offset_slot_hint = offset_slot; > + /* > + * When the field is multikey, the offset slot points not at the > + * data. It points at 'extra' array of offsets for this multikey > + * index. That array can only be accessed if index in that array > + * is known. > + */ > + if (field->is_multikey_part && multikey_idx == MULTIKEY_NONE) > + goto parse; > offset_slot_access: > /* Indexed field */ > offset = field_map_get_offset(field_map, offset_slot, I'm sure that your check must be moved for two lines up. I mean the check must be done before setting *offset_slot_hint. As I understood offset_slot_hint will contain a hint for further tuple_field_raw_by_path calls with the same path. That is a kind of agreement, we may call tuple_field_raw_by_path twice and must get the same results. But in your code you set *offset_slot_hint before a check that could go to 'parse' label. Meanwhile in the second call of tuple_field_raw_by_path it'll check *offset_slot_hint and will go to 'offset_slot_access' label. That's wrong. From sergepetrenko at tarantool.org Tue Aug 11 15:32:18 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Tue, 11 Aug 2020 15:32:18 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_1/1=5D_box=3A_snapshot_sho?= =?utf-8?q?uld_not_include_rolled_back_data?= In-Reply-To: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> References: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> Message-ID: <1597149138.527428518@f456.i.mail.ru> Hi! Thanks for the patch, LGTM. -- Serge?Petrenko ? ? >???????, 11 ??????? 2020, 1:18 +03:00 ?? Vladislav Shpilevoy : >? >Box.snapshot() could include rolled back data in case synchronous >transaction ROLLBACK arrived during WAL rotation in preparation of >a checkpoint. > >More specifically, snapshot consists of fixating the engines' >content (creation of a read-view), doing WAL rotation, and writing >the snapshot itself. All data changes after content fixation won't >go into the snap. So if ROLLBACK arrives during WAL rotation, the >fixated content will have rolled back data, not present in the >newest dataset. > >The patch makes it fail if during WAL rotation anything was rolled >back. The bug sometimes appeared in an existing test about qsync >snapshots, but with a very poor reproducibility. In a new test >file it is reproduced 100% without the patch. > >Closes #5167 >--- >Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5167-flaky-qsync-snapshots >Issue: https://github.com/tarantool/tarantool/issues/5167 > >@ChangeLog >* Snapshot could contain changes from a rolled back synchronous transaction (gh-5167). > >?src/box/gc.c | 12 +- >?src/box/wal.c | 4 + >?src/lib/core/errinj.h | 1 + >?test/box/errinj.result | 1 + >?.../gh-5167-qsync-rollback-snap.result | 165 ++++++++++++++++++ >?.../gh-5167-qsync-rollback-snap.test.lua | 67 +++++++ >?test/replication/suite.ini | 2 +- >?7 files changed, 250 insertions(+), 2 deletions(-) >?create mode 100644 test/replication/gh-5167-qsync-rollback-snap.result >?create mode 100644 test/replication/gh-5167-qsync-rollback-snap.test.lua > >diff --git a/src/box/gc.c b/src/box/gc.c >index d50a64d66..76f7c6325 100644 >--- a/src/box/gc.c >+++ b/src/box/gc.c >@@ -382,6 +382,7 @@ gc_do_checkpoint(bool is_scheduled) >?{ >? int rc; >? struct wal_checkpoint checkpoint; >+ int64_t limbo_rollback_count = txn_limbo.rollback_count; >? >? assert(!gc.checkpoint_is_in_progress); >? gc.checkpoint_is_in_progress = true; >@@ -396,7 +397,16 @@ gc_do_checkpoint(bool is_scheduled) >? rc = wal_begin_checkpoint(&checkpoint); >? if (rc != 0) >? goto out; >- >+ /* >+ * Check if the checkpoint contains rolled back data. That makes the >+ * checkpoint not self-sufficient - it needs the xlog file with >+ * ROLLBACK. Drop it. >+ */ >+ if (txn_limbo.rollback_count != limbo_rollback_count) { >+ rc = -1; >+ diag_set(ClientError, ER_SYNC_ROLLBACK); >+ goto out; >+ } >? /* >? * Wait the confirms on all "sync" transactions before >? * create a snapshot. >diff --git a/src/box/wal.c b/src/box/wal.c >index 220e68245..d8c92aa36 100644 >--- a/src/box/wal.c >+++ b/src/box/wal.c >@@ -662,6 +662,7 @@ wal_begin_checkpoint_f(struct cbus_call_msg *data) >? } >? vclock_copy(&msg->vclock, &writer->vclock); >? msg->wal_size = writer->checkpoint_wal_size; >+ ERROR_INJECT_SLEEP(ERRINJ_WAL_DELAY); >? return 0; >?} >? >@@ -1272,6 +1273,9 @@ wal_write_async(struct journal *journal, struct journal_entry *entry) >? writer->last_entry = entry; >? batch->approx_len += entry->approx_len; >? writer->wal_pipe.n_input += entry->n_rows * XROW_IOVMAX; >+#ifndef NDEBUG >+ ++errinj(ERRINJ_WAL_WRITE_COUNT, ERRINJ_INT)->iparam; >+#endif >? cpipe_flush_input(&writer->wal_pipe); >? return 0; >? >diff --git a/src/lib/core/errinj.h b/src/lib/core/errinj.h >index aace8736f..814c57c2e 100644 >--- a/src/lib/core/errinj.h >+++ b/src/lib/core/errinj.h >@@ -82,6 +82,7 @@ struct errinj { >? _(ERRINJ_WAL_DELAY, ERRINJ_BOOL, {.bparam = false}) \ >? _(ERRINJ_WAL_DELAY_COUNTDOWN, ERRINJ_INT, {.iparam = -1}) \ >? _(ERRINJ_WAL_FALLOCATE, ERRINJ_INT, {.iparam = 0}) \ >+ _(ERRINJ_WAL_WRITE_COUNT, ERRINJ_INT, {.iparam = 0}) \ >? _(ERRINJ_INDEX_ALLOC, ERRINJ_BOOL, {.bparam = false}) \ >? _(ERRINJ_TUPLE_ALLOC, ERRINJ_BOOL, {.bparam = false}) \ >? _(ERRINJ_TUPLE_FIELD, ERRINJ_BOOL, {.bparam = false}) \ >diff --git a/test/box/errinj.result b/test/box/errinj.result >index 4bea0f46f..613d22c51 100644 >--- a/test/box/errinj.result >+++ b/test/box/errinj.result >@@ -114,6 +114,7 @@ evals >???- ERRINJ_WAL_ROTATE: false >???- ERRINJ_WAL_SYNC: false >???- ERRINJ_WAL_WRITE: false >+ - ERRINJ_WAL_WRITE_COUNT: 3 >???- ERRINJ_WAL_WRITE_DISK: false >???- ERRINJ_WAL_WRITE_EOF: false >???- ERRINJ_WAL_WRITE_PARTIAL: -1 >diff --git a/test/replication/gh-5167-qsync-rollback-snap.result b/test/replication/gh-5167-qsync-rollback-snap.result >new file mode 100644 >index 000000000..06f58526c >--- /dev/null >+++ b/test/replication/gh-5167-qsync-rollback-snap.result >@@ -0,0 +1,165 @@ >+-- test-run result file version 2 >+test_run = require('test_run').new() >+ | --- >+ | ... >+engine = test_run:get_cfg('engine') >+ | --- >+ | ... >+ >+orig_synchro_quorum = box.cfg.replication_synchro_quorum >+ | --- >+ | ... >+orig_synchro_timeout = box.cfg.replication_synchro_timeout >+ | --- >+ | ... >+box.schema.user.grant('guest', 'super') >+ | --- >+ | ... >+ >+test_run:cmd('create server replica with rpl_master=default,\ >+ script="replication/replica.lua"') >+ | --- >+ | - true >+ | ... >+test_run:cmd('start server replica with wait=True, wait_load=True') >+ | --- >+ | - true >+ | ... >+ >+-- >+-- gh-5167: >+-- >+fiber = require('fiber') >+ | --- >+ | ... >+box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} >+ | --- >+ | ... >+_ = box.schema.space.create('sync', {is_sync = true, engine = engine}) >+ | --- >+ | ... >+_ = box.space.sync:create_index('pk') >+ | --- >+ | ... >+-- Write something to flush the current master's state to replica. >+_ = box.space.sync:insert{1} >+ | --- >+ | ... >+_ = box.space.sync:delete{1} >+ | --- >+ | ... >+ >+box.cfg{replication_synchro_quorum = 3} >+ | --- >+ | ... >+ok, err = nil >+ | --- >+ | ... >+f = fiber.create(function() \ >+ ok, err = pcall(box.space.sync.insert, box.space.sync, {1}) \ >+end) >+ | --- >+ | ... >+ >+test_run:switch('replica') >+ | --- >+ | - true >+ | ... >+fiber = require('fiber') >+ | --- >+ | ... >+test_run:wait_cond(function() return box.space.sync:count() == 1 end) >+ | --- >+ | - true >+ | ... >+-- Snapshot will stuck in WAL thread on rotation before starting wait on the >+-- limbo. >+box.error.injection.set("ERRINJ_WAL_DELAY", true) >+ | --- >+ | - ok >+ | ... >+wal_write_count = box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") >+ | --- >+ | ... >+ok, err = nil >+ | --- >+ | ... >+f = fiber.create(function() ok, err = pcall(box.snapshot) end) >+ | --- >+ | ... >+ >+test_run:switch('default') >+ | --- >+ | - true >+ | ... >+box.cfg{replication_synchro_timeout = 0.0001} >+ | --- >+ | ... >+test_run:wait_cond(function() return f:status() == 'dead' end) >+ | --- >+ | - true >+ | ... >+ok, err >+ | --- >+ | - false >+ | - Quorum collection for a synchronous transaction is timed out >+ | ... >+box.space.sync:select{} >+ | --- >+ | - [] >+ | ... >+ >+test_run:switch('replica') >+ | --- >+ | - true >+ | ... >+-- Rollback was received. Note, it is not legit to look for space:count() == 0. >+-- Because ideally ROLLBACK should not be applied before written to WAL. That >+-- means count() will be > 0 until WAL write succeeds. >+test_run:wait_cond(function() \ >+ return box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") > wal_write_count \ >+end) >+ | --- >+ | - true >+ | ... >+-- Now WAL rotation is done. Snapshot will fail, because will see that a >+-- rollback happened during that. Meaning that the rotated WAL contains >+-- not confirmed data, and it can't be used as a checkpoint. >+box.error.injection.set("ERRINJ_WAL_DELAY", false) >+ | --- >+ | - ok >+ | ... >+test_run:wait_cond(function() return f:status() == 'dead' end) >+ | --- >+ | - true >+ | ... >+ok, err >+ | --- >+ | - false >+ | - A rollback for a synchronous transaction is received >+ | ... >+ >+test_run:switch('default') >+ | --- >+ | - true >+ | ... >+box.space.sync:drop() >+ | --- >+ | ... >+test_run:cmd('stop server replica') >+ | --- >+ | - true >+ | ... >+test_run:cmd('delete server replica') >+ | --- >+ | - true >+ | ... >+box.schema.user.revoke('guest', 'super') >+ | --- >+ | ... >+box.cfg{ \ >+ replication_synchro_quorum = orig_synchro_quorum, \ >+ replication_synchro_timeout = orig_synchro_timeout, \ >+} >+ | --- >+ | ... >diff --git a/test/replication/gh-5167-qsync-rollback-snap.test.lua b/test/replication/gh-5167-qsync-rollback-snap.test.lua >new file mode 100644 >index 000000000..475727e61 >--- /dev/null >+++ b/test/replication/gh-5167-qsync-rollback-snap.test.lua >@@ -0,0 +1,67 @@ >+test_run = require('test_run').new() >+engine = test_run:get_cfg('engine') >+ >+orig_synchro_quorum = box.cfg.replication_synchro_quorum >+orig_synchro_timeout = box.cfg.replication_synchro_timeout >+box.schema.user.grant('guest', 'super') >+ >+test_run:cmd('create server replica with rpl_master=default,\ >+ script="replication/replica.lua"') >+test_run:cmd('start server replica with wait=True, wait_load=True') >+ >+-- >+-- gh-5167: >+-- >+fiber = require('fiber') >+box.cfg{replication_synchro_quorum = 2, replication_synchro_timeout = 1000} >+_ = box.schema.space.create('sync', {is_sync = true, engine = engine}) >+_ = box.space.sync:create_index('pk') >+-- Write something to flush the current master's state to replica. >+_ = box.space.sync:insert{1} >+_ = box.space.sync:delete{1} >+ >+box.cfg{replication_synchro_quorum = 3} >+ok, err = nil >+f = fiber.create(function() \ >+ ok, err = pcall(box.space.sync.insert, box.space.sync, {1}) \ >+end) >+ >+test_run:switch('replica') >+fiber = require('fiber') >+test_run:wait_cond(function() return box.space.sync:count() == 1 end) >+-- Snapshot will stuck in WAL thread on rotation before starting wait on the >+-- limbo. >+box.error.injection.set("ERRINJ_WAL_DELAY", true) >+wal_write_count = box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") >+ok, err = nil >+f = fiber.create(function() ok, err = pcall(box.snapshot) end) >+ >+test_run:switch('default') >+box.cfg{replication_synchro_timeout = 0.0001} >+test_run:wait_cond(function() return f:status() == 'dead' end) >+ok, err >+box.space.sync:select{} >+ >+test_run:switch('replica') >+-- Rollback was received. Note, it is not legit to look for space:count() == 0. >+-- Because ideally ROLLBACK should not be applied before written to WAL. That >+-- means count() will be > 0 until WAL write succeeds. >+test_run:wait_cond(function() \ >+ return box.error.injection.get("ERRINJ_WAL_WRITE_COUNT") > wal_write_count \ >+end) >+-- Now WAL rotation is done. Snapshot will fail, because will see that a >+-- rollback happened during that. Meaning that the rotated WAL contains >+-- not confirmed data, and it can't be used as a checkpoint. >+box.error.injection.set("ERRINJ_WAL_DELAY", false) >+test_run:wait_cond(function() return f:status() == 'dead' end) >+ok, err >+ >+test_run:switch('default') >+box.space.sync:drop() >+test_run:cmd('stop server replica') >+test_run:cmd('delete server replica') >+box.schema.user.revoke('guest', 'super') >+box.cfg{ \ >+ replication_synchro_quorum = orig_synchro_quorum, \ >+ replication_synchro_timeout = orig_synchro_timeout, \ >+} >diff --git a/test/replication/suite.ini b/test/replication/suite.ini >index 73f73eb89..247289d58 100644 >--- a/test/replication/suite.ini >+++ b/test/replication/suite.ini >@@ -3,7 +3,7 @@ core = tarantool >?script = master.lua >?description = tarantool/box, replication >?disabled = consistent.test.lua >-release_disabled = catch.test.lua errinj.test.lua gc.test.lua gc_no_space.test.lua before_replace.test.lua qsync_advanced.test.lua qsync_errinj.test.lua quorum.test.lua recover_missing_xlog.test.lua sync.test.lua long_row_timeout.test.lua gh-4739-vclock-assert.test.lua gh-4730-applier-rollback.test.lua gh-5140-qsync-casc-rollback.test.lua gh-5144-qsync-dup-confirm.test.lua >+release_disabled = catch.test.lua errinj.test.lua gc.test.lua gc_no_space.test.lua before_replace.test.lua qsync_advanced.test.lua qsync_errinj.test.lua quorum.test.lua recover_missing_xlog.test.lua sync.test.lua long_row_timeout.test.lua gh-4739-vclock-assert.test.lua gh-4730-applier-rollback.test.lua gh-5140-qsync-casc-rollback.test.lua gh-5144-qsync-dup-confirm.test.lua gh-5167-rollback-snap.test.lua >?config = suite.cfg >?lua_libs = lua/fast_replica.lua lua/rlimit.lua >?use_unix_sockets = True >-- >2.21.1 (Apple Git-122.3) >? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alyapunov at tarantool.org Tue Aug 11 21:50:35 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Tue, 11 Aug 2020 21:50:35 +0300 Subject: [Tarantool-patches] [PATCH 2/2] tuple: fix access by JSON path starting from '[*]' In-Reply-To: <21871bd97383aa16c15d8bdad20775f2a85548ae.1596584571.git.v.shpilevoy@tarantool.org> References: <21871bd97383aa16c15d8bdad20775f2a85548ae.1596584571.git.v.shpilevoy@tarantool.org> Message-ID: LGTM On 8/5/20 2:45 AM, Vladislav Shpilevoy wrote: > Tuple JSON field access crashed when '[*]' was used as a first > part of the JSON path. The patch makes it treated like 'field not > found'. > > Follow-up #5224 > --- > src/box/tuple.c | 3 ++- > test/box/gh-5224-multikey-field-access.result | 9 +++++++++ > test/box/gh-5224-multikey-field-access.test.lua | 6 ++++++ > 3 files changed, 17 insertions(+), 1 deletion(-) From imun at tarantool.org Tue Aug 11 23:13:24 2020 From: imun at tarantool.org (Igor Munkin) Date: Tue, 11 Aug 2020 23:13:24 +0300 Subject: [Tarantool-patches] [RFC] rfc: luajit metrics In-Reply-To: <20200724172003.GF894@tarantool.org> References: <20200721113451.25817-1-skaplun@tarantool.org> <20200721113717.22804-1-skaplun@tarantool.org> <20200723101525.GC894@tarantool.org> <20200724111829.GA4086@root> <20200724172003.GF894@tarantool.org> Message-ID: <20200811201324.GY18920@tarantool.org> Sergos, On 24.07.20, Sergey Ostanevich wrote: > Hi! > > > On 24 Jul 14:18, Sergey Kaplun wrote: > > > > + /* > > > > + * Overall number of snap restores for all traces > > > > > > Snap restores needs some explanation. > > > > Ok, sounds reasonable. AFAIK number of snap restores equals to amount of > > trace exits. May be this will be more informative for users. > > > > Well. What this will tell to you as a user running a Lua code? Consider > the number is grows - ??? And what if it shrinks?? What does a pretty high lymphocyte percentage tell you? This might be either allergic reaction you're facing right now, or infection, or something else. You can say nothing considering only this one index. is just a single JIT-related index and it shows nothing per se. Since this index is absolute, it doesn't shrink. > > I believe it is tighly bound to the total number of traces and the size > of code. It's closely related to the overall number of side exits leading to VM. NB: side exits leading to the side trace are not counted here (since snapshot is "replayed" and the corresponding code is added prior to the side trace enter), but trace exits violating the loop condition guards are (this is the way trace execution leaves the recorded loop). > But what should it disclose to the user - how should it react, > refactor its code? Wild guess: there *might* be irrelevant traces. At the same time it may be just a noise, since "succeeded" trace exits are not counted (but we can implement it within in VM). Anyway, the user has to proceed with the degradation investigation at first. > > > > > > > + jit_snap_restore: 0 > > > > + gc_freed: 2239391 > > > > + strhash_hit: 53759 > > > > + gc_steps_finalize: 0 > > > > + gc_allocated: 3609318 > > > > + gc_steps_atomic: 6 > > > > + gc_steps_sweep: 296 > > > > + gc_steps_sweepstring: 17920 > > > > + jit_trace_abort: 0 > > > > + strhash_miss: 6874 > > > > > > I wonder if those keys can be sorted? Consider, I want to find > > > something by the name. Sorting helps greatly. > > > > It can be implement by user as simple as: > > > > | local metrics = xjit.getmetrics() > > | local sorted = {} > > | for k, v in pairs(metrics.incremental) do table.insert(sorted, k) end > > | table.sort(sorted) > > | for _, k in ipairs(sorted) do print(k, metrics.incremental[k]) end > > | > > | --[[ > > | gc_allocated 8920 > > | gc_freed 3528 > > | gc_steps_atomic 0 > > | gc_steps_finalize 0 > > | gc_steps_pause 0 > > | gc_steps_propagate 62 > > | gc_steps_sweep 0 > > | gc_steps_sweepstring 0 > > | jit_snap_restore 0 > > | jit_trace_abort 0 > > | strhash_hit 118 > > | strhash_miss 7 > > | ]] > > > > For me it is not that simple, frankly. Not in understanding - in use. > Can we just incorporate the code into the getmetrics()? No, yields a table (unordered map if you want), not a string. And you question relates to stats view, not its storage. I personally prefer inspect module[1], that serializes a table sorting its keys as you can see below: | $ LUA_PATH="$HOME/.luarocks/share/lua/5.1/?.lua;;" \ | ./luajit -e 'print(require("inspect")(misc.getmetrics()))' | { | cdatanum = 0, | gc_allocated = 93583, | gc_freed = 28408, | gc_steps_atomic = 0, | gc_steps_finalize = 0, | gc_steps_pause = 1, | gc_steps_propagate = 218, | gc_steps_sweep = 0, | gc_steps_sweepstring = 0, | gc_total = 65175, | jit_mcode_size = 0, | jit_snap_restore = 0, | jit_trace_abort = 0, | jit_trace_num = 0, | strhash_hit = 958, | strhash_miss = 447, | strnum = 447, | tabnum = 69, | udatanum = 4 | } > > Thanks, > Sergos > > > > > > > -- > > Best regards, > > Sergey Kaplun [1]: https://github.com/kikito/inspect.lua -- Best regards, IM From v.shpilevoy at tarantool.org Wed Aug 12 00:24:14 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 11 Aug 2020 23:24:14 +0200 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: <53095890-c0b6-6eae-b3b1-9898e7ca05e3@tarantool.org> References: <53095890-c0b6-6eae-b3b1-9898e7ca05e3@tarantool.org> Message-ID: <8d39159c-31d3-1394-fb04-4793da531772@tarantool.org> Hi! Thanks for the review! > On 8/5/20 2:45 AM, Vladislav Shpilevoy wrote: >> ????????????? goto parse; >> ????????? if (offset_slot_hint != NULL) >> ????????????? *offset_slot_hint = offset_slot; >> +??????? /* >> +???????? * When the field is multikey, the offset slot points not at the >> +???????? * data. It points at 'extra' array of offsets for this multikey >> +???????? * index. That array can only be accessed if index in that array >> +???????? * is known. >> +???????? */ >> +??????? if (field->is_multikey_part && multikey_idx == MULTIKEY_NONE) >> +??????????? goto parse; >> ? offset_slot_access: >> ????????? /* Indexed field */ >> ????????? offset = field_map_get_offset(field_map, offset_slot, > I'm sure that your check must be moved for two lines up. I mean the check > must be done before setting *offset_slot_hint. > > As I understood offset_slot_hint will contain a hint for further tuple_field_raw_by_path > calls with the same path. That is a kind of agreement, we may call tuple_field_raw_by_path > twice and must get the same results. > > But in your code you set *offset_slot_hint before a check that could go to 'parse' label. > Meanwhile in the second call of tuple_field_raw_by_path it'll check *offset_slot_hint and > will go to 'offset_slot_access' label. That's wrong. You would be right if not the fact that there is always a guarantee, that if offset_slot_hint != NULL, then either multikey_idx != MULTIKEY_NONE or it is not a multikey part. It is unreachable. So it wouldn't be correct to put it 2 lines above, nor it wouldn't be incorrect - it does not change anything. But it is possible to put it *instead*. Into 'else' branch. Then it will be -1 condition check. New patch for this file: ==================== diff --git a/src/box/tuple.h b/src/box/tuple.h index 4752323e4..09ebeecf3 100644 --- a/src/box/tuple.h +++ b/src/box/tuple.h @@ -626,8 +626,28 @@ tuple_field_raw_by_path(struct tuple_format *format, const char *tuple, offset_slot = field->offset_slot; if (offset_slot == TUPLE_OFFSET_SLOT_NIL) goto parse; - if (offset_slot_hint != NULL) + if (offset_slot_hint != NULL) { *offset_slot_hint = offset_slot; + /* + * Hint is never requested for a multikey field without + * providing a concrete multikey index. + */ + assert(!field->is_multikey_part || + (multikey_idx != MULTIKEY_NONE && + field->is_multikey_part)); + } else if (field->is_multikey_part && + multikey_idx == MULTIKEY_NONE) { + /* + * When the field is multikey, the offset slot points + * not at the data. It points at 'extra' array of + * offsets for this multikey index. That array can only + * be accessed if index in that array is known. It is + * not known when the field is accessed not in an index. + * For example, in an application's Lua code by a JSON + * path. + */ + goto parse; + } offset_slot_access: /* Indexed field */ offset = field_map_get_offset(field_map, offset_slot, From v.shpilevoy at tarantool.org Wed Aug 12 02:01:55 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 12 Aug 2020 01:01:55 +0200 Subject: [Tarantool-patches] [PATCH 1/1] box: snapshot should not include rolled back data In-Reply-To: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> References: <6fc5876bfde067f19973a5d701b55bf4826f2a9e.1597097775.git.v.shpilevoy@tarantool.org> Message-ID: Pushed to master and 2.5. From imun at tarantool.org Wed Aug 12 12:39:32 2020 From: imun at tarantool.org (Igor Munkin) Date: Wed, 12 Aug 2020 12:39:32 +0300 Subject: [Tarantool-patches] [PATCH 1/2] metrics: add counters for metrics interested in In-Reply-To: <20200727022546.GA11579@root> References: <20200721113451.25817-1-skaplun@tarantool.org> <20200721113451.25817-2-skaplun@tarantool.org> <20200723201219.GQ18920@tarantool.org> <20200727022546.GA11579@root> Message-ID: <20200812093932.GZ18920@tarantool.org> Sergey, On 27.07.20, Sergey Kaplun wrote: > Igor, > > On 23.07.20, Igor Munkin wrote: > > Sergey, > > > > Thanks for the patch! Please consider my comments below. > > > > On 21.07.20, Sergey Kaplun wrote: > > > > > > > This counter is not incremented if cdata is allocated on trace. Consider > > the following test: > > | $ LUA_PATH="$HOME/.luarocks/share/lua/5.1/?.lua;;" ./luajit -joff < > | heredoc> local t = { } > > | heredoc> for i = 1, 1000 do > > | heredoc> table.insert(t, require('ffi').new('uint64_t', i)) > > | heredoc> end > > | heredoc> t = nil > > | heredoc> collectgarbage('collect') > > | heredoc> print(require('inspect')(misc.getmetrics())) > > | heredoc> EOF > > | { > > | global = { > > | cdatanum = 0, > > | gc_total = 74722, > > | jit_mcode_size = 0, > > | jit_trace_num = 0, > > | strnum = 534, > > | tabnum = 77, > > | udatanum = 5 > > | }, > > | incremental = { > > | gc_allocated = 173092, > > | gc_freed = 98370, > > | gc_steps_atomic = 1, > > | gc_steps_finalize = 0, > > | gc_steps_pause = 3, > > | gc_steps_propagate = 553, > > | gc_steps_sweep = 64, > > | gc_steps_sweepstring = 1024, > > | jit_snap_restore = 0, > > | jit_trace_abort = 0, > > | strhash_hit = 4021, > > | strhash_miss = 538 > > | } > > | } > > | $ LUA_PATH="$HOME/.luarocks/share/lua/5.1/?.lua;;" ./luajit -jon < > | heredoc> local t = { } > > | heredoc> for i = 1, 1000 do > > | heredoc> table.insert(t, require('ffi').new('uint64_t', i)) > > | heredoc> end > > | heredoc> t = nil > > | heredoc> collectgarbage('collect') > > | heredoc> print(require('inspect')(misc.getmetrics())) > > | heredoc> EOF > > | { > > | global = { > > | cdatanum = -941, > > | gc_total = 76561, > > | jit_mcode_size = 65536, > > | jit_trace_num = 3, > > | strnum = 534, > > | tabnum = 77, > > | udatanum = 5 > > | }, > > | incremental = { > > | gc_allocated = 145299, > > | gc_freed = 68738, > > | gc_steps_atomic = 3, > > | gc_steps_finalize = 0, > > | gc_steps_pause = 2, > > | gc_steps_propagate = 505, > > | gc_steps_sweep = 64, > > | gc_steps_sweepstring = 1024, > > | jit_snap_restore = 6, > > | jit_trace_abort = 0, > > | strhash_hit = 3085, > > | strhash_miss = 538 > > | } > > | } > > > > As you can see global.cdatanum is negative when JIT is on. You can take > > a look on the compiled trace by yourself. The root cause is CNEW IR > > semantics: for VLA/VLS cdata types call is emitted and > > the counters are fine, but for other cdata types JIT compiler emits just > > a call that so this object is not counted. Thereby JIT > > has to emit the corresponding instructions while assembling CNEW IR. > > I will elaborate on this later. > > Can we use an approach like this? Kinda. > > diff --git a/src/lj_asm_x86.h b/src/lj_asm_x86.h > index 3e189b1..a32a8c4 100644 > --- a/src/lj_asm_x86.h > +++ b/src/lj_asm_x86.h > @@ -1835,6 +1835,12 @@ static void asm_cnew(ASMState *as, IRIns *ir) > return; > } > > + /* Global state live longer than given trace. Global state is a Lua universe, so this line is too obvious. Typo: s/live/lives/. > + ** Increment cdatanum counter by address directly. > + */ > + emit_gmroi(as, XG_ARITHi(XOg_ADD), RID_NONE, > + ptr2addr(&J2G(as->J)->gc.cdatanum), (int32_t)1); I see no reason for explicit cast here: * the parameter is type * IIRC, integral literals are 32 bit integer types There is also an additional check within and imm8 addition is emitted for your case. Besides, other emitters are still left unpatched. > + > /* Combine initialization of marked, gct and ctypeid. */ > emit_movtomro(as, RID_ECX, RID_RET, offsetof(GCcdata, marked)); > emit_gri(as, XG_ARITHi(XOg_OR), RID_ECX, > > > > > > > > -- > > Best regards, > > IM > > I've fixed your other comments in [1]. Great, thanks! > > [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-July/018866.html > > -- > Best regards, > Sergey Kaplun -- Best regards, IM From yaroslav.dynnikov at tarantool.org Wed Aug 12 15:59:21 2020 From: yaroslav.dynnikov at tarantool.org (Yaroslav Dynnikov) Date: Wed, 12 Aug 2020 15:59:21 +0300 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported In-Reply-To: <4a43127c-6e11-8f28-27ee-62e24fa65a4b@tarantool.org> References: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> <4a43127c-6e11-8f28-27ee-62e24fa65a4b@tarantool.org> Message-ID: Hi, Vlad. Number 7 is the libcurl age. According to the documentation, age should be set to the version of this functionality by the time you write your program. This way, libcurl will always return a proper struct that your program understands, while programs in the future might get a different struct. See https://curl.haxx.se/libcurl/c/curl_version_info.html So it's safe enough to rely on the structure. And I think both checkes (libssl and libz) are important enough to be tracked by tests. We don't want them to be silently gone, do we? Best regards Yaroslav Dynnikov On Tue, 11 Aug 2020 at 01:49, Vladislav Shpilevoy wrote: > Hi! Thanks for the patch! > > > diff --git a/test/box-tap/gh-5223-curl-exports.test.lua > b/test/box-tap/gh-5223-curl-exports.test.lua > > new file mode 100755 > > index 000000000..300d60b07 > > --- /dev/null > > +++ b/test/box-tap/gh-5223-curl-exports.test.lua > > @@ -0,0 +1,177 @@ > > +#!/usr/bin/env tarantool > > + > > +local tap = require('tap') > > +local ffi = require('ffi') > > +ffi.cdef([[ > > + void *dlsym(void *handle, const char *symbol); > > + struct curl_version_info_data { > > 1. I don't think you need to copy-paste the entire structure. Anyway it is > returned from curl_version_info() by pointer, so you can just announce the > pointer type. And in the test check that ffi.C.curl_version_info(7) > returned > something not nil. I don't think you need to access any fields of this > struct. It is going to lead to UB in case the struct will change anyhow in > the sources. > > > + int age; /* see description below */ > > + const char *version; /* human readable string */ > > + unsigned int version_num; /* numeric representation */ > > + const char *host; /* human readable string */ > > + int features; /* bitmask, see below */ > > + char *ssl_version; /* human readable string */ > > + long ssl_version_num; /* not used, always zero */ > > + const char *libz_version; /* human readable string */ > > + const char * const *protocols; /* protocols */ > > + > > + /* when 'age' is CURLVERSION_SECOND or higher, the members > below exist */ > > + const char *ares; /* human readable string */ > > + int ares_num; /* number */ > > + > > + /* when 'age' is CURLVERSION_THIRD or higher, the members below > exist */ > > + const char *libidn; /* human readable string */ > > + > > + /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the > members > > + below exist */ > > + int iconv_ver_num; /* '_libiconv_version' if iconv > support enabled */ > > + > > + const char *libssh_version; /* human readable string */ > > + > > + /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the > members > > + below exist */ > > + unsigned int brotli_ver_num; /* Numeric Brotli version > > + (MAJOR << 24) | (MINOR << 12) | > PATCH */ > > + const char *brotli_version; /* human readable string. */ > > + > > + /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the > members > > + below exist */ > > + unsigned int nghttp2_ver_num; /* Numeric nghttp2 version > > + (MAJOR << 16) | (MINOR << 8) | > PATCH */ > > + const char *nghttp2_version; /* human readable string. */ > > + > > + const char *quic_version; /* human readable quic (+ HTTP/3) > library + > > + version or NULL */ > > + > > + /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the > members > > + below exist */ > > + const char *cainfo; /* the built-in default > CURLOPT_CAINFO, might > > + be NULL */ > > + const char *capath; /* the built-in default > CURLOPT_CAPATH, might > > + be NULL */ > > + }; > > + > > + struct curl_version_info_data *curl_version_info(int age); > > +]]) > > + > > +local info = ffi.C.curl_version_info(7) > > 2. Why '7'? > > > +local test = tap.test('curl-features') > > +test:plan(3) > > + > -------------- next part -------------- An HTML attachment was scrubbed... URL: From olegrok at tarantool.org Wed Aug 12 16:03:50 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Wed, 12 Aug 2020 16:03:50 +0300 Subject: [Tarantool-patches] [PATCH 1/2] src: return back import of table.clear() method In-Reply-To: <1f495519687c8e037c638ccabab28e23882a41df.1595943364.git.sergeyb@tarantool.org> References: <1f495519687c8e037c638ccabab28e23882a41df.1595943364.git.sergeyb@tarantool.org> Message-ID: <0f75a16f-c1a7-992b-29fb-54358861cd70@tarantool.org> Hi! Thanks for your patch! I think it shouldn't be placed in "src/lua/trigger.lua". I believe "src/lua/table.lua" is more appropriate place. Of course with comment why it should be done e.g. "This require modifies global "table" module and adds "clear" function to it". On 28/07/2020 16:52, sergeyb at tarantool.org wrote: > From: Sergey Bronnikov > > Import of 'table.clear' module has been removed > to fix luacheck warning about unused variable in > commit 3af79e70b5e1e9b1d69b97f3031a299132a02d2f > and method table.clear() became unavailable in Tarantool. > > Part of #5210 > --- > src/lua/trigger.lua | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/src/lua/trigger.lua b/src/lua/trigger.lua > index 1330ecdd4..066329ea6 100644 > --- a/src/lua/trigger.lua > +++ b/src/lua/trigger.lua > @@ -1,4 +1,5 @@ > local fun = require('fun') > +local _ = require('table.clear') BTW, "require('table.clear')" should be enough without "local _ =" > > -- > -- Checks that argument is a callable, i.e. a function or a table From alyapunov at tarantool.org Wed Aug 12 16:05:13 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Wed, 12 Aug 2020 16:05:13 +0300 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: <8d39159c-31d3-1394-fb04-4793da531772@tarantool.org> References: <53095890-c0b6-6eae-b3b1-9898e7ca05e3@tarantool.org> <8d39159c-31d3-1394-fb04-4793da531772@tarantool.org> Message-ID: <184a98de-9f49-be82-39a5-84b3c7510697@tarantool.org> Hi! thanks for the fix, as for me it became more understandable. See one minor comment below and LGTM. On 8/12/20 12:24 AM, Vladislav Shpilevoy wrote: > Hi! Thanks for the review! > > + if (offset_slot_hint != NULL) { > *offset_slot_hint = offset_slot; > + /* > + * Hint is never requested for a multikey field without > + * providing a concrete multikey index. > + */ > + assert(!field->is_multikey_part || > + (multikey_idx != MULTIKEY_NONE && > + field->is_multikey_part)); The last '&& field->is_multikey_part' is excess. > + } else if (field->is_multikey_part && > + multikey_idx == MULTIKEY_NONE) { > + /* > + * When the field is multikey, the offset slot points > + * not at the data. It points at 'extra' array of > + * offsets for this multikey index. That array can only > + * be accessed if index in that array is known. It is > + * not known when the field is accessed not in an index. > + * For example, in an application's Lua code by a JSON > + * path. > + */ > + goto parse; > + } > offset_slot_access: > /* Indexed field */ > offset = field_map_get_offset(field_map, offset_slot, From imeevma at tarantool.org Wed Aug 12 18:15:42 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:42 +0300 Subject: [Tarantool-patches] [PATCH v1 0/7] sql: properly check arguments types of built-in functions Message-ID: This patch-set makes SQL to use the ApplyType opcode to validate the argument types of built-in functions. https://github.com/tarantool/tarantool/issues/4159 https://github.com/tarantool/tarantool/tree/imeevma/gh-4159-rework-sql-builtins @ChangeLog - Built-in function argument types are now properly checked (gh-4159). Mergen Imeev (7): box: add has_vararg option for functions sql: do not return UNSIGNED in built-in functions sql: move built-in function definitions in _func box: add param_list to 'struct func' sql: check built-in functions argument types sql: VARBINARY and STRING in built-in functions sql: refactor sql/func.c src/box/alter.cc | 12 +- src/box/bootstrap.snap | Bin 5976 -> 6291 bytes src/box/func.c | 1 + src/box/func_def.c | 5 + src/box/func_def.h | 8 + src/box/lua/call.c | 2 + src/box/lua/upgrade.lua | 143 +++ src/box/sql/expr.c | 4 + src/box/sql/func.c | 894 +++---------------- src/box/sql/resolve.c | 2 +- src/box/sql/select.c | 26 + src/box/sql/sqlInt.h | 14 + src/box/sql/vdbeapi.c | 2 +- test/box-py/bootstrap.result | 2 +- test/sql-tap/cse.test.lua | 4 +- test/sql-tap/func.test.lua | 36 +- test/sql-tap/orderby1.test.lua | 2 +- test/sql-tap/position.test.lua | 6 +- test/sql/boolean.result | 32 +- test/sql/checks.result | 8 - test/sql/checks.test.lua | 2 - test/sql/types.result | 1489 +++++++++++++++++++++++++++++++- test/sql/types.test.lua | 250 ++++++ 23 files changed, 2092 insertions(+), 852 deletions(-) -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:45 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:45 +0300 Subject: [Tarantool-patches] [PATCH v1 1/7] box: add has_vararg option for functions In-Reply-To: References: Message-ID: The has_vararg option allows us to work with functions with a variable number of arguments. This is required for built-in SQL functions. Suppose this option is TRUE for a built-in SQL function. Then: 1) If param_list is empty, all arguments can be of any type. 2) If the length of param_list is not less than the number of the given arguments, the types of the given arguments must be compatible with the corresponding types described in param_list. 3) If the length of param_list is less than the number of given arguments, the rest of the arguments must be compatible with the last type in param_list. Part of #4159 --- src/box/func_def.c | 5 +++++ src/box/func_def.h | 6 ++++++ 2 files changed, 11 insertions(+) diff --git a/src/box/func_def.c b/src/box/func_def.c index 11d2bdb84..be12ce970 100644 --- a/src/box/func_def.c +++ b/src/box/func_def.c @@ -40,10 +40,13 @@ const char *func_aggregate_strs[] = {"none", "group"}; const struct func_opts func_opts_default = { /* .is_multikey = */ false, + /* .has_vararg = */ false, }; const struct opt_def func_opts_reg[] = { OPT_DEF("is_multikey", OPT_BOOL, struct func_opts, is_multikey), + OPT_DEF("has_vararg", OPT_BOOL, struct func_opts, has_vararg), + OPT_END, }; int @@ -51,6 +54,8 @@ func_opts_cmp(struct func_opts *o1, struct func_opts *o2) { if (o1->is_multikey != o2->is_multikey) return o1->is_multikey - o2->is_multikey; + if (o1->has_vararg != o2->has_vararg) + return o1->has_vararg - o2->has_vararg; return 0; } diff --git a/src/box/func_def.h b/src/box/func_def.h index d99d89190..89d5a404a 100644 --- a/src/box/func_def.h +++ b/src/box/func_def.h @@ -68,6 +68,12 @@ struct func_opts { * packed in array. */ bool is_multikey; + /** + * TRUE if the function can have a variable number of arguments. + * + * Currently only used in built-in SQL functions. + */ + bool has_vararg; }; extern const struct func_opts func_opts_default; -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:47 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:47 +0300 Subject: [Tarantool-patches] [PATCH v1 2/7] sql: do not return UNSIGNED in built-in functions In-Reply-To: References: Message-ID: <3d38cd163a24bf14298a16ee7231086a4df56015.1597244875.git.imeevma@gmail.com> This patch forces functions to return INTEGER instead of UNSIGNED. Part of #4159 --- src/box/sql/vdbeapi.c | 2 +- test/sql/types.result | 12 ++++++++++++ test/sql/types.test.lua | 6 ++++++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c index 7c59ef83f..d1eeaf114 100644 --- a/src/box/sql/vdbeapi.c +++ b/src/box/sql/vdbeapi.c @@ -328,7 +328,7 @@ sql_result_double(sql_context * pCtx, double rVal) void sql_result_uint(sql_context *ctx, uint64_t u_val) { - mem_set_u64(ctx->pOut, u_val); + mem_set_int(ctx->pOut, u_val, false); } void diff --git a/test/sql/types.result b/test/sql/types.result index 442245186..95f7713e8 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2795,3 +2795,15 @@ box.execute([[DROP TABLE ts;]]) --- - row_count: 1 ... +-- +-- gh-4159: Make sure that functions returns values of type INTEGER +-- instead of values of type UNSIGNED. +-- +box.execute([[SELECT typeof(length('abc'));]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 0270d9f8a..fff0057bd 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -623,3 +623,9 @@ box.execute([[DROP TABLE tb;]]) box.execute([[DROP TABLE tt;]]) box.execute([[DROP TABLE tv;]]) box.execute([[DROP TABLE ts;]]) + +-- +-- gh-4159: Make sure that functions returns values of type INTEGER +-- instead of values of type UNSIGNED. +-- +box.execute([[SELECT typeof(length('abc'));]]) -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:48 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:48 +0300 Subject: [Tarantool-patches] [PATCH v1 3/7] sql: move built-in function definitions in _func In-Reply-To: References: Message-ID: <202a7e7207c04e0196964afca93427dbf208ceec.1597244875.git.imeevma@gmail.com> This patch moves SQL built-in function definitions to _func. This helps create an unified way to check the types of arguments. It also allows users to see these definitions. Part of #4159 --- src/box/bootstrap.snap | Bin 5976 -> 6291 bytes src/box/lua/upgrade.lua | 143 +++++++++++++++++++++++++++++++++++ src/box/sql/func.c | 7 +- src/box/sql/resolve.c | 2 +- test/box-py/bootstrap.result | 2 +- test/sql/types.result | 4 +- 6 files changed, 148 insertions(+), 10 deletions(-) diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap index 8bd4f7ce24216a8bcced6aa97c20c83eb7a02c77..9163165b13b27771414243650051e900b3af8def 100644 GIT binary patch literal 6291 zcmV;E7;NWLPC-x#FfK7O3RY!ub7^mGIv_GGHZCwNGc+w{V`5}vGBPz{W(rAeb97;D zV`VxZWMnmFWnwZnEi*P`Gc7bUF)=MTV`MTdGiG8mH#svmHDoj}3RXjGZ)0mZAbWiZ z3e~y`y3H6G0M66h;g_WV00000D77#B08ov<04mfRFi8+_T4gZIFvAQp%phAsj|nab zs5-VJi8n60MkAamBQlQBNCoHrH^?UiRPEJsf;YUCfCAn|Fad9Q3xNa=L@C+a&NsW8>y)xaO~6w8M6IEMPd4ni>wDQODuc at M)CFY2K)u<{NXG zoW{R?_wP59_uKDc7j73yxOq{yg`4Oe*_I#KA at Rr_*);9T?%~U_X8JZm*nVFs5Z&|aYJ!psL3kJvZg25+V at VOh)LUm6wsF+>5gGp#k zN9zhBW(UFKcCgOw!iCvEConts1-FAc`1Ilw4Boy3gH2w7!Kdx^KU35HHQWEsE=;sN zTxwgoths-$w=#G4-)rtaO}*Zug-ez#T4T|o^_48HUc%yPr7Ny>wUs5Ut0-wr#fsL? zDOG?TN}~ew(AgBAUz0!&O`_5EAkAibq$K%jNxs^iC`hy}C{ku!PofN6HGR`tQ>s_4 zq3Z%QciCc^>Mkcx^4jD{UXPsQHLpfZj!ce@$0bI`qY@@Q$rA68DDnPClB_;uNV57Y zI+E3^tUis%Xk0`^Gp-^=nr%PYxQGz>Aw|3wLd5%_AzVlZ=Rt-z2O^|_KZ0mr5D*R2 zKp#J*rg{WTP4VzCH3ihe`yD*I*8${xjvc+@(9!#i9KGAXK_u3=K_t?!K_uqqj2bi< zG*<4a+GEXP%n0VIB%cOz`h+3l*fZZi=1&ab?#1R-)2tYbpBT)|vFH7}{!aD_x$fUP z+0gmfa7nCJSKa(=Vk=qD&rb~IW(}P)yqXTj#KfArTnV$11^vWeZelTF+}*xwV>1r^ ziNW0T!O?Z6Qklp^%78Eo_5x5F0a7FlB))K{xd0;uYZK|(>K)YLTgWw2PRhhWL!$IH z`lD|eQ!V^MNHG(W{ufJjsf)+2 zTz9I~T=Y&&%v{+94qkU!eBCZ|vVrN=Wru2Rfmr(MRJh}l2T%_^bo=)^c+l?X0Gi#&v11?kF5bbRgVuE;$E?c+4(Ke2HUkQ~ zYTTgVprD#8s+ at yMXTPQ{qPPQj4DPO(!zF=#Oy#nBnY(VycHhNC!2 z>+ at wL3&PQ;sOJ|j9F2N<(T4`@;)@xS3omk1GDktT=)yH>!G&wgVhh*k8nMt~8?VS> z8?C@%8 at I8FEBZ@?75$x}ivIRj3M$?=iYeX~3Mt;_6H(B(oPc7+)x;BD at 0IZ7V#103 zmuRAYC79^H|0I_99|}Nj^Lj3#DyEZ6fU)|cl=4`n*}vdOuU z$$4 at KKu+iCZ@|I-8qoPagAFn=9I)UhP;}SO^Icsxi#%Ujd`Icpzfqz{xxf!F_5tfbW~9}u zv0xH^^#j|CR|1f0>cj*11OO8Nb^^c&4eoNLiq!7_AZ zXlY0(78A09kyL!JP!%C6Kc at VkVLS^;f+4BA^b585%VeW<|4JATj#|lzka$jd9eeI> zk*Gfwo0wL&v1{M-LcagkKIQKAUi-8ItVT-H2}cY6C at nB^o%)@%x~lT~q?MXNaSKJO zb=9oA7GLgSsXr5!T3q*B#9G_Us&hWyxqr`HH5~?1C|;lBqH at _a*TU5AEv>3L*ym)4 z)IY2I?LWoBV4A=Y3U~jh6FVKfO62!%tENue#mKmOafyjgx7)aT5v%7ms~hX9It|d( z(%StzcVW}Gdof*Nr6pBe``4HIN at c!@$*zU^^W7}! z at 0D}7exbBau`rP$a*Syc4-xs9*xevgnA_m;^Vd}aM!vvabouEb28N5JEt0f2%0h+( z(G^2h;2$^|griF;Y*R2vQH!Dr6kMLz;)IqavM_;Vi7QGhC830bl_RPcL8XW(M9_wq z36T^85)e{;h~fj39-i>vWQQg?9OYny1Ctw;*r23_Bs3tI;fM^f7>Zyp@&Zw`$QKb8 zhO{7rg`pLMR0xs+5EOo#5ai{9PY)A#lmJA+juCW-kRt>fAl%qvLr;w^8hK{mk#Q%+ z2QWPN;fi&DW4KDDgrj}>^mzf-5rS#owQ=*;355}Z;VPNZv+LsPm)OXby4#}eTKGpx zw{BOey9K^QCXQ9fkZ_c5;+mHjSILmDvVLR&e|=v#I|P7iJ6$SUoMHHTDdMdGCEqb+Zz3 at z0XBO zM|yQtqbWMnmVfizMalZN2*q5IUklTpzfW5zpFZ)HE8WkDwQ#`1S?KJXaJgF;TrLY^ z*`P`H+x43GiUz&jEyu)58Te{9-j^m;!eG}^Vk;NeYCNBrTN78Yz}Lf>xiK-7IlLR0 z+Y(Qyz>BJP%hh%^C#K0UQxlWKGb@~!Bu=XG+VO5^oERw%5|_*UbWAJ-4xgTlSWLSd zQx=YbBvVUbC{#7y&bP!*-u!w=?1Tc|n4S_hnc!)Ok0i)lVWSAl+>)3H1YHA(FK1Hq zm}*J9V2P77;HCTRY-yPt4+|qeJnLyq|3g_b__l`Fc*wB-#o&BOyu;U;lvqc9bEej$tKo7# zFjzPT)Dr2d*>pQw`t;lJzHDI}MPHBU*JI)vE*x7e_jKbmu?_Cq{l+W+JJZn8hTr4x|Mybfh&wP5CyZ at l^U&=<_w9=x`vXk at lVJty?`5V`RFq at QY+uH zJ>M}s)?ppC=Q_S;`>fA<)?s?4=eoSdy1dJFY{&OZk9T;7?^(xoe3$oGpLdv^>6k9- z at h98K_ at g3K*9p7bL*5y5>=Q^&-d)#GRzT(yn%7XX8xZgGfR<`oR# zGq+1rTV!BHt9kfc+2WC5Ro)U$W(Ks-5Vw at IK_o18jRhZ`5K6trOBwG42~lNdMMp7Y z`DTTrLvLg)m at 23W!2r`x704MfMUS)#S68$t)tR_7IvKv)jlsnV{ytA2=k{?rVd^7< z-GKH7h$J9Rs81uq#n@@gpurDZNY$Iq>#bPu1_&c%wGWA)?^J?F5>J at 6iOrICQK+Tt zL-kkEWH at f2sHCX=;mATa+^w?x;|GcOPkLt6B?}k~`n2j%9T4(sMx$(z#R>pAHHYm% zyCqXfh?X-k_eZE6tg%*s^PXj^SU~?QO(z-n*P0?_@|zK%2I%TX_#&PFypLBr at ymgH z1tf;)lE*HLMlcE>QqW~v%IwO4aAyx5VDBGp9&pz-s2pFPi$NAI0_sPBdR*m`!%FKT zs~0p}X6l!@H95CEE^Xc*M#Cm2F at K9}$A=Y18gw>TKmwW;d7e=QL6BZ$@ZcZvj`}}M z+6tPYNL&E>Sv$x^V>Yv9`o=(i+`Xc$i~#20~ZYHL%tiy9;}qO0uI_VM0N; z%JL-ok5MfhFpH`Wb9*=P(UGi2Z0Y!xK)J3GHWnzqWpF;VAMC=pRjcYf#mrw~lhW at qI^ z#I0G3{&dJ^qK<(ii=HEXGPAWU_IJ?(4KSN2b6wuPpuEoPKlclSBbO$GZZRVKk)lwa zMfIhsafyNKc30)>t%4(^<**P0kM3uS$Br?jqga8g7#=_y)QGKRdVnP;jp7f{yZ}I# zy4GXy at y&oiXfuVNFVDLzfj?OWJDng{yeTf1J at 0vkQ%$Q5loHNM88h zknbywHwK at dT#v*>3PLD5k?-h#vtMM0?dgFKqL8b^sfEN3#`q&z(QP|E%qAZEPiM3= zOn4R=#_Yc;6gMKMhaQkQT#VfeCRX z1}&sdI`*WvK0k(vcoHuA5z}lvv4`nAIR90)uo#1WRz>A6gm?}t{V61smc61pUt#&J zBG0>CNyrSINO>C^oMzz;b8MAXTwd|r at V+!V0+O>Oa^WT at RfHL^9Jvw@^Z9bG8lMBv zR6^w7`#DYu_5z%@{e9F}-^iIW1Dd4Ab1nO*!ZuR zzo)#T`{uXp!EnM)%=H-hu7^ZRfkiC6EZ`K(trj z`29i?R-e{ z1?CF`#Q$HJd7+pB@&=?!^GEDZ_}^HCU=a2}Y&iZrjs at -?hPWv%rkg8mr5rVJ=*~29 zujSkVTn$foc&p5t(k#xZ5*@Ezz{B_+9Z3(@mT<6Pz&C-L;SOyZpL7myTTmb0^60R8 zx3yOpaM5`=2N9&Ql^3BUq(5ru3k_DRQDQ#eIkE5c%wqg1q0z-DxX{>jh8Fy#Y83I^ zw>XK-z~Fm_Vg|%1bs<_{y(Qddp^nKrcjo^se(OP;AmF-_5c8anln(_}Bq$9VYp{pK z6Vfs|)AJJha(K6TPw}NozS1q455AheeDftI$(RbJq?A9i4VFYSN0+%sEuA+zEc^OX zpaeP{@MzK56QWp6oQLB)9u7e%@%@1vF3v>9?!mx>$fzskHdA4wH7sdn=Z|PB=1K5| zK;e9X6W|@vLjy^@sCJdrm?Kqq!N at hEi)kN;AYP0ztK)X7j3(;E7f48Mhpndx0TCyk z_pV03qQ_jy at IygM^7K*>sqSycOQDs#;ara*L=>Tt=E7A~EGa7x$ef~GqgEwt1n==o zAuIukKNQ`IhU7s>ehKYFb1KnQA|^MJ{h_at at +v~EQU<(rIw28}3Sx`ng$ox at K??>z zswB7uG5W8XK#;h2Lp-G9oHBT{WPyX^jzhR8xCk at 4qnJXFI^`G&i|^NpKzHJsd&i2J zq8bOXa}P at ba6uTC4u-houwfXhsvAOnZ90@O70eHT#37EMxXFH-u+~pg6QS)Tt1YO9 zhd-$pOU6M<@yMO7ppyoJmMmnk+}`YuieZlSEsT}|E{O7!$q|O$kq-!xxW=!DJCG5m zTm5X at d5l2a*qo23M=A>#jo{SrRLX&Rt8n%c~ zKcpNY8gF6%B2o};ha2qICK$>bUl1;DG9-*HYm%7d+}+qu#qngDizFjCIIa)OTBuN7 zA>tO*x#gfoV4no)!!1%KI67~@lW at D4Piq0_hsQKbsGtA{9fj*4Mtu}B4l+0Fh?A?l zk%p0#JYZ0ku}Bl8`yfX36axkm7c7XMtGv;MlavH#P at eHfm8I(-Mt>Aj2oe`;h?lCI zQHGb4EO1bUF-VoA;~+-+G(!fH8`g-CUEWB;Nm>H9C=2f*x6+-GauDrB@&TBOq}p_B z+T-5gD0`QkI+GPl#}gSV_9WVnD)x?J?pl!tmZ8&xn~mAn4U$yo$b_q=MwC4#5y}26|J20L$x{Ty at L4Qqb_762URRB+F#d6h zhUSQXZsAgzy;3M75-ukpn>4hvt^l)h1>AjmyoF?F;*ZExp2&nnIs#E>{7qoJ;5zB!rmqpc_;~A#5>??KA0g7q%tZ at C z4P#s!-Z958)-zJSErZ5jDf30~J_84zZsoz~0UVn`Q49 z;zd`Gj&2qhjCcEtCBzzOms|@vqm9GX`@FsNYCB8Bp;gtVcIXM|1(XV`tw1bsIJ!RB z7pmU*WCfeu#ipg;r=574oSPOSHZofO>L!#AT0ykS(3KiD1uAk>a>)BWLqhbOH&m at q zL)ZXS9tUZAcmRMDMjr*pgDNvVKx5)@i^_x#&=@!lybwj~b{c9N44Kf!z4 at ySb3O)7 z7znyRy|IDQ+)U;QByD4A7i$Z#nj4$vvyOlVtDb_veBR7}=5FDlAd}7;kdX}B&`{1D z-;F7Z2 at qq^VJew!OB7FBX`Ewapa#W8niT_wk&-}<9kvl{Tzhi8*%t&{Q1lX*4EIe; z=>FFn1(AHb$uQV3L(x%%0E7txHrPJu at 6a4=_z}eh*7AE+YKb`B<<` zj7K-QX(8mLPnAItuVMmDiEmgJ$y;G3S*gl!GXk7qep^O>$h|!3_wM6lL?1E(>usrV JfHuFnz!wkj#k+nW%pPgt zBZ0)WtpN`tMI1&W;^gZsQ&N;l)}E- at Bxx#$t*xi7PvyfprWdR(4G!J07hhyME2Rh12qO5 z03$jVXaJ1pRDgk60$>Z!*eC$j5|Pg}34qnqumK~c0b2t`L_iG~k?1 at i4HzdHfB+|Q zBANqOVgUxQ#40o_ktH4rAQGvnh(zbqk(fnv8maP`cPgLx#(XB9 at vqFbrA}kH{AbCPXF6(|2w%f)%I+$Y2l*g{=M$Xyxo7R zxqlS(dQX-uShQ$;C5zTpu(X=#N~@Qww7S(*n6##{r1g|4T05ms3HoPkO3*)PQi5JZ z2K`g1R@;+AqwR^J^s_1bYJ$zp6Ukc_w6cWq?C!9cPYn?E>A-6&NqF0E!k zI2Q90v}OeycC=B}*kEhng)FuvOe4yLeFSlH5$V6%CUe at dOZWJ@=2NXQ3yZh=XO?YK zbPEd+_0eoFv+T=Jzq$0wW~1`RGm_0w6Olu{`}fN=R%v2Xzo?7KT-0moQ6>(;VY&CL z)U5k{CHE{+)qJ=;f1BDp{-nB5oo3fJbushgnm1V8sQb2k*i;kKY`YTE%m%ge*Q{1i zRrjuiT1D9--$oP<#yBj_YzT)27dlOu?B2ZxS*O$H9bVQUqgyiOVJAc`l(jI|1ejJ4`TjJ0a5 z0)kPK;V>LV!#E7+c#PpN99H`ShBXtyVN+AiE at C)r%E`r_+H;F8XiqJ;$WzEX^~_?6 z)`*1`t?`O1T4QUp0*h^|;)-pg!isI&#wn`kuM|}DH;O6x+g~W8c;6?YcwZ-=c%4l= zQS)NL37XduO at O^w!`Dj*Ci+ieiT;sLqW}JnNaBAZkododBmQ?1VZ{GK6!E`P2qLna zK at 72eAcR=|4?(Pd{XPKke-A(W&w~&DJ9p?Ie;s(pKZhOi&mRXJYI$zRftJSx9E`q> zs*Al9>W-pDpQnZ!yr;ni?`NpNJ7u7u<`u&XG%pxrVE(K_yZ3q_hVET}p?g0sbl-iL z3sN4;kCbmAOv+1n7Qnn;p#|?%V8QzoR`4DL6}&$o1 at ANgg_@o|C(!ipF(KWm at 9ODR zQFMDfCG^>z1U}o3P{Q`xi$H?*A&j6s2qI{&{RbgrzXycu^?ZhBC!Yu%V>OaWwe50G`a~r+tV$HW1PV`@ou>tkcnaJ z-!9jSTH+UL7alIE%x{T-hHZAapDqV%LUmz$To@|X`z5hVs5He2ZHkpDN=8hyCp4ue zOiD?#4(6n>0d+_>efaRvylul at VRMcRs6*mb$TyU({R;&ORU7mG;~udcW=9&0Dib2{ zPk)fjSS7$pMVoklp8#J1z)t`?nZaDja3RAqhDi()&}u1L`o)%fKvr*-dP-rGqLg?^ zyCkS29TFI%FK&=>M^8sy+#Ge8;jCs(MmH;?rXpJrm!d7J&npovL$VCPGUN)86(T00 zC7Eof3bt~?2Mj~d1_A*Gm=n?CxL+WcLr(YpU+EC`2T&5V+GM)o=O+*`M( zzm^)APPegZ&-6pS|JNSn?e<=K)DA`yh1q<=g+CNFn7Ky%-cs8%`8^UVMW2{`!o{}f zR#x4Yw_54XMI{&4J-b>eyVq3C=R5cBd7Gj{A^OCslk6syU2{D|{qEwrqJw`;lveuZ zl)wF>TKEf-H^brXA8q0%q&I2x{mZ6k6LUE-?(Q-z8ESSLcXzdVW^=l=eyYy|TQ05J z&vPF#jk~+35-%ya>e|0XbcvOctpCkdUHcbR;v^(+QvM2!=n^9x>H4!7{XVl at KkJ?{ zb+Oo{%+{sSOqR_l%Caw4HB}o_dG at H^UFrUQ^Ih!vYZdjoj at N0LQC7;+`FW`OaG13` z at 9)>0={JiObItJ+q;fx8%z%KJLH*dzF+$uu^m$W3Q;9KGJ<79nRqtq*;Q^N z7L``{Mwi&gNRAQR!bL at XR?#OWQpB at n+cwCiN4=&wRsNp8S5jrZi^#5r`t!|n_4i3R zT)$G1|>8knE{Cm zM`ADn!|Vkk7mD~P3L^qGqy-@?1X+Pdg`pIHpzz}aA0`O(>A{Cb2|Gv#8bOB$IYPhz z!i^7X?5UxL#@CEIGVsK>1H*$C9RPXB>|hzbnl0h5KY#xC!E}aV+IMZ-{B1s8$YS_v zw&d*EeZ3MF=}I^2>du9MxHQ}LWV+el>n^eEYL>Kx!}u<$S&8x0EJ-WNP`J8B&)UFr zNt=W*qo(#F*Yf>K`n6F0A^mYS~poBUqCLnV$~&61inoKA^vOkWc>Ep=gh zNGmrg*G5Q3NNR4`qu;gn*~kb4r2TP1)4(}32k z`;oyhF%=7bz8o3c5>JuS!->Huv6KpYIGv4mYvNiQ8x#`LuyRmHtaQce;mkB~5}ce) z2iz at jls8R0UDFnZVkO3g#82GVe!QO&J81*-A#oE5dSQG?%w&QnB}S4!mxYfaAcI5V zB at pMEesLA1ONjxMC6whgA>n*Vm1avy!u1ju- zgB&o1r&D4g18`AMKqVeRy| z3jOVHy&mw^g?Sje^nSi3-tppoHm4>I`cS!F66^3brzFnN%h;TDwww<5MGNDQ`q^$f z-VKcc at _Jsn@D2CpYXbU~*hYJ^>7sBwCa%GMJYX0lrcvDSa=so;3=_{_$>5%lSOzhN zxTu4JE1FV>W&i*HKmY(U8w3!9)vS}S3KIZ;fr5j=tR+E#ZH#FzNgQ`WroiFLan{tlfTbgB|Gh;!Y%xx% zk`>T&gv_tCNQW)PNma4}nvRh9wHE2H#W<--RzTAcGQZX$9kv)JRmloyIzr~xTBO4k z& zms0^Veo$FNG9_{HVB12IACUv5#L1D+WDJ8VimHj)MNwheCI<_JBg>zG6CoF&93d!0 z*n_nd?LDdvi0g!)gr0Bd6nzMDc7%LM*Idzd673F-Lj`&$W$i at vfDRXw!lDOjpVg4TWT9;t%mqR z&k!jQ|E|NE0M{Uu!G2Sxiy5j?Spl}kbnKTxR8dclo|6x{^!UVw(*B~T at tlU#AAeSe zm?5Yhvcd1|Kz#S{kuNWhdN*{j!Er`ggN$DUp~sUAFM?X*OokRgEE)2UdopebL3wLG z_=%)b<9Ndm#bNXn at So_H1fgeo52Fuc2|?9rKLP`16x}=-w at 6F`rN^5LFC++_9#1wK zAdk$=FzGEYWXxew!{s1S6E@%D;y443dFEfaSL>BvG~L*LRyn_X11QRbP!$Fh2cPP5 zq_};L4EXUWUj|03mi1JOrh at DF7uU>UCVgAa3x5sjGIZ#fK&fo`6|1<4!G9HtW!L0B znmN^#k552kR^~~^uG;!d@;ZwB0Yt_{F^{@N>7;EiZ45pXcA z?ws_=1?{Z4>QOa%V*zlnw>erc>ZE4#&Ll6WKDb5 at n=(^4P94g6gNGIwOJ7p*H&gZy z+`NlEcm??x!7ZUwogwr!UD|v#=B!~%Ji{78^!%8orm;JuE$QrgiJhT97VVAWf4l93 zt?M$%NaPHt;?!Gl zawQP-zla5M>{O)yvCbvRtM+6HuLc3P{CK7rIYkq=@IQX at X%x-TMJTDIORdhcxK at Rh z0BqP7dy;xIS#w7SUR>ZwI|j|^`Ox at D65h=Bx*mjAo5;DEt~WIb^CPDqd&MoM8=&^?H5aDdD6c~xUjP0I2&r|AFDg>DCt- at wx! zkpLkdivC6CrlHBci0uSPN>PZ?0XL+KLX(R7DwI4WiG}tw0wogGd!ok+sxE-snjD1a zHCi|`%sP5TP+D9=YB at nT7#&HXj-em7Ab;#!do)CcR(n7NWbxm+p*L;N<>!RF-K-m$ z1Ji6HswE{dAdJflhR{V~!|am-H^fh!QxAiw!PbDV08RY6X7smRYqAg`vDolYTUC!~4aU!gB|Z~N@%XuF at pYG(em3vdzTOoN#a(}7Se*ehq)I(XqwT2w)rF~B-C zdP;?Up&zy&9%#IKHC%^Qdq4$2kfnq^OAY+7qZ~@hC`cm)m`x2&iO at f^gBQdDjd!nx z>(FWss2~Wkl+b6Xfj at SXLunZWX~Y1tso^OR`iFK{kl0Rvhx`Y!4_LYYScj?h1)Gq< zN*5>#Dg*n`@kQf6{EjV742uZ90hHD{NZC*sxsT5)a at isKmzaNfJaP9H`JagVN9PyW ziSQ*_X15vnK2fCWT!qpr-Q4x3`9!m)ZGS?-!0$<>DgF~AP&SFrflxizEobZ+c<~@v zG&0kq*m?2L^mwER35-VEI}0Zx<$@IB!glL9x(y-nIO4;|R;IQaHw+ZsR-xm0T#9xw z;-C at 3Yb7g@guCzP$I2WH1jq*}2#q5NWEFj4eKh|wl0!OPZUkCYFbqoARpk5qm^Q%3 zUtY9 at _@U2)>;{vD;a5zxlJMmep>gtE678o1B^M at V6gjw7NEC*xiE4qlVelnVP9thL zJ6tcK2xQQoFL)t5(6R<Z_EH_iMk+W8`~-UKj{0bSve$kR!lyC$Hm;CNd_+G|)jSs0^6k2CiYy#{K5z zOr!;8#y(CbN<{A1N|9)SgzgV88Be)RxS~=DmdFh1f;NMLhix~=wQM*DEW-|s#mgK& z$Fex+HW5}(j=?pi*}i2FcCXTg7#tl`5m3wT;Q{x{+a(T%X=z0qX^g=jE0!<#0HmKAE3X z?Qdlztz~*kV16F^EuOyXbFv$Ai!@dUeVmS-AE`DA>Q_1aKPrC244$$f4BZ>7gR2c- z9D06I7kqkPC<;+e7>HKYja at d|_qf11VYfb79Mvf$v9bdkc9#O*lvDM zjT`tN2r=(}V)SBiSR!VRSBgr9a3?4`d*wNA@*YW<$t}%sllMrP#BMz)0X*Zmx;QY*H}-|Ua*)SWMg1^N_8RIlVbPImSXCMmzA;fFre G5UuU4uRFp3 diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua index add791cd7..54fc82dc6 100644 --- a/src/box/lua/upgrade.lua +++ b/src/box/lua/upgrade.lua @@ -971,6 +971,148 @@ local function upgrade_to_2_3_1() create_session_settings_space() end +-------------------------------------------------------------------------------- +-- Tarantool 2.5.2 +-------------------------------------------------------------------------------- + +local function update_sql_builtin_functions() + local _func = box.space[box.schema.FUNC_ID] + local updates + + _func:run_triggers(false) + updates = {{'=', 'param_list', {'number'}}, {'=', 'returns', 'number'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ABS', updates) + + updates = {{'=', 'param_list', {'number'}}, {'=', 'returns', 'number'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('AVG', updates) + _func.index[2]:update('SUM', updates) + _func.index[2]:update('TOTAL', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('CHAR', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'integer'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('CHARACTER_LENGTH', updates) + _func.index[2]:update('CHAR_LENGTH', updates) + _func.index[2]:update('LENGTH', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('COALESCE', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'integer'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('COUNT', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('GREATEST', updates) + _func.index[2]:update('LEAST', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar'}}, + {'=', 'returns', 'string'}, {'=', 'aggregate', 'group'}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('GROUP_CONCAT', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('HEX', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar'}}, + {'=', 'returns', 'scalar'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('IFNULL', updates) + _func.index[2]:update('NULLIF', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar', 'scalar'}}, + {'=', 'returns', 'boolean'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('LIKE', updates) + + updates = {{'=', 'param_list', {'scalar', 'double'}}, + {'=', 'returns', 'scalar'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LIKELIHOOD', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LIKELY', updates) + _func.index[2]:update('UNLIKELY', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LOWER', updates) + _func.index[2]:update('QUOTE', updates) + _func.index[2]:update('SOUNDEX', updates) + _func.index[2]:update('TYPEOF', updates) + _func.index[2]:update('UPPER', updates) + _func.index[2]:update('UNICODE', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('MAX', updates) + _func.index[2]:update('MIN', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar'}}, + {'=', 'returns', 'integer'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('POSITION', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('PRINTF', updates) + _func.index[2]:update('TRIM', updates) + + updates = {{'=', 'returns', 'integer'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('RANDOM', updates) + + updates = {{'=', 'returns', 'integer'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ROW_COUNT', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, + {'=', 'returns', 'varbinary'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('RANDOMBLOB', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, + {'=', 'returns', 'varbinary'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ZEROBLOB', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar', 'scalar'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('REPLACE', updates) + + updates = {{'=', 'param_list', {'double', 'unsigned'}}, + {'=', 'returns', 'double'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('ROUND', updates) + + updates = {{'=', 'param_list', {'scalar', 'integer', 'integer'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('SUBSTR', updates) + + updates = {{'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('VERSION', updates) + _func:run_triggers(true) +end + +local function upgrade_to_2_5_2() + update_sql_builtin_functions() +end + -------------------------------------------------------------------------------- local function get_version() @@ -1007,6 +1149,7 @@ local function upgrade(options) {version = mkversion(2, 2, 1), func = upgrade_to_2_2_1, auto = true}, {version = mkversion(2, 3, 0), func = upgrade_to_2_3_0, auto = true}, {version = mkversion(2, 3, 1), func = upgrade_to_2_3_1, auto = true}, + {version = mkversion(2, 5, 2), func = upgrade_to_2_5_2, auto = true}, } for _, handler in ipairs(handlers) do diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 487cdafe1..df15d303a 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2188,7 +2188,7 @@ sql_func_by_signature(const char *name, int argc) if (base == NULL || !base->def->exports.sql) return NULL; - if (base->def->param_count != -1 && base->def->param_count != argc) + if (!base->def->opts.has_vararg && base->def->param_count != argc) return NULL; return base; } @@ -2929,11 +2929,6 @@ func_sql_builtin_new(struct func_def *def) func->flags = sql_builtins[idx].flags; func->call = sql_builtins[idx].call; func->finalize = sql_builtins[idx].finalize; - def->param_count = sql_builtins[idx].param_count; - def->is_deterministic = sql_builtins[idx].is_deterministic; - def->returns = sql_builtins[idx].returns; - def->aggregate = sql_builtins[idx].aggregate; - def->exports.sql = sql_builtins[idx].export_to_sql; return &func->base; } diff --git a/src/box/sql/resolve.c b/src/box/sql/resolve.c index 6f625dc18..5238555c3 100644 --- a/src/box/sql/resolve.c +++ b/src/box/sql/resolve.c @@ -614,7 +614,7 @@ resolveExprStep(Walker * pWalker, Expr * pExpr) pNC->nErr++; return WRC_Abort; } - if (func->def->param_count != -1 && + if (!func->def->opts.has_vararg && func->def->param_count != n) { uint32_t argc = func->def->param_count; const char *err = tt_sprintf("%d", argc); diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result index 0876e77a6..c56f07cef 100644 --- a/test/box-py/bootstrap.result +++ b/test/box-py/bootstrap.result @@ -4,7 +4,7 @@ box.internal.bootstrap() box.space._schema:select{} --- - - ['max_id', 511] - - ['version', 2, 3, 1] + - ['version', 2, 5, 2] ... box.space._cluster:select{} --- diff --git a/test/sql/types.result b/test/sql/types.result index 95f7713e8..2498f3a48 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -231,7 +231,7 @@ box.execute("SELECT s LIKE NULL FROM t1;") --- - metadata: - name: COLUMN_1 - type: integer + type: boolean rows: - [null] ... @@ -257,7 +257,7 @@ box.execute("SELECT NULL LIKE s FROM t1;") --- - metadata: - name: COLUMN_1 - type: integer + type: boolean rows: - [null] ... -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:50 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:50 +0300 Subject: [Tarantool-patches] [PATCH v1 4/7] box: add param_list to 'struct func' In-Reply-To: References: Message-ID: <7b30cdd9124c50b4f7e12801f6b0567097f00a4d.1597244875.git.imeevma@gmail.com> This is needed to create an uniform way to check the types of arguments of SQL built-in functions. Part of #4159 --- src/box/alter.cc | 12 ++++++++++-- src/box/func.c | 1 + src/box/func_def.h | 2 ++ src/box/lua/call.c | 2 ++ src/box/sql/func.c | 1 + 5 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/box/alter.cc b/src/box/alter.cc index ba96d9c62..0914a7615 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -3293,7 +3293,11 @@ func_def_new_from_tuple(struct tuple *tuple) diag_set(OutOfMemory, def_sz, "malloc", "def"); return NULL; } - auto def_guard = make_scoped_guard([=] { free(def); }); + def->param_list = NULL; + auto def_guard = make_scoped_guard([=] { + free(def->param_list); + free(def); + }); if (func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid) != 0) return NULL; if (def->fid > BOX_FUNCTION_MAX) { @@ -3403,6 +3407,8 @@ func_def_new_from_tuple(struct tuple *tuple) if (param_list == NULL) return NULL; uint32_t argc = mp_decode_array(¶m_list); + uint32_t size = sizeof(enum field_type) * argc; + def->param_list = (enum field_type *)malloc(size); for (uint32_t i = 0; i < argc; i++) { if (mp_typeof(*param_list) != MP_STR) { diag_set(ClientError, ER_FIELD_TYPE, @@ -3412,7 +3418,8 @@ func_def_new_from_tuple(struct tuple *tuple) } uint32_t len; const char *str = mp_decode_str(¶m_list, &len); - if (STRN2ENUM(field_type, str, len) == field_type_MAX) { + def->param_list[i] = STRN2ENUM(field_type, str, len); + if (def->param_list[i] == field_type_MAX) { diag_set(ClientError, ER_CREATE_FUNCTION, def->name, "invalid argument type"); return NULL; @@ -3433,6 +3440,7 @@ func_def_new_from_tuple(struct tuple *tuple) /* By default export to Lua, but not other frontends. */ def->exports.lua = true; def->param_count = 0; + assert(def->param_list == NULL); } if (func_def_check(def) != 0) return NULL; diff --git a/src/box/func.c b/src/box/func.c index 8087c953f..1d20f8872 100644 --- a/src/box/func.c +++ b/src/box/func.c @@ -510,6 +510,7 @@ func_c_destroy(struct func *base) { assert(base->vtab == &func_c_vtab); assert(base != NULL && base->def->language == FUNC_LANGUAGE_C); + free(base->def->param_list); struct func_c *func = (struct func_c *) base; func_c_unload(func); TRASH(base); diff --git a/src/box/func_def.h b/src/box/func_def.h index 89d5a404a..8b86167af 100644 --- a/src/box/func_def.h +++ b/src/box/func_def.h @@ -117,6 +117,8 @@ struct func_def { bool is_sandboxed; /** The count of function's input arguments. */ int param_count; + /** List of input arguments to the function. */ + enum field_type *param_list; /** The type of the value returned by function. */ enum field_type returns; /** Function aggregate option. */ diff --git a/src/box/lua/call.c b/src/box/lua/call.c index 0315e720c..1dc4589dd 100644 --- a/src/box/lua/call.c +++ b/src/box/lua/call.c @@ -783,6 +783,7 @@ func_lua_destroy(struct func *func) { assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA); assert(func->vtab == &func_lua_vtab); + free(func->def->param_list); TRASH(func); free(func); } @@ -812,6 +813,7 @@ func_persistent_lua_destroy(struct func *base) assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA && base->def->body != NULL); assert(base->vtab == &func_persistent_lua_vtab); + free(base->def->param_list); struct func_lua *func = (struct func_lua *) base; func_persistent_lua_unload(func); free(func); diff --git a/src/box/sql/func.c b/src/box/sql/func.c index df15d303a..91755380d 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2937,6 +2937,7 @@ func_sql_builtin_destroy(struct func *func) { assert(func->vtab == &func_sql_builtin_vtab); assert(func->def->language == FUNC_LANGUAGE_SQL_BUILTIN); + free(func->def->param_list); free(func); } -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:52 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:52 +0300 Subject: [Tarantool-patches] [PATCH v1 5/7] sql: check built-in functions argument types In-Reply-To: References: Message-ID: <0dfadabbd47f651890eac5b5c40733d11bb0dd1c.1597244875.git.imeevma@gmail.com> This patch creates a uniform way to check the argument types of SQL built-in functions. Prior to this patch, argument types were checked inside functions. They are now checked in most cases in the ApplyType opcode. The only case where there is additional validation in a function is when a function can take either STRING or VARBINARY as an argument. In this case, the definition says that the function accepts a SCALAR, but in fact only STRING or VARBINARY should be accepted. This case will be completely fixed in the next patches of the patch set. Part of #4159 --- src/box/sql/expr.c | 4 ++++ src/box/sql/select.c | 26 ++++++++++++++++++++++++++ src/box/sql/sqlInt.h | 14 ++++++++++++++ test/sql-tap/func.test.lua | 22 +++++++++++----------- test/sql/boolean.result | 2 +- test/sql/checks.result | 8 -------- test/sql/checks.test.lua | 2 -- test/sql/types.result | 2 +- 8 files changed, 57 insertions(+), 23 deletions(-) diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index bc2182446..e2c8e1385 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -4120,6 +4120,10 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) } else { r1 = 0; } + if (func->def->language == FUNC_LANGUAGE_SQL_BUILTIN) { + sql_emit_func_arg_type_check(v, func, r1, + nFarg); + } if (sql_func_flag_is_set(func, SQL_FUNC_NEEDCOLL)) { sqlVdbeAddOp4(v, OP_CollSeq, 0, 0, 0, (char *)coll, P4_COLLSEQ); diff --git a/src/box/sql/select.c b/src/box/sql/select.c index b0554a172..49f01eb0d 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -124,6 +124,31 @@ clearSelect(sql * db, Select * p, int bFree) } } +void +sql_emit_func_arg_type_check(struct Vdbe *vdbe, struct func *func, int reg, + uint32_t argc) +{ + if (argc == 0 || func->def->param_list == NULL) + return; + assert(func->def->param_count > 0); + uint32_t len = (uint32_t)func->def->param_count; + assert(len > 0); + size_t size = (argc + 1) * sizeof(enum field_type); + enum field_type *types = sqlDbMallocZero(sql_get(), size); + if (argc <= len) { + for (uint32_t i = 0; i < argc; ++i) + types[i] = func->def->param_list[i]; + } else { + for (uint32_t i = 0; i < len; ++i) + types[i] = func->def->param_list[i]; + for (uint32_t i = len; i < argc; ++i) + types[i] = func->def->param_list[len - 1]; + } + types[argc] = field_type_MAX; + sqlVdbeAddOp4(vdbe, OP_ApplyType, reg, argc, 0, (char *)types, + P4_DYNAMIC); +} + /* * Initialize a SelectDest structure. */ @@ -5420,6 +5445,7 @@ updateAccumulator(Parse * pParse, AggInfo * pAggInfo) vdbe_insert_distinct(pParse, pF->iDistinct, pF->reg_eph, addrNext, 1, regAgg); } + sql_emit_func_arg_type_check(v, pF->func, regAgg, nArg); if (sql_func_flag_is_set(pF->func, SQL_FUNC_NEEDCOLL)) { struct coll *coll = NULL; struct ExprList_item *pItem; diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index adf90d824..d02614140 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -3883,6 +3883,20 @@ sql_index_type_str(struct sql *db, const struct index_def *idx_def); void sql_emit_table_types(struct Vdbe *v, struct space_def *def, int reg); +/** + * Code an OP_ApplyType opcode that try to cast implicitly types + * for given range of register starting from @a reg. These values + * then will be used as arguments of a function. + * + * @param vdbe VDBE. + * @param func Definition of the function. + * @param reg Register where types will be placed. + * @param argc Number of arguments. + */ +void +sql_emit_func_arg_type_check(struct Vdbe *vdbe, struct func *func, + int reg, uint32_t argc); + enum field_type sql_type_result(enum field_type lhs, enum field_type rhs); diff --git a/test/sql-tap/func.test.lua b/test/sql-tap/func.test.lua index 3c088920f..82cf350ea 100755 --- a/test/sql-tap/func.test.lua +++ b/test/sql-tap/func.test.lua @@ -412,13 +412,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "func-4.4.2", [[ SELECT abs(t1) FROM tbl1 ]], { -- - 0.0, 0.0, 0.0, 0.0, 0.0 + 1, "Type mismatch: can not convert this to number" -- }) @@ -502,13 +502,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "func-4.13", [[ SELECT round(t1,2) FROM tbl1 ]], { -- - 0.0, 0.0, 0.0, 0.0, 0.0 + 1, "Type mismatch: can not convert this to double" -- }) @@ -893,7 +893,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.5", [[ - SELECT sum(x) FROM (SELECT '9223372036' || '854775807' AS x + SELECT sum(x) FROM (SELECT CAST('9223372036' || '854775807' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -904,7 +904,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.6", [[ - SELECT typeof(sum(x)) FROM (SELECT '9223372036' || '854775807' AS x + SELECT typeof(sum(x)) FROM (SELECT CAST('9223372036' || '854775807' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -915,7 +915,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.7", [[ - SELECT typeof(sum(x)) FROM (SELECT '9223372036' || '854775808' AS x + SELECT typeof(sum(x)) FROM (SELECT CAST('9223372036' || '854775808' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -926,7 +926,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.8", [[ - SELECT sum(x)>0.0 FROM (SELECT '9223372036' || '854775808' AS x + SELECT sum(x)>0.0 FROM (SELECT CAST('9223372036' || '854775808' AS INTEGER) AS x UNION ALL SELECT -9223372036850000000) ]], { -- @@ -985,7 +985,7 @@ test:do_execsql_test( test:do_execsql_test( "func-9.5", [[ - SELECT length(randomblob(32)), length(randomblob(-5)), + SELECT length(randomblob(32)), length(randomblob(0)), length(randomblob(2000)) ]], { -- @@ -2918,7 +2918,7 @@ test:do_catchsql_test( SELECT ROUND(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to numeric" + 1, "Type mismatch: can not convert varbinary to double" -- }) @@ -2928,7 +2928,7 @@ test:do_catchsql_test( SELECT RANDOMBLOB(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to numeric" + 1, "Type mismatch: can not convert varbinary to unsigned" -- }) diff --git a/test/sql/boolean.result b/test/sql/boolean.result index 51ec5820b..d366aca7d 100644 --- a/test/sql/boolean.result +++ b/test/sql/boolean.result @@ -276,7 +276,7 @@ SELECT is_boolean('true'); SELECT abs(a) FROM t0; | --- | - null - | - 'Inconsistent types: expected number got boolean' + | - 'Type mismatch: can not convert FALSE to number' | ... SELECT lower(a) FROM t0; | --- diff --git a/test/sql/checks.result b/test/sql/checks.result index 7b18e5d6b..3c942fb23 100644 --- a/test/sql/checks.result +++ b/test/sql/checks.result @@ -519,14 +519,6 @@ s:insert({1, 'string'}) --- - error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' ... -s:insert({1, {map=true}}) ---- -- error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' -... -s:insert({1, {'a', 'r','r','a','y'}}) ---- -- error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' -... s:insert({1, 3.14}) --- - error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' diff --git a/test/sql/checks.test.lua b/test/sql/checks.test.lua index 301f8ea69..b55abe955 100644 --- a/test/sql/checks.test.lua +++ b/test/sql/checks.test.lua @@ -173,8 +173,6 @@ s:format({{name='X', type='integer'}, {name='Z', type='any'}}) _ = s:create_index('pk', {parts = {1, 'integer'}}) _ = box.space._ck_constraint:insert({s.id, 'complex2', false, 'SQL', 'typeof(coalesce(z,0))==\'integer\'', true}) s:insert({1, 'string'}) -s:insert({1, {map=true}}) -s:insert({1, {'a', 'r','r','a','y'}}) s:insert({1, 3.14}) s:insert({1, 666}) s:drop() diff --git a/test/sql/types.result b/test/sql/types.result index 2498f3a48..82a82117b 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -1322,7 +1322,7 @@ box.execute("SELECT upper(v) FROM t;") box.execute("SELECT abs(v) FROM t;") --- - null -- 'Inconsistent types: expected number got varbinary' +- 'Type mismatch: can not convert varbinary to number' ... box.execute("SELECT typeof(v) FROM t;") --- -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:53 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:53 +0300 Subject: [Tarantool-patches] [PATCH v1 6/7] sql: VARBINARY and STRING in built-in functions In-Reply-To: References: Message-ID: <4441a1221dfa0c5b3c04b14fb0bbf1f019eaa9c3.1597244875.git.imeevma@gmail.com> This patch forces SQL built-in functions to accept VARBINARY arguments and treat them as STRING arguments if they were given in place of STRING arguments. Closes #4159 --- src/box/sql/func.c | 89 +- test/sql-tap/cse.test.lua | 4 +- test/sql-tap/func.test.lua | 14 +- test/sql-tap/orderby1.test.lua | 2 +- test/sql-tap/position.test.lua | 6 +- test/sql/boolean.result | 30 +- test/sql/types.result | 1471 +++++++++++++++++++++++++++++++- test/sql/types.test.lua | 244 ++++++ 8 files changed, 1758 insertions(+), 102 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 91755380d..694bd0e83 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -463,34 +463,23 @@ typeofFunc(sql_context * context, int NotUsed, sql_value ** argv) static void lengthFunc(sql_context * context, int argc, sql_value ** argv) { - int len; - assert(argc == 1); UNUSED_PARAMETER(argc); - switch (sql_value_type(argv[0])) { - case MP_BIN: - case MP_ARRAY: - case MP_MAP: - case MP_INT: - case MP_UINT: - case MP_BOOL: - case MP_DOUBLE:{ - sql_result_uint(context, sql_value_bytes(argv[0])); - break; - } - case MP_STR:{ - const unsigned char *z = sql_value_text(argv[0]); - if (z == 0) - return; - len = sql_utf8_char_count(z, sql_value_bytes(argv[0])); - sql_result_uint(context, len); - break; - } - default:{ - sql_result_null(context); - break; - } + if (sql_value_type(argv[0]) == MP_NIL) + return sql_result_null(context); + if (sql_value_type(argv[0]) == MP_BIN) + return sql_result_uint(context, sql_value_bytes(argv[0])); + if (sql_value_type(argv[0]) != MP_STR) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(argv[0]), "string or varbinary"); + context->is_aborted = true; + return; } + const unsigned char *str = sql_value_text(argv[0]); + if (str == NULL) + return sql_result_null(context); + int len = sql_utf8_char_count(str, sql_value_bytes(argv[0])); + return sql_result_uint(context, len); } /* @@ -576,9 +565,9 @@ position_func(struct sql_context *context, int argc, struct Mem **argv) if (haystack_type != MP_STR && haystack_type != MP_BIN) inconsistent_type_arg = haystack; if (inconsistent_type_arg != NULL) { - diag_set(ClientError, ER_INCONSISTENT_TYPES, - "text or varbinary", - mem_type_to_str(inconsistent_type_arg)); + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(inconsistent_type_arg), + "string or varbinary"); context->is_aborted = true; return; } @@ -748,13 +737,20 @@ substrFunc(sql_context * context, int argc, sql_value ** argv) if (z == 0) return; assert(len == sql_value_bytes(argv[0])); - } else { + } else if (p0type == MP_NIL) { + return sql_result_null(context); + } else if (p0type == MP_STR) { z = sql_value_text(argv[0]); if (z == 0) return; len = 0; if (p1 < 0) len = sql_utf8_char_count(z, sql_value_bytes(argv[0])); + } else { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(argv[0]), "string or varbinary"); + context->is_aborted = true; + return; } if (argc == 3) { p2 = sql_value_int(argv[2]); @@ -906,10 +902,15 @@ case_type##ICUFunc(sql_context *context, int argc, sql_value **argv) \ const char *z2; \ int n; \ UNUSED_PARAMETER(argc); \ - int arg_type = sql_value_type(argv[0]); \ - if (mp_type_is_bloblike(arg_type)) { \ - diag_set(ClientError, ER_INCONSISTENT_TYPES, "text", \ - "varbinary"); \ + enum mp_type type = sql_value_type(argv[0]); \ + if (type == MP_NIL) \ + return sql_result_null(context); \ + if (type == MP_BIN) \ + return sql_result_value(context, argv[0]); \ + if (type != MP_STR) { \ + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, \ + sql_value_to_diag_str(argv[0]), \ + "string or varbinary"); \ context->is_aborted = true; \ return; \ } \ @@ -1452,6 +1453,13 @@ quoteFunc(sql_context * context, int argc, sql_value ** argv) static void unicodeFunc(sql_context * context, int argc, sql_value ** argv) { + enum mp_type type = sql_value_type(argv[0]); + if (type != MP_NIL && type != MP_STR && type != MP_BIN) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(argv[0]), "string or varbinary"); + context->is_aborted = true; + return; + } const unsigned char *z = sql_value_text(argv[0]); (void)argc; if (z && z[0]) @@ -1568,6 +1576,17 @@ replaceFunc(sql_context * context, int argc, sql_value ** argv) int loopLimit; /* Last zStr[] that might match zPattern[] */ int i, j; /* Loop counters */ + for (int i = 0; i < 3; ++i) { + enum mp_type type = sql_value_type(argv[i]); + if (type != MP_NIL && type != MP_STR && type != MP_BIN) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(argv[i]), + "string or varbinary"); + context->is_aborted = true; + return; + } + } + assert(argc == 3); UNUSED_PARAMETER(argc); zStr = sql_value_text(argv[0]); @@ -1880,9 +1899,9 @@ soundexFunc(sql_context * context, int argc, sql_value ** argv) }; assert(argc == 1); enum mp_type mp_type = sql_value_type(argv[0]); - if (mp_type_is_bloblike(mp_type)) { + if (mp_type != MP_NIL && mp_type != MP_STR && mp_type != MP_BIN) { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "text"); + sql_value_to_diag_str(argv[0]), "string or varbinary"); context->is_aborted = true; return; } diff --git a/test/sql-tap/cse.test.lua b/test/sql-tap/cse.test.lua index 341b6de01..cfb642cfe 100755 --- a/test/sql-tap/cse.test.lua +++ b/test/sql-tap/cse.test.lua @@ -198,7 +198,7 @@ test:do_execsql_test( test:do_execsql_test( "cse-1.13", [[ - SELECT upper(b), typeof(b), b FROM t1 + SELECT upper(CAST(b AS STRING)), typeof(b), b FROM t1 ]], { -- "11", "integer", 11, "21", "integer", 21 @@ -208,7 +208,7 @@ test:do_execsql_test( test:do_execsql_test( "cse-1.14", [[ - SELECT b, typeof(b), upper(b), typeof(b), b FROM t1 + SELECT b, typeof(b), upper(CAST(b AS STRING)), typeof(b), b FROM t1 ]], { -- 11, "integer", "11", "integer", 11, 21, "integer", "21", "integer", 21 diff --git a/test/sql-tap/func.test.lua b/test/sql-tap/func.test.lua index 82cf350ea..29d6e422c 100755 --- a/test/sql-tap/func.test.lua +++ b/test/sql-tap/func.test.lua @@ -95,7 +95,7 @@ test:do_execsql_test( test:do_execsql_test( "func-1.4", [[ - SELECT coalesce(length(a),-1) FROM t2 + SELECT coalesce(length(CAST(a AS STRING)),-1) FROM t2 ]], { -- 1, -1, 3, -1, 5 @@ -197,7 +197,7 @@ test:do_execsql_test( test:do_execsql_test( "func-2.9", [[ - SELECT substr(a,1,1) FROM t2 + SELECT substr(CAST(a AS STRING),1,1) FROM t2 ]], { -- "1", "", "3", "", "6" @@ -207,7 +207,7 @@ test:do_execsql_test( test:do_execsql_test( "func-2.10", [[ - SELECT substr(a,2,2) FROM t2 + SELECT substr(CAST(a AS STRING),2,2) FROM t2 ]], { -- "", "", "45", "", "78" @@ -763,7 +763,7 @@ test:do_execsql_test( test:do_execsql_test( "func-5.3", [[ - SELECT upper(a), lower(a) FROM t2 + SELECT upper(CAST(a AS STRING)), lower(CAST(a AS STRING)) FROM t2 ]], { -- "1","1","","","345","345","","","67890","67890" @@ -797,7 +797,7 @@ test:do_execsql_test( test:do_execsql_test( "func-6.2", [[ - SELECT coalesce(upper(a),'nil') FROM t2 + SELECT coalesce(upper(CAST(a AS STRING)),'nil') FROM t2 ]], { -- "1","nil","345","nil","67890" @@ -2932,13 +2932,13 @@ test:do_catchsql_test( -- }) -test:do_catchsql_test( +test:do_execsql_test( "func-76.3", [[ SELECT SOUNDEX(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to text" + "?000" -- }) diff --git a/test/sql-tap/orderby1.test.lua b/test/sql-tap/orderby1.test.lua index 51e8d301f..95a8de487 100755 --- a/test/sql-tap/orderby1.test.lua +++ b/test/sql-tap/orderby1.test.lua @@ -735,7 +735,7 @@ test:do_execsql_test( SELECT ( SELECT 'hardware' FROM ( SELECT 'software' ORDER BY 'firmware' ASC, 'sportswear' DESC - ) GROUP BY 1 HAVING length(b) <> 0 + ) GROUP BY 1 HAVING length(CAST(b AS STRING)) <> 0 ) FROM abc; ]], { diff --git a/test/sql-tap/position.test.lua b/test/sql-tap/position.test.lua index e0455abc9..a84e11cd0 100755 --- a/test/sql-tap/position.test.lua +++ b/test/sql-tap/position.test.lua @@ -228,7 +228,7 @@ test:do_test( return test:catchsql "SELECT position(34, 12345);" end, { -- - 1, "Inconsistent types: expected text or varbinary got unsigned" + 1, "Type mismatch: can not convert 12345 to string or varbinary" -- }) @@ -238,7 +238,7 @@ test:do_test( return test:catchsql "SELECT position(34, 123456.78);" end, { -- - 1, "Inconsistent types: expected text or varbinary got real" + 1, "Type mismatch: can not convert 123456.78 to string or varbinary" -- }) @@ -248,7 +248,7 @@ test:do_test( return test:catchsql "SELECT position(x'3334', 123456.78);" end, { -- - 1, "Inconsistent types: expected text or varbinary got real" + 1, "Type mismatch: can not convert 123456.78 to string or varbinary" -- }) diff --git a/test/sql/boolean.result b/test/sql/boolean.result index d366aca7d..743183f7d 100644 --- a/test/sql/boolean.result +++ b/test/sql/boolean.result @@ -280,25 +280,13 @@ SELECT abs(a) FROM t0; | ... SELECT lower(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: string - | rows: - | - ['false'] - | - ['true'] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string or varbinary' | ... SELECT upper(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: string - | rows: - | - ['FALSE'] - | - ['TRUE'] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string or varbinary' | ... SELECT quote(a) FROM t0; | --- @@ -314,14 +302,8 @@ SELECT quote(a) FROM t0; -- gh-4462: LENGTH didn't take BOOLEAN arguments. SELECT length(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: integer - | rows: - | - [5] - | - [4] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string or varbinary' | ... SELECT typeof(a) FROM t0; | --- diff --git a/test/sql/types.result b/test/sql/types.result index 82a82117b..c3eb4e27d 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -830,19 +830,13 @@ box.execute("DELETE FROM t WHERE i < 18446744073709551613;") ... box.execute("SELECT lower(i) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['18446744073709551613'] +- null +- 'Type mismatch: can not convert 18446744073709551613 to string or varbinary' ... box.execute("SELECT upper(i) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['18446744073709551613'] +- null +- 'Type mismatch: can not convert 18446744073709551613 to string or varbinary' ... box.execute("SELECT abs(i) FROM t;") --- @@ -1311,13 +1305,19 @@ box.execute("SELECT group_concat(v) FROM t;") ... box.execute("SELECT lower(v) FROM t;") --- -- null -- 'Inconsistent types: expected text got varbinary' +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['abc'] ... box.execute("SELECT upper(v) FROM t;") --- -- null -- 'Inconsistent types: expected text got varbinary' +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['abc'] ... box.execute("SELECT abs(v) FROM t;") --- @@ -1879,25 +1879,13 @@ box.execute("SELECT group_concat(d) FROM t;") ... box.execute("SELECT lower(d) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['10.0'] - - ['-2.0'] - - ['3.3'] - - ['1.8e+19'] +- null +- 'Type mismatch: can not convert 10.0 to string or varbinary' ... box.execute("SELECT upper(d) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['10.0'] - - ['-2.0'] - - ['3.3'] - - ['1.8E+19'] +- null +- 'Type mismatch: can not convert 10.0 to string or varbinary' ... box.execute("SELECT abs(d) FROM t;") --- @@ -2807,3 +2795,1426 @@ box.execute([[SELECT typeof(length('abc'));]]) rows: - ['integer'] ... +-- Make sure the function argument types are checked. +box.execute([[SELECT abs(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT abs(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT abs(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT abs(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT abs('a');]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT abs(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT char(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT char(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ["\x01"] +... +box.execute([[SELECT char(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ["\x01"] +... +box.execute([[SELECT char(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT char('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT char(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[SELECT character_length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT character_length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT character_length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT character_length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT character_length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT character_length(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT char_length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT char_length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT char_length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT char_length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT char_length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT char_length(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT coalesce(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT coalesce(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT coalesce(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT coalesce(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT coalesce('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT coalesce(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT greatest(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT greatest(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT greatest(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT greatest(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT greatest('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT greatest(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT hex(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['2D31'] +... +box.execute([[SELECT hex(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['31'] +... +box.execute([[SELECT hex(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['312E35'] +... +box.execute([[SELECT hex(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['54525545'] +... +box.execute([[SELECT hex('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['61'] +... +box.execute([[SELECT hex(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['33'] +... +box.execute([[SELECT ifnull(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT ifnull(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT ifnull(1.5, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT ifnull(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT ifnull('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT ifnull(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT least(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT least(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT least(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT least(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT least('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT least(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT length(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT likelihood(-1, -1);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(1, 1);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(1.5, 1.5);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(true, true);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood('a', 'a');]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(X'33', X'33');]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likely(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [-1] +... +box.execute([[SELECT likely(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT likely(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT likely(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: boolean + rows: + - [true] +... +box.execute([[SELECT likely('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT likely(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ['3'] +... +box.execute([[SELECT lower(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT lower(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT lower(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT lower(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT lower('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT lower(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT nullif(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT position(-1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT position(1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT position(1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT position(true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT position('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT position(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT printf(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['-1'] +... +box.execute([[SELECT printf(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1'] +... +box.execute([[SELECT printf(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT printf(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT printf('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT printf(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT quote(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [-1] +... +box.execute([[SELECT quote(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [1] +... +box.execute([[SELECT quote(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT quote(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT quote('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['''a'''] +... +box.execute([[SELECT quote(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['X''33'''] +... +box.execute([[SELECT randomblob(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT randomblob(0);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - [null] +... +box.execute([[SELECT randomblob(0.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - [null] +... +box.execute([[SELECT randomblob(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT randomblob('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT randomblob(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[SELECT replace(-1, -1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT replace(1, 1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT replace(1.5, 1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT replace(true, true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT replace('a', 'a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT replace(X'33', X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT round(-1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT round(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1] +... +box.execute([[SELECT round(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT round(true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to double' +... +box.execute([[SELECT round('a', 'a');]]) +--- +- null +- 'Type mismatch: can not convert a to double' +... +box.execute([[SELECT round(X'33', X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to double' +... +box.execute([[SELECT soundex(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT soundex(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT soundex(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT soundex(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT soundex('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['A000'] +... +box.execute([[SELECT soundex(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['?000'] +... +box.execute([[SELECT substr(-1, -1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT substr(1, 1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT substr(1.5, 1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT substr(true, true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to integer' +... +box.execute([[SELECT substr('a', 'a', 'a');]]) +--- +- null +- 'Type mismatch: can not convert a to integer' +... +box.execute([[SELECT substr(X'33', X'33', X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to integer' +... +box.execute([[SELECT typeof(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... +box.execute([[SELECT typeof(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... +box.execute([[SELECT typeof(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['double'] +... +box.execute([[SELECT typeof(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['boolean'] +... +box.execute([[SELECT typeof('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['string'] +... +box.execute([[SELECT typeof(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['varbinary'] +... +box.execute([[SELECT unicode(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT unicode(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT unicode(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT unicode(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT unicode('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [97] +... +box.execute([[SELECT unicode(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [51] +... +box.execute([[SELECT unlikely(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [-1] +... +box.execute([[SELECT unlikely(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT unlikely(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT unlikely(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: boolean + rows: + - [true] +... +box.execute([[SELECT unlikely('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT unlikely(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ['3'] +... +box.execute([[SELECT upper(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string or varbinary' +... +box.execute([[SELECT upper(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string or varbinary' +... +box.execute([[SELECT upper(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string or varbinary' +... +box.execute([[SELECT upper(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string or varbinary' +... +box.execute([[SELECT upper('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['A'] +... +box.execute([[SELECT upper(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT zeroblob(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT zeroblob(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ["\0"] +... +box.execute([[SELECT zeroblob(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ["\0"] +... +box.execute([[SELECT zeroblob(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT zeroblob('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT zeroblob(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, u UNSIGNED, d DOUBLE, b BOOLEAN, s STRING, v VARBINARY);]]) +--- +- row_count: 1 +... +box.execute([[INSERT INTO t VALUES (-1, 1, 1.5, true, 'a', X'33');]]) +--- +- row_count: 1 +... +box.execute([[SELECT avg(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT avg(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT avg(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT avg(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT avg(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT avg(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT count(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT group_concat(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['-1'] +... +box.execute([[SELECT group_concat(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1'] +... +box.execute([[SELECT group_concat(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT group_concat(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT group_concat(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT group_concat(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT max(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT max(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT max(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT max(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT max(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT max(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT min(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT min(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT min(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT min(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT min(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT min(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT sum(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT sum(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT sum(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT sum(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT sum(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT sum(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT total(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT total(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT total(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT total(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT total(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT total(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[DROP TABLE t;]]) +--- +- row_count: 1 +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index fff0057bd..ac9128548 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -629,3 +629,247 @@ box.execute([[DROP TABLE ts;]]) -- instead of values of type UNSIGNED. -- box.execute([[SELECT typeof(length('abc'));]]) + +-- Make sure the function argument types are checked. +box.execute([[SELECT abs(-1);]]) +box.execute([[SELECT abs(1);]]) +box.execute([[SELECT abs(1.5);]]) +box.execute([[SELECT abs(true);]]) +box.execute([[SELECT abs('a');]]) +box.execute([[SELECT abs(X'33');]]) + +box.execute([[SELECT char(-1);]]) +box.execute([[SELECT char(1);]]) +box.execute([[SELECT char(1.5);]]) +box.execute([[SELECT char(true);]]) +box.execute([[SELECT char('a');]]) +box.execute([[SELECT char(X'33');]]) + +box.execute([[SELECT character_length(-1);]]) +box.execute([[SELECT character_length(1);]]) +box.execute([[SELECT character_length(1.5);]]) +box.execute([[SELECT character_length(true);]]) +box.execute([[SELECT character_length('a');]]) +box.execute([[SELECT character_length(X'33');]]) + +box.execute([[SELECT char_length(-1);]]) +box.execute([[SELECT char_length(1);]]) +box.execute([[SELECT char_length(1.5);]]) +box.execute([[SELECT char_length(true);]]) +box.execute([[SELECT char_length('a');]]) +box.execute([[SELECT char_length(X'33');]]) + +box.execute([[SELECT coalesce(-1, -1);]]) +box.execute([[SELECT coalesce(1, 1);]]) +box.execute([[SELECT coalesce(1.5, 1.5);]]) +box.execute([[SELECT coalesce(true, true);]]) +box.execute([[SELECT coalesce('a', 'a');]]) +box.execute([[SELECT coalesce(X'33', X'33');]]) + +box.execute([[SELECT greatest(-1, -1);]]) +box.execute([[SELECT greatest(1, 1);]]) +box.execute([[SELECT greatest(1.5, 1.5);]]) +box.execute([[SELECT greatest(true, true);]]) +box.execute([[SELECT greatest('a', 'a');]]) +box.execute([[SELECT greatest(X'33', X'33');]]) + +box.execute([[SELECT hex(-1);]]) +box.execute([[SELECT hex(1);]]) +box.execute([[SELECT hex(1.5);]]) +box.execute([[SELECT hex(true);]]) +box.execute([[SELECT hex('a');]]) +box.execute([[SELECT hex(X'33');]]) + +box.execute([[SELECT ifnull(-1, -1);]]) +box.execute([[SELECT ifnull(1, 1);]]) +box.execute([[SELECT ifnull(1.5, 1);]]) +box.execute([[SELECT ifnull(true, true);]]) +box.execute([[SELECT ifnull('a', 'a');]]) +box.execute([[SELECT ifnull(X'33', X'33');]]) + +box.execute([[SELECT least(-1, -1);]]) +box.execute([[SELECT least(1, 1);]]) +box.execute([[SELECT least(1.5, 1.5);]]) +box.execute([[SELECT least(true, true);]]) +box.execute([[SELECT least('a', 'a');]]) +box.execute([[SELECT least(X'33', X'33');]]) + +box.execute([[SELECT length(-1);]]) +box.execute([[SELECT length(1);]]) +box.execute([[SELECT length(1.5);]]) +box.execute([[SELECT length(true);]]) +box.execute([[SELECT length('a');]]) +box.execute([[SELECT length(X'33');]]) + +box.execute([[SELECT likelihood(-1, -1);]]) +box.execute([[SELECT likelihood(1, 1);]]) +box.execute([[SELECT likelihood(1.5, 1.5);]]) +box.execute([[SELECT likelihood(true, true);]]) +box.execute([[SELECT likelihood('a', 'a');]]) +box.execute([[SELECT likelihood(X'33', X'33');]]) + +box.execute([[SELECT likely(-1);]]) +box.execute([[SELECT likely(1);]]) +box.execute([[SELECT likely(1.5);]]) +box.execute([[SELECT likely(true);]]) +box.execute([[SELECT likely('a');]]) +box.execute([[SELECT likely(X'33');]]) + +box.execute([[SELECT lower(-1);]]) +box.execute([[SELECT lower(1);]]) +box.execute([[SELECT lower(1.5);]]) +box.execute([[SELECT lower(true);]]) +box.execute([[SELECT lower('a');]]) +box.execute([[SELECT lower(X'33');]]) + +box.execute([[SELECT nullif(-1, -1);]]) +box.execute([[SELECT nullif(1, 1);]]) +box.execute([[SELECT nullif(1.5, 1.5);]]) +box.execute([[SELECT nullif(true, true);]]) +box.execute([[SELECT nullif('a', 'a');]]) +box.execute([[SELECT nullif(X'33', X'33');]]) + +box.execute([[SELECT position(-1, -1);]]) +box.execute([[SELECT position(1, 1);]]) +box.execute([[SELECT position(1.5, 1.5);]]) +box.execute([[SELECT position(true, true);]]) +box.execute([[SELECT position('a', 'a');]]) +box.execute([[SELECT position(X'33', X'33');]]) + +box.execute([[SELECT printf(-1);]]) +box.execute([[SELECT printf(1);]]) +box.execute([[SELECT printf(1.5);]]) +box.execute([[SELECT printf(true);]]) +box.execute([[SELECT printf('a');]]) +box.execute([[SELECT printf(X'33');]]) + +box.execute([[SELECT quote(-1);]]) +box.execute([[SELECT quote(1);]]) +box.execute([[SELECT quote(1.5);]]) +box.execute([[SELECT quote(true);]]) +box.execute([[SELECT quote('a');]]) +box.execute([[SELECT quote(X'33');]]) + +box.execute([[SELECT randomblob(-1);]]) +box.execute([[SELECT randomblob(0);]]) +box.execute([[SELECT randomblob(0.5);]]) +box.execute([[SELECT randomblob(true);]]) +box.execute([[SELECT randomblob('a');]]) +box.execute([[SELECT randomblob(X'33');]]) + +box.execute([[SELECT replace(-1, -1, -1);]]) +box.execute([[SELECT replace(1, 1, 1);]]) +box.execute([[SELECT replace(1.5, 1.5, 1.5);]]) +box.execute([[SELECT replace(true, true, true);]]) +box.execute([[SELECT replace('a', 'a', 'a');]]) +box.execute([[SELECT replace(X'33', X'33', X'33');]]) + +box.execute([[SELECT round(-1, -1);]]) +box.execute([[SELECT round(1, 1);]]) +box.execute([[SELECT round(1.5, 1.5);]]) +box.execute([[SELECT round(true, true);]]) +box.execute([[SELECT round('a', 'a');]]) +box.execute([[SELECT round(X'33', X'33');]]) + +box.execute([[SELECT soundex(-1);]]) +box.execute([[SELECT soundex(1);]]) +box.execute([[SELECT soundex(1.5);]]) +box.execute([[SELECT soundex(true);]]) +box.execute([[SELECT soundex('a');]]) +box.execute([[SELECT soundex(X'33');]]) + +box.execute([[SELECT substr(-1, -1, -1);]]) +box.execute([[SELECT substr(1, 1, 1);]]) +box.execute([[SELECT substr(1.5, 1.5, 1.5);]]) +box.execute([[SELECT substr(true, true, true);]]) +box.execute([[SELECT substr('a', 'a', 'a');]]) +box.execute([[SELECT substr(X'33', X'33', X'33');]]) + +box.execute([[SELECT typeof(-1);]]) +box.execute([[SELECT typeof(1);]]) +box.execute([[SELECT typeof(1.5);]]) +box.execute([[SELECT typeof(true);]]) +box.execute([[SELECT typeof('a');]]) +box.execute([[SELECT typeof(X'33');]]) + +box.execute([[SELECT unicode(-1);]]) +box.execute([[SELECT unicode(1);]]) +box.execute([[SELECT unicode(1.5);]]) +box.execute([[SELECT unicode(true);]]) +box.execute([[SELECT unicode('a');]]) +box.execute([[SELECT unicode(X'33');]]) + +box.execute([[SELECT unlikely(-1);]]) +box.execute([[SELECT unlikely(1);]]) +box.execute([[SELECT unlikely(1.5);]]) +box.execute([[SELECT unlikely(true);]]) +box.execute([[SELECT unlikely('a');]]) +box.execute([[SELECT unlikely(X'33');]]) + +box.execute([[SELECT upper(-1);]]) +box.execute([[SELECT upper(1);]]) +box.execute([[SELECT upper(1.5);]]) +box.execute([[SELECT upper(true);]]) +box.execute([[SELECT upper('a');]]) +box.execute([[SELECT upper(X'33');]]) + +box.execute([[SELECT zeroblob(-1);]]) +box.execute([[SELECT zeroblob(1);]]) +box.execute([[SELECT zeroblob(1.5);]]) +box.execute([[SELECT zeroblob(true);]]) +box.execute([[SELECT zeroblob('a');]]) +box.execute([[SELECT zeroblob(X'33');]]) + +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, u UNSIGNED, d DOUBLE, b BOOLEAN, s STRING, v VARBINARY);]]) +box.execute([[INSERT INTO t VALUES (-1, 1, 1.5, true, 'a', X'33');]]) + +box.execute([[SELECT avg(i) FROM t;]]) +box.execute([[SELECT avg(u) FROM t;]]) +box.execute([[SELECT avg(d) FROM t;]]) +box.execute([[SELECT avg(b) FROM t;]]) +box.execute([[SELECT avg(s) FROM t;]]) +box.execute([[SELECT avg(v) FROM t;]]) + +box.execute([[SELECT count(i) FROM t;]]) +box.execute([[SELECT count(u) FROM t;]]) +box.execute([[SELECT count(d) FROM t;]]) +box.execute([[SELECT count(b) FROM t;]]) +box.execute([[SELECT count(s) FROM t;]]) +box.execute([[SELECT count(v) FROM t;]]) + +box.execute([[SELECT group_concat(i) FROM t;]]) +box.execute([[SELECT group_concat(u) FROM t;]]) +box.execute([[SELECT group_concat(d) FROM t;]]) +box.execute([[SELECT group_concat(b) FROM t;]]) +box.execute([[SELECT group_concat(s) FROM t;]]) +box.execute([[SELECT group_concat(v) FROM t;]]) + +box.execute([[SELECT max(i) FROM t;]]) +box.execute([[SELECT max(u) FROM t;]]) +box.execute([[SELECT max(d) FROM t;]]) +box.execute([[SELECT max(b) FROM t;]]) +box.execute([[SELECT max(s) FROM t;]]) +box.execute([[SELECT max(v) FROM t;]]) + +box.execute([[SELECT min(i) FROM t;]]) +box.execute([[SELECT min(u) FROM t;]]) +box.execute([[SELECT min(d) FROM t;]]) +box.execute([[SELECT min(b) FROM t;]]) +box.execute([[SELECT min(s) FROM t;]]) +box.execute([[SELECT min(v) FROM t;]]) + +box.execute([[SELECT sum(i) FROM t;]]) +box.execute([[SELECT sum(u) FROM t;]]) +box.execute([[SELECT sum(d) FROM t;]]) +box.execute([[SELECT sum(b) FROM t;]]) +box.execute([[SELECT sum(s) FROM t;]]) +box.execute([[SELECT sum(v) FROM t;]]) + +box.execute([[SELECT total(i) FROM t;]]) +box.execute([[SELECT total(u) FROM t;]]) +box.execute([[SELECT total(d) FROM t;]]) +box.execute([[SELECT total(b) FROM t;]]) +box.execute([[SELECT total(s) FROM t;]]) +box.execute([[SELECT total(v) FROM t;]]) + +box.execute([[DROP TABLE t;]]) -- 2.25.1 From imeevma at tarantool.org Wed Aug 12 18:15:55 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Wed, 12 Aug 2020 18:15:55 +0300 Subject: [Tarantool-patches] [PATCH v1 7/7] sql: refactor sql/func.c In-Reply-To: References: Message-ID: <797828a6c2faec9a2945b85b7be521bf1c707e6b.1597244875.git.imeevma@gmail.com> After changing the way of checking the types of arguments, some of the code in sql/func.c is no longer used. This patch removes this code. Follow-up of #4159 --- src/box/sql/func.c | 797 +++++---------------------------------------- 1 file changed, 83 insertions(+), 714 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 694bd0e83..651864189 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -493,44 +493,21 @@ absFunc(sql_context * context, int argc, sql_value ** argv) { assert(argc == 1); UNUSED_PARAMETER(argc); - switch (sql_value_type(argv[0])) { - case MP_UINT: { - sql_result_uint(context, sql_value_uint64(argv[0])); - break; - } - case MP_INT: { + enum mp_type type = sql_value_type(argv[0]); + if (type == MP_NIL) + return sql_result_null(context); + if (type == MP_UINT) + return sql_result_uint(context, sql_value_uint64(argv[0])); + if (type == MP_INT) { int64_t value = sql_value_int64(argv[0]); assert(value < 0); - sql_result_uint(context, -value); - break; - } - case MP_NIL:{ - /* IMP: R-37434-19929 Abs(X) returns NULL if X is NULL. */ - sql_result_null(context); - break; - } - case MP_BOOL: - case MP_BIN: - case MP_ARRAY: - case MP_MAP: { - diag_set(ClientError, ER_INCONSISTENT_TYPES, "number", - mem_type_to_str(argv[0])); - context->is_aborted = true; - return; - } - default:{ - /* Because sql_value_double() returns 0.0 if the argument is not - * something that can be converted into a number, we have: - * IMP: R-01992-00519 Abs(X) returns 0.0 if X is a string or blob - * that cannot be converted to a numeric value. - */ - double rVal = sql_value_double(argv[0]); - if (rVal < 0) - rVal = -rVal; - sql_result_double(context, rVal); - break; - } + return sql_result_uint(context, -value); } + assert(type == MP_DOUBLE); + double value = sql_value_double(argv[0]); + if (value < 0) + value = -value; + return sql_result_double(context, value); } /** @@ -835,19 +812,14 @@ roundFunc(sql_context * context, int argc, sql_value ** argv) if (argc == 2) { if (sql_value_is_null(argv[1])) return; + assert(sql_value_type(argv[1]) == MP_UINT); n = sql_value_int(argv[1]); if (n < 0) n = 0; } if (sql_value_is_null(argv[0])) return; - enum mp_type mp_type = sql_value_type(argv[0]); - if (mp_type_is_bloblike(mp_type)) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "numeric"); - context->is_aborted = true; - return; - } + assert(sql_value_type(argv[0]) == MP_DOUBLE); r = sql_value_double(argv[0]); /* If Y==0 and X will fit in a 64-bit int, * handle the rounding directly, @@ -989,12 +961,7 @@ randomBlob(sql_context * context, int argc, sql_value ** argv) unsigned char *p; assert(argc == 1); UNUSED_PARAMETER(argc); - if (mp_type_is_bloblike(sql_value_type(argv[0]))) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "numeric"); - context->is_aborted = true; - return; - } + assert(sql_value_type(argv[0]) == MP_UINT); n = sql_value_int(argv[0]); if (n < 1) return; @@ -1546,6 +1513,7 @@ zeroblobFunc(sql_context * context, int argc, sql_value ** argv) i64 n; assert(argc == 1); UNUSED_PARAMETER(argc); + assert(sql_value_type(argv[0]) == MP_UINT); n = sql_value_int64(argv[0]); if (n < 0) n = 0; @@ -1968,18 +1936,10 @@ sum_step(struct sql_context *context, int argc, sql_value **argv) assert(argc == 1); UNUSED_PARAMETER(argc); struct SumCtx *p = sql_aggregate_context(context, sizeof(*p)); - int type = sql_value_type(argv[0]); + enum mp_type type = sql_value_type(argv[0]); if (type == MP_NIL || p == NULL) return; - if (type != MP_DOUBLE && type != MP_INT && type != MP_UINT) { - if (mem_apply_numeric_type(argv[0]) != 0) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "number"); - context->is_aborted = true; - return; - } - type = sql_value_type(argv[0]); - } + assert(type == MP_DOUBLE || type == MP_INT || type == MP_UINT); p->cnt++; if (type == MP_INT || type == MP_UINT) { int64_t v = sql_value_int64(argv[0]); @@ -2246,663 +2206,72 @@ static struct { uint16_t flags; void (*call)(sql_context *ctx, int argc, sql_value **argv); void (*finalize)(sql_context *ctx); - /** Members below are related to struct func_def. */ - bool is_deterministic; - int param_count; - enum field_type returns; - enum func_aggregate aggregate; - bool export_to_sql; } sql_builtins[] = { - {.name = "ABS", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = absFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "AVG", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .is_deterministic = false, - .aggregate = FUNC_AGGREGATE_GROUP, - .flags = 0, - .call = sum_step, - .finalize = avgFinalize, - .export_to_sql = true, - }, { - .name = "CEIL", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CEILING", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CHAR", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .is_deterministic = true, - .aggregate = FUNC_AGGREGATE_NONE, - .flags = 0, - .call = charFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "CHARACTER_LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "CHAR_LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "COALESCE", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_COALESCE, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "COUNT", - .param_count = -1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = countStep, - .finalize = countFinalize, - .export_to_sql = true, - }, { - .name = "CURRENT_DATE", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CURRENT_TIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CURRENT_TIMESTAMP", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "DATE", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "DATETIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EVERY", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXISTS", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXP", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXTRACT", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "FLOOR", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "GREATER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "GREATEST", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, - .call = minmaxFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "GROUP_CONCAT", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = groupConcatStep, - .finalize = groupConcatFinalize, - .export_to_sql = true, - }, { - .name = "HEX", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = hexFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "IFNULL", - .param_count = 2, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_COALESCE, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "JULIANDAY", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LEAST", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, - .call = minmaxFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_LENGTH, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LESSER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LIKE", - .param_count = -1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_LIKE, - .call = likeFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LIKELIHOOD", - .param_count = 2, - .returns = FIELD_TYPE_BOOLEAN, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LIKELY", - .param_count = 1, - .returns = FIELD_TYPE_BOOLEAN, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LN", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LOWER", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, - .call = LowerICUFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "MAX", - .param_count = 1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, - .call = minmaxStep, - .finalize = minMaxFinalize, - .export_to_sql = true, - }, { - .name = "MIN", - .param_count = 1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, - .call = minmaxStep, - .finalize = minMaxFinalize, - .export_to_sql = true, - }, { - .name = "MOD", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "NULLIF", - .param_count = 2, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL, - .call = nullifFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "OCTET_LENGTH", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "POSITION", - .param_count = 2, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL, - .call = position_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "POWER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "PRINTF", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = printfFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "QUOTE", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = quoteFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "RANDOM", - .param_count = 0, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .call = randomFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "RANDOMBLOB", - .param_count = 1, - .returns = FIELD_TYPE_VARBINARY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .call = randomBlob, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "REPLACE", - .param_count = 3, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = replaceFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ROUND", - .param_count = -1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = roundFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ROW_COUNT", - .param_count = 0, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = sql_row_count, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SOME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "SOUNDEX", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = soundexFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SQRT", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "STRFTIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "SUBSTR", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = substrFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SUM", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = sum_step, - .finalize = sumFinalize, - .export_to_sql = true, - }, { - .name = "TIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "TOTAL", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = sum_step, - .finalize = totalFinalize, - .export_to_sql = true, - }, { - .name = "TRIM", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = trim_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "TYPEOF", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_TYPEOF, - .call = typeofFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UNICODE", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = unicodeFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UNLIKELY", - .param_count = 1, - .returns = FIELD_TYPE_BOOLEAN, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UPPER", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, - .call = UpperICUFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "VERSION", - .param_count = 0, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = sql_func_version, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ZEROBLOB", - .param_count = 1, - .returns = FIELD_TYPE_VARBINARY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = zeroblobFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "_sql_stat_get", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "_sql_stat_init", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "_sql_stat_push", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, + {"ABS", 0, absFunc, NULL}, + {"AVG", 0, sum_step, avgFinalize}, + {"CEIL", 0, sql_builtin_stub, NULL}, + {"CEILING", 0, sql_builtin_stub, NULL}, + {"CHAR", 0, charFunc, NULL}, + {"CHARACTER_LENGTH", 0, lengthFunc, NULL}, + {"CHAR_LENGTH", 0, lengthFunc, NULL}, + {"COALESCE", SQL_FUNC_COALESCE, sql_builtin_stub, NULL}, + {"COUNT", 0, countStep, countFinalize}, + {"CURRENT_DATE", 0, sql_builtin_stub, NULL}, + {"CURRENT_TIME", 0, sql_builtin_stub, NULL}, + {"CURRENT_TIMESTAMP", 0, sql_builtin_stub, NULL}, + {"DATE", 0, sql_builtin_stub, NULL}, + {"DATETIME", 0, sql_builtin_stub, NULL}, + {"EVERY", 0, sql_builtin_stub, NULL}, + {"EXISTS", 0, sql_builtin_stub, NULL}, + {"EXP", 0, sql_builtin_stub, NULL}, + {"EXTRACT", 0, sql_builtin_stub, NULL}, + {"FLOOR", 0, sql_builtin_stub, NULL}, + {"GREATER", 0, sql_builtin_stub, NULL}, + {"GREATEST", SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, minmaxFunc, NULL}, + {"GROUP_CONCAT", 0, groupConcatStep, groupConcatFinalize}, + {"HEX", 0, hexFunc, NULL}, + {"IFNULL", SQL_FUNC_COALESCE, sql_builtin_stub, NULL}, + {"JULIANDAY", 0, sql_builtin_stub, NULL}, + {"LEAST", SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, minmaxFunc, NULL}, + {"LENGTH", SQL_FUNC_LENGTH, lengthFunc, NULL}, + {"LESSER", 0, sql_builtin_stub, NULL}, + {"LIKE", SQL_FUNC_NEEDCOLL | SQL_FUNC_LIKE, likeFunc, NULL}, + {"LIKELIHOOD", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"LIKELY", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"LN", 0, sql_builtin_stub, NULL}, + {"LOWER", SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, LowerICUFunc, NULL}, + {"MAX", SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, minmaxStep, minMaxFinalize}, + {"MIN", SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, minmaxStep, minMaxFinalize}, + {"MOD", 0, sql_builtin_stub, NULL}, + {"NULLIF", SQL_FUNC_NEEDCOLL, nullifFunc, NULL}, + {"OCTET_LENGTH", 0, sql_builtin_stub, NULL}, + {"POSITION", SQL_FUNC_NEEDCOLL, position_func, NULL}, + {"POWER", 0, sql_builtin_stub, NULL}, + {"PRINTF", 0, printfFunc, NULL}, + {"QUOTE", 0, quoteFunc, NULL}, + {"RANDOM", 0, randomFunc, NULL}, + {"RANDOMBLOB", 0, randomBlob, NULL}, + {"REPLACE", SQL_FUNC_DERIVEDCOLL, replaceFunc, NULL}, + {"ROUND", 0, roundFunc, NULL}, + {"ROW_COUNT", 0, sql_row_count, NULL}, + {"SOME", 0, sql_builtin_stub, NULL}, + {"SOUNDEX", 0, soundexFunc, NULL}, + {"SQRT", 0, sql_builtin_stub, NULL}, + {"STRFTIME", 0, sql_builtin_stub, NULL}, + {"SUBSTR", SQL_FUNC_DERIVEDCOLL, substrFunc, NULL}, + {"SUM", 0, sum_step, sumFinalize}, + {"TIME", 0, sql_builtin_stub, NULL}, + {"TOTAL", 0, sum_step, totalFinalize}, + {"TRIM", SQL_FUNC_DERIVEDCOLL, trim_func, NULL}, + {"TYPEOF", SQL_FUNC_TYPEOF, typeofFunc, NULL}, + {"UNICODE", 0, unicodeFunc, NULL}, + {"UNLIKELY", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"UPPER", SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, UpperICUFunc, NULL}, + {"VERSION", 0, sql_func_version, NULL}, + {"ZEROBLOB", 0, zeroblobFunc, NULL}, + {"_sql_stat_get", 0, sql_builtin_stub, NULL}, + {"_sql_stat_init", 0, sql_builtin_stub, NULL}, + {"_sql_stat_push", 0, sql_builtin_stub, NULL}, }; static struct func_vtab func_sql_builtin_vtab; -- 2.25.1 From v.shpilevoy at tarantool.org Wed Aug 12 23:10:13 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 12 Aug 2020 22:10:13 +0200 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported In-Reply-To: References: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> <4a43127c-6e11-8f28-27ee-62e24fa65a4b@tarantool.org> Message-ID: On 12.08.2020 14:59, Yaroslav Dynnikov wrote: > Hi, Vlad. > > Number 7 is the libcurl age. According to the documentation, age should be set to the version of this functionality by the time you write your program. This way, libcurl will always return a proper struct that your program understands, while programs in the future might get a different struct. See https://curl.haxx.se/libcurl/c/curl_version_info.html > > So it's safe enough to rely on the structure. And I think both checkes (libssl and libz) are important enough to be tracked by tests. We don't want them to be silently gone, do we? Still the test is overcomplicated, I think. But at least it should be stable, if with the same age number all curl versions will return the same curl_version_info_data. LGTM. From v.shpilevoy at tarantool.org Wed Aug 12 23:34:35 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 12 Aug 2020 22:34:35 +0200 Subject: [Tarantool-patches] [PATCH 1/2] tuple: fix multikey field JSON access crash In-Reply-To: <184a98de-9f49-be82-39a5-84b3c7510697@tarantool.org> References: <53095890-c0b6-6eae-b3b1-9898e7ca05e3@tarantool.org> <8d39159c-31d3-1394-fb04-4793da531772@tarantool.org> <184a98de-9f49-be82-39a5-84b3c7510697@tarantool.org> Message-ID: <8b323479-64fd-f6d1-d7ec-69de463e0988@tarantool.org> >> +??????? if (offset_slot_hint != NULL) { >> ????????????? *offset_slot_hint = offset_slot; >> +??????????? /* >> +???????????? * Hint is never requested for a multikey field without >> +???????????? * providing a concrete multikey index. >> +???????????? */ >> +??????????? assert(!field->is_multikey_part || >> +?????????????????? (multikey_idx != MULTIKEY_NONE && >> +??????????????? field->is_multikey_part)); > The last '&& field->is_multikey_part' is excess. Indeed, a stupid mistake. Somewhy I was sure it was needed yesterday. Dropped now. ==================== assert(!field->is_multikey_part || - (multikey_idx != MULTIKEY_NONE && - field->is_multikey_part)); + multikey_idx != MULTIKEY_NONE); ==================== From v.shpilevoy at tarantool.org Wed Aug 12 23:34:38 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 12 Aug 2020 22:34:38 +0200 Subject: [Tarantool-patches] [PATCH 0/2] JSON field multikey crash In-Reply-To: References: Message-ID: <6b013a97-18b1-62a0-5d3f-5bb53458e093@tarantool.org> Pushed to master, 2.5, 2.4. From v.shpilevoy at tarantool.org Thu Aug 13 00:23:42 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 12 Aug 2020 23:23:42 +0200 Subject: [Tarantool-patches] [PATCH V2] vinyl: rework upsert operation In-Reply-To: References: Message-ID: <02049f5a-702b-d2c7-6873-c5b26e9de005@tarantool.org> Hi! Thanks for the patch! > Closes #5105 > --- > Issues: > https://github.com/tarantool/tarantool/issues/5105 > > @ChangeLog: > - upserts now follow associative property: result of several upserts > doesn't depend on the order of their application (gh-5105); In my previous review I tried to explain that any squash now won't work, if we want to fix 5105. It does not matter if you check for type match. Because still there are overflows. Consider this test: box.cfg{} format = {} format[1] = {name = 'f1', type = 'unsigned'} format[2] = {name = 'f2', type = 'unsigned'} s = box.schema.space.create('test', {engine = 'vinyl', format = format}) _ = s:create_index('pk') uint_max = 18446744073709551615ULL s:replace{1, uint_max - 2} box.snapshot() s:upsert({1, 0}, {{'+', 2, 1}}) s:upsert({1, 0}, {{'+', 2, 1}}) s:upsert({1, 0}, {{'+', 2, 1}}) box.snapshot() s:select() --- - - [1, 18446744073709551614] ... Last 2 upserts were ignored. Even though only the last one should have been. Indeed, if I comment it out, the result becomes correct: box.cfg{} format = {} format[1] = {name = 'f1', type = 'unsigned'} format[2] = {name = 'f2', type = 'unsigned'} s = box.schema.space.create('test', {engine = 'vinyl', format = format}) _ = s:create_index('pk') uint_max = 18446744073709551615ULL s:replace{1, uint_max - 2} box.snapshot() s:upsert({1, 0}, {{'+', 2, 1}}) s:upsert({1, 0}, {{'+', 2, 1}}) -- s:upsert({1, 0}, {{'+', 2, 1}}) box.snapshot() s:select() --- - - [1, 18446744073709551615] ... Also I tried to make it depend on order, not on being applied or not. And used this test: box.cfg{} format = {} format[1] = {name = 'f1', type = 'unsigned'} format[2] = {name = 'f2', type = 'unsigned'} s = box.schema.space.create('test', {engine = 'vinyl', format = format}) _ = s:create_index('pk') uint_max = 18446744073709551615ULL s:replace{1, uint_max - 2, 0} box.snapshot() s:upsert({1, 0, 0}, {{'+', 2, 1}}) s:upsert({1, 0, 0}, {{'+', 2, 1}}) -- I wanted to check, if +0.5 will prevent squashing, because -- it does not satisfy format. And that if I make it after all -- the other upserts, the bug will appear again. s:upsert({1, 0, 0}, {{'+', 2, 0.5}}) s:upsert({1, 0, 0}, {{'+', 2, 1}}) box.snapshot() s:select() But got another bug instead: main/103/interactive I> vinyl checkpoint started main/105/checkpoint_daemon I> scheduled next checkpoint for Thu Aug 13 00:52:21 2020 main/107/vinyl.scheduler I> 512/0: dump started vinyl.dump.0/103/task vy_stmt.c:174 E> ER_VINYL_MAX_TUPLE_SIZE: Failed to allocate 1347440769 bytes for tuple: tuple is too large. Check 'vinyl_max_tuple_size' configuration option. main/107/vinyl.scheduler vy_stmt.c:174 E> ER_VINYL_MAX_TUPLE_SIZE: Failed to allocate 1347440769 bytes for tuple: tuple is too large. Check 'vinyl_max_tuple_size' configuration option. main/107/vinyl.scheduler vy_scheduler.c:1292 E> 512/0: dump failed main/107/vinyl.scheduler vy_scheduler.c:2058 W> throttling scheduler for 1 second(s) snapshot/101/main I> saving snapshot `./00000000000000000008.snap.inprogress' snapshot/101/main I> done main/103/interactive vy_scheduler.c:740 E> vinyl checkpoint failed: Failed to allocate 1347440769 bytes for tuple: tuple is too large. Check 'vinyl_max_tuple_size' configuration option. --- - error: 'Failed to allocate 1347440769 bytes for tuple: tuple is too large. Check ''vinyl_max_tuple_size'' configuration option.' ... Seems the tuple is corrupted somewhere, because a few numbers clearly can't occupy 1347440769 bytes. From kyukhin at tarantool.org Thu Aug 13 14:48:59 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Thu, 13 Aug 2020 14:48:59 +0300 Subject: [Tarantool-patches] [PATCH] Ensure all curl symbols are exported In-Reply-To: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> References: <20200806131955.3400088-1-yaroslav.dynnikov@tarantool.org> Message-ID: <20200813114859.axmmoc7z2c227v6j@tarantool.org> Hello, On 06 ??? 16:19, Yaroslav Dynnikov wrote: > In the recent update of libcurl (2.5.0-278-g807c7fa58) its layout has > changed: private function `Curl_version_init()` which used to fill-in > info structure was eliminated. As a result, no symbols for > `libcurl_la-version.o` remained used, so it wasn't included in tarantool > binary. And `curl_version` and `curl_version_info` symbols went missing. > > According to libcurl naming conventions all exported symbols are named > as `curl_*`. This patch lists them all explicitly in `exprots.h` and > adds the test. > > Close #5223 > > Issue: https://github.com/tarantool/tarantool/issues/5223 > Branch: https://github.com/tarantool/tarantool/tree/rosik/gh-5223-missing-curl-symbols I've checked your patch into master. In future, could you please append a Relase Notes entry to your patches. -- Regards, Kirill Yukhin From v.shpilevoy at tarantool.org Fri Aug 14 00:58:20 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 13 Aug 2020 23:58:20 +0200 Subject: [Tarantool-patches] [PATCH 1/1] xrow: introduce struct synchro_request Message-ID: All requests saved to WAL and transmitted through network have their own request structure with parameters: - struct request for DML; - struct call_request for CALL/EVAL; - struct auth_request for AUTH; - struct ballot for VOTE; - struct sql_request for SQL; - struct greeting for greeting. It is done for a reason - not to pass all the request parameters into each function one by one, and manage them all at once instead. For synchronous requests IPROTO_CONFIRM and IPROTO_ROLLBACK it was not done. Because so far it was not too hard to carry just 2 parameters: lsn and replica_id, from their body. But it will be changed in #5129. Because in fact these requests have more parameters, but they were filled by txn module, since synchro requests were saved to WAL via transactions (due to lack of alternative API to access WAL). After #5129 it will be necessary to save LSN and replica_id of the request author. This patch introduces struct synchro_request to simplify extension of the synchro parameters. Closes #5151 Needed for #5129 --- Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5151-synchro_request Issue: https://github.com/tarantool/tarantool/issues/5151 src/box/applier.cc | 32 +++++------------------ src/box/box.cc | 21 ++++----------- src/box/txn_limbo.c | 56 ++++++++++++++++++++++++++-------------- src/box/txn_limbo.h | 15 +++-------- src/box/xrow.c | 52 +++++++++---------------------------- src/box/xrow.h | 63 +++++++++++++++++++++------------------------ 6 files changed, 93 insertions(+), 146 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index a953d293e..98fb87375 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -275,26 +275,14 @@ process_nop(struct request *request) * or rollback some of its entries. */ static int -process_confirm_rollback(struct request *request, bool is_confirm) +process_synchro_row(struct request *request) { assert(iproto_type_is_synchro_request(request->header->type)); - uint32_t replica_id; struct txn *txn = in_txn(); - int64_t lsn = 0; - int res = 0; - if (is_confirm) - res = xrow_decode_confirm(request->header, &replica_id, &lsn); - else - res = xrow_decode_rollback(request->header, &replica_id, &lsn); - if (res == -1) - return -1; - - if (replica_id != txn_limbo.instance_id) { - diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, replica_id, - txn_limbo.instance_id); + struct synchro_request syn_req; + if (xrow_decode_synchro(request->header, &syn_req) != 0) return -1; - } assert(txn->n_applier_rows == 0); /* * This is not really a transaction. It just uses txn API @@ -306,16 +294,9 @@ process_confirm_rollback(struct request *request, bool is_confirm) if (txn_begin_stmt(txn, NULL) != 0) return -1; - - if (txn_commit_stmt(txn, request) == 0) { - if (is_confirm) - txn_limbo_read_confirm(&txn_limbo, lsn); - else - txn_limbo_read_rollback(&txn_limbo, lsn); - return 0; - } else { + if (txn_commit_stmt(txn, request) != 0) return -1; - } + return txn_limbo_process(&txn_limbo, &syn_req); } static int @@ -324,8 +305,7 @@ apply_row(struct xrow_header *row) struct request request; if (iproto_type_is_synchro_request(row->type)) { request.header = row; - return process_confirm_rollback(&request, - row->type == IPROTO_CONFIRM); + return process_synchro_row(&request); } if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; diff --git a/src/box/box.cc b/src/box/box.cc index 83eef5d98..8e811e9c1 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -367,22 +367,11 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) { struct request request; if (iproto_type_is_synchro_request(row->type)) { - uint32_t replica_id; - int64_t lsn; - switch(row->type) { - case IPROTO_CONFIRM: - if (xrow_decode_confirm(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_confirm(&txn_limbo, lsn); - break; - case IPROTO_ROLLBACK: - if (xrow_decode_rollback(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_rollback(&txn_limbo, lsn); - break; - } + struct synchro_request syn_req; + if (xrow_decode_synchro(row, &syn_req) != 0) + diag_raise(); + if (txn_limbo_process(&txn_limbo, &syn_req) != 0) + diag_raise(); return; } xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index a2043c17a..944161c30 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -31,6 +31,7 @@ #include "txn.h" #include "txn_limbo.h" #include "replication.h" +#include "iproto_constants.h" struct txn_limbo txn_limbo; @@ -272,11 +273,15 @@ complete: } static void -txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, - bool is_confirm) +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { assert(lsn > 0); + struct synchro_request req; + req.type = type; + req.replica_id = limbo->instance_id; + req.lsn = lsn; + struct xrow_header row; struct request request = { .header = &row, @@ -286,19 +291,7 @@ txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, if (txn == NULL) goto rollback; - int res = 0; - if (is_confirm) { - res = xrow_encode_confirm(&row, &txn->region, - limbo->instance_id, lsn); - } else { - /* - * This LSN is the first to be rolled back, so - * the last "safe" lsn is lsn - 1. - */ - res = xrow_encode_rollback(&row, &txn->region, - limbo->instance_id, lsn); - } - if (res == -1) + if (xrow_encode_synchro(&row, &txn->region, &req) != 0) goto rollback; /* * This is not really a transaction. It just uses txn API @@ -325,7 +318,7 @@ rollback: * problems are fixed. Or retry automatically with some period. */ panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, is_confirm ? "CONFIRM" : "ROLLBACK"); + "%s\n", lsn, iproto_type_name(type)); } /** @@ -338,10 +331,11 @@ txn_limbo_write_confirm(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->confirmed_lsn = lsn; - txn_limbo_write_confirm_rollback(limbo, lsn, true); + txn_limbo_write_synchro(limbo, IPROTO_CONFIRM, lsn); } -void +/** Confirm all the entries <= @a lsn. */ +static void txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -390,11 +384,12 @@ txn_limbo_write_rollback(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->is_in_rollback = true; - txn_limbo_write_confirm_rollback(limbo, lsn, false); + txn_limbo_write_synchro(limbo, IPROTO_ROLLBACK, lsn); limbo->is_in_rollback = false; } -void +/** Rollback all the entries >= @a lsn. */ +static void txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -577,6 +572,27 @@ complete: return 0; } +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) +{ + if (req->replica_id != limbo->instance_id) { + diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, req->replica_id, + limbo->instance_id); + return -1; + } + switch (req->type) { + case IPROTO_CONFIRM: + txn_limbo_read_confirm(limbo, req->lsn); + break; + case IPROTO_ROLLBACK: + txn_limbo_read_rollback(limbo, req->lsn); + break; + default: + unreachable(); + } + return 0; +} + void txn_limbo_force_empty(struct txn_limbo *limbo, int64_t confirm_lsn) { diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h index 04ee7ea5c..eaf662987 100644 --- a/src/box/txn_limbo.h +++ b/src/box/txn_limbo.h @@ -39,6 +39,7 @@ extern "C" { #endif /* defined(__cplusplus) */ struct txn; +struct synchro_request; /** * Transaction and its quorum metadata, to be stored in limbo. @@ -245,17 +246,9 @@ txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn); int txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry); -/** - * Confirm all the entries up to the given master's LSN. - */ -void -txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn); - -/** - * Rollback all the entries starting with given master's LSN. - */ -void -txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn); +/** Execute a synchronous replication request. */ +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req); /** * Waiting for confirmation of all "sync" transactions diff --git a/src/box/xrow.c b/src/box/xrow.c index 0c797a9d5..4b5d4356f 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,13 +893,13 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -static int -xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn, int type) +int +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req) { size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(replica_id) + mp_sizeof_uint(IPROTO_LSN) + - mp_sizeof_uint(lsn); + mp_sizeof_uint(req->replica_id) + + mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); char *buf = (char *)region_alloc(region, len); if (buf == NULL) { diag_set(OutOfMemory, len, "region_alloc", "buf"); @@ -909,9 +909,9 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, pos = mp_encode_map(pos, 2); pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, replica_id); + pos = mp_encode_uint(pos, req->replica_id); pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, lsn); + pos = mp_encode_uint(pos, req->lsn); memset(row, 0, sizeof(*row)); @@ -919,30 +919,13 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, row->body[0].iov_len = len; row->bodycnt = 1; - row->type = type; + row->type = req->type; return 0; } int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_CONFIRM); -} - -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_ROLLBACK); -} - -static int -xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, - int64_t *lsn) +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req) { if (row->bodycnt == 0) { diag_set(ClientError, ER_INVALID_MSGPACK, "request body"); @@ -977,30 +960,19 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, } switch (key) { case IPROTO_REPLICA_ID: - *replica_id = mp_decode_uint(&d); + req->replica_id = mp_decode_uint(&d); break; case IPROTO_LSN: - *lsn = mp_decode_uint(&d); + req->lsn = mp_decode_uint(&d); break; default: mp_next(&d); } } + req->type = row->type; return 0; } -int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - -int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - int xrow_to_iovec(const struct xrow_header *row, struct iovec *out) { diff --git a/src/box/xrow.h b/src/box/xrow.h index e21ede5a3..02dca74e5 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -216,54 +216,51 @@ xrow_encode_dml(const struct request *request, struct region *region, struct iovec *iov); /** - * Encode the CONFIRM to row body and set row type to - * IPROTO_CONFIRM. - * @param row xrow header. - * @param region Region to use to encode the confirmation body. - * @param replica_id master's instance id. - * @param lsn last confirmed lsn. - * @retval -1 on error. - * @retval 0 success. + * Synchronous replication request - confirmation or rollback of + * pending synchronous transactions. */ -int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +struct synchro_request { + /** Operation type - IPROTO_ROLLBACK or IPROTO_CONFIRM. */ + uint32_t type; + /** + * ID of the instance owning the pending transactions. + * Note, it may be not the same instance, who created this + * request. An instance can make an operation on foreign + * synchronous transactions in case a new master tries to + * finish transactions of an old master. + */ + uint32_t replica_id; + /** + * Operation LSN. + * In case of CONFIRM it means 'confirm all + * transactions with lsn <= this value'. + * In case of ROLLBACK it means 'rollback all transactions + * with lsn >= this value'. + */ + int64_t lsn; +}; /** - * Decode the CONFIRM request body. + * Encode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn last confirmed lsn. + * @param region Region to use to encode the confirmation body. + * @param req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); - -/** - * Encode the ROLLBACK row body and set row type to - * IPROTO_ROLLBACK. - * @param row xrow header. - * @param region Region to use to encode the rollback body. - * @param replica_id master's instance id. - * @param lsn lsn to rollback from, including it. - * @retval -1 on error. - * @retval 0 success. - */ -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req); /** - * Decode the ROLLBACK row body. + * Decode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn lsn to rollback from, including it. + * @param[out] req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); /** * CALL/EVAL request. -- 2.21.1 (Apple Git-122.3) From gorcunov at gmail.com Fri Aug 14 01:17:57 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Fri, 14 Aug 2020 01:17:57 +0300 Subject: [Tarantool-patches] [PATCH 1/1] xrow: introduce struct synchro_request In-Reply-To: References: Message-ID: <20200813221757.GB2074@grain> On Thu, Aug 13, 2020 at 11:58:20PM +0200, Vladislav Shpilevoy wrote: ... > +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) > { > assert(lsn > 0); > > + struct synchro_request req; > + req.type = type; > + req.replica_id = limbo->instance_id; > + req.lsn = lsn; > + Vlad, while you're at this code, could we please use designated initialization from the very beginning, ie struct synchro_request req = { .type = type, .replica_id = limbo->instance_id, .lsn = lsn, }; (the alignment is up to you though). Such initialization won't allow a bug to happen when we get a structure extension and other non updated fields will be zeroified. Other that this looks great! From alexander.turenko at tarantool.org Fri Aug 14 05:44:42 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Fri, 14 Aug 2020 05:44:42 +0300 Subject: [Tarantool-patches] [PATCH v2 2/3] Fix test box-tap/cfg.test.lua on openSuSE In-Reply-To: <99ba534fc775257849ba5b3d9b6bdd5711c817cb.1594126705.git.avtikhon@tarantool.org> References: <99ba534fc775257849ba5b3d9b6bdd5711c817cb.1594126705.git.avtikhon@tarantool.org> Message-ID: <20200814024442.4lgthzt576txghch@tkn_work_nb> > Fix test box-tap/cfg.test.lua on openSuSE > > Found that test fails on its subtest checking: > "gh-2872 bootstrap is aborted if vinyl_dir > contains vylog files left from previous runs" > > Debugging src/box/xlog.c found that all checks > were correct, but at function: > src/box/vy_log.c:vy_log_bootstrap() > the check on of the errno on ENOENT blocked the > negative return from function: > src/box/xlog.c:xdir_scan() > > Found that errno was already set to ENOENT before > the xdir_scan() call. To fix the issue the errno > must be clean before the call to xdir_scan, because > we are interesting in it only from this function. > The same issue fixed at function: > src/box/vy_log.c:vy_log_begin_recovery() > > Closes #4594 > Needed for #4562 My wish is to find the following information in the commit message: - The header: the subsystem prefix (vinyl) and what is fixed (vinyl_dir existence check at bootstrap). - What was wrong and becomes right (high-level or from a user perspective): when memtx_dir is not exists, but vinyl_dir exists and errno is set to ENOENT, box configuration succeeds, however it is should not. - What is the reason of the wrong behaviour: not all failure paths in xdir_scan() are set errno, but the caller assumes it. - From where the problem appears: openSUSE and box-tap/cfg.test.lua. > --- > src/box/vy_log.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/src/box/vy_log.c b/src/box/vy_log.c > index 311985c72..86599fd15 100644 > --- a/src/box/vy_log.c > +++ b/src/box/vy_log.c > @@ -1014,6 +1014,7 @@ vy_log_rebootstrap(void) > int > vy_log_bootstrap(void) > { > + errno = 0; > if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > return -1; You spotted the problem right. A bit more context: Usual C convention is to report success or failure using a return value and set errno at any error. So a caller usually just checks a return value and if it means a failure (usually -1), it checks errno to determine an exact reason. Usual convention in tarantool is a bit different: we use a special diagnostics area to report a reason of a failure. Not all failure paths of xdir_scan() sets errno (including our 'invalid instance UUID' case), so we cannot be sure that errno is not remains unchanged after a failure of the function. So, again, your fix is correct. However the approach with checking errno against ENOENT (No such file or directory) is not good (with or without your patch). For example: - What if xdir_scan() would be changed in future and, say, some call will rewrite errno after the opendir() call? - What if some other call inside xdir_scan() will set ENOENT: say, open() in xdir_open_cursor() due to some race? We lean on implementation details of the callee, not its contract. I think this way is too fragile and we should either: - check whether the directory exists before xdir_scan() call - or pass a flag to xdir_scan() whether the directory should exist. The latter looks better for me: it does not lead to code duplication. See the proposed diff below the email. BTW, there is the simple way to test the problem on any OS: | diff --git a/test/box-tap/cfg.test.lua b/test/box-tap/cfg.test.lua | index 569b5f463..2eda2f3c2 100755 | --- a/test/box-tap/cfg.test.lua | +++ b/test/box-tap/cfg.test.lua | @@ -392,6 +392,8 @@ s:create_index('pk') | os.exit(0) | ]], vinyl_dir)) | code = string.format([[ | +local errno = require('errno') | +errno(errno.ENOENT) | box.cfg{vinyl_dir = '%s'} | os.exit(0) | ]], vinyl_dir) Of course, we should not add it right to this test case, let's copy it somewhere and add separately. ---- diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..da3c50e87 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..2b894d680 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -559,7 +559,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..74f761994 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..3400eb75f 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); * snapshot or scan through all logs. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } From alexander.turenko at tarantool.org Fri Aug 14 05:55:29 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Fri, 14 Aug 2020 05:55:29 +0300 Subject: [Tarantool-patches] [PATCH v3 1/3] test: fix for OpenSuSE luajit tests build In-Reply-To: <1d2eb0604408a837fa2e3feecf924b2b1b216741.1594218821.git.avtikhon@tarantool.org> References: <0e52466494236c1f337b36f3248494f2cdeb4c4f.1594218821.git.avtikhon@tarantool.org> <1d2eb0604408a837fa2e3feecf924b2b1b216741.1594218821.git.avtikhon@tarantool.org> Message-ID: <20200814025529.fs3icxm26nsg7s6c@tkn_work_nb> This fix is land already in [1]. Please, rebase and update the patchset and resend it. I commented the patch re vinyl_dir existence check in [2] and merged necessary packpack support in [3]. The patch re rpmspec looks good at the brief glance. Don't forget about cover letter. [1]: https://github.com/tarantool/tarantool/commit/f526debcd84ae2d7bdc6c172f9a75d894ecc15dd [2]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019031.html [3]: https://github.com/packpack/packpack/pull/121 WBR, Alexander Turenko. On Wed, Jul 08, 2020 at 05:39:41PM +0300, Alexander V. Tikhonov wrote: > Found that OpenSUSE toolchain adds '--no-undefined' linked flag leading > to fails while building tests. The changes suppress this flag since > dynamic libraries are loaded via Tarantool executable and use its > symbols. So it is completely OK to have undefined symbols at build time. > Feel free to adjust it on your own. > > Needed for #4562 > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > Issue: https://github.com/tarantool/tarantool/issues/4562 > > test/CMakeLists.txt | 5 +++++ > 1 file changed, 5 insertions(+) > > diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt > index 697d1b21d..93de3d68b 100644 > --- a/test/CMakeLists.txt > +++ b/test/CMakeLists.txt > @@ -23,6 +23,11 @@ endfunction() > add_compile_flags("C;CXX" > "-Wno-unused-parameter") > > +# The dynamic libraries will be loaded from tarantool executable > +# and will use symbols from it. So it is completely okay to have > +# unresolved symbols at build time. > +string(REPLACE "-Wl,--no-undefined" "" CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS}") > + > if(POLICY CMP0037) > if(CMAKE_VERSION VERSION_LESS 3.11) > # cmake below 3.11 reserves name test. Use old policy. > -- > 2.17.1 > From avtikhon at tarantool.org Fri Aug 14 12:59:56 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 14 Aug 2020 12:59:56 +0300 Subject: [Tarantool-patches] [PATCH v4 0/2] gitlab-ci: implement openSUSE testing Message-ID: Implement openSUSE testing in gitlab-ci. It needed the following changes 1) packpack openSUSE build implementation - commited [3]. 2) added '--no-undefined' linker flag leading to fails while building tests - commited [1]. 3) new fix suggestion from A.Turenko with new test [2]. 4) patch for implementation testing for openSUSE. [1]: https://github.com/tarantool/tarantool/commit/f526debcd84ae2d7bdc6c172f9a75d894ecc15dd [2]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019031.html [3]: https://github.com/packpack/packpack/pull/121 Alexander V. Tikhonov (2): vinyl: check vinyl_dir existence at bootstrap gitlab-ci: add openSUSE packages build jobs --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci Issue: https://github.com/tarantool/tarantool/issues/4562 Issue: https://github.com/tarantool/tarantool/issues/4594 .gitlab-ci.yml | 24 ++++++++++++++++++++++++ rpm/tarantool.spec | 16 +++++++++++----- src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 ++-- src/box/vy_log.c | 4 ++-- src/box/wal.c | 2 +- src/box/xlog.c | 4 +++- src/box/xlog.h | 6 +++--- test/box-tap/cfg.test.lua | 23 ++++++++++++++++++++++- tools/update_repo.sh | 6 ++++-- 10 files changed, 73 insertions(+), 18 deletions(-) -- 2.17.1 From avtikhon at tarantool.org Fri Aug 14 12:59:57 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 14 Aug 2020 12:59:57 +0300 Subject: [Tarantool-patches] [PATCH v4 1/2] vinyl: check vinyl_dir existence at bootstrap In-Reply-To: References: Message-ID: <4eb7b26b9f80393b19536f49ab3985d72ba13d89.1597398597.git.avtikhon@tarantool.org> During implementation of openSUSE got failed box-tap/cfg.test.lua test. Found that when memtx_dir wasn't existed, while vinyl_dir existed and errno was set to ENOENT, box configuration succeeded, but it shouldn't. Reason of this wrong behaviour was that not all of the failure paths in xdir_scan() were set errno, but the caller assumed it. Usual C convention is to report success or failure using a return value and set errno at any error. So a caller usually just checks a return value and if it means a failure (usually -1), it checks errno to determine an exact reason. Usual convention in tarantool is a bit different: we use a special diagnostics area to report a reason of a failure. Not all failure paths of xdir_scan() sets errno (including our 'invalid instance UUID' case), so we cannot be sure that errno is not remains unchanged after a failure of the function. However the solution with checking errno against ENOENT (No such file or directory) is not good. For example: - What if xdir_scan() would be changed in future and, say, some call will rewrite errno after the opendir() call? - What if some other call inside xdir_scan() will set ENOENT: say, open() in xdir_open_cursor() due to some race? We lean on implementation details of the callee, not its contract. This way is too fragile and it should either check whether the directory exists before xdir_scan() call or pass a flag to xdir_scan() whether the directory should exist. Decided to use second variant - it does not lead to code duplication. Added subtest to box-tap/cfg.test.lua test file, to check the currently fixed issue. Closes #4594 Needed for #4562 Co-authored-by: Alexander Turenko --- src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 ++-- src/box/vy_log.c | 4 ++-- src/box/wal.c | 2 +- src/box/xlog.c | 4 +++- src/box/xlog.h | 6 +++--- test/box-tap/cfg.test.lua | 23 ++++++++++++++++++++++- 7 files changed, 34 insertions(+), 11 deletions(-) diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..da3c50e87 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..2b894d680 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -559,7 +559,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..74f761994 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..3400eb75f 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); * snapshot or scan through all logs. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } diff --git a/test/box-tap/cfg.test.lua b/test/box-tap/cfg.test.lua index 569b5f463..a60aa848e 100755 --- a/test/box-tap/cfg.test.lua +++ b/test/box-tap/cfg.test.lua @@ -6,7 +6,7 @@ local socket = require('socket') local fio = require('fio') local uuid = require('uuid') local msgpack = require('msgpack') -test:plan(108) +test:plan(109) -------------------------------------------------------------------------------- -- Invalid values @@ -605,5 +605,26 @@ test:ok(not box.info.listen:match(':0'), 'real port in info.listen') box.cfg{listen = box.NULL} test:is(nil, box.info.listen, 'cfg.listen reset drops info.listen') +-- +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and +-- errno is set to ENOENT, box configuration succeeds, however it +-- should not +-- +vinyl_dir = fio.tempdir() +run_script(string.format([[ +box.cfg{vinyl_dir = '%s'} +s = box.schema.space.create('test', {engine = 'vinyl'}) +s:create_index('pk') +os.exit(0) +]], vinyl_dir)) +code = string.format([[ +local errno = require('errno') +errno(errno.ENOENT) +box.cfg{vinyl_dir = '%s'} +os.exit(0) +]], vinyl_dir) +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") +fio.rmtree(vinyl_dir) + test:check() os.exit(0) -- 2.17.1 From avtikhon at tarantool.org Fri Aug 14 12:59:58 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 14 Aug 2020 12:59:58 +0300 Subject: [Tarantool-patches] [PATCH v4 2/2] gitlab-ci: add openSUSE packages build jobs In-Reply-To: References: Message-ID: Implemented openSUSE packages build with testing for images: opensuse-leap:15.[0-2] Added %{sle_version} checks in Tarantool spec file according to https://en.opensuse.org/openSUSE:Packaging_for_Leap#RPM_Distro_Version_Macros Added opensuse-leap of 15.1 and 15.2 versions to Gitlab-CI packages building/deploing jobs. Closes #4562 --- .gitlab-ci.yml | 24 ++++++++++++++++++++++++ rpm/tarantool.spec | 16 +++++++++++----- tools/update_repo.sh | 6 ++++-- 3 files changed, 39 insertions(+), 7 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 0ead08711..05d40b013 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -430,6 +430,18 @@ debian_10: OS: 'debian' DIST: 'buster' +opensuse_15_1: + <<: *pack_definition + variables: + OS: 'opensuse-leap' + DIST: '15.1' + +opensuse_15_2: + <<: *pack_definition + variables: + OS: 'opensuse-leap' + DIST: '15.2' + # Deploy sources_deploy: @@ -527,6 +539,18 @@ debian_10_deploy: OS: 'debian' DIST: 'buster' +opensuse_15_1_deploy: + <<: *deploy_definition + variables: + OS: 'opensuse-leap' + DIST: '15.1' + +opensuse_15_2_deploy: + <<: *deploy_definition + variables: + OS: 'opensuse-leap' + DIST: '15.2' + # Static builds static_build: diff --git a/rpm/tarantool.spec b/rpm/tarantool.spec index 88b1d6b5c..eedc0312c 100644 --- a/rpm/tarantool.spec +++ b/rpm/tarantool.spec @@ -1,5 +1,5 @@ # Enable systemd for on RHEL >= 7 and Fedora >= 15 -%if (0%{?fedora} >= 15 || 0%{?rhel} >= 7) +%if (0%{?fedora} >= 15 || 0%{?rhel} >= 7 || 0%{?sle_version} >= 1500) %bcond_without systemd %else %bcond_with systemd @@ -7,7 +7,7 @@ BuildRequires: cmake >= 2.8 BuildRequires: make -%if (0%{?fedora} >= 22 || 0%{?rhel} >= 7) +%if (0%{?fedora} >= 22 || 0%{?rhel} >= 7 || 0%{?sle_version} >= 1500) # RHEL 6 requires devtoolset BuildRequires: gcc >= 4.5 BuildRequires: gcc-c++ >= 4.5 @@ -53,6 +53,12 @@ Requires(preun): initscripts %bcond_without backtrace %endif +# openSuSE sets its own build directory in its macros, but we +# want to use in-source build there to simplify the RPM spec. +%if (0%{?sle_version} >= 1500) +%global __builddir . +%endif + %if %{with backtrace} BuildRequires: libunwind-devel # @@ -78,7 +84,7 @@ BuildRequires: python2-six >= 1.9.0 BuildRequires: python2-gevent >= 1.0 BuildRequires: python2-yaml >= 3.0.9 %else -%if (0%{?rhel} != 6) +%if (0%{?rhel} != 6 || 0%{?sle_version} >= 1500) BuildRequires: python >= 2.7 BuildRequires: python-six >= 1.9.0 BuildRequires: python-gevent >= 1.0 @@ -105,7 +111,7 @@ Requires: /etc/services # Deps for built-in package manager # https://github.com/tarantool/tarantool/issues/2612 Requires: openssl -%if (0%{?fedora} >= 22 || 0%{?rhel} >= 8) +%if (0%{?fedora} >= 22 || 0%{?rhel} >= 8 || 0%{?sle_version} >= 1500) # RHEL <= 7 doesn't support Recommends: Recommends: tarantool-devel Recommends: git-core @@ -164,7 +170,7 @@ rm -rf %{buildroot}%{_datarootdir}/doc/tarantool/ %check %if "%{_ci}" == "travis" -%if (0%{?fedora} >= 22 || 0%{?rhel} >= 7) +%if (0%{?fedora} >= 22 || 0%{?rhel} >= 7 || 0%{?sle_version} >= 1500) cd test && ./test-run.py --force -j 1 unit/ app/ app-tap/ box/ box-tap/ engine/ vinyl/ %endif %else diff --git a/tools/update_repo.sh b/tools/update_repo.sh index 5a68e3e05..02def11ff 100755 --- a/tools/update_repo.sh +++ b/tools/update_repo.sh @@ -6,7 +6,7 @@ rm_dir='rm -rf' mk_dir='mkdir -p' ws_prefix=/tmp/tarantool_repo_s3 -alloss='ubuntu debian el fedora' +alloss='ubuntu debian el fedora opensuse-leap' product=tarantool remove= force= @@ -31,6 +31,8 @@ function get_os_dists { alldists='6 7 8' elif [ "$os" == "fedora" ]; then alldists='27 28 29 30 31' + elif [ "$os" == "opensuse-leap" ]; then + alldists='15.0 15.1 15.2' fi echo "$alldists" @@ -934,7 +936,7 @@ if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then # unlock the publishing $rm_file $ws_lockfile popd -elif [ "$os" == "el" -o "$os" == "fedora" ]; then +elif [ "$os" == "el" -o "$os" == "fedora" -o "$os" == "opensuse-leap" ]; then # RPM packages structure needs different paths for binaries and sources # packages, in this way it is needed to call the packages registering # script twice with the given format: -- 2.17.1 From i.kosarev at tarantool.org Fri Aug 14 13:41:21 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Fri, 14 Aug 2020 13:41:21 +0300 Subject: [Tarantool-patches] [PATCH v2] tuple: drop extra restrictions for multikey index Message-ID: <20200814104121.6460-1-i.kosarev@tarantool.org> Multikey index did not work properly with nullable root field in tuple_raw_multikey_count(). Now it is fixed and corresponding restrictions are dropped. This also means that we can drop implicit nullability update for array/map fields and make all fields nullable by default, as it was until e1d3fe8ab8eed65394ad17409401a93b6fcdc435 (tuple format: don't allow null where array/map is expected), as far as default non-nullability itself doesn't solve any real problems while providing confusing behavior (gh-5027). Follow-up #5027 Closes #5192 --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5192-fix-multikey-index-restrictions Issue: https://github.com/tarantool/tarantool/issues/5192 @ChangeLog: * Dropped restrictions on nullable multikey index root. All fields are now nullable by default as it was before 2.2.1 (gh-5192). Changes in v2: - removed insignificant changes - fixed tests comments src/box/memtx_space.c | 8 --- src/box/tuple.c | 5 +- src/box/tuple_format.c | 20 ------ src/box/vinyl.c | 8 --- test/engine/gh-5027-fields-nullability.result | 71 +++---------------- .../gh-5027-fields-nullability.test.lua | 35 +++------ .../gh-5192-multikey-root-nullability.result | 62 ++++++++++++++++ ...gh-5192-multikey-root-nullability.test.lua | 20 ++++++ test/engine/json.result | 20 +++--- test/engine/json.test.lua | 6 +- test/engine/multikey.result | 8 +-- test/engine/multikey.test.lua | 4 +- 12 files changed, 123 insertions(+), 144 deletions(-) create mode 100644 test/engine/gh-5192-multikey-root-nullability.result create mode 100644 test/engine/gh-5192-multikey-root-nullability.test.lua diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index 8452ab430..d30ce44b8 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -604,14 +604,6 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) return -1; } } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } switch (index_def->type) { case HASH: if (! index_def->opts.is_unique) { diff --git a/src/box/tuple.c b/src/box/tuple.c index 9f0f24c64..94a3a96ba 100644 --- a/src/box/tuple.c +++ b/src/box/tuple.c @@ -554,7 +554,10 @@ tuple_raw_multikey_count(struct tuple_format *format, const char *data, NULL, MULTIKEY_NONE); if (array_raw == NULL) return 0; - assert(mp_typeof(*array_raw) == MP_ARRAY); + enum mp_type type = mp_typeof(*array_raw); + if (type == MP_NIL) + return 0; + assert(type == MP_ARRAY); return mp_decode_array(&array_raw); } diff --git a/src/box/tuple_format.c b/src/box/tuple_format.c index bae6c67cd..9b817d3cf 100644 --- a/src/box/tuple_format.c +++ b/src/box/tuple_format.c @@ -242,24 +242,6 @@ tuple_field_ensure_child_compatibility(struct tuple_field *parent, return 0; } -/** - * Tuple fields are nullable by default. However, it is not ok - * for array/map fields, as far as their implicit nullability - * might break field accessors expectations, provide confusing - * error messages and cause incorrect behaviour of - * tuple_multikey_count(). Thus array/map fields have to be - * non-nullable by default, which means we have to update default - * nullability for them. - */ -static void -tuple_field_update_nullability(struct tuple_field *field) -{ - if ((field->type == FIELD_TYPE_ARRAY || - field->type == FIELD_TYPE_MAP) && - tuple_field_is_nullable(field)) - field->nullable_action = ON_CONFLICT_ACTION_DEFAULT; -} - /** * Given a field number and a path, add the corresponding field * to the tuple format, allocating intermediate fields if @@ -335,13 +317,11 @@ tuple_format_add_field(struct tuple_format *format, uint32_t fieldno, parent->offset_slot = *current_slot; } } - tuple_field_update_nullability(parent); parent->is_key_part = true; next->is_multikey_part = is_multikey; parent = next; token_count++; } - tuple_field_update_nullability(parent); /* * The path has already been verified by the * key_def_decode_parts function. diff --git a/src/box/vinyl.c b/src/box/vinyl.c index 32301d7ba..5fa3ea3a5 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -658,14 +658,6 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) diag_set(ClientError, ER_NULLABLE_PRIMARY, space_name(space)); return -1; } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } /* Check that there are no ANY, ARRAY, MAP parts */ for (uint32_t i = 0; i < key_def->part_count; i++) { struct key_part *part = &key_def->parts[i]; diff --git a/test/engine/gh-5027-fields-nullability.result b/test/engine/gh-5027-fields-nullability.result index 1121727f6..382944d53 100644 --- a/test/engine/gh-5027-fields-nullability.result +++ b/test/engine/gh-5027-fields-nullability.result @@ -12,19 +12,19 @@ _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=true}}}) | --- | ... -s:replace{1} +s:replace{1} -- ok | --- | - [1] | ... -s:replace{1, box.NULL} +s:replace{1, box.NULL} -- ok | --- | - [1, null] | ... -s:replace{1, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL} -- ok | --- | - [1, null, null] | ... -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL, box.NULL} -- ok | --- | - [1, null, null, null] | ... @@ -41,79 +41,26 @@ _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) | --- | ... -s:replace{1} +s:replace{1} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL} +s:replace{1, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL, 5} +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok | --- | - [1, null, null, null, 5] | ... s:drop() | --- | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - [1, null] - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | - error: multikey index does not support nullable root field - | ... -s:replace{2, box.NULL} - | --- - | - [2, null] - | ... -s:drop() - | --- - | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array'}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | ... -s:replace{2, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -s:replace{3, {}} - | --- - | - [3, []] - | ... -s:drop() - | --- - | ... diff --git a/test/engine/gh-5027-fields-nullability.test.lua b/test/engine/gh-5027-fields-nullability.test.lua index 960103d6c..664883d0d 100644 --- a/test/engine/gh-5027-fields-nullability.test.lua +++ b/test/engine/gh-5027-fields-nullability.test.lua @@ -3,35 +3,18 @@ test_run = require('test_run').new() s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=true}}}) -s:replace{1} -s:replace{1, box.NULL} -s:replace{1, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1} -- ok +s:replace{1, box.NULL} -- ok +s:replace{1, box.NULL, box.NULL} -- ok +s:replace{1, box.NULL, box.NULL, box.NULL} -- ok s:drop() s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) -s:replace{1} -s:replace{1, box.NULL} -s:replace{1, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL, 5} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array'}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:replace{3, {}} +s:replace{1} -- error +s:replace{1, box.NULL} -- error +s:replace{1, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok s:drop() diff --git a/test/engine/gh-5192-multikey-root-nullability.result b/test/engine/gh-5192-multikey-root-nullability.result new file mode 100644 index 000000000..dcbb5067f --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.result @@ -0,0 +1,62 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- ok + | --- + | - [1, null] + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- ok + | --- + | - [2, null] + | ... +_ = s:delete(2) + | --- + | ... +s:truncate() + | --- + | ... +s:drop() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array'}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +s:replace{3, {}} -- ok + | --- + | - [3, []] + | ... +s:drop() + | --- + | ... diff --git a/test/engine/gh-5192-multikey-root-nullability.test.lua b/test/engine/gh-5192-multikey-root-nullability.test.lua new file mode 100644 index 000000000..5a8305bf9 --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.test.lua @@ -0,0 +1,20 @@ +test_run = require('test_run').new() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- ok +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- ok +_ = s:delete(2) +s:truncate() +s:drop() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array'}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- error +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- error +s:replace{3, {}} -- ok +s:drop() \ No newline at end of file diff --git a/test/engine/json.result b/test/engine/json.result index 2175266fc..b8fd9a1b6 100644 --- a/test/engine/json.result +++ b/test/engine/json.result @@ -738,12 +738,11 @@ _ = s:create_index('sk', {parts = {{'[2][1].a', 'unsigned'}}}) ... s:insert{1, box.NULL} -- error --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{2, {box.NULL}} -- error --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{3} -- error --- @@ -764,16 +763,15 @@ s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} --- ... -s:insert{7, box.NULL} -- error +s:insert{7, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [7, null] ... -s:insert{8, {box.NULL}} -- error +s:insert{8, {box.NULL}} -- ok --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- [8, [null]] ... --- Skipping nullable fields is okay though. +-- Skipping nullable fields is also okay. s:insert{9} -- ok --- - [9] @@ -792,7 +790,9 @@ s:insert{12, {{a = box.NULL}}} -- ok ... s.index.sk:select() --- -- - [9] +- - [7, null] + - [8, [null]] + - [9] - [10, []] - [11, [{'b': 1}]] - [12, [{'a': null}]] diff --git a/test/engine/json.test.lua b/test/engine/json.test.lua index 4771c3162..371bbad91 100644 --- a/test/engine/json.test.lua +++ b/test/engine/json.test.lua @@ -220,9 +220,9 @@ s:insert{4, {}} -- error s:insert{5, {{b = 1}}} -- error s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} -s:insert{7, box.NULL} -- error -s:insert{8, {box.NULL}} -- error --- Skipping nullable fields is okay though. +s:insert{7, box.NULL} -- ok +s:insert{8, {box.NULL}} -- ok +-- Skipping nullable fields is also okay. s:insert{9} -- ok s:insert{10, {}} -- ok s:insert{11, {{b = 1}}} -- ok diff --git a/test/engine/multikey.result b/test/engine/multikey.result index 968be4cc3..ff72983ec 100644 --- a/test/engine/multikey.result +++ b/test/engine/multikey.result @@ -794,9 +794,9 @@ _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) --- ... -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [1, null] ... s:insert{2, {box.NULL}} -- error --- @@ -814,9 +814,9 @@ s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} --- ... -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [5, null] ... s:insert{6, {box.NULL}} -- ok --- diff --git a/test/engine/multikey.test.lua b/test/engine/multikey.test.lua index ed7033494..653a9c9b1 100644 --- a/test/engine/multikey.test.lua +++ b/test/engine/multikey.test.lua @@ -214,12 +214,12 @@ s:drop() s = box.schema.space.create('test', {engine = engine}) _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok s:insert{2, {box.NULL}} -- error s:insert{3, {}} -- ok s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok s:insert{6, {box.NULL}} -- ok s:insert{7, {}} -- ok s:insert{8, {2}} -- ok -- 2.17.1 From i.kosarev at tarantool.org Fri Aug 14 13:46:55 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Fri, 14 Aug 2020 13:46:55 +0300 Subject: [Tarantool-patches] [PATCH v2] iproto: close socket explicitly before wal_dir at exit Message-ID: <20200814104655.6760-1-i.kosarev@tarantool.org> From: Ilya Kosarev tarantool instance didn't close socket explicitly which could cause hot standby instance to fail to bind in case it tries to bind before socket is closed by OS. Now it is fixed by closing socket explicitly before wal_dir. Closes #3967 --- https://github.com/tarantool/tarantool/tree/i.kosarev/gh-3967-close-socket-explicitly https://github.com/tarantool/tarantool/issues/3967 src/box/box.cc | 1 + src/box/iproto.cc | 13 +++++++++++++ src/box/iproto.h | 3 +++ 3 files changed, 17 insertions(+) diff --git a/src/box/box.cc b/src/box/box.cc index e7ec2891c..73d94f79b 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -1666,6 +1666,7 @@ box_free(void) tuple_free(); port_free(); #endif + iproto_free(); replication_free(); sequence_free(); gc_free(); diff --git a/src/box/iproto.cc b/src/box/iproto.cc index bea35ce87..b1d24910f 100644 --- a/src/box/iproto.cc +++ b/src/box/iproto.cc @@ -2080,3 +2080,16 @@ iproto_set_msg_max(int new_iproto_msg_max) iproto_do_cfg(&cfg_msg); cpipe_set_max_input(&net_pipe, new_iproto_msg_max / 2); } + +void +iproto_free() +{ + tt_pthread_cancel(net_cord.id); + tt_pthread_join(net_cord.id, NULL); + /* + * Close socket descriptor to prevent hot standby instance + * failing to bind in case it tries to bind + * before socket is closed by OS. + */ + close(binary.ev.fd); +} diff --git a/src/box/iproto.h b/src/box/iproto.h index b9a6cf8f7..8f3607ffc 100644 --- a/src/box/iproto.h +++ b/src/box/iproto.h @@ -80,6 +80,9 @@ iproto_listen(const char *uri); void iproto_set_msg_max(int iproto_msg_max); +void +iproto_free(); + #endif /* defined(__cplusplus) */ #endif -- 2.17.1 From i.kosarev at tarantool.org Fri Aug 14 13:47:21 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Fri, 14 Aug 2020 13:47:21 +0300 Subject: [Tarantool-patches] [PATCH] iproto: make iproto thread more independent from tx Message-ID: <20200814104721.6962-1-i.kosarev@tarantool.org> On connection, an evio service callback is invoked to accept it. The next step after acception was to process connection to tx thread through cbus. This meant that any connection interaction involves tx thread even before we get to decode what does the client want from us. Consequently, a number of problems appears. The main one is that we might get descriptor leak in case of unresponsive tx thread (for example, when building secondary index). There are some other cases where we might not want to spend precious tx time to process the connection in case iproto can do it all alone. This patch allows iproto to accept connection and send greeting by itself. The connection is initialized in tx thread when the real request comes through iproto_msg_decode(). In case request type was not recognized we can also send reply with an error without using tx. It is planned to add more iproto logic to prevent extra interaction with tx thread. This patch already to some extent solves descriptors leakage problem as far as connection establishes and stays in fetch_schema state while tx thread is unresponsive. The other user visible change is that on_connect triggers won't run on connections that don't provide any input, as reflected in bad_trigger.test.py. Part of #3776 --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-3776-handling-connections-in-iproto Issue: https://github.com/tarantool/tarantool/issues/3776 src/box/iproto.cc | 287 ++++++++++++++++++++------------ test/box-py/bad_trigger.result | 1 + test/box-py/bad_trigger.test.py | 22 ++- 3 files changed, 202 insertions(+), 108 deletions(-) diff --git a/src/box/iproto.cc b/src/box/iproto.cc index b8f65e5eca..a027d15c1d 100644 --- a/src/box/iproto.cc +++ b/src/box/iproto.cc @@ -245,7 +245,7 @@ iproto_msg_new(struct iproto_connection *con); static void iproto_resume(void); -static void +static int iproto_msg_decode(struct iproto_msg *msg, const char **pos, const char *reqend, bool *stop_input); @@ -256,6 +256,9 @@ iproto_msg_delete(struct iproto_msg *msg) iproto_resume(); } +static inline void +iproto_connection_delete(struct iproto_connection *con); + /** * A single global queue for all requests in all connections. All * requests from all connections are processed concurrently. @@ -280,6 +283,11 @@ static struct cord net_cord; * in the tx thread. */ static struct slab_cache net_slabc; +/** + * Slab cache used for allocating memory for output network buffers + * in the iproto thread. + */ +static struct slab_cache iproto_slabc; struct rmean *rmean_net; @@ -298,6 +306,9 @@ const char *rmean_net_strings[IPROTO_LAST] = { "REQUESTS", }; +static int +tx_init_connect(struct iproto_msg *msg); + static void tx_process_destroy(struct cmsg *m); @@ -650,7 +661,18 @@ iproto_connection_try_to_start_destroy(struct iproto_connection *con) * other parts of the connection. */ con->state = IPROTO_CONNECTION_DESTROYED; - cpipe_push(&tx_pipe, &con->destroy_msg); + if (con->session != NULL) + cpipe_push(&tx_pipe, &con->destroy_msg); + else { + /* + * In case session was not created we can safely destroy + * not involving tx thread. Thus we also need to destroy + * obuf, which still belongs to iproto thread. + */ + obuf_destroy(&con->obuf[0]); + obuf_destroy(&con->obuf[1]); + iproto_connection_delete(con); + } } /** @@ -677,9 +699,18 @@ iproto_connection_close(struct iproto_connection *con) * is done only once. */ con->p_ibuf->wpos -= con->parse_size; - cpipe_push(&tx_pipe, &con->disconnect_msg); assert(con->state == IPROTO_CONNECTION_ALIVE); con->state = IPROTO_CONNECTION_CLOSED; + rlist_del(&con->in_stop_list); + if (con->session != NULL) + cpipe_push(&tx_pipe, &con->disconnect_msg); + else + /* + * In case session was not created we can safely + * try to start destroy not involving tx thread. + */ + iproto_connection_try_to_start_destroy(con); + return; } else if (con->state == IPROTO_CONNECTION_PENDING_DESTROY) { iproto_connection_try_to_start_destroy(con); } else { @@ -809,6 +840,7 @@ iproto_enqueue_batch(struct iproto_connection *con, struct ibuf *in) assert(rlist_empty(&con->in_stop_list)); int n_requests = 0; bool stop_input = false; + bool obuf_in_iproto = (con->session == NULL); const char *errmsg; while (con->parse_size != 0 && !stop_input) { if (iproto_check_msg_max()) { @@ -853,12 +885,20 @@ err_msgpack: msg->len = reqend - reqstart; /* total request length */ - iproto_msg_decode(msg, &pos, reqend, &stop_input); - /* - * This can't throw, but should not be - * done in case of exception. - */ - cpipe_push_input(&tx_pipe, &msg->base); + if (iproto_msg_decode(msg, &pos, reqend, &stop_input) == 0) { + if (obuf_in_iproto) { + /* + * If session was not created yet and obuf is + * still in iproto we need to destroy it. New + * one will be created in tx thread if needed. + */ + obuf_destroy(&con->obuf[0]); + obuf_destroy(&con->obuf[1]); + obuf_in_iproto = false; + } + cpipe_push_input(&tx_pipe, &msg->base); + } + n_requests++; /* Request is parsed */ assert(reqend > reqstart); @@ -1105,8 +1145,8 @@ iproto_connection_new(int fd) ev_io_init(&con->output, iproto_connection_on_output, fd, EV_WRITE); ibuf_create(&con->ibuf[0], cord_slab_cache(), iproto_readahead); ibuf_create(&con->ibuf[1], cord_slab_cache(), iproto_readahead); - obuf_create(&con->obuf[0], &net_slabc, iproto_readahead); - obuf_create(&con->obuf[1], &net_slabc, iproto_readahead); + obuf_create(&con->obuf[0], &iproto_slabc, iproto_readahead); + obuf_create(&con->obuf[1], &iproto_slabc, iproto_readahead); con->p_ibuf = &con->ibuf[0]; con->tx.p_obuf = &con->obuf[0]; iproto_wpos_create(&con->wpos, con->tx.p_obuf); @@ -1134,10 +1174,6 @@ iproto_connection_delete(struct iproto_connection *con) assert(!evio_has_fd(&con->input)); assert(con->session == NULL); assert(con->state == IPROTO_CONNECTION_DESTROYED); - /* - * The output buffers must have been deleted - * in tx thread. - */ ibuf_destroy(&con->ibuf[0]); ibuf_destroy(&con->ibuf[1]); assert(con->obuf[0].pos == 0 && @@ -1172,6 +1208,9 @@ tx_reply_error(struct iproto_msg *msg); static void tx_reply_iproto_error(struct cmsg *m); +static void +tx_accept_wpos(struct iproto_connection *con, const struct iproto_wpos *wpos); + static void net_send_msg(struct cmsg *msg); @@ -1244,7 +1283,7 @@ static const struct cmsg_hop error_route[] = { { net_send_error, NULL }, }; -static void +static int iproto_msg_decode(struct iproto_msg *msg, const char **pos, const char *reqend, bool *stop_input) { @@ -1314,13 +1353,27 @@ iproto_msg_decode(struct iproto_msg *msg, const char **pos, const char *reqend, (uint32_t) type); goto error; } - return; + return 0; error: /** Log and send the error. */ diag_log(); diag_create(&msg->diag); diag_move(&fiber()->diag, &msg->diag); - cmsg_init(&msg->base, error_route); + if (msg->connection->session != NULL) { + cmsg_init(&msg->base, error_route); + return 0; + } + /* + * In case session was not created we can process error path + * without tx thread. + */ + tx_accept_wpos(msg->connection, &msg->wpos); + struct obuf *out = msg->connection->tx.p_obuf; + iproto_reply_error(out, diag_last_error(&msg->diag), + msg->header.sync, ::schema_version); + iproto_wpos_create(&msg->wpos, out); + net_send_error(&(msg->base)); + return -1; } static void @@ -1382,10 +1435,6 @@ tx_process_destroy(struct cmsg *m) session_destroy(con->session); con->session = NULL; /* safety */ } - /* - * Got to be done in iproto thread since - * that's where the memory is allocated. - */ obuf_destroy(&con->obuf[0]); obuf_destroy(&con->obuf[1]); } @@ -1478,13 +1527,21 @@ tx_accept_wpos(struct iproto_connection *con, const struct iproto_wpos *wpos) } } -static inline struct iproto_msg * -tx_accept_msg(struct cmsg *m) +static inline int +tx_accept_msg(struct cmsg *m, struct iproto_msg **msg) { - struct iproto_msg *msg = (struct iproto_msg *) m; - tx_accept_wpos(msg->connection, &msg->wpos); - tx_fiber_init(msg->connection->session, msg->header.sync); - return msg; + *msg = (struct iproto_msg *) m; + /* + * In case session was not created we need to init connection in tx and + * create it here. + */ + if ((*msg)->connection->session == NULL && tx_init_connect(*msg) != 0) { + (*msg)->close_connection = true; + return -1; + } + tx_accept_wpos((*msg)->connection, &(*msg)->wpos); + tx_fiber_init((*msg)->connection->session, (*msg)->header.sync); + return 0; } /** @@ -1507,7 +1564,8 @@ tx_reply_error(struct iproto_msg *msg) static void tx_reply_iproto_error(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; + tx_accept_msg(m, &msg); struct obuf *out = msg->connection->tx.p_obuf; iproto_reply_error(out, diag_last_error(&msg->diag), msg->header.sync, ::schema_version); @@ -1527,7 +1585,9 @@ tx_inject_delay(void) static void tx_process1(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; + if (tx_accept_msg(m, &msg) != 0) + goto error; if (tx_check_schema(msg->header.schema_version)) goto error; @@ -1553,17 +1613,20 @@ error: static void tx_process_select(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; + struct request *req; struct obuf *out; struct obuf_svp svp; struct port port; int count; int rc; - struct request *req = &msg->dml; + if (tx_accept_msg(m, &msg) != 0) + goto error; if (tx_check_schema(msg->header.schema_version)) goto error; tx_inject_delay(); + req = &msg->dml; rc = box_select(req->space_id, req->index_id, req->iterator, req->offset, req->limit, req->key, req->key_end, &port); @@ -1607,7 +1670,9 @@ tx_process_call_on_yield(struct trigger *trigger, void *event) static void tx_process_call(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; + if (tx_accept_msg(m, &msg) != 0) + goto error; if (tx_check_schema(msg->header.schema_version)) goto error; @@ -1686,13 +1751,15 @@ error: static void tx_process_misc(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); - struct iproto_connection *con = msg->connection; - struct obuf *out = con->tx.p_obuf; + struct iproto_msg *msg; + if (tx_accept_msg(m, &msg) != 0) + goto error; if (tx_check_schema(msg->header.schema_version)) goto error; try { + struct iproto_connection *con = msg->connection; + struct obuf *out = con->tx.p_obuf; struct ballot ballot; switch (msg->header.type) { case IPROTO_AUTH: @@ -1729,7 +1796,7 @@ error: static void tx_process_sql(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; struct obuf *out; struct port port; struct sql_bind *bind = NULL; @@ -1738,6 +1805,8 @@ tx_process_sql(struct cmsg *m) uint32_t len; bool is_unprepare = false; + if (tx_accept_msg(m, &msg) != 0) + goto error; if (tx_check_schema(msg->header.schema_version)) goto error; assert(msg->header.type == IPROTO_EXECUTE || @@ -1825,7 +1894,11 @@ error: static void tx_process_replication(struct cmsg *m) { - struct iproto_msg *msg = tx_accept_msg(m); + struct iproto_msg *msg; + if (tx_accept_msg(m, &msg) != 0) { + tx_reply_error(msg); + return; + } struct iproto_connection *con = msg->connection; struct ev_io io; coio_create(&io, con->input.fd); @@ -1865,6 +1938,29 @@ tx_process_replication(struct cmsg *m) } } +static int +net_check_connection(struct iproto_msg *msg) +{ + if (!msg->close_connection) + return 0; + + struct iproto_connection *con = msg->connection; + struct obuf *out = msg->wpos.obuf; + int64_t nwr = sio_writev(con->output.fd, out->iov, + obuf_iovcnt(out)); + + if (nwr > 0) { + /* Count statistics. */ + rmean_collect(rmean_net, IPROTO_SENT, nwr); + } else if (nwr < 0 && ! sio_wouldblock(errno)) { + diag_log(); + } + assert(iproto_connection_is_idle(con)); + iproto_connection_close(con); + iproto_msg_delete(msg); + return -1; +} + static void net_send_msg(struct cmsg *m) { @@ -1881,6 +1977,9 @@ net_send_msg(struct cmsg *m) } con->wend = msg->wpos; + if (net_check_connection(msg) != 0) + return; + if (evio_has_fd(&con->output)) { if (! ev_is_active(&con->output)) ev_feed_event(con->loop, &con->output, EV_WRITE); @@ -1910,6 +2009,10 @@ net_end_join(struct cmsg *m) struct iproto_connection *con = msg->connection; msg->p_ibuf->rpos += msg->len; + + if (net_check_connection(msg) != 0) + return; + iproto_msg_delete(msg); assert(! ev_is_active(&con->input)); @@ -1928,6 +2031,10 @@ net_end_subscribe(struct cmsg *m) struct iproto_connection *con = msg->connection; msg->p_ibuf->rpos += msg->len; + + if (net_check_connection(msg) != 0) + return; + iproto_msg_delete(msg); assert(! ev_is_active(&con->input)); @@ -1936,81 +2043,49 @@ net_end_subscribe(struct cmsg *m) } /** - * Handshake a connection: invoke the on-connect trigger - * and possibly authenticate. Try to send the client an error - * upon a failure. + * Handshake a connection: prepare greeting for it. */ static void -tx_process_connect(struct cmsg *m) +iproto_process_connect(struct iproto_msg *msg) { - struct iproto_msg *msg = (struct iproto_msg *) m; struct iproto_connection *con = msg->connection; struct obuf *out = msg->connection->tx.p_obuf; - try { /* connect. */ - con->session = session_create(SESSION_TYPE_BINARY); - if (con->session == NULL) - diag_raise(); - con->session->meta.connection = con; - tx_fiber_init(con->session, 0); - char *greeting = (char *) static_alloc(IPROTO_GREETING_SIZE); - /* TODO: dirty read from tx thread */ - struct tt_uuid uuid = INSTANCE_UUID; - random_bytes(con->salt, IPROTO_SALT_SIZE); - greeting_encode(greeting, tarantool_version_id(), &uuid, - con->salt, IPROTO_SALT_SIZE); - obuf_dup_xc(out, greeting, IPROTO_GREETING_SIZE); - if (! rlist_empty(&session_on_connect)) { - if (session_run_on_connect_triggers(con->session) != 0) - diag_raise(); - } - iproto_wpos_create(&msg->wpos, out); - } catch (Exception *e) { - tx_reply_error(msg); - msg->close_connection = true; - } -} - -/** - * Send a response to connect to the client or close the - * connection in case on_connect trigger failed. - */ -static void -net_send_greeting(struct cmsg *m) -{ - struct iproto_msg *msg = (struct iproto_msg *) m; - struct iproto_connection *con = msg->connection; - if (msg->close_connection) { - struct obuf *out = msg->wpos.obuf; - int64_t nwr = sio_writev(con->output.fd, out->iov, - obuf_iovcnt(out)); - - if (nwr > 0) { - /* Count statistics. */ - rmean_collect(rmean_net, IPROTO_SENT, nwr); - } else if (nwr < 0 && ! sio_wouldblock(errno)) { - diag_log(); - } - assert(iproto_connection_is_idle(con)); - iproto_connection_close(con); - iproto_msg_delete(msg); - return; - } + char *greeting = (char *) static_alloc(IPROTO_GREETING_SIZE); + struct tt_uuid uuid = INSTANCE_UUID; + random_bytes(con->salt, IPROTO_SALT_SIZE); + greeting_encode(greeting, tarantool_version_id(), &uuid, + con->salt, IPROTO_SALT_SIZE); + obuf_dup_xc(out, greeting, IPROTO_GREETING_SIZE); + iproto_wpos_create(&msg->wpos, out); con->wend = msg->wpos; - /* - * Connect is synchronous, so no one could have been - * messing up with the connection while it was in - * progress. - */ assert(evio_has_fd(&con->output)); - /* Handshake OK, start reading input. */ ev_feed_event(con->loop, &con->output, EV_WRITE); iproto_msg_delete(msg); } -static const struct cmsg_hop connect_route[] = { - { tx_process_connect, &net_pipe }, - { net_send_greeting, NULL }, -}; +static int +tx_init_connect(struct iproto_msg *msg) +{ + struct iproto_connection *con = msg->connection; + obuf_create(&con->obuf[0], &net_slabc, iproto_readahead); + obuf_create(&con->obuf[1], &net_slabc, iproto_readahead); + con->tx.p_obuf = &con->obuf[0]; + iproto_wpos_create(&con->wpos, con->tx.p_obuf); + iproto_wpos_create(&con->wend, con->tx.p_obuf); + + con->session = session_create(SESSION_TYPE_BINARY); + if (con->session == NULL) + return -1; + con->session->meta.connection = con; + + tx_fiber_init(con->session, 0); + if (! rlist_empty(&session_on_connect)) { + if (session_run_on_connect_triggers(con->session) != 0) + return -1; + } + + return 0; +} /** }}} */ @@ -2037,11 +2112,10 @@ iproto_on_accept(struct evio_service * /* service */, int fd, mempool_free(&iproto_connection_pool, con); return -1; } - cmsg_init(&msg->base, connect_route); msg->p_ibuf = con->p_ibuf; msg->wpos = con->wpos; msg->close_connection = false; - cpipe_push(&tx_pipe, &msg->base); + iproto_process_connect(msg); return 0; } @@ -2054,6 +2128,8 @@ static struct evio_service binary; /* iproto binary listener */ static int net_cord_f(va_list /* ap */) { + slab_cache_create(&iproto_slabc, &runtime); + mempool_create(&iproto_msg_pool, &cord()->slabc, sizeof(struct iproto_msg)); mempool_create(&iproto_connection_pool, &cord()->slabc, @@ -2297,7 +2373,8 @@ iproto_listen(const char *uri) size_t iproto_mem_used(void) { - return slab_cache_used(&net_cord.slabc) + slab_cache_used(&net_slabc); + return slab_cache_used(&net_cord.slabc) + + slab_cache_used(&net_slabc) + slab_cache_used(&iproto_slabc); } size_t diff --git a/test/box-py/bad_trigger.result b/test/box-py/bad_trigger.result index 5d064b7648..bfa9c2b759 100644 --- a/test/box-py/bad_trigger.result +++ b/test/box-py/bad_trigger.result @@ -14,6 +14,7 @@ type(box.session.on_connect(f1)) - function ... greeting: True +Nothing to read yet: Resource temporarily unavailable fixheader: True error code 32 error message: [string "function f1() nosuchfunction() end"]:1: attempt to call global 'nosuchfunction' (a nil value) diff --git a/test/box-py/bad_trigger.test.py b/test/box-py/bad_trigger.test.py index 7d200b9218..7f45f5e713 100644 --- a/test/box-py/bad_trigger.test.py +++ b/test/box-py/bad_trigger.test.py @@ -2,7 +2,7 @@ from lib.box_connection import BoxConnection from lib.tarantool_connection import TarantoolConnection from tarantool import NetworkError from tarantool.const import IPROTO_GREETING_SIZE, IPROTO_CODE, IPROTO_ERROR, \ - REQUEST_TYPE_ERROR + REQUEST_TYPE_ERROR, REQUEST_TYPE_PING import socket import msgpack @@ -26,9 +26,25 @@ s = conn.socket # Read greeting print 'greeting: ', len(s.recv(IPROTO_GREETING_SIZE)) == IPROTO_GREETING_SIZE -# Read error packet +# Check soscket IPROTO_FIXHEADER_SIZE = 5 -fixheader = s.recv(IPROTO_FIXHEADER_SIZE) +s.setblocking(False) +fixheader = None +try: + fixheader = s.recv(IPROTO_FIXHEADER_SIZE) +except socket.error as err: + print 'Nothing to read yet:', str(err).split(']')[1] +else: + print 'Received fixheader' +s.setblocking(True) + +# Send ping +query = msgpack.dumps({ IPROTO_CODE : REQUEST_TYPE_PING }) +s.send(msgpack.dumps(len(query)) + query) + +# Read error packet +if not fixheader: + fixheader = s.recv(IPROTO_FIXHEADER_SIZE) print 'fixheader: ', len(fixheader) == IPROTO_FIXHEADER_SIZE unpacker.feed(fixheader) packet_len = unpacker.unpack() -- 2.17.1 From avtikhon at tarantool.org Fri Aug 14 15:42:10 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 14 Aug 2020 15:42:10 +0300 Subject: [Tarantool-patches] [PATCH v2 2/3] Fix test box-tap/cfg.test.lua on openSuSE In-Reply-To: <20200814024442.4lgthzt576txghch@tkn_work_nb> References: <99ba534fc775257849ba5b3d9b6bdd5711c817cb.1594126705.git.avtikhon@tarantool.org> <20200814024442.4lgthzt576txghch@tkn_work_nb> Message-ID: <20200814124210.GA4609@hpalx> Hi Alexander, thanks a lot for a deep investigation. I've used your patch and rewrote the commit message. Also I've made there all your suggestions. On Fri, Aug 14, 2020 at 05:44:42AM +0300, Alexander Turenko wrote: > > Fix test box-tap/cfg.test.lua on openSuSE > > > > Found that test fails on its subtest checking: > > "gh-2872 bootstrap is aborted if vinyl_dir > > contains vylog files left from previous runs" > > > > Debugging src/box/xlog.c found that all checks > > were correct, but at function: > > src/box/vy_log.c:vy_log_bootstrap() > > the check on of the errno on ENOENT blocked the > > negative return from function: > > src/box/xlog.c:xdir_scan() > > > > Found that errno was already set to ENOENT before > > the xdir_scan() call. To fix the issue the errno > > must be clean before the call to xdir_scan, because > > we are interesting in it only from this function. > > The same issue fixed at function: > > src/box/vy_log.c:vy_log_begin_recovery() > > > > Closes #4594 > > Needed for #4562 > > My wish is to find the following information in the commit message: > > - The header: the subsystem prefix (vinyl) and what is fixed (vinyl_dir > existence check at bootstrap). > - What was wrong and becomes right (high-level or from a user > perspective): when memtx_dir is not exists, but vinyl_dir exists and > errno is set to ENOENT, box configuration succeeds, however it is > should not. > - What is the reason of the wrong behaviour: not all failure paths in > xdir_scan() are set errno, but the caller assumes it. > - From where the problem appears: openSUSE and box-tap/cfg.test.lua. > > > --- > > src/box/vy_log.c | 2 ++ > > 1 file changed, 2 insertions(+) > > > > > diff --git a/src/box/vy_log.c b/src/box/vy_log.c > > index 311985c72..86599fd15 100644 > > --- a/src/box/vy_log.c > > +++ b/src/box/vy_log.c > > @@ -1014,6 +1014,7 @@ vy_log_rebootstrap(void) > > int > > vy_log_bootstrap(void) > > { > > + errno = 0; > > if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > > return -1; > > You spotted the problem right. > > A bit more context: > > Usual C convention is to report success or failure using a return > value and set errno at any error. So a caller usually just checks a > return value and if it means a failure (usually -1), it checks errno > to determine an exact reason. > > Usual convention in tarantool is a bit different: we use a special > diagnostics area to report a reason of a failure. > > Not all failure paths of xdir_scan() sets errno (including our > 'invalid instance UUID' case), so we cannot be sure that errno is > not remains unchanged after a failure of the function. > > So, again, your fix is correct. > > However the approach with checking errno against ENOENT (No such file or > directory) is not good (with or without your patch). For example: > > - What if xdir_scan() would be changed in future and, say, some call > will rewrite errno after the opendir() call? > - What if some other call inside xdir_scan() will set ENOENT: say, > open() in xdir_open_cursor() due to some race? > > We lean on implementation details of the callee, not its contract. I > think this way is too fragile and we should either: > > - check whether the directory exists before xdir_scan() call > - or pass a flag to xdir_scan() whether the directory should exist. > > The latter looks better for me: it does not lead to code duplication. > See the proposed diff below the email. > > BTW, there is the simple way to test the problem on any OS: > > | diff --git a/test/box-tap/cfg.test.lua b/test/box-tap/cfg.test.lua > | index 569b5f463..2eda2f3c2 100755 > | --- a/test/box-tap/cfg.test.lua > | +++ b/test/box-tap/cfg.test.lua > | @@ -392,6 +392,8 @@ s:create_index('pk') > | os.exit(0) > | ]], vinyl_dir)) > | code = string.format([[ > | +local errno = require('errno') > | +errno(errno.ENOENT) > | box.cfg{vinyl_dir = '%s'} > | os.exit(0) > | ]], vinyl_dir) > > Of course, we should not add it right to this test case, let's copy it > somewhere and add separately. > > ---- > > diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c > index dfd6fce6e..9f079a6b5 100644 > --- a/src/box/memtx_engine.c > +++ b/src/box/memtx_engine.c > @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, > &xlog_opts_default); > memtx->snap_dir.force_recovery = force_recovery; > > - if (xdir_scan(&memtx->snap_dir) != 0) > + if (xdir_scan(&memtx->snap_dir, true) != 0) > goto fail; > > /* > diff --git a/src/box/recovery.cc b/src/box/recovery.cc > index d1a503cfc..cd33e7635 100644 > --- a/src/box/recovery.cc > +++ b/src/box/recovery.cc > @@ -121,7 +121,7 @@ void > recovery_scan(struct recovery *r, struct vclock *end_vclock, > struct vclock *gc_vclock) > { > - xdir_scan_xc(&r->wal_dir); > + xdir_scan_xc(&r->wal_dir, true); > > if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || > vclock_compare(end_vclock, &r->vclock) < 0) { > @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, > struct vclock *clock; > > if (scan_dir) > - xdir_scan_xc(&r->wal_dir); > + xdir_scan_xc(&r->wal_dir, true); > > if (xlog_cursor_is_open(&r->cursor)) { > /* If there's a WAL open, recover from it first. */ > diff --git a/src/box/vy_log.c b/src/box/vy_log.c > index 311985c72..da3c50e87 100644 > --- a/src/box/vy_log.c > +++ b/src/box/vy_log.c > @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) > int > vy_log_bootstrap(void) > { > - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > + if (xdir_scan(&vy_log.dir, false) < 0) > return -1; > if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) > return vy_log_rebootstrap(); > @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) > * because vinyl might not be even in use. Complain only > * on an attempt to write a vylog. > */ > - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > + if (xdir_scan(&vy_log.dir, false) < 0) > return NULL; > > if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { > diff --git a/src/box/wal.c b/src/box/wal.c > index d8c92aa36..2b894d680 100644 > --- a/src/box/wal.c > +++ b/src/box/wal.c > @@ -559,7 +559,7 @@ wal_enable(void) > * existing WAL files. Required for garbage collection, > * see wal_collect_garbage(). > */ > - if (xdir_scan(&writer->wal_dir)) > + if (xdir_scan(&writer->wal_dir, true)) > return -1; > > /* Open the most recent WAL file. */ > diff --git a/src/box/xlog.c b/src/box/xlog.c > index 6ccd3d68d..74f761994 100644 > --- a/src/box/xlog.c > +++ b/src/box/xlog.c > @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > * @return nothing. > */ > int > -xdir_scan(struct xdir *dir) > +xdir_scan(struct xdir *dir, bool is_dir_required) > { > DIR *dh = opendir(dir->dirname); /* log dir */ > int64_t *signatures = NULL; /* log file names */ > size_t s_count = 0, s_capacity = 0; > > if (dh == NULL) { > + if (!is_dir_required && errno == ENOENT) > + return 0; > diag_set(SystemError, "error reading directory '%s'", > dir->dirname); > return -1; > diff --git a/src/box/xlog.h b/src/box/xlog.h > index 9ffce598b..3400eb75f 100644 > --- a/src/box/xlog.h > +++ b/src/box/xlog.h > @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); > * snapshot or scan through all logs. > */ > int > -xdir_scan(struct xdir *dir); > +xdir_scan(struct xdir *dir, bool is_dir_required); > > /** > * Check that a directory exists and is writable. > @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > #include "exception.h" > > static inline void > -xdir_scan_xc(struct xdir *dir) > +xdir_scan_xc(struct xdir *dir, bool is_dir_required) > { > - if (xdir_scan(dir) == -1) > + if (xdir_scan(dir, is_dir_required) == -1) > diag_raise(); > } > From i.kosarev at tarantool.org Fri Aug 14 17:51:43 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Fri, 14 Aug 2020 17:51:43 +0300 Subject: [Tarantool-patches] [PATCH v3] tuple: drop extra restrictions for multikey index Message-ID: <20200814145143.23720-1-i.kosarev@tarantool.org> Multikey index did not work properly with nullable root field in tuple_raw_multikey_count(). Now it is fixed and corresponding restrictions are dropped. This also means that we can drop implicit nullability update for array/map fields and make all fields nullable by default, as it was until e1d3fe8ab8eed65394ad17409401a93b6fcdc435 (tuple format: don't allow null where array/map is expected), as far as default non-nullability itself doesn't solve any real problems while providing confusing behavior (gh-5027). Follow-up #5027 Closes #5192 @TarantoolBot document Title: tuple: allow nullable multikey index root Update the documentation for space_object:create_index() to reflect that it is now safe for multikey index root to be nullable. Previously it was not safe to insert, for example, box.NULL as an array field which elements are the part of multikey index due to error in counter of multikey index keys in tuple. It was partly fixed using default non-nullability for tuple fields. In 2.3.3 the restriction on nullable multikey index root was introduced. Now the counter problem is fixed and all tuple fields are nullable by default as before 2.2.1. --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5192-fix-multikey-index-restrictions Issue: https://github.com/tarantool/tarantool/issues/5192 @ChangeLog: * Dropped restrictions on nullable multikey index root. They were introduced due to inaccuracy in multikey index realization. It is now fixed. Also all fields are now nullable by default as it was before 2.2.1 (gh-5192). Changes in v2: - removed insignificant changes - fixed tests comments Changes in v3: - fixed some nits - added docbot request src/box/memtx_space.c | 8 --- src/box/tuple.c | 5 +- src/box/tuple_format.c | 20 ------ src/box/vinyl.c | 8 --- test/engine/gh-5027-fields-nullability.result | 63 ++----------------- .../gh-5027-fields-nullability.test.lua | 27 ++------ .../gh-5192-multikey-root-nullability.result | 59 +++++++++++++++++ ...gh-5192-multikey-root-nullability.test.lua | 19 ++++++ test/engine/json.result | 20 +++--- test/engine/json.test.lua | 6 +- test/engine/multikey.result | 8 +-- test/engine/multikey.test.lua | 4 +- 12 files changed, 111 insertions(+), 136 deletions(-) create mode 100644 test/engine/gh-5192-multikey-root-nullability.result create mode 100644 test/engine/gh-5192-multikey-root-nullability.test.lua diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index 8452ab4303..d30ce44b82 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -604,14 +604,6 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) return -1; } } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } switch (index_def->type) { case HASH: if (! index_def->opts.is_unique) { diff --git a/src/box/tuple.c b/src/box/tuple.c index e77753a5ba..f3965476ea 100644 --- a/src/box/tuple.c +++ b/src/box/tuple.c @@ -555,7 +555,10 @@ tuple_raw_multikey_count(struct tuple_format *format, const char *data, NULL, MULTIKEY_NONE); if (array_raw == NULL) return 0; - assert(mp_typeof(*array_raw) == MP_ARRAY); + enum mp_type type = mp_typeof(*array_raw); + if (type == MP_NIL) + return 0; + assert(type == MP_ARRAY); return mp_decode_array(&array_raw); } diff --git a/src/box/tuple_format.c b/src/box/tuple_format.c index bae6c67cda..9b817d3cf7 100644 --- a/src/box/tuple_format.c +++ b/src/box/tuple_format.c @@ -242,24 +242,6 @@ tuple_field_ensure_child_compatibility(struct tuple_field *parent, return 0; } -/** - * Tuple fields are nullable by default. However, it is not ok - * for array/map fields, as far as their implicit nullability - * might break field accessors expectations, provide confusing - * error messages and cause incorrect behaviour of - * tuple_multikey_count(). Thus array/map fields have to be - * non-nullable by default, which means we have to update default - * nullability for them. - */ -static void -tuple_field_update_nullability(struct tuple_field *field) -{ - if ((field->type == FIELD_TYPE_ARRAY || - field->type == FIELD_TYPE_MAP) && - tuple_field_is_nullable(field)) - field->nullable_action = ON_CONFLICT_ACTION_DEFAULT; -} - /** * Given a field number and a path, add the corresponding field * to the tuple format, allocating intermediate fields if @@ -335,13 +317,11 @@ tuple_format_add_field(struct tuple_format *format, uint32_t fieldno, parent->offset_slot = *current_slot; } } - tuple_field_update_nullability(parent); parent->is_key_part = true; next->is_multikey_part = is_multikey; parent = next; token_count++; } - tuple_field_update_nullability(parent); /* * The path has already been verified by the * key_def_decode_parts function. diff --git a/src/box/vinyl.c b/src/box/vinyl.c index 32301d7bab..5fa3ea3a54 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -658,14 +658,6 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) diag_set(ClientError, ER_NULLABLE_PRIMARY, space_name(space)); return -1; } - if (key_def->is_multikey && - key_def->multikey_fieldno < space->def->field_count && - space->def->fields[key_def->multikey_fieldno].is_nullable) { - diag_set(ClientError, ER_UNSUPPORTED, - "multikey index", - "nullable root field"); - return -1; - } /* Check that there are no ANY, ARRAY, MAP parts */ for (uint32_t i = 0; i < key_def->part_count; i++) { struct key_part *part = &key_def->parts[i]; diff --git a/test/engine/gh-5027-fields-nullability.result b/test/engine/gh-5027-fields-nullability.result index 1121727f6f..1e2709a24e 100644 --- a/test/engine/gh-5027-fields-nullability.result +++ b/test/engine/gh-5027-fields-nullability.result @@ -41,79 +41,26 @@ _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) | --- | ... -s:replace{1} +s:replace{1} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL} +s:replace{1, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL} +s:replace{1, box.NULL, box.NULL, box.NULL} -- error | --- | - error: Tuple field 5 required by space format is missing | ... -s:replace{1, box.NULL, box.NULL, box.NULL, 5} +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok | --- | - [1, null, null, null, 5] | ... s:drop() | --- | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - [1, null] - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | - error: multikey index does not support nullable root field - | ... -s:replace{2, box.NULL} - | --- - | - [2, null] - | ... -s:drop() - | --- - | ... - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) - | --- - | ... -_ = s:format({{name='id'}, {name='data', type='array'}}) - | --- - | ... -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) - | --- - | ... -s:replace{1, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) - | --- - | ... -s:replace{2, box.NULL} - | --- - | - error: 'Tuple field 2 type does not match one required by operation: expected array' - | ... -s:replace{3, {}} - | --- - | - [3, []] - | ... -s:drop() - | --- - | ... diff --git a/test/engine/gh-5027-fields-nullability.test.lua b/test/engine/gh-5027-fields-nullability.test.lua index 960103d6c4..8553d488c0 100644 --- a/test/engine/gh-5027-fields-nullability.test.lua +++ b/test/engine/gh-5027-fields-nullability.test.lua @@ -12,26 +12,9 @@ s:drop() s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) _ = s:create_index('i1', {parts={{1, 'unsigned'}}}) _ = s:create_index('i2', {parts={{5, 'unsigned', is_nullable=false}}}) -s:replace{1} -s:replace{1, box.NULL} -s:replace{1, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL} -s:replace{1, box.NULL, box.NULL, box.NULL, 5} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:drop() - -s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) -_ = s:format({{name='id'}, {name='data', type='array'}}) -_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) -s:replace{1, box.NULL} -_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) -s:replace{2, box.NULL} -s:replace{3, {}} +s:replace{1} -- error +s:replace{1, box.NULL} -- error +s:replace{1, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL} -- error +s:replace{1, box.NULL, box.NULL, box.NULL, 5} -- ok s:drop() diff --git a/test/engine/gh-5192-multikey-root-nullability.result b/test/engine/gh-5192-multikey-root-nullability.result new file mode 100644 index 0000000000..2a9cd66ed6 --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.result @@ -0,0 +1,59 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- ok + | --- + | - [1, null] + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- ok + | --- + | - [2, null] + | ... +_ = s:delete(2) + | --- + | ... +s:drop() + | --- + | ... + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) + | --- + | ... +_ = s:format({{name='id'}, {name='data', type='array'}}) + | --- + | ... +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) + | --- + | ... +s:replace{1, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) + | --- + | ... +s:replace{2, box.NULL} -- error + | --- + | - error: 'Tuple field 2 type does not match one required by operation: expected array' + | ... +s:replace{3, {}} -- ok + | --- + | - [3, []] + | ... +s:drop() + | --- + | ... diff --git a/test/engine/gh-5192-multikey-root-nullability.test.lua b/test/engine/gh-5192-multikey-root-nullability.test.lua new file mode 100644 index 0000000000..6eb0eaf059 --- /dev/null +++ b/test/engine/gh-5192-multikey-root-nullability.test.lua @@ -0,0 +1,19 @@ +test_run = require('test_run').new() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array', is_nullable=true}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- ok +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- ok +_ = s:delete(2) +s:drop() + +s = box.schema.space.create('gh-5027', {engine=test_run:get_cfg('engine')}) +_ = s:format({{name='id'}, {name='data', type='array'}}) +_ = s:create_index('i1', {parts={{1, 'unsigned'}}}) +s:replace{1, box.NULL} -- error +_ = s:create_index('i2', {parts={{field=2, path='[*].key', type='string'}}}) +s:replace{2, box.NULL} -- error +s:replace{3, {}} -- ok +s:drop() diff --git a/test/engine/json.result b/test/engine/json.result index 2175266fcc..b8fd9a1b69 100644 --- a/test/engine/json.result +++ b/test/engine/json.result @@ -738,12 +738,11 @@ _ = s:create_index('sk', {parts = {{'[2][1].a', 'unsigned'}}}) ... s:insert{1, box.NULL} -- error --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{2, {box.NULL}} -- error --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- error: Tuple field [2][1]["a"] required by space format is missing ... s:insert{3} -- error --- @@ -764,16 +763,15 @@ s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} --- ... -s:insert{7, box.NULL} -- error +s:insert{7, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [7, null] ... -s:insert{8, {box.NULL}} -- error +s:insert{8, {box.NULL}} -- ok --- -- error: 'Tuple field [2][1] type does not match one required by operation: expected - map' +- [8, [null]] ... --- Skipping nullable fields is okay though. +-- Skipping nullable fields is also okay. s:insert{9} -- ok --- - [9] @@ -792,7 +790,9 @@ s:insert{12, {{a = box.NULL}}} -- ok ... s.index.sk:select() --- -- - [9] +- - [7, null] + - [8, [null]] + - [9] - [10, []] - [11, [{'b': 1}]] - [12, [{'a': null}]] diff --git a/test/engine/json.test.lua b/test/engine/json.test.lua index 4771c3162a..371bbad91d 100644 --- a/test/engine/json.test.lua +++ b/test/engine/json.test.lua @@ -220,9 +220,9 @@ s:insert{4, {}} -- error s:insert{5, {{b = 1}}} -- error s:insert{6, {{a = 1}}} -- ok s.index.sk:alter{parts = {{'[2][1].a', 'unsigned', is_nullable = true}}} -s:insert{7, box.NULL} -- error -s:insert{8, {box.NULL}} -- error --- Skipping nullable fields is okay though. +s:insert{7, box.NULL} -- ok +s:insert{8, {box.NULL}} -- ok +-- Skipping nullable fields is also okay. s:insert{9} -- ok s:insert{10, {}} -- ok s:insert{11, {{b = 1}}} -- ok diff --git a/test/engine/multikey.result b/test/engine/multikey.result index 968be4cc3b..ff72983ecd 100644 --- a/test/engine/multikey.result +++ b/test/engine/multikey.result @@ -794,9 +794,9 @@ _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) --- ... -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [1, null] ... s:insert{2, {box.NULL}} -- error --- @@ -814,9 +814,9 @@ s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} --- ... -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok --- -- error: 'Tuple field 2 type does not match one required by operation: expected array' +- [5, null] ... s:insert{6, {box.NULL}} -- ok --- diff --git a/test/engine/multikey.test.lua b/test/engine/multikey.test.lua index ed70334949..653a9c9b17 100644 --- a/test/engine/multikey.test.lua +++ b/test/engine/multikey.test.lua @@ -214,12 +214,12 @@ s:drop() s = box.schema.space.create('test', {engine = engine}) _ = s:create_index('pk') _ = s:create_index('sk', {parts = {{'[2][*]', 'unsigned'}}}) -s:insert{1, box.NULL} -- error +s:insert{1, box.NULL} -- ok s:insert{2, {box.NULL}} -- error s:insert{3, {}} -- ok s:insert{4, {1}} -- ok s.index.sk:alter{parts = {{'[2][*]', 'unsigned', is_nullable = true}}} -s:insert{5, box.NULL} -- still error +s:insert{5, box.NULL} -- ok s:insert{6, {box.NULL}} -- ok s:insert{7, {}} -- ok s:insert{8, {2}} -- ok -- 2.17.1 From imeevma at tarantool.org Fri Aug 14 18:04:52 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:04:52 +0300 Subject: [Tarantool-patches] [PATCH v2 00/10] sql: properly check arguments types of built-in functions Message-ID: This patch-set makes SQL to use the ApplyType opcode to validate the argument types of built-in functions. https://github.com/tarantool/tarantool/issues/4159 https://github.com/tarantool/tarantool/tree/imeevma/gh-4159-rework-sql-builtins @ChangeLog - Built-in function argument types are now properly checked (gh-4159). Changes in v2: - Changed an approach to check types of STRING/VARBINARY arguments. - Added a new patch that changes signature of trim(). Mergen Imeev (10): sql: do not return UNSIGNED in built-in functions sql: fix functions return types sql: change signature of trim() box: add new options for functions sql: use has_vararg for built-in functions sql: add overloaded versions of the functions sql: move built-in function definitions in _func box: add param_list to 'struct func' sql: check built-in functions argument types sql: refactor sql/func.c src/box/alter.cc | 12 +- src/box/bootstrap.snap | Bin 5976 -> 6446 bytes src/box/func.c | 1 + src/box/func_def.c | 13 + src/box/func_def.h | 20 + src/box/lua/call.c | 2 + src/box/lua/upgrade.lua | 192 ++++ src/box/sql/analyze.c | 6 +- src/box/sql/expr.c | 33 +- src/box/sql/func.c | 879 +++--------------- src/box/sql/parse.y | 35 +- src/box/sql/resolve.c | 2 +- src/box/sql/select.c | 26 + src/box/sql/sqlInt.h | 19 +- src/box/sql/vdbeapi.c | 2 +- src/box/sql/vdbemem.c | 2 +- test/box-py/bootstrap.result | 6 +- test/box/access.result | 2 +- test/box/access.test.lua | 2 +- test/box/access_bin.result | 2 +- test/box/access_bin.test.lua | 2 +- test/box/access_sysview.result | 8 +- test/box/function1.result | 6 +- test/sql-tap/cse.test.lua | 24 +- test/sql-tap/func.test.lua | 46 +- test/sql-tap/orderby1.test.lua | 2 +- test/sql-tap/position.test.lua | 16 +- test/sql-tap/substr.test.lua | 2 +- test/sql/boolean.result | 32 +- test/sql/checks.result | 8 - test/sql/checks.test.lua | 2 - test/sql/collation.result | 8 + test/sql/collation.test.lua | 1 + test/sql/types.result | 1547 +++++++++++++++++++++++++++++++- test/sql/types.test.lua | 264 ++++++ test/wal_off/func_max.result | 8 +- 36 files changed, 2290 insertions(+), 942 deletions(-) -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:04:54 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:04:54 +0300 Subject: [Tarantool-patches] [PATCH v2 01/10] sql: do not return UNSIGNED in built-in functions In-Reply-To: References: Message-ID: <3a825a5a3d61933f5e76d6594f33ad304dd07559.1597417321.git.imeevma@gmail.com> This patch forces functions to return INTEGER instead of UNSIGNED. --- src/box/sql/vdbeapi.c | 2 +- test/sql/types.result | 12 ++++++++++++ test/sql/types.test.lua | 6 ++++++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c index 7c59ef83f..d1eeaf114 100644 --- a/src/box/sql/vdbeapi.c +++ b/src/box/sql/vdbeapi.c @@ -328,7 +328,7 @@ sql_result_double(sql_context * pCtx, double rVal) void sql_result_uint(sql_context *ctx, uint64_t u_val) { - mem_set_u64(ctx->pOut, u_val); + mem_set_int(ctx->pOut, u_val, false); } void diff --git a/test/sql/types.result b/test/sql/types.result index 442245186..95f7713e8 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2795,3 +2795,15 @@ box.execute([[DROP TABLE ts;]]) --- - row_count: 1 ... +-- +-- gh-4159: Make sure that functions returns values of type INTEGER +-- instead of values of type UNSIGNED. +-- +box.execute([[SELECT typeof(length('abc'));]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 0270d9f8a..fff0057bd 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -623,3 +623,9 @@ box.execute([[DROP TABLE tb;]]) box.execute([[DROP TABLE tt;]]) box.execute([[DROP TABLE tv;]]) box.execute([[DROP TABLE ts;]]) + +-- +-- gh-4159: Make sure that functions returns values of type INTEGER +-- instead of values of type UNSIGNED. +-- +box.execute([[SELECT typeof(length('abc'));]]) -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:04:55 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:04:55 +0300 Subject: [Tarantool-patches] [PATCH v2 02/10] sql: fix functions return types In-Reply-To: References: Message-ID: <804a579f9ba60138c0bf579da4c6a3c253d2c272.1597417321.git.imeevma@gmail.com> This patch fixes incorrect return types in SQL built-in function definitions. --- src/box/sql/func.c | 10 +++++----- test/sql/types.result | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 487cdafe1..affb285aa 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2466,7 +2466,7 @@ static struct { }, { .name = "IFNULL", .param_count = 2, - .returns = FIELD_TYPE_INTEGER, + .returns = FIELD_TYPE_SCALAR, .aggregate = FUNC_AGGREGATE_NONE, .is_deterministic = true, .flags = SQL_FUNC_COALESCE, @@ -2516,7 +2516,7 @@ static struct { }, { .name = "LIKE", .param_count = -1, - .returns = FIELD_TYPE_INTEGER, + .returns = FIELD_TYPE_BOOLEAN, .aggregate = FUNC_AGGREGATE_NONE, .is_deterministic = true, .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_LIKE, @@ -2526,7 +2526,7 @@ static struct { }, { .name = "LIKELIHOOD", .param_count = 2, - .returns = FIELD_TYPE_BOOLEAN, + .returns = FIELD_TYPE_SCALAR, .aggregate = FUNC_AGGREGATE_NONE, .is_deterministic = true, .flags = SQL_FUNC_UNLIKELY, @@ -2536,7 +2536,7 @@ static struct { }, { .name = "LIKELY", .param_count = 1, - .returns = FIELD_TYPE_BOOLEAN, + .returns = FIELD_TYPE_SCALAR, .aggregate = FUNC_AGGREGATE_NONE, .is_deterministic = true, .flags = SQL_FUNC_UNLIKELY, @@ -2686,7 +2686,7 @@ static struct { }, { .name = "ROUND", .param_count = -1, - .returns = FIELD_TYPE_INTEGER, + .returns = FIELD_TYPE_DOUBLE, .aggregate = FUNC_AGGREGATE_NONE, .is_deterministic = true, .flags = 0, diff --git a/test/sql/types.result b/test/sql/types.result index 95f7713e8..2498f3a48 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -231,7 +231,7 @@ box.execute("SELECT s LIKE NULL FROM t1;") --- - metadata: - name: COLUMN_1 - type: integer + type: boolean rows: - [null] ... @@ -257,7 +257,7 @@ box.execute("SELECT NULL LIKE s FROM t1;") --- - metadata: - name: COLUMN_1 - type: integer + type: boolean rows: - [null] ... -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:04:57 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:04:57 +0300 Subject: [Tarantool-patches] [PATCH v2 03/10] sql: change signature of trim() In-Reply-To: References: Message-ID: This patch changes the signature of the SQL built-in trim() function. This makes it easier to define a function in _func and fixes a bug where the function loses collation when the BOTH, LEADING, or TRAILING keywords are specified. --- src/box/sql/func.c | 56 +++++++++++++++++-------------------- src/box/sql/parse.y | 35 +++++++++++------------ test/sql/collation.result | 8 ++++++ test/sql/collation.test.lua | 1 + 4 files changed, 51 insertions(+), 49 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index affb285aa..e5da21191 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -1764,10 +1764,6 @@ trim_func_one_arg(struct sql_context *context, sql_value *arg) /** * Normalize args from @a argv input array when it has two args. * - * Case: TRIM( FROM ) - * If user has specified only, call trimming - * procedure with TRIM_BOTH as the flags and that trimming set. - * * Case: TRIM(LEADING/TRAILING/BOTH FROM ) * If user has specified side keyword only, then call trimming * procedure with the specified side and " " as the trimming set. @@ -1776,32 +1772,30 @@ static void trim_func_two_args(struct sql_context *context, sql_value *arg1, sql_value *arg2) { - const unsigned char *input_str, *trim_set; - if ((input_str = sql_value_text(arg2)) == NULL) - return; - - int input_str_sz = sql_value_bytes(arg2); - if (sql_value_type(arg1) == MP_INT || sql_value_type(arg1) == MP_UINT) { - uint8_t len_one = 1; - trim_procedure(context, sql_value_int(arg1), - (const unsigned char *) " ", &len_one, 1, - input_str, input_str_sz); - } else if ((trim_set = sql_value_text(arg1)) != NULL) { - int trim_set_sz = sql_value_bytes(arg1); - uint8_t *char_len; - int char_cnt = trim_prepare_char_len(context, trim_set, - trim_set_sz, &char_len); - if (char_cnt == -1) - return; - trim_procedure(context, TRIM_BOTH, trim_set, char_len, char_cnt, - input_str, input_str_sz); - sql_free(char_len); - } + assert(sql_value_type(arg2) == MP_UINT); + enum mp_type type = sql_value_type(arg1); + if (type == MP_NIL) + return sql_result_null(context); + const unsigned char *input_str = sql_value_text(arg1); + const unsigned char *trim_set; + + int input_str_sz = sql_value_bytes(arg1); + uint8_t len_one = 1; + if (type == MP_BIN) + trim_set = (const unsigned char *) "\0"; + else + trim_set = (const unsigned char *) " "; + trim_procedure(context, sql_value_int(arg2), trim_set, &len_one, 1, + input_str, input_str_sz); } /** * Normalize args from @a argv input array when it has three args. * + * Case: TRIM( FROM ) + * If user has specified only, call trimming + * procedure with TRIM_BOTH as the flags and that trimming set. + * * Case: TRIM(LEADING/TRAILING/BOTH FROM ) * If user has specified side keyword and , then * call trimming procedure with that args. @@ -1810,20 +1804,20 @@ static void trim_func_three_args(struct sql_context *context, sql_value *arg1, sql_value *arg2, sql_value *arg3) { - assert(sql_value_type(arg1) == MP_INT || sql_value_type(arg1) == MP_UINT); + assert(sql_value_type(arg2) == MP_UINT); const unsigned char *input_str, *trim_set; - if ((input_str = sql_value_text(arg3)) == NULL || - (trim_set = sql_value_text(arg2)) == NULL) + if ((input_str = sql_value_text(arg1)) == NULL || + (trim_set = sql_value_text(arg3)) == NULL) return; - int trim_set_sz = sql_value_bytes(arg2); - int input_str_sz = sql_value_bytes(arg3); + int trim_set_sz = sql_value_bytes(arg3); + int input_str_sz = sql_value_bytes(arg1); uint8_t *char_len; int char_cnt = trim_prepare_char_len(context, trim_set, trim_set_sz, &char_len); if (char_cnt == -1) return; - trim_procedure(context, sql_value_int(arg1), trim_set, char_len, + trim_procedure(context, sql_value_int(arg2), trim_set, char_len, char_cnt, input_str, input_str_sz); sql_free(char_len); } diff --git a/src/box/sql/parse.y b/src/box/sql/parse.y index 995875566..9c4bf491b 100644 --- a/src/box/sql/parse.y +++ b/src/box/sql/parse.y @@ -1123,32 +1123,31 @@ expr(A) ::= TRIM(X) LP trim_operands(Y) RP(E). { %type trim_operands {struct ExprList *} %destructor trim_operands {sql_expr_list_delete(pParse->db, $$);} -trim_operands(A) ::= trim_from_clause(F) expr(Y). { - A = sql_expr_list_append(pParse->db, F, Y.pExpr); +trim_operands(A) ::= trim_specification(N) expr(Z) FROM expr(Y). { + A = sql_expr_list_append(pParse->db, NULL, Y.pExpr); + struct Expr *p = sql_expr_new_dequoted(pParse->db, TK_INTEGER, + &sqlIntTokens[N]); + A = sql_expr_list_append(pParse->db, A, p); + A = sql_expr_list_append(pParse->db, A, Z.pExpr); } -trim_operands(A) ::= expr(Y). { +trim_operands(A) ::= trim_specification(N) FROM expr(Y). { A = sql_expr_list_append(pParse->db, NULL, Y.pExpr); + struct Expr *p = sql_expr_new_dequoted(pParse->db, TK_INTEGER, + &sqlIntTokens[N]); + A = sql_expr_list_append(pParse->db, A, p); } -%type trim_from_clause {struct ExprList *} -%destructor trim_from_clause {sql_expr_list_delete(pParse->db, $$);} - -/* - * The following two rules cover three cases of keyword - * (LEADING/TRAILING/BOTH) and combination. - * The case when both of them are absent is disallowed. - */ -trim_from_clause(A) ::= expr(Y) FROM. { +trim_operands(A) ::= expr(Z) FROM expr(Y). { A = sql_expr_list_append(pParse->db, NULL, Y.pExpr); + struct Expr *p = sql_expr_new_dequoted(pParse->db, TK_INTEGER, + &sqlIntTokens[TRIM_BOTH]); + A = sql_expr_list_append(pParse->db, A, p); + A = sql_expr_list_append(pParse->db, A, Z.pExpr); } -trim_from_clause(A) ::= trim_specification(N) expr_optional(Y) FROM. { - struct Expr *p = sql_expr_new_dequoted(pParse->db, TK_INTEGER, - &sqlIntTokens[N]); - A = sql_expr_list_append(pParse->db, NULL, p); - if (Y != NULL) - A = sql_expr_list_append(pParse->db, A, Y); +trim_operands(A) ::= expr(Y). { + A = sql_expr_list_append(pParse->db, NULL, Y.pExpr); } %type expr_optional {struct Expr *} diff --git a/test/sql/collation.result b/test/sql/collation.result index bfc89e1b8..23ff3d06e 100644 --- a/test/sql/collation.result +++ b/test/sql/collation.result @@ -1133,6 +1133,14 @@ box.execute("SELECT DISTINCT trim(s2) FROM jj;") rows: - ['A'] ... +box.execute("SELECT DISTINCT trim(BOTH FROM s2) FROM jj;") +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['A'] +... box.execute("INSERT INTO jj VALUES (3, 'aS'), (4, 'AS');") --- - row_count: 2 diff --git a/test/sql/collation.test.lua b/test/sql/collation.test.lua index 407fc19dc..8c2e8a133 100644 --- a/test/sql/collation.test.lua +++ b/test/sql/collation.test.lua @@ -308,6 +308,7 @@ box.execute("DROP TABLE qms4;") box.execute("CREATE TABLE jj (s1 INT PRIMARY KEY, s2 VARCHAR(3) COLLATE \"unicode_ci\");") box.execute("INSERT INTO jj VALUES (1,'A'), (2,'a')") box.execute("SELECT DISTINCT trim(s2) FROM jj;") +box.execute("SELECT DISTINCT trim(BOTH FROM s2) FROM jj;") box.execute("INSERT INTO jj VALUES (3, 'aS'), (4, 'AS');") box.execute("SELECT DISTINCT replace(s2, 'S', 's') FROM jj;") box.execute("SELECT DISTINCT substr(s2, 1, 1) FROM jj;") -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:04:59 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:04:59 +0300 Subject: [Tarantool-patches] [PATCH v2 04/10] box: add new options for functions In-Reply-To: References: Message-ID: <65c71dc224b0365954c396486c87738fb9ca5baf.1597417321.git.imeevma@gmail.com> The has_vararg option allows us to work with functions with a variable number of arguments. This is required for built-in SQL functions. Suppose this option is TRUE for a built-in SQL function. Then: 1) If param_list is empty, all arguments can be of any type. 2) If the length of param_list is not less than the number of the given arguments, the types of the given arguments must be compatible with the corresponding types described in param_list. 3) If the length of param_list is less than the number of given arguments, the rest of the arguments must be compatible with the last type in param_list. The is_overloaded option allows us to deal with functions that can take STRING or VARBINARY arguments. In case the first of these arguments is of type VARBINARY, we use the overloaded version of the function. By default, we use the STRING version of the function. The has_overload option indicates that the function has an overloaded version that accepts VARBINARY. See an explanation of the is_overloaded option. Part of #4159 --- src/box/func_def.c | 13 +++++++++++++ src/box/func_def.h | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/src/box/func_def.c b/src/box/func_def.c index 11d2bdb84..29929aea3 100644 --- a/src/box/func_def.c +++ b/src/box/func_def.c @@ -40,10 +40,17 @@ const char *func_aggregate_strs[] = {"none", "group"}; const struct func_opts func_opts_default = { /* .is_multikey = */ false, + /* .is_overloaded = */ false, + /* .has_overload = */ false, + /* .has_vararg = */ false, }; const struct opt_def func_opts_reg[] = { OPT_DEF("is_multikey", OPT_BOOL, struct func_opts, is_multikey), + OPT_DEF("is_overloaded", OPT_BOOL, struct func_opts, is_overloaded), + OPT_DEF("has_overload", OPT_BOOL, struct func_opts, has_overload), + OPT_DEF("has_vararg", OPT_BOOL, struct func_opts, has_vararg), + OPT_END, }; int @@ -51,6 +58,12 @@ func_opts_cmp(struct func_opts *o1, struct func_opts *o2) { if (o1->is_multikey != o2->is_multikey) return o1->is_multikey - o2->is_multikey; + if (o1->is_overloaded != o2->is_overloaded) + return o1->is_overloaded - o2->is_overloaded; + if (o1->has_overload != o2->has_overload) + return o1->has_overload - o2->has_overload; + if (o1->has_vararg != o2->has_vararg) + return o1->has_vararg - o2->has_vararg; return 0; } diff --git a/src/box/func_def.h b/src/box/func_def.h index d99d89190..bab2186da 100644 --- a/src/box/func_def.h +++ b/src/box/func_def.h @@ -68,6 +68,24 @@ struct func_opts { * packed in array. */ bool is_multikey; + /** + * True if this function is overloaded version of another function. + * + * Currently only used in built-in SQL functions. + */ + bool is_overloaded; + /** + * True if this function has overloaded version. + * + * Currently only used in built-in SQL functions. + */ + bool has_overload; + /** + * True if the function can have a variable number of arguments. + * + * Currently only used in built-in SQL functions. + */ + bool has_vararg; }; extern const struct func_opts func_opts_default; -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:00 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:00 +0300 Subject: [Tarantool-patches] [PATCH v2 05/10] sql: use has_vararg for built-in functions In-Reply-To: References: Message-ID: <4ab6e3c46bd2d5d597b33e55b650af91e040708b.1597417321.git.imeevma@gmail.com> After this patch, the has_varargs option will be used to determine that the function has a variable number of arguments. Part of #4159 --- src/box/sql/func.c | 3 ++- src/box/sql/resolve.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index e5da21191..1afee4924 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2182,7 +2182,7 @@ sql_func_by_signature(const char *name, int argc) if (base == NULL || !base->def->exports.sql) return NULL; - if (base->def->param_count != -1 && base->def->param_count != argc) + if (!base->def->opts.has_vararg && base->def->param_count != argc) return NULL; return base; } @@ -2928,6 +2928,7 @@ func_sql_builtin_new(struct func_def *def) def->returns = sql_builtins[idx].returns; def->aggregate = sql_builtins[idx].aggregate; def->exports.sql = sql_builtins[idx].export_to_sql; + def->opts.has_vararg = sql_builtins[idx].param_count == -1; return &func->base; } diff --git a/src/box/sql/resolve.c b/src/box/sql/resolve.c index 6f625dc18..5238555c3 100644 --- a/src/box/sql/resolve.c +++ b/src/box/sql/resolve.c @@ -614,7 +614,7 @@ resolveExprStep(Walker * pWalker, Expr * pExpr) pNC->nErr++; return WRC_Abort; } - if (func->def->param_count != -1 && + if (!func->def->opts.has_vararg && func->def->param_count != n) { uint32_t argc = func->def->param_count; const char *err = tt_sprintf("%d", argc); -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:02 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:02 +0300 Subject: [Tarantool-patches] [PATCH v2 06/10] sql: add overloaded versions of the functions In-Reply-To: References: Message-ID: <34e3d319f2955e21f9fc1d358b778128cea9ed79.1597417321.git.imeevma@gmail.com> This patch adds overload function definitions for the length(), substr(), trim(), and position() functions. Part of #4159 --- src/box/sql/analyze.c | 6 ++--- src/box/sql/expr.c | 29 ++++++++++++++++++++--- src/box/sql/func.c | 54 ++++++++++++++++++++++++++++++++++++++++--- src/box/sql/sqlInt.h | 5 ++-- src/box/sql/vdbemem.c | 2 +- 5 files changed, 84 insertions(+), 12 deletions(-) diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c index f74f9b358..d304b266e 100644 --- a/src/box/sql/analyze.c +++ b/src/box/sql/analyze.c @@ -716,7 +716,7 @@ callStatGet(Vdbe * v, int regStat4, int iParam, int regOut) { assert(regOut != regStat4 && regOut != regStat4 + 1); sqlVdbeAddOp2(v, OP_Integer, iParam, regStat4 + 1); - struct func *func = sql_func_by_signature("_sql_stat_get", 2); + struct func *func = sql_func_by_signature("_sql_stat_get", false, 2); assert(func != NULL); sqlVdbeAddOp4(v, OP_BuiltinFunction0, 0, regStat4, regOut, (char *)func, P4_FUNC); @@ -856,7 +856,7 @@ vdbe_emit_analyze_space(struct Parse *parse, struct space *space) sqlVdbeAddOp2(v, OP_Integer, part_count, stat4_reg + 1); sqlVdbeAddOp2(v, OP_Integer, part_count, stat4_reg + 2); struct func *init_func = - sql_func_by_signature("_sql_stat_init", 3); + sql_func_by_signature("_sql_stat_init", false, 3); assert(init_func != NULL); sqlVdbeAddOp4(v, OP_BuiltinFunction0, 0, stat4_reg + 1, stat4_reg, (char *)init_func, P4_FUNC); @@ -957,7 +957,7 @@ vdbe_emit_analyze_space(struct Parse *parse, struct space *space) pk_part_count, key_reg); assert(chng_reg == (stat4_reg + 1)); struct func *push_func = - sql_func_by_signature("_sql_stat_push", 3); + sql_func_by_signature("_sql_stat_push", false, 3); assert(push_func != NULL); sqlVdbeAddOp4(v, OP_BuiltinFunction0, 1, stat4_reg, tmp_reg, (char *)push_func, P4_FUNC); diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index bc2182446..99ca91bba 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -328,8 +328,15 @@ sql_expr_coll(Parse *parse, Expr *p, bool *is_explicit_coll, uint32_t *coll_id, if (op == TK_FUNCTION) { uint32_t arg_count = p->x.pList == NULL ? 0 : p->x.pList->nExpr; + bool has_blob_arg = false; + if (arg_count > 0) { + enum field_type type = + sql_expr_type(p->x.pList->a[0].pExpr); + has_blob_arg = type == FIELD_TYPE_VARBINARY; + } struct func *func = - sql_func_by_signature(p->u.zToken, arg_count); + sql_func_by_signature(p->u.zToken, has_blob_arg, + arg_count); if (func == NULL) break; if (sql_func_flag_is_set(func, SQL_FUNC_DERIVEDCOLL) && @@ -3975,8 +3982,15 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) } nFarg = pFarg ? pFarg->nExpr : 0; assert(!ExprHasProperty(pExpr, EP_IntValue)); + bool has_blob_arg = false; + if (nFarg > 0) { + enum field_type type = + sql_expr_type(pExpr->x.pList->a[0].pExpr); + has_blob_arg = type == FIELD_TYPE_VARBINARY; + } zId = pExpr->u.zToken; - struct func *func = sql_func_by_signature(zId, nFarg); + struct func *func = + sql_func_by_signature(zId, has_blob_arg, nFarg); if (func == NULL) { diag_set(ClientError, ER_NO_SUCH_FUNCTION, zId); @@ -5448,9 +5462,18 @@ analyzeAggregate(Walker * pWalker, Expr * pExpr) uint32_t argc = pExpr->x.pList != NULL ? pExpr->x.pList->nExpr : 0; + bool has_blob_arg = false; + if (argc > 0) { + enum field_type type = + sql_expr_type(pExpr->x.pList->a[0].pExpr); + has_blob_arg = + type == FIELD_TYPE_VARBINARY; + } pItem->func = sql_func_by_signature( - name, argc); + name, + has_blob_arg, + argc); assert(pItem->func != NULL); assert(pItem->func->def-> language == diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 1afee4924..ae1842824 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2169,19 +2169,27 @@ sql_is_like_func(struct Expr *expr) expr->x.pList->nExpr != 2) return 0; assert(!ExprHasProperty(expr, EP_xIsSelect)); - struct func *func = sql_func_by_signature(expr->u.zToken, 2); + struct func *func = sql_func_by_signature(expr->u.zToken, false, 2); if (func == NULL || !sql_func_flag_is_set(func, SQL_FUNC_LIKE)) return 0; return 1; } struct func * -sql_func_by_signature(const char *name, int argc) +sql_func_by_signature(const char *name, bool has_blob_arg, int argc) { struct func *base = func_by_name(name, strlen(name)); - if (base == NULL || !base->def->exports.sql) + if (base == NULL || !base->def->exports.sql || + base->def->opts.is_overloaded) return NULL; + if (has_blob_arg && base->def->opts.has_overload) { + const char *overload_name = tt_sprintf("%s_VARBINARY", name); + base = func_by_name(overload_name, strlen(overload_name)); + assert(base != NULL && base->def->exports.sql); + assert(base->def->opts.is_overloaded); + } + if (!base->def->opts.has_vararg && base->def->param_count != argc) return NULL; return base; @@ -2497,6 +2505,16 @@ static struct { .call = lengthFunc, .finalize = NULL, .export_to_sql = true, + }, { + .name = "LENGTH_VARBINARY", + .param_count = 1, + .returns = FIELD_TYPE_INTEGER, + .aggregate = FUNC_AGGREGATE_NONE, + .is_deterministic = true, + .flags = SQL_FUNC_LENGTH, + .call = lengthFunc, + .finalize = NULL, + .export_to_sql = true, }, { .name = "LESSER", .call = sql_builtin_stub, @@ -2617,6 +2635,16 @@ static struct { .call = position_func, .finalize = NULL, .export_to_sql = true, + }, { + .name = "POSITION_VARBINARY", + .param_count = 2, + .returns = FIELD_TYPE_INTEGER, + .aggregate = FUNC_AGGREGATE_NONE, + .is_deterministic = true, + .flags = 0, + .call = position_func, + .finalize = NULL, + .export_to_sql = true, }, { .name = "POWER", .call = sql_builtin_stub, @@ -2747,6 +2775,16 @@ static struct { .call = substrFunc, .finalize = NULL, .export_to_sql = true, + }, { + .name = "SUBSTR_VARBINARY", + .param_count = -1, + .returns = FIELD_TYPE_STRING, + .aggregate = FUNC_AGGREGATE_NONE, + .is_deterministic = true, + .flags = SQL_FUNC_DERIVEDCOLL, + .call = substrFunc, + .finalize = NULL, + .export_to_sql = true, }, { .name = "SUM", .param_count = 1, @@ -2787,6 +2825,16 @@ static struct { .call = trim_func, .finalize = NULL, .export_to_sql = true, + }, { + .name = "TRIM_VARBINARY", + .param_count = -1, + .returns = FIELD_TYPE_STRING, + .aggregate = FUNC_AGGREGATE_NONE, + .is_deterministic = true, + .flags = SQL_FUNC_DERIVEDCOLL, + .call = trim_func, + .finalize = NULL, + .export_to_sql = true, }, { .name = "TYPEOF", .param_count = 1, diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index adf90d824..9ff1dd3ff 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -4441,13 +4441,14 @@ sql_func_flag_is_set(struct func *func, uint16_t flag) * A SQL method to find a function in a hash by its name and * count of arguments. Only functions that have 'SQL' engine * export field set true and have exactly the same signature - * are returned. + * are returned. If has_blob_arg is set to true, the varbinary + * version of the function is returned is the functions has one. * * Returns not NULL function pointer when a valid and exported * to SQL engine function is found and NULL otherwise. */ struct func * -sql_func_by_signature(const char *name, int argc); +sql_func_by_signature(const char *name, bool has_blob_arg, int argc); /** * Generate VDBE code to halt execution with correct error if diff --git a/src/box/sql/vdbemem.c b/src/box/sql/vdbemem.c index 8e9ebf7ab..3af96c37c 100644 --- a/src/box/sql/vdbemem.c +++ b/src/box/sql/vdbemem.c @@ -1327,7 +1327,7 @@ valueFromFunction(sql * db, /* The database connection */ pList = p->x.pList; if (pList) nVal = pList->nExpr; - struct func *func = sql_func_by_signature(p->u.zToken, nVal); + struct func *func = sql_func_by_signature(p->u.zToken, false, nVal); if (func == NULL || func->def->language != FUNC_LANGUAGE_SQL_BUILTIN || !func->def->is_deterministic || sql_func_flag_is_set(func, SQL_FUNC_NEEDCOLL)) -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:03 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:03 +0300 Subject: [Tarantool-patches] [PATCH v2 07/10] sql: move built-in function definitions in _func In-Reply-To: References: Message-ID: <1fc825807e36bd97b97a28756b3e9da2c270de5b.1597417321.git.imeevma@gmail.com> This patch moves SQL built-in function definitions to _func. This helps create an unified way to check the types of arguments. It also allows users to see these definitions. Also, this patch enables overloading for length(), trim(), position() and substr() functions. Part of #4159 --- src/box/bootstrap.snap | Bin 5976 -> 6446 bytes src/box/lua/upgrade.lua | 192 +++++++++++++++++++++++++++++++++ src/box/sql/func.c | 6 -- test/box-py/bootstrap.result | 6 +- test/box/access.result | 2 +- test/box/access.test.lua | 2 +- test/box/access_bin.result | 2 +- test/box/access_bin.test.lua | 2 +- test/box/access_sysview.result | 8 +- test/box/function1.result | 6 +- test/wal_off/func_max.result | 8 +- 11 files changed, 212 insertions(+), 22 deletions(-) diff --git a/src/box/bootstrap.snap b/src/box/bootstrap.snap index 8bd4f7ce24216a8bcced6aa97c20c83eb7a02c77..17e5e2c65ea977620fae969da73b5449c9227078 100644 GIT binary patch literal 6446 zcmV+}8PVobPC-x#FfK7O3RY!ub7^mGIv_GGHZCwNG&3z{VK`$kV=*x at VG2oZb97;D zV`VxZI5{{mH#s$AEnzWaFfBARFg7h>IWRXZIb~sDV>vZ4Wn(xq3RXjGZ)0mZAbWiZ z3e~y`y3H810M4`=qUWUm00000D77#B08nMc0NU5=Fi8-2Xl2a6Gw=*N1Dp2d#+F*A z60ngZM6#TmglvJq_#EX=$!Jge7J1w0|Kzo1%_UPNi at +(ZlQ~>}FT|LOA)HF=P_aH# zqoo}H1NZ{X0v~|TwrSK1iB%s^r5O^3L=UQr#wjWz(gUMWYU+P9$Y%#qt<{qQG`!uQ z1Kw)D0dIMm!3G#;FoRx@!Pl@&4qK=N7|<%X;A{A#z=E&gbAk$J5`0lGJ2SzTJ4`~& zNbp6AI`cIw^JV617(vw6aQK9v%oiU%5CI?Zp#u=K!v-E`hfof*!*+OVU^sk&8V;XM z$D!5mX?)6LUZ+gv8#9?q#=m~|?{|~;+wWo*ZnsIec~Q89o9G_dmLJ(6 at yH%oGVROm z;mfjQUiM{$n`MJ6%W5IZKGz`2s+whiEBnl<)(il(PP5hvFhGMUkj|h9EZ~&cq*G?0 z2q?g^2PD9X>HGrCqy&NoY<->-w^^gI;nw$YyupzU-iqmmTba+rb at tCMgAjw=TgTQc^JZ zwB7zcLi)dE`~S)1h!&@S8Ub9^+`rdanY;V%HTR#UUdP!|zyJhjEx;D7wG7~DBLQ4p zq{Y>)&SIoBmLjdMWYPL*1q;waX;*+AI07SkTk*^k~$`GxK zN&vGi0ICdKHGR_?QmR+3q3Z%QciCc^>aHnN at +t*@yb4XpYhHt-9HR&wk5Gb+$0t#| z6DZy{dEz}2Cs}>Unq>6}#3ZX$S$!^%(YStqW?V{^G~0f(aV7cjNR)VgB#HMvNO;c? z&W-qRUZh6 at i5Ss9o+28kfkTKuK?&i3f(kMO3Tj7)_dtSp|6|8{A3%EV at uPPhK6>BL zgGkiDgGkJ=gGkIz96D$>a;)4{wZEF#z!A(=L_Q7X^jYJ^v1h)4OdT7<-HXkwrdY8W zIx*OpW6%3{{hjO=a^1goMy>hTa7nCJSKa(=;wl-Bj!q1Arp%f#yqXSo#KfArTnVd^ z@#w^0XJR&D+}*xwW77ky6N8>nikX=7 zzvwEn>bA_{@psLg+@)0&clXb2>#8Xh=JD~;Eikw3%TvFz^xJ0R^2l at Y%ux at MLcaU= zOf$$bu%}Pd#APOGR`qEU at 8D?eJuh`zeE*XBHo0mp+ at 8NqT|9o}x>K#@qIYUy=1MbD zc-?97b-U0R1=Fp|4%OTOuk_cgUei_gu7i3_+auq`)D6Zsnnu=Rgr~?5A}0bJ1{8Mb;6ZbfW5)~!j^Zc_ z#U`^?gd3Q;xsii~vVntzs&RvbVxe?w&|)}>qce=7I>#dnM{$%k=c`J_gQHGP&op2- z>hwfo4h?!n8Z)RFXk at 2kc7m31hHE6l4A(eD8LrVaib2LUhB3xAf+5B>ZsQkW^w$e8 z`rE}9{p~LoUcB!XUA(UrT)fXLwxDrgp~Z~riY&ffE8)v!1s45RaYg at BSkZs~Q&jQ4 z6jb~l#T5TPp^)PLC!+ZO=>!y3&nBK=KNC){e~Bj8&weGC_&luuh4?_+N;1q z_HkCo9*zpxXWtda?Qh_?-A$L<-QMO+TTH=CTSTEvTWpJ5Ddge_DCFW6S1#rvn-jVV z-Rka>O?SJi1tq#!Poj&ZMRbYoRhARW^}KoY<$2OWnNGfJa;{`@o?HUCl at iEJNCG*K zud(_inD9RdYyKahgf0RJy at xPD?;wcKstKWO1Gq=g*E{<4n3#tRoeih^HL;By<NAVwwPv^N+UTY}vsz|yV0vCtI-D?I2N2; z(UN7R1q%_76(TA{LsW)lLo*cCl&Sr4NhvQmax{0*@HgmOQ~TwT*bVYsrEC9Isq*9k zQ^0r#>}XB_DwJ3`+x0I?r!h3Pfx%O1kiL&W#M0?1!k^Ozp++V zReqnWQd1~ysbsaTij~*m%iSyWXW}x8>z<2OYr9!h&F4G!@42g{!(a-?mu;6C81Y|{Qhm#)QPzS8Fw!lMG at +D8+R{W z_1tE4Uwu`R0lHaQyPxMSY#Mhjrb}D|g{o`+8dD}FdcyuUQ+4g%bcu(Yc%bq(XiS+{ zs7cqq&FS~K)%sa?wW*0jS8ck*C*5q at tfp=I@>M^%z?Em8`mLqz?>FDXuD at PWzqh-u z;vgnf#qR5IYRbZjw$`n?lphYo*g6N4M zCjw}q3KLY806>W;Nk~CL=!hUAq7(szh$llh5u!;DOn{L60OZ4x9)5UGvO^LbkmPVA z>hslugOMAG*g&KP9vX(sAVh{BG3TZj=XJH*Sx^=r! z-7WAfGI5MbHiV;m6W6 at Nm`XPE+#(FJ#iMU)Uy{@*U=GkuN8KMPesR)9*0=&uW<8Z+ zX4R{xuKlb0UcbU6hEd6ew6<`UEHhI}IGzuRldzCn_;#iDGm2LWlWt)oXYH0-Vq3H4 z#7E59FPB8FTmuOO39Z>}gMQcEXCNRUy}D|U6(VZOzxn2(Wc^!&;sTOi3)7#!PunJ+ zJ~07Sx}OtoF at cE(ShG{Yzd2}ru%uGhp`Na*!$IVLt5!B at NSx-@Z-2)mvV zX9>YpjhNa}63?n|OG!*bwd36m zaN;2kypXtD?x$no0H);hY`i+!<&Lm$6=axN5>t_?`F6e~1^~^km&8*#;DPBWu at nxN zmRM+moD~kzK+G+Pqh!!Ekoa<@RPU&k#85Es`hv?T at skUd*og&Ry5G*08nEMC;U*RM zZa&*i3p1f$x0J-P33j_&(h>tnP*bvH;U#BM-pyyL at oGx8E&OAqiw>8=-E2*)Bno%4 zDRGiAa?tFMbTu9mBZ(lF`{|N!Ond|a19rJ3_IW_+#>Dhn;vNUETHx!hOeB)-x8p6b zkp_ at HTyMwZ#6_UAq?CMJbhjK66FKrcF)h`6y%X>dr_`sr at pwO*78YVm;`w?`9OS+E zYDiAJ^OtZwB?jVaNJ{*pqZv~}($#RepO-7_qiKos)oi+*EoJ)ccwMvth)bvxi12Vh z$QV;H13*9k0HYcN5X1zvw9+ON009gpbhvQRl7JC0;czGzOhzfBa*#rZG64W!1T#=G z*8xqXuSAJO2fHAj9icpHP_y&76Od#7P7{ghYfiGGt;>XAJ^s~{CvtjP5Go*D=`CH- zExFP=zp{I}rCV~PS9+&wx};ljF4;}5^vbU3mM+PiUg at 1( z^DW(yJH65?x#uNak~_WfE4k)Nx at 32H=T~;kmvl+4{H9lO&6jjbuJrbzGNbZ9H#=Oa zPROnl;GTHHqoImmGht5K97juG at O8JYuZbs}e!XZ5ub2_yyG(YCIwcF99=`+LV>?(Vz zY*=3D1`G0xGr7aRooE7)#%DzXWU{=na#GMUa*s$A5YjlnE_4NorktX~>KiVOAOd)! z=;~(07cPfzA;Et`7}!O5FTd#W2>NJ10X9UUgeIt2Gup+_s>+~I4p>O-nNOprr2qDy zBgMyuZ{u?6gvMpN9v7e0Mb)8O$s;^WOWAKIG$}MAxu>XP_uJseDh%Ac1=ipWS=l~4 z%ZwzFL}LDu$@=~o5C*9^4O)vGTL9=SIo1O0(@`n3iairXe?SHZaKva$#Aa}n+S(%x z;Gd#xYyN3WDlDHPrQ-GFdISFl4^;;x}m(_kRMk^3)7yf1bzh zo(k0bjBdYe$ve&sU*B$P7|IQFiiX)ORb3%PR9h^=P>+zM-Q=7kuCdIS-ci?lGq7 at d z`%{R&H9_3|W{IF0X7r<6sSDW$FT=XNSj4!l!ur64P%3G*`og at BEVgSv;VNIj8V_e_ z#Jq0OX&(`XY3|<-qT}JT0d!2}Wd#g|%MoJIOC~ar(gKJr71 at CaW0(g5=*aQ^`I!iK zWWIwNR>U>alZtyjRF3~a`$c@`5$kM!?%DDskKf~yX`{yxBek}X zzuDk&h)m0M;_-5L44fDQe%AM8+H15bQv&dCOc$^iQQTcN at SFzgLZpdb=;14G{w{&aCLZW4?^89T^w|8-!28q7%V_+VU1j!ErGAs9Vh^cWDOA>CxmwlEB%Bw)r{ z!g7q>TM at fi6@55BhWn>6PGH8cY4`#a#T$}rH~0D;ggXg5_y+hBY`x`^^&~^5;adJziPvHx^IOGgzGD1>0qFu!Dj5C>G31K%Zxv0u z%d0JBlKINRNT7m6YWcCswnCnu4?V9Gc?6Nk?g$Xdq!fo{043+jp3HA*yly~Qie3gX z40-_K6oFn)nR?$wf$@!E%f)Wws2tI$+U9LjUYN_aIvg9$DIAk42wekSx$Xw*C8|I at +mR|m*c#=?}mF|7ZakH9uI*V!I%mzNSc95|y*suk~P zn6QS25j>y3w6~yWn|VwlGoTlQ)a|n*V9~^hEO_{#XN)j0Nt?Z|xVlsvQVJ9_&{ z70MjCAhL{raaxg7TY^6OLl zEW(X2!O^?<^E63|QRn_1lcZV^=Ql_|L8^^C1Bu*1z;n zG=N3NU0TkELc5Xs(mAhgH57l at 6{u6haU!6 at w(mj0Q^gY;DG<(_qS>R8y*6Y4Sxt|0 zawU5x`WmH>Hzp7zwG+*%M3KIR+(h?>=DqaQn5l{Y9HwcMQwZ*NO2i9yEMVKJ at rj5o zEbDBj4yYS^EP;Y-ga at K(;0FnBOnGe%xx(Ga((tk06zRKotO!6OH}it-+cM&PO42l5 z@?{|vU(DVaVHUm^LdVaBjsG&Lh*CSS>jribnBvygElT|WU1MggRkN!Rs%{Hy5$QJN zq>Zz5QPvnjDl|F|F$$>spp at GMAQqQd=+n340uCGzo?ydT4ZUR$7CrJGNB?~wLR@~e zzwh-B;_{9G`7DgXr;``$;7&NY{W4LO$5>Po at i7j|ZI_XhmYlXTUgZI)PlPpG6V1$c zM?_pGYYZb4nw^3gBvkx>C|a-~ZPih at 7@EPO9WwsOo{l1Rnk9gbK+JlKi6p(yY)DQ- zNv(0SGD={7(QM^3u2USn5dKLD)y#KU0Q5aD4GwaTV1)9^=tx8KvD6M}!wbSC4Z2sO zundl&#@}=#0B+w)amhbi)_QJk{TttqjS7_WxM4$-1>?4gu!^6Vy`ehDPVP_k`*M$|oV}@bF z3MEgo#0QQpo!IBzjYqSXgNZ3mX&Hz6CDC?D5hsF9LwoREE5V};d!tio42#z~zI z148n%eM9C!dntt6!XQdNcVxg^orh%33mG$4$3bW&CKi|!rBRyl!d at oQ-w?&~ja0LN zfCYpq01!@ZQtm=hFp!c55fe#J-ALzF9d0C!hE0i7{LX8Ey{}2U28g4#$<7!YbZQ)< zHBANq6YOJsZ0f#3+*^nJeBcQZyNRR;&N&aE)IFYhJF^pxESEtMQF5wC#fkrvkx>?n zYcqs5?8o8LhYTAOe&-1_i>J at AM%fSwbUTQ IAJq`8?W$`(1ONa4 literal 5976 zcmV-e7pLe`PC-x#FfK7O3RY!ub7^mGIv_GGGcGVKGC49WXEHQ3GGb;i3Q2BrbYX5| zWjY`=Fl01gGdD6VGGjS5Ei`5^GA&{=GB_HuFnz!wkj#k+nW%pPgt zBZ0)WtpN`tMI1&W;^gZsQ&N;l)}E- at Bxx#$t*xi7PvyfprWdR(4G!J07hhyME2Rh12qO5 z03$jVXaJ1pRDgk60$>Z!*eC$j5|Pg}34qnqumK~c0b2t`L_iG~k?1 at i4HzdHfB+|Q zBANqOVgUxQ#40o_ktH4rAQGvnh(zbqk(fnv8maP`cPgLx#(XB9 at vqFbrA}kH{AbCPXF6(|2w%f)%I+$Y2l*g{=M$Xyxo7R zxqlS(dQX-uShQ$;C5zTpu(X=#N~@Qww7S(*n6##{r1g|4T05ms3HoPkO3*)PQi5JZ z2K`g1R@;+AqwR^J^s_1bYJ$zp6Ukc_w6cWq?C!9cPYn?E>A-6&NqF0E!k zI2Q90v}OeycC=B}*kEhng)FuvOe4yLeFSlH5$V6%CUe at dOZWJ@=2NXQ3yZh=XO?YK zbPEd+_0eoFv+T=Jzq$0wW~1`RGm_0w6Olu{`}fN=R%v2Xzo?7KT-0moQ6>(;VY&CL z)U5k{CHE{+)qJ=;f1BDp{-nB5oo3fJbushgnm1V8sQb2k*i;kKY`YTE%m%ge*Q{1i zRrjuiT1D9--$oP<#yBj_YzT)27dlOu?B2ZxS*O$H9bVQUqgyiOVJAc`l(jI|1ejJ4`TjJ0a5 z0)kPK;V>LV!#E7+c#PpN99H`ShBXtyVN+AiE at C)r%E`r_+H;F8XiqJ;$WzEX^~_?6 z)`*1`t?`O1T4QUp0*h^|;)-pg!isI&#wn`kuM|}DH;O6x+g~W8c;6?YcwZ-=c%4l= zQS)NL37XduO at O^w!`Dj*Ci+ieiT;sLqW}JnNaBAZkododBmQ?1VZ{GK6!E`P2qLna zK at 72eAcR=|4?(Pd{XPKke-A(W&w~&DJ9p?Ie;s(pKZhOi&mRXJYI$zRftJSx9E`q> zs*Al9>W-pDpQnZ!yr;ni?`NpNJ7u7u<`u&XG%pxrVE(K_yZ3q_hVET}p?g0sbl-iL z3sN4;kCbmAOv+1n7Qnn;p#|?%V8QzoR`4DL6}&$o1 at ANgg_@o|C(!ipF(KWm at 9ODR zQFMDfCG^>z1U}o3P{Q`xi$H?*A&j6s2qI{&{RbgrzXycu^?ZhBC!Yu%V>OaWwe50G`a~r+tV$HW1PV`@ou>tkcnaJ z-!9jSTH+UL7alIE%x{T-hHZAapDqV%LUmz$To@|X`z5hVs5He2ZHkpDN=8hyCp4ue zOiD?#4(6n>0d+_>efaRvylul at VRMcRs6*mb$TyU({R;&ORU7mG;~udcW=9&0Dib2{ zPk)fjSS7$pMVoklp8#J1z)t`?nZaDja3RAqhDi()&}u1L`o)%fKvr*-dP-rGqLg?^ zyCkS29TFI%FK&=>M^8sy+#Ge8;jCs(MmH;?rXpJrm!d7J&npovL$VCPGUN)86(T00 zC7Eof3bt~?2Mj~d1_A*Gm=n?CxL+WcLr(YpU+EC`2T&5V+GM)o=O+*`M( zzm^)APPegZ&-6pS|JNSn?e<=K)DA`yh1q<=g+CNFn7Ky%-cs8%`8^UVMW2{`!o{}f zR#x4Yw_54XMI{&4J-b>eyVq3C=R5cBd7Gj{A^OCslk6syU2{D|{qEwrqJw`;lveuZ zl)wF>TKEf-H^brXA8q0%q&I2x{mZ6k6LUE-?(Q-z8ESSLcXzdVW^=l=eyYy|TQ05J z&vPF#jk~+35-%ya>e|0XbcvOctpCkdUHcbR;v^(+QvM2!=n^9x>H4!7{XVl at KkJ?{ zb+Oo{%+{sSOqR_l%Caw4HB}o_dG at H^UFrUQ^Ih!vYZdjoj at N0LQC7;+`FW`OaG13` z at 9)>0={JiObItJ+q;fx8%z%KJLH*dzF+$uu^m$W3Q;9KGJ<79nRqtq*;Q^N z7L``{Mwi&gNRAQR!bL at XR?#OWQpB at n+cwCiN4=&wRsNp8S5jrZi^#5r`t!|n_4i3R zT)$G1|>8knE{Cm zM`ADn!|Vkk7mD~P3L^qGqy-@?1X+Pdg`pIHpzz}aA0`O(>A{Cb2|Gv#8bOB$IYPhz z!i^7X?5UxL#@CEIGVsK>1H*$C9RPXB>|hzbnl0h5KY#xC!E}aV+IMZ-{B1s8$YS_v zw&d*EeZ3MF=}I^2>du9MxHQ}LWV+el>n^eEYL>Kx!}u<$S&8x0EJ-WNP`J8B&)UFr zNt=W*qo(#F*Yf>K`n6F0A^mYS~poBUqCLnV$~&61inoKA^vOkWc>Ep=gh zNGmrg*G5Q3NNR4`qu;gn*~kb4r2TP1)4(}32k z`;oyhF%=7bz8o3c5>JuS!->Huv6KpYIGv4mYvNiQ8x#`LuyRmHtaQce;mkB~5}ce) z2iz at jls8R0UDFnZVkO3g#82GVe!QO&J81*-A#oE5dSQG?%w&QnB}S4!mxYfaAcI5V zB at pMEesLA1ONjxMC6whgA>n*Vm1avy!u1ju- zgB&o1r&D4g18`AMKqVeRy| z3jOVHy&mw^g?Sje^nSi3-tppoHm4>I`cS!F66^3brzFnN%h;TDwww<5MGNDQ`q^$f z-VKcc at _Jsn@D2CpYXbU~*hYJ^>7sBwCa%GMJYX0lrcvDSa=so;3=_{_$>5%lSOzhN zxTu4JE1FV>W&i*HKmY(U8w3!9)vS}S3KIZ;fr5j=tR+E#ZH#FzNgQ`WroiFLan{tlfTbgB|Gh;!Y%xx% zk`>T&gv_tCNQW)PNma4}nvRh9wHE2H#W<--RzTAcGQZX$9kv)JRmloyIzr~xTBO4k z& zms0^Veo$FNG9_{HVB12IACUv5#L1D+WDJ8VimHj)MNwheCI<_JBg>zG6CoF&93d!0 z*n_nd?LDdvi0g!)gr0Bd6nzMDc7%LM*Idzd673F-Lj`&$W$i at vfDRXw!lDOjpVg4TWT9;t%mqR z&k!jQ|E|NE0M{Uu!G2Sxiy5j?Spl}kbnKTxR8dclo|6x{^!UVw(*B~T at tlU#AAeSe zm?5Yhvcd1|Kz#S{kuNWhdN*{j!Er`ggN$DUp~sUAFM?X*OokRgEE)2UdopebL3wLG z_=%)b<9Ndm#bNXn at So_H1fgeo52Fuc2|?9rKLP`16x}=-w at 6F`rN^5LFC++_9#1wK zAdk$=FzGEYWXxew!{s1S6E@%D;y443dFEfaSL>BvG~L*LRyn_X11QRbP!$Fh2cPP5 zq_};L4EXUWUj|03mi1JOrh at DF7uU>UCVgAa3x5sjGIZ#fK&fo`6|1<4!G9HtW!L0B znmN^#k552kR^~~^uG;!d@;ZwB0Yt_{F^{@N>7;EiZ45pXcA z?ws_=1?{Z4>QOa%V*zlnw>erc>ZE4#&Ll6WKDb5 at n=(^4P94g6gNGIwOJ7p*H&gZy z+`NlEcm??x!7ZUwogwr!UD|v#=B!~%Ji{78^!%8orm;JuE$QrgiJhT97VVAWf4l93 zt?M$%NaPHt;?!Gl zawQP-zla5M>{O)yvCbvRtM+6HuLc3P{CK7rIYkq=@IQX at X%x-TMJTDIORdhcxK at Rh z0BqP7dy;xIS#w7SUR>ZwI|j|^`Ox at D65h=Bx*mjAo5;DEt~WIb^CPDqd&MoM8=&^?H5aDdD6c~xUjP0I2&r|AFDg>DCt- at wx! zkpLkdivC6CrlHBci0uSPN>PZ?0XL+KLX(R7DwI4WiG}tw0wogGd!ok+sxE-snjD1a zHCi|`%sP5TP+D9=YB at nT7#&HXj-em7Ab;#!do)CcR(n7NWbxm+p*L;N<>!RF-K-m$ z1Ji6HswE{dAdJflhR{V~!|am-H^fh!QxAiw!PbDV08RY6X7smRYqAg`vDolYTUC!~4aU!gB|Z~N@%XuF at pYG(em3vdzTOoN#a(}7Se*ehq)I(XqwT2w)rF~B-C zdP;?Up&zy&9%#IKHC%^Qdq4$2kfnq^OAY+7qZ~@hC`cm)m`x2&iO at f^gBQdDjd!nx z>(FWss2~Wkl+b6Xfj at SXLunZWX~Y1tso^OR`iFK{kl0Rvhx`Y!4_LYYScj?h1)Gq< zN*5>#Dg*n`@kQf6{EjV742uZ90hHD{NZC*sxsT5)a at isKmzaNfJaP9H`JagVN9PyW ziSQ*_X15vnK2fCWT!qpr-Q4x3`9!m)ZGS?-!0$<>DgF~AP&SFrflxizEobZ+c<~@v zG&0kq*m?2L^mwER35-VEI}0Zx<$@IB!glL9x(y-nIO4;|R;IQaHw+ZsR-xm0T#9xw z;-C at 3Yb7g@guCzP$I2WH1jq*}2#q5NWEFj4eKh|wl0!OPZUkCYFbqoARpk5qm^Q%3 zUtY9 at _@U2)>;{vD;a5zxlJMmep>gtE678o1B^M at V6gjw7NEC*xiE4qlVelnVP9thL zJ6tcK2xQQoFL)t5(6R<Z_EH_iMk+W8`~-UKj{0bSve$kR!lyC$Hm;CNd_+G|)jSs0^6k2CiYy#{K5z zOr!;8#y(CbN<{A1N|9)SgzgV88Be)RxS~=DmdFh1f;NMLhix~=wQM*DEW-|s#mgK& z$Fex+HW5}(j=?pi*}i2FcCXTg7#tl`5m3wT;Q{x{+a(T%X=z0qX^g=jE0!<#0HmKAE3X z?Qdlztz~*kV16F^EuOyXbFv$Ai!@dUeVmS-AE`DA>Q_1aKPrC244$$f4BZ>7gR2c- z9D06I7kqkPC<;+e7>HKYja at d|_qf11VYfb79Mvf$v9bdkc9#O*lvDM zjT`tN2r=(}V)SBiSR!VRSBgr9a3?4`d*wNA@*YW<$t}%sllMrP#BMz)0X*Zmx;QY*H}-|Ua*)SWMg1^N_8RIlVbPImSXCMmzA;fFre G5UuU4uRFp3 diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua index add791cd7..290741940 100644 --- a/src/box/lua/upgrade.lua +++ b/src/box/lua/upgrade.lua @@ -971,6 +971,197 @@ local function upgrade_to_2_3_1() create_session_settings_space() end +-------------------------------------------------------------------------------- +-- Tarantool 2.5.2 +-------------------------------------------------------------------------------- + +local function update_sql_builtin_functions() + local _func = box.space[box.schema.FUNC_ID] + local _priv = box.space[box.schema.PRIV_ID] + local updates + local datetime = os.date("%Y-%m-%d %H:%M:%S") + local overloaded_function_list = { + "LENGTH_VARBINARY", "POSITION_VARBINARY", "TRIM_VARBINARY", + "SUBSTR_VARBINARY", + } + + for _, v in pairs(overloaded_function_list) do + local t = _func:auto_increment({ADMIN, v, 1, 'SQL_BUILTIN', '', + 'function', {}, 'any', 'none', 'none', + false, false, true, {}, setmap({}), '', + datetime, datetime}) + _priv:replace{ADMIN, PUBLIC, 'function', t.id, box.priv.X} + end + + _func:run_triggers(false) + updates = {{'=', 'param_list', {'number'}}, {'=', 'returns', 'number'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ABS', updates) + + updates = {{'=', 'param_list', {'number'}}, {'=', 'returns', 'number'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('AVG', updates) + _func.index[2]:update('SUM', updates) + _func.index[2]:update('TOTAL', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('CHAR', updates) + + updates = {{'=', 'param_list', {'string'}}, {'=', 'returns', 'integer'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('CHARACTER_LENGTH', updates) + _func.index[2]:update('CHAR_LENGTH', updates) + + updates = {{'=', 'param_list', {'string'}}, {'=', 'returns', 'integer'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_overload = true}}} + _func.index[2]:update('LENGTH', updates) + + updates = {{'=', 'param_list', {'varbinary'}}, {'=', 'returns', 'integer'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {is_overloaded = true}}} + _func.index[2]:update('LENGTH_VARBINARY', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('COALESCE', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'integer'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('COUNT', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('GREATEST', updates) + _func.index[2]:update('LEAST', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar'}}, + {'=', 'returns', 'string'}, {'=', 'aggregate', 'group'}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('GROUP_CONCAT', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('HEX', updates) + + updates = {{'=', 'param_list', {'scalar', 'scalar'}}, + {'=', 'returns', 'scalar'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('IFNULL', updates) + _func.index[2]:update('NULLIF', updates) + + updates = {{'=', 'param_list', {'string', 'string', 'string'}}, + {'=', 'returns', 'boolean'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('LIKE', updates) + + updates = {{'=', 'param_list', {'scalar', 'double'}}, + {'=', 'returns', 'scalar'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LIKELIHOOD', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LIKELY', updates) + _func.index[2]:update('UNLIKELY', updates) + + updates = {{'=', 'param_list', {'string'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('LOWER', updates) + _func.index[2]:update('SOUNDEX', updates) + _func.index[2]:update('UNICODE', updates) + _func.index[2]:update('UPPER', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('QUOTE', updates) + _func.index[2]:update('TYPEOF', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'scalar'}, + {'=', 'aggregate', 'group'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('MAX', updates) + _func.index[2]:update('MIN', updates) + + updates = {{'=', 'param_list', {'string', 'string'}}, + {'=', 'returns', 'integer'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_overload = true}}} + _func.index[2]:update('POSITION', updates) + + updates = {{'=', 'param_list', {'varbinary', 'varbinary'}}, + {'=', 'returns', 'integer'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {is_overloaded = true}}} + _func.index[2]:update('POSITION_VARBINARY', updates) + + updates = {{'=', 'param_list', {'scalar'}}, {'=', 'returns', 'string'}, + {'=', 'is_deterministic', true}, {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('PRINTF', updates) + + updates = {{'=', 'param_list', {'string', 'unsigned', 'string'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true, has_overload = true}}} + _func.index[2]:update('TRIM', updates) + + updates = {{'=', 'param_list', {'varbinary', 'unsigned', 'varbinary'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true, is_overloaded = true}}} + _func.index[2]:update('TRIM_VARBINARY', updates) + + updates = {{'=', 'returns', 'integer'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('RANDOM', updates) + + updates = {{'=', 'returns', 'integer'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ROW_COUNT', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, + {'=', 'returns', 'varbinary'}, {'=', 'exports', {'SQL'}}} + _func.index[2]:update('RANDOMBLOB', updates) + + updates = {{'=', 'param_list', {'unsigned'}}, + {'=', 'returns', 'varbinary'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('ZEROBLOB', updates) + + updates = {{'=', 'param_list', {'string', 'string', 'string'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('REPLACE', updates) + + updates = {{'=', 'param_list', {'double', 'unsigned'}}, + {'=', 'returns', 'double'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, {'=', 'opts', {has_vararg = true}}} + _func.index[2]:update('ROUND', updates) + + updates = {{'=', 'param_list', {'string', 'integer', 'integer'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true, has_overload = true}}} + _func.index[2]:update('SUBSTR', updates) + + updates = {{'=', 'param_list', {'varbinary', 'integer', 'integer'}}, + {'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}, + {'=', 'opts', {has_vararg = true, is_overloaded = true}}} + _func.index[2]:update('SUBSTR_VARBINARY', updates) + + updates = {{'=', 'returns', 'string'}, {'=', 'is_deterministic', true}, + {'=', 'exports', {'SQL'}}} + _func.index[2]:update('VERSION', updates) + _func:run_triggers(true) +end + +local function upgrade_to_2_5_2() + update_sql_builtin_functions() +end + -------------------------------------------------------------------------------- local function get_version() @@ -1007,6 +1198,7 @@ local function upgrade(options) {version = mkversion(2, 2, 1), func = upgrade_to_2_2_1, auto = true}, {version = mkversion(2, 3, 0), func = upgrade_to_2_3_0, auto = true}, {version = mkversion(2, 3, 1), func = upgrade_to_2_3_1, auto = true}, + {version = mkversion(2, 5, 2), func = upgrade_to_2_5_2, auto = true}, } for _, handler in ipairs(handlers) do diff --git a/src/box/sql/func.c b/src/box/sql/func.c index ae1842824..1fbffa535 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2971,12 +2971,6 @@ func_sql_builtin_new(struct func_def *def) func->flags = sql_builtins[idx].flags; func->call = sql_builtins[idx].call; func->finalize = sql_builtins[idx].finalize; - def->param_count = sql_builtins[idx].param_count; - def->is_deterministic = sql_builtins[idx].is_deterministic; - def->returns = sql_builtins[idx].returns; - def->aggregate = sql_builtins[idx].aggregate; - def->exports.sql = sql_builtins[idx].export_to_sql; - def->opts.has_vararg = sql_builtins[idx].param_count == -1; return &func->base; } diff --git a/test/box-py/bootstrap.result b/test/box-py/bootstrap.result index 0876e77a6..289289443 100644 --- a/test/box-py/bootstrap.result +++ b/test/box-py/bootstrap.result @@ -4,7 +4,7 @@ box.internal.bootstrap() box.space._schema:select{} --- - - ['max_id', 511] - - ['version', 2, 3, 1] + - ['version', 2, 5, 2] ... box.space._cluster:select{} --- @@ -242,6 +242,10 @@ box.space._priv:select{} - [1, 2, 'function', 65, 4] - [1, 2, 'function', 66, 4] - [1, 2, 'function', 67, 4] + - [1, 2, 'function', 68, 4] + - [1, 2, 'function', 69, 4] + - [1, 2, 'function', 70, 4] + - [1, 2, 'function', 71, 4] - [1, 2, 'space', 276, 2] - [1, 2, 'space', 277, 1] - [1, 2, 'space', 281, 1] diff --git a/test/box/access.result b/test/box/access.result index 20b1b8b35..b72fa8b02 100644 --- a/test/box/access.result +++ b/test/box/access.result @@ -703,7 +703,7 @@ box.schema.func.exists(1) --- - true ... -box.schema.func.exists(68) +box.schema.func.exists(72) --- - false ... diff --git a/test/box/access.test.lua b/test/box/access.test.lua index 3e083a383..759d0a94f 100644 --- a/test/box/access.test.lua +++ b/test/box/access.test.lua @@ -282,7 +282,7 @@ box.schema.user.exists{} box.schema.func.exists('nosuchfunc') box.schema.func.exists('guest') box.schema.func.exists(1) -box.schema.func.exists(68) +box.schema.func.exists(72) box.schema.func.exists('box.schema.user.info') box.schema.func.exists() box.schema.func.exists(nil) diff --git a/test/box/access_bin.result b/test/box/access_bin.result index c58f331d3..d345e5996 100644 --- a/test/box/access_bin.result +++ b/test/box/access_bin.result @@ -298,7 +298,7 @@ box.schema.user.grant('guest', 'execute', 'universe') function f1() return box.space._func:get(1)[4] end --- ... -function f2() return box.space._func:get(68)[4] end +function f2() return box.space._func:get(72)[4] end --- ... box.schema.func.create('f1') diff --git a/test/box/access_bin.test.lua b/test/box/access_bin.test.lua index 41d5f4245..13d8041a9 100644 --- a/test/box/access_bin.test.lua +++ b/test/box/access_bin.test.lua @@ -112,7 +112,7 @@ test:drop() -- notice that guest can execute stuff, but can't read space _func box.schema.user.grant('guest', 'execute', 'universe') function f1() return box.space._func:get(1)[4] end -function f2() return box.space._func:get(68)[4] end +function f2() return box.space._func:get(72)[4] end box.schema.func.create('f1') box.schema.func.create('f2',{setuid=true}) c = net.connect(box.cfg.listen) diff --git a/test/box/access_sysview.result b/test/box/access_sysview.result index 799d19f03..f9ffae127 100644 --- a/test/box/access_sysview.result +++ b/test/box/access_sysview.result @@ -258,11 +258,11 @@ box.session.su('guest') ... #box.space._vpriv:select{} --- -- 82 +- 86 ... #box.space._vfunc:select{} --- -- 67 +- 71 ... #box.space._vcollation:select{} --- @@ -290,11 +290,11 @@ box.session.su('guest') ... #box.space._vpriv:select{} --- -- 82 +- 86 ... #box.space._vfunc:select{} --- -- 67 +- 71 ... #box.space._vsequence:select{} --- diff --git a/test/box/function1.result b/test/box/function1.result index 928cd5758..3cce0f4e2 100644 --- a/test/box/function1.result +++ b/test/box/function1.result @@ -97,7 +97,7 @@ box.func["function1.args"] exports: lua: true sql: false - id: 68 + id: 72 setuid: false is_multikey: false is_deterministic: false @@ -593,7 +593,7 @@ func exports: lua: true sql: false - id: 68 + id: 72 setuid: false is_multikey: false is_deterministic: false @@ -665,7 +665,7 @@ func exports: lua: true sql: false - id: 68 + id: 72 setuid: false is_multikey: false is_deterministic: false diff --git a/test/wal_off/func_max.result b/test/wal_off/func_max.result index 78db38d6b..1aef99c66 100644 --- a/test/wal_off/func_max.result +++ b/test/wal_off/func_max.result @@ -42,11 +42,11 @@ test_run:cmd("setopt delimiter ''"); ... func_limit() --- -- error: 'Failed to create function ''func31934'': function id is too big' +- error: 'Failed to create function ''func31930'': function id is too big' ... drop_limit_func() --- -- error: Function 'func31934' does not exist +- error: Function 'func31930' does not exist ... box.schema.user.create('testuser') --- @@ -62,11 +62,11 @@ session.su('testuser') ... func_limit() --- -- error: 'Failed to create function ''func31934'': function id is too big' +- error: 'Failed to create function ''func31930'': function id is too big' ... drop_limit_func() --- -- error: Function 'func31934' does not exist +- error: Function 'func31930' does not exist ... session.su('admin') --- -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:05 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:05 +0300 Subject: [Tarantool-patches] [PATCH v2 08/10] box: add param_list to 'struct func' In-Reply-To: References: Message-ID: <14bca5aab8484f9fcd0c93b29c183a8133b71d35.1597417321.git.imeevma@gmail.com> This is needed to create an uniform way to check the types of arguments of SQL built-in functions. Part of #4159 --- src/box/alter.cc | 12 ++++++++++-- src/box/func.c | 1 + src/box/func_def.h | 2 ++ src/box/lua/call.c | 2 ++ src/box/sql/func.c | 1 + 5 files changed, 16 insertions(+), 2 deletions(-) diff --git a/src/box/alter.cc b/src/box/alter.cc index ba96d9c62..0914a7615 100644 --- a/src/box/alter.cc +++ b/src/box/alter.cc @@ -3293,7 +3293,11 @@ func_def_new_from_tuple(struct tuple *tuple) diag_set(OutOfMemory, def_sz, "malloc", "def"); return NULL; } - auto def_guard = make_scoped_guard([=] { free(def); }); + def->param_list = NULL; + auto def_guard = make_scoped_guard([=] { + free(def->param_list); + free(def); + }); if (func_def_get_ids_from_tuple(tuple, &def->fid, &def->uid) != 0) return NULL; if (def->fid > BOX_FUNCTION_MAX) { @@ -3403,6 +3407,8 @@ func_def_new_from_tuple(struct tuple *tuple) if (param_list == NULL) return NULL; uint32_t argc = mp_decode_array(¶m_list); + uint32_t size = sizeof(enum field_type) * argc; + def->param_list = (enum field_type *)malloc(size); for (uint32_t i = 0; i < argc; i++) { if (mp_typeof(*param_list) != MP_STR) { diag_set(ClientError, ER_FIELD_TYPE, @@ -3412,7 +3418,8 @@ func_def_new_from_tuple(struct tuple *tuple) } uint32_t len; const char *str = mp_decode_str(¶m_list, &len); - if (STRN2ENUM(field_type, str, len) == field_type_MAX) { + def->param_list[i] = STRN2ENUM(field_type, str, len); + if (def->param_list[i] == field_type_MAX) { diag_set(ClientError, ER_CREATE_FUNCTION, def->name, "invalid argument type"); return NULL; @@ -3433,6 +3440,7 @@ func_def_new_from_tuple(struct tuple *tuple) /* By default export to Lua, but not other frontends. */ def->exports.lua = true; def->param_count = 0; + assert(def->param_list == NULL); } if (func_def_check(def) != 0) return NULL; diff --git a/src/box/func.c b/src/box/func.c index 8087c953f..1d20f8872 100644 --- a/src/box/func.c +++ b/src/box/func.c @@ -510,6 +510,7 @@ func_c_destroy(struct func *base) { assert(base->vtab == &func_c_vtab); assert(base != NULL && base->def->language == FUNC_LANGUAGE_C); + free(base->def->param_list); struct func_c *func = (struct func_c *) base; func_c_unload(func); TRASH(base); diff --git a/src/box/func_def.h b/src/box/func_def.h index bab2186da..712a5123b 100644 --- a/src/box/func_def.h +++ b/src/box/func_def.h @@ -129,6 +129,8 @@ struct func_def { bool is_sandboxed; /** The count of function's input arguments. */ int param_count; + /** List of input arguments to the function. */ + enum field_type *param_list; /** The type of the value returned by function. */ enum field_type returns; /** Function aggregate option. */ diff --git a/src/box/lua/call.c b/src/box/lua/call.c index 0315e720c..1dc4589dd 100644 --- a/src/box/lua/call.c +++ b/src/box/lua/call.c @@ -783,6 +783,7 @@ func_lua_destroy(struct func *func) { assert(func != NULL && func->def->language == FUNC_LANGUAGE_LUA); assert(func->vtab == &func_lua_vtab); + free(func->def->param_list); TRASH(func); free(func); } @@ -812,6 +813,7 @@ func_persistent_lua_destroy(struct func *base) assert(base != NULL && base->def->language == FUNC_LANGUAGE_LUA && base->def->body != NULL); assert(base->vtab == &func_persistent_lua_vtab); + free(base->def->param_list); struct func_lua *func = (struct func_lua *) base; func_persistent_lua_unload(func); free(func); diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 1fbffa535..c289f2de0 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -2979,6 +2979,7 @@ func_sql_builtin_destroy(struct func *func) { assert(func->vtab == &func_sql_builtin_vtab); assert(func->def->language == FUNC_LANGUAGE_SQL_BUILTIN); + free(func->def->param_list); free(func); } -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:11 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:11 +0300 Subject: [Tarantool-patches] [PATCH v2 09/10] sql: check built-in functions argument types In-Reply-To: References: Message-ID: <30b965327d52f04ef5adc433593b73dc3c7ce676.1597417321.git.imeevma@gmail.com> This patch creates a uniform way to check the argument types of SQL built-in functions. Prior to this patch, argument types were checked inside functions. They are now checked in the ApplyType opcode. Closes #4159 --- src/box/sql/expr.c | 4 + src/box/sql/select.c | 26 + src/box/sql/sqlInt.h | 14 + test/sql-tap/cse.test.lua | 24 +- test/sql-tap/func.test.lua | 46 +- test/sql-tap/orderby1.test.lua | 2 +- test/sql-tap/position.test.lua | 16 +- test/sql-tap/substr.test.lua | 2 +- test/sql/boolean.result | 32 +- test/sql/checks.result | 8 - test/sql/checks.test.lua | 2 - test/sql/types.result | 1535 +++++++++++++++++++++++++++++++- test/sql/types.test.lua | 258 ++++++ 13 files changed, 1828 insertions(+), 141 deletions(-) diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index 99ca91bba..68b55f0e4 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -4134,6 +4134,10 @@ sqlExprCodeTarget(Parse * pParse, Expr * pExpr, int target) } else { r1 = 0; } + if (func->def->language == FUNC_LANGUAGE_SQL_BUILTIN) { + sql_emit_func_arg_type_check(v, func, r1, + nFarg); + } if (sql_func_flag_is_set(func, SQL_FUNC_NEEDCOLL)) { sqlVdbeAddOp4(v, OP_CollSeq, 0, 0, 0, (char *)coll, P4_COLLSEQ); diff --git a/src/box/sql/select.c b/src/box/sql/select.c index b0554a172..49f01eb0d 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -124,6 +124,31 @@ clearSelect(sql * db, Select * p, int bFree) } } +void +sql_emit_func_arg_type_check(struct Vdbe *vdbe, struct func *func, int reg, + uint32_t argc) +{ + if (argc == 0 || func->def->param_list == NULL) + return; + assert(func->def->param_count > 0); + uint32_t len = (uint32_t)func->def->param_count; + assert(len > 0); + size_t size = (argc + 1) * sizeof(enum field_type); + enum field_type *types = sqlDbMallocZero(sql_get(), size); + if (argc <= len) { + for (uint32_t i = 0; i < argc; ++i) + types[i] = func->def->param_list[i]; + } else { + for (uint32_t i = 0; i < len; ++i) + types[i] = func->def->param_list[i]; + for (uint32_t i = len; i < argc; ++i) + types[i] = func->def->param_list[len - 1]; + } + types[argc] = field_type_MAX; + sqlVdbeAddOp4(vdbe, OP_ApplyType, reg, argc, 0, (char *)types, + P4_DYNAMIC); +} + /* * Initialize a SelectDest structure. */ @@ -5420,6 +5445,7 @@ updateAccumulator(Parse * pParse, AggInfo * pAggInfo) vdbe_insert_distinct(pParse, pF->iDistinct, pF->reg_eph, addrNext, 1, regAgg); } + sql_emit_func_arg_type_check(v, pF->func, regAgg, nArg); if (sql_func_flag_is_set(pF->func, SQL_FUNC_NEEDCOLL)) { struct coll *coll = NULL; struct ExprList_item *pItem; diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index 9ff1dd3ff..38fa83df0 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -3883,6 +3883,20 @@ sql_index_type_str(struct sql *db, const struct index_def *idx_def); void sql_emit_table_types(struct Vdbe *v, struct space_def *def, int reg); +/** + * Code an OP_ApplyType opcode that try to cast implicitly types + * for given range of register starting from @a reg. These values + * then will be used as arguments of a function. + * + * @param vdbe VDBE. + * @param func Definition of the function. + * @param reg Register where types will be placed. + * @param argc Number of arguments. + */ +void +sql_emit_func_arg_type_check(struct Vdbe *vdbe, struct func *func, + int reg, uint32_t argc); + enum field_type sql_type_result(enum field_type lhs, enum field_type rhs); diff --git a/test/sql-tap/cse.test.lua b/test/sql-tap/cse.test.lua index 341b6de01..18ddbf47c 100755 --- a/test/sql-tap/cse.test.lua +++ b/test/sql-tap/cse.test.lua @@ -1,6 +1,6 @@ #!/usr/bin/env tarantool test = require("sqltester") -test:plan(118) +test:plan(116) --!./tcltestrunner.lua -- 2008 April 1 @@ -193,28 +193,6 @@ test:do_execsql_test( -- }) - - -test:do_execsql_test( - "cse-1.13", - [[ - SELECT upper(b), typeof(b), b FROM t1 - ]], { - -- - "11", "integer", 11, "21", "integer", 21 - -- - }) - -test:do_execsql_test( - "cse-1.14", - [[ - SELECT b, typeof(b), upper(b), typeof(b), b FROM t1 - ]], { - -- - 11, "integer", "11", "integer", 11, 21, "integer", "21", "integer", 21 - -- - }) - -- Overflow the column cache. Create queries involving more and more -- columns until the cache overflows. Verify correct operation throughout. -- diff --git a/test/sql-tap/func.test.lua b/test/sql-tap/func.test.lua index 3c088920f..575a21007 100755 --- a/test/sql-tap/func.test.lua +++ b/test/sql-tap/func.test.lua @@ -1,6 +1,6 @@ #!/usr/bin/env tarantool test = require("sqltester") -test:plan(14694) +test:plan(14693) --!./tcltestrunner.lua -- 2001 September 15 @@ -95,7 +95,7 @@ test:do_execsql_test( test:do_execsql_test( "func-1.4", [[ - SELECT coalesce(length(a),-1) FROM t2 + SELECT coalesce(length(CAST(a AS STRING)),-1) FROM t2 ]], { -- 1, -1, 3, -1, 5 @@ -197,7 +197,7 @@ test:do_execsql_test( test:do_execsql_test( "func-2.9", [[ - SELECT substr(a,1,1) FROM t2 + SELECT substr(CAST(a AS STRING),1,1) FROM t2 ]], { -- "1", "", "3", "", "6" @@ -207,7 +207,7 @@ test:do_execsql_test( test:do_execsql_test( "func-2.10", [[ - SELECT substr(a,2,2) FROM t2 + SELECT substr(CAST(a AS STRING),2,2) FROM t2 ]], { -- "", "", "45", "", "78" @@ -412,13 +412,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "func-4.4.2", [[ SELECT abs(t1) FROM tbl1 ]], { -- - 0.0, 0.0, 0.0, 0.0, 0.0 + 1, "Type mismatch: can not convert this to number" -- }) @@ -502,13 +502,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "func-4.13", [[ SELECT round(t1,2) FROM tbl1 ]], { -- - 0.0, 0.0, 0.0, 0.0, 0.0 + 1, "Type mismatch: can not convert this to double" -- }) @@ -760,18 +760,6 @@ test:do_execsql_test( -- }) -test:do_execsql_test( - "func-5.3", - [[ - SELECT upper(a), lower(a) FROM t2 - ]], { - -- - "1","1","","","345","345","","","67890","67890" - -- - }) - - - test:do_catchsql_test( "func-5.5", [[ @@ -797,7 +785,7 @@ test:do_execsql_test( test:do_execsql_test( "func-6.2", [[ - SELECT coalesce(upper(a),'nil') FROM t2 + SELECT coalesce(upper(CAST(a AS STRING)),'nil') FROM t2 ]], { -- "1","nil","345","nil","67890" @@ -893,7 +881,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.5", [[ - SELECT sum(x) FROM (SELECT '9223372036' || '854775807' AS x + SELECT sum(x) FROM (SELECT CAST('9223372036' || '854775807' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -904,7 +892,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.6", [[ - SELECT typeof(sum(x)) FROM (SELECT '9223372036' || '854775807' AS x + SELECT typeof(sum(x)) FROM (SELECT CAST('9223372036' || '854775807' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -915,7 +903,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.7", [[ - SELECT typeof(sum(x)) FROM (SELECT '9223372036' || '854775808' AS x + SELECT typeof(sum(x)) FROM (SELECT CAST('9223372036' || '854775808' AS INTEGER) AS x UNION ALL SELECT -9223372036854775807) ]], { -- @@ -926,7 +914,7 @@ test:do_execsql_test( test:do_execsql_test( "func-8.8", [[ - SELECT sum(x)>0.0 FROM (SELECT '9223372036' || '854775808' AS x + SELECT sum(x)>0.0 FROM (SELECT CAST('9223372036' || '854775808' AS INTEGER) AS x UNION ALL SELECT -9223372036850000000) ]], { -- @@ -985,7 +973,7 @@ test:do_execsql_test( test:do_execsql_test( "func-9.5", [[ - SELECT length(randomblob(32)), length(randomblob(-5)), + SELECT length(randomblob(32)), length(randomblob(0)), length(randomblob(2000)) ]], { -- @@ -2918,7 +2906,7 @@ test:do_catchsql_test( SELECT ROUND(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to numeric" + 1, "Type mismatch: can not convert varbinary to double" -- }) @@ -2928,7 +2916,7 @@ test:do_catchsql_test( SELECT RANDOMBLOB(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to numeric" + 1, "Type mismatch: can not convert varbinary to unsigned" -- }) @@ -2938,7 +2926,7 @@ test:do_catchsql_test( SELECT SOUNDEX(X'FF') ]], { -- - 1, "Type mismatch: can not convert varbinary to text" + 1, "Type mismatch: can not convert varbinary to string" -- }) diff --git a/test/sql-tap/orderby1.test.lua b/test/sql-tap/orderby1.test.lua index 51e8d301f..95a8de487 100755 --- a/test/sql-tap/orderby1.test.lua +++ b/test/sql-tap/orderby1.test.lua @@ -735,7 +735,7 @@ test:do_execsql_test( SELECT ( SELECT 'hardware' FROM ( SELECT 'software' ORDER BY 'firmware' ASC, 'sportswear' DESC - ) GROUP BY 1 HAVING length(b) <> 0 + ) GROUP BY 1 HAVING length(CAST(b AS STRING)) <> 0 ) FROM abc; ]], { diff --git a/test/sql-tap/position.test.lua b/test/sql-tap/position.test.lua index e0455abc9..08fee3796 100755 --- a/test/sql-tap/position.test.lua +++ b/test/sql-tap/position.test.lua @@ -228,7 +228,7 @@ test:do_test( return test:catchsql "SELECT position(34, 12345);" end, { -- - 1, "Inconsistent types: expected text or varbinary got unsigned" + 1, "Type mismatch: can not convert 34 to string" -- }) @@ -238,7 +238,7 @@ test:do_test( return test:catchsql "SELECT position(34, 123456.78);" end, { -- - 1, "Inconsistent types: expected text or varbinary got real" + 1, "Type mismatch: can not convert 34 to string" -- }) @@ -248,7 +248,7 @@ test:do_test( return test:catchsql "SELECT position(x'3334', 123456.78);" end, { -- - 1, "Inconsistent types: expected text or varbinary got real" + 1, "Type mismatch: can not convert 123456.78 to varbinary" -- }) @@ -554,7 +554,7 @@ test:do_test( return test:catchsql("SELECT position('x', x'78c3a4e282ac79');") end, { -- - 1, "Inconsistent types: expected text got varbinary" + 1, "Type mismatch: can not convert varbinary to string" -- }) @@ -564,7 +564,7 @@ test:do_test( return test:catchsql "SELECT position('y', x'78c3a4e282ac79');" end, { -- - 1, "Inconsistent types: expected text got varbinary" + 1, "Type mismatch: can not convert varbinary to string" -- }) @@ -614,7 +614,7 @@ test:do_test( return test:catchsql "SELECT position(x'79', 'x??y');" end, { -- - 1, "Inconsistent types: expected varbinary got text" + 1, "Type mismatch: can not convert x??y to varbinary" -- }) @@ -624,7 +624,7 @@ test:do_test( return test:catchsql "SELECT position(x'a4', 'x??y');" end, { -- - 1, "Inconsistent types: expected varbinary got text" + 1, "Type mismatch: can not convert x??y to varbinary" -- }) @@ -634,7 +634,7 @@ test:do_test( return test:catchsql "SELECT position('y', x'78c3a4e282ac79');" end, { -- - 1, "Inconsistent types: expected text got varbinary" + 1, "Type mismatch: can not convert varbinary to string" -- }) diff --git a/test/sql-tap/substr.test.lua b/test/sql-tap/substr.test.lua index a9e656e6d..a970f9e93 100755 --- a/test/sql-tap/substr.test.lua +++ b/test/sql-tap/substr.test.lua @@ -25,7 +25,7 @@ test:plan(93) -- test:execsql [[ CREATE TABLE t1(id integer primary key --autoincrement - , t text, b SCALAR) + , t text, b VARBINARY) ]] local function substr_test(id, string, i1, i2, result) diff --git a/test/sql/boolean.result b/test/sql/boolean.result index 51ec5820b..b525e908c 100644 --- a/test/sql/boolean.result +++ b/test/sql/boolean.result @@ -276,29 +276,17 @@ SELECT is_boolean('true'); SELECT abs(a) FROM t0; | --- | - null - | - 'Inconsistent types: expected number got boolean' + | - 'Type mismatch: can not convert FALSE to number' | ... SELECT lower(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: string - | rows: - | - ['false'] - | - ['true'] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string' | ... SELECT upper(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: string - | rows: - | - ['FALSE'] - | - ['TRUE'] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string' | ... SELECT quote(a) FROM t0; | --- @@ -314,14 +302,8 @@ SELECT quote(a) FROM t0; -- gh-4462: LENGTH didn't take BOOLEAN arguments. SELECT length(a) FROM t0; | --- - | - metadata: - | - name: COLUMN_1 - | type: integer - | rows: - | - [5] - | - [4] - | - [null] - | - [null] + | - null + | - 'Type mismatch: can not convert FALSE to string' | ... SELECT typeof(a) FROM t0; | --- diff --git a/test/sql/checks.result b/test/sql/checks.result index 7b18e5d6b..3c942fb23 100644 --- a/test/sql/checks.result +++ b/test/sql/checks.result @@ -519,14 +519,6 @@ s:insert({1, 'string'}) --- - error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' ... -s:insert({1, {map=true}}) ---- -- error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' -... -s:insert({1, {'a', 'r','r','a','y'}}) ---- -- error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' -... s:insert({1, 3.14}) --- - error: 'Check constraint failed ''complex2'': typeof(coalesce(z,0))==''integer''' diff --git a/test/sql/checks.test.lua b/test/sql/checks.test.lua index 301f8ea69..b55abe955 100644 --- a/test/sql/checks.test.lua +++ b/test/sql/checks.test.lua @@ -173,8 +173,6 @@ s:format({{name='X', type='integer'}, {name='Z', type='any'}}) _ = s:create_index('pk', {parts = {1, 'integer'}}) _ = box.space._ck_constraint:insert({s.id, 'complex2', false, 'SQL', 'typeof(coalesce(z,0))==\'integer\'', true}) s:insert({1, 'string'}) -s:insert({1, {map=true}}) -s:insert({1, {'a', 'r','r','a','y'}}) s:insert({1, 3.14}) s:insert({1, 666}) s:drop() diff --git a/test/sql/types.result b/test/sql/types.result index 2498f3a48..bd503c700 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -215,25 +215,22 @@ box.execute("INSERT INTO t1 VALUES (randomblob(5));") box.execute("SELECT * FROM t1 WHERE s LIKE 'blob';") --- - null -- 'Inconsistent types: expected text got varbinary' +- 'Type mismatch: can not convert varbinary to string' ... box.execute("SELECT * FROM t1 WHERE 'blob' LIKE s;") --- - null -- 'Inconsistent types: expected text got varbinary' +- 'Type mismatch: can not convert varbinary to string' ... box.execute("SELECT * FROM t1 WHERE 'blob' LIKE x'0000';") --- - null -- 'Inconsistent types: expected text got varbinary' +- 'Type mismatch: can not convert varbinary to string' ... box.execute("SELECT s LIKE NULL FROM t1;") --- -- metadata: - - name: COLUMN_1 - type: boolean - rows: - - [null] +- null +- 'Type mismatch: can not convert varbinary to string' ... box.execute("DELETE FROM t1;") --- @@ -246,20 +243,17 @@ box.execute("INSERT INTO t1 VALUES (1);") box.execute("SELECT * FROM t1 WHERE s LIKE 'int';") --- - null -- 'Inconsistent types: expected text got unsigned' +- 'Type mismatch: can not convert 1 to string' ... box.execute("SELECT * FROM t1 WHERE 'int' LIKE 4;") --- - null -- 'Inconsistent types: expected text got unsigned' +- 'Type mismatch: can not convert 4 to string' ... box.execute("SELECT NULL LIKE s FROM t1;") --- -- metadata: - - name: COLUMN_1 - type: boolean - rows: - - [null] +- null +- 'Type mismatch: can not convert 1 to string' ... box.space.T1:drop() --- @@ -830,19 +824,13 @@ box.execute("DELETE FROM t WHERE i < 18446744073709551613;") ... box.execute("SELECT lower(i) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['18446744073709551613'] +- null +- 'Type mismatch: can not convert 18446744073709551613 to string' ... box.execute("SELECT upper(i) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['18446744073709551613'] +- null +- 'Type mismatch: can not convert 18446744073709551613 to string' ... box.execute("SELECT abs(i) FROM t;") --- @@ -1312,17 +1300,17 @@ box.execute("SELECT group_concat(v) FROM t;") box.execute("SELECT lower(v) FROM t;") --- - null -- 'Inconsistent types: expected text got varbinary' +- 'Type mismatch: can not convert varbinary to string' ... box.execute("SELECT upper(v) FROM t;") --- - null -- 'Inconsistent types: expected text got varbinary' +- 'Type mismatch: can not convert varbinary to string' ... box.execute("SELECT abs(v) FROM t;") --- - null -- 'Inconsistent types: expected number got varbinary' +- 'Type mismatch: can not convert varbinary to number' ... box.execute("SELECT typeof(v) FROM t;") --- @@ -1879,25 +1867,13 @@ box.execute("SELECT group_concat(d) FROM t;") ... box.execute("SELECT lower(d) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['10.0'] - - ['-2.0'] - - ['3.3'] - - ['1.8e+19'] +- null +- 'Type mismatch: can not convert 10.0 to string' ... box.execute("SELECT upper(d) FROM t;") --- -- metadata: - - name: COLUMN_1 - type: string - rows: - - ['10.0'] - - ['-2.0'] - - ['3.3'] - - ['1.8E+19'] +- null +- 'Type mismatch: can not convert 10.0 to string' ... box.execute("SELECT abs(d) FROM t;") --- @@ -2807,3 +2783,1474 @@ box.execute([[SELECT typeof(length('abc'));]]) rows: - ['integer'] ... +-- Make sure the function argument types are checked. +box.execute([[SELECT abs(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT abs(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT abs(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT abs(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT abs('a');]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT abs(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT char(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT char(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ["\x01"] +... +box.execute([[SELECT char(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ["\x01"] +... +box.execute([[SELECT char(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT char('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT char(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[SELECT character_length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT character_length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT character_length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT character_length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT character_length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT character_length(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT char_length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT char_length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT char_length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT char_length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT char_length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT char_length(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT coalesce(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT coalesce(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT coalesce(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT coalesce(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT coalesce('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT coalesce(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT greatest(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT greatest(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT greatest(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT greatest(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT greatest('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT greatest(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT hex(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['2D31'] +... +box.execute([[SELECT hex(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['31'] +... +box.execute([[SELECT hex(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['312E35'] +... +box.execute([[SELECT hex(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['54525545'] +... +box.execute([[SELECT hex('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['61'] +... +box.execute([[SELECT hex(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['33'] +... +box.execute([[SELECT ifnull(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT ifnull(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT ifnull(1.5, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT ifnull(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT ifnull('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT ifnull(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT least(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT least(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT least(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT least(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT least('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT least(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT length(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT length(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT length(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT length(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT length('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT length(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT likelihood(-1, -1);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(1, 1);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(1.5, 1.5);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(true, true);]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood('a', 'a');]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likelihood(X'33', X'33');]]) +--- +- null +- Illegal parameters, second argument to likelihood() must be a constant between 0.0 + and 1.0 +... +box.execute([[SELECT likely(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [-1] +... +box.execute([[SELECT likely(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT likely(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT likely(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: boolean + rows: + - [true] +... +box.execute([[SELECT likely('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT likely(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ['3'] +... +box.execute([[SELECT lower(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT lower(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT lower(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT lower(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT lower('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT lower(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT nullif(-1, -1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(true, true);]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT nullif(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [null] +... +box.execute([[SELECT position(-1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT position(1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT position(1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT position(true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT position('a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT position(X'33', X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT printf(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['-1'] +... +box.execute([[SELECT printf(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1'] +... +box.execute([[SELECT printf(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT printf(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT printf('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT printf(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT quote(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [-1] +... +box.execute([[SELECT quote(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [1] +... +box.execute([[SELECT quote(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT quote(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT quote('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['''a'''] +... +box.execute([[SELECT quote(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['X''33'''] +... +box.execute([[SELECT randomblob(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT randomblob(0);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - [null] +... +box.execute([[SELECT randomblob(0.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - [null] +... +box.execute([[SELECT randomblob(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT randomblob('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT randomblob(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[SELECT replace(-1, -1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT replace(1, 1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT replace(1.5, 1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT replace(true, true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT replace('a', 'a', 'a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT replace(X'33', X'33', X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT round(-1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT round(1, 1);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1] +... +box.execute([[SELECT round(1.5, 1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT round(true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to double' +... +box.execute([[SELECT round('a', 'a');]]) +--- +- null +- 'Type mismatch: can not convert a to double' +... +box.execute([[SELECT round(X'33', X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to double' +... +box.execute([[SELECT soundex(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT soundex(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT soundex(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT soundex(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT soundex('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['A000'] +... +box.execute([[SELECT soundex(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT substr(-1, -1, -1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT substr(1, 1, 1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT substr(1.5, 1.5, 1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT substr(true, true, true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT substr('a', 'a', 'a');]]) +--- +- null +- 'Type mismatch: can not convert a to integer' +... +box.execute([[SELECT substr(X'33', X'33', X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to integer' +... +box.execute([[SELECT typeof(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... +box.execute([[SELECT typeof(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['integer'] +... +box.execute([[SELECT typeof(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['double'] +... +box.execute([[SELECT typeof(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['boolean'] +... +box.execute([[SELECT typeof('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['string'] +... +box.execute([[SELECT typeof(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['varbinary'] +... +box.execute([[SELECT unicode(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT unicode(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT unicode(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT unicode(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT unicode('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - [97] +... +box.execute([[SELECT unicode(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT unlikely(-1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [-1] +... +box.execute([[SELECT unlikely(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT unlikely(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1.5] +... +box.execute([[SELECT unlikely(true);]]) +--- +- metadata: + - name: COLUMN_1 + type: boolean + rows: + - [true] +... +box.execute([[SELECT unlikely('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT unlikely(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ['3'] +... +box.execute([[SELECT upper(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT upper(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT upper(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT upper(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT upper('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['A'] +... +box.execute([[SELECT upper(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[SELECT zeroblob(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to unsigned' +... +box.execute([[SELECT zeroblob(1);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ["\0"] +... +box.execute([[SELECT zeroblob(1.5);]]) +--- +- metadata: + - name: COLUMN_1 + type: varbinary + rows: + - ["\0"] +... +box.execute([[SELECT zeroblob(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to unsigned' +... +box.execute([[SELECT zeroblob('a');]]) +--- +- null +- 'Type mismatch: can not convert a to unsigned' +... +box.execute([[SELECT zeroblob(X'33');]]) +--- +- null +- 'Type mismatch: can not convert varbinary to unsigned' +... +box.execute([[SELECT trim(-1);]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT trim(1);]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT trim(1.5);]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT trim(true);]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT trim('a');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT trim(X'33');]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT -1 like -1;]]) +--- +- null +- 'Type mismatch: can not convert -1 to string' +... +box.execute([[SELECT 1 like 1;]]) +--- +- null +- 'Type mismatch: can not convert 1 to string' +... +box.execute([[SELECT 1.5 like 1.5;]]) +--- +- null +- 'Type mismatch: can not convert 1.5 to string' +... +box.execute([[SELECT true like true;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to string' +... +box.execute([[SELECT 'a' like 'a';]]) +--- +- metadata: + - name: COLUMN_1 + type: boolean + rows: + - [true] +... +box.execute([[SELECT X'33' like X'33';]]) +--- +- null +- 'Type mismatch: can not convert varbinary to string' +... +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, u UNSIGNED, d DOUBLE, b BOOLEAN, s STRING, v VARBINARY);]]) +--- +- row_count: 1 +... +box.execute([[INSERT INTO t VALUES (-1, 1, 1.5, true, 'a', X'33');]]) +--- +- row_count: 1 +... +box.execute([[SELECT avg(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT avg(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT avg(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT avg(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT avg(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT avg(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT count(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT count(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: integer + rows: + - [1] +... +box.execute([[SELECT group_concat(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['-1'] +... +box.execute([[SELECT group_concat(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1'] +... +box.execute([[SELECT group_concat(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['1.5'] +... +box.execute([[SELECT group_concat(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['TRUE'] +... +box.execute([[SELECT group_concat(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['a'] +... +box.execute([[SELECT group_concat(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: string + rows: + - ['3'] +... +box.execute([[SELECT max(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT max(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT max(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT max(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT max(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT max(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT min(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [-1] +... +box.execute([[SELECT min(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1] +... +box.execute([[SELECT min(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [1.5] +... +box.execute([[SELECT min(b) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - [true] +... +box.execute([[SELECT min(s) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['a'] +... +box.execute([[SELECT min(v) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT sum(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT sum(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT sum(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT sum(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT sum(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT sum(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[SELECT total(i) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [-1] +... +box.execute([[SELECT total(u) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1] +... +box.execute([[SELECT total(d) FROM t;]]) +--- +- metadata: + - name: COLUMN_1 + type: number + rows: + - [1.5] +... +box.execute([[SELECT total(b) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert TRUE to number' +... +box.execute([[SELECT total(s) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert a to number' +... +box.execute([[SELECT total(v) FROM t;]]) +--- +- null +- 'Type mismatch: can not convert varbinary to number' +... +box.execute([[DROP TABLE t;]]) +--- +- row_count: 1 +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index fff0057bd..61483e7e9 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -629,3 +629,261 @@ box.execute([[DROP TABLE ts;]]) -- instead of values of type UNSIGNED. -- box.execute([[SELECT typeof(length('abc'));]]) + +-- Make sure the function argument types are checked. +box.execute([[SELECT abs(-1);]]) +box.execute([[SELECT abs(1);]]) +box.execute([[SELECT abs(1.5);]]) +box.execute([[SELECT abs(true);]]) +box.execute([[SELECT abs('a');]]) +box.execute([[SELECT abs(X'33');]]) + +box.execute([[SELECT char(-1);]]) +box.execute([[SELECT char(1);]]) +box.execute([[SELECT char(1.5);]]) +box.execute([[SELECT char(true);]]) +box.execute([[SELECT char('a');]]) +box.execute([[SELECT char(X'33');]]) + +box.execute([[SELECT character_length(-1);]]) +box.execute([[SELECT character_length(1);]]) +box.execute([[SELECT character_length(1.5);]]) +box.execute([[SELECT character_length(true);]]) +box.execute([[SELECT character_length('a');]]) +box.execute([[SELECT character_length(X'33');]]) + +box.execute([[SELECT char_length(-1);]]) +box.execute([[SELECT char_length(1);]]) +box.execute([[SELECT char_length(1.5);]]) +box.execute([[SELECT char_length(true);]]) +box.execute([[SELECT char_length('a');]]) +box.execute([[SELECT char_length(X'33');]]) + +box.execute([[SELECT coalesce(-1, -1);]]) +box.execute([[SELECT coalesce(1, 1);]]) +box.execute([[SELECT coalesce(1.5, 1.5);]]) +box.execute([[SELECT coalesce(true, true);]]) +box.execute([[SELECT coalesce('a', 'a');]]) +box.execute([[SELECT coalesce(X'33', X'33');]]) + +box.execute([[SELECT greatest(-1, -1);]]) +box.execute([[SELECT greatest(1, 1);]]) +box.execute([[SELECT greatest(1.5, 1.5);]]) +box.execute([[SELECT greatest(true, true);]]) +box.execute([[SELECT greatest('a', 'a');]]) +box.execute([[SELECT greatest(X'33', X'33');]]) + +box.execute([[SELECT hex(-1);]]) +box.execute([[SELECT hex(1);]]) +box.execute([[SELECT hex(1.5);]]) +box.execute([[SELECT hex(true);]]) +box.execute([[SELECT hex('a');]]) +box.execute([[SELECT hex(X'33');]]) + +box.execute([[SELECT ifnull(-1, -1);]]) +box.execute([[SELECT ifnull(1, 1);]]) +box.execute([[SELECT ifnull(1.5, 1);]]) +box.execute([[SELECT ifnull(true, true);]]) +box.execute([[SELECT ifnull('a', 'a');]]) +box.execute([[SELECT ifnull(X'33', X'33');]]) + +box.execute([[SELECT least(-1, -1);]]) +box.execute([[SELECT least(1, 1);]]) +box.execute([[SELECT least(1.5, 1.5);]]) +box.execute([[SELECT least(true, true);]]) +box.execute([[SELECT least('a', 'a');]]) +box.execute([[SELECT least(X'33', X'33');]]) + +box.execute([[SELECT length(-1);]]) +box.execute([[SELECT length(1);]]) +box.execute([[SELECT length(1.5);]]) +box.execute([[SELECT length(true);]]) +box.execute([[SELECT length('a');]]) +box.execute([[SELECT length(X'33');]]) + +box.execute([[SELECT likelihood(-1, -1);]]) +box.execute([[SELECT likelihood(1, 1);]]) +box.execute([[SELECT likelihood(1.5, 1.5);]]) +box.execute([[SELECT likelihood(true, true);]]) +box.execute([[SELECT likelihood('a', 'a');]]) +box.execute([[SELECT likelihood(X'33', X'33');]]) + +box.execute([[SELECT likely(-1);]]) +box.execute([[SELECT likely(1);]]) +box.execute([[SELECT likely(1.5);]]) +box.execute([[SELECT likely(true);]]) +box.execute([[SELECT likely('a');]]) +box.execute([[SELECT likely(X'33');]]) + +box.execute([[SELECT lower(-1);]]) +box.execute([[SELECT lower(1);]]) +box.execute([[SELECT lower(1.5);]]) +box.execute([[SELECT lower(true);]]) +box.execute([[SELECT lower('a');]]) +box.execute([[SELECT lower(X'33');]]) + +box.execute([[SELECT nullif(-1, -1);]]) +box.execute([[SELECT nullif(1, 1);]]) +box.execute([[SELECT nullif(1.5, 1.5);]]) +box.execute([[SELECT nullif(true, true);]]) +box.execute([[SELECT nullif('a', 'a');]]) +box.execute([[SELECT nullif(X'33', X'33');]]) + +box.execute([[SELECT position(-1, -1);]]) +box.execute([[SELECT position(1, 1);]]) +box.execute([[SELECT position(1.5, 1.5);]]) +box.execute([[SELECT position(true, true);]]) +box.execute([[SELECT position('a', 'a');]]) +box.execute([[SELECT position(X'33', X'33');]]) + +box.execute([[SELECT printf(-1);]]) +box.execute([[SELECT printf(1);]]) +box.execute([[SELECT printf(1.5);]]) +box.execute([[SELECT printf(true);]]) +box.execute([[SELECT printf('a');]]) +box.execute([[SELECT printf(X'33');]]) + +box.execute([[SELECT quote(-1);]]) +box.execute([[SELECT quote(1);]]) +box.execute([[SELECT quote(1.5);]]) +box.execute([[SELECT quote(true);]]) +box.execute([[SELECT quote('a');]]) +box.execute([[SELECT quote(X'33');]]) + +box.execute([[SELECT randomblob(-1);]]) +box.execute([[SELECT randomblob(0);]]) +box.execute([[SELECT randomblob(0.5);]]) +box.execute([[SELECT randomblob(true);]]) +box.execute([[SELECT randomblob('a');]]) +box.execute([[SELECT randomblob(X'33');]]) + +box.execute([[SELECT replace(-1, -1, -1);]]) +box.execute([[SELECT replace(1, 1, 1);]]) +box.execute([[SELECT replace(1.5, 1.5, 1.5);]]) +box.execute([[SELECT replace(true, true, true);]]) +box.execute([[SELECT replace('a', 'a', 'a');]]) +box.execute([[SELECT replace(X'33', X'33', X'33');]]) + +box.execute([[SELECT round(-1, -1);]]) +box.execute([[SELECT round(1, 1);]]) +box.execute([[SELECT round(1.5, 1.5);]]) +box.execute([[SELECT round(true, true);]]) +box.execute([[SELECT round('a', 'a');]]) +box.execute([[SELECT round(X'33', X'33');]]) + +box.execute([[SELECT soundex(-1);]]) +box.execute([[SELECT soundex(1);]]) +box.execute([[SELECT soundex(1.5);]]) +box.execute([[SELECT soundex(true);]]) +box.execute([[SELECT soundex('a');]]) +box.execute([[SELECT soundex(X'33');]]) + +box.execute([[SELECT substr(-1, -1, -1);]]) +box.execute([[SELECT substr(1, 1, 1);]]) +box.execute([[SELECT substr(1.5, 1.5, 1.5);]]) +box.execute([[SELECT substr(true, true, true);]]) +box.execute([[SELECT substr('a', 'a', 'a');]]) +box.execute([[SELECT substr(X'33', X'33', X'33');]]) + +box.execute([[SELECT typeof(-1);]]) +box.execute([[SELECT typeof(1);]]) +box.execute([[SELECT typeof(1.5);]]) +box.execute([[SELECT typeof(true);]]) +box.execute([[SELECT typeof('a');]]) +box.execute([[SELECT typeof(X'33');]]) + +box.execute([[SELECT unicode(-1);]]) +box.execute([[SELECT unicode(1);]]) +box.execute([[SELECT unicode(1.5);]]) +box.execute([[SELECT unicode(true);]]) +box.execute([[SELECT unicode('a');]]) +box.execute([[SELECT unicode(X'33');]]) + +box.execute([[SELECT unlikely(-1);]]) +box.execute([[SELECT unlikely(1);]]) +box.execute([[SELECT unlikely(1.5);]]) +box.execute([[SELECT unlikely(true);]]) +box.execute([[SELECT unlikely('a');]]) +box.execute([[SELECT unlikely(X'33');]]) + +box.execute([[SELECT upper(-1);]]) +box.execute([[SELECT upper(1);]]) +box.execute([[SELECT upper(1.5);]]) +box.execute([[SELECT upper(true);]]) +box.execute([[SELECT upper('a');]]) +box.execute([[SELECT upper(X'33');]]) + +box.execute([[SELECT zeroblob(-1);]]) +box.execute([[SELECT zeroblob(1);]]) +box.execute([[SELECT zeroblob(1.5);]]) +box.execute([[SELECT zeroblob(true);]]) +box.execute([[SELECT zeroblob('a');]]) +box.execute([[SELECT zeroblob(X'33');]]) + +box.execute([[SELECT trim(-1);]]) +box.execute([[SELECT trim(1);]]) +box.execute([[SELECT trim(1.5);]]) +box.execute([[SELECT trim(true);]]) +box.execute([[SELECT trim('a');]]) +box.execute([[SELECT trim(X'33');]]) + +box.execute([[SELECT -1 like -1;]]) +box.execute([[SELECT 1 like 1;]]) +box.execute([[SELECT 1.5 like 1.5;]]) +box.execute([[SELECT true like true;]]) +box.execute([[SELECT 'a' like 'a';]]) +box.execute([[SELECT X'33' like X'33';]]) + +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, u UNSIGNED, d DOUBLE, b BOOLEAN, s STRING, v VARBINARY);]]) +box.execute([[INSERT INTO t VALUES (-1, 1, 1.5, true, 'a', X'33');]]) + +box.execute([[SELECT avg(i) FROM t;]]) +box.execute([[SELECT avg(u) FROM t;]]) +box.execute([[SELECT avg(d) FROM t;]]) +box.execute([[SELECT avg(b) FROM t;]]) +box.execute([[SELECT avg(s) FROM t;]]) +box.execute([[SELECT avg(v) FROM t;]]) + +box.execute([[SELECT count(i) FROM t;]]) +box.execute([[SELECT count(u) FROM t;]]) +box.execute([[SELECT count(d) FROM t;]]) +box.execute([[SELECT count(b) FROM t;]]) +box.execute([[SELECT count(s) FROM t;]]) +box.execute([[SELECT count(v) FROM t;]]) + +box.execute([[SELECT group_concat(i) FROM t;]]) +box.execute([[SELECT group_concat(u) FROM t;]]) +box.execute([[SELECT group_concat(d) FROM t;]]) +box.execute([[SELECT group_concat(b) FROM t;]]) +box.execute([[SELECT group_concat(s) FROM t;]]) +box.execute([[SELECT group_concat(v) FROM t;]]) + +box.execute([[SELECT max(i) FROM t;]]) +box.execute([[SELECT max(u) FROM t;]]) +box.execute([[SELECT max(d) FROM t;]]) +box.execute([[SELECT max(b) FROM t;]]) +box.execute([[SELECT max(s) FROM t;]]) +box.execute([[SELECT max(v) FROM t;]]) + +box.execute([[SELECT min(i) FROM t;]]) +box.execute([[SELECT min(u) FROM t;]]) +box.execute([[SELECT min(d) FROM t;]]) +box.execute([[SELECT min(b) FROM t;]]) +box.execute([[SELECT min(s) FROM t;]]) +box.execute([[SELECT min(v) FROM t;]]) + +box.execute([[SELECT sum(i) FROM t;]]) +box.execute([[SELECT sum(u) FROM t;]]) +box.execute([[SELECT sum(d) FROM t;]]) +box.execute([[SELECT sum(b) FROM t;]]) +box.execute([[SELECT sum(s) FROM t;]]) +box.execute([[SELECT sum(v) FROM t;]]) + +box.execute([[SELECT total(i) FROM t;]]) +box.execute([[SELECT total(u) FROM t;]]) +box.execute([[SELECT total(d) FROM t;]]) +box.execute([[SELECT total(b) FROM t;]]) +box.execute([[SELECT total(s) FROM t;]]) +box.execute([[SELECT total(v) FROM t;]]) + +box.execute([[DROP TABLE t;]]) \ No newline at end of file -- 2.25.1 From imeevma at tarantool.org Fri Aug 14 18:05:15 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 14 Aug 2020 18:05:15 +0300 Subject: [Tarantool-patches] [PATCH v2 10/10] sql: refactor sql/func.c In-Reply-To: References: Message-ID: <847d1f4dd9b5b582f178e23361bbb746d2f12c04.1597417321.git.imeevma@gmail.com> After changing the way of checking the types of arguments, some of the code in sql/func.c is no longer used. This patch removes this code. Follow-up of #4159 --- src/box/sql/func.c | 841 +++++---------------------------------------- 1 file changed, 87 insertions(+), 754 deletions(-) diff --git a/src/box/sql/func.c b/src/box/sql/func.c index c289f2de0..de1e4d13d 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -504,44 +504,21 @@ absFunc(sql_context * context, int argc, sql_value ** argv) { assert(argc == 1); UNUSED_PARAMETER(argc); - switch (sql_value_type(argv[0])) { - case MP_UINT: { - sql_result_uint(context, sql_value_uint64(argv[0])); - break; - } - case MP_INT: { + enum mp_type type = sql_value_type(argv[0]); + if (type == MP_NIL) + return sql_result_null(context); + if (type == MP_UINT) + return sql_result_uint(context, sql_value_uint64(argv[0])); + if (type == MP_INT) { int64_t value = sql_value_int64(argv[0]); assert(value < 0); - sql_result_uint(context, -value); - break; - } - case MP_NIL:{ - /* IMP: R-37434-19929 Abs(X) returns NULL if X is NULL. */ - sql_result_null(context); - break; - } - case MP_BOOL: - case MP_BIN: - case MP_ARRAY: - case MP_MAP: { - diag_set(ClientError, ER_INCONSISTENT_TYPES, "number", - mem_type_to_str(argv[0])); - context->is_aborted = true; - return; - } - default:{ - /* Because sql_value_double() returns 0.0 if the argument is not - * something that can be converted into a number, we have: - * IMP: R-01992-00519 Abs(X) returns 0.0 if X is a string or blob - * that cannot be converted to a numeric value. - */ - double rVal = sql_value_double(argv[0]); - if (rVal < 0) - rVal = -rVal; - sql_result_double(context, rVal); - break; - } + return sql_result_uint(context, -value); } + assert(type == MP_DOUBLE); + double value = sql_value_double(argv[0]); + if (value < 0) + value = -value; + return sql_result_double(context, value); } /** @@ -839,19 +816,14 @@ roundFunc(sql_context * context, int argc, sql_value ** argv) if (argc == 2) { if (sql_value_is_null(argv[1])) return; + assert(sql_value_type(argv[1]) == MP_UINT); n = sql_value_int(argv[1]); if (n < 0) n = 0; } if (sql_value_is_null(argv[0])) return; - enum mp_type mp_type = sql_value_type(argv[0]); - if (mp_type_is_bloblike(mp_type)) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "numeric"); - context->is_aborted = true; - return; - } + assert(sql_value_type(argv[0]) == MP_DOUBLE); r = sql_value_double(argv[0]); /* If Y==0 and X will fit in a 64-bit int, * handle the rounding directly, @@ -988,12 +960,7 @@ randomBlob(sql_context * context, int argc, sql_value ** argv) unsigned char *p; assert(argc == 1); UNUSED_PARAMETER(argc); - if (mp_type_is_bloblike(sql_value_type(argv[0]))) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "numeric"); - context->is_aborted = true; - return; - } + assert(sql_value_type(argv[0]) == MP_UINT); n = sql_value_int(argv[0]); if (n < 1) return; @@ -1538,6 +1505,7 @@ zeroblobFunc(sql_context * context, int argc, sql_value ** argv) i64 n; assert(argc == 1); UNUSED_PARAMETER(argc); + assert(sql_value_type(argv[0]) == MP_UINT); n = sql_value_int64(argv[0]); if (n < 0) n = 0; @@ -1943,18 +1911,10 @@ sum_step(struct sql_context *context, int argc, sql_value **argv) assert(argc == 1); UNUSED_PARAMETER(argc); struct SumCtx *p = sql_aggregate_context(context, sizeof(*p)); - int type = sql_value_type(argv[0]); + enum mp_type type = sql_value_type(argv[0]); if (type == MP_NIL || p == NULL) return; - if (type != MP_DOUBLE && type != MP_INT && type != MP_UINT) { - if (mem_apply_numeric_type(argv[0]) != 0) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(argv[0]), "number"); - context->is_aborted = true; - return; - } - type = sql_value_type(argv[0]); - } + assert(type == MP_DOUBLE || type == MP_INT || type == MP_UINT); p->cnt++; if (type == MP_INT || type == MP_UINT) { int64_t v = sql_value_int64(argv[0]); @@ -2229,703 +2189,76 @@ static struct { uint16_t flags; void (*call)(sql_context *ctx, int argc, sql_value **argv); void (*finalize)(sql_context *ctx); - /** Members below are related to struct func_def. */ - bool is_deterministic; - int param_count; - enum field_type returns; - enum func_aggregate aggregate; - bool export_to_sql; } sql_builtins[] = { - {.name = "ABS", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = absFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "AVG", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .is_deterministic = false, - .aggregate = FUNC_AGGREGATE_GROUP, - .flags = 0, - .call = sum_step, - .finalize = avgFinalize, - .export_to_sql = true, - }, { - .name = "CEIL", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CEILING", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CHAR", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .is_deterministic = true, - .aggregate = FUNC_AGGREGATE_NONE, - .flags = 0, - .call = charFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "CHARACTER_LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "CHAR_LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "COALESCE", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_COALESCE, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "COUNT", - .param_count = -1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = countStep, - .finalize = countFinalize, - .export_to_sql = true, - }, { - .name = "CURRENT_DATE", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CURRENT_TIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "CURRENT_TIMESTAMP", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "DATE", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "DATETIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EVERY", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXISTS", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXP", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "EXTRACT", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "FLOOR", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "GREATER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "GREATEST", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, - .call = minmaxFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "GROUP_CONCAT", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = groupConcatStep, - .finalize = groupConcatFinalize, - .export_to_sql = true, - }, { - .name = "HEX", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = hexFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "IFNULL", - .param_count = 2, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_COALESCE, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "JULIANDAY", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LEAST", - .param_count = -1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, - .call = minmaxFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LENGTH", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_LENGTH, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LENGTH_VARBINARY", - .param_count = 1, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_LENGTH, - .call = lengthFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LESSER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LIKE", - .param_count = -1, - .returns = FIELD_TYPE_BOOLEAN, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_LIKE, - .call = likeFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LIKELIHOOD", - .param_count = 2, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LIKELY", - .param_count = 1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "LN", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "LOWER", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, - .call = LowerICUFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "MAX", - .param_count = 1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, - .call = minmaxStep, - .finalize = minMaxFinalize, - .export_to_sql = true, - }, { - .name = "MIN", - .param_count = 1, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, - .call = minmaxStep, - .finalize = minMaxFinalize, - .export_to_sql = true, - }, { - .name = "MOD", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "NULLIF", - .param_count = 2, - .returns = FIELD_TYPE_SCALAR, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL, - .call = nullifFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "OCTET_LENGTH", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "POSITION", - .param_count = 2, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_NEEDCOLL, - .call = position_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "POSITION_VARBINARY", - .param_count = 2, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = position_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "POWER", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "PRINTF", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = printfFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "QUOTE", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = quoteFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "RANDOM", - .param_count = 0, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .call = randomFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "RANDOMBLOB", - .param_count = 1, - .returns = FIELD_TYPE_VARBINARY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .call = randomBlob, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "REPLACE", - .param_count = 3, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = replaceFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ROUND", - .param_count = -1, - .returns = FIELD_TYPE_DOUBLE, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = roundFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ROW_COUNT", - .param_count = 0, - .returns = FIELD_TYPE_INTEGER, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = sql_row_count, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SOME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "SOUNDEX", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = soundexFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SQRT", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "STRFTIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "SUBSTR", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = substrFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SUBSTR_VARBINARY", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = substrFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "SUM", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = sum_step, - .finalize = sumFinalize, - .export_to_sql = true, - }, { - .name = "TIME", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "TOTAL", - .param_count = 1, - .returns = FIELD_TYPE_NUMBER, - .aggregate = FUNC_AGGREGATE_GROUP, - .is_deterministic = false, - .flags = 0, - .call = sum_step, - .finalize = totalFinalize, - .export_to_sql = true, - }, { - .name = "TRIM", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = trim_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "TRIM_VARBINARY", - .param_count = -1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL, - .call = trim_func, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "TYPEOF", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_TYPEOF, - .call = typeofFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UNICODE", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = unicodeFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UNLIKELY", - .param_count = 1, - .returns = FIELD_TYPE_BOOLEAN, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_UNLIKELY, - .call = sql_builtin_stub, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "UPPER", - .param_count = 1, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, - .call = UpperICUFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "VERSION", - .param_count = 0, - .returns = FIELD_TYPE_STRING, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = sql_func_version, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "ZEROBLOB", - .param_count = 1, - .returns = FIELD_TYPE_VARBINARY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = true, - .flags = 0, - .call = zeroblobFunc, - .finalize = NULL, - .export_to_sql = true, - }, { - .name = "_sql_stat_get", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "_sql_stat_init", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, { - .name = "_sql_stat_push", - .call = sql_builtin_stub, - .export_to_sql = false, - .param_count = -1, - .returns = FIELD_TYPE_ANY, - .aggregate = FUNC_AGGREGATE_NONE, - .is_deterministic = false, - .flags = 0, - .finalize = NULL, - }, + {"ABS", 0, absFunc, NULL}, + {"AVG", 0, sum_step, avgFinalize}, + {"CEIL", 0, sql_builtin_stub, NULL}, + {"CEILING", 0, sql_builtin_stub, NULL}, + {"CHAR", 0, charFunc, NULL}, + {"CHARACTER_LENGTH", 0, lengthFunc, NULL}, + {"CHAR_LENGTH", 0, lengthFunc, NULL}, + {"COALESCE", SQL_FUNC_COALESCE, sql_builtin_stub, NULL}, + {"COUNT", 0, countStep, countFinalize}, + {"CURRENT_DATE", 0, sql_builtin_stub, NULL}, + {"CURRENT_TIME", 0, sql_builtin_stub, NULL}, + {"CURRENT_TIMESTAMP", 0, sql_builtin_stub, NULL}, + {"DATE", 0, sql_builtin_stub, NULL}, + {"DATETIME", 0, sql_builtin_stub, NULL}, + {"EVERY", 0, sql_builtin_stub, NULL}, + {"EXISTS", 0, sql_builtin_stub, NULL}, + {"EXP", 0, sql_builtin_stub, NULL}, + {"EXTRACT", 0, sql_builtin_stub, NULL}, + {"FLOOR", 0, sql_builtin_stub, NULL}, + {"GREATER", 0, sql_builtin_stub, NULL}, + {"GREATEST", SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, minmaxFunc, NULL}, + {"GROUP_CONCAT", 0, groupConcatStep, groupConcatFinalize}, + {"HEX", 0, hexFunc, NULL}, + {"IFNULL", SQL_FUNC_COALESCE, sql_builtin_stub, NULL}, + {"JULIANDAY", 0, sql_builtin_stub, NULL}, + {"LEAST", SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, minmaxFunc, NULL}, + {"LENGTH", SQL_FUNC_LENGTH, lengthFunc, NULL}, + {"LENGTH_VARBINARY", SQL_FUNC_LENGTH, lengthFunc, NULL}, + {"LESSER", 0, sql_builtin_stub, NULL}, + {"LIKE", SQL_FUNC_NEEDCOLL | SQL_FUNC_LIKE, likeFunc, NULL}, + {"LIKELIHOOD", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"LIKELY", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"LN", 0, sql_builtin_stub, NULL}, + {"LOWER", SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, LowerICUFunc, NULL}, + {"MAX", SQL_FUNC_NEEDCOLL | SQL_FUNC_MAX, minmaxStep, minMaxFinalize}, + {"MIN", SQL_FUNC_NEEDCOLL | SQL_FUNC_MIN, minmaxStep, minMaxFinalize}, + {"MOD", 0, sql_builtin_stub, NULL}, + {"NULLIF", SQL_FUNC_NEEDCOLL, nullifFunc, NULL}, + {"OCTET_LENGTH", 0, sql_builtin_stub, NULL}, + {"POSITION", SQL_FUNC_NEEDCOLL, position_func, NULL}, + {"POSITION_VARBINARY", SQL_FUNC_NEEDCOLL, position_func, NULL}, + {"POWER", 0, sql_builtin_stub, NULL}, + {"PRINTF", 0, printfFunc, NULL}, + {"QUOTE", 0, quoteFunc, NULL}, + {"RANDOM", 0, randomFunc, NULL}, + {"RANDOMBLOB", 0, randomBlob, NULL}, + {"REPLACE", SQL_FUNC_DERIVEDCOLL, replaceFunc, NULL}, + {"ROUND", 0, roundFunc, NULL}, + {"ROW_COUNT", 0, sql_row_count, NULL}, + {"SOME", 0, sql_builtin_stub, NULL}, + {"SOUNDEX", 0, soundexFunc, NULL}, + {"SQRT", 0, sql_builtin_stub, NULL}, + {"STRFTIME", 0, sql_builtin_stub, NULL}, + {"SUBSTR", SQL_FUNC_DERIVEDCOLL, substrFunc, NULL}, + {"SUBSTR_VARBINARY", SQL_FUNC_DERIVEDCOLL, substrFunc, NULL}, + {"SUM", 0, sum_step, sumFinalize}, + {"TIME", 0, sql_builtin_stub, NULL}, + {"TOTAL", 0, sum_step, totalFinalize}, + {"TRIM", SQL_FUNC_DERIVEDCOLL, trim_func, NULL}, + {"TRIM_VARBINARY", SQL_FUNC_DERIVEDCOLL, trim_func, NULL}, + {"TYPEOF", SQL_FUNC_TYPEOF, typeofFunc, NULL}, + {"UNICODE", 0, unicodeFunc, NULL}, + {"UNLIKELY", SQL_FUNC_UNLIKELY, sql_builtin_stub, NULL}, + {"UPPER", SQL_FUNC_DERIVEDCOLL | SQL_FUNC_NEEDCOLL, UpperICUFunc, NULL}, + {"VERSION", 0, sql_func_version, NULL}, + {"ZEROBLOB", 0, zeroblobFunc, NULL}, + {"_sql_stat_get", 0, sql_builtin_stub, NULL}, + {"_sql_stat_init", 0, sql_builtin_stub, NULL}, + {"_sql_stat_push", 0, sql_builtin_stub, NULL}, }; static struct func_vtab func_sql_builtin_vtab; -- 2.25.1 From gorcunov at gmail.com Sat Aug 15 00:14:34 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:34 +0300 Subject: [Tarantool-patches] [PATCH v7 0/8] qsync: write CONFIRM/ROLLBACK without txn engine Message-ID: <20200814211442.667099-1-gorcunov@gmail.com> In this series we write CONFIRM/ROLLBACK messages into the WAL directly without involving the txn engine. Vlad, take a look please, once time permit. Note this branch carries yours "xrow: introduce struct synchro_request" which I fetched and applier manually since it wasn't yet in master branch. First 4 patches you've read already and hopefully I addressed all your comments. issue https://github.com/tarantool/tarantool/issues/5129 branch gorcunov/gh-5129-journal-7 v3: - bootstrap journal left NULL for async write - journal_write_async_cb_t type for async callback - struct synchro_body_bin type for encoded message - xrow_encode_synchro helper to operate with synchro_body_bin v7: - rebase on master - rework applier code Cyrill Gorcunov (8): journal: bind asynchronous write completion to an entry journal: add journal_entry_create helper qsync: provide a binary form of syncro entries qsync: direct write of CONFIRM/ROLLBACK into a journal applier: factor out latch locking applier: add shorthands to queue access applier: process synchro requests without txn engine applier: drop process_synchro_row src/box/applier.cc | 295 ++++++++++++++++++++++++++++++++++---------- src/box/box.cc | 15 +-- src/box/journal.c | 8 +- src/box/journal.h | 36 ++++-- src/box/txn.c | 2 +- src/box/txn_limbo.c | 69 ++++++----- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++- src/box/wal.h | 4 +- src/box/xrow.c | 41 +++--- src/box/xrow.h | 20 ++- 11 files changed, 351 insertions(+), 160 deletions(-) -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:35 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:35 +0300 Subject: [Tarantool-patches] [PATCH v7 1/8] journal: bind asynchronous write completion to an entry In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-2-gorcunov@gmail.com> In commit 77ba0e3504464131fe81c672d508d0275be2173a we've redesigned wal journal operations such that asynchronous write completion is a single instance per journal. It turned out that such simplification is too tight and doesn't allow us to pass entries into the journal with custom completions. Thus lets allow back such ability. We will need it to be able to write "confirm" records into wal directly without touching transactions code at all. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/box.cc | 15 ++++++++------- src/box/journal.c | 2 ++ src/box/journal.h | 20 +++++++++++--------- src/box/txn.c | 2 +- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++++++++----------- src/box/wal.h | 4 ++-- 7 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index 8e811e9c1..faffd5769 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -348,7 +348,7 @@ recovery_journal_write(struct journal *base, * Since there're no actual writes, fire a * journal_async_complete callback right away. */ - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -357,7 +357,7 @@ recovery_journal_create(struct vclock *v) { static struct recovery_journal journal; journal_create(&journal.base, recovery_journal_write, - txn_complete_async, recovery_journal_write); + recovery_journal_write); journal.vclock = v; journal_set(&journal.base); } @@ -2182,8 +2182,10 @@ engine_init() static int bootstrap_journal_write(struct journal *base, struct journal_entry *entry) { + (void)base; + entry->res = 0; - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -2569,8 +2571,8 @@ box_cfg_xc(void) int64_t wal_max_size = box_check_wal_max_size(cfg_geti64("wal_max_size")); enum wal_mode wal_mode = box_check_wal_mode(cfg_gets("wal_mode")); - if (wal_init(wal_mode, txn_complete_async, cfg_gets("wal_dir"), - wal_max_size, &INSTANCE_UUID, on_wal_garbage_collection, + if (wal_init(wal_mode, cfg_gets("wal_dir"), wal_max_size, + &INSTANCE_UUID, on_wal_garbage_collection, on_wal_checkpoint_threshold) != 0) { diag_raise(); } @@ -2617,8 +2619,7 @@ box_cfg_xc(void) } struct journal bootstrap_journal; - journal_create(&bootstrap_journal, NULL, txn_complete_async, - bootstrap_journal_write); + journal_create(&bootstrap_journal, NULL, bootstrap_journal_write); journal_set(&bootstrap_journal); auto bootstrap_journal_guard = make_scoped_guard([] { journal_set(NULL); diff --git a/src/box/journal.c b/src/box/journal.c index f1e89aaa2..48af9157b 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -36,6 +36,7 @@ struct journal *current_journal = NULL; struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data) { struct journal_entry *entry; @@ -50,6 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } + entry->write_async_cb = write_async_cb; entry->complete_data = complete_data; entry->approx_len = 0; entry->n_rows = n_rows; diff --git a/src/box/journal.h b/src/box/journal.h index 1a10e66c3..4b019fecf 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -42,6 +42,8 @@ extern "C" { struct xrow_header; struct journal_entry; +typedef void (*journal_write_async_f)(struct journal_entry *entry); + /** * An entry for an abstract journal. * Simply put, a write ahead log request. @@ -61,6 +63,10 @@ struct journal_entry { * A journal entry completion callback argument. */ void *complete_data; + /** + * Asynchronous write completion function. + */ + journal_write_async_f write_async_cb; /** * Approximate size of this request when encoded. */ @@ -84,6 +90,7 @@ struct region; */ struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data); /** @@ -96,22 +103,19 @@ struct journal { int (*write_async)(struct journal *journal, struct journal_entry *entry); - /** Asynchronous write completion */ - void (*write_async_cb)(struct journal_entry *entry); - /** Synchronous write */ int (*write)(struct journal *journal, struct journal_entry *entry); }; /** - * Finalize a single entry. + * Complete asynchronous write. */ static inline void -journal_async_complete(struct journal *journal, struct journal_entry *entry) +journal_async_complete(struct journal_entry *entry) { - assert(journal->write_async_cb != NULL); - journal->write_async_cb(entry); + assert(entry->write_async_cb != NULL); + entry->write_async_cb(entry); } /** @@ -173,12 +177,10 @@ static inline void journal_create(struct journal *journal, int (*write_async)(struct journal *journal, struct journal_entry *entry), - void (*write_async_cb)(struct journal_entry *entry), int (*write)(struct journal *journal, struct journal_entry *entry)) { journal->write_async = write_async; - journal->write_async_cb = write_async_cb; journal->write = write; } diff --git a/src/box/txn.c b/src/box/txn.c index 9c21258c5..cc1f496c5 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -551,7 +551,7 @@ txn_journal_entry_new(struct txn *txn) /* Save space for an additional NOP row just in case. */ req = journal_entry_new(txn->n_new_rows + txn->n_applier_rows + 1, - &txn->region, txn); + &txn->region, txn_complete_async, txn); if (req == NULL) return NULL; diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..de4c5205c 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -818,7 +818,7 @@ vy_log_tx_flush(struct vy_log_tx *tx) size_t used = region_used(&fiber()->gc); struct journal_entry *entry; - entry = journal_entry_new(tx_size, &fiber()->gc, NULL); + entry = journal_entry_new(tx_size, &fiber()->gc, NULL, NULL); if (entry == NULL) goto err; diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..045006b60 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -266,10 +266,9 @@ xlog_write_entry(struct xlog *l, struct journal_entry *entry) static void tx_schedule_queue(struct stailq *queue) { - struct wal_writer *writer = &wal_writer_singleton; struct journal_entry *req, *tmp; stailq_foreach_entry_safe(req, tmp, queue, fifo) - journal_async_complete(&writer->base, req); + journal_async_complete(req); } /** @@ -403,9 +402,8 @@ tx_notify_checkpoint(struct cmsg *msg) */ static void wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, - void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, - int64_t wal_max_size, const struct tt_uuid *instance_uuid, + const char *wal_dirname, int64_t wal_max_size, + const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { @@ -415,7 +413,6 @@ wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, journal_create(&writer->base, wal_mode == WAL_NONE ? wal_write_none_async : wal_write_async, - wall_async_cb, wal_mode == WAL_NONE ? wal_write_none : wal_write); @@ -525,15 +522,15 @@ wal_open(struct wal_writer *writer) } int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { /* Initialize the state. */ struct wal_writer *writer = &wal_writer_singleton; - wal_writer_create(writer, wal_mode, wall_async_cb, wal_dirname, - wal_max_size, instance_uuid, on_garbage_collection, + wal_writer_create(writer, wal_mode, wal_dirname, wal_max_size, + instance_uuid, on_garbage_collection, on_checkpoint_threshold); /* Start WAL thread. */ @@ -1314,7 +1311,7 @@ wal_write_none_async(struct journal *journal, vclock_merge(&writer->vclock, &vclock_diff); vclock_copy(&replicaset.vclock, &writer->vclock); entry->res = vclock_sum(&writer->vclock); - journal_async_complete(journal, entry); + journal_async_complete(entry); return 0; } diff --git a/src/box/wal.h b/src/box/wal.h index f348dc636..9d0cada46 100644 --- a/src/box/wal.h +++ b/src/box/wal.h @@ -81,8 +81,8 @@ typedef void (*wal_on_checkpoint_threshold_f)(void); * Start WAL thread and initialize WAL writer. */ int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold); -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:36 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:36 +0300 Subject: [Tarantool-patches] [PATCH v7 2/8] journal: add journal_entry_create helper In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-3-gorcunov@gmail.com> To create raw journal entries. We will use it to write confirm/rollback entries. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/journal.c | 8 ++------ src/box/journal.h | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/src/box/journal.c b/src/box/journal.c index 48af9157b..cb320b557 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -51,11 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } - entry->write_async_cb = write_async_cb; - entry->complete_data = complete_data; - entry->approx_len = 0; - entry->n_rows = n_rows; - entry->res = -1; - + journal_entry_create(entry, n_rows, 0, write_async_cb, + complete_data); return entry; } diff --git a/src/box/journal.h b/src/box/journal.h index 4b019fecf..5d8d5a726 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -83,6 +83,22 @@ struct journal_entry { struct region; +/** + * Initialize a new journal entry. + */ +static inline void +journal_entry_create(struct journal_entry *entry, size_t n_rows, + size_t approx_len, + journal_write_async_f write_async_cb, + void *complete_data) +{ + entry->write_async_cb = write_async_cb; + entry->complete_data = complete_data; + entry->approx_len = approx_len; + entry->n_rows = n_rows; + entry->res = -1; +} + /** * Create a new journal entry. * -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:37 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:37 +0300 Subject: [Tarantool-patches] [PATCH v7 3/8] qsync: provide a binary form of syncro entries In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-4-gorcunov@gmail.com> These msgpack entries will be needed to write them down to a journal without involving txn engine. Same time we would like to be able to allocate them on stack, for this sake the binary form is predefined. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 9 +++++++-- src/box/xrow.c | 41 ++++++++++++++++++----------------------- src/box/xrow.h | 20 +++++++++++++++----- 3 files changed, 40 insertions(+), 30 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index 944161c30..ed8c10419 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -282,6 +282,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) req.replica_id = limbo->instance_id; req.lsn = lsn; + /* + * This is a synchronous commit so we can + * use body and row allocated on a stack. + */ + struct synchro_body_bin body; struct xrow_header row; struct request request = { .header = &row, @@ -291,8 +296,8 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) if (txn == NULL) goto rollback; - if (xrow_encode_synchro(&row, &txn->region, &req) != 0) - goto rollback; + xrow_encode_synchro(&row, &body, &req); + /* * This is not really a transaction. It just uses txn API * to put the data into WAL. And obviously it should not diff --git a/src/box/xrow.c b/src/box/xrow.c index 4b5d4356f..03a4abdda 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,35 +893,30 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req) { - size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(req->replica_id) + - mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); - char *buf = (char *)region_alloc(region, len); - if (buf == NULL) { - diag_set(OutOfMemory, len, "region_alloc", "buf"); - return -1; - } - char *pos = buf; - - pos = mp_encode_map(pos, 2); - pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, req->replica_id); - pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, req->lsn); + /* + * A map with two elements. We don't compress + * numbers to have this structure constant in size, + * which allows us to preallocate it on stack. + */ + body->m_body = 0x80 | 2; + body->k_replica_id = IPROTO_REPLICA_ID; + body->m_replica_id = 0xce; + body->v_replica_id = mp_bswap_u32(req->replica_id); + body->k_lsn = IPROTO_LSN; + body->m_lsn = 0xcf; + body->v_lsn = mp_bswap_u64(req->lsn); memset(row, 0, sizeof(*row)); - row->body[0].iov_base = buf; - row->body[0].iov_len = len; - row->bodycnt = 1; - row->type = req->type; - - return 0; + row->body[0].iov_base = (void *)body; + row->body[0].iov_len = sizeof(*body); + row->bodycnt = 1; } int diff --git a/src/box/xrow.h b/src/box/xrow.h index 02dca74e5..20e82034d 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -240,16 +240,26 @@ struct synchro_request { int64_t lsn; }; +/** Synchro request xrow's body in MsgPack format. */ +struct PACKED synchro_body_bin { + uint8_t m_body; + uint8_t k_replica_id; + uint8_t m_replica_id; + uint32_t v_replica_id; + uint8_t k_lsn; + uint8_t m_lsn; + uint64_t v_lsn; +}; + /** * Encode synchronous replication request. * @param row xrow header. - * @param region Region to use to encode the confirmation body. + * @param body Desination to use to encode the confirmation body. * @param req Request parameters. - * @retval -1 on error. - * @retval 0 success. */ -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req); /** -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:38 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:38 +0300 Subject: [Tarantool-patches] [PATCH v7 4/8] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-5-gorcunov@gmail.com> When we need to write CONFIRM or ROLLBACK message (which is a binary record in msgpack format) into a journal we use txn code to allocate a new transaction, encode there a message and pass it to walk the long txn path before it hit the journal. This is not only resource wasting but also somehow strange from architectural point of view. Instead lets encode a record on the stack and write it to the journal directly. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 64 ++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index ed8c10419..447630d23 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -32,6 +32,7 @@ #include "txn_limbo.h" #include "replication.h" #include "iproto_constants.h" +#include "journal.h" struct txn_limbo txn_limbo; @@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) return 0; } +/** + * A callback for synchronous write: txn_limbo_write_synchro fiber + * waiting to proceed once a record is written to WAL. + */ +static void +txn_limbo_write_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + fiber_wakeup(entry->complete_data); +} + static void txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { @@ -284,46 +296,34 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) /* * This is a synchronous commit so we can - * use body and row allocated on a stack. + * allocate everything on a stack. */ struct synchro_body_bin body; struct xrow_header row; - struct request request = { - .header = &row, - }; + char buf[sizeof(struct journal_entry) + + sizeof(struct xrow_header *)]; - struct txn *txn = txn_begin(); - if (txn == NULL) - goto rollback; + struct journal_entry *entry = (struct journal_entry *)buf; + entry->rows[0] = &row; xrow_encode_synchro(&row, &body, &req); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - goto rollback; - if (txn_commit_stmt(txn, &request) != 0) - goto rollback; - if (txn_commit(txn) != 0) - goto rollback; - return; + journal_entry_create(entry, 1, xrow_approx_len(&row), + txn_limbo_write_cb, fiber()); -rollback: - /* - * XXX: the stub is supposed to be removed once it is defined what to do - * when a synchro request WAL write fails. One of the possible - * solutions: log the error, keep the limbo queue as is and probably put - * in rollback mode. Then provide a hook to call manually when WAL - * problems are fixed. Or retry automatically with some period. - */ - panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, iproto_type_name(type)); + if (journal_write(entry) != 0 || entry->res < 0) { + diag_set(ClientError, ER_WAL_IO); + diag_log(); + /* + * XXX: the stub is supposed to be removed once it is defined what to do + * when a synchro request WAL write fails. One of the possible + * solutions: log the error, keep the limbo queue as is and probably put + * in rollback mode. Then provide a hook to call manually when WAL + * problems are fixed. Or retry automatically with some period. + */ + panic("Could not write a synchro request to WAL: lsn = %lld, type = " + "%s\n", lsn, type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); + } } /** -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:39 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:39 +0300 Subject: [Tarantool-patches] [PATCH v7 5/8] applier: factor out latch locking In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-6-gorcunov@gmail.com> We will need to reuse this helpers. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 43 +++++++++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 98fb87375..60689f6d3 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -799,6 +799,29 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) return 0; } +/* + * In a full mesh topology, the same set of changes + * may arrive via two concurrently running appliers. + * Hence we need a latch to strictly order all changes + * that belong to the same server id. + */ +static inline struct latch * +applier_lock(uint32_t replica_id) +{ + struct replica *replica = replica_by_id(replica_id); + struct latch *latch = (replica ? &replica->order_latch : + &replicaset.applier.order_latch); + latch_lock(latch); + return latch; +} + +static inline void +applier_unlock(struct latch *latch) +{ + assert(latch != NULL); + latch_unlock(latch); +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -811,19 +834,11 @@ applier_apply_tx(struct stailq *rows) struct applier_tx_row, next)->row; struct xrow_header *last_row; last_row = &stailq_last_entry(rows, struct applier_tx_row, next)->row; - struct replica *replica = replica_by_id(first_row->replica_id); - /* - * In a full mesh topology, the same set of changes - * may arrive via two concurrently running appliers. - * Hence we need a latch to strictly order all changes - * that belong to the same server id. - */ - struct latch *latch = (replica ? &replica->order_latch : - &replicaset.applier.order_latch); - latch_lock(latch); + struct latch *latch = applier_lock(first_row->replica_id); + if (vclock_get(&replicaset.applier.vclock, last_row->replica_id) >= last_row->lsn) { - latch_unlock(latch); + applier_unlock(latch); return 0; } else if (vclock_get(&replicaset.applier.vclock, first_row->replica_id) >= first_row->lsn) { @@ -855,7 +870,7 @@ applier_apply_tx(struct stailq *rows) struct txn *txn = txn_begin(); struct applier_tx_row *item; if (txn == NULL) { - latch_unlock(latch); + applier_unlock(latch); return -1; } stailq_foreach_entry(item, rows, next) { @@ -930,12 +945,12 @@ applier_apply_tx(struct stailq *rows) */ vclock_follow(&replicaset.applier.vclock, last_row->replica_id, last_row->lsn); - latch_unlock(latch); + applier_unlock(latch); return 0; rollback: txn_rollback(txn); fail: - latch_unlock(latch); + applier_unlock(latch); fiber_gc(); return -1; } -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:40 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:40 +0300 Subject: [Tarantool-patches] [PATCH v7 6/8] applier: add shorthands to queue access In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-7-gorcunov@gmail.com> We need to access first and last xrow in a queue frenquently and opencoded variants are too ugly. Lets provide shorthands (we will reuse them in qsync packets handling as well). Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 60689f6d3..a71516282 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -648,6 +648,26 @@ struct applier_tx_row { struct xrow_header row; }; +/** + * Get first xrow from a list. + */ +static inline struct xrow_header * +applier_first_row(struct stailq *rows) +{ + return &stailq_first_entry(rows, + struct applier_tx_row, next)->row; +} + +/** + * Get last xrow from a list. + */ +static inline struct xrow_header * +applier_last_row(struct stailq *rows) +{ + return &stailq_last_entry(rows, + struct applier_tx_row, next)->row; +} + static struct applier_tx_row * applier_read_tx_row(struct applier *applier) { @@ -749,8 +769,7 @@ applier_read_tx(struct applier *applier, struct stailq *rows) } stailq_add_tail(rows, &tx_row->next); - } while (!stailq_last_entry(rows, struct applier_tx_row, - next)->row.is_commit); + } while (!applier_last_row(rows)->is_commit); } static int @@ -830,10 +849,8 @@ applier_unlock(struct latch *latch) static int applier_apply_tx(struct stailq *rows) { - struct xrow_header *first_row = &stailq_first_entry(rows, - struct applier_tx_row, next)->row; - struct xrow_header *last_row; - last_row = &stailq_last_entry(rows, struct applier_tx_row, next)->row; + struct xrow_header *first_row = applier_first_row(rows); + struct xrow_header *last_row = applier_last_row(rows); struct latch *latch = applier_lock(first_row->replica_id); if (vclock_get(&replicaset.applier.vclock, @@ -849,9 +866,7 @@ applier_apply_tx(struct stailq *rows) */ struct xrow_header *tmp; while (true) { - tmp = &stailq_first_entry(rows, - struct applier_tx_row, - next)->row; + tmp = applier_first_row(rows); if (tmp->lsn <= vclock_get(&replicaset.applier.vclock, tmp->replica_id)) { stailq_shift(rows); @@ -1133,12 +1148,12 @@ applier_subscribe(struct applier *applier) applier_read_tx(applier, &rows); applier->last_row_time = ev_monotonic_now(loop()); + /* * In case of an heartbeat message wake a writer up * and check applier state. */ - if (stailq_first_entry(&rows, struct applier_tx_row, - next)->row.lsn == 0) + if (applier_first_row(&rows)->lsn == 0) applier_signal_ack(applier); else if (applier_apply_tx(&rows) != 0) diag_raise(); -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:41 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:41 +0300 Subject: [Tarantool-patches] [PATCH v7 7/8] applier: process synchro requests without txn engine In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-8-gorcunov@gmail.com> Transaction processing code is very heavy simply because trasactions are carrying various data and involves a number of other mechanisms to procceed. In turn, when we receive confirm or rollback packed from another node in a cluster we just need to inspect limbo queue and write this packed into a WAL journal. So calling a bunch of txn engine helpers is simply waste of cycles. Thus lets rather handle them in a special light way: - allocate synchro_entry structure which would carry the journal entry itself and encoded message - process limbo queue to mark confirmed/rollback'ed messages - finally write this synchro_entry into a journal Which is a way more simplier. Part-of #5129 Suggedsted-by: Vladislav Shpilevoy Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 179 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 172 insertions(+), 7 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index a71516282..a1ce7a23f 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -51,8 +51,10 @@ #include "schema.h" #include "txn.h" #include "box.h" +#include "xrow.h" #include "scoped_guard.h" #include "txn_limbo.h" +#include "journal.h" STRS(applier_state, applier_STATE); @@ -841,6 +843,151 @@ applier_unlock(struct latch *latch) latch_unlock(latch); } +struct synchro_entry { + /** An applier initiated the syncho request. */ + struct applier *applier; + + /** Encoded form of a synchro record. */ + struct synchro_body_bin body_bin; + + /** xrow to write, used by the journal engine. */ + struct xrow_header row; + + /** + * The journal entry itself. Note since + * it has unsized array it must be the + * last entry in the structure. + */ + struct journal_entry journal_entry; +}; + +static void +synchro_entry_delete(struct synchro_entry *entry) +{ + free(entry); +} + +/** + * Async write journal completion. + */ +static void +apply_synchro_row_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + struct synchro_entry *synchro_entry = + (struct synchro_entry *)entry->complete_data; + struct applier *applier = synchro_entry->applier; + + /* + * We can reuse triggers, they are allocated when + * applier get subscribed and since packets handling + * is processed after the subscribtion phase the triggers + * will be alive. + */ + if (entry->res < 0) { + trigger_run(&replicaset.applier.on_rollback, applier); + /* + * Restore the last written vlock value. + */ + vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); + diag_set(ClientError, ER_WAL_IO); + diag_log(); + } else { + trigger_run(&replicaset.applier.on_wal_write, applier); + } + + synchro_entry_delete(synchro_entry); +} + +/** + * Allocate a new synchro_entry to be passed to + * the journal engine in async write way. + */ +static struct synchro_entry * +synchro_entry_new(struct applier *applier, + struct xrow_header *applier_row, + struct synchro_request *req) +{ + struct synchro_entry *entry; + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); + + /* + * For simplicity we use malloc here but + * probably should provide some cache similar + * to txn cache. + */ + entry = (struct synchro_entry *)malloc(size); + if (entry == NULL) { + diag_set(OutOfMemory, size, "malloc", "synchro_entry"); + return NULL; + } + + struct journal_entry *journal_entry = &entry->journal_entry; + struct synchro_body_bin *body_bin = &entry->body_bin; + struct xrow_header *row = &entry->row; + + entry->applier = applier; + journal_entry->rows[0] = row; + + xrow_encode_synchro(row, body_bin, req); + + row->lsn = applier_row->lsn; + row->replica_id = applier_row->replica_id; + + journal_entry_create(journal_entry, 1, xrow_approx_len(row), + apply_synchro_row_cb, entry); + return entry; +} + +/* + * Process a synchro request from incoming applier packet + * without using txn engine, for a speed sake. + */ +static int +apply_synchro_row(struct applier *applier, struct xrow_header *row) +{ + assert(iproto_type_is_synchro_request(row->type)); + + struct latch *latch = applier_lock(row->replica_id); + if (vclock_get(&replicaset.applier.vclock, + row->replica_id) >= row->lsn) { + applier_unlock(latch); + return 0; + } + + struct synchro_request req; + if (xrow_decode_synchro(row, &req) != 0) + goto out; + + if (txn_limbo_process(&txn_limbo, &req)) + goto out; + + struct synchro_entry *entry; + entry = synchro_entry_new(applier, row, &req); + if (entry == NULL) + goto out; + + if (journal_write_async(&entry->journal_entry) != 0) { + diag_set(ClientError, ER_WAL_IO); + goto out; + } + + /* + * In case if something get wrong the journal completion + * handler will set the applier's vclock back to last + * successfully WAL written value. + */ + vclock_follow(&replicaset.applier.vclock, + row->replica_id, row->lsn); + applier_unlock(latch); + return 0; + +out: + diag_log(); + applier_unlock(latch); + return -1; +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -1118,7 +1265,13 @@ applier_subscribe(struct applier *applier) applier->lag = TIMEOUT_INFINITY; - /* Register triggers to handle WAL writes and rollbacks. */ + /* + * Register triggers to handle WAL writes and rollbacks. + * + * Note we use them for syncronous packets handling as well + * thus when changing make sure that synchro handling won't + * be broken. + */ struct trigger on_wal_write; trigger_create(&on_wal_write, applier_on_wal_write, applier, NULL); trigger_add(&replicaset.applier.on_wal_write, &on_wal_write); @@ -1148,15 +1301,27 @@ applier_subscribe(struct applier *applier) applier_read_tx(applier, &rows); applier->last_row_time = ev_monotonic_now(loop()); + struct xrow_header *row = applier_first_row(&rows); - /* - * In case of an heartbeat message wake a writer up - * and check applier state. - */ - if (applier_first_row(&rows)->lsn == 0) + if (row->lsn == 0) { + /* + * In case of an heartbeat message + * wake a writer up and check + * the applier state. + */ applier_signal_ack(applier); - else if (applier_apply_tx(&rows) != 0) + } else if (iproto_type_is_synchro_request(row->type)) { + /* + * Make sure synchro messages are never reached + * in a batch (this is by design for simplicity + * sake). + */ + assert(stailq_first(&rows) == stailq_last(&rows)); + if (apply_synchro_row(applier, row) != 0) + diag_raise(); + } else if (applier_apply_tx(&rows) != 0) { diag_raise(); + } if (ibuf_used(ibuf) == 0) ibuf_reset(ibuf); -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 00:14:42 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 00:14:42 +0300 Subject: [Tarantool-patches] [PATCH v7 8/8] applier: drop process_synchro_row In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200814211442.667099-9-gorcunov@gmail.com> Handling of synchro requests now are passed via separate apply_synchro_row helper so we no longer need process_synchro_row and can drop it. Closes #5129 Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 38 +++++--------------------------------- 1 file changed, 5 insertions(+), 33 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index a1ce7a23f..7652e1acd 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -270,45 +270,17 @@ process_nop(struct request *request) return txn_commit_stmt(txn, request); } -/* - * CONFIRM/ROLLBACK rows aren't dml requests and require special - * handling: instead of performing some operations on spaces, - * processing these requests requires txn_limbo to either confirm - * or rollback some of its entries. - */ static int -process_synchro_row(struct request *request) +apply_row(struct xrow_header *row) { - assert(iproto_type_is_synchro_request(request->header->type)); - struct txn *txn = in_txn(); + struct request request; - struct synchro_request syn_req; - if (xrow_decode_synchro(request->header, &syn_req) != 0) - return -1; - assert(txn->n_applier_rows == 0); /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. + * Synchro requests must never use txn engine, + * instead they are handled separately. */ - txn_set_flag(txn, TXN_FORCE_ASYNC); + assert(!iproto_type_is_synchro_request(row->type)); - if (txn_begin_stmt(txn, NULL) != 0) - return -1; - if (txn_commit_stmt(txn, request) != 0) - return -1; - return txn_limbo_process(&txn_limbo, &syn_req); -} - -static int -apply_row(struct xrow_header *row) -{ - struct request request; - if (iproto_type_is_synchro_request(row->type)) { - request.header = row; - return process_synchro_row(&request); - } if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; if (request.type == IPROTO_NOP) -- 2.26.2 From gorcunov at gmail.com Sat Aug 15 11:38:51 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 11:38:51 +0300 Subject: [Tarantool-patches] [PATCH v7 9/8] txn: txn_add_redo -- drop synchro processing In-Reply-To: <20200814211442.667099-1-gorcunov@gmail.com> Message-ID: <20200815083851.GD2074@grain> Since we no longer use txn engine for synchro packets processing this code is never executed. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- Happen to miss this yesterday. I pushed into the same branch. src/box/txn.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/src/box/txn.c b/src/box/txn.c index cc1f496c5..b2d342355 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -82,14 +82,7 @@ txn_add_redo(struct txn *txn, struct txn_stmt *stmt, struct request *request) */ struct space *space = stmt->space; row->group_id = space != NULL ? space_group_id(space) : 0; - /* - * Sychronous replication entries are supplementary and - * aren't valid dml requests. They're encoded manually. - */ - if (likely(!iproto_type_is_synchro_request(row->type))) - row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); - else - row->bodycnt = xrow_header_dup_body(row, &txn->region); + row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); if (row->bodycnt < 0) return -1; stmt->row = row; -- 2.26.2 From v.shpilevoy at tarantool.org Sat Aug 15 16:24:11 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 15:24:11 +0200 Subject: [Tarantool-patches] [PATCH 1/1] xrow: introduce struct synchro_request In-Reply-To: <20200813221757.GB2074@grain> References: <20200813221757.GB2074@grain> Message-ID: Hi! Thanks for the review! On 14.08.2020 00:17, Cyrill Gorcunov wrote: > On Thu, Aug 13, 2020 at 11:58:20PM +0200, Vladislav Shpilevoy wrote: > ... >> +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >> { >> assert(lsn > 0); >> >> + struct synchro_request req; >> + req.type = type; >> + req.replica_id = limbo->instance_id; >> + req.lsn = lsn; >> + > > Vlad, while you're at this code, could we please use designated > initialization from the very beginning, ie > > struct synchro_request req = { > .type = type, > .replica_id = limbo->instance_id, > .lsn = lsn, > }; > > (the alignment is up to you though). Such initialization won't > allow a bug to happen when we get a structure extension and > other non updated fields will be zeroified. Ok, done: ==================== @@ -277,10 +277,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { assert(lsn > 0); - struct synchro_request req; - req.type = type; - req.replica_id = limbo->instance_id; - req.lsn = lsn; + struct synchro_request req = { + .type = type, + .replica_id = limbo->instance_id, + .lsn = lsn, + }; ==================== New complete patch: ==================== diff --git a/src/box/applier.cc b/src/box/applier.cc index a953d293e..98fb87375 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -275,26 +275,14 @@ process_nop(struct request *request) * or rollback some of its entries. */ static int -process_confirm_rollback(struct request *request, bool is_confirm) +process_synchro_row(struct request *request) { assert(iproto_type_is_synchro_request(request->header->type)); - uint32_t replica_id; struct txn *txn = in_txn(); - int64_t lsn = 0; - int res = 0; - if (is_confirm) - res = xrow_decode_confirm(request->header, &replica_id, &lsn); - else - res = xrow_decode_rollback(request->header, &replica_id, &lsn); - if (res == -1) - return -1; - - if (replica_id != txn_limbo.instance_id) { - diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, replica_id, - txn_limbo.instance_id); + struct synchro_request syn_req; + if (xrow_decode_synchro(request->header, &syn_req) != 0) return -1; - } assert(txn->n_applier_rows == 0); /* * This is not really a transaction. It just uses txn API @@ -306,16 +294,9 @@ process_confirm_rollback(struct request *request, bool is_confirm) if (txn_begin_stmt(txn, NULL) != 0) return -1; - - if (txn_commit_stmt(txn, request) == 0) { - if (is_confirm) - txn_limbo_read_confirm(&txn_limbo, lsn); - else - txn_limbo_read_rollback(&txn_limbo, lsn); - return 0; - } else { + if (txn_commit_stmt(txn, request) != 0) return -1; - } + return txn_limbo_process(&txn_limbo, &syn_req); } static int @@ -324,8 +305,7 @@ apply_row(struct xrow_header *row) struct request request; if (iproto_type_is_synchro_request(row->type)) { request.header = row; - return process_confirm_rollback(&request, - row->type == IPROTO_CONFIRM); + return process_synchro_row(&request); } if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; diff --git a/src/box/box.cc b/src/box/box.cc index 83eef5d98..8e811e9c1 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -367,22 +367,11 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) { struct request request; if (iproto_type_is_synchro_request(row->type)) { - uint32_t replica_id; - int64_t lsn; - switch(row->type) { - case IPROTO_CONFIRM: - if (xrow_decode_confirm(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_confirm(&txn_limbo, lsn); - break; - case IPROTO_ROLLBACK: - if (xrow_decode_rollback(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_rollback(&txn_limbo, lsn); - break; - } + struct synchro_request syn_req; + if (xrow_decode_synchro(row, &syn_req) != 0) + diag_raise(); + if (txn_limbo_process(&txn_limbo, &syn_req) != 0) + diag_raise(); return; } xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index a2043c17a..c6a4e5efc 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -31,6 +31,7 @@ #include "txn.h" #include "txn_limbo.h" #include "replication.h" +#include "iproto_constants.h" struct txn_limbo txn_limbo; @@ -272,11 +273,16 @@ complete: } static void -txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, - bool is_confirm) +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { assert(lsn > 0); + struct synchro_request req = { + .type = type, + .replica_id = limbo->instance_id, + .lsn = lsn, + }; + struct xrow_header row; struct request request = { .header = &row, @@ -286,19 +292,7 @@ txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, if (txn == NULL) goto rollback; - int res = 0; - if (is_confirm) { - res = xrow_encode_confirm(&row, &txn->region, - limbo->instance_id, lsn); - } else { - /* - * This LSN is the first to be rolled back, so - * the last "safe" lsn is lsn - 1. - */ - res = xrow_encode_rollback(&row, &txn->region, - limbo->instance_id, lsn); - } - if (res == -1) + if (xrow_encode_synchro(&row, &txn->region, &req) != 0) goto rollback; /* * This is not really a transaction. It just uses txn API @@ -325,7 +319,7 @@ rollback: * problems are fixed. Or retry automatically with some period. */ panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, is_confirm ? "CONFIRM" : "ROLLBACK"); + "%s\n", lsn, iproto_type_name(type)); } /** @@ -338,10 +332,11 @@ txn_limbo_write_confirm(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->confirmed_lsn = lsn; - txn_limbo_write_confirm_rollback(limbo, lsn, true); + txn_limbo_write_synchro(limbo, IPROTO_CONFIRM, lsn); } -void +/** Confirm all the entries <= @a lsn. */ +static void txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -390,11 +385,12 @@ txn_limbo_write_rollback(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->is_in_rollback = true; - txn_limbo_write_confirm_rollback(limbo, lsn, false); + txn_limbo_write_synchro(limbo, IPROTO_ROLLBACK, lsn); limbo->is_in_rollback = false; } -void +/** Rollback all the entries >= @a lsn. */ +static void txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -577,6 +573,27 @@ complete: return 0; } +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) +{ + if (req->replica_id != limbo->instance_id) { + diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, req->replica_id, + limbo->instance_id); + return -1; + } + switch (req->type) { + case IPROTO_CONFIRM: + txn_limbo_read_confirm(limbo, req->lsn); + break; + case IPROTO_ROLLBACK: + txn_limbo_read_rollback(limbo, req->lsn); + break; + default: + unreachable(); + } + return 0; +} + void txn_limbo_force_empty(struct txn_limbo *limbo, int64_t confirm_lsn) { diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h index 04ee7ea5c..eaf662987 100644 --- a/src/box/txn_limbo.h +++ b/src/box/txn_limbo.h @@ -39,6 +39,7 @@ extern "C" { #endif /* defined(__cplusplus) */ struct txn; +struct synchro_request; /** * Transaction and its quorum metadata, to be stored in limbo. @@ -245,17 +246,9 @@ txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn); int txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry); -/** - * Confirm all the entries up to the given master's LSN. - */ -void -txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn); - -/** - * Rollback all the entries starting with given master's LSN. - */ -void -txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn); +/** Execute a synchronous replication request. */ +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req); /** * Waiting for confirmation of all "sync" transactions diff --git a/src/box/xrow.c b/src/box/xrow.c index 0c797a9d5..bf174c701 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,13 +893,13 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -static int -xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn, int type) +int +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req) { size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(replica_id) + mp_sizeof_uint(IPROTO_LSN) + - mp_sizeof_uint(lsn); + mp_sizeof_uint(req->replica_id) + + mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); char *buf = (char *)region_alloc(region, len); if (buf == NULL) { diag_set(OutOfMemory, len, "region_alloc", "buf"); @@ -909,9 +909,9 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, pos = mp_encode_map(pos, 2); pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, replica_id); + pos = mp_encode_uint(pos, req->replica_id); pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, lsn); + pos = mp_encode_uint(pos, req->lsn); memset(row, 0, sizeof(*row)); @@ -919,30 +919,13 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, row->body[0].iov_len = len; row->bodycnt = 1; - row->type = type; + row->type = req->type; return 0; } int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_CONFIRM); -} - -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_ROLLBACK); -} - -static int -xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, - int64_t *lsn) +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req) { if (row->bodycnt == 0) { diag_set(ClientError, ER_INVALID_MSGPACK, "request body"); @@ -960,6 +943,7 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, return -1; } + memset(req, 0, sizeof(*req)); d = data; uint32_t map_size = mp_decode_map(&d); for (uint32_t i = 0; i < map_size; i++) { @@ -977,30 +961,19 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, } switch (key) { case IPROTO_REPLICA_ID: - *replica_id = mp_decode_uint(&d); + req->replica_id = mp_decode_uint(&d); break; case IPROTO_LSN: - *lsn = mp_decode_uint(&d); + req->lsn = mp_decode_uint(&d); break; default: mp_next(&d); } } + req->type = row->type; return 0; } -int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - -int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - int xrow_to_iovec(const struct xrow_header *row, struct iovec *out) { diff --git a/src/box/xrow.h b/src/box/xrow.h index e21ede5a3..02dca74e5 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -216,54 +216,51 @@ xrow_encode_dml(const struct request *request, struct region *region, struct iovec *iov); /** - * Encode the CONFIRM to row body and set row type to - * IPROTO_CONFIRM. - * @param row xrow header. - * @param region Region to use to encode the confirmation body. - * @param replica_id master's instance id. - * @param lsn last confirmed lsn. - * @retval -1 on error. - * @retval 0 success. + * Synchronous replication request - confirmation or rollback of + * pending synchronous transactions. */ -int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +struct synchro_request { + /** Operation type - IPROTO_ROLLBACK or IPROTO_CONFIRM. */ + uint32_t type; + /** + * ID of the instance owning the pending transactions. + * Note, it may be not the same instance, who created this + * request. An instance can make an operation on foreign + * synchronous transactions in case a new master tries to + * finish transactions of an old master. + */ + uint32_t replica_id; + /** + * Operation LSN. + * In case of CONFIRM it means 'confirm all + * transactions with lsn <= this value'. + * In case of ROLLBACK it means 'rollback all transactions + * with lsn >= this value'. + */ + int64_t lsn; +}; /** - * Decode the CONFIRM request body. + * Encode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn last confirmed lsn. + * @param region Region to use to encode the confirmation body. + * @param req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); - -/** - * Encode the ROLLBACK row body and set row type to - * IPROTO_ROLLBACK. - * @param row xrow header. - * @param region Region to use to encode the rollback body. - * @param replica_id master's instance id. - * @param lsn lsn to rollback from, including it. - * @retval -1 on error. - * @retval 0 success. - */ -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req); /** - * Decode the ROLLBACK row body. + * Decode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn lsn to rollback from, including it. + * @param[out] req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); /** * CALL/EVAL request. From gorcunov at gmail.com Sat Aug 15 16:38:16 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 16:38:16 +0300 Subject: [Tarantool-patches] [PATCH 1/1] xrow: introduce struct synchro_request In-Reply-To: References: <20200813221757.GB2074@grain> Message-ID: <20200815133816.GE2074@grain> On Sat, Aug 15, 2020 at 03:24:11PM +0200, Vladislav Shpilevoy wrote: > > Ok, done: > > ==================== > @@ -277,10 +277,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) > { > assert(lsn > 0); > > - struct synchro_request req; > - req.type = type; > - req.replica_id = limbo->instance_id; > - req.lsn = lsn; > + struct synchro_request req = { > + .type = type, > + .replica_id = limbo->instance_id, > + .lsn = lsn, > + }; > Ack. Thank you! From v.shpilevoy at tarantool.org Sat Aug 15 18:04:27 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 17:04:27 +0200 Subject: [Tarantool-patches] [PATCH v7 4/8] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200814211442.667099-5-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-5-gorcunov@gmail.com> Message-ID: <19c845c0-6cf9-288b-1cb6-942681d93ec3@tarantool.org> Hi! Thanks for the patch! See 3 comments below. On 14.08.2020 23:14, Cyrill Gorcunov wrote: > When we need to write CONFIRM or ROLLBACK message (which is > a binary record in msgpack format) into a journal we use txn code > to allocate a new transaction, encode there a message and pass it > to walk the long txn path before it hit the journal. This is not > only resource wasting but also somehow strange from architectural > point of view. > > Instead lets encode a record on the stack and write it to the journal > directly. > > Part-of #5129 > > Signed-off-by: Cyrill Gorcunov > --- > src/box/txn_limbo.c | 64 ++++++++++++++++++++++----------------------- > 1 file changed, 32 insertions(+), 32 deletions(-) > > diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c > index ed8c10419..447630d23 100644 > --- a/src/box/txn_limbo.c > +++ b/src/box/txn_limbo.c > @@ -284,46 +296,34 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) > > /* > * This is a synchronous commit so we can > - * use body and row allocated on a stack. > + * allocate everything on a stack. > */ 1. Unnecessary change. > struct synchro_body_bin body; > struct xrow_header row; > - struct request request = { > - .header = &row, > - }; > + char buf[sizeof(struct journal_entry) + > + sizeof(struct xrow_header *)]; 2. Is there a guarantee, that 'buf' will be aligned by at least 8 bytes? > > - struct txn *txn = txn_begin(); > - if (txn == NULL) > - goto rollback; > + struct journal_entry *entry = (struct journal_entry *)buf; > + entry->rows[0] = &row; > > xrow_encode_synchro(&row, &body, &req); > > - /* > - * This is not really a transaction. It just uses txn API > - * to put the data into WAL. And obviously it should not > - * go to the limbo and block on the very same sync > - * transaction which it tries to confirm now. > - */ > - txn_set_flag(txn, TXN_FORCE_ASYNC); > - > - if (txn_begin_stmt(txn, NULL) != 0) > - goto rollback; > - if (txn_commit_stmt(txn, &request) != 0) > - goto rollback; > - if (txn_commit(txn) != 0) > - goto rollback; > - return; > + journal_entry_create(entry, 1, xrow_approx_len(&row), > + txn_limbo_write_cb, fiber()); > > -rollback: > - /* > - * XXX: the stub is supposed to be removed once it is defined what to do > - * when a synchro request WAL write fails. One of the possible > - * solutions: log the error, keep the limbo queue as is and probably put > - * in rollback mode. Then provide a hook to call manually when WAL > - * problems are fixed. Or retry automatically with some period. > - */ > - panic("Could not write a synchro request to WAL: lsn = %lld, type = " > - "%s\n", lsn, iproto_type_name(type)); > + if (journal_write(entry) != 0 || entry->res < 0) { > + diag_set(ClientError, ER_WAL_IO); > + diag_log(); > + /* > + * XXX: the stub is supposed to be removed once it is defined what to do > + * when a synchro request WAL write fails. One of the possible > + * solutions: log the error, keep the limbo queue as is and probably put > + * in rollback mode. Then provide a hook to call manually when WAL > + * problems are fixed. Or retry automatically with some period. 3. Out of 80 symbols. > + */ > + panic("Could not write a synchro request to WAL: lsn = %lld, type = " > + "%s\n", lsn, type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); > + } > } > > /** > From v.shpilevoy at tarantool.org Sat Aug 15 18:04:32 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 17:04:32 +0200 Subject: [Tarantool-patches] [PATCH v7 5/8] applier: factor out latch locking In-Reply-To: <20200814211442.667099-6-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-6-gorcunov@gmail.com> Message-ID: <0da9d013-6b94-fc4d-17d4-379dcf51bbed@tarantool.org> Thanks for the patch! On 14.08.2020 23:14, Cyrill Gorcunov wrote: > We will need to reuse this helpers. > > Part-of #5129 > > Signed-off-by: Cyrill Gorcunov > --- > src/box/applier.cc | 43 +++++++++++++++++++++++++++++-------------- > 1 file changed, 29 insertions(+), 14 deletions(-) > > diff --git a/src/box/applier.cc b/src/box/applier.cc > index 98fb87375..60689f6d3 100644 > --- a/src/box/applier.cc > +++ b/src/box/applier.cc > @@ -799,6 +799,29 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) > return 0; > } > > +/* > + * In a full mesh topology, the same set of changes > + * may arrive via two concurrently running appliers. > + * Hence we need a latch to strictly order all changes > + * that belong to the same server id. > + */ > +static inline struct latch * > +applier_lock(uint32_t replica_id) > +{ > + struct replica *replica = replica_by_id(replica_id); > + struct latch *latch = (replica ? &replica->order_latch : We usually use != NULL to check if a pointer is not NULL. To emphasize it is not a boolean variable in the code. > + &replicaset.applier.order_latch); > + latch_lock(latch); > + return latch; > +} From v.shpilevoy at tarantool.org Sat Aug 15 18:06:05 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 17:06:05 +0200 Subject: [Tarantool-patches] [PATCH v7 7/8] applier: process synchro requests without txn engine In-Reply-To: <20200814211442.667099-8-gorcunov@gmail.com> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-8-gorcunov@gmail.com> Message-ID: Thanks for the patch! See 10 comments below. On 14.08.2020 23:14, Cyrill Gorcunov wrote: > Transaction processing code is very heavy simply because > trasactions are carrying various data and involves a number > of other mechanisms to procceed. 1. trasactions -> transactions. procceed -> proceed. > In turn, when we receive confirm or rollback packed from > another node in a cluster we just need to inspect limbo > queue and write this packed into a WAL journal. So calling > a bunch of txn engine helpers is simply waste of cycles. > > Thus lets rather handle them in a special light way: > > - allocate synchro_entry structure which would carry > the journal entry itself and encoded message > - process limbo queue to mark confirmed/rollback'ed > messages > - finally write this synchro_entry into a journal > > Which is a way more simplier. 2. 'more simplier' -> simpler. Otherwise looks like '????? ?????'. > Part-of #5129 > > Suggedsted-by: Vladislav Shpilevoy > Signed-off-by: Cyrill Gorcunov > --- > src/box/applier.cc | 179 +++++++++++++++++++++++++++++++++++++++++++-- > 1 file changed, 172 insertions(+), 7 deletions(-) > > diff --git a/src/box/applier.cc b/src/box/applier.cc > index a71516282..a1ce7a23f 100644 > --- a/src/box/applier.cc > +++ b/src/box/applier.cc > @@ -841,6 +843,151 @@ applier_unlock(struct latch *latch) > latch_unlock(latch); > } > > +struct synchro_entry { > + /** An applier initiated the syncho request. */ > + struct applier *applier; 3. Actually 'applier' is not needed. I looked around and realized it is never used. I even dropped it and nothing changed. > + > + /** Encoded form of a synchro record. */ > + struct synchro_body_bin body_bin; > + > + /** xrow to write, used by the journal engine. */ > + struct xrow_header row; > + > + /** > + * The journal entry itself. Note since > + * it has unsized array it must be the > + * last entry in the structure. > + */ > + struct journal_entry journal_entry; > +}; > + > +static void > +synchro_entry_delete(struct synchro_entry *entry) > +{ > + free(entry); > +} > + > +/** > + * Async write journal completion. > + */ > +static void > +apply_synchro_row_cb(struct journal_entry *entry) > +{ > + assert(entry->complete_data != NULL); > + struct synchro_entry *synchro_entry = > + (struct synchro_entry *)entry->complete_data; > + struct applier *applier = synchro_entry->applier; > + > + /* > + * We can reuse triggers, they are allocated when > + * applier get subscribed and since packets handling > + * is processed after the subscribtion phase the triggers > + * will be alive. 4. subscribtion -> subscription. Also I don't think I understood the comment. > + */ > + if (entry->res < 0) { > + trigger_run(&replicaset.applier.on_rollback, applier); > + /* > + * Restore the last written vlock value. > + */ > + vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); > + diag_set(ClientError, ER_WAL_IO); > + diag_log(); 5. Error should be set before running on_rollback, and it should be installed into replicaset.applier.diag. > + } else { > + trigger_run(&replicaset.applier.on_wal_write, applier); > + } > + > + synchro_entry_delete(synchro_entry); > +} > + > +/** > + * Allocate a new synchro_entry to be passed to > + * the journal engine in async write way. > + */ > +static struct synchro_entry * > +synchro_entry_new(struct applier *applier, > + struct xrow_header *applier_row, > + struct synchro_request *req) > +{ > + struct synchro_entry *entry; > + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); 6. Why don't you just add 'struct xrow_header*[1]' to the end of struct synchro_entry? There is no a case, when the entry is needed without the xrow_header pointer in the end. > + > + /* > + * For simplicity we use malloc here but > + * probably should provide some cache similar > + * to txn cache. > + */ > + entry = (struct synchro_entry *)malloc(size); > + if (entry == NULL) { > + diag_set(OutOfMemory, size, "malloc", "synchro_entry"); > + return NULL; > + } > + > + struct journal_entry *journal_entry = &entry->journal_entry; > + struct synchro_body_bin *body_bin = &entry->body_bin; > + struct xrow_header *row = &entry->row; > + > + entry->applier = applier; > + journal_entry->rows[0] = row; > + > + xrow_encode_synchro(row, body_bin, req); > + > + row->lsn = applier_row->lsn; > + row->replica_id = applier_row->replica_id; > + > + journal_entry_create(journal_entry, 1, xrow_approx_len(row), > + apply_synchro_row_cb, entry); > + return entry; > +} > + > +/* > + * Process a synchro request from incoming applier packet > + * without using txn engine, for a speed sake. 7. It is not about speed. Txn module is fast, it is one of the hottest and most optimized places in the whole code base. And this is exactly why synchro requests *should not* be there - they slow down and complicate txn, not vice versa. > + */ > +static int > +apply_synchro_row(struct applier *applier, struct xrow_header *row) > +{ > + assert(iproto_type_is_synchro_request(row->type)); > + > + struct latch *latch = applier_lock(row->replica_id); > + if (vclock_get(&replicaset.applier.vclock, > + row->replica_id) >= row->lsn) { > + applier_unlock(latch); > + return 0; > + } > + > + struct synchro_request req; > + if (xrow_decode_synchro(row, &req) != 0) > + goto out; > + > + if (txn_limbo_process(&txn_limbo, &req)) > + goto out; > + > + struct synchro_entry *entry; > + entry = synchro_entry_new(applier, row, &req); > + if (entry == NULL) > + goto out; > + > + if (journal_write_async(&entry->journal_entry) != 0) { > + diag_set(ClientError, ER_WAL_IO); > + goto out; > + } > + > + /* > + * In case if something get wrong the journal completion > + * handler will set the applier's vclock back to last > + * successfully WAL written value. > + */ > + vclock_follow(&replicaset.applier.vclock, > + row->replica_id, row->lsn); > + applier_unlock(latch); 8. Code duplication is too big. And I wouldn't mind if it was just applier locks, but vclock propagation and rollback is not that simple. I think we should do all that inside {applier_apply_tx()}. Because technically you apply tx - the synchro row is stored inside {struct stailq *rows} which is the tx. I moved it into {applier_apply_tx()}, and the code became smaller, simpler, with less duplication, and even less diff. It also allows to drop the commits 5/8 and 6/8. Take a look and lets discuss. > + return 0; > + > +out: > + diag_log(); > + applier_unlock(latch); > + return -1; > +} > + > /** > * Apply all rows in the rows queue as a single transaction. > * > @@ -1148,15 +1301,27 @@ applier_subscribe(struct applier *applier) > applier_read_tx(applier, &rows); > > applier->last_row_time = ev_monotonic_now(loop()); > + struct xrow_header *row = applier_first_row(&rows); > > - /* > - * In case of an heartbeat message wake a writer up > - * and check applier state. > - */ > - if (applier_first_row(&rows)->lsn == 0) > + if (row->lsn == 0) { > + /* > + * In case of an heartbeat message > + * wake a writer up and check > + * the applier state. > + */ > applier_signal_ack(applier); > - else if (applier_apply_tx(&rows) != 0) > + } else if (iproto_type_is_synchro_request(row->type)) { > + /* > + * Make sure synchro messages are never reached > + * in a batch (this is by design for simplicity > + * sake). 9. It is not about simplicity. It is about being not necessary. Transactions exist for DML and DDL (which is also DML on system spaces) only. For other WAL writes transactions in their common sense don't exist. So each row is a 'transaction'. In future we may want to change that and, for example, incorporate several synchro requests into a 'tx' (don't know why would we need that, but it is technically possible). > + */ > + assert(stailq_first(&rows) == stailq_last(&rows)); > + if (apply_synchro_row(applier, row) != 0) > + diag_raise(); > + } else if (applier_apply_tx(&rows) != 0) { > diag_raise(); > + } > > if (ibuf_used(ibuf) == 0) > ibuf_reset(ibuf); > 10. Consider my changes on top of this commit on your branch. Below I paste my diff squashed into your commit (on the branch they are not squashed). ==================== diff --git a/src/box/applier.cc b/src/box/applier.cc index a71516282..dfa62b72a 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -51,8 +51,10 @@ #include "schema.h" #include "txn.h" #include "box.h" +#include "xrow.h" #include "scoped_guard.h" #include "txn_limbo.h" +#include "journal.h" STRS(applier_state, applier_STATE); @@ -772,19 +774,9 @@ applier_read_tx(struct applier *applier, struct stailq *rows) } while (!applier_last_row(rows)->is_commit); } -static int -applier_txn_rollback_cb(struct trigger *trigger, void *event) +static void +applier_rollback_by_wal_io(void) { - (void) trigger; - struct txn *txn = (struct txn *) event; - /* - * Synchronous transaction rollback due to receiving a - * ROLLBACK entry is a normal event and requires no - * special handling. - */ - if (txn->signature == TXN_SIGNATURE_SYNC_ROLLBACK) - return 0; - /* * Setup shared applier diagnostic area. * @@ -793,19 +785,32 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) * diag use per-applier diag instead all the time * (which actually already present in the structure). * - * But remember that transactions are asynchronous - * and rollback may happen a way latter after it - * passed to the journal engine. + * But remember that WAL writes are asynchronous and + * rollback may happen a way later after it was passed to + * the journal engine. */ diag_set(ClientError, ER_WAL_IO); diag_set_error(&replicaset.applier.diag, diag_last_error(diag_get())); - /* Broadcast the rollback event across all appliers. */ - trigger_run(&replicaset.applier.on_rollback, event); - + /* Broadcast the rollback across all appliers. */ + trigger_run(&replicaset.applier.on_rollback, NULL); /* Rollback applier vclock to the committed one. */ vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); +} + +static int +applier_txn_rollback_cb(struct trigger *trigger, void *event) +{ + (void) trigger; + struct txn *txn = (struct txn *) event; + /* + * Synchronous transaction rollback due to receiving a + * ROLLBACK entry is a normal event and requires no + * special handling. + */ + if (txn->signature != TXN_SIGNATURE_SYNC_ROLLBACK) + applier_rollback_by_wal_io(); return 0; } @@ -841,6 +846,110 @@ applier_unlock(struct latch *latch) latch_unlock(latch); } +struct synchro_entry { + /** Encoded form of a synchro record. */ + struct synchro_body_bin body_bin; + + /** xrow to write, used by the journal engine. */ + struct xrow_header row; + + /** + * The journal entry itself. Note since + * it has unsized array it must be the + * last entry in the structure. + */ + struct journal_entry journal_entry; +}; + +static void +synchro_entry_delete(struct synchro_entry *entry) +{ + free(entry); +} + +/** + * Async write journal completion. + */ +static void +apply_synchro_row_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + struct synchro_entry *synchro_entry = + (struct synchro_entry *)entry->complete_data; + if (entry->res < 0) + applier_rollback_by_wal_io(); + else + trigger_run(&replicaset.applier.on_wal_write, NULL); + + synchro_entry_delete(synchro_entry); +} + +/** + * Allocate a new synchro_entry to be passed to + * the journal engine in async write way. + */ +static struct synchro_entry * +synchro_entry_new(struct xrow_header *applier_row, + struct synchro_request *req) +{ + struct synchro_entry *entry; + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); + + /* + * For simplicity we use malloc here but + * probably should provide some cache similar + * to txn cache. + */ + entry = (struct synchro_entry *)malloc(size); + if (entry == NULL) { + diag_set(OutOfMemory, size, "malloc", "synchro_entry"); + return NULL; + } + + struct journal_entry *journal_entry = &entry->journal_entry; + struct synchro_body_bin *body_bin = &entry->body_bin; + struct xrow_header *row = &entry->row; + + journal_entry->rows[0] = row; + + xrow_encode_synchro(row, body_bin, req); + + row->lsn = applier_row->lsn; + row->replica_id = applier_row->replica_id; + + journal_entry_create(journal_entry, 1, xrow_approx_len(row), + apply_synchro_row_cb, entry); + return entry; +} + +/** Process a synchro request. */ +static int +apply_synchro_row(struct xrow_header *row) +{ + assert(iproto_type_is_synchro_request(row->type)); + + struct synchro_request req; + if (xrow_decode_synchro(row, &req) != 0) + goto err; + + if (txn_limbo_process(&txn_limbo, &req)) + goto err; + + struct synchro_entry *entry; + entry = synchro_entry_new(row, &req); + if (entry == NULL) + goto err; + + if (journal_write_async(&entry->journal_entry) != 0) { + diag_set(ClientError, ER_WAL_IO); + goto err; + } + return 0; +err: + diag_log(); + return -1; +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -876,13 +985,26 @@ applier_apply_tx(struct stailq *rows) } } + if (unlikely(iproto_type_is_synchro_request(first_row->type))) { + /* + * Synchro messages are not transactions, in terms + * of DML. Always sent and written isolated from + * each other. + */ + assert(first_row == last_row); + if (apply_synchro_row(first_row) != 0) + diag_raise(); + goto success; + } + /** * Explicitly begin the transaction so that we can * control fiber->gc life cycle and, in case of apply * conflict safely access failed xrow object and allocate * IPROTO_NOP on gc. */ - struct txn *txn = txn_begin(); + struct txn *txn; + txn = txn_begin(); struct applier_tx_row *item; if (txn == NULL) { applier_unlock(latch); @@ -951,6 +1073,7 @@ applier_apply_tx(struct stailq *rows) if (txn_commit_async(txn) < 0) goto fail; +success: /* * The transaction was sent to journal so promote vclock. * @@ -1118,7 +1241,13 @@ applier_subscribe(struct applier *applier) applier->lag = TIMEOUT_INFINITY; - /* Register triggers to handle WAL writes and rollbacks. */ + /* + * Register triggers to handle WAL writes and rollbacks. + * + * Note we use them for syncronous packets handling as well + * thus when changing make sure that synchro handling won't + * be broken. + */ struct trigger on_wal_write; trigger_create(&on_wal_write, applier_on_wal_write, applier, NULL); trigger_add(&replicaset.applier.on_wal_write, &on_wal_write); From v.shpilevoy at tarantool.org Sat Aug 15 18:06:09 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 17:06:09 +0200 Subject: [Tarantool-patches] [PATCH v7 9/8] txn: txn_add_redo -- drop synchro processing In-Reply-To: <20200815083851.GD2074@grain> References: <20200815083851.GD2074@grain> Message-ID: <5618a6d1-1b65-a24e-e6d9-6332882f3fdb@tarantool.org> Thanks for the patch! On 15.08.2020 10:38, Cyrill Gorcunov wrote: > Since we no longer use txn engine for synchro > packets processing this code is never executed. > > Part-of #5129 > > Signed-off-by: Cyrill Gorcunov > --- > Happen to miss this yesterday. I pushed into the same branch. > > src/box/txn.c | 9 +-------- > 1 file changed, 1 insertion(+), 8 deletions(-) > > diff --git a/src/box/txn.c b/src/box/txn.c > index cc1f496c5..b2d342355 100644 > --- a/src/box/txn.c > +++ b/src/box/txn.c > @@ -82,14 +82,7 @@ txn_add_redo(struct txn *txn, struct txn_stmt *stmt, struct request *request) > */ > struct space *space = stmt->space; > row->group_id = space != NULL ? space_group_id(space) : 0; > - /* > - * Sychronous replication entries are supplementary and > - * aren't valid dml requests. They're encoded manually. > - */ > - if (likely(!iproto_type_is_synchro_request(row->type))) > - row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); > - else > - row->bodycnt = xrow_header_dup_body(row, &txn->region); xrow_header_dup_body() can be deleted now. > + row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); > if (row->bodycnt < 0) > return -1; > stmt->row = row; > From v.shpilevoy at tarantool.org Sat Aug 15 18:16:44 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 15 Aug 2020 17:16:44 +0200 Subject: [Tarantool-patches] [PATCH 1/1] applier: drop a couple of unnecessary arguments Message-ID: <7e16e45e171cb34edaff69b0f660421f1b67b4a1.1597504571.git.v.shpilevoy@tarantool.org> Applier on_rollback and on_wal_write don't need any arguments - they either work with a global state, or with the signaled applier stored inside the trigger. However into on_wal_write() and on_rollback() was passed the transaction object, unused. Even if it would be used, it should have been fixed, because soon these triggers will be fired not only for traditional 'txn' transactions. They will be used by the synchro request WAL writes too - they don't have 'transactions'. Part of #5129 --- Branch: http://github.com/tarantool/tarantool/tree/gerold103/applier_triggers_args Issue: https://github.com/tarantool/tarantool/issues/5129 src/box/applier.cc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index a953d293e..34cb0e4b9 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -802,8 +802,8 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) diag_set_error(&replicaset.applier.diag, diag_last_error(diag_get())); - /* Broadcast the rollback event across all appliers. */ - trigger_run(&replicaset.applier.on_rollback, event); + /* Broadcast the rollback across all appliers. */ + trigger_run(&replicaset.applier.on_rollback, NULL); /* Rollback applier vclock to the committed one. */ vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); @@ -814,8 +814,9 @@ static int applier_txn_wal_write_cb(struct trigger *trigger, void *event) { (void) trigger; - /* Broadcast the commit event across all appliers. */ - trigger_run(&replicaset.applier.on_wal_write, event); + (void) event; + /* Broadcast the WAL write across all appliers. */ + trigger_run(&replicaset.applier.on_wal_write, NULL); return 0; } -- 2.21.1 (Apple Git-122.3) From gorcunov at gmail.com Sat Aug 15 19:26:07 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 19:26:07 +0300 Subject: [Tarantool-patches] [PATCH v7 4/8] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <19c845c0-6cf9-288b-1cb6-942681d93ec3@tarantool.org> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-5-gorcunov@gmail.com> <19c845c0-6cf9-288b-1cb6-942681d93ec3@tarantool.org> Message-ID: <20200815162607.GF2074@grain> On Sat, Aug 15, 2020 at 05:04:27PM +0200, Vladislav Shpilevoy wrote: > > + char buf[sizeof(struct journal_entry) + > > + sizeof(struct xrow_header *)]; > > 2. Is there a guarantee, that 'buf' will be aligned by at least > 8 bytes? Yup. On 32bit machines it would be 4 bytes aligned, on 64bit -- 8 byte (this is basically how stack management works by default). I'll address the rest of comments. Thanks! From gorcunov at gmail.com Sat Aug 15 19:27:00 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Sat, 15 Aug 2020 19:27:00 +0300 Subject: [Tarantool-patches] [PATCH v7 5/8] applier: factor out latch locking In-Reply-To: <0da9d013-6b94-fc4d-17d4-379dcf51bbed@tarantool.org> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-6-gorcunov@gmail.com> <0da9d013-6b94-fc4d-17d4-379dcf51bbed@tarantool.org> Message-ID: <20200815162700.GG2074@grain> On Sat, Aug 15, 2020 at 05:04:32PM +0200, Vladislav Shpilevoy wrote: > > +static inline struct latch * > > +applier_lock(uint32_t replica_id) > > +{ > > + struct replica *replica = replica_by_id(replica_id); > > + struct latch *latch = (replica ? &replica->order_latch : > > We usually use != NULL to check if a pointer is not NULL. To emphasize > it is not a boolean variable in the code. I simply copied this code from old place. But sure, will add. From avtikhon at tarantool.org Sun Aug 16 23:01:34 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Sun, 16 Aug 2020 23:01:34 +0300 Subject: [Tarantool-patches] [PATCH v1] test: fix issue on first replica in drop_cluster() Message-ID: <27c2f93a6602f14b882484b4f0c7ec4b8748c371.1597607968.git.avtikhon@tarantool.org> Found flaky failed test replication/box_set_replication_stress.test.lua on drop_cluster() routine, like: --- replication/box_set_replication_stress.result Fri Aug 14 18:28:41 2020 +++ var/004_replication/box_set_replication_stress.result Sat Aug 15 15:19:44 2020 @@ -34,5 +34,3 @@ -- Cleanup. test_run:drop_cluster(SERVERS) - | --- - | ... Found that drop_cluster() routine from test-run repository failed in stop() routine from lib/tarantool_server.py:TarantoolServer class. It failed to stop 1st replica which used in test to switch on/off the replication 1000 times. It happend because stop() routine used SIGTERM by default which couldn't kill the first replica in some situations. It happend when both replca processes were alive and tried to read and write data into their sockets, but sockets of the first replica were already unreachable while second replica were alive. In this situation SIGTERM signal was not enough to stop the first replica and test-run hanged in wait_stop() in lib/tarantool_server.py:TarantoolServer class till test-run stopped the test by its general timeout of 2 minutes. To fix the issue the only possible way was to use SIGKILL instead of SIGTERM to be sure that the process will not wait for sockets closing and would be killed w/o waiting of it. SIGKILL could be used by default in drop_cluster() routine, but seems that this change was not good for detecting the other issues of the other tests. So it was decided to use SIGKILL just in this test as the additional option for "stop server" test-run call. Closes #5244 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5244-replication-box-stress-drop-replica Issue: https://github.com/tarantool/tarantool/issues/5244 .../replication/box_set_replication_stress.result | 15 ++++++++++++++- .../box_set_replication_stress.test.lua | 5 ++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/test/replication/box_set_replication_stress.result b/test/replication/box_set_replication_stress.result index e683c0643..225f33ecb 100644 --- a/test/replication/box_set_replication_stress.result +++ b/test/replication/box_set_replication_stress.result @@ -33,6 +33,19 @@ test_run:cmd("switch default") | ... -- Cleanup. -test_run:drop_cluster(SERVERS) +test_run:cmd('stop server master_quorum1 with signal=SIGKILL') | --- + | - true + | ... +test_run:cmd('delete server master_quorum1') + | --- + | - true + | ... +test_run:cmd('stop server master_quorum2 with signal=SIGKILL') + | --- + | - true + | ... +test_run:cmd('delete server master_quorum2') + | --- + | - true | ... diff --git a/test/replication/box_set_replication_stress.test.lua b/test/replication/box_set_replication_stress.test.lua index 407e91e0f..88652b0b4 100644 --- a/test/replication/box_set_replication_stress.test.lua +++ b/test/replication/box_set_replication_stress.test.lua @@ -14,4 +14,7 @@ end test_run:cmd("switch default") -- Cleanup. -test_run:drop_cluster(SERVERS) +test_run:cmd('stop server master_quorum1 with signal=SIGKILL') +test_run:cmd('delete server master_quorum1') +test_run:cmd('stop server master_quorum2 with signal=SIGKILL') +test_run:cmd('delete server master_quorum2') -- 2.17.1 From avtikhon at tarantool.org Sun Aug 16 23:41:38 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Sun, 16 Aug 2020 23:41:38 +0300 Subject: [Tarantool-patches] [PATCH v1] tarantoolctl fails to stop server Message-ID: When server process couldn't be stopped with SIGTERM as described in [1], then tarantoolctl didn't check its status and returned success exit code. To fix it, check of the process after SIGTERM send should be added and if the process was still alive than it should be killed with SIGKILL. Also some time delay added before the process status check to be sure that it completely gone. Closed #5245 [1] - https://github.com/tarantool/tarantool/issues/5244 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5245-tarantoolctl-kills Issue: https://github.com/tarantool/tarantool/issues/5244 extra/dist/tarantoolctl.in | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/extra/dist/tarantoolctl.in b/extra/dist/tarantoolctl.in index 90caf58ad..7bf3808f8 100755 --- a/extra/dist/tarantoolctl.in +++ b/extra/dist/tarantoolctl.in @@ -600,6 +600,15 @@ local function stop() fio.unlink(pid_file) return 1 end + fiber.sleep(1) + if fio.stat(pid_file) ~= nil then + log.info("Kill not terminated process %d", pid) + if ffi.C.kill(pid, 9) < 0 then + log.error("Can't kill process %d: %s", pid, errno.strerror()) + fio.unlink(pid_file) + return 1 + end + end return 0 end -- 2.17.1 From avtikhon at tarantool.org Sun Aug 16 23:58:43 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Sun, 16 Aug 2020 23:58:43 +0300 Subject: [Tarantool-patches] [PATCH v1] Output for sockets issues cuts its names in logs Message-ID: <6d7c812486f707fec0e8c653c8a98b893d655d1a.1597611447.git.avtikhon@tarantool.org> Found issue in log files: 2020-08-10 11:47:07.191 [73298] iproto sio.c:268 !> SystemError writev(1), called on fd 31, aka unix/:/Users/tntmac01.tarantool.i/tnt/test/var/001_r: Broken pipe when it cut the socket name. The length of the socket name based on: src/lib/core/sio.h:enum { SERVICE_NAME_MAXLEN = 32 }; src/lib/core/sio.c: int name_size = 2 * SERVICE_NAME_MAXLEN; Decided to increase multiplication of the name_size from 2 to 4. Closes #5246 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5246-logs-output-length Issue: https://github.com/tarantool/tarantool/issues/5246 src/lib/core/sio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/core/sio.c b/src/lib/core/sio.c index 97a512eee..631f792e8 100644 --- a/src/lib/core/sio.c +++ b/src/lib/core/sio.c @@ -51,7 +51,7 @@ sio_socketname(int fd) { /* Preserve errno */ int save_errno = errno; - int name_size = 2 * SERVICE_NAME_MAXLEN; + int name_size = 4 * SERVICE_NAME_MAXLEN; char *name = static_alloc(name_size); int n = snprintf(name, name_size, "fd %d", fd); if (fd >= 0) { -- 2.17.1 From alexander.turenko at tarantool.org Mon Aug 17 00:26:59 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Mon, 17 Aug 2020 00:26:59 +0300 Subject: [Tarantool-patches] [PATCH v4 1/2] vinyl: check vinyl_dir existence at bootstrap In-Reply-To: <4eb7b26b9f80393b19536f49ab3985d72ba13d89.1597398597.git.avtikhon@tarantool.org> References: <4eb7b26b9f80393b19536f49ab3985d72ba13d89.1597398597.git.avtikhon@tarantool.org> Message-ID: <20200816212659.t4q55vctby3imgd7@tkn_work_nb> > vinyl: check vinyl_dir existence at bootstrap It looks like you added the check, but it was present before the change. 'Fix check' better describes the change. > During implementation of openSUSE got failed box-tap/cfg.test.lua test. > Found that when memtx_dir wasn't existed, while vinyl_dir existed and > errno was set to ENOENT, box configuration succeeded, but it shouldn't. > Reason of this wrong behaviour was that not all of the failure paths in > xdir_scan() were set errno, but the caller assumed it. In fact this sentence describes everything that I asked in [1]. However, I don't mind including more information. But it should be clear for a reader: see the suggestions below. [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019031.html > Usual C convention is to report success or failure using a return > value and set errno at any error. So a caller usually just checks a > return value and if it means a failure (usually -1), it checks errno > to determine an exact reason. > > Usual convention in tarantool is a bit different: we use a special > diagnostics area to report a reason of a failure. > > Not all failure paths of xdir_scan() sets errno (including our > 'invalid instance UUID' case), so we cannot be sure that errno is > not remains unchanged after a failure of the function. > > However the solution with checking errno against ENOENT (No such file > or directory) is not good. For example: > > - What if xdir_scan() would be changed in future and, say, some call > will rewrite errno after the opendir() call? > - What if some other call inside xdir_scan() will set ENOENT: say, > open() in xdir_open_cursor() due to some race? > > We lean on implementation details of the callee, not its contract. This > way is too fragile and it should either check whether the directory > exists before xdir_scan() call or pass a flag to xdir_scan() whether > the directory should exist. Decided to use second variant - it does not > lead to code duplication. (Yes, there is a lot of text below. But I'm sure it deserves a time to read it. I hope it sounds friendly, not edgy.) This part of the commit message is like a bag of facts. But what is the idea or ideas you want to express? I'll give a couple of examples from your message. There are facts about failure reporting conventions. But why it is highlighted here? No conclusions, no connections to the rest of the message. Or, say, copy of parts of the discussion about possible solutions: `errno = 0;` vs `is_dir_required` without mention of the former. A reader has no chance to understand why it is written here. I don't want to continue pointing places, where a reader will be confused: you can (and should) do it youself. It is better to give general suggestions, how to explain things. I'm not much experienced here, to be honest, but there are points I would share. First of all, I strongly suggest to avoid copying of another person wording. When you write an explanation youself, you will ask youself questions about it. Whether it is correct? Whether it gives ideas you want to express? How to better structurize it? The only way to improve this skill is practice. I suggest to decide about core ideas, read all materials (code, test results, discussions), but than move the maretials aside and explain the ideas in your words. This is the main point of this email, so I'll stop here again. Don't copy-paste. My past review comments had sense in its context, but just confuses a reader here. Some of them were given as background information for you, some comments compare different ways to solve the problem. It is the information for you: to analyze and decide how to make your patch better. So, again: decide about core ideas and express them. In your words. Of course, you may find youself on the point "I don't know how to express it". That's the key moment. Here I usually ask myself whether I really understand what is going on. I got back to the code and materials, re-read them, perform additional tests, experiment with different solutions. If things become clear for me after this, I can continue describing them. But sometimes it appears that I'm unable to defend a choosen way to solve a problem and I'm going to rewrite my code. It may also appear that my description becomes large and vague: an idea is missed between details. Maybe cut it off entirely? Or give just core idea, without details? Or structurize to make it clear where the idea itself and where details that can be skipped by a reader? The last suggestion is to track context of a reader. Imagine that you know nothing about the problem. After the first paragraph you got ideas it gives. The next paragraph should be clear in this context. For example, it may start with: - 'There are pitfalls a developer should aware', - 'There are alternative solutions', - 'The implementation lean on the following assumptions', - 'The problem appears only under the following conditions', - or, say, just 'Usage example', - maybe even just 'Usual convention is' if it is connected to the general context later. All those clauses marks how paragraphs (ideas, in fact) are connected to each other and allows a reader to don't lose context while reading. > Added subtest to box-tap/cfg.test.lua test file, to check the currently > fixed issue. This is redundant, IMHO. Almost every change should be accompanied by a test case. > diff --git a/test/box-tap/cfg.test.lua b/test/box-tap/cfg.test.lua > index 569b5f463..a60aa848e 100755 > --- a/test/box-tap/cfg.test.lua > +++ b/test/box-tap/cfg.test.lua Our guidelines suggest to extract a bugfix test into a separate file. From avtikhon at tarantool.org Mon Aug 17 08:29:14 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Mon, 17 Aug 2020 08:29:14 +0300 Subject: [Tarantool-patches] [PATCH v5 1/2] vinyl: fix check vinyl_dir existence at bootstrap Message-ID: <07d6fd4508eda75a98bb9ea49dd58b6b14fbd99a.1597641988.git.avtikhon@tarantool.org> During implementation of openSUSE got failed box-tap/cfg.test.lua test. Found that when memtx_dir didn't exist and vinyl_dir existed and also errno was set to ENOENT, box configuration succeeded, but it shouldn't. Reason of this wrong behaviour was that not all of the failure paths in xdir_scan() set errno, but the caller assumed it. Debugging src/box/xlog.c found that all checks were correct, but at: src/box/vy_log.c:vy_log_bootstrap() src/box/vy_log.c:vy_log_begin_recovery() the checks on of the errno on ENOENT blocked the negative return from: src/box/xlog.c:xdir_scan() Found that errno was already set to ENOENT before the xdir_scan() call. To fix the issue the errno could be clean before the call to xdir_scan, because we are interesting in it only from xdir_scan function. After discussions found that there were alternative better solution to fix it. The fix with resetting errno was not good because xdir_scan() was not system call in real and some internal routines could set it to ENOENT itself, so it couldn't be controled from outside of function. To be sure in behaviour of the changing errno decided to pass a flag to xdir_scan() if the directory should exist. Closes #4594 Needed for #4562 Co-authored-by: Alexander Turenko --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci Issue: https://github.com/tarantool/tarantool/issues/4562 src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 +- src/box/vy_log.c | 4 +- src/box/wal.c | 2 +- src/box/xlog.c | 4 +- src/box/xlog.h | 6 +-- .../gh-4562-errno-at-xdir_scan.test.lua | 47 +++++++++++++++++++ 7 files changed, 59 insertions(+), 10 deletions(-) create mode 100755 test/box-tap/gh-4562-errno-at-xdir_scan.test.lua diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..da3c50e87 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..2b894d680 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -559,7 +559,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..74f761994 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..3400eb75f 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); * snapshot or scan through all logs. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua new file mode 100755 index 000000000..cbf7b1f35 --- /dev/null +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua @@ -0,0 +1,47 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local test = tap.test('cfg') +local fio = require('fio') +test:plan(1) + +local tarantool_bin = arg[-1] +local PANIC = 256 + +function run_script(code) + local dir = fio.tempdir() + local script_path = fio.pathjoin(dir, 'script.lua') + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, + tonumber('0777', 8)) + script:write(code) + script:write("\nos.exit(0)") + script:close() + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] + local res = os.execute(string.format(cmd, dir, tarantool_bin)) + fio.rmtree(dir) + return res +end + +-- +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and +-- errno is set to ENOENT, box configuration succeeds, however it +-- should not +-- +vinyl_dir = fio.tempdir() +run_script(string.format([[ +box.cfg{vinyl_dir = '%s'} +s = box.schema.space.create('test', {engine = 'vinyl'}) +s:create_index('pk') +os.exit(0) +]], vinyl_dir)) +code = string.format([[ +local errno = require('errno') +errno(errno.ENOENT) +box.cfg{vinyl_dir = '%s'} +os.exit(0) +]], vinyl_dir) +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") +fio.rmtree(vinyl_dir) + +test:check() +os.exit(0) -- 2.17.1 From avtikhon at tarantool.org Mon Aug 17 08:33:21 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Mon, 17 Aug 2020 08:33:21 +0300 Subject: [Tarantool-patches] [PATCH v4 1/2] vinyl: check vinyl_dir existence at bootstrap In-Reply-To: <20200816212659.t4q55vctby3imgd7@tkn_work_nb> References: <4eb7b26b9f80393b19536f49ab3985d72ba13d89.1597398597.git.avtikhon@tarantool.org> <20200816212659.t4q55vctby3imgd7@tkn_work_nb> Message-ID: <20200817053321.GA13807@hpalx> Hi Alexander, thanks a lot for your comments, I've corrected topic message of the commit as you suggested and completely rewrote the commit message in myown words. Also test split from the common one. On Mon, Aug 17, 2020 at 12:26:59AM +0300, Alexander Turenko wrote: > > vinyl: check vinyl_dir existence at bootstrap > > It looks like you added the check, but it was present before the change. > 'Fix check' better describes the change. > > > During implementation of openSUSE got failed box-tap/cfg.test.lua test. > > Found that when memtx_dir wasn't existed, while vinyl_dir existed and > > errno was set to ENOENT, box configuration succeeded, but it shouldn't. > > Reason of this wrong behaviour was that not all of the failure paths in > > xdir_scan() were set errno, but the caller assumed it. > > In fact this sentence describes everything that I asked in [1]. > However, I don't mind including more information. But it should be clear > for a reader: see the suggestions below. > > [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019031.html > > > Usual C convention is to report success or failure using a return > > value and set errno at any error. So a caller usually just checks a > > return value and if it means a failure (usually -1), it checks errno > > to determine an exact reason. > > > > Usual convention in tarantool is a bit different: we use a special > > diagnostics area to report a reason of a failure. > > > > Not all failure paths of xdir_scan() sets errno (including our > > 'invalid instance UUID' case), so we cannot be sure that errno is > > not remains unchanged after a failure of the function. > > > > However the solution with checking errno against ENOENT (No such file > > or directory) is not good. For example: > > > > - What if xdir_scan() would be changed in future and, say, some call > > will rewrite errno after the opendir() call? > > - What if some other call inside xdir_scan() will set ENOENT: say, > > open() in xdir_open_cursor() due to some race? > > > > We lean on implementation details of the callee, not its contract. This > > way is too fragile and it should either check whether the directory > > exists before xdir_scan() call or pass a flag to xdir_scan() whether > > the directory should exist. Decided to use second variant - it does not > > lead to code duplication. > > (Yes, there is a lot of text below. But I'm sure it deserves a time to > read it. I hope it sounds friendly, not edgy.) > > This part of the commit message is like a bag of facts. But what is the > idea or ideas you want to express? > > I'll give a couple of examples from your message. There are facts about > failure reporting conventions. But why it is highlighted here? No > conclusions, no connections to the rest of the message. Or, say, copy of > parts of the discussion about possible solutions: `errno = 0;` vs > `is_dir_required` without mention of the former. A reader has no chance > to understand why it is written here. > > I don't want to continue pointing places, where a reader will be > confused: you can (and should) do it youself. It is better to give > general suggestions, how to explain things. I'm not much experienced > here, to be honest, but there are points I would share. > > First of all, I strongly suggest to avoid copying of another person > wording. When you write an explanation youself, you will ask youself > questions about it. Whether it is correct? Whether it gives ideas you > want to express? How to better structurize it? The only way to improve > this skill is practice. > > I suggest to decide about core ideas, read all materials (code, test > results, discussions), but than move the maretials aside and explain the > ideas in your words. > > This is the main point of this email, so I'll stop here again. Don't > copy-paste. My past review comments had sense in its context, but just > confuses a reader here. Some of them were given as background > information for you, some comments compare different ways to solve the > problem. It is the information for you: to analyze and decide how to > make your patch better. So, again: decide about core ideas and express > them. In your words. > > Of course, you may find youself on the point "I don't know how to > express it". That's the key moment. Here I usually ask myself whether I > really understand what is going on. I got back to the code and > materials, re-read them, perform additional tests, experiment with > different solutions. If things become clear for me after this, I can > continue describing them. But sometimes it appears that I'm unable to > defend a choosen way to solve a problem and I'm going to rewrite my > code. > > It may also appear that my description becomes large and vague: an idea > is missed between details. Maybe cut it off entirely? Or give just core > idea, without details? Or structurize to make it clear where the idea > itself and where details that can be skipped by a reader? > > The last suggestion is to track context of a reader. Imagine that you > know nothing about the problem. After the first paragraph you got ideas > it gives. The next paragraph should be clear in this context. For > example, it may start with: > > - 'There are pitfalls a developer should aware', > - 'There are alternative solutions', > - 'The implementation lean on the following assumptions', > - 'The problem appears only under the following conditions', > - or, say, just 'Usage example', > - maybe even just 'Usual convention is' if it is connected to the > general context later. > > All those clauses marks how paragraphs (ideas, in fact) are connected to > each other and allows a reader to don't lose context while reading. > > > Added subtest to box-tap/cfg.test.lua test file, to check the currently > > fixed issue. > > This is redundant, IMHO. Almost every change should be accompanied by a > test case. > > > diff --git a/test/box-tap/cfg.test.lua b/test/box-tap/cfg.test.lua > > index 569b5f463..a60aa848e 100755 > > --- a/test/box-tap/cfg.test.lua > > +++ b/test/box-tap/cfg.test.lua > > Our guidelines suggest to extract a bugfix test into a separate file. From gorcunov at gmail.com Mon Aug 17 11:02:34 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 11:02:34 +0300 Subject: [Tarantool-patches] [PATCH 1/1] applier: drop a couple of unnecessary arguments In-Reply-To: <7e16e45e171cb34edaff69b0f660421f1b67b4a1.1597504571.git.v.shpilevoy@tarantool.org> References: <7e16e45e171cb34edaff69b0f660421f1b67b4a1.1597504571.git.v.shpilevoy@tarantool.org> Message-ID: <20200817080234.GH2074@grain> On Sat, Aug 15, 2020 at 05:16:44PM +0200, Vladislav Shpilevoy wrote: > Applier on_rollback and on_wal_write don't need any arguments - > they either work with a global state, or with the signaled applier > stored inside the trigger. > > However into on_wal_write() and on_rollback() was passed the > transaction object, unused. > > Even if it would be used, it should have been fixed, because soon > these triggers will be fired not only for traditional 'txn' > transactions. They will be used by the synchro request WAL writes > too - they don't have 'transactions'. > > Part of #5129 > --- Acked-by: Cyrill Gorcunov From gorcunov at gmail.com Mon Aug 17 11:03:03 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 11:03:03 +0300 Subject: [Tarantool-patches] [PATCH v7 9/8] txn: txn_add_redo -- drop synchro processing In-Reply-To: <5618a6d1-1b65-a24e-e6d9-6332882f3fdb@tarantool.org> References: <20200815083851.GD2074@grain> <5618a6d1-1b65-a24e-e6d9-6332882f3fdb@tarantool.org> Message-ID: <20200817080303.GI2074@grain> On Sat, Aug 15, 2020 at 05:06:09PM +0200, Vladislav Shpilevoy wrote: > > xrow_header_dup_body() can be deleted now. Thanks! From gorcunov at gmail.com Mon Aug 17 15:42:35 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 15:42:35 +0300 Subject: [Tarantool-patches] [PATCH v7 7/8] applier: process synchro requests without txn engine In-Reply-To: References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-8-gorcunov@gmail.com> Message-ID: <20200817124235.GJ2074@grain> On Sat, Aug 15, 2020 at 05:06:05PM +0200, Vladislav Shpilevoy wrote: > > +static struct synchro_entry * > > +synchro_entry_new(struct applier *applier, > > + struct xrow_header *applier_row, > > + struct synchro_request *req) > > +{ > > + struct synchro_entry *entry; > > + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); > > 6. Why don't you just add 'struct xrow_header*[1]' to the end of > struct synchro_entry? There is no a case, when the entry is needed > without the xrow_header pointer in the end. This is forbidden by asan and some other compilers we've in travis runs. I've been already trying. Thanks for all comments, Vlad. I've merged your changes. Once tests are passed I'll send new series out. From gorcunov at gmail.com Mon Aug 17 16:39:09 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:09 +0300 Subject: [Tarantool-patches] [PATCH v8 0/9] qsync: write CONFIRM/ROLLBACK without txn engine Message-ID: <20200817133918.875558-1-gorcunov@gmail.com> In this series we write CONFIRM/ROLLBACK messages into the WAL directly without involving the txn engine. Vlad, take a look please, once time permit. I merged your changes into "applier: process synchro requests without txn engine" and dropped separate latch locking helper. Still accessing first and last entries in a queue is done via separate helpers since it is a way better look. First 4 patches you've read already and hopefully I addressed all your comments. issue https://github.com/tarantool/tarantool/issues/5129 branch gorcunov/gh-5129-journal-8 v3: - bootstrap journal left NULL for async write - journal_write_async_cb_t type for async callback - struct synchro_body_bin type for encoded message - xrow_encode_synchro helper to operate with synchro_body_bin v7: - rebase on master - rework applier code v8: - move synchro requests processing into applier_apply_tx (by Vlad) - drop synchro processing from txn_add_redo Cyrill Gorcunov (8): journal: bind asynchronous write completion to an entry journal: add journal_entry_create helper qsync: provide a binary form of syncro entries qsync: direct write of CONFIRM/ROLLBACK into a journal applier: add shorthands to queue access applier: process synchro requests without txn engine txn: txn_add_redo -- drop synchro processing xrow: drop xrow_header_dup_body Vladislav Shpilevoy (1): xrow: introduce struct synchro_request src/box/applier.cc | 237 +++++++++++++++++++++++++++++++++----------- src/box/box.cc | 36 +++---- src/box/journal.c | 8 +- src/box/journal.h | 36 +++++-- src/box/txn.c | 11 +- src/box/txn_limbo.c | 123 +++++++++++++---------- src/box/txn_limbo.h | 15 +-- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++-- src/box/wal.h | 4 +- src/box/xrow.c | 94 +++++------------- src/box/xrow.h | 85 ++++++++-------- 12 files changed, 377 insertions(+), 293 deletions(-) -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:10 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:10 +0300 Subject: [Tarantool-patches] [PATCH v8 1/9] xrow: introduce struct synchro_request In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-2-gorcunov@gmail.com> From: Vladislav Shpilevoy All requests saved to WAL and transmitted through network have their own request structure with parameters: - struct request for DML; - struct call_request for CALL/EVAL; - struct auth_request for AUTH; - struct ballot for VOTE; - struct sql_request for SQL; - struct greeting for greeting. It is done for a reason - not to pass all the request parameters into each function one by one, and manage them all at once instead. For synchronous requests IPROTO_CONFIRM and IPROTO_ROLLBACK it was not done. Because so far it was not too hard to carry just 2 parameters: lsn and replica_id, from their body. But it will be changed in #5129. Because in fact these requests have more parameters, but they were filled by txn module, since synchro requests were saved to WAL via transactions (due to lack of alternative API to access WAL). After #5129 it will be necessary to save LSN and replica_id of the request author. This patch introduces struct synchro_request to simplify extension of the synchro parameters. Closes #5151 Needed for #5129 --- src/box/applier.cc | 32 +++++------------------ src/box/box.cc | 21 ++++----------- src/box/txn_limbo.c | 56 ++++++++++++++++++++++++++-------------- src/box/txn_limbo.h | 15 +++-------- src/box/xrow.c | 52 +++++++++---------------------------- src/box/xrow.h | 63 +++++++++++++++++++++------------------------ 6 files changed, 93 insertions(+), 146 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index a953d293e..98fb87375 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -275,26 +275,14 @@ process_nop(struct request *request) * or rollback some of its entries. */ static int -process_confirm_rollback(struct request *request, bool is_confirm) +process_synchro_row(struct request *request) { assert(iproto_type_is_synchro_request(request->header->type)); - uint32_t replica_id; struct txn *txn = in_txn(); - int64_t lsn = 0; - int res = 0; - if (is_confirm) - res = xrow_decode_confirm(request->header, &replica_id, &lsn); - else - res = xrow_decode_rollback(request->header, &replica_id, &lsn); - if (res == -1) - return -1; - - if (replica_id != txn_limbo.instance_id) { - diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, replica_id, - txn_limbo.instance_id); + struct synchro_request syn_req; + if (xrow_decode_synchro(request->header, &syn_req) != 0) return -1; - } assert(txn->n_applier_rows == 0); /* * This is not really a transaction. It just uses txn API @@ -306,16 +294,9 @@ process_confirm_rollback(struct request *request, bool is_confirm) if (txn_begin_stmt(txn, NULL) != 0) return -1; - - if (txn_commit_stmt(txn, request) == 0) { - if (is_confirm) - txn_limbo_read_confirm(&txn_limbo, lsn); - else - txn_limbo_read_rollback(&txn_limbo, lsn); - return 0; - } else { + if (txn_commit_stmt(txn, request) != 0) return -1; - } + return txn_limbo_process(&txn_limbo, &syn_req); } static int @@ -324,8 +305,7 @@ apply_row(struct xrow_header *row) struct request request; if (iproto_type_is_synchro_request(row->type)) { request.header = row; - return process_confirm_rollback(&request, - row->type == IPROTO_CONFIRM); + return process_synchro_row(&request); } if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; diff --git a/src/box/box.cc b/src/box/box.cc index 83eef5d98..8e811e9c1 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -367,22 +367,11 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) { struct request request; if (iproto_type_is_synchro_request(row->type)) { - uint32_t replica_id; - int64_t lsn; - switch(row->type) { - case IPROTO_CONFIRM: - if (xrow_decode_confirm(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_confirm(&txn_limbo, lsn); - break; - case IPROTO_ROLLBACK: - if (xrow_decode_rollback(row, &replica_id, &lsn) < 0) - diag_raise(); - assert(txn_limbo.instance_id == replica_id); - txn_limbo_read_rollback(&txn_limbo, lsn); - break; - } + struct synchro_request syn_req; + if (xrow_decode_synchro(row, &syn_req) != 0) + diag_raise(); + if (txn_limbo_process(&txn_limbo, &syn_req) != 0) + diag_raise(); return; } xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index a2043c17a..944161c30 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -31,6 +31,7 @@ #include "txn.h" #include "txn_limbo.h" #include "replication.h" +#include "iproto_constants.h" struct txn_limbo txn_limbo; @@ -272,11 +273,15 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) } static void -txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, - bool is_confirm) +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { assert(lsn > 0); + struct synchro_request req; + req.type = type; + req.replica_id = limbo->instance_id; + req.lsn = lsn; + struct xrow_header row; struct request request = { .header = &row, @@ -286,19 +291,7 @@ txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, if (txn == NULL) goto rollback; - int res = 0; - if (is_confirm) { - res = xrow_encode_confirm(&row, &txn->region, - limbo->instance_id, lsn); - } else { - /* - * This LSN is the first to be rolled back, so - * the last "safe" lsn is lsn - 1. - */ - res = xrow_encode_rollback(&row, &txn->region, - limbo->instance_id, lsn); - } - if (res == -1) + if (xrow_encode_synchro(&row, &txn->region, &req) != 0) goto rollback; /* * This is not really a transaction. It just uses txn API @@ -325,7 +318,7 @@ txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, * problems are fixed. Or retry automatically with some period. */ panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, is_confirm ? "CONFIRM" : "ROLLBACK"); + "%s\n", lsn, iproto_type_name(type)); } /** @@ -338,10 +331,11 @@ txn_limbo_write_confirm(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->confirmed_lsn = lsn; - txn_limbo_write_confirm_rollback(limbo, lsn, true); + txn_limbo_write_synchro(limbo, IPROTO_CONFIRM, lsn); } -void +/** Confirm all the entries <= @a lsn. */ +static void txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -390,11 +384,12 @@ txn_limbo_write_rollback(struct txn_limbo *limbo, int64_t lsn) assert(lsn > limbo->confirmed_lsn); assert(!limbo->is_in_rollback); limbo->is_in_rollback = true; - txn_limbo_write_confirm_rollback(limbo, lsn, false); + txn_limbo_write_synchro(limbo, IPROTO_ROLLBACK, lsn); limbo->is_in_rollback = false; } -void +/** Rollback all the entries >= @a lsn. */ +static void txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn) { assert(limbo->instance_id != REPLICA_ID_NIL); @@ -577,6 +572,27 @@ txn_limbo_wait_confirm(struct txn_limbo *limbo) return 0; } +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) +{ + if (req->replica_id != limbo->instance_id) { + diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, req->replica_id, + limbo->instance_id); + return -1; + } + switch (req->type) { + case IPROTO_CONFIRM: + txn_limbo_read_confirm(limbo, req->lsn); + break; + case IPROTO_ROLLBACK: + txn_limbo_read_rollback(limbo, req->lsn); + break; + default: + unreachable(); + } + return 0; +} + void txn_limbo_force_empty(struct txn_limbo *limbo, int64_t confirm_lsn) { diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h index 04ee7ea5c..eaf662987 100644 --- a/src/box/txn_limbo.h +++ b/src/box/txn_limbo.h @@ -39,6 +39,7 @@ extern "C" { #endif /* defined(__cplusplus) */ struct txn; +struct synchro_request; /** * Transaction and its quorum metadata, to be stored in limbo. @@ -245,17 +246,9 @@ txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn); int txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry); -/** - * Confirm all the entries up to the given master's LSN. - */ -void -txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn); - -/** - * Rollback all the entries starting with given master's LSN. - */ -void -txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn); +/** Execute a synchronous replication request. */ +int +txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req); /** * Waiting for confirmation of all "sync" transactions diff --git a/src/box/xrow.c b/src/box/xrow.c index 0c797a9d5..4b5d4356f 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,13 +893,13 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -static int -xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn, int type) +int +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req) { size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(replica_id) + mp_sizeof_uint(IPROTO_LSN) + - mp_sizeof_uint(lsn); + mp_sizeof_uint(req->replica_id) + + mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); char *buf = (char *)region_alloc(region, len); if (buf == NULL) { diag_set(OutOfMemory, len, "region_alloc", "buf"); @@ -909,9 +909,9 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, pos = mp_encode_map(pos, 2); pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, replica_id); + pos = mp_encode_uint(pos, req->replica_id); pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, lsn); + pos = mp_encode_uint(pos, req->lsn); memset(row, 0, sizeof(*row)); @@ -919,30 +919,13 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, row->body[0].iov_len = len; row->bodycnt = 1; - row->type = type; + row->type = req->type; return 0; } int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_CONFIRM); -} - -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn) -{ - return xrow_encode_confirm_rollback(row, region, replica_id, lsn, - IPROTO_ROLLBACK); -} - -static int -xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, - int64_t *lsn) +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req) { if (row->bodycnt == 0) { diag_set(ClientError, ER_INVALID_MSGPACK, "request body"); @@ -977,30 +960,19 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, } switch (key) { case IPROTO_REPLICA_ID: - *replica_id = mp_decode_uint(&d); + req->replica_id = mp_decode_uint(&d); break; case IPROTO_LSN: - *lsn = mp_decode_uint(&d); + req->lsn = mp_decode_uint(&d); break; default: mp_next(&d); } } + req->type = row->type; return 0; } -int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - -int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) -{ - return xrow_decode_confirm_rollback(row, replica_id, lsn); -} - int xrow_to_iovec(const struct xrow_header *row, struct iovec *out) { diff --git a/src/box/xrow.h b/src/box/xrow.h index e21ede5a3..02dca74e5 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -216,54 +216,51 @@ xrow_encode_dml(const struct request *request, struct region *region, struct iovec *iov); /** - * Encode the CONFIRM to row body and set row type to - * IPROTO_CONFIRM. - * @param row xrow header. - * @param region Region to use to encode the confirmation body. - * @param replica_id master's instance id. - * @param lsn last confirmed lsn. - * @retval -1 on error. - * @retval 0 success. + * Synchronous replication request - confirmation or rollback of + * pending synchronous transactions. */ -int -xrow_encode_confirm(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +struct synchro_request { + /** Operation type - IPROTO_ROLLBACK or IPROTO_CONFIRM. */ + uint32_t type; + /** + * ID of the instance owning the pending transactions. + * Note, it may be not the same instance, who created this + * request. An instance can make an operation on foreign + * synchronous transactions in case a new master tries to + * finish transactions of an old master. + */ + uint32_t replica_id; + /** + * Operation LSN. + * In case of CONFIRM it means 'confirm all + * transactions with lsn <= this value'. + * In case of ROLLBACK it means 'rollback all transactions + * with lsn >= this value'. + */ + int64_t lsn; +}; /** - * Decode the CONFIRM request body. + * Encode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn last confirmed lsn. + * @param region Region to use to encode the confirmation body. + * @param req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); - -/** - * Encode the ROLLBACK row body and set row type to - * IPROTO_ROLLBACK. - * @param row xrow header. - * @param region Region to use to encode the rollback body. - * @param replica_id master's instance id. - * @param lsn lsn to rollback from, including it. - * @retval -1 on error. - * @retval 0 success. - */ -int -xrow_encode_rollback(struct xrow_header *row, struct region *region, - uint32_t replica_id, int64_t lsn); +xrow_encode_synchro(struct xrow_header *row, struct region *region, + const struct synchro_request *req); /** - * Decode the ROLLBACK row body. + * Decode synchronous replication request. * @param row xrow header. - * @param[out] replica_id master's instance id. - * @param[out] lsn lsn to rollback from, including it. + * @param[out] req Request parameters. * @retval -1 on error. * @retval 0 success. */ int -xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); +xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); /** * CALL/EVAL request. -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:11 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:11 +0300 Subject: [Tarantool-patches] [PATCH v8 2/9] journal: bind asynchronous write completion to an entry In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-3-gorcunov@gmail.com> In commit 77ba0e3504464131fe81c672d508d0275be2173a we've redesigned wal journal operations such that asynchronous write completion is a single instance per journal. It turned out that such simplification is too tight and doesn't allow us to pass entries into the journal with custom completions. Thus lets allow back such ability. We will need it to be able to write "confirm" records into wal directly without touching transactions code at all. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/box.cc | 15 ++++++++------- src/box/journal.c | 2 ++ src/box/journal.h | 20 +++++++++++--------- src/box/txn.c | 2 +- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++++++++----------- src/box/wal.h | 4 ++-- 7 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index 8e811e9c1..faffd5769 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -348,7 +348,7 @@ recovery_journal_write(struct journal *base, * Since there're no actual writes, fire a * journal_async_complete callback right away. */ - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -357,7 +357,7 @@ recovery_journal_create(struct vclock *v) { static struct recovery_journal journal; journal_create(&journal.base, recovery_journal_write, - txn_complete_async, recovery_journal_write); + recovery_journal_write); journal.vclock = v; journal_set(&journal.base); } @@ -2182,8 +2182,10 @@ engine_init() static int bootstrap_journal_write(struct journal *base, struct journal_entry *entry) { + (void)base; + entry->res = 0; - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -2569,8 +2571,8 @@ box_cfg_xc(void) int64_t wal_max_size = box_check_wal_max_size(cfg_geti64("wal_max_size")); enum wal_mode wal_mode = box_check_wal_mode(cfg_gets("wal_mode")); - if (wal_init(wal_mode, txn_complete_async, cfg_gets("wal_dir"), - wal_max_size, &INSTANCE_UUID, on_wal_garbage_collection, + if (wal_init(wal_mode, cfg_gets("wal_dir"), wal_max_size, + &INSTANCE_UUID, on_wal_garbage_collection, on_wal_checkpoint_threshold) != 0) { diag_raise(); } @@ -2617,8 +2619,7 @@ box_cfg_xc(void) } struct journal bootstrap_journal; - journal_create(&bootstrap_journal, NULL, txn_complete_async, - bootstrap_journal_write); + journal_create(&bootstrap_journal, NULL, bootstrap_journal_write); journal_set(&bootstrap_journal); auto bootstrap_journal_guard = make_scoped_guard([] { journal_set(NULL); diff --git a/src/box/journal.c b/src/box/journal.c index f1e89aaa2..48af9157b 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -36,6 +36,7 @@ struct journal *current_journal = NULL; struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data) { struct journal_entry *entry; @@ -50,6 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } + entry->write_async_cb = write_async_cb; entry->complete_data = complete_data; entry->approx_len = 0; entry->n_rows = n_rows; diff --git a/src/box/journal.h b/src/box/journal.h index 1a10e66c3..4b019fecf 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -42,6 +42,8 @@ extern "C" { struct xrow_header; struct journal_entry; +typedef void (*journal_write_async_f)(struct journal_entry *entry); + /** * An entry for an abstract journal. * Simply put, a write ahead log request. @@ -61,6 +63,10 @@ struct journal_entry { * A journal entry completion callback argument. */ void *complete_data; + /** + * Asynchronous write completion function. + */ + journal_write_async_f write_async_cb; /** * Approximate size of this request when encoded. */ @@ -84,6 +90,7 @@ struct region; */ struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data); /** @@ -96,22 +103,19 @@ struct journal { int (*write_async)(struct journal *journal, struct journal_entry *entry); - /** Asynchronous write completion */ - void (*write_async_cb)(struct journal_entry *entry); - /** Synchronous write */ int (*write)(struct journal *journal, struct journal_entry *entry); }; /** - * Finalize a single entry. + * Complete asynchronous write. */ static inline void -journal_async_complete(struct journal *journal, struct journal_entry *entry) +journal_async_complete(struct journal_entry *entry) { - assert(journal->write_async_cb != NULL); - journal->write_async_cb(entry); + assert(entry->write_async_cb != NULL); + entry->write_async_cb(entry); } /** @@ -173,12 +177,10 @@ static inline void journal_create(struct journal *journal, int (*write_async)(struct journal *journal, struct journal_entry *entry), - void (*write_async_cb)(struct journal_entry *entry), int (*write)(struct journal *journal, struct journal_entry *entry)) { journal->write_async = write_async; - journal->write_async_cb = write_async_cb; journal->write = write; } diff --git a/src/box/txn.c b/src/box/txn.c index 9c21258c5..cc1f496c5 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -551,7 +551,7 @@ txn_journal_entry_new(struct txn *txn) /* Save space for an additional NOP row just in case. */ req = journal_entry_new(txn->n_new_rows + txn->n_applier_rows + 1, - &txn->region, txn); + &txn->region, txn_complete_async, txn); if (req == NULL) return NULL; diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..de4c5205c 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -818,7 +818,7 @@ vy_log_tx_flush(struct vy_log_tx *tx) size_t used = region_used(&fiber()->gc); struct journal_entry *entry; - entry = journal_entry_new(tx_size, &fiber()->gc, NULL); + entry = journal_entry_new(tx_size, &fiber()->gc, NULL, NULL); if (entry == NULL) goto err; diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..045006b60 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -266,10 +266,9 @@ xlog_write_entry(struct xlog *l, struct journal_entry *entry) static void tx_schedule_queue(struct stailq *queue) { - struct wal_writer *writer = &wal_writer_singleton; struct journal_entry *req, *tmp; stailq_foreach_entry_safe(req, tmp, queue, fifo) - journal_async_complete(&writer->base, req); + journal_async_complete(req); } /** @@ -403,9 +402,8 @@ tx_notify_checkpoint(struct cmsg *msg) */ static void wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, - void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, - int64_t wal_max_size, const struct tt_uuid *instance_uuid, + const char *wal_dirname, int64_t wal_max_size, + const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { @@ -415,7 +413,6 @@ wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, journal_create(&writer->base, wal_mode == WAL_NONE ? wal_write_none_async : wal_write_async, - wall_async_cb, wal_mode == WAL_NONE ? wal_write_none : wal_write); @@ -525,15 +522,15 @@ wal_open(struct wal_writer *writer) } int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { /* Initialize the state. */ struct wal_writer *writer = &wal_writer_singleton; - wal_writer_create(writer, wal_mode, wall_async_cb, wal_dirname, - wal_max_size, instance_uuid, on_garbage_collection, + wal_writer_create(writer, wal_mode, wal_dirname, wal_max_size, + instance_uuid, on_garbage_collection, on_checkpoint_threshold); /* Start WAL thread. */ @@ -1314,7 +1311,7 @@ wal_write_none_async(struct journal *journal, vclock_merge(&writer->vclock, &vclock_diff); vclock_copy(&replicaset.vclock, &writer->vclock); entry->res = vclock_sum(&writer->vclock); - journal_async_complete(journal, entry); + journal_async_complete(entry); return 0; } diff --git a/src/box/wal.h b/src/box/wal.h index f348dc636..9d0cada46 100644 --- a/src/box/wal.h +++ b/src/box/wal.h @@ -81,8 +81,8 @@ typedef void (*wal_on_checkpoint_threshold_f)(void); * Start WAL thread and initialize WAL writer. */ int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold); -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:12 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:12 +0300 Subject: [Tarantool-patches] [PATCH v8 3/9] journal: add journal_entry_create helper In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-4-gorcunov@gmail.com> To create raw journal entries. We will use it to write confirm/rollback entries. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/journal.c | 8 ++------ src/box/journal.h | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/src/box/journal.c b/src/box/journal.c index 48af9157b..cb320b557 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -51,11 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } - entry->write_async_cb = write_async_cb; - entry->complete_data = complete_data; - entry->approx_len = 0; - entry->n_rows = n_rows; - entry->res = -1; - + journal_entry_create(entry, n_rows, 0, write_async_cb, + complete_data); return entry; } diff --git a/src/box/journal.h b/src/box/journal.h index 4b019fecf..5d8d5a726 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -83,6 +83,22 @@ struct journal_entry { struct region; +/** + * Initialize a new journal entry. + */ +static inline void +journal_entry_create(struct journal_entry *entry, size_t n_rows, + size_t approx_len, + journal_write_async_f write_async_cb, + void *complete_data) +{ + entry->write_async_cb = write_async_cb; + entry->complete_data = complete_data; + entry->approx_len = approx_len; + entry->n_rows = n_rows; + entry->res = -1; +} + /** * Create a new journal entry. * -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:13 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:13 +0300 Subject: [Tarantool-patches] [PATCH v8 4/9] qsync: provide a binary form of syncro entries In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-5-gorcunov@gmail.com> These msgpack entries will be needed to write them down to a journal without involving txn engine. Same time we would like to be able to allocate them on stack, for this sake the binary form is predefined. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 9 +++++++-- src/box/xrow.c | 41 ++++++++++++++++++----------------------- src/box/xrow.h | 20 +++++++++++++++----- 3 files changed, 40 insertions(+), 30 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index 944161c30..ed8c10419 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -282,6 +282,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) req.replica_id = limbo->instance_id; req.lsn = lsn; + /* + * This is a synchronous commit so we can + * use body and row allocated on a stack. + */ + struct synchro_body_bin body; struct xrow_header row; struct request request = { .header = &row, @@ -291,8 +296,8 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) if (txn == NULL) goto rollback; - if (xrow_encode_synchro(&row, &txn->region, &req) != 0) - goto rollback; + xrow_encode_synchro(&row, &body, &req); + /* * This is not really a transaction. It just uses txn API * to put the data into WAL. And obviously it should not diff --git a/src/box/xrow.c b/src/box/xrow.c index 4b5d4356f..03a4abdda 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,35 +893,30 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req) { - size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(req->replica_id) + - mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); - char *buf = (char *)region_alloc(region, len); - if (buf == NULL) { - diag_set(OutOfMemory, len, "region_alloc", "buf"); - return -1; - } - char *pos = buf; - - pos = mp_encode_map(pos, 2); - pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, req->replica_id); - pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, req->lsn); + /* + * A map with two elements. We don't compress + * numbers to have this structure constant in size, + * which allows us to preallocate it on stack. + */ + body->m_body = 0x80 | 2; + body->k_replica_id = IPROTO_REPLICA_ID; + body->m_replica_id = 0xce; + body->v_replica_id = mp_bswap_u32(req->replica_id); + body->k_lsn = IPROTO_LSN; + body->m_lsn = 0xcf; + body->v_lsn = mp_bswap_u64(req->lsn); memset(row, 0, sizeof(*row)); - row->body[0].iov_base = buf; - row->body[0].iov_len = len; - row->bodycnt = 1; - row->type = req->type; - - return 0; + row->body[0].iov_base = (void *)body; + row->body[0].iov_len = sizeof(*body); + row->bodycnt = 1; } int diff --git a/src/box/xrow.h b/src/box/xrow.h index 02dca74e5..20e82034d 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -240,16 +240,26 @@ struct synchro_request { int64_t lsn; }; +/** Synchro request xrow's body in MsgPack format. */ +struct PACKED synchro_body_bin { + uint8_t m_body; + uint8_t k_replica_id; + uint8_t m_replica_id; + uint32_t v_replica_id; + uint8_t k_lsn; + uint8_t m_lsn; + uint64_t v_lsn; +}; + /** * Encode synchronous replication request. * @param row xrow header. - * @param region Region to use to encode the confirmation body. + * @param body Desination to use to encode the confirmation body. * @param req Request parameters. - * @retval -1 on error. - * @retval 0 success. */ -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req); /** -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:14 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:14 +0300 Subject: [Tarantool-patches] [PATCH v8 5/9] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-6-gorcunov@gmail.com> When we need to write CONFIRM or ROLLBACK message (which is a binary record in msgpack format) into a journal we use txn code to allocate a new transaction, encode there a message and pass it to walk the long txn path before it hit the journal. This is not only resource wasting but also somehow strange from architectural point of view. Instead lets encode a record on the stack and write it to the journal directly. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 64 ++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index ed8c10419..447630d23 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -32,6 +32,7 @@ #include "txn_limbo.h" #include "replication.h" #include "iproto_constants.h" +#include "journal.h" struct txn_limbo txn_limbo; @@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) return 0; } +/** + * A callback for synchronous write: txn_limbo_write_synchro fiber + * waiting to proceed once a record is written to WAL. + */ +static void +txn_limbo_write_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + fiber_wakeup(entry->complete_data); +} + static void txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { @@ -284,46 +296,34 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) /* * This is a synchronous commit so we can - * use body and row allocated on a stack. + * allocate everything on a stack. */ struct synchro_body_bin body; struct xrow_header row; - struct request request = { - .header = &row, - }; + char buf[sizeof(struct journal_entry) + + sizeof(struct xrow_header *)]; - struct txn *txn = txn_begin(); - if (txn == NULL) - goto rollback; + struct journal_entry *entry = (struct journal_entry *)buf; + entry->rows[0] = &row; xrow_encode_synchro(&row, &body, &req); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - goto rollback; - if (txn_commit_stmt(txn, &request) != 0) - goto rollback; - if (txn_commit(txn) != 0) - goto rollback; - return; + journal_entry_create(entry, 1, xrow_approx_len(&row), + txn_limbo_write_cb, fiber()); -rollback: - /* - * XXX: the stub is supposed to be removed once it is defined what to do - * when a synchro request WAL write fails. One of the possible - * solutions: log the error, keep the limbo queue as is and probably put - * in rollback mode. Then provide a hook to call manually when WAL - * problems are fixed. Or retry automatically with some period. - */ - panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, iproto_type_name(type)); + if (journal_write(entry) != 0 || entry->res < 0) { + diag_set(ClientError, ER_WAL_IO); + diag_log(); + /* + * XXX: the stub is supposed to be removed once it is defined what to do + * when a synchro request WAL write fails. One of the possible + * solutions: log the error, keep the limbo queue as is and probably put + * in rollback mode. Then provide a hook to call manually when WAL + * problems are fixed. Or retry automatically with some period. + */ + panic("Could not write a synchro request to WAL: lsn = %lld, type = " + "%s\n", lsn, type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); + } } /** -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:15 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:15 +0300 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-7-gorcunov@gmail.com> We need to access first and last xrow in a queue frenquently and opencoded variants are too ugly. Lets provide shorthands. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 98fb87375..860a18681 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -648,6 +648,26 @@ struct applier_tx_row { struct xrow_header row; }; +/** + * Get first xrow from a list. + */ +static inline struct xrow_header * +applier_first_row(struct stailq *rows) +{ + return &stailq_first_entry(rows, + struct applier_tx_row, next)->row; +} + +/** + * Get last xrow from a list. + */ +static inline struct xrow_header * +applier_last_row(struct stailq *rows) +{ + return &stailq_last_entry(rows, + struct applier_tx_row, next)->row; +} + static struct applier_tx_row * applier_read_tx_row(struct applier *applier) { @@ -749,8 +769,7 @@ applier_read_tx(struct applier *applier, struct stailq *rows) } stailq_add_tail(rows, &tx_row->next); - } while (!stailq_last_entry(rows, struct applier_tx_row, - next)->row.is_commit); + } while (!applier_last_row(rows)->is_commit); } static int @@ -807,10 +826,8 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) static int applier_apply_tx(struct stailq *rows) { - struct xrow_header *first_row = &stailq_first_entry(rows, - struct applier_tx_row, next)->row; - struct xrow_header *last_row; - last_row = &stailq_last_entry(rows, struct applier_tx_row, next)->row; + struct xrow_header *first_row = applier_first_row(rows); + struct xrow_header *last_row = applier_last_row(rows); struct replica *replica = replica_by_id(first_row->replica_id); /* * In a full mesh topology, the same set of changes @@ -834,9 +851,7 @@ applier_apply_tx(struct stailq *rows) */ struct xrow_header *tmp; while (true) { - tmp = &stailq_first_entry(rows, - struct applier_tx_row, - next)->row; + tmp = applier_first_row(rows); if (tmp->lsn <= vclock_get(&replicaset.applier.vclock, tmp->replica_id)) { stailq_shift(rows); @@ -1122,8 +1137,7 @@ applier_subscribe(struct applier *applier) * In case of an heartbeat message wake a writer up * and check applier state. */ - if (stailq_first_entry(&rows, struct applier_tx_row, - next)->row.lsn == 0) + if (applier_first_row(&rows)->lsn == 0) applier_signal_ack(applier); else if (applier_apply_tx(&rows) != 0) diag_raise(); -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:16 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:16 +0300 Subject: [Tarantool-patches] [PATCH v8 7/9] applier: process synchro requests without txn engine In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-8-gorcunov@gmail.com> Transaction processing code is very heavy simply because transactions are carrying various data and involves a number of other mechanisms to proceed. In turn, when we receive confirm or rollback packed from another node in a cluster we just need to inspect limbo queue and write this packed into a WAL journal. So calling a bunch of txn engine helpers is simply waste of cycles. Thus lets rather handle them in a special light way: - allocate synchro_entry structure which would carry the journal entry itself and encoded message - process limbo queue to mark confirmed/rollback'ed messages - finally write this synchro_entry into a journal Which is a way simplier. Part-of #5129 Suggedsted-by: Vladislav Shpilevoy Co-developed-by: Vladislav Shpilevoy Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 169 +++++++++++++++++++++++++++++++++++++++------ 1 file changed, 149 insertions(+), 20 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 860a18681..83f6da461 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -51,8 +51,10 @@ #include "schema.h" #include "txn.h" #include "box.h" +#include "xrow.h" #include "scoped_guard.h" #include "txn_limbo.h" +#include "journal.h" STRS(applier_state, applier_STATE); @@ -772,19 +774,9 @@ applier_read_tx(struct applier *applier, struct stailq *rows) } while (!applier_last_row(rows)->is_commit); } -static int -applier_txn_rollback_cb(struct trigger *trigger, void *event) +static void +applier_rollback_by_wal_io(void) { - (void) trigger; - struct txn *txn = (struct txn *) event; - /* - * Synchronous transaction rollback due to receiving a - * ROLLBACK entry is a normal event and requires no - * special handling. - */ - if (txn->signature == TXN_SIGNATURE_SYNC_ROLLBACK) - return 0; - /* * Setup shared applier diagnostic area. * @@ -793,19 +785,32 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) * diag use per-applier diag instead all the time * (which actually already present in the structure). * - * But remember that transactions are asynchronous - * and rollback may happen a way latter after it - * passed to the journal engine. + * But remember that WAL writes are asynchronous and + * rollback may happen a way later after it was passed to + * the journal engine. */ diag_set(ClientError, ER_WAL_IO); diag_set_error(&replicaset.applier.diag, diag_last_error(diag_get())); - /* Broadcast the rollback event across all appliers. */ - trigger_run(&replicaset.applier.on_rollback, event); - + /* Broadcast the rollback across all appliers. */ + trigger_run(&replicaset.applier.on_rollback, NULL); /* Rollback applier vclock to the committed one. */ vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); +} + +static int +applier_txn_rollback_cb(struct trigger *trigger, void *event) +{ + (void) trigger; + struct txn *txn = (struct txn *) event; + /* + * Synchronous transaction rollback due to receiving a + * ROLLBACK entry is a normal event and requires no + * special handling. + */ + if (txn->signature != TXN_SIGNATURE_SYNC_ROLLBACK) + applier_rollback_by_wal_io(); return 0; } @@ -818,6 +823,110 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) return 0; } +struct synchro_entry { + /** Encoded form of a synchro record. */ + struct synchro_body_bin body_bin; + + /** xrow to write, used by the journal engine. */ + struct xrow_header row; + + /** + * The journal entry itself. Note since + * it has unsized array it must be the + * last entry in the structure. + */ + struct journal_entry journal_entry; +}; + +static void +synchro_entry_delete(struct synchro_entry *entry) +{ + free(entry); +} + +/** + * Async write journal completion. + */ +static void +apply_synchro_row_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + struct synchro_entry *synchro_entry = + (struct synchro_entry *)entry->complete_data; + if (entry->res < 0) + applier_rollback_by_wal_io(); + else + trigger_run(&replicaset.applier.on_wal_write, NULL); + + synchro_entry_delete(synchro_entry); +} + +/** + * Allocate a new synchro_entry to be passed to + * the journal engine in async write way. + */ +static struct synchro_entry * +synchro_entry_new(struct xrow_header *applier_row, + struct synchro_request *req) +{ + struct synchro_entry *entry; + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); + + /* + * For simplicity we use malloc here but + * probably should provide some cache similar + * to txn cache. + */ + entry = (struct synchro_entry *)malloc(size); + if (entry == NULL) { + diag_set(OutOfMemory, size, "malloc", "synchro_entry"); + return NULL; + } + + struct journal_entry *journal_entry = &entry->journal_entry; + struct synchro_body_bin *body_bin = &entry->body_bin; + struct xrow_header *row = &entry->row; + + journal_entry->rows[0] = row; + + xrow_encode_synchro(row, body_bin, req); + + row->lsn = applier_row->lsn; + row->replica_id = applier_row->replica_id; + + journal_entry_create(journal_entry, 1, xrow_approx_len(row), + apply_synchro_row_cb, entry); + return entry; +} + +/** Process a synchro request. */ +static int +apply_synchro_row(struct xrow_header *row) +{ + assert(iproto_type_is_synchro_request(row->type)); + + struct synchro_request req; + if (xrow_decode_synchro(row, &req) != 0) + goto err; + + if (txn_limbo_process(&txn_limbo, &req)) + goto err; + + struct synchro_entry *entry; + entry = synchro_entry_new(row, &req); + if (entry == NULL) + goto err; + + if (journal_write_async(&entry->journal_entry) != 0) { + diag_set(ClientError, ER_WAL_IO); + goto err; + } + return 0; +err: + diag_log(); + return -1; +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -861,13 +970,26 @@ applier_apply_tx(struct stailq *rows) } } + if (unlikely(iproto_type_is_synchro_request(first_row->type))) { + /* + * Synchro messages are not transactions, in terms + * of DML. Always sent and written isolated from + * each other. + */ + assert(first_row == last_row); + if (apply_synchro_row(first_row) != 0) + diag_raise(); + goto success; + } + /** * Explicitly begin the transaction so that we can * control fiber->gc life cycle and, in case of apply * conflict safely access failed xrow object and allocate * IPROTO_NOP on gc. */ - struct txn *txn = txn_begin(); + struct txn *txn; + txn = txn_begin(); struct applier_tx_row *item; if (txn == NULL) { latch_unlock(latch); @@ -936,6 +1058,7 @@ applier_apply_tx(struct stailq *rows) if (txn_commit_async(txn) < 0) goto fail; +success: /* * The transaction was sent to journal so promote vclock. * @@ -1103,7 +1226,13 @@ applier_subscribe(struct applier *applier) applier->lag = TIMEOUT_INFINITY; - /* Register triggers to handle WAL writes and rollbacks. */ + /* + * Register triggers to handle WAL writes and rollbacks. + * + * Note we use them for syncronous packets handling as well + * thus when changing make sure that synchro handling won't + * be broken. + */ struct trigger on_wal_write; trigger_create(&on_wal_write, applier_on_wal_write, applier, NULL); trigger_add(&replicaset.applier.on_wal_write, &on_wal_write); -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:17 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:17 +0300 Subject: [Tarantool-patches] [PATCH v8 8/9] txn: txn_add_redo -- drop synchro processing In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-9-gorcunov@gmail.com> Since we no longer use txn engine for synchro packets processing this code is never executed. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/src/box/txn.c b/src/box/txn.c index cc1f496c5..b2d342355 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -82,14 +82,7 @@ txn_add_redo(struct txn *txn, struct txn_stmt *stmt, struct request *request) */ struct space *space = stmt->space; row->group_id = space != NULL ? space_group_id(space) : 0; - /* - * Sychronous replication entries are supplementary and - * aren't valid dml requests. They're encoded manually. - */ - if (likely(!iproto_type_is_synchro_request(row->type))) - row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); - else - row->bodycnt = xrow_header_dup_body(row, &txn->region); + row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); if (row->bodycnt < 0) return -1; stmt->row = row; -- 2.26.2 From gorcunov at gmail.com Mon Aug 17 16:39:18 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 17 Aug 2020 16:39:18 +0300 Subject: [Tarantool-patches] [PATCH v8 9/9] xrow: drop xrow_header_dup_body In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <20200817133918.875558-10-gorcunov@gmail.com> We no longer use it. Closes #5129 Signed-off-by: Cyrill Gorcunov --- src/box/xrow.c | 15 --------------- src/box/xrow.h | 8 -------- 2 files changed, 23 deletions(-) diff --git a/src/box/xrow.c b/src/box/xrow.c index 03a4abdda..9aa2fae00 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -220,21 +220,6 @@ xrow_header_decode(struct xrow_header *header, const char **pos, return 0; } -int -xrow_header_dup_body(struct xrow_header *row, struct region *region) -{ - assert(row->bodycnt == 1); - size_t size = row->body[0].iov_len; - char *copy = (char *)region_alloc(region, size); - if (copy == NULL) { - diag_set(OutOfMemory, size, "region_alloc", "copy"); - return -1; - } - memcpy(copy, row->body[0].iov_base, size); - row->body[0].iov_base = copy; - return 1; -} - /** * @pre pos points at a valid msgpack */ diff --git a/src/box/xrow.h b/src/box/xrow.h index 20e82034d..58d47b12d 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -141,14 +141,6 @@ int xrow_header_decode(struct xrow_header *header, const char **pos, const char *end, bool end_is_exact); -/** - * Duplicate the xrow's body onto the given region. - * @retval -1 Error. - * @retval >= 0 Iov count in the body. - */ -int -xrow_header_dup_body(struct xrow_header *header, struct region *region); - /** * DML request. */ -- 2.26.2 From sergepetrenko at tarantool.org Mon Aug 17 18:51:00 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Mon, 17 Aug 2020 18:51:00 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_1/1=5D_xrow=3A_introduce_s?= =?utf-8?q?truct_synchro=5Frequest?= In-Reply-To: References: <20200813221757.GB2074@grain> Message-ID: <1597679460.185815010@f745.i.mail.ru> ? Hi! Thanks for the patch! LGTM. ? ? ? ? >???????, 15 ??????? 2020, 16:24 +03:00 ?? Vladislav Shpilevoy : >? >Hi! Thanks for the review! > >On 14.08.2020 00:17, Cyrill Gorcunov wrote: >> On Thu, Aug 13, 2020 at 11:58:20PM +0200, Vladislav Shpilevoy wrote: >> ... >>> +txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >>> { >>> assert(lsn > 0); >>> >>> + struct synchro_request req; >>> + req.type = type; >>> + req.replica_id = limbo->instance_id; >>> + req.lsn = lsn; >>> + >> >> Vlad, while you're at this code, could we please use designated >> initialization from the very beginning, ie >> >> struct synchro_request req = { >> .type = type, >> .replica_id = limbo->instance_id, >> .lsn = lsn, >> }; >> >> (the alignment is up to you though). Such initialization won't >> allow a bug to happen when we get a structure extension and >> other non updated fields will be zeroified. >Ok, done: > >==================== >@@ -277,10 +277,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >?{ >? assert(lsn > 0); >? >- struct synchro_request req; >- req.type = type; >- req.replica_id = limbo->instance_id; >- req.lsn = lsn; >+ struct synchro_request req = { >+ .type = type, >+ .replica_id = limbo->instance_id, >+ .lsn = lsn, >+ }; >? >==================== > >New complete patch: > >==================== >diff --git a/src/box/applier.cc b/src/box/applier.cc >index a953d293e..98fb87375 100644 >--- a/src/box/applier.cc >+++ b/src/box/applier.cc >@@ -275,26 +275,14 @@ process_nop(struct request *request) >??* or rollback some of its entries. >??*/ >?static int >-process_confirm_rollback(struct request *request, bool is_confirm) >+process_synchro_row(struct request *request) >?{ >? assert(iproto_type_is_synchro_request(request->header->type)); >- uint32_t replica_id; >? struct txn *txn = in_txn(); >- int64_t lsn = 0; >? >- int res = 0; >- if (is_confirm) >- res = xrow_decode_confirm(request->header, &replica_id, &lsn); >- else >- res = xrow_decode_rollback(request->header, &replica_id, &lsn); >- if (res == -1) >- return -1; >- >- if (replica_id != txn_limbo.instance_id) { >- diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, replica_id, >- txn_limbo.instance_id); >+ struct synchro_request syn_req; >+ if (xrow_decode_synchro(request->header, &syn_req) != 0) >? return -1; >- } >? assert(txn->n_applier_rows == 0); >? /* >? * This is not really a transaction. It just uses txn API >@@ -306,16 +294,9 @@ process_confirm_rollback(struct request *request, bool is_confirm) >? >? if (txn_begin_stmt(txn, NULL) != 0) >? return -1; >- >- if (txn_commit_stmt(txn, request) == 0) { >- if (is_confirm) >- txn_limbo_read_confirm(&txn_limbo, lsn); >- else >- txn_limbo_read_rollback(&txn_limbo, lsn); >- return 0; >- } else { >+ if (txn_commit_stmt(txn, request) != 0) >? return -1; >- } >+ return txn_limbo_process(&txn_limbo, &syn_req); >?} >? >?static int >@@ -324,8 +305,7 @@ apply_row(struct xrow_header *row) >? struct request request; >? if (iproto_type_is_synchro_request(row->type)) { >? request.header = row; >- return process_confirm_rollback(&request, >- row->type == IPROTO_CONFIRM); >+ return process_synchro_row(&request); >? } >? if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) >? return -1; >diff --git a/src/box/box.cc b/src/box/box.cc >index 83eef5d98..8e811e9c1 100644 >--- a/src/box/box.cc >+++ b/src/box/box.cc >@@ -367,22 +367,11 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) >?{ >? struct request request; >? if (iproto_type_is_synchro_request(row->type)) { >- uint32_t replica_id; >- int64_t lsn; >- switch(row->type) { >- case IPROTO_CONFIRM: >- if (xrow_decode_confirm(row, &replica_id, &lsn) < 0) >- diag_raise(); >- assert(txn_limbo.instance_id == replica_id); >- txn_limbo_read_confirm(&txn_limbo, lsn); >- break; >- case IPROTO_ROLLBACK: >- if (xrow_decode_rollback(row, &replica_id, &lsn) < 0) >- diag_raise(); >- assert(txn_limbo.instance_id == replica_id); >- txn_limbo_read_rollback(&txn_limbo, lsn); >- break; >- } >+ struct synchro_request syn_req; >+ if (xrow_decode_synchro(row, &syn_req) != 0) >+ diag_raise(); >+ if (txn_limbo_process(&txn_limbo, &syn_req) != 0) >+ diag_raise(); >? return; >? } >? xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); >diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c >index a2043c17a..c6a4e5efc 100644 >--- a/src/box/txn_limbo.c >+++ b/src/box/txn_limbo.c >@@ -31,6 +31,7 @@ >?#include "txn.h" >?#include "txn_limbo.h" >?#include "replication.h" >+#include "iproto_constants.h" >? >?struct txn_limbo txn_limbo; >? >@@ -272,11 +273,16 @@ complete: >?} >? >?static void >-txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, >- bool is_confirm) >+txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >?{ >? assert(lsn > 0); >? >+ struct synchro_request req = { >+ .type = type, >+ .replica_id = limbo->instance_id, >+ .lsn = lsn, >+ }; >+ >? struct xrow_header row; >? struct request request = { >? .header = &row, >@@ -286,19 +292,7 @@ txn_limbo_write_confirm_rollback(struct txn_limbo *limbo, int64_t lsn, >? if (txn == NULL) >? goto rollback; >? >- int res = 0; >- if (is_confirm) { >- res = xrow_encode_confirm(&row, &txn->region, >- limbo->instance_id, lsn); >- } else { >- /* >- * This LSN is the first to be rolled back, so >- * the last "safe" lsn is lsn - 1. >- */ >- res = xrow_encode_rollback(&row, &txn->region, >- limbo->instance_id, lsn); >- } >- if (res == -1) >+ if (xrow_encode_synchro(&row, &txn->region, &req) != 0) >? goto rollback; >? /* >? * This is not really a transaction. It just uses txn API >@@ -325,7 +319,7 @@ rollback: >? * problems are fixed. Or retry automatically with some period. >? */ >? panic("Could not write a synchro request to WAL: lsn = %lld, type = " >- "%s\n", lsn, is_confirm ? "CONFIRM" : "ROLLBACK"); >+ "%s\n", lsn, iproto_type_name(type)); >?} >? >?/** >@@ -338,10 +332,11 @@ txn_limbo_write_confirm(struct txn_limbo *limbo, int64_t lsn) >? assert(lsn > limbo->confirmed_lsn); >? assert(!limbo->is_in_rollback); >? limbo->confirmed_lsn = lsn; >- txn_limbo_write_confirm_rollback(limbo, lsn, true); >+ txn_limbo_write_synchro(limbo, IPROTO_CONFIRM, lsn); >?} >? >-void >+/** Confirm all the entries <= @a lsn. */ >+static void >?txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn) >?{ >? assert(limbo->instance_id != REPLICA_ID_NIL); >@@ -390,11 +385,12 @@ txn_limbo_write_rollback(struct txn_limbo *limbo, int64_t lsn) >? assert(lsn > limbo->confirmed_lsn); >? assert(!limbo->is_in_rollback); >? limbo->is_in_rollback = true; >- txn_limbo_write_confirm_rollback(limbo, lsn, false); >+ txn_limbo_write_synchro(limbo, IPROTO_ROLLBACK, lsn); >? limbo->is_in_rollback = false; >?} >? >-void >+/** Rollback all the entries >= @a lsn. */ >+static void >?txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn) >?{ >? assert(limbo->instance_id != REPLICA_ID_NIL); >@@ -577,6 +573,27 @@ complete: >? return 0; >?} >? >+int >+txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req) >+{ >+ if (req->replica_id != limbo->instance_id) { >+ diag_set(ClientError, ER_SYNC_MASTER_MISMATCH, req->replica_id, >+ limbo->instance_id); >+ return -1; >+ } >+ switch (req->type) { >+ case IPROTO_CONFIRM: >+ txn_limbo_read_confirm(limbo, req->lsn); >+ break; >+ case IPROTO_ROLLBACK: >+ txn_limbo_read_rollback(limbo, req->lsn); >+ break; >+ default: >+ unreachable(); >+ } >+ return 0; >+} >+ >?void >?txn_limbo_force_empty(struct txn_limbo *limbo, int64_t confirm_lsn) >?{ >diff --git a/src/box/txn_limbo.h b/src/box/txn_limbo.h >index 04ee7ea5c..eaf662987 100644 >--- a/src/box/txn_limbo.h >+++ b/src/box/txn_limbo.h >@@ -39,6 +39,7 @@ extern "C" { >?#endif /* defined(__cplusplus) */ >? >?struct txn; >+struct synchro_request; >? >?/** >??* Transaction and its quorum metadata, to be stored in limbo. >@@ -245,17 +246,9 @@ txn_limbo_ack(struct txn_limbo *limbo, uint32_t replica_id, int64_t lsn); >?int >?txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry); >? >-/** >- * Confirm all the entries up to the given master's LSN. >- */ >-void >-txn_limbo_read_confirm(struct txn_limbo *limbo, int64_t lsn); >- >-/** >- * Rollback all the entries starting with given master's LSN. >- */ >-void >-txn_limbo_read_rollback(struct txn_limbo *limbo, int64_t lsn); >+/** Execute a synchronous replication request. */ >+int >+txn_limbo_process(struct txn_limbo *limbo, const struct synchro_request *req); >? >?/** >??* Waiting for confirmation of all "sync" transactions >diff --git a/src/box/xrow.c b/src/box/xrow.c >index 0c797a9d5..bf174c701 100644 >--- a/src/box/xrow.c >+++ b/src/box/xrow.c >@@ -893,13 +893,13 @@ xrow_encode_dml(const struct request *request, struct region *region, >? return iovcnt; >?} >? >-static int >-xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, >- uint32_t replica_id, int64_t lsn, int type) >+int >+xrow_encode_synchro(struct xrow_header *row, struct region *region, >+ const struct synchro_request *req) >?{ >? size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + >- mp_sizeof_uint(replica_id) + mp_sizeof_uint(IPROTO_LSN) + >- mp_sizeof_uint(lsn); >+ mp_sizeof_uint(req->replica_id) + >+ mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); >? char *buf = (char *)region_alloc(region, len); >? if (buf == NULL) { >? diag_set(OutOfMemory, len, "region_alloc", "buf"); >@@ -909,9 +909,9 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, >? >? pos = mp_encode_map(pos, 2); >? pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); >- pos = mp_encode_uint(pos, replica_id); >+ pos = mp_encode_uint(pos, req->replica_id); >? pos = mp_encode_uint(pos, IPROTO_LSN); >- pos = mp_encode_uint(pos, lsn); >+ pos = mp_encode_uint(pos, req->lsn); >? >? memset(row, 0, sizeof(*row)); >? >@@ -919,30 +919,13 @@ xrow_encode_confirm_rollback(struct xrow_header *row, struct region *region, >? row->body[0].iov_len = len; >? row->bodycnt = 1; >? >- row->type = type; >+ row->type = req->type; >? >? return 0; >?} >? >?int >-xrow_encode_confirm(struct xrow_header *row, struct region *region, >- uint32_t replica_id, int64_t lsn) >-{ >- return xrow_encode_confirm_rollback(row, region, replica_id, lsn, >- IPROTO_CONFIRM); >-} >- >-int >-xrow_encode_rollback(struct xrow_header *row, struct region *region, >- uint32_t replica_id, int64_t lsn) >-{ >- return xrow_encode_confirm_rollback(row, region, replica_id, lsn, >- IPROTO_ROLLBACK); >-} >- >-static int >-xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, >- int64_t *lsn) >+xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req) >?{ >? if (row->bodycnt == 0) { >? diag_set(ClientError, ER_INVALID_MSGPACK, "request body"); >@@ -960,6 +943,7 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, >? return -1; >? } >? >+ memset(req, 0, sizeof(*req)); >? d = data; >? uint32_t map_size = mp_decode_map(&d); >? for (uint32_t i = 0; i < map_size; i++) { >@@ -977,30 +961,19 @@ xrow_decode_confirm_rollback(struct xrow_header *row, uint32_t *replica_id, >? } >? switch (key) { >? case IPROTO_REPLICA_ID: >- *replica_id = mp_decode_uint(&d); >+ req->replica_id = mp_decode_uint(&d); >? break; >? case IPROTO_LSN: >- *lsn = mp_decode_uint(&d); >+ req->lsn = mp_decode_uint(&d); >? break; >? default: >? mp_next(&d); >? } >? } >+ req->type = row->type; >? return 0; >?} >? >-int >-xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) >-{ >- return xrow_decode_confirm_rollback(row, replica_id, lsn); >-} >- >-int >-xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn) >-{ >- return xrow_decode_confirm_rollback(row, replica_id, lsn); >-} >- >?int >?xrow_to_iovec(const struct xrow_header *row, struct iovec *out) >?{ >diff --git a/src/box/xrow.h b/src/box/xrow.h >index e21ede5a3..02dca74e5 100644 >--- a/src/box/xrow.h >+++ b/src/box/xrow.h >@@ -216,54 +216,51 @@ xrow_encode_dml(const struct request *request, struct region *region, >? struct iovec *iov); >? >?/** >- * Encode the CONFIRM to row body and set row type to >- * IPROTO_CONFIRM. >- * @param row xrow header. >- * @param region Region to use to encode the confirmation body. >- * @param replica_id master's instance id. >- * @param lsn last confirmed lsn. >- * @retval -1 on error. >- * @retval 0 success. >+ * Synchronous replication request - confirmation or rollback of >+ * pending synchronous transactions. >??*/ >-int >-xrow_encode_confirm(struct xrow_header *row, struct region *region, >- uint32_t replica_id, int64_t lsn); >+struct synchro_request { >+ /** Operation type - IPROTO_ROLLBACK or IPROTO_CONFIRM. */ >+ uint32_t type; >+ /** >+ * ID of the instance owning the pending transactions. >+ * Note, it may be not the same instance, who created this >+ * request. An instance can make an operation on foreign >+ * synchronous transactions in case a new master tries to >+ * finish transactions of an old master. >+ */ >+ uint32_t replica_id; >+ /** >+ * Operation LSN. >+ * In case of CONFIRM it means 'confirm all >+ * transactions with lsn <= this value'. >+ * In case of ROLLBACK it means 'rollback all transactions >+ * with lsn >= this value'. >+ */ >+ int64_t lsn; >+}; >? >?/** >- * Decode the CONFIRM request body. >+ * Encode synchronous replication request. >??* @param row xrow header. >- * @param[out] replica_id master's instance id. >- * @param[out] lsn last confirmed lsn. >+ * @param region Region to use to encode the confirmation body. >+ * @param req Request parameters. >??* @retval -1 on error. >??* @retval 0 success. >??*/ >?int >-xrow_decode_confirm(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); >- >-/** >- * Encode the ROLLBACK row body and set row type to >- * IPROTO_ROLLBACK. >- * @param row xrow header. >- * @param region Region to use to encode the rollback body. >- * @param replica_id master's instance id. >- * @param lsn lsn to rollback from, including it. >- * @retval -1 on error. >- * @retval 0 success. >- */ >-int >-xrow_encode_rollback(struct xrow_header *row, struct region *region, >- uint32_t replica_id, int64_t lsn); >+xrow_encode_synchro(struct xrow_header *row, struct region *region, >+ const struct synchro_request *req); >? >?/** >- * Decode the ROLLBACK row body. >+ * Decode synchronous replication request. >??* @param row xrow header. >- * @param[out] replica_id master's instance id. >- * @param[out] lsn lsn to rollback from, including it. >+ * @param[out] req Request parameters. >??* @retval -1 on error. >??* @retval 0 success. >??*/ >?int >-xrow_decode_rollback(struct xrow_header *row, uint32_t *replica_id, int64_t *lsn); >+xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); >? >?/** >??* CALL/EVAL request. >? ? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Mon Aug 17 23:22:50 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 22:22:50 +0200 Subject: [Tarantool-patches] [PATCH 1/1] xrow: introduce struct synchro_request In-Reply-To: References: Message-ID: <5c0be123-78d1-daf7-62f4-344d43534ad2@tarantool.org> Pushed to master and 2.5. From v.shpilevoy at tarantool.org Mon Aug 17 23:22:52 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 22:22:52 +0200 Subject: [Tarantool-patches] [PATCH 1/1] applier: drop a couple of unnecessary arguments In-Reply-To: <7e16e45e171cb34edaff69b0f660421f1b67b4a1.1597504571.git.v.shpilevoy@tarantool.org> References: <7e16e45e171cb34edaff69b0f660421f1b67b4a1.1597504571.git.v.shpilevoy@tarantool.org> Message-ID: <489bd27f-24c3-f56c-343a-f5b32b0723f1@tarantool.org> Pushed to master and 2.5. From v.shpilevoy at tarantool.org Mon Aug 17 23:49:00 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 22:49:00 +0200 Subject: [Tarantool-patches] [PATCH v7 7/8] applier: process synchro requests without txn engine In-Reply-To: <20200817124235.GJ2074@grain> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-8-gorcunov@gmail.com> <20200817124235.GJ2074@grain> Message-ID: <41e3fd4b-6d04-a5ab-3a6c-858929456ee9@tarantool.org> On 17.08.2020 14:42, Cyrill Gorcunov wrote: > On Sat, Aug 15, 2020 at 05:06:05PM +0200, Vladislav Shpilevoy wrote: >>> +static struct synchro_entry * >>> +synchro_entry_new(struct applier *applier, >>> + struct xrow_header *applier_row, >>> + struct synchro_request *req) >>> +{ >>> + struct synchro_entry *entry; >>> + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); >> >> 6. Why don't you just add 'struct xrow_header*[1]' to the end of >> struct synchro_entry? There is no a case, when the entry is needed >> without the xrow_header pointer in the end. > > This is forbidden by asan and some other compilers we've in travis runs. > I've been already trying. We use clang and gcc, just 2. To workaround that I suggest to add -Wno-gnu-variable-sized-type-not-at-end to compiler.cmake (with that flag it works, at least on clang - I need you to check it on gcc). From v.shpilevoy at tarantool.org Mon Aug 17 23:49:19 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 22:49:19 +0200 Subject: [Tarantool-patches] [PATCH v8 5/9] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200817133918.875558-6-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-6-gorcunov@gmail.com> Message-ID: > diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c > index ed8c10419..447630d23 100644 > --- a/src/box/txn_limbo.c > +++ b/src/box/txn_limbo.c > @@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) > @@ -284,46 +296,34 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) > -rollback: > - /* > - * XXX: the stub is supposed to be removed once it is defined what to do > - * when a synchro request WAL write fails. One of the possible > - * solutions: log the error, keep the limbo queue as is and probably put > - * in rollback mode. Then provide a hook to call manually when WAL > - * problems are fixed. Or retry automatically with some period. > - */ > - panic("Could not write a synchro request to WAL: lsn = %lld, type = " > - "%s\n", lsn, iproto_type_name(type)); > + if (journal_write(entry) != 0 || entry->res < 0) { > + diag_set(ClientError, ER_WAL_IO); > + diag_log(); > + /* > + * XXX: the stub is supposed to be removed once it is defined what to do > + * when a synchro request WAL write fails. One of the possible > + * solutions: log the error, keep the limbo queue as is and probably put > + * in rollback mode. Then provide a hook to call manually when WAL > + * problems are fixed. Or retry automatically with some period. Still out of 80 symbols. > + */ > + panic("Could not write a synchro request to WAL: lsn = %lld, type = " > + "%s\n", lsn, type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); > + } > } > > /** > From v.shpilevoy at tarantool.org Mon Aug 17 23:49:23 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 22:49:23 +0200 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: <20200817133918.875558-7-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-7-gorcunov@gmail.com> Message-ID: <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> It seems this commit is not needed - I dropped it and nothing changed. Even no merge/rebase conflicts. On 17.08.2020 15:39, Cyrill Gorcunov wrote: > We need to access first and last xrow in a queue > frenquently and opencoded variants are too ugly. frenquently -> frequently. From v.shpilevoy at tarantool.org Tue Aug 18 00:24:40 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 17 Aug 2020 23:24:40 +0200 Subject: [Tarantool-patches] [PATCH v8 0/9] qsync: write CONFIRM/ROLLBACK without txn engine In-Reply-To: <20200817133918.875558-1-gorcunov@gmail.com> References: <20200817133918.875558-1-gorcunov@gmail.com> Message-ID: <78aa1ff6-34c1-767c-7f50-21c93236d8e7@tarantool.org> process_synchro_row() is still present on the branch. From gorcunov at gmail.com Tue Aug 18 00:54:00 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 18 Aug 2020 00:54:00 +0300 Subject: [Tarantool-patches] [PATCH v8 0/9] qsync: write CONFIRM/ROLLBACK without txn engine In-Reply-To: <78aa1ff6-34c1-767c-7f50-21c93236d8e7@tarantool.org> References: <20200817133918.875558-1-gorcunov@gmail.com> <78aa1ff6-34c1-767c-7f50-21c93236d8e7@tarantool.org> Message-ID: <20200817215400.GK2074@grain> On Mon, Aug 17, 2020 at 11:24:40PM +0200, Vladislav Shpilevoy wrote: > process_synchro_row() is still present on the branch. Sigh. Overlooked, thanks! I've force-pushed the branch, just in case here is a commit on top. We can simply drop this helper it harmless. --- diff --git a/src/box/applier.cc b/src/box/applier.cc index 83f6da461..5d7c35ca6 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -270,45 +270,11 @@ process_nop(struct request *request) return txn_commit_stmt(txn, request); } -/* - * CONFIRM/ROLLBACK rows aren't dml requests and require special - * handling: instead of performing some operations on spaces, - * processing these requests requires txn_limbo to either confirm - * or rollback some of its entries. - */ -static int -process_synchro_row(struct request *request) -{ - assert(iproto_type_is_synchro_request(request->header->type)); - struct txn *txn = in_txn(); - - struct synchro_request syn_req; - if (xrow_decode_synchro(request->header, &syn_req) != 0) - return -1; - assert(txn->n_applier_rows == 0); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - return -1; - if (txn_commit_stmt(txn, request) != 0) - return -1; - return txn_limbo_process(&txn_limbo, &syn_req); -} - static int apply_row(struct xrow_header *row) { struct request request; - if (iproto_type_is_synchro_request(row->type)) { - request.header = row; - return process_synchro_row(&request); - } + assert(!iproto_type_is_synchro_request(row->type)); if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; if (request.type == IPROTO_NOP) From gorcunov at gmail.com Tue Aug 18 01:14:03 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 18 Aug 2020 01:14:03 +0300 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-7-gorcunov@gmail.com> <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> Message-ID: <20200817221403.GL2074@grain> On Mon, Aug 17, 2020 at 10:49:23PM +0200, Vladislav Shpilevoy wrote: > It seems this commit is not needed - I dropped it and nothing changed. > Even no merge/rebase conflicts. > > On 17.08.2020 15:39, Cyrill Gorcunov wrote: > > We need to access first and last xrow in a queue > > frenquently and opencoded variants are too ugly. > > frenquently -> frequently. The code is ugly as hell without it :/ We use them not once in the code that's why I made helpers. But I won't insist, drop it if you prefer. From gorcunov at gmail.com Tue Aug 18 01:16:21 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 18 Aug 2020 01:16:21 +0300 Subject: [Tarantool-patches] [PATCH v8 5/9] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-6-gorcunov@gmail.com> Message-ID: <20200817221621.GM2074@grain> On Mon, Aug 17, 2020 at 10:49:19PM +0200, Vladislav Shpilevoy wrote: > > + if (journal_write(entry) != 0 || entry->res < 0) { > > + diag_set(ClientError, ER_WAL_IO); > > + diag_log(); > > + /* > > + * XXX: the stub is supposed to be removed once it is defined what to do > > + * when a synchro request WAL write fails. One of the possible > > + * solutions: log the error, keep the limbo queue as is and probably put > > + * in rollback mode. Then provide a hook to call manually when WAL > > + * problems are fixed. Or retry automatically with some period. > > Still out of 80 symbols. Shame on me :( You already pointed and I remember this problem, but then been rebasing and it flew out of my head. I could force push update if this is the only problem. From gorcunov at gmail.com Tue Aug 18 01:23:56 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 18 Aug 2020 01:23:56 +0300 Subject: [Tarantool-patches] [PATCH v8 5/9] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-6-gorcunov@gmail.com> Message-ID: <20200817222356.GN2074@grain> When we need to write CONFIRM or ROLLBACK message (which is a binary record in msgpack format) into a journal we use txn code to allocate a new transaction, encode there a message and pass it to walk the long txn path before it hit the journal. This is not only resource wasting but also somehow strange from architectural point of view. Instead lets encode a record on the stack and write it to the journal directly. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- Force-pushed into the same branch src/box/txn_limbo.c | 66 +++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index ed8c10419..53fcdf137 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -32,6 +32,7 @@ #include "txn_limbo.h" #include "replication.h" #include "iproto_constants.h" +#include "journal.h" struct txn_limbo txn_limbo; @@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) return 0; } +/** + * A callback for synchronous write: txn_limbo_write_synchro fiber + * waiting to proceed once a record is written to WAL. + */ +static void +txn_limbo_write_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + fiber_wakeup(entry->complete_data); +} + static void txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { @@ -284,46 +296,36 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) /* * This is a synchronous commit so we can - * use body and row allocated on a stack. + * allocate everything on a stack. */ struct synchro_body_bin body; struct xrow_header row; - struct request request = { - .header = &row, - }; + char buf[sizeof(struct journal_entry) + + sizeof(struct xrow_header *)]; - struct txn *txn = txn_begin(); - if (txn == NULL) - goto rollback; + struct journal_entry *entry = (struct journal_entry *)buf; + entry->rows[0] = &row; xrow_encode_synchro(&row, &body, &req); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - goto rollback; - if (txn_commit_stmt(txn, &request) != 0) - goto rollback; - if (txn_commit(txn) != 0) - goto rollback; - return; + journal_entry_create(entry, 1, xrow_approx_len(&row), + txn_limbo_write_cb, fiber()); -rollback: - /* - * XXX: the stub is supposed to be removed once it is defined what to do - * when a synchro request WAL write fails. One of the possible - * solutions: log the error, keep the limbo queue as is and probably put - * in rollback mode. Then provide a hook to call manually when WAL - * problems are fixed. Or retry automatically with some period. - */ - panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, iproto_type_name(type)); + if (journal_write(entry) != 0 || entry->res < 0) { + diag_set(ClientError, ER_WAL_IO); + diag_log(); + /* + * XXX: the stub is supposed to be removed once it is defined + * what to do when a synchro request WAL write fails. One of + * the possible solutions: log the error, keep the limbo + * queue as is and probably put in rollback mode. Then + * provide a hook to call manually when WAL problems are fixed. + * Or retry automatically with some period. + */ + panic("Could not write a synchro request to WAL: " + "lsn = %lld, type = %s\n", lsn, + type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); + } } /** -- 2.26.2 From gorcunov at gmail.com Tue Aug 18 11:08:47 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Tue, 18 Aug 2020 11:08:47 +0300 Subject: [Tarantool-patches] [PATCH v7 7/8] applier: process synchro requests without txn engine In-Reply-To: <41e3fd4b-6d04-a5ab-3a6c-858929456ee9@tarantool.org> References: <20200814211442.667099-1-gorcunov@gmail.com> <20200814211442.667099-8-gorcunov@gmail.com> <20200817124235.GJ2074@grain> <41e3fd4b-6d04-a5ab-3a6c-858929456ee9@tarantool.org> Message-ID: <20200818080847.GO2074@grain> On Mon, Aug 17, 2020 at 10:49:00PM +0200, Vladislav Shpilevoy wrote: > On 17.08.2020 14:42, Cyrill Gorcunov wrote: > > On Sat, Aug 15, 2020 at 05:06:05PM +0200, Vladislav Shpilevoy wrote: > >>> +static struct synchro_entry * > >>> +synchro_entry_new(struct applier *applier, > >>> + struct xrow_header *applier_row, > >>> + struct synchro_request *req) > >>> +{ > >>> + struct synchro_entry *entry; > >>> + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); > >> > >> 6. Why don't you just add 'struct xrow_header*[1]' to the end of > >> struct synchro_entry? There is no a case, when the entry is needed > >> without the xrow_header pointer in the end. > > > > This is forbidden by asan and some other compilers we've in travis runs. > > I've been already trying. > > We use clang and gcc, just 2. To workaround that I suggest to add > -Wno-gnu-variable-sized-type-not-at-end to compiler.cmake (with that > flag it works, at least on clang - I need you to check it on gcc). I think this could be addressed later (if ever) -- we allocate an entry dinamically and we have similar thing in general xrow allocation, so in this POV at least they all are unified. From v.shpilevoy at tarantool.org Tue Aug 18 22:18:12 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 18 Aug 2020 21:18:12 +0200 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: <20200817221403.GL2074@grain> References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-7-gorcunov@gmail.com> <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> <20200817221403.GL2074@grain> Message-ID: On 18.08.2020 00:14, Cyrill Gorcunov wrote: > On Mon, Aug 17, 2020 at 10:49:23PM +0200, Vladislav Shpilevoy wrote: >> It seems this commit is not needed - I dropped it and nothing changed. >> Even no merge/rebase conflicts. >> >> On 17.08.2020 15:39, Cyrill Gorcunov wrote: >>> We need to access first and last xrow in a queue >>> frenquently and opencoded variants are too ugly. >> >> frenquently -> frequently. > > The code is ugly as hell without it :/ There are many ugly things, but it does not mean we need to rush changing them. It was discussed already many times. Please, drop this commit. It is not necessary. It does not interfere with the other patches in a single line. From alexander.turenko at tarantool.org Wed Aug 19 00:10:14 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Wed, 19 Aug 2020 00:10:14 +0300 Subject: [Tarantool-patches] [PATCH v5 1/2] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <07d6fd4508eda75a98bb9ea49dd58b6b14fbd99a.1597641988.git.avtikhon@tarantool.org> References: <07d6fd4508eda75a98bb9ea49dd58b6b14fbd99a.1597641988.git.avtikhon@tarantool.org> Message-ID: <20200818211014.22yrvxpoyhdokonb@tkn_work_nb> Thanks for the update! Now we can discuss the message. I left comments, where something confuses me in the message and the test. WBR, Alexander Turenko. On Mon, Aug 17, 2020 at 08:29:14AM +0300, Alexander V. Tikhonov wrote: > During implementation of openSUSE got failed box-tap/cfg.test.lua test. Implementation of openSUSE... build? > Found that when memtx_dir didn't exist and vinyl_dir existed and also > errno was set to ENOENT, box configuration succeeded, but it shouldn't. > Reason of this wrong behaviour was that not all of the failure paths in > xdir_scan() set errno, but the caller assumed it. > > Debugging src/box/xlog.c found that all checks were correct, but at: > > src/box/vy_log.c:vy_log_bootstrap() > src/box/vy_log.c:vy_log_begin_recovery() > > the checks on of the errno on ENOENT blocked the negative return from: > > src/box/xlog.c:xdir_scan() 'blocked negative return' is not clear for me. I guess that you want to express the following two points: - A negative return value is not considered as an error when errno is set to ENOENT. - The idea of this check is to handle the situation when vinyl_dir is not exists. I would rephrase it. > > Found that errno was already set to ENOENT before the xdir_scan() call. > To fix the issue the errno could be clean before the call to xdir_scan, > because we are interesting in it only from xdir_scan function. Strictly speaking, the reason is different: because xdir_scan() may return a negative value and leave errno unchanged. > > After discussions found that there were alternative better solution to > fix it. The fix with resetting errno was not good because xdir_scan() > was not system call in real and some internal routines could set it > to ENOENT itself, so it couldn't be controled from outside of function. 'internal routines could set it to ENOENT itself' -- I would say that something that is not vinyl_dir existence check. One more point: I don't see how it may happen except due to a race: - a file name is obtained from readdir(), - the file is deleted by another process, - we open the file and fail with ENOENT. And I'm not sure the race is possible. I want to say that I'm unable to state that 'internal routines could set it to ENOENT': I don't know, in fact. But the implementation can be changed in a future and there is no guarantee that returning with -1 and ENOENT means only lack of vinyl_dir. I mean, I would highlight that the sentence does not assert that the situation is possible with the current implementation. Just that we have no corresponding guarantee. Typo: was not system call -> is not a system call Typo: controled -> controlled. > > To be sure in behaviour of the changing errno decided to pass a flag to > xdir_scan() if the directory should exist. 'behaviour of the changing errno' is vague. I guess the idea of the sentence is that the variant with the flag is better, because (I guess) it is more explicit and should be less fragile. > > Closes #4594 > Needed for #4562 > > Co-authored-by: Alexander Turenko > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > Issue: https://github.com/tarantool/tarantool/issues/4562 > diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > new file mode 100755 > index 000000000..cbf7b1f35 > --- /dev/null > +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > @@ -0,0 +1,47 @@ > +#!/usr/bin/env tarantool > + > +local tap = require('tap') > +local test = tap.test('cfg') > +local fio = require('fio') > +test:plan(1) > + > +local tarantool_bin = arg[-1] > +local PANIC = 256 > + > +function run_script(code) Nit: It is the rule of thumb to don't fill the global namespace with fields (variables and functions) if you actually don't need it. > + local dir = fio.tempdir() > + local script_path = fio.pathjoin(dir, 'script.lua') > + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, > + tonumber('0777', 8)) > + script:write(code) > + script:write("\nos.exit(0)") > + script:close() > + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] > + local res = os.execute(string.format(cmd, dir, tarantool_bin)) > + fio.rmtree(dir) > + return res > +end > + > +-- > +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and > +-- errno is set to ENOENT, box configuration succeeds, however it > +-- should not > +-- > +vinyl_dir = fio.tempdir() Same here, it would be good to use 'local' here. > +run_script(string.format([[ > +box.cfg{vinyl_dir = '%s'} > +s = box.schema.space.create('test', {engine = 'vinyl'}) > +s:create_index('pk') > +os.exit(0) > +]], vinyl_dir)) > +code = string.format([[ Same here. > +local errno = require('errno') > +errno(errno.ENOENT) > +box.cfg{vinyl_dir = '%s'} > +os.exit(0) > +]], vinyl_dir) > +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") > +fio.rmtree(vinyl_dir) I propose to clarify the test steps a bit: | vinyl_dir = <...> | | -- Fill vinyl_dir. | run_script(<...>) | | -- Verify the case described above. | local code = <...> | test:is(<...>) | | -- Remove vinyl_dir. | fio.rmtree(vinyl_dir) > + > +test:check() > +os.exit(0) I suggest to highlight a test status in the exit code (it follows the Lua Style Guide proposal [1]): | os.exit(test:check() and 0 or 1) [1]: https://github.com/tarantool/doc/issues/1004 From alexander.turenko at tarantool.org Wed Aug 19 00:11:33 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Wed, 19 Aug 2020 00:11:33 +0300 Subject: [Tarantool-patches] [PATCH v4 2/2] gitlab-ci: add openSUSE packages build jobs In-Reply-To: References: Message-ID: <20200818211133.fmcfzas2darqcd6e@tkn_work_nb> LGTM. WBR, Alexander Turenko. On Fri, Aug 14, 2020 at 12:59:58PM +0300, Alexander V. Tikhonov wrote: > Implemented openSUSE packages build with testing for images: > opensuse-leap:15.[0-2] > > Added %{sle_version} checks in Tarantool spec file according to > https://en.opensuse.org/openSUSE:Packaging_for_Leap#RPM_Distro_Version_Macros > > Added opensuse-leap of 15.1 and 15.2 versions to Gitlab-CI packages > building/deploing jobs. > > Closes #4562 > --- > .gitlab-ci.yml | 24 ++++++++++++++++++++++++ > rpm/tarantool.spec | 16 +++++++++++----- > tools/update_repo.sh | 6 ++++-- > 3 files changed, 39 insertions(+), 7 deletions(-) From avtikhon at tarantool.org Wed Aug 19 09:04:48 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Wed, 19 Aug 2020 09:04:48 +0300 Subject: [Tarantool-patches] [PATCH v5 1/2] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <20200818211014.22yrvxpoyhdokonb@tkn_work_nb> References: <07d6fd4508eda75a98bb9ea49dd58b6b14fbd99a.1597641988.git.avtikhon@tarantool.org> <20200818211014.22yrvxpoyhdokonb@tkn_work_nb> Message-ID: <20200819060448.GA24723@hpalx> Hi Alexander, thanks for the review, please check my comments below. On Wed, Aug 19, 2020 at 12:10:14AM +0300, Alexander Turenko wrote: > Thanks for the update! Now we can discuss the message. > > I left comments, where something confuses me in the message and the > test. > > WBR, Alexander Turenko. > > On Mon, Aug 17, 2020 at 08:29:14AM +0300, Alexander V. Tikhonov wrote: > > During implementation of openSUSE got failed box-tap/cfg.test.lua test. > > Implementation of openSUSE... build? Corrected. > > > Found that when memtx_dir didn't exist and vinyl_dir existed and also > > errno was set to ENOENT, box configuration succeeded, but it shouldn't. > > Reason of this wrong behaviour was that not all of the failure paths in > > xdir_scan() set errno, but the caller assumed it. > > > > Debugging src/box/xlog.c found that all checks were correct, but at: > > > > src/box/vy_log.c:vy_log_bootstrap() > > src/box/vy_log.c:vy_log_begin_recovery() > > > > the checks on of the errno on ENOENT blocked the negative return from: > > > > src/box/xlog.c:xdir_scan() > > 'blocked negative return' is not clear for me. I guess that you want to > express the following two points: > > - A negative return value is not considered as an error when errno is > set to ENOENT. > - The idea of this check is to handle the situation when vinyl_dir is > not exists. > > I would rephrase it. > Right I've rewrote it an mentioned these points. > > > > Found that errno was already set to ENOENT before the xdir_scan() call. > > To fix the issue the errno could be clean before the call to xdir_scan, > > because we are interesting in it only from xdir_scan function. > > Strictly speaking, the reason is different: because xdir_scan() may > return a negative value and leave errno unchanged. > Right, I've changed the comment to make it more clear. > > > > After discussions found that there were alternative better solution to > > fix it. The fix with resetting errno was not good because xdir_scan() > > was not system call in real and some internal routines could set it > > to ENOENT itself, so it couldn't be controled from outside of function. > > 'internal routines could set it to ENOENT itself' -- I would say that > something that is not vinyl_dir existence check. > > One more point: I don't see how it may happen except due to a race: > > - a file name is obtained from readdir(), > - the file is deleted by another process, > - we open the file and fail with ENOENT. > > And I'm not sure the race is possible. I want to say that I'm unable to > state that 'internal routines could set it to ENOENT': I don't know, in > fact. But the implementation can be changed in a future and there is no > guarantee that returning with -1 and ENOENT means only lack of > vinyl_dir. > > I mean, I would highlight that the sentence does not assert that the > situation is possible with the current implementation. Just that we have > no corresponding guarantee. > Right, the xdir_scan() is not system call and can be changed in the way that can brake the check. > Typo: was not system call -> is not a system call > > Typo: controled -> controlled. > Corrected. > > > > To be sure in behaviour of the changing errno decided to pass a flag to > > xdir_scan() if the directory should exist. > > 'behaviour of the changing errno' is vague. I guess the idea of the > sentence is that the variant with the flag is better, because (I guess) > it is more explicit and should be less fragile. > Here I've added more comments. > > > > Closes #4594 > > Needed for #4562 > > > > Co-authored-by: Alexander Turenko > > --- > > > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > > Issue: https://github.com/tarantool/tarantool/issues/4562 > > > diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > > new file mode 100755 > > index 000000000..cbf7b1f35 > > --- /dev/null > > +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > > @@ -0,0 +1,47 @@ > > +#!/usr/bin/env tarantool > > + > > +local tap = require('tap') > > +local test = tap.test('cfg') > > +local fio = require('fio') > > +test:plan(1) > > + > > +local tarantool_bin = arg[-1] > > +local PANIC = 256 > > + > > +function run_script(code) > > Nit: It is the rule of thumb to don't fill the global namespace with > fields (variables and functions) if you actually don't need it. > Corrected. > > + local dir = fio.tempdir() > > + local script_path = fio.pathjoin(dir, 'script.lua') > > + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, > > + tonumber('0777', 8)) > > + script:write(code) > > + script:write("\nos.exit(0)") > > + script:close() > > + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] > > + local res = os.execute(string.format(cmd, dir, tarantool_bin)) > > + fio.rmtree(dir) > > + return res > > +end > > + > > +-- > > +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and > > +-- errno is set to ENOENT, box configuration succeeds, however it > > +-- should not > > +-- > > +vinyl_dir = fio.tempdir() > > Same here, it would be good to use 'local' here. > Corrected. > > +run_script(string.format([[ > > +box.cfg{vinyl_dir = '%s'} > > +s = box.schema.space.create('test', {engine = 'vinyl'}) > > +s:create_index('pk') > > +os.exit(0) > > +]], vinyl_dir)) > > +code = string.format([[ > > Same here. > Corrected. > > +local errno = require('errno') > > +errno(errno.ENOENT) > > +box.cfg{vinyl_dir = '%s'} > > +os.exit(0) > > +]], vinyl_dir) > > +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") > > +fio.rmtree(vinyl_dir) > > I propose to clarify the test steps a bit: > Corrected. > | vinyl_dir = <...> > | > | -- Fill vinyl_dir. > | run_script(<...>) > | > | -- Verify the case described above. > | local code = <...> > | test:is(<...>) > | > | -- Remove vinyl_dir. > | fio.rmtree(vinyl_dir) > > > + > > +test:check() > > +os.exit(0) > > I suggest to highlight a test status in the exit code (it follows the > Lua Style Guide proposal [1]): > > | os.exit(test:check() and 0 or 1) > > [1]: https://github.com/tarantool/doc/issues/1004 Corrected. From avtikhon at tarantool.org Wed Aug 19 09:17:39 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Wed, 19 Aug 2020 09:17:39 +0300 Subject: [Tarantool-patches] [PATCH v6] vinyl: fix check vinyl_dir existence at bootstrap Message-ID: <344b8b406afa1464aae59f9be231ff78f053b996.1597817683.git.avtikhon@tarantool.org> During implementation of openSUSE build with testing got failed test box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and vinyl_dir existed and also errno was set to ENOENT, box configuration succeeded, but it shouldn't. Reason of this wrong behaviour was that not all of the failure paths in xdir_scan() set errno, but the caller assumed it. Debugging the issue found that after xdir_scan() there was incorrect check for errno when it returned negative values. xdir_scan() is not system call and negative return value from it doesn't mean that errno whould be set too. Found that in situations when errno was left from previous commands before xdir_scan() and xdir_scan() returned negative value by itself than the check was wrong. The previous logic of the check was to catch the error ENOENT from inside the xdir_scan() function to handle the situation when vinyl_dir was not exist. In this way errno should be reseted before xdir_scan() call to give the ability for xdir_scan() to use return value without errno set and correctly handle errno from inside the xdir_scan(). After discussions found that there was alternative better solution to fix it. As mentioned above xdir_scan() function is not system call and can be changed inside it in any possible way. So check outside of this function on errno could be broken, because of the xdir_scan() changes. To avoid of it we must avoid of errno checks outside of the function. Better solution was to use the flag in xdir_scan(), to check if the directory should exist. So errno check was removed and instead of it the check for vinyl_dir existence using flag added. Closes #4594 Needed for #4562 Co-authored-by: Alexander Turenko --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci Issue: https://github.com/tarantool/tarantool/issues/4594 Issue: https://github.com/tarantool/tarantool/issues/4562 src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 +- src/box/vy_log.c | 4 +- src/box/wal.c | 2 +- src/box/xlog.c | 4 +- src/box/xlog.h | 6 +-- .../gh-4562-errno-at-xdir_scan.test.lua | 54 +++++++++++++++++++ 7 files changed, 66 insertions(+), 10 deletions(-) create mode 100755 test/box-tap/gh-4562-errno-at-xdir_scan.test.lua diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..da3c50e87 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..2b894d680 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -559,7 +559,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..74f761994 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..3400eb75f 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); * snapshot or scan through all logs. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua new file mode 100755 index 000000000..e6dd68d54 --- /dev/null +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua @@ -0,0 +1,54 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local test = tap.test('cfg') +local fio = require('fio') +test:plan(1) + +local tarantool_bin = arg[-1] +local PANIC = 256 + +local function run_script(code) + local dir = fio.tempdir() + local script_path = fio.pathjoin(dir, 'script.lua') + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, + tonumber('0777', 8)) + script:write(code) + script:write("\nos.exit(0)") + script:close() + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] + local res = os.execute(string.format(cmd, dir, tarantool_bin)) + fio.rmtree(dir) + return res +end + +-- +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and +-- errno is set to ENOENT, box configuration succeeds, however it +-- should not +-- + +-- create vinyl_dir as temporary path +local vinyl_dir = fio.tempdir() + +-- fill vinyl_dir +run_script(string.format([[ +box.cfg{vinyl_dir = '%s'} +s = box.schema.space.create('test', {engine = 'vinyl'}) +s:create_index('pk') +os.exit(0) +]], vinyl_dir)) + +-- verify the case described above +local code = string.format([[ +local errno = require('errno') +errno(errno.ENOENT) +box.cfg{vinyl_dir = '%s'} +os.exit(0) +]], vinyl_dir) +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") + +-- remove vinyl_dir +fio.rmtree(vinyl_dir) + +os.exit(test:check() and 0 or 1) -- 2.17.1 From avtikhon at tarantool.org Wed Aug 19 11:17:49 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Wed, 19 Aug 2020 11:17:49 +0300 Subject: [Tarantool-patches] [PATCH v1] test: flaky replication/anon.test.lua test Message-ID: <0002fadff7c250b4e3a810ef3e8d8ab6cec9ed63.1597824995.git.avtikhon@tarantool.org> Found flaky issues multi running replication/anon.test.lua test on the single worker: [007] --- replication/anon.result Fri Jun 5 09:02:25 2020 [007] +++ replication/anon.reject Mon Jun 8 01:19:37 2020 [007] @@ -55,7 +55,7 @@ [007] [007] box.info.status [007] | --- [007] - | - running [007] + | - orphan [007] | ... [007] box.info.id [007] | --- [094] --- replication/anon.result Sat Jun 20 06:02:43 2020 [094] +++ replication/anon.reject Tue Jun 23 19:35:28 2020 [094] @@ -154,7 +154,7 @@ [094] -- Test box.info.replication_anon. [094] box.info.replication_anon [094] | --- [094] - | - count: 1 [094] + | - count: 2 [094] | ... [094] #box.info.replication_anon() [094] | --- [094] It happend because replications may stay active from the previous runs on the common tarantool instance at the test-run worker. To avoid of it added restarting of the tarantool instance at the very start of the test. Closes #5058 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5058-repl-anon Issue: https://github.com/tarantool/tarantool/issues/5058 test/replication/anon.result | 2 ++ test/replication/anon.test.lua | 1 + test/replication/suite.ini | 1 - 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/test/replication/anon.result b/test/replication/anon.result index 2b880b94b..dd470797c 100644 --- a/test/replication/anon.result +++ b/test/replication/anon.result @@ -8,6 +8,8 @@ vclock_diff = require('fast_replica').vclock_diff test_run = env.new() | --- | ... +test_run:cmd('restart server default') + | -- diff --git a/test/replication/anon.test.lua b/test/replication/anon.test.lua index b42602d50..2cc77595b 100644 --- a/test/replication/anon.test.lua +++ b/test/replication/anon.test.lua @@ -1,6 +1,7 @@ env = require('test_run') vclock_diff = require('fast_replica').vclock_diff test_run = env.new() +test_run:cmd('restart server default') -- diff --git a/test/replication/suite.ini b/test/replication/suite.ini index ab9c3dabd..49d81e54d 100644 --- a/test/replication/suite.ini +++ b/test/replication/suite.ini @@ -22,5 +22,4 @@ fragile = errinj.test.lua ; gh-3870 recover_missing_xlog.test.lua ; gh-4989 box_set_replication_stress.test.lua ; gh-4992 gh-4986 gh-4605-empty-password.test.lua ; gh-5030 - anon.test.lua ; gh-5058 status.test.lua ; gh-5110 -- 2.17.1 From i.kosarev at tarantool.org Wed Aug 19 20:23:24 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Wed, 19 Aug 2020 20:23:24 +0300 Subject: [Tarantool-patches] [PATCH] test: concurrent tuple update segfault on bitset index iteration Message-ID: <20200819172324.6188-1-i.kosarev@tarantool.org> Concurrent tuple update could segfault on BITSET_ALL_NOT_SET iterator usage. Fixed in 850054b2dbca257076c3f7c22e00564ac55b70d5. Closes #1088 --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-1088-concurrent-tuple-update-segfault-with-bitset-index Issue: https://github.com/tarantool/tarantool/issues/1088 test/box/bitset.result | 86 ++++++++++++++++++++++++++++++++++++++++ test/box/bitset.test.lua | 42 ++++++++++++++++++++ 2 files changed, 128 insertions(+) diff --git a/test/box/bitset.result b/test/box/bitset.result index bf44773ef3..5da068385c 100644 --- a/test/box/bitset.result +++ b/test/box/bitset.result @@ -2020,3 +2020,89 @@ s:drop() box.schema.func.drop('s') --- ... +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration +test_run = require('test_run').new() +--- +... +fiber = require('fiber') +--- +... +s = box.schema.space.create('gh-1088') +--- +... +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) +--- +... +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) +--- +... +for i = 1, 100 do s:insert{i, 0, i - 1} end +--- +... +counter = 0 +--- +... +test_run:cmd("setopt delimiter ';'") +--- +- true +... +function update() + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do + counter = counter + 1 + s:update(t[1], {{'+', 3, 11}}) + fiber.sleep(0) + end + fiber.self():cancel() +end; +--- +... +test_run:cmd("setopt delimiter ''"); +--- +- true +... +fibers = {} +--- +... +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end +--- +... +updating = true +--- +... +test_run:cmd("setopt delimiter ';'") +--- +- true +... +while updating do + updating = false + for _, f in pairs(fibers) do + if f:status() ~= 'dead' then updating = true end + end + fiber.sleep(0.001) +end; +--- +... +test_run:cmd("setopt delimiter ''"); +--- +- true +... +s:get(1) +--- +- [1, 0, 1100] +... +s:get(2) +--- +- [2, 0, 1101] +... +s:get(3) +--- +- [3, 0, 1102] +... +s:get(4) +--- +- [4, 0, 1103] +... +counter -- total updates counter +--- +- 10000 +... diff --git a/test/box/bitset.test.lua b/test/box/bitset.test.lua index d644d34e0b..dd432edeb0 100644 --- a/test/box/bitset.test.lua +++ b/test/box/bitset.test.lua @@ -162,3 +162,45 @@ _ = s:create_index('pk') _ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) s:drop() box.schema.func.drop('s') + +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration +test_run = require('test_run').new() +fiber = require('fiber') + +s = box.schema.space.create('gh-1088') +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) +for i = 1, 100 do s:insert{i, 0, i - 1} end + +counter = 0 +test_run:cmd("setopt delimiter ';'") +function update() + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do + counter = counter + 1 + s:update(t[1], {{'+', 3, 11}}) + fiber.sleep(0) + end + fiber.self():cancel() +end; +test_run:cmd("setopt delimiter ''"); + +fibers = {} +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end + +updating = true +test_run:cmd("setopt delimiter ';'") +while updating do + updating = false + for _, f in pairs(fibers) do + if f:status() ~= 'dead' then updating = true end + end + fiber.sleep(0.001) +end; +test_run:cmd("setopt delimiter ''"); + +s:get(1) +s:get(2) +s:get(3) +s:get(4) + +counter -- total updates counter -- 2.17.1 From v.shpilevoy at tarantool.org Wed Aug 19 23:37:30 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Wed, 19 Aug 2020 22:37:30 +0200 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-7-gorcunov@gmail.com> <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> <20200817221403.GL2074@grain> Message-ID: <052d70b3-a11c-05db-15e7-87bd1b86ba19@tarantool.org> Hi! Today you said you did all the review fixes. I see that the branch didn't change, and this comment is still not addressed. On 18.08.2020 21:18, Vladislav Shpilevoy wrote: > On 18.08.2020 00:14, Cyrill Gorcunov wrote: >> On Mon, Aug 17, 2020 at 10:49:23PM +0200, Vladislav Shpilevoy wrote: >>> It seems this commit is not needed - I dropped it and nothing changed. >>> Even no merge/rebase conflicts. >>> >>> On 17.08.2020 15:39, Cyrill Gorcunov wrote: >>>> We need to access first and last xrow in a queue >>>> frenquently and opencoded variants are too ugly. >>> >>> frenquently -> frequently. >> >> The code is ugly as hell without it :/ > > There are many ugly things, but it does not mean we need to rush > changing them. It was discussed already many times. Please, drop > this commit. It is not necessary. It does not interfere with the > other patches in a single line. From gorcunov at gmail.com Wed Aug 19 23:49:27 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Wed, 19 Aug 2020 23:49:27 +0300 Subject: [Tarantool-patches] [PATCH v8 6/9] applier: add shorthands to queue access In-Reply-To: <052d70b3-a11c-05db-15e7-87bd1b86ba19@tarantool.org> References: <20200817133918.875558-1-gorcunov@gmail.com> <20200817133918.875558-7-gorcunov@gmail.com> <4f76adf1-3185-d4e2-3a88-8d312c9e5951@tarantool.org> <20200817221403.GL2074@grain> <052d70b3-a11c-05db-15e7-87bd1b86ba19@tarantool.org> Message-ID: <20200819204927.GP2074@grain> On Wed, Aug 19, 2020 at 10:37:30PM +0200, Vladislav Shpilevoy wrote: > Hi! Today you said you did all the review fixes. I see that > the branch didn't change, and this comment is still not > addressed. > >> > >> The code is ugly as hell without it :/ > > > > There are many ugly things, but it does not mean we need to rush > > changing them. It was discussed already many times. Please, drop > > this commit. It is not necessary. It does not interfere with the > > other patches in a single line. Hmm, seems this your reply somehow drown in my inbox :( I'm pretty damn sure that when we're changin the code we should improve it if we can. I'll update the branch and remove this commit. From gorcunov at gmail.com Thu Aug 20 00:34:35 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:35 +0300 Subject: [Tarantool-patches] [PATCH v9 0/7] qsync: write CONFIRM/ROLLBACK without txn engine Message-ID: <20200819213442.1099018-1-gorcunov@gmail.com> In this series we write CONFIRM/ROLLBACK messages into the WAL directly without involving the txn engine. Vlad, take a look please, once time permit. Since the series reaches v9 I desided to resend it instead of continue replying old thread, just for a final review. First 4 patches you've read already and hopefully I addressed all your comments. issue https://github.com/tarantool/tarantool/issues/5129 branch gorcunov/gh-5129-journal-9 v3: - bootstrap journal left NULL for async write - journal_write_async_cb_t type for async callback - struct synchro_body_bin type for encoded message - xrow_encode_synchro helper to operate with synchro_body_bin v7: - rebase on master - rework applier code v8: - move synchro requests processing into applier_apply_tx (by Vlad) - drop synchro processing from txn_add_redo v9: - rebase on master branch - drop "applier: add shorthands to queue access" from the series Cyrill Gorcunov (7): journal: bind asynchronous write completion to an entry journal: add journal_entry_create helper qsync: provide a binary form of syncro entries qsync: direct write of CONFIRM/ROLLBACK into a journal applier: process synchro requests without txn engine txn: txn_add_redo -- drop synchro processing xrow: drop xrow_header_dup_body src/box/applier.cc | 200 ++++++++++++++++++++++++++++++++------------ src/box/box.cc | 15 ++-- src/box/journal.c | 8 +- src/box/journal.h | 36 ++++++-- src/box/txn.c | 11 +-- src/box/txn_limbo.c | 71 +++++++++------- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++--- src/box/wal.h | 4 +- src/box/xrow.c | 56 ++++--------- src/box/xrow.h | 28 ++++--- 11 files changed, 271 insertions(+), 179 deletions(-) base-commit: ee07eab4da1d00da6ed848f1833cacd32b71c6eb -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:36 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:36 +0300 Subject: [Tarantool-patches] [PATCH v9 1/7] journal: bind asynchronous write completion to an entry In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-2-gorcunov@gmail.com> In commit 77ba0e3504464131fe81c672d508d0275be2173a we've redesigned wal journal operations such that asynchronous write completion is a single instance per journal. It turned out that such simplification is too tight and doesn't allow us to pass entries into the journal with custom completions. Thus lets allow back such ability. We will need it to be able to write "confirm" records into wal directly without touching transactions code at all. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/box.cc | 15 ++++++++------- src/box/journal.c | 2 ++ src/box/journal.h | 20 +++++++++++--------- src/box/txn.c | 2 +- src/box/vy_log.c | 2 +- src/box/wal.c | 19 ++++++++----------- src/box/wal.h | 4 ++-- 7 files changed, 33 insertions(+), 31 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index 8e811e9c1..faffd5769 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -348,7 +348,7 @@ recovery_journal_write(struct journal *base, * Since there're no actual writes, fire a * journal_async_complete callback right away. */ - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -357,7 +357,7 @@ recovery_journal_create(struct vclock *v) { static struct recovery_journal journal; journal_create(&journal.base, recovery_journal_write, - txn_complete_async, recovery_journal_write); + recovery_journal_write); journal.vclock = v; journal_set(&journal.base); } @@ -2182,8 +2182,10 @@ engine_init() static int bootstrap_journal_write(struct journal *base, struct journal_entry *entry) { + (void)base; + entry->res = 0; - journal_async_complete(base, entry); + journal_async_complete(entry); return 0; } @@ -2569,8 +2571,8 @@ box_cfg_xc(void) int64_t wal_max_size = box_check_wal_max_size(cfg_geti64("wal_max_size")); enum wal_mode wal_mode = box_check_wal_mode(cfg_gets("wal_mode")); - if (wal_init(wal_mode, txn_complete_async, cfg_gets("wal_dir"), - wal_max_size, &INSTANCE_UUID, on_wal_garbage_collection, + if (wal_init(wal_mode, cfg_gets("wal_dir"), wal_max_size, + &INSTANCE_UUID, on_wal_garbage_collection, on_wal_checkpoint_threshold) != 0) { diag_raise(); } @@ -2617,8 +2619,7 @@ box_cfg_xc(void) } struct journal bootstrap_journal; - journal_create(&bootstrap_journal, NULL, txn_complete_async, - bootstrap_journal_write); + journal_create(&bootstrap_journal, NULL, bootstrap_journal_write); journal_set(&bootstrap_journal); auto bootstrap_journal_guard = make_scoped_guard([] { journal_set(NULL); diff --git a/src/box/journal.c b/src/box/journal.c index f1e89aaa2..48af9157b 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -36,6 +36,7 @@ struct journal *current_journal = NULL; struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data) { struct journal_entry *entry; @@ -50,6 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } + entry->write_async_cb = write_async_cb; entry->complete_data = complete_data; entry->approx_len = 0; entry->n_rows = n_rows; diff --git a/src/box/journal.h b/src/box/journal.h index 1a10e66c3..4b019fecf 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -42,6 +42,8 @@ extern "C" { struct xrow_header; struct journal_entry; +typedef void (*journal_write_async_f)(struct journal_entry *entry); + /** * An entry for an abstract journal. * Simply put, a write ahead log request. @@ -61,6 +63,10 @@ struct journal_entry { * A journal entry completion callback argument. */ void *complete_data; + /** + * Asynchronous write completion function. + */ + journal_write_async_f write_async_cb; /** * Approximate size of this request when encoded. */ @@ -84,6 +90,7 @@ struct region; */ struct journal_entry * journal_entry_new(size_t n_rows, struct region *region, + journal_write_async_f write_async_cb, void *complete_data); /** @@ -96,22 +103,19 @@ struct journal { int (*write_async)(struct journal *journal, struct journal_entry *entry); - /** Asynchronous write completion */ - void (*write_async_cb)(struct journal_entry *entry); - /** Synchronous write */ int (*write)(struct journal *journal, struct journal_entry *entry); }; /** - * Finalize a single entry. + * Complete asynchronous write. */ static inline void -journal_async_complete(struct journal *journal, struct journal_entry *entry) +journal_async_complete(struct journal_entry *entry) { - assert(journal->write_async_cb != NULL); - journal->write_async_cb(entry); + assert(entry->write_async_cb != NULL); + entry->write_async_cb(entry); } /** @@ -173,12 +177,10 @@ static inline void journal_create(struct journal *journal, int (*write_async)(struct journal *journal, struct journal_entry *entry), - void (*write_async_cb)(struct journal_entry *entry), int (*write)(struct journal *journal, struct journal_entry *entry)) { journal->write_async = write_async; - journal->write_async_cb = write_async_cb; journal->write = write; } diff --git a/src/box/txn.c b/src/box/txn.c index 9c21258c5..cc1f496c5 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -551,7 +551,7 @@ txn_journal_entry_new(struct txn *txn) /* Save space for an additional NOP row just in case. */ req = journal_entry_new(txn->n_new_rows + txn->n_applier_rows + 1, - &txn->region, txn); + &txn->region, txn_complete_async, txn); if (req == NULL) return NULL; diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..de4c5205c 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -818,7 +818,7 @@ vy_log_tx_flush(struct vy_log_tx *tx) size_t used = region_used(&fiber()->gc); struct journal_entry *entry; - entry = journal_entry_new(tx_size, &fiber()->gc, NULL); + entry = journal_entry_new(tx_size, &fiber()->gc, NULL, NULL); if (entry == NULL) goto err; diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..045006b60 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -266,10 +266,9 @@ xlog_write_entry(struct xlog *l, struct journal_entry *entry) static void tx_schedule_queue(struct stailq *queue) { - struct wal_writer *writer = &wal_writer_singleton; struct journal_entry *req, *tmp; stailq_foreach_entry_safe(req, tmp, queue, fifo) - journal_async_complete(&writer->base, req); + journal_async_complete(req); } /** @@ -403,9 +402,8 @@ tx_notify_checkpoint(struct cmsg *msg) */ static void wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, - void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, - int64_t wal_max_size, const struct tt_uuid *instance_uuid, + const char *wal_dirname, int64_t wal_max_size, + const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { @@ -415,7 +413,6 @@ wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, journal_create(&writer->base, wal_mode == WAL_NONE ? wal_write_none_async : wal_write_async, - wall_async_cb, wal_mode == WAL_NONE ? wal_write_none : wal_write); @@ -525,15 +522,15 @@ wal_open(struct wal_writer *writer) } int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold) { /* Initialize the state. */ struct wal_writer *writer = &wal_writer_singleton; - wal_writer_create(writer, wal_mode, wall_async_cb, wal_dirname, - wal_max_size, instance_uuid, on_garbage_collection, + wal_writer_create(writer, wal_mode, wal_dirname, wal_max_size, + instance_uuid, on_garbage_collection, on_checkpoint_threshold); /* Start WAL thread. */ @@ -1314,7 +1311,7 @@ wal_write_none_async(struct journal *journal, vclock_merge(&writer->vclock, &vclock_diff); vclock_copy(&replicaset.vclock, &writer->vclock); entry->res = vclock_sum(&writer->vclock); - journal_async_complete(journal, entry); + journal_async_complete(entry); return 0; } diff --git a/src/box/wal.h b/src/box/wal.h index f348dc636..9d0cada46 100644 --- a/src/box/wal.h +++ b/src/box/wal.h @@ -81,8 +81,8 @@ typedef void (*wal_on_checkpoint_threshold_f)(void); * Start WAL thread and initialize WAL writer. */ int -wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), - const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, +wal_init(enum wal_mode wal_mode, const char *wal_dirname, + int64_t wal_max_size, const struct tt_uuid *instance_uuid, wal_on_garbage_collection_f on_garbage_collection, wal_on_checkpoint_threshold_f on_checkpoint_threshold); -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:37 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:37 +0300 Subject: [Tarantool-patches] [PATCH v9 2/7] journal: add journal_entry_create helper In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-3-gorcunov@gmail.com> To create raw journal entries. We will use it to write confirm/rollback entries. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/journal.c | 8 ++------ src/box/journal.h | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 6 deletions(-) diff --git a/src/box/journal.c b/src/box/journal.c index 48af9157b..cb320b557 100644 --- a/src/box/journal.c +++ b/src/box/journal.c @@ -51,11 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, return NULL; } - entry->write_async_cb = write_async_cb; - entry->complete_data = complete_data; - entry->approx_len = 0; - entry->n_rows = n_rows; - entry->res = -1; - + journal_entry_create(entry, n_rows, 0, write_async_cb, + complete_data); return entry; } diff --git a/src/box/journal.h b/src/box/journal.h index 4b019fecf..5d8d5a726 100644 --- a/src/box/journal.h +++ b/src/box/journal.h @@ -83,6 +83,22 @@ struct journal_entry { struct region; +/** + * Initialize a new journal entry. + */ +static inline void +journal_entry_create(struct journal_entry *entry, size_t n_rows, + size_t approx_len, + journal_write_async_f write_async_cb, + void *complete_data) +{ + entry->write_async_cb = write_async_cb; + entry->complete_data = complete_data; + entry->approx_len = approx_len; + entry->n_rows = n_rows; + entry->res = -1; +} + /** * Create a new journal entry. * -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:38 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:38 +0300 Subject: [Tarantool-patches] [PATCH v9 3/7] qsync: provide a binary form of syncro entries In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-4-gorcunov@gmail.com> These msgpack entries will be needed to write them down to a journal without involving txn engine. Same time we would like to be able to allocate them on stack, for this sake the binary form is predefined. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 9 +++++++-- src/box/xrow.c | 41 ++++++++++++++++++----------------------- src/box/xrow.h | 20 +++++++++++++++----- 3 files changed, 40 insertions(+), 30 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index c6a4e5efc..e458dad75 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -283,6 +283,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) .lsn = lsn, }; + /* + * This is a synchronous commit so we can + * use body and row allocated on a stack. + */ + struct synchro_body_bin body; struct xrow_header row; struct request request = { .header = &row, @@ -292,8 +297,8 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) if (txn == NULL) goto rollback; - if (xrow_encode_synchro(&row, &txn->region, &req) != 0) - goto rollback; + xrow_encode_synchro(&row, &body, &req); + /* * This is not really a transaction. It just uses txn API * to put the data into WAL. And obviously it should not diff --git a/src/box/xrow.c b/src/box/xrow.c index bf174c701..9c6fb4fc1 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -893,35 +893,30 @@ xrow_encode_dml(const struct request *request, struct region *region, return iovcnt; } -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req) { - size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + - mp_sizeof_uint(req->replica_id) + - mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); - char *buf = (char *)region_alloc(region, len); - if (buf == NULL) { - diag_set(OutOfMemory, len, "region_alloc", "buf"); - return -1; - } - char *pos = buf; - - pos = mp_encode_map(pos, 2); - pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); - pos = mp_encode_uint(pos, req->replica_id); - pos = mp_encode_uint(pos, IPROTO_LSN); - pos = mp_encode_uint(pos, req->lsn); + /* + * A map with two elements. We don't compress + * numbers to have this structure constant in size, + * which allows us to preallocate it on stack. + */ + body->m_body = 0x80 | 2; + body->k_replica_id = IPROTO_REPLICA_ID; + body->m_replica_id = 0xce; + body->v_replica_id = mp_bswap_u32(req->replica_id); + body->k_lsn = IPROTO_LSN; + body->m_lsn = 0xcf; + body->v_lsn = mp_bswap_u64(req->lsn); memset(row, 0, sizeof(*row)); - row->body[0].iov_base = buf; - row->body[0].iov_len = len; - row->bodycnt = 1; - row->type = req->type; - - return 0; + row->body[0].iov_base = (void *)body; + row->body[0].iov_len = sizeof(*body); + row->bodycnt = 1; } int diff --git a/src/box/xrow.h b/src/box/xrow.h index 02dca74e5..20e82034d 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -240,16 +240,26 @@ struct synchro_request { int64_t lsn; }; +/** Synchro request xrow's body in MsgPack format. */ +struct PACKED synchro_body_bin { + uint8_t m_body; + uint8_t k_replica_id; + uint8_t m_replica_id; + uint32_t v_replica_id; + uint8_t k_lsn; + uint8_t m_lsn; + uint64_t v_lsn; +}; + /** * Encode synchronous replication request. * @param row xrow header. - * @param region Region to use to encode the confirmation body. + * @param body Desination to use to encode the confirmation body. * @param req Request parameters. - * @retval -1 on error. - * @retval 0 success. */ -int -xrow_encode_synchro(struct xrow_header *row, struct region *region, +void +xrow_encode_synchro(struct xrow_header *row, + struct synchro_body_bin *body, const struct synchro_request *req); /** -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:39 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:39 +0300 Subject: [Tarantool-patches] [PATCH v9 4/7] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-5-gorcunov@gmail.com> When we need to write CONFIRM or ROLLBACK message (which is a binary record in msgpack format) into a journal we use txn code to allocate a new transaction, encode there a message and pass it to walk the long txn path before it hit the journal. This is not only resource wasting but also somehow strange from architectural point of view. Instead lets encode a record on the stack and write it to the journal directly. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn_limbo.c | 66 +++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 32 deletions(-) diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c index e458dad75..4b90d7fa5 100644 --- a/src/box/txn_limbo.c +++ b/src/box/txn_limbo.c @@ -32,6 +32,7 @@ #include "txn_limbo.h" #include "replication.h" #include "iproto_constants.h" +#include "journal.h" struct txn_limbo txn_limbo; @@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) return 0; } +/** + * A callback for synchronous write: txn_limbo_write_synchro fiber + * waiting to proceed once a record is written to WAL. + */ +static void +txn_limbo_write_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + fiber_wakeup(entry->complete_data); +} + static void txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) { @@ -285,46 +297,36 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) /* * This is a synchronous commit so we can - * use body and row allocated on a stack. + * allocate everything on a stack. */ struct synchro_body_bin body; struct xrow_header row; - struct request request = { - .header = &row, - }; + char buf[sizeof(struct journal_entry) + + sizeof(struct xrow_header *)]; - struct txn *txn = txn_begin(); - if (txn == NULL) - goto rollback; + struct journal_entry *entry = (struct journal_entry *)buf; + entry->rows[0] = &row; xrow_encode_synchro(&row, &body, &req); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - goto rollback; - if (txn_commit_stmt(txn, &request) != 0) - goto rollback; - if (txn_commit(txn) != 0) - goto rollback; - return; + journal_entry_create(entry, 1, xrow_approx_len(&row), + txn_limbo_write_cb, fiber()); -rollback: - /* - * XXX: the stub is supposed to be removed once it is defined what to do - * when a synchro request WAL write fails. One of the possible - * solutions: log the error, keep the limbo queue as is and probably put - * in rollback mode. Then provide a hook to call manually when WAL - * problems are fixed. Or retry automatically with some period. - */ - panic("Could not write a synchro request to WAL: lsn = %lld, type = " - "%s\n", lsn, iproto_type_name(type)); + if (journal_write(entry) != 0 || entry->res < 0) { + diag_set(ClientError, ER_WAL_IO); + diag_log(); + /* + * XXX: the stub is supposed to be removed once it is defined + * what to do when a synchro request WAL write fails. One of + * the possible solutions: log the error, keep the limbo + * queue as is and probably put in rollback mode. Then + * provide a hook to call manually when WAL problems are fixed. + * Or retry automatically with some period. + */ + panic("Could not write a synchro request to WAL: " + "lsn = %lld, type = %s\n", lsn, + type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); + } } /** -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:40 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:40 +0300 Subject: [Tarantool-patches] [PATCH v9 5/7] applier: process synchro requests without txn engine In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-6-gorcunov@gmail.com> Transaction processing code is very heavy simply because transactions are carrying various data and involves a number of other mechanisms to proceed. In turn, when we receive confirm or rollback packed from another node in a cluster we just need to inspect limbo queue and write this packed into a WAL journal. So calling a bunch of txn engine helpers is simply waste of cycles. Thus lets rather handle them in a special light way: - allocate synchro_entry structure which would carry the journal entry itself and encoded message - process limbo queue to mark confirmed/rollback'ed messages - finally write this synchro_entry into a journal Which is a way simplier. Part-of #5129 Suggedsted-by: Vladislav Shpilevoy Co-developed-by: Vladislav Shpilevoy Signed-off-by: Cyrill Gorcunov --- src/box/applier.cc | 200 +++++++++++++++++++++++++++++++++------------ 1 file changed, 148 insertions(+), 52 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 1387d518c..c1d07ca54 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -51,8 +51,10 @@ #include "schema.h" #include "txn.h" #include "box.h" +#include "xrow.h" #include "scoped_guard.h" #include "txn_limbo.h" +#include "journal.h" STRS(applier_state, applier_STATE); @@ -268,45 +270,11 @@ process_nop(struct request *request) return txn_commit_stmt(txn, request); } -/* - * CONFIRM/ROLLBACK rows aren't dml requests and require special - * handling: instead of performing some operations on spaces, - * processing these requests requires txn_limbo to either confirm - * or rollback some of its entries. - */ -static int -process_synchro_row(struct request *request) -{ - assert(iproto_type_is_synchro_request(request->header->type)); - struct txn *txn = in_txn(); - - struct synchro_request syn_req; - if (xrow_decode_synchro(request->header, &syn_req) != 0) - return -1; - assert(txn->n_applier_rows == 0); - /* - * This is not really a transaction. It just uses txn API - * to put the data into WAL. And obviously it should not - * go to the limbo and block on the very same sync - * transaction which it tries to confirm now. - */ - txn_set_flag(txn, TXN_FORCE_ASYNC); - - if (txn_begin_stmt(txn, NULL) != 0) - return -1; - if (txn_commit_stmt(txn, request) != 0) - return -1; - return txn_limbo_process(&txn_limbo, &syn_req); -} - static int apply_row(struct xrow_header *row) { struct request request; - if (iproto_type_is_synchro_request(row->type)) { - request.header = row; - return process_synchro_row(&request); - } + assert(!iproto_type_is_synchro_request(row->type)); if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) return -1; if (request.type == IPROTO_NOP) @@ -753,19 +721,9 @@ applier_read_tx(struct applier *applier, struct stailq *rows) next)->row.is_commit); } -static int -applier_txn_rollback_cb(struct trigger *trigger, void *event) +static void +applier_rollback_by_wal_io(void) { - (void) trigger; - struct txn *txn = (struct txn *) event; - /* - * Synchronous transaction rollback due to receiving a - * ROLLBACK entry is a normal event and requires no - * special handling. - */ - if (txn->signature == TXN_SIGNATURE_SYNC_ROLLBACK) - return 0; - /* * Setup shared applier diagnostic area. * @@ -774,9 +732,9 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) * diag use per-applier diag instead all the time * (which actually already present in the structure). * - * But remember that transactions are asynchronous - * and rollback may happen a way latter after it - * passed to the journal engine. + * But remember that WAL writes are asynchronous and + * rollback may happen a way later after it was passed to + * the journal engine. */ diag_set(ClientError, ER_WAL_IO); diag_set_error(&replicaset.applier.diag, @@ -787,6 +745,20 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) /* Rollback applier vclock to the committed one. */ vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); +} + +static int +applier_txn_rollback_cb(struct trigger *trigger, void *event) +{ + (void) trigger; + struct txn *txn = (struct txn *) event; + /* + * Synchronous transaction rollback due to receiving a + * ROLLBACK entry is a normal event and requires no + * special handling. + */ + if (txn->signature != TXN_SIGNATURE_SYNC_ROLLBACK) + applier_rollback_by_wal_io(); return 0; } @@ -800,6 +772,110 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) return 0; } +struct synchro_entry { + /** Encoded form of a synchro record. */ + struct synchro_body_bin body_bin; + + /** xrow to write, used by the journal engine. */ + struct xrow_header row; + + /** + * The journal entry itself. Note since + * it has unsized array it must be the + * last entry in the structure. + */ + struct journal_entry journal_entry; +}; + +static void +synchro_entry_delete(struct synchro_entry *entry) +{ + free(entry); +} + +/** + * Async write journal completion. + */ +static void +apply_synchro_row_cb(struct journal_entry *entry) +{ + assert(entry->complete_data != NULL); + struct synchro_entry *synchro_entry = + (struct synchro_entry *)entry->complete_data; + if (entry->res < 0) + applier_rollback_by_wal_io(); + else + trigger_run(&replicaset.applier.on_wal_write, NULL); + + synchro_entry_delete(synchro_entry); +} + +/** + * Allocate a new synchro_entry to be passed to + * the journal engine in async write way. + */ +static struct synchro_entry * +synchro_entry_new(struct xrow_header *applier_row, + struct synchro_request *req) +{ + struct synchro_entry *entry; + size_t size = sizeof(*entry) + sizeof(struct xrow_header *); + + /* + * For simplicity we use malloc here but + * probably should provide some cache similar + * to txn cache. + */ + entry = (struct synchro_entry *)malloc(size); + if (entry == NULL) { + diag_set(OutOfMemory, size, "malloc", "synchro_entry"); + return NULL; + } + + struct journal_entry *journal_entry = &entry->journal_entry; + struct synchro_body_bin *body_bin = &entry->body_bin; + struct xrow_header *row = &entry->row; + + journal_entry->rows[0] = row; + + xrow_encode_synchro(row, body_bin, req); + + row->lsn = applier_row->lsn; + row->replica_id = applier_row->replica_id; + + journal_entry_create(journal_entry, 1, xrow_approx_len(row), + apply_synchro_row_cb, entry); + return entry; +} + +/** Process a synchro request. */ +static int +apply_synchro_row(struct xrow_header *row) +{ + assert(iproto_type_is_synchro_request(row->type)); + + struct synchro_request req; + if (xrow_decode_synchro(row, &req) != 0) + goto err; + + if (txn_limbo_process(&txn_limbo, &req)) + goto err; + + struct synchro_entry *entry; + entry = synchro_entry_new(row, &req); + if (entry == NULL) + goto err; + + if (journal_write_async(&entry->journal_entry) != 0) { + diag_set(ClientError, ER_WAL_IO); + goto err; + } + return 0; +err: + diag_log(); + return -1; +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -847,13 +923,26 @@ applier_apply_tx(struct stailq *rows) } } + if (unlikely(iproto_type_is_synchro_request(first_row->type))) { + /* + * Synchro messages are not transactions, in terms + * of DML. Always sent and written isolated from + * each other. + */ + assert(first_row == last_row); + if (apply_synchro_row(first_row) != 0) + diag_raise(); + goto success; + } + /** * Explicitly begin the transaction so that we can * control fiber->gc life cycle and, in case of apply * conflict safely access failed xrow object and allocate * IPROTO_NOP on gc. */ - struct txn *txn = txn_begin(); + struct txn *txn; + txn = txn_begin(); struct applier_tx_row *item; if (txn == NULL) { latch_unlock(latch); @@ -922,6 +1011,7 @@ applier_apply_tx(struct stailq *rows) if (txn_commit_async(txn) < 0) goto fail; +success: /* * The transaction was sent to journal so promote vclock. * @@ -1089,7 +1179,13 @@ applier_subscribe(struct applier *applier) applier->lag = TIMEOUT_INFINITY; - /* Register triggers to handle WAL writes and rollbacks. */ + /* + * Register triggers to handle WAL writes and rollbacks. + * + * Note we use them for syncronous packets handling as well + * thus when changing make sure that synchro handling won't + * be broken. + */ struct trigger on_wal_write; trigger_create(&on_wal_write, applier_on_wal_write, applier, NULL); trigger_add(&replicaset.applier.on_wal_write, &on_wal_write); -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:41 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:41 +0300 Subject: [Tarantool-patches] [PATCH v9 6/7] txn: txn_add_redo -- drop synchro processing In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-7-gorcunov@gmail.com> Since we no longer use txn engine for synchro packets processing this code is never executed. Part-of #5129 Signed-off-by: Cyrill Gorcunov --- src/box/txn.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/src/box/txn.c b/src/box/txn.c index cc1f496c5..b2d342355 100644 --- a/src/box/txn.c +++ b/src/box/txn.c @@ -82,14 +82,7 @@ txn_add_redo(struct txn *txn, struct txn_stmt *stmt, struct request *request) */ struct space *space = stmt->space; row->group_id = space != NULL ? space_group_id(space) : 0; - /* - * Sychronous replication entries are supplementary and - * aren't valid dml requests. They're encoded manually. - */ - if (likely(!iproto_type_is_synchro_request(row->type))) - row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); - else - row->bodycnt = xrow_header_dup_body(row, &txn->region); + row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); if (row->bodycnt < 0) return -1; stmt->row = row; -- 2.26.2 From gorcunov at gmail.com Thu Aug 20 00:34:42 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 00:34:42 +0300 Subject: [Tarantool-patches] [PATCH v9 7/7] xrow: drop xrow_header_dup_body In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: <20200819213442.1099018-8-gorcunov@gmail.com> We no longer use it. Closes #5129 Signed-off-by: Cyrill Gorcunov --- src/box/xrow.c | 15 --------------- src/box/xrow.h | 8 -------- 2 files changed, 23 deletions(-) diff --git a/src/box/xrow.c b/src/box/xrow.c index 9c6fb4fc1..95ddb1fe7 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -220,21 +220,6 @@ xrow_header_decode(struct xrow_header *header, const char **pos, return 0; } -int -xrow_header_dup_body(struct xrow_header *row, struct region *region) -{ - assert(row->bodycnt == 1); - size_t size = row->body[0].iov_len; - char *copy = (char *)region_alloc(region, size); - if (copy == NULL) { - diag_set(OutOfMemory, size, "region_alloc", "copy"); - return -1; - } - memcpy(copy, row->body[0].iov_base, size); - row->body[0].iov_base = copy; - return 1; -} - /** * @pre pos points at a valid msgpack */ diff --git a/src/box/xrow.h b/src/box/xrow.h index 20e82034d..58d47b12d 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -141,14 +141,6 @@ int xrow_header_decode(struct xrow_header *header, const char **pos, const char *end, bool end_is_exact); -/** - * Duplicate the xrow's body onto the given region. - * @retval -1 Error. - * @retval >= 0 Iov count in the body. - */ -int -xrow_header_dup_body(struct xrow_header *header, struct region *region); - /** * DML request. */ -- 2.26.2 From v.shpilevoy at tarantool.org Thu Aug 20 01:20:23 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 20 Aug 2020 00:20:23 +0200 Subject: [Tarantool-patches] [PATCH v2 2/2] sql: support column addition In-Reply-To: <3FF95208-4C8E-4094-A005-7A50415A8651@tarantool.org> References: <20200403152752.8923-1-roman.habibov@tarantool.org> <20200403152752.8923-3-roman.habibov@tarantool.org> <5CF72787-A1F0-4C48-BA8F-08F02B6960F6@tarantool.org> <3FF95208-4C8E-4094-A005-7A50415A8651@tarantool.org> Message-ID: <8c4390aa-6b11-0d86-89f5-917226b75610@tarantool.org> Hi! Thanks for the patch! >>> + parser->has_autoinc = false; >>> region_create(&parser->region, &cord()->slabc); >>> } >>> >>> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h >>> index aa6a470f8..3143ec521 100644 >>> --- a/src/box/sql/sqlInt.h >>> +++ b/src/box/sql/sqlInt.h >>> @@ -2249,12 +2249,26 @@ struct Parse { >>> struct enable_entity_def enable_entity_def; >>> }; >>> /** >>> - * Table def is not part of union since information >>> - * being held must survive till the end of parsing of >>> - * whole CREATE TABLE statement (to pass it to >>> - * sqlEndTable() function). >>> + * Table def or column def is not part of union since >>> + * information being held must survive till the end of >>> + * parsing of whole or >>> + * statement (to pass it to >>> + * sqlEndTable() sql_create_column_end() function). >>> */ >>> struct create_table_def create_table_def; >>> + struct create_column_def create_column_def; >>> + /** >>> + * FK and CK constraints appeared in a or >>> + * a statement. >>> + */ >>> + struct rlist fkeys; >>> + struct rlist checks; >>> + uint32_t fkey_count; >>> + uint32_t check_count; >>> + /** True, if column to be created has . */ >>> + bool has_autoinc; >> >> 27. What column? This is struct Parse, it is not a column. > I know, but I haven't come up with anything better. Why can't autoinc field be moved into create_column_def, since this is related to only one column? Can you move struct rlist fkeys and uint32_t fkey_count into a new struct like 'struct create_fkeys_def;', and the same for checks? From v.shpilevoy at tarantool.org Thu Aug 20 01:20:28 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 20 Aug 2020 00:20:28 +0200 Subject: [Tarantool-patches] [PATCH v3 1/4] sql: rename TK_COLUMN to TK_COLUMN_NAME In-Reply-To: <20200811003338.45084-2-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> <20200811003338.45084-2-roman.habibov@tarantool.org> Message-ID: Thanks for the patch! > diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h > index adf90d824..beb83ce95 100644 > --- a/src/box/sql/sqlInt.h > +++ b/src/box/sql/sqlInt.h > @@ -1587,20 +1587,20 @@ struct Expr { > #if SQL_MAX_EXPR_DEPTH>0 > int nHeight; /* Height of the tree headed by this node */ > #endif > - int iTable; /* TK_COLUMN: cursor number of table holding column > + int iTable; /* TK_COLUMN_NAME: cursor number of table holding column > * TK_REGISTER: register number > * TK_TRIGGER: 1 -> new, 0 -> old > * EP_Unlikely: 134217728 times likelihood > * TK_SELECT: 1st register of result vector > */ > - ynVar iColumn; /* TK_COLUMN: column index. > + ynVar iColumn; /* TK_COLUMN_NAME: column index. Does not this look wrong to you? - 'COLUMN_NAME: column index'. In some other places TK_COLUMN_NAME also designates presense of a column index, not name. Probably a better name would be TK_COLUMN_REF. > * TK_VARIABLE: variable number (always >= 1). > * TK_SELECT_COLUMN: column of the result vector > */ > i16 iAgg; /* Which entry in pAggInfo->aCol[] or ->aFunc[] */ > i16 iRightJoinTable; /* If EP_FromJoin, the right table of the join */ > u8 op2; /* TK_REGISTER: original value of Expr.op > - * TK_COLUMN: the value of p5 for OP_Column > + * TK_COLUMN_NAME: the value of p5 for OP_Column > * TK_AGG_FUNCTION: nesting depth > */ > AggInfo *pAggInfo; /* Used by TK_AGG_COLUMN and TK_AGG_FUNCTION */ From v.shpilevoy at tarantool.org Thu Aug 20 01:20:30 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 20 Aug 2020 00:20:30 +0200 Subject: [Tarantool-patches] [PATCH v3 4/4] sql: support column addition In-Reply-To: <20200811003338.45084-5-roman.habibov@tarantool.org> References: <20200811003338.45084-1-roman.habibov@tarantool.org> <20200811003338.45084-5-roman.habibov@tarantool.org> Message-ID: <41180845-f742-d78d-c692-2e4244705319@tarantool.org> Thanks for the patch! See 20 comments below. > diff --git a/src/box/sql/build.c b/src/box/sql/build.c > index 9013bc86f..6f3d2747d 100644 > --- a/src/box/sql/build.c > +++ b/src/box/sql/build.c > @@ -285,48 +285,113 @@ sql_field_retrieve(Parse *parser, struct space_def *space_def, uint32_t id) > return field; > } > > -/* > - * Add a new column to the table currently being constructed. > +/** > + * Make shallow copy of @a space on region. > * > - * The parser calls this routine once for each column declaration > - * in a CREATE TABLE statement. sqlStartTable() gets called > - * first to get things going. Then this routine is called for each > - * column. > + * Function is used to add a new column to an existing space with > + * statement. Copy space def and index > + * array to create constraints appeared in the statement. The > + * index array copy will be modified by adding new elements to it. > + * It is necessary, because the statement may contain several > + * index definitions (constraints). > */ > +static struct space * > +sql_shallow_space_copy(struct Parse *parse, struct space *space) > +{ > + assert(space->def != NULL); > + struct space *ret = sql_ephemeral_space_new(parse, space->def->name); > + if (ret == NULL) > + return NULL; > + ret->index_count = space->index_count; > + ret->index_id_max = space->index_id_max; > + uint32_t indexes_sz = sizeof(struct index *) * (ret->index_count); > + ret->index = (struct index **) malloc(indexes_sz); 1. Why can't you use parser's region? > + if (ret->index == NULL) { > + diag_set(OutOfMemory, indexes_sz, "realloc", "ret->index"); 2. It is not realloc, it is malloc. 3. Seems you need to set parser->is_aborted = true in case of an error. As far as I understand, it is a contract of each function taking Parse argument. > + return NULL; > + } > + memcpy(ret->index, space->index, indexes_sz); > + memcpy(ret->def, space->def, sizeof(struct space_def)); > + ret->def->opts.is_temporary = true; > + ret->def->opts.is_ephemeral = true; > + if (ret->def->field_count != 0) { > + uint32_t fields_size = 0; > + ret->def->fields = > + region_alloc_array(&parse->region, > + typeof(struct field_def), > + ret->def->field_count, &fields_size); > + if (ret->def->fields == NULL) { > + diag_set(OutOfMemory, fields_size, "region_alloc", > + "ret->def->fields"); > + free(ret->index); > + return NULL; > + } > + memcpy(ret->def->fields, space->def->fields, fields_size); > + } > + > + return ret; > +} > @@ -334,18 +399,86 @@ sqlAddColumn(Parse * pParse, Token * pName, struct type_def *type_def) > +void > +sql_create_column_end(struct Parse *parse) > +{ > + struct space *space = parse->create_column_def.space; > + assert(space != NULL); > + struct space_def *def = space->def; > + struct field_def *field = &def->fields[def->field_count - 1]; > + if (field->nullable_action == ON_CONFLICT_ACTION_DEFAULT) { > + field->nullable_action = ON_CONFLICT_ACTION_NONE; > + field->is_nullable = true; > + } > + /* > + * Encode the format array and emit code to update _space. > + */ > + uint32_t table_stmt_sz = 0; > + struct region *region = &parse->region; > + char *table_stmt = sql_encode_table(region, def, &table_stmt_sz); > + char *raw = sqlDbMallocRaw(parse->db, table_stmt_sz); > + if (table_stmt == NULL || raw == NULL) { > + parse->is_aborted = true; > + return; > + } > + memcpy(raw, table_stmt, table_stmt_sz); > + > + struct Vdbe *v = sqlGetVdbe(parse); > + assert(v != NULL); > + > + struct space *system_space = space_by_id(BOX_SPACE_ID); > + assert(system_space != NULL); > + int cursor = parse->nTab++; > + vdbe_emit_open_cursor(parse, cursor, 0, system_space); > + sqlVdbeChangeP5(v, OPFLAG_SYSTEMSP); > + > + int key_reg = ++parse->nMem; > + sqlVdbeAddOp2(v, OP_Integer, def->id, key_reg); > + int addr = sqlVdbeAddOp4Int(v, OP_Found, cursor, 0, key_reg, 1); > + sqlVdbeAddOp2(v, OP_Halt, -1, ON_CONFLICT_ACTION_ABORT); > + sqlVdbeJumpHere(v, addr); > + > + int tuple_reg = sqlGetTempRange(parse, box_space_field_MAX + 1); 4. You need to call sqlReleaseTempRange() somewhere. > + for (int i = 0; i < box_space_field_MAX - 1; ++i) > + sqlVdbeAddOp3(v, OP_Column, cursor, i, tuple_reg + i); > + sqlVdbeAddOp1(v, OP_Close, cursor); > + > + sqlVdbeAddOp2(v, OP_Integer, def->field_count, tuple_reg + 4); > + sqlVdbeAddOp4(v, OP_Blob, table_stmt_sz, tuple_reg + 6, > + SQL_SUBTYPE_MSGPACK, raw, P4_DYNAMIC); > + sqlVdbeAddOp3(v, OP_MakeRecord, tuple_reg, box_space_field_MAX, > + tuple_reg + box_space_field_MAX); > + sqlVdbeAddOp4(v, OP_IdxReplace, tuple_reg + box_space_field_MAX, 0, 0, > + (char *) system_space, P4_SPACEPTR); > + sql_vdbe_create_constraints(parse, key_reg); > + > + /* > + * Clean up array allocated in sql_shallow_space_copy(). > + */ > + free(space->index); 5. It may happen that sql_create_column_end() is never called. For example, if an error is encountered somewhere inside column definition, after sql_create_column_start() is called. > } > > void > sql_column_add_nullable_action(struct Parse *parser, > enum on_conflict_action nullable_action) > { > - struct space *space = parser->create_table_def.new_space; > - if (space == NULL || NEVER(space->def->field_count < 1)) > + assert(parser->create_column_def.space != NULL); > + struct space_def *def = parser->create_column_def.space->def; > + if (NEVER(def->field_count < 1)) > return; > - struct space_def *def = space->def; > struct field_def *field = &def->fields[def->field_count - 1]; > if (field->nullable_action != ON_CONFLICT_ACTION_DEFAULT && > nullable_action != field->nullable_action) { > @@ -364,51 +497,42 @@ sql_column_add_nullable_action(struct Parse *parser, > } > > /* > - * The expression is the default value for the most recently added column > - * of the table currently under construction. > + * The expression is the default value for the most recently added > + * column. > * > * Default value expressions must be constant. Raise an exception if this > * is not the case. > * > * This routine is called by the parser while in the middle of > - * parsing a CREATE TABLE statement. > + * parsing a or a > + * statement. > */ > void > sqlAddDefaultValue(Parse * pParse, ExprSpan * pSpan) > { > sql *db = pParse->db; > - struct space *p = pParse->create_table_def.new_space; > - if (p != NULL) { > - assert(p->def->opts.is_ephemeral); > - struct space_def *def = p->def; > - if (!sqlExprIsConstantOrFunction > - (pSpan->pExpr, db->init.busy)) { > - const char *column_name = > - def->fields[def->field_count - 1].name; > - diag_set(ClientError, ER_CREATE_SPACE, def->name, > - tt_sprintf("default value of column '%s' is "\ > - "not constant", column_name)); > + assert(pParse->create_column_def.space != NULL); > + struct space_def *def = pParse->create_column_def.space->def; > + struct field_def *field = &def->fields[def->field_count - 1]; > + if (!sqlExprIsConstantOrFunction(pSpan->pExpr, db->init.busy)) { > + diag_set(ClientError, ER_CREATE_SPACE, def->name, > + tt_sprintf("default value of column '%s' is not " > + "constant", field->name)); > + pParse->is_aborted = true; > + } else { > + struct region *region = &pParse->region; > + uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); > + field->default_value = region_alloc(region, default_length + 1); > + if (field->default_value == NULL) { > + diag_set(OutOfMemory, default_length + 1, > + "region_alloc", "field->default_value"); > pParse->is_aborted = true; > - } else { > - assert(def != NULL); > - struct field_def *field = > - &def->fields[def->field_count - 1]; > - struct region *region = &pParse->region; > - uint32_t default_length = (int)(pSpan->zEnd - pSpan->zStart); > - field->default_value = region_alloc(region, > - default_length + 1); > - if (field->default_value == NULL) { > - diag_set(OutOfMemory, default_length + 1, > - "region_alloc", > - "field->default_value"); > - pParse->is_aborted = true; > - return; > - } > - strncpy(field->default_value, pSpan->zStart, > - default_length); > - field->default_value[default_length] = '\0'; > + goto add_default_value_exit; > } > + strncpy(field->default_value, pSpan->zStart, default_length); > + field->default_value[default_length] = '\0'; > } > +add_default_value_exit: > sql_expr_delete(db, pSpan->pExpr, false); 6. Was it necessary to make so many changes? Wouldn't it work if you would just replace struct space *p = pParse->create_table_def.new_space; with struct space *p = pParse->create_column_def.space; ? > } > > @@ -574,8 +700,10 @@ sql_create_check_contraint(struct Parse *parser) > (struct alter_entity_def *) create_ck_def; > assert(alter_def->entity_type == ENTITY_TYPE_CK); > (void) alter_def; > - struct space *space = parser->create_table_def.new_space; > - bool is_alter = space == NULL; > + struct space *space = parser->create_column_def.space; > + if (space == NULL) > + space = parser->create_table_def.new_space; 7. Why in some places you check create_table_def.new_space != NULL first, and in some places create_column_def.space != NULL first? Is the order important somehow? > + bool is_alter_add_constr = space == NULL; > /* Prepare payload for ck constraint definition. */ > struct region *region = &parser->region;> @@ -704,8 +845,7 @@ sql_column_collation(struct space_def *def, uint32_t column, uint32_t *coll_id) > * > * In cases mentioned above collation is fetched by id. > */ > - if (space == NULL) { > - assert(def->opts.is_ephemeral); > + if (def->opts.is_ephemeral) { 8. space_by_id() above is not needed, if the definition is ephemeral now. It can be moved below this condition so as not to call it when the result is not going to be used anyway. > assert(column < (uint32_t)def->field_count); > *coll_id = def->fields[column].coll_id; > struct coll_id *collation = coll_by_id(*coll_id); > @@ -1148,15 +1292,21 @@ resolve_link(struct Parse *parse_context, const struct space_def *def, > > /** > * Emit code to create sequences, indexes, check and foreign key > - * constraints appeared in . > + * constraints appeared in or > + * . > */ > static void > sql_vdbe_create_constraints(struct Parse *parse, int reg_space_id) > { > assert(reg_space_id != 0); > struct space *space = parse->create_table_def.new_space; > - assert(space != NULL); > + bool is_alter = space == NULL; > uint32_t i = 0; > + if (is_alter) { > + space = parse->create_column_def.space; > + i = space_by_name(space->def->name)->index_count; 9. Why do you need the original space for that? sql_shallow_space_copy() copies index_count as well. > + } > + assert(space != NULL); > for (; i < space->index_count; ++i) { > struct index *idx = space->index[i]; > vdbe_emit_create_index(parse, space->def, idx->def, > @@ -1908,6 +2077,8 @@ sql_create_foreign_key(struct Parse *parse_context) > goto tnt_error; > } > memset(fk_parse, 0, sizeof(*fk_parse)); > + if (parse_context->create_column_def.space != NULL) > + child_space = space; 10. Why? > rlist_add_entry(&parse_context->fkeys, fk_parse, link); > } > struct Token *parent = create_fk_def->parent_name; > @@ -1920,28 +2091,45 @@ sql_create_foreign_key(struct Parse *parse_context) > struct space *parent_space = space_by_name(parent_name); > - if (parent_space == NULL) { > - if (is_self_referenced) { > - struct fk_constraint_parse *fk = > - rlist_first_entry(&parse_context->fkeys, > - struct fk_constraint_parse, > - link); > - fk->selfref_cols = parent_cols; > - fk->is_self_referenced = true; > - } else { > - diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name);; > - goto tnt_error; > - } > + if (parent_space == NULL && !is_self_referenced) { > + diag_set(ClientError, ER_NO_SUCH_SPACE, parent_name); > + goto tnt_error; > + } > + if (is_self_referenced) { > + struct fk_constraint_parse *fk = > + rlist_first_entry(&parse_context->fkeys, > + struct fk_constraint_parse, > + link); > + fk->selfref_cols = parent_cols; > + fk->is_self_referenced = true; > } ^^^ 11. This refactoring seems unnecessary. What changed here functionally? > - if (!is_alter) { > + if (!is_alter_add_constr) { > if (create_def->name.n == 0) { > - constraint_name = > - sqlMPrintf(db, "fk_unnamed_%s_%d", > - space->def->name, > - ++parse_context->fkey_count); > + uint32_t idx = ++parse_context->fkey_count; > + /* > + * If it is we > + * should count the existing FK > + * constraints in the space and form a > + * name based on this. > + */ > + if (table_def->new_space == NULL) { > + struct space *original_space = > + space_by_name(space->def->name); > + assert(original_space != NULL); > + struct rlist *child_fk = > + &original_space->child_fk_constraint; > + if (!rlist_empty(child_fk)) { 12. You don't need to check for emptiness. rlist_foreach_entry() works fine with an empty list. > + struct fk_constraint *fk; > + rlist_foreach_entry(fk, child_fk, > + in_child_space) > + idx++; > + } > + } > + constraint_name = sqlMPrintf(db, "fk_unnamed_%s_%d", > + space->def->name, idx); > } else { > constraint_name = > sql_name_from_token(db, &create_def->name); > @@ -2001,7 +2189,8 @@ sql_create_foreign_key(struct Parse *parse_context) > } > int actions = create_fk_def->actions; > fk_def->field_count = child_cols_count; > - fk_def->child_id = child_space != NULL ? child_space->def->id : 0; > + fk_def->child_id = table_def->new_space == NULL ? > + child_space->def->id : 0; 13. Why? > fk_def->parent_id = parent_space != NULL ? parent_space->def->id : 0; > fk_def->is_deferred = create_constr_def->is_deferred; > fk_def->match = (enum fk_constraint_match) (create_fk_def->match); > @@ -2420,10 +2613,8 @@ sql_create_index(struct Parse *parse) { > } > goto exit_create_index; > } > - } else { > - if (parse->create_table_def.new_space == NULL) > - goto exit_create_index; > - space = parse->create_table_def.new_space; > + } else if (space == NULL) { 14. Why not !is_create_table_or_add_col? > + goto exit_create_index; > } > struct space_def *def = space->def; > > diff --git a/src/box/sql/parse.y b/src/box/sql/parse.y > index 995875566..0c9887851 100644 > --- a/src/box/sql/parse.y > +++ b/src/box/sql/parse.y > @@ -281,9 +286,11 @@ nm(A) ::= id(A). { > } > } > > -// "carglist" is a list of additional constraints that come after the > -// column name and column type in a CREATE TABLE statement. > -// > +/* 15. Out-of-function comments are started from /**. > + * "carglist" is a list of additional constraints and clauses that > + * come after the column name and column type in a > + * or statement. > + */ > carglist ::= carglist ccons. > carglist ::= . > %type cconsname { struct Token } > diff --git a/src/box/sql/parse_def.h b/src/box/sql/parse_def.h > index 1105fda6e..336914c57 100644 > --- a/src/box/sql/parse_def.h > +++ b/src/box/sql/parse_def.h > @@ -207,6 +209,14 @@ struct create_table_def { > struct space *new_space; > }; > > +struct create_column_def { > + struct create_entity_def base; > + /** Shallow space_def copy. */ 16. Not space_def. > + struct space *space; > + /** Column type. */ > + struct type_def *type_def; > +}; > + > struct create_view_def { > struct create_entity_def base; > /** > diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h > index fa87e7bd2..32142a871 100644 > --- a/src/box/sql/sqlInt.h > +++ b/src/box/sql/sqlInt.h > @@ -2860,15 +2864,30 @@ struct space *sqlResultSetOfSelect(Parse *, Select *); > > struct space * > sqlStartTable(Parse *, Token *); > -void sqlAddColumn(Parse *, Token *, struct type_def *); > + > +/** > + * Add new field to the format of ephemeral space in > + * create_table_def. If it is create shallow copy of > + * the existing space and add field to its format. 17. It fills create_column_def, not create_table_def. > + */ > +void > +sql_create_column_start(struct Parse *parse); > + > diff --git a/test/sql/add-column.result b/test/sql/add-column.result > new file mode 100644 > index 000000000..f86259105 > --- /dev/null > +++ b/test/sql/add-column.result > @@ -0,0 +1,471 @@ > +-- test-run result file version 2 > +-- > +-- gh-3075: Check statement. > +-- > +CREATE TABLE t1 (a INT PRIMARY KEY); > + | --- > + | - row_count: 1 > + | ... > + > +-- > +-- COLUMN keyword is optional. Check it here, but omit it below. > +-- > +ALTER TABLE t1 ADD COLUMN b INT; > + | --- > + | - row_count: 0 18. It seems you need to return row_count 1. To be consistent with other ALTER TABLE expressions. > + | ... > + > +-- > +-- A column with the same name already exists. > +-- > +ALTER TABLE t1 ADD b SCALAR; > + | --- > + | - null > + | - Space field 'B' is duplicate > + | ... > + > +-- > +-- Can't add column to a view. > +-- > +CREATE VIEW v AS SELECT * FROM t1; > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE v ADD b INT; > + | --- > + | - null > + | - Can't add column 'B'. 'V' is a view 19. What if I do the same via direct replace into _space in Lua? > + | ... > +DROP VIEW v; > + | --- > + | - row_count: 1 > + | ... > + > +-- > +-- Check PRIMARY KEY constraint works with an added column. > +-- > +CREATE TABLE pk_check (a INT CONSTRAINT pk PRIMARY KEY); > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE pk_check DROP CONSTRAINT pk; > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE pk_check ADD b INT PRIMARY KEY; > + | --- > + | - row_count: 0 > + | ... > +INSERT INTO pk_check VALUES (1, 1); > + | --- > + | - row_count: 1 > + | ... > +INSERT INTO pk_check VALUES (1, 1); > + | --- > + | - null > + | - Duplicate key exists in unique index 'pk_unnamed_PK_CHECK_1' in space 'PK_CHECK' > + | ... > +DROP TABLE pk_check; > + | --- > + | - row_count: 1 > + | ... > + > +-- > +-- Check UNIQUE constraint works with an added column. > +-- > +CREATE TABLE unique_check (a INT PRIMARY KEY); > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE unique_check ADD b INT UNIQUE; > + | --- > + | - row_count: 0 > + | ... > +INSERT INTO unique_check VALUES (1, 1); > + | --- > + | - row_count: 1 > + | ... > +INSERT INTO unique_check VALUES (2, 1); > + | --- > + | - null > + | - Duplicate key exists in unique index 'unique_unnamed_UNIQUE_CHECK_2' in space 'UNIQUE_CHECK' > + | ... > +DROP TABLE unique_check; > + | --- > + | - row_count: 1 > + | ... > + > +-- > +-- Check CHECK constraint works with an added column. > +-- > +CREATE TABLE ck_check (a INT PRIMARY KEY); > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE ck_check ADD b INT CHECK (b > 0); > + | --- > + | - row_count: 0 > + | ... > +INSERT INTO ck_check VALUES (1, 0); > + | --- > + | - null > + | - 'Check constraint failed ''ck_unnamed_CK_CHECK_1'': b > 0' > + | ... > +DROP TABLE ck_check; > + | --- > + | - row_count: 1 > + | ... > + > +-- > +-- Check FOREIGN KEY constraint works with an added column. > +-- > +CREATE TABLE fk_check (a INT PRIMARY KEY); > + | --- > + | - row_count: 1 > + | ... > +ALTER TABLE fk_check ADD b INT REFERENCES t1(a); > + | --- > + | - row_count: 0 > + | ... > +INSERT INTO fk_check VALUES (0, 1); > + | --- > + | - null > + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' > + | ... > +INSERT INTO fk_check VALUES (2, 0); > + | --- > + | - null > + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' > + | ... > +INSERT INTO fk_check VALUES (2, 1); > + | --- > + | - null > + | - 'Failed to execute SQL statement: FOREIGN KEY constraint failed' 20. It is worth adding one more test with a successfull insertion. From alexander.turenko at tarantool.org Thu Aug 20 01:35:52 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Thu, 20 Aug 2020 01:35:52 +0300 Subject: [Tarantool-patches] [PATCH v6] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <344b8b406afa1464aae59f9be231ff78f053b996.1597817683.git.avtikhon@tarantool.org> References: <344b8b406afa1464aae59f9be231ff78f053b996.1597817683.git.avtikhon@tarantool.org> Message-ID: <20200819223552.nplchepqb67tojxr@tkn_work_nb> This description becomes much better! On Wed, Aug 19, 2020 at 09:17:39AM +0300, Alexander V. Tikhonov wrote: > During implementation of openSUSE build with testing got failed test > box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and > vinyl_dir existed and also errno was set to ENOENT, box configuration > succeeded, but it shouldn't. Reason of this wrong behaviour was that > not all of the failure paths in xdir_scan() set errno, but the caller > assumed it. > > Debugging the issue found that after xdir_scan() there was incorrect > check for errno when it returned negative values. xdir_scan() is not > system call and negative return value from it doesn't mean that errno > whould be set too. Found that in situations when errno was left from > previous commands before xdir_scan() and xdir_scan() returned negative > value by itself than the check was wrong. The previous logic of the > check was to catch the error ENOENT from inside the xdir_scan() > function to handle the situation when vinyl_dir was not exist. In this > way errno should be reseted before xdir_scan() call to give the ability > for xdir_scan() to use return value without errno set and correctly > handle errno from inside the xdir_scan(). 'from inside' can be interpreted as 'to catch from inside', not as 'ENOENT from inside'. The same for 'handle errno for inside': 'to handle from inside' or 'errno from inside'? 'to use return value without errno set' is unclear for me. Are not it is intended to give the same idea as the previous 'errno should be reset before xdir_scan()'? If so, it look redundant for me. Typo: whould. Typo: reseted -> reset (irregular verb). > > After discussions found that there was alternative better solution to > fix it. As mentioned above xdir_scan() function is not system call and > can be changed inside it in any possible way. So check outside of this > function on errno could be broken, because of the xdir_scan() changes. > To avoid of it we must avoid of errno checks outside of the function. > Better solution was to use the flag in xdir_scan(), to check if the > directory should exist. So errno check was removed and instead of it > the check for vinyl_dir existence using flag added. > > Closes #4594 > Needed for #4562 > > Co-authored-by: Alexander Turenko > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > Issue: https://github.com/tarantool/tarantool/issues/4594 > Issue: https://github.com/tarantool/tarantool/issues/4562 From v.shpilevoy at tarantool.org Thu Aug 20 01:43:45 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 20 Aug 2020 00:43:45 +0200 Subject: [Tarantool-patches] [PATCH v9 4/7] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <20200819213442.1099018-5-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-5-gorcunov@gmail.com> Message-ID: <4c0358f3-130f-66d1-770d-c6659b6bbdbd@tarantool.org> Thanks for the patch! > diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c > index e458dad75..4b90d7fa5 100644 > --- a/src/box/txn_limbo.c > +++ b/src/box/txn_limbo.c > @@ -285,46 +297,36 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) > > /* > * This is a synchronous commit so we can > - * use body and row allocated on a stack. > + * allocate everything on a stack. > */ > struct synchro_body_bin body; > struct xrow_header row; > - struct request request = { > - .header = &row, > - }; > + char buf[sizeof(struct journal_entry) + > + sizeof(struct xrow_header *)]; > > - struct txn *txn = txn_begin(); > - if (txn == NULL) > - goto rollback; > + struct journal_entry *entry = (struct journal_entry *)buf; > + entry->rows[0] = &row; > > xrow_encode_synchro(&row, &body, &req); > > - /* > - * This is not really a transaction. It just uses txn API > - * to put the data into WAL. And obviously it should not > - * go to the limbo and block on the very same sync > - * transaction which it tries to confirm now. > - */ > - txn_set_flag(txn, TXN_FORCE_ASYNC); > - > - if (txn_begin_stmt(txn, NULL) != 0) > - goto rollback; > - if (txn_commit_stmt(txn, &request) != 0) > - goto rollback; > - if (txn_commit(txn) != 0) > - goto rollback; > - return; > + journal_entry_create(entry, 1, xrow_approx_len(&row), > + txn_limbo_write_cb, fiber()); > > -rollback: > - /* > - * XXX: the stub is supposed to be removed once it is defined what to do > - * when a synchro request WAL write fails. One of the possible > - * solutions: log the error, keep the limbo queue as is and probably put > - * in rollback mode. Then provide a hook to call manually when WAL > - * problems are fixed. Or retry automatically with some period. > - */ > - panic("Could not write a synchro request to WAL: lsn = %lld, type = " > - "%s\n", lsn, iproto_type_name(type)); > + if (journal_write(entry) != 0 || entry->res < 0) { > + diag_set(ClientError, ER_WAL_IO); > + diag_log(); > + /* > + * XXX: the stub is supposed to be removed once it is defined > + * what to do when a synchro request WAL write fails. One of > + * the possible solutions: log the error, keep the limbo > + * queue as is and probably put in rollback mode. Then > + * provide a hook to call manually when WAL problems are fixed. > + * Or retry automatically with some period. > + */ > + panic("Could not write a synchro request to WAL: " > + "lsn = %lld, type = %s\n", lsn, > + type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); Why did you inline iproto_type_name(type)? From avtikhon at tarantool.org Thu Aug 20 07:08:41 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 20 Aug 2020 07:08:41 +0300 Subject: [Tarantool-patches] [PATCH v1] asan/lsan: cleanup suppression lists Message-ID: <076c9037cb2e19264feff4687112622a58af5435.1597896446.git.avtikhon@tarantool.org> Removed asan/lsan suppresions for issues that were not reproduced. Removed skip condition files for tests that passed testing. Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/asan-restore Issue: https://github.com/tarantool/tarantool/issues/4360 asan/asan.supp | 10 --------- asan/lsan.supp | 45 ++------------------------------------ test/app-tap/json.skipcond | 7 ------ test/unit/guard.skipcond | 7 ------ 4 files changed, 2 insertions(+), 67 deletions(-) delete mode 100644 test/app-tap/json.skipcond delete mode 100644 test/unit/guard.skipcond diff --git a/asan/asan.supp b/asan/asan.supp index 79c13ec7d..24cb0845a 100644 --- a/asan/asan.supp +++ b/asan/asan.supp @@ -5,16 +5,6 @@ #fun:* #src:* -# !test: app-tap/json.test.lua -# source: third_party/lua-cjson/lua_cjson.c -fun:json_decode - -# test: unit/base64.test.lua -# source: third_party/base64.c -fun:base64_decode_block -# source: test/unit/base64.c -fun:base64_test - # !test: unit/msgpack.test # source: src/lib/msgpuck/test/msgpuck.c fun:test_mp_print diff --git a/asan/lsan.supp b/asan/lsan.supp index 3273c3baf..1e297d999 100644 --- a/asan/lsan.supp +++ b/asan/lsan.supp @@ -44,54 +44,17 @@ leak:tt_bitset_iterator_init # source: /lib/x86_64-linux-gnu/libc.so* leak:libc.so* -# test: box-tap/schema-mt.test.lua +# test: box-tap/schema_mt.test.lua # source: src/lib/core/coio_task.c leak:coio_on_start # source: src/lib/salad/mhash.h leak:mh_i32ptr_new -# test: replication/misc.test.lua -# source: src/box/vy_log.c -leak:vy_recovery_new_f -# source: src/lib/salad/mhash.h -leak:mh_i64ptr_new - # test: sql-tap/gh2250-trigger-chain-limit.test.lua # source: src/lib/core/exception.cc leak:Exception::operator new -# test: sql-tap/trigger9.test.lua -# source: src/lib/core/fiber.c -leak:cord_start - -# test: sql-tap/tkt-7bbfb7d442.test.lua -# test: sql-tap/view.test.lua -# test: sql-tap/with1.test.lua -# test: sql-tap/with2.test.lua -# source: src/box/sql/malloc.c -leak:sql_sized_malloc - -# test: swim/errinj.test.lua -# test: swim/swim.test.lua -# source: src/lib/swim/swim.c -leak:swim_member_new -leak:swim_update_member_payload - -# !test: unit/bps_tree.test.lua -# source: src/lib/salad/bps_tree.h -leak:bps_tree_test_create_leaf -leak:bps_tree_test_process_insert_leaf - -# !test: unit/heap.test.lua -# source: test/unit/heap.c -leak:test_random_delete_workload -leak:test_delete_last_node - -# !test: unit/heap_iterator.test.lua -# source: src/lib/salad/heap.h -leak:test_heap_reserve - -# !test: unit/swim.test.lua +# test: unit/swim.test.lua # source: src/lib/swim/swim_io.c leak:swim_scheduler_set_codec @@ -99,10 +62,6 @@ leak:swim_scheduler_set_codec # source: src/lib/core/fiber.h leak:fiber_cxx_invoke -# test: vinyl/errinj_ddl.test.lua -# source: src/box/vy_stmt.c -leak:vy_stmt_alloc - # test: vinyl/recover.test.lua # source: src/lib/core/fiber.c leak:cord_costart_thread_func diff --git a/test/app-tap/json.skipcond b/test/app-tap/json.skipcond deleted file mode 100644 index e46fd1088..000000000 --- a/test/app-tap/json.skipcond +++ /dev/null @@ -1,7 +0,0 @@ -import os - -# Disabled at ASAN build due to issue #4360. -if os.getenv("ASAN") == 'ON': - self.skip = 1 - -# vim: set ft=python: diff --git a/test/unit/guard.skipcond b/test/unit/guard.skipcond deleted file mode 100644 index e46fd1088..000000000 --- a/test/unit/guard.skipcond +++ /dev/null @@ -1,7 +0,0 @@ -import os - -# Disabled at ASAN build due to issue #4360. -if os.getenv("ASAN") == 'ON': - self.skip = 1 - -# vim: set ft=python: -- 2.17.1 From gorcunov at gmail.com Thu Aug 20 10:13:21 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Thu, 20 Aug 2020 10:13:21 +0300 Subject: [Tarantool-patches] [PATCH v9 4/7] qsync: direct write of CONFIRM/ROLLBACK into a journal In-Reply-To: <4c0358f3-130f-66d1-770d-c6659b6bbdbd@tarantool.org> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-5-gorcunov@gmail.com> <4c0358f3-130f-66d1-770d-c6659b6bbdbd@tarantool.org> Message-ID: <20200820071321.GQ2074@grain> On Thu, Aug 20, 2020 at 12:43:45AM +0200, Vladislav Shpilevoy wrote: > > + if (journal_write(entry) != 0 || entry->res < 0) { > > + diag_set(ClientError, ER_WAL_IO); > > + diag_log(); > > + /* > > + * XXX: the stub is supposed to be removed once it is defined > > + * what to do when a synchro request WAL write fails. One of > > + * the possible solutions: log the error, keep the limbo > > + * queue as is and probably put in rollback mode. Then > > + * provide a hook to call manually when WAL problems are fixed. > > + * Or retry automatically with some period. > > + */ > > + panic("Could not write a synchro request to WAL: " > > + "lsn = %lld, type = %s\n", lsn, > > + type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); > > Why did you inline iproto_type_name(type)? It happened to sneak in from old commits (I've cherry picked it, and there were no such code previously). I've fixed it and pushed into the branch. Thanks for noticing! From avtikhon at tarantool.org Thu Aug 20 11:02:13 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 20 Aug 2020 11:02:13 +0300 Subject: [Tarantool-patches] [PATCH msgpack v1] test: correct buffer size to fix ASAN error Message-ID: Found ASAN error: [001] + ok 206 - ================================================================= [001] +==6889==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000031 at pc 0x0000005a72e7 bp 0x7ffe47c30c80 sp 0x7ffe47c30c78 [001] +WRITE of size 1 at 0x604000000031 thread T0 [001] + #0 0x5a72e6 in mp_store_u8 /tarantool/src/lib/msgpuck/msgpuck.h:258:1 [001] + #1 0x5a72e6 in mp_encode_uint /tarantool/src/lib/msgpuck/msgpuck.h:1768 [001] + #2 0x4fa657 in test_mp_print /tarantool/src/lib/msgpuck/test/msgpuck.c:957:16 [001] + #3 0x509024 in main /tarantool/src/lib/msgpuck/test/msgpuck.c:1331:2 [001] + #4 0x7f3658fd909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) [001] + #5 0x41f339 in _start (/tnt/test/unit/msgpack.test+0x41f339) [001] + [001] +0x604000000031 is located 0 bytes to the right of 33-byte region [0x604000000010,0x604000000031) [001] +allocated by thread T0 here: [001] + #0 0x4cace3 in malloc (/tnt/test/unit/msgpack.test+0x4cace3) [001] + #1 0x4fa5db in test_mp_print /tarantool/src/lib/msgpuck/test/msgpuck.c:945:18 [001] + #2 0x509024 in main /tarantool/src/lib/msgpuck/test/msgpuck.c:1331:2 [001] + #3 0x7f3658fd909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) [001] + [001] +SUMMARY: AddressSanitizer: heap-buffer-overflow /tarantool/src/lib/msgpuck/msgpuck.h:258:1 in mp_store_u8 [001] +Shadow bytes around the buggy address: [001] + 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [001] + 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [001] + 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [001] + 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [001] + 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [001] +=>0x0c087fff8000: fa fa 00 00 00 00[01]fa fa fa fa fa fa fa fa fa [001] + 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [001] + 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [001] + 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [001] + 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [001] + 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa [001] +Shadow byte legend (one shadow byte represents 8 application bytes): [001] + Addressable: 00 [001] + Partially addressable: 01 02 03 04 05 06 07 [001] + Heap left redzone: fa [001] + Freed heap region: fd [001] + Stack left redzone: f1 [001] + Stack mid redzone: f2 [001] + Stack right redzone: f3 [001] + Stack after return: f5 [001] + Stack use after scope: f8 [001] + Global redzone: f9 [001] + Global init order: f6 [001] + Poisoned by user: f7 [001] + Container overflow: fc [001] + Array cookie: ac [001] + Intra object redzone: bb [001] + ASan internal: fe [001] + Left alloca redzone: ca Invetigated the buffer size that was allocated - it was 33 bytes, but it needed 34. The fix was to increase this buffer. Part of tarantool/tarantool#4360 --- Github: https://github.com/tarantool/msgpuck/tree/avtikhon/gh-4360-fix-asan-error Issue: https://github.com/tarantool/tarantool/issues/4360 test/msgpuck.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/msgpuck.c b/test/msgpuck.c index 2d9bcbf..6613234 100644 --- a/test/msgpuck.c +++ b/test/msgpuck.c @@ -940,7 +940,7 @@ test_mp_print() /* Test mp_snprint max nesting depth. */ int mp_buff_sz = MP_PRINT_MAX_DEPTH * mp_sizeof_array(1) + - mp_sizeof_uint(1); + mp_sizeof_uint(1) + 1; int exp_str_sz = 2 * (MP_PRINT_MAX_DEPTH + 1) + 3 + 1; char *mp_buff = malloc(mp_buff_sz); char *exp_str = malloc(exp_str_sz); -- 2.17.1 From avtikhon at tarantool.org Thu Aug 20 11:05:57 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 20 Aug 2020 11:05:57 +0300 Subject: [Tarantool-patches] [PATCH v1] test: remove asan suppression for unit/msgpack Message-ID: <87c28236a265e7ab43072bc89b0a3a11339f7631.1597910696.git.avtikhon@tarantool.org> ASAN should the issue in msgpuck repository in file test/msgpack.c which was the cause of the fail in unit/msgpack test. The issue was fixed in msgpuck repository and ASAN suppression was removed for it. Also removed skip condition file, which blocked the test when it failed. Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/asan-restore Issue: https://github.com/tarantool/tarantool/issues/4360 asan/asan.supp | 4 ---- src/lib/msgpuck | 2 +- test/unit/msgpack.skipcond | 7 ------- 3 files changed, 1 insertion(+), 12 deletions(-) delete mode 100644 test/unit/msgpack.skipcond diff --git a/asan/asan.supp b/asan/asan.supp index 24cb0845a..95a662461 100644 --- a/asan/asan.supp +++ b/asan/asan.supp @@ -4,7 +4,3 @@ # File format: #fun:* #src:* - -# !test: unit/msgpack.test -# source: src/lib/msgpuck/test/msgpuck.c -fun:test_mp_print diff --git a/src/lib/msgpuck b/src/lib/msgpuck index 3cc63d3d1..a843861f7 160000 --- a/src/lib/msgpuck +++ b/src/lib/msgpuck @@ -1 +1 @@ -Subproject commit 3cc63d3d1d17c022894d8fb095c5d3dc3ae270aa +Subproject commit a843861f7124f257bbd505fac8dbfb5708ecc855 diff --git a/test/unit/msgpack.skipcond b/test/unit/msgpack.skipcond deleted file mode 100644 index e46fd1088..000000000 --- a/test/unit/msgpack.skipcond +++ /dev/null @@ -1,7 +0,0 @@ -import os - -# Disabled at ASAN build due to issue #4360. -if os.getenv("ASAN") == 'ON': - self.skip = 1 - -# vim: set ft=python: -- 2.17.1 From kyukhin at tarantool.org Thu Aug 20 14:33:42 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Thu, 20 Aug 2020 14:33:42 +0300 Subject: [Tarantool-patches] [PATCH v1] asan/lsan: cleanup suppression lists In-Reply-To: <076c9037cb2e19264feff4687112622a58af5435.1597896446.git.avtikhon@tarantool.org> References: <076c9037cb2e19264feff4687112622a58af5435.1597896446.git.avtikhon@tarantool.org> Message-ID: <20200820113342.4zpujqqjha226eae@tarantool.org> Hello, On 20 ??? 07:08, Alexander V. Tikhonov wrote: > Removed asan/lsan suppresions for issues that were not reproduced. > Removed skip condition files for tests that passed testing. > > Part of #4360 > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/asan-restore > Issue: https://github.com/tarantool/tarantool/issues/4360 I've checked your patch into 1.10, 2.4, 2.5 and master. -- Regards, Kirill Yukhin From avtikhon at tarantool.org Thu Aug 20 17:25:38 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 20 Aug 2020 17:25:38 +0300 Subject: [Tarantool-patches] [PATCH v6] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <20200819223552.nplchepqb67tojxr@tkn_work_nb> References: <344b8b406afa1464aae59f9be231ff78f053b996.1597817683.git.avtikhon@tarantool.org> <20200819223552.nplchepqb67tojxr@tkn_work_nb> Message-ID: <20200820142538.GA26887@hpalx> Hi Alexander, thanks a lot for your help. As we discussed the commit message I've made all the changes in it and commited to branch. On Thu, Aug 20, 2020 at 01:35:52AM +0300, Alexander Turenko wrote: > This description becomes much better! > > On Wed, Aug 19, 2020 at 09:17:39AM +0300, Alexander V. Tikhonov wrote: > > During implementation of openSUSE build with testing got failed test > > box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and > > vinyl_dir existed and also errno was set to ENOENT, box configuration > > succeeded, but it shouldn't. Reason of this wrong behaviour was that > > not all of the failure paths in xdir_scan() set errno, but the caller > > assumed it. > > > > Debugging the issue found that after xdir_scan() there was incorrect > > check for errno when it returned negative values. xdir_scan() is not > > system call and negative return value from it doesn't mean that errno > > whould be set too. Found that in situations when errno was left from > > previous commands before xdir_scan() and xdir_scan() returned negative > > value by itself than the check was wrong. The previous logic of the > > check was to catch the error ENOENT from inside the xdir_scan() > > function to handle the situation when vinyl_dir was not exist. In this > > way errno should be reseted before xdir_scan() call to give the ability > > for xdir_scan() to use return value without errno set and correctly > > handle errno from inside the xdir_scan(). > > 'from inside' can be interpreted as 'to catch from inside', not as > 'ENOENT from inside'. The same for 'handle errno for inside': 'to handle > from inside' or 'errno from inside'? > > 'to use return value without errno set' is unclear for me. Are not it is > intended to give the same idea as the previous 'errno should be reset > before xdir_scan()'? If so, it look redundant for me. > > Typo: whould. > > Typo: reseted -> reset (irregular verb). > > > > > After discussions found that there was alternative better solution to > > fix it. As mentioned above xdir_scan() function is not system call and > > can be changed inside it in any possible way. So check outside of this > > function on errno could be broken, because of the xdir_scan() changes. > > To avoid of it we must avoid of errno checks outside of the function. > > Better solution was to use the flag in xdir_scan(), to check if the > > directory should exist. So errno check was removed and instead of it > > the check for vinyl_dir existence using flag added. > > > > Closes #4594 > > Needed for #4562 > > > > Co-authored-by: Alexander Turenko > > --- > > > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > > Issue: https://github.com/tarantool/tarantool/issues/4594 > > Issue: https://github.com/tarantool/tarantool/issues/4562 From alexander.turenko at tarantool.org Thu Aug 20 22:48:06 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Thu, 20 Aug 2020 22:48:06 +0300 Subject: [Tarantool-patches] [PATCH v6] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <20200820142538.GA26887@hpalx> References: <344b8b406afa1464aae59f9be231ff78f053b996.1597817683.git.avtikhon@tarantool.org> <20200819223552.nplchepqb67tojxr@tkn_work_nb> <20200820142538.GA26887@hpalx> Message-ID: <20200820194806.nmuhufhgwokjayrz@tkn_work_nb> Please, resend the patchset to Nikita Pettik and Aleksandr Lyapunov (because it is related to vinyl). Or extract this particular path to its own branch and send it as a singleton patch. WBR, Alexander Turenko. On Thu, Aug 20, 2020 at 05:25:38PM +0300, Alexander V. Tikhonov wrote: > Hi Alexander, thanks a lot for your help. As we discussed the commit > message I've made all the changes in it and commited to branch. I think it is good enough. LGTM. Cited below for the history. > commit e18b7e8f56787a9099429ff9e6fc6c1185d74723 > Author: Alexander V. Tikhonov > Date: Fri Aug 14 11:18:25 2020 +0300 > > vinyl: fix check vinyl_dir existence at bootstrap > > During implementation of openSUSE build with testing got failed test > box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and > vinyl_dir existed and also errno was set to ENOENT, box configuration > succeeded, but it shouldn't. Reason of this wrong behavior was that > not all of the failure paths in xdir_scan() set errno, but the caller > assumed it. > > Debugging the issue found that after xdir_scan() there was incorrect > check for errno when it returned negative values. xdir_scan() is not > system call and negative return value from it doesn't mean that errno > would be set too. Found that in situations when errno was left from > previous commands before xdir_scan() and xdir_scan() returned negative > value by itself it produced the wrong check. > > The previous failed logic of the check was to catch the error ENOENT > which set in the xdir_scan() function to handle the situation when > vinyl_dir was not exist. It failed, because checking ENOENT outside > the xdir_scan() function, we had to be sure that ENOENT had come from > xdir_scan() function call indeed and not from any other functions > before. To be sure in it possible fix could be reset errno before > xdir_scan() call, because errno could be passed from any other function > before call to xdir_scan(). > > As mentioned above xdir_scan() function is not system call and can be > changed in any possible way and it can return any result value without > need to setup errno. So check outside of this function on errno could > be broken. > > To avoid of it we must avoid of errno checks outside of the function. > Better solution is to use the flag in xdir_scan(), to check if the > directory should exist. So errno check was removed and instead of it > the check for vinyl_dir existence using flag added. > > Closes #4594 > Needed for #4562 > > Co-authored-by: Alexander Turenko From avtikhon at tarantool.org Fri Aug 21 07:56:56 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Fri, 21 Aug 2020 07:56:56 +0300 Subject: [Tarantool-patches] [PATCH v1] vinyl: fix check vinyl_dir existence at bootstrap Message-ID: During implementation of openSUSE build with testing got failed test box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and vinyl_dir existed and also errno was set to ENOENT, box configuration succeeded, but it shouldn't. Reason of this wrong behavior was that not all of the failure paths in xdir_scan() set errno, but the caller assumed it. Debugging the issue found that after xdir_scan() there was incorrect check for errno when it returned negative values. xdir_scan() is not system call and negative return value from it doesn't mean that errno would be set too. Found that in situations when errno was left from previous commands before xdir_scan() and xdir_scan() returned negative value by itself it produced the wrong check. The previous failed logic of the check was to catch the error ENOENT which set in the xdir_scan() function to handle the situation when vinyl_dir was not exist. It failed, because checking ENOENT outside the xdir_scan() function, we had to be sure that ENOENT had come from xdir_scan() function call indeed and not from any other functions before. To be sure in it possible fix could be reset errno before xdir_scan() call, because errno could be passed from any other function before call to xdir_scan(). As mentioned above xdir_scan() function is not system call and can be changed in any possible way and it can return any result value without need to setup errno. So check outside of this function on errno could be broken. To avoid of it we must avoid of errno checks outside of the function. Better solution is to use the flag in xdir_scan(), to check if the directory should exist. So errno check was removed and instead of it the check for vinyl_dir existence using flag added. Closes #4594 Needed for #4562 Co-authored-by: Alexander Turenko --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-vinyl-fix Issue: https://github.com/tarantool/tarantool/issues/4594 Issue: https://github.com/tarantool/tarantool/issues/4562 Review: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019130.html src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 +- src/box/vy_log.c | 4 +- src/box/wal.c | 2 +- src/box/xlog.c | 4 +- src/box/xlog.h | 6 +-- .../gh-4562-errno-at-xdir_scan.test.lua | 54 +++++++++++++++++++ 7 files changed, 66 insertions(+), 10 deletions(-) create mode 100755 test/box-tap/gh-4562-errno-at-xdir_scan.test.lua diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index 311985c72..da3c50e87 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index d8c92aa36..2b894d680 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -559,7 +559,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..74f761994 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..3400eb75f 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -187,7 +187,7 @@ xdir_destroy(struct xdir *dir); * snapshot or scan through all logs. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +821,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua new file mode 100755 index 000000000..e6dd68d54 --- /dev/null +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua @@ -0,0 +1,54 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local test = tap.test('cfg') +local fio = require('fio') +test:plan(1) + +local tarantool_bin = arg[-1] +local PANIC = 256 + +local function run_script(code) + local dir = fio.tempdir() + local script_path = fio.pathjoin(dir, 'script.lua') + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, + tonumber('0777', 8)) + script:write(code) + script:write("\nos.exit(0)") + script:close() + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] + local res = os.execute(string.format(cmd, dir, tarantool_bin)) + fio.rmtree(dir) + return res +end + +-- +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and +-- errno is set to ENOENT, box configuration succeeds, however it +-- should not +-- + +-- create vinyl_dir as temporary path +local vinyl_dir = fio.tempdir() + +-- fill vinyl_dir +run_script(string.format([[ +box.cfg{vinyl_dir = '%s'} +s = box.schema.space.create('test', {engine = 'vinyl'}) +s:create_index('pk') +os.exit(0) +]], vinyl_dir)) + +-- verify the case described above +local code = string.format([[ +local errno = require('errno') +errno(errno.ENOENT) +box.cfg{vinyl_dir = '%s'} +os.exit(0) +]], vinyl_dir) +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") + +-- remove vinyl_dir +fio.rmtree(vinyl_dir) + +os.exit(test:check() and 0 or 1) -- 2.17.1 From sergepetrenko at tarantool.org Fri Aug 21 10:48:14 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 10:48:14 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_1/7=5D_journal=3A_bind_?= =?utf-8?q?asynchronous_write_completion_to_an_entry?= In-Reply-To: <20200819213442.1099018-2-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-2-gorcunov@gmail.com> Message-ID: <1597996094.668971732@f326.i.mail.ru> Hi! Thanks for the patch! LGTM. ? >???????, 20 ??????? 2020, 0:35 +03:00 ?? Cyrill Gorcunov : >? >In commit 77ba0e3504464131fe81c672d508d0275be2173a we've redesigned >wal journal operations such that asynchronous write completion >is a single instance per journal. > >It turned out that such simplification is too tight and doesn't >allow us to pass entries into the journal with custom completions. > >Thus lets allow back such ability. We will need it to be able >to write "confirm" records into wal directly without touching >transactions code at all. > >Part-of #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/box.cc | 15 ++++++++------- >?src/box/journal.c | 2 ++ >?src/box/journal.h | 20 +++++++++++--------- >?src/box/txn.c | 2 +- >?src/box/vy_log.c | 2 +- >?src/box/wal.c | 19 ++++++++----------- >?src/box/wal.h | 4 ++-- >?7 files changed, 33 insertions(+), 31 deletions(-) > >diff --git a/src/box/box.cc b/src/box/box.cc >index 8e811e9c1..faffd5769 100644 >--- a/src/box/box.cc >+++ b/src/box/box.cc >@@ -348,7 +348,7 @@ recovery_journal_write(struct journal *base, >? * Since there're no actual writes, fire a >? * journal_async_complete callback right away. >? */ >- journal_async_complete(base, entry); >+ journal_async_complete(entry); >? return 0; >?} >? >@@ -357,7 +357,7 @@ recovery_journal_create(struct vclock *v) >?{ >? static struct recovery_journal journal; >? journal_create(&journal.base, recovery_journal_write, >- txn_complete_async, recovery_journal_write); >+ recovery_journal_write); >? journal.vclock = v; >? journal_set(&journal.base); >?} >@@ -2182,8 +2182,10 @@ engine_init() >?static int >?bootstrap_journal_write(struct journal *base, struct journal_entry *entry) >?{ >+ (void)base; >+ >? entry->res = 0; >- journal_async_complete(base, entry); >+ journal_async_complete(entry); >? return 0; >?} >? >@@ -2569,8 +2571,8 @@ box_cfg_xc(void) >? >? int64_t wal_max_size = box_check_wal_max_size(cfg_geti64("wal_max_size")); >? enum wal_mode wal_mode = box_check_wal_mode(cfg_gets("wal_mode")); >- if (wal_init(wal_mode, txn_complete_async, cfg_gets("wal_dir"), >- wal_max_size, &INSTANCE_UUID, on_wal_garbage_collection, >+ if (wal_init(wal_mode, cfg_gets("wal_dir"), wal_max_size, >+ &INSTANCE_UUID, on_wal_garbage_collection, >? on_wal_checkpoint_threshold) != 0) { >? diag_raise(); >? } >@@ -2617,8 +2619,7 @@ box_cfg_xc(void) >? } >? >? struct journal bootstrap_journal; >- journal_create(&bootstrap_journal, NULL, txn_complete_async, >- bootstrap_journal_write); >+ journal_create(&bootstrap_journal, NULL, bootstrap_journal_write); >? journal_set(&bootstrap_journal); >? auto bootstrap_journal_guard = make_scoped_guard([] { >? journal_set(NULL); >diff --git a/src/box/journal.c b/src/box/journal.c >index f1e89aaa2..48af9157b 100644 >--- a/src/box/journal.c >+++ b/src/box/journal.c >@@ -36,6 +36,7 @@ struct journal *current_journal = NULL; >? >?struct journal_entry * >?journal_entry_new(size_t n_rows, struct region *region, >+ journal_write_async_f write_async_cb, >? void *complete_data) >?{ >? struct journal_entry *entry; >@@ -50,6 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, >? return NULL; >? } >? >+ entry->write_async_cb = write_async_cb; >? entry->complete_data = complete_data; >? entry->approx_len = 0; >? entry->n_rows = n_rows; >diff --git a/src/box/journal.h b/src/box/journal.h >index 1a10e66c3..4b019fecf 100644 >--- a/src/box/journal.h >+++ b/src/box/journal.h >@@ -42,6 +42,8 @@ extern "C" { >?struct xrow_header; >?struct journal_entry; >? >+typedef void (*journal_write_async_f)(struct journal_entry *entry); >+ >?/** >??* An entry for an abstract journal. >??* Simply put, a write ahead log request. >@@ -61,6 +63,10 @@ struct journal_entry { >? * A journal entry completion callback argument. >? */ >? void *complete_data; >+ /** >+ * Asynchronous write completion function. >+ */ >+ journal_write_async_f write_async_cb; >? /** >? * Approximate size of this request when encoded. >? */ >@@ -84,6 +90,7 @@ struct region; >??*/ >?struct journal_entry * >?journal_entry_new(size_t n_rows, struct region *region, >+ journal_write_async_f write_async_cb, >? void *complete_data); >? >?/** >@@ -96,22 +103,19 @@ struct journal { >? int (*write_async)(struct journal *journal, >? struct journal_entry *entry); >? >- /** Asynchronous write completion */ >- void (*write_async_cb)(struct journal_entry *entry); >- >? /** Synchronous write */ >? int (*write)(struct journal *journal, >? struct journal_entry *entry); >?}; >? >?/** >- * Finalize a single entry. >+ * Complete asynchronous write. >??*/ >?static inline void >-journal_async_complete(struct journal *journal, struct journal_entry *entry) >+journal_async_complete(struct journal_entry *entry) >?{ >- assert(journal->write_async_cb != NULL); >- journal->write_async_cb(entry); >+ assert(entry->write_async_cb != NULL); >+ entry->write_async_cb(entry); >?} >? >?/** >@@ -173,12 +177,10 @@ static inline void >?journal_create(struct journal *journal, >? int (*write_async)(struct journal *journal, >? struct journal_entry *entry), >- void (*write_async_cb)(struct journal_entry *entry), >? int (*write)(struct journal *journal, >? struct journal_entry *entry)) >?{ >? journal->write_async = write_async; >- journal->write_async_cb = write_async_cb; >? journal->write = write; >?} >? >diff --git a/src/box/txn.c b/src/box/txn.c >index 9c21258c5..cc1f496c5 100644 >--- a/src/box/txn.c >+++ b/src/box/txn.c >@@ -551,7 +551,7 @@ txn_journal_entry_new(struct txn *txn) >? >? /* Save space for an additional NOP row just in case. */ >? req = journal_entry_new(txn->n_new_rows + txn->n_applier_rows + 1, >- &txn->region, txn); >+ &txn->region, txn_complete_async, txn); >? if (req == NULL) >? return NULL; >? >diff --git a/src/box/vy_log.c b/src/box/vy_log.c >index 311985c72..de4c5205c 100644 >--- a/src/box/vy_log.c >+++ b/src/box/vy_log.c >@@ -818,7 +818,7 @@ vy_log_tx_flush(struct vy_log_tx *tx) >? size_t used = region_used(&fiber()->gc); >? >? struct journal_entry *entry; >- entry = journal_entry_new(tx_size, &fiber()->gc, NULL); >+ entry = journal_entry_new(tx_size, &fiber()->gc, NULL, NULL); >? if (entry == NULL) >? goto err; >? >diff --git a/src/box/wal.c b/src/box/wal.c >index d8c92aa36..045006b60 100644 >--- a/src/box/wal.c >+++ b/src/box/wal.c >@@ -266,10 +266,9 @@ xlog_write_entry(struct xlog *l, struct journal_entry *entry) >?static void >?tx_schedule_queue(struct stailq *queue) >?{ >- struct wal_writer *writer = &wal_writer_singleton; >? struct journal_entry *req, *tmp; >? stailq_foreach_entry_safe(req, tmp, queue, fifo) >- journal_async_complete(&writer->base, req); >+ journal_async_complete(req); >?} >? >?/** >@@ -403,9 +402,8 @@ tx_notify_checkpoint(struct cmsg *msg) >??*/ >?static void >?wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, >- void (*wall_async_cb)(struct journal_entry *entry), >- const char *wal_dirname, >- int64_t wal_max_size, const struct tt_uuid *instance_uuid, >+ const char *wal_dirname, int64_t wal_max_size, >+ const struct tt_uuid *instance_uuid, >? wal_on_garbage_collection_f on_garbage_collection, >? wal_on_checkpoint_threshold_f on_checkpoint_threshold) >?{ >@@ -415,7 +413,6 @@ wal_writer_create(struct wal_writer *writer, enum wal_mode wal_mode, >? journal_create(&writer->base, >? wal_mode == WAL_NONE ? >? wal_write_none_async : wal_write_async, >- wall_async_cb, >? wal_mode == WAL_NONE ? >? wal_write_none : wal_write); >? >@@ -525,15 +522,15 @@ wal_open(struct wal_writer *writer) >?} >? >?int >-wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), >- const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, >+wal_init(enum wal_mode wal_mode, const char *wal_dirname, >+ int64_t wal_max_size, const struct tt_uuid *instance_uuid, >? wal_on_garbage_collection_f on_garbage_collection, >? wal_on_checkpoint_threshold_f on_checkpoint_threshold) >?{ >? /* Initialize the state. */ >? struct wal_writer *writer = &wal_writer_singleton; >- wal_writer_create(writer, wal_mode, wall_async_cb, wal_dirname, >- wal_max_size, instance_uuid, on_garbage_collection, >+ wal_writer_create(writer, wal_mode, wal_dirname, wal_max_size, >+ instance_uuid, on_garbage_collection, >? on_checkpoint_threshold); >? >? /* Start WAL thread. */ >@@ -1314,7 +1311,7 @@ wal_write_none_async(struct journal *journal, >? vclock_merge(&writer->vclock, &vclock_diff); >? vclock_copy(&replicaset.vclock, &writer->vclock); >? entry->res = vclock_sum(&writer->vclock); >- journal_async_complete(journal, entry); >+ journal_async_complete(entry); >? return 0; >?} >? >diff --git a/src/box/wal.h b/src/box/wal.h >index f348dc636..9d0cada46 100644 >--- a/src/box/wal.h >+++ b/src/box/wal.h >@@ -81,8 +81,8 @@ typedef void (*wal_on_checkpoint_threshold_f)(void); >??* Start WAL thread and initialize WAL writer. >??*/ >?int >-wal_init(enum wal_mode wal_mode, void (*wall_async_cb)(struct journal_entry *entry), >- const char *wal_dirname, int64_t wal_max_size, const struct tt_uuid *instance_uuid, >+wal_init(enum wal_mode wal_mode, const char *wal_dirname, >+ int64_t wal_max_size, const struct tt_uuid *instance_uuid, >? wal_on_garbage_collection_f on_garbage_collection, >? wal_on_checkpoint_threshold_f on_checkpoint_threshold); >? >-- >2.26.2 ? ? -- Serge?Petrenko ? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 21 10:51:06 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 10:51:06 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_2/7=5D_journal=3A_add_j?= =?utf-8?q?ournal=5Fentry=5Fcreate_helper?= In-Reply-To: <20200819213442.1099018-3-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-3-gorcunov@gmail.com> Message-ID: <1597996266.340182040@f326.i.mail.ru> Thanks for the patch! LGTM. ? >???????, 20 ??????? 2020, 0:35 +03:00 ?? Cyrill Gorcunov : >? >To create raw journal entries. We will use it >to write confirm/rollback entries. > >Part-of #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/journal.c | 8 ++------ >?src/box/journal.h | 16 ++++++++++++++++ >?2 files changed, 18 insertions(+), 6 deletions(-) > >diff --git a/src/box/journal.c b/src/box/journal.c >index 48af9157b..cb320b557 100644 >--- a/src/box/journal.c >+++ b/src/box/journal.c >@@ -51,11 +51,7 @@ journal_entry_new(size_t n_rows, struct region *region, >? return NULL; >? } >? >- entry->write_async_cb = write_async_cb; >- entry->complete_data = complete_data; >- entry->approx_len = 0; >- entry->n_rows = n_rows; >- entry->res = -1; >- >+ journal_entry_create(entry, n_rows, 0, write_async_cb, >+ complete_data); >? return entry; >?} >diff --git a/src/box/journal.h b/src/box/journal.h >index 4b019fecf..5d8d5a726 100644 >--- a/src/box/journal.h >+++ b/src/box/journal.h >@@ -83,6 +83,22 @@ struct journal_entry { >? >?struct region; >? >+/** >+ * Initialize a new journal entry. >+ */ >+static inline void >+journal_entry_create(struct journal_entry *entry, size_t n_rows, >+ size_t approx_len, >+ journal_write_async_f write_async_cb, >+ void *complete_data) >+{ >+ entry->write_async_cb = write_async_cb; >+ entry->complete_data = complete_data; >+ entry->approx_len = approx_len; >+ entry->n_rows = n_rows; >+ entry->res = -1; >+} >+ >?/** >??* Create a new journal entry. >??* >-- >2.26.2 ? -- Serge?Petrenko ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 21 11:15:44 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 11:15:44 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_3/7=5D_qsync=3A_provide?= =?utf-8?q?_a_binary_form_of_syncro_entries?= In-Reply-To: <20200819213442.1099018-4-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-4-gorcunov@gmail.com> Message-ID: <1597997744.808663195@f348.i.mail.ru> LGTM. ? ? >???????, 20 ??????? 2020, 0:35 +03:00 ?? Cyrill Gorcunov : >? >These msgpack entries will be needed to write them >down to a journal without involving txn engine. Same >time we would like to be able to allocate them on stack, >for this sake the binary form is predefined. > >Part-of #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/txn_limbo.c | 9 +++++++-- >?src/box/xrow.c | 41 ++++++++++++++++++----------------------- >?src/box/xrow.h | 20 +++++++++++++++----- >?3 files changed, 40 insertions(+), 30 deletions(-) > >diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c >index c6a4e5efc..e458dad75 100644 >--- a/src/box/txn_limbo.c >+++ b/src/box/txn_limbo.c >@@ -283,6 +283,11 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >? .lsn = lsn, >? }; >? >+ /* >+ * This is a synchronous commit so we can >+ * use body and row allocated on a stack. >+ */ >+ struct synchro_body_bin body; >? struct xrow_header row; >? struct request request = { >? .header = &row, >@@ -292,8 +297,8 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >? if (txn == NULL) >? goto rollback; >? >- if (xrow_encode_synchro(&row, &txn->region, &req) != 0) >- goto rollback; >+ xrow_encode_synchro(&row, &body, &req); >+ >? /* >? * This is not really a transaction. It just uses txn API >? * to put the data into WAL. And obviously it should not >diff --git a/src/box/xrow.c b/src/box/xrow.c >index bf174c701..9c6fb4fc1 100644 >--- a/src/box/xrow.c >+++ b/src/box/xrow.c >@@ -893,35 +893,30 @@ xrow_encode_dml(const struct request *request, struct region *region, >? return iovcnt; >?} >? >-int >-xrow_encode_synchro(struct xrow_header *row, struct region *region, >+void >+xrow_encode_synchro(struct xrow_header *row, >+ struct synchro_body_bin *body, >? const struct synchro_request *req) >?{ >- size_t len = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_REPLICA_ID) + >- mp_sizeof_uint(req->replica_id) + >- mp_sizeof_uint(IPROTO_LSN) + mp_sizeof_uint(req->lsn); >- char *buf = (char *)region_alloc(region, len); >- if (buf == NULL) { >- diag_set(OutOfMemory, len, "region_alloc", "buf"); >- return -1; >- } >- char *pos = buf; >- >- pos = mp_encode_map(pos, 2); >- pos = mp_encode_uint(pos, IPROTO_REPLICA_ID); >- pos = mp_encode_uint(pos, req->replica_id); >- pos = mp_encode_uint(pos, IPROTO_LSN); >- pos = mp_encode_uint(pos, req->lsn); >+ /* >+ * A map with two elements. We don't compress >+ * numbers to have this structure constant in size, >+ * which allows us to preallocate it on stack. >+ */ >+ body->m_body = 0x80 | 2; >+ body->k_replica_id = IPROTO_REPLICA_ID; >+ body->m_replica_id = 0xce; >+ body->v_replica_id = mp_bswap_u32(req->replica_id); >+ body->k_lsn = IPROTO_LSN; >+ body->m_lsn = 0xcf; >+ body->v_lsn = mp_bswap_u64(req->lsn); >? >? memset(row, 0, sizeof(*row)); >? >- row->body[0].iov_base = buf; >- row->body[0].iov_len = len; >- row->bodycnt = 1; >- >? row->type = req->type; >- >- return 0; >+ row->body[0].iov_base = (void *)body; >+ row->body[0].iov_len = sizeof(*body); >+ row->bodycnt = 1; >?} >? >?int >diff --git a/src/box/xrow.h b/src/box/xrow.h >index 02dca74e5..20e82034d 100644 >--- a/src/box/xrow.h >+++ b/src/box/xrow.h >@@ -240,16 +240,26 @@ struct synchro_request { >? int64_t lsn; >?}; >? >+/** Synchro request xrow's body in MsgPack format. */ >+struct PACKED synchro_body_bin { >+ uint8_t m_body; >+ uint8_t k_replica_id; >+ uint8_t m_replica_id; >+ uint32_t v_replica_id; >+ uint8_t k_lsn; >+ uint8_t m_lsn; >+ uint64_t v_lsn; >+}; >+ >?/** >??* Encode synchronous replication request. >??* @param row xrow header. >- * @param region Region to use to encode the confirmation body. >+ * @param body Desination to use to encode the confirmation body. >??* @param req Request parameters. >- * @retval -1 on error. >- * @retval 0 success. >??*/ >-int >-xrow_encode_synchro(struct xrow_header *row, struct region *region, >+void >+xrow_encode_synchro(struct xrow_header *row, >+ struct synchro_body_bin *body, >? const struct synchro_request *req); >? >?/** >-- >2.26.2 >? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 21 11:36:57 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 11:36:57 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_4/7=5D_qsync=3A_direct_?= =?utf-8?q?write_of_CONFIRM/ROLLBACK_into_a_journal?= In-Reply-To: <20200819213442.1099018-5-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-5-gorcunov@gmail.com> Message-ID: <1597999017.380175980@f519.i.mail.ru> Thanks for the patch! LGTM. ? ? >???????, 20 ??????? 2020, 0:35 +03:00 ?? Cyrill Gorcunov : >? >When we need to write CONFIRM or ROLLBACK message (which is >a binary record in msgpack format) into a journal we use txn code >to allocate a new transaction, encode there a message and pass it >to walk the long txn path before it hit the journal. This is not >only resource wasting but also somehow strange from architectural >point of view. > >Instead lets encode a record on the stack and write it to the journal >directly. > >Part-of #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/txn_limbo.c | 66 +++++++++++++++++++++++---------------------- >?1 file changed, 34 insertions(+), 32 deletions(-) > >diff --git a/src/box/txn_limbo.c b/src/box/txn_limbo.c >index e458dad75..4b90d7fa5 100644 >--- a/src/box/txn_limbo.c >+++ b/src/box/txn_limbo.c >@@ -32,6 +32,7 @@ >?#include "txn_limbo.h" >?#include "replication.h" >?#include "iproto_constants.h" >+#include "journal.h" >? >?struct txn_limbo txn_limbo; >? >@@ -272,6 +273,17 @@ txn_limbo_wait_complete(struct txn_limbo *limbo, struct txn_limbo_entry *entry) >? return 0; >?} >? >+/** >+ * A callback for synchronous write: txn_limbo_write_synchro fiber >+ * waiting to proceed once a record is written to WAL. >+ */ >+static void >+txn_limbo_write_cb(struct journal_entry *entry) >+{ >+ assert(entry->complete_data != NULL); >+ fiber_wakeup(entry->complete_data); >+} >+ >?static void >?txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >?{ >@@ -285,46 +297,36 @@ txn_limbo_write_synchro(struct txn_limbo *limbo, uint32_t type, int64_t lsn) >? >? /* >? * This is a synchronous commit so we can >- * use body and row allocated on a stack. >+ * allocate everything on a stack. >? */ >? struct synchro_body_bin body; >? struct xrow_header row; >- struct request request = { >- .header = &row, >- }; >+ char buf[sizeof(struct journal_entry) + >+ sizeof(struct xrow_header *)]; >? >- struct txn *txn = txn_begin(); >- if (txn == NULL) >- goto rollback; >+ struct journal_entry *entry = (struct journal_entry *)buf; >+ entry->rows[0] = &row; >? >? xrow_encode_synchro(&row, &body, &req); >? >- /* >- * This is not really a transaction. It just uses txn API >- * to put the data into WAL. And obviously it should not >- * go to the limbo and block on the very same sync >- * transaction which it tries to confirm now. >- */ >- txn_set_flag(txn, TXN_FORCE_ASYNC); >- >- if (txn_begin_stmt(txn, NULL) != 0) >- goto rollback; >- if (txn_commit_stmt(txn, &request) != 0) >- goto rollback; >- if (txn_commit(txn) != 0) >- goto rollback; >- return; >+ journal_entry_create(entry, 1, xrow_approx_len(&row), >+ txn_limbo_write_cb, fiber()); >? >-rollback: >- /* >- * XXX: the stub is supposed to be removed once it is defined what to do >- * when a synchro request WAL write fails. One of the possible >- * solutions: log the error, keep the limbo queue as is and probably put >- * in rollback mode. Then provide a hook to call manually when WAL >- * problems are fixed. Or retry automatically with some period. >- */ >- panic("Could not write a synchro request to WAL: lsn = %lld, type = " >- "%s\n", lsn, iproto_type_name(type)); >+ if (journal_write(entry) != 0 || entry->res < 0) { >+ diag_set(ClientError, ER_WAL_IO); >+ diag_log(); >+ /* >+ * XXX: the stub is supposed to be removed once it is defined >+ * what to do when a synchro request WAL write fails. One of >+ * the possible solutions: log the error, keep the limbo >+ * queue as is and probably put in rollback mode. Then >+ * provide a hook to call manually when WAL problems are fixed. >+ * Or retry automatically with some period. >+ */ >+ panic("Could not write a synchro request to WAL: " >+ "lsn = %lld, type = %s\n", lsn, >+ type == IPROTO_CONFIRM ? "CONFIRM" : "ROLLBACK"); >+ } >?} >? >?/** >-- >2.26.2 ? ? -- Serge?Petrenko ? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From imeevma at tarantool.org Fri Aug 21 11:40:48 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 11:40:48 +0300 Subject: [Tarantool-patches] [PATCH v1 0/2] sql: remove implicit cast from operations Message-ID: This patch-set removes implicit string-to-number conversion from arithmetic and bitwise operations. https://github.com/tarantool/tarantool/issues/3809 https://github.com/tarantool/tarantool/tree/imeevma/gh-3809-follow-up @ChangeLog - Strings are no longer implicitly converted to numbers in arithmetic and bitwise operations. Mergen Imeev (2): sql: remove implicit cast in arithmetic operations sql: remove implicit cast in bitwise operations src/box/sql/vdbe.c | 130 +++++++++++++----------- test/sql-tap/misc1.test.lua | 6 +- test/sql-tap/misc3.test.lua | 42 +------- test/sql-tap/tkt-a8a0d2996a.test.lua | 146 --------------------------- test/sql/types.result | 110 +++++++++++++++++++- test/sql/types.test.lua | 29 ++++++ 6 files changed, 207 insertions(+), 256 deletions(-) delete mode 100755 test/sql-tap/tkt-a8a0d2996a.test.lua -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 11:40:50 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 11:40:50 +0300 Subject: [Tarantool-patches] [PATCH v1 1/2] sql: remove implicit cast in arithmetic operations In-Reply-To: References: Message-ID: This patch removes the implicit conversion from STRING to NUMBER from arithmetic operations. However, INTEGER can still be implicitly converted to DOUBLE if the second operand is of type DOUBLE. Follow-up #3809 --- src/box/sql/vdbe.c | 71 ++++--------- test/sql-tap/misc1.test.lua | 6 +- test/sql-tap/misc3.test.lua | 42 +------- test/sql-tap/tkt-a8a0d2996a.test.lua | 146 --------------------------- test/sql/types.result | 61 ++++++++++- test/sql/types.test.lua | 15 +++ 6 files changed, 97 insertions(+), 244 deletions(-) delete mode 100755 test/sql-tap/tkt-a8a0d2996a.test.lua diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 14ddb5160..c0143a6b1 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -522,41 +522,6 @@ mem_convert_to_numeric(struct Mem *mem, enum field_type type) return mem_convert_to_integer(mem); } -/* - * pMem currently only holds a string type (or maybe a BLOB that we can - * interpret as a string if we want to). Compute its corresponding - * numeric type, if has one. Set the pMem->u.r and pMem->u.i fields - * accordingly. - */ -static u16 SQL_NOINLINE computeNumericType(Mem *pMem) -{ - assert((pMem->flags & (MEM_Int | MEM_UInt | MEM_Real)) == 0); - assert((pMem->flags & (MEM_Str|MEM_Blob))!=0); - if (sqlAtoF(pMem->z, &pMem->u.r, pMem->n)==0) - return 0; - bool is_neg; - if (sql_atoi64(pMem->z, (int64_t *) &pMem->u.i, &is_neg, pMem->n) == 0) - return is_neg ? MEM_Int : MEM_UInt; - return MEM_Real; -} - -/* - * Return the numeric type for pMem, either MEM_Int or MEM_Real or both or - * none. - * - * Unlike mem_apply_numeric_type(), this routine does not modify pMem->flags. - * But it does set pMem->u.r and pMem->u.i appropriately. - */ -static u16 numericType(Mem *pMem) -{ - if ((pMem->flags & (MEM_Int | MEM_UInt | MEM_Real)) != 0) - return pMem->flags & (MEM_Int | MEM_UInt | MEM_Real); - if (pMem->flags & (MEM_Str|MEM_Blob)) { - return computeNumericType(pMem); - } - return 0; -} - #ifdef SQL_DEBUG /* * Write a nice string representation of the contents of cell pMem @@ -1681,27 +1646,24 @@ case OP_Subtract: /* same as TK_MINUS, in1, in2, out3 */ case OP_Multiply: /* same as TK_STAR, in1, in2, out3 */ case OP_Divide: /* same as TK_SLASH, in1, in2, out3 */ case OP_Remainder: { /* same as TK_REM, in1, in2, out3 */ - u32 flags; /* Combined MEM_* flags from both inputs */ - u16 type1; /* Numeric type of left operand */ - u16 type2; /* Numeric type of right operand */ i64 iA; /* Integer value of left operand */ i64 iB; /* Integer value of right operand */ double rA; /* Real value of left operand */ double rB; /* Real value of right operand */ pIn1 = &aMem[pOp->p1]; - type1 = numericType(pIn1); pIn2 = &aMem[pOp->p2]; - type2 = numericType(pIn2); + enum mp_type type1 = mem_mp_type(pIn1); + enum mp_type type2 = mem_mp_type(pIn2); pOut = vdbe_prepare_null_out(p, pOp->p3); - flags = pIn1->flags | pIn2->flags; - if ((flags & MEM_Null)!=0) goto arithmetic_result_is_null; - if ((type1 & (MEM_Int | MEM_UInt)) != 0 && - (type2 & (MEM_Int | MEM_UInt)) != 0) { + if (type1 == MP_NIL || type2 == MP_NIL) + goto arithmetic_result_is_null; + if ((type1 == MP_INT || type1 == MP_UINT) && + (type2 == MP_INT || type2 == MP_UINT)) { iA = pIn1->u.i; iB = pIn2->u.i; - bool is_lhs_neg = pIn1->flags & MEM_Int; - bool is_rhs_neg = pIn2->flags & MEM_Int; + bool is_lhs_neg = type1 == MP_INT; + bool is_rhs_neg = type2 == MP_INT; bool is_res_neg; switch( pOp->opcode) { case OP_Add: { @@ -1742,17 +1704,28 @@ case OP_Remainder: { /* same as TK_REM, in1, in2, out3 */ } mem_set_int(pOut, iB, is_res_neg); } else { - if (sqlVdbeRealValue(pIn1, &rA) != 0) { + if (!mp_type_is_numeric(type1)) { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn1), "numeric"); goto abort_due_to_error; } - if (sqlVdbeRealValue(pIn2, &rB) != 0) { + if (!mp_type_is_numeric(type2)) { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn2), "numeric"); goto abort_due_to_error; } - assert(((type1 | type2) & MEM_Real) != 0); + if (type1 == MP_DOUBLE) + rA = pIn1->u.r; + else if (type1 == MP_INT) + rA = (double) pIn1->u.i; + else + rA = (double) pIn1->u.u; + if (type2 == MP_DOUBLE) + rB = pIn2->u.r; + else if (type2 == MP_INT) + rB = (double) pIn2->u.i; + else + rB = (double) pIn2->u.u; switch( pOp->opcode) { case OP_Add: rB += rA; break; case OP_Subtract: rB -= rA; break; diff --git a/test/sql-tap/misc1.test.lua b/test/sql-tap/misc1.test.lua index c0136d04c..005dc0cb6 100755 --- a/test/sql-tap/misc1.test.lua +++ b/test/sql-tap/misc1.test.lua @@ -68,7 +68,7 @@ test:do_test( cmd = cmd .. ")" test:execsql(cmd) end - return test:execsql("SELECT x50 FROM manycol ORDER BY x80+0") + return test:execsql("SELECT x50 FROM manycol ORDER BY CAST(x80 AS NUMBER)+0") end, { -- "50", "150", "250", "350", "450", "550", "650", "750", "850", "950", "1050" @@ -531,7 +531,7 @@ test:do_test( "misc1-10.7", function() where = string.gsub(where, "x0=0", "x0=100") - return test:catchsql("UPDATE manycol SET x1=CAST(x1+1 AS STRING) "..where.."") + return test:catchsql("UPDATE manycol SET x1=CAST(CAST(x1 AS NUMBER)+1 AS STRING) "..where.."") end, { -- 0 @@ -553,7 +553,7 @@ test:do_execsql_test( -- } {0 {}} test:do_execsql_test( "misc1-10.9", - "UPDATE manycol SET x1=CAST(x1+1 AS STRING) "..where + "UPDATE manycol SET x1=CAST(CAST(x1 AS NUMBER)+1 AS STRING) "..where --"UPDATE manycol SET x1=x1+1 $::where AND rowid>0" , {}) diff --git a/test/sql-tap/misc3.test.lua b/test/sql-tap/misc3.test.lua index d0e45e872..7ee1a9e36 100755 --- a/test/sql-tap/misc3.test.lua +++ b/test/sql-tap/misc3.test.lua @@ -1,7 +1,7 @@ #!/usr/bin/env tarantool test = require("sqltester") local json = require("json") -test:plan(34) +test:plan(30) --!./tcltestrunner.lua -- 2003 December 17 @@ -141,46 +141,6 @@ test:do_execsql_test( -- }) -test:do_execsql_test( - "misc3-2.6", - [[ - SELECT '-2.0e-127' * '-0.5e27' - ]], { - -- - 1e-100 - -- - }) - -test:do_execsql_test( - "misc3-2.7", - [[ - SELECT '+2.0e-127' * '-0.5e27' - ]], { - -- - -1e-100 - -- - }) - -test:do_execsql_test( - "misc3-2.8", - [[ - SELECT 2.0e-27 * '+0.5e+127' - ]], { - -- - 1e+100 - -- - }) - -test:do_execsql_test( - "misc3-2.9", - [[ - SELECT 2.0e-27 * '+0.000005e+132' - ]], { - -- - 1e+100 - -- - }) - -- Ticket #522. Make sure integer overflow is handled properly in -- indices. -- diff --git a/test/sql-tap/tkt-a8a0d2996a.test.lua b/test/sql-tap/tkt-a8a0d2996a.test.lua deleted file mode 100755 index ddaeb60de..000000000 --- a/test/sql-tap/tkt-a8a0d2996a.test.lua +++ /dev/null @@ -1,146 +0,0 @@ -#!/usr/bin/env tarantool -test = require("sqltester") -test:plan(12) - ---!./tcltestrunner.lua --- 2014-03-24 --- --- The author disclaims copyright to this source code. In place of --- a legal notice, here is a blessing: --- --- May you do good and not evil. --- May you find forgiveness for yourself and forgive others. --- May you share freely, never taking more than you give. --- -------------------------------------------------------------------------- --- --- Tests to verify that arithmetic operators do not change the type of --- input operands. Ticket [a8a0d2996a] --- --- ["set","testdir",[["file","dirname",["argv0"]]]] --- ["source",[["testdir"],"\/tester.tcl"]] -testprefix = "tkt-a8a0d2996a" -test:do_execsql_test( - 1.0, - [[ - CREATE TABLE t(id INT PRIMARY KEY, x TEXT UNIQUE, y TEXT); - INSERT INTO t VALUES(1, '1','1'); - SELECT typeof(x), typeof(y) FROM t WHERE 1=x+0 AND y=='1'; - ]], { - -- <1.0> - "string", "string" - -- - }) - -test:do_execsql_test( - 1.1, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x-0 AND y=='1'; - ]], { - -- <1.1> - "string", "string" - -- - }) - -test:do_execsql_test( - 1.2, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x*1 AND y=='1'; - ]], { - -- <1.2> - "string", "string" - -- - }) - -test:do_execsql_test( - 1.3, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x/1 AND y=='1'; - ]], { - -- <1.3> - "string", "string" - -- - }) - -test:do_execsql_test( - 1.4, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x%4 AND y=='1'; - ]], { - -- <1.4> - "string", "string" - -- - }) - -test:do_execsql_test( - 3.0, - [[ - UPDATE t SET x='1.0'; - SELECT typeof(x), typeof(y) FROM t WHERE 1=x+0 AND y=='1'; - ]], { - -- <3.0> - "string", "string" - -- - }) - -test:do_execsql_test( - 3.1, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x-0 AND y=='1'; - ]], { - -- <3.1> - "string", "string" - -- - }) - -test:do_execsql_test( - 3.2, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x*1 AND y=='1'; - ]], { - -- <3.2> - "string", "string" - -- - }) - -test:do_execsql_test( - 3.3, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x/1 AND y=='1'; - ]], { - -- <3.3> - "string", "string" - -- - }) - -test:do_execsql_test( - 3.4, - [[ - SELECT typeof(x), typeof(y) FROM t WHERE 1=x%4 AND y=='1'; - ]], { - -- <3.4> - "string", "string" - -- - }) - -test:do_execsql_test( - 4.0, - [[ - SELECT 1+1.; - ]], { - -- <4.0> - 2.0 - -- - }) - -test:do_execsql_test( - 4.1, - [[ - SELECT '1.23e64'/'1.0000e+62'; - ]], { - -- <4.1> - 123.0 - -- - }) - -test:finish_test() diff --git a/test/sql/types.result b/test/sql/types.result index 442245186..caedbf409 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -310,11 +310,8 @@ box.execute('SELECT 1 + 1.1;') ... box.execute('SELECT \'9223372036854\' + 1;') --- -- metadata: - - name: COLUMN_1 - type: integer - rows: - - [9223372036855] +- null +- 'Type mismatch: can not convert 9223372036854 to numeric' ... -- Fix BOOLEAN bindings. box.execute('SELECT ?', {true}) @@ -2795,3 +2792,57 @@ box.execute([[DROP TABLE ts;]]) --- - row_count: 1 ... +-- +-- Make sure there is no implicit string-to-number conversion in arithmetic +-- operations. +-- +box.execute([[SELECT '1' + 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT '1' % 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT '1' * 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT '1' / 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT '1' - 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT 1 + '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT 1 % '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT 1 * '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT 1 / '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT 1 - '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 0270d9f8a..844a6b670 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -623,3 +623,18 @@ box.execute([[DROP TABLE tb;]]) box.execute([[DROP TABLE tt;]]) box.execute([[DROP TABLE tv;]]) box.execute([[DROP TABLE ts;]]) + +-- +-- Make sure there is no implicit string-to-number conversion in arithmetic +-- operations. +-- +box.execute([[SELECT '1' + 2;]]) +box.execute([[SELECT '1' % 2;]]) +box.execute([[SELECT '1' * 2;]]) +box.execute([[SELECT '1' / 2;]]) +box.execute([[SELECT '1' - 2;]]) +box.execute([[SELECT 1 + '2';]]) +box.execute([[SELECT 1 % '2';]]) +box.execute([[SELECT 1 * '2';]]) +box.execute([[SELECT 1 / '2';]]) +box.execute([[SELECT 1 - '2';]]) -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 11:40:52 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 11:40:52 +0300 Subject: [Tarantool-patches] [PATCH v1 2/2] sql: remove implicit cast in bitwise operations In-Reply-To: References: Message-ID: <56f685097fb1120e36bf03114a32d567e668a2a2.1597998754.git.imeevma@gmail.com> This patch removes the implicit conversion from STRING to INTEGER from bitwise operations. However, DOUBLE can still be implicitly converted to INTEGER. Follow-up #3809 --- src/box/sql/vdbe.c | 59 ++++++++++++++++++++++++++++++++--------- test/sql/types.result | 49 ++++++++++++++++++++++++++++++++++ test/sql/types.test.lua | 14 ++++++++++ 3 files changed, 110 insertions(+), 12 deletions(-) diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index c0143a6b1..a8db6d076 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -1989,19 +1989,44 @@ case OP_ShiftRight: { /* same as TK_RSHIFT, in1, in2, out3 */ pIn1 = &aMem[pOp->p1]; pIn2 = &aMem[pOp->p2]; + enum mp_type type1 = mem_mp_type(pIn1); + enum mp_type type2 = mem_mp_type(pIn2); pOut = vdbe_prepare_null_out(p, pOp->p3); - if ((pIn1->flags | pIn2->flags) & MEM_Null) { + if (type1 == MP_NIL || type2 == MP_NIL) { /* Force NULL be of type INTEGER. */ pOut->field_type = FIELD_TYPE_INTEGER; break; } - bool unused; - if (sqlVdbeIntValue(pIn2, (int64_t *) &iA, &unused) != 0) { + if (type2 == MP_DOUBLE) { + double r = pIn2->u.r; + if (r >= 0 && r < (double)UINT64_MAX) { + iB = (int64_t)(uint64_t)r; + type2 = MP_UINT; + } else if (r >= (double)INT64_MIN && r < 0) { + iB = (int64_t)r; + type2 = MP_INT; + } + } + if (type2 == MP_INT || type2 == MP_UINT) { + iB = pIn2->u.i; + } else { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn2), "integer"); goto abort_due_to_error; } - if (sqlVdbeIntValue(pIn1, (int64_t *) &iB, &unused) != 0) { + if (type1 == MP_DOUBLE) { + double r = pIn1->u.r; + if (r >= 0 && r < (double)UINT64_MAX) { + iA = (int64_t)(uint64_t)r; + type1 = MP_UINT; + } else if (r >= (double)INT64_MIN && r < 0) { + iA = (int64_t)r; + type1 = MP_INT; + } + } + if (type1 == MP_INT || type1 == MP_UINT) { + iA = pIn1->u.i; + } else { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn1), "integer"); goto abort_due_to_error; @@ -2621,19 +2646,29 @@ case OP_Not: { /* same as TK_NOT, in1, out2 */ */ case OP_BitNot: { /* same as TK_BITNOT, in1, out2 */ pIn1 = &aMem[pOp->p1]; + enum mp_type type = mem_mp_type(pIn1); pOut = vdbe_prepare_null_out(p, pOp->p2); /* Force NULL be of type INTEGER. */ pOut->field_type = FIELD_TYPE_INTEGER; - if ((pIn1->flags & MEM_Null)==0) { - int64_t i; - bool is_neg; - if (sqlVdbeIntValue(pIn1, &i, &is_neg) != 0) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(pIn1), "integer"); - goto abort_due_to_error; + int64_t i; + if (type == MP_DOUBLE) { + double r = pIn1->u.r; + if (r >= 0 && r < (double)UINT64_MAX) { + i = (int64_t)(uint64_t)r; + type = MP_UINT; + } else if (r >= (double)INT64_MIN && r < 0) { + i = (int64_t)r; + type = MP_INT; } - mem_set_i64(pOut, ~i); } + if (type == MP_INT || type == MP_UINT) { + i = pIn1->u.i; + } else { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(pIn1), "integer"); + goto abort_due_to_error; + } + mem_set_i64(pOut, ~i); break; } diff --git a/test/sql/types.result b/test/sql/types.result index caedbf409..601e5beca 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2846,3 +2846,52 @@ box.execute([[SELECT 1 - '2';]]) - null - 'Type mismatch: can not convert 2 to numeric' ... +-- +-- Make sure there is no implicit string-to-number conversion in bitwise +-- operations. +-- +box.execute([[SELECT '1' | 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' & 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' << 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' >> 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT ~'1';]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT 1 | '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 & '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 << '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 >> '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 844a6b670..2d2f41da2 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -638,3 +638,17 @@ box.execute([[SELECT 1 % '2';]]) box.execute([[SELECT 1 * '2';]]) box.execute([[SELECT 1 / '2';]]) box.execute([[SELECT 1 - '2';]]) + +-- +-- Make sure there is no implicit string-to-number conversion in bitwise +-- operations. +-- +box.execute([[SELECT '1' | 2;]]) +box.execute([[SELECT '1' & 2;]]) +box.execute([[SELECT '1' << 2;]]) +box.execute([[SELECT '1' >> 2;]]) +box.execute([[SELECT ~'1';]]) +box.execute([[SELECT 1 | '2';]]) +box.execute([[SELECT 1 & '2';]]) +box.execute([[SELECT 1 << '2';]]) +box.execute([[SELECT 1 >> '2';]]) -- 2.25.1 From korablev at tarantool.org Fri Aug 21 11:46:25 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Fri, 21 Aug 2020 08:46:25 +0000 Subject: [Tarantool-patches] [PATCH v3] tuple: drop extra restrictions for multikey index In-Reply-To: <20200814145143.23720-1-i.kosarev@tarantool.org> References: <20200814145143.23720-1-i.kosarev@tarantool.org> Message-ID: <20200821084625.GA6452@tarantool.org> On 14 Aug 17:51, Ilya Kosarev wrote: > Multikey index did not work properly with nullable root field in > tuple_raw_multikey_count(). Now it is fixed and corresponding > restrictions are dropped. This also means that we can drop implicit > nullability update for array/map fields and make all fields nullable > by default, as it was until e1d3fe8ab8eed65394ad17409401a93b6fcdc435 > (tuple format: don't allow null where array/map is expected), as far as > default non-nullability itself doesn't solve any real problems while > providing confusing behavior (gh-5027). > > Follow-up #5027 > Closes #5192 > > @TarantoolBot document > Title: tuple: allow nullable multikey index root > Update the documentation for space_object:create_index() to reflect > that it is now safe for multikey index root to be nullable. > Previously it was not safe to insert, for example, box.NULL as an array > field which elements are the part of multikey index due to error in > counter of multikey index keys in tuple. It was partly fixed using > default non-nullability for tuple fields. In 2.3.3 the restriction on > nullable multikey index root was introduced. Now the counter problem is > fixed and all tuple fields are nullable by default as before 2.2.1. > --- > Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5192-fix-multikey-index-restrictions > Issue: https://github.com/tarantool/tarantool/issues/5192 > > @ChangeLog: > * Dropped restrictions on nullable multikey index root. They were > introduced due to inaccuracy in multikey index realization. It is > now fixed. Also all fields are now nullable by default as it was > before 2.2.1 (gh-5192). > > Changes in v2: > - removed insignificant changes > - fixed tests comments > > Changes in v3: > - fixed some nits > - added docbot request LGTM now. From sergepetrenko at tarantool.org Fri Aug 21 11:51:13 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 11:51:13 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_5/7=5D_applier=3A_proce?= =?utf-8?q?ss_synchro_requests_without_txn_engine?= In-Reply-To: <20200819213442.1099018-6-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-6-gorcunov@gmail.com> Message-ID: <1597999873.222856578@f425.i.mail.ru> Hi! Thanks for the patch. LGTM with one comment which?s up to you. ? ? >???????, 20 ??????? 2020, 0:36 +03:00 ?? Cyrill Gorcunov : >? >Transaction processing code is very heavy simply because >transactions are carrying various data and involves a number >of other mechanisms to proceed. > >In turn, when we receive confirm or rollback packed from >another node in a cluster we just need to inspect limbo >queue and write this packed into a WAL journal. So calling >a bunch of txn engine helpers is simply waste of cycles. > >Thus lets rather handle them in a special light way: > >?- allocate synchro_entry structure which would carry >???the journal entry itself and encoded message >?- process limbo queue to mark confirmed/rollback'ed >???messages >?- finally write this synchro_entry into a journal > >Which is a way simplier. > >Part-of #5129 > >Suggedsted-by: Vladislav Shpilevoy < v.shpilevoy at tarantool.org > >Co-developed-by: Vladislav Shpilevoy < v.shpilevoy at tarantool.org > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/applier.cc | 200 +++++++++++++++++++++++++++++++++------------ >?1 file changed, 148 insertions(+), 52 deletions(-) > >diff --git a/src/box/applier.cc b/src/box/applier.cc >index 1387d518c..c1d07ca54 100644 >--- a/src/box/applier.cc >+++ b/src/box/applier.cc >@@ -51,8 +51,10 @@ >?#include "schema.h" >?#include "txn.h" >?#include "box.h" >+#include "xrow.h" >?#include "scoped_guard.h" >?#include "txn_limbo.h" >+#include "journal.h" >? >?STRS(applier_state, applier_STATE); >? >@@ -268,45 +270,11 @@ process_nop(struct request *request) >? return txn_commit_stmt(txn, request); >?} >? >-/* >- * CONFIRM/ROLLBACK rows aren't dml requests and require special >- * handling: instead of performing some operations on spaces, >- * processing these requests requires txn_limbo to either confirm >- * or rollback some of its entries. >- */ >-static int >-process_synchro_row(struct request *request) >-{ >- assert(iproto_type_is_synchro_request(request->header->type)); >- struct txn *txn = in_txn(); >- >- struct synchro_request syn_req; >- if (xrow_decode_synchro(request->header, &syn_req) != 0) >- return -1; >- assert(txn->n_applier_rows == 0); >- /* >- * This is not really a transaction. It just uses txn API >- * to put the data into WAL. And obviously it should not >- * go to the limbo and block on the very same sync >- * transaction which it tries to confirm now. >- */ >- txn_set_flag(txn, TXN_FORCE_ASYNC); >- >- if (txn_begin_stmt(txn, NULL) != 0) >- return -1; >- if (txn_commit_stmt(txn, request) != 0) >- return -1; >- return txn_limbo_process(&txn_limbo, &syn_req); >-} >- >?static int >?apply_row(struct xrow_header *row) >?{ >? struct request request; >- if (iproto_type_is_synchro_request(row->type)) { >- request.header = row; >- return process_synchro_row(&request); >- } >+ assert(!iproto_type_is_synchro_request(row->type)); >? if (xrow_decode_dml(row, &request, dml_request_key_map(row->type)) != 0) >? return -1; >? if (request.type == IPROTO_NOP) >@@ -753,19 +721,9 @@ applier_read_tx(struct applier *applier, struct stailq *rows) >? next)->row.is_commit); >?} >? >-static int >-applier_txn_rollback_cb(struct trigger *trigger, void *event) >+static void >+applier_rollback_by_wal_io(void) >?{ >- (void) trigger; >- struct txn *txn = (struct txn *) event; >- /* >- * Synchronous transaction rollback due to receiving a >- * ROLLBACK entry is a normal event and requires no >- * special handling. >- */ >- if (txn->signature == TXN_SIGNATURE_SYNC_ROLLBACK) >- return 0; >- >? /* >? * Setup shared applier diagnostic area. >? * >@@ -774,9 +732,9 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) >? * diag use per-applier diag instead all the time >? * (which actually already present in the structure). >? * >- * But remember that transactions are asynchronous >- * and rollback may happen a way latter after it >- * passed to the journal engine. >+ * But remember that WAL writes are asynchronous and >+ * rollback may happen a way later after it was passed to >+ * the journal engine. >? */ >? diag_set(ClientError, ER_WAL_IO); >? diag_set_error(&replicaset.applier.diag, >@@ -787,6 +745,20 @@ applier_txn_rollback_cb(struct trigger *trigger, void *event) >? >? /* Rollback applier vclock to the committed one. */ >? vclock_copy(&replicaset.applier.vclock, &replicaset.vclock); >+} >+ >+static int >+applier_txn_rollback_cb(struct trigger *trigger, void *event) >+{ >+ (void) trigger; >+ struct txn *txn = (struct txn *) event; >+ /* >+ * Synchronous transaction rollback due to receiving a >+ * ROLLBACK entry is a normal event and requires no >+ * special handling. >+ */ >+ if (txn->signature != TXN_SIGNATURE_SYNC_ROLLBACK) >+ applier_rollback_by_wal_io(); >? return 0; >?} >? >@@ -800,6 +772,110 @@ applier_txn_wal_write_cb(struct trigger *trigger, void *event) >? return 0; >?} >? >+struct synchro_entry { >+ /** Encoded form of a synchro record. */ >+ struct synchro_body_bin body_bin; >+ >+ /** xrow to write, used by the journal engine. */ >+ struct xrow_header row; >+ >+ /** >+ * The journal entry itself. Note since >+ * it has unsized array it must be the >+ * last entry in the structure. >+ */ >+ struct journal_entry journal_entry; >+}; >+ >+static void >+synchro_entry_delete(struct synchro_entry *entry) >+{ >+ free(entry); >+} >+ >+/** >+ * Async write journal completion. >+ */ >+static void >+apply_synchro_row_cb(struct journal_entry *entry) >+{ >+ assert(entry->complete_data != NULL); >+ struct synchro_entry *synchro_entry = >+ (struct synchro_entry *)entry->complete_data; >+ if (entry->res < 0) >+ applier_rollback_by_wal_io(); >+ else >+ trigger_run(&replicaset.applier.on_wal_write, NULL); >+ >+ synchro_entry_delete(synchro_entry); >+} >+ >+/** >+ * Allocate a new synchro_entry to be passed to >+ * the journal engine in async write way. >+ */ >+static struct synchro_entry * >+synchro_entry_new(struct xrow_header *applier_row, >+ struct synchro_request *req) >+{ >+ struct synchro_entry *entry; >+ size_t size = sizeof(*entry) + sizeof(struct xrow_header *); >+ >+ /* >+ * For simplicity we use malloc here but >+ * probably should provide some cache similar >+ * to txn cache. >+ */ >+ entry = (struct synchro_entry *)malloc(size); >+ if (entry == NULL) { >+ diag_set(OutOfMemory, size, "malloc", "synchro_entry"); >+ return NULL; >+ } >+ >+ struct journal_entry *journal_entry = &entry->journal_entry; >+ struct synchro_body_bin *body_bin = &entry->body_bin; >+ struct xrow_header *row = &entry->row; >+ >+ journal_entry->rows[0] = row; >+ >+ xrow_encode_synchro(row, body_bin, req); >+ >+ row->lsn = applier_row->lsn; >+ row->replica_id = applier_row->replica_id; >+ >+ journal_entry_create(journal_entry, 1, xrow_approx_len(row), >+ apply_synchro_row_cb, entry); >+ return entry; >+} >+ >+/** Process a synchro request. */ >+static int >+apply_synchro_row(struct xrow_header *row) >+{ >+ assert(iproto_type_is_synchro_request(row->type)); >+ >+ struct synchro_request req; >+ if (xrow_decode_synchro(row, &req) != 0) >+ goto err; >+ >+ if (txn_limbo_process(&txn_limbo, &req)) >+ goto err; >+ >+ struct synchro_entry *entry; >+ entry = synchro_entry_new(row, &req); >+ if (entry == NULL) >+ goto err; >+ >+ if (journal_write_async(&entry->journal_entry) != 0) { >+ diag_set(ClientError, ER_WAL_IO); >+ goto err; >+ } >+ return 0; >+err: >+ diag_log(); >+ return -1; >+} >+ >?/** >??* Apply all rows in the rows queue as a single transaction. >??* >@@ -847,13 +923,26 @@ applier_apply_tx(struct stailq *rows) >? } >? } >? >+ if (unlikely(iproto_type_is_synchro_request(first_row->type))) { >+ /* >+ * Synchro messages are not transactions, in terms >+ * of DML. Always sent and written isolated from >+ * each other. >+ */ >+ assert(first_row == last_row); >+ if (apply_synchro_row(first_row) != 0) >+ diag_raise(); >+ goto success; >+ } >+ >? /** >? * Explicitly begin the transaction so that we can >? * control fiber->gc life cycle and, in case of apply >? * conflict safely access failed xrow object and allocate >? * IPROTO_NOP on gc. >? */ >- struct txn *txn = txn_begin(); >+ struct txn *txn; >+ txn = txn_begin(); ? Why this change? ? >? struct applier_tx_row *item; >? if (txn == NULL) { >? latch_unlock(latch); >@@ -922,6 +1011,7 @@ applier_apply_tx(struct stailq *rows) >? if (txn_commit_async(txn) < 0) >? goto fail; >? >+success: >? /* >? * The transaction was sent to journal so promote vclock. >? * >@@ -1089,7 +1179,13 @@ applier_subscribe(struct applier *applier) >? >? applier->lag = TIMEOUT_INFINITY; >? >- /* Register triggers to handle WAL writes and rollbacks. */ >+ /* >+ * Register triggers to handle WAL writes and rollbacks. >+ * >+ * Note we use them for syncronous packets handling as well >+ * thus when changing make sure that synchro handling won't >+ * be broken. >+ */ >? struct trigger on_wal_write; >? trigger_create(&on_wal_write, applier_on_wal_write, applier, NULL); >? trigger_add(&replicaset.applier.on_wal_write, &on_wal_write); >-- >2.26.2 >? ? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 21 11:52:14 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 11:52:14 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_6/7=5D_txn=3A_txn=5Fadd?= =?utf-8?q?=5Fredo_--_drop_synchro_processing?= In-Reply-To: <20200819213442.1099018-7-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-7-gorcunov@gmail.com> Message-ID: <1597999934.69982009@f106.i.mail.ru> LGTM. ? ? ? >???????, 20 ??????? 2020, 0:36 +03:00 ?? Cyrill Gorcunov : >? >Since we no longer use txn engine for synchro >packets processing this code is never executed. > >Part-of #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/txn.c | 9 +-------- >?1 file changed, 1 insertion(+), 8 deletions(-) > >diff --git a/src/box/txn.c b/src/box/txn.c >index cc1f496c5..b2d342355 100644 >--- a/src/box/txn.c >+++ b/src/box/txn.c >@@ -82,14 +82,7 @@ txn_add_redo(struct txn *txn, struct txn_stmt *stmt, struct request *request) >? */ >? struct space *space = stmt->space; >? row->group_id = space != NULL ? space_group_id(space) : 0; >- /* >- * Sychronous replication entries are supplementary and >- * aren't valid dml requests. They're encoded manually. >- */ >- if (likely(!iproto_type_is_synchro_request(row->type))) >- row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); >- else >- row->bodycnt = xrow_header_dup_body(row, &txn->region); >+ row->bodycnt = xrow_encode_dml(request, &txn->region, row->body); >? if (row->bodycnt < 0) >? return -1; >? stmt->row = row; >-- >2.26.2 -- Serge?Petrenko ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 21 11:57:23 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Fri, 21 Aug 2020 11:57:23 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_7/7=5D_xrow=3A_drop_xro?= =?utf-8?q?w=5Fheader=5Fdup=5Fbody?= In-Reply-To: <20200819213442.1099018-8-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-8-gorcunov@gmail.com> Message-ID: <1598000242.195231897@f106.i.mail.ru> LGTM. ? ? >???????, 20 ??????? 2020, 0:36 +03:00 ?? Cyrill Gorcunov : >? >We no longer use it. > >Closes #5129 > >Signed-off-by: Cyrill Gorcunov < gorcunov at gmail.com > >--- >?src/box/xrow.c | 15 --------------- >?src/box/xrow.h | 8 -------- >?2 files changed, 23 deletions(-) > >diff --git a/src/box/xrow.c b/src/box/xrow.c >index 9c6fb4fc1..95ddb1fe7 100644 >--- a/src/box/xrow.c >+++ b/src/box/xrow.c >@@ -220,21 +220,6 @@ xrow_header_decode(struct xrow_header *header, const char **pos, >? return 0; >?} >? >-int >-xrow_header_dup_body(struct xrow_header *row, struct region *region) >-{ >- assert(row->bodycnt == 1); >- size_t size = row->body[0].iov_len; >- char *copy = (char *)region_alloc(region, size); >- if (copy == NULL) { >- diag_set(OutOfMemory, size, "region_alloc", "copy"); >- return -1; >- } >- memcpy(copy, row->body[0].iov_base, size); >- row->body[0].iov_base = copy; >- return 1; >-} >- >?/** >??* @pre pos points at a valid msgpack >??*/ >diff --git a/src/box/xrow.h b/src/box/xrow.h >index 20e82034d..58d47b12d 100644 >--- a/src/box/xrow.h >+++ b/src/box/xrow.h >@@ -141,14 +141,6 @@ int >?xrow_header_decode(struct xrow_header *header, const char **pos, >? const char *end, bool end_is_exact); >? >-/** >- * Duplicate the xrow's body onto the given region. >- * @retval -1 Error. >- * @retval >= 0 Iov count in the body. >- */ >-int >-xrow_header_dup_body(struct xrow_header *header, struct region *region); >- >?/** >??* DML request. >??*/ >-- >2.26.2 >? ? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From korablev at tarantool.org Fri Aug 21 11:59:03 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Fri, 21 Aug 2020 08:59:03 +0000 Subject: [Tarantool-patches] [PATCH v1 1/2] sql: remove implicit cast in arithmetic operations In-Reply-To: References: Message-ID: <20200821085903.GB6452@tarantool.org> On 21 Aug 11:40, imeevma at tarantool.org wrote: > This patch removes the implicit conversion from STRING to NUMBER from > arithmetic operations. However, INTEGER can still be implicitly > converted to DOUBLE if the second operand is of type DOUBLE. > > Follow-up #3809 LGTM From imeevma at tarantool.org Fri Aug 21 12:19:47 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:47 +0300 Subject: [Tarantool-patches] [PATCH v5 0/6] sql; remove implicit cast for comparison Message-ID: This patch-set removes implicit cast from STRING to NUMBER and vice versa for comparison. https://github.com/tarantool/tarantool/issues/4230 https://github.com/tarantool/tarantool/tree/imeevma/gh-4230-remove-implicit-cast-for-comparison @ChangeLog - Implicit cast from STRING to number and vice versa for comparison removed (gh-4230). Changes in v5: - Patches were moved in a new patch-set. - Patch-set was simplified since implicit cast for assignment was removed. Mergen Imeev (6): sql: remove unused DOUBLE to INTEGER conversion sql: add implicit cast between numbers in OP_Seek* sql: change comparison between numbers using index sql: remove implicit cast from comparison opcodes sql: fix implicit cast in opcode MustBeInt sql: remove implicit cast from MakeRecord opcode src/box/sql/analyze.c | 6 +- src/box/sql/delete.c | 15 +- src/box/sql/expr.c | 17 +- src/box/sql/fk_constraint.c | 12 +- src/box/sql/select.c | 26 +- src/box/sql/sqlInt.h | 2 + src/box/sql/update.c | 23 +- src/box/sql/vdbe.c | 536 ++++++++++++++++++-------- src/box/sql/wherecode.c | 103 +---- test/sql-tap/identifier_case.test.lua | 6 +- test/sql-tap/in1.test.lua | 4 +- test/sql-tap/in3.test.lua | 26 +- test/sql-tap/in4.test.lua | 4 +- test/sql-tap/insert3.test.lua | 2 +- test/sql-tap/join.test.lua | 8 +- test/sql-tap/misc1.test.lua | 32 +- test/sql-tap/select1.test.lua | 4 +- test/sql-tap/select7.test.lua | 2 +- test/sql-tap/subquery.test.lua | 4 +- test/sql-tap/tkt-9a8b09f8e6.test.lua | 508 ------------------------ test/sql-tap/tkt3493.test.lua | 40 +- test/sql-tap/transitive1.test.lua | 12 +- test/sql-tap/where2.test.lua | 183 +-------- test/sql-tap/where5.test.lua | 12 +- test/sql/boolean.result | 76 +--- test/sql/types.result | 286 +++++++++++++- test/sql/types.test.lua | 63 +++ 27 files changed, 875 insertions(+), 1137 deletions(-) delete mode 100755 test/sql-tap/tkt-9a8b09f8e6.test.lua -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:49 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:49 +0300 Subject: [Tarantool-patches] [PATCH v5 1/6] sql: remove unused DOUBLE to INTEGER conversion In-Reply-To: References: Message-ID: This patch removes the unused DOUBLE to INTEGER conversion from OP_Seek* opcodes. This transformation is not used due to changes in the ApplyType opcode. The next few patches will introduce new rules for converting numbers (not just DOUBLE to INTEGER), and the implicit conversion within the ApplyType opcode will be disabled for this case. Part of #4230 --- src/box/sql/vdbe.c | 106 ++-------------------------------------- src/box/sql/wherecode.c | 27 ---------- 2 files changed, 5 insertions(+), 128 deletions(-) diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 14ddb5160..7405009a7 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -3350,7 +3350,7 @@ case OP_Close: { break; } -/* Opcode: SeekGE P1 P2 P3 P4 P5 +/* Opcode: SeekGE P1 P2 P3 P4 * * Synopsis: key=r[P3 at P4] * * If cursor P1 refers to an SQL table (B-Tree that uses integer keys), @@ -3373,14 +3373,9 @@ case OP_Close: { * from the beginning toward the end. In other words, the cursor is * configured to use Next, not Prev. * - * If P5 is not zero, than it is offset of integer fields in input - * vector. Force corresponding value to be INTEGER, in case it - * is floating point value. Alongside with that, type of - * iterator may be changed: a > 1.5 -> a >= 2. - * * See also: Found, NotFound, SeekLt, SeekGt, SeekLe */ -/* Opcode: SeekGT P1 P2 P3 P4 P5 +/* Opcode: SeekGT P1 P2 P3 P4 * * Synopsis: key=r[P3 at P4] * * If cursor P1 refers to an SQL table (B-Tree that uses integer keys), @@ -3396,12 +3391,9 @@ case OP_Close: { * from the beginning toward the end. In other words, the cursor is * configured to use Next, not Prev. * - * If P5 is not zero, than it is offset of integer fields in input - * vector. Force corresponding value to be INTEGER. - * - * P5 has the same meaning as for SeekGE. + * See also: Found, NotFound, SeekLt, SeekGe, SeekLe */ -/* Opcode: SeekLT P1 P2 P3 P4 P5 +/* Opcode: SeekLT P1 P2 P3 P4 * * Synopsis: key=r[P3 at P4] * * If cursor P1 refers to an SQL table (B-Tree that uses integer keys), @@ -3417,11 +3409,9 @@ case OP_Close: { * from the end toward the beginning. In other words, the cursor is * configured to use Prev, not Next. * - * P5 has the same meaning as for SeekGE. - * * See also: Found, NotFound, SeekGt, SeekGe, SeekLe */ -/* Opcode: SeekLE P1 P2 P3 P4 P5 +/* Opcode: SeekLE P1 P2 P3 P4 * * Synopsis: key=r[P3 at P4] * * If cursor P1 refers to an SQL table (B-Tree that uses integer keys), @@ -3444,8 +3434,6 @@ case OP_Close: { * The IdxGE opcode will be skipped if this opcode succeeds, but the * IdxGE opcode will be used on subsequent loop iterations. * - * P5 has the same meaning as for SeekGE. - * * See also: Found, NotFound, SeekGt, SeekGe, SeekLt */ case OP_SeekLT: /* jump, in3 */ @@ -3457,7 +3445,6 @@ case OP_SeekGT: { /* jump, in3 */ VdbeCursor *pC; /* The cursor to seek */ UnpackedRecord r; /* The key to seek for */ int nField; /* Number of columns or fields in the key */ - i64 iKey; /* The id we are to seek to */ int eqOnly; /* Only interested in == results */ assert(pOp->p1>=0 && pOp->p1nCursor); @@ -3475,86 +3462,6 @@ case OP_SeekGT: { /* jump, in3 */ #ifdef SQL_DEBUG pC->seekOp = pOp->opcode; #endif - iKey = 0; - /* - * In case floating value is intended to be passed to - * iterator over integer field, we must truncate it to - * integer value and change type of iterator: - * a > 1.5 -> a >= 2 - */ - int int_field = pOp->p5; - bool is_neg = false; - - if (int_field > 0) { - /* The input value in P3 might be of any type: integer, real, string, - * blob, or NULL. But it needs to be an integer before we can do - * the seek, so convert it. - */ - pIn3 = &aMem[int_field]; - if ((pIn3->flags & MEM_Null) != 0) - goto skip_truncate; - if ((pIn3->flags & MEM_Str) != 0) - mem_apply_numeric_type(pIn3); - int64_t i; - if ((pIn3->flags & MEM_Int) == MEM_Int) { - i = pIn3->u.i; - is_neg = true; - } else if ((pIn3->flags & MEM_UInt) == MEM_UInt) { - i = pIn3->u.u; - is_neg = false; - } else if ((pIn3->flags & MEM_Real) == MEM_Real) { - if (pIn3->u.r > INT64_MAX) - i = INT64_MAX; - else if (pIn3->u.r < INT64_MIN) - i = INT64_MIN; - else - i = pIn3->u.r; - is_neg = i < 0; - } else { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(pIn3), "integer"); - goto abort_due_to_error; - } - iKey = i; - - /* If the P3 value could not be converted into an integer without - * loss of information, then special processing is required... - */ - if ((pIn3->flags & (MEM_Int | MEM_UInt)) == 0) { - if ((pIn3->flags & MEM_Real)==0) { - /* If the P3 value cannot be converted into any kind of a number, - * then the seek is not possible, so jump to P2 - */ - VdbeBranchTaken(1,2); goto jump_to_p2; - break; - } - - /* If the approximation iKey is larger than the actual real search - * term, substitute >= for > and < for <=. e.g. if the search term - * is 4.9 and the integer approximation 5: - * - * (x > 4.9) -> (x >= 5) - * (x <= 4.9) -> (x < 5) - */ - if (pIn3->u.r<(double)iKey) { - assert(OP_SeekGE==(OP_SeekGT-1)); - assert(OP_SeekLT==(OP_SeekLE-1)); - assert((OP_SeekLE & 0x0001)==(OP_SeekGT & 0x0001)); - if ((oc & 0x0001)==(OP_SeekGT & 0x0001)) oc--; - } - - /* If the approximation iKey is smaller than the actual real search - * term, substitute <= for < and > for >=. - */ - else if (pIn3->u.r>(double)iKey) { - assert(OP_SeekLE==(OP_SeekLT+1)); - assert(OP_SeekGT==(OP_SeekGE+1)); - assert((OP_SeekLT & 0x0001)==(OP_SeekGE & 0x0001)); - if ((oc & 0x0001)==(OP_SeekLT & 0x0001)) oc++; - } - } - } -skip_truncate: /* * For a cursor with the OPFLAG_SEEKEQ hint, only the * OP_SeekGE and OP_SeekLE opcodes are allowed, and these @@ -3577,9 +3484,6 @@ skip_truncate: r.key_def = pC->key_def; r.nField = (u16)nField; - if (int_field > 0) - mem_set_int(&aMem[int_field], iKey, is_neg); - r.default_rc = ((1 & (oc - OP_SeekLT)) ? -1 : +1); assert(oc!=OP_SeekGT || r.default_rc==-1); assert(oc!=OP_SeekLE || r.default_rc==-1); diff --git a/src/box/sql/wherecode.c b/src/box/sql/wherecode.c index 6d8768865..97ba59151 100644 --- a/src/box/sql/wherecode.c +++ b/src/box/sql/wherecode.c @@ -910,10 +910,6 @@ sqlWhereCodeOneLoopStart(WhereInfo * pWInfo, /* Complete information about the W enum field_type *end_types = NULL; u8 bSeekPastNull = 0; /* True to seek past initial nulls */ u8 bStopAtNull = 0; /* Add condition to terminate at NULLs */ - int force_integer_reg = -1; /* If non-negative: number of - * column which must be converted - * to integer type, used for IPK. - */ struct index_def *idx_def = pLoop->index_def; assert(idx_def != NULL); @@ -1120,21 +1116,6 @@ sqlWhereCodeOneLoopStart(WhereInfo * pWInfo, /* Complete information about the W 1); } } - /* Inequality constraint comes always at the end of list. */ - part_count = idx_def->key_def->part_count; - if (pRangeStart != NULL) { - /* - * nEq == 0 means that filter condition - * contains only inequality. - */ - uint32_t ineq_idx = nEq == 0 ? 0 : nEq - 1; - assert(ineq_idx < part_count); - enum field_type ineq_type = - idx_def->key_def->parts[ineq_idx].type; - if (ineq_type == FIELD_TYPE_INTEGER || - ineq_type == FIELD_TYPE_UNSIGNED) - force_integer_reg = regBase + nEq; - } emit_apply_type(pParse, regBase, nConstraint - bSeekPastNull, start_types); if (pLoop->nSkip > 0 && nConstraint == pLoop->nSkip) { @@ -1152,14 +1133,6 @@ sqlWhereCodeOneLoopStart(WhereInfo * pWInfo, /* Complete information about the W } sqlVdbeAddOp4Int(v, op, iIdxCur, addrNxt, regBase, nConstraint); - /* If this is Seek* opcode, and IPK is detected in the - * constraints vector: force it to be integer. - */ - if ((op == OP_SeekGE || op == OP_SeekGT - || op == OP_SeekLE || op == OP_SeekLT) - && force_integer_reg > 0) { - sqlVdbeChangeP5(v, force_integer_reg); - } VdbeCoverage(v); VdbeCoverageIf(v, op == OP_Rewind); testcase(op == OP_Rewind); -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:51 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:51 +0300 Subject: [Tarantool-patches] [PATCH v5 2/6] sql: add implicit cast between numbers in OP_Seek* In-Reply-To: References: Message-ID: This patch adds new rules for implicit casting between numbers in OP_Seek * opcodes. They are still not used because the ApplyType opcode is converting numbers, but this will be changed in the next patch. Conversion within the ApplyType opcode can affect the result of comparison operations. Part of #4230 --- src/box/sql/vdbe.c | 332 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 332 insertions(+) diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 7405009a7..822d7e177 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -522,6 +522,268 @@ mem_convert_to_numeric(struct Mem *mem, enum field_type type) return mem_convert_to_integer(mem); } +/** + * Convert the numeric value contained in the MEM to UNSIGNED. + * @see mem_prepare_for_cmp() for more info. + * + * @param mem The MEM that contains the numeric value. + * @param[in][out] oc Operation code. + * @retval 0 if the conversion was successful, -1 otherwise. + */ +static int +mem_prepare_for_cmp_to_uint(struct Mem *mem, int *oc, int eqOnly) +{ + if ((mem->flags & MEM_Int) != 0) { + if (eqOnly == 1) + return 0; + if (*oc == OP_SeekGT) { + mem_set_u64(mem, 0); + *oc = OP_SeekGE; + return 0; + } + if (*oc == OP_SeekGE) { + mem_set_u64(mem, 0); + return 0; + } + if (*oc == OP_SeekLT) + return 0; + assert(*oc == OP_SeekLE); + return 0; + } + assert((mem->flags & MEM_Real) != 0); + double d = mem->u.r; + if (d >= 0.0 && d < (double)UINT64_MAX && d == (double)(uint64_t)d) + return mem_convert_to_unsigned(mem); + if (eqOnly == 1) + return 0; + if (*oc == OP_SeekGT) { + if (d >= (double)UINT64_MAX) + return 0; + if (d < 0) { + mem_set_u64(mem, 0); + *oc = OP_SeekGE; + return 0; + } + assert((double)(uint64_t)d < d); + return mem_convert_to_unsigned(mem); + } + if (*oc == OP_SeekGE) { + if (d >= (double)UINT64_MAX) + return 0; + if (d < 0) { + mem_set_u64(mem, 0); + return 0; + } + assert((double)(uint64_t)d < d); + *oc = OP_SeekGT; + return mem_convert_to_unsigned(mem); + } + if (*oc == OP_SeekLT) { + if (d >= (double)UINT64_MAX) { + *oc = OP_SeekLE; + mem_set_u64(mem, UINT64_MAX); + return 0; + } + if (d < 0) + return 0; + assert((double)(uint64_t)d < d); + *oc = OP_SeekLE; + return mem_convert_to_unsigned(mem); + } + assert(*oc == OP_SeekLE); + if (d >= (double)UINT64_MAX) { + mem_set_u64(mem, UINT64_MAX); + return 0; + } + if (d < 0) + return 0; + assert((double)(uint64_t)d < d); + return mem_convert_to_unsigned(mem); +} + +/** + * Convert the numeric value contained in the MEM to INTEGER. + * @see mem_prepare_for_cmp() for more info. + * + * @param mem The MEM that contains the numeric value. + * @param[in][out] oc Operation code. + * @retval 0 if the conversion was successful, -1 otherwise. + */ +static int +mem_prepare_for_cmp_to_int(struct Mem *mem, int *oc, int eqOnly) +{ + assert((mem->flags & MEM_Real) != 0); + double d = mem->u.r; + if (d >= 0.0 && d < (double)UINT64_MAX && d == (double)(uint64_t)d) + return mem_convert_to_integer(mem); + if (d >= (double)INT64_MIN && d < (double)INT64_MAX && + d == (double)(uint64_t)d) + return mem_convert_to_integer(mem); + if (eqOnly == 1) + return 0; + if (*oc == OP_SeekGT) { + if (d >= (double)UINT64_MAX) + return 0; + if (d < (double)INT64_MIN) { + mem_set_i64(mem, INT64_MIN); + *oc = OP_SeekGE; + return 0; + } + if (d > 0 || (double)(int64_t)d < d) + return mem_convert_to_integer(mem); + *oc = OP_SeekGE; + return mem_convert_to_integer(mem); + } + if (*oc == OP_SeekGE) { + if (d >= (double)UINT64_MAX) + return 0; + if (d < (double)INT64_MIN) { + mem_set_i64(mem, INT64_MIN); + return 0; + } + if (d > 0 || (double)(int64_t)d < d) { + *oc = OP_SeekGT; + return mem_convert_to_integer(mem); + } + return mem_convert_to_integer(mem); + } + if (*oc == OP_SeekLT) { + if (d >= (double)UINT64_MAX) { + *oc = OP_SeekLE; + mem_set_int(mem, UINT64_MAX, false); + return 0; + } + if (d < (double)INT64_MIN) + return 0; + if (d > 0 || (double)(int64_t)d < d) { + *oc = OP_SeekLE; + return mem_convert_to_integer(mem); + } + return mem_convert_to_integer(mem); + } + assert(*oc == OP_SeekLE); + if (d >= (double)UINT64_MAX) { + mem_set_int(mem, UINT64_MAX, false); + return 0; + } + if (d > 0 || (double)(int64_t)d < d) + return mem_convert_to_integer(mem); + *oc = OP_SeekLT; + return mem_convert_to_integer(mem); +} + +/** + * Convert the numeric value contained in the MEM to DOUBLE. + * @see mem_prepare_for_cmp() for more info. + * + * @param mem The MEM that contains the numeric value. + * @param[in][out] oc Operation code. + * @retval 0 if the conversion was successful, -1 otherwise. + */ +static int +mem_prepare_for_cmp_to_double(struct Mem *mem, int *oc, int eqOnly) +{ + if ((mem->flags & MEM_Int) != 0) { + int64_t i = mem->u.i; + if (i == (int64_t)(double)i) + return mem_convert_to_double(mem); + if (eqOnly == 1) + return 0; + double d = (double)i; + if (*oc == OP_SeekGT) { + if (d == (double)INT64_MAX || i < (int64_t)d) + *oc = OP_SeekGE; + return mem_convert_to_double(mem); + } + if (*oc == OP_SeekGE) { + if (d != (double)INT64_MAX && i > (int64_t)d) + *oc = OP_SeekGT; + return mem_convert_to_double(mem); + } + if (*oc == OP_SeekLT) { + if (d != (double)INT64_MAX && i > (int64_t)d) + *oc = OP_SeekLE; + return mem_convert_to_double(mem); + } + assert(*oc == OP_SeekLE); + if (d == (double)INT64_MAX || i < (int64_t)d) + *oc = OP_SeekLT; + return mem_convert_to_double(mem); + } + assert((mem->flags & MEM_UInt) != 0); + uint64_t u = mem->u.u; + if (u == (uint64_t)(double)u) + return mem_convert_to_double(mem); + if (eqOnly == 1) + return 0; + double d = (double)u; + if (*oc == OP_SeekGT) { + if (d == (double)UINT64_MAX || u < (uint64_t)d) + *oc = OP_SeekGE; + return mem_convert_to_double(mem); + } + if (*oc == OP_SeekGE) { + if (d != (double)UINT64_MAX && u > (uint64_t)d) + *oc = OP_SeekGT; + return mem_convert_to_double(mem); + } + if (*oc == OP_SeekLT) { + if (d != (double)UINT64_MAX && u > (uint64_t)d) + *oc = OP_SeekLE; + return mem_convert_to_double(mem); + } + assert(*oc == OP_SeekLE); + if (d == (double)UINT64_MAX || u < (uint64_t)d) + *oc = OP_SeekLT; + return mem_convert_to_double(mem); +} + +/** + * Convert the numeric value contained in the MEM to another + * numeric type according to the specified operation. If the + * conversion is successful, we will get the converted MEM. If the + * conversion fails, the MEM will not be changed. + * + * There are two reasons why the MEM might not convert. + * 1) MEM conversion affects the result of the operation. + * For example: + * CREATE TABLE t (i INT PRIMARY KEY); + * ... + * SELECT * FROM t WHERE i = 1.5; + * + * 2) After conversion, nothing will be found as a result of the + * operation. + * For example: + * CREATE TABLE t (i INT PRIMARY KEY); + * ... + * SELECT * FROM t WHERE i > 2^100; + * + * + * If the conversion is successful, the operation can also change. + * For example: + * CREATE TABLE t (i INT PRIMARY KEY); + * ... + * SELECT * FROM t WHERE i > -(2^100); + * + * The value becomes INT64_MIN after conversion and the operation + * becomes '> ='. + * + * @param mem The MEM that contains the numeric value. + * @param type The type to convert to. + * @param[in][out] oc Operation code. + * @retval 0 if the conversion was successful, -1 otherwise. + */ +static int +mem_prepare_for_cmp(struct Mem *mem, enum field_type type, int *oc, int eqOnly) +{ + if (type == FIELD_TYPE_UNSIGNED) + return mem_prepare_for_cmp_to_uint(mem, oc, eqOnly); + if (type == FIELD_TYPE_INTEGER) + return mem_prepare_for_cmp_to_int(mem, oc, eqOnly); + assert(type == FIELD_TYPE_DOUBLE); + return mem_prepare_for_cmp_to_double(mem, oc, eqOnly); +} + /* * pMem currently only holds a string type (or maybe a BLOB that we can * interpret as a string if we want to). Compute its corresponding @@ -3491,6 +3753,74 @@ case OP_SeekGT: { /* jump, in3 */ assert(oc!=OP_SeekLT || r.default_rc==+1); r.aMem = &aMem[pOp->p3]; + bool is_on_region = false; + struct region *region = &fiber()->gc; + size_t region_svp = 0; + for (int i = 0; i < r.nField; ++i) { + struct Mem *mem = &r.aMem[i]; + enum field_type type = r.key_def->parts[i].type; + /* + * We already know that MEM_type and field type + * are comparable. If they are not compatible, we + * should try to convert MEM to field type. + */ + if (!mem_is_type_compatible(mem, type)) { + /* + * We cannot make changes to original MEMs + * since they will be used in OP_Idx*. So + * we should copy them and make changes to + * the copies. + */ + if (!is_on_region) { + region_svp = region_used(region); + uint32_t size = sizeof(struct Mem) * r.nField; + r.aMem = region_aligned_alloc(region, size, + alignof(struct Mem)); + if (r.aMem == NULL) { + diag_set(OutOfMemory, size, + "region_aligned_alloc", + "r.aMem"); + goto abort_due_to_error; + } + memcpy(r.aMem, &aMem[pOp->p3], size); + is_on_region = true; + mem = &r.aMem[i]; + } + /* + * In cases where we can change the MEM + * according to the field type and opcode, + * we will get the converted MEM after + * this function. + * + * There are two cases where MEM does not + * change: + * 1) any resultof conversion can affect + * the result of the operation; + * 2) after conversion nothing will be + * found as a result of the operation. + * + * Examples can be found in description of + * the function. + */ + if (mem_prepare_for_cmp(mem, type, &oc, eqOnly) != 0) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(mem), + field_type_strs[type]); + goto abort_due_to_error; + } + /* + * if the MEM type and the field type are + * still not compatible, then the + * conversion failed and we won't find + * anything. + */ + if (!mem_is_type_compatible(mem, type)) { + res = 1; + goto seek_not_found; + } + } + } + #ifdef SQL_DEBUG { int i; for(i=0; ip2>0); VdbeBranchTaken(res!=0,2); if (res) { -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:53 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:53 +0300 Subject: [Tarantool-patches] [PATCH v5 3/6] sql: change comparison between numbers using index In-Reply-To: References: Message-ID: <96a8dec05827608e00503fd33d2cb0d2a1076052.1598000242.git.imeevma@gmail.com> This patch disables number conversions in ApplyType in wherecode.c. This allows conversions between numbers introduced in previous commit to be used. Part of #4230 --- src/box/sql/sqlInt.h | 2 + src/box/sql/vdbe.c | 3 +- src/box/sql/wherecode.c | 76 +--------------------------- test/sql-tap/in4.test.lua | 4 +- test/sql-tap/join.test.lua | 4 +- test/sql-tap/tkt-9a8b09f8e6.test.lua | 8 +-- test/sql/types.result | 54 ++++++++++++++++++++ test/sql/types.test.lua | 12 +++++ 8 files changed, 79 insertions(+), 84 deletions(-) diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index adf90d824..1e6f0f41f 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -2309,6 +2309,8 @@ struct Parse { #define OPFLAG_SYSTEMSP 0x20 /* OP_Open**: set if space pointer * points to system space. */ +/** OP_ApplyType: Do not convert numbers. */ +#define OPFLAG_DO_NOT_CONVERT_NUMBERS 0x01 /** * Prepare vdbe P5 flags for OP_{IdxInsert, IdxReplace, Update} diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 822d7e177..a377ceae7 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -3137,7 +3137,8 @@ case OP_ApplyType: { if (!mp_type_is_numeric(mem_mp_type(pIn1))) goto type_mismatch; /* Try to convert numeric-to-numeric. */ - if (mem_convert_to_numeric(pIn1, type) != 0) + if ((pOp->p5 & OPFLAG_DO_NOT_CONVERT_NUMBERS) == 0 && + mem_convert_to_numeric(pIn1, type) != 0) goto type_mismatch; } pIn1++; diff --git a/src/box/sql/wherecode.c b/src/box/sql/wherecode.c index 97ba59151..78d580801 100644 --- a/src/box/sql/wherecode.c +++ b/src/box/sql/wherecode.c @@ -374,6 +374,7 @@ emit_apply_type(Parse *pParse, int base, int n, enum field_type *types) types, n); sqlVdbeAddOp4(v, OP_ApplyType, base, n, 0, (char *) types_dup, P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); sql_expr_type_cache_change(pParse, base, n); } } @@ -1045,77 +1046,6 @@ sqlWhereCodeOneLoopStart(WhereInfo * pWInfo, /* Complete information about the W } struct index_def *idx_pk = space->index[0]->def; uint32_t pk_part_count = idx_pk->key_def->part_count; - /* - * Tarantool's iterator over integer fields doesn't - * tolerate floating point values. Hence, if term - * is equality comparison and value of operand is - * not integer, we can skip it since it always - * results in false: INT a == 0.5 -> false; - * It is done using OP_MustBeInt facilities. - * In case term is greater comparison (a > ?), we - * should notify OP_SeekGT to process truncation of - * floating point value: a > 0.5 -> a >= 1; - * It is done by setting P5 flag for OP_Seek*. - * It is worth mentioning that we do not need - * this step when it comes for less (<) comparison - * of nullable field. Key is NULL in this case: - * values are ordered as NULL, ... NULL, min_value, - * so to fetch min value we pass NULL to GT iterator. - * The only exception is less comparison in - * conjunction with ORDER BY DESC clause: - * in such situation we use LE iterator and - * truncated value to compare. But then - * pRangeStart == NULL. - * This procedure is correct for compound index: - * only one comparison of less/greater type can be - * used at the same time. For instance, - * a < 1.5 AND b > 0.5 is handled by SeekGT using - * column a and fetching column b from tuple and - * OP_Le comparison. - * - * Note that OP_ApplyType, which is emitted before - * OP_Seek** doesn't truncate floating point to - * integer. That's why we need this routine. - * Also, note that terms are separated by OR - * predicates, so we consider term as sequence - * of AND'ed predicates. - */ - size_t addrs_sz; - int *seek_addrs = region_alloc_array(&pParse->region, - typeof(seek_addrs[0]), nEq, - &addrs_sz); - if (seek_addrs == NULL) { - diag_set(OutOfMemory, addrs_sz, "region_alloc_array", - "seek_addrs"); - pParse->is_aborted = true; - return 0; - } - memset(seek_addrs, 0, addrs_sz); - for (int i = 0; i < nEq; i++) { - enum field_type type = idx_def->key_def->parts[i].type; - if (type == FIELD_TYPE_INTEGER || - type == FIELD_TYPE_UNSIGNED) { - /* - * OP_MustBeInt consider NULLs as - * non-integer values, so firstly - * check whether value is NULL or not. - */ - seek_addrs[i] = sqlVdbeAddOp1(v, OP_IsNull, - regBase); - sqlVdbeAddOp2(v, OP_MustBeInt, regBase + i, - addrNxt); - start_types[i] = FIELD_TYPE_SCALAR; - /* - * We need to notify column cache - * that type of value may change - * so we should fetch value from - * tuple again rather then copy - * from register. - */ - sql_expr_type_cache_change(pParse, regBase + i, - 1); - } - } emit_apply_type(pParse, regBase, nConstraint - bSeekPastNull, start_types); if (pLoop->nSkip > 0 && nConstraint == pLoop->nSkip) { @@ -1127,10 +1057,6 @@ sqlWhereCodeOneLoopStart(WhereInfo * pWInfo, /* Complete information about the W op = aStartOp[(start_constraints << 2) + (startEq << 1) + bRev]; assert(op != 0); - for (uint32_t i = 0; i < nEq; ++i) { - if (seek_addrs[i] != 0) - sqlVdbeJumpHere(v, seek_addrs[i]); - } sqlVdbeAddOp4Int(v, op, iIdxCur, addrNxt, regBase, nConstraint); VdbeCoverage(v); diff --git a/test/sql-tap/in4.test.lua b/test/sql-tap/in4.test.lua index 5c01ccdab..e0bf671d9 100755 --- a/test/sql-tap/in4.test.lua +++ b/test/sql-tap/in4.test.lua @@ -147,13 +147,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "in4-2.8", [[ SELECT b FROM t2 WHERE a IN ('', '0.0.0', '2') ]], { -- - "two" + 1, "Type mismatch: can not convert to integer" -- }) diff --git a/test/sql-tap/join.test.lua b/test/sql-tap/join.test.lua index 840b780a3..7a1346094 100755 --- a/test/sql-tap/join.test.lua +++ b/test/sql-tap/join.test.lua @@ -1028,13 +1028,13 @@ test:do_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "join-11.9", [[ SELECT * FROM t1 NATURAL JOIN t2 ]], { -- - "one", "1", "two", "2" + 1, "Type mismatch: can not convert 1 to integer" -- }) diff --git a/test/sql-tap/tkt-9a8b09f8e6.test.lua b/test/sql-tap/tkt-9a8b09f8e6.test.lua index ca3a5427a..1314d0aad 100755 --- a/test/sql-tap/tkt-9a8b09f8e6.test.lua +++ b/test/sql-tap/tkt-9a8b09f8e6.test.lua @@ -183,13 +183,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( 3.3, [[ SELECT x FROM t2 WHERE x IN ('1'); ]], { -- <3.3> - 1 + 1, "Type mismatch: can not convert 1 to integer" -- }) @@ -213,13 +213,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( 3.7, [[ SELECT x FROM t2 WHERE '1' IN (x); ]], { -- <3.7> - 1 + 1, "Type mismatch: can not convert 1 to integer" -- }) diff --git a/test/sql/types.result b/test/sql/types.result index 442245186..8810a9f82 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2795,3 +2795,57 @@ box.execute([[DROP TABLE ts;]]) --- - row_count: 1 ... +-- +-- gh-4230: Make sure the comparison between numbers that use +-- index is working correctly. +-- +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, a DOUBLE);]]) +--- +- row_count: 1 +... +box.execute([[INSERT INTO t VALUES (1, ?);]], {2^60}) +--- +- row_count: 1 +... +box.execute([[SELECT * FROM t WHERE i > ?]], {2^70}) +--- +- metadata: + - name: I + type: integer + - name: A + type: double + rows: [] +... +box.execute([[SELECT * FROM t WHERE i > ?]], {-2^70}) +--- +- metadata: + - name: I + type: integer + - name: A + type: double + rows: + - [1, 1152921504606846976] +... +box.execute([[SELECT * FROM t WHERE a = ?]], {2ULL^60ULL - 1ULL}) +--- +- metadata: + - name: I + type: integer + - name: A + type: double + rows: [] +... +box.execute([[SELECT * FROM t WHERE a > ?]], {2ULL^60ULL - 1ULL}) +--- +- metadata: + - name: I + type: integer + - name: A + type: double + rows: + - [1, 1152921504606846976] +... +box.execute([[DROP TABLE t;]]) +--- +- row_count: 1 +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 0270d9f8a..a23b12801 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -623,3 +623,15 @@ box.execute([[DROP TABLE tb;]]) box.execute([[DROP TABLE tt;]]) box.execute([[DROP TABLE tv;]]) box.execute([[DROP TABLE ts;]]) + +-- +-- gh-4230: Make sure the comparison between numbers that use +-- index is working correctly. +-- +box.execute([[CREATE TABLE t (i INTEGER PRIMARY KEY, a DOUBLE);]]) +box.execute([[INSERT INTO t VALUES (1, ?);]], {2^60}) +box.execute([[SELECT * FROM t WHERE i > ?]], {2^70}) +box.execute([[SELECT * FROM t WHERE i > ?]], {-2^70}) +box.execute([[SELECT * FROM t WHERE a = ?]], {2ULL^60ULL - 1ULL}) +box.execute([[SELECT * FROM t WHERE a > ?]], {2ULL^60ULL - 1ULL}) +box.execute([[DROP TABLE t;]]) -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:54 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:54 +0300 Subject: [Tarantool-patches] [PATCH v5 4/6] sql: remove implicit cast from comparison opcodes In-Reply-To: References: Message-ID: This patch removes implicit casting from STRING to number and vice versa from comparison opcodes. Part of #4230 --- src/box/sql/vdbe.c | 52 ++- test/sql-tap/identifier_case.test.lua | 6 +- test/sql-tap/in1.test.lua | 4 +- test/sql-tap/insert3.test.lua | 2 +- test/sql-tap/join.test.lua | 4 +- test/sql-tap/misc1.test.lua | 32 +- test/sql-tap/select1.test.lua | 4 +- test/sql-tap/select7.test.lua | 2 +- test/sql-tap/subquery.test.lua | 4 +- test/sql-tap/tkt-9a8b09f8e6.test.lua | 508 -------------------------- test/sql-tap/tkt3493.test.lua | 40 +- test/sql-tap/transitive1.test.lua | 12 +- test/sql-tap/where2.test.lua | 183 +--------- test/sql-tap/where5.test.lua | 12 +- test/sql/types.result | 7 +- 15 files changed, 86 insertions(+), 786 deletions(-) delete mode 100755 test/sql-tap/tkt-9a8b09f8e6.test.lua diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index a377ceae7..b326c4ba4 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -2577,22 +2577,17 @@ case OP_Ge: { /* same as TK_GE, jump, in1, in3 */ } else { enum field_type type = pOp->p5 & FIELD_TYPE_MASK; if (sql_type_is_numeric(type)) { - if ((flags1 | flags3)&MEM_Str) { - if ((flags1 & MEM_Str) == MEM_Str) { - mem_apply_numeric_type(pIn1); - testcase( flags3!=pIn3->flags); /* Possible if pIn1==pIn3 */ - flags3 = pIn3->flags; - } - if ((flags3 & MEM_Str) == MEM_Str) { - if (mem_apply_numeric_type(pIn3) != 0) { - diag_set(ClientError, - ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(pIn3), - "numeric"); - goto abort_due_to_error; - } - - } + if ((flags1 & MEM_Str) == MEM_Str) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(pIn1), + "numeric"); + goto abort_due_to_error; + } + if ((flags3 & MEM_Str) == MEM_Str) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(pIn3), + "numeric"); + goto abort_due_to_error; } /* Handle the common case of integer comparison here, as an * optimization, to avoid a call to sqlMemCompare() @@ -2625,22 +2620,17 @@ case OP_Ge: { /* same as TK_GE, jump, in1, in3 */ goto compare_op; } } else if (type == FIELD_TYPE_STRING) { - if ((flags1 & MEM_Str) == 0 && - (flags1 & (MEM_Int | MEM_UInt | MEM_Real)) != 0) { - testcase( pIn1->flags & MEM_Int); - testcase( pIn1->flags & MEM_Real); - sqlVdbeMemStringify(pIn1); - testcase( (flags1&MEM_Dyn) != (pIn1->flags&MEM_Dyn)); - flags1 = (pIn1->flags & ~MEM_TypeMask) | (flags1 & MEM_TypeMask); - assert(pIn1!=pIn3); + if ((flags1 & MEM_Str) == 0) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + mem_type_to_str(pIn3), + mem_type_to_str(pIn1)); + goto abort_due_to_error; } - if ((flags3 & MEM_Str) == 0 && - (flags3 & (MEM_Int | MEM_UInt | MEM_Real)) != 0) { - testcase( pIn3->flags & MEM_Int); - testcase( pIn3->flags & MEM_Real); - sqlVdbeMemStringify(pIn3); - testcase( (flags3&MEM_Dyn) != (pIn3->flags&MEM_Dyn)); - flags3 = (pIn3->flags & ~MEM_TypeMask) | (flags3 & MEM_TypeMask); + if ((flags3 & MEM_Str) == 0) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + mem_type_to_str(pIn1), + mem_type_to_str(pIn3)); + goto abort_due_to_error; } } assert(pOp->p4type==P4_COLLSEQ || pOp->p4.pColl==0); diff --git a/test/sql-tap/identifier_case.test.lua b/test/sql-tap/identifier_case.test.lua index 2a00626fc..1d56ffb44 100755 --- a/test/sql-tap/identifier_case.test.lua +++ b/test/sql-tap/identifier_case.test.lua @@ -242,11 +242,11 @@ data = { { 2, [[ 'a' < 'b' collate "binary" ]], {0, {true}}}, { 3, [[ 'a' < 'b' collate 'binary' ]], {1, [[Syntax error at line 1 near ''binary'']]}}, { 4, [[ 'a' < 'b' collate "unicode" ]], {0, {true}}}, - { 5, [[ 5 < 'b' collate "unicode" ]], {0, {true}}}, + { 5, [[ 5 < 'b' collate "unicode" ]], {1, "Type mismatch: can not convert b to numeric"}}, { 6, [[ 5 < 'b' collate unicode ]], {1,"Collation 'UNICODE' does not exist"}}, - { 7, [[ 5 < 'b' collate "unicode_ci" ]], {0, {true}}}, + { 7, [[ 5 < 'b' collate "unicode_ci" ]], {1, "Type mismatch: can not convert b to numeric"}}, { 8, [[ 5 < 'b' collate NONE ]], {1, "Collation 'NONE' does not exist"}}, - { 9, [[ 5 < 'b' collate "none" ]], {0, {true}}}, + { 9, [[ 5 < 'b' collate "none" ]], {1, "Type mismatch: can not convert b to numeric"}}, } for _, row in ipairs(data) do diff --git a/test/sql-tap/in1.test.lua b/test/sql-tap/in1.test.lua index 570cc1779..e2f498889 100755 --- a/test/sql-tap/in1.test.lua +++ b/test/sql-tap/in1.test.lua @@ -637,12 +637,12 @@ test:do_test( "in-11.2", function() -- The '2' should be coerced into 2 because t6.b is NUMERIC - return test:execsql [[ + return test:catchsql [[ SELECT * FROM t6 WHERE b IN ('2'); ]] end, { -- - 1, 2 + 1, "Type mismatch: can not convert 2 to numeric" -- }) diff --git a/test/sql-tap/insert3.test.lua b/test/sql-tap/insert3.test.lua index b92bc508e..3276f0db2 100755 --- a/test/sql-tap/insert3.test.lua +++ b/test/sql-tap/insert3.test.lua @@ -59,7 +59,7 @@ test:do_execsql_test( [[ CREATE TABLE log2(rowid INTEGER PRIMARY KEY AUTOINCREMENT, x TEXT UNIQUE,y INT ); CREATE TRIGGER r2 BEFORE INSERT ON t1 FOR EACH ROW BEGIN - UPDATE log2 SET y=y+1 WHERE x=new.b; + UPDATE log2 SET y=y+1 WHERE x=CAST(new.b AS STRING); INSERT OR IGNORE INTO log2(x, y) VALUES(CAST(new.b AS STRING),1); END; INSERT INTO t1(a, b) VALUES('hi', 453); diff --git a/test/sql-tap/join.test.lua b/test/sql-tap/join.test.lua index 7a1346094..a78913836 100755 --- a/test/sql-tap/join.test.lua +++ b/test/sql-tap/join.test.lua @@ -1038,13 +1038,13 @@ test:do_catchsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "join-11.10", [[ SELECT * FROM t2 NATURAL JOIN t1 ]], { -- - 1, "one", 2, "two" + 1, "Type mismatch: can not convert 1 to numeric" -- }) diff --git a/test/sql-tap/misc1.test.lua b/test/sql-tap/misc1.test.lua index c0136d04c..66666878e 100755 --- a/test/sql-tap/misc1.test.lua +++ b/test/sql-tap/misc1.test.lua @@ -88,7 +88,7 @@ test:do_execsql_test( test:do_execsql_test( "misc1-1.4", [[ - SELECT x75 FROM manycol WHERE x50=350 + SELECT x75 FROM manycol WHERE x50='350' ]], { -- "375" @@ -98,7 +98,7 @@ test:do_execsql_test( test:do_execsql_test( "misc1-1.5", [[ - SELECT x50 FROM manycol WHERE x99=599 + SELECT x50 FROM manycol WHERE x99='599' ]], { -- "550" @@ -109,7 +109,7 @@ test:do_test( "misc1-1.6", function() test:execsql("CREATE INDEX manycol_idx1 ON manycol(x99)") - return test:execsql("SELECT x50 FROM manycol WHERE x99=899") + return test:execsql("SELECT x50 FROM manycol WHERE x99='899'") end, { -- "850" @@ -129,7 +129,7 @@ test:do_execsql_test( test:do_test( "misc1-1.8", function() - test:execsql("DELETE FROM manycol WHERE x98=1234") + test:execsql("DELETE FROM manycol WHERE x98='1234'") return test:execsql("SELECT count(*) FROM manycol") end, { -- @@ -140,7 +140,7 @@ test:do_test( test:do_test( "misc1-1.9", function() - test:execsql("DELETE FROM manycol WHERE x98=998") + test:execsql("DELETE FROM manycol WHERE x98='998'") return test:execsql("SELECT count(*) FROM manycol") end, { -- @@ -151,7 +151,7 @@ test:do_test( test:do_test( "misc1-1.10", function() - test:execsql("DELETE FROM manycol WHERE x99=500") + test:execsql("DELETE FROM manycol WHERE x99='500'") return test:execsql("SELECT count(*) FROM manycol") end, { -- @@ -162,7 +162,7 @@ test:do_test( test:do_test( "misc1-1.11", function() - test:execsql("DELETE FROM manycol WHERE x99=599") + test:execsql("DELETE FROM manycol WHERE x99='599'") return test:execsql("SELECT count(*) FROM manycol") end, { -- @@ -479,9 +479,9 @@ local where = "" test:do_test( "misc1-10.1", function() - where = "WHERE x0>=0" + where = "WHERE x0>='0'" for i = 1, 99, 1 do - where = where .. " AND x"..i.."<>0" + where = where .. " AND x"..i.."<>'0'" end return test:catchsql("SELECT count(*) FROM manycol "..where.."") end, { @@ -496,7 +496,7 @@ test:do_test( test:do_test( "misc1-10.3", function() - where = string.gsub(where,"x0>=0", "x0=0") + where = string.gsub(where,"x0>='0'", "x0='0'") return test:catchsql("DELETE FROM manycol "..where.."") end, { -- @@ -520,7 +520,7 @@ test:do_execsql_test( test:do_execsql_test( "misc1-10.6", [[ - SELECT x1 FROM manycol WHERE x0=100 + SELECT x1 FROM manycol WHERE x0='100' ]], { -- "101" @@ -530,7 +530,7 @@ test:do_execsql_test( test:do_test( "misc1-10.7", function() - where = string.gsub(where, "x0=0", "x0=100") + where = string.gsub(where, "x0='0'", "x0='100'") return test:catchsql("UPDATE manycol SET x1=CAST(x1+1 AS STRING) "..where.."") end, { -- @@ -541,7 +541,7 @@ test:do_test( test:do_execsql_test( "misc1-10.8", [[ - SELECT x1 FROM manycol WHERE x0=100 + SELECT x1 FROM manycol WHERE x0='100' ]], { -- "102" @@ -563,7 +563,7 @@ test:do_execsql_test( test:do_execsql_test( "misc1-10.10", [[ - SELECT x1 FROM manycol WHERE x0=100 + SELECT x1 FROM manycol WHERE x0='100' ]], { -- "103" @@ -619,13 +619,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "misc1-12.2", [[ SELECT '0'==0.0 ]], { -- - true + 1, "Type mismatch: can not convert 0 to numeric" -- }) diff --git a/test/sql-tap/select1.test.lua b/test/sql-tap/select1.test.lua index d8be38da6..8c91b4d31 100755 --- a/test/sql-tap/select1.test.lua +++ b/test/sql-tap/select1.test.lua @@ -1913,7 +1913,7 @@ test:do_execsql_test( test:do_execsql_test( "select1-12.7", [[ - SELECT * FROM t3 WHERE a=(SELECT 1); + SELECT * FROM t3 WHERE a=(SELECT '1'); ]], { -- 0, "1", "2" @@ -1923,7 +1923,7 @@ test:do_execsql_test( test:do_execsql_test( "select1-12.8", [[ - SELECT * FROM t3 WHERE a=(SELECT 2); + SELECT * FROM t3 WHERE a=(SELECT '2'); ]], { -- diff --git a/test/sql-tap/select7.test.lua b/test/sql-tap/select7.test.lua index e1e43c557..0d1390fd6 100755 --- a/test/sql-tap/select7.test.lua +++ b/test/sql-tap/select7.test.lua @@ -256,7 +256,7 @@ test:do_execsql_test( DROP TABLE IF EXISTS t5; CREATE TABLE t5(a TEXT primary key, b INT); INSERT INTO t5 VALUES('123', 456); - SELECT typeof(a), a FROM t5 GROUP BY a HAVING a "string", "123" diff --git a/test/sql-tap/subquery.test.lua b/test/sql-tap/subquery.test.lua index e0771825e..bad702de9 100755 --- a/test/sql-tap/subquery.test.lua +++ b/test/sql-tap/subquery.test.lua @@ -284,13 +284,13 @@ test:do_execsql_test( -- }) -test:do_execsql_test( +test:do_catchsql_test( "subquery-2.3.2", [[ SELECT a IN (10.0, 20) FROM t3; ]], { -- - false + 1, "Type mismatch: can not convert text to real" -- }) diff --git a/test/sql-tap/tkt-9a8b09f8e6.test.lua b/test/sql-tap/tkt-9a8b09f8e6.test.lua deleted file mode 100755 index 1314d0aad..000000000 --- a/test/sql-tap/tkt-9a8b09f8e6.test.lua +++ /dev/null @@ -1,508 +0,0 @@ -#!/usr/bin/env tarantool -test = require("sqltester") -test:plan(47) - ---!./tcltestrunner.lua --- 2014 June 26 --- --- The author disclaims copyright to this source code. In place of --- a legal notice, here is a blessing: --- --- May you do good and not evil. --- May you find forgiveness for yourself and forgive others. --- May you share freely, never taking more than you give. --- -------------------------------------------------------------------------- --- This file implements regression tests for sql library. --- --- This file implements tests to verify that ticket [9a8b09f8e6] has been --- fixed. --- --- ["set","testdir",[["file","dirname",["argv0"]]]] --- ["source",[["testdir"],"\/tester.tcl"]] -testprefix = "tkt-9a8b09f8e6" --- MUST_WORK_TEST -if (0 > 0) - then -end -test:do_execsql_test( - 1.1, - [[ - CREATE TABLE t1(x TEXT primary key); - INSERT INTO t1 VALUES('1'); - ]], { - -- <1.1> - - -- - }) - -test:do_execsql_test( - 1.2, - [[ - CREATE TABLE t2(x INTEGER primary key); - INSERT INTO t2 VALUES(1); - ]], { - -- <1.2> - - -- - }) - -test:do_execsql_test( - 1.3, - [[ - CREATE TABLE t3(x NUMBER primary key); - INSERT INTO t3 VALUES(1.0); - ]], { - -- <1.3> - - -- - }) - -test:do_execsql_test( - 1.4, - [[ - CREATE TABLE t4(x NUMBER primary key); - INSERT INTO t4 VALUES(1.11); - ]], { - -- <1.4> - - -- - }) - -test:do_execsql_test( - 1.5, - [[ - CREATE TABLE t5(id INT primary key, x INT , y TEXT); - INSERT INTO t5 VALUES(1, 1, 'one'); - INSERT INTO t5 VALUES(2, 1, 'two'); - INSERT INTO t5 VALUES(3, 1.0, 'three'); - INSERT INTO t5 VALUES(4, 1.0, 'four'); - ]], { - -- <1.5> - - -- - }) - -test:do_execsql_test( - 2.1, - [[ - SELECT x FROM t1 WHERE x IN (1); - ]], { - -- <2.1> - "1" - -- - }) - -test:do_execsql_test( - 2.2, - [[ - SELECT x FROM t1 WHERE x IN (1.0); - ]], { - -- <2.2> - "1" - -- - }) - -test:do_execsql_test( - 2.3, - [[ - SELECT x FROM t1 WHERE x IN ('1'); - ]], { - -- <2.3> - "1" - -- - }) - -test:do_execsql_test( - 2.4, - [[ - SELECT x FROM t1 WHERE x IN ('1.0'); - ]], { - -- <2.4> - - -- - }) - -test:do_execsql_test( - 2.5, - [[ - SELECT x FROM t1 WHERE 1 IN (x); - ]], { - -- <2.5> - "1" - -- - }) - -test:do_execsql_test( - 2.6, - [[ - SELECT x FROM t1 WHERE 1.0 IN (x); - ]], { - -- <2.6> - "1" - -- - }) - -test:do_execsql_test( - 2.7, - [[ - SELECT x FROM t1 WHERE '1' IN (x); - ]], { - -- <2.7> - "1" - -- - }) - -test:do_execsql_test( - 2.8, - [[ - SELECT x FROM t1 WHERE '1.0' IN (x); - ]], { - -- <2.8> - - -- - }) - -test:do_execsql_test( - 3.1, - [[ - SELECT x FROM t2 WHERE x IN (1); - ]], { - -- <3.1> - 1 - -- - }) - -test:do_execsql_test( - 3.2, - [[ - SELECT x FROM t2 WHERE x IN (1.0); - ]], { - -- <3.2> - 1 - -- - }) - -test:do_catchsql_test( - 3.3, - [[ - SELECT x FROM t2 WHERE x IN ('1'); - ]], { - -- <3.3> - 1, "Type mismatch: can not convert 1 to integer" - -- - }) - -test:do_execsql_test( - 3.5, - [[ - SELECT x FROM t2 WHERE 1 IN (x); - ]], { - -- <3.5> - 1 - -- - }) - -test:do_execsql_test( - 3.6, - [[ - SELECT x FROM t2 WHERE 1.0 IN (x); - ]], { - -- <3.6> - 1 - -- - }) - -test:do_catchsql_test( - 3.7, - [[ - SELECT x FROM t2 WHERE '1' IN (x); - ]], { - -- <3.7> - 1, "Type mismatch: can not convert 1 to integer" - -- - }) - -test:do_execsql_test( - 4.1, - [[ - SELECT x FROM t3 WHERE x IN (1); - ]], { - -- <4.1> - 1.0 - -- - }) - -test:do_execsql_test( - 4.2, - [[ - SELECT x FROM t3 WHERE x IN (1.0); - ]], { - -- <4.2> - 1.0 - -- - }) - -test:do_catchsql_test( - 4.3, - [[ - SELECT x FROM t3 WHERE x IN ('1'); - ]], { - -- <4.3> - 1, "Type mismatch: can not convert 1 to number" - -- - }) - -test:do_catchsql_test( - 4.4, - [[ - SELECT x FROM t3 WHERE x IN ('1.0'); - ]], { - -- <4.4> - 1, "Type mismatch: can not convert 1.0 to number" - -- - }) - -test:do_execsql_test( - 4.5, - [[ - SELECT x FROM t3 WHERE 1 IN (x); - ]], { - -- <4.5> - 1.0 - -- - }) - -test:do_execsql_test( - 4.6, - [[ - SELECT x FROM t3 WHERE 1.0 IN (x); - ]], { - -- <4.6> - 1.0 - -- - }) - -test:do_catchsql_test( - 4.7, - [[ - SELECT x FROM t3 WHERE '1' IN (x); - ]], { - -- <4.7> - 1, "Type mismatch: can not convert 1 to number" - -- - }) - -test:do_catchsql_test( - 4.8, - [[ - SELECT x FROM t3 WHERE '1.0' IN (x); - ]], { - -- <4.8> - 1, "Type mismatch: can not convert 1.0 to number" - -- - }) - -test:do_execsql_test( - 5.1, - [[ - SELECT x FROM t4 WHERE x IN (1); - ]], { - -- <5.1> - - -- - }) - -test:do_execsql_test( - 5.2, - [[ - SELECT x FROM t4 WHERE x IN (1.0); - ]], { - -- <5.2> - - -- - }) - -test:do_catchsql_test( - 5.3, - [[ - SELECT x FROM t4 WHERE x IN ('1'); - ]], { - -- <5.3> - 1, "Type mismatch: can not convert 1 to number" - -- - }) - -test:do_catchsql_test( - 5.4, - [[ - SELECT x FROM t4 WHERE x IN ('1.0'); - ]], { - -- <5.4> - 1, "Type mismatch: can not convert 1.0 to number" - -- - }) - -test:do_execsql_test( - 5.5, - [[ - SELECT x FROM t4 WHERE x IN (1.11); - ]], { - -- <5.5> - 1.11 - -- - }) - -test:do_catchsql_test( - 5.6, - [[ - SELECT x FROM t4 WHERE x IN ('1.11'); - ]], { - -- <5.6> - 1, "Type mismatch: can not convert 1.11 to number" - -- - }) - -test:do_execsql_test( - 5.7, - [[ - SELECT x FROM t4 WHERE 1 IN (x); - ]], { - -- <5.7> - - -- - }) - -test:do_execsql_test( - 5.8, - [[ - SELECT x FROM t4 WHERE 1.0 IN (x); - ]], { - -- <5.8> - - -- - }) - -test:do_catchsql_test( - 5.9, - [[ - SELECT x FROM t4 WHERE '1' IN (x); - ]], { - -- <5.9> - 1, "Type mismatch: can not convert 1 to number" - -- - }) - -test:do_catchsql_test( - 5.10, - [[ - SELECT x FROM t4 WHERE '1.0' IN (x); - ]], { - -- <5.10> - 1, "Type mismatch: can not convert 1.0 to number" - -- - }) - -test:do_execsql_test( - 5.11, - [[ - SELECT x FROM t4 WHERE 1.11 IN (x); - ]], { - -- <5.11> - 1.11 - -- - }) - -test:do_catchsql_test( - 5.12, - [[ - SELECT x FROM t4 WHERE '1.11' IN (x); - ]], { - -- <5.12> - 1, "Type mismatch: can not convert 1.11 to number" - -- - }) - -test:do_execsql_test( - 6.1, - [[ - SELECT x, y FROM t5 WHERE x IN (1); - ]], { - -- <6.1> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.2, - [[ - SELECT x, y FROM t5 WHERE x IN (1.0); - ]], { - -- <6.2> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.3, - [[ - SELECT x, y FROM t5 WHERE x IN ('1'); - ]], { - -- <6.3> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.4, - [[ - SELECT x, y FROM t5 WHERE x IN ('1.0'); - ]], { - -- <6.4> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.5, - [[ - SELECT x, y FROM t5 WHERE 1 IN (x); - ]], { - -- <6.5> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.6, - [[ - SELECT x, y FROM t5 WHERE 1.0 IN (x); - ]], { - -- <6.6> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.7, - [[ - SELECT x, y FROM t5 WHERE '1' IN (x); - ]], { - -- <6.7> - 1, "one", 1, "two", 1, "three", 1.0, "four" - -- - }) - -test:do_execsql_test( - 6.8, - [[ - SELECT x, y FROM t5 WHERE '1.0' IN (x); - ]], { - -- <6.8> - 1, "one", 1, "two", 1, "three", 1, "four" - -- - }) - - - -test:finish_test() diff --git a/test/sql-tap/tkt3493.test.lua b/test/sql-tap/tkt3493.test.lua index de77e61e9..82ba828d0 100755 --- a/test/sql-tap/tkt3493.test.lua +++ b/test/sql-tap/tkt3493.test.lua @@ -1,6 +1,6 @@ #!/usr/bin/env tarantool test = require("sqltester") -test:plan(26) +test:plan(25) --!./tcltestrunner.lua -- 2008 October 13 @@ -45,7 +45,7 @@ test:do_execsql_test( [[ SELECT CASE - WHEN B.val = 1 THEN 'XYZ' + WHEN B.val = '1' THEN 'XYZ' ELSE A.val END AS Col1 FROM B @@ -63,7 +63,7 @@ test:do_execsql_test( [[ SELECT DISTINCT CASE - WHEN B.val = 1 THEN 'XYZ' + WHEN B.val = '1' THEN 'XYZ' ELSE A.val END AS Col1 FROM B @@ -79,7 +79,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-1.4", [[ - SELECT b.val, CASE WHEN b.val = 1 THEN 'xyz' ELSE b.val END AS col1 FROM b; + SELECT b.val, CASE WHEN b.val = '1' THEN 'xyz' ELSE b.val END AS col1 FROM b; ]], { -- "1", "xyz", "2", "2" @@ -91,7 +91,7 @@ test:do_execsql_test( [[ SELECT DISTINCT b.val, - CASE WHEN b.val = 1 THEN 'xyz' ELSE b.val END AS col1 + CASE WHEN b.val = '1' THEN 'xyz' ELSE b.val END AS col1 FROM b; ]], { -- @@ -126,23 +126,13 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.2.1", [[ - SELECT a=123 FROM t1 GROUP BY a + SELECT a='123' FROM t1 GROUP BY a ]], { -- true -- }) -test:do_execsql_test( - "tkt3493-2.2.2", - [[ - SELECT a=123 FROM t1 - ]], { - -- - true - -- - }) - test:do_execsql_test( "tkt3493-2.2.3", [[ @@ -156,7 +146,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.2.4", [[ - SELECT count(*), a=123 FROM t1 + SELECT count(*), a='123' FROM t1 ]], { -- 1, true @@ -166,7 +156,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.2.5", [[ - SELECT count(*), +a=123 FROM t1 + SELECT count(*), +a='123' FROM t1 ]], { -- 1, true @@ -176,7 +166,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.3.3", [[ - SELECT b='456' FROM t1 GROUP BY a + SELECT b = 456 FROM t1 GROUP BY a ]], { -- true @@ -186,7 +176,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.3.1", [[ - SELECT b='456' FROM t1 GROUP BY b + SELECT b = 456 FROM t1 GROUP BY b ]], { -- true @@ -196,7 +186,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.3.2", [[ - SELECT b='456' FROM t1 + SELECT b = 456 FROM t1 ]], { -- true @@ -206,7 +196,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.4.1", [[ - SELECT typeof(a), a FROM t1 GROUP BY a HAVING a=123 + SELECT typeof(a), a FROM t1 GROUP BY a HAVING a='123' ]], { -- "string", "123" @@ -216,7 +206,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.4.2", [[ - SELECT typeof(a), a FROM t1 GROUP BY b HAVING a=123 + SELECT typeof(a), a FROM t1 GROUP BY b HAVING a='123' ]], { -- "string", "123" @@ -226,7 +216,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.5.1", [[ - SELECT typeof(b), b FROM t1 GROUP BY a HAVING b='456' + SELECT typeof(b), b FROM t1 GROUP BY a HAVING b=456 ]], { -- "integer", 456 @@ -236,7 +226,7 @@ test:do_execsql_test( test:do_execsql_test( "tkt3493-2.5.2", [[ - SELECT typeof(b), b FROM t1 GROUP BY b HAVING b='456' + SELECT typeof(b), b FROM t1 GROUP BY b HAVING b=456 ]], { -- "integer", 456 diff --git a/test/sql-tap/transitive1.test.lua b/test/sql-tap/transitive1.test.lua index 96895b4a7..cc7e066bf 100755 --- a/test/sql-tap/transitive1.test.lua +++ b/test/sql-tap/transitive1.test.lua @@ -63,7 +63,7 @@ test:do_execsql_test( INSERT INTO t2 VALUES(2, 20,20,'20'); INSERT INTO t2 VALUES(3, 3,3,'3'); - SELECT a,b,c FROM t2 WHERE a=b AND c=b AND c=20; + SELECT a,b,c FROM t2 WHERE a=b AND c=CAST(b AS STRING) AND c='20'; ]], { -- 20, 20, "20" @@ -73,7 +73,7 @@ test:do_execsql_test( test:do_execsql_test( "transitive1-210", [[ - SELECT a,b,c FROM t2 WHERE a=b AND c=b AND c>='20' ORDER BY +a; + SELECT a,b,c FROM t2 WHERE a=b AND c=CAST(b AS STRING) AND c>='20' ORDER BY +a; ]], { -- 3, 3, "3", 20, 20, "20" @@ -83,7 +83,7 @@ test:do_execsql_test( test:do_execsql_test( "transitive1-220", [[ - SELECT a,b,c FROM t2 WHERE a=b AND c=b AND c<='20' ORDER BY +a; + SELECT a,b,c FROM t2 WHERE a=b AND c=CAST(b AS STRING) AND c<='20' ORDER BY +a; ]], { -- 20, 20, "20", 100, 100, "100" @@ -402,7 +402,7 @@ test:do_execsql_test( [[ CREATE TABLE x(i INTEGER PRIMARY KEY, y TEXT); INSERT INTO x VALUES(10, '10'); - SELECT * FROM x WHERE x.y>='1' AND x.y<'2' AND x.i=x.y; + SELECT * FROM x WHERE x.y>='1' AND x.y<'2' AND CAST(x.i AS STRING)=x.y; ]], { -- 10, "10" @@ -430,7 +430,7 @@ test:do_execsql_test( [[ CREATE TABLE t3(i INTEGER PRIMARY KEY, t TEXT); INSERT INTO t3 VALUES(10, '10'); - SELECT * FROM t3 WHERE i=t AND t = '10 '; + SELECT * FROM t3 WHERE CAST(i AS STRING)=t AND t = '10 '; ]], { -- @@ -443,7 +443,7 @@ test:do_execsql_test( CREATE TABLE u1(x TEXT PRIMARY KEY, y INTEGER, z TEXT); CREATE INDEX i1 ON u1(x); INSERT INTO u1 VALUES('00013', 13, '013'); - SELECT * FROM u1 WHERE x=y AND y=z AND z='013'; + SELECT * FROM u1 WHERE CAST(x AS INTEGER)=y AND y=CAST(z AS INTEGER) AND z='013'; ]], { -- "00013",13,"013" diff --git a/test/sql-tap/where2.test.lua b/test/sql-tap/where2.test.lua index f267be8e6..7348a855a 100755 --- a/test/sql-tap/where2.test.lua +++ b/test/sql-tap/where2.test.lua @@ -4,7 +4,7 @@ yaml = require("yaml") fio = require("fio") ffi = require("ffi") -test:plan(74) +test:plan(62) ffi.cdef[[ int dup(int oldfd); @@ -622,181 +622,12 @@ test:do_test( -- }) --- if X(356, "X!cmd", [=[["expr","[permutation] != \"no_optimization\""]]=]) --- then - -- Ticket #2249. Make sure the OR optimization is not attempted if - -- comparisons between columns of different affinities are needed. - -- - test:do_test( - "where2-6.7", - function() - test:execsql [[ - CREATE TABLE t2249a(a TEXT PRIMARY KEY, x VARCHAR(100)); - CREATE TABLE t2249b(b INTEGER PRIMARY KEY); - INSERT INTO t2249a(a) VALUES('0123'); - INSERT INTO t2249b VALUES(123); - ]] - return queryplan([[ - -- Because a is type TEXT and b is type INTEGER, both a and b - -- will attempt to convert to NUMERIC before the comparison. - -- They will thus compare equal. - -- - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a=b; - ]]) - end, { - -- - 123, '0123', "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.9", - function() - return queryplan([[ - -- The + operator doesn't affect RHS. - -- - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a=+b; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.9.2", - function() - -- The same thing but with the expression flipped around. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE +b=a - ]]) - end, { - -- - 123, "0123","nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.10", - function() - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE +a=+b; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.11", - function() - -- This will not attempt the OR optimization because of the a=b - -- comparison. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a=b OR a='hello'; - ]]) - end, { - -- - 123, '0123', "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.11.2", - function() - -- Permutations of the expression terms. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE b=a OR a='hello'; - ]]) - end, { - -- - 123, '0123', "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.11.3", - function() - -- Permutations of the expression terms. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE 'hello'=a OR b=a; - ]]) - end, { - -- - 123, '0123', "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.11.4", - function() - -- Permutations of the expression terms. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a='hello' OR b=a; - ]]) - end, { - -- - 123, '0123', "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - -- These tests are not run if subquery support is not included in the - -- build. This is because these tests test the "a = 1 OR a = 2" to - -- "a IN (1, 2)" optimisation transformation, which is not enabled if - -- subqueries and the IN operator is not available. - -- - test:do_test( - "where2-6.12", - function() - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a=+b OR a='hello'; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.12.2", - function() - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a='hello' OR +b=a; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.12.3", - function() - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE +b=a OR a='hello'; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - test:do_test( - "where2-6.13", - function() - -- The addition of +a on the second term disabled the OR optimization. - -- But we should still get the same empty-set result as in where2-6.9. - return queryplan([[ - SELECT b,a FROM t2249b CROSS JOIN t2249a WHERE a=+b OR +a='hello'; - ]]) - end, { - -- - 123, "0123", "nosort", "T2249B", "*", "T2249A", "*" - -- - }) - - + test:execsql [[ + CREATE TABLE t2249a(a TEXT PRIMARY KEY, x VARCHAR(100)); + CREATE TABLE t2249b(b INTEGER PRIMARY KEY); + INSERT INTO t2249a(a) VALUES('0123'); + INSERT INTO t2249b VALUES(123); + ]] -- Variations on the order of terms in a WHERE clause in order -- to make sure the OR optimizer can recognize them all. diff --git a/test/sql-tap/where5.test.lua b/test/sql-tap/where5.test.lua index 3aefcaca5..4a197aac2 100755 --- a/test/sql-tap/where5.test.lua +++ b/test/sql-tap/where5.test.lua @@ -34,7 +34,7 @@ test:do_test("where5-1.0", function() INSERT INTO t3 SELECT CAST(x AS INTEGER) FROM t1; ]] return test:execsql [[ - SELECT * FROM t1 WHERE x<0 + SELECT * FROM t1 WHERE x<'0' ]] end, { -- @@ -43,7 +43,7 @@ end, { }) test:do_execsql_test("where5-1.1", [[ - SELECT * FROM t1 WHERE x<=0 + SELECT * FROM t1 WHERE x<='0' ]], { -- '-1', '0' @@ -51,7 +51,7 @@ test:do_execsql_test("where5-1.1", [[ }) test:do_execsql_test("where5-1.2", [[ - SELECT * FROM t1 WHERE x=0 + SELECT * FROM t1 WHERE x='0' ]], { -- '0' @@ -59,7 +59,7 @@ test:do_execsql_test("where5-1.2", [[ }) test:do_execsql_test("where5-1.3", [[ - SELECT * FROM t1 WHERE x>=0 + SELECT * FROM t1 WHERE x>='0' ]], { -- '0', '1' @@ -67,7 +67,7 @@ test:do_execsql_test("where5-1.3", [[ }) test:do_execsql_test("where5-1.4", [[ - SELECT * FROM t1 WHERE x>0 + SELECT * FROM t1 WHERE x>'0' ]], { -- '1' @@ -75,7 +75,7 @@ test:do_execsql_test("where5-1.4", [[ }) test:do_execsql_test("where5-1.5", [[ - SELECT * FROM t1 WHERE x<>0 + SELECT * FROM t1 WHERE x<>'0' ]], { -- '-1', '1' diff --git a/test/sql/types.result b/test/sql/types.result index 8810a9f82..b40f45029 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -608,11 +608,8 @@ box.execute("SELECT 18446744073709551615.0 > 18446744073709551615") ... box.execute("SELECT 18446744073709551615 IN ('18446744073709551615', 18446744073709551615.0)") --- -- metadata: - - name: COLUMN_1 - type: boolean - rows: - - [true] +- null +- 'Type mismatch: can not convert 18446744073709551615 to numeric' ... box.execute("SELECT 1 LIMIT 18446744073709551615;") --- -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:56 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:56 +0300 Subject: [Tarantool-patches] [PATCH v5 5/6] sql: fix implicit cast in opcode MustBeInt In-Reply-To: References: Message-ID: <660af07f4a4cf50567ee5ca845cfa538f316e9f4.1598000242.git.imeevma@gmail.com> This patch removes implicit casting from STRING to number and vice versa from MustBeInt opcode. Part of #4230 --- src/box/sql/vdbe.c | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index b326c4ba4..a0ddbaf60 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -2352,16 +2352,20 @@ case OP_AddImm: { /* in1 */ */ case OP_MustBeInt: { /* jump, in1 */ pIn1 = &aMem[pOp->p1]; - if ((pIn1->flags & (MEM_Int | MEM_UInt)) == 0) { - mem_apply_type(pIn1, FIELD_TYPE_INTEGER); - if ((pIn1->flags & (MEM_Int | MEM_UInt)) == 0) { - if (pOp->p2==0) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(pIn1), "integer"); - goto abort_due_to_error; - } else { - goto jump_to_p2; - } + if (mem_is_type_compatible(pIn1, FIELD_TYPE_INTEGER)) + break; + if ((pIn1->flags & MEM_Real) != 0) { + double d = pIn1->u.r; + if (d == (double)(int64_t)d || d == (double)(uint64_t)d) + mem_convert_to_integer(pIn1); + } + if (!mem_is_type_compatible(pIn1, FIELD_TYPE_INTEGER)) { + if (pOp->p2==0) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(pIn1), "integer"); + goto abort_due_to_error; + } else { + goto jump_to_p2; } } break; -- 2.25.1 From imeevma at tarantool.org Fri Aug 21 12:19:58 2020 From: imeevma at tarantool.org (imeevma at tarantool.org) Date: Fri, 21 Aug 2020 12:19:58 +0300 Subject: [Tarantool-patches] [PATCH v5 6/6] sql: remove implicit cast from MakeRecord opcode In-Reply-To: References: Message-ID: <2446ef040a7f9d235cfc14bccf4dcaa7b6254e73.1598000242.git.imeevma@gmail.com> This patch removes implicit casting from MakeRecord opcode. Closes #4230 @TarantoolBot document Title: remove implicit cast for comparison After this patch-set, there will be no implicit casts for comparison. This means that the values ?of the field types STRING, BOOLEAN and VARBINARY can be compared with the values of the same field type. Any numerical value can be compared with any other numerical value. Example: ``` tarantool> box.execute([[SELECT '1' > 0;]]) --- - null - 'Type mismatch: can not convert 1 to numeric' ... tarantool> box.execute([[SELECT true > X'33';]]) --- - null - 'Type mismatch: can not convert boolean to varbinary' ... tarantool> box.execute([[SELECT 1.23 > 123;]]) --- - metadata: - name: 1.23 > 123 type: boolean rows: - [false] ... ``` --- src/box/sql/analyze.c | 6 +- src/box/sql/delete.c | 15 ++- src/box/sql/expr.c | 17 ++- src/box/sql/fk_constraint.c | 12 +- src/box/sql/select.c | 26 +++-- src/box/sql/update.c | 23 ++-- src/box/sql/vdbe.c | 19 +-- test/sql-tap/in3.test.lua | 26 +---- test/sql/boolean.result | 76 ++++-------- test/sql/types.result | 225 ++++++++++++++++++++++++++++++++++++ test/sql/types.test.lua | 51 ++++++++ 11 files changed, 363 insertions(+), 133 deletions(-) diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c index f74f9b358..3efcd041a 100644 --- a/src/box/sql/analyze.c +++ b/src/box/sql/analyze.c @@ -969,8 +969,10 @@ vdbe_emit_analyze_space(struct Parse *parse, struct space *space) FIELD_TYPE_STRING, FIELD_TYPE_STRING, field_type_MAX }; - sqlVdbeAddOp4(v, OP_MakeRecord, tab_name_reg, 4, tmp_reg, - (char *)types, sizeof(types)); + sqlVdbeAddOp4(v, OP_ApplyType, tab_name_reg, 4, 0, + (char *)types, P4_STATIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, tab_name_reg, 4, tmp_reg); sqlVdbeAddOp4(v, OP_IdxInsert, tmp_reg, 0, 0, (char *)stat1, P4_SPACEPTR); /* Add the entries to the stat4 table. */ diff --git a/src/box/sql/delete.c b/src/box/sql/delete.c index 68abd1f58..40a52aadc 100644 --- a/src/box/sql/delete.c +++ b/src/box/sql/delete.c @@ -329,11 +329,16 @@ sql_table_delete_from(struct Parse *parse, struct SrcList *tab_list, */ key_len = 0; struct index *pk = space_index(space, 0); - enum field_type *types = is_view ? NULL : - sql_index_type_str(parse->db, - pk->def); - sqlVdbeAddOp4(v, OP_MakeRecord, reg_pk, pk_len, - reg_key, (char *)types, P4_DYNAMIC); + if (!is_view) { + enum field_type *types = + sql_index_type_str(parse->db, pk->def); + sqlVdbeAddOp4(v, OP_ApplyType, reg_pk, pk_len, + 0, (char *)types, P4_DYNAMIC); + sqlVdbeChangeP5(v, + OPFLAG_DO_NOT_CONVERT_NUMBERS); + } + sqlVdbeAddOp3(v, OP_MakeRecord, reg_pk, pk_len, + reg_key); /* Set flag to save memory allocating one * by malloc. */ diff --git a/src/box/sql/expr.c b/src/box/sql/expr.c index bc2182446..d553b80db 100644 --- a/src/box/sql/expr.c +++ b/src/box/sql/expr.c @@ -2886,11 +2886,18 @@ sqlCodeSubselect(Parse * pParse, /* Parsing context */ jmpIfDynamic = -1; } r3 = sqlExprCodeTarget(pParse, pE2, r1); - enum field_type types[2] = - { lhs_type, field_type_MAX }; - sqlVdbeAddOp4(v, OP_MakeRecord, r3, - 1, r2, (char *)types, - sizeof(types)); + uint32_t size = + 2 * sizeof(enum field_type); + enum field_type *types= + sqlDbMallocZero(pParse->db, + size); + types[0] = lhs_type; + types[1] = field_type_MAX; + sqlVdbeAddOp4(v, OP_ApplyType, r3, 1, 0, + (char *)types, P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, r3, 1, + r2); sql_expr_type_cache_change(pParse, r3, 1); sqlVdbeAddOp2(v, OP_IdxInsert, r2, diff --git a/src/box/sql/fk_constraint.c b/src/box/sql/fk_constraint.c index 482220a95..50f9ebf74 100644 --- a/src/box/sql/fk_constraint.c +++ b/src/box/sql/fk_constraint.c @@ -264,11 +264,13 @@ fk_constraint_lookup_parent(struct Parse *parse_context, struct space *parent, } struct index *idx = space_index(parent, referenced_idx); assert(idx != NULL); - sqlVdbeAddOp4(v, OP_MakeRecord, temp_regs, field_count, - rec_reg, - (char *) sql_index_type_str(parse_context->db, - idx->def), - P4_DYNAMIC); + sqlVdbeAddOp4(v, OP_ApplyType, temp_regs, field_count, 0, + (char *) sql_index_type_str(parse_context->db, + idx->def), + P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, temp_regs, field_count, + rec_reg); sqlVdbeAddOp4Int(v, OP_Found, cursor, ok_label, rec_reg, 0); sqlReleaseTempReg(parse_context, rec_reg); sqlReleaseTempRange(parse_context, temp_regs, field_count); diff --git a/src/box/sql/select.c b/src/box/sql/select.c index b0554a172..4c7d8001c 100644 --- a/src/box/sql/select.c +++ b/src/box/sql/select.c @@ -1274,9 +1274,13 @@ selectInnerLoop(Parse * pParse, /* The parser context */ field_type_sequence_dup(pParse, pDest->dest_type, nResultCol); - sqlVdbeAddOp4(v, OP_MakeRecord, regResult, - nResultCol, r1, (char *)types, - P4_DYNAMIC); + sqlVdbeAddOp4(v, OP_ApplyType, regResult, + nResultCol, 0, (char *)types, + P4_DYNAMIC); + sqlVdbeChangeP5(v, + OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, regResult, + nResultCol, r1); sql_expr_type_cache_change(pParse, regResult, nResultCol); @@ -1696,9 +1700,11 @@ generateSortTail(Parse * pParse, /* Parsing context */ enum field_type *types = field_type_sequence_dup(pParse, pDest->dest_type, nColumn); - sqlVdbeAddOp4(v, OP_MakeRecord, regRow, nColumn, - regTupleid, (char *)types, - P4_DYNAMIC); + sqlVdbeAddOp4(v, OP_ApplyType, regRow, nColumn, 0, + (char *)types, P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, regRow, nColumn, + regTupleid); sql_expr_type_cache_change(pParse, regRow, nColumn); sqlVdbeAddOp2(v, OP_IdxInsert, regTupleid, pDest->reg_eph); break; @@ -3138,9 +3144,11 @@ generateOutputSubroutine(struct Parse *parse, struct Select *p, enum field_type *types = field_type_sequence_dup(parse, dest->dest_type, in->nSdst); - sqlVdbeAddOp4(v, OP_MakeRecord, in->iSdst, - in->nSdst, r1, (char *)types, - P4_DYNAMIC); + sqlVdbeAddOp4(v, OP_ApplyType, in->iSdst, in->nSdst, 0, + (char *)types, P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, in->iSdst, in->nSdst, + r1); sql_expr_type_cache_change(parse, in->iSdst, in->nSdst); sqlVdbeAddOp2(v, OP_IdxInsert, r1, dest->reg_eph); diff --git a/src/box/sql/update.c b/src/box/sql/update.c index 24c7cfa27..7c80dcc4e 100644 --- a/src/box/sql/update.c +++ b/src/box/sql/update.c @@ -251,11 +251,14 @@ sqlUpdate(Parse * pParse, /* The parser context */ nKey = pk_part_count; regKey = iPk; } else { - enum field_type *types = is_view ? NULL : - sql_index_type_str(pParse->db, - pPk->def); - sqlVdbeAddOp4(v, OP_MakeRecord, iPk, pk_part_count, - regKey, (char *) types, P4_DYNAMIC); + if (!is_view) { + enum field_type *types = + sql_index_type_str(pParse->db, pPk->def); + sqlVdbeAddOp4(v, OP_ApplyType, iPk, pk_part_count, 0, + (char *)types, P4_DYNAMIC); + sqlVdbeChangeP5(v, OPFLAG_DO_NOT_CONVERT_NUMBERS); + } + sqlVdbeAddOp3(v, OP_MakeRecord, iPk, pk_part_count, regKey); /* * Set flag to save memory allocating one by * malloc. @@ -423,9 +426,13 @@ sqlUpdate(Parse * pParse, /* The parser context */ enum field_type *types = sql_index_type_str(pParse->db, pPk->def); - sqlVdbeAddOp4(v, OP_MakeRecord, iPk, - pk_part_count, key_reg, - (char *) types, P4_DYNAMIC); + sqlVdbeAddOp4(v, OP_ApplyType, iPk, + pk_part_count, 0, (char *)types, + P4_DYNAMIC); + sqlVdbeChangeP5(v, + OPFLAG_DO_NOT_CONVERT_NUMBERS); + sqlVdbeAddOp3(v, OP_MakeRecord, iPk, + pk_part_count, key_reg); } else { assert(nKey == 0); key_reg = regKey; diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index a0ddbaf60..544619b03 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -3145,24 +3145,17 @@ type_mismatch: break; } -/* Opcode: MakeRecord P1 P2 P3 P4 P5 +/* Opcode: MakeRecord P1 P2 P3 * P5 * Synopsis: r[P3]=mkrec(r[P1 at P2]) * * Convert P2 registers beginning with P1 into the [record format] * use as a data record in a database table or as a key * in an index. The OP_Column opcode can decode the record later. * - * P4 may be a string that is P2 characters long. The nth character of the - * string indicates the column type that should be used for the nth - * field of the index key. - * - * If P4 is NULL then all index fields have type SCALAR. - * * If P5 is not NULL then record under construction is intended to be inserted * into ephemeral space. Thus, sort of memory optimization can be performed. */ case OP_MakeRecord: { - Mem *pRec; /* The new record */ Mem *pData0; /* First field to be combined into the record */ Mem MAYBE_UNUSED *pLast; /* Last field of the record */ int nField; /* Number of fields in the record */ @@ -3184,7 +3177,6 @@ case OP_MakeRecord: { * of the record to data0. */ nField = pOp->p1; - enum field_type *types = pOp->p4.types; bIsEphemeral = pOp->p5; assert(nField>0 && pOp->p2>0 && pOp->p2+nField<=(p->nMem+1 - p->nCursor)+1); pData0 = &aMem[nField]; @@ -3195,15 +3187,6 @@ case OP_MakeRecord: { assert(pOp->p3p1 || pOp->p3>=pOp->p1+pOp->p2); pOut = vdbe_prepare_null_out(p, pOp->p3); - /* Apply the requested types to all inputs */ - assert(pData0<=pLast); - if (types != NULL) { - pRec = pData0; - do { - mem_apply_type(pRec++, *(types++)); - } while(types[0] != field_type_MAX); - } - struct region *region = &fiber()->gc; size_t used = region_used(region); uint32_t tuple_size; diff --git a/test/sql-tap/in3.test.lua b/test/sql-tap/in3.test.lua index a6d842962..7f3abbae0 100755 --- a/test/sql-tap/in3.test.lua +++ b/test/sql-tap/in3.test.lua @@ -1,6 +1,6 @@ #!/usr/bin/env tarantool test = require("sqltester") -test:plan(28) +test:plan(26) --!./tcltestrunner.lua -- 2007 November 29 @@ -334,18 +334,6 @@ test:do_test( -- }) -test:do_test( - "in3-3.5", - function() - -- Numeric affinity should be applied to each side before the comparison - -- takes place. Therefore we cannot use index t1_i1, which has no affinity. - return exec_neph(" SELECT y IN (SELECT a FROM t1) FROM t2 ") - end, { - -- - 1, true - -- - }) - test:do_test( "in3-3.6", function() @@ -358,18 +346,6 @@ test:do_test( -- }) -test:do_test( - "in3-3.7", - function() - -- Numeric affinity is applied before the comparison takes place. - -- Making it impossible to use index t1_i3. - return exec_neph(" SELECT y IN (SELECT c FROM t1) FROM t2 ") - end, { - -- - 1, true - -- - }) - ----------------------------------------------------------------------- -- -- Test using a multi-column index. diff --git a/test/sql/boolean.result b/test/sql/boolean.result index 51ec5820b..d3eca833c 100644 --- a/test/sql/boolean.result +++ b/test/sql/boolean.result @@ -2256,19 +2256,13 @@ SELECT false IN (SELECT a1 FROM t6 LIMIT 1); | ... SELECT true IN (1, 1.2, 'true', false); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 1 to boolean' | ... SELECT false IN (1, 1.2, 'true', false); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [true] + | - null + | - 'Type mismatch: can not convert 1 to boolean' | ... SELECT a, a IN (true) FROM t; @@ -2328,14 +2322,8 @@ SELECT a, a IN (SELECT a1 FROM t6) FROM t; | ... SELECT a, a IN (1, 1.2, 'true', false) FROM t; | --- - | - metadata: - | - name: A - | type: boolean - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false, true] - | - [true, false] + | - null + | - 'Type mismatch: can not convert 1 to boolean' | ... SELECT true BETWEEN true AND true; @@ -3860,19 +3848,13 @@ SELECT a2, b, b != a2 FROM t6, t7; SELECT true IN (0, 1, 2, 3); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0 to boolean' | ... SELECT false IN (0, 1, 2, 3); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0 to boolean' | ... SELECT true IN (SELECT b FROM t7); | --- @@ -3886,14 +3868,8 @@ SELECT false IN (SELECT b FROM t7); | ... SELECT a1, a1 IN (0, 1, 2, 3) FROM t6 | --- - | - metadata: - | - name: A1 - | type: boolean - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false, false] - | - [true, false] + | - null + | - 'Type mismatch: can not convert 0 to boolean' | ... SELECT true BETWEEN 0 and 10; @@ -5005,35 +4981,23 @@ SELECT a2, c, c != a2 FROM t6, t8; SELECT true IN (0.1, 1.2, 2.3, 3.4); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0.1 to boolean' | ... SELECT false IN (0.1, 1.2, 2.3, 3.4); | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0.1 to boolean' | ... SELECT a1 IN (0.1, 1.2, 2.3, 3.4) FROM t6 LIMIT 1; | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0.1 to boolean' | ... SELECT a2 IN (0.1, 1.2, 2.3, 3.4) FROM t6 LIMIT 1; | --- - | - metadata: - | - name: COLUMN_1 - | type: boolean - | rows: - | - [false] + | - null + | - 'Type mismatch: can not convert 0.1 to boolean' | ... SELECT true IN (SELECT c FROM t8); | --- diff --git a/test/sql/types.result b/test/sql/types.result index b40f45029..29aa90d7b 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2846,3 +2846,228 @@ box.execute([[DROP TABLE t;]]) --- - row_count: 1 ... +-- +-- Make sure that there is no implicit cast between string and +-- number. +-- +box.execute([[SELECT '1' > 0;]]); +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT 1 > '0';]]); +--- +- null +- 'Type mismatch: can not convert 0 to numeric' +... +box.execute([[CREATE TABLE t (i INT PRIMARY KEY, d DOUBLE, n NUMBER, s STRING);]]) +--- +- row_count: 1 +... +box.execute([[INSERT INTO t VALUES (1, 1.0, 1, '2'), (2, 2.0, 2.0, '2');]]) +--- +- row_count: 2 +... +box.execute([[SELECT * from t WHERE i > s;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT * from t WHERE s > i;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT * from t WHERE d > s;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT * from t WHERE s > d;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT * from t WHERE i = 1 and n > s;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT * from t WHERE i = 2 and s > n;]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT i FROM t WHERE i in (1);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE d in (1);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE n in (1);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE s in (1);]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT i FROM t WHERE i in (1.0);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE d in (1.0);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE n in (1.0);]]) +--- +- metadata: + - name: I + type: integer + rows: + - [1] +... +box.execute([[SELECT i FROM t WHERE s in (1.0);]]) +--- +- null +- 'Type mismatch: can not convert 2 to numeric' +... +box.execute([[SELECT i FROM t WHERE i in ('1');]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT i FROM t WHERE d in ('1');]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT i FROM t WHERE n in ('1');]]) +--- +- null +- 'Type mismatch: can not convert 1 to numeric' +... +box.execute([[SELECT i FROM t WHERE s in ('1');]]) +--- +- metadata: + - name: I + type: integer + rows: [] +... +box.execute([[SELECT i FROM t WHERE i in ('1.0');]]) +--- +- null +- 'Type mismatch: can not convert 1.0 to integer' +... +box.execute([[SELECT i FROM t WHERE d in ('1.0');]]) +--- +- null +- 'Type mismatch: can not convert 1.0 to numeric' +... +box.execute([[SELECT i FROM t WHERE n in ('1.0');]]) +--- +- null +- 'Type mismatch: can not convert 1.0 to numeric' +... +box.execute([[SELECT i FROM t WHERE s in ('1.0');]]) +--- +- metadata: + - name: I + type: integer + rows: [] +... +box.execute([[DROP TABLE t;]]) +--- +- row_count: 1 +... +-- Comparison with SCALAR. +box.execute([[CREATE TABLE t(a SCALAR PRIMARY KEY);]]) +--- +- row_count: 1 +... +box.execute([[INSERT INTO t VALUES (1), (2.2), ('3');]]); +--- +- row_count: 3 +... +box.execute([[SELECT a FROM t WHERE a > 1]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a > 1.0]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a > '1']]); +--- +- metadata: + - name: A + type: scalar + rows: + - ['3'] +... +box.execute([[SELECT a FROM t WHERE a < 1]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a < 1.0]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a < '1']]); +--- +- metadata: + - name: A + type: scalar + rows: + - [1] + - [2.2] +... +box.execute([[SELECT a FROM t WHERE a = 1]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a = 1.0]]); +--- +- null +- 'Type mismatch: can not convert 3 to numeric' +... +box.execute([[SELECT a FROM t WHERE a = '1']]); +--- +- metadata: + - name: A + type: scalar + rows: [] +... +box.execute([[DROP TABLE t;]]) +--- +- row_count: 1 +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index a23b12801..09619653b 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -635,3 +635,54 @@ box.execute([[SELECT * FROM t WHERE i > ?]], {-2^70}) box.execute([[SELECT * FROM t WHERE a = ?]], {2ULL^60ULL - 1ULL}) box.execute([[SELECT * FROM t WHERE a > ?]], {2ULL^60ULL - 1ULL}) box.execute([[DROP TABLE t;]]) + +-- +-- Make sure that there is no implicit cast between string and +-- number. +-- +box.execute([[SELECT '1' > 0;]]); +box.execute([[SELECT 1 > '0';]]); +box.execute([[CREATE TABLE t (i INT PRIMARY KEY, d DOUBLE, n NUMBER, s STRING);]]) +box.execute([[INSERT INTO t VALUES (1, 1.0, 1, '2'), (2, 2.0, 2.0, '2');]]) +box.execute([[SELECT * from t WHERE i > s;]]) +box.execute([[SELECT * from t WHERE s > i;]]) +box.execute([[SELECT * from t WHERE d > s;]]) +box.execute([[SELECT * from t WHERE s > d;]]) +box.execute([[SELECT * from t WHERE i = 1 and n > s;]]) +box.execute([[SELECT * from t WHERE i = 2 and s > n;]]) + +box.execute([[SELECT i FROM t WHERE i in (1);]]) +box.execute([[SELECT i FROM t WHERE d in (1);]]) +box.execute([[SELECT i FROM t WHERE n in (1);]]) +box.execute([[SELECT i FROM t WHERE s in (1);]]) + +box.execute([[SELECT i FROM t WHERE i in (1.0);]]) +box.execute([[SELECT i FROM t WHERE d in (1.0);]]) +box.execute([[SELECT i FROM t WHERE n in (1.0);]]) +box.execute([[SELECT i FROM t WHERE s in (1.0);]]) + +box.execute([[SELECT i FROM t WHERE i in ('1');]]) +box.execute([[SELECT i FROM t WHERE d in ('1');]]) +box.execute([[SELECT i FROM t WHERE n in ('1');]]) +box.execute([[SELECT i FROM t WHERE s in ('1');]]) + +box.execute([[SELECT i FROM t WHERE i in ('1.0');]]) +box.execute([[SELECT i FROM t WHERE d in ('1.0');]]) +box.execute([[SELECT i FROM t WHERE n in ('1.0');]]) +box.execute([[SELECT i FROM t WHERE s in ('1.0');]]) + +box.execute([[DROP TABLE t;]]) + +-- Comparison with SCALAR. +box.execute([[CREATE TABLE t(a SCALAR PRIMARY KEY);]]) +box.execute([[INSERT INTO t VALUES (1), (2.2), ('3');]]); +box.execute([[SELECT a FROM t WHERE a > 1]]); +box.execute([[SELECT a FROM t WHERE a > 1.0]]); +box.execute([[SELECT a FROM t WHERE a > '1']]); +box.execute([[SELECT a FROM t WHERE a < 1]]); +box.execute([[SELECT a FROM t WHERE a < 1.0]]); +box.execute([[SELECT a FROM t WHERE a < '1']]); +box.execute([[SELECT a FROM t WHERE a = 1]]); +box.execute([[SELECT a FROM t WHERE a = 1.0]]); +box.execute([[SELECT a FROM t WHERE a = '1']]); +box.execute([[DROP TABLE t;]]) -- 2.25.1 From korablev at tarantool.org Fri Aug 21 12:21:30 2020 From: korablev at tarantool.org (Nikita Pettik) Date: Fri, 21 Aug 2020 09:21:30 +0000 Subject: [Tarantool-patches] [PATCH v1 2/2] sql: remove implicit cast in bitwise operations In-Reply-To: <56f685097fb1120e36bf03114a32d567e668a2a2.1597998754.git.imeevma@gmail.com> References: <56f685097fb1120e36bf03114a32d567e668a2a2.1597998754.git.imeevma@gmail.com> Message-ID: <20200821092130.GC6452@tarantool.org> On 21 Aug 11:40, imeevma at tarantool.org wrote: > This patch removes the implicit conversion from STRING to INTEGER from > bitwise operations. However, DOUBLE can still be implicitly converted to > INTEGER. I see no test involving doubles in bitwise operations. Does it make any sense at all? > Follow-up #3809 > --- > src/box/sql/vdbe.c | 59 ++++++++++++++++++++++++++++++++--------- > test/sql/types.result | 49 ++++++++++++++++++++++++++++++++++ > test/sql/types.test.lua | 14 ++++++++++ > 3 files changed, 110 insertions(+), 12 deletions(-) > > diff --git a/test/sql/types.result b/test/sql/types.result > index caedbf409..601e5beca 100644 > --- a/test/sql/types.result > +++ b/test/sql/types.result > @@ -2846,3 +2846,52 @@ box.execute([[SELECT 1 - '2';]]) > - null > - 'Type mismatch: can not convert 2 to numeric' > ... > +-- > +-- Make sure there is no implicit string-to-number conversion in bitwise > +-- operations. > +-- > +box.execute([[SELECT '1' | 2;]]) > +--- > +- null > +- 'Type mismatch: can not convert 1 to integer' > +... > +box.execute([[SELECT '1' & 2;]]) > +--- > +- null > +- 'Type mismatch: can not convert 1 to integer' > +... > +box.execute([[SELECT '1' << 2;]]) > +--- > +- null > +- 'Type mismatch: can not convert 1 to integer' > +... > +box.execute([[SELECT '1' >> 2;]]) > +--- > +- null > +- 'Type mismatch: can not convert 1 to integer' > +... > +box.execute([[SELECT ~'1';]]) > +--- > +- null > +- 'Type mismatch: can not convert 1 to integer' > +... > +box.execute([[SELECT 1 | '2';]]) > +--- > +- null > +- 'Type mismatch: can not convert 2 to integer' > +... > +box.execute([[SELECT 1 & '2';]]) > +--- > +- null > +- 'Type mismatch: can not convert 2 to integer' > +... > +box.execute([[SELECT 1 << '2';]]) > +--- > +- null > +- 'Type mismatch: can not convert 2 to integer' > +... > +box.execute([[SELECT 1 >> '2';]]) > +--- > +- null > +- 'Type mismatch: can not convert 2 to integer' > +... > diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua > index 844a6b670..2d2f41da2 100644 > --- a/test/sql/types.test.lua > +++ b/test/sql/types.test.lua > @@ -638,3 +638,17 @@ box.execute([[SELECT 1 % '2';]]) > box.execute([[SELECT 1 * '2';]]) > box.execute([[SELECT 1 / '2';]]) > box.execute([[SELECT 1 - '2';]]) > + > +-- > +-- Make sure there is no implicit string-to-number conversion in bitwise > +-- operations. > +-- > +box.execute([[SELECT '1' | 2;]]) > +box.execute([[SELECT '1' & 2;]]) > +box.execute([[SELECT '1' << 2;]]) > +box.execute([[SELECT '1' >> 2;]]) > +box.execute([[SELECT ~'1';]]) > +box.execute([[SELECT 1 | '2';]]) > +box.execute([[SELECT 1 & '2';]]) > +box.execute([[SELECT 1 << '2';]]) > +box.execute([[SELECT 1 >> '2';]]) > -- > 2.25.1 > From imeevma at tarantool.org Fri Aug 21 15:34:09 2020 From: imeevma at tarantool.org (Mergen Imeev) Date: Fri, 21 Aug 2020 15:34:09 +0300 Subject: [Tarantool-patches] [PATCH v1 2/2] sql: remove implicit cast in bitwise operations In-Reply-To: <20200821092130.GC6452@tarantool.org> References: <56f685097fb1120e36bf03114a32d567e668a2a2.1597998754.git.imeevma@gmail.com> <20200821092130.GC6452@tarantool.org> Message-ID: <20200821123409.GA219863@tarantool.org> Hi! Thank you for the review. My answer and new patch below. On Fri, Aug 21, 2020 at 09:21:30AM +0000, Nikita Pettik wrote: > On 21 Aug 11:40, imeevma at tarantool.org wrote: > > This patch removes the implicit conversion from STRING to INTEGER from > > bitwise operations. However, DOUBLE can still be implicitly converted to > > INTEGER. > > I see no test involving doubles in bitwise operations. Does it make > any sense at all? Even if it doesn't, it is legal to implicitly convert DOUBLE to INTEGER. However, when I tried to add tests for this case, I found an error in my patch. I re-made patch. Now in these opcodes we convert MEM to INTEGER, which I tried to avoid in previous patch. I did this to fix a bug where result of the operation is DOUBLE if one of the operands is DOUBLE. It didn't help, the result still has DOUBLE type. I decided to left conversion since it looks right here because all operands must be INTEGERS. This wouldn't work for arithmetic operations though. New patch: >From 3515ada4b363062cf9caa5d550ea40770e8a5e65 Mon Sep 17 00:00:00 2001 From: Mergen Imeev Date: Tue, 18 Aug 2020 18:18:59 +0300 Subject: [PATCH] sql: remove implicit cast in bitwise operations This patch removes the implicit conversion from STRING to INTEGER from bitwise operations. However, DOUBLE can still be implicitly converted to INTEGER. Follow-up #3809 diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index c0143a6b1..42228c435 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -1989,23 +1989,26 @@ case OP_ShiftRight: { /* same as TK_RSHIFT, in1, in2, out3 */ pIn1 = &aMem[pOp->p1]; pIn2 = &aMem[pOp->p2]; + enum mp_type type1 = mem_mp_type(pIn1); + enum mp_type type2 = mem_mp_type(pIn2); pOut = vdbe_prepare_null_out(p, pOp->p3); - if ((pIn1->flags | pIn2->flags) & MEM_Null) { + if (type1 == MP_NIL || type2 == MP_NIL) { /* Force NULL be of type INTEGER. */ pOut->field_type = FIELD_TYPE_INTEGER; break; } - bool unused; - if (sqlVdbeIntValue(pIn2, (int64_t *) &iA, &unused) != 0) { + if (mem_convert_to_integer(pIn2) != 0) { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn2), "integer"); goto abort_due_to_error; } - if (sqlVdbeIntValue(pIn1, (int64_t *) &iB, &unused) != 0) { + if (mem_convert_to_integer(pIn1) != 0) { diag_set(ClientError, ER_SQL_TYPE_MISMATCH, sql_value_to_diag_str(pIn1), "integer"); goto abort_due_to_error; } + iA = pIn2->u.i; + iB = pIn1->u.i; op = pOp->opcode; if (op==OP_BitAnd) { iA &= iB; @@ -2621,19 +2624,19 @@ case OP_Not: { /* same as TK_NOT, in1, out2 */ */ case OP_BitNot: { /* same as TK_BITNOT, in1, out2 */ pIn1 = &aMem[pOp->p1]; + enum mp_type type = mem_mp_type(pIn1); pOut = vdbe_prepare_null_out(p, pOp->p2); /* Force NULL be of type INTEGER. */ pOut->field_type = FIELD_TYPE_INTEGER; - if ((pIn1->flags & MEM_Null)==0) { - int64_t i; - bool is_neg; - if (sqlVdbeIntValue(pIn1, &i, &is_neg) != 0) { - diag_set(ClientError, ER_SQL_TYPE_MISMATCH, - sql_value_to_diag_str(pIn1), "integer"); - goto abort_due_to_error; - } - mem_set_i64(pOut, ~i); + if (type == MP_NIL) { + break; + } + if (mem_convert_to_integer(pIn1) != 0) { + diag_set(ClientError, ER_SQL_TYPE_MISMATCH, + sql_value_to_diag_str(pIn1), "integer"); + goto abort_due_to_error; } + mem_set_i64(pOut, ~pIn1->u.i); break; } diff --git a/test/sql/types.result b/test/sql/types.result index caedbf409..70247471e 100644 --- a/test/sql/types.result +++ b/test/sql/types.result @@ -2846,3 +2846,93 @@ box.execute([[SELECT 1 - '2';]]) - null - 'Type mismatch: can not convert 2 to numeric' ... +-- +-- Make sure there is no implicit string-to-number conversion in bitwise +-- operations. +-- +box.execute([[SELECT '1' | 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' & 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' << 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT '1' >> 2;]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT ~'1';]]) +--- +- null +- 'Type mismatch: can not convert 1 to integer' +... +box.execute([[SELECT 1 | '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 & '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 << '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +box.execute([[SELECT 1 >> '2';]]) +--- +- null +- 'Type mismatch: can not convert 2 to integer' +... +-- Make sure that DOUBLE implicitly cast to INTEGER in bitwise operations. +box.execute([[SELECT 3.5 | 1.3;]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [3] +... +box.execute([[SELECT 3.5 & 1.3;]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1] +... +box.execute([[SELECT 3.5 << 1.3;]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [6] +... +box.execute([[SELECT 3.5 >> 1.3;]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [1] +... +box.execute([[SELECT ~3.5;]]) +--- +- metadata: + - name: COLUMN_1 + type: double + rows: + - [-4] +... diff --git a/test/sql/types.test.lua b/test/sql/types.test.lua index 844a6b670..c000f4d13 100644 --- a/test/sql/types.test.lua +++ b/test/sql/types.test.lua @@ -638,3 +638,24 @@ box.execute([[SELECT 1 % '2';]]) box.execute([[SELECT 1 * '2';]]) box.execute([[SELECT 1 / '2';]]) box.execute([[SELECT 1 - '2';]]) + +-- +-- Make sure there is no implicit string-to-number conversion in bitwise +-- operations. +-- +box.execute([[SELECT '1' | 2;]]) +box.execute([[SELECT '1' & 2;]]) +box.execute([[SELECT '1' << 2;]]) +box.execute([[SELECT '1' >> 2;]]) +box.execute([[SELECT ~'1';]]) +box.execute([[SELECT 1 | '2';]]) +box.execute([[SELECT 1 & '2';]]) +box.execute([[SELECT 1 << '2';]]) +box.execute([[SELECT 1 >> '2';]]) + +-- Make sure that DOUBLE implicitly cast to INTEGER in bitwise operations. +box.execute([[SELECT 3.5 | 1.3;]]) +box.execute([[SELECT 3.5 & 1.3;]]) +box.execute([[SELECT 3.5 << 1.3;]]) +box.execute([[SELECT 3.5 >> 1.3;]]) +box.execute([[SELECT ~3.5;]]) From olegrok at tarantool.org Fri Aug 21 19:26:07 2020 From: olegrok at tarantool.org (olegrok at tarantool.org) Date: Fri, 21 Aug 2020 19:26:07 +0300 Subject: [Tarantool-patches] [PATCH] exports: allow to use json tools via FFI Message-ID: <20200821162607.68179-1-olegrok@tarantool.org> From: Oleg Babin This patch exports some json functions to be used via FFI. We solve following problem: currently we don't have any tools to inspect jsonpaths. E.g. user wants to ban or restrict an access to some tuple fields, it could be some system fields. Tarantool doesn't have hidden fields and to solve such problem we should fairly parse input jsonpaths. Before this patch user should write its own or some external tools. This patch allows to use functions from built-in json-lexer directly from Tarantool via FFI. Part of #5203 --- Issue: https://github.com/tarantool/tarantool/issues/5203 Branch: https://github.com/tarantool/tarantool/tree/olegrok/5203-expose-json-helpers Also issue contains an example of usage this feature. @Changelog: - Some symbols from tarantool json moudule are exported now (gh-5203). src/exports.h | 4 ++ test/box-tap/gh-5203-json-exports.test.lua | 82 ++++++++++++++++++++++ 2 files changed, 86 insertions(+) create mode 100755 test/box-tap/gh-5203-json-exports.test.lua diff --git a/src/exports.h b/src/exports.h index 7cf283e5b..83c90a8bc 100644 --- a/src/exports.h +++ b/src/exports.h @@ -152,6 +152,10 @@ EXPORT(ibuf_create) EXPORT(ibuf_destroy) EXPORT(ibuf_reinit) EXPORT(ibuf_reserve_slow) +EXPORT(json_lexer_next_token) +EXPORT(json_path_cmp) +EXPORT(json_path_validate) +EXPORT(json_path_multikey_offset) EXPORT(lbox_socket_local_resolve) EXPORT(lbox_socket_nonblock) EXPORT(log_format) diff --git a/test/box-tap/gh-5203-json-exports.test.lua b/test/box-tap/gh-5203-json-exports.test.lua new file mode 100755 index 000000000..1b8eb9afa --- /dev/null +++ b/test/box-tap/gh-5203-json-exports.test.lua @@ -0,0 +1,82 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local ffi = require('ffi') +ffi.cdef([[ + void *dlsym(void *handle, const char *symbol); + /** + * Lexer for JSON paths: + * , <.field>, <[123]>, <['field']> and their combinations. + */ + struct json_lexer { + /** Source string. */ + const char *src; + /** Length of string. */ + int src_len; + /** Current lexer's offset in bytes. */ + int offset; + /** Current lexer's offset in symbols. */ + int symbol_count; + /** + * Base field offset for emitted JSON_TOKEN_NUM tokens, + * e.g. 0 for C and 1 for Lua. + */ + int index_base; + }; + + enum json_token_type { + JSON_TOKEN_NUM, + JSON_TOKEN_STR, + JSON_TOKEN_ANY, + /** Lexer reached end of path. */ + JSON_TOKEN_END, + }; + + int + json_lexer_next_token(struct json_lexer *lexer, struct json_token *token); + + int + json_path_cmp(const char *a, int a_len, const char *b, int b_len, + int index_base); + + int + json_path_validate(const char *path, int path_len, int index_base); + + int + json_path_multikey_offset(const char *path, int path_len, int index_base); +]]) + +local test = tap.test('json-features') +test:plan(1) + +local RTLD_DEFAULT +-- See `man 3 dlsym`: +-- RTLD_DEFAULT +-- Find the first occurrence of the desired symbol using the default +-- shared object search order. The search will include global symbols +-- in the executable and its dependencies, as well as symbols in shared +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. +if jit.os == "OSX" then + RTLD_DEFAULT = ffi.cast("void *", -2LL) +else + RTLD_DEFAULT = ffi.cast("void *", 0LL) +end + +local json_symbols = { + 'json_lexer_next_token', + 'json_path_cmp', + 'json_path_validate', + 'json_path_multikey_offset', +} + +test:test('json_symbols', function(t) + t:plan(#json_symbols) + for _, sym in ipairs(json_symbols) do + t:ok( + ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, + ('Symbol %q found'):format(sym) + ) + end +end) + +os.exit(test:check() and 0 or 1) -- 2.23.0 From tsafin at tarantool.org Fri Aug 21 19:50:20 2020 From: tsafin at tarantool.org (Timur Safin) Date: Fri, 21 Aug 2020 19:50:20 +0300 Subject: [Tarantool-patches] [PATCH] exports: allow to use json tools via FFI In-Reply-To: <20200821162607.68179-1-olegrok@tarantool.org> References: <20200821162607.68179-1-olegrok@tarantool.org> Message-ID: <000001d677db$305d4390$9117cab0$@tarantool.org> LGTM as fairly trivial patch Thanks, Timur : From: olegrok at tarantool.org : Subject: [PATCH] exports: allow to use json tools via FFI : : From: Oleg Babin : : This patch exports some json functions to be used via FFI. We solve : following problem: currently we don't have any tools to inspect : jsonpaths. E.g. user wants to ban or restrict an access to some : tuple fields, it could be some system fields. Tarantool doesn't : have hidden fields and to solve such problem we should fairly : parse input jsonpaths. Before this patch user should write its own : or some external tools. This patch allows to use functions from : built-in json-lexer directly from Tarantool via FFI. : : Part of #5203 : --- : Issue: https://github.com/tarantool/tarantool/issues/5203 : Branch: https://github.com/tarantool/tarantool/tree/olegrok/5203-expose- : json-helpers : Also issue contains an example of usage this feature. : : @Changelog: : - Some symbols from tarantool json moudule are exported now : (gh-5203). : : src/exports.h | 4 ++ : test/box-tap/gh-5203-json-exports.test.lua | 82 ++++++++++++++++++++++ : 2 files changed, 86 insertions(+) : create mode 100755 test/box-tap/gh-5203-json-exports.test.lua : : diff --git a/src/exports.h b/src/exports.h : index 7cf283e5b..83c90a8bc 100644 : --- a/src/exports.h : +++ b/src/exports.h : @@ -152,6 +152,10 @@ EXPORT(ibuf_create) : EXPORT(ibuf_destroy) : EXPORT(ibuf_reinit) : EXPORT(ibuf_reserve_slow) : +EXPORT(json_lexer_next_token) : +EXPORT(json_path_cmp) : +EXPORT(json_path_validate) : +EXPORT(json_path_multikey_offset) : EXPORT(lbox_socket_local_resolve) : EXPORT(lbox_socket_nonblock) : EXPORT(log_format) : diff --git a/test/box-tap/gh-5203-json-exports.test.lua b/test/box-tap/gh- : 5203-json-exports.test.lua : new file mode 100755 : index 000000000..1b8eb9afa : --- /dev/null : +++ b/test/box-tap/gh-5203-json-exports.test.lua : @@ -0,0 +1,82 @@ : +#!/usr/bin/env tarantool : + : +local tap = require('tap') : +local ffi = require('ffi') : +ffi.cdef([[ : + void *dlsym(void *handle, const char *symbol); : + /** : + * Lexer for JSON paths: : + * , <.field>, <[123]>, <['field']> and their combinations. : + */ : + struct json_lexer { : + /** Source string. */ : + const char *src; : + /** Length of string. */ : + int src_len; : + /** Current lexer's offset in bytes. */ : + int offset; : + /** Current lexer's offset in symbols. */ : + int symbol_count; : + /** : + * Base field offset for emitted JSON_TOKEN_NUM tokens, : + * e.g. 0 for C and 1 for Lua. : + */ : + int index_base; : + }; : + : + enum json_token_type { : + JSON_TOKEN_NUM, : + JSON_TOKEN_STR, : + JSON_TOKEN_ANY, : + /** Lexer reached end of path. */ : + JSON_TOKEN_END, : + }; : + : + int : + json_lexer_next_token(struct json_lexer *lexer, struct json_token : *token); : + : + int : + json_path_cmp(const char *a, int a_len, const char *b, int b_len, : + int index_base); : + : + int : + json_path_validate(const char *path, int path_len, int index_base); : + : + int : + json_path_multikey_offset(const char *path, int path_len, int : index_base); : +]]) : + : +local test = tap.test('json-features') : +test:plan(1) : + : +local RTLD_DEFAULT : +-- See `man 3 dlsym`: : +-- RTLD_DEFAULT : +-- Find the first occurrence of the desired symbol using the default : +-- shared object search order. The search will include global symbols : +-- in the executable and its dependencies, as well as symbols in shared : +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. : +if jit.os == "OSX" then : + RTLD_DEFAULT = ffi.cast("void *", -2LL) : +else : + RTLD_DEFAULT = ffi.cast("void *", 0LL) : +end : + : +local json_symbols = { : + 'json_lexer_next_token', : + 'json_path_cmp', : + 'json_path_validate', : + 'json_path_multikey_offset', : +} : + : +test:test('json_symbols', function(t) : + t:plan(#json_symbols) : + for _, sym in ipairs(json_symbols) do : + t:ok( : + ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, : + ('Symbol %q found'):format(sym) : + ) : + end : +end) : + : +os.exit(test:check() and 0 or 1) : -- : 2.23.0 From v.shpilevoy at tarantool.org Sat Aug 22 00:56:58 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Fri, 21 Aug 2020 23:56:58 +0200 Subject: [Tarantool-patches] [PATCH] exports: allow to use json tools via FFI In-Reply-To: <20200821162607.68179-1-olegrok@tarantool.org> References: <20200821162607.68179-1-olegrok@tarantool.org> Message-ID: <8b318f99-f2f3-9671-96e6-0119781c39d1@tarantool.org> Hi! Thanks for the patch! On 21.08.2020 18:26, olegrok at tarantool.org wrote: > From: Oleg Babin > > This patch exports some json functions to be used via FFI. We solve > following problem: currently we don't have any tools to inspect > jsonpaths. E.g. user wants to ban or restrict an access to some > tuple fields, it could be some system fields. Tarantool doesn't > have hidden fields and to solve such problem we should fairly > parse input jsonpaths. Before this patch user should write its own > or some external tools. This patch allows to use functions from > built-in json-lexer directly from Tarantool via FFI. I understand the problem - you don't want to implement your own tool, but you must realize, that exporting some internal structs and functions via FFI is not safe. We won't care about "backward compatibility" of internal functions. And JSON code is quite mutable, it may change in future, if we will rework multikeys, for example. And that will make your code unusable on newer versions. Even probably inside one version, because we won't bump version number just for the sake of internal code change. Even worse is that you took not only general stuff like json_lexer_next_token, but also json_path_multikey_offset - this is heavily internal, why do you need it? I can't upvote this patch, sorry. I propose you to implement a JSON path parser in Lua, or copy-paste the C code from json.h/c. and create your own C module. The whole json.c is 666 lines, including the multikey support and json trees, which occupy most of the space and you don't need them. The parser is trivial, one day work to implement it from the scratch with the tests. Please, don't use the internal structs and functions. You can't just export everything you need if by luck it happened we have it somewhere internally implemented in C. It is not safe. The only thing you need to implement the same JSON path parser is utf8 module, and you have it. From v.shpilevoy at tarantool.org Sat Aug 22 00:59:44 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Fri, 21 Aug 2020 23:59:44 +0200 Subject: [Tarantool-patches] [PATCH v9 5/7] applier: process synchro requests without txn engine In-Reply-To: <1597999873.222856578@f425.i.mail.ru> References: <20200819213442.1099018-1-gorcunov@gmail.com> <20200819213442.1099018-6-gorcunov@gmail.com> <1597999873.222856578@f425.i.mail.ru> Message-ID: Hi! Thanks for the review! > diff --git a/src/box/applier.cc b/src/box/applier.cc > index 1387d518c..c1d07ca54 100644 > --- a/src/box/applier.cc > +++ b/src/box/applier.cc > @@ -847,13 +923,26 @@ applier_apply_tx(struct stailq *rows) > ? } > ? } > ? > + if (unlikely(iproto_type_is_synchro_request(first_row->type))) { > + /* > + * Synchro messages are not transactions, in terms > + * of DML. Always sent and written isolated from > + * each other. > + */ > + assert(first_row == last_row); > + if (apply_synchro_row(first_row) != 0) > + diag_raise(); > + goto success; > + } > + > ? /** > ? * Explicitly begin the transaction so that we can > ? * control fiber->gc life cycle and, in case of apply > ? * conflict safely access failed xrow object and allocate > ? * IPROTO_NOP on gc. > ? */ > - struct txn *txn = txn_begin(); > + struct txn *txn; > + txn = txn_begin(); > > ? > Why this change? In C++ you can't declare and assign variables bypassing labels (at least in clang). It won't compile. Here a new label is added - 'success:'. But it appeared, that it still allows to declare a variable and assign it on a next line. So here it is done. From v.shpilevoy at tarantool.org Sat Aug 22 01:46:02 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 00:46:02 +0200 Subject: [Tarantool-patches] [PATCH 1/1] box: introduce space:alter() Message-ID: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> There was no way to change certain space parameters without its recreation or manual update of internal system space _space. Even if some of them were legal to update: field_count, owner, flag of being temporary, is_sync flag. The patch introduces function space:alter(), which accepts a subset of parameters from box.schema.space.create which are mutable, and 'name' parameter. There is a method space:rename(), but still the parameter is added to space:alter() too, to be consistent with index:alter(), which also accepts a new name. Closes #5155 @TarantoolBot document Title: New function space:alter(options) Space objects in Lua (stored in `box.space` table) now have a new method: `space:alter(options)`. The method accepts a table with parameters `field_count`, `user`, `format`, `temporary`, `is_sync`, and `name`. All parameters have the same meaning as in `box.schema.space.create(name, options)`. Note, `name` parameter in `box.schema.space.create` is separated from `options` table. It is not so in `space:alter(options)` - here all parameters are specified in the `options` table. The function does not return anything in case of success, and throws an error when fails. >From 'Synchronous replication' page, from 'Limitations and known problems' it is necessary to delete the note about "no way to enable synchronous replication for existing spaces". Instead it is necessary to say, that it can be enabled using `space:alter({is_sync = true})`. And can be disabled by setting `is_sync = false`. https://www.tarantool.io/en/doc/2.5/book/replication/repl_sync/#limitations-and-known-problems The function will appear in >= 2.5.2. --- Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5155-space-alter Issue: https://github.com/tarantool/tarantool/issues/5155 @ChangeLog * New function `space:alter(options)` to change some space settings without recreation nor touching `_space` space. src/box/lua/schema.lua | 63 +++++++ test/box/alter.result | 378 ++++++++++++++++++++++++++++++++++++++++ test/box/alter.test.lua | 142 +++++++++++++++ 3 files changed, 583 insertions(+) diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua index 3c3fa3ef8..7fe1ea7a2 100644 --- a/src/box/lua/schema.lua +++ b/src/box/lua/schema.lua @@ -539,6 +539,64 @@ box.schema.space.rename = function(space_id, space_name) _space:update(space_id, {{"=", 3, space_name}}) end +local alter_space_template = { + field_count = 'number', + user = 'string, number', + format = 'table', + temporary = 'boolean', + is_sync = 'boolean', + name = 'string', +} + +box.schema.space.alter = function(space_id, options) + local space = box.space[space_id] + if not space then + box.error(box.error.NO_SUCH_SPACE, '#'..tostring(space_id)) + end + check_param_table(options, alter_space_template) + + local _space = box.space._space + local tuple = _space:get({space.id}) + assert(tuple ~= nil) + + local owner + if options.user then + owner = user_or_role_resolve(options.user) + if not owner then + box.error(box.error.NO_SUCH_USER, options.user) + end + else + owner = tuple.owner + end + + local name = options.name or tuple.name + local field_count = options.field_count or tuple.field_count + local flags = tuple.flags + + if options.temporary ~= nil then + flags.temporary = options.temporary + end + + if options.is_sync ~= nil then + flags.is_sync = options.is_sync + end + + local format + if options.format ~= nil then + format = update_format(options.format) + else + format = tuple.format + end + + tuple = tuple:totable() + tuple[2] = owner + tuple[3] = name + tuple[5] = field_count + tuple[6] = flags + tuple[7] = format + _space:replace(tuple) +end + box.schema.index = {} local function update_index_parts_1_6_0(parts) @@ -1718,6 +1776,11 @@ space_mt.rename = function(space, name) check_space_exists(space) return box.schema.space.rename(space.id, name) end +space_mt.alter = function(space, options) + check_space_arg(space, 'alter') + check_space_exists(space) + return box.schema.space.alter(space.id, options) +end space_mt.create_index = function(space, name, options) check_space_arg(space, 'create_index') check_space_exists(space) diff --git a/test/box/alter.result b/test/box/alter.result index f150faac1..237c2d8aa 100644 --- a/test/box/alter.result +++ b/test/box/alter.result @@ -1181,3 +1181,381 @@ s:drop() box.internal.collation.drop('test') --- ... +-- +-- gh-5155: space:alter(), added in order to make it possible to turn on/off +-- is_sync, but it is handy for other options too. +-- +s = box.schema.create_space('test') +--- +... +pk = s:create_index('pk') +--- +... +-- Bad usage. +s:alter({unknown = true}) +--- +- error: Illegal parameters, unexpected option 'unknown' +... +ok, err = pcall(s.alter, {}) +--- +... +assert(err:match("Use space:alter") ~= nil) +--- +- true +... +-- Alter field_count. +s:replace{1} +--- +- [1] +... +-- Can't update on non-empty space. +s:alter({field_count = 2}) +--- +- error: Tuple field count 1 does not match space field count 2 +... +s:delete{1} +--- +- [1] +... +-- Can update on empty space. +s:alter({field_count = 2}) +--- +... +s:replace{1} +--- +- error: Tuple field count 1 does not match space field count 2 +... +s:replace{1, 1} +--- +- [1, 1] +... +-- When not specified or nil - ignored. +s:alter({field_count = nil}) +--- +... +s:replace{2} +--- +- error: Tuple field count 1 does not match space field count 2 +... +s:replace{2, 2} +--- +- [2, 2] +... +-- Set to 0 drops the restriction. +s:alter({field_count = 0}) +--- +... +s:truncate() +--- +... +s:replace{1} +--- +- [1] +... +s:delete{1} +--- +- [1] +... +-- Invalid values. +s:alter({field_count = box.NULL}) +--- +- error: Illegal parameters, options parameter 'field_count' should be of type number +... +s:alter({field_count = 'string'}) +--- +- error: Illegal parameters, options parameter 'field_count' should be of type number +... +-- Alter owner. +owner1 = box.space._space:get{s.id}.owner +--- +... +box.schema.user.create('test') +--- +... +-- When not specified or nil - ignored. +s:alter({user = nil}) +--- +... +owner2 = box.space._space:get{s.id}.owner +--- +... +assert(owner2 == owner1) +--- +- true +... +s:alter({user = 'test'}) +--- +... +owner2 = box.space._space:get{s.id}.owner +--- +... +assert(owner2 ~= owner1) +--- +- true +... +s:alter({user = owner1}) +--- +... +owner2 = box.space._space:get{s.id}.owner +--- +... +assert(owner2 == owner1) +--- +- true +... +box.schema.user.drop('test') +--- +... +-- Invalid values. +s:alter({user = box.NULL}) +--- +- error: 'Illegal parameters, options parameter ''user'' should be one of types: string, + number' +... +s:alter({user = true}) +--- +- error: 'Illegal parameters, options parameter ''user'' should be one of types: string, + number' +... +s:alter({user = 'not_existing'}) +--- +- error: User 'not_existing' is not found +... +-- Alter format. +format = {{name = 'field1', type = 'unsigned'}} +--- +... +s:alter({format = format}) +--- +... +s:format() +--- +- [{'name': 'field1', 'type': 'unsigned'}] +... +-- When not specified or nil - ignored. +s:alter({format = nil}) +--- +... +s:format() +--- +- [{'name': 'field1', 'type': 'unsigned'}] +... +t = s:replace{1} +--- +... +assert(t.field1 == 1) +--- +- true +... +s:alter({format = {}}) +--- +... +assert(t.field1 == nil) +--- +- true +... +s:delete{1} +--- +- [1] +... +-- Invalid values. +s:alter({format = box.NULL}) +--- +- error: Illegal parameters, options parameter 'format' should be of type table +... +s:alter({format = true}) +--- +- error: Illegal parameters, options parameter 'format' should be of type table +... +s:alter({format = {{{1, 2, 3, 4}}}}) +--- +- error: 'Illegal parameters, format[1]: name (string) is expected' +... +-- +-- Alter temporary. +-- +s:alter({temporary = true}) +--- +... +assert(s.temporary) +--- +- true +... +-- When not specified or nil - ignored. +s:alter({temporary = nil}) +--- +... +assert(s.temporary) +--- +- true +... +s:alter({temporary = false}) +--- +... +assert(not s.temporary) +--- +- true +... +-- Ensure absence is not treated like 'true'. +s:alter({temporary = nil}) +--- +... +assert(not s.temporary) +--- +- true +... +-- Invalid values. +s:alter({temporary = box.NULL}) +--- +- error: Illegal parameters, options parameter 'temporary' should be of type boolean +... +s:alter({temporary = 100}) +--- +- error: Illegal parameters, options parameter 'temporary' should be of type boolean +... +-- +-- Alter is_sync. +-- +old_synchro_quorum = box.cfg.replication_synchro_quorum +--- +... +old_synchro_timeout = box.cfg.replication_synchro_timeout +--- +... +box.cfg{ \ + replication_synchro_quorum = 2, \ + replication_synchro_timeout = 0.001, \ +} +--- +... +s:alter({is_sync = true}) +--- +... +assert(s.is_sync) +--- +- true +... +s:replace{1} +--- +- error: Quorum collection for a synchronous transaction is timed out +... +-- When not specified or nil - ignored. +s:alter({is_sync = nil}) +--- +... +assert(s.is_sync) +--- +- true +... +s:alter({is_sync = false}) +--- +... +assert(not s.is_sync) +--- +- true +... +-- Ensure absence is not treated like 'true'. +s:alter({is_sync = nil}) +--- +... +assert(not s.is_sync) +--- +- true +... +-- Invalid values. +s:alter({is_sync = box.NULL}) +--- +- error: Illegal parameters, options parameter 'is_sync' should be of type boolean +... +s:alter({is_sync = 100}) +--- +- error: Illegal parameters, options parameter 'is_sync' should be of type boolean +... +s:replace{1} +--- +- [1] +... +s:delete{1} +--- +- [1] +... +box.cfg{ \ + replication_synchro_quorum = old_synchro_quorum, \ + replication_synchro_timeout = old_synchro_timeout, \ +} +--- +... +-- Alter name. +s:alter({name = 'test2'}) +--- +... +assert(box.space.test2 ~= nil) +--- +- true +... +assert(box.space.test == nil) +--- +- true +... +assert(s.name == 'test2') +--- +- true +... +-- When not specified or nil - ignored. +s:alter({name = nil}) +--- +... +assert(box.space.test2 ~= nil) +--- +- true +... +assert(box.space.test == nil) +--- +- true +... +assert(s.name == 'test2') +--- +- true +... +s:alter({name = '_space'}) +--- +- error: Duplicate key exists in unique index 'name' in space '_space' +... +s:alter({name = 'test'}) +--- +... +assert(box.space.test ~= nil) +--- +- true +... +assert(box.space.test2 == nil) +--- +- true +... +assert(s.name == 'test') +--- +- true +... +-- Invalid values. +s:alter({name = box.NULL}) +--- +- error: Illegal parameters, options parameter 'name' should be of type string +... +s:alter({name = 100}) +--- +- error: Illegal parameters, options parameter 'name' should be of type string +... +s:drop() +--- +... +s:alter({}) +--- +- error: Space 'test' does not exist +... +ok, err = pcall(box.schema.space.alter, s.id, {}) +--- +... +assert(err:match('does not exist') ~= nil) +--- +- true +... diff --git a/test/box/alter.test.lua b/test/box/alter.test.lua index 3cb0c4f84..abd08e2fa 100644 --- a/test/box/alter.test.lua +++ b/test/box/alter.test.lua @@ -478,3 +478,145 @@ validate_indexes() s:drop() box.internal.collation.drop('test') + +-- +-- gh-5155: space:alter(), added in order to make it possible to turn on/off +-- is_sync, but it is handy for other options too. +-- +s = box.schema.create_space('test') +pk = s:create_index('pk') + +-- Bad usage. +s:alter({unknown = true}) +ok, err = pcall(s.alter, {}) +assert(err:match("Use space:alter") ~= nil) + +-- Alter field_count. +s:replace{1} +-- Can't update on non-empty space. +s:alter({field_count = 2}) +s:delete{1} +-- Can update on empty space. +s:alter({field_count = 2}) +s:replace{1} +s:replace{1, 1} +-- When not specified or nil - ignored. +s:alter({field_count = nil}) +s:replace{2} +s:replace{2, 2} +-- Set to 0 drops the restriction. +s:alter({field_count = 0}) +s:truncate() +s:replace{1} +s:delete{1} +-- Invalid values. +s:alter({field_count = box.NULL}) +s:alter({field_count = 'string'}) + +-- Alter owner. +owner1 = box.space._space:get{s.id}.owner +box.schema.user.create('test') +-- When not specified or nil - ignored. +s:alter({user = nil}) +owner2 = box.space._space:get{s.id}.owner +assert(owner2 == owner1) +s:alter({user = 'test'}) +owner2 = box.space._space:get{s.id}.owner +assert(owner2 ~= owner1) +s:alter({user = owner1}) +owner2 = box.space._space:get{s.id}.owner +assert(owner2 == owner1) +box.schema.user.drop('test') +-- Invalid values. +s:alter({user = box.NULL}) +s:alter({user = true}) +s:alter({user = 'not_existing'}) + +-- Alter format. +format = {{name = 'field1', type = 'unsigned'}} +s:alter({format = format}) +s:format() +-- When not specified or nil - ignored. +s:alter({format = nil}) +s:format() +t = s:replace{1} +assert(t.field1 == 1) +s:alter({format = {}}) +assert(t.field1 == nil) +s:delete{1} +-- Invalid values. +s:alter({format = box.NULL}) +s:alter({format = true}) +s:alter({format = {{{1, 2, 3, 4}}}}) + +-- +-- Alter temporary. +-- +s:alter({temporary = true}) +assert(s.temporary) +-- When not specified or nil - ignored. +s:alter({temporary = nil}) +assert(s.temporary) +s:alter({temporary = false}) +assert(not s.temporary) +-- Ensure absence is not treated like 'true'. +s:alter({temporary = nil}) +assert(not s.temporary) +-- Invalid values. +s:alter({temporary = box.NULL}) +s:alter({temporary = 100}) + +-- +-- Alter is_sync. +-- +old_synchro_quorum = box.cfg.replication_synchro_quorum +old_synchro_timeout = box.cfg.replication_synchro_timeout +box.cfg{ \ + replication_synchro_quorum = 2, \ + replication_synchro_timeout = 0.001, \ +} +s:alter({is_sync = true}) +assert(s.is_sync) +s:replace{1} +-- When not specified or nil - ignored. +s:alter({is_sync = nil}) +assert(s.is_sync) +s:alter({is_sync = false}) +assert(not s.is_sync) +-- Ensure absence is not treated like 'true'. +s:alter({is_sync = nil}) +assert(not s.is_sync) +-- Invalid values. +s:alter({is_sync = box.NULL}) +s:alter({is_sync = 100}) +s:replace{1} +s:delete{1} +box.cfg{ \ + replication_synchro_quorum = old_synchro_quorum, \ + replication_synchro_timeout = old_synchro_timeout, \ +} + +-- Alter name. +s:alter({name = 'test2'}) +assert(box.space.test2 ~= nil) +assert(box.space.test == nil) +assert(s.name == 'test2') +-- When not specified or nil - ignored. +s:alter({name = nil}) +assert(box.space.test2 ~= nil) +assert(box.space.test == nil) +assert(s.name == 'test2') +s:alter({name = '_space'}) +s:alter({name = 'test'}) +assert(box.space.test ~= nil) +assert(box.space.test2 == nil) +assert(s.name == 'test') +-- Invalid values. +s:alter({name = box.NULL}) +s:alter({name = 100}) + +s:drop() + +s:alter({}) +ok, err = pcall(box.schema.space.alter, s.id, {}) +assert(err:match('does not exist') ~= nil) -- 2.21.1 (Apple Git-122.3) From v.shpilevoy at tarantool.org Sat Aug 22 17:23:35 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:23:35 +0200 Subject: [Tarantool-patches] [PATCH v2 01/10] sql: do not return UNSIGNED in built-in functions In-Reply-To: <3a825a5a3d61933f5e76d6594f33ad304dd07559.1597417321.git.imeevma@gmail.com> References: <3a825a5a3d61933f5e76d6594f33ad304dd07559.1597417321.git.imeevma@gmail.com> Message-ID: <737d1ce0-0d00-84f8-412d-fb805ebbdbbc@tarantool.org> Hi! Thanks for the patch! On 14.08.2020 17:04, imeevma at tarantool.org wrote: > This patch forces functions to return INTEGER instead of UNSIGNED. Why? I don't see a reason why length() would need to return int instead of uint. > diff --git a/test/sql/types.result b/test/sql/types.result > index 442245186..95f7713e8 100644 > --- a/test/sql/types.result > +++ b/test/sql/types.result > @@ -2795,3 +2795,15 @@ box.execute([[DROP TABLE ts;]]) > --- > - row_count: 1 > ... > +-- > +-- gh-4159: Make sure that functions returns values of type INTEGER > +-- instead of values of type UNSIGNED. What are the other functions besides length()? > +-- > +box.execute([[SELECT typeof(length('abc'));]]) From v.shpilevoy at tarantool.org Sat Aug 22 17:24:12 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:24:12 +0200 Subject: [Tarantool-patches] [PATCH v2 02/10] sql: fix functions return types In-Reply-To: <804a579f9ba60138c0bf579da4c6a3c253d2c272.1597417321.git.imeevma@gmail.com> References: <804a579f9ba60138c0bf579da4c6a3c253d2c272.1597417321.git.imeevma@gmail.com> Message-ID: Thanks for the patch! On 14.08.2020 17:04, imeevma at tarantool.org wrote: > This patch fixes incorrect return types in SQL built-in function definitions. Why are they incorrect? You need to provide more descriptive message. It took for me some time to scan the sqlite doc to find what do these functions do and why do you change their types. > --- > src/box/sql/func.c | 10 +++++----- > test/sql/types.result | 4 ++-- > 2 files changed, 7 insertions(+), 7 deletions(-) > > diff --git a/src/box/sql/func.c b/src/box/sql/func.c > index 487cdafe1..affb285aa 100644 > --- a/src/box/sql/func.c > +++ b/src/box/sql/func.c "UNLIKELY" still returns boolean. But according to the doc, it should return the single argument unchanged. So also should be SCALAR. Please, check the other functions too. Also you need to add tests on the changed types. For example, you changed likely() return type, but I don't see any tests failing. So looks like it is not covered. From v.shpilevoy at tarantool.org Sat Aug 22 17:25:16 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:25:16 +0200 Subject: [Tarantool-patches] [PATCH v2 00/10] sql: properly check arguments types of built-in functions In-Reply-To: References: Message-ID: <32b0f61a-5bba-6447-37e7-545482debafd@tarantool.org> Thanks for the patchset! Should there be a docbot request somewhere? It seems the visible behaviour is changed. From v.shpilevoy at tarantool.org Sat Aug 22 17:26:21 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:26:21 +0200 Subject: [Tarantool-patches] [PATCH v2 03/10] sql: change signature of trim() In-Reply-To: References: Message-ID: <8e1ae2cb-5bcb-16cd-7086-2cd6e8dbeca4@tarantool.org> Thanks for the patch! On 14.08.2020 17:04, imeevma at tarantool.org wrote: > This patch changes the signature of the SQL built-in trim() function. > This makes it easier to define a function in _func and fixes a bug where > the function loses collation when the BOTH, LEADING, or TRAILING > keywords are specified. I am not sure I understand. Did you break the backward compatibility by changing the public function? Or did you change only internal implementation? What exactly has changed? Where is refactoring, and where is the bugfix? What is so hard about its signature now, that it does not allow to define it in _func? If this is a bugfix, it should be on a separate branch, with changelog, so as it could be pushed to the older versions, according to our policy. > diff --git a/src/box/sql/func.c b/src/box/sql/func.c > index affb285aa..e5da21191 100644 > --- a/src/box/sql/func.c > +++ b/src/box/sql/func.c > @@ -1776,32 +1772,30 @@ static void > trim_func_two_args(struct sql_context *context, sql_value *arg1, > sql_value *arg2) > { > - const unsigned char *input_str, *trim_set; > - if ((input_str = sql_value_text(arg2)) == NULL) > - return; > - > - int input_str_sz = sql_value_bytes(arg2); > - if (sql_value_type(arg1) == MP_INT || sql_value_type(arg1) == MP_UINT) { > - uint8_t len_one = 1; > - trim_procedure(context, sql_value_int(arg1), > - (const unsigned char *) " ", &len_one, 1, > - input_str, input_str_sz); > - } else if ((trim_set = sql_value_text(arg1)) != NULL) { > - int trim_set_sz = sql_value_bytes(arg1); > - uint8_t *char_len; > - int char_cnt = trim_prepare_char_len(context, trim_set, > - trim_set_sz, &char_len); > - if (char_cnt == -1) > - return; > - trim_procedure(context, TRIM_BOTH, trim_set, char_len, char_cnt, > - input_str, input_str_sz); > - sql_free(char_len); > - } > + assert(sql_value_type(arg2) == MP_UINT); > + enum mp_type type = sql_value_type(arg1); > + if (type == MP_NIL) > + return sql_result_null(context); > + const unsigned char *input_str = sql_value_text(arg1); > + const unsigned char *trim_set; > + > + int input_str_sz = sql_value_bytes(arg1); > + uint8_t len_one = 1; > + if (type == MP_BIN) > + trim_set = (const unsigned char *) "\0"; > + else > + trim_set = (const unsigned char *) " "; > + trim_procedure(context, sql_value_int(arg2), trim_set, &len_one, 1, > + input_str, input_str_sz); Why did you move handling of the "TRIM( FROM )" case to another function? That breaks the trim functions idea of having trim_func_one_arg, trim_func_two_args, and trim_func_three_args separated. After your patch trim_func_three_args handles both 2 and 3 arguments, and trim_func_two_args handles not both options with 2 arguments. You either need to keep the different argument count handling inside the existing functions, or change their names somehow to reflect the new behaviour. From v.shpilevoy at tarantool.org Sat Aug 22 17:28:53 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:28:53 +0200 Subject: [Tarantool-patches] [PATCH v2 04/10] box: add new options for functions In-Reply-To: <65c71dc224b0365954c396486c87738fb9ca5baf.1597417321.git.imeevma@gmail.com> References: <65c71dc224b0365954c396486c87738fb9ca5baf.1597417321.git.imeevma@gmail.com> Message-ID: Thanks for the patch! On 14.08.2020 17:04, imeevma at tarantool.org wrote: > The has_vararg option allows us to work with functions with a variable > number of arguments. This is required for built-in SQL functions. > > Suppose this option is TRUE for a built-in SQL function. Then: > 1) If param_list is empty, all arguments can be of any type. > 2) If the length of param_list is not less than the number of the given > arguments, the types of the given arguments must be compatible with the > corresponding types described in param_list. > 3) If the length of param_list is less than the number of given > arguments, the rest of the arguments must be compatible with the last > type in param_list. How is it going to be used? For example, how will it work with printf(), where tail of the arguments can be of any type? Do you define it as scalar then? > The is_overloaded option allows us to deal with functions that can take > STRING or VARBINARY arguments. In case the first of these arguments is > of type VARBINARY, we use the overloaded version of the function. By > default, we use the STRING version of the function. > > The has_overload option indicates that the function has an overloaded > version that accepts VARBINARY. See an explanation of the is_overloaded > option. In addition to this whole idea with overloads being a huge crutch which also affects the public API with adding some new intricate options, it also looks like has_overload is entirely a runtime option. Why is it even stored in _func? What if I have a function taking STRING, and I want to add VARBINARY, I need to update 2 records in _func for that? Looks really inconvenient. I would try to think more into how not to store these options in _func. Because you are trying to describe some code from the executable's file using the storage. From there most of the problems arise - new complicated options for _func, hard upgrade process, and unnecessary changes of the schema. Why can't you specify all these flags inside sql_builtins array, and copy them into func_def from there? It already works fine this way. From v.shpilevoy at tarantool.org Sat Aug 22 17:29:47 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:29:47 +0200 Subject: [Tarantool-patches] [PATCH v2 06/10] sql: add overloaded versions of the functions In-Reply-To: <34e3d319f2955e21f9fc1d358b778128cea9ed79.1597417321.git.imeevma@gmail.com> References: <34e3d319f2955e21f9fc1d358b778128cea9ed79.1597417321.git.imeevma@gmail.com> Message-ID: <81667c8f-41b6-6d75-a781-3b8f34a50466@tarantool.org> Thanks for the patch! On 14.08.2020 17:05, imeevma at tarantool.org wrote: > This patch adds overload function definitions for the length(), > substr(), trim(), and position() functions. You should describe how the overload works. How name collisions are resolved. Also I want to say I am against these 'overloads' for string/varbinary. The code looks ugly with the new argument 'has_blob_arg' and with changing the function name. Also it adds **2** new flags to think about, when you work with the functions. It should be either more generic with proper function names mangling like in C++ (this is also bad), or it should not exist and the types should be resolved inside single function taking both string and blob. Currently the 'overloaded' VARBINARY functions even return exactly the same types as their non-VARBINARY versions (what is probably a bug - SUBSTR on blob returns string?) anyway. Or it should be separated into different functions defined explicitly. So as user would be aware he needs to use LENGTH for strings, LENGTH_BIN for byte length of blobs, etc. Otherwise you are trying to invent some kind of implicit casts, but you cast functions instead of types. Which is not better. From v.shpilevoy at tarantool.org Sat Aug 22 17:30:17 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:30:17 +0200 Subject: [Tarantool-patches] [PATCH v2 07/10] sql: move built-in function definitions in _func In-Reply-To: <1fc825807e36bd97b97a28756b3e9da2c270de5b.1597417321.git.imeevma@gmail.com> References: <1fc825807e36bd97b97a28756b3e9da2c270de5b.1597417321.git.imeevma@gmail.com> Message-ID: <8929d59f-ef32-9bbb-0d1b-b0b4677dc877@tarantool.org> Thanks for the patch! On 14.08.2020 17:05, imeevma at tarantool.org wrote: > This patch moves SQL built-in function definitions to _func. This helps > create an unified way to check the types of arguments. I think this all can be done by fetching more info from sql_builtins array. As I explained in one of the previous emails. > It also allows > users to see these definitions. Also, this patch enables overloading for > length(), trim(), position() and substr() functions. Tbh, I can't imagine what a monster user would need to look at these numerous flags. See no point in exposing it into _func. > diff --git a/src/box/sql/func.c b/src/box/sql/func.c > index ae1842824..1fbffa535 100644 > --- a/src/box/sql/func.c > +++ b/src/box/sql/func.c > @@ -2971,12 +2971,6 @@ func_sql_builtin_new(struct func_def *def) > func->flags = sql_builtins[idx].flags; > func->call = sql_builtins[idx].call; > func->finalize = sql_builtins[idx].finalize; > - def->param_count = sql_builtins[idx].param_count; > - def->is_deterministic = sql_builtins[idx].is_deterministic; > - def->returns = sql_builtins[idx].returns; > - def->aggregate = sql_builtins[idx].aggregate; > - def->exports.sql = sql_builtins[idx].export_to_sql; > - def->opts.has_vararg = sql_builtins[idx].param_count == -1; I think this should be extended, not removed. > return &func->base; > } > From v.shpilevoy at tarantool.org Sat Aug 22 17:30:24 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:30:24 +0200 Subject: [Tarantool-patches] [PATCH v2 08/10] box: add param_list to 'struct func' In-Reply-To: <14bca5aab8484f9fcd0c93b29c183a8133b71d35.1597417321.git.imeevma@gmail.com> References: <14bca5aab8484f9fcd0c93b29c183a8133b71d35.1597417321.git.imeevma@gmail.com> Message-ID: <7a30ccd3-c969-90af-88be-275f06c65ad3@tarantool.org> Thanks for the patch! > diff --git a/src/box/alter.cc b/src/box/alter.cc > index ba96d9c62..0914a7615 100644 > --- a/src/box/alter.cc > +++ b/src/box/alter.cc > @@ -3293,7 +3293,11 @@ func_def_new_from_tuple(struct tuple *tuple) > diag_set(OutOfMemory, def_sz, "malloc", "def"); > return NULL; > } > - auto def_guard = make_scoped_guard([=] { free(def); }); > + def->param_list = NULL; > + auto def_guard = make_scoped_guard([=] { > + free(def->param_list); > + free(def); > + }); Would be better to patch func_def_sizeof() and allocate in one block. From v.shpilevoy at tarantool.org Sat Aug 22 17:31:06 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Sat, 22 Aug 2020 16:31:06 +0200 Subject: [Tarantool-patches] [PATCH v2 10/10] sql: refactor sql/func.c In-Reply-To: <847d1f4dd9b5b582f178e23361bbb746d2f12c04.1597417321.git.imeevma@gmail.com> References: <847d1f4dd9b5b582f178e23361bbb746d2f12c04.1597417321.git.imeevma@gmail.com> Message-ID: <59a52e9c-c405-55e2-55a5-22bac51471e5@tarantool.org> Thanks for the patch! See 3 comments below. On 14.08.2020 17:05, imeevma at tarantool.org wrote: > After changing the way of checking the types of arguments, some of the > code in sql/func.c is no longer used. This patch removes this code. > > Follow-up of #4159 > --- > src/box/sql/func.c | 841 +++++---------------------------------------- > 1 file changed, 87 insertions(+), 754 deletions(-) > > diff --git a/src/box/sql/func.c b/src/box/sql/func.c > index c289f2de0..de1e4d13d 100644 > --- a/src/box/sql/func.c > +++ b/src/box/sql/func.c 1. Why didn't you simplify some funtions? - lengthFunc(). Is it allowed to pass numbers into it legally? Perhaps it is, I just don't remember. - position_func(). It has a some complicated check of argument types. I think now you can make it a bit simpler. - lower/upper(). They still check for non-blob argument. - like(). Also still checks for non-str types. > @@ -504,44 +504,21 @@ absFunc(sql_context * context, int argc, sql_value ** argv) > { > assert(argc == 1); > UNUSED_PARAMETER(argc); > - switch (sql_value_type(argv[0])) { > - case MP_UINT: { > - sql_result_uint(context, sql_value_uint64(argv[0])); > - break; > - } > - case MP_INT: { > + enum mp_type type = sql_value_type(argv[0]); > + if (type == MP_NIL) > + return sql_result_null(context); > + if (type == MP_UINT) > + return sql_result_uint(context, sql_value_uint64(argv[0])); > + if (type == MP_INT) { 2. Would look better as a switch. > int64_t value = sql_value_int64(argv[0]); > assert(value < 0); > - sql_result_uint(context, -value); > - break; > - } > @@ -2229,703 +2189,76 @@ static struct { > uint16_t flags; > void (*call)(sql_context *ctx, int argc, sql_value **argv); > void (*finalize)(sql_context *ctx); > - /** Members below are related to struct func_def. */ > - bool is_deterministic; > - int param_count; > - enum field_type returns; > - enum func_aggregate aggregate; > - bool export_to_sql; > } sql_builtins[] = { > - {.name = "ABS", > - .param_count = 1, > - .returns = FIELD_TYPE_NUMBER, > - .aggregate = FUNC_AGGREGATE_NONE, > - .is_deterministic = true, > - .flags = 0, > - .call = absFunc, > - .finalize = NULL, > - .export_to_sql = true, > - }, { 3. These removals I don't like. I think we don't need to expose anything into _func. All could be done in there instead, and would look simpler, IMO. As I explained in the previous emails in this thread. From sergepetrenko at tarantool.org Sun Aug 23 15:15:45 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Sun, 23 Aug 2020 15:15:45 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_v9_5/7=5D_applier=3A_proce?= =?utf-8?q?ss_synchro_requests_without_txn_engine?= In-Reply-To: References: <20200819213442.1099018-1-gorcunov@gmail.com> <1597999873.222856578@f425.i.mail.ru> Message-ID: <1598184945.737467032@f75.i.mail.ru> ? >???????, 22 ??????? 2020, 0:59 +03:00 ?? Vladislav Shpilevoy : >? >Hi! Thanks for the review! >? >> diff --git a/src/box/applier.cc b/src/box/applier.cc >> index 1387d518c..c1d07ca54 100644 >> --- a/src/box/applier.cc >> +++ b/src/box/applier.cc >> @@ -847,13 +923,26 @@ applier_apply_tx(struct stailq *rows) >> ? } >> ? } >> ? >> + if (unlikely(iproto_type_is_synchro_request(first_row->type))) { >> + /* >> + * Synchro messages are not transactions, in terms >> + * of DML. Always sent and written isolated from >> + * each other. >> + */ >> + assert(first_row == last_row); >> + if (apply_synchro_row(first_row) != 0) >> + diag_raise(); >> + goto success; >> + } >> + >> ? /** >> ? * Explicitly begin the transaction so that we can >> ? * control fiber->gc life cycle and, in case of apply >> ? * conflict safely access failed xrow object and allocate >> ? * IPROTO_NOP on gc. >> ? */ >> - struct txn *txn = txn_begin(); >> + struct txn *txn; >> + txn = txn_begin(); >> >> ? >> Why this change? >In C++ you can't declare and assign variables bypassing labels (at >least in clang). It won't compile. Here a new label is added - >'success:'. > >But it appeared, that it still allows to declare a variable and >assign it on a next line. So here it is done. I see. Thanks for the explanation! ? ? -- Serge?Petrenko ? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Sun Aug 23 15:53:32 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Sun, 23 Aug 2020 15:53:32 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_1/1=5D_box=3A_introduce_sp?= =?utf-8?b?YWNlOmFsdGVyKCk=?= In-Reply-To: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> References: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> Message-ID: <1598187212.879211088@f470.i.mail.ru> Hi!?Thanks for the patch! ? >???????, 22 ??????? 2020, 1:46 +03:00 ?? Vladislav Shpilevoy : >? >There was no way to change certain space parameters without its >recreation or manual update of internal system space _space. Even >if some of them were legal to update: field_count, owner, flag of >being temporary, is_sync flag. > >The patch introduces function space:alter(), which accepts a >subset of parameters from box.schema.space.create which are >mutable, and 'name' parameter. There is a method space:rename(), >but still the parameter is added to space:alter() too, to be >consistent with index:alter(), which also accepts a new name. > >Closes #5155 > >@TarantoolBot document >Title: New function space:alter(options) > >Space objects in Lua (stored in `box.space` table) now have a new >method: `space:alter(options)`. > >The method accepts a table with parameters `field_count`, `user`, >`format`, `temporary`, `is_sync`, and `name`. All parameters have >the same meaning as in `box.schema.space.create(name, options)`. > >Note, `name` parameter in `box.schema.space.create` is separated >from `options` table. It is not so in `space:alter(options)` - >here all parameters are specified in the `options` table. > >The function does not return anything in case of success, and >throws an error when fails. > >From 'Synchronous replication' page, from 'Limitations and known >problems' it is necessary to delete the note about "no way to >enable synchronous replication for existing spaces". Instead it >is necessary to say, that it can be enabled using >`space:alter({is_sync = true})`. And can be disabled by setting >`is_sync = false`. >https://www.tarantool.io/en/doc/2.5/book/replication/repl_sync/#limitations-and-known-problems > >The function will appear in >= 2.5.2. >--- >Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-5155-space-alter >Issue: https://github.com/tarantool/tarantool/issues/5155 > >@ChangeLog >* New function `space:alter(options)` to change some space settings without recreation nor touching `_space` space. > >?src/box/lua/schema.lua | 63 +++++++ >?test/box/alter.result | 378 ++++++++++++++++++++++++++++++++++++++++ >?test/box/alter.test.lua | 142 +++++++++++++++ >?3 files changed, 583 insertions(+) > >diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua >index 3c3fa3ef8..7fe1ea7a2 100644 >--- a/src/box/lua/schema.lua >+++ b/src/box/lua/schema.lua >@@ -539,6 +539,64 @@ box.schema.space.rename = function(space_id, space_name) >?????_space:update(space_id, {{"=", 3, space_name}}) >?end >? >+local alter_space_template = { >+ field_count = 'number', >+ user = 'string, number', >+ format = 'table', >+ temporary = 'boolean', >+ is_sync = 'boolean', >+ name = 'string', >+} >+ >+box.schema.space.alter = function(space_id, options) >+ local space = box.space[space_id] >+ if not space then >+ box.error(box.error.NO_SUCH_SPACE, '#'..tostring(space_id)) >+ end >+ check_param_table(options, alter_space_template) >+ >+ local _space = box.space._space >+ local tuple = _space:get({space.id}) >+ assert(tuple ~= nil) >+ >+ local owner >+ if options.user then >+ owner = user_or_role_resolve(options.user) >+ if not owner then >+ box.error(box.error.NO_SUCH_USER, options.user) >+ end >+ else >+ owner = tuple.owner >+ end >+ >+ local name = options.name or tuple.name >+ local field_count = options.field_count or tuple.field_count >+ local flags = tuple.flags >+ >+ if options.temporary ~= nil then >+ flags.temporary = options.temporary >+ end >+ >+ if options.is_sync ~= nil then >+ flags.is_sync = options.is_sync >+ end >+ >+ local format >+ if options.format ~= nil then >+ format = update_format(options.format) >+ else >+ format = tuple.format >+ end >+ >+ tuple = tuple:totable() >+ tuple[2] = owner >+ tuple[3] = name >+ tuple[5] = field_count >+ tuple[6] = flags >+ tuple[7] = format >+ _space:replace(tuple) >+end >+ >?box.schema.index = {} >? >?local function update_index_parts_1_6_0(parts) >@@ -1718,6 +1776,11 @@ space_mt.rename = function(space, name) >?????check_space_exists(space) >?????return box.schema.space.rename(space.id, name) >?end >+space_mt.alter = function(space, options) >+ check_space_arg(space, 'alter') >+ check_space_exists(space) >+ return box.schema.space.alter(space.id, options) >+end >?space_mt.create_index = function(space, name, options) >?????check_space_arg(space, 'create_index') >?????check_space_exists(space) >diff --git a/test/box/alter.result b/test/box/alter.result >index f150faac1..237c2d8aa 100644 >--- a/test/box/alter.result >+++ b/test/box/alter.result >@@ -1181,3 +1181,381 @@ s:drop() >?box.internal.collation.drop('test') >?--- >?... >+-- >+-- gh-5155: space:alter(), added in order to make it possible to turn on/off >+-- is_sync, but it is handy for other options too. >+-- >+s = box.schema.create_space('test') >+--- >+... >+pk = s:create_index('pk') >+--- >+... >+-- Bad usage. >+s:alter({unknown = true}) >+--- >+- error: Illegal parameters, unexpected option 'unknown' >+... >+ok, err = pcall(s.alter, {}) >+--- >+... >+assert(err:match("Use space:alter") ~= nil) >+--- >+- true >+... >+-- Alter field_count. >+s:replace{1} >+--- >+- [1] >+... >+-- Can't update on non-empty space. >+s:alter({field_count = 2}) >+--- >+- error: Tuple field count 1 does not match space field count 2 >+... >+s:delete{1} >+--- >+- [1] >+... >+-- Can update on empty space. >+s:alter({field_count = 2}) >+--- >+... >+s:replace{1} >+--- >+- error: Tuple field count 1 does not match space field count 2 >+... >+s:replace{1, 1} >+--- >+- [1, 1] >+... >+-- When not specified or nil - ignored. >+s:alter({field_count = nil}) >+--- >+... >+s:replace{2} >+--- >+- error: Tuple field count 1 does not match space field count 2 >+... >+s:replace{2, 2} >+--- >+- [2, 2] >+... >+-- Set to 0 drops the restriction. >+s:alter({field_count = 0}) >+--- >+... >+s:truncate() >+--- >+... >+s:replace{1} >+--- >+- [1] >+... >+s:delete{1} >+--- >+- [1] >+... >+-- Invalid values. >+s:alter({field_count = box.NULL}) >+--- >+- error: Illegal parameters, options parameter 'field_count' should be of type number >+... >+s:alter({field_count = 'string'}) >+--- >+- error: Illegal parameters, options parameter 'field_count' should be of type number >+... >+-- Alter owner. >+owner1 = box.space._space:get{s.id}.owner >+--- >+... >+box.schema.user.create('test') >+--- >+... >+-- When not specified or nil - ignored. >+s:alter({user = nil}) >+--- >+... >+owner2 = box.space._space:get{s.id}.owner >+--- >+... >+assert(owner2 == owner1) >+--- >+- true >+... >+s:alter({user = 'test'}) >+--- >+... >+owner2 = box.space._space:get{s.id}.owner >+--- >+... >+assert(owner2 ~= owner1) >+--- >+- true >+... >+s:alter({user = owner1}) >+--- >+... >+owner2 = box.space._space:get{s.id}.owner >+--- >+... >+assert(owner2 == owner1) >+--- >+- true >+... >+box.schema.user.drop('test') >+--- >+... >+-- Invalid values. >+s:alter({user = box.NULL}) >+--- >+- error: 'Illegal parameters, options parameter ''user'' should be one of types: string, >+ number' >+... >+s:alter({user = true}) >+--- >+- error: 'Illegal parameters, options parameter ''user'' should be one of types: string, >+ number' >+... >+s:alter({user = 'not_existing'}) >+--- >+- error: User 'not_existing' is not found >+... >+-- Alter format. >+format = {{name = 'field1', type = 'unsigned'}} >+--- >+... >+s:alter({format = format}) >+--- >+... >+s:format() >+--- >+- [{'name': 'field1', 'type': 'unsigned'}] >+... >+-- When not specified or nil - ignored. >+s:alter({format = nil}) >+--- >+... >+s:format() >+--- >+- [{'name': 'field1', 'type': 'unsigned'}] >+... >+t = s:replace{1} >+--- >+... >+assert(t.field1 == 1) >+--- >+- true >+... >+s:alter({format = {}}) >+--- >+... >+assert(t.field1 == nil) >+--- >+- true >+... >+s:delete{1} >+--- >+- [1] >+... >+-- Invalid values. >+s:alter({format = box.NULL}) >+--- >+- error: Illegal parameters, options parameter 'format' should be of type table >+... >+s:alter({format = true}) >+--- >+- error: Illegal parameters, options parameter 'format' should be of type table >+... >+s:alter({format = {{{1, 2, 3, 4}}}}) >+--- >+- error: 'Illegal parameters, format[1]: name (string) is expected' >+... >+-- >+-- Alter temporary. >+-- >+s:alter({temporary = true}) >+--- >+... >+assert(s.temporary) >+--- >+- true >+... >+-- When not specified or nil - ignored. >+s:alter({temporary = nil}) >+--- >+... >+assert(s.temporary) >+--- >+- true >+... >+s:alter({temporary = false}) >+--- >+... >+assert(not s.temporary) >+--- >+- true >+... >+-- Ensure absence is not treated like 'true'. >+s:alter({temporary = nil}) >+--- >+... >+assert(not s.temporary) >+--- >+- true >+... >+-- Invalid values. >+s:alter({temporary = box.NULL}) >+--- >+- error: Illegal parameters, options parameter 'temporary' should be of type boolean >+... >+s:alter({temporary = 100}) >+--- >+- error: Illegal parameters, options parameter 'temporary' should be of type boolean >+... >+-- >+-- Alter is_sync. >+-- >+old_synchro_quorum = box.cfg.replication_synchro_quorum >+--- >+... >+old_synchro_timeout = box.cfg.replication_synchro_timeout >+--- >+... >+box.cfg{ \ >+ replication_synchro_quorum = 2, \ >+ replication_synchro_timeout = 0.001, \ >+} >+--- >+... >+s:alter({is_sync = true}) >+--- >+... >+assert(s.is_sync) >+--- >+- true >+... >+s:replace{1} >+--- >+- error: Quorum collection for a synchronous transaction is timed out >+... >+-- When not specified or nil - ignored. >+s:alter({is_sync = nil}) >+--- >+... >+assert(s.is_sync) >+--- >+- true >+... >+s:alter({is_sync = false}) >+--- >+... >+assert(not s.is_sync) >+--- >+- true >+... >+-- Ensure absence is not treated like 'true'. >+s:alter({is_sync = nil}) >+--- >+... >+assert(not s.is_sync) >+--- >+- true >+... >+-- Invalid values. >+s:alter({is_sync = box.NULL}) >+--- >+- error: Illegal parameters, options parameter 'is_sync' should be of type boolean >+... >+s:alter({is_sync = 100}) >+--- >+- error: Illegal parameters, options parameter 'is_sync' should be of type boolean >+... >+s:replace{1} >+--- >+- [1] >+... >+s:delete{1} >+--- >+- [1] >+... >+box.cfg{ \ >+ replication_synchro_quorum = old_synchro_quorum, \ >+ replication_synchro_timeout = old_synchro_timeout, \ >+} >+--- >+... >+-- Alter name. >+s:alter({name = 'test2'}) >+--- >+... >+assert(box.space.test2 ~= nil) >+--- >+- true >+... >+assert(box.space.test == nil) >+--- >+- true >+... >+assert(s.name == 'test2') >+--- >+- true >+... >+-- When not specified or nil - ignored. >+s:alter({name = nil}) >+--- >+... >+assert(box.space.test2 ~= nil) >+--- >+- true >+... >+assert(box.space.test == nil) >+--- >+- true >+... >+assert(s.name == 'test2') >+--- >+- true >+... >+s:alter({name = '_space'}) >+--- >+- error: Duplicate key exists in unique index 'name' in space '_space' >+... >+s:alter({name = 'test'}) >+--- >+... >+assert(box.space.test ~= nil) >+--- >+- true >+... >+assert(box.space.test2 == nil) >+--- >+- true >+... >+assert(s.name == 'test') >+--- >+- true >+... >+-- Invalid values. >+s:alter({name = box.NULL}) >+--- >+- error: Illegal parameters, options parameter 'name' should be of type string >+... >+s:alter({name = 100}) >+--- >+- error: Illegal parameters, options parameter 'name' should be of type string >+... >+s:drop() >+--- >+... >+s:alter({}) >+--- >+- error: Space 'test' does not exist >+... >+ok, err = pcall(box.schema.space.alter, s.id, {}) >+--- >+... >+assert(err:match('does not exist') ~= nil) >+--- >+- true >+... >diff --git a/test/box/alter.test.lua b/test/box/alter.test.lua >index 3cb0c4f84..abd08e2fa 100644 >--- a/test/box/alter.test.lua >+++ b/test/box/alter.test.lua >@@ -478,3 +478,145 @@ validate_indexes() >? >?s:drop() >?box.internal.collation.drop('test') >+ >+-- >+-- gh-5155: space:alter(), added in order to make it possible to turn on/off >+-- is_sync, but it is handy for other options too. >+-- >+s = box.schema.create_space('test') >+pk = s:create_index('pk') >+ >+-- Bad usage. >+s:alter({unknown = true}) >+ok, err = pcall(s.alter, {}) >+assert(err:match("Use space:alter") ~= nil) >+ >+-- Alter field_count. >+s:replace{1} >+-- Can't update on non-empty space. >+s:alter({field_count = 2}) >+s:delete{1} >+-- Can update on empty space. >+s:alter({field_count = 2}) >+s:replace{1} >+s:replace{1, 1} >+-- When not specified or nil - ignored. >+s:alter({field_count = nil}) >+s:replace{2} >+s:replace{2, 2} >+-- Set to 0 drops the restriction. >+s:alter({field_count = 0}) >+s:truncate() >+s:replace{1} >+s:delete{1} >+-- Invalid values. >+s:alter({field_count = box.NULL}) >+s:alter({field_count = 'string'}) >+ >+-- Alter owner. >+owner1 = box.space._space:get{s.id}.owner >+box.schema.user.create('test') >+-- When not specified or nil - ignored. >+s:alter({user = nil}) >+owner2 = box.space._space:get{s.id}.owner >+assert(owner2 == owner1) >+s:alter({user = 'test'}) >+owner2 = box.space._space:get{s.id}.owner >+assert(owner2 ~= owner1) >+s:alter({user = owner1}) >+owner2 = box.space._space:get{s.id}.owner >+assert(owner2 == owner1) >+box.schema.user.drop('test') >+-- Invalid values. >+s:alter({user = box.NULL}) >+s:alter({user = true}) >+s:alter({user = 'not_existing'}) >+ >+-- Alter format. >+format = {{name = 'field1', type = 'unsigned'}} >+s:alter({format = format}) >+s:format() >+-- When not specified or nil - ignored. >+s:alter({format = nil}) >+s:format() >+t = s:replace{1} >+assert(t.field1 == 1) >+s:alter({format = {}}) >+assert(t.field1 == nil) >+s:delete{1} >+-- Invalid values. >+s:alter({format = box.NULL}) >+s:alter({format = true}) >+s:alter({format = {{{1, 2, 3, 4}}}}) >+ >+-- >+-- Alter temporary. >+-- >+s:alter({temporary = true}) >+assert(s.temporary) >+-- When not specified or nil - ignored. >+s:alter({temporary = nil}) >+assert(s.temporary) >+s:alter({temporary = false}) >+assert(not s.temporary) >+-- Ensure absence is not treated like 'true'. >+s:alter({temporary = nil}) >+assert(not s.temporary) >+-- Invalid values. >+s:alter({temporary = box.NULL}) >+s:alter({temporary = 100}) >+ >+-- >+-- Alter is_sync. >+-- >+old_synchro_quorum = box.cfg.replication_synchro_quorum >+old_synchro_timeout = box.cfg.replication_synchro_timeout >+box.cfg{ \ >+ replication_synchro_quorum = 2, \ >+ replication_synchro_timeout = 0.001, \ >+} >+s:alter({is_sync = true}) >+assert(s.is_sync) >+s:replace{1} >+-- When not specified or nil - ignored. >+s:alter({is_sync = nil}) >+assert(s.is_sync) >+s:alter({is_sync = false}) >+assert(not s.is_sync) >+-- Ensure absence is not treated like 'true'. >+s:alter({is_sync = nil}) >+assert(not s.is_sync) >+-- Invalid values. >+s:alter({is_sync = box.NULL}) >+s:alter({is_sync = 100}) >+s:replace{1} >+s:delete{1} >+box.cfg{ \ >+ replication_synchro_quorum = old_synchro_quorum, \ >+ replication_synchro_timeout = old_synchro_timeout, \ >+} >+ >+-- Alter name. >+s:alter({name = 'test2'}) >+assert(box.space.test2 ~= nil) >+assert(box.space.test == nil) >+assert(s.name == 'test2') >+-- When not specified or nil - ignored. >+s:alter({name = nil}) >+assert(box.space.test2 ~= nil) >+assert(box.space.test == nil) >+assert(s.name == 'test2') >+s:alter({name = '_space'}) >+s:alter({name = 'test'}) >+assert(box.space.test ~= nil) >+assert(box.space.test2 == nil) >+assert(s.name == 'test') >+-- Invalid values. >+s:alter({name = box.NULL}) >+s:alter({name = 100}) >+ >+s:drop() >+ >+s:alter({}) >+ok, err = pcall(box.schema.space.alter, s.id, {}) >+assert(err:match('does not exist') ~= nil) >-- >2.21.1 (Apple Git-122.3) ? ? LGTM. -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From olegrok at tarantool.org Mon Aug 24 10:12:39 2020 From: olegrok at tarantool.org (Oleg Babin) Date: Mon, 24 Aug 2020 10:12:39 +0300 Subject: [Tarantool-patches] [PATCH] exports: allow to use json tools via FFI In-Reply-To: <8b318f99-f2f3-9671-96e6-0119781c39d1@tarantool.org> References: <20200821162607.68179-1-olegrok@tarantool.org> <8b318f99-f2f3-9671-96e6-0119781c39d1@tarantool.org> Message-ID: <1f559012-7827-7eff-7f65-8363e349a961@tarantool.org> Hi! Thanks for your comments. See my answers below. On 22/08/2020 00:56, Vladislav Shpilevoy wrote: > Hi! Thanks for the patch! > > On 21.08.2020 18:26,olegrok at tarantool.org wrote: >> From: Oleg Babin >> >> This patch exports some json functions to be used via FFI. We solve >> following problem: currently we don't have any tools to inspect >> jsonpaths. E.g. user wants to ban or restrict an access to some >> tuple fields, it could be some system fields. Tarantool doesn't >> have hidden fields and to solve such problem we should fairly >> parse input jsonpaths. Before this patch user should write its own >> or some external tools. This patch allows to use functions from >> built-in json-lexer directly from Tarantool via FFI. > I understand the problem - you don't want to implement your own tool, > but you must realize, that exporting some internal structs and functions > via FFI is not safe. We won't care about "backward compatibility" of > internal functions. And JSON code is quite mutable, it may change in > future, if we will rework multikeys, for example. And that will make > your code unusable on newer versions. Even probably inside one version, > because we won't bump version number just for the sake of internal code > change. I understand it's OK in general for FFI. If I use some internal functions it's my responsibility to fix my wrappers/libraries to use it if something will change. > Even worse is that you took not only general stuff like json_lexer_next_token, > but also json_path_multikey_offset - this is heavily internal, why do you > need it? I don't really need it. I exported it just for consistency (to have full set of json_path_* functions). > I can't upvote this patch, sorry. I propose you to implement a JSON path > parser in Lua, or copy-paste the C code from json.h/c. and create your > own C module. The whole json.c is 666 lines, including the multikey > support and json trees, which occupy most of the space and you don't need > them. The parser is trivial, one day work to implement it from the scratch > with the tests. Please, don't use the internal structs and functions. > > You can't just export everything you need if by luck it happened we have it > somewhere internally implemented in C. It is not safe. > > The only thing you need to implement the same JSON path parser is utf8 module, > and you have it. Creation of some new module will add new external dependency that I don't want to introduce. I'm not sure that performance of Lua-implemented module will quite high as in case with FFI. But OK I understand your point. From gorcunov at gmail.com Mon Aug 24 10:15:57 2020 From: gorcunov at gmail.com (Cyrill Gorcunov) Date: Mon, 24 Aug 2020 10:15:57 +0300 Subject: [Tarantool-patches] [PATCH 1/1] box: introduce space:alter() In-Reply-To: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> References: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> Message-ID: <20200824071557.GD1174993@grain> On Sat, Aug 22, 2020 at 12:46:02AM +0200, Vladislav Shpilevoy wrote: > There was no way to change certain space parameters without its > recreation or manual update of internal system space _space. Even > if some of them were legal to update: field_count, owner, flag of > being temporary, is_sync flag. > > The patch introduces function space:alter(), which accepts a > subset of parameters from box.schema.space.create which are > mutable, and 'name' parameter. There is a method space:rename(), > but still the parameter is added to space:alter() too, to be > consistent with index:alter(), which also accepts a new name. > > Closes #5155 Ack From avtikhon at tarantool.org Mon Aug 24 11:35:47 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Mon, 24 Aug 2020 11:35:47 +0300 Subject: [Tarantool-patches] [PATCH v1] asan: fix leak in AccessDeniedError Message-ID: <7a05931c80eed95e5cf20518c8521ecfb11da66d.1598258073.git.avtikhon@tarantool.org> In asan/lsan check found common leaks after strdup() function, because of its internal allocations in AccessDeniedError class for m_object_name, m_object_type, m_access_type buffers: Indirect leak of 24 byte(s) in 4 object(s) allocated from: #0 0x50b550 in __interceptor_strdup (/tnt/src/tarantool+0x50b550) #1 0xd71a98 in AccessDeniedError::AccessDeniedError(char const*, unsigned int, char const*, char const*, char const*, char const*, bool) /tarantool/src/box/error.cc:309:18 #2 0xd71c5b in BuildAccessDeniedError /tarantool/src/box/error.cc:319:14 #3 0x567864 in access_check_space /tarantool/src/box/space.c:91:5 #4 0x55e58b in check_index(unsigned int, unsigned int, space**, index**) /tarantool/src/box/index.cc:172:6 #5 0x55e58b in box_index_max /tarantool/src/box/index.cc:296 #6 0x2abfea88 () To fix the found issues better to use local memory allocation in stack for these buffers. In the same situation in a common CustomError class m_custom_type buffer was locally allocated with 64 size. So the buffers were changed from strdup() function internal allocation to local setup with the same size. Suppresion "leak:AccessDeniedError::AccessDeniedError" removed from asan suppressions file. Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/asan-access-fix Issue: https://github.com/tarantool/tarantool/issues/4360 asan/lsan.supp | 6 ------ src/box/error.cc | 10 +++++++--- src/box/error.h | 9 +++------ 3 files changed, 10 insertions(+), 15 deletions(-) diff --git a/asan/lsan.supp b/asan/lsan.supp index 1e297d999..1275b7d0e 100644 --- a/asan/lsan.supp +++ b/asan/lsan.supp @@ -30,12 +30,6 @@ leak:gconv_init # source: third_party/luajit leak:lj_BC_FUNCC -# test: box/access.test.lua -# test: box/access_bin.test.lua -# test: box/access_misc.test.lua -# source: src/box/error.cc -leak:AccessDeniedError::AccessDeniedError - # test: box/bitset.test.lua # source: src/lib/bitset/iterator.c leak:tt_bitset_iterator_init diff --git a/src/box/error.cc b/src/box/error.cc index c3c2af3ab..4e112cc50 100644 --- a/src/box/error.cc +++ b/src/box/error.cc @@ -304,9 +304,13 @@ AccessDeniedError::AccessDeniedError(const char *file, unsigned int line, */ if (run_trigers) trigger_run(&on_access_denied, (void *) &ctx); - m_object_type = strdup(object_type); - m_access_type = strdup(access_type); - m_object_name = strdup(object_name); + strncpy(m_object_type, object_type, sizeof(m_object_type) - 1); + m_object_type[sizeof(m_object_type) - 1] = '\0'; + strncpy(m_access_type, access_type, sizeof(m_access_type) - 1); + m_access_type[sizeof(m_access_type) - 1] = '\0'; + strncpy(m_object_name, object_name, sizeof(m_object_name) - 1); + m_object_name[sizeof(m_object_name) - 1] = '\0'; + } struct error * diff --git a/src/box/error.h b/src/box/error.h index 988b98255..4c61ed74d 100644 --- a/src/box/error.h +++ b/src/box/error.h @@ -246,9 +246,6 @@ public: ~AccessDeniedError() { - free(m_object_name); - free(m_object_type); - free(m_access_type); } const char * @@ -271,11 +268,11 @@ public: private: /** Type of object the required access was denied to */ - char *m_object_type; + char m_object_type[64]; /** Name of object the required access was denied to */ - char *m_object_name; + char m_object_name[64]; /** Type of declined access */ - char *m_access_type; + char m_access_type[64]; }; /** -- 2.17.1 From huston.mavr at gmail.com Mon Aug 24 11:44:02 2020 From: huston.mavr at gmail.com (Alexandr Barulev) Date: Mon, 24 Aug 2020 11:44:02 +0300 Subject: [Tarantool-patches] [PATCH] build: refactor static build process In-Reply-To: References: <20200622181649.10100-1-huston.mavr@gmail.com> <20200727223734.cnxrdzik2cyt3ey4@tkn_work_nb> Message-ID: I've modified this patch: disabled building libunwind with minidebuginfo to prevent linking libunwind with liblzma. Here is a diff: diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt index 53ceb609c..d90a642e6 100644 --- a/static-build/CMakeLists.txt +++ b/static-build/CMakeLists.txt @@ -185,6 +185,7 @@ else() --prefix= --disable-shared --enable-static + --disable-minidebuginfo # to prevent linking with liblzma STEP_TARGETS download ) endif() ??, 6 ???. 2020 ?. ? 16:32, Alexandr Barulev : > I?ve squashed commit and changed it?s message; > Also I?ve sended diff at previous answer > > https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build > > ??, 5 ???. 2020 ?. ? 20:08, Mavr Huston : > >> Hi, thanks for the review! >> >> libicu installs as ExternalProject_Add too, its missed in commit message; >> >> Problem with curses and ncurses was on macOS and linux, because libcurses >> is an entire copy of libncurses, and tarantool links with system >> libcurses instead of libncurses installed as tarantool dependency, but >> module FindCurses.cmkae provides workaround for this problem - >> CURSES_NEED_NCURSES flag.- to use ncurses instead of curses. (i will fix >> this part at commit message) >> >> About disable-shred flag, used at libcurl building - we want to link only >> with >> static libraries, so we prevent creating unused .so. >> >> I've renamed static_build_no_deps_* jobs after review to >> static_build_cmake_* >> >> Also about such path tarantool-prefix/* - it's a cmake >> ExternalProject_Add() >> default path (i've also added comment at .travis.mk) >> >> Useless comments "Init macOS test env" deleted. >> >> > if (BUILD_STATIC) >> > - set(LIBZ_LIB_NAME libz.a) >> > + find_library(LIBZ_LIBRARY NAMES libz.a) >> > else() >> > - set(LIBZ_LIB_NAME z) >> > + find_library(LIBZ_LIBRARY NAMES z) >> > endif() >> > - find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) >> Here we simplified code, by deleting useless variable. >> >> I've added commentaries to cmake/compiler.cmake about libunwind on macOS >> and about ignoring flag -static-libstdc++ on macOS >> >> I've fixed static-build for using system compiler: gcc/g++ on linux >> and clang/clang++ on macOS >> >> I've refactored IF (NOT APPLE) condition to IF (APPLE) at >> static-build/CMakeLists.txt >> >> I've mentioned macOS dependencies at static-build/README.md xcode-tools >> and >> others, also I've added example with CMAKE_TARANTOOL_ARGS. >> >> Added commentaries about _EP_INSTALL_DIR at static-build/CMakeLists.txt >> >> Also deleted unused use_unix_sockets_iproto = True >> >> Also deleted curl-features.test.lua, because after rebase on master it >> fails, >> due to missing curl_version_info symbol at tarantool binary. This symbol >> lost >> after #807c7fa584f21ee955b2a14623d70f7510a3650d (build: update curl >> submodule >> to 7.71.1 version ) >> >> >> After pass the review I'll squash this changes to base commit and update >> commit >> message. >> >> Here is a diff of changes: >> ===================================== >> >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml >> index 5ec5dd9b9..c9aef3dc7 100644 >> --- a/.gitlab-ci.yml >> +++ b/.gitlab-ci.yml >> @@ -534,14 +534,14 @@ static_build: >> script: >> - ${GITLAB_MAKE} test_static_build >> >> -static_build_no_deps_linux: >> +static_build_cmake_linux: >> <<: *docker_test_definition >> script: >> - - ${GITLAB_MAKE} test_static_build_no_deps_linux >> + - ${GITLAB_MAKE} test_static_build_cmake_linux >> >> -static_build_no_deps_osx_15: >> +static_build_cmake_osx_15: >> stage: test >> tags: >> - osx_15 >> script: >> - - ${GITLAB_MAKE} test_static_build_no_deps_osx >> + - ${GITLAB_MAKE} test_static_build_cmake_osx >> diff --git a/.travis.mk b/.travis.mk >> index 64862348f..482672429 100644 >> --- a/.travis.mk >> +++ b/.travis.mk >> @@ -149,8 +149,8 @@ test_static_build: deps_debian_static >> CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk >> test_debian_no_deps >> >> # New static build >> - >> -test_static_build_no_deps_linux: >> +# builddir used in this target - is a default build path from cmake >> ExternalProject_Add() >> +test_static_build_cmake_linux: >> cd static-build && cmake . && make -j && ctest -V >> cd test && /usr/bin/python test-run.py --force \ >> --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build >> $(TEST_RUN_EXTRA_PARAMS) >> @@ -218,7 +218,6 @@ INIT_TEST_ENV_OSX=\ >> rm -rf /tmp/tnt >> >> test_osx_no_deps: build_osx >> - # Init macOS test env >> ${INIT_TEST_ENV_OSX}; \ >> cd test && ./test-run.py --vardir /tmp/tnt --force >> $(TEST_RUN_EXTRA_PARAMS) >> >> @@ -233,9 +232,9 @@ base_deps_osx: >> brew install --force ${STATIC_OSX_PKGS} || brew upgrade >> ${STATIC_OSX_PKGS} >> pip install --force-reinstall -r test-run/requirements.txt >> >> -test_static_build_no_deps_osx: base_deps_osx >> +# builddir used in this target - is a default build path from cmake >> ExternalProject_Add() >> +test_static_build_cmake_osx: base_deps_osx >> cd static-build && cmake . && make -j && ctest -V >> - # Init macOS test env >> ${INIT_TEST_ENV_OSX}; \ >> cd test && ./test-run.py --vardir /tmp/tnt \ >> --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ >> diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake >> index 8422181d6..afe480679 100644 >> --- a/cmake/FindReadline.cmake >> +++ b/cmake/FindReadline.cmake >> @@ -14,6 +14,14 @@ if(BUILD_STATIC) >> if (NOT CURSES_INFO_LIBRARY) >> set(CURSES_INFO_LIBRARY "") >> endif() >> + >> + # From Modules/FindCurses.cmake: >> + # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the >> + # ``find_package(Curses)`` call if NCurses functionality is required. >> + # This flag is set for linking with required library (installed >> + # via static-build/CMakeLists.txt). If this variable won't be set >> + # then tarantool binary links with system library curses which is an >> + # entire copy of ncurses >> set(CURSES_NEED_NCURSES TRUE) >> endif() >> >> diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake >> index 14f1e1186..db2ae6227 100644 >> --- a/cmake/compiler.cmake >> +++ b/cmake/compiler.cmake >> @@ -131,6 +131,8 @@ set(CMAKE_REQUIRED_INCLUDES "") >> if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) >> set(UNWIND_LIB_NAME libunwind.a) >> else() >> + # libunwind can't be compiled on macOS. >> + # But there exists libunwind.dylib as a part of MacOSSDK >> set(UNWIND_LIB_NAME unwind) >> endif() >> find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES >> ${UNWIND_LIB_NAME}) >> @@ -192,6 +194,9 @@ if (ENABLE_BACKTRACE) >> find_package_message(UNWIND_LIBRARIES "Found unwind" >> "${UNWIND_LIBRARIES}") >> endif() >> >> +# On macOS there is no '-static-libstdc++' flag and it's use will >> +# raise following error: >> +# error: argument unused during compilation: '-static-libstdc++' >> if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) >> # Static linking for c++ routines >> add_compile_flags("C;CXX" "-static-libstdc++") >> diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt >> index 86582af0a..53ceb609c 100644 >> --- a/static-build/CMakeLists.txt >> +++ b/static-build/CMakeLists.txt >> @@ -9,11 +9,18 @@ set(NCURSES_VERSION 6.2) >> set(READLINE_VERSION 8.0) >> set(UNWIND_VERSION 1.3-rc1) >> >> -find_program(C_COMPILER gcc) >> -find_program(CXX_COMPILER g++) >> +if (APPLE) >> + find_program(C_COMPILER clang) >> + find_program(CXX_COMPILER clang++) >> +else() >> + find_program(C_COMPILER gcc) >> + find_program(CXX_COMPILER g++) >> +endif() >> set(CMAKE_C_COMPILER ${C_COMPILER}) >> set(CMAKE_CXX_COMPILER ${CXX_COMPILER}) >> >> +# Install all libraries required by tarantool at current build dir >> + >> # >> # OpenSSL >> # >> @@ -80,7 +87,18 @@ ExternalProject_Add(readline >> # >> # ICONV >> # >> -if (NOT APPLE) >> +if (APPLE) >> + ExternalProject_Add(iconv >> + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz >> + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} >> + /configure >> + --prefix= >> + --disable-shared >> + --enable-static >> + --with-gnu-ld >> + STEP_TARGETS download >> + ) >> +else() >> # In linux iconv is embedded into glibc >> # So we find system header and copy it locally >> find_path(ICONV_INCLUDE_DIR iconv.h) >> @@ -101,20 +119,11 @@ if (NOT APPLE) >> add_custom_target(iconv >> DEPENDS >> "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h" >> ) >> + # This is a hack for further getting install directory of library >> + # by ExternalProject_Get_Property >> set_target_properties(iconv >> PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX} >> ) >> -else() >> - ExternalProject_Add(iconv >> - URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz >> - CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} >> - /configure >> - --prefix= >> - --disable-shared >> - --enable-static >> - --with-gnu-ld >> - STEP_TARGETS download >> - ) >> endif() >> >> # >> @@ -162,6 +171,8 @@ if (APPLE) >> endif() >> >> add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES}) >> + # This is a hack for further getting install directory of library >> + # by ExternalProject_Get_Property >> set_target_properties(unwind >> PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX} >> ) >> @@ -178,6 +189,8 @@ else() >> ) >> endif() >> >> +# Get install directories of builded libraries for building >> +# tarantool with custon CMAKE_PREFIX_PATH >> foreach(PROJ openssl icu zlib ncurses readline iconv unwind) >> ExternalProject_Get_Property(${PROJ} install_dir) >> set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir}) >> @@ -197,16 +210,14 @@ ExternalProject_Add(tarantool >> -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH} >> -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE >> -DOPENSSL_USE_STATIC_LIBS=TRUE >> - -DCMAKE_BUILD_TYPE=Debug >> -DBUILD_STATIC=TRUE >> -DENABLE_DIST=TRUE >> -DENABLE_BACKTRACE=TRUE >> - -DPACKAGE:STRING=${PACKAGE_NAME} >> -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} >> -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} >> ${CMAKE_TARANTOOL_ARGS} >> - BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j >> STEP_TARGETS build >> + BUILD_COMMAND $(MAKE) >> ) >> >> enable_testing() >> diff --git a/static-build/README.md b/static-build/README.md >> index 29fe085c3..0019e963f 100644 >> --- a/static-build/README.md >> +++ b/static-build/README.md >> @@ -13,6 +13,24 @@ yum install -y \ >> python-msgpack python-yaml python-argparse python-six python-gevent >> ``` >> >> +MacOS: >> + >> +Before you start please install default Xcode Tools by Apple: >> + >> +``` >> +sudo xcode-select --install >> +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer >> +``` >> + >> +Install brew using command from >> +[Homebrew repository instructions](https://github.com/Homebrew/inst) >> + >> +After that run next script: >> + >> +```bash >> + brew install autoconf automake libtool cmake >> file://$${PWD}/tools/brew_taps/tntpython2.rbs >> + pip install --force-reinstall -r test-run/requirements.txt >> +``` >> >> ### Usage >> >> @@ -21,3 +39,20 @@ cmake . >> make -j >> ctest -V >> ``` >> + >> +## Customize your build >> + >> +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` >> variable >> + >> +### Usage >> + >> +There is three types of `CMAKE_BUILD_TYPE`: >> +* Debug - default >> +* Release >> +* RelWithDebInfo >> + >> +And you want to build tarantool with RelWithDebInfo: >> + >> +```bash >> +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" . >> +``` >> diff --git a/static-build/test/static-build/curl-features.test.lua >> b/static-build/test/static-build/curl-features.test.lua >> deleted file mode 100755 >> index 57b1c4306..000000000 >> --- a/static-build/test/static-build/curl-features.test.lua >> +++ /dev/null >> @@ -1,67 +0,0 @@ >> -#!/usr/bin/env tarantool >> - >> -local tap = require('tap') >> -local ffi = require('ffi') >> -ffi.cdef([[ >> - struct curl_version_info_data { >> - int age; /* see description below */ >> - const char *version; /* human readable string */ >> - unsigned int version_num; /* numeric representation */ >> - const char *host; /* human readable string */ >> - int features; /* bitmask, see below */ >> - char *ssl_version; /* human readable string */ >> - long ssl_version_num; /* not used, always zero */ >> - const char *libz_version; /* human readable string */ >> - const char * const *protocols; /* protocols */ >> - >> - /* when 'age' is CURLVERSION_SECOND or higher, the members below >> exist */ >> - const char *ares; /* human readable string */ >> - int ares_num; /* number */ >> - >> - /* when 'age' is CURLVERSION_THIRD or higher, the members below >> exist */ >> - const char *libidn; /* human readable string */ >> - >> - /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the >> members >> - below exist */ >> - int iconv_ver_num; /* '_libiconv_version' if iconv support >> enabled */ >> - >> - const char *libssh_version; /* human readable string */ >> - >> - /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the >> members >> - below exist */ >> - unsigned int brotli_ver_num; /* Numeric Brotli version >> - (MAJOR << 24) | (MINOR << 12) | >> PATCH */ >> - const char *brotli_version; /* human readable string. */ >> - >> - /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the >> members >> - below exist */ >> - unsigned int nghttp2_ver_num; /* Numeric nghttp2 version >> - (MAJOR << 16) | (MINOR << 8) | >> PATCH */ >> - const char *nghttp2_version; /* human readable string. */ >> - >> - const char *quic_version; /* human readable quic (+ HTTP/3) >> library + >> - version or NULL */ >> - >> - /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the >> members >> - below exist */ >> - const char *cainfo; /* the built-in default >> CURLOPT_CAINFO, might >> - be NULL */ >> - const char *capath; /* the built-in default >> CURLOPT_CAPATH, might >> - be NULL */ >> - }; >> - >> - struct curl_version_info_data *curl_version_info(int age); >> -]]) >> - >> -local info = ffi.C.curl_version_info(7) >> -local test = tap.test('curl-features') >> -test:plan(2) >> - >> -if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then >> - test:diag('ssl_version: ' .. ffi.string(info.ssl_version)) >> -end >> -if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then >> - test:diag('libz_version: ' .. ffi.string(info.libz_version)) >> -end >> - >> -os.exit(test:check() and 0 or 1) >> diff --git a/static-build/test/static-build/suite.ini >> b/static-build/test/static-build/suite.ini >> index 4da3d5d2f..92e349466 100644 >> --- a/static-build/test/static-build/suite.ini >> +++ b/static-build/test/static-build/suite.ini >> @@ -3,4 +3,3 @@ core = app >> description = Static build tests >> script = box.lua >> is_parallel = True >> -use_unix_sockets_iproto = True >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From v.shpilevoy at tarantool.org Tue Aug 25 00:16:47 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 24 Aug 2020 23:16:47 +0200 Subject: [Tarantool-patches] [PATCH v9 0/7] qsync: write CONFIRM/ROLLBACK without txn engine In-Reply-To: <20200819213442.1099018-1-gorcunov@gmail.com> References: <20200819213442.1099018-1-gorcunov@gmail.com> Message-ID: Pushed to master and 2.5. From v.shpilevoy at tarantool.org Tue Aug 25 00:40:16 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 24 Aug 2020 23:40:16 +0200 Subject: [Tarantool-patches] [PATCH 1/1] box: introduce space:alter() In-Reply-To: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> References: <46c58e05328a4b9e85d6837fe922e308bd120ad4.1598049870.git.v.shpilevoy@tarantool.org> Message-ID: <7d87d70f-e815-37b0-159c-9433c60af2c1@tarantool.org> Pushed to master and 2.5. From v.shpilevoy at tarantool.org Tue Aug 25 00:49:55 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Mon, 24 Aug 2020 23:49:55 +0200 Subject: [Tarantool-patches] [PATCH msgpack v1] test: correct buffer size to fix ASAN error In-Reply-To: References: Message-ID: <4be7d009-a6b3-0bee-17b6-c225d64f520a@tarantool.org> Hi! Thanks for the patch! See 2 comments below. On 20.08.2020 10:02, Alexander V. Tikhonov wrote: > Found ASAN error: > > [001] + ok 206 - ================================================================= > [001] +==6889==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x604000000031 at pc 0x0000005a72e7 bp 0x7ffe47c30c80 sp 0x7ffe47c30c78 > [001] +WRITE of size 1 at 0x604000000031 thread T0 > [001] + #0 0x5a72e6 in mp_store_u8 /tarantool/src/lib/msgpuck/msgpuck.h:258:1 > [001] + #1 0x5a72e6 in mp_encode_uint /tarantool/src/lib/msgpuck/msgpuck.h:1768 > [001] + #2 0x4fa657 in test_mp_print /tarantool/src/lib/msgpuck/test/msgpuck.c:957:16 > [001] + #3 0x509024 in main /tarantool/src/lib/msgpuck/test/msgpuck.c:1331:2 > [001] + #4 0x7f3658fd909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) > [001] + #5 0x41f339 in _start (/tnt/test/unit/msgpack.test+0x41f339) > [001] + > [001] +0x604000000031 is located 0 bytes to the right of 33-byte region [0x604000000010,0x604000000031) > [001] +allocated by thread T0 here: > [001] + #0 0x4cace3 in malloc (/tnt/test/unit/msgpack.test+0x4cace3) > [001] + #1 0x4fa5db in test_mp_print /tarantool/src/lib/msgpuck/test/msgpuck.c:945:18 > [001] + #2 0x509024 in main /tarantool/src/lib/msgpuck/test/msgpuck.c:1331:2 > [001] + #3 0x7f3658fd909a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) > [001] + > [001] +SUMMARY: AddressSanitizer: heap-buffer-overflow /tarantool/src/lib/msgpuck/msgpuck.h:258:1 in mp_store_u8 > [001] +Shadow bytes around the buggy address: > [001] + 0x0c087fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [001] + 0x0c087fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [001] + 0x0c087fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [001] + 0x0c087fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [001] + 0x0c087fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > [001] +=>0x0c087fff8000: fa fa 00 00 00 00[01]fa fa fa fa fa fa fa fa fa > [001] + 0x0c087fff8010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > [001] + 0x0c087fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > [001] + 0x0c087fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > [001] + 0x0c087fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > [001] + 0x0c087fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa > [001] +Shadow byte legend (one shadow byte represents 8 application bytes): > [001] + Addressable: 00 > [001] + Partially addressable: 01 02 03 04 05 06 07 > [001] + Heap left redzone: fa > [001] + Freed heap region: fd > [001] + Stack left redzone: f1 > [001] + Stack mid redzone: f2 > [001] + Stack right redzone: f3 > [001] + Stack after return: f5 > [001] + Stack use after scope: f8 > [001] + Global redzone: f9 > [001] + Global init order: f6 > [001] + Poisoned by user: f7 > [001] + Container overflow: fc > [001] + Array cookie: ac > [001] + Intra object redzone: bb > [001] + ASan internal: fe > [001] + Left alloca redzone: ca > > Invetigated the buffer size that was allocated - it was 33 bytes, but it 1. Invetigated -> Investigated. > needed 34. The fix was to increase this buffer. > > Part of tarantool/tarantool#4360 > --- > > Github: https://github.com/tarantool/msgpuck/tree/avtikhon/gh-4360-fix-asan-error > Issue: https://github.com/tarantool/tarantool/issues/4360 > > test/msgpuck.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/test/msgpuck.c b/test/msgpuck.c > index 2d9bcbf..6613234 100644 > --- a/test/msgpuck.c > +++ b/test/msgpuck.c > @@ -940,7 +940,7 @@ test_mp_print() > > /* Test mp_snprint max nesting depth. */ > int mp_buff_sz = MP_PRINT_MAX_DEPTH * mp_sizeof_array(1) + > - mp_sizeof_uint(1); > + mp_sizeof_uint(1) + 1; 2. Better change to this: int mp_buff_sz = (MP_PRINT_MAX_DEPTH + 1) * mp_sizeof_array(1) + mp_sizeof_uint(1); To emphasize, that +1 comes from one another mp_encode_array(1). > int exp_str_sz = 2 * (MP_PRINT_MAX_DEPTH + 1) + 3 + 1; > char *mp_buff = malloc(mp_buff_sz); > char *exp_str = malloc(exp_str_sz); From v.shpilevoy at tarantool.org Tue Aug 25 01:01:26 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Tue, 25 Aug 2020 00:01:26 +0200 Subject: [Tarantool-patches] [PATCH v1] asan: fix leak in AccessDeniedError In-Reply-To: <7a05931c80eed95e5cf20518c8521ecfb11da66d.1598258073.git.avtikhon@tarantool.org> References: <7a05931c80eed95e5cf20518c8521ecfb11da66d.1598258073.git.avtikhon@tarantool.org> Message-ID: Thanks for the patch! See 3 comments below. On 24.08.2020 10:35, Alexander V. Tikhonov wrote: > In asan/lsan check found common leaks after strdup() function, > because of its internal allocations in AccessDeniedError class > for m_object_name, m_object_type, m_access_type buffers: > > Indirect leak of 24 byte(s) in 4 object(s) allocated from: > #0 0x50b550 in __interceptor_strdup (/tnt/src/tarantool+0x50b550) > #1 0xd71a98 in AccessDeniedError::AccessDeniedError(char const*, unsigned int, char const*, char const*, char const*, char const*, bool) /tarantool/src/box/error.cc:309:18 > #2 0xd71c5b in BuildAccessDeniedError /tarantool/src/box/error.cc:319:14 > #3 0x567864 in access_check_space /tarantool/src/box/space.c:91:5 > #4 0x55e58b in check_index(unsigned int, unsigned int, space**, index**) /tarantool/src/box/index.cc:172:6 > #5 0x55e58b in box_index_max /tarantool/src/box/index.cc:296 > #6 0x2abfea88 () > > To fix the found issues better to use local memory allocation in stack > for these buffers. In the same situation in a common CustomError class > m_custom_type buffer was locally allocated with 64 size. So the buffers > were changed from strdup() function internal allocation to local setup > with the same size. > > Suppresion "leak:AccessDeniedError::AccessDeniedError" removed from 1. Suppresion -> Suppression. > asan suppressions file. > > Part of #4360 > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/asan-access-fix > Issue: https://github.com/tarantool/tarantool/issues/4360 > > asan/lsan.supp | 6 ------ > src/box/error.cc | 10 +++++++--- > src/box/error.h | 9 +++------ > 3 files changed, 10 insertions(+), 15 deletions(-) > > diff --git a/asan/lsan.supp b/asan/lsan.supp > index 1e297d999..1275b7d0e 100644 > --- a/asan/lsan.supp > +++ b/asan/lsan.supp > @@ -30,12 +30,6 @@ leak:gconv_init > # source: third_party/luajit > leak:lj_BC_FUNCC > > -# test: box/access.test.lua > -# test: box/access_bin.test.lua > -# test: box/access_misc.test.lua > -# source: src/box/error.cc > -leak:AccessDeniedError::AccessDeniedError > - > # test: box/bitset.test.lua > # source: src/lib/bitset/iterator.c > leak:tt_bitset_iterator_init > diff --git a/src/box/error.cc b/src/box/error.cc > index c3c2af3ab..4e112cc50 100644 > --- a/src/box/error.cc > +++ b/src/box/error.cc > @@ -304,9 +304,13 @@ AccessDeniedError::AccessDeniedError(const char *file, unsigned int line, > */ > if (run_trigers) > trigger_run(&on_access_denied, (void *) &ctx); > - m_object_type = strdup(object_type); > - m_access_type = strdup(access_type); > - m_object_name = strdup(object_name); > + strncpy(m_object_type, object_type, sizeof(m_object_type) - 1); > + m_object_type[sizeof(m_object_type) - 1] = '\0'; > + strncpy(m_access_type, access_type, sizeof(m_access_type) - 1); > + m_access_type[sizeof(m_access_type) - 1] = '\0'; > + strncpy(m_object_name, object_name, sizeof(m_object_name) - 1); > + m_object_name[sizeof(m_object_name) - 1] = '\0'; 2. Please, use tabs, not whitespaces. > + > } > > struct error * > diff --git a/src/box/error.h b/src/box/error.h > index 988b98255..4c61ed74d 100644 > --- a/src/box/error.h > +++ b/src/box/error.h > @@ -246,9 +246,6 @@ public: > > ~AccessDeniedError() > { > - free(m_object_name); > - free(m_object_type); > - free(m_access_type); > } > > const char * > @@ -271,11 +268,11 @@ public: > > private: > /** Type of object the required access was denied to */ > - char *m_object_type; > + char m_object_type[64]; > /** Name of object the required access was denied to */ > - char *m_object_name; > + char m_object_name[64]; > /** Type of declined access */ > - char *m_access_type; > + char m_access_type[64]; 3. It does not look like a good idea. CustomError uses fixed size buffer for error message, which is considered not as important as error object attributes. We can't allocate everything inside the error object structure. What will happen when CustomError gets payload? It definitely won't fit into a fixed buffer. It is necessary to investigate deeper, why the destructor wasn't called. Any why ASAN does not consider the AccessDeniedError object leaked itself, because it is also on the heap. > }; > > /** > From kyukhin at tarantool.org Tue Aug 25 12:42:57 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Tue, 25 Aug 2020 12:42:57 +0300 Subject: [Tarantool-patches] [PATCH v3] tuple: drop extra restrictions for multikey index In-Reply-To: <20200814145143.23720-1-i.kosarev@tarantool.org> References: <20200814145143.23720-1-i.kosarev@tarantool.org> Message-ID: <20200825094256.baafdpufr3otktsh@tarantool.org> Hello, On 14 ??? 17:51, Ilya Kosarev wrote: > Multikey index did not work properly with nullable root field in > tuple_raw_multikey_count(). Now it is fixed and corresponding > restrictions are dropped. This also means that we can drop implicit > nullability update for array/map fields and make all fields nullable > by default, as it was until e1d3fe8ab8eed65394ad17409401a93b6fcdc435 > (tuple format: don't allow null where array/map is expected), as far as > default non-nullability itself doesn't solve any real problems while > providing confusing behavior (gh-5027). > > Follow-up #5027 > Closes #5192 > > @TarantoolBot document > Title: tuple: allow nullable multikey index root > Update the documentation for space_object:create_index() to reflect > that it is now safe for multikey index root to be nullable. > Previously it was not safe to insert, for example, box.NULL as an array > field which elements are the part of multikey index due to error in > counter of multikey index keys in tuple. It was partly fixed using > default non-nullability for tuple fields. In 2.3.3 the restriction on > nullable multikey index root was introduced. Now the counter problem is > fixed and all tuple fields are nullable by default as before 2.2.1. > --- > Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-5192-fix-multikey-index-restrictions > Issue: https://github.com/tarantool/tarantool/issues/5192 > > @ChangeLog: > * Dropped restrictions on nullable multikey index root. They were > introduced due to inaccuracy in multikey index realization. It is > now fixed. Also all fields are now nullable by default as it was > before 2.2.1 (gh-5192). LGTM. I've checked your patch into 2.4, 2.5 and master. -- Regards, Kirill Yukhin From sergeyb at tarantool.org Tue Aug 25 15:49:39 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Tue, 25 Aug 2020 15:49:39 +0300 Subject: [Tarantool-patches] [PATCH 2/3] replication: test clear_synchro_queue function In-Reply-To: <8c3b7d4c-22e8-108a-b15e-ef3c89510a63@tarantool.org> References: <2e5a2d3948eafebd1488ab120ad0458838661d8a.1594314820.git.sergeyb@tarantool.org> <8c3b7d4c-22e8-108a-b15e-ef3c89510a63@tarantool.org> Message-ID: <20200825124939.GB47610@pony.bronevichok.ru> Vlad, thanks for review! Patches updated in a branch and please see my answers inline. On 00:00 Tue 21 Jul , Vladislav Shpilevoy wrote: > Thanks for the patch! > > See 8 comments below. > > On 09.07.2020 19:16, sergeyb at tarantool.org wrote: > > From: Sergey Bronnikov > > > > Part of #5055 > > Part of #4849 > > --- > > test/replication/qsync_basic.result | 85 +++++++++++++++++++++++++++ > > test/replication/qsync_basic.test.lua | 31 ++++++++++ > > 2 files changed, 116 insertions(+) > > > > diff --git a/test/replication/qsync_basic.result b/test/replication/qsync_basic.result > > index ab4be0c7e..464df75a7 100644 > > --- a/test/replication/qsync_basic.result > > +++ b/test/replication/qsync_basic.result > > @@ -32,6 +32,14 @@ s2.is_sync > > | - false > > | ... > > > > +-- > > +-- gh-4849: clear synchro queue with unconfigured box > > +-- > > +box.ctl.clear_synchro_queue() > > 1. Enough to test that it does not crash here. No need to > check result. Well, removed assignment to variable. > > + | --- > > + | - -1 > > + | ... > > + > > -- Net.box takes sync into account. > > box.schema.user.grant('guest', 'super') > > | --- > > @@ -553,6 +561,82 @@ box.space.sync:select{7} > > | - - [7] > > | ... > > > > +-- > > +-- gh-4849: clear synchro queue on a replica > > +-- > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 2} > > 2. If you want this timeout to fail, it is too big. The test will be too > long. If it is supposed not to fail, then it is way too small. If you want > it to fail, it should be something like 0.001. If you want it to hold, it > should be 1000 to be sure. Keep in mind that you can change the timeout on > fly. That allows to build quite complex reactive tests completely event-based. set replication_synchro_timeout value to 1000 > > + | --- > > + | ... > > +f1 = fiber.create(box.space.sync.replace, box.space.sync, {9}) > > 3. You need to extract the exact result value. Use pcall for that, and print > its result after the fiber is dead. See other examples with fiber.create() in > qsync test suite. The same for the next test case. Done. > > + | --- > > + | ... > > +f1:status() > > + | --- > > + | - suspended > > + | ... > > +test_run:switch('replica') > > + | --- > > + | - true > > 4. Better wait until the value is delivered. Otherwise you can switch to > replica before master finishes WAL write, and the queue will be empty here. > Try > > test_run:wait_cond(function() return box.space.sync:get{9} ~= nil end) Agree, added wait_cond(). > > + | ... > > +box.ctl.clear_synchro_queue() > > + | --- > > + | - 0 > > + | ... > > +box.space.sync:select{9} > > + | --- > > + | - [] > > 5. If select returns 9 before queue clean, and returns empty afterwards, > it means the queue was cleared. So here the queue size is not really needed, > as you can see. removed this and next select{} statements > > + | ... > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +box.space.sync:select{9} > > + | --- > > + | - [] > > + | ... > > +f1:status() > > + | --- > > + | - dead > > + | ... > > + > > +-- > > +-- gh-4849: clear synchro queue on a master > > 6. Since the previous test the replica's WAL is different from master's. > I am not sure the replication is still alive. Test with clear_synchro_queue() on master keeps master alive at the end of test so I moved it before the same test for the replica. Also added note for others that cluster may be in a broken state here. > > +-- > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 2} > > + | --- > > + | ... > > +f1 = fiber.create(box.space.sync.replace, box.space.sync, {10}) > > + | --- > > + | ... > > +f1:status() > > + | --- > > + | - suspended > > + | ... > > +box.ctl.clear_synchro_queue() > > + | --- > > + | - -2 > > + | ... > > +box.space.sync:select{10} > > + | --- > > + | - - [10] > > 7. Why is it 10? The quorum is not reached, it should have been rolled back. Added "box.cfg{replication_synchro_timeout = 0.1}" before clear_synchro_queue() call and now tx is rolled back. Also I've added checks of current value with wait_cond() that is more reliable than select{} on a master and replica. Updated output looks like this: box.ctl.clear_synchro_queue() | --- | ... test_run:switch('replica') | --- | - true | ... test_run:wait_cond(function() return box.space.sync:get{10} == nil end) | --- | - true | ... test_run:switch('default') | --- | - true | ... test_run:wait_cond(function() return f:status() == 'dead' end) | --- | - true | ... ok, err | --- | - false | - Quorum collection for a synchronous transaction is timed out | ... test_run:wait_cond(function() return box.space.sync:get{10} == nil end) | --- | - true | ... > > + | ... > > +test_run:switch('replica') > > + | --- > > + | - true > > + | ... > > +box.space.sync:select{10} > > + | --- > > + | - - [10] > > + | ... > > + > > -- Cleanup. > > test_run:cmd('switch default') > > | --- > > @@ -576,6 +660,7 @@ test_run:cmd('delete server replica') > > | ... > > box.space.test:drop() > > | --- > > + | - error: A rollback for a synchronous transaction is received > > 8. Why is it changed? Perhaps it is because in previous version of test I haven't wait fibers 'dead'. There is no error now: box.space.test:drop() | --- | ... box.space.sync:drop() | --- | ... > > > | ... > > box.space.sync:drop() > > | --- From alexander.turenko at tarantool.org Tue Aug 25 16:21:50 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Tue, 25 Aug 2020 16:21:50 +0300 Subject: [Tarantool-patches] [PATCH] build: refactor static build process In-Reply-To: References: <20200622181649.10100-1-huston.mavr@gmail.com> <20200727223734.cnxrdzik2cyt3ey4@tkn_work_nb> Message-ID: <20200825132150.tetyh4tr46zy25ti@tkn_work_nb> Thanks for fixes and added comments! > diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt > index 86582af0a..53ceb609c 100644 > --- a/static-build/CMakeLists.txt > +++ b/static-build/CMakeLists.txt > @@ -9,11 +9,18 @@ set(NCURSES_VERSION 6.2) > set(READLINE_VERSION 8.0) > set(UNWIND_VERSION 1.3-rc1) > > -find_program(C_COMPILER gcc) > -find_program(CXX_COMPILER g++) > +if (APPLE) > + find_program(C_COMPILER clang) > + find_program(CXX_COMPILER clang++) > +else() > + find_program(C_COMPILER gcc) > + find_program(CXX_COMPILER g++) > +endif() How about just don't set it at all? From huston.mavr at gmail.com Tue Aug 25 17:01:08 2020 From: huston.mavr at gmail.com (HustonMmmavr) Date: Tue, 25 Aug 2020 17:01:08 +0300 Subject: [Tarantool-patches] [PATCH v2] build: refactor static build process Message-ID: <20200825140108.52090-1-huston.mavr@gmail.com> From: Yaroslav Dynnikov Refactored static build process to use static-build/CMakeLists.txt instead of Dockerfile.staticbuild (this allows to support static build on macOS). Following third-party dependencies for static build are installed via cmake `ExternalProject_Add`: - OpenSSL - Zlib - Ncurses - Readline - Unwind - ICU * Added support static build for macOS * Prevented linking tarantool binary with system libcurses (which is entire copy of libncurses) by setting flag `CURSES_NEED_NCURSES` to TRUE at file cmake/FindReadline.cmake. (This flag defined at FindCurses.cmake module and it's a workaround for linking with correct library) * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for staic build at file cmake/BuildLibCURL.cmake: - disable building shared libcurl libraries (by setting `--disable-shared` option) - disable hiding libcurl symbols (by setting `--disable-symbol-hiding` option) - prevent linking libcurl with system libz by settign `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option) * Removed Dockerfile.staticbuild * Added new gitlab.ci jobs to test new style static build: - static_build_cmake_linux - static_build_cmake_osx_15 * Removed static_docker_build gitlab.ci job Closes #5095 --- .gitlab-ci.yml | 11 +- .travis.mk | 52 +++- Dockerfile.staticbuild | 98 ------- cmake/BuildLibCURL.cmake | 18 +- cmake/FindReadline.cmake | 10 + cmake/compiler.cmake | 24 +- cmake/os.cmake | 5 +- static-build/.gitignore | 4 + static-build/CMakeLists.txt | 240 ++++++++++++++++++ static-build/README.md | 58 +++++ static-build/test/CheckDependencies.cmake | 43 ++++ static-build/test/static-build/box.lua | 3 + .../test/static-build/exports.test.lua | 148 +++++++++++ static-build/test/static-build/suite.ini | 5 + .../test/static-build/traceback.test.lua | 15 ++ static-build/test/test-run.py | 1 + 16 files changed, 608 insertions(+), 127 deletions(-) delete mode 100644 Dockerfile.staticbuild create mode 100644 static-build/.gitignore create mode 100644 static-build/CMakeLists.txt create mode 100644 static-build/README.md create mode 100644 static-build/test/CheckDependencies.cmake create mode 100755 static-build/test/static-build/box.lua create mode 100755 static-build/test/static-build/exports.test.lua create mode 100644 static-build/test/static-build/suite.ini create mode 100755 static-build/test/static-build/traceback.test.lua create mode 120000 static-build/test/test-run.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 0ead08711..c9aef3dc7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -534,9 +534,14 @@ static_build: script: - ${GITLAB_MAKE} test_static_build -static_docker_build: +static_build_cmake_linux: + <<: *docker_test_definition + script: + - ${GITLAB_MAKE} test_static_build_cmake_linux + +static_build_cmake_osx_15: stage: test tags: - - deploy_test + - osx_15 script: - - ${GITLAB_MAKE} test_static_docker_build + - ${GITLAB_MAKE} test_static_build_cmake_osx diff --git a/.travis.mk b/.travis.mk index efc05cf05..482672429 100644 --- a/.travis.mk +++ b/.travis.mk @@ -148,8 +148,12 @@ deps_debian_static: test_static_build: deps_debian_static CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk test_debian_no_deps -test_static_docker_build: - docker build --no-cache --network=host --build-arg RUN_TESTS=ON -f Dockerfile.staticbuild . +# New static build +# builddir used in this target - is a default build path from cmake ExternalProject_Add() +test_static_build_cmake_linux: + cd static-build && cmake . && make -j && ctest -V + cd test && /usr/bin/python test-run.py --force \ + --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build $(TEST_RUN_EXTRA_PARAMS) # ################### # Static Analysis @@ -193,15 +197,16 @@ build_osx: cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS} make -j -test_osx_no_deps: build_osx - # Limits: Increase the maximum number of open file descriptors on macOS: - # Travis-ci needs the "ulimit -n " call - # Gitlab-ci needs the "launchctl limit maxfiles " call - # Also gitlib-ci needs the password to change the limits, while - # travis-ci runs under root user. Limit setup must be in the same - # call as tests runs call. - # Tests: Temporary excluded replication/ suite with some tests - # from other suites by issues #4357 and #4370 + +# Limits: Increase the maximum number of open file descriptors on macOS: +# Travis-ci needs the "ulimit -n " call +# Gitlab-ci needs the "launchctl limit maxfiles " call +# Also gitlib-ci needs the password to change the limits, while +# travis-ci runs under root user. Limit setup must be in the same +# call as tests runs call. +# Tests: Temporary excluded replication/ suite with some tests +# from other suites by issues #4357 and #4370 +INIT_TEST_ENV_OSX=\ sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \ launchctl limit maxfiles || : ; \ ulimit -n ${MAX_FILES} || : ; \ @@ -210,11 +215,32 @@ test_osx_no_deps: build_osx launchctl limit maxproc || : ; \ ulimit -u ${MAX_PROC} || : ; \ ulimit -u ; \ - rm -rf /tmp/tnt ; \ - cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) + rm -rf /tmp/tnt + +test_osx_no_deps: build_osx + ${INIT_TEST_ENV_OSX}; \ + cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) test_osx: deps_osx test_osx_no_deps +# Static macOS build + +STATIC_OSX_PKGS=autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rb +base_deps_osx: + brew update || echo | /usr/bin/ruby -e \ + "$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" + brew install --force ${STATIC_OSX_PKGS} || brew upgrade ${STATIC_OSX_PKGS} + pip install --force-reinstall -r test-run/requirements.txt + +# builddir used in this target - is a default build path from cmake ExternalProject_Add() +test_static_build_cmake_osx: base_deps_osx + cd static-build && cmake . && make -j && ctest -V + ${INIT_TEST_ENV_OSX}; \ + cd test && ./test-run.py --vardir /tmp/tnt \ + --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ + --force $(TEST_RUN_EXTRA_PARAMS) + + ########### # FreeBSD # ########### diff --git a/Dockerfile.staticbuild b/Dockerfile.staticbuild deleted file mode 100644 index f67f46f5e..000000000 --- a/Dockerfile.staticbuild +++ /dev/null @@ -1,98 +0,0 @@ -FROM centos:7 - -RUN yum install -y epel-release -RUN yum install -y yum install https://centos7.iuscommunity.org/ius-release.rpm - -RUN set -x \ - && yum -y install \ - libstdc++ \ - libstdc++-static \ - readline \ - openssl \ - lz4 \ - binutils \ - ncurses \ - libgomp \ - lua \ - curl \ - tar \ - zip \ - unzip \ - libunwind \ - zlib \ - && yum -y install \ - perl \ - gcc-c++ \ - cmake \ - lz4-devel \ - binutils-devel \ - lua-devel \ - make \ - git \ - autoconf \ - automake \ - libtool \ - wget - -RUN yum -y install ncurses-static readline-static zlib-static pcre-static glibc-static - -RUN yum -y install python-devel python-pip - -RUN set -x && \ - cd / && \ - curl -O -L https://www.openssl.org/source/openssl-1.1.1f.tar.gz && \ - tar -xvf openssl-1.1.1f.tar.gz && \ - cd openssl-1.1.1f && \ - ./config --libdir=lib && \ - make -j && make install - -RUN set -x && \ - cd / && \ - curl -O -L https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz && \ - tar -xvf icu4c-62_1-src.tgz && \ - cd icu/source && \ - ./configure --with-data-packaging=static --enable-static --enable-shared && \ - make -j && make install - -RUN set -x && \ - cd / && \ - curl -O -L http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.3-rc1.tar.gz && \ - tar -xvf libunwind-1.3-rc1.tar.gz && \ - cd libunwind-1.3-rc1 && \ - ./configure --enable-static --enable-shared && \ - make -j && make install - -COPY . /tarantool - -WORKDIR /tarantool - -RUN set -x && \ - git submodule init && \ - git submodule update - -# Cleanup for 'build' directory added, because it purges all artefacts -# produced for curl build, including the old configuration in build/curl -RUN set -x && \ - find . -name 'CMakeFiles' -type d -exec rm -rf {} + && \ - find . -name 'CMakeCache.txt' -type f -delete && \ - rm -rf build test/small test/luajit-tap - -RUN pip install -r /tarantool/test-run/requirements.txt - -RUN set -x && \ - cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DENABLE_DIST:BOOL=ON \ - -DBUILD_STATIC=ON \ - -DOPENSSL_USE_STATIC_LIBS=ON \ - -DOPENSSL_ROOT_DIR=/usr/local \ - . && \ - make -j && make install - -ARG RUN_TESTS -RUN if [ -n "${RUN_TESTS}" ]; then \ - set -x && \ - cd test && \ - /usr/bin/python test-run.py --force; \ - fi - -ENTRYPOINT ["/bin/bash"] diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake index 5f8b15a63..365c14284 100644 --- a/cmake/BuildLibCURL.cmake +++ b/cmake/BuildLibCURL.cmake @@ -4,15 +4,20 @@ macro(curl_build) set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work) set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest) + message(STATUS "Looking for zlib") + find_path(ZLIB_INCLUDE_DIR zlib.h) + message(STATUS "Looking for zlib.h - ${ZLIB_INCLUDE_DIR}") if (BUILD_STATIC) - set(LIBZ_LIB_NAME libz.a) + find_library(LIBZ_LIBRARY NAMES libz.a) else() - set(LIBZ_LIB_NAME z) + find_library(LIBZ_LIBRARY NAMES z) endif() - find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) - if ("${LIBZ_LIBRARY}" STREQUAL "LIBZ_LIBRARY-NOTFOUND") + message(STATUS "Looking for libz - ${LIBZ_LIBRARY}") + + if (NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY) message(FATAL_ERROR "Unable to find zlib") endif() + get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR} DIRECTORY) # Use the same OpenSSL library for libcurl as is used for # tarantool itself. @@ -88,9 +93,10 @@ macro(curl_build) --prefix --enable-static - --enable-shared + --disable-shared + --disable-symbol-hiding - --with-zlib + --with-zlib=${FOUND_ZLIB_ROOT_DIR} ${LIBCURL_OPENSSL_OPT} --with-ca-fallback diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake index c48bdcb3e..afe480679 100644 --- a/cmake/FindReadline.cmake +++ b/cmake/FindReadline.cmake @@ -14,7 +14,17 @@ if(BUILD_STATIC) if (NOT CURSES_INFO_LIBRARY) set(CURSES_INFO_LIBRARY "") endif() + + # From Modules/FindCurses.cmake: + # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the + # ``find_package(Curses)`` call if NCurses functionality is required. + # This flag is set for linking with required library (installed + # via static-build/CMakeLists.txt). If this variable won't be set + # then tarantool binary links with system library curses which is an + # entire copy of ncurses + set(CURSES_NEED_NCURSES TRUE) endif() + find_package(Curses) if(NOT CURSES_FOUND) find_package(Termcap) diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake index 5a1141ebd..db2ae6227 100644 --- a/cmake/compiler.cmake +++ b/cmake/compiler.cmake @@ -33,10 +33,6 @@ if (CMAKE_COMPILER_IS_GNUCC) Your GCC version is ${CMAKE_CXX_COMPILER_VERSION}, please update ") endif() -else() - if (BUILD_STATIC) - message(FATAL_ERROR "Static build is supported for GCC only") - endif() endif() # @@ -120,10 +116,23 @@ set (CMAKE_CXX_FLAGS_RELWITHDEBINFO unset(CC_DEBUG_OPT) +message(STATUS "Looking for libunwind.h") +find_path(UNWIND_INCLUDE_DIR libunwind.h) +message(STATUS "Looking for libunwind.h - ${UNWIND_INCLUDE_DIR}") + +if (UNWIND_INCLUDE_DIR) + include_directories(${UNWIND_INCLUDE_DIR}) +endif() + +set(CMAKE_REQUIRED_INCLUDES ${UNWIND_INCLUDE_DIR}) check_include_file(libunwind.h HAVE_LIBUNWIND_H) -if(BUILD_STATIC) +set(CMAKE_REQUIRED_INCLUDES "") + +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) set(UNWIND_LIB_NAME libunwind.a) else() + # libunwind can't be compiled on macOS. + # But there exists libunwind.dylib as a part of MacOSSDK set(UNWIND_LIB_NAME unwind) endif() find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME}) @@ -185,7 +194,10 @@ if (ENABLE_BACKTRACE) find_package_message(UNWIND_LIBRARIES "Found unwind" "${UNWIND_LIBRARIES}") endif() -if(BUILD_STATIC) +# On macOS there is no '-static-libstdc++' flag and it's use will +# raise following error: +# error: argument unused during compilation: '-static-libstdc++' +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) # Static linking for c++ routines add_compile_flags("C;CXX" "-static-libstdc++") endif() diff --git a/cmake/os.cmake b/cmake/os.cmake index 905be61df..276a79b42 100644 --- a/cmake/os.cmake +++ b/cmake/os.cmake @@ -107,7 +107,10 @@ elseif (${CMAKE_SYSTEM_NAME} STREQUAL "Darwin") # Latest versions of Homebrew wont 'link --force' for libraries, that were # preinstalled in system. So we'll use this dirty hack - find_program(HOMEBREW_EXECUTABLE brew) + + if (NOT BUILD_STATIC) + find_program(HOMEBREW_EXECUTABLE brew) + endif() if(EXISTS ${HOMEBREW_EXECUTABLE}) execute_process(COMMAND ${HOMEBREW_EXECUTABLE} --prefix OUTPUT_VARIABLE HOMEBREW_PREFIX diff --git a/static-build/.gitignore b/static-build/.gitignore new file mode 100644 index 000000000..c8028a870 --- /dev/null +++ b/static-build/.gitignore @@ -0,0 +1,4 @@ +*-prefix +/Makefile +/test/var +/build diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt new file mode 100644 index 000000000..d90a642e6 --- /dev/null +++ b/static-build/CMakeLists.txt @@ -0,0 +1,240 @@ +cmake_minimum_required(VERSION 2.8) + +project(tarantool-env NONE) + +include(ExternalProject) +set(OPENSSL_VERSION 1.1.1f) +set(ZLIB_VERSION 1.2.11) +set(NCURSES_VERSION 6.2) +set(READLINE_VERSION 8.0) +set(UNWIND_VERSION 1.3-rc1) + +if (APPLE) + find_program(C_COMPILER clang) + find_program(CXX_COMPILER clang++) +else() + find_program(C_COMPILER gcc) + find_program(CXX_COMPILER g++) +endif() +set(CMAKE_C_COMPILER ${C_COMPILER}) +set(CMAKE_CXX_COMPILER ${CXX_COMPILER}) + +# Install all libraries required by tarantool at current build dir + +# +# OpenSSL +# +ExternalProject_Add(openssl + URL https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + /config + --prefix= + --libdir=lib + no-shared + INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install_sw +) + +# +# ICU +# +ExternalProject_Add(icu + URL https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + /source/configure + --with-data-packaging=static + --prefix= + --disable-shared + --enable-static +) + +# +# ZLIB +# +ExternalProject_Add(zlib + URL https://zlib.net/zlib-${ZLIB_VERSION}.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + /configure + --prefix= + --static + TEST_COMMAND ${CMAKE_MAKE_PROGRAM} check +) + +# +# Ncurses +# +ExternalProject_Add(ncurses + URL https://ftp.gnu.org/gnu/ncurses/ncurses-${NCURSES_VERSION}.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + /configure + --prefix= +) + +# +# ReadLine +# +ExternalProject_Add(readline + URL https://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + /configure + --prefix= + --disable-shared + # STEP_TARGETS download +) + +# +# ICONV +# +if (APPLE) + ExternalProject_Add(iconv + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + /configure + --prefix= + --disable-shared + --enable-static + --with-gnu-ld + STEP_TARGETS download + ) +else() + # In linux iconv is embedded into glibc + # So we find system header and copy it locally + find_path(ICONV_INCLUDE_DIR iconv.h) + if(NOT ICONV_INCLUDE_DIR) + message(FATAL_ERROR "iconv include header not found") + endif() + + set(ICONV_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix") + + add_custom_command( + OUTPUT "${ICONV_INSTALL_PREFIX}/include/iconv.h" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${ICONV_INSTALL_PREFIX}/include" + COMMAND ${CMAKE_COMMAND} -E copy + "${ICONV_INCLUDE_DIR}/iconv.h" + "${ICONV_INSTALL_PREFIX}/include/iconv.h" + ) + add_custom_target(iconv + DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h" + ) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property + set_target_properties(iconv + PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX} + ) +endif() + +# +# Unwind +# +if (APPLE) + # On macOS libunwind is a part of MacOSX.sdk + # So we need to find library and header and + # copy it locally + find_path(UNWIND_INCLUDE_DIR libunwind.h) + find_library(UNWIND_LIBRARY libunwind.dylib + PATH_SUFFIXES system + ) + + set(UNWIND_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/unwind-prefix") + + set(UNWIND_DEPENDENCIES) + + if (UNWIND_INCLUDE_DIR AND UNWIND_LIBRARY) + add_custom_command( + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/unwind.h" + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/libunwind.h" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${UNWIND_INSTALL_PREFIX}/include" + COMMAND ${CMAKE_COMMAND} -E copy + "${UNWIND_INCLUDE_DIR}/libunwind.h" + "${UNWIND_INCLUDE_DIR}/unwind.h" + "${UNWIND_INSTALL_PREFIX}/include/" + ) + add_custom_command( + OUTPUT "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${UNWIND_INSTALL_PREFIX}/lib" + COMMAND ${CMAKE_COMMAND} -E copy + "${UNWIND_LIBRARY}" + "${UNWIND_INSTALL_PREFIX}/lib/" + ) + set(UNWIND_DEPENDENCIES + ${UNWIND_DEPENDENCIES} + "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib" + "${UNWIND_INSTALL_PREFIX}/include/libunwind.h" + ) + else() + message(STATUS "Unwind not found") + endif() + + add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES}) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property + set_target_properties(unwind + PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX} + ) +else() + ExternalProject_Add(unwind + URL https://download.savannah.nongnu.org/releases/libunwind/libunwind-${UNWIND_VERSION}.tar.gz + CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + /configure + --prefix= + --disable-shared + --enable-static + --disable-minidebuginfo # to prevent linking with liblzma + STEP_TARGETS download + ) +endif() + +# Get install directories of builded libraries for building +# tarantool with custon CMAKE_PREFIX_PATH +foreach(PROJ openssl icu zlib ncurses readline iconv unwind) + ExternalProject_Get_Property(${PROJ} install_dir) + set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir}) + set(TARANTOOL_DEPENDS ${PROJ} ${TARANTOOL_DEPENDS}) + message(STATUS "Add external project ${PROJ} in ${install_dir}") +endforeach() + +ExternalProject_Add(tarantool + DEPENDS ${TARANTOOL_DEPENDS} + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/.. + LIST_SEPARATOR : + CMAKE_ARGS + # Override LOCALSTATEDIR to avoid cmake "special" cases: + # https://cmake.org/cmake/help/v3.4/module/GNUInstallDirs.html#special-cases + -DCMAKE_INSTALL_LOCALSTATEDIR=/var + -DCMAKE_INSTALL_PREFIX= + -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH} + -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE + -DOPENSSL_USE_STATIC_LIBS=TRUE + -DBUILD_STATIC=TRUE + -DENABLE_DIST=TRUE + -DENABLE_BACKTRACE=TRUE + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + ${CMAKE_TARANTOOL_ARGS} + STEP_TARGETS build + BUILD_COMMAND $(MAKE) +) + +enable_testing() +ExternalProject_Get_Property(tarantool binary_dir) +SET(TARANTOOL_BINARY_DIR ${binary_dir}) + +add_test( + NAME check-dependencies + COMMAND ${CMAKE_COMMAND} + -D FILE=${TARANTOOL_BINARY_DIR}/src/tarantool + -P CheckDependencies.cmake + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test +) + +add_test( + NAME test-run-static + COMMAND ./test-run.py --builddir ${TARANTOOL_BINARY_DIR} + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test +) diff --git a/static-build/README.md b/static-build/README.md new file mode 100644 index 000000000..0019e963f --- /dev/null +++ b/static-build/README.md @@ -0,0 +1,58 @@ +# Tarantool static build tooling + +These files help to prepare environment for building Tarantool +statically. And builds it. + +## Prerequisites + +CentOS: + +```bash +yum install -y \ + git perl gcc cmake make gcc-c++ libstdc++-static autoconf automake libtool \ + python-msgpack python-yaml python-argparse python-six python-gevent +``` + +MacOS: + +Before you start please install default Xcode Tools by Apple: + +``` +sudo xcode-select --install +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer +``` + +Install brew using command from +[Homebrew repository instructions](https://github.com/Homebrew/inst) + +After that run next script: + +```bash + brew install autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rbs + pip install --force-reinstall -r test-run/requirements.txt +``` + +### Usage + +```bash +cmake . +make -j +ctest -V +``` + +## Customize your build + +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` variable + +### Usage + +There is three types of `CMAKE_BUILD_TYPE`: +* Debug - default +* Release +* RelWithDebInfo + +And you want to build tarantool with RelWithDebInfo: + +```bash +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" . +``` diff --git a/static-build/test/CheckDependencies.cmake b/static-build/test/CheckDependencies.cmake new file mode 100644 index 000000000..49e91e7fa --- /dev/null +++ b/static-build/test/CheckDependencies.cmake @@ -0,0 +1,43 @@ +## This is a cmake-based test, it checks that tarantool static binary +# has no dependencies except allowed ones. + +include(GetPrerequisites) +if(NOT FILE) + message(FATAL_ERROR "Usage: " + "${CMAKE_COMMAND} -DFILE= -P CheckDependencies.cmake") +elseif(NOT EXISTS ${FILE}) + message(FATAL_ERROR "${FILE}: No such file") +endif() + +get_prerequisites(${FILE} DEPENDENCIES 0 0 "" "") + +if (APPLE) + set(ALLOWLIST + libSystem + CoreFoundation + libc++ + ) +elseif(UNIX) + set(ALLOWLIST + libdl + librt + libc + libm + libgcc_s + libpthread + ) +else() + message(FATAL_ERROR "Unknown platform") +endif() + +foreach(DEPENDENCY_FILE ${DEPENDENCIES}) + message("Dependency: ${DEPENDENCY_FILE}") +endforeach() + +foreach(DEPENDENCY_FILE ${DEPENDENCIES}) + get_filename_component(libname ${DEPENDENCY_FILE} NAME_WE) + list (FIND ALLOWLIST ${libname} _index) + if (_index EQUAL -1) + message(FATAL_ERROR "Blocklisted dependency: ${DEPENDENCY_FILE}") + endif() +endforeach() diff --git a/static-build/test/static-build/box.lua b/static-build/test/static-build/box.lua new file mode 100755 index 000000000..bad8a9055 --- /dev/null +++ b/static-build/test/static-build/box.lua @@ -0,0 +1,3 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) diff --git a/static-build/test/static-build/exports.test.lua b/static-build/test/static-build/exports.test.lua new file mode 100755 index 000000000..63dc163a9 --- /dev/null +++ b/static-build/test/static-build/exports.test.lua @@ -0,0 +1,148 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local ffi = require('ffi') +ffi.cdef([[ + void *dlsym(void *handle, const char *symbol); +]]) + +local test = tap.test('exports') + + +local RTLD_DEFAULT +-- See `man 3 dlsym`: +-- RTLD_DEFAULT +-- Find the first occurrence of the desired symbol using the default +-- shared object search order. The search will include global symbols +-- in the executable and its dependencies, as well as symbols in shared +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. +if jit.os == "OSX" then + RTLD_DEFAULT = ffi.cast("void *", -2LL) +else + RTLD_DEFAULT = ffi.cast("void *", 0LL) +end + +local function check_symbol(sym) + test:ok(ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, ('Symbol %q found'):format(sym)) +end + +local check_symbols = { + -- FFI + + 'guava', + 'base64_decode', + 'base64_encode', + 'SHA1internal', + 'random_bytes', + 'fiber_time', + 'ibuf_create', + 'ibuf_destroy', + 'port_destroy', + 'csv_create', + 'csv_destroy', + 'title_get', + 'title_update', + 'tnt_iconv', + 'tnt_iconv_open', + 'tnt_iconv_close', + 'exception_get_int', + 'exception_get_string', + + 'tarantool_lua_ibuf', + 'uuid_nil', + 'tt_uuid_create', + 'tt_uuid_str', + 'tt_uuid_is_equal', + 'tt_uuid_is_nil', + 'tt_uuid_bswap', + 'tt_uuid_from_string', + 'log_level', + 'log_format', + 'uri_parse', + 'uri_format', + 'PMurHash32', + 'PMurHash32_Process', + 'PMurHash32_Result', + 'crc32_calc', + 'mp_encode_double', + 'mp_encode_float', + 'mp_encode_decimal', + 'mp_decode_double', + 'mp_decode_float', + 'mp_decode_extl', + 'mp_sizeof_decimal', + 'decimal_unpack', + + 'log_type', + 'say_set_log_level', + 'say_logrotate', + 'say_set_log_format', + 'tarantool_uptime', + 'tarantool_exit', + 'log_pid', + 'space_by_id', + 'space_run_triggers', + 'space_bsize', + 'box_schema_version', + + 'crypto_EVP_MD_CTX_new', + 'crypto_EVP_MD_CTX_free', + 'crypto_HMAC_CTX_new', + 'crypto_HMAC_CTX_free', + 'crypto_stream_new', + 'crypto_stream_begin', + 'crypto_stream_append', + 'crypto_stream_commit', + 'crypto_stream_delete', + + -- Module API + + '_say', + 'swim_cfg', + 'swim_quit', + 'fiber_new', + 'fiber_cancel', + 'coio_wait', + 'coio_close', + 'coio_call', + 'coio_getaddrinfo', + 'luaT_call', + 'box_txn', + 'box_select', + 'clock_realtime', + 'string_strip_helper', + + -- Lua / LuaJIT + + 'lua_newstate', + 'lua_close', + 'luaL_loadstring', + 'luaJIT_profile_start', + 'luaJIT_profile_stop', + 'luaJIT_profile_dumpstack', + + 'ERR_error_string', + 'ERR_get_error', + + 'EVP_get_digestbyname', + 'EVP_get_cipherbyname', + 'EVP_CIPHER_CTX_new', + 'EVP_CIPHER_CTX_free', + 'EVP_CIPHER_block_size', + 'HMAC_Init_ex', + 'HMAC_Update', + 'HMAC_Final', + + 'ZSTD_compress', + 'ZSTD_decompress', + 'ZSTD_free', + 'ZSTD_malloc', + 'ZSTD_versionString', +} + +test:plan(#check_symbols) +for _, sym in ipairs(check_symbols) do + check_symbol(sym) +end + +os.exit(test:check() and 0 or 1) diff --git a/static-build/test/static-build/suite.ini b/static-build/test/static-build/suite.ini new file mode 100644 index 000000000..92e349466 --- /dev/null +++ b/static-build/test/static-build/suite.ini @@ -0,0 +1,5 @@ +[default] +core = app +description = Static build tests +script = box.lua +is_parallel = True diff --git a/static-build/test/static-build/traceback.test.lua b/static-build/test/static-build/traceback.test.lua new file mode 100755 index 000000000..32e1ed8b5 --- /dev/null +++ b/static-build/test/static-build/traceback.test.lua @@ -0,0 +1,15 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local fiber = require('fiber') + +local test = tap.test('traceback') +test:plan(1) + +local info = fiber.info()[fiber.id()] +test:ok(info.backtrace ~= nil, 'fiber.info() has backtrace') +for _, l in pairs(info.backtrace or {}) do + test:diag('%s: %s', next(l)) +end + +os.exit(test:check() and 0 or 1) diff --git a/static-build/test/test-run.py b/static-build/test/test-run.py new file mode 120000 index 000000000..02187cdc5 --- /dev/null +++ b/static-build/test/test-run.py @@ -0,0 +1 @@ +../../test-run/test-run.py \ No newline at end of file -- 2.26.2 From alexander.turenko at tarantool.org Tue Aug 25 17:51:03 2020 From: alexander.turenko at tarantool.org (Alexander Turenko) Date: Tue, 25 Aug 2020 17:51:03 +0300 Subject: [Tarantool-patches] [PATCH v2] build: refactor static build process In-Reply-To: <20200825140108.52090-1-huston.mavr@gmail.com> References: <20200825140108.52090-1-huston.mavr@gmail.com> Message-ID: <20200825145103.hiqmjkimuk3al3rt@tkn_work_nb> I looked very briefly (not thoroughly at all) on this iteration. There is nothing that confuses me (except few tiny comments below). I hope Igor will do thorough review. WBR, Alexander Turenko. > +if (APPLE) > + find_program(C_COMPILER clang) > + find_program(CXX_COMPILER clang++) > +else() > + find_program(C_COMPILER gcc) > + find_program(CXX_COMPILER g++) > +endif() Can we just leave it default? In offline discussion Alexandr B. said that tarantool builds with gcc, but icu with clang that gives some problem. Possible solution is to pass ${CMAKE_C_COMPILER} (and CXX too where necessary) to a subproject as we do for c-ares and curl. It seems it is already done, so maybe it worth to re-check whether it solves the problem. Anyway, if we really need to set a compiler here explicitly, I don't mind. Just noted that this way is somewhat unusual as I see. > diff --git a/static-build/test/static-build/box.lua b/static-build/test/static-build/box.lua > new file mode 100755 > index 000000000..bad8a9055 > --- /dev/null > +++ b/static-build/test/static-build/box.lua > @@ -0,0 +1,3 @@ > +#!/usr/bin/env tarantool > + > +require('console').listen(os.getenv('ADMIN')) Is looks redundant, see the comment below. > diff --git a/static-build/test/static-build/suite.ini b/static-build/test/static-build/suite.ini > new file mode 100644 > index 000000000..92e349466 > --- /dev/null > +++ b/static-build/test/static-build/suite.ini > @@ -0,0 +1,5 @@ > +[default] > +core = app > +description = Static build tests > +script = box.lua > +is_parallel = True 'script' does not have sense for 'core = app' test, it is for 'core = tarantool' tests. From alyapunov at tarantool.org Tue Aug 25 18:23:03 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Tue, 25 Aug 2020 18:23:03 +0300 Subject: [Tarantool-patches] [PATCH] test: concurrent tuple update segfault on bitset index iteration In-Reply-To: <20200819172324.6188-1-i.kosarev@tarantool.org> References: <20200819172324.6188-1-i.kosarev@tarantool.org> Message-ID: Hi, thanks for the patch, LGTM. On 8/19/20 8:23 PM, Ilya Kosarev wrote: > Concurrent tuple update could segfault on BITSET_ALL_NOT_SET iterator > usage. Fixed in 850054b2dbca257076c3f7c22e00564ac55b70d5. > > Closes #1088 > --- > Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-1088-concurrent-tuple-update-segfault-with-bitset-index > Issue: https://github.com/tarantool/tarantool/issues/1088 > > test/box/bitset.result | 86 ++++++++++++++++++++++++++++++++++++++++ > test/box/bitset.test.lua | 42 ++++++++++++++++++++ > 2 files changed, 128 insertions(+) > > diff --git a/test/box/bitset.result b/test/box/bitset.result > index bf44773ef3..5da068385c 100644 > --- a/test/box/bitset.result > +++ b/test/box/bitset.result > @@ -2020,3 +2020,89 @@ s:drop() > box.schema.func.drop('s') > --- > ... > +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration > +test_run = require('test_run').new() > +--- > +... > +fiber = require('fiber') > +--- > +... > +s = box.schema.space.create('gh-1088') > +--- > +... > +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) > +--- > +... > +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) > +--- > +... > +for i = 1, 100 do s:insert{i, 0, i - 1} end > +--- > +... > +counter = 0 > +--- > +... > +test_run:cmd("setopt delimiter ';'") > +--- > +- true > +... > +function update() > + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do > + counter = counter + 1 > + s:update(t[1], {{'+', 3, 11}}) > + fiber.sleep(0) > + end > + fiber.self():cancel() > +end; > +--- > +... > +test_run:cmd("setopt delimiter ''"); > +--- > +- true > +... > +fibers = {} > +--- > +... > +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end > +--- > +... > +updating = true > +--- > +... > +test_run:cmd("setopt delimiter ';'") > +--- > +- true > +... > +while updating do > + updating = false > + for _, f in pairs(fibers) do > + if f:status() ~= 'dead' then updating = true end > + end > + fiber.sleep(0.001) > +end; > +--- > +... > +test_run:cmd("setopt delimiter ''"); > +--- > +- true > +... > +s:get(1) > +--- > +- [1, 0, 1100] > +... > +s:get(2) > +--- > +- [2, 0, 1101] > +... > +s:get(3) > +--- > +- [3, 0, 1102] > +... > +s:get(4) > +--- > +- [4, 0, 1103] > +... > +counter -- total updates counter > +--- > +- 10000 > +... > diff --git a/test/box/bitset.test.lua b/test/box/bitset.test.lua > index d644d34e0b..dd432edeb0 100644 > --- a/test/box/bitset.test.lua > +++ b/test/box/bitset.test.lua > @@ -162,3 +162,45 @@ _ = s:create_index('pk') > _ = s:create_index('idx', {type = 'bitset', func = box.func.s.id, parts = {{1, 'unsigned'}}}) > s:drop() > box.schema.func.drop('s') > + > +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration > +test_run = require('test_run').new() > +fiber = require('fiber') > + > +s = box.schema.space.create('gh-1088') > +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) > +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) > +for i = 1, 100 do s:insert{i, 0, i - 1} end > + > +counter = 0 > +test_run:cmd("setopt delimiter ';'") > +function update() > + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do > + counter = counter + 1 > + s:update(t[1], {{'+', 3, 11}}) > + fiber.sleep(0) > + end > + fiber.self():cancel() > +end; > +test_run:cmd("setopt delimiter ''"); > + > +fibers = {} > +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end > + > +updating = true > +test_run:cmd("setopt delimiter ';'") > +while updating do > + updating = false > + for _, f in pairs(fibers) do > + if f:status() ~= 'dead' then updating = true end > + end > + fiber.sleep(0.001) > +end; > +test_run:cmd("setopt delimiter ''"); > + > +s:get(1) > +s:get(2) > +s:get(3) > +s:get(4) > + > +counter -- total updates counter From avtikhon at tarantool.org Wed Aug 26 09:18:16 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Wed, 26 Aug 2020 09:18:16 +0300 Subject: [Tarantool-patches] [PATCH v1] test: restore skpicond for asan unit/guard Message-ID: <6f35dd8da45de00586fb570608fd799ea10165df.1598422646.git.avtikhon@tarantool.org> Found that after commit: eec54b5741c421da51a4b58cac774b77d2dff6fc "asan/lsan: cleanup suppression lists" where was removed blocking file unit/guard.skipcond for asan testing, it was needed much more fixes from #4609, to enable this test on asan. After investigation the group of needed patches to be ported decided not to use it for 2.4 release branch and to restore the skip condition file to block the issue with the test to restore the release branch stability. Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4360-skip-test-guard-2.4 Issue: https://github.com/tarantool/tarantool/issues/4360 test/unit/guard.skipcond | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 test/unit/guard.skipcond diff --git a/test/unit/guard.skipcond b/test/unit/guard.skipcond new file mode 100644 index 000000000..e46fd1088 --- /dev/null +++ b/test/unit/guard.skipcond @@ -0,0 +1,7 @@ +import os + +# Disabled at ASAN build due to issue #4360. +if os.getenv("ASAN") == 'ON': + self.skip = 1 + +# vim: set ft=python: -- 2.17.1 From sergepetrenko at tarantool.org Wed Aug 26 10:31:37 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Wed, 26 Aug 2020 10:31:37 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_2/3=5D_replication=3A_test?= =?utf-8?q?_clear=5Fsynchro=5Fqueue_function?= In-Reply-To: <20200825124939.GB47610@pony.bronevichok.ru> References: <8c3b7d4c-22e8-108a-b15e-ef3c89510a63@tarantool.org> <20200825124939.GB47610@pony.bronevichok.ru> Message-ID: <1598427097.413655454@f128.i.mail.ru> Hi, Sergey! Thanks for the patch and the fixes. New version LGTM. I couldn?t find the other patches in my mailbox though, could you please resend them? If you need my review, of course. >???????, 25 ??????? 2020, 15:49 +03:00 ?? Sergey Bronnikov : >? >Vlad, thanks for review! >Patches updated in a branch and please see my answers inline. > >On 00:00 Tue 21 Jul , Vladislav Shpilevoy wrote: >> Thanks for the patch! >> >> See 8 comments below. >> >> On 09.07.2020 19:16, sergeyb at tarantool.org wrote: >> > From: Sergey Bronnikov < sergeyb at tarantool.org > >> > >> > Part of #5055 >> > Part of #4849 >> > --- >> > test/replication/qsync_basic.result | 85 +++++++++++++++++++++++++++ >> > test/replication/qsync_basic.test.lua | 31 ++++++++++ >> > 2 files changed, 116 insertions(+) >> > >> > diff --git a/test/replication/qsync_basic.result b/test/replication/qsync_basic.result >> > index ab4be0c7e..464df75a7 100644 >> > --- a/test/replication/qsync_basic.result >> > +++ b/test/replication/qsync_basic.result >> > @@ -32,6 +32,14 @@ s2.is_sync >> > | - false >> > | ... >> > >> > +-- >> > +-- gh-4849: clear synchro queue with unconfigured box >> > +-- >> > +box.ctl.clear_synchro_queue() >> >> 1. Enough to test that it does not crash here. No need to >> check result. > >Well, removed assignment to variable. > >> > + | --- >> > + | - -1 >> > + | ... >> > + >> > -- Net.box takes sync into account. >> > box.schema.user.grant('guest', 'super') >> > | --- >> > @@ -553,6 +561,82 @@ box.space.sync:select{7} >> > | - - [7] >> > | ... >> > >> > +-- >> > +-- gh-4849: clear synchro queue on a replica >> > +-- >> > +test_run:switch('default') >> > + | --- >> > + | - true >> > + | ... >> > +box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 2} >> >> 2. If you want this timeout to fail, it is too big. The test will be too >> long. If it is supposed not to fail, then it is way too small. If you want >> it to fail, it should be something like 0.001. If you want it to hold, it >> should be 1000 to be sure. Keep in mind that you can change the timeout on >> fly. That allows to build quite complex reactive tests completely event-based. > >set replication_synchro_timeout value to 1000 > >> > + | --- >> > + | ... >> > +f1 = fiber.create(box.space.sync.replace, box.space.sync, {9}) >> >> 3. You need to extract the exact result value. Use pcall for that, and print >> its result after the fiber is dead. See other examples with fiber.create() in >> qsync test suite. The same for the next test case. > >Done. > >> > + | --- >> > + | ... >> > +f1:status() >> > + | --- >> > + | - suspended >> > + | ... >> > +test_run:switch('replica') >> > + | --- >> > + | - true >> >> 4. Better wait until the value is delivered. Otherwise you can switch to >> replica before master finishes WAL write, and the queue will be empty here. >> Try >> >> test_run:wait_cond(function() return box.space.sync:get{9} ~= nil end) > >Agree, added wait_cond(). > >> > + | ... >> > +box.ctl.clear_synchro_queue() >> > + | --- >> > + | - 0 >> > + | ... >> > +box.space.sync:select{9} >> > + | --- >> > + | - [] >> >> 5. If select returns 9 before queue clean, and returns empty afterwards, >> it means the queue was cleared. So here the queue size is not really needed, >> as you can see. > >removed this and next select{} statements > >> > + | ... >> > +test_run:switch('default') >> > + | --- >> > + | - true >> > + | ... >> > +box.space.sync:select{9} >> > + | --- >> > + | - [] >> > + | ... >> > +f1:status() >> > + | --- >> > + | - dead >> > + | ... >> > + >> > +-- >> > +-- gh-4849: clear synchro queue on a master >> >> 6. Since the previous test the replica's WAL is different from master's. >> I am not sure the replication is still alive. > >Test with clear_synchro_queue() on master keeps master alive at the end >of test so I moved it before the same test for the replica. Also added >note for others that cluster may be in a broken state here. > >> > +-- >> > +test_run:switch('default') >> > + | --- >> > + | - true >> > + | ... >> > +box.cfg{replication_synchro_quorum = 3, replication_synchro_timeout = 2} >> > + | --- >> > + | ... >> > +f1 = fiber.create(box.space.sync.replace, box.space.sync, {10}) >> > + | --- >> > + | ... >> > +f1:status() >> > + | --- >> > + | - suspended >> > + | ... >> > +box.ctl.clear_synchro_queue() >> > + | --- >> > + | - -2 >> > + | ... >> > +box.space.sync:select{10} >> > + | --- >> > + | - - [10] >> >> 7. Why is it 10? The quorum is not reached, it should have been rolled back. > >Added "box.cfg{replication_synchro_timeout = 0.1}" before clear_synchro_queue() >call and now tx is rolled back. Also I've added checks of current value with >wait_cond() that is more reliable than select{} on a master and replica. >Updated output looks like this: > >box.ctl.clear_synchro_queue() >?| --- >?| ... >test_run:switch('replica') >?| --- >?| - true >?| ... >test_run:wait_cond(function() return box.space.sync:get{10} == nil end) >?| --- >?| - true >?| ... >test_run:switch('default') >?| --- >?| - true >?| ... >test_run:wait_cond(function() return f:status() == 'dead' end) >?| --- >?| - true >?| ... >ok, err >?| --- >?| - false >?| - Quorum collection for a synchronous transaction is timed out >?| ... >test_run:wait_cond(function() return box.space.sync:get{10} == nil end) >?| --- >?| - true >?| ... > >> > + | ... >> > +test_run:switch('replica') >> > + | --- >> > + | - true >> > + | ... >> > +box.space.sync:select{10} >> > + | --- >> > + | - - [10] >> > + | ... >> > + >> > -- Cleanup. >> > test_run:cmd('switch default') >> > | --- >> > @@ -576,6 +660,7 @@ test_run:cmd('delete server replica') >> > | ... >> > box.space.test:drop() >> > | --- >> > + | - error: A rollback for a synchronous transaction is received >> >> 8. Why is it changed? > >Perhaps it is because in previous version of test I haven't wait fibers 'dead'. >There is no error now: > >box.space.test:drop() >?| --- >?| ... >box.space.sync:drop() >?| --- >?| ... > >> >> > | ... >> > box.space.sync:drop() >> > | --- ? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Wed Aug 26 10:52:32 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:32 +0300 Subject: [Tarantool-patches] [RAFT 00/10] raft implementation Message-ID: From: sergepetrenko The patchset contains current part of raft implementation: persistent raft state and raft status messages broadcasting together with some follow-up fixes. The raft state machine still has to be implemented. Vladislav Shpilevoy (9): raft: introduce persistent raft state [tosquash] raft: return raft_request to xrow [tosquash] raft: introduce IPROTO_RAFT_VCLOCK [tosquash] xrow: refactor raft request codec [tosquash] raft: don't fill raft_request manually [tosquash] raft: rename curr_leader to leader [tosquash] raft: rename raft_process to raft_process_recovery [tosquash] applier: handler error at raft row appliance [tosquash] relay: move raft broadcast details into relay sergepetrenko (1): raft: relay status updates to followers src/box/CMakeLists.txt | 1 + src/box/applier.cc | 35 ++++++- src/box/box.cc | 23 ++++- src/box/iproto_constants.h | 15 +++ src/box/lua/misc.cc | 35 +++++++ src/box/memtx_engine.c | 40 ++++++++ src/box/raft.c | 181 +++++++++++++++++++++++++++++++++++++ src/box/raft.h | 81 +++++++++++++++++ src/box/relay.cc | 62 ++++++++++++- src/box/relay.h | 9 ++ src/box/xrow.c | 113 +++++++++++++++++++++++ src/box/xrow.h | 15 +++ 12 files changed, 603 insertions(+), 7 deletions(-) create mode 100644 src/box/raft.c create mode 100644 src/box/raft.h -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:33 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:33 +0300 Subject: [Tarantool-patches] [RAFT 01/10] raft: introduce persistent raft state In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy box.internal.raft_*() helper functions were introduced to test the persistency. Any state change is saved into WAL and into snapshot. --- src/box/CMakeLists.txt | 1 + src/box/box.cc | 8 +++ src/box/iproto_constants.h | 13 ++++ src/box/lua/misc.cc | 35 +++++++++++ src/box/memtx_engine.c | 35 +++++++++++ src/box/raft.c | 120 +++++++++++++++++++++++++++++++++++++ src/box/raft.h | 61 +++++++++++++++++++ src/box/xrow.c | 56 +++++++++++++++++ src/box/xrow.h | 12 ++++ 9 files changed, 341 insertions(+) create mode 100644 src/box/raft.c create mode 100644 src/box/raft.h diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt index b8b2689d2..29c3bfe79 100644 --- a/src/box/CMakeLists.txt +++ b/src/box/CMakeLists.txt @@ -170,6 +170,7 @@ add_library(box STATIC port.c txn.c txn_limbo.c + raft.c box.cc gc.c checkpoint_schedule.c diff --git a/src/box/box.cc b/src/box/box.cc index faffd5769..c0adccc6a 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -78,6 +78,7 @@ #include "sequence.h" #include "sql_stmt_cache.h" #include "msgpack.h" +#include "raft.h" #include "trivia/util.h" static char status[64] = "unknown"; @@ -374,6 +375,13 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) diag_raise(); return; } + if (iproto_type_is_raft_request(row->type)) { + struct raft_request raft_req; + if (xrow_decode_raft(row, &raft_req) != 0) + diag_raise(); + raft_process(&raft_req); + return; + } xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); if (request.type != IPROTO_NOP) { struct space *space = space_cache_find_xc(request.space_id); diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h index 4f5a2b195..8a11626b3 100644 --- a/src/box/iproto_constants.h +++ b/src/box/iproto_constants.h @@ -219,6 +219,8 @@ enum iproto_type { /** The maximum typecode used for box.stat() */ IPROTO_TYPE_STAT_MAX, + IPROTO_RAFT = 30, + /** A confirmation message for synchronous transactions. */ IPROTO_CONFIRM = 40, /** A rollback message for synchronous transactions. */ @@ -258,6 +260,11 @@ enum iproto_type { /** IPROTO type name by code */ extern const char *iproto_type_strs[]; +enum iproto_raft_keys { + IPROTO_RAFT_TERM = 0, + IPROTO_RAFT_VOTE = 1, +}; + /** * Returns IPROTO type name by @a type code. * @param type IPROTO type. @@ -332,6 +339,12 @@ iproto_type_is_synchro_request(uint32_t type) return type == IPROTO_CONFIRM || type == IPROTO_ROLLBACK; } +static inline bool +iproto_type_is_raft_request(uint32_t type) +{ + return type == IPROTO_RAFT; +} + /** This is an error. */ static inline bool iproto_type_is_error(uint32_t type) diff --git a/src/box/lua/misc.cc b/src/box/lua/misc.cc index 5da84b35a..98e98abe2 100644 --- a/src/box/lua/misc.cc +++ b/src/box/lua/misc.cc @@ -40,6 +40,8 @@ #include "box/tuple.h" #include "box/tuple_format.h" #include "box/lua/tuple.h" +#include "box/raft.h" +#include "box/xrow.h" #include "mpstream/mpstream.h" static uint32_t CTID_STRUCT_TUPLE_FORMAT_PTR; @@ -246,12 +248,45 @@ lbox_tuple_format_new(struct lua_State *L) /* }}} */ +static int +lbox_raft_new_term(struct lua_State *L) +{ + uint64_t min_term = luaL_checkuint64(L, 1); + raft_new_term(min_term); + return 0; +} + +static int +lbox_raft_vote(struct lua_State *L) +{ + uint64_t vote_for = luaL_checkuint64(L, 1); + if (vote_for > UINT32_MAX) + return luaL_error(L, "Invalid vote"); + raft_vote(vote_for); + return 0; +} + +static int +lbox_raft_get(struct lua_State *L) +{ + lua_createtable(L, 0, 2); + luaL_pushuint64(L, raft.term); + lua_setfield(L, -2, "term"); + luaL_pushuint64(L, raft.vote); + lua_setfield(L, -2, "vote"); + return 1; +} + void box_lua_misc_init(struct lua_State *L) { static const struct luaL_Reg boxlib_internal[] = { {"select", lbox_select}, {"new_tuple_format", lbox_tuple_format_new}, + /* Temporary helpers to sanity test raft persistency. */ + {"raft_new_term", lbox_raft_new_term}, + {"raft_vote", lbox_raft_vote}, + {"raft_get", lbox_raft_get}, {NULL, NULL} }; diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..26274de80 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -48,6 +48,7 @@ #include "replication.h" #include "schema.h" #include "gc.h" +#include "raft.h" /* sync snapshot every 16MB */ #define SNAP_SYNC_INTERVAL (1 << 24) @@ -200,12 +201,25 @@ memtx_engine_recover_snapshot(struct memtx_engine *memtx, return 0; } +static int +memtx_engine_recover_raft(const struct xrow_header *row) +{ + assert(row->type == IPROTO_RAFT); + struct raft_request req; + if (xrow_decode_raft(row, &req) != 0) + return -1; + raft_process(&req); + return 0; +} + static int memtx_engine_recover_snapshot_row(struct memtx_engine *memtx, struct xrow_header *row) { assert(row->bodycnt == 1); /* always 1 for read */ if (row->type != IPROTO_INSERT) { + if (row->type == IPROTO_RAFT) + return memtx_engine_recover_raft(row); diag_set(ClientError, ER_UNKNOWN_REQUEST_TYPE, (uint32_t) row->type); return -1; @@ -477,6 +491,7 @@ struct checkpoint { /** The vclock of the snapshot file. */ struct vclock vclock; struct xdir dir; + struct raft_request raft; /** * Do nothing, just touch the snapshot file - the * checkpoint already exists. @@ -501,6 +516,7 @@ checkpoint_new(const char *snap_dirname, uint64_t snap_io_rate_limit) opts.free_cache = true; xdir_create(&ckpt->dir, snap_dirname, SNAP, &INSTANCE_UUID, &opts); vclock_create(&ckpt->vclock); + raft_serialize(&ckpt->raft); ckpt->touch = false; return ckpt; } @@ -572,6 +588,23 @@ checkpoint_add_space(struct space *sp, void *data) return 0; }; +static int +checkpoint_write_raft(struct xlog *l, const struct raft_request *req) +{ + struct xrow_header row; + struct region *region = &fiber()->gc; + uint32_t svp = region_used(region); + int rc = -1; + if (xrow_encode_raft(&row, region, req) != 0) + goto finish; + if (checkpoint_write_row(l, &row) != 0) + goto finish; + rc = 0; +finish: + region_truncate(region, svp); + return rc; +} + static int checkpoint_f(va_list ap) { @@ -607,6 +640,8 @@ checkpoint_f(va_list ap) if (rc != 0) goto fail; } + if (checkpoint_write_raft(&snap, &ckpt->raft) != 0) + goto fail; if (xlog_flush(&snap) < 0) goto fail; diff --git a/src/box/raft.c b/src/box/raft.c new file mode 100644 index 000000000..5465f46b6 --- /dev/null +++ b/src/box/raft.c @@ -0,0 +1,120 @@ +/* + * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include "raft.h" + +#include "error.h" +#include "journal.h" +#include "xrow.h" +#include "small/region.h" + +/** Raft state of this instance. */ +struct raft raft = { + .term = 0, + .vote = 0, +}; + +void +raft_process(const struct raft_request *req) +{ + if (req->term != 0) + raft.term = req->term; + if (req->vote != 0) + raft.vote = req->vote; +} + +void +raft_serialize(struct raft_request *req) +{ + req->term = raft.term; + req->vote = raft.vote; +} + +static void +raft_write_cb(struct journal_entry *entry) +{ + fiber_wakeup(entry->complete_data); +} + +static void +raft_write_request(const struct raft_request *req) +{ + struct region *region = &fiber()->gc; + uint32_t svp = region_used(region); + struct xrow_header row; + char buf[sizeof(struct journal_entry) + + sizeof(struct xrow_header *)]; + struct journal_entry *entry = (struct journal_entry *)buf; + entry->rows[0] = &row; + + if (xrow_encode_raft(&row, region, req) != 0) + goto fail; + journal_entry_create(entry, 1, xrow_approx_len(&row), raft_write_cb, + fiber()); + + if (journal_write(entry) != 0 || entry->res < 0) { + diag_set(ClientError, ER_WAL_IO); + diag_log(); + goto fail; + } + region_truncate(region, svp); + return; +fail: + /* + * XXX: the stub is supposed to be removed once it is defined what to do + * when a raft request WAL write fails. + */ + panic("Could not write a raft request to WAL\n"); +} + +void +raft_new_term(uint64_t min_new_term) +{ + if (raft.term < min_new_term) + raft.term = min_new_term + 1; + else + ++raft.term; + + struct raft_request req; + memset(&req, 0, sizeof(req)); + req.term = raft.term; + raft_write_request(&req); +} + +void +raft_vote(uint32_t vote_for) +{ + raft.vote = vote_for; + + struct raft_request req; + memset(&req, 0, sizeof(req)); + req.vote = vote_for; + raft_write_request(&req); +} diff --git a/src/box/raft.h b/src/box/raft.h new file mode 100644 index 000000000..1f392033d --- /dev/null +++ b/src/box/raft.h @@ -0,0 +1,61 @@ +#pragma once +/* + * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ +#include + +struct raft_request; + +#if defined(__cplusplus) +extern "C" { +#endif + +struct raft { + uint64_t term; + uint32_t vote; +}; + +extern struct raft raft; + +void +raft_new_term(uint64_t min_new_term); + +void +raft_vote(uint32_t vote_for); + +void +raft_process(const struct raft_request *req); + +void +raft_serialize(struct raft_request *req); + +#if defined(__cplusplus) +} +#endif diff --git a/src/box/xrow.c b/src/box/xrow.c index 95ddb1fe7..1923bacfc 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -954,6 +954,62 @@ xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req) return 0; } +int +xrow_encode_raft(struct xrow_header *row, struct region *region, + const struct raft_request *r) +{ + size_t size = mp_sizeof_map(2) + + mp_sizeof_uint(IPROTO_RAFT_TERM) + + mp_sizeof_uint(r->term) + + mp_sizeof_uint(IPROTO_RAFT_VOTE) + + mp_sizeof_uint(r->vote); + char *buf = region_alloc(region, size); + if (buf == NULL) { + diag_set(OutOfMemory, size, "region_alloc", "buf"); + return -1; + } + memset(row, 0, sizeof(*row)); + row->type = IPROTO_RAFT; + row->body[0].iov_base = buf; + row->body[0].iov_len = size; + row->group_id = GROUP_LOCAL; + row->bodycnt = 1; + buf = mp_encode_map(buf, 2); + buf = mp_encode_uint(buf, IPROTO_RAFT_TERM); + buf = mp_encode_uint(buf, r->term); + buf = mp_encode_uint(buf, IPROTO_RAFT_VOTE); + buf = mp_encode_uint(buf, r->vote); + return 0; +} + +int +xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) +{ + /* TODO: handle bad format. */ + assert(row->type == IPROTO_RAFT); + assert(row->bodycnt == 1); + assert(row->group_id == GROUP_LOCAL); + memset(r, 0, sizeof(*r)); + const char *pos = row->body[0].iov_base; + uint32_t map_size = mp_decode_map(&pos); + for (uint32_t i = 0; i < map_size; ++i) + { + uint64_t key = mp_decode_uint(&pos); + switch (key) { + case IPROTO_RAFT_TERM: + r->term = mp_decode_uint(&pos); + break; + case IPROTO_RAFT_VOTE: + r->vote = mp_decode_uint(&pos); + break; + default: + mp_next(&pos); + break; + } + } + return 0; +} + int xrow_to_iovec(const struct xrow_header *row, struct iovec *out) { diff --git a/src/box/xrow.h b/src/box/xrow.h index 58d47b12d..c234f6f88 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -264,6 +264,18 @@ xrow_encode_synchro(struct xrow_header *row, int xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); +struct raft_request { + uint64_t term; + uint32_t vote; +}; + +int +xrow_encode_raft(struct xrow_header *row, struct region *region, + const struct raft_request *r); + +int +xrow_decode_raft(const struct xrow_header *row, struct raft_request *r); + /** * CALL/EVAL request. */ -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:34 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:34 +0300 Subject: [Tarantool-patches] [RAFT 02/10] raft: relay status updates to followers In-Reply-To: References: Message-ID: From: sergepetrenko The patch introduces a new type of system message used to notify the followers of the instance's raft status updates. It's relay's responsibility to deliver the new system rows to its peers. The notification system reuses and extends the same row type used to persist raft state in WAL and snapshot. Part of #1146 Part of #5204 --- src/box/applier.cc | 34 +++++++++++++++++++--- src/box/box.cc | 17 ++++++++++- src/box/iproto_constants.h | 1 + src/box/raft.c | 58 ++++++++++++++++++++++++++++++++++++++ src/box/raft.h | 39 +++++++++++++++++++++++-- src/box/relay.cc | 34 ++++++++++++++++++++-- src/box/relay.h | 14 +++++++++ src/box/xrow.c | 38 +++++++++++++++++++++++-- src/box/xrow.h | 5 +--- 9 files changed, 225 insertions(+), 15 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index c1d07ca54..f27436b79 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -55,6 +55,7 @@ #include "scoped_guard.h" #include "txn_limbo.h" #include "journal.h" +#include "raft.h" STRS(applier_state, applier_STATE); @@ -298,6 +299,8 @@ apply_final_join_row(struct xrow_header *row) */ if (iproto_type_is_synchro_request(row->type)) return 0; + if (iproto_type_is_raft_request(row->type)) + return 0; struct txn *txn = txn_begin(); if (txn == NULL) return -1; @@ -876,6 +879,23 @@ err: return -1; } +static int +apply_raft_row(struct xrow_header *row) +{ + assert(iproto_type_is_raft_request(row->type)); + + struct raft_request req; + struct vclock candidate_clock; + req.vclock = &candidate_clock; + + if (xrow_decode_raft(row, &req) != 0) + return -1; + + raft_process_msg(&req); + + return 0; +} + /** * Apply all rows in the rows queue as a single transaction. * @@ -1219,11 +1239,17 @@ applier_subscribe(struct applier *applier) * In case of an heartbeat message wake a writer up * and check applier state. */ - if (stailq_first_entry(&rows, struct applier_tx_row, - next)->row.lsn == 0) - applier_signal_ack(applier); - else if (applier_apply_tx(&rows) != 0) + struct xrow_header *first_row = + &stailq_first_entry(&rows, struct applier_tx_row, + next)->row; + if (first_row->lsn == 0) { + if (unlikely(iproto_type_is_raft_request(first_row->type))) + apply_raft_row(first_row); + else + applier_signal_ack(applier); + } else if (applier_apply_tx(&rows) != 0) { diag_raise(); + } if (ibuf_used(ibuf) == 0) ibuf_reset(ibuf); diff --git a/src/box/box.cc b/src/box/box.cc index c0adccc6a..8323de531 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -2050,7 +2050,22 @@ box_process_subscribe(struct ev_io *io, struct xrow_header *header) tt_uuid_str(&replica_uuid), sio_socketname(io->fd)); say_info("remote vclock %s local vclock %s", vclock_to_string(&replica_clock), vclock_to_string(&vclock)); - + /* + * Send out the current raft state of the instance. + */ + if (raft.state != RAFT_STATE_NONE) { + struct raft_request req; + req.term = raft.term; + req.vote = raft.vote; + req.state = raft.state; + /* + * Omit the candidate vclock, since we've just + * sent it in subscribe response. + */ + req.vclock = NULL; + xrow_encode_raft(&row, &fiber()->gc, &req); + coio_write_xrow(io, &row); + } /* * Replica clock is used in gc state and recovery * initialization, so we need to replace the remote 0-th diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h index 8a11626b3..4217ce2e0 100644 --- a/src/box/iproto_constants.h +++ b/src/box/iproto_constants.h @@ -263,6 +263,7 @@ extern const char *iproto_type_strs[]; enum iproto_raft_keys { IPROTO_RAFT_TERM = 0, IPROTO_RAFT_VOTE = 1, + IPROTO_RAFT_STATE = 2, }; /** diff --git a/src/box/raft.c b/src/box/raft.c index 5465f46b6..839a7dfeb 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -34,11 +34,15 @@ #include "journal.h" #include "xrow.h" #include "small/region.h" +#include "replication.h" +#include "relay.h" /** Raft state of this instance. */ struct raft raft = { .term = 0, .vote = 0, + .curr_leader = 0, + .state = RAFT_STATE_NONE, }; void @@ -50,9 +54,36 @@ raft_process(const struct raft_request *req) raft.vote = req->vote; } +void +raft_process_msg(const struct raft_request *req) +{ + if (req->term > raft.term) { + // Update term. + // The logic will be similar, but the code + // below is for testing purposes. + raft.term = req->term; + } + if (req->vote > 0) { + // Check whether the vote's for us. + } + switch (req->state) { + case RAFT_STATE_FOLLOWER: + break; + case RAFT_STATE_CANDIDATE: + // Perform voting logic. + break; + case RAFT_STATE_LEADER: + // Switch to a new leader. + break; + default: + break; + } +} + void raft_serialize(struct raft_request *req) { + memset(req, 0, sizeof(*req)); req->term = raft.term; req->vote = raft.vote; } @@ -84,6 +115,9 @@ raft_write_request(const struct raft_request *req) diag_log(); goto fail; } + + raft_broadcast(req); + region_truncate(region, svp); return; fail: @@ -118,3 +152,27 @@ raft_vote(uint32_t vote_for) req.vote = vote_for; raft_write_request(&req); } + +void +raft_free_msg(struct cmsg *msg) +{ + free((void *)msg->route); + free(msg); +} + +void +raft_broadcast(const struct raft_request *req) +{ + replicaset_foreach(replica) { + if (replica->relay != NULL && replica->id != REPLICA_ID_NIL && + relay_get_state(replica->relay) == RELAY_FOLLOW) { + // TODO: think of a proper allocator. + struct raft_broadcast_msg *raft_msg = + calloc(1, sizeof(*raft_msg)); + raft_msg->req = *req; + struct cmsg_hop *route = calloc(2, sizeof(*route)); + relay_push_raft_msg(replica->relay, &raft_msg->base, + route); + } + } +} diff --git a/src/box/raft.h b/src/box/raft.h index 1f392033d..9cb39dd24 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -30,32 +30,67 @@ * SUCH DAMAGE. */ #include - -struct raft_request; +#include "cbus.h" #if defined(__cplusplus) extern "C" { #endif +enum raft_state { + RAFT_STATE_NONE = 0, + RAFT_STATE_FOLLOWER = 1, + RAFT_STATE_CANDIDATE = 2, + RAFT_STATE_LEADER = 3 +}; + struct raft { uint64_t term; uint32_t vote; + uint32_t curr_leader; + enum raft_state state; }; extern struct raft raft; +struct raft_request { + uint64_t term; + uint32_t vote; + enum raft_state state; + struct vclock *vclock; +}; + +struct raft_broadcast_msg { + struct cmsg base; + struct raft_request req; +}; + void raft_new_term(uint64_t min_new_term); void raft_vote(uint32_t vote_for); +/** Process a raft entry stored in WAL/snapshot. */ void raft_process(const struct raft_request *req); +/** Process a raft status message coming from the network. */ +void +raft_process_msg(const struct raft_request *req); + void raft_serialize(struct raft_request *req); +void +raft_free_msg(struct cmsg *msg); + +/** + * Broadcast the changes in this instance's raft status to all + * the followers. + */ +void +raft_broadcast(const struct raft_request *req); + #if defined(__cplusplus) } #endif diff --git a/src/box/relay.cc b/src/box/relay.cc index a7843a8c2..be252cad1 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -54,6 +54,7 @@ #include "xstream.h" #include "wal.h" #include "txn_limbo.h" +#include "raft.h" /** * Cbus message to send status updates from relay to tx thread. @@ -773,13 +774,40 @@ relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row) relay_send(relay, row); } +static void +relay_send_raft(struct relay *relay, struct raft_request *req) +{ + struct xrow_header packet; + xrow_encode_raft(&packet, &fiber()->gc, req); + relay_send(relay, &packet); +} + +static void +relay_send_raft_msg(struct cmsg *msg) +{ + struct raft_broadcast_msg *raft_msg = (struct raft_broadcast_msg *)msg; + struct relay *relay = container_of(msg->route[0].pipe, struct relay, + tx_pipe); + relay_send_raft(relay, &raft_msg->req); +} + +void +relay_push_raft_msg(struct relay *relay, struct cmsg *msg, + struct cmsg_hop *route) +{ + route[0].f = relay_send_raft_msg; + route[0].pipe = &relay->tx_pipe; + route[1].f = raft_free_msg; + route[1].pipe = NULL; + cmsg_init(msg, route); + cpipe_push(&relay->relay_pipe, msg); +} + /** Send a single row to the client. */ static void relay_send_row(struct xstream *stream, struct xrow_header *packet) { struct relay *relay = container_of(stream, struct relay, stream); - assert(iproto_type_is_dml(packet->type) || - iproto_type_is_synchro_request(packet->type)); if (packet->group_id == GROUP_LOCAL) { /* * We do not relay replica-local rows to other @@ -796,6 +824,8 @@ relay_send_row(struct xstream *stream, struct xrow_header *packet) packet->group_id = GROUP_DEFAULT; packet->bodycnt = 0; } + assert(iproto_type_is_dml(packet->type) || + iproto_type_is_synchro_request(packet->type)); /* Check if the rows from the instance are filtered. */ if ((1 << packet->replica_id & relay->id_filter) != 0) return; diff --git a/src/box/relay.h b/src/box/relay.h index 0632fa912..c2c30cd11 100644 --- a/src/box/relay.h +++ b/src/box/relay.h @@ -41,6 +41,8 @@ struct relay; struct replica; struct tt_uuid; struct vclock; +struct cmsg; +struct cmsg_hop; enum relay_state { /** @@ -93,6 +95,18 @@ relay_vclock(const struct relay *relay); double relay_last_row_time(const struct relay *relay); +/** + * Initialize a raft status message with the route to relay and + * back and push the message to relay. + * + * @param relay relay. + * @param msg a preallocated status message. + * @param route a preallocated message route. + */ +void +relay_push_raft_msg(struct relay *relay, struct cmsg *msg, + struct cmsg_hop *route); + #if defined(__cplusplus) } /* extern "C" */ #endif /* defined(__cplusplus) */ diff --git a/src/box/xrow.c b/src/box/xrow.c index 1923bacfc..f60b12cfc 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -44,6 +44,7 @@ #include "scramble.h" #include "iproto_constants.h" #include "mpstream/mpstream.h" +#include "raft.h" static_assert(IPROTO_DATA < 0x7f && IPROTO_METADATA < 0x7f && IPROTO_SQL_INFO < 0x7f, "encoded IPROTO_BODY keys must fit into "\ @@ -958,11 +959,26 @@ int xrow_encode_raft(struct xrow_header *row, struct region *region, const struct raft_request *r) { + assert(mp_sizeof_map(2) == mp_sizeof_map(4)); + /* + * Term and vote are encoded every time for the sake of + * snapshot, while state and vclock are optional. + */ size_t size = mp_sizeof_map(2) + mp_sizeof_uint(IPROTO_RAFT_TERM) + mp_sizeof_uint(r->term) + mp_sizeof_uint(IPROTO_RAFT_VOTE) + mp_sizeof_uint(r->vote); + + size += (r->state != 0) * (mp_sizeof_uint(IPROTO_RAFT_STATE) + + mp_sizeof_uint(r->state)); + if (r->vclock != NULL) { + size += mp_sizeof_uint(IPROTO_VCLOCK) + + mp_sizeof_vclock_ignore0(r->vclock); + } + + int map_size = 2 + (r->state != 0) + (r->vclock != NULL); + char *buf = region_alloc(region, size); if (buf == NULL) { diag_set(OutOfMemory, size, "region_alloc", "buf"); @@ -974,11 +990,20 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, row->body[0].iov_len = size; row->group_id = GROUP_LOCAL; row->bodycnt = 1; - buf = mp_encode_map(buf, 2); + buf = mp_encode_map(buf, map_size); buf = mp_encode_uint(buf, IPROTO_RAFT_TERM); buf = mp_encode_uint(buf, r->term); buf = mp_encode_uint(buf, IPROTO_RAFT_VOTE); buf = mp_encode_uint(buf, r->vote); + if (r->state != 0) { + buf = mp_encode_uint(buf, IPROTO_RAFT_STATE); + buf = mp_encode_uint(buf, r->state); + } + if (r->vclock != NULL) { + buf = mp_encode_uint(buf, IPROTO_VCLOCK); + buf = mp_encode_vclock_ignore0(buf, r->vclock); + } + return 0; } @@ -989,7 +1014,7 @@ xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) assert(row->type == IPROTO_RAFT); assert(row->bodycnt == 1); assert(row->group_id == GROUP_LOCAL); - memset(r, 0, sizeof(*r)); + memset(r, 0, sizeof(*r) - sizeof(struct vclock *)); const char *pos = row->body[0].iov_base; uint32_t map_size = mp_decode_map(&pos); for (uint32_t i = 0; i < map_size; ++i) @@ -1002,6 +1027,15 @@ xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) case IPROTO_RAFT_VOTE: r->vote = mp_decode_uint(&pos); break; + case IPROTO_RAFT_STATE: + r->state = mp_decode_uint(&pos); + break; + case IPROTO_VCLOCK: + if (r->vclock != NULL) + mp_decode_vclock_ignore0(&pos, r->vclock); + else + mp_next(&pos); + break; default: mp_next(&pos); break; diff --git a/src/box/xrow.h b/src/box/xrow.h index c234f6f88..3f37dc18f 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -264,10 +264,7 @@ xrow_encode_synchro(struct xrow_header *row, int xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); -struct raft_request { - uint64_t term; - uint32_t vote; -}; +struct raft_request; int xrow_encode_raft(struct xrow_header *row, struct region *region, -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:35 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:35 +0300 Subject: [Tarantool-patches] [RAFT 03/10] [tosquash] raft: return raft_request to xrow In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy Xrow is the owner of all the IProto and xlog codecs. So the raft reaquest's definition belongs here, just like any other request. --- src/box/raft.c | 1 - src/box/raft.h | 9 +-------- src/box/xrow.c | 1 - src/box/xrow.h | 7 ++++++- 4 files changed, 7 insertions(+), 11 deletions(-) diff --git a/src/box/raft.c b/src/box/raft.c index 839a7dfeb..227846596 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -32,7 +32,6 @@ #include "error.h" #include "journal.h" -#include "xrow.h" #include "small/region.h" #include "replication.h" #include "relay.h" diff --git a/src/box/raft.h b/src/box/raft.h index 9cb39dd24..b11ae7b1d 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -29,7 +29,7 @@ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ -#include +#include "xrow.h" #include "cbus.h" #if defined(__cplusplus) @@ -52,13 +52,6 @@ struct raft { extern struct raft raft; -struct raft_request { - uint64_t term; - uint32_t vote; - enum raft_state state; - struct vclock *vclock; -}; - struct raft_broadcast_msg { struct cmsg base; struct raft_request req; diff --git a/src/box/xrow.c b/src/box/xrow.c index f60b12cfc..ed3f77a15 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -44,7 +44,6 @@ #include "scramble.h" #include "iproto_constants.h" #include "mpstream/mpstream.h" -#include "raft.h" static_assert(IPROTO_DATA < 0x7f && IPROTO_METADATA < 0x7f && IPROTO_SQL_INFO < 0x7f, "encoded IPROTO_BODY keys must fit into "\ diff --git a/src/box/xrow.h b/src/box/xrow.h index 3f37dc18f..5d571a821 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -264,7 +264,12 @@ xrow_encode_synchro(struct xrow_header *row, int xrow_decode_synchro(const struct xrow_header *row, struct synchro_request *req); -struct raft_request; +struct raft_request { + uint64_t term; + uint32_t vote; + uint32_t state; + struct vclock *vclock; +}; int xrow_encode_raft(struct xrow_header *row, struct region *region, -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:36 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:36 +0300 Subject: [Tarantool-patches] [RAFT 04/10] [tosquash] raft: introduce IPROTO_RAFT_VCLOCK In-Reply-To: References: Message-ID: <4fb599846ea343540312cd833f4dfe2d00771eb2.1598427905.git.sergepetrenko@tarantool.org> From: Vladislav Shpilevoy IPROTO_RAFT keys are stored in their own isolated dictionary, so no need to reuse other keys. Better implement the raft's own keyset with IPROTO_RAFT_* keys. --- src/box/iproto_constants.h | 1 + src/box/xrow.c | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h index 4217ce2e0..3ec397d3c 100644 --- a/src/box/iproto_constants.h +++ b/src/box/iproto_constants.h @@ -264,6 +264,7 @@ enum iproto_raft_keys { IPROTO_RAFT_TERM = 0, IPROTO_RAFT_VOTE = 1, IPROTO_RAFT_STATE = 2, + IPROTO_RAFT_VCLOCK = 3, }; /** diff --git a/src/box/xrow.c b/src/box/xrow.c index ed3f77a15..836de3575 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -972,7 +972,7 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, size += (r->state != 0) * (mp_sizeof_uint(IPROTO_RAFT_STATE) + mp_sizeof_uint(r->state)); if (r->vclock != NULL) { - size += mp_sizeof_uint(IPROTO_VCLOCK) + + size += mp_sizeof_uint(IPROTO_RAFT_VCLOCK) + mp_sizeof_vclock_ignore0(r->vclock); } @@ -999,7 +999,7 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, buf = mp_encode_uint(buf, r->state); } if (r->vclock != NULL) { - buf = mp_encode_uint(buf, IPROTO_VCLOCK); + buf = mp_encode_uint(buf, IPROTO_RAFT_VCLOCK); buf = mp_encode_vclock_ignore0(buf, r->vclock); } @@ -1029,7 +1029,7 @@ xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) case IPROTO_RAFT_STATE: r->state = mp_decode_uint(&pos); break; - case IPROTO_VCLOCK: + case IPROTO_RAFT_VCLOCK: if (r->vclock != NULL) mp_decode_vclock_ignore0(&pos, r->vclock); else -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:37 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:37 +0300 Subject: [Tarantool-patches] [RAFT 05/10] [tosquash] xrow: refactor raft request codec In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy There were a few issues: - In decode() memset was used without raft_request tail assuming that vclock storage pointer is always in the end. Better not to assume such things. The patch makes the decode() caller to pass vclock storage explicitly if necessary. - Little improvement over the readibility of the encoder to make more 'patterned'. Consisting of a sequence of similar code blocks of kind: if (is_field_set) { ++map_size; size += mp_sizeof_uint(field_key) + mp_sizeof_...(field_value); } ... if (is_field_set) { pos = mp_encode_uint(pos, field_key); pos = mp_encode_...(pos, field_value); } Instead of unique handling of certain fields. Also the vote is now not encoded into each message, because no need in that. - Added malformed packet handling. Note, that we don't consider the invalid MessagePack case. But probably should do it eventually. Other requests don't handle it, because they ar checked either by iproto thread or by relay thread. Need to see if raft message are also already validated by some mp_check() in relay. --- src/box/applier.cc | 4 +-- src/box/box.cc | 3 +- src/box/memtx_engine.c | 3 +- src/box/xrow.c | 76 +++++++++++++++++++++++++++--------------- src/box/xrow.h | 3 +- 5 files changed, 57 insertions(+), 32 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index f27436b79..8e6d1b2a4 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -886,9 +886,7 @@ apply_raft_row(struct xrow_header *row) struct raft_request req; struct vclock candidate_clock; - req.vclock = &candidate_clock; - - if (xrow_decode_raft(row, &req) != 0) + if (xrow_decode_raft(row, &req, &candidate_clock) != 0) return -1; raft_process_msg(&req); diff --git a/src/box/box.cc b/src/box/box.cc index 8323de531..b871f45e2 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -377,7 +377,8 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) } if (iproto_type_is_raft_request(row->type)) { struct raft_request raft_req; - if (xrow_decode_raft(row, &raft_req) != 0) + /* Vclock is never persisted in WAL by Raft. */ + if (xrow_decode_raft(row, &raft_req, NULL) != 0) diag_raise(); raft_process(&raft_req); return; diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index 26274de80..a034baa6c 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -206,7 +206,8 @@ memtx_engine_recover_raft(const struct xrow_header *row) { assert(row->type == IPROTO_RAFT); struct raft_request req; - if (xrow_decode_raft(row, &req) != 0) + /* Vclock is never persisted in WAL by Raft. */ + if (xrow_decode_raft(row, &req, NULL) != 0) return -1; raft_process(&req); return 0; diff --git a/src/box/xrow.c b/src/box/xrow.c index 836de3575..11fdacc0d 100644 --- a/src/box/xrow.c +++ b/src/box/xrow.c @@ -958,25 +958,29 @@ int xrow_encode_raft(struct xrow_header *row, struct region *region, const struct raft_request *r) { - assert(mp_sizeof_map(2) == mp_sizeof_map(4)); /* - * Term and vote are encoded every time for the sake of - * snapshot, while state and vclock are optional. + * Terms is encoded always. Sometimes the rest can be even ignored if + * the term is too old. */ - size_t size = mp_sizeof_map(2) + - mp_sizeof_uint(IPROTO_RAFT_TERM) + - mp_sizeof_uint(r->term) + - mp_sizeof_uint(IPROTO_RAFT_VOTE) + - mp_sizeof_uint(r->vote); - - size += (r->state != 0) * (mp_sizeof_uint(IPROTO_RAFT_STATE) + - mp_sizeof_uint(r->state)); + int map_size = 1; + size_t size = mp_sizeof_uint(IPROTO_RAFT_TERM) + + mp_sizeof_uint(r->term); + if (r->vote != 0) { + ++map_size; + size += mp_sizeof_uint(IPROTO_RAFT_VOTE) + + mp_sizeof_uint(r->vote); + } + if (r->state != 0) { + ++map_size; + size += mp_sizeof_uint(IPROTO_RAFT_STATE) + + mp_sizeof_uint(r->state); + } if (r->vclock != NULL) { + ++map_size; size += mp_sizeof_uint(IPROTO_RAFT_VCLOCK) + - mp_sizeof_vclock_ignore0(r->vclock); + mp_sizeof_vclock_ignore0(r->vclock); } - - int map_size = 2 + (r->state != 0) + (r->vclock != NULL); + size += mp_sizeof_map(map_size); char *buf = region_alloc(region, size); if (buf == NULL) { @@ -992,8 +996,10 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, buf = mp_encode_map(buf, map_size); buf = mp_encode_uint(buf, IPROTO_RAFT_TERM); buf = mp_encode_uint(buf, r->term); - buf = mp_encode_uint(buf, IPROTO_RAFT_VOTE); - buf = mp_encode_uint(buf, r->vote); + if (r->vote != 0) { + buf = mp_encode_uint(buf, IPROTO_RAFT_VOTE); + buf = mp_encode_uint(buf, r->vote); + } if (r->state != 0) { buf = mp_encode_uint(buf, IPROTO_RAFT_STATE); buf = mp_encode_uint(buf, r->state); @@ -1002,38 +1008,52 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, buf = mp_encode_uint(buf, IPROTO_RAFT_VCLOCK); buf = mp_encode_vclock_ignore0(buf, r->vclock); } - return 0; } int -xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) +xrow_decode_raft(const struct xrow_header *row, struct raft_request *r, + struct vclock *vclock) { - /* TODO: handle bad format. */ assert(row->type == IPROTO_RAFT); - assert(row->bodycnt == 1); - assert(row->group_id == GROUP_LOCAL); - memset(r, 0, sizeof(*r) - sizeof(struct vclock *)); - const char *pos = row->body[0].iov_base; + if (row->bodycnt != 1 || row->group_id != GROUP_LOCAL) { + diag_set(ClientError, ER_INVALID_MSGPACK, + "malformed raft request"); + return -1; + } + memset(r, 0, sizeof(*r)); + r->vclock = vclock; + + const char *begin = row->body[0].iov_base; + const char *end = begin + row->body[0].iov_len; + const char *pos = begin; uint32_t map_size = mp_decode_map(&pos); for (uint32_t i = 0; i < map_size; ++i) { + if (mp_typeof(*pos) != MP_UINT) + goto bad_msgpack; uint64_t key = mp_decode_uint(&pos); switch (key) { case IPROTO_RAFT_TERM: + if (mp_typeof(*pos) != MP_UINT) + goto bad_msgpack; r->term = mp_decode_uint(&pos); break; case IPROTO_RAFT_VOTE: + if (mp_typeof(*pos) != MP_UINT) + goto bad_msgpack; r->vote = mp_decode_uint(&pos); break; case IPROTO_RAFT_STATE: + if (mp_typeof(*pos) != MP_UINT) + goto bad_msgpack; r->state = mp_decode_uint(&pos); break; case IPROTO_RAFT_VCLOCK: - if (r->vclock != NULL) - mp_decode_vclock_ignore0(&pos, r->vclock); - else + if (r->vclock == NULL) mp_next(&pos); + else if (mp_decode_vclock_ignore0(&pos, r->vclock) != 0) + goto bad_msgpack; break; default: mp_next(&pos); @@ -1041,6 +1061,10 @@ xrow_decode_raft(const struct xrow_header *row, struct raft_request *r) } } return 0; + +bad_msgpack: + xrow_on_decode_err(begin, end, ER_INVALID_MSGPACK, "raft body"); + return -1; } int diff --git a/src/box/xrow.h b/src/box/xrow.h index 5d571a821..c627102dd 100644 --- a/src/box/xrow.h +++ b/src/box/xrow.h @@ -276,7 +276,8 @@ xrow_encode_raft(struct xrow_header *row, struct region *region, const struct raft_request *r); int -xrow_decode_raft(const struct xrow_header *row, struct raft_request *r); +xrow_decode_raft(const struct xrow_header *row, struct raft_request *r, + struct vclock *vclock); /** * CALL/EVAL request. -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:38 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:38 +0300 Subject: [Tarantool-patches] [RAFT 06/10] [tosquash] raft: don't fill raft_request manually In-Reply-To: References: Message-ID: <1fb0f4b22ccfea68e73f040f5404eebc2fe0154d.1598427905.git.sergepetrenko@tarantool.org> From: Vladislav Shpilevoy At 'subscribe' the raft complete state was sent to the peer. But it was filled manually into struct raft_request. There is a function raft_serialize() exactly to avoid such manual work. The patch extends the serializer with vclock argument. Since raft does not manage vclocks and needs them provided externally (so far, perhaps that will change). --- src/box/box.cc | 5 +---- src/box/memtx_engine.c | 6 +++++- src/box/raft.c | 7 ++++++- src/box/raft.h | 2 +- 4 files changed, 13 insertions(+), 7 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index b871f45e2..e7eb79e9f 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -2056,14 +2056,11 @@ box_process_subscribe(struct ev_io *io, struct xrow_header *header) */ if (raft.state != RAFT_STATE_NONE) { struct raft_request req; - req.term = raft.term; - req.vote = raft.vote; - req.state = raft.state; /* * Omit the candidate vclock, since we've just * sent it in subscribe response. */ - req.vclock = NULL; + raft_serialize(&req, NULL); xrow_encode_raft(&row, &fiber()->gc, &req); coio_write_xrow(io, &row); } diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index a034baa6c..7de12a569 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -517,7 +517,11 @@ checkpoint_new(const char *snap_dirname, uint64_t snap_io_rate_limit) opts.free_cache = true; xdir_create(&ckpt->dir, snap_dirname, SNAP, &INSTANCE_UUID, &opts); vclock_create(&ckpt->vclock); - raft_serialize(&ckpt->raft); + /* + * Don't encode vclock, because it is stored in the snapshot header + * anyway. + */ + raft_serialize(&ckpt->raft, NULL); ckpt->touch = false; return ckpt; } diff --git a/src/box/raft.c b/src/box/raft.c index 227846596..1d25459e9 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -80,11 +80,16 @@ raft_process_msg(const struct raft_request *req) } void -raft_serialize(struct raft_request *req) +raft_serialize(struct raft_request *req, struct vclock *vclock) { memset(req, 0, sizeof(*req)); req->term = raft.term; req->vote = raft.vote; + req->state = raft.state; + /* + * Raft does not own vclock, so it always expects it passed externally. + */ + req->vclock = vclock; } static void diff --git a/src/box/raft.h b/src/box/raft.h index b11ae7b1d..c95a51873 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -72,7 +72,7 @@ void raft_process_msg(const struct raft_request *req); void -raft_serialize(struct raft_request *req); +raft_serialize(struct raft_request *req, struct vclock *vclock); void raft_free_msg(struct cmsg *msg); -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:39 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:39 +0300 Subject: [Tarantool-patches] [RAFT 07/10] [tosquash] raft: rename curr_leader to leader In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy Because it will be consistent with 'vote', which is not 'curr_vote'. --- src/box/raft.c | 2 +- src/box/raft.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/box/raft.c b/src/box/raft.c index 1d25459e9..714a1518a 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -40,7 +40,7 @@ struct raft raft = { .term = 0, .vote = 0, - .curr_leader = 0, + .leader = 0, .state = RAFT_STATE_NONE, }; diff --git a/src/box/raft.h b/src/box/raft.h index c95a51873..927aa8f5f 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -46,7 +46,7 @@ enum raft_state { struct raft { uint64_t term; uint32_t vote; - uint32_t curr_leader; + uint32_t leader; enum raft_state state; }; -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:40 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:40 +0300 Subject: [Tarantool-patches] [RAFT 08/10] [tosquash] raft: rename raft_process to raft_process_recovery In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy There is another 'process' for remote messages from other raft nodes. In order to make these functions more clearly separated, the old function used for local state recovery is renamed to raft_process_recovery. This should be squashed into the first commit about persistent raft state. --- src/box/box.cc | 2 +- src/box/memtx_engine.c | 2 +- src/box/raft.c | 14 +++++++++++++- src/box/raft.h | 2 +- 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index e7eb79e9f..d01de2519 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -380,7 +380,7 @@ apply_wal_row(struct xstream *stream, struct xrow_header *row) /* Vclock is never persisted in WAL by Raft. */ if (xrow_decode_raft(row, &raft_req, NULL) != 0) diag_raise(); - raft_process(&raft_req); + raft_process_recovery(&raft_req); return; } xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type)); diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index 7de12a569..b0b744db8 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -209,7 +209,7 @@ memtx_engine_recover_raft(const struct xrow_header *row) /* Vclock is never persisted in WAL by Raft. */ if (xrow_decode_raft(row, &req, NULL) != 0) return -1; - raft_process(&req); + raft_process_recovery(&req); return 0; } diff --git a/src/box/raft.c b/src/box/raft.c index 714a1518a..34c1cf7aa 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -45,12 +45,24 @@ struct raft raft = { }; void -raft_process(const struct raft_request *req) +raft_process_recovery(const struct raft_request *req) { if (req->term != 0) raft.term = req->term; if (req->vote != 0) raft.vote = req->vote; + /* + * Role is never persisted. If recovery is happening, the + * node was restarted, and the former role can be false + * anyway. + */ + assert(req->state == RAFT_STATE_NONE); + /* + * Vclock is always persisted by some other subsystem - WAL, snapshot. + * It is used only to decide to whom to give the vote during election, + * as a part of the volatile state. + */ + assert(req->vclock == NULL); } void diff --git a/src/box/raft.h b/src/box/raft.h index 927aa8f5f..be071c215 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -65,7 +65,7 @@ raft_vote(uint32_t vote_for); /** Process a raft entry stored in WAL/snapshot. */ void -raft_process(const struct raft_request *req); +raft_process_recovery(const struct raft_request *req); /** Process a raft status message coming from the network. */ void -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:52:41 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:52:41 +0300 Subject: [Tarantool-patches] [RAFT 09/10] [tosquash] applier: handler error at raft row appliance In-Reply-To: References: Message-ID: <180461b1efe572162acb9b50e485684dc8d74ff2.1598427905.git.sergepetrenko@tarantool.org> From: Vladislav Shpilevoy --- src/box/applier.cc | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/box/applier.cc b/src/box/applier.cc index 8e6d1b2a4..b17ac5363 100644 --- a/src/box/applier.cc +++ b/src/box/applier.cc @@ -1241,10 +1241,13 @@ applier_subscribe(struct applier *applier) &stailq_first_entry(&rows, struct applier_tx_row, next)->row; if (first_row->lsn == 0) { - if (unlikely(iproto_type_is_raft_request(first_row->type))) - apply_raft_row(first_row); - else + if (unlikely(iproto_type_is_raft_request( + first_row->type))) { + if (apply_raft_row(first_row) != 0) + diag_raise(); + } else { applier_signal_ack(applier); + } } else if (applier_apply_tx(&rows) != 0) { diag_raise(); } -- 2.20.1 (Apple Git-117) From sergepetrenko at tarantool.org Wed Aug 26 10:53:29 2020 From: sergepetrenko at tarantool.org (Serge Petrenko) Date: Wed, 26 Aug 2020 10:53:29 +0300 Subject: [Tarantool-patches] [RAFT 10/10] [tosquash] relay: move raft broadcast details into relay In-Reply-To: References: Message-ID: From: Vladislav Shpilevoy Raft did some allocations, cbus and cmsg initializations, in order to broadcast its state update. Also it didn't copy vclock value so it could point at invalid memory. The patch moves all the details about pushing a raft update into relay.cc file, and fixes the vclock copying. --- src/box/raft.c | 15 +---------- src/box/raft.h | 8 ------ src/box/relay.cc | 68 ++++++++++++++++++++++++++++++++++-------------- src/box/relay.h | 11 +++----- 4 files changed, 52 insertions(+), 50 deletions(-) diff --git a/src/box/raft.c b/src/box/raft.c index 34c1cf7aa..e40d778af 100644 --- a/src/box/raft.c +++ b/src/box/raft.c @@ -169,26 +169,13 @@ raft_vote(uint32_t vote_for) raft_write_request(&req); } -void -raft_free_msg(struct cmsg *msg) -{ - free((void *)msg->route); - free(msg); -} - void raft_broadcast(const struct raft_request *req) { replicaset_foreach(replica) { if (replica->relay != NULL && replica->id != REPLICA_ID_NIL && relay_get_state(replica->relay) == RELAY_FOLLOW) { - // TODO: think of a proper allocator. - struct raft_broadcast_msg *raft_msg = - calloc(1, sizeof(*raft_msg)); - raft_msg->req = *req; - struct cmsg_hop *route = calloc(2, sizeof(*route)); - relay_push_raft_msg(replica->relay, &raft_msg->base, - route); + relay_push_raft(replica->relay, req); } } } diff --git a/src/box/raft.h b/src/box/raft.h index be071c215..e14173057 100644 --- a/src/box/raft.h +++ b/src/box/raft.h @@ -52,11 +52,6 @@ struct raft { extern struct raft raft; -struct raft_broadcast_msg { - struct cmsg base; - struct raft_request req; -}; - void raft_new_term(uint64_t min_new_term); @@ -74,9 +69,6 @@ raft_process_msg(const struct raft_request *req); void raft_serialize(struct raft_request *req, struct vclock *vclock); -void -raft_free_msg(struct cmsg *msg); - /** * Broadcast the changes in this instance's raft status to all * the followers. diff --git a/src/box/relay.cc b/src/box/relay.cc index be252cad1..53a90f826 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -774,33 +774,61 @@ relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row) relay_send(relay, row); } -static void -relay_send_raft(struct relay *relay, struct raft_request *req) -{ - struct xrow_header packet; - xrow_encode_raft(&packet, &fiber()->gc, req); - relay_send(relay, &packet); -} +struct relay_raft_msg { + struct cmsg base; + struct cmsg_hop route; + struct raft_request req; + struct vclock vclock; + struct relay *relay; +}; static void -relay_send_raft_msg(struct cmsg *msg) +relay_raft_msg_send(struct cmsg *base) { - struct raft_broadcast_msg *raft_msg = (struct raft_broadcast_msg *)msg; - struct relay *relay = container_of(msg->route[0].pipe, struct relay, - tx_pipe); - relay_send_raft(relay, &raft_msg->req); + struct relay_raft_msg *msg = (struct relay_raft_msg *)base; + struct xrow_header row; + xrow_encode_raft(&row, &fiber()->gc, &msg->req); + try { + relay_send(msg->relay, &row); + } catch (Exception *e) { + relay_set_error(msg->relay, e); + fiber_cancel(fiber()); + } + free(msg); } void -relay_push_raft_msg(struct relay *relay, struct cmsg *msg, - struct cmsg_hop *route) +relay_push_raft(struct relay *relay, const struct raft_request *req) { - route[0].f = relay_send_raft_msg; - route[0].pipe = &relay->tx_pipe; - route[1].f = raft_free_msg; - route[1].pipe = NULL; - cmsg_init(msg, route); - cpipe_push(&relay->relay_pipe, msg); + /* + * XXX: the message should be preallocated. It should + * work like Kharon in IProto. Relay should have 2 raft + * messages rotating. When one is sent, the other can be + * updated and a flag is set. When the first message is + * sent, the control returns to TX thread, sees the set + * flag, rotates the buffers, and sends it again. And so + * on. This is how it can work in future, with 0 heap + * allocations. Current solution with alloc-per-update is + * good enough as a start. Another option - wait until all + * is moved to WAL thread, where this will all happen + * in one thread and will be much simpler. + */ + struct relay_raft_msg *msg = + (struct relay_raft_msg *)malloc(sizeof(*msg)); + if (msg == NULL) { + panic("Couldn't allocate raft message"); + return; + } + msg->req = *req; + if (req->vclock != NULL) { + msg->req.vclock = &msg->vclock; + vclock_copy(&msg->vclock, req->vclock); + } + msg->route.f = relay_raft_msg_send; + msg->route.pipe = NULL; + cmsg_init(&msg->base, &msg->route); + msg->relay = relay; + cpipe_push(&relay->relay_pipe, &msg->base); } /** Send a single row to the client. */ diff --git a/src/box/relay.h b/src/box/relay.h index c2c30cd11..4d291698d 100644 --- a/src/box/relay.h +++ b/src/box/relay.h @@ -96,16 +96,11 @@ double relay_last_row_time(const struct relay *relay); /** - * Initialize a raft status message with the route to relay and - * back and push the message to relay. - * - * @param relay relay. - * @param msg a preallocated status message. - * @param route a preallocated message route. + * Send a Raft update request to the relay channel. It is not + * guaranteed that it will be delivered. The connection may break. */ void -relay_push_raft_msg(struct relay *relay, struct cmsg *msg, - struct cmsg_hop *route); +relay_push_raft(struct relay *relay, const struct raft_request *req); #if defined(__cplusplus) } /* extern "C" */ -- 2.20.1 (Apple Git-117) From avtikhon at tarantool.org Wed Aug 26 17:24:35 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Wed, 26 Aug 2020 17:24:35 +0300 Subject: [Tarantool-patches] [PATCH v1] test: fix flaky box/on_shutdown.test.lua on asan Message-ID: Found that box/on_shutdown.test.lua test fails on asan build with: 2020-08-26 09:04:06.750 [42629] main/102/on_shutdown [string "_ = box.ctl.on_shutdown(function() log.warn("..."]:1 W> on_shutdown 5 Starting instance proxy... Run console at unix/:/tnt/test/var/001_box/proxy.control Start failed: builtin/box/console.lua:865: failed to create server unix/:/tnt/test/var/001_box/proxy.control: Address already in use It happened on ASAN build, because server stop routine test-run/lib/preprocessor.py:TestState.server_stop() -> test-run/lib/tarantool_server.py:TarantoolServer.stop() needs some delay to free the proxy.control socket created by test-run/lib/preprocessor.py:TestState.server_start() -> tarantoolctl:process_local() To fix the issue added fiber.sleep() to give the needed delay. Closes #5260 Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5260-on-shutdown-test Issue: https://github.com/tarantool/tarantool/issues/5260 Issue: https://github.com/tarantool/tarantool/issues/4360 test/box/on_shutdown.result | 5 +++++ test/box/on_shutdown.skipcond | 7 ------- test/box/on_shutdown.test.lua | 3 +++ 3 files changed, 8 insertions(+), 7 deletions(-) delete mode 100644 test/box/on_shutdown.skipcond diff --git a/test/box/on_shutdown.result b/test/box/on_shutdown.result index ccbdf45cb..ea3f8ca4a 100644 --- a/test/box/on_shutdown.result +++ b/test/box/on_shutdown.result @@ -142,6 +142,11 @@ test_run:cmd("stop server test") --- - true ... +-- On ASAN build server stop needs some delay to free the proxy.control +-- socket created by tarantoolctl:process_local(), check gh-5260. +fiber.sleep(0.1) +--- +... test_run:cmd("start server test") --- - true diff --git a/test/box/on_shutdown.skipcond b/test/box/on_shutdown.skipcond deleted file mode 100644 index e46fd1088..000000000 --- a/test/box/on_shutdown.skipcond +++ /dev/null @@ -1,7 +0,0 @@ -import os - -# Disabled at ASAN build due to issue #4360. -if os.getenv("ASAN") == 'ON': - self.skip = 1 - -# vim: set ft=python: diff --git a/test/box/on_shutdown.test.lua b/test/box/on_shutdown.test.lua index 2a9143404..91ff36f55 100644 --- a/test/box/on_shutdown.test.lua +++ b/test/box/on_shutdown.test.lua @@ -58,6 +58,9 @@ fiber.sleep(0.1) -- The server should be already stopped by os.exit(), -- but start doesn't work without a prior call to stop. test_run:cmd("stop server test") +-- On ASAN build server stop needs some delay to free the proxy.control +-- socket created by tarantoolctl:process_local(), check gh-5260. +fiber.sleep(0.1) test_run:cmd("start server test") test_run:wait_log('test', 'on_shutdown 5', nil, 30, {noreset=true}) -- make sure we exited because of os.exit(), not a signal. -- 2.17.1 From sergeyb at tarantool.org Wed Aug 26 17:45:38 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Wed, 26 Aug 2020 17:45:38 +0300 Subject: [Tarantool-patches] [PATCH 3/3] replication: add test with random leaders promotion and demotion In-Reply-To: <3c7b2274-8443-2be7-c181-2c7026ab0fec@tarantool.org> References: <3c7b2274-8443-2be7-c181-2c7026ab0fec@tarantool.org> Message-ID: <20200826144538.GC47610@pony.bronevichok.ru> Vlad, thanks for review! Patch updated in a branch. To make sure patch doesn't make test flaky I run test 100 times using 10 workers in parallel without problems. ../../test/test-run.py --builddir=/home/s.bronnikov/tarantool/build --vardir=/home/s.bronnikov/tarantool /build/test/var -j 10 $(yes replication/qsync_random_leader.test.lua | head -n 100) On 00:01 Tue 21 Jul , Vladislav Shpilevoy wrote: > Thanks for the patch! > > See 11 comments below. > > > diff --git a/test/replication/qsync.lua b/test/replication/qsync.lua > > new file mode 100644 > > index 000000000..383aa5272 > > --- /dev/null > > +++ b/test/replication/qsync.lua > > @@ -0,0 +1,62 @@ > > +#!/usr/bin/env tarantool > > + > > +-- get instance name from filename (qsync1.lua => qsync1) > > +local INSTANCE_ID = string.match(arg[0], "%d") > > + > > +local SOCKET_DIR = require('fio').cwd() > > + > > +local TIMEOUT = tonumber(arg[1]) > > + > > +local function instance_uri(instance_id) > > + return SOCKET_DIR..'/qsync'..instance_id..'.sock'; > > +end > > + > > +-- start console first > > +require('console').listen(os.getenv('ADMIN')) > > + > > +box.cfg({ > > + listen = instance_uri(INSTANCE_ID); > > + replication_timeout = TIMEOUT; > > 1. Why do you need the custom replication_timeout? It is actually a copy-paste from original cluster initialization script, removed replication_timeout. > > + replication_sync_lag = 0.01; > > 2. Why do you need the lag setting? the same as above > > + replication_connect_quorum = 3; > > + replication = { > > + instance_uri(1); > > + instance_uri(2); > > + instance_uri(3); > > + instance_uri(4); > > + instance_uri(5); > > + instance_uri(6); > > + instance_uri(7); > > + instance_uri(8); > > + instance_uri(9); > > + instance_uri(10); > > + instance_uri(11); > > + instance_uri(12); > > + instance_uri(13); > > + instance_uri(14); > > + instance_uri(15); > > + instance_uri(16); > > + instance_uri(17); > > + instance_uri(18); > > + instance_uri(19); > > + instance_uri(20); > > + instance_uri(21); > > + instance_uri(22); > > + instance_uri(23); > > + instance_uri(24); > > + instance_uri(25); > > + instance_uri(26); > > + instance_uri(27); > > + instance_uri(28); > > + instance_uri(29); > > + instance_uri(30); > > + instance_uri(31); > > 3. Seems like in the test you use only 3 instances, not 32. Also the > quorum is set to 3. in updated test 5 instances are in use, others removed in initialization script > > + }; > > +}) > > + > > +box.once("bootstrap", function() > > + local test_run = require('test_run').new() > > + box.schema.user.grant("guest", 'replication') > > + box.schema.space.create('test', {engine = test_run:get_cfg('engine')}) > > + box.space.test:create_index('primary') > > 4. Where do you use this space? space has been renamed to "sync" and used it in a test > > +end) > > diff --git a/test/replication/qsync_random_leader.result b/test/replication/qsync_random_leader.result > > new file mode 100644 > > index 000000000..cb1b5e232 > > --- /dev/null > > +++ b/test/replication/qsync_random_leader.result > > @@ -0,0 +1,123 @@ > > +-- test-run result file version 2 > > +os = require('os') > > + | --- > > + | ... > > +env = require('test_run') > > + | --- > > + | ... > > +math = require('math') > > + | --- > > + | ... > > +fiber = require('fiber') > > + | --- > > + | ... > > +test_run = env.new() > > + | --- > > + | ... > > +engine = test_run:get_cfg('engine') > > + | --- > > + | ... > > + > > +NUM_INSTANCES = 3 > > + | --- > > + | ... > > +BROKEN_QUORUM = NUM_INSTANCES + 1 > > + | --- > > + | ... > > + > > +SERVERS = {} > > + | --- > > + | ... > > +test_run:cmd("setopt delimiter ';'") > > + | --- > > + | - true > > + | ... > > +for i=1,NUM_INSTANCES do > > + SERVERS[i] = 'qsync' .. i > > +end; > > + | --- > > + | ... > > +test_run:cmd("setopt delimiter ''"); > > 5. Please, lets be consistent and use either \ or the delimiter. Currently > it is irrational - you use \ for big code blocks, and a custom delimiter for > tiny blocks which could even be one line. Personally, I would use \ > everywhere. replaced constructions with delimiters to multiline statements > > + | --- > > + | - true > > + | ... > > +SERVERS -- print instance names > > + | --- > > + | - - qsync1 > > + | - qsync2 > > + | - qsync3 > > + | ... > > + > > +random = function(excluded_num, min, max) \ > > 6. Would be better to align all \ by 80 in this file. Makes easier to add > new longer lines in future without moving all the old \. Done. > > + math.randomseed(os.time()) \ > > + local r = math.random(min, max) \ > > + if (r == excluded_num) then \ > > + return random(excluded_num, min, max) \ > > + end \ > > + return r \ > > +end > > + | --- > > + | ... > > + > > +test_run:create_cluster(SERVERS, "replication", {args="0.1"}) > > + | --- > > + | ... > > +test_run:wait_fullmesh(SERVERS) > > + | --- > > + | ... > > +current_leader_id = 1 > > + | --- > > + | ... > > +test_run:switch(SERVERS[current_leader_id]) > > + | --- > > + | - true > > + | ... > > +box.cfg{replication_synchro_quorum=3, replication_synchro_timeout=0.1} > > 7. The timeout is tiny. It will lead to flakiness sooner or later, 100%. increased to 1 sec > > + | --- > > + | ... > > +_ = box.schema.space.create('sync', {is_sync=true}) > > + | --- > > + | ... > > +_ = box.space.sync:create_index('pk') > > + | --- > > + | ... > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > + > > +-- Testcase body. > > +for i=1,10 do \ > > + new_leader_id = random(current_leader_id, 1, #SERVERS) \ > > + test_run:switch(SERVERS[new_leader_id]) \ > > + box.cfg{read_only=false} \ > > + fiber = require('fiber') \ > > + f1 = fiber.create(function() box.space.sync:delete{} end) \ > > 8. Delete without a key will fail. You would notice it if you > would check results of the DML operations. Please, do that via pcall. > replaced delete{} with truncate{} > > + f2 = fiber.create(function() for i=1,10000 do box.space.sync:insert{i} end end) \ > > 9. You have \ exactly to avoid such long lines. splitted for shorter lines > > > + f1.status() \ > > + f2.status() \ > > 10. Output is not printed inside one statement. This whole cycle is > one statement because of \, so these status() calls are useless. removed > > + test_run:switch('default') \ > > + test_run:switch(SERVERS[current_leader_id]) \ > > + box.cfg{read_only=true} \ > > + test_run:switch('default') \ > > + current_leader_id = new_leader_id \ > > + fiber.sleep(0.1) \ > > 11. Why do you need this fiber.sleep()? I don't remember the reason to add it, but test works fine without it. So I removed it in updated patch. > > +end > > + | --- > > + | ... > > + > > +-- Teardown. > > +test_run:switch(SERVERS[current_leader_id]) > > + | --- > > + | - true > > + | ... > > +box.space.sync:drop() > > + | --- > > + | ... > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +test_run:drop_cluster(SERVERS) > > + | --- > > + | ... -- sergeyb@ From sergeyb at tarantool.org Wed Aug 26 17:48:37 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Wed, 26 Aug 2020 17:48:37 +0300 Subject: [Tarantool-patches] [PATCH 2/3] replication: test clear_synchro_queue function In-Reply-To: <1598427097.413655454@f128.i.mail.ru> References: <8c3b7d4c-22e8-108a-b15e-ef3c89510a63@tarantool.org> <20200825124939.GB47610@pony.bronevichok.ru> <1598427097.413655454@f128.i.mail.ru> Message-ID: <20200826144837.GD47610@pony.bronevichok.ru> On 10:31 Wed 26 Aug , Serge Petrenko wrote: > > Hi, Sergey! > Thanks for the patch and the fixes. New version LGTM. I couldn?t find > the other patches in my mailbox though, could you please resend them? If > you need my review, of course. Thanks for review! Second patch in a series updated [1]. You are in CC. Branch: ligurio/gh-4842-qsync-testing 1. https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019197.html From imun at tarantool.org Wed Aug 26 17:48:37 2020 From: imun at tarantool.org (Igor Munkin) Date: Wed, 26 Aug 2020 17:48:37 +0300 Subject: [Tarantool-patches] [PATCH v2 1/2] core: introduce various platform metrics In-Reply-To: <35a19def79a9cbc46dabdfa579869af9e4e589fb.1595794764.git.skaplun@tarantool.org> References: <35a19def79a9cbc46dabdfa579869af9e4e589fb.1595794764.git.skaplun@tarantool.org> Message-ID: <20200826144837.GA18920@tarantool.org> Sergey, Thanks for the patch! It looks OK in general, except several nits, I left below. On 26.07.20, Sergey Kaplun wrote: > This patch introduces the following counters: > - overall amount of allocated tables, cdata and udata objects > - number of incremental GC steps grouped by GC state > - number of string hashes hits and misses > - amount of allocated and freed memory > - number of trace aborts and restored snapshots Typo: we usually use whitespace prior to the list bullets (as you did in the previous version). > > Interfaces to obtain these metrics via both Lua and C API are > introduced in the next patch. > > Part of tarantool/tarantool#5187 > --- > src/lj_cdata.c | 2 ++ > src/lj_cdata.h | 2 ++ > src/lj_gc.c | 4 ++++ > src/lj_gc.h | 6 +----- > src/lj_jit.h | 3 +++ > src/lj_obj.h | 22 ++++++++++++++++++++++ > src/lj_snap.c | 1 + > src/lj_state.c | 2 +- > src/lj_str.c | 5 +++++ > src/lj_tab.c | 2 ++ > src/lj_trace.c | 5 ++++- > src/lj_udata.c | 2 ++ > 12 files changed, 49 insertions(+), 7 deletions(-) > > diff --git a/src/lj_jit.h b/src/lj_jit.h > index 7eb3d2a..90c1914 100644 > --- a/src/lj_jit.h > +++ b/src/lj_jit.h > @@ -475,6 +475,9 @@ typedef struct jit_State { > size_t szmcarea; /* Size of current mcode area. */ > size_t szallmcarea; /* Total size of all allocated mcode areas. */ > > + size_t nsnaprestore; /* Overall number of snap restores for this jit_State. */ > + size_t ntraceabort; /* Overall number of abort traces for this jit_State. */ Why did you emphasize that the counters relate to *this jit_State*? There are no such mentions elsewhere. > + > TValue errinfo; /* Additional info element for trace errors. */ > > #if LJ_HASPROFILE > diff --git a/src/lj_obj.h b/src/lj_obj.h > index f368578..18df173 100644 > --- a/src/lj_obj.h > +++ b/src/lj_obj.h > @@ -578,6 +589,9 @@ typedef struct GCState { > uint8_t state; /* GC state. */ > uint8_t nocdatafin; /* No cdata finalizer called. */ > uint8_t unused2; > + size_t freed; /* Total amount of freed memory. */ > + size_t allocated; /* Total amount of allocated memory. */ > + size_t state_count[GCSmax]; /* Count of incremental GC steps per state. */ One more time: consider the structure alignment and reorder the introduced fields to avoid excess padding. > MSize sweepstr; /* Sweep position in string table. */ > GCRef root; /* List of all collectable objects. */ > MRef sweep; /* Sweep position in root list. */ > @@ -602,6 +622,8 @@ typedef struct global_State { > BloomFilter next[2]; > } strbloom; > #endif > + size_t strhash_hit; /* Strings amount founded in string hash. */ Typo: s/founded/found/. > + size_t strhash_miss; /* Strings amount allocated and put into string hash. */ > lua_Alloc allocf; /* Memory allocator. */ > void *allocd; /* Memory allocator data. */ > GCState gc; /* Garbage collector. */ > > -- > 2.24.1 > Furthermore, please address the comments I left regarding the patch you've made for CNEW IR[1] and squash it with this one. Sergos, do we need other JIT architectures to be patched in scope of this series or Sergey can just add the corresponding preprocessor condition to stub the issue for now? [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019012.html -- Best regards, IM From sergeyb at tarantool.org Wed Aug 26 18:10:10 2020 From: sergeyb at tarantool.org (sergeyb at tarantool.org) Date: Wed, 26 Aug 2020 18:10:10 +0300 Subject: [Tarantool-patches] [PATCH v1] replication: change space sync mode in a loop Message-ID: <4c64406bcbe4f52629eba5a5c4dc1c9ea0115dea.1598454255.git.sergeyb@tarantool.org> From: Sergey Bronnikov New regression tests covers cases when one can change synchronous mode of space to asynchronous and vice versa. Closes #5055 Part of #5144 --- Branch: ligurio/gh-4842-qsync-change-mode CI: https://gitlab.com/tarantool/tarantool/-/pipelines/182271234 test/replication/qsync_sync_mode.result | 164 ++++++++++++++++++++++ test/replication/qsync_sync_mode.test.lua | 90 ++++++++++++ 2 files changed, 254 insertions(+) create mode 100644 test/replication/qsync_sync_mode.result create mode 100644 test/replication/qsync_sync_mode.test.lua diff --git a/test/replication/qsync_sync_mode.result b/test/replication/qsync_sync_mode.result new file mode 100644 index 000000000..f2f95ec0f --- /dev/null +++ b/test/replication/qsync_sync_mode.result @@ -0,0 +1,164 @@ +-- test-run result file version 2 +env = require('test_run') + | --- + | ... +test_run = env.new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... +fiber = require('fiber') + | --- + | ... +math = require('math') + | --- + | ... +math.randomseed(os.time()) + | --- + | ... + +orig_synchro_quorum = box.cfg.replication_synchro_quorum + | --- + | ... +orig_synchro_timeout = box.cfg.replication_synchro_timeout + | --- + | ... + +disable_sync_mode = function() \ + local s = box.space._space:get(box.space.sync.id) \ + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ + box.space._space:replace(new_s) \ +end + | --- + | ... + +enable_sync_mode = function() \ + local s = box.space._space:get(box.space.sync.id) \ + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ + box.space._space:replace(new_s) \ +end + | --- + | ... + +set_random_sync_mode = function() \ + if (math.random(1, 10) > 5) then \ + enable_sync_mode() \ + else \ + disable_sync_mode() \ + end \ +end + | --- + | ... + +set_random_quorum = function(n) \ + box.cfg{replication_synchro_quorum=math.random(1, n)} \ +end + | --- + | ... + +box.schema.user.grant('guest', 'replication') + | --- + | ... + +-- Setup an async cluster with two instances. +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica with wait=True, wait_load=True') + | --- + | - true + | ... + +-- Write data to a leader, enable and disable sync mode in background in a +-- loop. Expected no data loss. +-- Testcase setup. +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) + | --- + | ... +_ = box.space.sync:create_index('pk') + | --- + | ... +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} + | --- + | ... +-- Testcase body. +for i = 1,10 do \ + set_random_sync_mode() \ + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ + test_run:switch('replica') \ + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ + end \ + test_run:switch('default') \ +end + | --- + | ... +-- Testcase cleanup. +test_run:switch('default') + | --- + | - true + | ... +box.space.sync:drop() + | --- + | ... + +-- Write data to a leader, enable and disable sync mode and change quorum value +-- in background in a loop. +-- Testcase setup. +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) + | --- + | ... +_ = box.space.sync:create_index('pk') + | --- + | ... +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} + | --- + | ... +-- Testcase body. +for i = 1,10 do \ + set_random_sync_mode() \ + set_random_quorum(5) \ + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ + test_run:switch('replica') \ + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ + end \ + test_run:switch('default') \ +end + | --- + | ... +-- Testcase cleanup. +test_run:switch('default') + | --- + | - true + | ... +box.space.sync:drop() + | --- + | ... + +-- Teardown. +test_run:cmd('switch default') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +test_run:cleanup_cluster() + | --- + | ... +box.schema.user.revoke('guest', 'replication') + | --- + | ... +box.cfg{ \ + replication_synchro_quorum = orig_synchro_quorum, \ + replication_synchro_timeout = orig_synchro_timeout, \ +} + | --- + | ... diff --git a/test/replication/qsync_sync_mode.test.lua b/test/replication/qsync_sync_mode.test.lua new file mode 100644 index 000000000..706261c10 --- /dev/null +++ b/test/replication/qsync_sync_mode.test.lua @@ -0,0 +1,90 @@ +env = require('test_run') +test_run = env.new() +engine = test_run:get_cfg('engine') +fiber = require('fiber') +math = require('math') +math.randomseed(os.time()) + +orig_synchro_quorum = box.cfg.replication_synchro_quorum +orig_synchro_timeout = box.cfg.replication_synchro_timeout + +disable_sync_mode = function() \ + local s = box.space._space:get(box.space.sync.id) \ + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ + box.space._space:replace(new_s) \ +end + +enable_sync_mode = function() \ + local s = box.space._space:get(box.space.sync.id) \ + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ + box.space._space:replace(new_s) \ +end + +set_random_sync_mode = function() \ + if (math.random(1, 10) > 5) then \ + enable_sync_mode() \ + else \ + disable_sync_mode() \ + end \ +end + +set_random_quorum = function(n) \ + box.cfg{replication_synchro_quorum=math.random(1, n)} \ +end + +box.schema.user.grant('guest', 'replication') + +-- Setup an async cluster with two instances. +test_run:cmd('create server replica with rpl_master=default,\ + script="replication/replica.lua"') +test_run:cmd('start server replica with wait=True, wait_load=True') + +-- Write data to a leader, enable and disable sync mode in background in a +-- loop. Expected no data loss. +-- Testcase setup. +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) +_ = box.space.sync:create_index('pk') +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} +-- Testcase body. +for i = 1,10 do \ + set_random_sync_mode() \ + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ + test_run:switch('replica') \ + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ + end \ + test_run:switch('default') \ +end +-- Testcase cleanup. +test_run:switch('default') +box.space.sync:drop() + +-- Write data to a leader, enable and disable sync mode and change quorum value +-- in background in a loop. +-- Testcase setup. +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) +_ = box.space.sync:create_index('pk') +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} +-- Testcase body. +for i = 1,10 do \ + set_random_sync_mode() \ + set_random_quorum(5) \ + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ + test_run:switch('replica') \ + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ + end \ + test_run:switch('default') \ +end +-- Testcase cleanup. +test_run:switch('default') +box.space.sync:drop() + +-- Teardown. +test_run:cmd('switch default') +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +test_run:cleanup_cluster() +box.schema.user.revoke('guest', 'replication') +box.cfg{ \ + replication_synchro_quorum = orig_synchro_quorum, \ + replication_synchro_timeout = orig_synchro_timeout, \ +} -- 2.26.2 From sergos at tarantool.org Wed Aug 26 18:52:07 2020 From: sergos at tarantool.org (Sergey Ostanevich) Date: Wed, 26 Aug 2020 18:52:07 +0300 Subject: [Tarantool-patches] [PATCH v2 1/2] core: introduce various platform metrics In-Reply-To: <20200826144837.GA18920@tarantool.org> References: <35a19def79a9cbc46dabdfa579869af9e4e589fb.1595794764.git.skaplun@tarantool.org> <20200826144837.GA18920@tarantool.org> Message-ID: <20200826155207.GA49@tarantool.org> On 26 ??? 17:48, Igor Munkin wrote: > Sergey, > > Thanks for the patch! It looks OK in general, except several nits, I > left below. > > On 26.07.20, Sergey Kaplun wrote: > > This patch introduces the following counters: > > - overall amount of allocated tables, cdata and udata objects > > - number of incremental GC steps grouped by GC state > > - number of string hashes hits and misses > > - amount of allocated and freed memory > > - number of trace aborts and restored snapshots > > Typo: we usually use whitespace prior to the list bullets (as you did in > the previous version). > > > > > Interfaces to obtain these metrics via both Lua and C API are > > introduced in the next patch. > > > > Part of tarantool/tarantool#5187 > > --- > > src/lj_cdata.c | 2 ++ > > src/lj_cdata.h | 2 ++ > > src/lj_gc.c | 4 ++++ > > src/lj_gc.h | 6 +----- > > src/lj_jit.h | 3 +++ > > src/lj_obj.h | 22 ++++++++++++++++++++++ > > src/lj_snap.c | 1 + > > src/lj_state.c | 2 +- > > src/lj_str.c | 5 +++++ > > src/lj_tab.c | 2 ++ > > src/lj_trace.c | 5 ++++- > > src/lj_udata.c | 2 ++ > > 12 files changed, 49 insertions(+), 7 deletions(-) > > > > > > > diff --git a/src/lj_jit.h b/src/lj_jit.h > > index 7eb3d2a..90c1914 100644 > > --- a/src/lj_jit.h > > +++ b/src/lj_jit.h > > @@ -475,6 +475,9 @@ typedef struct jit_State { > > size_t szmcarea; /* Size of current mcode area. */ > > size_t szallmcarea; /* Total size of all allocated mcode areas. */ > > > > + size_t nsnaprestore; /* Overall number of snap restores for this jit_State. */ > > + size_t ntraceabort; /* Overall number of abort traces for this jit_State. */ > > Why did you emphasize that the counters relate to *this jit_State*? > There are no such mentions elsewhere. > > > + > > TValue errinfo; /* Additional info element for trace errors. */ > > > > #if LJ_HASPROFILE > > diff --git a/src/lj_obj.h b/src/lj_obj.h > > index f368578..18df173 100644 > > --- a/src/lj_obj.h > > +++ b/src/lj_obj.h > > > > > @@ -578,6 +589,9 @@ typedef struct GCState { > > uint8_t state; /* GC state. */ > > uint8_t nocdatafin; /* No cdata finalizer called. */ > > uint8_t unused2; > > + size_t freed; /* Total amount of freed memory. */ > > + size_t allocated; /* Total amount of allocated memory. */ > > + size_t state_count[GCSmax]; /* Count of incremental GC steps per state. */ > > One more time: consider the structure alignment and reorder the > introduced fields to avoid excess padding. > > > MSize sweepstr; /* Sweep position in string table. */ > > GCRef root; /* List of all collectable objects. */ > > MRef sweep; /* Sweep position in root list. */ > > > > > @@ -602,6 +622,8 @@ typedef struct global_State { > > BloomFilter next[2]; > > } strbloom; > > #endif > > + size_t strhash_hit; /* Strings amount founded in string hash. */ > > Typo: s/founded/found/. > > > + size_t strhash_miss; /* Strings amount allocated and put into string hash. */ > > lua_Alloc allocf; /* Memory allocator. */ > > void *allocd; /* Memory allocator data. */ > > GCState gc; /* Garbage collector. */ > > > > > > > -- > > 2.24.1 > > > > Furthermore, please address the comments I left regarding the patch > you've made for CNEW IR[1] and squash it with this one. > > Sergos, do we need other JIT architectures to be patched in scope of > this series or Sergey can just add the corresponding preprocessor > condition to stub the issue for now? AFAU the CNEW IR appeared in src/lj_asm_x86.h so I didn't get what preprocessor condition do you mean. Also, the inconsistency with the allocated CDATA counter will appear anyways, so it needs to be implemented for other targets also. Looks like a not-so-big deal? BTW, AlexanderTi made a successful build and a little less successful run on an ARM emulator that took some reasonable time. So there's a way to test the change for ARM. Both PPC and MIPS I suppose would be just fine to test for successful build. > > > [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019012.html > > -- > Best regards, > IM From sergepetrenko at tarantool.org Thu Aug 27 13:49:42 2020 From: sergepetrenko at tarantool.org (=?UTF-8?B?U2VyZ2UgUGV0cmVua28=?=) Date: Thu, 27 Aug 2020 13:49:42 +0300 Subject: [Tarantool-patches] =?utf-8?q?=5BPATCH_3/3=5D_replication=3A_add_?= =?utf-8?q?test_with_random_leaders_promotion_and_demotion?= In-Reply-To: <20200826144538.GC47610@pony.bronevichok.ru> References: <3c7b2274-8443-2be7-c181-2c7026ab0fec@tarantool.org> <20200826144538.GC47610@pony.bronevichok.ru> Message-ID: <1598525382.271317018@f511.i.mail.ru> Hi! Thanks for the fixes! I?m pasting parts of the patch below to comment on. ? ? +box.cfg({ +? ? listen = instance_uri(INSTANCE_ID); +? ? replication_connect_quorum = 3; +? ? replication = { +? ? ? ? instance_uri(1); +? ? ? ? instance_uri(2); +? ? ? ? instance_uri(3); +? ? ? ? instance_uri(4); +? ? ? ? instance_uri(5); +? ? }; +}) ? ? * You should?either omit?`replication_connect_quorum` at all, or set it to 5. Omitting it will have the same effect. I think you meant `replication_synchro_quorum` here, then it makes sense to set it to 3. Also ? `replication_synchro_timeout` should be set here, I?ll mention it ???????? again below. ? ? + +NUM_INSTANCES = 5 +BROKEN_QUORUM = NUM_INSTANCES + 1 + ? ? * BROKEN_QUORUM?assigned but never used. ? ? + +test_run:create_cluster(SERVERS, "replication", {args="0.1"}) +test_run:wait_fullmesh(SERVERS) ? ? * You?re passing some argument?to qsync1, ??qsync5 instances, but you never use ?it. ? ? +current_leader_id = 1 +test_run:switch(SERVERS[current_leader_id]) +box.cfg{replication_synchro_timeout=1} ? ? * You should set `replication_synchro_timeout`?on every instance, not only on qsync1 so ? you better move this?box.cfg call?to the instance file. Besides, the timeout should be bigger (much bigger), like Vlad said. We typically use 30 seconds for various replication timeouts. It?s fairly?common when a test is stable on your machine, but is flaky on testing machines. ? ? +test_run:switch('default') + +-- Testcase body. +for i=1,10 do? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? new_leader_id = random(current_leader_id, 1, #SERVERS) ? ? ? ? ? ? ? ? ? ? \ +? ? test_run:switch(SERVERS[new_leader_id])? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? box.cfg{read_only=false} ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? f1 = fiber.create(function() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? ? ? pcall(box.space.sync:truncate{}) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? end) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ ? ? * Why put truncate call in a separate fiber? Why use truncate at all? You may just replace all your `insert` calls below with `replace`, and then truncate won?t be needed. This is up to you though. ? ? +? ? f2 = fiber.create(function() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? ? ? for i=1,10000 do box.space.sync:insert{i} end? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? end) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ ? * So you?re testing a case when a leader has some unconfirmed transactions in limbo and then a leader change happens. Then you need to call `clear_synchro_queue` on a new leader to wait for?confirmation of old txns. Otherwise the new leader fails to insert its data, but the test doesn?t show this, because you don?t check fiber state or `insert()` return values. ? +? ? test_run:switch('default') ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? test_run:switch(SERVERS[current_leader_id])? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ ? ? * The 2 lines above are useless. ? ? +? ? box.cfg{read_only=true}? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? test_run:switch('default') ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +? ? current_leader_id = new_leader_id? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ +end + +-- Teardown. ? ? ? -- Serge?Petrenko -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Thu Aug 27 14:27:14 2020 From: sergepetrenko at tarantool.org (Sergey Petrenko) Date: Thu, 27 Aug 2020 14:27:14 +0300 Subject: [Tarantool-patches] [PATCH v1] replication: change space sync mode in a loop In-Reply-To: <4c64406bcbe4f52629eba5a5c4dc1c9ea0115dea.1598454255.git.sergeyb@tarantool.org> References: <4c64406bcbe4f52629eba5a5c4dc1c9ea0115dea.1598454255.git.sergeyb@tarantool.org> Message-ID: <347d6e10-deb8-d349-e1df-033b5075903f@tarantool.org> Hi! Thanks for the patch! Please see my comments below. 26.08.2020 18:10, sergeyb at tarantool.org ?????: > From: Sergey Bronnikov > > New regression tests covers cases when one can change synchronous mode > of space to asynchronous and vice versa. > > Closes #5055 > Part of #5144 > --- > > Branch: ligurio/gh-4842-qsync-change-mode > CI: https://gitlab.com/tarantool/tarantool/-/pipelines/182271234 > > test/replication/qsync_sync_mode.result | 164 ++++++++++++++++++++++ > test/replication/qsync_sync_mode.test.lua | 90 ++++++++++++ > 2 files changed, 254 insertions(+) > create mode 100644 test/replication/qsync_sync_mode.result > create mode 100644 test/replication/qsync_sync_mode.test.lua > > diff --git a/test/replication/qsync_sync_mode.result b/test/replication/qsync_sync_mode.result > new file mode 100644 > index 000000000..f2f95ec0f > --- /dev/null > +++ b/test/replication/qsync_sync_mode.result > @@ -0,0 +1,164 @@ > +-- test-run result file version 2 > +env = require('test_run') > + | --- > + | ... > +test_run = env.new() > + | --- > + | ... > +engine = test_run:get_cfg('engine') > + | --- > + | ... > +fiber = require('fiber') > + | --- > + | ... > +math = require('math') > + | --- > + | ... > +math.randomseed(os.time()) > + | --- > + | ... > + > +orig_synchro_quorum = box.cfg.replication_synchro_quorum > + | --- > + | ... > +orig_synchro_timeout = box.cfg.replication_synchro_timeout > + | --- > + | ... > + > +disable_sync_mode = function() \ > + local s = box.space._space:get(box.space.sync.id) \ > + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ > + box.space._space:replace(new_s) \ > +end > + | --- > + | ... > + > +enable_sync_mode = function() \ > + local s = box.space._space:get(box.space.sync.id) \ > + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ > + box.space._space:replace(new_s) \ > +end > + | --- > + | ... > + > +set_random_sync_mode = function() \ > + if (math.random(1, 10) > 5) then \ > + enable_sync_mode() \ > + else \ > + disable_sync_mode() \ > + end \ > +end > + | --- > + | ... > + > +set_random_quorum = function(n) \ > + box.cfg{replication_synchro_quorum=math.random(1, n)} \ > +end > + | --- > + | ... > + > +box.schema.user.grant('guest', 'replication') > + | --- > + | ... > + > +-- Setup an async cluster with two instances. > +test_run:cmd('create server replica with rpl_master=default,\ > + script="replication/replica.lua"') > + | --- > + | - true > + | ... > +test_run:cmd('start server replica with wait=True, wait_load=True') > + | --- > + | - true > + | ... > + > +-- Write data to a leader, enable and disable sync mode in background in a > +-- loop. Expected no data loss. > +-- Testcase setup. > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > + | --- > + | ... > +_ = box.space.sync:create_index('pk') > + | --- > + | ... > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > + | --- > + | ... > +-- Testcase body. > +for i = 1,10 do \ > + set_random_sync_mode() \ > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > + test_run:switch('replica') \ > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > + end \ > + test_run:switch('default') \ > +end > + | --- > + | ... > +-- Testcase cleanup. > +test_run:switch('default') > + | --- > + | - true > + | ... > +box.space.sync:drop() > + | --- > + | ... > + > +-- Write data to a leader, enable and disable sync mode and change quorum value > +-- in background in a loop. > +-- Testcase setup. > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > + | --- > + | ... > +_ = box.space.sync:create_index('pk') > + | --- > + | ... > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > + | --- > + | ... > +-- Testcase body. > +for i = 1,10 do \ > + set_random_sync_mode() \ > + set_random_quorum(5) \ > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > + test_run:switch('replica') \ > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > + end \ > + test_run:switch('default') \ > +end > + | --- > + | ... > +-- Testcase cleanup. > +test_run:switch('default') > + | --- > + | - true > + | ... > +box.space.sync:drop() > + | --- > + | ... > + > +-- Teardown. > +test_run:cmd('switch default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server replica') > + | --- > + | - true > + | ... > +test_run:cmd('delete server replica') > + | --- > + | - true > + | ... > +test_run:cleanup_cluster() > + | --- > + | ... > +box.schema.user.revoke('guest', 'replication') > + | --- > + | ... > +box.cfg{ \ > + replication_synchro_quorum = orig_synchro_quorum, \ > + replication_synchro_timeout = orig_synchro_timeout, \ > +} > + | --- > + | ... > diff --git a/test/replication/qsync_sync_mode.test.lua b/test/replication/qsync_sync_mode.test.lua > new file mode 100644 > index 000000000..706261c10 > --- /dev/null > +++ b/test/replication/qsync_sync_mode.test.lua > @@ -0,0 +1,90 @@ > +env = require('test_run') > +test_run = env.new() > +engine = test_run:get_cfg('engine') > +fiber = require('fiber') > +math = require('math') > +math.randomseed(os.time()) > + > +orig_synchro_quorum = box.cfg.replication_synchro_quorum > +orig_synchro_timeout = box.cfg.replication_synchro_timeout > + > +disable_sync_mode = function() \ > + local s = box.space._space:get(box.space.sync.id) \ > + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ > + box.space._space:replace(new_s) \ > +end > + > +enable_sync_mode = function() \ > + local s = box.space._space:get(box.space.sync.id) \ > + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ > + box.space._space:replace(new_s) \ > +end > + Vlad has pushed a patch with `space:alter` a couple of days ago. So now you may say `space:alter{is_sync=true}`, `space:alter{is_sync=false}` It does the same work you do, but looks much simpler. > +set_random_sync_mode = function() \ > + if (math.random(1, 10) > 5) then \ > + enable_sync_mode() \ > + else \ > + disable_sync_mode() \ > + end \ > +end > + > +set_random_quorum = function(n) \ > + box.cfg{replication_synchro_quorum=math.random(1, n)} \ > +end > + > +box.schema.user.grant('guest', 'replication') > + > +-- Setup an async cluster with two instances. > +test_run:cmd('create server replica with rpl_master=default,\ > + script="replication/replica.lua"') > +test_run:cmd('start server replica with wait=True, wait_load=True') > + > +-- Write data to a leader, enable and disable sync mode in background in a > +-- loop. Expected no data loss. But sync mode is not set in background, it's set in the same loop where insertions happen. > +-- Testcase setup. > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > +_ = box.space.sync:create_index('pk') > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} Is such a tiny timeout intended? I see you wrap the errors in a pcall, but still, it may happen that you get a timeout 10 out of 10 times and then you'll test nothing. We usually use 30 second replication timeouts. > +-- Testcase body. > +for i = 1,10 do \ > + set_random_sync_mode() \ > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > + test_run:switch('replica') \ > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > + end \ > + test_run:switch('default') \ > +end > +-- Testcase cleanup. > +test_run:switch('default') > +box.space.sync:drop() > + > +-- Write data to a leader, enable and disable sync mode and change quorum value > +-- in background in a loop. > +-- Testcase setup. > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > +_ = box.space.sync:create_index('pk') > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} Ok, I see why you need a small timeout in this testcase. Maybe set a small timeout only when the insertion is expected to fail(quorum = 3 or 4)?. And use a 30 sec timeout otherwise. > +-- Testcase body. > +for i = 1,10 do \ > + set_random_sync_mode() \ > + set_random_quorum(5) \ Math.random() range is inclusive, so it'd be more fair, to choose quorum from 1 to 4. Then there'd be 2 cases of successful write, quorum 1 and 2, and 2 cases of failed write, quorum 3 and 4. > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > + test_run:switch('replica') \ > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > + end \ > + test_run:switch('default') \ > +end > +-- Testcase cleanup. > +test_run:switch('default') > +box.space.sync:drop() > + > +-- Teardown. > +test_run:cmd('switch default') > +test_run:cmd('stop server replica') > +test_run:cmd('delete server replica') > +test_run:cleanup_cluster() > +box.schema.user.revoke('guest', 'replication') > +box.cfg{ \ > + replication_synchro_quorum = orig_synchro_quorum, \ > + replication_synchro_timeout = orig_synchro_timeout, \ > +} From alyapunov at tarantool.org Thu Aug 27 15:28:39 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Thu, 27 Aug 2020 15:28:39 +0300 Subject: [Tarantool-patches] [PATCH v1] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: References: Message-ID: Hi! Thanks for the patch! See 2 minor comments below. On 21.08.2020 07:56, Alexander V. Tikhonov wrote: > As mentioned above xdir_scan() function is not system call and can be > changed in any possible way and it can return any result value without > need to setup errno. So check outside of this function on errno could > be broken. > > To avoid of it we must avoid of errno checks outside of the function. I would say "To avoid that we must not check errno after call of the function." > diff --git a/src/box/xlog.c b/src/box/xlog.c > index 6ccd3d68d..74f761994 100644 > --- a/src/box/xlog.c > +++ b/src/box/xlog.c > @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > * @return nothing. > */ > int > -xdir_scan(struct xdir *dir) > +xdir_scan(struct xdir *dir, bool is_dir_required) The function comment also needs fixing. The new argument must have a description. The comment was already wrong though, it doesn't describe return value. Another problem is that the comment should be near function declaration (in xlog.h file) rather than definition. I would be glad if you fixed it. From avtikhon at tarantool.org Thu Aug 27 17:06:58 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 27 Aug 2020 17:06:58 +0300 Subject: [Tarantool-patches] [PATCH v1] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: References: Message-ID: <20200827140658.GA16950@hpalx> Hi Alexander, thanks for the review. I've made all changes as you suggested, please check the new version of the patch. On Thu, Aug 27, 2020 at 03:28:39PM +0300, Aleksandr Lyapunov wrote: > Hi! Thanks for the patch! See 2 minor comments below. > > On 21.08.2020 07:56, Alexander V. Tikhonov wrote: > > As mentioned above xdir_scan() function is not system call and can be > > changed in any possible way and it can return any result value without > > need to setup errno. So check outside of this function on errno could > > be broken. > > > > To avoid of it we must avoid of errno checks outside of the function. > I would say "To avoid that we must not check errno after call of the > function." > > diff --git a/src/box/xlog.c b/src/box/xlog.c > > index 6ccd3d68d..74f761994 100644 > > --- a/src/box/xlog.c > > +++ b/src/box/xlog.c > > @@ -511,13 +511,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > > * @return nothing. > > */ > > int > > -xdir_scan(struct xdir *dir) > > +xdir_scan(struct xdir *dir, bool is_dir_required) > The function comment also needs fixing. > The new argument must have a description. > The comment was already wrong though, it doesn't describe return value. > Another problem is that the comment should be near function > declaration (in xlog.h file) rather than definition. > I would be glad if you fixed it. > From avtikhon at tarantool.org Thu Aug 27 17:12:52 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Thu, 27 Aug 2020 17:12:52 +0300 Subject: [Tarantool-patches] [PATCH v2] vinyl: fix check vinyl_dir existence at bootstrap Message-ID: <77367c061961d33a6fd11325f4c6abb443ffb15a.1598537487.git.avtikhon@tarantool.org> During implementation of openSUSE build with testing got failed test box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and vinyl_dir existed and also errno was set to ENOENT, box configuration succeeded, but it shouldn't. Reason of this wrong behavior was that not all of the failure paths in xdir_scan() set errno, but the caller assumed it. Debugging the issue found that after xdir_scan() there was incorrect check for errno when it returned negative values. xdir_scan() is not system call and negative return value from it doesn't mean that errno would be set too. Found that in situations when errno was left from previous commands before xdir_scan() and xdir_scan() returned negative value by itself it produced the wrong check. The previous failed logic of the check was to catch the error ENOENT which set in the xdir_scan() function to handle the situation when vinyl_dir was not exist. It failed, because checking ENOENT outside the xdir_scan() function, we had to be sure that ENOENT had come from xdir_scan() function call indeed and not from any other functions before. To be sure in it possible fix could be reset errno before xdir_scan() call, because errno could be passed from any other function before call to xdir_scan(). As mentioned above xdir_scan() function is not system call and can be changed in any possible way and it can return any result value without need to setup errno. So check outside of this function on errno could be broken. To avoid that we must not check errno after call of the function. Better solution is to use the flag in xdir_scan(), to check if the directory should exist. So errno check was removed and instead of it the check for vinyl_dir existence using flag added. Closes #4594 Needed for #4562 Co-authored-by: Alexander Turenko --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-vinyl-fix Issue: https://github.com/tarantool/tarantool/issues/4594 Issue: https://github.com/tarantool/tarantool/issues/4562 Review: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019130.html src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 4 +- src/box/vy_log.c | 4 +- src/box/wal.c | 2 +- src/box/xlog.c | 14 ++++- src/box/xlog.h | 7 +-- .../gh-4562-errno-at-xdir_scan.test.lua | 54 +++++++++++++++++++ 7 files changed, 76 insertions(+), 11 deletions(-) create mode 100755 test/box-tap/gh-4562-errno-at-xdir_scan.test.lua diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index dfd6fce6e..9f079a6b5 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, &xlog_opts_default); memtx->snap_dir.force_recovery = force_recovery; - if (xdir_scan(&memtx->snap_dir) != 0) + if (xdir_scan(&memtx->snap_dir, true) != 0) goto fail; /* diff --git a/src/box/recovery.cc b/src/box/recovery.cc index d1a503cfc..cd33e7635 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -121,7 +121,7 @@ void recovery_scan(struct recovery *r, struct vclock *end_vclock, struct vclock *gc_vclock) { - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || vclock_compare(end_vclock, &r->vclock) < 0) { @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, struct vclock *clock; if (scan_dir) - xdir_scan_xc(&r->wal_dir); + xdir_scan_xc(&r->wal_dir, true); if (xlog_cursor_is_open(&r->cursor)) { /* If there's a WAL open, recover from it first. */ diff --git a/src/box/vy_log.c b/src/box/vy_log.c index de4c5205c..d23b1c18a 100644 --- a/src/box/vy_log.c +++ b/src/box/vy_log.c @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) int vy_log_bootstrap(void) { - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return -1; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) return vy_log_rebootstrap(); @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) * because vinyl might not be even in use. Complain only * on an attempt to write a vylog. */ - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) + if (xdir_scan(&vy_log.dir, false) < 0) return NULL; if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { diff --git a/src/box/wal.c b/src/box/wal.c index 045006b60..ea707aa5e 100644 --- a/src/box/wal.c +++ b/src/box/wal.c @@ -556,7 +556,7 @@ wal_enable(void) * existing WAL files. Required for garbage collection, * see wal_collect_garbage(). */ - if (xdir_scan(&writer->wal_dir)) + if (xdir_scan(&writer->wal_dir, true)) return -1; /* Open the most recent WAL file. */ diff --git a/src/box/xlog.c b/src/box/xlog.c index 6ccd3d68d..974f460be 100644 --- a/src/box/xlog.c +++ b/src/box/xlog.c @@ -487,6 +487,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * replication set (see also _cluster system space and vclock.h * comments). * + * @param dir - directory to scan + * @param is_dir_required - flag set if the directory should exist + * + * @return: + * 0 - on success or flag 'is_dir_required' was set to False in + * xdir_scan() arguments and opendir() failed with errno ENOENT; + * -1 - if opendir() failed, with other than ENOENT errno either + * when 'is_dir_required' was set to True in xdir_scan() arguments. + * * This function tries to avoid re-reading a file if * it is already in the set of files "known" to the log * dir object. This is done to speed up local hot standby and @@ -508,16 +517,17 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, * silence conditions such as out of memory or lack of OS * resources. * - * @return nothing. */ int -xdir_scan(struct xdir *dir) +xdir_scan(struct xdir *dir, bool is_dir_required) { DIR *dh = opendir(dir->dirname); /* log dir */ int64_t *signatures = NULL; /* log file names */ size_t s_count = 0, s_capacity = 0; if (dh == NULL) { + if (!is_dir_required && errno == ENOENT) + return 0; diag_set(SystemError, "error reading directory '%s'", dir->dirname); return -1; diff --git a/src/box/xlog.h b/src/box/xlog.h index 9ffce598b..5b1f42ce1 100644 --- a/src/box/xlog.h +++ b/src/box/xlog.h @@ -185,9 +185,10 @@ xdir_destroy(struct xdir *dir); * index with all log files (or snapshots) in the directory. * Must be used if it is necessary to find the last log/ * snapshot or scan through all logs. + * Function arguments described in xlog.c source file. */ int -xdir_scan(struct xdir *dir); +xdir_scan(struct xdir *dir, bool is_dir_required); /** * Check that a directory exists and is writable. @@ -821,9 +822,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, #include "exception.h" static inline void -xdir_scan_xc(struct xdir *dir) +xdir_scan_xc(struct xdir *dir, bool is_dir_required) { - if (xdir_scan(dir) == -1) + if (xdir_scan(dir, is_dir_required) == -1) diag_raise(); } diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua new file mode 100755 index 000000000..e6dd68d54 --- /dev/null +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua @@ -0,0 +1,54 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local test = tap.test('cfg') +local fio = require('fio') +test:plan(1) + +local tarantool_bin = arg[-1] +local PANIC = 256 + +local function run_script(code) + local dir = fio.tempdir() + local script_path = fio.pathjoin(dir, 'script.lua') + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, + tonumber('0777', 8)) + script:write(code) + script:write("\nos.exit(0)") + script:close() + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] + local res = os.execute(string.format(cmd, dir, tarantool_bin)) + fio.rmtree(dir) + return res +end + +-- +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and +-- errno is set to ENOENT, box configuration succeeds, however it +-- should not +-- + +-- create vinyl_dir as temporary path +local vinyl_dir = fio.tempdir() + +-- fill vinyl_dir +run_script(string.format([[ +box.cfg{vinyl_dir = '%s'} +s = box.schema.space.create('test', {engine = 'vinyl'}) +s:create_index('pk') +os.exit(0) +]], vinyl_dir)) + +-- verify the case described above +local code = string.format([[ +local errno = require('errno') +errno(errno.ENOENT) +box.cfg{vinyl_dir = '%s'} +os.exit(0) +]], vinyl_dir) +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") + +-- remove vinyl_dir +fio.rmtree(vinyl_dir) + +os.exit(test:check() and 0 or 1) -- 2.17.1 From alyapunov at tarantool.org Thu Aug 27 17:16:03 2020 From: alyapunov at tarantool.org (Aleksandr Lyapunov) Date: Thu, 27 Aug 2020 17:16:03 +0300 Subject: [Tarantool-patches] [PATCH v2] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <77367c061961d33a6fd11325f4c6abb443ffb15a.1598537487.git.avtikhon@tarantool.org> References: <77367c061961d33a6fd11325f4c6abb443ffb15a.1598537487.git.avtikhon@tarantool.org> Message-ID: <174b6002-0b28-4cf3-f064-b3c19bce3b9b@tarantool.org> Hi and thanks again, great, LGTM! On 27.08.2020 17:12, Alexander V. Tikhonov wrote: > During implementation of openSUSE build with testing got failed test > box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and > vinyl_dir existed and also errno was set to ENOENT, box configuration > succeeded, but it shouldn't. Reason of this wrong behavior was that > not all of the failure paths in xdir_scan() set errno, but the caller > assumed it. > > Debugging the issue found that after xdir_scan() there was incorrect > check for errno when it returned negative values. xdir_scan() is not > system call and negative return value from it doesn't mean that errno > would be set too. Found that in situations when errno was left from > previous commands before xdir_scan() and xdir_scan() returned negative > value by itself it produced the wrong check. > > The previous failed logic of the check was to catch the error ENOENT > which set in the xdir_scan() function to handle the situation when > vinyl_dir was not exist. It failed, because checking ENOENT outside > the xdir_scan() function, we had to be sure that ENOENT had come from > xdir_scan() function call indeed and not from any other functions > before. To be sure in it possible fix could be reset errno before > xdir_scan() call, because errno could be passed from any other function > before call to xdir_scan(). > > As mentioned above xdir_scan() function is not system call and can be > changed in any possible way and it can return any result value without > need to setup errno. So check outside of this function on errno could > be broken. > > To avoid that we must not check errno after call of the function. > Better solution is to use the flag in xdir_scan(), to check if the > directory should exist. So errno check was removed and instead of it > the check for vinyl_dir existence using flag added. > > Closes #4594 > Needed for #4562 > > Co-authored-by: Alexander Turenko > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-vinyl-fix > Issue: https://github.com/tarantool/tarantool/issues/4594 > Issue: https://github.com/tarantool/tarantool/issues/4562 > Review: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019130.html > > src/box/memtx_engine.c | 2 +- > src/box/recovery.cc | 4 +- > src/box/vy_log.c | 4 +- > src/box/wal.c | 2 +- > src/box/xlog.c | 14 ++++- > src/box/xlog.h | 7 +-- > .../gh-4562-errno-at-xdir_scan.test.lua | 54 +++++++++++++++++++ > 7 files changed, 76 insertions(+), 11 deletions(-) > create mode 100755 test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > > diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c > index dfd6fce6e..9f079a6b5 100644 > --- a/src/box/memtx_engine.c > +++ b/src/box/memtx_engine.c > @@ -992,7 +992,7 @@ memtx_engine_new(const char *snap_dirname, bool force_recovery, > &xlog_opts_default); > memtx->snap_dir.force_recovery = force_recovery; > > - if (xdir_scan(&memtx->snap_dir) != 0) > + if (xdir_scan(&memtx->snap_dir, true) != 0) > goto fail; > > /* > diff --git a/src/box/recovery.cc b/src/box/recovery.cc > index d1a503cfc..cd33e7635 100644 > --- a/src/box/recovery.cc > +++ b/src/box/recovery.cc > @@ -121,7 +121,7 @@ void > recovery_scan(struct recovery *r, struct vclock *end_vclock, > struct vclock *gc_vclock) > { > - xdir_scan_xc(&r->wal_dir); > + xdir_scan_xc(&r->wal_dir, true); > > if (xdir_last_vclock(&r->wal_dir, end_vclock) < 0 || > vclock_compare(end_vclock, &r->vclock) < 0) { > @@ -307,7 +307,7 @@ recover_remaining_wals(struct recovery *r, struct xstream *stream, > struct vclock *clock; > > if (scan_dir) > - xdir_scan_xc(&r->wal_dir); > + xdir_scan_xc(&r->wal_dir, true); > > if (xlog_cursor_is_open(&r->cursor)) { > /* If there's a WAL open, recover from it first. */ > diff --git a/src/box/vy_log.c b/src/box/vy_log.c > index de4c5205c..d23b1c18a 100644 > --- a/src/box/vy_log.c > +++ b/src/box/vy_log.c > @@ -1014,7 +1014,7 @@ vy_log_rebootstrap(void) > int > vy_log_bootstrap(void) > { > - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > + if (xdir_scan(&vy_log.dir, false) < 0) > return -1; > if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) >= 0) > return vy_log_rebootstrap(); > @@ -1036,7 +1036,7 @@ vy_log_begin_recovery(const struct vclock *vclock) > * because vinyl might not be even in use. Complain only > * on an attempt to write a vylog. > */ > - if (xdir_scan(&vy_log.dir) < 0 && errno != ENOENT) > + if (xdir_scan(&vy_log.dir, false) < 0) > return NULL; > > if (xdir_last_vclock(&vy_log.dir, &vy_log.last_checkpoint) < 0) { > diff --git a/src/box/wal.c b/src/box/wal.c > index 045006b60..ea707aa5e 100644 > --- a/src/box/wal.c > +++ b/src/box/wal.c > @@ -556,7 +556,7 @@ wal_enable(void) > * existing WAL files. Required for garbage collection, > * see wal_collect_garbage(). > */ > - if (xdir_scan(&writer->wal_dir)) > + if (xdir_scan(&writer->wal_dir, true)) > return -1; > > /* Open the most recent WAL file. */ > diff --git a/src/box/xlog.c b/src/box/xlog.c > index 6ccd3d68d..974f460be 100644 > --- a/src/box/xlog.c > +++ b/src/box/xlog.c > @@ -487,6 +487,15 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > * replication set (see also _cluster system space and vclock.h > * comments). > * > + * @param dir - directory to scan > + * @param is_dir_required - flag set if the directory should exist > + * > + * @return: > + * 0 - on success or flag 'is_dir_required' was set to False in > + * xdir_scan() arguments and opendir() failed with errno ENOENT; > + * -1 - if opendir() failed, with other than ENOENT errno either > + * when 'is_dir_required' was set to True in xdir_scan() arguments. > + * > * This function tries to avoid re-reading a file if > * it is already in the set of files "known" to the log > * dir object. This is done to speed up local hot standby and > @@ -508,16 +517,17 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > * silence conditions such as out of memory or lack of OS > * resources. > * > - * @return nothing. > */ > int > -xdir_scan(struct xdir *dir) > +xdir_scan(struct xdir *dir, bool is_dir_required) > { > DIR *dh = opendir(dir->dirname); /* log dir */ > int64_t *signatures = NULL; /* log file names */ > size_t s_count = 0, s_capacity = 0; > > if (dh == NULL) { > + if (!is_dir_required && errno == ENOENT) > + return 0; > diag_set(SystemError, "error reading directory '%s'", > dir->dirname); > return -1; > diff --git a/src/box/xlog.h b/src/box/xlog.h > index 9ffce598b..5b1f42ce1 100644 > --- a/src/box/xlog.h > +++ b/src/box/xlog.h > @@ -185,9 +185,10 @@ xdir_destroy(struct xdir *dir); > * index with all log files (or snapshots) in the directory. > * Must be used if it is necessary to find the last log/ > * snapshot or scan through all logs. > + * Function arguments described in xlog.c source file. > */ > int > -xdir_scan(struct xdir *dir); > +xdir_scan(struct xdir *dir, bool is_dir_required); > > /** > * Check that a directory exists and is writable. > @@ -821,9 +822,9 @@ xdir_open_cursor(struct xdir *dir, int64_t signature, > #include "exception.h" > > static inline void > -xdir_scan_xc(struct xdir *dir) > +xdir_scan_xc(struct xdir *dir, bool is_dir_required) > { > - if (xdir_scan(dir) == -1) > + if (xdir_scan(dir, is_dir_required) == -1) > diag_raise(); > } > > diff --git a/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > new file mode 100755 > index 000000000..e6dd68d54 > --- /dev/null > +++ b/test/box-tap/gh-4562-errno-at-xdir_scan.test.lua > @@ -0,0 +1,54 @@ > +#!/usr/bin/env tarantool > + > +local tap = require('tap') > +local test = tap.test('cfg') > +local fio = require('fio') > +test:plan(1) > + > +local tarantool_bin = arg[-1] > +local PANIC = 256 > + > +local function run_script(code) > + local dir = fio.tempdir() > + local script_path = fio.pathjoin(dir, 'script.lua') > + local script = fio.open(script_path, {'O_CREAT', 'O_WRONLY', 'O_APPEND'}, > + tonumber('0777', 8)) > + script:write(code) > + script:write("\nos.exit(0)") > + script:close() > + local cmd = [[/bin/sh -c 'cd "%s" && "%s" ./script.lua 2> /dev/null']] > + local res = os.execute(string.format(cmd, dir, tarantool_bin)) > + fio.rmtree(dir) > + return res > +end > + > +-- > +-- gh-4594: when memtx_dir is not exists, but vinyl_dir exists and > +-- errno is set to ENOENT, box configuration succeeds, however it > +-- should not > +-- > + > +-- create vinyl_dir as temporary path > +local vinyl_dir = fio.tempdir() > + > +-- fill vinyl_dir > +run_script(string.format([[ > +box.cfg{vinyl_dir = '%s'} > +s = box.schema.space.create('test', {engine = 'vinyl'}) > +s:create_index('pk') > +os.exit(0) > +]], vinyl_dir)) > + > +-- verify the case described above > +local code = string.format([[ > +local errno = require('errno') > +errno(errno.ENOENT) > +box.cfg{vinyl_dir = '%s'} > +os.exit(0) > +]], vinyl_dir) > +test:is(run_script(code), PANIC, "bootstrap with ENOENT from non-empty vinyl_dir") > + > +-- remove vinyl_dir > +fio.rmtree(vinyl_dir) > + > +os.exit(test:check() and 0 or 1) From imun at tarantool.org Thu Aug 27 21:25:34 2020 From: imun at tarantool.org (Igor Munkin) Date: Thu, 27 Aug 2020 21:25:34 +0300 Subject: [Tarantool-patches] [PATCH v2 2/2] metrics: add C and Lua API In-Reply-To: <97bddbd3a946463b135fb6ae559b0d72d4f2148c.1595794764.git.skaplun@tarantool.org> References: <97bddbd3a946463b135fb6ae559b0d72d4f2148c.1595794764.git.skaplun@tarantool.org> Message-ID: <20200827182534.GB18920@tarantool.org> Sergey, Thanks for the patch! Please consider my comments below. On 26.07.20, Sergey Kaplun wrote: > This patch adds C and Lua API for luajit metrics. Metrics include GC > statistic, JIT information, strhash hit/miss counters and amount of > different objects. C API provides with aditional header > that contains structure `luam_metrics` and `luaM_metrics` function > definition. Lua userspace expanded with builtin library (named `misc`) > with corresponding method `getmetrics`. Well, let's make the commit message a bit clearer: | This patch introduces both C and Lua API for LuaJIT platform | metrics implemented in scope of the previous patch. New | header provides C interface that fills the given | structure with the platform metrics. Additionally | module is loaded to Lua space and provides | method that yields the corresponding metrics via table. Feel free to change it on your own. > > Part of tarantool/tarantool#5187 > --- This comment relates to all created files: I see no reason to violate LuaJIT code style, so please adjust the sources considering the current practices. > Makefile | 2 +- > src/Makefile | 5 ++-- > src/Makefile.dep | 3 ++ > src/lib_init.c | 2 ++ > src/lib_misc.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++ > src/lj_misc_capi.c | 59 ++++++++++++++++++++++++++++++++++++ > src/lmisclib.h | 61 +++++++++++++++++++++++++++++++++++++ > src/luaconf.h | 1 + > 8 files changed, 205 insertions(+), 3 deletions(-) > create mode 100644 src/lib_misc.c > create mode 100644 src/lj_misc_capi.c > create mode 100644 src/lmisclib.h > > diff --git a/src/Makefile b/src/Makefile > index 827d4a4..fac69bc 100644 > --- a/src/Makefile > +++ b/src/Makefile > @@ -480,13 +480,14 @@ LJVM_BOUT= $(LJVM_S) > LJVM_MODE= elfasm > > LJLIB_O= lib_base.o lib_math.o lib_bit.o lib_string.o lib_table.o \ > - lib_io.o lib_os.o lib_package.o lib_debug.o lib_jit.o lib_ffi.o > + lib_io.o lib_os.o lib_package.o lib_debug.o lib_jit.o lib_ffi.o \ > + lib_misc.o > LJLIB_C= $(LJLIB_O:.o=.c) > > LJCORE_O= lj_gc.o lj_err.o lj_char.o lj_bc.o lj_obj.o lj_buf.o \ > lj_str.o lj_tab.o lj_func.o lj_udata.o lj_meta.o lj_debug.o \ > lj_state.o lj_dispatch.o lj_vmevent.o lj_vmmath.o lj_strscan.o \ > - lj_strfmt.o lj_strfmt_num.o lj_api.o lj_profile.o \ > + lj_strfmt.o lj_strfmt_num.o lj_api.o lj_misc_capi.o lj_profile.o \ Why did you add in the middle of LJCORE_O? > lj_lex.o lj_parse.o lj_bcread.o lj_bcwrite.o lj_load.o \ > lj_ir.o lj_opt_mem.o lj_opt_fold.o lj_opt_narrow.o \ > lj_opt_dce.o lj_opt_loop.o lj_opt_split.o lj_opt_sink.o \ > diff --git a/src/Makefile.dep b/src/Makefile.dep > index 2b1cb5e..1c3d8bd 100644 > --- a/src/Makefile.dep > +++ b/src/Makefile.dep lmisclib.h is missing in lib_init.o deplist. > @@ -41,6 +41,9 @@ lib_string.o: lib_string.c lua.h luaconf.h lauxlib.h lualib.h lj_obj.h \ > lib_table.o: lib_table.c lua.h luaconf.h lauxlib.h lualib.h lj_obj.h \ > lj_def.h lj_arch.h lj_gc.h lj_err.h lj_errmsg.h lj_buf.h lj_str.h \ > lj_tab.h lj_ff.h lj_ffdef.h lj_lib.h lj_libdef.h Object file list is sorted, so why do you violate this order? > +lib_misc.o: lib_misc.c lua.h lmisclib.h luaconf.h lj_obj.h lj_tab.h lj_str.h \ > + lj_arch.h lj_lib.h lj_vm.h lj_libdef.h Minor: luaconf.h is required for lua.h, not for lmisclib.h, so the order should be the same as in other lib_*.o targets. This comment also relates to lj_arch.h. lj_def.h is missing and lj_vm.h is excess. > +lj_misc_capi.o: lj_misc_capi.c lua.h lmisclib.h luaconf.h lj_obj.h lj_arch.h lj_def.h, lj_jit.h and lj_dispatch.h are missing. > lj_alloc.o: lj_alloc.c lj_def.h lua.h luaconf.h lj_arch.h lj_alloc.h > lj_api.o: lj_api.c lj_obj.h lua.h luaconf.h lj_def.h lj_arch.h lj_gc.h \ > lj_err.h lj_errmsg.h lj_debug.h lj_str.h lj_tab.h lj_func.h lj_udata.h \ > diff --git a/src/lib_misc.c b/src/lib_misc.c > new file mode 100644 > index 0000000..ef20921 > --- /dev/null > +++ b/src/lib_misc.c > @@ -0,0 +1,75 @@ > +/* > + * Lua interface to tarantool-specific extensions to the public Lua/C API. It relates neither to Lua/C API nor to Tarantool. This translation unit simply provides miscellaneous builtin functions extending Lua Reference manual. > + * > + * Major portions taken verbatim or adapted from the LuaVela interpreter. > + * Copyright (C) 2015-2019 IPONWEB Ltd. > + */ > + > +#define lib_misc_c > +#define LUA_LIB > + > +#include "lua.h" > +#include "lmisclib.h" > + > +#include "lj_obj.h" > +#include "lj_str.h" > +#include "lj_tab.h" > +#include "lj_ff.h" This include seems to be excess. > +#include "lj_lib.h" > + > +/* ------------------------------------------------------------------------ */ > + > +static LJ_AINLINE void setnumfield(struct lua_State *L, GCtab *t, > + const char *name, int64_t val) > +{ > + GCstr *key = lj_str_new(L, name, strlen(name)); looks more convenient here. Consider the following: | setnumV(lj_tab_setstr(L, t, lj_str_newz(L, name), (double)val)); > + setnumV(lj_tab_setstr(L, t, key), (double)val); Well, I've already mentioned my concerns while discussing offline the first series. Since I see no corresponding changes, I leave them in this reply. User will face precision loss if any counter exceeds 1 << 53. Consider the following example: | $ luajit -e 'print(2^53 == 2^53 + 1)' | true I guess some counters (e.g. gc_freed and gc_allocated) should be cdata 64-bit numbers instead of the Lua (i.e. doubles) ones. Thoughts? > +} > + > +#define LJLIB_MODULE_misc > + > +LJLIB_CF(misc_getmetrics) > +{ > + lua_createtable(L, 0, 19); > + GCtab *m = tabV(L->top - 1); > + > + struct luam_Metrics metrics; Please, declare local variables in the beginning of the block. > + luaM_metrics(L, &metrics); > + > + setnumfield(L, m, "strnum", metrics.strnum); > + setnumfield(L, m, "tabnum", metrics.tabnum); > + setnumfield(L, m, "udatanum", metrics.udatanum); > + setnumfield(L, m, "cdatanum", metrics.cdatanum); > + > + setnumfield(L, m, "gc_total", metrics.gc_total); > + > + setnumfield(L, m, "jit_mcode_size", metrics.jit_mcode_size); > + setnumfield(L, m, "jit_trace_num", metrics.jit_trace_num); This ordering looks odd. It differs from structure fields ordering. It also differs from the order fields are filled within . It's simply a mess, please adjust the ordering to a single way (the one from structure definition is fine). Furthermore, I don't get the rule you split blocks with whitespace... > + > + setnumfield(L, m, "gc_freed", metrics.gc_freed); > + setnumfield(L, m, "gc_allocated", metrics.gc_allocated); > + > + setnumfield(L, m, "gc_steps_pause", metrics.gc_steps_pause); > + setnumfield(L, m, "gc_steps_propagate", metrics.gc_steps_propagate); > + setnumfield(L, m, "gc_steps_atomic", metrics.gc_steps_atomic); > + setnumfield(L, m, "gc_steps_sweepstring", metrics.gc_steps_sweepstring); > + setnumfield(L, m, "gc_steps_sweep", metrics.gc_steps_sweep); > + setnumfield(L, m, "gc_steps_finalize", metrics.gc_steps_finalize); > + > + setnumfield(L, m, "jit_snap_restore", metrics.jit_snap_restore); > + setnumfield(L, m, "jit_trace_abort", metrics.jit_trace_abort); > + > + setnumfield(L, m, "strhash_hit", metrics.strhash_hit); > + setnumfield(L, m, "strhash_miss", metrics.strhash_miss); > + return 1; > +} > + > +/* ------------------------------------------------------------------------ */ > + > +#include "lj_libdef.h" > + > +LUALIB_API int luaopen_misc(struct lua_State *L) > +{ > + LJ_LIB_REG(L, LUAM_MISCLIBNAME, misc); > + return 1; > +} > diff --git a/src/lj_misc_capi.c b/src/lj_misc_capi.c Minor: fits better to the current LJ naming, doesn't it? > new file mode 100644 > index 0000000..5ae98b9 > --- /dev/null > +++ b/src/lj_misc_capi.c > @@ -0,0 +1,59 @@ > +/* > + * Tarantool-specific extensions to the public Lua/C API. Again, these interfaces doesn't relate to Tarantool. > + * > + * Major portions taken verbatim or adapted from the LuaVela. > + * Copyright (C) 2015-2019 IPONWEB Ltd. > + */ > + > +#include "lua.h" > +#include "lmisclib.h" > + > +#include "lj_obj.h" > +#include "lj_gc.h" This include seems to be excess. > +#include "lj_dispatch.h" > + > +#if LJ_HASJIT > +#include "lj_jit.h" > +#endif > + > +LUAM_API struct luam_Metrics * OK, I guess is enough here. > +luaM_metrics(lua_State *L, struct luam_Metrics *dst) Since there is no "src", just "metrics" are fine. > +{ I see not a single word or check for the case when dst is NULL. If you forbid NULL argument with the function contract, please add the corresponding assert. Otherwise the behaviour should be adjusted to it. > + memset(dst, 0, sizeof(*dst)); I was misguided with this , it looked excess. Later, I realized that the fields in the resulting structure are present despite the build flags, but their initialization respects the way LuaJIT is built. It's definitely worth to mention this fact with the comment nearby. > + global_State *g = G(L); > + GCState *gc = &g->gc; > +#if LJ_HASJIT > + jit_State *J = G2J(g); > +#endif > + > + dst->strhash_hit = g->strhash_hit; > + dst->strhash_miss = g->strhash_miss; > + > + dst->strnum = g->strnum; > + dst->tabnum = gc->tabnum; > + dst->udatanum = gc->udatanum; > +#if LJ_HASFFI > + dst->cdatanum = gc->cdatanum; > +#endif > + > + dst->gc_total = gc->total; > + dst->gc_freed = gc->freed; > + dst->gc_allocated = gc->allocated; > + > + dst->gc_steps_pause = gc->state_count[GCSpause]; > + dst->gc_steps_propagate = gc->state_count[GCSpropagate]; > + dst->gc_steps_atomic = gc->state_count[GCSatomic]; > + dst->gc_steps_sweepstring = gc->state_count[GCSsweepstring]; > + dst->gc_steps_sweep = gc->state_count[GCSsweep]; > + dst->gc_steps_finalize = gc->state_count[GCSfinalize]; > + > +#if LJ_HASJIT > + dst->jit_snap_restore = J->nsnaprestore; > + dst->jit_trace_abort = J->ntraceabort; > + Minor: Still don't get the rule you split blocks with whitespace... > + dst->jit_mcode_size = J->szallmcarea; > + dst->jit_trace_num = J->freetrace; > +#endif > + > + return dst; > +} > diff --git a/src/lmisclib.h b/src/lmisclib.h > new file mode 100644 > index 0000000..87423c1 > --- /dev/null > +++ b/src/lmisclib.h > @@ -0,0 +1,61 @@ > +/* Meh, you've written at least one sentence in the new LJ private sources, but there is not a word here (in the public header, to be distributed). I guess, this is the most important new file to be described. > + * Major portions taken verbatim or adapted from the LuaVela. > + * Copyright (C) 2015-2019 IPONWEB Ltd. > + */ > + > +#ifndef _LMISCLIB_H_INCLUDED > +#define _LMISCLIB_H_INCLUDED > + > +#include "lua.h" > + > +/* API for obtaining various metrics from the platform. */ > + > +struct luam_Metrics { > + /* > + * Strings amount founded in string hash Typo: s/founded/found/. > + * instead of allocation of new one. > + */ > + size_t strhash_hit; > + /* Strings amount allocated and put into string hash. */ > + size_t strhash_miss; > + > + size_t strnum; /* Amount of allocated string objects. */ > + size_t tabnum; /* Amount of allocated table objects. */ > + size_t udatanum; /* Amount of allocated udata objects. */ > + size_t cdatanum; /* Amount of allocated cdata objects. */ Why do you mix inline comments with those going prior to the field? Please choose a single way to write comments here. > + > + /* Memory currently allocated. */ > + size_t gc_total; > + /* Total amount of freed memory. */ > + size_t gc_freed; > + /* Total amount of allocated memory. */ > + size_t gc_allocated; > + > + /* Count of incremental GC steps per state. */ > + size_t gc_steps_pause; > + size_t gc_steps_propagate; > + size_t gc_steps_atomic; > + size_t gc_steps_sweepstring; > + size_t gc_steps_sweep; > + size_t gc_steps_finalize; > + > + /* > + * Overall number of snap restores (and number of stopped Strictly saying this is not true, since execution can leave the trace without restoring the snapshot (via ). > + * trace executions) for given jit_State. A mess again: you mention the fact snap restores are counted for the "given jit_State", *but* the overall traces amount is not? This remark looks irrelevant and a bit misguiding. I believe it can be dropped. > + */ > + size_t jit_snap_restore; > + /* Overall number of abort traces for given jit_State. */ > + size_t jit_trace_abort; > + /* Total size of all allocated machine code areas. */ > + size_t jit_mcode_size; > + /* Amount of JIT traces. */ > + unsigned int jit_trace_num; > +}; > + > +LUAM_API struct luam_Metrics *luaM_metrics(lua_State *L, > + struct luam_Metrics *dst); > + > +#define LUAM_MISCLIBNAME "misc" > +LUALIB_API int luaopen_misc(lua_State *L); > + > +#endif /* _LMISCLIB_H_INCLUDED */ > diff --git a/src/luaconf.h b/src/luaconf.h > index 60cb928..cf01e36 100644 > --- a/src/luaconf.h > +++ b/src/luaconf.h > @@ -144,6 +144,7 @@ > #endif > > #define LUALIB_API LUA_API > +#define LUAM_API LUA_API Minor: LUAMISC_API name looks more suitable to me (it visually fits LUALIB_API). LUAM_API would be fine if LUALIB_API was named as LUAL_API. > > /* Support for internal assertions. */ > #if defined(LUA_USE_ASSERT) || defined(LUA_USE_APICHECK) > -- > 2.24.1 > And last but not least: what about tests? -- Best regards, IM From imun at tarantool.org Thu Aug 27 21:42:19 2020 From: imun at tarantool.org (Igor Munkin) Date: Thu, 27 Aug 2020 21:42:19 +0300 Subject: [Tarantool-patches] [PATCH v2 1/2] core: introduce various platform metrics In-Reply-To: <20200826155207.GA49@tarantool.org> References: <35a19def79a9cbc46dabdfa579869af9e4e589fb.1595794764.git.skaplun@tarantool.org> <20200826144837.GA18920@tarantool.org> <20200826155207.GA49@tarantool.org> Message-ID: <20200827184219.GC18920@tarantool.org> Sergos, On 26.08.20, Sergey Ostanevich wrote: > On 26 ??? 17:48, Igor Munkin wrote: > > > > Furthermore, please address the comments I left regarding the patch > > you've made for CNEW IR[1] and squash it with this one. > > > > Sergos, do we need other JIT architectures to be patched in scope of > > this series or Sergey can just add the corresponding preprocessor > > condition to stub the issue for now? > > AFAU the CNEW IR appeared in src/lj_asm_x86.h so I didn't get what > preprocessor condition do you mean. Well, I clarify my point a bit: Sergey can implement cdata counter only for x86 platform both in compiler and interpreter. Other platforms will report nothing (i.e. zeros) for this metric and the issue can be fixed on demand. > Also, the inconsistency with the allocated CDATA counter will appear > anyways, so it needs to be implemented for other targets also. Looks > like a not-so-big deal? Yes, but I believe testing will take much more time than implementation. > BTW, AlexanderTi made a successful build and a little less successful > run on an ARM emulator that took some reasonable time. So there's a > way to test the change for ARM. Both PPC and MIPS I suppose would be > just fine to test for successful build. OK then, so I guess Sergey can make the patch for other arches as well. > > > > > > > [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019012.html > > > > -- > > Best regards, > > IM -- Best regards, IM From imun at tarantool.org Thu Aug 27 22:18:37 2020 From: imun at tarantool.org (Igor Munkin) Date: Thu, 27 Aug 2020 22:18:37 +0300 Subject: [Tarantool-patches] [PATCH v2] rfc: luajit metrics In-Reply-To: <20200726204236.20385-1-skaplun@tarantool.org> References: <20200726204236.20385-1-skaplun@tarantool.org> Message-ID: <20200827191837.GD18920@tarantool.org> Sergey, Thanks, this RFC is almost great! Please consider my comments below. On 26.07.20, Sergey Kaplun wrote: > Part of #5187 > --- > > This patch adds RFC to LuaJIT metrics interfaces. Nevertheless name > `misc` for builtin library is not good and should be discussed, because > tons of user modules can use that name for their own libraies. > > Branch: https://github.com/tarantool/tarantool/tree/skaplun/5187-luajit-metrics > Issue: https://github.com/tarantool/tarantool/issues/5187 > > Changes in v2: > - Fixed typos > - Made comments more verbose > - Avoided flushing any of metrics after each call of luaM_metrics() > > doc/rfc/5187-luajit-metrics.md | 126 +++++++++++++++++++++++++++++++++ > 1 file changed, 126 insertions(+) > create mode 100644 doc/rfc/5187-luajit-metrics.md > > diff --git a/doc/rfc/5187-luajit-metrics.md b/doc/rfc/5187-luajit-metrics.md > new file mode 100644 > index 000000000..2bd64cff4 > --- /dev/null > +++ b/doc/rfc/5187-luajit-metrics.md > @@ -0,0 +1,126 @@ > +# LuaJIT metrics > + > +* **Status**: In progress > +* **Start date**: 17-07-2020 > +* **Authors**: Sergey Kaplun @Buristan skaplun at tarantool.org, > + Igor Munkin @igormunkin imun at tarantool.org, > + Sergey Ostanevich @sergos sergos at tarantool.org > +* **Issues**: [#5187](https://github.com/tarantool/tarantool/issues/5187) > + > +## Summary > + > +LuaJIT metrics provide extra information about the Lua state. They consists of Typo: s/consists/consist/. > +GC metrics (overall amount of objects and memory usage), JIT stats (both > +related to the compiled traces and the engine itself), string hash hits/misses. > + > +## Background and motivation > + > +One can be curious about their application performance. We are going to provide > +various metrics about the several platform subsystems behaviour. GC pressure > +produced by user code can weight down all application performance. Irrelevant > +traces compiled by the JIT engine can just burn CPU time with no benefits as a > +result. String hash collisions can lead to DoS caused by a single request. All > +these metrics should be well monitored by users wanting to improve the > +performance of their application. > + > +## Detailed design > + > +For C API we introduce additional extension header that provides > +interfaces for new LuaJIT C API extensions. The first interface in this header > +will be the following: I propose the following rewording: | The additional header is introduced to extend the existing | LuaJIT C API with new interfaces. The first function provided via this | header is the following: > + > +``` > +/* API for obtaining various metrics from the platform. */ Typo: s/metrics from the platform/platform metrics/. > + > +LUAM_API struct luam_Metrics *luaM_metrics(lua_State *L, > + struct luam_Metrics *dst); Please, address the comments I left regarding the function signature here[1]. > +``` > + > +This function fills the structure pointed by `dst` with the corresponding > +metrics related to Lua state anchored to the given coroutine `L`. The result of > +the function is a pointer to the filled structure (the same `dst` points to). > + > +The `struct luam_Metrics` has the following definition: > + > +``` > +struct luam_Metrics { Please, address the comments I left regarding the structure definition here[1]. > + /* > + * Strings amount founded in string hash > + * instead of allocation of new one. > + */ > + size_t strhash_hit; > + /* Strings amount allocated and put into string hash. */ > + size_t strhash_miss; > + > + size_t strnum; /* Amount of allocated string objects. */ > + size_t tabnum; /* Amount of allocated table objects. */ > + size_t udatanum; /* Amount of allocated udata objects. */ > + size_t cdatanum; /* Amount of allocated cdata objects. */ > + > + /* Memory currently allocated. */ > + size_t gc_total; > + /* Total amount of freed memory. */ > + size_t gc_freed; > + /* Total amount of allocated memory. */ > + size_t gc_allocated; > + > + /* Count of incremental GC steps per state. */ > + size_t gc_steps_pause; > + size_t gc_steps_propagate; > + size_t gc_steps_atomic; > + size_t gc_steps_sweepstring; > + size_t gc_steps_sweep; > + size_t gc_steps_finalize; > + > + /* > + * Overall number of snap restores (and number of stopped > + * trace executions) for given jit_State. > + */ > + size_t jit_snap_restore; > + /* Overall number of abort traces for given jit_State. */ > + size_t jit_trace_abort; > + /* Total size of all allocated machine code areas. */ > + size_t jit_mcode_size; > + /* Amount of JIT traces. */ > + unsigned int jit_trace_num; > +}; > +``` > + > +All metrics are collected throughout the platform uptime. But some of them > +(namely `strhash_hit`, `strhash_miss`, `gc_freed`, `gc_allocated`, > +`gc_steps_pause`, `gc_steps_propagate`, `gc_steps_atomic`, > +`gc_steps_sweepstring`, `gc_steps_sweep`, `gc_steps_finalize`, > +`jit_snap_restore` and `jit_trace_abort`) increase monotonic and can overflow. Ouch, let's list these metrics in a bullet list for better readability. Typo: s/monotonic/monotonically/. > +They make sense only with comparing with their value from a previous > +`luaM_metrics()` call. > + > +There is also a complement introduced for Lua space -- `misc.getmetrics()`. > +This function is just a wrapper for `luaM_metrics()` returning a Lua table with > +the similar metrics. Its usage is quite simple: > +``` > +$ ./src/tarantool > +Tarantool 2.5.0-267-gbf047ad44 > +type 'help' for interactive help > +tarantool> misc.getmetrics() > +--- > +- tabnum: 1812 > + gc_total: 1369927 > + strnum: 5767 > + jit_trace_num: 0 > + cdatanum: 89 > + jit_mcode_size: 0 > + udatanum: 17 > + jit_snap_restore: 0 > + gc_freed: 2239391 > + strhash_hit: 53759 > + gc_steps_finalize: 0 > + gc_allocated: 3609318 > + gc_steps_atomic: 6 > + gc_steps_sweep: 296 > + gc_steps_sweepstring: 17920 > + jit_trace_abort: 0 > + strhash_miss: 6874 > + gc_steps_propagate: 10106 > + gc_steps_pause: 7 > +... > +``` > -- > 2.24.1 > Otherwise, LGTM. [1]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019208.html -- Best regards, IM From v.shpilevoy at tarantool.org Thu Aug 27 23:36:07 2020 From: v.shpilevoy at tarantool.org (Vladislav Shpilevoy) Date: Thu, 27 Aug 2020 22:36:07 +0200 Subject: [Tarantool-patches] [RAFT 02/10] raft: relay status updates to followers In-Reply-To: References: Message-ID: <82f89c29-2cce-4a5d-9698-0a240f4d263c@tarantool.org> Hi! Thanks for the patch! > diff --git a/src/box/relay.cc b/src/box/relay.cc > index a7843a8c2..be252cad1 100644 > --- a/src/box/relay.cc > +++ b/src/box/relay.cc > @@ -773,13 +774,40 @@ relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row) > relay_send(relay, row); > } > > +static void > +relay_send_raft(struct relay *relay, struct raft_request *req) > +{ > + struct xrow_header packet; > + xrow_encode_raft(&packet, &fiber()->gc, req); > + relay_send(relay, &packet); > +} > + > +static void > +relay_send_raft_msg(struct cmsg *msg) > +{ > + struct raft_broadcast_msg *raft_msg = (struct raft_broadcast_msg *)msg; > + struct relay *relay = container_of(msg->route[0].pipe, struct relay, > + tx_pipe); > + relay_send_raft(relay, &raft_msg->req); > +} > + > +void > +relay_push_raft_msg(struct relay *relay, struct cmsg *msg, > + struct cmsg_hop *route) > +{ > + route[0].f = relay_send_raft_msg; > + route[0].pipe = &relay->tx_pipe; > + route[1].f = raft_free_msg; > + route[1].pipe = NULL; > + cmsg_init(msg, route); > + cpipe_push(&relay->relay_pipe, msg); > +} > + > /** Send a single row to the client. */ > static void > relay_send_row(struct xstream *stream, struct xrow_header *packet) > { > struct relay *relay = container_of(stream, struct relay, stream); > - assert(iproto_type_is_dml(packet->type) || > - iproto_type_is_synchro_request(packet->type)); > if (packet->group_id == GROUP_LOCAL) { > /* > * We do not relay replica-local rows to other > @@ -796,6 +824,8 @@ relay_send_row(struct xstream *stream, struct xrow_header *packet) > packet->group_id = GROUP_DEFAULT; > packet->bodycnt = 0; > } > + assert(iproto_type_is_dml(packet->type) || > + iproto_type_is_synchro_request(packet->type)); Why did you move this check, if Raft uses relay_send() anyway? Not relay_send_row(). > /* Check if the rows from the instance are filtered. */ > if ((1 << packet->replica_id & relay->id_filter) != 0) > return; From huston.mavr at gmail.com Fri Aug 28 00:43:56 2020 From: huston.mavr at gmail.com (Alexandr Barulev) Date: Fri, 28 Aug 2020 00:43:56 +0300 Subject: [Tarantool-patches] [PATCH v2] build: refactor static build process In-Reply-To: <20200825145103.hiqmjkimuk3al3rt@tkn_work_nb> References: <20200825140108.52090-1-huston.mavr@gmail.com> <20200825145103.hiqmjkimuk3al3rt@tkn_work_nb> Message-ID: Hello, thanks for the new iteration of review! I deleted unused box.lua script and reference to it in suite.ini at static-build tests. I added comments about explicitly setting compilers at static-build/CMakeLists.txt. Also I fixed building libncurses/libcurses for correct work of FindCurses.cmake module. After this change there is no need to set CURSES_NEED_NCURSES at cmake/FindReadline.cmake, so I checkouted cmake/FindReadline.cmake from master and update patch commit massage. And fixed .travis.mk static_build_cmake_* jobs - build tarantool with -DCMAKE_BUILD_TYPE=RelWithDebInfo cmake option. Here is a new commit message (deleted part about CURSES_NEED_NCURSES): build: refactor static build process Refactored static build process to use static-build/CMakeLists.txt instead of Dockerfile.staticbuild (this allows to support static build on macOS). Following third-party dependencies for static build are installed via cmake `ExternalProject_Add`: - OpenSSL - Zlib - Ncurses - Readline - Unwind - ICU * Added support static build for macOS * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for staic build at file cmake/BuildLibCURL.cmake: - disable building shared libcurl libraries (by setting `--disable-shared` option) - disable hiding libcurl symbols (by setting `--disable-symbol-hiding` option) - prevent linking libcurl with system libz by settign `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option) * Removed Dockerfile.staticbuild * Added new gitlab.ci jobs to test new style static build: - static_build_cmake_linux - static_build_cmake_osx_15 * Removed static_docker_build gitlab.ci job Closes #5095 Here is a link to branch: https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build And here is a diff: diff --git a/.travis.mk b/.travis.mk index 482672429..ccd9d6db1 100644 --- a/.travis.mk +++ b/.travis.mk @@ -151,7 +151,8 @@ test_static_build: deps_debian_static # New static build # builddir used in this target - is a default build path from cmake ExternalProject_Add() test_static_build_cmake_linux: - cd static-build && cmake . && make -j && ctest -V + cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \ + make -j && ctest -V cd test && /usr/bin/python test-run.py --force \ --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build $(TEST_RUN_EXTRA_PARAMS) @@ -234,7 +235,8 @@ base_deps_osx: # builddir used in this target - is a default build path from cmake ExternalProject_Add() test_static_build_cmake_osx: base_deps_osx - cd static-build && cmake . && make -j && ctest -V + cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \ + make -j && ctest -V ${INIT_TEST_ENV_OSX}; \ cd test && ./test-run.py --vardir /tmp/tnt \ --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake index afe480679..c48bdcb3e 100644 --- a/cmake/FindReadline.cmake +++ b/cmake/FindReadline.cmake @@ -14,17 +14,7 @@ if(BUILD_STATIC) if (NOT CURSES_INFO_LIBRARY) set(CURSES_INFO_LIBRARY "") endif() - - # From Modules/FindCurses.cmake: - # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the - # ``find_package(Curses)`` call if NCurses functionality is required. - # This flag is set for linking with required library (installed - # via static-build/CMakeLists.txt). If this variable won't be set - # then tarantool binary links with system library curses which is an - # entire copy of ncurses - set(CURSES_NEED_NCURSES TRUE) endif() - find_package(Curses) if(NOT CURSES_FOUND) find_package(Termcap) diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt index d90a642e6..ecfdd0455 100644 --- a/static-build/CMakeLists.txt +++ b/static-build/CMakeLists.txt @@ -9,6 +9,12 @@ set(NCURSES_VERSION 6.2) set(READLINE_VERSION 8.0) set(UNWIND_VERSION 1.3-rc1) +# Set compilers explicitly for further configuring dependencies with +# these compilers. This gonna solve libicu building problem in case when +# at dependency configure stage no compiler specified and clang compiler +# exists on linux machine, libicu sources would be compiled with clang +# while for other dependencies (including tarantool) gcc would be used. +# This behaviour causes problem at tarantool linkage stage. if (APPLE) find_program(C_COMPILER clang) find_program(CXX_COMPILER clang++) @@ -70,6 +76,15 @@ ExternalProject_Add(ncurses CXX=${CMAKE_CXX_COMPILER} /configure --prefix= + + # This flag enables creation of libcurses.a as a symlink to libncurses.a + # and disables subdir creation `ncurses` at /include. It is + # necessary for correct work of FindCurses.cmake module (this module is + # builtin at cmake package) which used in cmake/FindReadline.cmake + --enable-overwrite + --with-termlib # enable building libtinfo to prevent linking + # with libtinfo from system directories + INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install.libs ) # diff --git a/static-build/test/static-build/box.lua b/static-build/test/static-build/box.lua deleted file mode 100755 index bad8a9055..000000000 --- a/static-build/test/static-build/box.lua +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env tarantool - -require('console').listen(os.getenv('ADMIN')) diff --git a/static-build/test/static-build/suite.ini b/static-build/test/static-build/suite.ini index 92e349466..5aabadd92 100644 --- a/static-build/test/static-build/suite.ini +++ b/static-build/test/static-build/suite.ini @@ -1,5 +1,4 @@ [default] core = app description = Static build tests -script = box.lua is_parallel = True ??, 25 ???. 2020 ?. ? 17:51, Alexander Turenko < alexander.turenko at tarantool.org>: > I looked very briefly (not thoroughly at all) on this iteration. > > There is nothing that confuses me (except few tiny comments below). > > I hope Igor will do thorough review. > > WBR, Alexander Turenko. > > > +if (APPLE) > > + find_program(C_COMPILER clang) > > + find_program(CXX_COMPILER clang++) > > +else() > > + find_program(C_COMPILER gcc) > > + find_program(CXX_COMPILER g++) > > +endif() > > Can we just leave it default? > > In offline discussion Alexandr B. said that tarantool builds with gcc, > but icu with clang that gives some problem. > > Possible solution is to pass ${CMAKE_C_COMPILER} (and CXX too where > necessary) to a subproject as we do for c-ares and curl. It seems it is > already done, so maybe it worth to re-check whether it solves the > problem. > > Anyway, if we really need to set a compiler here explicitly, I don't > mind. Just noted that this way is somewhat unusual as I see. > > > diff --git a/static-build/test/static-build/box.lua > b/static-build/test/static-build/box.lua > > new file mode 100755 > > index 000000000..bad8a9055 > > --- /dev/null > > +++ b/static-build/test/static-build/box.lua > > @@ -0,0 +1,3 @@ > > +#!/usr/bin/env tarantool > > + > > +require('console').listen(os.getenv('ADMIN')) > > Is looks redundant, see the comment below. > > > diff --git a/static-build/test/static-build/suite.ini > b/static-build/test/static-build/suite.ini > > new file mode 100644 > > index 000000000..92e349466 > > --- /dev/null > > +++ b/static-build/test/static-build/suite.ini > > @@ -0,0 +1,5 @@ > > +[default] > > +core = app > > +description = Static build tests > > +script = box.lua > > +is_parallel = True > > 'script' does not have sense for 'core = app' test, it is for 'core = > tarantool' tests. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergepetrenko at tarantool.org Fri Aug 28 13:10:59 2020 From: sergepetrenko at tarantool.org (Sergey Petrenko) Date: Fri, 28 Aug 2020 13:10:59 +0300 Subject: [Tarantool-patches] [RAFT 02/10] raft: relay status updates to followers In-Reply-To: <82f89c29-2cce-4a5d-9698-0a240f4d263c@tarantool.org> References: <82f89c29-2cce-4a5d-9698-0a240f4d263c@tarantool.org> Message-ID: <93f8a2ab-165e-09c3-f23c-effaacf6e0bf@tarantool.org> 27.08.2020 23:36, Vladislav Shpilevoy ?????: > Hi! Thanks for the patch! Hi! Thanks for the review! >> diff --git a/src/box/relay.cc b/src/box/relay.cc >> index a7843a8c2..be252cad1 100644 >> --- a/src/box/relay.cc >> +++ b/src/box/relay.cc >> @@ -773,13 +774,40 @@ relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row) >> relay_send(relay, row); >> } >> >> +static void >> +relay_send_raft(struct relay *relay, struct raft_request *req) >> +{ >> + struct xrow_header packet; >> + xrow_encode_raft(&packet, &fiber()->gc, req); >> + relay_send(relay, &packet); >> +} >> + >> +static void >> +relay_send_raft_msg(struct cmsg *msg) >> +{ >> + struct raft_broadcast_msg *raft_msg = (struct raft_broadcast_msg *)msg; >> + struct relay *relay = container_of(msg->route[0].pipe, struct relay, >> + tx_pipe); >> + relay_send_raft(relay, &raft_msg->req); >> +} >> + >> +void >> +relay_push_raft_msg(struct relay *relay, struct cmsg *msg, >> + struct cmsg_hop *route) >> +{ >> + route[0].f = relay_send_raft_msg; >> + route[0].pipe = &relay->tx_pipe; >> + route[1].f = raft_free_msg; >> + route[1].pipe = NULL; >> + cmsg_init(msg, route); >> + cpipe_push(&relay->relay_pipe, msg); >> +} >> + >> /** Send a single row to the client. */ >> static void >> relay_send_row(struct xstream *stream, struct xrow_header *packet) >> { >> struct relay *relay = container_of(stream, struct relay, stream); >> - assert(iproto_type_is_dml(packet->type) || >> - iproto_type_is_synchro_request(packet->type)); >> if (packet->group_id == GROUP_LOCAL) { >> /* >> * We do not relay replica-local rows to other >> @@ -796,6 +824,8 @@ relay_send_row(struct xstream *stream, struct xrow_header *packet) >> packet->group_id = GROUP_DEFAULT; >> packet->bodycnt = 0; >> } >> + assert(iproto_type_is_dml(packet->type) || >> + iproto_type_is_synchro_request(packet->type)); > Why did you move this check, if Raft uses relay_send() anyway? Not > relay_send_row(). When relay sends a WAL to the replica, local raft rows end up in relay_send_row(). The rows are not sent, since they're local, but they fail this assertion, so I moved it below the locality check. > >> /* Check if the rows from the instance are filtered. */ >> if ((1 << packet->replica_id & relay->id_filter) != 0) >> return; From avtikhon at tarantool.org Mon Aug 31 11:15:06 2020 From: avtikhon at tarantool.org (Alexander V. Tikhonov) Date: Mon, 31 Aug 2020 11:15:06 +0300 Subject: [Tarantool-patches] [PATCH v1] test: fix flaky box/on_shutdown.test.lua on asan Message-ID: Found that box/on_shutdown.test.lua test fails on asan build with: 2020-08-26 09:04:06.750 [42629] main/102/on_shutdown [string "_ = box.ctl.on_shutdown(function() log.warn("..."]:1 W> on_shutdown 5 Starting instance proxy... Run console at unix/:/tnt/test/var/001_box/proxy.control Start failed: builtin/box/console.lua:865: failed to create server unix/:/tnt/test/var/001_box/proxy.control: Address already in use It happened on ASAN build, because server stop routine test-run/lib/preprocessor.py:TestState.server_stop() -> test-run/lib/tarantool_server.py:TarantoolServer.stop() needs to free the proxy.control socket created by test-run/lib/preprocessor.py:TestState.server_start() -> tarantoolctl:process_local() On some builds like ASAN server stop routine needs more time to free the 'proxy.control' socket. So instead of time delay before server restart need to use garbage collector to be sure that it will be freed. Closes #5260 Part of #4360 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-5260-on-shutdown-test Issue: https://github.com/tarantool/tarantool/issues/5260 Issue: https://github.com/tarantool/tarantool/issues/4360 test/box/on_shutdown.result | 7 ++++++- test/box/on_shutdown.skipcond | 7 ------- test/box/on_shutdown.test.lua | 6 +++++- 3 files changed, 11 insertions(+), 9 deletions(-) delete mode 100644 test/box/on_shutdown.skipcond diff --git a/test/box/on_shutdown.result b/test/box/on_shutdown.result index ccbdf45cb..cf5c36c8d 100644 --- a/test/box/on_shutdown.result +++ b/test/box/on_shutdown.result @@ -133,8 +133,13 @@ test_run:eval("test", "_ = fiber.new(function() os.exit() while true do end end) --- - [] ... -fiber.sleep(0.1) +-- On some builds like ASAN server stop routine needs more time to free +-- the 'proxy.control' socket created by tarantoolctl:process_local(), +-- check gh-5260. So instead of time delay before server restart need +-- to use garbage collector to be sure that it will be freed. +collectgarbage('collect') --- +- 0 ... -- The server should be already stopped by os.exit(), -- but start doesn't work without a prior call to stop. diff --git a/test/box/on_shutdown.skipcond b/test/box/on_shutdown.skipcond deleted file mode 100644 index e46fd1088..000000000 --- a/test/box/on_shutdown.skipcond +++ /dev/null @@ -1,7 +0,0 @@ -import os - -# Disabled at ASAN build due to issue #4360. -if os.getenv("ASAN") == 'ON': - self.skip = 1 - -# vim: set ft=python: diff --git a/test/box/on_shutdown.test.lua b/test/box/on_shutdown.test.lua index 2a9143404..5437443b3 100644 --- a/test/box/on_shutdown.test.lua +++ b/test/box/on_shutdown.test.lua @@ -54,7 +54,11 @@ test_run:cmd("switch default") -- instance to make sure test_run doesn't lose connection to the -- shutting down instance. test_run:eval("test", "_ = fiber.new(function() os.exit() while true do end end)") -fiber.sleep(0.1) +-- On some builds like ASAN server stop routine needs more time to free +-- the 'proxy.control' socket created by tarantoolctl:process_local(), +-- check gh-5260. So instead of time delay before server restart need +-- to use garbage collector to be sure that it will be freed. +collectgarbage('collect') -- The server should be already stopped by os.exit(), -- but start doesn't work without a prior call to stop. test_run:cmd("stop server test") -- 2.17.1 From kyukhin at tarantool.org Mon Aug 31 12:51:32 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Mon, 31 Aug 2020 12:51:32 +0300 Subject: [Tarantool-patches] [PATCH v2] vinyl: fix check vinyl_dir existence at bootstrap In-Reply-To: <77367c061961d33a6fd11325f4c6abb443ffb15a.1598537487.git.avtikhon@tarantool.org> References: <77367c061961d33a6fd11325f4c6abb443ffb15a.1598537487.git.avtikhon@tarantool.org> Message-ID: <20200831095132.q2afqorjvhpby6de@tarantool.org> Hello, On 27 ??? 17:12, Alexander V. Tikhonov wrote: > During implementation of openSUSE build with testing got failed test > box-tap/cfg.test.lua. Found that when memtx_dir didn't exist and > vinyl_dir existed and also errno was set to ENOENT, box configuration > succeeded, but it shouldn't. Reason of this wrong behavior was that > not all of the failure paths in xdir_scan() set errno, but the caller > assumed it. > > Debugging the issue found that after xdir_scan() there was incorrect > check for errno when it returned negative values. xdir_scan() is not > system call and negative return value from it doesn't mean that errno > would be set too. Found that in situations when errno was left from > previous commands before xdir_scan() and xdir_scan() returned negative > value by itself it produced the wrong check. > > The previous failed logic of the check was to catch the error ENOENT > which set in the xdir_scan() function to handle the situation when > vinyl_dir was not exist. It failed, because checking ENOENT outside > the xdir_scan() function, we had to be sure that ENOENT had come from > xdir_scan() function call indeed and not from any other functions > before. To be sure in it possible fix could be reset errno before > xdir_scan() call, because errno could be passed from any other function > before call to xdir_scan(). > > As mentioned above xdir_scan() function is not system call and can be > changed in any possible way and it can return any result value without > need to setup errno. So check outside of this function on errno could > be broken. > > To avoid that we must not check errno after call of the function. > Better solution is to use the flag in xdir_scan(), to check if the > directory should exist. So errno check was removed and instead of it > the check for vinyl_dir existence using flag added. > > Closes #4594 > Needed for #4562 > > Co-authored-by: Alexander Turenko > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-vinyl-fix > Issue: https://github.com/tarantool/tarantool/issues/4594 > Issue: https://github.com/tarantool/tarantool/issues/4562 > Review: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019130.html LGTM. I've checked your patch into 1.10, 2.4, 2.5 and master. -- Regards, Kirill Yukhin From sergeyb at tarantool.org Mon Aug 31 13:05:39 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Mon, 31 Aug 2020 13:05:39 +0300 Subject: [Tarantool-patches] [PATCH 3/3] replication: add test with random leaders promotion and demotion In-Reply-To: <1598525382.271317018@f511.i.mail.ru> References: <3c7b2274-8443-2be7-c181-2c7026ab0fec@tarantool.org> <20200826144538.GC47610@pony.bronevichok.ru> <1598525382.271317018@f511.i.mail.ru> Message-ID: <20200831100539.GA94343@pony.bronevichok.ru> Sergey, thanks for review! See my comments inline, test updated in a branch. On 13:49 Thu 27 Aug , Serge Petrenko wrote: > > Hi! Thanks for the fixes! > I?m pasting parts of the patch below to comment on. > > +box.cfg({ > +? ? listen = instance_uri(INSTANCE_ID); > +? ? replication_connect_quorum = 3; > +? ? replication = { > +? ? ? ? instance_uri(1); > +? ? ? ? instance_uri(2); > +? ? ? ? instance_uri(3); > +? ? ? ? instance_uri(4); > +? ? ? ? instance_uri(5); > +? ? }; > +}) > ? > ? > * You should?either omit?`replication_connect_quorum` at all, or set it to 5. > Omitting it will have the same effect. > I think you meant `replication_synchro_quorum` here, then it makes sense > to set it to 3. Also ? `replication_synchro_timeout` should be set here, > I?ll mention it ???????? again below. removed replication_connect_quorum at all > + > +NUM_INSTANCES = 5 > +BROKEN_QUORUM = NUM_INSTANCES + 1 > + > ? > ? > * BROKEN_QUORUM?assigned but never used. removed > ? > + > +test_run:create_cluster(SERVERS, "replication", {args="0.1"}) > +test_run:wait_fullmesh(SERVERS) > ? > ? > * You?re passing some argument?to qsync1, ??qsync5 instances, but you never use ?it. removed argument > ? > +current_leader_id = 1 > +test_run:switch(SERVERS[current_leader_id]) > +box.cfg{replication_synchro_timeout=1} > ? > ? > * You should set `replication_synchro_timeout`?on every instance, not only > on qsync1 so ? you better move this?box.cfg call?to the instance file. > Besides, the timeout should be bigger (much bigger), like Vlad said. > We typically use 30 seconds for various replication timeouts. > It?s fairly?common when a test is stable on your machine, but is flaky on > testing machines. increased timeout up to 1000 and set synchro settings in a qsync.lua file > +test_run:switch('default') > + > +-- Testcase body. > +for i=1,10 do? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? new_leader_id = random(current_leader_id, 1, #SERVERS) ? ? ? ? ? ? ? ? ? ? \ > +? ? test_run:switch(SERVERS[new_leader_id])? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? box.cfg{read_only=false} ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? f1 = fiber.create(function() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? ? ? pcall(box.space.sync:truncate{}) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? end) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > ? > ? > * Why put truncate call in a separate fiber? > > Why use truncate at all? You may just replace all your `insert` calls below > with `replace`, and then truncate won?t be needed. This is up to you > though. As far as I remember original idea was to cleanup before next insert. I rewrote a body of test and keep all inserted values in a space and after loop check amount of inserted values. > ? > +? ? f2 = fiber.create(function() ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? ? ? for i=1,10000 do box.space.sync:insert{i} end? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? end) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > ? > * So you?re testing a case when a leader has some unconfirmed transactions in > limbo and then a leader change happens. Then you need to call > `clear_synchro_queue` on a new leader to wait for?confirmation of old txns. Otherwise > the new leader fails to insert its data, but the test doesn?t show this, because you > don?t check fiber state or `insert()` return values. added clear_synchro_queue() call to updated test > +? ? test_run:switch('default') ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? test_run:switch(SERVERS[current_leader_id])? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > ? > ? > * The 2 lines above are useless. removed in updated test: -- Testcase body. for i=1,100 do \ test_run:eval(SERVERS[current_leader_id], \ "box.cfg{replication_synchro_quorum=6}") \ test_run:eval(SERVERS[current_leader_id], \ string.format("box.space.sync:insert{%d}", i)) \ new_leader_id = random(current_leader_id, 1, #SERVERS) \ test_run:eval(SERVERS[new_leader_id], \ "box.cfg{replication_synchro_quorum=3}") \ test_run:eval(SERVERS[new_leader_id], "box.ctl.clear_synchro_queue()") \ replica = random(new_leader_id, 1, #SERVERS) \ test_run:eval(SERVERS[replica], \ string.format("box.space.sync:get{%d}", i)) \ test_run:switch('default') \ current_leader_id = new_leader_id \ end test_run:switch('qsync1') box.space.sync:count() -- 100 > ? > +? ? box.cfg{read_only=true}? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? test_run:switch('default') ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +? ? current_leader_id = new_leader_id? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \ > +end > + > +-- Teardown. > -- > Serge?Petrenko -- sergeyb@ From kyukhin at tarantool.org Mon Aug 31 13:17:47 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Mon, 31 Aug 2020 13:17:47 +0300 Subject: [Tarantool-patches] [PATCH v4 0/2] gitlab-ci: implement openSUSE testing In-Reply-To: References: Message-ID: <20200831101747.vxfnlshnk3eyupw5@tarantool.org> Hello, On 14 ??? 12:59, Alexander V. Tikhonov wrote: > Implement openSUSE testing in gitlab-ci. It needed the following changes > > 1) packpack openSUSE build implementation - commited [3]. > 2) added '--no-undefined' linker flag leading to fails while building tests - commited [1]. > 3) new fix suggestion from A.Turenko with new test [2]. > 4) patch for implementation testing for openSUSE. > > [1]: https://github.com/tarantool/tarantool/commit/f526debcd84ae2d7bdc6c172f9a75d894ecc15dd > [2]: https://lists.tarantool.org/pipermail/tarantool-patches/2020-August/019031.html > [3]: https://github.com/packpack/packpack/pull/121 > > Alexander V. Tikhonov (2): > vinyl: check vinyl_dir existence at bootstrap > gitlab-ci: add openSUSE packages build jobs > > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4562-suse-pack-full-ci > Issue: https://github.com/tarantool/tarantool/issues/4562 > Issue: https://github.com/tarantool/tarantool/issues/4594 I've checked your patch into 1.10, 2.4, 2.5 and master. -- Regards, Kirill Yukhin From i.kosarev at tarantool.org Mon Aug 31 13:59:54 2020 From: i.kosarev at tarantool.org (Ilya Kosarev) Date: Mon, 31 Aug 2020 13:59:54 +0300 Subject: [Tarantool-patches] [PATCH v2] test: concurrent tuple update segfault on bitset index iteration Message-ID: <20200831105954.32408-1-i.kosarev@tarantool.org> Concurrent tuple update could segfault on BITSET_ALL_NOT_SET iterator usage. Fixed in 850054b2dbca257076c3f7c22e00564ac55b70d5. This patch introduces corresponding test. Closes #1088 --- Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-1088-concurrent-tuple-update-segfault-with-bitset-index Issue: https://github.com/tarantool/tarantool/issues/1088 Changes in v2: - put test in separate file ...gh-1088-concurrent-bitset-iteration.result | 94 +++++++++++++++++++ ...-1088-concurrent-bitset-iteration.test.lua | 42 +++++++++ 2 files changed, 136 insertions(+) create mode 100644 test/engine/gh-1088-concurrent-bitset-iteration.result create mode 100644 test/engine/gh-1088-concurrent-bitset-iteration.test.lua diff --git a/test/engine/gh-1088-concurrent-bitset-iteration.result b/test/engine/gh-1088-concurrent-bitset-iteration.result new file mode 100644 index 0000000000..242c745628 --- /dev/null +++ b/test/engine/gh-1088-concurrent-bitset-iteration.result @@ -0,0 +1,94 @@ +-- test-run result file version 2 +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration + +test_run = require('test_run').new() + | --- + | ... +fiber = require('fiber') + | --- + | ... + +s = box.schema.space.create('gh-1088') + | --- + | ... +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) + | --- + | ... +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) + | --- + | ... +for i = 1, 100 do s:insert{i, 0, i - 1} end + | --- + | ... + +counter = 0 + | --- + | ... +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +function update() + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do + counter = counter + 1 + s:update(t[1], {{'+', 3, 11}}) + fiber.sleep(0) + end + fiber.self():cancel() +end; + | --- + | ... +test_run:cmd("setopt delimiter ''"); + | --- + | - true + | ... + +fibers = {} + | --- + | ... +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end + | --- + | ... + +updating = true + | --- + | ... +test_run:cmd("setopt delimiter ';'") + | --- + | - true + | ... +while updating do + updating = false + for _, f in pairs(fibers) do + if f:status() ~= 'dead' then updating = true end + end + fiber.sleep(0.001) +end; + | --- + | ... +test_run:cmd("setopt delimiter ''"); + | --- + | - true + | ... + +s:get(1) + | --- + | - [1, 0, 1100] + | ... +s:get(2) + | --- + | - [2, 0, 1101] + | ... +s:get(3) + | --- + | - [3, 0, 1102] + | ... +s:get(4) + | --- + | - [4, 0, 1103] + | ... + +counter -- total updates counter + | --- + | - 10000 + | ... diff --git a/test/engine/gh-1088-concurrent-bitset-iteration.test.lua b/test/engine/gh-1088-concurrent-bitset-iteration.test.lua new file mode 100644 index 0000000000..f57d1ecabe --- /dev/null +++ b/test/engine/gh-1088-concurrent-bitset-iteration.test.lua @@ -0,0 +1,42 @@ +-- gh-1088 concurrent tuple update segfaults on BITSET_ALL_NOT_SET iteration + +test_run = require('test_run').new() +fiber = require('fiber') + +s = box.schema.space.create('gh-1088') +_ = s:create_index('primary', {type = 'hash', parts = {1, 'num'}}) +_ = s:create_index('bitset', {unique = false, type = 'BITSET', parts = {2, 'num'}}) +for i = 1, 100 do s:insert{i, 0, i - 1} end + +counter = 0 +test_run:cmd("setopt delimiter ';'") +function update() + for _, t in s.index.bitset:pairs(1, {iterator = box.index.BITS_ALL_NOT_SET}) do + counter = counter + 1 + s:update(t[1], {{'+', 3, 11}}) + fiber.sleep(0) + end + fiber.self():cancel() +end; +test_run:cmd("setopt delimiter ''"); + +fibers = {} +for _ = 1, 100 do table.insert(fibers, fiber.create(update)) end + +updating = true +test_run:cmd("setopt delimiter ';'") +while updating do + updating = false + for _, f in pairs(fibers) do + if f:status() ~= 'dead' then updating = true end + end + fiber.sleep(0.001) +end; +test_run:cmd("setopt delimiter ''"); + +s:get(1) +s:get(2) +s:get(3) +s:get(4) + +counter -- total updates counter -- 2.17.1 From sergeyb at tarantool.org Mon Aug 31 14:06:42 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Mon, 31 Aug 2020 14:06:42 +0300 Subject: [Tarantool-patches] [PATCH v1] replication: change space sync mode in a loop In-Reply-To: <347d6e10-deb8-d349-e1df-033b5075903f@tarantool.org> References: <4c64406bcbe4f52629eba5a5c4dc1c9ea0115dea.1598454255.git.sergeyb@tarantool.org> <347d6e10-deb8-d349-e1df-033b5075903f@tarantool.org> Message-ID: <20200831110642.GB94343@pony.bronevichok.ru> Sergey, thanks for review! See my comments inline. Test has been updated in a branch. On 14:27 Thu 27 Aug , Sergey Petrenko wrote: > Hi! Thanks for the patch! > > Please see my comments below. > > 26.08.2020 18:10, sergeyb at tarantool.org ?????: > > From: Sergey Bronnikov > > > > New regression tests covers cases when one can change synchronous mode > > of space to asynchronous and vice versa. > > > > Closes #5055 > > Part of #5144 > > --- > > > > Branch: ligurio/gh-4842-qsync-change-mode > > CI: https://gitlab.com/tarantool/tarantool/-/pipelines/182271234 > > > > test/replication/qsync_sync_mode.result | 164 ++++++++++++++++++++++ > > test/replication/qsync_sync_mode.test.lua | 90 ++++++++++++ > > 2 files changed, 254 insertions(+) > > create mode 100644 test/replication/qsync_sync_mode.result > > create mode 100644 test/replication/qsync_sync_mode.test.lua > > > > diff --git a/test/replication/qsync_sync_mode.result b/test/replication/qsync_sync_mode.result > > new file mode 100644 > > index 000000000..f2f95ec0f > > --- /dev/null > > +++ b/test/replication/qsync_sync_mode.result > > @@ -0,0 +1,164 @@ > > +-- test-run result file version 2 > > +env = require('test_run') > > + | --- > > + | ... > > +test_run = env.new() > > + | --- > > + | ... > > +engine = test_run:get_cfg('engine') > > + | --- > > + | ... > > +fiber = require('fiber') > > + | --- > > + | ... > > +math = require('math') > > + | --- > > + | ... > > +math.randomseed(os.time()) > > + | --- > > + | ... > > + > > +orig_synchro_quorum = box.cfg.replication_synchro_quorum > > + | --- > > + | ... > > +orig_synchro_timeout = box.cfg.replication_synchro_timeout > > + | --- > > + | ... > > + > > +disable_sync_mode = function() \ > > + local s = box.space._space:get(box.space.sync.id) \ > > + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ > > + box.space._space:replace(new_s) \ > > +end > > + | --- > > + | ... > > + > > +enable_sync_mode = function() \ > > + local s = box.space._space:get(box.space.sync.id) \ > > + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ > > + box.space._space:replace(new_s) \ > > +end > > + | --- > > + | ... > > + > > +set_random_sync_mode = function() \ > > + if (math.random(1, 10) > 5) then \ > > + enable_sync_mode() \ > > + else \ > > + disable_sync_mode() \ > > + end \ > > +end > > + | --- > > + | ... > > + > > +set_random_quorum = function(n) \ > > + box.cfg{replication_synchro_quorum=math.random(1, n)} \ > > +end > > + | --- > > + | ... > > + > > +box.schema.user.grant('guest', 'replication') > > + | --- > > + | ... > > + > > +-- Setup an async cluster with two instances. > > +test_run:cmd('create server replica with rpl_master=default,\ > > + script="replication/replica.lua"') > > + | --- > > + | - true > > + | ... > > +test_run:cmd('start server replica with wait=True, wait_load=True') > > + | --- > > + | - true > > + | ... > > + > > +-- Write data to a leader, enable and disable sync mode in background in a > > +-- loop. Expected no data loss. > > +-- Testcase setup. > > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > > + | --- > > + | ... > > +_ = box.space.sync:create_index('pk') > > + | --- > > + | ... > > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > > + | --- > > + | ... > > +-- Testcase body. > > +for i = 1,10 do \ > > + set_random_sync_mode() \ > > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > > + test_run:switch('replica') \ > > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > > + end \ > > + test_run:switch('default') \ > > +end > > + | --- > > + | ... > > +-- Testcase cleanup. > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +box.space.sync:drop() > > + | --- > > + | ... > > + > > +-- Write data to a leader, enable and disable sync mode and change quorum value > > +-- in background in a loop. > > +-- Testcase setup. > > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > > + | --- > > + | ... > > +_ = box.space.sync:create_index('pk') > > + | --- > > + | ... > > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > > + | --- > > + | ... > > +-- Testcase body. > > +for i = 1,10 do \ > > + set_random_sync_mode() \ > > + set_random_quorum(5) \ > > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > > + test_run:switch('replica') \ > > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > > + end \ > > + test_run:switch('default') \ > > +end > > + | --- > > + | ... > > +-- Testcase cleanup. > > +test_run:switch('default') > > + | --- > > + | - true > > + | ... > > +box.space.sync:drop() > > + | --- > > + | ... > > + > > +-- Teardown. > > +test_run:cmd('switch default') > > + | --- > > + | - true > > + | ... > > +test_run:cmd('stop server replica') > > + | --- > > + | - true > > + | ... > > +test_run:cmd('delete server replica') > > + | --- > > + | - true > > + | ... > > +test_run:cleanup_cluster() > > + | --- > > + | ... > > +box.schema.user.revoke('guest', 'replication') > > + | --- > > + | ... > > +box.cfg{ \ > > + replication_synchro_quorum = orig_synchro_quorum, \ > > + replication_synchro_timeout = orig_synchro_timeout, \ > > +} > > + | --- > > + | ... > > diff --git a/test/replication/qsync_sync_mode.test.lua b/test/replication/qsync_sync_mode.test.lua > > new file mode 100644 > > index 000000000..706261c10 > > --- /dev/null > > +++ b/test/replication/qsync_sync_mode.test.lua > > @@ -0,0 +1,90 @@ > > +env = require('test_run') > > +test_run = env.new() > > +engine = test_run:get_cfg('engine') > > +fiber = require('fiber') > > +math = require('math') > > +math.randomseed(os.time()) > > + > > +orig_synchro_quorum = box.cfg.replication_synchro_quorum > > +orig_synchro_timeout = box.cfg.replication_synchro_timeout > > + > > +disable_sync_mode = function() \ > > + local s = box.space._space:get(box.space.sync.id) \ > > + local new_s = s:update({{'=', 6, {is_sync=false}}}) \ > > + box.space._space:replace(new_s) \ > > +end > > + > > +enable_sync_mode = function() \ > > + local s = box.space._space:get(box.space.sync.id) \ > > + local new_s = s:update({{'=', 6, {is_sync=true}}}) \ > > + box.space._space:replace(new_s) \ > > +end > > + > > Vlad has pushed a patch with `space:alter` a couple of days ago. > So now you may say `space:alter{is_sync=true}`, `space:alter{is_sync=false}` > It does the same work you do, but looks much simpler. used alter in updated patch: -- Testcase body. for i = 1,100 do \ box.space.sync:alter{is_sync=random_boolean()} \ box.space.sync:insert{i} \ test_run:switch('replica') \ test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ test_run:switch('default') \ end > > +set_random_sync_mode = function() \ > > + if (math.random(1, 10) > 5) then \ > > + enable_sync_mode() \ > > + else \ > > + disable_sync_mode() \ > > + end \ > > +end > > + > > +set_random_quorum = function(n) \ > > + box.cfg{replication_synchro_quorum=math.random(1, n)} \ > > +end > > + > > +box.schema.user.grant('guest', 'replication') > > + > > +-- Setup an async cluster with two instances. > > +test_run:cmd('create server replica with rpl_master=default,\ > > + script="replication/replica.lua"') > > +test_run:cmd('start server replica with wait=True, wait_load=True') > > + > > +-- Write data to a leader, enable and disable sync mode in background in a > > +-- loop. Expected no data loss. > > But sync mode is not set in background, it's set in the same loop where > insertions happen. in updated version I set sync mode just before an insert, description in comment has been updated > > +-- Testcase setup. > > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > > +_ = box.space.sync:create_index('pk') > > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > > Is such a tiny timeout intended? I see you wrap the errors in a pcall, but > still, > it may happen that you get a timeout 10 out of 10 times and then you'll > test nothing. We usually use 30 second replication timeouts. Fixed by setting it to 1000. > > +-- Testcase body. > > +for i = 1,10 do \ > > + set_random_sync_mode() \ > > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > > + test_run:switch('replica') \ > > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > > + end \ > > + test_run:switch('default') \ > > +end > > +-- Testcase cleanup. > > +test_run:switch('default') > > +box.space.sync:drop() > > + > > +-- Write data to a leader, enable and disable sync mode and change quorum value > > +-- in background in a loop. > > +-- Testcase setup. > > +_ = box.schema.space.create('sync', {is_sync=true, engine=engine}) > > +_ = box.space.sync:create_index('pk') > > +box.cfg{replication_synchro_quorum=2, replication_synchro_timeout=0.001} > > Ok, I see why you need a small timeout in this testcase. Maybe set a small > timeout > only when the insertion is expected to fail(quorum = 3 or 4)?. And use a 30 > sec timeout > otherwise. I have rewrote tests to consider to cases: - positive, when we have achievable quorum, set mode to sync or async on a master and then make an insert. Expected successful writes always. - negative, when we change mode to sync or async with tx's in a limbo. So comment above is not relevant now. > > +-- Testcase body. > > +for i = 1,10 do \ > > + set_random_sync_mode() \ > > + set_random_quorum(5) \ > > Math.random() range is inclusive, so it'd be more fair, to choose quorum > from 1 to 4. > Then there'd be 2 cases of successful write, quorum 1 and 2, and 2 cases of > failed write, > quorum 3 and 4. removed set_random_quorum() in updated tests > > + if pcall(box.space.sync.insert, box.space.sync, {i}) then \ > > + test_run:switch('replica') \ > > + test_run:wait_cond(function() return box.space.sync:get{i} ~= nil end) \ > > + end \ > > + test_run:switch('default') \ > > +end > > +-- Testcase cleanup. > > +test_run:switch('default') > > +box.space.sync:drop() > > + > > +-- Teardown. > > +test_run:cmd('switch default') > > +test_run:cmd('stop server replica') > > +test_run:cmd('delete server replica') > > +test_run:cleanup_cluster() > > +box.schema.user.revoke('guest', 'replication') > > +box.cfg{ \ > > + replication_synchro_quorum = orig_synchro_quorum, \ > > + replication_synchro_timeout = orig_synchro_timeout, \ > > +} -- sergeyb@ From sergeyb at tarantool.org Mon Aug 31 14:14:24 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Mon, 31 Aug 2020 14:14:24 +0300 Subject: [Tarantool-patches] [PATCH 1/2] src: return back import of table.clear() method In-Reply-To: <20200728164128.7sbgbjw3qiayuouc@tkn_work_nb> References: <1f495519687c8e037c638ccabab28e23882a41df.1595943364.git.sergeyb@tarantool.org> <20200728164128.7sbgbjw3qiayuouc@tkn_work_nb> Message-ID: <20200831111424.GC94343@pony.bronevichok.ru> Alexander, thanks for review! See my comments inline. Patch has been updated in a branch and squashed to a single commit. On 19:41 Tue 28 Jul , Alexander Turenko wrote: > > src: return back import of table.clear() method > > We usually mark a subsystem using a prefix: 'app' or 'lua' would fit > good here. set prefix 'lua' in updated commit message > BTW, why you splitted the code and test changes? I know, there are cons > and pros, but we usually include tests into the same commit as a source > change. >From my point of view it is better to split functional changes and tests. Agree with you that there are different pros et contras regarding it and I don't want to argue about it. I have squashed changes to a single commit. > Upside: It works as the documentation of changes; a formal description > of the change in addition to informal one within a commit message. > > Downside: A bit more work is required to verify the test on different > tarantool version. > > > Import of 'table.clear' module has been removed > > to fix luacheck warning about unused variable in > > commit 3af79e70b5e1e9b1d69b97f3031a299132a02d2f > > Nit: We usually use form git_commit_hash ('header of the commit') when > mention a commit. Done. > > and method table.clear() became unavailable in Tarantool. > > I would clarify that `table.clear` is not available until an explicit > `require('table.clear')` call. Done in commit message. > > --- a/src/lua/trigger.lua > > +++ b/src/lua/trigger.lua > > @@ -1,4 +1,5 @@ > > local fun = require('fun') > > +local _ = require('table.clear') > > I would add a comment that would explain why it is necessary. Just to > don't confuse a reader of the code. Done: local table = require('table') +-- require modifies global "table" module and adds "clear" function to it. +-- Lua applications like Cartridge relies on it. +local _ = require('table.clear') + table.copy = table_shallowcopy > It seems it was added non-intentionally in > 8a3dae66f6f2c8dffac39bbda1b38d1e17dfcc63 ('On schema reload callback for > net.box module') (not used anywhere in sources). However now we see that > the membership module leans on the fact that it is available without > explicit 'require'. > > I would look for more appropriate place for it: there is nothing special > in trigger.lua. Maybe src/lua/table.lua or src/lua/init.lua. The former > looks better for me. I chose a file src/lua/table.lua and moved module import to it. > WBR, Alexander Turenko. -- sergeyb@ From sergeyb at tarantool.org Mon Aug 31 14:18:28 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Mon, 31 Aug 2020 14:18:28 +0300 Subject: [Tarantool-patches] [PATCH 1/2] src: return back import of table.clear() method In-Reply-To: <0f75a16f-c1a7-992b-29fb-54358861cd70@tarantool.org> References: <1f495519687c8e037c638ccabab28e23882a41df.1595943364.git.sergeyb@tarantool.org> <0f75a16f-c1a7-992b-29fb-54358861cd70@tarantool.org> Message-ID: <20200831111828.GD94343@pony.bronevichok.ru> Hi, Oleg! Thanks for review! On 16:03 Wed 12 Aug , Oleg Babin wrote: > Hi! Thanks for your patch! > > I think it shouldn't be placed in "src/lua/trigger.lua". I believe > "src/lua/table.lua" is more appropriate place. Agree, "src/lua/table.lua" is more appropriate place for it, updated. > Of course with comment why it should be done e.g. "This require modifies > global "table" module and adds "clear" function to it". Done. > On 28/07/2020 16:52, sergeyb at tarantool.org wrote: > > From: Sergey Bronnikov > > > > Import of 'table.clear' module has been removed > > to fix luacheck warning about unused variable in > > commit 3af79e70b5e1e9b1d69b97f3031a299132a02d2f > > and method table.clear() became unavailable in Tarantool. > > > > Part of #5210 > > --- > > src/lua/trigger.lua | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/src/lua/trigger.lua b/src/lua/trigger.lua > > index 1330ecdd4..066329ea6 100644 > > --- a/src/lua/trigger.lua > > +++ b/src/lua/trigger.lua > > @@ -1,4 +1,5 @@ > > local fun = require('fun') > > +local _ = require('table.clear') > > BTW, "require('table.clear')" should be enough without "local _ =" Removed it. > > -- > > -- Checks that argument is a callable, i.e. a function or a table -- sergeyb@ From sergeyb at tarantool.org Mon Aug 31 14:20:01 2020 From: sergeyb at tarantool.org (Sergey Bronnikov) Date: Mon, 31 Aug 2020 14:20:01 +0300 Subject: [Tarantool-patches] [PATCH 2/2] test: add regression test for table.clear() In-Reply-To: <20200728164130.j7dvrpz7azdonqqa@tkn_work_nb> References: <20200728164130.j7dvrpz7azdonqqa@tkn_work_nb> Message-ID: <20200831112001.GE94343@pony.bronevichok.ru> Alexander, thanks for review! On 19:41 Tue 28 Jul , Alexander Turenko wrote: > > --- /dev/null > > +++ b/test/box-tap/gh-5210-table-clear.test.lua > > I would place it into app-tap, because it is not related to box. ok, moved to app-tap suite. > > +t = {a = 1, b = 2} > > I would use 'local' here. Added. > > +test:is(table.clear(t), nil, 'table clear') > > I would check that the table is cleared: say, using `next(t)`. It must > return `nil` for an empty table. Added is_deeply() to make sure it is empty: local t = {a = 1, b = 2} test:is(table.clear(t), nil, 'table clear') test:is_deeply(t, {}, 'table is clear') > WBR, Alexander Turenko. -- sergeyb@ From kyukhin at tarantool.org Mon Aug 31 14:36:13 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Mon, 31 Aug 2020 14:36:13 +0300 Subject: [Tarantool-patches] [PATCH v2] test: concurrent tuple update segfault on bitset index iteration In-Reply-To: <20200831105954.32408-1-i.kosarev@tarantool.org> References: <20200831105954.32408-1-i.kosarev@tarantool.org> Message-ID: <20200831113613.nfj3rpwy3guttadc@tarantool.org> Hello, On 31 ??? 13:59, Ilya Kosarev wrote: > Concurrent tuple update could segfault on BITSET_ALL_NOT_SET iterator > usage. Fixed in 850054b2dbca257076c3f7c22e00564ac55b70d5. This patch > introduces corresponding test. > > Closes #1088 > --- > Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-1088-concurrent-tuple-update-segfault-with-bitset-index > Issue: https://github.com/tarantool/tarantool/issues/1088 I've checked your patch into 1.10, 2.4, 2.5 and master. -- Regards, Kirill Yukhin From kyukhin at tarantool.org Mon Aug 31 14:39:28 2020 From: kyukhin at tarantool.org (Kirill Yukhin) Date: Mon, 31 Aug 2020 14:39:28 +0300 Subject: [Tarantool-patches] [PATCH v1] update_repo: correct fix for RPMs on missing metadata In-Reply-To: <9d538c87b2a0106b325c1ec104206a9f38f207df.1595855406.git.avtikhon@tarantool.org> References: <9d538c87b2a0106b325c1ec104206a9f38f207df.1595855406.git.avtikhon@tarantool.org> Message-ID: <20200831113928.x4mwgonzluow3ri4@tarantool.org> Hello, On 27 ??? 16:11, Alexander V. Tikhonov wrote: > Found that removing RPMs additional call to its removement is > needed, when no metadata was found. > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/correct_rpm_remove LGTM. I've checked your patch into 1.10, 2.4, 2.5 and master. -- Regards, Kirill Yukhin From huston.mavr at gmail.com Mon Aug 31 20:37:20 2020 From: huston.mavr at gmail.com (Alexandr Barulev) Date: Mon, 31 Aug 2020 20:37:20 +0300 Subject: [Tarantool-patches] [PATCH v2] build: refactor static build process In-Reply-To: References: <20200825140108.52090-1-huston.mavr@gmail.com> <20200825145103.hiqmjkimuk3al3rt@tkn_work_nb> Message-ID: Hello! Here is a new fix for building curses/ncurses library. Curses uses terminfo db (https://linux.die.net/man/5/terminfo). Previous version of static-build patch didn't link with terminfo database and this led to the problem that backspace and arrows doesn't work at tarantool terminal. Now this behaviour is fixed by setting search paths of terminfo db. Diff: diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt index ecfdd0455..d07cae176 100644 --- a/static-build/CMakeLists.txt +++ b/static-build/CMakeLists.txt @@ -82,9 +82,18 @@ ExternalProject_Add(ncurses # necessary for correct work of FindCurses.cmake module (this module is # builtin at cmake package) which used in cmake/FindReadline.cmake --enable-overwrite - --with-termlib # enable building libtinfo to prevent linking - # with libtinfo from system directories - INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install.libs + + # enable building libtinfo to prevent linking with libtinfo from system + # directories + --with-termlib + + # set search paths for terminfo db + --with-terminfo-dirs=/lib/terminfo:/usr/share/terminfo:/etc/terminfo + + # disable install created terminfo db, use db from system + --disable-db-install + --without-progs + --without-manpages ) # ??, 28 ???. 2020 ?. ? 00:43, Alexandr Barulev : > Hello, thanks for the new iteration of review! > > I deleted unused box.lua script and reference to it in suite.ini > at static-build tests. I added comments about explicitly setting > compilers at static-build/CMakeLists.txt. > > Also I fixed building libncurses/libcurses for correct work of > FindCurses.cmake module. After this change there is no need to set > CURSES_NEED_NCURSES at cmake/FindReadline.cmake, so I checkouted > cmake/FindReadline.cmake from master and update patch commit massage. > > And fixed .travis.mk static_build_cmake_* jobs - build tarantool > with -DCMAKE_BUILD_TYPE=RelWithDebInfo cmake option. > > > Here is a new commit message (deleted part about CURSES_NEED_NCURSES): > > build: refactor static build process > > Refactored static build process to use static-build/CMakeLists.txt > instead of Dockerfile.staticbuild (this allows to support static > build on macOS). Following third-party dependencies for static build > are installed via cmake `ExternalProject_Add`: > - OpenSSL > - Zlib > - Ncurses > - Readline > - Unwind > - ICU > > * Added support static build for macOS > * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for staic > build at file cmake/BuildLibCURL.cmake: > - disable building shared libcurl libraries (by setting > `--disable-shared` option) > - disable hiding libcurl symbols (by setting > `--disable-symbol-hiding` option) > - prevent linking libcurl with system libz by settign > `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option) > * Removed Dockerfile.staticbuild > * Added new gitlab.ci jobs to test new style static build: > - static_build_cmake_linux > - static_build_cmake_osx_15 > * Removed static_docker_build gitlab.ci job > > Closes #5095 > > > Here is a link to branch: > https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build > > > And here is a diff: > > diff --git a/.travis.mk b/.travis.mk > index 482672429..ccd9d6db1 100644 > --- a/.travis.mk > +++ b/.travis.mk > @@ -151,7 +151,8 @@ test_static_build: deps_debian_static > # New static build > # builddir used in this target - is a default build path from cmake > ExternalProject_Add() > test_static_build_cmake_linux: > - cd static-build && cmake . && make -j && ctest -V > + cd static-build && cmake > -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" > . && \ > + make -j && ctest -V > cd test && /usr/bin/python test-run.py --force \ > --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build > $(TEST_RUN_EXTRA_PARAMS) > > @@ -234,7 +235,8 @@ base_deps_osx: > > # builddir used in this target - is a default build path from cmake > ExternalProject_Add() > test_static_build_cmake_osx: base_deps_osx > - cd static-build && cmake . && make -j && ctest -V > + cd static-build && cmake > -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" > . && \ > + make -j && ctest -V > ${INIT_TEST_ENV_OSX}; \ > cd test && ./test-run.py --vardir /tmp/tnt \ > --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ > diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake > index afe480679..c48bdcb3e 100644 > --- a/cmake/FindReadline.cmake > +++ b/cmake/FindReadline.cmake > @@ -14,17 +14,7 @@ if(BUILD_STATIC) > if (NOT CURSES_INFO_LIBRARY) > set(CURSES_INFO_LIBRARY "") > endif() > - > - # From Modules/FindCurses.cmake: > - # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the > - # ``find_package(Curses)`` call if NCurses functionality is required. > - # This flag is set for linking with required library (installed > - # via static-build/CMakeLists.txt). If this variable won't be set > - # then tarantool binary links with system library curses which is an > - # entire copy of ncurses > - set(CURSES_NEED_NCURSES TRUE) > endif() > - > find_package(Curses) > if(NOT CURSES_FOUND) > find_package(Termcap) > diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt > index d90a642e6..ecfdd0455 100644 > --- a/static-build/CMakeLists.txt > +++ b/static-build/CMakeLists.txt > @@ -9,6 +9,12 @@ set(NCURSES_VERSION 6.2) > set(READLINE_VERSION 8.0) > set(UNWIND_VERSION 1.3-rc1) > > +# Set compilers explicitly for further configuring dependencies with > +# these compilers. This gonna solve libicu building problem in case when > +# at dependency configure stage no compiler specified and clang compiler > +# exists on linux machine, libicu sources would be compiled with clang > +# while for other dependencies (including tarantool) gcc would be used. > +# This behaviour causes problem at tarantool linkage stage. > if (APPLE) > find_program(C_COMPILER clang) > find_program(CXX_COMPILER clang++) > @@ -70,6 +76,15 @@ ExternalProject_Add(ncurses > CXX=${CMAKE_CXX_COMPILER} > /configure > --prefix= > + > + # This flag enables creation of libcurses.a as a symlink to > libncurses.a > + # and disables subdir creation `ncurses` at > /include. It is > + # necessary for correct work of FindCurses.cmake module (this > module is > + # builtin at cmake package) which used in cmake/FindReadline.cmake > + --enable-overwrite > + --with-termlib # enable building libtinfo to prevent > linking > + # with libtinfo from system directories > + INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install.libs > ) > > # > diff --git a/static-build/test/static-build/box.lua > b/static-build/test/static-build/box.lua > deleted file mode 100755 > index bad8a9055..000000000 > --- a/static-build/test/static-build/box.lua > +++ /dev/null > @@ -1,3 +0,0 @@ > -#!/usr/bin/env tarantool > - > -require('console').listen(os.getenv('ADMIN')) > diff --git a/static-build/test/static-build/suite.ini > b/static-build/test/static-build/suite.ini > index 92e349466..5aabadd92 100644 > --- a/static-build/test/static-build/suite.ini > +++ b/static-build/test/static-build/suite.ini > @@ -1,5 +1,4 @@ > [default] > core = app > description = Static build tests > -script = box.lua > is_parallel = True > > ??, 25 ???. 2020 ?. ? 17:51, Alexander Turenko < > alexander.turenko at tarantool.org>: > >> I looked very briefly (not thoroughly at all) on this iteration. >> >> There is nothing that confuses me (except few tiny comments below). >> >> I hope Igor will do thorough review. >> >> WBR, Alexander Turenko. >> >> > +if (APPLE) >> > + find_program(C_COMPILER clang) >> > + find_program(CXX_COMPILER clang++) >> > +else() >> > + find_program(C_COMPILER gcc) >> > + find_program(CXX_COMPILER g++) >> > +endif() >> >> Can we just leave it default? >> >> In offline discussion Alexandr B. said that tarantool builds with gcc, >> but icu with clang that gives some problem. >> >> Possible solution is to pass ${CMAKE_C_COMPILER} (and CXX too where >> necessary) to a subproject as we do for c-ares and curl. It seems it is >> already done, so maybe it worth to re-check whether it solves the >> problem. >> >> Anyway, if we really need to set a compiler here explicitly, I don't >> mind. Just noted that this way is somewhat unusual as I see. >> >> > diff --git a/static-build/test/static-build/box.lua >> b/static-build/test/static-build/box.lua >> > new file mode 100755 >> > index 000000000..bad8a9055 >> > --- /dev/null >> > +++ b/static-build/test/static-build/box.lua >> > @@ -0,0 +1,3 @@ >> > +#!/usr/bin/env tarantool >> > + >> > +require('console').listen(os.getenv('ADMIN')) >> >> Is looks redundant, see the comment below. >> >> > diff --git a/static-build/test/static-build/suite.ini >> b/static-build/test/static-build/suite.ini >> > new file mode 100644 >> > index 000000000..92e349466 >> > --- /dev/null >> > +++ b/static-build/test/static-build/suite.ini >> > @@ -0,0 +1,5 @@ >> > +[default] >> > +core = app >> > +description = Static build tests >> > +script = box.lua >> > +is_parallel = True >> >> 'script' does not have sense for 'core = app' test, it is for 'core = >> tarantool' tests. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: