From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id DD4996EC55; Wed, 28 Jul 2021 11:20:01 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org DD4996EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1627460402; bh=cjO2sbcdoPzl2/HrjtQEIzpyGG7jPhkqecaUr/9/I/I=; h=To:References:In-Reply-To:Date:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Ma9CDemL+2IbyN9i/oDKsFsOFVjWBwytrTtoz2qIxpC1JbQBlFbngohIx7EXyjEKH /gUXknJ8RFYelQknOZu+WCGQGwmnjDkURJJBqE8COPgK2089Aus6YrrDkYv4N1dJP4 9ChAvAVrqvS6P0M77QfejsA9RgSVrK3x/cd3gE6g= Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id B0BE16EC55 for ; Wed, 28 Jul 2021 11:20:00 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B0BE16EC55 Received: by smtp41.i.mail.ru with esmtpa (envelope-from ) id 1m8enH-0000j2-Io; Wed, 28 Jul 2021 11:19:59 +0300 To: References: <2939f293071df83ba612c431895adea49c9cb8de.1627430045.git.tsafin@tarantool.org> In-Reply-To: <2939f293071df83ba612c431895adea49c9cb8de.1627430045.git.tsafin@tarantool.org> Date: Wed, 28 Jul 2021 11:19:55 +0300 Message-ID: <006001d78389$59524100$0bf6c300$@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQJV0Gco4AoCzJGZ3DysJL5OF67LSgHmZayfqkw9O9A= Content-Language: ru X-4EC0790: 10 X-7564579A: 78E4E2B564C1792B X-77F55803: 4F1203BC0FB41BD941C43E597735A9C34755E0A9F196FCB739C645213AB7C8E0182A05F5380850400DB493E02AA918F98BD692D2DA30D6A03B8B7F3B086650153F03AB155850947B X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE71B1B5393BB9C1313EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063748C26B83F2B024408638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8772765C5FF8B6C22F3ED8CE2D4DAC194117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE7B96B19DC40933219D765CF6BB0EE041D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE3251EFD5447B32ED6C0837EA9F3D19764C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637B8F435DEDE9E76EBEFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: 0D63561A33F958A575C479871C66D277ADA226D703FD82E97E5D796A28D8A3ADD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75D299BBC8C521FD19410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D16EA493CC1FD9F8480E828D74337E3A6805EB8806267B4AB0E9A8EDB824303761E63A5D74FC642F1D7E09C32AA3244C4C61028F4137FECF22BBD4721CAC213EC3B3ADDA61883BB5927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojiF1u9eOpfTTRnJf2YMVcBQ== X-Mailru-Sender: B5B6A6EBBD94DAD8C47A93838048902D4A35D261CDD15203573789B4D0C40F29285477FE812B1CBC1EC9E4A2C82A33BC8C24925A86E657CE0C70AEE3C9A96FBAB3D7EE8ED63280BE112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v2 06/11] box, datetime: datetime comparison for indices X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Timur Safin via Tarantool-patches Reply-To: Timur Safin Cc: TML Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" As testing in release_lto revealed there is subtle problem with index sorting if we insert date which are very close=20 to each over in their nanoseconds parts https://github.com/tsafin/tarantool/runs/3176791534#step:5:5954 [022] Test failed! Result content mismatch:=20 [022] --- engine/datetime.result Tue Jul 27 23:04:00 2021=20 [022] +++ /tmp/tnt/rejects/engine/datetime.reject Tue Jul 27 23:18:29 = 2021=20 [022] @@ -70,7 +70,8 @@=20 [022]=20 [022] err=20 [022] | ---=20 [022] - | - null=20 [022] + | - - 2021-07-27T23:18:29.451Z=20 [022] + | - 2021-07-27T23:18:29.451020Z=20 [022] | ...=20 [022] box.space.T:drop()=20 [022] | ---=20 [022]=20 [022] Last 15 lines of Tarantool Log file [Instance = "box"][/tmp/tnt/022_engine/box.log]: =09 With expected engine/datetime.result: =09 for i =3D 1, #a - 1 do\ if tostring(a[i][1]) >=3D tostring(a[i+1][1]) then\ err =3D {a[i][1], a[i+1][1]}\ break\ end\ end | --- | ... =09 err | --- | - null | ... We have 2 dates of values 2021-07-27T23:18:29.451Z and=20 2021-07-27T23:18:29.451020Z which are in wrong order in the index. Will need to take into account nanoseconds part. Timur : From: Timur Safin : Subject: [PATCH v2 06/11] box, datetime: datetime comparison for = indices :=20 : * storage hints implemented for datetime_t values; : * proper comparison for indices of datetime type. :=20 : Part of #5941 : Part of #5946 : --- : src/box/field_def.c | 18 ++++++++ : src/box/field_def.h | 3 ++ : src/box/memtx_space.c | 3 +- : src/box/tuple_compare.cc | 30 ++++++++++++++ : src/box/vinyl.c | 3 +- : test/engine/datetime.result | 77 = +++++++++++++++++++++++++++++++++++ : test/engine/datetime.test.lua | 35 ++++++++++++++++ : 7 files changed, 165 insertions(+), 4 deletions(-) : create mode 100644 test/engine/datetime.result : create mode 100644 test/engine/datetime.test.lua :=20 : diff --git a/src/box/field_def.c b/src/box/field_def.c : index 6964e3e9f..aaf5f9cff 100644 : --- a/src/box/field_def.c : +++ b/src/box/field_def.c : @@ -193,3 +193,21 @@ field_type_by_name(const char *name, size_t len) : return FIELD_TYPE_ANY; : return field_type_MAX; : } : + : +const bool field_type_index_allowed[] =3D : + { : + /* [FIELD_TYPE_ANY] =3D */ false, : + /* [FIELD_TYPE_UNSIGNED] =3D */ true, : + /* [FIELD_TYPE_STRING] =3D */ true, : + /* [FIELD_TYPE_NUMBER] =3D */ true, : + /* [FIELD_TYPE_DOUBLE] =3D */ true, : + /* [FIELD_TYPE_INTEGER] =3D */ true, : + /* [FIELD_TYPE_BOOLEAN] =3D */ true, : + /* [FIELD_TYPE_VARBINARY]=3D */ true, : + /* [FIELD_TYPE_SCALAR] =3D */ true, : + /* [FIELD_TYPE_DECIMAL] =3D */ true, : + /* [FIELD_TYPE_UUID] =3D */ true, : + /* [FIELD_TYPE_ARRAY] =3D */ false, : + /* [FIELD_TYPE_MAP] =3D */ false, : + /* [FIELD_TYPE_DATETIME] =3D */ true, : +}; : diff --git a/src/box/field_def.h b/src/box/field_def.h : index 120b2a93d..bd02418df 100644 : --- a/src/box/field_def.h : +++ b/src/box/field_def.h : @@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[]; : extern const struct opt_def field_def_reg[]; : extern const struct field_def field_def_default; :=20 : +/** helper table for checking allowed indices for types */ : +extern const bool field_type_index_allowed[]; : + : /** : * @brief Field definition : * Contains information about of one tuple field. : diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c : index b71318d24..1ab16122e 100644 : --- a/src/box/memtx_space.c : +++ b/src/box/memtx_space.c : @@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space, = struct : index_def *index_def) : /* Check that there are no ANY, ARRAY, MAP parts */ : for (uint32_t i =3D 0; i < key_def->part_count; i++) { : struct key_part *part =3D &key_def->parts[i]; : - if (part->type <=3D FIELD_TYPE_ANY || : - part->type >=3D FIELD_TYPE_ARRAY) { : + if (!field_type_index_allowed[part->type]) { : diag_set(ClientError, ER_MODIFY_INDEX, : index_def->name, space_name(space), : tt_sprintf("field type '%s' is not supported", : diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc : index 48c833643..4adceebd3 100644 : --- a/src/box/tuple_compare.cc : +++ b/src/box/tuple_compare.cc : @@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a, = enum : mp_type a_type, : field_b, b_type); : case FIELD_TYPE_UUID: : return mp_compare_uuid(field_a, field_b); : + case FIELD_TYPE_DATETIME: : + return mp_compare_datetime(field_a, field_b); : default: : unreachable(); : return 0; : @@ -1630,6 +1632,18 @@ hint_uuid_raw(const char *data) : return hint_create(MP_CLASS_UUID, val); : } :=20 : +static inline hint_t : +hint_datetime(struct datetime_t *date) : +{ : + /* : + * Use at most HINT_VALUE_BITS from datetime : + * seconds field as a hint value : + */ : + uint64_t val =3D (uint64_t)date->secs & HINT_VALUE_MAX; : + : + return hint_create(MP_CLASS_DATETIME, val); : +} : + : static inline uint64_t : hint_str_raw(const char *s, uint32_t len) : { : @@ -1761,6 +1775,17 @@ field_hint_uuid(const char *field) : return hint_uuid_raw(data); : } :=20 : +static inline hint_t : +field_hint_datetime(const char *field) : +{ : + assert(mp_typeof(*field) =3D=3D MP_EXT); : + int8_t ext_type; : + uint32_t len =3D mp_decode_extl(&field, &ext_type); : + assert(ext_type =3D=3D MP_DATETIME); : + struct datetime_t date; : + return hint_datetime(datetime_unpack(&field, len, &date)); : +} : + : static inline hint_t : field_hint_string(const char *field, struct coll *coll) : { : @@ -1849,6 +1874,8 @@ field_hint(const char *field, struct coll *coll) : return field_hint_decimal(field); : case FIELD_TYPE_UUID: : return field_hint_uuid(field); : + case FIELD_TYPE_DATETIME: : + return field_hint_datetime(field); : default: : unreachable(); : } : @@ -1963,6 +1990,9 @@ key_def_set_hint_func(struct key_def *def) : case FIELD_TYPE_UUID: : key_def_set_hint_func(def); : break; : + case FIELD_TYPE_DATETIME: : + key_def_set_hint_func(def); : + break; : default: : /* Invalid key definition. */ : def->key_hint =3D NULL; : diff --git a/src/box/vinyl.c b/src/box/vinyl.c : index c80b2d99b..360d1fa70 100644 : --- a/src/box/vinyl.c : +++ b/src/box/vinyl.c : @@ -662,8 +662,7 @@ vinyl_space_check_index_def(struct space *space, = struct : index_def *index_def) : /* Check that there are no ANY, ARRAY, MAP parts */ : for (uint32_t i =3D 0; i < key_def->part_count; i++) { : struct key_part *part =3D &key_def->parts[i]; : - if (part->type <=3D FIELD_TYPE_ANY || : - part->type >=3D FIELD_TYPE_ARRAY) { : + if (!field_type_index_allowed[part->type]) { : diag_set(ClientError, ER_MODIFY_INDEX, : index_def->name, space_name(space), : tt_sprintf("field type '%s' is not supported", : diff --git a/test/engine/datetime.result b/test/engine/datetime.result : new file mode 100644 : index 000000000..3ff474dee : --- /dev/null : +++ b/test/engine/datetime.result : @@ -0,0 +1,77 @@ : +-- test-run result file version 2 : +env =3D require('test_run') : + | --- : + | ... : +test_run =3D env.new() : + | --- : + | ... : +engine =3D test_run:get_cfg('engine') : + | --- : + | ... : + : +date =3D require('datetime') : + | --- : + | ... : + : +_ =3D box.schema.space.create('T', {engine =3D engine}) : + | --- : + | ... : +_ =3D box.space.T:create_index('pk', {parts=3D{1,'datetime'}}) : + | --- : + | ... : + : +box.space.T:insert{date('1970-01-01')}\ : +box.space.T:insert{date('1970-01-02')}\ : +box.space.T:insert{date('1970-01-03')}\ : +box.space.T:insert{date('2000-01-01')} : + | --- : + | ... : + : +o =3D box.space.T:select{} : + | --- : + | ... : +assert(tostring(o[1][1]) =3D=3D '1970-01-01T00:00Z') : + | --- : + | - true : + | ... : +assert(tostring(o[2][1]) =3D=3D '1970-01-02T00:00Z') : + | --- : + | - true : + | ... : +assert(tostring(o[3][1]) =3D=3D '1970-01-03T00:00Z') : + | --- : + | - true : + | ... : +assert(tostring(o[4][1]) =3D=3D '2000-01-01T00:00Z') : + | --- : + | - true : + | ... : + : +for i =3D 1,16 do\ : + box.space.T:insert{date.now()}\ : +end : + | --- : + | ... : + : +a =3D box.space.T:select{} : + | --- : + | ... : +err =3D nil : + | --- : + | ... : +for i =3D 1, #a - 1 do\ : + if tostring(a[i][1]) >=3D tostring(a[i+1][1]) then\ : + err =3D {a[i][1], a[i+1][1]}\ : + break\ : + end\ : +end : + | --- : + | ... : + : +err : + | --- : + | - null : + | ... : +box.space.T:drop() : + | --- : + | ... : diff --git a/test/engine/datetime.test.lua = b/test/engine/datetime.test.lua : new file mode 100644 : index 000000000..180d9bb72 : --- /dev/null : +++ b/test/engine/datetime.test.lua : @@ -0,0 +1,35 @@ : +env =3D require('test_run') : +test_run =3D env.new() : +engine =3D test_run:get_cfg('engine') : + : +date =3D require('datetime') : + : +_ =3D box.schema.space.create('T', {engine =3D engine}) : +_ =3D box.space.T:create_index('pk', {parts=3D{1,'datetime'}}) : + : +box.space.T:insert{date('1970-01-01')}\ : +box.space.T:insert{date('1970-01-02')}\ : +box.space.T:insert{date('1970-01-03')}\ : +box.space.T:insert{date('2000-01-01')} : + : +o =3D box.space.T:select{} : +assert(tostring(o[1][1]) =3D=3D '1970-01-01T00:00Z') : +assert(tostring(o[2][1]) =3D=3D '1970-01-02T00:00Z') : +assert(tostring(o[3][1]) =3D=3D '1970-01-03T00:00Z') : +assert(tostring(o[4][1]) =3D=3D '2000-01-01T00:00Z') : + : +for i =3D 1,16 do\ : + box.space.T:insert{date.now()}\ : +end : + : +a =3D box.space.T:select{} : +err =3D nil : +for i =3D 1, #a - 1 do\ : + if tostring(a[i][1]) >=3D tostring(a[i+1][1]) then\ : + err =3D {a[i][1], a[i+1][1]}\ : + break\ : + end\ : +end : + : +err : +box.space.T:drop() : -- : 2.29.2