From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 10DC76EC42; Mon, 16 Aug 2021 02:59:59 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 10DC76EC42 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1629071999; bh=RuV5JmXtdcpv+Rpmdkt1DhszUiwMrvRZBZBHO+BrI5E=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=C4gHzTrVw892DD/f7oIwbX3nDrbKiX9dbXL+yCPGA+h+/EhoRQCyz4rkCsP7UidXu rsh5dk7Sfti5BHdGaw6gUYkRDJSfHSlqA+LP2WzgJCc32qGmxBCczhuVl9CP0nmRXV 3Vqd/eWQTQxR2q4RgIRkLW6F9/60hytLXDk1q9o4= Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3F1306EC40 for ; Mon, 16 Aug 2021 02:59:57 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3F1306EC40 Received: by smtp41.i.mail.ru with esmtpa (envelope-from ) id 1mFQ2l-00047O-Kb; Mon, 16 Aug 2021 02:59:56 +0300 To: v.shpilevoy@tarantool.org, imun@tarantool.org, imeevma@tarantool.org, tarantool-patches@dev.tarantool.org Date: Mon, 16 Aug 2021 02:59:34 +0300 Message-Id: X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD9BCE6B93DE0C6C3914462CDB1732D383C182A05F5380850406E4457CDAE45E13AB49A9D077D034E915A6FFC8389FB2E1C1C4944741452B39F X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7D6964C9E324ABA58EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637D81244D2CF6B2D5F8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8162B92B5124DFFA6C504E353992DF7F2117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC60CDF180582EB8FBA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751FCB629EEF1311BF91D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EED76C6ED7039589DE287C8E22D4AE2A51D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE3CE135D2742255B35AD7EC71F1DB88427C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F79006372A310894EFA5E9A9EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458F0AFF96BAACF4158235E5A14AD4A4A4625E192CAD1D9E79D0B18DC6AC13D9A1C5445EB5C6A65B8BB X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C5B73F950BC6E7FFBF9CC349F881611946A70779428AE474F9C2B6934AE262D3EE7EAB7254005DCED41D154D92A01BDF11E0A4E2319210D9B64D260DF9561598F01A9E91200F654B068F972559010CA1C8E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D71B56C992F8AF194E655BE96A7C205BD30617EDD46D5462824D38D1B88B8760D80255F651019A051D7E09C32AA3244C77413F597E81D018A99882DC8384D3005A1673A01BA68E408D5DD81C2BAB7D1D X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojVT2don9h9KpYJedHgutaqA== X-Mailru-Sender: B5B6A6EBBD94DAD833E4B9A85F720031806AC487ADECA354D2516FD19E323BB28E5F0331A5842FB91EC9E4A2C82A33BC8C24925A86E657CE0C70AEE3C9A96FBAB3D7EE8ED63280BE112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Timur Safin via Tarantool-patches Reply-To: Timur Safin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" * Version #5 changes: - After performance evaluations it was decided to switch data type used by `datetime.secs` from `int64_t` (which is boxed in LuaJIT) to `double` (which is unboxed number). - please see perf/datetime*.cc perf test as a benchmark we used for comparison of 2 veriants of C implementation. Results are here - https://gist.github.com/tsafin/618fbf847d258f6e7f5a75fdf9ea945b * Version #4 changes: A lot. - Got rid of closures usage here and there. Benchmarks revealed there is 20x speed difference of hash with closure handlers to the case when we check keys using sequence of simple ifs. And despite the fact that it looks ugler, we have updated all places which used to use closures - so much performance impact does not worth it; - c-dt library now is accompanied with corresponding unit-tests; - due to a new (old) way of exporting public symbols, which is now used by Tarantool build, we have modified c-dt sources to use approach similar to that used by xxhash (or ICU) - "namespaces" for public symbols. If there is DT_NAMESPACE macro (e.g. DT_NAMESPACE=tnt_) defined at the moment of compilation we rename all relevant public symbols to use that "namespace" as prefix #define dt_from_rdn DT_NAME(DT_NAMESPACE, dt_from_rdn) C sources continue to use original name (now as a macro), but in Lua ffi we have to use renamed symbol - `tnt_dt_from_rdn`. - We have separated code which implement MessagePack related funcionality from generic (i.e. stringization) support for datetime types. Stringization is in `src/core/datetime.c`, but MessagePack support is in `src/core/mp_datetime.c`. - `ctime` and `asctime` now use reentrant versions (i.e. `ctime_r` instead of `ctime`, and `asctime_r` instead of `asctime`). This allows us to avoid nasty problem with GC where it was possible to corrupt returned static buffer regardless the fact that it was immediately passed to `ffi.string`. `test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua` updated to check our new api `datetime.ctime`, and `datetime.asctime`. - Introduced MessagePack unit-test which check C implementation (mp encode/decode, mp_snprint, mp_fprint) - Some special efforts have been taken to make sure we deal correctly with datetime stamps which were before Unix Epoch (1970-01-01), i.e. for negative stamps or timezones against epoch. - And last, but not the least: in addition to usual getters like `timestamp`, `nanoseconds`, `epoch`, we have introduced function modifiers `to_utc()` or `to_tz(offset)` which allow to manipulate with datetime timezone `.offset` field. Actually it does not do much now (only reattributing object with different offset, but not changed actual timestamp, which is UTC normalized). But different timestamp modifies textual representation, which simplify debugging. * Version #3 changes: - renamed `struct datetime_t` to `struct datetime`, and `struct datetime_interval_t` to `struct datetime_interval`; - significantly reworked arguments checks in module api entries; - fixed datetime comparisons; - changed hints calculation to take into account fractional part; - provided more comments here and there; NB! There are MacOSX problems due to GLIBC specific code used (as Vlad has already pointed out) - so additional patch, making it more cross-compatible coming here soon... * Version #2 changes: - fixed problem with overloaded '-' and '+' operations for datetime arguments; - fixed messagepack serialization problems; - heavily documented MessagePack serialization schema in the code; - introduced working implementation of datetime hints for storage engines; - made interval related names be more consistent, renamed durations and period to intervals, i.e. t_datetime_duration to datetime_interval_t, duration_* to interval_*, period to interval; - properly implemented all reasonable cases of datetime+interval arithmetic; - moved all initialization code to utils.c; - renamed core/mp_datetime.c to core/datetime.c because it makes more sense now; * Version #1 - initial RFC In brief -------- This patchset implements datetime lua support in box, with serialization to messagepack, yaml, json and lua mode. Also it contains storage engines' indices implementation for datetime type introduced. * Current implementation is heavily influenced by Sci-Lua lua-time module https://github.com/stepelu/lua-time e.g. you could find very similar approach for handling of operations with year or month long intervals (which should be handled differently than usual intervals of seconds, or days). * But internally we actually use Christian Hanson' c-dt module https://github.com/chansen/c-dt (though it has been modified slightly for cleaner integration into our cmake-based build process). Datetime Module API ------------------- Small draft of documentation for our `datetime` module API has been extracted to the discussion topic there - https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988 We try to update documentation there once any relevant change introdcued to the implementaton. Messagepack serialization schema -------------------------------- In short it looks like: - now we introduce new MP_EXT extension type #4; - we may save 1 required and 2 optional fields for datetime field; In all gory details it's explained in MessagePack serialization schema depicted here: https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990 https://github.com/tarantool/tarantool/issues/5941 https://github.com/tarantool/tarantool/issues/5946 https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v4 Timur Safin (8): build: add Christian Hansen c-dt to the build lua: built-in module datetime lua, datetime: display datetime box, datetime: messagepack support for datetime box, datetime: datetime comparison for indices lua, datetime: time intervals support datetime: perf test for datetime parser datetime: changelog for datetime module .gitmodules | 3 + CMakeLists.txt | 8 + .../gh-5941-datetime-type-support.md | 4 + cmake/BuildCDT.cmake | 10 + extra/exports | 33 + perf/CMakeLists.txt | 3 + perf/datetime-common.h | 105 +++ perf/datetime-compare.cc | 213 +++++ perf/datetime-parser.cc | 105 +++ src/CMakeLists.txt | 5 +- src/box/field_def.c | 53 +- src/box/field_def.h | 4 + src/box/lua/serialize_lua.c | 7 +- src/box/memtx_space.c | 3 +- src/box/msgpack.c | 7 +- src/box/tuple_compare.cc | 77 ++ src/box/vinyl.c | 3 +- src/lib/core/CMakeLists.txt | 5 +- src/lib/core/datetime.c | 176 ++++ src/lib/core/datetime.h | 115 +++ src/lib/core/mp_datetime.c | 189 ++++ src/lib/core/mp_datetime.h | 89 ++ src/lib/core/mp_extension_types.h | 1 + src/lib/mpstream/mpstream.c | 11 + src/lib/mpstream/mpstream.h | 4 + src/lua/datetime.lua | 880 ++++++++++++++++++ src/lua/init.c | 4 +- src/lua/msgpack.c | 12 + src/lua/msgpackffi.lua | 18 + src/lua/serializer.c | 4 + src/lua/serializer.h | 2 + src/lua/utils.c | 28 +- src/lua/utils.h | 12 + test/app-tap/datetime.test.lua | 370 ++++++++ .../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 +- test/engine/datetime.result | 77 ++ test/engine/datetime.test.lua | 35 + test/unit/CMakeLists.txt | 3 +- test/unit/datetime.c | 381 ++++++++ test/unit/datetime.result | 471 ++++++++++ third_party/c-dt | 1 + third_party/lua-cjson/lua_cjson.c | 8 + third_party/lua-yaml/lyaml.cc | 6 +- 43 files changed, 3591 insertions(+), 28 deletions(-) create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md create mode 100644 cmake/BuildCDT.cmake create mode 100644 perf/datetime-common.h create mode 100644 perf/datetime-compare.cc create mode 100644 perf/datetime-parser.cc create mode 100644 src/lib/core/datetime.c create mode 100644 src/lib/core/datetime.h create mode 100644 src/lib/core/mp_datetime.c create mode 100644 src/lib/core/mp_datetime.h create mode 100644 src/lua/datetime.lua create mode 100755 test/app-tap/datetime.test.lua create mode 100644 test/engine/datetime.result create mode 100644 test/engine/datetime.test.lua create mode 100644 test/unit/datetime.c create mode 100644 test/unit/datetime.result create mode 160000 third_party/c-dt -- 2.29.2