* Version #2: - fixed problem with overloaded '-' and '+' operations for datetime arguments; - fixed messagepack serialization problems; - heavily documented MessagePack serialization schema in the code; - introduced working implementation of datetime hints for storage engines; - made interval related names be more consistent, renamed durations and period to intervals, i.e. t_datetime_duration to datetime_interval_t, duration_* to interval_*, period to interval; - properly implemented all reasonable cases of datetime+interval arithmetic; - moved all initialization code to utils.c; - renamed core/mp_datetime.c to core/datetime.c because it makes more sense now; * Version #1 - initial RFC series In brief -------- This patchset implements datetime lua support in box, with serialization to messagepack, yaml, json and lua mode. Also it contains storage engines' indices implementation for datetime type introduced. * Current implementation is heavily influenced by Sci-Lua lua-time module https://github.com/stepelu/lua-time e.g. you could find very similar approach for handling of operations with year or month long intervals (which should be handled differently than usual intervals of seconds, or days). * But internally we actually use Christian Hanson' c-dt module https://github.com/chansen/c-dt (though it has been modified slightly for cleaner integration into cmake build process) Datetime Module API ------------------- We used to have here draft documentation of datetime module api, but for a convenience it has been extracted to the discussion topic there - https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988 Messagepack serialization schema -------------------------------- In short it looks like: - now we introduce new MP_EXT extension type #4; - we may save 1 required and 2 optional fields for datetime field; In details it's explained in MessagePack serialization schema depicted here: https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990 https://github.com/tarantool/tarantool/issues/5941 https://github.com/tarantool/tarantool/issues/5946 https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v3 Timur Safin (11): build: add Christian Hansen c-dt to the build lua: built-in module datetime lua, datetime: datetime tests lua, datetime: display datetime box, datetime: add messagepack support for datetime box, datetime: datetime comparison for indices lua, datetime: proper datetime encoding lua, datetime: calculated attributes for datetimes lua, datetime: time intervals support lua, datetime: unixtime, timestamp setters in datetime.lua datetime: changelog for datetime module .gitmodules | 3 + CMakeLists.txt | 8 + .../gh-5941-datetime-type-support.md | 4 + cmake/BuildCDT.cmake | 6 + src/CMakeLists.txt | 3 + src/box/field_def.c | 52 +- src/box/field_def.h | 4 + src/box/lua/serialize_lua.c | 7 +- src/box/memtx_space.c | 3 +- src/box/msgpack.c | 7 +- src/box/tuple_compare.cc | 50 + src/box/vinyl.c | 3 +- src/exports.h | 29 + src/lib/core/CMakeLists.txt | 4 +- src/lib/core/datetime.c | 251 ++++ src/lib/core/datetime.h | 114 ++ src/lib/core/mp_extension_types.h | 1 + src/lib/mpstream/mpstream.c | 11 + src/lib/mpstream/mpstream.h | 4 + src/lua/datetime.lua | 1016 +++++++++++++++++ src/lua/init.c | 4 +- src/lua/msgpack.c | 12 + src/lua/msgpackffi.lua | 18 + src/lua/serializer.c | 4 + src/lua/serializer.h | 2 + src/lua/utils.c | 28 +- src/lua/utils.h | 12 + test/app-tap/datetime.test.lua | 367 ++++++ test/engine/datetime.result | 77 ++ test/engine/datetime.test.lua | 35 + test/unit/CMakeLists.txt | 2 + test/unit/datetime.c | 220 ++++ test/unit/datetime.result | 358 ++++++ third_party/c-dt | 1 + third_party/lua-cjson/lua_cjson.c | 8 + third_party/lua-yaml/lyaml.cc | 6 +- 36 files changed, 2709 insertions(+), 25 deletions(-) create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md create mode 100644 cmake/BuildCDT.cmake create mode 100755 src/lib/core/datetime.c create mode 100644 src/lib/core/datetime.h create mode 100644 src/lua/datetime.lua create mode 100755 test/app-tap/datetime.test.lua create mode 100644 test/engine/datetime.result create mode 100644 test/engine/datetime.test.lua create mode 100644 test/unit/datetime.c create mode 100644 test/unit/datetime.result create mode 160000 third_party/c-dt -- 2.29.2
* Integrated chansen/c-dt parser as 3rd party module to the Tarantool cmake build process. * Points to tsafin/c-dt instead iof original chansen/c-dt to have easier build integration, because there is additional commit which integrated cmake support --- .gitmodules | 3 +++ CMakeLists.txt | 8 ++++++++ cmake/BuildCDT.cmake | 6 ++++++ src/CMakeLists.txt | 1 + third_party/c-dt | 1 + 5 files changed, 19 insertions(+) create mode 100644 cmake/BuildCDT.cmake create mode 160000 third_party/c-dt diff --git a/.gitmodules b/.gitmodules index f2f91ee72..aa3fbae4e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -43,3 +43,6 @@ [submodule "third_party/xxHash"] path = third_party/xxHash url = https://github.com/tarantool/xxHash +[submodule "third_party/c-dt"] + path = third_party/c-dt + url = https://github.com/tarantool/c-dt.git diff --git a/CMakeLists.txt b/CMakeLists.txt index f9e1a7f79..81ee92933 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -571,6 +571,14 @@ endif() # zstd # +# +# Chritian Hanson c-dt +# + +include(BuildCDT) +libccdt_build() +add_dependencies(build_bundled_libs cdt) + # # Third-Party misc # diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake new file mode 100644 index 000000000..a19abb35c --- /dev/null +++ b/cmake/BuildCDT.cmake @@ -0,0 +1,6 @@ +macro(libccdt_build) + set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/) + set(LIBCDT_LIBRARIES cdt) + + add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt) +endmacro() diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index c0e272bd9..ef6a295d5 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -202,6 +202,7 @@ set (common_libraries ${ICONV_LIBRARIES} ${OPENSSL_LIBRARIES} ${XXHASH_LIBRARIES} + ${LIBCDT_LIBRARIES} ) if (TARGET_OS_LINUX OR TARGET_OS_DEBIAN_FREEBSD) diff --git a/third_party/c-dt b/third_party/c-dt new file mode 160000 index 000000000..8b61c4ea0 --- /dev/null +++ b/third_party/c-dt @@ -0,0 +1 @@ +Subproject commit 8b61c4ea006efefc3a068f8df4a156bf5c725c89 -- 2.29.2
* created a new Tarantool built-in module `datetime`; * register cdef types for this module; * export some `dt_*` functions from `c-dt` library; * lua implementationis of `asctime` and `strftime`; * datetime parsing unit tests, with and withput timezones; * c test for reversible strftime roundtrip; Part of #5941 --- src/CMakeLists.txt | 2 + src/exports.h | 21 ++ src/lib/core/datetime.h | 61 ++++ src/lua/datetime.lua | 581 ++++++++++++++++++++++++++++++++++++++ src/lua/init.c | 4 +- src/lua/utils.c | 27 ++ src/lua/utils.h | 12 + test/unit/CMakeLists.txt | 2 + test/unit/datetime.c | 221 +++++++++++++++ test/unit/datetime.result | 358 +++++++++++++++++++++++ 10 files changed, 1288 insertions(+), 1 deletion(-) create mode 100644 src/lib/core/datetime.h create mode 100644 src/lua/datetime.lua create mode 100644 test/unit/datetime.c create mode 100644 test/unit/datetime.result diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index ef6a295d5..e0499e57f 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua) lua_source(lua_sources lua/httpc.lua) lua_source(lua_sources lua/iconv.lua) lua_source(lua_sources lua/swim.lua) +lua_source(lua_sources lua/datetime.lua) + # LuaJIT jit.* library lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua) lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua) diff --git a/src/exports.h b/src/exports.h index 5bb3e6a2b..db40c03a4 100644 --- a/src/exports.h +++ b/src/exports.h @@ -531,3 +531,24 @@ EXPORT(uri_format) EXPORT(uri_parse) EXPORT(uuid_nil) EXPORT(uuid_unpack) +EXPORT(dt_from_rdn) +EXPORT(dt_from_yd) +EXPORT(dt_from_ymd) +EXPORT(dt_from_yqd) +EXPORT(dt_from_ywd) +EXPORT(dt_to_yd) +EXPORT(dt_to_ymd) +EXPORT(dt_to_yqd) +EXPORT(dt_to_ywd) +EXPORT(dt_rdn) +EXPORT(dt_dow) +EXPORT(dt_parse_iso_date) +EXPORT(dt_parse_iso_time) +EXPORT(dt_parse_iso_time_basic) +EXPORT(dt_parse_iso_time_extended) +EXPORT(dt_parse_iso_zone) +EXPORT(dt_parse_iso_zone_basic) +EXPORT(dt_parse_iso_zone_extended) +EXPORT(dt_parse_iso_zone_lenient) +EXPORT(dt_from_struct_tm) +EXPORT(dt_to_struct_tm) diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h new file mode 100644 index 000000000..403bf1c64 --- /dev/null +++ b/src/lib/core/datetime.h @@ -0,0 +1,61 @@ +#pragma once +/* + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <c-dt/dt_core.h> +#include <stdint.h> +#include <stdbool.h> + +#if defined(__cplusplus) +extern "C" { +#endif /* defined(__cplusplus) */ + +/** + * datetime structure consisting of: + */ +struct datetime_t { + int64_t secs; ///< seconds since epoch + int32_t nsec; ///< nanoseconds if any + int32_t offset; ///< offset in minutes from GMT +}; + +/** + * Date/time delta structure + */ +struct datetime_interval_t { + int64_t secs; ///< relative seconds delta + int32_t nsec; ///< nanoseconds delta +}; + +#if defined(__cplusplus) +} /* extern "C" */ +#endif /* defined(__cplusplus) */ + diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua new file mode 100644 index 000000000..0996ca5a2 --- /dev/null +++ b/src/lua/datetime.lua @@ -0,0 +1,581 @@ +local ffi = require('ffi') +local cdt = ffi.C + +ffi.cdef [[ + + typedef int dt_t; + + // dt_core.h + typedef enum { + DT_MON = 1, + DT_MONDAY = 1, + DT_TUE = 2, + DT_TUESDAY = 2, + DT_WED = 3, + DT_WEDNESDAY = 3, + DT_THU = 4, + DT_THURSDAY = 4, + DT_FRI = 5, + DT_FRIDAY = 5, + DT_SAT = 6, + DT_SATURDAY = 6, + DT_SUN = 7, + DT_SUNDAY = 7, + } dt_dow_t; + + dt_t dt_from_rdn (int n); + dt_t dt_from_yd (int y, int d); + dt_t dt_from_ymd (int y, int m, int d); + dt_t dt_from_yqd (int y, int q, int d); + dt_t dt_from_ywd (int y, int w, int d); + + void dt_to_yd (dt_t dt, int *y, int *d); + void dt_to_ymd (dt_t dt, int *y, int *m, int *d); + void dt_to_yqd (dt_t dt, int *y, int *q, int *d); + void dt_to_ywd (dt_t dt, int *y, int *w, int *d); + + int dt_rdn (dt_t dt); + dt_dow_t dt_dow (dt_t dt); + + // dt_parse_iso.h + size_t dt_parse_iso_date (const char *str, size_t len, dt_t *dt); + + size_t dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec); + size_t dt_parse_iso_time_basic (const char *str, size_t len, int *sod, int *nsec); + size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec); + + size_t dt_parse_iso_zone (const char *str, size_t len, int *offset); + size_t dt_parse_iso_zone_basic (const char *str, size_t len, int *offset); + size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset); + size_t dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset); + + // dt_tm.h + dt_t dt_from_struct_tm (const struct tm *tm); + void dt_to_struct_tm (dt_t dt, struct tm *tm); + + // <asm-generic/posix_types.h> + typedef long __kernel_long_t; + typedef unsigned long __kernel_ulong_t; + // /usr/include/x86_64-linux-gnu/bits/types/time_t.h + typedef long time_t; + + + // <time.h> + typedef __kernel_long_t __kernel_time_t; + typedef __kernel_long_t __kernel_suseconds_t; + + struct timespec { + __kernel_time_t tv_sec; /* seconds */ + long tv_nsec; /* nanoseconds */ + }; + + struct timeval { + __kernel_time_t tv_sec; /* seconds */ + __kernel_suseconds_t tv_usec; /* microseconds */ + }; + + struct timezone { + int tz_minuteswest; /* minutes west of Greenwich */ + int tz_dsttime; /* type of dst correction */ + }; + + // /usr/include/x86_64-linux-gnu/sys/time.h + typedef struct timezone * __timezone_ptr_t; + + /* Get the current time of day and timezone information, + putting it into *TV and *TZ. If TZ is NULL, *TZ is not filled. + Returns 0 on success, -1 on errors. + + NOTE: This form of timezone information is obsolete. + Use the functions and variables declared in <time.h> instead. */ + int gettimeofday (struct timeval *__tv, struct timezone * __tz); + + // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h + /* ISO C `broken-down time' structure. */ + struct tm + { + int tm_sec; /* Seconds. [0-60] (1 leap second) */ + int tm_min; /* Minutes. [0-59] */ + int tm_hour; /* Hours. [0-23] */ + int tm_mday; /* Day. [1-31] */ + int tm_mon; /* Month. [0-11] */ + int tm_year; /* Year - 1900. */ + int tm_wday; /* Day of week. [0-6] */ + int tm_yday; /* Days in year.[0-365] */ + int tm_isdst; /* DST. [-1/0/1]*/ + + long int tm_gmtoff; /* Seconds east of UTC. */ + const char *tm_zone;/* Timezone abbreviation. */ + }; + + // <time.h> + /* Return the current time and put it in *TIMER if TIMER is not NULL. */ + time_t time (time_t *__timer); + + /* Format TP into S according to FORMAT. + Write no more than MAXSIZE characters and return the number + of characters written, or 0 if it would exceed MAXSIZE. */ + size_t strftime (char * __s, size_t __maxsize, const char * __format, + const struct tm * __tp); + + /* Parse S according to FORMAT and store binary time information in TP. + The return value is a pointer to the first unparsed character in S. */ + char *strptime (const char * __s, const char * __fmt, struct tm *__tp); + + /* Return the `struct tm' representation of *TIMER in UTC, + using *TP to store the result. */ + struct tm *gmtime_r (const time_t * __timer, struct tm * __tp); + + /* Return the `struct tm' representation of *TIMER in local time, + using *TP to store the result. */ + struct tm *localtime_r (const time_t * __timer, struct tm * __tp); + + /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n" + that is the representation of TP in this format. */ + char *asctime (const struct tm *__tp); + + /* Equivalent to `asctime (localtime (timer))'. */ + char *ctime (const time_t *__timer); + +]] + +local native = ffi.C + +local SECS_PER_DAY = 86400 +local NANOS_PER_SEC = 1000000000LL + +-- c-dt/dt_config.h + +-- Unix, January 1, 1970, Thursday +local DT_EPOCH_1970_OFFSET = 719163LL + + +local datetime_t = ffi.typeof('struct datetime_t') +local interval_t = ffi.typeof('struct datetime_interval_t') + +local function interval_new() + local interval = ffi.new(interval_t) + return interval +end + +local function adjusted_secs(dt) + return dt.secs - dt.offset * 60 +end + +local function datetime_sub(lhs, rhs) + local s1 = adjusted_secs(lhs) + local s2 = adjusted_secs(rhs) + local d = interval_new() + d.secs = s2 - s1 + d.nsec = rhs.nsec - lhs.nsec + if d.nsec < 0 then + d.secs = d.secs - 1 + d.nsec = d.nsec + NANOS_PER_SEC + end + return d +end + +local function datetime_add(lhs, rhs) + local s1 = adjusted_secs(lhs) + local s2 = adjusted_secs(rhs) + local d = interval_new() + d.secs = s2 + s1 + d.nsec = rhs.nsec + lhs.nsec + if d.nsec >= NANOS_PER_SEC then + d.secs = d.secs + 1 + d.nsec = d.nsec - NANOS_PER_SEC + end + return d +end + +local function datetime_eq(lhs, rhs) + -- we usually don't need to check nullness + -- but older tarantool console will call us checking for equality to nil + if rhs == nil then + return false + end + return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) +end + + +local function datetime_lt(lhs, rhs) + return (lhs.secs < rhs.secs) or + (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) +end + +local function datetime_le(lhs, rhs) + return (lhs.secs <= rhs.secs) or + (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) +end + +local function datetime_serialize(self) + -- Allow YAML, MsgPack and JSON to dump objects with sockets + return { secs = self.secs, nsec = self.nsec, tz = self.offset } +end + +local function interval_serialize(self) + -- Allow YAML and JSON to dump objects with sockets + return { secs = self.secs, nsec = self.nsec } +end + +local datetime_mt = { + -- __tostring = datetime_tostring, + __serialize = datetime_serialize, + __eq = datetime_eq, + __lt = datetime_lt, + __le = datetime_le, + __sub = datetime_sub, + __add = datetime_add, + + nanoseconds = function(self) + return tonumber(self.secs*NANOS_PER_SEC + self.nsec) + end, + microseconds = function(self) + return tonumber(self.secs*1e6 + self.nsec*1e3) + end, + seconds = function(self) + return tonumber(self.secs + self.nsec*1e3) + end, + minutes = function(self) + return tonumber((self._ticks/(1e6*60))%60) + end, + hours = function(self) + return tonumber(self._ticks/(1e6*60*60)) + end, + +} + +local interval_mt = { + -- __tostring = interval_tostring, + __serialize = interval_serialize, + __eq = datetime_eq, + __lt = datetime_lt, + __le = datetime_le, +} + +local function datetime_new_raw(secs, nsec, offset) + local dt_obj = ffi.new(datetime_t) + dt_obj.secs = secs + dt_obj.nsec = nsec + dt_obj.offset = offset + return dt_obj +end + +local function local_rd(o) + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET +end + +local function local_dt(o) + return cdt.dt_from_rdn(local_rd(o)) +end + +local function mk_timestamp(dt, sp, fp, offset) + local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0 + local spV = sp ~= nil and sp or 0 + local fpV = fp ~= nil and fp or 0 + local ofsV = offset ~= nil and offset or 0 + return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV) +end + +-- create @datetime_t given object @o fields +local function datetime_new(o) + if o == nil then + return datetime_new_raw(0, 0, 0) + end + local secs = 0 + local nsec = 0 + local offset = 0 + local easy_way = false + local y, M, d, ymd + y, M, d, ymd = 0, 0, 0, false + + local h, m, s, frac, hms + h, m, s, frac, hms = 0, 0, 0, 0, false + + local dt = 0 + + for key, value in pairs(o) do + local handlers = { + secs = function(v) + secs = v + easy_way = true + end, + + nsec = function(v) + nsec = v + easy_way = true + end, + + offset = function (v) + offset = v + easy_way = true + end, + + year = function(v) + assert(v > 0 and v < 10000) + y = v + ymd = true + end, + + month = function(v) + assert(v > 0 and v < 12 ) + M = v + ymd = true + end, + + day = function(v) + assert(v > 0 and v < 32) + d = v + ymd = true + end, + + hour = function(v) + assert(v >= 0 and v < 24) + h = v + hms = true + end, + + minute = function(v) + assert(v >= 0 and v < 60) + m = v + hms = true + end, + + second = function(v) + assert(v >= 0 and v < 61) + frac = v % 1 + if frac then + s = v - (v % 1) + else + s = v + end + hms = true + end, + + -- tz offset in minutes + tz = function(v) + assert(v >= 0 and v <= 720) + offset = v + end + } + handlers[key](value) + end + + -- .sec, .nsec, .offset + if easy_way then + return datetime_new_raw(secs, nsec, offset) + end + + -- .year, .month, .day + if ymd then + dt = dt + cdt.dt_from_ymd(y, M, d) + end + + -- .hour, .minute, .second + if hms then + secs = h * 3600 + m * 60 + s + end + + return mk_timestamp(dt, secs, frac, offset) +end + + +-- simple parse functions: +-- parse_date/parse_time/parse_zone + +--[[ + Basic Extended + 20121224 2012-12-24 Calendar date (ISO 8601) + 2012359 2012-359 Ordinal date (ISO 8601) + 2012W521 2012-W52-1 Week date (ISO 8601) + 2012Q485 2012-Q4-85 Quarter date +]] + +local function parse_date(str) + local dt = ffi.new('dt_t[1]') + local rc = cdt.dt_parse_iso_date(str, #str, dt) + assert(rc > 0) + return mk_timestamp(dt[0]) +end + +--[[ + Basic Extended + T12 N/A + T1230 T12:30 + T123045 T12:30:45 + T123045.123456789 T12:30:45.123456789 + T123045,123456789 T12:30:45,123456789 + + The time designator [T] may be omitted. +]] +local function parse_time(str) + local sp = ffi.new('int[1]') + local fp = ffi.new('int[1]') + local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) + assert(rc > 0) + return mk_timestamp(nil, sp[0], fp[0]) +end + +--[[ + Basic Extended + Z N/A + ±hh N/A + ±hhmm ±hh:mm +]] +local function parse_zone(str) + local offset = ffi.new('int[1]') + local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) + return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len) +end + + +--[[ + aggregated parse functions + assumes to deal with date T time time_zone + at once + + date [T] time [ ] time_zone +]] +local function parse_str(str) + local dt = ffi.new('dt_t[1]') + local len = #str + local n = cdt.dt_parse_iso_date(str, len, dt) + local dt_ = dt[0] + if n == 0 or len == n then + return mk_timestamp(dt_) + end + + str = str:sub(tonumber(n) + 1) + + local ch = str:sub(1,1) + if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then + return mk_timestamp(dt_) + end + + str = str:sub(2) + len = #str + + local sp = ffi.new('int[1]') + local fp = ffi.new('int[1]') + local n = cdt.dt_parse_iso_time(str, len, sp, fp) + if n == 0 then + return mk_timestamp(dt_) + end + local sp_ = sp[0] + local fp_ = fp[0] + if len == n then + return mk_timestamp(dt_, sp_, fp_) + end + + str = str:sub(tonumber(n) + 1) + + if str:sub(1,1) == ' ' then + str = str:sub(2) + end + + len = #str + + local offset = ffi.new('int[1]') + n = cdt.dt_parse_iso_zone_lenient(str, len, offset) + if n == 0 then + return mk_timestamp(dt_, sp_, fp_) + end + return mk_timestamp(dt_, sp_, fp_, offset[0]) +end + +local function datetime_from(o) + if o == nil or type(o) == 'table' then + return datetime_new(o) + elseif type(o) == 'string' then + return parse_str(o) + end +end + +local function local_now() + local p_tv = ffi.new ' struct timeval [1] ' + local rc = native.gettimeofday(p_tv, nil) + assert(rc == 0) + + local secs = p_tv[0].tv_sec + local nsec = p_tv[0].tv_usec * 1000 + + local p_time = ffi.new 'time_t[1]' + local p_tm = ffi.new 'struct tm[1]' + native.time(p_time) + native.localtime_r(p_time, p_tm) + -- local dt = cdt.dt_from_struct_tm(p_tm) + local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes + + return datetime_new_raw(secs, nsec, ofs) -- FIXME +end + +local function datetime_to_tm_ptr(o) + local p_tm = ffi.new 'struct tm[1]' + assert(ffi.typeof(o) == datetime_t) + -- dt_to_struct_tm() fills only date data + cdt.dt_to_struct_tm(local_dt(o), p_tm) + + -- calculate the smaller data (hour, minute, + -- seconds) using datetime seconds value + local seconds_of_day = o.secs % 86400 + local hour = (seconds_of_day / 3600) % 24 + local minute = (seconds_of_day / 60) % 60 + p_tm[0].tm_sec = seconds_of_day % 60 + p_tm[0].tm_min = minute + p_tm[0].tm_hour = hour + + p_tm[0].tm_gmtoff = o.offset * 60 + + return p_tm +end + +local function asctime(o) + assert(ffi.typeof(o) == datetime_t) + local p_tm = datetime_to_tm_ptr(o) + return ffi.string(native.asctime(p_tm)) +end + +local function ctime(o) + assert(ffi.typeof(o) == datetime_t) + local p_time = ffi.new 'time_t[1]' + p_time[0] = o.secs + return ffi.string(native.ctime(p_time)) +end + +local function strftime(fmt, o) + assert(ffi.typeof(o) == datetime_t) + local sz = 50 + local buff = ffi.new('char[?]', sz) + local p_tm = datetime_to_tm_ptr(o) + native.strftime(buff, sz, fmt, p_tm) + return ffi.string(buff) +end + +-- strftime may be redirected to datetime:fmt("format") +local function datetime_fmt() +end + + +ffi.metatype(interval_t, interval_mt) +ffi.metatype(datetime_t, datetime_mt) + +return setmetatable( + { + datetime = datetime_new, + interval = interval_new, + + parse = parse_str, + parse_date = parse_date, + parse_time = parse_time, + parse_zone = parse_zone, + fmt = datetime_fmt, + + now = local_now, + -- strptime = strptime; + strftime = strftime, + asctime = asctime, + ctime = ctime, + }, { + __call = function(self, ...) return datetime_from(...) end + } +) diff --git a/src/lua/init.c b/src/lua/init.c index f9738025d..127e935d7 100644 --- a/src/lua/init.c +++ b/src/lua/init.c @@ -129,7 +129,8 @@ extern char strict_lua[], parse_lua[], process_lua[], humanize_lua[], - memprof_lua[] + memprof_lua[], + datetime_lua[] ; static const char *lua_modules[] = { @@ -184,6 +185,7 @@ static const char *lua_modules[] = { "memprof.process", process_lua, "memprof.humanize", humanize_lua, "memprof", memprof_lua, + "datetime", datetime_lua, NULL }; diff --git a/src/lua/utils.c b/src/lua/utils.c index 34cec0eed..6aeb18cfe 100644 --- a/src/lua/utils.c +++ b/src/lua/utils.c @@ -47,6 +47,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR; static uint32_t CTID_CHAR_PTR; static uint32_t CTID_CONST_CHAR_PTR; uint32_t CTID_UUID; +uint32_t CTID_DATETIME = 0; +uint32_t CTID_INTERVAL = 0; + void * luaL_pushcdata(struct lua_State *L, uint32_t ctypeid) @@ -107,6 +110,12 @@ luaL_pushuuid(struct lua_State *L) return luaL_pushcdata(L, CTID_UUID); } +struct datetime_t * +luaL_pushdatetime(struct lua_State *L) +{ + return luaL_pushcdata(L, CTID_DATETIME); +} + int luaL_iscdata(struct lua_State *L, int idx) { @@ -712,6 +721,24 @@ tarantool_lua_utils_init(struct lua_State *L) CTID_UUID = luaL_ctypeid(L, "struct tt_uuid"); assert(CTID_UUID != 0); + rc = luaL_cdef(L, "struct datetime_t {" + "int64_t secs;" + "int32_t nsec;" + "int32_t offset;" + "};"); + assert(rc == 0); + (void) rc; + CTID_DATETIME = luaL_ctypeid(L, "struct datetime_t"); + assert(CTID_DATETIME != 0); + rc = luaL_cdef(L, "struct datetime_interval_t {" + "int64_t secs;" + "int32_t nsec;" + "};"); + assert(rc == 0); + (void) rc; + CTID_INTERVAL = luaL_ctypeid(L, "struct datetime_interval_t"); + assert(CTID_INTERVAL != 0); + lua_pushcfunction(L, luaT_newthread_wrapper); luaT_newthread_ref = luaL_ref(L, LUA_REGISTRYINDEX); return 0; diff --git a/src/lua/utils.h b/src/lua/utils.h index 947d9240b..afd41c75b 100644 --- a/src/lua/utils.h +++ b/src/lua/utils.h @@ -59,6 +59,7 @@ struct lua_State; struct ibuf; typedef struct ibuf box_ibuf_t; struct tt_uuid; +struct datetime_t; /** * Single global lua_State shared by core and modules. @@ -69,10 +70,21 @@ struct tt_uuid; extern struct lua_State *tarantool_L; extern uint32_t CTID_UUID; +extern uint32_t CTID_DATETIME; +extern uint32_t CTID_INTERVAL; struct tt_uuid * luaL_pushuuid(struct lua_State *L); +/** + * @brief Push cdata of a datetime_t type onto the stack. + * @param L Lua State + * @sa luaL_pushcdata + * @return memory associated with this datetime_t data + */ +struct datetime_t * +luaL_pushdatetime(struct lua_State *L); + /** \cond public */ /** diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt index 5bb7cd6e7..f8320aebd 100644 --- a/test/unit/CMakeLists.txt +++ b/test/unit/CMakeLists.txt @@ -56,6 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c) target_link_libraries(uuid.test uuid unit) add_executable(random.test random.c core_test_utils.c) target_link_libraries(random.test core unit) +add_executable(datetime.test datetime.c) +target_link_libraries(datetime.test cdt unit) add_executable(bps_tree.test bps_tree.cc) target_link_libraries(bps_tree.test small misc) diff --git a/test/unit/datetime.c b/test/unit/datetime.c new file mode 100644 index 000000000..b6f568c03 --- /dev/null +++ b/test/unit/datetime.c @@ -0,0 +1,221 @@ +#include "dt.h" +#include <assert.h> +#include <stdint.h> +#include <string.h> +#include <time.h> + +#include "unit.h" + +const char sample[] = "2012-12-24T15:30Z"; + +#define S(s) {s, sizeof(s) - 1} +struct { + const char * sz; + size_t len; +} tests[] = { + S("2012-12-24 15:30Z"), + S("2012-12-24 15:30z"), + S("2012-12-24 15:30"), + S("2012-12-24 16:30+01:00"), + S("2012-12-24 16:30+0100"), + S("2012-12-24 16:30+01"), + S("2012-12-24 14:30-01:00"), + S("2012-12-24 14:30-0100"), + S("2012-12-24 14:30-01"), + S("2012-12-24 15:30:00Z"), + S("2012-12-24 15:30:00z"), + S("2012-12-24 15:30:00"), + S("2012-12-24 16:30:00+01:00"), + S("2012-12-24 16:30:00+0100"), + S("2012-12-24 14:30:00-01:00"), + S("2012-12-24 14:30:00-0100"), + S("2012-12-24 15:30:00.123456Z"), + S("2012-12-24 15:30:00.123456z"), + S("2012-12-24 15:30:00.123456"), + S("2012-12-24 16:30:00.123456+01:00"), + S("2012-12-24 16:30:00.123456+01"), + S("2012-12-24 14:30:00.123456-01:00"), + S("2012-12-24 14:30:00.123456-01"), + S("2012-12-24t15:30Z"), + S("2012-12-24t15:30z"), + S("2012-12-24t15:30"), + S("2012-12-24t16:30+01:00"), + S("2012-12-24t16:30+0100"), + S("2012-12-24t14:30-01:00"), + S("2012-12-24t14:30-0100"), + S("2012-12-24t15:30:00Z"), + S("2012-12-24t15:30:00z"), + S("2012-12-24t15:30:00"), + S("2012-12-24t16:30:00+01:00"), + S("2012-12-24t16:30:00+0100"), + S("2012-12-24t14:30:00-01:00"), + S("2012-12-24t14:30:00-0100"), + S("2012-12-24t15:30:00.123456Z"), + S("2012-12-24t15:30:00.123456z"), + S("2012-12-24t16:30:00.123456+01:00"), + S("2012-12-24t14:30:00.123456-01:00"), + S("2012-12-24 16:30 +01:00"), + S("2012-12-24 14:30 -01:00"), + S("2012-12-24 15:30 UTC"), + S("2012-12-24 16:30 UTC+1"), + S("2012-12-24 16:30 UTC+01"), + S("2012-12-24 16:30 UTC+0100"), + S("2012-12-24 16:30 UTC+01:00"), + S("2012-12-24 14:30 UTC-1"), + S("2012-12-24 14:30 UTC-01"), + S("2012-12-24 14:30 UTC-01:00"), + S("2012-12-24 14:30 UTC-0100"), + S("2012-12-24 15:30 GMT"), + S("2012-12-24 16:30 GMT+1"), + S("2012-12-24 16:30 GMT+01"), + S("2012-12-24 16:30 GMT+0100"), + S("2012-12-24 16:30 GMT+01:00"), + S("2012-12-24 14:30 GMT-1"), + S("2012-12-24 14:30 GMT-01"), + S("2012-12-24 14:30 GMT-01:00"), + S("2012-12-24 14:30 GMT-0100"), + S("2012-12-24 14:30 -01:00"), + S("2012-12-24 16:30:00 +01:00"), + S("2012-12-24 14:30:00 -01:00"), + S("2012-12-24 16:30:00.123456 +01:00"), + S("2012-12-24 14:30:00.123456 -01:00"), + S("2012-12-24 15:30:00.123456 -00:00"), + S("20121224T1630+01:00"), + S("2012-12-24T1630+01:00"), + S("20121224T16:30+01"), + S("20121224T16:30 +01"), +}; +#undef S + +#define DIM(a) (sizeof(a) / sizeof(a[0])) + +// p5-time-moment/src/moment_parse.c: parse_string_lenient() +static int +parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np, + int32_t *op) +{ + size_t n; + dt_t dt; + char c; + int sod = 0, nanosecond = 0, offset = 0; + + n = dt_parse_iso_date(str, len, &dt); + if (!n) + return 1; + if (n == len) + goto exit; + + c = str[n++]; + if (!(c == 'T' || c == 't' || c == ' ')) + return 1; + + str += n; + len -= n; + + n = dt_parse_iso_time(str, len, &sod, &nanosecond); + if (!n) + return 1; + if (n == len) + goto exit; + + if (str[n] == ' ') + n++; + + str += n; + len -= n; + + n = dt_parse_iso_zone_lenient(str, len, &offset); + if (!n || n != len) + return 1; + +exit: + *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60; + *np = nanosecond; + *op = offset; + + return 0; +} + +// avoid introducing external datetime.h dependency +// - just copy paste it for today +#define SECS_PER_DAY 86400 +#define NANOS_PER_SEC 1000000000 +#define DT_EPOCH_1970_OFFSET 719163 + + +struct datetime_t { + int64_t secs; + int32_t nsec; + int32_t offset; +}; + +static int +local_rd(const struct datetime_t * dt) { + return (int)(dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET; +} + +static int +local_dt(const struct datetime_t * dt) { + return dt_from_rdn(local_rd(dt)); +} + + +struct tm* +datetime_to_tm(struct datetime_t * dt) +{ + static struct tm tm; + + memset(&tm, 0, sizeof(tm)); + dt_to_struct_tm(local_dt(dt), &tm); + + int seconds_of_day = dt->secs % 86400; + tm.tm_hour = (seconds_of_day / 3600) % 24; + tm.tm_min = (seconds_of_day / 60) % 60; + tm.tm_sec = seconds_of_day % 60; + + return &tm; +} + +static void datetime_test(void) +{ + size_t index; + int64_t secs_expected; + int32_t nanosecs; + int32_t ofs; + + plan(355); + parse_datetime(sample, sizeof(sample) - 1, + &secs_expected, &nanosecs, &ofs); + + for (index = 0; index < DIM(tests); index++) { + int64_t secs; + int rc = parse_datetime(tests[index].sz, tests[index].len, + &secs, &nanosecs, &ofs); + is(rc, 0, "correct parse_datetime return value for '%s'", + tests[index].sz); + is(secs, secs_expected, "correct parse_datetime output " + "seconds for '%s", tests[index].sz); + + // check that stringized literal produces the same date + // time fields + static char buff[40]; + struct datetime_t dt = {secs, nanosecs, ofs}; + // datetime_to_tm returns time in GMT zone + struct tm * p_tm = datetime_to_tm(&dt); + size_t len = strftime(buff, sizeof buff, "%F %T%z", p_tm); + ok(len > 0, "strftime"); + rc = parse_datetime(buff, len, &dt.secs, &dt.nsec, &dt.offset); + is(rc, 0, "correct parse_datetime return value for '%s'", buff); + is(secs, dt.secs, + "reversible seconds via strftime for '%s", buff); + } +} + +int +main(void) +{ + plan(1); + datetime_test(); + + return check_plan(); +} diff --git a/test/unit/datetime.result b/test/unit/datetime.result new file mode 100644 index 000000000..f106fa769 --- /dev/null +++ b/test/unit/datetime.result @@ -0,0 +1,358 @@ +1..1 + 1..355 + ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z' + ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z + ok 3 - strftime + ok 4 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 5 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 6 - correct parse_datetime return value for '2012-12-24 15:30z' + ok 7 - correct parse_datetime output seconds for '2012-12-24 15:30z + ok 8 - strftime + ok 9 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 10 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 11 - correct parse_datetime return value for '2012-12-24 15:30' + ok 12 - correct parse_datetime output seconds for '2012-12-24 15:30 + ok 13 - strftime + ok 14 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 15 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 16 - correct parse_datetime return value for '2012-12-24 16:30+01:00' + ok 17 - correct parse_datetime output seconds for '2012-12-24 16:30+01:00 + ok 18 - strftime + ok 19 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 20 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 21 - correct parse_datetime return value for '2012-12-24 16:30+0100' + ok 22 - correct parse_datetime output seconds for '2012-12-24 16:30+0100 + ok 23 - strftime + ok 24 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 25 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 26 - correct parse_datetime return value for '2012-12-24 16:30+01' + ok 27 - correct parse_datetime output seconds for '2012-12-24 16:30+01 + ok 28 - strftime + ok 29 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 30 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 31 - correct parse_datetime return value for '2012-12-24 14:30-01:00' + ok 32 - correct parse_datetime output seconds for '2012-12-24 14:30-01:00 + ok 33 - strftime + ok 34 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 35 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 36 - correct parse_datetime return value for '2012-12-24 14:30-0100' + ok 37 - correct parse_datetime output seconds for '2012-12-24 14:30-0100 + ok 38 - strftime + ok 39 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 40 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 41 - correct parse_datetime return value for '2012-12-24 14:30-01' + ok 42 - correct parse_datetime output seconds for '2012-12-24 14:30-01 + ok 43 - strftime + ok 44 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 45 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 46 - correct parse_datetime return value for '2012-12-24 15:30:00Z' + ok 47 - correct parse_datetime output seconds for '2012-12-24 15:30:00Z + ok 48 - strftime + ok 49 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 50 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 51 - correct parse_datetime return value for '2012-12-24 15:30:00z' + ok 52 - correct parse_datetime output seconds for '2012-12-24 15:30:00z + ok 53 - strftime + ok 54 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 55 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 56 - correct parse_datetime return value for '2012-12-24 15:30:00' + ok 57 - correct parse_datetime output seconds for '2012-12-24 15:30:00 + ok 58 - strftime + ok 59 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 60 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 61 - correct parse_datetime return value for '2012-12-24 16:30:00+01:00' + ok 62 - correct parse_datetime output seconds for '2012-12-24 16:30:00+01:00 + ok 63 - strftime + ok 64 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 65 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 66 - correct parse_datetime return value for '2012-12-24 16:30:00+0100' + ok 67 - correct parse_datetime output seconds for '2012-12-24 16:30:00+0100 + ok 68 - strftime + ok 69 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 70 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 71 - correct parse_datetime return value for '2012-12-24 14:30:00-01:00' + ok 72 - correct parse_datetime output seconds for '2012-12-24 14:30:00-01:00 + ok 73 - strftime + ok 74 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 75 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 76 - correct parse_datetime return value for '2012-12-24 14:30:00-0100' + ok 77 - correct parse_datetime output seconds for '2012-12-24 14:30:00-0100 + ok 78 - strftime + ok 79 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 80 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 81 - correct parse_datetime return value for '2012-12-24 15:30:00.123456Z' + ok 82 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456Z + ok 83 - strftime + ok 84 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 85 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 86 - correct parse_datetime return value for '2012-12-24 15:30:00.123456z' + ok 87 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456z + ok 88 - strftime + ok 89 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 90 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 91 - correct parse_datetime return value for '2012-12-24 15:30:00.123456' + ok 92 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 + ok 93 - strftime + ok 94 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 95 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 96 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01:00' + ok 97 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01:00 + ok 98 - strftime + ok 99 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 100 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 101 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01' + ok 102 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01 + ok 103 - strftime + ok 104 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 105 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 106 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01:00' + ok 107 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01:00 + ok 108 - strftime + ok 109 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 110 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 111 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01' + ok 112 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01 + ok 113 - strftime + ok 114 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 115 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 116 - correct parse_datetime return value for '2012-12-24t15:30Z' + ok 117 - correct parse_datetime output seconds for '2012-12-24t15:30Z + ok 118 - strftime + ok 119 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 120 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 121 - correct parse_datetime return value for '2012-12-24t15:30z' + ok 122 - correct parse_datetime output seconds for '2012-12-24t15:30z + ok 123 - strftime + ok 124 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 125 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 126 - correct parse_datetime return value for '2012-12-24t15:30' + ok 127 - correct parse_datetime output seconds for '2012-12-24t15:30 + ok 128 - strftime + ok 129 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 130 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 131 - correct parse_datetime return value for '2012-12-24t16:30+01:00' + ok 132 - correct parse_datetime output seconds for '2012-12-24t16:30+01:00 + ok 133 - strftime + ok 134 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 135 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 136 - correct parse_datetime return value for '2012-12-24t16:30+0100' + ok 137 - correct parse_datetime output seconds for '2012-12-24t16:30+0100 + ok 138 - strftime + ok 139 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 140 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 141 - correct parse_datetime return value for '2012-12-24t14:30-01:00' + ok 142 - correct parse_datetime output seconds for '2012-12-24t14:30-01:00 + ok 143 - strftime + ok 144 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 145 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 146 - correct parse_datetime return value for '2012-12-24t14:30-0100' + ok 147 - correct parse_datetime output seconds for '2012-12-24t14:30-0100 + ok 148 - strftime + ok 149 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 150 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 151 - correct parse_datetime return value for '2012-12-24t15:30:00Z' + ok 152 - correct parse_datetime output seconds for '2012-12-24t15:30:00Z + ok 153 - strftime + ok 154 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 155 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 156 - correct parse_datetime return value for '2012-12-24t15:30:00z' + ok 157 - correct parse_datetime output seconds for '2012-12-24t15:30:00z + ok 158 - strftime + ok 159 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 160 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 161 - correct parse_datetime return value for '2012-12-24t15:30:00' + ok 162 - correct parse_datetime output seconds for '2012-12-24t15:30:00 + ok 163 - strftime + ok 164 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 165 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 166 - correct parse_datetime return value for '2012-12-24t16:30:00+01:00' + ok 167 - correct parse_datetime output seconds for '2012-12-24t16:30:00+01:00 + ok 168 - strftime + ok 169 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 170 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 171 - correct parse_datetime return value for '2012-12-24t16:30:00+0100' + ok 172 - correct parse_datetime output seconds for '2012-12-24t16:30:00+0100 + ok 173 - strftime + ok 174 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 175 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 176 - correct parse_datetime return value for '2012-12-24t14:30:00-01:00' + ok 177 - correct parse_datetime output seconds for '2012-12-24t14:30:00-01:00 + ok 178 - strftime + ok 179 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 180 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 181 - correct parse_datetime return value for '2012-12-24t14:30:00-0100' + ok 182 - correct parse_datetime output seconds for '2012-12-24t14:30:00-0100 + ok 183 - strftime + ok 184 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 185 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 186 - correct parse_datetime return value for '2012-12-24t15:30:00.123456Z' + ok 187 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456Z + ok 188 - strftime + ok 189 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 190 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 191 - correct parse_datetime return value for '2012-12-24t15:30:00.123456z' + ok 192 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456z + ok 193 - strftime + ok 194 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 195 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 196 - correct parse_datetime return value for '2012-12-24t16:30:00.123456+01:00' + ok 197 - correct parse_datetime output seconds for '2012-12-24t16:30:00.123456+01:00 + ok 198 - strftime + ok 199 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 200 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 201 - correct parse_datetime return value for '2012-12-24t14:30:00.123456-01:00' + ok 202 - correct parse_datetime output seconds for '2012-12-24t14:30:00.123456-01:00 + ok 203 - strftime + ok 204 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 205 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 206 - correct parse_datetime return value for '2012-12-24 16:30 +01:00' + ok 207 - correct parse_datetime output seconds for '2012-12-24 16:30 +01:00 + ok 208 - strftime + ok 209 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 210 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 211 - correct parse_datetime return value for '2012-12-24 14:30 -01:00' + ok 212 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00 + ok 213 - strftime + ok 214 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 215 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 216 - correct parse_datetime return value for '2012-12-24 15:30 UTC' + ok 217 - correct parse_datetime output seconds for '2012-12-24 15:30 UTC + ok 218 - strftime + ok 219 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 220 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 221 - correct parse_datetime return value for '2012-12-24 16:30 UTC+1' + ok 222 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+1 + ok 223 - strftime + ok 224 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 225 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 226 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01' + ok 227 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01 + ok 228 - strftime + ok 229 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 230 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 231 - correct parse_datetime return value for '2012-12-24 16:30 UTC+0100' + ok 232 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+0100 + ok 233 - strftime + ok 234 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 235 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 236 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01:00' + ok 237 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01:00 + ok 238 - strftime + ok 239 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 240 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 241 - correct parse_datetime return value for '2012-12-24 14:30 UTC-1' + ok 242 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-1 + ok 243 - strftime + ok 244 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 245 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 246 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01' + ok 247 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01 + ok 248 - strftime + ok 249 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 250 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 251 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01:00' + ok 252 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01:00 + ok 253 - strftime + ok 254 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 255 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 256 - correct parse_datetime return value for '2012-12-24 14:30 UTC-0100' + ok 257 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-0100 + ok 258 - strftime + ok 259 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 260 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 261 - correct parse_datetime return value for '2012-12-24 15:30 GMT' + ok 262 - correct parse_datetime output seconds for '2012-12-24 15:30 GMT + ok 263 - strftime + ok 264 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 265 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 266 - correct parse_datetime return value for '2012-12-24 16:30 GMT+1' + ok 267 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+1 + ok 268 - strftime + ok 269 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 270 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 271 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01' + ok 272 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01 + ok 273 - strftime + ok 274 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 275 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 276 - correct parse_datetime return value for '2012-12-24 16:30 GMT+0100' + ok 277 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+0100 + ok 278 - strftime + ok 279 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 280 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 281 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01:00' + ok 282 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01:00 + ok 283 - strftime + ok 284 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 285 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 286 - correct parse_datetime return value for '2012-12-24 14:30 GMT-1' + ok 287 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-1 + ok 288 - strftime + ok 289 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 290 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 291 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01' + ok 292 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01 + ok 293 - strftime + ok 294 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 295 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 296 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01:00' + ok 297 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01:00 + ok 298 - strftime + ok 299 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 300 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 301 - correct parse_datetime return value for '2012-12-24 14:30 GMT-0100' + ok 302 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-0100 + ok 303 - strftime + ok 304 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 305 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 306 - correct parse_datetime return value for '2012-12-24 14:30 -01:00' + ok 307 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00 + ok 308 - strftime + ok 309 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 310 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 311 - correct parse_datetime return value for '2012-12-24 16:30:00 +01:00' + ok 312 - correct parse_datetime output seconds for '2012-12-24 16:30:00 +01:00 + ok 313 - strftime + ok 314 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 315 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 316 - correct parse_datetime return value for '2012-12-24 14:30:00 -01:00' + ok 317 - correct parse_datetime output seconds for '2012-12-24 14:30:00 -01:00 + ok 318 - strftime + ok 319 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 320 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 321 - correct parse_datetime return value for '2012-12-24 16:30:00.123456 +01:00' + ok 322 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456 +01:00 + ok 323 - strftime + ok 324 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 325 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 326 - correct parse_datetime return value for '2012-12-24 14:30:00.123456 -01:00' + ok 327 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456 -01:00 + ok 328 - strftime + ok 329 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 330 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 331 - correct parse_datetime return value for '2012-12-24 15:30:00.123456 -00:00' + ok 332 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 -00:00 + ok 333 - strftime + ok 334 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 335 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 336 - correct parse_datetime return value for '20121224T1630+01:00' + ok 337 - correct parse_datetime output seconds for '20121224T1630+01:00 + ok 338 - strftime + ok 339 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 340 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 341 - correct parse_datetime return value for '2012-12-24T1630+01:00' + ok 342 - correct parse_datetime output seconds for '2012-12-24T1630+01:00 + ok 343 - strftime + ok 344 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 345 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 346 - correct parse_datetime return value for '20121224T16:30+01' + ok 347 - correct parse_datetime output seconds for '20121224T16:30+01 + ok 348 - strftime + ok 349 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 350 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 + ok 351 - correct parse_datetime return value for '20121224T16:30 +01' + ok 352 - correct parse_datetime output seconds for '20121224T16:30 +01 + ok 353 - strftime + ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' + ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 +ok 1 - subtests -- 2.29.2
* created app-tap test for new builtin module `datetime.lua` * added case to check datetime string formatting using: - asctime (gmt time); - ctime (local TZ time); - strftime (using given format). * added positive/negative checks to datetime test - extended api of datetime.parse_date, .parse_time, .parse_time_zone with a length of parsed (sub)string; - this allows us to check partially valid strings like "20121224 Foo bar". Part of #5941 --- src/lua/datetime.lua | 14 ++- test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++ 2 files changed, 197 insertions(+), 8 deletions(-) create mode 100755 test/app-tap/datetime.test.lua diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua index 0996ca5a2..f4d2d7737 100644 --- a/src/lua/datetime.lua +++ b/src/lua/datetime.lua @@ -318,7 +318,7 @@ local function datetime_new(o) end, month = function(v) - assert(v > 0 and v < 12 ) + assert(v > 0 and v < 13 ) M = v ymd = true end, @@ -393,9 +393,8 @@ end local function parse_date(str) local dt = ffi.new('dt_t[1]') - local rc = cdt.dt_parse_iso_date(str, #str, dt) - assert(rc > 0) - return mk_timestamp(dt[0]) + local len = cdt.dt_parse_iso_date(str, #str, dt) + return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len) end --[[ @@ -411,9 +410,8 @@ end local function parse_time(str) local sp = ffi.new('int[1]') local fp = ffi.new('int[1]') - local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) - assert(rc > 0) - return mk_timestamp(nil, sp[0], fp[0]) + local len = cdt.dt_parse_iso_time(str, #str, sp, fp) + return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil, tonumber(len) end --[[ @@ -448,7 +446,7 @@ local function parse_str(str) str = str:sub(tonumber(n) + 1) local ch = str:sub(1,1) - if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then + if ch:match('[Tt ]') == nil then return mk_timestamp(dt_) end diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua new file mode 100755 index 000000000..09c968858 --- /dev/null +++ b/test/app-tap/datetime.test.lua @@ -0,0 +1,191 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local test = tap.test("errno") +local date = require('datetime') + +test:plan(5) + +test:test("Simple tests for parser", function(test) + test:plan(2) + test:ok(date("1970-01-01T01:00:00Z") == + date {year=1970, month=1, day=1, hour=1, minute=0, second=0}) + test:ok(date("1970-01-01T02:00:00+02:00") == + date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120}) +end) + +test:test("Multiple tests for parser (with nanoseconds)", function(test) + test:plan(165) + -- borrowed from p5-time-moments/t/180_from_string.t + local tests = + { + { '1970-01-01T00:00:00Z', 0, 0, 0 }, + { '1970-01-01T02:00:00+02:00', 0, 0, 120 }, + { '1970-01-01T01:30:00+01:30', 0, 0, 90 }, + { '1970-01-01T01:00:00+01:00', 0, 0, 60 }, + { '1970-01-01T00:01:00+00:01', 0, 0, 1 }, + { '1970-01-01T00:00:00+00:00', 0, 0, 0 }, + { '1969-12-31T23:59:00-00:01', 0, 0, -1 }, + { '1969-12-31T23:00:00-01:00', 0, 0, -60 }, + { '1969-12-31T22:30:00-01:30', 0, 0, -90 }, + { '1969-12-31T22:00:00-02:00', 0, 0, -120 }, + { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 }, + { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 }, + { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 }, + { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 }, + { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 }, + { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 }, + { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 }, + { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 }, + { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 }, + { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 }, + { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 }, + { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 }, + { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 }, + { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 }, + { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 }, + { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 }, + { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 }, + { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 }, + { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 }, + { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 }, + { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 }, + { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 }, + { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 }, + { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 }, + { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 }, + { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 }, + { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 }, + { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 }, + { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 }, + { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 }, + { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 }, + { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 }, + { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 }, + { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 }, + { '1970-01-01T00:00:00.0Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.00Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.0000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.00000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.000000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 }, + { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 }, + { '1973-11-29T21:33:09Z', 123456789, 0, 0 }, + { '2013-10-28T17:51:56Z', 1382982716, 0, 0 }, + -- { '9999-12-31T23:59:59Z', 253402300799, 0, 0 }, + } + for _, value in ipairs(tests) do + local str, epoch, nsec, offset + str, epoch, nsec, offset = unpack(value) + local dt = date(str) + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch)) + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec)) + test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset)) + end +end) + +local ffi = require('ffi') + +ffi.cdef [[ + void tzset(void); +]] + +test:test("Datetime string formatting", function(test) + test:plan(7) + local str = "1970-01-01" + local t = date(str) + test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, tonumber(t.secs))) + test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec)) + test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset)) + test:ok(date.asctime(t) == 'Thu Jan 1 00:00:00 1970\n', ('%s: asctime'):format(str)) + -- ctime() is local timezone dependent. To make sure that + -- test is deterministic we enforce timezone via TZ environment + -- manipulations and calling tzset() + + -- redefine timezone to be always GMT-2 + os.setenv('TZ', 'GMT-2') + ffi.C.tzset() + test:ok(date.ctime(t) == 'Thu Jan 1 02:00:00 1970\n', ('%s: ctime with timezone'):format(str)) + test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime #1'):format(str)) + test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January 1970', ('%s: strftime #2'):format(str)) +end) + +test:test("Parse iso date - valid strings", function(test) + test:plan(32) + local good = { + {2012, 12, 24, "20121224", 8 }, + {2012, 12, 24, "20121224 Foo bar", 8 }, + {2012, 12, 24, "2012-12-24", 10 }, + {2012, 12, 24, "2012-12-24 23:59:59", 10 }, + {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 }, + {2012, 12, 24, "2012359", 7 }, + {2012, 12, 24, "2012359T235959+0130", 7 }, + {2012, 12, 24, "2012-359", 8 }, + {2012, 12, 24, "2012W521", 8 }, + {2012, 12, 24, "2012-W52-1", 10 }, + {2012, 12, 24, "2012Q485", 8 }, + {2012, 12, 24, "2012-Q4-85", 10 }, + { 1, 1, 1, "0001-Q1-01", 10 }, + { 1, 1, 1, "0001-W01-1", 10 }, + { 1, 1, 1, "0001-01-01", 10 }, + { 1, 1, 1, "0001-001", 8 }, + } + + for _, value in ipairs(good) do + local year, month, day, str, date_part_len; + year, month, day, str, date_part_len = unpack(value) + local expected_date = date{year = year, month = month, day = day} + local date_part, len + date_part, len = date.parse_date(str) + test:ok(len == date_part_len, ('%s: length check %d'):format(str, len)) + test:ok(expected_date == date_part, ('%s: expected date'):format(str)) + end +end) + +test:test("Parse iso date - invalid strings", function(test) + test:plan(62) + local bad = { + "20121232" , -- Invalid day of month + "2012-12-310", -- Invalid day of month + "2012-13-24" , -- Invalid month + "2012367" , -- Invalid day of year + "2012-000" , -- Invalid day of year + "2012W533" , -- Invalid week of year + "2012-W52-8" , -- Invalid day of week + "2012Q495" , -- Invalid day of quarter + "2012-Q5-85" , -- Invalid quarter + "20123670" , -- Trailing digit + "201212320" , -- Trailing digit + "2012-12" , -- Reduced accuracy + "2012-Q4" , -- Reduced accuracy + "2012-Q42" , -- Invalid + "2012-Q1-1" , -- Invalid day of quarter + "2012Q--420" , -- Invalid + "2012-Q-420" , -- Invalid + "2012Q11" , -- Incomplete + "2012Q1234" , -- Trailing digit + "2012W12" , -- Incomplete + "2012W1234" , -- Trailing digit + "2012W-123" , -- Invalid + "2012-W12" , -- Incomplete + "2012-W12-12", -- Trailing digit + "2012U1234" , -- Invalid + "2012-1234" , -- Invalid + "2012-X1234" , -- Invalid + "0000-Q1-01" , -- Year less than 0001 + "0000-W01-1" , -- Year less than 0001 + "0000-01-01" , -- Year less than 0001 + "0000-001" , -- Year less than 0001 + } + + for _, str in ipairs(bad) do + local date_part, len + date_part, len = date.parse_date(str) + test:ok(len == 0, ('%s: length check %d'):format(str, len)) + test:ok(date_part == nil, ('%s: empty date check %s'):format(str, date_part)) + end +end) + +os.exit(test:check() and 0 or 1) -- 2.29.2
* introduced output routine for converting datetime to their default output format. * use this routine for tostring() in datetime.lua Part of #5941 --- src/exports.h | 1 + src/lib/core/CMakeLists.txt | 1 + src/lib/core/datetime.c | 85 +++++++++++++++++++++++++++++++++++++ src/lib/core/datetime.h | 14 ++++++ src/lua/datetime.lua | 18 ++++++-- 5 files changed, 116 insertions(+), 3 deletions(-) create mode 100755 src/lib/core/datetime.c diff --git a/src/exports.h b/src/exports.h index db40c03a4..1a03db636 100644 --- a/src/exports.h +++ b/src/exports.h @@ -552,3 +552,4 @@ EXPORT(dt_parse_iso_zone_extended) EXPORT(dt_parse_iso_zone_lenient) EXPORT(dt_from_struct_tm) EXPORT(dt_to_struct_tm) +EXPORT(datetime_to_string) diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt index 2cd4d0b4f..8bc776b82 100644 --- a/src/lib/core/CMakeLists.txt +++ b/src/lib/core/CMakeLists.txt @@ -30,6 +30,7 @@ set(core_sources decimal.c mp_decimal.c cord_buf.c + datetime.c ) if (TARGET_OS_NETBSD) diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c new file mode 100755 index 000000000..65f813a70 --- /dev/null +++ b/src/lib/core/datetime.c @@ -0,0 +1,85 @@ +/* + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * 1. Redistributions of source code must retain the above + * copyright notice, this list of conditions and the + * following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include <string.h> + +#include "trivia/util.h" +#include "datetime.h" + +int +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len) +{ + char * src = buf; + int offset = date->offset; + int64_t secs = date->secs + offset * 60; + dt_t dt = dt_from_rdn((secs / SECS_PER_DAY) + 719163); + + int year, month, day, sec, ns, sign; + dt_to_ymd(dt, &year, &month, &day); + + int hour = (secs / 3600) % 24, + minute = (secs / 60) % 60; + ; + sec = secs % 60; + ns = date->nsec; + uint32_t sz; + sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d", + year, month, day, hour, minute); + buf += sz; len -= sz; + if (sec || ns) { + sz = snprintf(buf, len, ":%02d", sec); + buf += sz; len -= sz; + if (ns) { + if ((ns % 1000000) == 0) + sz = snprintf(buf, len, ".%03d", ns / 1000000); + else if ((ns % 1000) == 0) + sz = snprintf(buf, len, ".%06d", ns / 1000); + else + sz = snprintf(buf, len, ".%09d", ns); + buf += sz; len -= sz; + } + } + if (offset == 0) { + strncpy(buf, "Z", len); + buf++; + len--; + } + else { + if (offset < 0) + sign = '-', offset = -offset; + else + sign = '+'; + + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60); + buf += sz; len -= sz; + } + return (buf - src); +} diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h index 403bf1c64..fdaff2d27 100644 --- a/src/lib/core/datetime.h +++ b/src/lib/core/datetime.h @@ -38,6 +38,11 @@ extern "C" { #endif /* defined(__cplusplus) */ +#ifndef SECS_PER_DAY +#define SECS_PER_DAY 86400 +#define NANOS_PER_SEC 1000000000 +#endif + /** * datetime structure consisting of: */ @@ -55,6 +60,15 @@ struct datetime_interval_t { int32_t nsec; ///< nanoseconds delta }; +/** + * Convert datetime to string using default format + * @param date source datetime value + * @param buf output character buffer + * @param len size ofoutput buffer + */ +int +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len); + #if defined(__cplusplus) } /* extern "C" */ #endif /* defined(__cplusplus) */ diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua index f4d2d7737..9ec06d8d8 100644 --- a/src/lua/datetime.lua +++ b/src/lua/datetime.lua @@ -53,6 +53,12 @@ ffi.cdef [[ dt_t dt_from_struct_tm (const struct tm *tm); void dt_to_struct_tm (dt_t dt, struct tm *tm); + // mp_datetime.c + + int + datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len); + + // <asm-generic/posix_types.h> typedef long __kernel_long_t; typedef unsigned long __kernel_ulong_t; @@ -549,8 +555,13 @@ local function strftime(fmt, o) return ffi.string(buff) end --- strftime may be redirected to datetime:fmt("format") -local function datetime_fmt() +local function datetime_tostring(o) + assert(ffi.typeof(o) == datetime_t) + local sz = 48 + local buff = ffi.new('char[?]', sz) + local len = native.datetime_to_string(o, buff, sz) + assert(len < sz) + return ffi.string(buff) end @@ -566,7 +577,8 @@ return setmetatable( parse_date = parse_date, parse_time = parse_time, parse_zone = parse_zone, - fmt = datetime_fmt, + + tostring = datetime_tostring, now = local_now, -- strptime = strptime; -- 2.29.2
Serialize datetime_t as newly introduced MP_EXT type. It saves 1 required integer field and upto 2 optional unsigned fields in very compact fashion. - secs is required field; - but nsec, offset are both optional; * json, yaml serialization formats, lua output mode supported; Part of #5941 Part of #5946 --- src/box/field_def.c | 34 ++++--- src/box/field_def.h | 1 + src/box/lua/serialize_lua.c | 7 +- src/box/msgpack.c | 7 +- src/box/tuple_compare.cc | 20 ++++ src/exports.h | 2 + src/lib/core/CMakeLists.txt | 3 +- src/lib/core/datetime.c | 161 ++++++++++++++++++++++++++++++ src/lib/core/datetime.h | 41 +++++++- src/lib/core/mp_extension_types.h | 1 + src/lib/mpstream/mpstream.c | 11 ++ src/lib/mpstream/mpstream.h | 4 + src/lua/msgpack.c | 12 +++ src/lua/msgpackffi.lua | 8 ++ src/lua/serializer.c | 4 + src/lua/serializer.h | 2 + src/lua/utils.c | 1 - test/unit/datetime.c | 1 - third_party/lua-cjson/lua_cjson.c | 8 ++ third_party/lua-yaml/lyaml.cc | 6 +- 20 files changed, 312 insertions(+), 22 deletions(-) diff --git a/src/box/field_def.c b/src/box/field_def.c index 51acb8025..6964e3e9f 100644 --- a/src/box/field_def.c +++ b/src/box/field_def.c @@ -67,11 +67,12 @@ const uint32_t field_mp_type[] = { /* [FIELD_TYPE_VARBINARY] = */ 1U << MP_BIN, /* [FIELD_TYPE_SCALAR] = */ (1U << MP_UINT) | (1U << MP_INT) | (1U << MP_FLOAT) | (1U << MP_DOUBLE) | (1U << MP_STR) | - (1U << MP_BIN) | (1U << MP_BOOL), + (1U << MP_BIN) | (1U << MP_BOOL) | (1U << MP_DATETIME), /* [FIELD_TYPE_DECIMAL] = */ 0, /* only MP_DECIMAL is supported */ /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */ /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY, /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP), + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is supported */ }; const uint32_t field_ext_type[] = { @@ -88,6 +89,7 @@ const uint32_t field_ext_type[] = { /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID, /* [FIELD_TYPE_ARRAY] = */ 0, /* [FIELD_TYPE_MAP] = */ 0, + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME, }; const char *field_type_strs[] = { @@ -104,6 +106,7 @@ const char *field_type_strs[] = { /* [FIELD_TYPE_UUID] = */ "uuid", /* [FIELD_TYPE_ARRAY] = */ "array", /* [FIELD_TYPE_MAP] = */ "map", + /* [FIELD_TYPE_DATETIME] = */ "datetime", }; const char *on_conflict_action_strs[] = { @@ -128,20 +131,21 @@ field_type_by_name_wrapper(const char *str, uint32_t len) * values can be stored in the j type. */ static const bool field_type_compatibility[] = { - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP */ -/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false, -/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false, -/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false, -/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false, -/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false, -/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false, -/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false, -/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false, -/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false, -/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false, -/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false, -/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false, -/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true, + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */ +/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false, false, +/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false, false, +/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false, false, +/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false, false, +/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false, false, +/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false, false, +/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false, false, +/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false, false, +/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false, false, +/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false, false, +/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false, false, +/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false, false, +/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true, false, +/* DATETIME */ true, false, false, false, false, false, false, false, false, false, false, false, false, true, }; bool diff --git a/src/box/field_def.h b/src/box/field_def.h index c5cfe5e86..120b2a93d 100644 --- a/src/box/field_def.h +++ b/src/box/field_def.h @@ -63,6 +63,7 @@ enum field_type { FIELD_TYPE_UUID, FIELD_TYPE_ARRAY, FIELD_TYPE_MAP, + FIELD_TYPE_DATETIME, field_type_MAX }; diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c index 7144305cf..5cfa6f5e0 100644 --- a/src/box/lua/serialize_lua.c +++ b/src/box/lua/serialize_lua.c @@ -768,7 +768,7 @@ static int dump_node(struct lua_dumper *d, struct node *nd, int indent) { struct luaL_field *field = &nd->field; - char buf[FPCONV_G_FMT_BUFSIZE]; + char buf[FPCONV_G_FMT_BUFSIZE + 8]; int ltype = lua_type(d->L, -1); const char *str = NULL; size_t len = 0; @@ -861,6 +861,11 @@ dump_node(struct lua_dumper *d, struct node *nd, int indent) str = tt_uuid_str(field->uuidval); len = UUID_STR_LEN; break; + case MP_DATETIME: + nd->mask |= NODE_QUOTE; + str = buf; + len = datetime_to_string(field->dateval, buf, sizeof buf); + break; default: d->err = EINVAL; snprintf(d->err_msg, sizeof(d->err_msg), diff --git a/src/box/msgpack.c b/src/box/msgpack.c index 1723dea4c..e53af548c 100644 --- a/src/box/msgpack.c +++ b/src/box/msgpack.c @@ -1,5 +1,5 @@ /* - * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file. + * Copyright 2020-2021, Tarantool AUTHORS, please see AUTHORS file. * * Redistribution and use in source and binary forms, with or * without modification, are permitted provided that the following @@ -35,6 +35,7 @@ #include "mp_decimal.h" #include "uuid/mp_uuid.h" #include "mp_error.h" +#include "datetime.h" static int msgpack_fprint_ext(FILE *file, const char **data, int depth) @@ -47,6 +48,8 @@ msgpack_fprint_ext(FILE *file, const char **data, int depth) return mp_fprint_decimal(file, data, len); case MP_UUID: return mp_fprint_uuid(file, data, len); + case MP_DATETIME: + return mp_fprint_datetime(file, data, len); case MP_ERROR: return mp_fprint_error(file, data, depth); default: @@ -65,6 +68,8 @@ msgpack_snprint_ext(char *buf, int size, const char **data, int depth) return mp_snprint_decimal(buf, size, data, len); case MP_UUID: return mp_snprint_uuid(buf, size, data, len); + case MP_DATETIME: + return mp_snprint_datetime(buf, size, data, len); case MP_ERROR: return mp_snprint_error(buf, size, data, depth); default: diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc index 98938fb39..48c833643 100644 --- a/src/box/tuple_compare.cc +++ b/src/box/tuple_compare.cc @@ -36,6 +36,7 @@ #include "lib/core/decimal.h" #include "lib/core/mp_decimal.h" #include "uuid/mp_uuid.h" +#include "core/datetime.h" #include "lib/core/mp_extension_types.h" /* {{{ tuple_compare */ @@ -76,6 +77,7 @@ enum mp_class { MP_CLASS_STR, MP_CLASS_BIN, MP_CLASS_UUID, + MP_CLASS_DATETIME, MP_CLASS_ARRAY, MP_CLASS_MAP, mp_class_max, @@ -99,6 +101,8 @@ static enum mp_class mp_ext_classes[] = { /* .MP_UNKNOWN_EXTENSION = */ mp_class_max, /* unsupported */ /* .MP_DECIMAL = */ MP_CLASS_NUMBER, /* .MP_UUID = */ MP_CLASS_UUID, + /* .MP_ERROR = */ mp_class_max, + /* .MP_DATETIME = */ MP_CLASS_DATETIME, }; static enum mp_class @@ -390,6 +394,19 @@ mp_compare_uuid(const char *field_a, const char *field_b) return memcmp(field_a + 2, field_b + 2, UUID_PACKED_LEN); } +static int +mp_compare_datetime(const char *lhs, const char *rhs) +{ + datetime_t lhs_dt, rhs_dt; + datetime_t *ret; + ret = mp_decode_datetime(&lhs, &lhs_dt); + assert(ret != NULL); + ret = mp_decode_datetime(&rhs, &rhs_dt); + assert(ret != NULL); + (void)ret; + return datetime_compare(&lhs_dt, &rhs_dt); +} + typedef int (*mp_compare_f)(const char *, const char *); static mp_compare_f mp_class_comparators[] = { /* .MP_CLASS_NIL = */ NULL, @@ -398,6 +415,7 @@ static mp_compare_f mp_class_comparators[] = { /* .MP_CLASS_STR = */ mp_compare_str, /* .MP_CLASS_BIN = */ mp_compare_bin, /* .MP_CLASS_UUID = */ mp_compare_uuid, + /* .MP_CLASS_DATETIME=*/ mp_compare_datetime, /* .MP_CLASS_ARRAY = */ NULL, /* .MP_CLASS_MAP = */ NULL, }; @@ -478,6 +496,8 @@ tuple_compare_field(const char *field_a, const char *field_b, return mp_compare_decimal(field_a, field_b); case FIELD_TYPE_UUID: return mp_compare_uuid(field_a, field_b); + case FIELD_TYPE_DATETIME: + return mp_compare_datetime(field_a, field_b); default: unreachable(); return 0; diff --git a/src/exports.h b/src/exports.h index 1a03db636..586444b65 100644 --- a/src/exports.h +++ b/src/exports.h @@ -531,6 +531,8 @@ EXPORT(uri_format) EXPORT(uri_parse) EXPORT(uuid_nil) EXPORT(uuid_unpack) +EXPORT(datetime_unpack) +EXPORT(datetime_pack) EXPORT(dt_from_rdn) EXPORT(dt_from_yd) EXPORT(dt_from_ymd) diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt index 8bc776b82..c3e68c36f 100644 --- a/src/lib/core/CMakeLists.txt +++ b/src/lib/core/CMakeLists.txt @@ -44,7 +44,8 @@ add_library(core STATIC ${core_sources}) target_link_libraries(core salad small uri decNumber bit ${LIBEV_LIBRARIES} ${LIBEIO_LIBRARIES} ${LIBCORO_LIBRARIES} - ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}) + ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES} + ${LIBCDT_LIBRARIES}) if (ENABLE_BACKTRACE AND NOT TARGET_OS_DARWIN) target_link_libraries(core gcc_s ${UNWIND_LIBRARIES}) diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c index 65f813a70..26ba8a702 100755 --- a/src/lib/core/datetime.c +++ b/src/lib/core/datetime.c @@ -33,6 +33,132 @@ #include "trivia/util.h" #include "datetime.h" +#include "msgpuck.h" +#include "mp_extension_types.h" + +/* + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length) + extension, which creates container of 1 to 3 integers. + + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+ + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset (uint) | + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+ + + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains: + + - signed integer seconds part (required). Depending on the value of + seconds it may be from 1 to 8 bytes positive or negative integer number; + + - [optional] fraction time in nanoseconds as unsigned integer. + If this value is 0 then it's not saved (unless there is offset field, + as below); + + - [optional] timzeone offset in minutes as unsigned integer. + If this field is 0 then it's not saved. + */ + +static inline uint32_t +mp_sizeof_Xint(int64_t n) +{ + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n); +} + +static inline char * +mp_encode_Xint(char *data, int64_t v) +{ + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v); +} + +static inline int64_t +mp_decode_Xint(const char **data) +{ + switch (mp_typeof(**data)) { + case MP_UINT: + return (int64_t)mp_decode_uint(data); + case MP_INT: + return mp_decode_int(data); + default: + mp_unreachable(); + } + return 0; +} + +uint32_t +mp_sizeof_datetime(const struct datetime_t *date) +{ + uint32_t sz = mp_sizeof_Xint(date->secs); + + // even if nanosecs == 0 we need to output anything + // if we have non-null tz offset + if (date->nsec != 0 || date->offset != 0) + sz += mp_sizeof_Xint(date->nsec); + if (date->offset) + sz += mp_sizeof_Xint(date->offset); + + return sz; +} + +struct datetime_t * +datetime_unpack(const char **data, uint32_t len, struct datetime_t *date) +{ + const char * svp = *data; + + memset(date, 0, sizeof(*date)); + + date->secs = mp_decode_Xint(data); + + len -= *data - svp; + if (len <= 0) + return date; + + svp = *data; + date->secs = mp_decode_Xint(data); + len -= *data - svp; + + if (len <= 0) + return date; + + date->offset = mp_decode_Xint(data); + + return date; +} + +struct datetime_t * +mp_decode_datetime(const char **data, struct datetime_t *date) +{ + if (mp_typeof(**data) != MP_EXT) + return NULL; + + int8_t type; + uint32_t len = mp_decode_extl(data, &type); + + if (type != MP_DATETIME || len == 0) { + return NULL; + } + return datetime_unpack(data, len, date); +} + +char * +datetime_pack(char *data, const struct datetime_t *date) +{ + data = mp_encode_Xint(data, date->secs); + if (date->nsec != 0 || date->offset != 0) + data = mp_encode_Xint(data, date->nsec); + if (date->offset) + data = mp_encode_Xint(data, date->offset); + + return data; +} + +char * +mp_encode_datetime(char *data, const struct datetime_t *date) +{ + uint32_t len = mp_sizeof_datetime(date); + + data = mp_encode_extl(data, MP_DATETIME, len); + + return datetime_pack(data, date); +} int datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len) @@ -83,3 +209,38 @@ datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len) } return (buf - src); } + +int +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len) +{ + struct datetime_t date = {0}; + + if (datetime_unpack(data, len, &date) == NULL) + return -1; + + return datetime_to_string(&date, buf, size); +} + +int +mp_fprint_datetime(FILE *file, const char **data, uint32_t len) +{ + struct datetime_t date; + + if (datetime_unpack(data, len, &date) == NULL) + return -1; + + char buf[48]; + datetime_to_string(&date, buf, sizeof buf); + + return fprintf(file, "%s", buf); +} + +int datetime_compare(const struct datetime_t *lhs, const struct datetime_t *rhs) +{ + int result = COMPARE_RESULT(lhs->secs, rhs->secs); + if (result != 0) + return result; + + return COMPARE_RESULT(lhs->nsec, rhs->nsec); +} + diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h index fdaff2d27..6e78d8ddb 100644 --- a/src/lib/core/datetime.h +++ b/src/lib/core/datetime.h @@ -30,9 +30,10 @@ * SUCH DAMAGE. */ -#include <c-dt/dt_core.h> +#include <c-dt/dt.h> #include <stdint.h> #include <stdbool.h> +#include <stdio.h> #if defined(__cplusplus) extern "C" { @@ -60,6 +61,38 @@ struct datetime_interval_t { int32_t nsec; ///< nanoseconds delta }; +int +datetime_compare(const struct datetime_t * lhs, + const struct datetime_t * rhs); + + +struct datetime_t * +datetime_unpack(const char **data, uint32_t len, struct datetime_t *date); + +/** + * Pack datetime_t data to the MessagePack buffer. + */ +char * +datetime_pack(char *data, const struct datetime_t *date); + +/** + * Calculate size of MessagePack buffer for datetime_t data. + */ +uint32_t +mp_sizeof_datetime(const struct datetime_t *date); + +/** + * Decode data from MessagePack buffer to datetime_t structure. + */ +struct datetime_t * +mp_decode_datetime(const char **data, struct datetime_t *date); + +/** + * Encode datetime_t structure to the MessagePack buffer. + */ +char * +mp_encode_datetime(char *data, const struct datetime_t *date); + /** * Convert datetime to string using default format * @param date source datetime value @@ -69,6 +102,12 @@ struct datetime_interval_t { int datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len); +int +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len); + +int +mp_fprint_datetime(FILE *file, const char **data, uint32_t len); + #if defined(__cplusplus) } /* extern "C" */ #endif /* defined(__cplusplus) */ diff --git a/src/lib/core/mp_extension_types.h b/src/lib/core/mp_extension_types.h index e3ff9f5d0..3b7eaee7c 100644 --- a/src/lib/core/mp_extension_types.h +++ b/src/lib/core/mp_extension_types.h @@ -44,6 +44,7 @@ enum mp_extension_type { MP_DECIMAL = 1, MP_UUID = 2, MP_ERROR = 3, + MP_DATETIME = 4, mp_extension_type_MAX, }; diff --git a/src/lib/mpstream/mpstream.c b/src/lib/mpstream/mpstream.c index 70ca29889..1077e3b19 100644 --- a/src/lib/mpstream/mpstream.c +++ b/src/lib/mpstream/mpstream.c @@ -35,6 +35,7 @@ #include "msgpuck.h" #include "mp_decimal.h" #include "uuid/mp_uuid.h" +#include "core/datetime.h" void mpstream_reserve_slow(struct mpstream *stream, size_t size) @@ -208,6 +209,16 @@ mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid) mpstream_advance(stream, pos - data); } +void +mpstream_encode_datetime(struct mpstream *stream, const struct datetime_t *val) +{ + char *data = mpstream_reserve(stream, mp_sizeof_datetime(val)); + if (data == NULL) + return; + char *pos = mp_encode_datetime(data, val); + mpstream_advance(stream, pos - data); +} + void mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n) { diff --git a/src/lib/mpstream/mpstream.h b/src/lib/mpstream/mpstream.h index a60add143..540e9a666 100644 --- a/src/lib/mpstream/mpstream.h +++ b/src/lib/mpstream/mpstream.h @@ -39,6 +39,7 @@ extern "C" { #endif /* defined(__cplusplus) */ struct tt_uuid; +struct datetime_t; /** * Ask the allocator to reserve at least size bytes. It can reserve @@ -145,6 +146,9 @@ mpstream_encode_decimal(struct mpstream *stream, const decimal_t *val); void mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid); +void +mpstream_encode_datetime(struct mpstream *stream, const struct datetime_t *dt); + /** Copies n bytes from memory area src to stream. */ void mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n); diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c index b6ecf2b1e..0a4ba8129 100644 --- a/src/lua/msgpack.c +++ b/src/lua/msgpack.c @@ -46,6 +46,7 @@ #include "lib/core/decimal.h" /* decimal_unpack() */ #include "lib/uuid/mp_uuid.h" /* mp_decode_uuid() */ #include "lib/core/mp_extension_types.h" +#include "datetime.h" #include "cord_buf.h" #include <fiber.h> @@ -200,6 +201,9 @@ restart: /* used by MP_EXT of unidentified subtype */ break; case MP_ERROR: return luamp_encode_extension(L, top, stream); + case MP_DATETIME: + mpstream_encode_datetime(stream, field->dateval); + break; default: /* Run trigger if type can't be encoded */ type = luamp_encode_extension(L, top, stream); @@ -333,6 +337,14 @@ luamp_decode(struct lua_State *L, struct luaL_serializer *cfg, goto ext_decode_err; return; } + case MP_DATETIME: + { + struct datetime_t * date = luaL_pushdatetime(L); + date = datetime_unpack(data, len, date); + if (date == NULL) + goto ext_decode_err; + return; + } default: /* reset data to the extension header */ *data = svp; diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua index 1d54f11b8..271be857a 100644 --- a/src/lua/msgpackffi.lua +++ b/src/lua/msgpackffi.lua @@ -36,6 +36,8 @@ decimal_t * decimal_unpack(const char **data, uint32_t len, decimal_t *dec); struct tt_uuid * uuid_unpack(const char **data, uint32_t len, struct tt_uuid *uuid); +struct datetime_t * +datetime_unpack(const char **data, uint32_t len, struct datetime_t *date); ]]) local strict_alignment = (jit.arch == 'arm') @@ -513,6 +515,12 @@ local ext_decoder = { builtin.uuid_unpack(data, len, uuid) return uuid end, + -- MP_DATETIME + [4] = function(data, len) + local dt = ffi.new("struct datetime_t") + builtin.datetime_unpack(data, len, dt) + return dt + end, } local function decode_ext(data) diff --git a/src/lua/serializer.c b/src/lua/serializer.c index 8db6746a3..c27e62c62 100644 --- a/src/lua/serializer.c +++ b/src/lua/serializer.c @@ -41,6 +41,7 @@ #include "lib/core/mp_extension_types.h" #include "lua/error.h" +#include "datetime.h" #include "trivia/util.h" #include "diag.h" #include "serializer_opts.h" @@ -544,6 +545,9 @@ luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg, opts != NULL && opts->error_marshaling_enabled) { field->ext_type = MP_ERROR; + } else if (cd->ctypeid == CTID_DATETIME) { + field->ext_type = MP_DATETIME; + field->dateval = (struct datetime_t*) cdata; } else { field->ext_type = MP_UNKNOWN_EXTENSION; } diff --git a/src/lua/serializer.h b/src/lua/serializer.h index 0a0501a74..52e51d279 100644 --- a/src/lua/serializer.h +++ b/src/lua/serializer.h @@ -52,6 +52,7 @@ extern "C" { #include <lauxlib.h> #include "trigger.h" +#include "lib/core/datetime.h" #include "lib/core/decimal.h" /* decimal_t */ #include "lib/core/mp_extension_types.h" #include "lua/error.h" @@ -223,6 +224,7 @@ struct luaL_field { uint32_t size; decimal_t *decval; struct tt_uuid *uuidval; + struct datetime_t *dateval; }; enum mp_type type; /* subtypes of MP_EXT */ diff --git a/src/lua/utils.c b/src/lua/utils.c index 6aeb18cfe..a959ed9dd 100644 --- a/src/lua/utils.c +++ b/src/lua/utils.c @@ -241,7 +241,6 @@ luaL_setcdatagc(struct lua_State *L, int idx) lua_pop(L, 1); } - /** * A helper to register a single type metatable. */ diff --git a/test/unit/datetime.c b/test/unit/datetime.c index b6f568c03..226b6fadb 100644 --- a/test/unit/datetime.c +++ b/test/unit/datetime.c @@ -139,7 +139,6 @@ exit: // avoid introducing external datetime.h dependency // - just copy paste it for today #define SECS_PER_DAY 86400 -#define NANOS_PER_SEC 1000000000 #define DT_EPOCH_1970_OFFSET 719163 diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c index 5123b9a74..d5ddc4ea0 100644 --- a/third_party/lua-cjson/lua_cjson.c +++ b/third_party/lua-cjson/lua_cjson.c @@ -52,6 +52,7 @@ #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */ #include "tt_static.h" #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */ +#include "core/datetime.h" #include "cord_buf.h" typedef enum { @@ -426,6 +427,13 @@ static void json_append_data(lua_State *l, struct luaL_serializer *cfg, case MP_UUID: return json_append_string(cfg, json, tt_uuid_str(field.uuidval), UUID_STR_LEN); + + case MP_DATETIME: + { + char buf[128]; + size_t sz = datetime_to_string(field.dateval, buf, sizeof buf); + return json_append_string(cfg, json, buf, sz); + } default: assert(false); } diff --git a/third_party/lua-yaml/lyaml.cc b/third_party/lua-yaml/lyaml.cc index 5469e9f4f..b76a45dfb 100644 --- a/third_party/lua-yaml/lyaml.cc +++ b/third_party/lua-yaml/lyaml.cc @@ -617,7 +617,7 @@ static int dump_node(struct lua_yaml_dumper *dumper) yaml_event_t ev; yaml_scalar_style_t style = YAML_PLAIN_SCALAR_STYLE; int is_binary = 0; - char buf[FPCONV_G_FMT_BUFSIZE]; + char buf[FPCONV_G_FMT_BUFSIZE + 8]; // FIXME - need extra space for datetime struct luaL_field field; bool unused; (void) unused; @@ -707,6 +707,10 @@ static int dump_node(struct lua_yaml_dumper *dumper) str = tt_uuid_str(field.uuidval); len = UUID_STR_LEN; break; + case MP_DATETIME: + len = datetime_to_string(field.dateval, buf, sizeof buf); + str = buf; + break; default: assert(0); /* checked by luaL_checkfield() */ } -- 2.29.2
* storage hints implemented for datetime_t values; * proper comparison for indices of datetime type. Part of #5941 Part of #5946 --- src/box/field_def.c | 18 ++++++++ src/box/field_def.h | 3 ++ src/box/memtx_space.c | 3 +- src/box/tuple_compare.cc | 30 ++++++++++++++ src/box/vinyl.c | 3 +- test/engine/datetime.result | 77 +++++++++++++++++++++++++++++++++++ test/engine/datetime.test.lua | 35 ++++++++++++++++ 7 files changed, 165 insertions(+), 4 deletions(-) create mode 100644 test/engine/datetime.result create mode 100644 test/engine/datetime.test.lua diff --git a/src/box/field_def.c b/src/box/field_def.c index 6964e3e9f..aaf5f9cff 100644 --- a/src/box/field_def.c +++ b/src/box/field_def.c @@ -193,3 +193,21 @@ field_type_by_name(const char *name, size_t len) return FIELD_TYPE_ANY; return field_type_MAX; } + +const bool field_type_index_allowed[] = + { + /* [FIELD_TYPE_ANY] = */ false, + /* [FIELD_TYPE_UNSIGNED] = */ true, + /* [FIELD_TYPE_STRING] = */ true, + /* [FIELD_TYPE_NUMBER] = */ true, + /* [FIELD_TYPE_DOUBLE] = */ true, + /* [FIELD_TYPE_INTEGER] = */ true, + /* [FIELD_TYPE_BOOLEAN] = */ true, + /* [FIELD_TYPE_VARBINARY]= */ true, + /* [FIELD_TYPE_SCALAR] = */ true, + /* [FIELD_TYPE_DECIMAL] = */ true, + /* [FIELD_TYPE_UUID] = */ true, + /* [FIELD_TYPE_ARRAY] = */ false, + /* [FIELD_TYPE_MAP] = */ false, + /* [FIELD_TYPE_DATETIME] = */ true, +}; diff --git a/src/box/field_def.h b/src/box/field_def.h index 120b2a93d..bd02418df 100644 --- a/src/box/field_def.h +++ b/src/box/field_def.h @@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[]; extern const struct opt_def field_def_reg[]; extern const struct field_def field_def_default; +/** helper table for checking allowed indices for types */ +extern const bool field_type_index_allowed[]; + /** * @brief Field definition * Contains information about of one tuple field. diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c index b71318d24..1ab16122e 100644 --- a/src/box/memtx_space.c +++ b/src/box/memtx_space.c @@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def) /* Check that there are no ANY, ARRAY, MAP parts */ for (uint32_t i = 0; i < key_def->part_count; i++) { struct key_part *part = &key_def->parts[i]; - if (part->type <= FIELD_TYPE_ANY || - part->type >= FIELD_TYPE_ARRAY) { + if (!field_type_index_allowed[part->type]) { diag_set(ClientError, ER_MODIFY_INDEX, index_def->name, space_name(space), tt_sprintf("field type '%s' is not supported", diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc index 48c833643..4adceebd3 100644 --- a/src/box/tuple_compare.cc +++ b/src/box/tuple_compare.cc @@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a, enum mp_type a_type, field_b, b_type); case FIELD_TYPE_UUID: return mp_compare_uuid(field_a, field_b); + case FIELD_TYPE_DATETIME: + return mp_compare_datetime(field_a, field_b); default: unreachable(); return 0; @@ -1630,6 +1632,18 @@ hint_uuid_raw(const char *data) return hint_create(MP_CLASS_UUID, val); } +static inline hint_t +hint_datetime(struct datetime_t *date) +{ + /* + * Use at most HINT_VALUE_BITS from datetime + * seconds field as a hint value + */ + uint64_t val = (uint64_t)date->secs & HINT_VALUE_MAX; + + return hint_create(MP_CLASS_DATETIME, val); +} + static inline uint64_t hint_str_raw(const char *s, uint32_t len) { @@ -1761,6 +1775,17 @@ field_hint_uuid(const char *field) return hint_uuid_raw(data); } +static inline hint_t +field_hint_datetime(const char *field) +{ + assert(mp_typeof(*field) == MP_EXT); + int8_t ext_type; + uint32_t len = mp_decode_extl(&field, &ext_type); + assert(ext_type == MP_DATETIME); + struct datetime_t date; + return hint_datetime(datetime_unpack(&field, len, &date)); +} + static inline hint_t field_hint_string(const char *field, struct coll *coll) { @@ -1849,6 +1874,8 @@ field_hint(const char *field, struct coll *coll) return field_hint_decimal(field); case FIELD_TYPE_UUID: return field_hint_uuid(field); + case FIELD_TYPE_DATETIME: + return field_hint_datetime(field); default: unreachable(); } @@ -1963,6 +1990,9 @@ key_def_set_hint_func(struct key_def *def) case FIELD_TYPE_UUID: key_def_set_hint_func<FIELD_TYPE_UUID>(def); break; + case FIELD_TYPE_DATETIME: + key_def_set_hint_func<FIELD_TYPE_DATETIME>(def); + break; default: /* Invalid key definition. */ def->key_hint = NULL; diff --git a/src/box/vinyl.c b/src/box/vinyl.c index c80b2d99b..360d1fa70 100644 --- a/src/box/vinyl.c +++ b/src/box/vinyl.c @@ -662,8 +662,7 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def) /* Check that there are no ANY, ARRAY, MAP parts */ for (uint32_t i = 0; i < key_def->part_count; i++) { struct key_part *part = &key_def->parts[i]; - if (part->type <= FIELD_TYPE_ANY || - part->type >= FIELD_TYPE_ARRAY) { + if (!field_type_index_allowed[part->type]) { diag_set(ClientError, ER_MODIFY_INDEX, index_def->name, space_name(space), tt_sprintf("field type '%s' is not supported", diff --git a/test/engine/datetime.result b/test/engine/datetime.result new file mode 100644 index 000000000..3ff474dee --- /dev/null +++ b/test/engine/datetime.result @@ -0,0 +1,77 @@ +-- test-run result file version 2 +env = require('test_run') + | --- + | ... +test_run = env.new() + | --- + | ... +engine = test_run:get_cfg('engine') + | --- + | ... + +date = require('datetime') + | --- + | ... + +_ = box.schema.space.create('T', {engine = engine}) + | --- + | ... +_ = box.space.T:create_index('pk', {parts={1,'datetime'}}) + | --- + | ... + +box.space.T:insert{date('1970-01-01')}\ +box.space.T:insert{date('1970-01-02')}\ +box.space.T:insert{date('1970-01-03')}\ +box.space.T:insert{date('2000-01-01')} + | --- + | ... + +o = box.space.T:select{} + | --- + | ... +assert(tostring(o[1][1]) == '1970-01-01T00:00Z') + | --- + | - true + | ... +assert(tostring(o[2][1]) == '1970-01-02T00:00Z') + | --- + | - true + | ... +assert(tostring(o[3][1]) == '1970-01-03T00:00Z') + | --- + | - true + | ... +assert(tostring(o[4][1]) == '2000-01-01T00:00Z') + | --- + | - true + | ... + +for i = 1,16 do\ + box.space.T:insert{date.now()}\ +end + | --- + | ... + +a = box.space.T:select{} + | --- + | ... +err = nil + | --- + | ... +for i = 1, #a - 1 do\ + if tostring(a[i][1]) >= tostring(a[i+1][1]) then\ + err = {a[i][1], a[i+1][1]}\ + break\ + end\ +end + | --- + | ... + +err + | --- + | - null + | ... +box.space.T:drop() + | --- + | ... diff --git a/test/engine/datetime.test.lua b/test/engine/datetime.test.lua new file mode 100644 index 000000000..180d9bb72 --- /dev/null +++ b/test/engine/datetime.test.lua @@ -0,0 +1,35 @@ +env = require('test_run') +test_run = env.new() +engine = test_run:get_cfg('engine') + +date = require('datetime') + +_ = box.schema.space.create('T', {engine = engine}) +_ = box.space.T:create_index('pk', {parts={1,'datetime'}}) + +box.space.T:insert{date('1970-01-01')}\ +box.space.T:insert{date('1970-01-02')}\ +box.space.T:insert{date('1970-01-03')}\ +box.space.T:insert{date('2000-01-01')} + +o = box.space.T:select{} +assert(tostring(o[1][1]) == '1970-01-01T00:00Z') +assert(tostring(o[2][1]) == '1970-01-02T00:00Z') +assert(tostring(o[3][1]) == '1970-01-03T00:00Z') +assert(tostring(o[4][1]) == '2000-01-01T00:00Z') + +for i = 1,16 do\ + box.space.T:insert{date.now()}\ +end + +a = box.space.T:select{} +err = nil +for i = 1, #a - 1 do\ + if tostring(a[i][1]) >= tostring(a[i+1][1]) then\ + err = {a[i][1], a[i+1][1]}\ + break\ + end\ +end + +err +box.space.T:drop() -- 2.29.2
* correct incorrect encoding of MP_EXT sizes for datetime messagepack values; * export necessary symbols for datetime messagepack size calculations so they will be available for Lua consumption. Part of #5941 --- src/exports.h | 2 ++ src/lib/core/datetime.c | 15 ++++++++++----- src/lua/msgpackffi.lua | 10 ++++++++++ 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/src/exports.h b/src/exports.h index 586444b65..3a1e8854c 100644 --- a/src/exports.h +++ b/src/exports.h @@ -437,6 +437,7 @@ EXPORT(mp_encode_decimal) EXPORT(mp_encode_double) EXPORT(mp_encode_float) EXPORT(mp_encode_uuid) +EXPORT(mp_encode_datetime) EXPORT(mp_ext_hint) EXPORT(mp_format) EXPORT(mp_fprint) @@ -446,6 +447,7 @@ EXPORT(mp_fprint_recursion) EXPORT(mp_parser_hint) EXPORT(mp_sizeof_decimal) EXPORT(mp_sizeof_uuid) +EXPORT(mp_sizeof_datetime) EXPORT(mp_snprint) EXPORT(mp_snprint_ext) EXPORT(mp_snprint_ext_default) diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c index 26ba8a702..76b06eae5 100755 --- a/src/lib/core/datetime.c +++ b/src/lib/core/datetime.c @@ -83,8 +83,8 @@ mp_decode_Xint(const char **data) return 0; } -uint32_t -mp_sizeof_datetime(const struct datetime_t *date) +static inline uint32_t +mp_sizeof_datetime_raw(const struct datetime_t *date) { uint32_t sz = mp_sizeof_Xint(date->secs); @@ -94,10 +94,15 @@ mp_sizeof_datetime(const struct datetime_t *date) sz += mp_sizeof_Xint(date->nsec); if (date->offset) sz += mp_sizeof_Xint(date->offset); - return sz; } +uint32_t +mp_sizeof_datetime(const struct datetime_t *date) +{ + return mp_sizeof_ext(mp_sizeof_datetime_raw(date)); +} + struct datetime_t * datetime_unpack(const char **data, uint32_t len, struct datetime_t *date) { @@ -112,7 +117,7 @@ datetime_unpack(const char **data, uint32_t len, struct datetime_t *date) return date; svp = *data; - date->secs = mp_decode_Xint(data); + date->nsec = mp_decode_Xint(data); len -= *data - svp; if (len <= 0) @@ -153,7 +158,7 @@ datetime_pack(char *data, const struct datetime_t *date) char * mp_encode_datetime(char *data, const struct datetime_t *date) { - uint32_t len = mp_sizeof_datetime(date); + uint32_t len = mp_sizeof_datetime_raw(date); data = mp_encode_extl(data, MP_DATETIME, len); diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua index 271be857a..c47d77acb 100644 --- a/src/lua/msgpackffi.lua +++ b/src/lua/msgpackffi.lua @@ -26,6 +26,10 @@ char * mp_encode_uuid(char *data, const struct tt_uuid *uuid); uint32_t mp_sizeof_uuid(); +uint32_t +mp_sizeof_datetime(const struct t_datetime_tz *date); +char * +mp_encode_datetime(char *data, const struct t_datetime_tz *date); float mp_decode_float(const char **data); double @@ -144,6 +148,11 @@ local function encode_uuid(buf, uuid) builtin.mp_encode_uuid(p, uuid) end +local function encode_datetime(buf, date) + local p = buf:alloc(builtin.mp_sizeof_datetime(date)) + builtin.mp_encode_datetime(p, date) +end + local function encode_int(buf, num) if num >= 0 then if num <= 0x7f then @@ -322,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float) on_encode(ffi.typeof('double'), encode_double) on_encode(ffi.typeof('decimal_t'), encode_decimal) on_encode(ffi.typeof('struct tt_uuid'), encode_uuid) +on_encode(ffi.typeof('struct datetime_t'), encode_datetime) -------------------------------------------------------------------------------- -- Decoder -- 2.29.2
* introduced a set of calculated attributes to data object, e.g.: - timestamp, seconds, microseconds, minute, or hours Part of #5941 --- src/lua/datetime.lua | 51 ++++++++++++++++++++++------------ test/app-tap/datetime.test.lua | 17 +++++++++++- 2 files changed, 50 insertions(+), 18 deletions(-) diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua index 9ec06d8d8..7a208cef9 100644 --- a/src/lua/datetime.lua +++ b/src/lua/datetime.lua @@ -224,6 +224,36 @@ local function interval_serialize(self) return { secs = self.secs, nsec = self.nsec } end +local datetime_index = function(self, key) + local attributes = { + timestamp = function(self) + return tonumber(self.secs) + self.nsec / 1e9 + end, + nanoseconds = function(self) + return tonumber(self.secs * 1e9 + self.nsec) + end, + microseconds = function(self) + return tonumber(self.secs * 1e6 + self.nsec / 1e3) + end, + milliseconds = function(self) + return tonumber(self.secs * 1e3 + self.nsec / 1e6) + end, + seconds = function(self) + return tonumber(self.secs) + self.nsec / 1e9 + end, + minutes = function(self) + return (tonumber(self.secs) + self.nsec / 1e9) / 60 + end, + hours = function(self) + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) + end, + days = function(self) + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24 + end, + } + return attributes[key] ~= nil and attributes[key](self) or nil +end + local datetime_mt = { -- __tostring = datetime_tostring, __serialize = datetime_serialize, @@ -232,23 +262,7 @@ local datetime_mt = { __le = datetime_le, __sub = datetime_sub, __add = datetime_add, - - nanoseconds = function(self) - return tonumber(self.secs*NANOS_PER_SEC + self.nsec) - end, - microseconds = function(self) - return tonumber(self.secs*1e6 + self.nsec*1e3) - end, - seconds = function(self) - return tonumber(self.secs + self.nsec*1e3) - end, - minutes = function(self) - return tonumber((self._ticks/(1e6*60))%60) - end, - hours = function(self) - return tonumber(self._ticks/(1e6*60*60)) - end, - + __index = datetime_index, } local interval_mt = { @@ -257,6 +271,9 @@ local interval_mt = { __eq = datetime_eq, __lt = datetime_lt, __le = datetime_le, + __sub = datetime_sub, + __add = datetime_add, + __index = datetime_index, } local function datetime_new_raw(secs, nsec, offset) diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua index 09c968858..407d89556 100755 --- a/test/app-tap/datetime.test.lua +++ b/test/app-tap/datetime.test.lua @@ -4,7 +4,7 @@ local tap = require('tap') local test = tap.test("errno") local date = require('datetime') -test:plan(5) +test:plan(6) test:test("Simple tests for parser", function(test) test:plan(2) @@ -188,4 +188,19 @@ test:test("Parse iso date - invalid strings", function(test) end end) +test:test("Parse tiny date into seconds and other parts", function(test) + test:plan(9) + local str = '19700101 00:00:30.528' + local tiny = date(str) + test:ok(tiny.secs == 30, ("secs of '%s'"):format(str)) + test:ok(tiny.nsec == 528000000, ("nsec of '%s'"):format(str)) + test:ok(tiny.nanoseconds == 30528000000, "nanoseconds") + test:ok(tiny.microseconds == 30528000, "microseconds") + test:ok(tiny.milliseconds == 30528, "milliseconds") + test:ok(tiny.seconds == 30.528, "seconds") + test:ok(tiny.timestamp == 30.528, "timestamp") + test:ok(tiny.minutes == 0.5088, "minuts") + test:ok(tiny.hours == 0.00848, "hours") +end) + os.exit(test:check() and 0 or 1) -- 2.29.2
* created few entry points (months(N), years(N), days(N), etc.) for easier datetime arithmetic; * additions/subtractions of years/months use `dt_add_years()` and `dt_add_months()` from 3rd party c-dt library; * also there are `:add{}` and `:sub{}` methods in datetime object to add or substract more complex intervals; * introduced `is_datetime()` and `is_interval()` helpers for checking of validity of passed arguments; * human-readable stringization implemented for interval objects. Note, that additions/subtractions completed for all _reasonable_ combinations of values of date and interval types; Time + Interval => Time Interval + Time => Time Time - Time => Interval Time - Interval => Time Interval + Interval => Interval Interval - Interval => Interval Part of #5941 --- src/exports.h | 3 + src/lua/datetime.lua | 556 +++++++++++++++++++++++++++------ test/app-tap/datetime.test.lua | 163 +++++++++- 3 files changed, 631 insertions(+), 91 deletions(-) diff --git a/src/exports.h b/src/exports.h index 3a1e8854c..6e7fe206d 100644 --- a/src/exports.h +++ b/src/exports.h @@ -535,6 +535,9 @@ EXPORT(uuid_nil) EXPORT(uuid_unpack) EXPORT(datetime_unpack) EXPORT(datetime_pack) +EXPORT(dt_add_months) +EXPORT(dt_add_years) +EXPORT(dt_add_quarters) EXPORT(dt_from_rdn) EXPORT(dt_from_yd) EXPORT(dt_from_ymd) diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua index 7a208cef9..1466b923f 100644 --- a/src/lua/datetime.lua +++ b/src/lua/datetime.lua @@ -37,6 +37,17 @@ ffi.cdef [[ int dt_rdn (dt_t dt); dt_dow_t dt_dow (dt_t dt); + // dt_arithmetic.h + typedef enum { + DT_EXCESS, + DT_LIMIT, + DT_SNAP + } dt_adjust_t; + + dt_t dt_add_years (dt_t dt, int delta, dt_adjust_t adjust); + dt_t dt_add_quarters (dt_t dt, int delta, dt_adjust_t adjust); + dt_t dt_add_months (dt_t dt, int delta, dt_adjust_t adjust); + // dt_parse_iso.h size_t dt_parse_iso_date (const char *str, size_t len, dt_t *dt); @@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL local datetime_t = ffi.typeof('struct datetime_t') local interval_t = ffi.typeof('struct datetime_interval_t') +ffi.cdef [[ + struct t_interval_months { + int m; + }; + + struct t_interval_years { + int y; + }; +]] +local interval_months_t = ffi.typeof('struct t_interval_months') +local interval_years_t = ffi.typeof('struct t_interval_years') + +local function is_interval(o) + return ffi.istype(interval_t, o) or + ffi.istype(interval_months_t, o) or + ffi.istype(interval_years_t, o) +end + +local function is_datetime(o) + return ffi.istype(o, datetime_t) +end + local function interval_new() local interval = ffi.new(interval_t) return interval end -local function adjusted_secs(dt) - return dt.secs - dt.offset * 60 +local function check_number(n, message, lvl) + if lvl == nil then + lvl = 2 + end + if type(n) ~= 'number' then + return error(('Usage: %s'):format(message), lvl) + end end -local function datetime_sub(lhs, rhs) - local s1 = adjusted_secs(lhs) - local s2 = adjusted_secs(rhs) - local d = interval_new() - d.secs = s2 - s1 - d.nsec = rhs.nsec - lhs.nsec - if d.nsec < 0 then - d.secs = d.secs - 1 - d.nsec = d.nsec + NANOS_PER_SEC +local function check_date(o, message, lvl) + if lvl == nil then + lvl = 2 + end + if not is_datetime(o) then + return error(('Usage: %s'):format(message), lvl) end - return d end -local function datetime_add(lhs, rhs) - local s1 = adjusted_secs(lhs) - local s2 = adjusted_secs(rhs) - local d = interval_new() - d.secs = s2 + s1 - d.nsec = rhs.nsec + lhs.nsec - if d.nsec >= NANOS_PER_SEC then - d.secs = d.secs + 1 - d.nsec = d.nsec - NANOS_PER_SEC +local function check_date_interval(o, message, lvl) + if lvl == nil then + lvl = 2 + end + if not (is_datetime(o) or is_interval(o)) then + return error(('Usage: %s'):format(message), lvl) end - return d end -local function datetime_eq(lhs, rhs) - -- we usually don't need to check nullness - -- but older tarantool console will call us checking for equality to nil - if rhs == nil then - return false +local function check_interval(o, message, lvl) + if lvl == nil then + lvl = 2 + end + if not is_interval(o) then + return error(('Usage: %s'):format(message), lvl) end +end + +local function check_str(o, message, lvl) + if lvl == nil then + lvl = 2 + end + if not type(o) == 'string' then + return error(('Usage: %s'):format(message), lvl) + end +end + +local function interval_years_new(y) + check_number(y, "years(number)") + local o = ffi.new(interval_years_t) + o.y = y + return o +end + +local function interval_months_new(m) + check_number(m, "months(number)") + local o = ffi.new(interval_months_t) + o.m = m + return o +end + +local function interval_weeks_new(w) + check_number(w, "weeks(number)") + local o = ffi.new(interval_t) + o.secs = w * SECS_PER_DAY * 7 + return o +end + +local function interval_days_new(d) + check_number(d, "days(number)") + local o = ffi.new(interval_t) + o.secs = d * SECS_PER_DAY + return o +end + +local function interval_hours_new(h) + check_number(h, "hours(number)") + local o = ffi.new(interval_t) + o.secs = h * 60 * 60 + return o +end + +local function interval_minutes_new(m) + check_number(m, "minutes(number)") + local o = ffi.new(interval_t) + o.secs = m * 60 + return o +end + +local function interval_seconds_new(s) + check_number(s, "seconds(number)") + local o = ffi.new(interval_t) + o.nsec = s % 1 * 1e9 + o.secs = s - (s % 1) + return o +end + +local function datetime_eq(lhs, rhs) + check_date_interval(lhs, "datetime:__eq(date or interval)") + check_date_interval(rhs, "datetime:__eq(date or interval)") return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) end local function datetime_lt(lhs, rhs) + check_date_interval(lhs, "datetime:__lt(date or interval)") + check_date_interval(rhs, "datetime:__lt(date or interval)") return (lhs.secs < rhs.secs) or (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) end local function datetime_le(lhs, rhs) + check_date_interval(lhs, "datetime:__le(date or interval)") + check_date_interval(rhs, "datetime:__le(date or interval)") return (lhs.secs <= rhs.secs) or (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) end @@ -224,19 +323,123 @@ local function interval_serialize(self) return { secs = self.secs, nsec = self.nsec } end +local function local_rd(o) + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET +end + +local function local_dt(o) + return cdt.dt_from_rdn(local_rd(o)) +end + +local function _normalize_nsec(secs, nsec) + if nsec < 0 then + secs = secs - 1 + nsec = nsec + NANOS_PER_SEC + elseif nsec >= NANOS_PER_SEC then + secs = secs + 1 + nsec = nsec - NANOS_PER_SEC + end + return secs, nsec +end + +-- addition or subtraction from date/time of a given interval +-- described via table direction should be +1 or -1 +local function interval_increment(self, o, direction) + assert(direction == -1 or direction == 1) + check_date(self, "interval_increment(date, object, -+1)") + assert(type(o) == 'table') + + local ym_updated = false + local dhms_updated = false + + local dt = local_dt(self) + local secs, nsec + secs, nsec = self.secs, self.nsec + + for key, value in pairs(o) do + local handlers = { + years = function(v) + assert(v > 0 and v < 10000) + dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT) + ym_updated = true + end, + + months = function(v) + assert(v > 0 and v < 13 ) + dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT) + ym_updated = true + end, + + weeks = function(v) + assert(v > 0 and v < 32) + secs = secs + direction * 7 * v * SECS_PER_DAY + dhms_updated = true + end, + + days = function(v) + assert(v > 0 and v < 32) + secs = secs + direction * v * SECS_PER_DAY + dhms_updated = true + end, + + hours = function(v) + assert(v >= 0 and v < 24) + secs = secs + direction * 60 * 60 * v + dhms_updated = true + end, + + minutes = function(v) + assert(v >= 0 and v < 60) + secs = secs + direction * 60 * v + end, + + seconds = function(v) + assert(v >= 0 and v < 61) + local s, frac + frac = v % 1 + if frac > 0 then + s = v - (v % 1) + else + s = v + end + secs = secs + direction * s + nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds + dhms_updated = true + end, + } + handlers[key](value) + end + + secs, nsec = _normalize_nsec(secs, nsec) + + -- .days, .hours, .minutes, .seconds + if dhms_updated then + self.secs = secs + self.nsec = nsec + end + + -- .years, .months updated + if ym_updated then + self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY + + secs % SECS_PER_DAY + end + + return self +end + local datetime_index = function(self, key) local attributes = { timestamp = function(self) return tonumber(self.secs) + self.nsec / 1e9 end, nanoseconds = function(self) - return tonumber(self.secs * 1e9 + self.nsec) + return self.secs * 1e9 + self.nsec end, microseconds = function(self) - return tonumber(self.secs * 1e6 + self.nsec / 1e3) + return self.secs * 1e6 + self.nsec / 1e3 end, milliseconds = function(self) - return tonumber(self.secs * 1e3 + self.nsec / 1e6) + return self.secs * 1e3 + self.nsec / 1e6 end, seconds = function(self) return tonumber(self.secs) + self.nsec / 1e9 @@ -250,32 +453,20 @@ local datetime_index = function(self, key) days = function(self) return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24 end, + add = function(self) + return function(self, o) + return interval_increment(self, o, 1) + end + end, + sub = function(self) + return function(self, o) + return interval_increment(self, o, -1) + end + end, } return attributes[key] ~= nil and attributes[key](self) or nil end -local datetime_mt = { - -- __tostring = datetime_tostring, - __serialize = datetime_serialize, - __eq = datetime_eq, - __lt = datetime_lt, - __le = datetime_le, - __sub = datetime_sub, - __add = datetime_add, - __index = datetime_index, -} - -local interval_mt = { - -- __tostring = interval_tostring, - __serialize = interval_serialize, - __eq = datetime_eq, - __lt = datetime_lt, - __le = datetime_le, - __sub = datetime_sub, - __add = datetime_add, - __index = datetime_index, -} - local function datetime_new_raw(secs, nsec, offset) local dt_obj = ffi.new(datetime_t) dt_obj.secs = secs @@ -284,14 +475,6 @@ local function datetime_new_raw(secs, nsec, offset) return dt_obj end -local function local_rd(o) - return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET -end - -local function local_dt(o) - return cdt.dt_from_rdn(local_rd(o)) -end - local function mk_timestamp(dt, sp, fp, offset) local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0 local spV = sp ~= nil and sp or 0 @@ -367,11 +550,12 @@ local function datetime_new(o) second = function(v) assert(v >= 0 and v < 61) frac = v % 1 - if frac then + if frac > 0 then s = v - (v % 1) else s = v end + frac = frac * 1e9 -- convert fraction to nanoseconds hms = true end, @@ -402,6 +586,153 @@ local function datetime_new(o) return mk_timestamp(dt, secs, frac, offset) end +local function datetime_tostring(o) + if ffi.typeof(o) == datetime_t then + local sz = 48 + local buff = ffi.new('char[?]', sz) + local len = native.datetime_to_string(o, buff, sz) + assert(len < sz) + return ffi.string(buff) + elseif ffi.typeof(o) == interval_years_t then + return ('%+d years'):format(o.y) + elseif ffi.typeof(o) == interval_months_t then + return ('%+d months'):format(o.m) + elseif ffi.typeof(o) == interval_t then + local ts = o.timestamp + local sign = '+' + + if ts < 0 then + ts = -ts + sign = '-' + end + + if ts < 60 then + return ('%s%s secs'):format(sign, ts) + elseif ts < 60 * 60 then + return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60) + elseif ts < 24 * 60 * 60 then + return ('%+d hours, %d minutes, %s seconds'):format( + o.hours, o.minutes % 60, ts % 60) + else + return ('%+d days, %d hours, %d minutes, %s seconds'):format( + o.days, o.hours % 24, o.minutes % 60, ts % 60) + end + end +end + +local function date_first(lhs, rhs) + if is_datetime(lhs) then + return lhs, rhs + else + return rhs, lhs + end +end + +--[[ +Matrix of subtraction operands eligibility and their result type + +| | datetime | interval | interval_months | interval_years | ++-----------------+-----------+----------+-----------------+----------------+ +| datetime | interval | datetime | datetime | datetime | +| interval | | interval | | | +| interval_months | | | interval_months | | +| interval_years | | | | interval_years | +]] +local function datetime_sub(lhs, rhs) + check_date_interval(lhs, "datetime:__sub(date or interval)") + local d, s = lhs, rhs + local left_t = ffi.typeof(d) + local right_t = ffi.typeof(s) + local o + + if left_t == datetime_t then + -- 1. left is date, right is date or generic interval + if (right_t == datetime_t or right_t == interval_t) then + o = right_t == datetime_t and interval_new() or datetime_new() + o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs, + lhs.nsec - rhs.nsec) + return o + -- 2. left is date, right is interval in months + elseif right_t == interval_months_t then + local dt = cdt.dt_add_months(local_dt(lhs), -rhs.m, cdt.DT_LIMIT) + return mk_timestamp(dt, lhs.secs % SECS_PER_DAY, + lhs.nsec, lhs.offset) + + -- 3. left is date, right is interval in years + elseif right_t == interval_years_t then + local dt = cdt.dt_add_years(local_dt(lhs), -rhs.y, cdt.DT_LIMIT) + return mk_timestamp(dt, lhs.secs % SECS_PER_DAY, + lhs.nsec, lhs.offset) + else + error("datetime:__sub(date or interval) - incompatible type of arguments", 2) + end + -- 4. both left and right are generic intervals + elseif left_t == interval_t and right_t == interval_t then + o = interval_new() + o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs, + lhs.nsec - rhs.nsec) + return o + -- 5. both left and right are intervals in months + elseif left_t == interval_months_t and right_t == interval_months_t then + return interval_months_new(lhs.m - rhs.m) + -- 5. both left and right are intervals in years + elseif left_t == interval_years_t and right_t == interval_years_t then + return interval_years_new(lhs.y - rhs.y) + else + error("datetime:__sub(date or interval) - incompatible type of arguments", 2) + end +end + +--[[ +Matrix of addition operands eligibility and their result type + +| | datetime | interval | interval_months | interval_years | ++-----------------+-----------+----------+-----------------+----------------+ +| datetime | datetime | datetime | datetime | datetime | +| interval | datetime | interval | | | +| interval_months | datetime | | interval_months | | +| interval_years | datetime | | | interval_years | +]] +local function datetime_add(lhs, rhs) + local d, s = date_first(lhs, rhs) + + check_date_interval(d, "datetime:__add(interval)") + check_interval(s, "datetime:__add(interval)") + local left_t = ffi.typeof(d) + local right_t = ffi.typeof(s) + local o + + -- 1. left is date, right is date or interval + if left_t == datetime_t and right_t == interval_t then + o = datetime_new() + o.secs, o.nsec = _normalize_nsec(d.secs + s.secs, d.nsec + s.nsec) + return o + -- 2. left is date, right is interval in months + elseif left_t == datetime_t and right_t == interval_months_t then + local dt = cdt.dt_add_months(local_dt(d), s.m, cdt.DT_LIMIT) + local secs = d.secs % SECS_PER_DAY + return mk_timestamp(dt, secs, d.nsec, d.offset or 0) + + -- 3. left is date, right is interval in years + elseif left_t == datetime_t and right_t == interval_years_t then + local dt = cdt.dt_add_years(local_dt(d), s.y, cdt.DT_LIMIT) + local secs = d.secs % SECS_PER_DAY + return mk_timestamp(dt, secs, d.nsec, d.offset or 0) + -- 4. both left and right are generic intervals + elseif left_t == interval_t and right_t == interval_t then + o = interval_new() + o.secs, o.nsec = _normalize_nsec(d.secs + s.secs, d.nsec + s.nsec) + return o + -- 5. both left and right are intervals in months + elseif left_t == interval_months_t and right_t == interval_months_t then + return interval_months_new(d.m + s.m) + -- 6. both left and right are intervals in years + elseif left_t == interval_years_t and right_t == interval_years_t then + return interval_years_new(d.y + s.y) + else + error("datetime:__add(date or interval) - incompatible type of arguments", 2) + end +end -- simple parse functions: -- parse_date/parse_time/parse_zone @@ -415,6 +746,7 @@ end ]] local function parse_date(str) + check_str("datetime.parse_date(string)") local dt = ffi.new('dt_t[1]') local len = cdt.dt_parse_iso_date(str, #str, dt) return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len) @@ -431,6 +763,7 @@ end The time designator [T] may be omitted. ]] local function parse_time(str) + check_str("datetime.parse_time(string)") local sp = ffi.new('int[1]') local fp = ffi.new('int[1]') local len = cdt.dt_parse_iso_time(str, #str, sp, fp) @@ -444,6 +777,7 @@ end ±hhmm ±hh:mm ]] local function parse_zone(str) + check_str("datetime.parse_zone(string)") local offset = ffi.new('int[1]') local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len) @@ -457,7 +791,8 @@ end date [T] time [ ] time_zone ]] -local function parse_str(str) +local function parse(str) + check_str("datetime.parse(string)") local dt = ffi.new('dt_t[1]') local len = #str local n = cdt.dt_parse_iso_date(str, len, dt) @@ -508,7 +843,7 @@ local function datetime_from(o) if o == nil or type(o) == 'table' then return datetime_new(o) elseif type(o) == 'string' then - return parse_str(o) + return parse(o) end end @@ -531,8 +866,8 @@ local function local_now() end local function datetime_to_tm_ptr(o) + assert(is_datetime(o)) local p_tm = ffi.new 'struct tm[1]' - assert(ffi.typeof(o) == datetime_t) -- dt_to_struct_tm() fills only date data cdt.dt_to_struct_tm(local_dt(o), p_tm) @@ -551,20 +886,21 @@ local function datetime_to_tm_ptr(o) end local function asctime(o) - assert(ffi.typeof(o) == datetime_t) + check_date(o, "datetime:asctime()") + local p_tm = datetime_to_tm_ptr(o) return ffi.string(native.asctime(p_tm)) end local function ctime(o) - assert(ffi.typeof(o) == datetime_t) + check_date(o, "datetime:ctime()") local p_time = ffi.new 'time_t[1]' p_time[0] = o.secs return ffi.string(native.ctime(p_time)) end local function strftime(fmt, o) - assert(ffi.typeof(o) == datetime_t) + check_date(o, "datetime.strftime(fmt, date)") local sz = 50 local buff = ffi.new('char[?]', sz) local p_tm = datetime_to_tm_ptr(o) @@ -572,36 +908,76 @@ local function strftime(fmt, o) return ffi.string(buff) end -local function datetime_tostring(o) - assert(ffi.typeof(o) == datetime_t) - local sz = 48 - local buff = ffi.new('char[?]', sz) - local len = native.datetime_to_string(o, buff, sz) - assert(len < sz) - return ffi.string(buff) -end +local datetime_mt = { + __tostring = datetime_tostring, + __serialize = datetime_serialize, + __eq = datetime_eq, + __lt = datetime_lt, + __le = datetime_le, + __sub = datetime_sub, + __add = datetime_add, + __index = datetime_index, + add = function(self, o) + self = interval_increment(self, o, 1) + return self + end, + sub = function(self, o) + self = interval_increment(self, o, -1) + return self + end +} + +local interval_mt = { + __tostring = datetime_tostring, + __serialize = interval_serialize, + __eq = datetime_eq, + __lt = datetime_lt, + __le = datetime_le, + __sub = datetime_sub, + __add = datetime_add, + __index = datetime_index, +} + +local interval_tiny_mt = { + __tostring = datetime_tostring, + __serialize = interval_serialize, + __sub = datetime_sub, + __add = datetime_add, + __index = datetime_index, +} ffi.metatype(interval_t, interval_mt) ffi.metatype(datetime_t, datetime_mt) +ffi.metatype(interval_years_t, interval_tiny_mt) +ffi.metatype(interval_months_t, interval_tiny_mt) return setmetatable( { - datetime = datetime_new, - interval = interval_new, - - parse = parse_str, - parse_date = parse_date, - parse_time = parse_time, - parse_zone = parse_zone, - - tostring = datetime_tostring, - - now = local_now, - -- strptime = strptime; - strftime = strftime, - asctime = asctime, - ctime = ctime, + new = datetime_new, + years = interval_years_new, + months = interval_months_new, + weeks = interval_weeks_new, + days = interval_days_new, + hours = interval_hours_new, + minutes = interval_minutes_new, + seconds = interval_seconds_new, + interval = interval_new, + + parse = parse, + parse_date = parse_date, + parse_time = parse_time, + parse_zone = parse_zone, + + tostring = datetime_tostring, + + now = local_now, + strftime = strftime, + asctime = asctime, + ctime = ctime, + + is_datetime = is_datetime, + is_interval = is_interval, }, { __call = function(self, ...) return datetime_from(...) end } diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua index 407d89556..21d13b3a6 100755 --- a/test/app-tap/datetime.test.lua +++ b/test/app-tap/datetime.test.lua @@ -4,7 +4,7 @@ local tap = require('tap') local test = tap.test("errno") local date = require('datetime') -test:plan(6) +test:plan(10) test:test("Simple tests for parser", function(test) test:plan(2) @@ -203,4 +203,165 @@ test:test("Parse tiny date into seconds and other parts", function(test) test:ok(tiny.hours == 0.00848, "hours") end) +test:test("Stringization of dates and intervals", function(test) + test:plan(13) + local str = '19700101Z' + local dt = date(str) + test:ok(tostring(dt) == '1970-01-01T00:00Z', ('tostring(%s)'):format(str)) + test:ok(tostring(date.seconds(12)) == '+12 secs', '+12 seconds') + test:ok(tostring(date.seconds(-12)) == '-12 secs', '-12 seconds') + test:ok(tostring(date.minutes(12)) == '+12 minutes, 0 seconds', '+12 minutes') + test:ok(tostring(date.minutes(-12)) == '-12 minutes, 0 seconds', '-12 minutes') + test:ok(tostring(date.hours(12)) == '+12 hours, 0 minutes, 0 seconds', + '+12 hours') + test:ok(tostring(date.hours(-12)) == '-12 hours, 0 minutes, 0 seconds', + '-12 hours') + test:ok(tostring(date.days(12)) == '+12 days, 0 hours, 0 minutes, 0 seconds', + '+12 days') + test:ok(tostring(date.days(-12)) == '-12 days, 0 hours, 0 minutes, 0 seconds', + '-12 days') + test:ok(tostring(date.months(5)) == '+5 months', '+5 months') + test:ok(tostring(date.months(-5)) == '-5 months', '-5 months') + test:ok(tostring(date.years(4)) == '+4 years', '+4 years') + test:ok(tostring(date.years(-4)) == '-4 years', '-4 years') +end) + +test:test("Time interval operations", function(test) + test:plan(12) + + -- check arithmetic with leap dates + local T = date('1972-02-29') + local M = date.months(2) + local Y = date.years(1) + test:ok(tostring(T + M) == '1972-04-29T00:00Z', ('T(%s) + M(%s'):format(T, M)) + test:ok(tostring(T + Y) == '1973-03-01T00:00Z', ('T(%s) + Y(%s'):format(T, Y)) + test:ok(tostring(T + M + Y) == '1973-04-30T00:00Z', + ('T(%s) + M(%s) + Y(%s'):format(T, M, Y)) + test:ok(tostring(T + Y + M) == '1973-05-01T00:00Z', + ('T(%s) + M(%s) + Y(%s'):format(T, M, Y)) + test:ok(tostring(T:add{years = 1, months = 2}) == '1973-04-30T00:00Z', + ('T:add{years=1,months=2}(%s)'):format(T)) + + -- check average, not leap dates + T = date('1970-01-08') + test:ok(tostring(T + M) == '1970-03-08T00:00Z', ('T(%s) + M(%s'):format(T, M)) + test:ok(tostring(T + Y) == '1971-01-08T00:00Z', ('T(%s) + Y(%s'):format(T, Y)) + test:ok(tostring(T + M + Y) == '1971-03-08T00:00Z', + ('T(%s) + M(%s) + Y(%s'):format(T, M, Y)) + test:ok(tostring(T + Y + M) == '1971-03-08T00:00Z', + ('T(%s) + Y(%s) + M(%s'):format(T, Y, M)) + test:ok(tostring(T:add{years = 1, months = 2}) == '1971-03-08T00:00Z', + ('T:add{years=1,months=2}(%s)'):format(T)) + + + -- subtraction of 2 dates + local T2 = date('19700103') + local T1 = date('1970-01-01') + test:ok(tostring(T2 - T1) == '+2 days, 0 hours, 0 minutes, 0 seconds', + ('T2(%s) - T1(%s'):format(T2, T1)) + test:ok(tostring(T1 - T2) == '-2 days, 0 hours, 0 minutes, 0 seconds', + ('T2(%s) - T1(%s'):format(T2, T1)) +end) + +local function catchadd(A, B) + return pcall(function() return A + B end) +end + +--[[ +Matrix of addition operands eligibility and their result type + +| | datetime | interval | interval_months | interval_years | ++-----------------+-----------+----------+-----------------+----------------+ +| datetime | datetime | datetime | datetime | datetime | +| interval | datetime | interval | | | +| interval_months | datetime | | interval_months | | +| interval_years | datetime | | | interval_years | +]] + +test:test("Matrix of allowed time and interval additions", function(test) + test:plan(20) + + -- check arithmetic with leap dates + local T1970 = date('1970-01-01') + local T2000 = date('2000-01-01') + local I1 = date.days(1) + local M2 = date.months(2) + local M10 = date.months(10) + local Y1 = date.years(1) + local Y5 = date.years(5) + + test:ok(catchadd(T1970, I1) == true, "status: T + I") + test:ok(catchadd(T1970, M2) == true, "status: T + M") + test:ok(catchadd(T1970, Y1) == true, "status: T + Y") + test:ok(catchadd(T1970, T2000) == false, "status: T + T") + test:ok(catchadd(I1, T1970) == true, "status: I + T") + test:ok(catchadd(M2, T1970) == true, "status: M + T") + test:ok(catchadd(Y1, T1970) == true, "status: Y + T") + test:ok(catchadd(I1, Y1) == false, "status: I + Y") + test:ok(catchadd(M2, Y1) == false, "status: M + Y") + test:ok(catchadd(I1, Y1) == false, "status: I + Y") + test:ok(catchadd(Y5, M10) == false, "status: Y + M") + test:ok(catchadd(Y5, I1) == false, "status: Y + I") + test:ok(catchadd(Y5, Y1) == true, "status: Y + Y") + + test:ok(tostring(T1970 + I1) == "1970-01-02T00:00Z", "value: T + I") + test:ok(tostring(T1970 + M2) == "1970-03-01T00:00Z", "value: T + M") + test:ok(tostring(T1970 + Y1) == "1971-01-01T00:00Z", "value: T + Y") + test:ok(tostring(I1 + T1970) == "1970-01-02T00:00Z", "value: I + T") + test:ok(tostring(M2 + T1970) == "1970-03-01T00:00Z", "value: M + T") + test:ok(tostring(Y1 + T1970) == "1971-01-01T00:00Z", "value: Y + T") + test:ok(tostring(Y5 + Y1) == "+6 years", "Y + Y") + +end) + +local function catchsub_status(A, B) + return pcall(function() return A - B end) +end + +--[[ +Matrix of subtraction operands eligibility and their result type + +| | datetime | interval | interval_months | interval_years | ++-----------------+-----------+----------+-----------------+----------------+ +| datetime | interval | datetime | datetime | datetime | +| interval | | interval | | | +| interval_months | | | interval_months | | +| interval_years | | | | interval_years | +]] +test:test("Matrix of allowed time and interval subtractions", function(test) + test:plan(18) + + -- check arithmetic with leap dates + local T1970 = date('1970-01-01') + local T2000 = date('2000-01-01') + local I1 = date.days(1) + local M2 = date.months(2) + local M10 = date.months(10) + local Y1 = date.years(1) + local Y5 = date.years(5) + + test:ok(catchsub_status(T1970, I1) == true, "status: T - I") + test:ok(catchsub_status(T1970, M2) == true, "status: T - M") + test:ok(catchsub_status(T1970, Y1) == true, "status: T - Y") + test:ok(catchsub_status(T1970, T2000) == true, "status: T - T") + test:ok(catchsub_status(I1, T1970) == false, "status: I + T") + test:ok(catchsub_status(M2, T1970) == false, "status: M + T") + test:ok(catchsub_status(Y1, T1970) == false, "status: Y + T") + test:ok(catchsub_status(I1, Y1) == false, "status: I - Y") + test:ok(catchsub_status(M2, Y1) == false, "status: M - Y") + test:ok(catchsub_status(I1, Y1) == false, "status: I - Y") + test:ok(catchsub_status(Y5, M10) == false, "status: Y - M") + test:ok(catchsub_status(Y5, I1) == false, "status: Y - I") + test:ok(catchsub_status(Y5, Y1) == true, "status: Y - Y") + + test:ok(tostring(T1970 - I1) == "1969-12-31T00:00Z", "value: T - I") + test:ok(tostring(T1970 - M2) == "1969-11-01T00:00Z", "value: T - M") + test:ok(tostring(T1970 - Y1) == "1969-01-01T00:00Z", "value: T - Y") + test:ok(tostring(T1970 - T2000) == "-10957 days, 0 hours, 0 minutes, 0 seconds", + "value: T - T") + test:ok(tostring(Y5 - Y1) == "+4 years", "value: Y - Y") + + +end) + os.exit(test:check() and 0 or 1) -- 2.29.2
* implemented proper range checks for date attributes values; * created `.unixtime` attribute, which is alias to `.secs`, with corresponding setter/getter; * similarly to `unixtime`, created virtual `timestamp` attribute setter. Which is a convenient way to simultaneously assign unixtime (seconds since epoch) and nanoseconds Part of #5941 --- src/lua/datetime.lua | 90 ++++++++++++++++++++++++++++++-------------- 1 file changed, 61 insertions(+), 29 deletions(-) diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua index 1466b923f..cc2ae119b 100644 --- a/src/lua/datetime.lua +++ b/src/lua/datetime.lua @@ -342,12 +342,33 @@ local function _normalize_nsec(secs, nsec) return secs, nsec end +local function seconds_fraction(v) + local seconds, fraction + fraction = v % 1 + if fraction > 0 then + seconds = v - (v % 1) + else + seconds = v + end + return seconds, fraction +end + +local function check_range(v, range, txt) + assert(#range == 2) + if not (v >= range[1] and v <= range[2]) then + error(('value %d of %s is out of allowed range [%d, %d]'): + format(v, txt, range[1], range[2])) + end +end + -- addition or subtraction from date/time of a given interval -- described via table direction should be +1 or -1 local function interval_increment(self, o, direction) assert(direction == -1 or direction == 1) - check_date(self, "interval_increment(date, object, -+1)") - assert(type(o) == 'table') + check_date(self, "interval_increment(date, object, direction)") + if type(o) ~= 'table' then + error('interval_increment(date, object, direction) - object expected', 2) + end local ym_updated = false local dhms_updated = false @@ -359,49 +380,43 @@ local function interval_increment(self, o, direction) for key, value in pairs(o) do local handlers = { years = function(v) - assert(v > 0 and v < 10000) + check_range(v, {0, 9999}, key) dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT) ym_updated = true end, months = function(v) - assert(v > 0 and v < 13 ) + check_range(v, {0, 12}, key) dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT) ym_updated = true end, weeks = function(v) - assert(v > 0 and v < 32) + check_range(v, {0, 52}, key) secs = secs + direction * 7 * v * SECS_PER_DAY dhms_updated = true end, days = function(v) - assert(v > 0 and v < 32) + check_range(v, {0, 31}, key) secs = secs + direction * v * SECS_PER_DAY dhms_updated = true end, hours = function(v) - assert(v >= 0 and v < 24) + check_range(v, {0, 23}, key) secs = secs + direction * 60 * 60 * v dhms_updated = true end, minutes = function(v) - assert(v >= 0 and v < 60) + check_range(v, {0, 59}, key) secs = secs + direction * 60 * v end, seconds = function(v) - assert(v >= 0 and v < 61) - local s, frac - frac = v % 1 - if frac > 0 then - s = v - (v % 1) - else - s = v - end + check_range(v, {0, 60}, key) + local s, frac = seconds_fraction(v) secs = secs + direction * s nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds dhms_updated = true @@ -429,6 +444,9 @@ end local datetime_index = function(self, key) local attributes = { + unixtime = function(self) + return self.secs + end, timestamp = function(self) return tonumber(self.secs) + self.nsec / 1e9 end, @@ -467,6 +485,24 @@ local datetime_index = function(self, key) return attributes[key] ~= nil and attributes[key](self) or nil end +local function datetime_newindex(self, key, value) + local attributes = { + unixtime = function(self, value) + self.secs = value + self.nsec, self.offset = 0, 0 + end, + timestamp = function(self, value) + local secs, frac = seconds_fraction(value) + self.secs = secs + self.nsec = frac * 1e9 + self.offset = 0 + end, + } + if attributes[key] ~= nil then + attributes[key](self, value) + end +end + local function datetime_new_raw(secs, nsec, offset) local dt_obj = ffi.new(datetime_t) dt_obj.secs = secs @@ -518,50 +554,45 @@ local function datetime_new(o) end, year = function(v) - assert(v > 0 and v < 10000) + check_range(v, {1, 9999}, key) y = v ymd = true end, month = function(v) - assert(v > 0 and v < 13 ) + check_range(v, {1, 12}, key) M = v ymd = true end, day = function(v) - assert(v > 0 and v < 32) + check_range(v, {1, 31}, key) d = v ymd = true end, hour = function(v) - assert(v >= 0 and v < 24) + check_range(v, {0, 23}, key) h = v hms = true end, minute = function(v) - assert(v >= 0 and v < 60) + check_range(v, {0, 59}, key) m = v hms = true end, second = function(v) - assert(v >= 0 and v < 61) - frac = v % 1 - if frac > 0 then - s = v - (v % 1) - else - s = v - end + check_range(v, {0, 60}, key) + s, frac = seconds_fraction(v) frac = frac * 1e9 -- convert fraction to nanoseconds hms = true end, -- tz offset in minutes tz = function(v) - assert(v >= 0 and v <= 720) + check_range(v, {0, 720}, key) offset = v end } @@ -918,6 +949,7 @@ local datetime_mt = { __sub = datetime_sub, __add = datetime_add, __index = datetime_index, + __newindex = datetime_newindex, add = function(self, o) self = interval_increment(self, o, 1) return self -- 2.29.2
Introduced new date/time/interval types support to lua and storage engines. Closes #5941 Closes #5946 --- changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md b/changelogs/unreleased/gh-5941-datetime-type-support.md new file mode 100644 index 000000000..3c755008e --- /dev/null +++ b/changelogs/unreleased/gh-5941-datetime-type-support.md @@ -0,0 +1,4 @@ +## feature/lua/datetime + + * Introduce new builtin module for date/time/interval support - `datetime.lua`. + Support of new datetime type in storage engines (gh-5941, gh-5946). -- 2.29.2
Thanks for your patch.
I can't say that it will be complete review but it's brief feedback.
I left comments for some patches. Currently it is not complete series,
in fact it looks like as one-two commits with portion of fixups on the
top of them.
I believe this module should be implemented in pure C. I don't think
that arithmetic will
work fast enough in C.
Probably datetime and interval should be logically splitted into 2 parts.
```
tarantool> require('datetime')
---
- weeks: 'function: 0x010ca0aa48'
minutes: 'function: 0x010ca694d8'
new: 'function: 0x010ca5e098'
strftime: 'function: 0x010caa8fb0'
tostring: 'function: 0x010ca2f700'
hours: 'function: 0x010ca671c0'
parse_date: 'function: 0x010ca96f80'
seconds: 'function: 0x010ca63b78'
parse_time: 'function: 0x010ca22208'
is_interval: 'function: 0x010caa6d20'
months: 'function: 0x010ca61130'
is_datetime: 'function: 0x010ca5d3a0'
asctime: 'function: 0x010caa8e28'
ctime: 'function: 0x010ca88260'
parse_zone: 'function: 0x010ca02eb0'
now: 'function: 0x010ca86e70'
days: 'function: 0x010ca81980'
years: 'function: 0x010ca61438'
parse: 'function: 0x010ca9ad38'
interval: 'function: 0x010c9feb60'
...
```
It's not clear that part is about intervals and datetime.
```
-- Currently
datetime.now() + datetime.week(1)
-- Possible version
datetime.now() + interval.week(1)
```
Interface is better than in previous time. Thanks for working on this
series.
On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * Version #2:
>
> - fixed problem with overloaded '-' and '+' operations for datetime
> arguments;
> - fixed messagepack serialization problems;
> - heavily documented MessagePack serialization schema in the code;
> - introduced working implementation of datetime hints for storage engines;
> - made interval related names be more consistent, renamed durations and period
> to intervals, i.e. t_datetime_duration to datetime_interval_t,
> duration_* to interval_*, period to interval;
> - properly implemented all reasonable cases of datetime+interval
> arithmetic;
> - moved all initialization code to utils.c;
> - renamed core/mp_datetime.c to core/datetime.c because it makes more
> sense now;
>
> * Version #1 - initial RFC series
>
> In brief
> --------
> This patchset implements datetime lua support in box, with serialization
> to messagepack, yaml, json and lua mode. Also it contains storage
> engines' indices implementation for datetime type introduced.
>
> * Current implementation is heavily influenced by Sci-Lua lua-time module
> https://github.com/stepelu/lua-time
> e.g. you could find very similar approach for handling of operations
> with year or month long intervals (which should be handled differently than
> usual intervals of seconds, or days).
>
> * But internally we actually use Christian Hanson' c-dt module
> https://github.com/chansen/c-dt
> (though it has been modified slightly for cleaner integration
> into cmake build process)
>
>
> Datetime Module API
> -------------------
>
> We used to have here draft documentation of datetime module api, but
> for a convenience it has been extracted to the discussion topic there -
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
>
> Messagepack serialization schema
> --------------------------------
>
> In short it looks like:
> - now we introduce new MP_EXT extension type #4;
> - we may save 1 required and 2 optional fields for datetime field;
>
> In details it's explained in MessagePack serialization schema depicted here:
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990
>
>
> https://github.com/tarantool/tarantool/issues/5941
> https://github.com/tarantool/tarantool/issues/5946
>
> https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v3
>
> Timur Safin (11):
> build: add Christian Hansen c-dt to the build
> lua: built-in module datetime
> lua, datetime: datetime tests
> lua, datetime: display datetime
> box, datetime: add messagepack support for datetime
> box, datetime: datetime comparison for indices
> lua, datetime: proper datetime encoding
> lua, datetime: calculated attributes for datetimes
> lua, datetime: time intervals support
> lua, datetime: unixtime, timestamp setters in datetime.lua
> datetime: changelog for datetime module
>
> .gitmodules | 3 +
> CMakeLists.txt | 8 +
> .../gh-5941-datetime-type-support.md | 4 +
> cmake/BuildCDT.cmake | 6 +
> src/CMakeLists.txt | 3 +
> src/box/field_def.c | 52 +-
> src/box/field_def.h | 4 +
> src/box/lua/serialize_lua.c | 7 +-
> src/box/memtx_space.c | 3 +-
> src/box/msgpack.c | 7 +-
> src/box/tuple_compare.cc | 50 +
> src/box/vinyl.c | 3 +-
> src/exports.h | 29 +
> src/lib/core/CMakeLists.txt | 4 +-
> src/lib/core/datetime.c | 251 ++++
> src/lib/core/datetime.h | 114 ++
> src/lib/core/mp_extension_types.h | 1 +
> src/lib/mpstream/mpstream.c | 11 +
> src/lib/mpstream/mpstream.h | 4 +
> src/lua/datetime.lua | 1016 +++++++++++++++++
> src/lua/init.c | 4 +-
> src/lua/msgpack.c | 12 +
> src/lua/msgpackffi.lua | 18 +
> src/lua/serializer.c | 4 +
> src/lua/serializer.h | 2 +
> src/lua/utils.c | 28 +-
> src/lua/utils.h | 12 +
> test/app-tap/datetime.test.lua | 367 ++++++
> test/engine/datetime.result | 77 ++
> test/engine/datetime.test.lua | 35 +
> test/unit/CMakeLists.txt | 2 +
> test/unit/datetime.c | 220 ++++
> test/unit/datetime.result | 358 ++++++
> third_party/c-dt | 1 +
> third_party/lua-cjson/lua_cjson.c | 8 +
> third_party/lua-yaml/lyaml.cc | 6 +-
> 36 files changed, 2709 insertions(+), 25 deletions(-)
> create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
> create mode 100644 cmake/BuildCDT.cmake
> create mode 100755 src/lib/core/datetime.c
> create mode 100644 src/lib/core/datetime.h
> create mode 100644 src/lua/datetime.lua
> create mode 100755 test/app-tap/datetime.test.lua
> create mode 100644 test/engine/datetime.result
> create mode 100644 test/engine/datetime.test.lua
> create mode 100644 test/unit/datetime.c
> create mode 100644 test/unit/datetime.result
> create mode 160000 third_party/c-dt
>
Thanks for your patch! Some places from prevous review are still not fixed for some reasons. Please be careful with our Lua style guide I ponted some obvious violations. Also it would be great to analyze module functions with our memprof. I think there are some places that could be optimized. On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: > * created a new Tarantool built-in module `datetime`; > * register cdef types for this module; > * export some `dt_*` functions from `c-dt` library; > * lua implementationis of `asctime` and `strftime`; > * datetime parsing unit tests, with and withput timezones; > * c test for reversible strftime roundtrip; > > Part of #5941 > --- > src/CMakeLists.txt | 2 + > src/exports.h | 21 ++ > src/lib/core/datetime.h | 61 ++++ > src/lua/datetime.lua | 581 ++++++++++++++++++++++++++++++++++++++ > src/lua/init.c | 4 +- > src/lua/utils.c | 27 ++ > src/lua/utils.h | 12 + > test/unit/CMakeLists.txt | 2 + > test/unit/datetime.c | 221 +++++++++++++++ > test/unit/datetime.result | 358 +++++++++++++++++++++++ > 10 files changed, 1288 insertions(+), 1 deletion(-) > create mode 100644 src/lib/core/datetime.h > create mode 100644 src/lua/datetime.lua > create mode 100644 test/unit/datetime.c > create mode 100644 test/unit/datetime.result > > diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt > index ef6a295d5..e0499e57f 100644 > --- a/src/CMakeLists.txt > +++ b/src/CMakeLists.txt > @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua) > lua_source(lua_sources lua/httpc.lua) > lua_source(lua_sources lua/iconv.lua) > lua_source(lua_sources lua/swim.lua) > +lua_source(lua_sources lua/datetime.lua) > + > # LuaJIT jit.* library > lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua) > lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua) > diff --git a/src/exports.h b/src/exports.h > index 5bb3e6a2b..db40c03a4 100644 > --- a/src/exports.h > +++ b/src/exports.h > @@ -531,3 +531,24 @@ EXPORT(uri_format) > EXPORT(uri_parse) > EXPORT(uuid_nil) > EXPORT(uuid_unpack) > +EXPORT(dt_from_rdn) > +EXPORT(dt_from_yd) > +EXPORT(dt_from_ymd) > +EXPORT(dt_from_yqd) > +EXPORT(dt_from_ywd) > +EXPORT(dt_to_yd) > +EXPORT(dt_to_ymd) > +EXPORT(dt_to_yqd) > +EXPORT(dt_to_ywd) > +EXPORT(dt_rdn) > +EXPORT(dt_dow) > +EXPORT(dt_parse_iso_date) > +EXPORT(dt_parse_iso_time) > +EXPORT(dt_parse_iso_time_basic) > +EXPORT(dt_parse_iso_time_extended) > +EXPORT(dt_parse_iso_zone) > +EXPORT(dt_parse_iso_zone_basic) > +EXPORT(dt_parse_iso_zone_extended) > +EXPORT(dt_parse_iso_zone_lenient) > +EXPORT(dt_from_struct_tm) > +EXPORT(dt_to_struct_tm) > diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h > new file mode 100644 > index 000000000..403bf1c64 > --- /dev/null > +++ b/src/lib/core/datetime.h > @@ -0,0 +1,61 @@ > +#pragma once > +/* > + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * 1. Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the > + * following disclaimer. > + * > + * 2. Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL > + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, > + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR > + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF > + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF > + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +#include <c-dt/dt_core.h> > +#include <stdint.h> > +#include <stdbool.h> > + > +#if defined(__cplusplus) > +extern "C" { > +#endif /* defined(__cplusplus) */ > + > +/** > + * datetime structure consisting of: > + */ > +struct datetime_t { > + int64_t secs; ///< seconds since epoch > + int32_t nsec; ///< nanoseconds if any > + int32_t offset; ///< offset in minutes from GMT > +}; > + > +/** > + * Date/time delta structure > + */ > +struct datetime_interval_t { > + int64_t secs; ///< relative seconds delta > + int32_t nsec; ///< nanoseconds delta > +}; > + > +#if defined(__cplusplus) > +} /* extern "C" */ > +#endif /* defined(__cplusplus) */ > + > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua > new file mode 100644 > index 000000000..0996ca5a2 > --- /dev/null > +++ b/src/lua/datetime.lua > @@ -0,0 +1,581 @@ > +local ffi = require('ffi') Do we have any benchmarks that shows that FFI is faster than Lua C API? Or it's just easy way to implement such module. > +local cdt = ffi.C > + > +ffi.cdef [[ > + > + typedef int dt_t; > + > + // dt_core.h > + typedef enum { > + DT_MON = 1, > + DT_MONDAY = 1, > + DT_TUE = 2, > + DT_TUESDAY = 2, > + DT_WED = 3, > + DT_WEDNESDAY = 3, > + DT_THU = 4, > + DT_THURSDAY = 4, > + DT_FRI = 5, > + DT_FRIDAY = 5, > + DT_SAT = 6, > + DT_SATURDAY = 6, > + DT_SUN = 7, > + DT_SUNDAY = 7, > + } dt_dow_t; > + > + dt_t dt_from_rdn (int n); > + dt_t dt_from_yd (int y, int d); > + dt_t dt_from_ymd (int y, int m, int d); > + dt_t dt_from_yqd (int y, int q, int d); > + dt_t dt_from_ywd (int y, int w, int d); > + > + void dt_to_yd (dt_t dt, int *y, int *d); > + void dt_to_ymd (dt_t dt, int *y, int *m, int *d); > + void dt_to_yqd (dt_t dt, int *y, int *q, int *d); > + void dt_to_ywd (dt_t dt, int *y, int *w, int *d); > + > + int dt_rdn (dt_t dt); > + dt_dow_t dt_dow (dt_t dt); > + > + // dt_parse_iso.h > + size_t dt_parse_iso_date (const char *str, size_t len, dt_t *dt); > + > + size_t dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec); > + size_t dt_parse_iso_time_basic (const char *str, size_t len, int *sod, int *nsec); > + size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec); > + > + size_t dt_parse_iso_zone (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_basic (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset); > + > + // dt_tm.h > + dt_t dt_from_struct_tm (const struct tm *tm); > + void dt_to_struct_tm (dt_t dt, struct tm *tm); > + > + // <asm-generic/posix_types.h> > + typedef long __kernel_long_t; > + typedef unsigned long __kernel_ulong_t; > + // /usr/include/x86_64-linux-gnu/bits/types/time_t.h > + typedef long time_t; > + > + > + // <time.h> > + typedef __kernel_long_t __kernel_time_t; > + typedef __kernel_long_t __kernel_suseconds_t; > + > + struct timespec { > + __kernel_time_t tv_sec; /* seconds */ > + long tv_nsec; /* nanoseconds */ > + }; > + > + struct timeval { > + __kernel_time_t tv_sec; /* seconds */ > + __kernel_suseconds_t tv_usec; /* microseconds */ > + }; > + > + struct timezone { > + int tz_minuteswest; /* minutes west of Greenwich */ > + int tz_dsttime; /* type of dst correction */ > + }; > + > + // /usr/include/x86_64-linux-gnu/sys/time.h > + typedef struct timezone * __timezone_ptr_t; > + > + /* Get the current time of day and timezone information, > + putting it into *TV and *TZ. If TZ is NULL, *TZ is not filled. > + Returns 0 on success, -1 on errors. > + > + NOTE: This form of timezone information is obsolete. > + Use the functions and variables declared in <time.h> instead. */ > + int gettimeofday (struct timeval *__tv, struct timezone * __tz); > + > + // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h > + /* ISO C `broken-down time' structure. */ > + struct tm > + { > + int tm_sec; /* Seconds. [0-60] (1 leap second) */ > + int tm_min; /* Minutes. [0-59] */ > + int tm_hour; /* Hours. [0-23] */ > + int tm_mday; /* Day. [1-31] */ > + int tm_mon; /* Month. [0-11] */ > + int tm_year; /* Year - 1900. */ > + int tm_wday; /* Day of week. [0-6] */ > + int tm_yday; /* Days in year.[0-365] */ > + int tm_isdst; /* DST. [-1/0/1]*/ > + > + long int tm_gmtoff; /* Seconds east of UTC. */ > + const char *tm_zone;/* Timezone abbreviation. */ > + }; > + > + // <time.h> > + /* Return the current time and put it in *TIMER if TIMER is not NULL. */ > + time_t time (time_t *__timer); > + > + /* Format TP into S according to FORMAT. > + Write no more than MAXSIZE characters and return the number > + of characters written, or 0 if it would exceed MAXSIZE. */ > + size_t strftime (char * __s, size_t __maxsize, const char * __format, > + const struct tm * __tp); > + > + /* Parse S according to FORMAT and store binary time information in TP. > + The return value is a pointer to the first unparsed character in S. */ > + char *strptime (const char * __s, const char * __fmt, struct tm *__tp); > + > + /* Return the `struct tm' representation of *TIMER in UTC, > + using *TP to store the result. */ > + struct tm *gmtime_r (const time_t * __timer, struct tm * __tp); > + > + /* Return the `struct tm' representation of *TIMER in local time, > + using *TP to store the result. */ > + struct tm *localtime_r (const time_t * __timer, struct tm * __tp); > + > + /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n" > + that is the representation of TP in this format. */ > + char *asctime (const struct tm *__tp); > + > + /* Equivalent to `asctime (localtime (timer))'. */ > + char *ctime (const time_t *__timer); > + > +]] > + > +local native = ffi.C You've aleady declared cdt as ffi.C. There is a redefinition here. Is it really needed. I don't see anybody use "native" for ffi.C. There are two ways - "builtin" and "C". I suppose to choose one of them. > + > +local SECS_PER_DAY = 86400 > +local NANOS_PER_SEC = 1000000000LL > + > +-- c-dt/dt_config.h > + > +-- Unix, January 1, 1970, Thursday > +local DT_EPOCH_1970_OFFSET = 719163LL To be honest it's not completely clear that such value means until I visited https://github.com/chansen/c-dt/blob/21b8cd1fcb984386b7d4552c16fdd03fafab2b6a/dt_config.h#L50. I think some comment is needed here. > + > + > +local datetime_t = ffi.typeof('struct datetime_t') > +local interval_t = ffi.typeof('struct datetime_interval_t') > + > +local function interval_new() > + local interval = ffi.new(interval_t) > + return interval > +end > + > +local function adjusted_secs(dt) > + return dt.secs - dt.offset * 60 > +end > + > +local function datetime_sub(lhs, rhs) > + local s1 = adjusted_secs(lhs) > + local s2 = adjusted_secs(rhs) > + local d = interval_new() > + d.secs = s2 - s1 > + d.nsec = rhs.nsec - lhs.nsec > + if d.nsec < 0 then > + d.secs = d.secs - 1 > + d.nsec = d.nsec + NANOS_PER_SEC > + end > + return d > +end > + > +local function datetime_add(lhs, rhs) > + local s1 = adjusted_secs(lhs) > + local s2 = adjusted_secs(rhs) > + local d = interval_new() > + d.secs = s2 + s1 > + d.nsec = rhs.nsec + lhs.nsec > + if d.nsec >= NANOS_PER_SEC then > + d.secs = d.secs + 1 > + d.nsec = d.nsec - NANOS_PER_SEC > + end > + return d > +end > + > +local function datetime_eq(lhs, rhs) > + -- we usually don't need to check nullness > + -- but older tarantool console will call us checking for equality to nil > + if rhs == nil then > + return false > + end > + return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) > +end > + As in the previous review it will fail on attempt to compare with not datetime value. tarantool> datetime.new() == newproxy() --- - error: 'builtin/datetime.lua:222: bad argument #1 to ''is_datetime'' (C type expected, got userdata)' ... Expected false. > + > +local function datetime_lt(lhs, rhs) > + return (lhs.secs < rhs.secs) or > + (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) > +end > + > +local function datetime_le(lhs, rhs) > + return (lhs.secs <= rhs.secs) or > + (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) > +end > + > +local function datetime_serialize(self) > + -- Allow YAML, MsgPack and JSON to dump objects with sockets > + return { secs = self.secs, nsec = self.nsec, tz = self.offset } > +end > + > +local function interval_serialize(self) > + -- Allow YAML and JSON to dump objects with sockets > + return { secs = self.secs, nsec = self.nsec } > +end > + > +local datetime_mt = { > + -- __tostring = datetime_tostring, > + __serialize = datetime_serialize, > + __eq = datetime_eq, > + __lt = datetime_lt, > + __le = datetime_le, > + __sub = datetime_sub, > + __add = datetime_add, > + > + nanoseconds = function(self) > + return tonumber(self.secs*NANOS_PER_SEC + self.nsec) I think there should be a space before and after "*". Here and below. > + end, > + microseconds = function(self) > + return tonumber(self.secs*1e6 + self.nsec*1e3) > + end, > + seconds = function(self) > + return tonumber(self.secs + self.nsec*1e3) > + end, > + minutes = function(self) > + return tonumber((self._ticks/(1e6*60))%60) > + end, > + hours = function(self) > + return tonumber(self._ticks/(1e6*60*60)) > + end, > + I think this empty line could be removed. > +} > + > +local interval_mt = { > + -- __tostring = interval_tostring, > + __serialize = interval_serialize, > + __eq = datetime_eq, > + __lt = datetime_lt, > + __le = datetime_le, > +} > + > +local function datetime_new_raw(secs, nsec, offset) > + local dt_obj = ffi.new(datetime_t) > + dt_obj.secs = secs > + dt_obj.nsec = nsec > + dt_obj.offset = offset > + return dt_obj > +end > + > +local function local_rd(o) > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET > +end > + > +local function local_dt(o) > + return cdt.dt_from_rdn(local_rd(o)) > +end > + > +local function mk_timestamp(dt, sp, fp, offset) > + local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0 > + local spV = sp ~= nil and sp or 0 > + local fpV = fp ~= nil and fp or 0 > + local ofsV = offset ~= nil and offset or 0 > + return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV) > +end > + > +-- create @datetime_t given object @o fields > +local function datetime_new(o) > + if o == nil then > + return datetime_new_raw(0, 0, 0) > + end > + local secs = 0 > + local nsec = 0 > + local offset = 0 > + local easy_way = false What does "easy_way" mean? > + local y, M, d, ymd > + y, M, d, ymd = 0, 0, 0, false > + > + local h, m, s, frac, hms > + h, m, s, frac, hms = 0, 0, 0, 0, false > + > + local dt = 0 > + > + for key, value in pairs(o) do > + local handlers = { Here you recreate handlers for each iteration of the loop. I think it should be reworked. Currenlty it's quite slow I think even if-elseif-else branches will work faster without creating redundant GC objects. > + secs = function(v) > + secs = v > + easy_way = true > + end, > + > + nsec = function(v) > + nsec = v > + easy_way = true > + end, > + > + offset = function (v) > + offset = v > + easy_way = true > + end, > + > + year = function(v) > + assert(v > 0 and v < 10000) Still there are some assertions that yield unclear error messages. > + y = v > + ymd = true > + end, > + > + month = function(v) > + assert(v > 0 and v < 12 ) > + M = v > + ymd = true > + end, > + > + day = function(v) > + assert(v > 0 and v < 32) > + d = v > + ymd = true > + end, > + > + hour = function(v) > + assert(v >= 0 and v < 24) > + h = v > + hms = true > + end, > + > + minute = function(v) > + assert(v >= 0 and v < 60) > + m = v > + hms = true > + end, > + > + second = function(v) > + assert(v >= 0 and v < 61) > + frac = v % 1 > + if frac then > + s = v - (v % 1) > + else > + s = v > + end > + hms = true > + end, > + > + -- tz offset in minutes > + tz = function(v) > + assert(v >= 0 and v <= 720) > + offset = v > + end > + } > + handlers[key](value) > + end > + > + -- .sec, .nsec, .offset > + if easy_way then > + return datetime_new_raw(secs, nsec, offset) > + end > + > + -- .year, .month, .day > + if ymd then > + dt = dt + cdt.dt_from_ymd(y, M, d) > + end > + > + -- .hour, .minute, .second > + if hms then > + secs = h * 3600 + m * 60 + s > + end > + > + return mk_timestamp(dt, secs, frac, offset) > +end > + > + > +-- simple parse functions: > +-- parse_date/parse_time/parse_zone > + > +--[[ > + Basic Extended > + 20121224 2012-12-24 Calendar date (ISO 8601) > + 2012359 2012-359 Ordinal date (ISO 8601) > + 2012W521 2012-W52-1 Week date (ISO 8601) > + 2012Q485 2012-Q4-85 Quarter date > +]] > + > +local function parse_date(str) > + local dt = ffi.new('dt_t[1]') > + local rc = cdt.dt_parse_iso_date(str, #str, dt) > + assert(rc > 0) > + return mk_timestamp(dt[0]) > +end > + > +--[[ > + Basic Extended > + T12 N/A > + T1230 T12:30 > + T123045 T12:30:45 > + T123045.123456789 T12:30:45.123456789 > + T123045,123456789 T12:30:45,123456789 > + > + The time designator [T] may be omitted. > +]] > +local function parse_time(str) > + local sp = ffi.new('int[1]') > + local fp = ffi.new('int[1]') > + local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) > + assert(rc > 0) > + return mk_timestamp(nil, sp[0], fp[0]) > +end > + > +--[[ > + Basic Extended > + Z N/A > + ±hh N/A > + ±hhmm ±hh:mm > +]] > +local function parse_zone(str) > + local offset = ffi.new('int[1]') > + local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) > + return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len) > +end > + > + > +--[[ > + aggregated parse functions > + assumes to deal with date T time time_zone > + at once > + > + date [T] time [ ] time_zone > +]] > +local function parse_str(str) > + local dt = ffi.new('dt_t[1]') > + local len = #str > + local n = cdt.dt_parse_iso_date(str, len, dt) > + local dt_ = dt[0] > + if n == 0 or len == n then > + return mk_timestamp(dt_) > + end > + > + str = str:sub(tonumber(n) + 1) > + > + local ch = str:sub(1,1) > + if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then > + return mk_timestamp(dt_) > + end > + > + str = str:sub(2) > + len = #str > + > + local sp = ffi.new('int[1]') > + local fp = ffi.new('int[1]') > + local n = cdt.dt_parse_iso_time(str, len, sp, fp) > + if n == 0 then > + return mk_timestamp(dt_) > + end > + local sp_ = sp[0] > + local fp_ = fp[0] > + if len == n then > + return mk_timestamp(dt_, sp_, fp_) > + end > + > + str = str:sub(tonumber(n) + 1) > + > + if str:sub(1,1) == ' ' then > + str = str:sub(2) > + end > + > + len = #str > + > + local offset = ffi.new('int[1]') > + n = cdt.dt_parse_iso_zone_lenient(str, len, offset) > + if n == 0 then > + return mk_timestamp(dt_, sp_, fp_) > + end > + return mk_timestamp(dt_, sp_, fp_, offset[0]) > +end > + > +local function datetime_from(o) > + if o == nil or type(o) == 'table' then > + return datetime_new(o) > + elseif type(o) == 'string' then > + return parse_str(o) > + end > +end > + > +local function local_now() > + local p_tv = ffi.new ' struct timeval [1] ' This line doesn't conform our code-style. Please wrap argument into brackets. The same for such places below. > + local rc = native.gettimeofday(p_tv, nil) > + assert(rc == 0) > + > + local secs = p_tv[0].tv_sec > + local nsec = p_tv[0].tv_usec * 1000 > + > + local p_time = ffi.new 'time_t[1]' > + local p_tm = ffi.new 'struct tm[1]' > + native.time(p_time) > + native.localtime_r(p_time, p_tm) > + -- local dt = cdt.dt_from_struct_tm(p_tm) > + local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes > + > + return datetime_new_raw(secs, nsec, ofs) -- FIXME Do you plan to fix it before merge? > +end > + > +local function datetime_to_tm_ptr(o) > + local p_tm = ffi.new 'struct tm[1]' > + assert(ffi.typeof(o) == datetime_t) > + -- dt_to_struct_tm() fills only date data > + cdt.dt_to_struct_tm(local_dt(o), p_tm) > + > + -- calculate the smaller data (hour, minute, > + -- seconds) using datetime seconds value > + local seconds_of_day = o.secs % 86400 > + local hour = (seconds_of_day / 3600) % 24 > + local minute = (seconds_of_day / 60) % 60 > + p_tm[0].tm_sec = seconds_of_day % 60 > + p_tm[0].tm_min = minute > + p_tm[0].tm_hour = hour > + > + p_tm[0].tm_gmtoff = o.offset * 60 > + > + return p_tm > +end > + > +local function asctime(o) > + assert(ffi.typeof(o) == datetime_t) > + local p_tm = datetime_to_tm_ptr(o) > + return ffi.string(native.asctime(p_tm)) > +end > + > +local function ctime(o) > + assert(ffi.typeof(o) == datetime_t) > + local p_time = ffi.new 'time_t[1]' > + p_time[0] = o.secs > + return ffi.string(native.ctime(p_time)) > +end > + > +local function strftime(fmt, o) > + assert(ffi.typeof(o) == datetime_t) > + local sz = 50 Why 50? > + local buff = ffi.new('char[?]', sz) > + local p_tm = datetime_to_tm_ptr(o) > + native.strftime(buff, sz, fmt, p_tm) > + return ffi.string(buff) > +end > + > +-- strftime may be redirected to datetime:fmt("format") > +local function datetime_fmt() > +end > + > + > +ffi.metatype(interval_t, interval_mt) > +ffi.metatype(datetime_t, datetime_mt) > + > +return setmetatable( > + { > + datetime = datetime_new, > + interval = interval_new, > + > + parse = parse_str, > + parse_date = parse_date, > + parse_time = parse_time, > + parse_zone = parse_zone, > + fmt = datetime_fmt, > + > + now = local_now, > + -- strptime = strptime; It should be dropped if you don't need it. > + strftime = strftime, > + asctime = asctime, > + ctime = ctime, > + }, { > + __call = function(self, ...) return datetime_from(...) end > + } > +) > diff --git a/src/lua/init.c b/src/lua/init.c > index f9738025d..127e935d7 100644 > --- a/src/lua/init.c > +++ b/src/lua/init.c > @@ -129,7 +129,8 @@ extern char strict_lua[], > parse_lua[], > process_lua[], > humanize_lua[], > - memprof_lua[] > + memprof_lua[], > + datetime_lua[] > ; > > static const char *lua_modules[] = { > @@ -184,6 +185,7 @@ static const char *lua_modules[] = { > "memprof.process", process_lua, > "memprof.humanize", humanize_lua, > "memprof", memprof_lua, > + "datetime", datetime_lua, > NULL > }; > > diff --git a/src/lua/utils.c b/src/lua/utils.c > index 34cec0eed..6aeb18cfe 100644 > --- a/src/lua/utils.c > +++ b/src/lua/utils.c > @@ -47,6 +47,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR; > static uint32_t CTID_CHAR_PTR; > static uint32_t CTID_CONST_CHAR_PTR; > uint32_t CTID_UUID; > +uint32_t CTID_DATETIME = 0; > +uint32_t CTID_INTERVAL = 0; > + > > void * > luaL_pushcdata(struct lua_State *L, uint32_t ctypeid) > @@ -107,6 +110,12 @@ luaL_pushuuid(struct lua_State *L) > return luaL_pushcdata(L, CTID_UUID); > } > > +struct datetime_t * > +luaL_pushdatetime(struct lua_State *L) > +{ > + return luaL_pushcdata(L, CTID_DATETIME); > +} > + > int > luaL_iscdata(struct lua_State *L, int idx) > { > @@ -712,6 +721,24 @@ tarantool_lua_utils_init(struct lua_State *L) > CTID_UUID = luaL_ctypeid(L, "struct tt_uuid"); > assert(CTID_UUID != 0); > > + rc = luaL_cdef(L, "struct datetime_t {" > + "int64_t secs;" > + "int32_t nsec;" > + "int32_t offset;" > + "};"); > + assert(rc == 0); > + (void) rc; > + CTID_DATETIME = luaL_ctypeid(L, "struct datetime_t"); > + assert(CTID_DATETIME != 0); > + rc = luaL_cdef(L, "struct datetime_interval_t {" > + "int64_t secs;" > + "int32_t nsec;" > + "};"); > + assert(rc == 0); > + (void) rc; > + CTID_INTERVAL = luaL_ctypeid(L, "struct datetime_interval_t"); > + assert(CTID_INTERVAL != 0); > + > lua_pushcfunction(L, luaT_newthread_wrapper); > luaT_newthread_ref = luaL_ref(L, LUA_REGISTRYINDEX); > return 0; > diff --git a/src/lua/utils.h b/src/lua/utils.h > index 947d9240b..afd41c75b 100644 > --- a/src/lua/utils.h > +++ b/src/lua/utils.h > @@ -59,6 +59,7 @@ struct lua_State; > struct ibuf; > typedef struct ibuf box_ibuf_t; > struct tt_uuid; > +struct datetime_t; > > /** > * Single global lua_State shared by core and modules. > @@ -69,10 +70,21 @@ struct tt_uuid; > extern struct lua_State *tarantool_L; > > extern uint32_t CTID_UUID; > +extern uint32_t CTID_DATETIME; > +extern uint32_t CTID_INTERVAL; > > struct tt_uuid * > luaL_pushuuid(struct lua_State *L); > > +/** > + * @brief Push cdata of a datetime_t type onto the stack. > + * @param L Lua State > + * @sa luaL_pushcdata > + * @return memory associated with this datetime_t data > + */ > +struct datetime_t * > +luaL_pushdatetime(struct lua_State *L); > + > /** \cond public */ > > /** > diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt > index 5bb7cd6e7..f8320aebd 100644 > --- a/test/unit/CMakeLists.txt > +++ b/test/unit/CMakeLists.txt > @@ -56,6 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c) > target_link_libraries(uuid.test uuid unit) > add_executable(random.test random.c core_test_utils.c) > target_link_libraries(random.test core unit) > +add_executable(datetime.test datetime.c) > +target_link_libraries(datetime.test cdt unit) > > add_executable(bps_tree.test bps_tree.cc) > target_link_libraries(bps_tree.test small misc) > diff --git a/test/unit/datetime.c b/test/unit/datetime.c > new file mode 100644 > index 000000000..b6f568c03 > --- /dev/null > +++ b/test/unit/datetime.c > @@ -0,0 +1,221 @@ > +#include "dt.h" > +#include <assert.h> > +#include <stdint.h> > +#include <string.h> > +#include <time.h> > + > +#include "unit.h" > + > +const char sample[] = "2012-12-24T15:30Z"; > + > +#define S(s) {s, sizeof(s) - 1} > +struct { > + const char * sz; > + size_t len; > +} tests[] = { > + S("2012-12-24 15:30Z"), > + S("2012-12-24 15:30z"), > + S("2012-12-24 15:30"), > + S("2012-12-24 16:30+01:00"), > + S("2012-12-24 16:30+0100"), > + S("2012-12-24 16:30+01"), > + S("2012-12-24 14:30-01:00"), > + S("2012-12-24 14:30-0100"), > + S("2012-12-24 14:30-01"), > + S("2012-12-24 15:30:00Z"), > + S("2012-12-24 15:30:00z"), > + S("2012-12-24 15:30:00"), > + S("2012-12-24 16:30:00+01:00"), > + S("2012-12-24 16:30:00+0100"), > + S("2012-12-24 14:30:00-01:00"), > + S("2012-12-24 14:30:00-0100"), > + S("2012-12-24 15:30:00.123456Z"), > + S("2012-12-24 15:30:00.123456z"), > + S("2012-12-24 15:30:00.123456"), > + S("2012-12-24 16:30:00.123456+01:00"), > + S("2012-12-24 16:30:00.123456+01"), > + S("2012-12-24 14:30:00.123456-01:00"), > + S("2012-12-24 14:30:00.123456-01"), > + S("2012-12-24t15:30Z"), > + S("2012-12-24t15:30z"), > + S("2012-12-24t15:30"), > + S("2012-12-24t16:30+01:00"), > + S("2012-12-24t16:30+0100"), > + S("2012-12-24t14:30-01:00"), > + S("2012-12-24t14:30-0100"), > + S("2012-12-24t15:30:00Z"), > + S("2012-12-24t15:30:00z"), > + S("2012-12-24t15:30:00"), > + S("2012-12-24t16:30:00+01:00"), > + S("2012-12-24t16:30:00+0100"), > + S("2012-12-24t14:30:00-01:00"), > + S("2012-12-24t14:30:00-0100"), > + S("2012-12-24t15:30:00.123456Z"), > + S("2012-12-24t15:30:00.123456z"), > + S("2012-12-24t16:30:00.123456+01:00"), > + S("2012-12-24t14:30:00.123456-01:00"), > + S("2012-12-24 16:30 +01:00"), > + S("2012-12-24 14:30 -01:00"), > + S("2012-12-24 15:30 UTC"), > + S("2012-12-24 16:30 UTC+1"), > + S("2012-12-24 16:30 UTC+01"), > + S("2012-12-24 16:30 UTC+0100"), > + S("2012-12-24 16:30 UTC+01:00"), > + S("2012-12-24 14:30 UTC-1"), > + S("2012-12-24 14:30 UTC-01"), > + S("2012-12-24 14:30 UTC-01:00"), > + S("2012-12-24 14:30 UTC-0100"), > + S("2012-12-24 15:30 GMT"), > + S("2012-12-24 16:30 GMT+1"), > + S("2012-12-24 16:30 GMT+01"), > + S("2012-12-24 16:30 GMT+0100"), > + S("2012-12-24 16:30 GMT+01:00"), > + S("2012-12-24 14:30 GMT-1"), > + S("2012-12-24 14:30 GMT-01"), > + S("2012-12-24 14:30 GMT-01:00"), > + S("2012-12-24 14:30 GMT-0100"), > + S("2012-12-24 14:30 -01:00"), > + S("2012-12-24 16:30:00 +01:00"), > + S("2012-12-24 14:30:00 -01:00"), > + S("2012-12-24 16:30:00.123456 +01:00"), > + S("2012-12-24 14:30:00.123456 -01:00"), > + S("2012-12-24 15:30:00.123456 -00:00"), > + S("20121224T1630+01:00"), > + S("2012-12-24T1630+01:00"), > + S("20121224T16:30+01"), > + S("20121224T16:30 +01"), > +}; > +#undef S > + > +#define DIM(a) (sizeof(a) / sizeof(a[0])) > + > +// p5-time-moment/src/moment_parse.c: parse_string_lenient() > +static int > +parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np, > + int32_t *op) > +{ > + size_t n; > + dt_t dt; > + char c; > + int sod = 0, nanosecond = 0, offset = 0; > + > + n = dt_parse_iso_date(str, len, &dt); > + if (!n) > + return 1; > + if (n == len) > + goto exit; > + > + c = str[n++]; > + if (!(c == 'T' || c == 't' || c == ' ')) > + return 1; > + > + str += n; > + len -= n; > + > + n = dt_parse_iso_time(str, len, &sod, &nanosecond); > + if (!n) > + return 1; > + if (n == len) > + goto exit; > + > + if (str[n] == ' ') > + n++; > + > + str += n; > + len -= n; > + > + n = dt_parse_iso_zone_lenient(str, len, &offset); > + if (!n || n != len) > + return 1; > + > +exit: > + *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60; > + *np = nanosecond; > + *op = offset; > + > + return 0; > +} > + > +// avoid introducing external datetime.h dependency > +// - just copy paste it for today > +#define SECS_PER_DAY 86400 > +#define NANOS_PER_SEC 1000000000 > +#define DT_EPOCH_1970_OFFSET 719163 > + > + > +struct datetime_t { > + int64_t secs; > + int32_t nsec; > + int32_t offset; > +}; > + > +static int > +local_rd(const struct datetime_t * dt) { > + return (int)(dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET; > +} > + > +static int > +local_dt(const struct datetime_t * dt) { > + return dt_from_rdn(local_rd(dt)); > +} > + > + > +struct tm* > +datetime_to_tm(struct datetime_t * dt) > +{ > + static struct tm tm; > + > + memset(&tm, 0, sizeof(tm)); > + dt_to_struct_tm(local_dt(dt), &tm); > + > + int seconds_of_day = dt->secs % 86400; > + tm.tm_hour = (seconds_of_day / 3600) % 24; > + tm.tm_min = (seconds_of_day / 60) % 60; > + tm.tm_sec = seconds_of_day % 60; > + > + return &tm; > +} > + > +static void datetime_test(void) > +{ > + size_t index; > + int64_t secs_expected; > + int32_t nanosecs; > + int32_t ofs; > + > + plan(355); > + parse_datetime(sample, sizeof(sample) - 1, > + &secs_expected, &nanosecs, &ofs); > + > + for (index = 0; index < DIM(tests); index++) { > + int64_t secs; > + int rc = parse_datetime(tests[index].sz, tests[index].len, > + &secs, &nanosecs, &ofs); > + is(rc, 0, "correct parse_datetime return value for '%s'", > + tests[index].sz); > + is(secs, secs_expected, "correct parse_datetime output " > + "seconds for '%s", tests[index].sz); > + > + // check that stringized literal produces the same date > + // time fields > + static char buff[40]; > + struct datetime_t dt = {secs, nanosecs, ofs}; > + // datetime_to_tm returns time in GMT zone > + struct tm * p_tm = datetime_to_tm(&dt); > + size_t len = strftime(buff, sizeof buff, "%F %T%z", p_tm); > + ok(len > 0, "strftime"); > + rc = parse_datetime(buff, len, &dt.secs, &dt.nsec, &dt.offset); > + is(rc, 0, "correct parse_datetime return value for '%s'", buff); > + is(secs, dt.secs, > + "reversible seconds via strftime for '%s", buff); > + } > +} > + > +int > +main(void) > +{ > + plan(1); > + datetime_test(); > + > + return check_plan(); > +} > diff --git a/test/unit/datetime.result b/test/unit/datetime.result > new file mode 100644 > index 000000000..f106fa769 > --- /dev/null > +++ b/test/unit/datetime.result > @@ -0,0 +1,358 @@ > +1..1 > + 1..355 > + ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z' > + ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z > + ok 3 - strftime > + ok 4 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 5 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 6 - correct parse_datetime return value for '2012-12-24 15:30z' > + ok 7 - correct parse_datetime output seconds for '2012-12-24 15:30z > + ok 8 - strftime > + ok 9 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 10 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 11 - correct parse_datetime return value for '2012-12-24 15:30' > + ok 12 - correct parse_datetime output seconds for '2012-12-24 15:30 > + ok 13 - strftime > + ok 14 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 15 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 16 - correct parse_datetime return value for '2012-12-24 16:30+01:00' > + ok 17 - correct parse_datetime output seconds for '2012-12-24 16:30+01:00 > + ok 18 - strftime > + ok 19 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 20 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 21 - correct parse_datetime return value for '2012-12-24 16:30+0100' > + ok 22 - correct parse_datetime output seconds for '2012-12-24 16:30+0100 > + ok 23 - strftime > + ok 24 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 25 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 26 - correct parse_datetime return value for '2012-12-24 16:30+01' > + ok 27 - correct parse_datetime output seconds for '2012-12-24 16:30+01 > + ok 28 - strftime > + ok 29 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 30 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 31 - correct parse_datetime return value for '2012-12-24 14:30-01:00' > + ok 32 - correct parse_datetime output seconds for '2012-12-24 14:30-01:00 > + ok 33 - strftime > + ok 34 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 35 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 36 - correct parse_datetime return value for '2012-12-24 14:30-0100' > + ok 37 - correct parse_datetime output seconds for '2012-12-24 14:30-0100 > + ok 38 - strftime > + ok 39 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 40 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 41 - correct parse_datetime return value for '2012-12-24 14:30-01' > + ok 42 - correct parse_datetime output seconds for '2012-12-24 14:30-01 > + ok 43 - strftime > + ok 44 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 45 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 46 - correct parse_datetime return value for '2012-12-24 15:30:00Z' > + ok 47 - correct parse_datetime output seconds for '2012-12-24 15:30:00Z > + ok 48 - strftime > + ok 49 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 50 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 51 - correct parse_datetime return value for '2012-12-24 15:30:00z' > + ok 52 - correct parse_datetime output seconds for '2012-12-24 15:30:00z > + ok 53 - strftime > + ok 54 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 55 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 56 - correct parse_datetime return value for '2012-12-24 15:30:00' > + ok 57 - correct parse_datetime output seconds for '2012-12-24 15:30:00 > + ok 58 - strftime > + ok 59 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 60 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 61 - correct parse_datetime return value for '2012-12-24 16:30:00+01:00' > + ok 62 - correct parse_datetime output seconds for '2012-12-24 16:30:00+01:00 > + ok 63 - strftime > + ok 64 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 65 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 66 - correct parse_datetime return value for '2012-12-24 16:30:00+0100' > + ok 67 - correct parse_datetime output seconds for '2012-12-24 16:30:00+0100 > + ok 68 - strftime > + ok 69 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 70 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 71 - correct parse_datetime return value for '2012-12-24 14:30:00-01:00' > + ok 72 - correct parse_datetime output seconds for '2012-12-24 14:30:00-01:00 > + ok 73 - strftime > + ok 74 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 75 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 76 - correct parse_datetime return value for '2012-12-24 14:30:00-0100' > + ok 77 - correct parse_datetime output seconds for '2012-12-24 14:30:00-0100 > + ok 78 - strftime > + ok 79 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 80 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 81 - correct parse_datetime return value for '2012-12-24 15:30:00.123456Z' > + ok 82 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456Z > + ok 83 - strftime > + ok 84 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 85 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 86 - correct parse_datetime return value for '2012-12-24 15:30:00.123456z' > + ok 87 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456z > + ok 88 - strftime > + ok 89 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 90 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 91 - correct parse_datetime return value for '2012-12-24 15:30:00.123456' > + ok 92 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 > + ok 93 - strftime > + ok 94 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 95 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 96 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01:00' > + ok 97 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01:00 > + ok 98 - strftime > + ok 99 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 100 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 101 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01' > + ok 102 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01 > + ok 103 - strftime > + ok 104 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 105 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 106 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01:00' > + ok 107 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01:00 > + ok 108 - strftime > + ok 109 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 110 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 111 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01' > + ok 112 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01 > + ok 113 - strftime > + ok 114 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 115 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 116 - correct parse_datetime return value for '2012-12-24t15:30Z' > + ok 117 - correct parse_datetime output seconds for '2012-12-24t15:30Z > + ok 118 - strftime > + ok 119 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 120 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 121 - correct parse_datetime return value for '2012-12-24t15:30z' > + ok 122 - correct parse_datetime output seconds for '2012-12-24t15:30z > + ok 123 - strftime > + ok 124 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 125 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 126 - correct parse_datetime return value for '2012-12-24t15:30' > + ok 127 - correct parse_datetime output seconds for '2012-12-24t15:30 > + ok 128 - strftime > + ok 129 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 130 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 131 - correct parse_datetime return value for '2012-12-24t16:30+01:00' > + ok 132 - correct parse_datetime output seconds for '2012-12-24t16:30+01:00 > + ok 133 - strftime > + ok 134 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 135 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 136 - correct parse_datetime return value for '2012-12-24t16:30+0100' > + ok 137 - correct parse_datetime output seconds for '2012-12-24t16:30+0100 > + ok 138 - strftime > + ok 139 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 140 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 141 - correct parse_datetime return value for '2012-12-24t14:30-01:00' > + ok 142 - correct parse_datetime output seconds for '2012-12-24t14:30-01:00 > + ok 143 - strftime > + ok 144 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 145 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 146 - correct parse_datetime return value for '2012-12-24t14:30-0100' > + ok 147 - correct parse_datetime output seconds for '2012-12-24t14:30-0100 > + ok 148 - strftime > + ok 149 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 150 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 151 - correct parse_datetime return value for '2012-12-24t15:30:00Z' > + ok 152 - correct parse_datetime output seconds for '2012-12-24t15:30:00Z > + ok 153 - strftime > + ok 154 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 155 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 156 - correct parse_datetime return value for '2012-12-24t15:30:00z' > + ok 157 - correct parse_datetime output seconds for '2012-12-24t15:30:00z > + ok 158 - strftime > + ok 159 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 160 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 161 - correct parse_datetime return value for '2012-12-24t15:30:00' > + ok 162 - correct parse_datetime output seconds for '2012-12-24t15:30:00 > + ok 163 - strftime > + ok 164 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 165 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 166 - correct parse_datetime return value for '2012-12-24t16:30:00+01:00' > + ok 167 - correct parse_datetime output seconds for '2012-12-24t16:30:00+01:00 > + ok 168 - strftime > + ok 169 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 170 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 171 - correct parse_datetime return value for '2012-12-24t16:30:00+0100' > + ok 172 - correct parse_datetime output seconds for '2012-12-24t16:30:00+0100 > + ok 173 - strftime > + ok 174 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 175 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 176 - correct parse_datetime return value for '2012-12-24t14:30:00-01:00' > + ok 177 - correct parse_datetime output seconds for '2012-12-24t14:30:00-01:00 > + ok 178 - strftime > + ok 179 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 180 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 181 - correct parse_datetime return value for '2012-12-24t14:30:00-0100' > + ok 182 - correct parse_datetime output seconds for '2012-12-24t14:30:00-0100 > + ok 183 - strftime > + ok 184 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 185 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 186 - correct parse_datetime return value for '2012-12-24t15:30:00.123456Z' > + ok 187 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456Z > + ok 188 - strftime > + ok 189 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 190 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 191 - correct parse_datetime return value for '2012-12-24t15:30:00.123456z' > + ok 192 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456z > + ok 193 - strftime > + ok 194 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 195 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 196 - correct parse_datetime return value for '2012-12-24t16:30:00.123456+01:00' > + ok 197 - correct parse_datetime output seconds for '2012-12-24t16:30:00.123456+01:00 > + ok 198 - strftime > + ok 199 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 200 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 201 - correct parse_datetime return value for '2012-12-24t14:30:00.123456-01:00' > + ok 202 - correct parse_datetime output seconds for '2012-12-24t14:30:00.123456-01:00 > + ok 203 - strftime > + ok 204 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 205 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 206 - correct parse_datetime return value for '2012-12-24 16:30 +01:00' > + ok 207 - correct parse_datetime output seconds for '2012-12-24 16:30 +01:00 > + ok 208 - strftime > + ok 209 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 210 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 211 - correct parse_datetime return value for '2012-12-24 14:30 -01:00' > + ok 212 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00 > + ok 213 - strftime > + ok 214 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 215 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 216 - correct parse_datetime return value for '2012-12-24 15:30 UTC' > + ok 217 - correct parse_datetime output seconds for '2012-12-24 15:30 UTC > + ok 218 - strftime > + ok 219 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 220 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 221 - correct parse_datetime return value for '2012-12-24 16:30 UTC+1' > + ok 222 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+1 > + ok 223 - strftime > + ok 224 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 225 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 226 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01' > + ok 227 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01 > + ok 228 - strftime > + ok 229 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 230 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 231 - correct parse_datetime return value for '2012-12-24 16:30 UTC+0100' > + ok 232 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+0100 > + ok 233 - strftime > + ok 234 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 235 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 236 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01:00' > + ok 237 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01:00 > + ok 238 - strftime > + ok 239 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 240 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 241 - correct parse_datetime return value for '2012-12-24 14:30 UTC-1' > + ok 242 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-1 > + ok 243 - strftime > + ok 244 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 245 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 246 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01' > + ok 247 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01 > + ok 248 - strftime > + ok 249 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 250 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 251 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01:00' > + ok 252 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01:00 > + ok 253 - strftime > + ok 254 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 255 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 256 - correct parse_datetime return value for '2012-12-24 14:30 UTC-0100' > + ok 257 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-0100 > + ok 258 - strftime > + ok 259 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 260 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 261 - correct parse_datetime return value for '2012-12-24 15:30 GMT' > + ok 262 - correct parse_datetime output seconds for '2012-12-24 15:30 GMT > + ok 263 - strftime > + ok 264 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 265 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 266 - correct parse_datetime return value for '2012-12-24 16:30 GMT+1' > + ok 267 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+1 > + ok 268 - strftime > + ok 269 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 270 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 271 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01' > + ok 272 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01 > + ok 273 - strftime > + ok 274 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 275 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 276 - correct parse_datetime return value for '2012-12-24 16:30 GMT+0100' > + ok 277 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+0100 > + ok 278 - strftime > + ok 279 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 280 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 281 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01:00' > + ok 282 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01:00 > + ok 283 - strftime > + ok 284 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 285 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 286 - correct parse_datetime return value for '2012-12-24 14:30 GMT-1' > + ok 287 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-1 > + ok 288 - strftime > + ok 289 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 290 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 291 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01' > + ok 292 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01 > + ok 293 - strftime > + ok 294 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 295 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 296 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01:00' > + ok 297 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01:00 > + ok 298 - strftime > + ok 299 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 300 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 301 - correct parse_datetime return value for '2012-12-24 14:30 GMT-0100' > + ok 302 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-0100 > + ok 303 - strftime > + ok 304 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 305 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 306 - correct parse_datetime return value for '2012-12-24 14:30 -01:00' > + ok 307 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00 > + ok 308 - strftime > + ok 309 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 310 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 311 - correct parse_datetime return value for '2012-12-24 16:30:00 +01:00' > + ok 312 - correct parse_datetime output seconds for '2012-12-24 16:30:00 +01:00 > + ok 313 - strftime > + ok 314 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 315 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 316 - correct parse_datetime return value for '2012-12-24 14:30:00 -01:00' > + ok 317 - correct parse_datetime output seconds for '2012-12-24 14:30:00 -01:00 > + ok 318 - strftime > + ok 319 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 320 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 321 - correct parse_datetime return value for '2012-12-24 16:30:00.123456 +01:00' > + ok 322 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456 +01:00 > + ok 323 - strftime > + ok 324 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 325 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 326 - correct parse_datetime return value for '2012-12-24 14:30:00.123456 -01:00' > + ok 327 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456 -01:00 > + ok 328 - strftime > + ok 329 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 330 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 331 - correct parse_datetime return value for '2012-12-24 15:30:00.123456 -00:00' > + ok 332 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 -00:00 > + ok 333 - strftime > + ok 334 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 335 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 336 - correct parse_datetime return value for '20121224T1630+01:00' > + ok 337 - correct parse_datetime output seconds for '20121224T1630+01:00 > + ok 338 - strftime > + ok 339 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 340 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 341 - correct parse_datetime return value for '2012-12-24T1630+01:00' > + ok 342 - correct parse_datetime output seconds for '2012-12-24T1630+01:00 > + ok 343 - strftime > + ok 344 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 345 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 346 - correct parse_datetime return value for '20121224T16:30+01' > + ok 347 - correct parse_datetime output seconds for '20121224T16:30+01 > + ok 348 - strftime > + ok 349 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 350 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > + ok 351 - correct parse_datetime return value for '20121224T16:30 +01' > + ok 352 - correct parse_datetime output seconds for '20121224T16:30 +01 > + ok 353 - strftime > + ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00+0000' > + ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00+0000 > +ok 1 - subtests
Thanks for your patch. To be honest I don't understand the structure of this changes. I expect that "datetime tests" will contain only tests. All unrelated to tests changes should be squashed to the previous patch. I put several comments below. On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: > * created app-tap test for new builtin module `datetime.lua` > * added case to check datetime string formatting using: > - asctime (gmt time); > - ctime (local TZ time); > - strftime (using given format). > > * added positive/negative checks to datetime test > - extended api of datetime.parse_date, .parse_time, .parse_time_zone > with a length of parsed (sub)string; > - this allows us to check partially valid strings like "20121224 Foo bar". > > Part of #5941 > --- > src/lua/datetime.lua | 14 ++- > test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++ > 2 files changed, 197 insertions(+), 8 deletions(-) > create mode 100755 test/app-tap/datetime.test.lua > > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua > index 0996ca5a2..f4d2d7737 100644 > --- a/src/lua/datetime.lua > +++ b/src/lua/datetime.lua > @@ -318,7 +318,7 @@ local function datetime_new(o) > end, > > month = function(v) > - assert(v > 0 and v < 12 ) > + assert(v > 0 and v < 13 ) > M = v > ymd = true > end, > @@ -393,9 +393,8 @@ end > > local function parse_date(str) > local dt = ffi.new('dt_t[1]') > - local rc = cdt.dt_parse_iso_date(str, #str, dt) > - assert(rc > 0) > - return mk_timestamp(dt[0]) > + local len = cdt.dt_parse_iso_date(str, #str, dt) > + return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len) > end > > --[[ > @@ -411,9 +410,8 @@ end > local function parse_time(str) > local sp = ffi.new('int[1]') > local fp = ffi.new('int[1]') > - local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) > - assert(rc > 0) > - return mk_timestamp(nil, sp[0], fp[0]) > + local len = cdt.dt_parse_iso_time(str, #str, sp, fp) > + return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil, tonumber(len) > end > > --[[ > @@ -448,7 +446,7 @@ local function parse_str(str) > str = str:sub(tonumber(n) + 1) > > local ch = str:sub(1,1) > - if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then > + if ch:match('[Tt ]') == nil then > return mk_timestamp(dt_) > end > > diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua > new file mode 100755 > index 000000000..09c968858 > --- /dev/null > +++ b/test/app-tap/datetime.test.lua > @@ -0,0 +1,191 @@ > +#!/usr/bin/env tarantool > + > +local tap = require('tap') > +local test = tap.test("errno") > +local date = require('datetime') > + > +test:plan(5) > + > +test:test("Simple tests for parser", function(test) > + test:plan(2) > + test:ok(date("1970-01-01T01:00:00Z") == > + date {year=1970, month=1, day=1, hour=1, minute=0, second=0}) > + test:ok(date("1970-01-01T02:00:00+02:00") == > + date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120}) > +end) > + > +test:test("Multiple tests for parser (with nanoseconds)", function(test) > + test:plan(165) > + -- borrowed from p5-time-moments/t/180_from_string.t > + local tests = > + { > + { '1970-01-01T00:00:00Z', 0, 0, 0 }, > + { '1970-01-01T02:00:00+02:00', 0, 0, 120 }, > + { '1970-01-01T01:30:00+01:30', 0, 0, 90 }, > + { '1970-01-01T01:00:00+01:00', 0, 0, 60 }, > + { '1970-01-01T00:01:00+00:01', 0, 0, 1 }, > + { '1970-01-01T00:00:00+00:00', 0, 0, 0 }, > + { '1969-12-31T23:59:00-00:01', 0, 0, -1 }, > + { '1969-12-31T23:00:00-01:00', 0, 0, -60 }, > + { '1969-12-31T22:30:00-01:30', 0, 0, -90 }, > + { '1969-12-31T22:00:00-02:00', 0, 0, -120 }, > + { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 }, > + { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 }, > + { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 }, > + { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 }, > + { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 }, > + { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 }, > + { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 }, > + { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 }, > + { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 }, > + { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 }, > + { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 }, > + { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 }, > + { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 }, > + { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 }, > + { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 }, > + { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 }, > + { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 }, > + { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 }, > + { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 }, > + { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 }, > + { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 }, > + { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 }, > + { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 }, > + { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 }, > + { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 }, > + { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 }, > + { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 }, > + { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 }, > + { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 }, > + { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 }, > + { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 }, > + { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 }, > + { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 }, > + { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 }, > + { '1970-01-01T00:00:00.0Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.00Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.0000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.00000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.000000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 }, > + { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 }, > + { '1973-11-29T21:33:09Z', 123456789, 0, 0 }, > + { '2013-10-28T17:51:56Z', 1382982716, 0, 0 }, > + -- { '9999-12-31T23:59:59Z', 253402300799, 0, 0 }, Why is it commented? > + } > + for _, value in ipairs(tests) do > + local str, epoch, nsec, offset > + str, epoch, nsec, offset = unpack(value) > + local dt = date(str) > + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch)) > + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec)) > + test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset)) > + end > +end) > + > +local ffi = require('ffi') > + > +ffi.cdef [[ > + void tzset(void); > +]] > + It's probably better to require all modules at the top of the file. > +test:test("Datetime string formatting", function(test) > + test:plan(7) > + local str = "1970-01-01" > + local t = date(str) > + test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, tonumber(t.secs))) > + test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec)) > + test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset)) > + test:ok(date.asctime(t) == 'Thu Jan 1 00:00:00 1970\n', ('%s: asctime'):format(str)) > + -- ctime() is local timezone dependent. To make sure that > + -- test is deterministic we enforce timezone via TZ environment > + -- manipulations and calling tzset() > + > + -- redefine timezone to be always GMT-2 > + os.setenv('TZ', 'GMT-2') > + ffi.C.tzset() > + test:ok(date.ctime(t) == 'Thu Jan 1 02:00:00 1970\n', ('%s: ctime with timezone'):format(str)) > + test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime #1'):format(str)) > + test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January 1970', ('%s: strftime #2'):format(str)) > +end) > + > +test:test("Parse iso date - valid strings", function(test) > + test:plan(32) > + local good = { > + {2012, 12, 24, "20121224", 8 }, > + {2012, 12, 24, "20121224 Foo bar", 8 }, > + {2012, 12, 24, "2012-12-24", 10 }, > + {2012, 12, 24, "2012-12-24 23:59:59", 10 }, > + {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 }, > + {2012, 12, 24, "2012359", 7 }, > + {2012, 12, 24, "2012359T235959+0130", 7 }, > + {2012, 12, 24, "2012-359", 8 }, > + {2012, 12, 24, "2012W521", 8 }, > + {2012, 12, 24, "2012-W52-1", 10 }, > + {2012, 12, 24, "2012Q485", 8 }, > + {2012, 12, 24, "2012-Q4-85", 10 }, > + { 1, 1, 1, "0001-Q1-01", 10 }, > + { 1, 1, 1, "0001-W01-1", 10 }, > + { 1, 1, 1, "0001-01-01", 10 }, > + { 1, 1, 1, "0001-001", 8 }, > + } > + > + for _, value in ipairs(good) do > + local year, month, day, str, date_part_len; I didn't see before that somebody uses semicolon after variables definition. It's about code-style. It is clear that this is syntactically correct. > + year, month, day, str, date_part_len = unpack(value) > + local expected_date = date{year = year, month = month, day = day} > + local date_part, len > + date_part, len = date.parse_date(str) > + test:ok(len == date_part_len, ('%s: length check %d'):format(str, len)) > + test:ok(expected_date == date_part, ('%s: expected date'):format(str)) > + end > +end) > + > +test:test("Parse iso date - invalid strings", function(test) > + test:plan(62) > + local bad = { > + "20121232" , -- Invalid day of month > + "2012-12-310", -- Invalid day of month > + "2012-13-24" , -- Invalid month > + "2012367" , -- Invalid day of year > + "2012-000" , -- Invalid day of year > + "2012W533" , -- Invalid week of year > + "2012-W52-8" , -- Invalid day of week > + "2012Q495" , -- Invalid day of quarter > + "2012-Q5-85" , -- Invalid quarter > + "20123670" , -- Trailing digit > + "201212320" , -- Trailing digit > + "2012-12" , -- Reduced accuracy > + "2012-Q4" , -- Reduced accuracy > + "2012-Q42" , -- Invalid > + "2012-Q1-1" , -- Invalid day of quarter > + "2012Q--420" , -- Invalid > + "2012-Q-420" , -- Invalid > + "2012Q11" , -- Incomplete > + "2012Q1234" , -- Trailing digit > + "2012W12" , -- Incomplete > + "2012W1234" , -- Trailing digit > + "2012W-123" , -- Invalid > + "2012-W12" , -- Incomplete > + "2012-W12-12", -- Trailing digit > + "2012U1234" , -- Invalid > + "2012-1234" , -- Invalid > + "2012-X1234" , -- Invalid > + "0000-Q1-01" , -- Year less than 0001 > + "0000-W01-1" , -- Year less than 0001 > + "0000-01-01" , -- Year less than 0001 > + "0000-001" , -- Year less than 0001 > + } > + > + for _, str in ipairs(bad) do > + local date_part, len > + date_part, len = date.parse_date(str) > + test:ok(len == 0, ('%s: length check %d'):format(str, len)) > + test:ok(date_part == nil, ('%s: empty date check %s'):format(str, date_part)) > + end > +end) > + > +os.exit(test:check() and 0 or 1)
Thanks for your patch. It's not review at all. I have a question:
why is datetime not scalar?
Also I think that encoding and type system changes should live in
separate patches.
On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * introduced output routine for converting datetime
> to their default output format.
>
> * use this routine for tostring() in datetime.lua
>
> Part of #5941
> ---
> src/exports.h | 1 +
> src/lib/core/CMakeLists.txt | 1 +
> src/lib/core/datetime.c | 85 +++++++++++++++++++++++++++++++++++++
> src/lib/core/datetime.h | 14 ++++++
> src/lua/datetime.lua | 18 ++++++--
> 5 files changed, 116 insertions(+), 3 deletions(-)
> create mode 100755 src/lib/core/datetime.c
>
> diff --git a/src/exports.h b/src/exports.h
> index db40c03a4..1a03db636 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -552,3 +552,4 @@ EXPORT(dt_parse_iso_zone_extended)
> EXPORT(dt_parse_iso_zone_lenient)
> EXPORT(dt_from_struct_tm)
> EXPORT(dt_to_struct_tm)
> +EXPORT(datetime_to_string)
> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
> index 2cd4d0b4f..8bc776b82 100644
> --- a/src/lib/core/CMakeLists.txt
> +++ b/src/lib/core/CMakeLists.txt
> @@ -30,6 +30,7 @@ set(core_sources
> decimal.c
> mp_decimal.c
> cord_buf.c
> + datetime.c
> )
>
> if (TARGET_OS_NETBSD)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> new file mode 100755
> index 000000000..65f813a70
> --- /dev/null
> +++ b/src/lib/core/datetime.c
> @@ -0,0 +1,85 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <string.h>
> +
> +#include "trivia/util.h"
> +#include "datetime.h"
> +
> +int
> +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len)
> +{
> + char * src = buf;
> + int offset = date->offset;
> + int64_t secs = date->secs + offset * 60;
> + dt_t dt = dt_from_rdn((secs / SECS_PER_DAY) + 719163);
> +
> + int year, month, day, sec, ns, sign;
> + dt_to_ymd(dt, &year, &month, &day);
> +
> + int hour = (secs / 3600) % 24,
> + minute = (secs / 60) % 60;
> + ;
> + sec = secs % 60;
> + ns = date->nsec;
> + uint32_t sz;
> + sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
> + year, month, day, hour, minute);
> + buf += sz; len -= sz;
> + if (sec || ns) {
> + sz = snprintf(buf, len, ":%02d", sec);
> + buf += sz; len -= sz;
> + if (ns) {
> + if ((ns % 1000000) == 0)
> + sz = snprintf(buf, len, ".%03d", ns / 1000000);
> + else if ((ns % 1000) == 0)
> + sz = snprintf(buf, len, ".%06d", ns / 1000);
> + else
> + sz = snprintf(buf, len, ".%09d", ns);
> + buf += sz; len -= sz;
> + }
> + }
> + if (offset == 0) {
> + strncpy(buf, "Z", len);
> + buf++;
> + len--;
> + }
> + else {
> + if (offset < 0)
> + sign = '-', offset = -offset;
> + else
> + sign = '+';
> +
> + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
> + buf += sz; len -= sz;
> + }
> + return (buf - src);
> +}
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 403bf1c64..fdaff2d27 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -38,6 +38,11 @@
> extern "C" {
> #endif /* defined(__cplusplus) */
>
> +#ifndef SECS_PER_DAY
> +#define SECS_PER_DAY 86400
> +#define NANOS_PER_SEC 1000000000
> +#endif
> +
> /**
> * datetime structure consisting of:
> */
> @@ -55,6 +60,15 @@ struct datetime_interval_t {
> int32_t nsec; ///< nanoseconds delta
> };
>
> +/**
> + * Convert datetime to string using default format
> + * @param date source datetime value
> + * @param buf output character buffer
> + * @param len size ofoutput buffer
> + */
> +int
> +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
> +
> #if defined(__cplusplus)
> } /* extern "C" */
> #endif /* defined(__cplusplus) */
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index f4d2d7737..9ec06d8d8 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -53,6 +53,12 @@ ffi.cdef [[
> dt_t dt_from_struct_tm (const struct tm *tm);
> void dt_to_struct_tm (dt_t dt, struct tm *tm);
>
> + // mp_datetime.c
> +
> + int
> + datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
> +
> +
> // <asm-generic/posix_types.h>
> typedef long __kernel_long_t;
> typedef unsigned long __kernel_ulong_t;
> @@ -549,8 +555,13 @@ local function strftime(fmt, o)
> return ffi.string(buff)
> end
>
> --- strftime may be redirected to datetime:fmt("format")
> -local function datetime_fmt()
> +local function datetime_tostring(o)
> + assert(ffi.typeof(o) == datetime_t)
> + local sz = 48
> + local buff = ffi.new('char[?]', sz)
> + local len = native.datetime_to_string(o, buff, sz)
> + assert(len < sz)
> + return ffi.string(buff)
> end
>
>
> @@ -566,7 +577,8 @@ return setmetatable(
> parse_date = parse_date,
> parse_time = parse_time,
> parse_zone = parse_zone,
> - fmt = datetime_fmt,
> +
> + tostring = datetime_tostring,
>
> now = local_now,
> -- strptime = strptime;
Thanks for your patch. One question below. Also update with arithmetic operations doesn't work. ``` tarantool> box.space.dt:update({v}, {{'+', 2, 3}}) --- - error: 'Argument type in operation ''+'' on field 2 does not match field type: expected a number' ... tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}}) --- - error: unsupported Lua type 'cdata' ... ``` On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: > * storage hints implemented for datetime_t values; > * proper comparison for indices of datetime type. > > > diff --git a/test/engine/datetime.result b/test/engine/datetime.result > new file mode 100644 > index 000000000..3ff474dee > --- /dev/null > +++ b/test/engine/datetime.result > @@ -0,0 +1,77 @@ > +-- test-run result file version 2 > +env = require('test_run') > + | --- > + | ... > +test_run = env.new() > + | --- > + | ... > +engine = test_run:get_cfg('engine') > + | --- > + | ... > + > +date = require('datetime') > + | --- > + | ... > + > +_ = box.schema.space.create('T', {engine = engine}) > + | --- > + | ... > +_ = box.space.T:create_index('pk', {parts={1,'datetime'}}) > + | --- > + | ... > + > +box.space.T:insert{date('1970-01-01')}\ > +box.space.T:insert{date('1970-01-02')}\ > +box.space.T:insert{date('1970-01-03')}\ > +box.space.T:insert{date('2000-01-01')} > + | --- > + | ... > + > +o = box.space.T:select{} > + | --- > + | ... > +assert(tostring(o[1][1]) == '1970-01-01T00:00Z') > + | --- > + | - true > + | ... > +assert(tostring(o[2][1]) == '1970-01-02T00:00Z') > + | --- > + | - true > + | ... > +assert(tostring(o[3][1]) == '1970-01-03T00:00Z') > + | --- > + | - true > + | ... > +assert(tostring(o[4][1]) == '2000-01-01T00:00Z') > + | --- > + | - true > + | ... > + > +for i = 1,16 do\ > + box.space.T:insert{date.now()}\ > +end > + | --- > + | ... > + > +a = box.space.T:select{} > + | --- > + | ... > +err = nil > + | --- > + | ... > +for i = 1, #a - 1 do\ > + if tostring(a[i][1]) >= tostring(a[i+1][1]) then\ Why do you compare string representation but not values itself? > + err = {a[i][1], a[i+1][1]}\ > + break\ > + end\ > +end > + | --- > + | ... > + > +err > + | --- > + | - null > + | ... > +box.space.T:drop() > + | --- > + | ... >
Thanks for your patch. This change should be squashed into the patch
where encoding was introduced.
On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * correct incorrect encoding of MP_EXT sizes for datetime
> messagepack values;
> * export necessary symbols for datetime messagepack size calculations
> so they will be available for Lua consumption.
>
> Part of #5941
> ---
> src/exports.h | 2 ++
> src/lib/core/datetime.c | 15 ++++++++++-----
> src/lua/msgpackffi.lua | 10 ++++++++++
> 3 files changed, 22 insertions(+), 5 deletions(-)
>
> diff --git a/src/exports.h b/src/exports.h
> index 586444b65..3a1e8854c 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -437,6 +437,7 @@ EXPORT(mp_encode_decimal)
> EXPORT(mp_encode_double)
> EXPORT(mp_encode_float)
> EXPORT(mp_encode_uuid)
> +EXPORT(mp_encode_datetime)
> EXPORT(mp_ext_hint)
> EXPORT(mp_format)
> EXPORT(mp_fprint)
> @@ -446,6 +447,7 @@ EXPORT(mp_fprint_recursion)
> EXPORT(mp_parser_hint)
> EXPORT(mp_sizeof_decimal)
> EXPORT(mp_sizeof_uuid)
> +EXPORT(mp_sizeof_datetime)
> EXPORT(mp_snprint)
> EXPORT(mp_snprint_ext)
> EXPORT(mp_snprint_ext_default)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index 26ba8a702..76b06eae5 100755
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -83,8 +83,8 @@ mp_decode_Xint(const char **data)
> return 0;
> }
>
> -uint32_t
> -mp_sizeof_datetime(const struct datetime_t *date)
> +static inline uint32_t
> +mp_sizeof_datetime_raw(const struct datetime_t *date)
> {
> uint32_t sz = mp_sizeof_Xint(date->secs);
>
> @@ -94,10 +94,15 @@ mp_sizeof_datetime(const struct datetime_t *date)
> sz += mp_sizeof_Xint(date->nsec);
> if (date->offset)
> sz += mp_sizeof_Xint(date->offset);
> -
> return sz;
> }
>
> +uint32_t
> +mp_sizeof_datetime(const struct datetime_t *date)
> +{
> + return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
> +}
> +
> struct datetime_t *
> datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
> {
> @@ -112,7 +117,7 @@ datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
> return date;
>
> svp = *data;
> - date->secs = mp_decode_Xint(data);
> + date->nsec = mp_decode_Xint(data);
> len -= *data - svp;
>
> if (len <= 0)
> @@ -153,7 +158,7 @@ datetime_pack(char *data, const struct datetime_t *date)
> char *
> mp_encode_datetime(char *data, const struct datetime_t *date)
> {
> - uint32_t len = mp_sizeof_datetime(date);
> + uint32_t len = mp_sizeof_datetime_raw(date);
>
> data = mp_encode_extl(data, MP_DATETIME, len);
>
> diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua
> index 271be857a..c47d77acb 100644
> --- a/src/lua/msgpackffi.lua
> +++ b/src/lua/msgpackffi.lua
> @@ -26,6 +26,10 @@ char *
> mp_encode_uuid(char *data, const struct tt_uuid *uuid);
> uint32_t
> mp_sizeof_uuid();
> +uint32_t
> +mp_sizeof_datetime(const struct t_datetime_tz *date);
> +char *
> +mp_encode_datetime(char *data, const struct t_datetime_tz *date);
> float
> mp_decode_float(const char **data);
> double
> @@ -144,6 +148,11 @@ local function encode_uuid(buf, uuid)
> builtin.mp_encode_uuid(p, uuid)
> end
>
> +local function encode_datetime(buf, date)
> + local p = buf:alloc(builtin.mp_sizeof_datetime(date))
> + builtin.mp_encode_datetime(p, date)
> +end
> +
> local function encode_int(buf, num)
> if num >= 0 then
> if num <= 0x7f then
> @@ -322,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float)
> on_encode(ffi.typeof('double'), encode_double)
> on_encode(ffi.typeof('decimal_t'), encode_decimal)
> on_encode(ffi.typeof('struct tt_uuid'), encode_uuid)
> +on_encode(ffi.typeof('struct datetime_t'), encode_datetime)
>
> --------------------------------------------------------------------------------
> -- Decoder
Thanks for your patch. See one comment below. Also this change should be squashed into the first patch. On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: > * introduced a set of calculated attributes to data object, e.g.: > - timestamp, seconds, microseconds, minute, or hours > > Part of #5941 > --- > src/lua/datetime.lua | 51 ++++++++++++++++++++++------------ > test/app-tap/datetime.test.lua | 17 +++++++++++- > 2 files changed, 50 insertions(+), 18 deletions(-) > > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua > index 9ec06d8d8..7a208cef9 100644 > --- a/src/lua/datetime.lua > +++ b/src/lua/datetime.lua > @@ -224,6 +224,36 @@ local function interval_serialize(self) > return { secs = self.secs, nsec = self.nsec } > end > > +local datetime_index = function(self, key) > + local attributes = { This change looks like pessimization because currently we need to recreate "attributes" table for each function call. I suggest to use `attributes` as `__index` but not such function. > + timestamp = function(self) > + return tonumber(self.secs) + self.nsec / 1e9 > + end, > + nanoseconds = function(self) > + return tonumber(self.secs * 1e9 + self.nsec) > + end, > + microseconds = function(self) > + return tonumber(self.secs * 1e6 + self.nsec / 1e3) > + end, > + milliseconds = function(self) > + return tonumber(self.secs * 1e3 + self.nsec / 1e6) > + end, > + seconds = function(self) > + return tonumber(self.secs) + self.nsec / 1e9 > + end, > + minutes = function(self) > + return (tonumber(self.secs) + self.nsec / 1e9) / 60 > + end, > + hours = function(self) > + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) > + end, > + days = function(self) > + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24 > + end, > + } > + return attributes[key] ~= nil and attributes[key](self) or nil > +end > + > local datetime_mt = { > -- __tostring = datetime_tostring, > __serialize = datetime_serialize, > @@ -232,23 +262,7 @@ local datetime_mt = { > __le = datetime_le, > __sub = datetime_sub, > __add = datetime_add, > - > - nanoseconds = function(self) > - return tonumber(self.secs*NANOS_PER_SEC + self.nsec) > - end, > - microseconds = function(self) > - return tonumber(self.secs*1e6 + self.nsec*1e3) > - end, > - seconds = function(self) > - return tonumber(self.secs + self.nsec*1e3) > - end, > - minutes = function(self) > - return tonumber((self._ticks/(1e6*60))%60) > - end, > - hours = function(self) > - return tonumber(self._ticks/(1e6*60*60)) > - end, > - > + __index = datetime_index, > } > > local interval_mt = { > @@ -257,6 +271,9 @@ local interval_mt = { > __eq = datetime_eq, > __lt = datetime_lt, > __le = datetime_le, > + __sub = datetime_sub, > + __add = datetime_add, > + __index = datetime_index, > } > > local function datetime_new_raw(secs, nsec, offset) > diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua > index 09c968858..407d89556 100755 > --- a/test/app-tap/datetime.test.lua > +++ b/test/app-tap/datetime.test.lua > @@ -4,7 +4,7 @@ local tap = require('tap') > local test = tap.test("errno") > local date = require('datetime') > > -test:plan(5) > +test:plan(6) > > test:test("Simple tests for parser", function(test) > test:plan(2) > @@ -188,4 +188,19 @@ test:test("Parse iso date - invalid strings", function(test) > end > end) > > +test:test("Parse tiny date into seconds and other parts", function(test) > + test:plan(9) > + local str = '19700101 00:00:30.528' > + local tiny = date(str) > + test:ok(tiny.secs == 30, ("secs of '%s'"):format(str)) > + test:ok(tiny.nsec == 528000000, ("nsec of '%s'"):format(str)) > + test:ok(tiny.nanoseconds == 30528000000, "nanoseconds") > + test:ok(tiny.microseconds == 30528000, "microseconds") > + test:ok(tiny.milliseconds == 30528, "milliseconds") > + test:ok(tiny.seconds == 30.528, "seconds") > + test:ok(tiny.timestamp == 30.528, "timestamp") > + test:ok(tiny.minutes == 0.5088, "minuts") > + test:ok(tiny.hours == 0.00848, "hours") > +end) > + > os.exit(test:check() and 0 or 1)
Thanks for your patch. I wrote several comments below. However it makes me think that such approach will work quite slow since all functions is implemented in Lua, all arithmetic is in Lua. On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: > * created few entry points (months(N), years(N), days(N), etc.) > for easier datetime arithmetic; > * additions/subtractions of years/months use `dt_add_years()` > and `dt_add_months()` from 3rd party c-dt library; > * also there are `:add{}` and `:sub{}` methods in datetime > object to add or substract more complex intervals; > * introduced `is_datetime()` and `is_interval()` helpers for checking > of validity of passed arguments; > * human-readable stringization implemented for interval objects. > > Note, that additions/subtractions completed for all _reasonable_ > combinations of values of date and interval types; > > Time + Interval => Time > Interval + Time => Time > Time - Time => Interval > Time - Interval => Time > Interval + Interval => Interval > Interval - Interval => Interval > > Part of #5941 > --- > src/exports.h | 3 + > src/lua/datetime.lua | 556 +++++++++++++++++++++++++++------ > test/app-tap/datetime.test.lua | 163 +++++++++- > 3 files changed, 631 insertions(+), 91 deletions(-) > > diff --git a/src/exports.h b/src/exports.h > index 3a1e8854c..6e7fe206d 100644 > --- a/src/exports.h > +++ b/src/exports.h > @@ -535,6 +535,9 @@ EXPORT(uuid_nil) > EXPORT(uuid_unpack) > EXPORT(datetime_unpack) > EXPORT(datetime_pack) > +EXPORT(dt_add_months) > +EXPORT(dt_add_years) > +EXPORT(dt_add_quarters) > EXPORT(dt_from_rdn) > EXPORT(dt_from_yd) > EXPORT(dt_from_ymd) > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua > index 7a208cef9..1466b923f 100644 > --- a/src/lua/datetime.lua > +++ b/src/lua/datetime.lua > @@ -37,6 +37,17 @@ ffi.cdef [[ > int dt_rdn (dt_t dt); > dt_dow_t dt_dow (dt_t dt); > > + // dt_arithmetic.h > + typedef enum { > + DT_EXCESS, > + DT_LIMIT, > + DT_SNAP > + } dt_adjust_t; > + > + dt_t dt_add_years (dt_t dt, int delta, dt_adjust_t adjust); > + dt_t dt_add_quarters (dt_t dt, int delta, dt_adjust_t adjust); > + dt_t dt_add_months (dt_t dt, int delta, dt_adjust_t adjust); > + > // dt_parse_iso.h > size_t dt_parse_iso_date (const char *str, size_t len, dt_t *dt); > > @@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL > > local datetime_t = ffi.typeof('struct datetime_t') > local interval_t = ffi.typeof('struct datetime_interval_t') > +ffi.cdef [[ > + struct t_interval_months { > + int m; > + }; > + > + struct t_interval_years { > + int y; > + }; > +]] > +local interval_months_t = ffi.typeof('struct t_interval_months') > +local interval_years_t = ffi.typeof('struct t_interval_years') > + > +local function is_interval(o) > + return ffi.istype(interval_t, o) or > + ffi.istype(interval_months_t, o) or > + ffi.istype(interval_years_t, o) > +end > + It will throw for non-cdata values: tarantool> ffi.istype(interval_t, o) --- - error: 'bad argument #1 to ''?'' (C type expected, got nil)' ... tarantool> ffi.istype(interval_t, 123) --- - error: 'bad argument #1 to ''?'' (C type expected, got nil)' ... > +local function is_datetime(o) > + return ffi.istype(o, datetime_t) > +end > + > > local function interval_new() > local interval = ffi.new(interval_t) > return interval > end > > -local function adjusted_secs(dt) > - return dt.secs - dt.offset * 60 > +local function check_number(n, message, lvl) > + if lvl == nil then > + lvl = 2 > + end > + if type(n) ~= 'number' then > + return error(('Usage: %s'):format(message), lvl) > + end > end > > -local function datetime_sub(lhs, rhs) > - local s1 = adjusted_secs(lhs) > - local s2 = adjusted_secs(rhs) > - local d = interval_new() > - d.secs = s2 - s1 > - d.nsec = rhs.nsec - lhs.nsec > - if d.nsec < 0 then > - d.secs = d.secs - 1 > - d.nsec = d.nsec + NANOS_PER_SEC > +local function check_date(o, message, lvl) > + if lvl == nil then > + lvl = 2 > + end > + if not is_datetime(o) then > + return error(('Usage: %s'):format(message), lvl) > end > - return d > end > > -local function datetime_add(lhs, rhs) > - local s1 = adjusted_secs(lhs) > - local s2 = adjusted_secs(rhs) > - local d = interval_new() > - d.secs = s2 + s1 > - d.nsec = rhs.nsec + lhs.nsec > - if d.nsec >= NANOS_PER_SEC then > - d.secs = d.secs + 1 > - d.nsec = d.nsec - NANOS_PER_SEC > +local function check_date_interval(o, message, lvl) > + if lvl == nil then > + lvl = 2 > + end > + if not (is_datetime(o) or is_interval(o)) then > + return error(('Usage: %s'):format(message), lvl) > end > - return d > end > > -local function datetime_eq(lhs, rhs) > - -- we usually don't need to check nullness > - -- but older tarantool console will call us checking for equality to nil > - if rhs == nil then > - return false > +local function check_interval(o, message, lvl) > + if lvl == nil then > + lvl = 2 > + end > + if not is_interval(o) then > + return error(('Usage: %s'):format(message), lvl) > end > +end > + > +local function check_str(o, message, lvl) > + if lvl == nil then > + lvl = 2 > + end > + if not type(o) == 'string' then > + return error(('Usage: %s'):format(message), lvl) > + end > +end > + > +local function interval_years_new(y) > + check_number(y, "years(number)") > + local o = ffi.new(interval_years_t) > + o.y = y > + return o > +end > + > +local function interval_months_new(m) > + check_number(m, "months(number)") > + local o = ffi.new(interval_months_t) > + o.m = m > + return o > +end > + > +local function interval_weeks_new(w) > + check_number(w, "weeks(number)") > + local o = ffi.new(interval_t) > + o.secs = w * SECS_PER_DAY * 7 > + return o > +end > + > +local function interval_days_new(d) > + check_number(d, "days(number)") > + local o = ffi.new(interval_t) > + o.secs = d * SECS_PER_DAY > + return o > +end > + > +local function interval_hours_new(h) > + check_number(h, "hours(number)") > + local o = ffi.new(interval_t) > + o.secs = h * 60 * 60 > + return o > +end > + > +local function interval_minutes_new(m) > + check_number(m, "minutes(number)") > + local o = ffi.new(interval_t) > + o.secs = m * 60 > + return o > +end > + > +local function interval_seconds_new(s) > + check_number(s, "seconds(number)") > + local o = ffi.new(interval_t) > + o.nsec = s % 1 * 1e9 > + o.secs = s - (s % 1) > + return o > +end > + > +local function datetime_eq(lhs, rhs) > + check_date_interval(lhs, "datetime:__eq(date or interval)") > + check_date_interval(rhs, "datetime:__eq(date or interval)") > return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) > end > > > local function datetime_lt(lhs, rhs) > + check_date_interval(lhs, "datetime:__lt(date or interval)") > + check_date_interval(rhs, "datetime:__lt(date or interval)") > return (lhs.secs < rhs.secs) or > (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) > end > > local function datetime_le(lhs, rhs) > + check_date_interval(lhs, "datetime:__le(date or interval)") > + check_date_interval(rhs, "datetime:__le(date or interval)") > return (lhs.secs <= rhs.secs) or > (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) > end > @@ -224,19 +323,123 @@ local function interval_serialize(self) > return { secs = self.secs, nsec = self.nsec } > end > > +local function local_rd(o) > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET > +end > + > +local function local_dt(o) > + return cdt.dt_from_rdn(local_rd(o)) > +end > + > +local function _normalize_nsec(secs, nsec) > + if nsec < 0 then > + secs = secs - 1 > + nsec = nsec + NANOS_PER_SEC > + elseif nsec >= NANOS_PER_SEC then > + secs = secs + 1 > + nsec = nsec - NANOS_PER_SEC > + end > + return secs, nsec > +end > + > +-- addition or subtraction from date/time of a given interval > +-- described via table direction should be +1 or -1 > +local function interval_increment(self, o, direction) > + assert(direction == -1 or direction == 1) > + check_date(self, "interval_increment(date, object, -+1)") > + assert(type(o) == 'table') > + > + local ym_updated = false > + local dhms_updated = false > + > + local dt = local_dt(self) > + local secs, nsec > + secs, nsec = self.secs, self.nsec > + > + for key, value in pairs(o) do > + local handlers = { The same as in one previous patch. It's too expensive to recreate table and functions for each simple action and for each iteration loop. > + years = function(v) > + assert(v > 0 and v < 10000) > + dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT) > + ym_updated = true > + end, > + > + months = function(v) > + assert(v > 0 and v < 13 ) > + dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT) > + ym_updated = true > + end, > + > + weeks = function(v) > + assert(v > 0 and v < 32) > + secs = secs + direction * 7 * v * SECS_PER_DAY > + dhms_updated = true > + end, > + > + days = function(v) > + assert(v > 0 and v < 32) > + secs = secs + direction * v * SECS_PER_DAY > + dhms_updated = true > + end, > + > + hours = function(v) > + assert(v >= 0 and v < 24) > + secs = secs + direction * 60 * 60 * v > + dhms_updated = true > + end, > + > + minutes = function(v) > + assert(v >= 0 and v < 60) > + secs = secs + direction * 60 * v > + end, > + > + seconds = function(v) > + assert(v >= 0 and v < 61) > + local s, frac > + frac = v % 1 > + if frac > 0 then > + s = v - (v % 1) > + else > + s = v > + end > + secs = secs + direction * s > + nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds > + dhms_updated = true > + end, > + } > + handlers[key](value) > + end > + > + secs, nsec = _normalize_nsec(secs, nsec) > + > + -- .days, .hours, .minutes, .seconds > + if dhms_updated then > + self.secs = secs > + self.nsec = nsec > + end > + > + -- .years, .months updated > + if ym_updated then > + self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY + > + secs % SECS_PER_DAY > + end > + > + return self > +end > + > local datetime_index = function(self, key) > local attributes = { > timestamp = function(self) > return tonumber(self.secs) + self.nsec / 1e9 > end, > nanoseconds = function(self) > - return tonumber(self.secs * 1e9 + self.nsec) > + return self.secs * 1e9 + self.nsec > end, > microseconds = function(self) > - return tonumber(self.secs * 1e6 + self.nsec / 1e3) > + return self.secs * 1e6 + self.nsec / 1e3 > end, > milliseconds = function(self) > - return tonumber(self.secs * 1e3 + self.nsec / 1e6) > + return self.secs * 1e3 + self.nsec / 1e6 > end, > seconds = function(self) > return tonumber(self.secs) + self.nsec / 1e9 > @@ -250,32 +453,20 @@ local datetime_index = function(self, key) > days = function(self) > return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24 > end, > + add = function(self) > + return function(self, o) > + return interval_increment(self, o, 1) > + end > + end, > + sub = function(self) > + return function(self, o) > + return interval_increment(self, o, -1) > + end > + end, > } > return attributes[key] ~= nil and attributes[key](self) or nil > end > > -local datetime_mt = { > - -- __tostring = datetime_tostring, > - __serialize = datetime_serialize, > - __eq = datetime_eq, > - __lt = datetime_lt, > - __le = datetime_le, > - __sub = datetime_sub, > - __add = datetime_add, > - __index = datetime_index, > -} > - > -local interval_mt = { > - -- __tostring = interval_tostring, > - __serialize = interval_serialize, > - __eq = datetime_eq, > - __lt = datetime_lt, > - __le = datetime_le, > - __sub = datetime_sub, > - __add = datetime_add, > - __index = datetime_index, > -} > - > local function datetime_new_raw(secs, nsec, offset) > local dt_obj = ffi.new(datetime_t) > dt_obj.secs = secs > @@ -284,14 +475,6 @@ local function datetime_new_raw(secs, nsec, offset) > return dt_obj > end > > -local function local_rd(o) > - return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET > -end > - > -local function local_dt(o) > - return cdt.dt_from_rdn(local_rd(o)) > -end > - > local function mk_timestamp(dt, sp, fp, offset) > local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0 > local spV = sp ~= nil and sp or 0 > @@ -367,11 +550,12 @@ local function datetime_new(o) > second = function(v) > assert(v >= 0 and v < 61) > frac = v % 1 > - if frac then > + if frac > 0 then > s = v - (v % 1) > else > s = v > end > + frac = frac * 1e9 -- convert fraction to nanoseconds > hms = true > end, > > @@ -402,6 +586,153 @@ local function datetime_new(o) > return mk_timestamp(dt, secs, frac, offset) > end > > +local function datetime_tostring(o) > + if ffi.typeof(o) == datetime_t then > + local sz = 48 > + local buff = ffi.new('char[?]', sz) > + local len = native.datetime_to_string(o, buff, sz) > + assert(len < sz) > + return ffi.string(buff) > + elseif ffi.typeof(o) == interval_years_t then > + return ('%+d years'):format(o.y) > + elseif ffi.typeof(o) == interval_months_t then > + return ('%+d months'):format(o.m) > + elseif ffi.typeof(o) == interval_t then > + local ts = o.timestamp > + local sign = '+' > + > + if ts < 0 then > + ts = -ts > + sign = '-' > + end > + > + if ts < 60 then > + return ('%s%s secs'):format(sign, ts) > + elseif ts < 60 * 60 then > + return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60) > + elseif ts < 24 * 60 * 60 then > + return ('%+d hours, %d minutes, %s seconds'):format( > + o.hours, o.minutes % 60, ts % 60) > + else > + return ('%+d days, %d hours, %d minutes, %s seconds'):format( > + o.days, o.hours % 24, o.minutes % 60, ts % 60) > + end > + end > +end > + > +local function date_first(lhs, rhs) > + if is_datetime(lhs) then > + return lhs, rhs > + else > + return rhs, lhs > + end > +end > + > +--[[ > +Matrix of subtraction operands eligibility and their result type > + > +| | datetime | interval | interval_months | interval_years | > ++-----------------+-----------+----------+-----------------+----------------+ > +| datetime | interval | datetime | datetime | datetime | > +| interval | | interval | | | > +| interval_months | | | interval_months | | > +| interval_years | | | | interval_years | > +]] > +local function datetime_sub(lhs, rhs) > + check_date_interval(lhs, "datetime:__sub(date or interval)") > + local d, s = lhs, rhs > + local left_t = ffi.typeof(d) > + local right_t = ffi.typeof(s) > + local o > + > + if left_t == datetime_t then > + -- 1. left is date, right is date or generic interval > + if (right_t == datetime_t or right_t == interval_t) then > + o = right_t == datetime_t and interval_new() or datetime_new() > + o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs, > + lhs.nsec - rhs.nsec) > + return o > + -- 2. left is date, right is interval in months > + elseif right_t == interval_months_t then > + local dt = cdt.dt_add_months(local_dt(lhs), -rhs.m, cdt.DT_LIMIT) > + return mk_timestamp(dt, lhs.secs % SECS_PER_DAY, > + lhs.nsec, lhs.offset) > + > + -- 3. left is date, right is interval in years > + elseif right_t == interval_years_t then > + local dt = cdt.dt_add_years(local_dt(lhs), -rhs.y, cdt.DT_LIMIT) > + return mk_timestamp(dt, lhs.secs % SECS_PER_DAY, > + lhs.nsec, lhs.offset) > + else > + error("datetime:__sub(date or interval) - incompatible type of arguments", 2) > + end > + -- 4. both left and right are generic intervals > + elseif left_t == interval_t and right_t == interval_t then > + o = interval_new() > + o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs, > + lhs.nsec - rhs.nsec) > + return o > + -- 5. both left and right are intervals in months > + elseif left_t == interval_months_t and right_t == interval_months_t then > + return interval_months_new(lhs.m - rhs.m) > + -- 5. both left and right are intervals in years > + elseif left_t == interval_years_t and right_t == interval_years_t then > + return interval_years_new(lhs.y - rhs.y) > + else > + error("datetime:__sub(date or interval) - incompatible type of arguments", 2) > + end > +end > + > +--[[ > +Matrix of addition operands eligibility and their result type > + > +| | datetime | interval | interval_months | interval_years | > ++-----------------+-----------+----------+-----------------+----------------+ > +| datetime | datetime | datetime | datetime | datetime | > +| interval | datetime | interval | | | > +| interval_months | datetime | | interval_months | | > +| interval_years | datetime | | | interval_years | > +]] > +local function datetime_add(lhs, rhs) > + local d, s = date_first(lhs, rhs) > + > + check_date_interval(d, "datetime:__add(interval)") > + check_interval(s, "datetime:__add(interval)") tarantool> return require('datetime').now() + 1 --- - error: '[string "return require(''datetime'').now() + 1"]:1: Usage: datetime:__add(interval)' ... Looks a bit confusing. User doesn't know about metamethods. > + local left_t = ffi.typeof(d) > + local right_t = ffi.typeof(s) > + local o > +
Thanks for your patch.
The comment is primarly the same as in previous patch.
Why do you use such slow approach and creates redundant tables and
function (temporary metatable in fact)
to use it only once? Please fix it.
Unixtime and timestamp is great but they loss precision. I think it
should be possible
go get timestamp with nanoseconds precision since datetime has
nanoseconds precision.
Also still it's hard how to convert timestamp to datetime value back.
On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * implemented proper range checks for date attributes values;
>
> * created `.unixtime` attribute, which is alias to `.secs`,
> with corresponding setter/getter;
>
> * similarly to `unixtime`, created virtual `timestamp` attribute
> setter. Which is a convenient way to simultaneously assign
> unixtime (seconds since epoch) and nanoseconds
>
> Part of #5941
> ---
> src/lua/datetime.lua | 90 ++++++++++++++++++++++++++++++--------------
> 1 file changed, 61 insertions(+), 29 deletions(-)
>
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 1466b923f..cc2ae119b 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -342,12 +342,33 @@ local function _normalize_nsec(secs, nsec)
> return secs, nsec
> end
>
> +local function seconds_fraction(v)
> + local seconds, fraction
> + fraction = v % 1
> + if fraction > 0 then
> + seconds = v - (v % 1)
> + else
> + seconds = v
> + end
> + return seconds, fraction
> +end
> +
> +local function check_range(v, range, txt)
> + assert(#range == 2)
> + if not (v >= range[1] and v <= range[2]) then
> + error(('value %d of %s is out of allowed range [%d, %d]'):
> + format(v, txt, range[1], range[2]))
> + end
> +end
> +
> -- addition or subtraction from date/time of a given interval
> -- described via table direction should be +1 or -1
> local function interval_increment(self, o, direction)
> assert(direction == -1 or direction == 1)
> - check_date(self, "interval_increment(date, object, -+1)")
> - assert(type(o) == 'table')
> + check_date(self, "interval_increment(date, object, direction)")
> + if type(o) ~= 'table' then
> + error('interval_increment(date, object, direction) - object expected', 2)
> + end
>
> local ym_updated = false
> local dhms_updated = false
> @@ -359,49 +380,43 @@ local function interval_increment(self, o, direction)
> for key, value in pairs(o) do
> local handlers = {
> years = function(v)
> - assert(v > 0 and v < 10000)
> + check_range(v, {0, 9999}, key)
> dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
> ym_updated = true
> end,
>
> months = function(v)
> - assert(v > 0 and v < 13 )
> + check_range(v, {0, 12}, key)
> dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
> ym_updated = true
> end,
>
> weeks = function(v)
> - assert(v > 0 and v < 32)
> + check_range(v, {0, 52}, key)
> secs = secs + direction * 7 * v * SECS_PER_DAY
> dhms_updated = true
> end,
>
> days = function(v)
> - assert(v > 0 and v < 32)
> + check_range(v, {0, 31}, key)
> secs = secs + direction * v * SECS_PER_DAY
> dhms_updated = true
> end,
>
> hours = function(v)
> - assert(v >= 0 and v < 24)
> + check_range(v, {0, 23}, key)
> secs = secs + direction * 60 * 60 * v
> dhms_updated = true
> end,
>
> minutes = function(v)
> - assert(v >= 0 and v < 60)
> + check_range(v, {0, 59}, key)
> secs = secs + direction * 60 * v
> end,
>
> seconds = function(v)
> - assert(v >= 0 and v < 61)
> - local s, frac
> - frac = v % 1
> - if frac > 0 then
> - s = v - (v % 1)
> - else
> - s = v
> - end
> + check_range(v, {0, 60}, key)
> + local s, frac = seconds_fraction(v)
> secs = secs + direction * s
> nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
> dhms_updated = true
> @@ -429,6 +444,9 @@ end
>
> local datetime_index = function(self, key)
> local attributes = {
> + unixtime = function(self)
> + return self.secs
> + end,
> timestamp = function(self)
> return tonumber(self.secs) + self.nsec / 1e9
> end,
> @@ -467,6 +485,24 @@ local datetime_index = function(self, key)
> return attributes[key] ~= nil and attributes[key](self) or nil
> end
>
> +local function datetime_newindex(self, key, value)
> + local attributes = {
> + unixtime = function(self, value)
> + self.secs = value
> + self.nsec, self.offset = 0, 0
> + end,
> + timestamp = function(self, value)
> + local secs, frac = seconds_fraction(value)
> + self.secs = secs
> + self.nsec = frac * 1e9
> + self.offset = 0
> + end,
> + }
> + if attributes[key] ~= nil then
> + attributes[key](self, value)
> + end
> +end
> +
> local function datetime_new_raw(secs, nsec, offset)
> local dt_obj = ffi.new(datetime_t)
> dt_obj.secs = secs
> @@ -518,50 +554,45 @@ local function datetime_new(o)
> end,
>
> year = function(v)
> - assert(v > 0 and v < 10000)
> + check_range(v, {1, 9999}, key)
> y = v
> ymd = true
> end,
>
> month = function(v)
> - assert(v > 0 and v < 13 )
> + check_range(v, {1, 12}, key)
> M = v
> ymd = true
> end,
>
> day = function(v)
> - assert(v > 0 and v < 32)
> + check_range(v, {1, 31}, key)
> d = v
> ymd = true
> end,
>
> hour = function(v)
> - assert(v >= 0 and v < 24)
> + check_range(v, {0, 23}, key)
> h = v
> hms = true
> end,
>
> minute = function(v)
> - assert(v >= 0 and v < 60)
> + check_range(v, {0, 59}, key)
> m = v
> hms = true
> end,
>
> second = function(v)
> - assert(v >= 0 and v < 61)
> - frac = v % 1
> - if frac > 0 then
> - s = v - (v % 1)
> - else
> - s = v
> - end
> + check_range(v, {0, 60}, key)
> + s, frac = seconds_fraction(v)
> frac = frac * 1e9 -- convert fraction to nanoseconds
> hms = true
> end,
>
> -- tz offset in minutes
> tz = function(v)
> - assert(v >= 0 and v <= 720)
> + check_range(v, {0, 720}, key)
> offset = v
> end
> }
> @@ -918,6 +949,7 @@ local datetime_mt = {
> __sub = datetime_sub,
> __add = datetime_add,
> __index = datetime_index,
> + __newindex = datetime_newindex,
> add = function(self, o)
> self = interval_increment(self, o, 1)
> return self
Thanks for the patch! This is a bad question to ask now, but why did you choose a dead library from 2015 with just 25 stars, without any tests, and without any README as a basic block for our datetime types? Is this the correct link I am using - https://github.com/chansen/c-dt? See 31 comment below. On 28.07.2021 12:34, Timur Safin wrote: > * created a new Tarantool built-in module `datetime`; > * register cdef types for this module; > * export some `dt_*` functions from `c-dt` library; > * lua implementationis of `asctime` and `strftime`; 1. implementationis -> implementations > * datetime parsing unit tests, with and withput timezones; 2. withput -> without > diff --git a/src/exports.h b/src/exports.h > index 5bb3e6a2b..db40c03a4 100644 > --- a/src/exports.h > +++ b/src/exports.h > @@ -531,3 +531,24 @@ EXPORT(uri_format) > EXPORT(uri_parse) > EXPORT(uuid_nil) > EXPORT(uuid_unpack) > +EXPORT(dt_from_rdn) > +EXPORT(dt_from_yd) > +EXPORT(dt_from_ymd) > +EXPORT(dt_from_yqd) > +EXPORT(dt_from_ywd) > +EXPORT(dt_to_yd) > +EXPORT(dt_to_ymd) > +EXPORT(dt_to_yqd) > +EXPORT(dt_to_ywd) > +EXPORT(dt_rdn) > +EXPORT(dt_dow) > +EXPORT(dt_parse_iso_date) > +EXPORT(dt_parse_iso_time) > +EXPORT(dt_parse_iso_time_basic) > +EXPORT(dt_parse_iso_time_extended) > +EXPORT(dt_parse_iso_zone) > +EXPORT(dt_parse_iso_zone_basic) > +EXPORT(dt_parse_iso_zone_extended) > +EXPORT(dt_parse_iso_zone_lenient) > +EXPORT(dt_from_struct_tm) > +EXPORT(dt_to_struct_tm) 3. Please, read the comment in the beginning of the file. The first sentence would be: Keep the symbols sorted by name for search and addition simplicity, to avoid duplicates. > diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h > new file mode 100644 > index 000000000..403bf1c64 > --- /dev/null > +++ b/src/lib/core/datetime.h > @@ -0,0 +1,61 @@ <...> > +#include <c-dt/dt_core.h> 4. Please, use "" instead of <> for non-system headers. > +#include <stdint.h> > +#include <stdbool.h> > + > +#if defined(__cplusplus) > +extern "C" { > +#endif /* defined(__cplusplus) */ > + > +/** > + * datetime structure consisting of: 5. Well, this comment isn't really helpful. Also we do not use _t suffix ever except for typedefs. Please, drop both these things. The same for datetime_interval_t. > + */ > +struct datetime_t { > + int64_t secs; ///< seconds since epoch > + int32_t nsec; ///< nanoseconds if any > + int32_t offset; ///< offset in minutes from GMT 6. We never use // nor /// nor ///< nor we write comments on the same line as the code (except for old legacy code we inherited from sqlite and some super old tarantool code. The same for datetime_interval_t. > +}; > + > +/** > + * Date/time delta structure > + */ > +struct datetime_interval_t { > + int64_t secs; ///< relative seconds delta > + int32_t nsec; ///< nanoseconds delta > +}; > + 7. Why do you need this file? It is not included anywhere. And you don't need to define the structs in C if you are using them in Lua only. You can just define them in Lua using ffi.cdef like it is done in some other places. > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua > new file mode 100644 > index 000000000..0996ca5a2 > --- /dev/null > +++ b/src/lua/datetime.lua > @@ -0,0 +1,581 @@ > +local ffi = require('ffi') > +local cdt = ffi.C > + > +ffi.cdef [[ > + > + typedef int dt_t; > + > + // dt_core.h > + typedef enum { > + DT_MON = 1, > + DT_MONDAY = 1, > + DT_TUE = 2, > + DT_TUESDAY = 2, > + DT_WED = 3, > + DT_WEDNESDAY = 3, > + DT_THU = 4, > + DT_THURSDAY = 4, > + DT_FRI = 5, > + DT_FRIDAY = 5, > + DT_SAT = 6, > + DT_SATURDAY = 6, > + DT_SUN = 7, > + DT_SUNDAY = 7, > + } dt_dow_t; > + > + dt_t dt_from_rdn (int n); > + dt_t dt_from_yd (int y, int d); > + dt_t dt_from_ymd (int y, int m, int d); > + dt_t dt_from_yqd (int y, int q, int d); > + dt_t dt_from_ywd (int y, int w, int d); > + > + void dt_to_yd (dt_t dt, int *y, int *d); > + void dt_to_ymd (dt_t dt, int *y, int *m, int *d); > + void dt_to_yqd (dt_t dt, int *y, int *q, int *d); > + void dt_to_ywd (dt_t dt, int *y, int *w, int *d); > + > + int dt_rdn (dt_t dt); > + dt_dow_t dt_dow (dt_t dt); > + > + // dt_parse_iso.h > + size_t dt_parse_iso_date (const char *str, size_t len, dt_t *dt); > + > + size_t dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec); > + size_t dt_parse_iso_time_basic (const char *str, size_t len, int *sod, int *nsec); > + size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec); > + > + size_t dt_parse_iso_zone (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_basic (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset); > + size_t dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset); > + > + // dt_tm.h > + dt_t dt_from_struct_tm (const struct tm *tm); > + void dt_to_struct_tm (dt_t dt, struct tm *tm); > + > + // <asm-generic/posix_types.h> > + typedef long __kernel_long_t; > + typedef unsigned long __kernel_ulong_t; > + // /usr/include/x86_64-linux-gnu/bits/types/time_t.h > + typedef long time_t; > + > + > + // <time.h> > + typedef __kernel_long_t __kernel_time_t; > + typedef __kernel_long_t __kernel_suseconds_t; > + > + struct timespec { > + __kernel_time_t tv_sec; /* seconds */ > + long tv_nsec; /* nanoseconds */ > + }; > + > + struct timeval { > + __kernel_time_t tv_sec; /* seconds */ > + __kernel_suseconds_t tv_usec; /* microseconds */ > + }; > + > + struct timezone { > + int tz_minuteswest; /* minutes west of Greenwich */ > + int tz_dsttime; /* type of dst correction */ > + }; > + > + // /usr/include/x86_64-linux-gnu/sys/time.h > + typedef struct timezone * __timezone_ptr_t; > + > + /* Get the current time of day and timezone information, > + putting it into *TV and *TZ. If TZ is NULL, *TZ is not filled. > + Returns 0 on success, -1 on errors. > + > + NOTE: This form of timezone information is obsolete. > + Use the functions and variables declared in <time.h> instead. */ > + int gettimeofday (struct timeval *__tv, struct timezone * __tz); > + > + // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h > + /* ISO C `broken-down time' structure. */ > + struct tm > + { > + int tm_sec; /* Seconds. [0-60] (1 leap second) */ > + int tm_min; /* Minutes. [0-59] */ > + int tm_hour; /* Hours. [0-23] */ > + int tm_mday; /* Day. [1-31] */ > + int tm_mon; /* Month. [0-11] */ > + int tm_year; /* Year - 1900. */ > + int tm_wday; /* Day of week. [0-6] */ > + int tm_yday; /* Days in year.[0-365] */ > + int tm_isdst; /* DST. [-1/0/1]*/ > + > + long int tm_gmtoff; /* Seconds east of UTC. */ > + const char *tm_zone;/* Timezone abbreviation. */ > + }; > + > + // <time.h> > + /* Return the current time and put it in *TIMER if TIMER is not NULL. */ > + time_t time (time_t *__timer); > + > + /* Format TP into S according to FORMAT. > + Write no more than MAXSIZE characters and return the number > + of characters written, or 0 if it would exceed MAXSIZE. */ > + size_t strftime (char * __s, size_t __maxsize, const char * __format, > + const struct tm * __tp); > + > + /* Parse S according to FORMAT and store binary time information in TP. > + The return value is a pointer to the first unparsed character in S. */ > + char *strptime (const char * __s, const char * __fmt, struct tm *__tp); > + > + /* Return the `struct tm' representation of *TIMER in UTC, > + using *TP to store the result. */ > + struct tm *gmtime_r (const time_t * __timer, struct tm * __tp); > + > + /* Return the `struct tm' representation of *TIMER in local time, > + using *TP to store the result. */ > + struct tm *localtime_r (const time_t * __timer, struct tm * __tp); > + > + /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n" > + that is the representation of TP in this format. */ > + char *asctime (const struct tm *__tp); > + > + /* Equivalent to `asctime (localtime (timer))'. */ > + char *ctime (const time_t *__timer); > + 8. The amount of platform-specific stuff you got here really bothers me. For instance, struct tm from /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h. Is it said anywhere that the struct has only these members and only in this order? The standard says "includes at least the following members". Not only them and no specific order. I would probably go with Lua C implementation if I would start this module. Too strong dependecy on C. Also I wouldn't be so sure FFI is faster than Lua C: https://github.com/tarantool/tarantool/issues/5896. > +]] > + > +local native = ffi.C 9. You already have cdt in the beginning of the file. > + > +local SECS_PER_DAY = 86400 > +local NANOS_PER_SEC = 1000000000LL > + > +-- c-dt/dt_config.h > + > +-- Unix, January 1, 1970, Thursday > +local DT_EPOCH_1970_OFFSET = 719163LL 10. I remember there was a good question somewhere - how are these datetimes supposed to store dates before 1970? For example, a database of historical events or documents. Can this library store dates before Christ? > + > + > +local datetime_t = ffi.typeof('struct datetime_t') > +local interval_t = ffi.typeof('struct datetime_interval_t') > + > +local function interval_new() > + local interval = ffi.new(interval_t) > + return interval > +end > + > +local function adjusted_secs(dt) > + return dt.secs - dt.offset * 60 > +end > + > +local function datetime_sub(lhs, rhs) > + local s1 = adjusted_secs(lhs) > + local s2 = adjusted_secs(rhs) 11. Did you think about storing the time in a unified format with offset already applied and add the offset only when you print the time or convert to something? It seems these `dt.offset * 60` might be expensive on each arith operation done again and again. > + local d = interval_new() > + d.secs = s2 - s1 > + d.nsec = rhs.nsec - lhs.nsec > + if d.nsec < 0 then 12. AFAIK, in Lua index operator '.' is costly. You might want to cache its values like this: local secs = s2 - s1 local nsec = rhs.nsec - lhs.nsec if nsec < 0 then d.secs = secs - 1 d.nsec = nsec + NANOS_PER_SEC else d.secs = secs d.nsec = nsec end The same in the other places where index operation is repeated more than once: datetime_lt, datetime_le, etc. > + d.secs = d.secs - 1 > + d.nsec = d.nsec + NANOS_PER_SEC > + end > + return d > +end > + > +local function datetime_add(lhs, rhs) > + local s1 = adjusted_secs(lhs) > + local s2 = adjusted_secs(rhs) > + local d = interval_new() > + d.secs = s2 + s1 > + d.nsec = rhs.nsec + lhs.nsec > + if d.nsec >= NANOS_PER_SEC then > + d.secs = d.secs + 1 > + d.nsec = d.nsec - NANOS_PER_SEC > + end > + return d > +end > + > +local function datetime_eq(lhs, rhs) > + -- we usually don't need to check nullness > + -- but older tarantool console will call us checking for equality to nil > + if rhs == nil then > + return false > + end > + return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) 13. You do not need () around the comparison operators. The same in all places below. Also you could make it shorter: return rhs ~= nil and lhs.secs == rhs.secs and lhs.nsec == rhs.nsec 14. Why don't you take 'offset' into account? > +end > + > + > +local function datetime_lt(lhs, rhs) > + return (lhs.secs < rhs.secs) or > + (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) > +end > + > +local function datetime_le(lhs, rhs) > + return (lhs.secs <= rhs.secs) or > + (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) > +end > + > +local function datetime_serialize(self) > + -- Allow YAML, MsgPack and JSON to dump objects with sockets > + return { secs = self.secs, nsec = self.nsec, tz = self.offset } 15. Why is 'offset' serialized as 'tz' and not just 'offset'? > +end > + > +local function interval_serialize(self) > + -- Allow YAML and JSON to dump objects with sockets > + return { secs = self.secs, nsec = self.nsec } > +end > + > +local datetime_mt = { > + -- __tostring = datetime_tostring, 16. Why is it commented out? Is it even tested then? > + __serialize = datetime_serialize, > + __eq = datetime_eq, > + __lt = datetime_lt, > + __le = datetime_le, > + __sub = datetime_sub, > + __add = datetime_add, > + > + nanoseconds = function(self) > + return tonumber(self.secs*NANOS_PER_SEC + self.nsec) > + end, 17. Please, separate function definitions with an empty line between them. > + microseconds = function(self) > + return tonumber(self.secs*1e6 + self.nsec*1e3) 18. Please, add whitespaces around binary operators. 19. Why do you do tonumber() for all results? You are loosing precision for big values. The same in all the places where you use tonumber(). > + end, > + seconds = function(self) > + return tonumber(self.secs + self.nsec*1e3) > + end, > + minutes = function(self) > + return tonumber((self._ticks/(1e6*60))%60) > + end, > + hours = function(self) > + return tonumber(self._ticks/(1e6*60*60)) 20. I don't see _ticks defined anywhere in our source code in any of the files. What is it? Is it tested? > + end, > + 21. Extra empty line. > +} > + > +local interval_mt = { > + -- __tostring = interval_tostring, 22. Ditto. Why is it commented out and not tested? > + __serialize = interval_serialize, > + __eq = datetime_eq, > + __lt = datetime_lt, > + __le = datetime_le, > +} > + > +local function datetime_new_raw(secs, nsec, offset) > + local dt_obj = ffi.new(datetime_t) > + dt_obj.secs = secs > + dt_obj.nsec = nsec > + dt_obj.offset = offset > + return dt_obj > +end > + > +local function local_rd(o) 23. What is 'rd' and 'dt'? > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET > +end > + > +local function local_dt(o) > + return cdt.dt_from_rdn(local_rd(o)) > +end > + > +local function mk_timestamp(dt, sp, fp, offset) > + local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0 > + local spV = sp ~= nil and sp or 0 > + local fpV = fp ~= nil and fp or 0 > + local ofsV = offset ~= nil and offset or 0 > + return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV) > +end > + > +-- create @datetime_t given object @o fields > +local function datetime_new(o) 24. Why such a wierd name 'o'? It looks like 0, super confusing. > + if o == nil then > + return datetime_new_raw(0, 0, 0) > + end > + local secs = 0 > + local nsec = 0 > + local offset = 0 > + local easy_way = false > + local y, M, d, ymd > + y, M, d, ymd = 0, 0, 0, false > + > + local h, m, s, frac, hms > + h, m, s, frac, hms = 0, 0, 0, 0, false 25. Please, try to define the variables on their own lines. Like you did a few lines above. Otherwise it is practically unreadable. > + > + local dt = 0 > + > + for key, value in pairs(o) do > + local handlers = { > + secs = function(v) > + secs = v > + easy_way = true 26. I don't understand what 'easy way' means. Please, elaborate what you are trying to achieve. > + end, 27. The usage of closures here might render all your FFI efforts useless, killing the performance. Please, try to define all methods of all objects only once in the root namespace of the file. Closure usage might be justified only for rarely created long living heavy objects like netbox. > + > + nsec = function(v) > + nsec = v > + easy_way = true > + end, > + > + offset = function (v) > + offset = v > + easy_way = true > + end, > + > + year = function(v) > + assert(v > 0 and v < 10000) > + y = v > + ymd = true > + end, > + > + month = function(v) > + assert(v > 0 and v < 12 ) > + M = v > + ymd = true > + end, > + > + day = function(v) > + assert(v > 0 and v < 32) > + d = v > + ymd = true > + end, > + > + hour = function(v) > + assert(v >= 0 and v < 24) > + h = v > + hms = true > + end, > + > + minute = function(v) > + assert(v >= 0 and v < 60) > + m = v > + hms = true > + end, > + > + second = function(v) > + assert(v >= 0 and v < 61) > + frac = v % 1 > + if frac then > + s = v - (v % 1) > + else > + s = v > + end > + hms = true > + end, > + > + -- tz offset in minutes > + tz = function(v) > + assert(v >= 0 and v <= 720) > + offset = v > + end > + } > + handlers[key](value) > + end > + > + -- .sec, .nsec, .offset > + if easy_way then > + return datetime_new_raw(secs, nsec, offset) > + end > + > + -- .year, .month, .day > + if ymd then > + dt = dt + cdt.dt_from_ymd(y, M, d) > + end > + > + -- .hour, .minute, .second > + if hms then > + secs = h * 3600 + m * 60 + s > + end > + > + return mk_timestamp(dt, secs, frac, offset) > +end > + > + > +-- simple parse functions: > +-- parse_date/parse_time/parse_zone 28. I wouldn't be so sure they are 'simple' and I can read their names below anyway. This makes this comment impractical. > + > +--[[ > + Basic Extended > + 20121224 2012-12-24 Calendar date (ISO 8601) > + 2012359 2012-359 Ordinal date (ISO 8601) > + 2012W521 2012-W52-1 Week date (ISO 8601) > + 2012Q485 2012-Q4-85 Quarter date > +]] > + > +local function parse_date(str) > + local dt = ffi.new('dt_t[1]') > + local rc = cdt.dt_parse_iso_date(str, #str, dt) > + assert(rc > 0) 29. Er ... And what if I pass a string in a wrong format? Users should not get assertions. tarantool> datetime.parse_date('121r1 r1r 13 13 13') --- - error: 'builtin/datetime.lua:397: assertion failed!' ... This shall not ever happen. Would you mind implementing proper error handling. The same in all the places where the errors are ignored. > + return mk_timestamp(dt[0]) > +end > + > +--[[ > + Basic Extended > + T12 N/A > + T1230 T12:30 > + T123045 T12:30:45 > + T123045.123456789 T12:30:45.123456789 > + T123045,123456789 T12:30:45,123456789 > + > + The time designator [T] may be omitted. > +]] > +local function parse_time(str) > + local sp = ffi.new('int[1]') > + local fp = ffi.new('int[1]') > + local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) > + assert(rc > 0) > + return mk_timestamp(nil, sp[0], fp[0]) > +end > + > +--[[ > + Basic Extended > + Z N/A > + ±hh N/A > + ±hhmm ±hh:mm 30. I would recommend to avoid non-ASCII symbls in the code. I remember I used unicode table borders to build nice schemas but people were complaining it ruins their terminal's output. > +]] > +local function parse_zone(str) > + local offset = ffi.new('int[1]') > + local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) > + return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len) 31. Please, keep 80 symbols border. I am going to stop here, will return later.
Hi! Thanks for the patch! On 28.07.2021 12:34, Timur Safin wrote: > * Integrated chansen/c-dt parser as 3rd party module to the > Tarantool cmake build process. > * Points to tsafin/c-dt instead iof original chansen/c-dt to iof -> of. But is it even true? Below I see url = https://github.com/tarantool/c-dt.git not tsafin/c-dt. After build I get in git status this: Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) (commit or discard the untracked or modified content in submodules) modified: third_party/c-dt (modified content, untracked content) It says: modified: Makefile and some untracked files: CMakeFiles/ cmake_install.cmake libcdt.a Why is even Makefile added to the repository if we are using cmake anyway?
: From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> : Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime : : Thanks for the patch! : : This is a bad question to ask now, but why did you choose a dead library : from 2015 with just 25 stars, without any tests, and without any README as : a basic block for our datetime types? Is this the correct link I am : using - https://github.com/chansen/c-dt? Good you asked :) That was data-driven decision and you could see details here https://github.com/tsafin/bench-timestamps In short c-dt is as almost as feature-rich as ICU or CCTZ, but much, much faster (and we know how to make it even faster if we have to). Originally I've learned about c-dt from perl' perl5-time-moment mentioned by Mons (which module was the fastest date/time implementation in Perl ecosystem). : : See 31 comment below. : : On 28.07.2021 12:34, Timur Safin wrote: : > * created a new Tarantool built-in module `datetime`; : > * register cdef types for this module; : > * export some `dt_*` functions from `c-dt` library; : > * lua implementationis of `asctime` and `strftime`; : : 1. implementationis -> implementations : : > * datetime parsing unit tests, with and withput timezones; : : 2. withput -> without Thanks! : : > diff --git a/src/exports.h b/src/exports.h : > index 5bb3e6a2b..db40c03a4 100644 : > --- a/src/exports.h : > +++ b/src/exports.h : > @@ -531,3 +531,24 @@ EXPORT(uri_format) : > EXPORT(uri_parse) : > EXPORT(uuid_nil) : > EXPORT(uuid_unpack) : > +EXPORT(dt_from_rdn) : > +EXPORT(dt_from_yd) : > +EXPORT(dt_from_ymd) : > +EXPORT(dt_from_yqd) : > +EXPORT(dt_from_ywd) : > +EXPORT(dt_to_yd) : > +EXPORT(dt_to_ymd) : > +EXPORT(dt_to_yqd) : > +EXPORT(dt_to_ywd) : > +EXPORT(dt_rdn) : > +EXPORT(dt_dow) : > +EXPORT(dt_parse_iso_date) : > +EXPORT(dt_parse_iso_time) : > +EXPORT(dt_parse_iso_time_basic) : > +EXPORT(dt_parse_iso_time_extended) : > +EXPORT(dt_parse_iso_zone) : > +EXPORT(dt_parse_iso_zone_basic) : > +EXPORT(dt_parse_iso_zone_extended) : > +EXPORT(dt_parse_iso_zone_lenient) : > +EXPORT(dt_from_struct_tm) : > +EXPORT(dt_to_struct_tm) : : 3. Please, read the comment in the beginning of the file. The : first sentence would be: : : Keep the symbols sorted by name for search and addition : simplicity, to avoid duplicates. Yes, I knew it, but forgot. BTW, when I tried to sort the whole block I have discovered that there are multiple of symbols, which are already in incorrect order but I decided to not touch irrelevant chunks, and only sorted d* See in the next iteration... : : > diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h : > new file mode 100644 : > index 000000000..403bf1c64 : > --- /dev/null : > +++ b/src/lib/core/datetime.h : > @@ -0,0 +1,61 @@ : : <...> : : > +#include <c-dt/dt_core.h> : : 4. Please, use "" instead of <> for non-system headers. Ok. : : > +#include <stdint.h> : > +#include <stdbool.h> : > + : > +#if defined(__cplusplus) : > +extern "C" { : > +#endif /* defined(__cplusplus) */ : > + : > +/** : > + * datetime structure consisting of: : : 5. Well, this comment isn't really helpful. Also we do not : use _t suffix ever except for typedefs. Please, drop both : these things. : : The same for datetime_interval_t. Will rename to `struct datetime`, and `struct datetime_interval` Accordingly. : : > + */ : > +struct datetime_t { : > + int64_t secs; ///< seconds since epoch : > + int32_t nsec; ///< nanoseconds if any : > + int32_t offset; ///< offset in minutes from GMT : : 6. We never use // nor /// nor ///< nor we write comments : on the same line as the code (except for old legacy code we : inherited from sqlite and some super old tarantool code. BTW, why? Single line comment // is the legal C construction since C99 (and always have been acceptable by Gnu C and major other compilers). Also did you know that ///< is doxygen construction for documentation of a struct member? (Asking, just to make sure) : : The same for datetime_interval_t. : : > +}; : > + : > +/** : > + * Date/time delta structure : > + */ : > +struct datetime_interval_t { : > + int64_t secs; ///< relative seconds delta : > + int32_t nsec; ///< nanoseconds delta : > +}; : > + : : 7. Why do you need this file? It is not included anywhere. : And you don't need to define the structs in C if you are : using them in Lua only. You can just define them in Lua : using ffi.cdef like it is done in some other places. I'll need to use it in C very soon, once I get to SQL parser INTERVAL 'xx' support. : : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua : > new file mode 100644 : > index 000000000..0996ca5a2 : > --- /dev/null : > +++ b/src/lua/datetime.lua : > @@ -0,0 +1,581 @@ : > +local ffi = require('ffi') : > +local cdt = ffi.C : > + : > +ffi.cdef [[ : > + : > + typedef int dt_t; : > + : > + // dt_core.h : > + typedef enum { : > + DT_MON = 1, : > + DT_MONDAY = 1, : > + DT_TUE = 2, : > + DT_TUESDAY = 2, : > + DT_WED = 3, : > + DT_WEDNESDAY = 3, : > + DT_THU = 4, : > + DT_THURSDAY = 4, : > + DT_FRI = 5, : > + DT_FRIDAY = 5, : > + DT_SAT = 6, : > + DT_SATURDAY = 6, : > + DT_SUN = 7, : > + DT_SUNDAY = 7, : > + } dt_dow_t; : > + : > + dt_t dt_from_rdn (int n); : > + dt_t dt_from_yd (int y, int d); : > + dt_t dt_from_ymd (int y, int m, int d); : > + dt_t dt_from_yqd (int y, int q, int d); : > + dt_t dt_from_ywd (int y, int w, int d); : > + : > + void dt_to_yd (dt_t dt, int *y, int *d); : > + void dt_to_ymd (dt_t dt, int *y, int *m, int *d); : > + void dt_to_yqd (dt_t dt, int *y, int *q, int *d); : > + void dt_to_ywd (dt_t dt, int *y, int *w, int *d); : > + : > + int dt_rdn (dt_t dt); : > + dt_dow_t dt_dow (dt_t dt); : > + : > + // dt_parse_iso.h : > + size_t dt_parse_iso_date (const char *str, size_t len, dt_t : *dt); : > + : > + size_t dt_parse_iso_time (const char *str, size_t len, int : *sod, int *nsec); : > + size_t dt_parse_iso_time_basic (const char *str, size_t len, int : *sod, int *nsec); : > + size_t dt_parse_iso_time_extended (const char *str, size_t len, int : *sod, int *nsec); : > + : > + size_t dt_parse_iso_zone (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_basic (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_extended (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_lenient (const char *str, size_t len, int : *offset); : > + : > + // dt_tm.h : > + dt_t dt_from_struct_tm (const struct tm *tm); : > + void dt_to_struct_tm (dt_t dt, struct tm *tm); : > + : > + // <asm-generic/posix_types.h> : > + typedef long __kernel_long_t; : > + typedef unsigned long __kernel_ulong_t; : > + // /usr/include/x86_64-linux-gnu/bits/types/time_t.h : > + typedef long time_t; : > + : > + : > + // <time.h> : > + typedef __kernel_long_t __kernel_time_t; : > + typedef __kernel_long_t __kernel_suseconds_t; : > + : > + struct timespec { : > + __kernel_time_t tv_sec; /* seconds */ : > + long tv_nsec; /* nanoseconds */ : > + }; : > + : > + struct timeval { : > + __kernel_time_t tv_sec; /* seconds */ : > + __kernel_suseconds_t tv_usec; /* microseconds */ : > + }; : > + : > + struct timezone { : > + int tz_minuteswest; /* minutes west of Greenwich */ : > + int tz_dsttime; /* type of dst correction */ : > + }; : > + : > + // /usr/include/x86_64-linux-gnu/sys/time.h : > + typedef struct timezone * __timezone_ptr_t; : > + : > + /* Get the current time of day and timezone information, : > + putting it into *TV and *TZ. If TZ is NULL, *TZ is not filled. : > + Returns 0 on success, -1 on errors. : > + : > + NOTE: This form of timezone information is obsolete. : > + Use the functions and variables declared in <time.h> instead. */ : > + int gettimeofday (struct timeval *__tv, struct timezone * __tz); : > + : > + // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h : > + /* ISO C `broken-down time' structure. */ : > + struct tm : > + { : > + int tm_sec; /* Seconds. [0-60] (1 leap second) */ : > + int tm_min; /* Minutes. [0-59] */ : > + int tm_hour; /* Hours. [0-23] */ : > + int tm_mday; /* Day. [1-31] */ : > + int tm_mon; /* Month. [0-11] */ : > + int tm_year; /* Year - 1900. */ : > + int tm_wday; /* Day of week. [0-6] */ : > + int tm_yday; /* Days in year.[0-365] */ : > + int tm_isdst; /* DST. [-1/0/1]*/ : > + : > + long int tm_gmtoff; /* Seconds east of UTC. */ : > + const char *tm_zone;/* Timezone abbreviation. */ : > + }; : > + : > + // <time.h> : > + /* Return the current time and put it in *TIMER if TIMER is not NULL. : */ : > + time_t time (time_t *__timer); : > + : > + /* Format TP into S according to FORMAT. : > + Write no more than MAXSIZE characters and return the number : > + of characters written, or 0 if it would exceed MAXSIZE. */ : > + size_t strftime (char * __s, size_t __maxsize, const char * __format, : > + const struct tm * __tp); : > + : > + /* Parse S according to FORMAT and store binary time information in : TP. : > + The return value is a pointer to the first unparsed character in S. : */ : > + char *strptime (const char * __s, const char * __fmt, struct tm : *__tp); : > + : > + /* Return the `struct tm' representation of *TIMER in UTC, : > + using *TP to store the result. */ : > + struct tm *gmtime_r (const time_t * __timer, struct tm * __tp); : > + : > + /* Return the `struct tm' representation of *TIMER in local time, : > + using *TP to store the result. */ : > + struct tm *localtime_r (const time_t * __timer, struct tm * __tp); : > + : > + /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n" : > + that is the representation of TP in this format. */ : > + char *asctime (const struct tm *__tp); : > + : > + /* Equivalent to `asctime (localtime (timer))'. */ : > + char *ctime (const time_t *__timer); : > + : : 8. The amount of platform-specific stuff you got here really bothers me. : For instance, struct tm from /usr/include/x86_64-linux- : gnu/bits/types/struct_tm.h. : : Is it said anywhere that the struct has only these members and only in this : order? The standard says "includes at least the following members". Not : only them and no specific order. Agreed that despite the fact that `struct tm` is defined pretty similarly in POSIX/SUSV https://pubs.opengroup.org/onlinepubs/009696699/basedefs/time.h.html actual layout of (at least last fields tm_gmtoff and tm_zone) might vary significantly. And we better hide this layout specific code (i.e. allocation and manipulations with fields) into accompanying C code. : : I would probably go with Lua C implementation if I would start this : module. Too strong dependecy on C. Also I wouldn't be so sure FFI is faster : than : Lua C: https://github.com/tarantool/tarantool/issues/5896. As you know - who is faster is complicated question. There are some subtle bug in LuaJIT with structure allocations which I believe was the major contributor to slowness reported in Decimal branch https://github.com/tarantool/tarantool/issues/692#issuecomment-505043929 but which has been narrowed down by Sergos and Mike has already committed patch to upstream in https://github.com/LuaJIT/LuaJIT/commit/ad65934fa08a65bfb0eb9528731a4394842cc173 but, I believe, it's not yet in our LuaJIT fork. I'd probably postpone total redesign from ffi to LuaC of this 1st version of module (because performance is not yet problem, functionality is), but rather wait and see how things would change once this struct-of-NYI would be landed to Tarantool code. : : > +]] : > + : > +local native = ffi.C : : 9. You already have cdt in the beginning of the file. Yes, thanks : : > + : > +local SECS_PER_DAY = 86400 : > +local NANOS_PER_SEC = 1000000000LL : > + : > +-- c-dt/dt_config.h : > + : > +-- Unix, January 1, 1970, Thursday : > +local DT_EPOCH_1970_OFFSET = 719163LL : : 10. I remember there was a good question somewhere - how are these : datetimes supposed to store dates before 1970? For example, a : database of historical events or documents. : : Can this library store dates before Christ? Yes, please see relevant discussion with Peter https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1055996 c-dt is properly handling positive and negative dt values, where dt is number of dates since Rata Die, January 1, 0001. For better MessagePack compactness of our typical data we shift beginning of our 0 to the Unix Epoch (1970-01-01), and our 0 is actually dt=719163. So our epoch-based second 0 would become dt value (719163) via this simple formula (secs / 86400) + 719163. Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die (0001-01-01) in dates. We might consider to further shift, though, our relative 0 to any date in time like 2000-01-01, whatever, if we would decide that that different base would provide us more compact storage. But I believe unix-epoch date, from statistical point of view, is more convenient for storing and using at runtime. : : > + : > + : > +local datetime_t = ffi.typeof('struct datetime_t') : > +local interval_t = ffi.typeof('struct datetime_interval_t') : > + : > +local function interval_new() : > + local interval = ffi.new(interval_t) : > + return interval : > +end : > + : > +local function adjusted_secs(dt) : > + return dt.secs - dt.offset * 60 : > +end : > + : > +local function datetime_sub(lhs, rhs) : > + local s1 = adjusted_secs(lhs) : > + local s2 = adjusted_secs(rhs) : : 11. Did you think about storing the time in a unified : format with offset already applied and add the offset only : when you print the time or convert to something? It seems : these `dt.offset * 60` might be expensive on each arith : operation done again and again. Exactly! That was bug in the early patches, which has been fixed in later version of this code. Sorry for the confusion created. Yes, secs are UC normalized, and offset if nominal (used only For output). : : > + local d = interval_new() : > + d.secs = s2 - s1 : > + d.nsec = rhs.nsec - lhs.nsec : > + if d.nsec < 0 then : : 12. AFAIK, in Lua index operator '.' is costly. : You might want to cache its values like this: : : local secs = s2 - s1 : local nsec = rhs.nsec - lhs.nsec : if nsec < 0 then : d.secs = secs - 1 : d.nsec = nsec + NANOS_PER_SEC : else : d.secs = secs : d.nsec = nsec : end : : The same in the other places where index operation : is repeated more than once: datetime_lt, datetime_le, : etc. Thanks, I'll review all relevant places and extract read references to locals. : : > + d.secs = d.secs - 1 : > + d.nsec = d.nsec + NANOS_PER_SEC : > + end : > + return d : > +end : > + : > +local function datetime_add(lhs, rhs) : > + local s1 = adjusted_secs(lhs) : > + local s2 = adjusted_secs(rhs) : > + local d = interval_new() : > + d.secs = s2 + s1 : > + d.nsec = rhs.nsec + lhs.nsec : > + if d.nsec >= NANOS_PER_SEC then : > + d.secs = d.secs + 1 : > + d.nsec = d.nsec - NANOS_PER_SEC : > + end : > + return d : > +end : > + : > +local function datetime_eq(lhs, rhs) : > + -- we usually don't need to check nullness : > + -- but older tarantool console will call us checking for equality to : nil : > + if rhs == nil then : > + return false : > + end : > + return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) : : 13. You do not need () around the comparison operators. The same in all : places : below. Also you could make it shorter: : : return rhs ~= nil and lhs.secs == rhs.secs and lhs.nsec == rhs.nsec Nice! Thanks! : : 14. Why don't you take 'offset' into account? Because it's not needed due to UTC-normalization. Sorry for the confusion with adjusted_sescs function. : : > +end : > + : > + : > +local function datetime_lt(lhs, rhs) : > + return (lhs.secs < rhs.secs) or : > + (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) : > +end : > + : > +local function datetime_le(lhs, rhs) : > + return (lhs.secs <= rhs.secs) or : > + (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) : > +end : > + : > +local function datetime_serialize(self) : > + -- Allow YAML, MsgPack and JSON to dump objects with sockets : > + return { secs = self.secs, nsec = self.nsec, tz = self.offset } : : 15. Why is 'offset' serialized as 'tz' and not just 'offset'? Yes, now you've asked it and it's looking confusing after rename. : : > +end : > + : > +local function interval_serialize(self) : > + -- Allow YAML and JSON to dump objects with sockets : > + return { secs = self.secs, nsec = self.nsec } : > +end : > + : > +local datetime_mt = { : > + -- __tostring = datetime_tostring, : : 16. Why is it commented out? Is it even tested then? It's implemented in the later patch in patchset series. : : > + __serialize = datetime_serialize, : > + __eq = datetime_eq, : > + __lt = datetime_lt, : > + __le = datetime_le, : > + __sub = datetime_sub, : > + __add = datetime_add, : > + : > + nanoseconds = function(self) : > + return tonumber(self.secs*NANOS_PER_SEC + self.nsec) : > + end, : : 17. Please, separate function definitions with an empty line : between them. Done! (Will be pushed later) : : > + microseconds = function(self) : > + return tonumber(self.secs*1e6 + self.nsec*1e3) : : 18. Please, add whitespaces around binary operators. Done! It was already such in the later commits which were part of in patchset. [Yes, apparently I need to squash few more commits, as Oleg Babin has asked elsewhere] : : 19. Why do you do tonumber() for all results? You are loosing : precision for big values. The same in all the places where you : use tonumber(). To be able to use it in places which expect numbers like %d in string.format. Though at the end of day I've came up to the version where I convert to Lua numbers not all (potentially fraction point) values, but with exception of nanoseconds / microseconds / milliseconds (which return integer values). : : > + end, : > + seconds = function(self) : > + return tonumber(self.secs + self.nsec*1e3) : > + end, : > + minutes = function(self) : > + return tonumber((self._ticks/(1e6*60))%60) : > + end, : > + hours = function(self) : > + return tonumber(self._ticks/(1e6*60*60)) : : 20. I don't see _ticks defined anywhere in our source code in : any of the files. What is it? Is it tested? Yes, that was miscopy from elsewhere. Fixed in newer commits. Sorry for the confusion. : : > + end, : > + : : 21. Extra empty line. Yup, but later, well, you know ... : : > +} : > + : > +local interval_mt = { : > + -- __tostring = interval_tostring, : : 22. Ditto. Why is it commented out and not tested? Tostring conversion was implemented later in a few commits. One for datetime stringization, and another one for interval. : : > + __serialize = interval_serialize, : > + __eq = datetime_eq, : > + __lt = datetime_lt, : > + __le = datetime_le, : > +} : > + : > +local function datetime_new_raw(secs, nsec, offset) : > + local dt_obj = ffi.new(datetime_t) : > + dt_obj.secs = secs : > + dt_obj.nsec = nsec : > + dt_obj.offset = offset : > + return dt_obj : > +end : > + : > +local function local_rd(o) : : 23. What is 'rd' and 'dt'? `dt` is type - data type from c-dt (alias to signed integer). Which shows a number of dates since Rata Die date (0001-01-01). It's signed 32-bit value, thus could represent huge range before 0001-01-01 as well. : : > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + : DT_EPOCH_1970_OFFSET : > +end : > + : > +local function local_dt(o) : > + return cdt.dt_from_rdn(local_rd(o)) : > +end : > + : > +local function mk_timestamp(dt, sp, fp, offset) : > + local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) : * SECS_PER_DAY or 0 : > + local spV = sp ~= nil and sp or 0 : > + local fpV = fp ~= nil and fp or 0 : > + local ofsV = offset ~= nil and offset or 0 : > + return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV) : > +end : > + : > +-- create @datetime_t given object @o fields : > +local function datetime_new(o) : : 24. Why such a wierd name 'o'? It looks like 0, super : confusing. o means object (with attributes) here. Will make it look longer, i.e. obj. : : > + if o == nil then : > + return datetime_new_raw(0, 0, 0) : > + end : > + local secs = 0 : > + local nsec = 0 : > + local offset = 0 : > + local easy_way = false : > + local y, M, d, ymd : > + y, M, d, ymd = 0, 0, 0, false : > + : > + local h, m, s, frac, hms : > + h, m, s, frac, hms = 0, 0, 0, 0, false : : 25. Please, try to define the variables on their own lines. : Like you did a few lines above. Otherwise it is practically : unreadable. Ok : : > + : > + local dt = 0 : > + : > + for key, value in pairs(o) do : > + local handlers = { : > + secs = function(v) : > + secs = v : > + easy_way = true : : 26. I don't understand what 'easy way' means. Please, elaborate : what you are trying to achieve. We could create datetime object 2 ways: - using raw attributes, without any conversion - i.e. secs, nsec, offset; - or using human-friendly year/month/day/hour/minutes/seconds; The easy_way is using raw mode - defining .secs, .nsec, .offset, but using The same object constructor interface. i.e. date = require 'datetime' T = date{ secs = 0, nsec = 0, offset = 0} Will create datetime object representing 1970-01-01T00:00Z moment. : : > + end, : : 27. The usage of closures here might render all your FFI efforts : useless, killing the performance. Please, try to define all : methods of all objects only once in the root namespace of the : file. Closure usage might be justified only for rarely created : long living heavy objects like netbox. Agreed that creating closures in each iteration of a loop was bad idea (as Oleg Babin has already pin-pointed it elsewhere). And we have to ate least to remove loop-invariant outside of cycle. But at the moment I do not consider the code creating object as very performance critical path, because I do not expect this path used frequently, more like after parsing of ido-8601 literals, which uses raw object construction approach. [I'll bench though the difference between using invariant closures and multiple ifs to see the size of a problem] : : > + : > + nsec = function(v) : > + nsec = v : > + easy_way = true : > + end, : > + : > + offset = function (v) : > + offset = v : > + easy_way = true : > + end, : > + : > + year = function(v) : > + assert(v > 0 and v < 10000) : > + y = v : > + ymd = true : > + end, : > + : > + month = function(v) : > + assert(v > 0 and v < 12 ) : > + M = v : > + ymd = true : > + end, : > + : > + day = function(v) : > + assert(v > 0 and v < 32) : > + d = v : > + ymd = true : > + end, : > + : > + hour = function(v) : > + assert(v >= 0 and v < 24) : > + h = v : > + hms = true : > + end, : > + : > + minute = function(v) : > + assert(v >= 0 and v < 60) : > + m = v : > + hms = true : > + end, : > + : > + second = function(v) : > + assert(v >= 0 and v < 61) : > + frac = v % 1 : > + if frac then : > + s = v - (v % 1) : > + else : > + s = v : > + end : > + hms = true : > + end, : > + : > + -- tz offset in minutes : > + tz = function(v) : > + assert(v >= 0 and v <= 720) : > + offset = v : > + end : > + } : > + handlers[key](value) : > + end : > + : > + -- .sec, .nsec, .offset : > + if easy_way then : > + return datetime_new_raw(secs, nsec, offset) : > + end : > + : > + -- .year, .month, .day : > + if ymd then : > + dt = dt + cdt.dt_from_ymd(y, M, d) : > + end : > + : > + -- .hour, .minute, .second : > + if hms then : > + secs = h * 3600 + m * 60 + s : > + end : > + : > + return mk_timestamp(dt, secs, frac, offset) : > +end : > + : > + : > +-- simple parse functions: : > +-- parse_date/parse_time/parse_zone : : 28. I wouldn't be so sure they are 'simple' and I can read : their names below anyway. This makes this comment impractical. :) ok : : > + : > +--[[ : > + Basic Extended : > + 20121224 2012-12-24 Calendar date (ISO 8601) : > + 2012359 2012-359 Ordinal date (ISO 8601) : > + 2012W521 2012-W52-1 Week date (ISO 8601) : > + 2012Q485 2012-Q4-85 Quarter date : > +]] : > + : > +local function parse_date(str) : > + local dt = ffi.new('dt_t[1]') : > + local rc = cdt.dt_parse_iso_date(str, #str, dt) : > + assert(rc > 0) : : 29. Er ... And what if I pass a string in a wrong format? Users : should not get assertions. : : tarantool> datetime.parse_date('121r1 r1r 13 13 13') : --- : - error: 'builtin/datetime.lua:397: assertion failed!' : ... : : This shall not ever happen. Would you mind implementing proper : error handling. The same in all the places where the errors are : ignored. Now, I've reworked approach how they handle errors (you could see It in later commits in the patch-set), and now it returns tuple - - parsed value - and the number of accepted input characters. tarantool> date.parse_date('121r1 r1r 13 13 13') --- - null - 0 ... Sometimes, while parsing composed strings using these partial methods (parse_date / parse_time / parse_time_zone) we need to know the number of symbols we have successfully accepted. That's why there is seconds value in returned tuple. Please give me know if there is better/more idiomatic approach used for such scenarios? : : > + return mk_timestamp(dt[0]) : > +end : > + : > +--[[ : > + Basic Extended : > + T12 N/A : > + T1230 T12:30 : > + T123045 T12:30:45 : > + T123045.123456789 T12:30:45.123456789 : > + T123045,123456789 T12:30:45,123456789 : > + : > + The time designator [T] may be omitted. : > +]] : > +local function parse_time(str) : > + local sp = ffi.new('int[1]') : > + local fp = ffi.new('int[1]') : > + local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) : > + assert(rc > 0) : > + return mk_timestamp(nil, sp[0], fp[0]) : > +end : > + : > +--[[ : > + Basic Extended : > + Z N/A : > + ±hh N/A : > + ±hhmm ±hh:mm : : 30. I would recommend to avoid non-ASCII symbls in the code. I : remember I used unicode table borders to build nice schemas : but people were complaining it ruins their terminal's output. Even if it's properly utf-8 encoded? That's odd. Will replace with + and - separately. : : > +]] : > +local function parse_zone(str) : > + local offset = ffi.new('int[1]') : > + local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) : > + return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, : tonumber(len) : : 31. Please, keep 80 symbols border. Yup : : I am going to stop here, will return later. Thanks! Timur
Hello Oleg, Thanks for quick review! : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in : module datetime : : Thanks for your patch! : : : Some places from prevous review are still not fixed for some reasons. Yup, that was oversight due to misunderstanding. Hopefully now I Understand it better. : : : Please be careful with our Lua style guide I ponted some obvious violations. : : Also it would be great to analyze module functions with our memprof. I : think there are some places that could be optimized. Hmm, hmm, probably a good idea. (Assuming using memprof is not a rocket science for such relatively newbie in LuaJIT as myself) ... : : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua : > new file mode 100644 : > index 000000000..0996ca5a2 : > --- /dev/null : > +++ b/src/lua/datetime.lua : > @@ -0,0 +1,581 @@ : > +local ffi = require('ffi') : : Do we have any benchmarks that shows that FFI is faster than Lua C API? : : Or it's just easy way to implement such module. Ok, here is a selection list which I've seen: - decimal is in LuaC; - uuid is in ffi; It looked equally acceptable and I've selected the one which is more convenient - via ffi. Now, you've pointed me to the probable performance issues we have with ffi, which are not there with LuaC. I believe they have been fixed with that Mike fix for NYI in structure allocations (after Sergos dirty fix) and once we will land this patch to our local fork those performance issues for structures allocations will gone. In any case, 1st version of module is not yet a proper moment to start to worry about performance, more rather about clarity and quality of API. We need to return to benchmarking of this code soon after code freeze lifted. : : : > +local cdt = ffi.C : > + : > +ffi.cdef [[ : > + : > + typedef int dt_t; : > + : > + // dt_core.h : > + typedef enum { : > + DT_MON = 1, : > + DT_MONDAY = 1, : > + DT_TUE = 2, : > + DT_TUESDAY = 2, : > + DT_WED = 3, : > + DT_WEDNESDAY = 3, : > + DT_THU = 4, : > + DT_THURSDAY = 4, : > + DT_FRI = 5, : > + DT_FRIDAY = 5, : > + DT_SAT = 6, : > + DT_SATURDAY = 6, : > + DT_SUN = 7, : > + DT_SUNDAY = 7, : > + } dt_dow_t; : > + : > + dt_t dt_from_rdn (int n); : > + dt_t dt_from_yd (int y, int d); : > + dt_t dt_from_ymd (int y, int m, int d); : > + dt_t dt_from_yqd (int y, int q, int d); : > + dt_t dt_from_ywd (int y, int w, int d); : > + : > + void dt_to_yd (dt_t dt, int *y, int *d); : > + void dt_to_ymd (dt_t dt, int *y, int *m, int *d); : > + void dt_to_yqd (dt_t dt, int *y, int *q, int *d); : > + void dt_to_ywd (dt_t dt, int *y, int *w, int *d); : > + : > + int dt_rdn (dt_t dt); : > + dt_dow_t dt_dow (dt_t dt); : > + : > + // dt_parse_iso.h : > + size_t dt_parse_iso_date (const char *str, size_t len, dt_t : *dt); : > + : > + size_t dt_parse_iso_time (const char *str, size_t len, int : *sod, int *nsec); : > + size_t dt_parse_iso_time_basic (const char *str, size_t len, int : *sod, int *nsec); : > + size_t dt_parse_iso_time_extended (const char *str, size_t len, int : *sod, int *nsec); : > + : > + size_t dt_parse_iso_zone (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_basic (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_extended (const char *str, size_t len, int : *offset); : > + size_t dt_parse_iso_zone_lenient (const char *str, size_t len, int : *offset); : > + : > + // dt_tm.h : > + dt_t dt_from_struct_tm (const struct tm *tm); : > + void dt_to_struct_tm (dt_t dt, struct tm *tm); : > + : > + // <asm-generic/posix_types.h> : > + typedef long __kernel_long_t; : > + typedef unsigned long __kernel_ulong_t; : > + // /usr/include/x86_64-linux-gnu/bits/types/time_t.h : > + typedef long time_t; : > + : > + : > + // <time.h> : > + typedef __kernel_long_t __kernel_time_t; : > + typedef __kernel_long_t __kernel_suseconds_t; : > + : > + struct timespec { : > + __kernel_time_t tv_sec; /* seconds */ : > + long tv_nsec; /* nanoseconds */ : > + }; : > + : > + struct timeval { : > + __kernel_time_t tv_sec; /* seconds */ : > + __kernel_suseconds_t tv_usec; /* microseconds */ : > + }; : > + : > + struct timezone { : > + int tz_minuteswest; /* minutes west of Greenwich */ : > + int tz_dsttime; /* type of dst correction */ : > + }; : > + : > + // /usr/include/x86_64-linux-gnu/sys/time.h : > + typedef struct timezone * __timezone_ptr_t; : > + : > + /* Get the current time of day and timezone information, : > + putting it into *TV and *TZ. If TZ is NULL, *TZ is not filled. : > + Returns 0 on success, -1 on errors. : > + : > + NOTE: This form of timezone information is obsolete. : > + Use the functions and variables declared in <time.h> instead. */ : > + int gettimeofday (struct timeval *__tv, struct timezone * __tz); : > + : > + // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h : > + /* ISO C `broken-down time' structure. */ : > + struct tm : > + { : > + int tm_sec; /* Seconds. [0-60] (1 leap second) */ : > + int tm_min; /* Minutes. [0-59] */ : > + int tm_hour; /* Hours. [0-23] */ : > + int tm_mday; /* Day. [1-31] */ : > + int tm_mon; /* Month. [0-11] */ : > + int tm_year; /* Year - 1900. */ : > + int tm_wday; /* Day of week. [0-6] */ : > + int tm_yday; /* Days in year.[0-365] */ : > + int tm_isdst; /* DST. [-1/0/1]*/ : > + : > + long int tm_gmtoff; /* Seconds east of UTC. */ : > + const char *tm_zone;/* Timezone abbreviation. */ : > + }; : > + : > + // <time.h> : > + /* Return the current time and put it in *TIMER if TIMER is not NULL. : */ : > + time_t time (time_t *__timer); : > + : > + /* Format TP into S according to FORMAT. : > + Write no more than MAXSIZE characters and return the number : > + of characters written, or 0 if it would exceed MAXSIZE. */ : > + size_t strftime (char * __s, size_t __maxsize, const char * __format, : > + const struct tm * __tp); : > + : > + /* Parse S according to FORMAT and store binary time information in : TP. : > + The return value is a pointer to the first unparsed character in S. : */ : > + char *strptime (const char * __s, const char * __fmt, struct tm : *__tp); : > + : > + /* Return the `struct tm' representation of *TIMER in UTC, : > + using *TP to store the result. */ : > + struct tm *gmtime_r (const time_t * __timer, struct tm * __tp); : > + : > + /* Return the `struct tm' representation of *TIMER in local time, : > + using *TP to store the result. */ : > + struct tm *localtime_r (const time_t * __timer, struct tm * __tp); : > + : > + /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n" : > + that is the representation of TP in this format. */ : > + char *asctime (const struct tm *__tp); : > + : > + /* Equivalent to `asctime (localtime (timer))'. */ : > + char *ctime (const time_t *__timer); : > + : > +]] : > + : > +local native = ffi.C : : You've aleady declared cdt as ffi.C. There is a redefinition here. Is it : really needed. Thanks for observation - that was copy-paste oversight from external module code (tsafin/c-dt-ffi), which is not making much sense if module is built-in. : : I don't see anybody use "native" for ffi.C. There are two ways - : "builtin" and "C". 'builtin` is better, yes. : : I suppose to choose one of them. : : > + : > +local SECS_PER_DAY = 86400 : > +local NANOS_PER_SEC = 1000000000LL : > + : > +-- c-dt/dt_config.h : > + : > +-- Unix, January 1, 1970, Thursday : > +local DT_EPOCH_1970_OFFSET = 719163LL : : To be honest it's not completely clear that such value means until : : I visited : https://github.com/chansen/c- : dt/blob/21b8cd1fcb984386b7d4552c16fdd03fafab2b6a/dt_config.h#L50. : : I think some comment is needed here. Yeah, having explained this same topic to Vlad I now realize it looks a little bit confusing :) : : : > + : > + : > +local datetime_t = ffi.typeof('struct datetime_t') : > +local interval_t = ffi.typeof('struct datetime_interval_t') : > + : > +local function interval_new() : > + local interval = ffi.new(interval_t) : > + return interval : > +end : > + : > +local function adjusted_secs(dt) : > + return dt.secs - dt.offset * 60 : > +end : > + : > +local function datetime_sub(lhs, rhs) : > + local s1 = adjusted_secs(lhs) : > + local s2 = adjusted_secs(rhs) : > + local d = interval_new() : > + d.secs = s2 - s1 : > + d.nsec = rhs.nsec - lhs.nsec : > + if d.nsec < 0 then : > + d.secs = d.secs - 1 : > + d.nsec = d.nsec + NANOS_PER_SEC : > + end : > + return d : > +end : > + : > +local function datetime_add(lhs, rhs) : > + local s1 = adjusted_secs(lhs) : > + local s2 = adjusted_secs(rhs) : > + local d = interval_new() : > + d.secs = s2 + s1 : > + d.nsec = rhs.nsec + lhs.nsec : > + if d.nsec >= NANOS_PER_SEC then : > + d.secs = d.secs + 1 : > + d.nsec = d.nsec - NANOS_PER_SEC : > + end : > + return d : > +end : > + : > +local function datetime_eq(lhs, rhs) : > + -- we usually don't need to check nullness : > + -- but older tarantool console will call us checking for equality to : nil : > + if rhs == nil then : > + return false : > + end : > + return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) : > +end : > + : : As in the previous review it will fail on attempt to compare with not : datetime value. : : tarantool> datetime.new() == newproxy() : --- : - error: 'builtin/datetime.lua:222: bad argument #1 to ''is_datetime'' : (C type expected, : got userdata)' : ... : : : Expected false. Understood. Finally! Here is the code with which I've ended up now: @@ -292,30 +295,32 @@ local function interval_seconds_new(s) return o end -local function datetime_eq(lhs, rhs) - check_date_interval(lhs, "datetime:__eq(date or interval)") - check_date_interval(rhs, "datetime:__eq(date or interval)") - return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec) +local function datetime_cmp(lhs, rhs) + if not is_date_interval(lhs) or + not is_date_interval(rhs) then + return nil + end + return (lhs.secs - rhs.secs) or (lhs.nsec - rhs.nsec) end +local function datetime_eq(lhs, rhs) + local rc = datetime_cmp(lhs, rhs) + return rc ~= nil and rc == 0 or false +end local function datetime_lt(lhs, rhs) - check_date_interval(lhs, "datetime:__lt(date or interval)") - check_date_interval(rhs, "datetime:__lt(date or interval)") - return (lhs.secs < rhs.secs) or - (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) + local rc = datetime_cmp(lhs, rhs) + return rc ~= nil and rc < 0 or false end local function datetime_le(lhs, rhs) - check_date_interval(lhs, "datetime:__le(date or interval)") - check_date_interval(rhs, "datetime:__le(date or interval)") - return (lhs.secs <= rhs.secs) or - (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) + local rc = datetime_cmp(lhs, rhs) + return rc ~= nil and rc <= 0 or false end : : : > + : > +local function datetime_lt(lhs, rhs) : > + return (lhs.secs < rhs.secs) or : > + (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec) : > +end : > + : > +local function datetime_le(lhs, rhs) : > + return (lhs.secs <= rhs.secs) or : > + (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec) : > +end : > + : > +local function datetime_serialize(self) : > + -- Allow YAML, MsgPack and JSON to dump objects with sockets : > + return { secs = self.secs, nsec = self.nsec, tz = self.offset } : > +end : > + : > +local function interval_serialize(self) : > + -- Allow YAML and JSON to dump objects with sockets : > + return { secs = self.secs, nsec = self.nsec } : > +end : > + : > +local datetime_mt = { : > + -- __tostring = datetime_tostring, : > + __serialize = datetime_serialize, : > + __eq = datetime_eq, : > + __lt = datetime_lt, : > + __le = datetime_le, : > + __sub = datetime_sub, : > + __add = datetime_add, : > + : > + nanoseconds = function(self) : > + return tonumber(self.secs*NANOS_PER_SEC + self.nsec) : I think there should be a space before and after "*". Here and below. : > + end, : > + microseconds = function(self) : > + return tonumber(self.secs*1e6 + self.nsec*1e3) : > + end, : > + seconds = function(self) : > + return tonumber(self.secs + self.nsec*1e3) : > + end, : > + minutes = function(self) : > + return tonumber((self._ticks/(1e6*60))%60) : > + end, : > + hours = function(self) : > + return tonumber(self._ticks/(1e6*60*60)) : > + end, : > + : I think this empty line could be removed. : > +} : > + : > +local interval_mt = { : > + -- __tostring = interval_tostring, : > + __serialize = interval_serialize, : > + __eq = datetime_eq, : > + __lt = datetime_lt, : > + __le = datetime_le, : > +} : > + : > +local function datetime_new_raw(secs, nsec, offset) : > + local dt_obj = ffi.new(datetime_t) : > + dt_obj.secs = secs : > + dt_obj.nsec = nsec : > + dt_obj.offset = offset : > + return dt_obj : > +end : > + : > +local function local_rd(o) : > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + : DT_EPOCH_1970_OFFSET : > +end : > + : > +local function local_dt(o) : > + return cdt.dt_from_rdn(local_rd(o)) : > +end : > + : > +local function mk_timestamp(dt, sp, fp, offset) : > + local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) : * SECS_PER_DAY or 0 : > + local spV = sp ~= nil and sp or 0 : > + local fpV = fp ~= nil and fp or 0 : > + local ofsV = offset ~= nil and offset or 0 : > + return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV) : > +end : > + : > +-- create @datetime_t given object @o fields : > +local function datetime_new(o) : > + if o == nil then : > + return datetime_new_raw(0, 0, 0) : > + end : > + local secs = 0 : > + local nsec = 0 : > + local offset = 0 : > + local easy_way = false : What does "easy_way" mean? As I've already tried to explain it to Vlad elsewhere: easy_way means direct usage of secs/nsec/offset attributed passed for creation of a new datetime object, and not parsing of human-readable year/month/day/ hour/minute/seconds attributes it has to deal with otherwise. : > + local y, M, d, ymd : > + y, M, d, ymd = 0, 0, 0, false : > + : > + local h, m, s, frac, hms : > + h, m, s, frac, hms = 0, 0, 0, 0, false : > + : > + local dt = 0 : > + : > + for key, value in pairs(o) do : > + local handlers = { : : Here you recreate handlers for each iteration of the loop. I think it : should be reworked. Yes, let me check how better will be if this handlers array will be invariant in the loop, and not created each iteration. I still prefer this tag to function approach because of a better clarity, but I need to verify in benchmark whether it provides acceptable performance. I do not expect thought that this attribute object creation approach would become a bootleneck in any case and might be on a critical path anywhere (I'd rather expect it for datetime objects created after parsing of massive bunch of log text), but need to look into timings we would have with invariant table. : : Currenlty it's quite slow I think even if-elseif-else branches will work : faster without : : creating redundant GC objects. Will see ... : : > + secs = function(v) : > + secs = v : > + easy_way = true : > + end, : > + : > + nsec = function(v) : > + nsec = v : > + easy_way = true : > + end, : > + : > + offset = function (v) : > + offset = v : > + easy_way = true : > + end, : > + : > + year = function(v) : > + assert(v > 0 and v < 10000) : Still there are some assertions that yield unclear error messages. They have been already reworked in a later patches of patchset. [Probably better to squash them all for datetime.lua, yes] : > + y = v : > + ymd = true : > + end, : > + : > + month = function(v) : > + assert(v > 0 and v < 12 ) : > + M = v : > + ymd = true : > + end, : > + : > + day = function(v) : > + assert(v > 0 and v < 32) : > + d = v : > + ymd = true : > + end, : > + : > + hour = function(v) : > + assert(v >= 0 and v < 24) : > + h = v : > + hms = true : > + end, : > + : > + minute = function(v) : > + assert(v >= 0 and v < 60) : > + m = v : > + hms = true : > + end, : > + : > + second = function(v) : > + assert(v >= 0 and v < 61) : > + frac = v % 1 : > + if frac then : > + s = v - (v % 1) : > + else : > + s = v : > + end : > + hms = true : > + end, : > + : > + -- tz offset in minutes : > + tz = function(v) : > + assert(v >= 0 and v <= 720) : > + offset = v : > + end : > + } : > + handlers[key](value) : > + end : > + : > + -- .sec, .nsec, .offset : > + if easy_way then : > + return datetime_new_raw(secs, nsec, offset) : > + end : > + : > + -- .year, .month, .day : > + if ymd then : > + dt = dt + cdt.dt_from_ymd(y, M, d) : > + end : > + : > + -- .hour, .minute, .second : > + if hms then : > + secs = h * 3600 + m * 60 + s : > + end : > + : > + return mk_timestamp(dt, secs, frac, offset) : > +end : > + : > + : > +-- simple parse functions: : > +-- parse_date/parse_time/parse_zone : > + : > +--[[ : > + Basic Extended : > + 20121224 2012-12-24 Calendar date (ISO 8601) : > + 2012359 2012-359 Ordinal date (ISO 8601) : > + 2012W521 2012-W52-1 Week date (ISO 8601) : > + 2012Q485 2012-Q4-85 Quarter date : > +]] : > + : > +local function parse_date(str) : > + local dt = ffi.new('dt_t[1]') : > + local rc = cdt.dt_parse_iso_date(str, #str, dt) : > + assert(rc > 0) : > + return mk_timestamp(dt[0]) : > +end : > + : > +--[[ : > + Basic Extended : > + T12 N/A : > + T1230 T12:30 : > + T123045 T12:30:45 : > + T123045.123456789 T12:30:45.123456789 : > + T123045,123456789 T12:30:45,123456789 : > + : > + The time designator [T] may be omitted. : > +]] : > +local function parse_time(str) : > + local sp = ffi.new('int[1]') : > + local fp = ffi.new('int[1]') : > + local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) : > + assert(rc > 0) : > + return mk_timestamp(nil, sp[0], fp[0]) : > +end : > + : > +--[[ : > + Basic Extended : > + Z N/A : > + ±hh N/A : > + ±hhmm ±hh:mm : > +]] : > +local function parse_zone(str) : > + local offset = ffi.new('int[1]') : > + local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset) : > + return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, : tonumber(len) : > +end : > + : > + : > +--[[ : > + aggregated parse functions : > + assumes to deal with date T time time_zone : > + at once : > + : > + date [T] time [ ] time_zone : > +]] : > +local function parse_str(str) : > + local dt = ffi.new('dt_t[1]') : > + local len = #str : > + local n = cdt.dt_parse_iso_date(str, len, dt) : > + local dt_ = dt[0] : > + if n == 0 or len == n then : > + return mk_timestamp(dt_) : > + end : > + : > + str = str:sub(tonumber(n) + 1) : > + : > + local ch = str:sub(1,1) : > + if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then : > + return mk_timestamp(dt_) : > + end : > + : > + str = str:sub(2) : > + len = #str : > + : > + local sp = ffi.new('int[1]') : > + local fp = ffi.new('int[1]') : > + local n = cdt.dt_parse_iso_time(str, len, sp, fp) : > + if n == 0 then : > + return mk_timestamp(dt_) : > + end : > + local sp_ = sp[0] : > + local fp_ = fp[0] : > + if len == n then : > + return mk_timestamp(dt_, sp_, fp_) : > + end : > + : > + str = str:sub(tonumber(n) + 1) : > + : > + if str:sub(1,1) == ' ' then : > + str = str:sub(2) : > + end : > + : > + len = #str : > + : > + local offset = ffi.new('int[1]') : > + n = cdt.dt_parse_iso_zone_lenient(str, len, offset) : > + if n == 0 then : > + return mk_timestamp(dt_, sp_, fp_) : > + end : > + return mk_timestamp(dt_, sp_, fp_, offset[0]) : > +end : > + : > +local function datetime_from(o) : > + if o == nil or type(o) == 'table' then : > + return datetime_new(o) : > + elseif type(o) == 'string' then : > + return parse_str(o) : > + end : > +end : > + : > +local function local_now() : > + local p_tv = ffi.new ' struct timeval [1] ' : : This line doesn't conform our code-style. Please wrap argument into : brackets. Yes, sorry. But for consistency with surrounding code I've put it inside of parentheses. : : The same for such places below. : : > + local rc = native.gettimeofday(p_tv, nil) : > + assert(rc == 0) : > + : > + local secs = p_tv[0].tv_sec : > + local nsec = p_tv[0].tv_usec * 1000 : > + : > + local p_time = ffi.new 'time_t[1]' : > + local p_tm = ffi.new 'struct tm[1]' : > + native.time(p_time) : > + native.localtime_r(p_time, p_tm) : > + -- local dt = cdt.dt_from_struct_tm(p_tm) : > + local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes : > + : > + return datetime_new_raw(secs, nsec, ofs) -- FIXME : Do you plan to fix it before merge? Thanks for spot! Done. : > +end : > + : > +local function datetime_to_tm_ptr(o) : > + local p_tm = ffi.new 'struct tm[1]' : > + assert(ffi.typeof(o) == datetime_t) : > + -- dt_to_struct_tm() fills only date data : > + cdt.dt_to_struct_tm(local_dt(o), p_tm) : > + : > + -- calculate the smaller data (hour, minute, : > + -- seconds) using datetime seconds value : > + local seconds_of_day = o.secs % 86400 : > + local hour = (seconds_of_day / 3600) % 24 : > + local minute = (seconds_of_day / 60) % 60 : > + p_tm[0].tm_sec = seconds_of_day % 60 : > + p_tm[0].tm_min = minute : > + p_tm[0].tm_hour = hour : > + : > + p_tm[0].tm_gmtoff = o.offset * 60 : > + : > + return p_tm : > +end : > + : > +local function asctime(o) : > + assert(ffi.typeof(o) == datetime_t) : > + local p_tm = datetime_to_tm_ptr(o) : > + return ffi.string(native.asctime(p_tm)) : > +end : > + : > +local function ctime(o) : > + assert(ffi.typeof(o) == datetime_t) : > + local p_time = ffi.new 'time_t[1]' : > + p_time[0] = o.secs : > + return ffi.string(native.ctime(p_time)) : > +end : > + : > +local function strftime(fmt, o) : > + assert(ffi.typeof(o) == datetime_t) : > + local sz = 50 : Why 50? Good question, for ISO-8601 formats, for normal date range (from 0000 till 9999 years) it's more like closer to 40, but well, there will be various timezones and all crazy kinds of input format combinations, so we need to either be precautious with larger sizes allocations. Or probably rely on glibc convention (which is apparently not in POSIX/SUSV) with 2 passes approach: 1. call with NULL so it will return size of required buffer; 2. then, after allocation, with the adjusted allocated buffer; https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html Something like that (beware of inline patch): ---------------------------------------------- local function strftime(fmt, o) check_date(o, "datetime.strftime(fmt, date)") - local sz = 50 - local buff = ffi.new('char[?]', sz) local p_tm = datetime_to_tm_ptr(o) - native.strftime(buff, sz, fmt, p_tm) + local sz = builtin.strftime(nil, 1024, fmt, p_tm) + local buff = ffi.new('char[?]', sz + 1) + builtin.strftime(buff, sz, fmt, p_tm) return ffi.string(buff) end ---------------------------------------------- : > + local buff = ffi.new('char[?]', sz) : > + local p_tm = datetime_to_tm_ptr(o) : > + native.strftime(buff, sz, fmt, p_tm) : > + return ffi.string(buff) : > +end : > + : > +-- strftime may be redirected to datetime:fmt("format") : > +local function datetime_fmt() : > +end : > + : > + : > +ffi.metatype(interval_t, interval_mt) : > +ffi.metatype(datetime_t, datetime_mt) : > + : > +return setmetatable( : > + { : > + datetime = datetime_new, : > + interval = interval_new, : > + : > + parse = parse_str, : > + parse_date = parse_date, : > + parse_time = parse_time, : > + parse_zone = parse_zone, : > + fmt = datetime_fmt, : > + : > + now = local_now, : > + -- strptime = strptime; : It should be dropped if you don't need it. : > + strftime = strftime, : > + asctime = asctime, : > + ctime = ctime, : > + }, { : > + __call = function(self, ...) return datetime_from(...) end : > + } : > +) Thanks, Timur
Hello Oleg, : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: : datetime tests : : Thanks for your patch. To be honest I don't understand the structure of : this changes. : : I expect that "datetime tests" will contain only tests. All unrelated to : tests changes : : should be squashed to the previous patch. Yup, sudden artefact of late minute squashes. Will copy to prior commit. : : : I put several comments below. : : : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: : > * created app-tap test for new builtin module `datetime.lua` : > * added case to check datetime string formatting using: : > - asctime (gmt time); : > - ctime (local TZ time); : > - strftime (using given format). : > : > * added positive/negative checks to datetime test : > - extended api of datetime.parse_date, .parse_time, .parse_time_zone : > with a length of parsed (sub)string; : > - this allows us to check partially valid strings like "20121224 Foo : bar". : > : > Part of #5941 : > --- : > src/lua/datetime.lua | 14 ++- : > test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++ : > 2 files changed, 197 insertions(+), 8 deletions(-) : > create mode 100755 test/app-tap/datetime.test.lua : > : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua : > index 0996ca5a2..f4d2d7737 100644 : > --- a/src/lua/datetime.lua : > +++ b/src/lua/datetime.lua : > @@ -318,7 +318,7 @@ local function datetime_new(o) : > end, : > : > month = function(v) : > - assert(v > 0 and v < 12 ) : > + assert(v > 0 and v < 13 ) : > M = v : > ymd = true : > end, : > @@ -393,9 +393,8 @@ end : > : > local function parse_date(str) : > local dt = ffi.new('dt_t[1]') : > - local rc = cdt.dt_parse_iso_date(str, #str, dt) : > - assert(rc > 0) : > - return mk_timestamp(dt[0]) : > + local len = cdt.dt_parse_iso_date(str, #str, dt) : > + return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len) : > end : > : > --[[ : > @@ -411,9 +410,8 @@ end : > local function parse_time(str) : > local sp = ffi.new('int[1]') : > local fp = ffi.new('int[1]') : > - local rc = cdt.dt_parse_iso_time(str, #str, sp, fp) : > - assert(rc > 0) : > - return mk_timestamp(nil, sp[0], fp[0]) : > + local len = cdt.dt_parse_iso_time(str, #str, sp, fp) : > + return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil, : tonumber(len) : > end : > : > --[[ : > @@ -448,7 +446,7 @@ local function parse_str(str) : > str = str:sub(tonumber(n) + 1) : > : > local ch = str:sub(1,1) : > - if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then : > + if ch:match('[Tt ]') == nil then : > return mk_timestamp(dt_) : > end : > : > diff --git a/test/app-tap/datetime.test.lua b/test/app- : tap/datetime.test.lua : > new file mode 100755 : > index 000000000..09c968858 : > --- /dev/null : > +++ b/test/app-tap/datetime.test.lua : > @@ -0,0 +1,191 @@ : > +#!/usr/bin/env tarantool : > + : > +local tap = require('tap') : > +local test = tap.test("errno") : > +local date = require('datetime') : > + : > +test:plan(5) : > + : > +test:test("Simple tests for parser", function(test) : > + test:plan(2) : > + test:ok(date("1970-01-01T01:00:00Z") == : > + date {year=1970, month=1, day=1, hour=1, minute=0, second=0}) : > + test:ok(date("1970-01-01T02:00:00+02:00") == : > + date {year=1970, month=1, day=1, hour=2, minute=0, second=0, : tz=120}) : > +end) : > + : > +test:test("Multiple tests for parser (with nanoseconds)", function(test) : > + test:plan(165) : > + -- borrowed from p5-time-moments/t/180_from_string.t : > + local tests = : > + { : > + { '1970-01-01T00:00:00Z', 0, 0, : 0 }, : > + { '1970-01-01T02:00:00+02:00', 0, 0, : 120 }, : > + { '1970-01-01T01:30:00+01:30', 0, 0, : 90 }, : > + { '1970-01-01T01:00:00+01:00', 0, 0, : 60 }, : > + { '1970-01-01T00:01:00+00:01', 0, 0, : 1 }, : > + { '1970-01-01T00:00:00+00:00', 0, 0, : 0 }, : > + { '1969-12-31T23:59:00-00:01', 0, 0, : -1 }, : > + { '1969-12-31T23:00:00-01:00', 0, 0, : -60 }, : > + { '1969-12-31T22:30:00-01:30', 0, 0, : -90 }, : > + { '1969-12-31T22:00:00-02:00', 0, 0, - : 120 }, : > + { '1970-01-01T00:00:00.123456789Z', 0, 123456789, : 0 }, : > + { '1970-01-01T00:00:00.12345678Z', 0, 123456780, : 0 }, : > + { '1970-01-01T00:00:00.1234567Z', 0, 123456700, : 0 }, : > + { '1970-01-01T00:00:00.123456Z', 0, 123456000, : 0 }, : > + { '1970-01-01T00:00:00.12345Z', 0, 123450000, : 0 }, : > + { '1970-01-01T00:00:00.1234Z', 0, 123400000, : 0 }, : > + { '1970-01-01T00:00:00.123Z', 0, 123000000, : 0 }, : > + { '1970-01-01T00:00:00.12Z', 0, 120000000, : 0 }, : > + { '1970-01-01T00:00:00.1Z', 0, 100000000, : 0 }, : > + { '1970-01-01T00:00:00.01Z', 0, 10000000, : 0 }, : > + { '1970-01-01T00:00:00.001Z', 0, 1000000, : 0 }, : > + { '1970-01-01T00:00:00.0001Z', 0, 100000, : 0 }, : > + { '1970-01-01T00:00:00.00001Z', 0, 10000, : 0 }, : > + { '1970-01-01T00:00:00.000001Z', 0, 1000, : 0 }, : > + { '1970-01-01T00:00:00.0000001Z', 0, 100, : 0 }, : > + { '1970-01-01T00:00:00.00000001Z', 0, 10, : 0 }, : > + { '1970-01-01T00:00:00.000000001Z', 0, 1, : 0 }, : > + { '1970-01-01T00:00:00.000000009Z', 0, 9, : 0 }, : > + { '1970-01-01T00:00:00.00000009Z', 0, 90, : 0 }, : > + { '1970-01-01T00:00:00.0000009Z', 0, 900, : 0 }, : > + { '1970-01-01T00:00:00.000009Z', 0, 9000, : 0 }, : > + { '1970-01-01T00:00:00.00009Z', 0, 90000, : 0 }, : > + { '1970-01-01T00:00:00.0009Z', 0, 900000, : 0 }, : > + { '1970-01-01T00:00:00.009Z', 0, 9000000, : 0 }, : > + { '1970-01-01T00:00:00.09Z', 0, 90000000, : 0 }, : > + { '1970-01-01T00:00:00.9Z', 0, 900000000, : 0 }, : > + { '1970-01-01T00:00:00.99Z', 0, 990000000, : 0 }, : > + { '1970-01-01T00:00:00.999Z', 0, 999000000, : 0 }, : > + { '1970-01-01T00:00:00.9999Z', 0, 999900000, : 0 }, : > + { '1970-01-01T00:00:00.99999Z', 0, 999990000, : 0 }, : > + { '1970-01-01T00:00:00.999999Z', 0, 999999000, : 0 }, : > + { '1970-01-01T00:00:00.9999999Z', 0, 999999900, : 0 }, : > + { '1970-01-01T00:00:00.99999999Z', 0, 999999990, : 0 }, : > + { '1970-01-01T00:00:00.999999999Z', 0, 999999999, : 0 }, : > + { '1970-01-01T00:00:00.0Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.00Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.0000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.00000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.000000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.0000000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.00000000Z', 0, 0, : 0 }, : > + { '1970-01-01T00:00:00.000000000Z', 0, 0, : 0 }, : > + { '1973-11-29T21:33:09Z', 123456789, 0, : 0 }, : > + { '2013-10-28T17:51:56Z', 1382982716, 0, : 0 }, : > + -- { '9999-12-31T23:59:59Z', 253402300799, : 0, 0 }, : : Why is it commented? It used to be failing to pass range checks, but now there is no reason (IIRC) need to restore this case. Thanks! : : > + } : > + for _, value in ipairs(tests) do : > + local str, epoch, nsec, offset : > + str, epoch, nsec, offset = unpack(value) : > + local dt = date(str) : > + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, : epoch)) : > + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec)) : > + test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, : offset)) : > + end : > +end) : > + : > +local ffi = require('ffi') : > + : > +ffi.cdef [[ : > + void tzset(void); : > +]] : > + : : : It's probably better to require all modules at the top of the file. Will do : : > +test:test("Datetime string formatting", function(test) : > + test:plan(7) : > + local str = "1970-01-01" : > + local t = date(str) : > + test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, : tonumber(t.secs))) : > + test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec)) : > + test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset)) : > + test:ok(date.asctime(t) == 'Thu Jan 1 00:00:00 1970\n', ('%s: : asctime'):format(str)) : > + -- ctime() is local timezone dependent. To make sure that : > + -- test is deterministic we enforce timezone via TZ environment : > + -- manipulations and calling tzset() : > + : > + -- redefine timezone to be always GMT-2 : > + os.setenv('TZ', 'GMT-2') : > + ffi.C.tzset() : > + test:ok(date.ctime(t) == 'Thu Jan 1 02:00:00 1970\n', ('%s: ctime : with timezone'):format(str)) : > + test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime : #1'):format(str)) : > + test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January : 1970', ('%s: strftime #2'):format(str)) : > +end) : > + : > +test:test("Parse iso date - valid strings", function(test) : > + test:plan(32) : > + local good = { : > + {2012, 12, 24, "20121224", 8 }, : > + {2012, 12, 24, "20121224 Foo bar", 8 }, : > + {2012, 12, 24, "2012-12-24", 10 }, : > + {2012, 12, 24, "2012-12-24 23:59:59", 10 }, : > + {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 }, : > + {2012, 12, 24, "2012359", 7 }, : > + {2012, 12, 24, "2012359T235959+0130", 7 }, : > + {2012, 12, 24, "2012-359", 8 }, : > + {2012, 12, 24, "2012W521", 8 }, : > + {2012, 12, 24, "2012-W52-1", 10 }, : > + {2012, 12, 24, "2012Q485", 8 }, : > + {2012, 12, 24, "2012-Q4-85", 10 }, : > + { 1, 1, 1, "0001-Q1-01", 10 }, : > + { 1, 1, 1, "0001-W01-1", 10 }, : > + { 1, 1, 1, "0001-01-01", 10 }, : > + { 1, 1, 1, "0001-001", 8 }, : > + } : > + : > + for _, value in ipairs(good) do : > + local year, month, day, str, date_part_len; : : I didn't see before that somebody uses semicolon after variables definition. Centures of programming in C/C++ give to know :) : : It's about code-style. It is clear that this is syntactically correct. : : > + year, month, day, str, date_part_len = unpack(value) : > + local expected_date = date{year = year, month = month, day = day} : > + local date_part, len : > + date_part, len = date.parse_date(str) : > + test:ok(len == date_part_len, ('%s: length check %d'):format(str, : len)) : > + test:ok(expected_date == date_part, ('%s: expected : date'):format(str)) : > + end : > +end) : > + : > +test:test("Parse iso date - invalid strings", function(test) : > + test:plan(62) : > + local bad = { : > + "20121232" , -- Invalid day of month : > + "2012-12-310", -- Invalid day of month : > + "2012-13-24" , -- Invalid month : > + "2012367" , -- Invalid day of year : > + "2012-000" , -- Invalid day of year : > + "2012W533" , -- Invalid week of year : > + "2012-W52-8" , -- Invalid day of week : > + "2012Q495" , -- Invalid day of quarter : > + "2012-Q5-85" , -- Invalid quarter : > + "20123670" , -- Trailing digit : > + "201212320" , -- Trailing digit : > + "2012-12" , -- Reduced accuracy : > + "2012-Q4" , -- Reduced accuracy : > + "2012-Q42" , -- Invalid : > + "2012-Q1-1" , -- Invalid day of quarter : > + "2012Q--420" , -- Invalid : > + "2012-Q-420" , -- Invalid : > + "2012Q11" , -- Incomplete : > + "2012Q1234" , -- Trailing digit : > + "2012W12" , -- Incomplete : > + "2012W1234" , -- Trailing digit : > + "2012W-123" , -- Invalid : > + "2012-W12" , -- Incomplete : > + "2012-W12-12", -- Trailing digit : > + "2012U1234" , -- Invalid : > + "2012-1234" , -- Invalid : > + "2012-X1234" , -- Invalid : > + "0000-Q1-01" , -- Year less than 0001 : > + "0000-W01-1" , -- Year less than 0001 : > + "0000-01-01" , -- Year less than 0001 : > + "0000-001" , -- Year less than 0001 : > + } : > + : > + for _, str in ipairs(bad) do : > + local date_part, len : > + date_part, len = date.parse_date(str) : > + test:ok(len == 0, ('%s: length check %d'):format(str, len)) : > + test:ok(date_part == nil, ('%s: empty date check %s'):format(str, : date_part)) : > + end : > +end) : > + : > +os.exit(test:check() and 0 or 1) Thanks, Timur
Hello Oleg, : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: : display datetime : : Thanks for your patch. It's not review at all. I have a question: : : why is datetime not scalar? Good question - it has to be scalar. It's some oversight from my side. : : : Also I think that encoding and type system changes should live in : separate patches. I'm not 100% sure that understand this your complain. Could you please clarify? Thanks, Timur
Hello Oleg, : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: : datetime comparison for indices : : Thanks for your patch. One question below. : : : Also update with arithmetic operations doesn't work. : : : ``` : : tarantool> box.space.dt:update({v}, {{'+', 2, 3}}) : --- : - error: 'Argument type in operation ''+'' on field 2 does not match : field type: expected : a number' : ... : : tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}}) : --- : - error: unsupported Lua type 'cdata' : ... : : ``` Oh, shit, I didn't take those operations into considerations. Do they work similarly with decimal? : : : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: : > * storage hints implemented for datetime_t values; : > * proper comparison for indices of datetime type. : > : > : > diff --git a/test/engine/datetime.result b/test/engine/datetime.result : > new file mode 100644 : > index 000000000..3ff474dee : > --- /dev/null : > +++ b/test/engine/datetime.result : > @@ -0,0 +1,77 @@ : > +-- test-run result file version 2 : > +env = require('test_run') : > + | --- : > + | ... : > +test_run = env.new() : > + | --- : > + | ... : > +engine = test_run:get_cfg('engine') : > + | --- : > + | ... : > + : > +date = require('datetime') : > + | --- : > + | ... : > + : > +_ = box.schema.space.create('T', {engine = engine}) : > + | --- : > + | ... : > +_ = box.space.T:create_index('pk', {parts={1,'datetime'}}) : > + | --- : > + | ... : > + : > +box.space.T:insert{date('1970-01-01')}\ : > +box.space.T:insert{date('1970-01-02')}\ : > +box.space.T:insert{date('1970-01-03')}\ : > +box.space.T:insert{date('2000-01-01')} : > + | --- : > + | ... : > + : > +o = box.space.T:select{} : > + | --- : > + | ... : > +assert(tostring(o[1][1]) == '1970-01-01T00:00Z') : > + | --- : > + | - true : > + | ... : > +assert(tostring(o[2][1]) == '1970-01-02T00:00Z') : > + | --- : > + | - true : > + | ... : > +assert(tostring(o[3][1]) == '1970-01-03T00:00Z') : > + | --- : > + | - true : > + | ... : > +assert(tostring(o[4][1]) == '2000-01-01T00:00Z') : > + | --- : > + | - true : > + | ... : > + : > +for i = 1,16 do\ : > + box.space.T:insert{date.now()}\ : > +end : > + | --- : > + | ... : > + : > +a = box.space.T:select{} : > + | --- : > + | ... : > +err = nil : > + | --- : > + | ... : > +for i = 1, #a - 1 do\ : > + if tostring(a[i][1]) >= tostring(a[i+1][1]) then\ : : Why do you compare string representation but not values itself? Good question! Yeah, there should be original values compared, without stringization. (And in this case, by some reason, it's easier to reproduce problem of incorrect sort ofder with sequential times from date.now(). Which used to be reproduced only on some configs at GH) : : : > + err = {a[i][1], a[i+1][1]}\ : > + break\ : > + end\ : > +end : > + | --- : > + | ... : > + : > +err : > + | --- : > + | - null : > + | ... : > +box.space.T:drop() : > + | --- : > + | ... : >
: From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: : proper datetime encoding : : Thanks for your patch. This change should be squashed into the patch : where encoding was introduced. Agreed. : : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: : > * correct incorrect encoding of MP_EXT sizes for datetime : > messagepack values; : > * export necessary symbols for datetime messagepack size calculations : > so they will be available for Lua consumption. : > : > Part of #5941 : > --- : > src/exports.h | 2 ++ : > src/lib/core/datetime.c | 15 ++++++++++----- : > src/lua/msgpackffi.lua | 10 ++++++++++ : > 3 files changed, 22 insertions(+), 5 deletions(-) : > Thanks, Timur
Hello Oleg again! : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: : calculated attributes for datetimes : : Thanks for your patch. See one comment below. : : Also this change should be squashed into the first patch. Agreed as well. : : : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: : > * introduced a set of calculated attributes to data object, e.g.: : > - timestamp, seconds, microseconds, minute, or hours : > : > Part of #5941 : > --- : > src/lua/datetime.lua | 51 ++++++++++++++++++++++------------ : > test/app-tap/datetime.test.lua | 17 +++++++++++- : > 2 files changed, 50 insertions(+), 18 deletions(-) : > : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua : > index 9ec06d8d8..7a208cef9 100644 : > --- a/src/lua/datetime.lua : > +++ b/src/lua/datetime.lua : > @@ -224,6 +224,36 @@ local function interval_serialize(self) : > return { secs = self.secs, nsec = self.nsec } : > end : > : > +local datetime_index = function(self, key) : > + local attributes = { : : This change looks like pessimization because currently we need to : recreate "attributes" table : : for each function call. I suggest to use `attributes` as `__index` but : not such function. What would you say if I simply precache this attributes table in local, and then proceed this way? ----------------------------------------- -local datetime_index = function(self, key) - local attributes = { +local datetime_index_handlers = { unixtime = function(self) return self.secs end, + timestamp = function(self) return tonumber(self.secs) + self.nsec / 1e9 end, + nanoseconds = function(self) return self.secs * 1e9 + self.nsec end, + microseconds = function(self) return self.secs * 1e6 + self.nsec / 1e3 end, + milliseconds = function(self) return self.secs * 1e3 + self.nsec / 1e6 end, + seconds = function(self) return tonumber(self.secs) + self.nsec / 1e9 end, + minutes = function(self) return (tonumber(self.secs) + self.nsec / 1e9) / 60 end, + hours = function(self) return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) end, + days = function(self) return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24 end, + add = function(self) return function(self, o) return interval_increment(self, o, 1) end end, + sub = function(self) return function(self, o) return interval_increment(self, o, -1) end end, } - return attributes[key] ~= nil and attributes[key](self) or nil + +local datetime_index = function(self, key) + return datetime_index_handlers[key] ~= nil and + datetime_index_handlers[key](self) or nil end ----------------------------------------- I mean we still carefully check existence of an asked key, and return nil if there is no such key at all. Is it much different than what you've meant? : : : > + timestamp = function(self) : > + return tonumber(self.secs) + self.nsec / 1e9 : > + end, : > + nanoseconds = function(self) : > + return tonumber(self.secs * 1e9 + self.nsec) : > + end, : > + microseconds = function(self) : > + return tonumber(self.secs * 1e6 + self.nsec / 1e3) : > + end, : > + milliseconds = function(self) : > + return tonumber(self.secs * 1e3 + self.nsec / 1e6) : > + end, : > + seconds = function(self) : > + return tonumber(self.secs) + self.nsec / 1e9 : > + end, : > + minutes = function(self) : > + return (tonumber(self.secs) + self.nsec / 1e9) / 60 : > + end, : > + hours = function(self) : > + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) : > + end, : > + days = function(self) : > + return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / : 24 : > + end, : > + } : > + return attributes[key] ~= nil and attributes[key](self) or nil : > +end : > + : > local datetime_mt = { : > -- __tostring = datetime_tostring, : > __serialize = datetime_serialize, Thanks, Timur
Hi there! : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time : intervals support : : Thanks for your patch. : : I wrote several comments below. : : : However it makes me think that such approach will work quite slow since : : all functions is implemented in Lua, all arithmetic is in Lua. I'm not that pessimistic here :) [Given the fact that I'd remove Out of a loops all handy handlers table creation] But we need to measure it first. : : ... : > @@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL : > : > local datetime_t = ffi.typeof('struct datetime_t') : > local interval_t = ffi.typeof('struct datetime_interval_t') : > +ffi.cdef [[ : > + struct t_interval_months { : > + int m; : > + }; : > + : > + struct t_interval_years { : > + int y; : > + }; : > +]] : > +local interval_months_t = ffi.typeof('struct t_interval_months') : > +local interval_years_t = ffi.typeof('struct t_interval_years') : > + : > +local function is_interval(o) : > + return ffi.istype(interval_t, o) or : > + ffi.istype(interval_months_t, o) or : > + ffi.istype(interval_years_t, o) : > +end : > + : : It will throw for non-cdata values: : : tarantool> ffi.istype(interval_t, o) : --- : - error: 'bad argument #1 to ''?'' (C type expected, got nil)' : ... : : tarantool> ffi.istype(interval_t, 123) : --- : - error: 'bad argument #1 to ''?'' (C type expected, got nil)' : ... Yes, thanks for pin-pointing it! I've modified it this way to avoid such errors: ------------------------------------------------ @@ -182,15 +181,19 @@ local interval_months_t = ffi.typeof('struct t_interval_months') local interval_years_t = ffi.typeof('struct t_interval_years') local function is_interval(o) - return ffi.istype(interval_t, o) or - ffi.istype(interval_months_t, o) or - ffi.istype(interval_years_t, o) + return type(o) == 'cdata' and + (ffi.istype(interval_t, o) or + ffi.istype(interval_months_t, o) or + ffi.istype(interval_years_t, o)) end local function is_datetime(o) - return ffi.istype(o, datetime_t) + return type(o) == 'cdata' and ffi.istype(o, datetime_t) end +local function is_date_interval(o) + return is_datetime(o) or is_interval(o) +end local function interval_new() local interval = ffi.new(interval_t) ------------------------------------------------ : > @@ -224,19 +323,123 @@ local function interval_serialize(self) : > return { secs = self.secs, nsec = self.nsec } : > end : > : > +local function local_rd(o) : > + return math.floor(tonumber(o.secs / SECS_PER_DAY)) + : DT_EPOCH_1970_OFFSET : > +end : > + : > +local function local_dt(o) : > + return cdt.dt_from_rdn(local_rd(o)) : > +end : > + : > +local function _normalize_nsec(secs, nsec) : > + if nsec < 0 then : > + secs = secs - 1 : > + nsec = nsec + NANOS_PER_SEC : > + elseif nsec >= NANOS_PER_SEC then : > + secs = secs + 1 : > + nsec = nsec - NANOS_PER_SEC : > + end : > + return secs, nsec : > +end : > + : > +-- addition or subtraction from date/time of a given interval : > +-- described via table direction should be +1 or -1 : > +local function interval_increment(self, o, direction) : > + assert(direction == -1 or direction == 1) : > + check_date(self, "interval_increment(date, object, -+1)") : > + assert(type(o) == 'table') : > + : > + local ym_updated = false : > + local dhms_updated = false : > + : > + local dt = local_dt(self) : > + local secs, nsec : > + secs, nsec = self.secs, self.nsec : > + : > + for key, value in pairs(o) do : > + local handlers = { : : The same as in one previous patch. It's too expensive to recreate table : and functions for each simple : : action and for each iteration loop. I've removed creation of a table, which is invariant in the loop, to outside of loop statement. Let me measure how it much would different than series of iffs. --------------------------------------------------------- @@ -377,52 +382,52 @@ local function interval_increment(self, o, direction) local secs, nsec secs, nsec = self.secs, self.nsec + local handlers = { + years = function(k, v) + check_range(v, {0, 9999}, k) + dt = builtin.dt_add_years(dt, direction * v, builtin.DT_LIMIT) + ym_updated = true + end, + + months = function(k, v) + check_range(v, {0, 12}, k) + dt = builtin.dt_add_months(dt, direction * v, builtin.DT_LIMIT) + ym_updated = true + end, + + weeks = function(k, v) + check_range(v, {0, 52}, k) + secs = secs + direction * 7 * v * SECS_PER_DAY + dhms_updated = true + dhms_updated = true + end, + + days = function(k, v) + check_range(v, {0, 31}, k) + secs = secs + direction * v * SECS_PER_DAY + dhms_updated = true + end, + + hours = function(k, v) + check_range(v, {0, 23}, k) + secs = secs + direction * 60 * 60 * v + dhms_updated = true + end, + + minutes = function(k, v) + check_range(v, {0, 59}, k) + secs = secs + direction * 60 * v + end, + + seconds = function(k, v) + check_range(v, {0, 60}, k) + local s, frac = seconds_fraction(v) + secs = secs + direction * s + nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds + dhms_updated = true + end, + } for key, value in pairs(o) do - local handlers = { - years = function(v) - check_range(v, {0, 9999}, key) - dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT) - ym_updated = true - end, - - months = function(v) - check_range(v, {0, 12}, key) - dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT) - ym_updated = true - end, - - weeks = function(v) - check_range(v, {0, 52}, key) - secs = secs + direction * 7 * v * SECS_PER_DAY - dhms_updated = true - end, - - days = function(v) - check_range(v, {0, 31}, key) - secs = secs + direction * v * SECS_PER_DAY - dhms_updated = true - end, - - hours = function(v) - check_range(v, {0, 23}, key) - secs = secs + direction * 60 * 60 * v - dhms_updated = true - end, - - minutes = function(v) - check_range(v, {0, 59}, key) - secs = secs + direction * 60 * v - end, - - seconds = function(v) - check_range(v, {0, 60}, key) - local s, frac = seconds_fraction(v) - secs = secs + direction * s - nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds - dhms_updated = true - end, - } - handlers[key](value) + handlers[key](key, value) end secs, nsec = _normalize_nsec(secs, nsec) --------------------------------------------------------- : : > + years = function(v) : > + assert(v > 0 and v < 10000) : > + dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT) : > + ym_updated = true : > + end, : > + : > + months = function(v) : > + assert(v > 0 and v < 13 ) : > + dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT) : > + ym_updated = true : > + end, : > + : > + weeks = function(v) : > + assert(v > 0 and v < 32) : > + secs = secs + direction * 7 * v * SECS_PER_DAY : > + dhms_updated = true : > + end, : > + : > + days = function(v) : > + assert(v > 0 and v < 32) : > + secs = secs + direction * v * SECS_PER_DAY : > + dhms_updated = true : > + end, : > + : > + hours = function(v) : > + assert(v >= 0 and v < 24) : > + secs = secs + direction * 60 * 60 * v : > + dhms_updated = true : > + end, : > + : > + minutes = function(v) : > + assert(v >= 0 and v < 60) : > + secs = secs + direction * 60 * v : > + end, : > + : > + seconds = function(v) : > + assert(v >= 0 and v < 61) : > + local s, frac : > + frac = v % 1 : > + if frac > 0 then : > + s = v - (v % 1) : > + else : > + s = v : > + end : > + secs = secs + direction * s : > + nsec = nsec + direction * frac * 1e9 -- convert fraction : to nanoseconds : > + dhms_updated = true : > + end, : > + } : > + handlers[key](value) : > + end : > + : > + secs, nsec = _normalize_nsec(secs, nsec) : > + : > + -- .days, .hours, .minutes, .seconds : > + if dhms_updated then : > + self.secs = secs : > + self.nsec = nsec : > + end : > + : > + -- .years, .months updated : > + if ym_updated then : > + self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * : SECS_PER_DAY + : > + secs % SECS_PER_DAY : > + end : > + : > + return self : > +end : > + : : tarantool> return require('datetime').now() + 1 : --- : - error: '[string "return require(''datetime'').now() + 1"]:1: Usage: : datetime:__add(interval)' : ... : : : Looks a bit confusing. User doesn't know about metamethods. Partially agreed. Though I assume that users of this code will be educated enough. But in any case, what would be your suggested error message here? Thanks, Timur
Hello Oleg, : From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: : unixtime, timestamp setters in datetime.lua : : Thanks for your patch. : : The comment is primarly the same as in previous patch. : : Why do you use such slow approach and creates redundant tables and : function (temporary metatable in fact) : : to use it only once? Please fix it. In previous emails I've shown how I'd deal with loop invariants. Should avoid this redundant temp objects problem. : : : Unixtime and timestamp is great but they loss precision. I think it : should be possible : : go get timestamp with nanoseconds precision since datetime has : nanoseconds precision. I'm not quite get it. Do you want to have nanoseconds exported via newindex? For direct assignments? It's not useful because of available .nsec field. Or may be you meant something else? : : : Also still it's hard how to convert timestamp to datetime value back. Timestamp is something secs.nsec, what looks complicated? I'm confused. Could you please elaborate? : : : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote: : > * implemented proper range checks for date attributes values; : > : > * created `.unixtime` attribute, which is alias to `.secs`, : > with corresponding setter/getter; : > : > * similarly to `unixtime`, created virtual `timestamp` attribute : > setter. Which is a convenient way to simultaneously assign : > unixtime (seconds since epoch) and nanoseconds : > : > Part of #5941 Thanks, Timur
Thanks for your answers. On 30.07.2021 22:00, Timur Safin wrote: > Ok, here is a selection list which I've seen: > - decimal is in LuaC; > - uuid is in ffi; > > It looked equally acceptable and I've selected the one which is > more convenient - via ffi. Now, you've pointed me to the probable > performance issues we have with ffi, which are not there with LuaC. > I believe they have been fixed with that Mike fix for NYI in > structure allocations (after Sergos dirty fix) and once we will land > this patch to our local fork those performance issues for structures > allocations will gone. > > In any case, 1st version of module is not yet a proper moment to start > to worry about performance, more rather about clarity and quality of API. > We need to return to benchmarking of this code soon after code freeze > lifted. I don't know why uuid is implemented via FFI. Uuid as module was introduced at least 8 yeard ago. Probably there were supporters of the theory that FFI/LuaJIT implementation could be fast. But currently we can see something like https://github.com/luafun/luafun/issues/31#issuecomment-277508076 and rewrite netbox in C. And it's not about patches in LuaJIT about structure initialization. You can easily cherry-pick it into our fork and show that FFI is really better than Lua C API. It's about an approach. We saw benchmarks when decimal was introduced but currently we didn't see anything. > : > + local y, M, d, ymd > : > + y, M, d, ymd = 0, 0, 0, false > : > + > : > + local h, m, s, frac, hms > : > + h, m, s, frac, hms = 0, 0, 0, 0, false > : > + > : > + local dt = 0 > : > + > : > + for key, value in pairs(o) do > : > + local handlers = { > : > : Here you recreate handlers for each iteration of the loop. I think it > : should be reworked. > > Yes, let me check how better will be if this handlers array will be invariant > in the loop, and not created each iteration. > > I still prefer this tag to function approach because of a better clarity, > but I need to verify in benchmark whether it provides acceptable performance. > > I do not expect thought that this attribute object creation approach would become > a bootleneck in any case and might be on a critical path anywhere (I'd rather expect > it for datetime objects created after parsing of massive bunch of log text), > but need to look into timings we would have with invariant table. > > : > : Currenlty it's quite slow I think even if-elseif-else branches will work > : faster without > : > : creating redundant GC objects. > > Will see ... I'll simplify your task a bit: local clock = require('clock') local mt_fun = function(_, key) local mt = { ['key'] = function() return 'value' end } return mt[key] ~= nil and mt[key]() end local mt_table = { ['key'] = function() return 'value' end } local t1 = setmetatable({}, {__index = mt_fun}) local t2 = setmetatable({}, {__index = mt_table}) local _ local start = clock.time() for _ = 1, 1e6 do _ = t1['key'] end print('function', clock.time() - start) collectgarbage() collectgarbage() local start = clock.time() for _ = 1, 1e6 do _ = t2['key'] end print('table', clock.time() - start) tarantool test.lua function 0.14332509040833 table 0.0002901554107666 The difference about x500 times on my Mac. > : > +end > : > + > : > +local function strftime(fmt, o) > : > + assert(ffi.typeof(o) == datetime_t) > : > + local sz = 50 > : Why 50? > > Good question, for ISO-8601 formats, for normal date range > (from 0000 till 9999 years) it's more like closer to 40, but > well, there will be various timezones and all crazy kinds of > input format combinations, so we need to either be precautious > with larger sizes allocations. On the one side you are right. But if user want to format string with length more than 50 it will be silently truncated. ``` tarantool> datetime.strftime('%d' .. string.rep('content', 50), datetime.new()) --- - 01contentcontentcontentcontentcontentcontentconten ... ``` I'm not sure that anybody will use strftime in such way but it's strange. Maybe you could use preallocated buffer for short strings and allocate huge if it's not enough. Some kind of optimistic approach. > Or probably rely on glibc convention (which is apparently not in > POSIX/SUSV) with 2 passes approach: > 1. call with NULL so it will return size of required buffer; > 2. then, after allocation, with the adjusted allocated buffer; > > https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html > https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html > > Something like that (beware of inline patch): > > ---------------------------------------------- > local function strftime(fmt, o) > check_date(o, "datetime.strftime(fmt, date)") > - local sz = 50 > - local buff = ffi.new('char[?]', sz) > local p_tm = datetime_to_tm_ptr(o) > - native.strftime(buff, sz, fmt, p_tm) > + local sz = builtin.strftime(nil, 1024, fmt, p_tm) > + local buff = ffi.new('char[?]', sz + 1) > + builtin.strftime(buff, sz, fmt, p_tm) > return ffi.string(buff) > end > ---------------------------------------------- > > : > + local buff = ffi.new('char[?]', sz) > : > + local p_tm = datetime_to_tm_ptr(o) > : > + native.strftime(buff, sz, fmt, p_tm) > : > + return ffi.string(buff) > : > +end > : > + > : > +-- strftime may be redirected to datetime:fmt("format") > : > +local function datetime_fmt() > : > +end > : > + > : > + > : > +ffi.metatype(interval_t, interval_mt) > : > +ffi.metatype(datetime_t, datetime_mt) > : > + > : > +return setmetatable( > : > + { > : > + datetime = datetime_new, > : > + interval = interval_new, > : > + > : > + parse = parse_str, > : > + parse_date = parse_date, > : > + parse_time = parse_time, > : > + parse_zone = parse_zone, > : > + fmt = datetime_fmt, > : > + > : > + now = local_now, > : > + -- strptime = strptime; > : It should be dropped if you don't need it. > : > + strftime = strftime, > : > + asctime = asctime, > : > + ctime = ctime, > : > + }, { > : > + __call = function(self, ...) return datetime_from(...) end > : > + } > : > +) > > Thanks, > Timur >
Thanks for your answers. See my answer below. On 31.07.2021 00:48, Timur Safin wrote: > : > : > : Also I think that encoding and type system changes should live in > : separate patches. > > I'm not 100% sure that understand this your complain. Could you please > clarify? > Hm... Probably I missed something. As I see now decimal encoding to msgpack and json/yaml... serializations are in the one patch (https://github.com/tarantool/tarantool/commit/485439e33196e26d120e622175f88b4edc7a5aa1). Seems my comment in not relevant, sorry.
Thanks for your reply. See my answer below.
On 31.07.2021 01:18, Timur Safin wrote:
> Hello Oleg,
>
>
> : From: Oleg Babin<olegrok@tarantool.org>
> : Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime:
> : datetime comparison for indices
> :
> : Thanks for your patch. One question below.
> :
> :
> : Also update with arithmetic operations doesn't work.
> :
> :
> : ```
> :
> : tarantool> box.space.dt:update({v}, {{'+', 2, 3}})
> : ---
> : - error: 'Argument type in operation ''+'' on field 2 does not match
> : field type: expected
> : a number'
> : ...
> :
> : tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}})
> : ---
> : - error: unsupported Lua type 'cdata'
> : ...
> :
> : ```
>
> Oh, shit, I didn't take those operations into considerations.
> Do they work similarly with decimal?
Looks like so. Moreover it seems we should support intervals as separate
type.
Currently the single way to update is get + sum + put/update (with =).
But I suppose native update will be needed anyway. At lest for SQL.
Will be it able to perform some operations with intervals and timestamps?
Thanks for your reply. See my comment below.
On 31.07.2021 01:30, Timur Safin wrote:
> Hello Oleg again!
>
> :
> :
> : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> : > * introduced a set of calculated attributes to data object, e.g.:
> : > - timestamp, seconds, microseconds, minute, or hours
> : >
> : > Part of #5941
> : > ---
> : > src/lua/datetime.lua | 51 ++++++++++++++++++++++------------
> : > test/app-tap/datetime.test.lua | 17 +++++++++++-
> : > 2 files changed, 50 insertions(+), 18 deletions(-)
> : >
> : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> : > index 9ec06d8d8..7a208cef9 100644
> : > --- a/src/lua/datetime.lua
> : > +++ b/src/lua/datetime.lua
> : > @@ -224,6 +224,36 @@ local function interval_serialize(self)
> : > return { secs = self.secs, nsec = self.nsec }
> : > end
> : >
> : > +local datetime_index = function(self, key)
> : > + local attributes = {
> :
> : This change looks like pessimization because currently we need to
> : recreate "attributes" table
> :
> : for each function call. I suggest to use `attributes` as `__index` but
> : not such function.
>
> What would you say if I simply precache this attributes table in local,
> and then proceed this way?
> -----------------------------------------
> -local datetime_index = function(self, key)
> - local attributes = {
> +local datetime_index_handlers = {
> unixtime = function(self)
> return self.secs
> end,
> +
> timestamp = function(self)
> return tonumber(self.secs) + self.nsec / 1e9
> end,
> +
> nanoseconds = function(self)
> return self.secs * 1e9 + self.nsec
> end,
> +
> microseconds = function(self)
> return self.secs * 1e6 + self.nsec / 1e3
> end,
> +
> milliseconds = function(self)
> return self.secs * 1e3 + self.nsec / 1e6
> end,
> +
> seconds = function(self)
> return tonumber(self.secs) + self.nsec / 1e9
> end,
> +
> minutes = function(self)
> return (tonumber(self.secs) + self.nsec / 1e9) / 60
> end,
> +
> hours = function(self)
> return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
> end,
> +
> days = function(self)
> return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
> end,
> +
> add = function(self)
> return function(self, o)
> return interval_increment(self, o, 1)
> end
> end,
> +
> sub = function(self)
> return function(self, o)
> return interval_increment(self, o, -1)
> end
> end,
> }
> - return attributes[key] ~= nil and attributes[key](self) or nil
> +
> +local datetime_index = function(self, key)
> + return datetime_index_handlers[key] ~= nil and
> + datetime_index_handlers[key](self) or nil
> end
> -----------------------------------------
>
> I mean we still carefully check existence of an asked key, and return nil if
> there is no such key at all. Is it much different than what you've meant?
>
It's again about temporary tables. I gave a benchmark in one of the
previous emails.
After changes it shoud be better. Thanks.
On 31.07.2021 01:58, Timur Safin wrote:
> :
> : tarantool> return require('datetime').now() + 1
> : ---
> : - error: '[string "return require(''datetime'').now() + 1"]:1: Usage:
> : datetime:__add(interval)'
> : ...
> :
> :
> : Looks a bit confusing. User doesn't know about metamethods.
>
> Partially agreed. Though I assume that users of this code will
> be educated enough. But in any case, what would be your suggested
> error message here?
>
In fact I don't know exactly.
But decimal shows something like:
```
tarantool> decimal.new(1) + 'abc'
---
- error: '[string "return decimal.new(1) + ''abc''"]:1: incorrect value
to convert
to decimal as 2 argument'
...
```
On 31.07.2021 02:11, Timur Safin wrote:
> Hello Oleg,
>
> :
> :
> : Unixtime and timestamp is great but they loss precision. I think it
> : should be possible
> :
> : go get timestamp with nanoseconds precision since datetime has
> : nanoseconds precision.
>
> I'm not quite get it. Do you want to have nanoseconds exported via newindex?
> For direct assignments? It's not useful because of available .nsec field.
> Or may be you meant something else?
>
> :
> :
> : Also still it's hard how to convert timestamp to datetime value back.
>
> Timestamp is something secs.nsec, what looks complicated? I'm confused.
> Could you please elaborate?
I expected something like.
tarantool> clock.time64()
---
- 1627712632369020000
...
This timestamp is in nanoseconds. It's unsigned long long value.
I tried to do something for datetime...
sec * 1e9 + nsec, but sec is null while nsec is not:
```
tarantool> dt.sec
---
- null
...
```
timestamp + nsec works in quite strage way:
```
tarantool> dt.timestamp * (1e9 * 1ULL) + dt.nsec
---
- 1627712485108074000
...
```
(I need to cast to ULL here to avoid precision lost).
But maybe my question is not relevant. There is a way to get nanoseconds
timestamp.
I just need to write some simple helper for my app. It's ok.
: From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time : intervals support : : : On 31.07.2021 01:58, Timur Safin wrote: : > : : > : tarantool> return require('datetime').now() + 1 : > : --- : > : - error: '[string "return require(''datetime'').now() + 1"]:1: Usage: : > : datetime:__add(interval)' : > : ... : > : : > : : > : Looks a bit confusing. User doesn't know about metamethods. : > : > Partially agreed. Though I assume that users of this code will : > be educated enough. But in any case, what would be your suggested : > error message here? : > : In fact I don't know exactly. : : But decimal shows something like: : : ``` : : tarantool> decimal.new(1) + 'abc' : --- : - error: '[string "return decimal.new(1) + ''abc''"]:1: incorrect value : to convert : to decimal as 2 argument' : ... : : ``` Very good, indeed. So the trick is to say that type is unexpected and optionally show of which value it's camplaining about. Like: Expected datetime or interval argument, but received '%s'.
: From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> : Subject: Re: [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the : build : : Hi! Thanks for the patch! : : On 28.07.2021 12:34, Timur Safin wrote: : > * Integrated chansen/c-dt parser as 3rd party module to the : > Tarantool cmake build process. : > * Points to tsafin/c-dt instead iof original chansen/c-dt to : : iof -> of. But is it even true? Below I see Thanks! : : url = https://github.com/tarantool/c-dt.git : : not tsafin/c-dt. We have relocated repository to Tarantool organization since then. : : After build I get in git status this: : : Changes not staged for commit: : (use "git add <file>..." to update what will be committed) : (use "git restore <file>..." to discard changes in working : directory) : (commit or discard the untracked or modified content in submodules) : modified: third_party/c-dt (modified content, untracked : content) : : It says: : : modified: Makefile : : and some untracked files: : : CMakeFiles/ : cmake_install.cmake : libcdt.a : : Why is even Makefile added to the repository if we are : using cmake anyway? Yeah, sounds like a small cmake configuration annoyance. Cmake here generates their artifacts just right under sources, and not under ${CMAKE_CURRENT_BINARY_DIR}. Will look how we could improve things, making it cleaner. Thanks, Timur
: > : From: Oleg Babin<olegrok@tarantool.org> : > : Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: : > : datetime comparison for indices : > : : > : Thanks for your patch. One question below. : > : : > : : > : Also update with arithmetic operations doesn't work. : > : : > : : > : ``` : > : : > : tarantool> box.space.dt:update({v}, {{'+', 2, 3}}) : > : --- : > : - error: 'Argument type in operation ''+'' on field 2 does not match : > : field type: expected : > : a number' : > : ... : > : : > : tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}}) : > : --- : > : - error: unsupported Lua type 'cdata' : > : ... : > : : > : ``` : > : > Oh, shit, I didn't take those operations into considerations. : > Do they work similarly with decimal? : : Looks like so. Moreover it seems we should support intervals as separate : type. : : Currently the single way to update is get + sum + put/update (with =). : : But I suppose native update will be needed anyway. At lest for SQL. : : Will be it able to perform some operations with intervals and timestamps? The original plan was to not persist intervals in the 1st iteration of implementation and store only full datetime values. Even for SQL I did not plan to allow schemas with interval as field type. INTERVALs in SQL expected to be present at run-time only at the Expressions, but then persisted as calculated datetime values. But, those box updates with intervals looks interesting, and we have to support, eventually. I don't know though whether it would require full interval support or the current partial will be enough? In any case - smells like we need to open ticket for that. Could you please do that with more details? Thanks, Timur
: From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: : unixtime, timestamp setters in datetime.lua : : : On 31.07.2021 02:11, Timur Safin wrote: : > Hello Oleg, : > : > : : > : : > : Unixtime and timestamp is great but they loss precision. I think it : > : should be possible : > : : > : go get timestamp with nanoseconds precision since datetime has : > : nanoseconds precision. : > : > I'm not quite get it. Do you want to have nanoseconds exported via : newindex? : > For direct assignments? It's not useful because of available .nsec field. : > Or may be you meant something else? : > : > : : > : : > : Also still it's hard how to convert timestamp to datetime value back. : > : > Timestamp is something secs.nsec, what looks complicated? I'm confused. : > Could you please elaborate? : : I expected something like. : : tarantool> clock.time64() : --- : - 1627712632369020000 : ... : : : This timestamp is in nanoseconds. It's unsigned long long value. : : I tried to do something for datetime... : : sec * 1e9 + nsec, but sec is null while nsec is not: : : ``` : : tarantool> dt.sec : --- : - null : ... : : ``` : : : timestamp + nsec works in quite strage way: : : ``` : : tarantool> dt.timestamp * (1e9 * 1ULL) + dt.nsec : --- : - 1627712485108074000 : ... : : ``` : : (I need to cast to ULL here to avoid precision lost). : : But maybe my question is not relevant. There is a way to get nanoseconds : timestamp. : : I just need to write some simple helper for my app. It's ok. There are nanoseconds / microseconds / milliseconds getters in datetime object, which operate on the original uint64_t. Do you want them to make read-write, not read-only? tarantool> T = date '1970-01-01' --- ... tarantool> T.timestamp --- - 0 ... tarantool> T.nanoseconds --- - 0 ... tarantool> T2000 = date '2000-01-01' --- ... tarantool> T.nanoseconds --- - 0 ... tarantool> T2000.timestamp --- - 946684800 ... tarantool> T2000.nanoseconds --- - 946684800000000000 ... tarantool> T2000.secs --- - 946684800 ... tarantool> ffi = require 'ffi' --- ... tarantool> ffi.typeof(T2000.nanoseconds) --- - ctype<int64_t> ... Regards, Timur
: From: Oleg Babin <olegrok@tarantool.org> : Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in : module datetime : : Thanks for your answers. : : On 30.07.2021 22:00, Timur Safin wrote: : > Ok, here is a selection list which I've seen: : > - decimal is in LuaC; : > - uuid is in ffi; : > : > It looked equally acceptable and I've selected the one which is : > more convenient - via ffi. Now, you've pointed me to the probable : > performance issues we have with ffi, which are not there with LuaC. : > I believe they have been fixed with that Mike fix for NYI in : > structure allocations (after Sergos dirty fix) and once we will land : > this patch to our local fork those performance issues for structures : > allocations will gone. : > : > In any case, 1st version of module is not yet a proper moment to start : > to worry about performance, more rather about clarity and quality of API. : > We need to return to benchmarking of this code soon after code freeze : > lifted. : : I don't know why uuid is implemented via FFI. Uuid as module was : introduced at least 8 yeard ago. : : Probably there were supporters of the theory that FFI/LuaJIT : implementation could be fast. : : But currently we can see something like : https://github.com/luafun/luafun/issues/31#issuecomment-277508076 : : and rewrite netbox in C. "I feel your pain, bro!" :) [here follows seemingly irrelevant speech, but actually it's on topic] I knew you have a plenty of LuaJIT experience, and sometimes, you might want to throw it out altogether. All that pain of debugging JIT performance issues. Woodoo and black magics of LuaJIT debugging. General unpredictability of Mike responses. And even lack of convenient tooling here and there. I do understand how all this could exhaust eventually, and make to loss of believe in LuaJIT usefulness. But we should not. We have skilled LuaJIT team here, which could handle all sorts of problems. We have huge amount of Lua code, and have no any reason to start the process of getting rid of it. Yes, there are cases which might hit some performance degradations, but usually there are reasonable workarounds. (Whom am I teaching here? You are much more aware of all these tricks than I am!) Having our background and grownup skillset, there are actually no much reasons to try to drastically reduce amount of LuaJIT usage, because we continue to be used as LuaJIT-based application server for all foreseeable future. [/end of seemingly irrelevant speech] To make this story short, I believe there is no much point to start rewrite of this module in LuaC before we have it published and landed to the Tarantool code-base. We need to make it feature complete first, then "blazingly fast". In that order. : : And it's not about patches in LuaJIT about structure initialization. You : can easily cherry-pick it into : : our fork and show that FFI is really better than Lua C API. It's about : an approach. We saw benchmarks : : when decimal was introduced but currently we didn't see anything. : : > : > + local y, M, d, ymd : > : > + y, M, d, ymd = 0, 0, 0, false : > : > + : > : > + local h, m, s, frac, hms : > : > + h, m, s, frac, hms = 0, 0, 0, 0, false : > : > + : > : > + local dt = 0 : > : > + : > : > + for key, value in pairs(o) do : > : > + local handlers = { : > : : > : Here you recreate handlers for each iteration of the loop. I think it : > : should be reworked. : > : > Yes, let me check how better will be if this handlers array will be : invariant : > in the loop, and not created each iteration. : > : > I still prefer this tag to function approach because of a better clarity, : > but I need to verify in benchmark whether it provides acceptable : performance. : > : > I do not expect thought that this attribute object creation approach would : become : > a bootleneck in any case and might be on a critical path anywhere (I'd : rather expect : > it for datetime objects created after parsing of massive bunch of log : text), : > but need to look into timings we would have with invariant table. : > : > : : > : Currenlty it's quite slow I think even if-elseif-else branches will work : > : faster without : > : : > : creating redundant GC objects. : > : > Will see ... : : I'll simplify your task a bit: : : local clock = require('clock') : : local mt_fun = function(_, key) : local mt = { : ['key'] = function() : return 'value' : end : } : return mt[key] ~= nil and mt[key]() : end : : local mt_table = { : ['key'] = function() : return 'value' : end : } : : local t1 = setmetatable({}, {__index = mt_fun}) : local t2 = setmetatable({}, {__index = mt_table}) : local _ : : local start = clock.time() : for _ = 1, 1e6 do : _ = t1['key'] : end : print('function', clock.time() - start) : : collectgarbage() : collectgarbage() : : local start = clock.time() : for _ = 1, 1e6 do : _ = t2['key'] : end : print('table', clock.time() - start) : : : tarantool test.lua : function 0.14332509040833 : table 0.0002901554107666 : : : The difference about x500 times on my Mac. Thanks! That's convincing! Thanks for this experiment, much appreciated! Interesting observation, though, that once we remove local mt table creation from inside of mt_fun to the module scope, making it invariant for the cycle. The difference is not that much huge. Debug build on WSL:: - your original version: ``` 19:22 $ ../build/src/tarantool bench-index.lua function 0.16711115837097 table 0.00081968307495117 ``` - modified version with invariant mt table: ``` 19:25 $ ../build/src/tarantool bench-index.lua function 0.00028538703918457 table 0.00028634071350098 ``` ```lua local clock = require('clock') local mt = { ['key'] = function() return 'value' end } local mt_fun = function(_, key) -- return mt[key]() return mt[key] ~= nil and mt[key]() end local mt_table = { ['key'] = function() return 'value' end } local t1 = setmetatable({}, {__index = mt_fun}) local t2 = setmetatable({}, {__index = mt_table}) local _ local start = clock.time() for _ = 1, 1e6 do _ = t1['key'] end print('function', clock.time() - start) collectgarbage() collectgarbage() local start = clock.time() for _ = 1, 1e6 do _ = t2['key'] end print('table', clock.time() - start) ``` So once again we have learned that it's not a good idea to create invariant table inside of loop, and simple loop invariant removal might change things dramatically. Thanks for the lesson! : : > : > +end : > : > + : > : > +local function strftime(fmt, o) : > : > + assert(ffi.typeof(o) == datetime_t) : > : > + local sz = 50 : > : Why 50? : > : > Good question, for ISO-8601 formats, for normal date range : > (from 0000 till 9999 years) it's more like closer to 40, but : > well, there will be various timezones and all crazy kinds of : > input format combinations, so we need to either be precautious : > with larger sizes allocations. : : On the one side you are right. But if user want to format string with : length more than 50 : : it will be silently truncated. No, no, no, please see my 2 pass variant suggested below. We will ask `strftime` first to calculate the length for us (passing NULL as buffer pointer), then allocate, and fill the allocated buffer with content the 2nd pass. : : : ``` : : tarantool> datetime.strftime('%d' .. string.rep('content', 50), : datetime.new()) : --- : - 01contentcontentcontentcontentcontentcontentconten : ... : : ``` : : I'm not sure that anybody will use strftime in such way but it's strange. : : Maybe you could use preallocated buffer for short strings and allocate : huge if : : it's not enough. Some kind of optimistic approach. : : : > Or probably rely on glibc convention (which is apparently not in : > POSIX/SUSV) with 2 passes approach: : > 1. call with NULL so it will return size of required buffer; : > 2. then, after allocation, with the adjusted allocated buffer; : > : > https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar- : Time.html : > https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html : > : > Something like that (beware of inline patch): : > : > ---------------------------------------------- : > local function strftime(fmt, o) : > check_date(o, "datetime.strftime(fmt, date)") : > - local sz = 50 : > - local buff = ffi.new('char[?]', sz) : > local p_tm = datetime_to_tm_ptr(o) : > - native.strftime(buff, sz, fmt, p_tm) : > + local sz = builtin.strftime(nil, 1024, fmt, p_tm) : > + local buff = ffi.new('char[?]', sz + 1) : > + builtin.strftime(buff, sz, fmt, p_tm) : > return ffi.string(buff) : > end : > ---------------------------------------------- Thanks, Timur
Hi! Thanks for the fixes! Although I would prefer you sending updates right under each non-trivial comment and pushing the changes. Otherwise I can't check if you really fixed the comments and if the fixes are better than it was before. On 30.07.2021 17:39, Timur Safin wrote: > : From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> > : Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime > : > : Thanks for the patch! > : > : This is a bad question to ask now, but why did you choose a dead library > : from 2015 with just 25 stars, without any tests, and without any README as > : a basic block for our datetime types? Is this the correct link I am > : using - https://github.com/chansen/c-dt? In all your responses the previous email's text is prefixed with ':' sign unlike it is done in other people's responses. In my email client it turns the email into a mess where I can't see where is the new text. Could you possibly start using '>' prefix? > : > + */ > : > +struct datetime_t { > : > + int64_t secs; ///< seconds since epoch > : > + int32_t nsec; ///< nanoseconds if any > : > + int32_t offset; ///< offset in minutes from GMT > : > : 6. We never use // nor /// nor ///< nor we write comments > : on the same line as the code (except for old legacy code we > : inherited from sqlite and some super old tarantool code. > > BTW, why? Single line comment // is the legal C construction > since C99 (and always have been acceptable by Gnu C and major > other compilers). > > Also did you know that ///< is doxygen construction for > documentation of a struct member? (Asking, just to make sure) The fact that something is a legal C construction does not mean it fits our code style. For doxygen in our rules we have /** as a comment opening. > : > : The same for datetime_interval_t. > : > : > +}; > : > + > : > +/** > : > + * Date/time delta structure > : > + */ > : > +struct datetime_interval_t { > : > + int64_t secs; ///< relative seconds delta > : > + int32_t nsec; ///< nanoseconds delta > : > +}; > : > + > : > : 7. Why do you need this file? It is not included anywhere. > : And you don't need to define the structs in C if you are > : using them in Lua only. You can just define them in Lua > : using ffi.cdef like it is done in some other places. > > I'll need to use it in C very soon, once I get to SQL parser > INTERVAL 'xx' support. You will need to introduce it in the same commit in which you will use it. We do not split patches into commits just to make them smaller. The split needs to make sense in terms of atomicity. > : > +local datetime_mt = { > : > + -- __tostring = datetime_tostring, > : > : 16. Why is it commented out? Is it even tested then? > > It's implemented in the later patch in patchset series. Well, then it should not be here in this commit. > : > + microseconds = function(self) > : > + return tonumber(self.secs*1e6 + self.nsec*1e3) > : > : 18. Please, add whitespaces around binary operators. > > Done! It was already such in the later commits which were part > of in patchset. [Yes, apparently I need to squash few more commits, > as Oleg Babin has asked elsewhere] It is not just about squashes. It is only about atomicity (and as a consequence - review simplicity). You can't just squash all the commits into one. You will need to change the older commits to make them right, then rebase the newer commits. > : > : 19. Why do you do tonumber() for all results? You are loosing > : precision for big values. The same in all the places where you > : use tonumber(). > > To be able to use it in places which expect numbers like %d in > string.format. That does not look like a good reason - you loose precision just to simplify the logs. Better make the logs use tonumber() than make the functions return imprecise results. > Though at the end of day I've came up to the version > where I convert to Lua numbers not all (potentially fraction point) > values, but with exception of nanoseconds / microseconds / milliseconds > (which return integer values). > > : > : > + end, > : > + > : > : 21. Extra empty line. > > Yup, but later, well, you know ... No, I couldn't understand. What 'later'? > : > : > +} > : > + > : > +local interval_mt = { > : > + -- __tostring = interval_tostring, > : > : 22. Ditto. Why is it commented out and not tested? > > Tostring conversion was implemented later in a few > commits. One for datetime stringization, and another > one for interval. Then it should be in the corresponding commit. > : > : > + __serialize = interval_serialize, > : > + __eq = datetime_eq, > : > + __lt = datetime_lt, > : > + __le = datetime_le, > : > +} > : > + > : > +local function datetime_new_raw(secs, nsec, offset) > : > + local dt_obj = ffi.new(datetime_t) > : > + dt_obj.secs = secs > : > + dt_obj.nsec = nsec > : > + dt_obj.offset = offset > : > + return dt_obj > : > +end > : > + > : > +local function local_rd(o) > : > : 23. What is 'rd' and 'dt'? > > `dt` is type - data type from c-dt (alias to signed integer). > Which shows a number of dates since Rata Die date (0001-01-01). > It's signed 32-bit value, thus could represent huge range before > 0001-01-01 as well. You didn't say what is 'rd'. > : > : > + end, > : > : 27. The usage of closures here might render all your FFI efforts > : useless, killing the performance. Please, try to define all > : methods of all objects only once in the root namespace of the > : file. Closure usage might be justified only for rarely created > : long living heavy objects like netbox. > > Agreed that creating closures in each iteration of a loop was > bad idea (as Oleg Babin has already pin-pointed it elsewhere). It is not only about in the loop. It is about having closures at all. I don't see why are they necessary here. It is not a complex state machine like netbox which needs closures. Please, rework the function not to have any closures in them. > : > + > : > +--[[ > : > + Basic Extended > : > + 20121224 2012-12-24 Calendar date (ISO 8601) > : > + 2012359 2012-359 Ordinal date (ISO 8601) > : > + 2012W521 2012-W52-1 Week date (ISO 8601) > : > + 2012Q485 2012-Q4-85 Quarter date > : > +]] > : > + > : > +local function parse_date(str) > : > + local dt = ffi.new('dt_t[1]') > : > + local rc = cdt.dt_parse_iso_date(str, #str, dt) > : > + assert(rc > 0) > : > : 29. Er ... And what if I pass a string in a wrong format? Users > : should not get assertions. > : > : tarantool> datetime.parse_date('121r1 r1r 13 13 13') > : --- > : - error: 'builtin/datetime.lua:397: assertion failed!' > : ... > : > : This shall not ever happen. Would you mind implementing proper > : error handling. The same in all the places where the errors are > : ignored. > > > Now, I've reworked approach how they handle errors (you could see > It in later commits in the patch-set), and now it returns tuple - > - parsed value > - and the number of accepted input characters. > > tarantool> date.parse_date('121r1 r1r 13 13 13') > --- > - null > - 0 > ... > > Sometimes, while parsing composed strings using these partial > methods (parse_date / parse_time / parse_time_zone) we need to > know the number of symbols we have successfully accepted. That's > why there is seconds value in returned tuple. > > Please give me know if there is better/more idiomatic approach > used for such scenarios? In the new code which is a part of the core we use exceptions. https://github.com/tarantool/doc/issues/1727 But I see it is still not documented, so for me 'null, pos' look fine so far.
Hello Vlad, Ok, ok, despite of 20 years (!!) long personal tradition (which I've learned from Larry Wall email style, e.g. https://www.nntp.perl.org/group/perl.perl5.porters/1999/09/msg444.html), I'm finally changing my citation style back to the usual `>`, so your mail client will be happy again. > From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> > Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime > > Hi! Thanks for the fixes! Although I would prefer you > sending updates right under each non-trivial comment and > pushing the changes. Otherwise I can't check if you really > fixed the comments and if the fixes are better than it was > before. I understand, but hard to follow (at least yet for me) unless I've reshuffled all changes back to the reasonable set of changes, which I'm ok to show indefinitely attached to the GitHub issue page. Personally I hate when thousands of temporary changes got attached forever to the issue, it's hard then to find the relevant ones. > In all your responses the previous email's text is prefixed with ':' sign > unlike it is done in other people's responses. In my email client it turns > the > email into a mess where I can't see where is the new text. Could you > possibly > start using '>' prefix? Now, I've switched. > > > : > + */ > > : > +struct datetime_t { > > : > + int64_t secs; ///< seconds since epoch > > : > + int32_t nsec; ///< nanoseconds if any > > : > + int32_t offset; ///< offset in minutes from GMT > > : > > : 6. We never use // nor /// nor ///< nor we write comments > > : on the same line as the code (except for old legacy code we > > : inherited from sqlite and some super old tarantool code. > > > > BTW, why? Single line comment // is the legal C construction > > since C99 (and always have been acceptable by Gnu C and major > > other compilers). > > > > Also did you know that ///< is doxygen construction for > > documentation of a struct member? (Asking, just to make sure) > > The fact that something is a legal C construction does not > mean it fits our code style. For doxygen in our rules we have > /** as a comment opening. Ok, ok, I've switched back to /**< ... */ here > > > : > > : The same for datetime_interval_t. > > : > > : > +}; > > : > + > > : > +/** > > : > + * Date/time delta structure > > : > + */ > > : > +struct datetime_interval_t { > > : > + int64_t secs; ///< relative seconds delta > > : > + int32_t nsec; ///< nanoseconds delta > > : > +}; > > : > + > > : > > : 7. Why do you need this file? It is not included anywhere. > > : And you don't need to define the structs in C if you are > > : using them in Lua only. You can just define them in Lua > > : using ffi.cdef like it is done in some other places. > > > > I'll need to use it in C very soon, once I get to SQL parser > > INTERVAL 'xx' support. > > You will need to introduce it in the same commit in which you > will use it. We do not split patches into commits just to make > them smaller. The split needs to make sense in terms of > atomicity. > Ok > > : > +local datetime_mt = { > > : > + -- __tostring = datetime_tostring, > > : > > : 16. Why is it commented out? Is it even tested then? > > > > It's implemented in the later patch in patchset series. > > Well, then it should not be here in this commit. > > > : > + microseconds = function(self) > > : > + return tonumber(self.secs*1e6 + self.nsec*1e3) > > : > > : 18. Please, add whitespaces around binary operators. > > > > Done! It was already such in the later commits which were part > > of in patchset. [Yes, apparently I need to squash few more commits, > > as Oleg Babin has asked elsewhere] > > It is not just about squashes. It is only about atomicity (and > as a consequence - review simplicity). You can't just squash all > the commits into one. You will need to change the older commits > to make them right, then rebase the newer commits. I see. BTW, what is the simplest approach here you could recommend? I mean what tools do you use for shuffling code between version layers? How to make it simple enough and painless? (I'd love to see training Here - I know how to do it the harder way in interactive rebase, but I suspect there are some helpers, which I missed, which allow to Make it easy way. If any?) > > > : > > : 19. Why do you do tonumber() for all results? You are loosing > > : precision for big values. The same in all the places where you > > : use tonumber(). > > > > To be able to use it in places which expect numbers like %d in > > string.format. > > That does not look like a good reason - you loose precision just > to simplify the logs. Better make the logs use tonumber() than > make the functions return imprecise results. FWIW, maximal integer value, which may be precisely saved to Lua Number, is ~9e15, which, if being assigned to datetime `.secs`, could represent this date: tarantool> T = date.new() --- ... tarantool> M = 9e15 --- ... tarantool> M --- - 9000000000000000 ... tarantool> T.secs = M --- ... tarantool> T --- - 2979311-04-03T16:00Z ... It's _well beyond_ currently planned supported date range. So better ergonomics while working with those calculated attributed does deserve some extra cast upon return of a value. We do not loss any precision if we work with expected date range. At the moment the claimed supported date range is [0000...9999] for Lua runtime, and is narrower [208BC...4147] if you need to store value, and keep it correctly sorted in index. > > > Though at the end of day I've came up to the version > > where I convert to Lua numbers not all (potentially fraction point) > > values, but with exception of nanoseconds / microseconds / milliseconds > > (which return integer values). > > > > : > > : > + end, > > : > + > > : > > : 21. Extra empty line. > > > > Yup, but later, well, you know ... > > No, I couldn't understand. What 'later'? That was referring to some later commit. > > > : > > : > +} > > : > + > > : > +local interval_mt = { > > : > + -- __tostring = interval_tostring, > > : > > : 22. Ditto. Why is it commented out and not tested? > > > > Tostring conversion was implemented later in a few > > commits. One for datetime stringization, and another > > one for interval. > > Then it should be in the corresponding commit. Yup. > > > : > > : > + __serialize = interval_serialize, > > : > + __eq = datetime_eq, > > : > + __lt = datetime_lt, > > : > + __le = datetime_le, > > : > +} > > : > + > > : > +local function datetime_new_raw(secs, nsec, offset) > > : > + local dt_obj = ffi.new(datetime_t) > > : > + dt_obj.secs = secs > > : > + dt_obj.nsec = nsec > > : > + dt_obj.offset = offset > > : > + return dt_obj > > : > +end > > : > + > > : > +local function local_rd(o) > > : > > : 23. What is 'rd' and 'dt'? > > > > `dt` is type - data type from c-dt (alias to signed integer). > > Which shows a number of dates since Rata Die date (0001-01-01). > > It's signed 32-bit value, thus could represent huge range before > > 0001-01-01 as well. > > You didn't say what is 'rd'. Rata Die date. > > > : > > : > + end, > > : > > : 27. The usage of closures here might render all your FFI efforts > > : useless, killing the performance. Please, try to define all > > : methods of all objects only once in the root namespace of the > > : file. Closure usage might be justified only for rarely created > > : long living heavy objects like netbox. > > > > Agreed that creating closures in each iteration of a loop was > > bad idea (as Oleg Babin has already pin-pointed it elsewhere). > > It is not only about in the loop. It is about having closures at all. > I don't see why are they necessary here. It is not a complex state > machine like netbox which needs closures. Please, rework the function > not to have any closures in them. datetime_new() is a good example of a function using closures called via tag attributes names, where I do need to have access to the locals from outer function, i.e. easy_way, ymd, hms variables which defines submode. Converting this code to the series of ifs may make dispatch slower (and less elegant), and after change which we have discussed with Oleg Babin elsewhere, when table creation as loop invariant moved to the module scope of above of a loop, then performance impact becoming negligible. While still keeping code ergonomics. Here is the deal - let me send fully feature complete patchset still using closures at a couple of places, we proceed with review before feature freeze, then we return back to performance/closure optimizations, but as a follow-up commit? > > > : > + > > : > +--[[ > > : > + Basic Extended > > : > + 20121224 2012-12-24 Calendar date (ISO 8601) > > : > + 2012359 2012-359 Ordinal date (ISO 8601) > > : > + 2012W521 2012-W52-1 Week date (ISO 8601) > > : > + 2012Q485 2012-Q4-85 Quarter date > > : > +]] > > : > + > > : > +local function parse_date(str) > > : > + local dt = ffi.new('dt_t[1]') > > : > + local rc = cdt.dt_parse_iso_date(str, #str, dt) > > : > + assert(rc > 0) > > : > > : 29. Er ... And what if I pass a string in a wrong format? Users > > : should not get assertions. > > : > > : tarantool> datetime.parse_date('121r1 r1r 13 13 13') > > : --- > > : - error: 'builtin/datetime.lua:397: assertion failed!' > > : ... > > : > > : This shall not ever happen. Would you mind implementing proper > > : error handling. The same in all the places where the errors are > > : ignored. > > > > > > Now, I've reworked approach how they handle errors (you could see > > It in later commits in the patch-set), and now it returns tuple - > > - parsed value > > - and the number of accepted input characters. > > > > tarantool> date.parse_date('121r1 r1r 13 13 13') > > --- > > - null > > - 0 > > ... > > > > Sometimes, while parsing composed strings using these partial > > methods (parse_date / parse_time / parse_time_zone) we need to > > know the number of symbols we have successfully accepted. That's > > why there is seconds value in returned tuple. > > > > Please give me know if there is better/more idiomatic approach > > used for such scenarios? > > In the new code which is a part of the core we use exceptions. > https://github.com/tarantool/doc/issues/1727 > > But I see it is still not documented, so for me 'null, pos' look > fine so far. Nice! I still need a time top move portions of a code between different patches in a new version of patchset, but I definitely plan to send it later tonight. Stay tuned! Thanks, Timur
>>> : > +local datetime_mt = {
>>> : > + -- __tostring = datetime_tostring,
>>> :
>>> : 16. Why is it commented out? Is it even tested then?
>>>
>>> It's implemented in the later patch in patchset series.
>>
>> Well, then it should not be here in this commit.
>>
>>> : > + microseconds = function(self)
>>> : > + return tonumber(self.secs*1e6 + self.nsec*1e3)
>>> :
>>> : 18. Please, add whitespaces around binary operators.
>>>
>>> Done! It was already such in the later commits which were part
>>> of in patchset. [Yes, apparently I need to squash few more commits,
>>> as Oleg Babin has asked elsewhere]
>>
>> It is not just about squashes. It is only about atomicity (and
>> as a consequence - review simplicity). You can't just squash all
>> the commits into one. You will need to change the older commits
>> to make them right, then rebase the newer commits.
>
> I see.
>
> BTW, what is the simplest approach here you could recommend?
> I mean what tools do you use for shuffling code between version layers?
> How to make it simple enough and painless? (I'd love to see training
> Here - I know how to do it the harder way in interactive rebase, but
> I suspect there are some helpers, which I missed, which allow to
> Make it easy way. If any?)
My main tools for doing review fixes per-commit are:
'git rebase -i', 'git commit --amend', 'git add -u'.
I couldn't find anything easier. These commands take a couple of
seconds to print and I doubt there is any tool doing it faster.