Tarantool development patches archive
 help / color / mirror / Atom feed
* [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support
@ 2021-07-28 10:34 Timur Safin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
                   ` (11 more replies)
  0 siblings, 12 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* Version #2:

  - fixed problem with overloaded '-' and '+' operations for datetime 
    arguments;
  - fixed messagepack serialization problems;
  - heavily documented MessagePack serialization schema in the code;
  - introduced working implementation of datetime hints for storage engines;
  - made interval related names be more consistent, renamed durations and period 
    to intervals, i.e. t_datetime_duration to datetime_interval_t, 
    duration_* to interval_*, period to interval;
  - properly implemented all reasonable cases of datetime+interval 
    arithmetic;
  - moved all initialization code to utils.c;
  - renamed core/mp_datetime.c to core/datetime.c because it makes more
    sense now;

* Version #1 - initial RFC series

In brief
--------
This patchset implements datetime lua support in box, with serialization 
to messagepack, yaml, json and lua mode. Also it contains storage 
engines' indices implementation for datetime type introduced.

* Current implementation is heavily influenced by Sci-Lua lua-time module
  https://github.com/stepelu/lua-time 
  e.g. you could find very similar approach for handling of operations
  with year or month long intervals (which should be handled differently than
  usual intervals of seconds, or days).

* But internally we actually use Christian Hanson' c-dt module
  https://github.com/chansen/c-dt 
  (though it has been modified slightly for cleaner integration
  into cmake build process)


Datetime Module API
-------------------

We used to have here draft documentation of datetime module api, but 
for a convenience it has been extracted to the discussion topic there - 
https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988

Messagepack serialization schema
--------------------------------

In short it looks like:
- now we introduce new MP_EXT extension type #4;
- we may save 1 required and 2 optional fields for datetime field;

In details it's explained in MessagePack serialization schema depicted here:
https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990


https://github.com/tarantool/tarantool/issues/5941
https://github.com/tarantool/tarantool/issues/5946

https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v3

Timur Safin (11):
  build: add Christian Hansen c-dt to the build
  lua: built-in module datetime
  lua, datetime: datetime tests
  lua, datetime: display datetime
  box, datetime: add messagepack support for datetime
  box, datetime: datetime comparison for indices
  lua, datetime: proper datetime encoding
  lua, datetime: calculated attributes for datetimes
  lua, datetime: time intervals support
  lua, datetime: unixtime, timestamp setters in datetime.lua
  datetime: changelog for datetime module

 .gitmodules                                   |    3 +
 CMakeLists.txt                                |    8 +
 .../gh-5941-datetime-type-support.md          |    4 +
 cmake/BuildCDT.cmake                          |    6 +
 src/CMakeLists.txt                            |    3 +
 src/box/field_def.c                           |   52 +-
 src/box/field_def.h                           |    4 +
 src/box/lua/serialize_lua.c                   |    7 +-
 src/box/memtx_space.c                         |    3 +-
 src/box/msgpack.c                             |    7 +-
 src/box/tuple_compare.cc                      |   50 +
 src/box/vinyl.c                               |    3 +-
 src/exports.h                                 |   29 +
 src/lib/core/CMakeLists.txt                   |    4 +-
 src/lib/core/datetime.c                       |  251 ++++
 src/lib/core/datetime.h                       |  114 ++
 src/lib/core/mp_extension_types.h             |    1 +
 src/lib/mpstream/mpstream.c                   |   11 +
 src/lib/mpstream/mpstream.h                   |    4 +
 src/lua/datetime.lua                          | 1016 +++++++++++++++++
 src/lua/init.c                                |    4 +-
 src/lua/msgpack.c                             |   12 +
 src/lua/msgpackffi.lua                        |   18 +
 src/lua/serializer.c                          |    4 +
 src/lua/serializer.h                          |    2 +
 src/lua/utils.c                               |   28 +-
 src/lua/utils.h                               |   12 +
 test/app-tap/datetime.test.lua                |  367 ++++++
 test/engine/datetime.result                   |   77 ++
 test/engine/datetime.test.lua                 |   35 +
 test/unit/CMakeLists.txt                      |    2 +
 test/unit/datetime.c                          |  220 ++++
 test/unit/datetime.result                     |  358 ++++++
 third_party/c-dt                              |    1 +
 third_party/lua-cjson/lua_cjson.c             |    8 +
 third_party/lua-yaml/lyaml.cc                 |    6 +-
 36 files changed, 2709 insertions(+), 25 deletions(-)
 create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
 create mode 100644 cmake/BuildCDT.cmake
 create mode 100755 src/lib/core/datetime.c
 create mode 100644 src/lib/core/datetime.h
 create mode 100644 src/lua/datetime.lua
 create mode 100755 test/app-tap/datetime.test.lua
 create mode 100644 test/engine/datetime.result
 create mode 100644 test/engine/datetime.test.lua
 create mode 100644 test/unit/datetime.c
 create mode 100644 test/unit/datetime.result
 create mode 160000 third_party/c-dt

-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 23:40   ` Vladislav Shpilevoy via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime Timur Safin via Tarantool-patches
                   ` (10 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* Integrated chansen/c-dt parser as 3rd party module to the
  Tarantool cmake build process.
* Points to tsafin/c-dt instead iof original chansen/c-dt to
  have easier build integration, because there is additional
  commit which integrated cmake support
---
 .gitmodules          | 3 +++
 CMakeLists.txt       | 8 ++++++++
 cmake/BuildCDT.cmake | 6 ++++++
 src/CMakeLists.txt   | 1 +
 third_party/c-dt     | 1 +
 5 files changed, 19 insertions(+)
 create mode 100644 cmake/BuildCDT.cmake
 create mode 160000 third_party/c-dt

diff --git a/.gitmodules b/.gitmodules
index f2f91ee72..aa3fbae4e 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -43,3 +43,6 @@
 [submodule "third_party/xxHash"]
 	path = third_party/xxHash
 	url = https://github.com/tarantool/xxHash
+[submodule "third_party/c-dt"]
+	path = third_party/c-dt
+	url = https://github.com/tarantool/c-dt.git
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f9e1a7f79..81ee92933 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -571,6 +571,14 @@ endif()
 # zstd
 #
 
+#
+# Chritian Hanson c-dt
+#
+
+include(BuildCDT)
+libccdt_build()
+add_dependencies(build_bundled_libs cdt)
+
 #
 # Third-Party misc
 #
diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
new file mode 100644
index 000000000..a19abb35c
--- /dev/null
+++ b/cmake/BuildCDT.cmake
@@ -0,0 +1,6 @@
+macro(libccdt_build)
+    set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/)
+    set(LIBCDT_LIBRARIES cdt)
+
+    add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt)
+endmacro()
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index c0e272bd9..ef6a295d5 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -202,6 +202,7 @@ set (common_libraries
     ${ICONV_LIBRARIES}
     ${OPENSSL_LIBRARIES}
     ${XXHASH_LIBRARIES}
+    ${LIBCDT_LIBRARIES}
 )
 
 if (TARGET_OS_LINUX OR TARGET_OS_DEBIAN_FREEBSD)
diff --git a/third_party/c-dt b/third_party/c-dt
new file mode 160000
index 000000000..8b61c4ea0
--- /dev/null
+++ b/third_party/c-dt
@@ -0,0 +1 @@
+Subproject commit 8b61c4ea006efefc3a068f8df4a156bf5c725c89
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-29 23:36   ` Vladislav Shpilevoy via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests Timur Safin via Tarantool-patches
                   ` (9 subsequent siblings)
  11 siblings, 2 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* created a new Tarantool built-in module `datetime`;
* register cdef types for this module;
* export some `dt_*` functions from `c-dt` library;
* lua implementationis of `asctime` and `strftime`;
* datetime parsing unit tests, with and withput timezones;
* c test for reversible strftime roundtrip;

Part of #5941
---
 src/CMakeLists.txt        |   2 +
 src/exports.h             |  21 ++
 src/lib/core/datetime.h   |  61 ++++
 src/lua/datetime.lua      | 581 ++++++++++++++++++++++++++++++++++++++
 src/lua/init.c            |   4 +-
 src/lua/utils.c           |  27 ++
 src/lua/utils.h           |  12 +
 test/unit/CMakeLists.txt  |   2 +
 test/unit/datetime.c      | 221 +++++++++++++++
 test/unit/datetime.result | 358 +++++++++++++++++++++++
 10 files changed, 1288 insertions(+), 1 deletion(-)
 create mode 100644 src/lib/core/datetime.h
 create mode 100644 src/lua/datetime.lua
 create mode 100644 test/unit/datetime.c
 create mode 100644 test/unit/datetime.result

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index ef6a295d5..e0499e57f 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
 lua_source(lua_sources lua/httpc.lua)
 lua_source(lua_sources lua/iconv.lua)
 lua_source(lua_sources lua/swim.lua)
+lua_source(lua_sources lua/datetime.lua)
+
 # LuaJIT jit.* library
 lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
 lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
diff --git a/src/exports.h b/src/exports.h
index 5bb3e6a2b..db40c03a4 100644
--- a/src/exports.h
+++ b/src/exports.h
@@ -531,3 +531,24 @@ EXPORT(uri_format)
 EXPORT(uri_parse)
 EXPORT(uuid_nil)
 EXPORT(uuid_unpack)
+EXPORT(dt_from_rdn)
+EXPORT(dt_from_yd)
+EXPORT(dt_from_ymd)
+EXPORT(dt_from_yqd)
+EXPORT(dt_from_ywd)
+EXPORT(dt_to_yd)
+EXPORT(dt_to_ymd)
+EXPORT(dt_to_yqd)
+EXPORT(dt_to_ywd)
+EXPORT(dt_rdn)
+EXPORT(dt_dow)
+EXPORT(dt_parse_iso_date)
+EXPORT(dt_parse_iso_time)
+EXPORT(dt_parse_iso_time_basic)
+EXPORT(dt_parse_iso_time_extended)
+EXPORT(dt_parse_iso_zone)
+EXPORT(dt_parse_iso_zone_basic)
+EXPORT(dt_parse_iso_zone_extended)
+EXPORT(dt_parse_iso_zone_lenient)
+EXPORT(dt_from_struct_tm)
+EXPORT(dt_to_struct_tm)
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
new file mode 100644
index 000000000..403bf1c64
--- /dev/null
+++ b/src/lib/core/datetime.h
@@ -0,0 +1,61 @@
+#pragma once
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <c-dt/dt_core.h>
+#include <stdint.h>
+#include <stdbool.h>
+
+#if defined(__cplusplus)
+extern "C" {
+#endif /* defined(__cplusplus) */
+
+/**
+ * datetime structure consisting of:
+ */
+struct datetime_t {
+	int64_t secs;	///< seconds since epoch
+	int32_t nsec;	///< nanoseconds if any
+	int32_t offset; ///< offset in minutes from GMT
+};
+
+/**
+ * Date/time delta structure
+ */
+struct datetime_interval_t {
+	int64_t secs; ///< relative seconds delta
+	int32_t nsec; ///< nanoseconds delta
+};
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
+
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
new file mode 100644
index 000000000..0996ca5a2
--- /dev/null
+++ b/src/lua/datetime.lua
@@ -0,0 +1,581 @@
+local ffi = require('ffi')
+local cdt = ffi.C
+
+ffi.cdef [[
+
+    typedef int dt_t;
+
+    // dt_core.h
+    typedef enum {
+        DT_MON       = 1,
+        DT_MONDAY    = 1,
+        DT_TUE       = 2,
+        DT_TUESDAY   = 2,
+        DT_WED       = 3,
+        DT_WEDNESDAY = 3,
+        DT_THU       = 4,
+        DT_THURSDAY  = 4,
+        DT_FRI       = 5,
+        DT_FRIDAY    = 5,
+        DT_SAT       = 6,
+        DT_SATURDAY  = 6,
+        DT_SUN       = 7,
+        DT_SUNDAY    = 7,
+    } dt_dow_t;
+
+    dt_t     dt_from_rdn     (int n);
+    dt_t     dt_from_yd      (int y, int d);
+    dt_t     dt_from_ymd     (int y, int m, int d);
+    dt_t     dt_from_yqd     (int y, int q, int d);
+    dt_t     dt_from_ywd     (int y, int w, int d);
+
+    void     dt_to_yd        (dt_t dt, int *y, int *d);
+    void     dt_to_ymd       (dt_t dt, int *y, int *m, int *d);
+    void     dt_to_yqd       (dt_t dt, int *y, int *q, int *d);
+    void     dt_to_ywd       (dt_t dt, int *y, int *w, int *d);
+
+    int      dt_rdn          (dt_t dt);
+    dt_dow_t dt_dow          (dt_t dt);
+
+    // dt_parse_iso.h
+    size_t dt_parse_iso_date          (const char *str, size_t len, dt_t *dt);
+
+    size_t dt_parse_iso_time          (const char *str, size_t len, int *sod, int *nsec);
+    size_t dt_parse_iso_time_basic    (const char *str, size_t len, int *sod, int *nsec);
+    size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec);
+
+    size_t dt_parse_iso_zone          (const char *str, size_t len, int *offset);
+    size_t dt_parse_iso_zone_basic    (const char *str, size_t len, int *offset);
+    size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset);
+    size_t dt_parse_iso_zone_lenient  (const char *str, size_t len, int *offset);
+
+    // dt_tm.h
+    dt_t    dt_from_struct_tm  (const struct tm *tm);
+    void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
+
+    // <asm-generic/posix_types.h>
+    typedef long            __kernel_long_t;
+    typedef unsigned long   __kernel_ulong_t;
+    // /usr/include/x86_64-linux-gnu/bits/types/time_t.h
+    typedef long            time_t;
+
+
+    // <time.h>
+    typedef __kernel_long_t	__kernel_time_t;
+    typedef __kernel_long_t	__kernel_suseconds_t;
+
+    struct timespec {
+        __kernel_time_t	        tv_sec;     /* seconds */
+        long                    tv_nsec;    /* nanoseconds */
+    };
+
+    struct timeval {
+        __kernel_time_t	        tv_sec;	    /* seconds */
+        __kernel_suseconds_t    tv_usec;    /* microseconds */
+    };
+
+    struct timezone {
+        int	tz_minuteswest;     /* minutes west of Greenwich */
+        int	tz_dsttime;	        /* type of dst correction */
+    };
+
+    // /usr/include/x86_64-linux-gnu/sys/time.h
+    typedef struct timezone * __timezone_ptr_t;
+
+    /* Get the current time of day and timezone information,
+       putting it into *TV and *TZ.  If TZ is NULL, *TZ is not filled.
+       Returns 0 on success, -1 on errors.
+
+       NOTE: This form of timezone information is obsolete.
+       Use the functions and variables declared in <time.h> instead.  */
+    int gettimeofday (struct timeval *__tv, struct timezone * __tz);
+
+    // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h
+    /* ISO C `broken-down time' structure.  */
+    struct tm
+    {
+        int tm_sec;	        /* Seconds.	[0-60] (1 leap second) */
+        int tm_min;	        /* Minutes.	[0-59] */
+        int tm_hour;        /* Hours.	[0-23] */
+        int tm_mday;        /* Day.		[1-31] */
+        int tm_mon;	        /* Month.	[0-11] */
+        int tm_year;        /* Year	- 1900.  */
+        int tm_wday;        /* Day of week.	[0-6] */
+        int tm_yday;        /* Days in year.[0-365]	*/
+        int tm_isdst;       /* DST.		[-1/0/1]*/
+
+        long int tm_gmtoff; /* Seconds east of UTC.  */
+        const char *tm_zone;/* Timezone abbreviation.  */
+    };
+
+    // <time.h>
+    /* Return the current time and put it in *TIMER if TIMER is not NULL.  */
+    time_t time (time_t *__timer);
+
+    /* Format TP into S according to FORMAT.
+    Write no more than MAXSIZE characters and return the number
+    of characters written, or 0 if it would exceed MAXSIZE.  */
+    size_t strftime (char * __s, size_t __maxsize, const char * __format,
+                     const struct tm * __tp);
+
+    /* Parse S according to FORMAT and store binary time information in TP.
+    The return value is a pointer to the first unparsed character in S.  */
+    char *strptime (const char * __s, const char * __fmt, struct tm *__tp);
+
+    /* Return the `struct tm' representation of *TIMER in UTC,
+    using *TP to store the result.  */
+    struct tm *gmtime_r (const time_t * __timer, struct tm * __tp);
+
+    /* Return the `struct tm' representation of *TIMER in local time,
+    using *TP to store the result.  */
+    struct tm *localtime_r (const time_t * __timer, struct tm * __tp);
+
+    /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n"
+    that is the representation of TP in this format.  */
+    char *asctime (const struct tm *__tp);
+
+    /* Equivalent to `asctime (localtime (timer))'.  */
+    char *ctime (const time_t *__timer);
+
+]]
+
+local native = ffi.C
+
+local SECS_PER_DAY     = 86400
+local NANOS_PER_SEC    = 1000000000LL
+
+-- c-dt/dt_config.h
+
+-- Unix, January 1, 1970, Thursday
+local DT_EPOCH_1970_OFFSET = 719163LL
+
+
+local datetime_t = ffi.typeof('struct datetime_t')
+local interval_t = ffi.typeof('struct datetime_interval_t')
+
+local function interval_new()
+    local interval = ffi.new(interval_t)
+    return interval
+end
+
+local function adjusted_secs(dt)
+    return dt.secs - dt.offset * 60
+end
+
+local function datetime_sub(lhs, rhs)
+    local s1 = adjusted_secs(lhs)
+    local s2 = adjusted_secs(rhs)
+    local d = interval_new()
+    d.secs = s2 - s1
+    d.nsec = rhs.nsec - lhs.nsec
+    if d.nsec < 0 then
+        d.secs = d.secs - 1
+        d.nsec = d.nsec + NANOS_PER_SEC
+    end
+    return d
+end
+
+local function datetime_add(lhs, rhs)
+    local s1 = adjusted_secs(lhs)
+    local s2 = adjusted_secs(rhs)
+    local d = interval_new()
+    d.secs = s2 + s1
+    d.nsec = rhs.nsec + lhs.nsec
+    if d.nsec >= NANOS_PER_SEC then
+        d.secs = d.secs + 1
+        d.nsec = d.nsec - NANOS_PER_SEC
+    end
+    return d
+end
+
+local function datetime_eq(lhs, rhs)
+    -- we usually don't need to check nullness
+    -- but older tarantool console will call us checking for equality to nil
+    if rhs == nil then
+        return false
+    end
+    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
+end
+
+
+local function datetime_lt(lhs, rhs)
+    return (lhs.secs < rhs.secs) or
+           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
+end
+
+local function datetime_le(lhs, rhs)
+    return (lhs.secs <= rhs.secs) or
+           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
+end
+
+local function datetime_serialize(self)
+    -- Allow YAML, MsgPack and JSON to dump objects with sockets
+    return { secs = self.secs, nsec = self.nsec, tz = self.offset }
+end
+
+local function interval_serialize(self)
+    -- Allow YAML and JSON to dump objects with sockets
+    return { secs = self.secs, nsec = self.nsec }
+end
+
+local datetime_mt = {
+    -- __tostring = datetime_tostring,
+    __serialize = datetime_serialize,
+    __eq = datetime_eq,
+    __lt = datetime_lt,
+    __le = datetime_le,
+    __sub = datetime_sub,
+    __add = datetime_add,
+
+    nanoseconds = function(self)
+        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
+    end,
+    microseconds = function(self)
+        return tonumber(self.secs*1e6 + self.nsec*1e3)
+    end,
+    seconds = function(self)
+        return tonumber(self.secs + self.nsec*1e3)
+    end,
+    minutes = function(self)
+        return tonumber((self._ticks/(1e6*60))%60)
+    end,
+    hours = function(self)
+        return tonumber(self._ticks/(1e6*60*60))
+    end,
+
+}
+
+local interval_mt = {
+    -- __tostring = interval_tostring,
+    __serialize = interval_serialize,
+    __eq = datetime_eq,
+    __lt = datetime_lt,
+    __le = datetime_le,
+}
+
+local function datetime_new_raw(secs, nsec, offset)
+    local dt_obj = ffi.new(datetime_t)
+    dt_obj.secs = secs
+    dt_obj.nsec = nsec
+    dt_obj.offset = offset
+    return dt_obj
+end
+
+local function local_rd(o)
+    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
+end
+
+local function local_dt(o)
+    return cdt.dt_from_rdn(local_rd(o))
+end
+
+local function mk_timestamp(dt, sp, fp, offset)
+    local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0
+    local spV = sp ~= nil and sp or 0
+    local fpV = fp ~= nil and fp or 0
+    local ofsV = offset ~= nil and offset or 0
+    return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV)
+end
+
+-- create @datetime_t given object @o fields
+local function datetime_new(o)
+    if o == nil then
+        return datetime_new_raw(0, 0, 0)
+    end
+    local secs = 0
+    local nsec = 0
+    local offset = 0
+    local easy_way = false
+    local y, M, d, ymd
+    y, M, d, ymd = 0, 0, 0, false
+
+    local h, m, s, frac, hms
+    h, m, s, frac, hms = 0, 0, 0, 0, false
+
+    local dt = 0
+
+    for key, value in pairs(o) do
+        local handlers = {
+            secs = function(v)
+                secs = v
+                easy_way = true
+            end,
+
+            nsec = function(v)
+                nsec = v
+                easy_way = true
+            end,
+
+            offset = function (v)
+                offset = v
+                easy_way = true
+            end,
+
+            year = function(v)
+                assert(v > 0 and v < 10000)
+                y = v
+                ymd = true
+            end,
+
+            month = function(v)
+                assert(v > 0 and v < 12 )
+                M = v
+                ymd = true
+            end,
+
+            day = function(v)
+                assert(v > 0 and v < 32)
+                d = v
+                ymd = true
+            end,
+
+            hour = function(v)
+                assert(v >= 0 and v < 24)
+                h = v
+                hms = true
+            end,
+
+            minute = function(v)
+                assert(v >= 0 and v < 60)
+                m = v
+                hms = true
+            end,
+
+            second = function(v)
+                assert(v >= 0 and v < 61)
+                frac = v % 1
+                if frac then
+                    s = v - (v % 1)
+                else
+                    s = v
+                end
+                hms = true
+            end,
+
+            -- tz offset in minutes
+            tz = function(v)
+                assert(v >= 0 and v <= 720)
+                offset = v
+            end
+        }
+        handlers[key](value)
+    end
+
+    -- .sec, .nsec, .offset
+    if easy_way then
+        return datetime_new_raw(secs, nsec, offset)
+    end
+
+    -- .year, .month, .day
+    if ymd then
+        dt = dt + cdt.dt_from_ymd(y, M, d)
+    end
+
+    -- .hour, .minute, .second
+    if hms then
+        secs = h * 3600 + m * 60 + s
+    end
+
+    return mk_timestamp(dt, secs, frac, offset)
+end
+
+
+-- simple parse functions:
+-- parse_date/parse_time/parse_zone
+
+--[[
+    Basic      Extended
+    20121224   2012-12-24   Calendar date   (ISO 8601)
+    2012359    2012-359     Ordinal date    (ISO 8601)
+    2012W521   2012-W52-1   Week date       (ISO 8601)
+    2012Q485   2012-Q4-85   Quarter date
+]]
+
+local function parse_date(str)
+    local dt = ffi.new('dt_t[1]')
+    local rc = cdt.dt_parse_iso_date(str, #str, dt)
+    assert(rc > 0)
+    return mk_timestamp(dt[0])
+end
+
+--[[
+    Basic               Extended
+    T12                 N/A
+    T1230               T12:30
+    T123045             T12:30:45
+    T123045.123456789   T12:30:45.123456789
+    T123045,123456789   T12:30:45,123456789
+
+    The time designator [T] may be omitted.
+]]
+local function parse_time(str)
+    local sp = ffi.new('int[1]')
+    local fp = ffi.new('int[1]')
+    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
+    assert(rc > 0)
+    return mk_timestamp(nil, sp[0], fp[0])
+end
+
+--[[
+    Basic    Extended
+    Z        N/A
+    ±hh      N/A
+    ±hhmm    ±hh:mm
+]]
+local function parse_zone(str)
+    local offset = ffi.new('int[1]')
+    local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
+    return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len)
+end
+
+
+--[[
+    aggregated parse functions
+    assumes to deal with date T time time_zone
+    at once
+
+    date [T] time [ ] time_zone
+]]
+local function parse_str(str)
+    local dt = ffi.new('dt_t[1]')
+    local len = #str
+    local n = cdt.dt_parse_iso_date(str, len, dt)
+    local dt_ = dt[0]
+    if n == 0 or len == n then
+        return mk_timestamp(dt_)
+    end
+
+    str = str:sub(tonumber(n) + 1)
+
+    local ch = str:sub(1,1)
+    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
+        return mk_timestamp(dt_)
+    end
+
+    str = str:sub(2)
+    len = #str
+
+    local sp = ffi.new('int[1]')
+    local fp = ffi.new('int[1]')
+    local n = cdt.dt_parse_iso_time(str, len, sp, fp)
+    if n == 0 then
+        return mk_timestamp(dt_)
+    end
+    local sp_ = sp[0]
+    local fp_ = fp[0]
+    if len == n then
+        return mk_timestamp(dt_, sp_, fp_)
+    end
+
+    str = str:sub(tonumber(n) + 1)
+
+    if str:sub(1,1) == ' ' then
+        str = str:sub(2)
+    end
+
+    len = #str
+
+    local offset = ffi.new('int[1]')
+    n = cdt.dt_parse_iso_zone_lenient(str, len, offset)
+    if n == 0 then
+        return mk_timestamp(dt_, sp_, fp_)
+    end
+    return mk_timestamp(dt_, sp_, fp_, offset[0])
+end
+
+local function datetime_from(o)
+    if o == nil or type(o) == 'table' then
+        return datetime_new(o)
+    elseif type(o) == 'string' then
+        return parse_str(o)
+    end
+end
+
+local function local_now()
+    local p_tv = ffi.new ' struct timeval [1] '
+    local rc = native.gettimeofday(p_tv, nil)
+    assert(rc == 0)
+
+    local secs = p_tv[0].tv_sec
+    local nsec = p_tv[0].tv_usec * 1000
+
+    local p_time = ffi.new 'time_t[1]'
+    local p_tm = ffi.new 'struct tm[1]'
+    native.time(p_time)
+    native.localtime_r(p_time, p_tm)
+    -- local dt = cdt.dt_from_struct_tm(p_tm)
+    local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes
+
+    return datetime_new_raw(secs, nsec, ofs) -- FIXME
+end
+
+local function datetime_to_tm_ptr(o)
+    local p_tm = ffi.new 'struct tm[1]'
+    assert(ffi.typeof(o) == datetime_t)
+    -- dt_to_struct_tm() fills only date data
+    cdt.dt_to_struct_tm(local_dt(o), p_tm)
+
+    -- calculate the smaller data (hour, minute,
+    -- seconds) using datetime seconds value
+    local seconds_of_day = o.secs % 86400
+    local hour = (seconds_of_day / 3600) % 24
+    local minute = (seconds_of_day / 60) % 60
+    p_tm[0].tm_sec = seconds_of_day % 60
+    p_tm[0].tm_min = minute
+    p_tm[0].tm_hour = hour
+
+    p_tm[0].tm_gmtoff = o.offset * 60
+
+    return p_tm
+end
+
+local function asctime(o)
+    assert(ffi.typeof(o) == datetime_t)
+    local p_tm = datetime_to_tm_ptr(o)
+    return ffi.string(native.asctime(p_tm))
+end
+
+local function ctime(o)
+    assert(ffi.typeof(o) == datetime_t)
+    local p_time = ffi.new 'time_t[1]'
+    p_time[0] = o.secs
+    return ffi.string(native.ctime(p_time))
+end
+
+local function strftime(fmt, o)
+    assert(ffi.typeof(o) == datetime_t)
+    local sz = 50
+    local buff = ffi.new('char[?]', sz)
+    local p_tm = datetime_to_tm_ptr(o)
+    native.strftime(buff, sz, fmt, p_tm)
+    return ffi.string(buff)
+end
+
+-- strftime may be redirected to datetime:fmt("format")
+local function datetime_fmt()
+end
+
+
+ffi.metatype(interval_t, interval_mt)
+ffi.metatype(datetime_t, datetime_mt)
+
+return setmetatable(
+    {
+        datetime = datetime_new,
+        interval = interval_new,
+
+        parse = parse_str,
+        parse_date = parse_date,
+        parse_time = parse_time,
+        parse_zone = parse_zone,
+        fmt = datetime_fmt,
+
+        now = local_now,
+    -- strptime = strptime;
+        strftime = strftime,
+        asctime = asctime,
+        ctime = ctime,
+    }, {
+        __call = function(self, ...) return datetime_from(...) end
+    }
+)
diff --git a/src/lua/init.c b/src/lua/init.c
index f9738025d..127e935d7 100644
--- a/src/lua/init.c
+++ b/src/lua/init.c
@@ -129,7 +129,8 @@ extern char strict_lua[],
 	parse_lua[],
 	process_lua[],
 	humanize_lua[],
-	memprof_lua[]
+	memprof_lua[],
+	datetime_lua[]
 ;
 
 static const char *lua_modules[] = {
@@ -184,6 +185,7 @@ static const char *lua_modules[] = {
 	"memprof.process", process_lua,
 	"memprof.humanize", humanize_lua,
 	"memprof", memprof_lua,
+	"datetime", datetime_lua,
 	NULL
 };
 
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 34cec0eed..6aeb18cfe 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -47,6 +47,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
 static uint32_t CTID_CHAR_PTR;
 static uint32_t CTID_CONST_CHAR_PTR;
 uint32_t CTID_UUID;
+uint32_t CTID_DATETIME = 0;
+uint32_t CTID_INTERVAL = 0;
+
 
 void *
 luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
@@ -107,6 +110,12 @@ luaL_pushuuid(struct lua_State *L)
 	return luaL_pushcdata(L, CTID_UUID);
 }
 
+struct datetime_t *
+luaL_pushdatetime(struct lua_State *L)
+{
+	return luaL_pushcdata(L, CTID_DATETIME);
+}
+
 int
 luaL_iscdata(struct lua_State *L, int idx)
 {
@@ -712,6 +721,24 @@ tarantool_lua_utils_init(struct lua_State *L)
 	CTID_UUID = luaL_ctypeid(L, "struct tt_uuid");
 	assert(CTID_UUID != 0);
 
+	rc = luaL_cdef(L, "struct datetime_t {"
+			  "int64_t secs;"
+			  "int32_t nsec;"
+			  "int32_t offset;"
+			  "};");
+	assert(rc == 0);
+	(void) rc;
+	CTID_DATETIME = luaL_ctypeid(L, "struct datetime_t");
+	assert(CTID_DATETIME != 0);
+	rc = luaL_cdef(L, "struct datetime_interval_t {"
+			  "int64_t secs;"
+			  "int32_t nsec;"
+			  "};");
+	assert(rc == 0);
+	(void) rc;
+	CTID_INTERVAL = luaL_ctypeid(L, "struct datetime_interval_t");
+	assert(CTID_INTERVAL != 0);
+
 	lua_pushcfunction(L, luaT_newthread_wrapper);
 	luaT_newthread_ref = luaL_ref(L, LUA_REGISTRYINDEX);
 	return 0;
diff --git a/src/lua/utils.h b/src/lua/utils.h
index 947d9240b..afd41c75b 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -59,6 +59,7 @@ struct lua_State;
 struct ibuf;
 typedef struct ibuf box_ibuf_t;
 struct tt_uuid;
+struct datetime_t;
 
 /**
  * Single global lua_State shared by core and modules.
@@ -69,10 +70,21 @@ struct tt_uuid;
 extern struct lua_State *tarantool_L;
 
 extern uint32_t CTID_UUID;
+extern uint32_t CTID_DATETIME;
+extern uint32_t CTID_INTERVAL;
 
 struct tt_uuid *
 luaL_pushuuid(struct lua_State *L);
 
+/**
+ * @brief Push cdata of a datetime_t type onto the stack.
+ * @param L Lua State
+ * @sa luaL_pushcdata
+ * @return memory associated with this datetime_t data
+ */
+struct datetime_t *
+luaL_pushdatetime(struct lua_State *L);
+
 /** \cond public */
 
 /**
diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
index 5bb7cd6e7..f8320aebd 100644
--- a/test/unit/CMakeLists.txt
+++ b/test/unit/CMakeLists.txt
@@ -56,6 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
 target_link_libraries(uuid.test uuid unit)
 add_executable(random.test random.c core_test_utils.c)
 target_link_libraries(random.test core unit)
+add_executable(datetime.test datetime.c)
+target_link_libraries(datetime.test cdt unit)
 
 add_executable(bps_tree.test bps_tree.cc)
 target_link_libraries(bps_tree.test small misc)
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
new file mode 100644
index 000000000..b6f568c03
--- /dev/null
+++ b/test/unit/datetime.c
@@ -0,0 +1,221 @@
+#include "dt.h"
+#include <assert.h>
+#include <stdint.h>
+#include <string.h>
+#include <time.h>
+
+#include "unit.h"
+
+const char sample[] = "2012-12-24T15:30Z";
+
+#define S(s) {s, sizeof(s) - 1}
+struct {
+	const char * sz;
+	size_t len;
+} tests[] = {
+	S("2012-12-24 15:30Z"),
+	S("2012-12-24 15:30z"),
+	S("2012-12-24 15:30"),
+	S("2012-12-24 16:30+01:00"),
+	S("2012-12-24 16:30+0100"),
+	S("2012-12-24 16:30+01"),
+	S("2012-12-24 14:30-01:00"),
+	S("2012-12-24 14:30-0100"),
+	S("2012-12-24 14:30-01"),
+	S("2012-12-24 15:30:00Z"),
+	S("2012-12-24 15:30:00z"),
+	S("2012-12-24 15:30:00"),
+	S("2012-12-24 16:30:00+01:00"),
+	S("2012-12-24 16:30:00+0100"),
+	S("2012-12-24 14:30:00-01:00"),
+	S("2012-12-24 14:30:00-0100"),
+	S("2012-12-24 15:30:00.123456Z"),
+	S("2012-12-24 15:30:00.123456z"),
+	S("2012-12-24 15:30:00.123456"),
+	S("2012-12-24 16:30:00.123456+01:00"),
+	S("2012-12-24 16:30:00.123456+01"),
+	S("2012-12-24 14:30:00.123456-01:00"),
+	S("2012-12-24 14:30:00.123456-01"),
+	S("2012-12-24t15:30Z"),
+	S("2012-12-24t15:30z"),
+	S("2012-12-24t15:30"),
+	S("2012-12-24t16:30+01:00"),
+	S("2012-12-24t16:30+0100"),
+	S("2012-12-24t14:30-01:00"),
+	S("2012-12-24t14:30-0100"),
+	S("2012-12-24t15:30:00Z"),
+	S("2012-12-24t15:30:00z"),
+	S("2012-12-24t15:30:00"),
+	S("2012-12-24t16:30:00+01:00"),
+	S("2012-12-24t16:30:00+0100"),
+	S("2012-12-24t14:30:00-01:00"),
+	S("2012-12-24t14:30:00-0100"),
+	S("2012-12-24t15:30:00.123456Z"),
+	S("2012-12-24t15:30:00.123456z"),
+	S("2012-12-24t16:30:00.123456+01:00"),
+	S("2012-12-24t14:30:00.123456-01:00"),
+	S("2012-12-24 16:30 +01:00"),
+	S("2012-12-24 14:30 -01:00"),
+	S("2012-12-24 15:30 UTC"),
+	S("2012-12-24 16:30 UTC+1"),
+	S("2012-12-24 16:30 UTC+01"),
+	S("2012-12-24 16:30 UTC+0100"),
+	S("2012-12-24 16:30 UTC+01:00"),
+	S("2012-12-24 14:30 UTC-1"),
+	S("2012-12-24 14:30 UTC-01"),
+	S("2012-12-24 14:30 UTC-01:00"),
+	S("2012-12-24 14:30 UTC-0100"),
+	S("2012-12-24 15:30 GMT"),
+	S("2012-12-24 16:30 GMT+1"),
+	S("2012-12-24 16:30 GMT+01"),
+	S("2012-12-24 16:30 GMT+0100"),
+	S("2012-12-24 16:30 GMT+01:00"),
+	S("2012-12-24 14:30 GMT-1"),
+	S("2012-12-24 14:30 GMT-01"),
+	S("2012-12-24 14:30 GMT-01:00"),
+	S("2012-12-24 14:30 GMT-0100"),
+	S("2012-12-24 14:30 -01:00"),
+	S("2012-12-24 16:30:00 +01:00"),
+	S("2012-12-24 14:30:00 -01:00"),
+	S("2012-12-24 16:30:00.123456 +01:00"),
+	S("2012-12-24 14:30:00.123456 -01:00"),
+	S("2012-12-24 15:30:00.123456 -00:00"),
+	S("20121224T1630+01:00"),
+	S("2012-12-24T1630+01:00"),
+	S("20121224T16:30+01"),
+	S("20121224T16:30 +01"),
+};
+#undef S
+
+#define DIM(a) (sizeof(a) / sizeof(a[0]))
+
+// p5-time-moment/src/moment_parse.c: parse_string_lenient()
+static int
+parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
+	       int32_t *op)
+{
+	size_t n;
+	dt_t dt;
+	char c;
+	int sod = 0, nanosecond = 0, offset = 0;
+
+	n = dt_parse_iso_date(str, len, &dt);
+	if (!n)
+		return 1;
+	if (n == len)
+		goto exit;
+
+	c = str[n++];
+	if (!(c == 'T' || c == 't' || c == ' '))
+		return 1;
+
+	str += n;
+	len -= n;
+
+	n = dt_parse_iso_time(str, len, &sod, &nanosecond);
+	if (!n)
+		return 1;
+	if (n == len)
+		goto exit;
+
+	if (str[n] == ' ')
+	n++;
+
+	str += n;
+	len -= n;
+
+	n = dt_parse_iso_zone_lenient(str, len, &offset);
+	if (!n || n != len)
+		return 1;
+
+exit:
+	*sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
+	*np = nanosecond;
+	*op = offset;
+
+	return 0;
+}
+
+// avoid introducing external datetime.h dependency
+// - just copy paste it for today
+#define SECS_PER_DAY      86400
+#define NANOS_PER_SEC     1000000000
+#define DT_EPOCH_1970_OFFSET 719163
+
+
+struct datetime_t {
+	int64_t secs;
+	int32_t nsec;
+	int32_t offset;
+};
+
+static int
+local_rd(const struct datetime_t * dt) {
+	return (int)(dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET;
+}
+
+static int
+local_dt(const struct datetime_t * dt) {
+	return dt_from_rdn(local_rd(dt));
+}
+
+
+struct tm*
+datetime_to_tm(struct datetime_t * dt)
+{
+	static struct tm tm;
+
+	memset(&tm, 0, sizeof(tm));
+	dt_to_struct_tm(local_dt(dt), &tm);
+
+	int seconds_of_day = dt->secs % 86400;
+	tm.tm_hour = (seconds_of_day / 3600) % 24;
+	tm.tm_min = (seconds_of_day / 60) % 60;
+	tm.tm_sec = seconds_of_day % 60;
+
+	return &tm;
+}
+
+static void datetime_test(void)
+{
+	size_t index;
+	int64_t secs_expected;
+	int32_t nanosecs;
+	int32_t ofs;
+
+	plan(355);
+	parse_datetime(sample, sizeof(sample) - 1,
+		       &secs_expected, &nanosecs, &ofs);
+
+	for (index = 0; index < DIM(tests); index++) {
+		int64_t secs;
+		int rc = parse_datetime(tests[index].sz, tests[index].len,
+						&secs, &nanosecs, &ofs);
+		is(rc, 0, "correct parse_datetime return value for '%s'",
+		   tests[index].sz);
+		is(secs, secs_expected, "correct parse_datetime output "
+		   "seconds for '%s", tests[index].sz);
+
+		// check that stringized literal produces the same date
+		// time fields
+		static char buff[40];
+		struct datetime_t dt = {secs, nanosecs, ofs};
+		// datetime_to_tm returns time in GMT zone
+		struct tm * p_tm = datetime_to_tm(&dt);
+		size_t len = strftime(buff, sizeof buff, "%F %T%z", p_tm);
+		ok(len > 0, "strftime");
+		rc = parse_datetime(buff, len, &dt.secs, &dt.nsec, &dt.offset);
+		is(rc, 0, "correct parse_datetime return value for '%s'", buff);
+		is(secs, dt.secs,
+		   "reversible seconds via strftime for '%s", buff);
+	}
+}
+
+int
+main(void)
+{
+	plan(1);
+	datetime_test();
+
+	return check_plan();
+}
diff --git a/test/unit/datetime.result b/test/unit/datetime.result
new file mode 100644
index 000000000..f106fa769
--- /dev/null
+++ b/test/unit/datetime.result
@@ -0,0 +1,358 @@
+1..1
+    1..355
+    ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z'
+    ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z
+    ok 3 - strftime
+    ok 4 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 5 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 6 - correct parse_datetime return value for '2012-12-24 15:30z'
+    ok 7 - correct parse_datetime output seconds for '2012-12-24 15:30z
+    ok 8 - strftime
+    ok 9 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 10 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 11 - correct parse_datetime return value for '2012-12-24 15:30'
+    ok 12 - correct parse_datetime output seconds for '2012-12-24 15:30
+    ok 13 - strftime
+    ok 14 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 15 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 16 - correct parse_datetime return value for '2012-12-24 16:30+01:00'
+    ok 17 - correct parse_datetime output seconds for '2012-12-24 16:30+01:00
+    ok 18 - strftime
+    ok 19 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 20 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 21 - correct parse_datetime return value for '2012-12-24 16:30+0100'
+    ok 22 - correct parse_datetime output seconds for '2012-12-24 16:30+0100
+    ok 23 - strftime
+    ok 24 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 25 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 26 - correct parse_datetime return value for '2012-12-24 16:30+01'
+    ok 27 - correct parse_datetime output seconds for '2012-12-24 16:30+01
+    ok 28 - strftime
+    ok 29 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 30 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 31 - correct parse_datetime return value for '2012-12-24 14:30-01:00'
+    ok 32 - correct parse_datetime output seconds for '2012-12-24 14:30-01:00
+    ok 33 - strftime
+    ok 34 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 35 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 36 - correct parse_datetime return value for '2012-12-24 14:30-0100'
+    ok 37 - correct parse_datetime output seconds for '2012-12-24 14:30-0100
+    ok 38 - strftime
+    ok 39 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 40 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 41 - correct parse_datetime return value for '2012-12-24 14:30-01'
+    ok 42 - correct parse_datetime output seconds for '2012-12-24 14:30-01
+    ok 43 - strftime
+    ok 44 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 45 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 46 - correct parse_datetime return value for '2012-12-24 15:30:00Z'
+    ok 47 - correct parse_datetime output seconds for '2012-12-24 15:30:00Z
+    ok 48 - strftime
+    ok 49 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 50 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 51 - correct parse_datetime return value for '2012-12-24 15:30:00z'
+    ok 52 - correct parse_datetime output seconds for '2012-12-24 15:30:00z
+    ok 53 - strftime
+    ok 54 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 55 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 56 - correct parse_datetime return value for '2012-12-24 15:30:00'
+    ok 57 - correct parse_datetime output seconds for '2012-12-24 15:30:00
+    ok 58 - strftime
+    ok 59 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 60 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 61 - correct parse_datetime return value for '2012-12-24 16:30:00+01:00'
+    ok 62 - correct parse_datetime output seconds for '2012-12-24 16:30:00+01:00
+    ok 63 - strftime
+    ok 64 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 65 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 66 - correct parse_datetime return value for '2012-12-24 16:30:00+0100'
+    ok 67 - correct parse_datetime output seconds for '2012-12-24 16:30:00+0100
+    ok 68 - strftime
+    ok 69 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 70 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 71 - correct parse_datetime return value for '2012-12-24 14:30:00-01:00'
+    ok 72 - correct parse_datetime output seconds for '2012-12-24 14:30:00-01:00
+    ok 73 - strftime
+    ok 74 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 75 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 76 - correct parse_datetime return value for '2012-12-24 14:30:00-0100'
+    ok 77 - correct parse_datetime output seconds for '2012-12-24 14:30:00-0100
+    ok 78 - strftime
+    ok 79 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 80 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 81 - correct parse_datetime return value for '2012-12-24 15:30:00.123456Z'
+    ok 82 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456Z
+    ok 83 - strftime
+    ok 84 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 85 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 86 - correct parse_datetime return value for '2012-12-24 15:30:00.123456z'
+    ok 87 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456z
+    ok 88 - strftime
+    ok 89 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 90 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 91 - correct parse_datetime return value for '2012-12-24 15:30:00.123456'
+    ok 92 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456
+    ok 93 - strftime
+    ok 94 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 95 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 96 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01:00'
+    ok 97 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01:00
+    ok 98 - strftime
+    ok 99 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 100 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 101 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01'
+    ok 102 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01
+    ok 103 - strftime
+    ok 104 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 105 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 106 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01:00'
+    ok 107 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01:00
+    ok 108 - strftime
+    ok 109 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 110 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 111 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01'
+    ok 112 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01
+    ok 113 - strftime
+    ok 114 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 115 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 116 - correct parse_datetime return value for '2012-12-24t15:30Z'
+    ok 117 - correct parse_datetime output seconds for '2012-12-24t15:30Z
+    ok 118 - strftime
+    ok 119 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 120 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 121 - correct parse_datetime return value for '2012-12-24t15:30z'
+    ok 122 - correct parse_datetime output seconds for '2012-12-24t15:30z
+    ok 123 - strftime
+    ok 124 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 125 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 126 - correct parse_datetime return value for '2012-12-24t15:30'
+    ok 127 - correct parse_datetime output seconds for '2012-12-24t15:30
+    ok 128 - strftime
+    ok 129 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 130 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 131 - correct parse_datetime return value for '2012-12-24t16:30+01:00'
+    ok 132 - correct parse_datetime output seconds for '2012-12-24t16:30+01:00
+    ok 133 - strftime
+    ok 134 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 135 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 136 - correct parse_datetime return value for '2012-12-24t16:30+0100'
+    ok 137 - correct parse_datetime output seconds for '2012-12-24t16:30+0100
+    ok 138 - strftime
+    ok 139 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 140 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 141 - correct parse_datetime return value for '2012-12-24t14:30-01:00'
+    ok 142 - correct parse_datetime output seconds for '2012-12-24t14:30-01:00
+    ok 143 - strftime
+    ok 144 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 145 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 146 - correct parse_datetime return value for '2012-12-24t14:30-0100'
+    ok 147 - correct parse_datetime output seconds for '2012-12-24t14:30-0100
+    ok 148 - strftime
+    ok 149 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 150 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 151 - correct parse_datetime return value for '2012-12-24t15:30:00Z'
+    ok 152 - correct parse_datetime output seconds for '2012-12-24t15:30:00Z
+    ok 153 - strftime
+    ok 154 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 155 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 156 - correct parse_datetime return value for '2012-12-24t15:30:00z'
+    ok 157 - correct parse_datetime output seconds for '2012-12-24t15:30:00z
+    ok 158 - strftime
+    ok 159 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 160 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 161 - correct parse_datetime return value for '2012-12-24t15:30:00'
+    ok 162 - correct parse_datetime output seconds for '2012-12-24t15:30:00
+    ok 163 - strftime
+    ok 164 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 165 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 166 - correct parse_datetime return value for '2012-12-24t16:30:00+01:00'
+    ok 167 - correct parse_datetime output seconds for '2012-12-24t16:30:00+01:00
+    ok 168 - strftime
+    ok 169 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 170 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 171 - correct parse_datetime return value for '2012-12-24t16:30:00+0100'
+    ok 172 - correct parse_datetime output seconds for '2012-12-24t16:30:00+0100
+    ok 173 - strftime
+    ok 174 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 175 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 176 - correct parse_datetime return value for '2012-12-24t14:30:00-01:00'
+    ok 177 - correct parse_datetime output seconds for '2012-12-24t14:30:00-01:00
+    ok 178 - strftime
+    ok 179 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 180 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 181 - correct parse_datetime return value for '2012-12-24t14:30:00-0100'
+    ok 182 - correct parse_datetime output seconds for '2012-12-24t14:30:00-0100
+    ok 183 - strftime
+    ok 184 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 185 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 186 - correct parse_datetime return value for '2012-12-24t15:30:00.123456Z'
+    ok 187 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456Z
+    ok 188 - strftime
+    ok 189 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 190 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 191 - correct parse_datetime return value for '2012-12-24t15:30:00.123456z'
+    ok 192 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456z
+    ok 193 - strftime
+    ok 194 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 195 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 196 - correct parse_datetime return value for '2012-12-24t16:30:00.123456+01:00'
+    ok 197 - correct parse_datetime output seconds for '2012-12-24t16:30:00.123456+01:00
+    ok 198 - strftime
+    ok 199 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 200 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 201 - correct parse_datetime return value for '2012-12-24t14:30:00.123456-01:00'
+    ok 202 - correct parse_datetime output seconds for '2012-12-24t14:30:00.123456-01:00
+    ok 203 - strftime
+    ok 204 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 205 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 206 - correct parse_datetime return value for '2012-12-24 16:30 +01:00'
+    ok 207 - correct parse_datetime output seconds for '2012-12-24 16:30 +01:00
+    ok 208 - strftime
+    ok 209 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 210 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 211 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
+    ok 212 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
+    ok 213 - strftime
+    ok 214 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 215 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 216 - correct parse_datetime return value for '2012-12-24 15:30 UTC'
+    ok 217 - correct parse_datetime output seconds for '2012-12-24 15:30 UTC
+    ok 218 - strftime
+    ok 219 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 220 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 221 - correct parse_datetime return value for '2012-12-24 16:30 UTC+1'
+    ok 222 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+1
+    ok 223 - strftime
+    ok 224 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 225 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 226 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01'
+    ok 227 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01
+    ok 228 - strftime
+    ok 229 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 230 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 231 - correct parse_datetime return value for '2012-12-24 16:30 UTC+0100'
+    ok 232 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+0100
+    ok 233 - strftime
+    ok 234 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 235 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 236 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01:00'
+    ok 237 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01:00
+    ok 238 - strftime
+    ok 239 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 240 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 241 - correct parse_datetime return value for '2012-12-24 14:30 UTC-1'
+    ok 242 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-1
+    ok 243 - strftime
+    ok 244 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 245 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 246 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01'
+    ok 247 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01
+    ok 248 - strftime
+    ok 249 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 250 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 251 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01:00'
+    ok 252 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01:00
+    ok 253 - strftime
+    ok 254 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 255 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 256 - correct parse_datetime return value for '2012-12-24 14:30 UTC-0100'
+    ok 257 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-0100
+    ok 258 - strftime
+    ok 259 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 260 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 261 - correct parse_datetime return value for '2012-12-24 15:30 GMT'
+    ok 262 - correct parse_datetime output seconds for '2012-12-24 15:30 GMT
+    ok 263 - strftime
+    ok 264 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 265 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 266 - correct parse_datetime return value for '2012-12-24 16:30 GMT+1'
+    ok 267 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+1
+    ok 268 - strftime
+    ok 269 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 270 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 271 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01'
+    ok 272 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01
+    ok 273 - strftime
+    ok 274 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 275 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 276 - correct parse_datetime return value for '2012-12-24 16:30 GMT+0100'
+    ok 277 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+0100
+    ok 278 - strftime
+    ok 279 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 280 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 281 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01:00'
+    ok 282 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01:00
+    ok 283 - strftime
+    ok 284 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 285 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 286 - correct parse_datetime return value for '2012-12-24 14:30 GMT-1'
+    ok 287 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-1
+    ok 288 - strftime
+    ok 289 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 290 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 291 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01'
+    ok 292 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01
+    ok 293 - strftime
+    ok 294 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 295 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 296 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01:00'
+    ok 297 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01:00
+    ok 298 - strftime
+    ok 299 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 300 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 301 - correct parse_datetime return value for '2012-12-24 14:30 GMT-0100'
+    ok 302 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-0100
+    ok 303 - strftime
+    ok 304 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 305 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 306 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
+    ok 307 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
+    ok 308 - strftime
+    ok 309 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 310 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 311 - correct parse_datetime return value for '2012-12-24 16:30:00 +01:00'
+    ok 312 - correct parse_datetime output seconds for '2012-12-24 16:30:00 +01:00
+    ok 313 - strftime
+    ok 314 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 315 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 316 - correct parse_datetime return value for '2012-12-24 14:30:00 -01:00'
+    ok 317 - correct parse_datetime output seconds for '2012-12-24 14:30:00 -01:00
+    ok 318 - strftime
+    ok 319 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 320 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 321 - correct parse_datetime return value for '2012-12-24 16:30:00.123456 +01:00'
+    ok 322 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456 +01:00
+    ok 323 - strftime
+    ok 324 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 325 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 326 - correct parse_datetime return value for '2012-12-24 14:30:00.123456 -01:00'
+    ok 327 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456 -01:00
+    ok 328 - strftime
+    ok 329 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 330 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 331 - correct parse_datetime return value for '2012-12-24 15:30:00.123456 -00:00'
+    ok 332 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 -00:00
+    ok 333 - strftime
+    ok 334 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 335 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 336 - correct parse_datetime return value for '20121224T1630+01:00'
+    ok 337 - correct parse_datetime output seconds for '20121224T1630+01:00
+    ok 338 - strftime
+    ok 339 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 340 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 341 - correct parse_datetime return value for '2012-12-24T1630+01:00'
+    ok 342 - correct parse_datetime output seconds for '2012-12-24T1630+01:00
+    ok 343 - strftime
+    ok 344 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 345 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 346 - correct parse_datetime return value for '20121224T16:30+01'
+    ok 347 - correct parse_datetime output seconds for '20121224T16:30+01
+    ok 348 - strftime
+    ok 349 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 350 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+    ok 351 - correct parse_datetime return value for '20121224T16:30 +01'
+    ok 352 - correct parse_datetime output seconds for '20121224T16:30 +01
+    ok 353 - strftime
+    ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
+    ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
+ok 1 - subtests
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime Timur Safin via Tarantool-patches
                   ` (8 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* created app-tap test for new builtin module `datetime.lua`
* added case to check datetime string formatting using:
  - asctime (gmt time);
  - ctime (local TZ time);
  - strftime (using given format).

* added positive/negative checks to datetime test
  - extended api of datetime.parse_date, .parse_time, .parse_time_zone
    with a length of parsed (sub)string;
  - this allows us to check partially valid strings like "20121224 Foo bar".

Part of #5941
---
 src/lua/datetime.lua           |  14 ++-
 test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++
 2 files changed, 197 insertions(+), 8 deletions(-)
 create mode 100755 test/app-tap/datetime.test.lua

diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 0996ca5a2..f4d2d7737 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -318,7 +318,7 @@ local function datetime_new(o)
             end,
 
             month = function(v)
-                assert(v > 0 and v < 12 )
+                assert(v > 0 and v < 13 )
                 M = v
                 ymd = true
             end,
@@ -393,9 +393,8 @@ end
 
 local function parse_date(str)
     local dt = ffi.new('dt_t[1]')
-    local rc = cdt.dt_parse_iso_date(str, #str, dt)
-    assert(rc > 0)
-    return mk_timestamp(dt[0])
+    local len = cdt.dt_parse_iso_date(str, #str, dt)
+    return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len)
 end
 
 --[[
@@ -411,9 +410,8 @@ end
 local function parse_time(str)
     local sp = ffi.new('int[1]')
     local fp = ffi.new('int[1]')
-    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
-    assert(rc > 0)
-    return mk_timestamp(nil, sp[0], fp[0])
+    local len = cdt.dt_parse_iso_time(str, #str, sp, fp)
+    return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil, tonumber(len)
 end
 
 --[[
@@ -448,7 +446,7 @@ local function parse_str(str)
     str = str:sub(tonumber(n) + 1)
 
     local ch = str:sub(1,1)
-    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
+    if ch:match('[Tt ]') == nil then
         return mk_timestamp(dt_)
     end
 
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
new file mode 100755
index 000000000..09c968858
--- /dev/null
+++ b/test/app-tap/datetime.test.lua
@@ -0,0 +1,191 @@
+#!/usr/bin/env tarantool
+
+local tap = require('tap')
+local test = tap.test("errno")
+local date = require('datetime')
+
+test:plan(5)
+
+test:test("Simple tests for parser", function(test)
+    test:plan(2)
+    test:ok(date("1970-01-01T01:00:00Z") ==
+            date {year=1970, month=1, day=1, hour=1, minute=0, second=0})
+    test:ok(date("1970-01-01T02:00:00+02:00") ==
+            date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120})
+end)
+
+test:test("Multiple tests for parser (with nanoseconds)", function(test)
+    test:plan(165)
+    -- borrowed from p5-time-moments/t/180_from_string.t
+    local tests =
+    {
+        { '1970-01-01T00:00:00Z',                       0,           0,    0 },
+        { '1970-01-01T02:00:00+02:00',                  0,           0,  120 },
+        { '1970-01-01T01:30:00+01:30',                  0,           0,   90 },
+        { '1970-01-01T01:00:00+01:00',                  0,           0,   60 },
+        { '1970-01-01T00:01:00+00:01',                  0,           0,    1 },
+        { '1970-01-01T00:00:00+00:00',                  0,           0,    0 },
+        { '1969-12-31T23:59:00-00:01',                  0,           0,   -1 },
+        { '1969-12-31T23:00:00-01:00',                  0,           0,  -60 },
+        { '1969-12-31T22:30:00-01:30',                  0,           0,  -90 },
+        { '1969-12-31T22:00:00-02:00',                  0,           0, -120 },
+        { '1970-01-01T00:00:00.123456789Z',             0,   123456789,    0 },
+        { '1970-01-01T00:00:00.12345678Z',              0,   123456780,    0 },
+        { '1970-01-01T00:00:00.1234567Z',               0,   123456700,    0 },
+        { '1970-01-01T00:00:00.123456Z',                0,   123456000,    0 },
+        { '1970-01-01T00:00:00.12345Z',                 0,   123450000,    0 },
+        { '1970-01-01T00:00:00.1234Z',                  0,   123400000,    0 },
+        { '1970-01-01T00:00:00.123Z',                   0,   123000000,    0 },
+        { '1970-01-01T00:00:00.12Z',                    0,   120000000,    0 },
+        { '1970-01-01T00:00:00.1Z',                     0,   100000000,    0 },
+        { '1970-01-01T00:00:00.01Z',                    0,    10000000,    0 },
+        { '1970-01-01T00:00:00.001Z',                   0,     1000000,    0 },
+        { '1970-01-01T00:00:00.0001Z',                  0,      100000,    0 },
+        { '1970-01-01T00:00:00.00001Z',                 0,       10000,    0 },
+        { '1970-01-01T00:00:00.000001Z',                0,        1000,    0 },
+        { '1970-01-01T00:00:00.0000001Z',               0,         100,    0 },
+        { '1970-01-01T00:00:00.00000001Z',              0,          10,    0 },
+        { '1970-01-01T00:00:00.000000001Z',             0,           1,    0 },
+        { '1970-01-01T00:00:00.000000009Z',             0,           9,    0 },
+        { '1970-01-01T00:00:00.00000009Z',              0,          90,    0 },
+        { '1970-01-01T00:00:00.0000009Z',               0,         900,    0 },
+        { '1970-01-01T00:00:00.000009Z',                0,        9000,    0 },
+        { '1970-01-01T00:00:00.00009Z',                 0,       90000,    0 },
+        { '1970-01-01T00:00:00.0009Z',                  0,      900000,    0 },
+        { '1970-01-01T00:00:00.009Z',                   0,     9000000,    0 },
+        { '1970-01-01T00:00:00.09Z',                    0,    90000000,    0 },
+        { '1970-01-01T00:00:00.9Z',                     0,   900000000,    0 },
+        { '1970-01-01T00:00:00.99Z',                    0,   990000000,    0 },
+        { '1970-01-01T00:00:00.999Z',                   0,   999000000,    0 },
+        { '1970-01-01T00:00:00.9999Z',                  0,   999900000,    0 },
+        { '1970-01-01T00:00:00.99999Z',                 0,   999990000,    0 },
+        { '1970-01-01T00:00:00.999999Z',                0,   999999000,    0 },
+        { '1970-01-01T00:00:00.9999999Z',               0,   999999900,    0 },
+        { '1970-01-01T00:00:00.99999999Z',              0,   999999990,    0 },
+        { '1970-01-01T00:00:00.999999999Z',             0,   999999999,    0 },
+        { '1970-01-01T00:00:00.0Z',                     0,           0,    0 },
+        { '1970-01-01T00:00:00.00Z',                    0,           0,    0 },
+        { '1970-01-01T00:00:00.000Z',                   0,           0,    0 },
+        { '1970-01-01T00:00:00.0000Z',                  0,           0,    0 },
+        { '1970-01-01T00:00:00.00000Z',                 0,           0,    0 },
+        { '1970-01-01T00:00:00.000000Z',                0,           0,    0 },
+        { '1970-01-01T00:00:00.0000000Z',               0,           0,    0 },
+        { '1970-01-01T00:00:00.00000000Z',              0,           0,    0 },
+        { '1970-01-01T00:00:00.000000000Z',             0,           0,    0 },
+        { '1973-11-29T21:33:09Z',               123456789,           0,    0 },
+        { '2013-10-28T17:51:56Z',              1382982716,           0,    0 },
+        -- { '9999-12-31T23:59:59Z',            253402300799,           0,    0 },
+    }
+    for _, value in ipairs(tests) do
+        local str, epoch, nsec, offset
+        str, epoch, nsec, offset = unpack(value)
+        local dt = date(str)
+        test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
+        test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
+        test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
+    end
+end)
+
+local ffi = require('ffi')
+
+ffi.cdef [[
+    void tzset(void);
+]]
+
+test:test("Datetime string formatting", function(test)
+    test:plan(7)
+    local str = "1970-01-01"
+    local t = date(str)
+    test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, tonumber(t.secs)))
+    test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec))
+    test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset))
+    test:ok(date.asctime(t) == 'Thu Jan  1 00:00:00 1970\n', ('%s: asctime'):format(str))
+    -- ctime() is local timezone dependent. To make sure that
+    -- test is deterministic we enforce timezone via TZ environment
+    -- manipulations and calling tzset()
+
+    -- redefine timezone to be always GMT-2
+    os.setenv('TZ', 'GMT-2')
+    ffi.C.tzset()
+    test:ok(date.ctime(t) == 'Thu Jan  1 02:00:00 1970\n', ('%s: ctime with timezone'):format(str))
+    test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime #1'):format(str))
+    test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January 1970', ('%s: strftime #2'):format(str))
+end)
+
+test:test("Parse iso date - valid strings", function(test)
+    test:plan(32)
+    local good = {
+        {2012, 12, 24, "20121224",                   8 },
+        {2012, 12, 24, "20121224  Foo bar",          8 },
+        {2012, 12, 24, "2012-12-24",                10 },
+        {2012, 12, 24, "2012-12-24 23:59:59",       10 },
+        {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 },
+        {2012, 12, 24, "2012359",                    7 },
+        {2012, 12, 24, "2012359T235959+0130",        7 },
+        {2012, 12, 24, "2012-359",                   8 },
+        {2012, 12, 24, "2012W521",                   8 },
+        {2012, 12, 24, "2012-W52-1",                10 },
+        {2012, 12, 24, "2012Q485",                   8 },
+        {2012, 12, 24, "2012-Q4-85",                10 },
+        {   1,  1,  1, "0001-Q1-01",                10 },
+        {   1,  1,  1, "0001-W01-1",                10 },
+        {   1,  1,  1, "0001-01-01",                10 },
+        {   1,  1,  1, "0001-001",                   8 },
+    }
+
+    for _, value in ipairs(good) do
+        local year, month, day, str, date_part_len;
+        year, month, day, str, date_part_len = unpack(value)
+        local expected_date = date{year = year, month = month, day = day}
+        local date_part, len
+        date_part, len = date.parse_date(str)
+        test:ok(len == date_part_len, ('%s: length check %d'):format(str, len))
+        test:ok(expected_date == date_part, ('%s: expected date'):format(str))
+    end
+end)
+
+test:test("Parse iso date - invalid strings", function(test)
+    test:plan(62)
+    local bad = {
+        "20121232"   , -- Invalid day of month
+        "2012-12-310", -- Invalid day of month
+        "2012-13-24" , -- Invalid month
+        "2012367"    , -- Invalid day of year
+        "2012-000"   , -- Invalid day of year
+        "2012W533"   , -- Invalid week of year
+        "2012-W52-8" , -- Invalid day of week
+        "2012Q495"   , -- Invalid day of quarter
+        "2012-Q5-85" , -- Invalid quarter
+        "20123670"   , -- Trailing digit
+        "201212320"  , -- Trailing digit
+        "2012-12"    , -- Reduced accuracy
+        "2012-Q4"    , -- Reduced accuracy
+        "2012-Q42"   , -- Invalid
+        "2012-Q1-1"  , -- Invalid day of quarter
+        "2012Q--420" , -- Invalid
+        "2012-Q-420" , -- Invalid
+        "2012Q11"    , -- Incomplete
+        "2012Q1234"  , -- Trailing digit
+        "2012W12"    , -- Incomplete
+        "2012W1234"  , -- Trailing digit
+        "2012W-123"  , -- Invalid
+        "2012-W12"   , -- Incomplete
+        "2012-W12-12", -- Trailing digit
+        "2012U1234"  , -- Invalid
+        "2012-1234"  , -- Invalid
+        "2012-X1234" , -- Invalid
+        "0000-Q1-01" , -- Year less than 0001
+        "0000-W01-1" , -- Year less than 0001
+        "0000-01-01" , -- Year less than 0001
+        "0000-001"   , -- Year less than 0001
+    }
+
+    for _, str in ipairs(bad) do
+        local date_part, len
+        date_part, len = date.parse_date(str)
+        test:ok(len == 0, ('%s: length check %d'):format(str, len))
+        test:ok(date_part == nil, ('%s: empty date check %s'):format(str, date_part))
+    end
+end)
+
+os.exit(test:check() and 0 or 1)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (2 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 05/11] box, datetime: add messagepack support for datetime Timur Safin via Tarantool-patches
                   ` (7 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* introduced output routine for converting datetime
  to their default output format.

* use this routine for tostring() in datetime.lua

Part of #5941
---
 src/exports.h               |  1 +
 src/lib/core/CMakeLists.txt |  1 +
 src/lib/core/datetime.c     | 85 +++++++++++++++++++++++++++++++++++++
 src/lib/core/datetime.h     | 14 ++++++
 src/lua/datetime.lua        | 18 ++++++--
 5 files changed, 116 insertions(+), 3 deletions(-)
 create mode 100755 src/lib/core/datetime.c

diff --git a/src/exports.h b/src/exports.h
index db40c03a4..1a03db636 100644
--- a/src/exports.h
+++ b/src/exports.h
@@ -552,3 +552,4 @@ EXPORT(dt_parse_iso_zone_extended)
 EXPORT(dt_parse_iso_zone_lenient)
 EXPORT(dt_from_struct_tm)
 EXPORT(dt_to_struct_tm)
+EXPORT(datetime_to_string)
diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
index 2cd4d0b4f..8bc776b82 100644
--- a/src/lib/core/CMakeLists.txt
+++ b/src/lib/core/CMakeLists.txt
@@ -30,6 +30,7 @@ set(core_sources
     decimal.c
     mp_decimal.c
     cord_buf.c
+    datetime.c
 )
 
 if (TARGET_OS_NETBSD)
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
new file mode 100755
index 000000000..65f813a70
--- /dev/null
+++ b/src/lib/core/datetime.c
@@ -0,0 +1,85 @@
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <string.h>
+
+#include "trivia/util.h"
+#include "datetime.h"
+
+int
+datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len)
+{
+	char * src = buf;
+	int offset = date->offset;
+	int64_t secs = date->secs + offset * 60;
+	dt_t dt = dt_from_rdn((secs / SECS_PER_DAY) + 719163);
+
+	int year, month, day, sec, ns, sign;
+	dt_to_ymd(dt, &year, &month, &day);
+
+	int hour = (secs / 3600) % 24,
+	    minute = (secs / 60) % 60;
+	;
+	sec = secs % 60;
+	ns = date->nsec;
+	uint32_t sz;
+	sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
+		      year, month, day, hour, minute);
+	buf += sz; len -= sz;
+	if (sec || ns) {
+		sz = snprintf(buf, len, ":%02d", sec);
+		buf += sz; len -= sz;
+		if (ns) {
+			if ((ns % 1000000) == 0)
+				sz = snprintf(buf, len, ".%03d", ns / 1000000);
+			else if ((ns % 1000) == 0)
+				sz = snprintf(buf, len, ".%06d", ns / 1000);
+			else
+				sz = snprintf(buf, len, ".%09d", ns);
+			buf += sz; len -= sz;
+		}
+	}
+	if (offset == 0) {
+		strncpy(buf, "Z", len);
+		buf++;
+		len--;
+	}
+	else {
+		if (offset < 0)
+			sign = '-', offset = -offset;
+		else
+			sign = '+';
+
+		sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
+		buf += sz; len -= sz;
+	}
+	return (buf - src);
+}
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 403bf1c64..fdaff2d27 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -38,6 +38,11 @@
 extern "C" {
 #endif /* defined(__cplusplus) */
 
+#ifndef SECS_PER_DAY
+#define SECS_PER_DAY	86400
+#define NANOS_PER_SEC	1000000000
+#endif
+
 /**
  * datetime structure consisting of:
  */
@@ -55,6 +60,15 @@ struct datetime_interval_t {
 	int32_t nsec; ///< nanoseconds delta
 };
 
+/**
+ * Convert datetime to string using default format
+ * @param date source datetime value
+ * @param buf output character buffer
+ * @param len size ofoutput buffer
+ */
+int
+datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
+
 #if defined(__cplusplus)
 } /* extern "C" */
 #endif /* defined(__cplusplus) */
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index f4d2d7737..9ec06d8d8 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -53,6 +53,12 @@ ffi.cdef [[
     dt_t    dt_from_struct_tm  (const struct tm *tm);
     void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
 
+    // mp_datetime.c
+
+    int
+    datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
+
+
     // <asm-generic/posix_types.h>
     typedef long            __kernel_long_t;
     typedef unsigned long   __kernel_ulong_t;
@@ -549,8 +555,13 @@ local function strftime(fmt, o)
     return ffi.string(buff)
 end
 
--- strftime may be redirected to datetime:fmt("format")
-local function datetime_fmt()
+local function datetime_tostring(o)
+    assert(ffi.typeof(o) == datetime_t)
+    local sz = 48
+    local buff = ffi.new('char[?]', sz)
+    local len = native.datetime_to_string(o, buff, sz)
+    assert(len < sz)
+    return ffi.string(buff)
 end
 
 
@@ -566,7 +577,8 @@ return setmetatable(
         parse_date = parse_date,
         parse_time = parse_time,
         parse_zone = parse_zone,
-        fmt = datetime_fmt,
+
+        tostring = datetime_tostring,
 
         now = local_now,
     -- strptime = strptime;
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 05/11] box, datetime: add messagepack support for datetime
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (3 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
                   ` (6 subsequent siblings)
  11 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

Serialize datetime_t as newly introduced MP_EXT type.
It saves 1 required integer field and upto 2 optional
unsigned fields in very compact fashion.
- secs is required field;
- but nsec, offset are both optional;

* json, yaml serialization formats, lua output mode
  supported;

Part of #5941
Part of #5946
---
 src/box/field_def.c               |  34 ++++---
 src/box/field_def.h               |   1 +
 src/box/lua/serialize_lua.c       |   7 +-
 src/box/msgpack.c                 |   7 +-
 src/box/tuple_compare.cc          |  20 ++++
 src/exports.h                     |   2 +
 src/lib/core/CMakeLists.txt       |   3 +-
 src/lib/core/datetime.c           | 161 ++++++++++++++++++++++++++++++
 src/lib/core/datetime.h           |  41 +++++++-
 src/lib/core/mp_extension_types.h |   1 +
 src/lib/mpstream/mpstream.c       |  11 ++
 src/lib/mpstream/mpstream.h       |   4 +
 src/lua/msgpack.c                 |  12 +++
 src/lua/msgpackffi.lua            |   8 ++
 src/lua/serializer.c              |   4 +
 src/lua/serializer.h              |   2 +
 src/lua/utils.c                   |   1 -
 test/unit/datetime.c              |   1 -
 third_party/lua-cjson/lua_cjson.c |   8 ++
 third_party/lua-yaml/lyaml.cc     |   6 +-
 20 files changed, 312 insertions(+), 22 deletions(-)

diff --git a/src/box/field_def.c b/src/box/field_def.c
index 51acb8025..6964e3e9f 100644
--- a/src/box/field_def.c
+++ b/src/box/field_def.c
@@ -67,11 +67,12 @@ const uint32_t field_mp_type[] = {
 	/* [FIELD_TYPE_VARBINARY] =  */ 1U << MP_BIN,
 	/* [FIELD_TYPE_SCALAR]   =  */ (1U << MP_UINT) | (1U << MP_INT) |
 		(1U << MP_FLOAT) | (1U << MP_DOUBLE) | (1U << MP_STR) |
-		(1U << MP_BIN) | (1U << MP_BOOL),
+		(1U << MP_BIN) | (1U << MP_BOOL) | (1U << MP_DATETIME),
 	/* [FIELD_TYPE_DECIMAL]  =  */ 0, /* only MP_DECIMAL is supported */
 	/* [FIELD_TYPE_UUID]     =  */ 0, /* only MP_UUID is supported */
 	/* [FIELD_TYPE_ARRAY]    =  */ 1U << MP_ARRAY,
 	/* [FIELD_TYPE_MAP]      =  */ (1U << MP_MAP),
+	/* [FIELD_TYPE_DATETIME] =  */ 0, /* only MP_DATETIME is supported */
 };
 
 const uint32_t field_ext_type[] = {
@@ -88,6 +89,7 @@ const uint32_t field_ext_type[] = {
 	/* [FIELD_TYPE_UUID]      = */ 1U << MP_UUID,
 	/* [FIELD_TYPE_ARRAY]     = */ 0,
 	/* [FIELD_TYPE_MAP]       = */ 0,
+	/* [FIELD_TYPE_DATETIME]  = */ 1U << MP_DATETIME,
 };
 
 const char *field_type_strs[] = {
@@ -104,6 +106,7 @@ const char *field_type_strs[] = {
 	/* [FIELD_TYPE_UUID]     = */ "uuid",
 	/* [FIELD_TYPE_ARRAY]    = */ "array",
 	/* [FIELD_TYPE_MAP]      = */ "map",
+	/* [FIELD_TYPE_DATETIME] = */ "datetime",
 };
 
 const char *on_conflict_action_strs[] = {
@@ -128,20 +131,21 @@ field_type_by_name_wrapper(const char *str, uint32_t len)
  * values can be stored in the j type.
  */
 static const bool field_type_compatibility[] = {
-	   /*   ANY   UNSIGNED  STRING   NUMBER  DOUBLE  INTEGER  BOOLEAN VARBINARY SCALAR  DECIMAL   UUID    ARRAY    MAP  */
-/*   ANY    */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   false,   false,
-/* UNSIGNED */ true,   true,    false,   true,    false,   true,    false,   false,  true,   false,  false,   false,   false,
-/*  STRING  */ true,   false,   true,    false,   false,   false,   false,   false,  true,   false,  false,   false,   false,
-/*  NUMBER  */ true,   false,   false,   true,    false,   false,   false,   false,  true,   false,  false,   false,   false,
-/*  DOUBLE  */ true,   false,   false,   true,    true,    false,   false,   false,  true,   false,  false,   false,   false,
-/*  INTEGER */ true,   false,   false,   true,    false,   true,    false,   false,  true,   false,  false,   false,   false,
-/*  BOOLEAN */ true,   false,   false,   false,   false,   false,   true,    false,  true,   false,  false,   false,   false,
-/* VARBINARY*/ true,   false,   false,   false,   false,   false,   false,   true,   true,   false,  false,   false,   false,
-/*  SCALAR  */ true,   false,   false,   false,   false,   false,   false,   false,  true,   false,  false,   false,   false,
-/*  DECIMAL */ true,   false,   false,   true,    false,   false,   false,   false,  true,   true,   false,   false,   false,
-/*   UUID   */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  true,    false,   false,
-/*   ARRAY  */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   true,    false,
-/*    MAP   */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   false,   true,
+	   /*   ANY   UNSIGNED  STRING   NUMBER  DOUBLE  INTEGER  BOOLEAN VARBINARY SCALAR  DECIMAL   UUID    ARRAY    MAP     DATETIME */
+/*   ANY    */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   false,   false,   false,
+/* UNSIGNED */ true,   true,    false,   true,    false,   true,    false,   false,  true,   false,  false,   false,   false,   false,
+/*  STRING  */ true,   false,   true,    false,   false,   false,   false,   false,  true,   false,  false,   false,   false,   false,
+/*  NUMBER  */ true,   false,   false,   true,    false,   false,   false,   false,  true,   false,  false,   false,   false,   false,
+/*  DOUBLE  */ true,   false,   false,   true,    true,    false,   false,   false,  true,   false,  false,   false,   false,   false,
+/*  INTEGER */ true,   false,   false,   true,    false,   true,    false,   false,  true,   false,  false,   false,   false,   false,
+/*  BOOLEAN */ true,   false,   false,   false,   false,   false,   true,    false,  true,   false,  false,   false,   false,   false,
+/* VARBINARY*/ true,   false,   false,   false,   false,   false,   false,   true,   true,   false,  false,   false,   false,   false,
+/*  SCALAR  */ true,   false,   false,   false,   false,   false,   false,   false,  true,   false,  false,   false,   false,   false,
+/*  DECIMAL */ true,   false,   false,   true,    false,   false,   false,   false,  true,   true,   false,   false,   false,   false,
+/*   UUID   */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  true,    false,   false,   false,
+/*   ARRAY  */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   true,    false,   false,
+/*    MAP   */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   false,   true,    false,
+/* DATETIME */ true,   false,   false,   false,   false,   false,   false,   false,  false,  false,  false,   false,   false,   true,
 };
 
 bool
diff --git a/src/box/field_def.h b/src/box/field_def.h
index c5cfe5e86..120b2a93d 100644
--- a/src/box/field_def.h
+++ b/src/box/field_def.h
@@ -63,6 +63,7 @@ enum field_type {
 	FIELD_TYPE_UUID,
 	FIELD_TYPE_ARRAY,
 	FIELD_TYPE_MAP,
+	FIELD_TYPE_DATETIME,
 	field_type_MAX
 };
 
diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
index 7144305cf..5cfa6f5e0 100644
--- a/src/box/lua/serialize_lua.c
+++ b/src/box/lua/serialize_lua.c
@@ -768,7 +768,7 @@ static int
 dump_node(struct lua_dumper *d, struct node *nd, int indent)
 {
 	struct luaL_field *field = &nd->field;
-	char buf[FPCONV_G_FMT_BUFSIZE];
+	char buf[FPCONV_G_FMT_BUFSIZE + 8];
 	int ltype = lua_type(d->L, -1);
 	const char *str = NULL;
 	size_t len = 0;
@@ -861,6 +861,11 @@ dump_node(struct lua_dumper *d, struct node *nd, int indent)
 			str = tt_uuid_str(field->uuidval);
 			len = UUID_STR_LEN;
 			break;
+		case MP_DATETIME:
+			nd->mask |= NODE_QUOTE;
+			str = buf;
+			len = datetime_to_string(field->dateval, buf, sizeof buf);
+			break;
 		default:
 			d->err = EINVAL;
 			snprintf(d->err_msg, sizeof(d->err_msg),
diff --git a/src/box/msgpack.c b/src/box/msgpack.c
index 1723dea4c..e53af548c 100644
--- a/src/box/msgpack.c
+++ b/src/box/msgpack.c
@@ -1,5 +1,5 @@
 /*
- * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file.
+ * Copyright 2020-2021, Tarantool AUTHORS, please see AUTHORS file.
  *
  * Redistribution and use in source and binary forms, with or
  * without modification, are permitted provided that the following
@@ -35,6 +35,7 @@
 #include "mp_decimal.h"
 #include "uuid/mp_uuid.h"
 #include "mp_error.h"
+#include "datetime.h"
 
 static int
 msgpack_fprint_ext(FILE *file, const char **data, int depth)
@@ -47,6 +48,8 @@ msgpack_fprint_ext(FILE *file, const char **data, int depth)
 		return mp_fprint_decimal(file, data, len);
 	case MP_UUID:
 		return mp_fprint_uuid(file, data, len);
+	case MP_DATETIME:
+		return mp_fprint_datetime(file, data, len);
 	case MP_ERROR:
 		return mp_fprint_error(file, data, depth);
 	default:
@@ -65,6 +68,8 @@ msgpack_snprint_ext(char *buf, int size, const char **data, int depth)
 		return mp_snprint_decimal(buf, size, data, len);
 	case MP_UUID:
 		return mp_snprint_uuid(buf, size, data, len);
+	case MP_DATETIME:
+		return mp_snprint_datetime(buf, size, data, len);
 	case MP_ERROR:
 		return mp_snprint_error(buf, size, data, depth);
 	default:
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index 98938fb39..48c833643 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -36,6 +36,7 @@
 #include "lib/core/decimal.h"
 #include "lib/core/mp_decimal.h"
 #include "uuid/mp_uuid.h"
+#include "core/datetime.h"
 #include "lib/core/mp_extension_types.h"
 
 /* {{{ tuple_compare */
@@ -76,6 +77,7 @@ enum mp_class {
 	MP_CLASS_STR,
 	MP_CLASS_BIN,
 	MP_CLASS_UUID,
+	MP_CLASS_DATETIME,
 	MP_CLASS_ARRAY,
 	MP_CLASS_MAP,
 	mp_class_max,
@@ -99,6 +101,8 @@ static enum mp_class mp_ext_classes[] = {
 	/* .MP_UNKNOWN_EXTENSION = */ mp_class_max, /* unsupported */
 	/* .MP_DECIMAL		 = */ MP_CLASS_NUMBER,
 	/* .MP_UUID		 = */ MP_CLASS_UUID,
+	/* .MP_ERROR		 = */ mp_class_max,
+	/* .MP_DATETIME		 = */ MP_CLASS_DATETIME,
 };
 
 static enum mp_class
@@ -390,6 +394,19 @@ mp_compare_uuid(const char *field_a, const char *field_b)
 	return memcmp(field_a + 2, field_b + 2, UUID_PACKED_LEN);
 }
 
+static int
+mp_compare_datetime(const char *lhs, const char *rhs)
+{
+	datetime_t lhs_dt, rhs_dt;
+	datetime_t *ret;
+	ret = mp_decode_datetime(&lhs, &lhs_dt);
+	assert(ret != NULL);
+	ret = mp_decode_datetime(&rhs, &rhs_dt);
+	assert(ret != NULL);
+	(void)ret;
+	return datetime_compare(&lhs_dt, &rhs_dt);
+}
+
 typedef int (*mp_compare_f)(const char *, const char *);
 static mp_compare_f mp_class_comparators[] = {
 	/* .MP_CLASS_NIL    = */ NULL,
@@ -398,6 +415,7 @@ static mp_compare_f mp_class_comparators[] = {
 	/* .MP_CLASS_STR    = */ mp_compare_str,
 	/* .MP_CLASS_BIN    = */ mp_compare_bin,
 	/* .MP_CLASS_UUID   = */ mp_compare_uuid,
+	/* .MP_CLASS_DATETIME=*/ mp_compare_datetime,
 	/* .MP_CLASS_ARRAY  = */ NULL,
 	/* .MP_CLASS_MAP    = */ NULL,
 };
@@ -478,6 +496,8 @@ tuple_compare_field(const char *field_a, const char *field_b,
 		return mp_compare_decimal(field_a, field_b);
 	case FIELD_TYPE_UUID:
 		return mp_compare_uuid(field_a, field_b);
+	case FIELD_TYPE_DATETIME:
+		return mp_compare_datetime(field_a, field_b);
 	default:
 		unreachable();
 		return 0;
diff --git a/src/exports.h b/src/exports.h
index 1a03db636..586444b65 100644
--- a/src/exports.h
+++ b/src/exports.h
@@ -531,6 +531,8 @@ EXPORT(uri_format)
 EXPORT(uri_parse)
 EXPORT(uuid_nil)
 EXPORT(uuid_unpack)
+EXPORT(datetime_unpack)
+EXPORT(datetime_pack)
 EXPORT(dt_from_rdn)
 EXPORT(dt_from_yd)
 EXPORT(dt_from_ymd)
diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
index 8bc776b82..c3e68c36f 100644
--- a/src/lib/core/CMakeLists.txt
+++ b/src/lib/core/CMakeLists.txt
@@ -44,7 +44,8 @@ add_library(core STATIC ${core_sources})
 
 target_link_libraries(core salad small uri decNumber bit ${LIBEV_LIBRARIES}
                       ${LIBEIO_LIBRARIES} ${LIBCORO_LIBRARIES}
-                      ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES})
+                      ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}
+                      ${LIBCDT_LIBRARIES})
 
 if (ENABLE_BACKTRACE AND NOT TARGET_OS_DARWIN)
     target_link_libraries(core gcc_s ${UNWIND_LIBRARIES})
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index 65f813a70..26ba8a702 100755
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -33,6 +33,132 @@
 
 #include "trivia/util.h"
 #include "datetime.h"
+#include "msgpuck.h"
+#include "mp_extension_types.h"
+
+/*
+  Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length)
+  extension, which creates container of 1 to 3 integers.
+
+  +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+  |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset (uint) |
+  +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+
+  MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
+
+  - signed integer seconds part (required). Depending on the value of
+    seconds it may be from 1 to 8 bytes positive or negative integer number;
+
+  - [optional] fraction time in nanoseconds as unsigned integer.
+    If this value is 0 then it's not saved (unless there is offset field,
+    as below);
+
+  - [optional] timzeone offset in minutes as unsigned integer.
+    If this field is 0 then it's not saved.
+ */
+
+static inline uint32_t
+mp_sizeof_Xint(int64_t n)
+{
+	return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
+}
+
+static inline char *
+mp_encode_Xint(char *data, int64_t v)
+{
+	return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
+}
+	
+static inline int64_t
+mp_decode_Xint(const char **data)
+{
+	switch (mp_typeof(**data)) {
+	case MP_UINT:
+		return (int64_t)mp_decode_uint(data);
+	case MP_INT:
+		return mp_decode_int(data);
+	default:
+		mp_unreachable();
+	}
+	return 0;
+}
+
+uint32_t
+mp_sizeof_datetime(const struct datetime_t *date)
+{
+	uint32_t sz = mp_sizeof_Xint(date->secs);
+
+	// even if nanosecs == 0 we need to output anything
+	// if we have non-null tz offset
+	if (date->nsec != 0 || date->offset != 0)
+		sz += mp_sizeof_Xint(date->nsec);
+	if (date->offset)
+		sz += mp_sizeof_Xint(date->offset);
+
+	return sz;
+}
+
+struct datetime_t *
+datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
+{
+	const char * svp = *data;
+
+	memset(date, 0, sizeof(*date));
+
+	date->secs = mp_decode_Xint(data);
+
+	len -= *data - svp;
+	if (len <= 0)
+		return date;
+
+	svp = *data;
+	date->secs = mp_decode_Xint(data);
+	len -= *data - svp;
+
+	if (len <= 0)
+		return date;
+
+	date->offset = mp_decode_Xint(data);
+
+	return date;
+}
+
+struct datetime_t *
+mp_decode_datetime(const char **data, struct datetime_t *date)
+{
+	if (mp_typeof(**data) != MP_EXT)
+		return NULL;
+
+	int8_t type;
+	uint32_t len = mp_decode_extl(data, &type);
+
+	if (type != MP_DATETIME || len == 0) {
+		return NULL;
+	}
+	return datetime_unpack(data, len, date);
+}
+
+char *
+datetime_pack(char *data, const struct datetime_t *date)
+{
+	data = mp_encode_Xint(data, date->secs);
+	if (date->nsec != 0 || date->offset != 0)
+		data = mp_encode_Xint(data, date->nsec);
+	if (date->offset)
+		data = mp_encode_Xint(data, date->offset);
+
+	return data;
+}
+
+char *
+mp_encode_datetime(char *data, const struct datetime_t *date)
+{
+	uint32_t len = mp_sizeof_datetime(date);
+
+	data = mp_encode_extl(data, MP_DATETIME, len);
+
+	return datetime_pack(data, date);
+}
 
 int
 datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len)
@@ -83,3 +209,38 @@ datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len)
 	}
 	return (buf - src);
 }
+
+int
+mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len)
+{
+	struct datetime_t date = {0};
+
+	if (datetime_unpack(data, len, &date) == NULL)
+		return -1;
+
+	return datetime_to_string(&date, buf, size);
+}
+
+int
+mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
+{
+	struct  datetime_t date;
+
+	if (datetime_unpack(data, len, &date) == NULL)
+		return -1;
+
+	char buf[48];
+	datetime_to_string(&date, buf, sizeof buf);
+
+	return fprintf(file, "%s", buf);
+}
+
+int datetime_compare(const struct datetime_t *lhs, const struct datetime_t *rhs)
+{
+	int result = COMPARE_RESULT(lhs->secs, rhs->secs);
+	if (result != 0)
+		return result;
+
+	return COMPARE_RESULT(lhs->nsec, rhs->nsec);
+}
+
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index fdaff2d27..6e78d8ddb 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -30,9 +30,10 @@
  * SUCH DAMAGE.
  */
 
-#include <c-dt/dt_core.h>
+#include <c-dt/dt.h>
 #include <stdint.h>
 #include <stdbool.h>
+#include <stdio.h>
 
 #if defined(__cplusplus)
 extern "C" {
@@ -60,6 +61,38 @@ struct datetime_interval_t {
 	int32_t nsec; ///< nanoseconds delta
 };
 
+int
+datetime_compare(const struct datetime_t * lhs,
+		 const struct datetime_t * rhs);
+
+
+struct datetime_t *
+datetime_unpack(const char **data, uint32_t len, struct datetime_t *date);
+
+/**
+ * Pack datetime_t data to the MessagePack buffer.
+ */
+char *
+datetime_pack(char *data, const struct datetime_t *date);
+
+/**
+ * Calculate size of MessagePack buffer for datetime_t data.
+ */
+uint32_t
+mp_sizeof_datetime(const struct datetime_t *date);
+
+/**
+ * Decode data from MessagePack buffer to datetime_t structure.
+ */
+struct datetime_t *
+mp_decode_datetime(const char **data, struct datetime_t *date);
+
+/**
+ * Encode datetime_t structure to the MessagePack buffer.
+ */
+char *
+mp_encode_datetime(char *data, const struct datetime_t *date);
+
 /**
  * Convert datetime to string using default format
  * @param date source datetime value
@@ -69,6 +102,12 @@ struct datetime_interval_t {
 int
 datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
 
+int
+mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len);
+
+int
+mp_fprint_datetime(FILE *file, const char **data, uint32_t len);
+
 #if defined(__cplusplus)
 } /* extern "C" */
 #endif /* defined(__cplusplus) */
diff --git a/src/lib/core/mp_extension_types.h b/src/lib/core/mp_extension_types.h
index e3ff9f5d0..3b7eaee7c 100644
--- a/src/lib/core/mp_extension_types.h
+++ b/src/lib/core/mp_extension_types.h
@@ -44,6 +44,7 @@ enum mp_extension_type {
     MP_DECIMAL = 1,
     MP_UUID = 2,
     MP_ERROR = 3,
+    MP_DATETIME = 4,
     mp_extension_type_MAX,
 };
 
diff --git a/src/lib/mpstream/mpstream.c b/src/lib/mpstream/mpstream.c
index 70ca29889..1077e3b19 100644
--- a/src/lib/mpstream/mpstream.c
+++ b/src/lib/mpstream/mpstream.c
@@ -35,6 +35,7 @@
 #include "msgpuck.h"
 #include "mp_decimal.h"
 #include "uuid/mp_uuid.h"
+#include "core/datetime.h"
 
 void
 mpstream_reserve_slow(struct mpstream *stream, size_t size)
@@ -208,6 +209,16 @@ mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid)
 	mpstream_advance(stream, pos - data);
 }
 
+void
+mpstream_encode_datetime(struct mpstream *stream, const struct datetime_t *val)
+{
+	char *data = mpstream_reserve(stream, mp_sizeof_datetime(val));
+	if (data == NULL)
+		return;
+	char *pos = mp_encode_datetime(data, val);
+	mpstream_advance(stream, pos - data);
+}
+
 void
 mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n)
 {
diff --git a/src/lib/mpstream/mpstream.h b/src/lib/mpstream/mpstream.h
index a60add143..540e9a666 100644
--- a/src/lib/mpstream/mpstream.h
+++ b/src/lib/mpstream/mpstream.h
@@ -39,6 +39,7 @@ extern "C" {
 #endif /* defined(__cplusplus) */
 
 struct tt_uuid;
+struct datetime_t;
 
 /**
 * Ask the allocator to reserve at least size bytes. It can reserve
@@ -145,6 +146,9 @@ mpstream_encode_decimal(struct mpstream *stream, const decimal_t *val);
 void
 mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid);
 
+void
+mpstream_encode_datetime(struct mpstream *stream, const struct datetime_t *dt);
+
 /** Copies n bytes from memory area src to stream. */
 void
 mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n);
diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c
index b6ecf2b1e..0a4ba8129 100644
--- a/src/lua/msgpack.c
+++ b/src/lua/msgpack.c
@@ -46,6 +46,7 @@
 #include "lib/core/decimal.h" /* decimal_unpack() */
 #include "lib/uuid/mp_uuid.h" /* mp_decode_uuid() */
 #include "lib/core/mp_extension_types.h"
+#include "datetime.h"
 
 #include "cord_buf.h"
 #include <fiber.h>
@@ -200,6 +201,9 @@ restart: /* used by MP_EXT of unidentified subtype */
 			break;
 		case MP_ERROR:
 			return luamp_encode_extension(L, top, stream);
+		case MP_DATETIME:
+			mpstream_encode_datetime(stream, field->dateval);
+			break;
 		default:
 			/* Run trigger if type can't be encoded */
 			type = luamp_encode_extension(L, top, stream);
@@ -333,6 +337,14 @@ luamp_decode(struct lua_State *L, struct luaL_serializer *cfg,
 				goto ext_decode_err;
 			return;
 		}
+		case MP_DATETIME:
+		{
+			struct datetime_t * date = luaL_pushdatetime(L);
+			date = datetime_unpack(data, len, date);
+			if (date == NULL)
+				goto ext_decode_err;
+			return;
+		}
 		default:
 			/* reset data to the extension header */
 			*data = svp;
diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua
index 1d54f11b8..271be857a 100644
--- a/src/lua/msgpackffi.lua
+++ b/src/lua/msgpackffi.lua
@@ -36,6 +36,8 @@ decimal_t *
 decimal_unpack(const char **data, uint32_t len, decimal_t *dec);
 struct tt_uuid *
 uuid_unpack(const char **data, uint32_t len, struct tt_uuid *uuid);
+struct datetime_t *
+datetime_unpack(const char **data, uint32_t len, struct datetime_t *date);
 ]])
 
 local strict_alignment = (jit.arch == 'arm')
@@ -513,6 +515,12 @@ local ext_decoder = {
         builtin.uuid_unpack(data, len, uuid)
         return uuid
     end,
+    -- MP_DATETIME
+    [4] = function(data, len)
+        local dt = ffi.new("struct datetime_t")
+        builtin.datetime_unpack(data, len, dt)
+        return dt
+    end,
 }
 
 local function decode_ext(data)
diff --git a/src/lua/serializer.c b/src/lua/serializer.c
index 8db6746a3..c27e62c62 100644
--- a/src/lua/serializer.c
+++ b/src/lua/serializer.c
@@ -41,6 +41,7 @@
 #include "lib/core/mp_extension_types.h"
 #include "lua/error.h"
 
+#include "datetime.h"
 #include "trivia/util.h"
 #include "diag.h"
 #include "serializer_opts.h"
@@ -544,6 +545,9 @@ luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
 				   opts != NULL &&
 				   opts->error_marshaling_enabled) {
 				field->ext_type = MP_ERROR;
+			} else if (cd->ctypeid == CTID_DATETIME) {
+				field->ext_type = MP_DATETIME;
+				field->dateval = (struct datetime_t*) cdata;
 			} else {
 				field->ext_type = MP_UNKNOWN_EXTENSION;
 			}
diff --git a/src/lua/serializer.h b/src/lua/serializer.h
index 0a0501a74..52e51d279 100644
--- a/src/lua/serializer.h
+++ b/src/lua/serializer.h
@@ -52,6 +52,7 @@ extern "C" {
 #include <lauxlib.h>
 
 #include "trigger.h"
+#include "lib/core/datetime.h"
 #include "lib/core/decimal.h" /* decimal_t */
 #include "lib/core/mp_extension_types.h"
 #include "lua/error.h"
@@ -223,6 +224,7 @@ struct luaL_field {
 		uint32_t size;
 		decimal_t *decval;
 		struct tt_uuid *uuidval;
+		struct datetime_t *dateval;
 	};
 	enum mp_type type;
 	/* subtypes of MP_EXT */
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 6aeb18cfe..a959ed9dd 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -241,7 +241,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
 	lua_pop(L, 1);
 }
 
-
 /**
  * A helper to register a single type metatable.
  */
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
index b6f568c03..226b6fadb 100644
--- a/test/unit/datetime.c
+++ b/test/unit/datetime.c
@@ -139,7 +139,6 @@ exit:
 // avoid introducing external datetime.h dependency
 // - just copy paste it for today
 #define SECS_PER_DAY      86400
-#define NANOS_PER_SEC     1000000000
 #define DT_EPOCH_1970_OFFSET 719163
 
 
diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c
index 5123b9a74..d5ddc4ea0 100644
--- a/third_party/lua-cjson/lua_cjson.c
+++ b/third_party/lua-cjson/lua_cjson.c
@@ -52,6 +52,7 @@
 #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
 #include "tt_static.h"
 #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */
+#include "core/datetime.h"
 #include "cord_buf.h"
 
 typedef enum {
@@ -426,6 +427,13 @@ static void json_append_data(lua_State *l, struct luaL_serializer *cfg,
         case MP_UUID:
             return json_append_string(cfg, json, tt_uuid_str(field.uuidval),
                                       UUID_STR_LEN);
+
+        case MP_DATETIME:
+        {
+            char buf[128];
+            size_t sz = datetime_to_string(field.dateval, buf, sizeof buf);
+            return json_append_string(cfg, json, buf, sz);
+        }
         default:
             assert(false);
         }
diff --git a/third_party/lua-yaml/lyaml.cc b/third_party/lua-yaml/lyaml.cc
index 5469e9f4f..b76a45dfb 100644
--- a/third_party/lua-yaml/lyaml.cc
+++ b/third_party/lua-yaml/lyaml.cc
@@ -617,7 +617,7 @@ static int dump_node(struct lua_yaml_dumper *dumper)
    yaml_event_t ev;
    yaml_scalar_style_t style = YAML_PLAIN_SCALAR_STYLE;
    int is_binary = 0;
-   char buf[FPCONV_G_FMT_BUFSIZE];
+   char buf[FPCONV_G_FMT_BUFSIZE + 8]; // FIXME - need extra space for datetime
    struct luaL_field field;
    bool unused;
    (void) unused;
@@ -707,6 +707,10 @@ static int dump_node(struct lua_yaml_dumper *dumper)
          str = tt_uuid_str(field.uuidval);
          len = UUID_STR_LEN;
          break;
+      case MP_DATETIME:
+         len = datetime_to_string(field.dateval, buf, sizeof buf);
+         str = buf;
+         break;
       default:
          assert(0); /* checked by luaL_checkfield() */
       }
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (4 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 05/11] box, datetime: add messagepack support for datetime Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:56   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding Timur Safin via Tarantool-patches
                   ` (5 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* storage hints implemented for datetime_t values;
* proper comparison for indices of datetime type.

Part of #5941
Part of #5946
---
 src/box/field_def.c           | 18 ++++++++
 src/box/field_def.h           |  3 ++
 src/box/memtx_space.c         |  3 +-
 src/box/tuple_compare.cc      | 30 ++++++++++++++
 src/box/vinyl.c               |  3 +-
 test/engine/datetime.result   | 77 +++++++++++++++++++++++++++++++++++
 test/engine/datetime.test.lua | 35 ++++++++++++++++
 7 files changed, 165 insertions(+), 4 deletions(-)
 create mode 100644 test/engine/datetime.result
 create mode 100644 test/engine/datetime.test.lua

diff --git a/src/box/field_def.c b/src/box/field_def.c
index 6964e3e9f..aaf5f9cff 100644
--- a/src/box/field_def.c
+++ b/src/box/field_def.c
@@ -193,3 +193,21 @@ field_type_by_name(const char *name, size_t len)
 		return FIELD_TYPE_ANY;
 	return field_type_MAX;
 }
+
+const bool field_type_index_allowed[] =
+    {
+	/* [FIELD_TYPE_ANY]      = */ false,
+	/* [FIELD_TYPE_UNSIGNED] = */ true,
+	/* [FIELD_TYPE_STRING]   = */ true,
+	/* [FIELD_TYPE_NUMBER]   = */ true,
+	/* [FIELD_TYPE_DOUBLE]   = */ true,
+	/* [FIELD_TYPE_INTEGER]  = */ true,
+	/* [FIELD_TYPE_BOOLEAN]  = */ true,
+	/* [FIELD_TYPE_VARBINARY]= */ true,
+	/* [FIELD_TYPE_SCALAR]   = */ true,
+	/* [FIELD_TYPE_DECIMAL]  = */ true,
+	/* [FIELD_TYPE_UUID]     = */ true,
+	/* [FIELD_TYPE_ARRAY]    = */ false,
+	/* [FIELD_TYPE_MAP]      = */ false,
+	/* [FIELD_TYPE_DATETIME] = */ true,
+};
diff --git a/src/box/field_def.h b/src/box/field_def.h
index 120b2a93d..bd02418df 100644
--- a/src/box/field_def.h
+++ b/src/box/field_def.h
@@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[];
 extern const struct opt_def field_def_reg[];
 extern const struct field_def field_def_default;
 
+/** helper table for checking allowed indices for types */
+extern const bool field_type_index_allowed[];
+
 /**
  * @brief Field definition
  * Contains information about of one tuple field.
diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
index b71318d24..1ab16122e 100644
--- a/src/box/memtx_space.c
+++ b/src/box/memtx_space.c
@@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
 	/* Check that there are no ANY, ARRAY, MAP parts */
 	for (uint32_t i = 0; i < key_def->part_count; i++) {
 		struct key_part *part = &key_def->parts[i];
-		if (part->type <= FIELD_TYPE_ANY ||
-		    part->type >= FIELD_TYPE_ARRAY) {
+		if (!field_type_index_allowed[part->type]) {
 			diag_set(ClientError, ER_MODIFY_INDEX,
 				 index_def->name, space_name(space),
 				 tt_sprintf("field type '%s' is not supported",
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index 48c833643..4adceebd3 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a, enum mp_type a_type,
 						   field_b, b_type);
 	case FIELD_TYPE_UUID:
 		return mp_compare_uuid(field_a, field_b);
+	case FIELD_TYPE_DATETIME:
+		return mp_compare_datetime(field_a, field_b);
 	default:
 		unreachable();
 		return 0;
@@ -1630,6 +1632,18 @@ hint_uuid_raw(const char *data)
 	return hint_create(MP_CLASS_UUID, val);
 }
 
+static inline hint_t
+hint_datetime(struct datetime_t *date)
+{
+	/*
+	 * Use at most HINT_VALUE_BITS from datetime
+	 * seconds field as a hint value
+	 */
+	uint64_t val = (uint64_t)date->secs & HINT_VALUE_MAX;
+
+	return hint_create(MP_CLASS_DATETIME, val);
+}
+
 static inline uint64_t
 hint_str_raw(const char *s, uint32_t len)
 {
@@ -1761,6 +1775,17 @@ field_hint_uuid(const char *field)
 	return hint_uuid_raw(data);
 }
 
+static inline hint_t
+field_hint_datetime(const char *field)
+{
+	assert(mp_typeof(*field) == MP_EXT);
+	int8_t ext_type;
+	uint32_t len = mp_decode_extl(&field, &ext_type);
+	assert(ext_type == MP_DATETIME);
+	struct datetime_t date;
+	return hint_datetime(datetime_unpack(&field, len, &date));
+}
+
 static inline hint_t
 field_hint_string(const char *field, struct coll *coll)
 {
@@ -1849,6 +1874,8 @@ field_hint(const char *field, struct coll *coll)
 		return field_hint_decimal(field);
 	case FIELD_TYPE_UUID:
 		return field_hint_uuid(field);
+	case FIELD_TYPE_DATETIME:
+		return field_hint_datetime(field);
 	default:
 		unreachable();
 	}
@@ -1963,6 +1990,9 @@ key_def_set_hint_func(struct key_def *def)
 	case FIELD_TYPE_UUID:
 		key_def_set_hint_func<FIELD_TYPE_UUID>(def);
 		break;
+	case FIELD_TYPE_DATETIME:
+		key_def_set_hint_func<FIELD_TYPE_DATETIME>(def);
+		break;
 	default:
 		/* Invalid key definition. */
 		def->key_hint = NULL;
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index c80b2d99b..360d1fa70 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -662,8 +662,7 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def)
 	/* Check that there are no ANY, ARRAY, MAP parts */
 	for (uint32_t i = 0; i < key_def->part_count; i++) {
 		struct key_part *part = &key_def->parts[i];
-		if (part->type <= FIELD_TYPE_ANY ||
-		    part->type >= FIELD_TYPE_ARRAY) {
+		if (!field_type_index_allowed[part->type]) {
 			diag_set(ClientError, ER_MODIFY_INDEX,
 				 index_def->name, space_name(space),
 				 tt_sprintf("field type '%s' is not supported",
diff --git a/test/engine/datetime.result b/test/engine/datetime.result
new file mode 100644
index 000000000..3ff474dee
--- /dev/null
+++ b/test/engine/datetime.result
@@ -0,0 +1,77 @@
+-- test-run result file version 2
+env = require('test_run')
+ | ---
+ | ...
+test_run = env.new()
+ | ---
+ | ...
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+
+date = require('datetime')
+ | ---
+ | ...
+
+_ = box.schema.space.create('T', {engine = engine})
+ | ---
+ | ...
+_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
+ | ---
+ | ...
+
+box.space.T:insert{date('1970-01-01')}\
+box.space.T:insert{date('1970-01-02')}\
+box.space.T:insert{date('1970-01-03')}\
+box.space.T:insert{date('2000-01-01')}
+ | ---
+ | ...
+
+o = box.space.T:select{}
+ | ---
+ | ...
+assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
+ | ---
+ | - true
+ | ...
+
+for i = 1,16 do\
+    box.space.T:insert{date.now()}\
+end
+ | ---
+ | ...
+
+a = box.space.T:select{}
+ | ---
+ | ...
+err = nil
+ | ---
+ | ...
+for i = 1, #a - 1 do\
+    if tostring(a[i][1]) >= tostring(a[i+1][1]) then\
+        err = {a[i][1], a[i+1][1]}\
+        break\
+    end\
+end
+ | ---
+ | ...
+
+err
+ | ---
+ | - null
+ | ...
+box.space.T:drop()
+ | ---
+ | ...
diff --git a/test/engine/datetime.test.lua b/test/engine/datetime.test.lua
new file mode 100644
index 000000000..180d9bb72
--- /dev/null
+++ b/test/engine/datetime.test.lua
@@ -0,0 +1,35 @@
+env = require('test_run')
+test_run = env.new()
+engine = test_run:get_cfg('engine')
+
+date = require('datetime')
+
+_ = box.schema.space.create('T', {engine = engine})
+_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
+
+box.space.T:insert{date('1970-01-01')}\
+box.space.T:insert{date('1970-01-02')}\
+box.space.T:insert{date('1970-01-03')}\
+box.space.T:insert{date('2000-01-01')}
+
+o = box.space.T:select{}
+assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
+assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
+assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
+assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
+
+for i = 1,16 do\
+    box.space.T:insert{date.now()}\
+end
+
+a = box.space.T:select{}
+err = nil
+for i = 1, #a - 1 do\
+    if tostring(a[i][1]) >= tostring(a[i+1][1]) then\
+        err = {a[i][1], a[i+1][1]}\
+        break\
+    end\
+end
+
+err
+box.space.T:drop()
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (5 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes Timur Safin via Tarantool-patches
                   ` (4 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* correct incorrect encoding of MP_EXT sizes for datetime
  messagepack values;
* export necessary symbols for datetime messagepack size calculations
  so they will be available for Lua consumption.

Part of #5941
---
 src/exports.h           |  2 ++
 src/lib/core/datetime.c | 15 ++++++++++-----
 src/lua/msgpackffi.lua  | 10 ++++++++++
 3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/src/exports.h b/src/exports.h
index 586444b65..3a1e8854c 100644
--- a/src/exports.h
+++ b/src/exports.h
@@ -437,6 +437,7 @@ EXPORT(mp_encode_decimal)
 EXPORT(mp_encode_double)
 EXPORT(mp_encode_float)
 EXPORT(mp_encode_uuid)
+EXPORT(mp_encode_datetime)
 EXPORT(mp_ext_hint)
 EXPORT(mp_format)
 EXPORT(mp_fprint)
@@ -446,6 +447,7 @@ EXPORT(mp_fprint_recursion)
 EXPORT(mp_parser_hint)
 EXPORT(mp_sizeof_decimal)
 EXPORT(mp_sizeof_uuid)
+EXPORT(mp_sizeof_datetime)
 EXPORT(mp_snprint)
 EXPORT(mp_snprint_ext)
 EXPORT(mp_snprint_ext_default)
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index 26ba8a702..76b06eae5 100755
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -83,8 +83,8 @@ mp_decode_Xint(const char **data)
 	return 0;
 }
 
-uint32_t
-mp_sizeof_datetime(const struct datetime_t *date)
+static inline uint32_t
+mp_sizeof_datetime_raw(const struct datetime_t *date)
 {
 	uint32_t sz = mp_sizeof_Xint(date->secs);
 
@@ -94,10 +94,15 @@ mp_sizeof_datetime(const struct datetime_t *date)
 		sz += mp_sizeof_Xint(date->nsec);
 	if (date->offset)
 		sz += mp_sizeof_Xint(date->offset);
-
 	return sz;
 }
 
+uint32_t
+mp_sizeof_datetime(const struct datetime_t *date)
+{
+	return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
+}
+
 struct datetime_t *
 datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
 {
@@ -112,7 +117,7 @@ datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
 		return date;
 
 	svp = *data;
-	date->secs = mp_decode_Xint(data);
+	date->nsec = mp_decode_Xint(data);
 	len -= *data - svp;
 
 	if (len <= 0)
@@ -153,7 +158,7 @@ datetime_pack(char *data, const struct datetime_t *date)
 char *
 mp_encode_datetime(char *data, const struct datetime_t *date)
 {
-	uint32_t len = mp_sizeof_datetime(date);
+	uint32_t len = mp_sizeof_datetime_raw(date);
 
 	data = mp_encode_extl(data, MP_DATETIME, len);
 
diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua
index 271be857a..c47d77acb 100644
--- a/src/lua/msgpackffi.lua
+++ b/src/lua/msgpackffi.lua
@@ -26,6 +26,10 @@ char *
 mp_encode_uuid(char *data, const struct tt_uuid *uuid);
 uint32_t
 mp_sizeof_uuid();
+uint32_t
+mp_sizeof_datetime(const struct t_datetime_tz *date);
+char *
+mp_encode_datetime(char *data, const struct t_datetime_tz *date);
 float
 mp_decode_float(const char **data);
 double
@@ -144,6 +148,11 @@ local function encode_uuid(buf, uuid)
     builtin.mp_encode_uuid(p, uuid)
 end
 
+local function encode_datetime(buf, date)
+    local p = buf:alloc(builtin.mp_sizeof_datetime(date))
+    builtin.mp_encode_datetime(p, date)
+end
+
 local function encode_int(buf, num)
     if num >= 0 then
         if num <= 0x7f then
@@ -322,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float)
 on_encode(ffi.typeof('double'), encode_double)
 on_encode(ffi.typeof('decimal_t'), encode_decimal)
 on_encode(ffi.typeof('struct tt_uuid'), encode_uuid)
+on_encode(ffi.typeof('struct datetime_t'), encode_datetime)
 
 --------------------------------------------------------------------------------
 -- Decoder
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (6 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support Timur Safin via Tarantool-patches
                   ` (3 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* introduced a set of calculated attributes to data object, e.g.:
  - timestamp, seconds, microseconds, minute, or hours

Part of #5941
---
 src/lua/datetime.lua           | 51 ++++++++++++++++++++++------------
 test/app-tap/datetime.test.lua | 17 +++++++++++-
 2 files changed, 50 insertions(+), 18 deletions(-)

diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 9ec06d8d8..7a208cef9 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -224,6 +224,36 @@ local function interval_serialize(self)
     return { secs = self.secs, nsec = self.nsec }
 end
 
+local datetime_index = function(self, key)
+    local attributes = {
+        timestamp = function(self)
+            return tonumber(self.secs) + self.nsec / 1e9
+        end,
+        nanoseconds = function(self)
+            return tonumber(self.secs * 1e9 + self.nsec)
+        end,
+        microseconds = function(self)
+            return tonumber(self.secs * 1e6 + self.nsec / 1e3)
+        end,
+        milliseconds = function(self)
+            return tonumber(self.secs * 1e3 + self.nsec / 1e6)
+        end,
+        seconds = function(self)
+            return tonumber(self.secs) + self.nsec / 1e9
+        end,
+        minutes = function(self)
+            return (tonumber(self.secs) + self.nsec / 1e9) / 60
+        end,
+        hours = function(self)
+            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
+        end,
+        days = function(self)
+            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
+        end,
+    }
+    return attributes[key] ~= nil and attributes[key](self) or nil
+end
+
 local datetime_mt = {
     -- __tostring = datetime_tostring,
     __serialize = datetime_serialize,
@@ -232,23 +262,7 @@ local datetime_mt = {
     __le = datetime_le,
     __sub = datetime_sub,
     __add = datetime_add,
-
-    nanoseconds = function(self)
-        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
-    end,
-    microseconds = function(self)
-        return tonumber(self.secs*1e6 + self.nsec*1e3)
-    end,
-    seconds = function(self)
-        return tonumber(self.secs + self.nsec*1e3)
-    end,
-    minutes = function(self)
-        return tonumber((self._ticks/(1e6*60))%60)
-    end,
-    hours = function(self)
-        return tonumber(self._ticks/(1e6*60*60))
-    end,
-
+    __index = datetime_index,
 }
 
 local interval_mt = {
@@ -257,6 +271,9 @@ local interval_mt = {
     __eq = datetime_eq,
     __lt = datetime_lt,
     __le = datetime_le,
+    __sub = datetime_sub,
+    __add = datetime_add,
+    __index = datetime_index,
 }
 
 local function datetime_new_raw(secs, nsec, offset)
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
index 09c968858..407d89556 100755
--- a/test/app-tap/datetime.test.lua
+++ b/test/app-tap/datetime.test.lua
@@ -4,7 +4,7 @@ local tap = require('tap')
 local test = tap.test("errno")
 local date = require('datetime')
 
-test:plan(5)
+test:plan(6)
 
 test:test("Simple tests for parser", function(test)
     test:plan(2)
@@ -188,4 +188,19 @@ test:test("Parse iso date - invalid strings", function(test)
     end
 end)
 
+test:test("Parse tiny date into seconds and other parts", function(test)
+    test:plan(9)
+    local str = '19700101 00:00:30.528'
+    local tiny = date(str)
+    test:ok(tiny.secs == 30, ("secs of '%s'"):format(str))
+    test:ok(tiny.nsec == 528000000, ("nsec of '%s'"):format(str))
+    test:ok(tiny.nanoseconds == 30528000000, "nanoseconds")
+    test:ok(tiny.microseconds == 30528000, "microseconds")
+    test:ok(tiny.milliseconds == 30528, "milliseconds")
+    test:ok(tiny.seconds == 30.528, "seconds")
+    test:ok(tiny.timestamp == 30.528, "timestamp")
+    test:ok(tiny.minutes == 0.5088, "minuts")
+    test:ok(tiny.hours == 0.00848, "hours")
+end)
+
 os.exit(test:check() and 0 or 1)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (7 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua Timur Safin via Tarantool-patches
                   ` (2 subsequent siblings)
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* created few entry points (months(N), years(N), days(N), etc.)
  for easier datetime arithmetic;
* additions/subtractions of years/months use `dt_add_years()`
  and `dt_add_months()` from 3rd party c-dt library;
* also there are `:add{}` and `:sub{}` methods in datetime
  object to add or substract more complex intervals;
* introduced `is_datetime()` and `is_interval()` helpers for checking
  of validity of passed arguments;
* human-readable stringization implemented for interval objects.

Note, that additions/subtractions completed for all _reasonable_
combinations of values of date and interval types;

	Time + Interval	=> Time
	Interval + Time => Time
	Time - Time 	=> Interval
	Time - Interval => Time
	Interval + Interval => Interval
	Interval - Interval => Interval

Part of #5941
---
 src/exports.h                  |   3 +
 src/lua/datetime.lua           | 556 +++++++++++++++++++++++++++------
 test/app-tap/datetime.test.lua | 163 +++++++++-
 3 files changed, 631 insertions(+), 91 deletions(-)

diff --git a/src/exports.h b/src/exports.h
index 3a1e8854c..6e7fe206d 100644
--- a/src/exports.h
+++ b/src/exports.h
@@ -535,6 +535,9 @@ EXPORT(uuid_nil)
 EXPORT(uuid_unpack)
 EXPORT(datetime_unpack)
 EXPORT(datetime_pack)
+EXPORT(dt_add_months)
+EXPORT(dt_add_years)
+EXPORT(dt_add_quarters)
 EXPORT(dt_from_rdn)
 EXPORT(dt_from_yd)
 EXPORT(dt_from_ymd)
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 7a208cef9..1466b923f 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -37,6 +37,17 @@ ffi.cdef [[
     int      dt_rdn          (dt_t dt);
     dt_dow_t dt_dow          (dt_t dt);
 
+    // dt_arithmetic.h
+    typedef enum {
+        DT_EXCESS,
+        DT_LIMIT,
+        DT_SNAP
+    } dt_adjust_t;
+
+    dt_t    dt_add_years        (dt_t dt, int delta, dt_adjust_t adjust);
+    dt_t    dt_add_quarters     (dt_t dt, int delta, dt_adjust_t adjust);
+    dt_t    dt_add_months       (dt_t dt, int delta, dt_adjust_t adjust);
+
     // dt_parse_iso.h
     size_t dt_parse_iso_date          (const char *str, size_t len, dt_t *dt);
 
@@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL
 
 local datetime_t = ffi.typeof('struct datetime_t')
 local interval_t = ffi.typeof('struct datetime_interval_t')
+ffi.cdef [[
+    struct t_interval_months {
+        int m;
+    };
+
+    struct t_interval_years {
+        int y;
+    };
+]]
+local interval_months_t = ffi.typeof('struct t_interval_months')
+local interval_years_t = ffi.typeof('struct t_interval_years')
+
+local function is_interval(o)
+    return ffi.istype(interval_t, o) or
+           ffi.istype(interval_months_t, o) or
+           ffi.istype(interval_years_t, o)
+end
+
+local function is_datetime(o)
+    return ffi.istype(o, datetime_t)
+end
+
 
 local function interval_new()
     local interval = ffi.new(interval_t)
     return interval
 end
 
-local function adjusted_secs(dt)
-    return dt.secs - dt.offset * 60
+local function check_number(n, message, lvl)
+    if lvl == nil then
+        lvl = 2
+    end
+    if type(n) ~= 'number' then
+        return error(('Usage: %s'):format(message), lvl)
+    end
 end
 
-local function datetime_sub(lhs, rhs)
-    local s1 = adjusted_secs(lhs)
-    local s2 = adjusted_secs(rhs)
-    local d = interval_new()
-    d.secs = s2 - s1
-    d.nsec = rhs.nsec - lhs.nsec
-    if d.nsec < 0 then
-        d.secs = d.secs - 1
-        d.nsec = d.nsec + NANOS_PER_SEC
+local function check_date(o, message, lvl)
+    if lvl == nil then
+        lvl = 2
+    end
+    if not is_datetime(o) then
+        return error(('Usage: %s'):format(message), lvl)
     end
-    return d
 end
 
-local function datetime_add(lhs, rhs)
-    local s1 = adjusted_secs(lhs)
-    local s2 = adjusted_secs(rhs)
-    local d = interval_new()
-    d.secs = s2 + s1
-    d.nsec = rhs.nsec + lhs.nsec
-    if d.nsec >= NANOS_PER_SEC then
-        d.secs = d.secs + 1
-        d.nsec = d.nsec - NANOS_PER_SEC
+local function check_date_interval(o, message, lvl)
+    if lvl == nil then
+        lvl = 2
+    end
+    if not (is_datetime(o) or is_interval(o)) then
+        return error(('Usage: %s'):format(message), lvl)
     end
-    return d
 end
 
-local function datetime_eq(lhs, rhs)
-    -- we usually don't need to check nullness
-    -- but older tarantool console will call us checking for equality to nil
-    if rhs == nil then
-        return false
+local function check_interval(o, message, lvl)
+    if lvl == nil then
+        lvl = 2
+    end
+    if not is_interval(o) then
+        return error(('Usage: %s'):format(message), lvl)
     end
+end
+
+local function check_str(o, message, lvl)
+    if lvl == nil then
+        lvl = 2
+    end
+    if not type(o) == 'string' then
+        return error(('Usage: %s'):format(message), lvl)
+    end
+end
+
+local function interval_years_new(y)
+    check_number(y, "years(number)")
+    local o = ffi.new(interval_years_t)
+    o.y = y
+    return o
+end
+
+local function interval_months_new(m)
+    check_number(m, "months(number)")
+    local o = ffi.new(interval_months_t)
+    o.m = m
+    return o
+end
+
+local function interval_weeks_new(w)
+    check_number(w, "weeks(number)")
+    local o = ffi.new(interval_t)
+    o.secs = w * SECS_PER_DAY * 7
+    return o
+end
+
+local function interval_days_new(d)
+    check_number(d, "days(number)")
+    local o = ffi.new(interval_t)
+    o.secs = d * SECS_PER_DAY
+    return o
+end
+
+local function interval_hours_new(h)
+    check_number(h, "hours(number)")
+    local o = ffi.new(interval_t)
+    o.secs = h * 60 * 60
+    return o
+end
+
+local function interval_minutes_new(m)
+    check_number(m, "minutes(number)")
+    local o = ffi.new(interval_t)
+    o.secs = m * 60
+    return o
+end
+
+local function interval_seconds_new(s)
+    check_number(s, "seconds(number)")
+    local o = ffi.new(interval_t)
+    o.nsec = s % 1 * 1e9
+    o.secs = s - (s % 1)
+    return o
+end
+
+local function datetime_eq(lhs, rhs)
+    check_date_interval(lhs, "datetime:__eq(date or interval)")
+    check_date_interval(rhs, "datetime:__eq(date or interval)")
     return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
 end
 
 
 local function datetime_lt(lhs, rhs)
+    check_date_interval(lhs, "datetime:__lt(date or interval)")
+    check_date_interval(rhs, "datetime:__lt(date or interval)")
     return (lhs.secs < rhs.secs) or
            (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
 end
 
 local function datetime_le(lhs, rhs)
+    check_date_interval(lhs, "datetime:__le(date or interval)")
+    check_date_interval(rhs, "datetime:__le(date or interval)")
     return (lhs.secs <= rhs.secs) or
            (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
 end
@@ -224,19 +323,123 @@ local function interval_serialize(self)
     return { secs = self.secs, nsec = self.nsec }
 end
 
+local function local_rd(o)
+    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
+end
+
+local function local_dt(o)
+    return cdt.dt_from_rdn(local_rd(o))
+end
+
+local function _normalize_nsec(secs, nsec)
+    if nsec < 0 then
+        secs = secs - 1
+        nsec = nsec + NANOS_PER_SEC
+    elseif nsec >= NANOS_PER_SEC then
+        secs = secs + 1
+        nsec = nsec - NANOS_PER_SEC
+    end
+    return secs, nsec
+end
+
+-- addition or subtraction from date/time of a given interval
+-- described via table direction should be +1 or -1
+local function interval_increment(self, o, direction)
+    assert(direction == -1 or direction == 1)
+    check_date(self, "interval_increment(date, object, -+1)")
+    assert(type(o) == 'table')
+
+    local ym_updated = false
+    local dhms_updated = false
+
+    local dt = local_dt(self)
+    local secs, nsec
+    secs, nsec = self.secs, self.nsec
+
+    for key, value in pairs(o) do
+        local handlers = {
+            years = function(v)
+                assert(v > 0 and v < 10000)
+                dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
+                ym_updated = true
+            end,
+
+            months = function(v)
+                assert(v > 0 and v < 13 )
+                dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
+                ym_updated = true
+            end,
+
+            weeks = function(v)
+                assert(v > 0 and v < 32)
+                secs = secs + direction * 7 * v * SECS_PER_DAY
+                dhms_updated = true
+            end,
+
+            days = function(v)
+                assert(v > 0 and v < 32)
+                secs = secs + direction * v * SECS_PER_DAY
+                dhms_updated = true
+            end,
+
+            hours = function(v)
+                assert(v >= 0 and v < 24)
+                secs = secs + direction * 60 * 60 * v
+                dhms_updated = true
+            end,
+
+            minutes = function(v)
+                assert(v >= 0 and v < 60)
+                secs = secs + direction * 60 * v
+            end,
+
+            seconds = function(v)
+                assert(v >= 0 and v < 61)
+                local s, frac
+                frac = v % 1
+                if frac > 0 then
+                    s = v - (v % 1)
+                else
+                    s = v
+                end
+                secs = secs + direction * s
+                nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
+                dhms_updated = true
+            end,
+        }
+        handlers[key](value)
+    end
+
+    secs, nsec = _normalize_nsec(secs, nsec)
+
+    -- .days, .hours, .minutes, .seconds
+    if dhms_updated then
+        self.secs = secs
+        self.nsec = nsec
+    end
+
+    -- .years, .months updated
+    if ym_updated then
+        self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY +
+                    secs % SECS_PER_DAY
+    end
+
+    return self
+end
+
 local datetime_index = function(self, key)
     local attributes = {
         timestamp = function(self)
             return tonumber(self.secs) + self.nsec / 1e9
         end,
         nanoseconds = function(self)
-            return tonumber(self.secs * 1e9 + self.nsec)
+            return self.secs * 1e9 + self.nsec
         end,
         microseconds = function(self)
-            return tonumber(self.secs * 1e6 + self.nsec / 1e3)
+            return self.secs * 1e6 + self.nsec / 1e3
         end,
         milliseconds = function(self)
-            return tonumber(self.secs * 1e3 + self.nsec / 1e6)
+            return self.secs * 1e3 + self.nsec / 1e6
         end,
         seconds = function(self)
             return tonumber(self.secs) + self.nsec / 1e9
@@ -250,32 +453,20 @@ local datetime_index = function(self, key)
         days = function(self)
             return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
         end,
+        add = function(self)
+            return function(self, o)
+                return interval_increment(self, o, 1)
+            end
+        end,
+        sub = function(self)
+            return function(self, o)
+                return interval_increment(self, o, -1)
+            end
+        end,
     }
     return attributes[key] ~= nil and attributes[key](self) or nil
 end
 
-local datetime_mt = {
-    -- __tostring = datetime_tostring,
-    __serialize = datetime_serialize,
-    __eq = datetime_eq,
-    __lt = datetime_lt,
-    __le = datetime_le,
-    __sub = datetime_sub,
-    __add = datetime_add,
-    __index = datetime_index,
-}
-
-local interval_mt = {
-    -- __tostring = interval_tostring,
-    __serialize = interval_serialize,
-    __eq = datetime_eq,
-    __lt = datetime_lt,
-    __le = datetime_le,
-    __sub = datetime_sub,
-    __add = datetime_add,
-    __index = datetime_index,
-}
-
 local function datetime_new_raw(secs, nsec, offset)
     local dt_obj = ffi.new(datetime_t)
     dt_obj.secs = secs
@@ -284,14 +475,6 @@ local function datetime_new_raw(secs, nsec, offset)
     return dt_obj
 end
 
-local function local_rd(o)
-    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
-end
-
-local function local_dt(o)
-    return cdt.dt_from_rdn(local_rd(o))
-end
-
 local function mk_timestamp(dt, sp, fp, offset)
     local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0
     local spV = sp ~= nil and sp or 0
@@ -367,11 +550,12 @@ local function datetime_new(o)
             second = function(v)
                 assert(v >= 0 and v < 61)
                 frac = v % 1
-                if frac then
+                if frac > 0 then
                     s = v - (v % 1)
                 else
                     s = v
                 end
+                frac = frac * 1e9 -- convert fraction to nanoseconds
                 hms = true
             end,
 
@@ -402,6 +586,153 @@ local function datetime_new(o)
     return mk_timestamp(dt, secs, frac, offset)
 end
 
+local function datetime_tostring(o)
+    if ffi.typeof(o) == datetime_t then
+        local sz = 48
+        local buff = ffi.new('char[?]', sz)
+        local len = native.datetime_to_string(o, buff, sz)
+        assert(len < sz)
+        return ffi.string(buff)
+    elseif ffi.typeof(o) == interval_years_t then
+        return ('%+d years'):format(o.y)
+    elseif ffi.typeof(o) == interval_months_t then
+        return ('%+d months'):format(o.m)
+    elseif ffi.typeof(o) == interval_t then
+        local ts = o.timestamp
+        local sign = '+'
+
+        if ts < 0 then
+            ts = -ts
+            sign = '-'
+        end
+
+        if ts < 60 then
+            return ('%s%s secs'):format(sign, ts)
+        elseif ts < 60 * 60 then
+            return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60)
+        elseif ts < 24 * 60 * 60 then
+            return ('%+d hours, %d minutes, %s seconds'):format(
+                    o.hours, o.minutes % 60, ts % 60)
+        else
+            return ('%+d days, %d hours, %d minutes, %s seconds'):format(
+                    o.days, o.hours % 24, o.minutes % 60, ts % 60)
+        end
+    end
+end
+
+local function date_first(lhs, rhs)
+    if is_datetime(lhs) then
+        return lhs, rhs
+    else
+        return rhs, lhs
+    end
+end
+
+--[[
+Matrix of subtraction operands eligibility and their result type
+
+|                 |  datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime        |  interval | datetime | datetime        | datetime       |
+| interval        |           | interval |                 |                |
+| interval_months |           |          | interval_months |                |
+| interval_years  |           |          |                 | interval_years |
+]]
+local function datetime_sub(lhs, rhs)
+    check_date_interval(lhs, "datetime:__sub(date or interval)")
+    local d, s = lhs, rhs
+    local left_t = ffi.typeof(d)
+    local right_t = ffi.typeof(s)
+    local o
+
+    if left_t == datetime_t then
+        -- 1. left is date, right is date or generic interval
+        if (right_t == datetime_t or right_t == interval_t) then
+            o = right_t == datetime_t and interval_new() or datetime_new()
+            o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs,
+                                            lhs.nsec - rhs.nsec)
+            return o
+        -- 2. left is date, right is interval in months
+        elseif right_t == interval_months_t then
+            local dt = cdt.dt_add_months(local_dt(lhs), -rhs.m, cdt.DT_LIMIT)
+            return mk_timestamp(dt, lhs.secs % SECS_PER_DAY,
+                                lhs.nsec, lhs.offset)
+
+        -- 3. left is date, right is interval in years
+        elseif right_t == interval_years_t then
+            local dt = cdt.dt_add_years(local_dt(lhs), -rhs.y, cdt.DT_LIMIT)
+            return mk_timestamp(dt, lhs.secs % SECS_PER_DAY,
+                                lhs.nsec, lhs.offset)
+        else
+            error("datetime:__sub(date or interval) - incompatible type of arguments", 2)
+        end
+    -- 4. both left and right are generic intervals
+    elseif left_t == interval_t and right_t == interval_t then
+        o = interval_new()
+        o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs,
+                                        lhs.nsec - rhs.nsec)
+        return o
+    -- 5. both left and right are intervals in months
+    elseif left_t == interval_months_t and right_t == interval_months_t then
+        return interval_months_new(lhs.m - rhs.m)
+    -- 5. both left and right are intervals in years
+    elseif left_t == interval_years_t and right_t == interval_years_t then
+        return interval_years_new(lhs.y - rhs.y)
+    else
+        error("datetime:__sub(date or interval) - incompatible type of arguments", 2)
+    end
+end
+
+--[[
+Matrix of addition operands eligibility and their result type
+
+|                 |  datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime        |  datetime | datetime | datetime        | datetime       |
+| interval        |  datetime | interval |                 |                |
+| interval_months |  datetime |          | interval_months |                |
+| interval_years  |  datetime |          |                 | interval_years |
+]]
+local function datetime_add(lhs, rhs)
+    local d, s = date_first(lhs, rhs)
+
+    check_date_interval(d, "datetime:__add(interval)")
+    check_interval(s, "datetime:__add(interval)")
+    local left_t = ffi.typeof(d)
+    local right_t = ffi.typeof(s)
+    local o
+
+    -- 1. left is date, right is date or interval
+    if left_t == datetime_t and right_t == interval_t then
+        o = datetime_new()
+        o.secs, o.nsec = _normalize_nsec(d.secs + s.secs, d.nsec + s.nsec)
+        return o
+    -- 2. left is date, right is interval in months
+    elseif left_t == datetime_t and right_t == interval_months_t then
+        local dt = cdt.dt_add_months(local_dt(d), s.m, cdt.DT_LIMIT)
+        local secs = d.secs % SECS_PER_DAY
+        return mk_timestamp(dt, secs, d.nsec, d.offset or 0)
+
+    -- 3. left is date, right is interval in years
+    elseif left_t == datetime_t and right_t == interval_years_t then
+        local dt = cdt.dt_add_years(local_dt(d), s.y, cdt.DT_LIMIT)
+        local secs = d.secs % SECS_PER_DAY
+        return mk_timestamp(dt, secs, d.nsec, d.offset or 0)
+    -- 4. both left and right are generic intervals
+    elseif left_t == interval_t and right_t == interval_t then
+        o = interval_new()
+        o.secs, o.nsec = _normalize_nsec(d.secs + s.secs, d.nsec + s.nsec)
+        return o
+    -- 5. both left and right are intervals in months
+    elseif left_t == interval_months_t and right_t == interval_months_t then
+        return interval_months_new(d.m + s.m)
+    -- 6. both left and right are intervals in years
+    elseif left_t == interval_years_t and right_t == interval_years_t then
+        return interval_years_new(d.y + s.y)
+    else
+        error("datetime:__add(date or interval) - incompatible type of arguments", 2)
+    end
+end
 
 -- simple parse functions:
 -- parse_date/parse_time/parse_zone
@@ -415,6 +746,7 @@ end
 ]]
 
 local function parse_date(str)
+    check_str("datetime.parse_date(string)")
     local dt = ffi.new('dt_t[1]')
     local len = cdt.dt_parse_iso_date(str, #str, dt)
     return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len)
@@ -431,6 +763,7 @@ end
     The time designator [T] may be omitted.
 ]]
 local function parse_time(str)
+    check_str("datetime.parse_time(string)")
     local sp = ffi.new('int[1]')
     local fp = ffi.new('int[1]')
     local len = cdt.dt_parse_iso_time(str, #str, sp, fp)
@@ -444,6 +777,7 @@ end
     ±hhmm    ±hh:mm
 ]]
 local function parse_zone(str)
+    check_str("datetime.parse_zone(string)")
     local offset = ffi.new('int[1]')
     local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
     return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len)
@@ -457,7 +791,8 @@ end
 
     date [T] time [ ] time_zone
 ]]
-local function parse_str(str)
+local function parse(str)
+    check_str("datetime.parse(string)")
     local dt = ffi.new('dt_t[1]')
     local len = #str
     local n = cdt.dt_parse_iso_date(str, len, dt)
@@ -508,7 +843,7 @@ local function datetime_from(o)
     if o == nil or type(o) == 'table' then
         return datetime_new(o)
     elseif type(o) == 'string' then
-        return parse_str(o)
+        return parse(o)
     end
 end
 
@@ -531,8 +866,8 @@ local function local_now()
 end
 
 local function datetime_to_tm_ptr(o)
+    assert(is_datetime(o))
     local p_tm = ffi.new 'struct tm[1]'
-    assert(ffi.typeof(o) == datetime_t)
     -- dt_to_struct_tm() fills only date data
     cdt.dt_to_struct_tm(local_dt(o), p_tm)
 
@@ -551,20 +886,21 @@ local function datetime_to_tm_ptr(o)
 end
 
 local function asctime(o)
-    assert(ffi.typeof(o) == datetime_t)
+    check_date(o, "datetime:asctime()")
+
     local p_tm = datetime_to_tm_ptr(o)
     return ffi.string(native.asctime(p_tm))
 end
 
 local function ctime(o)
-    assert(ffi.typeof(o) == datetime_t)
+    check_date(o, "datetime:ctime()")
     local p_time = ffi.new 'time_t[1]'
     p_time[0] = o.secs
     return ffi.string(native.ctime(p_time))
 end
 
 local function strftime(fmt, o)
-    assert(ffi.typeof(o) == datetime_t)
+    check_date(o, "datetime.strftime(fmt, date)")
     local sz = 50
     local buff = ffi.new('char[?]', sz)
     local p_tm = datetime_to_tm_ptr(o)
@@ -572,36 +908,76 @@ local function strftime(fmt, o)
     return ffi.string(buff)
 end
 
-local function datetime_tostring(o)
-    assert(ffi.typeof(o) == datetime_t)
-    local sz = 48
-    local buff = ffi.new('char[?]', sz)
-    local len = native.datetime_to_string(o, buff, sz)
-    assert(len < sz)
-    return ffi.string(buff)
-end
 
+local datetime_mt = {
+    __tostring = datetime_tostring,
+    __serialize = datetime_serialize,
+    __eq = datetime_eq,
+    __lt = datetime_lt,
+    __le = datetime_le,
+    __sub = datetime_sub,
+    __add = datetime_add,
+    __index = datetime_index,
+    add = function(self, o)
+        self = interval_increment(self, o, 1)
+        return self
+    end,
+    sub = function(self, o)
+        self = interval_increment(self, o, -1)
+        return self
+    end
+}
+
+local interval_mt = {
+    __tostring = datetime_tostring,
+    __serialize = interval_serialize,
+    __eq = datetime_eq,
+    __lt = datetime_lt,
+    __le = datetime_le,
+    __sub = datetime_sub,
+    __add = datetime_add,
+    __index = datetime_index,
+}
+
+local interval_tiny_mt = {
+    __tostring = datetime_tostring,
+    __serialize = interval_serialize,
+    __sub = datetime_sub,
+    __add = datetime_add,
+    __index = datetime_index,
+}
 
 ffi.metatype(interval_t, interval_mt)
 ffi.metatype(datetime_t, datetime_mt)
+ffi.metatype(interval_years_t, interval_tiny_mt)
+ffi.metatype(interval_months_t, interval_tiny_mt)
 
 return setmetatable(
     {
-        datetime = datetime_new,
-        interval = interval_new,
-
-        parse = parse_str,
-        parse_date = parse_date,
-        parse_time = parse_time,
-        parse_zone = parse_zone,
-
-        tostring = datetime_tostring,
-
-        now = local_now,
-    -- strptime = strptime;
-        strftime = strftime,
-        asctime = asctime,
-        ctime = ctime,
+        new         = datetime_new,
+        years       = interval_years_new,
+        months      = interval_months_new,
+        weeks       = interval_weeks_new,
+        days        = interval_days_new,
+        hours       = interval_hours_new,
+        minutes     = interval_minutes_new,
+        seconds     = interval_seconds_new,
+        interval    = interval_new,
+
+        parse       = parse,
+        parse_date  = parse_date,
+        parse_time  = parse_time,
+        parse_zone  = parse_zone,
+
+        tostring    = datetime_tostring,
+
+        now         = local_now,
+        strftime    = strftime,
+        asctime     = asctime,
+        ctime       = ctime,
+
+        is_datetime = is_datetime,
+        is_interval = is_interval,
     }, {
         __call = function(self, ...) return datetime_from(...) end
     }
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
index 407d89556..21d13b3a6 100755
--- a/test/app-tap/datetime.test.lua
+++ b/test/app-tap/datetime.test.lua
@@ -4,7 +4,7 @@ local tap = require('tap')
 local test = tap.test("errno")
 local date = require('datetime')
 
-test:plan(6)
+test:plan(10)
 
 test:test("Simple tests for parser", function(test)
     test:plan(2)
@@ -203,4 +203,165 @@ test:test("Parse tiny date into seconds and other parts", function(test)
     test:ok(tiny.hours == 0.00848, "hours")
 end)
 
+test:test("Stringization of dates and intervals", function(test)
+    test:plan(13)
+    local str = '19700101Z'
+    local dt = date(str)
+    test:ok(tostring(dt) == '1970-01-01T00:00Z', ('tostring(%s)'):format(str))
+    test:ok(tostring(date.seconds(12)) == '+12 secs', '+12 seconds')
+    test:ok(tostring(date.seconds(-12)) == '-12 secs', '-12 seconds')
+    test:ok(tostring(date.minutes(12)) == '+12 minutes, 0 seconds', '+12 minutes')
+    test:ok(tostring(date.minutes(-12)) == '-12 minutes, 0 seconds', '-12 minutes')
+    test:ok(tostring(date.hours(12)) == '+12 hours, 0 minutes, 0 seconds',
+            '+12 hours')
+    test:ok(tostring(date.hours(-12)) == '-12 hours, 0 minutes, 0 seconds',
+            '-12 hours')
+    test:ok(tostring(date.days(12)) == '+12 days, 0 hours, 0 minutes, 0 seconds',
+            '+12 days')
+    test:ok(tostring(date.days(-12)) == '-12 days, 0 hours, 0 minutes, 0 seconds',
+            '-12 days')
+    test:ok(tostring(date.months(5)) == '+5 months', '+5 months')
+    test:ok(tostring(date.months(-5)) == '-5 months', '-5 months')
+    test:ok(tostring(date.years(4)) == '+4 years', '+4 years')
+    test:ok(tostring(date.years(-4)) == '-4 years', '-4 years')
+end)
+
+test:test("Time interval operations", function(test)
+    test:plan(12)
+
+    -- check arithmetic with leap dates
+    local T = date('1972-02-29')
+    local M = date.months(2)
+    local Y = date.years(1)
+    test:ok(tostring(T + M) == '1972-04-29T00:00Z', ('T(%s) + M(%s'):format(T, M))
+    test:ok(tostring(T + Y) == '1973-03-01T00:00Z', ('T(%s) + Y(%s'):format(T, Y))
+    test:ok(tostring(T + M + Y) == '1973-04-30T00:00Z',
+            ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+    test:ok(tostring(T + Y + M) == '1973-05-01T00:00Z',
+            ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+    test:ok(tostring(T:add{years = 1, months = 2}) == '1973-04-30T00:00Z',
+            ('T:add{years=1,months=2}(%s)'):format(T))
+
+    -- check average, not leap dates
+    T = date('1970-01-08')
+    test:ok(tostring(T + M) == '1970-03-08T00:00Z', ('T(%s) + M(%s'):format(T, M))
+    test:ok(tostring(T + Y) == '1971-01-08T00:00Z', ('T(%s) + Y(%s'):format(T, Y))
+    test:ok(tostring(T + M + Y) == '1971-03-08T00:00Z',
+            ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+    test:ok(tostring(T + Y + M) == '1971-03-08T00:00Z',
+            ('T(%s) + Y(%s) + M(%s'):format(T, Y, M))
+    test:ok(tostring(T:add{years = 1, months = 2}) == '1971-03-08T00:00Z',
+            ('T:add{years=1,months=2}(%s)'):format(T))
+
+
+    -- subtraction of 2 dates
+    local T2 = date('19700103')
+    local T1 = date('1970-01-01')
+    test:ok(tostring(T2 - T1) == '+2 days, 0 hours, 0 minutes, 0 seconds',
+            ('T2(%s) - T1(%s'):format(T2, T1))
+    test:ok(tostring(T1 - T2) == '-2 days, 0 hours, 0 minutes, 0 seconds',
+            ('T2(%s) - T1(%s'):format(T2, T1))
+end)
+
+local function catchadd(A, B)
+    return pcall(function() return A + B end)
+end
+
+--[[
+Matrix of addition operands eligibility and their result type
+
+|                 |  datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime        |  datetime | datetime | datetime        | datetime       |
+| interval        |  datetime | interval |                 |                |
+| interval_months |  datetime |          | interval_months |                |
+| interval_years  |  datetime |          |                 | interval_years |
+]]
+
+test:test("Matrix of allowed time and interval additions", function(test)
+    test:plan(20)
+
+    -- check arithmetic with leap dates
+    local T1970 = date('1970-01-01')
+    local T2000 = date('2000-01-01')
+    local I1 = date.days(1)
+    local M2 = date.months(2)
+    local M10 = date.months(10)
+    local Y1 = date.years(1)
+    local Y5 = date.years(5)
+
+    test:ok(catchadd(T1970, I1) == true, "status: T + I")
+    test:ok(catchadd(T1970, M2) == true, "status: T + M")
+    test:ok(catchadd(T1970, Y1) == true, "status: T + Y")
+    test:ok(catchadd(T1970, T2000) == false, "status: T + T")
+    test:ok(catchadd(I1, T1970) == true, "status: I + T")
+    test:ok(catchadd(M2, T1970) == true, "status: M + T")
+    test:ok(catchadd(Y1, T1970) == true, "status: Y + T")
+    test:ok(catchadd(I1, Y1) == false, "status: I + Y")
+    test:ok(catchadd(M2, Y1) == false, "status: M + Y")
+    test:ok(catchadd(I1, Y1) == false, "status: I + Y")
+    test:ok(catchadd(Y5, M10) == false, "status: Y + M")
+    test:ok(catchadd(Y5, I1) == false, "status: Y + I")
+    test:ok(catchadd(Y5, Y1) == true, "status: Y + Y")
+
+    test:ok(tostring(T1970 + I1) == "1970-01-02T00:00Z", "value: T + I")
+    test:ok(tostring(T1970 + M2) == "1970-03-01T00:00Z", "value: T + M")
+    test:ok(tostring(T1970 + Y1) == "1971-01-01T00:00Z", "value: T + Y")
+    test:ok(tostring(I1 + T1970) == "1970-01-02T00:00Z", "value: I + T")
+    test:ok(tostring(M2 + T1970) == "1970-03-01T00:00Z", "value: M + T")
+    test:ok(tostring(Y1 + T1970) == "1971-01-01T00:00Z", "value: Y + T")
+    test:ok(tostring(Y5 + Y1) == "+6 years", "Y + Y")
+
+end)
+
+local function catchsub_status(A, B)
+    return pcall(function() return A - B end)
+end
+
+--[[
+Matrix of subtraction operands eligibility and their result type
+
+|                 |  datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime        |  interval | datetime | datetime        | datetime       |
+| interval        |           | interval |                 |                |
+| interval_months |           |          | interval_months |                |
+| interval_years  |           |          |                 | interval_years |
+]]
+test:test("Matrix of allowed time and interval subtractions", function(test)
+    test:plan(18)
+
+    -- check arithmetic with leap dates
+    local T1970 = date('1970-01-01')
+    local T2000 = date('2000-01-01')
+    local I1 = date.days(1)
+    local M2 = date.months(2)
+    local M10 = date.months(10)
+    local Y1 = date.years(1)
+    local Y5 = date.years(5)
+
+    test:ok(catchsub_status(T1970, I1) == true, "status: T - I")
+    test:ok(catchsub_status(T1970, M2) == true, "status: T - M")
+    test:ok(catchsub_status(T1970, Y1) == true, "status: T - Y")
+    test:ok(catchsub_status(T1970, T2000) == true, "status: T - T")
+    test:ok(catchsub_status(I1, T1970) == false, "status: I + T")
+    test:ok(catchsub_status(M2, T1970) == false, "status: M + T")
+    test:ok(catchsub_status(Y1, T1970) == false, "status: Y + T")
+    test:ok(catchsub_status(I1, Y1) == false, "status: I - Y")
+    test:ok(catchsub_status(M2, Y1) == false, "status: M - Y")
+    test:ok(catchsub_status(I1, Y1) == false, "status: I - Y")
+    test:ok(catchsub_status(Y5, M10) == false, "status: Y - M")
+    test:ok(catchsub_status(Y5, I1) == false, "status: Y - I")
+    test:ok(catchsub_status(Y5, Y1) == true, "status: Y - Y")
+
+    test:ok(tostring(T1970 - I1) == "1969-12-31T00:00Z", "value: T - I")
+    test:ok(tostring(T1970 - M2) == "1969-11-01T00:00Z", "value: T - M")
+    test:ok(tostring(T1970 - Y1) == "1969-01-01T00:00Z", "value: T - Y")
+    test:ok(tostring(T1970 - T2000) == "-10957 days, 0 hours, 0 minutes, 0 seconds",
+            "value: T - T")
+    test:ok(tostring(Y5 - Y1) == "+4 years", "value: Y - Y")
+
+
+end)
+
 os.exit(test:check() and 0 or 1)
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (8 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 11/11] datetime: changelog for datetime module Timur Safin via Tarantool-patches
  2021-07-29 18:55 ` [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Oleg Babin via Tarantool-patches
  11 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

* implemented proper range checks for date attributes values;

* created `.unixtime` attribute, which is alias to `.secs`,
  with corresponding setter/getter;

* similarly to `unixtime`, created virtual `timestamp` attribute
  setter. Which is a convenient way to simultaneously assign
  unixtime (seconds since epoch) and nanoseconds

Part of #5941
---
 src/lua/datetime.lua | 90 ++++++++++++++++++++++++++++++--------------
 1 file changed, 61 insertions(+), 29 deletions(-)

diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 1466b923f..cc2ae119b 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -342,12 +342,33 @@ local function _normalize_nsec(secs, nsec)
     return secs, nsec
 end
 
+local function seconds_fraction(v)
+    local seconds, fraction
+    fraction = v % 1
+    if fraction > 0 then
+        seconds = v - (v % 1)
+    else
+        seconds = v
+    end
+    return seconds, fraction
+end
+
+local function check_range(v, range, txt)
+    assert(#range == 2)
+    if not (v >= range[1] and v <= range[2]) then
+        error(('value %d of %s is out of allowed range [%d, %d]'):
+              format(v, txt, range[1], range[2]))
+    end
+end
+
 -- addition or subtraction from date/time of a given interval
 -- described via table direction should be +1 or -1
 local function interval_increment(self, o, direction)
     assert(direction == -1 or direction == 1)
-    check_date(self, "interval_increment(date, object, -+1)")
-    assert(type(o) == 'table')
+    check_date(self, "interval_increment(date, object, direction)")
+    if type(o) ~= 'table' then
+        error('interval_increment(date, object, direction) - object expected', 2)
+    end
 
     local ym_updated = false
     local dhms_updated = false
@@ -359,49 +380,43 @@ local function interval_increment(self, o, direction)
     for key, value in pairs(o) do
         local handlers = {
             years = function(v)
-                assert(v > 0 and v < 10000)
+                check_range(v, {0, 9999}, key)
                 dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
                 ym_updated = true
             end,
 
             months = function(v)
-                assert(v > 0 and v < 13 )
+                check_range(v, {0, 12}, key)
                 dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
                 ym_updated = true
             end,
 
             weeks = function(v)
-                assert(v > 0 and v < 32)
+                check_range(v, {0, 52}, key)
                 secs = secs + direction * 7 * v * SECS_PER_DAY
                 dhms_updated = true
             end,
 
             days = function(v)
-                assert(v > 0 and v < 32)
+                check_range(v, {0, 31}, key)
                 secs = secs + direction * v * SECS_PER_DAY
                 dhms_updated = true
             end,
 
             hours = function(v)
-                assert(v >= 0 and v < 24)
+                check_range(v, {0, 23}, key)
                 secs = secs + direction * 60 * 60 * v
                 dhms_updated = true
             end,
 
             minutes = function(v)
-                assert(v >= 0 and v < 60)
+                check_range(v, {0, 59}, key)
                 secs = secs + direction * 60 * v
             end,
 
             seconds = function(v)
-                assert(v >= 0 and v < 61)
-                local s, frac
-                frac = v % 1
-                if frac > 0 then
-                    s = v - (v % 1)
-                else
-                    s = v
-                end
+                check_range(v, {0, 60}, key)
+                local s, frac = seconds_fraction(v)
                 secs = secs + direction * s
                 nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
                 dhms_updated = true
@@ -429,6 +444,9 @@ end
 
 local datetime_index = function(self, key)
     local attributes = {
+        unixtime = function(self)
+            return self.secs
+        end,
         timestamp = function(self)
             return tonumber(self.secs) + self.nsec / 1e9
         end,
@@ -467,6 +485,24 @@ local datetime_index = function(self, key)
     return attributes[key] ~= nil and attributes[key](self) or nil
 end
 
+local function datetime_newindex(self, key, value)
+    local attributes = {
+        unixtime = function(self, value)
+            self.secs = value
+            self.nsec, self.offset = 0, 0
+        end,
+        timestamp = function(self, value)
+            local secs, frac = seconds_fraction(value)
+            self.secs = secs
+            self.nsec = frac * 1e9
+            self.offset = 0
+        end,
+    }
+    if attributes[key] ~= nil then
+        attributes[key](self, value)
+    end
+end
+
 local function datetime_new_raw(secs, nsec, offset)
     local dt_obj = ffi.new(datetime_t)
     dt_obj.secs = secs
@@ -518,50 +554,45 @@ local function datetime_new(o)
             end,
 
             year = function(v)
-                assert(v > 0 and v < 10000)
+                check_range(v, {1, 9999}, key)
                 y = v
                 ymd = true
             end,
 
             month = function(v)
-                assert(v > 0 and v < 13 )
+                check_range(v, {1, 12}, key)
                 M = v
                 ymd = true
             end,
 
             day = function(v)
-                assert(v > 0 and v < 32)
+                check_range(v, {1, 31}, key)
                 d = v
                 ymd = true
             end,
 
             hour = function(v)
-                assert(v >= 0 and v < 24)
+                check_range(v, {0, 23}, key)
                 h = v
                 hms = true
             end,
 
             minute = function(v)
-                assert(v >= 0 and v < 60)
+                check_range(v, {0, 59}, key)
                 m = v
                 hms = true
             end,
 
             second = function(v)
-                assert(v >= 0 and v < 61)
-                frac = v % 1
-                if frac > 0 then
-                    s = v - (v % 1)
-                else
-                    s = v
-                end
+                check_range(v, {0, 60}, key)
+                s, frac = seconds_fraction(v)
                 frac = frac * 1e9 -- convert fraction to nanoseconds
                 hms = true
             end,
 
             -- tz offset in minutes
             tz = function(v)
-                assert(v >= 0 and v <= 720)
+                check_range(v, {0, 720}, key)
                 offset = v
             end
         }
@@ -918,6 +949,7 @@ local datetime_mt = {
     __sub = datetime_sub,
     __add = datetime_add,
     __index = datetime_index,
+    __newindex = datetime_newindex,
     add = function(self, o)
         self = interval_increment(self, o, 1)
         return self
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* [Tarantool-patches] [PATCH resend v2 11/11] datetime: changelog for datetime module
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (9 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua Timur Safin via Tarantool-patches
@ 2021-07-28 10:34 ` Timur Safin via Tarantool-patches
  2021-07-29 18:55 ` [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Oleg Babin via Tarantool-patches
  11 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-28 10:34 UTC (permalink / raw)
  To: v.shpilevoy; +Cc: tarantool-patches

Introduced new date/time/interval types support to lua and storage engines.

Closes #5941
Closes #5946
---
 changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md

diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md b/changelogs/unreleased/gh-5941-datetime-type-support.md
new file mode 100644
index 000000000..3c755008e
--- /dev/null
+++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
@@ -0,0 +1,4 @@
+## feature/lua/datetime
+
+ * Introduce new builtin module for date/time/interval support - `datetime.lua`.
+   Support of new datetime type in storage engines (gh-5941, gh-5946).
-- 
2.29.2


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support
  2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
                   ` (10 preceding siblings ...)
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 11/11] datetime: changelog for datetime module Timur Safin via Tarantool-patches
@ 2021-07-29 18:55 ` Oleg Babin via Tarantool-patches
  11 siblings, 0 replies; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:55 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch.


I can't say that it will be complete review but it's brief feedback.

I left comments for some patches. Currently it is not complete series,

in fact it looks like as one-two commits with portion of fixups on the 
top of them.


I believe this module should be implemented in pure C. I don't think 
that arithmetic will

work fast enough in C.


Probably datetime and interval should be logically splitted into 2 parts.

```

tarantool> require('datetime')
---
- weeks: 'function: 0x010ca0aa48'
   minutes: 'function: 0x010ca694d8'
   new: 'function: 0x010ca5e098'
   strftime: 'function: 0x010caa8fb0'
   tostring: 'function: 0x010ca2f700'
   hours: 'function: 0x010ca671c0'
   parse_date: 'function: 0x010ca96f80'
   seconds: 'function: 0x010ca63b78'
   parse_time: 'function: 0x010ca22208'
   is_interval: 'function: 0x010caa6d20'
   months: 'function: 0x010ca61130'
   is_datetime: 'function: 0x010ca5d3a0'
   asctime: 'function: 0x010caa8e28'
   ctime: 'function: 0x010ca88260'
   parse_zone: 'function: 0x010ca02eb0'
   now: 'function: 0x010ca86e70'
   days: 'function: 0x010ca81980'
   years: 'function: 0x010ca61438'
   parse: 'function: 0x010ca9ad38'
   interval: 'function: 0x010c9feb60'
...

```


It's not clear that part is about intervals and datetime.

```

-- Currently

datetime.now() + datetime.week(1)


-- Possible version

datetime.now() + interval.week(1)

```


Interface is better than in previous time. Thanks for working on this 
series.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * Version #2:
>
>    - fixed problem with overloaded '-' and '+' operations for datetime
>      arguments;
>    - fixed messagepack serialization problems;
>    - heavily documented MessagePack serialization schema in the code;
>    - introduced working implementation of datetime hints for storage engines;
>    - made interval related names be more consistent, renamed durations and period
>      to intervals, i.e. t_datetime_duration to datetime_interval_t,
>      duration_* to interval_*, period to interval;
>    - properly implemented all reasonable cases of datetime+interval
>      arithmetic;
>    - moved all initialization code to utils.c;
>    - renamed core/mp_datetime.c to core/datetime.c because it makes more
>      sense now;
>
> * Version #1 - initial RFC series
>
> In brief
> --------
> This patchset implements datetime lua support in box, with serialization
> to messagepack, yaml, json and lua mode. Also it contains storage
> engines' indices implementation for datetime type introduced.
>
> * Current implementation is heavily influenced by Sci-Lua lua-time module
>    https://github.com/stepelu/lua-time  
>    e.g. you could find very similar approach for handling of operations
>    with year or month long intervals (which should be handled differently than
>    usual intervals of seconds, or days).
>
> * But internally we actually use Christian Hanson' c-dt module
>    https://github.com/chansen/c-dt  
>    (though it has been modified slightly for cleaner integration
>    into cmake build process)
>
>
> Datetime Module API
> -------------------
>
> We used to have here draft documentation of datetime module api, but
> for a convenience it has been extracted to the discussion topic there -
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
>
> Messagepack serialization schema
> --------------------------------
>
> In short it looks like:
> - now we introduce new MP_EXT extension type #4;
> - we may save 1 required and 2 optional fields for datetime field;
>
> In details it's explained in MessagePack serialization schema depicted here:
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990
>
>
> https://github.com/tarantool/tarantool/issues/5941
> https://github.com/tarantool/tarantool/issues/5946
>
> https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v3
>
> Timur Safin (11):
>    build: add Christian Hansen c-dt to the build
>    lua: built-in module datetime
>    lua, datetime: datetime tests
>    lua, datetime: display datetime
>    box, datetime: add messagepack support for datetime
>    box, datetime: datetime comparison for indices
>    lua, datetime: proper datetime encoding
>    lua, datetime: calculated attributes for datetimes
>    lua, datetime: time intervals support
>    lua, datetime: unixtime, timestamp setters in datetime.lua
>    datetime: changelog for datetime module
>
>   .gitmodules                                   |    3 +
>   CMakeLists.txt                                |    8 +
>   .../gh-5941-datetime-type-support.md          |    4 +
>   cmake/BuildCDT.cmake                          |    6 +
>   src/CMakeLists.txt                            |    3 +
>   src/box/field_def.c                           |   52 +-
>   src/box/field_def.h                           |    4 +
>   src/box/lua/serialize_lua.c                   |    7 +-
>   src/box/memtx_space.c                         |    3 +-
>   src/box/msgpack.c                             |    7 +-
>   src/box/tuple_compare.cc                      |   50 +
>   src/box/vinyl.c                               |    3 +-
>   src/exports.h                                 |   29 +
>   src/lib/core/CMakeLists.txt                   |    4 +-
>   src/lib/core/datetime.c                       |  251 ++++
>   src/lib/core/datetime.h                       |  114 ++
>   src/lib/core/mp_extension_types.h             |    1 +
>   src/lib/mpstream/mpstream.c                   |   11 +
>   src/lib/mpstream/mpstream.h                   |    4 +
>   src/lua/datetime.lua                          | 1016 +++++++++++++++++
>   src/lua/init.c                                |    4 +-
>   src/lua/msgpack.c                             |   12 +
>   src/lua/msgpackffi.lua                        |   18 +
>   src/lua/serializer.c                          |    4 +
>   src/lua/serializer.h                          |    2 +
>   src/lua/utils.c                               |   28 +-
>   src/lua/utils.h                               |   12 +
>   test/app-tap/datetime.test.lua                |  367 ++++++
>   test/engine/datetime.result                   |   77 ++
>   test/engine/datetime.test.lua                 |   35 +
>   test/unit/CMakeLists.txt                      |    2 +
>   test/unit/datetime.c                          |  220 ++++
>   test/unit/datetime.result                     |  358 ++++++
>   third_party/c-dt                              |    1 +
>   third_party/lua-cjson/lua_cjson.c             |    8 +
>   third_party/lua-yaml/lyaml.cc                 |    6 +-
>   36 files changed, 2709 insertions(+), 25 deletions(-)
>   create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
>   create mode 100644 cmake/BuildCDT.cmake
>   create mode 100755 src/lib/core/datetime.c
>   create mode 100644 src/lib/core/datetime.h
>   create mode 100644 src/lua/datetime.lua
>   create mode 100755 test/app-tap/datetime.test.lua
>   create mode 100644 test/engine/datetime.result
>   create mode 100644 test/engine/datetime.test.lua
>   create mode 100644 test/unit/datetime.c
>   create mode 100644 test/unit/datetime.result
>   create mode 160000 third_party/c-dt
>

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime Timur Safin via Tarantool-patches
@ 2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-30 19:00     ` Timur Safin via Tarantool-patches
  2021-07-29 23:36   ` Vladislav Shpilevoy via Tarantool-patches
  1 sibling, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:55 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch!


Some places from prevous review are still not fixed for some reasons.


Please be careful with our Lua style guide I ponted some obvious violations.

Also it would be great to analyze module functions with our memprof. I 
think there are some

places that could be optimized.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * created a new Tarantool built-in module `datetime`;
> * register cdef types for this module;
> * export some `dt_*` functions from `c-dt` library;
> * lua implementationis of `asctime` and `strftime`;
> * datetime parsing unit tests, with and withput timezones;
> * c test for reversible strftime roundtrip;
>
> Part of #5941
> ---
>   src/CMakeLists.txt        |   2 +
>   src/exports.h             |  21 ++
>   src/lib/core/datetime.h   |  61 ++++
>   src/lua/datetime.lua      | 581 ++++++++++++++++++++++++++++++++++++++
>   src/lua/init.c            |   4 +-
>   src/lua/utils.c           |  27 ++
>   src/lua/utils.h           |  12 +
>   test/unit/CMakeLists.txt  |   2 +
>   test/unit/datetime.c      | 221 +++++++++++++++
>   test/unit/datetime.result | 358 +++++++++++++++++++++++
>   10 files changed, 1288 insertions(+), 1 deletion(-)
>   create mode 100644 src/lib/core/datetime.h
>   create mode 100644 src/lua/datetime.lua
>   create mode 100644 test/unit/datetime.c
>   create mode 100644 test/unit/datetime.result
>
> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> index ef6a295d5..e0499e57f 100644
> --- a/src/CMakeLists.txt
> +++ b/src/CMakeLists.txt
> @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
>   lua_source(lua_sources lua/httpc.lua)
>   lua_source(lua_sources lua/iconv.lua)
>   lua_source(lua_sources lua/swim.lua)
> +lua_source(lua_sources lua/datetime.lua)
> +
>   # LuaJIT jit.* library
>   lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
>   lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
> diff --git a/src/exports.h b/src/exports.h
> index 5bb3e6a2b..db40c03a4 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -531,3 +531,24 @@ EXPORT(uri_format)
>   EXPORT(uri_parse)
>   EXPORT(uuid_nil)
>   EXPORT(uuid_unpack)
> +EXPORT(dt_from_rdn)
> +EXPORT(dt_from_yd)
> +EXPORT(dt_from_ymd)
> +EXPORT(dt_from_yqd)
> +EXPORT(dt_from_ywd)
> +EXPORT(dt_to_yd)
> +EXPORT(dt_to_ymd)
> +EXPORT(dt_to_yqd)
> +EXPORT(dt_to_ywd)
> +EXPORT(dt_rdn)
> +EXPORT(dt_dow)
> +EXPORT(dt_parse_iso_date)
> +EXPORT(dt_parse_iso_time)
> +EXPORT(dt_parse_iso_time_basic)
> +EXPORT(dt_parse_iso_time_extended)
> +EXPORT(dt_parse_iso_zone)
> +EXPORT(dt_parse_iso_zone_basic)
> +EXPORT(dt_parse_iso_zone_extended)
> +EXPORT(dt_parse_iso_zone_lenient)
> +EXPORT(dt_from_struct_tm)
> +EXPORT(dt_to_struct_tm)
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> new file mode 100644
> index 000000000..403bf1c64
> --- /dev/null
> +++ b/src/lib/core/datetime.h
> @@ -0,0 +1,61 @@
> +#pragma once
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + *    copyright notice, this list of conditions and the
> + *    following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + *    copyright notice, this list of conditions and the following
> + *    disclaimer in the documentation and/or other materials
> + *    provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <c-dt/dt_core.h>
> +#include <stdint.h>
> +#include <stdbool.h>
> +
> +#if defined(__cplusplus)
> +extern "C" {
> +#endif /* defined(__cplusplus) */
> +
> +/**
> + * datetime structure consisting of:
> + */
> +struct datetime_t {
> +	int64_t secs;	///< seconds since epoch
> +	int32_t nsec;	///< nanoseconds if any
> +	int32_t offset; ///< offset in minutes from GMT
> +};
> +
> +/**
> + * Date/time delta structure
> + */
> +struct datetime_interval_t {
> +	int64_t secs; ///< relative seconds delta
> +	int32_t nsec; ///< nanoseconds delta
> +};
> +
> +#if defined(__cplusplus)
> +} /* extern "C" */
> +#endif /* defined(__cplusplus) */
> +
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> new file mode 100644
> index 000000000..0996ca5a2
> --- /dev/null
> +++ b/src/lua/datetime.lua
> @@ -0,0 +1,581 @@
> +local ffi = require('ffi')

Do we have any benchmarks that shows that FFI is faster than Lua C API?

Or it's just easy way to implement such module.


> +local cdt = ffi.C
> +
> +ffi.cdef [[
> +
> +    typedef int dt_t;
> +
> +    // dt_core.h
> +    typedef enum {
> +        DT_MON       = 1,
> +        DT_MONDAY    = 1,
> +        DT_TUE       = 2,
> +        DT_TUESDAY   = 2,
> +        DT_WED       = 3,
> +        DT_WEDNESDAY = 3,
> +        DT_THU       = 4,
> +        DT_THURSDAY  = 4,
> +        DT_FRI       = 5,
> +        DT_FRIDAY    = 5,
> +        DT_SAT       = 6,
> +        DT_SATURDAY  = 6,
> +        DT_SUN       = 7,
> +        DT_SUNDAY    = 7,
> +    } dt_dow_t;
> +
> +    dt_t     dt_from_rdn     (int n);
> +    dt_t     dt_from_yd      (int y, int d);
> +    dt_t     dt_from_ymd     (int y, int m, int d);
> +    dt_t     dt_from_yqd     (int y, int q, int d);
> +    dt_t     dt_from_ywd     (int y, int w, int d);
> +
> +    void     dt_to_yd        (dt_t dt, int *y, int *d);
> +    void     dt_to_ymd       (dt_t dt, int *y, int *m, int *d);
> +    void     dt_to_yqd       (dt_t dt, int *y, int *q, int *d);
> +    void     dt_to_ywd       (dt_t dt, int *y, int *w, int *d);
> +
> +    int      dt_rdn          (dt_t dt);
> +    dt_dow_t dt_dow          (dt_t dt);
> +
> +    // dt_parse_iso.h
> +    size_t dt_parse_iso_date          (const char *str, size_t len, dt_t *dt);
> +
> +    size_t dt_parse_iso_time          (const char *str, size_t len, int *sod, int *nsec);
> +    size_t dt_parse_iso_time_basic    (const char *str, size_t len, int *sod, int *nsec);
> +    size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec);
> +
> +    size_t dt_parse_iso_zone          (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_basic    (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_lenient  (const char *str, size_t len, int *offset);
> +
> +    // dt_tm.h
> +    dt_t    dt_from_struct_tm  (const struct tm *tm);
> +    void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
> +
> +    // <asm-generic/posix_types.h>
> +    typedef long            __kernel_long_t;
> +    typedef unsigned long   __kernel_ulong_t;
> +    // /usr/include/x86_64-linux-gnu/bits/types/time_t.h
> +    typedef long            time_t;
> +
> +
> +    // <time.h>
> +    typedef __kernel_long_t	__kernel_time_t;
> +    typedef __kernel_long_t	__kernel_suseconds_t;
> +
> +    struct timespec {
> +        __kernel_time_t	        tv_sec;     /* seconds */
> +        long                    tv_nsec;    /* nanoseconds */
> +    };
> +
> +    struct timeval {
> +        __kernel_time_t	        tv_sec;	    /* seconds */
> +        __kernel_suseconds_t    tv_usec;    /* microseconds */
> +    };
> +
> +    struct timezone {
> +        int	tz_minuteswest;     /* minutes west of Greenwich */
> +        int	tz_dsttime;	        /* type of dst correction */
> +    };
> +
> +    // /usr/include/x86_64-linux-gnu/sys/time.h
> +    typedef struct timezone * __timezone_ptr_t;
> +
> +    /* Get the current time of day and timezone information,
> +       putting it into *TV and *TZ.  If TZ is NULL, *TZ is not filled.
> +       Returns 0 on success, -1 on errors.
> +
> +       NOTE: This form of timezone information is obsolete.
> +       Use the functions and variables declared in <time.h> instead.  */
> +    int gettimeofday (struct timeval *__tv, struct timezone * __tz);
> +
> +    // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h
> +    /* ISO C `broken-down time' structure.  */
> +    struct tm
> +    {
> +        int tm_sec;	        /* Seconds.	[0-60] (1 leap second) */
> +        int tm_min;	        /* Minutes.	[0-59] */
> +        int tm_hour;        /* Hours.	[0-23] */
> +        int tm_mday;        /* Day.		[1-31] */
> +        int tm_mon;	        /* Month.	[0-11] */
> +        int tm_year;        /* Year	- 1900.  */
> +        int tm_wday;        /* Day of week.	[0-6] */
> +        int tm_yday;        /* Days in year.[0-365]	*/
> +        int tm_isdst;       /* DST.		[-1/0/1]*/
> +
> +        long int tm_gmtoff; /* Seconds east of UTC.  */
> +        const char *tm_zone;/* Timezone abbreviation.  */
> +    };
> +
> +    // <time.h>
> +    /* Return the current time and put it in *TIMER if TIMER is not NULL.  */
> +    time_t time (time_t *__timer);
> +
> +    /* Format TP into S according to FORMAT.
> +    Write no more than MAXSIZE characters and return the number
> +    of characters written, or 0 if it would exceed MAXSIZE.  */
> +    size_t strftime (char * __s, size_t __maxsize, const char * __format,
> +                     const struct tm * __tp);
> +
> +    /* Parse S according to FORMAT and store binary time information in TP.
> +    The return value is a pointer to the first unparsed character in S.  */
> +    char *strptime (const char * __s, const char * __fmt, struct tm *__tp);
> +
> +    /* Return the `struct tm' representation of *TIMER in UTC,
> +    using *TP to store the result.  */
> +    struct tm *gmtime_r (const time_t * __timer, struct tm * __tp);
> +
> +    /* Return the `struct tm' representation of *TIMER in local time,
> +    using *TP to store the result.  */
> +    struct tm *localtime_r (const time_t * __timer, struct tm * __tp);
> +
> +    /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n"
> +    that is the representation of TP in this format.  */
> +    char *asctime (const struct tm *__tp);
> +
> +    /* Equivalent to `asctime (localtime (timer))'.  */
> +    char *ctime (const time_t *__timer);
> +
> +]]
> +
> +local native = ffi.C

You've aleady declared cdt as ffi.C. There is a redefinition here. Is it 
really needed.

I don't see anybody use "native" for ffi.C. There are two ways - 
"builtin" and "C".

I suppose to choose one of them.

> +
> +local SECS_PER_DAY     = 86400
> +local NANOS_PER_SEC    = 1000000000LL
> +
> +-- c-dt/dt_config.h
> +
> +-- Unix, January 1, 1970, Thursday
> +local DT_EPOCH_1970_OFFSET = 719163LL

To be honest it's not completely clear that such value means until

I visited 
https://github.com/chansen/c-dt/blob/21b8cd1fcb984386b7d4552c16fdd03fafab2b6a/dt_config.h#L50.

I think some comment is needed here.


> +
> +
> +local datetime_t = ffi.typeof('struct datetime_t')
> +local interval_t = ffi.typeof('struct datetime_interval_t')
> +
> +local function interval_new()
> +    local interval = ffi.new(interval_t)
> +    return interval
> +end
> +
> +local function adjusted_secs(dt)
> +    return dt.secs - dt.offset * 60
> +end
> +
> +local function datetime_sub(lhs, rhs)
> +    local s1 = adjusted_secs(lhs)
> +    local s2 = adjusted_secs(rhs)
> +    local d = interval_new()
> +    d.secs = s2 - s1
> +    d.nsec = rhs.nsec - lhs.nsec
> +    if d.nsec < 0 then
> +        d.secs = d.secs - 1
> +        d.nsec = d.nsec + NANOS_PER_SEC
> +    end
> +    return d
> +end
> +
> +local function datetime_add(lhs, rhs)
> +    local s1 = adjusted_secs(lhs)
> +    local s2 = adjusted_secs(rhs)
> +    local d = interval_new()
> +    d.secs = s2 + s1
> +    d.nsec = rhs.nsec + lhs.nsec
> +    if d.nsec >= NANOS_PER_SEC then
> +        d.secs = d.secs + 1
> +        d.nsec = d.nsec - NANOS_PER_SEC
> +    end
> +    return d
> +end
> +
> +local function datetime_eq(lhs, rhs)
> +    -- we usually don't need to check nullness
> +    -- but older tarantool console will call us checking for equality to nil
> +    if rhs == nil then
> +        return false
> +    end
> +    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
> +end
> +

As in the previous review it will fail on attempt to compare with not 
datetime value.

tarantool> datetime.new() == newproxy()
---
- error: 'builtin/datetime.lua:222: bad argument #1 to ''is_datetime'' 
(C type expected,
     got userdata)'
...


Expected false.


> +
> +local function datetime_lt(lhs, rhs)
> +    return (lhs.secs < rhs.secs) or
> +           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
> +end
> +
> +local function datetime_le(lhs, rhs)
> +    return (lhs.secs <= rhs.secs) or
> +           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
> +end
> +
> +local function datetime_serialize(self)
> +    -- Allow YAML, MsgPack and JSON to dump objects with sockets
> +    return { secs = self.secs, nsec = self.nsec, tz = self.offset }
> +end
> +
> +local function interval_serialize(self)
> +    -- Allow YAML and JSON to dump objects with sockets
> +    return { secs = self.secs, nsec = self.nsec }
> +end
> +
> +local datetime_mt = {
> +    -- __tostring = datetime_tostring,
> +    __serialize = datetime_serialize,
> +    __eq = datetime_eq,
> +    __lt = datetime_lt,
> +    __le = datetime_le,
> +    __sub = datetime_sub,
> +    __add = datetime_add,
> +
> +    nanoseconds = function(self)
> +        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
I think there should be a space before and after "*". Here and below.
> +    end,
> +    microseconds = function(self)
> +        return tonumber(self.secs*1e6 + self.nsec*1e3)
> +    end,
> +    seconds = function(self)
> +        return tonumber(self.secs + self.nsec*1e3)
> +    end,
> +    minutes = function(self)
> +        return tonumber((self._ticks/(1e6*60))%60)
> +    end,
> +    hours = function(self)
> +        return tonumber(self._ticks/(1e6*60*60))
> +    end,
> +
I think this empty line could be removed.
> +}
> +
> +local interval_mt = {
> +    -- __tostring = interval_tostring,
> +    __serialize = interval_serialize,
> +    __eq = datetime_eq,
> +    __lt = datetime_lt,
> +    __le = datetime_le,
> +}
> +
> +local function datetime_new_raw(secs, nsec, offset)
> +    local dt_obj = ffi.new(datetime_t)
> +    dt_obj.secs = secs
> +    dt_obj.nsec = nsec
> +    dt_obj.offset = offset
> +    return dt_obj
> +end
> +
> +local function local_rd(o)
> +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
> +end
> +
> +local function local_dt(o)
> +    return cdt.dt_from_rdn(local_rd(o))
> +end
> +
> +local function mk_timestamp(dt, sp, fp, offset)
> +    local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0
> +    local spV = sp ~= nil and sp or 0
> +    local fpV = fp ~= nil and fp or 0
> +    local ofsV = offset ~= nil and offset or 0
> +    return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV)
> +end
> +
> +-- create @datetime_t given object @o fields
> +local function datetime_new(o)
> +    if o == nil then
> +        return datetime_new_raw(0, 0, 0)
> +    end
> +    local secs = 0
> +    local nsec = 0
> +    local offset = 0
> +    local easy_way = false
What does "easy_way" mean?
> +    local y, M, d, ymd
> +    y, M, d, ymd = 0, 0, 0, false
> +
> +    local h, m, s, frac, hms
> +    h, m, s, frac, hms = 0, 0, 0, 0, false
> +
> +    local dt = 0
> +
> +    for key, value in pairs(o) do
> +        local handlers = {

Here you recreate handlers for each iteration of the loop. I think it 
should be reworked.

Currenlty it's quite slow I think even if-elseif-else branches will work 
faster without

creating redundant GC objects.

> +            secs = function(v)
> +                secs = v
> +                easy_way = true
> +            end,
> +
> +            nsec = function(v)
> +                nsec = v
> +                easy_way = true
> +            end,
> +
> +            offset = function (v)
> +                offset = v
> +                easy_way = true
> +            end,
> +
> +            year = function(v)
> +                assert(v > 0 and v < 10000)
Still there are some assertions that yield unclear error messages.
> +                y = v
> +                ymd = true
> +            end,
> +
> +            month = function(v)
> +                assert(v > 0 and v < 12 )
> +                M = v
> +                ymd = true
> +            end,
> +
> +            day = function(v)
> +                assert(v > 0 and v < 32)
> +                d = v
> +                ymd = true
> +            end,
> +
> +            hour = function(v)
> +                assert(v >= 0 and v < 24)
> +                h = v
> +                hms = true
> +            end,
> +
> +            minute = function(v)
> +                assert(v >= 0 and v < 60)
> +                m = v
> +                hms = true
> +            end,
> +
> +            second = function(v)
> +                assert(v >= 0 and v < 61)
> +                frac = v % 1
> +                if frac then
> +                    s = v - (v % 1)
> +                else
> +                    s = v
> +                end
> +                hms = true
> +            end,
> +
> +            -- tz offset in minutes
> +            tz = function(v)
> +                assert(v >= 0 and v <= 720)
> +                offset = v
> +            end
> +        }
> +        handlers[key](value)
> +    end
> +
> +    -- .sec, .nsec, .offset
> +    if easy_way then
> +        return datetime_new_raw(secs, nsec, offset)
> +    end
> +
> +    -- .year, .month, .day
> +    if ymd then
> +        dt = dt + cdt.dt_from_ymd(y, M, d)
> +    end
> +
> +    -- .hour, .minute, .second
> +    if hms then
> +        secs = h * 3600 + m * 60 + s
> +    end
> +
> +    return mk_timestamp(dt, secs, frac, offset)
> +end
> +
> +
> +-- simple parse functions:
> +-- parse_date/parse_time/parse_zone
> +
> +--[[
> +    Basic      Extended
> +    20121224   2012-12-24   Calendar date   (ISO 8601)
> +    2012359    2012-359     Ordinal date    (ISO 8601)
> +    2012W521   2012-W52-1   Week date       (ISO 8601)
> +    2012Q485   2012-Q4-85   Quarter date
> +]]
> +
> +local function parse_date(str)
> +    local dt = ffi.new('dt_t[1]')
> +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
> +    assert(rc > 0)
> +    return mk_timestamp(dt[0])
> +end
> +
> +--[[
> +    Basic               Extended
> +    T12                 N/A
> +    T1230               T12:30
> +    T123045             T12:30:45
> +    T123045.123456789   T12:30:45.123456789
> +    T123045,123456789   T12:30:45,123456789
> +
> +    The time designator [T] may be omitted.
> +]]
> +local function parse_time(str)
> +    local sp = ffi.new('int[1]')
> +    local fp = ffi.new('int[1]')
> +    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
> +    assert(rc > 0)
> +    return mk_timestamp(nil, sp[0], fp[0])
> +end
> +
> +--[[
> +    Basic    Extended
> +    Z        N/A
> +    ±hh      N/A
> +    ±hhmm    ±hh:mm
> +]]
> +local function parse_zone(str)
> +    local offset = ffi.new('int[1]')
> +    local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
> +    return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len)
> +end
> +
> +
> +--[[
> +    aggregated parse functions
> +    assumes to deal with date T time time_zone
> +    at once
> +
> +    date [T] time [ ] time_zone
> +]]
> +local function parse_str(str)
> +    local dt = ffi.new('dt_t[1]')
> +    local len = #str
> +    local n = cdt.dt_parse_iso_date(str, len, dt)
> +    local dt_ = dt[0]
> +    if n == 0 or len == n then
> +        return mk_timestamp(dt_)
> +    end
> +
> +    str = str:sub(tonumber(n) + 1)
> +
> +    local ch = str:sub(1,1)
> +    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
> +        return mk_timestamp(dt_)
> +    end
> +
> +    str = str:sub(2)
> +    len = #str
> +
> +    local sp = ffi.new('int[1]')
> +    local fp = ffi.new('int[1]')
> +    local n = cdt.dt_parse_iso_time(str, len, sp, fp)
> +    if n == 0 then
> +        return mk_timestamp(dt_)
> +    end
> +    local sp_ = sp[0]
> +    local fp_ = fp[0]
> +    if len == n then
> +        return mk_timestamp(dt_, sp_, fp_)
> +    end
> +
> +    str = str:sub(tonumber(n) + 1)
> +
> +    if str:sub(1,1) == ' ' then
> +        str = str:sub(2)
> +    end
> +
> +    len = #str
> +
> +    local offset = ffi.new('int[1]')
> +    n = cdt.dt_parse_iso_zone_lenient(str, len, offset)
> +    if n == 0 then
> +        return mk_timestamp(dt_, sp_, fp_)
> +    end
> +    return mk_timestamp(dt_, sp_, fp_, offset[0])
> +end
> +
> +local function datetime_from(o)
> +    if o == nil or type(o) == 'table' then
> +        return datetime_new(o)
> +    elseif type(o) == 'string' then
> +        return parse_str(o)
> +    end
> +end
> +
> +local function local_now()
> +    local p_tv = ffi.new ' struct timeval [1] '

This line doesn't conform our code-style. Please wrap argument into 
brackets.

The same for such places below.

> +    local rc = native.gettimeofday(p_tv, nil)
> +    assert(rc == 0)
> +
> +    local secs = p_tv[0].tv_sec
> +    local nsec = p_tv[0].tv_usec * 1000
> +
> +    local p_time = ffi.new 'time_t[1]'
> +    local p_tm = ffi.new 'struct tm[1]'
> +    native.time(p_time)
> +    native.localtime_r(p_time, p_tm)
> +    -- local dt = cdt.dt_from_struct_tm(p_tm)
> +    local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes
> +
> +    return datetime_new_raw(secs, nsec, ofs) -- FIXME
Do you plan to fix it before merge?
> +end
> +
> +local function datetime_to_tm_ptr(o)
> +    local p_tm = ffi.new 'struct tm[1]'
> +    assert(ffi.typeof(o) == datetime_t)
> +    -- dt_to_struct_tm() fills only date data
> +    cdt.dt_to_struct_tm(local_dt(o), p_tm)
> +
> +    -- calculate the smaller data (hour, minute,
> +    -- seconds) using datetime seconds value
> +    local seconds_of_day = o.secs % 86400
> +    local hour = (seconds_of_day / 3600) % 24
> +    local minute = (seconds_of_day / 60) % 60
> +    p_tm[0].tm_sec = seconds_of_day % 60
> +    p_tm[0].tm_min = minute
> +    p_tm[0].tm_hour = hour
> +
> +    p_tm[0].tm_gmtoff = o.offset * 60
> +
> +    return p_tm
> +end
> +
> +local function asctime(o)
> +    assert(ffi.typeof(o) == datetime_t)
> +    local p_tm = datetime_to_tm_ptr(o)
> +    return ffi.string(native.asctime(p_tm))
> +end
> +
> +local function ctime(o)
> +    assert(ffi.typeof(o) == datetime_t)
> +    local p_time = ffi.new 'time_t[1]'
> +    p_time[0] = o.secs
> +    return ffi.string(native.ctime(p_time))
> +end
> +
> +local function strftime(fmt, o)
> +    assert(ffi.typeof(o) == datetime_t)
> +    local sz = 50
Why 50?
> +    local buff = ffi.new('char[?]', sz)
> +    local p_tm = datetime_to_tm_ptr(o)
> +    native.strftime(buff, sz, fmt, p_tm)
> +    return ffi.string(buff)
> +end
> +
> +-- strftime may be redirected to datetime:fmt("format")
> +local function datetime_fmt()
> +end
> +
> +
> +ffi.metatype(interval_t, interval_mt)
> +ffi.metatype(datetime_t, datetime_mt)
> +
> +return setmetatable(
> +    {
> +        datetime = datetime_new,
> +        interval = interval_new,
> +
> +        parse = parse_str,
> +        parse_date = parse_date,
> +        parse_time = parse_time,
> +        parse_zone = parse_zone,
> +        fmt = datetime_fmt,
> +
> +        now = local_now,
> +    -- strptime = strptime;
It should be dropped if you don't need it.
> +        strftime = strftime,
> +        asctime = asctime,
> +        ctime = ctime,
> +    }, {
> +        __call = function(self, ...) return datetime_from(...) end
> +    }
> +)
> diff --git a/src/lua/init.c b/src/lua/init.c
> index f9738025d..127e935d7 100644
> --- a/src/lua/init.c
> +++ b/src/lua/init.c
> @@ -129,7 +129,8 @@ extern char strict_lua[],
>   	parse_lua[],
>   	process_lua[],
>   	humanize_lua[],
> -	memprof_lua[]
> +	memprof_lua[],
> +	datetime_lua[]
>   ;
>   
>   static const char *lua_modules[] = {
> @@ -184,6 +185,7 @@ static const char *lua_modules[] = {
>   	"memprof.process", process_lua,
>   	"memprof.humanize", humanize_lua,
>   	"memprof", memprof_lua,
> +	"datetime", datetime_lua,
>   	NULL
>   };
>   
> diff --git a/src/lua/utils.c b/src/lua/utils.c
> index 34cec0eed..6aeb18cfe 100644
> --- a/src/lua/utils.c
> +++ b/src/lua/utils.c
> @@ -47,6 +47,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
>   static uint32_t CTID_CHAR_PTR;
>   static uint32_t CTID_CONST_CHAR_PTR;
>   uint32_t CTID_UUID;
> +uint32_t CTID_DATETIME = 0;
> +uint32_t CTID_INTERVAL = 0;
> +
>   
>   void *
>   luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
> @@ -107,6 +110,12 @@ luaL_pushuuid(struct lua_State *L)
>   	return luaL_pushcdata(L, CTID_UUID);
>   }
>   
> +struct datetime_t *
> +luaL_pushdatetime(struct lua_State *L)
> +{
> +	return luaL_pushcdata(L, CTID_DATETIME);
> +}
> +
>   int
>   luaL_iscdata(struct lua_State *L, int idx)
>   {
> @@ -712,6 +721,24 @@ tarantool_lua_utils_init(struct lua_State *L)
>   	CTID_UUID = luaL_ctypeid(L, "struct tt_uuid");
>   	assert(CTID_UUID != 0);
>   
> +	rc = luaL_cdef(L, "struct datetime_t {"
> +			  "int64_t secs;"
> +			  "int32_t nsec;"
> +			  "int32_t offset;"
> +			  "};");
> +	assert(rc == 0);
> +	(void) rc;
> +	CTID_DATETIME = luaL_ctypeid(L, "struct datetime_t");
> +	assert(CTID_DATETIME != 0);
> +	rc = luaL_cdef(L, "struct datetime_interval_t {"
> +			  "int64_t secs;"
> +			  "int32_t nsec;"
> +			  "};");
> +	assert(rc == 0);
> +	(void) rc;
> +	CTID_INTERVAL = luaL_ctypeid(L, "struct datetime_interval_t");
> +	assert(CTID_INTERVAL != 0);
> +
>   	lua_pushcfunction(L, luaT_newthread_wrapper);
>   	luaT_newthread_ref = luaL_ref(L, LUA_REGISTRYINDEX);
>   	return 0;
> diff --git a/src/lua/utils.h b/src/lua/utils.h
> index 947d9240b..afd41c75b 100644
> --- a/src/lua/utils.h
> +++ b/src/lua/utils.h
> @@ -59,6 +59,7 @@ struct lua_State;
>   struct ibuf;
>   typedef struct ibuf box_ibuf_t;
>   struct tt_uuid;
> +struct datetime_t;
>   
>   /**
>    * Single global lua_State shared by core and modules.
> @@ -69,10 +70,21 @@ struct tt_uuid;
>   extern struct lua_State *tarantool_L;
>   
>   extern uint32_t CTID_UUID;
> +extern uint32_t CTID_DATETIME;
> +extern uint32_t CTID_INTERVAL;
>   
>   struct tt_uuid *
>   luaL_pushuuid(struct lua_State *L);
>   
> +/**
> + * @brief Push cdata of a datetime_t type onto the stack.
> + * @param L Lua State
> + * @sa luaL_pushcdata
> + * @return memory associated with this datetime_t data
> + */
> +struct datetime_t *
> +luaL_pushdatetime(struct lua_State *L);
> +
>   /** \cond public */
>   
>   /**
> diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
> index 5bb7cd6e7..f8320aebd 100644
> --- a/test/unit/CMakeLists.txt
> +++ b/test/unit/CMakeLists.txt
> @@ -56,6 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
>   target_link_libraries(uuid.test uuid unit)
>   add_executable(random.test random.c core_test_utils.c)
>   target_link_libraries(random.test core unit)
> +add_executable(datetime.test datetime.c)
> +target_link_libraries(datetime.test cdt unit)
>   
>   add_executable(bps_tree.test bps_tree.cc)
>   target_link_libraries(bps_tree.test small misc)
> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
> new file mode 100644
> index 000000000..b6f568c03
> --- /dev/null
> +++ b/test/unit/datetime.c
> @@ -0,0 +1,221 @@
> +#include "dt.h"
> +#include <assert.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <time.h>
> +
> +#include "unit.h"
> +
> +const char sample[] = "2012-12-24T15:30Z";
> +
> +#define S(s) {s, sizeof(s) - 1}
> +struct {
> +	const char * sz;
> +	size_t len;
> +} tests[] = {
> +	S("2012-12-24 15:30Z"),
> +	S("2012-12-24 15:30z"),
> +	S("2012-12-24 15:30"),
> +	S("2012-12-24 16:30+01:00"),
> +	S("2012-12-24 16:30+0100"),
> +	S("2012-12-24 16:30+01"),
> +	S("2012-12-24 14:30-01:00"),
> +	S("2012-12-24 14:30-0100"),
> +	S("2012-12-24 14:30-01"),
> +	S("2012-12-24 15:30:00Z"),
> +	S("2012-12-24 15:30:00z"),
> +	S("2012-12-24 15:30:00"),
> +	S("2012-12-24 16:30:00+01:00"),
> +	S("2012-12-24 16:30:00+0100"),
> +	S("2012-12-24 14:30:00-01:00"),
> +	S("2012-12-24 14:30:00-0100"),
> +	S("2012-12-24 15:30:00.123456Z"),
> +	S("2012-12-24 15:30:00.123456z"),
> +	S("2012-12-24 15:30:00.123456"),
> +	S("2012-12-24 16:30:00.123456+01:00"),
> +	S("2012-12-24 16:30:00.123456+01"),
> +	S("2012-12-24 14:30:00.123456-01:00"),
> +	S("2012-12-24 14:30:00.123456-01"),
> +	S("2012-12-24t15:30Z"),
> +	S("2012-12-24t15:30z"),
> +	S("2012-12-24t15:30"),
> +	S("2012-12-24t16:30+01:00"),
> +	S("2012-12-24t16:30+0100"),
> +	S("2012-12-24t14:30-01:00"),
> +	S("2012-12-24t14:30-0100"),
> +	S("2012-12-24t15:30:00Z"),
> +	S("2012-12-24t15:30:00z"),
> +	S("2012-12-24t15:30:00"),
> +	S("2012-12-24t16:30:00+01:00"),
> +	S("2012-12-24t16:30:00+0100"),
> +	S("2012-12-24t14:30:00-01:00"),
> +	S("2012-12-24t14:30:00-0100"),
> +	S("2012-12-24t15:30:00.123456Z"),
> +	S("2012-12-24t15:30:00.123456z"),
> +	S("2012-12-24t16:30:00.123456+01:00"),
> +	S("2012-12-24t14:30:00.123456-01:00"),
> +	S("2012-12-24 16:30 +01:00"),
> +	S("2012-12-24 14:30 -01:00"),
> +	S("2012-12-24 15:30 UTC"),
> +	S("2012-12-24 16:30 UTC+1"),
> +	S("2012-12-24 16:30 UTC+01"),
> +	S("2012-12-24 16:30 UTC+0100"),
> +	S("2012-12-24 16:30 UTC+01:00"),
> +	S("2012-12-24 14:30 UTC-1"),
> +	S("2012-12-24 14:30 UTC-01"),
> +	S("2012-12-24 14:30 UTC-01:00"),
> +	S("2012-12-24 14:30 UTC-0100"),
> +	S("2012-12-24 15:30 GMT"),
> +	S("2012-12-24 16:30 GMT+1"),
> +	S("2012-12-24 16:30 GMT+01"),
> +	S("2012-12-24 16:30 GMT+0100"),
> +	S("2012-12-24 16:30 GMT+01:00"),
> +	S("2012-12-24 14:30 GMT-1"),
> +	S("2012-12-24 14:30 GMT-01"),
> +	S("2012-12-24 14:30 GMT-01:00"),
> +	S("2012-12-24 14:30 GMT-0100"),
> +	S("2012-12-24 14:30 -01:00"),
> +	S("2012-12-24 16:30:00 +01:00"),
> +	S("2012-12-24 14:30:00 -01:00"),
> +	S("2012-12-24 16:30:00.123456 +01:00"),
> +	S("2012-12-24 14:30:00.123456 -01:00"),
> +	S("2012-12-24 15:30:00.123456 -00:00"),
> +	S("20121224T1630+01:00"),
> +	S("2012-12-24T1630+01:00"),
> +	S("20121224T16:30+01"),
> +	S("20121224T16:30 +01"),
> +};
> +#undef S
> +
> +#define DIM(a) (sizeof(a) / sizeof(a[0]))
> +
> +// p5-time-moment/src/moment_parse.c: parse_string_lenient()
> +static int
> +parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
> +	       int32_t *op)
> +{
> +	size_t n;
> +	dt_t dt;
> +	char c;
> +	int sod = 0, nanosecond = 0, offset = 0;
> +
> +	n = dt_parse_iso_date(str, len, &dt);
> +	if (!n)
> +		return 1;
> +	if (n == len)
> +		goto exit;
> +
> +	c = str[n++];
> +	if (!(c == 'T' || c == 't' || c == ' '))
> +		return 1;
> +
> +	str += n;
> +	len -= n;
> +
> +	n = dt_parse_iso_time(str, len, &sod, &nanosecond);
> +	if (!n)
> +		return 1;
> +	if (n == len)
> +		goto exit;
> +
> +	if (str[n] == ' ')
> +	n++;
> +
> +	str += n;
> +	len -= n;
> +
> +	n = dt_parse_iso_zone_lenient(str, len, &offset);
> +	if (!n || n != len)
> +		return 1;
> +
> +exit:
> +	*sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
> +	*np = nanosecond;
> +	*op = offset;
> +
> +	return 0;
> +}
> +
> +// avoid introducing external datetime.h dependency
> +// - just copy paste it for today
> +#define SECS_PER_DAY      86400
> +#define NANOS_PER_SEC     1000000000
> +#define DT_EPOCH_1970_OFFSET 719163
> +
> +
> +struct datetime_t {
> +	int64_t secs;
> +	int32_t nsec;
> +	int32_t offset;
> +};
> +
> +static int
> +local_rd(const struct datetime_t * dt) {
> +	return (int)(dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET;
> +}
> +
> +static int
> +local_dt(const struct datetime_t * dt) {
> +	return dt_from_rdn(local_rd(dt));
> +}
> +
> +
> +struct tm*
> +datetime_to_tm(struct datetime_t * dt)
> +{
> +	static struct tm tm;
> +
> +	memset(&tm, 0, sizeof(tm));
> +	dt_to_struct_tm(local_dt(dt), &tm);
> +
> +	int seconds_of_day = dt->secs % 86400;
> +	tm.tm_hour = (seconds_of_day / 3600) % 24;
> +	tm.tm_min = (seconds_of_day / 60) % 60;
> +	tm.tm_sec = seconds_of_day % 60;
> +
> +	return &tm;
> +}
> +
> +static void datetime_test(void)
> +{
> +	size_t index;
> +	int64_t secs_expected;
> +	int32_t nanosecs;
> +	int32_t ofs;
> +
> +	plan(355);
> +	parse_datetime(sample, sizeof(sample) - 1,
> +		       &secs_expected, &nanosecs, &ofs);
> +
> +	for (index = 0; index < DIM(tests); index++) {
> +		int64_t secs;
> +		int rc = parse_datetime(tests[index].sz, tests[index].len,
> +						&secs, &nanosecs, &ofs);
> +		is(rc, 0, "correct parse_datetime return value for '%s'",
> +		   tests[index].sz);
> +		is(secs, secs_expected, "correct parse_datetime output "
> +		   "seconds for '%s", tests[index].sz);
> +
> +		// check that stringized literal produces the same date
> +		// time fields
> +		static char buff[40];
> +		struct datetime_t dt = {secs, nanosecs, ofs};
> +		// datetime_to_tm returns time in GMT zone
> +		struct tm * p_tm = datetime_to_tm(&dt);
> +		size_t len = strftime(buff, sizeof buff, "%F %T%z", p_tm);
> +		ok(len > 0, "strftime");
> +		rc = parse_datetime(buff, len, &dt.secs, &dt.nsec, &dt.offset);
> +		is(rc, 0, "correct parse_datetime return value for '%s'", buff);
> +		is(secs, dt.secs,
> +		   "reversible seconds via strftime for '%s", buff);
> +	}
> +}
> +
> +int
> +main(void)
> +{
> +	plan(1);
> +	datetime_test();
> +
> +	return check_plan();
> +}
> diff --git a/test/unit/datetime.result b/test/unit/datetime.result
> new file mode 100644
> index 000000000..f106fa769
> --- /dev/null
> +++ b/test/unit/datetime.result
> @@ -0,0 +1,358 @@
> +1..1
> +    1..355
> +    ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z'
> +    ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z
> +    ok 3 - strftime
> +    ok 4 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 5 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 6 - correct parse_datetime return value for '2012-12-24 15:30z'
> +    ok 7 - correct parse_datetime output seconds for '2012-12-24 15:30z
> +    ok 8 - strftime
> +    ok 9 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 10 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 11 - correct parse_datetime return value for '2012-12-24 15:30'
> +    ok 12 - correct parse_datetime output seconds for '2012-12-24 15:30
> +    ok 13 - strftime
> +    ok 14 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 15 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 16 - correct parse_datetime return value for '2012-12-24 16:30+01:00'
> +    ok 17 - correct parse_datetime output seconds for '2012-12-24 16:30+01:00
> +    ok 18 - strftime
> +    ok 19 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 20 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 21 - correct parse_datetime return value for '2012-12-24 16:30+0100'
> +    ok 22 - correct parse_datetime output seconds for '2012-12-24 16:30+0100
> +    ok 23 - strftime
> +    ok 24 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 25 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 26 - correct parse_datetime return value for '2012-12-24 16:30+01'
> +    ok 27 - correct parse_datetime output seconds for '2012-12-24 16:30+01
> +    ok 28 - strftime
> +    ok 29 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 30 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 31 - correct parse_datetime return value for '2012-12-24 14:30-01:00'
> +    ok 32 - correct parse_datetime output seconds for '2012-12-24 14:30-01:00
> +    ok 33 - strftime
> +    ok 34 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 35 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 36 - correct parse_datetime return value for '2012-12-24 14:30-0100'
> +    ok 37 - correct parse_datetime output seconds for '2012-12-24 14:30-0100
> +    ok 38 - strftime
> +    ok 39 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 40 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 41 - correct parse_datetime return value for '2012-12-24 14:30-01'
> +    ok 42 - correct parse_datetime output seconds for '2012-12-24 14:30-01
> +    ok 43 - strftime
> +    ok 44 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 45 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 46 - correct parse_datetime return value for '2012-12-24 15:30:00Z'
> +    ok 47 - correct parse_datetime output seconds for '2012-12-24 15:30:00Z
> +    ok 48 - strftime
> +    ok 49 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 50 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 51 - correct parse_datetime return value for '2012-12-24 15:30:00z'
> +    ok 52 - correct parse_datetime output seconds for '2012-12-24 15:30:00z
> +    ok 53 - strftime
> +    ok 54 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 55 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 56 - correct parse_datetime return value for '2012-12-24 15:30:00'
> +    ok 57 - correct parse_datetime output seconds for '2012-12-24 15:30:00
> +    ok 58 - strftime
> +    ok 59 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 60 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 61 - correct parse_datetime return value for '2012-12-24 16:30:00+01:00'
> +    ok 62 - correct parse_datetime output seconds for '2012-12-24 16:30:00+01:00
> +    ok 63 - strftime
> +    ok 64 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 65 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 66 - correct parse_datetime return value for '2012-12-24 16:30:00+0100'
> +    ok 67 - correct parse_datetime output seconds for '2012-12-24 16:30:00+0100
> +    ok 68 - strftime
> +    ok 69 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 70 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 71 - correct parse_datetime return value for '2012-12-24 14:30:00-01:00'
> +    ok 72 - correct parse_datetime output seconds for '2012-12-24 14:30:00-01:00
> +    ok 73 - strftime
> +    ok 74 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 75 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 76 - correct parse_datetime return value for '2012-12-24 14:30:00-0100'
> +    ok 77 - correct parse_datetime output seconds for '2012-12-24 14:30:00-0100
> +    ok 78 - strftime
> +    ok 79 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 80 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 81 - correct parse_datetime return value for '2012-12-24 15:30:00.123456Z'
> +    ok 82 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456Z
> +    ok 83 - strftime
> +    ok 84 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 85 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 86 - correct parse_datetime return value for '2012-12-24 15:30:00.123456z'
> +    ok 87 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456z
> +    ok 88 - strftime
> +    ok 89 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 90 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 91 - correct parse_datetime return value for '2012-12-24 15:30:00.123456'
> +    ok 92 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456
> +    ok 93 - strftime
> +    ok 94 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 95 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 96 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01:00'
> +    ok 97 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01:00
> +    ok 98 - strftime
> +    ok 99 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 100 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 101 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01'
> +    ok 102 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01
> +    ok 103 - strftime
> +    ok 104 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 105 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 106 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01:00'
> +    ok 107 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01:00
> +    ok 108 - strftime
> +    ok 109 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 110 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 111 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01'
> +    ok 112 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01
> +    ok 113 - strftime
> +    ok 114 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 115 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 116 - correct parse_datetime return value for '2012-12-24t15:30Z'
> +    ok 117 - correct parse_datetime output seconds for '2012-12-24t15:30Z
> +    ok 118 - strftime
> +    ok 119 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 120 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 121 - correct parse_datetime return value for '2012-12-24t15:30z'
> +    ok 122 - correct parse_datetime output seconds for '2012-12-24t15:30z
> +    ok 123 - strftime
> +    ok 124 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 125 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 126 - correct parse_datetime return value for '2012-12-24t15:30'
> +    ok 127 - correct parse_datetime output seconds for '2012-12-24t15:30
> +    ok 128 - strftime
> +    ok 129 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 130 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 131 - correct parse_datetime return value for '2012-12-24t16:30+01:00'
> +    ok 132 - correct parse_datetime output seconds for '2012-12-24t16:30+01:00
> +    ok 133 - strftime
> +    ok 134 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 135 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 136 - correct parse_datetime return value for '2012-12-24t16:30+0100'
> +    ok 137 - correct parse_datetime output seconds for '2012-12-24t16:30+0100
> +    ok 138 - strftime
> +    ok 139 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 140 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 141 - correct parse_datetime return value for '2012-12-24t14:30-01:00'
> +    ok 142 - correct parse_datetime output seconds for '2012-12-24t14:30-01:00
> +    ok 143 - strftime
> +    ok 144 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 145 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 146 - correct parse_datetime return value for '2012-12-24t14:30-0100'
> +    ok 147 - correct parse_datetime output seconds for '2012-12-24t14:30-0100
> +    ok 148 - strftime
> +    ok 149 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 150 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 151 - correct parse_datetime return value for '2012-12-24t15:30:00Z'
> +    ok 152 - correct parse_datetime output seconds for '2012-12-24t15:30:00Z
> +    ok 153 - strftime
> +    ok 154 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 155 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 156 - correct parse_datetime return value for '2012-12-24t15:30:00z'
> +    ok 157 - correct parse_datetime output seconds for '2012-12-24t15:30:00z
> +    ok 158 - strftime
> +    ok 159 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 160 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 161 - correct parse_datetime return value for '2012-12-24t15:30:00'
> +    ok 162 - correct parse_datetime output seconds for '2012-12-24t15:30:00
> +    ok 163 - strftime
> +    ok 164 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 165 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 166 - correct parse_datetime return value for '2012-12-24t16:30:00+01:00'
> +    ok 167 - correct parse_datetime output seconds for '2012-12-24t16:30:00+01:00
> +    ok 168 - strftime
> +    ok 169 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 170 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 171 - correct parse_datetime return value for '2012-12-24t16:30:00+0100'
> +    ok 172 - correct parse_datetime output seconds for '2012-12-24t16:30:00+0100
> +    ok 173 - strftime
> +    ok 174 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 175 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 176 - correct parse_datetime return value for '2012-12-24t14:30:00-01:00'
> +    ok 177 - correct parse_datetime output seconds for '2012-12-24t14:30:00-01:00
> +    ok 178 - strftime
> +    ok 179 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 180 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 181 - correct parse_datetime return value for '2012-12-24t14:30:00-0100'
> +    ok 182 - correct parse_datetime output seconds for '2012-12-24t14:30:00-0100
> +    ok 183 - strftime
> +    ok 184 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 185 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 186 - correct parse_datetime return value for '2012-12-24t15:30:00.123456Z'
> +    ok 187 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456Z
> +    ok 188 - strftime
> +    ok 189 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 190 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 191 - correct parse_datetime return value for '2012-12-24t15:30:00.123456z'
> +    ok 192 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456z
> +    ok 193 - strftime
> +    ok 194 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 195 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 196 - correct parse_datetime return value for '2012-12-24t16:30:00.123456+01:00'
> +    ok 197 - correct parse_datetime output seconds for '2012-12-24t16:30:00.123456+01:00
> +    ok 198 - strftime
> +    ok 199 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 200 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 201 - correct parse_datetime return value for '2012-12-24t14:30:00.123456-01:00'
> +    ok 202 - correct parse_datetime output seconds for '2012-12-24t14:30:00.123456-01:00
> +    ok 203 - strftime
> +    ok 204 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 205 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 206 - correct parse_datetime return value for '2012-12-24 16:30 +01:00'
> +    ok 207 - correct parse_datetime output seconds for '2012-12-24 16:30 +01:00
> +    ok 208 - strftime
> +    ok 209 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 210 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 211 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
> +    ok 212 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
> +    ok 213 - strftime
> +    ok 214 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 215 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 216 - correct parse_datetime return value for '2012-12-24 15:30 UTC'
> +    ok 217 - correct parse_datetime output seconds for '2012-12-24 15:30 UTC
> +    ok 218 - strftime
> +    ok 219 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 220 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 221 - correct parse_datetime return value for '2012-12-24 16:30 UTC+1'
> +    ok 222 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+1
> +    ok 223 - strftime
> +    ok 224 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 225 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 226 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01'
> +    ok 227 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01
> +    ok 228 - strftime
> +    ok 229 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 230 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 231 - correct parse_datetime return value for '2012-12-24 16:30 UTC+0100'
> +    ok 232 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+0100
> +    ok 233 - strftime
> +    ok 234 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 235 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 236 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01:00'
> +    ok 237 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01:00
> +    ok 238 - strftime
> +    ok 239 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 240 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 241 - correct parse_datetime return value for '2012-12-24 14:30 UTC-1'
> +    ok 242 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-1
> +    ok 243 - strftime
> +    ok 244 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 245 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 246 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01'
> +    ok 247 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01
> +    ok 248 - strftime
> +    ok 249 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 250 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 251 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01:00'
> +    ok 252 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01:00
> +    ok 253 - strftime
> +    ok 254 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 255 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 256 - correct parse_datetime return value for '2012-12-24 14:30 UTC-0100'
> +    ok 257 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-0100
> +    ok 258 - strftime
> +    ok 259 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 260 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 261 - correct parse_datetime return value for '2012-12-24 15:30 GMT'
> +    ok 262 - correct parse_datetime output seconds for '2012-12-24 15:30 GMT
> +    ok 263 - strftime
> +    ok 264 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 265 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 266 - correct parse_datetime return value for '2012-12-24 16:30 GMT+1'
> +    ok 267 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+1
> +    ok 268 - strftime
> +    ok 269 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 270 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 271 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01'
> +    ok 272 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01
> +    ok 273 - strftime
> +    ok 274 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 275 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 276 - correct parse_datetime return value for '2012-12-24 16:30 GMT+0100'
> +    ok 277 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+0100
> +    ok 278 - strftime
> +    ok 279 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 280 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 281 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01:00'
> +    ok 282 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01:00
> +    ok 283 - strftime
> +    ok 284 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 285 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 286 - correct parse_datetime return value for '2012-12-24 14:30 GMT-1'
> +    ok 287 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-1
> +    ok 288 - strftime
> +    ok 289 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 290 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 291 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01'
> +    ok 292 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01
> +    ok 293 - strftime
> +    ok 294 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 295 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 296 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01:00'
> +    ok 297 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01:00
> +    ok 298 - strftime
> +    ok 299 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 300 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 301 - correct parse_datetime return value for '2012-12-24 14:30 GMT-0100'
> +    ok 302 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-0100
> +    ok 303 - strftime
> +    ok 304 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 305 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 306 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
> +    ok 307 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
> +    ok 308 - strftime
> +    ok 309 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 310 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 311 - correct parse_datetime return value for '2012-12-24 16:30:00 +01:00'
> +    ok 312 - correct parse_datetime output seconds for '2012-12-24 16:30:00 +01:00
> +    ok 313 - strftime
> +    ok 314 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 315 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 316 - correct parse_datetime return value for '2012-12-24 14:30:00 -01:00'
> +    ok 317 - correct parse_datetime output seconds for '2012-12-24 14:30:00 -01:00
> +    ok 318 - strftime
> +    ok 319 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 320 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 321 - correct parse_datetime return value for '2012-12-24 16:30:00.123456 +01:00'
> +    ok 322 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456 +01:00
> +    ok 323 - strftime
> +    ok 324 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 325 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 326 - correct parse_datetime return value for '2012-12-24 14:30:00.123456 -01:00'
> +    ok 327 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456 -01:00
> +    ok 328 - strftime
> +    ok 329 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 330 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 331 - correct parse_datetime return value for '2012-12-24 15:30:00.123456 -00:00'
> +    ok 332 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 -00:00
> +    ok 333 - strftime
> +    ok 334 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 335 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 336 - correct parse_datetime return value for '20121224T1630+01:00'
> +    ok 337 - correct parse_datetime output seconds for '20121224T1630+01:00
> +    ok 338 - strftime
> +    ok 339 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 340 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 341 - correct parse_datetime return value for '2012-12-24T1630+01:00'
> +    ok 342 - correct parse_datetime output seconds for '2012-12-24T1630+01:00
> +    ok 343 - strftime
> +    ok 344 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 345 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 346 - correct parse_datetime return value for '20121224T16:30+01'
> +    ok 347 - correct parse_datetime output seconds for '20121224T16:30+01
> +    ok 348 - strftime
> +    ok 349 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 350 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +    ok 351 - correct parse_datetime return value for '20121224T16:30 +01'
> +    ok 352 - correct parse_datetime output seconds for '20121224T16:30 +01
> +    ok 353 - strftime
> +    ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00+0000'
> +    ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00+0000
> +ok 1 - subtests

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests Timur Safin via Tarantool-patches
@ 2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-30 20:45     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:55 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch. To be honest I don't understand the structure of 
this changes.

I expect that "datetime tests" will contain only tests. All unrelated to 
tests changes

should be squashed to the previous patch.


I put several comments below.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * created app-tap test for new builtin module `datetime.lua`
> * added case to check datetime string formatting using:
>    - asctime (gmt time);
>    - ctime (local TZ time);
>    - strftime (using given format).
>
> * added positive/negative checks to datetime test
>    - extended api of datetime.parse_date, .parse_time, .parse_time_zone
>      with a length of parsed (sub)string;
>    - this allows us to check partially valid strings like "20121224 Foo bar".
>
> Part of #5941
> ---
>   src/lua/datetime.lua           |  14 ++-
>   test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++
>   2 files changed, 197 insertions(+), 8 deletions(-)
>   create mode 100755 test/app-tap/datetime.test.lua
>
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 0996ca5a2..f4d2d7737 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -318,7 +318,7 @@ local function datetime_new(o)
>               end,
>   
>               month = function(v)
> -                assert(v > 0 and v < 12 )
> +                assert(v > 0 and v < 13 )
>                   M = v
>                   ymd = true
>               end,
> @@ -393,9 +393,8 @@ end
>   
>   local function parse_date(str)
>       local dt = ffi.new('dt_t[1]')
> -    local rc = cdt.dt_parse_iso_date(str, #str, dt)
> -    assert(rc > 0)
> -    return mk_timestamp(dt[0])
> +    local len = cdt.dt_parse_iso_date(str, #str, dt)
> +    return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len)
>   end
>   
>   --[[
> @@ -411,9 +410,8 @@ end
>   local function parse_time(str)
>       local sp = ffi.new('int[1]')
>       local fp = ffi.new('int[1]')
> -    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
> -    assert(rc > 0)
> -    return mk_timestamp(nil, sp[0], fp[0])
> +    local len = cdt.dt_parse_iso_time(str, #str, sp, fp)
> +    return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil, tonumber(len)
>   end
>   
>   --[[
> @@ -448,7 +446,7 @@ local function parse_str(str)
>       str = str:sub(tonumber(n) + 1)
>   
>       local ch = str:sub(1,1)
> -    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
> +    if ch:match('[Tt ]') == nil then
>           return mk_timestamp(dt_)
>       end
>   
> diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
> new file mode 100755
> index 000000000..09c968858
> --- /dev/null
> +++ b/test/app-tap/datetime.test.lua
> @@ -0,0 +1,191 @@
> +#!/usr/bin/env tarantool
> +
> +local tap = require('tap')
> +local test = tap.test("errno")
> +local date = require('datetime')
> +
> +test:plan(5)
> +
> +test:test("Simple tests for parser", function(test)
> +    test:plan(2)
> +    test:ok(date("1970-01-01T01:00:00Z") ==
> +            date {year=1970, month=1, day=1, hour=1, minute=0, second=0})
> +    test:ok(date("1970-01-01T02:00:00+02:00") ==
> +            date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120})
> +end)
> +
> +test:test("Multiple tests for parser (with nanoseconds)", function(test)
> +    test:plan(165)
> +    -- borrowed from p5-time-moments/t/180_from_string.t
> +    local tests =
> +    {
> +        { '1970-01-01T00:00:00Z',                       0,           0,    0 },
> +        { '1970-01-01T02:00:00+02:00',                  0,           0,  120 },
> +        { '1970-01-01T01:30:00+01:30',                  0,           0,   90 },
> +        { '1970-01-01T01:00:00+01:00',                  0,           0,   60 },
> +        { '1970-01-01T00:01:00+00:01',                  0,           0,    1 },
> +        { '1970-01-01T00:00:00+00:00',                  0,           0,    0 },
> +        { '1969-12-31T23:59:00-00:01',                  0,           0,   -1 },
> +        { '1969-12-31T23:00:00-01:00',                  0,           0,  -60 },
> +        { '1969-12-31T22:30:00-01:30',                  0,           0,  -90 },
> +        { '1969-12-31T22:00:00-02:00',                  0,           0, -120 },
> +        { '1970-01-01T00:00:00.123456789Z',             0,   123456789,    0 },
> +        { '1970-01-01T00:00:00.12345678Z',              0,   123456780,    0 },
> +        { '1970-01-01T00:00:00.1234567Z',               0,   123456700,    0 },
> +        { '1970-01-01T00:00:00.123456Z',                0,   123456000,    0 },
> +        { '1970-01-01T00:00:00.12345Z',                 0,   123450000,    0 },
> +        { '1970-01-01T00:00:00.1234Z',                  0,   123400000,    0 },
> +        { '1970-01-01T00:00:00.123Z',                   0,   123000000,    0 },
> +        { '1970-01-01T00:00:00.12Z',                    0,   120000000,    0 },
> +        { '1970-01-01T00:00:00.1Z',                     0,   100000000,    0 },
> +        { '1970-01-01T00:00:00.01Z',                    0,    10000000,    0 },
> +        { '1970-01-01T00:00:00.001Z',                   0,     1000000,    0 },
> +        { '1970-01-01T00:00:00.0001Z',                  0,      100000,    0 },
> +        { '1970-01-01T00:00:00.00001Z',                 0,       10000,    0 },
> +        { '1970-01-01T00:00:00.000001Z',                0,        1000,    0 },
> +        { '1970-01-01T00:00:00.0000001Z',               0,         100,    0 },
> +        { '1970-01-01T00:00:00.00000001Z',              0,          10,    0 },
> +        { '1970-01-01T00:00:00.000000001Z',             0,           1,    0 },
> +        { '1970-01-01T00:00:00.000000009Z',             0,           9,    0 },
> +        { '1970-01-01T00:00:00.00000009Z',              0,          90,    0 },
> +        { '1970-01-01T00:00:00.0000009Z',               0,         900,    0 },
> +        { '1970-01-01T00:00:00.000009Z',                0,        9000,    0 },
> +        { '1970-01-01T00:00:00.00009Z',                 0,       90000,    0 },
> +        { '1970-01-01T00:00:00.0009Z',                  0,      900000,    0 },
> +        { '1970-01-01T00:00:00.009Z',                   0,     9000000,    0 },
> +        { '1970-01-01T00:00:00.09Z',                    0,    90000000,    0 },
> +        { '1970-01-01T00:00:00.9Z',                     0,   900000000,    0 },
> +        { '1970-01-01T00:00:00.99Z',                    0,   990000000,    0 },
> +        { '1970-01-01T00:00:00.999Z',                   0,   999000000,    0 },
> +        { '1970-01-01T00:00:00.9999Z',                  0,   999900000,    0 },
> +        { '1970-01-01T00:00:00.99999Z',                 0,   999990000,    0 },
> +        { '1970-01-01T00:00:00.999999Z',                0,   999999000,    0 },
> +        { '1970-01-01T00:00:00.9999999Z',               0,   999999900,    0 },
> +        { '1970-01-01T00:00:00.99999999Z',              0,   999999990,    0 },
> +        { '1970-01-01T00:00:00.999999999Z',             0,   999999999,    0 },
> +        { '1970-01-01T00:00:00.0Z',                     0,           0,    0 },
> +        { '1970-01-01T00:00:00.00Z',                    0,           0,    0 },
> +        { '1970-01-01T00:00:00.000Z',                   0,           0,    0 },
> +        { '1970-01-01T00:00:00.0000Z',                  0,           0,    0 },
> +        { '1970-01-01T00:00:00.00000Z',                 0,           0,    0 },
> +        { '1970-01-01T00:00:00.000000Z',                0,           0,    0 },
> +        { '1970-01-01T00:00:00.0000000Z',               0,           0,    0 },
> +        { '1970-01-01T00:00:00.00000000Z',              0,           0,    0 },
> +        { '1970-01-01T00:00:00.000000000Z',             0,           0,    0 },
> +        { '1973-11-29T21:33:09Z',               123456789,           0,    0 },
> +        { '2013-10-28T17:51:56Z',              1382982716,           0,    0 },
> +        -- { '9999-12-31T23:59:59Z',            253402300799,           0,    0 },

Why is it commented?

> +    }
> +    for _, value in ipairs(tests) do
> +        local str, epoch, nsec, offset
> +        str, epoch, nsec, offset = unpack(value)
> +        local dt = date(str)
> +        test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
> +        test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
> +        test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
> +    end
> +end)
> +
> +local ffi = require('ffi')
> +
> +ffi.cdef [[
> +    void tzset(void);
> +]]
> +


It's probably better to require all modules at the top of the file.

> +test:test("Datetime string formatting", function(test)
> +    test:plan(7)
> +    local str = "1970-01-01"
> +    local t = date(str)
> +    test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, tonumber(t.secs)))
> +    test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec))
> +    test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset))
> +    test:ok(date.asctime(t) == 'Thu Jan  1 00:00:00 1970\n', ('%s: asctime'):format(str))
> +    -- ctime() is local timezone dependent. To make sure that
> +    -- test is deterministic we enforce timezone via TZ environment
> +    -- manipulations and calling tzset()
> +
> +    -- redefine timezone to be always GMT-2
> +    os.setenv('TZ', 'GMT-2')
> +    ffi.C.tzset()
> +    test:ok(date.ctime(t) == 'Thu Jan  1 02:00:00 1970\n', ('%s: ctime with timezone'):format(str))
> +    test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime #1'):format(str))
> +    test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January 1970', ('%s: strftime #2'):format(str))
> +end)
> +
> +test:test("Parse iso date - valid strings", function(test)
> +    test:plan(32)
> +    local good = {
> +        {2012, 12, 24, "20121224",                   8 },
> +        {2012, 12, 24, "20121224  Foo bar",          8 },
> +        {2012, 12, 24, "2012-12-24",                10 },
> +        {2012, 12, 24, "2012-12-24 23:59:59",       10 },
> +        {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 },
> +        {2012, 12, 24, "2012359",                    7 },
> +        {2012, 12, 24, "2012359T235959+0130",        7 },
> +        {2012, 12, 24, "2012-359",                   8 },
> +        {2012, 12, 24, "2012W521",                   8 },
> +        {2012, 12, 24, "2012-W52-1",                10 },
> +        {2012, 12, 24, "2012Q485",                   8 },
> +        {2012, 12, 24, "2012-Q4-85",                10 },
> +        {   1,  1,  1, "0001-Q1-01",                10 },
> +        {   1,  1,  1, "0001-W01-1",                10 },
> +        {   1,  1,  1, "0001-01-01",                10 },
> +        {   1,  1,  1, "0001-001",                   8 },
> +    }
> +
> +    for _, value in ipairs(good) do
> +        local year, month, day, str, date_part_len;

I didn't see before that somebody uses semicolon after variables definition.

It's about code-style. It is clear that this is syntactically correct.

> +        year, month, day, str, date_part_len = unpack(value)
> +        local expected_date = date{year = year, month = month, day = day}
> +        local date_part, len
> +        date_part, len = date.parse_date(str)
> +        test:ok(len == date_part_len, ('%s: length check %d'):format(str, len))
> +        test:ok(expected_date == date_part, ('%s: expected date'):format(str))
> +    end
> +end)
> +
> +test:test("Parse iso date - invalid strings", function(test)
> +    test:plan(62)
> +    local bad = {
> +        "20121232"   , -- Invalid day of month
> +        "2012-12-310", -- Invalid day of month
> +        "2012-13-24" , -- Invalid month
> +        "2012367"    , -- Invalid day of year
> +        "2012-000"   , -- Invalid day of year
> +        "2012W533"   , -- Invalid week of year
> +        "2012-W52-8" , -- Invalid day of week
> +        "2012Q495"   , -- Invalid day of quarter
> +        "2012-Q5-85" , -- Invalid quarter
> +        "20123670"   , -- Trailing digit
> +        "201212320"  , -- Trailing digit
> +        "2012-12"    , -- Reduced accuracy
> +        "2012-Q4"    , -- Reduced accuracy
> +        "2012-Q42"   , -- Invalid
> +        "2012-Q1-1"  , -- Invalid day of quarter
> +        "2012Q--420" , -- Invalid
> +        "2012-Q-420" , -- Invalid
> +        "2012Q11"    , -- Incomplete
> +        "2012Q1234"  , -- Trailing digit
> +        "2012W12"    , -- Incomplete
> +        "2012W1234"  , -- Trailing digit
> +        "2012W-123"  , -- Invalid
> +        "2012-W12"   , -- Incomplete
> +        "2012-W12-12", -- Trailing digit
> +        "2012U1234"  , -- Invalid
> +        "2012-1234"  , -- Invalid
> +        "2012-X1234" , -- Invalid
> +        "0000-Q1-01" , -- Year less than 0001
> +        "0000-W01-1" , -- Year less than 0001
> +        "0000-01-01" , -- Year less than 0001
> +        "0000-001"   , -- Year less than 0001
> +    }
> +
> +    for _, str in ipairs(bad) do
> +        local date_part, len
> +        date_part, len = date.parse_date(str)
> +        test:ok(len == 0, ('%s: length check %d'):format(str, len))
> +        test:ok(date_part == nil, ('%s: empty date check %s'):format(str, date_part))
> +    end
> +end)
> +
> +os.exit(test:check() and 0 or 1)

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime Timur Safin via Tarantool-patches
@ 2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
  2021-07-30 21:48     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:55 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch. It's not review at all. I have a question:

why is datetime not scalar?


Also I think that encoding and type system changes should live in 
separate patches.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * introduced output routine for converting datetime
>    to their default output format.
>
> * use this routine for tostring() in datetime.lua
>
> Part of #5941
> ---
>   src/exports.h               |  1 +
>   src/lib/core/CMakeLists.txt |  1 +
>   src/lib/core/datetime.c     | 85 +++++++++++++++++++++++++++++++++++++
>   src/lib/core/datetime.h     | 14 ++++++
>   src/lua/datetime.lua        | 18 ++++++--
>   5 files changed, 116 insertions(+), 3 deletions(-)
>   create mode 100755 src/lib/core/datetime.c
>
> diff --git a/src/exports.h b/src/exports.h
> index db40c03a4..1a03db636 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -552,3 +552,4 @@ EXPORT(dt_parse_iso_zone_extended)
>   EXPORT(dt_parse_iso_zone_lenient)
>   EXPORT(dt_from_struct_tm)
>   EXPORT(dt_to_struct_tm)
> +EXPORT(datetime_to_string)
> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
> index 2cd4d0b4f..8bc776b82 100644
> --- a/src/lib/core/CMakeLists.txt
> +++ b/src/lib/core/CMakeLists.txt
> @@ -30,6 +30,7 @@ set(core_sources
>       decimal.c
>       mp_decimal.c
>       cord_buf.c
> +    datetime.c
>   )
>   
>   if (TARGET_OS_NETBSD)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> new file mode 100755
> index 000000000..65f813a70
> --- /dev/null
> +++ b/src/lib/core/datetime.c
> @@ -0,0 +1,85 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + *    copyright notice, this list of conditions and the
> + *    following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + *    copyright notice, this list of conditions and the following
> + *    disclaimer in the documentation and/or other materials
> + *    provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <string.h>
> +
> +#include "trivia/util.h"
> +#include "datetime.h"
> +
> +int
> +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len)
> +{
> +	char * src = buf;
> +	int offset = date->offset;
> +	int64_t secs = date->secs + offset * 60;
> +	dt_t dt = dt_from_rdn((secs / SECS_PER_DAY) + 719163);
> +
> +	int year, month, day, sec, ns, sign;
> +	dt_to_ymd(dt, &year, &month, &day);
> +
> +	int hour = (secs / 3600) % 24,
> +	    minute = (secs / 60) % 60;
> +	;
> +	sec = secs % 60;
> +	ns = date->nsec;
> +	uint32_t sz;
> +	sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
> +		      year, month, day, hour, minute);
> +	buf += sz; len -= sz;
> +	if (sec || ns) {
> +		sz = snprintf(buf, len, ":%02d", sec);
> +		buf += sz; len -= sz;
> +		if (ns) {
> +			if ((ns % 1000000) == 0)
> +				sz = snprintf(buf, len, ".%03d", ns / 1000000);
> +			else if ((ns % 1000) == 0)
> +				sz = snprintf(buf, len, ".%06d", ns / 1000);
> +			else
> +				sz = snprintf(buf, len, ".%09d", ns);
> +			buf += sz; len -= sz;
> +		}
> +	}
> +	if (offset == 0) {
> +		strncpy(buf, "Z", len);
> +		buf++;
> +		len--;
> +	}
> +	else {
> +		if (offset < 0)
> +			sign = '-', offset = -offset;
> +		else
> +			sign = '+';
> +
> +		sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
> +		buf += sz; len -= sz;
> +	}
> +	return (buf - src);
> +}
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 403bf1c64..fdaff2d27 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -38,6 +38,11 @@
>   extern "C" {
>   #endif /* defined(__cplusplus) */
>   
> +#ifndef SECS_PER_DAY
> +#define SECS_PER_DAY	86400
> +#define NANOS_PER_SEC	1000000000
> +#endif
> +
>   /**
>    * datetime structure consisting of:
>    */
> @@ -55,6 +60,15 @@ struct datetime_interval_t {
>   	int32_t nsec; ///< nanoseconds delta
>   };
>   
> +/**
> + * Convert datetime to string using default format
> + * @param date source datetime value
> + * @param buf output character buffer
> + * @param len size ofoutput buffer
> + */
> +int
> +datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
> +
>   #if defined(__cplusplus)
>   } /* extern "C" */
>   #endif /* defined(__cplusplus) */
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index f4d2d7737..9ec06d8d8 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -53,6 +53,12 @@ ffi.cdef [[
>       dt_t    dt_from_struct_tm  (const struct tm *tm);
>       void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
>   
> +    // mp_datetime.c
> +
> +    int
> +    datetime_to_string(const struct datetime_t * date, char *buf, uint32_t len);
> +
> +
>       // <asm-generic/posix_types.h>
>       typedef long            __kernel_long_t;
>       typedef unsigned long   __kernel_ulong_t;
> @@ -549,8 +555,13 @@ local function strftime(fmt, o)
>       return ffi.string(buff)
>   end
>   
> --- strftime may be redirected to datetime:fmt("format")
> -local function datetime_fmt()
> +local function datetime_tostring(o)
> +    assert(ffi.typeof(o) == datetime_t)
> +    local sz = 48
> +    local buff = ffi.new('char[?]', sz)
> +    local len = native.datetime_to_string(o, buff, sz)
> +    assert(len < sz)
> +    return ffi.string(buff)
>   end
>   
>   
> @@ -566,7 +577,8 @@ return setmetatable(
>           parse_date = parse_date,
>           parse_time = parse_time,
>           parse_zone = parse_zone,
> -        fmt = datetime_fmt,
> +
> +        tostring = datetime_tostring,
>   
>           now = local_now,
>       -- strptime = strptime;

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
@ 2021-07-29 18:56   ` Oleg Babin via Tarantool-patches
  2021-07-30 22:18     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:56 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch. One question below.


Also update with arithmetic operations doesn't work.


```

tarantool> box.space.dt:update({v}, {{'+', 2, 3}})
---
- error: 'Argument type in operation ''+'' on field 2 does not match 
field type: expected
     a number'
...

tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}})
---
- error: unsupported Lua type 'cdata'
...

```


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * storage hints implemented for datetime_t values;
> * proper comparison for indices of datetime type.
>
>
> diff --git a/test/engine/datetime.result b/test/engine/datetime.result
> new file mode 100644
> index 000000000..3ff474dee
> --- /dev/null
> +++ b/test/engine/datetime.result
> @@ -0,0 +1,77 @@
> +-- test-run result file version 2
> +env = require('test_run')
> + | ---
> + | ...
> +test_run = env.new()
> + | ---
> + | ...
> +engine = test_run:get_cfg('engine')
> + | ---
> + | ...
> +
> +date = require('datetime')
> + | ---
> + | ...
> +
> +_ = box.schema.space.create('T', {engine = engine})
> + | ---
> + | ...
> +_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
> + | ---
> + | ...
> +
> +box.space.T:insert{date('1970-01-01')}\
> +box.space.T:insert{date('1970-01-02')}\
> +box.space.T:insert{date('1970-01-03')}\
> +box.space.T:insert{date('2000-01-01')}
> + | ---
> + | ...
> +
> +o = box.space.T:select{}
> + | ---
> + | ...
> +assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
> + | ---
> + | - true
> + | ...
> +assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
> + | ---
> + | - true
> + | ...
> +assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
> + | ---
> + | - true
> + | ...
> +assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
> + | ---
> + | - true
> + | ...
> +
> +for i = 1,16 do\
> +    box.space.T:insert{date.now()}\
> +end
> + | ---
> + | ...
> +
> +a = box.space.T:select{}
> + | ---
> + | ...
> +err = nil
> + | ---
> + | ...
> +for i = 1, #a - 1 do\
> +    if tostring(a[i][1]) >= tostring(a[i+1][1]) then\

Why do you compare string representation but not values itself?


> +        err = {a[i][1], a[i+1][1]}\
> +        break\
> +    end\
> +end
> + | ---
> + | ...
> +
> +err
> + | ---
> + | - null
> + | ...
> +box.space.T:drop()
> + | ---
> + | ...
>

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding Timur Safin via Tarantool-patches
@ 2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
  2021-07-30 22:20     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:57 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch. This change should be squashed into the patch 
where encoding was introduced.

On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * correct incorrect encoding of MP_EXT sizes for datetime
>    messagepack values;
> * export necessary symbols for datetime messagepack size calculations
>    so they will be available for Lua consumption.
>
> Part of #5941
> ---
>   src/exports.h           |  2 ++
>   src/lib/core/datetime.c | 15 ++++++++++-----
>   src/lua/msgpackffi.lua  | 10 ++++++++++
>   3 files changed, 22 insertions(+), 5 deletions(-)
>
> diff --git a/src/exports.h b/src/exports.h
> index 586444b65..3a1e8854c 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -437,6 +437,7 @@ EXPORT(mp_encode_decimal)
>   EXPORT(mp_encode_double)
>   EXPORT(mp_encode_float)
>   EXPORT(mp_encode_uuid)
> +EXPORT(mp_encode_datetime)
>   EXPORT(mp_ext_hint)
>   EXPORT(mp_format)
>   EXPORT(mp_fprint)
> @@ -446,6 +447,7 @@ EXPORT(mp_fprint_recursion)
>   EXPORT(mp_parser_hint)
>   EXPORT(mp_sizeof_decimal)
>   EXPORT(mp_sizeof_uuid)
> +EXPORT(mp_sizeof_datetime)
>   EXPORT(mp_snprint)
>   EXPORT(mp_snprint_ext)
>   EXPORT(mp_snprint_ext_default)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index 26ba8a702..76b06eae5 100755
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -83,8 +83,8 @@ mp_decode_Xint(const char **data)
>   	return 0;
>   }
>   
> -uint32_t
> -mp_sizeof_datetime(const struct datetime_t *date)
> +static inline uint32_t
> +mp_sizeof_datetime_raw(const struct datetime_t *date)
>   {
>   	uint32_t sz = mp_sizeof_Xint(date->secs);
>   
> @@ -94,10 +94,15 @@ mp_sizeof_datetime(const struct datetime_t *date)
>   		sz += mp_sizeof_Xint(date->nsec);
>   	if (date->offset)
>   		sz += mp_sizeof_Xint(date->offset);
> -
>   	return sz;
>   }
>   
> +uint32_t
> +mp_sizeof_datetime(const struct datetime_t *date)
> +{
> +	return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
> +}
> +
>   struct datetime_t *
>   datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
>   {
> @@ -112,7 +117,7 @@ datetime_unpack(const char **data, uint32_t len, struct datetime_t *date)
>   		return date;
>   
>   	svp = *data;
> -	date->secs = mp_decode_Xint(data);
> +	date->nsec = mp_decode_Xint(data);
>   	len -= *data - svp;
>   
>   	if (len <= 0)
> @@ -153,7 +158,7 @@ datetime_pack(char *data, const struct datetime_t *date)
>   char *
>   mp_encode_datetime(char *data, const struct datetime_t *date)
>   {
> -	uint32_t len = mp_sizeof_datetime(date);
> +	uint32_t len = mp_sizeof_datetime_raw(date);
>   
>   	data = mp_encode_extl(data, MP_DATETIME, len);
>   
> diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua
> index 271be857a..c47d77acb 100644
> --- a/src/lua/msgpackffi.lua
> +++ b/src/lua/msgpackffi.lua
> @@ -26,6 +26,10 @@ char *
>   mp_encode_uuid(char *data, const struct tt_uuid *uuid);
>   uint32_t
>   mp_sizeof_uuid();
> +uint32_t
> +mp_sizeof_datetime(const struct t_datetime_tz *date);
> +char *
> +mp_encode_datetime(char *data, const struct t_datetime_tz *date);
>   float
>   mp_decode_float(const char **data);
>   double
> @@ -144,6 +148,11 @@ local function encode_uuid(buf, uuid)
>       builtin.mp_encode_uuid(p, uuid)
>   end
>   
> +local function encode_datetime(buf, date)
> +    local p = buf:alloc(builtin.mp_sizeof_datetime(date))
> +    builtin.mp_encode_datetime(p, date)
> +end
> +
>   local function encode_int(buf, num)
>       if num >= 0 then
>           if num <= 0x7f then
> @@ -322,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float)
>   on_encode(ffi.typeof('double'), encode_double)
>   on_encode(ffi.typeof('decimal_t'), encode_decimal)
>   on_encode(ffi.typeof('struct tt_uuid'), encode_uuid)
> +on_encode(ffi.typeof('struct datetime_t'), encode_datetime)
>   
>   --------------------------------------------------------------------------------
>   -- Decoder

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes Timur Safin via Tarantool-patches
@ 2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
  2021-07-30 22:30     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:57 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch. See one comment below.

Also this change should be squashed into the first patch.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * introduced a set of calculated attributes to data object, e.g.:
>    - timestamp, seconds, microseconds, minute, or hours
>
> Part of #5941
> ---
>   src/lua/datetime.lua           | 51 ++++++++++++++++++++++------------
>   test/app-tap/datetime.test.lua | 17 +++++++++++-
>   2 files changed, 50 insertions(+), 18 deletions(-)
>
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 9ec06d8d8..7a208cef9 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -224,6 +224,36 @@ local function interval_serialize(self)
>       return { secs = self.secs, nsec = self.nsec }
>   end
>   
> +local datetime_index = function(self, key)
> +    local attributes = {

This change looks like pessimization because currently we need to 
recreate "attributes" table

for each function call. I suggest to use `attributes` as `__index` but 
not such function.


> +        timestamp = function(self)
> +            return tonumber(self.secs) + self.nsec / 1e9
> +        end,
> +        nanoseconds = function(self)
> +            return tonumber(self.secs * 1e9 + self.nsec)
> +        end,
> +        microseconds = function(self)
> +            return tonumber(self.secs * 1e6 + self.nsec / 1e3)
> +        end,
> +        milliseconds = function(self)
> +            return tonumber(self.secs * 1e3 + self.nsec / 1e6)
> +        end,
> +        seconds = function(self)
> +            return tonumber(self.secs) + self.nsec / 1e9
> +        end,
> +        minutes = function(self)
> +            return (tonumber(self.secs) + self.nsec / 1e9) / 60
> +        end,
> +        hours = function(self)
> +            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
> +        end,
> +        days = function(self)
> +            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
> +        end,
> +    }
> +    return attributes[key] ~= nil and attributes[key](self) or nil
> +end
> +
>   local datetime_mt = {
>       -- __tostring = datetime_tostring,
>       __serialize = datetime_serialize,
> @@ -232,23 +262,7 @@ local datetime_mt = {
>       __le = datetime_le,
>       __sub = datetime_sub,
>       __add = datetime_add,
> -
> -    nanoseconds = function(self)
> -        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
> -    end,
> -    microseconds = function(self)
> -        return tonumber(self.secs*1e6 + self.nsec*1e3)
> -    end,
> -    seconds = function(self)
> -        return tonumber(self.secs + self.nsec*1e3)
> -    end,
> -    minutes = function(self)
> -        return tonumber((self._ticks/(1e6*60))%60)
> -    end,
> -    hours = function(self)
> -        return tonumber(self._ticks/(1e6*60*60))
> -    end,
> -
> +    __index = datetime_index,
>   }
>   
>   local interval_mt = {
> @@ -257,6 +271,9 @@ local interval_mt = {
>       __eq = datetime_eq,
>       __lt = datetime_lt,
>       __le = datetime_le,
> +    __sub = datetime_sub,
> +    __add = datetime_add,
> +    __index = datetime_index,
>   }
>   
>   local function datetime_new_raw(secs, nsec, offset)
> diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
> index 09c968858..407d89556 100755
> --- a/test/app-tap/datetime.test.lua
> +++ b/test/app-tap/datetime.test.lua
> @@ -4,7 +4,7 @@ local tap = require('tap')
>   local test = tap.test("errno")
>   local date = require('datetime')
>   
> -test:plan(5)
> +test:plan(6)
>   
>   test:test("Simple tests for parser", function(test)
>       test:plan(2)
> @@ -188,4 +188,19 @@ test:test("Parse iso date - invalid strings", function(test)
>       end
>   end)
>   
> +test:test("Parse tiny date into seconds and other parts", function(test)
> +    test:plan(9)
> +    local str = '19700101 00:00:30.528'
> +    local tiny = date(str)
> +    test:ok(tiny.secs == 30, ("secs of '%s'"):format(str))
> +    test:ok(tiny.nsec == 528000000, ("nsec of '%s'"):format(str))
> +    test:ok(tiny.nanoseconds == 30528000000, "nanoseconds")
> +    test:ok(tiny.microseconds == 30528000, "microseconds")
> +    test:ok(tiny.milliseconds == 30528, "milliseconds")
> +    test:ok(tiny.seconds == 30.528, "seconds")
> +    test:ok(tiny.timestamp == 30.528, "timestamp")
> +    test:ok(tiny.minutes == 0.5088, "minuts")
> +    test:ok(tiny.hours == 0.00848, "hours")
> +end)
> +
>   os.exit(test:check() and 0 or 1)

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support Timur Safin via Tarantool-patches
@ 2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
  2021-07-30 22:58     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:58 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch.

I wrote several comments below.


However it makes me think that such approach will work quite slow since

all functions is implemented in Lua, all arithmetic is in Lua.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * created few entry points (months(N), years(N), days(N), etc.)
>    for easier datetime arithmetic;
> * additions/subtractions of years/months use `dt_add_years()`
>    and `dt_add_months()` from 3rd party c-dt library;
> * also there are `:add{}` and `:sub{}` methods in datetime
>    object to add or substract more complex intervals;
> * introduced `is_datetime()` and `is_interval()` helpers for checking
>    of validity of passed arguments;
> * human-readable stringization implemented for interval objects.
>
> Note, that additions/subtractions completed for all _reasonable_
> combinations of values of date and interval types;
>
> 	Time + Interval	=> Time
> 	Interval + Time => Time
> 	Time - Time 	=> Interval
> 	Time - Interval => Time
> 	Interval + Interval => Interval
> 	Interval - Interval => Interval
>
> Part of #5941
> ---
>   src/exports.h                  |   3 +
>   src/lua/datetime.lua           | 556 +++++++++++++++++++++++++++------
>   test/app-tap/datetime.test.lua | 163 +++++++++-
>   3 files changed, 631 insertions(+), 91 deletions(-)
>
> diff --git a/src/exports.h b/src/exports.h
> index 3a1e8854c..6e7fe206d 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -535,6 +535,9 @@ EXPORT(uuid_nil)
>   EXPORT(uuid_unpack)
>   EXPORT(datetime_unpack)
>   EXPORT(datetime_pack)
> +EXPORT(dt_add_months)
> +EXPORT(dt_add_years)
> +EXPORT(dt_add_quarters)
>   EXPORT(dt_from_rdn)
>   EXPORT(dt_from_yd)
>   EXPORT(dt_from_ymd)
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 7a208cef9..1466b923f 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -37,6 +37,17 @@ ffi.cdef [[
>       int      dt_rdn          (dt_t dt);
>       dt_dow_t dt_dow          (dt_t dt);
>   
> +    // dt_arithmetic.h
> +    typedef enum {
> +        DT_EXCESS,
> +        DT_LIMIT,
> +        DT_SNAP
> +    } dt_adjust_t;
> +
> +    dt_t    dt_add_years        (dt_t dt, int delta, dt_adjust_t adjust);
> +    dt_t    dt_add_quarters     (dt_t dt, int delta, dt_adjust_t adjust);
> +    dt_t    dt_add_months       (dt_t dt, int delta, dt_adjust_t adjust);
> +
>       // dt_parse_iso.h
>       size_t dt_parse_iso_date          (const char *str, size_t len, dt_t *dt);
>   
> @@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL
>   
>   local datetime_t = ffi.typeof('struct datetime_t')
>   local interval_t = ffi.typeof('struct datetime_interval_t')
> +ffi.cdef [[
> +    struct t_interval_months {
> +        int m;
> +    };
> +
> +    struct t_interval_years {
> +        int y;
> +    };
> +]]
> +local interval_months_t = ffi.typeof('struct t_interval_months')
> +local interval_years_t = ffi.typeof('struct t_interval_years')
> +
> +local function is_interval(o)
> +    return ffi.istype(interval_t, o) or
> +           ffi.istype(interval_months_t, o) or
> +           ffi.istype(interval_years_t, o)
> +end
> +

It will throw for non-cdata values:

tarantool> ffi.istype(interval_t, o)
---
- error: 'bad argument #1 to ''?'' (C type expected, got nil)'
...

tarantool> ffi.istype(interval_t, 123)
---
- error: 'bad argument #1 to ''?'' (C type expected, got nil)'
...


> +local function is_datetime(o)
> +    return ffi.istype(o, datetime_t)
> +end
> +
>   
>   local function interval_new()
>       local interval = ffi.new(interval_t)
>       return interval
>   end
>   
> -local function adjusted_secs(dt)
> -    return dt.secs - dt.offset * 60
> +local function check_number(n, message, lvl)
> +    if lvl == nil then
> +        lvl = 2
> +    end
> +    if type(n) ~= 'number' then
> +        return error(('Usage: %s'):format(message), lvl)
> +    end
>   end
>   
> -local function datetime_sub(lhs, rhs)
> -    local s1 = adjusted_secs(lhs)
> -    local s2 = adjusted_secs(rhs)
> -    local d = interval_new()
> -    d.secs = s2 - s1
> -    d.nsec = rhs.nsec - lhs.nsec
> -    if d.nsec < 0 then
> -        d.secs = d.secs - 1
> -        d.nsec = d.nsec + NANOS_PER_SEC
> +local function check_date(o, message, lvl)
> +    if lvl == nil then
> +        lvl = 2
> +    end
> +    if not is_datetime(o) then
> +        return error(('Usage: %s'):format(message), lvl)
>       end
> -    return d
>   end
>   
> -local function datetime_add(lhs, rhs)
> -    local s1 = adjusted_secs(lhs)
> -    local s2 = adjusted_secs(rhs)
> -    local d = interval_new()
> -    d.secs = s2 + s1
> -    d.nsec = rhs.nsec + lhs.nsec
> -    if d.nsec >= NANOS_PER_SEC then
> -        d.secs = d.secs + 1
> -        d.nsec = d.nsec - NANOS_PER_SEC
> +local function check_date_interval(o, message, lvl)
> +    if lvl == nil then
> +        lvl = 2
> +    end
> +    if not (is_datetime(o) or is_interval(o)) then
> +        return error(('Usage: %s'):format(message), lvl)
>       end
> -    return d
>   end
>   
> -local function datetime_eq(lhs, rhs)
> -    -- we usually don't need to check nullness
> -    -- but older tarantool console will call us checking for equality to nil
> -    if rhs == nil then
> -        return false
> +local function check_interval(o, message, lvl)
> +    if lvl == nil then
> +        lvl = 2
> +    end
> +    if not is_interval(o) then
> +        return error(('Usage: %s'):format(message), lvl)
>       end
> +end
> +
> +local function check_str(o, message, lvl)
> +    if lvl == nil then
> +        lvl = 2
> +    end
> +    if not type(o) == 'string' then
> +        return error(('Usage: %s'):format(message), lvl)
> +    end
> +end
> +
> +local function interval_years_new(y)
> +    check_number(y, "years(number)")
> +    local o = ffi.new(interval_years_t)
> +    o.y = y
> +    return o
> +end
> +
> +local function interval_months_new(m)
> +    check_number(m, "months(number)")
> +    local o = ffi.new(interval_months_t)
> +    o.m = m
> +    return o
> +end
> +
> +local function interval_weeks_new(w)
> +    check_number(w, "weeks(number)")
> +    local o = ffi.new(interval_t)
> +    o.secs = w * SECS_PER_DAY * 7
> +    return o
> +end
> +
> +local function interval_days_new(d)
> +    check_number(d, "days(number)")
> +    local o = ffi.new(interval_t)
> +    o.secs = d * SECS_PER_DAY
> +    return o
> +end
> +
> +local function interval_hours_new(h)
> +    check_number(h, "hours(number)")
> +    local o = ffi.new(interval_t)
> +    o.secs = h * 60 * 60
> +    return o
> +end
> +
> +local function interval_minutes_new(m)
> +    check_number(m, "minutes(number)")
> +    local o = ffi.new(interval_t)
> +    o.secs = m * 60
> +    return o
> +end
> +
> +local function interval_seconds_new(s)
> +    check_number(s, "seconds(number)")
> +    local o = ffi.new(interval_t)
> +    o.nsec = s % 1 * 1e9
> +    o.secs = s - (s % 1)
> +    return o
> +end
> +
> +local function datetime_eq(lhs, rhs)
> +    check_date_interval(lhs, "datetime:__eq(date or interval)")
> +    check_date_interval(rhs, "datetime:__eq(date or interval)")
>       return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
>   end
>   
>   
>   local function datetime_lt(lhs, rhs)
> +    check_date_interval(lhs, "datetime:__lt(date or interval)")
> +    check_date_interval(rhs, "datetime:__lt(date or interval)")
>       return (lhs.secs < rhs.secs) or
>              (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
>   end
>   
>   local function datetime_le(lhs, rhs)
> +    check_date_interval(lhs, "datetime:__le(date or interval)")
> +    check_date_interval(rhs, "datetime:__le(date or interval)")
>       return (lhs.secs <= rhs.secs) or
>              (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
>   end
> @@ -224,19 +323,123 @@ local function interval_serialize(self)
>       return { secs = self.secs, nsec = self.nsec }
>   end
>   
> +local function local_rd(o)
> +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
> +end
> +
> +local function local_dt(o)
> +    return cdt.dt_from_rdn(local_rd(o))
> +end
> +
> +local function _normalize_nsec(secs, nsec)
> +    if nsec < 0 then
> +        secs = secs - 1
> +        nsec = nsec + NANOS_PER_SEC
> +    elseif nsec >= NANOS_PER_SEC then
> +        secs = secs + 1
> +        nsec = nsec - NANOS_PER_SEC
> +    end
> +    return secs, nsec
> +end
> +
> +-- addition or subtraction from date/time of a given interval
> +-- described via table direction should be +1 or -1
> +local function interval_increment(self, o, direction)
> +    assert(direction == -1 or direction == 1)
> +    check_date(self, "interval_increment(date, object, -+1)")
> +    assert(type(o) == 'table')
> +
> +    local ym_updated = false
> +    local dhms_updated = false
> +
> +    local dt = local_dt(self)
> +    local secs, nsec
> +    secs, nsec = self.secs, self.nsec
> +
> +    for key, value in pairs(o) do
> +        local handlers = {

The same as in one previous patch. It's too expensive to recreate table 
and functions for each simple

action and for each iteration loop.

> +            years = function(v)
> +                assert(v > 0 and v < 10000)
> +                dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
> +                ym_updated = true
> +            end,
> +
> +            months = function(v)
> +                assert(v > 0 and v < 13 )
> +                dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
> +                ym_updated = true
> +            end,
> +
> +            weeks = function(v)
> +                assert(v > 0 and v < 32)
> +                secs = secs + direction * 7 * v * SECS_PER_DAY
> +                dhms_updated = true
> +            end,
> +
> +            days = function(v)
> +                assert(v > 0 and v < 32)
> +                secs = secs + direction * v * SECS_PER_DAY
> +                dhms_updated = true
> +            end,
> +
> +            hours = function(v)
> +                assert(v >= 0 and v < 24)
> +                secs = secs + direction * 60 * 60 * v
> +                dhms_updated = true
> +            end,
> +
> +            minutes = function(v)
> +                assert(v >= 0 and v < 60)
> +                secs = secs + direction * 60 * v
> +            end,
> +
> +            seconds = function(v)
> +                assert(v >= 0 and v < 61)
> +                local s, frac
> +                frac = v % 1
> +                if frac > 0 then
> +                    s = v - (v % 1)
> +                else
> +                    s = v
> +                end
> +                secs = secs + direction * s
> +                nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
> +                dhms_updated = true
> +            end,
> +        }
> +        handlers[key](value)
> +    end
> +
> +    secs, nsec = _normalize_nsec(secs, nsec)
> +
> +    -- .days, .hours, .minutes, .seconds
> +    if dhms_updated then
> +        self.secs = secs
> +        self.nsec = nsec
> +    end
> +
> +    -- .years, .months updated
> +    if ym_updated then
> +        self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY +
> +                    secs % SECS_PER_DAY
> +    end
> +
> +    return self
> +end
> +
>   local datetime_index = function(self, key)
>       local attributes = {
>           timestamp = function(self)
>               return tonumber(self.secs) + self.nsec / 1e9
>           end,
>           nanoseconds = function(self)
> -            return tonumber(self.secs * 1e9 + self.nsec)
> +            return self.secs * 1e9 + self.nsec
>           end,
>           microseconds = function(self)
> -            return tonumber(self.secs * 1e6 + self.nsec / 1e3)
> +            return self.secs * 1e6 + self.nsec / 1e3
>           end,
>           milliseconds = function(self)
> -            return tonumber(self.secs * 1e3 + self.nsec / 1e6)
> +            return self.secs * 1e3 + self.nsec / 1e6
>           end,
>           seconds = function(self)
>               return tonumber(self.secs) + self.nsec / 1e9
> @@ -250,32 +453,20 @@ local datetime_index = function(self, key)
>           days = function(self)
>               return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
>           end,
> +        add = function(self)
> +            return function(self, o)
> +                return interval_increment(self, o, 1)
> +            end
> +        end,
> +        sub = function(self)
> +            return function(self, o)
> +                return interval_increment(self, o, -1)
> +            end
> +        end,
>       }
>       return attributes[key] ~= nil and attributes[key](self) or nil
>   end
>   
> -local datetime_mt = {
> -    -- __tostring = datetime_tostring,
> -    __serialize = datetime_serialize,
> -    __eq = datetime_eq,
> -    __lt = datetime_lt,
> -    __le = datetime_le,
> -    __sub = datetime_sub,
> -    __add = datetime_add,
> -    __index = datetime_index,
> -}
> -
> -local interval_mt = {
> -    -- __tostring = interval_tostring,
> -    __serialize = interval_serialize,
> -    __eq = datetime_eq,
> -    __lt = datetime_lt,
> -    __le = datetime_le,
> -    __sub = datetime_sub,
> -    __add = datetime_add,
> -    __index = datetime_index,
> -}
> -
>   local function datetime_new_raw(secs, nsec, offset)
>       local dt_obj = ffi.new(datetime_t)
>       dt_obj.secs = secs
> @@ -284,14 +475,6 @@ local function datetime_new_raw(secs, nsec, offset)
>       return dt_obj
>   end
>   
> -local function local_rd(o)
> -    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
> -end
> -
> -local function local_dt(o)
> -    return cdt.dt_from_rdn(local_rd(o))
> -end
> -
>   local function mk_timestamp(dt, sp, fp, offset)
>       local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0
>       local spV = sp ~= nil and sp or 0
> @@ -367,11 +550,12 @@ local function datetime_new(o)
>               second = function(v)
>                   assert(v >= 0 and v < 61)
>                   frac = v % 1
> -                if frac then
> +                if frac > 0 then
>                       s = v - (v % 1)
>                   else
>                       s = v
>                   end
> +                frac = frac * 1e9 -- convert fraction to nanoseconds
>                   hms = true
>               end,
>   
> @@ -402,6 +586,153 @@ local function datetime_new(o)
>       return mk_timestamp(dt, secs, frac, offset)
>   end
>   
> +local function datetime_tostring(o)
> +    if ffi.typeof(o) == datetime_t then
> +        local sz = 48
> +        local buff = ffi.new('char[?]', sz)
> +        local len = native.datetime_to_string(o, buff, sz)
> +        assert(len < sz)
> +        return ffi.string(buff)
> +    elseif ffi.typeof(o) == interval_years_t then
> +        return ('%+d years'):format(o.y)
> +    elseif ffi.typeof(o) == interval_months_t then
> +        return ('%+d months'):format(o.m)
> +    elseif ffi.typeof(o) == interval_t then
> +        local ts = o.timestamp
> +        local sign = '+'
> +
> +        if ts < 0 then
> +            ts = -ts
> +            sign = '-'
> +        end
> +
> +        if ts < 60 then
> +            return ('%s%s secs'):format(sign, ts)
> +        elseif ts < 60 * 60 then
> +            return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60)
> +        elseif ts < 24 * 60 * 60 then
> +            return ('%+d hours, %d minutes, %s seconds'):format(
> +                    o.hours, o.minutes % 60, ts % 60)
> +        else
> +            return ('%+d days, %d hours, %d minutes, %s seconds'):format(
> +                    o.days, o.hours % 24, o.minutes % 60, ts % 60)
> +        end
> +    end
> +end
> +
> +local function date_first(lhs, rhs)
> +    if is_datetime(lhs) then
> +        return lhs, rhs
> +    else
> +        return rhs, lhs
> +    end
> +end
> +
> +--[[
> +Matrix of subtraction operands eligibility and their result type
> +
> +|                 |  datetime | interval | interval_months | interval_years |
> ++-----------------+-----------+----------+-----------------+----------------+
> +| datetime        |  interval | datetime | datetime        | datetime       |
> +| interval        |           | interval |                 |                |
> +| interval_months |           |          | interval_months |                |
> +| interval_years  |           |          |                 | interval_years |
> +]]
> +local function datetime_sub(lhs, rhs)
> +    check_date_interval(lhs, "datetime:__sub(date or interval)")
> +    local d, s = lhs, rhs
> +    local left_t = ffi.typeof(d)
> +    local right_t = ffi.typeof(s)
> +    local o
> +
> +    if left_t == datetime_t then
> +        -- 1. left is date, right is date or generic interval
> +        if (right_t == datetime_t or right_t == interval_t) then
> +            o = right_t == datetime_t and interval_new() or datetime_new()
> +            o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs,
> +                                            lhs.nsec - rhs.nsec)
> +            return o
> +        -- 2. left is date, right is interval in months
> +        elseif right_t == interval_months_t then
> +            local dt = cdt.dt_add_months(local_dt(lhs), -rhs.m, cdt.DT_LIMIT)
> +            return mk_timestamp(dt, lhs.secs % SECS_PER_DAY,
> +                                lhs.nsec, lhs.offset)
> +
> +        -- 3. left is date, right is interval in years
> +        elseif right_t == interval_years_t then
> +            local dt = cdt.dt_add_years(local_dt(lhs), -rhs.y, cdt.DT_LIMIT)
> +            return mk_timestamp(dt, lhs.secs % SECS_PER_DAY,
> +                                lhs.nsec, lhs.offset)
> +        else
> +            error("datetime:__sub(date or interval) - incompatible type of arguments", 2)
> +        end
> +    -- 4. both left and right are generic intervals
> +    elseif left_t == interval_t and right_t == interval_t then
> +        o = interval_new()
> +        o.secs, o.nsec = _normalize_nsec(lhs.secs - rhs.secs,
> +                                        lhs.nsec - rhs.nsec)
> +        return o
> +    -- 5. both left and right are intervals in months
> +    elseif left_t == interval_months_t and right_t == interval_months_t then
> +        return interval_months_new(lhs.m - rhs.m)
> +    -- 5. both left and right are intervals in years
> +    elseif left_t == interval_years_t and right_t == interval_years_t then
> +        return interval_years_new(lhs.y - rhs.y)
> +    else
> +        error("datetime:__sub(date or interval) - incompatible type of arguments", 2)
> +    end
> +end
> +
> +--[[
> +Matrix of addition operands eligibility and their result type
> +
> +|                 |  datetime | interval | interval_months | interval_years |
> ++-----------------+-----------+----------+-----------------+----------------+
> +| datetime        |  datetime | datetime | datetime        | datetime       |
> +| interval        |  datetime | interval |                 |                |
> +| interval_months |  datetime |          | interval_months |                |
> +| interval_years  |  datetime |          |                 | interval_years |
> +]]
> +local function datetime_add(lhs, rhs)
> +    local d, s = date_first(lhs, rhs)
> +
> +    check_date_interval(d, "datetime:__add(interval)")
> +    check_interval(s, "datetime:__add(interval)")

tarantool> return require('datetime').now() + 1
---
- error: '[string "return require(''datetime'').now() + 1"]:1: Usage: 
datetime:__add(interval)'
...


Looks a bit confusing. User doesn't know about metamethods.


> +    local left_t = ffi.typeof(d)
> +    local right_t = ffi.typeof(s)
> +    local o
> +

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua Timur Safin via Tarantool-patches
@ 2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
  2021-07-30 23:11     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-29 18:58 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your patch.

The comment is primarly the same as in previous patch.

Why do you use such slow approach and creates redundant tables and 
function (temporary metatable in fact)

to use it only once? Please fix it.


Unixtime and timestamp is great but they loss precision. I think it 
should be possible

go get timestamp with nanoseconds precision since datetime has 
nanoseconds precision.


Also still it's hard how to convert timestamp to datetime value back.


On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> * implemented proper range checks for date attributes values;
>
> * created `.unixtime` attribute, which is alias to `.secs`,
>    with corresponding setter/getter;
>
> * similarly to `unixtime`, created virtual `timestamp` attribute
>    setter. Which is a convenient way to simultaneously assign
>    unixtime (seconds since epoch) and nanoseconds
>
> Part of #5941
> ---
>   src/lua/datetime.lua | 90 ++++++++++++++++++++++++++++++--------------
>   1 file changed, 61 insertions(+), 29 deletions(-)
>
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 1466b923f..cc2ae119b 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -342,12 +342,33 @@ local function _normalize_nsec(secs, nsec)
>       return secs, nsec
>   end
>   
> +local function seconds_fraction(v)
> +    local seconds, fraction
> +    fraction = v % 1
> +    if fraction > 0 then
> +        seconds = v - (v % 1)
> +    else
> +        seconds = v
> +    end
> +    return seconds, fraction
> +end
> +
> +local function check_range(v, range, txt)
> +    assert(#range == 2)
> +    if not (v >= range[1] and v <= range[2]) then
> +        error(('value %d of %s is out of allowed range [%d, %d]'):
> +              format(v, txt, range[1], range[2]))
> +    end
> +end
> +
>   -- addition or subtraction from date/time of a given interval
>   -- described via table direction should be +1 or -1
>   local function interval_increment(self, o, direction)
>       assert(direction == -1 or direction == 1)
> -    check_date(self, "interval_increment(date, object, -+1)")
> -    assert(type(o) == 'table')
> +    check_date(self, "interval_increment(date, object, direction)")
> +    if type(o) ~= 'table' then
> +        error('interval_increment(date, object, direction) - object expected', 2)
> +    end
>   
>       local ym_updated = false
>       local dhms_updated = false
> @@ -359,49 +380,43 @@ local function interval_increment(self, o, direction)
>       for key, value in pairs(o) do
>           local handlers = {
>               years = function(v)
> -                assert(v > 0 and v < 10000)
> +                check_range(v, {0, 9999}, key)
>                   dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
>                   ym_updated = true
>               end,
>   
>               months = function(v)
> -                assert(v > 0 and v < 13 )
> +                check_range(v, {0, 12}, key)
>                   dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
>                   ym_updated = true
>               end,
>   
>               weeks = function(v)
> -                assert(v > 0 and v < 32)
> +                check_range(v, {0, 52}, key)
>                   secs = secs + direction * 7 * v * SECS_PER_DAY
>                   dhms_updated = true
>               end,
>   
>               days = function(v)
> -                assert(v > 0 and v < 32)
> +                check_range(v, {0, 31}, key)
>                   secs = secs + direction * v * SECS_PER_DAY
>                   dhms_updated = true
>               end,
>   
>               hours = function(v)
> -                assert(v >= 0 and v < 24)
> +                check_range(v, {0, 23}, key)
>                   secs = secs + direction * 60 * 60 * v
>                   dhms_updated = true
>               end,
>   
>               minutes = function(v)
> -                assert(v >= 0 and v < 60)
> +                check_range(v, {0, 59}, key)
>                   secs = secs + direction * 60 * v
>               end,
>   
>               seconds = function(v)
> -                assert(v >= 0 and v < 61)
> -                local s, frac
> -                frac = v % 1
> -                if frac > 0 then
> -                    s = v - (v % 1)
> -                else
> -                    s = v
> -                end
> +                check_range(v, {0, 60}, key)
> +                local s, frac = seconds_fraction(v)
>                   secs = secs + direction * s
>                   nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
>                   dhms_updated = true
> @@ -429,6 +444,9 @@ end
>   
>   local datetime_index = function(self, key)
>       local attributes = {
> +        unixtime = function(self)
> +            return self.secs
> +        end,
>           timestamp = function(self)
>               return tonumber(self.secs) + self.nsec / 1e9
>           end,
> @@ -467,6 +485,24 @@ local datetime_index = function(self, key)
>       return attributes[key] ~= nil and attributes[key](self) or nil
>   end
>   
> +local function datetime_newindex(self, key, value)
> +    local attributes = {
> +        unixtime = function(self, value)
> +            self.secs = value
> +            self.nsec, self.offset = 0, 0
> +        end,
> +        timestamp = function(self, value)
> +            local secs, frac = seconds_fraction(value)
> +            self.secs = secs
> +            self.nsec = frac * 1e9
> +            self.offset = 0
> +        end,
> +    }
> +    if attributes[key] ~= nil then
> +        attributes[key](self, value)
> +    end
> +end
> +
>   local function datetime_new_raw(secs, nsec, offset)
>       local dt_obj = ffi.new(datetime_t)
>       dt_obj.secs = secs
> @@ -518,50 +554,45 @@ local function datetime_new(o)
>               end,
>   
>               year = function(v)
> -                assert(v > 0 and v < 10000)
> +                check_range(v, {1, 9999}, key)
>                   y = v
>                   ymd = true
>               end,
>   
>               month = function(v)
> -                assert(v > 0 and v < 13 )
> +                check_range(v, {1, 12}, key)
>                   M = v
>                   ymd = true
>               end,
>   
>               day = function(v)
> -                assert(v > 0 and v < 32)
> +                check_range(v, {1, 31}, key)
>                   d = v
>                   ymd = true
>               end,
>   
>               hour = function(v)
> -                assert(v >= 0 and v < 24)
> +                check_range(v, {0, 23}, key)
>                   h = v
>                   hms = true
>               end,
>   
>               minute = function(v)
> -                assert(v >= 0 and v < 60)
> +                check_range(v, {0, 59}, key)
>                   m = v
>                   hms = true
>               end,
>   
>               second = function(v)
> -                assert(v >= 0 and v < 61)
> -                frac = v % 1
> -                if frac > 0 then
> -                    s = v - (v % 1)
> -                else
> -                    s = v
> -                end
> +                check_range(v, {0, 60}, key)
> +                s, frac = seconds_fraction(v)
>                   frac = frac * 1e9 -- convert fraction to nanoseconds
>                   hms = true
>               end,
>   
>               -- tz offset in minutes
>               tz = function(v)
> -                assert(v >= 0 and v <= 720)
> +                check_range(v, {0, 720}, key)
>                   offset = v
>               end
>           }
> @@ -918,6 +949,7 @@ local datetime_mt = {
>       __sub = datetime_sub,
>       __add = datetime_add,
>       __index = datetime_index,
> +    __newindex = datetime_newindex,
>       add = function(self, o)
>           self = interval_increment(self, o, 1)
>           return self

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime Timur Safin via Tarantool-patches
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
@ 2021-07-29 23:36   ` Vladislav Shpilevoy via Tarantool-patches
  2021-07-30 15:39     ` Timur Safin via Tarantool-patches
  1 sibling, 1 reply; 46+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-29 23:36 UTC (permalink / raw)
  To: Timur Safin; +Cc: tarantool-patches

Thanks for the patch!

This is a bad question to ask now, but why did you choose a dead library
from 2015 with just 25 stars, without any tests, and without any README as
a basic block for our datetime types? Is this the correct link I am
using - https://github.com/chansen/c-dt?

See 31 comment below.

On 28.07.2021 12:34, Timur Safin wrote:
> * created a new Tarantool built-in module `datetime`;
> * register cdef types for this module;
> * export some `dt_*` functions from `c-dt` library;
> * lua implementationis of `asctime` and `strftime`;

1. implementationis -> implementations

> * datetime parsing unit tests, with and withput timezones;

2. withput -> without

> diff --git a/src/exports.h b/src/exports.h
> index 5bb3e6a2b..db40c03a4 100644
> --- a/src/exports.h
> +++ b/src/exports.h
> @@ -531,3 +531,24 @@ EXPORT(uri_format)
>  EXPORT(uri_parse)
>  EXPORT(uuid_nil)
>  EXPORT(uuid_unpack)
> +EXPORT(dt_from_rdn)
> +EXPORT(dt_from_yd)
> +EXPORT(dt_from_ymd)
> +EXPORT(dt_from_yqd)
> +EXPORT(dt_from_ywd)
> +EXPORT(dt_to_yd)
> +EXPORT(dt_to_ymd)
> +EXPORT(dt_to_yqd)
> +EXPORT(dt_to_ywd)
> +EXPORT(dt_rdn)
> +EXPORT(dt_dow)
> +EXPORT(dt_parse_iso_date)
> +EXPORT(dt_parse_iso_time)
> +EXPORT(dt_parse_iso_time_basic)
> +EXPORT(dt_parse_iso_time_extended)
> +EXPORT(dt_parse_iso_zone)
> +EXPORT(dt_parse_iso_zone_basic)
> +EXPORT(dt_parse_iso_zone_extended)
> +EXPORT(dt_parse_iso_zone_lenient)
> +EXPORT(dt_from_struct_tm)
> +EXPORT(dt_to_struct_tm)

3. Please, read the comment in the beginning of the file. The
first sentence would be:

	Keep the symbols sorted by name for search and addition
	simplicity, to avoid duplicates.

> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> new file mode 100644
> index 000000000..403bf1c64
> --- /dev/null
> +++ b/src/lib/core/datetime.h
> @@ -0,0 +1,61 @@

<...>

> +#include <c-dt/dt_core.h>

4. Please, use "" instead of <> for non-system headers.

> +#include <stdint.h>
> +#include <stdbool.h>
> +
> +#if defined(__cplusplus)
> +extern "C" {
> +#endif /* defined(__cplusplus) */
> +
> +/**
> + * datetime structure consisting of:

5. Well, this comment isn't really helpful. Also we do not
use _t suffix ever except for typedefs. Please, drop both
these things.

The same for datetime_interval_t.

> + */
> +struct datetime_t {
> +	int64_t secs;	///< seconds since epoch
> +	int32_t nsec;	///< nanoseconds if any
> +	int32_t offset; ///< offset in minutes from GMT

6. We never use // nor /// nor ///< nor we write comments
on the same line as the code (except for old legacy code we
inherited from sqlite and some super old tarantool code.

The same for datetime_interval_t.

> +};
> +
> +/**
> + * Date/time delta structure
> + */
> +struct datetime_interval_t {
> +	int64_t secs; ///< relative seconds delta
> +	int32_t nsec; ///< nanoseconds delta
> +};
> +

7. Why do you need this file? It is not included anywhere.
And you don't need to define the structs in C if you are
using them in Lua only. You can just define them in Lua
using ffi.cdef like it is done in some other places.

> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> new file mode 100644
> index 000000000..0996ca5a2
> --- /dev/null
> +++ b/src/lua/datetime.lua
> @@ -0,0 +1,581 @@
> +local ffi = require('ffi')
> +local cdt = ffi.C
> +
> +ffi.cdef [[
> +
> +    typedef int dt_t;
> +
> +    // dt_core.h
> +    typedef enum {
> +        DT_MON       = 1,
> +        DT_MONDAY    = 1,
> +        DT_TUE       = 2,
> +        DT_TUESDAY   = 2,
> +        DT_WED       = 3,
> +        DT_WEDNESDAY = 3,
> +        DT_THU       = 4,
> +        DT_THURSDAY  = 4,
> +        DT_FRI       = 5,
> +        DT_FRIDAY    = 5,
> +        DT_SAT       = 6,
> +        DT_SATURDAY  = 6,
> +        DT_SUN       = 7,
> +        DT_SUNDAY    = 7,
> +    } dt_dow_t;
> +
> +    dt_t     dt_from_rdn     (int n);
> +    dt_t     dt_from_yd      (int y, int d);
> +    dt_t     dt_from_ymd     (int y, int m, int d);
> +    dt_t     dt_from_yqd     (int y, int q, int d);
> +    dt_t     dt_from_ywd     (int y, int w, int d);
> +
> +    void     dt_to_yd        (dt_t dt, int *y, int *d);
> +    void     dt_to_ymd       (dt_t dt, int *y, int *m, int *d);
> +    void     dt_to_yqd       (dt_t dt, int *y, int *q, int *d);
> +    void     dt_to_ywd       (dt_t dt, int *y, int *w, int *d);
> +
> +    int      dt_rdn          (dt_t dt);
> +    dt_dow_t dt_dow          (dt_t dt);
> +
> +    // dt_parse_iso.h
> +    size_t dt_parse_iso_date          (const char *str, size_t len, dt_t *dt);
> +
> +    size_t dt_parse_iso_time          (const char *str, size_t len, int *sod, int *nsec);
> +    size_t dt_parse_iso_time_basic    (const char *str, size_t len, int *sod, int *nsec);
> +    size_t dt_parse_iso_time_extended (const char *str, size_t len, int *sod, int *nsec);
> +
> +    size_t dt_parse_iso_zone          (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_basic    (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_extended (const char *str, size_t len, int *offset);
> +    size_t dt_parse_iso_zone_lenient  (const char *str, size_t len, int *offset);
> +
> +    // dt_tm.h
> +    dt_t    dt_from_struct_tm  (const struct tm *tm);
> +    void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
> +
> +    // <asm-generic/posix_types.h>
> +    typedef long            __kernel_long_t;
> +    typedef unsigned long   __kernel_ulong_t;
> +    // /usr/include/x86_64-linux-gnu/bits/types/time_t.h
> +    typedef long            time_t;
> +
> +
> +    // <time.h>
> +    typedef __kernel_long_t	__kernel_time_t;
> +    typedef __kernel_long_t	__kernel_suseconds_t;
> +
> +    struct timespec {
> +        __kernel_time_t	        tv_sec;     /* seconds */
> +        long                    tv_nsec;    /* nanoseconds */
> +    };
> +
> +    struct timeval {
> +        __kernel_time_t	        tv_sec;	    /* seconds */
> +        __kernel_suseconds_t    tv_usec;    /* microseconds */
> +    };
> +
> +    struct timezone {
> +        int	tz_minuteswest;     /* minutes west of Greenwich */
> +        int	tz_dsttime;	        /* type of dst correction */
> +    };
> +
> +    // /usr/include/x86_64-linux-gnu/sys/time.h
> +    typedef struct timezone * __timezone_ptr_t;
> +
> +    /* Get the current time of day and timezone information,
> +       putting it into *TV and *TZ.  If TZ is NULL, *TZ is not filled.
> +       Returns 0 on success, -1 on errors.
> +
> +       NOTE: This form of timezone information is obsolete.
> +       Use the functions and variables declared in <time.h> instead.  */
> +    int gettimeofday (struct timeval *__tv, struct timezone * __tz);
> +
> +    // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h
> +    /* ISO C `broken-down time' structure.  */
> +    struct tm
> +    {
> +        int tm_sec;	        /* Seconds.	[0-60] (1 leap second) */
> +        int tm_min;	        /* Minutes.	[0-59] */
> +        int tm_hour;        /* Hours.	[0-23] */
> +        int tm_mday;        /* Day.		[1-31] */
> +        int tm_mon;	        /* Month.	[0-11] */
> +        int tm_year;        /* Year	- 1900.  */
> +        int tm_wday;        /* Day of week.	[0-6] */
> +        int tm_yday;        /* Days in year.[0-365]	*/
> +        int tm_isdst;       /* DST.		[-1/0/1]*/
> +
> +        long int tm_gmtoff; /* Seconds east of UTC.  */
> +        const char *tm_zone;/* Timezone abbreviation.  */
> +    };
> +
> +    // <time.h>
> +    /* Return the current time and put it in *TIMER if TIMER is not NULL.  */
> +    time_t time (time_t *__timer);
> +
> +    /* Format TP into S according to FORMAT.
> +    Write no more than MAXSIZE characters and return the number
> +    of characters written, or 0 if it would exceed MAXSIZE.  */
> +    size_t strftime (char * __s, size_t __maxsize, const char * __format,
> +                     const struct tm * __tp);
> +
> +    /* Parse S according to FORMAT and store binary time information in TP.
> +    The return value is a pointer to the first unparsed character in S.  */
> +    char *strptime (const char * __s, const char * __fmt, struct tm *__tp);
> +
> +    /* Return the `struct tm' representation of *TIMER in UTC,
> +    using *TP to store the result.  */
> +    struct tm *gmtime_r (const time_t * __timer, struct tm * __tp);
> +
> +    /* Return the `struct tm' representation of *TIMER in local time,
> +    using *TP to store the result.  */
> +    struct tm *localtime_r (const time_t * __timer, struct tm * __tp);
> +
> +    /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n"
> +    that is the representation of TP in this format.  */
> +    char *asctime (const struct tm *__tp);
> +
> +    /* Equivalent to `asctime (localtime (timer))'.  */
> +    char *ctime (const time_t *__timer);
> +

8. The amount of platform-specific stuff you got here really bothers me.
For instance, struct tm from /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h.

Is it said anywhere that the struct has only these members and only in this
order? The standard says "includes at least the following members". Not
only them and no specific order.

I would probably go with Lua C implementation if I would start this
module. Too strong dependecy on C. Also I wouldn't be so sure FFI is faster than
Lua C: https://github.com/tarantool/tarantool/issues/5896.

> +]]
> +
> +local native = ffi.C

9. You already have cdt in the beginning of the file.

> +
> +local SECS_PER_DAY     = 86400
> +local NANOS_PER_SEC    = 1000000000LL
> +
> +-- c-dt/dt_config.h
> +
> +-- Unix, January 1, 1970, Thursday
> +local DT_EPOCH_1970_OFFSET = 719163LL

10. I remember there was a good question somewhere - how are these
datetimes supposed to store dates before 1970? For example, a
database of historical events or documents.

Can this library store dates before Christ?

> +
> +
> +local datetime_t = ffi.typeof('struct datetime_t')
> +local interval_t = ffi.typeof('struct datetime_interval_t')
> +
> +local function interval_new()
> +    local interval = ffi.new(interval_t)
> +    return interval
> +end
> +
> +local function adjusted_secs(dt)
> +    return dt.secs - dt.offset * 60
> +end
> +
> +local function datetime_sub(lhs, rhs)
> +    local s1 = adjusted_secs(lhs)
> +    local s2 = adjusted_secs(rhs)

11. Did you think about storing the time in a unified
format with offset already applied and add the offset only
when you print the time or convert to something? It seems
these `dt.offset * 60` might be expensive on each arith
operation done again and again.

> +    local d = interval_new()
> +    d.secs = s2 - s1
> +    d.nsec = rhs.nsec - lhs.nsec
> +    if d.nsec < 0 then

12. AFAIK, in Lua index operator '.' is costly.
You might want to cache its values like this:

	local secs = s2 - s1
	local nsec = rhs.nsec - lhs.nsec
	if nsec < 0 then
		d.secs = secs - 1
		d.nsec = nsec + NANOS_PER_SEC
	else
		d.secs = secs
		d.nsec = nsec
	end

The same in the other places where index operation
is repeated more than once: datetime_lt, datetime_le,
etc.

> +        d.secs = d.secs - 1
> +        d.nsec = d.nsec + NANOS_PER_SEC
> +    end
> +    return d
> +end
> +
> +local function datetime_add(lhs, rhs)
> +    local s1 = adjusted_secs(lhs)
> +    local s2 = adjusted_secs(rhs)
> +    local d = interval_new()
> +    d.secs = s2 + s1
> +    d.nsec = rhs.nsec + lhs.nsec
> +    if d.nsec >= NANOS_PER_SEC then
> +        d.secs = d.secs + 1
> +        d.nsec = d.nsec - NANOS_PER_SEC
> +    end
> +    return d
> +end
> +
> +local function datetime_eq(lhs, rhs)
> +    -- we usually don't need to check nullness
> +    -- but older tarantool console will call us checking for equality to nil
> +    if rhs == nil then
> +        return false
> +    end
> +    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)

13. You do not need () around the comparison operators. The same in all places
below. Also you could make it shorter:

	return rhs ~= nil and lhs.secs == rhs.secs and lhs.nsec == rhs.nsec

14. Why don't you take 'offset' into account?

> +end
> +
> +
> +local function datetime_lt(lhs, rhs)
> +    return (lhs.secs < rhs.secs) or
> +           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
> +end
> +
> +local function datetime_le(lhs, rhs)
> +    return (lhs.secs <= rhs.secs) or
> +           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
> +end
> +
> +local function datetime_serialize(self)
> +    -- Allow YAML, MsgPack and JSON to dump objects with sockets
> +    return { secs = self.secs, nsec = self.nsec, tz = self.offset }

15. Why is 'offset' serialized as 'tz' and not just 'offset'?

> +end
> +
> +local function interval_serialize(self)
> +    -- Allow YAML and JSON to dump objects with sockets
> +    return { secs = self.secs, nsec = self.nsec }
> +end
> +
> +local datetime_mt = {
> +    -- __tostring = datetime_tostring,

16. Why is it commented out? Is it even tested then?

> +    __serialize = datetime_serialize,
> +    __eq = datetime_eq,
> +    __lt = datetime_lt,
> +    __le = datetime_le,
> +    __sub = datetime_sub,
> +    __add = datetime_add,
> +
> +    nanoseconds = function(self)
> +        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
> +    end,

17. Please, separate function definitions with an empty line
between them.

> +    microseconds = function(self)
> +        return tonumber(self.secs*1e6 + self.nsec*1e3)

18. Please, add whitespaces around binary operators.

19. Why do you do tonumber() for all results? You are loosing
precision for big values. The same in all the places where you
use tonumber().

> +    end,
> +    seconds = function(self)
> +        return tonumber(self.secs + self.nsec*1e3)
> +    end,
> +    minutes = function(self)
> +        return tonumber((self._ticks/(1e6*60))%60)
> +    end,
> +    hours = function(self)
> +        return tonumber(self._ticks/(1e6*60*60))

20. I don't see _ticks defined anywhere in our source code in
any of the files. What is it? Is it tested?

> +    end,
> +

21. Extra empty line.

> +}
> +
> +local interval_mt = {
> +    -- __tostring = interval_tostring,

22. Ditto. Why is it commented out and not tested?

> +    __serialize = interval_serialize,
> +    __eq = datetime_eq,
> +    __lt = datetime_lt,
> +    __le = datetime_le,
> +}
> +
> +local function datetime_new_raw(secs, nsec, offset)
> +    local dt_obj = ffi.new(datetime_t)
> +    dt_obj.secs = secs
> +    dt_obj.nsec = nsec
> +    dt_obj.offset = offset
> +    return dt_obj
> +end
> +
> +local function local_rd(o)

23. What is 'rd' and 'dt'?

> +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) + DT_EPOCH_1970_OFFSET
> +end
> +
> +local function local_dt(o)
> +    return cdt.dt_from_rdn(local_rd(o))
> +end
> +
> +local function mk_timestamp(dt, sp, fp, offset)
> +    local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY or 0
> +    local spV = sp ~= nil and sp or 0
> +    local fpV = fp ~= nil and fp or 0
> +    local ofsV = offset ~= nil and offset or 0
> +    return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV)
> +end
> +
> +-- create @datetime_t given object @o fields
> +local function datetime_new(o)

24. Why such a wierd name 'o'? It looks like 0, super
confusing.

> +    if o == nil then
> +        return datetime_new_raw(0, 0, 0)
> +    end
> +    local secs = 0
> +    local nsec = 0
> +    local offset = 0
> +    local easy_way = false
> +    local y, M, d, ymd
> +    y, M, d, ymd = 0, 0, 0, false
> +
> +    local h, m, s, frac, hms
> +    h, m, s, frac, hms = 0, 0, 0, 0, false

25. Please, try to define the variables on their own lines.
Like you did a few lines above. Otherwise it is practically
unreadable.

> +
> +    local dt = 0
> +
> +    for key, value in pairs(o) do
> +        local handlers = {
> +            secs = function(v)
> +                secs = v
> +                easy_way = true

26. I don't understand what 'easy way' means. Please, elaborate
what you are trying to achieve.

> +            end,

27. The usage of closures here might render all your FFI efforts
useless, killing the performance. Please, try to define all
methods of all objects only once in the root namespace of the
file. Closure usage might be justified only for rarely created
long living heavy objects like netbox.

> +
> +            nsec = function(v)
> +                nsec = v
> +                easy_way = true
> +            end,
> +
> +            offset = function (v)
> +                offset = v
> +                easy_way = true
> +            end,
> +
> +            year = function(v)
> +                assert(v > 0 and v < 10000)
> +                y = v
> +                ymd = true
> +            end,
> +
> +            month = function(v)
> +                assert(v > 0 and v < 12 )
> +                M = v
> +                ymd = true
> +            end,
> +
> +            day = function(v)
> +                assert(v > 0 and v < 32)
> +                d = v
> +                ymd = true
> +            end,
> +
> +            hour = function(v)
> +                assert(v >= 0 and v < 24)
> +                h = v
> +                hms = true
> +            end,
> +
> +            minute = function(v)
> +                assert(v >= 0 and v < 60)
> +                m = v
> +                hms = true
> +            end,
> +
> +            second = function(v)
> +                assert(v >= 0 and v < 61)
> +                frac = v % 1
> +                if frac then
> +                    s = v - (v % 1)
> +                else
> +                    s = v
> +                end
> +                hms = true
> +            end,
> +
> +            -- tz offset in minutes
> +            tz = function(v)
> +                assert(v >= 0 and v <= 720)
> +                offset = v
> +            end
> +        }
> +        handlers[key](value)
> +    end
> +
> +    -- .sec, .nsec, .offset
> +    if easy_way then
> +        return datetime_new_raw(secs, nsec, offset)
> +    end
> +
> +    -- .year, .month, .day
> +    if ymd then
> +        dt = dt + cdt.dt_from_ymd(y, M, d)
> +    end
> +
> +    -- .hour, .minute, .second
> +    if hms then
> +        secs = h * 3600 + m * 60 + s
> +    end
> +
> +    return mk_timestamp(dt, secs, frac, offset)
> +end
> +
> +
> +-- simple parse functions:
> +-- parse_date/parse_time/parse_zone

28. I wouldn't be so sure they are 'simple' and I can read
their names below anyway. This makes this comment impractical.

> +
> +--[[
> +    Basic      Extended
> +    20121224   2012-12-24   Calendar date   (ISO 8601)
> +    2012359    2012-359     Ordinal date    (ISO 8601)
> +    2012W521   2012-W52-1   Week date       (ISO 8601)
> +    2012Q485   2012-Q4-85   Quarter date
> +]]
> +
> +local function parse_date(str)
> +    local dt = ffi.new('dt_t[1]')
> +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
> +    assert(rc > 0)

29. Er ... And what if I pass a string in a wrong format? Users
should not get assertions.

	tarantool> datetime.parse_date('121r1 r1r 13 13 13')
	---
	- error: 'builtin/datetime.lua:397: assertion failed!'
	...

This shall not ever happen. Would you mind implementing proper
error handling. The same in all the places where the errors are
ignored.

> +    return mk_timestamp(dt[0])
> +end
> +
> +--[[
> +    Basic               Extended
> +    T12                 N/A
> +    T1230               T12:30
> +    T123045             T12:30:45
> +    T123045.123456789   T12:30:45.123456789
> +    T123045,123456789   T12:30:45,123456789
> +
> +    The time designator [T] may be omitted.
> +]]
> +local function parse_time(str)
> +    local sp = ffi.new('int[1]')
> +    local fp = ffi.new('int[1]')
> +    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
> +    assert(rc > 0)
> +    return mk_timestamp(nil, sp[0], fp[0])
> +end
> +
> +--[[
> +    Basic    Extended
> +    Z        N/A
> +    ±hh      N/A
> +    ±hhmm    ±hh:mm

30. I would recommend to avoid non-ASCII symbls in the code. I
remember I used unicode table borders to build nice schemas
but people were complaining it ruins their terminal's output.

> +]]
> +local function parse_zone(str)
> +    local offset = ffi.new('int[1]')
> +    local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
> +    return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil, tonumber(len)

31. Please, keep 80 symbols border.

I am going to stop here, will return later.

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build
  2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
@ 2021-07-29 23:40   ` Vladislav Shpilevoy via Tarantool-patches
  2021-07-31  9:22     ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-07-29 23:40 UTC (permalink / raw)
  To: Timur Safin; +Cc: tarantool-patches

Hi! Thanks for the patch!

On 28.07.2021 12:34, Timur Safin wrote:
> * Integrated chansen/c-dt parser as 3rd party module to the
>   Tarantool cmake build process.
> * Points to tsafin/c-dt instead iof original chansen/c-dt to

iof -> of. But is it even true? Below I see

	url = https://github.com/tarantool/c-dt.git

not tsafin/c-dt.

After build I get in git status this:

	Changes not staged for commit:
	  (use "git add <file>..." to update what will be committed)
	  (use "git restore <file>..." to discard changes in working directory)
	  (commit or discard the untracked or modified content in submodules)
		modified:   third_party/c-dt (modified content, untracked content)

It says:

	modified:   Makefile

and some untracked files:

	CMakeFiles/
	cmake_install.cmake
	libcdt.a

Why is even Makefile added to the repository if we are
using cmake anyway?

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-29 23:36   ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-30 15:39     ` Timur Safin via Tarantool-patches
  2021-08-01 17:01       ` Vladislav Shpilevoy via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 15:39 UTC (permalink / raw)
  To: 'Vladislav Shpilevoy'; +Cc: tarantool-patches

: From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
: Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime
: 
: Thanks for the patch!
: 
: This is a bad question to ask now, but why did you choose a dead library
: from 2015 with just 25 stars, without any tests, and without any README as
: a basic block for our datetime types? Is this the correct link I am
: using - https://github.com/chansen/c-dt?

Good you asked :) That was data-driven decision and you could see details here
https://github.com/tsafin/bench-timestamps

In short c-dt is as almost as feature-rich as ICU or CCTZ, but much, much 
faster (and we know how to make it even faster if we have to).
Originally I've learned about c-dt from perl' perl5-time-moment mentioned by Mons
(which module was the fastest date/time implementation in Perl ecosystem).

: 
: See 31 comment below.
: 
: On 28.07.2021 12:34, Timur Safin wrote:
: > * created a new Tarantool built-in module `datetime`;
: > * register cdef types for this module;
: > * export some `dt_*` functions from `c-dt` library;
: > * lua implementationis of `asctime` and `strftime`;
: 
: 1. implementationis -> implementations

: 
: > * datetime parsing unit tests, with and withput timezones;
: 
: 2. withput -> without

Thanks!

: 
: > diff --git a/src/exports.h b/src/exports.h
: > index 5bb3e6a2b..db40c03a4 100644
: > --- a/src/exports.h
: > +++ b/src/exports.h
: > @@ -531,3 +531,24 @@ EXPORT(uri_format)
: >  EXPORT(uri_parse)
: >  EXPORT(uuid_nil)
: >  EXPORT(uuid_unpack)
: > +EXPORT(dt_from_rdn)
: > +EXPORT(dt_from_yd)
: > +EXPORT(dt_from_ymd)
: > +EXPORT(dt_from_yqd)
: > +EXPORT(dt_from_ywd)
: > +EXPORT(dt_to_yd)
: > +EXPORT(dt_to_ymd)
: > +EXPORT(dt_to_yqd)
: > +EXPORT(dt_to_ywd)
: > +EXPORT(dt_rdn)
: > +EXPORT(dt_dow)
: > +EXPORT(dt_parse_iso_date)
: > +EXPORT(dt_parse_iso_time)
: > +EXPORT(dt_parse_iso_time_basic)
: > +EXPORT(dt_parse_iso_time_extended)
: > +EXPORT(dt_parse_iso_zone)
: > +EXPORT(dt_parse_iso_zone_basic)
: > +EXPORT(dt_parse_iso_zone_extended)
: > +EXPORT(dt_parse_iso_zone_lenient)
: > +EXPORT(dt_from_struct_tm)
: > +EXPORT(dt_to_struct_tm)
: 
: 3. Please, read the comment in the beginning of the file. The
: first sentence would be:
: 
: 	Keep the symbols sorted by name for search and addition
: 	simplicity, to avoid duplicates.

Yes, I knew it, but forgot. 
BTW, when I tried to sort the whole block I have discovered that
there are multiple of symbols, which are already in incorrect order
but I decided to not touch irrelevant chunks, and only sorted d*
See in the next iteration...
: 
: > diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
: > new file mode 100644
: > index 000000000..403bf1c64
: > --- /dev/null
: > +++ b/src/lib/core/datetime.h
: > @@ -0,0 +1,61 @@
: 
: <...>
: 
: > +#include <c-dt/dt_core.h>
: 
: 4. Please, use "" instead of <> for non-system headers.

Ok.

: 
: > +#include <stdint.h>
: > +#include <stdbool.h>
: > +
: > +#if defined(__cplusplus)
: > +extern "C" {
: > +#endif /* defined(__cplusplus) */
: > +
: > +/**
: > + * datetime structure consisting of:
: 
: 5. Well, this comment isn't really helpful. Also we do not
: use _t suffix ever except for typedefs. Please, drop both
: these things.
: 
: The same for datetime_interval_t.

Will rename to `struct datetime`, and `struct datetime_interval` 
Accordingly.

: 
: > + */
: > +struct datetime_t {
: > +	int64_t secs;	///< seconds since epoch
: > +	int32_t nsec;	///< nanoseconds if any
: > +	int32_t offset; ///< offset in minutes from GMT
: 
: 6. We never use // nor /// nor ///< nor we write comments
: on the same line as the code (except for old legacy code we
: inherited from sqlite and some super old tarantool code.

BTW, why? Single line comment // is the legal C construction 
since C99 (and always have been acceptable by Gnu C and major 
other compilers). 

Also did you know that ///< is doxygen construction for 
documentation of a struct member? (Asking, just to make sure)

: 
: The same for datetime_interval_t.
: 
: > +};
: > +
: > +/**
: > + * Date/time delta structure
: > + */
: > +struct datetime_interval_t {
: > +	int64_t secs; ///< relative seconds delta
: > +	int32_t nsec; ///< nanoseconds delta
: > +};
: > +
: 
: 7. Why do you need this file? It is not included anywhere.
: And you don't need to define the structs in C if you are
: using them in Lua only. You can just define them in Lua
: using ffi.cdef like it is done in some other places.

I'll need to use it in C very soon, once I get to SQL parser
INTERVAL 'xx' support.

: 
: > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
: > new file mode 100644
: > index 000000000..0996ca5a2
: > --- /dev/null
: > +++ b/src/lua/datetime.lua
: > @@ -0,0 +1,581 @@
: > +local ffi = require('ffi')
: > +local cdt = ffi.C
: > +
: > +ffi.cdef [[
: > +
: > +    typedef int dt_t;
: > +
: > +    // dt_core.h
: > +    typedef enum {
: > +        DT_MON       = 1,
: > +        DT_MONDAY    = 1,
: > +        DT_TUE       = 2,
: > +        DT_TUESDAY   = 2,
: > +        DT_WED       = 3,
: > +        DT_WEDNESDAY = 3,
: > +        DT_THU       = 4,
: > +        DT_THURSDAY  = 4,
: > +        DT_FRI       = 5,
: > +        DT_FRIDAY    = 5,
: > +        DT_SAT       = 6,
: > +        DT_SATURDAY  = 6,
: > +        DT_SUN       = 7,
: > +        DT_SUNDAY    = 7,
: > +    } dt_dow_t;
: > +
: > +    dt_t     dt_from_rdn     (int n);
: > +    dt_t     dt_from_yd      (int y, int d);
: > +    dt_t     dt_from_ymd     (int y, int m, int d);
: > +    dt_t     dt_from_yqd     (int y, int q, int d);
: > +    dt_t     dt_from_ywd     (int y, int w, int d);
: > +
: > +    void     dt_to_yd        (dt_t dt, int *y, int *d);
: > +    void     dt_to_ymd       (dt_t dt, int *y, int *m, int *d);
: > +    void     dt_to_yqd       (dt_t dt, int *y, int *q, int *d);
: > +    void     dt_to_ywd       (dt_t dt, int *y, int *w, int *d);
: > +
: > +    int      dt_rdn          (dt_t dt);
: > +    dt_dow_t dt_dow          (dt_t dt);
: > +
: > +    // dt_parse_iso.h
: > +    size_t dt_parse_iso_date          (const char *str, size_t len, dt_t
: *dt);
: > +
: > +    size_t dt_parse_iso_time          (const char *str, size_t len, int
: *sod, int *nsec);
: > +    size_t dt_parse_iso_time_basic    (const char *str, size_t len, int
: *sod, int *nsec);
: > +    size_t dt_parse_iso_time_extended (const char *str, size_t len, int
: *sod, int *nsec);
: > +
: > +    size_t dt_parse_iso_zone          (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_basic    (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_extended (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_lenient  (const char *str, size_t len, int
: *offset);
: > +
: > +    // dt_tm.h
: > +    dt_t    dt_from_struct_tm  (const struct tm *tm);
: > +    void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
: > +
: > +    // <asm-generic/posix_types.h>
: > +    typedef long            __kernel_long_t;
: > +    typedef unsigned long   __kernel_ulong_t;
: > +    // /usr/include/x86_64-linux-gnu/bits/types/time_t.h
: > +    typedef long            time_t;
: > +
: > +
: > +    // <time.h>
: > +    typedef __kernel_long_t	__kernel_time_t;
: > +    typedef __kernel_long_t	__kernel_suseconds_t;
: > +
: > +    struct timespec {
: > +        __kernel_time_t	        tv_sec;     /* seconds */
: > +        long                    tv_nsec;    /* nanoseconds */
: > +    };
: > +
: > +    struct timeval {
: > +        __kernel_time_t	        tv_sec;	    /* seconds */
: > +        __kernel_suseconds_t    tv_usec;    /* microseconds */
: > +    };
: > +
: > +    struct timezone {
: > +        int	tz_minuteswest;     /* minutes west of Greenwich */
: > +        int	tz_dsttime;	        /* type of dst correction */
: > +    };
: > +
: > +    // /usr/include/x86_64-linux-gnu/sys/time.h
: > +    typedef struct timezone * __timezone_ptr_t;
: > +
: > +    /* Get the current time of day and timezone information,
: > +       putting it into *TV and *TZ.  If TZ is NULL, *TZ is not filled.
: > +       Returns 0 on success, -1 on errors.
: > +
: > +       NOTE: This form of timezone information is obsolete.
: > +       Use the functions and variables declared in <time.h> instead.  */
: > +    int gettimeofday (struct timeval *__tv, struct timezone * __tz);
: > +
: > +    // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h
: > +    /* ISO C `broken-down time' structure.  */
: > +    struct tm
: > +    {
: > +        int tm_sec;	        /* Seconds.	[0-60] (1 leap second) */
: > +        int tm_min;	        /* Minutes.	[0-59] */
: > +        int tm_hour;        /* Hours.	[0-23] */
: > +        int tm_mday;        /* Day.		[1-31] */
: > +        int tm_mon;	        /* Month.	[0-11] */
: > +        int tm_year;        /* Year	- 1900.  */
: > +        int tm_wday;        /* Day of week.	[0-6] */
: > +        int tm_yday;        /* Days in year.[0-365]	*/
: > +        int tm_isdst;       /* DST.		[-1/0/1]*/
: > +
: > +        long int tm_gmtoff; /* Seconds east of UTC.  */
: > +        const char *tm_zone;/* Timezone abbreviation.  */
: > +    };
: > +
: > +    // <time.h>
: > +    /* Return the current time and put it in *TIMER if TIMER is not NULL.
: */
: > +    time_t time (time_t *__timer);
: > +
: > +    /* Format TP into S according to FORMAT.
: > +    Write no more than MAXSIZE characters and return the number
: > +    of characters written, or 0 if it would exceed MAXSIZE.  */
: > +    size_t strftime (char * __s, size_t __maxsize, const char * __format,
: > +                     const struct tm * __tp);
: > +
: > +    /* Parse S according to FORMAT and store binary time information in
: TP.
: > +    The return value is a pointer to the first unparsed character in S.
: */
: > +    char *strptime (const char * __s, const char * __fmt, struct tm
: *__tp);
: > +
: > +    /* Return the `struct tm' representation of *TIMER in UTC,
: > +    using *TP to store the result.  */
: > +    struct tm *gmtime_r (const time_t * __timer, struct tm * __tp);
: > +
: > +    /* Return the `struct tm' representation of *TIMER in local time,
: > +    using *TP to store the result.  */
: > +    struct tm *localtime_r (const time_t * __timer, struct tm * __tp);
: > +
: > +    /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n"
: > +    that is the representation of TP in this format.  */
: > +    char *asctime (const struct tm *__tp);
: > +
: > +    /* Equivalent to `asctime (localtime (timer))'.  */
: > +    char *ctime (const time_t *__timer);
: > +
: 
: 8. The amount of platform-specific stuff you got here really bothers me.
: For instance, struct tm from /usr/include/x86_64-linux-
: gnu/bits/types/struct_tm.h.
: 
: Is it said anywhere that the struct has only these members and only in this
: order? The standard says "includes at least the following members". Not
: only them and no specific order.

Agreed that despite the fact that `struct tm` is defined pretty similarly 
in POSIX/SUSV https://pubs.opengroup.org/onlinepubs/009696699/basedefs/time.h.html
actual layout of (at least last fields tm_gmtoff and tm_zone) might vary
significantly. And we better hide this layout specific code (i.e. allocation
and manipulations with fields) into accompanying C code. 

: 
: I would probably go with Lua C implementation if I would start this
: module. Too strong dependecy on C. Also I wouldn't be so sure FFI is faster
: than
: Lua C: https://github.com/tarantool/tarantool/issues/5896.

As you know - who is faster is complicated question. There are some
subtle bug in LuaJIT with structure allocations which I believe was the
major contributor to slowness reported in Decimal branch 
https://github.com/tarantool/tarantool/issues/692#issuecomment-505043929 
but which has been narrowed down by Sergos and Mike has already committed
patch to upstream in https://github.com/LuaJIT/LuaJIT/commit/ad65934fa08a65bfb0eb9528731a4394842cc173
but, I believe, it's not yet in our LuaJIT fork. 

I'd probably postpone total redesign from ffi to LuaC of this 1st version
of module (because performance is not yet problem, functionality is), but
rather wait and see how things would change once this struct-of-NYI would
be landed to Tarantool code. 
: 
: > +]]
: > +
: > +local native = ffi.C
: 
: 9. You already have cdt in the beginning of the file.

Yes, thanks

: 
: > +
: > +local SECS_PER_DAY     = 86400
: > +local NANOS_PER_SEC    = 1000000000LL
: > +
: > +-- c-dt/dt_config.h
: > +
: > +-- Unix, January 1, 1970, Thursday
: > +local DT_EPOCH_1970_OFFSET = 719163LL
: 
: 10. I remember there was a good question somewhere - how are these
: datetimes supposed to store dates before 1970? For example, a
: database of historical events or documents.
: 
: Can this library store dates before Christ?

Yes, please see relevant discussion with Peter https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1055996
c-dt is properly handling positive and negative dt values, 
where dt is number of dates since Rata Die, January 1, 0001.
For better MessagePack compactness of our typical data we shift beginning
of our 0 to the Unix Epoch (1970-01-01), and our 0 is actually 
dt=719163.
So our epoch-based second 0 would become dt value (719163) via this simple
formula (secs / 86400) + 719163. Where 719163 is an offset of Unix Epoch
(1970-01-01) since Rata Die (0001-01-01) in dates.

We might consider to further shift, though, our relative 0 
to any date in time like 2000-01-01, whatever, if we would decide that
that different base would provide us more compact storage. But I believe
unix-epoch date, from statistical point of view, is more convenient for 
storing and using at runtime.

: 
: > +
: > +
: > +local datetime_t = ffi.typeof('struct datetime_t')
: > +local interval_t = ffi.typeof('struct datetime_interval_t')
: > +
: > +local function interval_new()
: > +    local interval = ffi.new(interval_t)
: > +    return interval
: > +end
: > +
: > +local function adjusted_secs(dt)
: > +    return dt.secs - dt.offset * 60
: > +end
: > +
: > +local function datetime_sub(lhs, rhs)
: > +    local s1 = adjusted_secs(lhs)
: > +    local s2 = adjusted_secs(rhs)
: 
: 11. Did you think about storing the time in a unified
: format with offset already applied and add the offset only
: when you print the time or convert to something? It seems
: these `dt.offset * 60` might be expensive on each arith
: operation done again and again.

Exactly! That was bug in the early patches, which has been fixed 
in later version of this code. Sorry for the confusion created. 
Yes, secs are UC normalized, and offset if nominal (used only
For output).

: 
: > +    local d = interval_new()
: > +    d.secs = s2 - s1
: > +    d.nsec = rhs.nsec - lhs.nsec
: > +    if d.nsec < 0 then
: 
: 12. AFAIK, in Lua index operator '.' is costly.
: You might want to cache its values like this:
: 
: 	local secs = s2 - s1
: 	local nsec = rhs.nsec - lhs.nsec
: 	if nsec < 0 then
: 		d.secs = secs - 1
: 		d.nsec = nsec + NANOS_PER_SEC
: 	else
: 		d.secs = secs
: 		d.nsec = nsec
: 	end
: 
: The same in the other places where index operation
: is repeated more than once: datetime_lt, datetime_le,
: etc.

Thanks, I'll review all relevant places and extract
read references to locals.

: 
: > +        d.secs = d.secs - 1
: > +        d.nsec = d.nsec + NANOS_PER_SEC
: > +    end
: > +    return d
: > +end
: > +
: > +local function datetime_add(lhs, rhs)
: > +    local s1 = adjusted_secs(lhs)
: > +    local s2 = adjusted_secs(rhs)
: > +    local d = interval_new()
: > +    d.secs = s2 + s1
: > +    d.nsec = rhs.nsec + lhs.nsec
: > +    if d.nsec >= NANOS_PER_SEC then
: > +        d.secs = d.secs + 1
: > +        d.nsec = d.nsec - NANOS_PER_SEC
: > +    end
: > +    return d
: > +end
: > +
: > +local function datetime_eq(lhs, rhs)
: > +    -- we usually don't need to check nullness
: > +    -- but older tarantool console will call us checking for equality to
: nil
: > +    if rhs == nil then
: > +        return false
: > +    end
: > +    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
: 
: 13. You do not need () around the comparison operators. The same in all
: places
: below. Also you could make it shorter:
: 
: 	return rhs ~= nil and lhs.secs == rhs.secs and lhs.nsec == rhs.nsec

Nice! Thanks!

: 
: 14. Why don't you take 'offset' into account?

Because it's not needed due to UTC-normalization. Sorry for the confusion 
with adjusted_sescs function. 

: 
: > +end
: > +
: > +
: > +local function datetime_lt(lhs, rhs)
: > +    return (lhs.secs < rhs.secs) or
: > +           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
: > +end
: > +
: > +local function datetime_le(lhs, rhs)
: > +    return (lhs.secs <= rhs.secs) or
: > +           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
: > +end
: > +
: > +local function datetime_serialize(self)
: > +    -- Allow YAML, MsgPack and JSON to dump objects with sockets
: > +    return { secs = self.secs, nsec = self.nsec, tz = self.offset }
: 
: 15. Why is 'offset' serialized as 'tz' and not just 'offset'?

Yes, now you've asked it and it's looking confusing after rename.

: 
: > +end
: > +
: > +local function interval_serialize(self)
: > +    -- Allow YAML and JSON to dump objects with sockets
: > +    return { secs = self.secs, nsec = self.nsec }
: > +end
: > +
: > +local datetime_mt = {
: > +    -- __tostring = datetime_tostring,
: 
: 16. Why is it commented out? Is it even tested then?

It's implemented in the later patch in patchset series.

: 
: > +    __serialize = datetime_serialize,
: > +    __eq = datetime_eq,
: > +    __lt = datetime_lt,
: > +    __le = datetime_le,
: > +    __sub = datetime_sub,
: > +    __add = datetime_add,
: > +
: > +    nanoseconds = function(self)
: > +        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
: > +    end,
: 
: 17. Please, separate function definitions with an empty line
: between them.

Done! (Will be pushed later) 

: 
: > +    microseconds = function(self)
: > +        return tonumber(self.secs*1e6 + self.nsec*1e3)
: 
: 18. Please, add whitespaces around binary operators.

Done! It was already such in the later commits which were part
of in patchset. [Yes, apparently I need to squash few more commits,
as Oleg Babin has asked elsewhere]

: 
: 19. Why do you do tonumber() for all results? You are loosing
: precision for big values. The same in all the places where you
: use tonumber().

To be able to use it in places which expect numbers like %d in
string.format. Though at the end of day I've came up to the version
where I convert to Lua numbers not all (potentially fraction point) 
values, but with exception of nanoseconds / microseconds / milliseconds
(which return integer values). 

: 
: > +    end,
: > +    seconds = function(self)
: > +        return tonumber(self.secs + self.nsec*1e3)
: > +    end,
: > +    minutes = function(self)
: > +        return tonumber((self._ticks/(1e6*60))%60)
: > +    end,
: > +    hours = function(self)
: > +        return tonumber(self._ticks/(1e6*60*60))
: 
: 20. I don't see _ticks defined anywhere in our source code in
: any of the files. What is it? Is it tested?

Yes, that was miscopy from elsewhere. Fixed in newer commits. 
Sorry for the confusion.

: 
: > +    end,
: > +
: 
: 21. Extra empty line.

Yup, but later, well, you know ...

: 
: > +}
: > +
: > +local interval_mt = {
: > +    -- __tostring = interval_tostring,
: 
: 22. Ditto. Why is it commented out and not tested?

Tostring conversion was implemented later in a few 
commits. One for datetime stringization, and another 
one for interval.

: 
: > +    __serialize = interval_serialize,
: > +    __eq = datetime_eq,
: > +    __lt = datetime_lt,
: > +    __le = datetime_le,
: > +}
: > +
: > +local function datetime_new_raw(secs, nsec, offset)
: > +    local dt_obj = ffi.new(datetime_t)
: > +    dt_obj.secs = secs
: > +    dt_obj.nsec = nsec
: > +    dt_obj.offset = offset
: > +    return dt_obj
: > +end
: > +
: > +local function local_rd(o)
: 
: 23. What is 'rd' and 'dt'?

`dt` is type - data type from c-dt (alias to signed integer).
Which shows a number of dates since Rata Die date (0001-01-01).
It's signed 32-bit value, thus could represent huge range before 
0001-01-01 as well. 

: 
: > +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) +
: DT_EPOCH_1970_OFFSET
: > +end
: > +
: > +local function local_dt(o)
: > +    return cdt.dt_from_rdn(local_rd(o))
: > +end
: > +
: > +local function mk_timestamp(dt, sp, fp, offset)
: > +    local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET)
: * SECS_PER_DAY or 0
: > +    local spV = sp ~= nil and sp or 0
: > +    local fpV = fp ~= nil and fp or 0
: > +    local ofsV = offset ~= nil and offset or 0
: > +    return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV)
: > +end
: > +
: > +-- create @datetime_t given object @o fields
: > +local function datetime_new(o)
: 
: 24. Why such a wierd name 'o'? It looks like 0, super
: confusing.

o means object (with attributes) here. Will make it look 
longer, i.e. obj.

: 
: > +    if o == nil then
: > +        return datetime_new_raw(0, 0, 0)
: > +    end
: > +    local secs = 0
: > +    local nsec = 0
: > +    local offset = 0
: > +    local easy_way = false
: > +    local y, M, d, ymd
: > +    y, M, d, ymd = 0, 0, 0, false
: > +
: > +    local h, m, s, frac, hms
: > +    h, m, s, frac, hms = 0, 0, 0, 0, false
: 
: 25. Please, try to define the variables on their own lines.
: Like you did a few lines above. Otherwise it is practically
: unreadable.

Ok

: 
: > +
: > +    local dt = 0
: > +
: > +    for key, value in pairs(o) do
: > +        local handlers = {
: > +            secs = function(v)
: > +                secs = v
: > +                easy_way = true
: 
: 26. I don't understand what 'easy way' means. Please, elaborate
: what you are trying to achieve.

We could create datetime object 2 ways:
- using raw attributes, without any conversion - i.e. secs, nsec, offset;
- or using human-friendly year/month/day/hour/minutes/seconds;

The easy_way is using raw mode - defining .secs, .nsec, .offset, but using
The same object constructor interface. i.e.

	date = require 'datetime'
	T = date{ secs = 0, nsec = 0, offset = 0}

Will create datetime object representing 1970-01-01T00:00Z moment.

: 
: > +            end,
: 
: 27. The usage of closures here might render all your FFI efforts
: useless, killing the performance. Please, try to define all
: methods of all objects only once in the root namespace of the
: file. Closure usage might be justified only for rarely created
: long living heavy objects like netbox.

Agreed that creating closures in each iteration of a loop was
bad idea (as Oleg Babin has already pin-pointed it elsewhere).
And we have to ate least to remove loop-invariant outside of cycle.
But at the moment I do not consider the code creating object 
as very performance critical path, because I do not expect this
path used frequently, more like after parsing of ido-8601 literals,
which uses raw object construction approach.

[I'll bench though the difference between using invariant closures 
and multiple ifs to see the size of a problem]

: 
: > +
: > +            nsec = function(v)
: > +                nsec = v
: > +                easy_way = true
: > +            end,
: > +
: > +            offset = function (v)
: > +                offset = v
: > +                easy_way = true
: > +            end,
: > +
: > +            year = function(v)
: > +                assert(v > 0 and v < 10000)
: > +                y = v
: > +                ymd = true
: > +            end,
: > +
: > +            month = function(v)
: > +                assert(v > 0 and v < 12 )
: > +                M = v
: > +                ymd = true
: > +            end,
: > +
: > +            day = function(v)
: > +                assert(v > 0 and v < 32)
: > +                d = v
: > +                ymd = true
: > +            end,
: > +
: > +            hour = function(v)
: > +                assert(v >= 0 and v < 24)
: > +                h = v
: > +                hms = true
: > +            end,
: > +
: > +            minute = function(v)
: > +                assert(v >= 0 and v < 60)
: > +                m = v
: > +                hms = true
: > +            end,
: > +
: > +            second = function(v)
: > +                assert(v >= 0 and v < 61)
: > +                frac = v % 1
: > +                if frac then
: > +                    s = v - (v % 1)
: > +                else
: > +                    s = v
: > +                end
: > +                hms = true
: > +            end,
: > +
: > +            -- tz offset in minutes
: > +            tz = function(v)
: > +                assert(v >= 0 and v <= 720)
: > +                offset = v
: > +            end
: > +        }
: > +        handlers[key](value)
: > +    end
: > +
: > +    -- .sec, .nsec, .offset
: > +    if easy_way then
: > +        return datetime_new_raw(secs, nsec, offset)
: > +    end
: > +
: > +    -- .year, .month, .day
: > +    if ymd then
: > +        dt = dt + cdt.dt_from_ymd(y, M, d)
: > +    end
: > +
: > +    -- .hour, .minute, .second
: > +    if hms then
: > +        secs = h * 3600 + m * 60 + s
: > +    end
: > +
: > +    return mk_timestamp(dt, secs, frac, offset)
: > +end
: > +
: > +
: > +-- simple parse functions:
: > +-- parse_date/parse_time/parse_zone
: 
: 28. I wouldn't be so sure they are 'simple' and I can read
: their names below anyway. This makes this comment impractical.

:) ok

: 
: > +
: > +--[[
: > +    Basic      Extended
: > +    20121224   2012-12-24   Calendar date   (ISO 8601)
: > +    2012359    2012-359     Ordinal date    (ISO 8601)
: > +    2012W521   2012-W52-1   Week date       (ISO 8601)
: > +    2012Q485   2012-Q4-85   Quarter date
: > +]]
: > +
: > +local function parse_date(str)
: > +    local dt = ffi.new('dt_t[1]')
: > +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
: > +    assert(rc > 0)
: 
: 29. Er ... And what if I pass a string in a wrong format? Users
: should not get assertions.
: 
: 	tarantool> datetime.parse_date('121r1 r1r 13 13 13')
: 	---
: 	- error: 'builtin/datetime.lua:397: assertion failed!'
: 	...
: 
: This shall not ever happen. Would you mind implementing proper
: error handling. The same in all the places where the errors are
: ignored.


Now, I've reworked approach how they handle errors (you could see
It in later commits in the patch-set), and now it returns tuple - 
- parsed value
- and the number of accepted input characters.

	tarantool> date.parse_date('121r1 r1r 13 13 13')
	---
	- null
	- 0
	...

Sometimes, while parsing composed strings using these partial
methods (parse_date / parse_time / parse_time_zone) we need to
know the number of symbols we have successfully accepted. That's
why there is seconds value in returned tuple.

Please give me know if there is better/more idiomatic approach
used for such scenarios?

: 
: > +    return mk_timestamp(dt[0])
: > +end
: > +
: > +--[[
: > +    Basic               Extended
: > +    T12                 N/A
: > +    T1230               T12:30
: > +    T123045             T12:30:45
: > +    T123045.123456789   T12:30:45.123456789
: > +    T123045,123456789   T12:30:45,123456789
: > +
: > +    The time designator [T] may be omitted.
: > +]]
: > +local function parse_time(str)
: > +    local sp = ffi.new('int[1]')
: > +    local fp = ffi.new('int[1]')
: > +    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
: > +    assert(rc > 0)
: > +    return mk_timestamp(nil, sp[0], fp[0])
: > +end
: > +
: > +--[[
: > +    Basic    Extended
: > +    Z        N/A
: > +    ±hh      N/A
: > +    ±hhmm    ±hh:mm
: 
: 30. I would recommend to avoid non-ASCII symbls in the code. I
: remember I used unicode table borders to build nice schemas
: but people were complaining it ruins their terminal's output.

Even if it's properly utf-8 encoded? That's odd. 
Will replace with + and - separately.

: 
: > +]]
: > +local function parse_zone(str)
: > +    local offset = ffi.new('int[1]')
: > +    local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
: > +    return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil,
: tonumber(len)
: 
: 31. Please, keep 80 symbols border.

Yup

: 
: I am going to stop here, will return later.

Thanks!
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 19:00     ` Timur Safin via Tarantool-patches
  2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 19:00 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg,

Thanks for quick review!

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in
: module datetime
: 
: Thanks for your patch!
: 
: 
: Some places from prevous review are still not fixed for some reasons.

Yup, that was oversight due to misunderstanding. Hopefully now I 
Understand it better.

: 
: 
: Please be careful with our Lua style guide I ponted some obvious violations.
: 
: Also it would be great to analyze module functions with our memprof. I
: think there are some places that could be optimized.

Hmm, hmm, probably a good idea. (Assuming using memprof is not a 
rocket science for such relatively newbie in LuaJIT as myself)

...
: 
: > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
: > new file mode 100644
: > index 000000000..0996ca5a2
: > --- /dev/null
: > +++ b/src/lua/datetime.lua
: > @@ -0,0 +1,581 @@
: > +local ffi = require('ffi')
: 
: Do we have any benchmarks that shows that FFI is faster than Lua C API?
: 
: Or it's just easy way to implement such module.

Ok, here is a selection list which I've seen:
- decimal is in LuaC;
- uuid is in ffi;

It looked equally acceptable and I've selected the one which is
more convenient - via ffi. Now, you've pointed me to the probable
performance issues we have with ffi, which are not there with LuaC.
I believe they have been fixed with that Mike fix for NYI in
structure allocations (after Sergos dirty fix) and once we will land
this patch to our local fork those performance issues for structures 
allocations will gone.

In any case, 1st version of module is not yet a proper moment to start
to worry about performance, more rather about clarity and quality of API.
We need to return to benchmarking of this code soon after code freeze
lifted.

: 
: 
: > +local cdt = ffi.C
: > +
: > +ffi.cdef [[
: > +
: > +    typedef int dt_t;
: > +
: > +    // dt_core.h
: > +    typedef enum {
: > +        DT_MON       = 1,
: > +        DT_MONDAY    = 1,
: > +        DT_TUE       = 2,
: > +        DT_TUESDAY   = 2,
: > +        DT_WED       = 3,
: > +        DT_WEDNESDAY = 3,
: > +        DT_THU       = 4,
: > +        DT_THURSDAY  = 4,
: > +        DT_FRI       = 5,
: > +        DT_FRIDAY    = 5,
: > +        DT_SAT       = 6,
: > +        DT_SATURDAY  = 6,
: > +        DT_SUN       = 7,
: > +        DT_SUNDAY    = 7,
: > +    } dt_dow_t;
: > +
: > +    dt_t     dt_from_rdn     (int n);
: > +    dt_t     dt_from_yd      (int y, int d);
: > +    dt_t     dt_from_ymd     (int y, int m, int d);
: > +    dt_t     dt_from_yqd     (int y, int q, int d);
: > +    dt_t     dt_from_ywd     (int y, int w, int d);
: > +
: > +    void     dt_to_yd        (dt_t dt, int *y, int *d);
: > +    void     dt_to_ymd       (dt_t dt, int *y, int *m, int *d);
: > +    void     dt_to_yqd       (dt_t dt, int *y, int *q, int *d);
: > +    void     dt_to_ywd       (dt_t dt, int *y, int *w, int *d);
: > +
: > +    int      dt_rdn          (dt_t dt);
: > +    dt_dow_t dt_dow          (dt_t dt);
: > +
: > +    // dt_parse_iso.h
: > +    size_t dt_parse_iso_date          (const char *str, size_t len, dt_t
: *dt);
: > +
: > +    size_t dt_parse_iso_time          (const char *str, size_t len, int
: *sod, int *nsec);
: > +    size_t dt_parse_iso_time_basic    (const char *str, size_t len, int
: *sod, int *nsec);
: > +    size_t dt_parse_iso_time_extended (const char *str, size_t len, int
: *sod, int *nsec);
: > +
: > +    size_t dt_parse_iso_zone          (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_basic    (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_extended (const char *str, size_t len, int
: *offset);
: > +    size_t dt_parse_iso_zone_lenient  (const char *str, size_t len, int
: *offset);
: > +
: > +    // dt_tm.h
: > +    dt_t    dt_from_struct_tm  (const struct tm *tm);
: > +    void    dt_to_struct_tm    (dt_t dt, struct tm *tm);
: > +
: > +    // <asm-generic/posix_types.h>
: > +    typedef long            __kernel_long_t;
: > +    typedef unsigned long   __kernel_ulong_t;
: > +    // /usr/include/x86_64-linux-gnu/bits/types/time_t.h
: > +    typedef long            time_t;
: > +
: > +
: > +    // <time.h>
: > +    typedef __kernel_long_t	__kernel_time_t;
: > +    typedef __kernel_long_t	__kernel_suseconds_t;
: > +
: > +    struct timespec {
: > +        __kernel_time_t	        tv_sec;     /* seconds */
: > +        long                    tv_nsec;    /* nanoseconds */
: > +    };
: > +
: > +    struct timeval {
: > +        __kernel_time_t	        tv_sec;	    /* seconds */
: > +        __kernel_suseconds_t    tv_usec;    /* microseconds */
: > +    };
: > +
: > +    struct timezone {
: > +        int	tz_minuteswest;     /* minutes west of Greenwich */
: > +        int	tz_dsttime;	        /* type of dst correction */
: > +    };
: > +
: > +    // /usr/include/x86_64-linux-gnu/sys/time.h
: > +    typedef struct timezone * __timezone_ptr_t;
: > +
: > +    /* Get the current time of day and timezone information,
: > +       putting it into *TV and *TZ.  If TZ is NULL, *TZ is not filled.
: > +       Returns 0 on success, -1 on errors.
: > +
: > +       NOTE: This form of timezone information is obsolete.
: > +       Use the functions and variables declared in <time.h> instead.  */
: > +    int gettimeofday (struct timeval *__tv, struct timezone * __tz);
: > +
: > +    // /usr/include/x86_64-linux-gnu/bits/types/struct_tm.h
: > +    /* ISO C `broken-down time' structure.  */
: > +    struct tm
: > +    {
: > +        int tm_sec;	        /* Seconds.	[0-60] (1 leap second) */
: > +        int tm_min;	        /* Minutes.	[0-59] */
: > +        int tm_hour;        /* Hours.	[0-23] */
: > +        int tm_mday;        /* Day.		[1-31] */
: > +        int tm_mon;	        /* Month.	[0-11] */
: > +        int tm_year;        /* Year	- 1900.  */
: > +        int tm_wday;        /* Day of week.	[0-6] */
: > +        int tm_yday;        /* Days in year.[0-365]	*/
: > +        int tm_isdst;       /* DST.		[-1/0/1]*/
: > +
: > +        long int tm_gmtoff; /* Seconds east of UTC.  */
: > +        const char *tm_zone;/* Timezone abbreviation.  */
: > +    };
: > +
: > +    // <time.h>
: > +    /* Return the current time and put it in *TIMER if TIMER is not NULL.
: */
: > +    time_t time (time_t *__timer);
: > +
: > +    /* Format TP into S according to FORMAT.
: > +    Write no more than MAXSIZE characters and return the number
: > +    of characters written, or 0 if it would exceed MAXSIZE.  */
: > +    size_t strftime (char * __s, size_t __maxsize, const char * __format,
: > +                     const struct tm * __tp);
: > +
: > +    /* Parse S according to FORMAT and store binary time information in
: TP.
: > +    The return value is a pointer to the first unparsed character in S.
: */
: > +    char *strptime (const char * __s, const char * __fmt, struct tm
: *__tp);
: > +
: > +    /* Return the `struct tm' representation of *TIMER in UTC,
: > +    using *TP to store the result.  */
: > +    struct tm *gmtime_r (const time_t * __timer, struct tm * __tp);
: > +
: > +    /* Return the `struct tm' representation of *TIMER in local time,
: > +    using *TP to store the result.  */
: > +    struct tm *localtime_r (const time_t * __timer, struct tm * __tp);
: > +
: > +    /* Return a string of the form "Day Mon dd hh:mm:ss yyyy\n"
: > +    that is the representation of TP in this format.  */
: > +    char *asctime (const struct tm *__tp);
: > +
: > +    /* Equivalent to `asctime (localtime (timer))'.  */
: > +    char *ctime (const time_t *__timer);
: > +
: > +]]
: > +
: > +local native = ffi.C
: 
: You've aleady declared cdt as ffi.C. There is a redefinition here. Is it
: really needed.

Thanks for observation - that was copy-paste oversight from external module
code (tsafin/c-dt-ffi), which is not making much sense if module is built-in.

: 
: I don't see anybody use "native" for ffi.C. There are two ways -
: "builtin" and "C".

'builtin` is better, yes. 

: 
: I suppose to choose one of them.
: 
: > +
: > +local SECS_PER_DAY     = 86400
: > +local NANOS_PER_SEC    = 1000000000LL
: > +
: > +-- c-dt/dt_config.h
: > +
: > +-- Unix, January 1, 1970, Thursday
: > +local DT_EPOCH_1970_OFFSET = 719163LL
: 
: To be honest it's not completely clear that such value means until
: 
: I visited
: https://github.com/chansen/c-
: dt/blob/21b8cd1fcb984386b7d4552c16fdd03fafab2b6a/dt_config.h#L50.
: 
: I think some comment is needed here.

Yeah, having explained this same topic to Vlad I now realize it looks
a little bit confusing :)

: 
: 
: > +
: > +
: > +local datetime_t = ffi.typeof('struct datetime_t')
: > +local interval_t = ffi.typeof('struct datetime_interval_t')
: > +
: > +local function interval_new()
: > +    local interval = ffi.new(interval_t)
: > +    return interval
: > +end
: > +
: > +local function adjusted_secs(dt)
: > +    return dt.secs - dt.offset * 60
: > +end
: > +
: > +local function datetime_sub(lhs, rhs)
: > +    local s1 = adjusted_secs(lhs)
: > +    local s2 = adjusted_secs(rhs)
: > +    local d = interval_new()
: > +    d.secs = s2 - s1
: > +    d.nsec = rhs.nsec - lhs.nsec
: > +    if d.nsec < 0 then
: > +        d.secs = d.secs - 1
: > +        d.nsec = d.nsec + NANOS_PER_SEC
: > +    end
: > +    return d
: > +end
: > +
: > +local function datetime_add(lhs, rhs)
: > +    local s1 = adjusted_secs(lhs)
: > +    local s2 = adjusted_secs(rhs)
: > +    local d = interval_new()
: > +    d.secs = s2 + s1
: > +    d.nsec = rhs.nsec + lhs.nsec
: > +    if d.nsec >= NANOS_PER_SEC then
: > +        d.secs = d.secs + 1
: > +        d.nsec = d.nsec - NANOS_PER_SEC
: > +    end
: > +    return d
: > +end
: > +
: > +local function datetime_eq(lhs, rhs)
: > +    -- we usually don't need to check nullness
: > +    -- but older tarantool console will call us checking for equality to
: nil
: > +    if rhs == nil then
: > +        return false
: > +    end
: > +    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
: > +end
: > +
: 
: As in the previous review it will fail on attempt to compare with not
: datetime value.
: 
: tarantool> datetime.new() == newproxy()
: ---
: - error: 'builtin/datetime.lua:222: bad argument #1 to ''is_datetime''
: (C type expected,
:      got userdata)'
: ...
: 
: 
: Expected false.

Understood. Finally!

Here is the code with which I've ended up now:

@@ -292,30 +295,32 @@ local function interval_seconds_new(s)
     return o
 end
 
-local function datetime_eq(lhs, rhs)
-    check_date_interval(lhs, "datetime:__eq(date or interval)")
-    check_date_interval(rhs, "datetime:__eq(date or interval)")
-    return (lhs.secs == rhs.secs) and (lhs.nsec == rhs.nsec)
+local function datetime_cmp(lhs, rhs)
+    if not is_date_interval(lhs) or
+       not is_date_interval(rhs) then
+       return nil
+    end
+    return (lhs.secs - rhs.secs) or (lhs.nsec - rhs.nsec)
 end
 
+local function datetime_eq(lhs, rhs)
+    local rc = datetime_cmp(lhs, rhs) 
+    return rc ~= nil and rc == 0 or false
+end
 
 local function datetime_lt(lhs, rhs)
-    check_date_interval(lhs, "datetime:__lt(date or interval)")
-    check_date_interval(rhs, "datetime:__lt(date or interval)")
-    return (lhs.secs < rhs.secs) or
-           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
+    local rc = datetime_cmp(lhs, rhs) 
+    return rc ~= nil and rc < 0 or false
 end
 
 local function datetime_le(lhs, rhs)
-    check_date_interval(lhs, "datetime:__le(date or interval)")
-    check_date_interval(rhs, "datetime:__le(date or interval)")
-    return (lhs.secs <= rhs.secs) or
-           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
+    local rc = datetime_cmp(lhs, rhs) 
+    return rc ~= nil and rc <= 0 or false
 end


: 
: 
: > +
: > +local function datetime_lt(lhs, rhs)
: > +    return (lhs.secs < rhs.secs) or
: > +           (lhs.secs == rhs.secs and lhs.nsec < rhs.nsec)
: > +end
: > +
: > +local function datetime_le(lhs, rhs)
: > +    return (lhs.secs <= rhs.secs) or
: > +           (lhs.secs == rhs.secs and lhs.nsec <= rhs.nsec)
: > +end
: > +
: > +local function datetime_serialize(self)
: > +    -- Allow YAML, MsgPack and JSON to dump objects with sockets
: > +    return { secs = self.secs, nsec = self.nsec, tz = self.offset }
: > +end
: > +
: > +local function interval_serialize(self)
: > +    -- Allow YAML and JSON to dump objects with sockets
: > +    return { secs = self.secs, nsec = self.nsec }
: > +end
: > +
: > +local datetime_mt = {
: > +    -- __tostring = datetime_tostring,
: > +    __serialize = datetime_serialize,
: > +    __eq = datetime_eq,
: > +    __lt = datetime_lt,
: > +    __le = datetime_le,
: > +    __sub = datetime_sub,
: > +    __add = datetime_add,
: > +
: > +    nanoseconds = function(self)
: > +        return tonumber(self.secs*NANOS_PER_SEC + self.nsec)
: I think there should be a space before and after "*". Here and below.
: > +    end,
: > +    microseconds = function(self)
: > +        return tonumber(self.secs*1e6 + self.nsec*1e3)
: > +    end,
: > +    seconds = function(self)
: > +        return tonumber(self.secs + self.nsec*1e3)
: > +    end,
: > +    minutes = function(self)
: > +        return tonumber((self._ticks/(1e6*60))%60)
: > +    end,
: > +    hours = function(self)
: > +        return tonumber(self._ticks/(1e6*60*60))
: > +    end,
: > +
: I think this empty line could be removed.
: > +}
: > +
: > +local interval_mt = {
: > +    -- __tostring = interval_tostring,
: > +    __serialize = interval_serialize,
: > +    __eq = datetime_eq,
: > +    __lt = datetime_lt,
: > +    __le = datetime_le,
: > +}
: > +
: > +local function datetime_new_raw(secs, nsec, offset)
: > +    local dt_obj = ffi.new(datetime_t)
: > +    dt_obj.secs = secs
: > +    dt_obj.nsec = nsec
: > +    dt_obj.offset = offset
: > +    return dt_obj
: > +end
: > +
: > +local function local_rd(o)
: > +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) +
: DT_EPOCH_1970_OFFSET
: > +end
: > +
: > +local function local_dt(o)
: > +    return cdt.dt_from_rdn(local_rd(o))
: > +end
: > +
: > +local function mk_timestamp(dt, sp, fp, offset)
: > +    local epochV = dt ~= nil and (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET)
: * SECS_PER_DAY or 0
: > +    local spV = sp ~= nil and sp or 0
: > +    local fpV = fp ~= nil and fp or 0
: > +    local ofsV = offset ~= nil and offset or 0
: > +    return datetime_new_raw (epochV + spV - ofsV * 60, fpV, ofsV)
: > +end
: > +
: > +-- create @datetime_t given object @o fields
: > +local function datetime_new(o)
: > +    if o == nil then
: > +        return datetime_new_raw(0, 0, 0)
: > +    end
: > +    local secs = 0
: > +    local nsec = 0
: > +    local offset = 0
: > +    local easy_way = false

: What does "easy_way" mean?

As I've already tried to explain it to Vlad elsewhere: easy_way means
direct usage of secs/nsec/offset attributed passed for creation of a 
new datetime object, and not parsing of human-readable year/month/day/
hour/minute/seconds attributes it has to deal with otherwise.

: > +    local y, M, d, ymd
: > +    y, M, d, ymd = 0, 0, 0, false
: > +
: > +    local h, m, s, frac, hms
: > +    h, m, s, frac, hms = 0, 0, 0, 0, false
: > +
: > +    local dt = 0
: > +
: > +    for key, value in pairs(o) do
: > +        local handlers = {
: 
: Here you recreate handlers for each iteration of the loop. I think it
: should be reworked.

Yes, let me check how better will be if this handlers array will be invariant
in the loop, and not created each iteration.

I still prefer this tag to function approach because of a better clarity,
but I need to verify in benchmark whether it provides acceptable performance.

I do not expect thought that this attribute object creation approach would become
a bootleneck in any case and might be on a critical path anywhere (I'd rather expect 
it for datetime objects created after parsing of massive bunch of log text), 
but need to look into timings we would have with invariant table.

: 
: Currenlty it's quite slow I think even if-elseif-else branches will work
: faster without
: 
: creating redundant GC objects.

Will see ...

: 
: > +            secs = function(v)
: > +                secs = v
: > +                easy_way = true
: > +            end,
: > +
: > +            nsec = function(v)
: > +                nsec = v
: > +                easy_way = true
: > +            end,
: > +
: > +            offset = function (v)
: > +                offset = v
: > +                easy_way = true
: > +            end,
: > +
: > +            year = function(v)
: > +                assert(v > 0 and v < 10000)
: Still there are some assertions that yield unclear error messages.

They have been already reworked in a later patches of patchset. 
[Probably better to squash them all for datetime.lua, yes]

: > +                y = v
: > +                ymd = true
: > +            end,
: > +
: > +            month = function(v)
: > +                assert(v > 0 and v < 12 )
: > +                M = v
: > +                ymd = true
: > +            end,
: > +
: > +            day = function(v)
: > +                assert(v > 0 and v < 32)
: > +                d = v
: > +                ymd = true
: > +            end,
: > +
: > +            hour = function(v)
: > +                assert(v >= 0 and v < 24)
: > +                h = v
: > +                hms = true
: > +            end,
: > +
: > +            minute = function(v)
: > +                assert(v >= 0 and v < 60)
: > +                m = v
: > +                hms = true
: > +            end,
: > +
: > +            second = function(v)
: > +                assert(v >= 0 and v < 61)
: > +                frac = v % 1
: > +                if frac then
: > +                    s = v - (v % 1)
: > +                else
: > +                    s = v
: > +                end
: > +                hms = true
: > +            end,
: > +
: > +            -- tz offset in minutes
: > +            tz = function(v)
: > +                assert(v >= 0 and v <= 720)
: > +                offset = v
: > +            end
: > +        }
: > +        handlers[key](value)
: > +    end
: > +
: > +    -- .sec, .nsec, .offset
: > +    if easy_way then
: > +        return datetime_new_raw(secs, nsec, offset)
: > +    end
: > +
: > +    -- .year, .month, .day
: > +    if ymd then
: > +        dt = dt + cdt.dt_from_ymd(y, M, d)
: > +    end
: > +
: > +    -- .hour, .minute, .second
: > +    if hms then
: > +        secs = h * 3600 + m * 60 + s
: > +    end
: > +
: > +    return mk_timestamp(dt, secs, frac, offset)
: > +end
: > +
: > +
: > +-- simple parse functions:
: > +-- parse_date/parse_time/parse_zone
: > +
: > +--[[
: > +    Basic      Extended
: > +    20121224   2012-12-24   Calendar date   (ISO 8601)
: > +    2012359    2012-359     Ordinal date    (ISO 8601)
: > +    2012W521   2012-W52-1   Week date       (ISO 8601)
: > +    2012Q485   2012-Q4-85   Quarter date
: > +]]
: > +
: > +local function parse_date(str)
: > +    local dt = ffi.new('dt_t[1]')
: > +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
: > +    assert(rc > 0)
: > +    return mk_timestamp(dt[0])
: > +end
: > +
: > +--[[
: > +    Basic               Extended
: > +    T12                 N/A
: > +    T1230               T12:30
: > +    T123045             T12:30:45
: > +    T123045.123456789   T12:30:45.123456789
: > +    T123045,123456789   T12:30:45,123456789
: > +
: > +    The time designator [T] may be omitted.
: > +]]
: > +local function parse_time(str)
: > +    local sp = ffi.new('int[1]')
: > +    local fp = ffi.new('int[1]')
: > +    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
: > +    assert(rc > 0)
: > +    return mk_timestamp(nil, sp[0], fp[0])
: > +end
: > +
: > +--[[
: > +    Basic    Extended
: > +    Z        N/A
: > +    ±hh      N/A
: > +    ±hhmm    ±hh:mm
: > +]]
: > +local function parse_zone(str)
: > +    local offset = ffi.new('int[1]')
: > +    local len = cdt.dt_parse_iso_zone_lenient(str, #str, offset)
: > +    return len > 0 and mk_timestamp(nil, nil, nil, offset[0]) or nil,
: tonumber(len)
: > +end
: > +
: > +
: > +--[[
: > +    aggregated parse functions
: > +    assumes to deal with date T time time_zone
: > +    at once
: > +
: > +    date [T] time [ ] time_zone
: > +]]
: > +local function parse_str(str)
: > +    local dt = ffi.new('dt_t[1]')
: > +    local len = #str
: > +    local n = cdt.dt_parse_iso_date(str, len, dt)
: > +    local dt_ = dt[0]
: > +    if n == 0 or len == n then
: > +        return mk_timestamp(dt_)
: > +    end
: > +
: > +    str = str:sub(tonumber(n) + 1)
: > +
: > +    local ch = str:sub(1,1)
: > +    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
: > +        return mk_timestamp(dt_)
: > +    end
: > +
: > +    str = str:sub(2)
: > +    len = #str
: > +
: > +    local sp = ffi.new('int[1]')
: > +    local fp = ffi.new('int[1]')
: > +    local n = cdt.dt_parse_iso_time(str, len, sp, fp)
: > +    if n == 0 then
: > +        return mk_timestamp(dt_)
: > +    end
: > +    local sp_ = sp[0]
: > +    local fp_ = fp[0]
: > +    if len == n then
: > +        return mk_timestamp(dt_, sp_, fp_)
: > +    end
: > +
: > +    str = str:sub(tonumber(n) + 1)
: > +
: > +    if str:sub(1,1) == ' ' then
: > +        str = str:sub(2)
: > +    end
: > +
: > +    len = #str
: > +
: > +    local offset = ffi.new('int[1]')
: > +    n = cdt.dt_parse_iso_zone_lenient(str, len, offset)
: > +    if n == 0 then
: > +        return mk_timestamp(dt_, sp_, fp_)
: > +    end
: > +    return mk_timestamp(dt_, sp_, fp_, offset[0])
: > +end
: > +
: > +local function datetime_from(o)
: > +    if o == nil or type(o) == 'table' then
: > +        return datetime_new(o)
: > +    elseif type(o) == 'string' then
: > +        return parse_str(o)
: > +    end
: > +end
: > +
: > +local function local_now()
: > +    local p_tv = ffi.new ' struct timeval [1] '
: 
: This line doesn't conform our code-style. Please wrap argument into
: brackets.

Yes, sorry. But for consistency with surrounding code I've put it
inside of parentheses.

: 
: The same for such places below.
: 
: > +    local rc = native.gettimeofday(p_tv, nil)
: > +    assert(rc == 0)
: > +
: > +    local secs = p_tv[0].tv_sec
: > +    local nsec = p_tv[0].tv_usec * 1000
: > +
: > +    local p_time = ffi.new 'time_t[1]'
: > +    local p_tm = ffi.new 'struct tm[1]'
: > +    native.time(p_time)
: > +    native.localtime_r(p_time, p_tm)
: > +    -- local dt = cdt.dt_from_struct_tm(p_tm)
: > +    local ofs = p_tm[0].tm_gmtoff / 60 -- convert seconds to minutes
: > +
: > +    return datetime_new_raw(secs, nsec, ofs) -- FIXME

: Do you plan to fix it before merge?

Thanks for spot! Done.

: > +end
: > +
: > +local function datetime_to_tm_ptr(o)
: > +    local p_tm = ffi.new 'struct tm[1]'
: > +    assert(ffi.typeof(o) == datetime_t)
: > +    -- dt_to_struct_tm() fills only date data
: > +    cdt.dt_to_struct_tm(local_dt(o), p_tm)
: > +
: > +    -- calculate the smaller data (hour, minute,
: > +    -- seconds) using datetime seconds value
: > +    local seconds_of_day = o.secs % 86400
: > +    local hour = (seconds_of_day / 3600) % 24
: > +    local minute = (seconds_of_day / 60) % 60
: > +    p_tm[0].tm_sec = seconds_of_day % 60
: > +    p_tm[0].tm_min = minute
: > +    p_tm[0].tm_hour = hour
: > +
: > +    p_tm[0].tm_gmtoff = o.offset * 60
: > +
: > +    return p_tm
: > +end
: > +
: > +local function asctime(o)
: > +    assert(ffi.typeof(o) == datetime_t)
: > +    local p_tm = datetime_to_tm_ptr(o)
: > +    return ffi.string(native.asctime(p_tm))
: > +end
: > +
: > +local function ctime(o)
: > +    assert(ffi.typeof(o) == datetime_t)
: > +    local p_time = ffi.new 'time_t[1]'
: > +    p_time[0] = o.secs
: > +    return ffi.string(native.ctime(p_time))
: > +end
: > +
: > +local function strftime(fmt, o)
: > +    assert(ffi.typeof(o) == datetime_t)
: > +    local sz = 50
: Why 50?

Good question, for ISO-8601 formats, for normal date range 
(from 0000 till 9999 years) it's more like closer to 40, but 
well, there will be various timezones and all crazy kinds of 
input format combinations, so we need to either be precautious 
with larger sizes allocations.

Or probably rely on glibc convention (which is apparently not in 
POSIX/SUSV) with 2 passes approach:
1. call with NULL so it will return size of required buffer;
2. then, after allocation, with the adjusted allocated buffer;

https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html

Something like that (beware of inline patch):

----------------------------------------------
local function strftime(fmt, o)
     check_date(o, "datetime.strftime(fmt, date)")
-    local sz = 50
-    local buff = ffi.new('char[?]', sz)
     local p_tm = datetime_to_tm_ptr(o)
-    native.strftime(buff, sz, fmt, p_tm)
+    local sz = builtin.strftime(nil, 1024, fmt, p_tm)
+    local buff = ffi.new('char[?]', sz + 1)
+    builtin.strftime(buff, sz, fmt, p_tm)
     return ffi.string(buff)
 end
----------------------------------------------

: > +    local buff = ffi.new('char[?]', sz)
: > +    local p_tm = datetime_to_tm_ptr(o)
: > +    native.strftime(buff, sz, fmt, p_tm)
: > +    return ffi.string(buff)
: > +end
: > +
: > +-- strftime may be redirected to datetime:fmt("format")
: > +local function datetime_fmt()
: > +end
: > +
: > +
: > +ffi.metatype(interval_t, interval_mt)
: > +ffi.metatype(datetime_t, datetime_mt)
: > +
: > +return setmetatable(
: > +    {
: > +        datetime = datetime_new,
: > +        interval = interval_new,
: > +
: > +        parse = parse_str,
: > +        parse_date = parse_date,
: > +        parse_time = parse_time,
: > +        parse_zone = parse_zone,
: > +        fmt = datetime_fmt,
: > +
: > +        now = local_now,
: > +    -- strptime = strptime;
: It should be dropped if you don't need it.
: > +        strftime = strftime,
: > +        asctime = asctime,
: > +        ctime = ctime,
: > +    }, {
: > +        __call = function(self, ...) return datetime_from(...) end
: > +    }
: > +)

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 20:45     ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 20:45 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg,

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime:
: datetime tests
: 
: Thanks for your patch. To be honest I don't understand the structure of
: this changes.
: 
: I expect that "datetime tests" will contain only tests. All unrelated to
: tests changes
: 
: should be squashed to the previous patch.

Yup, sudden artefact of late minute squashes. Will copy to prior commit.


: 
: 
: I put several comments below.
: 
: 
: On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
: > * created app-tap test for new builtin module `datetime.lua`
: > * added case to check datetime string formatting using:
: >    - asctime (gmt time);
: >    - ctime (local TZ time);
: >    - strftime (using given format).
: >
: > * added positive/negative checks to datetime test
: >    - extended api of datetime.parse_date, .parse_time, .parse_time_zone
: >      with a length of parsed (sub)string;
: >    - this allows us to check partially valid strings like "20121224 Foo
: bar".
: >
: > Part of #5941
: > ---
: >   src/lua/datetime.lua           |  14 ++-
: >   test/app-tap/datetime.test.lua | 191 +++++++++++++++++++++++++++++++++
: >   2 files changed, 197 insertions(+), 8 deletions(-)
: >   create mode 100755 test/app-tap/datetime.test.lua
: >
: > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
: > index 0996ca5a2..f4d2d7737 100644
: > --- a/src/lua/datetime.lua
: > +++ b/src/lua/datetime.lua
: > @@ -318,7 +318,7 @@ local function datetime_new(o)
: >               end,
: >
: >               month = function(v)
: > -                assert(v > 0 and v < 12 )
: > +                assert(v > 0 and v < 13 )
: >                   M = v
: >                   ymd = true
: >               end,
: > @@ -393,9 +393,8 @@ end
: >
: >   local function parse_date(str)
: >       local dt = ffi.new('dt_t[1]')
: > -    local rc = cdt.dt_parse_iso_date(str, #str, dt)
: > -    assert(rc > 0)
: > -    return mk_timestamp(dt[0])
: > +    local len = cdt.dt_parse_iso_date(str, #str, dt)
: > +    return len > 0 and mk_timestamp(dt[0]) or nil, tonumber(len)
: >   end
: >
: >   --[[
: > @@ -411,9 +410,8 @@ end
: >   local function parse_time(str)
: >       local sp = ffi.new('int[1]')
: >       local fp = ffi.new('int[1]')
: > -    local rc = cdt.dt_parse_iso_time(str, #str, sp, fp)
: > -    assert(rc > 0)
: > -    return mk_timestamp(nil, sp[0], fp[0])
: > +    local len = cdt.dt_parse_iso_time(str, #str, sp, fp)
: > +    return len > 0 and mk_timestamp(nil, sp[0], fp[0]) or nil,
: tonumber(len)
: >   end
: >
: >   --[[
: > @@ -448,7 +446,7 @@ local function parse_str(str)
: >       str = str:sub(tonumber(n) + 1)
: >
: >       local ch = str:sub(1,1)
: > -    if ch ~= 't' and ch ~= 'T' and ch ~= ' ' then
: > +    if ch:match('[Tt ]') == nil then
: >           return mk_timestamp(dt_)
: >       end
: >
: > diff --git a/test/app-tap/datetime.test.lua b/test/app-
: tap/datetime.test.lua
: > new file mode 100755
: > index 000000000..09c968858
: > --- /dev/null
: > +++ b/test/app-tap/datetime.test.lua
: > @@ -0,0 +1,191 @@
: > +#!/usr/bin/env tarantool
: > +
: > +local tap = require('tap')
: > +local test = tap.test("errno")
: > +local date = require('datetime')
: > +
: > +test:plan(5)
: > +
: > +test:test("Simple tests for parser", function(test)
: > +    test:plan(2)
: > +    test:ok(date("1970-01-01T01:00:00Z") ==
: > +            date {year=1970, month=1, day=1, hour=1, minute=0, second=0})
: > +    test:ok(date("1970-01-01T02:00:00+02:00") ==
: > +            date {year=1970, month=1, day=1, hour=2, minute=0, second=0,
: tz=120})
: > +end)
: > +
: > +test:test("Multiple tests for parser (with nanoseconds)", function(test)
: > +    test:plan(165)
: > +    -- borrowed from p5-time-moments/t/180_from_string.t
: > +    local tests =
: > +    {
: > +        { '1970-01-01T00:00:00Z',                       0,           0,
: 0 },
: > +        { '1970-01-01T02:00:00+02:00',                  0,           0,
: 120 },
: > +        { '1970-01-01T01:30:00+01:30',                  0,           0,
: 90 },
: > +        { '1970-01-01T01:00:00+01:00',                  0,           0,
: 60 },
: > +        { '1970-01-01T00:01:00+00:01',                  0,           0,
: 1 },
: > +        { '1970-01-01T00:00:00+00:00',                  0,           0,
: 0 },
: > +        { '1969-12-31T23:59:00-00:01',                  0,           0,
: -1 },
: > +        { '1969-12-31T23:00:00-01:00',                  0,           0,
: -60 },
: > +        { '1969-12-31T22:30:00-01:30',                  0,           0,
: -90 },
: > +        { '1969-12-31T22:00:00-02:00',                  0,           0, -
: 120 },
: > +        { '1970-01-01T00:00:00.123456789Z',             0,   123456789,
: 0 },
: > +        { '1970-01-01T00:00:00.12345678Z',              0,   123456780,
: 0 },
: > +        { '1970-01-01T00:00:00.1234567Z',               0,   123456700,
: 0 },
: > +        { '1970-01-01T00:00:00.123456Z',                0,   123456000,
: 0 },
: > +        { '1970-01-01T00:00:00.12345Z',                 0,   123450000,
: 0 },
: > +        { '1970-01-01T00:00:00.1234Z',                  0,   123400000,
: 0 },
: > +        { '1970-01-01T00:00:00.123Z',                   0,   123000000,
: 0 },
: > +        { '1970-01-01T00:00:00.12Z',                    0,   120000000,
: 0 },
: > +        { '1970-01-01T00:00:00.1Z',                     0,   100000000,
: 0 },
: > +        { '1970-01-01T00:00:00.01Z',                    0,    10000000,
: 0 },
: > +        { '1970-01-01T00:00:00.001Z',                   0,     1000000,
: 0 },
: > +        { '1970-01-01T00:00:00.0001Z',                  0,      100000,
: 0 },
: > +        { '1970-01-01T00:00:00.00001Z',                 0,       10000,
: 0 },
: > +        { '1970-01-01T00:00:00.000001Z',                0,        1000,
: 0 },
: > +        { '1970-01-01T00:00:00.0000001Z',               0,         100,
: 0 },
: > +        { '1970-01-01T00:00:00.00000001Z',              0,          10,
: 0 },
: > +        { '1970-01-01T00:00:00.000000001Z',             0,           1,
: 0 },
: > +        { '1970-01-01T00:00:00.000000009Z',             0,           9,
: 0 },
: > +        { '1970-01-01T00:00:00.00000009Z',              0,          90,
: 0 },
: > +        { '1970-01-01T00:00:00.0000009Z',               0,         900,
: 0 },
: > +        { '1970-01-01T00:00:00.000009Z',                0,        9000,
: 0 },
: > +        { '1970-01-01T00:00:00.00009Z',                 0,       90000,
: 0 },
: > +        { '1970-01-01T00:00:00.0009Z',                  0,      900000,
: 0 },
: > +        { '1970-01-01T00:00:00.009Z',                   0,     9000000,
: 0 },
: > +        { '1970-01-01T00:00:00.09Z',                    0,    90000000,
: 0 },
: > +        { '1970-01-01T00:00:00.9Z',                     0,   900000000,
: 0 },
: > +        { '1970-01-01T00:00:00.99Z',                    0,   990000000,
: 0 },
: > +        { '1970-01-01T00:00:00.999Z',                   0,   999000000,
: 0 },
: > +        { '1970-01-01T00:00:00.9999Z',                  0,   999900000,
: 0 },
: > +        { '1970-01-01T00:00:00.99999Z',                 0,   999990000,
: 0 },
: > +        { '1970-01-01T00:00:00.999999Z',                0,   999999000,
: 0 },
: > +        { '1970-01-01T00:00:00.9999999Z',               0,   999999900,
: 0 },
: > +        { '1970-01-01T00:00:00.99999999Z',              0,   999999990,
: 0 },
: > +        { '1970-01-01T00:00:00.999999999Z',             0,   999999999,
: 0 },
: > +        { '1970-01-01T00:00:00.0Z',                     0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.00Z',                    0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.000Z',                   0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.0000Z',                  0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.00000Z',                 0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.000000Z',                0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.0000000Z',               0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.00000000Z',              0,           0,
: 0 },
: > +        { '1970-01-01T00:00:00.000000000Z',             0,           0,
: 0 },
: > +        { '1973-11-29T21:33:09Z',               123456789,           0,
: 0 },
: > +        { '2013-10-28T17:51:56Z',              1382982716,           0,
: 0 },
: > +        -- { '9999-12-31T23:59:59Z',            253402300799,
: 0,    0 },
: 
: Why is it commented?

It used to be failing to pass range checks, but now there is no reason (IIRC)
need to restore this case. Thanks!

: 
: > +    }
: > +    for _, value in ipairs(tests) do
: > +        local str, epoch, nsec, offset
: > +        str, epoch, nsec, offset = unpack(value)
: > +        local dt = date(str)
: > +        test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str,
: epoch))
: > +        test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
: > +        test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str,
: offset))
: > +    end
: > +end)
: > +
: > +local ffi = require('ffi')
: > +
: > +ffi.cdef [[
: > +    void tzset(void);
: > +]]
: > +
: 
: 
: It's probably better to require all modules at the top of the file.

Will do

: 
: > +test:test("Datetime string formatting", function(test)
: > +    test:plan(7)
: > +    local str = "1970-01-01"
: > +    local t = date(str)
: > +    test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str,
: tonumber(t.secs)))
: > +    test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec))
: > +    test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset))
: > +    test:ok(date.asctime(t) == 'Thu Jan  1 00:00:00 1970\n', ('%s:
: asctime'):format(str))
: > +    -- ctime() is local timezone dependent. To make sure that
: > +    -- test is deterministic we enforce timezone via TZ environment
: > +    -- manipulations and calling tzset()
: > +
: > +    -- redefine timezone to be always GMT-2
: > +    os.setenv('TZ', 'GMT-2')
: > +    ffi.C.tzset()
: > +    test:ok(date.ctime(t) == 'Thu Jan  1 02:00:00 1970\n', ('%s: ctime
: with timezone'):format(str))
: > +    test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime
: #1'):format(str))
: > +    test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January
: 1970', ('%s: strftime #2'):format(str))
: > +end)
: > +
: > +test:test("Parse iso date - valid strings", function(test)
: > +    test:plan(32)
: > +    local good = {
: > +        {2012, 12, 24, "20121224",                   8 },
: > +        {2012, 12, 24, "20121224  Foo bar",          8 },
: > +        {2012, 12, 24, "2012-12-24",                10 },
: > +        {2012, 12, 24, "2012-12-24 23:59:59",       10 },
: > +        {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 },
: > +        {2012, 12, 24, "2012359",                    7 },
: > +        {2012, 12, 24, "2012359T235959+0130",        7 },
: > +        {2012, 12, 24, "2012-359",                   8 },
: > +        {2012, 12, 24, "2012W521",                   8 },
: > +        {2012, 12, 24, "2012-W52-1",                10 },
: > +        {2012, 12, 24, "2012Q485",                   8 },
: > +        {2012, 12, 24, "2012-Q4-85",                10 },
: > +        {   1,  1,  1, "0001-Q1-01",                10 },
: > +        {   1,  1,  1, "0001-W01-1",                10 },
: > +        {   1,  1,  1, "0001-01-01",                10 },
: > +        {   1,  1,  1, "0001-001",                   8 },
: > +    }
: > +
: > +    for _, value in ipairs(good) do
: > +        local year, month, day, str, date_part_len;
: 
: I didn't see before that somebody uses semicolon after variables definition.

Centures of programming in C/C++ give to know :)

: 
: It's about code-style. It is clear that this is syntactically correct.
: 
: > +        year, month, day, str, date_part_len = unpack(value)
: > +        local expected_date = date{year = year, month = month, day = day}
: > +        local date_part, len
: > +        date_part, len = date.parse_date(str)
: > +        test:ok(len == date_part_len, ('%s: length check %d'):format(str,
: len))
: > +        test:ok(expected_date == date_part, ('%s: expected
: date'):format(str))
: > +    end
: > +end)
: > +
: > +test:test("Parse iso date - invalid strings", function(test)
: > +    test:plan(62)
: > +    local bad = {
: > +        "20121232"   , -- Invalid day of month
: > +        "2012-12-310", -- Invalid day of month
: > +        "2012-13-24" , -- Invalid month
: > +        "2012367"    , -- Invalid day of year
: > +        "2012-000"   , -- Invalid day of year
: > +        "2012W533"   , -- Invalid week of year
: > +        "2012-W52-8" , -- Invalid day of week
: > +        "2012Q495"   , -- Invalid day of quarter
: > +        "2012-Q5-85" , -- Invalid quarter
: > +        "20123670"   , -- Trailing digit
: > +        "201212320"  , -- Trailing digit
: > +        "2012-12"    , -- Reduced accuracy
: > +        "2012-Q4"    , -- Reduced accuracy
: > +        "2012-Q42"   , -- Invalid
: > +        "2012-Q1-1"  , -- Invalid day of quarter
: > +        "2012Q--420" , -- Invalid
: > +        "2012-Q-420" , -- Invalid
: > +        "2012Q11"    , -- Incomplete
: > +        "2012Q1234"  , -- Trailing digit
: > +        "2012W12"    , -- Incomplete
: > +        "2012W1234"  , -- Trailing digit
: > +        "2012W-123"  , -- Invalid
: > +        "2012-W12"   , -- Incomplete
: > +        "2012-W12-12", -- Trailing digit
: > +        "2012U1234"  , -- Invalid
: > +        "2012-1234"  , -- Invalid
: > +        "2012-X1234" , -- Invalid
: > +        "0000-Q1-01" , -- Year less than 0001
: > +        "0000-W01-1" , -- Year less than 0001
: > +        "0000-01-01" , -- Year less than 0001
: > +        "0000-001"   , -- Year less than 0001
: > +    }
: > +
: > +    for _, str in ipairs(bad) do
: > +        local date_part, len
: > +        date_part, len = date.parse_date(str)
: > +        test:ok(len == 0, ('%s: length check %d'):format(str, len))
: > +        test:ok(date_part == nil, ('%s: empty date check %s'):format(str,
: date_part))
: > +    end
: > +end)
: > +
: > +os.exit(test:check() and 0 or 1)

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime
  2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 21:48     ` Timur Safin via Tarantool-patches
  2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 21:48 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg,

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime:
: display datetime
: 
: Thanks for your patch. It's not review at all. I have a question:
: 
: why is datetime not scalar?

Good question - it has to be scalar. It's some oversight from my side. 

: 
: 
: Also I think that encoding and type system changes should live in
: separate patches.

I'm not 100% sure that understand this your complain. Could you please
clarify?

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices
  2021-07-29 18:56   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 22:18     ` Timur Safin via Tarantool-patches
  2021-07-31  6:30       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 22:18 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg,


: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime:
: datetime comparison for indices
: 
: Thanks for your patch. One question below.
: 
: 
: Also update with arithmetic operations doesn't work.
: 
: 
: ```
: 
: tarantool> box.space.dt:update({v}, {{'+', 2, 3}})
: ---
: - error: 'Argument type in operation ''+'' on field 2 does not match
: field type: expected
:      a number'
: ...
: 
: tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}})
: ---
: - error: unsupported Lua type 'cdata'
: ...
: 
: ```

Oh, shit, I didn't take those operations into considerations.
Do they work similarly with decimal?


: 
: 
: On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
: > * storage hints implemented for datetime_t values;
: > * proper comparison for indices of datetime type.
: >
: >
: > diff --git a/test/engine/datetime.result b/test/engine/datetime.result
: > new file mode 100644
: > index 000000000..3ff474dee
: > --- /dev/null
: > +++ b/test/engine/datetime.result
: > @@ -0,0 +1,77 @@
: > +-- test-run result file version 2
: > +env = require('test_run')
: > + | ---
: > + | ...
: > +test_run = env.new()
: > + | ---
: > + | ...
: > +engine = test_run:get_cfg('engine')
: > + | ---
: > + | ...
: > +
: > +date = require('datetime')
: > + | ---
: > + | ...
: > +
: > +_ = box.schema.space.create('T', {engine = engine})
: > + | ---
: > + | ...
: > +_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
: > + | ---
: > + | ...
: > +
: > +box.space.T:insert{date('1970-01-01')}\
: > +box.space.T:insert{date('1970-01-02')}\
: > +box.space.T:insert{date('1970-01-03')}\
: > +box.space.T:insert{date('2000-01-01')}
: > + | ---
: > + | ...
: > +
: > +o = box.space.T:select{}
: > + | ---
: > + | ...
: > +assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
: > + | ---
: > + | - true
: > + | ...
: > +assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
: > + | ---
: > + | - true
: > + | ...
: > +assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
: > + | ---
: > + | - true
: > + | ...
: > +assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
: > + | ---
: > + | - true
: > + | ...
: > +
: > +for i = 1,16 do\
: > +    box.space.T:insert{date.now()}\
: > +end
: > + | ---
: > + | ...
: > +
: > +a = box.space.T:select{}
: > + | ---
: > + | ...
: > +err = nil
: > + | ---
: > + | ...
: > +for i = 1, #a - 1 do\
: > +    if tostring(a[i][1]) >= tostring(a[i+1][1]) then\
: 
: Why do you compare string representation but not values itself?

Good question! Yeah, there should be original values compared, without
stringization. (And in this case, by some reason, it's easier to 
reproduce problem of incorrect sort ofder with sequential times from
date.now(). Which used to be reproduced only on some configs at GH)

: 
: 
: > +        err = {a[i][1], a[i+1][1]}\
: > +        break\
: > +    end\
: > +end
: > + | ---
: > + | ...
: > +
: > +err
: > + | ---
: > + | - null
: > + | ...
: > +box.space.T:drop()
: > + | ---
: > + | ...
: >


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding
  2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 22:20     ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 22:20 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches


: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime:
: proper datetime encoding
: 
: Thanks for your patch. This change should be squashed into the patch
: where encoding was introduced.

Agreed.

: 
: On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
: > * correct incorrect encoding of MP_EXT sizes for datetime
: >    messagepack values;
: > * export necessary symbols for datetime messagepack size calculations
: >    so they will be available for Lua consumption.
: >
: > Part of #5941
: > ---
: >   src/exports.h           |  2 ++
: >   src/lib/core/datetime.c | 15 ++++++++++-----
: >   src/lua/msgpackffi.lua  | 10 ++++++++++
: >   3 files changed, 22 insertions(+), 5 deletions(-)
: >

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes
  2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 22:30     ` Timur Safin via Tarantool-patches
  2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 22:30 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg again!

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime:
: calculated attributes for datetimes
: 
: Thanks for your patch. See one comment below.
: 
: Also this change should be squashed into the first patch.

Agreed as well. 

: 
: 
: On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
: > * introduced a set of calculated attributes to data object, e.g.:
: >    - timestamp, seconds, microseconds, minute, or hours
: >
: > Part of #5941
: > ---
: >   src/lua/datetime.lua           | 51 ++++++++++++++++++++++------------
: >   test/app-tap/datetime.test.lua | 17 +++++++++++-
: >   2 files changed, 50 insertions(+), 18 deletions(-)
: >
: > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
: > index 9ec06d8d8..7a208cef9 100644
: > --- a/src/lua/datetime.lua
: > +++ b/src/lua/datetime.lua
: > @@ -224,6 +224,36 @@ local function interval_serialize(self)
: >       return { secs = self.secs, nsec = self.nsec }
: >   end
: >
: > +local datetime_index = function(self, key)
: > +    local attributes = {
: 
: This change looks like pessimization because currently we need to
: recreate "attributes" table
: 
: for each function call. I suggest to use `attributes` as `__index` but
: not such function.

What would you say if I simply precache this attributes table in local,
and then proceed this way?
-----------------------------------------
-local datetime_index = function(self, key)
-    local attributes = {
+local datetime_index_handlers = {
         unixtime = function(self)
             return self.secs
         end,
+
         timestamp = function(self)
             return tonumber(self.secs) + self.nsec / 1e9
         end,
+
         nanoseconds = function(self)
             return self.secs * 1e9 + self.nsec
         end,
+
         microseconds = function(self)
             return self.secs * 1e6 + self.nsec / 1e3
         end,
+
         milliseconds = function(self)
             return self.secs * 1e3 + self.nsec / 1e6
         end,
+
         seconds = function(self)
             return tonumber(self.secs) + self.nsec / 1e9
         end,
+
         minutes = function(self)
             return (tonumber(self.secs) + self.nsec / 1e9) / 60
         end,
+
         hours = function(self)
             return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
         end,
+
         days = function(self)
             return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
         end,
+
         add = function(self)
             return function(self, o)
                 return interval_increment(self, o, 1)
             end
         end,
+
         sub = function(self)
             return function(self, o)
                 return interval_increment(self, o, -1)
             end
         end,
     }
-    return attributes[key] ~= nil and attributes[key](self) or nil
+
+local datetime_index = function(self, key)
+    return datetime_index_handlers[key] ~= nil and
+           datetime_index_handlers[key](self) or nil
 end
-----------------------------------------

I mean we still carefully check existence of an asked key, and return nil if
there is no such key at all. Is it much different than what you've meant?


: 
: 
: > +        timestamp = function(self)
: > +            return tonumber(self.secs) + self.nsec / 1e9
: > +        end,
: > +        nanoseconds = function(self)
: > +            return tonumber(self.secs * 1e9 + self.nsec)
: > +        end,
: > +        microseconds = function(self)
: > +            return tonumber(self.secs * 1e6 + self.nsec / 1e3)
: > +        end,
: > +        milliseconds = function(self)
: > +            return tonumber(self.secs * 1e3 + self.nsec / 1e6)
: > +        end,
: > +        seconds = function(self)
: > +            return tonumber(self.secs) + self.nsec / 1e9
: > +        end,
: > +        minutes = function(self)
: > +            return (tonumber(self.secs) + self.nsec / 1e9) / 60
: > +        end,
: > +        hours = function(self)
: > +            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
: > +        end,
: > +        days = function(self)
: > +            return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) /
: 24
: > +        end,
: > +    }
: > +    return attributes[key] ~= nil and attributes[key](self) or nil
: > +end
: > +
: >   local datetime_mt = {
: >       -- __tostring = datetime_tostring,
: >       __serialize = datetime_serialize,

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support
  2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 22:58     ` Timur Safin via Tarantool-patches
  2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 22:58 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hi there!

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time
: intervals support
: 
: Thanks for your patch.
: 
: I wrote several comments below.
: 
: 
: However it makes me think that such approach will work quite slow since
: 
: all functions is implemented in Lua, all arithmetic is in Lua.

I'm not that pessimistic here :) [Given the fact that I'd remove
Out of a loops all handy handlers table creation] But we need
to measure it first. 
: 
: 
...
: > @@ -158,58 +169,146 @@ local DT_EPOCH_1970_OFFSET = 719163LL
: >
: >   local datetime_t = ffi.typeof('struct datetime_t')
: >   local interval_t = ffi.typeof('struct datetime_interval_t')
: > +ffi.cdef [[
: > +    struct t_interval_months {
: > +        int m;
: > +    };
: > +
: > +    struct t_interval_years {
: > +        int y;
: > +    };
: > +]]
: > +local interval_months_t = ffi.typeof('struct t_interval_months')
: > +local interval_years_t = ffi.typeof('struct t_interval_years')
: > +
: > +local function is_interval(o)
: > +    return ffi.istype(interval_t, o) or
: > +           ffi.istype(interval_months_t, o) or
: > +           ffi.istype(interval_years_t, o)
: > +end
: > +
: 
: It will throw for non-cdata values:
: 
: tarantool> ffi.istype(interval_t, o)
: ---
: - error: 'bad argument #1 to ''?'' (C type expected, got nil)'
: ...
: 
: tarantool> ffi.istype(interval_t, 123)
: ---
: - error: 'bad argument #1 to ''?'' (C type expected, got nil)'
: ...

Yes, thanks for pin-pointing it! I've modified it this way to 
avoid such errors:
------------------------------------------------
@@ -182,15 +181,19 @@ local interval_months_t = ffi.typeof('struct t_interval_months')
 local interval_years_t = ffi.typeof('struct t_interval_years')
 
 local function is_interval(o)
-    return ffi.istype(interval_t, o) or
-           ffi.istype(interval_months_t, o) or
-           ffi.istype(interval_years_t, o)
+    return type(o) == 'cdata' and 
+           (ffi.istype(interval_t, o) or
+            ffi.istype(interval_months_t, o) or
+            ffi.istype(interval_years_t, o))
 end
 
 local function is_datetime(o)
-    return ffi.istype(o, datetime_t)
+    return type(o) == 'cdata' and ffi.istype(o, datetime_t)
 end
 
+local function is_date_interval(o)
+    return is_datetime(o) or is_interval(o)
+end
 
 local function interval_new()
     local interval = ffi.new(interval_t)
------------------------------------------------

: > @@ -224,19 +323,123 @@ local function interval_serialize(self)
: >       return { secs = self.secs, nsec = self.nsec }
: >   end
: >
: > +local function local_rd(o)
: > +    return math.floor(tonumber(o.secs / SECS_PER_DAY)) +
: DT_EPOCH_1970_OFFSET
: > +end
: > +
: > +local function local_dt(o)
: > +    return cdt.dt_from_rdn(local_rd(o))
: > +end
: > +
: > +local function _normalize_nsec(secs, nsec)
: > +    if nsec < 0 then
: > +        secs = secs - 1
: > +        nsec = nsec + NANOS_PER_SEC
: > +    elseif nsec >= NANOS_PER_SEC then
: > +        secs = secs + 1
: > +        nsec = nsec - NANOS_PER_SEC
: > +    end
: > +    return secs, nsec
: > +end
: > +
: > +-- addition or subtraction from date/time of a given interval
: > +-- described via table direction should be +1 or -1
: > +local function interval_increment(self, o, direction)
: > +    assert(direction == -1 or direction == 1)
: > +    check_date(self, "interval_increment(date, object, -+1)")
: > +    assert(type(o) == 'table')
: > +
: > +    local ym_updated = false
: > +    local dhms_updated = false
: > +
: > +    local dt = local_dt(self)
: > +    local secs, nsec
: > +    secs, nsec = self.secs, self.nsec
: > +
: > +    for key, value in pairs(o) do
: > +        local handlers = {
: 
: The same as in one previous patch. It's too expensive to recreate table
: and functions for each simple
: 
: action and for each iteration loop.

I've removed creation of a table, which is invariant in the loop, to
outside of loop statement. Let me measure how it much would different
than series of iffs. 

---------------------------------------------------------
@@ -377,52 +382,52 @@ local function interval_increment(self, o, direction)
     local secs, nsec
     secs, nsec = self.secs, self.nsec
 
+    local handlers = {
+        years = function(k, v)
+            check_range(v, {0, 9999}, k)
+            dt = builtin.dt_add_years(dt, direction * v, builtin.DT_LIMIT)
+            ym_updated = true
+        end,
+
+        months = function(k, v)
+            check_range(v, {0, 12}, k)
+            dt = builtin.dt_add_months(dt, direction * v, builtin.DT_LIMIT)
+            ym_updated = true
+        end,
+
+        weeks = function(k, v)
+            check_range(v, {0, 52}, k)
+            secs = secs + direction * 7 * v * SECS_PER_DAY
+            dhms_updated = true
+            dhms_updated = true
+        end,
+
+        days = function(k, v)
+            check_range(v, {0, 31}, k)
+            secs = secs + direction * v * SECS_PER_DAY
+            dhms_updated = true
+        end,
+
+        hours = function(k, v)
+            check_range(v, {0, 23}, k)
+            secs = secs + direction * 60 * 60 * v
+            dhms_updated = true
+        end,
+
+        minutes = function(k, v)
+            check_range(v, {0, 59}, k)
+            secs = secs + direction * 60 * v
+        end,
+
+        seconds = function(k, v)
+            check_range(v, {0, 60}, k)
+            local s, frac = seconds_fraction(v)
+            secs = secs + direction * s
+            nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
+            dhms_updated = true
+        end,
+    }
     for key, value in pairs(o) do
-        local handlers = {
-            years = function(v)
-                check_range(v, {0, 9999}, key)
-                dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
-                ym_updated = true
-            end,
-
-            months = function(v)
-                check_range(v, {0, 12}, key)
-                dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
-                ym_updated = true
-            end,
-
-            weeks = function(v)
-                check_range(v, {0, 52}, key)
-                secs = secs + direction * 7 * v * SECS_PER_DAY
-                dhms_updated = true
-            end,
-
-            days = function(v)
-                check_range(v, {0, 31}, key)
-                secs = secs + direction * v * SECS_PER_DAY
-                dhms_updated = true
-            end,
-
-            hours = function(v)
-                check_range(v, {0, 23}, key)
-                secs = secs + direction * 60 * 60 * v
-                dhms_updated = true
-            end,
-
-            minutes = function(v)
-                check_range(v, {0, 59}, key)
-                secs = secs + direction * 60 * v
-            end,
-
-            seconds = function(v)
-                check_range(v, {0, 60}, key)
-                local s, frac = seconds_fraction(v)
-                secs = secs + direction * s
-                nsec = nsec + direction * frac * 1e9 -- convert fraction to nanoseconds
-                dhms_updated = true
-            end,
-        }
-        handlers[key](value)
+        handlers[key](key, value)
     end
 
     secs, nsec = _normalize_nsec(secs, nsec)
---------------------------------------------------------

: 
: > +            years = function(v)
: > +                assert(v > 0 and v < 10000)
: > +                dt = cdt.dt_add_years(dt, direction * v, cdt.DT_LIMIT)
: > +                ym_updated = true
: > +            end,
: > +
: > +            months = function(v)
: > +                assert(v > 0 and v < 13 )
: > +                dt = cdt.dt_add_months(dt, direction * v, cdt.DT_LIMIT)
: > +                ym_updated = true
: > +            end,
: > +
: > +            weeks = function(v)
: > +                assert(v > 0 and v < 32)
: > +                secs = secs + direction * 7 * v * SECS_PER_DAY
: > +                dhms_updated = true
: > +            end,
: > +
: > +            days = function(v)
: > +                assert(v > 0 and v < 32)
: > +                secs = secs + direction * v * SECS_PER_DAY
: > +                dhms_updated = true
: > +            end,
: > +
: > +            hours = function(v)
: > +                assert(v >= 0 and v < 24)
: > +                secs = secs + direction * 60 * 60 * v
: > +                dhms_updated = true
: > +            end,
: > +
: > +            minutes = function(v)
: > +                assert(v >= 0 and v < 60)
: > +                secs = secs + direction * 60 * v
: > +            end,
: > +
: > +            seconds = function(v)
: > +                assert(v >= 0 and v < 61)
: > +                local s, frac
: > +                frac = v % 1
: > +                if frac > 0 then
: > +                    s = v - (v % 1)
: > +                else
: > +                    s = v
: > +                end
: > +                secs = secs + direction * s
: > +                nsec = nsec + direction * frac * 1e9 -- convert fraction
: to nanoseconds
: > +                dhms_updated = true
: > +            end,
: > +        }
: > +        handlers[key](value)
: > +    end
: > +
: > +    secs, nsec = _normalize_nsec(secs, nsec)
: > +
: > +    -- .days, .hours, .minutes, .seconds
: > +    if dhms_updated then
: > +        self.secs = secs
: > +        self.nsec = nsec
: > +    end
: > +
: > +    -- .years, .months updated
: > +    if ym_updated then
: > +        self.secs = (cdt.dt_rdn(dt) - DT_EPOCH_1970_OFFSET) *
: SECS_PER_DAY +
: > +                    secs % SECS_PER_DAY
: > +    end
: > +
: > +    return self
: > +end
: > +

: 
: tarantool> return require('datetime').now() + 1
: ---
: - error: '[string "return require(''datetime'').now() + 1"]:1: Usage:
: datetime:__add(interval)'
: ...
: 
: 
: Looks a bit confusing. User doesn't know about metamethods.

Partially agreed. Though I assume that users of this code will 
be educated enough. But in any case, what would be your suggested
error message here? 

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua
  2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
@ 2021-07-30 23:11     ` Timur Safin via Tarantool-patches
  2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-30 23:11 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

Hello Oleg,

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime:
: unixtime, timestamp setters in datetime.lua
: 
: Thanks for your patch.
: 
: The comment is primarly the same as in previous patch.
: 
: Why do you use such slow approach and creates redundant tables and
: function (temporary metatable in fact)
: 
: to use it only once? Please fix it.

In previous emails I've shown how I'd deal with loop invariants. Should
avoid this redundant temp objects problem.

: 
: 
: Unixtime and timestamp is great but they loss precision. I think it
: should be possible
: 
: go get timestamp with nanoseconds precision since datetime has
: nanoseconds precision.

I'm not quite get it. Do you want to have nanoseconds exported via newindex?
For direct assignments? It's not useful because of available .nsec field.
Or may be you meant something else?

: 
: 
: Also still it's hard how to convert timestamp to datetime value back.

Timestamp is something secs.nsec, what looks complicated? I'm confused.
Could you please elaborate?

: 
: 
: On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
: > * implemented proper range checks for date attributes values;
: >
: > * created `.unixtime` attribute, which is alias to `.secs`,
: >    with corresponding setter/getter;
: >
: > * similarly to `unixtime`, created virtual `timestamp` attribute
: >    setter. Which is a convenient way to simultaneously assign
: >    unixtime (seconds since epoch) and nanoseconds
: >
: > Part of #5941

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-30 19:00     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
  2021-07-31 16:51         ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:29 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your answers.

On 30.07.2021 22:00, Timur Safin wrote:
> Ok, here is a selection list which I've seen:
> - decimal is in LuaC;
> - uuid is in ffi;
>
> It looked equally acceptable and I've selected the one which is
> more convenient - via ffi. Now, you've pointed me to the probable
> performance issues we have with ffi, which are not there with LuaC.
> I believe they have been fixed with that Mike fix for NYI in
> structure allocations (after Sergos dirty fix) and once we will land
> this patch to our local fork those performance issues for structures
> allocations will gone.
>
> In any case, 1st version of module is not yet a proper moment to start
> to worry about performance, more rather about clarity and quality of API.
> We need to return to benchmarking of this code soon after code freeze
> lifted.

I don't know why uuid is implemented via FFI. Uuid as module was 
introduced at least 8 yeard ago.

Probably there were supporters of the theory that FFI/LuaJIT 
implementation could be fast.

But currently we can see something like 
https://github.com/luafun/luafun/issues/31#issuecomment-277508076

and rewrite netbox in C.

And it's not about patches in LuaJIT about structure initialization. You 
can easily cherry-pick it into

our fork and show that FFI is really better than Lua C API. It's about 
an approach. We saw benchmarks

when decimal was introduced but currently we didn't see anything.

> : > +    local y, M, d, ymd
> : > +    y, M, d, ymd = 0, 0, 0, false
> : > +
> : > +    local h, m, s, frac, hms
> : > +    h, m, s, frac, hms = 0, 0, 0, 0, false
> : > +
> : > +    local dt = 0
> : > +
> : > +    for key, value in pairs(o) do
> : > +        local handlers = {
> :
> : Here you recreate handlers for each iteration of the loop. I think it
> : should be reworked.
>
> Yes, let me check how better will be if this handlers array will be invariant
> in the loop, and not created each iteration.
>
> I still prefer this tag to function approach because of a better clarity,
> but I need to verify in benchmark whether it provides acceptable performance.
>
> I do not expect thought that this attribute object creation approach would become
> a bootleneck in any case and might be on a critical path anywhere (I'd rather expect
> it for datetime objects created after parsing of massive bunch of log text),
> but need to look into timings we would have with invariant table.
>
> :
> : Currenlty it's quite slow I think even if-elseif-else branches will work
> : faster without
> :
> : creating redundant GC objects.
>
> Will see ...

I'll simplify your task a bit:

local clock = require('clock')

local mt_fun = function(_, key)
     local mt = {
         ['key'] = function()
             return 'value'
         end
     }
     return mt[key] ~= nil and mt[key]()
end

local mt_table = {
     ['key'] = function()
         return 'value'
     end
}

local t1 = setmetatable({}, {__index = mt_fun})
local t2 = setmetatable({}, {__index = mt_table})
local _

local start = clock.time()
for _ = 1, 1e6 do
     _ = t1['key']
end
print('function', clock.time() - start)

collectgarbage()
collectgarbage()

local start = clock.time()
for _ = 1, 1e6 do
     _ = t2['key']
end
print('table', clock.time() - start)


tarantool test.lua
function    0.14332509040833
table    0.0002901554107666


The difference about x500 times on my Mac.

> : > +end
> : > +
> : > +local function strftime(fmt, o)
> : > +    assert(ffi.typeof(o) == datetime_t)
> : > +    local sz = 50
> : Why 50?
>
> Good question, for ISO-8601 formats, for normal date range
> (from 0000 till 9999 years) it's more like closer to 40, but
> well, there will be various timezones and all crazy kinds of
> input format combinations, so we need to either be precautious
> with larger sizes allocations.

On the one side you are right. But if user want to format string with 
length more than 50

it will be silently truncated.


```

tarantool> datetime.strftime('%d' .. string.rep('content', 50), 
datetime.new())
---
- 01contentcontentcontentcontentcontentcontentconten
...

```

I'm not sure that anybody will use strftime in such way but it's strange.

Maybe you could use preallocated buffer for short strings and allocate 
huge if

it's not enough. Some kind of optimistic approach.


> Or probably rely on glibc convention (which is apparently not in
> POSIX/SUSV) with 2 passes approach:
> 1. call with NULL so it will return size of required buffer;
> 2. then, after allocation, with the adjusted allocated buffer;
>
> https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
> https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html
>
> Something like that (beware of inline patch):
>
> ----------------------------------------------
> local function strftime(fmt, o)
>       check_date(o, "datetime.strftime(fmt, date)")
> -    local sz = 50
> -    local buff = ffi.new('char[?]', sz)
>       local p_tm = datetime_to_tm_ptr(o)
> -    native.strftime(buff, sz, fmt, p_tm)
> +    local sz = builtin.strftime(nil, 1024, fmt, p_tm)
> +    local buff = ffi.new('char[?]', sz + 1)
> +    builtin.strftime(buff, sz, fmt, p_tm)
>       return ffi.string(buff)
>   end
> ----------------------------------------------
>
> : > +    local buff = ffi.new('char[?]', sz)
> : > +    local p_tm = datetime_to_tm_ptr(o)
> : > +    native.strftime(buff, sz, fmt, p_tm)
> : > +    return ffi.string(buff)
> : > +end
> : > +
> : > +-- strftime may be redirected to datetime:fmt("format")
> : > +local function datetime_fmt()
> : > +end
> : > +
> : > +
> : > +ffi.metatype(interval_t, interval_mt)
> : > +ffi.metatype(datetime_t, datetime_mt)
> : > +
> : > +return setmetatable(
> : > +    {
> : > +        datetime = datetime_new,
> : > +        interval = interval_new,
> : > +
> : > +        parse = parse_str,
> : > +        parse_date = parse_date,
> : > +        parse_time = parse_time,
> : > +        parse_zone = parse_zone,
> : > +        fmt = datetime_fmt,
> : > +
> : > +        now = local_now,
> : > +    -- strptime = strptime;
> : It should be dropped if you don't need it.
> : > +        strftime = strftime,
> : > +        asctime = asctime,
> : > +        ctime = ctime,
> : > +    }, {
> : > +        __call = function(self, ...) return datetime_from(...) end
> : > +    }
> : > +)
>
> Thanks,
> Timur
>

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime
  2021-07-30 21:48     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:29 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your answers. See my answer below.

On 31.07.2021 00:48, Timur Safin wrote:
> :
> :
> : Also I think that encoding and type system changes should live in
> : separate patches.
>
> I'm not 100% sure that understand this your complain. Could you please
> clarify?
>
Hm... Probably I missed something. As I see now decimal encoding to msgpack

and json/yaml... serializations are in the one patch 
(https://github.com/tarantool/tarantool/commit/485439e33196e26d120e622175f88b4edc7a5aa1).

Seems my comment in not relevant, sorry.


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices
  2021-07-30 22:18     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:30       ` Oleg Babin via Tarantool-patches
  2021-07-31  9:31         ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:30 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your reply. See my answer below.

On 31.07.2021 01:18, Timur Safin wrote:
> Hello Oleg,
>
>
> : From: Oleg Babin<olegrok@tarantool.org>
> : Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime:
> : datetime comparison for indices
> :
> : Thanks for your patch. One question below.
> :
> :
> : Also update with arithmetic operations doesn't work.
> :
> :
> : ```
> :
> : tarantool> box.space.dt:update({v}, {{'+', 2, 3}})
> : ---
> : - error: 'Argument type in operation ''+'' on field 2 does not match
> : field type: expected
> :      a number'
> : ...
> :
> : tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}})
> : ---
> : - error: unsupported Lua type 'cdata'
> : ...
> :
> : ```
>
> Oh, shit, I didn't take those operations into considerations.
> Do they work similarly with decimal?

Looks like so. Moreover it seems we should support intervals as separate 
type.

Currently the single way to update is get + sum + put/update (with =).

But I suppose native update will be needed anyway. At lest for SQL.

Will be it able to perform some operations with intervals and timestamps?


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes
  2021-07-30 22:30     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:31 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches

Thanks for your reply. See my comment below.

On 31.07.2021 01:30, Timur Safin wrote:
> Hello Oleg again!
>
> :
> :
> : On 28.07.2021 13:34, Timur Safin via Tarantool-patches wrote:
> : > * introduced a set of calculated attributes to data object, e.g.:
> : >    - timestamp, seconds, microseconds, minute, or hours
> : >
> : > Part of #5941
> : > ---
> : >   src/lua/datetime.lua           | 51 ++++++++++++++++++++++------------
> : >   test/app-tap/datetime.test.lua | 17 +++++++++++-
> : >   2 files changed, 50 insertions(+), 18 deletions(-)
> : >
> : > diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> : > index 9ec06d8d8..7a208cef9 100644
> : > --- a/src/lua/datetime.lua
> : > +++ b/src/lua/datetime.lua
> : > @@ -224,6 +224,36 @@ local function interval_serialize(self)
> : >       return { secs = self.secs, nsec = self.nsec }
> : >   end
> : >
> : > +local datetime_index = function(self, key)
> : > +    local attributes = {
> :
> : This change looks like pessimization because currently we need to
> : recreate "attributes" table
> :
> : for each function call. I suggest to use `attributes` as `__index` but
> : not such function.
>
> What would you say if I simply precache this attributes table in local,
> and then proceed this way?
> -----------------------------------------
> -local datetime_index = function(self, key)
> -    local attributes = {
> +local datetime_index_handlers = {
>           unixtime = function(self)
>               return self.secs
>           end,
> +
>           timestamp = function(self)
>               return tonumber(self.secs) + self.nsec / 1e9
>           end,
> +
>           nanoseconds = function(self)
>               return self.secs * 1e9 + self.nsec
>           end,
> +
>           microseconds = function(self)
>               return self.secs * 1e6 + self.nsec / 1e3
>           end,
> +
>           milliseconds = function(self)
>               return self.secs * 1e3 + self.nsec / 1e6
>           end,
> +
>           seconds = function(self)
>               return tonumber(self.secs) + self.nsec / 1e9
>           end,
> +
>           minutes = function(self)
>               return (tonumber(self.secs) + self.nsec / 1e9) / 60
>           end,
> +
>           hours = function(self)
>               return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60)
>           end,
> +
>           days = function(self)
>               return (tonumber(self.secs) + self.nsec / 1e9) / (60 * 60) / 24
>           end,
> +
>           add = function(self)
>               return function(self, o)
>                   return interval_increment(self, o, 1)
>               end
>           end,
> +
>           sub = function(self)
>               return function(self, o)
>                   return interval_increment(self, o, -1)
>               end
>           end,
>       }
> -    return attributes[key] ~= nil and attributes[key](self) or nil
> +
> +local datetime_index = function(self, key)
> +    return datetime_index_handlers[key] ~= nil and
> +           datetime_index_handlers[key](self) or nil
>   end
> -----------------------------------------
>
> I mean we still carefully check existence of an asked key, and return nil if
> there is no such key at all. Is it much different than what you've meant?
>
It's again about temporary tables. I gave a benchmark in one of the 
previous emails.

After changes it shoud be better. Thanks.


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support
  2021-07-30 22:58     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  2021-07-31  9:20         ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:31 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches


On 31.07.2021 01:58, Timur Safin wrote:
> :
> : tarantool> return require('datetime').now() + 1
> : ---
> : - error: '[string "return require(''datetime'').now() + 1"]:1: Usage:
> : datetime:__add(interval)'
> : ...
> :
> :
> : Looks a bit confusing. User doesn't know about metamethods.
>
> Partially agreed. Though I assume that users of this code will
> be educated enough. But in any case, what would be your suggested
> error message here?
>
In fact I don't know exactly.

But decimal shows something like:

```

tarantool> decimal.new(1) + 'abc'
---
- error: '[string "return decimal.new(1) + ''abc''"]:1: incorrect value 
to convert
     to decimal as 2 argument'
...

```


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua
  2021-07-30 23:11     ` Timur Safin via Tarantool-patches
@ 2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
  2021-07-31  9:54         ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Oleg Babin via Tarantool-patches @ 2021-07-31  6:31 UTC (permalink / raw)
  To: Timur Safin, v.shpilevoy; +Cc: tarantool-patches


On 31.07.2021 02:11, Timur Safin wrote:
> Hello Oleg,
>
> :
> :
> : Unixtime and timestamp is great but they loss precision. I think it
> : should be possible
> :
> : go get timestamp with nanoseconds precision since datetime has
> : nanoseconds precision.
>
> I'm not quite get it. Do you want to have nanoseconds exported via newindex?
> For direct assignments? It's not useful because of available .nsec field.
> Or may be you meant something else?
>
> :
> :
> : Also still it's hard how to convert timestamp to datetime value back.
>
> Timestamp is something secs.nsec, what looks complicated? I'm confused.
> Could you please elaborate?

I expected something like.

tarantool> clock.time64()
---
- 1627712632369020000
...


This timestamp is in nanoseconds. It's unsigned long long value.

I tried to do something for datetime...

sec * 1e9 + nsec, but sec is null while nsec is not:

```

tarantool> dt.sec
---
- null
...

```


timestamp + nsec works in quite strage way:

```

tarantool> dt.timestamp * (1e9 * 1ULL) + dt.nsec
---
- 1627712485108074000
...

```

(I need to cast to ULL here to avoid precision lost).

But maybe my question is not relevant. There is a way to get nanoseconds 
timestamp.

I just need to write some simple helper for my app. It's ok.


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support
  2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
@ 2021-07-31  9:20         ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-31  9:20 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time
: intervals support
: 
: 
: On 31.07.2021 01:58, Timur Safin wrote:
: > :
: > : tarantool> return require('datetime').now() + 1
: > : ---
: > : - error: '[string "return require(''datetime'').now() + 1"]:1: Usage:
: > : datetime:__add(interval)'
: > : ...
: > :
: > :
: > : Looks a bit confusing. User doesn't know about metamethods.
: >
: > Partially agreed. Though I assume that users of this code will
: > be educated enough. But in any case, what would be your suggested
: > error message here?
: >
: In fact I don't know exactly.
: 
: But decimal shows something like:
: 
: ```
: 
: tarantool> decimal.new(1) + 'abc'
: ---
: - error: '[string "return decimal.new(1) + ''abc''"]:1: incorrect value
: to convert
:      to decimal as 2 argument'
: ...
: 
: ```

Very good, indeed. So the trick is to say that type is unexpected and optionally show of which value it's camplaining about.

Like:

Expected datetime or interval argument, but received '%s'.



^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build
  2021-07-29 23:40   ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-07-31  9:22     ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-31  9:22 UTC (permalink / raw)
  To: 'Vladislav Shpilevoy'; +Cc: tarantool-patches

: From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
: Subject: Re: [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the
: build
: 
: Hi! Thanks for the patch!
: 
: On 28.07.2021 12:34, Timur Safin wrote:
: > * Integrated chansen/c-dt parser as 3rd party module to the
: >   Tarantool cmake build process.
: > * Points to tsafin/c-dt instead iof original chansen/c-dt to
: 
: iof -> of. But is it even true? Below I see

Thanks!

: 
: 	url = https://github.com/tarantool/c-dt.git
: 
: not tsafin/c-dt.

We have relocated repository to Tarantool organization since then.

: 
: After build I get in git status this:
: 
: 	Changes not staged for commit:
: 	  (use "git add <file>..." to update what will be committed)
: 	  (use "git restore <file>..." to discard changes in working
: directory)
: 	  (commit or discard the untracked or modified content in submodules)
: 		modified:   third_party/c-dt (modified content, untracked
: content)
: 
: It says:
: 
: 	modified:   Makefile
: 
: and some untracked files:
: 
: 	CMakeFiles/
: 	cmake_install.cmake
: 	libcdt.a
: 
: Why is even Makefile added to the repository if we are
: using cmake anyway?

Yeah, sounds like a small cmake configuration annoyance. Cmake here generates their 
artifacts just right under sources, and not under ${CMAKE_CURRENT_BINARY_DIR}. Will
look how we could improve things, making it cleaner. 

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices
  2021-07-31  6:30       ` Oleg Babin via Tarantool-patches
@ 2021-07-31  9:31         ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-31  9:31 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

: > : From: Oleg Babin<olegrok@tarantool.org>
: > : Subject: Re: [Tarantool-patches] [PATCH resend v2 06/11] box, datetime:
: > : datetime comparison for indices
: > :
: > : Thanks for your patch. One question below.
: > :
: > :
: > : Also update with arithmetic operations doesn't work.
: > :
: > :
: > : ```
: > :
: > : tarantool> box.space.dt:update({v}, {{'+', 2, 3}})
: > : ---
: > : - error: 'Argument type in operation ''+'' on field 2 does not match
: > : field type: expected
: > :      a number'
: > : ...
: > :
: > : tarantool> box.space.dt:update({v}, {{'+', 2, dt.week(1)}})
: > : ---
: > : - error: unsupported Lua type 'cdata'
: > : ...
: > :
: > : ```
: >
: > Oh, shit, I didn't take those operations into considerations.
: > Do they work similarly with decimal?
: 
: Looks like so. Moreover it seems we should support intervals as separate
: type.
: 
: Currently the single way to update is get + sum + put/update (with =).
: 
: But I suppose native update will be needed anyway. At lest for SQL.
: 
: Will be it able to perform some operations with intervals and timestamps?


The original plan was to not persist intervals in the 1st iteration
of implementation and store only full datetime values. Even for 
SQL I did not plan to allow schemas with interval as field type. 
INTERVALs in SQL expected to be present at run-time only at the 
Expressions, but then persisted as calculated datetime values.

But, those box updates with intervals looks interesting, and we have
to support, eventually. I don't know though whether it would
require full interval support or the current partial will
be enough? 

In any case - smells like we need to open ticket for that. Could
you please do that with more details?

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua
  2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
@ 2021-07-31  9:54         ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-31  9:54 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime:
: unixtime, timestamp setters in datetime.lua
: 
: 
: On 31.07.2021 02:11, Timur Safin wrote:
: > Hello Oleg,
: >
: > :
: > :
: > : Unixtime and timestamp is great but they loss precision. I think it
: > : should be possible
: > :
: > : go get timestamp with nanoseconds precision since datetime has
: > : nanoseconds precision.
: >
: > I'm not quite get it. Do you want to have nanoseconds exported via
: newindex?
: > For direct assignments? It's not useful because of available .nsec field.
: > Or may be you meant something else?
: >
: > :
: > :
: > : Also still it's hard how to convert timestamp to datetime value back.
: >
: > Timestamp is something secs.nsec, what looks complicated? I'm confused.
: > Could you please elaborate?
: 
: I expected something like.
: 
: tarantool> clock.time64()
: ---
: - 1627712632369020000
: ...
: 
: 
: This timestamp is in nanoseconds. It's unsigned long long value.
: 
: I tried to do something for datetime...
: 
: sec * 1e9 + nsec, but sec is null while nsec is not:
: 
: ```
: 
: tarantool> dt.sec
: ---
: - null
: ...
: 
: ```
: 
: 
: timestamp + nsec works in quite strage way:
: 
: ```
: 
: tarantool> dt.timestamp * (1e9 * 1ULL) + dt.nsec
: ---
: - 1627712485108074000
: ...
: 
: ```
: 
: (I need to cast to ULL here to avoid precision lost).
: 
: But maybe my question is not relevant. There is a way to get nanoseconds
: timestamp.
: 
: I just need to write some simple helper for my app. It's ok.

There are nanoseconds / microseconds / milliseconds getters in datetime
object, which operate on the original uint64_t. Do you want them to make
read-write, not read-only?

tarantool> T = date '1970-01-01'
---
...

tarantool> T.timestamp
---
- 0
...

tarantool> T.nanoseconds
---
- 0
...

tarantool> T2000 = date '2000-01-01'
---
...

tarantool> T.nanoseconds
---
- 0
...

tarantool> T2000.timestamp
---
- 946684800
...

tarantool> T2000.nanoseconds
---
- 946684800000000000
...

tarantool> T2000.secs
---
- 946684800
...

tarantool> ffi = require 'ffi'
---
...

tarantool> ffi.typeof(T2000.nanoseconds)
---
- ctype<int64_t>
...

Regards,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
@ 2021-07-31 16:51         ` Timur Safin via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-07-31 16:51 UTC (permalink / raw)
  To: 'Oleg Babin', v.shpilevoy; +Cc: tarantool-patches

: From: Oleg Babin <olegrok@tarantool.org>
: Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in
: module datetime
: 
: Thanks for your answers.
: 
: On 30.07.2021 22:00, Timur Safin wrote:
: > Ok, here is a selection list which I've seen:
: > - decimal is in LuaC;
: > - uuid is in ffi;
: >
: > It looked equally acceptable and I've selected the one which is
: > more convenient - via ffi. Now, you've pointed me to the probable
: > performance issues we have with ffi, which are not there with LuaC.
: > I believe they have been fixed with that Mike fix for NYI in
: > structure allocations (after Sergos dirty fix) and once we will land
: > this patch to our local fork those performance issues for structures
: > allocations will gone.
: >
: > In any case, 1st version of module is not yet a proper moment to start
: > to worry about performance, more rather about clarity and quality of API.
: > We need to return to benchmarking of this code soon after code freeze
: > lifted.
: 
: I don't know why uuid is implemented via FFI. Uuid as module was
: introduced at least 8 yeard ago.
: 
: Probably there were supporters of the theory that FFI/LuaJIT
: implementation could be fast.
: 
: But currently we can see something like
: https://github.com/luafun/luafun/issues/31#issuecomment-277508076
: 
: and rewrite netbox in C.

"I feel your pain, bro!" :)

[here follows seemingly irrelevant speech, but actually it's on topic]

I knew you have a plenty of LuaJIT experience, and sometimes, 
you might want to throw it out altogether. All that pain of debugging JIT
performance issues. Woodoo and black magics of LuaJIT debugging. General 
unpredictability of Mike responses. And even lack of convenient tooling
here and there. I do understand how all this could exhaust eventually, 
and make to loss of believe in LuaJIT usefulness.

But we should not. We have skilled LuaJIT team here, which could handle
all sorts of problems. We have huge amount of Lua code, and have no any
reason to start the process of getting rid of it. Yes, there are cases 
which might hit some performance degradations, but usually there are 
reasonable workarounds. (Whom am I teaching here? You are much more aware
of all these tricks than I am!)

Having our background and grownup skillset, there are actually no much 
reasons to try to drastically reduce amount of LuaJIT usage, because 
we continue to be used as LuaJIT-based application server for all 
foreseeable future.
[/end of seemingly irrelevant speech]

To make this story short, I believe there is no much point to start 
rewrite of this module in LuaC before we have it published and landed to 
the Tarantool code-base. We need to make it feature complete first, then 
"blazingly fast". In that order.

: 
: And it's not about patches in LuaJIT about structure initialization. You
: can easily cherry-pick it into
: 
: our fork and show that FFI is really better than Lua C API. It's about
: an approach. We saw benchmarks
: 
: when decimal was introduced but currently we didn't see anything.
: 
: > : > +    local y, M, d, ymd
: > : > +    y, M, d, ymd = 0, 0, 0, false
: > : > +
: > : > +    local h, m, s, frac, hms
: > : > +    h, m, s, frac, hms = 0, 0, 0, 0, false
: > : > +
: > : > +    local dt = 0
: > : > +
: > : > +    for key, value in pairs(o) do
: > : > +        local handlers = {
: > :
: > : Here you recreate handlers for each iteration of the loop. I think it
: > : should be reworked.
: >
: > Yes, let me check how better will be if this handlers array will be
: invariant
: > in the loop, and not created each iteration.
: >
: > I still prefer this tag to function approach because of a better clarity,
: > but I need to verify in benchmark whether it provides acceptable
: performance.
: >
: > I do not expect thought that this attribute object creation approach would
: become
: > a bootleneck in any case and might be on a critical path anywhere (I'd
: rather expect
: > it for datetime objects created after parsing of massive bunch of log
: text),
: > but need to look into timings we would have with invariant table.
: >
: > :
: > : Currenlty it's quite slow I think even if-elseif-else branches will work
: > : faster without
: > :
: > : creating redundant GC objects.
: >
: > Will see ...
: 
: I'll simplify your task a bit:
: 
: local clock = require('clock')
: 
: local mt_fun = function(_, key)
:      local mt = {
:          ['key'] = function()
:              return 'value'
:          end
:      }
:      return mt[key] ~= nil and mt[key]()
: end
: 
: local mt_table = {
:      ['key'] = function()
:          return 'value'
:      end
: }
: 
: local t1 = setmetatable({}, {__index = mt_fun})
: local t2 = setmetatable({}, {__index = mt_table})
: local _
: 
: local start = clock.time()
: for _ = 1, 1e6 do
:      _ = t1['key']
: end
: print('function', clock.time() - start)
: 
: collectgarbage()
: collectgarbage()
: 
: local start = clock.time()
: for _ = 1, 1e6 do
:      _ = t2['key']
: end
: print('table', clock.time() - start)
: 
: 
: tarantool test.lua
: function    0.14332509040833
: table    0.0002901554107666
: 
: 
: The difference about x500 times on my Mac.

Thanks! That's convincing! Thanks for this experiment,
much appreciated!

Interesting observation, though, that once we remove local 
mt table creation from inside of mt_fun to the module scope, making
it invariant for the cycle. The difference is not that much
huge. 

Debug build on WSL::
- your original version:

```
19:22 $ ../build/src/tarantool bench-index.lua 
function        0.16711115837097
table   0.00081968307495117
```

- modified version with invariant mt table:

```
19:25 $ ../build/src/tarantool bench-index.lua 
function        0.00028538703918457
table   0.00028634071350098
```

```lua
local clock = require('clock')

local mt = {
    ['key'] = function()
        return 'value'
    end
}
local mt_fun = function(_, key)
    -- return mt[key]()
    return mt[key] ~= nil and mt[key]()
end

local mt_table = {
    ['key'] = function()
        return 'value'
    end
}

local t1 = setmetatable({}, {__index = mt_fun})
local t2 = setmetatable({}, {__index = mt_table})
local _

local start = clock.time()
for _ = 1, 1e6 do
     _ = t1['key']
end
print('function', clock.time() - start)

collectgarbage()
collectgarbage()

local start = clock.time()
for _ = 1, 1e6 do
     _ = t2['key']
end
print('table', clock.time() - start)
```

So once again we have learned that it's not a good idea
to create invariant table inside of loop, and simple 
loop invariant removal might change things dramatically.

Thanks for the lesson!
: 
: > : > +end
: > : > +
: > : > +local function strftime(fmt, o)
: > : > +    assert(ffi.typeof(o) == datetime_t)
: > : > +    local sz = 50
: > : Why 50?
: >
: > Good question, for ISO-8601 formats, for normal date range
: > (from 0000 till 9999 years) it's more like closer to 40, but
: > well, there will be various timezones and all crazy kinds of
: > input format combinations, so we need to either be precautious
: > with larger sizes allocations.
: 
: On the one side you are right. But if user want to format string with
: length more than 50
: 
: it will be silently truncated.

No, no, no, please see my 2 pass variant suggested below. We will ask
`strftime` first to calculate the length for us (passing NULL as buffer
pointer), then allocate, and fill the allocated buffer with content 
the 2nd pass. 

: 
: 
: ```
: 
: tarantool> datetime.strftime('%d' .. string.rep('content', 50),
: datetime.new())
: ---
: - 01contentcontentcontentcontentcontentcontentconten
: ...
: 
: ```
: 
: I'm not sure that anybody will use strftime in such way but it's strange.
: 
: Maybe you could use preallocated buffer for short strings and allocate
: huge if
: 
: it's not enough. Some kind of optimistic approach.
: 
: 
: > Or probably rely on glibc convention (which is apparently not in
: > POSIX/SUSV) with 2 passes approach:
: > 1. call with NULL so it will return size of required buffer;
: > 2. then, after allocation, with the adjusted allocated buffer;
: >
: > https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-
: Time.html
: > https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html
: >
: > Something like that (beware of inline patch):
: >
: > ----------------------------------------------
: > local function strftime(fmt, o)
: >       check_date(o, "datetime.strftime(fmt, date)")
: > -    local sz = 50
: > -    local buff = ffi.new('char[?]', sz)
: >       local p_tm = datetime_to_tm_ptr(o)
: > -    native.strftime(buff, sz, fmt, p_tm)
: > +    local sz = builtin.strftime(nil, 1024, fmt, p_tm)
: > +    local buff = ffi.new('char[?]', sz + 1)
: > +    builtin.strftime(buff, sz, fmt, p_tm)
: >       return ffi.string(buff)
: >   end
: > ----------------------------------------------

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-07-30 15:39     ` Timur Safin via Tarantool-patches
@ 2021-08-01 17:01       ` Vladislav Shpilevoy via Tarantool-patches
  2021-08-01 20:23         ` Timur Safin via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-08-01 17:01 UTC (permalink / raw)
  To: Timur Safin; +Cc: tarantool-patches

Hi! Thanks for the fixes! Although I would prefer you
sending updates right under each non-trivial comment and
pushing the changes. Otherwise I can't check if you really
fixed the comments and if the fixes are better than it was
before.

On 30.07.2021 17:39, Timur Safin wrote:
> : From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
> : Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime
> : 
> : Thanks for the patch!
> : 
> : This is a bad question to ask now, but why did you choose a dead library
> : from 2015 with just 25 stars, without any tests, and without any README as
> : a basic block for our datetime types? Is this the correct link I am
> : using - https://github.com/chansen/c-dt?

In all your responses the previous email's text is prefixed with ':' sign
unlike it is done in other people's responses. In my email client it turns the
email into a mess where I can't see where is the new text. Could you possibly
start using '>' prefix?

> : > + */
> : > +struct datetime_t {
> : > +	int64_t secs;	///< seconds since epoch
> : > +	int32_t nsec;	///< nanoseconds if any
> : > +	int32_t offset; ///< offset in minutes from GMT
> : 
> : 6. We never use // nor /// nor ///< nor we write comments
> : on the same line as the code (except for old legacy code we
> : inherited from sqlite and some super old tarantool code.
> 
> BTW, why? Single line comment // is the legal C construction 
> since C99 (and always have been acceptable by Gnu C and major 
> other compilers). 
> 
> Also did you know that ///< is doxygen construction for 
> documentation of a struct member? (Asking, just to make sure)

The fact that something is a legal C construction does not
mean it fits our code style. For doxygen in our rules we have
/** as a comment opening.

> : 
> : The same for datetime_interval_t.
> : 
> : > +};
> : > +
> : > +/**
> : > + * Date/time delta structure
> : > + */
> : > +struct datetime_interval_t {
> : > +	int64_t secs; ///< relative seconds delta
> : > +	int32_t nsec; ///< nanoseconds delta
> : > +};
> : > +
> : 
> : 7. Why do you need this file? It is not included anywhere.
> : And you don't need to define the structs in C if you are
> : using them in Lua only. You can just define them in Lua
> : using ffi.cdef like it is done in some other places.
> 
> I'll need to use it in C very soon, once I get to SQL parser
> INTERVAL 'xx' support.

You will need to introduce it in the same commit in which you
will use it. We do not split patches into commits just to make
them smaller. The split needs to make sense in terms of
atomicity.

> : > +local datetime_mt = {
> : > +    -- __tostring = datetime_tostring,
> : 
> : 16. Why is it commented out? Is it even tested then?
> 
> It's implemented in the later patch in patchset series.

Well, then it should not be here in this commit.

> : > +    microseconds = function(self)
> : > +        return tonumber(self.secs*1e6 + self.nsec*1e3)
> : 
> : 18. Please, add whitespaces around binary operators.
> 
> Done! It was already such in the later commits which were part
> of in patchset. [Yes, apparently I need to squash few more commits,
> as Oleg Babin has asked elsewhere]

It is not just about squashes. It is only about atomicity (and
as a consequence - review simplicity). You can't just squash all
the commits into one. You will need to change the older commits
to make them right, then rebase the newer commits.

> : 
> : 19. Why do you do tonumber() for all results? You are loosing
> : precision for big values. The same in all the places where you
> : use tonumber().
> 
> To be able to use it in places which expect numbers like %d in
> string.format.

That does not look like a good reason - you loose precision just
to simplify the logs. Better make the logs use tonumber() than
make the functions return imprecise results.

> Though at the end of day I've came up to the version
> where I convert to Lua numbers not all (potentially fraction point) 
> values, but with exception of nanoseconds / microseconds / milliseconds
> (which return integer values). 
> 
> : 
> : > +    end,
> : > +
> : 
> : 21. Extra empty line.
> 
> Yup, but later, well, you know ...

No, I couldn't understand. What 'later'?

> : 
> : > +}
> : > +
> : > +local interval_mt = {
> : > +    -- __tostring = interval_tostring,
> : 
> : 22. Ditto. Why is it commented out and not tested?
> 
> Tostring conversion was implemented later in a few 
> commits. One for datetime stringization, and another 
> one for interval.

Then it should be in the corresponding commit.

> : 
> : > +    __serialize = interval_serialize,
> : > +    __eq = datetime_eq,
> : > +    __lt = datetime_lt,
> : > +    __le = datetime_le,
> : > +}
> : > +
> : > +local function datetime_new_raw(secs, nsec, offset)
> : > +    local dt_obj = ffi.new(datetime_t)
> : > +    dt_obj.secs = secs
> : > +    dt_obj.nsec = nsec
> : > +    dt_obj.offset = offset
> : > +    return dt_obj
> : > +end
> : > +
> : > +local function local_rd(o)
> : 
> : 23. What is 'rd' and 'dt'?
> 
> `dt` is type - data type from c-dt (alias to signed integer).
> Which shows a number of dates since Rata Die date (0001-01-01).
> It's signed 32-bit value, thus could represent huge range before 
> 0001-01-01 as well. 

You didn't say what is 'rd'.

> : 
> : > +            end,
> : 
> : 27. The usage of closures here might render all your FFI efforts
> : useless, killing the performance. Please, try to define all
> : methods of all objects only once in the root namespace of the
> : file. Closure usage might be justified only for rarely created
> : long living heavy objects like netbox.
> 
> Agreed that creating closures in each iteration of a loop was
> bad idea (as Oleg Babin has already pin-pointed it elsewhere).

It is not only about in the loop. It is about having closures at all.
I don't see why are they necessary here. It is not a complex state
machine like netbox which needs closures. Please, rework the function
not to have any closures in them.

> : > +
> : > +--[[
> : > +    Basic      Extended
> : > +    20121224   2012-12-24   Calendar date   (ISO 8601)
> : > +    2012359    2012-359     Ordinal date    (ISO 8601)
> : > +    2012W521   2012-W52-1   Week date       (ISO 8601)
> : > +    2012Q485   2012-Q4-85   Quarter date
> : > +]]
> : > +
> : > +local function parse_date(str)
> : > +    local dt = ffi.new('dt_t[1]')
> : > +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
> : > +    assert(rc > 0)
> : 
> : 29. Er ... And what if I pass a string in a wrong format? Users
> : should not get assertions.
> : 
> : 	tarantool> datetime.parse_date('121r1 r1r 13 13 13')
> : 	---
> : 	- error: 'builtin/datetime.lua:397: assertion failed!'
> : 	...
> : 
> : This shall not ever happen. Would you mind implementing proper
> : error handling. The same in all the places where the errors are
> : ignored.
> 
> 
> Now, I've reworked approach how they handle errors (you could see
> It in later commits in the patch-set), and now it returns tuple - 
> - parsed value
> - and the number of accepted input characters.
> 
> 	tarantool> date.parse_date('121r1 r1r 13 13 13')
> 	---
> 	- null
> 	- 0
> 	...
> 
> Sometimes, while parsing composed strings using these partial
> methods (parse_date / parse_time / parse_time_zone) we need to
> know the number of symbols we have successfully accepted. That's
> why there is seconds value in returned tuple.
> 
> Please give me know if there is better/more idiomatic approach
> used for such scenarios?

In the new code which is a part of the core we use exceptions.
https://github.com/tarantool/doc/issues/1727

But I see it is still not documented, so for me 'null, pos' look
fine so far.

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-08-01 17:01       ` Vladislav Shpilevoy via Tarantool-patches
@ 2021-08-01 20:23         ` Timur Safin via Tarantool-patches
  2021-08-04 23:57           ` Vladislav Shpilevoy via Tarantool-patches
  0 siblings, 1 reply; 46+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-01 20:23 UTC (permalink / raw)
  To: 'Vladislav Shpilevoy'; +Cc: tarantool-patches

Hello Vlad,

Ok, ok, despite of 20 years (!!) long personal tradition (which I've 
learned from Larry Wall email style, e.g. https://www.nntp.perl.org/group/perl.perl5.porters/1999/09/msg444.html),
I'm finally changing my citation style back to the usual `>`, so your
mail client will be happy again. 

> From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
> Subject: Re: [PATCH resend v2 02/11] lua: built-in module datetime
> 
> Hi! Thanks for the fixes! Although I would prefer you
> sending updates right under each non-trivial comment and
> pushing the changes. Otherwise I can't check if you really
> fixed the comments and if the fixes are better than it was
> before.

I understand, but hard to follow (at least yet for me) unless
I've reshuffled all changes back to the reasonable set of changes,
which I'm ok to show indefinitely attached to the GitHub issue page.
Personally I hate when thousands of temporary changes got attached 
forever to the issue, it's hard then to find the relevant ones.

> In all your responses the previous email's text is prefixed with ':' sign
> unlike it is done in other people's responses. In my email client it turns
> the
> email into a mess where I can't see where is the new text. Could you
> possibly
> start using '>' prefix?

Now, I've switched. 

> 
> > : > + */
> > : > +struct datetime_t {
> > : > +	int64_t secs;	///< seconds since epoch
> > : > +	int32_t nsec;	///< nanoseconds if any
> > : > +	int32_t offset; ///< offset in minutes from GMT
> > :
> > : 6. We never use // nor /// nor ///< nor we write comments
> > : on the same line as the code (except for old legacy code we
> > : inherited from sqlite and some super old tarantool code.
> >
> > BTW, why? Single line comment // is the legal C construction
> > since C99 (and always have been acceptable by Gnu C and major
> > other compilers).
> >
> > Also did you know that ///< is doxygen construction for
> > documentation of a struct member? (Asking, just to make sure)
> 
> The fact that something is a legal C construction does not
> mean it fits our code style. For doxygen in our rules we have
> /** as a comment opening.

Ok, ok, I've switched back to /**< ... */ here

> 
> > :
> > : The same for datetime_interval_t.
> > :
> > : > +};
> > : > +
> > : > +/**
> > : > + * Date/time delta structure
> > : > + */
> > : > +struct datetime_interval_t {
> > : > +	int64_t secs; ///< relative seconds delta
> > : > +	int32_t nsec; ///< nanoseconds delta
> > : > +};
> > : > +
> > :
> > : 7. Why do you need this file? It is not included anywhere.
> > : And you don't need to define the structs in C if you are
> > : using them in Lua only. You can just define them in Lua
> > : using ffi.cdef like it is done in some other places.
> >
> > I'll need to use it in C very soon, once I get to SQL parser
> > INTERVAL 'xx' support.
> 
> You will need to introduce it in the same commit in which you
> will use it. We do not split patches into commits just to make
> them smaller. The split needs to make sense in terms of
> atomicity.
> 

Ok


> > : > +local datetime_mt = {
> > : > +    -- __tostring = datetime_tostring,
> > :
> > : 16. Why is it commented out? Is it even tested then?
> >
> > It's implemented in the later patch in patchset series.
> 
> Well, then it should not be here in this commit.
> 
> > : > +    microseconds = function(self)
> > : > +        return tonumber(self.secs*1e6 + self.nsec*1e3)
> > :
> > : 18. Please, add whitespaces around binary operators.
> >
> > Done! It was already such in the later commits which were part
> > of in patchset. [Yes, apparently I need to squash few more commits,
> > as Oleg Babin has asked elsewhere]
> 
> It is not just about squashes. It is only about atomicity (and
> as a consequence - review simplicity). You can't just squash all
> the commits into one. You will need to change the older commits
> to make them right, then rebase the newer commits.

I see.

BTW, what is the simplest approach here you could recommend?
I mean what tools do you use for shuffling code between version layers?
How to make it simple enough and painless? (I'd love to see training 
Here - I know how to do it the harder way in interactive rebase, but
I suspect there are some helpers, which I missed, which allow to 
Make it easy way. If any?)

> 
> > :
> > : 19. Why do you do tonumber() for all results? You are loosing
> > : precision for big values. The same in all the places where you
> > : use tonumber().
> >
> > To be able to use it in places which expect numbers like %d in
> > string.format.
> 
> That does not look like a good reason - you loose precision just
> to simplify the logs. Better make the logs use tonumber() than
> make the functions return imprecise results.

FWIW, maximal integer value, which may be precisely saved to Lua 
Number, is ~9e15, which, if being assigned to datetime `.secs`, 
could represent this date:

	tarantool> T = date.new()
	---
	...
	
	tarantool> M = 9e15
	---
	...
	
	tarantool> M
	---
	- 9000000000000000
	...
	
	tarantool> T.secs = M
	---
	...
	
	tarantool> T
	---
	- 2979311-04-03T16:00Z
	...

It's _well beyond_ currently planned supported date range.

So better ergonomics while working with those calculated attributed
does deserve some extra cast upon return of a value. We do not loss
any precision if we work with expected date range.

At the moment the claimed supported date range is [0000...9999] for
Lua runtime, and is narrower [208BC...4147] if you need to store value,
and keep it correctly sorted in index.

> 
> > Though at the end of day I've came up to the version
> > where I convert to Lua numbers not all (potentially fraction point)
> > values, but with exception of nanoseconds / microseconds / milliseconds
> > (which return integer values).
> >
> > :
> > : > +    end,
> > : > +
> > :
> > : 21. Extra empty line.
> >
> > Yup, but later, well, you know ...
> 
> No, I couldn't understand. What 'later'?

That was referring to some later commit. 

> 
> > :
> > : > +}
> > : > +
> > : > +local interval_mt = {
> > : > +    -- __tostring = interval_tostring,
> > :
> > : 22. Ditto. Why is it commented out and not tested?
> >
> > Tostring conversion was implemented later in a few
> > commits. One for datetime stringization, and another
> > one for interval.
> 
> Then it should be in the corresponding commit.

Yup.

> 
> > :
> > : > +    __serialize = interval_serialize,
> > : > +    __eq = datetime_eq,
> > : > +    __lt = datetime_lt,
> > : > +    __le = datetime_le,
> > : > +}
> > : > +
> > : > +local function datetime_new_raw(secs, nsec, offset)
> > : > +    local dt_obj = ffi.new(datetime_t)
> > : > +    dt_obj.secs = secs
> > : > +    dt_obj.nsec = nsec
> > : > +    dt_obj.offset = offset
> > : > +    return dt_obj
> > : > +end
> > : > +
> > : > +local function local_rd(o)
> > :
> > : 23. What is 'rd' and 'dt'?
> >
> > `dt` is type - data type from c-dt (alias to signed integer).
> > Which shows a number of dates since Rata Die date (0001-01-01).
> > It's signed 32-bit value, thus could represent huge range before
> > 0001-01-01 as well.
> 
> You didn't say what is 'rd'.

Rata Die date.

> 
> > :
> > : > +            end,
> > :
> > : 27. The usage of closures here might render all your FFI efforts
> > : useless, killing the performance. Please, try to define all
> > : methods of all objects only once in the root namespace of the
> > : file. Closure usage might be justified only for rarely created
> > : long living heavy objects like netbox.
> >
> > Agreed that creating closures in each iteration of a loop was
> > bad idea (as Oleg Babin has already pin-pointed it elsewhere).
> 
> It is not only about in the loop. It is about having closures at all.
> I don't see why are they necessary here. It is not a complex state
> machine like netbox which needs closures. Please, rework the function
> not to have any closures in them.

datetime_new() is a good example of a function using closures
called via tag attributes names, where I do need to have access 
to the locals from outer function, i.e. easy_way, ymd, hms variables
which defines submode. 
Converting this code to the series of ifs may make dispatch
slower (and less elegant), and after change which we have discussed 
with Oleg Babin elsewhere, when table creation as loop invariant
moved to the module scope of above of a loop, then performance
impact becoming negligible. While still keeping code ergonomics.

Here is the deal - let me send fully feature complete patchset
still using closures at a couple of places, we proceed with review
before feature freeze, then we return back to performance/closure
optimizations, but as a follow-up commit?

> 
> > : > +
> > : > +--[[
> > : > +    Basic      Extended
> > : > +    20121224   2012-12-24   Calendar date   (ISO 8601)
> > : > +    2012359    2012-359     Ordinal date    (ISO 8601)
> > : > +    2012W521   2012-W52-1   Week date       (ISO 8601)
> > : > +    2012Q485   2012-Q4-85   Quarter date
> > : > +]]
> > : > +
> > : > +local function parse_date(str)
> > : > +    local dt = ffi.new('dt_t[1]')
> > : > +    local rc = cdt.dt_parse_iso_date(str, #str, dt)
> > : > +    assert(rc > 0)
> > :
> > : 29. Er ... And what if I pass a string in a wrong format? Users
> > : should not get assertions.
> > :
> > : 	tarantool> datetime.parse_date('121r1 r1r 13 13 13')
> > : 	---
> > : 	- error: 'builtin/datetime.lua:397: assertion failed!'
> > : 	...
> > :
> > : This shall not ever happen. Would you mind implementing proper
> > : error handling. The same in all the places where the errors are
> > : ignored.
> >
> >
> > Now, I've reworked approach how they handle errors (you could see
> > It in later commits in the patch-set), and now it returns tuple -
> > - parsed value
> > - and the number of accepted input characters.
> >
> > 	tarantool> date.parse_date('121r1 r1r 13 13 13')
> > 	---
> > 	- null
> > 	- 0
> > 	...
> >
> > Sometimes, while parsing composed strings using these partial
> > methods (parse_date / parse_time / parse_time_zone) we need to
> > know the number of symbols we have successfully accepted. That's
> > why there is seconds value in returned tuple.
> >
> > Please give me know if there is better/more idiomatic approach
> > used for such scenarios?
> 
> In the new code which is a part of the core we use exceptions.
> https://github.com/tarantool/doc/issues/1727
> 
> But I see it is still not documented, so for me 'null, pos' look
> fine so far.

Nice!

I still need a time top move portions of a code between different 
patches in a new version of patchset, but I definitely plan to send
it later tonight. Stay tuned!

Thanks,
Timur


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime
  2021-08-01 20:23         ` Timur Safin via Tarantool-patches
@ 2021-08-04 23:57           ` Vladislav Shpilevoy via Tarantool-patches
  0 siblings, 0 replies; 46+ messages in thread
From: Vladislav Shpilevoy via Tarantool-patches @ 2021-08-04 23:57 UTC (permalink / raw)
  To: Timur Safin; +Cc: tarantool-patches

>>> : > +local datetime_mt = {
>>> : > +    -- __tostring = datetime_tostring,
>>> :
>>> : 16. Why is it commented out? Is it even tested then?
>>>
>>> It's implemented in the later patch in patchset series.
>>
>> Well, then it should not be here in this commit.
>>
>>> : > +    microseconds = function(self)
>>> : > +        return tonumber(self.secs*1e6 + self.nsec*1e3)
>>> :
>>> : 18. Please, add whitespaces around binary operators.
>>>
>>> Done! It was already such in the later commits which were part
>>> of in patchset. [Yes, apparently I need to squash few more commits,
>>> as Oleg Babin has asked elsewhere]
>>
>> It is not just about squashes. It is only about atomicity (and
>> as a consequence - review simplicity). You can't just squash all
>> the commits into one. You will need to change the older commits
>> to make them right, then rebase the newer commits.
> 
> I see.
> 
> BTW, what is the simplest approach here you could recommend?
> I mean what tools do you use for shuffling code between version layers?
> How to make it simple enough and painless? (I'd love to see training 
> Here - I know how to do it the harder way in interactive rebase, but
> I suspect there are some helpers, which I missed, which allow to 
> Make it easy way. If any?)

My main tools for doing review fixes per-commit are:
'git rebase -i', 'git commit --amend', 'git add -u'.

I couldn't find anything easier. These commands take a couple of
seconds to print and I doubt there is any tool doing it faster.

^ permalink raw reply	[flat|nested] 46+ messages in thread

end of thread, other threads:[~2021-08-04 23:58 UTC | newest]

Thread overview: 46+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-28 10:34 [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 01/11] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
2021-07-29 23:40   ` Vladislav Shpilevoy via Tarantool-patches
2021-07-31  9:22     ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime Timur Safin via Tarantool-patches
2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
2021-07-30 19:00     ` Timur Safin via Tarantool-patches
2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
2021-07-31 16:51         ` Timur Safin via Tarantool-patches
2021-07-29 23:36   ` Vladislav Shpilevoy via Tarantool-patches
2021-07-30 15:39     ` Timur Safin via Tarantool-patches
2021-08-01 17:01       ` Vladislav Shpilevoy via Tarantool-patches
2021-08-01 20:23         ` Timur Safin via Tarantool-patches
2021-08-04 23:57           ` Vladislav Shpilevoy via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 03/11] lua, datetime: datetime tests Timur Safin via Tarantool-patches
2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
2021-07-30 20:45     ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 04/11] lua, datetime: display datetime Timur Safin via Tarantool-patches
2021-07-29 18:55   ` Oleg Babin via Tarantool-patches
2021-07-30 21:48     ` Timur Safin via Tarantool-patches
2021-07-31  6:29       ` Oleg Babin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 05/11] box, datetime: add messagepack support for datetime Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 06/11] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
2021-07-29 18:56   ` Oleg Babin via Tarantool-patches
2021-07-30 22:18     ` Timur Safin via Tarantool-patches
2021-07-31  6:30       ` Oleg Babin via Tarantool-patches
2021-07-31  9:31         ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 07/11] lua, datetime: proper datetime encoding Timur Safin via Tarantool-patches
2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
2021-07-30 22:20     ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 08/11] lua, datetime: calculated attributes for datetimes Timur Safin via Tarantool-patches
2021-07-29 18:57   ` Oleg Babin via Tarantool-patches
2021-07-30 22:30     ` Timur Safin via Tarantool-patches
2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 09/11] lua, datetime: time intervals support Timur Safin via Tarantool-patches
2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
2021-07-30 22:58     ` Timur Safin via Tarantool-patches
2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
2021-07-31  9:20         ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 10/11] lua, datetime: unixtime, timestamp setters in datetime.lua Timur Safin via Tarantool-patches
2021-07-29 18:58   ` Oleg Babin via Tarantool-patches
2021-07-30 23:11     ` Timur Safin via Tarantool-patches
2021-07-31  6:31       ` Oleg Babin via Tarantool-patches
2021-07-31  9:54         ` Timur Safin via Tarantool-patches
2021-07-28 10:34 ` [Tarantool-patches] [PATCH resend v2 11/11] datetime: changelog for datetime module Timur Safin via Tarantool-patches
2021-07-29 18:55 ` [Tarantool-patches] [PATCH resend v2 00/11] Initial datetime support Oleg Babin via Tarantool-patches

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox