* [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 15:50 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime Timur Safin via Tarantool-patches
` (8 subsequent siblings)
9 siblings, 2 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
* Integrated chansen/c-dt parser as 3rd party module to the
Tarantool cmake build process;
* Points to tarantool/c-dt instead of original chansen/c-dt to
have easier build integration, because there are additional
commits which have integrated cmake support and have established
symbols renaming facilities (similar to those we see in xxhash
or icu).
We have to be able to rename externally public symbols to avoid
name clashes with 3rd party modules. We prefix c-dt symbols
in the Tarantool build with `tnt_` prefix;
* took special care that generated build artifacts not override
in-source files, but use different build/ directory.
* added datetime parsing unit tests, for literals - with and
without trailing timezones;
* also we check that strftime is reversible and produce consistent
results after roundtrip from/to strings;
* discovered the harder way that on *BSD/MacOSX `strftime()` format
`%z` outputs local time-zone if passed `tm_gmtoff` is 0.
This behaviour is different to that we observed on Linux, thus we
might have different execution results. Made test to not use `%z`
and only operate with normalized date time formats `%F` and `%T`
Part of #5941
---
.gitmodules | 3 +
CMakeLists.txt | 8 +
cmake/BuildCDT.cmake | 8 +
src/CMakeLists.txt | 3 +-
test/unit/CMakeLists.txt | 3 +-
test/unit/datetime.c | 223 ++++++++++++++++++++++++
test/unit/datetime.result | 358 ++++++++++++++++++++++++++++++++++++++
third_party/c-dt | 1 +
8 files changed, 605 insertions(+), 2 deletions(-)
create mode 100644 cmake/BuildCDT.cmake
create mode 100644 test/unit/datetime.c
create mode 100644 test/unit/datetime.result
create mode 160000 third_party/c-dt
diff --git a/.gitmodules b/.gitmodules
index f2f91ee72..aa3fbae4e 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -43,3 +43,6 @@
[submodule "third_party/xxHash"]
path = third_party/xxHash
url = https://github.com/tarantool/xxHash
+[submodule "third_party/c-dt"]
+ path = third_party/c-dt
+ url = https://github.com/tarantool/c-dt.git
diff --git a/CMakeLists.txt b/CMakeLists.txt
index e25b81eac..53c86f2a5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -571,6 +571,14 @@ endif()
# zstd
#
+#
+# Chritian Hanson c-dt
+#
+
+include(BuildCDT)
+libccdt_build()
+add_dependencies(build_bundled_libs cdt)
+
#
# Third-Party misc
#
diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
new file mode 100644
index 000000000..343fb1b99
--- /dev/null
+++ b/cmake/BuildCDT.cmake
@@ -0,0 +1,8 @@
+macro(libccdt_build)
+ set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/)
+ set(LIBCDT_LIBRARIES cdt)
+
+ file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
+ add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
+ ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
+endmacro()
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index adb03b3f4..97b0cb326 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -193,7 +193,8 @@ target_link_libraries(server core coll http_parser bit uri uuid swim swim_udp
# Rule of thumb: if exporting a symbol from a static library, list the
# library here.
set (reexport_libraries server core misc bitset csv swim swim_udp swim_ev
- shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES} ${CURL_LIBRARIES} ${XXHASH_LIBRARIES})
+ shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}
+ ${CURL_LIBRARIES} ${XXHASH_LIBRARIES} ${LIBCDT_LIBRARIES})
set (common_libraries
${reexport_libraries}
diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
index 5bb7cd6e7..31b183a8f 100644
--- a/test/unit/CMakeLists.txt
+++ b/test/unit/CMakeLists.txt
@@ -56,7 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
target_link_libraries(uuid.test uuid unit)
add_executable(random.test random.c core_test_utils.c)
target_link_libraries(random.test core unit)
-
+add_executable(datetime.test datetime.c)
+target_link_libraries(datetime.test cdt unit)
add_executable(bps_tree.test bps_tree.cc)
target_link_libraries(bps_tree.test small misc)
add_executable(bps_tree_iterator.test bps_tree_iterator.cc)
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
new file mode 100644
index 000000000..64c19dac4
--- /dev/null
+++ b/test/unit/datetime.c
@@ -0,0 +1,223 @@
+#include "dt.h"
+#include <assert.h>
+#include <stdint.h>
+#include <string.h>
+#include <time.h>
+
+#include "unit.h"
+
+static const char sample[] = "2012-12-24T15:30Z";
+
+#define S(s) {s, sizeof(s) - 1}
+struct {
+ const char * sz;
+ size_t len;
+} tests[] = {
+ S("2012-12-24 15:30Z"),
+ S("2012-12-24 15:30z"),
+ S("2012-12-24 15:30"),
+ S("2012-12-24 16:30+01:00"),
+ S("2012-12-24 16:30+0100"),
+ S("2012-12-24 16:30+01"),
+ S("2012-12-24 14:30-01:00"),
+ S("2012-12-24 14:30-0100"),
+ S("2012-12-24 14:30-01"),
+ S("2012-12-24 15:30:00Z"),
+ S("2012-12-24 15:30:00z"),
+ S("2012-12-24 15:30:00"),
+ S("2012-12-24 16:30:00+01:00"),
+ S("2012-12-24 16:30:00+0100"),
+ S("2012-12-24 14:30:00-01:00"),
+ S("2012-12-24 14:30:00-0100"),
+ S("2012-12-24 15:30:00.123456Z"),
+ S("2012-12-24 15:30:00.123456z"),
+ S("2012-12-24 15:30:00.123456"),
+ S("2012-12-24 16:30:00.123456+01:00"),
+ S("2012-12-24 16:30:00.123456+01"),
+ S("2012-12-24 14:30:00.123456-01:00"),
+ S("2012-12-24 14:30:00.123456-01"),
+ S("2012-12-24t15:30Z"),
+ S("2012-12-24t15:30z"),
+ S("2012-12-24t15:30"),
+ S("2012-12-24t16:30+01:00"),
+ S("2012-12-24t16:30+0100"),
+ S("2012-12-24t14:30-01:00"),
+ S("2012-12-24t14:30-0100"),
+ S("2012-12-24t15:30:00Z"),
+ S("2012-12-24t15:30:00z"),
+ S("2012-12-24t15:30:00"),
+ S("2012-12-24t16:30:00+01:00"),
+ S("2012-12-24t16:30:00+0100"),
+ S("2012-12-24t14:30:00-01:00"),
+ S("2012-12-24t14:30:00-0100"),
+ S("2012-12-24t15:30:00.123456Z"),
+ S("2012-12-24t15:30:00.123456z"),
+ S("2012-12-24t16:30:00.123456+01:00"),
+ S("2012-12-24t14:30:00.123456-01:00"),
+ S("2012-12-24 16:30 +01:00"),
+ S("2012-12-24 14:30 -01:00"),
+ S("2012-12-24 15:30 UTC"),
+ S("2012-12-24 16:30 UTC+1"),
+ S("2012-12-24 16:30 UTC+01"),
+ S("2012-12-24 16:30 UTC+0100"),
+ S("2012-12-24 16:30 UTC+01:00"),
+ S("2012-12-24 14:30 UTC-1"),
+ S("2012-12-24 14:30 UTC-01"),
+ S("2012-12-24 14:30 UTC-01:00"),
+ S("2012-12-24 14:30 UTC-0100"),
+ S("2012-12-24 15:30 GMT"),
+ S("2012-12-24 16:30 GMT+1"),
+ S("2012-12-24 16:30 GMT+01"),
+ S("2012-12-24 16:30 GMT+0100"),
+ S("2012-12-24 16:30 GMT+01:00"),
+ S("2012-12-24 14:30 GMT-1"),
+ S("2012-12-24 14:30 GMT-01"),
+ S("2012-12-24 14:30 GMT-01:00"),
+ S("2012-12-24 14:30 GMT-0100"),
+ S("2012-12-24 14:30 -01:00"),
+ S("2012-12-24 16:30:00 +01:00"),
+ S("2012-12-24 14:30:00 -01:00"),
+ S("2012-12-24 16:30:00.123456 +01:00"),
+ S("2012-12-24 14:30:00.123456 -01:00"),
+ S("2012-12-24 15:30:00.123456 -00:00"),
+ S("20121224T1630+01:00"),
+ S("2012-12-24T1630+01:00"),
+ S("20121224T16:30+01"),
+ S("20121224T16:30 +01"),
+};
+#undef S
+
+#define DIM(a) (sizeof(a) / sizeof(a[0]))
+
+/* p5-time-moment/src/moment_parse.c: parse_string_lenient() */
+static int
+parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
+ int32_t *op)
+{
+ size_t n;
+ dt_t dt;
+ char c;
+ int sod = 0, nanosecond = 0, offset = 0;
+
+ n = dt_parse_iso_date(str, len, &dt);
+ if (!n)
+ return 1;
+ if (n == len)
+ goto exit;
+
+ c = str[n++];
+ if (!(c == 'T' || c == 't' || c == ' '))
+ return 1;
+
+ str += n;
+ len -= n;
+
+ n = dt_parse_iso_time(str, len, &sod, &nanosecond);
+ if (!n)
+ return 1;
+ if (n == len)
+ goto exit;
+
+ if (str[n] == ' ')
+ n++;
+
+ str += n;
+ len -= n;
+
+ n = dt_parse_iso_zone_lenient(str, len, &offset);
+ if (!n || n != len)
+ return 1;
+
+exit:
+ *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
+ *np = nanosecond;
+ *op = offset;
+
+ return 0;
+}
+
+/* avoid introducing external datetime.h dependency -
+ just copy paste it for today
+*/
+#define SECS_PER_DAY 86400
+#define DT_EPOCH_1970_OFFSET 719163
+
+struct datetime {
+ double secs;
+ int32_t nsec;
+ int32_t offset;
+};
+
+static int
+local_rd(const struct datetime *dt)
+{
+ return (int)((int64_t)dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET;
+}
+
+static int
+local_dt(const struct datetime *dt)
+{
+ return dt_from_rdn(local_rd(dt));
+}
+
+struct tm *
+datetime_to_tm(struct datetime *dt)
+{
+ static struct tm tm;
+
+ memset(&tm, 0, sizeof(tm));
+ dt_to_struct_tm(local_dt(dt), &tm);
+
+ int seconds_of_day = (int64_t)dt->secs % 86400;
+ tm.tm_hour = (seconds_of_day / 3600) % 24;
+ tm.tm_min = (seconds_of_day / 60) % 60;
+ tm.tm_sec = seconds_of_day % 60;
+
+ return &tm;
+}
+
+static void datetime_test(void)
+{
+ size_t index;
+ int64_t secs_expected;
+ int32_t nanosecs;
+ int32_t ofs;
+
+ plan(355);
+ parse_datetime(sample, sizeof(sample) - 1,
+ &secs_expected, &nanosecs, &ofs);
+
+ for (index = 0; index < DIM(tests); index++) {
+ int64_t secs;
+ int rc = parse_datetime(tests[index].sz, tests[index].len,
+ &secs, &nanosecs, &ofs);
+ is(rc, 0, "correct parse_datetime return value for '%s'",
+ tests[index].sz);
+ is(secs, secs_expected, "correct parse_datetime output "
+ "seconds for '%s", tests[index].sz);
+
+ /* check that stringized literal produces the same date */
+ /* time fields */
+ static char buff[40];
+ struct datetime dt = {secs, nanosecs, ofs};
+ /* datetime_to_tm returns time in GMT zone */
+ struct tm * p_tm = datetime_to_tm(&dt);
+ size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
+ ok(len > 0, "strftime");
+ int64_t parsed_secs;
+ int32_t parsed_nsecs, parsed_ofs;
+ rc = parse_datetime(buff, len, &parsed_secs, &parsed_nsecs, &parsed_ofs);
+ is(rc, 0, "correct parse_datetime return value for '%s'", buff);
+ is(secs, parsed_secs,
+ "reversible seconds via strftime for '%s", buff);
+ }
+}
+
+int
+main(void)
+{
+ plan(1);
+ datetime_test();
+
+ return check_plan();
+}
diff --git a/test/unit/datetime.result b/test/unit/datetime.result
new file mode 100644
index 000000000..33997d9df
--- /dev/null
+++ b/test/unit/datetime.result
@@ -0,0 +1,358 @@
+1..1
+ 1..355
+ ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z'
+ ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z
+ ok 3 - strftime
+ ok 4 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 5 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 6 - correct parse_datetime return value for '2012-12-24 15:30z'
+ ok 7 - correct parse_datetime output seconds for '2012-12-24 15:30z
+ ok 8 - strftime
+ ok 9 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 10 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 11 - correct parse_datetime return value for '2012-12-24 15:30'
+ ok 12 - correct parse_datetime output seconds for '2012-12-24 15:30
+ ok 13 - strftime
+ ok 14 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 15 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 16 - correct parse_datetime return value for '2012-12-24 16:30+01:00'
+ ok 17 - correct parse_datetime output seconds for '2012-12-24 16:30+01:00
+ ok 18 - strftime
+ ok 19 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 20 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 21 - correct parse_datetime return value for '2012-12-24 16:30+0100'
+ ok 22 - correct parse_datetime output seconds for '2012-12-24 16:30+0100
+ ok 23 - strftime
+ ok 24 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 25 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 26 - correct parse_datetime return value for '2012-12-24 16:30+01'
+ ok 27 - correct parse_datetime output seconds for '2012-12-24 16:30+01
+ ok 28 - strftime
+ ok 29 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 30 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 31 - correct parse_datetime return value for '2012-12-24 14:30-01:00'
+ ok 32 - correct parse_datetime output seconds for '2012-12-24 14:30-01:00
+ ok 33 - strftime
+ ok 34 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 35 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 36 - correct parse_datetime return value for '2012-12-24 14:30-0100'
+ ok 37 - correct parse_datetime output seconds for '2012-12-24 14:30-0100
+ ok 38 - strftime
+ ok 39 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 40 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 41 - correct parse_datetime return value for '2012-12-24 14:30-01'
+ ok 42 - correct parse_datetime output seconds for '2012-12-24 14:30-01
+ ok 43 - strftime
+ ok 44 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 45 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 46 - correct parse_datetime return value for '2012-12-24 15:30:00Z'
+ ok 47 - correct parse_datetime output seconds for '2012-12-24 15:30:00Z
+ ok 48 - strftime
+ ok 49 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 50 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 51 - correct parse_datetime return value for '2012-12-24 15:30:00z'
+ ok 52 - correct parse_datetime output seconds for '2012-12-24 15:30:00z
+ ok 53 - strftime
+ ok 54 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 55 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 56 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 57 - correct parse_datetime output seconds for '2012-12-24 15:30:00
+ ok 58 - strftime
+ ok 59 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 60 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 61 - correct parse_datetime return value for '2012-12-24 16:30:00+01:00'
+ ok 62 - correct parse_datetime output seconds for '2012-12-24 16:30:00+01:00
+ ok 63 - strftime
+ ok 64 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 65 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 66 - correct parse_datetime return value for '2012-12-24 16:30:00+0100'
+ ok 67 - correct parse_datetime output seconds for '2012-12-24 16:30:00+0100
+ ok 68 - strftime
+ ok 69 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 70 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 71 - correct parse_datetime return value for '2012-12-24 14:30:00-01:00'
+ ok 72 - correct parse_datetime output seconds for '2012-12-24 14:30:00-01:00
+ ok 73 - strftime
+ ok 74 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 75 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 76 - correct parse_datetime return value for '2012-12-24 14:30:00-0100'
+ ok 77 - correct parse_datetime output seconds for '2012-12-24 14:30:00-0100
+ ok 78 - strftime
+ ok 79 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 80 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 81 - correct parse_datetime return value for '2012-12-24 15:30:00.123456Z'
+ ok 82 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456Z
+ ok 83 - strftime
+ ok 84 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 85 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 86 - correct parse_datetime return value for '2012-12-24 15:30:00.123456z'
+ ok 87 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456z
+ ok 88 - strftime
+ ok 89 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 90 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 91 - correct parse_datetime return value for '2012-12-24 15:30:00.123456'
+ ok 92 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456
+ ok 93 - strftime
+ ok 94 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 95 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 96 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01:00'
+ ok 97 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01:00
+ ok 98 - strftime
+ ok 99 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 100 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 101 - correct parse_datetime return value for '2012-12-24 16:30:00.123456+01'
+ ok 102 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456+01
+ ok 103 - strftime
+ ok 104 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 105 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 106 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01:00'
+ ok 107 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01:00
+ ok 108 - strftime
+ ok 109 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 110 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 111 - correct parse_datetime return value for '2012-12-24 14:30:00.123456-01'
+ ok 112 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456-01
+ ok 113 - strftime
+ ok 114 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 115 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 116 - correct parse_datetime return value for '2012-12-24t15:30Z'
+ ok 117 - correct parse_datetime output seconds for '2012-12-24t15:30Z
+ ok 118 - strftime
+ ok 119 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 120 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 121 - correct parse_datetime return value for '2012-12-24t15:30z'
+ ok 122 - correct parse_datetime output seconds for '2012-12-24t15:30z
+ ok 123 - strftime
+ ok 124 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 125 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 126 - correct parse_datetime return value for '2012-12-24t15:30'
+ ok 127 - correct parse_datetime output seconds for '2012-12-24t15:30
+ ok 128 - strftime
+ ok 129 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 130 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 131 - correct parse_datetime return value for '2012-12-24t16:30+01:00'
+ ok 132 - correct parse_datetime output seconds for '2012-12-24t16:30+01:00
+ ok 133 - strftime
+ ok 134 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 135 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 136 - correct parse_datetime return value for '2012-12-24t16:30+0100'
+ ok 137 - correct parse_datetime output seconds for '2012-12-24t16:30+0100
+ ok 138 - strftime
+ ok 139 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 140 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 141 - correct parse_datetime return value for '2012-12-24t14:30-01:00'
+ ok 142 - correct parse_datetime output seconds for '2012-12-24t14:30-01:00
+ ok 143 - strftime
+ ok 144 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 145 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 146 - correct parse_datetime return value for '2012-12-24t14:30-0100'
+ ok 147 - correct parse_datetime output seconds for '2012-12-24t14:30-0100
+ ok 148 - strftime
+ ok 149 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 150 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 151 - correct parse_datetime return value for '2012-12-24t15:30:00Z'
+ ok 152 - correct parse_datetime output seconds for '2012-12-24t15:30:00Z
+ ok 153 - strftime
+ ok 154 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 155 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 156 - correct parse_datetime return value for '2012-12-24t15:30:00z'
+ ok 157 - correct parse_datetime output seconds for '2012-12-24t15:30:00z
+ ok 158 - strftime
+ ok 159 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 160 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 161 - correct parse_datetime return value for '2012-12-24t15:30:00'
+ ok 162 - correct parse_datetime output seconds for '2012-12-24t15:30:00
+ ok 163 - strftime
+ ok 164 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 165 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 166 - correct parse_datetime return value for '2012-12-24t16:30:00+01:00'
+ ok 167 - correct parse_datetime output seconds for '2012-12-24t16:30:00+01:00
+ ok 168 - strftime
+ ok 169 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 170 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 171 - correct parse_datetime return value for '2012-12-24t16:30:00+0100'
+ ok 172 - correct parse_datetime output seconds for '2012-12-24t16:30:00+0100
+ ok 173 - strftime
+ ok 174 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 175 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 176 - correct parse_datetime return value for '2012-12-24t14:30:00-01:00'
+ ok 177 - correct parse_datetime output seconds for '2012-12-24t14:30:00-01:00
+ ok 178 - strftime
+ ok 179 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 180 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 181 - correct parse_datetime return value for '2012-12-24t14:30:00-0100'
+ ok 182 - correct parse_datetime output seconds for '2012-12-24t14:30:00-0100
+ ok 183 - strftime
+ ok 184 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 185 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 186 - correct parse_datetime return value for '2012-12-24t15:30:00.123456Z'
+ ok 187 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456Z
+ ok 188 - strftime
+ ok 189 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 190 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 191 - correct parse_datetime return value for '2012-12-24t15:30:00.123456z'
+ ok 192 - correct parse_datetime output seconds for '2012-12-24t15:30:00.123456z
+ ok 193 - strftime
+ ok 194 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 195 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 196 - correct parse_datetime return value for '2012-12-24t16:30:00.123456+01:00'
+ ok 197 - correct parse_datetime output seconds for '2012-12-24t16:30:00.123456+01:00
+ ok 198 - strftime
+ ok 199 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 200 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 201 - correct parse_datetime return value for '2012-12-24t14:30:00.123456-01:00'
+ ok 202 - correct parse_datetime output seconds for '2012-12-24t14:30:00.123456-01:00
+ ok 203 - strftime
+ ok 204 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 205 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 206 - correct parse_datetime return value for '2012-12-24 16:30 +01:00'
+ ok 207 - correct parse_datetime output seconds for '2012-12-24 16:30 +01:00
+ ok 208 - strftime
+ ok 209 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 210 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 211 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
+ ok 212 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
+ ok 213 - strftime
+ ok 214 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 215 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 216 - correct parse_datetime return value for '2012-12-24 15:30 UTC'
+ ok 217 - correct parse_datetime output seconds for '2012-12-24 15:30 UTC
+ ok 218 - strftime
+ ok 219 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 220 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 221 - correct parse_datetime return value for '2012-12-24 16:30 UTC+1'
+ ok 222 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+1
+ ok 223 - strftime
+ ok 224 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 225 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 226 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01'
+ ok 227 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01
+ ok 228 - strftime
+ ok 229 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 230 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 231 - correct parse_datetime return value for '2012-12-24 16:30 UTC+0100'
+ ok 232 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+0100
+ ok 233 - strftime
+ ok 234 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 235 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 236 - correct parse_datetime return value for '2012-12-24 16:30 UTC+01:00'
+ ok 237 - correct parse_datetime output seconds for '2012-12-24 16:30 UTC+01:00
+ ok 238 - strftime
+ ok 239 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 240 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 241 - correct parse_datetime return value for '2012-12-24 14:30 UTC-1'
+ ok 242 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-1
+ ok 243 - strftime
+ ok 244 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 245 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 246 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01'
+ ok 247 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01
+ ok 248 - strftime
+ ok 249 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 250 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 251 - correct parse_datetime return value for '2012-12-24 14:30 UTC-01:00'
+ ok 252 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-01:00
+ ok 253 - strftime
+ ok 254 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 255 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 256 - correct parse_datetime return value for '2012-12-24 14:30 UTC-0100'
+ ok 257 - correct parse_datetime output seconds for '2012-12-24 14:30 UTC-0100
+ ok 258 - strftime
+ ok 259 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 260 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 261 - correct parse_datetime return value for '2012-12-24 15:30 GMT'
+ ok 262 - correct parse_datetime output seconds for '2012-12-24 15:30 GMT
+ ok 263 - strftime
+ ok 264 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 265 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 266 - correct parse_datetime return value for '2012-12-24 16:30 GMT+1'
+ ok 267 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+1
+ ok 268 - strftime
+ ok 269 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 270 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 271 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01'
+ ok 272 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01
+ ok 273 - strftime
+ ok 274 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 275 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 276 - correct parse_datetime return value for '2012-12-24 16:30 GMT+0100'
+ ok 277 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+0100
+ ok 278 - strftime
+ ok 279 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 280 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 281 - correct parse_datetime return value for '2012-12-24 16:30 GMT+01:00'
+ ok 282 - correct parse_datetime output seconds for '2012-12-24 16:30 GMT+01:00
+ ok 283 - strftime
+ ok 284 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 285 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 286 - correct parse_datetime return value for '2012-12-24 14:30 GMT-1'
+ ok 287 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-1
+ ok 288 - strftime
+ ok 289 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 290 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 291 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01'
+ ok 292 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01
+ ok 293 - strftime
+ ok 294 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 295 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 296 - correct parse_datetime return value for '2012-12-24 14:30 GMT-01:00'
+ ok 297 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-01:00
+ ok 298 - strftime
+ ok 299 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 300 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 301 - correct parse_datetime return value for '2012-12-24 14:30 GMT-0100'
+ ok 302 - correct parse_datetime output seconds for '2012-12-24 14:30 GMT-0100
+ ok 303 - strftime
+ ok 304 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 305 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 306 - correct parse_datetime return value for '2012-12-24 14:30 -01:00'
+ ok 307 - correct parse_datetime output seconds for '2012-12-24 14:30 -01:00
+ ok 308 - strftime
+ ok 309 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 310 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 311 - correct parse_datetime return value for '2012-12-24 16:30:00 +01:00'
+ ok 312 - correct parse_datetime output seconds for '2012-12-24 16:30:00 +01:00
+ ok 313 - strftime
+ ok 314 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 315 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 316 - correct parse_datetime return value for '2012-12-24 14:30:00 -01:00'
+ ok 317 - correct parse_datetime output seconds for '2012-12-24 14:30:00 -01:00
+ ok 318 - strftime
+ ok 319 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 320 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 321 - correct parse_datetime return value for '2012-12-24 16:30:00.123456 +01:00'
+ ok 322 - correct parse_datetime output seconds for '2012-12-24 16:30:00.123456 +01:00
+ ok 323 - strftime
+ ok 324 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 325 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 326 - correct parse_datetime return value for '2012-12-24 14:30:00.123456 -01:00'
+ ok 327 - correct parse_datetime output seconds for '2012-12-24 14:30:00.123456 -01:00
+ ok 328 - strftime
+ ok 329 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 330 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 331 - correct parse_datetime return value for '2012-12-24 15:30:00.123456 -00:00'
+ ok 332 - correct parse_datetime output seconds for '2012-12-24 15:30:00.123456 -00:00
+ ok 333 - strftime
+ ok 334 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 335 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 336 - correct parse_datetime return value for '20121224T1630+01:00'
+ ok 337 - correct parse_datetime output seconds for '20121224T1630+01:00
+ ok 338 - strftime
+ ok 339 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 340 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 341 - correct parse_datetime return value for '2012-12-24T1630+01:00'
+ ok 342 - correct parse_datetime output seconds for '2012-12-24T1630+01:00
+ ok 343 - strftime
+ ok 344 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 345 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 346 - correct parse_datetime return value for '20121224T16:30+01'
+ ok 347 - correct parse_datetime output seconds for '20121224T16:30+01
+ ok 348 - strftime
+ ok 349 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 350 - reversible seconds via strftime for '2012-12-24 15:30:00
+ ok 351 - correct parse_datetime return value for '20121224T16:30 +01'
+ ok 352 - correct parse_datetime output seconds for '20121224T16:30 +01
+ ok 353 - strftime
+ ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00'
+ ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00
+ok 1 - subtests
diff --git a/third_party/c-dt b/third_party/c-dt
new file mode 160000
index 000000000..5b1398ca8
--- /dev/null
+++ b/third_party/c-dt
@@ -0,0 +1 @@
+Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
@ 2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:24 ` Safin Timur via Tarantool-patches
2021-08-17 15:50 ` Vladimir Davydov via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:15 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * Integrated chansen/c-dt parser as 3rd party module to the
> Tarantool cmake build process;
> * Points to tarantool/c-dt instead of original chansen/c-dt to
> have easier build integration, because there are additional
> commits which have integrated cmake support and have established
> symbols renaming facilities (similar to those we see in xxhash
> or icu).
> We have to be able to rename externally public symbols to avoid
> name clashes with 3rd party modules. We prefix c-dt symbols
> in the Tarantool build with `tnt_` prefix;
> * took special care that generated build artifacts not override
> in-source files, but use different build/ directory.
>
> * added datetime parsing unit tests, for literals - with and
> without trailing timezones;
> * also we check that strftime is reversible and produce consistent
> results after roundtrip from/to strings;
> * discovered the harder way that on *BSD/MacOSX `strftime()` format
> `%z` outputs local time-zone if passed `tm_gmtoff` is 0.
> This behaviour is different to that we observed on Linux, thus we
> might have different execution results. Made test to not use `%z`
> and only operate with normalized date time formats `%F` and `%T`
>
> Part of #5941
Hi! Thanks for the patch!
Please, find a couple of style comments below.
Also I think you may squash all the commits regarding c-dt cmake integration
into one. And push all the commits from your branch to tarantool/c-dt
master.
It's not good that they live on a separate branch
> ---
> .gitmodules | 3 +
> CMakeLists.txt | 8 +
> cmake/BuildCDT.cmake | 8 +
> src/CMakeLists.txt | 3 +-
> test/unit/CMakeLists.txt | 3 +-
> test/unit/datetime.c | 223 ++++++++++++++++++++++++
> test/unit/datetime.result | 358 ++++++++++++++++++++++++++++++++++++++
> third_party/c-dt | 1 +
> 8 files changed, 605 insertions(+), 2 deletions(-)
> create mode 100644 cmake/BuildCDT.cmake
> create mode 100644 test/unit/datetime.c
> create mode 100644 test/unit/datetime.result
> create mode 160000 third_party/c-dt
>
> diff --git a/.gitmodules b/.gitmodules
> index f2f91ee72..aa3fbae4e 100644
> --- a/.gitmodules
> +++ b/.gitmodules
> @@ -43,3 +43,6 @@
> [submodule "third_party/xxHash"]
> path = third_party/xxHash
> url =https://github.com/tarantool/xxHash
> +[submodule "third_party/c-dt"]
> + path = third_party/c-dt
> + url =https://github.com/tarantool/c-dt.git
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index e25b81eac..53c86f2a5 100644
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -571,6 +571,14 @@ endif()
> # zstd
> #
>
> +#
> +# Chritian Hanson c-dt
> +#
> +
> +include(BuildCDT)
> +libccdt_build()
> +add_dependencies(build_bundled_libs cdt)
> +
> #
> # Third-Party misc
> #
> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
> new file mode 100644
> index 000000000..343fb1b99
> --- /dev/null
> +++ b/cmake/BuildCDT.cmake
> @@ -0,0 +1,8 @@
> +macro(libccdt_build)
> + set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/)
> + set(LIBCDT_LIBRARIES cdt)
> +
> + file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
> + add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
> + ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
> +endmacro()
> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> index adb03b3f4..97b0cb326 100644
> --- a/src/CMakeLists.txt
> +++ b/src/CMakeLists.txt
> @@ -193,7 +193,8 @@ target_link_libraries(server core coll http_parser bit uri uuid swim swim_udp
> # Rule of thumb: if exporting a symbol from a static library, list the
> # library here.
> set (reexport_libraries server core misc bitset csv swim swim_udp swim_ev
> - shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES} ${CURL_LIBRARIES} ${XXHASH_LIBRARIES})
> + shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}
> + ${CURL_LIBRARIES} ${XXHASH_LIBRARIES} ${LIBCDT_LIBRARIES})
>
> set (common_libraries
> ${reexport_libraries}
> diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
> index 5bb7cd6e7..31b183a8f 100644
> --- a/test/unit/CMakeLists.txt
> +++ b/test/unit/CMakeLists.txt
> @@ -56,7 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
> target_link_libraries(uuid.test uuid unit)
> add_executable(random.test random.c core_test_utils.c)
> target_link_libraries(random.test core unit)
> -
> +add_executable(datetime.test datetime.c)
> +target_link_libraries(datetime.test cdt unit)
> add_executable(bps_tree.test bps_tree.cc)
> target_link_libraries(bps_tree.test small misc)
> add_executable(bps_tree_iterator.test bps_tree_iterator.cc)
> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
> new file mode 100644
> index 000000000..64c19dac4
> --- /dev/null
> +++ b/test/unit/datetime.c
I see that you only test datetime parsing in this test.
Not the datetime module itself.
Maybe worth renaming the test to datetime_parse, or c-dt,
or any other name you find suitable?
P.S. never mind, I see more tests are added to this file later on.
<stripped>
> +
> +/* avoid introducing external datetime.h dependency -
> + just copy paste it for today
> +*/
Please, fix comment formatting:
/* Something you have to say. */
/*
* Something you have to say
* spanning a couple of lines.
*/
> +#define SECS_PER_DAY 86400
> +#define DT_EPOCH_1970_OFFSET 719163
> +
> +struct datetime {
> + double secs;
> + int32_t nsec;
> + int32_t offset;
> +};
> +
> +static int
> +local_rd(const struct datetime *dt)
> +{
> + return (int)((int64_t)dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET;
> +}
> +
> +static int
> +local_dt(const struct datetime *dt)
> +{
> + return dt_from_rdn(local_rd(dt));
> +}
> +
> +struct tm *
> +datetime_to_tm(struct datetime *dt)
> +{
> + static struct tm tm;
> +
> + memset(&tm, 0, sizeof(tm));
> + dt_to_struct_tm(local_dt(dt), &tm);
> +
> + int seconds_of_day = (int64_t)dt->secs % 86400;
> + tm.tm_hour = (seconds_of_day / 3600) % 24;
> + tm.tm_min = (seconds_of_day / 60) % 60;
> + tm.tm_sec = seconds_of_day % 60;
> +
> + return &tm;
> +}
> +
> +static void datetime_test(void)
> +{
> + size_t index;
> + int64_t secs_expected;
> + int32_t nanosecs;
> + int32_t ofs;
> +
> + plan(355);
> + parse_datetime(sample, sizeof(sample) - 1,
> + &secs_expected, &nanosecs, &ofs);
> +
> + for (index = 0; index < DIM(tests); index++) {
> + int64_t secs;
> + int rc = parse_datetime(tests[index].sz, tests[index].len,
> + &secs, &nanosecs, &ofs);
Please, fix argument alignment here.
> + is(rc, 0, "correct parse_datetime return value for '%s'",
> + tests[index].sz);
> + is(secs, secs_expected, "correct parse_datetime output "
> + "seconds for '%s", tests[index].sz);
> +
> + /* check that stringized literal produces the same date */
> + /* time fields */
Same as above, please fix comment formatting.
> + static char buff[40];
> + struct datetime dt = {secs, nanosecs, ofs};
> + /* datetime_to_tm returns time in GMT zone */
> + struct tm * p_tm = datetime_to_tm(&dt);
> + size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
> + ok(len > 0, "strftime");
> + int64_t parsed_secs;
> + int32_t parsed_nsecs, parsed_ofs;
> + rc = parse_datetime(buff, len, &parsed_secs, &parsed_nsecs, &parsed_ofs);
> + is(rc, 0, "correct parse_datetime return value for '%s'", buff);
> + is(secs, parsed_secs,
> + "reversible seconds via strftime for '%s", buff);
> + }
> +}
> +
> +int
> +main(void)
> +{
> + plan(1);
> + datetime_test();
> +
> + return check_plan();
> +}
> diff --git a/test/unit/datetime.result b/test/unit/datetime.result
> new file mode 100644
> index 000000000..33997d9df
> --- /dev/null
> +++ b/test/unit/datetime.result
<stripped>
> diff --git a/third_party/c-dt b/third_party/c-dt
> new file mode 160000
> index 000000000..5b1398ca8
> --- /dev/null
> +++ b/third_party/c-dt
> @@ -0,0 +1 @@
> +Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
> --
> Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:24 ` Safin Timur via Tarantool-patches
2021-08-18 8:56 ` Serge Petrenko via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:24 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
Thanks, for quick review, please see my notes below...
On 17.08.2021 15:15, Serge Petrenko wrote:
>
> Hi! Thanks for the patch!
>
> Please, find a couple of style comments below.
>
> Also I think you may squash all the commits regarding c-dt cmake
> integration
> into one. And push all the commits from your branch to tarantool/c-dt
> master.
> It's not good that they live on a separate branch
I wanted to keep original Christian Hansen' master intact, that's why
I've pushed all our changes to the separate branch. Agreed though they
make no much sense separately, and squashed them together. Also renamed
branch `cmake` to `tarantool-master` and has made it's default in
repository.
>
>> ---
>> .gitmodules | 3 +
>> CMakeLists.txt | 8 +
>> cmake/BuildCDT.cmake | 8 +
>> src/CMakeLists.txt | 3 +-
>> test/unit/CMakeLists.txt | 3 +-
>> test/unit/datetime.c | 223 ++++++++++++++++++++++++
>> test/unit/datetime.result | 358 ++++++++++++++++++++++++++++++++++++++
>> third_party/c-dt | 1 +
>> 8 files changed, 605 insertions(+), 2 deletions(-)
>> create mode 100644 cmake/BuildCDT.cmake
>> create mode 100644 test/unit/datetime.c
>> create mode 100644 test/unit/datetime.result
>> create mode 160000 third_party/c-dt
>>
>> diff --git a/.gitmodules b/.gitmodules
>> index f2f91ee72..aa3fbae4e 100644
>> --- a/.gitmodules
>> +++ b/.gitmodules
>> @@ -43,3 +43,6 @@
>> [submodule "third_party/xxHash"]
>> path = third_party/xxHash
>> url =https://github.com/tarantool/xxHash
>> +[submodule "third_party/c-dt"]
>> + path = third_party/c-dt
>> + url =https://github.com/tarantool/c-dt.git
>> diff --git a/CMakeLists.txt b/CMakeLists.txt
>> index e25b81eac..53c86f2a5 100644
>> --- a/CMakeLists.txt
>> +++ b/CMakeLists.txt
>> @@ -571,6 +571,14 @@ endif()
>> # zstd
>> #
>> +#
>> +# Chritian Hanson c-dt
BTW, I've spelled his name wrongly. Fixed and push-forced updated branch.
>> +#
>> +
>> +include(BuildCDT)
>> +libccdt_build()
>> +add_dependencies(build_bundled_libs cdt)
>> +
>> #
>> # Third-Party misc
>> #
>> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
>> new file mode 100644
>> index 000000000..343fb1b99
>> --- /dev/null
>> +++ b/cmake/BuildCDT.cmake
>> @@ -0,0 +1,8 @@
>> +macro(libccdt_build)
>> + set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/)
>> + set(LIBCDT_LIBRARIES cdt)
>> +
>> + file(MAKE_DIRECTORY
>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>> + add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
>> +
>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>> +endmacro()
>> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
>> index adb03b3f4..97b0cb326 100644
>> --- a/src/CMakeLists.txt
>> +++ b/src/CMakeLists.txt
>> @@ -193,7 +193,8 @@ target_link_libraries(server core coll http_parser
>> bit uri uuid swim swim_udp
>> # Rule of thumb: if exporting a symbol from a static library, list the
>> # library here.
>> set (reexport_libraries server core misc bitset csv swim swim_udp
>> swim_ev
>> - shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES}
>> ${ICU_LIBRARIES} ${CURL_LIBRARIES} ${XXHASH_LIBRARIES})
>> + shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}
>> + ${CURL_LIBRARIES} ${XXHASH_LIBRARIES} ${LIBCDT_LIBRARIES})
>> set (common_libraries
>> ${reexport_libraries}
>> diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
>> index 5bb7cd6e7..31b183a8f 100644
>> --- a/test/unit/CMakeLists.txt
>> +++ b/test/unit/CMakeLists.txt
>> @@ -56,7 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
>> target_link_libraries(uuid.test uuid unit)
>> add_executable(random.test random.c core_test_utils.c)
>> target_link_libraries(random.test core unit)
>> -
>> +add_executable(datetime.test datetime.c)
>> +target_link_libraries(datetime.test cdt unit)
>> add_executable(bps_tree.test bps_tree.cc)
>> target_link_libraries(bps_tree.test small misc)
>> add_executable(bps_tree_iterator.test bps_tree_iterator.cc)
>> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
>> new file mode 100644
>> index 000000000..64c19dac4
>> --- /dev/null
>> +++ b/test/unit/datetime.c
>
> I see that you only test datetime parsing in this test.
> Not the datetime module itself.
> Maybe worth renaming the test to datetime_parse, or c-dt,
> or any other name you find suitable?
>
> P.S. never mind, I see more tests are added to this file later on.
Yes.
>
>
> <stripped>
>> +
>> +/* avoid introducing external datetime.h dependency -
>> + just copy paste it for today
>> +*/
>
> Please, fix comment formatting:
>
> /* Something you have to say. */
>
> /*
> * Something you have to say
> * spanning a couple of lines.
> */
>
I've deleted this block of code with comment, after I've introduced
datetime.h. So I'd rather delete this comment.
>> +#define SECS_PER_DAY 86400
>> +#define DT_EPOCH_1970_OFFSET 719163
>> +
>> +struct datetime {
>> + double secs;
>> + int32_t nsec;
>> + int32_t offset;
>> +};
>> +
>> +static int
>> +local_rd(const struct datetime *dt)
>> +{
>> + return (int)((int64_t)dt->secs / SECS_PER_DAY) +
>> DT_EPOCH_1970_OFFSET;
>> +}
>> +
>> +static int
>> +local_dt(const struct datetime *dt)
>> +{
>> + return dt_from_rdn(local_rd(dt));
>> +}
>> +
>> +struct tm *
>> +datetime_to_tm(struct datetime *dt)
>> +{
>> + static struct tm tm;
>> +
>> + memset(&tm, 0, sizeof(tm));
>> + dt_to_struct_tm(local_dt(dt), &tm);
>> +
>> + int seconds_of_day = (int64_t)dt->secs % 86400;
>> + tm.tm_hour = (seconds_of_day / 3600) % 24;
>> + tm.tm_min = (seconds_of_day / 60) % 60;
>> + tm.tm_sec = seconds_of_day % 60;
>> +
>> + return &tm;
>> +}
>> +
>> +static void datetime_test(void)
>> +{
>> + size_t index;
>> + int64_t secs_expected;
>> + int32_t nanosecs;
>> + int32_t ofs;
>> +
>> + plan(355);
>> + parse_datetime(sample, sizeof(sample) - 1,
>> + &secs_expected, &nanosecs, &ofs);
>> +
>> + for (index = 0; index < DIM(tests); index++) {
>> + int64_t secs;
>> + int rc = parse_datetime(tests[index].sz, tests[index].len,
>> + &secs, &nanosecs, &ofs);
>
>
> Please, fix argument alignment here.
Fixed.
>
>
>> + is(rc, 0, "correct parse_datetime return value for '%s'",
>> + tests[index].sz);
>> + is(secs, secs_expected, "correct parse_datetime output "
>> + "seconds for '%s", tests[index].sz);
>> +
>> + /* check that stringized literal produces the same date */
>> + /* time fields */
>
>
> Same as above, please fix comment formatting.
>
Ditto.
>
>> + static char buff[40];
>> + struct datetime dt = {secs, nanosecs, ofs};
>> + /* datetime_to_tm returns time in GMT zone */
>> + struct tm * p_tm = datetime_to_tm(&dt);
>> + size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
>> + ok(len > 0, "strftime");
>> + int64_t parsed_secs;
>> + int32_t parsed_nsecs, parsed_ofs;
>> + rc = parse_datetime(buff, len, &parsed_secs, &parsed_nsecs,
>> &parsed_ofs);
>> + is(rc, 0, "correct parse_datetime return value for '%s'", buff);
>> + is(secs, parsed_secs,
>> + "reversible seconds via strftime for '%s", buff);
>> + }
>> +}
>> +
>> +int
>> +main(void)
>> +{
>> + plan(1);
>> + datetime_test();
>> +
>> + return check_plan();
>> +}
>> diff --git a/test/unit/datetime.result b/test/unit/datetime.result
>> new file mode 100644
>> index 000000000..33997d9df
>> --- /dev/null
>> +++ b/test/unit/datetime.result
>
> <stripped>
>
>> diff --git a/third_party/c-dt b/third_party/c-dt
>> new file mode 160000
>> index 000000000..5b1398ca8
>> --- /dev/null
>> +++ b/third_party/c-dt
>> @@ -0,0 +1 @@
>> +Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
>> --
>> Serge Petrenko
>
Here is incremental update for this patch so far (applied to branch, but
not yet pushed - to accumulate all changes after Vova feedback):
--------------------------------------------------------------------
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 53c86f2a5..8037c30a7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -572,7 +572,7 @@ endif()
#
#
-# Chritian Hanson c-dt
+# Christian Hansen c-dt
#
include(BuildCDT)
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
index 64c19dac4..827930fc4 100644
--- a/test/unit/datetime.c
+++ b/test/unit/datetime.c
@@ -136,16 +136,13 @@ exit:
return 0;
}
-/* avoid introducing external datetime.h dependency -
- just copy paste it for today
-*/
-#define SECS_PER_DAY 86400
+#define SECS_PER_DAY 86400
#define DT_EPOCH_1970_OFFSET 719163
struct datetime {
- double secs;
- int32_t nsec;
- int32_t offset;
+ double secs;
+ uint32_t nsec;
+ int32_t offset;
};
static int
@@ -190,14 +187,16 @@ static void datetime_test(void)
for (index = 0; index < DIM(tests); index++) {
int64_t secs;
int rc = parse_datetime(tests[index].sz, tests[index].len,
- &secs, &nanosecs, &ofs);
+ &secs, &nanosecs, &ofs);
is(rc, 0, "correct parse_datetime return value for '%s'",
tests[index].sz);
is(secs, secs_expected, "correct parse_datetime output "
"seconds for '%s", tests[index].sz);
- /* check that stringized literal produces the same date */
- /* time fields */
+ /*
+ * check that stringized literal produces the same date
+ * time fields
+ */
static char buff[40];
struct datetime dt = {secs, nanosecs, ofs};
/* datetime_to_tm returns time in GMT zone */
diff --git a/third_party/c-dt b/third_party/c-dt
index 5b1398ca8..3cbbbc7f0 160000
--- a/third_party/c-dt
+++ b/third_party/c-dt
@@ -1 +1 @@
-Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
+Subproject commit 3cbbbc7f032cfa67a8df9f81101403249825d7f3
--------------------------------------------------------------------
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-17 23:24 ` Safin Timur via Tarantool-patches
@ 2021-08-18 8:56 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-18 8:56 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
18.08.2021 02:24, Safin Timur пишет:
> Thanks, for quick review, please see my notes below...
>
> On 17.08.2021 15:15, Serge Petrenko wrote:
>>
>> Hi! Thanks for the patch!
>>
>> Please, find a couple of style comments below.
>>
>> Also I think you may squash all the commits regarding c-dt cmake
>> integration
>> into one. And push all the commits from your branch to tarantool/c-dt
>> master.
>> It's not good that they live on a separate branch
>
> I wanted to keep original Christian Hansen' master intact, that's why
> I've pushed all our changes to the separate branch. Agreed though
> they make no much sense separately, and squashed them together. Also
> renamed branch `cmake` to `tarantool-master` and has made it's default
> in repository.
Yep, that's fine.
>
>>
>>> ---
>>> .gitmodules | 3 +
>>> CMakeLists.txt | 8 +
>>> cmake/BuildCDT.cmake | 8 +
>>> src/CMakeLists.txt | 3 +-
>>> test/unit/CMakeLists.txt | 3 +-
>>> test/unit/datetime.c | 223 ++++++++++++++++++++++++
>>> test/unit/datetime.result | 358
>>> ++++++++++++++++++++++++++++++++++++++
>>> third_party/c-dt | 1 +
>>> 8 files changed, 605 insertions(+), 2 deletions(-)
>>> create mode 100644 cmake/BuildCDT.cmake
>>> create mode 100644 test/unit/datetime.c
>>> create mode 100644 test/unit/datetime.result
>>> create mode 160000 third_party/c-dt
>>>
>>> diff --git a/.gitmodules b/.gitmodules
>>> index f2f91ee72..aa3fbae4e 100644
>>> --- a/.gitmodules
>>> +++ b/.gitmodules
>>> @@ -43,3 +43,6 @@
>>> [submodule "third_party/xxHash"]
>>> path = third_party/xxHash
>>> url =https://github.com/tarantool/xxHash
>>> +[submodule "third_party/c-dt"]
>>> + path = third_party/c-dt
>>> + url =https://github.com/tarantool/c-dt.git
>>> diff --git a/CMakeLists.txt b/CMakeLists.txt
>>> index e25b81eac..53c86f2a5 100644
>>> --- a/CMakeLists.txt
>>> +++ b/CMakeLists.txt
>>> @@ -571,6 +571,14 @@ endif()
>>> # zstd
>>> #
>>> +#
>>> +# Chritian Hanson c-dt
>
> BTW, I've spelled his name wrongly. Fixed and push-forced updated branch.
>
>>> +#
>>> +
>>> +include(BuildCDT)
>>> +libccdt_build()
>>> +add_dependencies(build_bundled_libs cdt)
>>> +
>>> #
>>> # Third-Party misc
>>> #
>>> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
>>> new file mode 100644
>>> index 000000000..343fb1b99
>>> --- /dev/null
>>> +++ b/cmake/BuildCDT.cmake
>>> @@ -0,0 +1,8 @@
>>> +macro(libccdt_build)
>>> + set(LIBCDT_INCLUDE_DIRS ${PROJECT_SOURCE_DIR}/third_party/c-dt/)
>>> + set(LIBCDT_LIBRARIES cdt)
>>> +
>>> + file(MAKE_DIRECTORY
>>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>>> + add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
>>> + ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>>> +endmacro()
>>> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
>>> index adb03b3f4..97b0cb326 100644
>>> --- a/src/CMakeLists.txt
>>> +++ b/src/CMakeLists.txt
>>> @@ -193,7 +193,8 @@ target_link_libraries(server core coll
>>> http_parser bit uri uuid swim swim_udp
>>> # Rule of thumb: if exporting a symbol from a static library, list
>>> the
>>> # library here.
>>> set (reexport_libraries server core misc bitset csv swim swim_udp
>>> swim_ev
>>> - shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES}
>>> ${ICU_LIBRARIES} ${CURL_LIBRARIES} ${XXHASH_LIBRARIES})
>>> + shutdown ${LUAJIT_LIBRARIES} ${MSGPUCK_LIBRARIES}
>>> ${ICU_LIBRARIES}
>>> + ${CURL_LIBRARIES} ${XXHASH_LIBRARIES} ${LIBCDT_LIBRARIES})
>>> set (common_libraries
>>> ${reexport_libraries}
>>> diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
>>> index 5bb7cd6e7..31b183a8f 100644
>>> --- a/test/unit/CMakeLists.txt
>>> +++ b/test/unit/CMakeLists.txt
>>> @@ -56,7 +56,8 @@ add_executable(uuid.test uuid.c core_test_utils.c)
>>> target_link_libraries(uuid.test uuid unit)
>>> add_executable(random.test random.c core_test_utils.c)
>>> target_link_libraries(random.test core unit)
>>> -
>>> +add_executable(datetime.test datetime.c)
>>> +target_link_libraries(datetime.test cdt unit)
>>> add_executable(bps_tree.test bps_tree.cc)
>>> target_link_libraries(bps_tree.test small misc)
>>> add_executable(bps_tree_iterator.test bps_tree_iterator.cc)
>>> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
>>> new file mode 100644
>>> index 000000000..64c19dac4
>>> --- /dev/null
>>> +++ b/test/unit/datetime.c
>>
>> I see that you only test datetime parsing in this test.
>> Not the datetime module itself.
>> Maybe worth renaming the test to datetime_parse, or c-dt,
>> or any other name you find suitable?
>>
>> P.S. never mind, I see more tests are added to this file later on.
>
> Yes.
>
>>
>>
>> <stripped>
>>> +
>>> +/* avoid introducing external datetime.h dependency -
>>> + just copy paste it for today
>>> +*/
>>
>> Please, fix comment formatting:
>>
>> /* Something you have to say. */
>>
>> /*
>> * Something you have to say
>> * spanning a couple of lines.
>> */
>>
>
> I've deleted this block of code with comment, after I've introduced
> datetime.h. So I'd rather delete this comment.
>
>
>>> +#define SECS_PER_DAY 86400
>>> +#define DT_EPOCH_1970_OFFSET 719163
>>> +
>>> +struct datetime {
>>> + double secs;
>>> + int32_t nsec;
>>> + int32_t offset;
>>> +};
>>> +
>>> +static int
>>> +local_rd(const struct datetime *dt)
>>> +{
>>> + return (int)((int64_t)dt->secs / SECS_PER_DAY) +
>>> DT_EPOCH_1970_OFFSET;
>>> +}
>>> +
>>> +static int
>>> +local_dt(const struct datetime *dt)
>>> +{
>>> + return dt_from_rdn(local_rd(dt));
>>> +}
>>> +
>>> +struct tm *
>>> +datetime_to_tm(struct datetime *dt)
>>> +{
>>> + static struct tm tm;
>>> +
>>> + memset(&tm, 0, sizeof(tm));
>>> + dt_to_struct_tm(local_dt(dt), &tm);
>>> +
>>> + int seconds_of_day = (int64_t)dt->secs % 86400;
>>> + tm.tm_hour = (seconds_of_day / 3600) % 24;
>>> + tm.tm_min = (seconds_of_day / 60) % 60;
>>> + tm.tm_sec = seconds_of_day % 60;
>>> +
>>> + return &tm;
>>> +}
>>> +
>>> +static void datetime_test(void)
>>> +{
>>> + size_t index;
>>> + int64_t secs_expected;
>>> + int32_t nanosecs;
>>> + int32_t ofs;
>>> +
>>> + plan(355);
>>> + parse_datetime(sample, sizeof(sample) - 1,
>>> + &secs_expected, &nanosecs, &ofs);
>>> +
>>> + for (index = 0; index < DIM(tests); index++) {
>>> + int64_t secs;
>>> + int rc = parse_datetime(tests[index].sz, tests[index].len,
>>> + &secs, &nanosecs, &ofs);
>>
>>
>> Please, fix argument alignment here.
>
> Fixed.
>
>>
>>
>>> + is(rc, 0, "correct parse_datetime return value for '%s'",
>>> + tests[index].sz);
>>> + is(secs, secs_expected, "correct parse_datetime output "
>>> + "seconds for '%s", tests[index].sz);
>>> +
>>> + /* check that stringized literal produces the same date */
>>> + /* time fields */
>>
>>
>> Same as above, please fix comment formatting.
>>
>
> Ditto.
>
>>
>>> + static char buff[40];
>>> + struct datetime dt = {secs, nanosecs, ofs};
>>> + /* datetime_to_tm returns time in GMT zone */
>>> + struct tm * p_tm = datetime_to_tm(&dt);
>>> + size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
>>> + ok(len > 0, "strftime");
>>> + int64_t parsed_secs;
>>> + int32_t parsed_nsecs, parsed_ofs;
>>> + rc = parse_datetime(buff, len, &parsed_secs, &parsed_nsecs,
>>> &parsed_ofs);
>>> + is(rc, 0, "correct parse_datetime return value for '%s'",
>>> buff);
>>> + is(secs, parsed_secs,
>>> + "reversible seconds via strftime for '%s", buff);
>>> + }
>>> +}
>>> +
>>> +int
>>> +main(void)
>>> +{
>>> + plan(1);
>>> + datetime_test();
>>> +
>>> + return check_plan();
>>> +}
>>> diff --git a/test/unit/datetime.result b/test/unit/datetime.result
>>> new file mode 100644
>>> index 000000000..33997d9df
>>> --- /dev/null
>>> +++ b/test/unit/datetime.result
>>
>> <stripped>
>>
>>> diff --git a/third_party/c-dt b/third_party/c-dt
>>> new file mode 160000
>>> index 000000000..5b1398ca8
>>> --- /dev/null
>>> +++ b/third_party/c-dt
>>> @@ -0,0 +1 @@
>>> +Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
>>> --
>>> Serge Petrenko
>>
>
> Here is incremental update for this patch so far (applied to branch,
> but not yet pushed - to accumulate all changes after Vova feedback):
> --------------------------------------------------------------------
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index 53c86f2a5..8037c30a7 100644
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -572,7 +572,7 @@ endif()
> #
>
> #
> -# Chritian Hanson c-dt
> +# Christian Hansen c-dt
> #
>
> include(BuildCDT)
> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
> index 64c19dac4..827930fc4 100644
> --- a/test/unit/datetime.c
> +++ b/test/unit/datetime.c
> @@ -136,16 +136,13 @@ exit:
> return 0;
> }
>
> -/* avoid introducing external datetime.h dependency -
> - just copy paste it for today
> -*/
> -#define SECS_PER_DAY 86400
> +#define SECS_PER_DAY 86400
> #define DT_EPOCH_1970_OFFSET 719163
>
> struct datetime {
> - double secs;
> - int32_t nsec;
> - int32_t offset;
> + double secs;
> + uint32_t nsec;
> + int32_t offset;
> };
>
> static int
> @@ -190,14 +187,16 @@ static void datetime_test(void)
> for (index = 0; index < DIM(tests); index++) {
> int64_t secs;
> int rc = parse_datetime(tests[index].sz,
> tests[index].len,
> - &secs, &nanosecs, &ofs);
> + &secs, &nanosecs, &ofs);
> is(rc, 0, "correct parse_datetime return value for '%s'",
> tests[index].sz);
> is(secs, secs_expected, "correct parse_datetime output "
> "seconds for '%s", tests[index].sz);
>
> - /* check that stringized literal produces the same
> date */
> - /* time fields */
> + /*
> + * check that stringized literal produces the same date
> + * time fields
> + */
> static char buff[40];
> struct datetime dt = {secs, nanosecs, ofs};
> /* datetime_to_tm returns time in GMT zone */
> diff --git a/third_party/c-dt b/third_party/c-dt
> index 5b1398ca8..3cbbbc7f0 160000
> --- a/third_party/c-dt
> +++ b/third_party/c-dt
> @@ -1 +1 @@
> -Subproject commit 5b1398ca8f53513985670a2e832b5f6e253addf7
> +Subproject commit 3cbbbc7f032cfa67a8df9f81101403249825d7f3
> --------------------------------------------------------------------
>
> Thanks,
> Timur
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 15:50 ` Vladimir Davydov via Tarantool-patches
2021-08-18 10:04 ` Safin Timur via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 15:50 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:35AM +0300, Timur Safin via Tarantool-patches wrote:
> * Integrated chansen/c-dt parser as 3rd party module to the
> Tarantool cmake build process;
> * Points to tarantool/c-dt instead of original chansen/c-dt to
> have easier build integration, because there are additional
> commits which have integrated cmake support and have established
> symbols renaming facilities (similar to those we see in xxhash
> or icu).
> We have to be able to rename externally public symbols to avoid
> name clashes with 3rd party modules. We prefix c-dt symbols
> in the Tarantool build with `tnt_` prefix;
I don't see this done in this patch. Looks like this is done by patch 2.
> * took special care that generated build artifacts not override
> in-source files, but use different build/ directory.
>
> * added datetime parsing unit tests, for literals - with and
> without trailing timezones;
> * also we check that strftime is reversible and produce consistent
> results after roundtrip from/to strings;
> * discovered the harder way that on *BSD/MacOSX `strftime()` format
> `%z` outputs local time-zone if passed `tm_gmtoff` is 0.
> This behaviour is different to that we observed on Linux, thus we
> might have different execution results. Made test to not use `%z`
> and only operate with normalized date time formats `%F` and `%T`
>
> Part of #5941
> ---
> .gitmodules | 3 +
> CMakeLists.txt | 8 +
> cmake/BuildCDT.cmake | 8 +
> src/CMakeLists.txt | 3 +-
> test/unit/CMakeLists.txt | 3 +-
> test/unit/datetime.c | 223 ++++++++++++++++++++++++
> test/unit/datetime.result | 358 ++++++++++++++++++++++++++++++++++++++
> third_party/c-dt | 1 +
> 8 files changed, 605 insertions(+), 2 deletions(-)
> create mode 100644 cmake/BuildCDT.cmake
> create mode 100644 test/unit/datetime.c
> create mode 100644 test/unit/datetime.result
> create mode 160000 third_party/c-dt
>
> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
> new file mode 100644
> index 000000000..64c19dac4
> --- /dev/null
> +++ b/test/unit/datetime.c
> @@ -0,0 +1,223 @@
> +#include "dt.h"
> +#include <assert.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <time.h>
> +
> +#include "unit.h"
> +
> +static const char sample[] = "2012-12-24T15:30Z";
> +
> +#define S(s) {s, sizeof(s) - 1}
> +struct {
> + const char * sz;
Extra space after '*'.
We usually name a variable that stores a zero-terminated 's' or 'str',
never 'sz'.
> +#define DIM(a) (sizeof(a) / sizeof(a[0]))
There's 'lengthof' helper already defined for this.
> +
> +/* p5-time-moment/src/moment_parse.c: parse_string_lenient() */
I don't understand the purpose of this comment.
> +static int
> +parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
> + int32_t *op)
What's 'sp', 'np', and 'op'? Please refrain from using confusing
abbreviations.
> +{
> + size_t n;
> + dt_t dt;
> + char c;
> + int sod = 0, nanosecond = 0, offset = 0;
> +
> + n = dt_parse_iso_date(str, len, &dt);
> + if (!n)
> + return 1;
> + if (n == len)
> + goto exit;
> +
> + c = str[n++];
> + if (!(c == 'T' || c == 't' || c == ' '))
> + return 1;
> +
> + str += n;
> + len -= n;
> +
> + n = dt_parse_iso_time(str, len, &sod, &nanosecond);
> + if (!n)
> + return 1;
> + if (n == len)
> + goto exit;
> +
> + if (str[n] == ' ')
> + n++;
Bad indentation.
> +
> + str += n;
> + len -= n;
> +
> + n = dt_parse_iso_zone_lenient(str, len, &offset);
> + if (!n || n != len)
> + return 1;
> +
> +exit:
> + *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
Please define and use appropriate constants to make the code easier for
understanding: DAYS_PER_YEAR, MINUTES_PER_HOUR, etc.
> + *np = nanosecond;
> + *op = offset;
> +
> + return 0;
> +}
> +
> +/* avoid introducing external datetime.h dependency -
> + just copy paste it for today
> +*/
Bad comment formatting.
> +#define SECS_PER_DAY 86400
> +#define DT_EPOCH_1970_OFFSET 719163
> +
> +struct datetime {
> + double secs;
> + int32_t nsec;
> + int32_t offset;
> +};
I see that this struct as well as some functions in this module are
defined in src/lib/core/datetime in patch 2, and then you remove struct
datetime from this test in patch 3, but leave the helper functions
datetime_to_tm, parse_datetime.
This makes me think that:
- You should squash patches 1-3.
- Both datetime and datetime manipulating functions (including
parse_datetime) should be defined only in src/lib/core/datetime
while test/unit should depend on src/lib/core/datetime.
> +
> +static int
> +local_rd(const struct datetime *dt)
> +{
> + return (int)((int64_t)dt->secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET;
> +}
> +
> +static int
> +local_dt(const struct datetime *dt)
> +{
> + return dt_from_rdn(local_rd(dt));
> +}
> +
> +struct tm *
> +datetime_to_tm(struct datetime *dt)
> +{
> + static struct tm tm;
> +
> + memset(&tm, 0, sizeof(tm));
> + dt_to_struct_tm(local_dt(dt), &tm);
> +
> + int seconds_of_day = (int64_t)dt->secs % 86400;
> + tm.tm_hour = (seconds_of_day / 3600) % 24;
> + tm.tm_min = (seconds_of_day / 60) % 60;
> + tm.tm_sec = seconds_of_day % 60;
> +
> + return &tm;
> +}
> +
> +static void datetime_test(void)
> +{
> + size_t index;
> + int64_t secs_expected;
> + int32_t nanosecs;
> + int32_t ofs;
offset?
> +
> + plan(355);
> + parse_datetime(sample, sizeof(sample) - 1,
> + &secs_expected, &nanosecs, &ofs);
> +
> + for (index = 0; index < DIM(tests); index++) {
> + int64_t secs;
> + int rc = parse_datetime(tests[index].sz, tests[index].len,
> + &secs, &nanosecs, &ofs);
> + is(rc, 0, "correct parse_datetime return value for '%s'",
> + tests[index].sz);
> + is(secs, secs_expected, "correct parse_datetime output "
> + "seconds for '%s", tests[index].sz);
> +
> + /* check that stringized literal produces the same date */
> + /* time fields */
> + static char buff[40];
> + struct datetime dt = {secs, nanosecs, ofs};
> + /* datetime_to_tm returns time in GMT zone */
> + struct tm * p_tm = datetime_to_tm(&dt);
Extra space after '*'.
> + size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
> + ok(len > 0, "strftime");
> + int64_t parsed_secs;
> + int32_t parsed_nsecs, parsed_ofs;
> + rc = parse_datetime(buff, len, &parsed_secs, &parsed_nsecs, &parsed_ofs);
> + is(rc, 0, "correct parse_datetime return value for '%s'", buff);
> + is(secs, parsed_secs,
> + "reversible seconds via strftime for '%s", buff);
> + }
> +}
> +
> +int
> +main(void)
> +{
> + plan(1);
> + datetime_test();
> +
> + return check_plan();
> +}
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build
2021-08-17 15:50 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 10:04 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 10:04 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
Thanks for your feedback, it's massive. Will take some time to process
all changes...
On 17.08.2021 18:50, Vladimir Davydov via Tarantool-patches wrote:
> On Mon, Aug 16, 2021 at 02:59:35AM +0300, Timur Safin via
> Tarantool-patches wrote:
>> * Integrated chansen/c-dt parser as 3rd party module to the
>> Tarantool cmake build process;
>> * Points to tarantool/c-dt instead of original chansen/c-dt to
>> have easier build integration, because there are additional
>> commits which have integrated cmake support and have established
>> symbols renaming facilities (similar to those we see in xxhash
>> or icu).
>> We have to be able to rename externally public symbols to avoid
>> name clashes with 3rd party modules. We prefix c-dt symbols
>> in the Tarantool build with `tnt_` prefix;
>
> I don't see this done in this patch. Looks like this is done by patch 2.
Indeed, that was extracted for better observability of a patch.
>
>> * took special care that generated build artifacts not override
>> in-source files, but use different build/ directory.
>>
>> * added datetime parsing unit tests, for literals - with and
>> without trailing timezones;
>> * also we check that strftime is reversible and produce consistent
>> results after roundtrip from/to strings;
>> * discovered the harder way that on *BSD/MacOSX `strftime()` format
>> `%z` outputs local time-zone if passed `tm_gmtoff` is 0.
>> This behaviour is different to that we observed on Linux, thus we
>> might have different execution results. Made test to not use `%z`
>> and only operate with normalized date time formats `%F` and `%T`
>>
>> Part of #5941
>> ---
>> .gitmodules | 3 +
>> CMakeLists.txt | 8 +
>> cmake/BuildCDT.cmake | 8 +
>> src/CMakeLists.txt | 3 +-
>> test/unit/CMakeLists.txt | 3 +-
>> test/unit/datetime.c | 223 ++++++++++++++++++++++++
>> test/unit/datetime.result | 358 ++++++++++++++++++++++++++++++++++++++
>> third_party/c-dt | 1 +
>> 8 files changed, 605 insertions(+), 2 deletions(-)
>> create mode 100644 cmake/BuildCDT.cmake
>> create mode 100644 test/unit/datetime.c
>> create mode 100644 test/unit/datetime.result
>> create mode 160000 third_party/c-dt
>>
>> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
>> new file mode 100644
>> index 000000000..64c19dac4
>> --- /dev/null
>> +++ b/test/unit/datetime.c
>> @@ -0,0 +1,223 @@
>> +#include "dt.h"
>> +#include <assert.h>
>> +#include <stdint.h>
>> +#include <string.h>
>> +#include <time.h>
>> +
>> +#include "unit.h"
>> +
>> +static const char sample[] = "2012-12-24T15:30Z";
>> +
>> +#define S(s) {s, sizeof(s) - 1}
>> +struct {
>> + const char * sz;
>
> Extra space after '*'.
>
> We usually name a variable that stores a zero-terminated 's' or 'str',
> never 'sz'.
>
>> +#define DIM(a) (sizeof(a) / sizeof(a[0]))
>
> There's 'lengthof' helper already defined for this.
Thanks! Updated.
>
>> +
>> +/* p5-time-moment/src/moment_parse.c: parse_string_lenient() */
>
> I don't understand the purpose of this comment.
That was referring to the original implementation of similar code in
Christian Hansen time-moment module for perl5. Agreed that it has not
much value right now, so removed it.
>
>> +static int
>> +parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
>> + int32_t *op)
>
> What's 'sp', 'np', and 'op'? Please refrain from using confusing
> abbreviations.
>
>> +{
>> + size_t n;
>> + dt_t dt;
>> + char c;
>> + int sod = 0, nanosecond = 0, offset = 0;
>> +
>> + n = dt_parse_iso_date(str, len, &dt);
>> + if (!n)
>> + return 1;
>> + if (n == len)
>> + goto exit;
>> +
>> + c = str[n++];
>> + if (!(c == 'T' || c == 't' || c == ' '))
>> + return 1;
>> +
>> + str += n;
>> + len -= n;
>> +
>> + n = dt_parse_iso_time(str, len, &sod, &nanosecond);
>> + if (!n)
>> + return 1;
>> + if (n == len)
>> + goto exit;
>> +
>> + if (str[n] == ' ')
>> + n++;
>
> Bad indentation.
Fixed.
>
>> +
>> + str += n;
>> + len -= n;
>> +
>> + n = dt_parse_iso_zone_lenient(str, len, &offset);
>> + if (!n || n != len)
>> + return 1;
>> +
>> +exit:
>> + *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
>
> Please define and use appropriate constants to make the code easier for
> understanding: DAYS_PER_YEAR, MINUTES_PER_HOUR, etc.
Changed to use DT_EPOCH_1970_OFFSET and SECS_PER_DAY.
>
>> + *np = nanosecond;
>> + *op = offset;
>> +
>> + return 0;
>> +}
>> +
>> +/* avoid introducing external datetime.h dependency -
>> + just copy paste it for today
>> +*/
>
> Bad comment formatting.
This comment gone. Thanks!
>
>> +#define SECS_PER_DAY 86400
>> +#define DT_EPOCH_1970_OFFSET 719163
>> +
>> +struct datetime {
>> + double secs;
>> + int32_t nsec;
>> + int32_t offset;
>> +};
>
> I see that this struct as well as some functions in this module are
> defined in src/lib/core/datetime in patch 2, and then you remove struct
> datetime from this test in patch 3, but leave the helper functions
> datetime_to_tm, parse_datetime.
>
> This makes me think that:
> - You should squash patches 1-3.
> - Both datetime and datetime manipulating functions (including
> parse_datetime) should be defined only in src/lib/core/datetime
> while test/unit should depend on src/lib/core/datetime.
Thats's quite reasonable suggestion (and was my original intention), but
for simplicity of review I tried split these huge patches into smaller
ones (look s not very successfully, and introduced some
inter-dependencies between patches). I'll squash.
>
>> +
>> +static int
>> +local_rd(const struct datetime *dt)
>> +{
>> + return (int)((int64_t)dt->secs / SECS_PER_DAY) +
> DT_EPOCH_1970_OFFSET;
>> +}
>> +
>> +static int
>> +local_dt(const struct datetime *dt)
>> +{
>> + return dt_from_rdn(local_rd(dt));
>> +}
>> +
>> +struct tm *
>> +datetime_to_tm(struct datetime *dt)
>> +{
>> + static struct tm tm;
>> +
>> + memset(&tm, 0, sizeof(tm));
>> + dt_to_struct_tm(local_dt(dt), &tm);
>> +
>> + int seconds_of_day = (int64_t)dt->secs % 86400;
>> + tm.tm_hour = (seconds_of_day / 3600) % 24;
>> + tm.tm_min = (seconds_of_day / 60) % 60;
>> + tm.tm_sec = seconds_of_day % 60;
>> +
>> + return &tm;
>> +}
>> +
>> +static void datetime_test(void)
>> +{
>> + size_t index;
>> + int64_t secs_expected;
>> + int32_t nanosecs;
>> + int32_t ofs;
>
> offset?
Yup, renamed.
>
>> +
>> + plan(355);
>> + parse_datetime(sample, sizeof(sample) - 1,
>> + &secs_expected, &nanosecs, &ofs);
>> +
>> + for (index = 0; index < DIM(tests); index++) {
>> + int64_t secs;
>> + int rc = parse_datetime(tests[index].sz, tests[index].len,
>> + &secs, &nanosecs, &ofs);
>> + is(rc, 0, "correct parse_datetime return value for '%s'",
>> + tests[index].sz);
>> + is(secs, secs_expected, "correct parse_datetime output "
>> + "seconds for '%s", tests[index].sz);
>> +
>> + /* check that stringized literal produces the same date */
>> + /* time fields */
>> + static char buff[40];
>> + struct datetime dt = {secs, nanosecs, ofs};
>> + /* datetime_to_tm returns time in GMT zone */
>> + struct tm * p_tm = datetime_to_tm(&dt);
>
> Extra space after '*'.
Removed
>
>> + size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
>> + ok(len > 0, "strftime");
>> + int64_t parsed_secs;
>> + int32_t parsed_nsecs, parsed_ofs;
>> + rc = parse_datetime(buff, len, &parsed_secs,
> &parsed_nsecs, &parsed_ofs);
>> + is(rc, 0, "correct parse_datetime return value for '%s'",
> buff);
>> + is(secs, parsed_secs,
>> + "reversible seconds via strftime for '%s", buff);
>> + }
>> +}
>> +
>> +int
>> +main(void)
>> +{
>> + plan(1);
>> + datetime_test();
>> +
>> + return check_plan();
>> +}
Here are incremental changes so far:
--------------------------------------------------------
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
index 18b24927e..bfc752c59 100644
--- a/test/unit/datetime.c
+++ b/test/unit/datetime.c
@@ -9,12 +9,13 @@
#include "mp_datetime.h"
#include "msgpuck.h"
#include "mp_extension_types.h"
+#include "trivia/util.h"
static const char sample[] = "2012-12-24T15:30Z";
#define S(s) {s, sizeof(s) - 1}
struct {
- const char * sz;
+ const char *str;
size_t len;
} tests[] = {
S("2012-12-24 15:30Z"),
@@ -91,17 +92,14 @@ struct {
};
#undef S
-#define DIM(a) (sizeof(a) / sizeof(a[0]))
-
-/* p5-time-moment/src/moment_parse.c: parse_string_lenient() */
static int
-parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
- int32_t *op)
+parse_datetime(const char *str, size_t len, int64_t *secs_p,
+ int32_t *nanosecs_p, int32_t *offset_p)
{
size_t n;
dt_t dt;
char c;
- int sod = 0, nanosecond = 0, offset = 0;
+ int sec_of_day = 0, nanosecond = 0, offset = 0;
n = dt_parse_iso_date(str, len, &dt);
if (!n)
@@ -116,14 +114,14 @@ parse_datetime(const char *str, size_t len,
int64_t *sp, int32_t *np,
str += n;
len -= n;
- n = dt_parse_iso_time(str, len, &sod, &nanosecond);
+ n = dt_parse_iso_time(str, len, &sec_of_day, &nanosecond);
if (!n)
return 1;
if (n == len)
goto exit;
if (str[n] == ' ')
- n++;
+ n++;
str += n;
len -= n;
@@ -133,9 +131,10 @@ parse_datetime(const char *str, size_t len, int64_t
*sp, int32_t *np,
return 1;
exit:
- *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
- *np = nanosecond;
- *op = offset;
+ *secs_p = ((int64_t)dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY +
+ sec_of_day - offset * 60;
+ *nanosecs_p = nanosecond;
+ *offset_p = offset;
return 0;
}
@@ -173,29 +172,30 @@ static void datetime_test(void)
size_t index;
int64_t secs_expected;
int32_t nanosecs;
- int32_t ofs;
+ int32_t offset;
plan(355);
parse_datetime(sample, sizeof(sample) - 1,
- &secs_expected, &nanosecs, &ofs);
+ &secs_expected, &nanosecs, &offset);
- for (index = 0; index < DIM(tests); index++) {
+ for (index = 0; index < lengthof(tests); index++) {
int64_t secs;
- int rc = parse_datetime(tests[index].sz, tests[index].len,
- &secs, &nanosecs, &ofs);
+ int rc = parse_datetime(tests[index].str, tests[index].len,
+ &secs, &nanosecs, &offset);
is(rc, 0, "correct parse_datetime return value for '%s'",
- tests[index].sz);
+ tests[index].str);
is(secs, secs_expected, "correct parse_datetime output "
- "seconds for '%s", tests[index].sz);
+ "seconds for '%s",
+ tests[index].str);
/*
* check that stringized literal produces the same date
* time fields
*/
static char buff[40];
- struct datetime dt = {secs, nanosecs, ofs};
+ struct datetime dt = {secs, nanosecs, offset};
/* datetime_to_tm returns time in GMT zone */
- struct tm * p_tm = datetime_to_tm(&dt);
+ struct tm *p_tm = datetime_to_tm(&dt);
size_t len = strftime(buff, sizeof buff, "%F %T", p_tm);
ok(len > 0, "strftime");
int64_t parsed_secs;
@@ -237,7 +237,7 @@ tostring_datetime_test(void)
size_t index;
plan(15);
- for (index = 0; index < DIM(tests); index++) {
+ for (index = 0; index < lengthof(tests); index++) {
struct datetime date = {
tests[index].secs,
tests[index].nsec,
@@ -282,7 +282,7 @@ mp_datetime_test()
size_t index;
plan(85);
- for (index = 0; index < DIM(tests); index++) {
+ for (index = 0; index < lengthof(tests); index++) {
struct datetime date = {
tests[index].secs,
tests[index].nsec,
--------------------------------------------------------
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 16:52 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime Timur Safin via Tarantool-patches
` (7 subsequent siblings)
9 siblings, 2 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
* created a new Tarantool built-in module `datetime`;
* register cdef types for this module;
* export some `dt_*` functions from `c-dt` library;
* implemented `ctime`, `asctime` and `strftime` as
ffi wrappers in C kernel code:
We used to use heavy ffi-related Lua code, but it appears
to be quite fragile, if we run this code on different OS
targets. To make this Lua code less platfrom dependent we
have moved all platform specifics to the C level.
Specifically we want to avoid dependence on `struct tm {}`
layout and GLIBC-specific `strftime` behaviour (which
differs to BSD/Mac LIBC behaviour).
* introduced simple datetime tests
Created app-tap test for new builtin module `datetime.lua`,
where we specifically check:
- cases of simple datetime string formatting using:
- asctime (gmt time);
- ctime (local TZ time);
- strftime (using given format).
- verify that we parse expected, and fail with unexpected:
- for that, in the datetime.lua, we have extended api of `parse_date()`,
`parse_time()`, and `parse_time_zone()` so they return
not only parsed object, but also a length of parsed substring;
- which allows us to parse even _partially_ valid strings like
"20121224 Foo bar".
- check calculated attributes to date object, e.g.:
- timestamp, seconds, microseconds, minute, or hours
- to_utc(), and to_tz() allow to switch timezone of a
datetime object. It's not changing much - only timezone
but that impacts textual representation of a date.
Part of #5941
---
cmake/BuildCDT.cmake | 2 +
| 26 +
src/CMakeLists.txt | 2 +
src/lib/core/CMakeLists.txt | 1 +
src/lib/core/datetime.c | 96 ++++
src/lib/core/datetime.h | 95 ++++
src/lua/datetime.lua | 500 ++++++++++++++++++
src/lua/init.c | 4 +-
src/lua/utils.c | 27 +
src/lua/utils.h | 12 +
test/app-tap/datetime.test.lua | 206 ++++++++
.../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 ++-
12 files changed, 1043 insertions(+), 2 deletions(-)
create mode 100644 src/lib/core/datetime.c
create mode 100644 src/lib/core/datetime.h
create mode 100644 src/lua/datetime.lua
create mode 100755 test/app-tap/datetime.test.lua
diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
index 343fb1b99..80b26c64a 100644
--- a/cmake/BuildCDT.cmake
+++ b/cmake/BuildCDT.cmake
@@ -5,4 +5,6 @@ macro(libccdt_build)
file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
+ set_target_properties(cdt PROPERTIES COMPILE_FLAGS "-DDT_NAMESPACE=tnt_")
+ add_definitions("-DDT_NAMESPACE=tnt_")
endmacro()
--git a/extra/exports b/extra/exports
index 9eaba1282..80eb92abd 100644
--- a/extra/exports
+++ b/extra/exports
@@ -148,8 +148,34 @@ csv_feed
csv_iterator_create
csv_next
csv_setopt
+datetime_asctime
+datetime_ctime
+datetime_now
+datetime_strftime
+decimal_unpack
decimal_from_string
decimal_unpack
+tnt_dt_dow
+tnt_dt_from_rdn
+tnt_dt_from_struct_tm
+tnt_dt_from_yd
+tnt_dt_from_ymd
+tnt_dt_from_yqd
+tnt_dt_from_ywd
+tnt_dt_parse_iso_date
+tnt_dt_parse_iso_time_basic
+tnt_dt_parse_iso_time_extended
+tnt_dt_parse_iso_time
+tnt_dt_parse_iso_zone_basic
+tnt_dt_parse_iso_zone_extended
+tnt_dt_parse_iso_zone_lenient
+tnt_dt_parse_iso_zone
+tnt_dt_rdn
+tnt_dt_to_struct_tm
+tnt_dt_to_yd
+tnt_dt_to_ymd
+tnt_dt_to_yqd
+tnt_dt_to_ywd
error_ref
error_set_prev
error_unref
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 97b0cb326..4473ff1da 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
lua_source(lua_sources lua/httpc.lua)
lua_source(lua_sources lua/iconv.lua)
lua_source(lua_sources lua/swim.lua)
+lua_source(lua_sources lua/datetime.lua)
+
# LuaJIT jit.* library
lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
index 2cd4d0b4f..8bc776b82 100644
--- a/src/lib/core/CMakeLists.txt
+++ b/src/lib/core/CMakeLists.txt
@@ -30,6 +30,7 @@ set(core_sources
decimal.c
mp_decimal.c
cord_buf.c
+ datetime.c
)
if (TARGET_OS_NETBSD)
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
new file mode 100644
index 000000000..c48295a6f
--- /dev/null
+++ b/src/lib/core/datetime.c
@@ -0,0 +1,96 @@
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <string.h>
+#include <time.h>
+
+#include "trivia/util.h"
+#include "datetime.h"
+
+static int
+local_dt(int64_t secs)
+{
+ return dt_from_rdn((int)(secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET);
+}
+
+static struct tm *
+datetime_to_tm(const struct datetime *date)
+{
+ static struct tm tm;
+
+ memset(&tm, 0, sizeof(tm));
+ int64_t secs = date->secs;
+ dt_to_struct_tm(local_dt(secs), &tm);
+
+ int seconds_of_day = (int64_t)date->secs % SECS_PER_DAY;
+ tm.tm_hour = (seconds_of_day / 3600) % 24;
+ tm.tm_min = (seconds_of_day / 60) % 60;
+ tm.tm_sec = seconds_of_day % 60;
+
+ return &tm;
+}
+
+void
+datetime_now(struct datetime *now)
+{
+ struct timeval tv;
+ gettimeofday(&tv, NULL);
+ now->secs = tv.tv_sec;
+ now->nsec = tv.tv_usec * 1000;
+
+ time_t now_seconds;
+ time(&now_seconds);
+ struct tm tm;
+ localtime_r(&now_seconds, &tm);
+ now->offset = tm.tm_gmtoff / 60;
+}
+
+char *
+datetime_asctime(const struct datetime *date, char *buf)
+{
+ struct tm *p_tm = datetime_to_tm(date);
+ return asctime_r(p_tm, buf);
+}
+
+char *
+datetime_ctime(const struct datetime *date, char *buf)
+{
+ time_t time = date->secs;
+ return ctime_r(&time, buf);
+}
+
+size_t
+datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
+ uint32_t len)
+{
+ struct tm *p_tm = datetime_to_tm(date);
+ return strftime(buf, len, fmt, p_tm);
+}
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
new file mode 100644
index 000000000..1a8d7e34f
--- /dev/null
+++ b/src/lib/core/datetime.h
@@ -0,0 +1,95 @@
+#pragma once
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdint.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include "c-dt/dt.h"
+
+#if defined(__cplusplus)
+extern "C"
+{
+#endif /* defined(__cplusplus) */
+
+#ifndef SECS_PER_DAY
+#define SECS_PER_DAY 86400
+#define DT_EPOCH_1970_OFFSET 719163
+#endif
+
+/**
+ * Full datetime structure representing moments
+ * since Unix Epoch (1970-01-01).
+ * Time is kept normalized to UTC, time-zone offset
+ * is informative only.
+ */
+struct datetime {
+ /** seconds since epoch */
+ double secs;
+ /** nanoseconds if any */
+ int32_t nsec;
+ /** offset in minutes from UTC */
+ int32_t offset;
+};
+
+/**
+ * Date/time interval structure
+ */
+struct datetime_interval {
+ /** relative seconds delta */
+ double secs;
+ /** nanoseconds delta */
+ int32_t nsec;
+};
+
+/**
+ * Convert datetime to string using default asctime format
+ * "Sun Sep 16 01:03:52 1973\n\0"
+ * Wrapper around reenterable asctime_r() version of POSIX function
+ * @param date source datetime value
+ * @sa datetime_ctime
+ */
+char *
+datetime_asctime(const struct datetime *date, char *buf);
+
+char *
+datetime_ctime(const struct datetime *date, char *buf);
+
+size_t
+datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
+ uint32_t len);
+
+void
+datetime_now(struct datetime * now);
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
new file mode 100644
index 000000000..ce579828f
--- /dev/null
+++ b/src/lua/datetime.lua
@@ -0,0 +1,500 @@
+local ffi = require('ffi')
+
+ffi.cdef [[
+
+ /*
+ `c-dt` library functions handles properly both positive and negative `dt`
+ values, where `dt` is a number of dates since Rata Die date (0001-01-01).
+
+ For better compactness of our typical data in MessagePack stream we shift
+ root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
+ actually dt = 719163.
+
+ So here is a simple formula how convert our epoch-based seconds to dt values
+ dt = (secs / 86400) + 719163
+ Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
+ (0001-01-01) in dates.
+
+ */
+ typedef int dt_t;
+
+ // dt_core.h
+ dt_t tnt_dt_from_rdn (int n);
+ dt_t tnt_dt_from_ymd (int y, int m, int d);
+
+ int tnt_dt_rdn (dt_t dt);
+
+ // dt_parse_iso.h
+ size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
+ size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec);
+ size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset);
+
+ // datetime.c
+ int
+ datetime_to_string(const struct datetime * date, char *buf, uint32_t len);
+
+ char *
+ datetime_asctime(const struct datetime *date, char *buf);
+
+ char *
+ datetime_ctime(const struct datetime *date, char *buf);
+
+ size_t
+ datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
+ uint32_t len);
+
+ void
+ datetime_now(struct datetime * now);
+
+]]
+
+local builtin = ffi.C
+local math_modf = math.modf
+
+local SECS_PER_DAY = 86400
+
+-- c-dt/dt_config.h
+
+-- Unix, January 1, 1970, Thursday
+local DT_EPOCH_1970_OFFSET = 719163
+
+
+local datetime_t = ffi.typeof('struct datetime')
+local interval_t = ffi.typeof('struct datetime_interval')
+
+local function is_interval(o)
+ return type(o) == 'cdata' and ffi.istype(interval_t, o)
+end
+
+local function is_datetime(o)
+ return type(o) == 'cdata' and ffi.istype(datetime_t, o)
+end
+
+local function is_date_interval(o)
+ return type(o) == 'cdata' and
+ (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
+end
+
+local function interval_new()
+ local interval = ffi.new(interval_t)
+ return interval
+end
+
+local function check_date(o, message)
+ if not is_datetime(o) then
+ return error(("%s: expected datetime, but received %s"):
+ format(message, o), 2)
+ end
+end
+
+local function check_str(s, message)
+ if not type(s) == 'string' then
+ return error(("%s: expected string, but received %s"):
+ format(message, s), 2)
+ end
+end
+
+local function check_range(v, range, txt)
+ assert(#range == 2)
+ if v < range[1] or v > range[2] then
+ error(('value %d of %s is out of allowed range [%d, %d]'):
+ format(v, txt, range[1], range[2]), 4)
+ end
+end
+
+local function datetime_cmp(lhs, rhs)
+ if not is_date_interval(lhs) or
+ not is_date_interval(rhs) then
+ return nil
+ end
+ local sdiff = lhs.secs - rhs.secs
+ return sdiff ~= 0 and sdiff or (lhs.nsec - rhs.nsec)
+end
+
+local function datetime_eq(lhs, rhs)
+ local rc = datetime_cmp(lhs, rhs)
+ return rc ~= nil and rc == 0
+end
+
+local function datetime_lt(lhs, rhs)
+ local rc = datetime_cmp(lhs, rhs)
+ return rc == nil and error('incompatible types for comparison', 2) or
+ rc < 0
+end
+
+local function datetime_le(lhs, rhs)
+ local rc = datetime_cmp(lhs, rhs)
+ return rc == nil and error('incompatible types for comparison', 2) or
+ rc <= 0
+end
+
+local function datetime_serialize(self)
+ return { secs = self.secs, nsec = self.nsec, offset = self.offset }
+end
+
+local function interval_serialize(self)
+ return { secs = self.secs, nsec = self.nsec }
+end
+
+local function datetime_new_raw(secs, nsec, offset)
+ local dt_obj = ffi.new(datetime_t)
+ dt_obj.secs = secs
+ dt_obj.nsec = nsec
+ dt_obj.offset = offset
+ return dt_obj
+end
+
+local function datetime_new_dt(dt, secs, frac, offset)
+ local epochV = dt ~= nil and (builtin.tnt_dt_rdn(dt) - DT_EPOCH_1970_OFFSET) *
+ SECS_PER_DAY or 0
+ local secsV = secs ~= nil and secs or 0
+ local fracV = frac ~= nil and frac or 0
+ local ofsV = offset ~= nil and offset or 0
+ return datetime_new_raw(epochV + secsV - ofsV * 60, fracV, ofsV)
+end
+
+-- create datetime given attribute values from obj
+-- in the "easy mode", providing builder with
+-- .secs, .nsec, .offset
+local function datetime_new_obj(obj, ...)
+ if obj == nil or type(obj) ~= 'table' then
+ return datetime_new_raw(obj, ...)
+ end
+ local secs = 0
+ local nsec = 0
+ local offset = 0
+
+ for key, value in pairs(obj) do
+ if key == 'secs' then
+ secs = value
+ elseif key == 'nsec' then
+ nsec = value
+ elseif key == 'offset' then
+ offset = value
+ else
+ error(('unknown attribute %s'):format(key), 2)
+ end
+ end
+
+ return datetime_new_raw(secs, nsec, offset)
+end
+
+-- create datetime given attribute values from obj
+local function datetime_new(obj)
+ if obj == nil or type(obj) ~= 'table' then
+ return datetime_new_raw(0, 0, 0)
+ end
+ local y = 0
+ local M = 0
+ local d = 0
+ local ymd = false
+
+ local h = 0
+ local m = 0
+ local s = 0
+ local frac = 0
+ local hms = false
+ local offset = 0
+
+ local dt = 0
+
+ for key, value in pairs(obj) do
+ if key == 'year' then
+ check_range(value, {1, 9999}, key)
+ y = value
+ ymd = true
+ elseif key == 'month' then
+ check_range(value, {1, 12}, key)
+ M = value
+ ymd = true
+ elseif key == 'day' then
+ check_range(value, {1, 31}, key)
+ d = value
+ ymd = true
+ elseif key == 'hour' then
+ check_range(value, {0, 23}, key)
+ h = value
+ hms = true
+ elseif key == 'min' or key == 'minute' then
+ check_range(value, {0, 59}, key)
+ m = value
+ hms = true
+ elseif key == 'sec' or key == 'second' then
+ check_range(value, {0, 60}, key)
+ s, frac = math_modf(value)
+ frac = frac * 1e9 -- convert fraction to nanoseconds
+ hms = true
+ elseif key == 'tz' then
+ -- tz offset in minutes
+ check_range(value, {0, 720}, key)
+ offset = value
+ elseif key == 'isdst' or key == 'wday' or key =='yday' then -- luacheck: ignore 542
+ -- ignore unused os.date attributes
+ else
+ error(('unknown attribute %s'):format(key), 2)
+ end
+ end
+
+ -- .year, .month, .day
+ if ymd then
+ dt = builtin.tnt_dt_from_ymd(y, M, d)
+ end
+
+ -- .hour, .minute, .second
+ local secs = 0
+ if hms then
+ secs = h * 3600 + m * 60 + s
+ end
+
+ return datetime_new_dt(dt, secs, frac, offset)
+end
+
+--[[
+ Basic Extended
+ 20121224 2012-12-24 Calendar date (ISO 8601)
+ 2012359 2012-359 Ordinal date (ISO 8601)
+ 2012W521 2012-W52-1 Week date (ISO 8601)
+ 2012Q485 2012-Q4-85 Quarter date
+]]
+
+local function parse_date(str)
+ check_str("datetime.parse_date()")
+ local dt = ffi.new('dt_t[1]')
+ local len = builtin.tnt_dt_parse_iso_date(str, #str, dt)
+ return len > 0 and datetime_new_dt(dt[0]) or nil, tonumber(len)
+end
+
+--[[
+ Basic Extended
+ T12 N/A
+ T1230 T12:30
+ T123045 T12:30:45
+ T123045.123456789 T12:30:45.123456789
+ T123045,123456789 T12:30:45,123456789
+
+ The time designator [T] may be omitted.
+]]
+local function parse_time(str)
+ check_str("datetime.parse_time()")
+ local sp = ffi.new('int[1]')
+ local fp = ffi.new('int[1]')
+ local len = builtin.tnt_dt_parse_iso_time(str, #str, sp, fp)
+ return len > 0 and datetime_new_dt(nil, sp[0], fp[0]) or nil,
+ tonumber(len)
+end
+
+--[[
+ Basic Extended
+ Z N/A
+ +hh N/A
+ -hh N/A
+ +hhmm +hh:mm
+ -hhmm -hh:mm
+]]
+local function parse_zone(str)
+ check_str("datetime.parse_zone()")
+ local offset = ffi.new('int[1]')
+ local len = builtin.tnt_dt_parse_iso_zone_lenient(str, #str, offset)
+ return len > 0 and datetime_new_dt(nil, nil, nil, offset[0]) or nil,
+ tonumber(len)
+end
+
+
+--[[
+ aggregated parse functions
+ assumes to deal with date T time time_zone
+ at once
+
+ date [T] time [ ] time_zone
+]]
+local function parse(str)
+ check_str("datetime.parse()")
+ local dt = ffi.new('dt_t[1]')
+ local len = #str
+ local n = builtin.tnt_dt_parse_iso_date(str, len, dt)
+ local dt_ = dt[0]
+ if n == 0 or len == n then
+ return datetime_new_dt(dt_)
+ end
+
+ str = str:sub(tonumber(n) + 1)
+
+ local ch = str:sub(1,1)
+ if ch:match('[Tt ]') == nil then
+ return datetime_new_dt(dt_)
+ end
+
+ str = str:sub(2)
+ len = #str
+
+ local sp = ffi.new('int[1]')
+ local fp = ffi.new('int[1]')
+ local n = builtin.tnt_dt_parse_iso_time(str, len, sp, fp)
+ if n == 0 then
+ return datetime_new_dt(dt_)
+ end
+ local sp_ = sp[0]
+ local fp_ = fp[0]
+ if len == n then
+ return datetime_new_dt(dt_, sp_, fp_)
+ end
+
+ str = str:sub(tonumber(n) + 1)
+
+ if str:sub(1,1) == ' ' then
+ str = str:sub(2)
+ end
+
+ len = #str
+
+ local offset = ffi.new('int[1]')
+ n = builtin.tnt_dt_parse_iso_zone_lenient(str, len, offset)
+ if n == 0 then
+ return datetime_new_dt(dt_, sp_, fp_)
+ end
+ return datetime_new_dt(dt_, sp_, fp_, offset[0])
+end
+
+local function datetime_from(o)
+ if o == nil or type(o) == 'table' then
+ return datetime_new(o)
+ elseif type(o) == 'string' then
+ return parse(o)
+ end
+end
+
+local function local_now()
+ local d = datetime_new_raw(0, 0, 0)
+ builtin.datetime_now(d)
+ return d
+end
+
+-- Change the time-zone to the provided target_offset
+-- Time `.secs`/`.nsec` are always UTC normalized, we need only to
+-- reattribute object with different `.offset`
+local function datetime_to_tz(self, tgt_ofs)
+ if self.offset == tgt_ofs then
+ return self
+ end
+ if type(tgt_ofs) == 'string' then
+ local obj = parse_zone(tgt_ofs)
+ if obj == nil then
+ error(('%s: invalid time-zone format %s'):format(self, tgt_ofs), 2)
+ else
+ tgt_ofs = obj.offset
+ end
+ end
+ return datetime_new_raw(self.secs, self.nsec, tgt_ofs)
+end
+
+local function datetime_index(self, key)
+ if key == 'epoch' or key == 'unixtime' then
+ return self.secs
+ elseif key == 'ts' or key == 'timestamp' then
+ return self.secs + self.nsec / 1e9
+ elseif key == 'ns' or key == 'nanoseconds' then
+ return self.secs * 1e9 + self.nsec
+ elseif key == 'us' or key == 'microseconds' then
+ return self.secs * 1e6 + self.nsec / 1e3
+ elseif key == 'ms' or key == 'milliseconds' then
+ return self.secs * 1e3 + self.nsec / 1e6
+ elseif key == 's' or key == 'seconds' then
+ return self.secs + self.nsec / 1e9
+ elseif key == 'm' or key == 'min' or key == 'minutes' then
+ return (self.secs + self.nsec / 1e9) / 60
+ elseif key == 'hr' or key == 'hours' then
+ return (self.secs + self.nsec / 1e9) / (60 * 60)
+ elseif key == 'd' or key == 'days' then
+ return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
+ elseif key == 'to_utc' then
+ return function(self)
+ return datetime_to_tz(self, 0)
+ end
+ elseif key == 'to_tz' then
+ return function(self, offset)
+ return datetime_to_tz(self, offset)
+ end
+ else
+ error(('unknown attribute %s'):format(key), 2)
+ end
+end
+
+local function datetime_newindex(self, key, value)
+ if key == 'epoch' or key == 'unixtime' then
+ self.secs = value
+ self.nsec, self.offset = 0, 0
+ elseif key == 'ts' or key == 'timestamp' then
+ local secs, frac = math_modf(value)
+ self.secs = secs
+ self.nsec = frac * 1e9
+ self.offset = 0
+ else
+ error(('assigning to unknown attribute %s'):format(key), 2)
+ end
+end
+
+-- sizeof("Wed Jun 30 21:49:08 1993\n")
+local buf_len = 26
+
+local function asctime(o)
+ check_date(o, "datetime:asctime()")
+ local buf = ffi.new('char[?]', buf_len)
+ return ffi.string(builtin.datetime_asctime(o, buf))
+end
+
+local function ctime(o)
+ check_date(o, "datetime:ctime()")
+ local buf = ffi.new('char[?]', buf_len)
+ return ffi.string(builtin.datetime_ctime(o, buf))
+end
+
+local function strftime(fmt, o)
+ check_date(o, "datetime.strftime()")
+ local sz = 128
+ local buff = ffi.new('char[?]', sz)
+ builtin.datetime_strftime(o, fmt, buff, sz)
+ return ffi.string(buff)
+end
+
+local datetime_mt = {
+ __serialize = datetime_serialize,
+ __eq = datetime_eq,
+ __lt = datetime_lt,
+ __le = datetime_le,
+ __index = datetime_index,
+ __newindex = datetime_newindex,
+}
+
+local interval_mt = {
+ __serialize = interval_serialize,
+ __eq = datetime_eq,
+ __lt = datetime_lt,
+ __le = datetime_le,
+ __index = datetime_index,
+}
+
+ffi.metatype(interval_t, interval_mt)
+ffi.metatype(datetime_t, datetime_mt)
+
+return setmetatable(
+ {
+ new = datetime_new,
+ new_raw = datetime_new_obj,
+ interval = interval_new,
+
+ parse = parse,
+ parse_date = parse_date,
+ parse_time = parse_time,
+ parse_zone = parse_zone,
+
+ now = local_now,
+ strftime = strftime,
+ asctime = asctime,
+ ctime = ctime,
+
+ is_datetime = is_datetime,
+ is_interval = is_interval,
+ }, {
+ __call = function(self, ...) return datetime_from(...) end
+ }
+)
diff --git a/src/lua/init.c b/src/lua/init.c
index f9738025d..127e935d7 100644
--- a/src/lua/init.c
+++ b/src/lua/init.c
@@ -129,7 +129,8 @@ extern char strict_lua[],
parse_lua[],
process_lua[],
humanize_lua[],
- memprof_lua[]
+ memprof_lua[],
+ datetime_lua[]
;
static const char *lua_modules[] = {
@@ -184,6 +185,7 @@ static const char *lua_modules[] = {
"memprof.process", process_lua,
"memprof.humanize", humanize_lua,
"memprof", memprof_lua,
+ "datetime", datetime_lua,
NULL
};
diff --git a/src/lua/utils.c b/src/lua/utils.c
index c71cd4857..2c89326f3 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -48,6 +48,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
uint32_t CTID_CHAR_PTR;
uint32_t CTID_CONST_CHAR_PTR;
uint32_t CTID_UUID;
+uint32_t CTID_DATETIME = 0;
+uint32_t CTID_INTERVAL = 0;
+
void *
luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
@@ -120,6 +123,12 @@ luaL_pushuuidstr(struct lua_State *L, const struct tt_uuid *uuid)
lua_pushlstring(L, str, UUID_STR_LEN);
}
+struct datetime *
+luaL_pushdatetime(struct lua_State *L)
+{
+ return luaL_pushcdata(L, CTID_DATETIME);
+}
+
int
luaL_iscdata(struct lua_State *L, int idx)
{
@@ -725,6 +734,24 @@ tarantool_lua_utils_init(struct lua_State *L)
CTID_UUID = luaL_ctypeid(L, "struct tt_uuid");
assert(CTID_UUID != 0);
+ rc = luaL_cdef(L, "struct datetime {"
+ "double secs;"
+ "int32_t nsec;"
+ "int32_t offset;"
+ "};");
+ assert(rc == 0);
+ (void) rc;
+ CTID_DATETIME = luaL_ctypeid(L, "struct datetime");
+ assert(CTID_DATETIME != 0);
+ rc = luaL_cdef(L, "struct datetime_interval {"
+ "double secs;"
+ "int32_t nsec;"
+ "};");
+ assert(rc == 0);
+ (void) rc;
+ CTID_INTERVAL = luaL_ctypeid(L, "struct datetime_interval");
+ assert(CTID_INTERVAL != 0);
+
lua_pushcfunction(L, luaT_newthread_wrapper);
luaT_newthread_ref = luaL_ref(L, LUA_REGISTRYINDEX);
return 0;
diff --git a/src/lua/utils.h b/src/lua/utils.h
index 45070b778..73495b607 100644
--- a/src/lua/utils.h
+++ b/src/lua/utils.h
@@ -59,6 +59,7 @@ struct lua_State;
struct ibuf;
typedef struct ibuf box_ibuf_t;
struct tt_uuid;
+struct datetime;
/**
* Single global lua_State shared by core and modules.
@@ -71,6 +72,8 @@ extern struct lua_State *tarantool_L;
extern uint32_t CTID_CHAR_PTR;
extern uint32_t CTID_CONST_CHAR_PTR;
extern uint32_t CTID_UUID;
+extern uint32_t CTID_DATETIME;
+extern uint32_t CTID_INTERVAL;
struct tt_uuid *
luaL_pushuuid(struct lua_State *L);
@@ -78,6 +81,15 @@ luaL_pushuuid(struct lua_State *L);
void
luaL_pushuuidstr(struct lua_State *L, const struct tt_uuid *uuid);
+/**
+ * @brief Push cdata of a datetime type onto the stack.
+ * @param L Lua State
+ * @sa luaL_pushcdata
+ * @return memory associated with this datetime data
+ */
+struct datetime *
+luaL_pushdatetime(struct lua_State *L);
+
/** \cond public */
/**
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
new file mode 100755
index 000000000..464d4bd49
--- /dev/null
+++ b/test/app-tap/datetime.test.lua
@@ -0,0 +1,206 @@
+#!/usr/bin/env tarantool
+
+local tap = require('tap')
+local test = tap.test("errno")
+local date = require('datetime')
+local ffi = require('ffi')
+
+
+test:plan(6)
+
+test:test("Simple tests for parser", function(test)
+ test:plan(2)
+ test:ok(date("1970-01-01T01:00:00Z") ==
+ date {year=1970, month=1, day=1, hour=1, minute=0, second=0})
+ test:ok(date("1970-01-01T02:00:00+02:00") ==
+ date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120})
+end)
+
+test:test("Multiple tests for parser (with nanoseconds)", function(test)
+ test:plan(168)
+ -- borrowed from p5-time-moments/t/180_from_string.t
+ local tests =
+ {
+ { '1970-01-01T00:00:00Z', 0, 0, 0 },
+ { '1970-01-01T02:00:00+02:00', 0, 0, 120 },
+ { '1970-01-01T01:30:00+01:30', 0, 0, 90 },
+ { '1970-01-01T01:00:00+01:00', 0, 0, 60 },
+ { '1970-01-01T00:01:00+00:01', 0, 0, 1 },
+ { '1970-01-01T00:00:00+00:00', 0, 0, 0 },
+ { '1969-12-31T23:59:00-00:01', 0, 0, -1 },
+ { '1969-12-31T23:00:00-01:00', 0, 0, -60 },
+ { '1969-12-31T22:30:00-01:30', 0, 0, -90 },
+ { '1969-12-31T22:00:00-02:00', 0, 0, -120 },
+ { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 },
+ { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 },
+ { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 },
+ { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 },
+ { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 },
+ { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 },
+ { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 },
+ { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 },
+ { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 },
+ { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 },
+ { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 },
+ { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 },
+ { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 },
+ { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 },
+ { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 },
+ { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 },
+ { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 },
+ { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 },
+ { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 },
+ { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 },
+ { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 },
+ { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 },
+ { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 },
+ { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 },
+ { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 },
+ { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 },
+ { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 },
+ { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 },
+ { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 },
+ { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 },
+ { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 },
+ { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 },
+ { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 },
+ { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 },
+ { '1970-01-01T00:00:00.0Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.00Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.0000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.00000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.000000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 },
+ { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 },
+ { '1973-11-29T21:33:09Z', 123456789, 0, 0 },
+ { '2013-10-28T17:51:56Z', 1382982716, 0, 0 },
+ { '9999-12-31T23:59:59Z', 253402300799, 0, 0 },
+ }
+ for _, value in ipairs(tests) do
+ local str, epoch, nsec, offset
+ str, epoch, nsec, offset = unpack(value)
+ local dt = date(str)
+ test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
+ test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
+ test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
+ end
+end)
+
+ffi.cdef [[
+ void tzset(void);
+]]
+
+test:test("Datetime string formatting", function(test)
+ test:plan(7)
+ local str = "1970-01-01"
+ local t = date(str)
+ test:ok(t.secs == 0, ('%s: t.secs == %d'):format(str, tonumber(t.secs)))
+ test:ok(t.nsec == 0, ('%s: t.nsec == %d'):format(str, t.nsec))
+ test:ok(t.offset == 0, ('%s: t.offset == %d'):format(str, t.offset))
+ test:ok(date.asctime(t) == 'Thu Jan 1 00:00:00 1970\n', ('%s: asctime'):format(str))
+ -- ctime() is local timezone dependent. To make sure that
+ -- test is deterministic we enforce timezone via TZ environment
+ -- manipulations and calling tzset()
+
+ -- redefine timezone to be always GMT-2
+ os.setenv('TZ', 'GMT-2')
+ ffi.C.tzset()
+ test:ok(date.ctime(t) == 'Thu Jan 1 02:00:00 1970\n', ('%s: ctime with timezone'):format(str))
+ test:ok(date.strftime('%d/%m/%Y', t) == '01/01/1970', ('%s: strftime #1'):format(str))
+ test:ok(date.strftime('%A %d. %B %Y', t) == 'Thursday 01. January 1970', ('%s: strftime #2'):format(str))
+end)
+
+test:test("Parse iso date - valid strings", function(test)
+ test:plan(32)
+ local good = {
+ {2012, 12, 24, "20121224", 8 },
+ {2012, 12, 24, "20121224 Foo bar", 8 },
+ {2012, 12, 24, "2012-12-24", 10 },
+ {2012, 12, 24, "2012-12-24 23:59:59", 10 },
+ {2012, 12, 24, "2012-12-24T00:00:00+00:00", 10 },
+ {2012, 12, 24, "2012359", 7 },
+ {2012, 12, 24, "2012359T235959+0130", 7 },
+ {2012, 12, 24, "2012-359", 8 },
+ {2012, 12, 24, "2012W521", 8 },
+ {2012, 12, 24, "2012-W52-1", 10 },
+ {2012, 12, 24, "2012Q485", 8 },
+ {2012, 12, 24, "2012-Q4-85", 10 },
+ { 1, 1, 1, "0001-Q1-01", 10 },
+ { 1, 1, 1, "0001-W01-1", 10 },
+ { 1, 1, 1, "0001-01-01", 10 },
+ { 1, 1, 1, "0001-001", 8 },
+ }
+
+ for _, value in ipairs(good) do
+ local year, month, day, str, date_part_len
+ year, month, day, str, date_part_len = unpack(value)
+ local expected_date = date{year = year, month = month, day = day}
+ local date_part, len
+ date_part, len = date.parse_date(str)
+ test:ok(len == date_part_len, ('%s: length check %d'):format(str, len))
+ test:ok(expected_date == date_part, ('%s: expected date'):format(str))
+ end
+end)
+
+test:test("Parse iso date - invalid strings", function(test)
+ test:plan(62)
+ local bad = {
+ "20121232" , -- Invalid day of month
+ "2012-12-310", -- Invalid day of month
+ "2012-13-24" , -- Invalid month
+ "2012367" , -- Invalid day of year
+ "2012-000" , -- Invalid day of year
+ "2012W533" , -- Invalid week of year
+ "2012-W52-8" , -- Invalid day of week
+ "2012Q495" , -- Invalid day of quarter
+ "2012-Q5-85" , -- Invalid quarter
+ "20123670" , -- Trailing digit
+ "201212320" , -- Trailing digit
+ "2012-12" , -- Reduced accuracy
+ "2012-Q4" , -- Reduced accuracy
+ "2012-Q42" , -- Invalid
+ "2012-Q1-1" , -- Invalid day of quarter
+ "2012Q--420" , -- Invalid
+ "2012-Q-420" , -- Invalid
+ "2012Q11" , -- Incomplete
+ "2012Q1234" , -- Trailing digit
+ "2012W12" , -- Incomplete
+ "2012W1234" , -- Trailing digit
+ "2012W-123" , -- Invalid
+ "2012-W12" , -- Incomplete
+ "2012-W12-12", -- Trailing digit
+ "2012U1234" , -- Invalid
+ "2012-1234" , -- Invalid
+ "2012-X1234" , -- Invalid
+ "0000-Q1-01" , -- Year less than 0001
+ "0000-W01-1" , -- Year less than 0001
+ "0000-01-01" , -- Year less than 0001
+ "0000-001" , -- Year less than 0001
+ }
+
+ for _, str in ipairs(bad) do
+ local date_part, len
+ date_part, len = date.parse_date(str)
+ test:ok(len == 0, ('%s: length check %d'):format(str, len))
+ test:ok(date_part == nil, ('%s: empty date check %s'):format(str, date_part))
+ end
+end)
+
+test:test("Parse tiny date into seconds and other parts", function(test)
+ test:plan(9)
+ local str = '19700101 00:00:30.528'
+ local tiny = date(str)
+ test:ok(tiny.secs == 30, ("secs of '%s'"):format(str))
+ test:ok(tiny.nsec == 528000000, ("nsec of '%s'"):format(str))
+ test:ok(tiny.nanoseconds == 30528000000, "nanoseconds")
+ test:ok(tiny.microseconds == 30528000, "microseconds")
+ test:ok(tiny.milliseconds == 30528, "milliseconds")
+ test:ok(tiny.seconds == 30.528, "seconds")
+ test:ok(tiny.timestamp == 30.528, "timestamp")
+ test:ok(tiny.minutes == 0.5088, "minutes")
+ test:ok(tiny.hours == 0.00848, "hours")
+end)
+
+os.exit(test:check() and 0 or 1)
diff --git a/test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua b/test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua
index 84c2944e5..7eb05f9e8 100755
--- a/test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua
+++ b/test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua
@@ -14,6 +14,7 @@ local uri = require('uri')
local json = require('json')
local msgpackffi = require('msgpackffi')
local decimal = require('decimal')
+local datetime = require('datetime')
local function test_uuid(test)
test:plan(1)
@@ -256,15 +257,86 @@ local function test_decimal(test)
test:ok(is_success, 'decimal str in gc')
end
+local function test_datetime_asctime(test)
+ test:plan(1)
+
+ local gc_count = 100
+ local iter_count = 1000
+ local is_success = true
+
+ local T1 = datetime('1970-01-01')
+ local T2 = datetime('2000-01-01')
+
+ local function datetime_asctime()
+ local str1 = datetime.asctime(T1)
+ local str2 = datetime.asctime(T2)
+ local str3 = datetime.asctime(T1)
+ local str4 = datetime.asctime(T2)
+ if str1 ~= str3 or str2 ~= str4 then
+ is_success = false
+ end
+ end
+
+ local function create_gc()
+ for _ = 1, gc_count do
+ ffi.gc(ffi.new('char[1]'), function() datetime_asctime() end)
+ end
+ end
+
+ for _ = 1, iter_count do
+ create_gc()
+ datetime_asctime()
+ end
+
+ test:ok(is_success, 'info datetime in gc')
+end
+
+local function test_datetime_ctime(test)
+ test:plan(1)
+
+ local gc_count = 100
+ local iter_count = 1000
+ local is_success = true
+
+ local T1 = datetime('1970-01-01')
+ local T2 = datetime('2000-01-01')
+
+ local function datetime_ctime()
+ local str1 = datetime.ctime(T1)
+ local str2 = datetime.ctime(T2)
+ local str3 = datetime.ctime(T1)
+ local str4 = datetime.ctime(T2)
+ if str1 ~= str3 or str2 ~= str4 then
+ is_success = false
+ end
+ end
+
+ local function create_gc()
+ for _ = 1, gc_count do
+ ffi.gc(ffi.new('char[1]'), function() datetime_ctime() end)
+ end
+ end
+
+ for _ = 1, iter_count do
+ create_gc()
+ datetime_ctime()
+ end
+
+ test:ok(is_success, 'info datetime in gc')
+end
+
+
box.cfg{}
local test = tap.test('gh-5632-6050-6259-gc-buf-reuse')
-test:plan(6)
+test:plan(8)
test:test('uuid in __gc', test_uuid)
test:test('uri in __gc', test_uri)
test:test('msgpackffi in __gc', test_msgpackffi)
test:test('json in __gc', test_json)
test:test('info uuid in __gc', test_info_uuid)
test:test('decimal str in __gc', test_decimal)
+test:test('datetime.asctime in __gc', test_datetime_asctime)
+test:test('datetime.ctime in __gc', test_datetime_ctime)
os.exit(test:check() and 0 or 1)
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime Timur Safin via Tarantool-patches
@ 2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:30 ` Safin Timur via Tarantool-patches
2021-08-17 16:52 ` Vladimir Davydov via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:15 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * created a new Tarantool built-in module `datetime`;
> * register cdef types for this module;
> * export some `dt_*` functions from `c-dt` library;
>
> * implemented `ctime`, `asctime` and `strftime` as
> ffi wrappers in C kernel code:
>
> We used to use heavy ffi-related Lua code, but it appears
> to be quite fragile, if we run this code on different OS
> targets. To make this Lua code less platfrom dependent we
> have moved all platform specifics to the C level.
> Specifically we want to avoid dependence on `struct tm {}`
> layout and GLIBC-specific `strftime` behaviour (which
> differs to BSD/Mac LIBC behaviour).
>
> * introduced simple datetime tests
>
> Created app-tap test for new builtin module `datetime.lua`,
> where we specifically check:
>
> - cases of simple datetime string formatting using:
> - asctime (gmt time);
> - ctime (local TZ time);
> - strftime (using given format).
>
> - verify that we parse expected, and fail with unexpected:
> - for that, in the datetime.lua, we have extended api of `parse_date()`,
> `parse_time()`, and `parse_time_zone()` so they return
> not only parsed object, but also a length of parsed substring;
> - which allows us to parse even _partially_ valid strings like
> "20121224 Foo bar".
>
> - check calculated attributes to date object, e.g.:
> - timestamp, seconds, microseconds, minute, or hours
> - to_utc(), and to_tz() allow to switch timezone of a
> datetime object. It's not changing much - only timezone
> but that impacts textual representation of a date.
>
> Part of #5941
Please, add a docbot request to the commit message.
Here it should say that you introduce lua datetime module
and describe shortly what the module does.
> ---
> cmake/BuildCDT.cmake | 2 +
> extra/exports | 26 +
> src/CMakeLists.txt | 2 +
> src/lib/core/CMakeLists.txt | 1 +
> src/lib/core/datetime.c | 96 ++++
> src/lib/core/datetime.h | 95 ++++
> src/lua/datetime.lua | 500 ++++++++++++++++++
> src/lua/init.c | 4 +-
> src/lua/utils.c | 27 +
> src/lua/utils.h | 12 +
> test/app-tap/datetime.test.lua | 206 ++++++++
> .../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 ++-
> 12 files changed, 1043 insertions(+), 2 deletions(-)
> create mode 100644 src/lib/core/datetime.c
> create mode 100644 src/lib/core/datetime.h
> create mode 100644 src/lua/datetime.lua
> create mode 100755 test/app-tap/datetime.test.lua
>
> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
> index 343fb1b99..80b26c64a 100644
> --- a/cmake/BuildCDT.cmake
> +++ b/cmake/BuildCDT.cmake
> @@ -5,4 +5,6 @@ macro(libccdt_build)
> file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
> add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
> + set_target_properties(cdt PROPERTIES COMPILE_FLAGS "-DDT_NAMESPACE=tnt_")
> + add_definitions("-DDT_NAMESPACE=tnt_")
> endmacro()
This change belongs to the previous commit, doesn't it?
> diff --git a/extra/exports b/extra/exports
> index 9eaba1282..80eb92abd 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -148,8 +148,34 @@ csv_feed
> csv_iterator_create
> csv_next
> csv_setopt
> +datetime_asctime
> +datetime_ctime
> +datetime_now
> +datetime_strftime
> +decimal_unpack
> decimal_from_string
> decimal_unpack
> +tnt_dt_dow
> +tnt_dt_from_rdn
> +tnt_dt_from_struct_tm
> +tnt_dt_from_yd
> +tnt_dt_from_ymd
> +tnt_dt_from_yqd
> +tnt_dt_from_ywd
> +tnt_dt_parse_iso_date
> +tnt_dt_parse_iso_time_basic
> +tnt_dt_parse_iso_time_extended
> +tnt_dt_parse_iso_time
> +tnt_dt_parse_iso_zone_basic
> +tnt_dt_parse_iso_zone_extended
> +tnt_dt_parse_iso_zone_lenient
> +tnt_dt_parse_iso_zone
> +tnt_dt_rdn
> +tnt_dt_to_struct_tm
> +tnt_dt_to_yd
> +tnt_dt_to_ymd
> +tnt_dt_to_yqd
> +tnt_dt_to_ywd
> error_ref
> error_set_prev
> error_unref
> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> index 97b0cb326..4473ff1da 100644
> --- a/src/CMakeLists.txt
> +++ b/src/CMakeLists.txt
> @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
> lua_source(lua_sources lua/httpc.lua)
> lua_source(lua_sources lua/iconv.lua)
> lua_source(lua_sources lua/swim.lua)
> +lua_source(lua_sources lua/datetime.lua)
> +
> # LuaJIT jit.* library
> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
> index 2cd4d0b4f..8bc776b82 100644
> --- a/src/lib/core/CMakeLists.txt
> +++ b/src/lib/core/CMakeLists.txt
> @@ -30,6 +30,7 @@ set(core_sources
> decimal.c
> mp_decimal.c
> cord_buf.c
> + datetime.c
> )
>
> if (TARGET_OS_NETBSD)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> new file mode 100644
> index 000000000..c48295a6f
> --- /dev/null
> +++ b/src/lib/core/datetime.c
> @@ -0,0 +1,96 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
As far as I know, we switched to the following license format recently:
/*
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
*/
See, for example:
./src/box/module_cache.c: * SPDX-License-Identifier: BSD-2-Clause
./src/box/lua/lib.c: * SPDX-License-Identifier: BSD-2-Clause
./src/box/lua/lib.h: * SPDX-License-Identifier: BSD-2-Clause
./src/box/module_cache.h: * SPDX-License-Identifier: BSD-2-Clause
./src/lib/core/cord_buf.c: * SPDX-License-Identifier: BSD-2-Clause
./src/lib/core/crash.c: * SPDX-License-Identifier: BSD-2-Clause
./src/lib/core/cord_buf.h: * SPDX-License-Identifier: BSD-2-Clause
./src/lib/core/crash.h: * SPDX-License-Identifier: BSD-2-Clause
<stripped>
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> new file mode 100644
> index 000000000..1a8d7e34f
> --- /dev/null
> +++ b/src/lib/core/datetime.h
> @@ -0,0 +1,95 @@
> +#pragma once
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
Same about the license.
And I'd move the "#pragma once" below the license comment.
Otherwise it's easily lost. Up to you.
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
AFAICS you don't need stdio included here.
> +#include "c-dt/dt.h"
> +
> +#if defined(__cplusplus)
> +extern "C"
> +{
> +#endif /* defined(__cplusplus) */
> +
> +#ifndef SECS_PER_DAY
> +#define SECS_PER_DAY 86400
> +#define DT_EPOCH_1970_OFFSET 719163
Please, add a short comment on what this is.
I had to spend some time googling to understand.
So, please mention that this is measured in days from 01-01-0001.
> +#endif
> +
> +/**
> + * Full datetime structure representing moments
> + * since Unix Epoch (1970-01-01).
> + * Time is kept normalized to UTC, time-zone offset
> + * is informative only.
> + */
> +struct datetime {
> + /** seconds since epoch */
> + double secs;
> + /** nanoseconds if any */
> + int32_t nsec;
As discussed, let's make nsec a uint32_t, since
nsec part is always positive.
> + /** offset in minutes from UTC */
> + int32_t offset;
> +};
> +
> +/**
> + * Date/time interval structure
> + */
> +struct datetime_interval {
> + /** relative seconds delta */
> + double secs;
> + /** nanoseconds delta */
> + int32_t nsec;
> +};
> +
Please start comments with a capital letter and end them with a dot.
> +/**
> + * Convert datetime to string using default asctime format
> + * "Sun Sep 16 01:03:52 1973\n\0"
> + * Wrapper around reenterable asctime_r() version of POSIX function
> + * @param date source datetime value
> + * @sa datetime_ctime
> + */
> +char *
> +datetime_asctime(const struct datetime *date, char *buf);
> +
> +char *
> +datetime_ctime(const struct datetime *date, char *buf);
> +
> +size_t
> +datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> + uint32_t len);
> +
> +void
> +datetime_now(struct datetime * now);
> +
> +#if defined(__cplusplus)
> +} /* extern "C" */
> +#endif /* defined(__cplusplus) */
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> new file mode 100644
> index 000000000..ce579828f
> --- /dev/null
> +++ b/src/lua/datetime.lua
> @@ -0,0 +1,500 @@
> +local ffi = require('ffi')
> +
> +ffi.cdef [[
> +
> + /*
> + `c-dt` library functions handles properly both positive and negative `dt`
> + values, where `dt` is a number of dates since Rata Die date (0001-01-01).
> +
> + For better compactness of our typical data in MessagePack stream we shift
> + root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
> + actually dt = 719163.
> +
> + So here is a simple formula how convert our epoch-based seconds to dt values
> + dt = (secs / 86400) + 719163
> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
> + (0001-01-01) in dates.
> +
> + */
I'd move the comments outside the ffi.cdef block. This way they'd get
proper highlighting, and it would be harder to mess something up
by accidentally deleting the "*/"
> + typedef int dt_t;
> +
> + // dt_core.h
> + dt_t tnt_dt_from_rdn (int n);
> + dt_t tnt_dt_from_ymd (int y, int m, int d);
> +
> + int tnt_dt_rdn (dt_t dt);
> +
> + // dt_parse_iso.h
> + size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
> + size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec);
> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset);
> +
> + // datetime.c
Also you may split the definitions into multiple ffi.cdef[[]] blocks
if you want to add some per-definition comments.
> + int
> + datetime_to_string(const struct datetime * date, char *buf, uint32_t len);
> +
> + char *
> + datetime_asctime(const struct datetime *date, char *buf);
> +
> + char *
> + datetime_ctime(const struct datetime *date, char *buf);
> +
> + size_t
> + datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> + uint32_t len);
> +
> + void
> + datetime_now(struct datetime * now);
> +
> +]]
<stripped>
> diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
> new file mode 100755
> index 000000000..464d4bd49
> --- /dev/null
> +++ b/test/app-tap/datetime.test.lua
> @@ -0,0 +1,206 @@
> +#!/usr/bin/env tarantool
> +
> +local tap = require('tap')
> +local test = tap.test("errno")
> +local date = require('datetime')
> +local ffi = require('ffi')
> +
> +
> +test:plan(6)
> +
> +test:test("Simple tests for parser", function(test)
> + test:plan(2)
> + test:ok(date("1970-01-01T01:00:00Z") ==
> + date {year=1970, month=1, day=1, hour=1, minute=0, second=0})
> + test:ok(date("1970-01-01T02:00:00+02:00") ==
> + date {year=1970, month=1, day=1, hour=2, minute=0, second=0, tz=120})
> +end)
> +
> +test:test("Multiple tests for parser (with nanoseconds)", function(test)
> + test:plan(168)
> + -- borrowed from p5-time-moments/t/180_from_string.t
> + local tests =
> + {
> + { '1970-01-01T00:00:00Z', 0, 0, 0 },
> + { '1970-01-01T02:00:00+02:00', 0, 0, 120 },
> + { '1970-01-01T01:30:00+01:30', 0, 0, 90 },
> + { '1970-01-01T01:00:00+01:00', 0, 0, 60 },
> + { '1970-01-01T00:01:00+00:01', 0, 0, 1 },
> + { '1970-01-01T00:00:00+00:00', 0, 0, 0 },
> + { '1969-12-31T23:59:00-00:01', 0, 0, -1 },
> + { '1969-12-31T23:00:00-01:00', 0, 0, -60 },
> + { '1969-12-31T22:30:00-01:30', 0, 0, -90 },
> + { '1969-12-31T22:00:00-02:00', 0, 0, -120 },
> + { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 },
> + { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 },
> + { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 },
> + { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 },
> + { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 },
> + { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 },
> + { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 },
> + { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 },
> + { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 },
> + { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 },
> + { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 },
> + { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 },
> + { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 },
> + { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 },
> + { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 },
> + { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 },
> + { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 },
> + { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 },
> + { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 },
> + { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 },
> + { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 },
> + { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 },
> + { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 },
> + { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 },
> + { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 },
> + { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 },
> + { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 },
> + { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 },
> + { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 },
> + { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 },
> + { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 },
> + { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 },
> + { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 },
> + { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 },
Красивое :)
> + { '1970-01-01T00:00:00.0Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.00Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.0000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.00000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.000000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 },
> + { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 },
> + { '1973-11-29T21:33:09Z', 123456789, 0, 0 },
> + { '2013-10-28T17:51:56Z', 1382982716, 0, 0 },
> + { '9999-12-31T23:59:59Z', 253402300799, 0, 0 },
> + }
> + for _, value in ipairs(tests) do
> + local str, epoch, nsec, offset
> + str, epoch, nsec, offset = unpack(value)
> + local dt = date(str)
> + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
> + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
> + test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
> + end
> +end)
> +
> +ffi.cdef [[
> + void tzset(void);
> +]]
> +
>
<stripped>
>
>
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:30 ` Safin Timur via Tarantool-patches
2021-08-18 8:56 ` Serge Petrenko via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:30 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
On 17.08.2021 15:15, Serge Petrenko wrote:
>
>
>> - check calculated attributes to date object, e.g.:
>> - timestamp, seconds, microseconds, minute, or hours
>> - to_utc(), and to_tz() allow to switch timezone of a
>> datetime object. It's not changing much - only timezone
>> but that impacts textual representation of a date.
>>
>> Part of #5941
>
> Please, add a docbot request to the commit message.
> Here it should say that you introduce lua datetime module
> and describe shortly what the module does.
I'm planning to cheat here - to not put detailed description to docbot
request part here, but rather to refer to the externally available
documentation I've written in
https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
Like:
@TarantoolBot document
Title: Introduced new built-in `datetime` module
`datetime` module has been introduced, which allows to parse ISO-8601
literals representing timestamps of various formats, and then manipulate
with date objects.
Please refer to https://github.com/tarantool/tarantool/discussions/6244
for more detailed description of module API.
>
>> ---
>> cmake/BuildCDT.cmake | 2 +
>> extra/exports | 26 +
>> src/CMakeLists.txt | 2 +
>> src/lib/core/CMakeLists.txt | 1 +
>> src/lib/core/datetime.c | 96 ++++
>> src/lib/core/datetime.h | 95 ++++
>> src/lua/datetime.lua | 500 ++++++++++++++++++
>> src/lua/init.c | 4 +-
>> src/lua/utils.c | 27 +
>> src/lua/utils.h | 12 +
>> test/app-tap/datetime.test.lua | 206 ++++++++
>> .../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 ++-
>> 12 files changed, 1043 insertions(+), 2 deletions(-)
>> create mode 100644 src/lib/core/datetime.c
>> create mode 100644 src/lib/core/datetime.h
>> create mode 100644 src/lua/datetime.lua
>> create mode 100755 test/app-tap/datetime.test.lua
>>
>> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
>> index 343fb1b99..80b26c64a 100644
>> --- a/cmake/BuildCDT.cmake
>> +++ b/cmake/BuildCDT.cmake
>> @@ -5,4 +5,6 @@ macro(libccdt_build)
>> file(MAKE_DIRECTORY
>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>> add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
>>
>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>> + set_target_properties(cdt PROPERTIES COMPILE_FLAGS
>> "-DDT_NAMESPACE=tnt_")
>> + add_definitions("-DDT_NAMESPACE=tnt_")
>> endmacro()
>
>
> This change belongs to the previous commit, doesn't it?
I considered that, but decided that for better observability purposes I
want to keep this rename of symbols via `tnt_` prefix closer to the code
where we exports those functions (please see `tnt_dt_dow` and others
below in the /extra/exports).
[But taking into account that we gonna squash these commits I don't care
that much now]
>
>
>> diff --git a/extra/exports b/extra/exports
>> index 9eaba1282..80eb92abd 100644
>> --- a/extra/exports
>> +++ b/extra/exports
>> @@ -148,8 +148,34 @@ csv_feed
>> csv_iterator_create
>> csv_next
>> csv_setopt
>> +datetime_asctime
>> +datetime_ctime
>> +datetime_now
>> +datetime_strftime
>> +decimal_unpack
>> decimal_from_string
>> decimal_unpack
These guys renamed here
|
V
>> +tnt_dt_dow
>> +tnt_dt_from_rdn
>> +tnt_dt_from_struct_tm
>> +tnt_dt_from_yd
>> +tnt_dt_from_ymd
>> +tnt_dt_from_yqd
>> +tnt_dt_from_ywd
>> +tnt_dt_parse_iso_date
>> +tnt_dt_parse_iso_time_basic
>> +tnt_dt_parse_iso_time_extended
>> +tnt_dt_parse_iso_time
>> +tnt_dt_parse_iso_zone_basic
>> +tnt_dt_parse_iso_zone_extended
>> +tnt_dt_parse_iso_zone_lenient
>> +tnt_dt_parse_iso_zone
>> +tnt_dt_rdn
>> +tnt_dt_to_struct_tm
>> +tnt_dt_to_yd
>> +tnt_dt_to_ymd
>> +tnt_dt_to_yqd
>> +tnt_dt_to_ywd
>> error_ref
>> error_set_prev
>> error_unref
>> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
>> index 97b0cb326..4473ff1da 100644
>> --- a/src/CMakeLists.txt
>> +++ b/src/CMakeLists.txt
>> @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
>> lua_source(lua_sources lua/httpc.lua)
>> lua_source(lua_sources lua/iconv.lua)
>> lua_source(lua_sources lua/swim.lua)
>> +lua_source(lua_sources lua/datetime.lua)
>> +
>> # LuaJIT jit.* library
>> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
>> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
>> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
>> index 2cd4d0b4f..8bc776b82 100644
>> --- a/src/lib/core/CMakeLists.txt
>> +++ b/src/lib/core/CMakeLists.txt
>> @@ -30,6 +30,7 @@ set(core_sources
>> decimal.c
>> mp_decimal.c
>> cord_buf.c
>> + datetime.c
>> )
>> if (TARGET_OS_NETBSD)
>> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
>> new file mode 100644
>> index 000000000..c48295a6f
>> --- /dev/null
>> +++ b/src/lib/core/datetime.c
>> @@ -0,0 +1,96 @@
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>
> As far as I know, we switched to the following license format recently:
>
> /*
> * SPDX-License-Identifier: BSD-2-Clause
> *
> * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
> */
Much shorter - I like it. Updated.
>
> See, for example:
>
> ./src/box/module_cache.c: * SPDX-License-Identifier: BSD-2-Clause
> ./src/box/lua/lib.c: * SPDX-License-Identifier: BSD-2-Clause
> ./src/box/lua/lib.h: * SPDX-License-Identifier: BSD-2-Clause
> ./src/box/module_cache.h: * SPDX-License-Identifier: BSD-2-Clause
> ./src/lib/core/cord_buf.c: * SPDX-License-Identifier: BSD-2-Clause
> ./src/lib/core/crash.c: * SPDX-License-Identifier: BSD-2-Clause
> ./src/lib/core/cord_buf.h: * SPDX-License-Identifier: BSD-2-Clause
> ./src/lib/core/crash.h: * SPDX-License-Identifier: BSD-2-Clause
>
>
> <stripped>
>
>
>> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
>> new file mode 100644
>> index 000000000..1a8d7e34f
>> --- /dev/null
>> +++ b/src/lib/core/datetime.h
>> @@ -0,0 +1,95 @@
>> +#pragma once
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>
> Same about the license.
Updated
>
> And I'd move the "#pragma once" below the license comment.
> Otherwise it's easily lost. Up to you.
On one side I agreed that it might left unrecognizable for untrained
eye. But on other side - the sooner compiler/preprocessor will see, the
better :)
[And there is already well established convention to put it at the first
line.]
So I've left it on the 1st line.
>
>> +#include <stdint.h>
>> +#include <stdbool.h>
>> +#include <stdio.h>
>
> AFAICS you don't need stdio included here.
Indeed!
>
>> +#include "c-dt/dt.h"
>> +
>> +#if defined(__cplusplus)
>> +extern "C"
>> +{
>> +#endif /* defined(__cplusplus) */
>> +
>> +#ifndef SECS_PER_DAY
>> +#define SECS_PER_DAY 86400
>> +#define DT_EPOCH_1970_OFFSET 719163
>
> Please, add a short comment on what this is.
> I had to spend some time googling to understand.
>
> So, please mention that this is measured in days from 01-01-0001.
I've written some explanation about these magic numbers. Now it's
verboser a bit:
--------------------------------------------------------
/**
* We count dates since so called "Rata Die" date
* January 1, 0001, Monday (as Day 1).
* But datetime structure keeps seconds since
* Unix "Epoch" date:
* Unix, January 1, 1970, Thursday
*
* The difference between Epoch (1970-01-01)
* and Rata Die (0001-01-01) is 719163 days.
*/
#ifndef SECS_PER_DAY
#define SECS_PER_DAY 86400
#define DT_EPOCH_1970_OFFSET 719163
#endif
--------------------------------------------------------
also, for the MP-related patch, I've added this comment, and defines
(might be used in asserts):
--------------------------------------------------------
/**
* c-dt library uses int as type for dt value, which
* represents the number of days since Rata Die date.
* This implies limits to the number of seconds we
* could safely store in our structures and then safely
* pass to c-dt functions.
*
* So supported ranges will be
* - for seconds [-185604722870400 .. 185480451417600]
* - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
*/
#define MAX_DT_DAY_VALUE (int64_t)INT_MAX
#define MIN_DT_DAY_VALUE (int64_t)INT_MIN
#define SECS_EPOCH_1970_OFFSET \
((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
#define MAX_EPOCH_SECS_VALUE \
(MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
#define MIN_EPOCH_SECS_VALUE \
(MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
--------------------------------------------------------
>
>> +#endif
>> +
>> +/**
>> + * Full datetime structure representing moments
>> + * since Unix Epoch (1970-01-01).
>> + * Time is kept normalized to UTC, time-zone offset
>> + * is informative only.
>> + */
>> +struct datetime {
>> + /** seconds since epoch */
>> + double secs;
>> + /** nanoseconds if any */
>> + int32_t nsec;
>
>
> As discussed, let's make nsec a uint32_t, since
> nsec part is always positive.
Changed.
>
>
>> + /** offset in minutes from UTC */
>> + int32_t offset;
>> +};
>> +
>> +/**
>> + * Date/time interval structure
>> + */
>> +struct datetime_interval {
>> + /** relative seconds delta */
>> + double secs;
>> + /** nanoseconds delta */
>> + int32_t nsec;
>> +};
>> +
>
>
> Please start comments with a capital letter and end them with a dot.
Done.
>
>
>> +/**
>> + * Convert datetime to string using default asctime format
>> + * "Sun Sep 16 01:03:52 1973\n\0"
>> + * Wrapper around reenterable asctime_r() version of POSIX function
>> + * @param date source datetime value
>> + * @sa datetime_ctime
>> + */
>> +char *
>> +datetime_asctime(const struct datetime *date, char *buf);
>> +
>> +char *
>> +datetime_ctime(const struct datetime *date, char *buf);
>> +
>> +size_t
>> +datetime_strftime(const struct datetime *date, const char *fmt, char
>> *buf,
>> + uint32_t len);
>> +
>> +void
>> +datetime_now(struct datetime * now);
>> +
>> +#if defined(__cplusplus)
>> +} /* extern "C" */
>> +#endif /* defined(__cplusplus) */
>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>> new file mode 100644
>> index 000000000..ce579828f
>> --- /dev/null
>> +++ b/src/lua/datetime.lua
>> @@ -0,0 +1,500 @@
>> +local ffi = require('ffi')
>> +
>> +ffi.cdef [[
>> +
>> + /*
>> + `c-dt` library functions handles properly both positive and
>> negative `dt`
>> + values, where `dt` is a number of dates since Rata Die date
>> (0001-01-01).
>> +
>> + For better compactness of our typical data in MessagePack stream
>> we shift
>> + root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
>> + actually dt = 719163.
>> +
>> + So here is a simple formula how convert our epoch-based seconds
>> to dt values
>> + dt = (secs / 86400) + 719163
>> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
>> + (0001-01-01) in dates.
>> +
>> + */
>
>
> I'd move the comments outside the ffi.cdef block. This way they'd get
> proper highlighting, and it would be harder to mess something up
> by accidentally deleting the "*/"
Extracted.
>
>
>> + typedef int dt_t;
>> +
>> + // dt_core.h
>> + dt_t tnt_dt_from_rdn (int n);
>> + dt_t tnt_dt_from_ymd (int y, int m, int d);
>> +
>> + int tnt_dt_rdn (dt_t dt);
>> +
>> + // dt_parse_iso.h
>> + size_t tnt_dt_parse_iso_date (const char *str, size_t
>> len, dt_t *dt);
>> + size_t tnt_dt_parse_iso_time (const char *str, size_t
>> len, int *sod, int *nsec);
>> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t
>> len, int *offset);
>> +
>> + // datetime.c
>
>
> Also you may split the definitions into multiple ffi.cdef[[]] blocks
> if you want to add some per-definition comments.
>
Have split it into several. Like that
------------------------------------
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index ce579828f..7601421b1 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -1,8 +1,6 @@
local ffi = require('ffi')
-ffi.cdef [[
-
- /*
+--[[
`c-dt` library functions handles properly both positive and
negative `dt`
values, where `dt` is a number of dates since Rata Die date
(0001-01-01).
@@ -14,22 +12,27 @@ ffi.cdef [[
dt = (secs / 86400) + 719163
Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
(0001-01-01) in dates.
+]]
- */
+-- dt_core.h definitions
+ffi.cdef [[
typedef int dt_t;
- // dt_core.h
dt_t tnt_dt_from_rdn (int n);
dt_t tnt_dt_from_ymd (int y, int m, int d);
int tnt_dt_rdn (dt_t dt);
+]]
- // dt_parse_iso.h
+-- dt_parse_iso.h definitions
+ffi.cdef [[
size_t tnt_dt_parse_iso_date (const char *str, size_t
len, dt_t *dt);
size_t tnt_dt_parse_iso_time (const char *str, size_t
len, int *sod, int *nsec);
size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t
len, int *offset);
+]]
- // datetime.c
+-- Tarantool functions - datetime.c
+ffi.cdef [[
int
datetime_to_string(const struct datetime * date, char *buf,
uint32_t len);
@@ -45,7 +48,6 @@ ffi.cdef [[
void
datetime_now(struct datetime * now);
-
]]
local builtin = ffi.C
------------------------------------
>
>> + int
>> + datetime_to_string(const struct datetime * date, char *buf,
>> uint32_t len);
>> +
>> + char *
>> + datetime_asctime(const struct datetime *date, char *buf);
>> +
>> + char *
>> + datetime_ctime(const struct datetime *date, char *buf);
>> +
>> + size_t
>> + datetime_strftime(const struct datetime *date, const char *fmt,
>> char *buf,
>> + uint32_t len);
>> +
>> + void
>> + datetime_now(struct datetime * now);
>> +
>> +]]
>
>
> <stripped>
>
>
>> diff --git a/test/app-tap/datetime.test.lua
>> b/test/app-tap/datetime.test.lua
>> new file mode 100755
>> index 000000000..464d4bd49
>> --- /dev/null
>> +++ b/test/app-tap/datetime.test.lua
>> @@ -0,0 +1,206 @@
>> +#!/usr/bin/env tarantool
>> +
>> +local tap = require('tap')
>> +local test = tap.test("errno")
>> +local date = require('datetime')
>> +local ffi = require('ffi')
>> +
>> +
>> +test:plan(6)
>> +
>> +test:test("Simple tests for parser", function(test)
>> + test:plan(2)
>> + test:ok(date("1970-01-01T01:00:00Z") ==
>> + date {year=1970, month=1, day=1, hour=1, minute=0,
>> second=0})
>> + test:ok(date("1970-01-01T02:00:00+02:00") ==
>> + date {year=1970, month=1, day=1, hour=2, minute=0,
>> second=0, tz=120})
>> +end)
>> +
>> +test:test("Multiple tests for parser (with nanoseconds)", function(test)
>> + test:plan(168)
>> + -- borrowed from p5-time-moments/t/180_from_string.t
>> + local tests =
>> + {
>> + { '1970-01-01T00:00:00Z', 0,
>> 0, 0 },
>> + { '1970-01-01T02:00:00+02:00', 0,
>> 0, 120 },
>> + { '1970-01-01T01:30:00+01:30', 0,
>> 0, 90 },
>> + { '1970-01-01T01:00:00+01:00', 0,
>> 0, 60 },
>> + { '1970-01-01T00:01:00+00:01', 0,
>> 0, 1 },
>> + { '1970-01-01T00:00:00+00:00', 0,
>> 0, 0 },
>> + { '1969-12-31T23:59:00-00:01', 0,
>> 0, -1 },
>> + { '1969-12-31T23:00:00-01:00', 0,
>> 0, -60 },
>> + { '1969-12-31T22:30:00-01:30', 0,
>> 0, -90 },
>> + { '1969-12-31T22:00:00-02:00', 0,
>> 0, -120 },
>> + { '1970-01-01T00:00:00.123456789Z', 0,
>> 123456789, 0 },
>> + { '1970-01-01T00:00:00.12345678Z', 0,
>> 123456780, 0 },
>> + { '1970-01-01T00:00:00.1234567Z', 0,
>> 123456700, 0 },
>> + { '1970-01-01T00:00:00.123456Z', 0,
>> 123456000, 0 },
>> + { '1970-01-01T00:00:00.12345Z', 0,
>> 123450000, 0 },
>> + { '1970-01-01T00:00:00.1234Z', 0,
>> 123400000, 0 },
>> + { '1970-01-01T00:00:00.123Z', 0,
>> 123000000, 0 },
>> + { '1970-01-01T00:00:00.12Z', 0,
>> 120000000, 0 },
>> + { '1970-01-01T00:00:00.1Z', 0,
>> 100000000, 0 },
>> + { '1970-01-01T00:00:00.01Z', 0,
>> 10000000, 0 },
>> + { '1970-01-01T00:00:00.001Z', 0,
>> 1000000, 0 },
>> + { '1970-01-01T00:00:00.0001Z', 0,
>> 100000, 0 },
>> + { '1970-01-01T00:00:00.00001Z', 0,
>> 10000, 0 },
>> + { '1970-01-01T00:00:00.000001Z', 0,
>> 1000, 0 },
>> + { '1970-01-01T00:00:00.0000001Z', 0,
>> 100, 0 },
>> + { '1970-01-01T00:00:00.00000001Z', 0,
>> 10, 0 },
>> + { '1970-01-01T00:00:00.000000001Z', 0,
>> 1, 0 },
>> + { '1970-01-01T00:00:00.000000009Z', 0,
>> 9, 0 },
>> + { '1970-01-01T00:00:00.00000009Z', 0,
>> 90, 0 },
>> + { '1970-01-01T00:00:00.0000009Z', 0,
>> 900, 0 },
>> + { '1970-01-01T00:00:00.000009Z', 0,
>> 9000, 0 },
>> + { '1970-01-01T00:00:00.00009Z', 0,
>> 90000, 0 },
>> + { '1970-01-01T00:00:00.0009Z', 0,
>> 900000, 0 },
>> + { '1970-01-01T00:00:00.009Z', 0,
>> 9000000, 0 },
>> + { '1970-01-01T00:00:00.09Z', 0,
>> 90000000, 0 },
>> + { '1970-01-01T00:00:00.9Z', 0,
>> 900000000, 0 },
>> + { '1970-01-01T00:00:00.99Z', 0,
>> 990000000, 0 },
>> + { '1970-01-01T00:00:00.999Z', 0,
>> 999000000, 0 },
>> + { '1970-01-01T00:00:00.9999Z', 0,
>> 999900000, 0 },
>> + { '1970-01-01T00:00:00.99999Z', 0,
>> 999990000, 0 },
>> + { '1970-01-01T00:00:00.999999Z', 0,
>> 999999000, 0 },
>> + { '1970-01-01T00:00:00.9999999Z', 0,
>> 999999900, 0 },
>> + { '1970-01-01T00:00:00.99999999Z', 0,
>> 999999990, 0 },
>> + { '1970-01-01T00:00:00.999999999Z', 0,
>> 999999999, 0 },
>
> Красивое :)
:)
>
>> + { '1970-01-01T00:00:00.0Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.00Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.0000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.00000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.000000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.0000000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.00000000Z', 0,
>> 0, 0 },
>> + { '1970-01-01T00:00:00.000000000Z', 0,
>> 0, 0 },
>> + { '1973-11-29T21:33:09Z', 123456789,
>> 0, 0 },
>> + { '2013-10-28T17:51:56Z', 1382982716,
>> 0, 0 },
>> + { '9999-12-31T23:59:59Z', 253402300799,
>> 0, 0 },
>> + }
>> + for _, value in ipairs(tests) do
>> + local str, epoch, nsec, offset
>> + str, epoch, nsec, offset = unpack(value)
>> + local dt = date(str)
>> + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str,
>> epoch))
>> + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str,
>> nsec))
>> + test:ok(dt.offset == offset, ('%s: dt.offset ==
>> %d'):format(str, offset))
>> + end
>> +end)
>> +
>> +ffi.cdef [[
>> + void tzset(void);
>> +]]
>> +
>>
>
>
> <stripped>
>
>
>
Here is (was) incremental patch. [Now it's slightly changed, with
MP-related defines, but you got the point]:
------------------------------------------------------------------
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index c48295a6f..719a4cd47 100644
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -1,32 +1,7 @@
/*
- * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * 1. Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the
- * following disclaimer.
+ * SPDX-License-Identifier: BSD-2-Clause
*
- * 2. Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
- * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
- * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
- * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
#include <string.h>
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 1a8d7e34f..88774110c 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -1,38 +1,12 @@
#pragma once
/*
- * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * 1. Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the
- * following disclaimer.
+ * SPDX-License-Identifier: BSD-2-Clause
*
- * 2. Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
- * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
- * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
- * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
#include <stdint.h>
#include <stdbool.h>
-#include <stdio.h>
#include "c-dt/dt.h"
#if defined(__cplusplus)
@@ -40,23 +14,34 @@ extern "C"
{
#endif /* defined(__cplusplus) */
+/**
+ * We count dates since so called "Rata Die" date
+ * January 1, 0001, Monday (as Day 1).
+ * But datetime structure keeps seconds since
+ * Unix "Epoch" date:
+ * Unix, January 1, 1970, Thursday
+ *
+ * The difference between Epoch (1970-01-01)
+ * and Rata Die (0001-01-01) is 719163 days.
+ */
+
#ifndef SECS_PER_DAY
#define SECS_PER_DAY 86400
#define DT_EPOCH_1970_OFFSET 719163
#endif
/**
- * Full datetime structure representing moments
- * since Unix Epoch (1970-01-01).
- * Time is kept normalized to UTC, time-zone offset
+ * datetime structure keeps number of seconds since
+ * Unix Epoch.
+ * Time is normalized by UTC, so time-zone offset
* is informative only.
*/
struct datetime {
- /** seconds since epoch */
+ /** Seconds since Epoch. */
double secs;
- /** nanoseconds if any */
- int32_t nsec;
- /** offset in minutes from UTC */
+ /** Nanoseconds, if any. */
+ uint32_t nsec;
+ /** Offset in minutes from UTC. */
int32_t offset;
};
@@ -64,10 +49,10 @@ struct datetime {
* Date/time interval structure
*/
struct datetime_interval {
- /** relative seconds delta */
+ /** Relative seconds delta. */
double secs;
- /** nanoseconds delta */
- int32_t nsec;
+ /** Nanoseconds delta, if any. */
+ uint32_t nsec;
};
/**
------------------------------------------------------------------
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-17 23:30 ` Safin Timur via Tarantool-patches
@ 2021-08-18 8:56 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-18 8:56 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
18.08.2021 02:30, Safin Timur пишет:
> On 17.08.2021 15:15, Serge Petrenko wrote:
>>
>>
>>> - check calculated attributes to date object, e.g.:
>>> - timestamp, seconds, microseconds, minute, or hours
>>> - to_utc(), and to_tz() allow to switch timezone of a
>>> datetime object. It's not changing much - only timezone
>>> but that impacts textual representation of a date.
>>>
>>> Part of #5941
>>
>> Please, add a docbot request to the commit message.
>> Here it should say that you introduce lua datetime module
>> and describe shortly what the module does.
>
> I'm planning to cheat here - to not put detailed description to docbot
> request part here, but rather to refer to the externally available
> documentation I've written in
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
>
> Like:
>
> @TarantoolBot document
> Title: Introduced new built-in `datetime` module
>
>
> `datetime` module has been introduced, which allows to parse ISO-8601
> literals representing timestamps of various formats, and then
> manipulate with date objects.
>
> Please refer to
> https://github.com/tarantool/tarantool/discussions/6244 for more
> detailed description of module API.
That's fine, I guess.
>
>>
>>> ---
>>> cmake/BuildCDT.cmake | 2 +
>>> extra/exports | 26 +
>>> src/CMakeLists.txt | 2 +
>>> src/lib/core/CMakeLists.txt | 1 +
>>> src/lib/core/datetime.c | 96 ++++
>>> src/lib/core/datetime.h | 95 ++++
>>> src/lua/datetime.lua | 500
>>> ++++++++++++++++++
>>> src/lua/init.c | 4 +-
>>> src/lua/utils.c | 27 +
>>> src/lua/utils.h | 12 +
>>> test/app-tap/datetime.test.lua | 206 ++++++++
>>> .../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 ++-
>>> 12 files changed, 1043 insertions(+), 2 deletions(-)
>>> create mode 100644 src/lib/core/datetime.c
>>> create mode 100644 src/lib/core/datetime.h
>>> create mode 100644 src/lua/datetime.lua
>>> create mode 100755 test/app-tap/datetime.test.lua
>>>
>>> diff --git a/cmake/BuildCDT.cmake b/cmake/BuildCDT.cmake
>>> index 343fb1b99..80b26c64a 100644
>>> --- a/cmake/BuildCDT.cmake
>>> +++ b/cmake/BuildCDT.cmake
>>> @@ -5,4 +5,6 @@ macro(libccdt_build)
>>> file(MAKE_DIRECTORY
>>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>>> add_subdirectory(${PROJECT_SOURCE_DIR}/third_party/c-dt
>>> ${CMAKE_CURRENT_BINARY_DIR}/third_party/c-dt/build/)
>>> + set_target_properties(cdt PROPERTIES COMPILE_FLAGS
>>> "-DDT_NAMESPACE=tnt_")
>>> + add_definitions("-DDT_NAMESPACE=tnt_")
>>> endmacro()
>>
>>
>> This change belongs to the previous commit, doesn't it?
>
> I considered that, but decided that for better observability purposes
> I want to keep this rename of symbols via `tnt_` prefix closer to the
> code where we exports those functions (please see `tnt_dt_dow` and
> others below in the /extra/exports).
>
> [But taking into account that we gonna squash these commits I don't
> care that much now]
>
>>
>>
>>> diff --git a/extra/exports b/extra/exports
>>> index 9eaba1282..80eb92abd 100644
>>> --- a/extra/exports
>>> +++ b/extra/exports
>>> @@ -148,8 +148,34 @@ csv_feed
>>> csv_iterator_create
>>> csv_next
>>> csv_setopt
>>> +datetime_asctime
>>> +datetime_ctime
>>> +datetime_now
>>> +datetime_strftime
>>> +decimal_unpack
>>> decimal_from_string
>>> decimal_unpack
>
> These guys renamed here
> |
> V
>>> +tnt_dt_dow
>>> +tnt_dt_from_rdn
>>> +tnt_dt_from_struct_tm
>>> +tnt_dt_from_yd
>>> +tnt_dt_from_ymd
>>> +tnt_dt_from_yqd
>>> +tnt_dt_from_ywd
>>> +tnt_dt_parse_iso_date
>>> +tnt_dt_parse_iso_time_basic
>>> +tnt_dt_parse_iso_time_extended
>>> +tnt_dt_parse_iso_time
>>> +tnt_dt_parse_iso_zone_basic
>>> +tnt_dt_parse_iso_zone_extended
>>> +tnt_dt_parse_iso_zone_lenient
>>> +tnt_dt_parse_iso_zone
>>> +tnt_dt_rdn
>>> +tnt_dt_to_struct_tm
>>> +tnt_dt_to_yd
>>> +tnt_dt_to_ymd
>>> +tnt_dt_to_yqd
>>> +tnt_dt_to_ywd
>>> error_ref
>>> error_set_prev
>>> error_unref
>>> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
>>> index 97b0cb326..4473ff1da 100644
>>> --- a/src/CMakeLists.txt
>>> +++ b/src/CMakeLists.txt
>>> @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
>>> lua_source(lua_sources lua/httpc.lua)
>>> lua_source(lua_sources lua/iconv.lua)
>>> lua_source(lua_sources lua/swim.lua)
>>> +lua_source(lua_sources lua/datetime.lua)
>>> +
>>> # LuaJIT jit.* library
>>> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
>>> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
>>> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
>>> index 2cd4d0b4f..8bc776b82 100644
>>> --- a/src/lib/core/CMakeLists.txt
>>> +++ b/src/lib/core/CMakeLists.txt
>>> @@ -30,6 +30,7 @@ set(core_sources
>>> decimal.c
>>> mp_decimal.c
>>> cord_buf.c
>>> + datetime.c
>>> )
>>> if (TARGET_OS_NETBSD)
>>> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
>>> new file mode 100644
>>> index 000000000..c48295a6f
>>> --- /dev/null
>>> +++ b/src/lib/core/datetime.c
>>> @@ -0,0 +1,96 @@
>>> +/*
>>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>>> + *
>>> + * Redistribution and use in source and binary forms, with or
>>> + * without modification, are permitted provided that the following
>>> + * conditions are met:
>>> + *
>>> + * 1. Redistributions of source code must retain the above
>>> + * copyright notice, this list of conditions and the
>>> + * following disclaimer.
>>> + *
>>> + * 2. Redistributions in binary form must reproduce the above
>>> + * copyright notice, this list of conditions and the following
>>> + * disclaimer in the documentation and/or other materials
>>> + * provided with the distribution.
>>> + *
>>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>>> + * SUCH DAMAGE.
>>> + */
>>> +
>>
>> As far as I know, we switched to the following license format recently:
>>
>> /*
>> * SPDX-License-Identifier: BSD-2-Clause
>> *
>> * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file.
>> */
>
> Much shorter - I like it. Updated.
>
>>
>> See, for example:
>>
>> ./src/box/module_cache.c: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/box/lua/lib.c: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/box/lua/lib.h: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/box/module_cache.h: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/lib/core/cord_buf.c: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/lib/core/crash.c: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/lib/core/cord_buf.h: * SPDX-License-Identifier: BSD-2-Clause
>> ./src/lib/core/crash.h: * SPDX-License-Identifier: BSD-2-Clause
>>
>>
>> <stripped>
>>
>>
>>> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
>>> new file mode 100644
>>> index 000000000..1a8d7e34f
>>> --- /dev/null
>>> +++ b/src/lib/core/datetime.h
>>> @@ -0,0 +1,95 @@
>>> +#pragma once
>>> +/*
>>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>>> + *
>>> + * Redistribution and use in source and binary forms, with or
>>> + * without modification, are permitted provided that the following
>>> + * conditions are met:
>>> + *
>>> + * 1. Redistributions of source code must retain the above
>>> + * copyright notice, this list of conditions and the
>>> + * following disclaimer.
>>> + *
>>> + * 2. Redistributions in binary form must reproduce the above
>>> + * copyright notice, this list of conditions and the following
>>> + * disclaimer in the documentation and/or other materials
>>> + * provided with the distribution.
>>> + *
>>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>>> + * SUCH DAMAGE.
>>> + */
>>> +
>>
>> Same about the license.
>
> Updated
>
>>
>> And I'd move the "#pragma once" below the license comment.
>> Otherwise it's easily lost. Up to you.
>
> On one side I agreed that it might left unrecognizable for untrained
> eye. But on other side - the sooner compiler/preprocessor will see,
> the better :)
>
> [And there is already well established convention to put it at the
> first line.]
>
> So I've left it on the 1st line.
>
>>
>>> +#include <stdint.h>
>>> +#include <stdbool.h>
>>> +#include <stdio.h>
>>
>> AFAICS you don't need stdio included here.
>
> Indeed!
>
>>
>>> +#include "c-dt/dt.h"
>>> +
>>> +#if defined(__cplusplus)
>>> +extern "C"
>>> +{
>>> +#endif /* defined(__cplusplus) */
>>> +
>>> +#ifndef SECS_PER_DAY
>>> +#define SECS_PER_DAY 86400
>>> +#define DT_EPOCH_1970_OFFSET 719163
>>
>> Please, add a short comment on what this is.
>> I had to spend some time googling to understand.
>>
>> So, please mention that this is measured in days from 01-01-0001.
>
> I've written some explanation about these magic numbers. Now it's
> verboser a bit:
> --------------------------------------------------------
> /**
> * We count dates since so called "Rata Die" date
> * January 1, 0001, Monday (as Day 1).
> * But datetime structure keeps seconds since
> * Unix "Epoch" date:
> * Unix, January 1, 1970, Thursday
> *
> * The difference between Epoch (1970-01-01)
> * and Rata Die (0001-01-01) is 719163 days.
> */
>
> #ifndef SECS_PER_DAY
> #define SECS_PER_DAY 86400
> #define DT_EPOCH_1970_OFFSET 719163
> #endif
> --------------------------------------------------------
>
> also, for the MP-related patch, I've added this comment, and defines
> (might be used in asserts):
>
> --------------------------------------------------------
> /**
> * c-dt library uses int as type for dt value, which
> * represents the number of days since Rata Die date.
> * This implies limits to the number of seconds we
> * could safely store in our structures and then safely
> * pass to c-dt functions.
> *
> * So supported ranges will be
> * - for seconds [-185604722870400 .. 185480451417600]
> * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
> */
> #define MAX_DT_DAY_VALUE (int64_t)INT_MAX
> #define MIN_DT_DAY_VALUE (int64_t)INT_MIN
> #define SECS_EPOCH_1970_OFFSET \
> ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> #define MAX_EPOCH_SECS_VALUE \
> (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> #define MIN_EPOCH_SECS_VALUE \
> (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> --------------------------------------------------------
>
>>
>>> +#endif
>>> +
>>> +/**
>>> + * Full datetime structure representing moments
>>> + * since Unix Epoch (1970-01-01).
>>> + * Time is kept normalized to UTC, time-zone offset
>>> + * is informative only.
>>> + */
>>> +struct datetime {
>>> + /** seconds since epoch */
>>> + double secs;
>>> + /** nanoseconds if any */
>>> + int32_t nsec;
>>
>>
>> As discussed, let's make nsec a uint32_t, since
>> nsec part is always positive.
>
> Changed.
>
>>
>>
>>> + /** offset in minutes from UTC */
>>> + int32_t offset;
>>> +};
>>> +
>>> +/**
>>> + * Date/time interval structure
>>> + */
>>> +struct datetime_interval {
>>> + /** relative seconds delta */
>>> + double secs;
>>> + /** nanoseconds delta */
>>> + int32_t nsec;
>>> +};
>>> +
>>
>>
>> Please start comments with a capital letter and end them with a dot.
>
> Done.
>
>>
>>
>>> +/**
>>> + * Convert datetime to string using default asctime format
>>> + * "Sun Sep 16 01:03:52 1973\n\0"
>>> + * Wrapper around reenterable asctime_r() version of POSIX function
>>> + * @param date source datetime value
>>> + * @sa datetime_ctime
>>> + */
>>> +char *
>>> +datetime_asctime(const struct datetime *date, char *buf);
>>> +
>>> +char *
>>> +datetime_ctime(const struct datetime *date, char *buf);
>>> +
>>> +size_t
>>> +datetime_strftime(const struct datetime *date, const char *fmt,
>>> char *buf,
>>> + uint32_t len);
>>> +
>>> +void
>>> +datetime_now(struct datetime * now);
>>> +
>>> +#if defined(__cplusplus)
>>> +} /* extern "C" */
>>> +#endif /* defined(__cplusplus) */
>>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>>> new file mode 100644
>>> index 000000000..ce579828f
>>> --- /dev/null
>>> +++ b/src/lua/datetime.lua
>>> @@ -0,0 +1,500 @@
>>> +local ffi = require('ffi')
>>> +
>>> +ffi.cdef [[
>>> +
>>> + /*
>>> + `c-dt` library functions handles properly both positive and
>>> negative `dt`
>>> + values, where `dt` is a number of dates since Rata Die date
>>> (0001-01-01).
>>> +
>>> + For better compactness of our typical data in MessagePack
>>> stream we shift
>>> + root of our time to the Unix Epoch date (1970-01-01), thus our
>>> 0 is
>>> + actually dt = 719163.
>>> +
>>> + So here is a simple formula how convert our epoch-based seconds
>>> to dt values
>>> + dt = (secs / 86400) + 719163
>>> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata
>>> Die
>>> + (0001-01-01) in dates.
>>> +
>>> + */
>>
>>
>> I'd move the comments outside the ffi.cdef block. This way they'd get
>> proper highlighting, and it would be harder to mess something up
>> by accidentally deleting the "*/"
>
> Extracted.
>
>>
>>
>>> + typedef int dt_t;
>>> +
>>> + // dt_core.h
>>> + dt_t tnt_dt_from_rdn (int n);
>>> + dt_t tnt_dt_from_ymd (int y, int m, int d);
>>> +
>>> + int tnt_dt_rdn (dt_t dt);
>>> +
>>> + // dt_parse_iso.h
>>> + size_t tnt_dt_parse_iso_date (const char *str, size_t
>>> len, dt_t *dt);
>>> + size_t tnt_dt_parse_iso_time (const char *str, size_t
>>> len, int *sod, int *nsec);
>>> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t
>>> len, int *offset);
>>> +
>>> + // datetime.c
>>
>>
>> Also you may split the definitions into multiple ffi.cdef[[]] blocks
>> if you want to add some per-definition comments.
>>
>
> Have split it into several. Like that
> ------------------------------------
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index ce579828f..7601421b1 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -1,8 +1,6 @@
> local ffi = require('ffi')
>
> -ffi.cdef [[
> -
> - /*
> +--[[
> `c-dt` library functions handles properly both positive and
> negative `dt`
> values, where `dt` is a number of dates since Rata Die date
> (0001-01-01).
>
> @@ -14,22 +12,27 @@ ffi.cdef [[
> dt = (secs / 86400) + 719163
> Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
> (0001-01-01) in dates.
> +]]
>
> - */
> +-- dt_core.h definitions
> +ffi.cdef [[
> typedef int dt_t;
>
> - // dt_core.h
> dt_t tnt_dt_from_rdn (int n);
> dt_t tnt_dt_from_ymd (int y, int m, int d);
>
> int tnt_dt_rdn (dt_t dt);
> +]]
>
> - // dt_parse_iso.h
> +-- dt_parse_iso.h definitions
> +ffi.cdef [[
> size_t tnt_dt_parse_iso_date (const char *str, size_t
> len, dt_t *dt);
> size_t tnt_dt_parse_iso_time (const char *str, size_t
> len, int *sod, int *nsec);
> size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t
> len, int *offset);
> +]]
>
> - // datetime.c
> +-- Tarantool functions - datetime.c
> +ffi.cdef [[
> int
> datetime_to_string(const struct datetime * date, char *buf,
> uint32_t len);
>
> @@ -45,7 +48,6 @@ ffi.cdef [[
>
> void
> datetime_now(struct datetime * now);
> -
> ]]
>
> local builtin = ffi.C
> ------------------------------------
>
>>
>>> + int
>>> + datetime_to_string(const struct datetime * date, char *buf,
>>> uint32_t len);
>>> +
>>> + char *
>>> + datetime_asctime(const struct datetime *date, char *buf);
>>> +
>>> + char *
>>> + datetime_ctime(const struct datetime *date, char *buf);
>>> +
>>> + size_t
>>> + datetime_strftime(const struct datetime *date, const char *fmt,
>>> char *buf,
>>> + uint32_t len);
>>> +
>>> + void
>>> + datetime_now(struct datetime * now);
>>> +
>>> +]]
>>
>>
>> <stripped>
>>
>>
>>> diff --git a/test/app-tap/datetime.test.lua
>>> b/test/app-tap/datetime.test.lua
>>> new file mode 100755
>>> index 000000000..464d4bd49
>>> --- /dev/null
>>> +++ b/test/app-tap/datetime.test.lua
>>> @@ -0,0 +1,206 @@
>>> +#!/usr/bin/env tarantool
>>> +
>>> +local tap = require('tap')
>>> +local test = tap.test("errno")
>>> +local date = require('datetime')
>>> +local ffi = require('ffi')
>>> +
>>> +
>>> +test:plan(6)
>>> +
>>> +test:test("Simple tests for parser", function(test)
>>> + test:plan(2)
>>> + test:ok(date("1970-01-01T01:00:00Z") ==
>>> + date {year=1970, month=1, day=1, hour=1, minute=0,
>>> second=0})
>>> + test:ok(date("1970-01-01T02:00:00+02:00") ==
>>> + date {year=1970, month=1, day=1, hour=2, minute=0,
>>> second=0, tz=120})
>>> +end)
>>> +
>>> +test:test("Multiple tests for parser (with nanoseconds)",
>>> function(test)
>>> + test:plan(168)
>>> + -- borrowed from p5-time-moments/t/180_from_string.t
>>> + local tests =
>>> + {
>>> + { '1970-01-01T00:00:00Z', 0, 0, 0 },
>>> + { '1970-01-01T02:00:00+02:00', 0, 0, 120 },
>>> + { '1970-01-01T01:30:00+01:30', 0, 0, 90 },
>>> + { '1970-01-01T01:00:00+01:00', 0, 0, 60 },
>>> + { '1970-01-01T00:01:00+00:01', 0, 0, 1 },
>>> + { '1970-01-01T00:00:00+00:00', 0, 0, 0 },
>>> + { '1969-12-31T23:59:00-00:01', 0, 0, -1 },
>>> + { '1969-12-31T23:00:00-01:00', 0, 0, -60 },
>>> + { '1969-12-31T22:30:00-01:30', 0, 0, -90 },
>>> + { '1969-12-31T22:00:00-02:00', 0, 0, -120 },
>>> + { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 },
>>> + { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 },
>>> + { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 },
>>> + { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 },
>>> + { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 },
>>> + { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 },
>>> + { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 },
>>> + { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 },
>>> + { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 },
>>> + { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 },
>>> + { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 },
>>> + { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 },
>>> + { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 },
>>> + { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 },
>>> + { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 },
>>> + { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 },
>>> + { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 },
>>> + { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 },
>>> + { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 },
>>> + { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 },
>>> + { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 },
>>> + { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 },
>>> + { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 },
>>> + { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 },
>>> + { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 },
>>> + { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 },
>>> + { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 },
>>> + { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 },
>>> + { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 },
>>> + { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 },
>>> + { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 },
>>> + { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 },
>>> + { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 },
>>> + { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 },
>>
>> Красивое :)
>
> :)
>
>>
>>> + { '1970-01-01T00:00:00.0Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.00Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.0000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.00000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.000000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 },
>>> + { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 },
>>> + { '1973-11-29T21:33:09Z', 123456789, 0, 0 },
>>> + { '2013-10-28T17:51:56Z', 1382982716, 0, 0 },
>>> + { '9999-12-31T23:59:59Z', 253402300799, 0, 0 },
>>> + }
>>> + for _, value in ipairs(tests) do
>>> + local str, epoch, nsec, offset
>>> + str, epoch, nsec, offset = unpack(value)
>>> + local dt = date(str)
>>> + test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str,
>>> epoch))
>>> + test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str,
>>> nsec))
>>> + test:ok(dt.offset == offset, ('%s: dt.offset ==
>>> %d'):format(str, offset))
>>> + end
>>> +end)
>>> +
>>> +ffi.cdef [[
>>> + void tzset(void);
>>> +]]
>>> +
>>>
>>
>>
>> <stripped>
>>
>>
>>
>
> Here is (was) incremental patch. [Now it's slightly changed, with
> MP-related defines, but you got the point]:
> ------------------------------------------------------------------
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index c48295a6f..719a4cd47 100644
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -1,32 +1,7 @@
> /*
> - * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> - *
> - * Redistribution and use in source and binary forms, with or
> - * without modification, are permitted provided that the following
> - * conditions are met:
> - *
> - * 1. Redistributions of source code must retain the above
> - * copyright notice, this list of conditions and the
> - * following disclaimer.
> + * SPDX-License-Identifier: BSD-2-Clause
> *
> - * 2. Redistributions in binary form must reproduce the above
> - * copyright notice, this list of conditions and the following
> - * disclaimer in the documentation and/or other materials
> - * provided with the distribution.
> - *
> - * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> - * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> - * SUCH DAMAGE.
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> #include <string.h>
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 1a8d7e34f..88774110c 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -1,38 +1,12 @@
> #pragma once
> /*
> - * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> - *
> - * Redistribution and use in source and binary forms, with or
> - * without modification, are permitted provided that the following
> - * conditions are met:
> - *
> - * 1. Redistributions of source code must retain the above
> - * copyright notice, this list of conditions and the
> - * following disclaimer.
> + * SPDX-License-Identifier: BSD-2-Clause
> *
> - * 2. Redistributions in binary form must reproduce the above
> - * copyright notice, this list of conditions and the following
> - * disclaimer in the documentation and/or other materials
> - * provided with the distribution.
> - *
> - * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> - * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> - * SUCH DAMAGE.
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> #include <stdint.h>
> #include <stdbool.h>
> -#include <stdio.h>
> #include "c-dt/dt.h"
>
> #if defined(__cplusplus)
> @@ -40,23 +14,34 @@ extern "C"
> {
> #endif /* defined(__cplusplus) */
>
> +/**
> + * We count dates since so called "Rata Die" date
> + * January 1, 0001, Monday (as Day 1).
> + * But datetime structure keeps seconds since
> + * Unix "Epoch" date:
> + * Unix, January 1, 1970, Thursday
> + *
> + * The difference between Epoch (1970-01-01)
> + * and Rata Die (0001-01-01) is 719163 days.
> + */
> +
> #ifndef SECS_PER_DAY
> #define SECS_PER_DAY 86400
> #define DT_EPOCH_1970_OFFSET 719163
> #endif
>
> /**
> - * Full datetime structure representing moments
> - * since Unix Epoch (1970-01-01).
> - * Time is kept normalized to UTC, time-zone offset
> + * datetime structure keeps number of seconds since
> + * Unix Epoch.
> + * Time is normalized by UTC, so time-zone offset
> * is informative only.
> */
> struct datetime {
> - /** seconds since epoch */
> + /** Seconds since Epoch. */
> double secs;
> - /** nanoseconds if any */
> - int32_t nsec;
> - /** offset in minutes from UTC */
> + /** Nanoseconds, if any. */
> + uint32_t nsec;
> + /** Offset in minutes from UTC. */
> int32_t offset;
> };
>
> @@ -64,10 +49,10 @@ struct datetime {
> * Date/time interval structure
> */
> struct datetime_interval {
> - /** relative seconds delta */
> + /** Relative seconds delta. */
> double secs;
> - /** nanoseconds delta */
> - int32_t nsec;
> + /** Nanoseconds delta, if any. */
> + uint32_t nsec;
> };
>
> /**
> ------------------------------------------------------------------
>
> Thanks,
> Timur
Thanks for the changes!
Looks good so far. I'll take a look at the whole series again
once you push the updates to Vladimir's comments
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 16:52 ` Vladimir Davydov via Tarantool-patches
2021-08-17 19:16 ` Vladimir Davydov via Tarantool-patches
2021-08-18 10:03 ` Safin Timur via Tarantool-patches
1 sibling, 2 replies; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 16:52 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:36AM +0300, Timur Safin via Tarantool-patches wrote:
> diff --git a/extra/exports b/extra/exports
> index 9eaba1282..80eb92abd 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -148,8 +148,34 @@ csv_feed
> csv_iterator_create
> csv_next
> csv_setopt
> +datetime_asctime
> +datetime_ctime
> +datetime_now
> +datetime_strftime
> +decimal_unpack
decimal_unpack?
> decimal_from_string
> decimal_unpack
> +tnt_dt_dow
> +tnt_dt_from_rdn
> +tnt_dt_from_struct_tm
> +tnt_dt_from_yd
> +tnt_dt_from_ymd
> +tnt_dt_from_yqd
> +tnt_dt_from_ywd
> +tnt_dt_parse_iso_date
> +tnt_dt_parse_iso_time_basic
> +tnt_dt_parse_iso_time_extended
> +tnt_dt_parse_iso_time
> +tnt_dt_parse_iso_zone_basic
> +tnt_dt_parse_iso_zone_extended
> +tnt_dt_parse_iso_zone_lenient
> +tnt_dt_parse_iso_zone
> +tnt_dt_rdn
> +tnt_dt_to_struct_tm
> +tnt_dt_to_yd
> +tnt_dt_to_ymd
> +tnt_dt_to_yqd
> +tnt_dt_to_ywd
> error_ref
> error_set_prev
> error_unref
> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> index 97b0cb326..4473ff1da 100644
> --- a/src/CMakeLists.txt
> +++ b/src/CMakeLists.txt
> @@ -51,6 +51,8 @@ lua_source(lua_sources ../third_party/luafun/fun.lua)
> lua_source(lua_sources lua/httpc.lua)
> lua_source(lua_sources lua/iconv.lua)
> lua_source(lua_sources lua/swim.lua)
> +lua_source(lua_sources lua/datetime.lua)
> +
> # LuaJIT jit.* library
> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bc.lua)
> lua_source(lua_sources ${LUAJIT_SOURCE_ROOT}/src/jit/bcsave.lua)
> diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
> index 2cd4d0b4f..8bc776b82 100644
> --- a/src/lib/core/CMakeLists.txt
> +++ b/src/lib/core/CMakeLists.txt
> @@ -30,6 +30,7 @@ set(core_sources
> decimal.c
> mp_decimal.c
> cord_buf.c
> + datetime.c
> )
>
> if (TARGET_OS_NETBSD)
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> new file mode 100644
> index 000000000..c48295a6f
> --- /dev/null
> +++ b/src/lib/core/datetime.c
> @@ -0,0 +1,96 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <string.h>
> +#include <time.h>
> +
> +#include "trivia/util.h"
> +#include "datetime.h"
> +
> +static int
> +local_dt(int64_t secs)
> +{
> + return dt_from_rdn((int)(secs / SECS_PER_DAY) + DT_EPOCH_1970_OFFSET);
> +}
I don't understand what this function does. Please add a comment.
> +
> +static struct tm *
> +datetime_to_tm(const struct datetime *date)
> +{
> + static struct tm tm;
> +
> + memset(&tm, 0, sizeof(tm));
> + int64_t secs = date->secs;
> + dt_to_struct_tm(local_dt(secs), &tm);
> +
> + int seconds_of_day = (int64_t)date->secs % SECS_PER_DAY;
> + tm.tm_hour = (seconds_of_day / 3600) % 24;
> + tm.tm_min = (seconds_of_day / 60) % 60;
> + tm.tm_sec = seconds_of_day % 60;
To make the code easier for understanding, please define and use
constants here and everywhere else in this patch: HOURS_PER_DAY,
MINUTES_PER_HOUR, NSECS_PER_USEC, etc.
> +
> + return &tm;
> +}
> +
> +void
> +datetime_now(struct datetime *now)
> +{
> + struct timeval tv;
> + gettimeofday(&tv, NULL);
> + now->secs = tv.tv_sec;
> + now->nsec = tv.tv_usec * 1000;
> +
> + time_t now_seconds;
> + time(&now_seconds);
Can't you use tv.tv_sec here?
> + struct tm tm;
> + localtime_r(&now_seconds, &tm);
> + now->offset = tm.tm_gmtoff / 60;
> +}
> +
> +char *
> +datetime_asctime(const struct datetime *date, char *buf)
> +{
> + struct tm *p_tm = datetime_to_tm(date);
> + return asctime_r(p_tm, buf);
> +}
> +
> +char *
> +datetime_ctime(const struct datetime *date, char *buf)
> +{
> + time_t time = date->secs;
> + return ctime_r(&time, buf);
> +}
> +
> +size_t
> +datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> + uint32_t len)
> +{
> + struct tm *p_tm = datetime_to_tm(date);
> + return strftime(buf, len, fmt, p_tm);
> +}
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> new file mode 100644
> index 000000000..1a8d7e34f
> --- /dev/null
> +++ b/src/lib/core/datetime.h
> @@ -0,0 +1,95 @@
> +#pragma once
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include <stdint.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include "c-dt/dt.h"
> +
> +#if defined(__cplusplus)
> +extern "C"
> +{
> +#endif /* defined(__cplusplus) */
> +
> +#ifndef SECS_PER_DAY
> +#define SECS_PER_DAY 86400
> +#define DT_EPOCH_1970_OFFSET 719163
I don't understand what this constant stores. Please add a comment.
> +#endif
> +
> +/**
> + * Full datetime structure representing moments
> + * since Unix Epoch (1970-01-01).
> + * Time is kept normalized to UTC, time-zone offset
> + * is informative only.
> + */
> +struct datetime {
> + /** seconds since epoch */
> + double secs;
Please add a comment explaining why you use 'double' instead of
an integer type.
> + /** nanoseconds if any */
> + int32_t nsec;
Why 'nsec', but 'secs'? This is inconsistent. Should be 'nsec' and 'sec'
or 'nsecs' and 'secs'.
> + /** offset in minutes from UTC */
> + int32_t offset;
Why do you use int32_t instead of int for these two members?
Same comments for the datetime_interval struct.
> +};
> +
> +/**
> + * Date/time interval structure
> + */
> +struct datetime_interval {
> + /** relative seconds delta */
> + double secs;
> + /** nanoseconds delta */
> + int32_t nsec;
> +};
> +
> +/**
> + * Convert datetime to string using default asctime format
> + * "Sun Sep 16 01:03:52 1973\n\0"
> + * Wrapper around reenterable asctime_r() version of POSIX function
> + * @param date source datetime value
> + * @sa datetime_ctime
> + */
> +char *
> +datetime_asctime(const struct datetime *date, char *buf);
> +
> +char *
> +datetime_ctime(const struct datetime *date, char *buf);
> +
> +size_t
> +datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> + uint32_t len);
> +
> +void
> +datetime_now(struct datetime * now);
Extra space after '*'.
Please add comments to all these functions, like you did for
datetime_asctime.
> +
> +#if defined(__cplusplus)
> +} /* extern "C" */
> +#endif /* defined(__cplusplus) */
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> new file mode 100644
> index 000000000..ce579828f
> --- /dev/null
> +++ b/src/lua/datetime.lua
> @@ -0,0 +1,500 @@
> +local ffi = require('ffi')
> +
> +ffi.cdef [[
> +
> + /*
> + `c-dt` library functions handles properly both positive and negative `dt`
> + values, where `dt` is a number of dates since Rata Die date (0001-01-01).
> +
> + For better compactness of our typical data in MessagePack stream we shift
> + root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
> + actually dt = 719163.
> +
> + So here is a simple formula how convert our epoch-based seconds to dt values
> + dt = (secs / 86400) + 719163
> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
> + (0001-01-01) in dates.
> +
> + */
> + typedef int dt_t;
> +
> + // dt_core.h
> + dt_t tnt_dt_from_rdn (int n);
> + dt_t tnt_dt_from_ymd (int y, int m, int d);
> +
> + int tnt_dt_rdn (dt_t dt);
> +
> + // dt_parse_iso.h
> + size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
> + size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec);
> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len, int *offset);
The line is too long. Please ensure that all lines are <= 80 characters
long.
> +
> + // datetime.c
> + int
> + datetime_to_string(const struct datetime * date, char *buf, uint32_t len);
> +
> + char *
> + datetime_asctime(const struct datetime *date, char *buf);
> +
> + char *
> + datetime_ctime(const struct datetime *date, char *buf);
> +
> + size_t
> + datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> + uint32_t len);
> +
> + void
> + datetime_now(struct datetime * now);
Extra space after '*'.
> +
> +]]
> +
> +local builtin = ffi.C
> +local math_modf = math.modf
> +
> +local SECS_PER_DAY = 86400
> +
> +-- c-dt/dt_config.h
> +
> +-- Unix, January 1, 1970, Thursday
> +local DT_EPOCH_1970_OFFSET = 719163
> +
> +
> +local datetime_t = ffi.typeof('struct datetime')
> +local interval_t = ffi.typeof('struct datetime_interval')
> +
> +local function is_interval(o)
> + return type(o) == 'cdata' and ffi.istype(interval_t, o)
The check for 'cdata' is redundant. ffi.istype alone should be enough
AFAIK.
> +end
> +
> +local function is_datetime(o)
> + return type(o) == 'cdata' and ffi.istype(datetime_t, o)
> +end
> +
> +local function is_date_interval(o)
> + return type(o) == 'cdata' and
> + (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
> +end
> +
> +local function interval_new()
> + local interval = ffi.new(interval_t)
> + return interval
> +end
> +
> +local function check_date(o, message)
> + if not is_datetime(o) then
> + return error(("%s: expected datetime, but received %s"):
> + format(message, o), 2)
> + end
> +end
> +
> +local function check_str(s, message)
> + if not type(s) == 'string' then
> + return error(("%s: expected string, but received %s"):
> + format(message, s), 2)
> + end
> +end
> +
> +local function check_range(v, range, txt)
> + assert(#range == 2)
> + if v < range[1] or v > range[2] then
> + error(('value %d of %s is out of allowed range [%d, %d]'):
> + format(v, txt, range[1], range[2]), 4)
> + end
> +end
> +
> +local function datetime_cmp(lhs, rhs)
> + if not is_date_interval(lhs) or
> + not is_date_interval(rhs) then
> + return nil
Bad indentation.
> + end
> + local sdiff = lhs.secs - rhs.secs
> + return sdiff ~= 0 and sdiff or (lhs.nsec - rhs.nsec)
> +end
> +
> +local function datetime_eq(lhs, rhs)
> + local rc = datetime_cmp(lhs, rhs)
> + return rc ~= nil and rc == 0
> +end
> +
> +local function datetime_lt(lhs, rhs)
> + local rc = datetime_cmp(lhs, rhs)
> + return rc == nil and error('incompatible types for comparison', 2) or
> + rc < 0
> +end
> +
> +local function datetime_le(lhs, rhs)
> + local rc = datetime_cmp(lhs, rhs)
> + return rc == nil and error('incompatible types for comparison', 2) or
> + rc <= 0
> +end
> +
> +local function datetime_serialize(self)
> + return { secs = self.secs, nsec = self.nsec, offset = self.offset }
> +end
> +
> +local function interval_serialize(self)
> + return { secs = self.secs, nsec = self.nsec }
> +end
> +
> +local function datetime_new_raw(secs, nsec, offset)
> + local dt_obj = ffi.new(datetime_t)
> + dt_obj.secs = secs
> + dt_obj.nsec = nsec
> + dt_obj.offset = offset
> + return dt_obj
> +end
> +
> +local function datetime_new_dt(dt, secs, frac, offset)
What's 'frac'? Please either add a comment or give it a better name.
> + local epochV = dt ~= nil and (builtin.tnt_dt_rdn(dt) - DT_EPOCH_1970_OFFSET) *
> + SECS_PER_DAY or 0
AFAIK we don't use camel-case for naming local variables in Lua code.
> + local secsV = secs ~= nil and secs or 0
This is equivalent to 'secs = secs or 0'
> + local fracV = frac ~= nil and frac or 0
> + local ofsV = offset ~= nil and offset or 0
> + return datetime_new_raw(epochV + secsV - ofsV * 60, fracV, ofsV)
> +end
> +
> +-- create datetime given attribute values from obj
> +-- in the "easy mode", providing builder with
> +-- .secs, .nsec, .offset
> +local function datetime_new_obj(obj, ...)
> + if obj == nil or type(obj) ~= 'table' then
type(obj) ~= 'table' implies 'obj' == nil
> + return datetime_new_raw(obj, ...)
> + end
> + local secs = 0
> + local nsec = 0
> + local offset = 0
> +
> + for key, value in pairs(obj) do
> + if key == 'secs' then
> + secs = value
> + elseif key == 'nsec' then
> + nsec = value
> + elseif key == 'offset' then
> + offset = value
> + else
> + error(('unknown attribute %s'):format(key), 2)
> + end
Hmm, do we really need this check? IMO this is less clear and probably
slower than accessing the table directly:
local secs = obj.secs
local nsecs = obj.nsecs
...
I never saw we do anything like this in Lua code.
Same comment for datetime_new and other places where you use pairs()
like this.
> + end
> +
> + return datetime_new_raw(secs, nsec, offset)
> +end
> +
> +-- create datetime given attribute values from obj
Bad comment - what's 'obj'?
> +local function datetime_new(obj)
> + if obj == nil or type(obj) ~= 'table' then
> + return datetime_new_raw(0, 0, 0)
> + end
> + local y = 0
> + local M = 0
> + local d = 0
> + local ymd = false
> +
> + local h = 0
> + local m = 0
> + local s = 0
> + local frac = 0
> + local hms = false
> + local offset = 0
> +
> + local dt = 0
> +
> + for key, value in pairs(obj) do
> + if key == 'year' then
> + check_range(value, {1, 9999}, key)
> + y = value
> + ymd = true
> + elseif key == 'month' then
> + check_range(value, {1, 12}, key)
> + M = value
> + ymd = true
> + elseif key == 'day' then
> + check_range(value, {1, 31}, key)
> + d = value
> + ymd = true
> + elseif key == 'hour' then
> + check_range(value, {0, 23}, key)
> + h = value
> + hms = true
> + elseif key == 'min' or key == 'minute' then
> + check_range(value, {0, 59}, key)
> + m = value
> + hms = true
> + elseif key == 'sec' or key == 'second' then
I don't think we should support both 'sec'/'min' and 'second'/'minute'
here. Since you want to be consistent with os.date() output, I would
leave 'sec'/'min' only.
> + check_range(value, {0, 60}, key)
> + s, frac = math_modf(value)
> + frac = frac * 1e9 -- convert fraction to nanoseconds
So 'frac' actually stores nanoseconds. Please rename accordingly.
> + hms = true
> + elseif key == 'tz' then
> + -- tz offset in minutes
Bad indentation.
> + check_range(value, {0, 720}, key)
> + offset = value
> + elseif key == 'isdst' or key == 'wday' or key =='yday' then -- luacheck: ignore 542
Missing space between '==' and 'yday'.
> + -- ignore unused os.date attributes
> + else
> + error(('unknown attribute %s'):format(key), 2)
> + end
> + end
> +
> + -- .year, .month, .day
> + if ymd then
> + dt = builtin.tnt_dt_from_ymd(y, M, d)
> + end
> +
> + -- .hour, .minute, .second
> + local secs = 0
> + if hms then
> + secs = h * 3600 + m * 60 + s
> + end
> +
> + return datetime_new_dt(dt, secs, frac, offset)
> +end
> +
> +--[[
> + Basic Extended
> + 20121224 2012-12-24 Calendar date (ISO 8601)
> + 2012359 2012-359 Ordinal date (ISO 8601)
> + 2012W521 2012-W52-1 Week date (ISO 8601)
> + 2012Q485 2012-Q4-85 Quarter date
> +]]
Please mention in the comment what this function returns.
Same for other 'parse' functions.
> +
> +local function parse_date(str)
> + check_str("datetime.parse_date()")
check_str(str, ...)
Here and everywhere else.
> + local dt = ffi.new('dt_t[1]')
> + local len = builtin.tnt_dt_parse_iso_date(str, #str, dt)
> + return len > 0 and datetime_new_dt(dt[0]) or nil, tonumber(len)
Is this tonumber() really necessary?
> +end
> +
> +--[[
> + Basic Extended
> + T12 N/A
> + T1230 T12:30
> + T123045 T12:30:45
> + T123045.123456789 T12:30:45.123456789
> + T123045,123456789 T12:30:45,123456789
> +
> + The time designator [T] may be omitted.
> +]]
> +local function parse_time(str)
> + check_str("datetime.parse_time()")
> + local sp = ffi.new('int[1]')
> + local fp = ffi.new('int[1]')
> + local len = builtin.tnt_dt_parse_iso_time(str, #str, sp, fp)
> + return len > 0 and datetime_new_dt(nil, sp[0], fp[0]) or nil,
> + tonumber(len)
> +end
> +
> +--[[
> + Basic Extended
> + Z N/A
> + +hh N/A
> + -hh N/A
> + +hhmm +hh:mm
> + -hhmm -hh:mm
> +]]
> +local function parse_zone(str)
> + check_str("datetime.parse_zone()")
> + local offset = ffi.new('int[1]')
> + local len = builtin.tnt_dt_parse_iso_zone_lenient(str, #str, offset)
> + return len > 0 and datetime_new_dt(nil, nil, nil, offset[0]) or nil,
> + tonumber(len)
> +end
> +
Extra new line.
> +
> +--[[
> + aggregated parse functions
> + assumes to deal with date T time time_zone
> + at once
> +
> + date [T] time [ ] time_zone
> +]]
> +local function parse(str)
> + check_str("datetime.parse()")
> + local dt = ffi.new('dt_t[1]')
> + local len = #str
> + local n = builtin.tnt_dt_parse_iso_date(str, len, dt)
> + local dt_ = dt[0]
> + if n == 0 or len == n then
> + return datetime_new_dt(dt_)
> + end
> +
> + str = str:sub(tonumber(n) + 1)
> +
> + local ch = str:sub(1,1)
Missing space after ','.
> + if ch:match('[Tt ]') == nil then
> + return datetime_new_dt(dt_)
> + end
> +
> + str = str:sub(2)
> + len = #str
> +
> + local sp = ffi.new('int[1]')
> + local fp = ffi.new('int[1]')
> + local n = builtin.tnt_dt_parse_iso_time(str, len, sp, fp)
> + if n == 0 then
> + return datetime_new_dt(dt_)
> + end
> + local sp_ = sp[0]
> + local fp_ = fp[0]
> + if len == n then
> + return datetime_new_dt(dt_, sp_, fp_)
> + end
> +
> + str = str:sub(tonumber(n) + 1)
> +
> + if str:sub(1,1) == ' ' then
Missing space after ','.
> + str = str:sub(2)
> + end
> +
> + len = #str
> +
> + local offset = ffi.new('int[1]')
> + n = builtin.tnt_dt_parse_iso_zone_lenient(str, len, offset)
> + if n == 0 then
> + return datetime_new_dt(dt_, sp_, fp_)
> + end
> + return datetime_new_dt(dt_, sp_, fp_, offset[0])
> +end
> +
> +local function datetime_from(o)
Please add a comment explaining what this function does.
> + if o == nil or type(o) == 'table' then
> + return datetime_new(o)
> + elseif type(o) == 'string' then
> + return parse(o)
> + end
> +end
> +
> +local function local_now()
Please add a comment explaining what this function does.
> + local d = datetime_new_raw(0, 0, 0)
> + builtin.datetime_now(d)
> + return d
> +end
> +
> +-- Change the time-zone to the provided target_offset
> +-- Time `.secs`/`.nsec` are always UTC normalized, we need only to
> +-- reattribute object with different `.offset`
It doesn't change the time-zone of the given object. It creates a new
object with the new timezone. Please fix the comment to avoid confusion.
> +local function datetime_to_tz(self, tgt_ofs)
> + if self.offset == tgt_ofs then
> + return self
> + end
> + if type(tgt_ofs) == 'string' then
> + local obj = parse_zone(tgt_ofs)
> + if obj == nil then
> + error(('%s: invalid time-zone format %s'):format(self, tgt_ofs), 2)
> + else
> + tgt_ofs = obj.offset
target_offset. Please don't use confusing abbreviations.
> + end
> + end
> + return datetime_new_raw(self.secs, self.nsec, tgt_ofs)
> +end
> +
> +local function datetime_index(self, key)
> + if key == 'epoch' or key == 'unixtime' then
> + return self.secs
> + elseif key == 'ts' or key == 'timestamp' then
> + return self.secs + self.nsec / 1e9
> + elseif key == 'ns' or key == 'nanoseconds' then
> + return self.secs * 1e9 + self.nsec
> + elseif key == 'us' or key == 'microseconds' then
> + return self.secs * 1e6 + self.nsec / 1e3
> + elseif key == 'ms' or key == 'milliseconds' then
> + return self.secs * 1e3 + self.nsec / 1e6
> + elseif key == 's' or key == 'seconds' then
> + return self.secs + self.nsec / 1e9
> + elseif key == 'm' or key == 'min' or key == 'minutes' then
> + return (self.secs + self.nsec / 1e9) / 60
> + elseif key == 'hr' or key == 'hours' then
> + return (self.secs + self.nsec / 1e9) / (60 * 60)
> + elseif key == 'd' or key == 'days' then
> + return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
1. There's so many ways to get the same information, for example m,
min, minutes. There should be exactly one way.
2. I'd expect datetime.hour return the number of hours in the current
day, like os.date(), but it returns the number of hours since the
Unix epoch instead. This is confusing and useless.
3. Please document the behavior somewhere in the comments to the code.
4. These if-else's are inefficient AFAIU.
> + elseif key == 'to_utc' then
> + return function(self)
> + return datetime_to_tz(self, 0)
> + end
> + elseif key == 'to_tz' then
> + return function(self, offset)
> + return datetime_to_tz(self, offset)
> + end
> + else
> + error(('unknown attribute %s'):format(key), 2)
> + end
> +end
> +
> +local function datetime_newindex(self, key, value)
> + if key == 'epoch' or key == 'unixtime' then
> + self.secs = value
> + self.nsec, self.offset = 0, 0
> + elseif key == 'ts' or key == 'timestamp' then
> + local secs, frac = math_modf(value)
> + self.secs = secs
> + self.nsec = frac * 1e9
> + self.offset = 0
Do we really want the datetime object to be mutable? If so, allowing to
set its value only to the time since the unix epoch doesn't look
particularly user-friendly.
> + else
> + error(('assigning to unknown attribute %s'):format(key), 2)
> + end
> +end
> +
> +-- sizeof("Wed Jun 30 21:49:08 1993\n")
> +local buf_len = 26
What if year > 9999?
> +
> +local function asctime(o)
> + check_date(o, "datetime:asctime()")
> + local buf = ffi.new('char[?]', buf_len)
I think it would be more efficient to define the buffer in a global
variable - we don't yield in this code so this should be fine. Also,
please give the buffer an appropriate name that would say that it is
used for datetime formatting.
> + return ffi.string(builtin.datetime_asctime(o, buf))
> +end
> +
> +local function ctime(o)
> + check_date(o, "datetime:ctime()")
> + local buf = ffi.new('char[?]', buf_len)
> + return ffi.string(builtin.datetime_ctime(o, buf))
> +end
> +
> +local function strftime(fmt, o)
> + check_date(o, "datetime.strftime()")
> + local sz = 128
> + local buff = ffi.new('char[?]', sz)
> + builtin.datetime_strftime(o, fmt, buff, sz)
> + return ffi.string(buff)
> +end
> +
> +local datetime_mt = {
> + __serialize = datetime_serialize,
> + __eq = datetime_eq,
> + __lt = datetime_lt,
> + __le = datetime_le,
> + __index = datetime_index,
> + __newindex = datetime_newindex,
> +}
> +
> +local interval_mt = {
> + __serialize = interval_serialize,
> + __eq = datetime_eq,
> + __lt = datetime_lt,
> + __le = datetime_le,
> + __index = datetime_index,
Why does datetime_mt has __newindex while interval_mt doesn't?
> +}
> +
> +ffi.metatype(interval_t, interval_mt)
> +ffi.metatype(datetime_t, datetime_mt)
> +
> +return setmetatable(
> + {
> + new = datetime_new,
> + new_raw = datetime_new_obj,
I'm not sure, we need to make the 'raw' function public.
> + interval = interval_new,
Why would anyone want to create a 0 interval?
> +
> + parse = parse,
> + parse_date = parse_date,
> + parse_time = parse_time,
> + parse_zone = parse_zone,
Creating a datetime object without a date sounds confusing.
> +
> + now = local_now,
> + strftime = strftime,
> + asctime = asctime,
> + ctime = ctime,
> +
> + is_datetime = is_datetime,
> + is_interval = is_interval,
I don't see any point in making these functions public.
> + }, {
> + __call = function(self, ...) return datetime_from(...) end
> + }
> +)
> diff --git a/src/lua/init.c b/src/lua/init.c
> index f9738025d..127e935d7 100644
> --- a/src/lua/init.c
> +++ b/src/lua/init.c
> @@ -129,7 +129,8 @@ extern char strict_lua[],
> parse_lua[],
> process_lua[],
> humanize_lua[],
> - memprof_lua[]
> + memprof_lua[],
> + datetime_lua[]
> ;
>
> static const char *lua_modules[] = {
> @@ -184,6 +185,7 @@ static const char *lua_modules[] = {
> "memprof.process", process_lua,
> "memprof.humanize", humanize_lua,
> "memprof", memprof_lua,
> + "datetime", datetime_lua,
> NULL
> };
>
> diff --git a/src/lua/utils.c b/src/lua/utils.c
> index c71cd4857..2c89326f3 100644
> --- a/src/lua/utils.c
> +++ b/src/lua/utils.c
> @@ -48,6 +48,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
> uint32_t CTID_CHAR_PTR;
> uint32_t CTID_CONST_CHAR_PTR;
> uint32_t CTID_UUID;
> +uint32_t CTID_DATETIME = 0;
> +uint32_t CTID_INTERVAL = 0;
Should be called CTID_DATETIME_INTERVAL.
Anyway, it's never used. Please remove.
> +
>
> void *
> luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
> @@ -120,6 +123,12 @@ luaL_pushuuidstr(struct lua_State *L, const struct tt_uuid *uuid)
> lua_pushlstring(L, str, UUID_STR_LEN);
> }
>
> +struct datetime *
> +luaL_pushdatetime(struct lua_State *L)
> +{
> + return luaL_pushcdata(L, CTID_DATETIME);
> +}
> +
This function isn't used in this patch. Please move it to patch 4.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-17 16:52 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-17 19:16 ` Vladimir Davydov via Tarantool-patches
2021-08-18 13:38 ` Safin Timur via Tarantool-patches
2021-08-18 10:03 ` Safin Timur via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 19:16 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Tue, Aug 17, 2021 at 07:52:43PM +0300, Vladimir Davydov wrote:
> On Mon, Aug 16, 2021 at 02:59:36AM +0300, Timur Safin via Tarantool-patches wrote:
> > +/**
> > + * Full datetime structure representing moments
> > + * since Unix Epoch (1970-01-01).
> > + * Time is kept normalized to UTC, time-zone offset
> > + * is informative only.
> > + */
> > +struct datetime {
> > + /** seconds since epoch */
> > + double secs;
>
> Please add a comment explaining why you use 'double' instead of
> an integer type.
Come to think of it, why don't you use two ints here? E.g. one for low
32 bits, another for high 32 bits, or one for years, another for seconds
in the year.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-17 19:16 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 13:38 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 13:38 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On 17.08.2021 22:16, Vladimir Davydov via Tarantool-patches wrote:
> On Tue, Aug 17, 2021 at 07:52:43PM +0300, Vladimir Davydov wrote:
>> On Mon, Aug 16, 2021 at 02:59:36AM +0300, Timur Safin via
> Tarantool-patches wrote:
>>> +/**
>>> + * Full datetime structure representing moments
>>> + * since Unix Epoch (1970-01-01).
>>> + * Time is kept normalized to UTC, time-zone offset
>>> + * is informative only.
>>> + */
>>> +struct datetime {
>>> + /** seconds since epoch */
>>> + double secs;
>>
>> Please add a comment explaining why you use 'double' instead of
>> an integer type.
>
> Come to think of it, why don't you use two ints here? E.g. one for low
> 32 bits, another for high 32 bits, or one for years, another for seconds
> in the year.
>
This IMHO is awkward, and would scream of using int64_t. Especially if
we would need to proceed this bit arithmetics in Lua. May play with that
idea on the subsequent patches, but do not see any performance reason to
do at the moment.
Once again - apparently double is perfectly ok from performance point of
view, and in our date range we want it to work with seconds it's not
loosing any data. Looks like good enough.
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-17 16:52 ` Vladimir Davydov via Tarantool-patches
2021-08-17 19:16 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 10:03 ` Safin Timur via Tarantool-patches
2021-08-18 10:06 ` Safin Timur via Tarantool-patches
2021-08-18 11:45 ` Vladimir Davydov via Tarantool-patches
1 sibling, 2 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 10:03 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On 17.08.2021 19:52, Vladimir Davydov via Tarantool-patches wrote:
> On Mon, Aug 16, 2021 at 02:59:36AM +0300, Timur Safin via
> Tarantool-patches wrote:
>> diff --git a/extra/exports b/extra/exports
>> index 9eaba1282..80eb92abd 100644
>> --- a/extra/exports
>> +++ b/extra/exports
>> @@ -148,8 +148,34 @@ csv_feed
>> csv_iterator_create
>> csv_next
>> csv_setopt
>> +datetime_asctime
>> +datetime_ctime
>> +datetime_now
>> +datetime_strftime
>> +decimal_unpack
>
> decimal_unpack?
Yes, bad copy paste. Supposed to become datetime_unpack. Has been
changed later in the patchset. But now, after Serge complain, I've
reshuffled and fixed original patch, not later.
>
>> decimal_from_string
>> decimal_unpack
>> +tnt_dt_dow
>> +tnt_dt_from_rdn
>> +tnt_dt_from_struct_tm
>> +tnt_dt_from_yd
>> +tnt_dt_from_ymd
>> +tnt_dt_from_yqd
>> +tnt_dt_from_ywd
>> +tnt_dt_parse_iso_date
>> +tnt_dt_parse_iso_time_basic
>> +tnt_dt_parse_iso_time_extended
>> +tnt_dt_parse_iso_time
>> +tnt_dt_parse_iso_zone_basic
>> +tnt_dt_parse_iso_zone_extended
>> +tnt_dt_parse_iso_zone_lenient
>> +tnt_dt_parse_iso_zone
>> +tnt_dt_rdn
>> +tnt_dt_to_struct_tm
>> +tnt_dt_to_yd
>> +tnt_dt_to_ymd
>> +tnt_dt_to_yqd
>> +tnt_dt_to_ywd
>> error_ref
>> error_set_prev
>> error_unref
...
>> new file mode 100644
>> index 000000000..c48295a6f
>> --- /dev/null
>> +++ b/src/lib/core/datetime.c
>> @@ -0,0 +1,96 @@
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>> +#include <string.h>
>> +#include <time.h>
>> +
>> +#include "trivia/util.h"
>> +#include "datetime.h"
>> +
>> +static int
>> +local_dt(int64_t secs)
>> +{
>> + return dt_from_rdn((int)(secs / SECS_PER_DAY) +
> DT_EPOCH_1970_OFFSET);
>> +}
>
> I don't understand what this function does. Please add a comment.
Good point. Added comment.
>
>> +
>> +static struct tm *
>> +datetime_to_tm(const struct datetime *date)
>> +{
>> + static struct tm tm;
>> +
>> + memset(&tm, 0, sizeof(tm));
>> + int64_t secs = date->secs;
>> + dt_to_struct_tm(local_dt(secs), &tm);
>> +
>> + int seconds_of_day = (int64_t)date->secs % SECS_PER_DAY;
>> + tm.tm_hour = (seconds_of_day / 3600) % 24;
>> + tm.tm_min = (seconds_of_day / 60) % 60;
>> + tm.tm_sec = seconds_of_day % 60;
>
> To make the code easier for understanding, please define and use
> constants here and everywhere else in this patch: HOURS_PER_DAY,
> MINUTES_PER_HOUR, NSECS_PER_USEC, etc.
>
>> +
>> + return &tm;
>> +}
>> +
>> +void
>> +datetime_now(struct datetime *now)
>> +{
>> + struct timeval tv;
>> + gettimeofday(&tv, NULL);
>> + now->secs = tv.tv_sec;
>> + now->nsec = tv.tv_usec * 1000;
>> +
>> + time_t now_seconds;
>> + time(&now_seconds);
>
> Can't you use tv.tv_sec here?
Nope. :)
Actually I don't want to know value returned from time() - all I need is
to eventually call localtime[_r]() to get local timezone. If you know
simpler way to determine local timezone without time, then please give
me know.
>> new file mode 100644
>> index 000000000..1a8d7e34f
>> --- /dev/null
>> +++ b/src/lib/core/datetime.h
>> @@ -0,0 +1,95 @@
>> +#pragma once
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>> +#include <stdint.h>
>> +#include <stdbool.h>
>> +#include <stdio.h>
>> +#include "c-dt/dt.h"
>> +
>> +#if defined(__cplusplus)
>> +extern "C"
>> +{
>> +#endif /* defined(__cplusplus) */
>> +
>> +#ifndef SECS_PER_DAY
>> +#define SECS_PER_DAY 86400
>> +#define DT_EPOCH_1970_OFFSET 719163
>
> I don't understand what this constant stores. Please add a comment.
I've documented them after Serge request this way:
------------------------------------------------------
/**
* We count dates since so called "Rata Die" date
* January 1, 0001, Monday (as Day 1).
* But datetime structure keeps seconds since
* Unix "Epoch" date:
* Unix, January 1, 1970, Thursday
*
* The difference between Epoch (1970-01-01)
* and Rata Die (0001-01-01) is 719163 days.
*/
#ifndef SECS_PER_DAY
#define SECS_PER_DAY 86400
#define DT_EPOCH_1970_OFFSET 719163
#endif
/**
* c-dt library uses int as type for dt value, which
* represents the number of days since Rata Die date.
* This implies limits to the number of seconds we
* could safely store in our structures and then safely
* pass to c-dt functions.
*
* So supported ranges will be
* - for seconds [-185604722870400 .. 185480451417600]
* - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
*/
#define MAX_DT_DAY_VALUE (int64_t)INT_MAX
#define MIN_DT_DAY_VALUE (int64_t)INT_MIN
#define SECS_EPOCH_1970_OFFSET \
((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
#define MAX_EPOCH_SECS_VALUE \
(MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
#define MIN_EPOCH_SECS_VALUE \
(MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
------------------------------------------------------
This is more than you probably were asking for at the moment, but bear
with me - we will need those MAX_EPOCH_SECS_VALUE/MIN_EPOCH_SECS_VALUE
in the messagepack serialization/deserialization code.
>
>> +#endif
>> +
>> +/**
>> + * Full datetime structure representing moments
>> + * since Unix Epoch (1970-01-01).
>> + * Time is kept normalized to UTC, time-zone offset
>> + * is informative only.
>> + */
>> +struct datetime {
>> + /** seconds since epoch */
>> + double secs;
>
> Please add a comment explaining why you use 'double' instead of
> an integer type.
>
>> + /** nanoseconds if any */
>> + int32_t nsec;
>
> Why 'nsec', but 'secs'? This is inconsistent. Should be 'nsec' and 'sec'
> or 'nsecs' and 'secs'.
Good point, will rname to use 'secs' and 'nsecs'. But rename will be
massive, so no incremental patch so far.
>
>> + /** offset in minutes from UTC */
>> + int32_t offset;
>
> Why do you use int32_t instead of int for these two members?
Because I need signed integer larger than 16-bit, but less or equal to
32-bit signed value. Simple int is not exactly what I want, depending on
platfrom it could be 4byte or 8bytes long. For example, ILP32, and LP64
do have 4-bytes long int, but there are(were) ILP64 hardware platforms
(like Cray) where int was as 64-bit as long or pointer.
int is not as specific as int32_t in this particular case.
>
> Same comments for the datetime_interval struct.
>
>> +};
>> +
>> +/**
>> + * Date/time interval structure
>> + */
>> +struct datetime_interval {
>> + /** relative seconds delta */
>> + double secs;
>> + /** nanoseconds delta */
>> + int32_t nsec;
>> +};
>> +
>> +/**
>> + * Convert datetime to string using default asctime format
>> + * "Sun Sep 16 01:03:52 1973\n\0"
>> + * Wrapper around reenterable asctime_r() version of POSIX function
>> + * @param date source datetime value
>> + * @sa datetime_ctime
>> + */
>> +char *
>> +datetime_asctime(const struct datetime *date, char *buf);
>> +
>> +char *
>> +datetime_ctime(const struct datetime *date, char *buf);
>> +
>> +size_t
>> +datetime_strftime(const struct datetime *date, const char *fmt, char
> *buf,
>> + uint32_t len);
>> +
>> +void
>> +datetime_now(struct datetime * now);
>
> Extra space after '*'.
Fixed.
>
> Please add comments to all these functions, like you did for
> datetime_asctime.
Added.
>
>> +
>> +#if defined(__cplusplus)
>> +} /* extern "C" */
>> +#endif /* defined(__cplusplus) */
>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>> new file mode 100644
>> index 000000000..ce579828f
>> --- /dev/null
>> +++ b/src/lua/datetime.lua
>> @@ -0,0 +1,500 @@
>> +local ffi = require('ffi')
>> +
>> +ffi.cdef [[
>> +
>> + /*
>> + `c-dt` library functions handles properly both positive and
> negative `dt`
>> + values, where `dt` is a number of dates since Rata Die date
> (0001-01-01).
>> +
>> + For better compactness of our typical data in MessagePack stream we
> shift
>> + root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
>> + actually dt = 719163.
>> +
>> + So here is a simple formula how convert our epoch-based seconds to
> dt values
>> + dt = (secs / 86400) + 719163
>> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
>> + (0001-01-01) in dates.
>> +
>> + */
>> + typedef int dt_t;
>> +
>> + // dt_core.h
>> + dt_t tnt_dt_from_rdn (int n);
>> + dt_t tnt_dt_from_ymd (int y, int m, int d);
>> +
>> + int tnt_dt_rdn (dt_t dt);
>> +
>> + // dt_parse_iso.h
>> + size_t tnt_dt_parse_iso_date (const char *str, size_t len,
> dt_t *dt);
>> + size_t tnt_dt_parse_iso_time (const char *str, size_t len,
> int *sod, int *nsec);
>> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len,
> int *offset);
>
> The line is too long. Please ensure that all lines are <= 80 characters
> long.
Reformatted.
>
>> +
>> + // datetime.c
>> + int
>> + datetime_to_string(const struct datetime * date, char *buf,
> uint32_t len);
>> +
>> + char *
>> + datetime_asctime(const struct datetime *date, char *buf);
>> +
>> + char *
>> + datetime_ctime(const struct datetime *date, char *buf);
>> +
>> + size_t
>> + datetime_strftime(const struct datetime *date, const char *fmt,
> char *buf,
>> + uint32_t len);
>> +
>> + void
>> + datetime_now(struct datetime * now);
>
> Extra space after '*'.
Removed
>
>> +
>> +]]
>> +
>> +local builtin = ffi.C
>> +local math_modf = math.modf
>> +
>> +local SECS_PER_DAY = 86400
>> +
>> +-- c-dt/dt_config.h
>> +
>> +-- Unix, January 1, 1970, Thursday
>> +local DT_EPOCH_1970_OFFSET = 719163
>> +
>> +
>> +local datetime_t = ffi.typeof('struct datetime')
>> +local interval_t = ffi.typeof('struct datetime_interval')
>> +
>> +local function is_interval(o)
>> + return type(o) == 'cdata' and ffi.istype(interval_t, o)
>
> The check for 'cdata' is redundant. ffi.istype alone should be enough
> AFAIK.
Wanted to object, but have verified and indeed, in that particular order
(first ffi metatype reference, then value) it's accepting every type.
Apparently I'd got wrong impression when I passed arguments in the wrong
order. Updated.
>
>> +end
>> +
>> +local function is_datetime(o)
>> + return type(o) == 'cdata' and ffi.istype(datetime_t, o)
>> +end
>> +
>> +local function is_date_interval(o)
>> + return type(o) == 'cdata' and
>> + (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
>> +end
>> +
>> +local function interval_new()
>> + local interval = ffi.new(interval_t)
>> + return interval
>> +end
>> +
>> +local function check_date(o, message)
>> + if not is_datetime(o) then
>> + return error(("%s: expected datetime, but received %s"):
>> + format(message, o), 2)
>> + end
>> +end
>> +
>> +local function check_str(s, message)
>> + if not type(s) == 'string' then
>> + return error(("%s: expected string, but received %s"):
>> + format(message, s), 2)
>> + end
>> +end
>> +
>> +local function check_range(v, range, txt)
>> + assert(#range == 2)
>> + if v < range[1] or v > range[2] then
>> + error(('value %d of %s is out of allowed range [%d, %d]'):
>> + format(v, txt, range[1], range[2]), 4)
>> + end
>> +end
>> +
>> +local function datetime_cmp(lhs, rhs)
>> + if not is_date_interval(lhs) or
>> + not is_date_interval(rhs) then
>> + return nil
>
> Bad indentation.
Fixed.
>
>> + end
>> + local sdiff = lhs.secs - rhs.secs
>> + return sdiff ~= 0 and sdiff or (lhs.nsec - rhs.nsec)
>> +end
>> +
>> +local function datetime_eq(lhs, rhs)
>> + local rc = datetime_cmp(lhs, rhs)
>> + return rc ~= nil and rc == 0
>> +end
>> +
>> +local function datetime_lt(lhs, rhs)
>> + local rc = datetime_cmp(lhs, rhs)
>> + return rc == nil and error('incompatible types for comparison', 2)
> or
>> + rc < 0
>> +end
>> +
>> +local function datetime_le(lhs, rhs)
>> + local rc = datetime_cmp(lhs, rhs)
>> + return rc == nil and error('incompatible types for comparison', 2)
> or
>> + rc <= 0
>> +end
>> +
>> +local function datetime_serialize(self)
>> + return { secs = self.secs, nsec = self.nsec, offset = self.offset }
>> +end
>> +
>> +local function interval_serialize(self)
>> + return { secs = self.secs, nsec = self.nsec }
>> +end
>> +
>> +local function datetime_new_raw(secs, nsec, offset)
>> + local dt_obj = ffi.new(datetime_t)
>> + dt_obj.secs = secs
>> + dt_obj.nsec = nsec
>> + dt_obj.offset = offset
>> + return dt_obj
>> +end
>> +
>> +local function datetime_new_dt(dt, secs, frac, offset)
>
> What's 'frac'? Please either add a comment or give it a better name.
fraction part. Renamed to full fraction.
>
>> + local epochV = dt ~= nil and (builtin.tnt_dt_rdn(dt) -
> DT_EPOCH_1970_OFFSET) *
>> + SECS_PER_DAY or 0
>
> AFAIK we don't use camel-case for naming local variables in Lua code.
>
>> + local secsV = secs ~= nil and secs or 0
>
> This is equivalent to 'secs = secs or 0'
Indeed. Fixed.
>
>> + local fracV = frac ~= nil and frac or 0
>> + local ofsV = offset ~= nil and offset or 0
>> + return datetime_new_raw(epochV + secsV - ofsV * 60, fracV, ofsV)
>> +end
>> +
>> +-- create datetime given attribute values from obj
>> +-- in the "easy mode", providing builder with
>> +-- .secs, .nsec, .offset
>> +local function datetime_new_obj(obj, ...)
>> + if obj == nil or type(obj) ~= 'table' then
>
> type(obj) ~= 'table' implies 'obj' == nil
I meant that obj may be simple number, not builder object.
>
>> + return datetime_new_raw(obj, ...)
>> + end
>> + local secs = 0
>> + local nsec = 0
>> + local offset = 0
>> +
>> + for key, value in pairs(obj) do
>> + if key == 'secs' then
>> + secs = value
>> + elseif key == 'nsec' then
>> + nsec = value
>> + elseif key == 'offset' then
>> + offset = value
>> + else
>> + error(('unknown attribute %s'):format(key), 2)
>> + end
>
> Hmm, do we really need this check? IMO this is less clear and probably
> slower than accessing the table directly:
>
> local secs = obj.secs
> local nsecs = obj.nsecs
> ...
Ok, let me put you to the context, where we have been iteration before.
Here we see smaller part of larger code which used to be emulating named
arguments for creating datetime using attributes from passed. It's a
standard idiom for emulating named arguments in a languages where there
is no such syntax, but there is json-like table/object/hash facilities.
https://www.lua.org/pil/5.3.html
The idea is to allow constructions like below (with any set of
attributes passed)
date = require 'datetime'
T = date.new{ year = 20201, month = 11, day = 15,
hour = 1, minute = 10, second = 01, tz = 180}
The problem was that there is set of attributes, which we want to use
for initialization via this same named attributes idiom, but directly
using .secs, .nsec, .offset attributes when we know what we are doing.
T = date.new_raw{ secs = 0, nsec = 0, offset = 180}
That's why this silly separate version has been created which strangely
parsing set of attributes accessible in the cdata object. (If there is
passed cdata object, but it's not)
[Worth to note, that at the moment in addition to initialization object
`date.new_raw` now handles gracefully this case of calling it without
creating initialization object, but passing arguments:
T = date.new_raw(0, 0, 180)
This is why there is check for object in obj `type(obj) ~= 'table'`
]
Now you have asked that I realized that ffi.new could do the
initialization magic for us, and instead of this for-loop we could
simply pass an object to the 2nd argument of ffi.new.
tarantool> o = { secs = 0, nsec = 0, offset = 180}
---
...
tarantool> ffi = require 'ffi'
---
...
tarantool> T = ffi.new('struct datetime', o)
---
...
tarantool> T
---
- 1970-01-01T03:00+03:00
...
The problem is - it will be silently ignoring bogus attributes we may
pass to this initialization object, which do not map directly to the set
of known fields in a structure initialized.
Which is not a problem for the code above, where we verify set of
attributes in an object, and bail out if there is any unknown attribute.
Performance-wise I do not expect such fancy initialization of objects
would be on a critical path. The expected scenario for average scenario
- creating datetime objects using textual strings (from logs). And this
is very fast, due to c-dt parsing speed.
So here is dilema:
- we could directly pass initialization object to the ffi.new. But allow
all kinds of unknown attributes;
- or we could check all attributes in a slightly slower loop, but
provide nice user visible diagnostics for an error.
What would you recommend? (I prefer implementation with checks and
human-understandable errors)
>
> I never saw we do anything like this in Lua code.
>
> Same comment for datetime_new and other places where you use pairs()
> like this.
>
It's still same approach to use initialization object for "named
arguments" idiom.
>> + end
>> +
>> + return datetime_new_raw(secs, nsec, offset)
>> +end
>> +
>> +-- create datetime given attribute values from obj
>
> Bad comment - what's 'obj'?
Yes, probably the confusion created because I was too terse here, and
not explained that obj is initialization object for "named arguments"
approach.
>
>> +local function datetime_new(obj)
>> + if obj == nil or type(obj) ~= 'table' then
>> + return datetime_new_raw(0, 0, 0)
>> + end
>> + local y = 0
>> + local M = 0
>> + local d = 0
>> + local ymd = false
>> +
>> + local h = 0
>> + local m = 0
>> + local s = 0
>> + local frac = 0
>> + local hms = false
>> + local offset = 0
>> +
>> + local dt = 0
>> +
>> + for key, value in pairs(obj) do
>> + if key == 'year' then
>> + check_range(value, {1, 9999}, key)
>> + y = value
>> + ymd = true
>> + elseif key == 'month' then
>> + check_range(value, {1, 12}, key)
>> + M = value
>> + ymd = true
>> + elseif key == 'day' then
>> + check_range(value, {1, 31}, key)
>> + d = value
>> + ymd = true
>> + elseif key == 'hour' then
>> + check_range(value, {0, 23}, key)
>> + h = value
>> + hms = true
>> + elseif key == 'min' or key == 'minute' then
>> + check_range(value, {0, 59}, key)
>> + m = value
>> + hms = true
>> + elseif key == 'sec' or key == 'second' then
>
> I don't think we should support both 'sec'/'min' and 'second'/'minute'
> here. Since you want to be consistent with os.date() output, I would
> leave 'sec'/'min' only.
Originally, there were only human-readable names like 'second' or
'minute', os.date compatibility added only lately. Because, it costs
nothing (comparing to all other overhead).
I'd prefer to have ability to use full readable names, not only those
from set of os.date. It costs nothing, but make it's friendlier.
>
>> + check_range(value, {0, 60}, key)
>> + s, frac = math_modf(value)
>> + frac = frac * 1e9 -- convert fraction to nanoseconds
>
> So 'frac' actually stores nanoseconds. Please rename accordingly.
Yes, this is fractional part of seconds, but represented as integers in
nanoseconds units. Renamed.
>> + hms = true
>> + elseif key == 'tz' then
>> + -- tz offset in minutes
>
> Bad indentation.
Fixed.
>
>> + check_range(value, {0, 720}, key)
>> + offset = value
>> + elseif key == 'isdst' or key == 'wday' or key =='yday' then --
> luacheck: ignore 542
>
> Missing space between '==' and 'yday'.
Fixed.
>
>> + -- ignore unused os.date attributes
>> + else
>> + error(('unknown attribute %s'):format(key), 2)
>> + end
>> + end
>> +
>> + -- .year, .month, .day
>> + if ymd then
>> + dt = builtin.tnt_dt_from_ymd(y, M, d)
>> + end
>> +
>> + -- .hour, .minute, .second
>> + local secs = 0
>> + if hms then
>> + secs = h * 3600 + m * 60 + s
>> + end
>> +
>> + return datetime_new_dt(dt, secs, frac, offset)
>> +end
>> +
>> +--[[
>> + Basic Extended
>> + 20121224 2012-12-24 Calendar date (ISO 8601)
>> + 2012359 2012-359 Ordinal date (ISO 8601)
>> + 2012W521 2012-W52-1 Week date (ISO 8601)
>> + 2012Q485 2012-Q4-85 Quarter date
>> +]]
>
> Please mention in the comment what this function returns.
> Same for other 'parse' functions.
Good point. Clarified.
>
>> +
>> +local function parse_date(str)
>> + check_str("datetime.parse_date()")
>
> check_str(str, ...)
>
> Here and everywhere else.
>
>> + local dt = ffi.new('dt_t[1]')
>> + local len = builtin.tnt_dt_parse_iso_date(str, #str, dt)
>> + return len > 0 and datetime_new_dt(dt[0]) or nil, tonumber(len)
>
> Is this tonumber() really necessary?
Yup, size_t is boxed in cdata.
```
tarantool> ffi = require 'ffi'
---
...
tarantool> ffi.cdef [[ size_t tnt_dt_parse_iso_date (const char *str,
size_t len, dt_t *dt); ]]
---
...
tarantool> ffi.cdef [[ typedef int dt_t; ]]
---
...
tarantool> dt = ffi.new 'dt_t[1]'
---
...
tarantool> r = ffi.C.tnt_dt_parse_iso_date('1970-01-01', 10, dt)
---
...
tarantool> dt[0]
---
- 719163
...
tarantool> type(r)
---
- cdata
...
tarantool> ffi.typeof(r)
---
- ctype<uint64_t>
...
```
>
>> +end
>> +
>> +--[[
>> + Basic Extended
>> + T12 N/A
>> + T1230 T12:30
>> + T123045 T12:30:45
>> + T123045.123456789 T12:30:45.123456789
>> + T123045,123456789 T12:30:45,123456789
>> +
>> + The time designator [T] may be omitted.
>> +]]
>> +local function parse_time(str)
>> + check_str("datetime.parse_time()")
>> + local sp = ffi.new('int[1]')
>> + local fp = ffi.new('int[1]')
>> + local len = builtin.tnt_dt_parse_iso_time(str, #str, sp, fp)
>> + return len > 0 and datetime_new_dt(nil, sp[0], fp[0]) or nil,
>> + tonumber(len)
>> +end
>> +
>> +--[[
>> + Basic Extended
>> + Z N/A
>> + +hh N/A
>> + -hh N/A
>> + +hhmm +hh:mm
>> + -hhmm -hh:mm
>> +]]
>> +local function parse_zone(str)
>> + check_str("datetime.parse_zone()")
>> + local offset = ffi.new('int[1]')
>> + local len = builtin.tnt_dt_parse_iso_zone_lenient(str, #str,
> offset)
>> + return len > 0 and datetime_new_dt(nil, nil, nil, offset[0]) or
> nil,
>> + tonumber(len)
>> +end
>> +
>
> Extra new line.
Removed
>
>> +
>> +--[[
>> + aggregated parse functions
>> + assumes to deal with date T time time_zone
>> + at once
>> +
>> + date [T] time [ ] time_zone
>> +]]
>> +local function parse(str)
>> + check_str("datetime.parse()")
>> + local dt = ffi.new('dt_t[1]')
>> + local len = #str
>> + local n = builtin.tnt_dt_parse_iso_date(str, len, dt)
>> + local dt_ = dt[0]
>> + if n == 0 or len == n then
>> + return datetime_new_dt(dt_)
>> + end
>> +
>> + str = str:sub(tonumber(n) + 1)
>> +
>> + local ch = str:sub(1,1)
>
> Missing space after ','.
Fixed
>
>> + if ch:match('[Tt ]') == nil then
>> + return datetime_new_dt(dt_)
>> + end
>> +
>> + str = str:sub(2)
>> + len = #str
>> +
>> + local sp = ffi.new('int[1]')
>> + local fp = ffi.new('int[1]')
>> + local n = builtin.tnt_dt_parse_iso_time(str, len, sp, fp)
>> + if n == 0 then
>> + return datetime_new_dt(dt_)
>> + end
>> + local sp_ = sp[0]
>> + local fp_ = fp[0]
>> + if len == n then
>> + return datetime_new_dt(dt_, sp_, fp_)
>> + end
>> +
>> + str = str:sub(tonumber(n) + 1)
>> +
>> + if str:sub(1,1) == ' ' then
>
> Missing space after ','.
Fixed
>
>> + str = str:sub(2)
>> + end
>> +
>> + len = #str
>> +
>> + local offset = ffi.new('int[1]')
>> + n = builtin.tnt_dt_parse_iso_zone_lenient(str, len, offset)
>> + if n == 0 then
>> + return datetime_new_dt(dt_, sp_, fp_)
>> + end
>> + return datetime_new_dt(dt_, sp_, fp_, offset[0])
>> +end
>> +
>> +local function datetime_from(o)
>
> Please add a comment explaining what this function does.
Added.
>
>> + if o == nil or type(o) == 'table' then
>> + return datetime_new(o)
>> + elseif type(o) == 'string' then
>> + return parse(o)
>> + end
>> +end
>> +
>> +local function local_now()
>
> Please add a comment explaining what this function does.
Added.
>
>> + local d = datetime_new_raw(0, 0, 0)
>> + builtin.datetime_now(d)
>> + return d
>> +end
>> +
>> +-- Change the time-zone to the provided target_offset
>> +-- Time `.secs`/`.nsec` are always UTC normalized, we need only to
>> +-- reattribute object with different `.offset`
>
> It doesn't change the time-zone of the given object. It creates a new
> object with the new timezone. Please fix the comment to avoid confusion.
Indeed. Implementation has been changed to make original object immune
to the timezone changes, but comment has not been updated. Updated now.
>
>> +local function datetime_to_tz(self, tgt_ofs)
>> + if self.offset == tgt_ofs then
>> + return self
>> + end
>> + if type(tgt_ofs) == 'string' then
>> + local obj = parse_zone(tgt_ofs)
>> + if obj == nil then
>> + error(('%s: invalid time-zone format %s'):format(self,
> tgt_ofs), 2)
>> + else
>> + tgt_ofs = obj.offset
>
> target_offset. Please don't use confusing abbreviations.
>
>> + end
>> + end
>> + return datetime_new_raw(self.secs, self.nsec, tgt_ofs)
>> +end
>> +
>> +local function datetime_index(self, key)
>> + if key == 'epoch' or key == 'unixtime' then
>> + return self.secs
>> + elseif key == 'ts' or key == 'timestamp' then
>> + return self.secs + self.nsec / 1e9
>> + elseif key == 'ns' or key == 'nanoseconds' then
>> + return self.secs * 1e9 + self.nsec
>> + elseif key == 'us' or key == 'microseconds' then
>> + return self.secs * 1e6 + self.nsec / 1e3
>> + elseif key == 'ms' or key == 'milliseconds' then
>> + return self.secs * 1e3 + self.nsec / 1e6
>> + elseif key == 's' or key == 'seconds' then
>> + return self.secs + self.nsec / 1e9
>> + elseif key == 'm' or key == 'min' or key == 'minutes' then
>> + return (self.secs + self.nsec / 1e9) / 60
>> + elseif key == 'hr' or key == 'hours' then
>> + return (self.secs + self.nsec / 1e9) / (60 * 60)
>> + elseif key == 'd' or key == 'days' then
>> + return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
>
> 1. There's so many ways to get the same information, for example m,
> min, minutes. There should be exactly one way.
Disagreed. I prefer friendlier approach.
>
> 2. I'd expect datetime.hour return the number of hours in the current
> day, like os.date(), but it returns the number of hours since the
> Unix epoch instead. This is confusing and useless.
Well, at least this is consistent with seconds and their fractions of
different units. If there would be demand for os.date compatible table
format, we might extend interface with corresponding accessor. But I do
not foresee it, because os.date is totally broken if we need correct
handling of timezone information. AFAIK it's always using local time
context.
>
> 3. Please document the behavior somewhere in the comments to the code.
Documented.
>
> 4. These if-else's are inefficient AFAIU.
Quite the contrary (if we compare to the idiomatic closure handlers
case). If/ifelse is the fastest way to handle such cases in Lua.
Here is earlier bench we have discussed with Vlad and Oleg Babin, and
after which I've changed from using hash based handlers to the series of
ifs.
https://gist.github.com/tsafin/31cc9b0872b6015904fcc90d97740770
>
>> + elseif key == 'to_utc' then
>> + return function(self)
>> + return datetime_to_tz(self, 0)
>> + end
>> + elseif key == 'to_tz' then
>> + return function(self, offset)
>> + return datetime_to_tz(self, offset)
>> + end
>> + else
>> + error(('unknown attribute %s'):format(key), 2)
>> + end
>> +end
>> +
>> +local function datetime_newindex(self, key, value)
>> + if key == 'epoch' or key == 'unixtime' then
>> + self.secs = value
>> + self.nsec, self.offset = 0, 0
>> + elseif key == 'ts' or key == 'timestamp' then
>> + local secs, frac = math_modf(value)
>> + self.secs = secs
>> + self.nsec = frac * 1e9
>> + self.offset = 0
>
> Do we really want the datetime object to be mutable? If so, allowing to
> set its value only to the time since the unix epoch doesn't look
> particularly user-friendly.
I do want. That was request from customer - to have ability to modify by
timestamp from Epoch.
>
>> + else
>> + error(('assigning to unknown attribute %s'):format(key), 2)
>> + end
>> +end
>> +
>> +-- sizeof("Wed Jun 30 21:49:08 1993\n")
>> +local buf_len = 26
>
> What if year > 9999?
Good point. IIRC both BSD libc and glibc have hardcoded length limit at
around this size and not ready to handle such huge dates properly.
>
>> +
>> +local function asctime(o)
>> + check_date(o, "datetime:asctime()")
>> + local buf = ffi.new('char[?]', buf_len)
>
> I think it would be more efficient to define the buffer in a global
> variable - we don't yield in this code so this should be fine. Also,
> please give the buffer an appropriate name that would say that it is
> used for datetime formatting.
Good point. Changed to using outer variables in asctime and ctime functions.
>
>> + return ffi.string(builtin.datetime_asctime(o, buf))
>> +end
>> +
>> +local function ctime(o)
>> + check_date(o, "datetime:ctime()")
>> + local buf = ffi.new('char[?]', buf_len)
>> + return ffi.string(builtin.datetime_ctime(o, buf))
>> +end
>> +
>> +local function strftime(fmt, o)
>> + check_date(o, "datetime.strftime()")
>> + local sz = 128
>> + local buff = ffi.new('char[?]', sz)
>> + builtin.datetime_strftime(o, fmt, buff, sz)
>> + return ffi.string(buff)
>> +end
>> +
>> +local datetime_mt = {
>> + __serialize = datetime_serialize,
>> + __eq = datetime_eq,
>> + __lt = datetime_lt,
>> + __le = datetime_le,
>> + __index = datetime_index,
>> + __newindex = datetime_newindex,
>> +}
>> +
>> +local interval_mt = {
>> + __serialize = interval_serialize,
>> + __eq = datetime_eq,
>> + __lt = datetime_lt,
>> + __le = datetime_le,
>> + __index = datetime_index,
>
> Why does datetime_mt has __newindex while interval_mt doesn't?
I did not foresee the need. Should we?
And semantically datetime has persistence and life-time, but intervals
have not. [And besides all we already provide a plenty of helpers for
creation of intervals]
>
>> +}
>> +
>> +ffi.metatype(interval_t, interval_mt)
>> +ffi.metatype(datetime_t, datetime_mt)
>> +
>> +return setmetatable(
>> + {
>> + new = datetime_new,
>> + new_raw = datetime_new_obj,
>
> I'm not sure, we need to make the 'raw' function public.
That's for second kind of constructors using initialization object but
with low level attributes {secs = 0, nsec = 0, offset}. It's not for
everybody, but still provides some ergonomics.
>
>> + interval = interval_new,
>
> Why would anyone want to create a 0 interval?
Good point, especiall taking into account that there is no way to modify
it, beyond directly modifying .secs and .nsec fields of cdata object.
Will delete it. We already have a plenty of helpers to construct
different kinds of intervals.
>
>> +
>> + parse = parse,
>> + parse_date = parse_date,
>
>> + parse_time = parse_time,
>> + parse_zone = parse_zone,
>
> Creating a datetime object without a date sounds confusing.
:) But I do foresee the need to parse parts of datetime string, and I
did not want to establish special type for that.
Now you have asked it I realized we may better create interval objects
of approapriate time. But in this case we need to add timezone to
interval record. So it will be able to represent timezone shifts.
Ok, heer is the deal:
- at the moment those partial parsed objects store their data to the
partial datetime object created. That's confusing but do not require
modifications in interval arithmetic.
- we may start to store partial time and timezone information into
generic intervals. But it would require extension of interval arithmetic
to be ready to shift by timezone delta. [Does it even make any sense?]
What do you think?
>
>> +
>> + now = local_now,
>> + strftime = strftime,
>> + asctime = asctime,
>> + ctime = ctime,
>> +
>> + is_datetime = is_datetime,
>> + is_interval = is_interval,
>
> I don't see any point in making these functions public.
I was asked by customer to introduce such.
>
>> + }, {
>> + __call = function(self, ...) return datetime_from(...) end
>> + }
>> +)
>> diff --git a/src/lua/init.c b/src/lua/init.c
>> index f9738025d..127e935d7 100644
>> --- a/src/lua/init.c
>> +++ b/src/lua/init.c
>> @@ -129,7 +129,8 @@ extern char strict_lua[],
>> parse_lua[],
>> process_lua[],
>> humanize_lua[],
>> - memprof_lua[]
>> + memprof_lua[],
>> + datetime_lua[]
>> ;
>>
>> static const char *lua_modules[] = {
>> @@ -184,6 +185,7 @@ static const char *lua_modules[] = {
>> "memprof.process", process_lua,
>> "memprof.humanize", humanize_lua,
>> "memprof", memprof_lua,
>> + "datetime", datetime_lua,
>> NULL
>> };
>>
>> diff --git a/src/lua/utils.c b/src/lua/utils.c
>> index c71cd4857..2c89326f3 100644
>> --- a/src/lua/utils.c
>> +++ b/src/lua/utils.c
>> @@ -48,6 +48,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
>> uint32_t CTID_CHAR_PTR;
>> uint32_t CTID_CONST_CHAR_PTR;
>> uint32_t CTID_UUID;
>> +uint32_t CTID_DATETIME = 0;
>> +uint32_t CTID_INTERVAL = 0;
>
> Should be called CTID_DATETIME_INTERVAL.
>
> Anyway, it's never used. Please remove.
It's used in later patches. Wil lsquash and rename.
>
>> +
>>
>> void *
>> luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
>> @@ -120,6 +123,12 @@ luaL_pushuuidstr(struct lua_State *L, const struct
> tt_uuid *uuid)
>> lua_pushlstring(L, str, UUID_STR_LEN);
>> }
>>
>> +struct datetime *
>> +luaL_pushdatetime(struct lua_State *L)
>> +{
>> + return luaL_pushcdata(L, CTID_DATETIME);
>> +}
>> +
>
> This function isn't used in this patch. Please move it to patch 4.
>
Looks like it will be used once squashed.
Thanks,
Timur
P.S.
I need to jump to the different location, so the rest of responses will
be send in a couple of hours. Please answer some questions - I 'm
interested in your opinion.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-18 10:03 ` Safin Timur via Tarantool-patches
@ 2021-08-18 10:06 ` Safin Timur via Tarantool-patches
2021-08-18 11:45 ` Vladimir Davydov via Tarantool-patches
1 sibling, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 10:06 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
Forgot to insert an inlined incremental changes introduced:
-----------------------------------------------------------
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index 6d92645f7..4cd93b264 100644
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -12,6 +12,11 @@
#include "trivia/util.h"
#include "datetime.h"
+/*
+ * Given the seconds from Epoch (1970-01-01) we calculate date
+ * since Rata Die (0001-01-01).
+ * DT_EPOCH_1970_OFFSET is the distance in days from Rata Die to Epoch.
+ */
static int
local_dt(int64_t secs)
{
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index b8d179600..278ae6a87 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -111,9 +111,25 @@ datetime_to_string(const struct datetime *date,
char *buf, int len);
char *
datetime_asctime(const struct datetime *date, char *buf);
+/**
+ * Convert datetime to string using default ctime format, using
+ * local timezone for representation.
+ * "Sun Sep 16 01:03:52 1973\n\0"
+ * Wrapper around reenterable ctime_r() version of POSIX function
+ * @param date source datetime value
+ * @sa datetime_asctime
+ */
char *
datetime_ctime(const struct datetime *date, char *buf);
+/**
+ * Convert datetime to string using default format provided
+ * Wrapper around standard strftime() function
+ * @param date source datetime value
+ * @param fmt format
+ * @param buf output buffer
+ * @param len size of output buffer
+ */
size_t
datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
uint32_t len);
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index ae22c598d..a0fa29167 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -16,51 +16,49 @@ local ffi = require('ffi')
-- dt_core.h definitions
ffi.cdef [[
- typedef int dt_t;
- dt_t tnt_dt_from_rdn (int n);
- dt_t tnt_dt_from_ymd (int y, int m, int d);
+typedef int dt_t;
+
+dt_t tnt_dt_from_rdn (int n);
+dt_t tnt_dt_from_ymd (int y, int m, int d);
+int tnt_dt_rdn (dt_t dt);
- int tnt_dt_rdn (dt_t dt);
]]
-- dt_arithmetic.h definitions
ffi.cdef [[
- typedef enum {
- DT_EXCESS,
- DT_LIMIT,
- DT_SNAP
- } dt_adjust_t;
-
- dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t
adjust);
- dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t
adjust);
- dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t
adjust);
+
+typedef enum {
+ DT_EXCESS,
+ DT_LIMIT,
+ DT_SNAP
+} dt_adjust_t;
+
+dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t adjust);
+dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t adjust);
+dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t adjust);
+
]]
-- dt_parse_iso.h definitions
ffi.cdef [[
- size_t tnt_dt_parse_iso_date (const char *str, size_t len,
dt_t *dt);
- size_t tnt_dt_parse_iso_time (const char *str, size_t len,
int *sod, int *nsec);
- size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len,
int *offset);
+
+size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
+size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod,
int *nsec);
+size_t tnt_dt_parse_iso_zone_lenient(const char *str, size_t len, int
*offset);
+
]]
-- Tarantool functions - datetime.c
ffi.cdef [[
- int
- datetime_to_string(const struct datetime * date, char *buf, int len);
-
- char *
- datetime_asctime(const struct datetime *date, char *buf);
- char *
- datetime_ctime(const struct datetime *date, char *buf);
+int datetime_to_string(const struct datetime * date, char *buf, int
len);
+char *datetime_asctime(const struct datetime *date, char *buf);
+char *datetime_ctime(const struct datetime *date, char *buf);
+size_t datetime_strftime(const struct datetime *date, const char *fmt,
char *buf,
+ uint32_t len);
+void datetime_now(struct datetime *now);
- size_t
- datetime_strftime(const struct datetime *date, const char *fmt,
char *buf,
- uint32_t len);
-
- void
- datetime_now(struct datetime * now);
]]
local builtin = ffi.C
@@ -92,22 +90,17 @@ local interval_months_t = ffi.typeof('struct
interval_months')
local interval_years_t = ffi.typeof('struct interval_years')
local function is_interval(o)
- return type(o) == 'cdata' and
- (ffi.istype(interval_t, o) or
- ffi.istype(interval_months_t, o) or
- ffi.istype(interval_years_t, o))
+ return ffi.istype(interval_t, o) or
+ ffi.istype(interval_months_t, o) or
+ ffi.istype(interval_years_t, o)
end
local function is_datetime(o)
- return type(o) == 'cdata' and ffi.istype(datetime_t, o)
+ return ffi.istype(datetime_t, o)
end
local function is_date_interval(o)
- return type(o) == 'cdata' and
- (ffi.istype(datetime_t, o) or
- ffi.istype(interval_t, o) or
- ffi.istype(interval_months_t, o) or
- ffi.istype(interval_years_t, o))
+ return is_datetime(o) or is_interval(o)
end
local function interval_new()
@@ -230,9 +223,8 @@ local function normalize_nsec(secs, nsec)
end
local function datetime_cmp(lhs, rhs)
- if not is_date_interval(lhs) or
- not is_date_interval(rhs) then
- return nil
+ if not is_date_interval(lhs) or not is_date_interval(rhs) then
+ return nil
end
local sdiff = lhs.secs - rhs.secs
return sdiff ~= 0 and sdiff or (lhs.nsec - rhs.nsec)
@@ -271,12 +263,12 @@ local function datetime_new_raw(secs, nsec, offset)
return dt_obj
end
-local function datetime_new_dt(dt, secs, frac, offset)
+local function datetime_new_dt(dt, secs, fraction, offset)
local epochV = dt ~= nil and (builtin.tnt_dt_rdn(dt) -
DT_EPOCH_1970_OFFSET) *
SECS_PER_DAY or 0
- local secsV = secs ~= nil and secs or 0
- local fracV = frac ~= nil and frac or 0
- local ofsV = offset ~= nil and offset or 0
+ local secsV = secs or 0
+ local fracV = fraction or 0
+ local ofsV = offset or 0
return datetime_new_raw(epochV + secsV - ofsV * 60, fracV, ofsV)
end
@@ -319,7 +311,7 @@ local function datetime_new(obj)
local h = 0
local m = 0
local s = 0
- local frac = 0
+ local nsec = 0
local hms = false
local offset = 0
@@ -348,14 +340,15 @@ local function datetime_new(obj)
hms = true
elseif key == 'sec' or key == 'second' then
check_range(value, {0, 60}, key)
- s, frac = math_modf(value)
- frac = frac * 1e9 -- convert fraction to nanoseconds
+ s, nsec = math_modf(value)
+ nsec = nsec * 1e9 -- convert fraction to nanoseconds
hms = true
elseif key == 'tz' then
- -- tz offset in minutes
+ -- tz offset in minutes
check_range(value, {0, 720}, key)
offset = value
- elseif key == 'isdst' or key == 'wday' or key =='yday' then --
luacheck: ignore 542
+ elseif key == 'isdst' or key == 'wday' or
+ key == 'yday' then -- luacheck: ignore 542
-- ignore unused os.date attributes
else
error(('unknown attribute %s'):format(key), 2)
@@ -373,7 +366,7 @@ local function datetime_new(obj)
secs = h * 3600 + m * 60 + s
end
- return datetime_new_dt(dt, secs, frac, offset)
+ return datetime_new_dt(dt, secs, nsec, offset)
end
local function datetime_tostring(o)
@@ -537,6 +530,9 @@ end
2012359 2012-359 Ordinal date (ISO 8601)
2012W521 2012-W52-1 Week date (ISO 8601)
2012Q485 2012-Q4-85 Quarter date
+
+ Returns pair of constructed datetime object, and length of string
+ which has been accepted by parser.
]]
local function parse_date(str)
@@ -555,6 +551,9 @@ end
T123045,123456789 T12:30:45,123456789
The time designator [T] may be omitted.
+
+ Returns pair of constructed datetime object, and length of string
+ which has been accepted by parser.
]]
local function parse_time(str)
check_str("datetime.parse_time()")
@@ -572,6 +571,9 @@ end
-hh N/A
+hhmm +hh:mm
-hhmm -hh:mm
+
+ Returns pair of constructed datetime object, and length of string
+ which has been accepted by parser.
]]
local function parse_zone(str)
check_str("datetime.parse_zone()")
@@ -581,13 +583,14 @@ local function parse_zone(str)
tonumber(len)
end
-
--[[
aggregated parse functions
assumes to deal with date T time time_zone
at once
date [T] time [ ] time_zone
+
+ Returns constructed datetime object.
]]
local function parse(str)
check_str("datetime.parse()")
@@ -601,7 +604,7 @@ local function parse(str)
str = str:sub(tonumber(n) + 1)
- local ch = str:sub(1,1)
+ local ch = str:sub(1, 1)
if ch:match('[Tt ]') == nil then
return datetime_new_dt(dt_)
end
@@ -623,7 +626,7 @@ local function parse(str)
str = str:sub(tonumber(n) + 1)
- if str:sub(1,1) == ' ' then
+ if str:sub(1, 1) == ' ' then
str = str:sub(2)
end
@@ -637,6 +640,11 @@ local function parse(str)
return datetime_new_dt(dt_, sp_, fp_, offset[0])
end
+--[[
+ Dispatch function to create datetime from string or table.
+ Creates default timeobject (pointing to Epoch date) if
+ called without arguments.
+]]
local function datetime_from(o)
if o == nil or type(o) == 'table' then
return datetime_new(o)
@@ -645,6 +653,10 @@ local function datetime_from(o)
end
end
+--[[
+ Create datetime object representing current time using microseconds
+ platform timer and local timezone information.
+]]
local function local_now()
local d = datetime_new_raw(0, 0, 0)
builtin.datetime_now(d)
@@ -720,9 +732,14 @@ local function interval_increment(self, o, direction)
return self
end
--- Change the time-zone to the provided target_offset
--- Time `.secs`/`.nsec` are always UTC normalized, we need only to
--- reattribute object with different `.offset`
+--[[
+ Create new object with modified to target_offset time-zone.
+ If target timezone does not differ to the original one - we
+ return original object unmodified.
+
+ Time `.secs`/`.nsec` are always UTC normalized, we need only to
+ reattribute object with different `.offset`
+]]
local function datetime_to_tz(self, tgt_ofs)
if self.offset == tgt_ofs then
return self
@@ -738,6 +755,29 @@ local function datetime_to_tz(self, tgt_ofs)
return datetime_new_raw(self.secs, self.nsec, tgt_ofs)
end
+--[[
+ Provide a set of accessor to datetime attributes:
+ - .epoch or .unixtime - return timestamp as seconds represented as
double
+ typed value, and measured since Epoch;
+ - .ts or .timestamp - the same timestamp, but with extended
precision and
+ fraction part;
+
+ All accessors below convert time distance from Epoch date to different
+ units:
+ - .ns or .nanoseconds - seconds and fraction converted to nanoseconds;
+ - .us or .microseconds - seconds and fraction converted to
microseconds;
+ - .ms or .milliseconds - seconds and fraction converted to
milliseconds;
+ - .s or .seconds - seconds with extended precision and fraction part;
+ - .m or .min or minutes - minutes with extended precision;
+ - .hr or .hours - hours with extended precision;
+ - .d or .days - days with extended precision;
+
+ .add or .sub methods provide friendlier way for interval arithmetics;
+
+ .to_utc and .to_tz allows to create equivalent datetime object but in
+ particular timezone or as UTC. Time moment remains the same, only
time-zone
+ information changed, thus textual representation will be impacted also.
+]]
local function datetime_index(self, key)
if key == 'epoch' or key == 'unixtime' then
return self.secs
@@ -778,6 +818,13 @@ local function datetime_index(self, key)
end
end
+--[[
+ Allow to change datetime attributes via:
+ - .epoch or .unixtime - change datetime via provided interval to
Unix Epoch;
+ - .ts or .timestamp - change both seconds and nanoseconds fields
via given
+ timestamp with extended precision. Timestamp fraction part changes
+ nanoseconds information.
+]]
local function datetime_newindex(self, key, value)
if key == 'epoch' or key == 'unixtime' then
self.secs = value
@@ -794,17 +841,18 @@ end
-- sizeof("Wed Jun 30 21:49:08 1993\n")
local buf_len = 26
+local asctime_buffer = ffi.new('char[?]', buf_len)
local function asctime(o)
check_date(o, "datetime:asctime()")
- local buf = ffi.new('char[?]', buf_len)
- return ffi.string(builtin.datetime_asctime(o, buf))
+ return ffi.string(builtin.datetime_asctime(o, asctime_buffer))
end
+local ctime_buffer = ffi.new('char[?]', buf_len)
+
local function ctime(o)
check_date(o, "datetime:ctime()")
- local buf = ffi.new('char[?]', buf_len)
- return ffi.string(builtin.datetime_ctime(o, buf))
+ return ffi.string(builtin.datetime_ctime(o, ctime_buffer))
end
local function strftime(fmt, o)
-----------------------------------------------------------
On 18.08.2021 13:03, Safin Timur wrote:
> On 17.08.2021 19:52, Vladimir Davydov via Tarantool-patches wrote:
>> On Mon, Aug 16, 2021 at 02:59:36AM +0300, Timur Safin via
>> Tarantool-patches wrote:
>>> diff --git a/extra/exports b/extra/exports
>>> index 9eaba1282..80eb92abd 100644
>>> --- a/extra/exports
>>> +++ b/extra/exports
>>> @@ -148,8 +148,34 @@ csv_feed
>>> csv_iterator_create
>>> csv_next
>>> csv_setopt
>>> +datetime_asctime
>>> +datetime_ctime
>>> +datetime_now
>>> +datetime_strftime
>>> +decimal_unpack
>>
>> decimal_unpack?
>
> Yes, bad copy paste. Supposed to become datetime_unpack. Has been
> changed later in the patchset. But now, after Serge complain, I've
> reshuffled and fixed original patch, not later.
>
>>
>>> decimal_from_string
>>> decimal_unpack
>>> +tnt_dt_dow
>>> +tnt_dt_from_rdn
>>> +tnt_dt_from_struct_tm
>>> +tnt_dt_from_yd
>>> +tnt_dt_from_ymd
>>> +tnt_dt_from_yqd
>>> +tnt_dt_from_ywd
>>> +tnt_dt_parse_iso_date
>>> +tnt_dt_parse_iso_time_basic
>>> +tnt_dt_parse_iso_time_extended
>>> +tnt_dt_parse_iso_time
>>> +tnt_dt_parse_iso_zone_basic
>>> +tnt_dt_parse_iso_zone_extended
>>> +tnt_dt_parse_iso_zone_lenient
>>> +tnt_dt_parse_iso_zone
>>> +tnt_dt_rdn
>>> +tnt_dt_to_struct_tm
>>> +tnt_dt_to_yd
>>> +tnt_dt_to_ymd
>>> +tnt_dt_to_yqd
>>> +tnt_dt_to_ywd
>>> error_ref
>>> error_set_prev
>>> error_unref
>
> ...
>
>>> new file mode 100644
>>> index 000000000..c48295a6f
>>> --- /dev/null
>>> +++ b/src/lib/core/datetime.c
>>> @@ -0,0 +1,96 @@
>>> +/*
>>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>>> + *
>>> + * Redistribution and use in source and binary forms, with or
>>> + * without modification, are permitted provided that the following
>>> + * conditions are met:
>>> + *
>>> + * 1. Redistributions of source code must retain the above
>>> + * copyright notice, this list of conditions and the
>>> + * following disclaimer.
>>> + *
>>> + * 2. Redistributions in binary form must reproduce the above
>>> + * copyright notice, this list of conditions and the following
>>> + * disclaimer in the documentation and/or other materials
>>> + * provided with the distribution.
>>> + *
>>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>>> + * SUCH DAMAGE.
>>> + */
>>> +
>>> +#include <string.h>
>>> +#include <time.h>
>>> +
>>> +#include "trivia/util.h"
>>> +#include "datetime.h"
>>> +
>>> +static int
>>> +local_dt(int64_t secs)
>>> +{
>>> + return dt_from_rdn((int)(secs / SECS_PER_DAY) +
>> DT_EPOCH_1970_OFFSET);
>>> +}
>>
>> I don't understand what this function does. Please add a comment.
>
> Good point. Added comment.
>
>>
>>> +
>>> +static struct tm *
>>> +datetime_to_tm(const struct datetime *date)
>>> +{
>>> + static struct tm tm;
>>> +
>>> + memset(&tm, 0, sizeof(tm));
>>> + int64_t secs = date->secs;
>>> + dt_to_struct_tm(local_dt(secs), &tm);
>>> +
>>> + int seconds_of_day = (int64_t)date->secs % SECS_PER_DAY;
>>> + tm.tm_hour = (seconds_of_day / 3600) % 24;
>>> + tm.tm_min = (seconds_of_day / 60) % 60;
>>> + tm.tm_sec = seconds_of_day % 60;
>>
>> To make the code easier for understanding, please define and use
>> constants here and everywhere else in this patch: HOURS_PER_DAY,
>> MINUTES_PER_HOUR, NSECS_PER_USEC, etc.
>>
>>> +
>>> + return &tm;
>>> +}
>>> +
>>> +void
>>> +datetime_now(struct datetime *now)
>>> +{
>>> + struct timeval tv;
>>> + gettimeofday(&tv, NULL);
>>> + now->secs = tv.tv_sec;
>>> + now->nsec = tv.tv_usec * 1000;
>>> +
>>> + time_t now_seconds;
>>> + time(&now_seconds);
>>
>> Can't you use tv.tv_sec here?
>
> Nope. :)
>
> Actually I don't want to know value returned from time() - all I need is
> to eventually call localtime[_r]() to get local timezone. If you know
> simpler way to determine local timezone without time, then please give
> me know.
>
>
>>> new file mode 100644
>>> index 000000000..1a8d7e34f
>>> --- /dev/null
>>> +++ b/src/lib/core/datetime.h
>>> @@ -0,0 +1,95 @@
>>> +#pragma once
>>> +/*
>>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>>> + *
>>> + * Redistribution and use in source and binary forms, with or
>>> + * without modification, are permitted provided that the following
>>> + * conditions are met:
>>> + *
>>> + * 1. Redistributions of source code must retain the above
>>> + * copyright notice, this list of conditions and the
>>> + * following disclaimer.
>>> + *
>>> + * 2. Redistributions in binary form must reproduce the above
>>> + * copyright notice, this list of conditions and the following
>>> + * disclaimer in the documentation and/or other materials
>>> + * provided with the distribution.
>>> + *
>>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>>> + * SUCH DAMAGE.
>>> + */
>>> +
>>> +#include <stdint.h>
>>> +#include <stdbool.h>
>>> +#include <stdio.h>
>>> +#include "c-dt/dt.h"
>>> +
>>> +#if defined(__cplusplus)
>>> +extern "C"
>>> +{
>>> +#endif /* defined(__cplusplus) */
>>> +
>>> +#ifndef SECS_PER_DAY
>>> +#define SECS_PER_DAY 86400
>>> +#define DT_EPOCH_1970_OFFSET 719163
>>
>> I don't understand what this constant stores. Please add a comment.
>
> I've documented them after Serge request this way:
>
> ------------------------------------------------------
> /**
> * We count dates since so called "Rata Die" date
> * January 1, 0001, Monday (as Day 1).
> * But datetime structure keeps seconds since
> * Unix "Epoch" date:
> * Unix, January 1, 1970, Thursday
> *
> * The difference between Epoch (1970-01-01)
> * and Rata Die (0001-01-01) is 719163 days.
> */
>
> #ifndef SECS_PER_DAY
> #define SECS_PER_DAY 86400
> #define DT_EPOCH_1970_OFFSET 719163
> #endif
>
> /**
> * c-dt library uses int as type for dt value, which
> * represents the number of days since Rata Die date.
> * This implies limits to the number of seconds we
> * could safely store in our structures and then safely
> * pass to c-dt functions.
> *
> * So supported ranges will be
> * - for seconds [-185604722870400 .. 185480451417600]
> * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
> */
> #define MAX_DT_DAY_VALUE (int64_t)INT_MAX
> #define MIN_DT_DAY_VALUE (int64_t)INT_MIN
> #define SECS_EPOCH_1970_OFFSET \
> ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> #define MAX_EPOCH_SECS_VALUE \
> (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> #define MIN_EPOCH_SECS_VALUE \
> (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> ------------------------------------------------------
>
> This is more than you probably were asking for at the moment, but bear
> with me - we will need those MAX_EPOCH_SECS_VALUE/MIN_EPOCH_SECS_VALUE
> in the messagepack serialization/deserialization code.
>
>>
>>> +#endif
>>> +
>>> +/**
>>> + * Full datetime structure representing moments
>>> + * since Unix Epoch (1970-01-01).
>>> + * Time is kept normalized to UTC, time-zone offset
>>> + * is informative only.
>>> + */
>>> +struct datetime {
>>> + /** seconds since epoch */
>>> + double secs;
>>
>> Please add a comment explaining why you use 'double' instead of
>> an integer type.
>>
>>> + /** nanoseconds if any */
>>> + int32_t nsec;
>>
>> Why 'nsec', but 'secs'? This is inconsistent. Should be 'nsec' and 'sec'
>> or 'nsecs' and 'secs'.
>
> Good point, will rname to use 'secs' and 'nsecs'. But rename will be
> massive, so no incremental patch so far.
>
>>
>>> + /** offset in minutes from UTC */
>>> + int32_t offset;
>>
>> Why do you use int32_t instead of int for these two members?
>
> Because I need signed integer larger than 16-bit, but less or equal to
> 32-bit signed value. Simple int is not exactly what I want, depending on
> platfrom it could be 4byte or 8bytes long. For example, ILP32, and LP64
> do have 4-bytes long int, but there are(were) ILP64 hardware platforms
> (like Cray) where int was as 64-bit as long or pointer.
>
> int is not as specific as int32_t in this particular case.
>
>>
>> Same comments for the datetime_interval struct.
>>
>>> +};
>>> +
>>> +/**
>>> + * Date/time interval structure
>>> + */
>>> +struct datetime_interval {
>>> + /** relative seconds delta */
>>> + double secs;
>>> + /** nanoseconds delta */
>>> + int32_t nsec;
>>> +};
>>> +
>>> +/**
>>> + * Convert datetime to string using default asctime format
>>> + * "Sun Sep 16 01:03:52 1973\n\0"
>>> + * Wrapper around reenterable asctime_r() version of POSIX function
>>> + * @param date source datetime value
>>> + * @sa datetime_ctime
>>> + */
>>> +char *
>>> +datetime_asctime(const struct datetime *date, char *buf);
>>> +
>>> +char *
>>> +datetime_ctime(const struct datetime *date, char *buf);
>>> +
>>> +size_t
>>> +datetime_strftime(const struct datetime *date, const char *fmt, char
>> *buf,
>>> + uint32_t len);
>>> +
>>> +void
>>> +datetime_now(struct datetime * now);
>>
>> Extra space after '*'.
>
> Fixed.
>
>>
>> Please add comments to all these functions, like you did for
>> datetime_asctime.
>
> Added.
>
>>
>>> +
>>> +#if defined(__cplusplus)
>>> +} /* extern "C" */
>>> +#endif /* defined(__cplusplus) */
>>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>>> new file mode 100644
>>> index 000000000..ce579828f
>>> --- /dev/null
>>> +++ b/src/lua/datetime.lua
>>> @@ -0,0 +1,500 @@
>>> +local ffi = require('ffi')
>>> +
>>> +ffi.cdef [[
>>> +
>>> + /*
>>> + `c-dt` library functions handles properly both positive and
>> negative `dt`
>>> + values, where `dt` is a number of dates since Rata Die date
>> (0001-01-01).
>>> +
>>> + For better compactness of our typical data in MessagePack stream we
>> shift
>>> + root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
>>> + actually dt = 719163.
>>> +
>>> + So here is a simple formula how convert our epoch-based seconds to
>> dt values
>>> + dt = (secs / 86400) + 719163
>>> + Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
>>> + (0001-01-01) in dates.
>>> +
>>> + */
>>> + typedef int dt_t;
>>> +
>>> + // dt_core.h
>>> + dt_t tnt_dt_from_rdn (int n);
>>> + dt_t tnt_dt_from_ymd (int y, int m, int d);
>>> +
>>> + int tnt_dt_rdn (dt_t dt);
>>> +
>>> + // dt_parse_iso.h
>>> + size_t tnt_dt_parse_iso_date (const char *str, size_t len,
>> dt_t *dt);
>>> + size_t tnt_dt_parse_iso_time (const char *str, size_t len,
>> int *sod, int *nsec);
>>> + size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len,
>> int *offset);
>>
>> The line is too long. Please ensure that all lines are <= 80 characters
>> long.
>
> Reformatted.
>
>>
>>> +
>>> + // datetime.c
>>> + int
>>> + datetime_to_string(const struct datetime * date, char *buf,
>> uint32_t len);
>>> +
>>> + char *
>>> + datetime_asctime(const struct datetime *date, char *buf);
>>> +
>>> + char *
>>> + datetime_ctime(const struct datetime *date, char *buf);
>>> +
>>> + size_t
>>> + datetime_strftime(const struct datetime *date, const char *fmt,
>> char *buf,
>>> + uint32_t len);
>>> +
>>> + void
>>> + datetime_now(struct datetime * now);
>>
>> Extra space after '*'.
>
> Removed
>
>>
>>> +
>>> +]]
>>> +
>>> +local builtin = ffi.C
>>> +local math_modf = math.modf
>>> +
>>> +local SECS_PER_DAY = 86400
>>> +
>>> +-- c-dt/dt_config.h
>>> +
>>> +-- Unix, January 1, 1970, Thursday
>>> +local DT_EPOCH_1970_OFFSET = 719163
>>> +
>>> +
>>> +local datetime_t = ffi.typeof('struct datetime')
>>> +local interval_t = ffi.typeof('struct datetime_interval')
>>> +
>>> +local function is_interval(o)
>>> + return type(o) == 'cdata' and ffi.istype(interval_t, o)
>>
>> The check for 'cdata' is redundant. ffi.istype alone should be enough
>> AFAIK.
>
> Wanted to object, but have verified and indeed, in that particular order
> (first ffi metatype reference, then value) it's accepting every type.
> Apparently I'd got wrong impression when I passed arguments in the wrong
> order. Updated.
>
>
>>
>>> +end
>>> +
>>> +local function is_datetime(o)
>>> + return type(o) == 'cdata' and ffi.istype(datetime_t, o)
>>> +end
>>> +
>>> +local function is_date_interval(o)
>>> + return type(o) == 'cdata' and
>>> + (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
>>> +end
>>> +
>>> +local function interval_new()
>>> + local interval = ffi.new(interval_t)
>>> + return interval
>>> +end
>>> +
>>> +local function check_date(o, message)
>>> + if not is_datetime(o) then
>>> + return error(("%s: expected datetime, but received %s"):
>>> + format(message, o), 2)
>>> + end
>>> +end
>>> +
>>> +local function check_str(s, message)
>>> + if not type(s) == 'string' then
>>> + return error(("%s: expected string, but received %s"):
>>> + format(message, s), 2)
>>> + end
>>> +end
>>> +
>>> +local function check_range(v, range, txt)
>>> + assert(#range == 2)
>>> + if v < range[1] or v > range[2] then
>>> + error(('value %d of %s is out of allowed range [%d, %d]'):
>>> + format(v, txt, range[1], range[2]), 4)
>>> + end
>>> +end
>>> +
>>> +local function datetime_cmp(lhs, rhs)
>>> + if not is_date_interval(lhs) or
>>> + not is_date_interval(rhs) then
>>> + return nil
>>
>> Bad indentation.
>
> Fixed.
>
>>
>>> + end
>>> + local sdiff = lhs.secs - rhs.secs
>>> + return sdiff ~= 0 and sdiff or (lhs.nsec - rhs.nsec)
>>> +end
>>> +
>>> +local function datetime_eq(lhs, rhs)
>>> + local rc = datetime_cmp(lhs, rhs)
>>> + return rc ~= nil and rc == 0
>>> +end
>>> +
>>> +local function datetime_lt(lhs, rhs)
>>> + local rc = datetime_cmp(lhs, rhs)
>>> + return rc == nil and error('incompatible types for comparison', 2)
>> or
>>> + rc < 0
>>> +end
>>> +
>>> +local function datetime_le(lhs, rhs)
>>> + local rc = datetime_cmp(lhs, rhs)
>>> + return rc == nil and error('incompatible types for comparison', 2)
>> or
>>> + rc <= 0
>>> +end
>>> +
>>> +local function datetime_serialize(self)
>>> + return { secs = self.secs, nsec = self.nsec, offset = self.offset }
>>> +end
>>> +
>>> +local function interval_serialize(self)
>>> + return { secs = self.secs, nsec = self.nsec }
>>> +end
>>> +
>>> +local function datetime_new_raw(secs, nsec, offset)
>>> + local dt_obj = ffi.new(datetime_t)
>>> + dt_obj.secs = secs
>>> + dt_obj.nsec = nsec
>>> + dt_obj.offset = offset
>>> + return dt_obj
>>> +end
>>> +
>>> +local function datetime_new_dt(dt, secs, frac, offset)
>>
>> What's 'frac'? Please either add a comment or give it a better name.
>
> fraction part. Renamed to full fraction.
>
>>
>>> + local epochV = dt ~= nil and (builtin.tnt_dt_rdn(dt) -
>> DT_EPOCH_1970_OFFSET) *
>>> + SECS_PER_DAY or 0
>>
>> AFAIK we don't use camel-case for naming local variables in Lua code.
>>
>>> + local secsV = secs ~= nil and secs or 0
>>
>> This is equivalent to 'secs = secs or 0'
>
> Indeed. Fixed.
>
>>
>>> + local fracV = frac ~= nil and frac or 0
>>> + local ofsV = offset ~= nil and offset or 0
>>> + return datetime_new_raw(epochV + secsV - ofsV * 60, fracV, ofsV)
>>> +end
>>> +
>>> +-- create datetime given attribute values from obj
>>> +-- in the "easy mode", providing builder with
>>> +-- .secs, .nsec, .offset
>>> +local function datetime_new_obj(obj, ...)
>>> + if obj == nil or type(obj) ~= 'table' then
>>
>> type(obj) ~= 'table' implies 'obj' == nil
>
> I meant that obj may be simple number, not builder object.
>
>>
>>> + return datetime_new_raw(obj, ...)
>>> + end
>>> + local secs = 0
>>> + local nsec = 0
>>> + local offset = 0
>>> +
>>> + for key, value in pairs(obj) do
>>> + if key == 'secs' then
>>> + secs = value
>>> + elseif key == 'nsec' then
>>> + nsec = value
>>> + elseif key == 'offset' then
>>> + offset = value
>>> + else
>>> + error(('unknown attribute %s'):format(key), 2)
>>> + end
>>
>> Hmm, do we really need this check? IMO this is less clear and probably
>> slower than accessing the table directly:
>>
>> local secs = obj.secs
>> local nsecs = obj.nsecs
>> ...
>
> Ok, let me put you to the context, where we have been iteration before.
> Here we see smaller part of larger code which used to be emulating named
> arguments for creating datetime using attributes from passed. It's a
> standard idiom for emulating named arguments in a languages where there
> is no such syntax, but there is json-like table/object/hash facilities.
>
> https://www.lua.org/pil/5.3.html
>
> The idea is to allow constructions like below (with any set of
> attributes passed)
>
> date = require 'datetime'
> T = date.new{ year = 20201, month = 11, day = 15,
> hour = 1, minute = 10, second = 01, tz = 180}
>
> The problem was that there is set of attributes, which we want to use
> for initialization via this same named attributes idiom, but directly
> using .secs, .nsec, .offset attributes when we know what we are doing.
>
> T = date.new_raw{ secs = 0, nsec = 0, offset = 180}
>
> That's why this silly separate version has been created which strangely
> parsing set of attributes accessible in the cdata object. (If there is
> passed cdata object, but it's not)
>
> [Worth to note, that at the moment in addition to initialization object
> `date.new_raw` now handles gracefully this case of calling it without
> creating initialization object, but passing arguments:
>
> T = date.new_raw(0, 0, 180)
>
> This is why there is check for object in obj `type(obj) ~= 'table'`
> ]
>
>
> Now you have asked that I realized that ffi.new could do the
> initialization magic for us, and instead of this for-loop we could
> simply pass an object to the 2nd argument of ffi.new.
>
> tarantool> o = { secs = 0, nsec = 0, offset = 180}
> ---
> ...
>
> tarantool> ffi = require 'ffi'
> ---
> ...
>
> tarantool> T = ffi.new('struct datetime', o)
> ---
> ...
>
> tarantool> T
> ---
> - 1970-01-01T03:00+03:00
> ...
>
> The problem is - it will be silently ignoring bogus attributes we may
> pass to this initialization object, which do not map directly to the set
> of known fields in a structure initialized.
>
> Which is not a problem for the code above, where we verify set of
> attributes in an object, and bail out if there is any unknown attribute.
>
> Performance-wise I do not expect such fancy initialization of objects
> would be on a critical path. The expected scenario for average scenario
> - creating datetime objects using textual strings (from logs). And this
> is very fast, due to c-dt parsing speed.
>
>
> So here is dilema:
> - we could directly pass initialization object to the ffi.new. But allow
> all kinds of unknown attributes;
> - or we could check all attributes in a slightly slower loop, but
> provide nice user visible diagnostics for an error.
>
> What would you recommend? (I prefer implementation with checks and
> human-understandable errors)
>
>
>>
>> I never saw we do anything like this in Lua code.
>>
>> Same comment for datetime_new and other places where you use pairs()
>> like this.
>>
>
> It's still same approach to use initialization object for "named
> arguments" idiom.
>
>>> + end
>>> +
>>> + return datetime_new_raw(secs, nsec, offset)
>>> +end
>>> +
>>> +-- create datetime given attribute values from obj
>>
>> Bad comment - what's 'obj'?
>
> Yes, probably the confusion created because I was too terse here, and
> not explained that obj is initialization object for "named arguments"
> approach.
>
>>
>>> +local function datetime_new(obj)
>>> + if obj == nil or type(obj) ~= 'table' then
>>> + return datetime_new_raw(0, 0, 0)
>>> + end
>>> + local y = 0
>>> + local M = 0
>>> + local d = 0
>>> + local ymd = false
>>> +
>>> + local h = 0
>>> + local m = 0
>>> + local s = 0
>>> + local frac = 0
>>> + local hms = false
>>> + local offset = 0
>>> +
>>> + local dt = 0
>>> +
>>> + for key, value in pairs(obj) do
>>> + if key == 'year' then
>>> + check_range(value, {1, 9999}, key)
>>> + y = value
>>> + ymd = true
>>> + elseif key == 'month' then
>>> + check_range(value, {1, 12}, key)
>>> + M = value
>>> + ymd = true
>>> + elseif key == 'day' then
>>> + check_range(value, {1, 31}, key)
>>> + d = value
>>> + ymd = true
>>> + elseif key == 'hour' then
>>> + check_range(value, {0, 23}, key)
>>> + h = value
>>> + hms = true
>>> + elseif key == 'min' or key == 'minute' then
>>> + check_range(value, {0, 59}, key)
>>> + m = value
>>> + hms = true
>>> + elseif key == 'sec' or key == 'second' then
>>
>> I don't think we should support both 'sec'/'min' and 'second'/'minute'
>> here. Since you want to be consistent with os.date() output, I would
>> leave 'sec'/'min' only.
>
> Originally, there were only human-readable names like 'second' or
> 'minute', os.date compatibility added only lately. Because, it costs
> nothing (comparing to all other overhead).
>
> I'd prefer to have ability to use full readable names, not only those
> from set of os.date. It costs nothing, but make it's friendlier.
>
>>
>>> + check_range(value, {0, 60}, key)
>>> + s, frac = math_modf(value)
>>> + frac = frac * 1e9 -- convert fraction to nanoseconds
>>
>> So 'frac' actually stores nanoseconds. Please rename accordingly.
>
> Yes, this is fractional part of seconds, but represented as integers in
> nanoseconds units. Renamed.
>
>>> + hms = true
>>> + elseif key == 'tz' then
>>> + -- tz offset in minutes
>>
>> Bad indentation.
>
> Fixed.
>
>>
>>> + check_range(value, {0, 720}, key)
>>> + offset = value
>>> + elseif key == 'isdst' or key == 'wday' or key =='yday' then --
>> luacheck: ignore 542
>>
>> Missing space between '==' and 'yday'.
>
> Fixed.
>
>>
>>> + -- ignore unused os.date attributes
>>> + else
>>> + error(('unknown attribute %s'):format(key), 2)
>>> + end
>>> + end
>>> +
>>> + -- .year, .month, .day
>>> + if ymd then
>>> + dt = builtin.tnt_dt_from_ymd(y, M, d)
>>> + end
>>> +
>>> + -- .hour, .minute, .second
>>> + local secs = 0
>>> + if hms then
>>> + secs = h * 3600 + m * 60 + s
>>> + end
>>> +
>>> + return datetime_new_dt(dt, secs, frac, offset)
>>> +end
>>> +
>>> +--[[
>>> + Basic Extended
>>> + 20121224 2012-12-24 Calendar date (ISO 8601)
>>> + 2012359 2012-359 Ordinal date (ISO 8601)
>>> + 2012W521 2012-W52-1 Week date (ISO 8601)
>>> + 2012Q485 2012-Q4-85 Quarter date
>>> +]]
>>
>> Please mention in the comment what this function returns.
>> Same for other 'parse' functions.
>
> Good point. Clarified.
>
>>
>>> +
>>> +local function parse_date(str)
>>> + check_str("datetime.parse_date()")
>>
>> check_str(str, ...)
>>
>> Here and everywhere else.
>>
>>> + local dt = ffi.new('dt_t[1]')
>>> + local len = builtin.tnt_dt_parse_iso_date(str, #str, dt)
>>> + return len > 0 and datetime_new_dt(dt[0]) or nil, tonumber(len)
>>
>> Is this tonumber() really necessary?
>
> Yup, size_t is boxed in cdata.
> ```
> tarantool> ffi = require 'ffi'
> ---
> ...
>
> tarantool> ffi.cdef [[ size_t tnt_dt_parse_iso_date (const char *str,
> size_t len, dt_t *dt); ]]
> ---
> ...
>
> tarantool> ffi.cdef [[ typedef int dt_t; ]]
> ---
> ...
>
>
> tarantool> dt = ffi.new 'dt_t[1]'
> ---
> ...
>
> tarantool> r = ffi.C.tnt_dt_parse_iso_date('1970-01-01', 10, dt)
> ---
> ...
>
> tarantool> dt[0]
> ---
> - 719163
> ...
>
> tarantool> type(r)
> ---
> - cdata
> ...
>
> tarantool> ffi.typeof(r)
> ---
> - ctype<uint64_t>
> ...
> ```
>
>>
>>> +end
>>> +
>>> +--[[
>>> + Basic Extended
>>> + T12 N/A
>>> + T1230 T12:30
>>> + T123045 T12:30:45
>>> + T123045.123456789 T12:30:45.123456789
>>> + T123045,123456789 T12:30:45,123456789
>>> +
>>> + The time designator [T] may be omitted.
>>> +]]
>>> +local function parse_time(str)
>>> + check_str("datetime.parse_time()")
>>> + local sp = ffi.new('int[1]')
>>> + local fp = ffi.new('int[1]')
>>> + local len = builtin.tnt_dt_parse_iso_time(str, #str, sp, fp)
>>> + return len > 0 and datetime_new_dt(nil, sp[0], fp[0]) or nil,
>>> + tonumber(len)
>>> +end
>>> +
>>> +--[[
>>> + Basic Extended
>>> + Z N/A
>>> + +hh N/A
>>> + -hh N/A
>>> + +hhmm +hh:mm
>>> + -hhmm -hh:mm
>>> +]]
>>> +local function parse_zone(str)
>>> + check_str("datetime.parse_zone()")
>>> + local offset = ffi.new('int[1]')
>>> + local len = builtin.tnt_dt_parse_iso_zone_lenient(str, #str,
>> offset)
>>> + return len > 0 and datetime_new_dt(nil, nil, nil, offset[0]) or
>> nil,
>>> + tonumber(len)
>>> +end
>>> +
>>
>> Extra new line.
>
> Removed
>
>>
>>> +
>>> +--[[
>>> + aggregated parse functions
>>> + assumes to deal with date T time time_zone
>>> + at once
>>> +
>>> + date [T] time [ ] time_zone
>>> +]]
>>> +local function parse(str)
>>> + check_str("datetime.parse()")
>>> + local dt = ffi.new('dt_t[1]')
>>> + local len = #str
>>> + local n = builtin.tnt_dt_parse_iso_date(str, len, dt)
>>> + local dt_ = dt[0]
>>> + if n == 0 or len == n then
>>> + return datetime_new_dt(dt_)
>>> + end
>>> +
>>> + str = str:sub(tonumber(n) + 1)
>>> +
>>> + local ch = str:sub(1,1)
>>
>> Missing space after ','.
>
> Fixed
>
>>
>>> + if ch:match('[Tt ]') == nil then
>>> + return datetime_new_dt(dt_)
>>> + end
>>> +
>>> + str = str:sub(2)
>>> + len = #str
>>> +
>>> + local sp = ffi.new('int[1]')
>>> + local fp = ffi.new('int[1]')
>>> + local n = builtin.tnt_dt_parse_iso_time(str, len, sp, fp)
>>> + if n == 0 then
>>> + return datetime_new_dt(dt_)
>>> + end
>>> + local sp_ = sp[0]
>>> + local fp_ = fp[0]
>>> + if len == n then
>>> + return datetime_new_dt(dt_, sp_, fp_)
>>> + end
>>> +
>>> + str = str:sub(tonumber(n) + 1)
>>> +
>>> + if str:sub(1,1) == ' ' then
>>
>> Missing space after ','.
>
> Fixed
>
>>
>>> + str = str:sub(2)
>>> + end
>>> +
>>> + len = #str
>>> +
>>> + local offset = ffi.new('int[1]')
>>> + n = builtin.tnt_dt_parse_iso_zone_lenient(str, len, offset)
>>> + if n == 0 then
>>> + return datetime_new_dt(dt_, sp_, fp_)
>>> + end
>>> + return datetime_new_dt(dt_, sp_, fp_, offset[0])
>>> +end
>>> +
>>> +local function datetime_from(o)
>>
>> Please add a comment explaining what this function does.
>
> Added.
>
>>
>>> + if o == nil or type(o) == 'table' then
>>> + return datetime_new(o)
>>> + elseif type(o) == 'string' then
>>> + return parse(o)
>>> + end
>>> +end
>>> +
>>> +local function local_now()
>>
>> Please add a comment explaining what this function does.
>
> Added.
>
>>
>>> + local d = datetime_new_raw(0, 0, 0)
>>> + builtin.datetime_now(d)
>>> + return d
>>> +end
>>> +
>>> +-- Change the time-zone to the provided target_offset
>>> +-- Time `.secs`/`.nsec` are always UTC normalized, we need only to
>>> +-- reattribute object with different `.offset`
>>
>> It doesn't change the time-zone of the given object. It creates a new
>> object with the new timezone. Please fix the comment to avoid confusion.
>
> Indeed. Implementation has been changed to make original object immune
> to the timezone changes, but comment has not been updated. Updated now.
>
>>
>>> +local function datetime_to_tz(self, tgt_ofs)
>>> + if self.offset == tgt_ofs then
>>> + return self
>>> + end
>>> + if type(tgt_ofs) == 'string' then
>>> + local obj = parse_zone(tgt_ofs)
>>> + if obj == nil then
>>> + error(('%s: invalid time-zone format %s'):format(self,
>> tgt_ofs), 2)
>>> + else
>>> + tgt_ofs = obj.offset
>>
>> target_offset. Please don't use confusing abbreviations.
>>
>>> + end
>>> + end
>>> + return datetime_new_raw(self.secs, self.nsec, tgt_ofs)
>>> +end
>>> +
>>> +local function datetime_index(self, key)
>>> + if key == 'epoch' or key == 'unixtime' then
>>> + return self.secs
>>> + elseif key == 'ts' or key == 'timestamp' then
>>> + return self.secs + self.nsec / 1e9
>>> + elseif key == 'ns' or key == 'nanoseconds' then
>>> + return self.secs * 1e9 + self.nsec
>>> + elseif key == 'us' or key == 'microseconds' then
>>> + return self.secs * 1e6 + self.nsec / 1e3
>>> + elseif key == 'ms' or key == 'milliseconds' then
>>> + return self.secs * 1e3 + self.nsec / 1e6
>>> + elseif key == 's' or key == 'seconds' then
>>> + return self.secs + self.nsec / 1e9
>>> + elseif key == 'm' or key == 'min' or key == 'minutes' then
>>> + return (self.secs + self.nsec / 1e9) / 60
>>> + elseif key == 'hr' or key == 'hours' then
>>> + return (self.secs + self.nsec / 1e9) / (60 * 60)
>>> + elseif key == 'd' or key == 'days' then
>>> + return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
>>
>> 1. There's so many ways to get the same information, for example m,
>> min, minutes. There should be exactly one way.
>
> Disagreed. I prefer friendlier approach.
>
>>
>> 2. I'd expect datetime.hour return the number of hours in the current
>> day, like os.date(), but it returns the number of hours since the
>> Unix epoch instead. This is confusing and useless.
>
> Well, at least this is consistent with seconds and their fractions of
> different units. If there would be demand for os.date compatible table
> format, we might extend interface with corresponding accessor. But I do
> not foresee it, because os.date is totally broken if we need correct
> handling of timezone information. AFAIK it's always using local time
> context.
>>
>> 3. Please document the behavior somewhere in the comments to the code.
>
> Documented.
>
>>
>> 4. These if-else's are inefficient AFAIU.
>
> Quite the contrary (if we compare to the idiomatic closure handlers
> case). If/ifelse is the fastest way to handle such cases in Lua.
> Here is earlier bench we have discussed with Vlad and Oleg Babin, and
> after which I've changed from using hash based handlers to the series of
> ifs.
>
> https://gist.github.com/tsafin/31cc9b0872b6015904fcc90d97740770
>
>
>>
>>> + elseif key == 'to_utc' then
>>> + return function(self)
>>> + return datetime_to_tz(self, 0)
>>> + end
>>> + elseif key == 'to_tz' then
>>> + return function(self, offset)
>>> + return datetime_to_tz(self, offset)
>>> + end
>>> + else
>>> + error(('unknown attribute %s'):format(key), 2)
>>> + end
>>> +end
>>> +
>>> +local function datetime_newindex(self, key, value)
>>> + if key == 'epoch' or key == 'unixtime' then
>>> + self.secs = value
>>> + self.nsec, self.offset = 0, 0
>>> + elseif key == 'ts' or key == 'timestamp' then
>>> + local secs, frac = math_modf(value)
>>> + self.secs = secs
>>> + self.nsec = frac * 1e9
>>> + self.offset = 0
>>
>> Do we really want the datetime object to be mutable? If so, allowing to
>> set its value only to the time since the unix epoch doesn't look
>> particularly user-friendly.
>
> I do want. That was request from customer - to have ability to modify by
> timestamp from Epoch.
>
>>
>>> + else
>>> + error(('assigning to unknown attribute %s'):format(key), 2)
>>> + end
>>> +end
>>> +
>>> +-- sizeof("Wed Jun 30 21:49:08 1993\n")
>>> +local buf_len = 26
>>
>> What if year > 9999?
>
> Good point. IIRC both BSD libc and glibc have hardcoded length limit at
> around this size and not ready to handle such huge dates properly.
>
>>
>>> +
>>> +local function asctime(o)
>>> + check_date(o, "datetime:asctime()")
>>> + local buf = ffi.new('char[?]', buf_len)
>>
>> I think it would be more efficient to define the buffer in a global
>> variable - we don't yield in this code so this should be fine. Also,
>> please give the buffer an appropriate name that would say that it is
>> used for datetime formatting.
>
> Good point. Changed to using outer variables in asctime and ctime
> functions.
>
>>
>>> + return ffi.string(builtin.datetime_asctime(o, buf))
>>> +end
>>> +
>>> +local function ctime(o)
>>> + check_date(o, "datetime:ctime()")
>>> + local buf = ffi.new('char[?]', buf_len)
>>> + return ffi.string(builtin.datetime_ctime(o, buf))
>>> +end
>>> +
>>> +local function strftime(fmt, o)
>>> + check_date(o, "datetime.strftime()")
>>> + local sz = 128
>>> + local buff = ffi.new('char[?]', sz)
>>> + builtin.datetime_strftime(o, fmt, buff, sz)
>>> + return ffi.string(buff)
>>> +end
>>> +
>>> +local datetime_mt = {
>>> + __serialize = datetime_serialize,
>>> + __eq = datetime_eq,
>>> + __lt = datetime_lt,
>>> + __le = datetime_le,
>>> + __index = datetime_index,
>>> + __newindex = datetime_newindex,
>>> +}
>>> +
>>> +local interval_mt = {
>>> + __serialize = interval_serialize,
>>> + __eq = datetime_eq,
>>> + __lt = datetime_lt,
>>> + __le = datetime_le,
>>> + __index = datetime_index,
>>
>> Why does datetime_mt has __newindex while interval_mt doesn't?
>
> I did not foresee the need. Should we?
>
> And semantically datetime has persistence and life-time, but intervals
> have not. [And besides all we already provide a plenty of helpers for
> creation of intervals]
>
>>
>>> +}
>>> +
>>> +ffi.metatype(interval_t, interval_mt)
>>> +ffi.metatype(datetime_t, datetime_mt)
>>> +
>>> +return setmetatable(
>>> + {
>>> + new = datetime_new,
>>> + new_raw = datetime_new_obj,
>>
>> I'm not sure, we need to make the 'raw' function public.
>
> That's for second kind of constructors using initialization object but
> with low level attributes {secs = 0, nsec = 0, offset}. It's not for
> everybody, but still provides some ergonomics.
>
>>
>>> + interval = interval_new,
>>
>> Why would anyone want to create a 0 interval?
>
> Good point, especiall taking into account that there is no way to modify
> it, beyond directly modifying .secs and .nsec fields of cdata object.
>
> Will delete it. We already have a plenty of helpers to construct
> different kinds of intervals.
>
>>
>>> +
>>> + parse = parse,
>>> + parse_date = parse_date,
>>
>>> + parse_time = parse_time,
>>> + parse_zone = parse_zone,
>>
>> Creating a datetime object without a date sounds confusing.
>
> :) But I do foresee the need to parse parts of datetime string, and I
> did not want to establish special type for that.
>
> Now you have asked it I realized we may better create interval objects
> of approapriate time. But in this case we need to add timezone to
> interval record. So it will be able to represent timezone shifts.
>
> Ok, heer is the deal:
> - at the moment those partial parsed objects store their data to the
> partial datetime object created. That's confusing but do not require
> modifications in interval arithmetic.
> - we may start to store partial time and timezone information into
> generic intervals. But it would require extension of interval arithmetic
> to be ready to shift by timezone delta. [Does it even make any sense?]
>
> What do you think?
>
>>
>>> +
>>> + now = local_now,
>>> + strftime = strftime,
>>> + asctime = asctime,
>>> + ctime = ctime,
>>> +
>>> + is_datetime = is_datetime,
>>> + is_interval = is_interval,
>>
>> I don't see any point in making these functions public.
>
> I was asked by customer to introduce such.
>
>>
>>> + }, {
>>> + __call = function(self, ...) return datetime_from(...) end
>>> + }
>>> +)
>>> diff --git a/src/lua/init.c b/src/lua/init.c
>>> index f9738025d..127e935d7 100644
>>> --- a/src/lua/init.c
>>> +++ b/src/lua/init.c
>>> @@ -129,7 +129,8 @@ extern char strict_lua[],
>>> parse_lua[],
>>> process_lua[],
>>> humanize_lua[],
>>> - memprof_lua[]
>>> + memprof_lua[],
>>> + datetime_lua[]
>>> ;
>>> static const char *lua_modules[] = {
>>> @@ -184,6 +185,7 @@ static const char *lua_modules[] = {
>>> "memprof.process", process_lua,
>>> "memprof.humanize", humanize_lua,
>>> "memprof", memprof_lua,
>>> + "datetime", datetime_lua,
>>> NULL
>>> };
>>> diff --git a/src/lua/utils.c b/src/lua/utils.c
>>> index c71cd4857..2c89326f3 100644
>>> --- a/src/lua/utils.c
>>> +++ b/src/lua/utils.c
>>> @@ -48,6 +48,9 @@ static uint32_t CTID_STRUCT_IBUF_PTR;
>>> uint32_t CTID_CHAR_PTR;
>>> uint32_t CTID_CONST_CHAR_PTR;
>>> uint32_t CTID_UUID;
>>> +uint32_t CTID_DATETIME = 0;
>>> +uint32_t CTID_INTERVAL = 0;
>>
>> Should be called CTID_DATETIME_INTERVAL.
>>
>> Anyway, it's never used. Please remove.
>
> It's used in later patches. Wil lsquash and rename.
>
>>
>>> +
>>> void *
>>> luaL_pushcdata(struct lua_State *L, uint32_t ctypeid)
>>> @@ -120,6 +123,12 @@ luaL_pushuuidstr(struct lua_State *L, const struct
>> tt_uuid *uuid)
>>> lua_pushlstring(L, str, UUID_STR_LEN);
>>> }
>>> +struct datetime *
>>> +luaL_pushdatetime(struct lua_State *L)
>>> +{
>>> + return luaL_pushcdata(L, CTID_DATETIME);
>>> +}
>>> +
>>
>> This function isn't used in this patch. Please move it to patch 4.
>>
>
> Looks like it will be used once squashed.
>
> Thanks,
> Timur
>
> P.S.
>
> I need to jump to the different location, so the rest of responses will
> be send in a couple of hours. Please answer some questions - I 'm
> interested in your opinion.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime
2021-08-18 10:03 ` Safin Timur via Tarantool-patches
2021-08-18 10:06 ` Safin Timur via Tarantool-patches
@ 2021-08-18 11:45 ` Vladimir Davydov via Tarantool-patches
1 sibling, 0 replies; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-18 11:45 UTC (permalink / raw)
To: Safin Timur; +Cc: Timur Safin via Tarantool-patches, v.shpilevoy
On Wed, Aug 18, 2021 at 01:03:27PM +0300, Safin Timur wrote:
> > > +void
> > > +datetime_now(struct datetime *now)
> > > +{
> > > + struct timeval tv;
> > > + gettimeofday(&tv, NULL);
> > > + now->secs = tv.tv_sec;
> > > + now->nsec = tv.tv_usec * 1000;
> > > +
> > > + time_t now_seconds;
> > > + time(&now_seconds);
> >
> > Can't you use tv.tv_sec here?
>
> Nope. :)
>
> Actually I don't want to know value returned from time() - all I need is to
> eventually call localtime[_r]() to get local timezone. If you know simpler
> way to determine local timezone without time, then please give me know.
AFAIU time() returns exactly the same value as tv.tv_sec, where tv is
the value returned by gettimeofday(), no?
Come to think of it, I'm not so sure it's a good idea to call
localtime() on each call to get the timezone. In comparison to
gettimeofday() it 10x slower. Maybe, it's worth caching the current
timezone?
vlad@esperanza:~/tmp$ cat bench.lua
clock = require('clock')
datetime = require('datetime')
LOOPS = 10000000
t1 = clock.monotonic()
for i = 1, LOOPS do
datetime.now()
end
t2 = clock.monotonic()
print(string.format('datetime.now: %f', t2 - t1))
t1 = clock.monotonic()
for i = 1, LOOPS do
clock.realtime()
end
t2 = clock.monotonic()
print(string.format('clock.realtime: %f', t2 - t1))
os.exit(0)
vlad@esperanza:~/tmp$ tarantool bench.lua
datetime.now: 2.190355
clock.realtime: 0.201188
> > > +#define DT_EPOCH_1970_OFFSET 719163
> >
> > I don't understand what this constant stores. Please add a comment.
>
> I've documented them after Serge request this way:
>
> ------------------------------------------------------
> /**
> * We count dates since so called "Rata Die" date
> * January 1, 0001, Monday (as Day 1).
> * But datetime structure keeps seconds since
> * Unix "Epoch" date:
> * Unix, January 1, 1970, Thursday
> *
> * The difference between Epoch (1970-01-01)
> * and Rata Die (0001-01-01) is 719163 days.
> */
>
> #ifndef SECS_PER_DAY
> #define SECS_PER_DAY 86400
> #define DT_EPOCH_1970_OFFSET 719163
> #endif
>
> /**
> * c-dt library uses int as type for dt value, which
> * represents the number of days since Rata Die date.
> * This implies limits to the number of seconds we
> * could safely store in our structures and then safely
> * pass to c-dt functions.
> *
> * So supported ranges will be
> * - for seconds [-185604722870400 .. 185480451417600]
> * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
> */
> #define MAX_DT_DAY_VALUE (int64_t)INT_MAX
> #define MIN_DT_DAY_VALUE (int64_t)INT_MIN
> #define SECS_EPOCH_1970_OFFSET \
> ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> #define MAX_EPOCH_SECS_VALUE \
> (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> #define MIN_EPOCH_SECS_VALUE \
> (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
I'd name them DT_MAX_DAY_VALUE, etc, with DT_ prefix. For consistency
with DT_EPOCH_1970_OFFSET and so as to emphasize that these constants
are related to datetime objects. SECS_PER_DAY, MINS_PER_DAY, etc are
fine without prefix, because they are universal.
> > > +local function datetime_new_dt(dt, secs, frac, offset)
> >
> > What's 'frac'? Please either add a comment or give it a better name.
>
> fraction part. Renamed to full fraction.
It's unclear what fraction is of. I'd rename it to 'nsecs', because
the function it is passed to (datetime_new_raw) actually takes
nanoseconds.
> > > +-- create datetime given attribute values from obj
> > > +-- in the "easy mode", providing builder with
> > > +-- .secs, .nsec, .offset
> > > +local function datetime_new_obj(obj, ...)
> > > + if obj == nil or type(obj) ~= 'table' then
> >
> > type(obj) ~= 'table' implies 'obj' == nil
>
> I meant that obj may be simple number, not builder object.
I understand, but
if type(obj) ~= 'table' then
will work for obj == nil (because type(obj) is nil ~= 'table).
That is, 'obj == nil' check is redundant.
>
> >
> > > + return datetime_new_raw(obj, ...)
> > > + end
> > > + local secs = 0
> > > + local nsec = 0
> > > + local offset = 0
> > > +
> > > + for key, value in pairs(obj) do
> > > + if key == 'secs' then
> > > + secs = value
> > > + elseif key == 'nsec' then
> > > + nsec = value
> > > + elseif key == 'offset' then
> > > + offset = value
> > > + else
> > > + error(('unknown attribute %s'):format(key), 2)
> > > + end
> >
> > Hmm, do we really need this check? IMO this is less clear and probably
> > slower than accessing the table directly:
> >
> > local secs = obj.secs
> > local nsecs = obj.nsecs
> > ...
>
> So here is dilema:
> - we could directly pass initialization object to the ffi.new. But allow all
> kinds of unknown attributes;
> - or we could check all attributes in a slightly slower loop, but provide
> nice user visible diagnostics for an error.
>
> What would you recommend? (I prefer implementation with checks and
> human-understandable errors)
I understand the dilemma. I don't know what we should do. Probably ask
someone who writes a lot of Lua code. I, personally, haven't seen checks
like this in our Lua code - we never check if there's bogus positional
arguments AFAICS. E.g. if you pass {tmieout = 123} to net.box.call() in
options, there will be no error (and apparently no timeout because of
the typo). I think that this is the Lua way, but I can't say for sure.
> > > +local function datetime_new(obj)
> > > + if obj == nil or type(obj) ~= 'table' then
> > > + return datetime_new_raw(0, 0, 0)
> > > + end
> > > + local y = 0
> > > + local M = 0
> > > + local d = 0
> > > + local ymd = false
> > > +
> > > + local h = 0
> > > + local m = 0
> > > + local s = 0
> > > + local frac = 0
> > > + local hms = false
> > > + local offset = 0
> > > +
> > > + local dt = 0
> > > +
> > > + for key, value in pairs(obj) do
> > > + if key == 'year' then
> > > + check_range(value, {1, 9999}, key)
> > > + y = value
> > > + ymd = true
> > > + elseif key == 'month' then
> > > + check_range(value, {1, 12}, key)
> > > + M = value
> > > + ymd = true
> > > + elseif key == 'day' then
> > > + check_range(value, {1, 31}, key)
> > > + d = value
> > > + ymd = true
> > > + elseif key == 'hour' then
> > > + check_range(value, {0, 23}, key)
> > > + h = value
> > > + hms = true
> > > + elseif key == 'min' or key == 'minute' then
> > > + check_range(value, {0, 59}, key)
> > > + m = value
> > > + hms = true
> > > + elseif key == 'sec' or key == 'second' then
> >
> > I don't think we should support both 'sec'/'min' and 'second'/'minute'
> > here. Since you want to be consistent with os.date() output, I would
> > leave 'sec'/'min' only.
>
> Originally, there were only human-readable names like 'second' or 'minute',
> os.date compatibility added only lately. Because, it costs nothing
> (comparing to all other overhead).
>
> I'd prefer to have ability to use full readable names, not only those from
> set of os.date. It costs nothing, but make it's friendlier.
'min' and 'sec' are common abbreviations so I wouldn't say they are not
human-readable.
> > > +local function datetime_index(self, key)
> > > + if key == 'epoch' or key == 'unixtime' then
> > > + return self.secs
> > > + elseif key == 'ts' or key == 'timestamp' then
> > > + return self.secs + self.nsec / 1e9
> > > + elseif key == 'ns' or key == 'nanoseconds' then
> > > + return self.secs * 1e9 + self.nsec
> > > + elseif key == 'us' or key == 'microseconds' then
> > > + return self.secs * 1e6 + self.nsec / 1e3
> > > + elseif key == 'ms' or key == 'milliseconds' then
> > > + return self.secs * 1e3 + self.nsec / 1e6
> > > + elseif key == 's' or key == 'seconds' then
> > > + return self.secs + self.nsec / 1e9
> > > + elseif key == 'm' or key == 'min' or key == 'minutes' then
> > > + return (self.secs + self.nsec / 1e9) / 60
> > > + elseif key == 'hr' or key == 'hours' then
> > > + return (self.secs + self.nsec / 1e9) / (60 * 60)
> > > + elseif key == 'd' or key == 'days' then
> > > + return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
> >
> > 1. There's so many ways to get the same information, for example m,
> > min, minutes. There should be exactly one way.
>
> Disagreed. I prefer friendlier approach.
I wouldn't call it 'friendly'. What exactly, as a user, should I use,
'minutes', 'min', or 'm'? Freedom of choice is a pain.
>
> >
> > 2. I'd expect datetime.hour return the number of hours in the current
> > day, like os.date(), but it returns the number of hours since the
> > Unix epoch instead. This is confusing and useless.
>
> Well, at least this is consistent with seconds and their fractions of
> different units. If there would be demand for os.date compatible table
> format, we might extend interface with corresponding accessor. But I do not
> foresee it, because os.date is totally broken if we need correct handling of
> timezone information. AFAIK it's always using local time context.
It's just that I don't see any point in knowing the number of hours that
passed since 1970. Better remove this method altogether then. To be
discussed.
> >
> > 3. Please document the behavior somewhere in the comments to the code.
>
> Documented.
>
> >
> > 4. These if-else's are inefficient AFAIU.
>
> Quite the contrary (if we compare to the idiomatic closure handlers case).
> If/ifelse is the fastest way to handle such cases in Lua.
> Here is earlier bench we have discussed with Vlad and Oleg Babin, and after
> which I've changed from using hash based handlers to the series of ifs.
>
> https://gist.github.com/tsafin/31cc9b0872b6015904fcc90d97740770
The test creates a new hashtable on each function invocation - no
surprise it's slow. The hashtable should be global.
>
>
> >
> > > + elseif key == 'to_utc' then
> > > + return function(self)
> > > + return datetime_to_tz(self, 0)
> > > + end
> > > + elseif key == 'to_tz' then
> > > + return function(self, offset)
> > > + return datetime_to_tz(self, offset)
> > > + end
> > > + else
> > > + error(('unknown attribute %s'):format(key), 2)
> > > + end
> > > +end
> > > +
> > > +local function datetime_newindex(self, key, value)
> > > + if key == 'epoch' or key == 'unixtime' then
> > > + self.secs = value
> > > + self.nsec, self.offset = 0, 0
> > > + elseif key == 'ts' or key == 'timestamp' then
> > > + local secs, frac = math_modf(value)
> > > + self.secs = secs
> > > + self.nsec = frac * 1e9
> > > + self.offset = 0
> >
> > Do we really want the datetime object to be mutable? If so, allowing to
> > set its value only to the time since the unix epoch doesn't look
> > particularly user-friendly.
>
> I do want. That was request from customer - to have ability to modify by
> timestamp from Epoch.
Why can't the user create a new datetime object instead?
To be discussed.
> > > +local datetime_mt = {
> > > + __serialize = datetime_serialize,
> > > + __eq = datetime_eq,
> > > + __lt = datetime_lt,
> > > + __le = datetime_le,
> > > + __index = datetime_index,
> > > + __newindex = datetime_newindex,
> > > +}
> > > +
> > > +local interval_mt = {
> > > + __serialize = interval_serialize,
> > > + __eq = datetime_eq,
> > > + __lt = datetime_lt,
> > > + __le = datetime_le,
> > > + __index = datetime_index,
> >
> > Why does datetime_mt has __newindex while interval_mt doesn't?
>
> I did not foresee the need. Should we?
>
> And semantically datetime has persistence and life-time, but intervals have
> not. [And besides all we already provide a plenty of helpers for creation of
> intervals]
You're right - there's no point in __newindex for interval. Resolved.
>
> >
> > > +}
> > > +
> > > +ffi.metatype(interval_t, interval_mt)
> > > +ffi.metatype(datetime_t, datetime_mt)
> > > +
> > > +return setmetatable(
> > > + {
> > > + new = datetime_new,
> > > + new_raw = datetime_new_obj,
> >
> > I'm not sure, we need to make the 'raw' function public.
>
> That's for second kind of constructors using initialization object but with
> low level attributes {secs = 0, nsec = 0, offset}. It's not for everybody,
> but still provides some ergonomics.
You're probably right. Still, to be discussed.
> > > +
> > > + parse = parse,
> > > + parse_date = parse_date,
> >
> > > + parse_time = parse_time,
> > > + parse_zone = parse_zone,
> >
> > Creating a datetime object without a date sounds confusing.
>
> :) But I do foresee the need to parse parts of datetime string, and I did
> not want to establish special type for that.
>
> Now you have asked it I realized we may better create interval objects of
> approapriate time. But in this case we need to add timezone to interval
> record. So it will be able to represent timezone shifts.
>
> Ok, heer is the deal:
> - at the moment those partial parsed objects store their data to the partial
> datetime object created. That's confusing but do not require modifications
> in interval arithmetic.
> - we may start to store partial time and timezone information into generic
> intervals. But it would require extension of interval arithmetic to be ready
> to shift by timezone delta. [Does it even make any sense?]
>
> What do you think?
No, I don't like the idea of storing a timezone shift in time interval.
IMO better introduce separate 'time' and 'date' objects, which would
store the time in the datetime struct under the hood. This needs to be
discussed.
>
> >
> > > +
> > > + now = local_now,
> > > + strftime = strftime,
> > > + asctime = asctime,
> > > + ctime = ctime,
> > > +
> > > + is_datetime = is_datetime,
> > > + is_interval = is_interval,
> >
> > I don't see any point in making these functions public.
>
> I was asked by customer to introduce such.
Okay, please add tests for them.
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 1/8] build: add Christian Hansen c-dt to the build Timur Safin via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 2/8] lua: built-in module datetime Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 17:06 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime Timur Safin via Tarantool-patches
` (6 subsequent siblings)
9 siblings, 2 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
* introduced output routine for converting datetime
to their default output format.
* use this routine for tostring() in datetime.lua
Part of #5941
---
| 1 +
src/lib/core/datetime.c | 71 ++++++++++++++++++
src/lib/core/datetime.h | 9 +++
src/lua/datetime.lua | 35 +++++++++
test/app-tap/datetime.test.lua | 131 ++++++++++++++++++---------------
test/unit/CMakeLists.txt | 2 +-
test/unit/datetime.c | 61 +++++++++++----
7 files changed, 236 insertions(+), 74 deletions(-)
--git a/extra/exports b/extra/exports
index 80eb92abd..2437e175c 100644
--- a/extra/exports
+++ b/extra/exports
@@ -152,6 +152,7 @@ datetime_asctime
datetime_ctime
datetime_now
datetime_strftime
+datetime_to_string
decimal_unpack
decimal_from_string
decimal_unpack
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index c48295a6f..c24a0df82 100644
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -29,6 +29,8 @@
* SUCH DAMAGE.
*/
+#include <assert.h>
+#include <limits.h>
#include <string.h>
#include <time.h>
@@ -94,3 +96,72 @@ datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
struct tm *p_tm = datetime_to_tm(date);
return strftime(buf, len, fmt, p_tm);
}
+
+#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
+
+/* NB! buf may be NULL, and we should handle it gracefully, returning
+ * calculated length of output string
+ */
+int
+datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
+{
+#define ADVANCE(sz) \
+ if (buf != NULL) { \
+ buf += sz; \
+ len -= sz; \
+ } \
+ ret += sz;
+
+ int offset = date->offset;
+ /* for negative offsets around Epoch date we could get
+ * negative secs value, which should be attributed to
+ * 1969-12-31, not 1970-01-01, thus we first shift
+ * epoch to Rata Die then divide by seconds per day,
+ * not in reverse
+ */
+ int64_t secs = (int64_t)date->secs + offset * 60 + SECS_EPOCH_1970_OFFSET;
+ assert((secs / SECS_PER_DAY) <= INT_MAX);
+ dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
+
+ int year, month, day, sec, ns, sign;
+ dt_to_ymd(dt, &year, &month, &day);
+
+ int hour = (secs / 3600) % 24,
+ minute = (secs / 60) % 60;
+ sec = secs % 60;
+ ns = date->nsec;
+
+ int ret = 0;
+ uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
+ year, month, day, hour, minute);
+ ADVANCE(sz);
+ if (sec || ns) {
+ sz = snprintf(buf, len, ":%02d", sec);
+ ADVANCE(sz);
+ if (ns) {
+ if ((ns % 1000000) == 0)
+ sz = snprintf(buf, len, ".%03d", ns / 1000000);
+ else if ((ns % 1000) == 0)
+ sz = snprintf(buf, len, ".%06d", ns / 1000);
+ else
+ sz = snprintf(buf, len, ".%09d", ns);
+ ADVANCE(sz);
+ }
+ }
+ if (offset == 0) {
+ sz = snprintf(buf, len, "Z");
+ ADVANCE(sz);
+ }
+ else {
+ if (offset < 0)
+ sign = '-', offset = -offset;
+ else
+ sign = '+';
+
+ sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
+ ADVANCE(sz);
+ }
+ return ret;
+}
+#undef ADVANCE
+
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 1a8d7e34f..964e76fcc 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -70,6 +70,15 @@ struct datetime_interval {
int32_t nsec;
};
+/**
+ * Convert datetime to string using default format
+ * @param date source datetime value
+ * @param buf output character buffer
+ * @param len size ofoutput buffer
+ */
+int
+datetime_to_string(const struct datetime *date, char *buf, uint32_t len);
+
/**
* Convert datetime to string using default asctime format
* "Sun Sep 16 01:03:52 1973\n\0"
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index ce579828f..4d946f194 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -249,6 +249,37 @@ local function datetime_new(obj)
return datetime_new_dt(dt, secs, frac, offset)
end
+local function datetime_tostring(o)
+ if ffi.typeof(o) == datetime_t then
+ local sz = 48
+ local buff = ffi.new('char[?]', sz)
+ local len = builtin.datetime_to_string(o, buff, sz)
+ assert(len < sz)
+ return ffi.string(buff)
+ elseif ffi.typeof(o) == interval_t then
+ local ts = o.timestamp
+ local sign = '+'
+
+ if ts < 0 then
+ ts = -ts
+ sign = '-'
+ end
+
+ if ts < 60 then
+ return ('%s%s secs'):format(sign, ts)
+ elseif ts < 60 * 60 then
+ return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60)
+ elseif ts < 24 * 60 * 60 then
+ return ('%+d hours, %d minutes, %s seconds'):format(
+ o.hours, o.minutes % 60, ts % 60)
+ else
+ return ('%+d days, %d hours, %d minutes, %s seconds'):format(
+ o.days, o.hours % 24, o.minutes % 60, ts % 60)
+ end
+ end
+end
+
+
--[[
Basic Extended
20121224 2012-12-24 Calendar date (ISO 8601)
@@ -457,6 +488,7 @@ local function strftime(fmt, o)
end
local datetime_mt = {
+ __tostring = datetime_tostring,
__serialize = datetime_serialize,
__eq = datetime_eq,
__lt = datetime_lt,
@@ -466,6 +498,7 @@ local datetime_mt = {
}
local interval_mt = {
+ __tostring = datetime_tostring,
__serialize = interval_serialize,
__eq = datetime_eq,
__lt = datetime_lt,
@@ -487,6 +520,8 @@ return setmetatable(
parse_time = parse_time,
parse_zone = parse_zone,
+ tostring = datetime_tostring,
+
now = local_now,
strftime = strftime,
asctime = asctime,
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
index 464d4bd49..244ec2575 100755
--- a/test/app-tap/datetime.test.lua
+++ b/test/app-tap/datetime.test.lua
@@ -6,7 +6,7 @@ local date = require('datetime')
local ffi = require('ffi')
-test:plan(6)
+test:plan(7)
test:test("Simple tests for parser", function(test)
test:plan(2)
@@ -17,74 +17,78 @@ test:test("Simple tests for parser", function(test)
end)
test:test("Multiple tests for parser (with nanoseconds)", function(test)
- test:plan(168)
+ test:plan(193)
-- borrowed from p5-time-moments/t/180_from_string.t
local tests =
{
- { '1970-01-01T00:00:00Z', 0, 0, 0 },
- { '1970-01-01T02:00:00+02:00', 0, 0, 120 },
- { '1970-01-01T01:30:00+01:30', 0, 0, 90 },
- { '1970-01-01T01:00:00+01:00', 0, 0, 60 },
- { '1970-01-01T00:01:00+00:01', 0, 0, 1 },
- { '1970-01-01T00:00:00+00:00', 0, 0, 0 },
- { '1969-12-31T23:59:00-00:01', 0, 0, -1 },
- { '1969-12-31T23:00:00-01:00', 0, 0, -60 },
- { '1969-12-31T22:30:00-01:30', 0, 0, -90 },
- { '1969-12-31T22:00:00-02:00', 0, 0, -120 },
- { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 },
- { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 },
- { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 },
- { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 },
- { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 },
- { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 },
- { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 },
- { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 },
- { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 },
- { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 },
- { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 },
- { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 },
- { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 },
- { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 },
- { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 },
- { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 },
- { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 },
- { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 },
- { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 },
- { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 },
- { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 },
- { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 },
- { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 },
- { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 },
- { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 },
- { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 },
- { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 },
- { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 },
- { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 },
- { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 },
- { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 },
- { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 },
- { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 },
- { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 },
- { '1970-01-01T00:00:00.0Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.00Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.0000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.00000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.000000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 },
- { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 },
- { '1973-11-29T21:33:09Z', 123456789, 0, 0 },
- { '2013-10-28T17:51:56Z', 1382982716, 0, 0 },
- { '9999-12-31T23:59:59Z', 253402300799, 0, 0 },
+ {'1970-01-01T00:00Z', 0, 0, 0, 1},
+ {'1970-01-01T02:00+02:00', 0, 0, 120, 1},
+ {'1970-01-01T01:30+01:30', 0, 0, 90, 1},
+ {'1970-01-01T01:00+01:00', 0, 0, 60, 1},
+ {'1970-01-01T00:01+00:01', 0, 0, 1, 1},
+ {'1970-01-01T00:00Z', 0, 0, 0, 1},
+ {'1969-12-31T23:59-00:01', 0, 0, -1, 1},
+ {'1969-12-31T23:00-01:00', 0, 0, -60, 1},
+ {'1969-12-31T22:30-01:30', 0, 0, -90, 1},
+ {'1969-12-31T22:00-02:00', 0, 0, -120, 1},
+ {'1970-01-01T00:00:00.123456789Z', 0, 123456789, 0, 1},
+ {'1970-01-01T00:00:00.12345678Z', 0, 123456780, 0, 0},
+ {'1970-01-01T00:00:00.1234567Z', 0, 123456700, 0, 0},
+ {'1970-01-01T00:00:00.123456Z', 0, 123456000, 0, 1},
+ {'1970-01-01T00:00:00.12345Z', 0, 123450000, 0, 0},
+ {'1970-01-01T00:00:00.1234Z', 0, 123400000, 0, 0},
+ {'1970-01-01T00:00:00.123Z', 0, 123000000, 0, 1},
+ {'1970-01-01T00:00:00.12Z', 0, 120000000, 0, 0},
+ {'1970-01-01T00:00:00.1Z', 0, 100000000, 0, 0},
+ {'1970-01-01T00:00:00.01Z', 0, 10000000, 0, 0},
+ {'1970-01-01T00:00:00.001Z', 0, 1000000, 0, 1},
+ {'1970-01-01T00:00:00.0001Z', 0, 100000, 0, 0},
+ {'1970-01-01T00:00:00.00001Z', 0, 10000, 0, 0},
+ {'1970-01-01T00:00:00.000001Z', 0, 1000, 0, 1},
+ {'1970-01-01T00:00:00.0000001Z', 0, 100, 0, 0},
+ {'1970-01-01T00:00:00.00000001Z', 0, 10, 0, 0},
+ {'1970-01-01T00:00:00.000000001Z', 0, 1, 0, 1},
+ {'1970-01-01T00:00:00.000000009Z', 0, 9, 0, 1},
+ {'1970-01-01T00:00:00.00000009Z', 0, 90, 0, 0},
+ {'1970-01-01T00:00:00.0000009Z', 0, 900, 0, 0},
+ {'1970-01-01T00:00:00.000009Z', 0, 9000, 0, 1},
+ {'1970-01-01T00:00:00.00009Z', 0, 90000, 0, 0},
+ {'1970-01-01T00:00:00.0009Z', 0, 900000, 0, 0},
+ {'1970-01-01T00:00:00.009Z', 0, 9000000, 0, 1},
+ {'1970-01-01T00:00:00.09Z', 0, 90000000, 0, 0},
+ {'1970-01-01T00:00:00.9Z', 0, 900000000, 0, 0},
+ {'1970-01-01T00:00:00.99Z', 0, 990000000, 0, 0},
+ {'1970-01-01T00:00:00.999Z', 0, 999000000, 0, 1},
+ {'1970-01-01T00:00:00.9999Z', 0, 999900000, 0, 0},
+ {'1970-01-01T00:00:00.99999Z', 0, 999990000, 0, 0},
+ {'1970-01-01T00:00:00.999999Z', 0, 999999000, 0, 1},
+ {'1970-01-01T00:00:00.9999999Z', 0, 999999900, 0, 0},
+ {'1970-01-01T00:00:00.99999999Z', 0, 999999990, 0, 0},
+ {'1970-01-01T00:00:00.999999999Z', 0, 999999999, 0, 1},
+ {'1970-01-01T00:00:00.0Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.00Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.0000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.00000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.000000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.0000000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.00000000Z', 0, 0, 0, 0},
+ {'1970-01-01T00:00:00.000000000Z', 0, 0, 0, 0},
+ {'1973-11-29T21:33:09Z', 123456789, 0, 0, 1},
+ {'2013-10-28T17:51:56Z', 1382982716, 0, 0, 1},
+ {'9999-12-31T23:59:59Z', 253402300799, 0, 0, 1},
}
for _, value in ipairs(tests) do
- local str, epoch, nsec, offset
- str, epoch, nsec, offset = unpack(value)
+ local str, epoch, nsec, offset, check
+ str, epoch, nsec, offset, check = unpack(value)
local dt = date(str)
test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
+ if check > 0 then
+ test:ok(str == tostring(dt), ('%s == tostring(%s)'):
+ format(str, tostring(dt)))
+ end
end
end)
@@ -203,4 +207,11 @@ test:test("Parse tiny date into seconds and other parts", function(test)
test:ok(tiny.hours == 0.00848, "hours")
end)
+test:test("Stringization of date", function(test)
+ test:plan(1)
+ local str = '19700101Z'
+ local dt = date(str)
+ test:ok(tostring(dt) == '1970-01-01T00:00Z', ('tostring(%s)'):format(str))
+end)
+
os.exit(test:check() and 0 or 1)
diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt
index 31b183a8f..8194a133f 100644
--- a/test/unit/CMakeLists.txt
+++ b/test/unit/CMakeLists.txt
@@ -57,7 +57,7 @@ target_link_libraries(uuid.test uuid unit)
add_executable(random.test random.c core_test_utils.c)
target_link_libraries(random.test core unit)
add_executable(datetime.test datetime.c)
-target_link_libraries(datetime.test cdt unit)
+target_link_libraries(datetime.test cdt core unit)
add_executable(bps_tree.test bps_tree.cc)
target_link_libraries(bps_tree.test small misc)
add_executable(bps_tree_iterator.test bps_tree_iterator.cc)
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
index 64c19dac4..1ae76003b 100644
--- a/test/unit/datetime.c
+++ b/test/unit/datetime.c
@@ -5,6 +5,7 @@
#include <time.h>
#include "unit.h"
+#include "datetime.h"
static const char sample[] = "2012-12-24T15:30Z";
@@ -136,18 +137,6 @@ exit:
return 0;
}
-/* avoid introducing external datetime.h dependency -
- just copy paste it for today
-*/
-#define SECS_PER_DAY 86400
-#define DT_EPOCH_1970_OFFSET 719163
-
-struct datetime {
- double secs;
- int32_t nsec;
- int32_t offset;
-};
-
static int
local_rd(const struct datetime *dt)
{
@@ -211,13 +200,59 @@ static void datetime_test(void)
is(secs, parsed_secs,
"reversible seconds via strftime for '%s", buff);
}
+ check_plan();
+}
+
+
+static void
+tostring_datetime_test(void)
+{
+ static struct {
+ const char *string;
+ int64_t secs;
+ uint32_t nsec;
+ uint32_t offset;
+ } tests[] = {
+ {"1970-01-01T02:00+02:00", 0, 0, 120},
+ {"1970-01-01T01:30+01:30", 0, 0, 90},
+ {"1970-01-01T01:00+01:00", 0, 0, 60},
+ {"1970-01-01T00:01+00:01", 0, 0, 1},
+ {"1970-01-01T00:00Z", 0, 0, 0},
+ {"1969-12-31T23:59-00:01", 0, 0, -1},
+ {"1969-12-31T23:00-01:00", 0, 0, -60},
+ {"1969-12-31T22:30-01:30", 0, 0, -90},
+ {"1969-12-31T22:00-02:00", 0, 0, -120},
+ {"1970-01-01T00:00:00.123456789Z", 0, 123456789, 0},
+ {"1970-01-01T00:00:00.123456Z", 0, 123456000, 0},
+ {"1970-01-01T00:00:00.123Z", 0, 123000000, 0},
+ {"1973-11-29T21:33:09Z", 123456789, 0, 0},
+ {"2013-10-28T17:51:56Z", 1382982716, 0, 0},
+ {"9999-12-31T23:59:59Z", 253402300799, 0, 0},
+ };
+ size_t index;
+
+ plan(15);
+ for (index = 0; index < DIM(tests); index++) {
+ struct datetime date = {
+ tests[index].secs,
+ tests[index].nsec,
+ tests[index].offset
+ };
+ char buf[48];
+ datetime_to_string(&date, buf, sizeof buf);
+ is(strcmp(buf, tests[index].string), 0,
+ "string '%s' expected, received '%s'",
+ tests[index].string, buf);
+ }
+ check_plan();
}
int
main(void)
{
- plan(1);
+ plan(2);
datetime_test();
+ tostring_datetime_test();
return check_plan();
}
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime Timur Safin via Tarantool-patches
@ 2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:32 ` Safin Timur via Tarantool-patches
2021-08-17 17:06 ` Vladimir Davydov via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:15 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * introduced output routine for converting datetime
> to their default output format.
>
> * use this routine for tostring() in datetime.lua
>
> Part of #5941
> ---
> extra/exports | 1 +
> src/lib/core/datetime.c | 71 ++++++++++++++++++
> src/lib/core/datetime.h | 9 +++
> src/lua/datetime.lua | 35 +++++++++
> test/app-tap/datetime.test.lua | 131 ++++++++++++++++++---------------
> test/unit/CMakeLists.txt | 2 +-
> test/unit/datetime.c | 61 +++++++++++----
> 7 files changed, 236 insertions(+), 74 deletions(-)
>
> diff --git a/extra/exports b/extra/exports
> index 80eb92abd..2437e175c 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -152,6 +152,7 @@ datetime_asctime
> datetime_ctime
> datetime_now
> datetime_strftime
> +datetime_to_string
> decimal_unpack
> decimal_from_string
> decimal_unpack
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index c48295a6f..c24a0df82 100644
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -29,6 +29,8 @@
> * SUCH DAMAGE.
> */
>
> +#include <assert.h>
> +#include <limits.h>
> #include <string.h>
> #include <time.h>
>
> @@ -94,3 +96,72 @@ datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> struct tm *p_tm = datetime_to_tm(date);
> return strftime(buf, len, fmt, p_tm);
> }
> +
> +#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> +
> +/* NB! buf may be NULL, and we should handle it gracefully, returning
> + * calculated length of output string
> + */
> +int
> +datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
> +{
> +#define ADVANCE(sz) \
> + if (buf != NULL) { \
> + buf += sz; \
> + len -= sz; \
> + } \
> + ret += sz;
> +
> + int offset = date->offset;
> + /* for negative offsets around Epoch date we could get
> + * negative secs value, which should be attributed to
> + * 1969-12-31, not 1970-01-01, thus we first shift
> + * epoch to Rata Die then divide by seconds per day,
> + * not in reverse
> + */
> + int64_t secs = (int64_t)date->secs + offset * 60 + SECS_EPOCH_1970_OFFSET;
> + assert((secs / SECS_PER_DAY) <= INT_MAX);
> + dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
> +
> + int year, month, day, sec, ns, sign;
> + dt_to_ymd(dt, &year, &month, &day);
> +
> + int hour = (secs / 3600) % 24,
> + minute = (secs / 60) % 60;
> + sec = secs % 60;
> + ns = date->nsec;
> +
> + int ret = 0;
> + uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
> + year, month, day, hour, minute);
> + ADVANCE(sz);
Please, replace snprintf + ADVANCE() with SNPRINT macro
from src/trivia/util.h
It does exactly what you need.
> + if (sec || ns) {
> + sz = snprintf(buf, len, ":%02d", sec);
> + ADVANCE(sz);
> + if (ns) {
> + if ((ns % 1000000) == 0)
> + sz = snprintf(buf, len, ".%03d", ns / 1000000);
> + else if ((ns % 1000) == 0)
> + sz = snprintf(buf, len, ".%06d", ns / 1000);
> + else
> + sz = snprintf(buf, len, ".%09d", ns);
> + ADVANCE(sz);
> + }
> + }
> + if (offset == 0) {
> + sz = snprintf(buf, len, "Z");
> + ADVANCE(sz);
> + }
> + else {
> + if (offset < 0)
> + sign = '-', offset = -offset;
> + else
> + sign = '+';
> +
> + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
> + ADVANCE(sz);
> + }
> + return ret;
> +}
> +#undef ADVANCE
> +
<stripped>
> diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
> index 464d4bd49..244ec2575 100755
> --- a/test/app-tap/datetime.test.lua
> +++ b/test/app-tap/datetime.test.lua
> @@ -6,7 +6,7 @@ local date = require('datetime')
> local ffi = require('ffi')
>
>
> -test:plan(6)
> +test:plan(7)
>
> test:test("Simple tests for parser", function(test)
> test:plan(2)
> @@ -17,74 +17,78 @@ test:test("Simple tests for parser", function(test)
> end)
>
> test:test("Multiple tests for parser (with nanoseconds)", function(test)
> - test:plan(168)
> + test:plan(193)
> -- borrowed from p5-time-moments/t/180_from_string.t
> local tests =
> {
> - { '1970-01-01T00:00:00Z', 0, 0, 0 },
> - { '1970-01-01T02:00:00+02:00', 0, 0, 120 },
> - { '1970-01-01T01:30:00+01:30', 0, 0, 90 },
> - { '1970-01-01T01:00:00+01:00', 0, 0, 60 },
> - { '1970-01-01T00:01:00+00:01', 0, 0, 1 },
> - { '1970-01-01T00:00:00+00:00', 0, 0, 0 },
> - { '1969-12-31T23:59:00-00:01', 0, 0, -1 },
> - { '1969-12-31T23:00:00-01:00', 0, 0, -60 },
> - { '1969-12-31T22:30:00-01:30', 0, 0, -90 },
> - { '1969-12-31T22:00:00-02:00', 0, 0, -120 },
> - { '1970-01-01T00:00:00.123456789Z', 0, 123456789, 0 },
> - { '1970-01-01T00:00:00.12345678Z', 0, 123456780, 0 },
> - { '1970-01-01T00:00:00.1234567Z', 0, 123456700, 0 },
> - { '1970-01-01T00:00:00.123456Z', 0, 123456000, 0 },
> - { '1970-01-01T00:00:00.12345Z', 0, 123450000, 0 },
> - { '1970-01-01T00:00:00.1234Z', 0, 123400000, 0 },
> - { '1970-01-01T00:00:00.123Z', 0, 123000000, 0 },
> - { '1970-01-01T00:00:00.12Z', 0, 120000000, 0 },
> - { '1970-01-01T00:00:00.1Z', 0, 100000000, 0 },
> - { '1970-01-01T00:00:00.01Z', 0, 10000000, 0 },
> - { '1970-01-01T00:00:00.001Z', 0, 1000000, 0 },
> - { '1970-01-01T00:00:00.0001Z', 0, 100000, 0 },
> - { '1970-01-01T00:00:00.00001Z', 0, 10000, 0 },
> - { '1970-01-01T00:00:00.000001Z', 0, 1000, 0 },
> - { '1970-01-01T00:00:00.0000001Z', 0, 100, 0 },
> - { '1970-01-01T00:00:00.00000001Z', 0, 10, 0 },
> - { '1970-01-01T00:00:00.000000001Z', 0, 1, 0 },
> - { '1970-01-01T00:00:00.000000009Z', 0, 9, 0 },
> - { '1970-01-01T00:00:00.00000009Z', 0, 90, 0 },
> - { '1970-01-01T00:00:00.0000009Z', 0, 900, 0 },
> - { '1970-01-01T00:00:00.000009Z', 0, 9000, 0 },
> - { '1970-01-01T00:00:00.00009Z', 0, 90000, 0 },
> - { '1970-01-01T00:00:00.0009Z', 0, 900000, 0 },
> - { '1970-01-01T00:00:00.009Z', 0, 9000000, 0 },
> - { '1970-01-01T00:00:00.09Z', 0, 90000000, 0 },
> - { '1970-01-01T00:00:00.9Z', 0, 900000000, 0 },
> - { '1970-01-01T00:00:00.99Z', 0, 990000000, 0 },
> - { '1970-01-01T00:00:00.999Z', 0, 999000000, 0 },
> - { '1970-01-01T00:00:00.9999Z', 0, 999900000, 0 },
> - { '1970-01-01T00:00:00.99999Z', 0, 999990000, 0 },
> - { '1970-01-01T00:00:00.999999Z', 0, 999999000, 0 },
> - { '1970-01-01T00:00:00.9999999Z', 0, 999999900, 0 },
> - { '1970-01-01T00:00:00.99999999Z', 0, 999999990, 0 },
> - { '1970-01-01T00:00:00.999999999Z', 0, 999999999, 0 },
> - { '1970-01-01T00:00:00.0Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.00Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.0000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.00000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.000000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.0000000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.00000000Z', 0, 0, 0 },
> - { '1970-01-01T00:00:00.000000000Z', 0, 0, 0 },
> - { '1973-11-29T21:33:09Z', 123456789, 0, 0 },
> - { '2013-10-28T17:51:56Z', 1382982716, 0, 0 },
> - { '9999-12-31T23:59:59Z', 253402300799, 0, 0 },
> + {'1970-01-01T00:00Z', 0, 0, 0, 1},
> + {'1970-01-01T02:00+02:00', 0, 0, 120, 1},
> + {'1970-01-01T01:30+01:30', 0, 0, 90, 1},
> + {'1970-01-01T01:00+01:00', 0, 0, 60, 1},
> + {'1970-01-01T00:01+00:01', 0, 0, 1, 1},
> + {'1970-01-01T00:00Z', 0, 0, 0, 1},
> + {'1969-12-31T23:59-00:01', 0, 0, -1, 1},
> + {'1969-12-31T23:00-01:00', 0, 0, -60, 1},
> + {'1969-12-31T22:30-01:30', 0, 0, -90, 1},
> + {'1969-12-31T22:00-02:00', 0, 0, -120, 1},
> + {'1970-01-01T00:00:00.123456789Z', 0, 123456789, 0, 1},
> + {'1970-01-01T00:00:00.12345678Z', 0, 123456780, 0, 0},
> + {'1970-01-01T00:00:00.1234567Z', 0, 123456700, 0, 0},
> + {'1970-01-01T00:00:00.123456Z', 0, 123456000, 0, 1},
> + {'1970-01-01T00:00:00.12345Z', 0, 123450000, 0, 0},
> + {'1970-01-01T00:00:00.1234Z', 0, 123400000, 0, 0},
> + {'1970-01-01T00:00:00.123Z', 0, 123000000, 0, 1},
> + {'1970-01-01T00:00:00.12Z', 0, 120000000, 0, 0},
> + {'1970-01-01T00:00:00.1Z', 0, 100000000, 0, 0},
> + {'1970-01-01T00:00:00.01Z', 0, 10000000, 0, 0},
> + {'1970-01-01T00:00:00.001Z', 0, 1000000, 0, 1},
> + {'1970-01-01T00:00:00.0001Z', 0, 100000, 0, 0},
> + {'1970-01-01T00:00:00.00001Z', 0, 10000, 0, 0},
> + {'1970-01-01T00:00:00.000001Z', 0, 1000, 0, 1},
> + {'1970-01-01T00:00:00.0000001Z', 0, 100, 0, 0},
> + {'1970-01-01T00:00:00.00000001Z', 0, 10, 0, 0},
> + {'1970-01-01T00:00:00.000000001Z', 0, 1, 0, 1},
> + {'1970-01-01T00:00:00.000000009Z', 0, 9, 0, 1},
> + {'1970-01-01T00:00:00.00000009Z', 0, 90, 0, 0},
> + {'1970-01-01T00:00:00.0000009Z', 0, 900, 0, 0},
> + {'1970-01-01T00:00:00.000009Z', 0, 9000, 0, 1},
> + {'1970-01-01T00:00:00.00009Z', 0, 90000, 0, 0},
> + {'1970-01-01T00:00:00.0009Z', 0, 900000, 0, 0},
> + {'1970-01-01T00:00:00.009Z', 0, 9000000, 0, 1},
> + {'1970-01-01T00:00:00.09Z', 0, 90000000, 0, 0},
> + {'1970-01-01T00:00:00.9Z', 0, 900000000, 0, 0},
> + {'1970-01-01T00:00:00.99Z', 0, 990000000, 0, 0},
> + {'1970-01-01T00:00:00.999Z', 0, 999000000, 0, 1},
> + {'1970-01-01T00:00:00.9999Z', 0, 999900000, 0, 0},
> + {'1970-01-01T00:00:00.99999Z', 0, 999990000, 0, 0},
> + {'1970-01-01T00:00:00.999999Z', 0, 999999000, 0, 1},
> + {'1970-01-01T00:00:00.9999999Z', 0, 999999900, 0, 0},
> + {'1970-01-01T00:00:00.99999999Z', 0, 999999990, 0, 0},
> + {'1970-01-01T00:00:00.999999999Z', 0, 999999999, 0, 1},
> + {'1970-01-01T00:00:00.0Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.00Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.0000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.00000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.000000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.0000000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.00000000Z', 0, 0, 0, 0},
> + {'1970-01-01T00:00:00.000000000Z', 0, 0, 0, 0},
> + {'1973-11-29T21:33:09Z', 123456789, 0, 0, 1},
> + {'2013-10-28T17:51:56Z', 1382982716, 0, 0, 1},
> + {'9999-12-31T23:59:59Z', 253402300799, 0, 0, 1},
Please, squash this change into the previous commit.
> }
> for _, value in ipairs(tests) do
> - local str, epoch, nsec, offset
> - str, epoch, nsec, offset = unpack(value)
> + local str, epoch, nsec, offset, check
> + str, epoch, nsec, offset, check = unpack(value)
> local dt = date(str)
> test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str, epoch))
> test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str, nsec))
> test:ok(dt.offset == offset, ('%s: dt.offset == %d'):format(str, offset))
> + if check > 0 then
> + test:ok(str == tostring(dt), ('%s == tostring(%s)'):
> + format(str, tostring(dt)))
> + end
> end
> end)
>
> @@ -203,4 +207,11 @@ test:test("Parse tiny date into seconds and other parts", function(test)
> test:ok(tiny.hours == 0.00848, "hours")
> end)
>
>
>
<stripped>
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:32 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:32 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
My responses below, thanks...
On 17.08.2021 15:15, Serge Petrenko wrote:
>
>
> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>> * introduced output routine for converting datetime
>> to their default output format.
>>
>> * use this routine for tostring() in datetime.lua
>>
>> Part of #5941
>> ---
>> extra/exports | 1 +
>> src/lib/core/datetime.c | 71 ++++++++++++++++++
>> src/lib/core/datetime.h | 9 +++
>> src/lua/datetime.lua | 35 +++++++++
>> test/app-tap/datetime.test.lua | 131 ++++++++++++++++++---------------
>> test/unit/CMakeLists.txt | 2 +-
>> test/unit/datetime.c | 61 +++++++++++----
>> 7 files changed, 236 insertions(+), 74 deletions(-)
>>
>> diff --git a/extra/exports b/extra/exports
>> index 80eb92abd..2437e175c 100644
>> --- a/extra/exports
>> +++ b/extra/exports
>> @@ -152,6 +152,7 @@ datetime_asctime
>> datetime_ctime
>> datetime_now
>> datetime_strftime
>> +datetime_to_string
>> decimal_unpack
>> decimal_from_string
>> decimal_unpack
>> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
>> index c48295a6f..c24a0df82 100644
>> --- a/src/lib/core/datetime.c
>> +++ b/src/lib/core/datetime.c
>> @@ -29,6 +29,8 @@
>> * SUCH DAMAGE.
>> */
>> +#include <assert.h>
>> +#include <limits.h>
>> #include <string.h>
>> #include <time.h>
>> @@ -94,3 +96,72 @@ datetime_strftime(const struct datetime *date,
>> const char *fmt, char *buf,
>> struct tm *p_tm = datetime_to_tm(date);
>> return strftime(buf, len, fmt, p_tm);
>> }
>> +
>> +#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET *
>> SECS_PER_DAY)
>> +
>> +/* NB! buf may be NULL, and we should handle it gracefully, returning
>> + * calculated length of output string
>> + */
>> +int
>> +datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
>> +{
>> +#define ADVANCE(sz) \
>> + if (buf != NULL) { \
>> + buf += sz; \
>> + len -= sz; \
>> + } \
>> + ret += sz;
>> +
>> + int offset = date->offset;
>> + /* for negative offsets around Epoch date we could get
>> + * negative secs value, which should be attributed to
>> + * 1969-12-31, not 1970-01-01, thus we first shift
>> + * epoch to Rata Die then divide by seconds per day,
>> + * not in reverse
>> + */
>> + int64_t secs = (int64_t)date->secs + offset * 60 +
>> SECS_EPOCH_1970_OFFSET;
>> + assert((secs / SECS_PER_DAY) <= INT_MAX);
>> + dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
>> +
>> + int year, month, day, sec, ns, sign;
>> + dt_to_ymd(dt, &year, &month, &day);
>> +
>> + int hour = (secs / 3600) % 24,
>> + minute = (secs / 60) % 60;
>> + sec = secs % 60;
>> + ns = date->nsec;
>> +
>> + int ret = 0;
>> + uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
>> + year, month, day, hour, minute);
>> + ADVANCE(sz);
>
>
> Please, replace snprintf + ADVANCE() with SNPRINT macro
> from src/trivia/util.h
>
> It does exactly what you need.
OMG! What a coincidence! Thanks!
Updated accordingly...
-----------------------------------------
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index c05197efd..bdaaff555 100644
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -78,15 +78,8 @@ datetime_strftime(const struct datetime *date, const
char *fmt, char *buf,
* calculated length of output string
*/
int
-datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
+datetime_to_string(const struct datetime *date, char *buf, int len)
{
-#define ADVANCE(sz) \
- if (buf != NULL) { \
- buf += sz; \
- len -= sz; \
- } \
- ret += sz;
-
int offset = date->offset;
/* for negative offsets around Epoch date we could get
* negative secs value, which should be attributed to
@@ -106,26 +99,24 @@ datetime_to_string(const struct datetime *date,
char *buf, uint32_t len)
sec = secs % 60;
ns = date->nsec;
- int ret = 0;
- uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
- year, month, day, hour, minute);
- ADVANCE(sz);
+ int sz = 0;
+ SNPRINT(sz, snprintf, buf, len, "%04d-%02d-%02dT%02d:%02d",
+ year, month, day, hour, minute);
if (sec || ns) {
- sz = snprintf(buf, len, ":%02d", sec);
- ADVANCE(sz);
+ SNPRINT(sz, snprintf, buf, len, ":%02d", sec);
if (ns) {
if ((ns % 1000000) == 0)
- sz = snprintf(buf, len, ".%03d", ns /
1000000);
+ SNPRINT(sz, snprintf, buf, len, ".%03d",
+ ns / 1000000);
else if ((ns % 1000) == 0)
- sz = snprintf(buf, len, ".%06d", ns / 1000);
+ SNPRINT(sz, snprintf, buf, len, ".%06d",
+ ns / 1000);
else
- sz = snprintf(buf, len, ".%09d", ns);
- ADVANCE(sz);
+ SNPRINT(sz, snprintf, buf, len, ".%09d",
ns);
}
}
if (offset == 0) {
- sz = snprintf(buf, len, "Z");
- ADVANCE(sz);
+ SNPRINT(sz, snprintf, buf, len, "Z");
}
else {
if (offset < 0)
@@ -133,10 +124,9 @@ datetime_to_string(const struct datetime *date,
char *buf, uint32_t len)
else
sign = '+';
- sz = snprintf(buf, len, "%c%02d:%02d", sign, offset /
60, offset % 60);
- ADVANCE(sz);
+ SNPRINT(sz, snprintf, buf, len, "%c%02d:%02d", sign,
+ offset / 60, offset % 60);
}
- return ret;
+ return sz;
}
-#undef ADVANCE
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 3c7a7d99d..688ab59ec 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -62,7 +62,7 @@ struct datetime_interval {
* @param len size ofoutput buffer
*/
int
-datetime_to_string(const struct datetime *date, char *buf, uint32_t len);
+datetime_to_string(const struct datetime *date, char *buf, int len);
/**
* Convert datetime to string using default asctime format
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 4d946f194..96ecd1fee 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -31,7 +31,7 @@ ffi.cdef [[
// datetime.c
int
- datetime_to_string(const struct datetime * date, char *buf,
uint32_t len);
+ datetime_to_string(const struct datetime * date, char *buf, int len);
char *
datetime_asctime(const struct datetime *date, char *buf);
-----------------------------------------
>
>
>> + if (sec || ns) {
>> + sz = snprintf(buf, len, ":%02d", sec);
>> + ADVANCE(sz);
>> + if (ns) {
>> + if ((ns % 1000000) == 0)
>> + sz = snprintf(buf, len, ".%03d", ns / 1000000);
>> + else if ((ns % 1000) == 0)
>> + sz = snprintf(buf, len, ".%06d", ns / 1000);
>> + else
>> + sz = snprintf(buf, len, ".%09d", ns);
>> + ADVANCE(sz);
>> + }
>> + }
>> + if (offset == 0) {
>> + sz = snprintf(buf, len, "Z");
>> + ADVANCE(sz);
>> + }
>> + else {
>> + if (offset < 0)
>> + sign = '-', offset = -offset;
>> + else
>> + sign = '+';
>> +
>> + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60,
>> offset % 60);
>> + ADVANCE(sz);
>> + }
>> + return ret;
>> +}
>> +#undef ADVANCE
>> +
>
>
> <stripped>
>
>
>> diff --git a/test/app-tap/datetime.test.lua
>> b/test/app-tap/datetime.test.lua
>> index 464d4bd49..244ec2575 100755
>> --- a/test/app-tap/datetime.test.lua
>> +++ b/test/app-tap/datetime.test.lua
>> @@ -6,7 +6,7 @@ local date = require('datetime')
>> local ffi = require('ffi')
>> -test:plan(6)
>> +test:plan(7)
>> test:test("Simple tests for parser", function(test)
>> test:plan(2)
>> @@ -17,74 +17,78 @@ test:test("Simple tests for parser", function(test)
>> end)
>> test:test("Multiple tests for parser (with nanoseconds)",
>> function(test)
>> - test:plan(168)
>> + test:plan(193)
>> -- borrowed from p5-time-moments/t/180_from_string.t
>> local tests =
>> {
>> - { '1970-01-01T00:00:00Z', 0,
>> 0, 0 },
>> - { '1970-01-01T02:00:00+02:00', 0,
>> 0, 120 },
>> - { '1970-01-01T01:30:00+01:30', 0,
>> 0, 90 },
>> - { '1970-01-01T01:00:00+01:00', 0,
>> 0, 60 },
>> - { '1970-01-01T00:01:00+00:01', 0,
>> 0, 1 },
>> - { '1970-01-01T00:00:00+00:00', 0,
>> 0, 0 },
>> - { '1969-12-31T23:59:00-00:01', 0,
>> 0, -1 },
>> - { '1969-12-31T23:00:00-01:00', 0,
>> 0, -60 },
>> - { '1969-12-31T22:30:00-01:30', 0,
>> 0, -90 },
>> - { '1969-12-31T22:00:00-02:00', 0,
>> 0, -120 },
>> - { '1970-01-01T00:00:00.123456789Z', 0,
>> 123456789, 0 },
>> - { '1970-01-01T00:00:00.12345678Z', 0,
>> 123456780, 0 },
>> - { '1970-01-01T00:00:00.1234567Z', 0,
>> 123456700, 0 },
>> - { '1970-01-01T00:00:00.123456Z', 0,
>> 123456000, 0 },
>> - { '1970-01-01T00:00:00.12345Z', 0,
>> 123450000, 0 },
>> - { '1970-01-01T00:00:00.1234Z', 0,
>> 123400000, 0 },
>> - { '1970-01-01T00:00:00.123Z', 0,
>> 123000000, 0 },
>> - { '1970-01-01T00:00:00.12Z', 0,
>> 120000000, 0 },
>> - { '1970-01-01T00:00:00.1Z', 0,
>> 100000000, 0 },
>> - { '1970-01-01T00:00:00.01Z', 0,
>> 10000000, 0 },
>> - { '1970-01-01T00:00:00.001Z', 0,
>> 1000000, 0 },
>> - { '1970-01-01T00:00:00.0001Z', 0,
>> 100000, 0 },
>> - { '1970-01-01T00:00:00.00001Z', 0,
>> 10000, 0 },
>> - { '1970-01-01T00:00:00.000001Z', 0,
>> 1000, 0 },
>> - { '1970-01-01T00:00:00.0000001Z', 0,
>> 100, 0 },
>> - { '1970-01-01T00:00:00.00000001Z', 0,
>> 10, 0 },
>> - { '1970-01-01T00:00:00.000000001Z', 0,
>> 1, 0 },
>> - { '1970-01-01T00:00:00.000000009Z', 0,
>> 9, 0 },
>> - { '1970-01-01T00:00:00.00000009Z', 0,
>> 90, 0 },
>> - { '1970-01-01T00:00:00.0000009Z', 0,
>> 900, 0 },
>> - { '1970-01-01T00:00:00.000009Z', 0,
>> 9000, 0 },
>> - { '1970-01-01T00:00:00.00009Z', 0,
>> 90000, 0 },
>> - { '1970-01-01T00:00:00.0009Z', 0,
>> 900000, 0 },
>> - { '1970-01-01T00:00:00.009Z', 0,
>> 9000000, 0 },
>> - { '1970-01-01T00:00:00.09Z', 0,
>> 90000000, 0 },
>> - { '1970-01-01T00:00:00.9Z', 0,
>> 900000000, 0 },
>> - { '1970-01-01T00:00:00.99Z', 0,
>> 990000000, 0 },
>> - { '1970-01-01T00:00:00.999Z', 0,
>> 999000000, 0 },
>> - { '1970-01-01T00:00:00.9999Z', 0,
>> 999900000, 0 },
>> - { '1970-01-01T00:00:00.99999Z', 0,
>> 999990000, 0 },
>> - { '1970-01-01T00:00:00.999999Z', 0,
>> 999999000, 0 },
>> - { '1970-01-01T00:00:00.9999999Z', 0,
>> 999999900, 0 },
>> - { '1970-01-01T00:00:00.99999999Z', 0,
>> 999999990, 0 },
>> - { '1970-01-01T00:00:00.999999999Z', 0,
>> 999999999, 0 },
>> - { '1970-01-01T00:00:00.0Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.00Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.0000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.00000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.000000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.0000000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.00000000Z', 0,
>> 0, 0 },
>> - { '1970-01-01T00:00:00.000000000Z', 0,
>> 0, 0 },
>> - { '1973-11-29T21:33:09Z', 123456789,
>> 0, 0 },
>> - { '2013-10-28T17:51:56Z', 1382982716,
>> 0, 0 },
>> - { '9999-12-31T23:59:59Z', 253402300799,
>> 0, 0 },
>> + {'1970-01-01T00:00Z', 0, 0, 0, 1},
>> + {'1970-01-01T02:00+02:00', 0, 0, 120, 1},
>> + {'1970-01-01T01:30+01:30', 0, 0, 90, 1},
>> + {'1970-01-01T01:00+01:00', 0, 0, 60, 1},
>> + {'1970-01-01T00:01+00:01', 0, 0, 1, 1},
>> + {'1970-01-01T00:00Z', 0, 0, 0, 1},
>> + {'1969-12-31T23:59-00:01', 0, 0, -1, 1},
>> + {'1969-12-31T23:00-01:00', 0, 0, -60, 1},
>> + {'1969-12-31T22:30-01:30', 0, 0, -90, 1},
>> + {'1969-12-31T22:00-02:00', 0, 0, -120, 1},
>> + {'1970-01-01T00:00:00.123456789Z', 0, 123456789, 0, 1},
>> + {'1970-01-01T00:00:00.12345678Z', 0, 123456780, 0, 0},
>> + {'1970-01-01T00:00:00.1234567Z', 0, 123456700, 0, 0},
>> + {'1970-01-01T00:00:00.123456Z', 0, 123456000, 0, 1},
>> + {'1970-01-01T00:00:00.12345Z', 0, 123450000, 0, 0},
>> + {'1970-01-01T00:00:00.1234Z', 0, 123400000, 0, 0},
>> + {'1970-01-01T00:00:00.123Z', 0, 123000000, 0, 1},
>> + {'1970-01-01T00:00:00.12Z', 0, 120000000, 0, 0},
>> + {'1970-01-01T00:00:00.1Z', 0, 100000000, 0, 0},
>> + {'1970-01-01T00:00:00.01Z', 0, 10000000, 0, 0},
>> + {'1970-01-01T00:00:00.001Z', 0, 1000000, 0, 1},
>> + {'1970-01-01T00:00:00.0001Z', 0, 100000, 0, 0},
>> + {'1970-01-01T00:00:00.00001Z', 0, 10000, 0, 0},
>> + {'1970-01-01T00:00:00.000001Z', 0, 1000, 0, 1},
>> + {'1970-01-01T00:00:00.0000001Z', 0, 100, 0, 0},
>> + {'1970-01-01T00:00:00.00000001Z', 0, 10, 0, 0},
>> + {'1970-01-01T00:00:00.000000001Z', 0, 1, 0, 1},
>> + {'1970-01-01T00:00:00.000000009Z', 0, 9, 0, 1},
>> + {'1970-01-01T00:00:00.00000009Z', 0, 90, 0, 0},
>> + {'1970-01-01T00:00:00.0000009Z', 0, 900, 0, 0},
>> + {'1970-01-01T00:00:00.000009Z', 0, 9000, 0, 1},
>> + {'1970-01-01T00:00:00.00009Z', 0, 90000, 0, 0},
>> + {'1970-01-01T00:00:00.0009Z', 0, 900000, 0, 0},
>> + {'1970-01-01T00:00:00.009Z', 0, 9000000, 0, 1},
>> + {'1970-01-01T00:00:00.09Z', 0, 90000000, 0, 0},
>> + {'1970-01-01T00:00:00.9Z', 0, 900000000, 0, 0},
>> + {'1970-01-01T00:00:00.99Z', 0, 990000000, 0, 0},
>> + {'1970-01-01T00:00:00.999Z', 0, 999000000, 0, 1},
>> + {'1970-01-01T00:00:00.9999Z', 0, 999900000, 0, 0},
>> + {'1970-01-01T00:00:00.99999Z', 0, 999990000, 0, 0},
>> + {'1970-01-01T00:00:00.999999Z', 0, 999999000, 0, 1},
>> + {'1970-01-01T00:00:00.9999999Z', 0, 999999900, 0, 0},
>> + {'1970-01-01T00:00:00.99999999Z', 0, 999999990, 0, 0},
>> + {'1970-01-01T00:00:00.999999999Z', 0, 999999999, 0, 1},
>> + {'1970-01-01T00:00:00.0Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.00Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.0000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.00000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.000000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.0000000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.00000000Z', 0, 0, 0, 0},
>> + {'1970-01-01T00:00:00.000000000Z', 0, 0, 0, 0},
>> + {'1973-11-29T21:33:09Z', 123456789, 0, 0, 1},
>> + {'2013-10-28T17:51:56Z', 1382982716, 0, 0, 1},
>> + {'9999-12-31T23:59:59Z', 253402300799, 0, 0, 1},
>
>
> Please, squash this change into the previous commit.
Surprisingly, that was original configuration of patches, then I've
split it to smaller chunks.
But yes, this test dependency shows that code should be in a single
patch. Will do. [Will not push updated branch yet, to accumulate changes
for Vova feedback]
>
>
>> }
>> for _, value in ipairs(tests) do
>> - local str, epoch, nsec, offset
>> - str, epoch, nsec, offset = unpack(value)
>> + local str, epoch, nsec, offset, check
>> + str, epoch, nsec, offset, check = unpack(value)
>> local dt = date(str)
>> test:ok(dt.secs == epoch, ('%s: dt.secs == %d'):format(str,
>> epoch))
>> test:ok(dt.nsec == nsec, ('%s: dt.nsec == %d'):format(str,
>> nsec))
>> test:ok(dt.offset == offset, ('%s: dt.offset ==
>> %d'):format(str, offset))
>> + if check > 0 then
>> + test:ok(str == tostring(dt), ('%s == tostring(%s)'):
>> + format(str, tostring(dt)))
>> + end
>> end
>> end)
>> @@ -203,4 +207,11 @@ test:test("Parse tiny date into seconds and other
>> parts", function(test)
>> test:ok(tiny.hours == 0.00848, "hours")
>> end)
>>
>>
>
>
> <stripped>
>
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime Timur Safin via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 17:06 ` Vladimir Davydov via Tarantool-patches
2021-08-18 14:10 ` Safin Timur via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 17:06 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:37AM +0300, Timur Safin via Tarantool-patches wrote:
> * introduced output routine for converting datetime
> to their default output format.
>
> * use this routine for tostring() in datetime.lua
>
> Part of #5941
> ---
> extra/exports | 1 +
> src/lib/core/datetime.c | 71 ++++++++++++++++++
> src/lib/core/datetime.h | 9 +++
> src/lua/datetime.lua | 35 +++++++++
> test/app-tap/datetime.test.lua | 131 ++++++++++++++++++---------------
> test/unit/CMakeLists.txt | 2 +-
> test/unit/datetime.c | 61 +++++++++++----
> 7 files changed, 236 insertions(+), 74 deletions(-)
>
> diff --git a/extra/exports b/extra/exports
> index 80eb92abd..2437e175c 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -152,6 +152,7 @@ datetime_asctime
> datetime_ctime
> datetime_now
> datetime_strftime
> +datetime_to_string
> decimal_unpack
> decimal_from_string
> decimal_unpack
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index c48295a6f..c24a0df82 100644
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -29,6 +29,8 @@
> * SUCH DAMAGE.
> */
>
> +#include <assert.h>
> +#include <limits.h>
> #include <string.h>
> #include <time.h>
>
> @@ -94,3 +96,72 @@ datetime_strftime(const struct datetime *date, const char *fmt, char *buf,
> struct tm *p_tm = datetime_to_tm(date);
> return strftime(buf, len, fmt, p_tm);
> }
> +
> +#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> +
> +/* NB! buf may be NULL, and we should handle it gracefully, returning
/**
* ...
(according to our coding style)
> + * calculated length of output string
> + */
> +int
> +datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
> +{
> +#define ADVANCE(sz) \
> + if (buf != NULL) { \
> + buf += sz; \
> + len -= sz; \
> + } \
> + ret += sz;
Please use SNPRINT helper.
> +
> + int offset = date->offset;
> + /* for negative offsets around Epoch date we could get
> + * negative secs value, which should be attributed to
> + * 1969-12-31, not 1970-01-01, thus we first shift
> + * epoch to Rata Die then divide by seconds per day,
> + * not in reverse
> + */
> + int64_t secs = (int64_t)date->secs + offset * 60 + SECS_EPOCH_1970_OFFSET;
> + assert((secs / SECS_PER_DAY) <= INT_MAX);
> + dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
> +
> + int year, month, day, sec, ns, sign;
> + dt_to_ymd(dt, &year, &month, &day);
> +
> + int hour = (secs / 3600) % 24,
> + minute = (secs / 60) % 60;
Please define on a separate line:
int hour = ...;
int minute = ...;
> + sec = secs % 60;
sec, secs, oh
Please use better names.
> + ns = date->nsec;
> +
> + int ret = 0;
> + uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
> + year, month, day, hour, minute);
> + ADVANCE(sz);
> + if (sec || ns) {
> + sz = snprintf(buf, len, ":%02d", sec);
> + ADVANCE(sz);
> + if (ns) {
> + if ((ns % 1000000) == 0)
> + sz = snprintf(buf, len, ".%03d", ns / 1000000);
> + else if ((ns % 1000) == 0)
> + sz = snprintf(buf, len, ".%06d", ns / 1000);
> + else
> + sz = snprintf(buf, len, ".%09d", ns);
> + ADVANCE(sz);
> + }
> + }
> + if (offset == 0) {
> + sz = snprintf(buf, len, "Z");
> + ADVANCE(sz);
> + }
> + else {
Bad formatting. Should be '} else {'.
> + if (offset < 0)
> + sign = '-', offset = -offset;
Please don't abuse ','.
> + else
> + sign = '+';
> +
> + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
> + ADVANCE(sz);
> + }
> + return ret;
> +}
> +#undef ADVANCE
> +
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 1a8d7e34f..964e76fcc 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -70,6 +70,15 @@ struct datetime_interval {
> int32_t nsec;
> };
>
> +/**
> + * Convert datetime to string using default format
> + * @param date source datetime value
> + * @param buf output character buffer
> + * @param len size ofoutput buffer
> + */
> +int
> +datetime_to_string(const struct datetime *date, char *buf, uint32_t len);
> +
This should be an snprint-like function: buf and len should go first.
This would allow us to use it in conjunction with SNPRINT.
> /**
> * Convert datetime to string using default asctime format
> * "Sun Sep 16 01:03:52 1973\n\0"
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index ce579828f..4d946f194 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -249,6 +249,37 @@ local function datetime_new(obj)
> return datetime_new_dt(dt, secs, frac, offset)
> end
>
> +local function datetime_tostring(o)
Please add a comment to this function with example output.
> + if ffi.typeof(o) == datetime_t then
> + local sz = 48
> + local buff = ffi.new('char[?]', sz)
> + local len = builtin.datetime_to_string(o, buff, sz)
> + assert(len < sz)
> + return ffi.string(buff)
> + elseif ffi.typeof(o) == interval_t then
Please define separate functions for interval and datetime.
> + local ts = o.timestamp
> + local sign = '+'
> +
> + if ts < 0 then
> + ts = -ts
> + sign = '-'
> + end
> +
> + if ts < 60 then
> + return ('%s%s secs'):format(sign, ts)
> + elseif ts < 60 * 60 then
> + return ('%+d minutes, %s seconds'):format(o.minutes, ts % 60)
> + elseif ts < 24 * 60 * 60 then
> + return ('%+d hours, %d minutes, %s seconds'):format(
> + o.hours, o.minutes % 60, ts % 60)
> + else
> + return ('%+d days, %d hours, %d minutes, %s seconds'):format(
> + o.days, o.hours % 24, o.minutes % 60, ts % 60)
> + end
> + end
> +end
> +
> +
> --[[
> Basic Extended
> 20121224 2012-12-24 Calendar date (ISO 8601)
> @@ -457,6 +488,7 @@ local function strftime(fmt, o)
> end
>
> local datetime_mt = {
> + __tostring = datetime_tostring,
> __serialize = datetime_serialize,
> __eq = datetime_eq,
> __lt = datetime_lt,
> @@ -466,6 +498,7 @@ local datetime_mt = {
> }
>
> local interval_mt = {
> + __tostring = datetime_tostring,
> __serialize = interval_serialize,
> __eq = datetime_eq,
> __lt = datetime_lt,
> @@ -487,6 +520,8 @@ return setmetatable(
> parse_time = parse_time,
> parse_zone = parse_zone,
>
> + tostring = datetime_tostring,
> +
I don't think we need this function in the module. Global tostring() is
enough.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime
2021-08-17 17:06 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 14:10 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 14:10 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On 17.08.2021 20:06, Vladimir Davydov via Tarantool-patches wrote:
> On Mon, Aug 16, 2021 at 02:59:37AM +0300, Timur Safin via
> Tarantool-patches wrote:
>> * introduced output routine for converting datetime
>> to their default output format.
>>
>> * use this routine for tostring() in datetime.lua
>>
>> Part of #5941
>> ---
>> extra/exports | 1 +
>> src/lib/core/datetime.c | 71 ++++++++++++++++++
>> src/lib/core/datetime.h | 9 +++
>> src/lua/datetime.lua | 35 +++++++++
>> test/app-tap/datetime.test.lua | 131 ++++++++++++++++++---------------
>> test/unit/CMakeLists.txt | 2 +-
>> test/unit/datetime.c | 61 +++++++++++----
>> 7 files changed, 236 insertions(+), 74 deletions(-)
>>
>> diff --git a/extra/exports b/extra/exports
>> index 80eb92abd..2437e175c 100644
>> --- a/extra/exports
>> +++ b/extra/exports
>> @@ -152,6 +152,7 @@ datetime_asctime
>> datetime_ctime
>> datetime_now
>> datetime_strftime
>> +datetime_to_string
>> decimal_unpack
>> decimal_from_string
>> decimal_unpack
>> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
>> index c48295a6f..c24a0df82 100644
>> --- a/src/lib/core/datetime.c
>> +++ b/src/lib/core/datetime.c
>> @@ -29,6 +29,8 @@
>> * SUCH DAMAGE.
>> */
>>
>> +#include <assert.h>
>> +#include <limits.h>
>> #include <string.h>
>> #include <time.h>
>>
>> @@ -94,3 +96,72 @@ datetime_strftime(const struct datetime *date, const
> char *fmt, char *buf,
>> struct tm *p_tm = datetime_to_tm(date);
>> return strftime(buf, len, fmt, p_tm);
>> }
>> +
>> +#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET *
> SECS_PER_DAY)
>> +
>> +/* NB! buf may be NULL, and we should handle it gracefully, returning
>
> /**
> * ...
>
> (according to our coding style)
AFAIK external comments (in headers) should be doxygen, but internal
comments (in the implementation) may be not in doxygen format. That's
internal comment.
>
>> + * calculated length of output string
>> + */
>> +int
>> +datetime_to_string(const struct datetime *date, char *buf, uint32_t
> len)
>> +{
>> +#define ADVANCE(sz) \
>> + if (buf != NULL) { \
>> + buf += sz; \
>> + len -= sz; \
>> + } \
>> + ret += sz;
>
> Please use SNPRINT helper.
Yes, already has have it reimplemented after Serge Petrenko comment.
It's a little bit shorter now. Nice!
>
>> +
>> + int offset = date->offset;
>> + /* for negative offsets around Epoch date we could get
>> + * negative secs value, which should be attributed to
>> + * 1969-12-31, not 1970-01-01, thus we first shift
>> + * epoch to Rata Die then divide by seconds per day,
>> + * not in reverse
>> + */
>> + int64_t secs = (int64_t)date->secs + offset * 60 +
> SECS_EPOCH_1970_OFFSET;
>> + assert((secs / SECS_PER_DAY) <= INT_MAX);
>> + dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
>> +
>> + int year, month, day, sec, ns, sign;
>> + dt_to_ymd(dt, &year, &month, &day);
>> +
>> + int hour = (secs / 3600) % 24,
>> + minute = (secs / 60) % 60;
>
> Please define on a separate line:
>
> int hour = ...;
> int minute = ...;
Done.
>
>> + sec = secs % 60;
>
> sec, secs, oh
>
> Please use better names.
Renamed.
>
>> + ns = date->nsec;
>> +
>> + int ret = 0;
>> + uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
>> + year, month, day, hour, minute);
>> + ADVANCE(sz);
>> + if (sec || ns) {
>> + sz = snprintf(buf, len, ":%02d", sec);
>> + ADVANCE(sz);
>> + if (ns) {
>> + if ((ns % 1000000) == 0)
>> + sz = snprintf(buf, len, ".%03d", ns /
> 1000000);
>> + else if ((ns % 1000) == 0)
>> + sz = snprintf(buf, len, ".%06d", ns /
> 1000);
>> + else
>> + sz = snprintf(buf, len, ".%09d", ns);
>> + ADVANCE(sz);
>> + }
>> + }
>> + if (offset == 0) {
>> + sz = snprintf(buf, len, "Z");
>> + ADVANCE(sz);
>> + }
>> + else {
>
> Bad formatting. Should be '} else {'.
Fixed.
>
>> + if (offset < 0)
>> + sign = '-', offset = -offset;
>
> Please don't abuse ','.
>
>> + else
>> + sign = '+';
>> +
>> + sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60,
> offset % 60);
>> + ADVANCE(sz);
>> + }
>> + return ret;
>> +}
>> +#undef ADVANCE
>> +
>> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
>> index 1a8d7e34f..964e76fcc 100644
>> --- a/src/lib/core/datetime.h
>> +++ b/src/lib/core/datetime.h
>> @@ -70,6 +70,15 @@ struct datetime_interval {
>> int32_t nsec;
>> };
>>
>> +/**
>> + * Convert datetime to string using default format
>> + * @param date source datetime value
>> + * @param buf output character buffer
>> + * @param len size ofoutput buffer
>> + */
>> +int
>> +datetime_to_string(const struct datetime *date, char *buf, uint32_t
> len);
>> +
>
> This should be an snprint-like function: buf and len should go first.
> This would allow us to use it in conjunction with SNPRINT.
Will reorder.
[BTW, if we keep argument at the same order then for x64 ABI, with
fastcall and upto 6 arguments passed via registers RDI, RSI, RDX, RCX,
R8, R9, then compiler may keep arguments in the same registers without
any spilling to the stack]
>
>> /**
>> * Convert datetime to string using default asctime format
>> * "Sun Sep 16 01:03:52 1973\n\0"
>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>> index ce579828f..4d946f194 100644
>> --- a/src/lua/datetime.lua
>> +++ b/src/lua/datetime.lua
>> @@ -249,6 +249,37 @@ local function datetime_new(obj)
>> return datetime_new_dt(dt, secs, frac, offset)
>> end
>>
>> +local function datetime_tostring(o)
>
> Please add a comment to this function with example output.
Documented.
>
>> + if ffi.typeof(o) == datetime_t then
>> + local sz = 48
>> + local buff = ffi.new('char[?]', sz)
>> + local len = builtin.datetime_to_string(o, buff, sz)
>> + assert(len < sz)
>> + return ffi.string(buff)
>> + elseif ffi.typeof(o) == interval_t then
>
> Please define separate functions for interval and datetime.
Yes, that will simplify. Part of a corresponding diff below...
----------------------------------------
@@ -373,9 +366,16 @@ local function datetime_new(obj)
secs = h * 3600 + m * 60 + s
end
- return datetime_new_dt(dt, secs, frac, offset)
+ return datetime_new_dt(dt, secs, nsec, offset)
end
+--[[
+ Convert to text datetime values
+
+ - datetime will use ISO-8601 forat:
+ 1970-01-01T00:00Z
+ 2021-08-18T16:57:08.981725+03:00
+]]
local function datetime_tostring(o)
if ffi.typeof(o) == datetime_t then
local sz = 48
@@ -383,7 +383,25 @@ local function datetime_tostring(o)
local len = builtin.datetime_to_string(o, buff, sz)
assert(len < sz)
return ffi.string(buff)
- elseif ffi.typeof(o) == interval_years_t then
+ end
+end
+
+--[[
+ Convert to text interval values of different types
+
+ - depending on a values stored there generic interval
+ values may display in following format:
+ +12 secs
+ -23 minutes, 0 seconds
+ +12 hours, 23 minutes, 1 seconds
+ -7 days, 23 hours, 23 minutes, 1 seconds
+ - years will be displayed as
+ +10 years
+ - months will be displayed as:
+ +2 months
+]]
+local function interval_tostring(o)
+ if ffi.typeof(o) == interval_years_t then
return ('%+d years'):format(o.y)
elseif ffi.typeof(o) == interval_months_t then
return ('%+d months'):format(o.m)
@@ -828,7 +901,7 @@ local datetime_mt = {
}
local interval_mt = {
- __tostring = datetime_tostring,
+ __tostring = interval_tostring,
__serialize = interval_serialize,
__eq = datetime_eq,
__lt = datetime_lt,
@@ -839,7 +912,7 @@ local interval_mt = {
}
local interval_tiny_mt = {
- __tostring = datetime_tostring,
+ __tostring = interval_tostring,
__serialize = interval_serialize,
__sub = datetime_sub,
__add = datetime_add,
----------------------------------------
>
>> + local ts = o.timestamp
>> + local sign = '+'
>> +
>> + if ts < 0 then
>> + ts = -ts
>> + sign = '-'
>> + end
>> +
>> + if ts < 60 then
>> + return ('%s%s secs'):format(sign, ts)
>> + elseif ts < 60 * 60 then
>> + return ('%+d minutes, %s seconds'):format(o.minutes, ts %
> 60)
>> + elseif ts < 24 * 60 * 60 then
>> + return ('%+d hours, %d minutes, %s seconds'):format(
>> + o.hours, o.minutes % 60, ts % 60)
>> + else
>> + return ('%+d days, %d hours, %d minutes, %s
> seconds'):format(
>> + o.days, o.hours % 24, o.minutes % 60, ts % 60)
>> + end
>> + end
>> +end
>> +
>> +
>> --[[
>> Basic Extended
>> 20121224 2012-12-24 Calendar date (ISO 8601)
>> @@ -457,6 +488,7 @@ local function strftime(fmt, o)
>> end
>>
>> local datetime_mt = {
>> + __tostring = datetime_tostring,
>> __serialize = datetime_serialize,
>> __eq = datetime_eq,
>> __lt = datetime_lt,
>> @@ -466,6 +498,7 @@ local datetime_mt = {
>> }
>>
>> local interval_mt = {
>> + __tostring = datetime_tostring,
>> __serialize = interval_serialize,
>> __eq = datetime_eq,
>> __lt = datetime_lt,
>> @@ -487,6 +520,8 @@ return setmetatable(
>> parse_time = parse_time,
>> parse_zone = parse_zone,
>>
>> + tostring = datetime_tostring,
>> +
>
> I don't think we need this function in the module. Global tostring() is
> enough.
>
-----------------------------------------
@@ -862,15 +935,12 @@ return setmetatable(
hours = interval_hours_new,
minutes = interval_minutes_new,
seconds = interval_seconds_new,
- interval = interval_new,
parse = parse,
parse_date = parse_date,
parse_time = parse_time,
parse_zone = parse_zone,
- tostring = datetime_tostring,
-
now = local_now,
strftime = strftime,
asctime = asctime,
-----------------------------------------
Deleted.
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (2 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 3/8] lua, datetime: display datetime Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-16 0:20 ` Safin Timur via Tarantool-patches
` (2 more replies)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
` (5 subsequent siblings)
9 siblings, 3 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
Serialize datetime_t as newly introduced MP_EXT type.
It saves 1 required integer field and upto 2 optional
unsigned fields in very compact fashion.
- secs is required field;
- but nsec, offset are both optional;
* json, yaml serialization formats, lua output mode
supported;
* exported symbols for datetime messagepack size calculations
so they are available for usage on Lua side.
Part of #5941
Part of #5946
---
| 5 +-
src/box/field_def.c | 35 +++---
src/box/field_def.h | 1 +
src/box/lua/serialize_lua.c | 7 +-
src/box/msgpack.c | 7 +-
src/box/tuple_compare.cc | 20 ++++
src/lib/core/CMakeLists.txt | 4 +-
src/lib/core/datetime.c | 9 ++
src/lib/core/datetime.h | 11 ++
src/lib/core/mp_datetime.c | 189 ++++++++++++++++++++++++++++++
src/lib/core/mp_datetime.h | 89 ++++++++++++++
src/lib/core/mp_extension_types.h | 1 +
src/lib/mpstream/mpstream.c | 11 ++
src/lib/mpstream/mpstream.h | 4 +
src/lua/msgpack.c | 12 ++
src/lua/msgpackffi.lua | 18 +++
src/lua/serializer.c | 4 +
src/lua/serializer.h | 2 +
src/lua/utils.c | 1 -
test/unit/datetime.c | 125 +++++++++++++++++++-
test/unit/datetime.result | 115 +++++++++++++++++-
third_party/lua-cjson/lua_cjson.c | 8 ++
third_party/lua-yaml/lyaml.cc | 6 +-
23 files changed, 661 insertions(+), 23 deletions(-)
create mode 100644 src/lib/core/mp_datetime.c
create mode 100644 src/lib/core/mp_datetime.h
--git a/extra/exports b/extra/exports
index 2437e175c..c34a5c2b5 100644
--- a/extra/exports
+++ b/extra/exports
@@ -151,9 +151,10 @@ csv_setopt
datetime_asctime
datetime_ctime
datetime_now
+datetime_pack
datetime_strftime
datetime_to_string
-decimal_unpack
+datetime_unpack
decimal_from_string
decimal_unpack
tnt_dt_dow
@@ -397,6 +398,7 @@ mp_decode_uint
mp_encode_array
mp_encode_bin
mp_encode_bool
+mp_encode_datetime
mp_encode_decimal
mp_encode_double
mp_encode_float
@@ -413,6 +415,7 @@ mp_next
mp_next_slowpath
mp_parser_hint
mp_sizeof_array
+mp_sizeof_datetime
mp_sizeof_decimal
mp_sizeof_str
mp_sizeof_uuid
diff --git a/src/box/field_def.c b/src/box/field_def.c
index 51acb8025..2682a42ee 100644
--- a/src/box/field_def.c
+++ b/src/box/field_def.c
@@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
/* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
/* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
/* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
+ /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is supported */
};
const uint32_t field_ext_type[] = {
@@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
/* [FIELD_TYPE_INTEGER] = */ 0,
/* [FIELD_TYPE_BOOLEAN] = */ 0,
/* [FIELD_TYPE_VARBINARY] = */ 0,
- /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID),
+ /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID) |
+ (1U << MP_DATETIME),
/* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
/* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
/* [FIELD_TYPE_ARRAY] = */ 0,
/* [FIELD_TYPE_MAP] = */ 0,
+ /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
};
const char *field_type_strs[] = {
@@ -104,6 +107,7 @@ const char *field_type_strs[] = {
/* [FIELD_TYPE_UUID] = */ "uuid",
/* [FIELD_TYPE_ARRAY] = */ "array",
/* [FIELD_TYPE_MAP] = */ "map",
+ /* [FIELD_TYPE_DATETIME] = */ "datetime",
};
const char *on_conflict_action_strs[] = {
@@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str, uint32_t len)
* values can be stored in the j type.
*/
static const bool field_type_compatibility[] = {
- /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
-/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false,
-/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false,
-/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false,
-/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false,
-/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false,
-/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false,
-/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false,
-/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false,
-/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false,
-/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false,
-/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false,
-/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false,
-/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true,
+ /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
+/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false, false,
+/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false, false,
+/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false, false,
+/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false, false,
+/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false, false,
+/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false, false,
+/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false, false,
+/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false, false,
+/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false, false,
+/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false, false,
+/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false, false,
+/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false, false,
+/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true, false,
+/* DATETIME */ true, false, false, false, false, false, false, false, true, false, false, false, false, true,
};
bool
diff --git a/src/box/field_def.h b/src/box/field_def.h
index c5cfe5e86..120b2a93d 100644
--- a/src/box/field_def.h
+++ b/src/box/field_def.h
@@ -63,6 +63,7 @@ enum field_type {
FIELD_TYPE_UUID,
FIELD_TYPE_ARRAY,
FIELD_TYPE_MAP,
+ FIELD_TYPE_DATETIME,
field_type_MAX
};
diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
index 1f791980f..51855011b 100644
--- a/src/box/lua/serialize_lua.c
+++ b/src/box/lua/serialize_lua.c
@@ -768,7 +768,7 @@ static int
dump_node(struct lua_dumper *d, struct node *nd, int indent)
{
struct luaL_field *field = &nd->field;
- char buf[FPCONV_G_FMT_BUFSIZE];
+ char buf[FPCONV_G_FMT_BUFSIZE + 8];
int ltype = lua_type(d->L, -1);
const char *str = NULL;
size_t len = 0;
@@ -861,6 +861,11 @@ dump_node(struct lua_dumper *d, struct node *nd, int indent)
str = tt_uuid_str(field->uuidval);
len = UUID_STR_LEN;
break;
+ case MP_DATETIME:
+ nd->mask |= NODE_QUOTE;
+ str = buf;
+ len = datetime_to_string(field->dateval, buf, sizeof buf);
+ break;
default:
d->err = EINVAL;
snprintf(d->err_msg, sizeof(d->err_msg),
diff --git a/src/box/msgpack.c b/src/box/msgpack.c
index 1723dea4c..12f4fd95a 100644
--- a/src/box/msgpack.c
+++ b/src/box/msgpack.c
@@ -1,5 +1,5 @@
/*
- * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file.
+ * Copyright 2020-2021, Tarantool AUTHORS, please see AUTHORS file.
*
* Redistribution and use in source and binary forms, with or
* without modification, are permitted provided that the following
@@ -35,6 +35,7 @@
#include "mp_decimal.h"
#include "uuid/mp_uuid.h"
#include "mp_error.h"
+#include "mp_datetime.h"
static int
msgpack_fprint_ext(FILE *file, const char **data, int depth)
@@ -47,6 +48,8 @@ msgpack_fprint_ext(FILE *file, const char **data, int depth)
return mp_fprint_decimal(file, data, len);
case MP_UUID:
return mp_fprint_uuid(file, data, len);
+ case MP_DATETIME:
+ return mp_fprint_datetime(file, data, len);
case MP_ERROR:
return mp_fprint_error(file, data, depth);
default:
@@ -65,6 +68,8 @@ msgpack_snprint_ext(char *buf, int size, const char **data, int depth)
return mp_snprint_decimal(buf, size, data, len);
case MP_UUID:
return mp_snprint_uuid(buf, size, data, len);
+ case MP_DATETIME:
+ return mp_snprint_datetime(buf, size, data, len);
case MP_ERROR:
return mp_snprint_error(buf, size, data, depth);
default:
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index 43cd29ce9..9a69f2a72 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -36,6 +36,7 @@
#include "lib/core/decimal.h"
#include "lib/core/mp_decimal.h"
#include "uuid/mp_uuid.h"
+#include "lib/core/mp_datetime.h"
#include "lib/core/mp_extension_types.h"
/* {{{ tuple_compare */
@@ -76,6 +77,7 @@ enum mp_class {
MP_CLASS_STR,
MP_CLASS_BIN,
MP_CLASS_UUID,
+ MP_CLASS_DATETIME,
MP_CLASS_ARRAY,
MP_CLASS_MAP,
mp_class_max,
@@ -99,6 +101,8 @@ static enum mp_class mp_ext_classes[] = {
/* .MP_UNKNOWN_EXTENSION = */ mp_class_max, /* unsupported */
/* .MP_DECIMAL = */ MP_CLASS_NUMBER,
/* .MP_UUID = */ MP_CLASS_UUID,
+ /* .MP_ERROR = */ mp_class_max,
+ /* .MP_DATETIME = */ MP_CLASS_DATETIME,
};
static enum mp_class
@@ -390,6 +394,19 @@ mp_compare_uuid(const char *field_a, const char *field_b)
return memcmp(field_a + 2, field_b + 2, UUID_PACKED_LEN);
}
+static int
+mp_compare_datetime(const char *lhs, const char *rhs)
+{
+ struct datetime lhs_dt, rhs_dt;
+ struct datetime *ret;
+ ret = mp_decode_datetime(&lhs, &lhs_dt);
+ assert(ret != NULL);
+ ret = mp_decode_datetime(&rhs, &rhs_dt);
+ assert(ret != NULL);
+ (void)ret;
+ return datetime_compare(&lhs_dt, &rhs_dt);
+}
+
typedef int (*mp_compare_f)(const char *, const char *);
static mp_compare_f mp_class_comparators[] = {
/* .MP_CLASS_NIL = */ NULL,
@@ -398,6 +415,7 @@ static mp_compare_f mp_class_comparators[] = {
/* .MP_CLASS_STR = */ mp_compare_str,
/* .MP_CLASS_BIN = */ mp_compare_bin,
/* .MP_CLASS_UUID = */ mp_compare_uuid,
+ /* .MP_CLASS_DATETIME=*/ mp_compare_datetime,
/* .MP_CLASS_ARRAY = */ NULL,
/* .MP_CLASS_MAP = */ NULL,
};
@@ -478,6 +496,8 @@ tuple_compare_field(const char *field_a, const char *field_b,
return mp_compare_decimal(field_a, field_b);
case FIELD_TYPE_UUID:
return mp_compare_uuid(field_a, field_b);
+ case FIELD_TYPE_DATETIME:
+ return mp_compare_datetime(field_a, field_b);
default:
unreachable();
return 0;
diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt
index 8bc776b82..61fc6548f 100644
--- a/src/lib/core/CMakeLists.txt
+++ b/src/lib/core/CMakeLists.txt
@@ -31,6 +31,7 @@ set(core_sources
mp_decimal.c
cord_buf.c
datetime.c
+ mp_datetime.c
)
if (TARGET_OS_NETBSD)
@@ -44,7 +45,8 @@ add_library(core STATIC ${core_sources})
target_link_libraries(core salad small uri decNumber bit ${LIBEV_LIBRARIES}
${LIBEIO_LIBRARIES} ${LIBCORO_LIBRARIES}
- ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES})
+ ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}
+ ${LIBCDT_LIBRARIES})
if (ENABLE_BACKTRACE AND NOT TARGET_OS_DARWIN)
target_link_libraries(core gcc_s ${UNWIND_LIBRARIES})
diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
index c24a0df82..baf9cc8ae 100644
--- a/src/lib/core/datetime.c
+++ b/src/lib/core/datetime.c
@@ -165,3 +165,12 @@ datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
}
#undef ADVANCE
+int
+datetime_compare(const struct datetime *lhs, const struct datetime *rhs)
+{
+ int result = COMPARE_RESULT(lhs->secs, rhs->secs);
+ if (result != 0)
+ return result;
+
+ return COMPARE_RESULT(lhs->nsec, rhs->nsec);
+}
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 964e76fcc..5122e422e 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -70,6 +70,17 @@ struct datetime_interval {
int32_t nsec;
};
+/**
+ * Compare arguments of a datetime type
+ * @param lhs left datetime argument
+ * @param rhs right datetime argument
+ * @retval < 0 if lhs less than rhs
+ * @retval = 0 if lhs and rhs equal
+ * @retval > 0 if lhs greater than rhs
+ */
+int
+datetime_compare(const struct datetime *lhs, const struct datetime *rhs);
+
/**
* Convert datetime to string using default format
* @param date source datetime value
diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
new file mode 100644
index 000000000..d0a3e562c
--- /dev/null
+++ b/src/lib/core/mp_datetime.c
@@ -0,0 +1,189 @@
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "mp_datetime.h"
+#include "msgpuck.h"
+#include "mp_extension_types.h"
+
+/*
+ Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length)
+ extension, which creates container of 1 to 3 integers.
+
+ +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+ |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset (uint) |
+ +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+
+ MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
+
+ - signed integer seconds part (required). Depending on the value of
+ seconds it may be from 1 to 8 bytes positive or negative integer number;
+
+ - [optional] fraction time in nanoseconds as unsigned integer.
+ If this value is 0 then it's not saved (unless there is offset field,
+ as below);
+
+ - [optional] timzeone offset in minutes as unsigned integer.
+ If this field is 0 then it's not saved.
+ */
+
+static inline uint32_t
+mp_sizeof_Xint(int64_t n)
+{
+ return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
+}
+
+static inline char *
+mp_encode_Xint(char *data, int64_t v)
+{
+ return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
+}
+
+static inline int64_t
+mp_decode_Xint(const char **data)
+{
+ switch (mp_typeof(**data)) {
+ case MP_UINT:
+ return (int64_t)mp_decode_uint(data);
+ case MP_INT:
+ return mp_decode_int(data);
+ default:
+ mp_unreachable();
+ }
+ return 0;
+}
+
+static inline uint32_t
+mp_sizeof_datetime_raw(const struct datetime *date)
+{
+ uint32_t sz = mp_sizeof_Xint(date->secs);
+
+ // even if nanosecs == 0 we need to output anything
+ // if we have non-null tz offset
+ if (date->nsec != 0 || date->offset != 0)
+ sz += mp_sizeof_Xint(date->nsec);
+ if (date->offset)
+ sz += mp_sizeof_Xint(date->offset);
+ return sz;
+}
+
+uint32_t
+mp_sizeof_datetime(const struct datetime *date)
+{
+ return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
+}
+
+struct datetime *
+datetime_unpack(const char **data, uint32_t len, struct datetime *date)
+{
+ const char * svp = *data;
+
+ memset(date, 0, sizeof(*date));
+
+ date->secs = mp_decode_Xint(data);
+
+ len -= *data - svp;
+ if (len <= 0)
+ return date;
+
+ svp = *data;
+ date->nsec = mp_decode_Xint(data);
+ len -= *data - svp;
+
+ if (len <= 0)
+ return date;
+
+ date->offset = mp_decode_Xint(data);
+
+ return date;
+}
+
+struct datetime *
+mp_decode_datetime(const char **data, struct datetime *date)
+{
+ if (mp_typeof(**data) != MP_EXT)
+ return NULL;
+
+ int8_t type;
+ uint32_t len = mp_decode_extl(data, &type);
+
+ if (type != MP_DATETIME || len == 0) {
+ return NULL;
+ }
+ return datetime_unpack(data, len, date);
+}
+
+char *
+datetime_pack(char *data, const struct datetime *date)
+{
+ data = mp_encode_Xint(data, date->secs);
+ if (date->nsec != 0 || date->offset != 0)
+ data = mp_encode_Xint(data, date->nsec);
+ if (date->offset)
+ data = mp_encode_Xint(data, date->offset);
+
+ return data;
+}
+
+char *
+mp_encode_datetime(char *data, const struct datetime *date)
+{
+ uint32_t len = mp_sizeof_datetime_raw(date);
+
+ data = mp_encode_extl(data, MP_DATETIME, len);
+
+ return datetime_pack(data, date);
+}
+
+int
+mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len)
+{
+ struct datetime date = {0};
+
+ if (datetime_unpack(data, len, &date) == NULL)
+ return -1;
+
+ return datetime_to_string(&date, buf, size);
+}
+
+int
+mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
+{
+ struct datetime date;
+
+ if (datetime_unpack(data, len, &date) == NULL)
+ return -1;
+
+ char buf[48];
+ datetime_to_string(&date, buf, sizeof buf);
+
+ return fprintf(file, "%s", buf);
+}
+
diff --git a/src/lib/core/mp_datetime.h b/src/lib/core/mp_datetime.h
new file mode 100644
index 000000000..9a4d2720c
--- /dev/null
+++ b/src/lib/core/mp_datetime.h
@@ -0,0 +1,89 @@
+#pragma once
+/*
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <stdio.h>
+#include "datetime.h"
+
+#if defined(__cplusplus)
+extern "C"
+{
+#endif /* defined(__cplusplus) */
+
+/**
+ * Unpack datetime data from MessagePack buffer.
+ * @sa datetime_pack
+ */
+struct datetime *
+datetime_unpack(const char **data, uint32_t len, struct datetime *date);
+
+/**
+ * Pack datetime data to MessagePack buffer.
+ * @sa datetime_unpack
+ */
+char *
+datetime_pack(char *data, const struct datetime *date);
+
+/**
+ * Calculate size of MessagePack buffer for datetime data.
+ */
+uint32_t
+mp_sizeof_datetime(const struct datetime *date);
+
+/**
+ * Decode data from MessagePack buffer to datetime structure.
+ */
+struct datetime *
+mp_decode_datetime(const char **data, struct datetime *date);
+
+/**
+ * Encode datetime structure to the MessagePack buffer.
+ */
+char *
+mp_encode_datetime(char *data, const struct datetime *date);
+
+/**
+ * Print datetime's string representation into a given buffer.
+ * @sa mp_snprint_decimal
+ */
+int
+mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len);
+
+/**
+ * Print datetime's string representation into a stream.
+ * @sa mp_fprint_decimal
+ */
+int
+mp_fprint_datetime(FILE *file, const char **data, uint32_t len);
+
+#if defined(__cplusplus)
+} /* extern "C" */
+#endif /* defined(__cplusplus) */
diff --git a/src/lib/core/mp_extension_types.h b/src/lib/core/mp_extension_types.h
index e3ff9f5d0..3b7eaee7c 100644
--- a/src/lib/core/mp_extension_types.h
+++ b/src/lib/core/mp_extension_types.h
@@ -44,6 +44,7 @@ enum mp_extension_type {
MP_DECIMAL = 1,
MP_UUID = 2,
MP_ERROR = 3,
+ MP_DATETIME = 4,
mp_extension_type_MAX,
};
diff --git a/src/lib/mpstream/mpstream.c b/src/lib/mpstream/mpstream.c
index 70ca29889..d3e1de965 100644
--- a/src/lib/mpstream/mpstream.c
+++ b/src/lib/mpstream/mpstream.c
@@ -35,6 +35,7 @@
#include "msgpuck.h"
#include "mp_decimal.h"
#include "uuid/mp_uuid.h"
+#include "mp_datetime.h"
void
mpstream_reserve_slow(struct mpstream *stream, size_t size)
@@ -208,6 +209,16 @@ mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid)
mpstream_advance(stream, pos - data);
}
+void
+mpstream_encode_datetime(struct mpstream *stream, const struct datetime *val)
+{
+ char *data = mpstream_reserve(stream, mp_sizeof_datetime(val));
+ if (data == NULL)
+ return;
+ char *pos = mp_encode_datetime(data, val);
+ mpstream_advance(stream, pos - data);
+}
+
void
mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n)
{
diff --git a/src/lib/mpstream/mpstream.h b/src/lib/mpstream/mpstream.h
index a60add143..94831160f 100644
--- a/src/lib/mpstream/mpstream.h
+++ b/src/lib/mpstream/mpstream.h
@@ -39,6 +39,7 @@ extern "C" {
#endif /* defined(__cplusplus) */
struct tt_uuid;
+struct datetime;
/**
* Ask the allocator to reserve at least size bytes. It can reserve
@@ -145,6 +146,9 @@ mpstream_encode_decimal(struct mpstream *stream, const decimal_t *val);
void
mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid);
+void
+mpstream_encode_datetime(struct mpstream *stream, const struct datetime *dt);
+
/** Copies n bytes from memory area src to stream. */
void
mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n);
diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c
index b6ecf2b1e..bca53f6b7 100644
--- a/src/lua/msgpack.c
+++ b/src/lua/msgpack.c
@@ -46,6 +46,7 @@
#include "lib/core/decimal.h" /* decimal_unpack() */
#include "lib/uuid/mp_uuid.h" /* mp_decode_uuid() */
#include "lib/core/mp_extension_types.h"
+#include "lib/core/mp_datetime.h"
#include "cord_buf.h"
#include <fiber.h>
@@ -200,6 +201,9 @@ restart: /* used by MP_EXT of unidentified subtype */
break;
case MP_ERROR:
return luamp_encode_extension(L, top, stream);
+ case MP_DATETIME:
+ mpstream_encode_datetime(stream, field->dateval);
+ break;
default:
/* Run trigger if type can't be encoded */
type = luamp_encode_extension(L, top, stream);
@@ -333,6 +337,14 @@ luamp_decode(struct lua_State *L, struct luaL_serializer *cfg,
goto ext_decode_err;
return;
}
+ case MP_DATETIME:
+ {
+ struct datetime *date = luaL_pushdatetime(L);
+ date = datetime_unpack(data, len, date);
+ if (date == NULL)
+ goto ext_decode_err;
+ return;
+ }
default:
/* reset data to the extension header */
*data = svp;
diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua
index 1d54f11b8..fb5e7d644 100644
--- a/src/lua/msgpackffi.lua
+++ b/src/lua/msgpackffi.lua
@@ -26,6 +26,10 @@ char *
mp_encode_uuid(char *data, const struct tt_uuid *uuid);
uint32_t
mp_sizeof_uuid();
+uint32_t
+mp_sizeof_datetime(const struct t_datetime_tz *date);
+char *
+mp_encode_datetime(char *data, const struct t_datetime_tz *date);
float
mp_decode_float(const char **data);
double
@@ -36,6 +40,8 @@ decimal_t *
decimal_unpack(const char **data, uint32_t len, decimal_t *dec);
struct tt_uuid *
uuid_unpack(const char **data, uint32_t len, struct tt_uuid *uuid);
+struct datetime *
+datetime_unpack(const char **data, uint32_t len, struct datetime *date);
]])
local strict_alignment = (jit.arch == 'arm')
@@ -142,6 +148,11 @@ local function encode_uuid(buf, uuid)
builtin.mp_encode_uuid(p, uuid)
end
+local function encode_datetime(buf, date)
+ local p = buf:alloc(builtin.mp_sizeof_datetime(date))
+ builtin.mp_encode_datetime(p, date)
+end
+
local function encode_int(buf, num)
if num >= 0 then
if num <= 0x7f then
@@ -320,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float)
on_encode(ffi.typeof('double'), encode_double)
on_encode(ffi.typeof('decimal_t'), encode_decimal)
on_encode(ffi.typeof('struct tt_uuid'), encode_uuid)
+on_encode(ffi.typeof('struct datetime'), encode_datetime)
--------------------------------------------------------------------------------
-- Decoder
@@ -513,6 +525,12 @@ local ext_decoder = {
builtin.uuid_unpack(data, len, uuid)
return uuid
end,
+ -- MP_DATETIME
+ [4] = function(data, len)
+ local dt = ffi.new("struct datetime")
+ builtin.datetime_unpack(data, len, dt)
+ return dt
+ end,
}
local function decode_ext(data)
diff --git a/src/lua/serializer.c b/src/lua/serializer.c
index 8db6746a3..24f4a5ff9 100644
--- a/src/lua/serializer.c
+++ b/src/lua/serializer.c
@@ -41,6 +41,7 @@
#include "lib/core/mp_extension_types.h"
#include "lua/error.h"
+#include "datetime.h"
#include "trivia/util.h"
#include "diag.h"
#include "serializer_opts.h"
@@ -544,6 +545,9 @@ luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg,
opts != NULL &&
opts->error_marshaling_enabled) {
field->ext_type = MP_ERROR;
+ } else if (cd->ctypeid == CTID_DATETIME) {
+ field->ext_type = MP_DATETIME;
+ field->dateval = (struct datetime *)cdata;
} else {
field->ext_type = MP_UNKNOWN_EXTENSION;
}
diff --git a/src/lua/serializer.h b/src/lua/serializer.h
index 0a0501a74..e7a240e0a 100644
--- a/src/lua/serializer.h
+++ b/src/lua/serializer.h
@@ -52,6 +52,7 @@ extern "C" {
#include <lauxlib.h>
#include "trigger.h"
+#include "lib/core/datetime.h"
#include "lib/core/decimal.h" /* decimal_t */
#include "lib/core/mp_extension_types.h"
#include "lua/error.h"
@@ -223,6 +224,7 @@ struct luaL_field {
uint32_t size;
decimal_t *decval;
struct tt_uuid *uuidval;
+ struct datetime *dateval;
};
enum mp_type type;
/* subtypes of MP_EXT */
diff --git a/src/lua/utils.c b/src/lua/utils.c
index 2c89326f3..771f6f278 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -254,7 +254,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
lua_pop(L, 1);
}
-
/**
* A helper to register a single type metatable.
*/
diff --git a/test/unit/datetime.c b/test/unit/datetime.c
index 1ae76003b..a72ac2253 100644
--- a/test/unit/datetime.c
+++ b/test/unit/datetime.c
@@ -6,6 +6,9 @@
#include "unit.h"
#include "datetime.h"
+#include "mp_datetime.h"
+#include "msgpuck.h"
+#include "mp_extension_types.h"
static const char sample[] = "2012-12-24T15:30Z";
@@ -247,12 +250,132 @@ tostring_datetime_test(void)
check_plan();
}
+static void
+mp_datetime_test()
+{
+ static struct {
+ int64_t secs;
+ uint32_t nsec;
+ uint32_t offset;
+ uint32_t len;
+ } tests[] = {
+ {/* '1970-01-01T02:00+02:00' */ 0, 0, 120, 6},
+ {/* '1970-01-01T01:30+01:30' */ 0, 0, 90, 6},
+ {/* '1970-01-01T01:00+01:00' */ 0, 0, 60, 6},
+ {/* '1970-01-01T00:01+00:01' */ 0, 0, 1, 6},
+ {/* '1970-01-01T00:00Z' */ 0, 0, 0, 3},
+ {/* '1969-12-31T23:59-00:01' */ 0, 0, -1, 6},
+ {/* '1969-12-31T23:00-01:00' */ 0, 0, -60, 6},
+ {/* '1969-12-31T22:30-01:30' */ 0, 0, -90, 6},
+ {/* '1969-12-31T22:00-02:00' */ 0, 0, -120, 6},
+ {/* '1970-01-01T00:00:00.123456789Z' */ 0, 123456789, 0, 9},
+ {/* '1970-01-01T00:00:00.123456Z' */ 0, 123456000, 0, 9},
+ {/* '1970-01-01T00:00:00.123Z' */ 0, 123000000, 0, 9},
+ {/* '1973-11-29T21:33:09Z' */ 123456789, 0, 0, 8},
+ {/* '2013-10-28T17:51:56Z' */ 1382982716, 0, 0, 8},
+ {/* '9999-12-31T23:59:59Z' */ 253402300799, 0, 0, 12},
+ {/* '9999-12-31T23:59:59.123456789Z' */ 253402300799, 123456789, 0, 17},
+ {/* '9999-12-31T23:59:59.123456789-02:00' */ 253402300799, 123456789, -120, 18},
+ };
+ size_t index;
+
+ plan(85);
+ for (index = 0; index < DIM(tests); index++) {
+ struct datetime date = {
+ tests[index].secs,
+ tests[index].nsec,
+ tests[index].offset
+ };
+ char buf[24], *data = buf;
+ const char *data1 = buf;
+ struct datetime ret;
+
+ char *end = mp_encode_datetime(data, &date);
+ uint32_t len = mp_sizeof_datetime(&date);
+ is(len, tests[index].len, "len %u, expected len %u",
+ len, tests[index].len);
+ is(end - data, len,
+ "mp_sizeof_datetime(%d) == encoded length %ld",
+ len, end - data);
+
+ struct datetime *rc = mp_decode_datetime(&data1, &ret);
+ is(rc, &ret, "mp_decode_datetime() return code");
+ is(data1, end, "mp_sizeof_uuid() == decoded length");
+ is(datetime_compare(&date, &ret), 0, "datetime_compare(&date, &ret)");
+ }
+ check_plan();
+}
+
+
+static int
+mp_fprint_ext_test(FILE *file, const char **data, int depth)
+{
+ (void)depth;
+ int8_t type;
+ uint32_t len = mp_decode_extl(data, &type);
+ if (type != MP_DATETIME)
+ return fprintf(file, "undefined");
+ return mp_fprint_datetime(file, data, len);
+}
+
+static int
+mp_snprint_ext_test(char *buf, int size, const char **data, int depth)
+{
+ (void)depth;
+ int8_t type;
+ uint32_t len = mp_decode_extl(data, &type);
+ if (type != MP_DATETIME)
+ return snprintf(buf, size, "undefined");
+ return mp_snprint_datetime(buf, size, data, len);
+}
+
+static void
+mp_print_test(void)
+{
+ plan(5);
+ header();
+
+ mp_snprint_ext = mp_snprint_ext_test;
+ mp_fprint_ext = mp_fprint_ext_test;
+
+ char sample[64];
+ char buffer[64];
+ char str[64];
+ struct datetime date = {0, 0, 0}; // 1970-01-01T00:00Z
+
+ mp_encode_datetime(buffer, &date);
+ int sz = datetime_to_string(&date, str, sizeof str);
+ int rc = mp_snprint(NULL, 0, buffer);
+ is(rc, sz, "correct mp_snprint size %u with empty buffer", rc);
+ rc = mp_snprint(str, sizeof(str), buffer);
+ is(rc, sz, "correct mp_snprint size %u", rc);
+ datetime_to_string(&date, sample, sizeof sample);
+ is(strcmp(str, sample), 0, "correct mp_snprint result");
+
+ FILE *f = tmpfile();
+ rc = mp_fprint(f, buffer);
+ is(rc, sz, "correct mp_fprint size %u", sz);
+ rewind(f);
+ rc = fread(str, 1, sizeof(str), f);
+ str[rc] = 0;
+ is(strcmp(str, sample), 0, "correct mp_fprint result %u", rc);
+ fclose(f);
+
+ mp_snprint_ext = mp_snprint_ext_default;
+ mp_fprint_ext = mp_fprint_ext_default;
+
+ footer();
+ check_plan();
+}
+
int
main(void)
{
- plan(2);
+ plan(4);
datetime_test();
tostring_datetime_test();
+ mp_datetime_test();
+ mp_print_test();
return check_plan();
}
diff --git a/test/unit/datetime.result b/test/unit/datetime.result
index 33997d9df..5f01c4344 100644
--- a/test/unit/datetime.result
+++ b/test/unit/datetime.result
@@ -1,4 +1,4 @@
-1..1
+1..4
1..355
ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z'
ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z
@@ -356,3 +356,116 @@
ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00'
ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00
ok 1 - subtests
+ 1..15
+ ok 1 - string '1970-01-01T02:00+02:00' expected, received '1970-01-01T02:00+02:00'
+ ok 2 - string '1970-01-01T01:30+01:30' expected, received '1970-01-01T01:30+01:30'
+ ok 3 - string '1970-01-01T01:00+01:00' expected, received '1970-01-01T01:00+01:00'
+ ok 4 - string '1970-01-01T00:01+00:01' expected, received '1970-01-01T00:01+00:01'
+ ok 5 - string '1970-01-01T00:00Z' expected, received '1970-01-01T00:00Z'
+ ok 6 - string '1969-12-31T23:59-00:01' expected, received '1969-12-31T23:59-00:01'
+ ok 7 - string '1969-12-31T23:00-01:00' expected, received '1969-12-31T23:00-01:00'
+ ok 8 - string '1969-12-31T22:30-01:30' expected, received '1969-12-31T22:30-01:30'
+ ok 9 - string '1969-12-31T22:00-02:00' expected, received '1969-12-31T22:00-02:00'
+ ok 10 - string '1970-01-01T00:00:00.123456789Z' expected, received '1970-01-01T00:00:00.123456789Z'
+ ok 11 - string '1970-01-01T00:00:00.123456Z' expected, received '1970-01-01T00:00:00.123456Z'
+ ok 12 - string '1970-01-01T00:00:00.123Z' expected, received '1970-01-01T00:00:00.123Z'
+ ok 13 - string '1973-11-29T21:33:09Z' expected, received '1973-11-29T21:33:09Z'
+ ok 14 - string '2013-10-28T17:51:56Z' expected, received '2013-10-28T17:51:56Z'
+ ok 15 - string '9999-12-31T23:59:59Z' expected, received '9999-12-31T23:59:59Z'
+ok 2 - subtests
+ 1..85
+ ok 1 - len 6, expected len 6
+ ok 2 - mp_sizeof_datetime(6) == encoded length 6
+ ok 3 - mp_decode_datetime() return code
+ ok 4 - mp_sizeof_uuid() == decoded length
+ ok 5 - datetime_compare(&date, &ret)
+ ok 6 - len 6, expected len 6
+ ok 7 - mp_sizeof_datetime(6) == encoded length 6
+ ok 8 - mp_decode_datetime() return code
+ ok 9 - mp_sizeof_uuid() == decoded length
+ ok 10 - datetime_compare(&date, &ret)
+ ok 11 - len 6, expected len 6
+ ok 12 - mp_sizeof_datetime(6) == encoded length 6
+ ok 13 - mp_decode_datetime() return code
+ ok 14 - mp_sizeof_uuid() == decoded length
+ ok 15 - datetime_compare(&date, &ret)
+ ok 16 - len 6, expected len 6
+ ok 17 - mp_sizeof_datetime(6) == encoded length 6
+ ok 18 - mp_decode_datetime() return code
+ ok 19 - mp_sizeof_uuid() == decoded length
+ ok 20 - datetime_compare(&date, &ret)
+ ok 21 - len 3, expected len 3
+ ok 22 - mp_sizeof_datetime(3) == encoded length 3
+ ok 23 - mp_decode_datetime() return code
+ ok 24 - mp_sizeof_uuid() == decoded length
+ ok 25 - datetime_compare(&date, &ret)
+ ok 26 - len 6, expected len 6
+ ok 27 - mp_sizeof_datetime(6) == encoded length 6
+ ok 28 - mp_decode_datetime() return code
+ ok 29 - mp_sizeof_uuid() == decoded length
+ ok 30 - datetime_compare(&date, &ret)
+ ok 31 - len 6, expected len 6
+ ok 32 - mp_sizeof_datetime(6) == encoded length 6
+ ok 33 - mp_decode_datetime() return code
+ ok 34 - mp_sizeof_uuid() == decoded length
+ ok 35 - datetime_compare(&date, &ret)
+ ok 36 - len 6, expected len 6
+ ok 37 - mp_sizeof_datetime(6) == encoded length 6
+ ok 38 - mp_decode_datetime() return code
+ ok 39 - mp_sizeof_uuid() == decoded length
+ ok 40 - datetime_compare(&date, &ret)
+ ok 41 - len 6, expected len 6
+ ok 42 - mp_sizeof_datetime(6) == encoded length 6
+ ok 43 - mp_decode_datetime() return code
+ ok 44 - mp_sizeof_uuid() == decoded length
+ ok 45 - datetime_compare(&date, &ret)
+ ok 46 - len 9, expected len 9
+ ok 47 - mp_sizeof_datetime(9) == encoded length 9
+ ok 48 - mp_decode_datetime() return code
+ ok 49 - mp_sizeof_uuid() == decoded length
+ ok 50 - datetime_compare(&date, &ret)
+ ok 51 - len 9, expected len 9
+ ok 52 - mp_sizeof_datetime(9) == encoded length 9
+ ok 53 - mp_decode_datetime() return code
+ ok 54 - mp_sizeof_uuid() == decoded length
+ ok 55 - datetime_compare(&date, &ret)
+ ok 56 - len 9, expected len 9
+ ok 57 - mp_sizeof_datetime(9) == encoded length 9
+ ok 58 - mp_decode_datetime() return code
+ ok 59 - mp_sizeof_uuid() == decoded length
+ ok 60 - datetime_compare(&date, &ret)
+ ok 61 - len 8, expected len 8
+ ok 62 - mp_sizeof_datetime(8) == encoded length 8
+ ok 63 - mp_decode_datetime() return code
+ ok 64 - mp_sizeof_uuid() == decoded length
+ ok 65 - datetime_compare(&date, &ret)
+ ok 66 - len 8, expected len 8
+ ok 67 - mp_sizeof_datetime(8) == encoded length 8
+ ok 68 - mp_decode_datetime() return code
+ ok 69 - mp_sizeof_uuid() == decoded length
+ ok 70 - datetime_compare(&date, &ret)
+ ok 71 - len 12, expected len 12
+ ok 72 - mp_sizeof_datetime(12) == encoded length 12
+ ok 73 - mp_decode_datetime() return code
+ ok 74 - mp_sizeof_uuid() == decoded length
+ ok 75 - datetime_compare(&date, &ret)
+ ok 76 - len 17, expected len 17
+ ok 77 - mp_sizeof_datetime(17) == encoded length 17
+ ok 78 - mp_decode_datetime() return code
+ ok 79 - mp_sizeof_uuid() == decoded length
+ ok 80 - datetime_compare(&date, &ret)
+ ok 81 - len 18, expected len 18
+ ok 82 - mp_sizeof_datetime(18) == encoded length 18
+ ok 83 - mp_decode_datetime() return code
+ ok 84 - mp_sizeof_uuid() == decoded length
+ ok 85 - datetime_compare(&date, &ret)
+ok 3 - subtests
+ 1..5
+ *** mp_print_test ***
+ ok 1 - correct mp_snprint size 17 with empty buffer
+ ok 2 - correct mp_snprint size 17
+ ok 3 - correct mp_snprint result
+ ok 4 - correct mp_fprint size 17
+ ok 5 - correct mp_fprint result 17
+ *** mp_print_test: done ***
+ok 4 - subtests
diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c
index 7a326075a..924e0419a 100644
--- a/third_party/lua-cjson/lua_cjson.c
+++ b/third_party/lua-cjson/lua_cjson.c
@@ -52,6 +52,7 @@
#include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
#include "tt_static.h"
#include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */
+#include "core/datetime.h"
#include "cord_buf.h"
typedef enum {
@@ -426,6 +427,13 @@ static void json_append_data(lua_State *l, struct luaL_serializer *cfg,
case MP_UUID:
return json_append_string(cfg, json, tt_uuid_str(field.uuidval),
UUID_STR_LEN);
+
+ case MP_DATETIME:
+ {
+ char buf[128];
+ size_t sz = datetime_to_string(field.dateval, buf, sizeof buf);
+ return json_append_string(cfg, json, buf, sz);
+ }
default:
assert(false);
}
diff --git a/third_party/lua-yaml/lyaml.cc b/third_party/lua-yaml/lyaml.cc
index 2b67dcc6a..034104737 100644
--- a/third_party/lua-yaml/lyaml.cc
+++ b/third_party/lua-yaml/lyaml.cc
@@ -617,7 +617,7 @@ static int dump_node(struct lua_yaml_dumper *dumper)
yaml_event_t ev;
yaml_scalar_style_t style = YAML_PLAIN_SCALAR_STYLE;
int is_binary = 0;
- char buf[FPCONV_G_FMT_BUFSIZE];
+ char buf[FPCONV_G_FMT_BUFSIZE + 8];
struct luaL_field field;
bool unused;
(void) unused;
@@ -707,6 +707,10 @@ static int dump_node(struct lua_yaml_dumper *dumper)
str = tt_uuid_str(field.uuidval);
len = UUID_STR_LEN;
break;
+ case MP_DATETIME:
+ len = datetime_to_string(field.dateval, buf, sizeof buf);
+ str = buf;
+ break;
default:
assert(0); /* checked by luaL_checkfield() */
}
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime Timur Safin via Tarantool-patches
@ 2021-08-16 0:20 ` Safin Timur via Tarantool-patches
2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 18:36 ` Vladimir Davydov via Tarantool-patches
2 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-16 0:20 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
On 16.08.2021 2:59, Timur Safin wrote:
> Serialize datetime_t as newly introduced MP_EXT type.
> It saves 1 required integer field and upto 2 optional
> unsigned fields in very compact fashion.
> - secs is required field;
> - but nsec, offset are both optional;
>
> * json, yaml serialization formats, lua output mode
> supported;
> * exported symbols for datetime messagepack size calculations
> so they are available for usage on Lua side.
>
> Part of #5941
> Part of #5946
...
> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
> new file mode 100644
> index 000000000..d0a3e562c
> --- /dev/null
> +++ b/src/lib/core/mp_datetime.c
...
> +
> +int
> +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len)
> +{
> + struct datetime date = {0};
Due to old gcc compiler used by CentOS7 I had to modify these 2
functions - see incremental patch below...
> +
> + if (datetime_unpack(data, len, &date) == NULL)
> + return -1;
> +
> + return datetime_to_string(&date, buf, size);
> +}
> +
> +int
> +mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
> +{
> + struct datetime date;
> +
> + if (datetime_unpack(data, len, &date) == NULL)
> + return -1;
> +
> + char buf[48];
> + datetime_to_string(&date, buf, sizeof buf);
> +
> + return fprintf(file, "%s", buf);
> +}
> +
----------------------------------------------------------------
03:15 $ git diff
diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
index d0a3e562c..7e475d5f1 100644
--- a/src/lib/core/mp_datetime.c
+++ b/src/lib/core/mp_datetime.c
@@ -165,7 +165,7 @@ mp_encode_datetime(char *data, const struct datetime
*date)
int
mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len)
{
- struct datetime date = {0};
+ struct datetime date = {0, 0, 0};
if (datetime_unpack(data, len, &date) == NULL)
return -1;
@@ -176,7 +176,7 @@ mp_snprint_datetime(char *buf, int size, const char
**data, uint32_t len)
int
mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
{
- struct datetime date;
+ struct datetime date = {0, 0, 0};
if (datetime_unpack(data, len, &date) == NULL)
return -1;
----------------------------------------------------------------
Sorry for the inconvenience!
[Applied and force pushed]
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-16 0:20 ` Safin Timur via Tarantool-patches
@ 2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:15 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 03:20, Safin Timur via Tarantool-patches пишет:
>
>
> On 16.08.2021 2:59, Timur Safin wrote:
>> Serialize datetime_t as newly introduced MP_EXT type.
>> It saves 1 required integer field and upto 2 optional
>> unsigned fields in very compact fashion.
>> - secs is required field;
>> - but nsec, offset are both optional;
>>
>> * json, yaml serialization formats, lua output mode
>> supported;
>> * exported symbols for datetime messagepack size calculations
>> so they are available for usage on Lua side.
>>
>> Part of #5941
>> Part of #5946
> ...
>
>> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
>> new file mode 100644
>> index 000000000..d0a3e562c
>> --- /dev/null
>> +++ b/src/lib/core/mp_datetime.c
> ...
>
>> +
>> +int
>> +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t
>> len)
>> +{
>> + struct datetime date = {0};
>
> Due to old gcc compiler used by CentOS7 I had to modify these 2
> functions - see incremental patch below...
>> +
>> + if (datetime_unpack(data, len, &date) == NULL)
>> + return -1;
>> +
>> + return datetime_to_string(&date, buf, size);
>> +}
>> +
>> +int
>> +mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
>> +{
>> + struct datetime date;
>> +
>> + if (datetime_unpack(data, len, &date) == NULL)
>> + return -1;
>> +
>> + char buf[48];
>> + datetime_to_string(&date, buf, sizeof buf);
>> +
>> + return fprintf(file, "%s", buf);
>> +}
>> +
>
> ----------------------------------------------------------------
> 03:15 $ git diff
> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
> index d0a3e562c..7e475d5f1 100644
> --- a/src/lib/core/mp_datetime.c
> +++ b/src/lib/core/mp_datetime.c
> @@ -165,7 +165,7 @@ mp_encode_datetime(char *data, const struct
> datetime *date)
> int
> mp_snprint_datetime(char *buf, int size, const char **data, uint32_t
> len)
> {
> - struct datetime date = {0};
> + struct datetime date = {0, 0, 0};
Please, use designated initializers here:
struct datetime date = {
.secs = ...,
.nsec = ...,
...
>
> if (datetime_unpack(data, len, &date) == NULL)
> return -1;
> @@ -176,7 +176,7 @@ mp_snprint_datetime(char *buf, int size, const
> char **data, uint32_t len)
> int
> mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
> {
> - struct datetime date;
> + struct datetime date = {0, 0, 0};
>
> if (datetime_unpack(data, len, &date) == NULL)
> return -1;
> ----------------------------------------------------------------
>
> Sorry for the inconvenience!
>
> [Applied and force pushed]
>
> Timur
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime Timur Safin via Tarantool-patches
2021-08-16 0:20 ` Safin Timur via Tarantool-patches
@ 2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:42 ` Safin Timur via Tarantool-patches
2021-08-17 18:36 ` Vladimir Davydov via Tarantool-patches
2 siblings, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:16 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> Serialize datetime_t as newly introduced MP_EXT type.
> It saves 1 required integer field and upto 2 optional
> unsigned fields in very compact fashion.
> - secs is required field;
> - but nsec, offset are both optional;
>
> * json, yaml serialization formats, lua output mode
> supported;
> * exported symbols for datetime messagepack size calculations
> so they are available for usage on Lua side.
>
> Part of #5941
> Part of #5946
> ---
> extra/exports | 5 +-
> src/box/field_def.c | 35 +++---
> src/box/field_def.h | 1 +
> src/box/lua/serialize_lua.c | 7 +-
> src/box/msgpack.c | 7 +-
> src/box/tuple_compare.cc | 20 ++++
> src/lib/core/CMakeLists.txt | 4 +-
> src/lib/core/datetime.c | 9 ++
> src/lib/core/datetime.h | 11 ++
> src/lib/core/mp_datetime.c | 189 ++++++++++++++++++++++++++++++
> src/lib/core/mp_datetime.h | 89 ++++++++++++++
> src/lib/core/mp_extension_types.h | 1 +
> src/lib/mpstream/mpstream.c | 11 ++
> src/lib/mpstream/mpstream.h | 4 +
> src/lua/msgpack.c | 12 ++
> src/lua/msgpackffi.lua | 18 +++
> src/lua/serializer.c | 4 +
> src/lua/serializer.h | 2 +
> src/lua/utils.c | 1 -
> test/unit/datetime.c | 125 +++++++++++++++++++-
> test/unit/datetime.result | 115 +++++++++++++++++-
> third_party/lua-cjson/lua_cjson.c | 8 ++
> third_party/lua-yaml/lyaml.cc | 6 +-
> 23 files changed, 661 insertions(+), 23 deletions(-)
> create mode 100644 src/lib/core/mp_datetime.c
> create mode 100644 src/lib/core/mp_datetime.h
>
> diff --git a/extra/exports b/extra/exports
> index 2437e175c..c34a5c2b5 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -151,9 +151,10 @@ csv_setopt
> datetime_asctime
> datetime_ctime
> datetime_now
> +datetime_pack
> datetime_strftime
> datetime_to_string
> -decimal_unpack
> +datetime_unpack
decimal_unpack should stay there.
> decimal_from_string
> decimal_unpack
> tnt_dt_dow
> @@ -397,6 +398,7 @@ mp_decode_uint
> mp_encode_array
> mp_encode_bin
> mp_encode_bool
> +mp_encode_datetime
> mp_encode_decimal
> mp_encode_double
> mp_encode_float
> @@ -413,6 +415,7 @@ mp_next
> mp_next_slowpath
> mp_parser_hint
> mp_sizeof_array
> +mp_sizeof_datetime
> mp_sizeof_decimal
> mp_sizeof_str
> mp_sizeof_uuid
> diff --git a/src/box/field_def.c b/src/box/field_def.c
> index 51acb8025..2682a42ee 100644
> --- a/src/box/field_def.c
> +++ b/src/box/field_def.c
> @@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
> /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
> /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
> /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
> + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is supported */
> };
>
> const uint32_t field_ext_type[] = {
> @@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
> /* [FIELD_TYPE_INTEGER] = */ 0,
> /* [FIELD_TYPE_BOOLEAN] = */ 0,
> /* [FIELD_TYPE_VARBINARY] = */ 0,
> - /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID),
> + /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID) |
> + (1U << MP_DATETIME),
> /* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
> /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
> /* [FIELD_TYPE_ARRAY] = */ 0,
> /* [FIELD_TYPE_MAP] = */ 0,
> + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
> };
>
> const char *field_type_strs[] = {
> @@ -104,6 +107,7 @@ const char *field_type_strs[] = {
> /* [FIELD_TYPE_UUID] = */ "uuid",
> /* [FIELD_TYPE_ARRAY] = */ "array",
> /* [FIELD_TYPE_MAP] = */ "map",
> + /* [FIELD_TYPE_DATETIME] = */ "datetime",
> };
>
> const char *on_conflict_action_strs[] = {
> @@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str, uint32_t len)
> * values can be stored in the j type.
> */
> static const bool field_type_compatibility[] = {
> - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
> -/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false,
> -/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false,
> -/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false,
> -/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false,
> -/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false,
> -/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false,
> -/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false,
> -/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false,
> -/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false,
> -/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false,
> -/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false,
> -/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false,
> -/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true,
> + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
> +/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false, false,
> +/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false, false,
> +/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false, false,
> +/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false, false,
> +/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false, false,
> +/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false, false,
> +/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false, false,
> +/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false, false,
> +/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false, false,
> +/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false, false,
> +/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false, false,
> +/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false, false,
> +/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true, false,
> +/* DATETIME */ true, false, false, false, false, false, false, false, true, false, false, false, false, true,
> };
>
> bool
> diff --git a/src/box/field_def.h b/src/box/field_def.h
> index c5cfe5e86..120b2a93d 100644
> --- a/src/box/field_def.h
> +++ b/src/box/field_def.h
> @@ -63,6 +63,7 @@ enum field_type {
> FIELD_TYPE_UUID,
> FIELD_TYPE_ARRAY,
> FIELD_TYPE_MAP,
> + FIELD_TYPE_DATETIME,
> field_type_MAX
> };
Please, define FIELD_TYPE_DATETIME higher.
Right after FIELD_TYPE_UUID.
This way you won't need to rework field type allowed in index check
in the next commit.
>
> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
> index 1f791980f..51855011b 100644
> --- a/src/box/lua/serialize_lua.c
> +++ b/src/box/lua/serialize_lua.c
> @@ -768,7 +768,7 @@ static int
> dump_node(struct lua_dumper *d, struct node *nd, int indent)
> {
> struct luaL_field *field = &nd->field;
> - char buf[FPCONV_G_FMT_BUFSIZE];
> + char buf[FPCONV_G_FMT_BUFSIZE + 8];
Why "+8"?
> int ltype = lua_type(d->L, -1);
> const char *str = NULL;
> size_t len = 0;
<stripped>
> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
> new file mode 100644
> index 000000000..d0a3e562c
> --- /dev/null
> +++ b/src/lib/core/mp_datetime.c
> @@ -0,0 +1,189 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
Same about the license.
Please, replace that with
/*
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
And do the same for all new files.
> +#include "mp_datetime.h"
> +#include "msgpuck.h"
> +#include "mp_extension_types.h"
> +
> +/*
> + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length)
> + extension, which creates container of 1 to 3 integers.
> +
> + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset (uint) |
> + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
The order should be 0xC7, len(uint8), 4, seconds, ...
according to
https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family
> +
> + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
> +
> + - signed integer seconds part (required). Depending on the value of
> + seconds it may be from 1 to 8 bytes positive or negative integer number;
> +
> + - [optional] fraction time in nanoseconds as unsigned integer.
> + If this value is 0 then it's not saved (unless there is offset field,
> + as below);
> +
> + - [optional] timzeone offset in minutes as unsigned integer.
> + If this field is 0 then it's not saved.
> + */
> +
> +static inline uint32_t
> +mp_sizeof_Xint(int64_t n)
> +{
> + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
> +}
> +
> +static inline char *
> +mp_encode_Xint(char *data, int64_t v)
> +{
> + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
> +}
> +
> +static inline int64_t
> +mp_decode_Xint(const char **data)
> +{
> + switch (mp_typeof(**data)) {
> + case MP_UINT:
> + return (int64_t)mp_decode_uint(data);
> + case MP_INT:
> + return mp_decode_int(data);
> + default:
> + mp_unreachable();
> + }
> + return 0;
> +}
I believe mp_decode_Xint and mp_encode_Xint
belong to a more generic file, but I couldn't find an
appropriate one. Up to you.
> +
> +static inline uint32_t
> +mp_sizeof_datetime_raw(const struct datetime *date)
> +{
> + uint32_t sz = mp_sizeof_Xint(date->secs);
> +
> + // even if nanosecs == 0 we need to output anything
> + // if we have non-null tz offset
Please, stick with our comment format:
/*
* Even if nanosecs == 0 we need to output anything
* if we have non-null tz offset
*/
> + if (date->nsec != 0 || date->offset != 0)
> + sz += mp_sizeof_Xint(date->nsec);
> + if (date->offset)
> + sz += mp_sizeof_Xint(date->offset);
> + return sz;
> +}
> +
> +uint32_t
> +mp_sizeof_datetime(const struct datetime *date)
> +{
> + return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
> +}
> +
> +struct datetime *
> +datetime_unpack(const char **data, uint32_t len, struct datetime *date)
> +{
> + const char * svp = *data;
> +
> + memset(date, 0, sizeof(*date));
> +
> + date->secs = mp_decode_Xint(data);
Please, leave a comment about date->secs possible range here.
Why is it ok to store a decoded int64_t in a double.
> +
> + len -= *data - svp;
> + if (len <= 0)
> + return date;
> +
> + svp = *data;
> + date->nsec = mp_decode_Xint(data);
> + len -= *data - svp;
> +
> + if (len <= 0)
> + return date;
> +
> + date->offset = mp_decode_Xint(data);
> +
> + return date;
> +}
> +
> +struct datetime *
> +mp_decode_datetime(const char **data, struct datetime *date)
> +{
> + if (mp_typeof(**data) != MP_EXT)
> + return NULL;
> +
> + int8_t type;
> + uint32_t len = mp_decode_extl(data, &type);
> +
> + if (type != MP_DATETIME || len == 0) {
> + return NULL;
Please, revert data to savepoint when decoding fails.
If mp_decode_extl or datetime_unpack fail, you mustn't
modify data.
> + }
> + return datetime_unpack(data, len, date);
> +}
> +
> +char *
> +datetime_pack(char *data, const struct datetime *date)
> +{
> + data = mp_encode_Xint(data, date->secs);
> + if (date->nsec != 0 || date->offset != 0)
> + data = mp_encode_Xint(data, date->nsec);
> + if (date->offset)
> + data = mp_encode_Xint(data, date->offset);
> +
> + return data;
> +}
<stripped>
> diff --git a/src/lua/serializer.h b/src/lua/serializer.h
> index 0a0501a74..e7a240e0a 100644
> --- a/src/lua/serializer.h
> +++ b/src/lua/serializer.h
> @@ -52,6 +52,7 @@ extern "C" {
> #include <lauxlib.h>
>
> #include "trigger.h"
> +#include "lib/core/datetime.h"
> #include "lib/core/decimal.h" /* decimal_t */
> #include "lib/core/mp_extension_types.h"
> #include "lua/error.h"
> @@ -223,6 +224,7 @@ struct luaL_field {
> uint32_t size;
> decimal_t *decval;
> struct tt_uuid *uuidval;
> + struct datetime *dateval;
> };
> enum mp_type type;
> /* subtypes of MP_EXT */
> diff --git a/src/lua/utils.c b/src/lua/utils.c
> index 2c89326f3..771f6f278 100644
> --- a/src/lua/utils.c
> +++ b/src/lua/utils.c
> @@ -254,7 +254,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
> lua_pop(L, 1);
> }
>
> -
Extraneous change. Please, remove.
> /**
> * A helper to register a single type metatable.
> */
> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
> index 1ae76003b..a72ac2253 100644
> --- a/test/unit/datetime.c
> +++ b/test/unit/datetime.c
> @@ -6,6 +6,9 @@
>
> #include "unit.h"
> #include "datetime.h"
> +#include "mp_datetime.h"
> +#include "msgpuck.h"
> +#include "mp_extension_types.h"
>
> static const char sample[] = "2012-12-24T15:30Z";
>
> @@ -247,12 +250,132 @@ tostring_datetime_test(void)
> check_plan();
> }
>
>
<stripped>
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:42 ` Safin Timur via Tarantool-patches
2021-08-18 9:01 ` Serge Petrenko via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:42 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
Thanks Sergey for your feedback, below you'll see few comments and
incremental patch...
On 17.08.2021 15:16, Serge Petrenko wrote:
>
>
> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>> Serialize datetime_t as newly introduced MP_EXT type.
>> It saves 1 required integer field and upto 2 optional
>> unsigned fields in very compact fashion.
>> - secs is required field;
>> - but nsec, offset are both optional;
>>
>> * json, yaml serialization formats, lua output mode
>> supported;
>> * exported symbols for datetime messagepack size calculations
>> so they are available for usage on Lua side.
>>
>> Part of #5941
>> Part of #5946
>> ---
>> extra/exports | 5 +-
>> src/box/field_def.c | 35 +++---
>> src/box/field_def.h | 1 +
>> src/box/lua/serialize_lua.c | 7 +-
>> src/box/msgpack.c | 7 +-
>> src/box/tuple_compare.cc | 20 ++++
>> src/lib/core/CMakeLists.txt | 4 +-
>> src/lib/core/datetime.c | 9 ++
>> src/lib/core/datetime.h | 11 ++
>> src/lib/core/mp_datetime.c | 189 ++++++++++++++++++++++++++++++
>> src/lib/core/mp_datetime.h | 89 ++++++++++++++
>> src/lib/core/mp_extension_types.h | 1 +
>> src/lib/mpstream/mpstream.c | 11 ++
>> src/lib/mpstream/mpstream.h | 4 +
>> src/lua/msgpack.c | 12 ++
>> src/lua/msgpackffi.lua | 18 +++
>> src/lua/serializer.c | 4 +
>> src/lua/serializer.h | 2 +
>> src/lua/utils.c | 1 -
>> test/unit/datetime.c | 125 +++++++++++++++++++-
>> test/unit/datetime.result | 115 +++++++++++++++++-
>> third_party/lua-cjson/lua_cjson.c | 8 ++
>> third_party/lua-yaml/lyaml.cc | 6 +-
>> 23 files changed, 661 insertions(+), 23 deletions(-)
>> create mode 100644 src/lib/core/mp_datetime.c
>> create mode 100644 src/lib/core/mp_datetime.h
>>
>> diff --git a/extra/exports b/extra/exports
>> index 2437e175c..c34a5c2b5 100644
>> --- a/extra/exports
>> +++ b/extra/exports
>> @@ -151,9 +151,10 @@ csv_setopt
>> datetime_asctime
>> datetime_ctime
>> datetime_now
>> +datetime_pack
>> datetime_strftime
>> datetime_to_string
>> -decimal_unpack
>> +datetime_unpack
>
>
> decimal_unpack should stay there.
It's there, but 2 lines below :)
That was me copy-pasted decimal_unpack a few patches before, but has not
changed it to datetime_unpack. I've corrected it in this patch.
I've now corrected the original appearance, now with correct name.
>
>
>> decimal_from_string
>> decimal_unpack
^^^ it was here
>> tnt_dt_dow
>> @@ -397,6 +398,7 @@ mp_decode_uint
>> mp_encode_array
>> mp_encode_bin
>> mp_encode_bool
>> +mp_encode_datetime
>> mp_encode_decimal
>> mp_encode_double
>> mp_encode_float
>> @@ -413,6 +415,7 @@ mp_next
>> mp_next_slowpath
>> mp_parser_hint
>> mp_sizeof_array
>> +mp_sizeof_datetime
>> mp_sizeof_decimal
>> mp_sizeof_str
>> mp_sizeof_uuid
>> diff --git a/src/box/field_def.c b/src/box/field_def.c
>> index 51acb8025..2682a42ee 100644
>> --- a/src/box/field_def.c
>> +++ b/src/box/field_def.c
>> @@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
>> /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
>> /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
>> /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
>> + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is
>> supported */
>> };
>> const uint32_t field_ext_type[] = {
>> @@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
>> /* [FIELD_TYPE_INTEGER] = */ 0,
>> /* [FIELD_TYPE_BOOLEAN] = */ 0,
>> /* [FIELD_TYPE_VARBINARY] = */ 0,
>> - /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID),
>> + /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U <<
>> MP_UUID) |
>> + (1U << MP_DATETIME),
>> /* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
>> /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
>> /* [FIELD_TYPE_ARRAY] = */ 0,
>> /* [FIELD_TYPE_MAP] = */ 0,
>> + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
>> };
>> const char *field_type_strs[] = {
>> @@ -104,6 +107,7 @@ const char *field_type_strs[] = {
>> /* [FIELD_TYPE_UUID] = */ "uuid",
>> /* [FIELD_TYPE_ARRAY] = */ "array",
>> /* [FIELD_TYPE_MAP] = */ "map",
>> + /* [FIELD_TYPE_DATETIME] = */ "datetime",
>> };
>> const char *on_conflict_action_strs[] = {
>> @@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str,
>> uint32_t len)
>> * values can be stored in the j type.
>> */
>> static const bool field_type_compatibility[] = {
>> - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN
>> VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
>> -/* ANY */ true, false, false, false, false, false,
>> false, false, false, false, false, false, false,
>> -/* UNSIGNED */ true, true, false, true, false, true,
>> false, false, true, false, false, false, false,
>> -/* STRING */ true, false, true, false, false, false,
>> false, false, true, false, false, false, false,
>> -/* NUMBER */ true, false, false, true, false, false,
>> false, false, true, false, false, false, false,
>> -/* DOUBLE */ true, false, false, true, true, false,
>> false, false, true, false, false, false, false,
>> -/* INTEGER */ true, false, false, true, false, true,
>> false, false, true, false, false, false, false,
>> -/* BOOLEAN */ true, false, false, false, false, false,
>> true, false, true, false, false, false, false,
>> -/* VARBINARY*/ true, false, false, false, false, false,
>> false, true, true, false, false, false, false,
>> -/* SCALAR */ true, false, false, false, false, false,
>> false, false, true, false, false, false, false,
>> -/* DECIMAL */ true, false, false, true, false, false,
>> false, false, true, true, false, false, false,
>> -/* UUID */ true, false, false, false, false, false,
>> false, false, false, false, true, false, false,
>> -/* ARRAY */ true, false, false, false, false, false,
>> false, false, false, false, false, true, false,
>> -/* MAP */ true, false, false, false, false, false,
>> false, false, false, false, false, false, true,
>> + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN
>> VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
>> +/* ANY */ true, false, false, false, false, false,
>> false, false, false, false, false, false, false, false,
>> +/* UNSIGNED */ true, true, false, true, false, true,
>> false, false, true, false, false, false, false, false,
>> +/* STRING */ true, false, true, false, false, false,
>> false, false, true, false, false, false, false, false,
>> +/* NUMBER */ true, false, false, true, false, false,
>> false, false, true, false, false, false, false, false,
>> +/* DOUBLE */ true, false, false, true, true, false,
>> false, false, true, false, false, false, false, false,
>> +/* INTEGER */ true, false, false, true, false, true,
>> false, false, true, false, false, false, false, false,
>> +/* BOOLEAN */ true, false, false, false, false, false,
>> true, false, true, false, false, false, false, false,
>> +/* VARBINARY*/ true, false, false, false, false, false,
>> false, true, true, false, false, false, false, false,
>> +/* SCALAR */ true, false, false, false, false, false,
>> false, false, true, false, false, false, false, false,
>> +/* DECIMAL */ true, false, false, true, false, false,
>> false, false, true, true, false, false, false, false,
>> +/* UUID */ true, false, false, false, false, false,
>> false, false, false, false, true, false, false, false,
>> +/* ARRAY */ true, false, false, false, false, false,
>> false, false, false, false, false, true, false, false,
>> +/* MAP */ true, false, false, false, false, false,
>> false, false, false, false, false, false, true, false,
>> +/* DATETIME */ true, false, false, false, false, false,
>> false, false, true, false, false, false, false, true,
>> };
>> bool
>> diff --git a/src/box/field_def.h b/src/box/field_def.h
>> index c5cfe5e86..120b2a93d 100644
>> --- a/src/box/field_def.h
>> +++ b/src/box/field_def.h
>> @@ -63,6 +63,7 @@ enum field_type {
>> FIELD_TYPE_UUID,
>> FIELD_TYPE_ARRAY,
>> FIELD_TYPE_MAP,
>> + FIELD_TYPE_DATETIME,
>> field_type_MAX
>> };
>
>
> Please, define FIELD_TYPE_DATETIME higher.
> Right after FIELD_TYPE_UUID.
>
> This way you won't need to rework field type allowed in index check
> in the next commit.
That's very straighforward and easy, my bad that I've overcomplicated it!
But I'll move the change to the next patch, as it'scorrectly has pointed
out by Vova, should be part of indices support,
>
>
>> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
>> index 1f791980f..51855011b 100644
>> --- a/src/box/lua/serialize_lua.c
>> +++ b/src/box/lua/serialize_lua.c
>> @@ -768,7 +768,7 @@ static int
>> dump_node(struct lua_dumper *d, struct node *nd, int indent)
>> {
>> struct luaL_field *field = &nd->field;
>> - char buf[FPCONV_G_FMT_BUFSIZE];
>> + char buf[FPCONV_G_FMT_BUFSIZE + 8];
>
>
> Why "+8"?
Well, because current FPCONV_G_FMT_BUFSIZE (32) was not enough for full
ISO-8601 literal with nanoseconds :)
Probably I should introduce some newer constant...
[Or, as Vova has suggested - just to use MAX from those 2 values, my
length and FPCONV_G_FMT_BUFSIZE.]
--------------------------------------------
diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
index 51855011b..eef3a4995 100644
--- a/src/box/lua/serialize_lua.c
+++ b/src/box/lua/serialize_lua.c
@@ -768,7 +768,7 @@ static int
dump_node(struct lua_dumper *d, struct node *nd, int indent)
{
struct luaL_field *field = &nd->field;
- char buf[FPCONV_G_FMT_BUFSIZE + 8];
+ char buf[MAX(FPCONV_G_FMT_BUFSIZE, DT_TO_STRING_BUFSIZE)];
int ltype = lua_type(d->L, -1);
const char *str = NULL;
size_t len = 0;
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index 497cd9f14..b8d179600 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -87,6 +87,11 @@ struct datetime_interval {
int
datetime_compare(const struct datetime *lhs, const struct datetime *rhs);
+/**
+ * Required size of datetime_to_string string buffer
+ */
+#define DT_TO_STRING_BUFSIZE 48
+
/**
* Convert datetime to string using default format
* @param date source datetime value
--------------------------------------------
>
>
>> int ltype = lua_type(d->L, -1);
>> const char *str = NULL;
>> size_t len = 0;
>
>
> <stripped>
>
>
>> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
>> new file mode 100644
>> index 000000000..d0a3e562c
>> --- /dev/null
>> +++ b/src/lib/core/mp_datetime.c
>> @@ -0,0 +1,189 @@
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>
> Same about the license.
> Please, replace that with
>
> /*
> * SPDX-License-Identifier: BSD-2-Clause
> *
> * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> And do the same for all new files.
Updated.
>
>> +#include "mp_datetime.h"
>> +#include "msgpuck.h"
>> +#include "mp_extension_types.h"
>> +
>> +/*
>> + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1
>> byte length)
>> + extension, which creates container of 1 to 3 integers.
>> +
>> +
>> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
>>
>> + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset
>> (uint) |
>> +
>> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
>>
>
> The order should be 0xC7, len(uint8), 4, seconds, ...
> according to
> https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family
Indeed, that was my misconception, thanks for correction!
[Updated picture in the patch and in the discussion -
https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990]
>
>> +
>> + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
>> +
>> + - signed integer seconds part (required). Depending on the value of
>> + seconds it may be from 1 to 8 bytes positive or negative integer
>> number;
>> +
>> + - [optional] fraction time in nanoseconds as unsigned integer.
>> + If this value is 0 then it's not saved (unless there is offset
>> field,
>> + as below);
>> +
>> + - [optional] timzeone offset in minutes as unsigned integer.
>> + If this field is 0 then it's not saved.
>> + */
>> +
>> +static inline uint32_t
>> +mp_sizeof_Xint(int64_t n)
>> +{
>> + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
>> +}
>> +
>> +static inline char *
>> +mp_encode_Xint(char *data, int64_t v)
>> +{
>> + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
>> +}
>> +
>> +static inline int64_t
>> +mp_decode_Xint(const char **data)
>> +{
>> + switch (mp_typeof(**data)) {
>> + case MP_UINT:
>> + return (int64_t)mp_decode_uint(data);
>> + case MP_INT:
>> + return mp_decode_int(data);
>> + default:
>> + mp_unreachable();
>> + }
>> + return 0;
>> +}
>
> I believe mp_decode_Xint and mp_encode_Xint
> belong to a more generic file, but I couldn't find an
> appropriate one. Up to you.
Yup, it was planned to be placed to more generic place once it would
become useful at least the 2nd time. And this time is actually 2nd (1st
was in SQL AST parser branch here
https://github.com/tarantool/tarantool/commit/55a4182ebfbed1a3c916fb7e326f8f7861776a7f#diff-e3f5bdfa58bcaed35b89f22e94be7ad472a6b37d656a129722ea0d5609503c6aR132-R143).
But that patchset has not yet landed to the master, so once again code
usage is 1st time and worth only local application. When I'll return to
distributed-sql AST parser I'll reshake them and put elsewhere.
>
>> +
>> +static inline uint32_t
>> +mp_sizeof_datetime_raw(const struct datetime *date)
>> +{
>> + uint32_t sz = mp_sizeof_Xint(date->secs);
>> +
>> + // even if nanosecs == 0 we need to output anything
>> + // if we have non-null tz offset
>
>
> Please, stick with our comment format:
Oh, yup, that slipt thru. Corrected.
>
> /*
> * Even if nanosecs == 0 we need to output anything
> * if we have non-null tz offset
> */
>
>
>> + if (date->nsec != 0 || date->offset != 0)
>> + sz += mp_sizeof_Xint(date->nsec);
>> + if (date->offset)
>> + sz += mp_sizeof_Xint(date->offset);
>> + return sz;
>> +}
>> +
>> +uint32_t
>> +mp_sizeof_datetime(const struct datetime *date)
>> +{
>> + return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
>> +}
>> +
>> +struct datetime *
>> +datetime_unpack(const char **data, uint32_t len, struct datetime *date)
>> +{
>> + const char * svp = *data;
>> +
>> + memset(date, 0, sizeof(*date));
>> +
>> + date->secs = mp_decode_Xint(data);
>
>
> Please, leave a comment about date->secs possible range here.
> Why is it ok to store a decoded int64_t in a double.
Yes, that's reasonable complain. I'll document dt supported range in the
datetime.h header, and to declare legal bounds there, so we could use
them later in asserts.
Please see incremental patch for this step below...
>
>
>> +
>> + len -= *data - svp;
>> + if (len <= 0)
>> + return date;
>> +
>> + svp = *data;
>> + date->nsec = mp_decode_Xint(data);
>> + len -= *data - svp;
>> +
>> + if (len <= 0)
>> + return date;
>> +
>> + date->offset = mp_decode_Xint(data);
>> +
>> + return date;
>> +}
>> +
>> +struct datetime *
>> +mp_decode_datetime(const char **data, struct datetime *date)
>> +{
>> + if (mp_typeof(**data) != MP_EXT)
>> + return NULL;
>> +
>> + int8_t type;
>> + uint32_t len = mp_decode_extl(data, &type);
>> +
>> + if (type != MP_DATETIME || len == 0) {
>> + return NULL;
>
>
> Please, revert data to savepoint when decoding fails.
> If mp_decode_extl or datetime_unpack fail, you mustn't
> modify data.
>
Didn't think about this case - will make sure data points to the
original location if fails.
>
>> + }
>> + return datetime_unpack(data, len, date);
>> +}
>> +
>> +char *
>> +datetime_pack(char *data, const struct datetime *date)
>> +{
>> + data = mp_encode_Xint(data, date->secs);
>> + if (date->nsec != 0 || date->offset != 0)
>> + data = mp_encode_Xint(data, date->nsec);
>> + if (date->offset)
>> + data = mp_encode_Xint(data, date->offset);
>> +
>> + return data;
>> +}
>
>
> <stripped>
>
>
>> diff --git a/src/lua/serializer.h b/src/lua/serializer.h
>> index 0a0501a74..e7a240e0a 100644
>> --- a/src/lua/serializer.h
>> +++ b/src/lua/serializer.h
>> @@ -52,6 +52,7 @@ extern "C" {
>> #include <lauxlib.h>
>> #include "trigger.h"
>> +#include "lib/core/datetime.h"
>> #include "lib/core/decimal.h" /* decimal_t */
>> #include "lib/core/mp_extension_types.h"
>> #include "lua/error.h"
>> @@ -223,6 +224,7 @@ struct luaL_field {
>> uint32_t size;
>> decimal_t *decval;
>> struct tt_uuid *uuidval;
>> + struct datetime *dateval;
>> };
>> enum mp_type type;
>> /* subtypes of MP_EXT */
>> diff --git a/src/lua/utils.c b/src/lua/utils.c
>> index 2c89326f3..771f6f278 100644
>> --- a/src/lua/utils.c
>> +++ b/src/lua/utils.c
>> @@ -254,7 +254,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
>> lua_pop(L, 1);
>> }
>> -
>
>
> Extraneous change. Please, remove.
Removed from the patch. Thanks!
>
>
>> /**
>> * A helper to register a single type metatable.
>> */
>> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
>> index 1ae76003b..a72ac2253 100644
>> --- a/test/unit/datetime.c
>> +++ b/test/unit/datetime.c
>> @@ -6,6 +6,9 @@
>> #include "unit.h"
>> #include "datetime.h"
>> +#include "mp_datetime.h"
>> +#include "msgpuck.h"
>> +#include "mp_extension_types.h"
>> static const char sample[] = "2012-12-24T15:30Z";
>> @@ -247,12 +250,132 @@ tostring_datetime_test(void)
>> check_plan();
>> }
>>
>
>
> <stripped>
>
-----------------------------------------------------
diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
index f98f7010d..df3c1c83d 100644
--- a/src/lib/core/datetime.h
+++ b/src/lib/core/datetime.h
@@ -5,6 +5,7 @@
* Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
+#include <limits.h>
#include <stdint.h>
#include <stdbool.h>
#include "c-dt/dt.h"
@@ -30,6 +31,26 @@ extern "C"
#define DT_EPOCH_1970_OFFSET 719163
#endif
+/**
+ * c-dt library uses int as type for dt value, which
+ * represents the number of days since Rata Die date.
+ * This implies limits to the number of seconds we
+ * could safely store in our structures and then safely
+ * pass to c-dt functions.
+ *
+ * So supported ranges will be
+ * - for seconds [-185604722870400 .. 185480451417600]
+ * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
+ */
+#define MAX_DT_DAY_VALUE (int64_t)INT_MAX
+#define MIN_DT_DAY_VALUE (int64_t)INT_MIN
+#define SECS_EPOCH_1970_OFFSET \
+ ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
+#define MAX_EPOCH_SECS_VALUE \
+ (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
+#define MIN_EPOCH_SECS_VALUE \
+ (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
+
/**
* datetime structure keeps number of seconds since
* Unix Epoch.
diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
index 7e475d5f1..963752c23 100644
--- a/src/lib/core/mp_datetime.c
+++ b/src/lib/core/mp_datetime.c
@@ -1,34 +1,12 @@
/*
- * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * 1. Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the
- * following disclaimer.
- *
- * 2. Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
+ * SPDX-License-Identifier: BSD-2-Clause
*
- * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
- * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
- * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
- * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
+#include <limits.h>
+#include <assert.h>
+
#include "mp_datetime.h"
#include "msgpuck.h"
#include "mp_extension_types.h"
@@ -37,9 +15,9 @@
Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte
length)
extension, which creates container of 1 to 3 integers.
-
+----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
- |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset
(uint) |
-
+----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+
+----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
+ |0xC7|len (uint8)| 4 | seconds (int) | nanoseconds (uint) | offset
(int) |
+
+----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
@@ -50,7 +28,7 @@
If this value is 0 then it's not saved (unless there is offset field,
as below);
- - [optional] timzeone offset in minutes as unsigned integer.
+ - [optional] timezone offset in minutes as signed integer.
If this field is 0 then it's not saved.
*/
@@ -80,17 +58,34 @@ mp_decode_Xint(const char **data)
return 0;
}
+#define check_secs(secs) \
+ assert((int64_t)(secs) <= MAX_EPOCH_SECS_VALUE);\
+ assert((int64_t)(secs) >= MIN_EPOCH_SECS_VALUE);
+
+#define check_nanosecs(nsec) assert((nsec) < 1000000000);
+
+#define check_tz_offset(offset) \
+ assert((offset) <= (12 * 60));\
+ assert((offset) >= (-12 * 60));
+
static inline uint32_t
mp_sizeof_datetime_raw(const struct datetime *date)
{
+ check_secs(date->secs);
uint32_t sz = mp_sizeof_Xint(date->secs);
- // even if nanosecs == 0 we need to output anything
- // if we have non-null tz offset
- if (date->nsec != 0 || date->offset != 0)
+ /*
+ * even if nanosecs == 0 we need to output something
+ * if we have a non-null tz offset
+ */
+ if (date->nsec != 0 || date->offset != 0) {
+ check_nanosecs(date->nsec);
sz += mp_sizeof_Xint(date->nsec);
- if (date->offset)
+ }
+ if (date->offset) {
+ check_tz_offset(date->offset);
sz += mp_sizeof_Xint(date->offset);
+ }
return sz;
}
@@ -103,24 +98,30 @@ mp_sizeof_datetime(const struct datetime *date)
struct datetime *
datetime_unpack(const char **data, uint32_t len, struct datetime *date)
{
- const char * svp = *data;
+ const char *svp = *data;
memset(date, 0, sizeof(*date));
- date->secs = mp_decode_Xint(data);
+ int64_t seconds = mp_decode_Xint(data);
+ check_secs(seconds);
+ date->secs = seconds;
len -= *data - svp;
if (len <= 0)
return date;
svp = *data;
- date->nsec = mp_decode_Xint(data);
+ uint64_t nanoseconds = mp_decode_uint(data);
+ check_nanosecs(nanoseconds);
+ date->nsec = nanoseconds;
len -= *data - svp;
if (len <= 0)
return date;
- date->offset = mp_decode_Xint(data);
+ int64_t offset = mp_decode_Xint(data);
+ check_tz_offset(offset);
+ date->offset = offset;
return date;
}
@@ -131,10 +132,12 @@ mp_decode_datetime(const char **data, struct
datetime *date)
if (mp_typeof(**data) != MP_EXT)
return NULL;
+ const char *svp = *data;
int8_t type;
uint32_t len = mp_decode_extl(data, &type);
if (type != MP_DATETIME || len == 0) {
+ *data = svp;
return NULL;
}
return datetime_unpack(data, len, date);
@@ -145,7 +148,7 @@ datetime_pack(char *data, const struct datetime *date)
{
data = mp_encode_Xint(data, date->secs);
if (date->nsec != 0 || date->offset != 0)
- data = mp_encode_Xint(data, date->nsec);
+ data = mp_encode_uint(data, date->nsec);
if (date->offset)
data = mp_encode_Xint(data, date->offset);
@@ -165,7 +168,9 @@ mp_encode_datetime(char *data, const struct datetime
*date)
int
mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len)
{
- struct datetime date = {0, 0, 0};
+ struct datetime date = {
+ .secs = 0, .nsec = 0, .offset = 0
+ };
if (datetime_unpack(data, len, &date) == NULL)
return -1;
@@ -176,7 +181,9 @@ mp_snprint_datetime(char *buf, int size, const char
**data, uint32_t len)
int
mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
{
- struct datetime date = {0, 0, 0};
+ struct datetime date = {
+ .secs = 0, .nsec = 0, .offset = 0
+ };
if (datetime_unpack(data, len, &date) == NULL)
return -1;
diff --git a/src/lib/core/mp_datetime.h b/src/lib/core/mp_datetime.h
index 9a4d2720c..92e94a243 100644
--- a/src/lib/core/mp_datetime.h
+++ b/src/lib/core/mp_datetime.h
@@ -1,33 +1,8 @@
#pragma once
/*
- * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * 1. Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the
- * following disclaimer.
+ * SPDX-License-Identifier: BSD-2-Clause
*
- * 2. Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
- * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
- * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
- * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
- * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
- * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
- * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
- * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
- * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
- * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
+ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
*/
#include <stdio.h>
-----------------------------------------------------
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-17 23:42 ` Safin Timur via Tarantool-patches
@ 2021-08-18 9:01 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-18 9:01 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
18.08.2021 02:42, Safin Timur пишет:
> Thanks Sergey for your feedback, below you'll see few comments and
> incremental patch...
>
> On 17.08.2021 15:16, Serge Petrenko wrote:
>>
>>
>> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>>> Serialize datetime_t as newly introduced MP_EXT type.
>>> It saves 1 required integer field and upto 2 optional
>>> unsigned fields in very compact fashion.
>>> - secs is required field;
>>> - but nsec, offset are both optional;
>>>
>>> * json, yaml serialization formats, lua output mode
>>> supported;
>>> * exported symbols for datetime messagepack size calculations
>>> so they are available for usage on Lua side.
>>>
>>> Part of #5941
>>> Part of #5946
>>> ---
>>> extra/exports | 5 +-
>>> src/box/field_def.c | 35 +++---
>>> src/box/field_def.h | 1 +
>>> src/box/lua/serialize_lua.c | 7 +-
>>> src/box/msgpack.c | 7 +-
>>> src/box/tuple_compare.cc | 20 ++++
>>> src/lib/core/CMakeLists.txt | 4 +-
>>> src/lib/core/datetime.c | 9 ++
>>> src/lib/core/datetime.h | 11 ++
>>> src/lib/core/mp_datetime.c | 189
>>> ++++++++++++++++++++++++++++++
>>> src/lib/core/mp_datetime.h | 89 ++++++++++++++
>>> src/lib/core/mp_extension_types.h | 1 +
>>> src/lib/mpstream/mpstream.c | 11 ++
>>> src/lib/mpstream/mpstream.h | 4 +
>>> src/lua/msgpack.c | 12 ++
>>> src/lua/msgpackffi.lua | 18 +++
>>> src/lua/serializer.c | 4 +
>>> src/lua/serializer.h | 2 +
>>> src/lua/utils.c | 1 -
>>> test/unit/datetime.c | 125 +++++++++++++++++++-
>>> test/unit/datetime.result | 115 +++++++++++++++++-
>>> third_party/lua-cjson/lua_cjson.c | 8 ++
>>> third_party/lua-yaml/lyaml.cc | 6 +-
>>> 23 files changed, 661 insertions(+), 23 deletions(-)
>>> create mode 100644 src/lib/core/mp_datetime.c
>>> create mode 100644 src/lib/core/mp_datetime.h
>>>
>>> diff --git a/extra/exports b/extra/exports
>>> index 2437e175c..c34a5c2b5 100644
>>> --- a/extra/exports
>>> +++ b/extra/exports
>>> @@ -151,9 +151,10 @@ csv_setopt
>>> datetime_asctime
>>> datetime_ctime
>>> datetime_now
>>> +datetime_pack
>>> datetime_strftime
>>> datetime_to_string
>>> -decimal_unpack
>>> +datetime_unpack
>>
>>
>> decimal_unpack should stay there.
>
> It's there, but 2 lines below :)
>
> That was me copy-pasted decimal_unpack a few patches before, but has
> not changed it to datetime_unpack. I've corrected it in this patch.
>
> I've now corrected the original appearance, now with correct name.
Ok, thanks!
>
>>
>>
>>> decimal_from_string
>>> decimal_unpack
>
> ^^^ it was here
>
>>> tnt_dt_dow
>>> @@ -397,6 +398,7 @@ mp_decode_uint
>>> mp_encode_array
>>> mp_encode_bin
>>> mp_encode_bool
>>> +mp_encode_datetime
>>> mp_encode_decimal
>>> mp_encode_double
>>> mp_encode_float
>>> @@ -413,6 +415,7 @@ mp_next
>>> mp_next_slowpath
>>> mp_parser_hint
>>> mp_sizeof_array
>>> +mp_sizeof_datetime
>>> mp_sizeof_decimal
>>> mp_sizeof_str
>>> mp_sizeof_uuid
>>> diff --git a/src/box/field_def.c b/src/box/field_def.c
>>> index 51acb8025..2682a42ee 100644
>>> --- a/src/box/field_def.c
>>> +++ b/src/box/field_def.c
>>> @@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
>>> /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
>>> /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
>>> /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
>>> + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is
>>> supported */
>>> };
>>> const uint32_t field_ext_type[] = {
>>> @@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
>>> /* [FIELD_TYPE_INTEGER] = */ 0,
>>> /* [FIELD_TYPE_BOOLEAN] = */ 0,
>>> /* [FIELD_TYPE_VARBINARY] = */ 0,
>>> - /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U <<
>>> MP_UUID),
>>> + /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U <<
>>> MP_UUID) |
>>> + (1U << MP_DATETIME),
>>> /* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
>>> /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
>>> /* [FIELD_TYPE_ARRAY] = */ 0,
>>> /* [FIELD_TYPE_MAP] = */ 0,
>>> + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
>>> };
>>> const char *field_type_strs[] = {
>>> @@ -104,6 +107,7 @@ const char *field_type_strs[] = {
>>> /* [FIELD_TYPE_UUID] = */ "uuid",
>>> /* [FIELD_TYPE_ARRAY] = */ "array",
>>> /* [FIELD_TYPE_MAP] = */ "map",
>>> + /* [FIELD_TYPE_DATETIME] = */ "datetime",
>>> };
>>> const char *on_conflict_action_strs[] = {
>>> @@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str,
>>> uint32_t len)
>>> * values can be stored in the j type.
>>> */
>>> static const bool field_type_compatibility[] = {
>>> - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER
>>> BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
>>> -/* ANY */ true, false, false, false, false, false,
>>> false, false, false, false, false, false, false,
>>> -/* UNSIGNED */ true, true, false, true, false, true,
>>> false, false, true, false, false, false, false,
>>> -/* STRING */ true, false, true, false, false, false,
>>> false, false, true, false, false, false, false,
>>> -/* NUMBER */ true, false, false, true, false, false,
>>> false, false, true, false, false, false, false,
>>> -/* DOUBLE */ true, false, false, true, true, false,
>>> false, false, true, false, false, false, false,
>>> -/* INTEGER */ true, false, false, true, false, true,
>>> false, false, true, false, false, false, false,
>>> -/* BOOLEAN */ true, false, false, false, false, false,
>>> true, false, true, false, false, false, false,
>>> -/* VARBINARY*/ true, false, false, false, false, false,
>>> false, true, true, false, false, false, false,
>>> -/* SCALAR */ true, false, false, false, false, false,
>>> false, false, true, false, false, false, false,
>>> -/* DECIMAL */ true, false, false, true, false, false,
>>> false, false, true, true, false, false, false,
>>> -/* UUID */ true, false, false, false, false, false,
>>> false, false, false, false, true, false, false,
>>> -/* ARRAY */ true, false, false, false, false, false,
>>> false, false, false, false, false, true, false,
>>> -/* MAP */ true, false, false, false, false, false,
>>> false, false, false, false, false, false, true,
>>> + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER
>>> BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
>>> +/* ANY */ true, false, false, false, false, false,
>>> false, false, false, false, false, false, false, false,
>>> +/* UNSIGNED */ true, true, false, true, false, true,
>>> false, false, true, false, false, false, false, false,
>>> +/* STRING */ true, false, true, false, false, false,
>>> false, false, true, false, false, false, false, false,
>>> +/* NUMBER */ true, false, false, true, false, false,
>>> false, false, true, false, false, false, false, false,
>>> +/* DOUBLE */ true, false, false, true, true, false,
>>> false, false, true, false, false, false, false, false,
>>> +/* INTEGER */ true, false, false, true, false, true,
>>> false, false, true, false, false, false, false, false,
>>> +/* BOOLEAN */ true, false, false, false, false, false,
>>> true, false, true, false, false, false, false, false,
>>> +/* VARBINARY*/ true, false, false, false, false, false,
>>> false, true, true, false, false, false, false, false,
>>> +/* SCALAR */ true, false, false, false, false, false,
>>> false, false, true, false, false, false, false, false,
>>> +/* DECIMAL */ true, false, false, true, false, false,
>>> false, false, true, true, false, false, false, false,
>>> +/* UUID */ true, false, false, false, false, false,
>>> false, false, false, false, true, false, false, false,
>>> +/* ARRAY */ true, false, false, false, false, false,
>>> false, false, false, false, false, true, false, false,
>>> +/* MAP */ true, false, false, false, false, false,
>>> false, false, false, false, false, false, true, false,
>>> +/* DATETIME */ true, false, false, false, false, false,
>>> false, false, true, false, false, false, false, true,
>>> };
>>> bool
>>> diff --git a/src/box/field_def.h b/src/box/field_def.h
>>> index c5cfe5e86..120b2a93d 100644
>>> --- a/src/box/field_def.h
>>> +++ b/src/box/field_def.h
>>> @@ -63,6 +63,7 @@ enum field_type {
>>> FIELD_TYPE_UUID,
>>> FIELD_TYPE_ARRAY,
>>> FIELD_TYPE_MAP,
>>> + FIELD_TYPE_DATETIME,
>>> field_type_MAX
>>> };
>>
>>
>> Please, define FIELD_TYPE_DATETIME higher.
>> Right after FIELD_TYPE_UUID.
>>
>> This way you won't need to rework field type allowed in index check
>> in the next commit.
>
> That's very straighforward and easy, my bad that I've overcomplicated it!
>
> But I'll move the change to the next patch, as it'scorrectly has
> pointed out by Vova, should be part of indices support,
>
Ok.
>
>>
>>
>>> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
>>> index 1f791980f..51855011b 100644
>>> --- a/src/box/lua/serialize_lua.c
>>> +++ b/src/box/lua/serialize_lua.c
>>> @@ -768,7 +768,7 @@ static int
>>> dump_node(struct lua_dumper *d, struct node *nd, int indent)
>>> {
>>> struct luaL_field *field = &nd->field;
>>> - char buf[FPCONV_G_FMT_BUFSIZE];
>>> + char buf[FPCONV_G_FMT_BUFSIZE + 8];
>>
>>
>> Why "+8"?
>
> Well, because current FPCONV_G_FMT_BUFSIZE (32) was not enough for
> full ISO-8601 literal with nanoseconds :)
>
> Probably I should introduce some newer constant...
>
> [Or, as Vova has suggested - just to use MAX from those 2 values, my
> length and FPCONV_G_FMT_BUFSIZE.]
>
> --------------------------------------------
> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
> index 51855011b..eef3a4995 100644
> --- a/src/box/lua/serialize_lua.c
> +++ b/src/box/lua/serialize_lua.c
> @@ -768,7 +768,7 @@ static int
> dump_node(struct lua_dumper *d, struct node *nd, int indent)
> {
> struct luaL_field *field = &nd->field;
> - char buf[FPCONV_G_FMT_BUFSIZE + 8];
> + char buf[MAX(FPCONV_G_FMT_BUFSIZE, DT_TO_STRING_BUFSIZE)];
> int ltype = lua_type(d->L, -1);
> const char *str = NULL;
> size_t len = 0;
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 497cd9f14..b8d179600 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -87,6 +87,11 @@ struct datetime_interval {
> int
> datetime_compare(const struct datetime *lhs, const struct datetime
> *rhs);
>
> +/**
> + * Required size of datetime_to_string string buffer
> + */
> +#define DT_TO_STRING_BUFSIZE 48
> +
> /**
> * Convert datetime to string using default format
> * @param date source datetime value
> --------------------------------------------
>
Looks good.
>
>>
>>
>>> int ltype = lua_type(d->L, -1);
>>> const char *str = NULL;
>>> size_t len = 0;
>>
>>
>> <stripped>
>>
>>
>>> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
>>> new file mode 100644
>>> index 000000000..d0a3e562c
>>> --- /dev/null
>>> +++ b/src/lib/core/mp_datetime.c
>>> @@ -0,0 +1,189 @@
>>> +/*
>>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>>> + *
>>> + * Redistribution and use in source and binary forms, with or
>>> + * without modification, are permitted provided that the following
>>> + * conditions are met:
>>> + *
>>> + * 1. Redistributions of source code must retain the above
>>> + * copyright notice, this list of conditions and the
>>> + * following disclaimer.
>>> + *
>>> + * 2. Redistributions in binary form must reproduce the above
>>> + * copyright notice, this list of conditions and the following
>>> + * disclaimer in the documentation and/or other materials
>>> + * provided with the distribution.
>>> + *
>>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>>> + * SUCH DAMAGE.
>>> + */
>>> +
>>
>> Same about the license.
>> Please, replace that with
>>
>> /*
>> * SPDX-License-Identifier: BSD-2-Clause
>> *
>> * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> */
>>
>> And do the same for all new files.
>
> Updated.
>
>>
>>> +#include "mp_datetime.h"
>>> +#include "msgpuck.h"
>>> +#include "mp_extension_types.h"
>>> +
>>> +/*
>>> + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1
>>> byte length)
>>> + extension, which creates container of 1 to 3 integers.
>>> +
>>> +
>>> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
>>>
>>> + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) |
>>> offset (uint) |
>>> +
>>> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
>>>
>>
>> The order should be 0xC7, len(uint8), 4, seconds, ...
>> according to
>> https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family
>
> Indeed, that was my misconception, thanks for correction!
> [Updated picture in the patch and in the discussion -
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990]
>
>>
>>> +
>>> + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
>>> +
>>> + - signed integer seconds part (required). Depending on the value of
>>> + seconds it may be from 1 to 8 bytes positive or negative
>>> integer number;
>>> +
>>> + - [optional] fraction time in nanoseconds as unsigned integer.
>>> + If this value is 0 then it's not saved (unless there is offset
>>> field,
>>> + as below);
>>> +
>>> + - [optional] timzeone offset in minutes as unsigned integer.
>>> + If this field is 0 then it's not saved.
>>> + */
>>> +
>>> +static inline uint32_t
>>> +mp_sizeof_Xint(int64_t n)
>>> +{
>>> + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
>>> +}
>>> +
>>> +static inline char *
>>> +mp_encode_Xint(char *data, int64_t v)
>>> +{
>>> + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
>>> +}
>>> +
>>> +static inline int64_t
>>> +mp_decode_Xint(const char **data)
>>> +{
>>> + switch (mp_typeof(**data)) {
>>> + case MP_UINT:
>>> + return (int64_t)mp_decode_uint(data);
>>> + case MP_INT:
>>> + return mp_decode_int(data);
>>> + default:
>>> + mp_unreachable();
>>> + }
>>> + return 0;
>>> +}
>>
>> I believe mp_decode_Xint and mp_encode_Xint
>> belong to a more generic file, but I couldn't find an
>> appropriate one. Up to you.
>
> Yup, it was planned to be placed to more generic place once it would
> become useful at least the 2nd time. And this time is actually 2nd
> (1st was in SQL AST parser branch here
> https://github.com/tarantool/tarantool/commit/55a4182ebfbed1a3c916fb7e326f8f7861776a7f#diff-e3f5bdfa58bcaed35b89f22e94be7ad472a6b37d656a129722ea0d5609503c6aR132-R143).
> But that patchset has not yet landed to the master, so once again code
> usage is 1st time and worth only local application. When I'll return
> to distributed-sql AST parser I'll reshake them and put elsewhere.
>
>
>>
>>> +
>>> +static inline uint32_t
>>> +mp_sizeof_datetime_raw(const struct datetime *date)
>>> +{
>>> + uint32_t sz = mp_sizeof_Xint(date->secs);
>>> +
>>> + // even if nanosecs == 0 we need to output anything
>>> + // if we have non-null tz offset
>>
>>
>> Please, stick with our comment format:
>
> Oh, yup, that slipt thru. Corrected.
>
>>
>> /*
>> * Even if nanosecs == 0 we need to output anything
>> * if we have non-null tz offset
>> */
>>
>>
>>> + if (date->nsec != 0 || date->offset != 0)
>>> + sz += mp_sizeof_Xint(date->nsec);
>>> + if (date->offset)
>>> + sz += mp_sizeof_Xint(date->offset);
>>> + return sz;
>>> +}
>>> +
>>> +uint32_t
>>> +mp_sizeof_datetime(const struct datetime *date)
>>> +{
>>> + return mp_sizeof_ext(mp_sizeof_datetime_raw(date));
>>> +}
>>> +
>>> +struct datetime *
>>> +datetime_unpack(const char **data, uint32_t len, struct datetime
>>> *date)
>>> +{
>>> + const char * svp = *data;
>>> +
>>> + memset(date, 0, sizeof(*date));
>>> +
>>> + date->secs = mp_decode_Xint(data);
>>
>>
>> Please, leave a comment about date->secs possible range here.
>> Why is it ok to store a decoded int64_t in a double.
>
> Yes, that's reasonable complain. I'll document dt supported range in
> the datetime.h header, and to declare legal bounds there, so we could
> use them later in asserts.
>
> Please see incremental patch for this step below...
>
>>
>>
>>> +
>>> + len -= *data - svp;
>>> + if (len <= 0)
>>> + return date;
>>> +
>>> + svp = *data;
>>> + date->nsec = mp_decode_Xint(data);
>>> + len -= *data - svp;
>>> +
>>> + if (len <= 0)
>>> + return date;
>>> +
>>> + date->offset = mp_decode_Xint(data);
>>> +
>>> + return date;
>>> +}
>>> +
>>> +struct datetime *
>>> +mp_decode_datetime(const char **data, struct datetime *date)
>>> +{
>>> + if (mp_typeof(**data) != MP_EXT)
>>> + return NULL;
>>> +
>>> + int8_t type;
>>> + uint32_t len = mp_decode_extl(data, &type);
>>> +
>>> + if (type != MP_DATETIME || len == 0) {
>>> + return NULL;
>>
>>
>> Please, revert data to savepoint when decoding fails.
>> If mp_decode_extl or datetime_unpack fail, you mustn't
>> modify data.
>>
>
> Didn't think about this case - will make sure data points to the
> original location if fails.
>
>>
>>> + }
>>> + return datetime_unpack(data, len, date);
>>> +}
>>> +
>>> +char *
>>> +datetime_pack(char *data, const struct datetime *date)
>>> +{
>>> + data = mp_encode_Xint(data, date->secs);
>>> + if (date->nsec != 0 || date->offset != 0)
>>> + data = mp_encode_Xint(data, date->nsec);
>>> + if (date->offset)
>>> + data = mp_encode_Xint(data, date->offset);
>>> +
>>> + return data;
>>> +}
>>
>>
>> <stripped>
>>
>>
>>> diff --git a/src/lua/serializer.h b/src/lua/serializer.h
>>> index 0a0501a74..e7a240e0a 100644
>>> --- a/src/lua/serializer.h
>>> +++ b/src/lua/serializer.h
>>> @@ -52,6 +52,7 @@ extern "C" {
>>> #include <lauxlib.h>
>>> #include "trigger.h"
>>> +#include "lib/core/datetime.h"
>>> #include "lib/core/decimal.h" /* decimal_t */
>>> #include "lib/core/mp_extension_types.h"
>>> #include "lua/error.h"
>>> @@ -223,6 +224,7 @@ struct luaL_field {
>>> uint32_t size;
>>> decimal_t *decval;
>>> struct tt_uuid *uuidval;
>>> + struct datetime *dateval;
>>> };
>>> enum mp_type type;
>>> /* subtypes of MP_EXT */
>>> diff --git a/src/lua/utils.c b/src/lua/utils.c
>>> index 2c89326f3..771f6f278 100644
>>> --- a/src/lua/utils.c
>>> +++ b/src/lua/utils.c
>>> @@ -254,7 +254,6 @@ luaL_setcdatagc(struct lua_State *L, int idx)
>>> lua_pop(L, 1);
>>> }
>>> -
>>
>>
>> Extraneous change. Please, remove.
>
> Removed from the patch. Thanks!
>
>>
>>
>>> /**
>>> * A helper to register a single type metatable.
>>> */
>>> diff --git a/test/unit/datetime.c b/test/unit/datetime.c
>>> index 1ae76003b..a72ac2253 100644
>>> --- a/test/unit/datetime.c
>>> +++ b/test/unit/datetime.c
>>> @@ -6,6 +6,9 @@
>>> #include "unit.h"
>>> #include "datetime.h"
>>> +#include "mp_datetime.h"
>>> +#include "msgpuck.h"
>>> +#include "mp_extension_types.h"
>>> static const char sample[] = "2012-12-24T15:30Z";
>>> @@ -247,12 +250,132 @@ tostring_datetime_test(void)
>>> check_plan();
>>> }
>>>
>>
>>
>> <stripped>
>>
>
>
> -----------------------------------------------------
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index f98f7010d..df3c1c83d 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -5,6 +5,7 @@
> * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> +#include <limits.h>
> #include <stdint.h>
> #include <stdbool.h>
> #include "c-dt/dt.h"
> @@ -30,6 +31,26 @@ extern "C"
> #define DT_EPOCH_1970_OFFSET 719163
> #endif
>
> +/**
> + * c-dt library uses int as type for dt value, which
> + * represents the number of days since Rata Die date.
> + * This implies limits to the number of seconds we
> + * could safely store in our structures and then safely
> + * pass to c-dt functions.
> + *
> + * So supported ranges will be
> + * - for seconds [-185604722870400 .. 185480451417600]
> + * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z]
> + */
> +#define MAX_DT_DAY_VALUE (int64_t)INT_MAX
> +#define MIN_DT_DAY_VALUE (int64_t)INT_MIN
> +#define SECS_EPOCH_1970_OFFSET \
> + ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
> +#define MAX_EPOCH_SECS_VALUE \
> + (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> +#define MIN_EPOCH_SECS_VALUE \
> + (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET)
> +
> /**
> * datetime structure keeps number of seconds since
> * Unix Epoch.
> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
> index 7e475d5f1..963752c23 100644
> --- a/src/lib/core/mp_datetime.c
> +++ b/src/lib/core/mp_datetime.c
> @@ -1,34 +1,12 @@
> /*
> - * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> - *
> - * Redistribution and use in source and binary forms, with or
> - * without modification, are permitted provided that the following
> - * conditions are met:
> - *
> - * 1. Redistributions of source code must retain the above
> - * copyright notice, this list of conditions and the
> - * following disclaimer.
> - *
> - * 2. Redistributions in binary form must reproduce the above
> - * copyright notice, this list of conditions and the following
> - * disclaimer in the documentation and/or other materials
> - * provided with the distribution.
> + * SPDX-License-Identifier: BSD-2-Clause
> *
> - * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> - * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> - * SUCH DAMAGE.
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> +#include <limits.h>
> +#include <assert.h>
> +
> #include "mp_datetime.h"
> #include "msgpuck.h"
> #include "mp_extension_types.h"
> @@ -37,9 +15,9 @@
> Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1
> byte length)
> extension, which creates container of 1 to 3 integers.
>
> -
> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> - |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset
> (uint) |
> -
> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> +
> +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> + |0xC7|len (uint8)| 4 | seconds (int) | nanoseconds (uint) | offset
> (int) |
> +
> +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
>
> MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
>
> @@ -50,7 +28,7 @@
> If this value is 0 then it's not saved (unless there is offset
> field,
> as below);
>
> - - [optional] timzeone offset in minutes as unsigned integer.
> + - [optional] timezone offset in minutes as signed integer.
> If this field is 0 then it's not saved.
> */
>
> @@ -80,17 +58,34 @@ mp_decode_Xint(const char **data)
> return 0;
> }
>
> +#define check_secs(secs) \
> + assert((int64_t)(secs) <= MAX_EPOCH_SECS_VALUE);\
> + assert((int64_t)(secs) >= MIN_EPOCH_SECS_VALUE);
> +
> +#define check_nanosecs(nsec) assert((nsec) < 1000000000);
> +
> +#define check_tz_offset(offset) \
> + assert((offset) <= (12 * 60));\
> + assert((offset) >= (-12 * 60));
> +
> static inline uint32_t
> mp_sizeof_datetime_raw(const struct datetime *date)
> {
> + check_secs(date->secs);
> uint32_t sz = mp_sizeof_Xint(date->secs);
>
> - // even if nanosecs == 0 we need to output anything
> - // if we have non-null tz offset
> - if (date->nsec != 0 || date->offset != 0)
> + /*
> + * even if nanosecs == 0 we need to output something
> + * if we have a non-null tz offset
> + */
> + if (date->nsec != 0 || date->offset != 0) {
> + check_nanosecs(date->nsec);
> sz += mp_sizeof_Xint(date->nsec);
> - if (date->offset)
> + }
> + if (date->offset) {
> + check_tz_offset(date->offset);
> sz += mp_sizeof_Xint(date->offset);
> + }
> return sz;
> }
>
> @@ -103,24 +98,30 @@ mp_sizeof_datetime(const struct datetime *date)
> struct datetime *
> datetime_unpack(const char **data, uint32_t len, struct datetime *date)
> {
> - const char * svp = *data;
> + const char *svp = *data;
>
> memset(date, 0, sizeof(*date));
>
> - date->secs = mp_decode_Xint(data);
> + int64_t seconds = mp_decode_Xint(data);
> + check_secs(seconds);
> + date->secs = seconds;
>
> len -= *data - svp;
> if (len <= 0)
> return date;
>
> svp = *data;
> - date->nsec = mp_decode_Xint(data);
> + uint64_t nanoseconds = mp_decode_uint(data);
> + check_nanosecs(nanoseconds);
> + date->nsec = nanoseconds;
> len -= *data - svp;
>
> if (len <= 0)
> return date;
>
> - date->offset = mp_decode_Xint(data);
> + int64_t offset = mp_decode_Xint(data);
> + check_tz_offset(offset);
> + date->offset = offset;
>
> return date;
> }
> @@ -131,10 +132,12 @@ mp_decode_datetime(const char **data, struct
> datetime *date)
> if (mp_typeof(**data) != MP_EXT)
> return NULL;
>
> + const char *svp = *data;
> int8_t type;
> uint32_t len = mp_decode_extl(data, &type);
>
> if (type != MP_DATETIME || len == 0) {
> + *data = svp;
> return NULL;
> }
> return datetime_unpack(data, len, date);
> @@ -145,7 +148,7 @@ datetime_pack(char *data, const struct datetime
> *date)
> {
> data = mp_encode_Xint(data, date->secs);
> if (date->nsec != 0 || date->offset != 0)
> - data = mp_encode_Xint(data, date->nsec);
> + data = mp_encode_uint(data, date->nsec);
> if (date->offset)
> data = mp_encode_Xint(data, date->offset);
>
> @@ -165,7 +168,9 @@ mp_encode_datetime(char *data, const struct
> datetime *date)
> int
> mp_snprint_datetime(char *buf, int size, const char **data, uint32_t
> len)
> {
> - struct datetime date = {0, 0, 0};
> + struct datetime date = {
> + .secs = 0, .nsec = 0, .offset = 0
> + };
>
> if (datetime_unpack(data, len, &date) == NULL)
> return -1;
> @@ -176,7 +181,9 @@ mp_snprint_datetime(char *buf, int size, const
> char **data, uint32_t len)
> int
> mp_fprint_datetime(FILE *file, const char **data, uint32_t len)
> {
> - struct datetime date = {0, 0, 0};
> + struct datetime date = {
> + .secs = 0, .nsec = 0, .offset = 0
> + };
>
> if (datetime_unpack(data, len, &date) == NULL)
> return -1;
> diff --git a/src/lib/core/mp_datetime.h b/src/lib/core/mp_datetime.h
> index 9a4d2720c..92e94a243 100644
> --- a/src/lib/core/mp_datetime.h
> +++ b/src/lib/core/mp_datetime.h
> @@ -1,33 +1,8 @@
> #pragma once
> /*
> - * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> - *
> - * Redistribution and use in source and binary forms, with or
> - * without modification, are permitted provided that the following
> - * conditions are met:
> - *
> - * 1. Redistributions of source code must retain the above
> - * copyright notice, this list of conditions and the
> - * following disclaimer.
> + * SPDX-License-Identifier: BSD-2-Clause
> *
> - * 2. Redistributions in binary form must reproduce the above
> - * copyright notice, this list of conditions and the following
> - * disclaimer in the documentation and/or other materials
> - * provided with the distribution.
> - *
> - * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> - * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> - * SUCH DAMAGE.
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> */
>
> #include <stdio.h>
>
> -----------------------------------------------------
>
> Thanks,
> Timur
Thanks for the changes!
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime Timur Safin via Tarantool-patches
2021-08-16 0:20 ` Safin Timur via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 18:36 ` Vladimir Davydov via Tarantool-patches
2021-08-18 14:27 ` Safin Timur via Tarantool-patches
2 siblings, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 18:36 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:38AM +0300, Timur Safin via Tarantool-patches wrote:
> Serialize datetime_t as newly introduced MP_EXT type.
> It saves 1 required integer field and upto 2 optional
> unsigned fields in very compact fashion.
> - secs is required field;
> - but nsec, offset are both optional;
>
> * json, yaml serialization formats, lua output mode
> supported;
> * exported symbols for datetime messagepack size calculations
> so they are available for usage on Lua side.
>
> Part of #5941
> Part of #5946
> ---
> extra/exports | 5 +-
> src/box/field_def.c | 35 +++---
> src/box/field_def.h | 1 +
> src/box/lua/serialize_lua.c | 7 +-
> src/box/msgpack.c | 7 +-
> src/box/tuple_compare.cc | 20 ++++
> src/lib/core/CMakeLists.txt | 4 +-
> src/lib/core/datetime.c | 9 ++
> src/lib/core/datetime.h | 11 ++
> src/lib/core/mp_datetime.c | 189 ++++++++++++++++++++++++++++++
> src/lib/core/mp_datetime.h | 89 ++++++++++++++
> src/lib/core/mp_extension_types.h | 1 +
> src/lib/mpstream/mpstream.c | 11 ++
> src/lib/mpstream/mpstream.h | 4 +
> src/lua/msgpack.c | 12 ++
> src/lua/msgpackffi.lua | 18 +++
> src/lua/serializer.c | 4 +
> src/lua/serializer.h | 2 +
> src/lua/utils.c | 1 -
> test/unit/datetime.c | 125 +++++++++++++++++++-
> test/unit/datetime.result | 115 +++++++++++++++++-
> third_party/lua-cjson/lua_cjson.c | 8 ++
> third_party/lua-yaml/lyaml.cc | 6 +-
Please add a Lua test checking that serialization (msgpack, yaml, json)
works fine. Should be added in this patch.
> 23 files changed, 661 insertions(+), 23 deletions(-)
> create mode 100644 src/lib/core/mp_datetime.c
> create mode 100644 src/lib/core/mp_datetime.h
>
> diff --git a/src/box/field_def.c b/src/box/field_def.c
> index 51acb8025..2682a42ee 100644
> --- a/src/box/field_def.c
> +++ b/src/box/field_def.c
> @@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
> /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
> /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
> /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
> + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is supported */
> };
>
> const uint32_t field_ext_type[] = {
> @@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
> /* [FIELD_TYPE_INTEGER] = */ 0,
> /* [FIELD_TYPE_BOOLEAN] = */ 0,
> /* [FIELD_TYPE_VARBINARY] = */ 0,
> - /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID),
> + /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U << MP_UUID) |
> + (1U << MP_DATETIME),
> /* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
> /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
> /* [FIELD_TYPE_ARRAY] = */ 0,
> /* [FIELD_TYPE_MAP] = */ 0,
> + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
> };
>
> const char *field_type_strs[] = {
> @@ -104,6 +107,7 @@ const char *field_type_strs[] = {
> /* [FIELD_TYPE_UUID] = */ "uuid",
> /* [FIELD_TYPE_ARRAY] = */ "array",
> /* [FIELD_TYPE_MAP] = */ "map",
> + /* [FIELD_TYPE_DATETIME] = */ "datetime",
> };
This doesn't belong to this patch. Please split in two:
- encoding datetime as msgpack/yaml/json - this patch
- indexing of datetime - next patch
>
> const char *on_conflict_action_strs[] = {
> @@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str, uint32_t len)
> * values can be stored in the j type.
> */
> static const bool field_type_compatibility[] = {
> - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
> -/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false,
> -/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false,
> -/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false,
> -/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false,
> -/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false,
> -/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false,
> -/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false,
> -/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false,
> -/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false,
> -/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false,
> -/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false,
> -/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false,
> -/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true,
> + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
> +/* ANY */ true, false, false, false, false, false, false, false, false, false, false, false, false, false,
> +/* UNSIGNED */ true, true, false, true, false, true, false, false, true, false, false, false, false, false,
> +/* STRING */ true, false, true, false, false, false, false, false, true, false, false, false, false, false,
> +/* NUMBER */ true, false, false, true, false, false, false, false, true, false, false, false, false, false,
> +/* DOUBLE */ true, false, false, true, true, false, false, false, true, false, false, false, false, false,
> +/* INTEGER */ true, false, false, true, false, true, false, false, true, false, false, false, false, false,
> +/* BOOLEAN */ true, false, false, false, false, false, true, false, true, false, false, false, false, false,
> +/* VARBINARY*/ true, false, false, false, false, false, false, true, true, false, false, false, false, false,
> +/* SCALAR */ true, false, false, false, false, false, false, false, true, false, false, false, false, false,
> +/* DECIMAL */ true, false, false, true, false, false, false, false, true, true, false, false, false, false,
> +/* UUID */ true, false, false, false, false, false, false, false, false, false, true, false, false, false,
> +/* ARRAY */ true, false, false, false, false, false, false, false, false, false, false, true, false, false,
> +/* MAP */ true, false, false, false, false, false, false, false, false, false, false, false, true, false,
> +/* DATETIME */ true, false, false, false, false, false, false, false, true, false, false, false, false, true,
> };
>
> bool
> diff --git a/src/box/field_def.h b/src/box/field_def.h
> index c5cfe5e86..120b2a93d 100644
> --- a/src/box/field_def.h
> +++ b/src/box/field_def.h
> @@ -63,6 +63,7 @@ enum field_type {
> FIELD_TYPE_UUID,
> FIELD_TYPE_ARRAY,
> FIELD_TYPE_MAP,
> + FIELD_TYPE_DATETIME,
> field_type_MAX
> };
>
> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
> index 1f791980f..51855011b 100644
> --- a/src/box/lua/serialize_lua.c
> +++ b/src/box/lua/serialize_lua.c
> @@ -768,7 +768,7 @@ static int
> dump_node(struct lua_dumper *d, struct node *nd, int indent)
> {
> struct luaL_field *field = &nd->field;
> - char buf[FPCONV_G_FMT_BUFSIZE];
> + char buf[FPCONV_G_FMT_BUFSIZE + 8];
Why +8? Better use max(FPCONV_G_FMT_BUFSIZE, <your buf size>).
> int ltype = lua_type(d->L, -1);
> const char *str = NULL;
> size_t len = 0;
> @@ -861,6 +861,11 @@ dump_node(struct lua_dumper *d, struct node *nd, int indent)
> str = tt_uuid_str(field->uuidval);
> len = UUID_STR_LEN;
> break;
> + case MP_DATETIME:
> + nd->mask |= NODE_QUOTE;
> + str = buf;
> + len = datetime_to_string(field->dateval, buf, sizeof buf);
sizeof(buf)
Please fix here and in other places.
> + break;
> default:
> d->err = EINVAL;
> snprintf(d->err_msg, sizeof(d->err_msg),
> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
> index 43cd29ce9..9a69f2a72 100644
> --- a/src/box/tuple_compare.cc
> +++ b/src/box/tuple_compare.cc
> @@ -36,6 +36,7 @@
> #include "lib/core/decimal.h"
> #include "lib/core/mp_decimal.h"
> #include "uuid/mp_uuid.h"
> +#include "lib/core/mp_datetime.h"
> #include "lib/core/mp_extension_types.h"
>
> /* {{{ tuple_compare */
> @@ -76,6 +77,7 @@ enum mp_class {
> MP_CLASS_STR,
> MP_CLASS_BIN,
> MP_CLASS_UUID,
> + MP_CLASS_DATETIME,
> MP_CLASS_ARRAY,
> MP_CLASS_MAP,
> mp_class_max,
> @@ -99,6 +101,8 @@ static enum mp_class mp_ext_classes[] = {
> /* .MP_UNKNOWN_EXTENSION = */ mp_class_max, /* unsupported */
> /* .MP_DECIMAL = */ MP_CLASS_NUMBER,
> /* .MP_UUID = */ MP_CLASS_UUID,
> + /* .MP_ERROR = */ mp_class_max,
> + /* .MP_DATETIME = */ MP_CLASS_DATETIME,
> };
>
> static enum mp_class
> @@ -390,6 +394,19 @@ mp_compare_uuid(const char *field_a, const char *field_b)
> return memcmp(field_a + 2, field_b + 2, UUID_PACKED_LEN);
> }
>
> +static int
> +mp_compare_datetime(const char *lhs, const char *rhs)
> +{
> + struct datetime lhs_dt, rhs_dt;
> + struct datetime *ret;
> + ret = mp_decode_datetime(&lhs, &lhs_dt);
> + assert(ret != NULL);
> + ret = mp_decode_datetime(&rhs, &rhs_dt);
> + assert(ret != NULL);
> + (void)ret;
> + return datetime_compare(&lhs_dt, &rhs_dt);
> +}
> +
Comparators is a part of indexing hence belong to the next patch.
> typedef int (*mp_compare_f)(const char *, const char *);
> static mp_compare_f mp_class_comparators[] = {
> /* .MP_CLASS_NIL = */ NULL,
> @@ -398,6 +415,7 @@ static mp_compare_f mp_class_comparators[] = {
> /* .MP_CLASS_STR = */ mp_compare_str,
> /* .MP_CLASS_BIN = */ mp_compare_bin,
> /* .MP_CLASS_UUID = */ mp_compare_uuid,
> + /* .MP_CLASS_DATETIME=*/ mp_compare_datetime,
> /* .MP_CLASS_ARRAY = */ NULL,
> /* .MP_CLASS_MAP = */ NULL,
> };
> @@ -478,6 +496,8 @@ tuple_compare_field(const char *field_a, const char *field_b,
> return mp_compare_decimal(field_a, field_b);
> case FIELD_TYPE_UUID:
> return mp_compare_uuid(field_a, field_b);
> + case FIELD_TYPE_DATETIME:
> + return mp_compare_datetime(field_a, field_b);
> default:
> unreachable();
> return 0;
> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
> index c24a0df82..baf9cc8ae 100644
> --- a/src/lib/core/datetime.c
> +++ b/src/lib/core/datetime.c
> @@ -165,3 +165,12 @@ datetime_to_string(const struct datetime *date, char *buf, uint32_t len)
> }
> #undef ADVANCE
>
> +int
> +datetime_compare(const struct datetime *lhs, const struct datetime *rhs)
> +{
> + int result = COMPARE_RESULT(lhs->secs, rhs->secs);
> + if (result != 0)
> + return result;
> +
> + return COMPARE_RESULT(lhs->nsec, rhs->nsec);
What about offset? Shouldn't you take it into account (convert
all dates to UTC)?
> +}
> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
> index 964e76fcc..5122e422e 100644
> --- a/src/lib/core/datetime.h
> +++ b/src/lib/core/datetime.h
> @@ -70,6 +70,17 @@ struct datetime_interval {
> int32_t nsec;
> };
>
> +/**
> + * Compare arguments of a datetime type
> + * @param lhs left datetime argument
> + * @param rhs right datetime argument
> + * @retval < 0 if lhs less than rhs
> + * @retval = 0 if lhs and rhs equal
> + * @retval > 0 if lhs greater than rhs
> + */
> +int
> +datetime_compare(const struct datetime *lhs, const struct datetime *rhs);
> +
> /**
> * Convert datetime to string using default format
> * @param date source datetime value
> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
> new file mode 100644
> index 000000000..d0a3e562c
> --- /dev/null
> +++ b/src/lib/core/mp_datetime.c
> @@ -0,0 +1,189 @@
> +/*
> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + * copyright notice, this list of conditions and the
> + * following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + * copyright notice, this list of conditions and the following
> + * disclaimer in the documentation and/or other materials
> + * provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#include "mp_datetime.h"
> +#include "msgpuck.h"
> +#include "mp_extension_types.h"
> +
> +/*
> + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length)
> + extension, which creates container of 1 to 3 integers.
> +
> + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset (uint) |
> + +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+
> +
> + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
> +
> + - signed integer seconds part (required). Depending on the value of
> + seconds it may be from 1 to 8 bytes positive or negative integer number;
> +
> + - [optional] fraction time in nanoseconds as unsigned integer.
> + If this value is 0 then it's not saved (unless there is offset field,
> + as below);
> +
> + - [optional] timzeone offset in minutes as unsigned integer.
> + If this field is 0 then it's not saved.
> + */
> +
> +static inline uint32_t
> +mp_sizeof_Xint(int64_t n)
mp_sizeof_xint - Let's not mix camel-case and underscores.
I think this should be put somewhere in a public header so that everyone
can use them: mp_utils.h?
> +{
> + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
> +}
> +
> +static inline char *
> +mp_encode_Xint(char *data, int64_t v)
> +{
> + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
> +}
> +
> +static inline int64_t
> +mp_decode_Xint(const char **data)
> +{
> + switch (mp_typeof(**data)) {
> + case MP_UINT:
> + return (int64_t)mp_decode_uint(data);
assert(value <= INT64_MAX) ?
> + case MP_INT:
> + return mp_decode_int(data);
> + default:
> + mp_unreachable();
> + }
> + return 0;
> +}
> +
> +static inline uint32_t
> +mp_sizeof_datetime_raw(const struct datetime *date)
> +{
> + uint32_t sz = mp_sizeof_Xint(date->secs);
> +
> + // even if nanosecs == 0 we need to output anything
> + // if we have non-null tz offset
> + if (date->nsec != 0 || date->offset != 0)
> + sz += mp_sizeof_Xint(date->nsec);
> + if (date->offset)
if (date->offset != 0)
Please be consistent.
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime
2021-08-17 18:36 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 14:27 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 14:27 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On 17.08.2021 21:36, Vladimir Davydov via Tarantool-patches wrote:
> On Mon, Aug 16, 2021 at 02:59:38AM +0300, Timur Safin via
> Tarantool-patches wrote:
>> Serialize datetime_t as newly introduced MP_EXT type.
>> It saves 1 required integer field and upto 2 optional
>> unsigned fields in very compact fashion.
>> - secs is required field;
>> - but nsec, offset are both optional;
>>
>> * json, yaml serialization formats, lua output mode
>> supported;
>> * exported symbols for datetime messagepack size calculations
>> so they are available for usage on Lua side.
>>
>> Part of #5941
>> Part of #5946
>> ---
>> extra/exports | 5 +-
>> src/box/field_def.c | 35 +++---
>> src/box/field_def.h | 1 +
>> src/box/lua/serialize_lua.c | 7 +-
>> src/box/msgpack.c | 7 +-
>> src/box/tuple_compare.cc | 20 ++++
>> src/lib/core/CMakeLists.txt | 4 +-
>> src/lib/core/datetime.c | 9 ++
>> src/lib/core/datetime.h | 11 ++
>> src/lib/core/mp_datetime.c | 189 ++++++++++++++++++++++++++++++
>> src/lib/core/mp_datetime.h | 89 ++++++++++++++
>> src/lib/core/mp_extension_types.h | 1 +
>> src/lib/mpstream/mpstream.c | 11 ++
>> src/lib/mpstream/mpstream.h | 4 +
>> src/lua/msgpack.c | 12 ++
>> src/lua/msgpackffi.lua | 18 +++
>> src/lua/serializer.c | 4 +
>> src/lua/serializer.h | 2 +
>> src/lua/utils.c | 1 -
>> test/unit/datetime.c | 125 +++++++++++++++++++-
>> test/unit/datetime.result | 115 +++++++++++++++++-
>> third_party/lua-cjson/lua_cjson.c | 8 ++
>> third_party/lua-yaml/lyaml.cc | 6 +-
>
> Please add a Lua test checking that serialization (msgpack, yaml, json)
> works fine. Should be added in this patch.
Ok, will move test here (once reshake patches)
>
>> 23 files changed, 661 insertions(+), 23 deletions(-)
>> create mode 100644 src/lib/core/mp_datetime.c
>> create mode 100644 src/lib/core/mp_datetime.h
>>
>> diff --git a/src/box/field_def.c b/src/box/field_def.c
>> index 51acb8025..2682a42ee 100644
>> --- a/src/box/field_def.c
>> +++ b/src/box/field_def.c
>> @@ -72,6 +72,7 @@ const uint32_t field_mp_type[] = {
>> /* [FIELD_TYPE_UUID] = */ 0, /* only MP_UUID is supported */
>> /* [FIELD_TYPE_ARRAY] = */ 1U << MP_ARRAY,
>> /* [FIELD_TYPE_MAP] = */ (1U << MP_MAP),
>> + /* [FIELD_TYPE_DATETIME] = */ 0, /* only MP_DATETIME is supported
> */
>> };
>>
>> const uint32_t field_ext_type[] = {
>> @@ -83,11 +84,13 @@ const uint32_t field_ext_type[] = {
>> /* [FIELD_TYPE_INTEGER] = */ 0,
>> /* [FIELD_TYPE_BOOLEAN] = */ 0,
>> /* [FIELD_TYPE_VARBINARY] = */ 0,
>> - /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U <<
> MP_UUID),
>> + /* [FIELD_TYPE_SCALAR] = */ (1U << MP_DECIMAL) | (1U <<
> MP_UUID) |
>> + (1U << MP_DATETIME),
>> /* [FIELD_TYPE_DECIMAL] = */ 1U << MP_DECIMAL,
>> /* [FIELD_TYPE_UUID] = */ 1U << MP_UUID,
>> /* [FIELD_TYPE_ARRAY] = */ 0,
>> /* [FIELD_TYPE_MAP] = */ 0,
>> + /* [FIELD_TYPE_DATETIME] = */ 1U << MP_DATETIME,
>> };
>>
>> const char *field_type_strs[] = {
>> @@ -104,6 +107,7 @@ const char *field_type_strs[] = {
>> /* [FIELD_TYPE_UUID] = */ "uuid",
>> /* [FIELD_TYPE_ARRAY] = */ "array",
>> /* [FIELD_TYPE_MAP] = */ "map",
>> + /* [FIELD_TYPE_DATETIME] = */ "datetime",
>> };
>
> This doesn't belong to this patch. Please split in two:
>
> - encoding datetime as msgpack/yaml/json - this patch
> - indexing of datetime - next patch
Understood. Very reasonable.
>
>>
>> const char *on_conflict_action_strs[] = {
>> @@ -128,20 +132,21 @@ field_type_by_name_wrapper(const char *str,
> uint32_t len)
>> * values can be stored in the j type.
>> */
>> static const bool field_type_compatibility[] = {
>> - /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN
> VARBINARY SCALAR DECIMAL UUID ARRAY MAP */
>> -/* ANY */ true, false, false, false, false, false,
> false, false, false, false, false, false, false,
>> -/* UNSIGNED */ true, true, false, true, false, true,
> false, false, true, false, false, false, false,
>> -/* STRING */ true, false, true, false, false, false,
> false, false, true, false, false, false, false,
>> -/* NUMBER */ true, false, false, true, false, false,
> false, false, true, false, false, false, false,
>> -/* DOUBLE */ true, false, false, true, true, false,
> false, false, true, false, false, false, false,
>> -/* INTEGER */ true, false, false, true, false, true,
> false, false, true, false, false, false, false,
>> -/* BOOLEAN */ true, false, false, false, false, false,
> true, false, true, false, false, false, false,
>> -/* VARBINARY*/ true, false, false, false, false, false,
> false, true, true, false, false, false, false,
>> -/* SCALAR */ true, false, false, false, false, false,
> false, false, true, false, false, false, false,
>> -/* DECIMAL */ true, false, false, true, false, false,
> false, false, true, true, false, false, false,
>> -/* UUID */ true, false, false, false, false, false,
> false, false, false, false, true, false, false,
>> -/* ARRAY */ true, false, false, false, false, false,
> false, false, false, false, false, true, false,
>> -/* MAP */ true, false, false, false, false, false,
> false, false, false, false, false, false, true,
>> + /* ANY UNSIGNED STRING NUMBER DOUBLE INTEGER BOOLEAN
> VARBINARY SCALAR DECIMAL UUID ARRAY MAP DATETIME */
>> +/* ANY */ true, false, false, false, false, false,
> false, false, false, false, false, false, false, false,
>> +/* UNSIGNED */ true, true, false, true, false, true,
> false, false, true, false, false, false, false, false,
>> +/* STRING */ true, false, true, false, false, false,
> false, false, true, false, false, false, false, false,
>> +/* NUMBER */ true, false, false, true, false, false,
> false, false, true, false, false, false, false, false,
>> +/* DOUBLE */ true, false, false, true, true, false,
> false, false, true, false, false, false, false, false,
>> +/* INTEGER */ true, false, false, true, false, true,
> false, false, true, false, false, false, false, false,
>> +/* BOOLEAN */ true, false, false, false, false, false,
> true, false, true, false, false, false, false, false,
>> +/* VARBINARY*/ true, false, false, false, false, false,
> false, true, true, false, false, false, false, false,
>> +/* SCALAR */ true, false, false, false, false, false,
> false, false, true, false, false, false, false, false,
>> +/* DECIMAL */ true, false, false, true, false, false,
> false, false, true, true, false, false, false, false,
>> +/* UUID */ true, false, false, false, false, false,
> false, false, false, false, true, false, false, false,
>> +/* ARRAY */ true, false, false, false, false, false,
> false, false, false, false, false, true, false, false,
>> +/* MAP */ true, false, false, false, false, false,
> false, false, false, false, false, false, true, false,
>> +/* DATETIME */ true, false, false, false, false, false,
> false, false, true, false, false, false, false, true,
>> };
>>
>> bool
>> diff --git a/src/box/field_def.h b/src/box/field_def.h
>> index c5cfe5e86..120b2a93d 100644
>> --- a/src/box/field_def.h
>> +++ b/src/box/field_def.h
>> @@ -63,6 +63,7 @@ enum field_type {
>> FIELD_TYPE_UUID,
>> FIELD_TYPE_ARRAY,
>> FIELD_TYPE_MAP,
>> + FIELD_TYPE_DATETIME,
>> field_type_MAX
>> };
>>
>> diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c
>> index 1f791980f..51855011b 100644
>> --- a/src/box/lua/serialize_lua.c
>> +++ b/src/box/lua/serialize_lua.c
>> @@ -768,7 +768,7 @@ static int
>> dump_node(struct lua_dumper *d, struct node *nd, int indent)
>> {
>> struct luaL_field *field = &nd->field;
>> - char buf[FPCONV_G_FMT_BUFSIZE];
>> + char buf[FPCONV_G_FMT_BUFSIZE + 8];
>
> Why +8? Better use max(FPCONV_G_FMT_BUFSIZE, <your buf size>).
>
>> int ltype = lua_type(d->L, -1);
>> const char *str = NULL;
>> size_t len = 0;
>> @@ -861,6 +861,11 @@ dump_node(struct lua_dumper *d, struct node *nd,
> int indent)
>> str = tt_uuid_str(field->uuidval);
>> len = UUID_STR_LEN;
>> break;
>> + case MP_DATETIME:
>> + nd->mask |= NODE_QUOTE;
>> + str = buf;
>> + len = datetime_to_string(field->dateval, buf,
> sizeof buf);
>
> sizeof(buf)
>
> Please fix here and in other places.
Well, but... Ok
(This is matter of taste though)
>
>> + break;
>> default:
>> d->err = EINVAL;
>> snprintf(d->err_msg, sizeof(d->err_msg),
>> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
>> index 43cd29ce9..9a69f2a72 100644
>> --- a/src/box/tuple_compare.cc
>> +++ b/src/box/tuple_compare.cc
>> @@ -36,6 +36,7 @@
>> #include "lib/core/decimal.h"
>> #include "lib/core/mp_decimal.h"
>> #include "uuid/mp_uuid.h"
>> +#include "lib/core/mp_datetime.h"
>> #include "lib/core/mp_extension_types.h"
>>
>> /* {{{ tuple_compare */
>> @@ -76,6 +77,7 @@ enum mp_class {
>> MP_CLASS_STR,
>> MP_CLASS_BIN,
>> MP_CLASS_UUID,
>> + MP_CLASS_DATETIME,
>> MP_CLASS_ARRAY,
>> MP_CLASS_MAP,
>> mp_class_max,
>> @@ -99,6 +101,8 @@ static enum mp_class mp_ext_classes[] = {
>> /* .MP_UNKNOWN_EXTENSION = */ mp_class_max, /* unsupported */
>> /* .MP_DECIMAL = */ MP_CLASS_NUMBER,
>> /* .MP_UUID = */ MP_CLASS_UUID,
>> + /* .MP_ERROR = */ mp_class_max,
>> + /* .MP_DATETIME = */ MP_CLASS_DATETIME,
>> };
>>
>> static enum mp_class
>> @@ -390,6 +394,19 @@ mp_compare_uuid(const char *field_a, const char
> *field_b)
>> return memcmp(field_a + 2, field_b + 2, UUID_PACKED_LEN);
>> }
>>
>> +static int
>> +mp_compare_datetime(const char *lhs, const char *rhs)
>> +{
>> + struct datetime lhs_dt, rhs_dt;
>> + struct datetime *ret;
>> + ret = mp_decode_datetime(&lhs, &lhs_dt);
>> + assert(ret != NULL);
>> + ret = mp_decode_datetime(&rhs, &rhs_dt);
>> + assert(ret != NULL);
>> + (void)ret;
>> + return datetime_compare(&lhs_dt, &rhs_dt);
>> +}
>> +
>
> Comparators is a part of indexing hence belong to the next patch.
Ok.
>
>> typedef int (*mp_compare_f)(const char *, const char *);
>> static mp_compare_f mp_class_comparators[] = {
>> /* .MP_CLASS_NIL = */ NULL,
>> @@ -398,6 +415,7 @@ static mp_compare_f mp_class_comparators[] = {
>> /* .MP_CLASS_STR = */ mp_compare_str,
>> /* .MP_CLASS_BIN = */ mp_compare_bin,
>> /* .MP_CLASS_UUID = */ mp_compare_uuid,
>> + /* .MP_CLASS_DATETIME=*/ mp_compare_datetime,
>> /* .MP_CLASS_ARRAY = */ NULL,
>> /* .MP_CLASS_MAP = */ NULL,
>> };
>> @@ -478,6 +496,8 @@ tuple_compare_field(const char *field_a, const char
> *field_b,
>> return mp_compare_decimal(field_a, field_b);
>> case FIELD_TYPE_UUID:
>> return mp_compare_uuid(field_a, field_b);
>> + case FIELD_TYPE_DATETIME:
>> + return mp_compare_datetime(field_a, field_b);
>> default:
>> unreachable();
>> return 0;
>> diff --git a/src/lib/core/datetime.c b/src/lib/core/datetime.c
>> index c24a0df82..baf9cc8ae 100644
>> --- a/src/lib/core/datetime.c
>> +++ b/src/lib/core/datetime.c
>> @@ -165,3 +165,12 @@ datetime_to_string(const struct datetime *date,
> char *buf, uint32_t len)
>> }
>> #undef ADVANCE
>>
>> +int
>> +datetime_compare(const struct datetime *lhs, const struct datetime
> *rhs)
>> +{
>> + int result = COMPARE_RESULT(lhs->secs, rhs->secs);
>> + if (result != 0)
>> + return result;
>> +
>> + return COMPARE_RESULT(lhs->nsec, rhs->nsec);
>
> What about offset? Shouldn't you take it into account (convert
> all dates to UTC)?
We keep datetime normalized, that's part of their design (to not waste
time in normalizations when need to proceed arithmetic). Information
about original time-zone is only for information/display purposes.
Changing of timezone will not impact values of .secs or .nsec, but will
change displayed text.
>
>> +}
>> diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h
>> index 964e76fcc..5122e422e 100644
>> --- a/src/lib/core/datetime.h
>> +++ b/src/lib/core/datetime.h
>> @@ -70,6 +70,17 @@ struct datetime_interval {
>> int32_t nsec;
>> };
>>
>> +/**
>> + * Compare arguments of a datetime type
>> + * @param lhs left datetime argument
>> + * @param rhs right datetime argument
>> + * @retval < 0 if lhs less than rhs
>> + * @retval = 0 if lhs and rhs equal
>> + * @retval > 0 if lhs greater than rhs
>> + */
>> +int
>> +datetime_compare(const struct datetime *lhs, const struct datetime
> *rhs);
>> +
>> /**
>> * Convert datetime to string using default format
>> * @param date source datetime value
>> diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
>> new file mode 100644
>> index 000000000..d0a3e562c
>> --- /dev/null
>> +++ b/src/lib/core/mp_datetime.c
>> @@ -0,0 +1,189 @@
>> +/*
>> + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file.
>> + *
>> + * Redistribution and use in source and binary forms, with or
>> + * without modification, are permitted provided that the following
>> + * conditions are met:
>> + *
>> + * 1. Redistributions of source code must retain the above
>> + * copyright notice, this list of conditions and the
>> + * following disclaimer.
>> + *
>> + * 2. Redistributions in binary form must reproduce the above
>> + * copyright notice, this list of conditions and the following
>> + * disclaimer in the documentation and/or other materials
>> + * provided with the distribution.
>> + *
>> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
>> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
>> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
>> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
>> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
>> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
>> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
>> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
>> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
>> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
>> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
>> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
>> + * SUCH DAMAGE.
>> + */
>> +
>> +#include "mp_datetime.h"
>> +#include "msgpuck.h"
>> +#include "mp_extension_types.h"
>> +
>> +/*
>> + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte
> length)
>> + extension, which creates container of 1 to 3 integers.
>> +
>> +
> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....
> +
>> + |0xC7| 4 |len (uint8)| seconds (int) | nanoseconds (uint) | offset
> (uint) |
>> +
> +----+---+-----------+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....
> +
>> +
>> + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains:
>> +
>> + - signed integer seconds part (required). Depending on the value of
>> + seconds it may be from 1 to 8 bytes positive or negative integer
> number;
>> +
>> + - [optional] fraction time in nanoseconds as unsigned integer.
>> + If this value is 0 then it's not saved (unless there is offset
> field,
>> + as below);
>> +
>> + - [optional] timzeone offset in minutes as unsigned integer.
>> + If this field is 0 then it's not saved.
>> + */
>> +
>> +static inline uint32_t
>> +mp_sizeof_Xint(int64_t n)
>
> mp_sizeof_xint - Let's not mix camel-case and underscores.
>
> I think this should be put somewhere in a public header so that everyone
> can use them: mp_utils.h?
Will create - you are the 2nd who is asking for that extraction. Pity we
do not have such mp utilities file already.
>
>> +{
>> + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
>> +}
>> +
>> +static inline char *
>> +mp_encode_Xint(char *data, int64_t v)
>> +{
>> + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
>> +}
>> +
>> +static inline int64_t
>> +mp_decode_Xint(const char **data)
>> +{
>> + switch (mp_typeof(**data)) {
>> + case MP_UINT:
>> + return (int64_t)mp_decode_uint(data);
>
> assert(value <= INT64_MAX) ?
That's why I kept this local for a moment. Taking into consideration the
nature of our data we are guaranteed to not ever approach any bound
ranges neither for unsigned nor for signed integers here.
But in generic header file it need to be able to proceed all types of
input data gracefully.
On the 2nd thought I believe this assert should be placed into
mp_encode_xint() function. To avoid garbage data to be generated.
>
>> + case MP_INT:
>> + return mp_decode_int(data);
>> + default:
>> + mp_unreachable();
>> + }
>> + return 0;
>> +}
>> +
>> +static inline uint32_t
>> +mp_sizeof_datetime_raw(const struct datetime *date)
>> +{
>> + uint32_t sz = mp_sizeof_Xint(date->secs);
>> +
>> + // even if nanosecs == 0 we need to output anything
>> + // if we have non-null tz offset
>> + if (date->nsec != 0 || date->offset != 0)
>> + sz += mp_sizeof_Xint(date->nsec);
>> + if (date->offset)
>
> if (date->offset != 0)
>
> Please be consistent.
Done.
Increment
-------------------------------------
diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c
index 963752c23..1c9e8a7df 100644
--- a/src/lib/core/mp_datetime.c
+++ b/src/lib/core/mp_datetime.c
@@ -33,19 +33,20 @@
*/
static inline uint32_t
-mp_sizeof_Xint(int64_t n)
+mp_sizeof_xint(int64_t n)
{
return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n);
}
static inline char *
-mp_encode_Xint(char *data, int64_t v)
+mp_encode_xint(char *data, int64_t v)
{
+ assert((uint64_t)v <= LONG_MAX);
return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v);
}
static inline int64_t
-mp_decode_Xint(const char **data)
+mp_decode_xint(const char **data)
{
switch (mp_typeof(**data)) {
case MP_UINT:
@@ -72,7 +73,7 @@ static inline uint32_t
mp_sizeof_datetime_raw(const struct datetime *date)
{
check_secs(date->secs);
- uint32_t sz = mp_sizeof_Xint(date->secs);
+ uint32_t sz = mp_sizeof_xint(date->secs);
/*
* even if nanosecs == 0 we need to output something
@@ -80,11 +81,11 @@ mp_sizeof_datetime_raw(const struct datetime *date)
*/
if (date->nsec != 0 || date->offset != 0) {
check_nanosecs(date->nsec);
- sz += mp_sizeof_Xint(date->nsec);
+ sz += mp_sizeof_xint(date->nsec);
}
- if (date->offset) {
+ if (date->offset != 0) {
check_tz_offset(date->offset);
- sz += mp_sizeof_Xint(date->offset);
+ sz += mp_sizeof_xint(date->offset);
}
return sz;
}
@@ -102,7 +103,7 @@ datetime_unpack(const char **data, uint32_t len,
struct datetime *date)
memset(date, 0, sizeof(*date));
- int64_t seconds = mp_decode_Xint(data);
+ int64_t seconds = mp_decode_xint(data);
check_secs(seconds);
date->secs = seconds;
@@ -119,7 +120,7 @@ datetime_unpack(const char **data, uint32_t len,
struct datetime *date)
if (len <= 0)
return date;
- int64_t offset = mp_decode_Xint(data);
+ int64_t offset = mp_decode_xint(data);
check_tz_offset(offset);
date->offset = offset;
@@ -146,11 +147,11 @@ mp_decode_datetime(const char **data, struct
datetime *date)
char *
datetime_pack(char *data, const struct datetime *date)
{
- data = mp_encode_Xint(data, date->secs);
+ data = mp_encode_xint(data, date->secs);
if (date->nsec != 0 || date->offset != 0)
data = mp_encode_uint(data, date->nsec);
if (date->offset)
- data = mp_encode_Xint(data, date->offset);
+ data = mp_encode_xint(data, date->offset);
return data;
}
-------------------------------------
>
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (3 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 4/8] box, datetime: messagepack support for datetime Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 19:05 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support Timur Safin via Tarantool-patches
` (4 subsequent siblings)
9 siblings, 2 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
* storage hints implemented for datetime_t values;
* proper comparison for indices of datetime type.
Part of #5941
Part of #5946
---
src/box/field_def.c | 18 ++++++++
src/box/field_def.h | 3 ++
src/box/memtx_space.c | 3 +-
src/box/tuple_compare.cc | 57 ++++++++++++++++++++++++++
src/box/vinyl.c | 3 +-
test/engine/datetime.result | 77 +++++++++++++++++++++++++++++++++++
test/engine/datetime.test.lua | 35 ++++++++++++++++
7 files changed, 192 insertions(+), 4 deletions(-)
create mode 100644 test/engine/datetime.result
create mode 100644 test/engine/datetime.test.lua
diff --git a/src/box/field_def.c b/src/box/field_def.c
index 2682a42ee..97033d0bb 100644
--- a/src/box/field_def.c
+++ b/src/box/field_def.c
@@ -194,3 +194,21 @@ field_type_by_name(const char *name, size_t len)
return FIELD_TYPE_ANY;
return field_type_MAX;
}
+
+const bool field_type_index_allowed[] =
+ {
+ /* [FIELD_TYPE_ANY] = */ false,
+ /* [FIELD_TYPE_UNSIGNED] = */ true,
+ /* [FIELD_TYPE_STRING] = */ true,
+ /* [FIELD_TYPE_NUMBER] = */ true,
+ /* [FIELD_TYPE_DOUBLE] = */ true,
+ /* [FIELD_TYPE_INTEGER] = */ true,
+ /* [FIELD_TYPE_BOOLEAN] = */ true,
+ /* [FIELD_TYPE_VARBINARY]= */ true,
+ /* [FIELD_TYPE_SCALAR] = */ true,
+ /* [FIELD_TYPE_DECIMAL] = */ true,
+ /* [FIELD_TYPE_UUID] = */ true,
+ /* [FIELD_TYPE_ARRAY] = */ false,
+ /* [FIELD_TYPE_MAP] = */ false,
+ /* [FIELD_TYPE_DATETIME] = */ true,
+};
diff --git a/src/box/field_def.h b/src/box/field_def.h
index 120b2a93d..bd02418df 100644
--- a/src/box/field_def.h
+++ b/src/box/field_def.h
@@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[];
extern const struct opt_def field_def_reg[];
extern const struct field_def field_def_default;
+/** helper table for checking allowed indices for types */
+extern const bool field_type_index_allowed[];
+
/**
* @brief Field definition
* Contains information about of one tuple field.
diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
index b71318d24..1ab16122e 100644
--- a/src/box/memtx_space.c
+++ b/src/box/memtx_space.c
@@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
/* Check that there are no ANY, ARRAY, MAP parts */
for (uint32_t i = 0; i < key_def->part_count; i++) {
struct key_part *part = &key_def->parts[i];
- if (part->type <= FIELD_TYPE_ANY ||
- part->type >= FIELD_TYPE_ARRAY) {
+ if (!field_type_index_allowed[part->type]) {
diag_set(ClientError, ER_MODIFY_INDEX,
index_def->name, space_name(space),
tt_sprintf("field type '%s' is not supported",
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index 9a69f2a72..110017853 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a, enum mp_type a_type,
field_b, b_type);
case FIELD_TYPE_UUID:
return mp_compare_uuid(field_a, field_b);
+ case FIELD_TYPE_DATETIME:
+ return mp_compare_datetime(field_a, field_b);
default:
unreachable();
return 0;
@@ -1538,6 +1540,21 @@ func_index_compare_with_key(struct tuple *tuple, hint_t tuple_hint,
#define HINT_VALUE_DOUBLE_MAX (exp2(HINT_VALUE_BITS - 1) - 1)
#define HINT_VALUE_DOUBLE_MIN (-exp2(HINT_VALUE_BITS - 1))
+/**
+ * We need to squeeze 64 bits of seconds and 32 bits of nanoseconds
+ * into 60 bits of hint value. The idea is to represent wide enough
+ * years range, and leave the rest of bits occupied from nanoseconds part:
+ * - 36 bits is enough for time range of [208BC..4147]
+ * - for nanoseconds there is left 24 bits, which are MSB part of
+ * 32-bit value
+ */
+#define HINT_VALUE_SECS_BITS 36
+#define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS - HINT_VALUE_SECS_BITS)
+#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
+#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
+#define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT - HINT_VALUE_NSEC_BITS)
+#define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
+
/*
* HINT_CLASS_BITS should be big enough to store any mp_class value.
* Note, ((1 << HINT_CLASS_BITS) - 1) is reserved for HINT_NONE.
@@ -1630,6 +1647,25 @@ hint_uuid_raw(const char *data)
return hint_create(MP_CLASS_UUID, val);
}
+static inline hint_t
+hint_datetime(struct datetime *date)
+{
+ /*
+ * Use at most HINT_VALUE_SECS_BITS from datetime
+ * seconds field as a hint value, and at MSB part
+ * of HINT_VALUE_NSEC_BITS from nanoseconds.
+ */
+ int64_t secs = date->secs;
+ int32_t nsec = date->nsec;
+ uint64_t val = secs <= HINT_VALUE_SECS_MIN ? 0 :
+ secs - HINT_VALUE_SECS_MIN;
+ if (val >= HINT_VALUE_SECS_MAX)
+ val = HINT_VALUE_SECS_MAX;
+ val <<= HINT_VALUE_NSEC_BITS;
+ val |= (nsec >> HINT_VALUE_NSEC_SHIFT) & HINT_VALUE_NSEC_MAX;
+ return hint_create(MP_CLASS_DATETIME, val);
+}
+
static inline uint64_t
hint_str_raw(const char *s, uint32_t len)
{
@@ -1761,6 +1797,17 @@ field_hint_uuid(const char *field)
return hint_uuid_raw(data);
}
+static inline hint_t
+field_hint_datetime(const char *field)
+{
+ assert(mp_typeof(*field) == MP_EXT);
+ int8_t ext_type;
+ uint32_t len = mp_decode_extl(&field, &ext_type);
+ assert(ext_type == MP_DATETIME);
+ struct datetime date;
+ return hint_datetime(datetime_unpack(&field, len, &date));
+}
+
static inline hint_t
field_hint_string(const char *field, struct coll *coll)
{
@@ -1812,6 +1859,11 @@ field_hint_scalar(const char *field, struct coll *coll)
}
case MP_UUID:
return hint_uuid_raw(field);
+ case MP_DATETIME:
+ {
+ struct datetime date;
+ return hint_datetime(datetime_unpack(&field, len, &date));
+ }
default:
unreachable();
}
@@ -1849,6 +1901,8 @@ field_hint(const char *field, struct coll *coll)
return field_hint_decimal(field);
case FIELD_TYPE_UUID:
return field_hint_uuid(field);
+ case FIELD_TYPE_DATETIME:
+ return field_hint_datetime(field);
default:
unreachable();
}
@@ -1963,6 +2017,9 @@ key_def_set_hint_func(struct key_def *def)
case FIELD_TYPE_UUID:
key_def_set_hint_func<FIELD_TYPE_UUID>(def);
break;
+ case FIELD_TYPE_DATETIME:
+ key_def_set_hint_func<FIELD_TYPE_DATETIME>(def);
+ break;
default:
/* Invalid key definition. */
def->key_hint = NULL;
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index c80b2d99b..360d1fa70 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -662,8 +662,7 @@ vinyl_space_check_index_def(struct space *space, struct index_def *index_def)
/* Check that there are no ANY, ARRAY, MAP parts */
for (uint32_t i = 0; i < key_def->part_count; i++) {
struct key_part *part = &key_def->parts[i];
- if (part->type <= FIELD_TYPE_ANY ||
- part->type >= FIELD_TYPE_ARRAY) {
+ if (!field_type_index_allowed[part->type]) {
diag_set(ClientError, ER_MODIFY_INDEX,
index_def->name, space_name(space),
tt_sprintf("field type '%s' is not supported",
diff --git a/test/engine/datetime.result b/test/engine/datetime.result
new file mode 100644
index 000000000..848a0aaec
--- /dev/null
+++ b/test/engine/datetime.result
@@ -0,0 +1,77 @@
+-- test-run result file version 2
+env = require('test_run')
+ | ---
+ | ...
+test_run = env.new()
+ | ---
+ | ...
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+
+date = require('datetime')
+ | ---
+ | ...
+
+_ = box.schema.space.create('T', {engine = engine})
+ | ---
+ | ...
+_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
+ | ---
+ | ...
+
+box.space.T:insert{date('1970-01-01')}\
+box.space.T:insert{date('1970-01-02')}\
+box.space.T:insert{date('1970-01-03')}\
+box.space.T:insert{date('2000-01-01')}
+ | ---
+ | ...
+
+o = box.space.T:select{}
+ | ---
+ | ...
+assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
+ | ---
+ | - true
+ | ...
+assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
+ | ---
+ | - true
+ | ...
+
+for i = 1,16 do\
+ box.space.T:insert{date.now()}\
+end
+ | ---
+ | ...
+
+a = box.space.T:select{}
+ | ---
+ | ...
+err = {}
+ | ---
+ | ...
+for i = 1, #a - 1 do\
+ if a[i][1] >= a[i+1][1] then\
+ table.insert(err, {a[i][1], a[i+1][1]})\
+ break\
+ end\
+end
+ | ---
+ | ...
+
+err
+ | ---
+ | - []
+ | ...
+box.space.T:drop()
+ | ---
+ | ...
diff --git a/test/engine/datetime.test.lua b/test/engine/datetime.test.lua
new file mode 100644
index 000000000..3685e4d4b
--- /dev/null
+++ b/test/engine/datetime.test.lua
@@ -0,0 +1,35 @@
+env = require('test_run')
+test_run = env.new()
+engine = test_run:get_cfg('engine')
+
+date = require('datetime')
+
+_ = box.schema.space.create('T', {engine = engine})
+_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
+
+box.space.T:insert{date('1970-01-01')}\
+box.space.T:insert{date('1970-01-02')}\
+box.space.T:insert{date('1970-01-03')}\
+box.space.T:insert{date('2000-01-01')}
+
+o = box.space.T:select{}
+assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
+assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
+assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
+assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
+
+for i = 1,16 do\
+ box.space.T:insert{date.now()}\
+end
+
+a = box.space.T:select{}
+err = {}
+for i = 1, #a - 1 do\
+ if a[i][1] >= a[i+1][1] then\
+ table.insert(err, {a[i][1], a[i+1][1]})\
+ break\
+ end\
+end
+
+err
+box.space.T:drop()
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
@ 2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:43 ` Safin Timur via Tarantool-patches
2021-08-17 19:05 ` Vladimir Davydov via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:16 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * storage hints implemented for datetime_t values;
> * proper comparison for indices of datetime type.
>
> Part of #5941
> Part of #5946
Please, add a docbot request stating that it's now possible to store
datetime values in spaces and create indexed datetime fields.
> ---
> src/box/field_def.c | 18 ++++++++
> src/box/field_def.h | 3 ++
> src/box/memtx_space.c | 3 +-
> src/box/tuple_compare.cc | 57 ++++++++++++++++++++++++++
> src/box/vinyl.c | 3 +-
> test/engine/datetime.result | 77 +++++++++++++++++++++++++++++++++++
> test/engine/datetime.test.lua | 35 ++++++++++++++++
> 7 files changed, 192 insertions(+), 4 deletions(-)
> create mode 100644 test/engine/datetime.result
> create mode 100644 test/engine/datetime.test.lua
>
> diff --git a/src/box/field_def.c b/src/box/field_def.c
> index 2682a42ee..97033d0bb 100644
> --- a/src/box/field_def.c
> +++ b/src/box/field_def.c
> @@ -194,3 +194,21 @@ field_type_by_name(const char *name, size_t len)
> return FIELD_TYPE_ANY;
> return field_type_MAX;
> }
> +
> +const bool field_type_index_allowed[] =
> + {
> + /* [FIELD_TYPE_ANY] = */ false,
> + /* [FIELD_TYPE_UNSIGNED] = */ true,
> + /* [FIELD_TYPE_STRING] = */ true,
> + /* [FIELD_TYPE_NUMBER] = */ true,
> + /* [FIELD_TYPE_DOUBLE] = */ true,
> + /* [FIELD_TYPE_INTEGER] = */ true,
> + /* [FIELD_TYPE_BOOLEAN] = */ true,
> + /* [FIELD_TYPE_VARBINARY]= */ true,
> + /* [FIELD_TYPE_SCALAR] = */ true,
> + /* [FIELD_TYPE_DECIMAL] = */ true,
> + /* [FIELD_TYPE_UUID] = */ true,
> + /* [FIELD_TYPE_ARRAY] = */ false,
> + /* [FIELD_TYPE_MAP] = */ false,
> + /* [FIELD_TYPE_DATETIME] = */ true,
> +};
You wouldn't need that array if you moved
FIELD_TYPE_DATETIME above FIELD_TYPE_ARRAY
in the previous commit.
Please, do so.
> diff --git a/src/box/field_def.h b/src/box/field_def.h
> index 120b2a93d..bd02418df 100644
> --- a/src/box/field_def.h
> +++ b/src/box/field_def.h
> @@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[];
> extern const struct opt_def field_def_reg[];
> extern const struct field_def field_def_default;
>
> +/** helper table for checking allowed indices for types */
> +extern const bool field_type_index_allowed[];
> +
> /**
> * @brief Field definition
> * Contains information about of one tuple field.
> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
> index b71318d24..1ab16122e 100644
> --- a/src/box/memtx_space.c
> +++ b/src/box/memtx_space.c
> @@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space, struct index_def *index_def)
> /* Check that there are no ANY, ARRAY, MAP parts */
> for (uint32_t i = 0; i < key_def->part_count; i++) {
> struct key_part *part = &key_def->parts[i];
> - if (part->type <= FIELD_TYPE_ANY ||
> - part->type >= FIELD_TYPE_ARRAY) {
> + if (!field_type_index_allowed[part->type]) {
> diag_set(ClientError, ER_MODIFY_INDEX,
> index_def->name, space_name(space),
> tt_sprintf("field type '%s' is not supported",
> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
> index 9a69f2a72..110017853 100644
> --- a/src/box/tuple_compare.cc
> +++ b/src/box/tuple_compare.cc
> @@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a, enum mp_type a_type,
> field_b, b_type);
> case FIELD_TYPE_UUID:
> return mp_compare_uuid(field_a, field_b);
> + case FIELD_TYPE_DATETIME:
> + return mp_compare_datetime(field_a, field_b);
> default:
> unreachable();
> return 0;
> @@ -1538,6 +1540,21 @@ func_index_compare_with_key(struct tuple *tuple, hint_t tuple_hint,
> #define HINT_VALUE_DOUBLE_MAX (exp2(HINT_VALUE_BITS - 1) - 1)
> #define HINT_VALUE_DOUBLE_MIN (-exp2(HINT_VALUE_BITS - 1))
>
> +/**
> + * We need to squeeze 64 bits of seconds and 32 bits of nanoseconds
> + * into 60 bits of hint value. The idea is to represent wide enough
> + * years range, and leave the rest of bits occupied from nanoseconds part:
> + * - 36 bits is enough for time range of [208BC..4147]
> + * - for nanoseconds there is left 24 bits, which are MSB part of
> + * 32-bit value
> + */
> +#define HINT_VALUE_SECS_BITS 36
> +#define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS - HINT_VALUE_SECS_BITS)
> +#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
Am I missing something?
n bits may store values from (-2^(n-1)) to 2^(n-1)-1
should be (1LL << (HINT_VALUE_SECS_BITS -1)) - 1 ?
> +#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
should be
#define HINT_VALUE_SECS_MIN (-(1LL << (HINT_VALUE_SECS_BITS - 1)))
?
> +#define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT - HINT_VALUE_NSEC_BITS)
> +#define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
> +
> /*
> * HINT_CLASS_BITS should be big enough to store any mp_class value.
> * Note, ((1 << HINT_CLASS_BITS) - 1) is reserved for HINT_NONE.
> @@ -1630,6 +1647,25 @@ hint_uuid_raw(const char *data)
> return hint_create(MP_CLASS_UUID, val);
> }
>
> +static inline hint_t
> +hint_datetime(struct datetime *date)
> +{
> + /*
> + * Use at most HINT_VALUE_SECS_BITS from datetime
> + * seconds field as a hint value, and at MSB part
> + * of HINT_VALUE_NSEC_BITS from nanoseconds.
> + */
> + int64_t secs = date->secs;
> + int32_t nsec = date->nsec;
> + uint64_t val = secs <= HINT_VALUE_SECS_MIN ? 0 :
> + secs - HINT_VALUE_SECS_MIN;
> + if (val >= HINT_VALUE_SECS_MAX)
> + val = HINT_VALUE_SECS_MAX;
> + val <<= HINT_VALUE_NSEC_BITS;
> + val |= (nsec >> HINT_VALUE_NSEC_SHIFT) & HINT_VALUE_NSEC_MAX;
> + return hint_create(MP_CLASS_DATETIME, val);
> +}
> +
<stripped>
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:43 ` Safin Timur via Tarantool-patches
2021-08-18 9:03 ` Serge Petrenko via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:43 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
On 17.08.2021 15:16, Serge Petrenko wrote:
>
>
> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>> * storage hints implemented for datetime_t values;
>> * proper comparison for indices of datetime type.
>>
>> Part of #5941
>> Part of #5946
>
>
> Please, add a docbot request stating that it's now possible to store
> datetime values in spaces and create indexed datetime fields.
Will use something like that:
@TarantoolBot document
Title: Storage support for datetime values
It's now possible to store datetime values in spaces and create
indexed datetime fields.
Please refer to https://github.com/tarantool/tarantool/discussions/6244
for more detailed description of a storage schema.
>
>
>> ---
>> src/box/field_def.c | 18 ++++++++
>> src/box/field_def.h | 3 ++
>> src/box/memtx_space.c | 3 +-
>> src/box/tuple_compare.cc | 57 ++++++++++++++++++++++++++
>> src/box/vinyl.c | 3 +-
>> test/engine/datetime.result | 77 +++++++++++++++++++++++++++++++++++
>> test/engine/datetime.test.lua | 35 ++++++++++++++++
>> 7 files changed, 192 insertions(+), 4 deletions(-)
>> create mode 100644 test/engine/datetime.result
>> create mode 100644 test/engine/datetime.test.lua
>>
>> diff --git a/src/box/field_def.c b/src/box/field_def.c
>> index 2682a42ee..97033d0bb 100644
>> --- a/src/box/field_def.c
>> +++ b/src/box/field_def.c
>> @@ -194,3 +194,21 @@ field_type_by_name(const char *name, size_t len)
>> return FIELD_TYPE_ANY;
>> return field_type_MAX;
>> }
>> +
>> +const bool field_type_index_allowed[] =
>> + {
>> + /* [FIELD_TYPE_ANY] = */ false,
>> + /* [FIELD_TYPE_UNSIGNED] = */ true,
>> + /* [FIELD_TYPE_STRING] = */ true,
>> + /* [FIELD_TYPE_NUMBER] = */ true,
>> + /* [FIELD_TYPE_DOUBLE] = */ true,
>> + /* [FIELD_TYPE_INTEGER] = */ true,
>> + /* [FIELD_TYPE_BOOLEAN] = */ true,
>> + /* [FIELD_TYPE_VARBINARY]= */ true,
>> + /* [FIELD_TYPE_SCALAR] = */ true,
>> + /* [FIELD_TYPE_DECIMAL] = */ true,
>> + /* [FIELD_TYPE_UUID] = */ true,
>> + /* [FIELD_TYPE_ARRAY] = */ false,
>> + /* [FIELD_TYPE_MAP] = */ false,
>> + /* [FIELD_TYPE_DATETIME] = */ true,
>> +};
>
>
> You wouldn't need that array if you moved
> FIELD_TYPE_DATETIME above FIELD_TYPE_ARRAY
> in the previous commit.
>
> Please, do so.
Yes, will change order and also move all field support code to this
patch (as Vova recommends).
>
>
>> diff --git a/src/box/field_def.h b/src/box/field_def.h
>> index 120b2a93d..bd02418df 100644
>> --- a/src/box/field_def.h
>> +++ b/src/box/field_def.h
>> @@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[];
>> extern const struct opt_def field_def_reg[];
>> extern const struct field_def field_def_default;
>> +/** helper table for checking allowed indices for types */
>> +extern const bool field_type_index_allowed[];
>> +
>> /**
>> * @brief Field definition
>> * Contains information about of one tuple field.
>> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
>> index b71318d24..1ab16122e 100644
>> --- a/src/box/memtx_space.c
>> +++ b/src/box/memtx_space.c
>> @@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space,
>> struct index_def *index_def)
>> /* Check that there are no ANY, ARRAY, MAP parts */
>> for (uint32_t i = 0; i < key_def->part_count; i++) {
>> struct key_part *part = &key_def->parts[i];
>> - if (part->type <= FIELD_TYPE_ANY ||
>> - part->type >= FIELD_TYPE_ARRAY) {
>> + if (!field_type_index_allowed[part->type]) {
>> diag_set(ClientError, ER_MODIFY_INDEX,
>> index_def->name, space_name(space),
>> tt_sprintf("field type '%s' is not supported",
>> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
>> index 9a69f2a72..110017853 100644
>> --- a/src/box/tuple_compare.cc
>> +++ b/src/box/tuple_compare.cc
>> @@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char *field_a,
>> enum mp_type a_type,
>> field_b, b_type);
>> case FIELD_TYPE_UUID:
>> return mp_compare_uuid(field_a, field_b);
>> + case FIELD_TYPE_DATETIME:
>> + return mp_compare_datetime(field_a, field_b);
>> default:
>> unreachable();
>> return 0;
>> @@ -1538,6 +1540,21 @@ func_index_compare_with_key(struct tuple
>> *tuple, hint_t tuple_hint,
>> #define HINT_VALUE_DOUBLE_MAX (exp2(HINT_VALUE_BITS - 1) - 1)
>> #define HINT_VALUE_DOUBLE_MIN (-exp2(HINT_VALUE_BITS - 1))
>> +/**
>> + * We need to squeeze 64 bits of seconds and 32 bits of nanoseconds
>> + * into 60 bits of hint value. The idea is to represent wide enough
>> + * years range, and leave the rest of bits occupied from nanoseconds
>> part:
>> + * - 36 bits is enough for time range of [208BC..4147]
>> + * - for nanoseconds there is left 24 bits, which are MSB part of
>> + * 32-bit value
>> + */
>> +#define HINT_VALUE_SECS_BITS 36
>> +#define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS - HINT_VALUE_SECS_BITS)
>> +#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
>
> Am I missing something?
> n bits may store values from (-2^(n-1)) to 2^(n-1)-1
>
> should be (1LL << (HINT_VALUE_SECS_BITS -1)) - 1 ?
>
>
>> +#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
>
>
>
>
> should be
>
> #define HINT_VALUE_SECS_MIN (-(1LL << (HINT_VALUE_SECS_BITS - 1)))
>
> ?
>
Yes, my definition made sense only when used as a mask (in prior version
of a code). Thus did not take into consideration a sign bit. You
absolutely correct that if seconds are signed then we have lesser number
of bits, and your definitions of HINT_VALUE_SECS_MAX/HINT_VALUE_SECS_MIN
should be used.
>
>> +#define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT -
>> HINT_VALUE_NSEC_BITS)
>> +#define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
>> +
>> /*
>> * HINT_CLASS_BITS should be big enough to store any mp_class value.
>> * Note, ((1 << HINT_CLASS_BITS) - 1) is reserved for HINT_NONE.
>> @@ -1630,6 +1647,25 @@ hint_uuid_raw(const char *data)
>> return hint_create(MP_CLASS_UUID, val);
>> }
>> +static inline hint_t
>> +hint_datetime(struct datetime *date)
>> +{
>> + /*
>> + * Use at most HINT_VALUE_SECS_BITS from datetime
>> + * seconds field as a hint value, and at MSB part
>> + * of HINT_VALUE_NSEC_BITS from nanoseconds.
>> + */
>> + int64_t secs = date->secs;
>> + int32_t nsec = date->nsec;
>> + uint64_t val = secs <= HINT_VALUE_SECS_MIN ? 0 :
>> + secs - HINT_VALUE_SECS_MIN;
>> + if (val >= HINT_VALUE_SECS_MAX)
>> + val = HINT_VALUE_SECS_MAX;
>> + val <<= HINT_VALUE_NSEC_BITS;
>> + val |= (nsec >> HINT_VALUE_NSEC_SHIFT) & HINT_VALUE_NSEC_MAX;
>> + return hint_create(MP_CLASS_DATETIME, val);
>> +}
>> +
>
> <stripped>
>
Patch increment here small (so far)
------------------------------------
diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
index 110017853..2478498ba 100644
--- a/src/box/tuple_compare.cc
+++ b/src/box/tuple_compare.cc
@@ -1550,8 +1550,8 @@ func_index_compare_with_key(struct tuple *tuple,
hint_t tuple_hint,
*/
#define HINT_VALUE_SECS_BITS 36
#define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS - HINT_VALUE_SECS_BITS)
-#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
-#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
+#define HINT_VALUE_SECS_MAX ((1LL << (HINT_VALUE_SECS_BITS - 1)) - 1)
+#define HINT_VALUE_SECS_MIN (-(1LL << (HINT_VALUE_SECS_BITS - 1)))
#define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT -
HINT_VALUE_NSEC_BITS)
#define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
------------------------------------
But please see code moves which will be done in the next version of a
patchset, so all field and indices changes will become part of a single
patch.
Regards,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-17 23:43 ` Safin Timur via Tarantool-patches
@ 2021-08-18 9:03 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-18 9:03 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
18.08.2021 02:43, Safin Timur пишет:
> On 17.08.2021 15:16, Serge Petrenko wrote:
>>
>>
>> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>>> * storage hints implemented for datetime_t values;
>>> * proper comparison for indices of datetime type.
>>>
>>> Part of #5941
>>> Part of #5946
>>
>>
>> Please, add a docbot request stating that it's now possible to store
>> datetime values in spaces and create indexed datetime fields.
>
> Will use something like that:
>
> @TarantoolBot document
>
> Title: Storage support for datetime values
>
> It's now possible to store datetime values in spaces and create
> indexed datetime fields.
>
> Please refer to https://github.com/tarantool/tarantool/discussions/6244
> for more detailed description of a storage schema.
Looks ok to me.
>
>>
>>
>>> ---
>>> src/box/field_def.c | 18 ++++++++
>>> src/box/field_def.h | 3 ++
>>> src/box/memtx_space.c | 3 +-
>>> src/box/tuple_compare.cc | 57 ++++++++++++++++++++++++++
>>> src/box/vinyl.c | 3 +-
>>> test/engine/datetime.result | 77
>>> +++++++++++++++++++++++++++++++++++
>>> test/engine/datetime.test.lua | 35 ++++++++++++++++
>>> 7 files changed, 192 insertions(+), 4 deletions(-)
>>> create mode 100644 test/engine/datetime.result
>>> create mode 100644 test/engine/datetime.test.lua
>>>
>>> diff --git a/src/box/field_def.c b/src/box/field_def.c
>>> index 2682a42ee..97033d0bb 100644
>>> --- a/src/box/field_def.c
>>> +++ b/src/box/field_def.c
>>> @@ -194,3 +194,21 @@ field_type_by_name(const char *name, size_t len)
>>> return FIELD_TYPE_ANY;
>>> return field_type_MAX;
>>> }
>>> +
>>> +const bool field_type_index_allowed[] =
>>> + {
>>> + /* [FIELD_TYPE_ANY] = */ false,
>>> + /* [FIELD_TYPE_UNSIGNED] = */ true,
>>> + /* [FIELD_TYPE_STRING] = */ true,
>>> + /* [FIELD_TYPE_NUMBER] = */ true,
>>> + /* [FIELD_TYPE_DOUBLE] = */ true,
>>> + /* [FIELD_TYPE_INTEGER] = */ true,
>>> + /* [FIELD_TYPE_BOOLEAN] = */ true,
>>> + /* [FIELD_TYPE_VARBINARY]= */ true,
>>> + /* [FIELD_TYPE_SCALAR] = */ true,
>>> + /* [FIELD_TYPE_DECIMAL] = */ true,
>>> + /* [FIELD_TYPE_UUID] = */ true,
>>> + /* [FIELD_TYPE_ARRAY] = */ false,
>>> + /* [FIELD_TYPE_MAP] = */ false,
>>> + /* [FIELD_TYPE_DATETIME] = */ true,
>>> +};
>>
>>
>> You wouldn't need that array if you moved
>> FIELD_TYPE_DATETIME above FIELD_TYPE_ARRAY
>> in the previous commit.
>>
>> Please, do so.
>
> Yes, will change order and also move all field support code to this
> patch (as Vova recommends).
>
>>
>>
>>> diff --git a/src/box/field_def.h b/src/box/field_def.h
>>> index 120b2a93d..bd02418df 100644
>>> --- a/src/box/field_def.h
>>> +++ b/src/box/field_def.h
>>> @@ -120,6 +120,9 @@ extern const uint32_t field_ext_type[];
>>> extern const struct opt_def field_def_reg[];
>>> extern const struct field_def field_def_default;
>>> +/** helper table for checking allowed indices for types */
>>> +extern const bool field_type_index_allowed[];
>>> +
>>> /**
>>> * @brief Field definition
>>> * Contains information about of one tuple field.
>>> diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
>>> index b71318d24..1ab16122e 100644
>>> --- a/src/box/memtx_space.c
>>> +++ b/src/box/memtx_space.c
>>> @@ -748,8 +748,7 @@ memtx_space_check_index_def(struct space *space,
>>> struct index_def *index_def)
>>> /* Check that there are no ANY, ARRAY, MAP parts */
>>> for (uint32_t i = 0; i < key_def->part_count; i++) {
>>> struct key_part *part = &key_def->parts[i];
>>> - if (part->type <= FIELD_TYPE_ANY ||
>>> - part->type >= FIELD_TYPE_ARRAY) {
>>> + if (!field_type_index_allowed[part->type]) {
>>> diag_set(ClientError, ER_MODIFY_INDEX,
>>> index_def->name, space_name(space),
>>> tt_sprintf("field type '%s' is not supported",
>>> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
>>> index 9a69f2a72..110017853 100644
>>> --- a/src/box/tuple_compare.cc
>>> +++ b/src/box/tuple_compare.cc
>>> @@ -538,6 +538,8 @@ tuple_compare_field_with_type(const char
>>> *field_a, enum mp_type a_type,
>>> field_b, b_type);
>>> case FIELD_TYPE_UUID:
>>> return mp_compare_uuid(field_a, field_b);
>>> + case FIELD_TYPE_DATETIME:
>>> + return mp_compare_datetime(field_a, field_b);
>>> default:
>>> unreachable();
>>> return 0;
>>> @@ -1538,6 +1540,21 @@ func_index_compare_with_key(struct tuple
>>> *tuple, hint_t tuple_hint,
>>> #define HINT_VALUE_DOUBLE_MAX (exp2(HINT_VALUE_BITS - 1) - 1)
>>> #define HINT_VALUE_DOUBLE_MIN (-exp2(HINT_VALUE_BITS - 1))
>>> +/**
>>> + * We need to squeeze 64 bits of seconds and 32 bits of nanoseconds
>>> + * into 60 bits of hint value. The idea is to represent wide enough
>>> + * years range, and leave the rest of bits occupied from
>>> nanoseconds part:
>>> + * - 36 bits is enough for time range of [208BC..4147]
>>> + * - for nanoseconds there is left 24 bits, which are MSB part of
>>> + * 32-bit value
>>> + */
>>> +#define HINT_VALUE_SECS_BITS 36
>>> +#define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS -
>>> HINT_VALUE_SECS_BITS)
>>> +#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
>>
>> Am I missing something?
>> n bits may store values from (-2^(n-1)) to 2^(n-1)-1
>>
>> should be (1LL << (HINT_VALUE_SECS_BITS -1)) - 1 ?
>>
>>
>>> +#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
>>
>>
>>
>>
>> should be
>>
>> #define HINT_VALUE_SECS_MIN (-(1LL << (HINT_VALUE_SECS_BITS - 1)))
>>
>> ?
>>
>
> Yes, my definition made sense only when used as a mask (in prior
> version of a code). Thus did not take into consideration a sign bit.
> You absolutely correct that if seconds are signed then we have lesser
> number of bits, and your definitions of
> HINT_VALUE_SECS_MAX/HINT_VALUE_SECS_MIN should be used.
>
>>
>>> +#define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT -
>>> HINT_VALUE_NSEC_BITS)
>>> +#define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
>>> +
>>> /*
>>> * HINT_CLASS_BITS should be big enough to store any mp_class value.
>>> * Note, ((1 << HINT_CLASS_BITS) - 1) is reserved for HINT_NONE.
>>> @@ -1630,6 +1647,25 @@ hint_uuid_raw(const char *data)
>>> return hint_create(MP_CLASS_UUID, val);
>>> }
>>> +static inline hint_t
>>> +hint_datetime(struct datetime *date)
>>> +{
>>> + /*
>>> + * Use at most HINT_VALUE_SECS_BITS from datetime
>>> + * seconds field as a hint value, and at MSB part
>>> + * of HINT_VALUE_NSEC_BITS from nanoseconds.
>>> + */
>>> + int64_t secs = date->secs;
>>> + int32_t nsec = date->nsec;
>>> + uint64_t val = secs <= HINT_VALUE_SECS_MIN ? 0 :
>>> + secs - HINT_VALUE_SECS_MIN;
>>> + if (val >= HINT_VALUE_SECS_MAX)
>>> + val = HINT_VALUE_SECS_MAX;
>>> + val <<= HINT_VALUE_NSEC_BITS;
>>> + val |= (nsec >> HINT_VALUE_NSEC_SHIFT) & HINT_VALUE_NSEC_MAX;
>>> + return hint_create(MP_CLASS_DATETIME, val);
>>> +}
>>> +
>>
>> <stripped>
>>
>
> Patch increment here small (so far)
> ------------------------------------
> diff --git a/src/box/tuple_compare.cc b/src/box/tuple_compare.cc
> index 110017853..2478498ba 100644
> --- a/src/box/tuple_compare.cc
> +++ b/src/box/tuple_compare.cc
> @@ -1550,8 +1550,8 @@ func_index_compare_with_key(struct tuple *tuple,
> hint_t tuple_hint,
> */
> #define HINT_VALUE_SECS_BITS 36
> #define HINT_VALUE_NSEC_BITS (HINT_VALUE_BITS - HINT_VALUE_SECS_BITS)
> -#define HINT_VALUE_SECS_MAX ((1LL << HINT_VALUE_SECS_BITS) - 1)
> -#define HINT_VALUE_SECS_MIN (-(1LL << HINT_VALUE_SECS_BITS))
> +#define HINT_VALUE_SECS_MAX ((1LL << (HINT_VALUE_SECS_BITS - 1)) - 1)
> +#define HINT_VALUE_SECS_MIN (-(1LL << (HINT_VALUE_SECS_BITS - 1)))
> #define HINT_VALUE_NSEC_SHIFT (sizeof(int) * CHAR_BIT -
> HINT_VALUE_NSEC_BITS)
> #define HINT_VALUE_NSEC_MAX ((1ULL << HINT_VALUE_NSEC_BITS) - 1)
>
> ------------------------------------
>
> But please see code moves which will be done in the next version of a
> patchset, so all field and indices changes will become part of a
> single patch.
Sure, I'll check out the new version once it's pushed.
>
> Regards,
> Timur
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 19:05 ` Vladimir Davydov via Tarantool-patches
2021-08-18 17:18 ` Safin Timur via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 19:05 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:39AM +0300, Timur Safin via Tarantool-patches wrote:
> diff --git a/test/engine/datetime.test.lua b/test/engine/datetime.test.lua
> new file mode 100644
> index 000000000..3685e4d4b
> --- /dev/null
> +++ b/test/engine/datetime.test.lua
> @@ -0,0 +1,35 @@
> +env = require('test_run')
> +test_run = env.new()
> +engine = test_run:get_cfg('engine')
> +
> +date = require('datetime')
> +
> +_ = box.schema.space.create('T', {engine = engine})
> +_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
> +
> +box.space.T:insert{date('1970-01-01')}\
> +box.space.T:insert{date('1970-01-02')}\
> +box.space.T:insert{date('1970-01-03')}\
> +box.space.T:insert{date('2000-01-01')}
We need more tests. Off the top of my head:
- unique constraint
- multi-part indexes
- index:get(key) / index:select(key)
- index:replace
- index:update / index:upsert
- dates before unix epoch and very big dates
- hint corner cases (when two different dates have the same hint)
- snapshot and recovery
> +
> +o = box.space.T:select{}
> +assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
> +assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
> +assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
> +assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
> +
> +for i = 1,16 do\
> + box.space.T:insert{date.now()}\
> +end
Please don't use now(), because it'd make it difficult to reproduce
a failure.
> +
> +a = box.space.T:select{}
> +err = {}
> +for i = 1, #a - 1 do\
> + if a[i][1] >= a[i+1][1] then\
> + table.insert(err, {a[i][1], a[i+1][1]})\
> + break\
> + end\
> +end
> +
> +err
> +box.space.T:drop()
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices
2021-08-17 19:05 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 17:18 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 17:18 UTC (permalink / raw)
To: Vladimir Davydov, Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On 17.08.2021 22:05, Vladimir Davydov via Tarantool-patches wrote:
> On Mon, Aug 16, 2021 at 02:59:39AM +0300, Timur Safin via
> Tarantool-patches wrote:
>> diff --git a/test/engine/datetime.test.lua
> b/test/engine/datetime.test.lua
>> new file mode 100644
>> index 000000000..3685e4d4b
>> --- /dev/null
>> +++ b/test/engine/datetime.test.lua
>> @@ -0,0 +1,35 @@
>> +env = require('test_run')
>> +test_run = env.new()
>> +engine = test_run:get_cfg('engine')
>> +
>> +date = require('datetime')
>> +
>> +_ = box.schema.space.create('T', {engine = engine})
>> +_ = box.space.T:create_index('pk', {parts={1,'datetime'}})
>> +
>> +box.space.T:insert{date('1970-01-01')}\
>> +box.space.T:insert{date('1970-01-02')}\
>> +box.space.T:insert{date('1970-01-03')}\
>> +box.space.T:insert{date('2000-01-01')}
>
> We need more tests. Off the top of my head:
> - unique constraint
> - multi-part indexes
> - index:get(key) / index:select(key)
> - index:replace
> - index:update / index:upsert
> - dates before unix epoch and very big dates
> - hint corner cases (when two different dates have the same hint)
> - snapshot and recovery
Sigh. Could you please to show the good example for such kind of tests
for any data types? Because I modelled my tests after uuid, and decimal,
and I didn't see that much to be checked usually.
>
>> +
>> +o = box.space.T:select{}
>> +assert(tostring(o[1][1]) == '1970-01-01T00:00Z')
>> +assert(tostring(o[2][1]) == '1970-01-02T00:00Z')
>> +assert(tostring(o[3][1]) == '1970-01-03T00:00Z')
>> +assert(tostring(o[4][1]) == '2000-01-01T00:00Z')
>> +
>> +for i = 1,16 do\
>> + box.space.T:insert{date.now()}\
>> +end
>
> Please don't use now(), because it'd make it difficult to reproduce
> a failure.
>
>> +
>> +a = box.space.T:select{}
>> +err = {}
>> +for i = 1, #a - 1 do\
>> + if a[i][1] >= a[i+1][1] then\
>> + table.insert(err, {a[i][1], a[i+1][1]})\
>> + break\
>> + end\
>> +end
>> +
>> +err
>> +box.space.T:drop()
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (4 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 5/8] box, datetime: datetime comparison for indices Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 18:52 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 7/8] datetime: perf test for datetime parser Timur Safin via Tarantool-patches
` (3 subsequent siblings)
9 siblings, 2 replies; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
* created few entry points (months(N), years(N), days(N), etc.)
for easier datetime arithmetic;
* additions/subtractions of years/months use `dt_add_years()`
and `dt_add_months()` from 3rd party c-dt library;
* also there are `:add{}` and `:sub{}` methods in datetime
object to add or substract more complex intervals;
* introduced `is_datetime()` and `is_interval()` helpers for checking
of validity of passed arguments;
* human-readable stringization implemented for interval objects.
Note, that additions/subtractions completed for all _reasonable_
combinations of values of date and interval types;
Time + Interval => Time
Interval + Time => Time
Time - Time => Interval
Time - Interval => Time
Interval + Interval => Interval
Interval - Interval => Interval
Part of #5941
---
| 3 +
src/lua/datetime.lua | 349 ++++++++++++++++++++++++++++++++-
test/app-tap/datetime.test.lua | 159 ++++++++++++++-
3 files changed, 506 insertions(+), 5 deletions(-)
--git a/extra/exports b/extra/exports
index c34a5c2b5..421dda51f 100644
--- a/extra/exports
+++ b/extra/exports
@@ -157,6 +157,9 @@ datetime_to_string
datetime_unpack
decimal_from_string
decimal_unpack
+tnt_dt_add_years
+tnt_dt_add_quarters
+tnt_dt_add_months
tnt_dt_dow
tnt_dt_from_rdn
tnt_dt_from_struct_tm
diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
index 4d946f194..5fd0565ac 100644
--- a/src/lua/datetime.lua
+++ b/src/lua/datetime.lua
@@ -24,6 +24,17 @@ ffi.cdef [[
int tnt_dt_rdn (dt_t dt);
+ // dt_arithmetic.h
+ typedef enum {
+ DT_EXCESS,
+ DT_LIMIT,
+ DT_SNAP
+ } dt_adjust_t;
+
+ dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t adjust);
+ dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t adjust);
+ dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t adjust);
+
// dt_parse_iso.h
size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec);
@@ -50,8 +61,10 @@ ffi.cdef [[
local builtin = ffi.C
local math_modf = math.modf
+local math_floor = math.floor
local SECS_PER_DAY = 86400
+local NANOS_PER_SEC = 1000000000
-- c-dt/dt_config.h
@@ -62,8 +75,23 @@ local DT_EPOCH_1970_OFFSET = 719163
local datetime_t = ffi.typeof('struct datetime')
local interval_t = ffi.typeof('struct datetime_interval')
+ffi.cdef [[
+ struct interval_months {
+ int m;
+ };
+
+ struct interval_years {
+ int y;
+ };
+]]
+local interval_months_t = ffi.typeof('struct interval_months')
+local interval_years_t = ffi.typeof('struct interval_years')
+
local function is_interval(o)
- return type(o) == 'cdata' and ffi.istype(interval_t, o)
+ return type(o) == 'cdata' and
+ (ffi.istype(interval_t, o) or
+ ffi.istype(interval_months_t, o) or
+ ffi.istype(interval_years_t, o))
end
local function is_datetime(o)
@@ -72,7 +100,10 @@ end
local function is_date_interval(o)
return type(o) == 'cdata' and
- (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
+ (ffi.istype(datetime_t, o) or
+ ffi.istype(interval_t, o) or
+ ffi.istype(interval_months_t, o) or
+ ffi.istype(interval_years_t, o))
end
local function interval_new()
@@ -80,6 +111,13 @@ local function interval_new()
return interval
end
+local function check_number(n, message)
+ if type(n) ~= 'number' then
+ return error(("%s: expected number, but received %s"):
+ format(message, n), 2)
+ end
+end
+
local function check_date(o, message)
if not is_datetime(o) then
return error(("%s: expected datetime, but received %s"):
@@ -87,6 +125,20 @@ local function check_date(o, message)
end
end
+local function check_date_interval(o, message)
+ if not is_datetime(o) and not is_interval(o) then
+ return error(("%s: expected datetime or interval, but received %s"):
+ format(message, o), 2)
+ end
+end
+
+local function check_interval(o, message)
+ if not is_interval(o) then
+ return error(("%s: expected interval, but received %s"):
+ format(message, o), 2)
+ end
+end
+
local function check_str(s, message)
if not type(s) == 'string' then
return error(("%s: expected string, but received %s"):
@@ -102,6 +154,77 @@ local function check_range(v, range, txt)
end
end
+local function interval_years_new(y)
+ check_number(y, "years(number)")
+ local o = ffi.new(interval_years_t)
+ o.y = y
+ return o
+end
+
+local function interval_months_new(m)
+ check_number(m, "months(number)")
+ local o = ffi.new(interval_months_t)
+ o.m = m
+ return o
+end
+
+local function interval_weeks_new(w)
+ check_number(w, "weeks(number)")
+ local o = ffi.new(interval_t)
+ o.secs = w * SECS_PER_DAY * 7
+ return o
+end
+
+local function interval_days_new(d)
+ check_number(d, "days(number)")
+ local o = ffi.new(interval_t)
+ o.secs = d * SECS_PER_DAY
+ return o
+end
+
+local function interval_hours_new(h)
+ check_number(h, "hours(number)")
+ local o = ffi.new(interval_t)
+ o.secs = h * 60 * 60
+ return o
+end
+
+local function interval_minutes_new(m)
+ check_number(m, "minutes(number)")
+ local o = ffi.new(interval_t)
+ o.secs = m * 60
+ return o
+end
+
+local function interval_seconds_new(s)
+ check_number(s, "seconds(number)")
+ local o = ffi.new(interval_t)
+ o.nsec = s % 1 * 1e9
+ o.secs = s - (s % 1)
+ return o
+end
+
+local SECS_EPOCH_OFFSET = (DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
+
+local function local_rd(secs)
+ return math_floor((secs + SECS_EPOCH_OFFSET) / SECS_PER_DAY)
+end
+
+local function local_dt(secs)
+ return builtin.tnt_dt_from_rdn(local_rd(secs))
+end
+
+local function normalize_nsec(secs, nsec)
+ if nsec < 0 then
+ secs = secs - 1
+ nsec = nsec + NANOS_PER_SEC
+ elseif nsec >= NANOS_PER_SEC then
+ secs = secs + 1
+ nsec = nsec - NANOS_PER_SEC
+ end
+ return secs, nsec
+end
+
local function datetime_cmp(lhs, rhs)
if not is_date_interval(lhs) or
not is_date_interval(rhs) then
@@ -256,6 +379,10 @@ local function datetime_tostring(o)
local len = builtin.datetime_to_string(o, buff, sz)
assert(len < sz)
return ffi.string(buff)
+ elseif ffi.typeof(o) == interval_years_t then
+ return ('%+d years'):format(o.y)
+ elseif ffi.typeof(o) == interval_months_t then
+ return ('%+d months'):format(o.m)
elseif ffi.typeof(o) == interval_t then
local ts = o.timestamp
local sign = '+'
@@ -279,6 +406,126 @@ local function datetime_tostring(o)
end
end
+local function date_first(lhs, rhs)
+ if is_datetime(lhs) then
+ return lhs, rhs
+ else
+ return rhs, lhs
+ end
+end
+
+local function error_incompatible(name)
+ error(("datetime:%s() - incompatible type of arguments"):
+ format(name), 3)
+end
+
+--[[
+Matrix of subtraction operands eligibility and their result type
+
+| | datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime | interval | datetime | datetime | datetime |
+| interval | | interval | | |
+| interval_months | | | interval_months | |
+| interval_years | | | | interval_years |
+]]
+local function datetime_sub(lhs, rhs)
+ check_date_interval(lhs, "operator -")
+ local d, s = lhs, rhs
+ local left_t = ffi.typeof(d)
+ local right_t = ffi.typeof(s)
+ local o
+
+ if left_t == datetime_t then
+ -- 1. left is date, right is date or generic interval
+ if right_t == datetime_t or right_t == interval_t then
+ o = right_t == datetime_t and interval_new() or datetime_new()
+ o.secs, o.nsec = normalize_nsec(lhs.secs - rhs.secs,
+ lhs.nsec - rhs.nsec)
+ return o
+ -- 2. left is date, right is interval in months
+ elseif right_t == interval_months_t then
+ local dt = local_dt(lhs.secs)
+ dt = builtin.tnt_dt_add_months(dt, -rhs.m, builtin.DT_LIMIT)
+ return datetime_new_dt(dt, lhs.secs % SECS_PER_DAY,
+ lhs.nsec, lhs.offset)
+
+ -- 3. left is date, right is interval in years
+ elseif right_t == interval_years_t then
+ local dt = local_dt(lhs.secs)
+ dt = builtin.tnt_dt_add_years(dt, -rhs.y, builtin.DT_LIMIT)
+ return datetime_new_dt(dt, lhs.secs % SECS_PER_DAY,
+ lhs.nsec, lhs.offset)
+ else
+ error_incompatible("operator -")
+ end
+ -- 4. both left and right are generic intervals
+ elseif left_t == interval_t and right_t == interval_t then
+ o = interval_new()
+ o.secs, o.nsec = normalize_nsec(lhs.secs - rhs.secs,
+ lhs.nsec - rhs.nsec)
+ return o
+ -- 5. both left and right are intervals in months
+ elseif left_t == interval_months_t and right_t == interval_months_t then
+ return interval_months_new(lhs.m - rhs.m)
+ -- 5. both left and right are intervals in years
+ elseif left_t == interval_years_t and right_t == interval_years_t then
+ return interval_years_new(lhs.y - rhs.y)
+ else
+ error_incompatible("operator -")
+ end
+end
+
+--[[
+Matrix of addition operands eligibility and their result type
+
+| | datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime | datetime | datetime | datetime | datetime |
+| interval | datetime | interval | | |
+| interval_months | datetime | | interval_months | |
+| interval_years | datetime | | | interval_years |
+]]
+local function datetime_add(lhs, rhs)
+ local d, s = date_first(lhs, rhs)
+
+ check_date_interval(d, "operator +")
+ check_interval(s, "operator +")
+ local left_t = ffi.typeof(d)
+ local right_t = ffi.typeof(s)
+ local o
+
+ -- 1. left is date, right is date or interval
+ if left_t == datetime_t and right_t == interval_t then
+ o = datetime_new()
+ o.secs, o.nsec = normalize_nsec(d.secs + s.secs, d.nsec + s.nsec)
+ return o
+ -- 2. left is date, right is interval in months
+ elseif left_t == datetime_t and right_t == interval_months_t then
+ local dt = builtin.tnt_dt_add_months(local_dt(d.secs), s.m, builtin.DT_LIMIT)
+ local secs = d.secs % SECS_PER_DAY
+ return datetime_new_dt(dt, secs, d.nsec, d.offset or 0)
+
+ -- 3. left is date, right is interval in years
+ elseif left_t == datetime_t and right_t == interval_years_t then
+ local dt = builtin.tnt_dt_add_years(local_dt(d.secs), s.y, builtin.DT_LIMIT)
+ local secs = d.secs % SECS_PER_DAY
+ return datetime_new_dt(dt, secs, d.nsec, d.offset or 0)
+ -- 4. both left and right are generic intervals
+ elseif left_t == interval_t and right_t == interval_t then
+ o = interval_new()
+ o.secs, o.nsec = normalize_nsec(d.secs + s.secs, d.nsec + s.nsec)
+ return o
+ -- 5. both left and right are intervals in months
+ elseif left_t == interval_months_t and right_t == interval_months_t then
+ return interval_months_new(d.m + s.m)
+ -- 6. both left and right are intervals in years
+ elseif left_t == interval_years_t and right_t == interval_years_t then
+ return interval_years_new(d.y + s.y)
+ else
+ error_incompatible("operator +")
+ end
+end
--[[
Basic Extended
@@ -400,6 +647,75 @@ local function local_now()
return d
end
+-- addition or subtraction from date/time of a given interval
+-- described via table direction should be +1 or -1
+local function interval_increment(self, o, direction)
+ assert(direction == -1 or direction == 1)
+ local title = direction > 0 and "datetime.add" or "datetime.sub"
+ check_date(self, title)
+ if type(o) ~= 'table' then
+ error(('%s - object expected'):format(title), 2)
+ end
+
+ local ym_updated = false
+ local dhms_updated = false
+
+ local secs, nsec
+ secs, nsec = self.secs, self.nsec
+ -- operations with intervals should be done using human dates
+ -- not UTC dates, thus we normalize to UTC
+ local dt = local_dt(secs)
+
+ for key, value in pairs(o) do
+ if key == 'years' then
+ check_range(value, {0, 9999}, key)
+ dt = builtin.tnt_dt_add_years(dt, direction * value, builtin.DT_LIMIT)
+ ym_updated = true
+ elseif key == 'months' then
+ check_range(value, {0, 12}, key)
+ dt = builtin.tnt_dt_add_months(dt, direction * value, builtin.DT_LIMIT)
+ ym_updated = true
+ elseif key == 'weeks' then
+ check_range(value, {0, 52}, key)
+ secs = secs + direction * 7 * value * SECS_PER_DAY
+ dhms_updated = true
+ elseif key == 'days' then
+ check_range(value, {0, 31}, key)
+ secs = secs + direction * value * SECS_PER_DAY
+ dhms_updated = true
+ elseif key == 'hours' then
+ check_range(value, {0, 23}, key)
+ secs = secs + direction * 60 * 60 * value
+ dhms_updated = true
+ elseif key == 'minutes' then
+ check_range(value, {0, 59}, key)
+ secs = secs + direction * 60 * value
+ elseif key == 'seconds' then
+ check_range(value, {0, 60}, key)
+ local s, frac = math.modf(value)
+ secs = secs + direction * s
+ nsec = nsec + direction * frac * 1e9
+ dhms_updated = true
+ end
+ end
+
+ secs, nsec = normalize_nsec(secs, nsec)
+
+ -- .days, .hours, .minutes, .seconds
+ if dhms_updated then
+ self.secs = secs
+ self.nsec = nsec
+ end
+
+ -- .years, .months updated
+ if ym_updated then
+ self.secs = (builtin.tnt_dt_rdn(dt) - DT_EPOCH_1970_OFFSET) * SECS_PER_DAY +
+ secs % SECS_PER_DAY
+ end
+
+ return self
+end
+
-- Change the time-zone to the provided target_offset
-- Time `.secs`/`.nsec` are always UTC normalized, we need only to
-- reattribute object with different `.offset`
@@ -437,6 +753,14 @@ local function datetime_index(self, key)
return (self.secs + self.nsec / 1e9) / (60 * 60)
elseif key == 'd' or key == 'days' then
return (self.secs + self.nsec / 1e9) / (24 * 60 * 60)
+ elseif key == 'add' then
+ return function(self, obj)
+ return interval_increment(self, obj, 1)
+ end
+ elseif key == 'sub' then
+ return function(self, obj)
+ return interval_increment(self, obj, -1)
+ end
elseif key == 'to_utc' then
return function(self)
return datetime_to_tz(self, 0)
@@ -493,6 +817,8 @@ local datetime_mt = {
__eq = datetime_eq,
__lt = datetime_lt,
__le = datetime_le,
+ __sub = datetime_sub,
+ __add = datetime_add,
__index = datetime_index,
__newindex = datetime_newindex,
}
@@ -503,16 +829,35 @@ local interval_mt = {
__eq = datetime_eq,
__lt = datetime_lt,
__le = datetime_le,
+ __sub = datetime_sub,
+ __add = datetime_add,
+ __index = datetime_index,
+}
+
+local interval_tiny_mt = {
+ __tostring = datetime_tostring,
+ __serialize = interval_serialize,
+ __sub = datetime_sub,
+ __add = datetime_add,
__index = datetime_index,
}
ffi.metatype(interval_t, interval_mt)
ffi.metatype(datetime_t, datetime_mt)
+ffi.metatype(interval_years_t, interval_tiny_mt)
+ffi.metatype(interval_months_t, interval_tiny_mt)
return setmetatable(
{
new = datetime_new,
new_raw = datetime_new_obj,
+ years = interval_years_new,
+ months = interval_months_new,
+ weeks = interval_weeks_new,
+ days = interval_days_new,
+ hours = interval_hours_new,
+ minutes = interval_minutes_new,
+ seconds = interval_seconds_new,
interval = interval_new,
parse = parse,
diff --git a/test/app-tap/datetime.test.lua b/test/app-tap/datetime.test.lua
index 244ec2575..a2d7fac57 100755
--- a/test/app-tap/datetime.test.lua
+++ b/test/app-tap/datetime.test.lua
@@ -6,7 +6,7 @@ local date = require('datetime')
local ffi = require('ffi')
-test:plan(7)
+test:plan(10)
test:test("Simple tests for parser", function(test)
test:plan(2)
@@ -207,11 +207,164 @@ test:test("Parse tiny date into seconds and other parts", function(test)
test:ok(tiny.hours == 0.00848, "hours")
end)
-test:test("Stringization of date", function(test)
- test:plan(1)
+test:test("Stringization of dates and intervals", function(test)
+ test:plan(13)
local str = '19700101Z'
local dt = date(str)
test:ok(tostring(dt) == '1970-01-01T00:00Z', ('tostring(%s)'):format(str))
+ test:ok(tostring(date.seconds(12)) == '+12 secs', '+12 seconds')
+ test:ok(tostring(date.seconds(-12)) == '-12 secs', '-12 seconds')
+ test:ok(tostring(date.minutes(12)) == '+12 minutes, 0 seconds', '+12 minutes')
+ test:ok(tostring(date.minutes(-12)) == '-12 minutes, 0 seconds', '-12 minutes')
+ test:ok(tostring(date.hours(12)) == '+12 hours, 0 minutes, 0 seconds',
+ '+12 hours')
+ test:ok(tostring(date.hours(-12)) == '-12 hours, 0 minutes, 0 seconds',
+ '-12 hours')
+ test:ok(tostring(date.days(12)) == '+12 days, 0 hours, 0 minutes, 0 seconds',
+ '+12 days')
+ test:ok(tostring(date.days(-12)) == '-12 days, 0 hours, 0 minutes, 0 seconds',
+ '-12 days')
+ test:ok(tostring(date.months(5)) == '+5 months', '+5 months')
+ test:ok(tostring(date.months(-5)) == '-5 months', '-5 months')
+ test:ok(tostring(date.years(4)) == '+4 years', '+4 years')
+ test:ok(tostring(date.years(-4)) == '-4 years', '-4 years')
+end)
+
+test:test("Time interval operations", function(test)
+ test:plan(12)
+
+ -- check arithmetic with leap dates
+ local T = date('1972-02-29')
+ local M = date.months(2)
+ local Y = date.years(1)
+ test:ok(tostring(T + M) == '1972-04-29T00:00Z', ('T(%s) + M(%s'):format(T, M))
+ test:ok(tostring(T + Y) == '1973-03-01T00:00Z', ('T(%s) + Y(%s'):format(T, Y))
+ test:ok(tostring(T + M + Y) == '1973-04-30T00:00Z',
+ ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+ test:ok(tostring(T + Y + M) == '1973-05-01T00:00Z',
+ ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+ test:ok(tostring(T:add{years = 1, months = 2}) == '1973-04-30T00:00Z',
+ ('T:add{years=1,months=2}(%s)'):format(T))
+
+ -- check average, not leap dates
+ T = date('1970-01-08')
+ test:ok(tostring(T + M) == '1970-03-08T00:00Z', ('T(%s) + M(%s'):format(T, M))
+ test:ok(tostring(T + Y) == '1971-01-08T00:00Z', ('T(%s) + Y(%s'):format(T, Y))
+ test:ok(tostring(T + M + Y) == '1971-03-08T00:00Z',
+ ('T(%s) + M(%s) + Y(%s'):format(T, M, Y))
+ test:ok(tostring(T + Y + M) == '1971-03-08T00:00Z',
+ ('T(%s) + Y(%s) + M(%s'):format(T, Y, M))
+ test:ok(tostring(T:add{years = 1, months = 2}) == '1971-03-08T00:00Z',
+ ('T:add{years=1,months=2}(%s)'):format(T))
+
+
+ -- subtraction of 2 dates
+ local T2 = date('19700103')
+ local T1 = date('1970-01-01')
+ test:ok(tostring(T2 - T1) == '+2 days, 0 hours, 0 minutes, 0 seconds',
+ ('T2(%s) - T1(%s'):format(T2, T1))
+ test:ok(tostring(T1 - T2) == '-2 days, 0 hours, 0 minutes, 0 seconds',
+ ('T2(%s) - T1(%s'):format(T2, T1))
+end)
+
+local function catchadd(A, B)
+ return pcall(function() return A + B end)
+end
+
+--[[
+Matrix of addition operands eligibility and their result type
+
+| | datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime | datetime | datetime | datetime | datetime |
+| interval | datetime | interval | | |
+| interval_months | datetime | | interval_months | |
+| interval_years | datetime | | | interval_years |
+]]
+
+test:test("Matrix of allowed time and interval additions", function(test)
+ test:plan(20)
+
+ -- check arithmetic with leap dates
+ local T1970 = date('1970-01-01')
+ local T2000 = date('2000-01-01')
+ local I1 = date.days(1)
+ local M2 = date.months(2)
+ local M10 = date.months(10)
+ local Y1 = date.years(1)
+ local Y5 = date.years(5)
+
+ test:ok(catchadd(T1970, I1) == true, "status: T + I")
+ test:ok(catchadd(T1970, M2) == true, "status: T + M")
+ test:ok(catchadd(T1970, Y1) == true, "status: T + Y")
+ test:ok(catchadd(T1970, T2000) == false, "status: T + T")
+ test:ok(catchadd(I1, T1970) == true, "status: I + T")
+ test:ok(catchadd(M2, T1970) == true, "status: M + T")
+ test:ok(catchadd(Y1, T1970) == true, "status: Y + T")
+ test:ok(catchadd(I1, Y1) == false, "status: I + Y")
+ test:ok(catchadd(M2, Y1) == false, "status: M + Y")
+ test:ok(catchadd(I1, Y1) == false, "status: I + Y")
+ test:ok(catchadd(Y5, M10) == false, "status: Y + M")
+ test:ok(catchadd(Y5, I1) == false, "status: Y + I")
+ test:ok(catchadd(Y5, Y1) == true, "status: Y + Y")
+
+ test:ok(tostring(T1970 + I1) == "1970-01-02T00:00Z", "value: T + I")
+ test:ok(tostring(T1970 + M2) == "1970-03-01T00:00Z", "value: T + M")
+ test:ok(tostring(T1970 + Y1) == "1971-01-01T00:00Z", "value: T + Y")
+ test:ok(tostring(I1 + T1970) == "1970-01-02T00:00Z", "value: I + T")
+ test:ok(tostring(M2 + T1970) == "1970-03-01T00:00Z", "value: M + T")
+ test:ok(tostring(Y1 + T1970) == "1971-01-01T00:00Z", "value: Y + T")
+ test:ok(tostring(Y5 + Y1) == "+6 years", "Y + Y")
+
+end)
+
+local function catchsub_status(A, B)
+ return pcall(function() return A - B end)
+end
+
+--[[
+Matrix of subtraction operands eligibility and their result type
+
+| | datetime | interval | interval_months | interval_years |
++-----------------+-----------+----------+-----------------+----------------+
+| datetime | interval | datetime | datetime | datetime |
+| interval | | interval | | |
+| interval_months | | | interval_months | |
+| interval_years | | | | interval_years |
+]]
+test:test("Matrix of allowed time and interval subtractions", function(test)
+ test:plan(18)
+
+ -- check arithmetic with leap dates
+ local T1970 = date('1970-01-01')
+ local T2000 = date('2000-01-01')
+ local I1 = date.days(1)
+ local M2 = date.months(2)
+ local M10 = date.months(10)
+ local Y1 = date.years(1)
+ local Y5 = date.years(5)
+
+ test:ok(catchsub_status(T1970, I1) == true, "status: T - I")
+ test:ok(catchsub_status(T1970, M2) == true, "status: T - M")
+ test:ok(catchsub_status(T1970, Y1) == true, "status: T - Y")
+ test:ok(catchsub_status(T1970, T2000) == true, "status: T - T")
+ test:ok(catchsub_status(I1, T1970) == false, "status: I + T")
+ test:ok(catchsub_status(M2, T1970) == false, "status: M + T")
+ test:ok(catchsub_status(Y1, T1970) == false, "status: Y + T")
+ test:ok(catchsub_status(I1, Y1) == false, "status: I - Y")
+ test:ok(catchsub_status(M2, Y1) == false, "status: M - Y")
+ test:ok(catchsub_status(I1, Y1) == false, "status: I - Y")
+ test:ok(catchsub_status(Y5, M10) == false, "status: Y - M")
+ test:ok(catchsub_status(Y5, I1) == false, "status: Y - I")
+ test:ok(catchsub_status(Y5, Y1) == true, "status: Y - Y")
+
+ test:ok(tostring(T1970 - I1) == "1969-12-31T00:00Z", "value: T - I")
+ test:ok(tostring(T1970 - M2) == "1969-11-01T00:00Z", "value: T - M")
+ test:ok(tostring(T1970 - Y1) == "1969-01-01T00:00Z", "value: T - Y")
+ test:ok(tostring(T1970 - T2000) == "-10957 days, 0 hours, 0 minutes, 0 seconds",
+ "value: T - T")
+ test:ok(tostring(Y5 - Y1) == "+4 years", "value: Y - Y")
+
end)
os.exit(test:check() and 0 or 1)
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support Timur Safin via Tarantool-patches
@ 2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:44 ` Safin Timur via Tarantool-patches
2021-08-17 18:52 ` Vladimir Davydov via Tarantool-patches
1 sibling, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:16 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * created few entry points (months(N), years(N), days(N), etc.)
> for easier datetime arithmetic;
> * additions/subtractions of years/months use `dt_add_years()`
> and `dt_add_months()` from 3rd party c-dt library;
> * also there are `:add{}` and `:sub{}` methods in datetime
> object to add or substract more complex intervals;
> * introduced `is_datetime()` and `is_interval()` helpers for checking
> of validity of passed arguments;
> * human-readable stringization implemented for interval objects.
>
> Note, that additions/subtractions completed for all _reasonable_
> combinations of values of date and interval types;
>
> Time + Interval => Time
> Interval + Time => Time
> Time - Time => Interval
> Time - Interval => Time
> Interval + Interval => Interval
> Interval - Interval => Interval
>
> Part of #5941
> ---
> extra/exports | 3 +
> src/lua/datetime.lua | 349 ++++++++++++++++++++++++++++++++-
> test/app-tap/datetime.test.lua | 159 ++++++++++++++-
> 3 files changed, 506 insertions(+), 5 deletions(-)
>
> diff --git a/extra/exports b/extra/exports
> index c34a5c2b5..421dda51f 100644
> --- a/extra/exports
> +++ b/extra/exports
> @@ -157,6 +157,9 @@ datetime_to_string
> datetime_unpack
> decimal_from_string
> decimal_unpack
> +tnt_dt_add_years
> +tnt_dt_add_quarters
> +tnt_dt_add_months
> tnt_dt_dow
> tnt_dt_from_rdn
> tnt_dt_from_struct_tm
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 4d946f194..5fd0565ac 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -24,6 +24,17 @@ ffi.cdef [[
>
> int tnt_dt_rdn (dt_t dt);
>
> + // dt_arithmetic.h
> + typedef enum {
> + DT_EXCESS,
> + DT_LIMIT,
> + DT_SNAP
> + } dt_adjust_t;
> +
> + dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t adjust);
> + dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t adjust);
> + dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t adjust);
> +
Same about comments inside ffi.cdef. Better avoid them.
Please, split the cdef into reasonable blocks with
comments (when you need them)
between the blocks.
> // dt_parse_iso.h
> size_t tnt_dt_parse_iso_date (const char *str, size_t len, dt_t *dt);
> size_t tnt_dt_parse_iso_time (const char *str, size_t len, int *sod, int *nsec);
> @@ -50,8 +61,10 @@ ffi.cdef [[
>
> local builtin = ffi.C
> local math_modf = math.modf
> +local math_floor = math.floor
>
> local SECS_PER_DAY = 86400
> +local NANOS_PER_SEC = 1000000000
>
> -- c-dt/dt_config.h
>
>
>
>
Everything else looks fine.
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:44 ` Safin Timur via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:44 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
On 17.08.2021 15:16, Serge Petrenko wrote:
>
>> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
>> index 4d946f194..5fd0565ac 100644
>> --- a/src/lua/datetime.lua
>> +++ b/src/lua/datetime.lua
>> @@ -24,6 +24,17 @@ ffi.cdef [[
>> int tnt_dt_rdn (dt_t dt);
>> + // dt_arithmetic.h
>> + typedef enum {
>> + DT_EXCESS,
>> + DT_LIMIT,
>> + DT_SNAP
>> + } dt_adjust_t;
>> +
>> + dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t
>> adjust);
>> + dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t
>> adjust);
>> + dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t
>> adjust);
>> +
>
>
> Same about comments inside ffi.cdef. Better avoid them.
>
> Please, split the cdef into reasonable blocks with
> comments (when you need them)
> between the blocks.
I've split this code into several ffi.cdef blocks, so at the moment the
header of a module looks like this
-----------------------------------------------------------------
--[[
`c-dt` library functions handles properly both positive and
negative `dt`
values, where `dt` is a number of dates since Rata Die date
(0001-01-01).
For better compactness of our typical data in MessagePack stream we
shift
root of our time to the Unix Epoch date (1970-01-01), thus our 0 is
actually dt = 719163.
So here is a simple formula how convert our epoch-based seconds to
dt values
dt = (secs / 86400) + 719163
Where 719163 is an offset of Unix Epoch (1970-01-01) since Rata Die
(0001-01-01) in dates.
]]
-- dt_core.h definitions
ffi.cdef [[
typedef int dt_t;
dt_t tnt_dt_from_rdn (int n);
dt_t tnt_dt_from_ymd (int y, int m, int d);
int tnt_dt_rdn (dt_t dt);
]]
-- dt_arithmetic.h definitions
ffi.cdef [[
typedef enum {
DT_EXCESS,
DT_LIMIT,
DT_SNAP
} dt_adjust_t;
dt_t tnt_dt_add_years (dt_t dt, int delta, dt_adjust_t
adjust);
dt_t tnt_dt_add_quarters (dt_t dt, int delta, dt_adjust_t
adjust);
dt_t tnt_dt_add_months (dt_t dt, int delta, dt_adjust_t
adjust);
]]
-- dt_parse_iso.h definitions
ffi.cdef [[
size_t tnt_dt_parse_iso_date (const char *str, size_t len,
dt_t *dt);
size_t tnt_dt_parse_iso_time (const char *str, size_t len,
int *sod, int *nsec);
size_t tnt_dt_parse_iso_zone_lenient (const char *str, size_t len,
int *offset);
]]
-- Tarantool functions - datetime.c
ffi.cdef [[
int
datetime_to_string(const struct datetime * date, char *buf, int len);
char *
datetime_asctime(const struct datetime *date, char *buf);
char *
datetime_ctime(const struct datetime *date, char *buf);
size_t
datetime_strftime(const struct datetime *date, const char *fmt,
char *buf,
uint32_t len);
void
datetime_now(struct datetime * now);
]]
-----------------------------------------------------------------
>
>
>> // dt_parse_iso.h
>> size_t tnt_dt_parse_iso_date (const char *str, size_t
>> len, dt_t *dt);
>> size_t tnt_dt_parse_iso_time (const char *str, size_t
>> len, int *sod, int *nsec);
>> @@ -50,8 +61,10 @@ ffi.cdef [[
>> local builtin = ffi.C
>> local math_modf = math.modf
>> +local math_floor = math.floor
>> local SECS_PER_DAY = 86400
>> +local NANOS_PER_SEC = 1000000000
>> -- c-dt/dt_config.h
>>
>>
>
> Everything else looks fine.
>
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support Timur Safin via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 18:52 ` Vladimir Davydov via Tarantool-patches
1 sibling, 0 replies; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 18:52 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:40AM +0300, Timur Safin via Tarantool-patches wrote:
> diff --git a/src/lua/datetime.lua b/src/lua/datetime.lua
> index 4d946f194..5fd0565ac 100644
> --- a/src/lua/datetime.lua
> +++ b/src/lua/datetime.lua
> @@ -62,8 +75,23 @@ local DT_EPOCH_1970_OFFSET = 719163
> local datetime_t = ffi.typeof('struct datetime')
> local interval_t = ffi.typeof('struct datetime_interval')
>
> +ffi.cdef [[
> + struct interval_months {
> + int m;
> + };
> +
> + struct interval_years {
> + int y;
> + };
> +]]
> +local interval_months_t = ffi.typeof('struct interval_months')
> +local interval_years_t = ffi.typeof('struct interval_years')
I disagree that this is required. interval_years and interval_months are
very vague notions - they depend on the date they are applied to.
Besides, supporting them complicates the code. I think that a time
interval must be exact. At the same time, I agree it may be convenient
to add a few years or months to a date. Why not add datetime methods
(add_years, add_months) instead of introducing two new kinds of
intervals?
> +
> local function is_interval(o)
> - return type(o) == 'cdata' and ffi.istype(interval_t, o)
> + return type(o) == 'cdata' and
> + (ffi.istype(interval_t, o) or
> + ffi.istype(interval_months_t, o) or
> + ffi.istype(interval_years_t, o))
> end
>
> local function is_datetime(o)
> @@ -72,7 +100,10 @@ end
>
> local function is_date_interval(o)
> return type(o) == 'cdata' and
> - (ffi.istype(interval_t, o) or ffi.istype(datetime_t, o))
> + (ffi.istype(datetime_t, o) or
> + ffi.istype(interval_t, o) or
> + ffi.istype(interval_months_t, o) or
> + ffi.istype(interval_years_t, o))
> end
>
> local function interval_new()
> @@ -80,6 +111,13 @@ local function interval_new()
> return interval
> end
>
> +local function check_number(n, message)
> + if type(n) ~= 'number' then
> + return error(("%s: expected number, but received %s"):
> + format(message, n), 2)
> + end
> +end
> +
> local function check_date(o, message)
> if not is_datetime(o) then
> return error(("%s: expected datetime, but received %s"):
> @@ -87,6 +125,20 @@ local function check_date(o, message)
> end
> end
>
> +local function check_date_interval(o, message)
> + if not is_datetime(o) and not is_interval(o) then
> + return error(("%s: expected datetime or interval, but received %s"):
> + format(message, o), 2)
> + end
> +end
> +
> +local function check_interval(o, message)
> + if not is_interval(o) then
> + return error(("%s: expected interval, but received %s"):
> + format(message, o), 2)
> + end
> +end
> +
> local function check_str(s, message)
> if not type(s) == 'string' then
> return error(("%s: expected string, but received %s"):
> @@ -102,6 +154,77 @@ local function check_range(v, range, txt)
> end
> end
>
> +local function interval_years_new(y)
> + check_number(y, "years(number)")
> + local o = ffi.new(interval_years_t)
> + o.y = y
> + return o
> +end
> +
> +local function interval_months_new(m)
> + check_number(m, "months(number)")
> + local o = ffi.new(interval_months_t)
> + o.m = m
> + return o
> +end
> +
> +local function interval_weeks_new(w)
> + check_number(w, "weeks(number)")
> + local o = ffi.new(interval_t)
> + o.secs = w * SECS_PER_DAY * 7
> + return o
> +end
> +
> +local function interval_days_new(d)
> + check_number(d, "days(number)")
> + local o = ffi.new(interval_t)
> + o.secs = d * SECS_PER_DAY
> + return o
> +end
> +
> +local function interval_hours_new(h)
> + check_number(h, "hours(number)")
> + local o = ffi.new(interval_t)
> + o.secs = h * 60 * 60
> + return o
> +end
> +
> +local function interval_minutes_new(m)
> + check_number(m, "minutes(number)")
> + local o = ffi.new(interval_t)
> + o.secs = m * 60
> + return o
> +end
> +
> +local function interval_seconds_new(s)
> + check_number(s, "seconds(number)")
> + local o = ffi.new(interval_t)
> + o.nsec = s % 1 * 1e9
> + o.secs = s - (s % 1)
> + return o
> +end
These functions should have been added by the patch that introduced the
datetime library to Lua, because without them the library doesn't seem
to be complete. Doing this in a separate patch doesn't ease review - in
fact it only confuses me as a reviewer, because now I have to jump
between two patches to see the whole picture. In general, please try to
split a patch set in such a way that each patch may be applied before
the rest without breaking anything, introducing dead code or a
half-working feature.
IMO the series should be split as follows:
1. Add datetime library to Lua + unit tests + Lua tests.
2. Add datetime serialization to msgpack/yaml/json + Lua tests.
3. Add datetime indexing support + Lua tests.
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 7/8] datetime: perf test for datetime parser
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (5 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 6/8] lua, datetime: time intervals support Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 19:13 ` Vladimir Davydov via Tarantool-patches
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module Timur Safin via Tarantool-patches
` (2 subsequent siblings)
9 siblings, 1 reply; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
It was told that if field `datetime.secs` would be `double` we should get
better performance in LuaJIT instead of `uint64_t` type, which is used at the
moment.
So we have created benchmark, which was comparing implementations of functions
from `datetime.c` if we would use `double` or `int64_t` for `datetime.secs` field.
Despite expectations, based on prior experience with floaing-point on x86
processors, comparison shows that `double` provides similar or
sometimes better timings. And picture stays consistent be it SSE2, AVX1 or
AVX2 code.
Part of #5941
---
perf/CMakeLists.txt | 3 +
perf/datetime-common.h | 105 +++++++++++++++++++
perf/datetime-compare.cc | 213 +++++++++++++++++++++++++++++++++++++++
perf/datetime-parser.cc | 105 +++++++++++++++++++
4 files changed, 426 insertions(+)
create mode 100644 perf/datetime-common.h
create mode 100644 perf/datetime-compare.cc
create mode 100644 perf/datetime-parser.cc
diff --git a/perf/CMakeLists.txt b/perf/CMakeLists.txt
index 3651de5b4..b5d7caf81 100644
--- a/perf/CMakeLists.txt
+++ b/perf/CMakeLists.txt
@@ -12,3 +12,6 @@ include_directories(${CMAKE_SOURCE_DIR}/third_party)
add_executable(tuple.perftest tuple.cc)
target_link_libraries(tuple.perftest core box tuple benchmark::benchmark)
+
+add_executable(datetime.perftest datetime-parser.cc datetime-compare.cc)
+target_link_libraries(datetime.perftest cdt core benchmark::benchmark)
diff --git a/perf/datetime-common.h b/perf/datetime-common.h
new file mode 100644
index 000000000..6fd4e1e3b
--- /dev/null
+++ b/perf/datetime-common.h
@@ -0,0 +1,105 @@
+#include <assert.h>
+#include <stdint.h>
+#include <string.h>
+#include <benchmark/benchmark.h>
+
+#include "dt.h"
+#include "datetime.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+static const char sample[] = "2012-12-24T15:30Z";
+
+#define S(s) \
+ { s, sizeof(s) - 1 }
+
+static struct
+{
+ const char *sz;
+ size_t len;
+} tests[] = {
+ S("2012-12-24 15:30Z"),
+ S("2012-12-24 15:30z"),
+ S("2012-12-24 15:30"),
+ S("2012-12-24 16:30+01:00"),
+ S("2012-12-24 16:30+0100"),
+ S("2012-12-24 16:30+01"),
+ S("2012-12-24 14:30-01:00"),
+ S("2012-12-24 14:30-0100"),
+ S("2012-12-24 14:30-01"),
+ S("2012-12-24 15:30:00Z"),
+ S("2012-12-24 15:30:00z"),
+ S("2012-12-24 15:30:00"),
+ S("2012-12-24 16:30:00+01:00"),
+ S("2012-12-24 16:30:00+0100"),
+ S("2012-12-24 14:30:00-01:00"),
+ S("2012-12-24 14:30:00-0100"),
+ S("2012-12-24 15:30:00.123456Z"),
+ S("2012-12-24 15:30:00.123456z"),
+ S("2012-12-24 15:30:00.123456"),
+ S("2012-12-24 16:30:00.123456+01:00"),
+ S("2012-12-24 16:30:00.123456+01"),
+ S("2012-12-24 14:30:00.123456-01:00"),
+ S("2012-12-24 14:30:00.123456-01"),
+ S("2012-12-24t15:30Z"),
+ S("2012-12-24t15:30z"),
+ S("2012-12-24t15:30"),
+ S("2012-12-24t16:30+01:00"),
+ S("2012-12-24t16:30+0100"),
+ S("2012-12-24t14:30-01:00"),
+ S("2012-12-24t14:30-0100"),
+ S("2012-12-24t15:30:00Z"),
+ S("2012-12-24t15:30:00z"),
+ S("2012-12-24t15:30:00"),
+ S("2012-12-24t16:30:00+01:00"),
+ S("2012-12-24t16:30:00+0100"),
+ S("2012-12-24t14:30:00-01:00"),
+ S("2012-12-24t14:30:00-0100"),
+ S("2012-12-24t15:30:00.123456Z"),
+ S("2012-12-24t15:30:00.123456z"),
+ S("2012-12-24t16:30:00.123456+01:00"),
+ S("2012-12-24t14:30:00.123456-01:00"),
+ S("2012-12-24 16:30 +01:00"),
+ S("2012-12-24 14:30 -01:00"),
+ S("2012-12-24 15:30 UTC"),
+ S("2012-12-24 16:30 UTC+1"),
+ S("2012-12-24 16:30 UTC+01"),
+ S("2012-12-24 16:30 UTC+0100"),
+ S("2012-12-24 16:30 UTC+01:00"),
+ S("2012-12-24 14:30 UTC-1"),
+ S("2012-12-24 14:30 UTC-01"),
+ S("2012-12-24 14:30 UTC-01:00"),
+ S("2012-12-24 14:30 UTC-0100"),
+ S("2012-12-24 15:30 GMT"),
+ S("2012-12-24 16:30 GMT+1"),
+ S("2012-12-24 16:30 GMT+01"),
+ S("2012-12-24 16:30 GMT+0100"),
+ S("2012-12-24 16:30 GMT+01:00"),
+ S("2012-12-24 14:30 GMT-1"),
+ S("2012-12-24 14:30 GMT-01"),
+ S("2012-12-24 14:30 GMT-01:00"),
+ S("2012-12-24 14:30 GMT-0100"),
+ S("2012-12-24 14:30 -01:00"),
+ S("2012-12-24 16:30:00 +01:00"),
+ S("2012-12-24 14:30:00 -01:00"),
+ S("2012-12-24 16:30:00.123456 +01:00"),
+ S("2012-12-24 14:30:00.123456 -01:00"),
+ S("2012-12-24 15:30:00.123456 -00:00"),
+ S("20121224T1630+01:00"),
+ S("2012-12-24T1630+01:00"),
+ S("20121224T16:30+01"),
+ S("20121224T16:30 +01"),
+};
+#undef S
+
+#define DIM(a) (sizeof(a) / sizeof(a[0]))
+
+int
+parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
+ int32_t *op);
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/perf/datetime-compare.cc b/perf/datetime-compare.cc
new file mode 100644
index 000000000..6e561a5eb
--- /dev/null
+++ b/perf/datetime-compare.cc
@@ -0,0 +1,213 @@
+#include "dt.h"
+#include <string.h>
+#include <assert.h>
+#include <limits.h>
+
+#include "datetime-common.h"
+
+template <typename T>
+struct datetime_bench
+{
+ T secs;
+ uint32_t nsec;
+ uint32_t offset;
+
+static struct datetime_bench date_array[];
+};
+template<typename T>
+struct datetime_bench<T> datetime_bench<T>::date_array[DIM(tests)];
+
+/// Parse 70 datetime literals of various lengths
+template <typename T>
+static void
+Assign70()
+{
+ size_t index;
+ int64_t secs_expected;
+ int nanosecs;
+ int ofs;
+ using dt_bench = datetime_bench<T>;
+
+ for (index = 0; index < DIM(tests); index++) {
+ int64_t secs;
+ int rc = parse_datetime(tests[index].sz, tests[index].len,
+ &secs, &nanosecs, &ofs);
+ assert(rc == 0);
+ dt_bench::date_array[index].secs = (T)secs;
+ dt_bench::date_array[index].nsec = nanosecs;
+ dt_bench::date_array[index].offset = ofs;
+ }
+}
+
+template <typename T>
+static void
+DateTime_Assign70(benchmark::State &state)
+{
+ for (auto _ : state)
+ Assign70<T>();
+}
+BENCHMARK_TEMPLATE1(DateTime_Assign70, uint64_t);
+BENCHMARK_TEMPLATE1(DateTime_Assign70, double);
+
+#define COMPARE_RESULT_BENCH(a, b) (a < b ? -1 : a > b)
+
+template <typename T>
+int datetime_compare(const struct datetime_bench<T> *lhs,
+ const struct datetime_bench<T> *rhs)
+{
+ int result = COMPARE_RESULT_BENCH(lhs->secs, rhs->secs);
+ if (result != 0)
+ return result;
+
+ return COMPARE_RESULT_BENCH(lhs->nsec, rhs->nsec);
+}
+
+template <typename T>
+static void
+AssignCompare70()
+{
+ size_t index;
+ int nanosecs;
+ int ofs;
+ using dt_bench = datetime_bench<T>;
+
+ size_t arrays_sz = DIM(tests);
+ for (index = 0; index < arrays_sz; index++) {
+ int64_t secs;
+ int rc = parse_datetime(tests[index].sz, tests[index].len,
+ &secs, &nanosecs, &ofs);
+ assert(rc == 0);
+ dt_bench::date_array[index].secs = (T)secs;
+ dt_bench::date_array[index].nsec = nanosecs;
+ dt_bench::date_array[index].offset = ofs;
+ }
+
+ for (index = 0; index < (arrays_sz - 1); index++) {
+ volatile int rc = datetime_compare<T>(&dt_bench::date_array[index],
+ &dt_bench::date_array[index + 1]);
+ assert(rc == 0 || rc == -1 || rc == 1);
+ }
+}
+
+template <typename T>
+static void
+DateTime_AssignCompare70(benchmark::State &state)
+{
+ for (auto _ : state)
+ AssignCompare70<T>();
+}
+BENCHMARK_TEMPLATE1(DateTime_AssignCompare70, uint64_t);
+BENCHMARK_TEMPLATE1(DateTime_AssignCompare70, double);
+
+template <typename T>
+static void
+Compare20()
+{
+ size_t index;
+ int nanosecs;
+ int ofs;
+ using dt_bench = datetime_bench<T>;
+
+ for (size_t i = 0; i < 10; i++) {
+ volatile int rc = datetime_compare<T>(&dt_bench::date_array[i],
+ &dt_bench::date_array[32 + i]);
+ assert(rc == 0 || rc == -1 || rc == 1);
+ }
+}
+
+template <typename T>
+static void
+DateTime_Compare20(benchmark::State &state)
+{
+ for (auto _ : state)
+ Compare20<T>();
+}
+BENCHMARK_TEMPLATE1(DateTime_Compare20, uint64_t);
+BENCHMARK_TEMPLATE1(DateTime_Compare20, double);
+
+
+#define SECS_EPOCH_1970_OFFSET ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY)
+
+template<typename T>
+int
+datetime_to_string(const struct datetime_bench<T> *date, char *buf, uint32_t len)
+{
+#define ADVANCE(sz) \
+ if (buf != NULL) { \
+ buf += sz; \
+ len -= sz; \
+ } \
+ ret += sz;
+
+ int offset = date->offset;
+ /* for negative offsets around Epoch date we could get
+ * negative secs value, which should be attributed to
+ * 1969-12-31, not 1970-01-01, thus we first shift
+ * epoch to Rata Die then divide by seconds per day,
+ * not in reverse
+ */
+ int64_t secs = (int64_t)date->secs + offset * 60 + SECS_EPOCH_1970_OFFSET;
+ assert((secs / SECS_PER_DAY) <= INT_MAX);
+ dt_t dt = dt_from_rdn(secs / SECS_PER_DAY);
+
+ int year, month, day, sec, ns, sign;
+ dt_to_ymd(dt, &year, &month, &day);
+
+ int hour = (secs / 3600) % 24,
+ minute = (secs / 60) % 60;
+ sec = secs % 60;
+ ns = date->nsec;
+
+ int ret = 0;
+ uint32_t sz = snprintf(buf, len, "%04d-%02d-%02dT%02d:%02d",
+ year, month, day, hour, minute);
+ ADVANCE(sz);
+ if (sec || ns) {
+ sz = snprintf(buf, len, ":%02d", sec);
+ ADVANCE(sz);
+ if (ns) {
+ if ((ns % 1000000) == 0)
+ sz = snprintf(buf, len, ".%03d", ns / 1000000);
+ else if ((ns % 1000) == 0)
+ sz = snprintf(buf, len, ".%06d", ns / 1000);
+ else
+ sz = snprintf(buf, len, ".%09d", ns);
+ ADVANCE(sz);
+ }
+ }
+ if (offset == 0) {
+ sz = snprintf(buf, len, "Z");
+ ADVANCE(sz);
+ }
+ else {
+ if (offset < 0)
+ sign = '-', offset = -offset;
+ else
+ sign = '+';
+
+ sz = snprintf(buf, len, "%c%02d:%02d", sign, offset / 60, offset % 60);
+ ADVANCE(sz);
+ }
+ return ret;
+}
+#undef ADVANCE
+
+template <typename T>
+static void
+ToString1()
+{
+ char buf[48];
+ struct datetime_bench<T> dateval = datetime_bench<T>::date_array[13];
+
+ volatile auto len = datetime_to_string<T>(&dateval, buf, sizeof buf);
+}
+
+template <typename T>
+static void
+DateTime_ToString1(benchmark::State &state)
+{
+ for (auto _ : state)
+ ToString1<T>();
+}
+BENCHMARK_TEMPLATE1(DateTime_ToString1, uint64_t);
+BENCHMARK_TEMPLATE1(DateTime_ToString1, double);
diff --git a/perf/datetime-parser.cc b/perf/datetime-parser.cc
new file mode 100644
index 000000000..61557fe8f
--- /dev/null
+++ b/perf/datetime-parser.cc
@@ -0,0 +1,105 @@
+#include "dt.h"
+#include <string.h>
+#include <assert.h>
+
+#include "datetime-common.h"
+
+/* p5-time-moment/src/moment_parse.c: parse_string_lenient() */
+int
+parse_datetime(const char *str, size_t len, int64_t *sp, int32_t *np,
+ int32_t *op)
+{
+ size_t n;
+ dt_t dt;
+ char c;
+ int sod = 0, nanosecond = 0, offset = 0;
+
+ n = dt_parse_iso_date(str, len, &dt);
+ if (!n)
+ return 1;
+ if (n == len)
+ goto exit;
+
+ c = str[n++];
+ if (!(c == 'T' || c == 't' || c == ' '))
+ return 1;
+
+ str += n;
+ len -= n;
+
+ n = dt_parse_iso_time(str, len, &sod, &nanosecond);
+ if (!n)
+ return 1;
+ if (n == len)
+ goto exit;
+
+ if (str[n] == ' ')
+ n++;
+
+ str += n;
+ len -= n;
+
+ n = dt_parse_iso_zone_lenient(str, len, &offset);
+ if (!n || n != len)
+ return 1;
+
+exit:
+ *sp = ((int64_t)dt_rdn(dt) - 719163) * 86400 + sod - offset * 60;
+ *np = nanosecond;
+ *op = offset;
+
+ return 0;
+}
+
+/// Parse 70 datetime literals of various lengths
+static void
+ParseTimeStamps()
+{
+ size_t index;
+ int64_t secs_expected;
+ int nanosecs;
+ int ofs;
+ parse_datetime(sample, sizeof(sample) - 1, &secs_expected,
+ &nanosecs, &ofs);
+
+ for (index = 0; index < DIM(tests); index++)
+ {
+ int64_t secs;
+ int rc = parse_datetime(tests[index].sz, tests[index].len,
+ &secs, &nanosecs, &ofs);
+ assert(rc == 0);
+ assert(secs == secs_expected);
+ }
+}
+
+static void
+CDT_Parse70(benchmark::State &state)
+{
+ for (auto _ : state)
+ ParseTimeStamps();
+}
+BENCHMARK(CDT_Parse70);
+
+/// Parse single datetime literal of longest length
+static void
+Parse1()
+{
+ const char civil_string[] = "2015-02-18T10:50:31.521345123+10:00";
+ int64_t secs;
+ int nanosecs;
+ int ofs;
+ int rc = parse_datetime(civil_string, sizeof(civil_string) - 1,
+ &secs, &nanosecs, &ofs);
+ assert(rc == 0);
+ assert(nanosecs == 521345123);
+}
+
+static void
+CDT_Parse1(benchmark::State &state)
+{
+ for (auto _ : state)
+ Parse1();
+}
+BENCHMARK(CDT_Parse1);
+
+BENCHMARK_MAIN();
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 7/8] datetime: perf test for datetime parser
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 7/8] datetime: perf test for datetime parser Timur Safin via Tarantool-patches
@ 2021-08-17 19:13 ` Vladimir Davydov via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-17 19:13 UTC (permalink / raw)
To: Timur Safin via Tarantool-patches; +Cc: v.shpilevoy
On Mon, Aug 16, 2021 at 02:59:41AM +0300, Timur Safin via Tarantool-patches wrote:
> It was told that if field `datetime.secs` would be `double` we should get
> better performance in LuaJIT instead of `uint64_t` type, which is used at the
> moment.
>
> So we have created benchmark, which was comparing implementations of functions
> from `datetime.c` if we would use `double` or `int64_t` for `datetime.secs` field.
>
> Despite expectations, based on prior experience with floaing-point on x86
> processors, comparison shows that `double` provides similar or
> sometimes better timings. And picture stays consistent be it SSE2, AVX1 or
> AVX2 code.
>
> Part of #5941
I don't think this belongs in the repository. My understanding of the
perf tests committed to the repository (may be wrong) is that they are
supposed to be used to ensure there's no performance degradation of a
certain subsystem. Ideally, they should be run automatically after each
commit. The test added by this patch just checks what implementation of
datetime is faster - with double or int64_t.
^ permalink raw reply [flat|nested] 50+ messages in thread
* [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (6 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 7/8] datetime: perf test for datetime parser Timur Safin via Tarantool-patches
@ 2021-08-15 23:59 ` Timur Safin via Tarantool-patches
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 12:15 ` [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Serge Petrenko via Tarantool-patches
[not found] ` <20210818082222.mofgheciutpipelz@esperanza>
9 siblings, 1 reply; 50+ messages in thread
From: Timur Safin via Tarantool-patches @ 2021-08-15 23:59 UTC (permalink / raw)
To: v.shpilevoy, imun, imeevma, tarantool-patches
Introduced new date/time/interval types support to lua and storage engines.
Closes #5941
Closes #5946
---
changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++
1 file changed, 4 insertions(+)
create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md b/changelogs/unreleased/gh-5941-datetime-type-support.md
new file mode 100644
index 000000000..3c755008e
--- /dev/null
+++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
@@ -0,0 +1,4 @@
+## feature/lua/datetime
+
+ * Introduce new builtin module for date/time/interval support - `datetime.lua`.
+ Support of new datetime type in storage engines (gh-5941, gh-5946).
--
2.29.2
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module Timur Safin via Tarantool-patches
@ 2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
2021-08-17 23:44 ` Safin Timur via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:16 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> Introduced new date/time/interval types support to lua and storage engines.
>
> Closes #5941
> Closes #5946
> ---
> changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++
> 1 file changed, 4 insertions(+)
> create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
>
> diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md b/changelogs/unreleased/gh-5941-datetime-type-support.md
> new file mode 100644
> index 000000000..3c755008e
> --- /dev/null
> +++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
> @@ -0,0 +1,4 @@
> +## feature/lua/datetime
> +
> + * Introduce new builtin module for date/time/interval support - `datetime.lua`.
> + Support of new datetime type in storage engines (gh-5941, gh-5946).
I'd extract the second line into a separate bullet.
Up to you.
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module
2021-08-17 12:16 ` Serge Petrenko via Tarantool-patches
@ 2021-08-17 23:44 ` Safin Timur via Tarantool-patches
2021-08-18 9:04 ` Serge Petrenko via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-17 23:44 UTC (permalink / raw)
To: Serge Petrenko; +Cc: tarantool-patches, v.shpilevoy
On 17.08.2021 15:16, Serge Petrenko wrote:
>
>
> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>> Introduced new date/time/interval types support to lua and storage
>> engines.
>>
>> Closes #5941
>> Closes #5946
>> ---
>> changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++
>> 1 file changed, 4 insertions(+)
>> create mode 100644
>> changelogs/unreleased/gh-5941-datetime-type-support.md
>>
>> diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md
>> b/changelogs/unreleased/gh-5941-datetime-type-support.md
>> new file mode 100644
>> index 000000000..3c755008e
>> --- /dev/null
>> +++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
>> @@ -0,0 +1,4 @@
>> +## feature/lua/datetime
>> +
>> + * Introduce new builtin module for date/time/interval support -
>> `datetime.lua`.
>> + Support of new datetime type in storage engines (gh-5941, gh-5946).
>
> I'd extract the second line into a separate bullet.
> Up to you.
>
>
Made some changes
-----------------------------------------------------------------
diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md
b/changelogs/unreleased/gh-5941-datetime-type-support.md
index 3c755008e..fb1f23077 100644
--- a/changelogs/unreleased/gh-5941-datetime-type-support.md
+++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
@@ -1,4 +1,6 @@
## feature/lua/datetime
- * Introduce new builtin module for date/time/interval support -
`datetime.lua`.
- Support of new datetime type in storage engines (gh-5941, gh-5946).
+ * Introduce new builtin module `datetime.lua` for date, time, and interval
+ support;
+ * Support of a new datetime type in storage engines allows to store
datetime
+ fields and build indices with them (gh-5941, gh-5946).
-----------------------------------------------------------------
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module
2021-08-17 23:44 ` Safin Timur via Tarantool-patches
@ 2021-08-18 9:04 ` Serge Petrenko via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-18 9:04 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
18.08.2021 02:44, Safin Timur пишет:
> On 17.08.2021 15:16, Serge Petrenko wrote:
>>
>>
>> 16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
>>> Introduced new date/time/interval types support to lua and storage
>>> engines.
>>>
>>> Closes #5941
>>> Closes #5946
>>> ---
>>> changelogs/unreleased/gh-5941-datetime-type-support.md | 4 ++++
>>> 1 file changed, 4 insertions(+)
>>> create mode 100644
>>> changelogs/unreleased/gh-5941-datetime-type-support.md
>>>
>>> diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md
>>> b/changelogs/unreleased/gh-5941-datetime-type-support.md
>>> new file mode 100644
>>> index 000000000..3c755008e
>>> --- /dev/null
>>> +++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
>>> @@ -0,0 +1,4 @@
>>> +## feature/lua/datetime
>>> +
>>> + * Introduce new builtin module for date/time/interval support -
>>> `datetime.lua`.
>>> + Support of new datetime type in storage engines (gh-5941, gh-5946).
>>
>> I'd extract the second line into a separate bullet.
>> Up to you.
>>
>>
>
> Made some changes
> -----------------------------------------------------------------
> diff --git a/changelogs/unreleased/gh-5941-datetime-type-support.md
> b/changelogs/unreleased/gh-5941-datetime-type-support.md
> index 3c755008e..fb1f23077 100644
> --- a/changelogs/unreleased/gh-5941-datetime-type-support.md
> +++ b/changelogs/unreleased/gh-5941-datetime-type-support.md
> @@ -1,4 +1,6 @@
> ## feature/lua/datetime
>
> - * Introduce new builtin module for date/time/interval support -
> `datetime.lua`.
> - Support of new datetime type in storage engines (gh-5941, gh-5946).
> + * Introduce new builtin module `datetime.lua` for date, time, and
> interval
> + support;
> + * Support of a new datetime type in storage engines allows to store
> datetime
> + fields and build indices with them (gh-5941, gh-5946).
> -----------------------------------------------------------------
>
> Thanks,
> Timur
Looks good, thanks!
This may be squashed into the 6th commit. Up to you.
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread
* Re: [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation
2021-08-15 23:59 [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation Timur Safin via Tarantool-patches
` (7 preceding siblings ...)
2021-08-15 23:59 ` [Tarantool-patches] [PATCH v5 8/8] datetime: changelog for datetime module Timur Safin via Tarantool-patches
@ 2021-08-17 12:15 ` Serge Petrenko via Tarantool-patches
[not found] ` <20210818082222.mofgheciutpipelz@esperanza>
9 siblings, 0 replies; 50+ messages in thread
From: Serge Petrenko via Tarantool-patches @ 2021-08-17 12:15 UTC (permalink / raw)
To: Timur Safin; +Cc: tarantool-patches, v.shpilevoy
16.08.2021 02:59, Timur Safin via Tarantool-patches пишет:
> * Version #5 changes:
>
> - After performance evaluations it was decided to switch data type used
> by `datetime.secs` from `int64_t` (which is boxed in LuaJIT) to
> `double` (which is unboxed number).
> - please see perf/datetime*.cc perf test as a benchmark we used for
> comparison of 2 veriants of C implementation. Results are here -
> https://gist.github.com/tsafin/618fbf847d258f6e7f5a75fdf9ea945b
Hi! Thanks for the patchset!
Generally looks ok. Most of my comments are style related,
except the ones covering patches 4, 5.
I like the amount of tests you introduce.
> * Version #4 changes:
>
> A lot.
>
> - Got rid of closures usage here and there. Benchmarks revealed there is
> 20x speed difference of hash with closure handlers to the case when we
> check keys using sequence of simple ifs.
> And despite the fact that it looks ugler, we have updated all places which
> used to use closures - so much performance impact does not worth it;
>
> - c-dt library now is accompanied with corresponding unit-tests;
>
> - due to a new (old) way of exporting public symbols, which is now used by
> Tarantool build, we have modified c-dt sources to use approach similar
> to that used by xxhash (or ICU) - "namespaces" for public symbols.
> If there is DT_NAMESPACE macro (e.g. DT_NAMESPACE=tnt_) defined at the
> moment of compilation we rename all relevant public symbols to
> use that "namespace" as prefix
>
> #define dt_from_rdn DT_NAME(DT_NAMESPACE, dt_from_rdn)
>
> C sources continue to use original name (now as a macro), but in Lua ffi
> we have to use renamed symbol - `tnt_dt_from_rdn`.
>
> - We have separated code which implement MessagePack related funcionality
> from generic (i.e. stringization) support for datetime types.
> Stringization is in `src/core/datetime.c`, but MessagePack support is in
> `src/core/mp_datetime.c`.
>
> - `ctime` and `asctime` now use reentrant versions (i.e. `ctime_r` instead of
> `ctime`, and `asctime_r` instead of `asctime`). This allows us to avoid
> nasty problem with GC where it was possible to corrupt returned
> static buffer regardless the fact that it was immediately passed
> to `ffi.string`.
> `test/app-tap/gh-5632-6050-6259-gc-buf-reuse.test.lua` updated to check our
> new api `datetime.ctime`, and `datetime.asctime`.
>
> - Introduced MessagePack unit-test which check C implementation (mp
> encode/decode, mp_snprint, mp_fprint)
>
> - Some special efforts have been taken to make sure we deal correctly
> with datetime stamps which were before Unix Epoch (1970-01-01), i.e.
> for negative stamps or timezones against epoch.
>
> - And last, but not the least: in addition to usual getters like
> `timestamp`, `nanoseconds`, `epoch`, we have introduced function
> modifiers `to_utc()` or `to_tz(offset)` which allow to manipulate with
> datetime timezone `.offset` field. Actually it does not do much now
> (only reattributing object with different offset, but not changed
> actual timestamp, which is UTC normalized). But different timestamp
> modifies textual representation, which simplify debugging.
>
> * Version #3 changes:
>
> - renamed `struct datetime_t` to `struct datetime`, and `struct
> datetime_interval_t` to `struct datetime_interval`;
> - significantly reworked arguments checks in module api entries;
> - fixed datetime comparisons;
> - changed hints calculation to take into account fractional part;
> - provided more comments here and there;
>
> NB! There are MacOSX problems due to GLIBC specific code used (as Vlad
> has already pointed out) - so additional patch, making it more
> cross-compatible coming here soon...
>
> * Version #2 changes:
>
> - fixed problem with overloaded '-' and '+' operations for datetime
> arguments;
> - fixed messagepack serialization problems;
> - heavily documented MessagePack serialization schema in the code;
> - introduced working implementation of datetime hints for storage engines;
> - made interval related names be more consistent, renamed durations and period
> to intervals, i.e. t_datetime_duration to datetime_interval_t,
> duration_* to interval_*, period to interval;
> - properly implemented all reasonable cases of datetime+interval
> arithmetic;
> - moved all initialization code to utils.c;
> - renamed core/mp_datetime.c to core/datetime.c because it makes more
> sense now;
>
> * Version #1 - initial RFC
>
> In brief
> --------
>
> This patchset implements datetime lua support in box, with serialization
> to messagepack, yaml, json and lua mode. Also it contains storage
> engines' indices implementation for datetime type introduced.
>
> * Current implementation is heavily influenced by Sci-Lua lua-time module
> https://github.com/stepelu/lua-time
> e.g. you could find very similar approach for handling of operations
> with year or month long intervals (which should be handled differently than
> usual intervals of seconds, or days).
>
> * But internally we actually use Christian Hanson' c-dt module
> https://github.com/chansen/c-dt
> (though it has been modified slightly for cleaner integration
> into our cmake-based build process).
>
>
> Datetime Module API
> -------------------
>
> Small draft of documentation for our `datetime` module API has been
> extracted to the discussion topic there -
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
> We try to update documentation there once any relevant change introdcued
> to the implementaton.
>
> Messagepack serialization schema
> --------------------------------
>
> In short it looks like:
> - now we introduce new MP_EXT extension type #4;
> - we may save 1 required and 2 optional fields for datetime field;
>
> In all gory details it's explained in MessagePack serialization schema depicted here:
> https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043990
>
>
> https://github.com/tarantool/tarantool/issues/5941
> https://github.com/tarantool/tarantool/issues/5946
>
> https://github.com/tarantool/tarantool/tree/tsafin/gh-5941-datetime-v4
>
> Timur Safin (8):
> build: add Christian Hansen c-dt to the build
> lua: built-in module datetime
> lua, datetime: display datetime
> box, datetime: messagepack support for datetime
> box, datetime: datetime comparison for indices
> lua, datetime: time intervals support
> datetime: perf test for datetime parser
> datetime: changelog for datetime module
>
> .gitmodules | 3 +
> CMakeLists.txt | 8 +
> .../gh-5941-datetime-type-support.md | 4 +
> cmake/BuildCDT.cmake | 10 +
> extra/exports | 33 +
> perf/CMakeLists.txt | 3 +
> perf/datetime-common.h | 105 +++
> perf/datetime-compare.cc | 213 +++++
> perf/datetime-parser.cc | 105 +++
> src/CMakeLists.txt | 5 +-
> src/box/field_def.c | 53 +-
> src/box/field_def.h | 4 +
> src/box/lua/serialize_lua.c | 7 +-
> src/box/memtx_space.c | 3 +-
> src/box/msgpack.c | 7 +-
> src/box/tuple_compare.cc | 77 ++
> src/box/vinyl.c | 3 +-
> src/lib/core/CMakeLists.txt | 5 +-
> src/lib/core/datetime.c | 176 ++++
> src/lib/core/datetime.h | 115 +++
> src/lib/core/mp_datetime.c | 189 ++++
> src/lib/core/mp_datetime.h | 89 ++
> src/lib/core/mp_extension_types.h | 1 +
> src/lib/mpstream/mpstream.c | 11 +
> src/lib/mpstream/mpstream.h | 4 +
> src/lua/datetime.lua | 880 ++++++++++++++++++
> src/lua/init.c | 4 +-
> src/lua/msgpack.c | 12 +
> src/lua/msgpackffi.lua | 18 +
> src/lua/serializer.c | 4 +
> src/lua/serializer.h | 2 +
> src/lua/utils.c | 28 +-
> src/lua/utils.h | 12 +
> test/app-tap/datetime.test.lua | 370 ++++++++
> .../gh-5632-6050-6259-gc-buf-reuse.test.lua | 74 +-
> test/engine/datetime.result | 77 ++
> test/engine/datetime.test.lua | 35 +
> test/unit/CMakeLists.txt | 3 +-
> test/unit/datetime.c | 381 ++++++++
> test/unit/datetime.result | 471 ++++++++++
> third_party/c-dt | 1 +
> third_party/lua-cjson/lua_cjson.c | 8 +
> third_party/lua-yaml/lyaml.cc | 6 +-
> 43 files changed, 3591 insertions(+), 28 deletions(-)
> create mode 100644 changelogs/unreleased/gh-5941-datetime-type-support.md
> create mode 100644 cmake/BuildCDT.cmake
> create mode 100644 perf/datetime-common.h
> create mode 100644 perf/datetime-compare.cc
> create mode 100644 perf/datetime-parser.cc
> create mode 100644 src/lib/core/datetime.c
> create mode 100644 src/lib/core/datetime.h
> create mode 100644 src/lib/core/mp_datetime.c
> create mode 100644 src/lib/core/mp_datetime.h
> create mode 100644 src/lua/datetime.lua
> create mode 100755 test/app-tap/datetime.test.lua
> create mode 100644 test/engine/datetime.result
> create mode 100644 test/engine/datetime.test.lua
> create mode 100644 test/unit/datetime.c
> create mode 100644 test/unit/datetime.result
> create mode 160000 third_party/c-dt
>
--
Serge Petrenko
^ permalink raw reply [flat|nested] 50+ messages in thread[parent not found: <20210818082222.mofgheciutpipelz@esperanza>]
* Re: [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation
[not found] ` <20210818082222.mofgheciutpipelz@esperanza>
@ 2021-08-18 8:25 ` Vladimir Davydov via Tarantool-patches
2021-08-18 13:24 ` Safin Timur via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-18 8:25 UTC (permalink / raw)
To: Timur Safin; +Cc: Vladimir Davydov, tarantool-patches, v.shpilevoy
[ += tarantool-patches - sorry dropped it from Cc by mistake ]
Putting aside minor hitches, like bad indentation here and there,
I have some concerns regarding the datetime API:
- I believe we shouldn't introduce interval in years/months, because
they depend on the date they are applied to. Better add datetime
methods instead - add_years / add_months.
- The API provides too many ways to extract the same information from a
datetime object (m, min, minute all mean the same). IMO a good API
should provide exactly one way to achieve a certain goal.
- AFAICS datetime.hour returns the number of hours passed since the
year 1970. This is confusing. I don't have any idea why anyone would
need this. As a user I expect them to return the hour of the current
day. Same for other similar methods, like month, minute, year.
- Creating a datetime without a date (with parse_time) looks weird:
tarantool> dt = datetime.parse_time('10:00:00')
---
...
tarantool> tostring(dt)
---
- 1970-01-01T10:00Z
...
Why 1970? I think that a datetime should always be created from a
date and optionally a time. If you want just time, you need another
kind of object - time. After all it's *date*time.
- datetime.days(2) + datetime.hours(12) + datetime.minutes(30)
looks cool, but I'm not sure it's efficient to create so many objects
and then sum them to construct an interval. It's surely acceptable
for compiled languages without garbage collection (like C++), but for
Lua I think it'd be more efficient to provide an API like this:
datetime.interval{days = 2, hours = 12, minutes = 30}
(I'm not a Lua expert so not sure about this)
- datetime.strftime, asctime, ctime - look too low level, which is no
surprise as they come from the C API. Judging by their names I can't
say that they present a datetime as a string in a certain format.
Maybe, rename strftime() to format() and make it a method of datetime
object (datetime:format)? Without a format, it could print ASCII
time. As for asctime, ctime, I'd drop them, because one can use
strftime to get the same result. Besides, they append \n to the
output, which looks weird:
tarantool> datetime.asctime(dt)
---
- 'Thu Jan 1 10:00:00 1970
'
...
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation
2021-08-18 8:25 ` Vladimir Davydov via Tarantool-patches
@ 2021-08-18 13:24 ` Safin Timur via Tarantool-patches
2021-08-18 14:22 ` Vladimir Davydov via Tarantool-patches
0 siblings, 1 reply; 50+ messages in thread
From: Safin Timur via Tarantool-patches @ 2021-08-18 13:24 UTC (permalink / raw)
To: Vladimir Davydov; +Cc: tarantool-patches, v.shpilevoy
Thanks, that's very practical approach to collect all complains spread
across big patchset to the single email - much easier to deal with.
And just in case, to make sure we are on the same line - could you
please read the module api documentation I've saved here
https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
(I keep it more or less updated after all API changes so far)
On 18.08.2021 11:25, Vladimir Davydov wrote:
> [ += tarantool-patches - sorry dropped it from Cc by mistake ]
>
> Putting aside minor hitches, like bad indentation here and there,
> I have some concerns regarding the datetime API:
>
> - I believe we shouldn't introduce interval in years/months, because
> they depend on the date they are applied to. Better add datetime
> methods instead - add_years / add_months.
Either I don't understand your objection, or you missed some the big
chunk of interval API I described here.
The reason why I had to introduce special interval types was this
problem of adding or subtraction generic interval. It's complicated to
calculate number of days when you need to subtract or add whole year,
it's even more complicated becoming when you need to add or subtract
several months (especially if they are spanning period longer than
year). c-dt does have special functions for this purpose
(
dt_t dt_add_years (dt_t dt, int delta, dt_adjust_t adjust);
dt_t dt_add_months (dt_t dt, int delta, dt_adjust_t adjust);
) which could reasonably handle all sort of truncations or snapping, if
month is shorter or longer than original one.
And having necessity to separate generic interval arithmetic for month,
or year (or possible quarters) intervals made me to use different cdata
type for their special storage. We may use any different way to
distinguish special years, like special field in the generic interval
and introduce kind of interval as another field. But approach use in
SciLua LuaTime https://github.com/stepelu/lua-time when we use different
cdata type as their tag for dispatching, looked most the elegant and
easiest one.
>
> - The API provides too many ways to extract the same information from a
> datetime object (m, min, minute all mean the same). IMO a good API
> should provide exactly one way to achieve a certain goal.
Wholeheartedly disagreed. With all due respect. There are many cases
when it's ok to provide several means for achieaving particular goal.
I'm ok in such environment with such mind-sets. (That may be explaining
why I liked C++ and Perl, and disliked Go and Python :))
I do foresee circumstances wen someone would like to use `date.now() +
date.weeks(2)`, while others would consistently use
`date.now():add{weeks = 2}`. Both approaches acceptable, and may be
supported equally.
>
> - AFAICS datetime.hour returns the number of hours passed since the
> year 1970. This is confusing. I don't have any idea why anyone would
> need this. As a user I expect them to return the hour of the current
> day. Same for other similar methods, like month, minute, year.
I've introduce minute, hour, and upward as accessors, soon after I've
provided .nanoseconds, .milliseconds and .seconds accessors.
Having this discussion here I realize that .hour may be not considered
as yet another accessor to seconds information, but as hour in a day,
this datetime is representing.
I suggest to remove these confusing accessors altogether, and open
ticket for later additions a method which would return such os.date-like
table. With attributes properly translated to
year/month/day/hour/minute/second.
>
> - Creating a datetime without a date (with parse_time) looks weird:
>
> tarantool> dt = datetime.parse_time('10:00:00')
> ---
> ...
>
> tarantool> tostring(dt)
> ---
> - 1970-01-01T10:00Z
> ...
>
> Why 1970? I think that a datetime should always be created from a
> date and optionally a time. If you want just time, you need another
> kind of object - time. After all it's *date*time.
Agreed, that parsing substrings of fuller datetime literal (parse_date,
parse_time, and parse_zone) are not that much useful without separate
entities date, time, and zone. I planned to extend usage of those api
while working with SQL datetime support and it's runtime. At the moment
we have datetime entity, and time without date is not yet making much
sense.
Suggestions, for a moment - disable/hide parse_time() and parse_zone().
parse_date() is perfectly legal now, it will use the same datastructures
where there will be empty information for seconds/nanoseconds since
beginning of a date.
>
> - datetime.days(2) + datetime.hours(12) + datetime.minutes(30)
>
> looks cool, but I'm not sure it's efficient to create so many objects
> and then sum them to construct an interval. It's surely acceptable
> for compiled languages without garbage collection (like C++), but for
> Lua I think it'd be more efficient to provide an API like this:
>
> datetime.interval{days = 2, hours = 12, minutes = 30}
I do have such API also, e.g. date:add{ days = 2, hours = 12, minutes =
30} they do modify original object though.
tarantool> date.now():add{days = 2, hours = 2}
---
- 2021-08-20T18:17:13.897261+03:00
...
>
> (I'm not a Lua expert so not sure about this)
>
> - datetime.strftime, asctime, ctime - look too low level, which is no
> surprise as they come from the C API. Judging by their names I can't
> say that they present a datetime as a string in a certain format.
> Maybe, rename strftime() to format() and make it a method of datetime
> object (datetime:format)? Without a format, it could print ASCII
> time. As for asctime, ctime, I'd drop them, because one can use
> strftime to get the same result. Besides, they append \n to the
> output, which looks weird:
>
> tarantool> datetime.asctime(dt)
> ---
> - 'Thu Jan 1 10:00:00 1970
>
> '
> ...
This is the way asctime/ctime were formatting date for centuries :)
Agreed though, that their predefined format looks strange, and almost
useless. And strftime is much more flexible. Will hide/delete their APIs.
The question is how different output of datetime:format() [without
specified format] should look like to the way we display by default
tostring(datetime)?
And should we introduce format() when we have tostring()?
[I still prefer it named strftime, because it's becoming clearly
apparent from the name what format should be used and how]
Thanks,
Timur
^ permalink raw reply [flat|nested] 50+ messages in thread* Re: [Tarantool-patches] [PATCH v5 0/8] Initial datetime implementation
2021-08-18 13:24 ` Safin Timur via Tarantool-patches
@ 2021-08-18 14:22 ` Vladimir Davydov via Tarantool-patches
0 siblings, 0 replies; 50+ messages in thread
From: Vladimir Davydov via Tarantool-patches @ 2021-08-18 14:22 UTC (permalink / raw)
To: Safin Timur; +Cc: tarantool-patches, v.shpilevoy
On Wed, Aug 18, 2021 at 04:24:16PM +0300, Safin Timur wrote:
> Thanks, that's very practical approach to collect all complains spread
> across big patchset to the single email - much easier to deal with.
>
> And just in case, to make sure we are on the same line - could you please
> read the module api documentation I've saved here https://github.com/tarantool/tarantool/discussions/6244#discussioncomment-1043988
> (I keep it more or less updated after all API changes so far)
>
>
> On 18.08.2021 11:25, Vladimir Davydov wrote:
> > [ += tarantool-patches - sorry dropped it from Cc by mistake ]
> >
> > Putting aside minor hitches, like bad indentation here and there,
> > I have some concerns regarding the datetime API:
> >
> > - I believe we shouldn't introduce interval in years/months, because
> > they depend on the date they are applied to. Better add datetime
> > methods instead - add_years / add_months.
>
> Either I don't understand your objection, or you missed some the big chunk
> of interval API I described here.
I missed that there's add/sub methods, sorry. (Because they are defined
in datetime:__index, not in the metatable, which is confusing, oh...)
Still, I don't see much point in defining both __add/__sub and add/sub.
If they have different behavior (like it seems they do in your patch
because they do not share implementation), it will be confusing. If
they are equivalent, then why would we need both? We don't have
decimal.add, only __add, for example.
Regarding datetime.years and datetime.months interval constructors,
quoting the RFC you wrote:
tarantool> T = date('1972-02-29')
---
...
tarantool> M = date.months(2)
---
...
tarantool> Y = date.years(1)
---
...
tarantool> T + M + Y
---
- 1973-04-30T00:00Z
...
tarantool> T + Y + M
---
- 1973-05-01T00:00Z
...
¯\_(ツ)_/¯
But even putting aside different results, I don't understand why
T + M + Y doesn't equal 1973-04-29 and T + Y + M doesn't equal
1973-05-29. Confused. Suggest to drop year and month intervals.
An interval must always be exact.
>
> The reason why I had to introduce special interval types was this problem of
> adding or subtraction generic interval. It's complicated to calculate number
> of days when you need to subtract or add whole year, it's even more
> complicated becoming when you need to add or subtract several months
> (especially if they are spanning period longer than year). c-dt does have
> special functions for this purpose
> (
> dt_t dt_add_years (dt_t dt, int delta, dt_adjust_t adjust);
> dt_t dt_add_months (dt_t dt, int delta, dt_adjust_t adjust);
> ) which could reasonably handle all sort of truncations or snapping, if
> month is shorter or longer than original one.
>
> And having necessity to separate generic interval arithmetic for month, or
> year (or possible quarters) intervals made me to use different cdata type
> for their special storage. We may use any different way to distinguish
> special years, like special field in the generic interval and introduce kind
> of interval as another field. But approach use in SciLua LuaTime
> https://github.com/stepelu/lua-time when we use different cdata type as
> their tag for dispatching, looked most the elegant and easiest one.
>
> >
> > - The API provides too many ways to extract the same information from a
> > datetime object (m, min, minute all mean the same). IMO a good API
> > should provide exactly one way to achieve a certain goal.
>
> Wholeheartedly disagreed. With all due respect. There are many cases when
> it's ok to provide several means for achieaving particular goal. I'm ok in
> such environment with such mind-sets. (That may be explaining why I liked
> C++ and Perl, and disliked Go and Python :))
>
> I do foresee circumstances wen someone would like to use `date.now() +
> date.weeks(2)`, while others would consistently use `date.now():add{weeks =
> 2}`. Both approaches acceptable, and may be supported equally.
IMO this syntactic sugaring only complicates developers' lives (suspect
they will end up writing guidelines what they can and can't use to keep
the code consistent, like they do in C++).
>
> >
> > - AFAICS datetime.hour returns the number of hours passed since the
> > year 1970. This is confusing. I don't have any idea why anyone would
> > need this. As a user I expect them to return the hour of the current
> > day. Same for other similar methods, like month, minute, year.
>
> I've introduce minute, hour, and upward as accessors, soon after I've
> provided .nanoseconds, .milliseconds and .seconds accessors.
>
> Having this discussion here I realize that .hour may be not considered as
> yet another accessor to seconds information, but as hour in a day, this
> datetime is representing.
>
> I suggest to remove these confusing accessors altogether, and open ticket
> for later additions a method which would return such os.date-like table.
> With attributes properly translated to year/month/day/hour/minute/second.
OK, but we need to agree first on what those accessors will look like:
- datetime members
- datetime methods
- a method that returns a table similar to the one returned by os.date
>
> >
> > - Creating a datetime without a date (with parse_time) looks weird:
> >
> > tarantool> dt = datetime.parse_time('10:00:00')
> > ---
> > ...
> > tarantool> tostring(dt)
> > ---
> > - 1970-01-01T10:00Z
> > ...
> >
> > Why 1970? I think that a datetime should always be created from a
> > date and optionally a time. If you want just time, you need another
> > kind of object - time. After all it's *date*time.
>
> Agreed, that parsing substrings of fuller datetime literal (parse_date,
> parse_time, and parse_zone) are not that much useful without separate
> entities date, time, and zone. I planned to extend usage of those api while
> working with SQL datetime support and it's runtime. At the moment we have
> datetime entity, and time without date is not yet making much sense.
>
> Suggestions, for a moment - disable/hide parse_time() and parse_zone().
>
> parse_date() is perfectly legal now, it will use the same datastructures
> where there will be empty information for seconds/nanoseconds since
> beginning of a date.
I'd prefer parse() to handle the case when there's no time in the string
if possible, but this is discussable. In fact, I'd prefer not to have
parse() at all - I'd like datetime(str) to handle construction from any
string, with or without time.
> > - datetime.strftime, asctime, ctime - look too low level, which is no
> > surprise as they come from the C API. Judging by their names I can't
> > say that they present a datetime as a string in a certain format.
> > Maybe, rename strftime() to format() and make it a method of datetime
> > object (datetime:format)? Without a format, it could print ASCII
> > time. As for asctime, ctime, I'd drop them, because one can use
> > strftime to get the same result. Besides, they append \n to the
> > output, which looks weird:
> >
> > tarantool> datetime.asctime(dt)
> > ---
> > - 'Thu Jan 1 10:00:00 1970
> > '
> > ...
>
> This is the way asctime/ctime were formatting date for centuries :)
>
> Agreed though, that their predefined format looks strange, and almost
> useless. And strftime is much more flexible. Will hide/delete their APIs.
>
> The question is how different output of datetime:format() [without specified
> format] should look like to the way we display by default
> tostring(datetime)?
>
> And should we introduce format() when we have tostring()?
>
> [I still prefer it named strftime, because it's becoming clearly apparent
> from the name what format should be used and how]
Guess strftime is fine. Python, uses strftime. Maybe, there's other Lua
APIs that have similar functionality (converting to string with options)
we could copycat?
^ permalink raw reply [flat|nested] 50+ messages in thread