From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 9E3A06EC40; Thu, 19 Aug 2021 05:57:58 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 9E3A06EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1629341878; bh=3lnD54r4+RG2+6Frsu3Jl+RkPHv8Gcic/jYgpXZvqFk=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=lAn4adGYeAlKKpbbS5y+/fSGpje/tV32S291oVn0NQY4xwUHU/vLojgIhDwW4KjT7 vwDZkdaUtSmBoVdPx2WmSjvakMtFbdN5h3Fj+2JCtGf3YPJR3KbO610z0tDQZUuBsJ 8ajuEIRCyHXFwu/oDm2i2RQsh4s+RiEkW36fLbMc= Received: from smtp33.i.mail.ru (smtp33.i.mail.ru [94.100.177.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id E61466EC42 for ; Thu, 19 Aug 2021 05:56:58 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E61466EC42 Received: by smtp33.i.mail.ru with esmtpa (envelope-from ) id 1mGYEi-00022X-At; Thu, 19 Aug 2021 05:56:56 +0300 To: vdavydov@tarantool.org, sergepetrenko@tarantool.org, tarantool-patches@dev.tarantool.org Date: Thu, 19 Aug 2021 05:56:30 +0300 Message-Id: <003a95dc7856f488a551eacb66b0920629a62ab7.1629341071.git.tsafin@tarantool.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD9BCE6B93DE0C6C3914462CDB1732D383C182A05F538085040F87CC1AACB603DB8575C8807434EC0E075A066DBA4B5D93011B4C4CA00ADB4CA X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7B9FBA884A7C9B8BAEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637C1CDCB5E4A85220F8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8C991508EE567F72B80FAB6976ABC2923117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EEFAD5A440E159F97D4782AAF36435267CD8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE30085B890FD2717DA302FCEF25BFAB345C4224003CC836476EA7A3FFF5B025636E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637427B078F297B269AEFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458F0AFF96BAACF4158235E5A14AD4A4A4625E192CAD1D9E79D0B18DC6AC13D9A1C7ADFC5E2280B23B4 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CF160826E4E1956AE273A1D8B66D6558B313E3F43093B4D529C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF5A3EDA775A1E0ED0699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3452F7993FB7281BFDE6F70B352EEC5A9457307D322FD81FE3EC8457D2E97724DD6DD90D4ED7674E881D7E09C32AA3244C2C4040936E5545462A3DAA1F1B44BEDEBBA718C7E6A9E042AD832FF50B3043B1 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojGSxK+6r6oBHDw6+VQhdW8w== X-Mailru-Sender: 6CA451E36783D721CBEA96CEA26D325D697BCC0F495034880BCFE74F4AE7C6B5B7CBEF92542CD7C82F97C478340294DCC77752E0C033A69E0F0C7111264B8915FF1320A92A5534336C18EFA0BB12DBB0 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v6 2/5] box, datetime: messagepack support for datetime X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Timur Safin via Tarantool-patches Reply-To: Timur Safin Cc: v.shpilevoy@tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Serialize `struct datetime` as newly introduced MP_EXT type. It saves 1 required integer field and upto 2 optional unsigned fields in very compact fashion. - secs is required field; - but nsec, offset are both optional; * json, yaml serialization formats, lua output mode supported; * exported symbols for datetime messagepack size calculations so they are available for usage on Lua side. * As a bonus we introduce core/mp_utils.h with set of helpers which simplify working with mp_sizeof*/mp_encode_*/mp_decode_* functions regardless of a signedness of an integer data we deal with. Part of #5941 Part of #5946 --- extra/exports | 3 + src/box/lua/serialize_lua.c | 8 +- src/box/msgpack.c | 7 +- src/lib/core/CMakeLists.txt | 4 +- src/lib/core/datetime.h | 24 +++++ src/lib/core/mp_datetime.c | 171 ++++++++++++++++++++++++++++++ src/lib/core/mp_datetime.h | 64 +++++++++++ src/lib/core/mp_extension_types.h | 1 + src/lib/core/mp_utils.h | 64 +++++++++++ src/lib/mpstream/mpstream.c | 11 ++ src/lib/mpstream/mpstream.h | 4 + src/lua/msgpack.c | 12 +++ src/lua/msgpackffi.lua | 18 ++++ src/lua/serializer.c | 4 + src/lua/serializer.h | 2 + test/unit/datetime.c | 124 +++++++++++++++++++++- test/unit/datetime.result | 115 +++++++++++++++++++- third_party/lua-cjson/lua_cjson.c | 8 ++ third_party/lua-yaml/lyaml.cc | 6 +- 19 files changed, 644 insertions(+), 6 deletions(-) create mode 100644 src/lib/core/mp_datetime.c create mode 100644 src/lib/core/mp_datetime.h create mode 100644 src/lib/core/mp_utils.h diff --git a/extra/exports b/extra/exports index 0e7392d61..f232c94e0 100644 --- a/extra/exports +++ b/extra/exports @@ -149,6 +149,7 @@ csv_iterator_create csv_next csv_setopt datetime_now +datetime_pack datetime_strftime datetime_to_string datetime_unpack @@ -378,6 +379,7 @@ mp_decode_uint mp_encode_array mp_encode_bin mp_encode_bool +mp_encode_datetime mp_encode_decimal mp_encode_double mp_encode_float @@ -394,6 +396,7 @@ mp_next mp_next_slowpath mp_parser_hint mp_sizeof_array +mp_sizeof_datetime mp_sizeof_decimal mp_sizeof_str mp_sizeof_uuid diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c index 1f791980f..2c1ed6abb 100644 --- a/src/box/lua/serialize_lua.c +++ b/src/box/lua/serialize_lua.c @@ -768,7 +768,7 @@ static int dump_node(struct lua_dumper *d, struct node *nd, int indent) { struct luaL_field *field = &nd->field; - char buf[FPCONV_G_FMT_BUFSIZE]; + char buf[MAX(FPCONV_G_FMT_BUFSIZE, DT_TO_STRING_BUFSIZE)]; int ltype = lua_type(d->L, -1); const char *str = NULL; size_t len = 0; @@ -861,6 +861,12 @@ dump_node(struct lua_dumper *d, struct node *nd, int indent) str = tt_uuid_str(field->uuidval); len = UUID_STR_LEN; break; + case MP_DATETIME: + nd->mask |= NODE_QUOTE; + str = buf; + len = datetime_to_string(field->dateval, buf, + sizeof(buf)); + break; default: d->err = EINVAL; snprintf(d->err_msg, sizeof(d->err_msg), diff --git a/src/box/msgpack.c b/src/box/msgpack.c index 1723dea4c..12f4fd95a 100644 --- a/src/box/msgpack.c +++ b/src/box/msgpack.c @@ -1,5 +1,5 @@ /* - * Copyright 2020, Tarantool AUTHORS, please see AUTHORS file. + * Copyright 2020-2021, Tarantool AUTHORS, please see AUTHORS file. * * Redistribution and use in source and binary forms, with or * without modification, are permitted provided that the following @@ -35,6 +35,7 @@ #include "mp_decimal.h" #include "uuid/mp_uuid.h" #include "mp_error.h" +#include "mp_datetime.h" static int msgpack_fprint_ext(FILE *file, const char **data, int depth) @@ -47,6 +48,8 @@ msgpack_fprint_ext(FILE *file, const char **data, int depth) return mp_fprint_decimal(file, data, len); case MP_UUID: return mp_fprint_uuid(file, data, len); + case MP_DATETIME: + return mp_fprint_datetime(file, data, len); case MP_ERROR: return mp_fprint_error(file, data, depth); default: @@ -65,6 +68,8 @@ msgpack_snprint_ext(char *buf, int size, const char **data, int depth) return mp_snprint_decimal(buf, size, data, len); case MP_UUID: return mp_snprint_uuid(buf, size, data, len); + case MP_DATETIME: + return mp_snprint_datetime(buf, size, data, len); case MP_ERROR: return mp_snprint_error(buf, size, data, depth); default: diff --git a/src/lib/core/CMakeLists.txt b/src/lib/core/CMakeLists.txt index 8bc776b82..61fc6548f 100644 --- a/src/lib/core/CMakeLists.txt +++ b/src/lib/core/CMakeLists.txt @@ -31,6 +31,7 @@ set(core_sources mp_decimal.c cord_buf.c datetime.c + mp_datetime.c ) if (TARGET_OS_NETBSD) @@ -44,7 +45,8 @@ add_library(core STATIC ${core_sources}) target_link_libraries(core salad small uri decNumber bit ${LIBEV_LIBRARIES} ${LIBEIO_LIBRARIES} ${LIBCORO_LIBRARIES} - ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES}) + ${MSGPUCK_LIBRARIES} ${ICU_LIBRARIES} + ${LIBCDT_LIBRARIES}) if (ENABLE_BACKTRACE AND NOT TARGET_OS_DARWIN) target_link_libraries(core gcc_s ${UNWIND_LIBRARIES}) diff --git a/src/lib/core/datetime.h b/src/lib/core/datetime.h index 71feefded..fb537e372 100644 --- a/src/lib/core/datetime.h +++ b/src/lib/core/datetime.h @@ -5,6 +5,7 @@ * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. */ +#include #include #include #include "c-dt/dt.h" @@ -30,8 +31,26 @@ extern "C" #define DT_EPOCH_1970_OFFSET 719163 #endif +/** + * c-dt library uses int as type for dt value, which + * represents the number of days since Rata Die date. + * This implies limits to the number of seconds we + * could safely store in our structures and then safely + * pass to c-dt functions. + * + * So supported ranges will be + * - for seconds [-185604722870400 .. 185480451417600] + * - for dates [-5879610-06-22T00:00Z .. 5879611-07-11T00:00Z] + */ +#define MAX_DT_DAY_VALUE (int64_t)INT_MAX +#define MIN_DT_DAY_VALUE (int64_t)INT_MIN #define SECS_EPOCH_1970_OFFSET \ ((int64_t)DT_EPOCH_1970_OFFSET * SECS_PER_DAY) +#define MAX_EPOCH_SECS_VALUE \ + (MAX_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET) +#define MIN_EPOCH_SECS_VALUE \ + (MIN_DT_DAY_VALUE * SECS_PER_DAY - SECS_EPOCH_1970_OFFSET) + /** * datetime structure keeps number of seconds since * Unix Epoch. @@ -57,6 +76,11 @@ struct datetime_interval { uint32_t nsec; }; +/** + * Required size of datetime_to_string string buffer + */ +#define DT_TO_STRING_BUFSIZE 48 + /** * Convert datetime to string using default format * @param date source datetime value diff --git a/src/lib/core/mp_datetime.c b/src/lib/core/mp_datetime.c new file mode 100644 index 000000000..1430517e4 --- /dev/null +++ b/src/lib/core/mp_datetime.c @@ -0,0 +1,171 @@ +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. + */ + +#include + +#include "msgpuck.h" +#include "mp_datetime.h" +#include "mp_extension_types.h" +#include "mp_utils.h" + +/* + Datetime MessagePack serialization schema is MP_EXT (0xC7 for 1 byte length) + extension, which creates container of 1 to 3 integers. + + +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+ + |0xC7|len (uint8)| 4 | seconds (int) | nanoseconds (uint) | offset (int) | + +----+-----------+---+====~~~~~~~====+-----~~~~~~~~-------+....~~~~~~~....+ + + MessagePack extension MP_EXT (0xC7), after 1-byte length, contains: + + - signed integer seconds part (required). Depending on the value of + seconds it may be from 1 to 8 bytes positive or negative integer number; + + - [optional] fraction time in nanoseconds as unsigned integer. + If this value is 0 then it's not saved (unless there is offset field, + as below); + + - [optional] timezone offset in minutes as signed integer. + If this field is 0 then it's not saved. + */ + + +#define check_secs(secs) \ + assert((int64_t)(secs) <= MAX_EPOCH_SECS_VALUE);\ + assert((int64_t)(secs) >= MIN_EPOCH_SECS_VALUE); + +#define check_nanosecs(nsec) assert((nsec) < 1000000000); + +#define check_tz_offset(offset) \ + assert((offset) <= (12 * 60));\ + assert((offset) >= (-12 * 60)); + +static inline uint32_t +mp_sizeof_datetime_raw(const struct datetime *date) +{ + check_secs(date->secs); + uint32_t sz = mp_sizeof_xint(date->secs); + + /* + * even if nanosecs == 0 we need to output something + * if we have a non-null tz offset + */ + if (date->nsec != 0 || date->offset != 0) { + check_nanosecs(date->nsec); + sz += mp_sizeof_xint(date->nsec); + } + if (date->offset != 0) { + check_tz_offset(date->offset); + sz += mp_sizeof_xint(date->offset); + } + return sz; +} + +uint32_t +mp_sizeof_datetime(const struct datetime *date) +{ + return mp_sizeof_ext(mp_sizeof_datetime_raw(date)); +} + +struct datetime * +datetime_unpack(const char **data, uint32_t len, struct datetime *date) +{ + const char *svp = *data; + + memset(date, 0, sizeof(*date)); + + int64_t seconds = mp_decode_xint(data); + check_secs(seconds); + date->secs = seconds; + + len -= *data - svp; + if (len <= 0) + return date; + + svp = *data; + uint64_t nanoseconds = mp_decode_uint(data); + check_nanosecs(nanoseconds); + date->nsec = nanoseconds; + len -= *data - svp; + + if (len <= 0) + return date; + + int64_t offset = mp_decode_xint(data); + check_tz_offset(offset); + date->offset = offset; + + return date; +} + +struct datetime * +mp_decode_datetime(const char **data, struct datetime *date) +{ + if (mp_typeof(**data) != MP_EXT) + return NULL; + + const char *svp = *data; + int8_t type; + uint32_t len = mp_decode_extl(data, &type); + + if (type != MP_DATETIME || len == 0) { + *data = svp; + return NULL; + } + return datetime_unpack(data, len, date); +} + +char * +datetime_pack(char *data, const struct datetime *date) +{ + data = mp_encode_xint(data, date->secs); + if (date->nsec != 0 || date->offset != 0) + data = mp_encode_uint(data, date->nsec); + if (date->offset) + data = mp_encode_xint(data, date->offset); + + return data; +} + +char * +mp_encode_datetime(char *data, const struct datetime *date) +{ + uint32_t len = mp_sizeof_datetime_raw(date); + + data = mp_encode_extl(data, MP_DATETIME, len); + + return datetime_pack(data, date); +} + +int +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len) +{ + struct datetime date = { + .secs = 0, .nsec = 0, .offset = 0 + }; + + if (datetime_unpack(data, len, &date) == NULL) + return -1; + + return datetime_to_string(&date, buf, size); +} + +int +mp_fprint_datetime(FILE *file, const char **data, uint32_t len) +{ + struct datetime date = { + .secs = 0, .nsec = 0, .offset = 0 + }; + + if (datetime_unpack(data, len, &date) == NULL) + return -1; + + char buf[DT_TO_STRING_BUFSIZE]; + datetime_to_string(&date, buf, sizeof(buf)); + + return fprintf(file, "%s", buf); +} + diff --git a/src/lib/core/mp_datetime.h b/src/lib/core/mp_datetime.h new file mode 100644 index 000000000..92e94a243 --- /dev/null +++ b/src/lib/core/mp_datetime.h @@ -0,0 +1,64 @@ +#pragma once +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. + */ + +#include +#include "datetime.h" + +#if defined(__cplusplus) +extern "C" +{ +#endif /* defined(__cplusplus) */ + +/** + * Unpack datetime data from MessagePack buffer. + * @sa datetime_pack + */ +struct datetime * +datetime_unpack(const char **data, uint32_t len, struct datetime *date); + +/** + * Pack datetime data to MessagePack buffer. + * @sa datetime_unpack + */ +char * +datetime_pack(char *data, const struct datetime *date); + +/** + * Calculate size of MessagePack buffer for datetime data. + */ +uint32_t +mp_sizeof_datetime(const struct datetime *date); + +/** + * Decode data from MessagePack buffer to datetime structure. + */ +struct datetime * +mp_decode_datetime(const char **data, struct datetime *date); + +/** + * Encode datetime structure to the MessagePack buffer. + */ +char * +mp_encode_datetime(char *data, const struct datetime *date); + +/** + * Print datetime's string representation into a given buffer. + * @sa mp_snprint_decimal + */ +int +mp_snprint_datetime(char *buf, int size, const char **data, uint32_t len); + +/** + * Print datetime's string representation into a stream. + * @sa mp_fprint_decimal + */ +int +mp_fprint_datetime(FILE *file, const char **data, uint32_t len); + +#if defined(__cplusplus) +} /* extern "C" */ +#endif /* defined(__cplusplus) */ diff --git a/src/lib/core/mp_extension_types.h b/src/lib/core/mp_extension_types.h index e3ff9f5d0..3b7eaee7c 100644 --- a/src/lib/core/mp_extension_types.h +++ b/src/lib/core/mp_extension_types.h @@ -44,6 +44,7 @@ enum mp_extension_type { MP_DECIMAL = 1, MP_UUID = 2, MP_ERROR = 3, + MP_DATETIME = 4, mp_extension_type_MAX, }; diff --git a/src/lib/core/mp_utils.h b/src/lib/core/mp_utils.h new file mode 100644 index 000000000..bcfecca2a --- /dev/null +++ b/src/lib/core/mp_utils.h @@ -0,0 +1,64 @@ +#pragma once +/* + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright 2021, Tarantool AUTHORS, please see AUTHORS file. + */ + +#include +#include + +#include "msgpuck.h" + +/** + * Convenient wrapper for determining size of integer value + * in MessagePack, regardless of it's sign. + * + * @param n signed 64-bit value + * @sa mp_sizeof_int + * @sa mp_sizeof_uint + */ +static inline uint32_t +mp_sizeof_xint(int64_t n) +{ + return n < 0 ? mp_sizeof_int(n) : mp_sizeof_uint(n); +} + +/** + * Convenient wrapper for encoding integer value + * to messagepack, regardless of it's sign. + * + * @param data output MessagePack buffer + * @param v 64-bit value to be encoded. + * @sa mp_encode_int + * @sa mp_encode_uint + */ +static inline char * +mp_encode_xint(char *data, int64_t v) +{ + assert(v < 0 || (uint64_t)v <= LONG_MAX); + return v < 0 ? mp_encode_int(data, v) : mp_encode_uint(data, v); +} + +/** + * Convenient wrapper for decoding to integer value + * from MessagePack, regardless of it's sign. + * + * @param data messagepack buffer + * @retval return signed, 64-bit value. + * @sa mp_decode_int + * @sa mp_decode_uint + */ +static inline int64_t +mp_decode_xint(const char **data) +{ + switch (mp_typeof(**data)) { + case MP_UINT: + return (int64_t)mp_decode_uint(data); + case MP_INT: + return mp_decode_int(data); + default: + mp_unreachable(); + } + return 0; +} diff --git a/src/lib/mpstream/mpstream.c b/src/lib/mpstream/mpstream.c index 70ca29889..d3e1de965 100644 --- a/src/lib/mpstream/mpstream.c +++ b/src/lib/mpstream/mpstream.c @@ -35,6 +35,7 @@ #include "msgpuck.h" #include "mp_decimal.h" #include "uuid/mp_uuid.h" +#include "mp_datetime.h" void mpstream_reserve_slow(struct mpstream *stream, size_t size) @@ -208,6 +209,16 @@ mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid) mpstream_advance(stream, pos - data); } +void +mpstream_encode_datetime(struct mpstream *stream, const struct datetime *val) +{ + char *data = mpstream_reserve(stream, mp_sizeof_datetime(val)); + if (data == NULL) + return; + char *pos = mp_encode_datetime(data, val); + mpstream_advance(stream, pos - data); +} + void mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n) { diff --git a/src/lib/mpstream/mpstream.h b/src/lib/mpstream/mpstream.h index a60add143..94831160f 100644 --- a/src/lib/mpstream/mpstream.h +++ b/src/lib/mpstream/mpstream.h @@ -39,6 +39,7 @@ extern "C" { #endif /* defined(__cplusplus) */ struct tt_uuid; +struct datetime; /** * Ask the allocator to reserve at least size bytes. It can reserve @@ -145,6 +146,9 @@ mpstream_encode_decimal(struct mpstream *stream, const decimal_t *val); void mpstream_encode_uuid(struct mpstream *stream, const struct tt_uuid *uuid); +void +mpstream_encode_datetime(struct mpstream *stream, const struct datetime *dt); + /** Copies n bytes from memory area src to stream. */ void mpstream_memcpy(struct mpstream *stream, const void *src, uint32_t n); diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c index b6ecf2b1e..bca53f6b7 100644 --- a/src/lua/msgpack.c +++ b/src/lua/msgpack.c @@ -46,6 +46,7 @@ #include "lib/core/decimal.h" /* decimal_unpack() */ #include "lib/uuid/mp_uuid.h" /* mp_decode_uuid() */ #include "lib/core/mp_extension_types.h" +#include "lib/core/mp_datetime.h" #include "cord_buf.h" #include @@ -200,6 +201,9 @@ restart: /* used by MP_EXT of unidentified subtype */ break; case MP_ERROR: return luamp_encode_extension(L, top, stream); + case MP_DATETIME: + mpstream_encode_datetime(stream, field->dateval); + break; default: /* Run trigger if type can't be encoded */ type = luamp_encode_extension(L, top, stream); @@ -333,6 +337,14 @@ luamp_decode(struct lua_State *L, struct luaL_serializer *cfg, goto ext_decode_err; return; } + case MP_DATETIME: + { + struct datetime *date = luaL_pushdatetime(L); + date = datetime_unpack(data, len, date); + if (date == NULL) + goto ext_decode_err; + return; + } default: /* reset data to the extension header */ *data = svp; diff --git a/src/lua/msgpackffi.lua b/src/lua/msgpackffi.lua index 1d54f11b8..fb5e7d644 100644 --- a/src/lua/msgpackffi.lua +++ b/src/lua/msgpackffi.lua @@ -26,6 +26,10 @@ char * mp_encode_uuid(char *data, const struct tt_uuid *uuid); uint32_t mp_sizeof_uuid(); +uint32_t +mp_sizeof_datetime(const struct t_datetime_tz *date); +char * +mp_encode_datetime(char *data, const struct t_datetime_tz *date); float mp_decode_float(const char **data); double @@ -36,6 +40,8 @@ decimal_t * decimal_unpack(const char **data, uint32_t len, decimal_t *dec); struct tt_uuid * uuid_unpack(const char **data, uint32_t len, struct tt_uuid *uuid); +struct datetime * +datetime_unpack(const char **data, uint32_t len, struct datetime *date); ]]) local strict_alignment = (jit.arch == 'arm') @@ -142,6 +148,11 @@ local function encode_uuid(buf, uuid) builtin.mp_encode_uuid(p, uuid) end +local function encode_datetime(buf, date) + local p = buf:alloc(builtin.mp_sizeof_datetime(date)) + builtin.mp_encode_datetime(p, date) +end + local function encode_int(buf, num) if num >= 0 then if num <= 0x7f then @@ -320,6 +331,7 @@ on_encode(ffi.typeof('float'), encode_float) on_encode(ffi.typeof('double'), encode_double) on_encode(ffi.typeof('decimal_t'), encode_decimal) on_encode(ffi.typeof('struct tt_uuid'), encode_uuid) +on_encode(ffi.typeof('struct datetime'), encode_datetime) -------------------------------------------------------------------------------- -- Decoder @@ -513,6 +525,12 @@ local ext_decoder = { builtin.uuid_unpack(data, len, uuid) return uuid end, + -- MP_DATETIME + [4] = function(data, len) + local dt = ffi.new("struct datetime") + builtin.datetime_unpack(data, len, dt) + return dt + end, } local function decode_ext(data) diff --git a/src/lua/serializer.c b/src/lua/serializer.c index 8db6746a3..24f4a5ff9 100644 --- a/src/lua/serializer.c +++ b/src/lua/serializer.c @@ -41,6 +41,7 @@ #include "lib/core/mp_extension_types.h" #include "lua/error.h" +#include "datetime.h" #include "trivia/util.h" #include "diag.h" #include "serializer_opts.h" @@ -544,6 +545,9 @@ luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg, opts != NULL && opts->error_marshaling_enabled) { field->ext_type = MP_ERROR; + } else if (cd->ctypeid == CTID_DATETIME) { + field->ext_type = MP_DATETIME; + field->dateval = (struct datetime *)cdata; } else { field->ext_type = MP_UNKNOWN_EXTENSION; } diff --git a/src/lua/serializer.h b/src/lua/serializer.h index 0a0501a74..e7a240e0a 100644 --- a/src/lua/serializer.h +++ b/src/lua/serializer.h @@ -52,6 +52,7 @@ extern "C" { #include #include "trigger.h" +#include "lib/core/datetime.h" #include "lib/core/decimal.h" /* decimal_t */ #include "lib/core/mp_extension_types.h" #include "lua/error.h" @@ -223,6 +224,7 @@ struct luaL_field { uint32_t size; decimal_t *decval; struct tt_uuid *uuidval; + struct datetime *dateval; }; enum mp_type type; /* subtypes of MP_EXT */ diff --git a/test/unit/datetime.c b/test/unit/datetime.c index 931636172..c6ba444f5 100644 --- a/test/unit/datetime.c +++ b/test/unit/datetime.c @@ -6,6 +6,9 @@ #include "unit.h" #include "datetime.h" +#include "mp_datetime.h" +#include "msgpuck.h" +#include "mp_extension_types.h" #include "trivia/util.h" static const char sample[] = "2012-12-24T15:30Z"; @@ -249,12 +252,131 @@ tostring_datetime_test(void) check_plan(); } +static void +mp_datetime_test() +{ + static struct { + int64_t secs; + uint32_t nsec; + uint32_t offset; + uint32_t len; + } tests[] = { + {/* '1970-01-01T02:00+02:00' */ 0, 0, 120, 6}, + {/* '1970-01-01T01:30+01:30' */ 0, 0, 90, 6}, + {/* '1970-01-01T01:00+01:00' */ 0, 0, 60, 6}, + {/* '1970-01-01T00:01+00:01' */ 0, 0, 1, 6}, + {/* '1970-01-01T00:00Z' */ 0, 0, 0, 3}, + {/* '1969-12-31T23:59-00:01' */ 0, 0, -1, 6}, + {/* '1969-12-31T23:00-01:00' */ 0, 0, -60, 6}, + {/* '1969-12-31T22:30-01:30' */ 0, 0, -90, 6}, + {/* '1969-12-31T22:00-02:00' */ 0, 0, -120, 6}, + {/* '1970-01-01T00:00:00.123456789Z' */ 0, 123456789, 0, 9}, + {/* '1970-01-01T00:00:00.123456Z' */ 0, 123456000, 0, 9}, + {/* '1970-01-01T00:00:00.123Z' */ 0, 123000000, 0, 9}, + {/* '1973-11-29T21:33:09Z' */ 123456789, 0, 0, 8}, + {/* '2013-10-28T17:51:56Z' */ 1382982716, 0, 0, 8}, + {/* '9999-12-31T23:59:59Z' */ 253402300799, 0, 0, 12}, + {/* '9999-12-31T23:59:59.123456789Z' */ 253402300799, 123456789, 0, 17}, + {/* '9999-12-31T23:59:59.123456789-02:00' */ 253402300799, 123456789, -120, 18}, + }; + size_t index; + + plan(68); + for (index = 0; index < lengthof(tests); index++) { + struct datetime date = { + tests[index].secs, + tests[index].nsec, + tests[index].offset + }; + char buf[24], *data = buf; + const char *data1 = buf; + struct datetime ret; + + char *end = mp_encode_datetime(data, &date); + uint32_t len = mp_sizeof_datetime(&date); + is(len, tests[index].len, "len %u, expected len %u", + len, tests[index].len); + is(end - data, len, + "mp_sizeof_datetime(%d) == encoded length %ld", + len, end - data); + + struct datetime *rc = mp_decode_datetime(&data1, &ret); + is(rc, &ret, "mp_decode_datetime() return code"); + is(data1, end, "mp_sizeof_uuid() == decoded length"); + } + check_plan(); +} + + +static int +mp_fprint_ext_test(FILE *file, const char **data, int depth) +{ + (void)depth; + int8_t type; + uint32_t len = mp_decode_extl(data, &type); + if (type != MP_DATETIME) + return fprintf(file, "undefined"); + return mp_fprint_datetime(file, data, len); +} + +static int +mp_snprint_ext_test(char *buf, int size, const char **data, int depth) +{ + (void)depth; + int8_t type; + uint32_t len = mp_decode_extl(data, &type); + if (type != MP_DATETIME) + return snprintf(buf, size, "undefined"); + return mp_snprint_datetime(buf, size, data, len); +} + +static void +mp_print_test(void) +{ + plan(5); + header(); + + mp_snprint_ext = mp_snprint_ext_test; + mp_fprint_ext = mp_fprint_ext_test; + + char sample[64]; + char buffer[64]; + char str[64]; + struct datetime date = {0, 0, 0}; // 1970-01-01T00:00Z + + mp_encode_datetime(buffer, &date); + int sz = datetime_to_string(&date, str, sizeof(str)); + int rc = mp_snprint(NULL, 0, buffer); + is(rc, sz, "correct mp_snprint size %u with empty buffer", rc); + rc = mp_snprint(str, sizeof(str), buffer); + is(rc, sz, "correct mp_snprint size %u", rc); + datetime_to_string(&date, sample, sizeof(sample)); + is(strcmp(str, sample), 0, "correct mp_snprint result"); + + FILE *f = tmpfile(); + rc = mp_fprint(f, buffer); + is(rc, sz, "correct mp_fprint size %u", sz); + rewind(f); + rc = fread(str, 1, sizeof(str), f); + str[rc] = 0; + is(strcmp(str, sample), 0, "correct mp_fprint result %u", rc); + fclose(f); + + mp_snprint_ext = mp_snprint_ext_default; + mp_fprint_ext = mp_fprint_ext_default; + + footer(); + check_plan(); +} + int main(void) { - plan(2); + plan(4); datetime_test(); tostring_datetime_test(); + mp_datetime_test(); + mp_print_test(); return check_plan(); } diff --git a/test/unit/datetime.result b/test/unit/datetime.result index 33997d9df..5f01c4344 100644 --- a/test/unit/datetime.result +++ b/test/unit/datetime.result @@ -1,4 +1,4 @@ -1..1 +1..4 1..355 ok 1 - correct parse_datetime return value for '2012-12-24 15:30Z' ok 2 - correct parse_datetime output seconds for '2012-12-24 15:30Z @@ -356,3 +356,116 @@ ok 354 - correct parse_datetime return value for '2012-12-24 15:30:00' ok 355 - reversible seconds via strftime for '2012-12-24 15:30:00 ok 1 - subtests + 1..15 + ok 1 - string '1970-01-01T02:00+02:00' expected, received '1970-01-01T02:00+02:00' + ok 2 - string '1970-01-01T01:30+01:30' expected, received '1970-01-01T01:30+01:30' + ok 3 - string '1970-01-01T01:00+01:00' expected, received '1970-01-01T01:00+01:00' + ok 4 - string '1970-01-01T00:01+00:01' expected, received '1970-01-01T00:01+00:01' + ok 5 - string '1970-01-01T00:00Z' expected, received '1970-01-01T00:00Z' + ok 6 - string '1969-12-31T23:59-00:01' expected, received '1969-12-31T23:59-00:01' + ok 7 - string '1969-12-31T23:00-01:00' expected, received '1969-12-31T23:00-01:00' + ok 8 - string '1969-12-31T22:30-01:30' expected, received '1969-12-31T22:30-01:30' + ok 9 - string '1969-12-31T22:00-02:00' expected, received '1969-12-31T22:00-02:00' + ok 10 - string '1970-01-01T00:00:00.123456789Z' expected, received '1970-01-01T00:00:00.123456789Z' + ok 11 - string '1970-01-01T00:00:00.123456Z' expected, received '1970-01-01T00:00:00.123456Z' + ok 12 - string '1970-01-01T00:00:00.123Z' expected, received '1970-01-01T00:00:00.123Z' + ok 13 - string '1973-11-29T21:33:09Z' expected, received '1973-11-29T21:33:09Z' + ok 14 - string '2013-10-28T17:51:56Z' expected, received '2013-10-28T17:51:56Z' + ok 15 - string '9999-12-31T23:59:59Z' expected, received '9999-12-31T23:59:59Z' +ok 2 - subtests + 1..85 + ok 1 - len 6, expected len 6 + ok 2 - mp_sizeof_datetime(6) == encoded length 6 + ok 3 - mp_decode_datetime() return code + ok 4 - mp_sizeof_uuid() == decoded length + ok 5 - datetime_compare(&date, &ret) + ok 6 - len 6, expected len 6 + ok 7 - mp_sizeof_datetime(6) == encoded length 6 + ok 8 - mp_decode_datetime() return code + ok 9 - mp_sizeof_uuid() == decoded length + ok 10 - datetime_compare(&date, &ret) + ok 11 - len 6, expected len 6 + ok 12 - mp_sizeof_datetime(6) == encoded length 6 + ok 13 - mp_decode_datetime() return code + ok 14 - mp_sizeof_uuid() == decoded length + ok 15 - datetime_compare(&date, &ret) + ok 16 - len 6, expected len 6 + ok 17 - mp_sizeof_datetime(6) == encoded length 6 + ok 18 - mp_decode_datetime() return code + ok 19 - mp_sizeof_uuid() == decoded length + ok 20 - datetime_compare(&date, &ret) + ok 21 - len 3, expected len 3 + ok 22 - mp_sizeof_datetime(3) == encoded length 3 + ok 23 - mp_decode_datetime() return code + ok 24 - mp_sizeof_uuid() == decoded length + ok 25 - datetime_compare(&date, &ret) + ok 26 - len 6, expected len 6 + ok 27 - mp_sizeof_datetime(6) == encoded length 6 + ok 28 - mp_decode_datetime() return code + ok 29 - mp_sizeof_uuid() == decoded length + ok 30 - datetime_compare(&date, &ret) + ok 31 - len 6, expected len 6 + ok 32 - mp_sizeof_datetime(6) == encoded length 6 + ok 33 - mp_decode_datetime() return code + ok 34 - mp_sizeof_uuid() == decoded length + ok 35 - datetime_compare(&date, &ret) + ok 36 - len 6, expected len 6 + ok 37 - mp_sizeof_datetime(6) == encoded length 6 + ok 38 - mp_decode_datetime() return code + ok 39 - mp_sizeof_uuid() == decoded length + ok 40 - datetime_compare(&date, &ret) + ok 41 - len 6, expected len 6 + ok 42 - mp_sizeof_datetime(6) == encoded length 6 + ok 43 - mp_decode_datetime() return code + ok 44 - mp_sizeof_uuid() == decoded length + ok 45 - datetime_compare(&date, &ret) + ok 46 - len 9, expected len 9 + ok 47 - mp_sizeof_datetime(9) == encoded length 9 + ok 48 - mp_decode_datetime() return code + ok 49 - mp_sizeof_uuid() == decoded length + ok 50 - datetime_compare(&date, &ret) + ok 51 - len 9, expected len 9 + ok 52 - mp_sizeof_datetime(9) == encoded length 9 + ok 53 - mp_decode_datetime() return code + ok 54 - mp_sizeof_uuid() == decoded length + ok 55 - datetime_compare(&date, &ret) + ok 56 - len 9, expected len 9 + ok 57 - mp_sizeof_datetime(9) == encoded length 9 + ok 58 - mp_decode_datetime() return code + ok 59 - mp_sizeof_uuid() == decoded length + ok 60 - datetime_compare(&date, &ret) + ok 61 - len 8, expected len 8 + ok 62 - mp_sizeof_datetime(8) == encoded length 8 + ok 63 - mp_decode_datetime() return code + ok 64 - mp_sizeof_uuid() == decoded length + ok 65 - datetime_compare(&date, &ret) + ok 66 - len 8, expected len 8 + ok 67 - mp_sizeof_datetime(8) == encoded length 8 + ok 68 - mp_decode_datetime() return code + ok 69 - mp_sizeof_uuid() == decoded length + ok 70 - datetime_compare(&date, &ret) + ok 71 - len 12, expected len 12 + ok 72 - mp_sizeof_datetime(12) == encoded length 12 + ok 73 - mp_decode_datetime() return code + ok 74 - mp_sizeof_uuid() == decoded length + ok 75 - datetime_compare(&date, &ret) + ok 76 - len 17, expected len 17 + ok 77 - mp_sizeof_datetime(17) == encoded length 17 + ok 78 - mp_decode_datetime() return code + ok 79 - mp_sizeof_uuid() == decoded length + ok 80 - datetime_compare(&date, &ret) + ok 81 - len 18, expected len 18 + ok 82 - mp_sizeof_datetime(18) == encoded length 18 + ok 83 - mp_decode_datetime() return code + ok 84 - mp_sizeof_uuid() == decoded length + ok 85 - datetime_compare(&date, &ret) +ok 3 - subtests + 1..5 + *** mp_print_test *** + ok 1 - correct mp_snprint size 17 with empty buffer + ok 2 - correct mp_snprint size 17 + ok 3 - correct mp_snprint result + ok 4 - correct mp_fprint size 17 + ok 5 - correct mp_fprint result 17 + *** mp_print_test: done *** +ok 4 - subtests diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c index 7a326075a..4d1b28ca1 100644 --- a/third_party/lua-cjson/lua_cjson.c +++ b/third_party/lua-cjson/lua_cjson.c @@ -52,6 +52,7 @@ #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */ #include "tt_static.h" #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */ +#include "core/datetime.h" #include "cord_buf.h" typedef enum { @@ -426,6 +427,13 @@ static void json_append_data(lua_State *l, struct luaL_serializer *cfg, case MP_UUID: return json_append_string(cfg, json, tt_uuid_str(field.uuidval), UUID_STR_LEN); + + case MP_DATETIME: + { + char buf[DT_TO_STRING_BUFSIZE]; + size_t sz = datetime_to_string(field.dateval, buf, sizeof(buf)); + return json_append_string(cfg, json, buf, sz); + } default: assert(false); } diff --git a/third_party/lua-yaml/lyaml.cc b/third_party/lua-yaml/lyaml.cc index 2b67dcc6a..33c7cdfa1 100644 --- a/third_party/lua-yaml/lyaml.cc +++ b/third_party/lua-yaml/lyaml.cc @@ -617,7 +617,7 @@ static int dump_node(struct lua_yaml_dumper *dumper) yaml_event_t ev; yaml_scalar_style_t style = YAML_PLAIN_SCALAR_STYLE; int is_binary = 0; - char buf[FPCONV_G_FMT_BUFSIZE]; + char buf[MAX(FPCONV_G_FMT_BUFSIZE, DT_TO_STRING_BUFSIZE)]; struct luaL_field field; bool unused; (void) unused; @@ -707,6 +707,10 @@ static int dump_node(struct lua_yaml_dumper *dumper) str = tt_uuid_str(field.uuidval); len = UUID_STR_LEN; break; + case MP_DATETIME: + len = datetime_to_string(field.dateval, buf, sizeof(buf)); + str = buf; + break; default: assert(0); /* checked by luaL_checkfield() */ } -- 2.29.2