From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 2DB6E6EC5D; Fri, 9 Apr 2021 20:57:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2DB6E6EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617991037; bh=iEKRWyA3mZO/AMRP98CmntwL/jAeYsteAUYXWIe5dr8=; h=To:Cc:Date:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=TQqXg7SyTFfJ3dPV1yQtRD8MycmLnZtaMsgWT7SfTQW5Js75rVQmaOZ7T/YgMCcKN kRHXAu4TYlGxgqyeGAxD1vIzI6GSkcYBYGxsHV4bbWpLIkNRflD1CEjyJGs2fVJ09V gjSYncpyid+7FCWOizHIRfAebDEq5mVHnxAdJZfI= Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 96CC76EC5D for ; Fri, 9 Apr 2021 20:57:15 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 96CC76EC5D Received: by smtpng2.m.smailru.net with esmtpa (envelope-from ) id 1lUvNa-000765-KO; Fri, 09 Apr 2021 20:57:15 +0300 To: v.shpilevoy@tarantool.org, tsafin@tarantool.org Cc: tarantool-patches@dev.tarantool.org Date: Fri, 9 Apr 2021 20:57:14 +0300 Message-Id: <8c81c3e86a628e5777145b30f69392dd5a0fd873.1617984948.git.imeevma@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD92FFCB8E6708E7480D608FE24BC85426BB1B55F651FED8C70182A05F538085040BE0FE6418ADB9BCD56D49C0D60F1400F4BA072753B59AFD3273631510F585982 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE70312E9A300D47E3BEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006370BACBAB4C30C4AEB8638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2A912E39E8BA51E5114A1BCF7FA3CEC904AEA611A897B002DD2E47CDBA5A96583C09775C1D3CA48CFCA5A41EBD8A3A0199FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3CA5A41EBD8A3A0199FA2833FD35BB23DF004C906525384302BEBFE083D3B9BA73A03B725D353964B0B7D0EA88DDEDAC722CA9DD8327EE4930A3850AC1BE2E735C6EABA9B74D0DA47B5C8C57E37DE458BEDA766A37F9254B7 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CD0035DD76F8A8A4F1F69C8823484AFD723C23A832C8A28DC9C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF0417BEADF48D1460699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3435BBF0AC4E3A921CB2C9FBABC4259AF86312798D0679B1BC2496E8EA2AE2D86D52A48A40BF11F4D71D7E09C32AA3244CB1DF74C1A3AFCBC72D4C46A211392E8364EE5813BBCA3A9DFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojyO2lHpuZu4QmvsAd27TxRA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A06382200CA42D916550A5DE8588DB4CE8A2C3483D72C36FC87018B9F80AB2734326CD2FB559BB5D741EB96352A0ABBE4FDA4210A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH v5 16/52] sql: rework vdbe_decode_msgpack_into_mem() X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mergen Imeev via Tarantool-patches Reply-To: imeevma@tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thank you for the review! My answers and new patch below. On 30.03.2021 02:02, Vladislav Shpilevoy wrote: > Thanks for the patch! > > See 3 comments below. > > On 23.03.2021 10:35, Mergen Imeev via Tarantool-patches wrote: >> The original vdbe_decode_msgpack_into_mem() returns a MEM that contains >> string and binary values as ephemeral. This patch renames this function >> to vdbe_decode_msgpack_into_mem_ephemeral() and introduces new >> vdbe_decode_msgpack_into_mem(), which returns a MEM that contains string >> and binary values in newly allocated memory. >> >> This patch actually changes behavior in this case: > > 1. Changes how? I don't see any changes in the tests. > I make this change because it doesn't affect test. Example of changed behaviour: CREATE TABLE t1(m VARBINARY primary key); INSERT INTO t1 VALUES(x'6178'), (x'6278'), (x'6379'); SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; Before this patch: tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'y'] - [1, 'y'] ... After this patch. tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'x'] - [1, 'y'] ... A bit more is written in issue #5890. >> CREATE TABLE t1(m VARBINARY primary key); >> INSERT INTO t1 VALUES(x'6178'), (x'6278'), (x'6379'); >> SELECT count(*), substr(m,2,1) AS m FROM t1 GROUP BY m; >> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; >> >> But it doesn't change behaviour for this: >> >> CREATE TABLE t2(m STRING primary key); >> INSERT INTO t2 VALUES('ax'), ('bx'), ('cy'); >> SELECT count(*), substr(m,2,1) AS m FROM t2 GROUP BY m; >> SELECT count(*), substr(m,2,1) AS mx FROM t2 GROUP BY mx; >> >> Part of #5818 >> Part of #5890 >> --- >> src/box/sql/mem.c | 16 +++++++++++++++- >> src/box/sql/mem.h | 17 ++++++++++++++++- >> src/box/sql/vdbe.c | 18 ------------------ >> src/box/sql/vdbeaux.c | 2 +- >> 4 files changed, 32 insertions(+), 21 deletions(-) >> >> diff --git a/src/box/sql/mem.c b/src/box/sql/mem.c >> index 3d42ac63c..a2316cc90 100644 >> --- a/src/box/sql/mem.c >> +++ b/src/box/sql/mem.c >> @@ -2253,7 +2253,8 @@ sqlVdbeRecordCompareMsgpack(const void *key1, >> } >> >> int >> -vdbe_decode_msgpack_into_mem(const char *buf, struct Mem *mem, uint32_t *len) >> +vdbe_decode_msgpack_into_ephemeral_mem(const char *buf, struct Mem *mem, >> + uint32_t *len) > > 2. The function name is getting Java vibes. I propose to rename it to > mem_from_mp_ephemeral() and mem_from_mp() correspondingly. They also should > start taking the mem as a first argument. > Thank you! Fixed. >> { >> const char *start_buf = buf; >> switch (mp_typeof(*buf)) { >> @@ -2354,6 +2355,19 @@ install_blob: >> return 0; >> } >> >> +int >> +vdbe_decode_msgpack_into_mem(const char *buf, struct Mem *mem, uint32_t *len) >> +{ >> + if (vdbe_decode_msgpack_into_ephemeral_mem(buf, mem, len) != 0) >> + return -1; >> + if ((mem->flags & (MEM_Str | MEM_Blob)) != 0) { >> + assert((mem->flags & MEM_Ephem) != 0); >> + if (sqlVdbeMemGrow(mem, mem->n, 1) != 0) >> + return -1; > > 3. Maybe it is worth adding a function like mem_materialize() or > mem_make_writable() for that kind of work. > Not sure. For now decided to not add a new function. >> + } >> + return 0; >> +} New patch: commit 8c81c3e86a628e5777145b30f69392dd5a0fd873 Author: Mergen Imeev Date: Sat Mar 13 15:43:38 2021 +0300 sql: rework vdbe_decode_msgpack_into_mem() The original vdbe_decode_msgpack_into_mem() returns a MEM that contains string and binary values as ephemeral. This patch renames this function to mem_from_mp_ephemeral() and introduces new function mem_from_mp(), which returns a MEM that contains string and binary values in newly allocated memory. This patch changes behavior for this query: CREATE TABLE t1(m VARBINARY primary key); INSERT INTO t1 VALUES(x'6178'), (x'6278'), (x'6379'); SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; Before this patch: tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'y'] - [1, 'y'] ... After this patch. tarantool> SELECT count(*), substr(m,2,1) AS mx FROM t1 GROUP BY mx; --- - metadata: - name: COLUMN_1 type: integer - name: MX type: string rows: - [2, 'x'] - [1, 'y'] ... Part of #5818 Closes #5890 diff --git a/src/box/sql/mem.c b/src/box/sql/mem.c index d56fe56c6..7d06e256c 100644 --- a/src/box/sql/mem.c +++ b/src/box/sql/mem.c @@ -2214,7 +2214,7 @@ sqlVdbeRecordCompareMsgpack(const void *key1, } int -vdbe_decode_msgpack_into_mem(const char *buf, struct Mem *mem, uint32_t *len) +mem_from_mp_ephemeral(struct Mem *mem, const char *buf, uint32_t *len) { const char *start_buf = buf; switch (mp_typeof(*buf)) { @@ -2315,6 +2315,19 @@ install_blob: return 0; } +int +mem_from_mp(struct Mem *mem, const char *buf, uint32_t *len) +{ + if (mem_from_mp_ephemeral(mem, buf, len) != 0) + return -1; + if ((mem->flags & (MEM_Str | MEM_Blob)) != 0) { + assert((mem->flags & MEM_Ephem) != 0); + if (sqlVdbeMemGrow(mem, mem->n, 1) != 0) + return -1; + } + return 0; +} + void mpstream_encode_vdbe_mem(struct mpstream *stream, struct Mem *var) { diff --git a/src/box/sql/mem.h b/src/box/sql/mem.h index 394055db9..55f8f0c9f 100644 --- a/src/box/sql/mem.h +++ b/src/box/sql/mem.h @@ -520,16 +520,30 @@ int sqlVdbeRecordCompareMsgpack(const void *key1, struct UnpackedRecord *key2); /** - * Decode msgpack and save value into VDBE memory cell. + * Decode msgpack and save value into VDBE memory cell. String and binary string + * values set as ephemeral. * + * @param mem Memory cell to write value into. * @param buf Buffer to deserialize msgpack from. + * @param len[out] Length of decoded part. + * @retval Return code: < 0 in case of error. + * @retval 0 on success. + */ +int +mem_from_mp_ephemeral(struct Mem *mem, const char *buf, uint32_t *len); + +/** + * Decode msgpack and save value into VDBE memory cell. String and binary string + * values copied to newly allocated memory. + * * @param mem Memory cell to write value into. + * @param buf Buffer to deserialize msgpack from. * @param len[out] Length of decoded part. * @retval Return code: < 0 in case of error. * @retval 0 on success. */ int -vdbe_decode_msgpack_into_mem(const char *buf, struct Mem *mem, uint32_t *len); +mem_from_mp(struct Mem *mem, const char *buf, uint32_t *len); /** * Perform encoding memory variable to stream. diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 0c19acff5..378c7a043 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -405,26 +405,8 @@ vdbe_field_ref_fetch(struct vdbe_field_ref *field_ref, uint32_t fieldno, assert(sqlVdbeCheckMemInvariants(dest_mem) != 0); const char *data = vdbe_field_ref_fetch_data(field_ref, fieldno); uint32_t dummy; - if (vdbe_decode_msgpack_into_mem(data, dest_mem, &dummy) != 0) + if (mem_from_mp(dest_mem, data, &dummy) != 0) return -1; - - /* - * Add 0 termination (at most for strings) - * Not sure why do we check MEM_Ephem - */ - if (mem_is_str(dest_mem) && mem_is_ephemeral(dest_mem)) { - int len = dest_mem->n; - if (dest_mem->szMalloc < len + 1) { - if (sqlVdbeMemGrow(dest_mem, len + 1, 1) != 0) - return -1; - } else { - dest_mem->z = - memcpy(dest_mem->zMalloc, dest_mem->z, len); - dest_mem->flags &= ~MEM_Ephem; - } - dest_mem->z[len] = 0; - dest_mem->flags |= MEM_Term; - } UPDATE_MAX_BLOBSIZE(dest_mem); return 0; } diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c index bec8a532a..dff108412 100644 --- a/src/box/sql/vdbeaux.c +++ b/src/box/sql/vdbeaux.c @@ -2358,7 +2358,7 @@ sqlVdbeRecordUnpackMsgpack(struct key_def *key_def, /* Information about the rec pMem->szMalloc = 0; pMem->z = 0; uint32_t sz = 0; - vdbe_decode_msgpack_into_mem(zParse, pMem, &sz); + mem_from_mp_ephemeral(pMem, zParse, &sz); assert(sz != 0); zParse += sz; pMem++; diff --git a/src/box/sql/vdbemem.c b/src/box/sql/vdbemem.c index 91cba9962..ba5c08a00 100644 --- a/src/box/sql/vdbemem.c +++ b/src/box/sql/vdbemem.c @@ -563,7 +563,7 @@ sql_stat4_column(struct sql *db, const char *record, uint32_t col_num, } } uint32_t unused; - return vdbe_decode_msgpack_into_mem(a, mem, &unused); + return mem_from_mp(mem, a, &unused); } /*