From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 385576EC5E; Fri, 9 Apr 2021 22:47:07 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 385576EC5E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617997627; bh=c6VvWOIjzy67j+wbec1ciuObzp/M9TcH7iQRx/3f7g0=; h=To:Cc:Date:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=D6Yb3B61LRy1vz9aSmTjd7wlXb+UnFmOrADiP46h11Od9WSRDRTgm94Rf6uurmhxJ BwH1+Rq84KJFCQq8fiBu20kPLNMHqmiYkaYVEBAc571UT2x2zC9QWZ6rk/vNIlNOkx 38OM8Zd28yExaShP9ptx1VJ183xOhOawEuMnAaVY= Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id E74D96EC63 for ; Fri, 9 Apr 2021 22:45:41 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E74D96EC63 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1lUx4X-0006dp-54; Fri, 09 Apr 2021 22:45:41 +0300 To: v.shpilevoy@tarantool.org, tsafin@tarantool.org Cc: tarantool-patches@dev.tarantool.org Date: Fri, 9 Apr 2021 22:45:40 +0300 Message-Id: <67af1300324e2548ffa9234e762cdc473ba88e19.1617984948.git.imeevma@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD92FFCB8E6708E7480B1C8842CE613979772AC4B34408C6525182A05F538085040F68CC58FC87B0BDD701051AE4EF60FE8957229D26FEF665069AB3141027135A2 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7495A032B936E882FEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063707C4856229E8E7E48638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B21E790EB8BB06260779EE3549E9B526DF618001F51B5FD3F9D2E47CDBA5A96583C09775C1D3CA48CF27ED053E960B195E117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE75A31C54DA8CF07A79FA2833FD35BB23DF004C906525384302BEBFE083D3B9BA73A03B725D353964B0B7D0EA88DDEDAC722CA9DD8327EE4930A3850AC1BE2E735C6EABA9B74D0DA47B5C8C57E37DE458BEDA766A37F9254B7 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2AD77751E876CB595E8F7B195E1C97831712C5303C9CF66A6A3BF5431196D8D52 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CD0035DD76F8A8A4F782A76CD96FC41F8774477F393256FFB9C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF0417BEADF48D1460699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3480A9008907CB2FD08D700DB1139C7A5C269CAE066787E8A54DA10A9FB6692A2D3D9ECF359B7F16BE1D7E09C32AA3244C16813C7050F37F2A9AEFEBBFF21887A17101BF96129E4011FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojyO2lHpuZu4SnEvdN/0BrpA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A06382290194DDB413D0FF9AE637CFA5E25CCF083D72C36FC87018B9F80AB2734326CD2FB559BB5D741EB96352A0ABBE4FDA4210A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH v5 29/52] sql: introduce mem_set_bin_*() functions X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mergen Imeev via Tarantool-patches Reply-To: imeevma@tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thank you for the review! My answers and new patch below. On 30.03.2021 02:05, Vladislav Shpilevoy wrote: > Thanks for the patch! > > See 2 comments below. > >> diff --git a/src/box/sql/mem.c b/src/box/sql/mem.c >> index 59a378e1b..5ee49cdca 100644 >> --- a/src/box/sql/mem.c >> +++ b/src/box/sql/mem.c >> @@ -410,6 +410,61 @@ mem_copy_string0(struct Mem *mem, const char *value) >> return 0; >> } >> >> +static inline void >> +mem_set_const_bin(struct Mem *mem, char *value, uint32_t size, int alloc_type) >> +{ >> + assert((alloc_type & (MEM_Static | MEM_Ephem)) != 0); >> + mem_clear(mem); >> + mem->z = value; >> + mem->n = size; >> + mem->flags = MEM_Blob | alloc_type; >> + mem->field_type = FIELD_TYPE_VARBINARY; >> +} >> + >> +static inline void >> +mem_set_dyn_bin(struct Mem *mem, char *value, uint32_t size, int alloc_type) >> +{ >> + assert((mem->flags & MEM_Dyn) == 0 || value != mem->z); >> + assert(mem->szMalloc == 0 || value != mem->zMalloc); >> + assert(alloc_type == MEM_Dyn || alloc_type == 0); >> + mem_destroy(mem); > > 1. Why is it destroy here and clear above? > There is a rule which I cannot understand - it says that MEM should contain either given allocated memory + sql_free() (MEM_Dyn) or "self-allocated" memory (which may be not self-allocated actually, but still). I see no reason for this, but I decided to leave it as it is for now. So, in case we set MEM_Dyn or allocated type (when MEM_Static, MEM_Ephem and MEM_Dyn unset) MEM have to free all memory. This is why we use destroy here. In case of MEM_Ephem or MEM_Static we only should free memory if it is of MEM_Dyn type. So that is why we use clear() there. >> + mem->z = value; >> + mem->n = size; >> + mem->flags = MEM_Blob | alloc_type; >> + mem->field_type = FIELD_TYPE_VARBINARY; >> + if (alloc_type == MEM_Dyn) { >> + mem->xDel = sql_free; >> + } else { >> + mem->xDel = NULL; >> + mem->zMalloc = mem->z; >> + mem->szMalloc = sqlDbMallocSize(mem->db, mem->zMalloc); >> + } >> +} >> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c >> index 5b5e5b0c8..0e51e4809 100644 >> --- a/src/box/sql/vdbeapi.c >> +++ b/src/box/sql/vdbeapi.c >> @@ -183,7 +182,13 @@ sql_result_blob(sql_context * pCtx, >> ) >> { >> assert(n >= 0); >> - if (sqlVdbeMemSetStr(pCtx->pOut, z, n, 0, xDel) != 0) >> + if (xDel == SQL_STATIC) >> + mem_set_static_binary(pCtx->pOut, (char *)z, n); >> + else if (xDel == SQL_DYNAMIC) >> + mem_set_allocated_binary(pCtx->pOut, (char *)z, n); >> + else if (xDel != SQL_TRANSIENT) >> + mem_set_dynamic_binary(pCtx->pOut, (char *)z, n); >> + else if (sqlVdbeMemSetStr(pCtx->pOut, z, n, 0, xDel) != 0) >> pCtx->is_aborted = true; > > 2. It seems to me you need to add a generic mem_set_binary which would > take the xdel argument. Repeating this tree of ifs in each usage place > is not any better. The same for the string API. I think the problem here is mostly with the way built-in functions returns their result. Don't you think that it is better to change sql_result_*() instead of adding of a generic mem_set_binary() function? About bindText() - I think it is better to move sql_bind_column() to sql/vdbeapi.c and inline such functions as bindText(), sql_bind_double(), etc. This is the only place they are used, after all. Still, I didn't do any of these changes. I plan to rework built-in functions later, along with removing them from _func. As for sql_bind_column() - I just not sure that this should be done in this patch-set. No problem to add one more patch to the set, though. What do you think? New patch: commit 67af1300324e2548ffa9234e762cdc473ba88e19 Author: Mergen Imeev Date: Tue Mar 16 10:17:33 2021 +0300 sql: introduce mem_set_bin_*() functions This patch introduces set of mem_set_bin_*() functions. These functions clears MEM and sets it to given binary value. Degree of clearing and type of allocation of the binary value is determined by the function used. Part of #5818 diff --git a/src/box/sql/mem.c b/src/box/sql/mem.c index 045c44e8f..2622cdd82 100644 --- a/src/box/sql/mem.c +++ b/src/box/sql/mem.c @@ -422,6 +422,61 @@ mem_copy_str0(struct Mem *mem, const char *value) return 0; } +static inline void +set_bin_const(struct Mem *mem, char *value, uint32_t size, int alloc_type) +{ + assert((alloc_type & (MEM_Static | MEM_Ephem)) != 0); + mem_clear(mem); + mem->z = value; + mem->n = size; + mem->flags = MEM_Blob | alloc_type; + mem->field_type = FIELD_TYPE_VARBINARY; +} + +static inline void +set_bin_dynamic(struct Mem *mem, char *value, uint32_t size, int alloc_type) +{ + assert((mem->flags & MEM_Dyn) == 0 || value != mem->z); + assert(mem->szMalloc == 0 || value != mem->zMalloc); + assert(alloc_type == MEM_Dyn || alloc_type == 0); + mem_destroy(mem); + mem->z = value; + mem->n = size; + mem->flags = MEM_Blob | alloc_type; + mem->field_type = FIELD_TYPE_VARBINARY; + if (alloc_type == MEM_Dyn) { + mem->xDel = sql_free; + } else { + mem->xDel = NULL; + mem->zMalloc = mem->z; + mem->szMalloc = sqlDbMallocSize(mem->db, mem->zMalloc); + } +} + +void +mem_set_bin_ephemeral(struct Mem *mem, char *value, uint32_t size) +{ + set_bin_const(mem, value, size, MEM_Ephem); +} + +void +mem_set_bin_static(struct Mem *mem, char *value, uint32_t size) +{ + set_bin_const(mem, value, size, MEM_Static); +} + +void +mem_set_bin_dynamic(struct Mem *mem, char *value, uint32_t size) +{ + set_bin_dynamic(mem, value, size, MEM_Dyn); +} + +void +mem_set_bin_allocated(struct Mem *mem, char *value, uint32_t size) +{ + set_bin_dynamic(mem, value, size, 0); +} + int mem_copy(struct Mem *to, const struct Mem *from) { @@ -2547,19 +2602,6 @@ mem_is_type_compatible(struct Mem *mem, enum field_type type) return field_mp_plain_type_is_compatible(type, mp_type, true); } -/* Allocate memory for internal VDBE structure on region. */ -int -vdbe_mem_alloc_blob_region(struct Mem *vdbe_mem, uint32_t size) -{ - vdbe_mem->n = size; - vdbe_mem->z = region_alloc(&fiber()->gc, size); - if (vdbe_mem->z == NULL) - return -1; - vdbe_mem->flags = MEM_Ephem | MEM_Blob; - assert(sqlVdbeCheckMemInvariants(vdbe_mem)); - return 0; -} - int sql_vdbemem_finalize(struct Mem *mem, struct func *func) { diff --git a/src/box/sql/mem.h b/src/box/sql/mem.h index d03651192..532860ff7 100644 --- a/src/box/sql/mem.h +++ b/src/box/sql/mem.h @@ -246,6 +246,34 @@ mem_copy_str(struct Mem *mem, const char *value, uint32_t len); int mem_copy_str0(struct Mem *mem, const char *value); +/** + * Clear MEM and set it to VARBINARY. The binary value belongs to another + * object. + */ +void +mem_set_bin_ephemeral(struct Mem *mem, char *value, uint32_t size); + +/** Clear MEM and set it to VARBINARY. The binary value is static. */ +void +mem_set_bin_static(struct Mem *mem, char *value, uint32_t size); + +/** + * Clear MEM and set it to VARBINARY. The binary value was allocated by another + * object and passed to MEM. MEMs with this allocation type must free given + * memory whenever the MEM changes. + */ +void +mem_set_bin_dynamic(struct Mem *mem, char *value, uint32_t size); + +/** + * Clear MEM and set it to VARBINARY. The binary value was allocated by another + * object and passed to MEM. MEMs with this allocation type only deallocate the + * string on destruction. Also, the memory may be reallocated if MEM is set to a + * different value of this allocation type. + */ +void +mem_set_bin_allocated(struct Mem *mem, char *value, uint32_t size); + /** * Copy content of MEM from one MEM to another. In case source MEM contains * string or binary and allocation type is not STATIC, this value is copied to @@ -606,9 +634,6 @@ mem_is_type_compatible(struct Mem *mem, enum field_type type); /** MEM manipulate functions. */ -int -vdbe_mem_alloc_blob_region(struct Mem *vdbe_mem, uint32_t size); - /** * Memory cell mem contains the context of an aggregate function. * This routine calls the finalize method for that function. The diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 943179c55..85fb7a26d 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -886,8 +886,15 @@ case OP_Null: { /* out2 */ case OP_Blob: { /* out2 */ assert(pOp->p1 <= SQL_MAX_LENGTH); pOut = vdbe_prepare_null_out(p, pOp->p2); - sqlVdbeMemSetStr(pOut, pOp->p4.z, pOp->p1, 0, 0); - if (pOp->p3!=0) { + if (pOp->p3 == 0) { + /* + * TODO: It is possible that vabinary should be stored as + * ephemeral or static depending on value. There is no way to + * determine right now, so it is stored as static. + */ + mem_set_bin_static(pOut, pOp->p4.z, pOp->p1); + } else { + sqlVdbeMemSetStr(pOut, pOp->p4.z, pOp->p1, 0, 0); pOut->flags |= MEM_Subtype; pOut->subtype = pOp->p3; } @@ -2201,9 +2208,7 @@ case OP_MakeRecord: { * sure previously allocated memory has gone. */ mem_destroy(pOut); - pOut->flags = MEM_Blob | MEM_Ephem; - pOut->n = tuple_size; - pOut->z = tuple; + mem_set_bin_ephemeral(pOut, tuple, tuple_size); } assert(sqlVdbeCheckMemInvariants(pOut)); assert(pOp->p3>0 && pOp->p3<=(p->nMem+1 - p->nCursor)); @@ -3286,9 +3291,14 @@ case OP_RowData: { } testcase( n==0); - if (vdbe_mem_alloc_blob_region(pOut, n) != 0) + char *buf = region_alloc(&fiber()->gc, n); + if (buf == NULL) { + diag_set(OutOfMemory, n, "region_alloc", "buf"); goto abort_due_to_error; - sqlCursorPayload(pCrsr, 0, n, pOut->z); + } + sqlCursorPayload(pCrsr, 0, n, buf); + mem_set_bin_ephemeral(pOut, buf, n); + assert(sqlVdbeCheckMemInvariants(pOut)); UPDATE_MAX_BLOBSIZE(pOut); REGISTER_TRACE(p, pOp->p2, pOut); break; diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c index ee095f36e..8c7a0b6cd 100644 --- a/src/box/sql/vdbeapi.c +++ b/src/box/sql/vdbeapi.c @@ -111,9 +111,8 @@ sql_metadata_is_full() * The following routines are used by user-defined functions to specify * the function result. * - * The setStrOrError() function calls sqlVdbeMemSetStr() to store the - * result as a string or blob but if the string or blob is too large, it - * then sets the error code. + * The setStrOrError() function sets the result as a string or blob but + * if the string or blob is too large, it then sets the error code. * * The invokeValueDestructor(P,X) routine invokes destructor function X() * on value P is not going to be used and need to be destroyed. @@ -183,7 +182,13 @@ sql_result_blob(sql_context * pCtx, ) { assert(n >= 0); - if (sqlVdbeMemSetStr(pCtx->pOut, z, n, 0, xDel) != 0) + if (xDel == SQL_STATIC) + mem_set_bin_static(pCtx->pOut, (char *)z, n); + else if (xDel == SQL_DYNAMIC) + mem_set_bin_allocated(pCtx->pOut, (char *)z, n); + else if (xDel != SQL_TRANSIENT) + mem_set_bin_dynamic(pCtx->pOut, (char *)z, n); + else if (sqlVdbeMemSetStr(pCtx->pOut, z, n, 0, xDel) != 0) pCtx->is_aborted = true; } @@ -832,7 +837,13 @@ sql_bind_blob(sql_stmt * pStmt, if (zData == NULL) return 0; struct Mem *var = &p->aVar[i - 1]; - if (sqlVdbeMemSetStr(var, zData, nData, 0, xDel) != 0) + if (xDel == SQL_STATIC) + mem_set_bin_static(var, (char *)zData, nData); + else if (xDel == SQL_DYNAMIC) + mem_set_bin_allocated(var, (char *)zData, nData); + else if (xDel != SQL_TRANSIENT) + mem_set_bin_dynamic(var, (char *)zData, nData); + else if (sqlVdbeMemSetStr(var, zData, nData, 0, xDel) != 0) return -1; return sql_bind_type(p, i, "varbinary"); }