From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 98A036ECE3; Tue, 19 Oct 2021 14:14:28 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 98A036ECE3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1634642068; bh=KptURt1pFflA+XAGaLLTvIpXuHY2z7ts6f7fm4pOd80=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=HhLRflVV4ClO2pIC/vcFVqCI4LwyhqILGU95sQYY1lLJbnXoxw8ZUwhqJy8TGVHRa fGnD1M/u8mDcGQ8gwU8hZR7raJqP3nZB0qjpbH22l4y4A/vX8h3R/wW7iCgERlF2yU PxyFWABsgp44RNH4wS3VD7p6emxDtxCCuVuX1wj8= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 6D7116ECE3 for ; Tue, 19 Oct 2021 14:14:27 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6D7116ECE3 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1mcn4c-0001WL-EU; Tue, 19 Oct 2021 14:14:26 +0300 Date: Tue, 19 Oct 2021 14:14:25 +0300 To: Vladislav Shpilevoy Cc: tarantool-patches@dev.tarantool.org Message-ID: <20211019111425.GA190172@tarantool.org> References: <3543f4417e240c74d1dea9a2b6e086aeca950167.1633092363.git.imeevma@gmail.com> <6dfd69ff-a807-b0d5-4896-4b5118ee2679@tarantool.org> <20211005094806.GE55311@tarantool.org> <38e0558f-2cce-bf78-0be9-92e9c60c2379@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <38e0558f-2cce-bf78-0be9-92e9c60c2379@tarantool.org> X-4EC0790: 10 X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD9C7814344C8C501C83238E3156CE19B78C95C7AD4390B7974182A05F538085040019C9A3106709267C32EF982C31D59E77156830E4F820E0533D87F368224BDB1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE74E2C4641A2CB07F2EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637E8D1333770DC60CDEA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F2CACE460DF841FF8694D399E9B1C9181BCC7F00164DA146DAFE8445B8C89999728AA50765F7900637D0FEED2715E18529389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC8989FD0BDF65E50FBF6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA7E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8B953A8A48A05D51F175ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A5E0C9EE1A5D99CA23B6DE2C9CC1B1369A0A74B04B45CA9EDAD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7557E988E9157162368E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34197948450CB5442AF76D6B5D4DB4D316A49F3D398ED16B7778428E15B06E2A8147D5350617AADC091D7E09C32AA3244CF3EC5BE573F1E23085C19F7BA7919B07408A6A02710B7304729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojSRIpe8siFhWeICJNq2KBrA== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D958C80E0CFE7CEC69DFD91911E5E978683D72C36FC87018B9F80AB2734326CD2FB559BB5D741EB96352A0ABBE4FDA4210A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v4 10/16] sql: refactor AVG() function X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mergen Imeev via Tarantool-patches Reply-To: Mergen Imeev Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thank you for the review! My answer, diff and new patch below. On Mon, Oct 11, 2021 at 11:50:39PM +0200, Vladislav Shpilevoy wrote: > Thanks for the fixes! > > >> @@ -141,17 +141,14 @@ fin_avg(struct sql_context *ctx) > >> assert(mem_is_null(ctx->pMem) || mem_is_bin(ctx->pMem)); > >> if (mem_is_null(ctx->pMem)) > >> return mem_set_null(ctx->pOut); > >> - struct Mem *tmp = (struct Mem *)ctx->pMem->z; > >> - uint32_t *count_val = (uint32_t *)(tmp + 1); > >> - struct Mem sum; > >> - mem_create(&sum); > >> - mem_copy_as_ephemeral(&sum, tmp); > >> - mem_destroy(tmp); > >> + struct Mem *sum = (struct Mem *)ctx->pMem->z; > >> + uint32_t *count_val = (uint32_t *)(sum + 1); > >> struct Mem count; > >> mem_create(&count); > >> mem_set_uint(&count, *count_val); > >> if (mem_div(&sum, &count, ctx->pOut) != 0) > >> ctx->is_aborted = true; > >> + mem_destroy(sum); > >> } > > This will work, however, I think it will create some unnecessary restrictions > > due to changes with pMem and pOut in a few patches. I suggest to apply part of > > you diff with exception of mem_destroy(), which I sugget to replace by assert(). > > We have full control over this tmp/sum mem and we know, that there will be no > > memory to free, so assert should be enough. > > > > What do you think of this diff? > > It looks the same as mine except you didn't call the destroy. I am fine with it, > but when I propose to wrap the check about a mem not needing a destroy into a > function. We should not use mem members as is when possible. It is a too > complicated structure. So far. > > Something like mem_is_trivial(). If it returns true, you don't need to > call mem_clear()/mem_destroy() and nothing will leak. Added mem_is_trivial(). Diff: diff --git a/src/box/sql/func.c b/src/box/sql/func.c index a811e55f9..8be553110 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -141,16 +141,13 @@ fin_avg(struct sql_context *ctx) assert(mem_is_null(ctx->pMem) || mem_is_bin(ctx->pMem)); if (mem_is_null(ctx->pMem)) return mem_set_null(ctx->pOut); - struct Mem *tmp = (struct Mem *)ctx->pMem->z; - uint32_t *count_val = (uint32_t *)(tmp + 1); - struct Mem sum; - mem_create(&sum); - mem_copy_as_ephemeral(&sum, tmp); - mem_destroy(tmp); + struct Mem *sum = (struct Mem *)ctx->pMem->z; + uint32_t *count_val = (uint32_t *)(sum + 1); + assert(mem_is_trivial(sum)); struct Mem count; mem_create(&count); mem_set_uint(&count, *count_val); - if (mem_div(&sum, &count, ctx->pOut) != 0) + if (mem_div(sum, &count, ctx->pOut) != 0) ctx->is_aborted = true; } diff --git a/src/box/sql/mem.h b/src/box/sql/mem.h index 7d5a750f5..52a63949a 100644 --- a/src/box/sql/mem.h +++ b/src/box/sql/mem.h @@ -237,6 +237,14 @@ mem_is_allocated(const struct Mem *mem) return mem_is_bytes(mem) && mem->z == mem->zMalloc; } +/** Return TRUE if MEM does not need to be freed or destroyed. */ +static inline bool +mem_is_trivial(const struct Mem *mem) +{ + return mem->szMalloc == 0 && (mem->flags & MEM_Dyn) == 0 && + (mem->type & (MEM_TYPE_FRAME | MEM_TYPE_AGG)) == 0; +} + static inline bool mem_is_cleared(const struct Mem *mem) { New patch: commit 18c50ab95a05c958cf1be016a482aa89f121f9b6 Author: Mergen Imeev Date: Thu Sep 9 18:19:53 2021 +0300 sql: refactor AVG() function Part of #4145 diff --git a/src/box/sql/func.c b/src/box/sql/func.c index c3c7ebec0..8be553110 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -102,6 +102,55 @@ fin_total(struct sql_context *ctx) mem_copy_as_ephemeral(ctx->pOut, ctx->pMem); } +/** Implementation of the AVG() function. */ +static void +step_avg(struct sql_context *ctx, int argc, struct Mem **argv) +{ + assert(argc == 1); + (void)argc; + assert(mem_is_null(ctx->pMem) || mem_is_bin(ctx->pMem)); + if (mem_is_null(argv[0])) + return; + struct Mem *mem; + uint32_t *count; + if (mem_is_null(ctx->pMem)) { + uint32_t size = sizeof(struct Mem) + sizeof(uint32_t); + mem = sqlDbMallocRawNN(sql_get(), size); + if (mem == NULL) { + ctx->is_aborted = true; + return; + } + count = (uint32_t *)(mem + 1); + mem_create(mem); + *count = 1; + mem_copy_as_ephemeral(mem, argv[0]); + mem_set_bin_allocated(ctx->pMem, (char *)mem, size); + return; + } + mem = (struct Mem *)ctx->pMem->z; + count = (uint32_t *)(mem + 1); + ++*count; + if (mem_add(mem, argv[0], mem) != 0) + ctx->is_aborted = true; +} + +/** Finalizer for the AVG() function. */ +static void +fin_avg(struct sql_context *ctx) +{ + assert(mem_is_null(ctx->pMem) || mem_is_bin(ctx->pMem)); + if (mem_is_null(ctx->pMem)) + return mem_set_null(ctx->pOut); + struct Mem *sum = (struct Mem *)ctx->pMem->z; + uint32_t *count_val = (uint32_t *)(sum + 1); + assert(mem_is_trivial(sum)); + struct Mem count; + mem_create(&count); + mem_set_uint(&count, *count_val); + if (mem_div(sum, &count, ctx->pOut) != 0) + ctx->is_aborted = true; +} + static const unsigned char * mem_as_ustr(struct Mem *mem) { @@ -1663,69 +1712,6 @@ soundexFunc(sql_context * context, int argc, sql_value ** argv) } } -/* - * An instance of the following structure holds the context of a - * sum() or avg() aggregate computation. - */ -typedef struct SumCtx SumCtx; -struct SumCtx { - struct Mem mem; - uint32_t count; -}; - -/* - * Routines used to compute the sum, average, and total. - * - * The SUM() function follows the (broken) SQL standard which means - * that it returns NULL if it sums over no inputs. TOTAL returns - * 0.0 in that case. In addition, TOTAL always returns a float where - * SUM might return an integer if it never encounters a floating point - * value. TOTAL never fails, but SUM might through an exception if - * it overflows an integer. - */ -static void -sum_step(struct sql_context *context, int argc, sql_value **argv) -{ - assert(argc == 1); - UNUSED_PARAMETER(argc); - struct SumCtx *p = sql_aggregate_context(context, sizeof(*p)); - if (p == NULL) { - context->is_aborted = true; - return; - } - if (p->count == 0) { - mem_create(&p->mem); - assert(context->func->def->returns == FIELD_TYPE_INTEGER || - context->func->def->returns == FIELD_TYPE_DOUBLE); - if (context->func->def->returns == FIELD_TYPE_INTEGER) - mem_set_uint(&p->mem, 0); - else - mem_set_double(&p->mem, 0.0); - } - if (argv[0]->type == MEM_TYPE_NULL) - return; - ++p->count; - assert(mem_is_num(argv[0])); - if (mem_add(&p->mem, argv[0], &p->mem) != 0) - context->is_aborted = true; -} - -static void -avgFinalize(sql_context * context) -{ - SumCtx *p; - p = sql_aggregate_context(context, 0); - if (p == NULL || p->count == 0) { - mem_set_null(context->pOut); - return; - } - struct Mem mem; - mem_create(&mem); - mem_set_uint(&mem, p->count); - if (mem_div(&p->mem, &mem, context->pOut) != 0) - context->is_aborted = true; -} - /* * The following structure keeps track of state information for the * count() aggregate function. @@ -2022,8 +2008,8 @@ struct sql_func_definition { static struct sql_func_definition definitions[] = { {"ABS", 1, {FIELD_TYPE_INTEGER}, FIELD_TYPE_INTEGER, absFunc, NULL}, {"ABS", 1, {FIELD_TYPE_DOUBLE}, FIELD_TYPE_DOUBLE, absFunc, NULL}, - {"AVG", 1, {FIELD_TYPE_INTEGER}, FIELD_TYPE_INTEGER, sum_step, avgFinalize}, - {"AVG", 1, {FIELD_TYPE_DOUBLE}, FIELD_TYPE_DOUBLE, sum_step, avgFinalize}, + {"AVG", 1, {FIELD_TYPE_INTEGER}, FIELD_TYPE_INTEGER, step_avg, fin_avg}, + {"AVG", 1, {FIELD_TYPE_DOUBLE}, FIELD_TYPE_DOUBLE, step_avg, fin_avg}, {"CHAR", -1, {FIELD_TYPE_INTEGER}, FIELD_TYPE_STRING, charFunc, NULL}, {"CHAR_LENGTH", 1, {FIELD_TYPE_STRING}, FIELD_TYPE_INTEGER, lengthFunc, NULL}, diff --git a/src/box/sql/mem.h b/src/box/sql/mem.h index 7d5a750f5..52a63949a 100644 --- a/src/box/sql/mem.h +++ b/src/box/sql/mem.h @@ -237,6 +237,14 @@ mem_is_allocated(const struct Mem *mem) return mem_is_bytes(mem) && mem->z == mem->zMalloc; } +/** Return TRUE if MEM does not need to be freed or destroyed. */ +static inline bool +mem_is_trivial(const struct Mem *mem) +{ + return mem->szMalloc == 0 && (mem->flags & MEM_Dyn) == 0 && + (mem->type & (MEM_TYPE_FRAME | MEM_TYPE_AGG)) == 0; +} + static inline bool mem_is_cleared(const struct Mem *mem) {