From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 3BB197030C; Mon, 24 May 2021 13:04:56 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3BB197030C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1621850696; bh=kE0Go4ZvHg7O3Xkb2Grsdc2lmGUuu2bqP0xIBlcC6hE=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=krNOO5zhBMQUh9Rx99sJ5gUV30ur4r5QjkdQsbnZm0iOFgmih2AjYkOpmvb89YYN3 R0P/2BwAVRA6IRDjue7BRINWTDcwQIgiNeQGDBY2F0/M0EkW9oj3rQ9JM9yii98Zy4 A59Br7PNHCZV95SiQpUUn307vZBQAEUOt+XwbJHo= Received: from smtp40.i.mail.ru (smtp40.i.mail.ru [94.100.177.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CA62D7030C for ; Mon, 24 May 2021 13:04:54 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CA62D7030C Received: by smtp40.i.mail.ru with esmtpa (envelope-from ) id 1ll7S9-0002kn-Pu; Mon, 24 May 2021 13:04:54 +0300 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org References: <2fb4c066558879eea74acab2e20b8a1c8f85d86b.1621778740.git.v.shpilevoy@tarantool.org> Message-ID: <6e775936-55ac-066e-c68f-743890f49a07@tarantool.org> Date: Mon, 24 May 2021 13:04:53 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.10.2 MIME-Version: 1.0 In-Reply-To: <2fb4c066558879eea74acab2e20b8a1c8f85d86b.1621778740.git.v.shpilevoy@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD91B019B01C53E51AFA94C0B24B2C939D4C80F0683D6F6F7C600894C459B0CD1B96B7BFBBCEC16693494951854AD43F5AD74AF30F0EEBEFC7D8E573506CDC45F9B X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78BAADB77C21FF6F2EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637D82ED515D6052E03EA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BD6CF32B5F8F9D404CE21157AB675B77E03DD57AE96BD7143CC7F00164DA146DAFE8445B8C89999728AA50765F7900637F6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F7900637F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA73AA81AA40904B5D9A18204E546F3947C0839144E5BB460BA2D242C3BD2E3F4C64AD6D5ED66289B52698AB9A7B718F8C46E0066C2D8992A16725E5C173C3A84C38ECE97E0F2853DA0BA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE75B51C8FB0C3E748C731C566533BA786AA5CC5B56E945C8DA X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24209795067102C07E8F7B195E1C9783175ECA14A47E8AAD9D5B09C2752DC36E1 X-C1DE0DAB: 0D63561A33F958A5610E5C35D1DA411499A4B59173CC46C71E56C04A4EBFC372D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA753C350047980234DB410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3407FE5477D6A8AF0808038151C44E7FC2BF50933F411C951251009417E7A7DD37F44C3A1FEC284CE61D7E09C32AA3244C64F8AF99C5967112FAB298597C6C4FB68A6D4CC6FBFAC251FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj+gjVyQcIK6LFnP1hA88xgA== X-Mailru-Sender: 583F1D7ACE8F49BD07526C4546A62CBF3F9C3F14258E352D3E9A6A9B85152199C313ED024E1BF76D23E75C7104EB1B885DEE61814008E47C7013064206BFB89F93956FB04BA385BE9437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 1/1] json: use cord_ibuf for encoding and decoding X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for your patch. I see strange effect. After a patch following script: ``` for i = 1, 1e9 do pcall(json.encode, function() end) end ``` produces quite strange effects with memory. After some time my system kills a process - also I see in htop that process consumes about 20% of memory. In contrast before the patch process uses 0.1% of memory and doesn't have any oscillations in "VIRT" and "RES" columns. Yes, it's a negative case but I believe such behaviour shouldn't be affected as well. Valid json encoding looks to be worked ok. One nit below. On 23.05.2021 17:06, Vladislav Shpilevoy wrote: > Lua json module used to have a global buffer for all encodings. It > was reused by each next encode(). > > This was not correct, because during encode() might happen a GC > step, which might call encode() again and spoil the global buffer. > > The same problem was already fixed for the global static buffer in > scope of #5632. Similarly to that time, the patch makes Lua json > module use cord_ibuf to prevent "concurrent" usage of the buffer > data. The global buffer is deleted. > > According to a few microbenchmarks it didn't affect the perf > anyhow. > > Core part of the patch is strbuf changes. Firstly, its destruction > is now optional, cord_ibuf can free itself on a next yield. > Secondly, its reallocation algorithm is kept intact - ibuf is used > as an allocator, not as the buffer itself. This is done so as not > to be too intrusive in the third party module which might need an > upgrade to the upstream in the future. > > Closes #6050 > --- > diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c > index 38e999870..85186d6d5 100644 > --- a/third_party/lua-cjson/lua_cjson.c > +++ b/third_party/lua-cjson/lua_cjson.c > @@ -51,8 +51,7 @@ > #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */ > #include "tt_static.h" > #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */ > - > -#define DEFAULT_ENCODE_KEEP_BUFFER 1 > +#include "cord_buf.h" > > typedef enum { > T_OBJ_BEGIN, > @@ -100,10 +99,6 @@ static struct luaL_serializer *luaL_json_default; > static json_token_type_t ch2token[256]; > static char escape2char[256]; /* Decoding */ > > -/* encode_buf is only allocated and used when > - * encode_keep_buffer is set */ > -static strbuf_t encode_buf; > - > typedef struct { > const char *data; > const char *ptr; > @@ -182,9 +177,6 @@ static int json_destroy_config(lua_State *l) > static void json_create_tokens() > { > int i; > -#if DEFAULT_ENCODE_KEEP_BUFFER > 0 > - strbuf_init(&encode_buf, 0); > -#endif > > /* Decoding init */ > > @@ -444,7 +436,9 @@ static int json_encode(lua_State *l) { > "expected 1 or 2 arguments"); > > /* Reuse existing buffer. */ > - strbuf_reset(&encode_buf); > + strbuf_t encode_buf; > + struct ibuf *ibuf = cord_ibuf_take(); > + strbuf_create(&encode_buf, -1, ibuf); Maybe it's better to use "0" here. I know it has the same effect but usually 0 is default value. But up to you.