From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 4E49B7030C; Mon, 24 May 2021 18:49:49 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 4E49B7030C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1621871389; bh=Jpw3lnjHDDKxzS3Sdlz3wI6EerZjqFcVbO0ZlfcGnYo=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=sxhQ9eO6TJm6k4MSCZ7yK5l1C8944qwrlnPtYsZBVFfR2MKjqjbXi5vwf/BxOkft0 47MNk2qD0HkigXqG6Vt6VQcaetvzzo6vh8W8a5MRHiL/Z8biiv6A4n25vYWDx/Hl7o ew5LXnmVTTlwPlObOen9+AMrMjYBm7bRPWY48JCw= Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id BD36A7030C for ; Mon, 24 May 2021 18:49:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BD36A7030C Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1llCpu-0005y3-QE; Mon, 24 May 2021 18:49:47 +0300 To: Oleg Babin , tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org References: <2fb4c066558879eea74acab2e20b8a1c8f85d86b.1621778740.git.v.shpilevoy@tarantool.org> <6e775936-55ac-066e-c68f-743890f49a07@tarantool.org> Message-ID: <06d4c252-3f41-d91b-6943-ec4cca520a99@tarantool.org> Date: Mon, 24 May 2021 17:49:45 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.10.2 MIME-Version: 1.0 In-Reply-To: <6e775936-55ac-066e-c68f-743890f49a07@tarantool.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD91B019B01C53E51AF07EDA974EE11B688BBB65244C6A6BC9F00894C459B0CD1B9D47F430A0944000400FADEACF7A67D548E2CFE174D4D9C6F03D3BFA7FB8E1556 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7BCC85671EC7A750CEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063703E3935C5A8197E98638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8B7D47733C76C1684462281B3885BB3D9117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8B2EE5AD8F952D28FBA471835C12D1D977C4224003CC8364762BB6847A3DEAEFB0F43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7CFA80D66F452D417A43847C11F186F3C59DAA53EE0834AAEE X-C1DE0DAB: 0D63561A33F958A586464BA8A7929F804577E156817F5221FFECEB363E0126FED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA753C350047980234DB410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A08F09726D78E07B9C697ABE7B185103A7D5E392BE2E429C94BC6D73608AF9B82362FC9D1A06E84A1D7E09C32AA3244C9FFA0087295B2D7B5707C7C230B58B95F522A1CF68F4BE05FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj+gjVyQcIK6IcH8SvSIc8WA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822AB3928582CA39A052F2D66E64AFD005A3841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH 1/1] json: use cord_ibuf for encoding and decoding X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the review! On 24.05.2021 12:04, Oleg Babin wrote: > Hi! Thanks for your patch. > > > I see strange effect. After a patch following script: > > ``` > > for i = 1, 1e9 do pcall(json.encode, function() end) end > > ``` > > produces quite strange effects with memory. After some time > > my system kills a process - also I see in htop that process consumes about 20% of memory. > > In contrast before the patch process uses 0.1% of memory and doesn't have any oscillations > > in "VIRT" and "RES" columns. Yes, it's a negative case but I believe such behaviour shouldn't be affected as well. This is happening because you didn't do any yields. Cord buffer is freed automatically when a yield happens. This is a workaround for not being able to use a global buffer, which wouldn't need freeing at all. This is a known issue with the cord buffer, and the only working alternative I see is to wrap all related Lua C calls into lua_pcall(). This leads to perf issues for the success case, because pcall does more work; because you usually need to re-push the arguments; and because pcall is not jitted AFAIK. For instance about arguments re-push, to use lua_pcall() in lua_cjson in json_encode() I would need to push the Lua json.encode(...) arguments on the stack again. I couldn't find any good solution for the error-case so far. The same issue exists now with all the code which used IBUF_SHARED/tarantool_ibuf and now uses cord_ibuf_take()/put(). It does not justify the problem though. I was thinking about using pcall anyway; about pushing a GC function on Lua stack to free the cord buffer; about having a global buffer for normal context and another global buffer per each level of GC recursion. The last idea is not possible to implement due to lack of a concept of GC level in our Lua implementation. The other ideas are going to hit the perf for the success case. All looks bad. Your particular example started working when I added a yield every 10k encodes. >> diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c >> index 38e999870..85186d6d5 100644 >> --- a/third_party/lua-cjson/lua_cjson.c >> +++ b/third_party/lua-cjson/lua_cjson.c >> @@ -182,9 +177,6 @@ static int json_destroy_config(lua_State *l) >>   static void json_create_tokens() >>   { >>       int i; >> -#if DEFAULT_ENCODE_KEEP_BUFFER > 0 >> -    strbuf_init(&encode_buf, 0); >> -#endif >>         /* Decoding init */ >>   @@ -444,7 +436,9 @@ static int json_encode(lua_State *l) { >>                     "expected 1 or 2 arguments"); >>         /* Reuse existing buffer. */ >> -    strbuf_reset(&encode_buf); >> +    strbuf_t encode_buf; >> +    struct ibuf *ibuf = cord_ibuf_take(); >> +    strbuf_create(&encode_buf, -1, ibuf); > > Maybe it's better to use "0" here. I know it has the same effect but usually 0 is default value. But up to you. 0 looks like "do not pre-allocate anything". I used the default value explicitly now: ==================== @@ -438,7 +438,7 @@ static int json_encode(lua_State *l) { /* Reuse existing buffer. */ strbuf_t encode_buf; struct ibuf *ibuf = cord_ibuf_take(); - strbuf_create(&encode_buf, -1, ibuf); + strbuf_create(&encode_buf, STRBUF_DEFAULT_SIZE, ibuf); struct luaL_serializer *cfg = luaL_checkserializer(l); if (lua_gettop(l) == 2) { ====================