Tarantool development patches archive
 help / color / mirror / Atom feed
From: Oleg Babin via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>,
	tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org
Subject: Re: [Tarantool-patches] [PATCH 1/1] json: use cord_ibuf for encoding and decoding
Date: Mon, 24 May 2021 13:04:53 +0300	[thread overview]
Message-ID: <6e775936-55ac-066e-c68f-743890f49a07@tarantool.org> (raw)
In-Reply-To: <2fb4c066558879eea74acab2e20b8a1c8f85d86b.1621778740.git.v.shpilevoy@tarantool.org>

Hi! Thanks for your patch.


I see strange effect. After a patch following script:

```

for i = 1, 1e9 do pcall(json.encode, function() end) end

```

produces quite strange effects with memory. After some time

my system kills a process - also I see in htop that process consumes 
about 20% of memory.

In contrast before the patch process uses 0.1% of memory and doesn't 
have any oscillations

in "VIRT" and "RES" columns. Yes, it's a negative case but I believe 
such behaviour shouldn't be affected as well.


Valid json encoding looks to be worked ok.

One nit below.


On 23.05.2021 17:06, Vladislav Shpilevoy wrote:
> Lua json module used to have a global buffer for all encodings. It
> was reused by each next encode().
>
> This was not correct, because during encode() might happen a GC
> step, which might call encode() again and spoil the global buffer.
>
> The same problem was already fixed for the global static buffer in
> scope of #5632. Similarly to that time, the patch makes Lua json
> module use cord_ibuf to prevent "concurrent" usage of the buffer
> data. The global buffer is deleted.
>
> According to a few microbenchmarks it didn't affect the perf
> anyhow.
>
> Core part of the patch is strbuf changes. Firstly, its destruction
> is now optional, cord_ibuf can free itself on a next yield.
> Secondly, its reallocation algorithm is kept intact - ibuf is used
> as an allocator, not as the buffer itself. This is done so as not
> to be too intrusive in the third party module which might need an
> upgrade to the upstream in the future.
>
> Closes #6050
> ---
> diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c
> index 38e999870..85186d6d5 100644
> --- a/third_party/lua-cjson/lua_cjson.c
> +++ b/third_party/lua-cjson/lua_cjson.c
> @@ -51,8 +51,7 @@
>   #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
>   #include "tt_static.h"
>   #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */
> -
> -#define DEFAULT_ENCODE_KEEP_BUFFER 1
> +#include "cord_buf.h"
>   
>   typedef enum {
>       T_OBJ_BEGIN,
> @@ -100,10 +99,6 @@ static struct luaL_serializer *luaL_json_default;
>   static json_token_type_t ch2token[256];
>   static char escape2char[256];  /* Decoding */
>   
> -/* encode_buf is only allocated and used when
> - * encode_keep_buffer is set */
> -static strbuf_t encode_buf;
> -
>   typedef struct {
>       const char *data;
>       const char *ptr;
> @@ -182,9 +177,6 @@ static int json_destroy_config(lua_State *l)
>   static void json_create_tokens()
>   {
>       int i;
> -#if DEFAULT_ENCODE_KEEP_BUFFER > 0
> -    strbuf_init(&encode_buf, 0);
> -#endif
>   
>       /* Decoding init */
>   
> @@ -444,7 +436,9 @@ static int json_encode(lua_State *l) {
>                     "expected 1 or 2 arguments");
>   
>       /* Reuse existing buffer. */
> -    strbuf_reset(&encode_buf);
> +    strbuf_t encode_buf;
> +    struct ibuf *ibuf = cord_ibuf_take();
> +    strbuf_create(&encode_buf, -1, ibuf);

Maybe it's better to use "0" here. I know it has the same effect but 
usually 0 is default value. But up to you.


  reply	other threads:[~2021-05-24 10:04 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-23 14:06 Vladislav Shpilevoy via Tarantool-patches
2021-05-24 10:04 ` Oleg Babin via Tarantool-patches [this message]
2021-05-24 15:49   ` Vladislav Shpilevoy via Tarantool-patches
2021-05-24 16:00     ` Oleg Babin via Tarantool-patches
2021-05-24 13:01 ` Serge Petrenko via Tarantool-patches
2021-05-24 13:05   ` Serge Petrenko via Tarantool-patches
2021-05-24 15:47     ` Vladislav Shpilevoy via Tarantool-patches
2021-05-24 15:47   ` Vladislav Shpilevoy via Tarantool-patches
2021-05-24 16:17     ` Serge Petrenko via Tarantool-patches
2021-05-25 21:20 ` Vladislav Shpilevoy via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6e775936-55ac-066e-c68f-743890f49a07@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=olegrok@tarantool.org \
    --cc=sergepetrenko@tarantool.org \
    --cc=v.shpilevoy@tarantool.org \
    --subject='Re: [Tarantool-patches] [PATCH 1/1] json: use cord_ibuf for encoding and decoding' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox