[Tarantool-patches] [PATCH 1/1] json: use cord_ibuf for encoding and decoding
Oleg Babin
olegrok at tarantool.org
Mon May 24 13:04:53 MSK 2021
Hi! Thanks for your patch.
I see strange effect. After a patch following script:
```
for i = 1, 1e9 do pcall(json.encode, function() end) end
```
produces quite strange effects with memory. After some time
my system kills a process - also I see in htop that process consumes
about 20% of memory.
In contrast before the patch process uses 0.1% of memory and doesn't
have any oscillations
in "VIRT" and "RES" columns. Yes, it's a negative case but I believe
such behaviour shouldn't be affected as well.
Valid json encoding looks to be worked ok.
One nit below.
On 23.05.2021 17:06, Vladislav Shpilevoy wrote:
> Lua json module used to have a global buffer for all encodings. It
> was reused by each next encode().
>
> This was not correct, because during encode() might happen a GC
> step, which might call encode() again and spoil the global buffer.
>
> The same problem was already fixed for the global static buffer in
> scope of #5632. Similarly to that time, the patch makes Lua json
> module use cord_ibuf to prevent "concurrent" usage of the buffer
> data. The global buffer is deleted.
>
> According to a few microbenchmarks it didn't affect the perf
> anyhow.
>
> Core part of the patch is strbuf changes. Firstly, its destruction
> is now optional, cord_ibuf can free itself on a next yield.
> Secondly, its reallocation algorithm is kept intact - ibuf is used
> as an allocator, not as the buffer itself. This is done so as not
> to be too intrusive in the third party module which might need an
> upgrade to the upstream in the future.
>
> Closes #6050
> ---
> diff --git a/third_party/lua-cjson/lua_cjson.c b/third_party/lua-cjson/lua_cjson.c
> index 38e999870..85186d6d5 100644
> --- a/third_party/lua-cjson/lua_cjson.c
> +++ b/third_party/lua-cjson/lua_cjson.c
> @@ -51,8 +51,7 @@
> #include "mp_extension_types.h" /* MP_DECIMAL, MP_UUID */
> #include "tt_static.h"
> #include "uuid/tt_uuid.h" /* tt_uuid_to_string(), UUID_STR_LEN */
> -
> -#define DEFAULT_ENCODE_KEEP_BUFFER 1
> +#include "cord_buf.h"
>
> typedef enum {
> T_OBJ_BEGIN,
> @@ -100,10 +99,6 @@ static struct luaL_serializer *luaL_json_default;
> static json_token_type_t ch2token[256];
> static char escape2char[256]; /* Decoding */
>
> -/* encode_buf is only allocated and used when
> - * encode_keep_buffer is set */
> -static strbuf_t encode_buf;
> -
> typedef struct {
> const char *data;
> const char *ptr;
> @@ -182,9 +177,6 @@ static int json_destroy_config(lua_State *l)
> static void json_create_tokens()
> {
> int i;
> -#if DEFAULT_ENCODE_KEEP_BUFFER > 0
> - strbuf_init(&encode_buf, 0);
> -#endif
>
> /* Decoding init */
>
> @@ -444,7 +436,9 @@ static int json_encode(lua_State *l) {
> "expected 1 or 2 arguments");
>
> /* Reuse existing buffer. */
> - strbuf_reset(&encode_buf);
> + strbuf_t encode_buf;
> + struct ibuf *ibuf = cord_ibuf_take();
> + strbuf_create(&encode_buf, -1, ibuf);
Maybe it's better to use "0" here. I know it has the same effect but
usually 0 is default value. But up to you.
More information about the Tarantool-patches
mailing list