From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 7BC7F6EC5D; Tue, 6 Apr 2021 01:05:33 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7BC7F6EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617660333; bh=6yAYUlomc8lcJUIprCyIJhLT+HTw2XyN+he9IQlfPGY=; h=In-Reply-To:Date:Cc:References:To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=gm7d8OCqRK2/pYF0k617eTnp9HHRcZP4JgTWdP6xP0dwXp8pfqLKotYQOgGMVxHTD BIDr7fkdMrx2LEaDh9kUS2b54BaIl5XLUE0sblyBSKL65UgwcJ4/C5VJXidmlIO7Kj bVqyUyP/Iq/K6doR2CiivxLuXVTgQb98oaQ/YezM= Received: from smtp16.mail.ru (smtp16.mail.ru [94.100.176.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CDDE26EC5D for ; Tue, 6 Apr 2021 01:05:31 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CDDE26EC5D Received: by smtp16.mail.ru with esmtpa (envelope-from ) id 1lTXLe-0001VV-RK; Tue, 06 Apr 2021 01:05:31 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) In-Reply-To: <20210314124220.GA16737@root> Date: Tue, 6 Apr 2021 01:05:29 +0300 Cc: tarantool-patches@dev.tarantool.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20210311054949.91263-1-roman.habibov@tarantool.org> <20210314124220.GA16737@root> To: Sergey Kaplun X-Mailer: Apple Mail (2.3608.120.23.2.1) X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9ED7173E37F4E32947427BE79D20CABD4F844F00D7AFD5C3C182A05F538085040858549F7269C4F6BECCF04B38DF222B6BE8B85DE0D51E2E24BC8197C92EBE898 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE77C1346FE4B18DC51EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063732727FD5A4355FD18638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C16EE06F5A270FE6AA631BC0D4F85C46DDC7A54DFE5D32A19A471835C12D1D9774AD6D5ED66289B5259CC434672EE6371117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658359CC434672EE6371117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CF7928A8123BBB9D52BA3038C0950A5D36C8A9BA7A39EFB766EC990983EF5C0329BA3038C0950A5D36D5E8D9A59859A8B625A4C53D6AAA6D4F76E601842F6C81A1F004C906525384307823802FF610243DF43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7C64E4DC1543031AB7C4224003CC83647689D4C264860C145E X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D0152948A36B1B65F355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A5A374457F6C96AD531239E6E7ACAA0F6DF8DD3ECFEDB37B94D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34BD150993ED26DDF42233B87E955B1BA8AA49EC9E5DFC256DCF5DE2320569F57892C8300B9890A8A31D7E09C32AA3244C02F0E4FE3B5DE169990EE9E414F04BC455E75C8D0ED9F6EE729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojM00ve/f+0omHgvObo5z35A== X-Mailru-Sender: ED747E36EB90C325A4602F7CA2CC58C18D4A7DA9B32B615C9A1DB7A9580279DA3FB1763410276AF21EC3B765AEBF8DFDC77752E0C033A69EF2501F26BCC01020D1D7FFF4A7B59B3C6C18EFA0BB12DBB0 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] serializer: serialize recursive structures X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Roman Khabibov via Tarantool-patches Reply-To: Roman Khabibov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for your review. Perf Script: local serializer =3D require('json') clock =3D require('clock') serializer.cfg({encode_max_depth =3D 64}) local avg =3D 0 for j =3D 1, 10 do local time =3D clock.monotonic() for i =3D 1, 150000 do local rec1 =3D {{{{{{{{{{{{{{{{}}}}}}}}}}}}}}}} serializer.encode(rec1) end local result =3D clock.monotonic() - time print(result) avg =3D avg + result end print("avg", avg / 10) JSON master r:tarantool r.khabibov$ ./src/tarantool a.lua 2.3421729998663 2.4643029998988 2.4067219998688 2.5030510001816 2.6378739997745 2.3903989996761 2.9206310003065 2.9560130001046 2.8415450002067 2.8488600002602 avg 2.6311571000144 my branch 2.3160850000568 2.4038570001721 2.4244269998744 2.4124370003119 2.4159540003166 2.4096140000038 2.4202209999785 2.3913590000011 2.4089310001582 2.4692329997197 avg 2.4072118000593 MSGPACK master 4.1559460000135 3.415306000039 3.2962070000358 3.0802390002646 2.8740380001254 2.7747510001063 2.7418200001121 2.7159779998474 2.8011710001156 3.1459960001521 avg 3.1001452000812 my branch 2.6075639999472 2.7236989997327 3.0367560000159 3.071974999737 4.0645679999143 3.3987269997597 3.0836050002836 3.1121310000308 2.9265290000476 3.1954760001972 avg 3.1221029999666 > On Mar 14, 2021, at 15:42, Sergey Kaplun = wrote: >=20 > Hi! >=20 > Thanks for the patch! >=20 > Side note: I've got into the swing of things by looking of previous > series of patches ([1][2]). AFAICS, there was idea to throw an error = on > recursive serialization, is there discussion anywhere about this > implementation instead the old one which I've missed? >=20 > I have some questions about current version of serializer: >=20 > Nit: we have no need to add a marker for structures that are not used: > | Tarantool 2.8.0-108-g195d44ddc > | tarantool> local a =3D {} local b =3D {a =3D a} b[b] =3D b a[b] =3D = a setmetatable(b, {__serialize =3D function(...) return {a} end}) return = a > | --- > | - &0 > | ? &1 > | - *0 > | : *0 > | ... >=20 > And when serialize is omitted, it works like expected: > | Tarantool 2.8.0-108-g195d44ddc > | tarantool> local a =3D {} local b =3D {a =3D a} b[b] =3D b a[b] =3D = a return a > | --- > | - &0 > | ? &1 > | a: *0 > | *1: *1 > | : *0 > | ... >=20 > In the old version it looks minimalistic and good: > | Tarantool 2.8.0-0-gefc30ccf8 > | tarantool> local a =3D {} local b =3D {a =3D a} b[b] =3D b a[b] =3D = a setmetatable(b, {__serialize =3D function(...) return {a} end}) return = a > | --- > | - &0 > | ? - *0 > | : *0 > | ... >=20 > Also, I found some inconsistency in showing repeated objects. > For the new version: > | Tarantool 2.8.0-108-g195d44ddc > | local a =3D {} local b =3D {} a[b] =3D b setmetatable(b, = {__serialize =3D function(...) return print end}) return a > | --- > | - 'function: builtin#29': *0 > | ... >=20 > But '&0' does not exist, so it looks confusing. Same story with = threads. >=20 > And for old version: >=20 > | Tarantool 2.8.0-0-gefc30ccf8 > | tarantool> local a =3D {} local b =3D {} a[b] =3D b setmetatable(b, = {__serialize =3D function(...) return print end}) return a > | --- > | - 'function: builtin#29': 'function: builtin#29' > | ... >=20 >=20 > Side note: I'm not sure that the next example is adequate, so I CC-ed = Sergos > to judge me. > The main idea is to change a serialization function inside = serialization. > It looks weird and useless, to be honest, but who knows. Here is the = example: >=20 > | Tarantool 2.8.0-108-g195d44ddc > | tarantool> local function example() > | > local a =3D {} > | > local b =3D {} > | > b[a] =3D a > | > local reta > | > local retb > | > local function swap_ser(o) > | > local newf > | > local f =3D getmetatable(o).__serialize > | > if f =3D=3D reta then > | > newf =3D retb > | > else > | > newf =3D reta > | > end > | > getmetatable(o).__serialize =3D newf > | > end > | > reta =3D function(o) swap_ser(o) return "a" end > | > retb =3D function(o) swap_ser(o) return "b" end > | > setmetatable(a, {__serialize =3D reta}) > | > return b > | > end return example() > | --- > | - a: *0 > | ... >=20 > Note, that '*0' looks even more confusing now :). >=20 > For the old version: > | Tarantool 2.8.0-0-gefc30ccf8 > | tarantool> local function example() > | > local a =3D {} > | > local b =3D {} > | > b[a] =3D a > | > local reta > | > local retb > | > local function swap_ser(o) > | > local newf > | > local f =3D getmetatable(o).__serialize > | > if f =3D=3D reta then > | > newf =3D retb > | > else > | > newf =3D reta > | > end > | > getmetatable(o).__serialize =3D newf > | > end > | > reta =3D function(o) swap_ser(o) return "a" end > | > retb =3D function(o) swap_ser(o) return "b" end > | > setmetatable(a, {__serialize =3D reta}) > | > return b > | > end return example() > | --- > | - a: b > | ... >=20 > Looks like we should check not only object itself, but that the = serialization > function is the same. >=20 > Moreover, I don't know, what can we do with situations, when a = serialize > function depends on some context that can be changed inside it: >=20 > The new version: > | Tarantool 2.8.0-108-g195d44ddc > | tarantool> local a =3D {} local b =3D {} b[a] =3D a local show_a =3D = true setmetatable(a, {__serialize =3D function() show_a =3D not show_a = if show_a then return "a" else return "b" end end}) return b > | --- > | - b: *0 > | ... >=20 > The old version works like expected: > | Tarantool 2.8.0-0-gefc30ccf8 > | tarantool> local a =3D {} local b =3D {} b[a] =3D a local show_a =3D = true setmetatable(a, {__serialize =3D function() show_a =3D not show_a = if show_a then return "a" else return "b" end end}) return b > | --- > | - b: a > | ... >=20 > Side note: It doesn't relate to issue and patch directly, but > deep recursive calls still lead to core: > | Tarantool 2.8.0-108-g195d44ddc > | tarantool> local a =3D {} a[a] =3D a local recf recf =3D function(t) = return setmetatable({}, {__serialize =3D recf}) end setmetatable(a, = {__serialize =3D recf}) return a > | Segmentation fault (core dumped) >=20 > With the backtrace that looks like the following: > | #8 0x000055fd217034d3 in lua_field_try_serialize (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1375, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:694 > | #9 0x000055fd2170387d in lua_field_inspect_table (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1375, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:740 > | #10 0x000055fd21704542 in luaL_tofield (L=3D0x40ffb378, = cfg=3D0x41c6e8c0, anchortable_index=3D3, opts=3D0x0, index=3D1375, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:944 > | #11 0x000055fd217034f8 in lua_field_try_serialize (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1374, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:695 > | #12 0x000055fd2170387d in lua_field_inspect_table (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1374, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:740 > | #13 0x000055fd21704542 in luaL_tofield (L=3D0x40ffb378, = cfg=3D0x41c6e8c0, anchortable_index=3D3, opts=3D0x0, index=3D1374, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:944 > | #14 0x000055fd217034f8 in lua_field_try_serialize (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1373, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:695 > | #15 0x000055fd2170387d in lua_field_inspect_table (L=3D0x40ffb378, = anchortable_index=3D3, cfg=3D0x41c6e8c0, idx=3D1373, = field=3D0x7f29425fd760) > | at /home/burii/reviews/tarantool/serialize/src/lua/utils.c:740 > | <4K lines below> >=20 > I suppose that it can be handled by checking of the serialize level of > structure (table). If its to high we can either raise an error or > return for the rest of the structure. At least it will be > good to create an issue follow-ups this problem. See updated tests. =20 > Since the answers to the questions above are critical to evaluate the > logic of the patch, I have not reviewed it yet. Most parts of the > comments below is about to add more comments to code or some = suggestions > about codestile. >=20 > On 11.03.21, Roman Khabibov wrote: >> Fix bug with bus error during serializing of recursive >> structures. >=20 > Please provide a little bit more description about your changes. > It helps me as a reviewer to check the patch and for other developers > it will be easier to understand these changes in the future. Done. >>=20 >> Closes #3228 >> --- >>=20 >> Branch: = https://github.com/tarantool/tarantool/tree/romanhabibov/gh-3228-visited-s= et >> Issue: https://github.com/tarantool/tarantool/issues/3228 >>=20 >> src/box/lua/call.c | 6 +- >> src/box/lua/execute.c | 2 +- >> src/box/lua/serialize_lua.c | 96 ++------ >> src/box/lua/tuple.c | 2 +- >> src/box/sql/func.c | 2 +- >> src/lua/msgpack.c | 10 +- >> src/lua/pickle.c | 2 +- >> src/lua/utils.c | 226 = ++++++++++++++++-- >> src/lua/utils.h | 43 +++- >> ...-3228-serializer-look-for-recursion.result | 67 ++++++ >> ...228-serializer-look-for-recursion.test.lua | 17 ++ >> test/swim/swim.result | 18 +- >> third_party/lua-cjson/lua_cjson.c | 4 +- >> third_party/lua-yaml/lyaml.cc | 88 +++---- >> 14 files changed, 390 insertions(+), 193 deletions(-) >> create mode 100644 = test/app/gh-3228-serializer-look-for-recursion.result >> create mode 100644 = test/app/gh-3228-serializer-look-for-recursion.test.lua >=20 > Please provide Changelog entry too. >>=20 >> diff --git a/src/box/lua/call.c b/src/box/lua/call.c >> index 0315e720c..e4907a3f7 100644 >> --- a/src/box/lua/call.c >> +++ b/src/box/lua/call.c >=20 > >=20 >> diff --git a/src/box/lua/serialize_lua.c = b/src/box/lua/serialize_lua.c >> index caa08a60f..ab612dfc2 100644 >> --- a/src/box/lua/serialize_lua.c >> +++ b/src/box/lua/serialize_lua.c >> @@ -215,7 +215,7 @@ trace_node(struct lua_dumper *d) >=20 > >=20 >> @@ -339,41 +339,16 @@ emit_node(struct lua_dumper *d, struct node = *nd, int indent, >> static const char * >> get_lua_anchor(struct lua_dumper *d) >> { >> - const char *s =3D ""; >> - >> - lua_pushvalue(d->L, -1); >> - lua_rawget(d->L, d->anchortable_index); >> - if (!lua_toboolean(d->L, -1)) { >> - lua_pop(d->L, 1); >> - return NULL; >> - } >> - >> - if (lua_isboolean(d->L, -1)) { >> - /* >> - * This element is referenced more >> - * than once but has not been named. >> - */ >> - char buf[32]; >> - snprintf(buf, sizeof(buf), "%u", d->anchor_number++); >> - lua_pop(d->L, 1); >> - lua_pushvalue(d->L, -1); >> - lua_pushstring(d->L, buf); >> - s =3D lua_tostring(d->L, -1); >> - lua_rawset(d->L, d->anchortable_index); >> - trace_anchor(s, false); >> - } else { >> - /* >> - * An aliased element. >> - * >> - * FIXME: Need an example to use. >> - * >> - * const char *str =3D lua_tostring(d->L, -1); >> - */ >> - const char *str =3D lua_tostring(d->L, -1); >> - trace_anchor(str, true); >> - lua_pop(d->L, 1); >> + const char *anchor =3D NULL; >> + int code =3D get_anchor(d->L, d->anchortable_index, = &d->anchor_number, >> + &anchor); >> + if (code =3D=3D 1) { >=20 > Please avoid using magic numbers. Here and below. Done. See enums. >> + trace_anchor(anchor, false); >> + } else if (code =3D=3D 2) { >> + trace_anchor(anchor, true); >> + return ""; >> } >> - return s; >> + return anchor; >> } >>=20 >> static void >> @@ -783,7 +758,7 @@ dump_node(struct lua_dumper *d, struct node *nd, = int indent) >=20 > >=20 >> @@ -881,49 +856,6 @@ dump_node(struct lua_dumper *d, struct node *nd, = int indent) >> return emit_node(d, nd, indent, str, len); >> } >>=20 >> -/** >> - * Find references to tables, we use it >> - * to find self references in tables. >> - */ >> -static void >> -find_references(struct lua_dumper *d) >> -{ >=20 > >=20 >> -} >> - >> /** >> * Dump recursively from the root node. >> */ >> @@ -935,7 +867,7 @@ dump_root(struct lua_dumper *d) >=20 > >=20 >> diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c >> index 3e6f043b4..4d417ec34 100644 >> --- a/src/box/lua/tuple.c >> +++ b/src/box/lua/tuple.c >=20 > >=20 >> diff --git a/src/box/sql/func.c b/src/box/sql/func.c >> index f15d27051..d4a7064f0 100644 >> --- a/src/box/sql/func.c >> +++ b/src/box/sql/func.c >=20 > >=20 >> diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c >> index 24b0d2ccd..412736572 100644 >> --- a/src/lua/msgpack.c >> +++ b/src/lua/msgpack.c >=20 > >=20 >> diff --git a/src/lua/pickle.c b/src/lua/pickle.c >> index 65208b5b3..86f0f77e8 100644 >> --- a/src/lua/pickle.c >> +++ b/src/lua/pickle.c >=20 > >=20 >> diff --git a/src/lua/utils.c b/src/lua/utils.c >> index b5a6ca5b7..64404e3fc 100644 >> --- a/src/lua/utils.c >> +++ b/src/lua/utils.c >> @@ -439,9 +439,54 @@ lua_gettable_wrapper(lua_State *L) >> return 1; >> } >>=20 >> +/** >> + * Check if node with @a idx serialized. >> + */ >> +static int >> +is_serialized(struct lua_State *L, int anchortable_index, int idx) >> +{ >> + if (anchortable_index =3D=3D 0) >> + return 0; >> + lua_pushvalue(L, idx); >> + lua_rawget(L, anchortable_index); >> + int is_serialized =3D 0; >> + if (lua_isnil(L, -1) =3D=3D 0) { >=20 > Nit: This comparison is good according to current Tarantool's code > style, but all old checks are using just `lua_is*()`. > Personally, I prefer to keep code style consistence inside one > translation unit. > Feel free to ignore. > Here and below. >> + lua_pushstring(L, "serialized"); >> + lua_rawget(L, -2); >> + is_serialized =3D lua_toboolean(L, -1); >> + lua_pop(L, 1); >> + } >> + lua_pop(L, 1); >> + return is_serialized; >> +} >> + >> +/** >> + * Mark node with @a idx as serialized. >> + */ >> +static void >> +mark_as_serialized(struct lua_State *L, int anchortable_index, int = idx) >> +{ >> + if (anchortable_index =3D=3D 0) >> + return; >> + /* Push copy of table. */ >=20 > For what? Feel free to be more descriptive. I added comments. >> + lua_pushvalue(L, idx); >> + lua_pushvalue(L, idx); >> + lua_rawget(L, anchortable_index); >> + if (lua_isnil(L, -1) =3D=3D 1) { >> + lua_pop(L, 1); >> + lua_newtable(L); >> + } >> + int flags_index =3D lua_gettop(L); >> + lua_pushstring(L, "serialized"); >> + lua_pushboolean(L, true); >> + lua_rawset(L, flags_index); >> + lua_rawset(L, anchortable_index); >> +} >> + >=20 > >=20 >> +void >> +find_references(struct lua_State *L, int anchortable_index, const = char *mode) >> +{ >> + int type =3D lua_type(L, -1); >> + if (type !=3D LUA_TTABLE || anchortable_index =3D=3D 0) >> + return; >> + >> + int node_index =3D lua_gettop(L); >> + lua_pushvalue(L, node_index); >> + >> + /* Get value with flags from anchor table. */ >> + lua_pushvalue(L, node_index); >> + lua_rawget(L, anchortable_index); >> + if (lua_isnil(L, -1) =3D=3D 1) { >> + lua_pop(L, 1); >> + lua_newtable(L); >> + } >> + int flags_index =3D lua_gettop(L); >> + >> + /* Get and increment counter. */ >> + lua_pushstring(L, mode); >> + lua_rawget(L, flags_index); >> + >> + int new_count =3D 0; >> + if (lua_isnil(L, -1) =3D=3D 1) { >> + new_count =3D 1; >> + lua_pop(L, 1); >> + } else { >> + lua_Integer number =3D lua_tointeger(L, -1); >> + lua_pop(L, 1); >> + if (number < 2) >> + new_count =3D ++number; >> + else >> + new_count =3D -1; >> + } >> + >> + /* Push new count to value with flags. */ >> + if (new_count =3D=3D -1) { >> + lua_pop(L, 2); >> + assert(node_index =3D=3D lua_gettop(L)); >> + return; >> + } >> + lua_pushstring(L, mode); >> + lua_pushinteger(L, new_count); >> + lua_rawset(L, flags_index); >> + >> + /* Check if node is already serialized. */ >> + bool already_serialized =3D false; >> + if (strcmp(mode, "after") =3D=3D 0) { >> + lua_pushstring(L, "serialized"); >> + lua_rawget(L, flags_index); >> + already_serialized =3D lua_toboolean(L, -1); >> + lua_pop(L, 1); >> + } >> + lua_rawset(L, anchortable_index); >> + assert(node_index =3D=3D lua_gettop(L)); >> + if (already_serialized =3D=3D true) >> + return; >> + >> + /* Recursively process other table values. */ >> + lua_pushnil(L); >> + while (lua_next(L, -2) !=3D 0) { >> + /* Find references on value. */ >> + find_references(L, anchortable_index, mode); >> + lua_pop(L, 1); >> + /* Find references on key. */ >> + find_references(L, anchortable_index, mode); >> + } >> +} >> + >> +int >> +get_anchor(struct lua_State *L, int anchortable_index, >> + unsigned int *anchor_number, const char **anchor) >> +{ >> + lua_pushvalue(L, -1); >> + lua_rawget(L, anchortable_index); >> + if (lua_isnil(L, -1) =3D=3D 1) { >> + lua_pop(L, 1); >> + *anchor =3D NULL; >> + return 0; >> + } >> + >> + lua_pushstring(L, "before"); >> + lua_rawget(L, -2); >> + const char *str =3D NULL; >> + if (lua_type(L, -1) =3D=3D LUA_TSTRING) >> + str =3D lua_tostring(L, -1); >> + lua_Integer number_before =3D lua_tointeger(L, -1); >> + lua_pop(L, 1); >> + lua_Integer number_after =3D 0; >> + if (str =3D=3D NULL) { >> + lua_pushstring(L, "after"); >> + lua_rawget(L, -2); >> + if (lua_type(L, -1) =3D=3D LUA_TSTRING) >> + str =3D lua_tostring(L, -1); >> + number_after =3D lua_tointeger(L, -1); >> + lua_pop(L, 1); >> + } >> + >> + *anchor =3D ""; >> + int ret =3D 0; >> + if ((number_before > 1 || number_after > 1) && str =3D=3D NULL) = { >> + /* >> + * This element is referenced more than once but >> + * has not been named. >> + */ >> + char buf[32]; >> + snprintf(buf, sizeof(buf), "%u", (*anchor_number)++); >=20 > According to your comment: > | @param[out] anchor_number Ptr to the number anchors. > Does it determines only by this function (as far as it is output = value)? > If yes, should you set *anchor_number to 0? And can it be more than 1? Done. >> + lua_pop(L, 1); >> + lua_pushvalue(L, -1); >> + >> + lua_pushvalue(L, -1); >> + lua_rawget(L, anchortable_index); >> + if (number_before > 1) >> + lua_pushstring(L, "before"); >> + else >> + lua_pushstring(L, "after"); >=20 > Looks like `lua_pushstring(L, number_before > 1 ? "before" : = "after");` > is enough, isn't it? It isn=E2=80=99t actually now. >> + lua_pushstring(L, buf); >> + *anchor =3D lua_tostring(L, -1); >> + lua_rawset(L, -3); >> + lua_rawset(L, anchortable_index); >> + ret =3D 1; >> + } else if (str !=3D NULL) { >> + /* This is an aliased element. */ >> + lua_pop(L, 1); >> + *anchor =3D str; >> + ret =3D 2; >> + } else { >> + lua_pop(L, 1); >> + *anchor =3D NULL; >> + ret =3D 0; >> + } >> + return ret; >> +} >> + >> /** >> * Call __serialize method of a table object by index >> * if the former exists. >> @@ -496,11 +677,13 @@ lua_field_inspect_ucdata(struct lua_State *L, = struct luaL_serializer *cfg, >=20 > >=20 >> diff --git a/src/lua/utils.h b/src/lua/utils.h >> index 37531676d..ef9c70dfb 100644 >> --- a/src/lua/utils.h >> +++ b/src/lua/utils.h >> @@ -354,6 +354,29 @@ struct luaL_field { >> bool compact; /* a flag used by YAML serializer = */ >> }; >>=20 >> +/** >> + * Find references to tables to look up self references in tables. >> + * >> + * @param L Lua stack. >> + * @param anchortable_index Index of anchor table on stack. >> + * @param mode When the function is called before or after dump >> + function. >> + */ >=20 > Please add some information about how this function modifies > table by `anchortable_index`. May be it makes sense to reference > this function by `luaL_tofield()` and `luaL_convertfield()`. Done. >> +void >> +find_references(struct lua_State *L, int anchortable_index, const = char *mode); >> + >> +/** >> + * Generate aliases and anchor numbers for self references. >> + * >> + * @param L Lua stack. >> + * @param anchortable_index Index of anchor table on stack. >> + * @param[out] anchor_number Ptr to the number anchors. >> + * @param[out] anchor Ptr to anchor string. >> + */ >> +int >> +get_anchor(struct lua_State *L, int anchortable_index, >> + unsigned int *anchor_number, const char **anchor); >> + >> /** >> * @brief Convert a value from the Lua stack to a lua_field = structure. >> * This function is designed for use with Lua bindings and data >> @@ -388,6 +411,7 @@ struct luaL_field { >=20 > >=20 >> diff --git a/test/app/gh-3228-serializer-look-for-recursion.result = b/test/app/gh-3228-serializer-look-for-recursion.result >> new file mode 100644 >> index 000000000..773470c8e >> --- /dev/null >> +++ b/test/app/gh-3228-serializer-look-for-recursion.result >=20 > >=20 >> diff --git a/test/app/gh-3228-serializer-look-for-recursion.test.lua = b/test/app/gh-3228-serializer-look-for-recursion.test.lua >> new file mode 100644 >> index 000000000..1c9b0375f >> --- /dev/null >> +++ b/test/app/gh-3228-serializer-look-for-recursion.test.lua >=20 > >=20 >> diff --git a/test/swim/swim.result b/test/swim/swim.result >> index bfc83c143..539131677 100644 >> --- a/test/swim/swim.result >> +++ b/test/swim/swim.result >=20 > >=20 >> diff --git a/third_party/lua-cjson/lua_cjson.c = b/third_party/lua-cjson/lua_cjson.c >> index 38e999870..67a9762bc 100644 >> --- a/third_party/lua-cjson/lua_cjson.c >> +++ b/third_party/lua-cjson/lua_cjson.c >=20 > >=20 >> diff --git a/third_party/lua-yaml/lyaml.cc = b/third_party/lua-yaml/lyaml.cc >> index 9c3a4a646..72f269e6c 100644 >> --- a/third_party/lua-yaml/lyaml.cc >> +++ b/third_party/lua-yaml/lyaml.cc >> @@ -500,38 +500,23 @@ usage_error: >> static int dump_node(struct lua_yaml_dumper *dumper); >>=20 >> static yaml_char_t *get_yaml_anchor(struct lua_yaml_dumper *dumper) { >> - const char *s =3D ""; >> - lua_pushvalue(dumper->L, -1); >> - lua_rawget(dumper->L, dumper->anchortable_index); >> - if (!lua_toboolean(dumper->L, -1)) { >> - lua_pop(dumper->L, 1); >> - return NULL; >> - } >> - >> - if (lua_isboolean(dumper->L, -1)) { >> - /* this element is referenced more than once but has not been = named */ >> - char buf[32]; >> - snprintf(buf, sizeof(buf), "%u", dumper->anchor_number++); >> - lua_pop(dumper->L, 1); >> - lua_pushvalue(dumper->L, -1); >> - lua_pushstring(dumper->L, buf); >> - s =3D lua_tostring(dumper->L, -1); >> - lua_rawset(dumper->L, dumper->anchortable_index); >> - } else { >> - /* this is an aliased element */ >> + const char *anchor =3D NULL; >> + if (get_anchor(dumper->L, dumper->anchortable_index, = &dumper->anchor_number, >> + &anchor) =3D=3D 2) { >> yaml_event_t ev; >> - const char *str =3D lua_tostring(dumper->L, -1); >> - if (!yaml_alias_event_initialize(&ev, (yaml_char_t *) str) || >> + if (!yaml_alias_event_initialize(&ev, (yaml_char_t *) anchor) = || >> !yaml_emitter_emit(&dumper->emitter, &ev)) >> luaL_error(dumper->L, OOM_ERRMSG); >> - lua_pop(dumper->L, 1); >> + return (yaml_char_t *)""; >> } >> - return (yaml_char_t *)s; >> + return (yaml_char_t *) anchor; >=20 > Nit: s/(yaml_char_t *) anchor/(yaml_char_t *)anchor/, according to the > previous change. Fixed. >> } >>=20 >> -static int dump_table(struct lua_yaml_dumper *dumper, struct = luaL_field *field){ >> +static int dump_table(struct lua_yaml_dumper *dumper, struct = luaL_field *field, >> + yaml_char_t *anchor) { >> yaml_event_t ev; >> - yaml_char_t *anchor =3D get_yaml_anchor(dumper); >> + if (anchor =3D=3D NULL) >> + anchor =3D get_yaml_anchor(dumper); >>=20 >> if (anchor && !*anchor) return 1; >>=20 >> @@ -556,10 +541,12 @@ static int dump_table(struct lua_yaml_dumper = *dumper, struct luaL_field *field){ >> yaml_emitter_emit(&dumper->emitter, &ev) !=3D 0 ? 1 : 0; >> } >>=20 >> -static int dump_array(struct lua_yaml_dumper *dumper, struct = luaL_field *field){ >> +static int dump_array(struct lua_yaml_dumper *dumper, struct = luaL_field *field, >> + yaml_char_t *anchor) { >> unsigned i; >> yaml_event_t ev; >> - yaml_char_t *anchor =3D get_yaml_anchor(dumper); >> + if (anchor =3D=3D NULL) >> + anchor =3D get_yaml_anchor(dumper); >>=20 >> if (anchor && !*anchor) >> return 1; >> @@ -621,8 +608,18 @@ static int dump_node(struct lua_yaml_dumper = *dumper) >> bool unused; >> (void) unused; >>=20 >> + yaml_char_t *anchor =3D NULL; >> + >> + /* >> + * Keep anchor to print it if luaL_checkfield() push new value >> + * on stack after serialization. >> + */ >> + if (lua_istable(dumper->L, -1) =3D=3D 1) >> + anchor =3D get_yaml_anchor(dumper); >> + >> int top =3D lua_gettop(dumper->L); >> - luaL_checkfield(dumper->L, dumper->cfg, top, &field); >> + luaL_checkfield(dumper->L, dumper->cfg, = dumper->anchortable_index, top, >> + &field); >> switch(field.type) { >> case MP_UINT: >> snprintf(buf, sizeof(buf) - 1, "%" PRIu64, field.ival); >> @@ -647,9 +644,9 @@ static int dump_node(struct lua_yaml_dumper = *dumper) >> len =3D strlen(buf); >> break; >> case MP_ARRAY: >> - return dump_array(dumper, &field); >> + return dump_array(dumper, &field, anchor); >> case MP_MAP: >> - return dump_table(dumper, &field); >> + return dump_table(dumper, &field, anchor); >> case MP_STR: >> str =3D lua_tolstring(dumper->L, -1, &len); >> if (yaml_is_null(str, len) || yaml_is_bool(str, len, &unused) = || >> @@ -745,35 +742,6 @@ static int append_output(void *arg, unsigned = char *buf, size_t len) { >> return 1; >> } >>=20 >> -static void find_references(struct lua_yaml_dumper *dumper) { >=20 > >=20 >> -} >> - >> int >> lua_yaml_encode(lua_State *L, struct luaL_serializer *serializer, >> const char *tag_handle, const char *tag_prefix) >> @@ -819,7 +787,7 @@ lua_yaml_encode(lua_State *L, struct = luaL_serializer *serializer, >> dumper.anchortable_index =3D lua_gettop(L); >> dumper.anchor_number =3D 0; >> lua_pushvalue(L, 1); /* push copy of arg we're processing */ >> - find_references(&dumper); >> + find_references(L, dumper.anchortable_index, "before"); >=20 > Side note: looks like depends on > now. Doesn't look like a real third party :). > Feel free to ignore. >=20 >> dump_document(&dumper); >> if (dumper.error) >> goto error; >> --=20 >> 2.24.3 (Apple Git-128) >>=20 >=20 > [1]: = https://lists.tarantool.org/pipermail/tarantool-patches/2020-November/0207= 71.html > [2]: = https://lists.tarantool.org/pipermail/tarantool-patches/2020-July/018419.h= tml >=20 > --=20 > Best regards, > Sergey Kaplun commit 61c84258d32f121c0a5e75761ae3b347d0f41b5d Author: Roman Khabibov Date: Fri Feb 5 16:55:56 2021 +0300 serializer: serialize recursive structures =20 Fix bug with bus error during serializing of recursive structures and print aliases if node become referenced after serialization. =20 This patch adds map with serialized nodes and results. The map is used to mark referenced nodes which appears during serializing. =20 Closes #3228 =20 @TarantoolBot __serialize parameter If __serialize parameter is initialized as a function then this function should be pure. A pure function is a function which returns identical values for identical arguments (no variation with local static variables, non-local variables, mutable reference arguments or input streams). Otherwise, the behavior is undefined. diff --git a/changelogs/unreleased/serializer-bug.md = b/changelogs/unreleased/serializer-bug.md new file mode 100755 index 000000000..96f359d64 --- /dev/null +++ b/changelogs/unreleased/serializer-bug.md @@ -0,0 +1,4 @@ +## bugfix/core + +* Recursive structures appeared after __serialize call don't lead +* to segfault anymore in lua and yaml serializers (gh-5392). diff --git a/src/box/lua/call.c b/src/box/lua/call.c index 0315e720c..e4907a3f7 100644 --- a/src/box/lua/call.c +++ b/src/box/lua/call.c @@ -193,7 +193,7 @@ luamp_encode_call_16(lua_State *L, struct = luaL_serializer *cfg, */ for (int i =3D 1; i <=3D nrets; ++i) { struct luaL_field field; - if (luaL_tofield(L, cfg, NULL, i, &field) < 0) + if (luaL_tofield(L, cfg, 0, NULL, i, &field) < = 0) return luaT_error(L); struct tuple *tuple; if (field.type =3D=3D MP_EXT && @@ -222,7 +222,7 @@ luamp_encode_call_16(lua_State *L, struct = luaL_serializer *cfg, * Inspect the first result */ struct luaL_field root; - if (luaL_tofield(L, cfg, NULL, 1, &root) < 0) + if (luaL_tofield(L, cfg, 0, NULL, 1, &root) < 0) return luaT_error(L); struct tuple *tuple; if (root.type =3D=3D MP_EXT && (tuple =3D luaT_istuple(L, 1)) !=3D= NULL) { @@ -252,7 +252,7 @@ luamp_encode_call_16(lua_State *L, struct = luaL_serializer *cfg, for (uint32_t t =3D 1; t <=3D root.size; t++) { lua_rawgeti(L, 1, t); struct luaL_field field; - if (luaL_tofield(L, cfg, NULL, -1, &field) < 0) + if (luaL_tofield(L, cfg, 0, NULL, -1, &field) < 0) return luaT_error(L); if (field.type =3D=3D MP_EXT && (tuple =3D = luaT_istuple(L, -1))) { tuple_to_mpstream(tuple, stream); diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c index 926a0a61c..76d271f5c 100644 --- a/src/box/lua/execute.c +++ b/src/box/lua/execute.c @@ -328,7 +328,7 @@ lua_sql_bind_decode(struct lua_State *L, struct = sql_bind *bind, int idx, int i) bind->name =3D NULL; bind->name_len =3D 0; } - if (luaL_tofield(L, luaL_msgpack_default, NULL, -1, &field) < 0) + if (luaL_tofield(L, luaL_msgpack_default, 0, NULL, -1, &field) < = 0) return -1; switch (field.type) { case MP_UINT: diff --git a/src/box/lua/serialize_lua.c b/src/box/lua/serialize_lua.c index caa08a60f..17b434ba5 100644 --- a/src/box/lua/serialize_lua.c +++ b/src/box/lua/serialize_lua.c @@ -115,6 +115,7 @@ struct lua_dumper { /** Anchors for self references */ int anchortable_index; unsigned int anchor_number; + int serialized_objs_idx; =20 /** Error message buffer */ char err_msg[256]; @@ -215,7 +216,7 @@ trace_node(struct lua_dumper *d) struct luaL_field field; =20 memset(&field, 0, sizeof(field)); - luaL_checkfield(d->L, d->cfg, lua_gettop(d->L), &field); + luaL_checkfield(d->L, d->cfg, 0, lua_gettop(d->L), &field); =20 if (field.type < lengthof(mp_type_names)) { if (field.type =3D=3D MP_EXT) { @@ -234,7 +235,7 @@ trace_node(struct lua_dumper *d) =20 memset(&field, 0, sizeof(field)); =20 - luaL_checkfield(d->L, d->cfg, top, &field); + luaL_checkfield(d->L, d->cfg, 0, top, &field); say_info("serializer-trace: node :\tfield type %s (%d)", type_str, field.type); } @@ -339,41 +340,16 @@ emit_node(struct lua_dumper *d, struct node *nd, = int indent, static const char * get_lua_anchor(struct lua_dumper *d) { - const char *s =3D ""; - - lua_pushvalue(d->L, -1); - lua_rawget(d->L, d->anchortable_index); - if (!lua_toboolean(d->L, -1)) { - lua_pop(d->L, 1); - return NULL; + const char *anchor =3D NULL; + int code =3D get_anchor(d->L, d->anchortable_index, = &d->anchor_number, + &anchor); + if (code =3D=3D GET_ANCHOR_NAMED_NOT) { + trace_anchor(anchor, false); + } else if (code =3D=3D GET_ANCHOR_ALIASED) { + trace_anchor(anchor, true); + return ""; } - - if (lua_isboolean(d->L, -1)) { - /* - * This element is referenced more - * than once but has not been named. - */ - char buf[32]; - snprintf(buf, sizeof(buf), "%u", d->anchor_number++); - lua_pop(d->L, 1); - lua_pushvalue(d->L, -1); - lua_pushstring(d->L, buf); - s =3D lua_tostring(d->L, -1); - lua_rawset(d->L, d->anchortable_index); - trace_anchor(s, false); - } else { - /* - * An aliased element. - * - * FIXME: Need an example to use. - * - * const char *str =3D lua_tostring(d->L, -1); - */ - const char *str =3D lua_tostring(d->L, -1); - trace_anchor(str, true); - lua_pop(d->L, 1); - } - return s; + return anchor; } =20 static void @@ -783,7 +759,7 @@ dump_node(struct lua_dumper *d, struct node *nd, int = indent) return -1; =20 memset(field, 0, sizeof(*field)); - luaL_checkfield(d->L, d->cfg, lua_gettop(d->L), field); + luaL_checkfield(d->L, d->cfg, 0, lua_gettop(d->L), field); =20 switch (field->type) { case MP_NIL: @@ -881,49 +857,6 @@ dump_node(struct lua_dumper *d, struct node *nd, = int indent) return emit_node(d, nd, indent, str, len); } =20 -/** - * Find references to tables, we use it - * to find self references in tables. - */ -static void -find_references(struct lua_dumper *d) -{ - int newval; - - if (lua_type(d->L, -1) !=3D LUA_TTABLE) - return; - - /* Copy of a table for self refs */ - lua_pushvalue(d->L, -1); - lua_rawget(d->L, d->anchortable_index); - if (lua_isnil(d->L, -1)) - newval =3D 0; - else if (!lua_toboolean(d->L, -1)) - newval =3D 1; - else - newval =3D -1; - lua_pop(d->L, 1); - - if (newval !=3D -1) { - lua_pushvalue(d->L, -1); - lua_pushboolean(d->L, newval); - lua_rawset(d->L, d->anchortable_index); - } - - if (newval !=3D 0) - return; - - /* - * Other values and keys in the table - */ - lua_pushnil(d->L); - while (lua_next(d->L, -2) !=3D 0) { - find_references(d); - lua_pop(d->L, 1); - find_references(d); - } -} - /** * Dump recursively from the root node. */ @@ -935,7 +868,8 @@ dump_root(struct lua_dumper *d) }; int ret; =20 - luaL_checkfield(d->L, d->cfg, lua_gettop(d->L), &nd.field); + luaL_checkfield(d->L, d->cfg, d->serialized_objs_idx, + lua_gettop(d->L), &nd.field); =20 if (nd.field.type !=3D MP_ARRAY || nd.field.size !=3D 1) { d->err =3D EINVAL; @@ -982,9 +916,13 @@ lua_encode(lua_State *L, struct luaL_serializer = *serializer, dumper.anchortable_index =3D lua_gettop(L); dumper.anchor_number =3D 0; =20 + lua_newtable(L); + dumper.serialized_objs_idx =3D lua_gettop(L); + /* Push copy of arg we're processing */ lua_pushvalue(L, 1); - find_references(&dumper); + find_references_and_serialize(dumper.L, = dumper.anchortable_index, + dumper.serialized_objs_idx); =20 if (dump_root(&dumper) !=3D 0) goto out; diff --git a/src/box/lua/tuple.c b/src/box/lua/tuple.c index f7198a025..2779cf786 100644 --- a/src/box/lua/tuple.c +++ b/src/box/lua/tuple.c @@ -427,7 +427,7 @@ luamp_convert_key(struct lua_State *L, struct = luaL_serializer *cfg, return tuple_to_mpstream(tuple, stream); =20 struct luaL_field field; - if (luaL_tofield(L, cfg, NULL, index, &field) < 0) + if (luaL_tofield(L, cfg, 0, NULL, index, &field) < 0) luaT_error(L); if (field.type =3D=3D MP_ARRAY) { lua_pushvalue(L, index); diff --git a/src/box/sql/func.c b/src/box/sql/func.c index 9b6179f3a..2c7181f84 100644 --- a/src/box/sql/func.c +++ b/src/box/sql/func.c @@ -264,7 +264,7 @@ port_lua_get_vdbemem(struct port *base, uint32_t = *size) return NULL; for (int i =3D 0; i < argc; i++) { struct luaL_field field; - if (luaL_tofield(L, luaL_msgpack_default, + if (luaL_tofield(L, luaL_msgpack_default, 0, NULL, -1 - i, &field) < 0) { goto error; } diff --git a/src/lua/msgpack.c b/src/lua/msgpack.c index 1e74a6a3c..f9503a275 100644 --- a/src/lua/msgpack.c +++ b/src/lua/msgpack.c @@ -157,11 +157,11 @@ restart: /* used by MP_EXT of unidentified subtype = */ lua_pushnil(L); /* first key */ while (lua_next(L, top) !=3D 0) { lua_pushvalue(L, -2); /* push a copy of key to = top */ - if (luaL_tofield(L, cfg, opts, lua_gettop(L), = field) < 0) + if (luaL_tofield(L, cfg, 0, opts, lua_gettop(L), = field) < 0) return luaT_error(L); luamp_encode_r(L, cfg, opts, stream, field, = level + 1); lua_pop(L, 1); /* pop a copy of key */ - if (luaL_tofield(L, cfg, opts, lua_gettop(L), = field) < 0) + if (luaL_tofield(L, cfg, 0, opts, lua_gettop(L), = field) < 0) return luaT_error(L); luamp_encode_r(L, cfg, opts, stream, field, = level + 1); lua_pop(L, 1); /* pop value */ @@ -182,7 +182,7 @@ restart: /* used by MP_EXT of unidentified subtype = */ mpstream_encode_array(stream, size); for (uint32_t i =3D 0; i < size; i++) { lua_rawgeti(L, top, i + 1); - if (luaL_tofield(L, cfg, opts, top + 1, field) < = 0) + if (luaL_tofield(L, cfg, 0, opts, top + 1, = field) < 0) return luaT_error(L); luamp_encode_r(L, cfg, opts, stream, field, = level + 1); lua_pop(L, 1); @@ -205,7 +205,7 @@ restart: /* used by MP_EXT of unidentified subtype = */ if (type !=3D MP_EXT) return type; /* Value has been packed by = the trigger */ /* Try to convert value to serializable type */ - luaL_convertfield(L, cfg, top, field); + luaL_convertfield(L, cfg, 0, top, field); /* handled by luaL_convertfield */ assert(field->type !=3D MP_EXT); assert(lua_gettop(L) =3D=3D top); @@ -230,7 +230,7 @@ luamp_encode(struct lua_State *L, struct = luaL_serializer *cfg, } =20 struct luaL_field field; - if (luaL_tofield(L, cfg, opts, lua_gettop(L), &field) < 0) + if (luaL_tofield(L, cfg, 0, opts, lua_gettop(L), &field) < 0) return luaT_error(L); enum mp_type top_type =3D luamp_encode_r(L, cfg, opts, stream, = &field, 0); =20 diff --git a/src/lua/pickle.c b/src/lua/pickle.c index 65208b5b3..86f0f77e8 100644 --- a/src/lua/pickle.c +++ b/src/lua/pickle.c @@ -80,7 +80,7 @@ lbox_pack(struct lua_State *L) if (i > nargs) luaL_error(L, "pickle.pack: argument count does = not match " "the format"); - luaL_checkfield(L, luaL_msgpack_default, i, &field); + luaL_checkfield(L, luaL_msgpack_default, 0, i, &field); switch (*format) { case 'B': case 'b': diff --git a/src/lua/utils.c b/src/lua/utils.c index b5a6ca5b7..0580f4094 100644 --- a/src/lua/utils.c +++ b/src/lua/utils.c @@ -441,7 +441,8 @@ lua_gettable_wrapper(lua_State *L) =20 static void lua_field_inspect_ucdata(struct lua_State *L, struct luaL_serializer = *cfg, - int idx, struct luaL_field *field) + int serialized_objs_idx, int idx, + struct luaL_field *field) { if (!cfg->encode_load_metatables) return; @@ -463,59 +464,280 @@ lua_field_inspect_ucdata(struct lua_State *L, = struct luaL_serializer *cfg, lua_pcall(L, 1, 1, 0); /* replace obj with the unpacked value */ lua_replace(L, idx); - if (luaL_tofield(L, cfg, NULL, idx, field) < 0) + if (luaL_tofield(L, cfg, serialized_objs_idx, NULL, idx, + field) < 0) luaT_error(L); } /* else ignore lua_gettable exceptions */ lua_settop(L, top); /* remove temporary objects */ } =20 +int +get_anchor(struct lua_State *L, int anchortable_index, + unsigned int *anchor_number, const char **anchor) +{ + lua_pushvalue(L, -1); + lua_rawget(L, anchortable_index); + if (lua_toboolean(L, -1) =3D=3D 0) { + /* This element is not referenced. */ + lua_pop(L, 1); + *anchor =3D NULL; + return 0; + } + + if (lua_isboolean(L, -1) !=3D 0) { + /* + * This element is referenced more than once but + * has not been named. + */ + char buf[32]; + snprintf(buf, sizeof(buf), "%u", (*anchor_number)++); + lua_pop(L, 1); + /* Generate a string anchor and push to table. */ + lua_pushvalue(L, -1); + lua_pushstring(L, buf); + *anchor =3D lua_tostring(L, -1); + lua_rawset(L, anchortable_index); + return GET_ANCHOR_NAMED_NOT; + } else { + /* This is an aliased element. */ + *anchor =3D lua_tostring(L, -1); + lua_pop(L, 1); + return GET_ANCHOR_ALIASED; + } + return GET_ANCHOR_NO_REFS; +} + +void +find_references_and_serialize(struct lua_State *L, int = anchortable_index, + int serialized_objs_idx) +{ + int type =3D lua_type(L, -1); + if (type !=3D LUA_TTABLE) + return; + int node_idx =3D lua_gettop(L); + + /* + * Check if the node is already serialized i.e. the record + * is already in the map of serialized objects. + */ + lua_pushvalue(L, node_idx); + lua_rawget(L, serialized_objs_idx); + bool srlzd =3D lua_isnil(L, -1) =3D=3D 0; + /* + * This flag indicates that the object is being serialized + * in the current function call. + */ + bool serialization =3D false; + + if (!srlzd && luaL_getmetafield(L, node_idx, LUAL_SERIALIZE) !=3D = 0 && + lua_isfunction(L, -1) !=3D 0) { + /* Delete nil after lua_rawget() above. */ + lua_replace(L, -2); + srlzd =3D true; + serialization =3D true; + /* + * Copy object itself and call serialize function + * for it. + */ + lua_pushvalue(L, node_idx); + if (lua_pcall(L, 1, 1, 0) !=3D 0) { + diag_set(LuajitError, lua_tostring(L, -1)); + luaT_error(L); + } + /* + * Add the result of serialization to the + * serialized_objs map. Key is node (node_idx), + * value is the result of serialization (now on + * the top of stack). + */ + lua_pushvalue(L, node_idx); + lua_pushvalue(L, -2); + lua_rawset(L, serialized_objs_idx); + } + + if (srlzd) { + if (lua_type(L, -1) =3D=3D LUA_TTABLE) { + /* + * Now we will process the new node - the + * result of serialization. It is + * necessary to take it into account in + * the anchor table. + */ + node_idx =3D lua_gettop(L); + } else { + lua_pop(L, 1); + return; + } + } else { + if (lua_isnil(L, -1) =3D=3D 0) + /* + * Pop an extra field thrown out + * by luaL_getmetafield(), it was + * not a function. + */ + lua_pop(L, 1); + /* Delete nil after lua_rawget() above. */ + lua_pop(L, 1); + } + /* + * Take an entry from the anchor table about the current + * node. + */ + lua_pushvalue(L, node_idx); + lua_rawget(L, anchortable_index); + int newval =3D -1; + if (lua_isnil(L, -1) =3D=3D 1) + /* + * The node has not yet been encountered, it is + * bypassed for the first time. + */ + newval =3D 0; + else if (lua_toboolean(L, -1) =3D=3D 0) + /* The node has already met once. */ + newval =3D 1; + lua_pop(L, 1); + if (newval !=3D -1) { + lua_pushvalue(L, node_idx); + lua_pushboolean(L, newval); + lua_rawset(L, anchortable_index); + } + if (srlzd && !serialization) { + /* + * The node has already been serialized not in the + * current call, so there is no point in going + * further in depth and checking the leaves. Pop + * the processed sterilization result. + */ + lua_pop(L, 1); + return; + } + if (newval) + /* The node has already met twice or more, so + * there is no point in going further in depth and + * checking the leaves. + */ + return; + + /* Recursively process other table values. */ + lua_pushnil(L); + while (lua_next(L, node_idx) !=3D 0) { + /* Find references on value. */ + find_references_and_serialize(L, anchortable_index, + serialized_objs_idx); + lua_pop(L, 1); + /* Find references on key. */ + find_references_and_serialize(L, anchortable_index, + serialized_objs_idx); + } + if (serialization) + /* + * The loop above processed the nodes inside the + * serialization result. Pop it, so as not to + * break the integrity of the loop of the previous + * call in recursion. + */ + lua_pop(L, 1); + + return; +} + +enum try_serialize_ret_code +{ + + TRY_SERIALIZE_ERROR =3D -1, + TRY_SERIALIZE_RES_TABLE_NOT =3D 0, + TRY_SERIALIZE_RES_FROM_MAP, + TRY_SERIALIZE_DEFAULT_TABLE, +}; + /** * Call __serialize method of a table object by index - * if the former exists. + * if the former exists or pull result of serialization (if + * exists) from table with index @a serialized_objs_idx if it + * isn't 0. * * If __serialize does not exist then function does nothing - * and the function returns 1; + * and the function returns TRY_SERIALIZE_DEFAULT_TABLE; * * If __serialize exists, is a function (which doesn't * raise any error) then a result of serialization - * replaces old value by the index and the function returns 0; + * replaces old value by the index and the function returns + * TRY_SERIALIZE_RES_TABLE_NOT or TRY_SERIALIZE_DEFAULT_TABLE; * * If the serialization is a hint string (like 'array' or 'map'), * then field->type, field->size and field->compact - * are set if necessary and the function returns 0; + * are set if necessary and the function returns + * TRY_SERIALIZE_RES_TABLE_NOT; * - * Otherwise it is an error, set diag and the funciton returns -1; + * Otherwise it is an error, set diag and the funciton returns + * TRY_SERIALIZE_ERROR; * * Return values: - * -1 - error occurs, diag is set, the top of guest stack - * is undefined. - * 0 - __serialize field is available in the metatable, - * the result value is put in the origin slot, - * encoding is finished. - * 1 - __serialize field is not available in the metatable, - * proceed with default table encoding. + * TRY_SERIALIZE_ERROR - error occurs, diag is set, the top of + * guest stack is undefined. + * TRY_SERIALIZE_RES_TABLE_NOT - __serialize field is available in + * the metatable, the result value is put in the origin slot, + * encoding is finished. + * TRY_SERIALIZE_RES_FROM_MAP - __serialize field is available in + * the metatable, the result value is table from serialized + * objects map. + * TRY_SERIALIZE_DEFAULT_TABLE - __serialize field is not + * available in the metatable, proceed with default table + * encoding. */ + static int -lua_field_try_serialize(struct lua_State *L, struct luaL_serializer = *cfg, - int idx, struct luaL_field *field) +lua_field_try_serialize(struct lua_State *L, int serialized_objs_idx, + struct luaL_serializer *cfg, int idx, + struct luaL_field *field) { if (luaL_getmetafield(L, idx, LUAL_SERIALIZE) =3D=3D 0) - return 1; + return TRY_SERIALIZE_DEFAULT_TABLE; if (lua_isfunction(L, -1)) { - /* copy object itself */ - lua_pushvalue(L, idx); - if (lua_pcall(L, 1, 1, 0) !=3D 0) { - diag_set(LuajitError, lua_tostring(L, -1)); - return -1; + if (serialized_objs_idx !=3D 0) { + /* + * Pop the __serialize function for the + * current node. It was already called in + * find_references_and_serialize(). + */ + lua_pop(L, 1); + /* + * Get the result of serialization from + * the map. + */ + lua_pushvalue(L, idx); + lua_rawget(L, serialized_objs_idx); + assert(lua_isnil(L, -1) =3D=3D 0); + + /* + * Replace the serialized node with a new + * result, if it is a table. + */ + if (lua_type(L, -1) =3D=3D LUA_TTABLE) { + lua_replace(L, idx); + return TRY_SERIALIZE_RES_FROM_MAP; + } + } else { + /* + * Serializer don't use map with + * serialized objects. Copy object itself + * and call __serialize for it. + */ + lua_pushvalue(L, idx); + if (lua_pcall(L, 1, 1, 0) !=3D 0) { + diag_set(LuajitError, lua_tostring(L, = -1)); + return TRY_SERIALIZE_ERROR; + } } - if (luaL_tofield(L, cfg, NULL, -1, field) !=3D 0) - return -1; + if (luaL_tofield(L, cfg, serialized_objs_idx, NULL, -1, + field) !=3D 0) + return TRY_SERIALIZE_ERROR; lua_replace(L, idx); - return 0; + return TRY_SERIALIZE_RES_TABLE_NOT; } if (!lua_isstring(L, -1)) { diag_set(LuajitError, "invalid " LUAL_SERIALIZE " = value"); - return -1; + return TRY_SERIALIZE_ERROR; } const char *type =3D lua_tostring(L, -1); if (strcmp(type, "array") =3D=3D 0 || strcmp(type, "seq") =3D=3D = 0 || @@ -533,16 +755,17 @@ lua_field_try_serialize(struct lua_State *L, = struct luaL_serializer *cfg, field->compact =3D true; } else { diag_set(LuajitError, "invalid " LUAL_SERIALIZE " = value"); - return -1; + return TRY_SERIALIZE_ERROR; } /* Remove value set by luaL_getmetafield. */ lua_pop(L, 1); - return 0; + return TRY_SERIALIZE_RES_TABLE_NOT; } =20 static int -lua_field_inspect_table(struct lua_State *L, struct luaL_serializer = *cfg, - int idx, struct luaL_field *field) +lua_field_inspect_table(struct lua_State *L, int serialized_objs_idx, + struct luaL_serializer *cfg, int idx, + struct luaL_field *field) { assert(lua_type(L, idx) =3D=3D LUA_TTABLE); uint32_t size =3D 0; @@ -550,14 +773,20 @@ lua_field_inspect_table(struct lua_State *L, = struct luaL_serializer *cfg, =20 if (cfg->encode_load_metatables) { int top =3D lua_gettop(L); - int res =3D lua_field_try_serialize(L, cfg, idx, field); - if (res =3D=3D -1) + int res =3D lua_field_try_serialize(L, = serialized_objs_idx, cfg, + idx, field); + if (res =3D=3D TRY_SERIALIZE_ERROR) return -1; assert(lua_gettop(L) =3D=3D top); (void)top; - if (res =3D=3D 0) + if (res =3D=3D TRY_SERIALIZE_RES_TABLE_NOT || + (res =3D=3D TRY_SERIALIZE_RES_FROM_MAP && + lua_type(L, -1) !=3D LUA_TTABLE)) return 0; - /* Fallthrough with res =3D=3D 1 */ + /* + * Fallthrough with res TRY_SERIALIZE_RES_FROM_MAP + * or TRY_SERIALIZE_DEFAULT_TABLE. + */ } =20 field->type =3D MP_ARRAY; @@ -612,14 +841,14 @@ lua_field_tostring(struct lua_State *L, struct = luaL_serializer *cfg, int idx, lua_call(L, 1, 1); lua_replace(L, idx); lua_settop(L, top); - if (luaL_tofield(L, cfg, NULL, idx, field) < 0) + if (luaL_tofield(L, cfg, 0, NULL, idx, field) < 0) luaT_error(L); } =20 int luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg, - const struct serializer_opts *opts, int index, - struct luaL_field *field) + int serialized_objs_idx, const struct serializer_opts = *opts, + int index, struct luaL_field *field) { if (index < 0) index =3D lua_gettop(L) + index + 1; @@ -753,7 +982,8 @@ luaL_tofield(struct lua_State *L, struct = luaL_serializer *cfg, case LUA_TTABLE: { field->compact =3D false; - return lua_field_inspect_table(L, cfg, index, field); + return lua_field_inspect_table(L, serialized_objs_idx, = cfg, + index, field); } case LUA_TLIGHTUSERDATA: case LUA_TUSERDATA: @@ -773,8 +1003,8 @@ luaL_tofield(struct lua_State *L, struct = luaL_serializer *cfg, } =20 void -luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int = idx, - struct luaL_field *field) +luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, + int serialized_objs_idx, int idx, struct luaL_field = *field) { if (idx < 0) idx =3D lua_gettop(L) + idx + 1; @@ -789,9 +1019,12 @@ luaL_convertfield(struct lua_State *L, struct = luaL_serializer *cfg, int idx, */ GCcdata *cd =3D cdataV(L->base + idx - 1); if (cd->ctypeid > CTID_CTYPEID) - lua_field_inspect_ucdata(L, cfg, idx, = field); + lua_field_inspect_ucdata(L, cfg, + = serialized_objs_idx, idx, + field); } else if (type =3D=3D LUA_TUSERDATA) { - lua_field_inspect_ucdata(L, cfg, idx, field); + lua_field_inspect_ucdata(L, cfg, = serialized_objs_idx, idx, + field); } } =20 diff --git a/src/lua/utils.h b/src/lua/utils.h index 4a164868b..cb1025495 100644 --- a/src/lua/utils.h +++ b/src/lua/utils.h @@ -353,6 +353,44 @@ struct luaL_field { bool compact; /* a flag used by YAML serializer = */ }; =20 +/** + * Find references to tables to look up self references in tables + * and serialize nodes if __serialize is function. Collect + * referenced nodes in a table with stack index @a + * anchortable_index and serialized nodes with results in a table + * with stack index @a serialized_objs_idx. + * + * @param L Lua stack. + * @param anchortable_index Index of anchor table on stack with + pairs: "node address" <-> "false (met once) or true + (met more than once)". + * @param serialized_objs_idx Index of serialized objects map on + * stack with pairs: "node address" <-> "result address". + */ +void +find_references_and_serialize(struct lua_State *L, int = anchortable_index, + int serialized_objs_idx); + +enum get_anchor_ret_code +{ + GET_ANCHOR_NO_REFS =3D 0, + GET_ANCHOR_NAMED_NOT, + GET_ANCHOR_ALIASED, +}; + +/** + * Generate aliases and anchor numbers for self references. + * + * @param L Lua stack. + * @param anchortable_index Index of anchor table on stack. + * @param[out] anchor_number Ptr to the number anchors in a + structure being under dumping. + * @param[out] anchor Ptr to anchor string. + */ +int +get_anchor(struct lua_State *L, int anchortable_index, + unsigned int *anchor_number, const char **anchor); + /** * @brief Convert a value from the Lua stack to a lua_field structure. * This function is designed for use with Lua bindings and data @@ -387,6 +425,8 @@ struct luaL_field { * * @param L stack * @param cfg configuration + * @param serialized_objs_idx Index of table with serialized + objects map. * @param opts the Lua serializer additional options. * @param index stack index * @param field conversion result @@ -396,8 +436,8 @@ struct luaL_field { */ int luaL_tofield(struct lua_State *L, struct luaL_serializer *cfg, - const struct serializer_opts *opts, int index, - struct luaL_field *field); + int serialized_objs_idx, const struct serializer_opts = *opts, + int index, struct luaL_field *field); =20 /** * @brief Try to convert userdata/cdata values using defined conversion = logic. @@ -405,18 +445,22 @@ luaL_tofield(struct lua_State *L, struct = luaL_serializer *cfg, * * @param L stack * @param cfg configuration + * @param serialized_objs_idx Index of table with serialized + objects map. * @param idx stack index * @param field conversion result */ void -luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, int = idx, - struct luaL_field *field); +luaL_convertfield(struct lua_State *L, struct luaL_serializer *cfg, + int serialized_objs_idx, int idx, struct luaL_field = *field); =20 /** * @brief A wrapper for luaL_tofield() and luaL_convertfield() that * tries to convert value or raise an error. * @param L stack * @param cfg configuration + * @param serialized_objs_idx Stack index of table with serialized + objects map. * @param idx stack index * @param field conversion result * @sa lua_tofield() @@ -432,14 +476,14 @@ luaL_convertfield(struct lua_State *L, struct = luaL_serializer *cfg, int idx, * (tostring) -> (nil) -> exception */ static inline void -luaL_checkfield(struct lua_State *L, struct luaL_serializer *cfg, int = idx, - struct luaL_field *field) +luaL_checkfield(struct lua_State *L, struct luaL_serializer *cfg, + int serialized_objs_idx, int idx, struct luaL_field = *field) { - if (luaL_tofield(L, cfg, NULL, idx, field) < 0) + if (luaL_tofield(L, cfg, serialized_objs_idx, NULL, idx, field) = < 0) luaT_error(L); if (field->type !=3D MP_EXT || field->ext_type !=3D = MP_UNKNOWN_EXTENSION) return; - luaL_convertfield(L, cfg, idx, field); + luaL_convertfield(L, cfg, serialized_objs_idx, idx, field); } =20 void diff --git a/test/app/gh-3228-serializer-look-for-recursion.result = b/test/app/gh-3228-serializer-look-for-recursion.result new file mode 100644 index 000000000..cb8c73de0 --- /dev/null +++ b/test/app/gh-3228-serializer-look-for-recursion.result @@ -0,0 +1,211 @@ +-- test-run result file version 2 +test_run =3D require('test_run').new() + | --- + | ... + +-- +-- gh-3228: Check that recursive structures are serialized +-- properly. +-- +setmetatable({}, {__serialize =3D function(a) return a end}) + | --- + | - [] + | ... +setmetatable({}, {__serialize =3D function(a) return {a} end}) + | --- + | - &0 + | - *0 + | ... +setmetatable({}, {__serialize =3D function(a) return {a, a} end}) + | --- + | - &0 + | - *0 + | - *0 + | ... +setmetatable({}, {__serialize =3D function(a) return {a, a, a} end}) + | --- + | - &0 + | - *0 + | - *0 + | - *0 + | ... +setmetatable({}, {__serialize =3D function(a) return {{a, a}, a} end}) + | --- + | - &0 + | - - *0 + | - *0 + | - *0 + | ... +setmetatable({}, {__serialize =3D function(a) return {a, 1} end}) + | --- + | - &0 + | - *0 + | - 1 + | ... +setmetatable({}, {__serialize =3D function(a) return {{{{a}}}} end}) + | --- + | - &0 + | - - - - *0 + | ... +setmetatable({}, {__serialize =3D function(a) return {{{{1}}}} end}) + | --- + | - - - - - 1 + | ... + +b =3D {} + | --- + | ... +setmetatable({b}, {__serialize =3D function(a) return {a_1 =3D a, a_2 =3D= a, b_1 =3D b, b_2 =3D b} end}) + | --- + | - &0 + | b_2: &1 [] + | a_2: *0 + | a_1: *0 + | b_1: *1 + | ... +setmetatable({b}, {__serialize =3D function(a) return {a_1 =3D a, a_2 =3D= {a, b}, b =3D b} end}) + | --- + | - &0 + | b: &1 [] + | a_2: + | - *0 + | - *1 + | a_1: *0 + | ... + +a =3D {} + | --- + | ... +b =3D {a =3D a} + | --- + | ... +b[b] =3D b + | --- + | ... +a[b] =3D a + | --- + | ... +setmetatable(b, {__serialize =3D function(...) return {a} end}) + | --- + | - &0 + | - &1 + | *0: *1 + | ... +a + | --- + | - &0 + | ? - *0 + | : *0 + | ... + +a =3D {} + | --- + | ... +b =3D {} + | --- + | ... +a[b] =3D b + | --- + | ... +setmetatable(b, {__serialize =3D function(...) return print end}) + | --- + | - 'function: builtin#29' + | ... +a + | --- + | - 'function: builtin#29': 'function: builtin#29' + | ... + +a =3D {} + | --- + | ... +a[a] =3D a + | --- + | ... +recf =3D function(t) return setmetatable({}, {__serialize =3D recf}) = end + | --- + | ... +setmetatable(a, {__serialize =3D recf}) return a + | --- + | - [] + | ... + +-- +-- __serialize function is pure, i.e. always returns identical +-- value for identical argument. Otherwise, the behavior is +-- undefined. So that, we ignore the side effects and just use the +-- value after the first serialization. +-- +a =3D {} + | --- + | ... +b =3D {} + | --- + | ... +b[a] =3D a + | --- + | ... +show_a =3D true + | --- + | ... +test_run:cmd('setopt delimiter ";"') + | --- + | - true + | ... +serialize =3D function() + show_a =3D not show_a + if show_a then + return "a" + else + return "b" end +end; + | --- + | ... +test_run:cmd('setopt delimiter ""'); + | --- + | - true + | ... +setmetatable(a, {__serialize =3D serialize}) + | --- + | - b + | ... +b + | --- + | - a: a + | ... + +test_run:cmd('setopt delimiter ";"') + | --- + | - true + | ... +function example() + local a =3D {} + local b =3D {} + b[a] =3D a + local reta + local retb + local function swap_ser(o) + local newf + local f =3D getmetatable(o).__serialize + if f =3D=3D reta then + newf =3D retb + else + newf =3D reta + end + getmetatable(o).__serialize =3D newf + end + reta =3D function(o) swap_ser(o) return "a" end + retb =3D function(o) swap_ser(o) return "b" end + setmetatable(a, {__serialize =3D reta}) + return b +end; + | --- + | ... +test_run:cmd('setopt delimiter ""'); + | --- + | - true + | ... +example() + | --- + | - a: a + | ... diff --git a/test/app/gh-3228-serializer-look-for-recursion.test.lua = b/test/app/gh-3228-serializer-look-for-recursion.test.lua new file mode 100644 index 000000000..bfa046a16 --- /dev/null +++ b/test/app/gh-3228-serializer-look-for-recursion.test.lua @@ -0,0 +1,83 @@ +test_run =3D require('test_run').new() + +-- +-- gh-3228: Check that recursive structures are serialized +-- properly. +-- +setmetatable({}, {__serialize =3D function(a) return a end}) +setmetatable({}, {__serialize =3D function(a) return {a} end}) +setmetatable({}, {__serialize =3D function(a) return {a, a} end}) +setmetatable({}, {__serialize =3D function(a) return {a, a, a} end}) +setmetatable({}, {__serialize =3D function(a) return {{a, a}, a} end}) +setmetatable({}, {__serialize =3D function(a) return {a, 1} end}) +setmetatable({}, {__serialize =3D function(a) return {{{{a}}}} end}) +setmetatable({}, {__serialize =3D function(a) return {{{{1}}}} end}) + +b =3D {} +setmetatable({b}, {__serialize =3D function(a) return {a_1 =3D a, a_2 =3D= a, b_1 =3D b, b_2 =3D b} end}) +setmetatable({b}, {__serialize =3D function(a) return {a_1 =3D a, a_2 =3D= {a, b}, b =3D b} end}) + +a =3D {} +b =3D {a =3D a} +b[b] =3D b +a[b] =3D a +setmetatable(b, {__serialize =3D function(...) return {a} end}) +a + +a =3D {} +b =3D {} +a[b] =3D b +setmetatable(b, {__serialize =3D function(...) return print end}) +a + +a =3D {} +a[a] =3D a +recf =3D function(t) return setmetatable({}, {__serialize =3D recf}) = end +setmetatable(a, {__serialize =3D recf}) return a + +-- +-- __serialize function is pure, i.e. always returns identical +-- value for identical argument. Otherwise, the behavior is +-- undefined. So that, we ignore the side effects and just use the +-- value after the first serialization. +-- +a =3D {} +b =3D {} +b[a] =3D a +show_a =3D true +test_run:cmd('setopt delimiter ";"') +serialize =3D function() + show_a =3D not show_a + if show_a then + return "a" + else + return "b" end +end; +test_run:cmd('setopt delimiter ""'); +setmetatable(a, {__serialize =3D serialize}) +b + +test_run:cmd('setopt delimiter ";"') +function example() + local a =3D {} + local b =3D {} + b[a] =3D a + local reta + local retb + local function swap_ser(o) + local newf + local f =3D getmetatable(o).__serialize + if f =3D=3D reta then + newf =3D retb + else + newf =3D reta + end + getmetatable(o).__serialize =3D newf + end + reta =3D function(o) swap_ser(o) return "a" end + retb =3D function(o) swap_ser(o) return "b" end + setmetatable(a, {__serialize =3D reta}) + return b +end; +test_run:cmd('setopt delimiter ""'); +example() diff --git a/test/swim/swim.result b/test/swim/swim.result index bfc83c143..539131677 100644 --- a/test/swim/swim.result +++ b/test/swim/swim.result @@ -1322,16 +1322,13 @@ m_list incarnation: cdata {generation =3D 0ULL, version =3D 1ULL} uuid: 00000000-0000-1000-8000-000000000002 payload_size: 0 - - uri: 127.0.0.1: - status: alive - incarnation: cdata {generation =3D 0ULL, version =3D 2ULL} - uuid: 00000000-0000-1000-8000-000000000001 - payload_size: 8 - - uri: 127.0.0.1: + - &0 + uri: 127.0.0.1: status: alive incarnation: cdata {generation =3D 0ULL, version =3D 2ULL} uuid: 00000000-0000-1000-8000-000000000001 payload_size: 8 + - *0 ... e_list --- @@ -1374,16 +1371,13 @@ fiber.sleep(0) -- Two events - status update to 'left', and 'drop'. m_list --- -- - uri: 127.0.0.1: - status: left - incarnation: cdata {generation =3D 0ULL, version =3D 1ULL} - uuid: 00000000-0000-1000-8000-000000000002 - payload_size: 0 - - uri: 127.0.0.1: +- - &0 + uri: 127.0.0.1: status: left incarnation: cdata {generation =3D 0ULL, version =3D 1ULL} uuid: 00000000-0000-1000-8000-000000000002 payload_size: 0 + - *0 ... e_list --- diff --git a/third_party/lua-cjson/lua_cjson.c = b/third_party/lua-cjson/lua_cjson.c index 38e999870..67a9762bc 100644 --- a/third_party/lua-cjson/lua_cjson.c +++ b/third_party/lua-cjson/lua_cjson.c @@ -354,7 +354,7 @@ static void json_append_object(lua_State *l, struct = luaL_serializer *cfg, comma =3D 1; =20 struct luaL_field field; - luaL_checkfield(l, cfg, -2, &field); + luaL_checkfield(l, cfg, 0, -2, &field); if (field.type =3D=3D MP_UINT) { strbuf_append_char(json, '"'); json_append_uint(cfg, json, field.ival); @@ -384,7 +384,7 @@ static void json_append_data(lua_State *l, struct = luaL_serializer *cfg, int current_depth, strbuf_t *json) { struct luaL_field field; - luaL_checkfield(l, cfg, -1, &field); + luaL_checkfield(l, cfg, 0, -1, &field); switch (field.type) { case MP_UINT: return json_append_uint(cfg, json, field.ival); diff --git a/third_party/lua-yaml/lyaml.cc = b/third_party/lua-yaml/lyaml.cc index 9c3a4a646..249603ba3 100644 --- a/third_party/lua-yaml/lyaml.cc +++ b/third_party/lua-yaml/lyaml.cc @@ -79,6 +79,7 @@ struct lua_yaml_dumper { lua_State *L; struct luaL_serializer *cfg; int anchortable_index; + int serialized_objs_idx; unsigned int anchor_number; yaml_emitter_t emitter; char error; @@ -500,33 +501,16 @@ usage_error: static int dump_node(struct lua_yaml_dumper *dumper); =20 static yaml_char_t *get_yaml_anchor(struct lua_yaml_dumper *dumper) { - const char *s =3D ""; - lua_pushvalue(dumper->L, -1); - lua_rawget(dumper->L, dumper->anchortable_index); - if (!lua_toboolean(dumper->L, -1)) { - lua_pop(dumper->L, 1); - return NULL; - } - - if (lua_isboolean(dumper->L, -1)) { - /* this element is referenced more than once but has not been = named */ - char buf[32]; - snprintf(buf, sizeof(buf), "%u", dumper->anchor_number++); - lua_pop(dumper->L, 1); - lua_pushvalue(dumper->L, -1); - lua_pushstring(dumper->L, buf); - s =3D lua_tostring(dumper->L, -1); - lua_rawset(dumper->L, dumper->anchortable_index); - } else { - /* this is an aliased element */ + const char *anchor =3D NULL; + if (get_anchor(dumper->L, dumper->anchortable_index, = &dumper->anchor_number, + &anchor) =3D=3D GET_ANCHOR_ALIASED) { yaml_event_t ev; - const char *str =3D lua_tostring(dumper->L, -1); - if (!yaml_alias_event_initialize(&ev, (yaml_char_t *) str) || + if (!yaml_alias_event_initialize(&ev, (yaml_char_t *) anchor) || !yaml_emitter_emit(&dumper->emitter, &ev)) luaL_error(dumper->L, OOM_ERRMSG); - lua_pop(dumper->L, 1); + return (yaml_char_t *)""; } - return (yaml_char_t *)s; + return (yaml_char_t *)anchor; } =20 static int dump_table(struct lua_yaml_dumper *dumper, struct luaL_field = *field){ @@ -622,7 +606,8 @@ static int dump_node(struct lua_yaml_dumper *dumper) (void) unused; =20 int top =3D lua_gettop(dumper->L); - luaL_checkfield(dumper->L, dumper->cfg, top, &field); + luaL_checkfield(dumper->L, dumper->cfg, dumper->serialized_objs_idx, + top, &field); switch(field.type) { case MP_UINT: snprintf(buf, sizeof(buf) - 1, "%" PRIu64, field.ival); @@ -745,35 +730,6 @@ static int append_output(void *arg, unsigned char = *buf, size_t len) { return 1; } =20 -static void find_references(struct lua_yaml_dumper *dumper) { - int newval =3D -1, type =3D lua_type(dumper->L, -1); - if (type !=3D LUA_TTABLE) - return; - - lua_pushvalue(dumper->L, -1); /* push copy of table */ - lua_rawget(dumper->L, dumper->anchortable_index); - if (lua_isnil(dumper->L, -1)) - newval =3D 0; - else if (!lua_toboolean(dumper->L, -1)) - newval =3D 1; - lua_pop(dumper->L, 1); - if (newval !=3D -1) { - lua_pushvalue(dumper->L, -1); - lua_pushboolean(dumper->L, newval); - lua_rawset(dumper->L, dumper->anchortable_index); - } - if (newval) - return; - - /* recursively process other table values */ - lua_pushnil(dumper->L); - while (lua_next(dumper->L, -2) !=3D 0) { - find_references(dumper); /* find references on value */ - lua_pop(dumper->L, 1); - find_references(dumper); /* find references on key */ - } -} - int lua_yaml_encode(lua_State *L, struct luaL_serializer *serializer, const char *tag_handle, const char *tag_prefix) @@ -817,9 +773,12 @@ lua_yaml_encode(lua_State *L, struct = luaL_serializer *serializer, =20 lua_newtable(L); dumper.anchortable_index =3D lua_gettop(L); + lua_newtable(L); + dumper.serialized_objs_idx =3D lua_gettop(L); dumper.anchor_number =3D 0; lua_pushvalue(L, 1); /* push copy of arg we're processing */ - find_references(&dumper); + find_references_and_serialize(dumper.L, dumper.anchortable_index, + dumper.serialized_objs_idx); dump_document(&dumper); if (dumper.error) goto error;