[Tarantool-patches] [PATCH] serializer: check for recursive serialization

[Roman Khabibov <roman.habibov@tarantool.org>]

Print error if object after serialization is the same.

Closes #3228
---

Branch: https://github.com/tarantool/tarantool/tree/romanhabibov/serialize-check
Issue: https://github.com/tarantool/tarantool/issues/3228

@ChangeLog:
* Fix bug with bus error when __serialize function generates infinite recursion (gh-3228).

 src/lua/utils.c                               |  5 +++++
 ...-3228-serializer-look-for-recursion.result | 19 +++++++++++++++++++
 ...228-serializer-look-for-recursion.test.lua |  8 ++++++++
 3 files changed, 32 insertions(+)
 create mode 100644 test/app/gh-3228-serializer-look-for-recursion.result
 create mode 100644 test/app/gh-3228-serializer-look-for-recursion.test.lua

diff --git a/src/lua/utils.c b/src/lua/utils.c
index 23fbdd4ad..d12f3675a 100644
--- a/src/lua/utils.c
+++ b/src/lua/utils.c
@@ -508,6 +508,11 @@ lua_field_try_serialize(struct lua_State *L, struct luaL_serializer *cfg,
 			diag_set(LuajitError, lua_tostring(L, -1));
 			return -1;
 		}
+		if (lua_rawequal(L, -2, -1) == 1) {
+			diag_set(LuajitError, "Bad __serialize function. It "
+				 "can't return the same value.");
+			return -1;
+		}
 		if (luaL_tofield(L, cfg, NULL, -1, field) != 0)
 			return -1;
 		lua_replace(L, idx);
diff --git a/test/app/gh-3228-serializer-look-for-recursion.result b/test/app/gh-3228-serializer-look-for-recursion.result
new file mode 100644
index 000000000..cd86ab06a
--- /dev/null
+++ b/test/app/gh-3228-serializer-look-for-recursion.result
@@ -0,0 +1,19 @@
+-- test-run result file version 2
+test_run = require('test_run').new()
+ | ---
+ | ...
+
+--
+-- gh-3228: Check the error message in the case of a __serialize
+-- function generating infinite recursion.
+--
+setmetatable({}, {__serialize = function(a) return a end})
+ | ---
+ | - error: 'console: an exception occurred when formatting the output: Bad __serialize
+ |     function. It can''t return the same value.'
+ | ...
+setmetatable({}, {__serialize = function(a, b, c) return a, b, c end})
+ | ---
+ | - error: 'console: an exception occurred when formatting the output: Bad __serialize
+ |     function. It can''t return the same value.'
+ | ...
diff --git a/test/app/gh-3228-serializer-look-for-recursion.test.lua b/test/app/gh-3228-serializer-look-for-recursion.test.lua
new file mode 100644
index 000000000..d3c76ef0c
--- /dev/null
+++ b/test/app/gh-3228-serializer-look-for-recursion.test.lua
@@ -0,0 +1,8 @@
+test_run = require('test_run').new()
+
+--
+-- gh-3228: Check the error message in the case of a __serialize
+-- function generating infinite recursion.
+--
+setmetatable({}, {__serialize = function(a) return a end})
+setmetatable({}, {__serialize = function(a, b, c) return a, b, c end})
-- 
2.24.3 (Apple Git-128)