Re: [Tarantool-patches] [PATCH v1 1/1] sql: do not reset region on select

[Mergen Imeev <imeevma@tarantool.org>]

Hi! Thank you for the review. Sorry for such late reply. My answers and
new patch below.

On Wed, Nov 04, 2020 at 11:14:38PM +0100, Vladislav Shpilevoy wrote:
> Hi! Thanks for the patch!
> 
> See 2 comments below.
> 
> On 02.11.2020 10:49, imeevma@tarantool.org wrote:
> > Prior to this patch, region on fiber was reset during select(), get(),
> > count(), max(), or min(). This would result in an error if one of these
> > operations was used in a user-defined function in SQL. After this patch,
> > these functions truncate region instead of resetting it.
> > 
> > Closes #5427
> > ---
> > https://github.com/tarantool/tarantool/issues/5427
> > https://github.com/tarantool/tarantool/tree/imeevma/gh-5427-lua-func-changes-result
> > 
> > @ChangeLog
> >  - Region truncated instead of resetting on select() (gh-5427).
> 
> 1. I suggest to make the message more understandable for users. They have
> no idea what a region is. For instance, consider
> 
> 	Memory corruption when SQL called Lua functions with box calls inside
> 
Thanks, added.

> >  src/box/box.cc                                |   5 +-
> >  src/box/index.cc                              |  20 +++
> >  src/box/txn.h                                 |   6 +-
> >  .../gh-5427-lua-func-changes-result.result    | 153 ++++++++++++++++++
> >  .../gh-5427-lua-func-changes-result.test.lua  |  86 ++++++++++
> >  5 files changed, 264 insertions(+), 6 deletions(-)
> >  create mode 100644 test/sql/gh-5427-lua-func-changes-result.result
> >  create mode 100644 test/sql/gh-5427-lua-func-changes-result.test.lua
> > 
> > diff --git a/src/box/box.cc b/src/box/box.cc
> > index 18568df3b..473ef52ad 100644
> > --- a/src/box/box.cc
> > +++ b/src/box/box.cc
> > @@ -1388,7 +1388,8 @@ box_select(uint32_t space_id, uint32_t index_id,
> >  	   struct port *port)
> >  {
> >  	(void)key_end;
> > -
> > +	struct region *region = &fiber()->gc;
> > +	size_t used = region_used(region);
> >  	rmean_collect(rmean_box, IPROTO_SELECT, 1);
> 
> 2. I must say I couldn't come up with a better idea so far. I remember I
> also proposed to start a transaction for each VDBE statement, but not sure
> it is a right thing to do.
> 
> It means, we probably should go for this one, but a bit polished.
> 
> In the current solution I don't like, that you basically just inlined
> txn_commit_ro_stmt(), but instead I would rather want you to patch it to
> make it work.
> 
> So for example txn_begin_ro_stmt inside remembers the current region size,
> and returns it via an out parameter struct txn_ro_savepoint *svp. Which
> you create on the stack.
> 
> Then, after you are done with the statement, you call txn_commit_ro_stmt,
> which takes const struct txn_ro_savepoint *svp, and does the region_truncate
> inside.
Thanks, fixed.



From adc5b1096409b941daae0b91b8b1d7b0118fca05 Mon Sep 17 00:00:00 2001
Message-Id: <adc5b1096409b941daae0b91b8b1d7b0118fca05.1608096646.git.imeevma@gmail.com>
From: Mergen Imeev <imeevma@gmail.com>
Date: Thu, 22 Oct 2020 14:21:28 +0300
Subject: [PATCH v1 1/1] sql: do not reset region on select

Prior to this patch, region on fiber was reset during select(), get(),
count(), max(), or min(). This would result in an error if one of these
operations was used in a user-defined function in SQL. After this patch,
these functions truncate region instead of resetting it.

Closes #5427
---
https://github.com/tarantool/tarantool/issues/5427
https://github.com/tarantool/tarantool/tree/imeevma/gh-5427-lua-func-changes-result

@ChangeLog
 - Fixed memory corruption when SQL called Lua functions with box calls inside (gh-5427).

 src/box/box.cc                                |   5 +-
 src/box/index.cc                              |  25 +--
 src/box/sql.c                                 |   5 +-
 src/box/txn.h                                 |  16 +-
 .../gh-5427-lua-func-changes-result.result    | 153 ++++++++++++++++++
 .../gh-5427-lua-func-changes-result.test.lua  |  86 ++++++++++
 6 files changed, 273 insertions(+), 17 deletions(-)
 create mode 100644 test/sql/gh-5427-lua-func-changes-result.result
 create mode 100644 test/sql/gh-5427-lua-func-changes-result.test.lua

diff --git a/src/box/box.cc b/src/box/box.cc
index a8bc3471d..7470f882c 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -1394,7 +1394,8 @@ box_select(uint32_t space_id, uint32_t index_id,
 	});
 
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 
 	struct iterator *it = index_create_iterator(index, type,
@@ -1428,7 +1429,7 @@ box_select(uint32_t space_id, uint32_t index_id,
 		txn_rollback_stmt(txn);
 		return -1;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	return 0;
 }
 
diff --git a/src/box/index.cc b/src/box/index.cc
index c2fc00867..179cc190f 100644
--- a/src/box/index.cc
+++ b/src/box/index.cc
@@ -239,13 +239,14 @@ box_index_get(uint32_t space_id, uint32_t index_id, const char *key,
 		return -1;
 	/* Start transaction in the engine. */
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 	if (index_get(index, key, part_count, result) != 0) {
 		txn_rollback_stmt(txn);
 		return -1;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	/* Count statistics. */
 	rmean_collect(rmean_box, IPROTO_SELECT, 1);
 	if (*result != NULL)
@@ -273,13 +274,14 @@ box_index_min(uint32_t space_id, uint32_t index_id, const char *key,
 		return -1;
 	/* Start transaction in the engine. */
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 	if (index_min(index, key, part_count, result) != 0) {
 		txn_rollback_stmt(txn);
 		return -1;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	if (*result != NULL)
 		tuple_bless(*result);
 	return 0;
@@ -305,13 +307,14 @@ box_index_max(uint32_t space_id, uint32_t index_id, const char *key,
 		return -1;
 	/* Start transaction in the engine. */
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 	if (index_max(index, key, part_count, result) != 0) {
 		txn_rollback_stmt(txn);
 		return -1;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	if (*result != NULL)
 		tuple_bless(*result);
 	return 0;
@@ -338,14 +341,15 @@ box_index_count(uint32_t space_id, uint32_t index_id, int type,
 		return -1;
 	/* Start transaction in the engine. */
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 	ssize_t count = index_count(index, itype, key, part_count);
 	if (count < 0) {
 		txn_rollback_stmt(txn);
 		return -1;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	return count;
 }
 
@@ -374,7 +378,8 @@ box_index_iterator(uint32_t space_id, uint32_t index_id, int type,
 	if (key_validate(index->def, itype, key, part_count))
 		return NULL;
 	struct txn *txn;
-	if (txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return NULL;
 	struct iterator *it = index_create_iterator(index, itype,
 						    key, part_count);
@@ -382,7 +387,7 @@ box_index_iterator(uint32_t space_id, uint32_t index_id, int type,
 		txn_rollback_stmt(txn);
 		return NULL;
 	}
-	txn_commit_ro_stmt(txn);
+	txn_commit_ro_stmt(txn, &svp);
 	rmean_collect(rmean_box, IPROTO_SELECT, 1);
 	return it;
 }
diff --git a/src/box/sql.c b/src/box/sql.c
index c55d0f4e5..3d968e56a 100644
--- a/src/box/sql.c
+++ b/src/box/sql.c
@@ -901,7 +901,8 @@ cursor_seek(BtCursor *pCur, int *pRes)
 
 	struct space *space = pCur->space;
 	struct txn *txn = NULL;
-	if (space->def->id != 0 && txn_begin_ro_stmt(space, &txn) != 0)
+	struct txn_ro_savepoint svp;
+	if (space->def->id != 0 && txn_begin_ro_stmt(space, &txn, &svp) != 0)
 		return -1;
 	struct iterator *it =
 		index_create_iterator(pCur->index, pCur->iter_type, key,
@@ -913,7 +914,7 @@ cursor_seek(BtCursor *pCur, int *pRes)
 		return -1;
 	}
 	if (txn != NULL)
-		txn_commit_ro_stmt(txn);
+		txn_commit_ro_stmt(txn, &svp);
 	pCur->iter = it;
 	pCur->eState = CURSOR_VALID;
 
diff --git a/src/box/txn.h b/src/box/txn.h
index dfa7c0ee4..1a5ec975e 100644
--- a/src/box/txn.h
+++ b/src/box/txn.h
@@ -258,6 +258,13 @@ struct txn_savepoint {
 	char name[1];
 };
 
+struct txn_ro_savepoint {
+	/** Region used during this transaction. */
+	struct region *region;
+	/** Savepoint for region. */
+	size_t region_used;
+};
+
 extern double too_long_threshold;
 
 /**
@@ -557,24 +564,27 @@ txn_begin_in_engine(struct engine *engine, struct txn *txn);
  * select.
  */
 static inline int
-txn_begin_ro_stmt(struct space *space, struct txn **txn)
+txn_begin_ro_stmt(struct space *space, struct txn **txn,
+		  struct txn_ro_savepoint *svp)
 {
 	*txn = in_txn();
 	if (*txn != NULL) {
 		struct engine *engine = space->engine;
 		return txn_begin_in_engine(engine, *txn);
 	}
+	svp->region = &fiber()->gc;
+	svp->region_used = region_used(svp->region);
 	return 0;
 }
 
 static inline void
-txn_commit_ro_stmt(struct txn *txn)
+txn_commit_ro_stmt(struct txn *txn, struct txn_ro_savepoint *svp)
 {
 	assert(txn == in_txn());
 	if (txn) {
 		/* nothing to do */
 	} else {
-		fiber_gc();
+		region_truncate(svp->region, svp->region_used);
 	}
 }
 
diff --git a/test/sql/gh-5427-lua-func-changes-result.result b/test/sql/gh-5427-lua-func-changes-result.result
new file mode 100644
index 000000000..f2e777a0e
--- /dev/null
+++ b/test/sql/gh-5427-lua-func-changes-result.result
@@ -0,0 +1,153 @@
+-- test-run result file version 2
+test_run = require('test_run').new()
+ | ---
+ | ...
+engine = test_run:get_cfg('engine')
+ | ---
+ | ...
+_ = box.space._session_settings:update('sql_default_engine', {{'=', 2, engine}})
+ | ---
+ | ...
+
+box.execute([[CREATE TABLE t (b STRING PRIMARY KEY);]])
+ | ---
+ | - row_count: 1
+ | ...
+box.execute([[INSERT INTO t VALUES ('aaaaaaaaaaaa');]])
+ | ---
+ | - row_count: 1
+ | ...
+test_run:cmd("setopt delimiter ';'");
+ | ---
+ | - true
+ | ...
+box.schema.func.create('CORRUPT_SELECT', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:select()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+ | ---
+ | ...
+box.schema.func.create('CORRUPT_GET', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:get('aaaaaaaaaaaa')
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+ | ---
+ | ...
+box.schema.func.create('CORRUPT_COUNT', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:count()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+ | ---
+ | ...
+box.schema.func.create('CORRUPT_MAX', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T.index[0]:max()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+ | ---
+ | ...
+box.schema.func.create('CORRUPT_MIN', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T.index[0]:min()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+ | ---
+ | ...
+test_run:cmd("setopt delimiter ''");
+ | ---
+ | - true
+ | ...
+
+box.execute([[select CORRUPT_SELECT(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+ | ---
+ | - metadata:
+ |   - name: COLUMN_1
+ |     type: integer
+ |   rows:
+ |   - [1]
+ | ...
+box.execute([[select CORRUPT_GET(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+ | ---
+ | - metadata:
+ |   - name: COLUMN_1
+ |     type: integer
+ |   rows:
+ |   - [1]
+ | ...
+box.execute([[select CORRUPT_COUNT(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+ | ---
+ | - metadata:
+ |   - name: COLUMN_1
+ |     type: integer
+ |   rows:
+ |   - [1]
+ | ...
+box.execute([[select CORRUPT_MAX(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+ | ---
+ | - metadata:
+ |   - name: COLUMN_1
+ |     type: integer
+ |   rows:
+ |   - [1]
+ | ...
+box.execute([[select CORRUPT_MIN(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+ | ---
+ | - metadata:
+ |   - name: COLUMN_1
+ |     type: integer
+ |   rows:
+ |   - [1]
+ | ...
+
+box.execute([[DROP TABLE t;]])
+ | ---
+ | - row_count: 1
+ | ...
diff --git a/test/sql/gh-5427-lua-func-changes-result.test.lua b/test/sql/gh-5427-lua-func-changes-result.test.lua
new file mode 100644
index 000000000..f2699dbe7
--- /dev/null
+++ b/test/sql/gh-5427-lua-func-changes-result.test.lua
@@ -0,0 +1,86 @@
+test_run = require('test_run').new()
+engine = test_run:get_cfg('engine')
+_ = box.space._session_settings:update('sql_default_engine', {{'=', 2, engine}})
+
+box.execute([[CREATE TABLE t (b STRING PRIMARY KEY);]])
+box.execute([[INSERT INTO t VALUES ('aaaaaaaaaaaa');]])
+test_run:cmd("setopt delimiter ';'");
+box.schema.func.create('CORRUPT_SELECT', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:select()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+box.schema.func.create('CORRUPT_GET', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:get('aaaaaaaaaaaa')
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+box.schema.func.create('CORRUPT_COUNT', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T:count()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+box.schema.func.create('CORRUPT_MAX', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T.index[0]:max()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+box.schema.func.create('CORRUPT_MIN', {
+    language = 'LUA',
+    returns = 'integer',
+    body = [[
+        function(x)
+            box.space.T.index[0]:min()
+            return 1
+        end]],
+    is_sandboxed = false,
+    param_list = { "string" },
+    exports = { 'LUA', 'SQL' },
+    is_deterministic = false,
+    if_not_exists = true
+});
+test_run:cmd("setopt delimiter ''");
+
+box.execute([[select CORRUPT_SELECT(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+box.execute([[select CORRUPT_GET(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+box.execute([[select CORRUPT_COUNT(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+box.execute([[select CORRUPT_MAX(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+box.execute([[select CORRUPT_MIN(t.b) from t where t.b = ? and t.b <= ? order by t.b;]], {"aaaaaaaaaaaa", "aaaaaaaaaaaa"})
+
+box.execute([[DROP TABLE t;]])
-- 
2.25.1