[Tarantool-patches] [PATCH v3 00/20] sql: prepared statements

[Tarantool-patches] [PATCH v3 00/20] sql: prepared statements

[Nikita Pettik]

Branch: https://github.com/tarantool/tarantool/tree/np/gh-2592-prepared-statements-v3
Issue: https://github.com/tarantool/tarantool/issues/2592

V1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012274.html
V2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012496.html

Changes in V3 (requested by server team):

 - Now there's no eviction policy, so statements reside in 'cache'
   until explicit deallocation via 'unprepare' call or session's
   disconect;
 - instead of string ids, now we use numeric ids which correspond
   to value of hash function applied to the string containing original
   SQL request;
 - in accordance with previous point, 'unprepare' support has been returned;
 - since there's no eviction policy, disconnect event may turn out to
   be expensive (in terms of deallocating all related to the session
   prepared statements). To remove possible spikes in workload, we
   maintain GC queue and reference counters for prepared statements.
   When all sessions (previously refed statement) unref it, statement
   gets into GC queue. In case of prepared statements holder is out
   of memory, GC process is launched: all statements in queue are
   deallocated;
 - to track available in scope of session prepared statements, we
   also maintain session-local map containing statement's IDs
   allocated in this session.

Nikita Pettik (20):
  sql: remove sql_prepare_v2()
  sql: refactor sql_prepare() and sqlPrepare()
  sql: move sql_prepare() declaration to box/execute.h
  sql: rename sqlPrepare() to sql_stmt_compile()
  sql: rename sql_finalize() to sql_stmt_finalize()
  sql: rename sql_reset() to sql_stmt_reset()
  sql: move sql_stmt_finalize() to execute.h
  port: increase padding of struct port
  port: add result set format and request type to port_sql
  sql: resurrect sql_bind_parameter_count() function
  sql: resurrect sql_bind_parameter_name()
  sql: add sql_stmt_schema_version()
  sql: introduce sql_stmt_sizeof() function
  box: increment schema_version on ddl operations
  sql: introduce sql_stmt_query_str() method
  sql: move sql_stmt_busy() declaration to box/execute.h
  sql: introduce holder for prepared statemets
  box: introduce prepared statements
  netbox: introduce prepared statements
  sql: add cache statistics to box.info

 src/box/CMakeLists.txt          |   1 +
 src/box/alter.cc                |   3 +
 src/box/bind.c                  |   2 +-
 src/box/box.cc                  |  26 ++
 src/box/box.h                   |   3 +
 src/box/ck_constraint.c         |   7 +-
 src/box/errcode.h               |   2 +
 src/box/execute.c               | 234 ++++++++++++-
 src/box/execute.h               |  60 ++++
 src/box/iproto.cc               |  68 +++-
 src/box/iproto_constants.c      |   7 +-
 src/box/iproto_constants.h      |   5 +
 src/box/lua/cfg.cc              |   9 +
 src/box/lua/execute.c           | 235 ++++++++++++-
 src/box/lua/execute.h           |   2 +-
 src/box/lua/info.c              |  25 ++
 src/box/lua/init.c              |   2 +-
 src/box/lua/load_cfg.lua        |   3 +
 src/box/lua/net_box.c           |  98 +++++-
 src/box/lua/net_box.lua         |  27 ++
 src/box/session.cc              |  35 ++
 src/box/session.h               |  17 +
 src/box/sql.c                   |   3 +
 src/box/sql/analyze.c           |  23 +-
 src/box/sql/legacy.c            |   3 +-
 src/box/sql/prepare.c           |  56 +--
 src/box/sql/sqlInt.h            |  51 +--
 src/box/sql/vdbe.c              |   4 +-
 src/box/sql/vdbe.h              |   2 +-
 src/box/sql/vdbeInt.h           |   1 -
 src/box/sql/vdbeapi.c           | 113 ++++--
 src/box/sql/vdbeaux.c           |   6 +-
 src/box/sql_stmt_cache.c        | 305 +++++++++++++++++
 src/box/sql_stmt_cache.h        | 153 +++++++++
 src/box/xrow.c                  |  23 +-
 src/box/xrow.h                  |   4 +-
 src/lib/core/port.h             |   2 +-
 test/app-tap/init_script.result |  37 +-
 test/box/admin.result           |   2 +
 test/box/cfg.result             |   7 +
 test/box/cfg.test.lua           |   1 +
 test/box/info.result            |   1 +
 test/box/misc.result            |   5 +
 test/sql/engine.cfg             |   4 +
 test/sql/iproto.result          |   2 +-
 test/sql/prepared.result        | 737 ++++++++++++++++++++++++++++++++++++++++
 test/sql/prepared.test.lua      | 282 +++++++++++++++
 47 files changed, 2503 insertions(+), 195 deletions(-)
 create mode 100644 src/box/sql_stmt_cache.c
 create mode 100644 src/box/sql_stmt_cache.h
 create mode 100644 test/sql/prepared.result
 create mode 100644 test/sql/prepared.test.lua

-- 
2.15.1

[Tarantool-patches] [PATCH v3 01/20] sql: remove sql_prepare_v2()

[Nikita Pettik]

There are two versions of the same function (sql_prepare()) which are
almost identical. Let's keep more relevant version sql_prepare_v2() but
rename it to sql_prepare() in order to avoid any mess.

Needed for #3292
---
 src/box/execute.c     |  2 +-
 src/box/sql/legacy.c  |  2 +-
 src/box/sql/prepare.c | 32 ++++----------------------------
 src/box/sql/sqlInt.h  | 25 +++++++++++--------------
 src/box/sql/vdbeapi.c |  2 +-
 5 files changed, 18 insertions(+), 45 deletions(-)

diff --git a/src/box/execute.c b/src/box/execute.c
index e8b012e5b..130a3f675 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -443,7 +443,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
 {
 	struct sql_stmt *stmt;
 	struct sql *db = sql_get();
-	if (sql_prepare_v2(db, sql, len, &stmt, NULL) != 0)
+	if (sql_prepare(db, sql, len, &stmt, NULL) != 0)
 		return -1;
 	assert(stmt != NULL);
 	port_sql_create(port, stmt);
diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
index 0b1370f4a..bfd1e32b9 100644
--- a/src/box/sql/legacy.c
+++ b/src/box/sql/legacy.c
@@ -70,7 +70,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
 		char **azVals = 0;
 
 		pStmt = 0;
-		rc = sql_prepare_v2(db, zSql, -1, &pStmt, &zLeftover);
+		rc = sql_prepare(db, zSql, -1, &pStmt, &zLeftover);
 		assert(rc == 0 || pStmt == NULL);
 		if (rc != 0)
 			continue;
diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
index 0ecc676e2..35e81212d 100644
--- a/src/box/sql/prepare.c
+++ b/src/box/sql/prepare.c
@@ -204,36 +204,12 @@ sqlReprepare(Vdbe * p)
 	return 0;
 }
 
-/*
- * Two versions of the official API.  Legacy and new use.  In the legacy
- * version, the original SQL text is not saved in the prepared statement
- * and so if a schema change occurs, an error is returned by
- * sql_step().  In the new version, the original SQL text is retained
- * and the statement is automatically recompiled if an schema change
- * occurs.
- */
-int
-sql_prepare(sql * db,		/* Database handle. */
-		const char *zSql,	/* UTF-8 encoded SQL statement. */
-		int nBytes,		/* Length of zSql in bytes. */
-		sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
-		const char **pzTail)	/* OUT: End of parsed string */
-{
-	int rc = sqlPrepare(db, zSql, nBytes, 0, 0, ppStmt, pzTail);
-	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
-	return rc;
-}
-
 int
-sql_prepare_v2(sql * db,	/* Database handle. */
-		   const char *zSql,	/* UTF-8 encoded SQL statement. */
-		   int nBytes,	/* Length of zSql in bytes. */
-		   sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
-		   const char **pzTail	/* OUT: End of parsed string */
-    )
+sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
+	    const char **sql_tail)
 {
-	int rc = sqlPrepare(db, zSql, nBytes, 1, 0, ppStmt, pzTail);
-	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
+	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
+	assert(rc == 0 || stmt == NULL || *stmt == NULL);
 	return rc;
 }
 
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 2594b73e0..7bd952a17 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -468,21 +468,18 @@ typedef void (*sql_destructor_type) (void *);
 #define SQL_STATIC      ((sql_destructor_type)0)
 #define SQL_TRANSIENT   ((sql_destructor_type)-1)
 
+/**
+ * Prepare (compile into VDBE byte-code) statement.
+ *
+ * @param db Database handle.
+ * @param sql UTF-8 encoded SQL statement.
+ * @param length Length of @param sql in bytes.
+ * @param[out] stmt A pointer to the prepared statement.
+ * @param[out] sql_tail End of parsed string.
+ */
 int
-sql_prepare(sql * db,	/* Database handle */
-		const char *zSql,	/* SQL statement, UTF-8 encoded */
-		int nByte,	/* Maximum length of zSql in bytes. */
-		sql_stmt ** ppStmt,	/* OUT: Statement handle */
-		const char **pzTail	/* OUT: Pointer to unused portion of zSql */
-	);
-
-int
-sql_prepare_v2(sql * db,	/* Database handle */
-		   const char *zSql,	/* SQL statement, UTF-8 encoded */
-		   int nByte,	/* Maximum length of zSql in bytes. */
-		   sql_stmt ** ppStmt,	/* OUT: Statement handle */
-		   const char **pzTail	/* OUT: Pointer to unused portion of zSql */
-	);
+sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
+	    const char **sql_tail);
 
 int
 sql_step(sql_stmt *);
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 685212d91..12449d3bc 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -452,7 +452,7 @@ sqlStep(Vdbe * p)
 		checkProfileCallback(db, p);
 
 	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
-		/* If this statement was prepared using sql_prepare_v2(), and an
+		/* If this statement was prepared using sql_prepare(), and an
 		 * error has occurred, then return an error.
 		 */
 		if (p->is_aborted)
-- 
2.15.1

[Tarantool-patches] [PATCH v3 02/20] sql: refactor sql_prepare() and sqlPrepare()

[Nikita Pettik]

- Removed saveSqlFlag as argument from sqlPrepare(). It was used to
  indicate that its caller is sql_prepare_v2() not sql_prepare().
  Since in previous commit we've left only one version of this function
  let's remove this flag at all.

- Removed struct db from list of sql_prepare() arguments. There's one
  global database handler and it can be obtained by sql_get() call.
  Hence, it makes no sense to pass around this argument.

Needed for #3292
---
 src/box/execute.c     |  3 +--
 src/box/sql/analyze.c | 16 +++++++---------
 src/box/sql/legacy.c  |  2 +-
 src/box/sql/prepare.c | 10 ++++------
 src/box/sql/sqlInt.h  |  3 +--
 src/box/sql/vdbe.h    |  2 +-
 src/box/sql/vdbeInt.h |  1 -
 src/box/sql/vdbeapi.c |  2 +-
 src/box/sql/vdbeaux.c |  5 +----
 9 files changed, 17 insertions(+), 27 deletions(-)

diff --git a/src/box/execute.c b/src/box/execute.c
index 130a3f675..0b21386b5 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -442,8 +442,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
 			struct region *region)
 {
 	struct sql_stmt *stmt;
-	struct sql *db = sql_get();
-	if (sql_prepare(db, sql, len, &stmt, NULL) != 0)
+	if (sql_prepare(sql, len, &stmt, NULL) != 0)
 		return -1;
 	assert(stmt != NULL);
 	port_sql_create(port, stmt);
diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
index b9858c8d6..e43011dd0 100644
--- a/src/box/sql/analyze.c
+++ b/src/box/sql/analyze.c
@@ -1336,14 +1336,13 @@ sample_compare(const void *a, const void *b, void *arg)
  * statistics (i.e. arrays lt, dt, dlt and avg_eq). 'load' query
  * is needed for
  *
- * @param db Database handler.
  * @param sql_select_prepare SELECT statement, see above.
  * @param sql_select_load SELECT statement, see above.
  * @param[out] stats Statistics is saved here.
  * @retval 0 on success, -1 otherwise.
  */
 static int
-load_stat_from_space(struct sql *db, const char *sql_select_prepare,
+load_stat_from_space(const char *sql_select_prepare,
 		     const char *sql_select_load, struct index_stat *stats)
 {
 	struct index **indexes = NULL;
@@ -1359,7 +1358,7 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
 		}
 	}
 	sql_stmt *stmt = NULL;
-	int rc = sql_prepare(db, sql_select_prepare, -1, &stmt, 0);
+	int rc = sql_prepare(sql_select_prepare, -1, &stmt, 0);
 	if (rc)
 		goto finalize;
 	uint32_t current_idx_count = 0;
@@ -1427,7 +1426,7 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
 	rc = sql_finalize(stmt);
 	if (rc)
 		goto finalize;
-	rc = sql_prepare(db, sql_select_load, -1, &stmt, 0);
+	rc = sql_prepare(sql_select_load, -1, &stmt, 0);
 	if (rc)
 		goto finalize;
 	struct index *prev_index = NULL;
@@ -1505,12 +1504,11 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
 }
 
 static int
-load_stat_to_index(struct sql *db, const char *sql_select_load,
-		   struct index_stat **stats)
+load_stat_to_index(const char *sql_select_load, struct index_stat **stats)
 {
 	assert(stats != NULL && *stats != NULL);
 	struct sql_stmt *stmt = NULL;
-	if (sql_prepare(db, sql_select_load, -1, &stmt, 0) != 0)
+	if (sql_prepare(sql_select_load, -1, &stmt, 0) != 0)
 		return -1;
 	uint32_t current_idx_count = 0;
 	while (sql_step(stmt) == SQL_ROW) {
@@ -1696,7 +1694,7 @@ sql_analysis_load(struct sql *db)
 	const char *load_query = "SELECT \"tbl\",\"idx\",\"neq\",\"nlt\","
 				 "\"ndlt\",\"sample\" FROM \"_sql_stat4\"";
 	/* Load the statistics from the _sql_stat4 table. */
-	if (load_stat_from_space(db, init_query, load_query, stats) != 0)
+	if (load_stat_from_space(init_query, load_query, stats) != 0)
 		goto fail;
 	/*
 	 * Now we have complete statistics for each index
@@ -1739,7 +1737,7 @@ sql_analysis_load(struct sql *db)
 	 */
 	const char *order_query = "SELECT \"tbl\",\"idx\" FROM "
 				  "\"_sql_stat4\" GROUP BY \"tbl\",\"idx\"";
-	if (load_stat_to_index(db, order_query, heap_stats) == 0)
+	if (load_stat_to_index(order_query, heap_stats) == 0)
 		return box_txn_commit();
 fail:
 	box_txn_rollback();
diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
index bfd1e32b9..16507b334 100644
--- a/src/box/sql/legacy.c
+++ b/src/box/sql/legacy.c
@@ -70,7 +70,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
 		char **azVals = 0;
 
 		pStmt = 0;
-		rc = sql_prepare(db, zSql, -1, &pStmt, &zLeftover);
+		rc = sql_prepare(zSql, -1, &pStmt, &zLeftover);
 		assert(rc == 0 || pStmt == NULL);
 		if (rc != 0)
 			continue;
diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
index 35e81212d..520b52d64 100644
--- a/src/box/sql/prepare.c
+++ b/src/box/sql/prepare.c
@@ -46,7 +46,6 @@ static int
 sqlPrepare(sql * db,	/* Database handle. */
 	       const char *zSql,	/* UTF-8 encoded SQL statement. */
 	       int nBytes,	/* Length of zSql in bytes. */
-	       int saveSqlFlag,	/* True to copy SQL text into the sql_stmt */
 	       Vdbe * pReprepare,	/* VM being reprepared */
 	       sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
 	       const char **pzTail	/* OUT: End of parsed string */
@@ -156,8 +155,7 @@ sqlPrepare(sql * db,	/* Database handle. */
 
 	if (db->init.busy == 0) {
 		Vdbe *pVdbe = sParse.pVdbe;
-		sqlVdbeSetSql(pVdbe, zSql, (int)(sParse.zTail - zSql),
-				  saveSqlFlag);
+		sqlVdbeSetSql(pVdbe, zSql, (int)(sParse.zTail - zSql));
 	}
 	if (sParse.pVdbe != NULL && (rc != 0 || db->mallocFailed)) {
 		sqlVdbeFinalize(sParse.pVdbe);
@@ -192,7 +190,7 @@ sqlReprepare(Vdbe * p)
 	zSql = sql_sql((sql_stmt *) p);
 	assert(zSql != 0);	/* Reprepare only called for prepare_v2() statements */
 	db = sqlVdbeDb(p);
-	if (sqlPrepare(db, zSql, -1, 0, p, &pNew, 0) != 0) {
+	if (sqlPrepare(db, zSql, -1, p, &pNew, 0) != 0) {
 		assert(pNew == 0);
 		return -1;
 	}
@@ -205,10 +203,10 @@ sqlReprepare(Vdbe * p)
 }
 
 int
-sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
+sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
 	    const char **sql_tail)
 {
-	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
+	int rc = sqlPrepare(sql_get(), sql, length, 0, stmt, sql_tail);
 	assert(rc == 0 || stmt == NULL || *stmt == NULL);
 	return rc;
 }
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 7bd952a17..ac1d8ce42 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -471,14 +471,13 @@ typedef void (*sql_destructor_type) (void *);
 /**
  * Prepare (compile into VDBE byte-code) statement.
  *
- * @param db Database handle.
  * @param sql UTF-8 encoded SQL statement.
  * @param length Length of @param sql in bytes.
  * @param[out] stmt A pointer to the prepared statement.
  * @param[out] sql_tail End of parsed string.
  */
 int
-sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
+sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
 	    const char **sql_tail);
 
 int
diff --git a/src/box/sql/vdbe.h b/src/box/sql/vdbe.h
index 582d48a1f..573577355 100644
--- a/src/box/sql/vdbe.h
+++ b/src/box/sql/vdbe.h
@@ -251,7 +251,7 @@ void sqlVdbeSetNumCols(Vdbe *, int);
 int sqlVdbeSetColName(Vdbe *, int, int, const char *, void (*)(void *));
 void sqlVdbeCountChanges(Vdbe *);
 sql *sqlVdbeDb(Vdbe *);
-void sqlVdbeSetSql(Vdbe *, const char *z, int n, int);
+void sqlVdbeSetSql(Vdbe *, const char *z, int n);
 void sqlVdbeSwap(Vdbe *, Vdbe *);
 VdbeOp *sqlVdbeTakeOpArray(Vdbe *, int *, int *);
 sql_value *sqlVdbeGetBoundValue(Vdbe *, int, u8);
diff --git a/src/box/sql/vdbeInt.h b/src/box/sql/vdbeInt.h
index 0f32b4cd6..078ebc34e 100644
--- a/src/box/sql/vdbeInt.h
+++ b/src/box/sql/vdbeInt.h
@@ -421,7 +421,6 @@ struct Vdbe {
 	bft explain:2;		/* True if EXPLAIN present on SQL command */
 	bft changeCntOn:1;	/* True to update the change-counter */
 	bft runOnlyOnce:1;	/* Automatically expire on reset */
-	bft isPrepareV2:1;	/* True if prepared with prepare_v2() */
 	u32 aCounter[5];	/* Counters used by sql_stmt_status() */
 	char *zSql;		/* Text of the SQL statement that generated this */
 	void *pFree;		/* Free this when deleting the vdbe */
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 12449d3bc..db7936e78 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -451,7 +451,7 @@ sqlStep(Vdbe * p)
 	if (rc != SQL_ROW)
 		checkProfileCallback(db, p);
 
-	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
+	if (rc != SQL_ROW && rc != SQL_DONE) {
 		/* If this statement was prepared using sql_prepare(), and an
 		 * error has occurred, then return an error.
 		 */
diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c
index a1d658648..619105820 100644
--- a/src/box/sql/vdbeaux.c
+++ b/src/box/sql/vdbeaux.c
@@ -89,14 +89,12 @@ sql_vdbe_prepare(struct Vdbe *vdbe)
  * Remember the SQL string for a prepared statement.
  */
 void
-sqlVdbeSetSql(Vdbe * p, const char *z, int n, int isPrepareV2)
+sqlVdbeSetSql(Vdbe * p, const char *z, int n)
 {
-	assert(isPrepareV2 == 1 || isPrepareV2 == 0);
 	if (p == 0)
 		return;
 	assert(p->zSql == 0);
 	p->zSql = sqlDbStrNDup(p->db, z, n);
-	p->isPrepareV2 = (u8) isPrepareV2;
 }
 
 /*
@@ -120,7 +118,6 @@ sqlVdbeSwap(Vdbe * pA, Vdbe * pB)
 	zTmp = pA->zSql;
 	pA->zSql = pB->zSql;
 	pB->zSql = zTmp;
-	pB->isPrepareV2 = pA->isPrepareV2;
 }
 
 /*
-- 
2.15.1

[Tarantool-patches] [PATCH v3 03/20] sql: move sql_prepare() declaration to box/execute.h

[Nikita Pettik]

We are going to split sql_prepare_and_execute() into several explicit
and logically separated steps:

1. sql_prepare() -- compile VDBE byte-code
2. sql_bind() -- bind variables (if there are any)
3. sql_execute() -- query (byte-code) execution in virtual machine

For instance, for dry-run we are interested only in query preparation.
Contrary, if we had prepared statement cache, we could skip query
preparation and handle only bind and execute steps.

To avoid inclusion of sql/sqlInt.h header (which gathers almost all SQL
specific functions and constants) let's move sql_prepare() to
box/execute.h header (which already holds sql_prepare_and_execute()).

Needed for #3292
---
 src/box/execute.h     | 12 ++++++++++++
 src/box/sql/analyze.c |  1 +
 src/box/sql/legacy.c  |  1 +
 src/box/sql/sqlInt.h  | 12 ------------
 4 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/src/box/execute.h b/src/box/execute.h
index a2fd4d1b7..a6000c08b 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -89,6 +89,18 @@ struct port_sql {
 
 extern const struct port_vtab port_sql_vtab;
 
+/**
+ * Prepare (compile into VDBE byte-code) statement.
+ *
+ * @param sql UTF-8 encoded SQL statement.
+ * @param length Length of @param sql in bytes.
+ * @param[out] stmt A pointer to the prepared statement.
+ * @param[out] sql_tail End of parsed string.
+ */
+int
+sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
+	    const char **sql_tail);
+
 #if defined(__cplusplus)
 } /* extern "C" { */
 #endif
diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
index e43011dd0..00ca15413 100644
--- a/src/box/sql/analyze.c
+++ b/src/box/sql/analyze.c
@@ -106,6 +106,7 @@
  */
 
 #include "box/box.h"
+#include "box/execute.h"
 #include "box/index.h"
 #include "box/key_def.h"
 #include "box/schema.h"
diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
index 16507b334..e3a2c77ca 100644
--- a/src/box/sql/legacy.c
+++ b/src/box/sql/legacy.c
@@ -37,6 +37,7 @@
  */
 
 #include "sqlInt.h"
+#include "box/execute.h"
 #include "box/session.h"
 
 /*
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index ac1d8ce42..3ca10778e 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -468,18 +468,6 @@ typedef void (*sql_destructor_type) (void *);
 #define SQL_STATIC      ((sql_destructor_type)0)
 #define SQL_TRANSIENT   ((sql_destructor_type)-1)
 
-/**
- * Prepare (compile into VDBE byte-code) statement.
- *
- * @param sql UTF-8 encoded SQL statement.
- * @param length Length of @param sql in bytes.
- * @param[out] stmt A pointer to the prepared statement.
- * @param[out] sql_tail End of parsed string.
- */
-int
-sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
-	    const char **sql_tail);
-
 int
 sql_step(sql_stmt *);
 
-- 
2.15.1

[Tarantool-patches] [PATCH v3 04/20] sql: rename sqlPrepare() to sql_stmt_compile()

[Nikita Pettik]

sql_prepare() is going not only to compile statement, but also to save it
to the prepared statement cache. So we'd better rename sqlPrepare()
which is static wrapper around sql_prepare() and make it non-static.
Where it is possible let's use sql_stmt_compile() instead of sql_prepare().

Needed for #2592
---
 src/box/execute.c     |  2 +-
 src/box/sql/analyze.c |  6 +++---
 src/box/sql/legacy.c  |  2 +-
 src/box/sql/prepare.c | 21 ++++++---------------
 src/box/sql/sqlInt.h  | 14 ++++++++++++++
 src/box/sql/vdbeapi.c |  2 +-
 6 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/src/box/execute.c b/src/box/execute.c
index 0b21386b5..af66447b5 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -442,7 +442,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
 			struct region *region)
 {
 	struct sql_stmt *stmt;
-	if (sql_prepare(sql, len, &stmt, NULL) != 0)
+	if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
 		return -1;
 	assert(stmt != NULL);
 	port_sql_create(port, stmt);
diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
index 00ca15413..42e2a1a2f 100644
--- a/src/box/sql/analyze.c
+++ b/src/box/sql/analyze.c
@@ -1359,7 +1359,7 @@ load_stat_from_space(const char *sql_select_prepare,
 		}
 	}
 	sql_stmt *stmt = NULL;
-	int rc = sql_prepare(sql_select_prepare, -1, &stmt, 0);
+	int rc = sql_stmt_compile(sql_select_prepare, -1, NULL, &stmt, 0);
 	if (rc)
 		goto finalize;
 	uint32_t current_idx_count = 0;
@@ -1427,7 +1427,7 @@ load_stat_from_space(const char *sql_select_prepare,
 	rc = sql_finalize(stmt);
 	if (rc)
 		goto finalize;
-	rc = sql_prepare(sql_select_load, -1, &stmt, 0);
+	rc = sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0);
 	if (rc)
 		goto finalize;
 	struct index *prev_index = NULL;
@@ -1509,7 +1509,7 @@ load_stat_to_index(const char *sql_select_load, struct index_stat **stats)
 {
 	assert(stats != NULL && *stats != NULL);
 	struct sql_stmt *stmt = NULL;
-	if (sql_prepare(sql_select_load, -1, &stmt, 0) != 0)
+	if (sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0) != 0)
 		return -1;
 	uint32_t current_idx_count = 0;
 	while (sql_step(stmt) == SQL_ROW) {
diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
index e3a2c77ca..458afd7f8 100644
--- a/src/box/sql/legacy.c
+++ b/src/box/sql/legacy.c
@@ -71,7 +71,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
 		char **azVals = 0;
 
 		pStmt = 0;
-		rc = sql_prepare(zSql, -1, &pStmt, &zLeftover);
+		rc = sql_stmt_compile(zSql, -1, NULL, &pStmt, &zLeftover);
 		assert(rc == 0 || pStmt == NULL);
 		if (rc != 0)
 			continue;
diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
index 520b52d64..73d6866a4 100644
--- a/src/box/sql/prepare.c
+++ b/src/box/sql/prepare.c
@@ -39,18 +39,11 @@
 #include "box/space.h"
 #include "box/session.h"
 
-/*
- * Compile the UTF-8 encoded SQL statement zSql into a statement handle.
- */
-static int
-sqlPrepare(sql * db,	/* Database handle. */
-	       const char *zSql,	/* UTF-8 encoded SQL statement. */
-	       int nBytes,	/* Length of zSql in bytes. */
-	       Vdbe * pReprepare,	/* VM being reprepared */
-	       sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
-	       const char **pzTail	/* OUT: End of parsed string */
-    )
+int
+sql_stmt_compile(const char *zSql, int nBytes, struct Vdbe *pReprepare,
+		 sql_stmt **ppStmt, const char **pzTail)
 {
+	struct sql *db = sql_get();
 	int rc = 0;	/* Result code */
 	Parse sParse;		/* Parsing context */
 	sql_parser_create(&sParse, db, current_session()->sql_flags);
@@ -185,12 +178,10 @@ sqlReprepare(Vdbe * p)
 {
 	sql_stmt *pNew;
 	const char *zSql;
-	sql *db;
 
 	zSql = sql_sql((sql_stmt *) p);
 	assert(zSql != 0);	/* Reprepare only called for prepare_v2() statements */
-	db = sqlVdbeDb(p);
-	if (sqlPrepare(db, zSql, -1, p, &pNew, 0) != 0) {
+	if (sql_stmt_compile(zSql, -1, p, &pNew, 0) != 0) {
 		assert(pNew == 0);
 		return -1;
 	}
@@ -206,7 +197,7 @@ int
 sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
 	    const char **sql_tail)
 {
-	int rc = sqlPrepare(sql_get(), sql, length, 0, stmt, sql_tail);
+	int rc = sql_stmt_compile(sql, length, 0, stmt, sql_tail);
 	assert(rc == 0 || stmt == NULL || *stmt == NULL);
 	return rc;
 }
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 3ca10778e..03deb733c 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -468,6 +468,20 @@ typedef void (*sql_destructor_type) (void *);
 #define SQL_STATIC      ((sql_destructor_type)0)
 #define SQL_TRANSIENT   ((sql_destructor_type)-1)
 
+/**
+ * Compile the UTF-8 encoded SQL statement into
+ * a statement handle (struct Vdbe).
+ *
+ * @param sql UTF-8 encoded SQL statement.
+ * @param sql_len Length of @sql in bytes.
+ * @param re_prepared VM being re-compiled. Can be NULL.
+ * @param[out] stmt A pointer to the compiled statement.
+ * @param[out] sql_tail End of parsed string.
+ */
+int
+sql_stmt_compile(const char *sql, int bytes_count, struct Vdbe *re_prepared,
+		 sql_stmt **stmt, const char **sql_tail);
+
 int
 sql_step(sql_stmt *);
 
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index db7936e78..ab8441bc5 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -71,7 +71,7 @@ invokeProfileCallback(sql * db, Vdbe * p)
 
 /*
  * The following routine destroys a virtual machine that is created by
- * the sql_compile() routine. The integer returned is an SQL_
+ * the sql_stmt_compile() routine. The integer returned is an SQL_
  * success/failure code that describes the result of executing the virtual
  * machine.
  */
-- 
2.15.1

[Tarantool-patches] [PATCH v3 05/20] sql: rename sql_finalize() to sql_stmt_finalize()

[Nikita Pettik]

Let's follow unified naming rules for SQL high level API which
manipulates on statements objects. To be more precise, let's use
'sql_stmt_' prefix for interface functions operating on statement
handles.
---
 src/box/bind.c          | 2 +-
 src/box/ck_constraint.c | 4 ++--
 src/box/execute.c       | 2 +-
 src/box/lua/execute.c   | 2 +-
 src/box/sql/analyze.c   | 6 +++---
 src/box/sql/sqlInt.h    | 2 +-
 src/box/sql/vdbe.c      | 4 ++--
 src/box/sql/vdbeapi.c   | 2 +-
 8 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/src/box/bind.c b/src/box/bind.c
index 7eea9fcc8..bbc1f56df 100644
--- a/src/box/bind.c
+++ b/src/box/bind.c
@@ -180,7 +180,7 @@ sql_bind_column(struct sql_stmt *stmt, const struct sql_bind *p,
 		 * Parameters are allocated within message pack,
 		 * received from the iproto thread. IProto thread
 		 * now is waiting for the response and it will not
-		 * free the packet until sql_finalize. So
+		 * free the packet until sql_stmt_finalize. So
 		 * there is no need to copy the packet and we can
 		 * use SQL_STATIC.
 		 */
diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
index a2c66ce00..551bdd397 100644
--- a/src/box/ck_constraint.c
+++ b/src/box/ck_constraint.c
@@ -141,7 +141,7 @@ ck_constraint_program_compile(struct ck_constraint_def *ck_constraint_def,
 		diag_set(ClientError, ER_CREATE_CK_CONSTRAINT,
 			 ck_constraint_def->name,
 			 box_error_message(box_error_last()));
-		sql_finalize((struct sql_stmt *) v);
+		sql_stmt_finalize((struct sql_stmt *) v);
 		return NULL;
 	}
 	return (struct sql_stmt *) v;
@@ -254,7 +254,7 @@ error:
 void
 ck_constraint_delete(struct ck_constraint *ck_constraint)
 {
-	sql_finalize(ck_constraint->stmt);
+	sql_stmt_finalize(ck_constraint->stmt);
 	ck_constraint_def_delete(ck_constraint->def);
 	TRASH(ck_constraint);
 	free(ck_constraint);
diff --git a/src/box/execute.c b/src/box/execute.c
index af66447b5..fb83e1194 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -100,7 +100,7 @@ static void
 port_sql_destroy(struct port *base)
 {
 	port_tuple_vtab.destroy(base);
-	sql_finalize(((struct port_sql *)base)->stmt);
+	sql_stmt_finalize(((struct port_sql *)base)->stmt);
 }
 
 const struct port_vtab port_sql_vtab = {
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index ffa3d4d2e..68adacf72 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -217,7 +217,7 @@ lua_sql_bind_list_decode(struct lua_State *L, struct sql_bind **out_bind,
 	size_t size = sizeof(struct sql_bind) * bind_count;
 	/*
 	 * Memory allocated here will be freed in
-	 * sql_finalize() or in txn_commit()/txn_rollback() if
+	 * sql_stmt_finalize() or in txn_commit()/txn_rollback() if
 	 * there is an active transaction.
 	 */
 	struct sql_bind *bind = (struct sql_bind *) region_alloc(region, size);
diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
index 42e2a1a2f..f74f9b358 100644
--- a/src/box/sql/analyze.c
+++ b/src/box/sql/analyze.c
@@ -1424,7 +1424,7 @@ load_stat_from_space(const char *sql_select_prepare,
 		current_idx_count++;
 
 	}
-	rc = sql_finalize(stmt);
+	rc = sql_stmt_finalize(stmt);
 	if (rc)
 		goto finalize;
 	rc = sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0);
@@ -1475,7 +1475,7 @@ load_stat_from_space(const char *sql_select_prepare,
 		sample->sample_key = region_alloc(&fiber()->gc,
 						  sample->key_size);
 		if (sample->sample_key == NULL) {
-			sql_finalize(stmt);
+			sql_stmt_finalize(stmt);
 			rc = -1;
 			diag_set(OutOfMemory, sample->key_size,
 				 "region", "sample_key");
@@ -1488,7 +1488,7 @@ load_stat_from_space(const char *sql_select_prepare,
 		}
 		stats[current_idx_count].sample_count++;
 	}
-	rc = sql_finalize(stmt);
+	rc = sql_stmt_finalize(stmt);
 	if (rc == 0 && prev_index != NULL)
 		init_avg_eq(prev_index, &stats[current_idx_count]);
 	assert(current_idx_count <= index_count);
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 03deb733c..cf0b946f1 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -521,7 +521,7 @@ sql_column_value(sql_stmt *,
 		     int iCol);
 
 int
-sql_finalize(sql_stmt * pStmt);
+sql_stmt_finalize(sql_stmt * pStmt);
 
 /*
  * Terminate the current execution of an SQL statement and reset
diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c
index ab86be9a9..336fd4a52 100644
--- a/src/box/sql/vdbe.c
+++ b/src/box/sql/vdbe.c
@@ -1084,7 +1084,7 @@ case OP_Yield: {            /* in1, jump */
  * automatically.
  *
  * P1 is the result code returned by sql_exec(),
- * sql_reset(), or sql_finalize().  For a normal halt,
+ * sql_reset(), or sql_stmt_finalize().  For a normal halt,
  * this should be 0.
  * For errors, it can be some other value.  If P1!=0 then P2 will
  * determine whether or not to rollback the current transaction.
@@ -2887,7 +2887,7 @@ case OP_MakeRecord: {
 	 * memory shouldn't be reused until it is written into WAL.
 	 *
 	 * However, if memory for ephemeral space is allocated
-	 * on region, it will be freed only in sql_finalize()
+	 * on region, it will be freed only in sql_stmt_finalize()
 	 * routine.
 	 */
 	if (bIsEphemeral) {
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index ab8441bc5..7463fb9d7 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -76,7 +76,7 @@ invokeProfileCallback(sql * db, Vdbe * p)
  * machine.
  */
 int
-sql_finalize(sql_stmt * pStmt)
+sql_stmt_finalize(sql_stmt * pStmt)
 {
 	if (pStmt == NULL)
 		return 0;
-- 
2.15.1

[Tarantool-patches] [PATCH v3 06/20] sql: rename sql_reset() to sql_stmt_reset()

[Nikita Pettik]

---
 src/box/ck_constraint.c | 2 +-
 src/box/sql/sqlInt.h    | 2 +-
 src/box/sql/vdbe.c      | 2 +-
 src/box/sql/vdbeapi.c   | 4 ++--
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
index 551bdd397..bc2a5e8f4 100644
--- a/src/box/ck_constraint.c
+++ b/src/box/ck_constraint.c
@@ -173,7 +173,7 @@ ck_constraint_program_run(struct ck_constraint *ck_constraint,
 	 * Get VDBE execution state and reset VM to run it
 	 * next time.
 	 */
-	return sql_reset(ck_constraint->stmt);
+	return sql_stmt_reset(ck_constraint->stmt);
 }
 
 int
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index cf0b946f1..b1e4ac2fa 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -532,7 +532,7 @@ sql_stmt_finalize(sql_stmt * pStmt);
  * @retval sql_ret_code Error code on error.
  */
 int
-sql_reset(struct sql_stmt *stmt);
+sql_stmt_reset(struct sql_stmt *stmt);
 
 int
 sql_exec(sql *,	/* An open database */
diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c
index 336fd4a52..8da007b4d 100644
--- a/src/box/sql/vdbe.c
+++ b/src/box/sql/vdbe.c
@@ -1084,7 +1084,7 @@ case OP_Yield: {            /* in1, jump */
  * automatically.
  *
  * P1 is the result code returned by sql_exec(),
- * sql_reset(), or sql_stmt_finalize().  For a normal halt,
+ * sql_stmt_reset(), or sql_stmt_finalize().  For a normal halt,
  * this should be 0.
  * For errors, it can be some other value.  If P1!=0 then P2 will
  * determine whether or not to rollback the current transaction.
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 7463fb9d7..b6bf9aa81 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -88,7 +88,7 @@ sql_stmt_finalize(sql_stmt * pStmt)
 }
 
 int
-sql_reset(sql_stmt * pStmt)
+sql_stmt_reset(sql_stmt *pStmt)
 {
 	assert(pStmt != NULL);
 	struct Vdbe *v = (Vdbe *) pStmt;
@@ -414,7 +414,7 @@ sqlStep(Vdbe * p)
 
 	assert(p);
 	if (p->magic != VDBE_MAGIC_RUN)
-		sql_reset((sql_stmt *) p);
+		sql_stmt_reset((sql_stmt *) p);
 
 	/* Check that malloc() has not failed. If it has, return early. */
 	db = p->db;
-- 
2.15.1

[Tarantool-patches] [PATCH v3 07/20] sql: move sql_stmt_finalize() to execute.h

[Nikita Pettik]

We are going to introduce prepared statement cache. On statement's
deallocation we should release all resources which is done by
sql_finalize(). Now it is declared in sql/sqlInt.h header, which
accumulates almost all SQL related functions. To avoid including such a
huge header to use single function, let's move its signature to
box/execute.h

Need for #2592
---
 src/box/ck_constraint.c | 1 +
 src/box/execute.h       | 3 +++
 src/box/sql/sqlInt.h    | 3 ---
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
index bc2a5e8f4..ff3f05587 100644
--- a/src/box/ck_constraint.c
+++ b/src/box/ck_constraint.c
@@ -29,6 +29,7 @@
  * SUCH DAMAGE.
  */
 #include "box/session.h"
+#include "execute.h"
 #include "bind.h"
 #include "ck_constraint.h"
 #include "errcode.h"
diff --git a/src/box/execute.h b/src/box/execute.h
index a6000c08b..ce1e7a67d 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -89,6 +89,9 @@ struct port_sql {
 
 extern const struct port_vtab port_sql_vtab;
 
+int
+sql_stmt_finalize(struct sql_stmt *stmt);
+
 /**
  * Prepare (compile into VDBE byte-code) statement.
  *
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index b1e4ac2fa..24da3ca11 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -520,9 +520,6 @@ sql_value *
 sql_column_value(sql_stmt *,
 		     int iCol);
 
-int
-sql_stmt_finalize(sql_stmt * pStmt);
-
 /*
  * Terminate the current execution of an SQL statement and reset
  * it back to its starting state so that it can be reused.
-- 
2.15.1

[Tarantool-patches] [PATCH v3 08/20] port: increase padding of struct port

[Nikita Pettik]

We are going to extend context of struct port_sql. One already inherits
struct port_tuple, which makes it size barely fits into 48 bytes of
padding of basic structure (struct port). Hence, let's increase padding
a bit to be able to add at least one more member to struct port_sql.
---
 src/lib/core/port.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/lib/core/port.h b/src/lib/core/port.h
index d61342287..bfdfa4656 100644
--- a/src/lib/core/port.h
+++ b/src/lib/core/port.h
@@ -122,7 +122,7 @@ struct port {
 	 * Implementation dependent content. Needed to declare
 	 * an abstract port instance on stack.
 	 */
-	char pad[48];
+	char pad[52];
 };
 
 /** Is not inlined just to be exported. */
-- 
2.15.1

[Tarantool-patches] [PATCH v3 09/20] port: add result set format and request type to port_sql

[Nikita Pettik]

Result set serialization formats of DQL and DML queries are different:
the last one contains number of affected rows and optionally list of
autoincremented ids; the first one comprises all meta-information
including column names of resulting set and their types. What is more,
serialization format is going to be different for execute and prepare
requests. So let's introduce separate member to struct port_sql
responsible for serialization format to be used.

Note that C standard specifies that enums are integers, but it does not
specify the size. Hence, let's use simple uint8 - mentioned enum are
small enough to fit into it.

What is more, prepared statement finalization is required only for
PREPARE-AND-EXECUTE requests. So let's keep flag indicating required
finalization as well.

Needed for #2592
---
 src/box/execute.c     | 34 +++++++++++++++++++++++++---------
 src/box/execute.h     | 21 +++++++++++++++++++++
 src/box/lua/execute.c | 22 +++++++++++++++-------
 3 files changed, 61 insertions(+), 16 deletions(-)

diff --git a/src/box/execute.c b/src/box/execute.c
index fb83e1194..3bc4988b7 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -100,7 +100,9 @@ static void
 port_sql_destroy(struct port *base)
 {
 	port_tuple_vtab.destroy(base);
-	sql_stmt_finalize(((struct port_sql *)base)->stmt);
+	struct port_sql *port_sql = (struct port_sql *) base;
+	if (port_sql->do_finalize)
+		sql_stmt_finalize(((struct port_sql *)base)->stmt);
 }
 
 const struct port_vtab port_sql_vtab = {
@@ -114,11 +116,15 @@ const struct port_vtab port_sql_vtab = {
 };
 
 static void
-port_sql_create(struct port *port, struct sql_stmt *stmt)
+port_sql_create(struct port *port, struct sql_stmt *stmt,
+		enum sql_serialization_format format, bool do_finalize)
 {
 	port_tuple_create(port);
-	((struct port_sql *)port)->stmt = stmt;
 	port->vtab = &port_sql_vtab;
+	struct port_sql *port_sql = (struct port_sql *) port;
+	port_sql->stmt = stmt;
+	port_sql->serialization_format = format;
+	port_sql->do_finalize = do_finalize;
 }
 
 /**
@@ -324,9 +330,10 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 {
 	assert(port->vtab == &port_sql_vtab);
 	sql *db = sql_get();
-	struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
-	int column_count = sql_column_count(stmt);
-	if (column_count > 0) {
+	struct port_sql *sql_port = (struct port_sql *)port;
+	struct sql_stmt *stmt = sql_port->stmt;
+	switch (sql_port->serialization_format) {
+	case DQL_EXECUTE: {
 		int keys = 2;
 		int size = mp_sizeof_map(keys);
 		char *pos = (char *) obuf_alloc(out, size);
@@ -335,7 +342,7 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 			return -1;
 		}
 		pos = mp_encode_map(pos, keys);
-		if (sql_get_metadata(stmt, out, column_count) != 0)
+		if (sql_get_metadata(stmt, out, sql_column_count(stmt)) != 0)
 			return -1;
 		size = mp_sizeof_uint(IPROTO_DATA);
 		pos = (char *) obuf_alloc(out, size);
@@ -346,7 +353,9 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 		pos = mp_encode_uint(pos, IPROTO_DATA);
 		if (port_tuple_vtab.dump_msgpack(port, out) < 0)
 			return -1;
-	} else {
+		break;
+	}
+	case DML_EXECUTE: {
 		int keys = 1;
 		assert(((struct port_tuple *)port)->size == 0);
 		struct stailq *autoinc_id_list =
@@ -395,6 +404,11 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 				      mp_encode_int(buf, id_entry->id);
 			}
 		}
+		break;
+	}
+	default: {
+		unreachable();
+	}
 	}
 	return 0;
 }
@@ -445,7 +459,9 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
 	if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
 		return -1;
 	assert(stmt != NULL);
-	port_sql_create(port, stmt);
+	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
+					   DQL_EXECUTE : DML_EXECUTE;
+	port_sql_create(port, stmt, format, true);
 	if (sql_bind(stmt, bind, bind_count) == 0 &&
 	    sql_execute(stmt, port, region) == 0)
 		return 0;
diff --git a/src/box/execute.h b/src/box/execute.h
index ce1e7a67d..c87e765cf 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -46,6 +46,17 @@ enum sql_info_key {
 	sql_info_key_MAX,
 };
 
+/**
+ * One of possible formats used to dump msgpack/Lua.
+ * For details see port_sql_dump_msgpack() and port_sql_dump_lua().
+ */
+enum sql_serialization_format {
+	DQL_EXECUTE = 0,
+	DML_EXECUTE = 1,
+	DQL_PREPARE = 2,
+	DML_PREPARE = 3,
+};
+
 extern const char *sql_info_key_strs[];
 
 struct region;
@@ -85,6 +96,16 @@ struct port_sql {
 	struct port_tuple port_tuple;
 	/* Prepared SQL statement. */
 	struct sql_stmt *stmt;
+	/**
+	 * Serialization format depends on type of SQL query: DML or
+	 * DQL; and on type of SQL request: execute or prepare.
+	 */
+	uint8_t serialization_format;
+	/**
+	 * There's no need in clean-up in case of PREPARE request:
+	 * statement remains in cache and will be deleted later.
+	 */
+	bool do_finalize;
 };
 
 extern const struct port_vtab port_sql_vtab;
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index 68adacf72..b164ffcaf 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -45,18 +45,21 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
 	assert(is_flat == false);
 	assert(port->vtab == &port_sql_vtab);
 	struct sql *db = sql_get();
-	struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
-	int column_count = sql_column_count(stmt);
-	if (column_count > 0) {
+	struct port_sql *port_sql = (struct port_sql *)port;
+	struct sql_stmt *stmt = port_sql->stmt;
+	switch (port_sql->serialization_format) {
+	case DQL_EXECUTE: {
 		lua_createtable(L, 0, 2);
-		lua_sql_get_metadata(stmt, L, column_count);
+		lua_sql_get_metadata(stmt, L, sql_column_count(stmt));
 		lua_setfield(L, -2, "metadata");
 		port_tuple_vtab.dump_lua(port, L, false);
 		lua_setfield(L, -2, "rows");
-	} else {
-		assert(((struct port_tuple *)port)->size == 0);
+		break;
+	}
+	case DML_EXECUTE: {
+		assert(((struct port_tuple *) port)->size == 0);
 		struct stailq *autoinc_id_list =
-			vdbe_autoinc_id_list((struct Vdbe *)stmt);
+			vdbe_autoinc_id_list((struct Vdbe *) stmt);
 		lua_createtable(L, 0, stailq_empty(autoinc_id_list) ? 1 : 2);
 
 		luaL_pushuint64(L, db->nChange);
@@ -77,6 +80,11 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
 				sql_info_key_strs[SQL_INFO_AUTOINCREMENT_IDS];
 			lua_setfield(L, -2, field_name);
 		}
+		break;
+	}
+	default: {
+		unreachable();
+	}
 	}
 }
 
-- 
2.15.1

[Tarantool-patches] [PATCH v3 10/20] sql: resurrect sql_bind_parameter_count() function

[Nikita Pettik]

This function is present in sql/vdbeapi.c source file, its prototype is
missing in any header file. It makes impossible to use it. Let's add
prototype declaration to sql/sqlInt.h (as other parameter
setters/getters) and refactor a bit in accordance with our codestyle.

Need for #2592
---
 src/box/sql/sqlInt.h  |  6 ++++++
 src/box/sql/vdbeapi.c | 10 +++-------
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 24da3ca11..a9faaa6e7 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -689,6 +689,12 @@ int
 sql_bind_zeroblob64(sql_stmt *, int,
 			sql_uint64);
 
+/**
+ * Return the number of wildcards that should be bound to.
+ */
+int
+sql_bind_parameter_count(const struct sql_stmt *stmt);
+
 /**
  * Perform pointer parameter binding for the prepared sql
  * statement.
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index b6bf9aa81..7fda525ce 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -1051,15 +1051,11 @@ sql_bind_zeroblob64(sql_stmt * pStmt, int i, sql_uint64 n)
 	return sql_bind_zeroblob(pStmt, i, n);
 }
 
-/*
- * Return the number of wildcards that can be potentially bound to.
- * This routine is added to support DBD::sql.
- */
 int
-sql_bind_parameter_count(sql_stmt * pStmt)
+sql_bind_parameter_count(const struct sql_stmt *stmt)
 {
-	Vdbe *p = (Vdbe *) pStmt;
-	return p ? p->nVar : 0;
+	struct Vdbe *p = (struct Vdbe *) stmt;
+	return p->nVar;
 }
 
 /*
-- 
2.15.1

[Tarantool-patches] [PATCH v3 11/20] sql: resurrect sql_bind_parameter_name()

[Nikita Pettik]

We may need to get name of parameter to be bound by its index position.
So let's resurrect sql_bind_parameter_name() - put its prototype to
sql/sqlInt.h header and update codestyle.

Need for #2592
---
 src/box/sql/sqlInt.h  |  8 ++++++++
 src/box/sql/vdbeapi.c | 14 ++++----------
 2 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index a9faaa6e7..09f638268 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -695,6 +695,14 @@ sql_bind_zeroblob64(sql_stmt *, int,
 int
 sql_bind_parameter_count(const struct sql_stmt *stmt);
 
+/**
+ * Return the name of a wildcard parameter. Return NULL if the index
+ * is out of range or if the wildcard is unnamed. Parameter's index
+ * is 0-based.
+ */
+const char *
+sql_bind_parameter_name(const struct sql_stmt *stmt, int i);
+
 /**
  * Perform pointer parameter binding for the prepared sql
  * statement.
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 7fda525ce..b1c556ec3 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -1058,18 +1058,12 @@ sql_bind_parameter_count(const struct sql_stmt *stmt)
 	return p->nVar;
 }
 
-/*
- * Return the name of a wildcard parameter.  Return NULL if the index
- * is out of range or if the wildcard is unnamed.
- *
- * The result is always UTF-8.
- */
 const char *
-sql_bind_parameter_name(sql_stmt * pStmt, int i)
+sql_bind_parameter_name(const struct sql_stmt *stmt, int i)
 {
-	Vdbe *p = (Vdbe *) pStmt;
-	if (p == 0)
-		return 0;
+	struct Vdbe *p = (struct Vdbe *) stmt;
+	if (p == NULL)
+		return NULL;
 	return sqlVListNumToName(p->pVList, i);
 }
 
-- 
2.15.1

[Tarantool-patches] [PATCH v3 12/20] sql: add sql_stmt_schema_version()

[Nikita Pettik]

Let's introduce interface function to get schema version of prepared
statement. It is required since sturct sql_stmt (i.e. prepared
statement) is an opaque object and in fact is an alias to struct Vdbe.
Statements with schema version different from the current one are
considered to be expired and should be re-compiled.

Needed for #2592
---
 src/box/sql/sqlInt.h  | 3 +++
 src/box/sql/vdbeapi.c | 7 +++++++
 2 files changed, 10 insertions(+)

diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 09f638268..7dfc29809 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -571,6 +571,9 @@ sql_column_name(sql_stmt *, int N);
 const char *
 sql_column_datatype(sql_stmt *, int N);
 
+uint32_t
+sql_stmt_schema_version(const struct sql_stmt *stmt);
+
 int
 sql_initialize(void);
 
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index b1c556ec3..7d9ce11e7 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -798,6 +798,13 @@ sql_column_decltype(sql_stmt * pStmt, int N)
 			  COLNAME_DECLTYPE);
 }
 
+uint32_t
+sql_stmt_schema_version(const struct sql_stmt *stmt)
+{
+	struct Vdbe *v = (struct Vdbe *) stmt;
+	return v->schema_ver;
+}
+
 /******************************* sql_bind_  **************************
  *
  * Routines used to attach values to wildcards in a compiled SQL statement.
-- 
2.15.1

[Tarantool-patches] [PATCH v3 13/20] sql: introduce sql_stmt_sizeof() function

[Nikita Pettik]

To implement memory quota of prepared statement cache, we have to
estimate size of prepared statement. This function attempts at that.

Part of #2592
---
 src/box/execute.h     |  8 ++++++++
 src/box/sql/vdbeapi.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 61 insertions(+)

diff --git a/src/box/execute.h b/src/box/execute.h
index c87e765cf..2dd4fca03 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -113,6 +113,14 @@ extern const struct port_vtab port_sql_vtab;
 int
 sql_stmt_finalize(struct sql_stmt *stmt);
 
+/**
+ * Calculate estimated size of memory occupied by VM.
+ * See sqlVdbeMakeReady() for details concerning allocated
+ * memory.
+ */
+size_t
+sql_stmt_est_size(const struct sql_stmt *stmt);
+
 /**
  * Prepare (compile into VDBE byte-code) statement.
  *
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 7d9ce11e7..2ac174112 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -805,6 +805,59 @@ sql_stmt_schema_version(const struct sql_stmt *stmt)
 	return v->schema_ver;
 }
 
+size_t
+sql_stmt_est_size(const struct sql_stmt *stmt)
+{
+	struct Vdbe *v = (struct Vdbe *) stmt;
+	size_t size = sizeof(*v);
+	/* Names and types of result set columns */
+	size += sizeof(struct Mem) * v->nResColumn * COLNAME_N;
+	/* Opcodes */
+	size += sizeof(struct VdbeOp) * v->nOp;
+	/* Memory cells */
+	size += sizeof(struct Mem) * v->nMem;
+	/* Bindings */
+	size += sizeof(struct Mem) * v->nVar;
+	/* Bindings included in the result set */
+	size += sizeof(uint32_t) * v->res_var_count;
+	/* Cursors */
+	size += sizeof(struct VdbeCursor *) * v->nCursor;
+
+	for (int i = 0; i < v->nOp; ++i) {
+		/* Estimate size of p4 operand. */
+		if (v->aOp[i].p4type == P4_NOTUSED)
+			continue;
+		switch (v->aOp[i].p4type) {
+		case P4_DYNAMIC:
+		case P4_STATIC:
+			if (v->aOp[i].opcode == OP_Blob ||
+			    v->aOp[i].opcode == OP_String)
+				size += v->aOp[i].p1;
+			else if (v->aOp[i].opcode == OP_String8)
+				size += strlen(v->aOp[i].p4.z);
+			break;
+		case P4_BOOL:
+			size += sizeof(v->aOp[i].p4.b);
+			break;
+		case P4_INT32:
+			size += sizeof(v->aOp[i].p4.i);
+			break;
+		case P4_UINT64:
+		case P4_INT64:
+			size += sizeof(*v->aOp[i].p4.pI64);
+			break;
+		case P4_REAL:
+			size += sizeof(*v->aOp[i].p4.pReal);
+			break;
+		default:
+			size += sizeof(v->aOp[i].p4.p);
+			break;
+		}
+	}
+	size += strlen(v->zSql);
+	return size;
+}
+
 /******************************* sql_bind_  **************************
  *
  * Routines used to attach values to wildcards in a compiled SQL statement.
-- 
2.15.1

[Tarantool-patches] [PATCH v3 14/20] box: increment schema_version on ddl operations

[Nikita Pettik]

Some DDL operations such as SQL trigger alter, check and foreign
constraint alter don't result in schema version change. On the other
hand, we are going to rely on schema version to determine expired
prepared statements: for instance, if FK constraint has been created
after DML statement preparation, the latter may ignore FK constraint
(instead of proper "statement has expired" error). Let's fix it and
account schema change on each DDL operation.

Need for #2592
---
 src/box/alter.cc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/box/alter.cc b/src/box/alter.cc
index bef25b605..f33c1dfd2 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -4773,6 +4773,7 @@ on_replace_dd_trigger(struct trigger * /* trigger */, void *event)
 
 	txn_stmt_on_rollback(stmt, on_rollback);
 	txn_stmt_on_commit(stmt, on_commit);
+	++schema_version;
 	return 0;
 }
 
@@ -5283,6 +5284,7 @@ on_replace_dd_fk_constraint(struct trigger * /* trigger*/, void *event)
 		space_reset_fk_constraint_mask(child_space);
 		space_reset_fk_constraint_mask(parent_space);
 	}
+	++schema_version;
 	return 0;
 }
 
@@ -5528,6 +5530,7 @@ on_replace_dd_ck_constraint(struct trigger * /* trigger*/, void *event)
 
 	if (trigger_run(&on_alter_space, space) != 0)
 		return -1;
+	++schema_version;
 	return 0;
 }
 
-- 
2.15.1

[Tarantool-patches] [PATCH v3 15/20] sql: introduce sql_stmt_query_str() method

[Nikita Pettik]

It is getter to fetch string of SQL query from prepared statement.

Needed for #2592
---
 src/box/execute.h     | 6 ++++++
 src/box/sql/vdbeapi.c | 7 +++++++
 2 files changed, 13 insertions(+)

diff --git a/src/box/execute.h b/src/box/execute.h
index 2dd4fca03..f3d6c38b3 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -121,6 +121,12 @@ sql_stmt_finalize(struct sql_stmt *stmt);
 size_t
 sql_stmt_est_size(const struct sql_stmt *stmt);
 
+/**
+ * Return string of SQL query.
+ */
+const char *
+sql_stmt_query_str(const struct sql_stmt *stmt);
+
 /**
  * Prepare (compile into VDBE byte-code) statement.
  *
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 2ac174112..7278d2ab3 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -858,6 +858,13 @@ sql_stmt_est_size(const struct sql_stmt *stmt)
 	return size;
 }
 
+const char *
+sql_stmt_query_str(const struct sql_stmt *stmt)
+{
+	const struct Vdbe *v = (const struct Vdbe *) stmt;
+	return v->zSql;
+}
+
 /******************************* sql_bind_  **************************
  *
  * Routines used to attach values to wildcards in a compiled SQL statement.
-- 
2.15.1

[Tarantool-patches] [PATCH v3 16/20] sql: move sql_stmt_busy() declaration to box/execute.h

[Nikita Pettik]

We are going to use it in box/execute.c and in SQL prepared statement
cache implementation. So to avoid including whole sqlInt.h let's move it
to relative small execute.h header. Let's also fix codestyle of this
function.

Needed for #2592
---
 src/box/execute.h     |  4 ++++
 src/box/sql/sqlInt.h  |  3 ---
 src/box/sql/vdbeapi.c | 10 ++++------
 src/box/sql/vdbeaux.c |  1 +
 4 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/src/box/execute.h b/src/box/execute.h
index f3d6c38b3..61c8e0281 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -127,6 +127,10 @@ sql_stmt_est_size(const struct sql_stmt *stmt);
 const char *
 sql_stmt_query_str(const struct sql_stmt *stmt);
 
+/** Return true if statement executes right now. */
+int
+sql_stmt_busy(const struct sql_stmt *stmt);
+
 /**
  * Prepare (compile into VDBE byte-code) statement.
  *
diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
index 7dfc29809..cbe46e790 100644
--- a/src/box/sql/sqlInt.h
+++ b/src/box/sql/sqlInt.h
@@ -718,9 +718,6 @@ sql_bind_parameter_name(const struct sql_stmt *stmt, int i);
 int
 sql_bind_ptr(struct sql_stmt *stmt, int i, void *ptr);
 
-int
-sql_stmt_busy(sql_stmt *);
-
 int
 sql_init_db(sql **db);
 
diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
index 7278d2ab3..01185af5f 100644
--- a/src/box/sql/vdbeapi.c
+++ b/src/box/sql/vdbeapi.c
@@ -1190,14 +1190,12 @@ sql_db_handle(sql_stmt * pStmt)
 	return pStmt ? ((Vdbe *) pStmt)->db : 0;
 }
 
-/*
- * Return true if the prepared statement is in need of being reset.
- */
 int
-sql_stmt_busy(sql_stmt * pStmt)
+sql_stmt_busy(const struct sql_stmt *stmt)
 {
-	Vdbe *v = (Vdbe *) pStmt;
-	return v != 0 && v->magic == VDBE_MAGIC_RUN && v->pc >= 0;
+	assert(stmt != NULL);
+	const struct Vdbe *v = (const struct Vdbe *) stmt;
+	return v->magic == VDBE_MAGIC_RUN && v->pc >= 0;
 }
 
 /*
diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c
index 619105820..afe1ecb2a 100644
--- a/src/box/sql/vdbeaux.c
+++ b/src/box/sql/vdbeaux.c
@@ -43,6 +43,7 @@
 #include "sqlInt.h"
 #include "vdbeInt.h"
 #include "tarantoolInt.h"
+#include "box/execute.h"
 
 /*
  * Create a new virtual database engine.
-- 
2.15.1

[Tarantool-patches] [PATCH v3 17/20] sql: introduce holder for prepared statemets

[Nikita Pettik]

This patch introduces holder (as data structure) to handle prepared
statements and a set of interface functions (insert, delete, find) to
operate on it. Holder under the hood is implemented as a global hash
(keys are values of hash function applied to the original string containing
SQL query; values are pointer to wrappers around compiled VDBE objects) and
GC queue. Each entry in hash has reference counter. When it reaches 0
value, entry is moved to GC queue. In case prepared statements holder is
out of memory, it launches GC process: each entry in GC queue is deleted
and all resources are released. Such approach allows to avoid workload
spikes on session's disconnect (since on such event all statements must
be deallocated).
Each session is extended with local hash to map statement ids available
for it. That is, session is allowed to execute and deallocate only
statements which are previously prepared in scope of this session.
On the other hand, global hash makes it possible to share same prepared
statement object among different sessions.
Size of cache is regulated via box.cfg{sql_cache_size} parameter.

Part of #2592
---
 src/box/CMakeLists.txt          |   1 +
 src/box/box.cc                  |  26 ++++
 src/box/box.h                   |   3 +
 src/box/errcode.h               |   1 +
 src/box/lua/cfg.cc              |   9 ++
 src/box/lua/load_cfg.lua        |   3 +
 src/box/session.cc              |  35 +++++
 src/box/session.h               |  17 +++
 src/box/sql.c                   |   3 +
 src/box/sql_stmt_cache.c        | 289 ++++++++++++++++++++++++++++++++++++++++
 src/box/sql_stmt_cache.h        | 145 ++++++++++++++++++++
 test/app-tap/init_script.result |  37 ++---
 test/box/admin.result           |   2 +
 test/box/cfg.result             |   7 +
 test/box/cfg.test.lua           |   1 +
 test/box/misc.result            |   1 +
 16 files changed, 562 insertions(+), 18 deletions(-)
 create mode 100644 src/box/sql_stmt_cache.c
 create mode 100644 src/box/sql_stmt_cache.h

diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
index 5cd5cba81..763bc3a4c 100644
--- a/src/box/CMakeLists.txt
+++ b/src/box/CMakeLists.txt
@@ -126,6 +126,7 @@ add_library(box STATIC
     sql.c
     bind.c
     execute.c
+    sql_stmt_cache.c
     wal.c
     call.c
     merger.c
diff --git a/src/box/box.cc b/src/box/box.cc
index b119c927b..2b0cfa32d 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -74,6 +74,7 @@
 #include "call.h"
 #include "func.h"
 #include "sequence.h"
+#include "sql_stmt_cache.h"
 
 static char status[64] = "unknown";
 
@@ -599,6 +600,17 @@ box_check_vinyl_options(void)
 	}
 }
 
+static int
+box_check_sql_cache_size(int size)
+{
+	if (size < 0) {
+		diag_set(ClientError, ER_CFG, "sql_cache_size",
+			 "must be non-negative");
+		return -1;
+	}
+	return 0;
+}
+
 void
 box_check_config()
 {
@@ -620,6 +632,7 @@ box_check_config()
 	box_check_memtx_memory(cfg_geti64("memtx_memory"));
 	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
 	box_check_vinyl_options();
+	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
 }
 
 /*
@@ -886,6 +899,17 @@ box_set_net_msg_max(void)
 				IPROTO_FIBER_POOL_SIZE_FACTOR);
 }
 
+int
+box_set_prepared_stmt_cache_size(void)
+{
+	int cache_sz = cfg_geti("sql_cache_size");
+	if (box_check_sql_cache_size(cache_sz) != 0)
+		return -1;
+	if (sql_stmt_cache_set_size(cache_sz) != 0)
+		return -1;
+	return 0;
+}
+
 /* }}} configuration bindings */
 
 /**
@@ -2096,6 +2120,8 @@ box_cfg_xc(void)
 	box_check_instance_uuid(&instance_uuid);
 	box_check_replicaset_uuid(&replicaset_uuid);
 
+	if (box_set_prepared_stmt_cache_size() != 0)
+		diag_raise();
 	box_set_net_msg_max();
 	box_set_readahead();
 	box_set_too_long_threshold();
diff --git a/src/box/box.h b/src/box/box.h
index ccd527bd5..6806f1fc2 100644
--- a/src/box/box.h
+++ b/src/box/box.h
@@ -236,6 +236,9 @@ void box_set_replication_sync_timeout(void);
 void box_set_replication_skip_conflict(void);
 void box_set_net_msg_max(void);
 
+int
+box_set_prepared_stmt_cache_size(void);
+
 extern "C" {
 #endif /* defined(__cplusplus) */
 
diff --git a/src/box/errcode.h b/src/box/errcode.h
index c660b1c70..ee44f61b3 100644
--- a/src/box/errcode.h
+++ b/src/box/errcode.h
@@ -258,6 +258,7 @@ struct errcode_record {
 	/*203 */_(ER_BOOTSTRAP_READONLY,	"Trying to bootstrap a local read-only instance as master") \
 	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
 	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
+	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
 
 /*
  * !IMPORTANT! Please follow instructions at start of the file
diff --git a/src/box/lua/cfg.cc b/src/box/lua/cfg.cc
index 4884ce013..439af3cf9 100644
--- a/src/box/lua/cfg.cc
+++ b/src/box/lua/cfg.cc
@@ -274,6 +274,14 @@ lbox_cfg_set_net_msg_max(struct lua_State *L)
 	return 0;
 }
 
+static int
+lbox_set_prepared_stmt_cache_size(struct lua_State *L)
+{
+	if (box_set_prepared_stmt_cache_size() != 0)
+		luaT_error(L);
+	return 0;
+}
+
 static int
 lbox_cfg_set_worker_pool_threads(struct lua_State *L)
 {
@@ -378,6 +386,7 @@ box_lua_cfg_init(struct lua_State *L)
 		{"cfg_set_replication_sync_timeout", lbox_cfg_set_replication_sync_timeout},
 		{"cfg_set_replication_skip_conflict", lbox_cfg_set_replication_skip_conflict},
 		{"cfg_set_net_msg_max", lbox_cfg_set_net_msg_max},
+		{"cfg_set_sql_cache_size", lbox_set_prepared_stmt_cache_size},
 		{NULL, NULL}
 	};
 
diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua
index 85617c8f0..4463f989c 100644
--- a/src/box/lua/load_cfg.lua
+++ b/src/box/lua/load_cfg.lua
@@ -81,6 +81,7 @@ local default_cfg = {
     feedback_host         = "https://feedback.tarantool.io",
     feedback_interval     = 3600,
     net_msg_max           = 768,
+    sql_cache_size        = 5 * 1024 * 1024,
 }
 
 -- types of available options
@@ -144,6 +145,7 @@ local template_cfg = {
     feedback_host         = 'string',
     feedback_interval     = 'number',
     net_msg_max           = 'number',
+    sql_cache_size        = 'number',
 }
 
 local function normalize_uri(port)
@@ -250,6 +252,7 @@ local dynamic_cfg = {
     instance_uuid           = check_instance_uuid,
     replicaset_uuid         = check_replicaset_uuid,
     net_msg_max             = private.cfg_set_net_msg_max,
+    sql_cache_size          = private.cfg_set_sql_cache_size,
 }
 
 --
diff --git a/src/box/session.cc b/src/box/session.cc
index 461d1cf25..881318252 100644
--- a/src/box/session.cc
+++ b/src/box/session.cc
@@ -36,6 +36,7 @@
 #include "user.h"
 #include "error.h"
 #include "tt_static.h"
+#include "sql_stmt_cache.h"
 
 const char *session_type_strs[] = {
 	"background",
@@ -141,6 +142,7 @@ session_create(enum session_type type)
 	session_set_type(session, type);
 	session->sql_flags = default_flags;
 	session->sql_default_engine = SQL_STORAGE_ENGINE_MEMTX;
+	session->sql_stmts = NULL;
 
 	/* For on_connect triggers. */
 	credentials_create(&session->credentials, guest_user);
@@ -178,6 +180,38 @@ session_create_on_demand()
 	return s;
 }
 
+bool
+session_check_stmt_id(struct session *session, uint32_t stmt_id)
+{
+	if (session->sql_stmts == NULL)
+		return false;
+	mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL);
+	return i != mh_end(session->sql_stmts);
+}
+
+int
+session_add_stmt_id(struct session *session, uint32_t id)
+{
+	if (session->sql_stmts == NULL) {
+		session->sql_stmts = mh_i32ptr_new();
+		if (session->sql_stmts == NULL) {
+			diag_set(OutOfMemory, 0, "mh_i32ptr_new",
+				 "session stmt hash");
+			return -1;
+		}
+	}
+	return sql_session_stmt_hash_add_id(session->sql_stmts, id);
+}
+
+void
+session_remove_stmt_id(struct session *session, uint32_t stmt_id)
+{
+	assert(session->sql_stmts != NULL);
+	mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL);
+	assert(i != mh_end(session->sql_stmts));
+	mh_i32ptr_del(session->sql_stmts, i, NULL);
+}
+
 /**
  * To quickly switch to admin user when executing
  * on_connect/on_disconnect triggers in iproto.
@@ -227,6 +261,7 @@ session_destroy(struct session *session)
 	struct mh_i64ptr_node_t node = { session->id, NULL };
 	mh_i64ptr_remove(session_registry, &node, NULL);
 	credentials_destroy(&session->credentials);
+	sql_session_stmt_hash_erase(session->sql_stmts);
 	mempool_free(&session_pool, session);
 }
 
diff --git a/src/box/session.h b/src/box/session.h
index eff3d7a67..6dfc7cba5 100644
--- a/src/box/session.h
+++ b/src/box/session.h
@@ -101,6 +101,11 @@ struct session {
 	const struct session_vtab *vtab;
 	/** Session metadata. */
 	union session_meta meta;
+	/**
+	 * ID of statements prepared in current session.
+	 * This map is allocated on demand.
+	 */
+	struct mh_i32ptr_t *sql_stmts;
 	/** Session user id and global grants */
 	struct credentials credentials;
 	/** Trigger for fiber on_stop to cleanup created on-demand session */
@@ -267,6 +272,18 @@ session_storage_cleanup(int sid);
 struct session *
 session_create(enum session_type type);
 
+/** Return true if given statement id belongs to the session. */
+bool
+session_check_stmt_id(struct session *session, uint32_t stmt_id);
+
+/** Add prepared statement ID to the session hash. */
+int
+session_add_stmt_id(struct session *session, uint32_t stmt_id);
+
+/** Remove prepared statement ID from the session hash. */
+void
+session_remove_stmt_id(struct session *session, uint32_t stmt_id);
+
 /**
  * Destroy a session.
  * Must be called by the networking layer on disconnect.
diff --git a/src/box/sql.c b/src/box/sql.c
index f1df55571..455fabeef 100644
--- a/src/box/sql.c
+++ b/src/box/sql.c
@@ -54,6 +54,7 @@
 #include "iproto_constants.h"
 #include "fk_constraint.h"
 #include "mpstream.h"
+#include "sql_stmt_cache.h"
 
 static sql *db = NULL;
 
@@ -74,6 +75,8 @@ sql_init()
 	if (sql_init_db(&db) != 0)
 		panic("failed to initialize SQL subsystem");
 
+	sql_stmt_cache_init();
+
 	assert(db != NULL);
 }
 
diff --git a/src/box/sql_stmt_cache.c b/src/box/sql_stmt_cache.c
new file mode 100644
index 000000000..742e4135c
--- /dev/null
+++ b/src/box/sql_stmt_cache.c
@@ -0,0 +1,289 @@
+/*
+ * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include "sql_stmt_cache.h"
+
+#include "assoc.h"
+#include "error.h"
+#include "execute.h"
+#include "diag.h"
+
+static struct sql_stmt_cache sql_stmt_cache;
+
+void
+sql_stmt_cache_init()
+{
+	sql_stmt_cache.hash = mh_i32ptr_new();
+	if (sql_stmt_cache.hash == NULL)
+		panic("out of memory");
+	sql_stmt_cache.mem_quota = 0;
+	sql_stmt_cache.mem_used = 0;
+	rlist_create(&sql_stmt_cache.gc_queue);
+}
+
+static size_t
+sql_cache_entry_sizeof(struct sql_stmt *stmt)
+{
+	return sql_stmt_est_size(stmt) + sizeof(struct stmt_cache_entry);
+}
+
+static void
+sql_cache_entry_delete(struct stmt_cache_entry *entry)
+{
+	assert(entry->refs == 0);
+	assert(! sql_stmt_busy(entry->stmt));
+	sql_stmt_finalize(entry->stmt);
+	TRASH(entry);
+	free(entry);
+}
+
+/**
+ * Remove statement entry from cache: firstly delete from hash,
+ * than remove from LRU list and account cache size changes,
+ * finally release occupied memory.
+ */
+static void
+sql_stmt_cache_delete(struct stmt_cache_entry *entry)
+{
+	if (sql_stmt_cache.last_found == entry)
+		sql_stmt_cache.last_found = NULL;
+	rlist_del(&entry->link);
+	sql_stmt_cache.mem_used -= sql_cache_entry_sizeof(entry->stmt);
+	sql_cache_entry_delete(entry);
+}
+
+static struct stmt_cache_entry *
+stmt_cache_find_entry(uint32_t stmt_id)
+{
+	if (sql_stmt_cache.last_found != NULL) {
+		const char *sql_str =
+			sql_stmt_query_str(sql_stmt_cache.last_found->stmt);
+		uint32_t last_stmt_id = sql_stmt_calculate_id(sql_str,
+							 strlen(sql_str));
+		if (last_stmt_id == stmt_id)
+			return sql_stmt_cache.last_found;
+		/* Fallthrough to slow hash search. */
+	}
+	struct mh_i32ptr_t *hash = sql_stmt_cache.hash;
+	mh_int_t stmt = mh_i32ptr_find(hash, stmt_id, NULL);
+	if (stmt == mh_end(hash))
+		return NULL;
+	struct stmt_cache_entry *entry = mh_i32ptr_node(hash, stmt)->val;
+	if (entry == NULL)
+		return NULL;
+	sql_stmt_cache.last_found = entry;
+	return entry;
+}
+
+static void
+sql_stmt_cache_gc()
+{
+	struct stmt_cache_entry *entry, *next;
+	rlist_foreach_entry_safe(entry, &sql_stmt_cache.gc_queue, link, next)
+		sql_stmt_cache_delete(entry);
+	assert(rlist_empty(&sql_stmt_cache.gc_queue));
+}
+
+/**
+ * Allocate new cache entry containing given prepared statement.
+ * Add it to the LRU cache list. Account cache size enlargement.
+ */
+static struct stmt_cache_entry *
+sql_cache_entry_new(struct sql_stmt *stmt)
+{
+	struct stmt_cache_entry *entry = malloc(sizeof(*entry));
+	if (entry == NULL) {
+		diag_set(OutOfMemory, sizeof(*entry), "malloc",
+			 "struct stmt_cache_entry");
+		return NULL;
+	}
+	entry->stmt = stmt;
+	entry->refs = 0;
+	return entry;
+}
+
+/**
+ * Return true if used memory (accounting new entry) for SQL
+ * prepared statement cache does not exceed the limit.
+ */
+static bool
+sql_cache_check_new_entry_size(size_t size)
+{
+	return (sql_stmt_cache.mem_used + size <= sql_stmt_cache.mem_quota);
+}
+
+static void
+sql_stmt_cache_entry_unref(struct stmt_cache_entry *entry)
+{
+	assert((int64_t)entry->refs - 1 >= 0);
+	if (--entry->refs == 0) {
+		/*
+		 * Remove entry from hash and add it to gc queue.
+		 * Resources are to be released in the nearest
+		 * GC cycle (see sql_stmt_cache_insert()).
+		 */
+		struct sql_stmt_cache *cache = &sql_stmt_cache;
+		const char *sql_str = sql_stmt_query_str(entry->stmt);
+		uint32_t stmt_id = sql_stmt_calculate_id(sql_str,
+							 strlen(sql_str));
+		mh_int_t i = mh_i32ptr_find(cache->hash, stmt_id, NULL);
+		assert(i != mh_end(cache->hash));
+		mh_i32ptr_del(cache->hash, i, NULL);
+		rlist_add(&sql_stmt_cache.gc_queue, &entry->link);
+		if (sql_stmt_cache.last_found == entry)
+			sql_stmt_cache.last_found = NULL;
+	}
+}
+
+void
+sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash)
+{
+	if (hash == NULL)
+		return;
+	mh_int_t i;
+	struct stmt_cache_entry *entry;
+	mh_foreach(hash, i) {
+		entry = (struct stmt_cache_entry *)
+			mh_i32ptr_node(hash, i)->val;
+		sql_stmt_cache_entry_unref(entry);
+	}
+	mh_i32ptr_delete(hash);
+}
+
+int
+sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id)
+{
+	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
+	const struct mh_i32ptr_node_t id_node = { stmt_id, entry };
+	struct mh_i32ptr_node_t *old_node = NULL;
+	mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL);
+	if (i == mh_end(hash)) {
+		diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node");
+		return -1;
+	}
+	assert(old_node == NULL);
+	entry->refs++;
+	return 0;
+}
+
+uint32_t
+sql_stmt_calculate_id(const char *sql_str, size_t len)
+{
+	return mh_strn_hash(sql_str, len);
+}
+
+void
+sql_stmt_unref(uint32_t stmt_id)
+{
+	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
+	assert(entry != NULL);
+	sql_stmt_cache_entry_unref(entry);
+}
+
+int
+sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt)
+{
+	const char *sql_str = sql_stmt_query_str(old_stmt);
+	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
+	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
+	uint32_t ref_count = entry->refs;
+	sql_stmt_cache_delete(entry);
+	if (sql_stmt_cache_insert(new_stmt) != 0) {
+		sql_stmt_finalize(new_stmt);
+		return -1;
+	}
+	/* Restore reference counter. */
+	entry = stmt_cache_find_entry(stmt_id);
+	entry->refs = ref_count;
+	return 0;
+}
+
+int
+sql_stmt_cache_insert(struct sql_stmt *stmt)
+{
+	assert(stmt != NULL);
+	struct sql_stmt_cache *cache = &sql_stmt_cache;
+	size_t new_entry_size = sql_cache_entry_sizeof(stmt);
+
+	if (! sql_cache_check_new_entry_size(new_entry_size))
+		sql_stmt_cache_gc();
+	/*
+	 * Test memory limit again. Raise an error if it is
+	 * still overcrowded.
+	 */
+	if (! sql_cache_check_new_entry_size(new_entry_size)) {
+		diag_set(ClientError, ER_SQL_PREPARE, "Memory limit for SQL "\
+			"prepared statements has been reached. Please, deallocate "\
+			"active statements or increase SQL cache size.");
+		return -1;
+	}
+	struct mh_i32ptr_t *hash = cache->hash;
+	struct stmt_cache_entry *entry = sql_cache_entry_new(stmt);
+	if (entry == NULL)
+		return -1;
+	const char *sql_str = sql_stmt_query_str(stmt);
+	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
+	assert(sql_stmt_cache_find(stmt_id) == NULL);
+	const struct mh_i32ptr_node_t id_node = { stmt_id, entry };
+	struct mh_i32ptr_node_t *old_node = NULL;
+	mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL);
+	if (i == mh_end(hash)) {
+		sql_cache_entry_delete(entry);
+		diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node");
+		return -1;
+	}
+	assert(old_node == NULL);
+	sql_stmt_cache.mem_used += sql_cache_entry_sizeof(stmt);
+	return 0;
+}
+
+struct sql_stmt *
+sql_stmt_cache_find(uint32_t stmt_id)
+{
+	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
+	if (entry == NULL)
+		return NULL;
+	return entry->stmt;
+}
+
+int
+sql_stmt_cache_set_size(size_t size)
+{
+	if (sql_stmt_cache.mem_used > size)
+		sql_stmt_cache_gc();
+	if (sql_stmt_cache.mem_used > size) {
+		diag_set(ClientError, ER_SQL_PREPARE, "Can't reduce memory "\
+			 "limit for SQL prepared statements: please, deallocate "\
+			 "active statements");
+		return -1;
+	}
+	sql_stmt_cache.mem_quota = size;
+	return 0;
+}
diff --git a/src/box/sql_stmt_cache.h b/src/box/sql_stmt_cache.h
new file mode 100644
index 000000000..f3935a27f
--- /dev/null
+++ b/src/box/sql_stmt_cache.h
@@ -0,0 +1,145 @@
+#ifndef INCLUDES_PREP_STMT_H
+#define INCLUDES_PREP_STMT_H
+/*
+ * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ *    copyright notice, this list of conditions and the
+ *    following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ *    copyright notice, this list of conditions and the following
+ *    disclaimer in the documentation and/or other materials
+ *    provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include <stdint.h>
+#include <stdio.h>
+
+#include "small/rlist.h"
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+struct sql_stmt;
+struct mh_i64ptr_t;
+
+struct stmt_cache_entry {
+	/** Prepared statement itself. */
+	struct sql_stmt *stmt;
+	/**
+	 * Link to the next entry. All statements are to be
+	 * evicted on the next gc cycle.
+	 */
+	struct rlist link;
+	/**
+	 * Reference counter. If it is == 0, entry gets
+	 * into GC queue.
+	 */
+	uint32_t refs;
+};
+
+/**
+ * Global prepared statements holder.
+ */
+struct sql_stmt_cache {
+	/** Size of memory currently occupied by prepared statements. */
+	size_t mem_used;
+	/** Max memory size that can be used for cache. */
+	size_t mem_quota;
+	/** Query id -> struct stmt_cahce_entry hash.*/
+	struct mh_i32ptr_t *hash;
+	/**
+	 * After deallocation statements are not deleted, but
+	 * moved to this list. GC process is triggered only
+	 * when memory limit has reached. It allows to reduce
+	 * workload on session's disconnect.
+	 */
+	struct rlist gc_queue;
+	/**
+	 * Last result of sql_stmt_cache_find() invocation.
+	 * Since during processing prepared statement it
+	 * may require to find the same statement several
+	 * times.
+	 */
+	struct stmt_cache_entry *last_found;
+};
+
+/**
+ * Initialize global cache for prepared statements. Called once
+ * during database setup (in sql_init()).
+ */
+void
+sql_stmt_cache_init();
+
+/**
+ * Erase session local hash: unref statements belong to this
+ * session and deallocate hash itself.
+ * @hash is assumed to be member of struct session @sql_stmts.
+ */
+void
+sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash);
+
+/**
+ * Add entry corresponding to prepared statement with given ID
+ * to session-local hash and increase its ref counter.
+ * @hash is assumed to be member of struct session @sql_stmts.
+ */
+int
+sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id);
+
+/**
+ * Prepared statement ID is supposed to be hash value
+ * of the original SQL query string.
+ */
+uint32_t
+sql_stmt_calculate_id(const char *sql_str, size_t len);
+
+/** Unref prepared statement entry in global holder. */
+void
+sql_stmt_unref(uint32_t stmt_id);
+
+int
+sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt);
+
+/**
+ * Save prepared statement to the prepared statement cache.
+ * Account cache size change. If the cache is full (i.e. memory
+ * quota is exceeded) diag error is raised. In case of success
+ * return id of prepared statement via output parameter @id.
+ */
+int
+sql_stmt_cache_insert(struct sql_stmt *stmt);
+
+/** Find entry by SQL string. In case of search fails it returns NULL. */
+struct sql_stmt *
+sql_stmt_cache_find(uint32_t stmt_id);
+
+
+/** Set prepared cache size limit. */
+int
+sql_stmt_cache_set_size(size_t size);
+
+#if defined(__cplusplus)
+} /* extern "C" { */
+#endif
+
+#endif
diff --git a/test/app-tap/init_script.result b/test/app-tap/init_script.result
index 799297ba0..551a0bbeb 100644
--- a/test/app-tap/init_script.result
+++ b/test/app-tap/init_script.result
@@ -31,24 +31,25 @@ box.cfg
 26	replication_sync_timeout:300
 27	replication_timeout:1
 28	slab_alloc_factor:1.05
-29	strip_core:true
-30	too_long_threshold:0.5
-31	vinyl_bloom_fpr:0.05
-32	vinyl_cache:134217728
-33	vinyl_dir:.
-34	vinyl_max_tuple_size:1048576
-35	vinyl_memory:134217728
-36	vinyl_page_size:8192
-37	vinyl_read_threads:1
-38	vinyl_run_count_per_level:2
-39	vinyl_run_size_ratio:3.5
-40	vinyl_timeout:60
-41	vinyl_write_threads:4
-42	wal_dir:.
-43	wal_dir_rescan_delay:2
-44	wal_max_size:268435456
-45	wal_mode:write
-46	worker_pool_threads:4
+29	sql_cache_size:5242880
+30	strip_core:true
+31	too_long_threshold:0.5
+32	vinyl_bloom_fpr:0.05
+33	vinyl_cache:134217728
+34	vinyl_dir:.
+35	vinyl_max_tuple_size:1048576
+36	vinyl_memory:134217728
+37	vinyl_page_size:8192
+38	vinyl_read_threads:1
+39	vinyl_run_count_per_level:2
+40	vinyl_run_size_ratio:3.5
+41	vinyl_timeout:60
+42	vinyl_write_threads:4
+43	wal_dir:.
+44	wal_dir_rescan_delay:2
+45	wal_max_size:268435456
+46	wal_mode:write
+47	worker_pool_threads:4
 --
 -- Test insert from detached fiber
 --
diff --git a/test/box/admin.result b/test/box/admin.result
index 6126f3a97..852c1cde8 100644
--- a/test/box/admin.result
+++ b/test/box/admin.result
@@ -83,6 +83,8 @@ cfg_filter(box.cfg)
     - 1
   - - slab_alloc_factor
     - 1.05
+  - - sql_cache_size
+    - 5242880
   - - strip_core
     - true
   - - too_long_threshold
diff --git a/test/box/cfg.result b/test/box/cfg.result
index 5370bb870..331f5e986 100644
--- a/test/box/cfg.result
+++ b/test/box/cfg.result
@@ -71,6 +71,8 @@ cfg_filter(box.cfg)
  |     - 1
  |   - - slab_alloc_factor
  |     - 1.05
+ |   - - sql_cache_size
+ |     - 5242880
  |   - - strip_core
  |     - true
  |   - - too_long_threshold
@@ -170,6 +172,8 @@ cfg_filter(box.cfg)
  |     - 1
  |   - - slab_alloc_factor
  |     - 1.05
+ |   - - sql_cache_size
+ |     - 5242880
  |   - - strip_core
  |     - true
  |   - - too_long_threshold
@@ -315,6 +319,9 @@ box.cfg{memtx_memory = box.cfg.memtx_memory}
 box.cfg{vinyl_memory = box.cfg.vinyl_memory}
  | ---
  | ...
+box.cfg{sql_cache_size = box.cfg.sql_cache_size}
+ | ---
+ | ...
 
 --------------------------------------------------------------------------------
 -- Test of default cfg options
diff --git a/test/box/cfg.test.lua b/test/box/cfg.test.lua
index 56ccb6767..e6a90d770 100644
--- a/test/box/cfg.test.lua
+++ b/test/box/cfg.test.lua
@@ -51,6 +51,7 @@ box.cfg{replicaset_uuid = '12345678-0123-5678-1234-abcdefabcdef'}
 
 box.cfg{memtx_memory = box.cfg.memtx_memory}
 box.cfg{vinyl_memory = box.cfg.vinyl_memory}
+box.cfg{sql_cache_size = box.cfg.sql_cache_size}
 
 --------------------------------------------------------------------------------
 -- Test of default cfg options
diff --git a/test/box/misc.result b/test/box/misc.result
index d2a20307a..7e5d28b70 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -554,6 +554,7 @@ t;
   203: box.error.BOOTSTRAP_READONLY
   204: box.error.SQL_FUNC_WRONG_RET_COUNT
   205: box.error.FUNC_INVALID_RETURN_TYPE
+  206: box.error.SQL_PREPARE
 ...
 test_run:cmd("setopt delimiter ''");
 ---
-- 
2.15.1

[Tarantool-patches] [PATCH v3 18/20] box: introduce prepared statements

[Nikita Pettik]

This patch introduces local prepared statements. Support of prepared
statements in IProto protocol and netbox is added in the next patch.

Prepared statement is an opaque instance of SQL Virtual Machine. It can
be executed several times without necessity of query recompilation. To
achieve this one can use box.prepare(...) function. It takes string of
SQL query to be prepared; returns extended set of meta-information
including statement's ID, parameter's types and names, types and names
of columns of the resulting set, count of parameters to be bound.  Lua
object representing result of :prepare() invocation also features two
methods - :execute() and :unprepare(). They correspond to
box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
automatically substitute string of prepared statement to be executed.
Statements are held in prepared statement cache - for details see
previous commit.  After schema changes all prepared statement located in
cache are considered to be expired - they must be re-prepared by
separate :prepare() call (or be invalidated with :unrepare()).

Two sessions can share one prepared statements. But in current
implementation if statement is executed by one session, another one is
not able to use it and will compile it from scratch and than execute.

SQL cache memory limit is regulated by box{sql_cache_size} which can be
set dynamically. However, it can be set to the value which is less than
the size of current free space in cache (since otherwise some statements
can disappear from cache).

Part of #2592
---
 src/box/errcode.h          |   1 +
 src/box/execute.c          | 114 ++++++++
 src/box/execute.h          |  16 +-
 src/box/lua/execute.c      | 213 +++++++++++++-
 src/box/lua/execute.h      |   2 +-
 src/box/lua/init.c         |   2 +-
 src/box/sql/prepare.c      |   9 -
 test/box/misc.result       |   3 +
 test/sql/engine.cfg        |   3 +
 test/sql/prepared.result   | 687 +++++++++++++++++++++++++++++++++++++++++++++
 test/sql/prepared.test.lua | 240 ++++++++++++++++
 11 files changed, 1267 insertions(+), 23 deletions(-)
 create mode 100644 test/sql/prepared.result
 create mode 100644 test/sql/prepared.test.lua

diff --git a/src/box/errcode.h b/src/box/errcode.h
index ee44f61b3..9e12f3a31 100644
--- a/src/box/errcode.h
+++ b/src/box/errcode.h
@@ -259,6 +259,7 @@ struct errcode_record {
 	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
 	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
 	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
+	/*207 */_(ER_WRONG_QUERY_ID,		"Prepared statement with id %u does not exist") \
 
 /*
  * !IMPORTANT! Please follow instructions at start of the file
diff --git a/src/box/execute.c b/src/box/execute.c
index 3bc4988b7..09224c23a 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -30,6 +30,7 @@
  */
 #include "execute.h"
 
+#include "assoc.h"
 #include "bind.h"
 #include "iproto_constants.h"
 #include "sql/sqlInt.h"
@@ -45,6 +46,8 @@
 #include "tuple.h"
 #include "sql/vdbe.h"
 #include "box/lua/execute.h"
+#include "box/sql_stmt_cache.h"
+#include "session.h"
 
 const char *sql_info_key_strs[] = {
 	"row_count",
@@ -413,6 +416,81 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 	return 0;
 }
 
+static bool
+sql_stmt_check_schema_version(struct sql_stmt *stmt)
+{
+	return sql_stmt_schema_version(stmt) == box_schema_version();
+}
+
+/**
+ * Re-compile statement and refresh global prepared statement
+ * cache with the newest value.
+ */
+static int
+sql_reprepare(struct sql_stmt **stmt)
+{
+	const char *sql_str = sql_stmt_query_str(*stmt);
+	struct sql_stmt *new_stmt;
+	if (sql_stmt_compile(sql_str, strlen(sql_str), NULL,
+			     &new_stmt, NULL) != 0)
+		return -1;
+	if (sql_stmt_cache_update(*stmt, new_stmt) != 0)
+		return -1;
+	*stmt = new_stmt;
+	return 0;
+}
+
+/**
+ * Compile statement and save it to the global holder;
+ * update session hash with prepared statement ID (if
+ * it's not already there).
+ */
+int
+sql_prepare(const char *sql, int len, struct port *port)
+{
+	uint32_t stmt_id = sql_stmt_calculate_id(sql, len);
+	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
+	if (stmt == NULL) {
+		if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
+			return -1;
+		if (sql_stmt_cache_insert(stmt) != 0) {
+			sql_stmt_finalize(stmt);
+			return -1;
+		}
+	} else {
+		if (! sql_stmt_check_schema_version(stmt)) {
+			if (sql_reprepare(&stmt) != 0)
+				return -1;
+		}
+	}
+	assert(stmt != NULL);
+	/* Add id to the list of available statements in session. */
+	if (!session_check_stmt_id(current_session(), stmt_id))
+		session_add_stmt_id(current_session(), stmt_id);
+	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
+					   DQL_PREPARE : DML_PREPARE;
+	port_sql_create(port, stmt, format, false);
+
+	return 0;
+}
+
+/**
+ * Deallocate prepared statement from current session:
+ * remove its ID from session-local hash and unref entry
+ * in global holder.
+ */
+int
+sql_unprepare(uint32_t stmt_id)
+{
+	if (!session_check_stmt_id(current_session(), stmt_id)) {
+		diag_set(ClientError, ER_WRONG_QUERY_ID, stmt_id);
+		return -1;
+	}
+	session_remove_stmt_id(current_session(), stmt_id);
+	sql_stmt_unref(stmt_id);
+	return 0;
+}
+
 /**
  * Execute prepared SQL statement.
  *
@@ -450,6 +528,42 @@ sql_execute(struct sql_stmt *stmt, struct port *port, struct region *region)
 	return 0;
 }
 
+int
+sql_execute_prepared(uint32_t stmt_id, const struct sql_bind *bind,
+		     uint32_t bind_count, struct port *port,
+		     struct region *region)
+{
+
+	if (!session_check_stmt_id(current_session(), stmt_id)) {
+		diag_set(ClientError, ER_WRONG_QUERY_ID, stmt_id);
+		return -1;
+	}
+	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
+	assert(stmt != NULL);
+	if (! sql_stmt_check_schema_version(stmt)) {
+		diag_set(ClientError, ER_SQL_EXECUTE, "statement has expired");
+		return -1;
+	}
+	if (sql_stmt_busy(stmt)) {
+		const char *sql_str = sql_stmt_query_str(stmt);
+		return sql_prepare_and_execute(sql_str, strlen(sql_str), bind,
+					       bind_count, port, region);
+	}
+	if (sql_bind(stmt, bind, bind_count) != 0)
+		return -1;
+	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
+					       DQL_EXECUTE : DML_EXECUTE;
+	port_sql_create(port, stmt, format, false);
+	if (sql_execute(stmt, port, region) != 0) {
+		port_destroy(port);
+		sql_stmt_reset(stmt);
+		return -1;
+	}
+	sql_stmt_reset(stmt);
+
+	return 0;
+}
+
 int
 sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
 			uint32_t bind_count, struct port *port,
diff --git a/src/box/execute.h b/src/box/execute.h
index 61c8e0281..5e2327f4a 100644
--- a/src/box/execute.h
+++ b/src/box/execute.h
@@ -62,6 +62,14 @@ extern const char *sql_info_key_strs[];
 struct region;
 struct sql_bind;
 
+int
+sql_unprepare(uint32_t stmt_id);
+
+int
+sql_execute_prepared(uint32_t query_id, const struct sql_bind *bind,
+		     uint32_t bind_count, struct port *port,
+		     struct region *region);
+
 /**
  * Prepare and execute an SQL statement.
  * @param sql SQL statement.
@@ -135,13 +143,11 @@ sql_stmt_busy(const struct sql_stmt *stmt);
  * Prepare (compile into VDBE byte-code) statement.
  *
  * @param sql UTF-8 encoded SQL statement.
- * @param length Length of @param sql in bytes.
- * @param[out] stmt A pointer to the prepared statement.
- * @param[out] sql_tail End of parsed string.
+ * @param len Length of @param sql in bytes.
+ * @param port Port to store request response.
  */
 int
-sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
-	    const char **sql_tail);
+sql_prepare(const char *sql, int len, struct port *port);
 
 #if defined(__cplusplus)
 } /* extern "C" { */
diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
index b164ffcaf..6cb1f5db9 100644
--- a/src/box/lua/execute.c
+++ b/src/box/lua/execute.c
@@ -5,6 +5,8 @@
 #include "box/port.h"
 #include "box/execute.h"
 #include "box/bind.h"
+#include "box/sql_stmt_cache.h"
+#include "box/schema.h"
 
 /**
  * Serialize a description of the prepared statement.
@@ -38,6 +40,101 @@ lua_sql_get_metadata(struct sql_stmt *stmt, struct lua_State *L,
 	}
 }
 
+static inline void
+lua_sql_get_params_metadata(struct sql_stmt *stmt, struct lua_State *L)
+{
+	int bind_count = sql_bind_parameter_count(stmt);
+	lua_createtable(L, bind_count, 0);
+	for (int i = 0; i < bind_count; ++i) {
+		lua_createtable(L, 0, 2);
+		const char *name = sql_bind_parameter_name(stmt, i);
+		if (name == NULL)
+			name = "?";
+		const char *type = "ANY";
+		lua_pushstring(L, name);
+		lua_setfield(L, -2, "name");
+		lua_pushstring(L, type);
+		lua_setfield(L, -2, "type");
+		lua_rawseti(L, -2, i + 1);
+	}
+}
+
+/** Forward declaration to avoid code movement. */
+static int
+lbox_execute(struct lua_State *L);
+
+/**
+ * Prepare SQL statement: compile it and save to the cache.
+ * In fact it is wrapper around box.execute() which unfolds
+ * it to box.execute(stmt.query_id).
+ */
+static int
+lbox_execute_prepared(struct lua_State *L)
+{
+	int top = lua_gettop(L);
+
+	if ((top != 1 && top != 2) || ! lua_istable(L, 1))
+		return luaL_error(L, "Usage: statement:execute([, params])");
+	lua_getfield(L, 1, "stmt_id");
+	if (!lua_isnumber(L, -1))
+		return luaL_error(L, "Query id is expected to be numeric");
+	lua_remove(L, 1);
+	if (top == 2) {
+		/*
+		 * Stack state (before remove operation):
+		 * 1 Prepared statement object (Lua table)
+		 * 2 Bindings (Lua table)
+		 * 3 Statement ID(fetched from PS table) - top of stack
+		 *
+		 * We should make it suitable to pass arguments to
+		 * lbox_execute(), i.e. after manipulations stack
+		 * should look like:
+		 * 1 Statement ID
+		 * 2 Bindings - top of stack
+		 * Since there's no swap operation, we firstly remove
+		 * PS object, then copy table of values to be bound to
+		 * the top of stack (push), and finally remove original
+		 * bindings from stack.
+		 */
+		lua_pushvalue(L, 1);
+		lua_remove(L, 1);
+	}
+	return lbox_execute(L);
+}
+
+/**
+ * Unprepare statement: remove it from prepared statements cache.
+ * This function can be called in two ways: as member of prepared
+ * statement handle (stmt:unprepare()) or as box.unprepare(stmt_id).
+ */
+static int
+lbox_unprepare(struct lua_State *L)
+{
+	int top = lua_gettop(L);
+
+	if (top != 1 || (! lua_istable(L, 1) && ! lua_isnumber(L, 1))) {
+		return luaL_error(L, "Usage: statement:unprepare() or "\
+				     "box.unprepare(stmt_id)");
+	}
+	lua_Integer stmt_id;
+	if (lua_istable(L, 1)) {
+		lua_getfield(L, -1, "stmt_id");
+		if (! lua_isnumber(L, -1)) {
+			return luaL_error(L, "Statement id is expected "\
+					     "to be numeric");
+		}
+		stmt_id = lua_tointeger(L, -1);
+		lua_pop(L, 1);
+	} else {
+		stmt_id = lua_tonumber(L, 1);
+	}
+	if (stmt_id < 0)
+		return luaL_error(L, "Statement id can't be negative");
+	if (sql_unprepare((uint32_t) stmt_id) != 0)
+		return luaT_push_nil_and_error(L);
+	return 0;
+}
+
 void
 port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
 {
@@ -82,7 +179,66 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
 		}
 		break;
 	}
-	default: {
+	case DQL_PREPARE: {
+		/* Format is following:
+		 * stmt_id,
+		 * param_count,
+		 * params {name, type},
+		 * metadata {name, type}
+		 * execute(), unprepare()
+		 */
+		lua_createtable(L, 0, 6);
+		/* query_id */
+		const char *sql_str = sql_stmt_query_str(port_sql->stmt);
+		luaL_pushuint64(L, sql_stmt_calculate_id(sql_str,
+							 strlen(sql_str)));
+		lua_setfield(L, -2, "stmt_id");
+		/* param_count */
+		luaL_pushuint64(L, sql_bind_parameter_count(stmt));
+		lua_setfield(L, -2, "param_count");
+		/* params map */
+		lua_sql_get_params_metadata(stmt, L);
+		lua_setfield(L, -2, "params");
+		/* metadata */
+		lua_sql_get_metadata(stmt, L, sql_column_count(stmt));
+		lua_setfield(L, -2, "metadata");
+		/* execute function */
+		lua_pushcfunction(L, lbox_execute_prepared);
+		lua_setfield(L, -2, "execute");
+		/* unprepare function */
+		lua_pushcfunction(L, lbox_unprepare);
+		lua_setfield(L, -2, "unprepare");
+		break;
+	}
+	case DML_PREPARE : {
+		assert(((struct port_tuple *) port)->size == 0);
+		/* Format is following:
+		 * stmt_id,
+		 * param_count,
+		 * params {name, type},
+		 * execute(), unprepare()
+		 */
+		lua_createtable(L, 0, 5);
+		/* query_id */
+		const char *sql_str = sql_stmt_query_str(port_sql->stmt);
+		luaL_pushuint64(L, sql_stmt_calculate_id(sql_str,
+							 strlen(sql_str)));
+		lua_setfield(L, -2, "stmt_id");
+		/* param_count */
+		luaL_pushuint64(L, sql_bind_parameter_count(stmt));
+		lua_setfield(L, -2, "param_count");
+		/* params map */
+		lua_sql_get_params_metadata(stmt, L);
+		lua_setfield(L, -2, "params");
+		/* execute function */
+		lua_pushcfunction(L, lbox_execute_prepared);
+		lua_setfield(L, -2, "execute");
+		/* unprepare function */
+		lua_pushcfunction(L, lbox_unprepare);
+		lua_setfield(L, -2, "unprepare");
+		break;
+	}
+	default:{
 		unreachable();
 	}
 	}
@@ -253,9 +409,8 @@ lbox_execute(struct lua_State *L)
 	int top = lua_gettop(L);
 
 	if ((top != 1 && top != 2) || ! lua_isstring(L, 1))
-		return luaL_error(L, "Usage: box.execute(sqlstring[, params])");
-
-	const char *sql = lua_tolstring(L, 1, &length);
+		return luaL_error(L, "Usage: box.execute(sqlstring[, params]) "
+				  "or box.execute(stmt_id[, params])");
 
 	if (top == 2) {
 		if (! lua_istable(L, 2))
@@ -264,9 +419,44 @@ lbox_execute(struct lua_State *L)
 		if (bind_count < 0)
 			return luaT_push_nil_and_error(L);
 	}
+	/*
+	 * lua_isstring() returns true for numeric values as well,
+	 * so test explicit type instead.
+	 */
+	if (lua_type(L, 1) == LUA_TSTRING) {
+		const char *sql = lua_tolstring(L, 1, &length);
+		if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
+					    &fiber()->gc) != 0)
+			return luaT_push_nil_and_error(L);
+	} else {
+		assert(lua_type(L, 1) == LUA_TNUMBER);
+		lua_Integer query_id = lua_tointeger(L, 1);
+		if (query_id < 0)
+			return luaL_error(L, "Statement id can't be negative");
+		if (sql_execute_prepared(query_id, bind, bind_count, &port,
+					 &fiber()->gc) != 0)
+			return luaT_push_nil_and_error(L);
+	}
+	port_dump_lua(&port, L, false);
+	port_destroy(&port);
+	return 1;
+}
 
-	if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
-				    &fiber()->gc) != 0)
+/**
+ * Prepare SQL statement: compile it and save to the cache.
+ */
+static int
+lbox_prepare(struct lua_State *L)
+{
+	size_t length;
+	struct port port;
+	int top = lua_gettop(L);
+
+	if ((top != 1 && top != 2) || ! lua_isstring(L, 1))
+		return luaL_error(L, "Usage: box.prepare(sqlstring)");
+
+	const char *sql = lua_tolstring(L, 1, &length);
+	if (sql_prepare(sql, length, &port) != 0)
 		return luaT_push_nil_and_error(L);
 	port_dump_lua(&port, L, false);
 	port_destroy(&port);
@@ -274,11 +464,20 @@ lbox_execute(struct lua_State *L)
 }
 
 void
-box_lua_execute_init(struct lua_State *L)
+box_lua_sql_init(struct lua_State *L)
 {
 	lua_getfield(L, LUA_GLOBALSINDEX, "box");
 	lua_pushstring(L, "execute");
 	lua_pushcfunction(L, lbox_execute);
 	lua_settable(L, -3);
+
+	lua_pushstring(L, "prepare");
+	lua_pushcfunction(L, lbox_prepare);
+	lua_settable(L, -3);
+
+	lua_pushstring(L, "unprepare");
+	lua_pushcfunction(L, lbox_unprepare);
+	lua_settable(L, -3);
+
 	lua_pop(L, 1);
 }
diff --git a/src/box/lua/execute.h b/src/box/lua/execute.h
index 23e193fa4..bafd67615 100644
--- a/src/box/lua/execute.h
+++ b/src/box/lua/execute.h
@@ -66,6 +66,6 @@ lua_sql_bind_list_decode(struct lua_State *L, struct sql_bind **out_bind,
 			 int idx);
 
 void
-box_lua_execute_init(struct lua_State *L);
+box_lua_sql_init(struct lua_State *L);
 
 #endif /* INCLUDES_TARANTOOL_LUA_EXECUTE_H */
diff --git a/src/box/lua/init.c b/src/box/lua/init.c
index 7ffed409d..7be520e09 100644
--- a/src/box/lua/init.c
+++ b/src/box/lua/init.c
@@ -314,7 +314,7 @@ box_lua_init(struct lua_State *L)
 	box_lua_ctl_init(L);
 	box_lua_session_init(L);
 	box_lua_xlog_init(L);
-	box_lua_execute_init(L);
+	box_lua_sql_init(L);
 	luaopen_net_box(L);
 	lua_pop(L, 1);
 	tarantool_lua_console_init(L);
diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
index 73d6866a4..5b1c6c581 100644
--- a/src/box/sql/prepare.c
+++ b/src/box/sql/prepare.c
@@ -193,15 +193,6 @@ sqlReprepare(Vdbe * p)
 	return 0;
 }
 
-int
-sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
-	    const char **sql_tail)
-{
-	int rc = sql_stmt_compile(sql, length, 0, stmt, sql_tail);
-	assert(rc == 0 || stmt == NULL || *stmt == NULL);
-	return rc;
-}
-
 void
 sql_parser_create(struct Parse *parser, struct sql *db, uint32_t sql_flags)
 {
diff --git a/test/box/misc.result b/test/box/misc.result
index 7e5d28b70..90923f28e 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -73,6 +73,7 @@ t
   - on_commit
   - on_rollback
   - once
+  - prepare
   - priv
   - rollback
   - rollback_to_savepoint
@@ -86,6 +87,7 @@ t
   - space
   - stat
   - tuple
+  - unprepare
 ...
 t = nil
 ---
@@ -555,6 +557,7 @@ t;
   204: box.error.SQL_FUNC_WRONG_RET_COUNT
   205: box.error.FUNC_INVALID_RETURN_TYPE
   206: box.error.SQL_PREPARE
+  207: box.error.WRONG_QUERY_ID
 ...
 test_run:cmd("setopt delimiter ''");
 ---
diff --git a/test/sql/engine.cfg b/test/sql/engine.cfg
index 284c42082..a1b4b0fc5 100644
--- a/test/sql/engine.cfg
+++ b/test/sql/engine.cfg
@@ -9,6 +9,9 @@
         "remote": {"remote": "true"},
         "local": {"remote": "false"}
     },
+    "prepared.test.lua": {
+        "local": {"remote": "false"}
+    },
     "*": {
         "memtx": {"engine": "memtx"},
         "vinyl": {"engine": "vinyl"}
diff --git a/test/sql/prepared.result b/test/sql/prepared.result
new file mode 100644
index 000000000..bd37cfdd7
--- /dev/null
+++ b/test/sql/prepared.result
@@ -0,0 +1,687 @@
+-- test-run result file version 2
+remote = require('net.box')
+ | ---
+ | ...
+test_run = require('test_run').new()
+ | ---
+ | ...
+fiber = require('fiber')
+ | ---
+ | ...
+
+-- Wrappers to make remote and local execution interface return
+-- same result pattern.
+--
+test_run:cmd("setopt delimiter ';'")
+ | ---
+ | - true
+ | ...
+execute = function(...)
+    local res, err = box.execute(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+ | ---
+ | ...
+prepare = function(...)
+    local res, err = box.prepare(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+ | ---
+ | ...
+unprepare = function(...)
+    local res, err = box.unprepare(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+ | ---
+ | ...
+
+test_run:cmd("setopt delimiter ''");
+ | ---
+ | - true
+ | ...
+
+-- Test local interface and basic capabilities of prepared statements.
+--
+execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
+ | ---
+ | - row_count: 1
+ | ...
+space = box.space.TEST
+ | ---
+ | ...
+space:replace{1, 2, '3'}
+ | ---
+ | - [1, 2, '3']
+ | ...
+space:replace{4, 5, '6'}
+ | ---
+ | - [4, 5, '6']
+ | ...
+space:replace{7, 8.5, '9'}
+ | ---
+ | - [7, 8.5, '9']
+ | ...
+s, e = prepare("SELECT * FROM test WHERE id = ? AND a = ?;")
+ | ---
+ | ...
+assert(e == nil)
+ | ---
+ | - true
+ | ...
+assert(s ~= nil)
+ | ---
+ | - true
+ | ...
+s.stmt_id
+ | ---
+ | - 3603193623
+ | ...
+s.metadata
+ | ---
+ | - - name: ID
+ |     type: integer
+ |   - name: A
+ |     type: number
+ |   - name: B
+ |     type: string
+ | ...
+s.params
+ | ---
+ | - - name: '?'
+ |     type: ANY
+ |   - name: '?'
+ |     type: ANY
+ | ...
+s.param_count
+ | ---
+ | - 2
+ | ...
+execute(s.stmt_id, {1, 2})
+ | ---
+ | - metadata:
+ |   - name: ID
+ |     type: integer
+ |   - name: A
+ |     type: number
+ |   - name: B
+ |     type: string
+ |   rows:
+ |   - [1, 2, '3']
+ | ...
+execute(s.stmt_id, {1, 3})
+ | ---
+ | - metadata:
+ |   - name: ID
+ |     type: integer
+ |   - name: A
+ |     type: number
+ |   - name: B
+ |     type: string
+ |   rows: []
+ | ...
+s:execute({1, 2})
+ | ---
+ | - metadata:
+ |   - name: ID
+ |     type: integer
+ |   - name: A
+ |     type: number
+ |   - name: B
+ |     type: string
+ |   rows:
+ |   - [1, 2, '3']
+ | ...
+s:execute({1, 3})
+ | ---
+ | - metadata:
+ |   - name: ID
+ |     type: integer
+ |   - name: A
+ |     type: number
+ |   - name: B
+ |     type: string
+ |   rows: []
+ | ...
+s:unprepare()
+ | ---
+ | ...
+
+-- Test preparation of different types of queries.
+-- Let's start from DDL. It doesn't make much sense since
+-- any prepared DDL statement can be executed once, but
+-- anyway make sure that no crashes occur.
+--
+s = prepare("CREATE INDEX i1 ON test(a)")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("DROP INDEX i1 ON test;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("CREATE VIEW v AS SELECT * FROM test;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("DROP VIEW v;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("ALTER TABLE test RENAME TO test1")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 0
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+box.execute("CREATE TABLE test2 (id INT PRIMARY KEY);")
+ | ---
+ | - row_count: 1
+ | ...
+s = prepare("ALTER TABLE test2 ADD CONSTRAINT fk1 FOREIGN KEY (id) REFERENCES test2")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+box.space.TEST2:drop()
+ | ---
+ | ...
+
+s = prepare("CREATE TRIGGER tr1 INSERT ON test1 FOR EACH ROW BEGIN DELETE FROM test1; END;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("DROP TRIGGER tr1;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("DROP TABLE test1;")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - error: 'Failed to execute SQL statement: statement has expired'
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- DQL
+--
+execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
+ | ---
+ | - row_count: 1
+ | ...
+space = box.space.TEST
+ | ---
+ | ...
+space:replace{1, 2, '3'}
+ | ---
+ | - [1, 2, '3']
+ | ...
+space:replace{4, 5, '6'}
+ | ---
+ | - [4, 5, '6']
+ | ...
+space:replace{7, 8.5, '9'}
+ | ---
+ | - [7, 8.5, '9']
+ | ...
+_ = prepare("SELECT a FROM test WHERE b = '3';")
+ | ---
+ | ...
+s = prepare("SELECT a FROM test WHERE b = '3';")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [2]
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [2]
+ | ...
+s:execute()
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [2]
+ | ...
+s:execute()
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [2]
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
+ | ---
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - metadata:
+ |   - name: count(*)
+ |     type: integer
+ |   - name: count(a - 3)
+ |     type: integer
+ |   - name: max(b)
+ |     type: scalar
+ |   - name: abs(id)
+ |     type: number
+ |   rows:
+ |   - [1, 1, '3', 1]
+ | ...
+execute(s.stmt_id)
+ | ---
+ | - metadata:
+ |   - name: count(*)
+ |     type: integer
+ |   - name: count(a - 3)
+ |     type: integer
+ |   - name: max(b)
+ |     type: scalar
+ |   - name: abs(id)
+ |     type: number
+ |   rows:
+ |   - [1, 1, '3', 1]
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- Let's try something a bit more complicated. For instance recursive
+-- query displaying Mandelbrot set.
+--
+s = prepare([[WITH RECURSIVE \
+                  xaxis(x) AS (VALUES(-2.0) UNION ALL SELECT x+0.05 FROM xaxis WHERE x<1.2), \
+                  yaxis(y) AS (VALUES(-1.0) UNION ALL SELECT y+0.1 FROM yaxis WHERE y<1.0), \
+                  m(iter, cx, cy, x, y) AS ( \
+                      SELECT 0, x, y, 0.0, 0.0 FROM xaxis, yaxis \
+                      UNION ALL \
+                      SELECT iter+1, cx, cy, x*x-y*y + cx, 2.0*x*y + cy FROM m \
+                          WHERE (x*x + y*y) < 4.0 AND iter<28), \
+                      m2(iter, cx, cy) AS ( \
+                          SELECT max(iter), cx, cy FROM m GROUP BY cx, cy), \
+                      a(t) AS ( \
+                          SELECT group_concat( substr(' .+*#', 1+LEAST(iter/7,4), 1), '') \
+                              FROM m2 GROUP BY cy) \
+                  SELECT group_concat(TRIM(TRAILING FROM t),x'0a') FROM a;]])
+ | ---
+ | ...
+
+res = execute(s.stmt_id)
+ | ---
+ | ...
+res.metadata
+ | ---
+ | - - name: group_concat(TRIM(TRAILING FROM t),x'0a')
+ |     type: string
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- Workflow with bindings is still the same.
+--
+s = prepare("SELECT a FROM test WHERE b = ?;")
+ | ---
+ | ...
+execute(s.stmt_id, {'6'})
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [5]
+ | ...
+execute(s.stmt_id, {'9'})
+ | ---
+ | - metadata:
+ |   - name: A
+ |     type: number
+ |   rows:
+ |   - [8.5]
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- DML
+s = prepare("INSERT INTO test VALUES (?, ?, ?);")
+ | ---
+ | ...
+execute(s.stmt_id, {5, 6, '7'})
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id, {6, 10, '7'})
+ | ---
+ | - row_count: 1
+ | ...
+execute(s.stmt_id, {9, 11, '7'})
+ | ---
+ | - row_count: 1
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- EXPLAIN and PRAGMA work fine as well.
+--
+s1 = prepare("EXPLAIN SELECT a FROM test WHERE b = '3';")
+ | ---
+ | ...
+res = execute(s1.stmt_id)
+ | ---
+ | ...
+res.metadata
+ | ---
+ | - - name: addr
+ |     type: integer
+ |   - name: opcode
+ |     type: text
+ |   - name: p1
+ |     type: integer
+ |   - name: p2
+ |     type: integer
+ |   - name: p3
+ |     type: integer
+ |   - name: p4
+ |     type: text
+ |   - name: p5
+ |     type: text
+ |   - name: comment
+ |     type: text
+ | ...
+assert(res.rows ~= nil)
+ | ---
+ | - true
+ | ...
+
+s2 = prepare("EXPLAIN QUERY PLAN SELECT a FROM test WHERE b = '3';")
+ | ---
+ | ...
+res = execute(s2.stmt_id)
+ | ---
+ | ...
+res.metadata
+ | ---
+ | - - name: selectid
+ |     type: integer
+ |   - name: order
+ |     type: integer
+ |   - name: from
+ |     type: integer
+ |   - name: detail
+ |     type: text
+ | ...
+assert(res.rows ~= nil)
+ | ---
+ | - true
+ | ...
+
+s3 = prepare("PRAGMA count_changes;")
+ | ---
+ | ...
+execute(s3.stmt_id)
+ | ---
+ | - metadata:
+ |   - name: defer_foreign_keys
+ |     type: integer
+ |   rows:
+ |   - [0]
+ | ...
+
+unprepare(s2.stmt_id)
+ | ---
+ | - null
+ | ...
+unprepare(s3.stmt_id)
+ | ---
+ | - null
+ | ...
+unprepare(s1.stmt_id)
+ | ---
+ | - null
+ | ...
+
+-- Setting cache size to 0 is possible only in case if
+-- there's no any prepared statements right now .
+--
+box.cfg{sql_cache_size = 0}
+ | ---
+ | ...
+prepare("SELECT a FROM test;")
+ | ---
+ | - error: 'Failed to prepare SQL statement: Memory limit for SQL prepared statements
+ |     has been reached. Please, deallocate active statements or increase SQL cache size.'
+ | ...
+box.cfg{sql_cache_size = 0}
+ | ---
+ | ...
+
+-- Still with small size everything should work.
+--
+box.cfg{sql_cache_size = 1500}
+ | ---
+ | ...
+
+test_run:cmd("setopt delimiter ';'");
+ | ---
+ | - true
+ | ...
+ok = nil
+res = nil
+_ = fiber.create(function()
+    for i = 1, 5 do
+        pcall(prepare, string.format("SELECT * FROM test WHERE a = %d;", i))
+    end
+    ok, res = pcall(prepare, "SELECT * FROM test WHERE b = '3';")
+end);
+ | ---
+ | ...
+while ok == nil do fiber.sleep(0.00001) end;
+ | ---
+ | ...
+assert(ok == false);
+ | ---
+ | - true
+ | ...
+res;
+ | ---
+ | - 'Failed to prepare SQL statement: Memory limit for SQL prepared statements has been
+ |   reached. Please, deallocate active statements or increase SQL cache size.'
+ | ...
+
+-- Check that after fiber is dead, its session gets rid of
+-- all prepared statements.
+--
+box.cfg{sql_cache_size = 0};
+ | ---
+ | ...
+box.cfg{sql_cache_size = 3000};
+ | ---
+ | ...
+
+-- Make sure that if prepared statement is busy (is executed
+-- right now), prepared statement is not used, i.e. statement
+-- is compiled from scratch, executed and finilized.
+--
+box.schema.func.create('SLEEP', {language = 'Lua',
+    body = 'function () fiber.sleep(0.1) return 1 end',
+    exports = {'LUA', 'SQL'}});
+ | ---
+ | ...
+
+s = prepare("SELECT id, SLEEP() FROM test;");
+ | ---
+ | ...
+assert(s ~= nil);
+ | ---
+ | - true
+ | ...
+
+function implicit_yield()
+    prepare("SELECT id, SLEEP() FROM test;")
+    execute("SELECT id, SLEEP() FROM test;")
+end;
+ | ---
+ | ...
+
+f1 = fiber.new(implicit_yield)
+f2 = fiber.new(implicit_yield)
+f1:set_joinable(true)
+f2:set_joinable(true)
+
+f1:join();
+ | ---
+ | ...
+f2:join();
+ | ---
+ | - true
+ | ...
+
+unprepare(s.stmt_id);
+ | ---
+ | - null
+ | ...
+
+test_run:cmd("setopt delimiter ''");
+ | ---
+ | - true
+ | ...
+
+box.cfg{sql_cache_size = 5 * 1024 * 1024}
+ | ---
+ | ...
+box.space.TEST:drop()
+ | ---
+ | ...
+box.schema.func.drop('SLEEP')
+ | ---
+ | ...
diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
new file mode 100644
index 000000000..49d2fb3ae
--- /dev/null
+++ b/test/sql/prepared.test.lua
@@ -0,0 +1,240 @@
+remote = require('net.box')
+test_run = require('test_run').new()
+fiber = require('fiber')
+
+-- Wrappers to make remote and local execution interface return
+-- same result pattern.
+--
+test_run:cmd("setopt delimiter ';'")
+execute = function(...)
+    local res, err = box.execute(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+prepare = function(...)
+    local res, err = box.prepare(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+unprepare = function(...)
+    local res, err = box.unprepare(...)
+    if err ~= nil then
+        error(err)
+    end
+    return res
+end;
+
+test_run:cmd("setopt delimiter ''");
+
+-- Test local interface and basic capabilities of prepared statements.
+--
+execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
+space = box.space.TEST
+space:replace{1, 2, '3'}
+space:replace{4, 5, '6'}
+space:replace{7, 8.5, '9'}
+s, e = prepare("SELECT * FROM test WHERE id = ? AND a = ?;")
+assert(e == nil)
+assert(s ~= nil)
+s.stmt_id
+s.metadata
+s.params
+s.param_count
+execute(s.stmt_id, {1, 2})
+execute(s.stmt_id, {1, 3})
+s:execute({1, 2})
+s:execute({1, 3})
+s:unprepare()
+
+-- Test preparation of different types of queries.
+-- Let's start from DDL. It doesn't make much sense since
+-- any prepared DDL statement can be executed once, but
+-- anyway make sure that no crashes occur.
+--
+s = prepare("CREATE INDEX i1 ON test(a)")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("DROP INDEX i1 ON test;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("CREATE VIEW v AS SELECT * FROM test;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("DROP VIEW v;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("ALTER TABLE test RENAME TO test1")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+box.execute("CREATE TABLE test2 (id INT PRIMARY KEY);")
+s = prepare("ALTER TABLE test2 ADD CONSTRAINT fk1 FOREIGN KEY (id) REFERENCES test2")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+box.space.TEST2:drop()
+
+s = prepare("CREATE TRIGGER tr1 INSERT ON test1 FOR EACH ROW BEGIN DELETE FROM test1; END;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("DROP TRIGGER tr1;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+s = prepare("DROP TABLE test1;")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+-- DQL
+--
+execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
+space = box.space.TEST
+space:replace{1, 2, '3'}
+space:replace{4, 5, '6'}
+space:replace{7, 8.5, '9'}
+_ = prepare("SELECT a FROM test WHERE b = '3';")
+s = prepare("SELECT a FROM test WHERE b = '3';")
+execute(s.stmt_id)
+execute(s.stmt_id)
+s:execute()
+s:execute()
+unprepare(s.stmt_id)
+
+s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
+execute(s.stmt_id)
+execute(s.stmt_id)
+unprepare(s.stmt_id)
+
+-- Let's try something a bit more complicated. For instance recursive
+-- query displaying Mandelbrot set.
+--
+s = prepare([[WITH RECURSIVE \
+                  xaxis(x) AS (VALUES(-2.0) UNION ALL SELECT x+0.05 FROM xaxis WHERE x<1.2), \
+                  yaxis(y) AS (VALUES(-1.0) UNION ALL SELECT y+0.1 FROM yaxis WHERE y<1.0), \
+                  m(iter, cx, cy, x, y) AS ( \
+                      SELECT 0, x, y, 0.0, 0.0 FROM xaxis, yaxis \
+                      UNION ALL \
+                      SELECT iter+1, cx, cy, x*x-y*y + cx, 2.0*x*y + cy FROM m \
+                          WHERE (x*x + y*y) < 4.0 AND iter<28), \
+                      m2(iter, cx, cy) AS ( \
+                          SELECT max(iter), cx, cy FROM m GROUP BY cx, cy), \
+                      a(t) AS ( \
+                          SELECT group_concat( substr(' .+*#', 1+LEAST(iter/7,4), 1), '') \
+                              FROM m2 GROUP BY cy) \
+                  SELECT group_concat(TRIM(TRAILING FROM t),x'0a') FROM a;]])
+
+res = execute(s.stmt_id)
+res.metadata
+unprepare(s.stmt_id)
+
+-- Workflow with bindings is still the same.
+--
+s = prepare("SELECT a FROM test WHERE b = ?;")
+execute(s.stmt_id, {'6'})
+execute(s.stmt_id, {'9'})
+unprepare(s.stmt_id)
+
+-- DML
+s = prepare("INSERT INTO test VALUES (?, ?, ?);")
+execute(s.stmt_id, {5, 6, '7'})
+execute(s.stmt_id, {6, 10, '7'})
+execute(s.stmt_id, {9, 11, '7'})
+unprepare(s.stmt_id)
+
+-- EXPLAIN and PRAGMA work fine as well.
+--
+s1 = prepare("EXPLAIN SELECT a FROM test WHERE b = '3';")
+res = execute(s1.stmt_id)
+res.metadata
+assert(res.rows ~= nil)
+
+s2 = prepare("EXPLAIN QUERY PLAN SELECT a FROM test WHERE b = '3';")
+res = execute(s2.stmt_id)
+res.metadata
+assert(res.rows ~= nil)
+
+s3 = prepare("PRAGMA count_changes;")
+execute(s3.stmt_id)
+
+unprepare(s2.stmt_id)
+unprepare(s3.stmt_id)
+unprepare(s1.stmt_id)
+
+-- Setting cache size to 0 is possible only in case if
+-- there's no any prepared statements right now .
+--
+box.cfg{sql_cache_size = 0}
+prepare("SELECT a FROM test;")
+box.cfg{sql_cache_size = 0}
+
+-- Still with small size everything should work.
+--
+box.cfg{sql_cache_size = 1500}
+
+test_run:cmd("setopt delimiter ';'");
+ok = nil
+res = nil
+_ = fiber.create(function()
+    for i = 1, 5 do
+        pcall(prepare, string.format("SELECT * FROM test WHERE a = %d;", i))
+    end
+    ok, res = pcall(prepare, "SELECT * FROM test WHERE b = '3';")
+end);
+while ok == nil do fiber.sleep(0.00001) end;
+assert(ok == false);
+res;
+
+-- Check that after fiber is dead, its session gets rid of
+-- all prepared statements.
+--
+box.cfg{sql_cache_size = 0};
+box.cfg{sql_cache_size = 3000};
+
+-- Make sure that if prepared statement is busy (is executed
+-- right now), prepared statement is not used, i.e. statement
+-- is compiled from scratch, executed and finilized.
+--
+box.schema.func.create('SLEEP', {language = 'Lua',
+    body = 'function () fiber.sleep(0.1) return 1 end',
+    exports = {'LUA', 'SQL'}});
+
+s = prepare("SELECT id, SLEEP() FROM test;");
+assert(s ~= nil);
+
+function implicit_yield()
+    prepare("SELECT id, SLEEP() FROM test;")
+    execute("SELECT id, SLEEP() FROM test;")
+end;
+
+f1 = fiber.new(implicit_yield)
+f2 = fiber.new(implicit_yield)
+f1:set_joinable(true)
+f2:set_joinable(true)
+
+f1:join();
+f2:join();
+
+unprepare(s.stmt_id);
+
+test_run:cmd("setopt delimiter ''");
+
+box.cfg{sql_cache_size = 5 * 1024 * 1024}
+box.space.TEST:drop()
+box.schema.func.drop('SLEEP')
-- 
2.15.1

[Tarantool-patches] [PATCH v3 19/20] netbox: introduce prepared statements

[Nikita Pettik]

This patch introduces support of prepared statements in IProto
protocol. To achieve this new IProto command is added - IPROTO_PREPARE
(key is 0x13). It is sent with one of two mandatory keys:
IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
and assumes integer value). Depending on body it means to prepare or
unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
corresponds to the count of parameters to be bound).

Part of #2592
---
 src/box/execute.c          |  83 ++++++++++++++++++++++++++++++
 src/box/iproto.cc          |  68 +++++++++++++++++++++----
 src/box/iproto_constants.c |   7 ++-
 src/box/iproto_constants.h |   5 ++
 src/box/lua/net_box.c      |  98 +++++++++++++++++++++++++++++++++--
 src/box/lua/net_box.lua    |  27 ++++++++++
 src/box/xrow.c             |  23 +++++++--
 src/box/xrow.h             |   4 +-
 test/box/misc.result       |   1 +
 test/sql/engine.cfg        |   1 +
 test/sql/iproto.result     |   2 +-
 test/sql/prepared.result   | 124 ++++++++++++++++++++++++++-------------------
 test/sql/prepared.test.lua |  76 +++++++++++++++++++--------
 13 files changed, 420 insertions(+), 99 deletions(-)

diff --git a/src/box/execute.c b/src/box/execute.c
index 09224c23a..7174d0d41 100644
--- a/src/box/execute.c
+++ b/src/box/execute.c
@@ -328,6 +328,68 @@ sql_get_metadata(struct sql_stmt *stmt, struct obuf *out, int column_count)
 	return 0;
 }
 
+static inline int
+sql_get_params_metadata(struct sql_stmt *stmt, struct obuf *out)
+{
+	int bind_count = sql_bind_parameter_count(stmt);
+	int size = mp_sizeof_uint(IPROTO_BIND_METADATA) +
+		   mp_sizeof_array(bind_count);
+	char *pos = (char *) obuf_alloc(out, size);
+	if (pos == NULL) {
+		diag_set(OutOfMemory, size, "obuf_alloc", "pos");
+		return -1;
+	}
+	pos = mp_encode_uint(pos, IPROTO_BIND_METADATA);
+	pos = mp_encode_array(pos, bind_count);
+	for (int i = 0; i < bind_count; ++i) {
+		size_t size = mp_sizeof_map(2) +
+			      mp_sizeof_uint(IPROTO_FIELD_NAME) +
+			      mp_sizeof_uint(IPROTO_FIELD_TYPE);
+		const char *name = sql_bind_parameter_name(stmt, i);
+		if (name == NULL)
+			name = "?";
+		const char *type = "ANY";
+		size += mp_sizeof_str(strlen(name));
+		size += mp_sizeof_str(strlen(type));
+		char *pos = (char *) obuf_alloc(out, size);
+		if (pos == NULL) {
+			diag_set(OutOfMemory, size, "obuf_alloc", "pos");
+			return -1;
+		}
+		pos = mp_encode_map(pos, 2);
+		pos = mp_encode_uint(pos, IPROTO_FIELD_NAME);
+		pos = mp_encode_str(pos, name, strlen(name));
+		pos = mp_encode_uint(pos, IPROTO_FIELD_TYPE);
+		pos = mp_encode_str(pos, type, strlen(type));
+	}
+	return 0;
+}
+
+static int
+sql_get_prepare_common_keys(struct sql_stmt *stmt, struct obuf *out, int keys)
+{
+	const char *sql_str = sql_stmt_query_str(stmt);
+	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
+	int size = mp_sizeof_map(keys) +
+		   mp_sizeof_uint(IPROTO_STMT_ID) +
+		   mp_sizeof_uint(stmt_id) +
+		   mp_sizeof_uint(IPROTO_BIND_COUNT) +
+		   mp_sizeof_uint(sql_bind_parameter_count(stmt));
+	char *pos = (char *) obuf_alloc(out, size);
+	if (pos == NULL) {
+		diag_set(OutOfMemory, size, "obuf_alloc", "pos");
+		return -1;
+	}
+	pos = mp_encode_map(pos, keys);
+	pos = mp_encode_uint(pos, IPROTO_STMT_ID);
+	pos = mp_encode_uint(pos, stmt_id);
+	pos = mp_encode_uint(pos, IPROTO_BIND_COUNT);
+	pos = mp_encode_uint(pos, sql_bind_parameter_count(stmt));
+	if (sql_get_params_metadata(stmt, out) != 0)
+		return -1;
+	return 0;
+}
+
 static int
 port_sql_dump_msgpack(struct port *port, struct obuf *out)
 {
@@ -409,6 +471,27 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
 		}
 		break;
 	}
+	case DQL_PREPARE: {
+		/* Format is following:
+		 * query_id,
+		 * param_count,
+		 * params {name, type},
+		 * metadata {name, type}
+		 */
+		int keys = 4;
+		if (sql_get_prepare_common_keys(stmt, out, keys) != 0)
+			return -1;
+		return sql_get_metadata(stmt, out, sql_column_count(stmt));
+	}
+	case DML_PREPARE: {
+		/* Format is following:
+		 * query_id,
+		 * param_count,
+		 * params {name, type},
+		 */
+		int keys = 3;
+		return sql_get_prepare_common_keys(stmt, out, keys);
+		}
 	default: {
 		unreachable();
 	}
diff --git a/src/box/iproto.cc b/src/box/iproto.cc
index c39b8e7bf..fac94658a 100644
--- a/src/box/iproto.cc
+++ b/src/box/iproto.cc
@@ -178,7 +178,7 @@ struct iproto_msg
 		struct call_request call;
 		/** Authentication request. */
 		struct auth_request auth;
-		/* SQL request, if this is the EXECUTE request. */
+		/* SQL request, if this is the EXECUTE/PREPARE request. */
 		struct sql_request sql;
 		/** In case of iproto parse error, saved diagnostics. */
 		struct diag diag;
@@ -1209,6 +1209,7 @@ static const struct cmsg_hop *dml_route[IPROTO_TYPE_STAT_MAX] = {
 	call_route,                             /* IPROTO_CALL */
 	sql_route,                              /* IPROTO_EXECUTE */
 	NULL,                                   /* IPROTO_NOP */
+	sql_route,                              /* IPROTO_PREPARE */
 };
 
 static const struct cmsg_hop join_route[] = {
@@ -1264,6 +1265,7 @@ iproto_msg_decode(struct iproto_msg *msg, const char **pos, const char *reqend,
 		cmsg_init(&msg->base, call_route);
 		break;
 	case IPROTO_EXECUTE:
+	case IPROTO_PREPARE:
 		if (xrow_decode_sql(&msg->header, &msg->sql) != 0)
 			goto error;
 		cmsg_init(&msg->base, sql_route);
@@ -1710,23 +1712,64 @@ tx_process_sql(struct cmsg *m)
 	int bind_count = 0;
 	const char *sql;
 	uint32_t len;
+	bool is_unprepare = false;
 
 	tx_fiber_init(msg->connection->session, msg->header.sync);
 
 	if (tx_check_schema(msg->header.schema_version))
 		goto error;
-	assert(msg->header.type == IPROTO_EXECUTE);
+	assert(msg->header.type == IPROTO_EXECUTE ||
+	       msg->header.type == IPROTO_PREPARE);
 	tx_inject_delay();
 	if (msg->sql.bind != NULL) {
 		bind_count = sql_bind_list_decode(msg->sql.bind, &bind);
 		if (bind_count < 0)
 			goto error;
 	}
-	sql = msg->sql.sql_text;
-	sql = mp_decode_str(&sql, &len);
-	if (sql_prepare_and_execute(sql, len, bind, bind_count, &port,
-				    &fiber()->gc) != 0)
-		goto error;
+	/*
+	 * There are four options:
+	 * 1. Prepare SQL query (IPROTO_PREPARE + SQL string);
+	 * 2. Unprepare SQL query (IPROTO_PREPARE + stmt id);
+	 * 3. Execute SQL query (IPROTO_EXECUTE + SQL string);
+	 * 4. Execute prepared query (IPROTO_EXECUTE + stmt id).
+	 */
+	if (msg->header.type == IPROTO_EXECUTE) {
+		if (msg->sql.sql_text != NULL) {
+			assert(msg->sql.stmt_id == NULL);
+			sql = msg->sql.sql_text;
+			sql = mp_decode_str(&sql, &len);
+			if (sql_prepare_and_execute(sql, len, bind, bind_count,
+						    &port, &fiber()->gc) != 0)
+				goto error;
+		} else {
+			assert(msg->sql.sql_text == NULL);
+			assert(msg->sql.stmt_id != NULL);
+			sql = msg->sql.stmt_id;
+			uint32_t stmt_id = mp_decode_uint(&sql);
+			if (sql_execute_prepared(stmt_id, bind, bind_count,
+						 &port, &fiber()->gc) != 0)
+				goto error;
+		}
+	} else {
+		/* IPROTO_PREPARE */
+		if (msg->sql.sql_text != NULL) {
+			assert(msg->sql.stmt_id == NULL);
+			sql = msg->sql.sql_text;
+			sql = mp_decode_str(&sql, &len);
+			if (sql_prepare(sql, len, &port) != 0)
+				goto error;
+		} else {
+			/* UNPREPARE */
+			assert(msg->sql.sql_text == NULL);
+			assert(msg->sql.stmt_id != NULL);
+			sql = msg->sql.stmt_id;
+			uint32_t stmt_id = mp_decode_uint(&sql);
+			if (sql_unprepare(stmt_id) != 0)
+				goto error;
+			is_unprepare = true;
+		}
+	}
+
 	/*
 	 * Take an obuf only after execute(). Else the buffer can
 	 * become out of date during yield.
@@ -1738,12 +1781,15 @@ tx_process_sql(struct cmsg *m)
 		port_destroy(&port);
 		goto error;
 	}
-	if (port_dump_msgpack(&port, out) != 0) {
+	/* Nothing to dump in case of UNPREPARE request. */
+	if (! is_unprepare) {
+		if (port_dump_msgpack(&port, out) != 0) {
+			port_destroy(&port);
+			obuf_rollback_to_svp(out, &header_svp);
+			goto error;
+		}
 		port_destroy(&port);
-		obuf_rollback_to_svp(out, &header_svp);
-		goto error;
 	}
-	port_destroy(&port);
 	iproto_reply_sql(out, &header_svp, msg->header.sync, schema_version);
 	iproto_wpos_create(&msg->wpos, out);
 	return;
diff --git a/src/box/iproto_constants.c b/src/box/iproto_constants.c
index 09ded1ecb..029d9888c 100644
--- a/src/box/iproto_constants.c
+++ b/src/box/iproto_constants.c
@@ -107,6 +107,7 @@ const char *iproto_type_strs[] =
 	"CALL",
 	"EXECUTE",
 	NULL, /* NOP */
+	"PREPARE",
 };
 
 #define bit(c) (1ULL<<IPROTO_##c)
@@ -124,6 +125,7 @@ const uint64_t iproto_body_key_map[IPROTO_TYPE_STAT_MAX] = {
 	0,                                                     /* CALL */
 	0,                                                     /* EXECUTE */
 	0,                                                     /* NOP */
+	0,                                                     /* PREPARE */
 };
 #undef bit
 
@@ -179,8 +181,8 @@ const char *iproto_key_strs[IPROTO_KEY_MAX] = {
 	"data",             /* 0x30 */
 	"error",            /* 0x31 */
 	"metadata",         /* 0x32 */
-	NULL,               /* 0x33 */
-	NULL,               /* 0x34 */
+	"bind meta",        /* 0x33 */
+	"bind count",       /* 0x34 */
 	NULL,               /* 0x35 */
 	NULL,               /* 0x36 */
 	NULL,               /* 0x37 */
@@ -195,6 +197,7 @@ const char *iproto_key_strs[IPROTO_KEY_MAX] = {
 	"SQL text",         /* 0x40 */
 	"SQL bind",         /* 0x41 */
 	"SQL info",         /* 0x42 */
+	"stmt id",          /* 0x43 */
 };
 
 const char *vy_page_info_key_strs[VY_PAGE_INFO_KEY_MAX] = {
diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h
index 5e8a7d483..34d0f49c6 100644
--- a/src/box/iproto_constants.h
+++ b/src/box/iproto_constants.h
@@ -110,6 +110,8 @@ enum iproto_key {
 	 * ]
 	 */
 	IPROTO_METADATA = 0x32,
+	IPROTO_BIND_METADATA = 0x33,
+	IPROTO_BIND_COUNT = 0x34,
 
 	/* Leave a gap between response keys and SQL keys. */
 	IPROTO_SQL_TEXT = 0x40,
@@ -120,6 +122,7 @@ enum iproto_key {
 	 * }
 	 */
 	IPROTO_SQL_INFO = 0x42,
+	IPROTO_STMT_ID = 0x43,
 	IPROTO_KEY_MAX
 };
 
@@ -203,6 +206,8 @@ enum iproto_type {
 	IPROTO_EXECUTE = 11,
 	/** No operation. Treated as DML, used to bump LSN. */
 	IPROTO_NOP = 12,
+	/** Prepare SQL statement. */
+	IPROTO_PREPARE = 13,
 	/** The maximum typecode used for box.stat() */
 	IPROTO_TYPE_STAT_MAX,
 
diff --git a/src/box/lua/net_box.c b/src/box/lua/net_box.c
index 001af95dc..aa8a15e30 100644
--- a/src/box/lua/net_box.c
+++ b/src/box/lua/net_box.c
@@ -570,10 +570,16 @@ netbox_encode_execute(lua_State *L)
 
 	mpstream_encode_map(&stream, 3);
 
-	size_t len;
-	const char *query = lua_tolstring(L, 3, &len);
-	mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
-	mpstream_encode_strn(&stream, query, len);
+	if (lua_type(L, 3) == LUA_TNUMBER) {
+		uint32_t query_id = lua_tointeger(L, 3);
+		mpstream_encode_uint(&stream, IPROTO_STMT_ID);
+		mpstream_encode_uint(&stream, query_id);
+	} else {
+		size_t len;
+		const char *query = lua_tolstring(L, 3, &len);
+		mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
+		mpstream_encode_strn(&stream, query, len);
+	}
 
 	mpstream_encode_uint(&stream, IPROTO_SQL_BIND);
 	luamp_encode_tuple(L, cfg, &stream, 4);
@@ -585,6 +591,32 @@ netbox_encode_execute(lua_State *L)
 	return 0;
 }
 
+static int
+netbox_encode_prepare(lua_State *L)
+{
+	if (lua_gettop(L) < 3)
+		return luaL_error(L, "Usage: netbox.encode_prepare(ibuf, "\
+				     "sync, query)");
+	struct mpstream stream;
+	size_t svp = netbox_prepare_request(L, &stream, IPROTO_PREPARE);
+
+	mpstream_encode_map(&stream, 1);
+
+	if (lua_type(L, 3) == LUA_TNUMBER) {
+		uint32_t query_id = lua_tointeger(L, 3);
+		mpstream_encode_uint(&stream, IPROTO_STMT_ID);
+		mpstream_encode_uint(&stream, query_id);
+	} else {
+		size_t len;
+		const char *query = lua_tolstring(L, 3, &len);
+		mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
+		mpstream_encode_strn(&stream, query, len);
+	};
+
+	netbox_encode_request(&stream, svp);
+	return 0;
+}
+
 /**
  * Decode IPROTO_DATA into tuples array.
  * @param L Lua stack to push result on.
@@ -752,6 +784,62 @@ netbox_decode_execute(struct lua_State *L)
 	return 2;
 }
 
+static int
+netbox_decode_prepare(struct lua_State *L)
+{
+	uint32_t ctypeid;
+	const char *data = *(const char **)luaL_checkcdata(L, 1, &ctypeid);
+	assert(mp_typeof(*data) == MP_MAP);
+	uint32_t map_size = mp_decode_map(&data);
+	int stmt_id_idx = 0, meta_idx = 0, bind_meta_idx = 0,
+	    bind_count_idx = 0;
+	uint32_t stmt_id = 0;
+	for (uint32_t i = 0; i < map_size; ++i) {
+		uint32_t key = mp_decode_uint(&data);
+		switch(key) {
+		case IPROTO_STMT_ID: {
+			stmt_id = mp_decode_uint(&data);
+			luaL_pushuint64(L, stmt_id);
+			stmt_id_idx = i - map_size;
+			break;
+		}
+		case IPROTO_METADATA: {
+			netbox_decode_metadata(L, &data);
+			meta_idx = i - map_size;
+			break;
+		}
+		case IPROTO_BIND_METADATA: {
+			netbox_decode_metadata(L, &data);
+			bind_meta_idx = i - map_size;
+			break;
+		}
+		default: {
+			assert(key == IPROTO_BIND_COUNT);
+			uint32_t bind_count = mp_decode_uint(&data);
+			luaL_pushuint64(L, bind_count);
+			bind_count_idx = i - map_size;
+			break;
+		}}
+	}
+	/* These fields must be present in response. */
+	assert(stmt_id_idx * bind_meta_idx * bind_count_idx != 0);
+	/* General meta is presented only in DQL responses. */
+	lua_createtable(L, 0, meta_idx != 0 ? 4 : 3);
+	lua_pushvalue(L, stmt_id_idx - 1);
+	lua_setfield(L, -2, "stmt_id");
+	lua_pushvalue(L, bind_count_idx - 1);
+	lua_setfield(L, -2, "param_count");
+	lua_pushvalue(L, bind_meta_idx - 1);
+	lua_setfield(L, -2, "params");
+	if (meta_idx != 0) {
+		lua_pushvalue(L, meta_idx - 1);
+		lua_setfield(L, -2, "metadata");
+	}
+
+	*(const char **)luaL_pushcdata(L, ctypeid) = data;
+	return 2;
+}
+
 int
 luaopen_net_box(struct lua_State *L)
 {
@@ -767,11 +855,13 @@ luaopen_net_box(struct lua_State *L)
 		{ "encode_update",  netbox_encode_update },
 		{ "encode_upsert",  netbox_encode_upsert },
 		{ "encode_execute", netbox_encode_execute},
+		{ "encode_prepare", netbox_encode_prepare},
 		{ "encode_auth",    netbox_encode_auth },
 		{ "decode_greeting",netbox_decode_greeting },
 		{ "communicate",    netbox_communicate },
 		{ "decode_select",  netbox_decode_select },
 		{ "decode_execute", netbox_decode_execute },
+		{ "decode_prepare", netbox_decode_prepare },
 		{ NULL, NULL}
 	};
 	/* luaL_register_module polutes _G */
diff --git a/src/box/lua/net_box.lua b/src/box/lua/net_box.lua
index c2e1bb9c4..b4811edfa 100644
--- a/src/box/lua/net_box.lua
+++ b/src/box/lua/net_box.lua
@@ -104,6 +104,8 @@ local method_encoder = {
     upsert  = internal.encode_upsert,
     select  = internal.encode_select,
     execute = internal.encode_execute,
+    prepare = internal.encode_prepare,
+    unprepare = internal.encode_prepare,
     get     = internal.encode_select,
     min     = internal.encode_select,
     max     = internal.encode_select,
@@ -128,6 +130,8 @@ local method_decoder = {
     upsert  = decode_nil,
     select  = internal.decode_select,
     execute = internal.decode_execute,
+    prepare = internal.decode_prepare,
+    unprepare = decode_nil,
     get     = decode_get,
     min     = decode_get,
     max     = decode_get,
@@ -1197,6 +1201,29 @@ function remote_methods:execute(query, parameters, sql_opts, netbox_opts)
                          sql_opts or {})
 end
 
+function remote_methods:prepare(query, parameters, sql_opts, netbox_opts)
+    check_remote_arg(self, "prepare")
+    if type(query) ~= "string" then
+        box.error(box.error.SQL_PREPARE, "expected string as SQL statement")
+    end
+    if sql_opts ~= nil then
+        box.error(box.error.UNSUPPORTED, "prepare", "options")
+    end
+    return self:_request('prepare', netbox_opts, nil, query)
+end
+
+function remote_methods:unprepare(query, parameters, sql_opts, netbox_opts)
+    check_remote_arg(self, "unprepare")
+    if type(query) ~= "number" then
+        box.error("query id is expected to be numeric")
+    end
+    if sql_opts ~= nil then
+        box.error(box.error.UNSUPPORTED, "unprepare", "options")
+    end
+    return self:_request('unprepare', netbox_opts, nil, query, parameters or {},
+                         sql_opts or {})
+end
+
 function remote_methods:wait_state(state, timeout)
     check_remote_arg(self, 'wait_state')
     if timeout == nil then
diff --git a/src/box/xrow.c b/src/box/xrow.c
index 18bf08971..88f308be5 100644
--- a/src/box/xrow.c
+++ b/src/box/xrow.c
@@ -576,9 +576,11 @@ error:
 	uint32_t map_size = mp_decode_map(&data);
 	request->sql_text = NULL;
 	request->bind = NULL;
+	request->stmt_id = NULL;
 	for (uint32_t i = 0; i < map_size; ++i) {
 		uint8_t key = *data;
-		if (key != IPROTO_SQL_BIND && key != IPROTO_SQL_TEXT) {
+		if (key != IPROTO_SQL_BIND && key != IPROTO_SQL_TEXT &&
+		    key != IPROTO_STMT_ID) {
 			mp_check(&data, end);   /* skip the key */
 			mp_check(&data, end);   /* skip the value */
 			continue;
@@ -588,12 +590,23 @@ error:
 			goto error;
 		if (key == IPROTO_SQL_BIND)
 			request->bind = value;
-		else
+		else if (key == IPROTO_SQL_TEXT)
 			request->sql_text = value;
+		else
+			request->stmt_id = value;
 	}
-	if (request->sql_text == NULL) {
-		xrow_on_decode_err(row->body[0].iov_base, end, ER_MISSING_REQUEST_FIELD,
-			 iproto_key_name(IPROTO_SQL_TEXT));
+	if (request->sql_text != NULL && request->stmt_id != NULL) {
+		xrow_on_decode_err(row->body[0].iov_base, end, ER_INVALID_MSGPACK,
+				   "SQL text and statement id are incompatible "\
+				   "options in one request: choose one");
+		return -1;
+	}
+	if (request->sql_text == NULL && request->stmt_id == NULL) {
+		xrow_on_decode_err(row->body[0].iov_base, end,
+				   ER_MISSING_REQUEST_FIELD,
+				   tt_sprintf("%s or %s",
+					      iproto_key_name(IPROTO_SQL_TEXT),
+					      iproto_key_name(IPROTO_STMT_ID)));
 		return -1;
 	}
 	if (data != end)
diff --git a/src/box/xrow.h b/src/box/xrow.h
index 60def2d3c..a4d8dc015 100644
--- a/src/box/xrow.h
+++ b/src/box/xrow.h
@@ -526,12 +526,14 @@ int
 iproto_reply_error(struct obuf *out, const struct error *e, uint64_t sync,
 		   uint32_t schema_version);
 
-/** EXECUTE request. */
+/** EXECUTE/PREPARE request. */
 struct sql_request {
 	/** SQL statement text. */
 	const char *sql_text;
 	/** MessagePack array of parameters. */
 	const char *bind;
+	/** ID of prepared statement. In this case @sql_text == NULL. */
+	const char *stmt_id;
 };
 
 /**
diff --git a/test/box/misc.result b/test/box/misc.result
index 90923f28e..79fd49442 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -250,6 +250,7 @@ t;
   - EVAL
   - CALL
   - ERROR
+  - PREPARE
   - REPLACE
   - UPSERT
   - AUTH
diff --git a/test/sql/engine.cfg b/test/sql/engine.cfg
index a1b4b0fc5..e38bec24e 100644
--- a/test/sql/engine.cfg
+++ b/test/sql/engine.cfg
@@ -10,6 +10,7 @@
         "local": {"remote": "false"}
     },
     "prepared.test.lua": {
+        "remote": {"remote": "true"},
         "local": {"remote": "false"}
     },
     "*": {
diff --git a/test/sql/iproto.result b/test/sql/iproto.result
index 67acd0ac1..4dfbfce50 100644
--- a/test/sql/iproto.result
+++ b/test/sql/iproto.result
@@ -119,7 +119,7 @@ cn:execute('select id as identifier from test where a = 5;')
 -- netbox API errors.
 cn:execute(100)
 ---
-- error: Syntax error near '100'
+- error: Prepared statement with id 100 does not exist
 ...
 cn:execute('select 1', nil, {dry_run = true})
 ---
diff --git a/test/sql/prepared.result b/test/sql/prepared.result
index bd37cfdd7..2f4983b00 100644
--- a/test/sql/prepared.result
+++ b/test/sql/prepared.result
@@ -12,34 +12,49 @@ fiber = require('fiber')
 -- Wrappers to make remote and local execution interface return
 -- same result pattern.
 --
-test_run:cmd("setopt delimiter ';'")
+is_remote = test_run:get_cfg('remote') == 'true'
  | ---
- | - true
  | ...
-execute = function(...)
-    local res, err = box.execute(...)
-    if err ~= nil then
-        error(err)
-    end
-    return res
-end;
+execute = nil
  | ---
  | ...
-prepare = function(...)
-    local res, err = box.prepare(...)
-    if err ~= nil then
-        error(err)
-    end
-    return res
-end;
+prepare = nil
+ | ---
+ | ...
+
+test_run:cmd("setopt delimiter ';'")
  | ---
+ | - true
  | ...
-unprepare = function(...)
-    local res, err = box.unprepare(...)
-    if err ~= nil then
-        error(err)
+if is_remote then
+    box.schema.user.grant('guest','read, write, execute', 'universe')
+    box.schema.user.grant('guest', 'create', 'space')
+    cn = remote.connect(box.cfg.listen)
+    execute = function(...) return cn:execute(...) end
+    prepare = function(...) return cn:prepare(...) end
+    unprepare = function(...) return cn:unprepare(...) end
+else
+    execute = function(...)
+        local res, err = box.execute(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
+    end
+    prepare = function(...)
+        local res, err = box.prepare(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
+    end
+    unprepare = function(...)
+        local res, err = box.unprepare(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
     end
-    return res
 end;
  | ---
  | ...
@@ -128,31 +143,26 @@ execute(s.stmt_id, {1, 3})
  |     type: string
  |   rows: []
  | ...
-s:execute({1, 2})
+
+test_run:cmd("setopt delimiter ';'")
  | ---
- | - metadata:
- |   - name: ID
- |     type: integer
- |   - name: A
- |     type: number
- |   - name: B
- |     type: string
- |   rows:
- |   - [1, 2, '3']
+ | - true
  | ...
-s:execute({1, 3})
+if not is_remote then
+    res = s:execute({1, 2})
+    assert(res ~= nil)
+    res = s:execute({1, 3})
+    assert(res ~= nil)
+end;
  | ---
- | - metadata:
- |   - name: ID
- |     type: integer
- |   - name: A
- |     type: number
- |   - name: B
- |     type: string
- |   rows: []
  | ...
-s:unprepare()
+test_run:cmd("setopt delimiter ''");
  | ---
+ | - true
+ | ...
+unprepare(s.stmt_id)
+ | ---
+ | - null
  | ...
 
 -- Test preparation of different types of queries.
@@ -338,6 +348,7 @@ _ = prepare("SELECT a FROM test WHERE b = '3';")
 s = prepare("SELECT a FROM test WHERE b = '3';")
  | ---
  | ...
+
 execute(s.stmt_id)
  | ---
  | - metadata:
@@ -354,21 +365,21 @@ execute(s.stmt_id)
  |   rows:
  |   - [2]
  | ...
-s:execute()
+test_run:cmd("setopt delimiter ';'")
  | ---
- | - metadata:
- |   - name: A
- |     type: number
- |   rows:
- |   - [2]
+ | - true
  | ...
-s:execute()
+if not is_remote then
+    res = s:execute()
+    assert(res ~= nil)
+    res = s:execute()
+    assert(res ~= nil)
+end;
  | ---
- | - metadata:
- |   - name: A
- |     type: number
- |   rows:
- |   - [2]
+ | ...
+test_run:cmd("setopt delimiter ''");
+ | ---
+ | - true
  | ...
 unprepare(s.stmt_id)
  | ---
@@ -671,6 +682,13 @@ unprepare(s.stmt_id);
  | - null
  | ...
 
+if is_remote then
+    cn:close()
+    box.schema.user.revoke('guest', 'read, write, execute', 'universe')
+    box.schema.user.revoke('guest', 'create', 'space')
+end;
+ | ---
+ | ...
 test_run:cmd("setopt delimiter ''");
  | ---
  | - true
diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
index 49d2fb3ae..c464cc21a 100644
--- a/test/sql/prepared.test.lua
+++ b/test/sql/prepared.test.lua
@@ -5,27 +5,40 @@ fiber = require('fiber')
 -- Wrappers to make remote and local execution interface return
 -- same result pattern.
 --
+is_remote = test_run:get_cfg('remote') == 'true'
+execute = nil
+prepare = nil
+
 test_run:cmd("setopt delimiter ';'")
-execute = function(...)
-    local res, err = box.execute(...)
-    if err ~= nil then
-        error(err)
+if is_remote then
+    box.schema.user.grant('guest','read, write, execute', 'universe')
+    box.schema.user.grant('guest', 'create', 'space')
+    cn = remote.connect(box.cfg.listen)
+    execute = function(...) return cn:execute(...) end
+    prepare = function(...) return cn:prepare(...) end
+    unprepare = function(...) return cn:unprepare(...) end
+else
+    execute = function(...)
+        local res, err = box.execute(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
     end
-    return res
-end;
-prepare = function(...)
-    local res, err = box.prepare(...)
-    if err ~= nil then
-        error(err)
+    prepare = function(...)
+        local res, err = box.prepare(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
     end
-    return res
-end;
-unprepare = function(...)
-    local res, err = box.unprepare(...)
-    if err ~= nil then
-        error(err)
+    unprepare = function(...)
+        local res, err = box.unprepare(...)
+        if err ~= nil then
+            error(err)
+        end
+        return res
     end
-    return res
 end;
 
 test_run:cmd("setopt delimiter ''");
@@ -46,9 +59,16 @@ s.params
 s.param_count
 execute(s.stmt_id, {1, 2})
 execute(s.stmt_id, {1, 3})
-s:execute({1, 2})
-s:execute({1, 3})
-s:unprepare()
+
+test_run:cmd("setopt delimiter ';'")
+if not is_remote then
+    res = s:execute({1, 2})
+    assert(res ~= nil)
+    res = s:execute({1, 3})
+    assert(res ~= nil)
+end;
+test_run:cmd("setopt delimiter ''");
+unprepare(s.stmt_id)
 
 -- Test preparation of different types of queries.
 -- Let's start from DDL. It doesn't make much sense since
@@ -111,10 +131,17 @@ space:replace{4, 5, '6'}
 space:replace{7, 8.5, '9'}
 _ = prepare("SELECT a FROM test WHERE b = '3';")
 s = prepare("SELECT a FROM test WHERE b = '3';")
+
 execute(s.stmt_id)
 execute(s.stmt_id)
-s:execute()
-s:execute()
+test_run:cmd("setopt delimiter ';'")
+if not is_remote then
+    res = s:execute()
+    assert(res ~= nil)
+    res = s:execute()
+    assert(res ~= nil)
+end;
+test_run:cmd("setopt delimiter ''");
 unprepare(s.stmt_id)
 
 s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
@@ -233,6 +260,11 @@ f2:join();
 
 unprepare(s.stmt_id);
 
+if is_remote then
+    cn:close()
+    box.schema.user.revoke('guest', 'read, write, execute', 'universe')
+    box.schema.user.revoke('guest', 'create', 'space')
+end;
 test_run:cmd("setopt delimiter ''");
 
 box.cfg{sql_cache_size = 5 * 1024 * 1024}
-- 
2.15.1

[Tarantool-patches] [PATCH v3 20/20] sql: add cache statistics to box.info

[Nikita Pettik]

To track current memory occupied by prepared statements and number of
them, let's extend box.info submodule with .sql statistics: now it
contains current total size of prepared statements and their count.

@TarantoolBot document
Title: Prepared statements in SQL

Now it is possible to prepare (i.e. compile into byte-code and save to
the cache) statement and execute it several times. Mechanism is similar
to ones in other DBs. Prepared statement is identified by numeric
ID, which are returned alongside with prepared statement handle.
Note that they are not sequential and represent value of hash function
applied to the string containing original SQL request.
Prepared statement holder is shared among all sessions. However, session
has access only to statements which have been prepared in scope of it.
There's no eviction policy like in any cache; to remove statement from
holder explicit unprepare request is required. Alternatively, session's
disconnect also removes statements from holder.
Several sessions can share one prepared statement, which will be
destroyed when all related sessions are disconnected or send unprepare
request. Memory limit for prepared statements is adjusted by
box.cfg{sql_cache_size} handle (can be set dynamically;

Any DDL operation leads to expiration of all prepared statements: they
should be manually removed or re-prepared.
Prepared statements are available in local mode (i.e. via box.prepare()
function) and are supported in IProto protocol. In the latter case
next IProto keys are used to make up/receive requests/responses:
IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
request, meanwhile IPROTO_STMT_ID - unprepare;
IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
be bound) are response keys. They are mandatory members of result of
IPROTO_PREPARE execution.

To track statistics of used memory and number of currently prepared
statements, box.info is extended with SQL statistics:

box.info:sql().cache.stmt_count - number of prepared statements;
box.info:sql().cache.size - size of occupied by prepared statements memory.

Typical workflow with prepared statements is following:

s = box.prepare("SELECT * FROM t WHERE id = ?;")
s:execute({1}) or box.execute(s.sql_str, {1})
s:execute({2}) or box.execute(s.sql_str, {2})
s:unprepare() or box.unprepare(s.query_id)

Structure of object is following (member : type):

- stmt_id: integer
  execute: function
  params: map [name : string, type : integer]
  unprepare: function
  metadata: map [name : string, type : integer]
  param_count: integer
...

In terms of remote connection:

cn = netbox:connect(addr)
s = cn:prepare("SELECT * FROM t WHERE id = ?;")
cn:execute(s.sql_str, {1})
cn:unprepare(s.query_id)

Closese #2592
---
 src/box/lua/info.c         | 25 +++++++++++++++++++++++++
 src/box/sql_stmt_cache.c   | 16 ++++++++++++++++
 src/box/sql_stmt_cache.h   |  8 ++++++++
 test/box/info.result       |  1 +
 test/sql/prepared.result   | 34 +++++++++++++++++++++++++++++++++-
 test/sql/prepared.test.lua | 12 +++++++++++-
 6 files changed, 94 insertions(+), 2 deletions(-)

diff --git a/src/box/lua/info.c b/src/box/lua/info.c
index e029e0e17..8933ea829 100644
--- a/src/box/lua/info.c
+++ b/src/box/lua/info.c
@@ -45,6 +45,7 @@
 #include "box/gc.h"
 #include "box/engine.h"
 #include "box/vinyl.h"
+#include "box/sql_stmt_cache.h"
 #include "main.h"
 #include "version.h"
 #include "box/box.h"
@@ -494,6 +495,29 @@ lbox_info_vinyl(struct lua_State *L)
 	return 1;
 }
 
+static int
+lbox_info_sql_call(struct lua_State *L)
+{
+	struct info_handler h;
+	luaT_info_handler_create(&h, L);
+	sql_stmt_cache_stat(&h);
+
+	return 1;
+}
+
+static int
+lbox_info_sql(struct lua_State *L)
+{
+	lua_newtable(L);
+	lua_newtable(L); /* metatable */
+	lua_pushstring(L, "__call");
+	lua_pushcfunction(L, lbox_info_sql_call);
+	lua_settable(L, -3);
+
+	lua_setmetatable(L, -2);
+	return 1;
+}
+
 static const struct luaL_Reg lbox_info_dynamic_meta[] = {
 	{"id", lbox_info_id},
 	{"uuid", lbox_info_uuid},
@@ -509,6 +533,7 @@ static const struct luaL_Reg lbox_info_dynamic_meta[] = {
 	{"memory", lbox_info_memory},
 	{"gc", lbox_info_gc},
 	{"vinyl", lbox_info_vinyl},
+	{"sql", lbox_info_sql},
 	{NULL, NULL}
 };
 
diff --git a/src/box/sql_stmt_cache.c b/src/box/sql_stmt_cache.c
index 742e4135c..a4f5f2745 100644
--- a/src/box/sql_stmt_cache.c
+++ b/src/box/sql_stmt_cache.c
@@ -34,6 +34,7 @@
 #include "error.h"
 #include "execute.h"
 #include "diag.h"
+#include "info/info.h"
 
 static struct sql_stmt_cache sql_stmt_cache;
 
@@ -48,6 +49,21 @@ sql_stmt_cache_init()
 	rlist_create(&sql_stmt_cache.gc_queue);
 }
 
+void
+sql_stmt_cache_stat(struct info_handler *h)
+{
+	info_begin(h);
+	info_table_begin(h, "cache");
+	info_append_int(h, "size", sql_stmt_cache.mem_used);
+	uint32_t entry_count = 0;
+	mh_int_t i;
+	mh_foreach(sql_stmt_cache.hash, i)
+		entry_count++;
+	info_append_int(h, "stmt_count", entry_count);
+	info_table_end(h);
+	info_end(h);
+}
+
 static size_t
 sql_cache_entry_sizeof(struct sql_stmt *stmt)
 {
diff --git a/src/box/sql_stmt_cache.h b/src/box/sql_stmt_cache.h
index f3935a27f..468cbc9a0 100644
--- a/src/box/sql_stmt_cache.h
+++ b/src/box/sql_stmt_cache.h
@@ -41,6 +41,7 @@ extern "C" {
 
 struct sql_stmt;
 struct mh_i64ptr_t;
+struct info_handler;
 
 struct stmt_cache_entry {
 	/** Prepared statement itself. */
@@ -90,6 +91,13 @@ struct sql_stmt_cache {
 void
 sql_stmt_cache_init();
 
+/**
+ * Store statistics concerning cache (current size and number
+ * of statements in it) into info handler @h.
+ */
+void
+sql_stmt_cache_stat(struct info_handler *h);
+
 /**
  * Erase session local hash: unref statements belong to this
  * session and deallocate hash itself.
diff --git a/test/box/info.result b/test/box/info.result
index af81f7add..2e84cbbe3 100644
--- a/test/box/info.result
+++ b/test/box/info.result
@@ -84,6 +84,7 @@ t
   - replication
   - ro
   - signature
+  - sql
   - status
   - uptime
   - uuid
diff --git a/test/sql/prepared.result b/test/sql/prepared.result
index 2f4983b00..9951a4e43 100644
--- a/test/sql/prepared.result
+++ b/test/sql/prepared.result
@@ -64,6 +64,21 @@ test_run:cmd("setopt delimiter ''");
  | - true
  | ...
 
+-- Check default cache statistics.
+--
+box.info.sql()
+ | ---
+ | - cache:
+ |     size: 0
+ |     stmt_count: 0
+ | ...
+box.info:sql()
+ | ---
+ | - cache:
+ |     size: 0
+ |     stmt_count: 0
+ | ...
+
 -- Test local interface and basic capabilities of prepared statements.
 --
 execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
@@ -144,6 +159,15 @@ execute(s.stmt_id, {1, 3})
  |   rows: []
  | ...
 
+assert(box.info.sql().cache.stmt_count ~= 0)
+ | ---
+ | - true
+ | ...
+assert(box.info.sql().cache.size ~= 0)
+ | ---
+ | - true
+ | ...
+
 test_run:cmd("setopt delimiter ';'")
  | ---
  | - true
@@ -584,8 +608,16 @@ unprepare(s1.stmt_id)
 -- Setting cache size to 0 is possible only in case if
 -- there's no any prepared statements right now .
 --
-box.cfg{sql_cache_size = 0}
+box.cfg{sql_cache_size = 0 }
+ | ---
+ | ...
+assert(box.info.sql().cache.stmt_count == 0)
  | ---
+ | - true
+ | ...
+assert(box.info.sql().cache.size == 0)
+ | ---
+ | - true
  | ...
 prepare("SELECT a FROM test;")
  | ---
diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
index c464cc21a..5820525d1 100644
--- a/test/sql/prepared.test.lua
+++ b/test/sql/prepared.test.lua
@@ -43,6 +43,11 @@ end;
 
 test_run:cmd("setopt delimiter ''");
 
+-- Check default cache statistics.
+--
+box.info.sql()
+box.info:sql()
+
 -- Test local interface and basic capabilities of prepared statements.
 --
 execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
@@ -60,6 +65,9 @@ s.param_count
 execute(s.stmt_id, {1, 2})
 execute(s.stmt_id, {1, 3})
 
+assert(box.info.sql().cache.stmt_count ~= 0)
+assert(box.info.sql().cache.size ~= 0)
+
 test_run:cmd("setopt delimiter ';'")
 if not is_remote then
     res = s:execute({1, 2})
@@ -207,7 +215,9 @@ unprepare(s1.stmt_id)
 -- Setting cache size to 0 is possible only in case if
 -- there's no any prepared statements right now .
 --
-box.cfg{sql_cache_size = 0}
+box.cfg{sql_cache_size = 0 }
+assert(box.info.sql().cache.stmt_count == 0)
+assert(box.info.sql().cache.size == 0)
 prepare("SELECT a FROM test;")
 box.cfg{sql_cache_size = 0}
 
-- 
2.15.1

Re: [Tarantool-patches] [PATCH v3 01/20] sql: remove sql_prepare_v2()

[Sergey Ostanevich]

Hi!

Thanks for the patch!

On 20 Dec 15:47, Nikita Pettik wrote:
> There are two versions of the same function (sql_prepare()) which are
> almost identical. Let's keep more relevant version sql_prepare_v2() but
> rename it to sql_prepare() in order to avoid any mess.
> 
> Needed for #3292
> ---
>  src/box/execute.c     |  2 +-
>  src/box/sql/legacy.c  |  2 +-
>  src/box/sql/prepare.c | 32 ++++----------------------------
>  src/box/sql/sqlInt.h  | 25 +++++++++++--------------
>  src/box/sql/vdbeapi.c |  2 +-
>  5 files changed, 18 insertions(+), 45 deletions(-)
> 
> diff --git a/src/box/execute.c b/src/box/execute.c
> index e8b012e5b..130a3f675 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -443,7 +443,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
>  {
>  	struct sql_stmt *stmt;
>  	struct sql *db = sql_get();
> -	if (sql_prepare_v2(db, sql, len, &stmt, NULL) != 0)
> +	if (sql_prepare(db, sql, len, &stmt, NULL) != 0)
>  		return -1;
>  	assert(stmt != NULL);
>  	port_sql_create(port, stmt);
> diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
> index 0b1370f4a..bfd1e32b9 100644
> --- a/src/box/sql/legacy.c
> +++ b/src/box/sql/legacy.c
> @@ -70,7 +70,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
>  		char **azVals = 0;
>  
>  		pStmt = 0;
> -		rc = sql_prepare_v2(db, zSql, -1, &pStmt, &zLeftover);
> +		rc = sql_prepare(db, zSql, -1, &pStmt, &zLeftover);
>  		assert(rc == 0 || pStmt == NULL);
>  		if (rc != 0)
>  			continue;
> diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> index 0ecc676e2..35e81212d 100644
> --- a/src/box/sql/prepare.c
> +++ b/src/box/sql/prepare.c
> @@ -204,36 +204,12 @@ sqlReprepare(Vdbe * p)
>  	return 0;
>  }
>  
> -/*
> - * Two versions of the official API.  Legacy and new use.  In the legacy
> - * version, the original SQL text is not saved in the prepared statement
> - * and so if a schema change occurs, an error is returned by
> - * sql_step().  In the new version, the original SQL text is retained
> - * and the statement is automatically recompiled if an schema change
> - * occurs.
> - */
> -int
> -sql_prepare(sql * db,		/* Database handle. */
> -		const char *zSql,	/* UTF-8 encoded SQL statement. */
> -		int nBytes,		/* Length of zSql in bytes. */
> -		sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> -		const char **pzTail)	/* OUT: End of parsed string */
> -{
> -	int rc = sqlPrepare(db, zSql, nBytes, 0, 0, ppStmt, pzTail);
> -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> -	return rc;
> -}
> -
>  int
> -sql_prepare_v2(sql * db,	/* Database handle. */
> -		   const char *zSql,	/* UTF-8 encoded SQL statement. */
> -		   int nBytes,	/* Length of zSql in bytes. */
> -		   sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> -		   const char **pzTail	/* OUT: End of parsed string */
> -    )
> +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> +	    const char **sql_tail)
>  {
> -	int rc = sqlPrepare(db, zSql, nBytes, 1, 0, ppStmt, pzTail);
> -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> +	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
> +	assert(rc == 0 || stmt == NULL || *stmt == NULL);
>  	return rc;
>  }
>  
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 2594b73e0..7bd952a17 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -468,21 +468,18 @@ typedef void (*sql_destructor_type) (void *);
>  #define SQL_STATIC      ((sql_destructor_type)0)
>  #define SQL_TRANSIENT   ((sql_destructor_type)-1)
>  
> +/**
> + * Prepare (compile into VDBE byte-code) statement.

Could you please extend the description with details on SQL text
preservance and recmopilation, same as it was in old version?

> + *
> + * @param db Database handle.
> + * @param sql UTF-8 encoded SQL statement.
> + * @param length Length of @param sql in bytes.
> + * @param[out] stmt A pointer to the prepared statement.
> + * @param[out] sql_tail End of parsed string.
> + */
>  int
> -sql_prepare(sql * db,	/* Database handle */
> -		const char *zSql,	/* SQL statement, UTF-8 encoded */
> -		int nByte,	/* Maximum length of zSql in bytes. */
> -		sql_stmt ** ppStmt,	/* OUT: Statement handle */
> -		const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> -	);
> -
> -int
> -sql_prepare_v2(sql * db,	/* Database handle */
> -		   const char *zSql,	/* SQL statement, UTF-8 encoded */
> -		   int nByte,	/* Maximum length of zSql in bytes. */
> -		   sql_stmt ** ppStmt,	/* OUT: Statement handle */
> -		   const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> -	);
> +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> +	    const char **sql_tail);
>  
>  int
>  sql_step(sql_stmt *);
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 685212d91..12449d3bc 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -452,7 +452,7 @@ sqlStep(Vdbe * p)
>  		checkProfileCallback(db, p);
>  
>  	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
> -		/* If this statement was prepared using sql_prepare_v2(), and an
> +		/* If this statement was prepared using sql_prepare(), and an
>  		 * error has occurred, then return an error.
>  		 */
>  		if (p->is_aborted)
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 17/20] sql: introduce holder for prepared statemets

[Sergey Ostanevich]

Hi!

Thank you for the patch! Below are two questions I have.

Thanks,
Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> This patch introduces holder (as data structure) to handle prepared
> statements and a set of interface functions (insert, delete, find) to
> operate on it. Holder under the hood is implemented as a global hash
> (keys are values of hash function applied to the original string containing
> SQL query; values are pointer to wrappers around compiled VDBE objects) and
> GC queue. Each entry in hash has reference counter. When it reaches 0
> value, entry is moved to GC queue. In case prepared statements holder is
> out of memory, it launches GC process: each entry in GC queue is deleted
> and all resources are released. Such approach allows to avoid workload
> spikes on session's disconnect (since on such event all statements must
> be deallocated).
> Each session is extended with local hash to map statement ids available
> for it. That is, session is allowed to execute and deallocate only
> statements which are previously prepared in scope of this session.
> On the other hand, global hash makes it possible to share same prepared
> statement object among different sessions.
> Size of cache is regulated via box.cfg{sql_cache_size} parameter.
> 
> Part of #2592
> ---
>  src/box/CMakeLists.txt          |   1 +
>  src/box/box.cc                  |  26 ++++
>  src/box/box.h                   |   3 +
>  src/box/errcode.h               |   1 +
>  src/box/lua/cfg.cc              |   9 ++
>  src/box/lua/load_cfg.lua        |   3 +
>  src/box/session.cc              |  35 +++++
>  src/box/session.h               |  17 +++
>  src/box/sql.c                   |   3 +
>  src/box/sql_stmt_cache.c        | 289 ++++++++++++++++++++++++++++++++++++++++
>  src/box/sql_stmt_cache.h        | 145 ++++++++++++++++++++
>  test/app-tap/init_script.result |  37 ++---
>  test/box/admin.result           |   2 +
>  test/box/cfg.result             |   7 +
>  test/box/cfg.test.lua           |   1 +
>  test/box/misc.result            |   1 +
>  16 files changed, 562 insertions(+), 18 deletions(-)
>  create mode 100644 src/box/sql_stmt_cache.c
>  create mode 100644 src/box/sql_stmt_cache.h
> 
> diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
> index 5cd5cba81..763bc3a4c 100644
> --- a/src/box/CMakeLists.txt
> +++ b/src/box/CMakeLists.txt
> @@ -126,6 +126,7 @@ add_library(box STATIC
>      sql.c
>      bind.c
>      execute.c
> +    sql_stmt_cache.c
>      wal.c
>      call.c
>      merger.c
> diff --git a/src/box/box.cc b/src/box/box.cc
> index b119c927b..2b0cfa32d 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -74,6 +74,7 @@
>  #include "call.h"
>  #include "func.h"
>  #include "sequence.h"
> +#include "sql_stmt_cache.h"
>  
>  static char status[64] = "unknown";
>  
> @@ -599,6 +600,17 @@ box_check_vinyl_options(void)
>  	}
>  }
>  
> +static int
> +box_check_sql_cache_size(int size)
> +{
> +	if (size < 0) {
> +		diag_set(ClientError, ER_CFG, "sql_cache_size",
> +			 "must be non-negative");
> +		return -1;
> +	}
> +	return 0;
> +}
> +
>  void
>  box_check_config()
>  {
> @@ -620,6 +632,7 @@ box_check_config()
>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>  	box_check_vinyl_options();
> +	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
>  }
>  
>  /*
> @@ -886,6 +899,17 @@ box_set_net_msg_max(void)
>  				IPROTO_FIBER_POOL_SIZE_FACTOR);
>  }
>  
> +int
> +box_set_prepared_stmt_cache_size(void)
> +{
> +	int cache_sz = cfg_geti("sql_cache_size");
> +	if (box_check_sql_cache_size(cache_sz) != 0)
> +		return -1;
> +	if (sql_stmt_cache_set_size(cache_sz) != 0)
> +		return -1;
> +	return 0;
> +}
> +
>  /* }}} configuration bindings */
>  
>  /**
> @@ -2096,6 +2120,8 @@ box_cfg_xc(void)
>  	box_check_instance_uuid(&instance_uuid);
>  	box_check_replicaset_uuid(&replicaset_uuid);
>  
> +	if (box_set_prepared_stmt_cache_size() != 0)
> +		diag_raise();
>  	box_set_net_msg_max();
>  	box_set_readahead();
>  	box_set_too_long_threshold();
> diff --git a/src/box/box.h b/src/box/box.h
> index ccd527bd5..6806f1fc2 100644
> --- a/src/box/box.h
> +++ b/src/box/box.h
> @@ -236,6 +236,9 @@ void box_set_replication_sync_timeout(void);
>  void box_set_replication_skip_conflict(void);
>  void box_set_net_msg_max(void);
>  
> +int
> +box_set_prepared_stmt_cache_size(void);
> +
>  extern "C" {
>  #endif /* defined(__cplusplus) */
>  
> diff --git a/src/box/errcode.h b/src/box/errcode.h
> index c660b1c70..ee44f61b3 100644
> --- a/src/box/errcode.h
> +++ b/src/box/errcode.h
> @@ -258,6 +258,7 @@ struct errcode_record {
>  	/*203 */_(ER_BOOTSTRAP_READONLY,	"Trying to bootstrap a local read-only instance as master") \
>  	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
>  	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
> +	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
>  
>  /*
>   * !IMPORTANT! Please follow instructions at start of the file
> diff --git a/src/box/lua/cfg.cc b/src/box/lua/cfg.cc
> index 4884ce013..439af3cf9 100644
> --- a/src/box/lua/cfg.cc
> +++ b/src/box/lua/cfg.cc
> @@ -274,6 +274,14 @@ lbox_cfg_set_net_msg_max(struct lua_State *L)
>  	return 0;
>  }
>  
> +static int
> +lbox_set_prepared_stmt_cache_size(struct lua_State *L)
> +{
> +	if (box_set_prepared_stmt_cache_size() != 0)
> +		luaT_error(L);
> +	return 0;
> +}
> +
>  static int
>  lbox_cfg_set_worker_pool_threads(struct lua_State *L)
>  {
> @@ -378,6 +386,7 @@ box_lua_cfg_init(struct lua_State *L)
>  		{"cfg_set_replication_sync_timeout", lbox_cfg_set_replication_sync_timeout},
>  		{"cfg_set_replication_skip_conflict", lbox_cfg_set_replication_skip_conflict},
>  		{"cfg_set_net_msg_max", lbox_cfg_set_net_msg_max},
> +		{"cfg_set_sql_cache_size", lbox_set_prepared_stmt_cache_size},
>  		{NULL, NULL}
>  	};
>  
> diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua
> index 85617c8f0..4463f989c 100644
> --- a/src/box/lua/load_cfg.lua
> +++ b/src/box/lua/load_cfg.lua
> @@ -81,6 +81,7 @@ local default_cfg = {
>      feedback_host         = "https://feedback.tarantool.io",
>      feedback_interval     = 3600,
>      net_msg_max           = 768,
> +    sql_cache_size        = 5 * 1024 * 1024,
>  }
>  
>  -- types of available options
> @@ -144,6 +145,7 @@ local template_cfg = {
>      feedback_host         = 'string',
>      feedback_interval     = 'number',
>      net_msg_max           = 'number',
> +    sql_cache_size        = 'number',
>  }
>  
>  local function normalize_uri(port)
> @@ -250,6 +252,7 @@ local dynamic_cfg = {
>      instance_uuid           = check_instance_uuid,
>      replicaset_uuid         = check_replicaset_uuid,
>      net_msg_max             = private.cfg_set_net_msg_max,
> +    sql_cache_size          = private.cfg_set_sql_cache_size,
>  }
>  
>  --
> diff --git a/src/box/session.cc b/src/box/session.cc
> index 461d1cf25..881318252 100644
> --- a/src/box/session.cc
> +++ b/src/box/session.cc
> @@ -36,6 +36,7 @@
>  #include "user.h"
>  #include "error.h"
>  #include "tt_static.h"
> +#include "sql_stmt_cache.h"
>  
>  const char *session_type_strs[] = {
>  	"background",
> @@ -141,6 +142,7 @@ session_create(enum session_type type)
>  	session_set_type(session, type);
>  	session->sql_flags = default_flags;
>  	session->sql_default_engine = SQL_STORAGE_ENGINE_MEMTX;
> +	session->sql_stmts = NULL;
>  
>  	/* For on_connect triggers. */
>  	credentials_create(&session->credentials, guest_user);
> @@ -178,6 +180,38 @@ session_create_on_demand()
>  	return s;
>  }
>  
> +bool
> +session_check_stmt_id(struct session *session, uint32_t stmt_id)
> +{
> +	if (session->sql_stmts == NULL)
> +		return false;
> +	mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL);
> +	return i != mh_end(session->sql_stmts);
> +}
> +
> +int
> +session_add_stmt_id(struct session *session, uint32_t id)
> +{
> +	if (session->sql_stmts == NULL) {
> +		session->sql_stmts = mh_i32ptr_new();
> +		if (session->sql_stmts == NULL) {
> +			diag_set(OutOfMemory, 0, "mh_i32ptr_new",
> +				 "session stmt hash");
> +			return -1;
> +		}
> +	}
> +	return sql_session_stmt_hash_add_id(session->sql_stmts, id);
> +}
> +
> +void
> +session_remove_stmt_id(struct session *session, uint32_t stmt_id)
> +{
> +	assert(session->sql_stmts != NULL);
> +	mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL);
> +	assert(i != mh_end(session->sql_stmts));
> +	mh_i32ptr_del(session->sql_stmts, i, NULL);
> +}
> +
>  /**
>   * To quickly switch to admin user when executing
>   * on_connect/on_disconnect triggers in iproto.
> @@ -227,6 +261,7 @@ session_destroy(struct session *session)
>  	struct mh_i64ptr_node_t node = { session->id, NULL };
>  	mh_i64ptr_remove(session_registry, &node, NULL);
>  	credentials_destroy(&session->credentials);
> +	sql_session_stmt_hash_erase(session->sql_stmts);
>  	mempool_free(&session_pool, session);
>  }
>  
> diff --git a/src/box/session.h b/src/box/session.h
> index eff3d7a67..6dfc7cba5 100644
> --- a/src/box/session.h
> +++ b/src/box/session.h
> @@ -101,6 +101,11 @@ struct session {
>  	const struct session_vtab *vtab;
>  	/** Session metadata. */
>  	union session_meta meta;
> +	/**
> +	 * ID of statements prepared in current session.
> +	 * This map is allocated on demand.
> +	 */
> +	struct mh_i32ptr_t *sql_stmts;
>  	/** Session user id and global grants */
>  	struct credentials credentials;
>  	/** Trigger for fiber on_stop to cleanup created on-demand session */
> @@ -267,6 +272,18 @@ session_storage_cleanup(int sid);
>  struct session *
>  session_create(enum session_type type);
>  
> +/** Return true if given statement id belongs to the session. */
> +bool
> +session_check_stmt_id(struct session *session, uint32_t stmt_id);
> +
> +/** Add prepared statement ID to the session hash. */
> +int
> +session_add_stmt_id(struct session *session, uint32_t stmt_id);
> +
> +/** Remove prepared statement ID from the session hash. */
> +void
> +session_remove_stmt_id(struct session *session, uint32_t stmt_id);
> +
>  /**
>   * Destroy a session.
>   * Must be called by the networking layer on disconnect.
> diff --git a/src/box/sql.c b/src/box/sql.c
> index f1df55571..455fabeef 100644
> --- a/src/box/sql.c
> +++ b/src/box/sql.c
> @@ -54,6 +54,7 @@
>  #include "iproto_constants.h"
>  #include "fk_constraint.h"
>  #include "mpstream.h"
> +#include "sql_stmt_cache.h"
>  
>  static sql *db = NULL;
>  
> @@ -74,6 +75,8 @@ sql_init()
>  	if (sql_init_db(&db) != 0)
>  		panic("failed to initialize SQL subsystem");
>  
> +	sql_stmt_cache_init();
> +
>  	assert(db != NULL);
>  }
>  
> diff --git a/src/box/sql_stmt_cache.c b/src/box/sql_stmt_cache.c
> new file mode 100644
> index 000000000..742e4135c
> --- /dev/null
> +++ b/src/box/sql_stmt_cache.c
> @@ -0,0 +1,289 @@
> +/*
> + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + *    copyright notice, this list of conditions and the
> + *    following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + *    copyright notice, this list of conditions and the following
> + *    disclaimer in the documentation and/or other materials
> + *    provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +#include "sql_stmt_cache.h"
> +
> +#include "assoc.h"
> +#include "error.h"
> +#include "execute.h"
> +#include "diag.h"
> +
> +static struct sql_stmt_cache sql_stmt_cache;
> +
> +void
> +sql_stmt_cache_init()
> +{
> +	sql_stmt_cache.hash = mh_i32ptr_new();
> +	if (sql_stmt_cache.hash == NULL)
> +		panic("out of memory");
> +	sql_stmt_cache.mem_quota = 0;
> +	sql_stmt_cache.mem_used = 0;
> +	rlist_create(&sql_stmt_cache.gc_queue);
> +}
> +
> +static size_t
> +sql_cache_entry_sizeof(struct sql_stmt *stmt)
> +{
> +	return sql_stmt_est_size(stmt) + sizeof(struct stmt_cache_entry);
> +}
> +
> +static void
> +sql_cache_entry_delete(struct stmt_cache_entry *entry)
> +{
> +	assert(entry->refs == 0);
> +	assert(! sql_stmt_busy(entry->stmt));
> +	sql_stmt_finalize(entry->stmt);
> +	TRASH(entry);
> +	free(entry);
> +}
> +
> +/**
> + * Remove statement entry from cache: firstly delete from hash,
> + * than remove from LRU list and account cache size changes,
> + * finally release occupied memory.
> + */
> +static void
> +sql_stmt_cache_delete(struct stmt_cache_entry *entry)
> +{
> +	if (sql_stmt_cache.last_found == entry)
> +		sql_stmt_cache.last_found = NULL;
> +	rlist_del(&entry->link);
> +	sql_stmt_cache.mem_used -= sql_cache_entry_sizeof(entry->stmt);
> +	sql_cache_entry_delete(entry);
> +}
> +
> +static struct stmt_cache_entry *
> +stmt_cache_find_entry(uint32_t stmt_id)
> +{
> +	if (sql_stmt_cache.last_found != NULL) {
> +		const char *sql_str =
> +			sql_stmt_query_str(sql_stmt_cache.last_found->stmt);
> +		uint32_t last_stmt_id = sql_stmt_calculate_id(sql_str,
> +							 strlen(sql_str));
> +		if (last_stmt_id == stmt_id)
> +			return sql_stmt_cache.last_found;
> +		/* Fallthrough to slow hash search. */
> +	}
> +	struct mh_i32ptr_t *hash = sql_stmt_cache.hash;
> +	mh_int_t stmt = mh_i32ptr_find(hash, stmt_id, NULL);
> +	if (stmt == mh_end(hash))
> +		return NULL;
> +	struct stmt_cache_entry *entry = mh_i32ptr_node(hash, stmt)->val;
> +	if (entry == NULL)
> +		return NULL;
> +	sql_stmt_cache.last_found = entry;
> +	return entry;
> +}
> +
> +static void
> +sql_stmt_cache_gc()
> +{
> +	struct stmt_cache_entry *entry, *next;
> +	rlist_foreach_entry_safe(entry, &sql_stmt_cache.gc_queue, link, next)
> +		sql_stmt_cache_delete(entry);
> +	assert(rlist_empty(&sql_stmt_cache.gc_queue));
> +}
> +
> +/**
> + * Allocate new cache entry containing given prepared statement.
> + * Add it to the LRU cache list. Account cache size enlargement.
> + */
> +static struct stmt_cache_entry *
> +sql_cache_entry_new(struct sql_stmt *stmt)
> +{
> +	struct stmt_cache_entry *entry = malloc(sizeof(*entry));
> +	if (entry == NULL) {
> +		diag_set(OutOfMemory, sizeof(*entry), "malloc",
> +			 "struct stmt_cache_entry");
> +		return NULL;
> +	}
> +	entry->stmt = stmt;
> +	entry->refs = 0;
> +	return entry;
> +}
> +
> +/**
> + * Return true if used memory (accounting new entry) for SQL
> + * prepared statement cache does not exceed the limit.
> + */
> +static bool
> +sql_cache_check_new_entry_size(size_t size)
> +{
> +	return (sql_stmt_cache.mem_used + size <= sql_stmt_cache.mem_quota);
> +}
> +
> +static void
> +sql_stmt_cache_entry_unref(struct stmt_cache_entry *entry)
> +{
> +	assert((int64_t)entry->refs - 1 >= 0);
> +	if (--entry->refs == 0) {
> +		/*
> +		 * Remove entry from hash and add it to gc queue.
> +		 * Resources are to be released in the nearest
> +		 * GC cycle (see sql_stmt_cache_insert()).
> +		 */
> +		struct sql_stmt_cache *cache = &sql_stmt_cache;
> +		const char *sql_str = sql_stmt_query_str(entry->stmt);
> +		uint32_t stmt_id = sql_stmt_calculate_id(sql_str,
> +							 strlen(sql_str));
> +		mh_int_t i = mh_i32ptr_find(cache->hash, stmt_id, NULL);
> +		assert(i != mh_end(cache->hash));
> +		mh_i32ptr_del(cache->hash, i, NULL);
> +		rlist_add(&sql_stmt_cache.gc_queue, &entry->link);
> +		if (sql_stmt_cache.last_found == entry)
> +			sql_stmt_cache.last_found = NULL;
> +	}
> +}
> +
> +void
> +sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash)
> +{
> +	if (hash == NULL)
> +		return;
> +	mh_int_t i;
> +	struct stmt_cache_entry *entry;
> +	mh_foreach(hash, i) {
> +		entry = (struct stmt_cache_entry *)
> +			mh_i32ptr_node(hash, i)->val;
> +		sql_stmt_cache_entry_unref(entry);
> +	}
> +	mh_i32ptr_delete(hash);
> +}
> +
> +int
> +sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id)
> +{
> +	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
> +	const struct mh_i32ptr_node_t id_node = { stmt_id, entry };
> +	struct mh_i32ptr_node_t *old_node = NULL;
> +	mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL);
> +	if (i == mh_end(hash)) {
> +		diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node");
> +		return -1;
> +	}
> +	assert(old_node == NULL);
> +	entry->refs++;
> +	return 0;
> +}
> +
> +uint32_t
> +sql_stmt_calculate_id(const char *sql_str, size_t len)
> +{
> +	return mh_strn_hash(sql_str, len);
> +}
> +
> +void
> +sql_stmt_unref(uint32_t stmt_id)
> +{
> +	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
> +	assert(entry != NULL);
> +	sql_stmt_cache_entry_unref(entry);
> +}
> +
> +int
> +sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt)
> +{
> +	const char *sql_str = sql_stmt_query_str(old_stmt);
> +	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
> +	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
> +	uint32_t ref_count = entry->refs;

Does this update can happen is such a way that statement can be
substituted by a different one, still with the same hash? In such a case
users of 'old_stmt' can see surprising results...

> +	sql_stmt_cache_delete(entry);
> +	if (sql_stmt_cache_insert(new_stmt) != 0) {
> +		sql_stmt_finalize(new_stmt);
> +		return -1;
> +	}
> +	/* Restore reference counter. */
> +	entry = stmt_cache_find_entry(stmt_id);
> +	entry->refs = ref_count;
> +	return 0;
> +}
> +
> +int
> +sql_stmt_cache_insert(struct sql_stmt *stmt)
> +{
> +	assert(stmt != NULL);
> +	struct sql_stmt_cache *cache = &sql_stmt_cache;
> +	size_t new_entry_size = sql_cache_entry_sizeof(stmt);
> +
> +	if (! sql_cache_check_new_entry_size(new_entry_size))
> +		sql_stmt_cache_gc();
> +	/*
> +	 * Test memory limit again. Raise an error if it is
> +	 * still overcrowded.
> +	 */
> +	if (! sql_cache_check_new_entry_size(new_entry_size)) {
> +		diag_set(ClientError, ER_SQL_PREPARE, "Memory limit for SQL "\
> +			"prepared statements has been reached. Please, deallocate "\
> +			"active statements or increase SQL cache size.");
> +		return -1;
> +	}
> +	struct mh_i32ptr_t *hash = cache->hash;
> +	struct stmt_cache_entry *entry = sql_cache_entry_new(stmt);
> +	if (entry == NULL)
> +		return -1;
> +	const char *sql_str = sql_stmt_query_str(stmt);
> +	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
> +	assert(sql_stmt_cache_find(stmt_id) == NULL);

Good for debug, but what will we have in release, consider we will never
have a chance to test all possible statements?

> +	const struct mh_i32ptr_node_t id_node = { stmt_id, entry };
> +	struct mh_i32ptr_node_t *old_node = NULL;
> +	mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL);
> +	if (i == mh_end(hash)) {
> +		sql_cache_entry_delete(entry);
> +		diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node");
> +		return -1;
> +	}
> +	assert(old_node == NULL);
> +	sql_stmt_cache.mem_used += sql_cache_entry_sizeof(stmt);
> +	return 0;
> +}
> +
> +struct sql_stmt *
> +sql_stmt_cache_find(uint32_t stmt_id)
> +{
> +	struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id);
> +	if (entry == NULL)
> +		return NULL;
> +	return entry->stmt;
> +}
> +
> +int
> +sql_stmt_cache_set_size(size_t size)
> +{
> +	if (sql_stmt_cache.mem_used > size)
> +		sql_stmt_cache_gc();
> +	if (sql_stmt_cache.mem_used > size) {
> +		diag_set(ClientError, ER_SQL_PREPARE, "Can't reduce memory "\
> +			 "limit for SQL prepared statements: please, deallocate "\
> +			 "active statements");
> +		return -1;
> +	}
> +	sql_stmt_cache.mem_quota = size;
> +	return 0;
> +}
> diff --git a/src/box/sql_stmt_cache.h b/src/box/sql_stmt_cache.h
> new file mode 100644
> index 000000000..f3935a27f
> --- /dev/null
> +++ b/src/box/sql_stmt_cache.h
> @@ -0,0 +1,145 @@
> +#ifndef INCLUDES_PREP_STMT_H
> +#define INCLUDES_PREP_STMT_H
> +/*
> + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * 1. Redistributions of source code must retain the above
> + *    copyright notice, this list of conditions and the
> + *    following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above
> + *    copyright notice, this list of conditions and the following
> + *    disclaimer in the documentation and/or other materials
> + *    provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
> + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
> + * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
> + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
> + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
> + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
> + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +#include <stdint.h>
> +#include <stdio.h>
> +
> +#include "small/rlist.h"
> +
> +#if defined(__cplusplus)
> +extern "C" {
> +#endif
> +
> +struct sql_stmt;
> +struct mh_i64ptr_t;
> +
> +struct stmt_cache_entry {
> +	/** Prepared statement itself. */
> +	struct sql_stmt *stmt;
> +	/**
> +	 * Link to the next entry. All statements are to be
> +	 * evicted on the next gc cycle.
> +	 */
> +	struct rlist link;
> +	/**
> +	 * Reference counter. If it is == 0, entry gets
> +	 * into GC queue.
> +	 */
> +	uint32_t refs;
> +};
> +
> +/**
> + * Global prepared statements holder.
> + */
> +struct sql_stmt_cache {
> +	/** Size of memory currently occupied by prepared statements. */
> +	size_t mem_used;
> +	/** Max memory size that can be used for cache. */
> +	size_t mem_quota;
> +	/** Query id -> struct stmt_cahce_entry hash.*/
> +	struct mh_i32ptr_t *hash;
> +	/**
> +	 * After deallocation statements are not deleted, but
> +	 * moved to this list. GC process is triggered only
> +	 * when memory limit has reached. It allows to reduce
> +	 * workload on session's disconnect.
> +	 */
> +	struct rlist gc_queue;
> +	/**
> +	 * Last result of sql_stmt_cache_find() invocation.
> +	 * Since during processing prepared statement it
> +	 * may require to find the same statement several
> +	 * times.
> +	 */
> +	struct stmt_cache_entry *last_found;
> +};
> +
> +/**
> + * Initialize global cache for prepared statements. Called once
> + * during database setup (in sql_init()).
> + */
> +void
> +sql_stmt_cache_init();
> +
> +/**
> + * Erase session local hash: unref statements belong to this
> + * session and deallocate hash itself.
> + * @hash is assumed to be member of struct session @sql_stmts.
> + */
> +void
> +sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash);
> +
> +/**
> + * Add entry corresponding to prepared statement with given ID
> + * to session-local hash and increase its ref counter.
> + * @hash is assumed to be member of struct session @sql_stmts.
> + */
> +int
> +sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id);
> +
> +/**
> + * Prepared statement ID is supposed to be hash value
> + * of the original SQL query string.
> + */
> +uint32_t
> +sql_stmt_calculate_id(const char *sql_str, size_t len);
> +
> +/** Unref prepared statement entry in global holder. */
> +void
> +sql_stmt_unref(uint32_t stmt_id);
> +
> +int
> +sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt);
> +
> +/**
> + * Save prepared statement to the prepared statement cache.
> + * Account cache size change. If the cache is full (i.e. memory
> + * quota is exceeded) diag error is raised. In case of success
> + * return id of prepared statement via output parameter @id.
> + */
> +int
> +sql_stmt_cache_insert(struct sql_stmt *stmt);
> +
> +/** Find entry by SQL string. In case of search fails it returns NULL. */
> +struct sql_stmt *
> +sql_stmt_cache_find(uint32_t stmt_id);
> +
> +
> +/** Set prepared cache size limit. */
> +int
> +sql_stmt_cache_set_size(size_t size);
> +
> +#if defined(__cplusplus)
> +} /* extern "C" { */
> +#endif
> +
> +#endif
> diff --git a/test/app-tap/init_script.result b/test/app-tap/init_script.result
> index 799297ba0..551a0bbeb 100644
> --- a/test/app-tap/init_script.result
> +++ b/test/app-tap/init_script.result
> @@ -31,24 +31,25 @@ box.cfg
>  26	replication_sync_timeout:300
>  27	replication_timeout:1
>  28	slab_alloc_factor:1.05
> -29	strip_core:true
> -30	too_long_threshold:0.5
> -31	vinyl_bloom_fpr:0.05
> -32	vinyl_cache:134217728
> -33	vinyl_dir:.
> -34	vinyl_max_tuple_size:1048576
> -35	vinyl_memory:134217728
> -36	vinyl_page_size:8192
> -37	vinyl_read_threads:1
> -38	vinyl_run_count_per_level:2
> -39	vinyl_run_size_ratio:3.5
> -40	vinyl_timeout:60
> -41	vinyl_write_threads:4
> -42	wal_dir:.
> -43	wal_dir_rescan_delay:2
> -44	wal_max_size:268435456
> -45	wal_mode:write
> -46	worker_pool_threads:4
> +29	sql_cache_size:5242880
> +30	strip_core:true
> +31	too_long_threshold:0.5
> +32	vinyl_bloom_fpr:0.05
> +33	vinyl_cache:134217728
> +34	vinyl_dir:.
> +35	vinyl_max_tuple_size:1048576
> +36	vinyl_memory:134217728
> +37	vinyl_page_size:8192
> +38	vinyl_read_threads:1
> +39	vinyl_run_count_per_level:2
> +40	vinyl_run_size_ratio:3.5
> +41	vinyl_timeout:60
> +42	vinyl_write_threads:4
> +43	wal_dir:.
> +44	wal_dir_rescan_delay:2
> +45	wal_max_size:268435456
> +46	wal_mode:write
> +47	worker_pool_threads:4
>  --
>  -- Test insert from detached fiber
>  --
> diff --git a/test/box/admin.result b/test/box/admin.result
> index 6126f3a97..852c1cde8 100644
> --- a/test/box/admin.result
> +++ b/test/box/admin.result
> @@ -83,6 +83,8 @@ cfg_filter(box.cfg)
>      - 1
>    - - slab_alloc_factor
>      - 1.05
> +  - - sql_cache_size
> +    - 5242880
>    - - strip_core
>      - true
>    - - too_long_threshold
> diff --git a/test/box/cfg.result b/test/box/cfg.result
> index 5370bb870..331f5e986 100644
> --- a/test/box/cfg.result
> +++ b/test/box/cfg.result
> @@ -71,6 +71,8 @@ cfg_filter(box.cfg)
>   |     - 1
>   |   - - slab_alloc_factor
>   |     - 1.05
> + |   - - sql_cache_size
> + |     - 5242880
>   |   - - strip_core
>   |     - true
>   |   - - too_long_threshold
> @@ -170,6 +172,8 @@ cfg_filter(box.cfg)
>   |     - 1
>   |   - - slab_alloc_factor
>   |     - 1.05
> + |   - - sql_cache_size
> + |     - 5242880
>   |   - - strip_core
>   |     - true
>   |   - - too_long_threshold
> @@ -315,6 +319,9 @@ box.cfg{memtx_memory = box.cfg.memtx_memory}
>  box.cfg{vinyl_memory = box.cfg.vinyl_memory}
>   | ---
>   | ...
> +box.cfg{sql_cache_size = box.cfg.sql_cache_size}
> + | ---
> + | ...
>  
>  --------------------------------------------------------------------------------
>  -- Test of default cfg options
> diff --git a/test/box/cfg.test.lua b/test/box/cfg.test.lua
> index 56ccb6767..e6a90d770 100644
> --- a/test/box/cfg.test.lua
> +++ b/test/box/cfg.test.lua
> @@ -51,6 +51,7 @@ box.cfg{replicaset_uuid = '12345678-0123-5678-1234-abcdefabcdef'}
>  
>  box.cfg{memtx_memory = box.cfg.memtx_memory}
>  box.cfg{vinyl_memory = box.cfg.vinyl_memory}
> +box.cfg{sql_cache_size = box.cfg.sql_cache_size}
>  
>  --------------------------------------------------------------------------------
>  -- Test of default cfg options
> diff --git a/test/box/misc.result b/test/box/misc.result
> index d2a20307a..7e5d28b70 100644
> --- a/test/box/misc.result
> +++ b/test/box/misc.result
> @@ -554,6 +554,7 @@ t;
>    203: box.error.BOOTSTRAP_READONLY
>    204: box.error.SQL_FUNC_WRONG_RET_COUNT
>    205: box.error.FUNC_INVALID_RETURN_TYPE
> +  206: box.error.SQL_PREPARE
>  ...
>  test_run:cmd("setopt delimiter ''");
>  ---
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 01/20] sql: remove sql_prepare_v2()

[Nikita Pettik]

On 23 Dec 17:03, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the patch!
> 
> On 20 Dec 15:47, Nikita Pettik wrote:
> >  			continue;
> > diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> > index 0ecc676e2..35e81212d 100644
> > --- a/src/box/sql/prepare.c
> > +++ b/src/box/sql/prepare.c
> > @@ -204,36 +204,12 @@ sqlReprepare(Vdbe * p)
> >  	return 0;
> >  }
> >  
> > -/*
> > - * Two versions of the official API.  Legacy and new use.  In the legacy
> > - * version, the original SQL text is not saved in the prepared statement
> > - * and so if a schema change occurs, an error is returned by
> > - * sql_step().  In the new version, the original SQL text is retained
> > - * and the statement is automatically recompiled if an schema change
> > - * occurs.
> > - */
> > -int
> > -sql_prepare(sql * db,		/* Database handle. */
> > -		const char *zSql,	/* UTF-8 encoded SQL statement. */
> > -		int nBytes,		/* Length of zSql in bytes. */
> > -		sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> > -		const char **pzTail)	/* OUT: End of parsed string */
> > -{
> > -	int rc = sqlPrepare(db, zSql, nBytes, 0, 0, ppStmt, pzTail);
> > -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> > -	return rc;
> > -}
> > -
> >  int
> > -sql_prepare_v2(sql * db,	/* Database handle. */
> > -		   const char *zSql,	/* UTF-8 encoded SQL statement. */
> > -		   int nBytes,	/* Length of zSql in bytes. */
> > -		   sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> > -		   const char **pzTail	/* OUT: End of parsed string */
> > -    )
> > +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> > +	    const char **sql_tail)
> >  {
> > -	int rc = sqlPrepare(db, zSql, nBytes, 1, 0, ppStmt, pzTail);
> > -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> > +	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
> > +	assert(rc == 0 || stmt == NULL || *stmt == NULL);
> >  	return rc;
> >  }
> >  
> > diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> > index 2594b73e0..7bd952a17 100644
> > --- a/src/box/sql/sqlInt.h
> > +++ b/src/box/sql/sqlInt.h
> > @@ -468,21 +468,18 @@ typedef void (*sql_destructor_type) (void *);
> >  #define SQL_STATIC      ((sql_destructor_type)0)
> >  #define SQL_TRANSIENT   ((sql_destructor_type)-1)
> >  
> > +/**
> > + * Prepare (compile into VDBE byte-code) statement.
> 
> Could you please extend the description with details on SQL text
> preservance and recmopilation, same as it was in old version?

It was obsolete and non-relevant (related purely to SQLite) comment.
It has nothing to do with current sql_prepare() behaviour.
 
> > + *
> > + * @param db Database handle.
> > + * @param sql UTF-8 encoded SQL statement.
> > + * @param length Length of @param sql in bytes.
> > + * @param[out] stmt A pointer to the prepared statement.
> > + * @param[out] sql_tail End of parsed string.
> > + */
> >  int
> > -sql_prepare(sql * db,	/* Database handle */
> > -		const char *zSql,	/* SQL statement, UTF-8 encoded */
> > -		int nByte,	/* Maximum length of zSql in bytes. */
> > -		sql_stmt ** ppStmt,	/* OUT: Statement handle */
> > -		const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> > -	);
> > -
> > -int
> > -sql_prepare_v2(sql * db,	/* Database handle */
> > -		   const char *zSql,	/* SQL statement, UTF-8 encoded */
> > -		   int nByte,	/* Maximum length of zSql in bytes. */
> > -		   sql_stmt ** ppStmt,	/* OUT: Statement handle */
> > -		   const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> > -	);
> > +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> > +	    const char **sql_tail);
> >  
> >  int
> >  sql_step(sql_stmt *);
> > diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> > index 685212d91..12449d3bc 100644
> > --- a/src/box/sql/vdbeapi.c
> > +++ b/src/box/sql/vdbeapi.c
> > @@ -452,7 +452,7 @@ sqlStep(Vdbe * p)
> >  		checkProfileCallback(db, p);
> >  
> >  	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
> > -		/* If this statement was prepared using sql_prepare_v2(), and an
> > +		/* If this statement was prepared using sql_prepare(), and an
> >  		 * error has occurred, then return an error.
> >  		 */
> >  		if (p->is_aborted)
> > -- 
> > 2.15.1
> > 

Re: [Tarantool-patches] [PATCH v3 02/20] sql: refactor sql_prepare() and sqlPrepare()

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> - Removed saveSqlFlag as argument from sqlPrepare(). It was used to
>   indicate that its caller is sql_prepare_v2() not sql_prepare().
>   Since in previous commit we've left only one version of this function
>   let's remove this flag at all.
> 
> - Removed struct db from list of sql_prepare() arguments. There's one
>   global database handler and it can be obtained by sql_get() call.
>   Hence, it makes no sense to pass around this argument.
> 
> Needed for #3292
> ---
>  src/box/execute.c     |  3 +--
>  src/box/sql/analyze.c | 16 +++++++---------
>  src/box/sql/legacy.c  |  2 +-
>  src/box/sql/prepare.c | 10 ++++------
>  src/box/sql/sqlInt.h  |  3 +--
>  src/box/sql/vdbe.h    |  2 +-
>  src/box/sql/vdbeInt.h |  1 -
>  src/box/sql/vdbeapi.c |  2 +-
>  src/box/sql/vdbeaux.c |  5 +----
>  9 files changed, 17 insertions(+), 27 deletions(-)
> 
> diff --git a/src/box/execute.c b/src/box/execute.c
> index 130a3f675..0b21386b5 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -442,8 +442,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
>  			struct region *region)
>  {
>  	struct sql_stmt *stmt;
> -	struct sql *db = sql_get();
> -	if (sql_prepare(db, sql, len, &stmt, NULL) != 0)
> +	if (sql_prepare(sql, len, &stmt, NULL) != 0)
>  		return -1;
>  	assert(stmt != NULL);
>  	port_sql_create(port, stmt);
> diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
> index b9858c8d6..e43011dd0 100644
> --- a/src/box/sql/analyze.c
> +++ b/src/box/sql/analyze.c
> @@ -1336,14 +1336,13 @@ sample_compare(const void *a, const void *b, void *arg)
>   * statistics (i.e. arrays lt, dt, dlt and avg_eq). 'load' query
>   * is needed for
>   *
> - * @param db Database handler.
>   * @param sql_select_prepare SELECT statement, see above.
>   * @param sql_select_load SELECT statement, see above.
>   * @param[out] stats Statistics is saved here.
>   * @retval 0 on success, -1 otherwise.
>   */
>  static int
> -load_stat_from_space(struct sql *db, const char *sql_select_prepare,
> +load_stat_from_space(const char *sql_select_prepare,
>  		     const char *sql_select_load, struct index_stat *stats)
>  {
>  	struct index **indexes = NULL;
> @@ -1359,7 +1358,7 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
>  		}
>  	}
>  	sql_stmt *stmt = NULL;
> -	int rc = sql_prepare(db, sql_select_prepare, -1, &stmt, 0);
> +	int rc = sql_prepare(sql_select_prepare, -1, &stmt, 0);
>  	if (rc)
>  		goto finalize;
>  	uint32_t current_idx_count = 0;
> @@ -1427,7 +1426,7 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
>  	rc = sql_finalize(stmt);
>  	if (rc)
>  		goto finalize;
> -	rc = sql_prepare(db, sql_select_load, -1, &stmt, 0);
> +	rc = sql_prepare(sql_select_load, -1, &stmt, 0);
>  	if (rc)
>  		goto finalize;
>  	struct index *prev_index = NULL;
> @@ -1505,12 +1504,11 @@ load_stat_from_space(struct sql *db, const char *sql_select_prepare,
>  }
>  
>  static int
> -load_stat_to_index(struct sql *db, const char *sql_select_load,
> -		   struct index_stat **stats)
> +load_stat_to_index(const char *sql_select_load, struct index_stat **stats)
>  {
>  	assert(stats != NULL && *stats != NULL);
>  	struct sql_stmt *stmt = NULL;
> -	if (sql_prepare(db, sql_select_load, -1, &stmt, 0) != 0)
> +	if (sql_prepare(sql_select_load, -1, &stmt, 0) != 0)
>  		return -1;
>  	uint32_t current_idx_count = 0;
>  	while (sql_step(stmt) == SQL_ROW) {
> @@ -1696,7 +1694,7 @@ sql_analysis_load(struct sql *db)
>  	const char *load_query = "SELECT \"tbl\",\"idx\",\"neq\",\"nlt\","
>  				 "\"ndlt\",\"sample\" FROM \"_sql_stat4\"";
>  	/* Load the statistics from the _sql_stat4 table. */
> -	if (load_stat_from_space(db, init_query, load_query, stats) != 0)
> +	if (load_stat_from_space(init_query, load_query, stats) != 0)
>  		goto fail;
>  	/*
>  	 * Now we have complete statistics for each index
> @@ -1739,7 +1737,7 @@ sql_analysis_load(struct sql *db)
>  	 */
>  	const char *order_query = "SELECT \"tbl\",\"idx\" FROM "
>  				  "\"_sql_stat4\" GROUP BY \"tbl\",\"idx\"";
> -	if (load_stat_to_index(db, order_query, heap_stats) == 0)
> +	if (load_stat_to_index(order_query, heap_stats) == 0)
>  		return box_txn_commit();
>  fail:
>  	box_txn_rollback();
> diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
> index bfd1e32b9..16507b334 100644
> --- a/src/box/sql/legacy.c
> +++ b/src/box/sql/legacy.c
> @@ -70,7 +70,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
>  		char **azVals = 0;
>  
>  		pStmt = 0;
> -		rc = sql_prepare(db, zSql, -1, &pStmt, &zLeftover);
> +		rc = sql_prepare(zSql, -1, &pStmt, &zLeftover);
>  		assert(rc == 0 || pStmt == NULL);
>  		if (rc != 0)
>  			continue;
> diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> index 35e81212d..520b52d64 100644
> --- a/src/box/sql/prepare.c
> +++ b/src/box/sql/prepare.c
> @@ -46,7 +46,6 @@ static int
>  sqlPrepare(sql * db,	/* Database handle. */
>  	       const char *zSql,	/* UTF-8 encoded SQL statement. */
>  	       int nBytes,	/* Length of zSql in bytes. */
> -	       int saveSqlFlag,	/* True to copy SQL text into the sql_stmt */
>  	       Vdbe * pReprepare,	/* VM being reprepared */
>  	       sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
>  	       const char **pzTail	/* OUT: End of parsed string */
> @@ -156,8 +155,7 @@ sqlPrepare(sql * db,	/* Database handle. */
>  
>  	if (db->init.busy == 0) {
>  		Vdbe *pVdbe = sParse.pVdbe;
> -		sqlVdbeSetSql(pVdbe, zSql, (int)(sParse.zTail - zSql),
> -				  saveSqlFlag);
> +		sqlVdbeSetSql(pVdbe, zSql, (int)(sParse.zTail - zSql));
>  	}
>  	if (sParse.pVdbe != NULL && (rc != 0 || db->mallocFailed)) {
>  		sqlVdbeFinalize(sParse.pVdbe);
> @@ -192,7 +190,7 @@ sqlReprepare(Vdbe * p)
>  	zSql = sql_sql((sql_stmt *) p);
>  	assert(zSql != 0);	/* Reprepare only called for prepare_v2() statements */
>  	db = sqlVdbeDb(p);
> -	if (sqlPrepare(db, zSql, -1, 0, p, &pNew, 0) != 0) {
> +	if (sqlPrepare(db, zSql, -1, p, &pNew, 0) != 0) {
>  		assert(pNew == 0);
>  		return -1;
>  	}
> @@ -205,10 +203,10 @@ sqlReprepare(Vdbe * p)
>  }
>  
>  int
> -sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> +sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
>  	    const char **sql_tail)
>  {
> -	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
> +	int rc = sqlPrepare(sql_get(), sql, length, 0, stmt, sql_tail);
>  	assert(rc == 0 || stmt == NULL || *stmt == NULL);
>  	return rc;
>  }
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 7bd952a17..ac1d8ce42 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -471,14 +471,13 @@ typedef void (*sql_destructor_type) (void *);
>  /**
>   * Prepare (compile into VDBE byte-code) statement.
>   *
> - * @param db Database handle.
>   * @param sql UTF-8 encoded SQL statement.
>   * @param length Length of @param sql in bytes.
>   * @param[out] stmt A pointer to the prepared statement.
>   * @param[out] sql_tail End of parsed string.
>   */
>  int
> -sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> +sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
>  	    const char **sql_tail);
>  
>  int
> diff --git a/src/box/sql/vdbe.h b/src/box/sql/vdbe.h
> index 582d48a1f..573577355 100644
> --- a/src/box/sql/vdbe.h
> +++ b/src/box/sql/vdbe.h
> @@ -251,7 +251,7 @@ void sqlVdbeSetNumCols(Vdbe *, int);
>  int sqlVdbeSetColName(Vdbe *, int, int, const char *, void (*)(void *));
>  void sqlVdbeCountChanges(Vdbe *);
>  sql *sqlVdbeDb(Vdbe *);
> -void sqlVdbeSetSql(Vdbe *, const char *z, int n, int);
> +void sqlVdbeSetSql(Vdbe *, const char *z, int n);
>  void sqlVdbeSwap(Vdbe *, Vdbe *);
>  VdbeOp *sqlVdbeTakeOpArray(Vdbe *, int *, int *);
>  sql_value *sqlVdbeGetBoundValue(Vdbe *, int, u8);
> diff --git a/src/box/sql/vdbeInt.h b/src/box/sql/vdbeInt.h
> index 0f32b4cd6..078ebc34e 100644
> --- a/src/box/sql/vdbeInt.h
> +++ b/src/box/sql/vdbeInt.h
> @@ -421,7 +421,6 @@ struct Vdbe {
>  	bft explain:2;		/* True if EXPLAIN present on SQL command */
>  	bft changeCntOn:1;	/* True to update the change-counter */
>  	bft runOnlyOnce:1;	/* Automatically expire on reset */
> -	bft isPrepareV2:1;	/* True if prepared with prepare_v2() */
>  	u32 aCounter[5];	/* Counters used by sql_stmt_status() */
>  	char *zSql;		/* Text of the SQL statement that generated this */
>  	void *pFree;		/* Free this when deleting the vdbe */
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 12449d3bc..db7936e78 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -451,7 +451,7 @@ sqlStep(Vdbe * p)
>  	if (rc != SQL_ROW)
>  		checkProfileCallback(db, p);
>  
> -	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
> +	if (rc != SQL_ROW && rc != SQL_DONE) {
>  		/* If this statement was prepared using sql_prepare(), and an
>  		 * error has occurred, then return an error.
>  		 */
> diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c
> index a1d658648..619105820 100644
> --- a/src/box/sql/vdbeaux.c
> +++ b/src/box/sql/vdbeaux.c
> @@ -89,14 +89,12 @@ sql_vdbe_prepare(struct Vdbe *vdbe)
>   * Remember the SQL string for a prepared statement.
>   */
>  void
> -sqlVdbeSetSql(Vdbe * p, const char *z, int n, int isPrepareV2)
> +sqlVdbeSetSql(Vdbe * p, const char *z, int n)
>  {
> -	assert(isPrepareV2 == 1 || isPrepareV2 == 0);
>  	if (p == 0)
>  		return;
>  	assert(p->zSql == 0);
>  	p->zSql = sqlDbStrNDup(p->db, z, n);
> -	p->isPrepareV2 = (u8) isPrepareV2;
>  }
>  
>  /*
> @@ -120,7 +118,6 @@ sqlVdbeSwap(Vdbe * pA, Vdbe * pB)
>  	zTmp = pA->zSql;
>  	pA->zSql = pB->zSql;
>  	pB->zSql = zTmp;
> -	pB->isPrepareV2 = pA->isPrepareV2;
>  }
>  
>  /*
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 03/20] sql: move sql_prepare() declaration to box/execute.h

[Sergey Ostanevich]

Hi!

Thanks for the patch. LGTM.

Regards,
Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> We are going to split sql_prepare_and_execute() into several explicit
> and logically separated steps:
> 
> 1. sql_prepare() -- compile VDBE byte-code
> 2. sql_bind() -- bind variables (if there are any)
> 3. sql_execute() -- query (byte-code) execution in virtual machine
> 
> For instance, for dry-run we are interested only in query preparation.
> Contrary, if we had prepared statement cache, we could skip query
> preparation and handle only bind and execute steps.
> 
> To avoid inclusion of sql/sqlInt.h header (which gathers almost all SQL
> specific functions and constants) let's move sql_prepare() to
> box/execute.h header (which already holds sql_prepare_and_execute()).
> 
> Needed for #3292
> ---
>  src/box/execute.h     | 12 ++++++++++++
>  src/box/sql/analyze.c |  1 +
>  src/box/sql/legacy.c  |  1 +
>  src/box/sql/sqlInt.h  | 12 ------------
>  4 files changed, 14 insertions(+), 12 deletions(-)
> 
> diff --git a/src/box/execute.h b/src/box/execute.h
> index a2fd4d1b7..a6000c08b 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -89,6 +89,18 @@ struct port_sql {
>  
>  extern const struct port_vtab port_sql_vtab;
>  
> +/**
> + * Prepare (compile into VDBE byte-code) statement.
> + *
> + * @param sql UTF-8 encoded SQL statement.
> + * @param length Length of @param sql in bytes.
> + * @param[out] stmt A pointer to the prepared statement.
> + * @param[out] sql_tail End of parsed string.
> + */
> +int
> +sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
> +	    const char **sql_tail);
> +
>  #if defined(__cplusplus)
>  } /* extern "C" { */
>  #endif
> diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
> index e43011dd0..00ca15413 100644
> --- a/src/box/sql/analyze.c
> +++ b/src/box/sql/analyze.c
> @@ -106,6 +106,7 @@
>   */
>  
>  #include "box/box.h"
> +#include "box/execute.h"
>  #include "box/index.h"
>  #include "box/key_def.h"
>  #include "box/schema.h"
> diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
> index 16507b334..e3a2c77ca 100644
> --- a/src/box/sql/legacy.c
> +++ b/src/box/sql/legacy.c
> @@ -37,6 +37,7 @@
>   */
>  
>  #include "sqlInt.h"
> +#include "box/execute.h"
>  #include "box/session.h"
>  
>  /*
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index ac1d8ce42..3ca10778e 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -468,18 +468,6 @@ typedef void (*sql_destructor_type) (void *);
>  #define SQL_STATIC      ((sql_destructor_type)0)
>  #define SQL_TRANSIENT   ((sql_destructor_type)-1)
>  
> -/**
> - * Prepare (compile into VDBE byte-code) statement.
> - *
> - * @param sql UTF-8 encoded SQL statement.
> - * @param length Length of @param sql in bytes.
> - * @param[out] stmt A pointer to the prepared statement.
> - * @param[out] sql_tail End of parsed string.
> - */
> -int
> -sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
> -	    const char **sql_tail);
> -
>  int
>  sql_step(sql_stmt *);
>  
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 04/20] sql: rename sqlPrepare() to sql_stmt_compile()

[Sergey Ostanevich]

Hi! 

Thanks for the patch! Some comments below

On 20 Dec 15:47, Nikita Pettik wrote:
> sql_prepare() is going not only to compile statement, but also to save it
> to the prepared statement cache. So we'd better rename sqlPrepare()
> which is static wrapper around sql_prepare() and make it non-static.
> Where it is possible let's use sql_stmt_compile() instead of sql_prepare().
> 
> Needed for #2592
> ---
>  src/box/execute.c     |  2 +-
>  src/box/sql/analyze.c |  6 +++---
>  src/box/sql/legacy.c  |  2 +-
>  src/box/sql/prepare.c | 21 ++++++---------------
>  src/box/sql/sqlInt.h  | 14 ++++++++++++++
>  src/box/sql/vdbeapi.c |  2 +-
>  6 files changed, 26 insertions(+), 21 deletions(-)
> 
> diff --git a/src/box/execute.c b/src/box/execute.c
> index 0b21386b5..af66447b5 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -442,7 +442,7 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
>  			struct region *region)
>  {
>  	struct sql_stmt *stmt;
> -	if (sql_prepare(sql, len, &stmt, NULL) != 0)
> +	if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
>  		return -1;
>  	assert(stmt != NULL);
>  	port_sql_create(port, stmt);
> diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
> index 00ca15413..42e2a1a2f 100644
> --- a/src/box/sql/analyze.c
> +++ b/src/box/sql/analyze.c
> @@ -1359,7 +1359,7 @@ load_stat_from_space(const char *sql_select_prepare,
>  		}
>  	}
>  	sql_stmt *stmt = NULL;
> -	int rc = sql_prepare(sql_select_prepare, -1, &stmt, 0);
> +	int rc = sql_stmt_compile(sql_select_prepare, -1, NULL, &stmt, 0);
>  	if (rc)
>  		goto finalize;
>  	uint32_t current_idx_count = 0;
> @@ -1427,7 +1427,7 @@ load_stat_from_space(const char *sql_select_prepare,
>  	rc = sql_finalize(stmt);
>  	if (rc)
>  		goto finalize;
> -	rc = sql_prepare(sql_select_load, -1, &stmt, 0);
> +	rc = sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0);
>  	if (rc)
>  		goto finalize;
>  	struct index *prev_index = NULL;
> @@ -1509,7 +1509,7 @@ load_stat_to_index(const char *sql_select_load, struct index_stat **stats)
>  {
>  	assert(stats != NULL && *stats != NULL);
>  	struct sql_stmt *stmt = NULL;
> -	if (sql_prepare(sql_select_load, -1, &stmt, 0) != 0)
> +	if (sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0) != 0)
>  		return -1;
>  	uint32_t current_idx_count = 0;
>  	while (sql_step(stmt) == SQL_ROW) {
> diff --git a/src/box/sql/legacy.c b/src/box/sql/legacy.c
> index e3a2c77ca..458afd7f8 100644
> --- a/src/box/sql/legacy.c
> +++ b/src/box/sql/legacy.c
> @@ -71,7 +71,7 @@ sql_exec(sql * db,	/* The database on which the SQL executes */
>  		char **azVals = 0;
>  
>  		pStmt = 0;
> -		rc = sql_prepare(zSql, -1, &pStmt, &zLeftover);
> +		rc = sql_stmt_compile(zSql, -1, NULL, &pStmt, &zLeftover);
>  		assert(rc == 0 || pStmt == NULL);
>  		if (rc != 0)
>  			continue;
> diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> index 520b52d64..73d6866a4 100644
> --- a/src/box/sql/prepare.c
> +++ b/src/box/sql/prepare.c
> @@ -39,18 +39,11 @@
>  #include "box/space.h"
>  #include "box/session.h"
>  
> -/*
> - * Compile the UTF-8 encoded SQL statement zSql into a statement handle.
> - */
> -static int
> -sqlPrepare(sql * db,	/* Database handle. */
> -	       const char *zSql,	/* UTF-8 encoded SQL statement. */
> -	       int nBytes,	/* Length of zSql in bytes. */
> -	       Vdbe * pReprepare,	/* VM being reprepared */
> -	       sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> -	       const char **pzTail	/* OUT: End of parsed string */
> -    )
> +int
> +sql_stmt_compile(const char *zSql, int nBytes, struct Vdbe *pReprepare,
> +		 sql_stmt **ppStmt, const char **pzTail)
>  {
> +	struct sql *db = sql_get();
>  	int rc = 0;	/* Result code */
>  	Parse sParse;		/* Parsing context */
>  	sql_parser_create(&sParse, db, current_session()->sql_flags);
> @@ -185,12 +178,10 @@ sqlReprepare(Vdbe * p)
>  {
>  	sql_stmt *pNew;
>  	const char *zSql;
> -	sql *db;
>  
>  	zSql = sql_sql((sql_stmt *) p);
>  	assert(zSql != 0);	/* Reprepare only called for prepare_v2() statements */

Comment is irrelevant, prepare_v2 is viped out?

> -	db = sqlVdbeDb(p);
> -	if (sqlPrepare(db, zSql, -1, p, &pNew, 0) != 0) {
> +	if (sql_stmt_compile(zSql, -1, p, &pNew, 0) != 0) {
>  		assert(pNew == 0);
>  		return -1;
>  	}
> @@ -206,7 +197,7 @@ int
>  sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
>  	    const char **sql_tail)
>  {
> -	int rc = sqlPrepare(sql_get(), sql, length, 0, stmt, sql_tail);
> +	int rc = sql_stmt_compile(sql, length, 0, stmt, sql_tail);
>  	assert(rc == 0 || stmt == NULL || *stmt == NULL);
>  	return rc;
>  }
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 3ca10778e..03deb733c 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -468,6 +468,20 @@ typedef void (*sql_destructor_type) (void *);
>  #define SQL_STATIC      ((sql_destructor_type)0)
>  #define SQL_TRANSIENT   ((sql_destructor_type)-1)
>  
> +/**
> + * Compile the UTF-8 encoded SQL statement into
> + * a statement handle (struct Vdbe).
> + *
> + * @param sql UTF-8 encoded SQL statement.
> + * @param sql_len Length of @sql in bytes.
> + * @param re_prepared VM being re-compiled. Can be NULL.
> + * @param[out] stmt A pointer to the compiled statement.
> + * @param[out] sql_tail End of parsed string.
> + */
> +int
> +sql_stmt_compile(const char *sql, int bytes_count, struct Vdbe *re_prepared,
> +		 sql_stmt **stmt, const char **sql_tail);
> +
>  int
>  sql_step(sql_stmt *);
>  
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index db7936e78..ab8441bc5 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -71,7 +71,7 @@ invokeProfileCallback(sql * db, Vdbe * p)
>  
>  /*
>   * The following routine destroys a virtual machine that is created by
> - * the sql_compile() routine. The integer returned is an SQL_
> + * the sql_stmt_compile() routine. The integer returned is an SQL_
>   * success/failure code that describes the result of executing the virtual
>   * machine.
>   */
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 05/20] sql: rename sql_finalize() to sql_stmt_finalize()

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos.

On 20 Dec 15:47, Nikita Pettik wrote:
> Let's follow unified naming rules for SQL high level API which
> manipulates on statements objects. To be more precise, let's use
> 'sql_stmt_' prefix for interface functions operating on statement
> handles.
> ---
>  src/box/bind.c          | 2 +-
>  src/box/ck_constraint.c | 4 ++--
>  src/box/execute.c       | 2 +-
>  src/box/lua/execute.c   | 2 +-
>  src/box/sql/analyze.c   | 6 +++---
>  src/box/sql/sqlInt.h    | 2 +-
>  src/box/sql/vdbe.c      | 4 ++--
>  src/box/sql/vdbeapi.c   | 2 +-
>  8 files changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/src/box/bind.c b/src/box/bind.c
> index 7eea9fcc8..bbc1f56df 100644
> --- a/src/box/bind.c
> +++ b/src/box/bind.c
> @@ -180,7 +180,7 @@ sql_bind_column(struct sql_stmt *stmt, const struct sql_bind *p,
>  		 * Parameters are allocated within message pack,
>  		 * received from the iproto thread. IProto thread
>  		 * now is waiting for the response and it will not
> -		 * free the packet until sql_finalize. So
> +		 * free the packet until sql_stmt_finalize. So
>  		 * there is no need to copy the packet and we can
>  		 * use SQL_STATIC.
>  		 */
> diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
> index a2c66ce00..551bdd397 100644
> --- a/src/box/ck_constraint.c
> +++ b/src/box/ck_constraint.c
> @@ -141,7 +141,7 @@ ck_constraint_program_compile(struct ck_constraint_def *ck_constraint_def,
>  		diag_set(ClientError, ER_CREATE_CK_CONSTRAINT,
>  			 ck_constraint_def->name,
>  			 box_error_message(box_error_last()));
> -		sql_finalize((struct sql_stmt *) v);
> +		sql_stmt_finalize((struct sql_stmt *) v);
>  		return NULL;
>  	}
>  	return (struct sql_stmt *) v;
> @@ -254,7 +254,7 @@ error:
>  void
>  ck_constraint_delete(struct ck_constraint *ck_constraint)
>  {
> -	sql_finalize(ck_constraint->stmt);
> +	sql_stmt_finalize(ck_constraint->stmt);
>  	ck_constraint_def_delete(ck_constraint->def);
>  	TRASH(ck_constraint);
>  	free(ck_constraint);
> diff --git a/src/box/execute.c b/src/box/execute.c
> index af66447b5..fb83e1194 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -100,7 +100,7 @@ static void
>  port_sql_destroy(struct port *base)
>  {
>  	port_tuple_vtab.destroy(base);
> -	sql_finalize(((struct port_sql *)base)->stmt);
> +	sql_stmt_finalize(((struct port_sql *)base)->stmt);
>  }
>  
>  const struct port_vtab port_sql_vtab = {
> diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
> index ffa3d4d2e..68adacf72 100644
> --- a/src/box/lua/execute.c
> +++ b/src/box/lua/execute.c
> @@ -217,7 +217,7 @@ lua_sql_bind_list_decode(struct lua_State *L, struct sql_bind **out_bind,
>  	size_t size = sizeof(struct sql_bind) * bind_count;
>  	/*
>  	 * Memory allocated here will be freed in
> -	 * sql_finalize() or in txn_commit()/txn_rollback() if
> +	 * sql_stmt_finalize() or in txn_commit()/txn_rollback() if
>  	 * there is an active transaction.
>  	 */
>  	struct sql_bind *bind = (struct sql_bind *) region_alloc(region, size);
> diff --git a/src/box/sql/analyze.c b/src/box/sql/analyze.c
> index 42e2a1a2f..f74f9b358 100644
> --- a/src/box/sql/analyze.c
> +++ b/src/box/sql/analyze.c
> @@ -1424,7 +1424,7 @@ load_stat_from_space(const char *sql_select_prepare,
>  		current_idx_count++;
>  
>  	}
> -	rc = sql_finalize(stmt);
> +	rc = sql_stmt_finalize(stmt);
>  	if (rc)
>  		goto finalize;
>  	rc = sql_stmt_compile(sql_select_load, -1, NULL, &stmt, 0);
> @@ -1475,7 +1475,7 @@ load_stat_from_space(const char *sql_select_prepare,
>  		sample->sample_key = region_alloc(&fiber()->gc,
>  						  sample->key_size);
>  		if (sample->sample_key == NULL) {
> -			sql_finalize(stmt);
> +			sql_stmt_finalize(stmt);
>  			rc = -1;
>  			diag_set(OutOfMemory, sample->key_size,
>  				 "region", "sample_key");
> @@ -1488,7 +1488,7 @@ load_stat_from_space(const char *sql_select_prepare,
>  		}
>  		stats[current_idx_count].sample_count++;
>  	}
> -	rc = sql_finalize(stmt);
> +	rc = sql_stmt_finalize(stmt);
>  	if (rc == 0 && prev_index != NULL)
>  		init_avg_eq(prev_index, &stats[current_idx_count]);
>  	assert(current_idx_count <= index_count);
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 03deb733c..cf0b946f1 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -521,7 +521,7 @@ sql_column_value(sql_stmt *,
>  		     int iCol);
>  
>  int
> -sql_finalize(sql_stmt * pStmt);
> +sql_stmt_finalize(sql_stmt * pStmt);
>  
>  /*
>   * Terminate the current execution of an SQL statement and reset
> diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c
> index ab86be9a9..336fd4a52 100644
> --- a/src/box/sql/vdbe.c
> +++ b/src/box/sql/vdbe.c
> @@ -1084,7 +1084,7 @@ case OP_Yield: {            /* in1, jump */
>   * automatically.
>   *
>   * P1 is the result code returned by sql_exec(),
> - * sql_reset(), or sql_finalize().  For a normal halt,
> + * sql_reset(), or sql_stmt_finalize().  For a normal halt,
>   * this should be 0.
>   * For errors, it can be some other value.  If P1!=0 then P2 will
>   * determine whether or not to rollback the current transaction.
> @@ -2887,7 +2887,7 @@ case OP_MakeRecord: {
>  	 * memory shouldn't be reused until it is written into WAL.
>  	 *
>  	 * However, if memory for ephemeral space is allocated
> -	 * on region, it will be freed only in sql_finalize()
> +	 * on region, it will be freed only in sql_stmt_finalize()
>  	 * routine.
>  	 */
>  	if (bIsEphemeral) {
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index ab8441bc5..7463fb9d7 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -76,7 +76,7 @@ invokeProfileCallback(sql * db, Vdbe * p)
>   * machine.
>   */
>  int
> -sql_finalize(sql_stmt * pStmt)
> +sql_stmt_finalize(sql_stmt * pStmt)
>  {
>  	if (pStmt == NULL)
>  		return 0;
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 06/20] sql: rename sql_reset() to sql_stmt_reset()

[Sergey Ostanevich]

Hi!

10x for the patch, LGTM.

Sergos


On 20 Dec 15:47, Nikita Pettik wrote:
> ---
>  src/box/ck_constraint.c | 2 +-
>  src/box/sql/sqlInt.h    | 2 +-
>  src/box/sql/vdbe.c      | 2 +-
>  src/box/sql/vdbeapi.c   | 4 ++--
>  4 files changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
> index 551bdd397..bc2a5e8f4 100644
> --- a/src/box/ck_constraint.c
> +++ b/src/box/ck_constraint.c
> @@ -173,7 +173,7 @@ ck_constraint_program_run(struct ck_constraint *ck_constraint,
>  	 * Get VDBE execution state and reset VM to run it
>  	 * next time.
>  	 */
> -	return sql_reset(ck_constraint->stmt);
> +	return sql_stmt_reset(ck_constraint->stmt);
>  }
>  
>  int
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index cf0b946f1..b1e4ac2fa 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -532,7 +532,7 @@ sql_stmt_finalize(sql_stmt * pStmt);
>   * @retval sql_ret_code Error code on error.
>   */
>  int
> -sql_reset(struct sql_stmt *stmt);
> +sql_stmt_reset(struct sql_stmt *stmt);
>  
>  int
>  sql_exec(sql *,	/* An open database */
> diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c
> index 336fd4a52..8da007b4d 100644
> --- a/src/box/sql/vdbe.c
> +++ b/src/box/sql/vdbe.c
> @@ -1084,7 +1084,7 @@ case OP_Yield: {            /* in1, jump */
>   * automatically.
>   *
>   * P1 is the result code returned by sql_exec(),
> - * sql_reset(), or sql_stmt_finalize().  For a normal halt,
> + * sql_stmt_reset(), or sql_stmt_finalize().  For a normal halt,
>   * this should be 0.
>   * For errors, it can be some other value.  If P1!=0 then P2 will
>   * determine whether or not to rollback the current transaction.
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 7463fb9d7..b6bf9aa81 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -88,7 +88,7 @@ sql_stmt_finalize(sql_stmt * pStmt)
>  }
>  
>  int
> -sql_reset(sql_stmt * pStmt)
> +sql_stmt_reset(sql_stmt *pStmt)
>  {
>  	assert(pStmt != NULL);
>  	struct Vdbe *v = (Vdbe *) pStmt;
> @@ -414,7 +414,7 @@ sqlStep(Vdbe * p)
>  
>  	assert(p);
>  	if (p->magic != VDBE_MAGIC_RUN)
> -		sql_reset((sql_stmt *) p);
> +		sql_stmt_reset((sql_stmt *) p);
>  
>  	/* Check that malloc() has not failed. If it has, return early. */
>  	db = p->db;
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 07/20] sql: move sql_stmt_finalize() to execute.h

[Sergey Ostanevich]

Hi!

Thanks for the patch, just one nit.
LGTM

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> We are going to introduce prepared statement cache. On statement's
> deallocation we should release all resources which is done by
> sql_finalize(). Now it is declared in sql/sqlInt.h header, which

  ^^^
sql_stmt_finalize() 

> accumulates almost all SQL related functions. To avoid including such a
> huge header to use single function, let's move its signature to
> box/execute.h
> 
> Need for #2592
> ---
>  src/box/ck_constraint.c | 1 +
>  src/box/execute.h       | 3 +++
>  src/box/sql/sqlInt.h    | 3 ---
>  3 files changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/src/box/ck_constraint.c b/src/box/ck_constraint.c
> index bc2a5e8f4..ff3f05587 100644
> --- a/src/box/ck_constraint.c
> +++ b/src/box/ck_constraint.c
> @@ -29,6 +29,7 @@
>   * SUCH DAMAGE.
>   */
>  #include "box/session.h"
> +#include "execute.h"
>  #include "bind.h"
>  #include "ck_constraint.h"
>  #include "errcode.h"
> diff --git a/src/box/execute.h b/src/box/execute.h
> index a6000c08b..ce1e7a67d 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -89,6 +89,9 @@ struct port_sql {
>  
>  extern const struct port_vtab port_sql_vtab;
>  
> +int
> +sql_stmt_finalize(struct sql_stmt *stmt);
> +
>  /**
>   * Prepare (compile into VDBE byte-code) statement.
>   *
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index b1e4ac2fa..24da3ca11 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -520,9 +520,6 @@ sql_value *
>  sql_column_value(sql_stmt *,
>  		     int iCol);
>  
> -int
> -sql_stmt_finalize(sql_stmt * pStmt);
> -
>  /*
>   * Terminate the current execution of an SQL statement and reset
>   * it back to its starting state so that it can be reused.
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 08/20] port: increase padding of struct port

[Sergey Ostanevich]

Hi!

Thanks for the patch!

See my comment below.

Sergos


On 20 Dec 15:47, Nikita Pettik wrote:
> We are going to extend context of struct port_sql. One already inherits
> struct port_tuple, which makes it size barely fits into 48 bytes of
> padding of basic structure (struct port). Hence, let's increase padding
> a bit to be able to add at least one more member to struct port_sql.
> ---
>  src/lib/core/port.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/lib/core/port.h b/src/lib/core/port.h
> index d61342287..bfdfa4656 100644
> --- a/src/lib/core/port.h
> +++ b/src/lib/core/port.h
> @@ -122,7 +122,7 @@ struct port {
>  	 * Implementation dependent content. Needed to declare
>  	 * an abstract port instance on stack.
>  	 */
> -	char pad[48];
> +	char pad[52];

The port_sql struct is containing the port_tuple and a 3 extra fields. I
agree that port_typle size is 48 bytes, but how do you plat to fit 3
fields in 4 extra bytes?

>  };
>  
>  /** Is not inlined just to be exported. */
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 10/20] sql: resurrect sql_bind_parameter_count() function

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> This function is present in sql/vdbeapi.c source file, its prototype is
> missing in any header file. It makes impossible to use it. Let's add
> prototype declaration to sql/sqlInt.h (as other parameter
> setters/getters) and refactor a bit in accordance with our codestyle.
> 
> Need for #2592
> ---
>  src/box/sql/sqlInt.h  |  6 ++++++
>  src/box/sql/vdbeapi.c | 10 +++-------
>  2 files changed, 9 insertions(+), 7 deletions(-)
> 
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 24da3ca11..a9faaa6e7 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -689,6 +689,12 @@ int
>  sql_bind_zeroblob64(sql_stmt *, int,
>  			sql_uint64);
>  
> +/**
> + * Return the number of wildcards that should be bound to.
> + */
> +int
> +sql_bind_parameter_count(const struct sql_stmt *stmt);
> +
>  /**
>   * Perform pointer parameter binding for the prepared sql
>   * statement.
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index b6bf9aa81..7fda525ce 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -1051,15 +1051,11 @@ sql_bind_zeroblob64(sql_stmt * pStmt, int i, sql_uint64 n)
>  	return sql_bind_zeroblob(pStmt, i, n);
>  }
>  
> -/*
> - * Return the number of wildcards that can be potentially bound to.
> - * This routine is added to support DBD::sql.
> - */
>  int
> -sql_bind_parameter_count(sql_stmt * pStmt)
> +sql_bind_parameter_count(const struct sql_stmt *stmt)
>  {
> -	Vdbe *p = (Vdbe *) pStmt;
> -	return p ? p->nVar : 0;
> +	struct Vdbe *p = (struct Vdbe *) stmt;
> +	return p->nVar;
>  }
>  
>  /*
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 11/20] sql: resurrect sql_bind_parameter_name()

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> We may need to get name of parameter to be bound by its index position.
> So let's resurrect sql_bind_parameter_name() - put its prototype to
> sql/sqlInt.h header and update codestyle.
> 
> Need for #2592
> ---
>  src/box/sql/sqlInt.h  |  8 ++++++++
>  src/box/sql/vdbeapi.c | 14 ++++----------
>  2 files changed, 12 insertions(+), 10 deletions(-)
> 
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index a9faaa6e7..09f638268 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -695,6 +695,14 @@ sql_bind_zeroblob64(sql_stmt *, int,
>  int
>  sql_bind_parameter_count(const struct sql_stmt *stmt);
>  
> +/**
> + * Return the name of a wildcard parameter. Return NULL if the index
> + * is out of range or if the wildcard is unnamed. Parameter's index
> + * is 0-based.
> + */
> +const char *
> +sql_bind_parameter_name(const struct sql_stmt *stmt, int i);
> +
>  /**
>   * Perform pointer parameter binding for the prepared sql
>   * statement.
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 7fda525ce..b1c556ec3 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -1058,18 +1058,12 @@ sql_bind_parameter_count(const struct sql_stmt *stmt)
>  	return p->nVar;
>  }
>  
> -/*
> - * Return the name of a wildcard parameter.  Return NULL if the index
> - * is out of range or if the wildcard is unnamed.
> - *
> - * The result is always UTF-8.
> - */
>  const char *
> -sql_bind_parameter_name(sql_stmt * pStmt, int i)
> +sql_bind_parameter_name(const struct sql_stmt *stmt, int i)
>  {
> -	Vdbe *p = (Vdbe *) pStmt;
> -	if (p == 0)
> -		return 0;
> +	struct Vdbe *p = (struct Vdbe *) stmt;
> +	if (p == NULL)
> +		return NULL;
>  	return sqlVListNumToName(p->pVList, i);
>  }
>  
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 09/20] port: add result set format and request type to port_sql

[Sergey Ostanevich]

Hi!

Thanks for the patch, this one LGTM.


Sergos


On 20 Dec 15:47, Nikita Pettik wrote:
> Result set serialization formats of DQL and DML queries are different:
> the last one contains number of affected rows and optionally list of
> autoincremented ids; the first one comprises all meta-information
> including column names of resulting set and their types. What is more,
> serialization format is going to be different for execute and prepare
> requests. So let's introduce separate member to struct port_sql
> responsible for serialization format to be used.
> 
> Note that C standard specifies that enums are integers, but it does not
> specify the size. Hence, let's use simple uint8 - mentioned enum are
> small enough to fit into it.
> 
> What is more, prepared statement finalization is required only for
> PREPARE-AND-EXECUTE requests. So let's keep flag indicating required
> finalization as well.
> 
> Needed for #2592
> ---
>  src/box/execute.c     | 34 +++++++++++++++++++++++++---------
>  src/box/execute.h     | 21 +++++++++++++++++++++
>  src/box/lua/execute.c | 22 +++++++++++++++-------
>  3 files changed, 61 insertions(+), 16 deletions(-)
> 
> diff --git a/src/box/execute.c b/src/box/execute.c
> index fb83e1194..3bc4988b7 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -100,7 +100,9 @@ static void
>  port_sql_destroy(struct port *base)
>  {
>  	port_tuple_vtab.destroy(base);
> -	sql_stmt_finalize(((struct port_sql *)base)->stmt);
> +	struct port_sql *port_sql = (struct port_sql *) base;
> +	if (port_sql->do_finalize)
> +		sql_stmt_finalize(((struct port_sql *)base)->stmt);
>  }
>  
>  const struct port_vtab port_sql_vtab = {
> @@ -114,11 +116,15 @@ const struct port_vtab port_sql_vtab = {
>  };
>  
>  static void
> -port_sql_create(struct port *port, struct sql_stmt *stmt)
> +port_sql_create(struct port *port, struct sql_stmt *stmt,
> +		enum sql_serialization_format format, bool do_finalize)
>  {
>  	port_tuple_create(port);
> -	((struct port_sql *)port)->stmt = stmt;
>  	port->vtab = &port_sql_vtab;
> +	struct port_sql *port_sql = (struct port_sql *) port;
> +	port_sql->stmt = stmt;
> +	port_sql->serialization_format = format;
> +	port_sql->do_finalize = do_finalize;
>  }
>  
>  /**
> @@ -324,9 +330,10 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  {
>  	assert(port->vtab == &port_sql_vtab);
>  	sql *db = sql_get();
> -	struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
> -	int column_count = sql_column_count(stmt);
> -	if (column_count > 0) {
> +	struct port_sql *sql_port = (struct port_sql *)port;
> +	struct sql_stmt *stmt = sql_port->stmt;
> +	switch (sql_port->serialization_format) {
> +	case DQL_EXECUTE: {
>  		int keys = 2;
>  		int size = mp_sizeof_map(keys);
>  		char *pos = (char *) obuf_alloc(out, size);
> @@ -335,7 +342,7 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  			return -1;
>  		}
>  		pos = mp_encode_map(pos, keys);
> -		if (sql_get_metadata(stmt, out, column_count) != 0)
> +		if (sql_get_metadata(stmt, out, sql_column_count(stmt)) != 0)
>  			return -1;
>  		size = mp_sizeof_uint(IPROTO_DATA);
>  		pos = (char *) obuf_alloc(out, size);
> @@ -346,7 +353,9 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  		pos = mp_encode_uint(pos, IPROTO_DATA);
>  		if (port_tuple_vtab.dump_msgpack(port, out) < 0)
>  			return -1;
> -	} else {
> +		break;
> +	}
> +	case DML_EXECUTE: {
>  		int keys = 1;
>  		assert(((struct port_tuple *)port)->size == 0);
>  		struct stailq *autoinc_id_list =
> @@ -395,6 +404,11 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  				      mp_encode_int(buf, id_entry->id);
>  			}
>  		}
> +		break;
> +	}
> +	default: {
> +		unreachable();
> +	}
>  	}
>  	return 0;
>  }
> @@ -445,7 +459,9 @@ sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
>  	if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
>  		return -1;
>  	assert(stmt != NULL);
> -	port_sql_create(port, stmt);
> +	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
> +					   DQL_EXECUTE : DML_EXECUTE;
> +	port_sql_create(port, stmt, format, true);
>  	if (sql_bind(stmt, bind, bind_count) == 0 &&
>  	    sql_execute(stmt, port, region) == 0)
>  		return 0;
> diff --git a/src/box/execute.h b/src/box/execute.h
> index ce1e7a67d..c87e765cf 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -46,6 +46,17 @@ enum sql_info_key {
>  	sql_info_key_MAX,
>  };
>  
> +/**
> + * One of possible formats used to dump msgpack/Lua.
> + * For details see port_sql_dump_msgpack() and port_sql_dump_lua().
> + */
> +enum sql_serialization_format {
> +	DQL_EXECUTE = 0,
> +	DML_EXECUTE = 1,
> +	DQL_PREPARE = 2,
> +	DML_PREPARE = 3,
> +};
> +
>  extern const char *sql_info_key_strs[];
>  
>  struct region;
> @@ -85,6 +96,16 @@ struct port_sql {
>  	struct port_tuple port_tuple;
>  	/* Prepared SQL statement. */
>  	struct sql_stmt *stmt;
> +	/**
> +	 * Serialization format depends on type of SQL query: DML or
> +	 * DQL; and on type of SQL request: execute or prepare.
> +	 */
> +	uint8_t serialization_format;
> +	/**
> +	 * There's no need in clean-up in case of PREPARE request:
> +	 * statement remains in cache and will be deleted later.
> +	 */
> +	bool do_finalize;
>  };
>  
>  extern const struct port_vtab port_sql_vtab;
> diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
> index 68adacf72..b164ffcaf 100644
> --- a/src/box/lua/execute.c
> +++ b/src/box/lua/execute.c
> @@ -45,18 +45,21 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
>  	assert(is_flat == false);
>  	assert(port->vtab == &port_sql_vtab);
>  	struct sql *db = sql_get();
> -	struct sql_stmt *stmt = ((struct port_sql *)port)->stmt;
> -	int column_count = sql_column_count(stmt);
> -	if (column_count > 0) {
> +	struct port_sql *port_sql = (struct port_sql *)port;
> +	struct sql_stmt *stmt = port_sql->stmt;
> +	switch (port_sql->serialization_format) {
> +	case DQL_EXECUTE: {
>  		lua_createtable(L, 0, 2);
> -		lua_sql_get_metadata(stmt, L, column_count);
> +		lua_sql_get_metadata(stmt, L, sql_column_count(stmt));
>  		lua_setfield(L, -2, "metadata");
>  		port_tuple_vtab.dump_lua(port, L, false);
>  		lua_setfield(L, -2, "rows");
> -	} else {
> -		assert(((struct port_tuple *)port)->size == 0);
> +		break;
> +	}
> +	case DML_EXECUTE: {
> +		assert(((struct port_tuple *) port)->size == 0);
>  		struct stailq *autoinc_id_list =
> -			vdbe_autoinc_id_list((struct Vdbe *)stmt);
> +			vdbe_autoinc_id_list((struct Vdbe *) stmt);
>  		lua_createtable(L, 0, stailq_empty(autoinc_id_list) ? 1 : 2);
>  
>  		luaL_pushuint64(L, db->nChange);
> @@ -77,6 +80,11 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
>  				sql_info_key_strs[SQL_INFO_AUTOINCREMENT_IDS];
>  			lua_setfield(L, -2, field_name);
>  		}
> +		break;
> +	}
> +	default: {
> +		unreachable();
> +	}
>  	}
>  }
>  
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 12/20] sql: add sql_stmt_schema_version()

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> Let's introduce interface function to get schema version of prepared
> statement. It is required since sturct sql_stmt (i.e. prepared
> statement) is an opaque object and in fact is an alias to struct Vdbe.
> Statements with schema version different from the current one are
> considered to be expired and should be re-compiled.
> 
> Needed for #2592
> ---
>  src/box/sql/sqlInt.h  | 3 +++
>  src/box/sql/vdbeapi.c | 7 +++++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 09f638268..7dfc29809 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -571,6 +571,9 @@ sql_column_name(sql_stmt *, int N);
>  const char *
>  sql_column_datatype(sql_stmt *, int N);
>  
> +uint32_t
> +sql_stmt_schema_version(const struct sql_stmt *stmt);
> +
>  int
>  sql_initialize(void);
>  
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index b1c556ec3..7d9ce11e7 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -798,6 +798,13 @@ sql_column_decltype(sql_stmt * pStmt, int N)
>  			  COLNAME_DECLTYPE);
>  }
>  
> +uint32_t
> +sql_stmt_schema_version(const struct sql_stmt *stmt)
> +{
> +	struct Vdbe *v = (struct Vdbe *) stmt;
> +	return v->schema_ver;
> +}
> +
>  /******************************* sql_bind_  **************************
>   *
>   * Routines used to attach values to wildcards in a compiled SQL statement.
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 13/20] sql: introduce sql_stmt_sizeof() function

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> To implement memory quota of prepared statement cache, we have to
> estimate size of prepared statement. This function attempts at that.
> 
> Part of #2592
> ---
>  src/box/execute.h     |  8 ++++++++
>  src/box/sql/vdbeapi.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 61 insertions(+)
> 
> diff --git a/src/box/execute.h b/src/box/execute.h
> index c87e765cf..2dd4fca03 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -113,6 +113,14 @@ extern const struct port_vtab port_sql_vtab;
>  int
>  sql_stmt_finalize(struct sql_stmt *stmt);
>  
> +/**
> + * Calculate estimated size of memory occupied by VM.
> + * See sqlVdbeMakeReady() for details concerning allocated
> + * memory.
> + */
> +size_t
> +sql_stmt_est_size(const struct sql_stmt *stmt);
> +
>  /**
>   * Prepare (compile into VDBE byte-code) statement.
>   *
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 7d9ce11e7..2ac174112 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -805,6 +805,59 @@ sql_stmt_schema_version(const struct sql_stmt *stmt)
>  	return v->schema_ver;
>  }
>  
> +size_t
> +sql_stmt_est_size(const struct sql_stmt *stmt)
> +{
> +	struct Vdbe *v = (struct Vdbe *) stmt;
> +	size_t size = sizeof(*v);
> +	/* Names and types of result set columns */
> +	size += sizeof(struct Mem) * v->nResColumn * COLNAME_N;
> +	/* Opcodes */
> +	size += sizeof(struct VdbeOp) * v->nOp;
> +	/* Memory cells */
> +	size += sizeof(struct Mem) * v->nMem;
> +	/* Bindings */
> +	size += sizeof(struct Mem) * v->nVar;
> +	/* Bindings included in the result set */
> +	size += sizeof(uint32_t) * v->res_var_count;
> +	/* Cursors */
> +	size += sizeof(struct VdbeCursor *) * v->nCursor;
> +
> +	for (int i = 0; i < v->nOp; ++i) {
> +		/* Estimate size of p4 operand. */
> +		if (v->aOp[i].p4type == P4_NOTUSED)
> +			continue;
> +		switch (v->aOp[i].p4type) {
> +		case P4_DYNAMIC:
> +		case P4_STATIC:
> +			if (v->aOp[i].opcode == OP_Blob ||
> +			    v->aOp[i].opcode == OP_String)
> +				size += v->aOp[i].p1;
> +			else if (v->aOp[i].opcode == OP_String8)
> +				size += strlen(v->aOp[i].p4.z);
> +			break;
> +		case P4_BOOL:
> +			size += sizeof(v->aOp[i].p4.b);
> +			break;
> +		case P4_INT32:
> +			size += sizeof(v->aOp[i].p4.i);
> +			break;
> +		case P4_UINT64:
> +		case P4_INT64:
> +			size += sizeof(*v->aOp[i].p4.pI64);
> +			break;
> +		case P4_REAL:
> +			size += sizeof(*v->aOp[i].p4.pReal);
> +			break;
> +		default:
> +			size += sizeof(v->aOp[i].p4.p);
> +			break;
> +		}
> +	}
> +	size += strlen(v->zSql);
> +	return size;
> +}
> +
>  /******************************* sql_bind_  **************************
>   *
>   * Routines used to attach values to wildcards in a compiled SQL statement.
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 14/20] box: increment schema_version on ddl operations

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> Some DDL operations such as SQL trigger alter, check and foreign
> constraint alter don't result in schema version change. On the other
> hand, we are going to rely on schema version to determine expired
> prepared statements: for instance, if FK constraint has been created
> after DML statement preparation, the latter may ignore FK constraint
> (instead of proper "statement has expired" error). Let's fix it and
> account schema change on each DDL operation.
> 
> Need for #2592
> ---
>  src/box/alter.cc | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/src/box/alter.cc b/src/box/alter.cc
> index bef25b605..f33c1dfd2 100644
> --- a/src/box/alter.cc
> +++ b/src/box/alter.cc
> @@ -4773,6 +4773,7 @@ on_replace_dd_trigger(struct trigger * /* trigger */, void *event)
>  
>  	txn_stmt_on_rollback(stmt, on_rollback);
>  	txn_stmt_on_commit(stmt, on_commit);
> +	++schema_version;
>  	return 0;
>  }
>  
> @@ -5283,6 +5284,7 @@ on_replace_dd_fk_constraint(struct trigger * /* trigger*/, void *event)
>  		space_reset_fk_constraint_mask(child_space);
>  		space_reset_fk_constraint_mask(parent_space);
>  	}
> +	++schema_version;
>  	return 0;
>  }
>  
> @@ -5528,6 +5530,7 @@ on_replace_dd_ck_constraint(struct trigger * /* trigger*/, void *event)
>  
>  	if (trigger_run(&on_alter_space, space) != 0)
>  		return -1;
> +	++schema_version;
>  	return 0;
>  }
>  
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 15/20] sql: introduce sql_stmt_query_str() method

[Sergey Ostanevich]

Hi!

Tahnks for the patch, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> It is getter to fetch string of SQL query from prepared statement.
> 
> Needed for #2592
> ---
>  src/box/execute.h     | 6 ++++++
>  src/box/sql/vdbeapi.c | 7 +++++++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/src/box/execute.h b/src/box/execute.h
> index 2dd4fca03..f3d6c38b3 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -121,6 +121,12 @@ sql_stmt_finalize(struct sql_stmt *stmt);
>  size_t
>  sql_stmt_est_size(const struct sql_stmt *stmt);
>  
> +/**
> + * Return string of SQL query.
> + */
> +const char *
> +sql_stmt_query_str(const struct sql_stmt *stmt);
> +
>  /**
>   * Prepare (compile into VDBE byte-code) statement.
>   *
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 2ac174112..7278d2ab3 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -858,6 +858,13 @@ sql_stmt_est_size(const struct sql_stmt *stmt)
>  	return size;
>  }
>  
> +const char *
> +sql_stmt_query_str(const struct sql_stmt *stmt)
> +{
> +	const struct Vdbe *v = (const struct Vdbe *) stmt;
> +	return v->zSql;
> +}
> +
>  /******************************* sql_bind_  **************************
>   *
>   * Routines used to attach values to wildcards in a compiled SQL statement.
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 16/20] sql: move sql_stmt_busy() declaration to box/execute.h

[Sergey Ostanevich]

Hi!

Thanks, LGTM.

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> We are going to use it in box/execute.c and in SQL prepared statement
> cache implementation. So to avoid including whole sqlInt.h let's move it
> to relative small execute.h header. Let's also fix codestyle of this
> function.
> 
> Needed for #2592
> ---
>  src/box/execute.h     |  4 ++++
>  src/box/sql/sqlInt.h  |  3 ---
>  src/box/sql/vdbeapi.c | 10 ++++------
>  src/box/sql/vdbeaux.c |  1 +
>  4 files changed, 9 insertions(+), 9 deletions(-)
> 
> diff --git a/src/box/execute.h b/src/box/execute.h
> index f3d6c38b3..61c8e0281 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -127,6 +127,10 @@ sql_stmt_est_size(const struct sql_stmt *stmt);
>  const char *
>  sql_stmt_query_str(const struct sql_stmt *stmt);
>  
> +/** Return true if statement executes right now. */
> +int
> +sql_stmt_busy(const struct sql_stmt *stmt);
> +
>  /**
>   * Prepare (compile into VDBE byte-code) statement.
>   *
> diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> index 7dfc29809..cbe46e790 100644
> --- a/src/box/sql/sqlInt.h
> +++ b/src/box/sql/sqlInt.h
> @@ -718,9 +718,6 @@ sql_bind_parameter_name(const struct sql_stmt *stmt, int i);
>  int
>  sql_bind_ptr(struct sql_stmt *stmt, int i, void *ptr);
>  
> -int
> -sql_stmt_busy(sql_stmt *);
> -
>  int
>  sql_init_db(sql **db);
>  
> diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> index 7278d2ab3..01185af5f 100644
> --- a/src/box/sql/vdbeapi.c
> +++ b/src/box/sql/vdbeapi.c
> @@ -1190,14 +1190,12 @@ sql_db_handle(sql_stmt * pStmt)
>  	return pStmt ? ((Vdbe *) pStmt)->db : 0;
>  }
>  
> -/*
> - * Return true if the prepared statement is in need of being reset.
> - */
>  int
> -sql_stmt_busy(sql_stmt * pStmt)
> +sql_stmt_busy(const struct sql_stmt *stmt)
>  {
> -	Vdbe *v = (Vdbe *) pStmt;
> -	return v != 0 && v->magic == VDBE_MAGIC_RUN && v->pc >= 0;
> +	assert(stmt != NULL);
> +	const struct Vdbe *v = (const struct Vdbe *) stmt;
> +	return v->magic == VDBE_MAGIC_RUN && v->pc >= 0;
>  }
>  
>  /*
> diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c
> index 619105820..afe1ecb2a 100644
> --- a/src/box/sql/vdbeaux.c
> +++ b/src/box/sql/vdbeaux.c
> @@ -43,6 +43,7 @@
>  #include "sqlInt.h"
>  #include "vdbeInt.h"
>  #include "tarantoolInt.h"
> +#include "box/execute.h"
>  
>  /*
>   * Create a new virtual database engine.
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 18/20] box: introduce prepared statements

[Sergey Ostanevich]

Hi!

Thanks for the patch, LGTM with just 2 nits below. 

Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> This patch introduces local prepared statements. Support of prepared
> statements in IProto protocol and netbox is added in the next patch.
> 
> Prepared statement is an opaque instance of SQL Virtual Machine. It can
> be executed several times without necessity of query recompilation. To
> achieve this one can use box.prepare(...) function. It takes string of
> SQL query to be prepared; returns extended set of meta-information
> including statement's ID, parameter's types and names, types and names
> of columns of the resulting set, count of parameters to be bound.  Lua
> object representing result of :prepare() invocation also features two
> methods - :execute() and :unprepare(). They correspond to
> box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
> automatically substitute string of prepared statement to be executed.
> Statements are held in prepared statement cache - for details see
> previous commit.  After schema changes all prepared statement located in
> cache are considered to be expired - they must be re-prepared by
> separate :prepare() call (or be invalidated with :unrepare()).
> 
> Two sessions can share one prepared statements. But in current
> implementation if statement is executed by one session, another one is
> not able to use it and will compile it from scratch and than execute.

It would be nice to mention plans on what should/will be done for
resolution of this. Also, my previous question on DDL during the
execution of the statement is valid - one session can ruin execution
of a prepared statement in another session with a DDL. Should there 
be some guard for DDL until all executions of prep statements are finished?

> 
> SQL cache memory limit is regulated by box{sql_cache_size} which can be
> set dynamically. However, it can be set to the value which is less than
> the size of current free space in cache (since otherwise some statements
> can disappear from cache).
> 
> Part of #2592
> ---
>  src/box/errcode.h          |   1 +
>  src/box/execute.c          | 114 ++++++++
>  src/box/execute.h          |  16 +-
>  src/box/lua/execute.c      | 213 +++++++++++++-
>  src/box/lua/execute.h      |   2 +-
>  src/box/lua/init.c         |   2 +-
>  src/box/sql/prepare.c      |   9 -
>  test/box/misc.result       |   3 +
>  test/sql/engine.cfg        |   3 +
>  test/sql/prepared.result   | 687 +++++++++++++++++++++++++++++++++++++++++++++
>  test/sql/prepared.test.lua | 240 ++++++++++++++++
>  11 files changed, 1267 insertions(+), 23 deletions(-)
>  create mode 100644 test/sql/prepared.result
>  create mode 100644 test/sql/prepared.test.lua
> 
> diff --git a/src/box/errcode.h b/src/box/errcode.h
> index ee44f61b3..9e12f3a31 100644
> --- a/src/box/errcode.h
> +++ b/src/box/errcode.h
> @@ -259,6 +259,7 @@ struct errcode_record {
>  	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
>  	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
>  	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
> +	/*207 */_(ER_WRONG_QUERY_ID,		"Prepared statement with id %u does not exist") \
>  
>  /*
>   * !IMPORTANT! Please follow instructions at start of the file
> diff --git a/src/box/execute.c b/src/box/execute.c
> index 3bc4988b7..09224c23a 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -30,6 +30,7 @@
>   */
>  #include "execute.h"
>  
> +#include "assoc.h"
>  #include "bind.h"
>  #include "iproto_constants.h"
>  #include "sql/sqlInt.h"
> @@ -45,6 +46,8 @@
>  #include "tuple.h"
>  #include "sql/vdbe.h"
>  #include "box/lua/execute.h"
> +#include "box/sql_stmt_cache.h"
> +#include "session.h"
>  
>  const char *sql_info_key_strs[] = {
>  	"row_count",
> @@ -413,6 +416,81 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  	return 0;
>  }
>  
> +static bool
> +sql_stmt_check_schema_version(struct sql_stmt *stmt)

The naming could be better - since you state something as true or false,
it should be definitive, like sql_stmt_schema_version_is_valid()

> +{
> +	return sql_stmt_schema_version(stmt) == box_schema_version();
> +}
> +
> +/**
> + * Re-compile statement and refresh global prepared statement
> + * cache with the newest value.
> + */
> +static int
> +sql_reprepare(struct sql_stmt **stmt)
> +{
> +	const char *sql_str = sql_stmt_query_str(*stmt);
> +	struct sql_stmt *new_stmt;
> +	if (sql_stmt_compile(sql_str, strlen(sql_str), NULL,
> +			     &new_stmt, NULL) != 0)
> +		return -1;
> +	if (sql_stmt_cache_update(*stmt, new_stmt) != 0)
> +		return -1;
> +	*stmt = new_stmt;
> +	return 0;
> +}
> +
> +/**
> + * Compile statement and save it to the global holder;
> + * update session hash with prepared statement ID (if
> + * it's not already there).
> + */
> +int
> +sql_prepare(const char *sql, int len, struct port *port)
> +{
> +	uint32_t stmt_id = sql_stmt_calculate_id(sql, len);
> +	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
> +	if (stmt == NULL) {
> +		if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
> +			return -1;
> +		if (sql_stmt_cache_insert(stmt) != 0) {
> +			sql_stmt_finalize(stmt);
> +			return -1;
> +		}
> +	} else {
> +		if (! sql_stmt_check_schema_version(stmt)) {

The unaries should not be space-delimited as per C style $3.1
https://www.tarantool.io/en/doc/2.2/dev_guide/c_style_guide/

> +			if (sql_reprepare(&stmt) != 0)
> +				return -1;
> +		}
> +	}
> +	assert(stmt != NULL);
> +	/* Add id to the list of available statements in session. */
> +	if (!session_check_stmt_id(current_session(), stmt_id))
> +		session_add_stmt_id(current_session(), stmt_id);
> +	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
> +					   DQL_PREPARE : DML_PREPARE;
> +	port_sql_create(port, stmt, format, false);
> +
> +	return 0;
> +}
> +
> +/**
> + * Deallocate prepared statement from current session:
> + * remove its ID from session-local hash and unref entry
> + * in global holder.
> + */
> +int
> +sql_unprepare(uint32_t stmt_id)
> +{
> +	if (!session_check_stmt_id(current_session(), stmt_id)) {
> +		diag_set(ClientError, ER_WRONG_QUERY_ID, stmt_id);
> +		return -1;
> +	}
> +	session_remove_stmt_id(current_session(), stmt_id);
> +	sql_stmt_unref(stmt_id);
> +	return 0;
> +}
> +
>  /**
>   * Execute prepared SQL statement.
>   *
> @@ -450,6 +528,42 @@ sql_execute(struct sql_stmt *stmt, struct port *port, struct region *region)
>  	return 0;
>  }
>  
> +int
> +sql_execute_prepared(uint32_t stmt_id, const struct sql_bind *bind,
> +		     uint32_t bind_count, struct port *port,
> +		     struct region *region)
> +{
> +
> +	if (!session_check_stmt_id(current_session(), stmt_id)) {
> +		diag_set(ClientError, ER_WRONG_QUERY_ID, stmt_id);
> +		return -1;
> +	}
> +	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
> +	assert(stmt != NULL);
> +	if (! sql_stmt_check_schema_version(stmt)) {
> +		diag_set(ClientError, ER_SQL_EXECUTE, "statement has expired");
> +		return -1;
> +	}
> +	if (sql_stmt_busy(stmt)) {
> +		const char *sql_str = sql_stmt_query_str(stmt);
> +		return sql_prepare_and_execute(sql_str, strlen(sql_str), bind,
> +					       bind_count, port, region);
> +	}
> +	if (sql_bind(stmt, bind, bind_count) != 0)
> +		return -1;
> +	enum sql_serialization_format format = sql_column_count(stmt) > 0 ?
> +					       DQL_EXECUTE : DML_EXECUTE;
> +	port_sql_create(port, stmt, format, false);
> +	if (sql_execute(stmt, port, region) != 0) {
> +		port_destroy(port);
> +		sql_stmt_reset(stmt);
> +		return -1;
> +	}
> +	sql_stmt_reset(stmt);
> +
> +	return 0;
> +}
> +
>  int
>  sql_prepare_and_execute(const char *sql, int len, const struct sql_bind *bind,
>  			uint32_t bind_count, struct port *port,
> diff --git a/src/box/execute.h b/src/box/execute.h
> index 61c8e0281..5e2327f4a 100644
> --- a/src/box/execute.h
> +++ b/src/box/execute.h
> @@ -62,6 +62,14 @@ extern const char *sql_info_key_strs[];
>  struct region;
>  struct sql_bind;
>  
> +int
> +sql_unprepare(uint32_t stmt_id);
> +
> +int
> +sql_execute_prepared(uint32_t query_id, const struct sql_bind *bind,
> +		     uint32_t bind_count, struct port *port,
> +		     struct region *region);
> +
>  /**
>   * Prepare and execute an SQL statement.
>   * @param sql SQL statement.
> @@ -135,13 +143,11 @@ sql_stmt_busy(const struct sql_stmt *stmt);
>   * Prepare (compile into VDBE byte-code) statement.
>   *
>   * @param sql UTF-8 encoded SQL statement.
> - * @param length Length of @param sql in bytes.
> - * @param[out] stmt A pointer to the prepared statement.
> - * @param[out] sql_tail End of parsed string.
> + * @param len Length of @param sql in bytes.
> + * @param port Port to store request response.
>   */
>  int
> -sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
> -	    const char **sql_tail);
> +sql_prepare(const char *sql, int len, struct port *port);
>  
>  #if defined(__cplusplus)
>  } /* extern "C" { */
> diff --git a/src/box/lua/execute.c b/src/box/lua/execute.c
> index b164ffcaf..6cb1f5db9 100644
> --- a/src/box/lua/execute.c
> +++ b/src/box/lua/execute.c
> @@ -5,6 +5,8 @@
>  #include "box/port.h"
>  #include "box/execute.h"
>  #include "box/bind.h"
> +#include "box/sql_stmt_cache.h"
> +#include "box/schema.h"
>  
>  /**
>   * Serialize a description of the prepared statement.
> @@ -38,6 +40,101 @@ lua_sql_get_metadata(struct sql_stmt *stmt, struct lua_State *L,
>  	}
>  }
>  
> +static inline void
> +lua_sql_get_params_metadata(struct sql_stmt *stmt, struct lua_State *L)
> +{
> +	int bind_count = sql_bind_parameter_count(stmt);
> +	lua_createtable(L, bind_count, 0);
> +	for (int i = 0; i < bind_count; ++i) {
> +		lua_createtable(L, 0, 2);
> +		const char *name = sql_bind_parameter_name(stmt, i);
> +		if (name == NULL)
> +			name = "?";
> +		const char *type = "ANY";
> +		lua_pushstring(L, name);
> +		lua_setfield(L, -2, "name");
> +		lua_pushstring(L, type);
> +		lua_setfield(L, -2, "type");
> +		lua_rawseti(L, -2, i + 1);
> +	}
> +}
> +
> +/** Forward declaration to avoid code movement. */
> +static int
> +lbox_execute(struct lua_State *L);
> +
> +/**
> + * Prepare SQL statement: compile it and save to the cache.
> + * In fact it is wrapper around box.execute() which unfolds
> + * it to box.execute(stmt.query_id).
> + */
> +static int
> +lbox_execute_prepared(struct lua_State *L)
> +{
> +	int top = lua_gettop(L);
> +
> +	if ((top != 1 && top != 2) || ! lua_istable(L, 1))
> +		return luaL_error(L, "Usage: statement:execute([, params])");
> +	lua_getfield(L, 1, "stmt_id");
> +	if (!lua_isnumber(L, -1))
> +		return luaL_error(L, "Query id is expected to be numeric");
> +	lua_remove(L, 1);
> +	if (top == 2) {
> +		/*
> +		 * Stack state (before remove operation):
> +		 * 1 Prepared statement object (Lua table)
> +		 * 2 Bindings (Lua table)
> +		 * 3 Statement ID(fetched from PS table) - top of stack
> +		 *
> +		 * We should make it suitable to pass arguments to
> +		 * lbox_execute(), i.e. after manipulations stack
> +		 * should look like:
> +		 * 1 Statement ID
> +		 * 2 Bindings - top of stack
> +		 * Since there's no swap operation, we firstly remove
> +		 * PS object, then copy table of values to be bound to
> +		 * the top of stack (push), and finally remove original
> +		 * bindings from stack.
> +		 */
> +		lua_pushvalue(L, 1);
> +		lua_remove(L, 1);
> +	}
> +	return lbox_execute(L);
> +}
> +
> +/**
> + * Unprepare statement: remove it from prepared statements cache.
> + * This function can be called in two ways: as member of prepared
> + * statement handle (stmt:unprepare()) or as box.unprepare(stmt_id).
> + */
> +static int
> +lbox_unprepare(struct lua_State *L)
> +{
> +	int top = lua_gettop(L);
> +
> +	if (top != 1 || (! lua_istable(L, 1) && ! lua_isnumber(L, 1))) {
> +		return luaL_error(L, "Usage: statement:unprepare() or "\
> +				     "box.unprepare(stmt_id)");
> +	}
> +	lua_Integer stmt_id;
> +	if (lua_istable(L, 1)) {
> +		lua_getfield(L, -1, "stmt_id");
> +		if (! lua_isnumber(L, -1)) {
> +			return luaL_error(L, "Statement id is expected "\
> +					     "to be numeric");
> +		}
> +		stmt_id = lua_tointeger(L, -1);
> +		lua_pop(L, 1);
> +	} else {
> +		stmt_id = lua_tonumber(L, 1);
> +	}
> +	if (stmt_id < 0)
> +		return luaL_error(L, "Statement id can't be negative");
> +	if (sql_unprepare((uint32_t) stmt_id) != 0)
> +		return luaT_push_nil_and_error(L);
> +	return 0;
> +}
> +
>  void
>  port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
>  {
> @@ -82,7 +179,66 @@ port_sql_dump_lua(struct port *port, struct lua_State *L, bool is_flat)
>  		}
>  		break;
>  	}
> -	default: {
> +	case DQL_PREPARE: {
> +		/* Format is following:
> +		 * stmt_id,
> +		 * param_count,
> +		 * params {name, type},
> +		 * metadata {name, type}
> +		 * execute(), unprepare()
> +		 */
> +		lua_createtable(L, 0, 6);
> +		/* query_id */
> +		const char *sql_str = sql_stmt_query_str(port_sql->stmt);
> +		luaL_pushuint64(L, sql_stmt_calculate_id(sql_str,
> +							 strlen(sql_str)));
> +		lua_setfield(L, -2, "stmt_id");
> +		/* param_count */
> +		luaL_pushuint64(L, sql_bind_parameter_count(stmt));
> +		lua_setfield(L, -2, "param_count");
> +		/* params map */
> +		lua_sql_get_params_metadata(stmt, L);
> +		lua_setfield(L, -2, "params");
> +		/* metadata */
> +		lua_sql_get_metadata(stmt, L, sql_column_count(stmt));
> +		lua_setfield(L, -2, "metadata");
> +		/* execute function */
> +		lua_pushcfunction(L, lbox_execute_prepared);
> +		lua_setfield(L, -2, "execute");
> +		/* unprepare function */
> +		lua_pushcfunction(L, lbox_unprepare);
> +		lua_setfield(L, -2, "unprepare");
> +		break;
> +	}
> +	case DML_PREPARE : {
> +		assert(((struct port_tuple *) port)->size == 0);
> +		/* Format is following:
> +		 * stmt_id,
> +		 * param_count,
> +		 * params {name, type},
> +		 * execute(), unprepare()
> +		 */
> +		lua_createtable(L, 0, 5);
> +		/* query_id */
> +		const char *sql_str = sql_stmt_query_str(port_sql->stmt);
> +		luaL_pushuint64(L, sql_stmt_calculate_id(sql_str,
> +							 strlen(sql_str)));
> +		lua_setfield(L, -2, "stmt_id");
> +		/* param_count */
> +		luaL_pushuint64(L, sql_bind_parameter_count(stmt));
> +		lua_setfield(L, -2, "param_count");
> +		/* params map */
> +		lua_sql_get_params_metadata(stmt, L);
> +		lua_setfield(L, -2, "params");
> +		/* execute function */
> +		lua_pushcfunction(L, lbox_execute_prepared);
> +		lua_setfield(L, -2, "execute");
> +		/* unprepare function */
> +		lua_pushcfunction(L, lbox_unprepare);
> +		lua_setfield(L, -2, "unprepare");
> +		break;
> +	}
> +	default:{
>  		unreachable();
>  	}
>  	}
> @@ -253,9 +409,8 @@ lbox_execute(struct lua_State *L)
>  	int top = lua_gettop(L);
>  
>  	if ((top != 1 && top != 2) || ! lua_isstring(L, 1))
> -		return luaL_error(L, "Usage: box.execute(sqlstring[, params])");
> -
> -	const char *sql = lua_tolstring(L, 1, &length);
> +		return luaL_error(L, "Usage: box.execute(sqlstring[, params]) "
> +				  "or box.execute(stmt_id[, params])");
>  
>  	if (top == 2) {
>  		if (! lua_istable(L, 2))
> @@ -264,9 +419,44 @@ lbox_execute(struct lua_State *L)
>  		if (bind_count < 0)
>  			return luaT_push_nil_and_error(L);
>  	}
> +	/*
> +	 * lua_isstring() returns true for numeric values as well,
> +	 * so test explicit type instead.
> +	 */
> +	if (lua_type(L, 1) == LUA_TSTRING) {
> +		const char *sql = lua_tolstring(L, 1, &length);
> +		if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
> +					    &fiber()->gc) != 0)
> +			return luaT_push_nil_and_error(L);
> +	} else {
> +		assert(lua_type(L, 1) == LUA_TNUMBER);
> +		lua_Integer query_id = lua_tointeger(L, 1);
> +		if (query_id < 0)
> +			return luaL_error(L, "Statement id can't be negative");
> +		if (sql_execute_prepared(query_id, bind, bind_count, &port,
> +					 &fiber()->gc) != 0)
> +			return luaT_push_nil_and_error(L);
> +	}
> +	port_dump_lua(&port, L, false);
> +	port_destroy(&port);
> +	return 1;
> +}
>  
> -	if (sql_prepare_and_execute(sql, length, bind, bind_count, &port,
> -				    &fiber()->gc) != 0)
> +/**
> + * Prepare SQL statement: compile it and save to the cache.
> + */
> +static int
> +lbox_prepare(struct lua_State *L)
> +{
> +	size_t length;
> +	struct port port;
> +	int top = lua_gettop(L);
> +
> +	if ((top != 1 && top != 2) || ! lua_isstring(L, 1))
> +		return luaL_error(L, "Usage: box.prepare(sqlstring)");
> +
> +	const char *sql = lua_tolstring(L, 1, &length);
> +	if (sql_prepare(sql, length, &port) != 0)
>  		return luaT_push_nil_and_error(L);
>  	port_dump_lua(&port, L, false);
>  	port_destroy(&port);
> @@ -274,11 +464,20 @@ lbox_execute(struct lua_State *L)
>  }
>  
>  void
> -box_lua_execute_init(struct lua_State *L)
> +box_lua_sql_init(struct lua_State *L)
>  {
>  	lua_getfield(L, LUA_GLOBALSINDEX, "box");
>  	lua_pushstring(L, "execute");
>  	lua_pushcfunction(L, lbox_execute);
>  	lua_settable(L, -3);
> +
> +	lua_pushstring(L, "prepare");
> +	lua_pushcfunction(L, lbox_prepare);
> +	lua_settable(L, -3);
> +
> +	lua_pushstring(L, "unprepare");
> +	lua_pushcfunction(L, lbox_unprepare);
> +	lua_settable(L, -3);
> +
>  	lua_pop(L, 1);
>  }
> diff --git a/src/box/lua/execute.h b/src/box/lua/execute.h
> index 23e193fa4..bafd67615 100644
> --- a/src/box/lua/execute.h
> +++ b/src/box/lua/execute.h
> @@ -66,6 +66,6 @@ lua_sql_bind_list_decode(struct lua_State *L, struct sql_bind **out_bind,
>  			 int idx);
>  
>  void
> -box_lua_execute_init(struct lua_State *L);
> +box_lua_sql_init(struct lua_State *L);
>  
>  #endif /* INCLUDES_TARANTOOL_LUA_EXECUTE_H */
> diff --git a/src/box/lua/init.c b/src/box/lua/init.c
> index 7ffed409d..7be520e09 100644
> --- a/src/box/lua/init.c
> +++ b/src/box/lua/init.c
> @@ -314,7 +314,7 @@ box_lua_init(struct lua_State *L)
>  	box_lua_ctl_init(L);
>  	box_lua_session_init(L);
>  	box_lua_xlog_init(L);
> -	box_lua_execute_init(L);
> +	box_lua_sql_init(L);
>  	luaopen_net_box(L);
>  	lua_pop(L, 1);
>  	tarantool_lua_console_init(L);
> diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> index 73d6866a4..5b1c6c581 100644
> --- a/src/box/sql/prepare.c
> +++ b/src/box/sql/prepare.c
> @@ -193,15 +193,6 @@ sqlReprepare(Vdbe * p)
>  	return 0;
>  }
>  
> -int
> -sql_prepare(const char *sql, int length, struct sql_stmt **stmt,
> -	    const char **sql_tail)
> -{
> -	int rc = sql_stmt_compile(sql, length, 0, stmt, sql_tail);
> -	assert(rc == 0 || stmt == NULL || *stmt == NULL);
> -	return rc;
> -}
> -
>  void
>  sql_parser_create(struct Parse *parser, struct sql *db, uint32_t sql_flags)
>  {
> diff --git a/test/box/misc.result b/test/box/misc.result
> index 7e5d28b70..90923f28e 100644
> --- a/test/box/misc.result
> +++ b/test/box/misc.result
> @@ -73,6 +73,7 @@ t
>    - on_commit
>    - on_rollback
>    - once
> +  - prepare
>    - priv
>    - rollback
>    - rollback_to_savepoint
> @@ -86,6 +87,7 @@ t
>    - space
>    - stat
>    - tuple
> +  - unprepare
>  ...
>  t = nil
>  ---
> @@ -555,6 +557,7 @@ t;
>    204: box.error.SQL_FUNC_WRONG_RET_COUNT
>    205: box.error.FUNC_INVALID_RETURN_TYPE
>    206: box.error.SQL_PREPARE
> +  207: box.error.WRONG_QUERY_ID
>  ...
>  test_run:cmd("setopt delimiter ''");
>  ---
> diff --git a/test/sql/engine.cfg b/test/sql/engine.cfg
> index 284c42082..a1b4b0fc5 100644
> --- a/test/sql/engine.cfg
> +++ b/test/sql/engine.cfg
> @@ -9,6 +9,9 @@
>          "remote": {"remote": "true"},
>          "local": {"remote": "false"}
>      },
> +    "prepared.test.lua": {
> +        "local": {"remote": "false"}
> +    },
>      "*": {
>          "memtx": {"engine": "memtx"},
>          "vinyl": {"engine": "vinyl"}
> diff --git a/test/sql/prepared.result b/test/sql/prepared.result
> new file mode 100644
> index 000000000..bd37cfdd7
> --- /dev/null
> +++ b/test/sql/prepared.result
> @@ -0,0 +1,687 @@
> +-- test-run result file version 2
> +remote = require('net.box')
> + | ---
> + | ...
> +test_run = require('test_run').new()
> + | ---
> + | ...
> +fiber = require('fiber')
> + | ---
> + | ...
> +
> +-- Wrappers to make remote and local execution interface return
> +-- same result pattern.
> +--
> +test_run:cmd("setopt delimiter ';'")
> + | ---
> + | - true
> + | ...
> +execute = function(...)
> +    local res, err = box.execute(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> + | ---
> + | ...
> +prepare = function(...)
> +    local res, err = box.prepare(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> + | ---
> + | ...
> +unprepare = function(...)
> +    local res, err = box.unprepare(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> + | ---
> + | ...
> +
> +test_run:cmd("setopt delimiter ''");
> + | ---
> + | - true
> + | ...
> +
> +-- Test local interface and basic capabilities of prepared statements.
> +--
> +execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> + | ---
> + | - row_count: 1
> + | ...
> +space = box.space.TEST
> + | ---
> + | ...
> +space:replace{1, 2, '3'}
> + | ---
> + | - [1, 2, '3']
> + | ...
> +space:replace{4, 5, '6'}
> + | ---
> + | - [4, 5, '6']
> + | ...
> +space:replace{7, 8.5, '9'}
> + | ---
> + | - [7, 8.5, '9']
> + | ...
> +s, e = prepare("SELECT * FROM test WHERE id = ? AND a = ?;")
> + | ---
> + | ...
> +assert(e == nil)
> + | ---
> + | - true
> + | ...
> +assert(s ~= nil)
> + | ---
> + | - true
> + | ...
> +s.stmt_id
> + | ---
> + | - 3603193623
> + | ...
> +s.metadata
> + | ---
> + | - - name: ID
> + |     type: integer
> + |   - name: A
> + |     type: number
> + |   - name: B
> + |     type: string
> + | ...
> +s.params
> + | ---
> + | - - name: '?'
> + |     type: ANY
> + |   - name: '?'
> + |     type: ANY
> + | ...
> +s.param_count
> + | ---
> + | - 2
> + | ...
> +execute(s.stmt_id, {1, 2})
> + | ---
> + | - metadata:
> + |   - name: ID
> + |     type: integer
> + |   - name: A
> + |     type: number
> + |   - name: B
> + |     type: string
> + |   rows:
> + |   - [1, 2, '3']
> + | ...
> +execute(s.stmt_id, {1, 3})
> + | ---
> + | - metadata:
> + |   - name: ID
> + |     type: integer
> + |   - name: A
> + |     type: number
> + |   - name: B
> + |     type: string
> + |   rows: []
> + | ...
> +s:execute({1, 2})
> + | ---
> + | - metadata:
> + |   - name: ID
> + |     type: integer
> + |   - name: A
> + |     type: number
> + |   - name: B
> + |     type: string
> + |   rows:
> + |   - [1, 2, '3']
> + | ...
> +s:execute({1, 3})
> + | ---
> + | - metadata:
> + |   - name: ID
> + |     type: integer
> + |   - name: A
> + |     type: number
> + |   - name: B
> + |     type: string
> + |   rows: []
> + | ...
> +s:unprepare()
> + | ---
> + | ...
> +
> +-- Test preparation of different types of queries.
> +-- Let's start from DDL. It doesn't make much sense since
> +-- any prepared DDL statement can be executed once, but
> +-- anyway make sure that no crashes occur.
> +--
> +s = prepare("CREATE INDEX i1 ON test(a)")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("DROP INDEX i1 ON test;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("CREATE VIEW v AS SELECT * FROM test;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("DROP VIEW v;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("ALTER TABLE test RENAME TO test1")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 0
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +box.execute("CREATE TABLE test2 (id INT PRIMARY KEY);")
> + | ---
> + | - row_count: 1
> + | ...
> +s = prepare("ALTER TABLE test2 ADD CONSTRAINT fk1 FOREIGN KEY (id) REFERENCES test2")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +box.space.TEST2:drop()
> + | ---
> + | ...
> +
> +s = prepare("CREATE TRIGGER tr1 INSERT ON test1 FOR EACH ROW BEGIN DELETE FROM test1; END;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("DROP TRIGGER tr1;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("DROP TABLE test1;")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - error: 'Failed to execute SQL statement: statement has expired'
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- DQL
> +--
> +execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> + | ---
> + | - row_count: 1
> + | ...
> +space = box.space.TEST
> + | ---
> + | ...
> +space:replace{1, 2, '3'}
> + | ---
> + | - [1, 2, '3']
> + | ...
> +space:replace{4, 5, '6'}
> + | ---
> + | - [4, 5, '6']
> + | ...
> +space:replace{7, 8.5, '9'}
> + | ---
> + | - [7, 8.5, '9']
> + | ...
> +_ = prepare("SELECT a FROM test WHERE b = '3';")
> + | ---
> + | ...
> +s = prepare("SELECT a FROM test WHERE b = '3';")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [2]
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [2]
> + | ...
> +s:execute()
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [2]
> + | ...
> +s:execute()
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [2]
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
> + | ---
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - metadata:
> + |   - name: count(*)
> + |     type: integer
> + |   - name: count(a - 3)
> + |     type: integer
> + |   - name: max(b)
> + |     type: scalar
> + |   - name: abs(id)
> + |     type: number
> + |   rows:
> + |   - [1, 1, '3', 1]
> + | ...
> +execute(s.stmt_id)
> + | ---
> + | - metadata:
> + |   - name: count(*)
> + |     type: integer
> + |   - name: count(a - 3)
> + |     type: integer
> + |   - name: max(b)
> + |     type: scalar
> + |   - name: abs(id)
> + |     type: number
> + |   rows:
> + |   - [1, 1, '3', 1]
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- Let's try something a bit more complicated. For instance recursive
> +-- query displaying Mandelbrot set.
> +--
> +s = prepare([[WITH RECURSIVE \
> +                  xaxis(x) AS (VALUES(-2.0) UNION ALL SELECT x+0.05 FROM xaxis WHERE x<1.2), \
> +                  yaxis(y) AS (VALUES(-1.0) UNION ALL SELECT y+0.1 FROM yaxis WHERE y<1.0), \
> +                  m(iter, cx, cy, x, y) AS ( \
> +                      SELECT 0, x, y, 0.0, 0.0 FROM xaxis, yaxis \
> +                      UNION ALL \
> +                      SELECT iter+1, cx, cy, x*x-y*y + cx, 2.0*x*y + cy FROM m \
> +                          WHERE (x*x + y*y) < 4.0 AND iter<28), \
> +                      m2(iter, cx, cy) AS ( \
> +                          SELECT max(iter), cx, cy FROM m GROUP BY cx, cy), \
> +                      a(t) AS ( \
> +                          SELECT group_concat( substr(' .+*#', 1+LEAST(iter/7,4), 1), '') \
> +                              FROM m2 GROUP BY cy) \
> +                  SELECT group_concat(TRIM(TRAILING FROM t),x'0a') FROM a;]])
> + | ---
> + | ...
> +
> +res = execute(s.stmt_id)
> + | ---
> + | ...
> +res.metadata
> + | ---
> + | - - name: group_concat(TRIM(TRAILING FROM t),x'0a')
> + |     type: string
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- Workflow with bindings is still the same.
> +--
> +s = prepare("SELECT a FROM test WHERE b = ?;")
> + | ---
> + | ...
> +execute(s.stmt_id, {'6'})
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [5]
> + | ...
> +execute(s.stmt_id, {'9'})
> + | ---
> + | - metadata:
> + |   - name: A
> + |     type: number
> + |   rows:
> + |   - [8.5]
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- DML
> +s = prepare("INSERT INTO test VALUES (?, ?, ?);")
> + | ---
> + | ...
> +execute(s.stmt_id, {5, 6, '7'})
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id, {6, 10, '7'})
> + | ---
> + | - row_count: 1
> + | ...
> +execute(s.stmt_id, {9, 11, '7'})
> + | ---
> + | - row_count: 1
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- EXPLAIN and PRAGMA work fine as well.
> +--
> +s1 = prepare("EXPLAIN SELECT a FROM test WHERE b = '3';")
> + | ---
> + | ...
> +res = execute(s1.stmt_id)
> + | ---
> + | ...
> +res.metadata
> + | ---
> + | - - name: addr
> + |     type: integer
> + |   - name: opcode
> + |     type: text
> + |   - name: p1
> + |     type: integer
> + |   - name: p2
> + |     type: integer
> + |   - name: p3
> + |     type: integer
> + |   - name: p4
> + |     type: text
> + |   - name: p5
> + |     type: text
> + |   - name: comment
> + |     type: text
> + | ...
> +assert(res.rows ~= nil)
> + | ---
> + | - true
> + | ...
> +
> +s2 = prepare("EXPLAIN QUERY PLAN SELECT a FROM test WHERE b = '3';")
> + | ---
> + | ...
> +res = execute(s2.stmt_id)
> + | ---
> + | ...
> +res.metadata
> + | ---
> + | - - name: selectid
> + |     type: integer
> + |   - name: order
> + |     type: integer
> + |   - name: from
> + |     type: integer
> + |   - name: detail
> + |     type: text
> + | ...
> +assert(res.rows ~= nil)
> + | ---
> + | - true
> + | ...
> +
> +s3 = prepare("PRAGMA count_changes;")
> + | ---
> + | ...
> +execute(s3.stmt_id)
> + | ---
> + | - metadata:
> + |   - name: defer_foreign_keys
> + |     type: integer
> + |   rows:
> + |   - [0]
> + | ...
> +
> +unprepare(s2.stmt_id)
> + | ---
> + | - null
> + | ...
> +unprepare(s3.stmt_id)
> + | ---
> + | - null
> + | ...
> +unprepare(s1.stmt_id)
> + | ---
> + | - null
> + | ...
> +
> +-- Setting cache size to 0 is possible only in case if
> +-- there's no any prepared statements right now .
> +--
> +box.cfg{sql_cache_size = 0}
> + | ---
> + | ...
> +prepare("SELECT a FROM test;")
> + | ---
> + | - error: 'Failed to prepare SQL statement: Memory limit for SQL prepared statements
> + |     has been reached. Please, deallocate active statements or increase SQL cache size.'
> + | ...
> +box.cfg{sql_cache_size = 0}
> + | ---
> + | ...
> +
> +-- Still with small size everything should work.
> +--
> +box.cfg{sql_cache_size = 1500}
> + | ---
> + | ...
> +
> +test_run:cmd("setopt delimiter ';'");
> + | ---
> + | - true
> + | ...
> +ok = nil
> +res = nil
> +_ = fiber.create(function()
> +    for i = 1, 5 do
> +        pcall(prepare, string.format("SELECT * FROM test WHERE a = %d;", i))
> +    end
> +    ok, res = pcall(prepare, "SELECT * FROM test WHERE b = '3';")
> +end);
> + | ---
> + | ...
> +while ok == nil do fiber.sleep(0.00001) end;
> + | ---
> + | ...
> +assert(ok == false);
> + | ---
> + | - true
> + | ...
> +res;
> + | ---
> + | - 'Failed to prepare SQL statement: Memory limit for SQL prepared statements has been
> + |   reached. Please, deallocate active statements or increase SQL cache size.'
> + | ...
> +
> +-- Check that after fiber is dead, its session gets rid of
> +-- all prepared statements.
> +--
> +box.cfg{sql_cache_size = 0};
> + | ---
> + | ...
> +box.cfg{sql_cache_size = 3000};
> + | ---
> + | ...
> +
> +-- Make sure that if prepared statement is busy (is executed
> +-- right now), prepared statement is not used, i.e. statement
> +-- is compiled from scratch, executed and finilized.
> +--
> +box.schema.func.create('SLEEP', {language = 'Lua',
> +    body = 'function () fiber.sleep(0.1) return 1 end',
> +    exports = {'LUA', 'SQL'}});
> + | ---
> + | ...
> +
> +s = prepare("SELECT id, SLEEP() FROM test;");
> + | ---
> + | ...
> +assert(s ~= nil);
> + | ---
> + | - true
> + | ...
> +
> +function implicit_yield()
> +    prepare("SELECT id, SLEEP() FROM test;")
> +    execute("SELECT id, SLEEP() FROM test;")
> +end;
> + | ---
> + | ...
> +
> +f1 = fiber.new(implicit_yield)
> +f2 = fiber.new(implicit_yield)
> +f1:set_joinable(true)
> +f2:set_joinable(true)
> +
> +f1:join();
> + | ---
> + | ...
> +f2:join();
> + | ---
> + | - true
> + | ...
> +
> +unprepare(s.stmt_id);
> + | ---
> + | - null
> + | ...
> +
> +test_run:cmd("setopt delimiter ''");
> + | ---
> + | - true
> + | ...
> +
> +box.cfg{sql_cache_size = 5 * 1024 * 1024}
> + | ---
> + | ...
> +box.space.TEST:drop()
> + | ---
> + | ...
> +box.schema.func.drop('SLEEP')
> + | ---
> + | ...
> diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
> new file mode 100644
> index 000000000..49d2fb3ae
> --- /dev/null
> +++ b/test/sql/prepared.test.lua
> @@ -0,0 +1,240 @@
> +remote = require('net.box')
> +test_run = require('test_run').new()
> +fiber = require('fiber')
> +
> +-- Wrappers to make remote and local execution interface return
> +-- same result pattern.
> +--
> +test_run:cmd("setopt delimiter ';'")
> +execute = function(...)
> +    local res, err = box.execute(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> +prepare = function(...)
> +    local res, err = box.prepare(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> +unprepare = function(...)
> +    local res, err = box.unprepare(...)
> +    if err ~= nil then
> +        error(err)
> +    end
> +    return res
> +end;
> +
> +test_run:cmd("setopt delimiter ''");
> +
> +-- Test local interface and basic capabilities of prepared statements.
> +--
> +execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> +space = box.space.TEST
> +space:replace{1, 2, '3'}
> +space:replace{4, 5, '6'}
> +space:replace{7, 8.5, '9'}
> +s, e = prepare("SELECT * FROM test WHERE id = ? AND a = ?;")
> +assert(e == nil)
> +assert(s ~= nil)
> +s.stmt_id
> +s.metadata
> +s.params
> +s.param_count
> +execute(s.stmt_id, {1, 2})
> +execute(s.stmt_id, {1, 3})
> +s:execute({1, 2})
> +s:execute({1, 3})
> +s:unprepare()
> +
> +-- Test preparation of different types of queries.
> +-- Let's start from DDL. It doesn't make much sense since
> +-- any prepared DDL statement can be executed once, but
> +-- anyway make sure that no crashes occur.
> +--
> +s = prepare("CREATE INDEX i1 ON test(a)")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("DROP INDEX i1 ON test;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("CREATE VIEW v AS SELECT * FROM test;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("DROP VIEW v;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("ALTER TABLE test RENAME TO test1")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +box.execute("CREATE TABLE test2 (id INT PRIMARY KEY);")
> +s = prepare("ALTER TABLE test2 ADD CONSTRAINT fk1 FOREIGN KEY (id) REFERENCES test2")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +box.space.TEST2:drop()
> +
> +s = prepare("CREATE TRIGGER tr1 INSERT ON test1 FOR EACH ROW BEGIN DELETE FROM test1; END;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("DROP TRIGGER tr1;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +s = prepare("DROP TABLE test1;")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +-- DQL
> +--
> +execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> +space = box.space.TEST
> +space:replace{1, 2, '3'}
> +space:replace{4, 5, '6'}
> +space:replace{7, 8.5, '9'}
> +_ = prepare("SELECT a FROM test WHERE b = '3';")
> +s = prepare("SELECT a FROM test WHERE b = '3';")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +s:execute()
> +s:execute()
> +unprepare(s.stmt_id)
> +
> +s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
> +execute(s.stmt_id)
> +execute(s.stmt_id)
> +unprepare(s.stmt_id)
> +
> +-- Let's try something a bit more complicated. For instance recursive
> +-- query displaying Mandelbrot set.
> +--
> +s = prepare([[WITH RECURSIVE \
> +                  xaxis(x) AS (VALUES(-2.0) UNION ALL SELECT x+0.05 FROM xaxis WHERE x<1.2), \
> +                  yaxis(y) AS (VALUES(-1.0) UNION ALL SELECT y+0.1 FROM yaxis WHERE y<1.0), \
> +                  m(iter, cx, cy, x, y) AS ( \
> +                      SELECT 0, x, y, 0.0, 0.0 FROM xaxis, yaxis \
> +                      UNION ALL \
> +                      SELECT iter+1, cx, cy, x*x-y*y + cx, 2.0*x*y + cy FROM m \
> +                          WHERE (x*x + y*y) < 4.0 AND iter<28), \
> +                      m2(iter, cx, cy) AS ( \
> +                          SELECT max(iter), cx, cy FROM m GROUP BY cx, cy), \
> +                      a(t) AS ( \
> +                          SELECT group_concat( substr(' .+*#', 1+LEAST(iter/7,4), 1), '') \
> +                              FROM m2 GROUP BY cy) \
> +                  SELECT group_concat(TRIM(TRAILING FROM t),x'0a') FROM a;]])
> +
> +res = execute(s.stmt_id)
> +res.metadata
> +unprepare(s.stmt_id)
> +
> +-- Workflow with bindings is still the same.
> +--
> +s = prepare("SELECT a FROM test WHERE b = ?;")
> +execute(s.stmt_id, {'6'})
> +execute(s.stmt_id, {'9'})
> +unprepare(s.stmt_id)
> +
> +-- DML
> +s = prepare("INSERT INTO test VALUES (?, ?, ?);")
> +execute(s.stmt_id, {5, 6, '7'})
> +execute(s.stmt_id, {6, 10, '7'})
> +execute(s.stmt_id, {9, 11, '7'})
> +unprepare(s.stmt_id)
> +
> +-- EXPLAIN and PRAGMA work fine as well.
> +--
> +s1 = prepare("EXPLAIN SELECT a FROM test WHERE b = '3';")
> +res = execute(s1.stmt_id)
> +res.metadata
> +assert(res.rows ~= nil)
> +
> +s2 = prepare("EXPLAIN QUERY PLAN SELECT a FROM test WHERE b = '3';")
> +res = execute(s2.stmt_id)
> +res.metadata
> +assert(res.rows ~= nil)
> +
> +s3 = prepare("PRAGMA count_changes;")
> +execute(s3.stmt_id)
> +
> +unprepare(s2.stmt_id)
> +unprepare(s3.stmt_id)
> +unprepare(s1.stmt_id)
> +
> +-- Setting cache size to 0 is possible only in case if
> +-- there's no any prepared statements right now .
> +--
> +box.cfg{sql_cache_size = 0}
> +prepare("SELECT a FROM test;")
> +box.cfg{sql_cache_size = 0}
> +
> +-- Still with small size everything should work.
> +--
> +box.cfg{sql_cache_size = 1500}
> +
> +test_run:cmd("setopt delimiter ';'");
> +ok = nil
> +res = nil
> +_ = fiber.create(function()
> +    for i = 1, 5 do
> +        pcall(prepare, string.format("SELECT * FROM test WHERE a = %d;", i))
> +    end
> +    ok, res = pcall(prepare, "SELECT * FROM test WHERE b = '3';")
> +end);
> +while ok == nil do fiber.sleep(0.00001) end;
> +assert(ok == false);
> +res;
> +
> +-- Check that after fiber is dead, its session gets rid of
> +-- all prepared statements.
> +--
> +box.cfg{sql_cache_size = 0};
> +box.cfg{sql_cache_size = 3000};
> +
> +-- Make sure that if prepared statement is busy (is executed
> +-- right now), prepared statement is not used, i.e. statement
> +-- is compiled from scratch, executed and finilized.
> +--
> +box.schema.func.create('SLEEP', {language = 'Lua',
> +    body = 'function () fiber.sleep(0.1) return 1 end',
> +    exports = {'LUA', 'SQL'}});
> +
> +s = prepare("SELECT id, SLEEP() FROM test;");
> +assert(s ~= nil);
> +
> +function implicit_yield()
> +    prepare("SELECT id, SLEEP() FROM test;")
> +    execute("SELECT id, SLEEP() FROM test;")
> +end;
> +
> +f1 = fiber.new(implicit_yield)
> +f2 = fiber.new(implicit_yield)
> +f1:set_joinable(true)
> +f2:set_joinable(true)
> +
> +f1:join();
> +f2:join();
> +
> +unprepare(s.stmt_id);
> +
> +test_run:cmd("setopt delimiter ''");
> +
> +box.cfg{sql_cache_size = 5 * 1024 * 1024}
> +box.space.TEST:drop()
> +box.schema.func.drop('SLEEP')
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 19/20] netbox: introduce prepared statements

[Sergey Ostanevich]

Hi!

Thanks for the patch, one nit and one question.

Regards,
Sergos



On 20 Dec 15:47, Nikita Pettik wrote:
> This patch introduces support of prepared statements in IProto
> protocol. To achieve this new IProto command is added - IPROTO_PREPARE
> (key is 0x13). It is sent with one of two mandatory keys:
> IPROTO_SQL_TEXT (0x40 and assumes string value) or IPROTO_STMT_ID (0x43
> and assumes integer value). Depending on body it means to prepare or
> unprepare SQL statement: IPROTO_SQL_TEXT implies prepare request,
> meanwhile IPROTO_STMT_ID - unprepare.  Also to reply on PREPARE request a
> few response keys are added: IPROTO_BIND_METADATA (0x33 and contains
> parameters metadata of type map) and IPROTO_BIND_COUNT (0x34 and
> corresponds to the count of parameters to be bound).
> 
> Part of #2592
> ---
>  src/box/execute.c          |  83 ++++++++++++++++++++++++++++++
>  src/box/iproto.cc          |  68 +++++++++++++++++++++----
>  src/box/iproto_constants.c |   7 ++-
>  src/box/iproto_constants.h |   5 ++
>  src/box/lua/net_box.c      |  98 +++++++++++++++++++++++++++++++++--
>  src/box/lua/net_box.lua    |  27 ++++++++++
>  src/box/xrow.c             |  23 +++++++--
>  src/box/xrow.h             |   4 +-
>  test/box/misc.result       |   1 +
>  test/sql/engine.cfg        |   1 +
>  test/sql/iproto.result     |   2 +-
>  test/sql/prepared.result   | 124 ++++++++++++++++++++++++++-------------------
>  test/sql/prepared.test.lua |  76 +++++++++++++++++++--------
>  13 files changed, 420 insertions(+), 99 deletions(-)
> 
> diff --git a/src/box/execute.c b/src/box/execute.c
> index 09224c23a..7174d0d41 100644
> --- a/src/box/execute.c
> +++ b/src/box/execute.c
> @@ -328,6 +328,68 @@ sql_get_metadata(struct sql_stmt *stmt, struct obuf *out, int column_count)
>  	return 0;
>  }
>  
> +static inline int
> +sql_get_params_metadata(struct sql_stmt *stmt, struct obuf *out)
> +{
> +	int bind_count = sql_bind_parameter_count(stmt);
> +	int size = mp_sizeof_uint(IPROTO_BIND_METADATA) +
> +		   mp_sizeof_array(bind_count);
> +	char *pos = (char *) obuf_alloc(out, size);
> +	if (pos == NULL) {
> +		diag_set(OutOfMemory, size, "obuf_alloc", "pos");
> +		return -1;
> +	}
> +	pos = mp_encode_uint(pos, IPROTO_BIND_METADATA);
> +	pos = mp_encode_array(pos, bind_count);
> +	for (int i = 0; i < bind_count; ++i) {
> +		size_t size = mp_sizeof_map(2) +
> +			      mp_sizeof_uint(IPROTO_FIELD_NAME) +
> +			      mp_sizeof_uint(IPROTO_FIELD_TYPE);
> +		const char *name = sql_bind_parameter_name(stmt, i);
> +		if (name == NULL)
> +			name = "?";
> +		const char *type = "ANY";
> +		size += mp_sizeof_str(strlen(name));
> +		size += mp_sizeof_str(strlen(type));
> +		char *pos = (char *) obuf_alloc(out, size);
> +		if (pos == NULL) {
> +			diag_set(OutOfMemory, size, "obuf_alloc", "pos");
> +			return -1;
> +		}
> +		pos = mp_encode_map(pos, 2);
> +		pos = mp_encode_uint(pos, IPROTO_FIELD_NAME);
> +		pos = mp_encode_str(pos, name, strlen(name));
> +		pos = mp_encode_uint(pos, IPROTO_FIELD_TYPE);
> +		pos = mp_encode_str(pos, type, strlen(type));
> +	}
> +	return 0;
> +}
> +
> +static int
> +sql_get_prepare_common_keys(struct sql_stmt *stmt, struct obuf *out, int keys)
> +{
> +	const char *sql_str = sql_stmt_query_str(stmt);
> +	uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str));
> +	int size = mp_sizeof_map(keys) +
> +		   mp_sizeof_uint(IPROTO_STMT_ID) +
> +		   mp_sizeof_uint(stmt_id) +
> +		   mp_sizeof_uint(IPROTO_BIND_COUNT) +
> +		   mp_sizeof_uint(sql_bind_parameter_count(stmt));
> +	char *pos = (char *) obuf_alloc(out, size);
> +	if (pos == NULL) {
> +		diag_set(OutOfMemory, size, "obuf_alloc", "pos");
> +		return -1;
> +	}
> +	pos = mp_encode_map(pos, keys);
> +	pos = mp_encode_uint(pos, IPROTO_STMT_ID);
> +	pos = mp_encode_uint(pos, stmt_id);
> +	pos = mp_encode_uint(pos, IPROTO_BIND_COUNT);
> +	pos = mp_encode_uint(pos, sql_bind_parameter_count(stmt));
> +	if (sql_get_params_metadata(stmt, out) != 0)
> +		return -1;
> +	return 0;
> +}
> +
>  static int
>  port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  {
> @@ -409,6 +471,27 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
>  		}
>  		break;
>  	}
> +	case DQL_PREPARE: {
> +		/* Format is following:
> +		 * query_id,
> +		 * param_count,
> +		 * params {name, type},
> +		 * metadata {name, type}
> +		 */
> +		int keys = 4;
> +		if (sql_get_prepare_common_keys(stmt, out, keys) != 0)
> +			return -1;
> +		return sql_get_metadata(stmt, out, sql_column_count(stmt));
> +	}
> +	case DML_PREPARE: {
> +		/* Format is following:
> +		 * query_id,
> +		 * param_count,
> +		 * params {name, type},
> +		 */
> +		int keys = 3;
> +		return sql_get_prepare_common_keys(stmt, out, keys);
> +		}
>  	default: {
>  		unreachable();
>  	}
> diff --git a/src/box/iproto.cc b/src/box/iproto.cc
> index c39b8e7bf..fac94658a 100644
> --- a/src/box/iproto.cc
> +++ b/src/box/iproto.cc
> @@ -178,7 +178,7 @@ struct iproto_msg
>  		struct call_request call;
>  		/** Authentication request. */
>  		struct auth_request auth;
> -		/* SQL request, if this is the EXECUTE request. */
> +		/* SQL request, if this is the EXECUTE/PREPARE request. */
>  		struct sql_request sql;
>  		/** In case of iproto parse error, saved diagnostics. */
>  		struct diag diag;
> @@ -1209,6 +1209,7 @@ static const struct cmsg_hop *dml_route[IPROTO_TYPE_STAT_MAX] = {
>  	call_route,                             /* IPROTO_CALL */
>  	sql_route,                              /* IPROTO_EXECUTE */
>  	NULL,                                   /* IPROTO_NOP */
> +	sql_route,                              /* IPROTO_PREPARE */
>  };
>  
>  static const struct cmsg_hop join_route[] = {
> @@ -1264,6 +1265,7 @@ iproto_msg_decode(struct iproto_msg *msg, const char **pos, const char *reqend,
>  		cmsg_init(&msg->base, call_route);
>  		break;
>  	case IPROTO_EXECUTE:
> +	case IPROTO_PREPARE:
>  		if (xrow_decode_sql(&msg->header, &msg->sql) != 0)
>  			goto error;
>  		cmsg_init(&msg->base, sql_route);
> @@ -1710,23 +1712,64 @@ tx_process_sql(struct cmsg *m)
>  	int bind_count = 0;
>  	const char *sql;
>  	uint32_t len;
> +	bool is_unprepare = false;
>  
>  	tx_fiber_init(msg->connection->session, msg->header.sync);
>  
>  	if (tx_check_schema(msg->header.schema_version))
>  		goto error;
> -	assert(msg->header.type == IPROTO_EXECUTE);
> +	assert(msg->header.type == IPROTO_EXECUTE ||
> +	       msg->header.type == IPROTO_PREPARE);
>  	tx_inject_delay();
>  	if (msg->sql.bind != NULL) {
>  		bind_count = sql_bind_list_decode(msg->sql.bind, &bind);
>  		if (bind_count < 0)
>  			goto error;
>  	}
> -	sql = msg->sql.sql_text;
> -	sql = mp_decode_str(&sql, &len);
> -	if (sql_prepare_and_execute(sql, len, bind, bind_count, &port,
> -				    &fiber()->gc) != 0)
> -		goto error;
> +	/*
> +	 * There are four options:
> +	 * 1. Prepare SQL query (IPROTO_PREPARE + SQL string);
> +	 * 2. Unprepare SQL query (IPROTO_PREPARE + stmt id);
> +	 * 3. Execute SQL query (IPROTO_EXECUTE + SQL string);
> +	 * 4. Execute prepared query (IPROTO_EXECUTE + stmt id).
> +	 */
> +	if (msg->header.type == IPROTO_EXECUTE) {
> +		if (msg->sql.sql_text != NULL) {
> +			assert(msg->sql.stmt_id == NULL);
> +			sql = msg->sql.sql_text;
> +			sql = mp_decode_str(&sql, &len);
> +			if (sql_prepare_and_execute(sql, len, bind, bind_count,
> +						    &port, &fiber()->gc) != 0)
> +				goto error;
> +		} else {
> +			assert(msg->sql.sql_text == NULL);
> +			assert(msg->sql.stmt_id != NULL);
> +			sql = msg->sql.stmt_id;
> +			uint32_t stmt_id = mp_decode_uint(&sql);
> +			if (sql_execute_prepared(stmt_id, bind, bind_count,
> +						 &port, &fiber()->gc) != 0)
> +				goto error;
> +		}
> +	} else {
> +		/* IPROTO_PREPARE */
> +		if (msg->sql.sql_text != NULL) {
> +			assert(msg->sql.stmt_id == NULL);
> +			sql = msg->sql.sql_text;
> +			sql = mp_decode_str(&sql, &len);
> +			if (sql_prepare(sql, len, &port) != 0)
> +				goto error;
> +		} else {
> +			/* UNPREPARE */
> +			assert(msg->sql.sql_text == NULL);
> +			assert(msg->sql.stmt_id != NULL);
> +			sql = msg->sql.stmt_id;
> +			uint32_t stmt_id = mp_decode_uint(&sql);
> +			if (sql_unprepare(stmt_id) != 0)
> +				goto error;
> +			is_unprepare = true;
> +		}
> +	}
> +
>  	/*
>  	 * Take an obuf only after execute(). Else the buffer can
>  	 * become out of date during yield.
> @@ -1738,12 +1781,15 @@ tx_process_sql(struct cmsg *m)
>  		port_destroy(&port);
>  		goto error;
>  	}
> -	if (port_dump_msgpack(&port, out) != 0) {
> +	/* Nothing to dump in case of UNPREPARE request. */
> +	if (! is_unprepare) {

Unary ops - no spaces.

> +		if (port_dump_msgpack(&port, out) != 0) {
> +			port_destroy(&port);
> +			obuf_rollback_to_svp(out, &header_svp);
> +			goto error;
> +		}
>  		port_destroy(&port);
> -		obuf_rollback_to_svp(out, &header_svp);
> -		goto error;
>  	}
> -	port_destroy(&port);
>  	iproto_reply_sql(out, &header_svp, msg->header.sync, schema_version);
>  	iproto_wpos_create(&msg->wpos, out);
>  	return;
> diff --git a/src/box/iproto_constants.c b/src/box/iproto_constants.c
> index 09ded1ecb..029d9888c 100644
> --- a/src/box/iproto_constants.c
> +++ b/src/box/iproto_constants.c
> @@ -107,6 +107,7 @@ const char *iproto_type_strs[] =
>  	"CALL",
>  	"EXECUTE",
>  	NULL, /* NOP */
> +	"PREPARE",
>  };
>  
>  #define bit(c) (1ULL<<IPROTO_##c)
> @@ -124,6 +125,7 @@ const uint64_t iproto_body_key_map[IPROTO_TYPE_STAT_MAX] = {
>  	0,                                                     /* CALL */
>  	0,                                                     /* EXECUTE */
>  	0,                                                     /* NOP */
> +	0,                                                     /* PREPARE */
>  };
>  #undef bit
>  
> @@ -179,8 +181,8 @@ const char *iproto_key_strs[IPROTO_KEY_MAX] = {
>  	"data",             /* 0x30 */
>  	"error",            /* 0x31 */
>  	"metadata",         /* 0x32 */
> -	NULL,               /* 0x33 */
> -	NULL,               /* 0x34 */
> +	"bind meta",        /* 0x33 */
> +	"bind count",       /* 0x34 */
>  	NULL,               /* 0x35 */
>  	NULL,               /* 0x36 */
>  	NULL,               /* 0x37 */
> @@ -195,6 +197,7 @@ const char *iproto_key_strs[IPROTO_KEY_MAX] = {
>  	"SQL text",         /* 0x40 */
>  	"SQL bind",         /* 0x41 */
>  	"SQL info",         /* 0x42 */
> +	"stmt id",          /* 0x43 */
>  };
>  
>  const char *vy_page_info_key_strs[VY_PAGE_INFO_KEY_MAX] = {
> diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h
> index 5e8a7d483..34d0f49c6 100644
> --- a/src/box/iproto_constants.h
> +++ b/src/box/iproto_constants.h
> @@ -110,6 +110,8 @@ enum iproto_key {
>  	 * ]
>  	 */
>  	IPROTO_METADATA = 0x32,
> +	IPROTO_BIND_METADATA = 0x33,
> +	IPROTO_BIND_COUNT = 0x34,
>  
>  	/* Leave a gap between response keys and SQL keys. */
>  	IPROTO_SQL_TEXT = 0x40,
> @@ -120,6 +122,7 @@ enum iproto_key {
>  	 * }
>  	 */
>  	IPROTO_SQL_INFO = 0x42,
> +	IPROTO_STMT_ID = 0x43,
>  	IPROTO_KEY_MAX
>  };
>  
> @@ -203,6 +206,8 @@ enum iproto_type {
>  	IPROTO_EXECUTE = 11,
>  	/** No operation. Treated as DML, used to bump LSN. */
>  	IPROTO_NOP = 12,
> +	/** Prepare SQL statement. */
> +	IPROTO_PREPARE = 13,
>  	/** The maximum typecode used for box.stat() */
>  	IPROTO_TYPE_STAT_MAX,
>  
> diff --git a/src/box/lua/net_box.c b/src/box/lua/net_box.c
> index 001af95dc..aa8a15e30 100644
> --- a/src/box/lua/net_box.c
> +++ b/src/box/lua/net_box.c
> @@ -570,10 +570,16 @@ netbox_encode_execute(lua_State *L)
>  
>  	mpstream_encode_map(&stream, 3);
>  
> -	size_t len;
> -	const char *query = lua_tolstring(L, 3, &len);
> -	mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
> -	mpstream_encode_strn(&stream, query, len);
> +	if (lua_type(L, 3) == LUA_TNUMBER) {
> +		uint32_t query_id = lua_tointeger(L, 3);
> +		mpstream_encode_uint(&stream, IPROTO_STMT_ID);
> +		mpstream_encode_uint(&stream, query_id);
> +	} else {
> +		size_t len;
> +		const char *query = lua_tolstring(L, 3, &len);
> +		mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
> +		mpstream_encode_strn(&stream, query, len);
> +	}
>  
>  	mpstream_encode_uint(&stream, IPROTO_SQL_BIND);
>  	luamp_encode_tuple(L, cfg, &stream, 4);
> @@ -585,6 +591,32 @@ netbox_encode_execute(lua_State *L)
>  	return 0;
>  }
>  
> +static int
> +netbox_encode_prepare(lua_State *L)
> +{
> +	if (lua_gettop(L) < 3)
> +		return luaL_error(L, "Usage: netbox.encode_prepare(ibuf, "\
> +				     "sync, query)");
> +	struct mpstream stream;
> +	size_t svp = netbox_prepare_request(L, &stream, IPROTO_PREPARE);
> +
> +	mpstream_encode_map(&stream, 1);
> +
> +	if (lua_type(L, 3) == LUA_TNUMBER) {
> +		uint32_t query_id = lua_tointeger(L, 3);
> +		mpstream_encode_uint(&stream, IPROTO_STMT_ID);
> +		mpstream_encode_uint(&stream, query_id);
> +	} else {
> +		size_t len;
> +		const char *query = lua_tolstring(L, 3, &len);
> +		mpstream_encode_uint(&stream, IPROTO_SQL_TEXT);
> +		mpstream_encode_strn(&stream, query, len);
> +	};
> +
> +	netbox_encode_request(&stream, svp);
> +	return 0;
> +}
> +
>  /**
>   * Decode IPROTO_DATA into tuples array.
>   * @param L Lua stack to push result on.
> @@ -752,6 +784,62 @@ netbox_decode_execute(struct lua_State *L)
>  	return 2;
>  }
>  
> +static int
> +netbox_decode_prepare(struct lua_State *L)
> +{
> +	uint32_t ctypeid;
> +	const char *data = *(const char **)luaL_checkcdata(L, 1, &ctypeid);
> +	assert(mp_typeof(*data) == MP_MAP);
> +	uint32_t map_size = mp_decode_map(&data);
> +	int stmt_id_idx = 0, meta_idx = 0, bind_meta_idx = 0,
> +	    bind_count_idx = 0;
> +	uint32_t stmt_id = 0;
> +	for (uint32_t i = 0; i < map_size; ++i) {
> +		uint32_t key = mp_decode_uint(&data);
> +		switch(key) {
> +		case IPROTO_STMT_ID: {
> +			stmt_id = mp_decode_uint(&data);
> +			luaL_pushuint64(L, stmt_id);
> +			stmt_id_idx = i - map_size;
> +			break;
> +		}
> +		case IPROTO_METADATA: {
> +			netbox_decode_metadata(L, &data);
> +			meta_idx = i - map_size;
> +			break;
> +		}
> +		case IPROTO_BIND_METADATA: {
> +			netbox_decode_metadata(L, &data);
> +			bind_meta_idx = i - map_size;
> +			break;
> +		}
> +		default: {
> +			assert(key == IPROTO_BIND_COUNT);
> +			uint32_t bind_count = mp_decode_uint(&data);
> +			luaL_pushuint64(L, bind_count);
> +			bind_count_idx = i - map_size;
> +			break;
> +		}}
> +	}
> +	/* These fields must be present in response. */
> +	assert(stmt_id_idx * bind_meta_idx * bind_count_idx != 0);
> +	/* General meta is presented only in DQL responses. */
> +	lua_createtable(L, 0, meta_idx != 0 ? 4 : 3);
> +	lua_pushvalue(L, stmt_id_idx - 1);
> +	lua_setfield(L, -2, "stmt_id");
> +	lua_pushvalue(L, bind_count_idx - 1);
> +	lua_setfield(L, -2, "param_count");
> +	lua_pushvalue(L, bind_meta_idx - 1);
> +	lua_setfield(L, -2, "params");
> +	if (meta_idx != 0) {
> +		lua_pushvalue(L, meta_idx - 1);
> +		lua_setfield(L, -2, "metadata");
> +	}
> +
> +	*(const char **)luaL_pushcdata(L, ctypeid) = data;
> +	return 2;
> +}
> +
>  int
>  luaopen_net_box(struct lua_State *L)
>  {
> @@ -767,11 +855,13 @@ luaopen_net_box(struct lua_State *L)
>  		{ "encode_update",  netbox_encode_update },
>  		{ "encode_upsert",  netbox_encode_upsert },
>  		{ "encode_execute", netbox_encode_execute},
> +		{ "encode_prepare", netbox_encode_prepare},
>  		{ "encode_auth",    netbox_encode_auth },
>  		{ "decode_greeting",netbox_decode_greeting },
>  		{ "communicate",    netbox_communicate },
>  		{ "decode_select",  netbox_decode_select },
>  		{ "decode_execute", netbox_decode_execute },
> +		{ "decode_prepare", netbox_decode_prepare },
>  		{ NULL, NULL}
>  	};
>  	/* luaL_register_module polutes _G */
> diff --git a/src/box/lua/net_box.lua b/src/box/lua/net_box.lua
> index c2e1bb9c4..b4811edfa 100644
> --- a/src/box/lua/net_box.lua
> +++ b/src/box/lua/net_box.lua
> @@ -104,6 +104,8 @@ local method_encoder = {
>      upsert  = internal.encode_upsert,
>      select  = internal.encode_select,
>      execute = internal.encode_execute,
> +    prepare = internal.encode_prepare,
> +    unprepare = internal.encode_prepare,
>      get     = internal.encode_select,
>      min     = internal.encode_select,
>      max     = internal.encode_select,
> @@ -128,6 +130,8 @@ local method_decoder = {
>      upsert  = decode_nil,
>      select  = internal.decode_select,
>      execute = internal.decode_execute,
> +    prepare = internal.decode_prepare,
> +    unprepare = decode_nil,

should it be internal.decode_prepare?

>      get     = decode_get,
>      min     = decode_get,
>      max     = decode_get,
> @@ -1197,6 +1201,29 @@ function remote_methods:execute(query, parameters, sql_opts, netbox_opts)
>                           sql_opts or {})
>  end
>  
> +function remote_methods:prepare(query, parameters, sql_opts, netbox_opts)
> +    check_remote_arg(self, "prepare")
> +    if type(query) ~= "string" then
> +        box.error(box.error.SQL_PREPARE, "expected string as SQL statement")
> +    end
> +    if sql_opts ~= nil then
> +        box.error(box.error.UNSUPPORTED, "prepare", "options")
> +    end
> +    return self:_request('prepare', netbox_opts, nil, query)
> +end
> +
> +function remote_methods:unprepare(query, parameters, sql_opts, netbox_opts)
> +    check_remote_arg(self, "unprepare")
> +    if type(query) ~= "number" then
> +        box.error("query id is expected to be numeric")
> +    end
> +    if sql_opts ~= nil then
> +        box.error(box.error.UNSUPPORTED, "unprepare", "options")
> +    end
> +    return self:_request('unprepare', netbox_opts, nil, query, parameters or {},
> +                         sql_opts or {})
> +end
> +
>  function remote_methods:wait_state(state, timeout)
>      check_remote_arg(self, 'wait_state')
>      if timeout == nil then
> diff --git a/src/box/xrow.c b/src/box/xrow.c
> index 18bf08971..88f308be5 100644
> --- a/src/box/xrow.c
> +++ b/src/box/xrow.c
> @@ -576,9 +576,11 @@ error:
>  	uint32_t map_size = mp_decode_map(&data);
>  	request->sql_text = NULL;
>  	request->bind = NULL;
> +	request->stmt_id = NULL;
>  	for (uint32_t i = 0; i < map_size; ++i) {
>  		uint8_t key = *data;
> -		if (key != IPROTO_SQL_BIND && key != IPROTO_SQL_TEXT) {
> +		if (key != IPROTO_SQL_BIND && key != IPROTO_SQL_TEXT &&
> +		    key != IPROTO_STMT_ID) {
>  			mp_check(&data, end);   /* skip the key */
>  			mp_check(&data, end);   /* skip the value */
>  			continue;
> @@ -588,12 +590,23 @@ error:
>  			goto error;
>  		if (key == IPROTO_SQL_BIND)
>  			request->bind = value;
> -		else
> +		else if (key == IPROTO_SQL_TEXT)
>  			request->sql_text = value;
> +		else
> +			request->stmt_id = value;
>  	}
> -	if (request->sql_text == NULL) {
> -		xrow_on_decode_err(row->body[0].iov_base, end, ER_MISSING_REQUEST_FIELD,
> -			 iproto_key_name(IPROTO_SQL_TEXT));
> +	if (request->sql_text != NULL && request->stmt_id != NULL) {
> +		xrow_on_decode_err(row->body[0].iov_base, end, ER_INVALID_MSGPACK,
> +				   "SQL text and statement id are incompatible "\
> +				   "options in one request: choose one");
> +		return -1;
> +	}
> +	if (request->sql_text == NULL && request->stmt_id == NULL) {
> +		xrow_on_decode_err(row->body[0].iov_base, end,
> +				   ER_MISSING_REQUEST_FIELD,
> +				   tt_sprintf("%s or %s",
> +					      iproto_key_name(IPROTO_SQL_TEXT),
> +					      iproto_key_name(IPROTO_STMT_ID)));
>  		return -1;
>  	}
>  	if (data != end)
> diff --git a/src/box/xrow.h b/src/box/xrow.h
> index 60def2d3c..a4d8dc015 100644
> --- a/src/box/xrow.h
> +++ b/src/box/xrow.h
> @@ -526,12 +526,14 @@ int
>  iproto_reply_error(struct obuf *out, const struct error *e, uint64_t sync,
>  		   uint32_t schema_version);
>  
> -/** EXECUTE request. */
> +/** EXECUTE/PREPARE request. */
>  struct sql_request {
>  	/** SQL statement text. */
>  	const char *sql_text;
>  	/** MessagePack array of parameters. */
>  	const char *bind;
> +	/** ID of prepared statement. In this case @sql_text == NULL. */
> +	const char *stmt_id;
>  };
>  
>  /**
> diff --git a/test/box/misc.result b/test/box/misc.result
> index 90923f28e..79fd49442 100644
> --- a/test/box/misc.result
> +++ b/test/box/misc.result
> @@ -250,6 +250,7 @@ t;
>    - EVAL
>    - CALL
>    - ERROR
> +  - PREPARE
>    - REPLACE
>    - UPSERT
>    - AUTH
> diff --git a/test/sql/engine.cfg b/test/sql/engine.cfg
> index a1b4b0fc5..e38bec24e 100644
> --- a/test/sql/engine.cfg
> +++ b/test/sql/engine.cfg
> @@ -10,6 +10,7 @@
>          "local": {"remote": "false"}
>      },
>      "prepared.test.lua": {
> +        "remote": {"remote": "true"},
>          "local": {"remote": "false"}
>      },
>      "*": {
> diff --git a/test/sql/iproto.result b/test/sql/iproto.result
> index 67acd0ac1..4dfbfce50 100644
> --- a/test/sql/iproto.result
> +++ b/test/sql/iproto.result
> @@ -119,7 +119,7 @@ cn:execute('select id as identifier from test where a = 5;')
>  -- netbox API errors.
>  cn:execute(100)
>  ---
> -- error: Syntax error near '100'
> +- error: Prepared statement with id 100 does not exist
>  ...
>  cn:execute('select 1', nil, {dry_run = true})
>  ---
> diff --git a/test/sql/prepared.result b/test/sql/prepared.result
> index bd37cfdd7..2f4983b00 100644
> --- a/test/sql/prepared.result
> +++ b/test/sql/prepared.result
> @@ -12,34 +12,49 @@ fiber = require('fiber')
>  -- Wrappers to make remote and local execution interface return
>  -- same result pattern.
>  --
> -test_run:cmd("setopt delimiter ';'")
> +is_remote = test_run:get_cfg('remote') == 'true'
>   | ---
> - | - true
>   | ...
> -execute = function(...)
> -    local res, err = box.execute(...)
> -    if err ~= nil then
> -        error(err)
> -    end
> -    return res
> -end;
> +execute = nil
>   | ---
>   | ...
> -prepare = function(...)
> -    local res, err = box.prepare(...)
> -    if err ~= nil then
> -        error(err)
> -    end
> -    return res
> -end;
> +prepare = nil
> + | ---
> + | ...
> +
> +test_run:cmd("setopt delimiter ';'")
>   | ---
> + | - true
>   | ...
> -unprepare = function(...)
> -    local res, err = box.unprepare(...)
> -    if err ~= nil then
> -        error(err)
> +if is_remote then
> +    box.schema.user.grant('guest','read, write, execute', 'universe')
> +    box.schema.user.grant('guest', 'create', 'space')
> +    cn = remote.connect(box.cfg.listen)
> +    execute = function(...) return cn:execute(...) end
> +    prepare = function(...) return cn:prepare(...) end
> +    unprepare = function(...) return cn:unprepare(...) end
> +else
> +    execute = function(...)
> +        local res, err = box.execute(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
> +    end
> +    prepare = function(...)
> +        local res, err = box.prepare(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
> +    end
> +    unprepare = function(...)
> +        local res, err = box.unprepare(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
>      end
> -    return res
>  end;
>   | ---
>   | ...
> @@ -128,31 +143,26 @@ execute(s.stmt_id, {1, 3})
>   |     type: string
>   |   rows: []
>   | ...
> -s:execute({1, 2})
> +
> +test_run:cmd("setopt delimiter ';'")
>   | ---
> - | - metadata:
> - |   - name: ID
> - |     type: integer
> - |   - name: A
> - |     type: number
> - |   - name: B
> - |     type: string
> - |   rows:
> - |   - [1, 2, '3']
> + | - true
>   | ...
> -s:execute({1, 3})
> +if not is_remote then
> +    res = s:execute({1, 2})
> +    assert(res ~= nil)
> +    res = s:execute({1, 3})
> +    assert(res ~= nil)
> +end;
>   | ---
> - | - metadata:
> - |   - name: ID
> - |     type: integer
> - |   - name: A
> - |     type: number
> - |   - name: B
> - |     type: string
> - |   rows: []
>   | ...
> -s:unprepare()
> +test_run:cmd("setopt delimiter ''");
>   | ---
> + | - true
> + | ...
> +unprepare(s.stmt_id)
> + | ---
> + | - null
>   | ...
>  
>  -- Test preparation of different types of queries.
> @@ -338,6 +348,7 @@ _ = prepare("SELECT a FROM test WHERE b = '3';")
>  s = prepare("SELECT a FROM test WHERE b = '3';")
>   | ---
>   | ...
> +
>  execute(s.stmt_id)
>   | ---
>   | - metadata:
> @@ -354,21 +365,21 @@ execute(s.stmt_id)
>   |   rows:
>   |   - [2]
>   | ...
> -s:execute()
> +test_run:cmd("setopt delimiter ';'")
>   | ---
> - | - metadata:
> - |   - name: A
> - |     type: number
> - |   rows:
> - |   - [2]
> + | - true
>   | ...
> -s:execute()
> +if not is_remote then
> +    res = s:execute()
> +    assert(res ~= nil)
> +    res = s:execute()
> +    assert(res ~= nil)
> +end;
>   | ---
> - | - metadata:
> - |   - name: A
> - |     type: number
> - |   rows:
> - |   - [2]
> + | ...
> +test_run:cmd("setopt delimiter ''");
> + | ---
> + | - true
>   | ...
>  unprepare(s.stmt_id)
>   | ---
> @@ -671,6 +682,13 @@ unprepare(s.stmt_id);
>   | - null
>   | ...
>  
> +if is_remote then
> +    cn:close()
> +    box.schema.user.revoke('guest', 'read, write, execute', 'universe')
> +    box.schema.user.revoke('guest', 'create', 'space')
> +end;
> + | ---
> + | ...
>  test_run:cmd("setopt delimiter ''");
>   | ---
>   | - true
> diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
> index 49d2fb3ae..c464cc21a 100644
> --- a/test/sql/prepared.test.lua
> +++ b/test/sql/prepared.test.lua
> @@ -5,27 +5,40 @@ fiber = require('fiber')
>  -- Wrappers to make remote and local execution interface return
>  -- same result pattern.
>  --
> +is_remote = test_run:get_cfg('remote') == 'true'
> +execute = nil
> +prepare = nil
> +
>  test_run:cmd("setopt delimiter ';'")
> -execute = function(...)
> -    local res, err = box.execute(...)
> -    if err ~= nil then
> -        error(err)
> +if is_remote then
> +    box.schema.user.grant('guest','read, write, execute', 'universe')
> +    box.schema.user.grant('guest', 'create', 'space')
> +    cn = remote.connect(box.cfg.listen)
> +    execute = function(...) return cn:execute(...) end
> +    prepare = function(...) return cn:prepare(...) end
> +    unprepare = function(...) return cn:unprepare(...) end
> +else
> +    execute = function(...)
> +        local res, err = box.execute(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
>      end
> -    return res
> -end;
> -prepare = function(...)
> -    local res, err = box.prepare(...)
> -    if err ~= nil then
> -        error(err)
> +    prepare = function(...)
> +        local res, err = box.prepare(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
>      end
> -    return res
> -end;
> -unprepare = function(...)
> -    local res, err = box.unprepare(...)
> -    if err ~= nil then
> -        error(err)
> +    unprepare = function(...)
> +        local res, err = box.unprepare(...)
> +        if err ~= nil then
> +            error(err)
> +        end
> +        return res
>      end
> -    return res
>  end;
>  
>  test_run:cmd("setopt delimiter ''");
> @@ -46,9 +59,16 @@ s.params
>  s.param_count
>  execute(s.stmt_id, {1, 2})
>  execute(s.stmt_id, {1, 3})
> -s:execute({1, 2})
> -s:execute({1, 3})
> -s:unprepare()
> +
> +test_run:cmd("setopt delimiter ';'")
> +if not is_remote then
> +    res = s:execute({1, 2})
> +    assert(res ~= nil)
> +    res = s:execute({1, 3})
> +    assert(res ~= nil)
> +end;
> +test_run:cmd("setopt delimiter ''");
> +unprepare(s.stmt_id)
>  
>  -- Test preparation of different types of queries.
>  -- Let's start from DDL. It doesn't make much sense since
> @@ -111,10 +131,17 @@ space:replace{4, 5, '6'}
>  space:replace{7, 8.5, '9'}
>  _ = prepare("SELECT a FROM test WHERE b = '3';")
>  s = prepare("SELECT a FROM test WHERE b = '3';")
> +
>  execute(s.stmt_id)
>  execute(s.stmt_id)
> -s:execute()
> -s:execute()
> +test_run:cmd("setopt delimiter ';'")
> +if not is_remote then
> +    res = s:execute()
> +    assert(res ~= nil)
> +    res = s:execute()
> +    assert(res ~= nil)
> +end;
> +test_run:cmd("setopt delimiter ''");
>  unprepare(s.stmt_id)
>  
>  s = prepare("SELECT count(*), count(a - 3), max(b), abs(id) FROM test WHERE b = '3';")
> @@ -233,6 +260,11 @@ f2:join();
>  
>  unprepare(s.stmt_id);
>  
> +if is_remote then
> +    cn:close()
> +    box.schema.user.revoke('guest', 'read, write, execute', 'universe')
> +    box.schema.user.revoke('guest', 'create', 'space')
> +end;
>  test_run:cmd("setopt delimiter ''");
>  
>  box.cfg{sql_cache_size = 5 * 1024 * 1024}
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 20/20] sql: add cache statistics to box.info

[Sergey Ostanevich]

Hi!

Thanks for the whole patchset!
I got one major question: in case session is ended - there should be a
refcnt reduction for all prepared stmts of this session. I didn't see
that in the code.

Regards,
Sergos

On 20 Dec 15:47, Nikita Pettik wrote:
> To track current memory occupied by prepared statements and number of
> them, let's extend box.info submodule with .sql statistics: now it
> contains current total size of prepared statements and their count.
> 
> @TarantoolBot document
> Title: Prepared statements in SQL
> 
> Now it is possible to prepare (i.e. compile into byte-code and save to
> the cache) statement and execute it several times. Mechanism is similar
> to ones in other DBs. Prepared statement is identified by numeric
> ID, which are returned alongside with prepared statement handle.
> Note that they are not sequential and represent value of hash function
> applied to the string containing original SQL request.
> Prepared statement holder is shared among all sessions. However, session
> has access only to statements which have been prepared in scope of it.
> There's no eviction policy like in any cache; to remove statement from
> holder explicit unprepare request is required. Alternatively, session's
> disconnect also removes statements from holder.
> Several sessions can share one prepared statement, which will be
> destroyed when all related sessions are disconnected or send unprepare
> request. Memory limit for prepared statements is adjusted by
> box.cfg{sql_cache_size} handle (can be set dynamically;
> 
> Any DDL operation leads to expiration of all prepared statements: they
> should be manually removed or re-prepared.

Does it mean that for user code there should be called unprepare(s.id) and
s = prepare(orginal sql text)? If I got it right from previous patches
the re-prepare is called automatically?

> Prepared statements are available in local mode (i.e. via box.prepare()
> function) and are supported in IProto protocol. In the latter case
> next IProto keys are used to make up/receive requests/responses:
> IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
> one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
> or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
> means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
> request, meanwhile IPROTO_STMT_ID - unprepare;
> IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
> and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
> be bound) are response keys. They are mandatory members of result of
> IPROTO_PREPARE execution.
> 
> To track statistics of used memory and number of currently prepared
> statements, box.info is extended with SQL statistics:
> 
> box.info:sql().cache.stmt_count - number of prepared statements;
> box.info:sql().cache.size - size of occupied by prepared statements memory.
> 
> Typical workflow with prepared statements is following:
> 
> s = box.prepare("SELECT * FROM t WHERE id = ?;")
> s:execute({1}) or box.execute(s.sql_str, {1})
> s:execute({2}) or box.execute(s.sql_str, {2})
> s:unprepare() or box.unprepare(s.query_id)
> 
> Structure of object is following (member : type):
> 
> - stmt_id: integer
>   execute: function
>   params: map [name : string, type : integer]
>   unprepare: function
>   metadata: map [name : string, type : integer]
>   param_count: integer
> ...
> 
> In terms of remote connection:
> 
> cn = netbox:connect(addr)
> s = cn:prepare("SELECT * FROM t WHERE id = ?;")
> cn:execute(s.sql_str, {1})
> cn:unprepare(s.query_id)
> 
> Closese #2592

Misprint above: Closes

> ---
>  src/box/lua/info.c         | 25 +++++++++++++++++++++++++
>  src/box/sql_stmt_cache.c   | 16 ++++++++++++++++
>  src/box/sql_stmt_cache.h   |  8 ++++++++
>  test/box/info.result       |  1 +
>  test/sql/prepared.result   | 34 +++++++++++++++++++++++++++++++++-
>  test/sql/prepared.test.lua | 12 +++++++++++-
>  6 files changed, 94 insertions(+), 2 deletions(-)
> 
> diff --git a/src/box/lua/info.c b/src/box/lua/info.c
> index e029e0e17..8933ea829 100644
> --- a/src/box/lua/info.c
> +++ b/src/box/lua/info.c
> @@ -45,6 +45,7 @@
>  #include "box/gc.h"
>  #include "box/engine.h"
>  #include "box/vinyl.h"
> +#include "box/sql_stmt_cache.h"
>  #include "main.h"
>  #include "version.h"
>  #include "box/box.h"
> @@ -494,6 +495,29 @@ lbox_info_vinyl(struct lua_State *L)
>  	return 1;
>  }
>  
> +static int
> +lbox_info_sql_call(struct lua_State *L)
> +{
> +	struct info_handler h;
> +	luaT_info_handler_create(&h, L);
> +	sql_stmt_cache_stat(&h);
> +
> +	return 1;
> +}
> +
> +static int
> +lbox_info_sql(struct lua_State *L)
> +{
> +	lua_newtable(L);
> +	lua_newtable(L); /* metatable */
> +	lua_pushstring(L, "__call");
> +	lua_pushcfunction(L, lbox_info_sql_call);
> +	lua_settable(L, -3);
> +
> +	lua_setmetatable(L, -2);
> +	return 1;
> +}
> +
>  static const struct luaL_Reg lbox_info_dynamic_meta[] = {
>  	{"id", lbox_info_id},
>  	{"uuid", lbox_info_uuid},
> @@ -509,6 +533,7 @@ static const struct luaL_Reg lbox_info_dynamic_meta[] = {
>  	{"memory", lbox_info_memory},
>  	{"gc", lbox_info_gc},
>  	{"vinyl", lbox_info_vinyl},
> +	{"sql", lbox_info_sql},
>  	{NULL, NULL}
>  };
>  
> diff --git a/src/box/sql_stmt_cache.c b/src/box/sql_stmt_cache.c
> index 742e4135c..a4f5f2745 100644
> --- a/src/box/sql_stmt_cache.c
> +++ b/src/box/sql_stmt_cache.c
> @@ -34,6 +34,7 @@
>  #include "error.h"
>  #include "execute.h"
>  #include "diag.h"
> +#include "info/info.h"
>  
>  static struct sql_stmt_cache sql_stmt_cache;
>  
> @@ -48,6 +49,21 @@ sql_stmt_cache_init()
>  	rlist_create(&sql_stmt_cache.gc_queue);
>  }
>  
> +void
> +sql_stmt_cache_stat(struct info_handler *h)
> +{
> +	info_begin(h);
> +	info_table_begin(h, "cache");
> +	info_append_int(h, "size", sql_stmt_cache.mem_used);
> +	uint32_t entry_count = 0;
> +	mh_int_t i;
> +	mh_foreach(sql_stmt_cache.hash, i)
> +		entry_count++;
> +	info_append_int(h, "stmt_count", entry_count);
> +	info_table_end(h);
> +	info_end(h);
> +}
> +
>  static size_t
>  sql_cache_entry_sizeof(struct sql_stmt *stmt)
>  {
> diff --git a/src/box/sql_stmt_cache.h b/src/box/sql_stmt_cache.h
> index f3935a27f..468cbc9a0 100644
> --- a/src/box/sql_stmt_cache.h
> +++ b/src/box/sql_stmt_cache.h
> @@ -41,6 +41,7 @@ extern "C" {
>  
>  struct sql_stmt;
>  struct mh_i64ptr_t;
> +struct info_handler;
>  
>  struct stmt_cache_entry {
>  	/** Prepared statement itself. */
> @@ -90,6 +91,13 @@ struct sql_stmt_cache {
>  void
>  sql_stmt_cache_init();
>  
> +/**
> + * Store statistics concerning cache (current size and number
> + * of statements in it) into info handler @h.
> + */
> +void
> +sql_stmt_cache_stat(struct info_handler *h);
> +
>  /**
>   * Erase session local hash: unref statements belong to this
>   * session and deallocate hash itself.
> diff --git a/test/box/info.result b/test/box/info.result
> index af81f7add..2e84cbbe3 100644
> --- a/test/box/info.result
> +++ b/test/box/info.result
> @@ -84,6 +84,7 @@ t
>    - replication
>    - ro
>    - signature
> +  - sql
>    - status
>    - uptime
>    - uuid
> diff --git a/test/sql/prepared.result b/test/sql/prepared.result
> index 2f4983b00..9951a4e43 100644
> --- a/test/sql/prepared.result
> +++ b/test/sql/prepared.result
> @@ -64,6 +64,21 @@ test_run:cmd("setopt delimiter ''");
>   | - true
>   | ...
>  
> +-- Check default cache statistics.
> +--
> +box.info.sql()
> + | ---
> + | - cache:
> + |     size: 0
> + |     stmt_count: 0
> + | ...
> +box.info:sql()
> + | ---
> + | - cache:
> + |     size: 0
> + |     stmt_count: 0
> + | ...
> +
>  -- Test local interface and basic capabilities of prepared statements.
>  --
>  execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> @@ -144,6 +159,15 @@ execute(s.stmt_id, {1, 3})
>   |   rows: []
>   | ...
>  
> +assert(box.info.sql().cache.stmt_count ~= 0)
> + | ---
> + | - true
> + | ...
> +assert(box.info.sql().cache.size ~= 0)
> + | ---
> + | - true
> + | ...
> +
>  test_run:cmd("setopt delimiter ';'")
>   | ---
>   | - true
> @@ -584,8 +608,16 @@ unprepare(s1.stmt_id)
>  -- Setting cache size to 0 is possible only in case if
>  -- there's no any prepared statements right now .
>  --
> -box.cfg{sql_cache_size = 0}
> +box.cfg{sql_cache_size = 0 }
> + | ---
> + | ...
> +assert(box.info.sql().cache.stmt_count == 0)
>   | ---
> + | - true
> + | ...
> +assert(box.info.sql().cache.size == 0)
> + | ---
> + | - true
>   | ...
>  prepare("SELECT a FROM test;")
>   | ---
> diff --git a/test/sql/prepared.test.lua b/test/sql/prepared.test.lua
> index c464cc21a..5820525d1 100644
> --- a/test/sql/prepared.test.lua
> +++ b/test/sql/prepared.test.lua
> @@ -43,6 +43,11 @@ end;
>  
>  test_run:cmd("setopt delimiter ''");
>  
> +-- Check default cache statistics.
> +--
> +box.info.sql()
> +box.info:sql()
> +
>  -- Test local interface and basic capabilities of prepared statements.
>  --
>  execute('CREATE TABLE test (id INT PRIMARY KEY, a NUMBER, b TEXT)')
> @@ -60,6 +65,9 @@ s.param_count
>  execute(s.stmt_id, {1, 2})
>  execute(s.stmt_id, {1, 3})
>  
> +assert(box.info.sql().cache.stmt_count ~= 0)
> +assert(box.info.sql().cache.size ~= 0)
> +
>  test_run:cmd("setopt delimiter ';'")
>  if not is_remote then
>      res = s:execute({1, 2})
> @@ -207,7 +215,9 @@ unprepare(s1.stmt_id)
>  -- Setting cache size to 0 is possible only in case if
>  -- there's no any prepared statements right now .
>  --
> -box.cfg{sql_cache_size = 0}
> +box.cfg{sql_cache_size = 0 }
> +assert(box.info.sql().cache.stmt_count == 0)
> +assert(box.info.sql().cache.size == 0)
>  prepare("SELECT a FROM test;")
>  box.cfg{sql_cache_size = 0}
>  
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 01/20] sql: remove sql_prepare_v2()

[Sergey Ostanevich]

Thanks,

LGTM.

Sergos

On 24 Dec 03:51, Nikita Pettik wrote:
> On 23 Dec 17:03, Sergey Ostanevich wrote:
> > Hi!
> > 
> > Thanks for the patch!
> > 
> > On 20 Dec 15:47, Nikita Pettik wrote:
> > >  			continue;
> > > diff --git a/src/box/sql/prepare.c b/src/box/sql/prepare.c
> > > index 0ecc676e2..35e81212d 100644
> > > --- a/src/box/sql/prepare.c
> > > +++ b/src/box/sql/prepare.c
> > > @@ -204,36 +204,12 @@ sqlReprepare(Vdbe * p)
> > >  	return 0;
> > >  }
> > >  
> > > -/*
> > > - * Two versions of the official API.  Legacy and new use.  In the legacy
> > > - * version, the original SQL text is not saved in the prepared statement
> > > - * and so if a schema change occurs, an error is returned by
> > > - * sql_step().  In the new version, the original SQL text is retained
> > > - * and the statement is automatically recompiled if an schema change
> > > - * occurs.
> > > - */
> > > -int
> > > -sql_prepare(sql * db,		/* Database handle. */
> > > -		const char *zSql,	/* UTF-8 encoded SQL statement. */
> > > -		int nBytes,		/* Length of zSql in bytes. */
> > > -		sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> > > -		const char **pzTail)	/* OUT: End of parsed string */
> > > -{
> > > -	int rc = sqlPrepare(db, zSql, nBytes, 0, 0, ppStmt, pzTail);
> > > -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> > > -	return rc;
> > > -}
> > > -
> > >  int
> > > -sql_prepare_v2(sql * db,	/* Database handle. */
> > > -		   const char *zSql,	/* UTF-8 encoded SQL statement. */
> > > -		   int nBytes,	/* Length of zSql in bytes. */
> > > -		   sql_stmt ** ppStmt,	/* OUT: A pointer to the prepared statement */
> > > -		   const char **pzTail	/* OUT: End of parsed string */
> > > -    )
> > > +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> > > +	    const char **sql_tail)
> > >  {
> > > -	int rc = sqlPrepare(db, zSql, nBytes, 1, 0, ppStmt, pzTail);
> > > -	assert(rc == 0 || ppStmt == NULL || *ppStmt == NULL);	/* VERIFY: F13021 */
> > > +	int rc = sqlPrepare(db, sql, length, 1, 0, stmt, sql_tail);
> > > +	assert(rc == 0 || stmt == NULL || *stmt == NULL);
> > >  	return rc;
> > >  }
> > >  
> > > diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h
> > > index 2594b73e0..7bd952a17 100644
> > > --- a/src/box/sql/sqlInt.h
> > > +++ b/src/box/sql/sqlInt.h
> > > @@ -468,21 +468,18 @@ typedef void (*sql_destructor_type) (void *);
> > >  #define SQL_STATIC      ((sql_destructor_type)0)
> > >  #define SQL_TRANSIENT   ((sql_destructor_type)-1)
> > >  
> > > +/**
> > > + * Prepare (compile into VDBE byte-code) statement.
> > 
> > Could you please extend the description with details on SQL text
> > preservance and recmopilation, same as it was in old version?
> 
> It was obsolete and non-relevant (related purely to SQLite) comment.
> It has nothing to do with current sql_prepare() behaviour.
>  
> > > + *
> > > + * @param db Database handle.
> > > + * @param sql UTF-8 encoded SQL statement.
> > > + * @param length Length of @param sql in bytes.
> > > + * @param[out] stmt A pointer to the prepared statement.
> > > + * @param[out] sql_tail End of parsed string.
> > > + */
> > >  int
> > > -sql_prepare(sql * db,	/* Database handle */
> > > -		const char *zSql,	/* SQL statement, UTF-8 encoded */
> > > -		int nByte,	/* Maximum length of zSql in bytes. */
> > > -		sql_stmt ** ppStmt,	/* OUT: Statement handle */
> > > -		const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> > > -	);
> > > -
> > > -int
> > > -sql_prepare_v2(sql * db,	/* Database handle */
> > > -		   const char *zSql,	/* SQL statement, UTF-8 encoded */
> > > -		   int nByte,	/* Maximum length of zSql in bytes. */
> > > -		   sql_stmt ** ppStmt,	/* OUT: Statement handle */
> > > -		   const char **pzTail	/* OUT: Pointer to unused portion of zSql */
> > > -	);
> > > +sql_prepare(struct sql *db, const char *sql, int length, struct sql_stmt **stmt,
> > > +	    const char **sql_tail);
> > >  
> > >  int
> > >  sql_step(sql_stmt *);
> > > diff --git a/src/box/sql/vdbeapi.c b/src/box/sql/vdbeapi.c
> > > index 685212d91..12449d3bc 100644
> > > --- a/src/box/sql/vdbeapi.c
> > > +++ b/src/box/sql/vdbeapi.c
> > > @@ -452,7 +452,7 @@ sqlStep(Vdbe * p)
> > >  		checkProfileCallback(db, p);
> > >  
> > >  	if (p->isPrepareV2 && rc != SQL_ROW && rc != SQL_DONE) {
> > > -		/* If this statement was prepared using sql_prepare_v2(), and an
> > > +		/* If this statement was prepared using sql_prepare(), and an
> > >  		 * error has occurred, then return an error.
> > >  		 */
> > >  		if (p->is_aborted)
> > > -- 
> > > 2.15.1
> > > 

Re: [Tarantool-patches] [PATCH v3 00/20] sql: prepared statements

[Nikita Pettik]

On 20 Dec 15:47, Nikita Pettik wrote:

I've pushed all preliminary patches to master (I've fixed Sergos's
nits for current version and Vlad's and Konstantin's comments for
previous patch version).

> Branch: https://github.com/tarantool/tarantool/tree/np/gh-2592-prepared-statements-v3
> Issue: https://github.com/tarantool/tarantool/issues/2592
> 
> V1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012274.html
> V2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012496.html
> 
> Changes in V3 (requested by server team):
> 
>  - Now there's no eviction policy, so statements reside in 'cache'
>    until explicit deallocation via 'unprepare' call or session's
>    disconect;
>  - instead of string ids, now we use numeric ids which correspond
>    to value of hash function applied to the string containing original
>    SQL request;
>  - in accordance with previous point, 'unprepare' support has been returned;
>  - since there's no eviction policy, disconnect event may turn out to
>    be expensive (in terms of deallocating all related to the session
>    prepared statements). To remove possible spikes in workload, we
>    maintain GC queue and reference counters for prepared statements.
>    When all sessions (previously refed statement) unref it, statement
>    gets into GC queue. In case of prepared statements holder is out
>    of memory, GC process is launched: all statements in queue are
>    deallocated;
>  - to track available in scope of session prepared statements, we
>    also maintain session-local map containing statement's IDs
>    allocated in this session.
> 
> Nikita Pettik (20):
>   sql: remove sql_prepare_v2()
>   sql: refactor sql_prepare() and sqlPrepare()
>   sql: move sql_prepare() declaration to box/execute.h
>   sql: rename sqlPrepare() to sql_stmt_compile()
>   sql: rename sql_finalize() to sql_stmt_finalize()
>   sql: rename sql_reset() to sql_stmt_reset()
>   sql: move sql_stmt_finalize() to execute.h
>   port: increase padding of struct port
>   port: add result set format and request type to port_sql
>   sql: resurrect sql_bind_parameter_count() function
>   sql: resurrect sql_bind_parameter_name()
>   sql: add sql_stmt_schema_version()
>   sql: introduce sql_stmt_sizeof() function
>   box: increment schema_version on ddl operations
>   sql: introduce sql_stmt_query_str() method
>   sql: move sql_stmt_busy() declaration to box/execute.h
>   sql: introduce holder for prepared statemets
>   box: introduce prepared statements
>   netbox: introduce prepared statements
>   sql: add cache statistics to box.info
> 
>  src/box/CMakeLists.txt          |   1 +
>  src/box/alter.cc                |   3 +
>  src/box/bind.c                  |   2 +-
>  src/box/box.cc                  |  26 ++
>  src/box/box.h                   |   3 +
>  src/box/ck_constraint.c         |   7 +-
>  src/box/errcode.h               |   2 +
>  src/box/execute.c               | 234 ++++++++++++-
>  src/box/execute.h               |  60 ++++
>  src/box/iproto.cc               |  68 +++-
>  src/box/iproto_constants.c      |   7 +-
>  src/box/iproto_constants.h      |   5 +
>  src/box/lua/cfg.cc              |   9 +
>  src/box/lua/execute.c           | 235 ++++++++++++-
>  src/box/lua/execute.h           |   2 +-
>  src/box/lua/info.c              |  25 ++
>  src/box/lua/init.c              |   2 +-
>  src/box/lua/load_cfg.lua        |   3 +
>  src/box/lua/net_box.c           |  98 +++++-
>  src/box/lua/net_box.lua         |  27 ++
>  src/box/session.cc              |  35 ++
>  src/box/session.h               |  17 +
>  src/box/sql.c                   |   3 +
>  src/box/sql/analyze.c           |  23 +-
>  src/box/sql/legacy.c            |   3 +-
>  src/box/sql/prepare.c           |  56 +--
>  src/box/sql/sqlInt.h            |  51 +--
>  src/box/sql/vdbe.c              |   4 +-
>  src/box/sql/vdbe.h              |   2 +-
>  src/box/sql/vdbeInt.h           |   1 -
>  src/box/sql/vdbeapi.c           | 113 ++++--
>  src/box/sql/vdbeaux.c           |   6 +-
>  src/box/sql_stmt_cache.c        | 305 +++++++++++++++++
>  src/box/sql_stmt_cache.h        | 153 +++++++++
>  src/box/xrow.c                  |  23 +-
>  src/box/xrow.h                  |   4 +-
>  src/lib/core/port.h             |   2 +-
>  test/app-tap/init_script.result |  37 +-
>  test/box/admin.result           |   2 +
>  test/box/cfg.result             |   7 +
>  test/box/cfg.test.lua           |   1 +
>  test/box/info.result            |   1 +
>  test/box/misc.result            |   5 +
>  test/sql/engine.cfg             |   4 +
>  test/sql/iproto.result          |   2 +-
>  test/sql/prepared.result        | 737 ++++++++++++++++++++++++++++++++++++++++
>  test/sql/prepared.test.lua      | 282 +++++++++++++++
>  47 files changed, 2503 insertions(+), 195 deletions(-)
>  create mode 100644 src/box/sql_stmt_cache.c
>  create mode 100644 src/box/sql_stmt_cache.h
>  create mode 100644 test/sql/prepared.result
>  create mode 100644 test/sql/prepared.test.lua
> 
> -- 
> 2.15.1
> 

Re: [Tarantool-patches] [PATCH v3 20/20] sql: add cache statistics to box.info

[Nikita Pettik]

On 25 Dec 23:53, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the whole patchset!
> I got one major question: in case session is ended - there should be a
> refcnt reduction for all prepared stmts of this session. I didn't see
> that in the code.

Look at session_destroy() which calls sql_session_stmt_hash_erase().
 
> Regards,
> Sergos
> 
> On 20 Dec 15:47, Nikita Pettik wrote:
> > To track current memory occupied by prepared statements and number of
> > them, let's extend box.info submodule with .sql statistics: now it
> > contains current total size of prepared statements and their count.
> > 
> > @TarantoolBot document
> > Title: Prepared statements in SQL
> > 
> > Now it is possible to prepare (i.e. compile into byte-code and save to
> > the cache) statement and execute it several times. Mechanism is similar
> > to ones in other DBs. Prepared statement is identified by numeric
> > ID, which are returned alongside with prepared statement handle.
> > Note that they are not sequential and represent value of hash function
> > applied to the string containing original SQL request.
> > Prepared statement holder is shared among all sessions. However, session
> > has access only to statements which have been prepared in scope of it.
> > There's no eviction policy like in any cache; to remove statement from
> > holder explicit unprepare request is required. Alternatively, session's
> > disconnect also removes statements from holder.
> > Several sessions can share one prepared statement, which will be
> > destroyed when all related sessions are disconnected or send unprepare
> > request. Memory limit for prepared statements is adjusted by
> > box.cfg{sql_cache_size} handle (can be set dynamically;
> > 
> > Any DDL operation leads to expiration of all prepared statements: they
> > should be manually removed or re-prepared.
> 
> Does it mean that for user code there should be called unprepare(s.id) and
> s = prepare(orginal sql text)? If I got it right from previous patches
> the re-prepare is called automatically?

In current versions there's no auto re-prepare, user should call
prepare again manually.

> 
> > Prepared statements are available in local mode (i.e. via box.prepare()
> > function) and are supported in IProto protocol. In the latter case
> > next IProto keys are used to make up/receive requests/responses:
> > IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
> > one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
> > or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
> > means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
> > request, meanwhile IPROTO_STMT_ID - unprepare;
> > IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
> > and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
> > be bound) are response keys. They are mandatory members of result of
> > IPROTO_PREPARE execution.
> > 
> > To track statistics of used memory and number of currently prepared
> > statements, box.info is extended with SQL statistics:
> > 
> > box.info:sql().cache.stmt_count - number of prepared statements;
> > box.info:sql().cache.size - size of occupied by prepared statements memory.
> > 
> > Typical workflow with prepared statements is following:
> > 
> > s = box.prepare("SELECT * FROM t WHERE id = ?;")
> > s:execute({1}) or box.execute(s.sql_str, {1})
> > s:execute({2}) or box.execute(s.sql_str, {2})
> > s:unprepare() or box.unprepare(s.query_id)
> > 
> > Structure of object is following (member : type):
> > 
> > - stmt_id: integer
> >   execute: function
> >   params: map [name : string, type : integer]
> >   unprepare: function
> >   metadata: map [name : string, type : integer]
> >   param_count: integer
> > ...
> > 
> > In terms of remote connection:
> > 
> > cn = netbox:connect(addr)
> > s = cn:prepare("SELECT * FROM t WHERE id = ?;")
> > cn:execute(s.sql_str, {1})
> > cn:unprepare(s.query_id)
> > 
> > Closese #2592
> 
> Misprint above: Closes

Thx, fixed.

Re: [Tarantool-patches] [PATCH v3 19/20] netbox: introduce prepared statements

[Nikita Pettik]

On 25 Dec 23:41, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the patch, one nit and one question.
> 
> Regards,
> Sergos
> 
> 
> 
> On 20 Dec 15:47, Nikita Pettik wrote:
> > @@ -1738,12 +1781,15 @@ tx_process_sql(struct cmsg *m)
> >  		port_destroy(&port);
> >  		goto error;
> >  	}
> > -	if (port_dump_msgpack(&port, out) != 0) {
> > +	/* Nothing to dump in case of UNPREPARE request. */
> > +	if (! is_unprepare) {
> 
> Unary ops - no spaces.

Ok, fixed.
 
> > +		if (port_dump_msgpack(&port, out) != 0) {
> > +			port_destroy(&port);
> > +			obuf_rollback_to_svp(out, &header_svp);
> > +			goto error;
> > +		}
> >  		port_destroy(&port);
> > -		obuf_rollback_to_svp(out, &header_svp);
> > -		goto error;
> >  	}
> > -	port_destroy(&port);
> >  	iproto_reply_sql(out, &header_svp, msg->header.sync, schema_version);
> >  	iproto_wpos_create(&msg->wpos, out);
> >  	return;
> > diff --git a/src/box/lua/net_box.lua b/src/box/lua/net_box.lua
> > index c2e1bb9c4..b4811edfa 100644
> > --- a/src/box/lua/net_box.lua
> > +++ b/src/box/lua/net_box.lua
> > @@ -104,6 +104,8 @@ local method_encoder = {
> >      upsert  = internal.encode_upsert,
> >      select  = internal.encode_select,
> >      execute = internal.encode_execute,
> > +    prepare = internal.encode_prepare,
> > +    unprepare = internal.encode_prepare,
> >      get     = internal.encode_select,
> >      min     = internal.encode_select,
> >      max     = internal.encode_select,
> > @@ -128,6 +130,8 @@ local method_decoder = {
> >      upsert  = decode_nil,
> >      select  = internal.decode_select,
> >      execute = internal.decode_execute,
> > +    prepare = internal.decode_prepare,
> > +    unprepare = decode_nil,

No. Result of prepare contains metadata (like names, types etc),
meanwhile result of unprepare contains only response status,
so there's nothing to decode.
 
> should it be internal.decode_prepare?
> 
> >      get     = decode_get,
> >      min     = decode_get,
> >      max     = decode_get,

Re: [Tarantool-patches] [PATCH v3 18/20] box: introduce prepared statements

[Nikita Pettik]

On 25 Dec 18:23, Sergey Ostanevich wrote:
> Hi!
> 
> Thanks for the patch, LGTM with just 2 nits below. 
> 
> Sergos
> 
> On 20 Dec 15:47, Nikita Pettik wrote:
> > This patch introduces local prepared statements. Support of prepared
> > statements in IProto protocol and netbox is added in the next patch.
> > 
> > Prepared statement is an opaque instance of SQL Virtual Machine. It can
> > be executed several times without necessity of query recompilation. To
> > achieve this one can use box.prepare(...) function. It takes string of
> > SQL query to be prepared; returns extended set of meta-information
> > including statement's ID, parameter's types and names, types and names
> > of columns of the resulting set, count of parameters to be bound.  Lua
> > object representing result of :prepare() invocation also features two
> > methods - :execute() and :unprepare(). They correspond to
> > box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
> > automatically substitute string of prepared statement to be executed.
> > Statements are held in prepared statement cache - for details see
> > previous commit.  After schema changes all prepared statement located in
> > cache are considered to be expired - they must be re-prepared by
> > separate :prepare() call (or be invalidated with :unrepare()).
> > 
> > Two sessions can share one prepared statements. But in current
> > implementation if statement is executed by one session, another one is
> > not able to use it and will compile it from scratch and than execute.
> 
> It would be nice to mention plans on what should/will be done for
> resolution of this.

I'm going to file an issue to provide optimization for this.
I suppose we can copy VM before executing it. But now VM
consists of several parts which are allocated/deallocated as
separate chunks. So to achieve fast copy and resources finalization
for VM it would be nice to copy them into one chunk before execution.

> Also, my previous question on DDL during the
> execution of the statement is valid - one session can ruin execution
> of a prepared statement in another session with a DDL. Should there 
> be some guard for DDL until all executions of prep statements are finished?

I don't think it is good idea. Firstly, it would lock DB for any DDL
until there's at least one statemenet under execution (taking into
account that Tarantool as a DB is assumed to be under load all the time
it means in fact that DDL would not be allowed at all). Secondly, it is
documented behaviour, so users should be aware of it.

> > SQL cache memory limit is regulated by box{sql_cache_size} which can be
> > set dynamically. However, it can be set to the value which is less than
> > the size of current free space in cache (since otherwise some statements
> > can disappear from cache).
> > 
> > Part of #2592
> > ---
> >  src/box/errcode.h          |   1 +
> >  src/box/execute.c          | 114 ++++++++
> >  src/box/execute.h          |  16 +-
> >  src/box/lua/execute.c      | 213 +++++++++++++-
> >  src/box/lua/execute.h      |   2 +-
> >  src/box/lua/init.c         |   2 +-
> >  src/box/sql/prepare.c      |   9 -
> >  test/box/misc.result       |   3 +
> >  test/sql/engine.cfg        |   3 +
> >  test/sql/prepared.result   | 687 +++++++++++++++++++++++++++++++++++++++++++++
> >  test/sql/prepared.test.lua | 240 ++++++++++++++++
> >  11 files changed, 1267 insertions(+), 23 deletions(-)
> >  create mode 100644 test/sql/prepared.result
> >  create mode 100644 test/sql/prepared.test.lua
> > 
> > diff --git a/src/box/errcode.h b/src/box/errcode.h
> > index ee44f61b3..9e12f3a31 100644
> > --- a/src/box/errcode.h
> > +++ b/src/box/errcode.h
> > @@ -259,6 +259,7 @@ struct errcode_record {
> >  	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
> >  	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
> >  	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
> > +	/*207 */_(ER_WRONG_QUERY_ID,		"Prepared statement with id %u does not exist") \
> >  
> >  /*
> >   * !IMPORTANT! Please follow instructions at start of the file
> > diff --git a/src/box/execute.c b/src/box/execute.c
> > index 3bc4988b7..09224c23a 100644
> > --- a/src/box/execute.c
> > +++ b/src/box/execute.c
> > @@ -30,6 +30,7 @@
> >   */
> >  #include "execute.h"
> >  
> > +#include "assoc.h"
> >  #include "bind.h"
> >  #include "iproto_constants.h"
> >  #include "sql/sqlInt.h"
> > @@ -45,6 +46,8 @@
> >  #include "tuple.h"
> >  #include "sql/vdbe.h"
> >  #include "box/lua/execute.h"
> > +#include "box/sql_stmt_cache.h"
> > +#include "session.h"
> >  
> >  const char *sql_info_key_strs[] = {
> >  	"row_count",
> > @@ -413,6 +416,81 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
> >  	return 0;
> >  }
> >  
> > +static bool
> > +sql_stmt_check_schema_version(struct sql_stmt *stmt)
> 
> The naming could be better - since you state something as true or false,
> it should be definitive, like sql_stmt_schema_version_is_valid()

Ok, let's rename it to sql_stmt_schema_version_is_valid()

> > +{
> > +	return sql_stmt_schema_version(stmt) == box_schema_version();
> > +}
> > +
> > +int
> > +sql_prepare(const char *sql, int len, struct port *port)
> > +{
> > +	uint32_t stmt_id = sql_stmt_calculate_id(sql, len);
> > +	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
> > +	if (stmt == NULL) {
> > +		if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
> > +			return -1;
> > +		if (sql_stmt_cache_insert(stmt) != 0) {
> > +			sql_stmt_finalize(stmt);
> > +			return -1;
> > +		}
> > +	} else {
> > +		if (! sql_stmt_check_schema_version(stmt)) {
> 
> The unaries should not be space-delimited as per C style $3.1
> https://www.tarantool.io/en/doc/2.2/dev_guide/c_style_guide/

Fixed.
 

Re: [Tarantool-patches] [PATCH v3 18/20] box: introduce prepared statements

[Sergey Ostanevich]

Hi!

Thanks for the answers, LGTM.

Sergos

On 30 Dec 12:27, Nikita Pettik wrote:
> On 25 Dec 18:23, Sergey Ostanevich wrote:
> > Hi!
> > 
> > Thanks for the patch, LGTM with just 2 nits below. 
> > 
> > Sergos
> > 
> > On 20 Dec 15:47, Nikita Pettik wrote:
> > > This patch introduces local prepared statements. Support of prepared
> > > statements in IProto protocol and netbox is added in the next patch.
> > > 
> > > Prepared statement is an opaque instance of SQL Virtual Machine. It can
> > > be executed several times without necessity of query recompilation. To
> > > achieve this one can use box.prepare(...) function. It takes string of
> > > SQL query to be prepared; returns extended set of meta-information
> > > including statement's ID, parameter's types and names, types and names
> > > of columns of the resulting set, count of parameters to be bound.  Lua
> > > object representing result of :prepare() invocation also features two
> > > methods - :execute() and :unprepare(). They correspond to
> > > box.execute(stmt.stmt_id) and box.unprepare(stmt.stmt_id), i.e.
> > > automatically substitute string of prepared statement to be executed.
> > > Statements are held in prepared statement cache - for details see
> > > previous commit.  After schema changes all prepared statement located in
> > > cache are considered to be expired - they must be re-prepared by
> > > separate :prepare() call (or be invalidated with :unrepare()).
> > > 
> > > Two sessions can share one prepared statements. But in current
> > > implementation if statement is executed by one session, another one is
> > > not able to use it and will compile it from scratch and than execute.
> > 
> > It would be nice to mention plans on what should/will be done for
> > resolution of this.
> 
> I'm going to file an issue to provide optimization for this.
> I suppose we can copy VM before executing it. But now VM
> consists of several parts which are allocated/deallocated as
> separate chunks. So to achieve fast copy and resources finalization
> for VM it would be nice to copy them into one chunk before execution.
> 
> > Also, my previous question on DDL during the
> > execution of the statement is valid - one session can ruin execution
> > of a prepared statement in another session with a DDL. Should there 
> > be some guard for DDL until all executions of prep statements are finished?
> 
> I don't think it is good idea. Firstly, it would lock DB for any DDL
> until there's at least one statemenet under execution (taking into
> account that Tarantool as a DB is assumed to be under load all the time
> it means in fact that DDL would not be allowed at all). Secondly, it is
> documented behaviour, so users should be aware of it.
> 
> > > SQL cache memory limit is regulated by box{sql_cache_size} which can be
> > > set dynamically. However, it can be set to the value which is less than
> > > the size of current free space in cache (since otherwise some statements
> > > can disappear from cache).
> > > 
> > > Part of #2592
> > > ---
> > >  src/box/errcode.h          |   1 +
> > >  src/box/execute.c          | 114 ++++++++
> > >  src/box/execute.h          |  16 +-
> > >  src/box/lua/execute.c      | 213 +++++++++++++-
> > >  src/box/lua/execute.h      |   2 +-
> > >  src/box/lua/init.c         |   2 +-
> > >  src/box/sql/prepare.c      |   9 -
> > >  test/box/misc.result       |   3 +
> > >  test/sql/engine.cfg        |   3 +
> > >  test/sql/prepared.result   | 687 +++++++++++++++++++++++++++++++++++++++++++++
> > >  test/sql/prepared.test.lua | 240 ++++++++++++++++
> > >  11 files changed, 1267 insertions(+), 23 deletions(-)
> > >  create mode 100644 test/sql/prepared.result
> > >  create mode 100644 test/sql/prepared.test.lua
> > > 
> > > diff --git a/src/box/errcode.h b/src/box/errcode.h
> > > index ee44f61b3..9e12f3a31 100644
> > > --- a/src/box/errcode.h
> > > +++ b/src/box/errcode.h
> > > @@ -259,6 +259,7 @@ struct errcode_record {
> > >  	/*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT,	"SQL expects exactly one argument returned from %s, got %d")\
> > >  	/*205 */_(ER_FUNC_INVALID_RETURN_TYPE,	"Function '%s' returned value of invalid type: expected %s got %s") \
> > >  	/*206 */_(ER_SQL_PREPARE,		"Failed to prepare SQL statement: %s") \
> > > +	/*207 */_(ER_WRONG_QUERY_ID,		"Prepared statement with id %u does not exist") \
> > >  
> > >  /*
> > >   * !IMPORTANT! Please follow instructions at start of the file
> > > diff --git a/src/box/execute.c b/src/box/execute.c
> > > index 3bc4988b7..09224c23a 100644
> > > --- a/src/box/execute.c
> > > +++ b/src/box/execute.c
> > > @@ -30,6 +30,7 @@
> > >   */
> > >  #include "execute.h"
> > >  
> > > +#include "assoc.h"
> > >  #include "bind.h"
> > >  #include "iproto_constants.h"
> > >  #include "sql/sqlInt.h"
> > > @@ -45,6 +46,8 @@
> > >  #include "tuple.h"
> > >  #include "sql/vdbe.h"
> > >  #include "box/lua/execute.h"
> > > +#include "box/sql_stmt_cache.h"
> > > +#include "session.h"
> > >  
> > >  const char *sql_info_key_strs[] = {
> > >  	"row_count",
> > > @@ -413,6 +416,81 @@ port_sql_dump_msgpack(struct port *port, struct obuf *out)
> > >  	return 0;
> > >  }
> > >  
> > > +static bool
> > > +sql_stmt_check_schema_version(struct sql_stmt *stmt)
> > 
> > The naming could be better - since you state something as true or false,
> > it should be definitive, like sql_stmt_schema_version_is_valid()
> 
> Ok, let's rename it to sql_stmt_schema_version_is_valid()
> 
> > > +{
> > > +	return sql_stmt_schema_version(stmt) == box_schema_version();
> > > +}
> > > +
> > > +int
> > > +sql_prepare(const char *sql, int len, struct port *port)
> > > +{
> > > +	uint32_t stmt_id = sql_stmt_calculate_id(sql, len);
> > > +	struct sql_stmt *stmt = sql_stmt_cache_find(stmt_id);
> > > +	if (stmt == NULL) {
> > > +		if (sql_stmt_compile(sql, len, NULL, &stmt, NULL) != 0)
> > > +			return -1;
> > > +		if (sql_stmt_cache_insert(stmt) != 0) {
> > > +			sql_stmt_finalize(stmt);
> > > +			return -1;
> > > +		}
> > > +	} else {
> > > +		if (! sql_stmt_check_schema_version(stmt)) {
> > 
> > The unaries should not be space-delimited as per C style $3.1
> > https://www.tarantool.io/en/doc/2.2/dev_guide/c_style_guide/
> 
> Fixed.
>  

Re: [Tarantool-patches] [PATCH v3 19/20] netbox: introduce prepared statements

[Sergey Ostanevich]

Hi!

Thanks, LGTM.


Sergos

On 30 Dec 11:58, Nikita Pettik wrote:
> On 25 Dec 23:41, Sergey Ostanevich wrote:
> > Hi!
> > 
> > Thanks for the patch, one nit and one question.
> > 
> > Regards,
> > Sergos
> > 
> > 
> > 
> > On 20 Dec 15:47, Nikita Pettik wrote:
> > > @@ -1738,12 +1781,15 @@ tx_process_sql(struct cmsg *m)
> > >  		port_destroy(&port);
> > >  		goto error;
> > >  	}
> > > -	if (port_dump_msgpack(&port, out) != 0) {
> > > +	/* Nothing to dump in case of UNPREPARE request. */
> > > +	if (! is_unprepare) {
> > 
> > Unary ops - no spaces.
> 
> Ok, fixed.
>  
> > > +		if (port_dump_msgpack(&port, out) != 0) {
> > > +			port_destroy(&port);
> > > +			obuf_rollback_to_svp(out, &header_svp);
> > > +			goto error;
> > > +		}
> > >  		port_destroy(&port);
> > > -		obuf_rollback_to_svp(out, &header_svp);
> > > -		goto error;
> > >  	}
> > > -	port_destroy(&port);
> > >  	iproto_reply_sql(out, &header_svp, msg->header.sync, schema_version);
> > >  	iproto_wpos_create(&msg->wpos, out);
> > >  	return;
> > > diff --git a/src/box/lua/net_box.lua b/src/box/lua/net_box.lua
> > > index c2e1bb9c4..b4811edfa 100644
> > > --- a/src/box/lua/net_box.lua
> > > +++ b/src/box/lua/net_box.lua
> > > @@ -104,6 +104,8 @@ local method_encoder = {
> > >      upsert  = internal.encode_upsert,
> > >      select  = internal.encode_select,
> > >      execute = internal.encode_execute,
> > > +    prepare = internal.encode_prepare,
> > > +    unprepare = internal.encode_prepare,
> > >      get     = internal.encode_select,
> > >      min     = internal.encode_select,
> > >      max     = internal.encode_select,
> > > @@ -128,6 +130,8 @@ local method_decoder = {
> > >      upsert  = decode_nil,
> > >      select  = internal.decode_select,
> > >      execute = internal.decode_execute,
> > > +    prepare = internal.decode_prepare,
> > > +    unprepare = decode_nil,
> 
> No. Result of prepare contains metadata (like names, types etc),
> meanwhile result of unprepare contains only response status,
> so there's nothing to decode.
>  
> > should it be internal.decode_prepare?
> > 
> > >      get     = decode_get,
> > >      min     = decode_get,
> > >      max     = decode_get,

Re: [Tarantool-patches] [PATCH v3 20/20] sql: add cache statistics to box.info

[Sergey Ostanevich]

Hi!

Got it with stmts removal. LGTM.

Sergos


On 30 Dec 11:46, Nikita Pettik wrote:
> On 25 Dec 23:53, Sergey Ostanevich wrote:
> > Hi!
> > 
> > Thanks for the whole patchset!
> > I got one major question: in case session is ended - there should be a
> > refcnt reduction for all prepared stmts of this session. I didn't see
> > that in the code.
> 
> Look at session_destroy() which calls sql_session_stmt_hash_erase().
>  
> > Regards,
> > Sergos
> > 
> > On 20 Dec 15:47, Nikita Pettik wrote:
> > > To track current memory occupied by prepared statements and number of
> > > them, let's extend box.info submodule with .sql statistics: now it
> > > contains current total size of prepared statements and their count.
> > > 
> > > @TarantoolBot document
> > > Title: Prepared statements in SQL
> > > 
> > > Now it is possible to prepare (i.e. compile into byte-code and save to
> > > the cache) statement and execute it several times. Mechanism is similar
> > > to ones in other DBs. Prepared statement is identified by numeric
> > > ID, which are returned alongside with prepared statement handle.
> > > Note that they are not sequential and represent value of hash function
> > > applied to the string containing original SQL request.
> > > Prepared statement holder is shared among all sessions. However, session
> > > has access only to statements which have been prepared in scope of it.
> > > There's no eviction policy like in any cache; to remove statement from
> > > holder explicit unprepare request is required. Alternatively, session's
> > > disconnect also removes statements from holder.
> > > Several sessions can share one prepared statement, which will be
> > > destroyed when all related sessions are disconnected or send unprepare
> > > request. Memory limit for prepared statements is adjusted by
> > > box.cfg{sql_cache_size} handle (can be set dynamically;
> > > 
> > > Any DDL operation leads to expiration of all prepared statements: they
> > > should be manually removed or re-prepared.
> > 
> > Does it mean that for user code there should be called unprepare(s.id) and
> > s = prepare(orginal sql text)? If I got it right from previous patches
> > the re-prepare is called automatically?
> 
> In current versions there's no auto re-prepare, user should call
> prepare again manually.
> 
> > 
> > > Prepared statements are available in local mode (i.e. via box.prepare()
> > > function) and are supported in IProto protocol. In the latter case
> > > next IProto keys are used to make up/receive requests/responses:
> > > IPROTO_PREPARE - new IProto command; key is 0x13. It can be sent with
> > > one of two mandatory keys: IPROTO_SQL_TEXT (0x40 and assumes string value)
> > > or IPROTO_STMT_ID (0x43 and assumes integer value). Depending on body it
> > > means to prepare or unprepare SQL statement: IPROTO_SQL_TEXT implies prepare
> > > request, meanwhile IPROTO_STMT_ID - unprepare;
> > > IPROTO_BIND_METADATA (0x33 and contains parameters metadata of type map)
> > > and IPROTO_BIND_COUNT (0x34 and corresponds to the count of parameters to
> > > be bound) are response keys. They are mandatory members of result of
> > > IPROTO_PREPARE execution.
> > > 
> > > To track statistics of used memory and number of currently prepared
> > > statements, box.info is extended with SQL statistics:
> > > 
> > > box.info:sql().cache.stmt_count - number of prepared statements;
> > > box.info:sql().cache.size - size of occupied by prepared statements memory.
> > > 
> > > Typical workflow with prepared statements is following:
> > > 
> > > s = box.prepare("SELECT * FROM t WHERE id = ?;")
> > > s:execute({1}) or box.execute(s.sql_str, {1})
> > > s:execute({2}) or box.execute(s.sql_str, {2})
> > > s:unprepare() or box.unprepare(s.query_id)
> > > 
> > > Structure of object is following (member : type):
> > > 
> > > - stmt_id: integer
> > >   execute: function
> > >   params: map [name : string, type : integer]
> > >   unprepare: function
> > >   metadata: map [name : string, type : integer]
> > >   param_count: integer
> > > ...
> > > 
> > > In terms of remote connection:
> > > 
> > > cn = netbox:connect(addr)
> > > s = cn:prepare("SELECT * FROM t WHERE id = ?;")
> > > cn:execute(s.sql_str, {1})
> > > cn:unprepare(s.query_id)
> > > 
> > > Closese #2592
> > 
> > Misprint above: Closes
> 
> Thx, fixed.
> 

Re: [Tarantool-patches] [PATCH v3 00/20] sql: prepared statements

[Kirill Yukhin]

Hello,

On 20 дек 15:47, Nikita Pettik wrote:
> Branch: https://github.com/tarantool/tarantool/tree/np/gh-2592-prepared-statements-v3
> Issue: https://github.com/tarantool/tarantool/issues/2592
> 
> V1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012274.html
> V2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012496.html
> 
> Changes in V3 (requested by server team):
> 
>  - Now there's no eviction policy, so statements reside in 'cache'
>    until explicit deallocation via 'unprepare' call or session's
>    disconect;
>  - instead of string ids, now we use numeric ids which correspond
>    to value of hash function applied to the string containing original
>    SQL request;
>  - in accordance with previous point, 'unprepare' support has been returned;
>  - since there's no eviction policy, disconnect event may turn out to
>    be expensive (in terms of deallocating all related to the session
>    prepared statements). To remove possible spikes in workload, we
>    maintain GC queue and reference counters for prepared statements.
>    When all sessions (previously refed statement) unref it, statement
>    gets into GC queue. In case of prepared statements holder is out
>    of memory, GC process is launched: all statements in queue are
>    deallocated;
>  - to track available in scope of session prepared statements, we
>    also maintain session-local map containing statement's IDs
>    allocated in this session.
> 
> Nikita Pettik (20):
>   sql: remove sql_prepare_v2()
>   sql: refactor sql_prepare() and sqlPrepare()
>   sql: move sql_prepare() declaration to box/execute.h
>   sql: rename sqlPrepare() to sql_stmt_compile()
>   sql: rename sql_finalize() to sql_stmt_finalize()
>   sql: rename sql_reset() to sql_stmt_reset()
>   sql: move sql_stmt_finalize() to execute.h
>   port: increase padding of struct port
>   port: add result set format and request type to port_sql
>   sql: resurrect sql_bind_parameter_count() function
>   sql: resurrect sql_bind_parameter_name()
>   sql: add sql_stmt_schema_version()
>   sql: introduce sql_stmt_sizeof() function
>   box: increment schema_version on ddl operations
>   sql: introduce sql_stmt_query_str() method
>   sql: move sql_stmt_busy() declaration to box/execute.h
>   sql: introduce holder for prepared statemets
>   box: introduce prepared statements
>   netbox: introduce prepared statements
>   sql: add cache statistics to box.info

I've checked your patchset into master.

--
Regards, Kirill Yukhin