From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp43.i.mail.ru (smtp43.i.mail.ru [94.100.177.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id B8C1346970E for ; Mon, 23 Dec 2019 23:54:30 +0300 (MSK) Date: Mon, 23 Dec 2019 23:54:29 +0300 From: Sergey Ostanevich Message-ID: <20191223205429.GB19594@tarantool.org> References: <43abc201ff09881dabfd3afa3e003a2ede3035f4.1576844632.git.korablev@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <43abc201ff09881dabfd3afa3e003a2ede3035f4.1576844632.git.korablev@tarantool.org> Subject: Re: [Tarantool-patches] [PATCH v3 17/20] sql: introduce holder for prepared statemets List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Nikita Pettik Cc: tarantool-patches@dev.tarantool.org Hi! Thank you for the patch! Below are two questions I have. Thanks, Sergos On 20 Dec 15:47, Nikita Pettik wrote: > This patch introduces holder (as data structure) to handle prepared > statements and a set of interface functions (insert, delete, find) to > operate on it. Holder under the hood is implemented as a global hash > (keys are values of hash function applied to the original string containing > SQL query; values are pointer to wrappers around compiled VDBE objects) and > GC queue. Each entry in hash has reference counter. When it reaches 0 > value, entry is moved to GC queue. In case prepared statements holder is > out of memory, it launches GC process: each entry in GC queue is deleted > and all resources are released. Such approach allows to avoid workload > spikes on session's disconnect (since on such event all statements must > be deallocated). > Each session is extended with local hash to map statement ids available > for it. That is, session is allowed to execute and deallocate only > statements which are previously prepared in scope of this session. > On the other hand, global hash makes it possible to share same prepared > statement object among different sessions. > Size of cache is regulated via box.cfg{sql_cache_size} parameter. > > Part of #2592 > --- > src/box/CMakeLists.txt | 1 + > src/box/box.cc | 26 ++++ > src/box/box.h | 3 + > src/box/errcode.h | 1 + > src/box/lua/cfg.cc | 9 ++ > src/box/lua/load_cfg.lua | 3 + > src/box/session.cc | 35 +++++ > src/box/session.h | 17 +++ > src/box/sql.c | 3 + > src/box/sql_stmt_cache.c | 289 ++++++++++++++++++++++++++++++++++++++++ > src/box/sql_stmt_cache.h | 145 ++++++++++++++++++++ > test/app-tap/init_script.result | 37 ++--- > test/box/admin.result | 2 + > test/box/cfg.result | 7 + > test/box/cfg.test.lua | 1 + > test/box/misc.result | 1 + > 16 files changed, 562 insertions(+), 18 deletions(-) > create mode 100644 src/box/sql_stmt_cache.c > create mode 100644 src/box/sql_stmt_cache.h > > diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt > index 5cd5cba81..763bc3a4c 100644 > --- a/src/box/CMakeLists.txt > +++ b/src/box/CMakeLists.txt > @@ -126,6 +126,7 @@ add_library(box STATIC > sql.c > bind.c > execute.c > + sql_stmt_cache.c > wal.c > call.c > merger.c > diff --git a/src/box/box.cc b/src/box/box.cc > index b119c927b..2b0cfa32d 100644 > --- a/src/box/box.cc > +++ b/src/box/box.cc > @@ -74,6 +74,7 @@ > #include "call.h" > #include "func.h" > #include "sequence.h" > +#include "sql_stmt_cache.h" > > static char status[64] = "unknown"; > > @@ -599,6 +600,17 @@ box_check_vinyl_options(void) > } > } > > +static int > +box_check_sql_cache_size(int size) > +{ > + if (size < 0) { > + diag_set(ClientError, ER_CFG, "sql_cache_size", > + "must be non-negative"); > + return -1; > + } > + return 0; > +} > + > void > box_check_config() > { > @@ -620,6 +632,7 @@ box_check_config() > box_check_memtx_memory(cfg_geti64("memtx_memory")); > box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size")); > box_check_vinyl_options(); > + box_check_sql_cache_size(cfg_geti("sql_cache_size")); > } > > /* > @@ -886,6 +899,17 @@ box_set_net_msg_max(void) > IPROTO_FIBER_POOL_SIZE_FACTOR); > } > > +int > +box_set_prepared_stmt_cache_size(void) > +{ > + int cache_sz = cfg_geti("sql_cache_size"); > + if (box_check_sql_cache_size(cache_sz) != 0) > + return -1; > + if (sql_stmt_cache_set_size(cache_sz) != 0) > + return -1; > + return 0; > +} > + > /* }}} configuration bindings */ > > /** > @@ -2096,6 +2120,8 @@ box_cfg_xc(void) > box_check_instance_uuid(&instance_uuid); > box_check_replicaset_uuid(&replicaset_uuid); > > + if (box_set_prepared_stmt_cache_size() != 0) > + diag_raise(); > box_set_net_msg_max(); > box_set_readahead(); > box_set_too_long_threshold(); > diff --git a/src/box/box.h b/src/box/box.h > index ccd527bd5..6806f1fc2 100644 > --- a/src/box/box.h > +++ b/src/box/box.h > @@ -236,6 +236,9 @@ void box_set_replication_sync_timeout(void); > void box_set_replication_skip_conflict(void); > void box_set_net_msg_max(void); > > +int > +box_set_prepared_stmt_cache_size(void); > + > extern "C" { > #endif /* defined(__cplusplus) */ > > diff --git a/src/box/errcode.h b/src/box/errcode.h > index c660b1c70..ee44f61b3 100644 > --- a/src/box/errcode.h > +++ b/src/box/errcode.h > @@ -258,6 +258,7 @@ struct errcode_record { > /*203 */_(ER_BOOTSTRAP_READONLY, "Trying to bootstrap a local read-only instance as master") \ > /*204 */_(ER_SQL_FUNC_WRONG_RET_COUNT, "SQL expects exactly one argument returned from %s, got %d")\ > /*205 */_(ER_FUNC_INVALID_RETURN_TYPE, "Function '%s' returned value of invalid type: expected %s got %s") \ > + /*206 */_(ER_SQL_PREPARE, "Failed to prepare SQL statement: %s") \ > > /* > * !IMPORTANT! Please follow instructions at start of the file > diff --git a/src/box/lua/cfg.cc b/src/box/lua/cfg.cc > index 4884ce013..439af3cf9 100644 > --- a/src/box/lua/cfg.cc > +++ b/src/box/lua/cfg.cc > @@ -274,6 +274,14 @@ lbox_cfg_set_net_msg_max(struct lua_State *L) > return 0; > } > > +static int > +lbox_set_prepared_stmt_cache_size(struct lua_State *L) > +{ > + if (box_set_prepared_stmt_cache_size() != 0) > + luaT_error(L); > + return 0; > +} > + > static int > lbox_cfg_set_worker_pool_threads(struct lua_State *L) > { > @@ -378,6 +386,7 @@ box_lua_cfg_init(struct lua_State *L) > {"cfg_set_replication_sync_timeout", lbox_cfg_set_replication_sync_timeout}, > {"cfg_set_replication_skip_conflict", lbox_cfg_set_replication_skip_conflict}, > {"cfg_set_net_msg_max", lbox_cfg_set_net_msg_max}, > + {"cfg_set_sql_cache_size", lbox_set_prepared_stmt_cache_size}, > {NULL, NULL} > }; > > diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua > index 85617c8f0..4463f989c 100644 > --- a/src/box/lua/load_cfg.lua > +++ b/src/box/lua/load_cfg.lua > @@ -81,6 +81,7 @@ local default_cfg = { > feedback_host = "https://feedback.tarantool.io", > feedback_interval = 3600, > net_msg_max = 768, > + sql_cache_size = 5 * 1024 * 1024, > } > > -- types of available options > @@ -144,6 +145,7 @@ local template_cfg = { > feedback_host = 'string', > feedback_interval = 'number', > net_msg_max = 'number', > + sql_cache_size = 'number', > } > > local function normalize_uri(port) > @@ -250,6 +252,7 @@ local dynamic_cfg = { > instance_uuid = check_instance_uuid, > replicaset_uuid = check_replicaset_uuid, > net_msg_max = private.cfg_set_net_msg_max, > + sql_cache_size = private.cfg_set_sql_cache_size, > } > > -- > diff --git a/src/box/session.cc b/src/box/session.cc > index 461d1cf25..881318252 100644 > --- a/src/box/session.cc > +++ b/src/box/session.cc > @@ -36,6 +36,7 @@ > #include "user.h" > #include "error.h" > #include "tt_static.h" > +#include "sql_stmt_cache.h" > > const char *session_type_strs[] = { > "background", > @@ -141,6 +142,7 @@ session_create(enum session_type type) > session_set_type(session, type); > session->sql_flags = default_flags; > session->sql_default_engine = SQL_STORAGE_ENGINE_MEMTX; > + session->sql_stmts = NULL; > > /* For on_connect triggers. */ > credentials_create(&session->credentials, guest_user); > @@ -178,6 +180,38 @@ session_create_on_demand() > return s; > } > > +bool > +session_check_stmt_id(struct session *session, uint32_t stmt_id) > +{ > + if (session->sql_stmts == NULL) > + return false; > + mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL); > + return i != mh_end(session->sql_stmts); > +} > + > +int > +session_add_stmt_id(struct session *session, uint32_t id) > +{ > + if (session->sql_stmts == NULL) { > + session->sql_stmts = mh_i32ptr_new(); > + if (session->sql_stmts == NULL) { > + diag_set(OutOfMemory, 0, "mh_i32ptr_new", > + "session stmt hash"); > + return -1; > + } > + } > + return sql_session_stmt_hash_add_id(session->sql_stmts, id); > +} > + > +void > +session_remove_stmt_id(struct session *session, uint32_t stmt_id) > +{ > + assert(session->sql_stmts != NULL); > + mh_int_t i = mh_i32ptr_find(session->sql_stmts, stmt_id, NULL); > + assert(i != mh_end(session->sql_stmts)); > + mh_i32ptr_del(session->sql_stmts, i, NULL); > +} > + > /** > * To quickly switch to admin user when executing > * on_connect/on_disconnect triggers in iproto. > @@ -227,6 +261,7 @@ session_destroy(struct session *session) > struct mh_i64ptr_node_t node = { session->id, NULL }; > mh_i64ptr_remove(session_registry, &node, NULL); > credentials_destroy(&session->credentials); > + sql_session_stmt_hash_erase(session->sql_stmts); > mempool_free(&session_pool, session); > } > > diff --git a/src/box/session.h b/src/box/session.h > index eff3d7a67..6dfc7cba5 100644 > --- a/src/box/session.h > +++ b/src/box/session.h > @@ -101,6 +101,11 @@ struct session { > const struct session_vtab *vtab; > /** Session metadata. */ > union session_meta meta; > + /** > + * ID of statements prepared in current session. > + * This map is allocated on demand. > + */ > + struct mh_i32ptr_t *sql_stmts; > /** Session user id and global grants */ > struct credentials credentials; > /** Trigger for fiber on_stop to cleanup created on-demand session */ > @@ -267,6 +272,18 @@ session_storage_cleanup(int sid); > struct session * > session_create(enum session_type type); > > +/** Return true if given statement id belongs to the session. */ > +bool > +session_check_stmt_id(struct session *session, uint32_t stmt_id); > + > +/** Add prepared statement ID to the session hash. */ > +int > +session_add_stmt_id(struct session *session, uint32_t stmt_id); > + > +/** Remove prepared statement ID from the session hash. */ > +void > +session_remove_stmt_id(struct session *session, uint32_t stmt_id); > + > /** > * Destroy a session. > * Must be called by the networking layer on disconnect. > diff --git a/src/box/sql.c b/src/box/sql.c > index f1df55571..455fabeef 100644 > --- a/src/box/sql.c > +++ b/src/box/sql.c > @@ -54,6 +54,7 @@ > #include "iproto_constants.h" > #include "fk_constraint.h" > #include "mpstream.h" > +#include "sql_stmt_cache.h" > > static sql *db = NULL; > > @@ -74,6 +75,8 @@ sql_init() > if (sql_init_db(&db) != 0) > panic("failed to initialize SQL subsystem"); > > + sql_stmt_cache_init(); > + > assert(db != NULL); > } > > diff --git a/src/box/sql_stmt_cache.c b/src/box/sql_stmt_cache.c > new file mode 100644 > index 000000000..742e4135c > --- /dev/null > +++ b/src/box/sql_stmt_cache.c > @@ -0,0 +1,289 @@ > +/* > + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file. > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * 1. Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the > + * following disclaimer. > + * > + * 2. Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL > + * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, > + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR > + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF > + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF > + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > +#include "sql_stmt_cache.h" > + > +#include "assoc.h" > +#include "error.h" > +#include "execute.h" > +#include "diag.h" > + > +static struct sql_stmt_cache sql_stmt_cache; > + > +void > +sql_stmt_cache_init() > +{ > + sql_stmt_cache.hash = mh_i32ptr_new(); > + if (sql_stmt_cache.hash == NULL) > + panic("out of memory"); > + sql_stmt_cache.mem_quota = 0; > + sql_stmt_cache.mem_used = 0; > + rlist_create(&sql_stmt_cache.gc_queue); > +} > + > +static size_t > +sql_cache_entry_sizeof(struct sql_stmt *stmt) > +{ > + return sql_stmt_est_size(stmt) + sizeof(struct stmt_cache_entry); > +} > + > +static void > +sql_cache_entry_delete(struct stmt_cache_entry *entry) > +{ > + assert(entry->refs == 0); > + assert(! sql_stmt_busy(entry->stmt)); > + sql_stmt_finalize(entry->stmt); > + TRASH(entry); > + free(entry); > +} > + > +/** > + * Remove statement entry from cache: firstly delete from hash, > + * than remove from LRU list and account cache size changes, > + * finally release occupied memory. > + */ > +static void > +sql_stmt_cache_delete(struct stmt_cache_entry *entry) > +{ > + if (sql_stmt_cache.last_found == entry) > + sql_stmt_cache.last_found = NULL; > + rlist_del(&entry->link); > + sql_stmt_cache.mem_used -= sql_cache_entry_sizeof(entry->stmt); > + sql_cache_entry_delete(entry); > +} > + > +static struct stmt_cache_entry * > +stmt_cache_find_entry(uint32_t stmt_id) > +{ > + if (sql_stmt_cache.last_found != NULL) { > + const char *sql_str = > + sql_stmt_query_str(sql_stmt_cache.last_found->stmt); > + uint32_t last_stmt_id = sql_stmt_calculate_id(sql_str, > + strlen(sql_str)); > + if (last_stmt_id == stmt_id) > + return sql_stmt_cache.last_found; > + /* Fallthrough to slow hash search. */ > + } > + struct mh_i32ptr_t *hash = sql_stmt_cache.hash; > + mh_int_t stmt = mh_i32ptr_find(hash, stmt_id, NULL); > + if (stmt == mh_end(hash)) > + return NULL; > + struct stmt_cache_entry *entry = mh_i32ptr_node(hash, stmt)->val; > + if (entry == NULL) > + return NULL; > + sql_stmt_cache.last_found = entry; > + return entry; > +} > + > +static void > +sql_stmt_cache_gc() > +{ > + struct stmt_cache_entry *entry, *next; > + rlist_foreach_entry_safe(entry, &sql_stmt_cache.gc_queue, link, next) > + sql_stmt_cache_delete(entry); > + assert(rlist_empty(&sql_stmt_cache.gc_queue)); > +} > + > +/** > + * Allocate new cache entry containing given prepared statement. > + * Add it to the LRU cache list. Account cache size enlargement. > + */ > +static struct stmt_cache_entry * > +sql_cache_entry_new(struct sql_stmt *stmt) > +{ > + struct stmt_cache_entry *entry = malloc(sizeof(*entry)); > + if (entry == NULL) { > + diag_set(OutOfMemory, sizeof(*entry), "malloc", > + "struct stmt_cache_entry"); > + return NULL; > + } > + entry->stmt = stmt; > + entry->refs = 0; > + return entry; > +} > + > +/** > + * Return true if used memory (accounting new entry) for SQL > + * prepared statement cache does not exceed the limit. > + */ > +static bool > +sql_cache_check_new_entry_size(size_t size) > +{ > + return (sql_stmt_cache.mem_used + size <= sql_stmt_cache.mem_quota); > +} > + > +static void > +sql_stmt_cache_entry_unref(struct stmt_cache_entry *entry) > +{ > + assert((int64_t)entry->refs - 1 >= 0); > + if (--entry->refs == 0) { > + /* > + * Remove entry from hash and add it to gc queue. > + * Resources are to be released in the nearest > + * GC cycle (see sql_stmt_cache_insert()). > + */ > + struct sql_stmt_cache *cache = &sql_stmt_cache; > + const char *sql_str = sql_stmt_query_str(entry->stmt); > + uint32_t stmt_id = sql_stmt_calculate_id(sql_str, > + strlen(sql_str)); > + mh_int_t i = mh_i32ptr_find(cache->hash, stmt_id, NULL); > + assert(i != mh_end(cache->hash)); > + mh_i32ptr_del(cache->hash, i, NULL); > + rlist_add(&sql_stmt_cache.gc_queue, &entry->link); > + if (sql_stmt_cache.last_found == entry) > + sql_stmt_cache.last_found = NULL; > + } > +} > + > +void > +sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash) > +{ > + if (hash == NULL) > + return; > + mh_int_t i; > + struct stmt_cache_entry *entry; > + mh_foreach(hash, i) { > + entry = (struct stmt_cache_entry *) > + mh_i32ptr_node(hash, i)->val; > + sql_stmt_cache_entry_unref(entry); > + } > + mh_i32ptr_delete(hash); > +} > + > +int > +sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id) > +{ > + struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id); > + const struct mh_i32ptr_node_t id_node = { stmt_id, entry }; > + struct mh_i32ptr_node_t *old_node = NULL; > + mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL); > + if (i == mh_end(hash)) { > + diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node"); > + return -1; > + } > + assert(old_node == NULL); > + entry->refs++; > + return 0; > +} > + > +uint32_t > +sql_stmt_calculate_id(const char *sql_str, size_t len) > +{ > + return mh_strn_hash(sql_str, len); > +} > + > +void > +sql_stmt_unref(uint32_t stmt_id) > +{ > + struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id); > + assert(entry != NULL); > + sql_stmt_cache_entry_unref(entry); > +} > + > +int > +sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt) > +{ > + const char *sql_str = sql_stmt_query_str(old_stmt); > + uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str)); > + struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id); > + uint32_t ref_count = entry->refs; Does this update can happen is such a way that statement can be substituted by a different one, still with the same hash? In such a case users of 'old_stmt' can see surprising results... > + sql_stmt_cache_delete(entry); > + if (sql_stmt_cache_insert(new_stmt) != 0) { > + sql_stmt_finalize(new_stmt); > + return -1; > + } > + /* Restore reference counter. */ > + entry = stmt_cache_find_entry(stmt_id); > + entry->refs = ref_count; > + return 0; > +} > + > +int > +sql_stmt_cache_insert(struct sql_stmt *stmt) > +{ > + assert(stmt != NULL); > + struct sql_stmt_cache *cache = &sql_stmt_cache; > + size_t new_entry_size = sql_cache_entry_sizeof(stmt); > + > + if (! sql_cache_check_new_entry_size(new_entry_size)) > + sql_stmt_cache_gc(); > + /* > + * Test memory limit again. Raise an error if it is > + * still overcrowded. > + */ > + if (! sql_cache_check_new_entry_size(new_entry_size)) { > + diag_set(ClientError, ER_SQL_PREPARE, "Memory limit for SQL "\ > + "prepared statements has been reached. Please, deallocate "\ > + "active statements or increase SQL cache size."); > + return -1; > + } > + struct mh_i32ptr_t *hash = cache->hash; > + struct stmt_cache_entry *entry = sql_cache_entry_new(stmt); > + if (entry == NULL) > + return -1; > + const char *sql_str = sql_stmt_query_str(stmt); > + uint32_t stmt_id = sql_stmt_calculate_id(sql_str, strlen(sql_str)); > + assert(sql_stmt_cache_find(stmt_id) == NULL); Good for debug, but what will we have in release, consider we will never have a chance to test all possible statements? > + const struct mh_i32ptr_node_t id_node = { stmt_id, entry }; > + struct mh_i32ptr_node_t *old_node = NULL; > + mh_int_t i = mh_i32ptr_put(hash, &id_node, &old_node, NULL); > + if (i == mh_end(hash)) { > + sql_cache_entry_delete(entry); > + diag_set(OutOfMemory, 0, "mh_i32ptr_put", "mh_i32ptr_node"); > + return -1; > + } > + assert(old_node == NULL); > + sql_stmt_cache.mem_used += sql_cache_entry_sizeof(stmt); > + return 0; > +} > + > +struct sql_stmt * > +sql_stmt_cache_find(uint32_t stmt_id) > +{ > + struct stmt_cache_entry *entry = stmt_cache_find_entry(stmt_id); > + if (entry == NULL) > + return NULL; > + return entry->stmt; > +} > + > +int > +sql_stmt_cache_set_size(size_t size) > +{ > + if (sql_stmt_cache.mem_used > size) > + sql_stmt_cache_gc(); > + if (sql_stmt_cache.mem_used > size) { > + diag_set(ClientError, ER_SQL_PREPARE, "Can't reduce memory "\ > + "limit for SQL prepared statements: please, deallocate "\ > + "active statements"); > + return -1; > + } > + sql_stmt_cache.mem_quota = size; > + return 0; > +} > diff --git a/src/box/sql_stmt_cache.h b/src/box/sql_stmt_cache.h > new file mode 100644 > index 000000000..f3935a27f > --- /dev/null > +++ b/src/box/sql_stmt_cache.h > @@ -0,0 +1,145 @@ > +#ifndef INCLUDES_PREP_STMT_H > +#define INCLUDES_PREP_STMT_H > +/* > + * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file. > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * 1. Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the > + * following disclaimer. > + * > + * 2. Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED > + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR > + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL > + * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, > + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR > + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF > + * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT > + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF > + * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > +#include > +#include > + > +#include "small/rlist.h" > + > +#if defined(__cplusplus) > +extern "C" { > +#endif > + > +struct sql_stmt; > +struct mh_i64ptr_t; > + > +struct stmt_cache_entry { > + /** Prepared statement itself. */ > + struct sql_stmt *stmt; > + /** > + * Link to the next entry. All statements are to be > + * evicted on the next gc cycle. > + */ > + struct rlist link; > + /** > + * Reference counter. If it is == 0, entry gets > + * into GC queue. > + */ > + uint32_t refs; > +}; > + > +/** > + * Global prepared statements holder. > + */ > +struct sql_stmt_cache { > + /** Size of memory currently occupied by prepared statements. */ > + size_t mem_used; > + /** Max memory size that can be used for cache. */ > + size_t mem_quota; > + /** Query id -> struct stmt_cahce_entry hash.*/ > + struct mh_i32ptr_t *hash; > + /** > + * After deallocation statements are not deleted, but > + * moved to this list. GC process is triggered only > + * when memory limit has reached. It allows to reduce > + * workload on session's disconnect. > + */ > + struct rlist gc_queue; > + /** > + * Last result of sql_stmt_cache_find() invocation. > + * Since during processing prepared statement it > + * may require to find the same statement several > + * times. > + */ > + struct stmt_cache_entry *last_found; > +}; > + > +/** > + * Initialize global cache for prepared statements. Called once > + * during database setup (in sql_init()). > + */ > +void > +sql_stmt_cache_init(); > + > +/** > + * Erase session local hash: unref statements belong to this > + * session and deallocate hash itself. > + * @hash is assumed to be member of struct session @sql_stmts. > + */ > +void > +sql_session_stmt_hash_erase(struct mh_i32ptr_t *hash); > + > +/** > + * Add entry corresponding to prepared statement with given ID > + * to session-local hash and increase its ref counter. > + * @hash is assumed to be member of struct session @sql_stmts. > + */ > +int > +sql_session_stmt_hash_add_id(struct mh_i32ptr_t *hash, uint32_t stmt_id); > + > +/** > + * Prepared statement ID is supposed to be hash value > + * of the original SQL query string. > + */ > +uint32_t > +sql_stmt_calculate_id(const char *sql_str, size_t len); > + > +/** Unref prepared statement entry in global holder. */ > +void > +sql_stmt_unref(uint32_t stmt_id); > + > +int > +sql_stmt_cache_update(struct sql_stmt *old_stmt, struct sql_stmt *new_stmt); > + > +/** > + * Save prepared statement to the prepared statement cache. > + * Account cache size change. If the cache is full (i.e. memory > + * quota is exceeded) diag error is raised. In case of success > + * return id of prepared statement via output parameter @id. > + */ > +int > +sql_stmt_cache_insert(struct sql_stmt *stmt); > + > +/** Find entry by SQL string. In case of search fails it returns NULL. */ > +struct sql_stmt * > +sql_stmt_cache_find(uint32_t stmt_id); > + > + > +/** Set prepared cache size limit. */ > +int > +sql_stmt_cache_set_size(size_t size); > + > +#if defined(__cplusplus) > +} /* extern "C" { */ > +#endif > + > +#endif > diff --git a/test/app-tap/init_script.result b/test/app-tap/init_script.result > index 799297ba0..551a0bbeb 100644 > --- a/test/app-tap/init_script.result > +++ b/test/app-tap/init_script.result > @@ -31,24 +31,25 @@ box.cfg > 26 replication_sync_timeout:300 > 27 replication_timeout:1 > 28 slab_alloc_factor:1.05 > -29 strip_core:true > -30 too_long_threshold:0.5 > -31 vinyl_bloom_fpr:0.05 > -32 vinyl_cache:134217728 > -33 vinyl_dir:. > -34 vinyl_max_tuple_size:1048576 > -35 vinyl_memory:134217728 > -36 vinyl_page_size:8192 > -37 vinyl_read_threads:1 > -38 vinyl_run_count_per_level:2 > -39 vinyl_run_size_ratio:3.5 > -40 vinyl_timeout:60 > -41 vinyl_write_threads:4 > -42 wal_dir:. > -43 wal_dir_rescan_delay:2 > -44 wal_max_size:268435456 > -45 wal_mode:write > -46 worker_pool_threads:4 > +29 sql_cache_size:5242880 > +30 strip_core:true > +31 too_long_threshold:0.5 > +32 vinyl_bloom_fpr:0.05 > +33 vinyl_cache:134217728 > +34 vinyl_dir:. > +35 vinyl_max_tuple_size:1048576 > +36 vinyl_memory:134217728 > +37 vinyl_page_size:8192 > +38 vinyl_read_threads:1 > +39 vinyl_run_count_per_level:2 > +40 vinyl_run_size_ratio:3.5 > +41 vinyl_timeout:60 > +42 vinyl_write_threads:4 > +43 wal_dir:. > +44 wal_dir_rescan_delay:2 > +45 wal_max_size:268435456 > +46 wal_mode:write > +47 worker_pool_threads:4 > -- > -- Test insert from detached fiber > -- > diff --git a/test/box/admin.result b/test/box/admin.result > index 6126f3a97..852c1cde8 100644 > --- a/test/box/admin.result > +++ b/test/box/admin.result > @@ -83,6 +83,8 @@ cfg_filter(box.cfg) > - 1 > - - slab_alloc_factor > - 1.05 > + - - sql_cache_size > + - 5242880 > - - strip_core > - true > - - too_long_threshold > diff --git a/test/box/cfg.result b/test/box/cfg.result > index 5370bb870..331f5e986 100644 > --- a/test/box/cfg.result > +++ b/test/box/cfg.result > @@ -71,6 +71,8 @@ cfg_filter(box.cfg) > | - 1 > | - - slab_alloc_factor > | - 1.05 > + | - - sql_cache_size > + | - 5242880 > | - - strip_core > | - true > | - - too_long_threshold > @@ -170,6 +172,8 @@ cfg_filter(box.cfg) > | - 1 > | - - slab_alloc_factor > | - 1.05 > + | - - sql_cache_size > + | - 5242880 > | - - strip_core > | - true > | - - too_long_threshold > @@ -315,6 +319,9 @@ box.cfg{memtx_memory = box.cfg.memtx_memory} > box.cfg{vinyl_memory = box.cfg.vinyl_memory} > | --- > | ... > +box.cfg{sql_cache_size = box.cfg.sql_cache_size} > + | --- > + | ... > > -------------------------------------------------------------------------------- > -- Test of default cfg options > diff --git a/test/box/cfg.test.lua b/test/box/cfg.test.lua > index 56ccb6767..e6a90d770 100644 > --- a/test/box/cfg.test.lua > +++ b/test/box/cfg.test.lua > @@ -51,6 +51,7 @@ box.cfg{replicaset_uuid = '12345678-0123-5678-1234-abcdefabcdef'} > > box.cfg{memtx_memory = box.cfg.memtx_memory} > box.cfg{vinyl_memory = box.cfg.vinyl_memory} > +box.cfg{sql_cache_size = box.cfg.sql_cache_size} > > -------------------------------------------------------------------------------- > -- Test of default cfg options > diff --git a/test/box/misc.result b/test/box/misc.result > index d2a20307a..7e5d28b70 100644 > --- a/test/box/misc.result > +++ b/test/box/misc.result > @@ -554,6 +554,7 @@ t; > 203: box.error.BOOTSTRAP_READONLY > 204: box.error.SQL_FUNC_WRONG_RET_COUNT > 205: box.error.FUNC_INVALID_RETURN_TYPE > + 206: box.error.SQL_PREPARE > ... > test_run:cmd("setopt delimiter ''"); > --- > -- > 2.15.1 >