[PATCH v2 0/4] Introduce replica local spaces

[PATCH v2 0/4] Introduce replica local spaces

[Vladimir Davydov]

This patch introduces a new space option, group_id, which defines how
a space is replicated. If it is 0 (default), the space is replicated
throughout the cluster. If it is 1, the space is not replicated.
For more information, see patch 4.

https://github.com/tarantool/tarantool/issues/3443
https://github.com/tarantool/tarantool/commits/dv/gh-3443-replica-local-spaces

v1: https://www.freelists.org/post/tarantool-patches/PATCH-06-Introduce-replica-local-spaces

Changes in v2:
 - Use group_id space option instead of is_local flag, as this is more
   flexible and will allow us to implement fully-functional replication
   groups in future (Kostja).
 - Refactor code used for committing NOP requests (Kostja).
 - Rename space_opts::temporary to is_temporary (Kostja).
 - Add a comment to request_create_from_tuple (Kostja).
 - Remove merged patches from the patch set (patches 1-3 from v1).

Vladimir Davydov (4):
  txn: do not require space id for nop requests
  xrow: make NOP requests bodiless
  Rename space_opts::temporary to is_temporary
  Introduce replica local spaces

 src/box/alter.cc                       |  15 ++-
 src/box/box.cc                         |  24 ++++-
 src/box/errcode.h                      |   1 +
 src/box/iproto_constants.c             |   6 +-
 src/box/iproto_constants.h             |   1 +
 src/box/lua/schema.lua                 |   8 +-
 src/box/lua/space.cc                   |   5 +
 src/box/lua/xlog.c                     |  20 ++--
 src/box/memtx_engine.c                 |  10 +-
 src/box/memtx_space.c                  |   4 +-
 src/box/relay.cc                       |  17 +++-
 src/box/replication.h                  |  14 +++
 src/box/request.c                      |   7 +-
 src/box/space.h                        |  12 ++-
 src/box/space_def.c                    |   6 +-
 src/box/space_def.h                    |   7 +-
 src/box/tuple_format.c                 |   2 +-
 src/box/tuple_format.h                 |   2 +-
 src/box/txn.c                          |  79 ++++++++-------
 src/box/vinyl.c                        |   7 +-
 src/box/xrow.c                         |  28 ++++--
 src/box/xrow.h                         |   1 +
 test/box/before_replace.result         |   2 +-
 test/box/before_replace.test.lua       |   2 +-
 test/box/misc.result                   |   5 +-
 test/box/on_replace.result             |   4 +-
 test/engine/iterator.result            |   2 +-
 test/replication/local_spaces.result   | 174 +++++++++++++++++++++++++++++++++
 test/replication/local_spaces.test.lua |  63 ++++++++++++
 test/replication/suite.cfg             |   1 +
 test/vinyl/ddl.result                  |   5 +
 test/vinyl/ddl.test.lua                |   3 +
 32 files changed, 452 insertions(+), 85 deletions(-)
 create mode 100644 test/replication/local_spaces.result
 create mode 100644 test/replication/local_spaces.test.lua

-- 
2.11.0

[PATCH v2 1/4] txn: do not require space id for nop requests

[Vladimir Davydov]

Currently, IPROTO_NOP can only be generated by a before_replace trigger,
when it returns the old tuple thus turning the original operation into a
NOP. In such a case we know the space id and we write it to the request
body. This allows us to dispatch NOP requests via DML route.

As a part of replica local spaces feature, we will substitute requests
operating on local spaces with NOP in relay in order to promote vclock
on replicas without actual data modification. Since space_id is stored
in request body, sending it to replicas would mean decoding the request
body in relay, which is an overkill. To avoid that, let's separate NOP
and DML paths and remove space_id from NOP requests.

Needed for #3443
---
 src/box/box.cc                   | 21 +++++++++++
 src/box/iproto_constants.c       |  2 +-
 src/box/request.c                |  7 ++--
 src/box/txn.c                    | 78 ++++++++++++++++++++--------------------
 test/box/before_replace.result   |  4 +--
 test/box/before_replace.test.lua |  2 +-
 test/box/on_replace.result       |  4 +--
 7 files changed, 72 insertions(+), 46 deletions(-)

diff --git a/src/box/box.cc b/src/box/box.cc
index ba0af95e..00a025b2 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -192,6 +192,22 @@ process_rw(struct request *request, struct space *space, struct tuple **result)
 	return rc;
 }
 
+/**
+ * Process a no-op request.
+ *
+ * A no-op request does not affect any space, but it
+ * promotes vclock and is written to WAL.
+ */
+static int
+process_nop(struct request *request)
+{
+	assert(request->type == IPROTO_NOP);
+	struct txn *txn = txn_begin_stmt(NULL);
+	if (txn == NULL)
+		return -1;
+	return txn_commit_stmt(txn, request);
+}
+
 void
 box_set_ro(bool ro)
 {
@@ -279,6 +295,11 @@ apply_row(struct xstream *stream, struct xrow_header *row)
 	(void) stream;
 	struct request request;
 	xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type));
+	if (request.type == IPROTO_NOP) {
+		if (process_nop(&request) != 0)
+			diag_raise();
+		return;
+	}
 	struct space *space = space_cache_find_xc(request.space_id);
 	if (process_rw(&request, space, NULL) != 0) {
 		say_error("error applying row: %s", request_str(&request));
diff --git a/src/box/iproto_constants.c b/src/box/iproto_constants.c
index 3adb7cd4..5c1d3a31 100644
--- a/src/box/iproto_constants.c
+++ b/src/box/iproto_constants.c
@@ -121,7 +121,7 @@ const uint64_t iproto_body_key_map[IPROTO_TYPE_STAT_MAX] = {
 	bit(SPACE_ID) | bit(OPS) | bit(TUPLE),                 /* UPSERT */
 	0,                                                     /* CALL */
 	0,                                                     /* reserved */
-	bit(SPACE_ID),                                         /* NOP */
+	0,                                                     /* NOP */
 };
 #undef bit
 
diff --git a/src/box/request.c b/src/box/request.c
index fda54a18..8690519c 100644
--- a/src/box/request.c
+++ b/src/box/request.c
@@ -50,7 +50,6 @@ request_create_from_tuple(struct request *request, struct space *space,
 			  struct tuple *old_tuple, struct tuple *new_tuple)
 {
 	memset(request, 0, sizeof(*request));
-	request->space_id = space->def->id;
 
 	if (old_tuple == new_tuple) {
 		/*
@@ -60,7 +59,11 @@ request_create_from_tuple(struct request *request, struct space *space,
 		request->type = IPROTO_NOP;
 		return 0;
 	}
-
+	/*
+	 * Space pointer may be zero in case of NOP, in which case
+	 * this line is not reached.
+	 */
+	request->space_id = space->def->id;
 	if (new_tuple == NULL) {
 		uint32_t size, key_size;
 		const char *data = tuple_data_range(old_tuple, &size);
diff --git a/src/box/txn.c b/src/box/txn.c
index b5ad39b0..80e5463d 100644
--- a/src/box/txn.c
+++ b/src/box/txn.c
@@ -106,13 +106,15 @@ txn_rollback_to_svp(struct txn *txn, struct stailq_entry *svp)
 	stailq_cut_tail(&txn->stmts, svp, &rollback);
 	stailq_reverse(&rollback);
 	stailq_foreach_entry(stmt, &rollback, next) {
-		engine_rollback_statement(txn->engine, txn, stmt);
+		if (stmt->space != NULL) {
+			engine_rollback_statement(txn->engine, txn, stmt);
+			stmt->space = NULL;
+		}
 		if (stmt->row != NULL) {
 			assert(txn->n_rows > 0);
 			txn->n_rows--;
 			stmt->row = NULL;
 		}
-		stmt->space = NULL;
 	}
 }
 
@@ -146,7 +148,6 @@ int
 txn_begin_in_engine(struct engine *engine, struct txn *txn)
 {
 	if (txn->engine == NULL) {
-		assert(stailq_empty(&txn->stmts));
 		txn->engine = engine;
 		return engine_begin(engine, txn);
 	} else if (txn->engine != engine) {
@@ -173,6 +174,15 @@ txn_begin_stmt(struct space *space)
 		return NULL;
 	}
 
+	struct txn_stmt *stmt = txn_stmt_new(txn);
+	if (stmt == NULL) {
+		if (txn->is_autocommit && txn->in_sub_stmt == 0)
+			txn_rollback();
+		return NULL;
+	}
+	if (space == NULL)
+		return txn;
+
 	if (trigger_run(&space->on_stmt_begin, txn) != 0)
 		goto fail;
 
@@ -180,19 +190,13 @@ txn_begin_stmt(struct space *space)
 	if (txn_begin_in_engine(engine, txn) != 0)
 		goto fail;
 
-	struct txn_stmt *stmt = txn_stmt_new(txn);
-	if (stmt == NULL)
-		goto fail;
 	stmt->space = space;
+	if (engine_begin_statement(engine, txn) != 0)
+		goto fail;
 
-	if (engine_begin_statement(engine, txn) != 0) {
-		txn_rollback_stmt();
-		return NULL;
-	}
 	return txn;
 fail:
-	if (txn->is_autocommit && txn->in_sub_stmt == 0)
-		txn_rollback();
+	txn_rollback_stmt();
 	return NULL;
 }
 
@@ -211,7 +215,7 @@ txn_commit_stmt(struct txn *txn, struct request *request)
 	struct txn_stmt *stmt = txn_current_stmt(txn);
 
 	/* Create WAL record for the write requests in non-temporary spaces */
-	if (!space_is_temporary(stmt->space)) {
+	if (stmt->space == NULL || !space_is_temporary(stmt->space)) {
 		if (txn_add_redo(stmt, request) != 0)
 			goto fail;
 		++txn->n_rows;
@@ -225,7 +229,7 @@ txn_commit_stmt(struct txn *txn, struct request *request)
 	 * - perhaps we should run triggers even for deletes which
 	 *   doesn't find any rows
 	 */
-	if (!rlist_empty(&stmt->space->on_replace) &&
+	if (stmt->space != NULL && !rlist_empty(&stmt->space->on_replace) &&
 	    stmt->space->run_triggers && (stmt->old_tuple || stmt->new_tuple)) {
 		if (trigger_run(&stmt->space->on_replace, txn) != 0)
 			goto fail;
@@ -239,7 +243,6 @@ fail:
 	return -1;
 }
 
-
 static int64_t
 txn_write_to_wal(struct txn *txn)
 {
@@ -288,32 +291,31 @@ txn_commit(struct txn *txn)
 {
 	assert(txn == in_txn());
 
-	assert(stailq_empty(&txn->stmts) || txn->engine);
-
 	/* Do transaction conflict resolving */
-	if (txn->engine) {
-		if (engine_prepare(txn->engine, txn) != 0)
-			goto fail;
+	if (txn->engine != NULL &&
+	    engine_prepare(txn->engine, txn) != 0)
+		goto fail;
 
-		if (txn->n_rows > 0) {
-			txn->signature = txn_write_to_wal(txn);
-			if (txn->signature < 0)
-				goto fail;
-		}
-		/*
-		 * The transaction is in the binary log. No action below
-		 * may throw. In case an error has happened, there is
-		 * no other option but terminate.
-		 */
-		if (txn->has_triggers &&
-		    trigger_run(&txn->on_commit, txn) != 0) {
-			diag_log();
-			unreachable();
-			panic("commit trigger failed");
-		}
+	if (txn->n_rows > 0) {
+		txn->signature = txn_write_to_wal(txn);
+		if (txn->signature < 0)
+			goto fail;
+	}
+	/*
+	 * The transaction is in the binary log. No action below
+	 * may throw. In case an error has happened, there is
+	 * no other option but terminate.
+	 */
+	if (txn->has_triggers &&
+	    trigger_run(&txn->on_commit, txn) != 0) {
+		diag_log();
+		unreachable();
+		panic("commit trigger failed");
+	}
 
+	if (txn->engine != NULL)
 		engine_commit(txn->engine, txn);
-	}
+
 	TRASH(txn);
 	/** Free volatile txn memory. */
 	fiber_gc();
@@ -478,7 +480,7 @@ box_txn_rollback_to_savepoint(box_txn_savepoint_t *svp)
 	}
 	struct txn_stmt *stmt = svp->stmt == NULL ? NULL :
 			stailq_entry(svp->stmt, struct txn_stmt, next);
-	if (stmt != NULL && stmt->space == NULL) {
+	if (stmt != NULL && stmt->space == NULL && stmt->row == NULL) {
 		/*
 		 * The statement at which this savepoint was
 		 * created has been rolled back.
diff --git a/test/box/before_replace.result b/test/box/before_replace.result
index 4f47b9fa..2b6c1801 100644
--- a/test/box/before_replace.result
+++ b/test/box/before_replace.result
@@ -660,9 +660,9 @@ row.HEADER.type
 ---
 - NOP
 ...
-row.BODY.space_id == s.id
+#row.BODY
 ---
-- true
+- 0
 ...
 -- gh-3128 before_replace with run_triggers
 s2 = box.schema.space.create("test2")
diff --git a/test/box/before_replace.test.lua b/test/box/before_replace.test.lua
index 22733c1d..9b4f49cf 100644
--- a/test/box/before_replace.test.lua
+++ b/test/box/before_replace.test.lua
@@ -221,7 +221,7 @@ path = fio.pathjoin(box.cfg.wal_dir, string.format('%020d.xlog', box.info.lsn -
 fun, param, state = xlog.pairs(path)
 state, row = fun(param, state)
 row.HEADER.type
-row.BODY.space_id == s.id
+#row.BODY
 
 -- gh-3128 before_replace with run_triggers
 s2 = box.schema.space.create("test2")
diff --git a/test/box/on_replace.result b/test/box/on_replace.result
index 8c52e128..fcdb4379 100644
--- a/test/box/on_replace.result
+++ b/test/box/on_replace.result
@@ -527,14 +527,14 @@ t = s:on_replace(function () s:rename('newname') end, t)
 ...
 s:replace({8, 9})
 ---
-- error: Space _space does not support multi-statement transactions
+- error: DDL does not support multi-statement transactions
 ...
 t = s:on_replace(function () s.index.pk:rename('newname') end, t)
 ---
 ...
 s:replace({9, 10})
 ---
-- error: Space _index does not support multi-statement transactions
+- error: DDL does not support multi-statement transactions
 ...
 s:select()
 ---
-- 
2.11.0

[PATCH v2 2/4] xrow: make NOP requests bodiless

[Vladimir Davydov]

A NOP request has no body, but since it is treated as DML, we still
encode a zero-size map for its body. This complicates conversion of
local requests to NOP in relay as we can't omit xrow_encode_dml (see
the next patch), so let's allow DML requests to be bodiless.

Needed for #3443
---
 src/box/box.cc                   |  1 -
 src/box/lua/xlog.c               | 15 ++++++++-------
 src/box/xrow.c                   | 19 ++++++++++---------
 test/box/before_replace.result   |  4 ++--
 test/box/before_replace.test.lua |  2 +-
 5 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/src/box/box.cc b/src/box/box.cc
index 00a025b2..ec38886d 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -291,7 +291,6 @@ recovery_journal_create(struct recovery_journal *journal, struct vclock *v)
 static inline void
 apply_row(struct xstream *stream, struct xrow_header *row)
 {
-	assert(row->bodycnt == 1); /* always 1 for read */
 	(void) stream;
 	struct request request;
 	xrow_decode_dml_xc(row, &request, dml_request_key_map(row->type));
diff --git a/src/box/lua/xlog.c b/src/box/lua/xlog.c
index 70384c1d..2271c829 100644
--- a/src/box/lua/xlog.c
+++ b/src/box/lua/xlog.c
@@ -219,13 +219,14 @@ lbox_xlog_parser_iterate(struct lua_State *L)
 
 	lua_settable(L, -3); /* HEADER */
 
-	assert(row.bodycnt == 1); /* always 1 for read */
-	lua_pushstring(L, "BODY");
-	lua_newtable(L);
-	lbox_xlog_parse_body(L, row.type, (char *)row.body[0].iov_base,
-			     row.body[0].iov_len);
-	lua_settable(L, -3);  /* BODY */
-
+	if (row.bodycnt > 0) {
+		assert(row.bodycnt == 1);
+		lua_pushstring(L, "BODY");
+		lua_newtable(L);
+		lbox_xlog_parse_body(L, row.type, row.body[0].iov_base,
+				     row.body[0].iov_len);
+		lua_settable(L, -3);  /* BODY */
+	}
 	return 2;
 }
 
diff --git a/src/box/xrow.c b/src/box/xrow.c
index 48fbff27..64d845f7 100644
--- a/src/box/xrow.c
+++ b/src/box/xrow.c
@@ -405,11 +405,12 @@ int
 xrow_decode_dml(struct xrow_header *row, struct request *request,
 		uint64_t key_map)
 {
-	if (row->bodycnt == 0) {
-		diag_set(ClientError, ER_INVALID_MSGPACK,
-			 "missing request body");
-		return -1;
-	}
+	memset(request, 0, sizeof(*request));
+	request->header = row;
+	request->type = row->type;
+
+	if (row->bodycnt == 0)
+		goto done;
 
 	assert(row->bodycnt == 1);
 	const char *data = (const char *) row->body[0].iov_base;
@@ -422,10 +423,6 @@ error:
 		return -1;
 	}
 
-	memset(request, 0, sizeof(*request));
-	request->header = row;
-	request->type = row->type;
-
 	uint32_t size = mp_decode_map(&data);
 	for (uint32_t i = 0; i < size; i++) {
 		if (! iproto_dml_body_has_key(data, end)) {
@@ -480,6 +477,7 @@ error:
 		diag_set(ClientError, ER_INVALID_MSGPACK, "packet end");
 		return -1;
 	}
+done:
 	if (key_map) {
 		enum iproto_key key = (enum iproto_key) bit_ctz_u64(key_map);
 		diag_set(ClientError, ER_MISSING_REQUEST_FIELD,
@@ -567,6 +565,9 @@ xrow_encode_dml(const struct request *request, struct iovec *iov)
 		map_size++;
 	}
 
+	if (map_size == 0)
+		return 0;
+
 	assert(pos <= begin + len);
 	mp_encode_map(begin, map_size);
 	iov[0].iov_base = begin;
diff --git a/test/box/before_replace.result b/test/box/before_replace.result
index 2b6c1801..cd0a3be1 100644
--- a/test/box/before_replace.result
+++ b/test/box/before_replace.result
@@ -660,9 +660,9 @@ row.HEADER.type
 ---
 - NOP
 ...
-#row.BODY
+row.BODY == nil
 ---
-- 0
+- true
 ...
 -- gh-3128 before_replace with run_triggers
 s2 = box.schema.space.create("test2")
diff --git a/test/box/before_replace.test.lua b/test/box/before_replace.test.lua
index 9b4f49cf..dd464384 100644
--- a/test/box/before_replace.test.lua
+++ b/test/box/before_replace.test.lua
@@ -221,7 +221,7 @@ path = fio.pathjoin(box.cfg.wal_dir, string.format('%020d.xlog', box.info.lsn -
 fun, param, state = xlog.pairs(path)
 state, row = fun(param, state)
 row.HEADER.type
-#row.BODY
+row.BODY == nil
 
 -- gh-3128 before_replace with run_triggers
 s2 = box.schema.space.create("test2")
-- 
2.11.0

[PATCH v2 3/4] Rename space_opts::temporary to is_temporary

[Vladimir Davydov]

We typically prefix all boolean variables with 'is_', so let's rename
space_opts::temporary to is_temporary for consistency.

While we are at it, let's also rename tuple_format::temporary to
is_temporary and use space_is_temporary() helper wherever we have
a space pointer.
---
 src/box/alter.cc       | 2 +-
 src/box/box.cc         | 2 +-
 src/box/memtx_engine.c | 2 +-
 src/box/memtx_space.c  | 4 ++--
 src/box/space.h        | 5 ++++-
 src/box/space_def.c    | 4 ++--
 src/box/space_def.h    | 2 +-
 src/box/tuple_format.c | 2 +-
 src/box/tuple_format.h | 2 +-
 src/box/vinyl.c        | 2 +-
 10 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/src/box/alter.cc b/src/box/alter.cc
index 509e4b7e..07dee7d9 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -390,7 +390,7 @@ space_opts_decode(struct space_opts *opts, const char *data)
 			while (isspace(*flags)) /* skip space */
 				flags++;
 			if (strncmp(flags, "temporary", strlen("temporary")) == 0)
-				opts->temporary = true;
+				opts->is_temporary = true;
 			flags = strchr(flags, ',');
 			if (flags)
 				flags++;
diff --git a/src/box/box.cc b/src/box/box.cc
index ec38886d..15b84374 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -976,7 +976,7 @@ box_process1(struct request *request, box_tuple_t **result)
 	struct space *space = space_cache_find(request->space_id);
 	if (space == NULL)
 		return -1;
-	if (!space->def->opts.temporary && box_check_writable() != 0)
+	if (!space_is_temporary(space) && box_check_writable() != 0)
 		return -1;
 	return process_rw(request, space, result);
 }
diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c
index 06858a84..74c6be8d 100644
--- a/src/box/memtx_engine.c
+++ b/src/box/memtx_engine.c
@@ -1221,7 +1221,7 @@ memtx_tuple_delete(struct tuple_format *format, struct tuple *tuple)
 		container_of(tuple, struct memtx_tuple, base);
 	if (memtx->alloc.free_mode != SMALL_DELAYED_FREE ||
 	    memtx_tuple->version == memtx->snapshot_version ||
-	    format->temporary)
+	    format->is_temporary)
 		smfree(&memtx->alloc, memtx_tuple, total);
 	else
 		smfree_delayed(&memtx->alloc, memtx_tuple, total);
diff --git a/src/box/memtx_space.c b/src/box/memtx_space.c
index aef7e788..32700071 100644
--- a/src/box/memtx_space.c
+++ b/src/box/memtx_space.c
@@ -830,7 +830,7 @@ memtx_space_prepare_alter(struct space *old_space, struct space *new_space)
 	struct memtx_space *new_memtx_space = (struct memtx_space *)new_space;
 
 	if (old_memtx_space->bsize != 0 &&
-	    old_space->def->opts.temporary != new_space->def->opts.temporary) {
+	    space_is_temporary(old_space) != space_is_temporary(new_space)) {
 		diag_set(ClientError, ER_ALTER_SPACE, old_space->def->name,
 			 "can not switch temporary flag on a non-empty space");
 		return -1;
@@ -896,7 +896,7 @@ memtx_space_new(struct memtx_engine *memtx,
 		return NULL;
 	}
 	format->engine = memtx;
-	format->temporary = def->opts.temporary;
+	format->is_temporary = def->opts.is_temporary;
 	format->exact_field_count = def->exact_field_count;
 	tuple_format_ref(format);
 
diff --git a/src/box/space.h b/src/box/space.h
index cbd46c9e..074e2462 100644
--- a/src/box/space.h
+++ b/src/box/space.h
@@ -193,7 +193,10 @@ space_name(const struct space *space)
 
 /** Return true if space is temporary. */
 static inline bool
-space_is_temporary(struct space *space) { return space->def->opts.temporary; }
+space_is_temporary(struct space *space)
+{
+	return space->def->opts.is_temporary;
+}
 
 void
 space_run_triggers(struct space *space, bool yesno);
diff --git a/src/box/space_def.c b/src/box/space_def.c
index 7349c214..ff35cb20 100644
--- a/src/box/space_def.c
+++ b/src/box/space_def.c
@@ -34,11 +34,11 @@
 #include "error.h"
 
 const struct space_opts space_opts_default = {
-	/* .temporary = */ false,
+	/* .is_temporary = */ false,
 };
 
 const struct opt_def space_opts_reg[] = {
-	OPT_DEF("temporary", OPT_BOOL, struct space_opts, temporary),
+	OPT_DEF("temporary", OPT_BOOL, struct space_opts, is_temporary),
 	OPT_END,
 };
 
diff --git a/src/box/space_def.h b/src/box/space_def.h
index 97c7e138..6cee6ad8 100644
--- a/src/box/space_def.h
+++ b/src/box/space_def.h
@@ -47,7 +47,7 @@ struct space_opts {
 	 * - changes are not written to WAL
 	 * - changes are not part of a snapshot
 	 */
-	bool temporary;
+	bool is_temporary;
 };
 
 extern const struct space_opts space_opts_default;
diff --git a/src/box/tuple_format.c b/src/box/tuple_format.c
index 486646ea..2e19d2e3 100644
--- a/src/box/tuple_format.c
+++ b/src/box/tuple_format.c
@@ -270,7 +270,7 @@ tuple_format_new(struct tuple_format_vtab *vtab, struct key_def * const *keys,
 	format->vtab = *vtab;
 	format->engine = NULL;
 	format->extra_size = extra_size;
-	format->temporary = false;
+	format->is_temporary = false;
 	if (tuple_format_register(format) < 0) {
 		tuple_format_destroy(format);
 		free(format);
diff --git a/src/box/tuple_format.h b/src/box/tuple_format.h
index 9da9be3e..c7dc48ff 100644
--- a/src/box/tuple_format.h
+++ b/src/box/tuple_format.h
@@ -128,7 +128,7 @@ struct tuple_format {
 	 * hence can be freed immediately while checkpointing is
 	 * in progress.
 	 */
-	bool temporary;
+	bool is_temporary;
 	/**
 	 * The number of extra bytes to reserve in tuples before
 	 * field map.
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 10edbeda..08a83bb5 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -590,7 +590,7 @@ vy_lsm_find_unique(struct space *space, uint32_t index_id)
 static int
 vinyl_engine_check_space_def(struct space_def *def)
 {
-	if (def->opts.temporary) {
+	if (def->opts.is_temporary) {
 		diag_set(ClientError, ER_ALTER_SPACE,
 			 def->name, "engine does not support temporary flag");
 		return -1;
-- 
2.11.0

[PATCH v2 4/4] Introduce replica local spaces

[Vladimir Davydov]

This patch introduces a new space option, group_id, which defines how
the space is replicated. If it is 0 (default), the space is replicated
throughout the entire cluster. If it is 1, the space is replica local,
i.e. all changes made to it are invisible to other replicas in the
cluster. Currently, no other value is permitted, but in future we will
use this option for setting up arbitrary replication groups in a
cluster. The option can only be set on space creation and cannot be
altered.

Technically, to support this feature, we introduce a new header key,
IPROTO_GROUP_ID, which is set to the space group id for all rows
corresponding to a space, both in xlog and in snap. Relay won't send
snapshot rows whose group_id is 1. As for xlog rows, they are
transformed to IPROTO_NOP so as to promote vclock on replicas without
any actual data modification.

The feature is currently supported for memtx spaces only, but it should
be easy to implement it for vinyl spaces as well.

@TarantoolBot document
Title: Document new space option - group_id
Group identifier defines how the space is replicated. Currently, only
two values are permitted: 0 and 1. If space.group_id is 0, changes made
to the space are replicated throughout the entire cluster. If it is 1,
then the space is replica local, i.e. changes made to it are invisible
to other replicas in the cluster.

Closes #3443
---
 src/box/alter.cc                       |  13 +++
 src/box/errcode.h                      |   1 +
 src/box/iproto_constants.c             |   4 +-
 src/box/iproto_constants.h             |   1 +
 src/box/lua/schema.lua                 |   8 +-
 src/box/lua/space.cc                   |   5 +
 src/box/lua/xlog.c                     |   5 +
 src/box/memtx_engine.c                 |   8 +-
 src/box/relay.cc                       |  17 +++-
 src/box/replication.h                  |  14 +++
 src/box/space.h                        |   7 ++
 src/box/space_def.c                    |   2 +
 src/box/space_def.h                    |   5 +
 src/box/txn.c                          |   1 +
 src/box/vinyl.c                        |   5 +
 src/box/xrow.c                         |   9 ++
 src/box/xrow.h                         |   1 +
 test/box/misc.result                   |   5 +-
 test/engine/iterator.result            |   2 +-
 test/replication/local_spaces.result   | 174 +++++++++++++++++++++++++++++++++
 test/replication/local_spaces.test.lua |  63 ++++++++++++
 test/replication/suite.cfg             |   1 +
 test/vinyl/ddl.result                  |   5 +
 test/vinyl/ddl.test.lua                |   3 +
 24 files changed, 348 insertions(+), 11 deletions(-)
 create mode 100644 test/replication/local_spaces.result
 create mode 100644 test/replication/local_spaces.test.lua

diff --git a/src/box/alter.cc b/src/box/alter.cc
index 07dee7d9..9d114b86 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -560,6 +560,15 @@ space_def_new_from_tuple(struct tuple *tuple, uint32_t errcode,
 	}
 	struct space_opts opts;
 	space_opts_decode(&opts, space_opts);
+	/*
+	 * Currently, only predefined replication groups
+	 * are supported.
+	 */
+	if (opts.group_id != GROUP_DEFAULT &&
+	    opts.group_id != GROUP_LOCAL) {
+		tnt_raise(ClientError, ER_NO_SUCH_GROUP,
+			  int2str(opts.group_id));
+	}
 	struct space_def *def =
 		space_def_new_xc(id, uid, exact_field_count, name, name_len,
 				 engine_name, engine_name_len, &opts, fields,
@@ -1631,6 +1640,10 @@ on_replace_dd_space(struct trigger * /* trigger */, void *event)
 			tnt_raise(ClientError, ER_ALTER_SPACE,
 				  space_name(old_space),
 				  "can not change space engine");
+		if (def->opts.group_id != space_group_id(old_space))
+			tnt_raise(ClientError, ER_ALTER_SPACE,
+				  space_name(old_space),
+				  "group id is immutable");
 		/*
 		 * Allow change of space properties, but do it
 		 * in WAL-error-safe mode.
diff --git a/src/box/errcode.h b/src/box/errcode.h
index f3987cf0..3d5f66af 100644
--- a/src/box/errcode.h
+++ b/src/box/errcode.h
@@ -207,6 +207,7 @@ struct errcode_record {
 	/*152 */_(ER_NULLABLE_PRIMARY,		"Primary index of the space '%s' can not contain nullable parts") \
 	/*153 */_(ER_NULLABLE_MISMATCH,		"Field %d is %s in space format, but %s in index parts") \
 	/*154 */_(ER_TRANSACTION_YIELD,		"Transaction has been aborted by a fiber yield") \
+	/*155 */_(ER_NO_SUCH_GROUP,		"Replication group '%s' does not exist") \
 
 /*
  * !IMPORTANT! Please follow instructions at start of the file
diff --git a/src/box/iproto_constants.c b/src/box/iproto_constants.c
index 5c1d3a31..3bc965bd 100644
--- a/src/box/iproto_constants.c
+++ b/src/box/iproto_constants.c
@@ -40,10 +40,10 @@ const unsigned char iproto_key_type[IPROTO_KEY_MAX] =
 		/* 0x04 */	MP_DOUBLE, /* IPROTO_TIMESTAMP */
 		/* 0x05 */	MP_UINT,   /* IPROTO_SCHEMA_VERSION */
 		/* 0x06 */	MP_UINT,   /* IPROTO_SERVER_VERSION */
+		/* 0x07 */	MP_UINT,   /* IPROTO_GROUP_ID */
 	/* }}} */
 
 	/* {{{ unused */
-		/* 0x07 */	MP_UINT,
 		/* 0x08 */	MP_UINT,
 		/* 0x09 */	MP_UINT,
 		/* 0x0a */	MP_UINT,
@@ -133,7 +133,7 @@ const char *iproto_key_strs[IPROTO_KEY_MAX] = {
 	"timestamp",        /* 0x04 */
 	"schema version",   /* 0x05 */
 	"server version",   /* 0x06 */
-	NULL,               /* 0x07 */
+	"group id",         /* 0x07 */
 	NULL,               /* 0x08 */
 	NULL,               /* 0x09 */
 	NULL,               /* 0x0a */
diff --git a/src/box/iproto_constants.h b/src/box/iproto_constants.h
index d1320de7..ccbf2da5 100644
--- a/src/box/iproto_constants.h
+++ b/src/box/iproto_constants.h
@@ -58,6 +58,7 @@ enum iproto_key {
 	IPROTO_TIMESTAMP = 0x04,
 	IPROTO_SCHEMA_VERSION = 0x05,
 	IPROTO_SERVER_VERSION = 0x06,
+	IPROTO_GROUP_ID = 0x07,
 	/* Leave a gap for other keys in the header. */
 	IPROTO_SPACE_ID = 0x10,
 	IPROTO_INDEX_ID = 0x11,
diff --git a/src/box/lua/schema.lua b/src/box/lua/schema.lua
index dc18f71b..356f05b6 100644
--- a/src/box/lua/schema.lua
+++ b/src/box/lua/schema.lua
@@ -418,6 +418,7 @@ box.schema.space.create = function(name, options)
         field_count = 'number',
         user = 'string, number',
         format = 'table',
+        group_id = 'number',
         temporary = 'boolean',
     }
     local options_defaults = {
@@ -461,6 +462,7 @@ box.schema.space.create = function(name, options)
     format = update_format(format)
     -- filter out global parameters from the options array
     local space_options = setmap({
+        group_id = options.group_id,
         temporary = options.temporary and true or nil,
     })
     _space:insert{id, uid, name, options.engine, options.field_count,
@@ -2369,7 +2371,11 @@ local function box_space_mt(tab)
     for k,v in pairs(tab) do
         -- skip system spaces and views
         if type(k) == 'string' and #k > 0 and k:sub(1,1) ~= '_' then
-            t[k] = { engine = v.engine, temporary = v.temporary }
+            t[k] = {
+                engine = v.engine,
+                group_id = v.group_id,
+                temporary = v.temporary,
+            }
         end
     end
     return t
diff --git a/src/box/lua/space.cc b/src/box/lua/space.cc
index 52438275..5a38a2ac 100644
--- a/src/box/lua/space.cc
+++ b/src/box/lua/space.cc
@@ -160,6 +160,11 @@ lbox_fillspace(struct lua_State *L, struct space *space, int i)
 	lua_pushnumber(L, space_id(space));
 	lua_settable(L, i);
 
+	/* space.group_id */
+	lua_pushstring(L, "group_id");
+	lua_pushnumber(L, space_group_id(space));
+	lua_settable(L, i);
+
 	/* space.is_temp */
 	lua_pushstring(L, "temporary");
 	lua_pushboolean(L, space_is_temporary(space));
diff --git a/src/box/lua/xlog.c b/src/box/lua/xlog.c
index 2271c829..3c7cab38 100644
--- a/src/box/lua/xlog.c
+++ b/src/box/lua/xlog.c
@@ -211,6 +211,11 @@ lbox_xlog_parser_iterate(struct lua_State *L)
 		lua_pushinteger(L, row.replica_id);
 		lua_settable(L, -3); /* replica_id */
 	}
+	if (row.group_id != 0) {
+		lbox_xlog_pushkey(L, iproto_key_name(IPROTO_GROUP_ID));
+		lua_pushinteger(L, row.group_id);
+		lua_settable(L, -3); /* group_id */
+	}
 	if (row.tm != 0) {
 		lbox_xlog_pushkey(L, iproto_key_name(IPROTO_TIMESTAMP));
 		lua_pushnumber(L, row.tm);
diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c
index 74c6be8d..c210afbe 100644
--- a/src/box/memtx_engine.c
+++ b/src/box/memtx_engine.c
@@ -555,19 +555,20 @@ checkpoint_write_row(struct xlog *l, struct xrow_header *row)
 }
 
 static int
-checkpoint_write_tuple(struct xlog *l, uint32_t space_id,
+checkpoint_write_tuple(struct xlog *l, struct space *space,
 		       const char *data, uint32_t size)
 {
 	struct request_replace_body body;
 	body.m_body = 0x82; /* map of two elements. */
 	body.k_space_id = IPROTO_SPACE_ID;
 	body.m_space_id = 0xce; /* uint32 */
-	body.v_space_id = mp_bswap_u32(space_id);
+	body.v_space_id = mp_bswap_u32(space_id(space));
 	body.k_tuple = IPROTO_TUPLE;
 
 	struct xrow_header row;
 	memset(&row, 0, sizeof(struct xrow_header));
 	row.type = IPROTO_INSERT;
+	row.group_id = space_group_id(space);
 
 	row.bodycnt = 2;
 	row.body[0].iov_base = &body;
@@ -692,8 +693,7 @@ checkpoint_f(va_list ap)
 		struct snapshot_iterator *it = entry->iterator;
 		for (data = it->next(it, &size); data != NULL;
 		     data = it->next(it, &size)) {
-			if (checkpoint_write_tuple(&snap,
-					space_id(entry->space),
+			if (checkpoint_write_tuple(&snap, entry->space,
 					data, size) != 0) {
 				xlog_close(&snap, false);
 				return -1;
diff --git a/src/box/relay.cc b/src/box/relay.cc
index c91e5aed..75c3d56a 100644
--- a/src/box/relay.cc
+++ b/src/box/relay.cc
@@ -620,7 +620,12 @@ static void
 relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row)
 {
 	struct relay *relay = container_of(stream, struct relay, stream);
-	relay_send(relay, row);
+	/*
+	 * Ignore replica local requests as we don't need to promote
+	 * vclock while sending a snapshot.
+	 */
+	if (row->group_id != GROUP_LOCAL)
+		relay_send(relay, row);
 }
 
 /** Send a single row to the client. */
@@ -630,6 +635,16 @@ relay_send_row(struct xstream *stream, struct xrow_header *packet)
 	struct relay *relay = container_of(stream, struct relay, stream);
 	assert(iproto_type_is_dml(packet->type));
 	/*
+	 * Transform replica local requests to IPROTO_NOP so as to
+	 * promote vclock on the replica without actually modifying
+	 * any data.
+	 */
+	if (packet->group_id == GROUP_LOCAL) {
+		packet->type = IPROTO_NOP;
+		packet->group_id = GROUP_DEFAULT;
+		packet->bodycnt = 0;
+	}
+	/*
 	 * We're feeding a WAL, thus responding to SUBSCRIBE request.
 	 * In that case, only send a row if it is not from the same replica
 	 * (i.e. don't send replica's own rows back) or if this row is
diff --git a/src/box/replication.h b/src/box/replication.h
index fdf995c3..6956837a 100644
--- a/src/box/replication.h
+++ b/src/box/replication.h
@@ -96,6 +96,20 @@ extern "C" {
 
 struct gc_consumer;
 
+/** Predefined replication group identifiers. */
+enum {
+	/**
+	 * Default replication group: changes made to the space
+	 * are replicated throughout the entire cluster.
+	 */
+	GROUP_DEFAULT = 0,
+	/**
+	 * Replica local space: changes made to the space are
+	 * not replicated.
+	 */
+	GROUP_LOCAL = 1,
+};
+
 static const int REPLICATION_CONNECT_QUORUM_ALL = INT_MAX;
 
 /**
diff --git a/src/box/space.h b/src/box/space.h
index 074e2462..ae32e6df 100644
--- a/src/box/space.h
+++ b/src/box/space.h
@@ -198,6 +198,13 @@ space_is_temporary(struct space *space)
 	return space->def->opts.is_temporary;
 }
 
+/** Return replication group id of a space. */
+static inline bool
+space_group_id(struct space *space)
+{
+	return space->def->opts.group_id;
+}
+
 void
 space_run_triggers(struct space *space, bool yesno);
 
diff --git a/src/box/space_def.c b/src/box/space_def.c
index ff35cb20..6243c2c4 100644
--- a/src/box/space_def.c
+++ b/src/box/space_def.c
@@ -34,10 +34,12 @@
 #include "error.h"
 
 const struct space_opts space_opts_default = {
+	/* .group_id = */ 0,
 	/* .is_temporary = */ false,
 };
 
 const struct opt_def space_opts_reg[] = {
+	OPT_DEF("group_id", OPT_UINT32, struct space_opts, group_id),
 	OPT_DEF("temporary", OPT_BOOL, struct space_opts, is_temporary),
 	OPT_END,
 };
diff --git a/src/box/space_def.h b/src/box/space_def.h
index 6cee6ad8..f66417d9 100644
--- a/src/box/space_def.h
+++ b/src/box/space_def.h
@@ -41,6 +41,11 @@ extern "C" {
 
 /** Space options */
 struct space_opts {
+	/**
+	 * Replication group identifier. Defines how changes
+	 * made to a space are replicated.
+	 */
+	uint32_t group_id;
         /**
 	 * The space is a temporary:
 	 * - it is empty at server start
diff --git a/src/box/txn.c b/src/box/txn.c
index 80e5463d..619780f3 100644
--- a/src/box/txn.c
+++ b/src/box/txn.c
@@ -61,6 +61,7 @@ txn_add_redo(struct txn_stmt *stmt, struct request *request)
 	/* Initialize members explicitly to save time on memset() */
 	row->type = request->type;
 	row->replica_id = 0;
+	row->group_id = stmt->space != NULL ? space_group_id(stmt->space) : 0;
 	row->lsn = 0;
 	row->sync = 0;
 	row->tm = 0;
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 08a83bb5..2d11fc8d 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -595,6 +595,11 @@ vinyl_engine_check_space_def(struct space_def *def)
 			 def->name, "engine does not support temporary flag");
 		return -1;
 	}
+	if (def->opts.group_id != 0) {
+		diag_set(ClientError, ER_ALTER_SPACE,
+			 def->name, "engine does not support group id");
+		return -1;
+	}
 	return 0;
 }
 
diff --git a/src/box/xrow.c b/src/box/xrow.c
index 64d845f7..11316906 100644
--- a/src/box/xrow.c
+++ b/src/box/xrow.c
@@ -97,6 +97,9 @@ error:
 		case IPROTO_REPLICA_ID:
 			header->replica_id = mp_decode_uint(pos);
 			break;
+		case IPROTO_GROUP_ID:
+			header->group_id = mp_decode_uint(pos);
+			break;
 		case IPROTO_LSN:
 			header->lsn = mp_decode_uint(pos);
 			break;
@@ -178,6 +181,12 @@ xrow_header_encode(const struct xrow_header *header, uint64_t sync,
 		map_size++;
 	}
 
+	if (header->group_id) {
+		d = mp_encode_uint(d, IPROTO_GROUP_ID);
+		d = mp_encode_uint(d, header->group_id);
+		map_size++;
+	}
+
 	if (header->lsn) {
 		d = mp_encode_uint(d, IPROTO_LSN);
 		d = mp_encode_uint(d, header->lsn);
diff --git a/src/box/xrow.h b/src/box/xrow.h
index 1bb5f103..92ea3c97 100644
--- a/src/box/xrow.h
+++ b/src/box/xrow.h
@@ -57,6 +57,7 @@ struct xrow_header {
 
 	uint32_t type;
 	uint32_t replica_id;
+	uint32_t group_id;
 	uint64_t sync;
 	int64_t lsn; /* LSN must be signed for correct comparison */
 	double tm;
diff --git a/test/box/misc.result b/test/box/misc.result
index 5c390ae8..9209c33a 100644
--- a/test/box/misc.result
+++ b/test/box/misc.result
@@ -398,8 +398,9 @@ t;
   - 'box.error.UPDATE_ARG_TYPE : 26'
   - 'box.error.CROSS_ENGINE_TRANSACTION : 81'
   - 'box.error.FORMAT_MISMATCH_INDEX_PART : 27'
-  - 'box.error.FUNCTION_TX_ACTIVE : 30'
   - 'box.error.injection : table: <address>
+  - 'box.error.FUNCTION_TX_ACTIVE : 30'
+  - 'box.error.ITERATOR_TYPE : 72'
   - 'box.error.TRANSACTION_YIELD : 154'
   - 'box.error.NO_SUCH_ENGINE : 57'
   - 'box.error.COMMIT_IN_SUB_STMT : 122'
@@ -449,7 +450,7 @@ t;
   - 'box.error.COMPRESSION : 119'
   - 'box.error.INVALID_ORDER : 68'
   - 'box.error.UNKNOWN : 0'
-  - 'box.error.ITERATOR_TYPE : 72'
+  - 'box.error.NO_SUCH_GROUP : 155'
   - 'box.error.TUPLE_FORMAT_LIMIT : 16'
   - 'box.error.DROP_PRIMARY_KEY : 17'
   - 'box.error.NULLABLE_PRIMARY : 152'
diff --git a/test/engine/iterator.result b/test/engine/iterator.result
index 05d892df..10097edb 100644
--- a/test/engine/iterator.result
+++ b/test/engine/iterator.result
@@ -4213,7 +4213,7 @@ s:replace{35}
 ...
 state, value = gen(param,state)
 ---
-- error: 'builtin/box/schema.lua:1032: usage: next(param, state)'
+- error: 'builtin/box/schema.lua:1034: usage: next(param, state)'
 ...
 value
 ---
diff --git a/test/replication/local_spaces.result b/test/replication/local_spaces.result
new file mode 100644
index 00000000..0eb5a2f6
--- /dev/null
+++ b/test/replication/local_spaces.result
@@ -0,0 +1,174 @@
+env = require('test_run')
+---
+...
+test_run = env.new()
+---
+...
+--
+-- gh-3443: Check that changes done to spaces marked as local
+-- are not replicated, but vclock is still promoted.
+--
+-- Currently, there are only two replication groups:
+-- 0 (global) and 1 (local)
+box.schema.space.create('test', {group_id = 2}) -- error
+---
+- error: Replication group '2' does not exist
+...
+s1 = box.schema.space.create('test1')
+---
+...
+_ = s1:create_index('pk')
+---
+...
+s2 = box.schema.space.create('test2', {group_id = 1})
+---
+...
+_ = s2:create_index('pk')
+---
+...
+s1.group_id
+---
+- 0
+...
+s2.group_id
+---
+- 1
+...
+-- group_id is immutable
+box.space._space:update(s1.id, {{'=', 6, {group_id = 1}}}) -- error
+---
+- error: 'Can''t modify space ''test1'': group id is immutable'
+...
+box.space._space:update(s2.id, {{'=', 6, {group_id = 0}}}) -- error
+---
+- error: 'Can''t modify space ''test2'': group id is immutable'
+...
+_ = s1:insert{1}
+---
+...
+_ = s2:insert{1}
+---
+...
+box.snapshot()
+---
+- ok
+...
+_ = s1:insert{2}
+---
+...
+_ = s2:insert{2}
+---
+...
+box.schema.user.grant('guest', 'replication')
+---
+...
+test_run:cmd("create server replica with rpl_master=default, script='replication/replica.lua'")
+---
+- true
+...
+test_run:cmd("start server replica")
+---
+- true
+...
+test_run:cmd("switch replica")
+---
+- true
+...
+box.space.test1.group_id
+---
+- 0
+...
+box.space.test2.group_id
+---
+- 1
+...
+box.space.test1:select()
+---
+- - [1]
+  - [2]
+...
+box.space.test2:select()
+---
+- []
+...
+for i = 1, 3 do box.space.test2:insert{i, i} end
+---
+...
+test_run:cmd("switch default")
+---
+- true
+...
+_ = s1:insert{3}
+---
+...
+_ = s2:insert{3}
+---
+...
+vclock = test_run:get_vclock('default')
+---
+...
+_ = test_run:wait_vclock('replica', vclock)
+---
+...
+test_run:cmd("switch replica")
+---
+- true
+...
+box.space.test1:select()
+---
+- - [1]
+  - [2]
+  - [3]
+...
+box.space.test2:select()
+---
+- - [1, 1]
+  - [2, 2]
+  - [3, 3]
+...
+test_run:cmd("restart server replica")
+box.space.test1:select()
+---
+- - [1]
+  - [2]
+  - [3]
+...
+box.space.test2:select()
+---
+- - [1, 1]
+  - [2, 2]
+  - [3, 3]
+...
+test_run:cmd("switch default")
+---
+- true
+...
+test_run:cmd("stop server replica")
+---
+- true
+...
+test_run:cmd("cleanup server replica")
+---
+- true
+...
+box.schema.user.revoke('guest', 'replication')
+---
+...
+s1:select()
+---
+- - [1]
+  - [2]
+  - [3]
+...
+s2:select()
+---
+- - [1]
+  - [2]
+  - [3]
+...
+s1:drop()
+---
+...
+s2:drop()
+---
+...
diff --git a/test/replication/local_spaces.test.lua b/test/replication/local_spaces.test.lua
new file mode 100644
index 00000000..700c1413
--- /dev/null
+++ b/test/replication/local_spaces.test.lua
@@ -0,0 +1,63 @@
+env = require('test_run')
+test_run = env.new()
+
+--
+-- gh-3443: Check that changes done to spaces marked as local
+-- are not replicated, but vclock is still promoted.
+--
+
+-- Currently, there are only two replication groups:
+-- 0 (global) and 1 (local)
+box.schema.space.create('test', {group_id = 2}) -- error
+
+s1 = box.schema.space.create('test1')
+_ = s1:create_index('pk')
+s2 = box.schema.space.create('test2', {group_id = 1})
+_ = s2:create_index('pk')
+s1.group_id
+s2.group_id
+
+-- group_id is immutable
+box.space._space:update(s1.id, {{'=', 6, {group_id = 1}}}) -- error
+box.space._space:update(s2.id, {{'=', 6, {group_id = 0}}}) -- error
+
+_ = s1:insert{1}
+_ = s2:insert{1}
+box.snapshot()
+_ = s1:insert{2}
+_ = s2:insert{2}
+
+box.schema.user.grant('guest', 'replication')
+test_run:cmd("create server replica with rpl_master=default, script='replication/replica.lua'")
+test_run:cmd("start server replica")
+
+test_run:cmd("switch replica")
+box.space.test1.group_id
+box.space.test2.group_id
+box.space.test1:select()
+box.space.test2:select()
+for i = 1, 3 do box.space.test2:insert{i, i} end
+
+test_run:cmd("switch default")
+_ = s1:insert{3}
+_ = s2:insert{3}
+vclock = test_run:get_vclock('default')
+_ = test_run:wait_vclock('replica', vclock)
+
+test_run:cmd("switch replica")
+box.space.test1:select()
+box.space.test2:select()
+test_run:cmd("restart server replica")
+box.space.test1:select()
+box.space.test2:select()
+
+test_run:cmd("switch default")
+test_run:cmd("stop server replica")
+test_run:cmd("cleanup server replica")
+box.schema.user.revoke('guest', 'replication')
+
+s1:select()
+s2:select()
+
+s1:drop()
+s2:drop()
diff --git a/test/replication/suite.cfg b/test/replication/suite.cfg
index 95e94e5a..283edcad 100644
--- a/test/replication/suite.cfg
+++ b/test/replication/suite.cfg
@@ -6,6 +6,7 @@
     "wal_off.test.lua": {},
     "hot_standby.test.lua": {},
     "rebootstrap.test.lua": {},
+    "local_spaces.test.lua": {},
     "*": {
         "memtx": {"engine": "memtx"},
         "vinyl": {"engine": "vinyl"}
diff --git a/test/vinyl/ddl.result b/test/vinyl/ddl.result
index 16ee7097..5f385123 100644
--- a/test/vinyl/ddl.result
+++ b/test/vinyl/ddl.result
@@ -44,6 +44,11 @@ space:create_index('pk', {bloom_fpr = 1.1})
 space:drop()
 ---
 ...
+-- vinyl does not support replica local spaces
+space = box.schema.space.create('test', {engine = 'vinyl', group_id = 1})
+---
+- error: 'Can''t modify space ''test'': engine does not support group id'
+...
 -- space secondary index create
 space = box.schema.space.create('test', { engine = 'vinyl' })
 ---
diff --git a/test/vinyl/ddl.test.lua b/test/vinyl/ddl.test.lua
index 95dd5a11..3894f003 100644
--- a/test/vinyl/ddl.test.lua
+++ b/test/vinyl/ddl.test.lua
@@ -12,6 +12,9 @@ space:create_index('pk', {bloom_fpr = 0})
 space:create_index('pk', {bloom_fpr = 1.1})
 space:drop()
 
+-- vinyl does not support replica local spaces
+space = box.schema.space.create('test', {engine = 'vinyl', group_id = 1})
+
 -- space secondary index create
 space = box.schema.space.create('test', { engine = 'vinyl' })
 index1 = space:create_index('primary')
-- 
2.11.0

Re: [tarantool-patches] [PATCH v2 4/4] Introduce replica local spaces

[Vladislav Shpilevoy]

On 09/07/2018 18:40, Vladimir Davydov wrote:
> This patch introduces a new space option, group_id, which defines how
> the space is replicated. If it is 0 (default), the space is replicated
> throughout the entire cluster. If it is 1, the space is replica local,
> i.e. all changes made to it are invisible to other replicas in the
> cluster. Currently, no other value is permitted, but in future we will
> use this option for setting up arbitrary replication groups in a
> cluster. The option can only be set on space creation and cannot be
> altered.
> 
> Technically, to support this feature, we introduce a new header key,
> IPROTO_GROUP_ID, which is set to the space group id for all rows
> corresponding to a space, both in xlog and in snap. Relay won't send
> snapshot rows whose group_id is 1. As for xlog rows, they are
> transformed to IPROTO_NOP so as to promote vclock on replicas without
> any actual data modification.
> 
> The feature is currently supported for memtx spaces only, but it should
> be easy to implement it for vinyl spaces as well.
> 
> @TarantoolBot document
> Title: Document new space option - group_id
> Group identifier defines how the space is replicated. Currently, only
> two values are permitted: 0 and 1. If space.group_id is 0, changes made
> to the space are replicated throughout the entire cluster. If it is 1,
> then the space is replica local, i.e. changes made to it are invisible
> to other replicas in the cluster.

Please, put the doc request after 'Closes'. The request must be at the
very end of the commit message.

> 
> Closes #3443
> ---

Re: [PATCH v2 1/4] txn: do not require space id for nop requests

[Konstantin Osipov]

* Vladimir Davydov <vdavydov.dev@gmail.com> [18/07/09 18:45]:
> Currently, IPROTO_NOP can only be generated by a before_replace trigger,
> when it returns the old tuple thus turning the original operation into a
> NOP. In such a case we know the space id and we write it to the request
> body. This allows us to dispatch NOP requests via DML route.
> 
> As a part of replica local spaces feature, we will substitute requests
> operating on local spaces with NOP in relay in order to promote vclock
> on replicas without actual data modification. Since space_id is stored
> in request body, sending it to replicas would mean decoding the request
> body in relay, which is an overkill. To avoid that, let's separate NOP
> and DML paths and remove space_id from NOP requests.
> 

I pushed this patch set.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov