From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: tarantool-patches@freelists.org
Subject: [PATCH 00/13] Join replicas off the current read view
Date: Sat, 10 Aug 2019 13:03:27 +0300 [thread overview]
Message-ID: <cover.1565430177.git.vdavydov.dev@gmail.com> (raw)
Currently, we join replicas off the last checkpoint. As a result, we
must keep all files corresponding to the last checkpoint. This means
that we must always create a memtx snapshot file on initial call to
box.cfg() even though it is virtually the same for all instances.
Besides, we must rotate the vylog file synchronously with snapshot
creation, otherwise we wouldn't be able to pull all vinyl files
corresponding to the last checkpoint. This interconnection between
vylog and xlog makes the code difficult to maintain.
Actually, nothing prevents us from relaying the current read view
instead of the last checkpoint on initial join, as both memtx and
vinyl support a consistent read view. This patch does the trick.
This is a step towards making vylog independent of checkpointing
and WAL.
https://github.com/tarantool/tarantool/issues/1271
https://github.com/tarantool/tarantool/commits/dv/gh-1271-rework-replica-join
Vladimir Davydov (13):
vinyl: embed engine in vy_env
vinyl: embed index in vy_lsm
vinyl: move reference counting from vy_lsm to index
vinyl: don't pin index for iterator lifetime
vinyl: don't exempt dropped indexes from dump and compaction
memtx: don't store pointers to index internals in iterator
memtx: use ref counting to pin indexes for snapshot
memtx: allow snapshot iterator to fail
memtx: enter small delayed free mode from snapshot iterator
wal: make wal_sync fail on write error
xrow: factor out helper for setting REPLACE request body
test: disable replication/on_schema_init
relay: join new replicas off read view
src/box/blackhole.c | 3 +-
src/box/box.cc | 53 +-
src/box/engine.c | 21 -
src/box/engine.h | 27 +-
src/box/index.cc | 2 +
src/box/index.h | 29 +-
src/box/iproto_constants.h | 11 +
src/box/lua/info.c | 3 +-
src/box/lua/stat.c | 3 +-
src/box/memtx_bitset.c | 10 +-
src/box/memtx_engine.c | 152 +----
src/box/memtx_engine.h | 28 +-
src/box/memtx_hash.c | 52 +-
src/box/memtx_tree.c | 100 ++--
src/box/relay.cc | 170 +++++-
src/box/relay.h | 2 +-
src/box/sequence.c | 24 +-
src/box/space.c | 4 +-
src/box/sysview.c | 3 +-
src/box/vinyl.c | 596 ++++++--------------
src/box/vinyl.h | 26 +-
src/box/vy_lsm.c | 6 +-
src/box/vy_lsm.h | 22 +-
src/box/vy_run.c | 6 -
src/box/vy_scheduler.c | 95 ++--
src/box/vy_scheduler.h | 10 +-
src/box/vy_tx.c | 10 +-
src/box/vy_tx.h | 8 +
src/box/wal.c | 29 +-
src/box/wal.h | 5 +-
src/lib/core/errinj.h | 2 +-
test/box/errinj.result | 38 +-
test/replication-py/cluster.result | 13 -
test/replication-py/cluster.test.py | 25 -
test/replication/join_without_snap.result | 88 +++
test/replication/join_without_snap.test.lua | 32 ++
test/replication/suite.cfg | 1 +
test/replication/suite.ini | 2 +-
test/unit/vy_point_lookup.c | 3 +-
test/vinyl/errinj.result | 4 +-
test/vinyl/errinj.test.lua | 4 +-
test/xlog/panic_on_broken_lsn.result | 31 +-
test/xlog/panic_on_broken_lsn.test.lua | 20 +-
43 files changed, 826 insertions(+), 947 deletions(-)
create mode 100644 test/replication/join_without_snap.result
create mode 100644 test/replication/join_without_snap.test.lua
--
2.20.1
next reply other threads:[~2019-08-10 10:03 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-10 10:03 Vladimir Davydov [this message]
2019-08-10 10:03 ` [PATCH 01/13] vinyl: embed engine in vy_env Vladimir Davydov
2019-08-12 22:14 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:09 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 02/13] vinyl: embed index in vy_lsm Vladimir Davydov
2019-08-12 22:14 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:09 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 03/13] vinyl: move reference counting from vy_lsm to index Vladimir Davydov
2019-08-12 22:16 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:09 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 04/13] vinyl: don't pin index for iterator lifetime Vladimir Davydov
2019-08-10 10:03 ` [PATCH 05/13] vinyl: don't exempt dropped indexes from dump and compaction Vladimir Davydov
2019-08-10 10:03 ` [PATCH 06/13] memtx: don't store pointers to index internals in iterator Vladimir Davydov
2019-08-12 22:21 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:10 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 07/13] memtx: use ref counting to pin indexes for snapshot Vladimir Davydov
2019-08-12 22:24 ` [tarantool-patches] " Konstantin Osipov
2019-08-13 10:56 ` Vladimir Davydov
2019-08-13 16:08 ` Georgy Kirichenko
2019-08-10 10:03 ` [PATCH 08/13] memtx: allow snapshot iterator to fail Vladimir Davydov
2019-08-12 22:25 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:10 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 09/13] memtx: enter small delayed free mode from snapshot iterator Vladimir Davydov
2019-08-12 22:27 ` [tarantool-patches] " Konstantin Osipov
2019-08-13 10:59 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 10/13] wal: make wal_sync fail on write error Vladimir Davydov
2019-08-12 22:29 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 16:48 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 11/13] xrow: factor out helper for setting REPLACE request body Vladimir Davydov
2019-08-12 22:29 ` [tarantool-patches] " Konstantin Osipov
2019-08-14 13:11 ` Vladimir Davydov
2019-08-10 10:03 ` [PATCH 12/13] test: disable replication/on_schema_init Vladimir Davydov
2019-08-12 22:31 ` [tarantool-patches] " Konstantin Osipov
2019-08-10 10:03 ` [PATCH 13/13] relay: join new replicas off read view Vladimir Davydov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1565430177.git.vdavydov.dev@gmail.com \
--to=vdavydov.dev@gmail.com \
--cc=tarantool-patches@freelists.org \
--subject='Re: [PATCH 00/13] Join replicas off the current read view' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox