From: Vladimir Davydov <vdavydov.dev@gmail.com> To: tarantool-patches@freelists.org Subject: [PATCH v2 0/7] Join replicas off the current read view Date: Mon, 19 Aug 2019 19:53:13 +0300 [thread overview] Message-ID: <cover.1566233187.git.vdavydov.dev@gmail.com> (raw) Currently, we join replicas off the last checkpoint. As a result, we must keep all files corresponding to the last checkpoint. This means that we must always create a memtx snapshot file on initial call to box.cfg() even though it is virtually the same for all instances. Besides, we must rotate the vylog file synchronously with snapshot creation, otherwise we wouldn't be able to pull all vinyl files corresponding to the last checkpoint. This interconnection between vylog and xlog makes the code difficult to maintain. Actually, nothing prevents us from relaying the current read view instead of the last checkpoint on initial join, as both memtx and vinyl support a consistent read view. This patch does the trick. This is a step towards making vylog independent of checkpointing and WAL. https://github.com/tarantool/tarantool/issues/1271 https://github.com/tarantool/tarantool/issues/4417 https://github.com/tarantool/tarantool/commits/dv/gh-1271-rework-replica-join Changes in v2: - Commit preparatory patches approved by Kostja and rebase on the latest master branch. - Fix the issue with box.on_schema_init and space.before_replace instead of disabling the test (see #4417). Vladimir Davydov (7): vinyl: don't pin index for iterator lifetime vinyl: don't exempt dropped indexes from dump and compaction vinyl: get rid of vy_env::join_lsn memtx: use ref counting to pin indexes for snapshot memtx: enter small delayed free mode from snapshot iterator space: get rid of apply_initial_join_row method relay: join new replicas off read view src/box/applier.cc | 32 +- src/box/blackhole.c | 4 +- src/box/box.cc | 33 +- src/box/engine.c | 21 - src/box/engine.h | 27 +- src/box/memtx_engine.c | 146 ++---- src/box/memtx_engine.h | 28 +- src/box/memtx_hash.c | 18 +- src/box/memtx_space.c | 30 -- src/box/memtx_tree.c | 19 +- src/box/relay.cc | 170 ++++++- src/box/relay.h | 2 +- src/box/space.c | 9 - src/box/space.h | 16 - src/box/sysview.c | 4 +- src/box/vinyl.c | 526 ++++---------------- src/box/vy_lsm.c | 4 + src/box/vy_lsm.h | 7 + src/box/vy_run.c | 6 - src/box/vy_scheduler.c | 95 ++-- src/box/vy_scheduler.h | 10 +- src/box/vy_tx.c | 12 +- src/box/vy_tx.h | 8 + src/box/vy_write_iterator.c | 8 +- src/lib/core/errinj.h | 2 +- test/box/errinj.result | 38 +- test/replication-py/cluster.result | 13 - test/replication-py/cluster.test.py | 25 - test/replication/join_without_snap.result | 88 ++++ test/replication/join_without_snap.test.lua | 32 ++ test/replication/on_schema_init.result | 6 + test/replication/on_schema_init.test.lua | 3 + test/replication/suite.cfg | 1 + test/vinyl/errinj.result | 4 +- test/vinyl/errinj.test.lua | 4 +- test/xlog/panic_on_broken_lsn.result | 31 +- test/xlog/panic_on_broken_lsn.test.lua | 20 +- 37 files changed, 610 insertions(+), 892 deletions(-) create mode 100644 test/replication/join_without_snap.result create mode 100644 test/replication/join_without_snap.test.lua -- 2.20.1
next reply other threads:[~2019-08-19 16:53 UTC|newest] Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-08-19 16:53 Vladimir Davydov [this message] 2019-08-19 16:53 ` [PATCH v2 1/7] vinyl: don't pin index for iterator lifetime Vladimir Davydov 2019-08-19 20:35 ` [tarantool-patches] " Konstantin Osipov 2019-08-19 16:53 ` [PATCH v2 2/7] vinyl: don't exempt dropped indexes from dump and compaction Vladimir Davydov 2019-08-19 20:47 ` [tarantool-patches] " Konstantin Osipov 2019-08-20 8:12 ` Vladimir Davydov 2019-08-20 9:02 ` Vladimir Davydov 2019-08-20 11:52 ` Konstantin Osipov 2019-08-20 14:16 ` Vladimir Davydov 2019-08-19 16:53 ` [PATCH v2 3/7] vinyl: get rid of vy_env::join_lsn Vladimir Davydov 2019-08-19 20:49 ` [tarantool-patches] " Konstantin Osipov 2019-08-19 16:53 ` [PATCH v2 4/7] memtx: use ref counting to pin indexes for snapshot Vladimir Davydov 2019-08-19 20:50 ` [tarantool-patches] " Konstantin Osipov 2019-08-19 16:53 ` [PATCH v2 5/7] memtx: enter small delayed free mode from snapshot iterator Vladimir Davydov 2019-08-19 20:51 ` [tarantool-patches] " Konstantin Osipov 2019-08-19 16:53 ` [PATCH v2 6/7] space: get rid of apply_initial_join_row method Vladimir Davydov 2019-08-19 20:54 ` [tarantool-patches] " Konstantin Osipov 2019-08-19 16:53 ` [PATCH v2 7/7] relay: join new replicas off read view Vladimir Davydov 2019-08-19 20:57 ` [tarantool-patches] " Konstantin Osipov 2019-08-20 8:16 ` Vladimir Davydov 2019-08-20 11:53 ` Konstantin Osipov 2019-08-20 12:05 ` Vladimir Davydov 2019-08-20 13:50 ` Konstantin Osipov 2019-08-20 14:03 ` Vladimir Davydov 2019-08-21 22:08 ` Konstantin Osipov 2019-08-22 8:05 ` Vladimir Davydov 2019-08-19 16:54 ` [PATCH v2 0/7] Join replicas off the current " Vladimir Davydov 2019-08-20 8:53 ` Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=cover.1566233187.git.vdavydov.dev@gmail.com \ --to=vdavydov.dev@gmail.com \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH v2 0/7] Join replicas off the current read view' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox