Tarantool development patches archive
 help / color / mirror / Atom feed
* [PATCH v2 0/7] Join replicas off the current read view
@ 2019-08-19 16:53 Vladimir Davydov
  2019-08-19 16:53 ` [PATCH v2 1/7] vinyl: don't pin index for iterator lifetime Vladimir Davydov
                   ` (8 more replies)
  0 siblings, 9 replies; 28+ messages in thread
From: Vladimir Davydov @ 2019-08-19 16:53 UTC (permalink / raw)
  To: tarantool-patches

Currently, we join replicas off the last checkpoint. As a result, we
must keep all files corresponding to the last checkpoint. This means
that we must always create a memtx snapshot file on initial call to
box.cfg() even though it is virtually the same for all instances.
Besides, we must rotate the vylog file synchronously with snapshot
creation, otherwise we wouldn't be able to pull all vinyl files
corresponding to the last checkpoint. This interconnection between
vylog and xlog makes the code difficult to maintain.

Actually, nothing prevents us from relaying the current read view
instead of the last checkpoint on initial join, as both memtx and
vinyl support a consistent read view. This patch does the trick.
This is a step towards making vylog independent of checkpointing
and WAL.

https://github.com/tarantool/tarantool/issues/1271
https://github.com/tarantool/tarantool/issues/4417
https://github.com/tarantool/tarantool/commits/dv/gh-1271-rework-replica-join

Changes in v2:
 - Commit preparatory patches approved by Kostja and rebase on
   the latest master branch.
 - Fix the issue with box.on_schema_init and space.before_replace
   instead of disabling the test (see #4417).

Vladimir Davydov (7):
  vinyl: don't pin index for iterator lifetime
  vinyl: don't exempt dropped indexes from dump and compaction
  vinyl: get rid of vy_env::join_lsn
  memtx: use ref counting to pin indexes for snapshot
  memtx: enter small delayed free mode from snapshot iterator
  space: get rid of apply_initial_join_row method
  relay: join new replicas off read view

 src/box/applier.cc                          |  32 +-
 src/box/blackhole.c                         |   4 +-
 src/box/box.cc                              |  33 +-
 src/box/engine.c                            |  21 -
 src/box/engine.h                            |  27 +-
 src/box/memtx_engine.c                      | 146 ++----
 src/box/memtx_engine.h                      |  28 +-
 src/box/memtx_hash.c                        |  18 +-
 src/box/memtx_space.c                       |  30 --
 src/box/memtx_tree.c                        |  19 +-
 src/box/relay.cc                            | 170 ++++++-
 src/box/relay.h                             |   2 +-
 src/box/space.c                             |   9 -
 src/box/space.h                             |  16 -
 src/box/sysview.c                           |   4 +-
 src/box/vinyl.c                             | 526 ++++----------------
 src/box/vy_lsm.c                            |   4 +
 src/box/vy_lsm.h                            |   7 +
 src/box/vy_run.c                            |   6 -
 src/box/vy_scheduler.c                      |  95 ++--
 src/box/vy_scheduler.h                      |  10 +-
 src/box/vy_tx.c                             |  12 +-
 src/box/vy_tx.h                             |   8 +
 src/box/vy_write_iterator.c                 |   8 +-
 src/lib/core/errinj.h                       |   2 +-
 test/box/errinj.result                      |  38 +-
 test/replication-py/cluster.result          |  13 -
 test/replication-py/cluster.test.py         |  25 -
 test/replication/join_without_snap.result   |  88 ++++
 test/replication/join_without_snap.test.lua |  32 ++
 test/replication/on_schema_init.result      |   6 +
 test/replication/on_schema_init.test.lua    |   3 +
 test/replication/suite.cfg                  |   1 +
 test/vinyl/errinj.result                    |   4 +-
 test/vinyl/errinj.test.lua                  |   4 +-
 test/xlog/panic_on_broken_lsn.result        |  31 +-
 test/xlog/panic_on_broken_lsn.test.lua      |  20 +-
 37 files changed, 610 insertions(+), 892 deletions(-)
 create mode 100644 test/replication/join_without_snap.result
 create mode 100644 test/replication/join_without_snap.test.lua

-- 
2.20.1

^ permalink raw reply	[flat|nested] 28+ messages in thread

end of thread, other threads:[~2019-08-22  8:05 UTC | newest]

Thread overview: 28+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-19 16:53 [PATCH v2 0/7] Join replicas off the current read view Vladimir Davydov
2019-08-19 16:53 ` [PATCH v2 1/7] vinyl: don't pin index for iterator lifetime Vladimir Davydov
2019-08-19 20:35   ` [tarantool-patches] " Konstantin Osipov
2019-08-19 16:53 ` [PATCH v2 2/7] vinyl: don't exempt dropped indexes from dump and compaction Vladimir Davydov
2019-08-19 20:47   ` [tarantool-patches] " Konstantin Osipov
2019-08-20  8:12     ` Vladimir Davydov
2019-08-20  9:02       ` Vladimir Davydov
2019-08-20 11:52       ` Konstantin Osipov
2019-08-20 14:16   ` Vladimir Davydov
2019-08-19 16:53 ` [PATCH v2 3/7] vinyl: get rid of vy_env::join_lsn Vladimir Davydov
2019-08-19 20:49   ` [tarantool-patches] " Konstantin Osipov
2019-08-19 16:53 ` [PATCH v2 4/7] memtx: use ref counting to pin indexes for snapshot Vladimir Davydov
2019-08-19 20:50   ` [tarantool-patches] " Konstantin Osipov
2019-08-19 16:53 ` [PATCH v2 5/7] memtx: enter small delayed free mode from snapshot iterator Vladimir Davydov
2019-08-19 20:51   ` [tarantool-patches] " Konstantin Osipov
2019-08-19 16:53 ` [PATCH v2 6/7] space: get rid of apply_initial_join_row method Vladimir Davydov
2019-08-19 20:54   ` [tarantool-patches] " Konstantin Osipov
2019-08-19 16:53 ` [PATCH v2 7/7] relay: join new replicas off read view Vladimir Davydov
2019-08-19 20:57   ` [tarantool-patches] " Konstantin Osipov
2019-08-20  8:16     ` Vladimir Davydov
2019-08-20 11:53       ` Konstantin Osipov
2019-08-20 12:05         ` Vladimir Davydov
2019-08-20 13:50           ` Konstantin Osipov
2019-08-20 14:03             ` Vladimir Davydov
2019-08-21 22:08               ` Konstantin Osipov
2019-08-22  8:05                 ` Vladimir Davydov
2019-08-19 16:54 ` [PATCH v2 0/7] Join replicas off the current " Vladimir Davydov
2019-08-20  8:53 ` Vladimir Davydov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox