[PATCH 00/25] vinyl: eliminate disk read on REPLACE/DELETE

[Vladimir Davydov <vdavydov.dev@gmail.com>]

This patch set optimizes REPLACE and DELETE operations in vinyl in
presence of secondary indexes: now they don't need to read the primary
key in order to delete the overwritten/deleted tuple from secondary
indexes, instead this job is handed over to primary index compaction
task, while the read iterator is tought to filter out overwritten
tuples that haven't been purged yet.

The primary difference of this patch set from the RFC version, which can
be found by the link

https://www.freelists.org/post/tarantool-patches/RFC-PATCH-0023-vinyl-eliminate-read-on-REPLACEDELETE

is that it resolves all the problems the RFC suffered from, namely
potential OOM during deferred DELETE generation, lost DELETEs on
restart, severe performance degradation in case a lot of tuples get
overwritten in memory, missing functional tests.

For more information about the algorithm and implementation details,
please see the final patch of the series.

https://github.com/tarantool/tarantool/issues/2129
https://github.com/tarantool/tarantool/commits/dv/gh-2129-vy-eliminate-read-on-replace-delete

Vladimir Davydov (25):
  vinyl: make point lookup always return the latest tuple version
  vinyl: simplify vy_squash_process
  vinyl: always get full tuple from pk after reading from secondary
    index
  vinyl: fold vy_replace_one and vy_replace_impl
  vinyl: fold vy_delete_impl
  vinyl: refactor unique check
  vinyl: check key uniqueness before modifying tx write set
  vinyl: remove env argument of vy_check_is_unique_{primary,secondary}
  vinyl: store full tuples in secondary index cache
  vinyl: do not free pending tasks on shutdown
  vinyl: store pointer to scheduler in struct vy_task
  vinyl: rename some members of vy_scheduler and vy_task struct
  vinyl: use cbus for communication between scheduler and worker threads
  vinyl: zap vy_scheduler::is_worker_pool_running
  vinyl: rename vy_task::status to is_failed
  xrow: allow to store flags in DML requests
  vinyl: pin last statement returned by write iterator explicitly
  vinyl: teach write iterator to return overwritten tuples
  vinyl: prepare write iterator heap comparator for deferred DELETEs
  vinyl: allow to skip certain statements on read
  vinyl: add function to create surrogate deletes from raw msgpack
  vinyl: remove pointless assertion from vy_stmt_new_surrogate_delete
  txn: add helper to detect transaction boundaries
  Introduce _vinyl_deferred_delete system space
  vinyl: eliminate disk read on REPLACE/DELETE

 src/box/bootstrap.snap              |  Bin 1540 -> 1605 bytes
 src/box/iproto_constants.c          |    4 +-
 src/box/iproto_constants.h          |    3 +-
 src/box/lua/space.cc                |    2 +
 src/box/lua/upgrade.lua             |   21 +
 src/box/schema.cc                   |   72 ++-
 src/box/schema_def.h                |    2 +
 src/box/txn.c                       |    3 +-
 src/box/txn.h                       |   11 +
 src/box/vinyl.c                     | 1019 ++++++++++++++++++-----------------
 src/box/vy_lsm.h                    |    5 +
 src/box/vy_mem.c                    |   20 +-
 src/box/vy_mem.h                    |    5 +
 src/box/vy_point_lookup.c           |   72 ++-
 src/box/vy_point_lookup.h           |   27 +-
 src/box/vy_read_iterator.c          |   61 +--
 src/box/vy_read_iterator.h          |   24 +
 src/box/vy_run.c                    |    7 +-
 src/box/vy_scheduler.c              |  685 +++++++++++++++++------
 src/box/vy_scheduler.h              |   33 +-
 src/box/vy_stmt.c                   |   12 +-
 src/box/vy_stmt.h                   |   56 +-
 src/box/vy_tx.c                     |  133 +++++
 src/box/vy_write_iterator.c         |  168 +++++-
 src/box/vy_write_iterator.h         |   27 +-
 src/box/xrow.c                      |    8 +
 src/box/xrow.h                      |    2 +
 test/app-tap/tarantoolctl.test.lua  |    2 +-
 test/box-py/bootstrap.result        |    5 +-
 test/box/access_misc.result         |    3 +
 test/box/access_sysview.result      |    2 +-
 test/box/alter.result               |    4 +-
 test/unit/vy_iterators_helper.c     |    5 +
 test/unit/vy_iterators_helper.h     |   12 +-
 test/unit/vy_point_lookup.c         |    6 +-
 test/unit/vy_write_iterator.c       |  232 +++++++-
 test/unit/vy_write_iterator.result  |   22 +-
 test/vinyl/deferred_delete.result   |  677 +++++++++++++++++++++++
 test/vinyl/deferred_delete.test.lua |  261 +++++++++
 test/vinyl/info.result              |   18 +-
 test/vinyl/info.test.lua            |    9 +-
 test/vinyl/layout.result            |   46 +-
 test/vinyl/quota.result             |    2 +-
 test/vinyl/tx_gap_lock.result       |   16 +-
 test/vinyl/tx_gap_lock.test.lua     |   10 +-
 test/vinyl/write_iterator.result    |    5 +
 test/vinyl/write_iterator.test.lua  |    3 +
 test/wal_off/alter.result           |    2 +-
 test/xlog/upgrade.result            |    5 +-
 49 files changed, 2953 insertions(+), 876 deletions(-)
 create mode 100644 test/vinyl/deferred_delete.result
 create mode 100644 test/vinyl/deferred_delete.test.lua

-- 
2.11.0