From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH 00/25] vinyl: eliminate disk read on REPLACE/DELETE Date: Fri, 27 Jul 2018 14:29:40 +0300 Message-Id: To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: This patch set optimizes REPLACE and DELETE operations in vinyl in presence of secondary indexes: now they don't need to read the primary key in order to delete the overwritten/deleted tuple from secondary indexes, instead this job is handed over to primary index compaction task, while the read iterator is tought to filter out overwritten tuples that haven't been purged yet. The primary difference of this patch set from the RFC version, which can be found by the link https://www.freelists.org/post/tarantool-patches/RFC-PATCH-0023-vinyl-eliminate-read-on-REPLACEDELETE is that it resolves all the problems the RFC suffered from, namely potential OOM during deferred DELETE generation, lost DELETEs on restart, severe performance degradation in case a lot of tuples get overwritten in memory, missing functional tests. For more information about the algorithm and implementation details, please see the final patch of the series. https://github.com/tarantool/tarantool/issues/2129 https://github.com/tarantool/tarantool/commits/dv/gh-2129-vy-eliminate-read-on-replace-delete Vladimir Davydov (25): vinyl: make point lookup always return the latest tuple version vinyl: simplify vy_squash_process vinyl: always get full tuple from pk after reading from secondary index vinyl: fold vy_replace_one and vy_replace_impl vinyl: fold vy_delete_impl vinyl: refactor unique check vinyl: check key uniqueness before modifying tx write set vinyl: remove env argument of vy_check_is_unique_{primary,secondary} vinyl: store full tuples in secondary index cache vinyl: do not free pending tasks on shutdown vinyl: store pointer to scheduler in struct vy_task vinyl: rename some members of vy_scheduler and vy_task struct vinyl: use cbus for communication between scheduler and worker threads vinyl: zap vy_scheduler::is_worker_pool_running vinyl: rename vy_task::status to is_failed xrow: allow to store flags in DML requests vinyl: pin last statement returned by write iterator explicitly vinyl: teach write iterator to return overwritten tuples vinyl: prepare write iterator heap comparator for deferred DELETEs vinyl: allow to skip certain statements on read vinyl: add function to create surrogate deletes from raw msgpack vinyl: remove pointless assertion from vy_stmt_new_surrogate_delete txn: add helper to detect transaction boundaries Introduce _vinyl_deferred_delete system space vinyl: eliminate disk read on REPLACE/DELETE src/box/bootstrap.snap | Bin 1540 -> 1605 bytes src/box/iproto_constants.c | 4 +- src/box/iproto_constants.h | 3 +- src/box/lua/space.cc | 2 + src/box/lua/upgrade.lua | 21 + src/box/schema.cc | 72 ++- src/box/schema_def.h | 2 + src/box/txn.c | 3 +- src/box/txn.h | 11 + src/box/vinyl.c | 1019 ++++++++++++++++++----------------- src/box/vy_lsm.h | 5 + src/box/vy_mem.c | 20 +- src/box/vy_mem.h | 5 + src/box/vy_point_lookup.c | 72 ++- src/box/vy_point_lookup.h | 27 +- src/box/vy_read_iterator.c | 61 +-- src/box/vy_read_iterator.h | 24 + src/box/vy_run.c | 7 +- src/box/vy_scheduler.c | 685 +++++++++++++++++------ src/box/vy_scheduler.h | 33 +- src/box/vy_stmt.c | 12 +- src/box/vy_stmt.h | 56 +- src/box/vy_tx.c | 133 +++++ src/box/vy_write_iterator.c | 168 +++++- src/box/vy_write_iterator.h | 27 +- src/box/xrow.c | 8 + src/box/xrow.h | 2 + test/app-tap/tarantoolctl.test.lua | 2 +- test/box-py/bootstrap.result | 5 +- test/box/access_misc.result | 3 + test/box/access_sysview.result | 2 +- test/box/alter.result | 4 +- test/unit/vy_iterators_helper.c | 5 + test/unit/vy_iterators_helper.h | 12 +- test/unit/vy_point_lookup.c | 6 +- test/unit/vy_write_iterator.c | 232 +++++++- test/unit/vy_write_iterator.result | 22 +- test/vinyl/deferred_delete.result | 677 +++++++++++++++++++++++ test/vinyl/deferred_delete.test.lua | 261 +++++++++ test/vinyl/info.result | 18 +- test/vinyl/info.test.lua | 9 +- test/vinyl/layout.result | 46 +- test/vinyl/quota.result | 2 +- test/vinyl/tx_gap_lock.result | 16 +- test/vinyl/tx_gap_lock.test.lua | 10 +- test/vinyl/write_iterator.result | 5 + test/vinyl/write_iterator.test.lua | 3 + test/wal_off/alter.result | 2 +- test/xlog/upgrade.result | 5 +- 49 files changed, 2953 insertions(+), 876 deletions(-) create mode 100644 test/vinyl/deferred_delete.result create mode 100644 test/vinyl/deferred_delete.test.lua -- 2.11.0