From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH v2 0/7] vinyl: eliminate disk read on REPLACE/DELETE Date: Tue, 21 Aug 2018 14:15:33 +0300 Message-Id: To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: This patch set optimizes REPLACE and DELETE operations in vinyl in presence of secondary indexes: now they don't need to read the primary key in order to delete the overwritten/deleted tuple from secondary indexes, instead this job is handed over to primary index compaction task, while the read iterator is tought to filter out overwritten tuples that haven't been purged yet. For more information about the algorithm and implementation details, please see the final patch of the series. https://github.com/tarantool/tarantool/issues/2129 https://github.com/tarantool/tarantool/commits/dv/gh-2129-vy-eliminate-read-on-replace-delete Changes in v2: - Remove merged patches and rebase on the latest 1.10. - Use id 257 for _vinyl_deferred_delete space, as it doesn't depend on other system spaces. - Refactor _vinyl_deferred_delete space proto creation code. - Use interface instead of callback for returning deferred DELETEs from write iterator. - Get rid of vy_mem::wal_deferred_delete_max_lsn. Account deferred DELETE WAL LSN in vy_mem::dump_lsn. - Don't encode flags for secondary index statements (so that there's no need to pass deferred DELETE handler to the write iterator on dump). v1: https://www.freelists.org/post/tarantool-patches/PATCH-0025-vinyl-eliminate-disk-read-on-REPLACEDELETE v0: https://www.freelists.org/post/tarantool-patches/RFC-PATCH-0023-vinyl-eliminate-read-on-REPLACEDELETE Vladimir Davydov (7): vinyl: do not store meta in secondary index runs vinyl: teach write iterator to return overwritten tuples vinyl: prepare write iterator heap comparator for deferred DELETEs vinyl: allow to skip certain statements on read Introduce _vinyl_deferred_delete system space vinyl: zap vy_mem::min_lsn and rename max_lsn to dump_lsn vinyl: eliminate disk read on REPLACE/DELETE src/box/bootstrap.snap | Bin 1540 -> 1607 bytes src/box/lua/space.cc | 2 + src/box/lua/upgrade.lua | 21 ++ src/box/schema.cc | 27 +- src/box/schema_def.h | 2 + src/box/vinyl.c | 261 ++++++++++++-- src/box/vy_lsm.h | 5 + src/box/vy_mem.c | 27 +- src/box/vy_mem.h | 16 +- src/box/vy_point_lookup.c | 32 ++ src/box/vy_point_lookup.h | 18 + src/box/vy_run.c | 7 +- src/box/vy_scheduler.c | 310 ++++++++++++++++- src/box/vy_stmt.c | 7 +- src/box/vy_stmt.h | 29 ++ src/box/vy_tx.c | 133 +++++++ src/box/vy_write_iterator.c | 153 +++++++- src/box/vy_write_iterator.h | 45 ++- test/app-tap/tarantoolctl.test.lua | 2 +- test/box-py/bootstrap.result | 7 +- test/box/access_misc.result | 5 +- test/box/access_sysview.result | 2 +- test/unit/vy_iterators_helper.c | 5 + test/unit/vy_iterators_helper.h | 12 +- test/unit/vy_mem.c | 11 +- test/unit/vy_mem.result | 22 +- test/unit/vy_point_lookup.c | 6 +- test/unit/vy_write_iterator.c | 254 +++++++++++++- test/unit/vy_write_iterator.result | 22 +- test/vinyl/deferred_delete.result | 677 ++++++++++++++++++++++++++++++++++++ test/vinyl/deferred_delete.test.lua | 261 ++++++++++++++ test/vinyl/info.result | 18 +- test/vinyl/info.test.lua | 9 +- test/vinyl/layout.result | 46 ++- test/vinyl/quota.result | 2 +- test/vinyl/tx_gap_lock.result | 16 +- test/vinyl/tx_gap_lock.test.lua | 10 +- test/vinyl/write_iterator.result | 5 + test/vinyl/write_iterator.test.lua | 3 + test/wal_off/alter.result | 2 +- test/xlog/upgrade.result | 7 +- 41 files changed, 2344 insertions(+), 155 deletions(-) create mode 100644 test/vinyl/deferred_delete.result create mode 100644 test/vinyl/deferred_delete.test.lua -- 2.11.0