From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH 0/3] Force compaction if there too many DELETEs Date: Sat, 20 Oct 2018 21:23:07 +0300 Message-Id: To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: Normally, space amplification of a vinyl index is confined by the configured LSM tree shape and shouldn't exceed 3-4. However, this might not hold in case indexed fields are frequently updated, e.g. s = box.schema.space.create('test', {engine = 'vinyl'}) s:create_index('primary') s:create_index('secondary', {parts = {2, 'unsigned'}}) s:update(key, {{'+', 2, 1}}) -- there a lot of updates like this Such a workload will generate tons of DELETE statement in the secondary index, and they won't be purged until major compaction is scheduled, which may not happen for a very very long time, resulting in space amplification getting as high as 100 or even 1000. This patch set fixes this issue by forcing major compaction of a range in case the number of DELETEs plus statements that they are supposed to purge exceeds a half of all statements in the range. This should prevent space and read amplification from growing to insane heights. Note, it doesn't close #3225, because that issue is also about forcing compaction in case the observed read amplification gets too high, which isn't addressed by this particular patch set. https://github.com/tarantool/tarantool/issues/3225 https://github.com/tarantool/tarantool/commits/dv/gh-3225-vy-force-compaction-when-there-are-too-many-deletes Vladimir Davydov (3): vinyl: remove useless local var from vy_range_update_compact_priority vinyl: account disk statements of each type vinyl: force major compaction if there are too many DELETEs src/box/iproto_constants.c | 1 + src/box/iproto_constants.h | 2 + src/box/vinyl.c | 6 ++ src/box/vy_lsm.c | 2 + src/box/vy_range.c | 27 +++++++-- src/box/vy_run.c | 82 +++++++++++++++++++++++++- src/box/vy_run.h | 10 ++++ src/box/vy_stat.h | 65 +++++++++++++++++++++ test/vinyl/compact.result | 40 +++++++++++++ test/vinyl/compact.test.lua | 18 ++++++ test/vinyl/deferred_delete.result | 26 +++++++-- test/vinyl/deferred_delete.test.lua | 15 ++++- test/vinyl/info.result | 113 +++++++++++++++++++++++++++++++++--- test/vinyl/info.test.lua | 36 ++++++++++++ test/vinyl/layout.result | 12 ++-- test/vinyl/update_optimize.result | 9 ++- test/vinyl/update_optimize.test.lua | 5 +- 17 files changed, 440 insertions(+), 29 deletions(-) -- 2.11.0