From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH v2 0/8] vinyl: compaction randomization and throttling Date: Thu, 24 Jan 2019 20:12:36 +0300 Message-Id: To: tarantool-patches@freelists.org List-ID: This patch set achieves two goals. The first one is randomizing compaction pace among ranges so as to avoid IO load bursts, which badly affect read and space amplification and complicate implementation of transaction throttling. The second goal is making sure that compaction always keeps up with dumps and throttling transactions if it doesn't. For more details, see comments to the individual patches and the GitHub issues. https://github.com/tarantool/tarantool/issues/3944 https://github.com/tarantool/tarantool/issues/3721 https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling Changes in v2: - Fixed compaction priority calculation that resulted in unstable compaction behavior and prevented randomization from smoothing out IO load generated by compaction. - Tuned automatic range sizing: don't limit the total number of ranges as this isn't going to play nicely with time series workloads; set max range size to 2 GB as we don't want compaction to take eternity no matter what; use 4 * dumps_per_compaction for range_count - better safe than sorry. - Fixed dumps_per_compaction update in case compaction was forced manually using index.compact(). We don't want to update this metric in such a case to avoid undesired side effects, such as splitting or coalescing ranges for no good reason. - Pushed the first two trivial patches and rebased. - Fixed a few failing tests. v1: https://www.freelists.org/post/tarantool-patches/PATCH-09-vinyl-compaction-randomization-and-throttling Vladimir Davydov (8): vinyl: use uncompressed run size for range split/coalesce/compaction vinyl: fix compaction priority calculation vinyl: rename lsm->range_heap to max_compaction_priority vinyl: keep track of dumps per compaction for each LSM tree vinyl: set range size automatically vinyl: randomize range compaction to avoid IO load spikes vinyl: introduce quota consumer types vinyl: throttle tx to ensure compaction keeps up with dumps src/box/alter.cc | 8 +- src/box/box.cc | 6 +- src/box/index_def.c | 2 +- src/box/lua/load_cfg.lua | 2 +- src/box/lua/space.cc | 6 +- src/box/vinyl.c | 39 ++++-- src/box/vy_log.c | 26 +++- src/box/vy_log.h | 10 +- src/box/vy_lsm.c | 91 ++++++++++++-- src/box/vy_lsm.h | 12 +- src/box/vy_quota.c | 132 ++++++++++++++++----- src/box/vy_quota.h | 97 ++++++++++++--- src/box/vy_range.c | 95 +++++++++++---- src/box/vy_range.h | 60 ++++++++-- src/box/vy_regulator.c | 98 +++++++++++++-- src/box/vy_regulator.h | 27 +++++ src/box/vy_run.c | 1 + src/box/vy_run.h | 20 ++++ src/box/vy_scheduler.c | 48 ++++++-- test/app-tap/init_script.result | 21 ++-- test/box-tap/cfg.test.lua | 3 +- test/box/admin.result | 2 - test/box/cfg.result | 4 - test/vinyl/ddl.result | 5 - test/vinyl/ddl.test.lua | 1 - test/vinyl/deferred_delete.result | 21 +++- test/vinyl/deferred_delete.test.lua | 11 +- test/vinyl/errinj.result | 4 +- test/vinyl/errinj.test.lua | 3 +- test/vinyl/gc.result | 3 +- test/vinyl/gc.test.lua | 3 +- test/vinyl/layout.result | 178 ++++++++++++++++++++------- test/vinyl/layout.test.lua | 5 +- test/vinyl/misc.result | 78 ++++++++++++ test/vinyl/misc.test.lua | 26 ++++ test/vinyl/replica_rejoin.result | 2 +- test/vinyl/replica_rejoin.test.lua | 2 +- test/vinyl/stat.result | 231 ++++++++++++++++++++++++++++-------- test/vinyl/stat.test.lua | 63 ++++++++++ test/vinyl/update_optimize.result | 76 ++++++------ test/vinyl/update_optimize.test.lua | 56 +++++---- test/vinyl/write_iterator.result | 20 +++- test/vinyl/write_iterator.test.lua | 10 +- 43 files changed, 1259 insertions(+), 349 deletions(-) -- 2.11.0