From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH 0/9] vinyl: compaction randomization and throttling Date: Mon, 21 Jan 2019 00:16:59 +0300 Message-Id: To: tarantool-patches@freelists.org List-ID: This patch set achieves two goals. The first one is randomizing compaction pace among ranges so as to avoid IO load bursts, which badly affect read and space amplification and complicate implementation of transaction throttling. The second goal is making sure that compaction always keeps up with dumps and throttling transactions if it doesn't. For more details, see comments to the individual patches and the GitHub issues. https://github.com/tarantool/tarantool/issues/3944 https://github.com/tarantool/tarantool/issues/3721 https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling Vladimir Davydov (9): vinyl: update lsm->range_heap in one go on dump completion vinyl: ignore unknown .run, .index and .vylog keys vinyl: use uncompressed run size for range split/coalesce/compaction vinyl: rename lsm->range_heap to max_compaction_priority vinyl: keep track of dumps per compaction for each LSM tree vinyl: set range size automatically vinyl: randomize range compaction to avoid IO load spikes vinyl: introduce quota consumer types vinyl: throttle tx to ensure compaction keeps up with dumps src/box/alter.cc | 8 +- src/box/box.cc | 6 +- src/box/index_def.c | 2 +- src/box/lua/load_cfg.lua | 2 +- src/box/lua/space.cc | 6 +- src/box/vinyl.c | 39 +++++-- src/box/vy_log.c | 33 ++++-- src/box/vy_log.h | 10 +- src/box/vy_lsm.c | 83 ++++++++++++-- src/box/vy_lsm.h | 12 +- src/box/vy_quota.c | 132 +++++++++++++++++----- src/box/vy_quota.h | 97 ++++++++++++++--- src/box/vy_range.c | 55 ++++++++-- src/box/vy_range.h | 60 +++++++--- src/box/vy_regulator.c | 96 ++++++++++++++-- src/box/vy_regulator.h | 27 +++++ src/box/vy_run.c | 13 +-- src/box/vy_run.h | 20 ++++ src/box/vy_scheduler.c | 40 ++++--- test/app-tap/init_script.result | 21 ++-- test/box-tap/cfg.test.lua | 3 +- test/box/admin.result | 2 - test/box/cfg.result | 4 - test/vinyl/ddl.result | 5 - test/vinyl/ddl.test.lua | 1 - test/vinyl/layout.result | 8 +- test/vinyl/misc.result | 78 +++++++++++++ test/vinyl/misc.test.lua | 26 +++++ test/vinyl/stat.result | 217 +++++++++++++++++++++++++++++-------- test/vinyl/stat.test.lua | 57 ++++++++++ test/vinyl/write_iterator.result | 16 +++ test/vinyl/write_iterator.test.lua | 6 + 32 files changed, 962 insertions(+), 223 deletions(-) -- 2.11.0