From: Vladimir Davydov <vdavydov.dev@gmail.com> To: tarantool-patches@freelists.org Subject: [PATCH v2 0/8] vinyl: compaction randomization and throttling Date: Thu, 24 Jan 2019 20:12:36 +0300 [thread overview] Message-ID: <cover.1548349067.git.vdavydov.dev@gmail.com> (raw) This patch set achieves two goals. The first one is randomizing compaction pace among ranges so as to avoid IO load bursts, which badly affect read and space amplification and complicate implementation of transaction throttling. The second goal is making sure that compaction always keeps up with dumps and throttling transactions if it doesn't. For more details, see comments to the individual patches and the GitHub issues. https://github.com/tarantool/tarantool/issues/3944 https://github.com/tarantool/tarantool/issues/3721 https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling Changes in v2: - Fixed compaction priority calculation that resulted in unstable compaction behavior and prevented randomization from smoothing out IO load generated by compaction. - Tuned automatic range sizing: don't limit the total number of ranges as this isn't going to play nicely with time series workloads; set max range size to 2 GB as we don't want compaction to take eternity no matter what; use 4 * dumps_per_compaction for range_count - better safe than sorry. - Fixed dumps_per_compaction update in case compaction was forced manually using index.compact(). We don't want to update this metric in such a case to avoid undesired side effects, such as splitting or coalescing ranges for no good reason. - Pushed the first two trivial patches and rebased. - Fixed a few failing tests. v1: https://www.freelists.org/post/tarantool-patches/PATCH-09-vinyl-compaction-randomization-and-throttling Vladimir Davydov (8): vinyl: use uncompressed run size for range split/coalesce/compaction vinyl: fix compaction priority calculation vinyl: rename lsm->range_heap to max_compaction_priority vinyl: keep track of dumps per compaction for each LSM tree vinyl: set range size automatically vinyl: randomize range compaction to avoid IO load spikes vinyl: introduce quota consumer types vinyl: throttle tx to ensure compaction keeps up with dumps src/box/alter.cc | 8 +- src/box/box.cc | 6 +- src/box/index_def.c | 2 +- src/box/lua/load_cfg.lua | 2 +- src/box/lua/space.cc | 6 +- src/box/vinyl.c | 39 ++++-- src/box/vy_log.c | 26 +++- src/box/vy_log.h | 10 +- src/box/vy_lsm.c | 91 ++++++++++++-- src/box/vy_lsm.h | 12 +- src/box/vy_quota.c | 132 ++++++++++++++++----- src/box/vy_quota.h | 97 ++++++++++++--- src/box/vy_range.c | 95 +++++++++++---- src/box/vy_range.h | 60 ++++++++-- src/box/vy_regulator.c | 98 +++++++++++++-- src/box/vy_regulator.h | 27 +++++ src/box/vy_run.c | 1 + src/box/vy_run.h | 20 ++++ src/box/vy_scheduler.c | 48 ++++++-- test/app-tap/init_script.result | 21 ++-- test/box-tap/cfg.test.lua | 3 +- test/box/admin.result | 2 - test/box/cfg.result | 4 - test/vinyl/ddl.result | 5 - test/vinyl/ddl.test.lua | 1 - test/vinyl/deferred_delete.result | 21 +++- test/vinyl/deferred_delete.test.lua | 11 +- test/vinyl/errinj.result | 4 +- test/vinyl/errinj.test.lua | 3 +- test/vinyl/gc.result | 3 +- test/vinyl/gc.test.lua | 3 +- test/vinyl/layout.result | 178 ++++++++++++++++++++------- test/vinyl/layout.test.lua | 5 +- test/vinyl/misc.result | 78 ++++++++++++ test/vinyl/misc.test.lua | 26 ++++ test/vinyl/replica_rejoin.result | 2 +- test/vinyl/replica_rejoin.test.lua | 2 +- test/vinyl/stat.result | 231 ++++++++++++++++++++++++++++-------- test/vinyl/stat.test.lua | 63 ++++++++++ test/vinyl/update_optimize.result | 76 ++++++------ test/vinyl/update_optimize.test.lua | 56 +++++---- test/vinyl/write_iterator.result | 20 +++- test/vinyl/write_iterator.test.lua | 10 +- 43 files changed, 1259 insertions(+), 349 deletions(-) -- 2.11.0
next reply other threads:[~2019-01-24 17:12 UTC|newest] Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-01-24 17:12 Vladimir Davydov [this message] 2019-01-24 17:12 ` [PATCH v2 1/8] vinyl: use uncompressed run size for range split/coalesce/compaction Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 2/8] vinyl: fix compaction priority calculation Vladimir Davydov 2019-02-08 17:18 ` [tarantool-patches] " Konstantin Osipov 2019-02-11 14:42 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 3/8] vinyl: rename lsm->range_heap to max_compaction_priority Vladimir Davydov 2019-02-08 17:19 ` [tarantool-patches] " Konstantin Osipov 2019-02-08 17:26 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 4/8] vinyl: keep track of dumps per compaction for each LSM tree Vladimir Davydov 2019-02-08 17:42 ` Vladimir Davydov 2019-02-11 18:17 ` [tarantool-patches] " Konstantin Osipov 2019-02-11 18:17 ` Konstantin Osipov 2019-02-12 10:15 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 5/8] vinyl: set range size automatically Vladimir Davydov 2019-02-11 18:21 ` [tarantool-patches] " Konstantin Osipov 2019-02-12 10:16 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 6/8] vinyl: randomize range compaction to avoid IO load spikes Vladimir Davydov 2019-02-11 18:21 ` [tarantool-patches] " Konstantin Osipov 2019-02-12 10:16 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 7/8] vinyl: introduce quota consumer types Vladimir Davydov 2019-02-12 15:48 ` Vladimir Davydov 2019-01-24 17:12 ` [PATCH v2 8/8] vinyl: throttle tx to ensure compaction keeps up with dumps Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=cover.1548349067.git.vdavydov.dev@gmail.com \ --to=vdavydov.dev@gmail.com \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH v2 0/8] vinyl: compaction randomization and throttling' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox