From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: tarantool-patches@freelists.org
Subject: [PATCH 0/9] vinyl: compaction randomization and throttling
Date: Mon, 21 Jan 2019 00:16:59 +0300 [thread overview]
Message-ID: <cover.1548017258.git.vdavydov.dev@gmail.com> (raw)
This patch set achieves two goals. The first one is randomizing
compaction pace among ranges so as to avoid IO load bursts, which badly
affect read and space amplification and complicate implementation of
transaction throttling. The second goal is making sure that compaction
always keeps up with dumps and throttling transactions if it doesn't.
For more details, see comments to the individual patches and the GitHub
issues.
https://github.com/tarantool/tarantool/issues/3944
https://github.com/tarantool/tarantool/issues/3721
https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling
Vladimir Davydov (9):
vinyl: update lsm->range_heap in one go on dump completion
vinyl: ignore unknown .run, .index and .vylog keys
vinyl: use uncompressed run size for range split/coalesce/compaction
vinyl: rename lsm->range_heap to max_compaction_priority
vinyl: keep track of dumps per compaction for each LSM tree
vinyl: set range size automatically
vinyl: randomize range compaction to avoid IO load spikes
vinyl: introduce quota consumer types
vinyl: throttle tx to ensure compaction keeps up with dumps
src/box/alter.cc | 8 +-
src/box/box.cc | 6 +-
src/box/index_def.c | 2 +-
src/box/lua/load_cfg.lua | 2 +-
src/box/lua/space.cc | 6 +-
src/box/vinyl.c | 39 +++++--
src/box/vy_log.c | 33 ++++--
src/box/vy_log.h | 10 +-
src/box/vy_lsm.c | 83 ++++++++++++--
src/box/vy_lsm.h | 12 +-
src/box/vy_quota.c | 132 +++++++++++++++++-----
src/box/vy_quota.h | 97 ++++++++++++++---
src/box/vy_range.c | 55 ++++++++--
src/box/vy_range.h | 60 +++++++---
src/box/vy_regulator.c | 96 ++++++++++++++--
src/box/vy_regulator.h | 27 +++++
src/box/vy_run.c | 13 +--
src/box/vy_run.h | 20 ++++
src/box/vy_scheduler.c | 40 ++++---
test/app-tap/init_script.result | 21 ++--
test/box-tap/cfg.test.lua | 3 +-
test/box/admin.result | 2 -
test/box/cfg.result | 4 -
test/vinyl/ddl.result | 5 -
test/vinyl/ddl.test.lua | 1 -
test/vinyl/layout.result | 8 +-
test/vinyl/misc.result | 78 +++++++++++++
test/vinyl/misc.test.lua | 26 +++++
test/vinyl/stat.result | 217 +++++++++++++++++++++++++++++--------
test/vinyl/stat.test.lua | 57 ++++++++++
test/vinyl/write_iterator.result | 16 +++
test/vinyl/write_iterator.test.lua | 6 +
32 files changed, 962 insertions(+), 223 deletions(-)
--
2.11.0
next reply other threads:[~2019-01-20 21:16 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-20 21:16 Vladimir Davydov [this message]
2019-01-20 21:17 ` [PATCH 1/9] vinyl: update lsm->range_heap in one go on dump completion Vladimir Davydov
2019-01-24 16:55 ` Vladimir Davydov
2019-02-05 16:37 ` [tarantool-patches] " Konstantin Osipov
2019-01-20 21:17 ` [PATCH 2/9] vinyl: ignore unknown .run, .index and .vylog keys Vladimir Davydov
2019-01-24 16:56 ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 3/9] vinyl: use uncompressed run size for range split/coalesce/compaction Vladimir Davydov
2019-01-21 9:42 ` Vladimir Davydov
2019-02-05 16:49 ` [tarantool-patches] " Konstantin Osipov
2019-02-06 8:55 ` Vladimir Davydov
2019-02-06 10:46 ` Konstantin Osipov
2019-02-06 10:55 ` Vladimir Davydov
2019-02-05 16:43 ` Konstantin Osipov
2019-02-06 16:48 ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 4/9] vinyl: rename lsm->range_heap to max_compaction_priority Vladimir Davydov
2019-01-20 21:17 ` [PATCH 5/9] vinyl: keep track of dumps per compaction for each LSM tree Vladimir Davydov
2019-02-05 16:58 ` [tarantool-patches] " Konstantin Osipov
2019-02-06 9:20 ` Vladimir Davydov
2019-02-06 16:54 ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 6/9] vinyl: set range size automatically Vladimir Davydov
2019-01-22 9:17 ` Vladimir Davydov
2019-02-05 17:09 ` [tarantool-patches] " Konstantin Osipov
2019-02-06 9:23 ` Vladimir Davydov
2019-02-06 17:04 ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 7/9] vinyl: randomize range compaction to avoid IO load spikes Vladimir Davydov
2019-01-22 12:54 ` Vladimir Davydov
2019-02-05 17:39 ` [tarantool-patches] " Konstantin Osipov
2019-02-06 8:53 ` Vladimir Davydov
2019-02-06 10:44 ` Konstantin Osipov
2019-02-06 10:52 ` Vladimir Davydov
2019-02-06 11:06 ` Konstantin Osipov
2019-02-06 11:49 ` Vladimir Davydov
2019-02-06 13:43 ` Konstantin Osipov
2019-02-06 14:00 ` Vladimir Davydov
2019-02-05 17:14 ` Konstantin Osipov
2019-01-20 21:17 ` [PATCH 8/9] vinyl: introduce quota consumer types Vladimir Davydov
2019-01-20 21:17 ` [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps Vladimir Davydov
2019-01-21 14:14 ` Vladimir Davydov
2019-01-22 9:09 ` Vladimir Davydov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1548017258.git.vdavydov.dev@gmail.com \
--to=vdavydov.dev@gmail.com \
--cc=tarantool-patches@freelists.org \
--subject='Re: [PATCH 0/9] vinyl: compaction randomization and throttling' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox