[PATCH v3 0/4] vinyl: compaction randomization and throttling
vdavydov.dev at gmail.com
Tue Feb 12 20:26:52 MSK 2019
This patch set achieves two goals. The first one is randomizing
compaction pace among ranges so as to avoid IO load bursts, which badly
affect read and space amplification and complicate implementation of
transaction throttling. The second goal is making sure that compaction
always keeps up with dumps and throttling transactions if it doesn't.
For more details, see comments to the individual patches and the GitHub
Changes in v3:
- Pushed all the patches related to compaction randomization. Now this
patch set is only about transaction throttling.
- Removed vy_quota_consumer_for_each_rate_limit (inlined it) as
requested by Kostja.
- Added a couple of patches addressing minor issues with vy_quota:
fairness and extra check.
- Changed the rate limit coefficient to 0.75 and updated the comment
as suggested by Kostja.
Changes in v2:
- Fixed compaction priority calculation that resulted in unstable
compaction behavior and prevented randomization from smoothing out
IO load generated by compaction.
- Tuned automatic range sizing: don't limit the total number of ranges
as this isn't going to play nicely with time series workloads; set
max range size to 2 GB as we don't want compaction to take eternity
no matter what; use 4 * dumps_per_compaction for range_count - better
safe than sorry.
- Fixed dumps_per_compaction update in case compaction was forced
manually using index.compact(). We don't want to update this metric
in such a case to avoid undesired side effects, such as splitting or
coalescing ranges for no good reason.
- Pushed the first two trivial patches and rebased.
- Fixed a few failing tests.
Vladimir Davydov (4):
vinyl: introduce quota consumer types
vinyl: remove extra quota check from vy_quota_use
vinyl: don't consume quota if wait queue isn't empty
vinyl: throttle tx to ensure compaction keeps up with dumps
src/box/vinyl.c | 36 +++++++----
src/box/vy_quota.c | 168 +++++++++++++++++++++++++++++++++++++------------
src/box/vy_quota.h | 97 +++++++++++++++++++++++-----
src/box/vy_regulator.c | 155 ++++++++++++++++++++++++++++++++++++++++++---
src/box/vy_regulator.h | 27 ++++++++
5 files changed, 410 insertions(+), 73 deletions(-)
More information about the Tarantool-patches