From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH v3 0/4] vinyl: compaction randomization and throttling Date: Tue, 12 Feb 2019 20:26:52 +0300 Message-Id: To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: This patch set achieves two goals. The first one is randomizing compaction pace among ranges so as to avoid IO load bursts, which badly affect read and space amplification and complicate implementation of transaction throttling. The second goal is making sure that compaction always keeps up with dumps and throttling transactions if it doesn't. For more details, see comments to the individual patches and the GitHub issues. https://github.com/tarantool/tarantool/issues/3944 https://github.com/tarantool/tarantool/issues/3721 https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling Changes in v3: - Pushed all the patches related to compaction randomization. Now this patch set is only about transaction throttling. - Removed vy_quota_consumer_for_each_rate_limit (inlined it) as requested by Kostja. - Added a couple of patches addressing minor issues with vy_quota: fairness and extra check. - Changed the rate limit coefficient to 0.75 and updated the comment as suggested by Kostja. v2: https://www.freelists.org/post/tarantool-patches/PATCH-v2-08-vinyl-compaction-randomization-and-throttling Changes in v2: - Fixed compaction priority calculation that resulted in unstable compaction behavior and prevented randomization from smoothing out IO load generated by compaction. - Tuned automatic range sizing: don't limit the total number of ranges as this isn't going to play nicely with time series workloads; set max range size to 2 GB as we don't want compaction to take eternity no matter what; use 4 * dumps_per_compaction for range_count - better safe than sorry. - Fixed dumps_per_compaction update in case compaction was forced manually using index.compact(). We don't want to update this metric in such a case to avoid undesired side effects, such as splitting or coalescing ranges for no good reason. - Pushed the first two trivial patches and rebased. - Fixed a few failing tests. v1: https://www.freelists.org/post/tarantool-patches/PATCH-09-vinyl-compaction-randomization-and-throttling Vladimir Davydov (4): vinyl: introduce quota consumer types vinyl: remove extra quota check from vy_quota_use vinyl: don't consume quota if wait queue isn't empty vinyl: throttle tx to ensure compaction keeps up with dumps src/box/vinyl.c | 36 +++++++---- src/box/vy_quota.c | 168 +++++++++++++++++++++++++++++++++++++------------ src/box/vy_quota.h | 97 +++++++++++++++++++++++----- src/box/vy_regulator.c | 155 ++++++++++++++++++++++++++++++++++++++++++--- src/box/vy_regulator.h | 27 ++++++++ 5 files changed, 410 insertions(+), 73 deletions(-) -- 2.11.0