Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: tarantool-patches@freelists.org
Subject: Re: [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps
Date: Mon, 21 Jan 2019 17:14:04 +0300	[thread overview]
Message-ID: <20190121141404.hdecfujunrtplons@esperanza> (raw)
In-Reply-To: <a48046dbdd143ba262592ad9ca3bdc0b3474257a.1548017258.git.vdavydov.dev@gmail.com>

On Mon, Jan 21, 2019 at 12:17:08AM +0300, Vladimir Davydov wrote:
> +void
> +vy_regulator_update_rate_limit(struct vy_regulator *regulator,
> +			       const struct vy_scheduler_stat *stat,
> +			       int compaction_threads)
> +{
> +	struct vy_scheduler_stat *last = &regulator->sched_stat_last;
> +	struct vy_scheduler_stat *recent = &regulator->sched_stat_recent;
> +	/*
> +	 * The maximal dump rate the database can handle while
> +	 * maintaining the current level of write amplification
> +	 * equals:
> +	 *
> +	 *                                        dump_output
> +	 *   max_dump_rate = compaction_rate * -----------------
> +	 *                                     compaction_output
> +	 *
> +	 * The average compaction rate can be estimated with:
> +	 *
> +	 *                                          compaction_output
> +	 *   compaction_rate = compaction_threads * -----------------
> +	 *                                           compaction_time
> +	 *
> +	 * Putting it all together and taking into account data
> +	 * compaction during memory dump, we get for the max
> +	 * transaction rate:
> +	 *
> +	 *                                 dump_input
> +	 *   max_tx_rate = max_dump_rate * ----------- =
> +	 *                                 dump_output
> +	 *
> +	 *                                        dump_input
> +	 *                 compaction_threads * ---------------
> +	 *                                      compaction_time
> +	 *
> +	 * We set the rate limit to half that to leave the database
> +	 * engine enough room needed for growing write amplification.
> +	 */
> +	recent->dump_count += stat->dump_count - last->dump_count;
> +	recent->dump_input += stat->dump_input - last->dump_input;
> +	recent->compaction_time += stat->compaction_time -
> +				   last->compaction_time;
> +	*last = *stat;
> +
> +	if (recent->compaction_time == 0 ||
> +	    recent->dump_input < (int)VY_DUMP_SIZE_ACCT_MIN)
> +		return;
> +
> +	double rate = 0.5 * compaction_threads * recent->dump_input /
> +						 recent->compaction_time;
> +	vy_quota_set_rate_limit(regulator->quota, VY_QUOTA_RESOURCE_DISK,
> +				MIN(rate, SIZE_MAX));
> +
> +	/*
> +	 * Periodically rotate statistics for quicker adaptation
> +	 * to workload changes.
> +	 */
> +	if (recent->dump_count > VY_RECENT_DUMP_COUNT) {
> +		recent->dump_count /= 2;
> +		recent->dump_input /= 2;
> +		recent->compaction_time /= 2;
> +	}
> +}

vinyl/deferreted_delete.test.lua generates very small dumps (several
KBs). If the disk is slow, which is the case on Travis CI, the overhead
associated with file creation will be too high in comparison to the
amount of work done by compaction, and so the rate limit will be set to
a very small value, stalling the test. I fixed this on the branch by
completely ignoring any dump less than 1 MB, as we do when estimating
dump bandwidth. The incremental diff is below.

diff --git a/src/box/vy_regulator.c b/src/box/vy_regulator.c
index b406cf97..75f1f798 100644
--- a/src/box/vy_regulator.c
+++ b/src/box/vy_regulator.c
@@ -322,16 +322,18 @@ vy_regulator_update_rate_limit(struct vy_regulator *regulator,
 	 * We set the rate limit to half that to leave the database
 	 * engine enough room needed for growing write amplification.
 	 */
-	recent->dump_count += stat->dump_count - last->dump_count;
-	recent->dump_input += stat->dump_input - last->dump_input;
-	recent->compaction_time += stat->compaction_time -
-				   last->compaction_time;
+	int32_t dump_count = stat->dump_count - last->dump_count;
+	int64_t dump_input = stat->dump_input - last->dump_input;
+	double compaction_time = stat->compaction_time - last->compaction_time;
 	*last = *stat;
 
-	if (recent->compaction_time == 0 ||
-	    recent->dump_input < (int)VY_DUMP_SIZE_ACCT_MIN)
+	if (dump_input < (ssize_t)VY_DUMP_SIZE_ACCT_MIN)
 		return;
 
+	recent->dump_count += dump_count;
+	recent->dump_input += dump_input;
+	recent->compaction_time += compaction_time;
+
 	double rate = 0.5 * compaction_threads * recent->dump_input /
 						 recent->compaction_time;
 	vy_quota_set_rate_limit(regulator->quota, VY_QUOTA_RESOURCE_DISK,

  reply	other threads:[~2019-01-21 14:14 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-20 21:16 [PATCH 0/9] vinyl: compaction randomization and throttling Vladimir Davydov
2019-01-20 21:17 ` [PATCH 1/9] vinyl: update lsm->range_heap in one go on dump completion Vladimir Davydov
2019-01-24 16:55   ` Vladimir Davydov
2019-02-05 16:37   ` [tarantool-patches] " Konstantin Osipov
2019-01-20 21:17 ` [PATCH 2/9] vinyl: ignore unknown .run, .index and .vylog keys Vladimir Davydov
2019-01-24 16:56   ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 3/9] vinyl: use uncompressed run size for range split/coalesce/compaction Vladimir Davydov
2019-01-21  9:42   ` Vladimir Davydov
2019-02-05 16:49     ` [tarantool-patches] " Konstantin Osipov
2019-02-06  8:55       ` Vladimir Davydov
2019-02-06 10:46         ` Konstantin Osipov
2019-02-06 10:55           ` Vladimir Davydov
2019-02-05 16:43   ` Konstantin Osipov
2019-02-06 16:48     ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 4/9] vinyl: rename lsm->range_heap to max_compaction_priority Vladimir Davydov
2019-01-20 21:17 ` [PATCH 5/9] vinyl: keep track of dumps per compaction for each LSM tree Vladimir Davydov
2019-02-05 16:58   ` [tarantool-patches] " Konstantin Osipov
2019-02-06  9:20     ` Vladimir Davydov
2019-02-06 16:54       ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 6/9] vinyl: set range size automatically Vladimir Davydov
2019-01-22  9:17   ` Vladimir Davydov
2019-02-05 17:09   ` [tarantool-patches] " Konstantin Osipov
2019-02-06  9:23     ` Vladimir Davydov
2019-02-06 17:04       ` Vladimir Davydov
2019-01-20 21:17 ` [PATCH 7/9] vinyl: randomize range compaction to avoid IO load spikes Vladimir Davydov
2019-01-22 12:54   ` Vladimir Davydov
2019-02-05 17:39     ` [tarantool-patches] " Konstantin Osipov
2019-02-06  8:53       ` Vladimir Davydov
2019-02-06 10:44         ` Konstantin Osipov
2019-02-06 10:52           ` Vladimir Davydov
2019-02-06 11:06             ` Konstantin Osipov
2019-02-06 11:49               ` Vladimir Davydov
2019-02-06 13:43                 ` Konstantin Osipov
2019-02-06 14:00                   ` Vladimir Davydov
2019-02-05 17:14   ` Konstantin Osipov
2019-01-20 21:17 ` [PATCH 8/9] vinyl: introduce quota consumer types Vladimir Davydov
2019-01-20 21:17 ` [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps Vladimir Davydov
2019-01-21 14:14   ` Vladimir Davydov [this message]
2019-01-22  9:09   ` Vladimir Davydov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190121141404.hdecfujunrtplons@esperanza \
    --to=vdavydov.dev@gmail.com \
    --cc=tarantool-patches@freelists.org \
    --subject='Re: [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox