From: Vladimir Davydov <vdavydov.dev@gmail.com> To: tarantool-patches@freelists.org Subject: Re: [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps Date: Tue, 22 Jan 2019 12:09:01 +0300 [thread overview] Message-ID: <20190122090901.6dyslnsi4httxbxe@esperanza> (raw) In-Reply-To: <a48046dbdd143ba262592ad9ca3bdc0b3474257a.1548017258.git.vdavydov.dev@gmail.com> On Mon, Jan 21, 2019 at 12:17:08AM +0300, Vladimir Davydov wrote: > +void > +vy_regulator_update_rate_limit(struct vy_regulator *regulator, > + const struct vy_scheduler_stat *stat, > + int compaction_threads) > +{ > + struct vy_scheduler_stat *last = ®ulator->sched_stat_last; > + struct vy_scheduler_stat *recent = ®ulator->sched_stat_recent; > + /* > + * The maximal dump rate the database can handle while > + * maintaining the current level of write amplification > + * equals: > + * > + * dump_output > + * max_dump_rate = compaction_rate * ----------------- > + * compaction_output > + * > + * The average compaction rate can be estimated with: > + * > + * compaction_output > + * compaction_rate = compaction_threads * ----------------- > + * compaction_time > + * > + * Putting it all together and taking into account data > + * compaction during memory dump, we get for the max > + * transaction rate: > + * > + * dump_input > + * max_tx_rate = max_dump_rate * ----------- = > + * dump_output > + * > + * dump_input > + * compaction_threads * --------------- > + * compaction_time > + * > + * We set the rate limit to half that to leave the database > + * engine enough room needed for growing write amplification. > + */ > + recent->dump_count += stat->dump_count - last->dump_count; > + recent->dump_input += stat->dump_input - last->dump_input; > + recent->compaction_time += stat->compaction_time - > + last->compaction_time; > + *last = *stat; > + > + if (recent->compaction_time == 0 || > + recent->dump_input < (int)VY_DUMP_SIZE_ACCT_MIN) > + return; > + > + double rate = 0.5 * compaction_threads * recent->dump_input / > + recent->compaction_time; ^^^ I ran some tests that showed that 0.75 down-shifting coefficient works fine as well: the instance didn't accumulate much of compaction debt while the average compaction thread utilization grew from 50-60% to about 80%. However, write amplification decreased from 4 to 3.6, which indicates that the instance didn't have enough breathing space for compaction to keep up with dumps. Setting the coefficient to a higher value, say 0.9, results in a very bad performance - compaction queue grows up to 100% and write amplification stays at about 2. Probably, we could make this coefficient dynamic - scale it from 0.5 up to 1 depending on the compaction queue size - but that's going to be tricky to implement while maintaining a steady RPS, without spikes and falls - we'll need to run many more tests. I think we should start with a static coefficient, which is predictable and results in a steady RPS, and then think of something more sophisticated if there's a demand for that. If so, what coefficient should we choose? 0.5 is safe while 0.75 yields higher RPS. The user can always increase the number of compaction threads if RPS happens to be too small due to regulator decisions, but still, which one is better? > + vy_quota_set_rate_limit(regulator->quota, VY_QUOTA_RESOURCE_DISK, > + MIN(rate, SIZE_MAX)); > + > + /* > + * Periodically rotate statistics for quicker adaptation > + * to workload changes. > + */ > + if (recent->dump_count > VY_RECENT_DUMP_COUNT) { > + recent->dump_count /= 2; > + recent->dump_input /= 2; > + recent->compaction_time /= 2; > + } > +}
prev parent reply other threads:[~2019-01-22 9:09 UTC|newest] Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-01-20 21:16 [PATCH 0/9] vinyl: compaction randomization and throttling Vladimir Davydov 2019-01-20 21:17 ` [PATCH 1/9] vinyl: update lsm->range_heap in one go on dump completion Vladimir Davydov 2019-01-24 16:55 ` Vladimir Davydov 2019-02-05 16:37 ` [tarantool-patches] " Konstantin Osipov 2019-01-20 21:17 ` [PATCH 2/9] vinyl: ignore unknown .run, .index and .vylog keys Vladimir Davydov 2019-01-24 16:56 ` Vladimir Davydov 2019-01-20 21:17 ` [PATCH 3/9] vinyl: use uncompressed run size for range split/coalesce/compaction Vladimir Davydov 2019-01-21 9:42 ` Vladimir Davydov 2019-02-05 16:49 ` [tarantool-patches] " Konstantin Osipov 2019-02-06 8:55 ` Vladimir Davydov 2019-02-06 10:46 ` Konstantin Osipov 2019-02-06 10:55 ` Vladimir Davydov 2019-02-05 16:43 ` Konstantin Osipov 2019-02-06 16:48 ` Vladimir Davydov 2019-01-20 21:17 ` [PATCH 4/9] vinyl: rename lsm->range_heap to max_compaction_priority Vladimir Davydov 2019-01-20 21:17 ` [PATCH 5/9] vinyl: keep track of dumps per compaction for each LSM tree Vladimir Davydov 2019-02-05 16:58 ` [tarantool-patches] " Konstantin Osipov 2019-02-06 9:20 ` Vladimir Davydov 2019-02-06 16:54 ` Vladimir Davydov 2019-01-20 21:17 ` [PATCH 6/9] vinyl: set range size automatically Vladimir Davydov 2019-01-22 9:17 ` Vladimir Davydov 2019-02-05 17:09 ` [tarantool-patches] " Konstantin Osipov 2019-02-06 9:23 ` Vladimir Davydov 2019-02-06 17:04 ` Vladimir Davydov 2019-01-20 21:17 ` [PATCH 7/9] vinyl: randomize range compaction to avoid IO load spikes Vladimir Davydov 2019-01-22 12:54 ` Vladimir Davydov 2019-02-05 17:39 ` [tarantool-patches] " Konstantin Osipov 2019-02-06 8:53 ` Vladimir Davydov 2019-02-06 10:44 ` Konstantin Osipov 2019-02-06 10:52 ` Vladimir Davydov 2019-02-06 11:06 ` Konstantin Osipov 2019-02-06 11:49 ` Vladimir Davydov 2019-02-06 13:43 ` Konstantin Osipov 2019-02-06 14:00 ` Vladimir Davydov 2019-02-05 17:14 ` Konstantin Osipov 2019-01-20 21:17 ` [PATCH 8/9] vinyl: introduce quota consumer types Vladimir Davydov 2019-01-20 21:17 ` [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps Vladimir Davydov 2019-01-21 14:14 ` Vladimir Davydov 2019-01-22 9:09 ` Vladimir Davydov [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190122090901.6dyslnsi4httxbxe@esperanza \ --to=vdavydov.dev@gmail.com \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH 9/9] vinyl: throttle tx to ensure compaction keeps up with dumps' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox