From: Vladimir Davydov <vdavydov.dev@gmail.com> To: kostja@tarantool.org Cc: tarantool-patches@freelists.org Subject: Re: [PATCH v2 09/11] vinyl: do not account zero dump bandwidth Date: Fri, 12 Oct 2018 16:27:21 +0300 [thread overview] Message-ID: <20181012132721.vbshvexo2igrhtur@esperanza> (raw) In-Reply-To: <13fecfb17369a9d4819eb82115722c37245b2db0.1538155645.git.vdavydov.dev@gmail.com> On Fri, Sep 28, 2018 at 08:40:07PM +0300, Vladimir Davydov wrote: > Since we free memory in 16 MB blocks (see SLAB_SIZE), it may occur that > we dump almost all data stored in a block but still have to leave it be, > because it contains data of a newer generation. If the memory limit is > small (as it is typically in tests), this may result in dumping 0 bytes. > In order not to disrupt statistics and throttling transactions in vain, > let's simply ignore such results. Normally, the memory limit should be > large enough for such granularity not to affect the measurements > (hundreds of megabytes) so this problem isn't worth putting more efforts > into. > > Needed for #1862 > --- > src/box/vy_regulator.c | 15 ++++++++++++++- > 1 file changed, 14 insertions(+), 1 deletion(-) > > diff --git a/src/box/vy_regulator.c b/src/box/vy_regulator.c > index 5ec5629f..682777fc 100644 > --- a/src/box/vy_regulator.c > +++ b/src/box/vy_regulator.c > @@ -200,7 +200,20 @@ vy_regulator_dump_complete(struct vy_regulator *regulator, > { > regulator->dump_in_progress = false; > > - if (dump_duration > 0) { > + /* > + * Update dump bandwidth. > + * > + * Note, since we free memory in 16 MB blocks (see SLAB_SIZE), > + * it may occur that we dump almost all data stored in a block > + * but still have to leave it be, because it contains data of > + * a newer generation. If the memory limit is small, this may > + * result in dumping 0 bytes. In order not to disrupt statistics > + * let's simply ignore such results. Normally, the memory limit > + * should be large enough for such granularity not to affect the > + * measurements (hundreds of megabytes) so this problem isn't > + * worth putting more efforts into. > + */ > + if (mem_dumped > 0 && dump_duration > 0) { > histogram_collect(regulator->dump_bw_hist, > mem_dumped / dump_duration); > /* Turns out this isn't enough. We'd better not account too small dumps, because such dumps have too high overhead associated with file creation. Our tests create a lot of small dumps using box.snapshot(). Taking them into account may slow down the overall test execution time or even break some tests. Let's ignore all dumps of size less than 1 MB for bandwidth estimation. The new patch is below. From 8d5858835c61ed290e753241e146cee50c41e6db Mon Sep 17 00:00:00 2001 From: Vladimir Davydov <vdavydov.dev@gmail.com> Date: Fri, 12 Oct 2018 16:12:06 +0300 Subject: [PATCH] vinyl: do not account small dumps for bandwidth estimation Small dumps (e.g. triggered by box.snapshot) have too high overhead associated with file creation so taking them into account for bandwidth estimation may result in erroneous transaction throttling. Let's ignore dumps of size less than 1 MB. Needed for #1862 diff --git a/src/box/vy_regulator.c b/src/box/vy_regulator.c index 6ecb5aa6..c6a56905 100644 --- a/src/box/vy_regulator.c +++ b/src/box/vy_regulator.c @@ -66,6 +66,13 @@ static const int VY_DUMP_BANDWIDTH_PCT = 10; */ static const size_t VY_DUMP_BANDWIDTH_DEFAULT = 10 * 1024 * 1024; +/** + * Do not take into account small dumps when estimating dump + * bandwidth, because they have too high overhead associated + * with file creation. + */ +static const size_t VY_DUMP_SIZE_ACCT_MIN = 1024 * 1024; + static void vy_regulator_trigger_dump(struct vy_regulator *regulator) { @@ -205,7 +212,7 @@ vy_regulator_dump_complete(struct vy_regulator *regulator, { regulator->dump_in_progress = false; - if (dump_duration > 0) { + if (mem_dumped >= VY_DUMP_SIZE_ACCT_MIN && dump_duration > 0) { histogram_collect(regulator->dump_bandwidth_hist, mem_dumped / dump_duration); /*
next prev parent reply other threads:[~2018-10-12 13:27 UTC|newest] Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-09-28 17:39 [PATCH v2 00/11] vinyl: transaction throttling infrastructure Vladimir Davydov 2018-09-28 17:39 ` [PATCH v2 01/11] vinyl: add helper to start scheduler and enable quota on startup Vladimir Davydov 2018-09-29 4:37 ` [tarantool-patches] " Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 02/11] vinyl: factor load regulator out of quota Vladimir Davydov 2018-09-29 5:00 ` [tarantool-patches] " Konstantin Osipov 2018-09-29 11:36 ` Vladimir Davydov [not found] ` <20180929114308.GA19162@chai> 2018-10-01 10:27 ` Vladimir Davydov 2018-10-01 10:31 ` Vladimir Davydov 2018-10-02 18:16 ` [tarantool-patches] " Konstantin Osipov 2018-10-03 8:49 ` Vladimir Davydov 2018-09-28 17:40 ` [PATCH v2 03/11] vinyl: minor refactoring of quota methods Vladimir Davydov 2018-09-29 5:01 ` [tarantool-patches] " Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 04/11] vinyl: move transaction size sanity check to quota Vladimir Davydov 2018-09-29 5:02 ` [tarantool-patches] " Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 05/11] vinyl: implement quota wait queue without fiber_cond Vladimir Davydov 2018-09-29 5:05 ` [tarantool-patches] " Konstantin Osipov 2018-09-29 11:44 ` Vladimir Davydov 2018-09-28 17:40 ` [PATCH v2 06/11] vinyl: enable quota upon recovery completion explicitly Vladimir Davydov 2018-09-29 5:06 ` [tarantool-patches] " Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 07/11] vinyl: zap vy_env::memory, read_threads, and write_threads Vladimir Davydov 2018-09-29 5:06 ` [tarantool-patches] " Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 08/11] vinyl: do not try to trigger dump in regulator if already in progress Vladimir Davydov 2018-09-28 17:40 ` [PATCH v2 09/11] vinyl: do not account zero dump bandwidth Vladimir Davydov 2018-10-12 13:27 ` Vladimir Davydov [this message] 2018-10-16 18:25 ` [tarantool-patches] " Konstantin Osipov 2018-10-17 8:44 ` Vladimir Davydov 2018-10-23 7:02 ` Konstantin Osipov 2018-09-28 17:40 ` [PATCH v2 10/11] vinyl: implement basic transaction throttling Vladimir Davydov 2018-09-28 17:40 ` [PATCH v2 11/11] vinyl: introduce quota consumer priorities Vladimir Davydov 2018-10-06 13:24 ` Konstantin Osipov 2018-10-08 11:10 ` Vladimir Davydov 2018-10-09 13:25 ` Vladimir Davydov 2018-10-11 7:02 ` Konstantin Osipov 2018-10-11 8:29 ` Vladimir Davydov 2018-10-03 9:06 ` [PATCH v2 00/11] vinyl: transaction throttling infrastructure Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181012132721.vbshvexo2igrhtur@esperanza \ --to=vdavydov.dev@gmail.com \ --cc=kostja@tarantool.org \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH v2 09/11] vinyl: do not account zero dump bandwidth' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox