From: Vladimir Davydov <vdavydov.dev@gmail.com> To: kostja@tarantool.org Cc: tarantool-patches@freelists.org Subject: [PATCH 4/7] vinyl: fix force compaction logic Date: Sun, 2 Sep 2018 23:18:57 +0300 [thread overview] Message-ID: <55436ba89b22a9f490c0642810986672a062cc73.1535917763.git.vdavydov.dev@gmail.com> (raw) In-Reply-To: <cover.1535917763.git.vdavydov.dev@gmail.com> In-Reply-To: <cover.1535917763.git.vdavydov.dev@gmail.com> This patch address a few problems index.compact() is suffering from, namely: - When a range is split or coalesced, it should inherit the value of needs_compaction flag from the source ranges. Currently, the flag is cleared so that the resulting range may be not compacted. - If a range has no slices, we shouldn't set needs_compaction flag for it, because obviously it can't be compacted, but we do. - The needs_compaction flag should be cleared as soon as we schedule a range for compaction, not when all slices have been compacted into one, as we presently expect, because the latter may never happen under a write-intensive load. --- src/box/vy_lsm.c | 9 +++++++-- src/box/vy_range.c | 16 ++-------------- src/box/vy_range.h | 8 ++------ src/box/vy_scheduler.c | 2 ++ 4 files changed, 13 insertions(+), 22 deletions(-) diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c index 15592fbf..a0d211f8 100644 --- a/src/box/vy_lsm.c +++ b/src/box/vy_lsm.c @@ -1040,6 +1040,7 @@ vy_lsm_split_range(struct vy_lsm *lsm, struct vy_range *range) if (new_slice != NULL) vy_range_add_slice(part, new_slice); } + part->needs_compaction = range->needs_compaction; part->compact_priority = range->compact_priority; } @@ -1147,6 +1148,8 @@ vy_lsm_coalesce_range(struct vy_lsm *lsm, struct vy_range *range) rlist_splice(&result->slices, &it->slices); result->slice_count += it->slice_count; vy_disk_stmt_counter_add(&result->count, &it->count); + if (it->needs_compaction) + result->needs_compaction = true; vy_range_delete(it); it = next; } @@ -1181,8 +1184,10 @@ vy_lsm_force_compaction(struct vy_lsm *lsm) struct vy_range_tree_iterator it; vy_range_tree_ifirst(lsm->tree, &it); - while ((range = vy_range_tree_inext(&it)) != NULL) - vy_range_force_compaction(range); + while ((range = vy_range_tree_inext(&it)) != NULL) { + range->needs_compaction = true; + vy_range_update_compact_priority(range, &lsm->opts); + } vy_range_heap_update_all(&lsm->range_heap); } diff --git a/src/box/vy_range.c b/src/box/vy_range.c index 6a55a018..ddcd2ed3 100644 --- a/src/box/vy_range.c +++ b/src/box/vy_range.c @@ -262,18 +262,6 @@ vy_range_remove_slice(struct vy_range *range, struct vy_slice *slice) vy_disk_stmt_counter_sub(&range->count, &slice->count); } -void -vy_range_force_compaction(struct vy_range *range) -{ - if (range->slice_count == 1) { - /* Already compacted. */ - assert(!range->needs_compaction); - return; - } - range->needs_compaction = true; - range->compact_priority = range->slice_count; -} - /** * To reduce write amplification caused by compaction, we follow * the LSM tree design. Runs in each range are divided into groups @@ -304,9 +292,9 @@ vy_range_update_compact_priority(struct vy_range *range, assert(opts->run_count_per_level > 0); assert(opts->run_size_ratio > 1); - if (range->slice_count == 1) { + if (range->slice_count <= 1) { /* Nothing to compact. */ - range->compact_priority = 1; + range->compact_priority = 0; range->needs_compaction = false; return; } diff --git a/src/box/vy_range.h b/src/box/vy_range.h index d7031e70..2ca19a1c 100644 --- a/src/box/vy_range.h +++ b/src/box/vy_range.h @@ -110,8 +110,8 @@ struct vy_range { * If this flag is set, the range must be scheduled for * major compaction, i.e. its compact_priority must be * raised to max (slice_count). The flag is set by - * vy_range_force_compaction() and cleared automatically - * when all slices of the range have been compacted. + * vy_lsm_force_compaction() and cleared when the range + * is scheduled for compaction. */ bool needs_compaction; /** Number of times the range was compacted. */ @@ -229,10 +229,6 @@ vy_range_add_slice_before(struct vy_range *range, struct vy_slice *slice, void vy_range_remove_slice(struct vy_range *range, struct vy_slice *slice); -/** Mark a range for major compaction. */ -void -vy_range_force_compaction(struct vy_range *range); - /** * Update compaction priority of a range. * diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c index 4959300e..a1ae3f54 100644 --- a/src/box/vy_scheduler.c +++ b/src/box/vy_scheduler.c @@ -1604,6 +1604,8 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_lsm *lsm, assert(n == 0); assert(new_run->dump_lsn >= 0); + range->needs_compaction = false; + task->range = range; task->new_run = new_run; task->wi = wi; -- 2.11.0
next prev parent reply other threads:[~2018-09-02 20:18 UTC|newest] Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-09-02 20:18 [PATCH 0/7] vinyl: improve stats for throttling Vladimir Davydov 2018-09-02 20:18 ` [PATCH 1/7] vinyl: fix accounting of secondary index cache statements Vladimir Davydov 2018-09-02 22:26 ` [tarantool-patches] " Konstantin Osipov 2018-09-02 20:18 ` [PATCH 2/7] vinyl: add global memory stats Vladimir Davydov 2018-09-02 22:27 ` [tarantool-patches] " Konstantin Osipov 2018-09-02 22:27 ` Konstantin Osipov 2018-09-03 8:10 ` Vladimir Davydov 2018-09-02 20:18 ` [PATCH 3/7] vinyl: add global disk stats Vladimir Davydov 2018-09-02 22:30 ` [tarantool-patches] " Konstantin Osipov 2018-09-02 20:18 ` Vladimir Davydov [this message] 2018-09-02 20:18 ` [PATCH 5/7] vinyl: update compact priority usual way on range split/coalesce Vladimir Davydov 2018-09-02 20:18 ` [PATCH 6/7] vinyl: keep track of compaction queue length and debt Vladimir Davydov 2018-09-02 20:19 ` [PATCH 7/7] vinyl: keep track of disk idle time Vladimir Davydov 2018-09-04 11:54 ` Vladimir Davydov 2018-09-04 17:23 ` Vladimir Davydov 2018-09-04 17:23 ` [PATCH 1/8] vinyl: add helper to check whether dump is in progress Vladimir Davydov 2018-09-06 7:33 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 2/8] vinyl: don't use mempool for allocating background tasks Vladimir Davydov 2018-09-06 7:33 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 3/8] vinyl: factor out worker pool from scheduler struct Vladimir Davydov 2018-09-06 7:34 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 4/8] vinyl: move worker allocation closer to task creation Vladimir Davydov 2018-09-06 7:35 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 5/8] vinyl: use separate thread pools for dump and compaction tasks Vladimir Davydov 2018-09-06 7:37 ` Konstantin Osipov 2018-09-06 9:48 ` Vladimir Davydov 2018-09-06 10:32 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 6/8] vinyl: zap vy_worker_pool::idle_worker_count Vladimir Davydov 2018-09-06 7:38 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 7/8] vinyl: don't start scheduler fiber until local recovery is complete Vladimir Davydov 2018-09-06 7:39 ` Konstantin Osipov 2018-09-04 17:23 ` [PATCH 8/8] vinyl: keep track of thread pool idle ratio Vladimir Davydov 2018-09-06 7:49 ` Konstantin Osipov 2018-09-06 8:18 ` Vladimir Davydov 2018-09-06 10:26 ` Konstantin Osipov 2018-09-06 10:52 ` Vladimir Davydov 2018-09-06 10:57 ` Konstantin Osipov 2018-09-06 11:59 ` Vladimir Davydov 2018-09-09 11:41 ` [PATCH 0/7] vinyl: improve stats for throttling Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=55436ba89b22a9f490c0642810986672a062cc73.1535917763.git.vdavydov.dev@gmail.com \ --to=vdavydov.dev@gmail.com \ --cc=kostja@tarantool.org \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH 4/7] vinyl: fix force compaction logic' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox