Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: tarantool-patches@freelists.org
Subject: [PATCH v2 0/8] vinyl: compaction randomization and throttling
Date: Thu, 24 Jan 2019 20:12:36 +0300	[thread overview]
Message-ID: <cover.1548349067.git.vdavydov.dev@gmail.com> (raw)

This patch set achieves two goals. The first one is randomizing
compaction pace among ranges so as to avoid IO load bursts, which badly
affect read and space amplification and complicate implementation of
transaction throttling. The second goal is making sure that compaction
always keeps up with dumps and throttling transactions if it doesn't.
For more details, see comments to the individual patches and the GitHub
issues.

https://github.com/tarantool/tarantool/issues/3944
https://github.com/tarantool/tarantool/issues/3721
https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling

Changes in v2:
 - Fixed compaction priority calculation that resulted in unstable
   compaction behavior and prevented randomization from smoothing out
   IO load generated by compaction.
 - Tuned automatic range sizing: don't limit the total number of ranges
   as this isn't going to play nicely with time series workloads; set
   max range size to 2 GB as we don't want compaction to take eternity
   no matter what; use 4 * dumps_per_compaction for range_count - better
   safe than sorry.
 - Fixed dumps_per_compaction update in case compaction was forced
   manually using index.compact(). We don't want to update this metric
   in such a case to avoid undesired side effects, such as splitting or
   coalescing ranges for no good reason.
 - Pushed the first two trivial patches and rebased.
 - Fixed a few failing tests.

v1: https://www.freelists.org/post/tarantool-patches/PATCH-09-vinyl-compaction-randomization-and-throttling

Vladimir Davydov (8):
  vinyl: use uncompressed run size for range split/coalesce/compaction
  vinyl: fix compaction priority calculation
  vinyl: rename lsm->range_heap to max_compaction_priority
  vinyl: keep track of dumps per compaction for each LSM tree
  vinyl: set range size automatically
  vinyl: randomize range compaction to avoid IO load spikes
  vinyl: introduce quota consumer types
  vinyl: throttle tx to ensure compaction keeps up with dumps

 src/box/alter.cc                    |   8 +-
 src/box/box.cc                      |   6 +-
 src/box/index_def.c                 |   2 +-
 src/box/lua/load_cfg.lua            |   2 +-
 src/box/lua/space.cc                |   6 +-
 src/box/vinyl.c                     |  39 ++++--
 src/box/vy_log.c                    |  26 +++-
 src/box/vy_log.h                    |  10 +-
 src/box/vy_lsm.c                    |  91 ++++++++++++--
 src/box/vy_lsm.h                    |  12 +-
 src/box/vy_quota.c                  | 132 ++++++++++++++++-----
 src/box/vy_quota.h                  |  97 ++++++++++++---
 src/box/vy_range.c                  |  95 +++++++++++----
 src/box/vy_range.h                  |  60 ++++++++--
 src/box/vy_regulator.c              |  98 +++++++++++++--
 src/box/vy_regulator.h              |  27 +++++
 src/box/vy_run.c                    |   1 +
 src/box/vy_run.h                    |  20 ++++
 src/box/vy_scheduler.c              |  48 ++++++--
 test/app-tap/init_script.result     |  21 ++--
 test/box-tap/cfg.test.lua           |   3 +-
 test/box/admin.result               |   2 -
 test/box/cfg.result                 |   4 -
 test/vinyl/ddl.result               |   5 -
 test/vinyl/ddl.test.lua             |   1 -
 test/vinyl/deferred_delete.result   |  21 +++-
 test/vinyl/deferred_delete.test.lua |  11 +-
 test/vinyl/errinj.result            |   4 +-
 test/vinyl/errinj.test.lua          |   3 +-
 test/vinyl/gc.result                |   3 +-
 test/vinyl/gc.test.lua              |   3 +-
 test/vinyl/layout.result            | 178 ++++++++++++++++++++-------
 test/vinyl/layout.test.lua          |   5 +-
 test/vinyl/misc.result              |  78 ++++++++++++
 test/vinyl/misc.test.lua            |  26 ++++
 test/vinyl/replica_rejoin.result    |   2 +-
 test/vinyl/replica_rejoin.test.lua  |   2 +-
 test/vinyl/stat.result              | 231 ++++++++++++++++++++++++++++--------
 test/vinyl/stat.test.lua            |  63 ++++++++++
 test/vinyl/update_optimize.result   |  76 ++++++------
 test/vinyl/update_optimize.test.lua |  56 +++++----
 test/vinyl/write_iterator.result    |  20 +++-
 test/vinyl/write_iterator.test.lua  |  10 +-
 43 files changed, 1259 insertions(+), 349 deletions(-)

-- 
2.11.0

             reply	other threads:[~2019-01-24 17:12 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-01-24 17:12 Vladimir Davydov [this message]
2019-01-24 17:12 ` [PATCH v2 1/8] vinyl: use uncompressed run size for range split/coalesce/compaction Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 2/8] vinyl: fix compaction priority calculation Vladimir Davydov
2019-02-08 17:18   ` [tarantool-patches] " Konstantin Osipov
2019-02-11 14:42     ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 3/8] vinyl: rename lsm->range_heap to max_compaction_priority Vladimir Davydov
2019-02-08 17:19   ` [tarantool-patches] " Konstantin Osipov
2019-02-08 17:26     ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 4/8] vinyl: keep track of dumps per compaction for each LSM tree Vladimir Davydov
2019-02-08 17:42   ` Vladimir Davydov
2019-02-11 18:17     ` [tarantool-patches] " Konstantin Osipov
2019-02-11 18:17   ` Konstantin Osipov
2019-02-12 10:15     ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 5/8] vinyl: set range size automatically Vladimir Davydov
2019-02-11 18:21   ` [tarantool-patches] " Konstantin Osipov
2019-02-12 10:16     ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 6/8] vinyl: randomize range compaction to avoid IO load spikes Vladimir Davydov
2019-02-11 18:21   ` [tarantool-patches] " Konstantin Osipov
2019-02-12 10:16     ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 7/8] vinyl: introduce quota consumer types Vladimir Davydov
2019-02-12 15:48   ` Vladimir Davydov
2019-01-24 17:12 ` [PATCH v2 8/8] vinyl: throttle tx to ensure compaction keeps up with dumps Vladimir Davydov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1548349067.git.vdavydov.dev@gmail.com \
    --to=vdavydov.dev@gmail.com \
    --cc=tarantool-patches@freelists.org \
    --subject='Re: [PATCH v2 0/8] vinyl: compaction randomization and throttling' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox