From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: kostja@tarantool.org
Cc: tarantool-patches@freelists.org
Subject: [PATCH 0/7] vinyl: improve stats for throttling
Date: Sun, 2 Sep 2018 23:18:53 +0300 [thread overview]
Message-ID: <cover.1535917763.git.vdavydov.dev@gmail.com> (raw)
To implement transaction throttling, we need (at least) two additional
metrics:
- Compaction debt, i.e. how much data have to be compacted to restore
the target LSM tree shape. This is needed to make decisions about
decreasing transaction rate.
- Disk idle time, i.e. how much time worker threads spend doing
nothing. This is needed to make decisions about loosening throttling.
This patch set adds those metrics. Along the way it also fixes a bug
in cache accounting (patch 1) and adds some useful global statistics
(patches 2 and 3).
https://github.com/tarantool/tarantool/issues/1862
https://github.com/tarantool/tarantool/commits/dv/vy-stats-for-throttling
Vladimir Davydov (7):
vinyl: fix accounting of secondary index cache statements
vinyl: add global memory stats
vinyl: add global disk stats
vinyl: fix force compaction logic
vinyl: update compact priority usual way on range split/coalesce
vinyl: keep track of compaction queue length and debt
vinyl: keep track of disk idle time
src/box/vinyl.c | 89 ++++++++++---------
src/box/vy_cache.c | 13 ++-
src/box/vy_cache.h | 5 +-
src/box/vy_lsm.c | 85 +++++++++++++-----
src/box/vy_lsm.h | 42 ++++++++-
src/box/vy_range.c | 42 ++++-----
src/box/vy_range.h | 17 ++--
src/box/vy_scheduler.c | 61 +++++++++++--
src/box/vy_scheduler.h | 12 +++
src/box/vy_stat.h | 39 +++++---
src/box/vy_tx.c | 19 ++++
src/box/vy_tx.h | 4 +
src/errinj.h | 1 +
test/box/errinj.result | 4 +-
test/unit/vy_iterators_helper.c | 2 +-
test/unit/vy_point_lookup.c | 2 +-
test/vinyl/cache.result | 55 +++++++++++-
test/vinyl/cache.test.lua | 26 +++++-
test/vinyl/errinj.result | 191 ++++++++++++++++++++++++++++++++++++++++
test/vinyl/errinj.test.lua | 56 ++++++++++++
test/vinyl/info.result | 159 ++++++++++++++++++++++++++++-----
test/vinyl/info.test.lua | 32 ++++++-
22 files changed, 809 insertions(+), 147 deletions(-)
--
2.11.0
next reply other threads:[~2018-09-02 20:18 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-02 20:18 Vladimir Davydov [this message]
2018-09-02 20:18 ` [PATCH 1/7] vinyl: fix accounting of secondary index cache statements Vladimir Davydov
2018-09-02 22:26 ` [tarantool-patches] " Konstantin Osipov
2018-09-02 20:18 ` [PATCH 2/7] vinyl: add global memory stats Vladimir Davydov
2018-09-02 22:27 ` [tarantool-patches] " Konstantin Osipov
2018-09-02 22:27 ` Konstantin Osipov
2018-09-03 8:10 ` Vladimir Davydov
2018-09-02 20:18 ` [PATCH 3/7] vinyl: add global disk stats Vladimir Davydov
2018-09-02 22:30 ` [tarantool-patches] " Konstantin Osipov
2018-09-02 20:18 ` [PATCH 4/7] vinyl: fix force compaction logic Vladimir Davydov
2018-09-02 20:18 ` [PATCH 5/7] vinyl: update compact priority usual way on range split/coalesce Vladimir Davydov
2018-09-02 20:18 ` [PATCH 6/7] vinyl: keep track of compaction queue length and debt Vladimir Davydov
2018-09-02 20:19 ` [PATCH 7/7] vinyl: keep track of disk idle time Vladimir Davydov
2018-09-04 11:54 ` Vladimir Davydov
2018-09-04 17:23 ` Vladimir Davydov
2018-09-04 17:23 ` [PATCH 1/8] vinyl: add helper to check whether dump is in progress Vladimir Davydov
2018-09-06 7:33 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 2/8] vinyl: don't use mempool for allocating background tasks Vladimir Davydov
2018-09-06 7:33 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 3/8] vinyl: factor out worker pool from scheduler struct Vladimir Davydov
2018-09-06 7:34 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 4/8] vinyl: move worker allocation closer to task creation Vladimir Davydov
2018-09-06 7:35 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 5/8] vinyl: use separate thread pools for dump and compaction tasks Vladimir Davydov
2018-09-06 7:37 ` Konstantin Osipov
2018-09-06 9:48 ` Vladimir Davydov
2018-09-06 10:32 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 6/8] vinyl: zap vy_worker_pool::idle_worker_count Vladimir Davydov
2018-09-06 7:38 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 7/8] vinyl: don't start scheduler fiber until local recovery is complete Vladimir Davydov
2018-09-06 7:39 ` Konstantin Osipov
2018-09-04 17:23 ` [PATCH 8/8] vinyl: keep track of thread pool idle ratio Vladimir Davydov
2018-09-06 7:49 ` Konstantin Osipov
2018-09-06 8:18 ` Vladimir Davydov
2018-09-06 10:26 ` Konstantin Osipov
2018-09-06 10:52 ` Vladimir Davydov
2018-09-06 10:57 ` Konstantin Osipov
2018-09-06 11:59 ` Vladimir Davydov
2018-09-09 11:41 ` [PATCH 0/7] vinyl: improve stats for throttling Vladimir Davydov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1535917763.git.vdavydov.dev@gmail.com \
--to=vdavydov.dev@gmail.com \
--cc=kostja@tarantool.org \
--cc=tarantool-patches@freelists.org \
--subject='Re: [PATCH 0/7] vinyl: improve stats for throttling' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox