From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: tarantool-patches@freelists.org
Subject: [PATCH 00/12] vinyl: statistics improvements
Date: Tue, 15 Jan 2019 17:17:09 +0300 [thread overview]
Message-ID: <cover.1547558871.git.vdavydov.dev@gmail.com> (raw)
This patch set adds a few metrics necessary for implementing compaction
randomization and transaction throttling, but it's useful on its own,
because it makes box.stat.vinyl() a little bit more useful when it comes
to performance analysis. Here's an example of box.stat.vinyl() output
with this patch set applied:
---
- tx:
conflict: 0
commit: 1979052
rollback: 0
statements: 2
transactions: 1
gap_locks: 0
read_views: 0
regulator:
dump_bandwidth: 10485760
dump_watermark: 20023725
write_rate: 7085581
memory:
tuple_cache: 0
tx: 2388
level0: 19394239
page_index: 4422529
bloom_filter: 1517177
disk:
data_compacted: 500330587
data: 762493299
index: 41814873
scheduler:
dump_time: 186.61679973663
tasks_inprogress: 3
dump_output: 2115930554
compaction_queue: 213022513
compaction_output: 4130054964
compaction_time: 737.99443827965
dump_count: 136
tasks_failed: 0
tasks_completed: 1839
dump_input: 2061676471
compaction_input: 5646476938
...
This patch set adds the 'scheduler' section, which exposes dump and
compaction related statistics, and 'disk.data_compacted', which is
needed for evaluating space amplification. See comments to individual
patches for more details.
Patches are mostly trivial - they primarily move pieces of code around
and expose internal counters.
What's still missing:
- disk compression ratio
- iterator statistics (in particular, read amplification)
However, they aren't required for compaction randomization or throttling
and so I leave them for later.
https://github.com/tarantool/tarantool/commits/dv/vy-stat-improvements
Vladimir Davydov (12):
test: rename vinyl/info to vinyl/stat
test: split vinyl/errinj
vinyl: rename dump/compact in/out to input/output
vinyl: rename compact to compaction
vinyl: bump range version in vy_range.c
vinyl: don't add dropped LSM trees to the scheduler during recovery
vinyl: move global dump/compaction statistics to scheduler
vinyl: add dump count to global scheduler statistics
vinyl: don't account secondary indexes to scheduler.dump_input
vinyl: add task accounting to global scheduler statistics
vinyl: add dump/compaction time to statistics
vinyl: add last level size to statistics
src/box/vinyl.c | 121 +--
src/box/vy_lsm.c | 88 +-
src/box/vy_lsm.h | 55 +-
src/box/vy_range.c | 27 +-
src/box/vy_range.h | 16 +-
src/box/vy_scheduler.c | 207 +++--
src/box/vy_scheduler.h | 15 +-
src/box/vy_stat.h | 57 +-
test/vinyl/ddl.result | 2 +-
test/vinyl/ddl.test.lua | 2 +-
test/vinyl/deferred_delete.result | 16 +-
test/vinyl/deferred_delete.test.lua | 16 +-
test/vinyl/errinj.result | 1168 +--------------------------
test/vinyl/errinj.test.lua | 504 +-----------
test/vinyl/errinj_ddl.result | 595 ++++++++++++++
test/vinyl/errinj_ddl.test.lua | 260 ++++++
test/vinyl/errinj_stat.result | 407 ++++++++++
test/vinyl/errinj_stat.test.lua | 129 +++
test/vinyl/errinj_tx.result | 435 ++++++++++
test/vinyl/errinj_tx.test.lua | 194 +++++
test/vinyl/errinj_vylog.result | 35 +
test/vinyl/errinj_vylog.test.lua | 18 +
test/vinyl/recovery_quota.result | 4 +-
test/vinyl/recovery_quota.test.lua | 4 +-
test/vinyl/snap_io_rate.result | 2 +-
test/vinyl/snap_io_rate.test.lua | 2 +-
test/vinyl/{info.lua => stat.lua} | 0
test/vinyl/{info.result => stat.result} | 493 ++++++++---
test/vinyl/{info.test.lua => stat.test.lua} | 93 ++-
test/vinyl/suite.ini | 2 +-
test/vinyl/update_optimize.result | 4 +-
test/vinyl/update_optimize.test.lua | 4 +-
test/vinyl/write_iterator.result | 20 +-
test/vinyl/write_iterator.test.lua | 20 +-
34 files changed, 2962 insertions(+), 2053 deletions(-)
create mode 100644 test/vinyl/errinj_ddl.result
create mode 100644 test/vinyl/errinj_ddl.test.lua
create mode 100644 test/vinyl/errinj_stat.result
create mode 100644 test/vinyl/errinj_stat.test.lua
create mode 100644 test/vinyl/errinj_tx.result
create mode 100644 test/vinyl/errinj_tx.test.lua
rename test/vinyl/{info.lua => stat.lua} (100%)
rename test/vinyl/{info.result => stat.result} (80%)
rename test/vinyl/{info.test.lua => stat.test.lua} (80%)
--
2.11.0
next reply other threads:[~2019-01-15 14:17 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-15 14:17 Vladimir Davydov [this message]
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
2019-01-17 11:32 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 02/12] test: split vinyl/errinj Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 03/12] vinyl: rename dump/compact in/out to input/output Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 04/12] vinyl: rename compact to compaction Vladimir Davydov
2019-01-17 11:34 ` [tarantool-patches] " Konstantin Osipov
2019-01-17 12:08 ` Vladimir Davydov
2019-01-17 13:51 ` Konstantin Osipov
2019-01-15 14:17 ` [PATCH 05/12] vinyl: bump range version in vy_range.c Vladimir Davydov
2019-01-15 14:17 ` [PATCH 06/12] vinyl: don't add dropped LSM trees to the scheduler during recovery Vladimir Davydov
2019-01-15 14:17 ` [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler Vladimir Davydov
2019-01-16 16:36 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 08/12] vinyl: add dump count to global scheduler statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 09/12] vinyl: don't account secondary indexes to scheduler.dump_input Vladimir Davydov
2019-01-15 14:17 ` [PATCH 10/12] vinyl: add task accounting to global scheduler statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 11/12] vinyl: add dump/compaction time to statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 12/12] vinyl: add last level size " Vladimir Davydov
2019-01-17 11:35 ` [tarantool-patches] " Konstantin Osipov
2019-01-17 11:32 ` [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements Konstantin Osipov
2019-01-17 12:06 ` Vladimir Davydov
2019-01-20 21:16 ` Vladimir Davydov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1547558871.git.vdavydov.dev@gmail.com \
--to=vdavydov.dev@gmail.com \
--cc=tarantool-patches@freelists.org \
--subject='Re: [PATCH 00/12] vinyl: statistics improvements' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox