From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH 00/12] vinyl: statistics improvements Date: Tue, 15 Jan 2019 17:17:09 +0300 Message-Id: To: tarantool-patches@freelists.org List-ID: This patch set adds a few metrics necessary for implementing compaction randomization and transaction throttling, but it's useful on its own, because it makes box.stat.vinyl() a little bit more useful when it comes to performance analysis. Here's an example of box.stat.vinyl() output with this patch set applied: --- - tx: conflict: 0 commit: 1979052 rollback: 0 statements: 2 transactions: 1 gap_locks: 0 read_views: 0 regulator: dump_bandwidth: 10485760 dump_watermark: 20023725 write_rate: 7085581 memory: tuple_cache: 0 tx: 2388 level0: 19394239 page_index: 4422529 bloom_filter: 1517177 disk: data_compacted: 500330587 data: 762493299 index: 41814873 scheduler: dump_time: 186.61679973663 tasks_inprogress: 3 dump_output: 2115930554 compaction_queue: 213022513 compaction_output: 4130054964 compaction_time: 737.99443827965 dump_count: 136 tasks_failed: 0 tasks_completed: 1839 dump_input: 2061676471 compaction_input: 5646476938 ... This patch set adds the 'scheduler' section, which exposes dump and compaction related statistics, and 'disk.data_compacted', which is needed for evaluating space amplification. See comments to individual patches for more details. Patches are mostly trivial - they primarily move pieces of code around and expose internal counters. What's still missing: - disk compression ratio - iterator statistics (in particular, read amplification) However, they aren't required for compaction randomization or throttling and so I leave them for later. https://github.com/tarantool/tarantool/commits/dv/vy-stat-improvements Vladimir Davydov (12): test: rename vinyl/info to vinyl/stat test: split vinyl/errinj vinyl: rename dump/compact in/out to input/output vinyl: rename compact to compaction vinyl: bump range version in vy_range.c vinyl: don't add dropped LSM trees to the scheduler during recovery vinyl: move global dump/compaction statistics to scheduler vinyl: add dump count to global scheduler statistics vinyl: don't account secondary indexes to scheduler.dump_input vinyl: add task accounting to global scheduler statistics vinyl: add dump/compaction time to statistics vinyl: add last level size to statistics src/box/vinyl.c | 121 +-- src/box/vy_lsm.c | 88 +- src/box/vy_lsm.h | 55 +- src/box/vy_range.c | 27 +- src/box/vy_range.h | 16 +- src/box/vy_scheduler.c | 207 +++-- src/box/vy_scheduler.h | 15 +- src/box/vy_stat.h | 57 +- test/vinyl/ddl.result | 2 +- test/vinyl/ddl.test.lua | 2 +- test/vinyl/deferred_delete.result | 16 +- test/vinyl/deferred_delete.test.lua | 16 +- test/vinyl/errinj.result | 1168 +-------------------------- test/vinyl/errinj.test.lua | 504 +----------- test/vinyl/errinj_ddl.result | 595 ++++++++++++++ test/vinyl/errinj_ddl.test.lua | 260 ++++++ test/vinyl/errinj_stat.result | 407 ++++++++++ test/vinyl/errinj_stat.test.lua | 129 +++ test/vinyl/errinj_tx.result | 435 ++++++++++ test/vinyl/errinj_tx.test.lua | 194 +++++ test/vinyl/errinj_vylog.result | 35 + test/vinyl/errinj_vylog.test.lua | 18 + test/vinyl/recovery_quota.result | 4 +- test/vinyl/recovery_quota.test.lua | 4 +- test/vinyl/snap_io_rate.result | 2 +- test/vinyl/snap_io_rate.test.lua | 2 +- test/vinyl/{info.lua => stat.lua} | 0 test/vinyl/{info.result => stat.result} | 493 ++++++++--- test/vinyl/{info.test.lua => stat.test.lua} | 93 ++- test/vinyl/suite.ini | 2 +- test/vinyl/update_optimize.result | 4 +- test/vinyl/update_optimize.test.lua | 4 +- test/vinyl/write_iterator.result | 20 +- test/vinyl/write_iterator.test.lua | 20 +- 34 files changed, 2962 insertions(+), 2053 deletions(-) create mode 100644 test/vinyl/errinj_ddl.result create mode 100644 test/vinyl/errinj_ddl.test.lua create mode 100644 test/vinyl/errinj_stat.result create mode 100644 test/vinyl/errinj_stat.test.lua create mode 100644 test/vinyl/errinj_tx.result create mode 100644 test/vinyl/errinj_tx.test.lua rename test/vinyl/{info.lua => stat.lua} (100%) rename test/vinyl/{info.result => stat.result} (80%) rename test/vinyl/{info.test.lua => stat.test.lua} (80%) -- 2.11.0