[PATCH 00/12] vinyl: statistics improvements

[Vladimir Davydov]

This patch set adds a few metrics necessary for implementing compaction
randomization and transaction throttling, but it's useful on its own,
because it makes box.stat.vinyl() a little bit more useful when it comes
to performance analysis. Here's an example of box.stat.vinyl() output
with this patch set applied:

---
- tx:
    conflict: 0
    commit: 1979052
    rollback: 0
    statements: 2
    transactions: 1
    gap_locks: 0
    read_views: 0
  regulator:
    dump_bandwidth: 10485760
    dump_watermark: 20023725
    write_rate: 7085581
  memory:
    tuple_cache: 0
    tx: 2388
    level0: 19394239
    page_index: 4422529
    bloom_filter: 1517177
  disk:
    data_compacted: 500330587
    data: 762493299
    index: 41814873
  scheduler:
    dump_time: 186.61679973663
    tasks_inprogress: 3
    dump_output: 2115930554
    compaction_queue: 213022513
    compaction_output: 4130054964
    compaction_time: 737.99443827965
    dump_count: 136
    tasks_failed: 0
    tasks_completed: 1839
    dump_input: 2061676471
    compaction_input: 5646476938
...

This patch set adds the 'scheduler' section, which exposes dump and
compaction related statistics, and 'disk.data_compacted', which is
needed for evaluating space amplification. See comments to individual
patches for more details.

Patches are mostly trivial - they primarily move pieces of code around
and expose internal counters.

What's still missing:
 - disk compression ratio
 - iterator statistics (in particular, read amplification)

However, they aren't required for compaction randomization or throttling
and so I leave them for later.

https://github.com/tarantool/tarantool/commits/dv/vy-stat-improvements

Vladimir Davydov (12):
  test: rename vinyl/info to vinyl/stat
  test: split vinyl/errinj
  vinyl: rename dump/compact in/out to input/output
  vinyl: rename compact to compaction
  vinyl: bump range version in vy_range.c
  vinyl: don't add dropped LSM trees to the scheduler during recovery
  vinyl: move global dump/compaction statistics to scheduler
  vinyl: add dump count to global scheduler statistics
  vinyl: don't account secondary indexes to scheduler.dump_input
  vinyl: add task accounting to global scheduler statistics
  vinyl: add dump/compaction time to statistics
  vinyl: add last level size to statistics

 src/box/vinyl.c                             |  121 +--
 src/box/vy_lsm.c                            |   88 +-
 src/box/vy_lsm.h                            |   55 +-
 src/box/vy_range.c                          |   27 +-
 src/box/vy_range.h                          |   16 +-
 src/box/vy_scheduler.c                      |  207 +++--
 src/box/vy_scheduler.h                      |   15 +-
 src/box/vy_stat.h                           |   57 +-
 test/vinyl/ddl.result                       |    2 +-
 test/vinyl/ddl.test.lua                     |    2 +-
 test/vinyl/deferred_delete.result           |   16 +-
 test/vinyl/deferred_delete.test.lua         |   16 +-
 test/vinyl/errinj.result                    | 1168 +--------------------------
 test/vinyl/errinj.test.lua                  |  504 +-----------
 test/vinyl/errinj_ddl.result                |  595 ++++++++++++++
 test/vinyl/errinj_ddl.test.lua              |  260 ++++++
 test/vinyl/errinj_stat.result               |  407 ++++++++++
 test/vinyl/errinj_stat.test.lua             |  129 +++
 test/vinyl/errinj_tx.result                 |  435 ++++++++++
 test/vinyl/errinj_tx.test.lua               |  194 +++++
 test/vinyl/errinj_vylog.result              |   35 +
 test/vinyl/errinj_vylog.test.lua            |   18 +
 test/vinyl/recovery_quota.result            |    4 +-
 test/vinyl/recovery_quota.test.lua          |    4 +-
 test/vinyl/snap_io_rate.result              |    2 +-
 test/vinyl/snap_io_rate.test.lua            |    2 +-
 test/vinyl/{info.lua => stat.lua}           |    0
 test/vinyl/{info.result => stat.result}     |  493 ++++++++---
 test/vinyl/{info.test.lua => stat.test.lua} |   93 ++-
 test/vinyl/suite.ini                        |    2 +-
 test/vinyl/update_optimize.result           |    4 +-
 test/vinyl/update_optimize.test.lua         |    4 +-
 test/vinyl/write_iterator.result            |   20 +-
 test/vinyl/write_iterator.test.lua          |   20 +-
 34 files changed, 2962 insertions(+), 2053 deletions(-)
 create mode 100644 test/vinyl/errinj_ddl.result
 create mode 100644 test/vinyl/errinj_ddl.test.lua
 create mode 100644 test/vinyl/errinj_stat.result
 create mode 100644 test/vinyl/errinj_stat.test.lua
 create mode 100644 test/vinyl/errinj_tx.result
 create mode 100644 test/vinyl/errinj_tx.test.lua
 rename test/vinyl/{info.lua => stat.lua} (100%)
 rename test/vinyl/{info.result => stat.result} (80%)
 rename test/vinyl/{info.test.lua => stat.test.lua} (80%)

-- 
2.11.0