[PATCH 0/9] vinyl: compaction randomization and throttling

Vladimir Davydov vdavydov.dev at gmail.com
Mon Jan 21 00:16:59 MSK 2019


This patch set achieves two goals. The first one is randomizing
compaction pace among ranges so as to avoid IO load bursts, which badly
affect read and space amplification and complicate implementation of
transaction throttling. The second goal is making sure that compaction
always keeps up with dumps and throttling transactions if it doesn't.
For more details, see comments to the individual patches and the GitHub
issues.

https://github.com/tarantool/tarantool/issues/3944
https://github.com/tarantool/tarantool/issues/3721
https://github.com/tarantool/tarantool/commits/dv/gh-3721-3944-vy-compaction-randomization-and-throttling

Vladimir Davydov (9):
  vinyl: update lsm->range_heap in one go on dump completion
  vinyl: ignore unknown .run, .index and .vylog keys
  vinyl: use uncompressed run size for range split/coalesce/compaction
  vinyl: rename lsm->range_heap to max_compaction_priority
  vinyl: keep track of dumps per compaction for each LSM tree
  vinyl: set range size automatically
  vinyl: randomize range compaction to avoid IO load spikes
  vinyl: introduce quota consumer types
  vinyl: throttle tx to ensure compaction keeps up with dumps

 src/box/alter.cc                   |   8 +-
 src/box/box.cc                     |   6 +-
 src/box/index_def.c                |   2 +-
 src/box/lua/load_cfg.lua           |   2 +-
 src/box/lua/space.cc               |   6 +-
 src/box/vinyl.c                    |  39 +++++--
 src/box/vy_log.c                   |  33 ++++--
 src/box/vy_log.h                   |  10 +-
 src/box/vy_lsm.c                   |  83 ++++++++++++--
 src/box/vy_lsm.h                   |  12 +-
 src/box/vy_quota.c                 | 132 +++++++++++++++++-----
 src/box/vy_quota.h                 |  97 ++++++++++++++---
 src/box/vy_range.c                 |  55 ++++++++--
 src/box/vy_range.h                 |  60 +++++++---
 src/box/vy_regulator.c             |  96 ++++++++++++++--
 src/box/vy_regulator.h             |  27 +++++
 src/box/vy_run.c                   |  13 +--
 src/box/vy_run.h                   |  20 ++++
 src/box/vy_scheduler.c             |  40 ++++---
 test/app-tap/init_script.result    |  21 ++--
 test/box-tap/cfg.test.lua          |   3 +-
 test/box/admin.result              |   2 -
 test/box/cfg.result                |   4 -
 test/vinyl/ddl.result              |   5 -
 test/vinyl/ddl.test.lua            |   1 -
 test/vinyl/layout.result           |   8 +-
 test/vinyl/misc.result             |  78 +++++++++++++
 test/vinyl/misc.test.lua           |  26 +++++
 test/vinyl/stat.result             | 217 +++++++++++++++++++++++++++++--------
 test/vinyl/stat.test.lua           |  57 ++++++++++
 test/vinyl/write_iterator.result   |  16 +++
 test/vinyl/write_iterator.test.lua |   6 +
 32 files changed, 962 insertions(+), 223 deletions(-)

-- 
2.11.0




More information about the Tarantool-patches mailing list