[PATCH 0/3] Force compaction if there too many DELETEs

[Vladimir Davydov]

Normally, space amplification of a vinyl index is confined by the
configured LSM tree shape and shouldn't exceed 3-4. However, this
might not hold in case indexed fields are frequently updated, e.g.

  s = box.schema.space.create('test', {engine = 'vinyl'})
  s:create_index('primary')
  s:create_index('secondary', {parts = {2, 'unsigned'}})

  s:update(key, {{'+', 2, 1}}) -- there a lot of updates like this

Such a workload will generate tons of DELETE statement in the secondary
index, and they won't be purged until major compaction is scheduled,
which may not happen for a very very long time, resulting in space
amplification getting as high as 100 or even 1000.

This patch set fixes this issue by forcing major compaction of a range
in case the number of DELETEs plus statements that they are supposed to
purge exceeds a half of all statements in the range. This should prevent
space and read amplification from growing to insane heights.

Note, it doesn't close #3225, because that issue is also about forcing
compaction in case the observed read amplification gets too high, which
isn't addressed by this particular patch set.

https://github.com/tarantool/tarantool/issues/3225
https://github.com/tarantool/tarantool/commits/dv/gh-3225-vy-force-compaction-when-there-are-too-many-deletes

Vladimir Davydov (3):
  vinyl: remove useless local var from vy_range_update_compact_priority
  vinyl: account disk statements of each type
  vinyl: force major compaction if there are too many DELETEs

 src/box/iproto_constants.c          |   1 +
 src/box/iproto_constants.h          |   2 +
 src/box/vinyl.c                     |   6 ++
 src/box/vy_lsm.c                    |   2 +
 src/box/vy_range.c                  |  27 +++++++--
 src/box/vy_run.c                    |  82 +++++++++++++++++++++++++-
 src/box/vy_run.h                    |  10 ++++
 src/box/vy_stat.h                   |  65 +++++++++++++++++++++
 test/vinyl/compact.result           |  40 +++++++++++++
 test/vinyl/compact.test.lua         |  18 ++++++
 test/vinyl/deferred_delete.result   |  26 +++++++--
 test/vinyl/deferred_delete.test.lua |  15 ++++-
 test/vinyl/info.result              | 113 +++++++++++++++++++++++++++++++++---
 test/vinyl/info.test.lua            |  36 ++++++++++++
 test/vinyl/layout.result            |  12 ++--
 test/vinyl/update_optimize.result   |   9 ++-
 test/vinyl/update_optimize.test.lua |   5 +-
 17 files changed, 440 insertions(+), 29 deletions(-)

-- 
2.11.0