Tarantool development patches archive
 help / color / mirror / Atom feed
* [PATCH] vinyl: do not apply run_count_per_level to the last level
@ 2018-11-16 17:00 Vladimir Davydov
  2018-11-18 16:37 ` Konstantin Osipov
  0 siblings, 1 reply; 3+ messages in thread
From: Vladimir Davydov @ 2018-11-16 17:00 UTC (permalink / raw)
  To: kostja; +Cc: tarantool-patches

Currently, run_count_per_level index option is applied to each LSM tree
level. As a result, we may end up storing each key run_count_per_level
times at the last level alone, which would result in prohibitive space
amplification. To avoid that, let's ignore run_count_per_level for the
last level.

Note, we have to tweak quite a few vinyl tests, because they implicitly
relied on the fact that producing run_count_per_level dumps would never
trigger compaction.

Closes #3657
---
https://github.com/tarantool/tarantool/issues/3657
https://github.com/tarantool/tarantool/commits/dv/gh-3657-vy-dont-store-more-than-one-run-at-last-level

 src/box/vy_range.c                  |  9 +++++
 test/vinyl/compact.result           | 23 ++++-------
 test/vinyl/compact.test.lua         | 15 +++----
 test/vinyl/deferred_delete.result   | 24 ++++++++++--
 test/vinyl/deferred_delete.test.lua | 14 +++++--
 test/vinyl/errinj.result            | 10 +++--
 test/vinyl/errinj.test.lua          |  3 +-
 test/vinyl/gc.result                | 10 +++--
 test/vinyl/gc.test.lua              |  7 ++--
 test/vinyl/info.result              |  4 +-
 test/vinyl/info.test.lua            |  4 +-
 test/vinyl/layout.result            | 78 ++++++++++++++++++++++---------------
 test/vinyl/layout.test.lua          |  3 ++
 test/vinyl/update_optimize.result   | 11 +++++-
 test/vinyl/update_optimize.test.lua |  5 ++-
 test/vinyl/upsert.result            |  8 +++-
 test/vinyl/upsert.test.lua          |  5 ++-
 17 files changed, 152 insertions(+), 81 deletions(-)

diff --git a/src/box/vy_range.c b/src/box/vy_range.c
index 0558098a..05023383 100644
--- a/src/box/vy_range.c
+++ b/src/box/vy_range.c
@@ -380,6 +380,15 @@ vy_range_update_compact_priority(struct vy_range *range,
 			est_new_run_size = total_size;
 		}
 	}
+
+	if (level_run_count > 1) {
+		/*
+		 * Do not store more than one run at the last level
+		 * to keep space amplification low.
+		 */
+		range->compact_priority = total_run_count;
+		range->compact_queue = total_stmt_count;
+	}
 }
 
 /**
diff --git a/test/vinyl/compact.result b/test/vinyl/compact.result
index a79292ee..1882c413 100644
--- a/test/vinyl/compact.result
+++ b/test/vinyl/compact.result
@@ -68,19 +68,6 @@ box.snapshot() -- create the second run
 ---
 - ok
 ...
-vyinfo().run_count == 2
----
-- true
-...
--- create a few more runs to trigger compaction
-space:insert({3, 3})
----
-- [3, 3]
-...
-box.snapshot()
----
-- ok
-...
 -- wait for compaction
 while vyinfo().run_count >= 2 do fiber.sleep(0.1) end
 ---
@@ -110,8 +97,9 @@ test_run:cmd("setopt delimiter ';'")
 ---
 - true
 ...
-function dump()
-    for i = 1, 10 do
+function dump(big)
+    local step = big and 1 or 5
+    for i = 1, 20, step do
         s:replace{i, digest.urandom(1000)}
     end
     box.snapshot()
@@ -135,7 +123,10 @@ test_run:cmd("setopt delimiter ''");
 ---
 - true
 ...
-dump()
+-- The first run should be big enough to prevent major compaction
+-- from kicking in on the next dump, because run_count_per_level
+-- is ignored on the last level (see gh-3657).
+dump(true)
 ---
 ...
 dump()
diff --git a/test/vinyl/compact.test.lua b/test/vinyl/compact.test.lua
index b9a1c109..6c1e5731 100644
--- a/test/vinyl/compact.test.lua
+++ b/test/vinyl/compact.test.lua
@@ -26,10 +26,6 @@ vyinfo().run_count == 1
 space:replace({2,2})
 space:upsert({2},{{'=',4,5}}) -- bad upsert
 box.snapshot() -- create the second run
-vyinfo().run_count == 2
--- create a few more runs to trigger compaction
-space:insert({3, 3})
-box.snapshot()
 
 -- wait for compaction
 while vyinfo().run_count >= 2 do fiber.sleep(0.1) end
@@ -47,8 +43,9 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
 _ = s:create_index('pk', {run_count_per_level = 100, page_size = 128, range_size = 1024})
 
 test_run:cmd("setopt delimiter ';'")
-function dump()
-    for i = 1, 10 do
+function dump(big)
+    local step = big and 1 or 5
+    for i = 1, 20, step do
         s:replace{i, digest.urandom(1000)}
     end
     box.snapshot()
@@ -66,7 +63,11 @@ function compact()
 end;
 test_run:cmd("setopt delimiter ''");
 
-dump()
+-- The first run should be big enough to prevent major compaction
+-- from kicking in on the next dump, because run_count_per_level
+-- is ignored on the last level (see gh-3657).
+dump(true)
+
 dump()
 dump()
 info() -- 1 range, 3 runs
diff --git a/test/vinyl/deferred_delete.result b/test/vinyl/deferred_delete.result
index 4a68c6c2..9a823d59 100644
--- a/test/vinyl/deferred_delete.result
+++ b/test/vinyl/deferred_delete.result
@@ -662,6 +662,9 @@ test_run:cmd("switch test")
 fiber = require('fiber')
 ---
 ...
+digest = require('digest')
+---
+...
 s = box.schema.space.create('test', {engine = 'vinyl'})
 ---
 ...
@@ -671,6 +674,18 @@ pk = s:create_index('pk', {run_count_per_level = 10})
 sk = s:create_index('sk', {run_count_per_level = 10, parts = {2, 'unsigned', 3, 'string'}, unique = false})
 ---
 ...
+-- Write a run big enough to prevent major compaction from kicking in
+-- (run_count_per_level is ignored on the last level - see gh-3657).
+dummy_rows = 100
+---
+...
+for i = 1, dummy_rows do s:replace{i + 1000, i + 1000, digest.urandom(100)} end
+---
+...
+box.snapshot()
+---
+- ok
+...
 pad = string.rep('x', 10 * 1024)
 ---
 ...
@@ -691,7 +706,7 @@ box.snapshot()
 ---
 - ok
 ...
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs
 ---
 - 240
 ...
@@ -711,7 +726,7 @@ sk:stat().disk.dump.count -- 1
 ---
 - 1
 ...
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
 ---
 - 360
 ...
@@ -725,10 +740,13 @@ pk = s.index.pk
 sk = s.index.sk
 ---
 ...
+dummy_rows = 100
+---
+...
 -- Should be 360, the same amount of statements as before restart.
 -- If we applied all deferred DELETEs, including the dumped ones,
 -- then there would be more.
-sk:stat().rows
+sk:stat().rows - dummy_rows
 ---
 - 360
 ...
diff --git a/test/vinyl/deferred_delete.test.lua b/test/vinyl/deferred_delete.test.lua
index e9e0e293..8d2cd0c1 100644
--- a/test/vinyl/deferred_delete.test.lua
+++ b/test/vinyl/deferred_delete.test.lua
@@ -248,11 +248,18 @@ test_run:cmd("start server test with args='1048576'")
 test_run:cmd("switch test")
 
 fiber = require('fiber')
+digest = require('digest')
 
 s = box.schema.space.create('test', {engine = 'vinyl'})
 pk = s:create_index('pk', {run_count_per_level = 10})
 sk = s:create_index('sk', {run_count_per_level = 10, parts = {2, 'unsigned', 3, 'string'}, unique = false})
 
+-- Write a run big enough to prevent major compaction from kicking in
+-- (run_count_per_level is ignored on the last level - see gh-3657).
+dummy_rows = 100
+for i = 1, dummy_rows do s:replace{i + 1000, i + 1000, digest.urandom(100)} end
+box.snapshot()
+
 pad = string.rep('x', 10 * 1024)
 for i = 1, 120 do s:replace{i, i, pad} end
 box.snapshot()
@@ -261,7 +268,7 @@ pad = string.rep('y', 10 * 1024)
 for i = 1, 120 do s:replace{i, i, pad} end
 box.snapshot()
 
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs
 
 box.stat.reset()
 
@@ -273,17 +280,18 @@ while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
 
 sk:stat().disk.dump.count -- 1
 
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
 
 test_run:cmd("restart server test with args='1048576'")
 s = box.space.test
 pk = s.index.pk
 sk = s.index.sk
+dummy_rows = 100
 
 -- Should be 360, the same amount of statements as before restart.
 -- If we applied all deferred DELETEs, including the dumped ones,
 -- then there would be more.
-sk:stat().rows
+sk:stat().rows - dummy_rows
 
 s:drop()
 
diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result
index 63d14c61..53345949 100644
--- a/test/vinyl/errinj.result
+++ b/test/vinyl/errinj.result
@@ -1867,6 +1867,10 @@ s:replace{1, 10}
 ---
 - [1, 10]
 ...
+s:replace{10, 100} -- to prevent last-level compaction (gh-3657)
+---
+- [10, 100]
+...
 box.snapshot()
 ---
 - ok
@@ -2137,9 +2141,6 @@ function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
 dump()
 ---
 ...
-dump()
----
-...
 i:stat().disk.compact.queue -- none
 ---
 - bytes_compressed: <bytes_compressed>
@@ -2158,6 +2159,9 @@ errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
 dump()
 ---
 ...
+dump()
+---
+...
 i:stat().disk.compact.queue -- 30 statements
 ---
 - bytes_compressed: <bytes_compressed>
diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua
index b6f697d3..c9293a38 100644
--- a/test/vinyl/errinj.test.lua
+++ b/test/vinyl/errinj.test.lua
@@ -748,6 +748,7 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
 _ = s:create_index('pk', {run_count_per_level = 10})
 _ = s:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
 s:replace{1, 10}
+s:replace{10, 100} -- to prevent last-level compaction (gh-3657)
 box.snapshot()
 s:replace{1, 20}
 box.snapshot()
@@ -860,11 +861,11 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
 i = s:create_index('pk', {run_count_per_level = 2})
 function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
 dump()
-dump()
 i:stat().disk.compact.queue -- none
 i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
 errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
 dump()
+dump()
 i:stat().disk.compact.queue -- 30 statements
 i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
 dump()
diff --git a/test/vinyl/gc.result b/test/vinyl/gc.result
index 3931219b..098c17c2 100644
--- a/test/vinyl/gc.result
+++ b/test/vinyl/gc.result
@@ -168,11 +168,14 @@ function count_runs() return #fio.glob(fio.pathjoin(box.cfg.vinyl_dir, s.id, s.i
 _ = s:replace{1}
 ---
 ...
+_ = s:replace{2} -- to prevent last-level compaction (gh-3657)
+---
+...
 box.snapshot()
 ---
 - ok
 ...
-_ = s:replace{2}
+_ = s:replace{3}
 ---
 ...
 box.snapshot()
@@ -186,12 +189,11 @@ count_runs() -- 2
 for i = 1, 20 do s:replace{i, string.rep('x', 100 * 1024)} end
 ---
 ...
-while s.index.pk:stat().disk.compact.count < 1 do fiber.sleep(0.001) end
+s.index.pk:compact()
 ---
 ...
-s.index.pk:stat().disk.compact.count -- 1
+while s.index.pk:stat().run_count > 1 do fiber.sleep(0.01) end
 ---
-- 1
 ...
 count_runs() -- 3 (compacted runs created after checkpoint are deleted)
 ---
diff --git a/test/vinyl/gc.test.lua b/test/vinyl/gc.test.lua
index 164fac12..b5e42d6b 100644
--- a/test/vinyl/gc.test.lua
+++ b/test/vinyl/gc.test.lua
@@ -84,15 +84,16 @@ _ = s:create_index('pk', {run_count_per_level = 3})
 function count_runs() return #fio.glob(fio.pathjoin(box.cfg.vinyl_dir, s.id, s.index.pk.id, '*.run')) end
 
 _ = s:replace{1}
+_ = s:replace{2} -- to prevent last-level compaction (gh-3657)
 box.snapshot()
-_ = s:replace{2}
+_ = s:replace{3}
 box.snapshot()
 
 count_runs() -- 2
 
 for i = 1, 20 do s:replace{i, string.rep('x', 100 * 1024)} end
-while s.index.pk:stat().disk.compact.count < 1 do fiber.sleep(0.001) end
-s.index.pk:stat().disk.compact.count -- 1
+s.index.pk:compact()
+while s.index.pk:stat().run_count > 1 do fiber.sleep(0.01) end
 
 count_runs() -- 3 (compacted runs created after checkpoint are deleted)
 
diff --git a/test/vinyl/info.result b/test/vinyl/info.result
index a47c9a1b..5a2935df 100644
--- a/test/vinyl/info.result
+++ b/test/vinyl/info.result
@@ -1298,7 +1298,7 @@ box.snapshot()
 i1:compact()
 ---
 ...
-wait(function() return i1:stat() end, st1, 'disk.compact.count', 1)
+while i1:stat().run_count > 1 do fiber.sleep(0.01) end
 ---
 ...
 box.snapshot()
@@ -1308,7 +1308,7 @@ box.snapshot()
 i2:compact()
 ---
 ...
-wait(function() return i2:stat() end, st2, 'disk.compact.count', 1)
+while i2:stat().run_count > 1 do fiber.sleep(0.01) end
 ---
 ...
 gst = gstat()
diff --git a/test/vinyl/info.test.lua b/test/vinyl/info.test.lua
index e5794a23..4e1a5207 100644
--- a/test/vinyl/info.test.lua
+++ b/test/vinyl/info.test.lua
@@ -383,10 +383,10 @@ i2:bsize() == st2.memory.index_size + st2.disk.index_size + st2.disk.bloom_size
 -- Then dump them and compact the secondary index.
 box.snapshot()
 i1:compact()
-wait(function() return i1:stat() end, st1, 'disk.compact.count', 1)
+while i1:stat().run_count > 1 do fiber.sleep(0.01) end
 box.snapshot()
 i2:compact()
-wait(function() return i2:stat() end, st2, 'disk.compact.count', 1)
+while i2:stat().run_count > 1 do fiber.sleep(0.01) end
 gst = gstat()
 st1 = i1:stat()
 st2 = i2:stat()
diff --git a/test/vinyl/layout.result b/test/vinyl/layout.result
index ee14cd51..1fd035c9 100644
--- a/test/vinyl/layout.result
+++ b/test/vinyl/layout.result
@@ -38,6 +38,8 @@ box.snapshot()
 space.index.sk:alter{parts = {{2, 'unsigned', is_nullable = true}}}
 ---
 ...
+-- Note, the first run is bigger than the second one to prevent
+-- last-level compaction (gh-3657).
 space:replace{'ЭЭЭ', box.NULL}
 ---
 - ['ЭЭЭ', null]
@@ -50,6 +52,10 @@ space:replace{'ёёё', box.NULL}
 ---
 - ['ёёё', null]
 ...
+space:replace{'ЭЮЯ', 666}
+---
+- ['ЭЮЯ', 666]
+...
 box.snapshot()
 ---
 - ok
@@ -126,16 +132,16 @@ test_run:cmd("push filter 'offset: .*' to 'offset: <offset>'")
 ...
 result
 ---
-- - - 00000000000000000010.vylog
+- - - 00000000000000000011.vylog
     - - HEADER:
           type: INSERT
         BODY:
           tuple: [0, {6: 512, 7: [{'field': 0, 'collation': 1, 'type': 'string'}],
-              9: 10, 12: 3, 13: 7}]
+              9: 11, 12: 3, 13: 7}]
       - HEADER:
           type: INSERT
         BODY:
-          tuple: [5, {2: 8, 9: 10}]
+          tuple: [5, {2: 8, 9: 11}]
       - HEADER:
           type: INSERT
         BODY:
@@ -156,11 +162,11 @@ result
           type: INSERT
         BODY:
           tuple: [0, {0: 2, 5: 1, 6: 512, 7: [{'field': 1, 'is_nullable': true, 'type': 'unsigned'}],
-              9: 10, 12: 4, 13: 7}]
+              9: 11, 12: 4, 13: 7}]
       - HEADER:
           type: INSERT
         BODY:
-          tuple: [5, {0: 2, 2: 6, 9: 10}]
+          tuple: [5, {0: 2, 2: 6, 9: 11}]
       - HEADER:
           type: INSERT
         BODY:
@@ -200,7 +206,7 @@ result
           timestamp: <timestamp>
           type: INSERT
         BODY:
-          tuple: [5, {0: 2, 2: 10, 9: 13}]
+          tuple: [5, {0: 2, 2: 10, 9: 14}]
       - HEADER:
           timestamp: <timestamp>
           type: INSERT
@@ -210,7 +216,7 @@ result
           timestamp: <timestamp>
           type: INSERT
         BODY:
-          tuple: [10, {0: 2, 9: 13}]
+          tuple: [10, {0: 2, 9: 14}]
       - HEADER:
           timestamp: <timestamp>
           type: INSERT
@@ -220,7 +226,7 @@ result
           timestamp: <timestamp>
           type: INSERT
         BODY:
-          tuple: [5, {2: 12, 9: 13}]
+          tuple: [5, {2: 12, 9: 14}]
       - HEADER:
           timestamp: <timestamp>
           type: INSERT
@@ -230,25 +236,25 @@ result
           timestamp: <timestamp>
           type: INSERT
         BODY:
-          tuple: [10, {9: 13}]
+          tuple: [10, {9: 14}]
   - - 00000000000000000008.index
     - - HEADER:
           type: RUNINFO
         BODY:
           min_lsn: 8
-          max_key: ['ЭЭЭ']
+          max_key: ['ЭЮЯ']
           page_count: 1
           bloom_filter: <bloom_filter>
-          max_lsn: 10
+          max_lsn: 11
           min_key: ['ёёё']
       - HEADER:
           type: PAGEINFO
         BODY:
           row_index_offset: <offset>
           offset: <offset>
-          size: 86
-          unpacked_size: 67
-          row_count: 3
+          size: 108
+          unpacked_size: 89
+          row_count: 4
           min_key: ['ёёё']
   - - 00000000000000000008.run
     - - HEADER:
@@ -267,18 +273,23 @@ result
         BODY:
           tuple: ['ЭЭЭ', null]
       - HEADER:
+          lsn: 11
+          type: REPLACE
+        BODY:
+          tuple: ['ЭЮЯ', 666]
+      - HEADER:
           type: ROWINDEX
         BODY:
-          row_index: "\0\0\0\0\0\0\0\x10\0\0\0 "
+          row_index: "\0\0\0\0\0\0\0\x10\0\0\0 \0\0\00"
   - - 00000000000000000012.index
     - - HEADER:
           type: RUNINFO
         BODY:
-          min_lsn: 11
+          min_lsn: 12
           max_key: ['ЮЮЮ']
           page_count: 1
           bloom_filter: <bloom_filter>
-          max_lsn: 13
+          max_lsn: 14
           min_key: ['ёёё']
       - HEADER:
           type: PAGEINFO
@@ -291,19 +302,19 @@ result
           min_key: ['ёёё']
   - - 00000000000000000012.run
     - - HEADER:
-          lsn: 11
+          lsn: 12
           type: REPLACE
         BODY:
           tuple: ['ёёё', 123]
           tuple_meta: {1: 1}
       - HEADER:
-          lsn: 13
+          lsn: 14
           type: REPLACE
         BODY:
           tuple: ['ююю', 789]
           tuple_meta: {1: 1}
       - HEADER:
-          lsn: 12
+          lsn: 13
           type: REPLACE
         BODY:
           tuple: ['ЮЮЮ', 456]
@@ -317,19 +328,19 @@ result
           type: RUNINFO
         BODY:
           min_lsn: 8
-          max_key: [null, 'ЭЭЭ']
+          max_key: [666, 'ЭЮЯ']
           page_count: 1
           bloom_filter: <bloom_filter>
-          max_lsn: 10
+          max_lsn: 11
           min_key: [null, 'ёёё']
       - HEADER:
           type: PAGEINFO
         BODY:
           row_index_offset: <offset>
           offset: <offset>
-          size: 86
-          unpacked_size: 67
-          row_count: 3
+          size: 108
+          unpacked_size: 89
+          row_count: 4
           min_key: [null, 'ёёё']
   - - 00000000000000000006.run
     - - HEADER:
@@ -348,18 +359,23 @@ result
         BODY:
           tuple: [null, 'ЭЭЭ']
       - HEADER:
+          lsn: 11
+          type: REPLACE
+        BODY:
+          tuple: [666, 'ЭЮЯ']
+      - HEADER:
           type: ROWINDEX
         BODY:
-          row_index: "\0\0\0\0\0\0\0\x10\0\0\0 "
+          row_index: "\0\0\0\0\0\0\0\x10\0\0\0 \0\0\00"
   - - 00000000000000000010.index
     - - HEADER:
           type: RUNINFO
         BODY:
-          min_lsn: 11
+          min_lsn: 12
           max_key: [789, 'ююю']
           page_count: 1
           bloom_filter: <bloom_filter>
-          max_lsn: 13
+          max_lsn: 14
           min_key: [123, 'ёёё']
       - HEADER:
           type: PAGEINFO
@@ -372,17 +388,17 @@ result
           min_key: [123, 'ёёё']
   - - 00000000000000000010.run
     - - HEADER:
-          lsn: 11
+          lsn: 12
           type: REPLACE
         BODY:
           tuple: [123, 'ёёё']
       - HEADER:
-          lsn: 12
+          lsn: 13
           type: REPLACE
         BODY:
           tuple: [456, 'ЮЮЮ']
       - HEADER:
-          lsn: 13
+          lsn: 14
           type: REPLACE
         BODY:
           tuple: [789, 'ююю']
diff --git a/test/vinyl/layout.test.lua b/test/vinyl/layout.test.lua
index 27bf5d40..60f22c76 100644
--- a/test/vinyl/layout.test.lua
+++ b/test/vinyl/layout.test.lua
@@ -17,9 +17,12 @@ box.snapshot()
 
 space.index.sk:alter{parts = {{2, 'unsigned', is_nullable = true}}}
 
+-- Note, the first run is bigger than the second one to prevent
+-- last-level compaction (gh-3657).
 space:replace{'ЭЭЭ', box.NULL}
 space:replace{'эээ', box.NULL}
 space:replace{'ёёё', box.NULL}
+space:replace{'ЭЮЯ', 666}
 box.snapshot()
 
 space:replace{'ёёё', 123}
diff --git a/test/vinyl/update_optimize.result b/test/vinyl/update_optimize.result
index 40f34151..bf9dff7e 100644
--- a/test/vinyl/update_optimize.result
+++ b/test/vinyl/update_optimize.result
@@ -728,6 +728,10 @@ s:insert{1, 10}
 ---
 - [1, 10]
 ...
+s:insert{10, 100} -- to prevent last-level compaction (gh-3657)
+---
+- [10, 100]
+...
 box.snapshot()
 ---
 - ok
@@ -743,9 +747,11 @@ box.snapshot()
 ---
 - ok
 ...
-s.index.sk:stat().rows -- INSERT in the first run + DELETE the second run
+-- Should be 3: INSERT{10, 1} and INSERT{100, 10} in the first run
+-- plus DELETE{10, 1} in the second run.
+s.index.sk:stat().rows
 ---
-- 2
+- 3
 ...
 s:insert{1, 20}
 ---
@@ -754,6 +760,7 @@ s:insert{1, 20}
 s.index.sk:select()
 ---
 - - [1, 20]
+  - [10, 100]
 ...
 s:drop()
 ---
diff --git a/test/vinyl/update_optimize.test.lua b/test/vinyl/update_optimize.test.lua
index a214dcae..67efba31 100644
--- a/test/vinyl/update_optimize.test.lua
+++ b/test/vinyl/update_optimize.test.lua
@@ -244,13 +244,16 @@ _ = s:create_index('pk')
 _ = s:create_index('sk', {parts = {2, 'unsigned'}, run_count_per_level = 10})
 
 s:insert{1, 10}
+s:insert{10, 100} -- to prevent last-level compaction (gh-3657)
 box.snapshot()
 
 s:update(1, {{'=', 2, 10}})
 s:delete(1)
 box.snapshot()
 
-s.index.sk:stat().rows -- INSERT in the first run + DELETE the second run
+-- Should be 3: INSERT{10, 1} and INSERT{100, 10} in the first run
+-- plus DELETE{10, 1} in the second run.
+s.index.sk:stat().rows
 
 s:insert{1, 20}
 s.index.sk:select()
diff --git a/test/vinyl/upsert.result b/test/vinyl/upsert.result
index e6c451d2..6c6d03d6 100644
--- a/test/vinyl/upsert.result
+++ b/test/vinyl/upsert.result
@@ -676,6 +676,10 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
 i = s:create_index('test', { run_count_per_level = 20 })
 ---
 ...
+-- Write a big run to prevent last-level compaction (gh-3657).
+for i = 101, 110 do s:replace{i, require('digest').urandom(50)} end
+---
+...
 s:replace({1, 1})
 ---
 - [1, 1]
@@ -690,9 +694,9 @@ s:upsert({1, 1}, {{'+', 1, 1}}) -- ignored due to primary key changed
 s:upsert({1, 1}, {{'+', 2, 1}}) -- applied to the previous statement
 ---
 ...
-s:select()
+s:get(1)
 ---
-- - [1, 2]
+- [1, 2]
 ...
 --
 -- gh-2520 use cache as a hint when applying upserts.
diff --git a/test/vinyl/upsert.test.lua b/test/vinyl/upsert.test.lua
index cb7cc81a..ef83e0f6 100644
--- a/test/vinyl/upsert.test.lua
+++ b/test/vinyl/upsert.test.lua
@@ -274,11 +274,14 @@ space:drop()
 s = box.schema.space.create('test', {engine = 'vinyl'})
 i = s:create_index('test', { run_count_per_level = 20 })
 
+-- Write a big run to prevent last-level compaction (gh-3657).
+for i = 101, 110 do s:replace{i, require('digest').urandom(50)} end
+
 s:replace({1, 1})
 box.snapshot()
 s:upsert({1, 1}, {{'+', 1, 1}}) -- ignored due to primary key changed
 s:upsert({1, 1}, {{'+', 2, 1}}) -- applied to the previous statement
-s:select()
+s:get(1)
 
 --
 -- gh-2520 use cache as a hint when applying upserts.
-- 
2.11.0

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] vinyl: do not apply run_count_per_level to the last level
  2018-11-16 17:00 [PATCH] vinyl: do not apply run_count_per_level to the last level Vladimir Davydov
@ 2018-11-18 16:37 ` Konstantin Osipov
  2018-11-19  8:20   ` Vladimir Davydov
  0 siblings, 1 reply; 3+ messages in thread
From: Konstantin Osipov @ 2018-11-18 16:37 UTC (permalink / raw)
  To: Vladimir Davydov; +Cc: tarantool-patches

* Vladimir Davydov <vdavydov.dev@gmail.com> [18/11/17 07:05]:
> Currently, run_count_per_level index option is applied to each LSM tree
> level. As a result, we may end up storing each key run_count_per_level
> times at the last level alone, which would result in prohibitive space
> amplification. To avoid that, let's ignore run_count_per_level for the
> last level.
> 
> Note, we have to tweak quite a few vinyl tests, because they implicitly
> relied on the fact that producing run_count_per_level dumps would never
> trigger compaction.

OK to push.

 

-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] vinyl: do not apply run_count_per_level to the last level
  2018-11-18 16:37 ` Konstantin Osipov
@ 2018-11-19  8:20   ` Vladimir Davydov
  0 siblings, 0 replies; 3+ messages in thread
From: Vladimir Davydov @ 2018-11-19  8:20 UTC (permalink / raw)
  To: Konstantin Osipov; +Cc: tarantool-patches

On Sun, Nov 18, 2018 at 07:37:52PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev@gmail.com> [18/11/17 07:05]:
> > Currently, run_count_per_level index option is applied to each LSM tree
> > level. As a result, we may end up storing each key run_count_per_level
> > times at the last level alone, which would result in prohibitive space
> > amplification. To avoid that, let's ignore run_count_per_level for the
> > last level.
> > 
> > Note, we have to tweak quite a few vinyl tests, because they implicitly
> > relied on the fact that producing run_count_per_level dumps would never
> > trigger compaction.
> 
> OK to push.

Pushed to 1.10.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2018-11-19  8:20 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-16 17:00 [PATCH] vinyl: do not apply run_count_per_level to the last level Vladimir Davydov
2018-11-18 16:37 ` Konstantin Osipov
2018-11-19  8:20   ` Vladimir Davydov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox