* [PATCH] vinyl: do not apply run_count_per_level to the last level
@ 2018-11-16 17:00 Vladimir Davydov
2018-11-18 16:37 ` Konstantin Osipov
0 siblings, 1 reply; 3+ messages in thread
From: Vladimir Davydov @ 2018-11-16 17:00 UTC (permalink / raw)
To: kostja; +Cc: tarantool-patches
Currently, run_count_per_level index option is applied to each LSM tree
level. As a result, we may end up storing each key run_count_per_level
times at the last level alone, which would result in prohibitive space
amplification. To avoid that, let's ignore run_count_per_level for the
last level.
Note, we have to tweak quite a few vinyl tests, because they implicitly
relied on the fact that producing run_count_per_level dumps would never
trigger compaction.
Closes #3657
---
https://github.com/tarantool/tarantool/issues/3657
https://github.com/tarantool/tarantool/commits/dv/gh-3657-vy-dont-store-more-than-one-run-at-last-level
src/box/vy_range.c | 9 +++++
test/vinyl/compact.result | 23 ++++-------
test/vinyl/compact.test.lua | 15 +++----
test/vinyl/deferred_delete.result | 24 ++++++++++--
test/vinyl/deferred_delete.test.lua | 14 +++++--
test/vinyl/errinj.result | 10 +++--
test/vinyl/errinj.test.lua | 3 +-
test/vinyl/gc.result | 10 +++--
test/vinyl/gc.test.lua | 7 ++--
test/vinyl/info.result | 4 +-
test/vinyl/info.test.lua | 4 +-
test/vinyl/layout.result | 78 ++++++++++++++++++++++---------------
test/vinyl/layout.test.lua | 3 ++
test/vinyl/update_optimize.result | 11 +++++-
test/vinyl/update_optimize.test.lua | 5 ++-
test/vinyl/upsert.result | 8 +++-
test/vinyl/upsert.test.lua | 5 ++-
17 files changed, 152 insertions(+), 81 deletions(-)
diff --git a/src/box/vy_range.c b/src/box/vy_range.c
index 0558098a..05023383 100644
--- a/src/box/vy_range.c
+++ b/src/box/vy_range.c
@@ -380,6 +380,15 @@ vy_range_update_compact_priority(struct vy_range *range,
est_new_run_size = total_size;
}
}
+
+ if (level_run_count > 1) {
+ /*
+ * Do not store more than one run at the last level
+ * to keep space amplification low.
+ */
+ range->compact_priority = total_run_count;
+ range->compact_queue = total_stmt_count;
+ }
}
/**
diff --git a/test/vinyl/compact.result b/test/vinyl/compact.result
index a79292ee..1882c413 100644
--- a/test/vinyl/compact.result
+++ b/test/vinyl/compact.result
@@ -68,19 +68,6 @@ box.snapshot() -- create the second run
---
- ok
...
-vyinfo().run_count == 2
----
-- true
-...
--- create a few more runs to trigger compaction
-space:insert({3, 3})
----
-- [3, 3]
-...
-box.snapshot()
----
-- ok
-...
-- wait for compaction
while vyinfo().run_count >= 2 do fiber.sleep(0.1) end
---
@@ -110,8 +97,9 @@ test_run:cmd("setopt delimiter ';'")
---
- true
...
-function dump()
- for i = 1, 10 do
+function dump(big)
+ local step = big and 1 or 5
+ for i = 1, 20, step do
s:replace{i, digest.urandom(1000)}
end
box.snapshot()
@@ -135,7 +123,10 @@ test_run:cmd("setopt delimiter ''");
---
- true
...
-dump()
+-- The first run should be big enough to prevent major compaction
+-- from kicking in on the next dump, because run_count_per_level
+-- is ignored on the last level (see gh-3657).
+dump(true)
---
...
dump()
diff --git a/test/vinyl/compact.test.lua b/test/vinyl/compact.test.lua
index b9a1c109..6c1e5731 100644
--- a/test/vinyl/compact.test.lua
+++ b/test/vinyl/compact.test.lua
@@ -26,10 +26,6 @@ vyinfo().run_count == 1
space:replace({2,2})
space:upsert({2},{{'=',4,5}}) -- bad upsert
box.snapshot() -- create the second run
-vyinfo().run_count == 2
--- create a few more runs to trigger compaction
-space:insert({3, 3})
-box.snapshot()
-- wait for compaction
while vyinfo().run_count >= 2 do fiber.sleep(0.1) end
@@ -47,8 +43,9 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
_ = s:create_index('pk', {run_count_per_level = 100, page_size = 128, range_size = 1024})
test_run:cmd("setopt delimiter ';'")
-function dump()
- for i = 1, 10 do
+function dump(big)
+ local step = big and 1 or 5
+ for i = 1, 20, step do
s:replace{i, digest.urandom(1000)}
end
box.snapshot()
@@ -66,7 +63,11 @@ function compact()
end;
test_run:cmd("setopt delimiter ''");
-dump()
+-- The first run should be big enough to prevent major compaction
+-- from kicking in on the next dump, because run_count_per_level
+-- is ignored on the last level (see gh-3657).
+dump(true)
+
dump()
dump()
info() -- 1 range, 3 runs
diff --git a/test/vinyl/deferred_delete.result b/test/vinyl/deferred_delete.result
index 4a68c6c2..9a823d59 100644
--- a/test/vinyl/deferred_delete.result
+++ b/test/vinyl/deferred_delete.result
@@ -662,6 +662,9 @@ test_run:cmd("switch test")
fiber = require('fiber')
---
...
+digest = require('digest')
+---
+...
s = box.schema.space.create('test', {engine = 'vinyl'})
---
...
@@ -671,6 +674,18 @@ pk = s:create_index('pk', {run_count_per_level = 10})
sk = s:create_index('sk', {run_count_per_level = 10, parts = {2, 'unsigned', 3, 'string'}, unique = false})
---
...
+-- Write a run big enough to prevent major compaction from kicking in
+-- (run_count_per_level is ignored on the last level - see gh-3657).
+dummy_rows = 100
+---
+...
+for i = 1, dummy_rows do s:replace{i + 1000, i + 1000, digest.urandom(100)} end
+---
+...
+box.snapshot()
+---
+- ok
+...
pad = string.rep('x', 10 * 1024)
---
...
@@ -691,7 +706,7 @@ box.snapshot()
---
- ok
...
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs
---
- 240
...
@@ -711,7 +726,7 @@ sk:stat().disk.dump.count -- 1
---
- 1
...
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
---
- 360
...
@@ -725,10 +740,13 @@ pk = s.index.pk
sk = s.index.sk
---
...
+dummy_rows = 100
+---
+...
-- Should be 360, the same amount of statements as before restart.
-- If we applied all deferred DELETEs, including the dumped ones,
-- then there would be more.
-sk:stat().rows
+sk:stat().rows - dummy_rows
---
- 360
...
diff --git a/test/vinyl/deferred_delete.test.lua b/test/vinyl/deferred_delete.test.lua
index e9e0e293..8d2cd0c1 100644
--- a/test/vinyl/deferred_delete.test.lua
+++ b/test/vinyl/deferred_delete.test.lua
@@ -248,11 +248,18 @@ test_run:cmd("start server test with args='1048576'")
test_run:cmd("switch test")
fiber = require('fiber')
+digest = require('digest')
s = box.schema.space.create('test', {engine = 'vinyl'})
pk = s:create_index('pk', {run_count_per_level = 10})
sk = s:create_index('sk', {run_count_per_level = 10, parts = {2, 'unsigned', 3, 'string'}, unique = false})
+-- Write a run big enough to prevent major compaction from kicking in
+-- (run_count_per_level is ignored on the last level - see gh-3657).
+dummy_rows = 100
+for i = 1, dummy_rows do s:replace{i + 1000, i + 1000, digest.urandom(100)} end
+box.snapshot()
+
pad = string.rep('x', 10 * 1024)
for i = 1, 120 do s:replace{i, i, pad} end
box.snapshot()
@@ -261,7 +268,7 @@ pad = string.rep('y', 10 * 1024)
for i = 1, 120 do s:replace{i, i, pad} end
box.snapshot()
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs
box.stat.reset()
@@ -273,17 +280,18 @@ while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
sk:stat().disk.dump.count -- 1
-sk:stat().rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
+sk:stat().rows - dummy_rows -- 120 old REPLACEs + 120 new REPLACEs + 120 deferred DELETEs
test_run:cmd("restart server test with args='1048576'")
s = box.space.test
pk = s.index.pk
sk = s.index.sk
+dummy_rows = 100
-- Should be 360, the same amount of statements as before restart.
-- If we applied all deferred DELETEs, including the dumped ones,
-- then there would be more.
-sk:stat().rows
+sk:stat().rows - dummy_rows
s:drop()
diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result
index 63d14c61..53345949 100644
--- a/test/vinyl/errinj.result
+++ b/test/vinyl/errinj.result
@@ -1867,6 +1867,10 @@ s:replace{1, 10}
---
- [1, 10]
...
+s:replace{10, 100} -- to prevent last-level compaction (gh-3657)
+---
+- [10, 100]
+...
box.snapshot()
---
- ok
@@ -2137,9 +2141,6 @@ function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
---
...
-dump()
----
-...
i:stat().disk.compact.queue -- none
---
- bytes_compressed: <bytes_compressed>
@@ -2158,6 +2159,9 @@ errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
dump()
---
...
+dump()
+---
+...
i:stat().disk.compact.queue -- 30 statements
---
- bytes_compressed: <bytes_compressed>
diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua
index b6f697d3..c9293a38 100644
--- a/test/vinyl/errinj.test.lua
+++ b/test/vinyl/errinj.test.lua
@@ -748,6 +748,7 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
_ = s:create_index('pk', {run_count_per_level = 10})
_ = s:create_index('sk', {unique = false, parts = {2, 'unsigned'}})
s:replace{1, 10}
+s:replace{10, 100} -- to prevent last-level compaction (gh-3657)
box.snapshot()
s:replace{1, 20}
box.snapshot()
@@ -860,11 +861,11 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
i = s:create_index('pk', {run_count_per_level = 2})
function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
-dump()
i:stat().disk.compact.queue -- none
i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
dump()
+dump()
i:stat().disk.compact.queue -- 30 statements
i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
dump()
diff --git a/test/vinyl/gc.result b/test/vinyl/gc.result
index 3931219b..098c17c2 100644
--- a/test/vinyl/gc.result
+++ b/test/vinyl/gc.result
@@ -168,11 +168,14 @@ function count_runs() return #fio.glob(fio.pathjoin(box.cfg.vinyl_dir, s.id, s.i
_ = s:replace{1}
---
...
+_ = s:replace{2} -- to prevent last-level compaction (gh-3657)
+---
+...
box.snapshot()
---
- ok
...
-_ = s:replace{2}
+_ = s:replace{3}
---
...
box.snapshot()
@@ -186,12 +189,11 @@ count_runs() -- 2
for i = 1, 20 do s:replace{i, string.rep('x', 100 * 1024)} end
---
...
-while s.index.pk:stat().disk.compact.count < 1 do fiber.sleep(0.001) end
+s.index.pk:compact()
---
...
-s.index.pk:stat().disk.compact.count -- 1
+while s.index.pk:stat().run_count > 1 do fiber.sleep(0.01) end
---
-- 1
...
count_runs() -- 3 (compacted runs created after checkpoint are deleted)
---
diff --git a/test/vinyl/gc.test.lua b/test/vinyl/gc.test.lua
index 164fac12..b5e42d6b 100644
--- a/test/vinyl/gc.test.lua
+++ b/test/vinyl/gc.test.lua
@@ -84,15 +84,16 @@ _ = s:create_index('pk', {run_count_per_level = 3})
function count_runs() return #fio.glob(fio.pathjoin(box.cfg.vinyl_dir, s.id, s.index.pk.id, '*.run')) end
_ = s:replace{1}
+_ = s:replace{2} -- to prevent last-level compaction (gh-3657)
box.snapshot()
-_ = s:replace{2}
+_ = s:replace{3}
box.snapshot()
count_runs() -- 2
for i = 1, 20 do s:replace{i, string.rep('x', 100 * 1024)} end
-while s.index.pk:stat().disk.compact.count < 1 do fiber.sleep(0.001) end
-s.index.pk:stat().disk.compact.count -- 1
+s.index.pk:compact()
+while s.index.pk:stat().run_count > 1 do fiber.sleep(0.01) end
count_runs() -- 3 (compacted runs created after checkpoint are deleted)
diff --git a/test/vinyl/info.result b/test/vinyl/info.result
index a47c9a1b..5a2935df 100644
--- a/test/vinyl/info.result
+++ b/test/vinyl/info.result
@@ -1298,7 +1298,7 @@ box.snapshot()
i1:compact()
---
...
-wait(function() return i1:stat() end, st1, 'disk.compact.count', 1)
+while i1:stat().run_count > 1 do fiber.sleep(0.01) end
---
...
box.snapshot()
@@ -1308,7 +1308,7 @@ box.snapshot()
i2:compact()
---
...
-wait(function() return i2:stat() end, st2, 'disk.compact.count', 1)
+while i2:stat().run_count > 1 do fiber.sleep(0.01) end
---
...
gst = gstat()
diff --git a/test/vinyl/info.test.lua b/test/vinyl/info.test.lua
index e5794a23..4e1a5207 100644
--- a/test/vinyl/info.test.lua
+++ b/test/vinyl/info.test.lua
@@ -383,10 +383,10 @@ i2:bsize() == st2.memory.index_size + st2.disk.index_size + st2.disk.bloom_size
-- Then dump them and compact the secondary index.
box.snapshot()
i1:compact()
-wait(function() return i1:stat() end, st1, 'disk.compact.count', 1)
+while i1:stat().run_count > 1 do fiber.sleep(0.01) end
box.snapshot()
i2:compact()
-wait(function() return i2:stat() end, st2, 'disk.compact.count', 1)
+while i2:stat().run_count > 1 do fiber.sleep(0.01) end
gst = gstat()
st1 = i1:stat()
st2 = i2:stat()
diff --git a/test/vinyl/layout.result b/test/vinyl/layout.result
index ee14cd51..1fd035c9 100644
--- a/test/vinyl/layout.result
+++ b/test/vinyl/layout.result
@@ -38,6 +38,8 @@ box.snapshot()
space.index.sk:alter{parts = {{2, 'unsigned', is_nullable = true}}}
---
...
+-- Note, the first run is bigger than the second one to prevent
+-- last-level compaction (gh-3657).
space:replace{'ЭЭЭ', box.NULL}
---
- ['ЭЭЭ', null]
@@ -50,6 +52,10 @@ space:replace{'ёёё', box.NULL}
---
- ['ёёё', null]
...
+space:replace{'ЭЮЯ', 666}
+---
+- ['ЭЮЯ', 666]
+...
box.snapshot()
---
- ok
@@ -126,16 +132,16 @@ test_run:cmd("push filter 'offset: .*' to 'offset: <offset>'")
...
result
---
-- - - 00000000000000000010.vylog
+- - - 00000000000000000011.vylog
- - HEADER:
type: INSERT
BODY:
tuple: [0, {6: 512, 7: [{'field': 0, 'collation': 1, 'type': 'string'}],
- 9: 10, 12: 3, 13: 7}]
+ 9: 11, 12: 3, 13: 7}]
- HEADER:
type: INSERT
BODY:
- tuple: [5, {2: 8, 9: 10}]
+ tuple: [5, {2: 8, 9: 11}]
- HEADER:
type: INSERT
BODY:
@@ -156,11 +162,11 @@ result
type: INSERT
BODY:
tuple: [0, {0: 2, 5: 1, 6: 512, 7: [{'field': 1, 'is_nullable': true, 'type': 'unsigned'}],
- 9: 10, 12: 4, 13: 7}]
+ 9: 11, 12: 4, 13: 7}]
- HEADER:
type: INSERT
BODY:
- tuple: [5, {0: 2, 2: 6, 9: 10}]
+ tuple: [5, {0: 2, 2: 6, 9: 11}]
- HEADER:
type: INSERT
BODY:
@@ -200,7 +206,7 @@ result
timestamp: <timestamp>
type: INSERT
BODY:
- tuple: [5, {0: 2, 2: 10, 9: 13}]
+ tuple: [5, {0: 2, 2: 10, 9: 14}]
- HEADER:
timestamp: <timestamp>
type: INSERT
@@ -210,7 +216,7 @@ result
timestamp: <timestamp>
type: INSERT
BODY:
- tuple: [10, {0: 2, 9: 13}]
+ tuple: [10, {0: 2, 9: 14}]
- HEADER:
timestamp: <timestamp>
type: INSERT
@@ -220,7 +226,7 @@ result
timestamp: <timestamp>
type: INSERT
BODY:
- tuple: [5, {2: 12, 9: 13}]
+ tuple: [5, {2: 12, 9: 14}]
- HEADER:
timestamp: <timestamp>
type: INSERT
@@ -230,25 +236,25 @@ result
timestamp: <timestamp>
type: INSERT
BODY:
- tuple: [10, {9: 13}]
+ tuple: [10, {9: 14}]
- - 00000000000000000008.index
- - HEADER:
type: RUNINFO
BODY:
min_lsn: 8
- max_key: ['ЭЭЭ']
+ max_key: ['ЭЮЯ']
page_count: 1
bloom_filter: <bloom_filter>
- max_lsn: 10
+ max_lsn: 11
min_key: ['ёёё']
- HEADER:
type: PAGEINFO
BODY:
row_index_offset: <offset>
offset: <offset>
- size: 86
- unpacked_size: 67
- row_count: 3
+ size: 108
+ unpacked_size: 89
+ row_count: 4
min_key: ['ёёё']
- - 00000000000000000008.run
- - HEADER:
@@ -267,18 +273,23 @@ result
BODY:
tuple: ['ЭЭЭ', null]
- HEADER:
+ lsn: 11
+ type: REPLACE
+ BODY:
+ tuple: ['ЭЮЯ', 666]
+ - HEADER:
type: ROWINDEX
BODY:
- row_index: "\0\0\0\0\0\0\0\x10\0\0\0 "
+ row_index: "\0\0\0\0\0\0\0\x10\0\0\0 \0\0\00"
- - 00000000000000000012.index
- - HEADER:
type: RUNINFO
BODY:
- min_lsn: 11
+ min_lsn: 12
max_key: ['ЮЮЮ']
page_count: 1
bloom_filter: <bloom_filter>
- max_lsn: 13
+ max_lsn: 14
min_key: ['ёёё']
- HEADER:
type: PAGEINFO
@@ -291,19 +302,19 @@ result
min_key: ['ёёё']
- - 00000000000000000012.run
- - HEADER:
- lsn: 11
+ lsn: 12
type: REPLACE
BODY:
tuple: ['ёёё', 123]
tuple_meta: {1: 1}
- HEADER:
- lsn: 13
+ lsn: 14
type: REPLACE
BODY:
tuple: ['ююю', 789]
tuple_meta: {1: 1}
- HEADER:
- lsn: 12
+ lsn: 13
type: REPLACE
BODY:
tuple: ['ЮЮЮ', 456]
@@ -317,19 +328,19 @@ result
type: RUNINFO
BODY:
min_lsn: 8
- max_key: [null, 'ЭЭЭ']
+ max_key: [666, 'ЭЮЯ']
page_count: 1
bloom_filter: <bloom_filter>
- max_lsn: 10
+ max_lsn: 11
min_key: [null, 'ёёё']
- HEADER:
type: PAGEINFO
BODY:
row_index_offset: <offset>
offset: <offset>
- size: 86
- unpacked_size: 67
- row_count: 3
+ size: 108
+ unpacked_size: 89
+ row_count: 4
min_key: [null, 'ёёё']
- - 00000000000000000006.run
- - HEADER:
@@ -348,18 +359,23 @@ result
BODY:
tuple: [null, 'ЭЭЭ']
- HEADER:
+ lsn: 11
+ type: REPLACE
+ BODY:
+ tuple: [666, 'ЭЮЯ']
+ - HEADER:
type: ROWINDEX
BODY:
- row_index: "\0\0\0\0\0\0\0\x10\0\0\0 "
+ row_index: "\0\0\0\0\0\0\0\x10\0\0\0 \0\0\00"
- - 00000000000000000010.index
- - HEADER:
type: RUNINFO
BODY:
- min_lsn: 11
+ min_lsn: 12
max_key: [789, 'ююю']
page_count: 1
bloom_filter: <bloom_filter>
- max_lsn: 13
+ max_lsn: 14
min_key: [123, 'ёёё']
- HEADER:
type: PAGEINFO
@@ -372,17 +388,17 @@ result
min_key: [123, 'ёёё']
- - 00000000000000000010.run
- - HEADER:
- lsn: 11
+ lsn: 12
type: REPLACE
BODY:
tuple: [123, 'ёёё']
- HEADER:
- lsn: 12
+ lsn: 13
type: REPLACE
BODY:
tuple: [456, 'ЮЮЮ']
- HEADER:
- lsn: 13
+ lsn: 14
type: REPLACE
BODY:
tuple: [789, 'ююю']
diff --git a/test/vinyl/layout.test.lua b/test/vinyl/layout.test.lua
index 27bf5d40..60f22c76 100644
--- a/test/vinyl/layout.test.lua
+++ b/test/vinyl/layout.test.lua
@@ -17,9 +17,12 @@ box.snapshot()
space.index.sk:alter{parts = {{2, 'unsigned', is_nullable = true}}}
+-- Note, the first run is bigger than the second one to prevent
+-- last-level compaction (gh-3657).
space:replace{'ЭЭЭ', box.NULL}
space:replace{'эээ', box.NULL}
space:replace{'ёёё', box.NULL}
+space:replace{'ЭЮЯ', 666}
box.snapshot()
space:replace{'ёёё', 123}
diff --git a/test/vinyl/update_optimize.result b/test/vinyl/update_optimize.result
index 40f34151..bf9dff7e 100644
--- a/test/vinyl/update_optimize.result
+++ b/test/vinyl/update_optimize.result
@@ -728,6 +728,10 @@ s:insert{1, 10}
---
- [1, 10]
...
+s:insert{10, 100} -- to prevent last-level compaction (gh-3657)
+---
+- [10, 100]
+...
box.snapshot()
---
- ok
@@ -743,9 +747,11 @@ box.snapshot()
---
- ok
...
-s.index.sk:stat().rows -- INSERT in the first run + DELETE the second run
+-- Should be 3: INSERT{10, 1} and INSERT{100, 10} in the first run
+-- plus DELETE{10, 1} in the second run.
+s.index.sk:stat().rows
---
-- 2
+- 3
...
s:insert{1, 20}
---
@@ -754,6 +760,7 @@ s:insert{1, 20}
s.index.sk:select()
---
- - [1, 20]
+ - [10, 100]
...
s:drop()
---
diff --git a/test/vinyl/update_optimize.test.lua b/test/vinyl/update_optimize.test.lua
index a214dcae..67efba31 100644
--- a/test/vinyl/update_optimize.test.lua
+++ b/test/vinyl/update_optimize.test.lua
@@ -244,13 +244,16 @@ _ = s:create_index('pk')
_ = s:create_index('sk', {parts = {2, 'unsigned'}, run_count_per_level = 10})
s:insert{1, 10}
+s:insert{10, 100} -- to prevent last-level compaction (gh-3657)
box.snapshot()
s:update(1, {{'=', 2, 10}})
s:delete(1)
box.snapshot()
-s.index.sk:stat().rows -- INSERT in the first run + DELETE the second run
+-- Should be 3: INSERT{10, 1} and INSERT{100, 10} in the first run
+-- plus DELETE{10, 1} in the second run.
+s.index.sk:stat().rows
s:insert{1, 20}
s.index.sk:select()
diff --git a/test/vinyl/upsert.result b/test/vinyl/upsert.result
index e6c451d2..6c6d03d6 100644
--- a/test/vinyl/upsert.result
+++ b/test/vinyl/upsert.result
@@ -676,6 +676,10 @@ s = box.schema.space.create('test', {engine = 'vinyl'})
i = s:create_index('test', { run_count_per_level = 20 })
---
...
+-- Write a big run to prevent last-level compaction (gh-3657).
+for i = 101, 110 do s:replace{i, require('digest').urandom(50)} end
+---
+...
s:replace({1, 1})
---
- [1, 1]
@@ -690,9 +694,9 @@ s:upsert({1, 1}, {{'+', 1, 1}}) -- ignored due to primary key changed
s:upsert({1, 1}, {{'+', 2, 1}}) -- applied to the previous statement
---
...
-s:select()
+s:get(1)
---
-- - [1, 2]
+- [1, 2]
...
--
-- gh-2520 use cache as a hint when applying upserts.
diff --git a/test/vinyl/upsert.test.lua b/test/vinyl/upsert.test.lua
index cb7cc81a..ef83e0f6 100644
--- a/test/vinyl/upsert.test.lua
+++ b/test/vinyl/upsert.test.lua
@@ -274,11 +274,14 @@ space:drop()
s = box.schema.space.create('test', {engine = 'vinyl'})
i = s:create_index('test', { run_count_per_level = 20 })
+-- Write a big run to prevent last-level compaction (gh-3657).
+for i = 101, 110 do s:replace{i, require('digest').urandom(50)} end
+
s:replace({1, 1})
box.snapshot()
s:upsert({1, 1}, {{'+', 1, 1}}) -- ignored due to primary key changed
s:upsert({1, 1}, {{'+', 2, 1}}) -- applied to the previous statement
-s:select()
+s:get(1)
--
-- gh-2520 use cache as a hint when applying upserts.
--
2.11.0
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] vinyl: do not apply run_count_per_level to the last level
2018-11-16 17:00 [PATCH] vinyl: do not apply run_count_per_level to the last level Vladimir Davydov
@ 2018-11-18 16:37 ` Konstantin Osipov
2018-11-19 8:20 ` Vladimir Davydov
0 siblings, 1 reply; 3+ messages in thread
From: Konstantin Osipov @ 2018-11-18 16:37 UTC (permalink / raw)
To: Vladimir Davydov; +Cc: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [18/11/17 07:05]:
> Currently, run_count_per_level index option is applied to each LSM tree
> level. As a result, we may end up storing each key run_count_per_level
> times at the last level alone, which would result in prohibitive space
> amplification. To avoid that, let's ignore run_count_per_level for the
> last level.
>
> Note, we have to tweak quite a few vinyl tests, because they implicitly
> relied on the fact that producing run_count_per_level dumps would never
> trigger compaction.
OK to push.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH] vinyl: do not apply run_count_per_level to the last level
2018-11-18 16:37 ` Konstantin Osipov
@ 2018-11-19 8:20 ` Vladimir Davydov
0 siblings, 0 replies; 3+ messages in thread
From: Vladimir Davydov @ 2018-11-19 8:20 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: tarantool-patches
On Sun, Nov 18, 2018 at 07:37:52PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev@gmail.com> [18/11/17 07:05]:
> > Currently, run_count_per_level index option is applied to each LSM tree
> > level. As a result, we may end up storing each key run_count_per_level
> > times at the last level alone, which would result in prohibitive space
> > amplification. To avoid that, let's ignore run_count_per_level for the
> > last level.
> >
> > Note, we have to tweak quite a few vinyl tests, because they implicitly
> > relied on the fact that producing run_count_per_level dumps would never
> > trigger compaction.
>
> OK to push.
Pushed to 1.10.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2018-11-19 8:20 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-16 17:00 [PATCH] vinyl: do not apply run_count_per_level to the last level Vladimir Davydov
2018-11-18 16:37 ` Konstantin Osipov
2018-11-19 8:20 ` Vladimir Davydov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox