* [PATCH 00/12] vinyl: statistics improvements
@ 2019-01-15 14:17 Vladimir Davydov
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
` (13 more replies)
0 siblings, 14 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
This patch set adds a few metrics necessary for implementing compaction
randomization and transaction throttling, but it's useful on its own,
because it makes box.stat.vinyl() a little bit more useful when it comes
to performance analysis. Here's an example of box.stat.vinyl() output
with this patch set applied:
---
- tx:
conflict: 0
commit: 1979052
rollback: 0
statements: 2
transactions: 1
gap_locks: 0
read_views: 0
regulator:
dump_bandwidth: 10485760
dump_watermark: 20023725
write_rate: 7085581
memory:
tuple_cache: 0
tx: 2388
level0: 19394239
page_index: 4422529
bloom_filter: 1517177
disk:
data_compacted: 500330587
data: 762493299
index: 41814873
scheduler:
dump_time: 186.61679973663
tasks_inprogress: 3
dump_output: 2115930554
compaction_queue: 213022513
compaction_output: 4130054964
compaction_time: 737.99443827965
dump_count: 136
tasks_failed: 0
tasks_completed: 1839
dump_input: 2061676471
compaction_input: 5646476938
...
This patch set adds the 'scheduler' section, which exposes dump and
compaction related statistics, and 'disk.data_compacted', which is
needed for evaluating space amplification. See comments to individual
patches for more details.
Patches are mostly trivial - they primarily move pieces of code around
and expose internal counters.
What's still missing:
- disk compression ratio
- iterator statistics (in particular, read amplification)
However, they aren't required for compaction randomization or throttling
and so I leave them for later.
https://github.com/tarantool/tarantool/commits/dv/vy-stat-improvements
Vladimir Davydov (12):
test: rename vinyl/info to vinyl/stat
test: split vinyl/errinj
vinyl: rename dump/compact in/out to input/output
vinyl: rename compact to compaction
vinyl: bump range version in vy_range.c
vinyl: don't add dropped LSM trees to the scheduler during recovery
vinyl: move global dump/compaction statistics to scheduler
vinyl: add dump count to global scheduler statistics
vinyl: don't account secondary indexes to scheduler.dump_input
vinyl: add task accounting to global scheduler statistics
vinyl: add dump/compaction time to statistics
vinyl: add last level size to statistics
src/box/vinyl.c | 121 +--
src/box/vy_lsm.c | 88 +-
src/box/vy_lsm.h | 55 +-
src/box/vy_range.c | 27 +-
src/box/vy_range.h | 16 +-
src/box/vy_scheduler.c | 207 +++--
src/box/vy_scheduler.h | 15 +-
src/box/vy_stat.h | 57 +-
test/vinyl/ddl.result | 2 +-
test/vinyl/ddl.test.lua | 2 +-
test/vinyl/deferred_delete.result | 16 +-
test/vinyl/deferred_delete.test.lua | 16 +-
test/vinyl/errinj.result | 1168 +--------------------------
test/vinyl/errinj.test.lua | 504 +-----------
test/vinyl/errinj_ddl.result | 595 ++++++++++++++
test/vinyl/errinj_ddl.test.lua | 260 ++++++
test/vinyl/errinj_stat.result | 407 ++++++++++
test/vinyl/errinj_stat.test.lua | 129 +++
test/vinyl/errinj_tx.result | 435 ++++++++++
test/vinyl/errinj_tx.test.lua | 194 +++++
test/vinyl/errinj_vylog.result | 35 +
test/vinyl/errinj_vylog.test.lua | 18 +
test/vinyl/recovery_quota.result | 4 +-
test/vinyl/recovery_quota.test.lua | 4 +-
test/vinyl/snap_io_rate.result | 2 +-
test/vinyl/snap_io_rate.test.lua | 2 +-
test/vinyl/{info.lua => stat.lua} | 0
test/vinyl/{info.result => stat.result} | 493 ++++++++---
test/vinyl/{info.test.lua => stat.test.lua} | 93 ++-
test/vinyl/suite.ini | 2 +-
test/vinyl/update_optimize.result | 4 +-
test/vinyl/update_optimize.test.lua | 4 +-
test/vinyl/write_iterator.result | 20 +-
test/vinyl/write_iterator.test.lua | 20 +-
34 files changed, 2962 insertions(+), 2053 deletions(-)
create mode 100644 test/vinyl/errinj_ddl.result
create mode 100644 test/vinyl/errinj_ddl.test.lua
create mode 100644 test/vinyl/errinj_stat.result
create mode 100644 test/vinyl/errinj_stat.test.lua
create mode 100644 test/vinyl/errinj_tx.result
create mode 100644 test/vinyl/errinj_tx.test.lua
rename test/vinyl/{info.lua => stat.lua} (100%)
rename test/vinyl/{info.result => stat.result} (80%)
rename test/vinyl/{info.test.lua => stat.test.lua} (80%)
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 01/12] test: rename vinyl/info to vinyl/stat
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-17 11:32 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 02/12] test: split vinyl/errinj Vladimir Davydov
` (12 subsequent siblings)
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
The test was called 'info' in the first place, because back when it was
introduced vinyl statistics were reported by 'info' method. Today, stats
are reported by 'stat' so let's rename the test as well to conform.
---
test/vinyl/{info.lua => stat.lua} | 0
test/vinyl/{info.result => stat.result} | 2 +-
test/vinyl/{info.test.lua => stat.test.lua} | 2 +-
3 files changed, 2 insertions(+), 2 deletions(-)
rename test/vinyl/{info.lua => stat.lua} (100%)
rename test/vinyl/{info.result => stat.result} (99%)
rename test/vinyl/{info.test.lua => stat.test.lua} (99%)
diff --git a/test/vinyl/info.lua b/test/vinyl/stat.lua
similarity index 100%
rename from test/vinyl/info.lua
rename to test/vinyl/stat.lua
diff --git a/test/vinyl/info.result b/test/vinyl/stat.result
similarity index 99%
rename from test/vinyl/info.result
rename to test/vinyl/stat.result
index 922728ab..c4a0e632 100644
--- a/test/vinyl/info.result
+++ b/test/vinyl/stat.result
@@ -4,7 +4,7 @@ test_run = require('test_run').new()
-- Since we store LSNs in data files, the data size may differ
-- from run to run. Deploy a new server to make sure it will be
-- the same so that we can check it.
-test_run:cmd('create server test with script = "vinyl/info.lua"')
+test_run:cmd('create server test with script = "vinyl/stat.lua"')
---
- true
...
diff --git a/test/vinyl/info.test.lua b/test/vinyl/stat.test.lua
similarity index 99%
rename from test/vinyl/info.test.lua
rename to test/vinyl/stat.test.lua
index ef2fd79e..8b296b9e 100644
--- a/test/vinyl/info.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -3,7 +3,7 @@ test_run = require('test_run').new()
-- Since we store LSNs in data files, the data size may differ
-- from run to run. Deploy a new server to make sure it will be
-- the same so that we can check it.
-test_run:cmd('create server test with script = "vinyl/info.lua"')
+test_run:cmd('create server test with script = "vinyl/stat.lua"')
test_run:cmd('start server test')
test_run:cmd('switch test')
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 02/12] test: split vinyl/errinj
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 03/12] vinyl: rename dump/compact in/out to input/output Vladimir Davydov
` (11 subsequent siblings)
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
This test is huge and takes long to complete. Let's move ddl, tx, and
stat related stuff to separate files.
---
test/vinyl/errinj.result | 1158 --------------------------------------
test/vinyl/errinj.test.lua | 494 ----------------
test/vinyl/errinj_ddl.result | 595 ++++++++++++++++++++
test/vinyl/errinj_ddl.test.lua | 260 +++++++++
test/vinyl/errinj_stat.result | 152 +++++
test/vinyl/errinj_stat.test.lua | 48 ++
test/vinyl/errinj_tx.result | 435 ++++++++++++++
test/vinyl/errinj_tx.test.lua | 194 +++++++
test/vinyl/errinj_vylog.result | 35 ++
test/vinyl/errinj_vylog.test.lua | 18 +
test/vinyl/suite.ini | 2 +-
11 files changed, 1738 insertions(+), 1653 deletions(-)
create mode 100644 test/vinyl/errinj_ddl.result
create mode 100644 test/vinyl/errinj_ddl.test.lua
create mode 100644 test/vinyl/errinj_stat.result
create mode 100644 test/vinyl/errinj_stat.test.lua
create mode 100644 test/vinyl/errinj_tx.result
create mode 100644 test/vinyl/errinj_tx.test.lua
diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result
index 23ab845b..a16475f5 100644
--- a/test/vinyl/errinj.result
+++ b/test/vinyl/errinj.result
@@ -1,6 +1,3 @@
---
--- gh-1681: vinyl: crash in vy_rollback on ER_WAL_WRITE
---
test_run = require('test_run').new()
---
...
@@ -17,42 +14,6 @@ errinj.set("ERRINJ_VY_SCHED_TIMEOUT", 0.040)
---
- ok
...
-s = box.schema.space.create('test', {engine='vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-function f() box.begin() s:insert{1, 'hi'} s:insert{2, 'bye'} box.commit() end
----
-...
-errinj.set("ERRINJ_WAL_WRITE", true)
----
-- ok
-...
-f()
----
-- error: Failed to write to disk
-...
-s:select{}
----
-- []
-...
-errinj.set("ERRINJ_WAL_WRITE", false)
----
-- ok
-...
-f()
----
-...
-s:select{}
----
-- - [1, 'hi']
- - [2, 'bye']
-...
-s:drop()
----
-...
--
-- Lost data in case of dump error
--
@@ -474,399 +435,6 @@ errinj.set("ERRINJ_VY_SQUASH_TIMEOUT", 0)
---
- ok
...
---https://github.com/tarantool/tarantool/issues/1842
---test error injection
-s = box.schema.space.create('test', {engine='vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-s:replace{0, 0}
----
-- [0, 0]
-...
-s:replace{1, 0}
----
-- [1, 0]
-...
-s:replace{2, 0}
----
-- [2, 0]
-...
-errinj.set("ERRINJ_WAL_WRITE", true)
----
-- ok
-...
-s:replace{3, 0}
----
-- error: Failed to write to disk
-...
-s:replace{4, 0}
----
-- error: Failed to write to disk
-...
-s:replace{5, 0}
----
-- error: Failed to write to disk
-...
-s:replace{6, 0}
----
-- error: Failed to write to disk
-...
-errinj.set("ERRINJ_WAL_WRITE", false)
----
-- ok
-...
-s:replace{7, 0}
----
-- [7, 0]
-...
-s:replace{8, 0}
----
-- [8, 0]
-...
-s:select{}
----
-- - [0, 0]
- - [1, 0]
- - [2, 0]
- - [7, 0]
- - [8, 0]
-...
-s:drop()
----
-...
-create_iterator = require('utils').create_iterator
----
-...
---iterator test
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
-fiber_status = 0
-
-function fiber_func()
- box.begin()
- s:replace{5, 5}
- fiber_status = 1
- local res = {pcall(box.commit) }
- fiber_status = 2
- return unpack(res)
-end;
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-s = box.schema.space.create('test', {engine='vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-fiber = require('fiber')
----
-...
-_ = s:replace{0, 0}
----
-...
-_ = s:replace{10, 0}
----
-...
-_ = s:replace{20, 0}
----
-...
-test_run:cmd("setopt delimiter ';'");
----
-- true
-...
-faced_trash = false
-for i = 1,100 do
- errinj.set("ERRINJ_WAL_WRITE", true)
- local f = fiber.create(fiber_func)
- local itr = create_iterator(s, {0}, {iterator='GE'})
- local first = itr.next()
- local second = itr.next()
- if (second[1] ~= 5 and second[1] ~= 10) then faced_trash = true end
- while fiber_status <= 1 do fiber.sleep(0.001) end
- local _,next = pcall(itr.next)
- _,next = pcall(itr.next)
- _,next = pcall(itr.next)
- errinj.set("ERRINJ_WAL_WRITE", false)
- s:delete{5}
-end;
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-faced_trash
----
-- false
-...
-s:drop()
----
-...
--- TX in prepared but not committed state
-s = box.schema.space.create('test', {engine='vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-fiber = require('fiber')
----
-...
-txn_proxy = require('txn_proxy')
----
-...
-s:replace{1, "original"}
----
-- [1, 'original']
-...
-s:replace{2, "original"}
----
-- [2, 'original']
-...
-s:replace{3, "original"}
----
-- [3, 'original']
-...
-c0 = txn_proxy.new()
----
-...
-c0:begin()
----
--
-...
-c1 = txn_proxy.new()
----
-...
-c1:begin()
----
--
-...
-c2 = txn_proxy.new()
----
-...
-c2:begin()
----
--
-...
-c3 = txn_proxy.new()
----
-...
-c3:begin()
----
--
-...
---
--- Prepared transactions
---
--- Pause WAL writer to cause all further calls to box.commit() to move
--- transactions into prepared, but not committed yet state.
-errinj.set("ERRINJ_WAL_DELAY", true)
----
-- ok
-...
-lsn = box.info.lsn
----
-...
-c0('s:replace{1, "c0"}')
----
-- - [1, 'c0']
-...
-c0('s:replace{2, "c0"}')
----
-- - [2, 'c0']
-...
-c0('s:replace{3, "c0"}')
----
-- - [3, 'c0']
-...
-_ = fiber.create(c0.commit, c0)
----
-...
-box.info.lsn == lsn
----
-- true
-...
-c1('s:replace{1, "c1"}')
----
-- - [1, 'c1']
-...
-c1('s:replace{2, "c1"}')
----
-- - [2, 'c1']
-...
-_ = fiber.create(c1.commit, c1)
----
-...
-box.info.lsn == lsn
----
-- true
-...
-c3('s:select{1}') -- c1 is visible
----
-- - [[1, 'c1']]
-...
-c2('s:replace{1, "c2"}')
----
-- - [1, 'c2']
-...
-c2('s:replace{3, "c2"}')
----
-- - [3, 'c2']
-...
-_ = fiber.create(c2.commit, c2)
----
-...
-box.info.lsn == lsn
----
-- true
-...
-c3('s:select{1}') -- c1 is visible, c2 is not
----
-- - [[1, 'c1']]
-...
-c3('s:select{2}') -- c1 is visible
----
-- - [[2, 'c1']]
-...
-c3('s:select{3}') -- c2 is not visible
----
-- - [[3, 'c0']]
-...
--- Resume WAL writer and wait until all transactions will been committed
-errinj.set("ERRINJ_WAL_DELAY", false)
----
-- ok
-...
-REQ_COUNT = 7
----
-...
-while box.info.lsn - lsn < REQ_COUNT do fiber.sleep(0.01) end
----
-...
-box.info.lsn == lsn + REQ_COUNT
----
-- true
-...
-c3('s:select{1}') -- c1 is visible, c2 is not
----
-- - [[1, 'c1']]
-...
-c3('s:select{2}') -- c1 is visible
----
-- - [[2, 'c1']]
-...
-c3('s:select{3}') -- c2 is not visible
----
-- - [[3, 'c0']]
-...
-c3:commit()
----
--
-...
-s:drop()
----
-...
---
--- Test mem restoration on a prepared and not commited statement
--- after moving iterator into read view.
---
-space = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-pk = space:create_index('pk')
----
-...
-space:replace{1}
----
-- [1]
-...
-space:replace{2}
----
-- [2]
-...
-space:replace{3}
----
-- [3]
-...
-last_read = nil
----
-...
-errinj.set("ERRINJ_WAL_DELAY", true)
----
-- ok
-...
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
--- block until wal_delay = false
--- send iterator to read view
--- flush mem and update index version to trigger iterator restore
-function fill_space()
- box.begin()
- space:replace{1}
- space:replace{2}
- space:replace{3}
- box.commit()
- space:replace{1, 1}
- box.snapshot()
-end;
----
-...
-function iterate_in_read_view()
- local i = create_iterator(space)
- last_read = i.next()
- fiber.sleep(100000)
- last_read = i.next()
-end;
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-f1 = fiber.create(fill_space)
----
-...
--- Prepared transaction is blocked due to wal_delay.
--- Start iterator with vlsn = INT64_MAX
-f2 = fiber.create(iterate_in_read_view)
----
-...
-last_read
----
-- [1]
-...
--- Finish prepared transaction and send to read view the iterator.
-errinj.set("ERRINJ_WAL_DELAY", false)
----
-- ok
-...
-while f1:status() ~= 'dead' do fiber.sleep(0.01) end
----
-...
-f2:wakeup()
----
-...
-while f2:status() ~= 'dead' do fiber.sleep(0.01) end
----
-...
-last_read
----
-- [2]
-...
-space:drop()
----
-...
--
-- Space drop in the middle of dump.
--
@@ -976,38 +544,6 @@ test_run:cmd("cleanup server test")
---
- true
...
---
--- If we logged an index creation in the metadata log before WAL write,
--- WAL failure would result in leaving the index record in vylog forever.
--- Since we use LSN to identify indexes in vylog, retrying index creation
--- would then lead to a duplicate index id in vylog and hence inability
--- to make a snapshot or recover.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-errinj.set('ERRINJ_WAL_IO', true)
----
-- ok
-...
-_ = s:create_index('pk')
----
-- error: Failed to write to disk
-...
-errinj.set('ERRINJ_WAL_IO', false)
----
-- ok
-...
-_ = s:create_index('pk')
----
-...
-box.snapshot()
----
-- ok
-...
-s:drop()
----
-...
s = box.schema.space.create('test', {engine = 'vinyl'})
---
...
@@ -1392,81 +928,6 @@ test_run:cmd("cleanup server low_quota")
- true
...
--
--- Check that ALTER is abroted if a tuple inserted during space
--- format change does not conform to the new format.
---
-format = {}
----
-...
-format[1] = {name = 'field1', type = 'unsigned'}
----
-...
-format[2] = {name = 'field2', type = 'string', is_nullable = true}
----
-...
-s = box.schema.space.create('test', {engine = 'vinyl', format = format})
----
-...
-_ = s:create_index('pk', {page_size = 16})
----
-...
-pad = string.rep('x', 16)
----
-...
-for i = 101, 200 do s:replace{i, pad} end
----
-...
-box.snapshot()
----
-- ok
-...
-ch = fiber.channel(1)
----
-...
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i, box.NULL}
- end
- ch:put(true)
-end);
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
----
-- ok
-...
-format[2].is_nullable = false
----
-...
-s:format(format) -- must fail
----
-- error: 'Tuple field 2 type does not match one required by operation: expected string'
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
----
-- ok
-...
-ch:get()
----
-- true
-...
-s:count() -- 200
----
-- 200
-...
-s:drop()
----
-...
---
-- gh-3437: if compaction races with checkpointing, it may remove
-- files needed for backup.
--
@@ -1533,317 +994,6 @@ s:drop()
---
...
--
--- gh-2449: change 'unique' index property from true to false
--- is done without index rebuild.
---
-s = box.schema.space.create('test', { engine = 'vinyl' })
----
-...
-_ = s:create_index('primary')
----
-...
-_ = s:create_index('secondary', {unique = true, parts = {2, 'unsigned'}})
----
-...
-s:insert{1, 10}
----
-- [1, 10]
-...
-box.snapshot()
----
-- ok
-...
-errinj.set("ERRINJ_VY_READ_PAGE", true);
----
-- ok
-...
-s.index.secondary:alter{unique = false} -- ok
----
-...
-s.index.secondary.unique
----
-- false
-...
-s.index.secondary:alter{unique = true} -- error
----
-- error: Error injection 'vinyl page read'
-...
-s.index.secondary.unique
----
-- false
-...
-errinj.set("ERRINJ_VY_READ_PAGE", false);
----
-- ok
-...
-s:insert{2, 10}
----
-- [2, 10]
-...
-s.index.secondary:select(10)
----
-- - [1, 10]
- - [2, 10]
-...
-s:drop()
----
-...
---
--- Check that ALTER is aborted if a tuple inserted during index build
--- doesn't conform to the new format.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-_ = s:create_index('pk', {page_size = 16})
----
-...
-pad = string.rep('x', 16)
----
-...
-for i = 101, 200 do s:replace{i, i, pad} end
----
-...
-box.snapshot()
----
-- ok
-...
-ch = fiber.channel(1)
----
-...
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i}
- end
- ch:put(true)
-end);
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
----
-- ok
-...
-s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
----
-- error: Tuple field 2 required by space format is missing
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
----
-- ok
-...
-ch:get()
----
-- true
-...
-s:count() -- 200
----
-- 200
-...
-s:drop()
----
-...
---
--- Check that ALTER is aborted if a tuple inserted during index build
--- violates unique constraint.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-_ = s:create_index('pk', {page_size = 16})
----
-...
-pad = string.rep('x', 16)
----
-...
-for i = 101, 200 do s:replace{i, i, pad} end
----
-...
-box.snapshot()
----
-- ok
-...
-ch = fiber.channel(1)
----
-...
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i, i + 1}
- end
- ch:put(true)
-end);
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
----
-- ok
-...
-s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
----
-- error: Duplicate key exists in unique index 'sk' in space 'test'
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
----
-- ok
-...
-ch:get()
----
-- true
-...
-s:count() -- 200
----
-- 200
-...
-s:drop()
----
-...
---
--- Check that modifications done to the space during the final dump
--- of a newly built index are recovered properly.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-for i = 1, 5 do s:replace{i, i} end
----
-...
-errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", true)
----
-- ok
-...
-ch = fiber.channel(1)
----
-...
-_ = fiber.create(function() s:create_index('sk', {parts = {2, 'integer'}}) ch:put(true) end)
----
-...
-fiber.sleep(0.01)
----
-...
-_ = s:delete{1}
----
-...
-_ = s:replace{2, -2}
----
-...
-_ = s:delete{2}
----
-...
-_ = s:replace{3, -3}
----
-...
-_ = s:replace{3, -2}
----
-...
-_ = s:replace{3, -1}
----
-...
-_ = s:delete{3}
----
-...
-_ = s:upsert({3, 3}, {{'=', 2, 1}})
----
-...
-_ = s:upsert({3, 3}, {{'=', 2, 2}})
----
-...
-_ = s:delete{3}
----
-...
-_ = s:replace{4, -1}
----
-...
-_ = s:replace{4, -2}
----
-...
-_ = s:replace{4, -4}
----
-...
-_ = s:upsert({5, 1}, {{'=', 2, 1}})
----
-...
-_ = s:upsert({5, 2}, {{'=', 2, -5}})
----
-...
-_ = s:replace{6, -6}
----
-...
-_ = s:upsert({7, -7}, {{'=', 2, -7}})
----
-...
-errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", false)
----
-- ok
-...
-ch:get()
----
-- true
-...
-s.index.sk:select()
----
-- - [7, -7]
- - [6, -6]
- - [5, -5]
- - [4, -4]
-...
-s.index.sk:stat().memory.rows
----
-- 27
-...
-test_run:cmd('restart server default')
-s = box.space.test
----
-...
-s.index.sk:select()
----
-- - [7, -7]
- - [6, -6]
- - [5, -5]
- - [4, -4]
-...
-s.index.sk:stat().memory.rows
----
-- 27
-...
-box.snapshot()
----
-- ok
-...
-s.index.sk:select()
----
-- - [7, -7]
- - [6, -6]
- - [5, -5]
- - [4, -4]
-...
-s.index.sk:stat().memory.rows
----
-- 0
-...
-s:drop()
----
-...
---
-- Check that tarantool doesn't hang or crash if error
-- occurs while writing a deferred DELETE to WAL.
--
@@ -1926,314 +1076,6 @@ s:drop()
---
...
--
--- gh-3458: check that rw transactions that started before DDL are
--- aborted.
---
-vinyl_cache = box.cfg.vinyl_cache
----
-...
-box.cfg{vinyl_cache = 0}
----
-...
-s1 = box.schema.space.create('test1', {engine = 'vinyl'})
----
-...
-_ = s1:create_index('pk', {page_size = 16})
----
-...
-s2 = box.schema.space.create('test2', {engine = 'vinyl'})
----
-...
-_ = s2:create_index('pk')
----
-...
-pad = string.rep('x', 16)
----
-...
-for i = 101, 200 do s1:replace{i, i, pad} end
----
-...
-box.snapshot()
----
-- ok
-...
-test_run:cmd("setopt delimiter ';'")
----
-- true
-...
-function async_replace(space, tuple, timeout)
- local c = fiber.channel(1)
- fiber.create(function()
- box.begin()
- space:replace(tuple)
- fiber.sleep(timeout)
- local status = pcall(box.commit)
- c:put(status)
- end)
- return c
-end;
----
-...
-test_run:cmd("setopt delimiter ''");
----
-- true
-...
-c1 = async_replace(s1, {1}, 0.01)
----
-...
-c2 = async_replace(s2, {1}, 0.01)
----
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
----
-- ok
-...
-s1:format{{'key', 'unsigned'}, {'value', 'unsigned'}}
----
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
----
-- ok
-...
-c1:get() -- false (transaction was aborted)
----
-- false
-...
-c2:get() -- true
----
-- true
-...
-s1:get(1) == nil
----
-- true
-...
-s2:get(1) ~= nil
----
-- true
-...
-s1:format()
----
-- [{'name': 'key', 'type': 'unsigned'}, {'name': 'value', 'type': 'unsigned'}]
-...
-s1:format{}
----
-...
-c1 = async_replace(s1, {2}, 0.01)
----
-...
-c2 = async_replace(s2, {2}, 0.01)
----
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
----
-- ok
-...
-_ = s1:create_index('sk', {parts = {2, 'unsigned'}})
----
-...
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
----
-- ok
-...
-c1:get() -- false (transaction was aborted)
----
-- false
-...
-c2:get() -- true
----
-- true
-...
-s1:get(2) == nil
----
-- true
-...
-s2:get(2) ~= nil
----
-- true
-...
-s1.index.pk:count() == s1.index.sk:count()
----
-- true
-...
-s1:drop()
----
-...
-s2:drop()
----
-...
-box.cfg{vinyl_cache = vinyl_cache}
----
-...
--- Transactions that reached WAL must not be aborted.
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-_ = s:create_index('pk')
----
-...
-errinj.set('ERRINJ_WAL_DELAY', true)
----
-- ok
-...
-_ = fiber.create(function() s:replace{1} end)
----
-...
-_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
----
-...
-fiber.sleep(0)
----
-...
-s:format{{'key', 'unsigned'}, {'value', 'unsigned'}} -- must fail
----
-- error: Tuple field 2 required by space format is missing
-...
-s:select()
----
-- - [1]
-...
-s:truncate()
----
-...
-errinj.set('ERRINJ_WAL_DELAY', true)
----
-- ok
-...
-_ = fiber.create(function() s:replace{1} end)
----
-...
-_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
----
-...
-fiber.sleep(0)
----
-...
-s:create_index('sk', {parts = {2, 'unsigned'}})
----
-- error: Tuple field 2 required by space format is missing
-...
-s:select()
----
-- - [1]
-...
-s:drop()
----
-...
---
--- Check disk.compact.queue stat.
---
-test_run:cmd("push filter 'bytes_compressed: .*' to 'bytes_compressed: <bytes_compressed>'")
----
-- true
-...
-s = box.schema.space.create('test', {engine = 'vinyl'})
----
-...
-i = s:create_index('pk', {run_count_per_level = 2})
----
-...
-function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
----
-...
-dump()
----
-...
-i:stat().disk.compact.queue -- none
----
-- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
-...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
----
-- true
-...
-errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
----
-- ok
-...
-dump()
----
-...
-dump()
----
-...
-i:stat().disk.compact.queue -- 30 statements
----
-- bytes_compressed: <bytes_compressed>
- pages: 3
- rows: 30
- bytes: 471
-...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
----
-- true
-...
-dump()
----
-...
-i:stat().disk.compact.queue -- 40 statements
----
-- bytes_compressed: <bytes_compressed>
- pages: 4
- rows: 40
- bytes: 628
-...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
----
-- true
-...
-dump()
----
-...
-i:stat().disk.compact.queue -- 50 statements
----
-- bytes_compressed: <bytes_compressed>
- pages: 5
- rows: 50
- bytes: 785
-...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
----
-- true
-...
-box.stat.reset() -- doesn't affect queue size
----
-...
-i:stat().disk.compact.queue -- 50 statements
----
-- bytes_compressed: <bytes_compressed>
- pages: 5
- rows: 50
- bytes: 785
-...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
----
-- true
-...
-errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
----
-- ok
-...
-while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
----
-...
-i:stat().disk.compact.queue -- none
----
-- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
-...
-s:drop()
----
-...
-test_run:cmd("clear filter")
----
-- true
-...
---
-- Check that an instance doesn't crash if a run file needed for
-- joining a replica is corrupted (see gh-3708).
--
diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua
index 59e9f81c..d4590fb4 100644
--- a/test/vinyl/errinj.test.lua
+++ b/test/vinyl/errinj.test.lua
@@ -1,21 +1,8 @@
---
--- gh-1681: vinyl: crash in vy_rollback on ER_WAL_WRITE
---
test_run = require('test_run').new()
fio = require('fio')
fiber = require('fiber')
errinj = box.error.injection
errinj.set("ERRINJ_VY_SCHED_TIMEOUT", 0.040)
-s = box.schema.space.create('test', {engine='vinyl'})
-_ = s:create_index('pk')
-function f() box.begin() s:insert{1, 'hi'} s:insert{2, 'bye'} box.commit() end
-errinj.set("ERRINJ_WAL_WRITE", true)
-f()
-s:select{}
-errinj.set("ERRINJ_WAL_WRITE", false)
-f()
-s:select{}
-s:drop()
--
-- Lost data in case of dump error
--
@@ -165,186 +152,6 @@ s:drop() -- index is gone
fiber.sleep(0.05)
errinj.set("ERRINJ_VY_SQUASH_TIMEOUT", 0)
---https://github.com/tarantool/tarantool/issues/1842
---test error injection
-s = box.schema.space.create('test', {engine='vinyl'})
-_ = s:create_index('pk')
-s:replace{0, 0}
-
-s:replace{1, 0}
-s:replace{2, 0}
-errinj.set("ERRINJ_WAL_WRITE", true)
-s:replace{3, 0}
-s:replace{4, 0}
-s:replace{5, 0}
-s:replace{6, 0}
-errinj.set("ERRINJ_WAL_WRITE", false)
-s:replace{7, 0}
-s:replace{8, 0}
-s:select{}
-
-s:drop()
-
-create_iterator = require('utils').create_iterator
-
---iterator test
-test_run:cmd("setopt delimiter ';'")
-
-fiber_status = 0
-
-function fiber_func()
- box.begin()
- s:replace{5, 5}
- fiber_status = 1
- local res = {pcall(box.commit) }
- fiber_status = 2
- return unpack(res)
-end;
-
-test_run:cmd("setopt delimiter ''");
-
-s = box.schema.space.create('test', {engine='vinyl'})
-_ = s:create_index('pk')
-fiber = require('fiber')
-
-_ = s:replace{0, 0}
-_ = s:replace{10, 0}
-_ = s:replace{20, 0}
-
-test_run:cmd("setopt delimiter ';'");
-
-faced_trash = false
-for i = 1,100 do
- errinj.set("ERRINJ_WAL_WRITE", true)
- local f = fiber.create(fiber_func)
- local itr = create_iterator(s, {0}, {iterator='GE'})
- local first = itr.next()
- local second = itr.next()
- if (second[1] ~= 5 and second[1] ~= 10) then faced_trash = true end
- while fiber_status <= 1 do fiber.sleep(0.001) end
- local _,next = pcall(itr.next)
- _,next = pcall(itr.next)
- _,next = pcall(itr.next)
- errinj.set("ERRINJ_WAL_WRITE", false)
- s:delete{5}
-end;
-
-test_run:cmd("setopt delimiter ''");
-
-faced_trash
-
-s:drop()
-
--- TX in prepared but not committed state
-s = box.schema.space.create('test', {engine='vinyl'})
-_ = s:create_index('pk')
-fiber = require('fiber')
-txn_proxy = require('txn_proxy')
-
-s:replace{1, "original"}
-s:replace{2, "original"}
-s:replace{3, "original"}
-
-c0 = txn_proxy.new()
-c0:begin()
-c1 = txn_proxy.new()
-c1:begin()
-c2 = txn_proxy.new()
-c2:begin()
-c3 = txn_proxy.new()
-c3:begin()
-
---
--- Prepared transactions
---
-
--- Pause WAL writer to cause all further calls to box.commit() to move
--- transactions into prepared, but not committed yet state.
-errinj.set("ERRINJ_WAL_DELAY", true)
-lsn = box.info.lsn
-c0('s:replace{1, "c0"}')
-c0('s:replace{2, "c0"}')
-c0('s:replace{3, "c0"}')
-_ = fiber.create(c0.commit, c0)
-box.info.lsn == lsn
-c1('s:replace{1, "c1"}')
-c1('s:replace{2, "c1"}')
-_ = fiber.create(c1.commit, c1)
-box.info.lsn == lsn
-c3('s:select{1}') -- c1 is visible
-c2('s:replace{1, "c2"}')
-c2('s:replace{3, "c2"}')
-_ = fiber.create(c2.commit, c2)
-box.info.lsn == lsn
-c3('s:select{1}') -- c1 is visible, c2 is not
-c3('s:select{2}') -- c1 is visible
-c3('s:select{3}') -- c2 is not visible
-
--- Resume WAL writer and wait until all transactions will been committed
-errinj.set("ERRINJ_WAL_DELAY", false)
-REQ_COUNT = 7
-while box.info.lsn - lsn < REQ_COUNT do fiber.sleep(0.01) end
-box.info.lsn == lsn + REQ_COUNT
-
-c3('s:select{1}') -- c1 is visible, c2 is not
-c3('s:select{2}') -- c1 is visible
-c3('s:select{3}') -- c2 is not visible
-c3:commit()
-
-s:drop()
-
---
--- Test mem restoration on a prepared and not commited statement
--- after moving iterator into read view.
---
-space = box.schema.space.create('test', {engine = 'vinyl'})
-pk = space:create_index('pk')
-space:replace{1}
-space:replace{2}
-space:replace{3}
-
-last_read = nil
-
-errinj.set("ERRINJ_WAL_DELAY", true)
-
-test_run:cmd("setopt delimiter ';'")
-
-function fill_space()
- box.begin()
- space:replace{1}
- space:replace{2}
- space:replace{3}
--- block until wal_delay = false
- box.commit()
--- send iterator to read view
- space:replace{1, 1}
--- flush mem and update index version to trigger iterator restore
- box.snapshot()
-end;
-
-function iterate_in_read_view()
- local i = create_iterator(space)
- last_read = i.next()
- fiber.sleep(100000)
- last_read = i.next()
-end;
-
-test_run:cmd("setopt delimiter ''");
-
-f1 = fiber.create(fill_space)
--- Prepared transaction is blocked due to wal_delay.
--- Start iterator with vlsn = INT64_MAX
-f2 = fiber.create(iterate_in_read_view)
-last_read
--- Finish prepared transaction and send to read view the iterator.
-errinj.set("ERRINJ_WAL_DELAY", false)
-while f1:status() ~= 'dead' do fiber.sleep(0.01) end
-f2:wakeup()
-while f2:status() ~= 'dead' do fiber.sleep(0.01) end
-last_read
-
-space:drop()
-
--
-- Space drop in the middle of dump.
--
@@ -389,21 +196,6 @@ t2 = fiber.time()
t2 - t1 < 1
test_run:cmd("cleanup server test")
---
--- If we logged an index creation in the metadata log before WAL write,
--- WAL failure would result in leaving the index record in vylog forever.
--- Since we use LSN to identify indexes in vylog, retrying index creation
--- would then lead to a duplicate index id in vylog and hence inability
--- to make a snapshot or recover.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
-errinj.set('ERRINJ_WAL_IO', true)
-_ = s:create_index('pk')
-errinj.set('ERRINJ_WAL_IO', false)
-_ = s:create_index('pk')
-box.snapshot()
-s:drop()
-
s = box.schema.space.create('test', {engine = 'vinyl'})
_ = s:create_index('i1', {parts = {1, 'unsigned'}})
@@ -541,41 +333,6 @@ test_run:cmd("stop server low_quota")
test_run:cmd("cleanup server low_quota")
--
--- Check that ALTER is abroted if a tuple inserted during space
--- format change does not conform to the new format.
---
-format = {}
-format[1] = {name = 'field1', type = 'unsigned'}
-format[2] = {name = 'field2', type = 'string', is_nullable = true}
-s = box.schema.space.create('test', {engine = 'vinyl', format = format})
-_ = s:create_index('pk', {page_size = 16})
-
-pad = string.rep('x', 16)
-for i = 101, 200 do s:replace{i, pad} end
-box.snapshot()
-
-ch = fiber.channel(1)
-test_run:cmd("setopt delimiter ';'")
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i, box.NULL}
- end
- ch:put(true)
-end);
-test_run:cmd("setopt delimiter ''");
-
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
-format[2].is_nullable = false
-s:format(format) -- must fail
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
-
-ch:get()
-
-s:count() -- 200
-s:drop()
-
---
-- gh-3437: if compaction races with checkpointing, it may remove
-- files needed for backup.
--
@@ -604,140 +361,6 @@ box.backup.stop()
s:drop()
--
--- gh-2449: change 'unique' index property from true to false
--- is done without index rebuild.
---
-s = box.schema.space.create('test', { engine = 'vinyl' })
-_ = s:create_index('primary')
-_ = s:create_index('secondary', {unique = true, parts = {2, 'unsigned'}})
-s:insert{1, 10}
-box.snapshot()
-errinj.set("ERRINJ_VY_READ_PAGE", true);
-s.index.secondary:alter{unique = false} -- ok
-s.index.secondary.unique
-s.index.secondary:alter{unique = true} -- error
-s.index.secondary.unique
-errinj.set("ERRINJ_VY_READ_PAGE", false);
-s:insert{2, 10}
-s.index.secondary:select(10)
-s:drop()
-
---
--- Check that ALTER is aborted if a tuple inserted during index build
--- doesn't conform to the new format.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
-_ = s:create_index('pk', {page_size = 16})
-
-pad = string.rep('x', 16)
-for i = 101, 200 do s:replace{i, i, pad} end
-box.snapshot()
-
-ch = fiber.channel(1)
-test_run:cmd("setopt delimiter ';'")
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i}
- end
- ch:put(true)
-end);
-test_run:cmd("setopt delimiter ''");
-
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
-s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
-
-ch:get()
-
-s:count() -- 200
-s:drop()
-
---
--- Check that ALTER is aborted if a tuple inserted during index build
--- violates unique constraint.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
-_ = s:create_index('pk', {page_size = 16})
-
-pad = string.rep('x', 16)
-for i = 101, 200 do s:replace{i, i, pad} end
-box.snapshot()
-
-ch = fiber.channel(1)
-test_run:cmd("setopt delimiter ';'")
-_ = fiber.create(function()
- fiber.sleep(0.01)
- for i = 1, 100 do
- s:replace{i, i + 1}
- end
- ch:put(true)
-end);
-test_run:cmd("setopt delimiter ''");
-
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
-s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
-
-ch:get()
-
-s:count() -- 200
-s:drop()
-
---
--- Check that modifications done to the space during the final dump
--- of a newly built index are recovered properly.
---
-s = box.schema.space.create('test', {engine = 'vinyl'})
-_ = s:create_index('pk')
-
-for i = 1, 5 do s:replace{i, i} end
-
-errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", true)
-ch = fiber.channel(1)
-_ = fiber.create(function() s:create_index('sk', {parts = {2, 'integer'}}) ch:put(true) end)
-
-fiber.sleep(0.01)
-
-_ = s:delete{1}
-_ = s:replace{2, -2}
-_ = s:delete{2}
-_ = s:replace{3, -3}
-_ = s:replace{3, -2}
-_ = s:replace{3, -1}
-_ = s:delete{3}
-_ = s:upsert({3, 3}, {{'=', 2, 1}})
-_ = s:upsert({3, 3}, {{'=', 2, 2}})
-_ = s:delete{3}
-_ = s:replace{4, -1}
-_ = s:replace{4, -2}
-_ = s:replace{4, -4}
-_ = s:upsert({5, 1}, {{'=', 2, 1}})
-_ = s:upsert({5, 2}, {{'=', 2, -5}})
-_ = s:replace{6, -6}
-_ = s:upsert({7, -7}, {{'=', 2, -7}})
-
-errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", false)
-ch:get()
-
-s.index.sk:select()
-s.index.sk:stat().memory.rows
-
-test_run:cmd('restart server default')
-
-s = box.space.test
-
-s.index.sk:select()
-s.index.sk:stat().memory.rows
-
-box.snapshot()
-
-s.index.sk:select()
-s.index.sk:stat().memory.rows
-
-s:drop()
-
---
-- Check that tarantool doesn't hang or crash if error
-- occurs while writing a deferred DELETE to WAL.
--
@@ -768,123 +391,6 @@ box.snapshot() -- ok
s:drop()
--
--- gh-3458: check that rw transactions that started before DDL are
--- aborted.
---
-vinyl_cache = box.cfg.vinyl_cache
-box.cfg{vinyl_cache = 0}
-
-s1 = box.schema.space.create('test1', {engine = 'vinyl'})
-_ = s1:create_index('pk', {page_size = 16})
-s2 = box.schema.space.create('test2', {engine = 'vinyl'})
-_ = s2:create_index('pk')
-
-pad = string.rep('x', 16)
-for i = 101, 200 do s1:replace{i, i, pad} end
-box.snapshot()
-
-test_run:cmd("setopt delimiter ';'")
-function async_replace(space, tuple, timeout)
- local c = fiber.channel(1)
- fiber.create(function()
- box.begin()
- space:replace(tuple)
- fiber.sleep(timeout)
- local status = pcall(box.commit)
- c:put(status)
- end)
- return c
-end;
-test_run:cmd("setopt delimiter ''");
-
-c1 = async_replace(s1, {1}, 0.01)
-c2 = async_replace(s2, {1}, 0.01)
-
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
-s1:format{{'key', 'unsigned'}, {'value', 'unsigned'}}
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
-
-c1:get() -- false (transaction was aborted)
-c2:get() -- true
-
-s1:get(1) == nil
-s2:get(1) ~= nil
-s1:format()
-s1:format{}
-
-c1 = async_replace(s1, {2}, 0.01)
-c2 = async_replace(s2, {2}, 0.01)
-
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
-_ = s1:create_index('sk', {parts = {2, 'unsigned'}})
-errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
-
-c1:get() -- false (transaction was aborted)
-c2:get() -- true
-
-s1:get(2) == nil
-s2:get(2) ~= nil
-s1.index.pk:count() == s1.index.sk:count()
-
-s1:drop()
-s2:drop()
-box.cfg{vinyl_cache = vinyl_cache}
-
--- Transactions that reached WAL must not be aborted.
-s = box.schema.space.create('test', {engine = 'vinyl'})
-_ = s:create_index('pk')
-
-errinj.set('ERRINJ_WAL_DELAY', true)
-_ = fiber.create(function() s:replace{1} end)
-_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
-
-fiber.sleep(0)
-s:format{{'key', 'unsigned'}, {'value', 'unsigned'}} -- must fail
-s:select()
-s:truncate()
-
-errinj.set('ERRINJ_WAL_DELAY', true)
-_ = fiber.create(function() s:replace{1} end)
-_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
-
-fiber.sleep(0)
-s:create_index('sk', {parts = {2, 'unsigned'}})
-s:select()
-s:drop()
-
---
--- Check disk.compact.queue stat.
---
-test_run:cmd("push filter 'bytes_compressed: .*' to 'bytes_compressed: <bytes_compressed>'")
-
-s = box.schema.space.create('test', {engine = 'vinyl'})
-i = s:create_index('pk', {run_count_per_level = 2})
-function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
-dump()
-i:stat().disk.compact.queue -- none
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
-errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
-dump()
-dump()
-i:stat().disk.compact.queue -- 30 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
-dump()
-i:stat().disk.compact.queue -- 40 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
-dump()
-i:stat().disk.compact.queue -- 50 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
-box.stat.reset() -- doesn't affect queue size
-i:stat().disk.compact.queue -- 50 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
-errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
-while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
-i:stat().disk.compact.queue -- none
-s:drop()
-
-test_run:cmd("clear filter")
-
---
-- Check that an instance doesn't crash if a run file needed for
-- joining a replica is corrupted (see gh-3708).
--
diff --git a/test/vinyl/errinj_ddl.result b/test/vinyl/errinj_ddl.result
new file mode 100644
index 00000000..4f5d59cd
--- /dev/null
+++ b/test/vinyl/errinj_ddl.result
@@ -0,0 +1,595 @@
+test_run = require('test_run').new()
+---
+...
+fiber = require('fiber')
+---
+...
+errinj = box.error.injection
+---
+...
+--
+-- Check that ALTER is abroted if a tuple inserted during space
+-- format change does not conform to the new format.
+--
+format = {}
+---
+...
+format[1] = {name = 'field1', type = 'unsigned'}
+---
+...
+format[2] = {name = 'field2', type = 'string', is_nullable = true}
+---
+...
+s = box.schema.space.create('test', {engine = 'vinyl', format = format})
+---
+...
+_ = s:create_index('pk', {page_size = 16})
+---
+...
+pad = string.rep('x', 16)
+---
+...
+for i = 101, 200 do s:replace{i, pad} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+ch = fiber.channel(1)
+---
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i, box.NULL}
+ end
+ ch:put(true)
+end);
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+---
+- ok
+...
+format[2].is_nullable = false
+---
+...
+s:format(format) -- must fail
+---
+- error: 'Tuple field 2 type does not match one required by operation: expected string'
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+---
+- ok
+...
+ch:get()
+---
+- true
+...
+s:count() -- 200
+---
+- 200
+...
+s:drop()
+---
+...
+--
+-- gh-2449: change 'unique' index property from true to false
+-- is done without index rebuild.
+--
+s = box.schema.space.create('test', { engine = 'vinyl' })
+---
+...
+_ = s:create_index('primary')
+---
+...
+_ = s:create_index('secondary', {unique = true, parts = {2, 'unsigned'}})
+---
+...
+s:insert{1, 10}
+---
+- [1, 10]
+...
+box.snapshot()
+---
+- ok
+...
+errinj.set("ERRINJ_VY_READ_PAGE", true);
+---
+- ok
+...
+s.index.secondary:alter{unique = false} -- ok
+---
+...
+s.index.secondary.unique
+---
+- false
+...
+s.index.secondary:alter{unique = true} -- error
+---
+- error: Error injection 'vinyl page read'
+...
+s.index.secondary.unique
+---
+- false
+...
+errinj.set("ERRINJ_VY_READ_PAGE", false);
+---
+- ok
+...
+s:insert{2, 10}
+---
+- [2, 10]
+...
+s.index.secondary:select(10)
+---
+- - [1, 10]
+ - [2, 10]
+...
+s:drop()
+---
+...
+--
+-- Check that ALTER is aborted if a tuple inserted during index build
+-- doesn't conform to the new format.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+_ = s:create_index('pk', {page_size = 16})
+---
+...
+pad = string.rep('x', 16)
+---
+...
+for i = 101, 200 do s:replace{i, i, pad} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+ch = fiber.channel(1)
+---
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i}
+ end
+ ch:put(true)
+end);
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+---
+- ok
+...
+s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
+---
+- error: Tuple field 2 required by space format is missing
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+---
+- ok
+...
+ch:get()
+---
+- true
+...
+s:count() -- 200
+---
+- 200
+...
+s:drop()
+---
+...
+--
+-- Check that ALTER is aborted if a tuple inserted during index build
+-- violates unique constraint.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+_ = s:create_index('pk', {page_size = 16})
+---
+...
+pad = string.rep('x', 16)
+---
+...
+for i = 101, 200 do s:replace{i, i, pad} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+ch = fiber.channel(1)
+---
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i, i + 1}
+ end
+ ch:put(true)
+end);
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+---
+- ok
+...
+s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
+---
+- error: Duplicate key exists in unique index 'sk' in space 'test'
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+---
+- ok
+...
+ch:get()
+---
+- true
+...
+s:count() -- 200
+---
+- 200
+...
+s:drop()
+---
+...
+--
+-- Check that modifications done to the space during the final dump
+-- of a newly built index are recovered properly.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+for i = 1, 5 do s:replace{i, i} end
+---
+...
+errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", true)
+---
+- ok
+...
+ch = fiber.channel(1)
+---
+...
+_ = fiber.create(function() s:create_index('sk', {parts = {2, 'integer'}}) ch:put(true) end)
+---
+...
+fiber.sleep(0.01)
+---
+...
+_ = s:delete{1}
+---
+...
+_ = s:replace{2, -2}
+---
+...
+_ = s:delete{2}
+---
+...
+_ = s:replace{3, -3}
+---
+...
+_ = s:replace{3, -2}
+---
+...
+_ = s:replace{3, -1}
+---
+...
+_ = s:delete{3}
+---
+...
+_ = s:upsert({3, 3}, {{'=', 2, 1}})
+---
+...
+_ = s:upsert({3, 3}, {{'=', 2, 2}})
+---
+...
+_ = s:delete{3}
+---
+...
+_ = s:replace{4, -1}
+---
+...
+_ = s:replace{4, -2}
+---
+...
+_ = s:replace{4, -4}
+---
+...
+_ = s:upsert({5, 1}, {{'=', 2, 1}})
+---
+...
+_ = s:upsert({5, 2}, {{'=', 2, -5}})
+---
+...
+_ = s:replace{6, -6}
+---
+...
+_ = s:upsert({7, -7}, {{'=', 2, -7}})
+---
+...
+errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", false)
+---
+- ok
+...
+ch:get()
+---
+- true
+...
+s.index.sk:select()
+---
+- - [7, -7]
+ - [6, -6]
+ - [5, -5]
+ - [4, -4]
+...
+s.index.sk:stat().memory.rows
+---
+- 27
+...
+test_run:cmd('restart server default')
+fiber = require('fiber')
+---
+...
+errinj = box.error.injection
+---
+...
+s = box.space.test
+---
+...
+s.index.sk:select()
+---
+- - [7, -7]
+ - [6, -6]
+ - [5, -5]
+ - [4, -4]
+...
+s.index.sk:stat().memory.rows
+---
+- 27
+...
+box.snapshot()
+---
+- ok
+...
+s.index.sk:select()
+---
+- - [7, -7]
+ - [6, -6]
+ - [5, -5]
+ - [4, -4]
+...
+s.index.sk:stat().memory.rows
+---
+- 0
+...
+s:drop()
+---
+...
+--
+-- gh-3458: check that rw transactions that started before DDL are
+-- aborted.
+--
+vinyl_cache = box.cfg.vinyl_cache
+---
+...
+box.cfg{vinyl_cache = 0}
+---
+...
+s1 = box.schema.space.create('test1', {engine = 'vinyl'})
+---
+...
+_ = s1:create_index('pk', {page_size = 16})
+---
+...
+s2 = box.schema.space.create('test2', {engine = 'vinyl'})
+---
+...
+_ = s2:create_index('pk')
+---
+...
+pad = string.rep('x', 16)
+---
+...
+for i = 101, 200 do s1:replace{i, i, pad} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+function async_replace(space, tuple, timeout)
+ local c = fiber.channel(1)
+ fiber.create(function()
+ box.begin()
+ space:replace(tuple)
+ fiber.sleep(timeout)
+ local status = pcall(box.commit)
+ c:put(status)
+ end)
+ return c
+end;
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+c1 = async_replace(s1, {1}, 0.01)
+---
+...
+c2 = async_replace(s2, {1}, 0.01)
+---
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+---
+- ok
+...
+s1:format{{'key', 'unsigned'}, {'value', 'unsigned'}}
+---
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+---
+- ok
+...
+c1:get() -- false (transaction was aborted)
+---
+- false
+...
+c2:get() -- true
+---
+- true
+...
+s1:get(1) == nil
+---
+- true
+...
+s2:get(1) ~= nil
+---
+- true
+...
+s1:format()
+---
+- [{'name': 'key', 'type': 'unsigned'}, {'name': 'value', 'type': 'unsigned'}]
+...
+s1:format{}
+---
+...
+c1 = async_replace(s1, {2}, 0.01)
+---
+...
+c2 = async_replace(s2, {2}, 0.01)
+---
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+---
+- ok
+...
+_ = s1:create_index('sk', {parts = {2, 'unsigned'}})
+---
+...
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+---
+- ok
+...
+c1:get() -- false (transaction was aborted)
+---
+- false
+...
+c2:get() -- true
+---
+- true
+...
+s1:get(2) == nil
+---
+- true
+...
+s2:get(2) ~= nil
+---
+- true
+...
+s1.index.pk:count() == s1.index.sk:count()
+---
+- true
+...
+s1:drop()
+---
+...
+s2:drop()
+---
+...
+box.cfg{vinyl_cache = vinyl_cache}
+---
+...
+-- Transactions that reached WAL must not be aborted.
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+errinj.set('ERRINJ_WAL_DELAY', true)
+---
+- ok
+...
+_ = fiber.create(function() s:replace{1} end)
+---
+...
+_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
+---
+...
+fiber.sleep(0)
+---
+...
+s:format{{'key', 'unsigned'}, {'value', 'unsigned'}} -- must fail
+---
+- error: Tuple field 2 required by space format is missing
+...
+s:select()
+---
+- - [1]
+...
+s:truncate()
+---
+...
+errinj.set('ERRINJ_WAL_DELAY', true)
+---
+- ok
+...
+_ = fiber.create(function() s:replace{1} end)
+---
+...
+_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
+---
+...
+fiber.sleep(0)
+---
+...
+s:create_index('sk', {parts = {2, 'unsigned'}})
+---
+- error: Tuple field 2 required by space format is missing
+...
+s:select()
+---
+- - [1]
+...
+s:drop()
+---
+...
diff --git a/test/vinyl/errinj_ddl.test.lua b/test/vinyl/errinj_ddl.test.lua
new file mode 100644
index 00000000..0948bc3d
--- /dev/null
+++ b/test/vinyl/errinj_ddl.test.lua
@@ -0,0 +1,260 @@
+test_run = require('test_run').new()
+fiber = require('fiber')
+errinj = box.error.injection
+
+--
+-- Check that ALTER is abroted if a tuple inserted during space
+-- format change does not conform to the new format.
+--
+format = {}
+format[1] = {name = 'field1', type = 'unsigned'}
+format[2] = {name = 'field2', type = 'string', is_nullable = true}
+s = box.schema.space.create('test', {engine = 'vinyl', format = format})
+_ = s:create_index('pk', {page_size = 16})
+
+pad = string.rep('x', 16)
+for i = 101, 200 do s:replace{i, pad} end
+box.snapshot()
+
+ch = fiber.channel(1)
+test_run:cmd("setopt delimiter ';'")
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i, box.NULL}
+ end
+ ch:put(true)
+end);
+test_run:cmd("setopt delimiter ''");
+
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+format[2].is_nullable = false
+s:format(format) -- must fail
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+
+ch:get()
+
+s:count() -- 200
+s:drop()
+
+--
+-- gh-2449: change 'unique' index property from true to false
+-- is done without index rebuild.
+--
+s = box.schema.space.create('test', { engine = 'vinyl' })
+_ = s:create_index('primary')
+_ = s:create_index('secondary', {unique = true, parts = {2, 'unsigned'}})
+s:insert{1, 10}
+box.snapshot()
+errinj.set("ERRINJ_VY_READ_PAGE", true);
+s.index.secondary:alter{unique = false} -- ok
+s.index.secondary.unique
+s.index.secondary:alter{unique = true} -- error
+s.index.secondary.unique
+errinj.set("ERRINJ_VY_READ_PAGE", false);
+s:insert{2, 10}
+s.index.secondary:select(10)
+s:drop()
+
+--
+-- Check that ALTER is aborted if a tuple inserted during index build
+-- doesn't conform to the new format.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+_ = s:create_index('pk', {page_size = 16})
+
+pad = string.rep('x', 16)
+for i = 101, 200 do s:replace{i, i, pad} end
+box.snapshot()
+
+ch = fiber.channel(1)
+test_run:cmd("setopt delimiter ';'")
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i}
+ end
+ ch:put(true)
+end);
+test_run:cmd("setopt delimiter ''");
+
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+
+ch:get()
+
+s:count() -- 200
+s:drop()
+
+--
+-- Check that ALTER is aborted if a tuple inserted during index build
+-- violates unique constraint.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+_ = s:create_index('pk', {page_size = 16})
+
+pad = string.rep('x', 16)
+for i = 101, 200 do s:replace{i, i, pad} end
+box.snapshot()
+
+ch = fiber.channel(1)
+test_run:cmd("setopt delimiter ';'")
+_ = fiber.create(function()
+ fiber.sleep(0.01)
+ for i = 1, 100 do
+ s:replace{i, i + 1}
+ end
+ ch:put(true)
+end);
+test_run:cmd("setopt delimiter ''");
+
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+s:create_index('sk', {parts = {2, 'unsigned'}}) -- must fail
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+
+ch:get()
+
+s:count() -- 200
+s:drop()
+
+--
+-- Check that modifications done to the space during the final dump
+-- of a newly built index are recovered properly.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+_ = s:create_index('pk')
+
+for i = 1, 5 do s:replace{i, i} end
+
+errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", true)
+ch = fiber.channel(1)
+_ = fiber.create(function() s:create_index('sk', {parts = {2, 'integer'}}) ch:put(true) end)
+
+fiber.sleep(0.01)
+
+_ = s:delete{1}
+_ = s:replace{2, -2}
+_ = s:delete{2}
+_ = s:replace{3, -3}
+_ = s:replace{3, -2}
+_ = s:replace{3, -1}
+_ = s:delete{3}
+_ = s:upsert({3, 3}, {{'=', 2, 1}})
+_ = s:upsert({3, 3}, {{'=', 2, 2}})
+_ = s:delete{3}
+_ = s:replace{4, -1}
+_ = s:replace{4, -2}
+_ = s:replace{4, -4}
+_ = s:upsert({5, 1}, {{'=', 2, 1}})
+_ = s:upsert({5, 2}, {{'=', 2, -5}})
+_ = s:replace{6, -6}
+_ = s:upsert({7, -7}, {{'=', 2, -7}})
+
+errinj.set("ERRINJ_VY_RUN_WRITE_DELAY", false)
+ch:get()
+
+s.index.sk:select()
+s.index.sk:stat().memory.rows
+
+test_run:cmd('restart server default')
+
+fiber = require('fiber')
+errinj = box.error.injection
+
+s = box.space.test
+
+s.index.sk:select()
+s.index.sk:stat().memory.rows
+
+box.snapshot()
+
+s.index.sk:select()
+s.index.sk:stat().memory.rows
+
+s:drop()
+
+--
+-- gh-3458: check that rw transactions that started before DDL are
+-- aborted.
+--
+vinyl_cache = box.cfg.vinyl_cache
+box.cfg{vinyl_cache = 0}
+
+s1 = box.schema.space.create('test1', {engine = 'vinyl'})
+_ = s1:create_index('pk', {page_size = 16})
+s2 = box.schema.space.create('test2', {engine = 'vinyl'})
+_ = s2:create_index('pk')
+
+pad = string.rep('x', 16)
+for i = 101, 200 do s1:replace{i, i, pad} end
+box.snapshot()
+
+test_run:cmd("setopt delimiter ';'")
+function async_replace(space, tuple, timeout)
+ local c = fiber.channel(1)
+ fiber.create(function()
+ box.begin()
+ space:replace(tuple)
+ fiber.sleep(timeout)
+ local status = pcall(box.commit)
+ c:put(status)
+ end)
+ return c
+end;
+test_run:cmd("setopt delimiter ''");
+
+c1 = async_replace(s1, {1}, 0.01)
+c2 = async_replace(s2, {1}, 0.01)
+
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+s1:format{{'key', 'unsigned'}, {'value', 'unsigned'}}
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+
+c1:get() -- false (transaction was aborted)
+c2:get() -- true
+
+s1:get(1) == nil
+s2:get(1) ~= nil
+s1:format()
+s1:format{}
+
+c1 = async_replace(s1, {2}, 0.01)
+c2 = async_replace(s2, {2}, 0.01)
+
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0.001)
+_ = s1:create_index('sk', {parts = {2, 'unsigned'}})
+errinj.set("ERRINJ_VY_READ_PAGE_TIMEOUT", 0)
+
+c1:get() -- false (transaction was aborted)
+c2:get() -- true
+
+s1:get(2) == nil
+s2:get(2) ~= nil
+s1.index.pk:count() == s1.index.sk:count()
+
+s1:drop()
+s2:drop()
+box.cfg{vinyl_cache = vinyl_cache}
+
+-- Transactions that reached WAL must not be aborted.
+s = box.schema.space.create('test', {engine = 'vinyl'})
+_ = s:create_index('pk')
+
+errinj.set('ERRINJ_WAL_DELAY', true)
+_ = fiber.create(function() s:replace{1} end)
+_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
+
+fiber.sleep(0)
+s:format{{'key', 'unsigned'}, {'value', 'unsigned'}} -- must fail
+s:select()
+s:truncate()
+
+errinj.set('ERRINJ_WAL_DELAY', true)
+_ = fiber.create(function() s:replace{1} end)
+_ = fiber.create(function() fiber.sleep(0.01) errinj.set('ERRINJ_WAL_DELAY', false) end)
+
+fiber.sleep(0)
+s:create_index('sk', {parts = {2, 'unsigned'}})
+s:select()
+s:drop()
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
new file mode 100644
index 00000000..fac56cee
--- /dev/null
+++ b/test/vinyl/errinj_stat.result
@@ -0,0 +1,152 @@
+test_run = require('test_run').new()
+---
+...
+-- Since we store LSNs in data files, the data size may differ
+-- from run to run. Deploy a new server to make sure it will be
+-- the same so that we can check it.
+test_run:cmd('create server test with script = "vinyl/stat.lua"')
+---
+- true
+...
+test_run:cmd('start server test')
+---
+- true
+...
+test_run:cmd('switch test')
+---
+- true
+...
+-- Compressed data size depends on the zstd version so let's
+-- filter it out.
+test_run:cmd("push filter 'bytes_compressed: .*' to 'bytes_compressed: <bytes_compressed>'")
+---
+- true
+...
+fiber = require('fiber')
+---
+...
+errinj = box.error.injection
+---
+...
+--
+-- Check disk.compact.queue stat.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+i = s:create_index('pk', {run_count_per_level = 2})
+---
+...
+function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
+---
+...
+dump()
+---
+...
+i:stat().disk.compact.queue -- none
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+...
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+---
+- true
+...
+errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
+---
+- ok
+...
+dump()
+---
+...
+dump()
+---
+...
+i:stat().disk.compact.queue -- 30 statements
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 3
+ rows: 30
+ bytes: 411
+...
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+---
+- true
+...
+dump()
+---
+...
+i:stat().disk.compact.queue -- 40 statements
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 4
+ rows: 40
+ bytes: 548
+...
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+---
+- true
+...
+dump()
+---
+...
+i:stat().disk.compact.queue -- 50 statements
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 5
+ rows: 50
+ bytes: 685
+...
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+---
+- true
+...
+box.stat.reset() -- doesn't affect queue size
+---
+...
+i:stat().disk.compact.queue -- 50 statements
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 5
+ rows: 50
+ bytes: 685
+...
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+---
+- true
+...
+errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
+---
+- ok
+...
+while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
+---
+...
+i:stat().disk.compact.queue -- none
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+...
+s:drop()
+---
+...
+test_run:cmd("clear filter")
+---
+- true
+...
+test_run:cmd('switch default')
+---
+- true
+...
+test_run:cmd('stop server test')
+---
+- true
+...
+test_run:cmd('cleanup server test')
+---
+- true
+...
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
new file mode 100644
index 00000000..1e0e63ae
--- /dev/null
+++ b/test/vinyl/errinj_stat.test.lua
@@ -0,0 +1,48 @@
+test_run = require('test_run').new()
+
+-- Since we store LSNs in data files, the data size may differ
+-- from run to run. Deploy a new server to make sure it will be
+-- the same so that we can check it.
+test_run:cmd('create server test with script = "vinyl/stat.lua"')
+test_run:cmd('start server test')
+test_run:cmd('switch test')
+
+-- Compressed data size depends on the zstd version so let's
+-- filter it out.
+test_run:cmd("push filter 'bytes_compressed: .*' to 'bytes_compressed: <bytes_compressed>'")
+
+fiber = require('fiber')
+errinj = box.error.injection
+
+--
+-- Check disk.compact.queue stat.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+i = s:create_index('pk', {run_count_per_level = 2})
+function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
+dump()
+i:stat().disk.compact.queue -- none
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
+dump()
+dump()
+i:stat().disk.compact.queue -- 30 statements
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+dump()
+i:stat().disk.compact.queue -- 40 statements
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+dump()
+i:stat().disk.compact.queue -- 50 statements
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+box.stat.reset() -- doesn't affect queue size
+i:stat().disk.compact.queue -- 50 statements
+i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
+while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
+i:stat().disk.compact.queue -- none
+s:drop()
+
+test_run:cmd("clear filter")
+test_run:cmd('switch default')
+test_run:cmd('stop server test')
+test_run:cmd('cleanup server test')
diff --git a/test/vinyl/errinj_tx.result b/test/vinyl/errinj_tx.result
new file mode 100644
index 00000000..a7583b2e
--- /dev/null
+++ b/test/vinyl/errinj_tx.result
@@ -0,0 +1,435 @@
+test_run = require('test_run').new()
+---
+...
+fiber = require('fiber')
+---
+...
+txn_proxy = require('txn_proxy')
+---
+...
+create_iterator = require('utils').create_iterator
+---
+...
+errinj = box.error.injection
+---
+...
+--
+-- gh-1681: vinyl: crash in vy_rollback on ER_WAL_WRITE
+--
+s = box.schema.space.create('test', {engine='vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+function f() box.begin() s:insert{1, 'hi'} s:insert{2, 'bye'} box.commit() end
+---
+...
+errinj.set("ERRINJ_WAL_WRITE", true)
+---
+- ok
+...
+f()
+---
+- error: Failed to write to disk
+...
+s:select{}
+---
+- []
+...
+errinj.set("ERRINJ_WAL_WRITE", false)
+---
+- ok
+...
+f()
+---
+...
+s:select{}
+---
+- - [1, 'hi']
+ - [2, 'bye']
+...
+s:drop()
+---
+...
+--https://github.com/tarantool/tarantool/issues/1842
+--test error injection
+s = box.schema.space.create('test', {engine='vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+s:replace{0, 0}
+---
+- [0, 0]
+...
+s:replace{1, 0}
+---
+- [1, 0]
+...
+s:replace{2, 0}
+---
+- [2, 0]
+...
+errinj.set("ERRINJ_WAL_WRITE", true)
+---
+- ok
+...
+s:replace{3, 0}
+---
+- error: Failed to write to disk
+...
+s:replace{4, 0}
+---
+- error: Failed to write to disk
+...
+s:replace{5, 0}
+---
+- error: Failed to write to disk
+...
+s:replace{6, 0}
+---
+- error: Failed to write to disk
+...
+errinj.set("ERRINJ_WAL_WRITE", false)
+---
+- ok
+...
+s:replace{7, 0}
+---
+- [7, 0]
+...
+s:replace{8, 0}
+---
+- [8, 0]
+...
+s:select{}
+---
+- - [0, 0]
+ - [1, 0]
+ - [2, 0]
+ - [7, 0]
+ - [8, 0]
+...
+s:drop()
+---
+...
+--iterator test
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+fiber_status = 0
+
+function fiber_func()
+ box.begin()
+ s:replace{5, 5}
+ fiber_status = 1
+ local res = {pcall(box.commit) }
+ fiber_status = 2
+ return unpack(res)
+end;
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+s = box.schema.space.create('test', {engine='vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+_ = s:replace{0, 0}
+---
+...
+_ = s:replace{10, 0}
+---
+...
+_ = s:replace{20, 0}
+---
+...
+test_run:cmd("setopt delimiter ';'");
+---
+- true
+...
+faced_trash = false
+for i = 1,100 do
+ errinj.set("ERRINJ_WAL_WRITE", true)
+ local f = fiber.create(fiber_func)
+ local itr = create_iterator(s, {0}, {iterator='GE'})
+ local first = itr.next()
+ local second = itr.next()
+ if (second[1] ~= 5 and second[1] ~= 10) then faced_trash = true end
+ while fiber_status <= 1 do fiber.sleep(0.001) end
+ local _,next = pcall(itr.next)
+ _,next = pcall(itr.next)
+ _,next = pcall(itr.next)
+ errinj.set("ERRINJ_WAL_WRITE", false)
+ s:delete{5}
+end;
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+faced_trash
+---
+- false
+...
+s:drop()
+---
+...
+-- TX in prepared but not committed state
+s = box.schema.space.create('test', {engine='vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+s:replace{1, "original"}
+---
+- [1, 'original']
+...
+s:replace{2, "original"}
+---
+- [2, 'original']
+...
+s:replace{3, "original"}
+---
+- [3, 'original']
+...
+c0 = txn_proxy.new()
+---
+...
+c0:begin()
+---
+-
+...
+c1 = txn_proxy.new()
+---
+...
+c1:begin()
+---
+-
+...
+c2 = txn_proxy.new()
+---
+...
+c2:begin()
+---
+-
+...
+c3 = txn_proxy.new()
+---
+...
+c3:begin()
+---
+-
+...
+--
+-- Prepared transactions
+--
+-- Pause WAL writer to cause all further calls to box.commit() to move
+-- transactions into prepared, but not committed yet state.
+errinj.set("ERRINJ_WAL_DELAY", true)
+---
+- ok
+...
+lsn = box.info.lsn
+---
+...
+c0('s:replace{1, "c0"}')
+---
+- - [1, 'c0']
+...
+c0('s:replace{2, "c0"}')
+---
+- - [2, 'c0']
+...
+c0('s:replace{3, "c0"}')
+---
+- - [3, 'c0']
+...
+_ = fiber.create(c0.commit, c0)
+---
+...
+box.info.lsn == lsn
+---
+- true
+...
+c1('s:replace{1, "c1"}')
+---
+- - [1, 'c1']
+...
+c1('s:replace{2, "c1"}')
+---
+- - [2, 'c1']
+...
+_ = fiber.create(c1.commit, c1)
+---
+...
+box.info.lsn == lsn
+---
+- true
+...
+c3('s:select{1}') -- c1 is visible
+---
+- - [[1, 'c1']]
+...
+c2('s:replace{1, "c2"}')
+---
+- - [1, 'c2']
+...
+c2('s:replace{3, "c2"}')
+---
+- - [3, 'c2']
+...
+_ = fiber.create(c2.commit, c2)
+---
+...
+box.info.lsn == lsn
+---
+- true
+...
+c3('s:select{1}') -- c1 is visible, c2 is not
+---
+- - [[1, 'c1']]
+...
+c3('s:select{2}') -- c1 is visible
+---
+- - [[2, 'c1']]
+...
+c3('s:select{3}') -- c2 is not visible
+---
+- - [[3, 'c0']]
+...
+-- Resume WAL writer and wait until all transactions will been committed
+errinj.set("ERRINJ_WAL_DELAY", false)
+---
+- ok
+...
+REQ_COUNT = 7
+---
+...
+while box.info.lsn - lsn < REQ_COUNT do fiber.sleep(0.01) end
+---
+...
+box.info.lsn == lsn + REQ_COUNT
+---
+- true
+...
+c3('s:select{1}') -- c1 is visible, c2 is not
+---
+- - [[1, 'c1']]
+...
+c3('s:select{2}') -- c1 is visible
+---
+- - [[2, 'c1']]
+...
+c3('s:select{3}') -- c2 is not visible
+---
+- - [[3, 'c0']]
+...
+c3:commit()
+---
+-
+...
+s:drop()
+---
+...
+--
+-- Test mem restoration on a prepared and not commited statement
+-- after moving iterator into read view.
+--
+space = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+pk = space:create_index('pk')
+---
+...
+space:replace{1}
+---
+- [1]
+...
+space:replace{2}
+---
+- [2]
+...
+space:replace{3}
+---
+- [3]
+...
+last_read = nil
+---
+...
+errinj.set("ERRINJ_WAL_DELAY", true)
+---
+- ok
+...
+test_run:cmd("setopt delimiter ';'")
+---
+- true
+...
+-- block until wal_delay = false
+-- send iterator to read view
+-- flush mem and update index version to trigger iterator restore
+function fill_space()
+ box.begin()
+ space:replace{1}
+ space:replace{2}
+ space:replace{3}
+ box.commit()
+ space:replace{1, 1}
+ box.snapshot()
+end;
+---
+...
+function iterate_in_read_view()
+ local i = create_iterator(space)
+ last_read = i.next()
+ fiber.sleep(100000)
+ last_read = i.next()
+end;
+---
+...
+test_run:cmd("setopt delimiter ''");
+---
+- true
+...
+f1 = fiber.create(fill_space)
+---
+...
+-- Prepared transaction is blocked due to wal_delay.
+-- Start iterator with vlsn = INT64_MAX
+f2 = fiber.create(iterate_in_read_view)
+---
+...
+last_read
+---
+- [1]
+...
+-- Finish prepared transaction and send to read view the iterator.
+errinj.set("ERRINJ_WAL_DELAY", false)
+---
+- ok
+...
+while f1:status() ~= 'dead' do fiber.sleep(0.01) end
+---
+...
+f2:wakeup()
+---
+...
+while f2:status() ~= 'dead' do fiber.sleep(0.01) end
+---
+...
+last_read
+---
+- [2]
+...
+space:drop()
+---
+...
diff --git a/test/vinyl/errinj_tx.test.lua b/test/vinyl/errinj_tx.test.lua
new file mode 100644
index 00000000..cf83b326
--- /dev/null
+++ b/test/vinyl/errinj_tx.test.lua
@@ -0,0 +1,194 @@
+test_run = require('test_run').new()
+fiber = require('fiber')
+txn_proxy = require('txn_proxy')
+create_iterator = require('utils').create_iterator
+errinj = box.error.injection
+
+--
+-- gh-1681: vinyl: crash in vy_rollback on ER_WAL_WRITE
+--
+s = box.schema.space.create('test', {engine='vinyl'})
+_ = s:create_index('pk')
+function f() box.begin() s:insert{1, 'hi'} s:insert{2, 'bye'} box.commit() end
+errinj.set("ERRINJ_WAL_WRITE", true)
+f()
+s:select{}
+errinj.set("ERRINJ_WAL_WRITE", false)
+f()
+s:select{}
+s:drop()
+
+--https://github.com/tarantool/tarantool/issues/1842
+--test error injection
+s = box.schema.space.create('test', {engine='vinyl'})
+_ = s:create_index('pk')
+s:replace{0, 0}
+
+s:replace{1, 0}
+s:replace{2, 0}
+errinj.set("ERRINJ_WAL_WRITE", true)
+s:replace{3, 0}
+s:replace{4, 0}
+s:replace{5, 0}
+s:replace{6, 0}
+errinj.set("ERRINJ_WAL_WRITE", false)
+s:replace{7, 0}
+s:replace{8, 0}
+s:select{}
+
+s:drop()
+
+--iterator test
+test_run:cmd("setopt delimiter ';'")
+
+fiber_status = 0
+
+function fiber_func()
+ box.begin()
+ s:replace{5, 5}
+ fiber_status = 1
+ local res = {pcall(box.commit) }
+ fiber_status = 2
+ return unpack(res)
+end;
+
+test_run:cmd("setopt delimiter ''");
+
+s = box.schema.space.create('test', {engine='vinyl'})
+_ = s:create_index('pk')
+
+_ = s:replace{0, 0}
+_ = s:replace{10, 0}
+_ = s:replace{20, 0}
+
+test_run:cmd("setopt delimiter ';'");
+
+faced_trash = false
+for i = 1,100 do
+ errinj.set("ERRINJ_WAL_WRITE", true)
+ local f = fiber.create(fiber_func)
+ local itr = create_iterator(s, {0}, {iterator='GE'})
+ local first = itr.next()
+ local second = itr.next()
+ if (second[1] ~= 5 and second[1] ~= 10) then faced_trash = true end
+ while fiber_status <= 1 do fiber.sleep(0.001) end
+ local _,next = pcall(itr.next)
+ _,next = pcall(itr.next)
+ _,next = pcall(itr.next)
+ errinj.set("ERRINJ_WAL_WRITE", false)
+ s:delete{5}
+end;
+
+test_run:cmd("setopt delimiter ''");
+
+faced_trash
+
+s:drop()
+
+-- TX in prepared but not committed state
+s = box.schema.space.create('test', {engine='vinyl'})
+_ = s:create_index('pk')
+
+s:replace{1, "original"}
+s:replace{2, "original"}
+s:replace{3, "original"}
+
+c0 = txn_proxy.new()
+c0:begin()
+c1 = txn_proxy.new()
+c1:begin()
+c2 = txn_proxy.new()
+c2:begin()
+c3 = txn_proxy.new()
+c3:begin()
+
+--
+-- Prepared transactions
+--
+
+-- Pause WAL writer to cause all further calls to box.commit() to move
+-- transactions into prepared, but not committed yet state.
+errinj.set("ERRINJ_WAL_DELAY", true)
+lsn = box.info.lsn
+c0('s:replace{1, "c0"}')
+c0('s:replace{2, "c0"}')
+c0('s:replace{3, "c0"}')
+_ = fiber.create(c0.commit, c0)
+box.info.lsn == lsn
+c1('s:replace{1, "c1"}')
+c1('s:replace{2, "c1"}')
+_ = fiber.create(c1.commit, c1)
+box.info.lsn == lsn
+c3('s:select{1}') -- c1 is visible
+c2('s:replace{1, "c2"}')
+c2('s:replace{3, "c2"}')
+_ = fiber.create(c2.commit, c2)
+box.info.lsn == lsn
+c3('s:select{1}') -- c1 is visible, c2 is not
+c3('s:select{2}') -- c1 is visible
+c3('s:select{3}') -- c2 is not visible
+
+-- Resume WAL writer and wait until all transactions will been committed
+errinj.set("ERRINJ_WAL_DELAY", false)
+REQ_COUNT = 7
+while box.info.lsn - lsn < REQ_COUNT do fiber.sleep(0.01) end
+box.info.lsn == lsn + REQ_COUNT
+
+c3('s:select{1}') -- c1 is visible, c2 is not
+c3('s:select{2}') -- c1 is visible
+c3('s:select{3}') -- c2 is not visible
+c3:commit()
+
+s:drop()
+
+--
+-- Test mem restoration on a prepared and not commited statement
+-- after moving iterator into read view.
+--
+space = box.schema.space.create('test', {engine = 'vinyl'})
+pk = space:create_index('pk')
+space:replace{1}
+space:replace{2}
+space:replace{3}
+
+last_read = nil
+
+errinj.set("ERRINJ_WAL_DELAY", true)
+
+test_run:cmd("setopt delimiter ';'")
+
+function fill_space()
+ box.begin()
+ space:replace{1}
+ space:replace{2}
+ space:replace{3}
+-- block until wal_delay = false
+ box.commit()
+-- send iterator to read view
+ space:replace{1, 1}
+-- flush mem and update index version to trigger iterator restore
+ box.snapshot()
+end;
+
+function iterate_in_read_view()
+ local i = create_iterator(space)
+ last_read = i.next()
+ fiber.sleep(100000)
+ last_read = i.next()
+end;
+
+test_run:cmd("setopt delimiter ''");
+
+f1 = fiber.create(fill_space)
+-- Prepared transaction is blocked due to wal_delay.
+-- Start iterator with vlsn = INT64_MAX
+f2 = fiber.create(iterate_in_read_view)
+last_read
+-- Finish prepared transaction and send to read view the iterator.
+errinj.set("ERRINJ_WAL_DELAY", false)
+while f1:status() ~= 'dead' do fiber.sleep(0.01) end
+f2:wakeup()
+while f2:status() ~= 'dead' do fiber.sleep(0.01) end
+last_read
+
+space:drop()
diff --git a/test/vinyl/errinj_vylog.result b/test/vinyl/errinj_vylog.result
index 9ced03df..0e3b79c4 100644
--- a/test/vinyl/errinj_vylog.result
+++ b/test/vinyl/errinj_vylog.result
@@ -5,6 +5,41 @@ fiber = require('fiber')
---
...
--
+-- Check that DDL operations are logged in vylog only after successful
+-- WAL write.
+--
+-- If we logged an index creation in the metadata log before WAL write,
+-- WAL failure would result in leaving the index record in vylog forever.
+-- Since we use LSN to identify indexes in vylog, retrying index creation
+-- would then lead to a duplicate index id in vylog and hence inability
+-- to make a snapshot or recover.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+box.error.injection.set('ERRINJ_WAL_IO', true)
+---
+- ok
+...
+_ = s:create_index('pk')
+---
+- error: Failed to write to disk
+...
+box.error.injection.set('ERRINJ_WAL_IO', false)
+---
+- ok
+...
+_ = s:create_index('pk')
+---
+...
+box.snapshot()
+---
+- ok
+...
+s:drop()
+---
+...
+--
-- Check that an error to commit a new run to vylog does not
-- break vinyl permanently.
--
diff --git a/test/vinyl/errinj_vylog.test.lua b/test/vinyl/errinj_vylog.test.lua
index c1fd517d..ce9e12e5 100644
--- a/test/vinyl/errinj_vylog.test.lua
+++ b/test/vinyl/errinj_vylog.test.lua
@@ -2,6 +2,24 @@ test_run = require('test_run').new()
fiber = require('fiber')
--
+-- Check that DDL operations are logged in vylog only after successful
+-- WAL write.
+--
+-- If we logged an index creation in the metadata log before WAL write,
+-- WAL failure would result in leaving the index record in vylog forever.
+-- Since we use LSN to identify indexes in vylog, retrying index creation
+-- would then lead to a duplicate index id in vylog and hence inability
+-- to make a snapshot or recover.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+box.error.injection.set('ERRINJ_WAL_IO', true)
+_ = s:create_index('pk')
+box.error.injection.set('ERRINJ_WAL_IO', false)
+_ = s:create_index('pk')
+box.snapshot()
+s:drop()
+
+--
-- Check that an error to commit a new run to vylog does not
-- break vinyl permanently.
--
diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini
index ace53e7f..d2a194d8 100644
--- a/test/vinyl/suite.ini
+++ b/test/vinyl/suite.ini
@@ -2,7 +2,7 @@
core = tarantool
description = vinyl integration tests
script = vinyl.lua
-release_disabled = errinj.test.lua errinj_gc.test.lua errinj_vylog.test.lua partial_dump.test.lua quota_timeout.test.lua recovery_quota.test.lua replica_rejoin.test.lua
+release_disabled = errinj.test.lua errinj_ddl.test.lua errinj_gc.test.lua errinj_stat.test.lua errinj_tx.test.lua errinj_vylog.test.lua partial_dump.test.lua quota_timeout.test.lua recovery_quota.test.lua replica_rejoin.test.lua
config = suite.cfg
lua_libs = suite.lua stress.lua large.lua txn_proxy.lua ../box/lua/utils.lua
use_unix_sockets = True
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 03/12] vinyl: rename dump/compact in/out to input/output
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
2019-01-15 14:17 ` [PATCH 02/12] test: split vinyl/errinj Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 04/12] vinyl: rename compact to compaction Vladimir Davydov
` (10 subsequent siblings)
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
'in' is a reserved keyword in Lua, so using 'in' as a map key was a bad
decision - one has to access it with [] rather than simply with dot.
Let's rename 'in' to 'input' and 'out' to 'output' both in the output
and in the code.
---
src/box/vinyl.c | 32 ++++++++---------
src/box/vy_lsm.c | 24 ++++++-------
src/box/vy_lsm.h | 8 ++---
src/box/vy_scheduler.c | 20 +++++------
src/box/vy_stat.h | 16 ++++-----
test/vinyl/recovery_quota.result | 4 +--
test/vinyl/recovery_quota.test.lua | 4 +--
test/vinyl/stat.result | 68 ++++++++++++++++++-------------------
test/vinyl/stat.test.lua | 8 ++---
test/vinyl/update_optimize.result | 4 +--
test/vinyl/update_optimize.test.lua | 4 +--
test/vinyl/write_iterator.result | 4 +--
test/vinyl/write_iterator.test.lua | 4 +--
13 files changed, 100 insertions(+), 100 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index ca987134..f6139898 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -303,13 +303,13 @@ vy_info_append_disk(struct vy_env *env, struct info_handler *h)
info_append_int(h, "index", stat->index);
info_table_begin(h, "dump");
- info_append_int(h, "in", stat->dump.in);
- info_append_int(h, "out", stat->dump.out);
+ info_append_int(h, "input", stat->dump.input);
+ info_append_int(h, "output", stat->dump.output);
info_table_end(h); /* dump */
info_table_begin(h, "compact");
- info_append_int(h, "in", stat->compact.in);
- info_append_int(h, "out", stat->compact.out);
+ info_append_int(h, "input", stat->compact.input);
+ info_append_int(h, "output", stat->compact.output);
info_append_int(h, "queue", stat->compact.queue);
info_table_end(h); /* compact */
@@ -414,13 +414,13 @@ vinyl_index_stat(struct index *index, struct info_handler *h)
info_table_end(h); /* iterator */
info_table_begin(h, "dump");
info_append_int(h, "count", stat->disk.dump.count);
- vy_info_append_stmt_counter(h, "in", &stat->disk.dump.in);
- vy_info_append_disk_stmt_counter(h, "out", &stat->disk.dump.out);
+ vy_info_append_stmt_counter(h, "input", &stat->disk.dump.input);
+ vy_info_append_disk_stmt_counter(h, "output", &stat->disk.dump.output);
info_table_end(h); /* dump */
info_table_begin(h, "compact");
info_append_int(h, "count", stat->disk.compact.count);
- vy_info_append_disk_stmt_counter(h, "in", &stat->disk.compact.in);
- vy_info_append_disk_stmt_counter(h, "out", &stat->disk.compact.out);
+ vy_info_append_disk_stmt_counter(h, "input", &stat->disk.compact.input);
+ vy_info_append_disk_stmt_counter(h, "output", &stat->disk.compact.output);
vy_info_append_disk_stmt_counter(h, "queue", &stat->disk.compact.queue);
info_table_end(h); /* compact */
info_append_int(h, "index_size", lsm->page_index_size);
@@ -476,13 +476,13 @@ vinyl_index_reset_stat(struct index *index)
/* Dump */
stat->disk.dump.count = 0;
- vy_stmt_counter_reset(&stat->disk.dump.in);
- vy_disk_stmt_counter_reset(&stat->disk.dump.out);
+ vy_stmt_counter_reset(&stat->disk.dump.input);
+ vy_disk_stmt_counter_reset(&stat->disk.dump.output);
/* Compaction */
stat->disk.compact.count = 0;
- vy_disk_stmt_counter_reset(&stat->disk.compact.in);
- vy_disk_stmt_counter_reset(&stat->disk.compact.out);
+ vy_disk_stmt_counter_reset(&stat->disk.compact.input);
+ vy_disk_stmt_counter_reset(&stat->disk.compact.output);
/* Cache */
cache_stat->lookup = 0;
@@ -515,10 +515,10 @@ vinyl_engine_reset_stat(struct engine *engine)
memset(&xm->stat, 0, sizeof(xm->stat));
struct vy_disk_stat *disk_stat = &env->lsm_env.disk_stat;
- disk_stat->dump.in = 0;
- disk_stat->dump.out = 0;
- disk_stat->compact.in = 0;
- disk_stat->compact.out = 0;
+ disk_stat->dump.input = 0;
+ disk_stat->dump.output = 0;
+ disk_stat->compact.input = 0;
+ disk_stat->compact.output = 0;
}
/** }}} Introspection */
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index 681b165b..69319174 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -765,28 +765,28 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
void
vy_lsm_acct_dump(struct vy_lsm *lsm,
- const struct vy_stmt_counter *in,
- const struct vy_disk_stmt_counter *out)
+ const struct vy_stmt_counter *input,
+ const struct vy_disk_stmt_counter *output)
{
lsm->stat.disk.dump.count++;
- vy_stmt_counter_add(&lsm->stat.disk.dump.in, in);
- vy_disk_stmt_counter_add(&lsm->stat.disk.dump.out, out);
+ vy_stmt_counter_add(&lsm->stat.disk.dump.input, input);
+ vy_disk_stmt_counter_add(&lsm->stat.disk.dump.output, output);
- lsm->env->disk_stat.dump.in += in->bytes;
- lsm->env->disk_stat.dump.out += out->bytes;
+ lsm->env->disk_stat.dump.input += input->bytes;
+ lsm->env->disk_stat.dump.output += output->bytes;
}
void
vy_lsm_acct_compaction(struct vy_lsm *lsm,
- const struct vy_disk_stmt_counter *in,
- const struct vy_disk_stmt_counter *out)
+ const struct vy_disk_stmt_counter *input,
+ const struct vy_disk_stmt_counter *output)
{
lsm->stat.disk.compact.count++;
- vy_disk_stmt_counter_add(&lsm->stat.disk.compact.in, in);
- vy_disk_stmt_counter_add(&lsm->stat.disk.compact.out, out);
+ vy_disk_stmt_counter_add(&lsm->stat.disk.compact.input, input);
+ vy_disk_stmt_counter_add(&lsm->stat.disk.compact.output, output);
- lsm->env->disk_stat.compact.in += in->bytes;
- lsm->env->disk_stat.compact.out += out->bytes;
+ lsm->env->disk_stat.compact.input += input->bytes;
+ lsm->env->disk_stat.compact.output += output->bytes;
}
int
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index d8d9f9d0..07dc1b5c 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -455,16 +455,16 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range);
*/
void
vy_lsm_acct_dump(struct vy_lsm *lsm,
- const struct vy_stmt_counter *in,
- const struct vy_disk_stmt_counter *out);
+ const struct vy_stmt_counter *input,
+ const struct vy_disk_stmt_counter *output);
/**
* Account compaction in LSM tree statistics.
*/
void
vy_lsm_acct_compaction(struct vy_lsm *lsm,
- const struct vy_disk_stmt_counter *in,
- const struct vy_disk_stmt_counter *out);
+ const struct vy_disk_stmt_counter *input,
+ const struct vy_disk_stmt_counter *output);
/**
* Allocate a new active in-memory index for an LSM tree while
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index 9221b973..72d2094e 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -1091,8 +1091,8 @@ vy_task_dump_complete(struct vy_task *task)
struct vy_lsm *lsm = task->lsm;
struct vy_run *new_run = task->new_run;
int64_t dump_lsn = new_run->dump_lsn;
- struct vy_disk_stmt_counter dump_out = new_run->count;
- struct vy_stmt_counter dump_in;
+ struct vy_disk_stmt_counter dump_output = new_run->count;
+ struct vy_stmt_counter dump_input;
struct tuple_format *key_format = lsm->env->key_format;
struct vy_mem *mem, *next_mem;
struct vy_slice **new_slices, *slice;
@@ -1214,15 +1214,15 @@ delete_mems:
* Delete dumped in-memory trees and account dump in
* LSM tree statistics.
*/
- vy_stmt_counter_reset(&dump_in);
+ vy_stmt_counter_reset(&dump_input);
rlist_foreach_entry_safe(mem, &lsm->sealed, in_sealed, next_mem) {
if (mem->generation > scheduler->dump_generation)
continue;
- vy_stmt_counter_add(&dump_in, &mem->count);
+ vy_stmt_counter_add(&dump_input, &mem->count);
vy_lsm_delete_mem(lsm, mem);
}
lsm->dump_lsn = MAX(lsm->dump_lsn, dump_lsn);
- vy_lsm_acct_dump(lsm, &dump_in, &dump_out);
+ vy_lsm_acct_dump(lsm, &dump_input, &dump_output);
/* The iterator has been cleaned up in a worker thread. */
task->wi->iface->close(task->wi);
@@ -1444,8 +1444,8 @@ vy_task_compact_complete(struct vy_task *task)
struct vy_lsm *lsm = task->lsm;
struct vy_range *range = task->range;
struct vy_run *new_run = task->new_run;
- struct vy_disk_stmt_counter compact_out = new_run->count;
- struct vy_disk_stmt_counter compact_in;
+ struct vy_disk_stmt_counter compact_output = new_run->count;
+ struct vy_disk_stmt_counter compact_input;
struct vy_slice *first_slice = task->first_slice;
struct vy_slice *last_slice = task->last_slice;
struct vy_slice *slice, *next_slice, *new_slice = NULL;
@@ -1547,12 +1547,12 @@ vy_task_compact_complete(struct vy_task *task)
vy_lsm_unacct_range(lsm, range);
if (new_slice != NULL)
vy_range_add_slice_before(range, new_slice, first_slice);
- vy_disk_stmt_counter_reset(&compact_in);
+ vy_disk_stmt_counter_reset(&compact_input);
for (slice = first_slice; ; slice = next_slice) {
next_slice = rlist_next_entry(slice, in_range);
vy_range_remove_slice(range, slice);
rlist_add_entry(&compacted_slices, slice, in_range);
- vy_disk_stmt_counter_add(&compact_in, &slice->count);
+ vy_disk_stmt_counter_add(&compact_input, &slice->count);
if (slice == last_slice)
break;
}
@@ -1560,7 +1560,7 @@ vy_task_compact_complete(struct vy_task *task)
range->version++;
vy_range_update_compact_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
- vy_lsm_acct_compaction(lsm, &compact_in, &compact_out);
+ vy_lsm_acct_compaction(lsm, &compact_input, &compact_output);
/*
* Unaccount unused runs and delete compacted slices.
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index 9bb09174..ae032d1d 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -148,18 +148,18 @@ struct vy_lsm_stat {
/* Number of completed tasks. */
int32_t count;
/** Number of input statements. */
- struct vy_stmt_counter in;
+ struct vy_stmt_counter input;
/** Number of output statements. */
- struct vy_disk_stmt_counter out;
+ struct vy_disk_stmt_counter output;
} dump;
/** Compaction statistics. */
struct {
/* Number of completed tasks. */
int32_t count;
/** Number of input statements. */
- struct vy_disk_stmt_counter in;
+ struct vy_disk_stmt_counter input;
/** Number of output statements. */
- struct vy_disk_stmt_counter out;
+ struct vy_disk_stmt_counter output;
/** Number of statements awaiting compaction. */
struct vy_disk_stmt_counter queue;
} compact;
@@ -215,12 +215,12 @@ struct vy_disk_stat {
int64_t data;
int64_t index;
struct {
- int64_t in;
- int64_t out;
+ int64_t input;
+ int64_t output;
} dump;
struct {
- int64_t in;
- int64_t out;
+ int64_t input;
+ int64_t output;
int64_t queue;
} compact;
};
diff --git a/test/vinyl/recovery_quota.result b/test/vinyl/recovery_quota.result
index 151f280b..002aca94 100644
--- a/test/vinyl/recovery_quota.result
+++ b/test/vinyl/recovery_quota.result
@@ -52,7 +52,7 @@ stat = box.space.test.index.pk:stat()
_ = var:insert{'put', stat.put.rows}
---
...
-_ = var:insert{'dump', stat.disk.dump.out.rows}
+_ = var:insert{'dump', stat.disk.dump.output.rows}
---
...
test_run:cmd('restart server test with args="2097152"')
@@ -74,7 +74,7 @@ var = box.space.var
dump_before = var:get('dump')[2]
---
...
-dump_after = stat.disk.dump.out.rows
+dump_after = stat.disk.dump.output.rows
---
...
put_before = var:get('put')[2]
diff --git a/test/vinyl/recovery_quota.test.lua b/test/vinyl/recovery_quota.test.lua
index 1e347667..b34e4d65 100644
--- a/test/vinyl/recovery_quota.test.lua
+++ b/test/vinyl/recovery_quota.test.lua
@@ -23,7 +23,7 @@ var = box.schema.space.create('var')
_ = var:create_index('pk', {parts = {1, 'string'}})
stat = box.space.test.index.pk:stat()
_ = var:insert{'put', stat.put.rows}
-_ = var:insert{'dump', stat.disk.dump.out.rows}
+_ = var:insert{'dump', stat.disk.dump.output.rows}
test_run:cmd('restart server test with args="2097152"')
-- Check that we do not exceed quota.
stat = box.stat.vinyl()
@@ -32,7 +32,7 @@ stat.memory.level0 <= box.cfg.vinyl_memory or {stat.memory.level0, box.cfg.vinyl
stat = box.space.test.index.pk:stat()
var = box.space.var
dump_before = var:get('dump')[2]
-dump_after = stat.disk.dump.out.rows
+dump_after = stat.disk.dump.output.rows
put_before = var:get('put')[2]
put_after = stat.put.rows
dump_after == 0 or dump_after
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index c4a0e632..315a5853 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -157,32 +157,32 @@ istat()
upserts: 0
deletes: 0
dump:
- in:
+ input:
rows: 0
bytes: 0
count: 0
- out:
+ output:
bytes_compressed: 0
pages: 0
rows: 0
bytes: 0
compact:
- in:
+ input:
bytes_compressed: 0
pages: 0
rows: 0
bytes: 0
count: 0
+ output:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
queue:
bytes_compressed: 0
pages: 0
rows: 0
bytes: 0
- out:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
iterator:
read:
bytes_compressed: 0
@@ -226,12 +226,12 @@ gstat()
---
- disk:
dump:
- in: 0
- out: 0
+ input: 0
+ output: 0
compact:
- in: 0
+ input: 0
+ output: 0
queue: 0
- out: 0
data: 0
index: 0
memory:
@@ -280,11 +280,11 @@ stat_diff(istat(), st)
run_count: 1
disk:
dump:
- in:
+ input:
rows: 25
bytes: 26525
count: 1
- out:
+ output:
bytes: 26049
pages: 7
bytes_compressed: <bytes_compressed>
@@ -320,11 +320,11 @@ stat_diff(istat(), st)
---
- disk:
dump:
- in:
+ input:
rows: 50
bytes: 53050
count: 1
- out:
+ output:
bytes: 52091
pages: 13
bytes_compressed: <bytes_compressed>
@@ -337,13 +337,13 @@ stat_diff(istat(), st)
statement:
replaces: 25
compact:
- in:
+ input:
bytes: 78140
pages: 20
bytes_compressed: <bytes_compressed>
rows: 75
count: 1
- out:
+ output:
bytes: 52091
pages: 13
bytes_compressed: <bytes_compressed>
@@ -701,19 +701,19 @@ box.rollback()
-- Global statistics.
--
-- dump and compaction totals
-gstat().disk.dump['in'] == istat().disk.dump['in'].bytes
+gstat().disk.dump.input == istat().disk.dump.input.bytes
---
- true
...
-gstat().disk.dump['out'] == istat().disk.dump['out'].bytes
+gstat().disk.dump.output == istat().disk.dump.output.bytes
---
- true
...
-gstat().disk.compact['in'] == istat().disk.compact['in'].bytes
+gstat().disk.compact.input == istat().disk.compact.input.bytes
---
- true
...
-gstat().disk.compact['out'] == istat().disk.compact['out'].bytes
+gstat().disk.compact.output == istat().disk.compact.output.bytes
---
- true
...
@@ -1007,32 +1007,32 @@ istat()
upserts: 0
deletes: 0
dump:
- in:
+ input:
rows: 0
bytes: 0
count: 0
- out:
+ output:
bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
compact:
- in:
+ input:
bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
count: 0
+ output:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
queue:
bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
- out:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
iterator:
read:
bytes_compressed: <bytes_compressed>
@@ -1076,12 +1076,12 @@ gstat()
---
- disk:
dump:
- in: 0
- out: 0
+ input: 0
+ output: 0
compact:
- in: 0
+ input: 0
+ output: 0
queue: 0
- out: 0
data: 104300
index: 1190
memory:
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 8b296b9e..56e544d1 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -205,10 +205,10 @@ box.rollback()
--
-- dump and compaction totals
-gstat().disk.dump['in'] == istat().disk.dump['in'].bytes
-gstat().disk.dump['out'] == istat().disk.dump['out'].bytes
-gstat().disk.compact['in'] == istat().disk.compact['in'].bytes
-gstat().disk.compact['out'] == istat().disk.compact['out'].bytes
+gstat().disk.dump.input == istat().disk.dump.input.bytes
+gstat().disk.dump.output == istat().disk.dump.output.bytes
+gstat().disk.compact.input == istat().disk.compact.input.bytes
+gstat().disk.compact.output == istat().disk.compact.output.bytes
-- use memory
st = gstat()
diff --git a/test/vinyl/update_optimize.result b/test/vinyl/update_optimize.result
index bf9dff7e..d8ff9fc4 100644
--- a/test/vinyl/update_optimize.result
+++ b/test/vinyl/update_optimize.result
@@ -16,7 +16,7 @@ index = space:create_index('primary', { run_count_per_level = 20 })
index2 = space:create_index('secondary', { parts = {5, 'unsigned'}, run_count_per_level = 20 })
---
...
-function dumped_stmt_count() return index:stat().disk.dump.out.rows + index2:stat().disk.dump.out.rows end
+function dumped_stmt_count() return index:stat().disk.dump.output.rows + index2:stat().disk.dump.output.rows end
---
...
box.snapshot()
@@ -217,7 +217,7 @@ index2 = space:create_index('secondary', { parts = {4, 'unsigned', 3, 'unsigned'
index3 = space:create_index('third', { parts = {5, 'unsigned'}, run_count_per_level = 20 })
---
...
-function dumped_stmt_count() return index:stat().disk.dump.out.rows + index2:stat().disk.dump.out.rows + index3:stat().disk.dump.out.rows end
+function dumped_stmt_count() return index:stat().disk.dump.output.rows + index2:stat().disk.dump.output.rows + index3:stat().disk.dump.output.rows end
---
...
box.snapshot()
diff --git a/test/vinyl/update_optimize.test.lua b/test/vinyl/update_optimize.test.lua
index 67efba31..41ff964b 100644
--- a/test/vinyl/update_optimize.test.lua
+++ b/test/vinyl/update_optimize.test.lua
@@ -8,7 +8,7 @@ fiber = require('fiber')
space = box.schema.space.create('test', { engine = 'vinyl' })
index = space:create_index('primary', { run_count_per_level = 20 })
index2 = space:create_index('secondary', { parts = {5, 'unsigned'}, run_count_per_level = 20 })
-function dumped_stmt_count() return index:stat().disk.dump.out.rows + index2:stat().disk.dump.out.rows end
+function dumped_stmt_count() return index:stat().disk.dump.output.rows + index2:stat().disk.dump.output.rows end
box.snapshot()
test_run:cmd("setopt delimiter ';'")
function wait_for_dump(index, old_count)
@@ -76,7 +76,7 @@ space = box.schema.space.create('test', { engine = 'vinyl' })
index = space:create_index('primary', { parts = {2, 'unsigned'}, run_count_per_level = 20 } )
index2 = space:create_index('secondary', { parts = {4, 'unsigned', 3, 'unsigned'}, run_count_per_level = 20 })
index3 = space:create_index('third', { parts = {5, 'unsigned'}, run_count_per_level = 20 })
-function dumped_stmt_count() return index:stat().disk.dump.out.rows + index2:stat().disk.dump.out.rows + index3:stat().disk.dump.out.rows end
+function dumped_stmt_count() return index:stat().disk.dump.output.rows + index2:stat().disk.dump.output.rows + index3:stat().disk.dump.output.rows end
box.snapshot()
index_run_count = index:stat().run_count
index2_run_count = index2:stat().run_count
diff --git a/test/vinyl/write_iterator.result b/test/vinyl/write_iterator.result
index cf1e426c..e6bf1b01 100644
--- a/test/vinyl/write_iterator.result
+++ b/test/vinyl/write_iterator.result
@@ -854,11 +854,11 @@ sk:stat().disk.compact.count -- 1
...
-- All INSERT+DELETE pairs should have been annihilated,
-- only padding is left.
-pk:stat().disk.compact.out.rows - PAD2 -- 0
+pk:stat().disk.compact.output.rows - PAD2 -- 0
---
- 0
...
-sk:stat().disk.compact.out.rows - PAD2 -- 0
+sk:stat().disk.compact.output.rows - PAD2 -- 0
---
- 0
...
diff --git a/test/vinyl/write_iterator.test.lua b/test/vinyl/write_iterator.test.lua
index a1de240f..a4ba42aa 100644
--- a/test/vinyl/write_iterator.test.lua
+++ b/test/vinyl/write_iterator.test.lua
@@ -361,8 +361,8 @@ pk:stat().disk.compact.count -- 1
sk:stat().disk.compact.count -- 1
-- All INSERT+DELETE pairs should have been annihilated,
-- only padding is left.
-pk:stat().disk.compact.out.rows - PAD2 -- 0
-sk:stat().disk.compact.out.rows - PAD2 -- 0
+pk:stat().disk.compact.output.rows - PAD2 -- 0
+sk:stat().disk.compact.output.rows - PAD2 -- 0
pk:select(1000, {iterator = 'LE'}) -- empty
sk:select(1000, {iterator = 'LE'}) -- empty
s:drop()
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 04/12] vinyl: rename compact to compaction
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (2 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 03/12] vinyl: rename dump/compact in/out to input/output Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-17 11:34 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 05/12] vinyl: bump range version in vy_range.c Vladimir Davydov
` (9 subsequent siblings)
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
compact_input sounds confusing, because 'compact' works as an adjective
here. Saving 3 characters per variable/stat name related to compaction
doesn't justify this. Let's rename 'compact' to 'compaction' both in
stats and in the code.
---
src/box/vinyl.c | 32 +++----
src/box/vy_lsm.c | 40 ++++-----
src/box/vy_lsm.h | 12 +--
src/box/vy_range.c | 24 +++---
src/box/vy_range.h | 16 ++--
src/box/vy_scheduler.c | 102 +++++++++++-----------
src/box/vy_scheduler.h | 6 +-
src/box/vy_stat.h | 4 +-
test/vinyl/ddl.result | 2 +-
test/vinyl/ddl.test.lua | 2 +-
test/vinyl/deferred_delete.result | 16 ++--
test/vinyl/deferred_delete.test.lua | 16 ++--
test/vinyl/errinj.result | 10 +--
test/vinyl/errinj.test.lua | 10 +--
test/vinyl/errinj_stat.result | 26 +++---
test/vinyl/errinj_stat.test.lua | 26 +++---
test/vinyl/snap_io_rate.result | 2 +-
test/vinyl/snap_io_rate.test.lua | 2 +-
test/vinyl/stat.result | 164 ++++++++++++++++++------------------
test/vinyl/stat.test.lua | 8 +-
test/vinyl/write_iterator.result | 20 ++---
test/vinyl/write_iterator.test.lua | 20 ++---
22 files changed, 282 insertions(+), 278 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index f6139898..8028fd2b 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -307,11 +307,11 @@ vy_info_append_disk(struct vy_env *env, struct info_handler *h)
info_append_int(h, "output", stat->dump.output);
info_table_end(h); /* dump */
- info_table_begin(h, "compact");
- info_append_int(h, "input", stat->compact.input);
- info_append_int(h, "output", stat->compact.output);
- info_append_int(h, "queue", stat->compact.queue);
- info_table_end(h); /* compact */
+ info_table_begin(h, "compaction");
+ info_append_int(h, "input", stat->compaction.input);
+ info_append_int(h, "output", stat->compaction.output);
+ info_append_int(h, "queue", stat->compaction.queue);
+ info_table_end(h); /* compaction */
info_table_end(h); /* disk */
}
@@ -417,12 +417,12 @@ vinyl_index_stat(struct index *index, struct info_handler *h)
vy_info_append_stmt_counter(h, "input", &stat->disk.dump.input);
vy_info_append_disk_stmt_counter(h, "output", &stat->disk.dump.output);
info_table_end(h); /* dump */
- info_table_begin(h, "compact");
- info_append_int(h, "count", stat->disk.compact.count);
- vy_info_append_disk_stmt_counter(h, "input", &stat->disk.compact.input);
- vy_info_append_disk_stmt_counter(h, "output", &stat->disk.compact.output);
- vy_info_append_disk_stmt_counter(h, "queue", &stat->disk.compact.queue);
- info_table_end(h); /* compact */
+ info_table_begin(h, "compaction");
+ info_append_int(h, "count", stat->disk.compaction.count);
+ vy_info_append_disk_stmt_counter(h, "input", &stat->disk.compaction.input);
+ vy_info_append_disk_stmt_counter(h, "output", &stat->disk.compaction.output);
+ vy_info_append_disk_stmt_counter(h, "queue", &stat->disk.compaction.queue);
+ info_table_end(h); /* compaction */
info_append_int(h, "index_size", lsm->page_index_size);
info_append_int(h, "bloom_size", lsm->bloom_size);
info_table_end(h); /* disk */
@@ -480,9 +480,9 @@ vinyl_index_reset_stat(struct index *index)
vy_disk_stmt_counter_reset(&stat->disk.dump.output);
/* Compaction */
- stat->disk.compact.count = 0;
- vy_disk_stmt_counter_reset(&stat->disk.compact.input);
- vy_disk_stmt_counter_reset(&stat->disk.compact.output);
+ stat->disk.compaction.count = 0;
+ vy_disk_stmt_counter_reset(&stat->disk.compaction.input);
+ vy_disk_stmt_counter_reset(&stat->disk.compaction.output);
/* Cache */
cache_stat->lookup = 0;
@@ -517,8 +517,8 @@ vinyl_engine_reset_stat(struct engine *engine)
struct vy_disk_stat *disk_stat = &env->lsm_env.disk_stat;
disk_stat->dump.input = 0;
disk_stat->dump.output = 0;
- disk_stat->compact.input = 0;
- disk_stat->compact.output = 0;
+ disk_stat->compaction.input = 0;
+ disk_stat->compaction.output = 0;
}
/** }}} Introspection */
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index 69319174..abadab5c 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -188,7 +188,7 @@ vy_lsm_new(struct vy_lsm_env *lsm_env, struct vy_cache_env *cache_env,
lsm->mem_format = format;
tuple_format_ref(lsm->mem_format);
lsm->in_dump.pos = UINT32_MAX;
- lsm->in_compact.pos = UINT32_MAX;
+ lsm->in_compaction.pos = UINT32_MAX;
lsm->space_id = index_def->space_id;
lsm->index_id = index_def->iid;
lsm->group_id = group_id;
@@ -234,13 +234,13 @@ vy_lsm_delete(struct vy_lsm *lsm)
{
assert(lsm->refs == 0);
assert(lsm->in_dump.pos == UINT32_MAX);
- assert(lsm->in_compact.pos == UINT32_MAX);
+ assert(lsm->in_compaction.pos == UINT32_MAX);
assert(vy_lsm_read_set_empty(&lsm->read_set));
assert(lsm->env->lsm_count > 0);
lsm->env->lsm_count--;
- lsm->env->disk_stat.compact.queue -=
- lsm->stat.disk.compact.queue.bytes;
+ lsm->env->disk_stat.compaction.queue -=
+ lsm->stat.disk.compaction.queue.bytes;
if (lsm->pk != NULL)
vy_lsm_unref(lsm->pk);
@@ -661,13 +661,13 @@ vy_lsm_generation(struct vy_lsm *lsm)
}
int
-vy_lsm_compact_priority(struct vy_lsm *lsm)
+vy_lsm_compaction_priority(struct vy_lsm *lsm)
{
struct heap_node *n = vy_range_heap_top(&lsm->range_heap);
if (n == NULL)
return 0;
struct vy_range *range = container_of(n, struct vy_range, heap_node);
- return range->compact_priority;
+ return range->compaction_priority;
}
void
@@ -749,18 +749,18 @@ void
vy_lsm_acct_range(struct vy_lsm *lsm, struct vy_range *range)
{
histogram_collect(lsm->run_hist, range->slice_count);
- vy_disk_stmt_counter_add(&lsm->stat.disk.compact.queue,
- &range->compact_queue);
- lsm->env->disk_stat.compact.queue += range->compact_queue.bytes;
+ vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.queue,
+ &range->compaction_queue);
+ lsm->env->disk_stat.compaction.queue += range->compaction_queue.bytes;
}
void
vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
{
histogram_discard(lsm->run_hist, range->slice_count);
- vy_disk_stmt_counter_sub(&lsm->stat.disk.compact.queue,
- &range->compact_queue);
- lsm->env->disk_stat.compact.queue -= range->compact_queue.bytes;
+ vy_disk_stmt_counter_sub(&lsm->stat.disk.compaction.queue,
+ &range->compaction_queue);
+ lsm->env->disk_stat.compaction.queue -= range->compaction_queue.bytes;
}
void
@@ -781,12 +781,12 @@ vy_lsm_acct_compaction(struct vy_lsm *lsm,
const struct vy_disk_stmt_counter *input,
const struct vy_disk_stmt_counter *output)
{
- lsm->stat.disk.compact.count++;
- vy_disk_stmt_counter_add(&lsm->stat.disk.compact.input, input);
- vy_disk_stmt_counter_add(&lsm->stat.disk.compact.output, output);
+ lsm->stat.disk.compaction.count++;
+ vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.input, input);
+ vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.output, output);
- lsm->env->disk_stat.compact.input += input->bytes;
- lsm->env->disk_stat.compact.output += output->bytes;
+ lsm->env->disk_stat.compaction.input += input->bytes;
+ lsm->env->disk_stat.compaction.output += output->bytes;
}
int
@@ -1061,7 +1061,7 @@ vy_lsm_split_range(struct vy_lsm *lsm, struct vy_range *range)
vy_range_add_slice(part, new_slice);
}
part->needs_compaction = range->needs_compaction;
- vy_range_update_compact_priority(part, &lsm->opts);
+ vy_range_update_compaction_priority(part, &lsm->opts);
}
/*
@@ -1178,7 +1178,7 @@ vy_lsm_coalesce_range(struct vy_lsm *lsm, struct vy_range *range)
* we don't need to compact the resulting range as long
* as it fits the configured LSM tree shape.
*/
- vy_range_update_compact_priority(result, &lsm->opts);
+ vy_range_update_compaction_priority(result, &lsm->opts);
vy_lsm_acct_range(lsm, result);
vy_lsm_add_range(lsm, result);
lsm->range_tree_version++;
@@ -1206,7 +1206,7 @@ vy_lsm_force_compaction(struct vy_lsm *lsm)
while ((range = vy_range_tree_inext(&it)) != NULL) {
vy_lsm_unacct_range(lsm, range);
range->needs_compaction = true;
- vy_range_update_compact_priority(range, &lsm->opts);
+ vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
}
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index 07dc1b5c..e7487995 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -221,7 +221,7 @@ struct vy_lsm {
vy_range_tree_t *tree;
/** Number of ranges in this LSM tree. */
int range_count;
- /** Heap of ranges, prioritized by compact_priority. */
+ /** Heap of ranges, prioritized by compaction_priority. */
heap_t range_heap;
/**
* List of all runs created for this LSM tree,
@@ -276,8 +276,8 @@ struct vy_lsm {
bool is_dumping;
/** Link in vy_scheduler->dump_heap. */
struct heap_node in_dump;
- /** Link in vy_scheduler->compact_heap. */
- struct heap_node in_compact;
+ /** Link in vy_scheduler->compaction_heap. */
+ struct heap_node in_compaction;
/**
* Interval tree containing reads from this LSM tree done by
* all active transactions. Linked by vy_tx_interval->in_lsm.
@@ -404,9 +404,9 @@ vy_lsm_recover(struct vy_lsm *lsm, struct vy_recovery *recovery,
int64_t
vy_lsm_generation(struct vy_lsm *lsm);
-/** Return max compact_priority among ranges of an LSM tree. */
+/** Return max compaction_priority among ranges of an LSM tree. */
int
-vy_lsm_compact_priority(struct vy_lsm *lsm);
+vy_lsm_compaction_priority(struct vy_lsm *lsm);
/** Add a run to the list of runs of an LSM tree. */
void
@@ -436,7 +436,7 @@ vy_lsm_remove_range(struct vy_lsm *lsm, struct vy_range *range);
* This function updates the following LSM tree statistics:
* - vy_lsm::run_hist after a slice is added to or removed from
* a range of the LSM tree.
- * - vy_lsm::stat::disk::compact::queue after compaction priority
+ * - vy_lsm::stat::disk::compaction::queue after compaction priority
* of a range is updated.
*/
void
diff --git a/src/box/vy_range.c b/src/box/vy_range.c
index 4e24e2a3..f489c95d 100644
--- a/src/box/vy_range.c
+++ b/src/box/vy_range.c
@@ -282,18 +282,18 @@ vy_range_remove_slice(struct vy_range *range, struct vy_slice *slice)
* ratio.
*
* Given a range, this function computes the maximal level that needs
- * to be compacted and sets @compact_priority to the number of runs in
- * this level and all preceding levels.
+ * to be compacted and sets @compaction_priority to the number of runs
+ * in this level and all preceding levels.
*/
void
-vy_range_update_compact_priority(struct vy_range *range,
- const struct index_opts *opts)
+vy_range_update_compaction_priority(struct vy_range *range,
+ const struct index_opts *opts)
{
assert(opts->run_count_per_level > 0);
assert(opts->run_size_ratio > 1);
- range->compact_priority = 0;
- vy_disk_stmt_counter_reset(&range->compact_queue);
+ range->compaction_priority = 0;
+ vy_disk_stmt_counter_reset(&range->compaction_queue);
if (range->slice_count <= 1) {
/* Nothing to compact. */
@@ -302,8 +302,8 @@ vy_range_update_compact_priority(struct vy_range *range,
}
if (range->needs_compaction) {
- range->compact_priority = range->slice_count;
- range->compact_queue = range->count;
+ range->compaction_priority = range->slice_count;
+ range->compaction_queue = range->count;
return;
}
@@ -372,8 +372,8 @@ vy_range_update_compact_priority(struct vy_range *range,
* for compaction. We compact all runs at
* this level and upper levels.
*/
- range->compact_priority = total_run_count;
- range->compact_queue = total_stmt_count;
+ range->compaction_priority = total_run_count;
+ range->compaction_queue = total_stmt_count;
est_new_run_size = total_stmt_count.bytes_compressed;
}
}
@@ -383,8 +383,8 @@ vy_range_update_compact_priority(struct vy_range *range,
* Do not store more than one run at the last level
* to keep space amplification low.
*/
- range->compact_priority = total_run_count;
- range->compact_queue = total_stmt_count;
+ range->compaction_priority = total_run_count;
+ range->compaction_queue = total_stmt_count;
}
}
diff --git a/src/box/vy_range.h b/src/box/vy_range.h
index 0830479d..05195d08 100644
--- a/src/box/vy_range.h
+++ b/src/box/vy_range.h
@@ -103,15 +103,15 @@ struct vy_range {
* The lower the level is scheduled for compaction,
* the bigger it tends to be because upper levels are
* taken in.
- * @sa vy_range_update_compact_priority() to see
+ * @sa vy_range_update_compaction_priority() to see
* how we decide how many runs to compact next time.
*/
- int compact_priority;
+ int compaction_priority;
/** Number of statements that need to be compacted. */
- struct vy_disk_stmt_counter compact_queue;
+ struct vy_disk_stmt_counter compaction_queue;
/**
* If this flag is set, the range must be scheduled for
- * major compaction, i.e. its compact_priority must be
+ * major compaction, i.e. its compaction_priority must be
* raised to max (slice_count). The flag is set by
* vy_lsm_force_compaction() and cleared when the range
* is scheduled for compaction.
@@ -132,7 +132,7 @@ struct vy_range {
/**
* Heap of all ranges of the same LSM tree, prioritized by
- * vy_range->compact_priority.
+ * vy_range->compaction_priority.
*/
#define HEAP_NAME vy_range_heap
static inline bool
@@ -140,7 +140,7 @@ vy_range_heap_less(struct heap_node *a, struct heap_node *b)
{
struct vy_range *r1 = container_of(a, struct vy_range, heap_node);
struct vy_range *r2 = container_of(b, struct vy_range, heap_node);
- return r1->compact_priority > r2->compact_priority;
+ return r1->compaction_priority > r2->compaction_priority;
}
#define HEAP_LESS(h, l, r) vy_range_heap_less(l, r)
#include "salad/heap.h"
@@ -239,8 +239,8 @@ vy_range_remove_slice(struct vy_range *range, struct vy_slice *slice);
* @param opts Index options.
*/
void
-vy_range_update_compact_priority(struct vy_range *range,
- const struct index_opts *opts);
+vy_range_update_compaction_priority(struct vy_range *range,
+ const struct index_opts *opts);
/**
* Check if a range needs to be split in two.
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index 72d2094e..ba20ef53 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -315,19 +315,19 @@ vy_dump_heap_less(struct heap_node *a, struct heap_node *b)
#undef HEAP_NAME
static bool
-vy_compact_heap_less(struct heap_node *a, struct heap_node *b)
+vy_compaction_heap_less(struct heap_node *a, struct heap_node *b)
{
- struct vy_lsm *i1 = container_of(a, struct vy_lsm, in_compact);
- struct vy_lsm *i2 = container_of(b, struct vy_lsm, in_compact);
+ struct vy_lsm *i1 = container_of(a, struct vy_lsm, in_compaction);
+ struct vy_lsm *i2 = container_of(b, struct vy_lsm, in_compaction);
/*
* Prefer LSM trees whose read amplification will be reduced
* most as a result of compaction.
*/
- return vy_lsm_compact_priority(i1) > vy_lsm_compact_priority(i2);
+ return vy_lsm_compaction_priority(i1) > vy_lsm_compaction_priority(i2);
}
-#define HEAP_NAME vy_compact_heap
-#define HEAP_LESS(h, l, r) vy_compact_heap_less(l, r)
+#define HEAP_NAME vy_compaction_heap
+#define HEAP_LESS(h, l, r) vy_compaction_heap_less(l, r)
#include "salad/heap.h"
@@ -459,16 +459,16 @@ vy_scheduler_create(struct vy_scheduler *scheduler, int write_threads,
*/
assert(write_threads > 1);
int dump_threads = MAX(1, write_threads / 4);
- int compact_threads = write_threads - dump_threads;
+ int compaction_threads = write_threads - dump_threads;
vy_worker_pool_create(&scheduler->dump_pool,
"dump", dump_threads);
- vy_worker_pool_create(&scheduler->compact_pool,
- "compact", compact_threads);
+ vy_worker_pool_create(&scheduler->compaction_pool,
+ "compaction", compaction_threads);
stailq_create(&scheduler->processed_tasks);
vy_dump_heap_create(&scheduler->dump_heap);
- vy_compact_heap_create(&scheduler->compact_heap);
+ vy_compaction_heap_create(&scheduler->compaction_heap);
diag_create(&scheduler->diag);
fiber_cond_create(&scheduler->dump_cond);
@@ -490,12 +490,12 @@ vy_scheduler_destroy(struct vy_scheduler *scheduler)
fiber_cond_signal(&scheduler->scheduler_cond);
vy_worker_pool_destroy(&scheduler->dump_pool);
- vy_worker_pool_destroy(&scheduler->compact_pool);
+ vy_worker_pool_destroy(&scheduler->compaction_pool);
diag_destroy(&scheduler->diag);
fiber_cond_destroy(&scheduler->dump_cond);
fiber_cond_destroy(&scheduler->scheduler_cond);
vy_dump_heap_destroy(&scheduler->dump_heap);
- vy_compact_heap_destroy(&scheduler->compact_heap);
+ vy_compaction_heap_destroy(&scheduler->compaction_heap);
TRASH(scheduler);
}
@@ -504,20 +504,22 @@ void
vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
assert(lsm->in_dump.pos == UINT32_MAX);
- assert(lsm->in_compact.pos == UINT32_MAX);
+ assert(lsm->in_compaction.pos == UINT32_MAX);
vy_dump_heap_insert(&scheduler->dump_heap, &lsm->in_dump);
- vy_compact_heap_insert(&scheduler->compact_heap, &lsm->in_compact);
+ vy_compaction_heap_insert(&scheduler->compaction_heap,
+ &lsm->in_compaction);
}
void
vy_scheduler_remove_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
assert(lsm->in_dump.pos != UINT32_MAX);
- assert(lsm->in_compact.pos != UINT32_MAX);
+ assert(lsm->in_compaction.pos != UINT32_MAX);
vy_dump_heap_delete(&scheduler->dump_heap, &lsm->in_dump);
- vy_compact_heap_delete(&scheduler->compact_heap, &lsm->in_compact);
+ vy_compaction_heap_delete(&scheduler->compaction_heap,
+ &lsm->in_compaction);
lsm->in_dump.pos = UINT32_MAX;
- lsm->in_compact.pos = UINT32_MAX;
+ lsm->in_compaction.pos = UINT32_MAX;
}
static void
@@ -526,13 +528,14 @@ vy_scheduler_update_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
if (lsm->is_dropped) {
/* Dropped LSM trees are exempted from scheduling. */
assert(lsm->in_dump.pos == UINT32_MAX);
- assert(lsm->in_compact.pos == UINT32_MAX);
+ assert(lsm->in_compaction.pos == UINT32_MAX);
return;
}
assert(lsm->in_dump.pos != UINT32_MAX);
- assert(lsm->in_compact.pos != UINT32_MAX);
+ assert(lsm->in_compaction.pos != UINT32_MAX);
vy_dump_heap_update(&scheduler->dump_heap, &lsm->in_dump);
- vy_compact_heap_update(&scheduler->compact_heap, &lsm->in_compact);
+ vy_compaction_heap_update(&scheduler->compaction_heap,
+ &lsm->in_compaction);
}
static void
@@ -1200,7 +1203,7 @@ vy_task_dump_complete(struct vy_task *task)
slice = new_slices[i];
vy_lsm_unacct_range(lsm, range);
vy_range_add_slice(range, slice);
- vy_range_update_compact_priority(range, &lsm->opts);
+ vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
if (!vy_range_is_scheduled(range))
vy_range_heap_update(&lsm->range_heap,
@@ -1427,7 +1430,7 @@ err:
}
static int
-vy_task_compact_execute(struct vy_task *task)
+vy_task_compaction_execute(struct vy_task *task)
{
struct errinj *errinj = errinj(ERRINJ_VY_COMPACTION_DELAY, ERRINJ_BOOL);
if (errinj != NULL && errinj->bparam) {
@@ -1438,14 +1441,14 @@ vy_task_compact_execute(struct vy_task *task)
}
static int
-vy_task_compact_complete(struct vy_task *task)
+vy_task_compaction_complete(struct vy_task *task)
{
struct vy_scheduler *scheduler = task->scheduler;
struct vy_lsm *lsm = task->lsm;
struct vy_range *range = task->range;
struct vy_run *new_run = task->new_run;
- struct vy_disk_stmt_counter compact_output = new_run->count;
- struct vy_disk_stmt_counter compact_input;
+ struct vy_disk_stmt_counter compaction_output = new_run->count;
+ struct vy_disk_stmt_counter compaction_input;
struct vy_slice *first_slice = task->first_slice;
struct vy_slice *last_slice = task->last_slice;
struct vy_slice *slice, *next_slice, *new_slice = NULL;
@@ -1547,20 +1550,20 @@ vy_task_compact_complete(struct vy_task *task)
vy_lsm_unacct_range(lsm, range);
if (new_slice != NULL)
vy_range_add_slice_before(range, new_slice, first_slice);
- vy_disk_stmt_counter_reset(&compact_input);
+ vy_disk_stmt_counter_reset(&compaction_input);
for (slice = first_slice; ; slice = next_slice) {
next_slice = rlist_next_entry(slice, in_range);
vy_range_remove_slice(range, slice);
rlist_add_entry(&compacted_slices, slice, in_range);
- vy_disk_stmt_counter_add(&compact_input, &slice->count);
+ vy_disk_stmt_counter_add(&compaction_input, &slice->count);
if (slice == last_slice)
break;
}
range->n_compactions++;
range->version++;
- vy_range_update_compact_priority(range, &lsm->opts);
+ vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
- vy_lsm_acct_compaction(lsm, &compact_input, &compact_output);
+ vy_lsm_acct_compaction(lsm, &compaction_input, &compaction_output);
/*
* Unaccount unused runs and delete compacted slices.
@@ -1586,7 +1589,7 @@ vy_task_compact_complete(struct vy_task *task)
}
static void
-vy_task_compact_abort(struct vy_task *task)
+vy_task_compaction_abort(struct vy_task *task)
{
struct vy_scheduler *scheduler = task->scheduler;
struct vy_lsm *lsm = task->lsm;
@@ -1614,13 +1617,13 @@ vy_task_compact_abort(struct vy_task *task)
}
static int
-vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
- struct vy_lsm *lsm, struct vy_task **p_task)
+vy_task_compaction_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
+ struct vy_lsm *lsm, struct vy_task **p_task)
{
- static struct vy_task_ops compact_ops = {
- .execute = vy_task_compact_execute,
- .complete = vy_task_compact_complete,
- .abort = vy_task_compact_abort,
+ static struct vy_task_ops compaction_ops = {
+ .execute = vy_task_compaction_execute,
+ .complete = vy_task_compaction_complete,
+ .abort = vy_task_compaction_abort,
};
struct heap_node *range_node;
@@ -1631,7 +1634,7 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
range_node = vy_range_heap_top(&lsm->range_heap);
assert(range_node != NULL);
range = container_of(range_node, struct vy_range, heap_node);
- assert(range->compact_priority > 1);
+ assert(range->compaction_priority > 1);
if (vy_lsm_split_range(lsm, range) ||
vy_lsm_coalesce_range(lsm, range)) {
@@ -1639,7 +1642,8 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
return 0;
}
- struct vy_task *task = vy_task_new(scheduler, worker, lsm, &compact_ops);
+ struct vy_task *task = vy_task_new(scheduler, worker, lsm,
+ &compaction_ops);
if (task == NULL)
goto err_task;
@@ -1648,7 +1652,7 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
goto err_run;
struct vy_stmt_stream *wi;
- bool is_last_level = (range->compact_priority == range->slice_count);
+ bool is_last_level = (range->compaction_priority == range->slice_count);
wi = vy_write_iterator_new(task->cmp_def, lsm->disk_format,
lsm->index_id == 0, is_last_level,
scheduler->read_views,
@@ -1658,7 +1662,7 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
goto err_wi;
struct vy_slice *slice;
- int n = range->compact_priority;
+ int n = range->compaction_priority;
rlist_foreach_entry(slice, &range->slices, in_range) {
if (vy_write_iterator_new_slice(wi, slice) != 0)
goto err_wi_sub;
@@ -1693,7 +1697,7 @@ vy_task_compact_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
say_info("%s: started compacting range %s, runs %d/%d",
vy_lsm_name(lsm), vy_range_str(range),
- range->compact_priority, range->slice_count);
+ range->compaction_priority, range->slice_count);
*p_task = task;
return 0;
@@ -1867,24 +1871,24 @@ no_task:
* Returns 0 on success, -1 on failure.
*/
static int
-vy_scheduler_peek_compact(struct vy_scheduler *scheduler,
- struct vy_task **ptask)
+vy_scheduler_peek_compaction(struct vy_scheduler *scheduler,
+ struct vy_task **ptask)
{
struct vy_worker *worker = NULL;
retry:
*ptask = NULL;
- struct heap_node *pn = vy_compact_heap_top(&scheduler->compact_heap);
+ struct heap_node *pn = vy_compaction_heap_top(&scheduler->compaction_heap);
if (pn == NULL)
goto no_task; /* nothing to do */
- struct vy_lsm *lsm = container_of(pn, struct vy_lsm, in_compact);
- if (vy_lsm_compact_priority(lsm) <= 1)
+ struct vy_lsm *lsm = container_of(pn, struct vy_lsm, in_compaction);
+ if (vy_lsm_compaction_priority(lsm) <= 1)
goto no_task; /* nothing to do */
if (worker == NULL) {
- worker = vy_worker_pool_get(&scheduler->compact_pool);
+ worker = vy_worker_pool_get(&scheduler->compaction_pool);
if (worker == NULL)
return 0; /* all workers are busy */
}
- if (vy_task_compact_new(scheduler, worker, lsm, ptask) != 0) {
+ if (vy_task_compaction_new(scheduler, worker, lsm, ptask) != 0) {
vy_worker_pool_put(worker);
return -1;
}
@@ -1907,7 +1911,7 @@ vy_schedule(struct vy_scheduler *scheduler, struct vy_task **ptask)
if (*ptask != NULL)
return 0;
- if (vy_scheduler_peek_compact(scheduler, ptask) != 0)
+ if (vy_scheduler_peek_compaction(scheduler, ptask) != 0)
goto fail;
if (*ptask != NULL)
return 0;
diff --git a/src/box/vy_scheduler.h b/src/box/vy_scheduler.h
index 96ce721b..5b09f964 100644
--- a/src/box/vy_scheduler.h
+++ b/src/box/vy_scheduler.h
@@ -75,7 +75,7 @@ struct vy_scheduler {
/** Pool of threads for performing background dumps. */
struct vy_worker_pool dump_pool;
/** Pool of threads for performing background compactions. */
- struct vy_worker_pool compact_pool;
+ struct vy_worker_pool compaction_pool;
/** Queue of processed tasks, linked by vy_task::in_processed. */
struct stailq processed_tasks;
/**
@@ -85,9 +85,9 @@ struct vy_scheduler {
heap_t dump_heap;
/**
* Heap of LSM trees, ordered by compaction priority,
- * linked by vy_lsm::in_compact.
+ * linked by vy_lsm::in_compaction.
*/
- heap_t compact_heap;
+ heap_t compaction_heap;
/** Last error seen by the scheduler. */
struct diag diag;
/**
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index ae032d1d..7ed55ff5 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -162,7 +162,7 @@ struct vy_lsm_stat {
struct vy_disk_stmt_counter output;
/** Number of statements awaiting compaction. */
struct vy_disk_stmt_counter queue;
- } compact;
+ } compaction;
} disk;
/** TX write set statistics. */
struct {
@@ -222,7 +222,7 @@ struct vy_disk_stat {
int64_t input;
int64_t output;
int64_t queue;
- } compact;
+ } compaction;
};
static inline int
diff --git a/test/vinyl/ddl.result b/test/vinyl/ddl.result
index dbffc3e5..68bb6b3a 100644
--- a/test/vinyl/ddl.result
+++ b/test/vinyl/ddl.result
@@ -110,7 +110,7 @@ box.snapshot()
- ok
...
-- wait for compaction to complete
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.01) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
---
...
pk:alter{parts = {2, 'unsigned'}} -- success: space is empty now
diff --git a/test/vinyl/ddl.test.lua b/test/vinyl/ddl.test.lua
index 7c504046..9b870f35 100644
--- a/test/vinyl/ddl.test.lua
+++ b/test/vinyl/ddl.test.lua
@@ -36,7 +36,7 @@ space:delete{2}
pk:alter{parts = {2, 'unsigned'}} -- error: mem/run not empty
box.snapshot()
-- wait for compaction to complete
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.01) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
pk:alter{parts = {2, 'unsigned'}} -- success: space is empty now
space:replace{1, 2}
-- gh-3508 - Altering primary index of a vinyl space doesn't work as expected
diff --git a/test/vinyl/deferred_delete.result b/test/vinyl/deferred_delete.result
index 5a432d07..29945f8d 100644
--- a/test/vinyl/deferred_delete.result
+++ b/test/vinyl/deferred_delete.result
@@ -150,7 +150,7 @@ box.snapshot()
pk:compact()
---
...
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
pk:stat().rows -- 5 new REPLACEs
@@ -237,13 +237,13 @@ box.snapshot()
i1:compact()
---
...
-while i1:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
i2:compact()
---
...
-while i2:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
i1:stat().rows -- 5 new REPLACEs
@@ -427,7 +427,7 @@ box.snapshot()
sk:compact()
---
...
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
sk:stat().run_count -- 0
@@ -549,7 +549,7 @@ box.snapshot()
pk:compact()
---
...
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-- Compact the secondary index to cleanup garbage.
@@ -560,7 +560,7 @@ box.snapshot()
sk:compact()
---
...
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
sk:select()
@@ -628,7 +628,7 @@ box.snapshot()
pk:compact()
---
...
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
sk:select() -- [1, 10, 'c']
@@ -725,7 +725,7 @@ box.stat.reset()
pk:compact()
---
...
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
sk:stat().disk.dump.count -- 1
diff --git a/test/vinyl/deferred_delete.test.lua b/test/vinyl/deferred_delete.test.lua
index 53f75a03..d38802da 100644
--- a/test/vinyl/deferred_delete.test.lua
+++ b/test/vinyl/deferred_delete.test.lua
@@ -57,7 +57,7 @@ box.cfg{vinyl_cache = vinyl_cache}
-- Compact the primary index to generate deferred DELETEs.
box.snapshot()
pk:compact()
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
pk:stat().rows -- 5 new REPLACEs
i1:stat().rows -- 10 old REPLACE + 5 new REPLACEs + 10 deferred DELETEs
i2:stat().rows -- ditto
@@ -87,9 +87,9 @@ i2:stat().rows -- ditto
-- Check that they cleanup garbage statements.
box.snapshot()
i1:compact()
-while i1:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
i2:compact()
-while i2:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
i1:stat().rows -- 5 new REPLACEs
i2:stat().rows -- ditto
box.stat.reset()
@@ -151,7 +151,7 @@ box.stat.vinyl().memory.tx
box.snapshot()
sk:compact()
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
sk:stat().run_count -- 0
s:drop()
@@ -201,12 +201,12 @@ sk:stat().rows -- 10 old REPLACEs + 5 new REPLACEs
-- Compact the primary index to generate deferred DELETEs.
box.snapshot()
pk:compact()
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
-- Compact the secondary index to cleanup garbage.
box.snapshot()
sk:compact()
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
sk:select()
@@ -234,7 +234,7 @@ box.snapshot()
-- Generate deferred DELETEs.
pk:compact()
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
sk:select() -- [1, 10, 'c']
box.snapshot()
@@ -280,7 +280,7 @@ box.stat.reset()
-- Deferred DELETEs won't fit in memory and trigger dump
-- of the secondary index.
pk:compact()
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
sk:stat().disk.dump.count -- 1
diff --git a/test/vinyl/errinj.result b/test/vinyl/errinj.result
index a16475f5..4a3df6ae 100644
--- a/test/vinyl/errinj.result
+++ b/test/vinyl/errinj.result
@@ -497,7 +497,7 @@ box.snapshot()
- ok
...
--
--- Check that all dump/compact tasks that are in progress at
+-- Check that all dump/compaction tasks that are in progress at
-- the time when the server stops are aborted immediately.
--
s = box.schema.space.create('test', {engine = 'vinyl'})
@@ -961,7 +961,7 @@ c = fiber.channel(1)
_ = fiber.create(function() box.snapshot() c:put(true) end)
---
...
-while s.index.pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while s.index.pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
errinj.set('ERRINJ_SNAP_COMMIT_DELAY', false)
@@ -1049,7 +1049,7 @@ s.index.pk:compact()
while box.stat.ERROR.total - errors == 0 do fiber.sleep(0.001) end
---
...
-s.index.pk:stat().disk.compact.count -- 0
+s.index.pk:stat().disk.compaction.count -- 0
---
- 0
...
@@ -1057,10 +1057,10 @@ errinj.set("ERRINJ_WAL_IO", false)
---
- ok
...
-while s.index.pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while s.index.pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-s.index.pk:stat().disk.compact.count -- 1
+s.index.pk:stat().disk.compaction.count -- 1
---
- 1
...
diff --git a/test/vinyl/errinj.test.lua b/test/vinyl/errinj.test.lua
index d4590fb4..c9d04aaf 100644
--- a/test/vinyl/errinj.test.lua
+++ b/test/vinyl/errinj.test.lua
@@ -179,7 +179,7 @@ box.error.injection.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
box.snapshot()
--
--- Check that all dump/compact tasks that are in progress at
+-- Check that all dump/compaction tasks that are in progress at
-- the time when the server stops are aborted immediately.
--
s = box.schema.space.create('test', {engine = 'vinyl'})
@@ -348,7 +348,7 @@ _ = s:replace{2}
errinj.set('ERRINJ_SNAP_COMMIT_DELAY', true)
c = fiber.channel(1)
_ = fiber.create(function() box.snapshot() c:put(true) end)
-while s.index.pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while s.index.pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
errinj.set('ERRINJ_SNAP_COMMIT_DELAY', false)
c:get()
-- Check that all files corresponding to the last checkpoint
@@ -381,10 +381,10 @@ errinj.set("ERRINJ_WAL_IO", true)
errors = box.stat.ERROR.total
s.index.pk:compact()
while box.stat.ERROR.total - errors == 0 do fiber.sleep(0.001) end
-s.index.pk:stat().disk.compact.count -- 0
+s.index.pk:stat().disk.compaction.count -- 0
errinj.set("ERRINJ_WAL_IO", false)
-while s.index.pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
-s.index.pk:stat().disk.compact.count -- 1
+while s.index.pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
+s.index.pk:stat().disk.compaction.count -- 1
errinj.set("ERRINJ_VY_SCHED_TIMEOUT", 0)
box.snapshot() -- ok
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
index fac56cee..9d7f64e9 100644
--- a/test/vinyl/errinj_stat.result
+++ b/test/vinyl/errinj_stat.result
@@ -29,7 +29,7 @@ errinj = box.error.injection
---
...
--
--- Check disk.compact.queue stat.
+-- Check disk.compaction.queue stat.
--
s = box.schema.space.create('test', {engine = 'vinyl'})
---
@@ -43,14 +43,14 @@ function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
---
...
-i:stat().disk.compact.queue -- none
+i:stat().disk.compaction.queue -- none
---
- bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
---
- true
...
@@ -64,56 +64,56 @@ dump()
dump()
---
...
-i:stat().disk.compact.queue -- 30 statements
+i:stat().disk.compaction.queue -- 30 statements
---
- bytes_compressed: <bytes_compressed>
pages: 3
rows: 30
bytes: 411
...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
---
- true
...
dump()
---
...
-i:stat().disk.compact.queue -- 40 statements
+i:stat().disk.compaction.queue -- 40 statements
---
- bytes_compressed: <bytes_compressed>
pages: 4
rows: 40
bytes: 548
...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
---
- true
...
dump()
---
...
-i:stat().disk.compact.queue -- 50 statements
+i:stat().disk.compaction.queue -- 50 statements
---
- bytes_compressed: <bytes_compressed>
pages: 5
rows: 50
bytes: 685
...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
---
- true
...
box.stat.reset() -- doesn't affect queue size
---
...
-i:stat().disk.compact.queue -- 50 statements
+i:stat().disk.compaction.queue -- 50 statements
---
- bytes_compressed: <bytes_compressed>
pages: 5
rows: 50
bytes: 685
...
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
---
- true
...
@@ -121,10 +121,10 @@ errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
---
- ok
...
-while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
+while i:stat().disk.compaction.count < 2 do fiber.sleep(0.01) end
---
...
-i:stat().disk.compact.queue -- none
+i:stat().disk.compaction.queue -- none
---
- bytes_compressed: <bytes_compressed>
pages: 0
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
index 1e0e63ae..6cd95cf9 100644
--- a/test/vinyl/errinj_stat.test.lua
+++ b/test/vinyl/errinj_stat.test.lua
@@ -15,31 +15,31 @@ fiber = require('fiber')
errinj = box.error.injection
--
--- Check disk.compact.queue stat.
+-- Check disk.compaction.queue stat.
--
s = box.schema.space.create('test', {engine = 'vinyl'})
i = s:create_index('pk', {run_count_per_level = 2})
function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
-i:stat().disk.compact.queue -- none
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue -- none
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
dump()
dump()
-i:stat().disk.compact.queue -- 30 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue -- 30 statements
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
dump()
-i:stat().disk.compact.queue -- 40 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue -- 40 statements
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
dump()
-i:stat().disk.compact.queue -- 50 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue -- 50 statements
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
box.stat.reset() -- doesn't affect queue size
-i:stat().disk.compact.queue -- 50 statements
-i:stat().disk.compact.queue.bytes == box.stat.vinyl().disk.compact.queue
+i:stat().disk.compaction.queue -- 50 statements
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
-while i:stat().disk.compact.count < 2 do fiber.sleep(0.01) end
-i:stat().disk.compact.queue -- none
+while i:stat().disk.compaction.count < 2 do fiber.sleep(0.01) end
+i:stat().disk.compaction.queue -- none
s:drop()
test_run:cmd("clear filter")
diff --git a/test/vinyl/snap_io_rate.result b/test/vinyl/snap_io_rate.result
index 073c9ff9..2390e139 100644
--- a/test/vinyl/snap_io_rate.result
+++ b/test/vinyl/snap_io_rate.result
@@ -86,7 +86,7 @@ box.snapshot()
---
- ok
...
-while s.index.primary:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while s.index.primary:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
t2 = fiber.time()
diff --git a/test/vinyl/snap_io_rate.test.lua b/test/vinyl/snap_io_rate.test.lua
index 27e88e6b..5a09bfd9 100644
--- a/test/vinyl/snap_io_rate.test.lua
+++ b/test/vinyl/snap_io_rate.test.lua
@@ -37,7 +37,7 @@ rate < box.cfg.snap_io_rate_limit or rate
fill()
t1 = fiber.time()
box.snapshot()
-while s.index.primary:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while s.index.primary:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
t2 = fiber.time()
-- dump + compaction => multiply by 2
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 315a5853..68aea30b 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -150,7 +150,23 @@ istat()
bytes: 0
disk:
index_size: 0
- rows: 0
+ compaction:
+ input:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
+ output:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ queue:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
statement:
inserts: 0
replaces: 0
@@ -166,40 +182,24 @@ istat()
pages: 0
rows: 0
bytes: 0
- compact:
- input:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- count: 0
- output:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- queue:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- iterator:
- read:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- bloom:
- hit: 0
- miss: 0
- lookup: 0
- get:
- rows: 0
- bytes: 0
bloom_size: 0
+ iterator:
+ read:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ bloom:
+ hit: 0
+ miss: 0
+ lookup: 0
+ get:
+ rows: 0
+ bytes: 0
+ bytes: 0
pages: 0
bytes_compressed: 0
- bytes: 0
+ rows: 0
txw:
bytes: 0
rows: 0
@@ -228,7 +228,7 @@ gstat()
dump:
input: 0
output: 0
- compact:
+ compaction:
input: 0
output: 0
queue: 0
@@ -291,12 +291,12 @@ stat_diff(istat(), st)
rows: 25
index_size: 294
rows: 25
- bytes: 26049
bytes_compressed: <bytes_compressed>
- bloom_size: 70
- statement:
- replaces: 25
pages: 7
+ bloom_size: 70
+ statement:
+ replaces: 25
+ bytes: 26049
bytes: 26049
put:
rows: 25
@@ -313,7 +313,7 @@ box.snapshot()
---
- ok
...
-wait(istat, st, 'disk.compact.count', 1)
+wait(istat, st, 'disk.compaction.count', 1)
---
...
stat_diff(istat(), st)
@@ -330,13 +330,7 @@ stat_diff(istat(), st)
bytes_compressed: <bytes_compressed>
rows: 50
index_size: 252
- rows: 25
- bytes: 26042
- bytes_compressed: <bytes_compressed>
- pages: 6
- statement:
- replaces: 25
- compact:
+ compaction:
input:
bytes: 78140
pages: 20
@@ -348,6 +342,12 @@ stat_diff(istat(), st)
pages: 13
bytes_compressed: <bytes_compressed>
rows: 50
+ rows: 25
+ bytes_compressed: <bytes_compressed>
+ pages: 6
+ statement:
+ replaces: 25
+ bytes: 26042
put:
rows: 50
bytes: 53050
@@ -572,7 +572,7 @@ box.snapshot()
---
- ok
...
-wait(istat, st, 'disk.compact.count', 2)
+wait(istat, st, 'disk.compaction.count', 2)
---
...
st = istat()
@@ -709,11 +709,11 @@ gstat().disk.dump.output == istat().disk.dump.output.bytes
---
- true
...
-gstat().disk.compact.input == istat().disk.compact.input.bytes
+gstat().disk.compaction.input == istat().disk.compaction.input.bytes
---
- true
...
-gstat().disk.compact.output == istat().disk.compact.output.bytes
+gstat().disk.compaction.output == istat().disk.compaction.output.bytes
---
- true
...
@@ -1000,7 +1000,23 @@ istat()
bytes: 0
disk:
index_size: 1050
- rows: 100
+ compaction:
+ input:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
+ output:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ queue:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
statement:
inserts: 0
replaces: 100
@@ -1016,40 +1032,24 @@ istat()
pages: 0
rows: 0
bytes: 0
- compact:
- input:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- count: 0
- output:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- queue:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- iterator:
- read:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- bloom:
- hit: 0
- miss: 0
- lookup: 0
- get:
- rows: 0
- bytes: 0
bloom_size: 140
- pages: 25
- bytes_compressed: <bytes_compressed>
+ iterator:
+ read:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ bloom:
+ hit: 0
+ miss: 0
+ lookup: 0
+ get:
+ rows: 0
+ bytes: 0
bytes: 104300
+ pages: 25
+ bytes_compressed: <bytes_compressed>
+ rows: 100
txw:
bytes: 0
rows: 0
@@ -1078,7 +1078,7 @@ gstat()
dump:
input: 0
output: 0
- compact:
+ compaction:
input: 0
output: 0
queue: 0
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 56e544d1..0173cf89 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -118,7 +118,7 @@ stat_diff(istat(), st)
st = istat()
for i = 1, 100, 2 do put(i) end
box.snapshot()
-wait(istat, st, 'disk.compact.count', 1)
+wait(istat, st, 'disk.compaction.count', 1)
stat_diff(istat(), st)
-- point lookup from disk + cache put
@@ -174,7 +174,7 @@ stat_diff(istat(), st, 'cache')
for i = 1, 100 do put(i) end
st = istat()
box.snapshot()
-wait(istat, st, 'disk.compact.count', 2)
+wait(istat, st, 'disk.compaction.count', 2)
st = istat()
st.range_count -- 2
st.run_count -- 2
@@ -207,8 +207,8 @@ box.rollback()
-- dump and compaction totals
gstat().disk.dump.input == istat().disk.dump.input.bytes
gstat().disk.dump.output == istat().disk.dump.output.bytes
-gstat().disk.compact.input == istat().disk.compact.input.bytes
-gstat().disk.compact.output == istat().disk.compact.output.bytes
+gstat().disk.compaction.input == istat().disk.compaction.input.bytes
+gstat().disk.compaction.output == istat().disk.compaction.output.bytes
-- use memory
st = gstat()
diff --git a/test/vinyl/write_iterator.result b/test/vinyl/write_iterator.result
index e6bf1b01..88a1c287 100644
--- a/test/vinyl/write_iterator.result
+++ b/test/vinyl/write_iterator.result
@@ -838,27 +838,27 @@ box.snapshot()
- ok
...
-- Wait for compaction.
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-pk:stat().disk.compact.count -- 1
+pk:stat().disk.compaction.count -- 1
---
- 1
...
-sk:stat().disk.compact.count -- 1
+sk:stat().disk.compaction.count -- 1
---
- 1
...
-- All INSERT+DELETE pairs should have been annihilated,
-- only padding is left.
-pk:stat().disk.compact.output.rows - PAD2 -- 0
+pk:stat().disk.compaction.output.rows - PAD2 -- 0
---
- 0
...
-sk:stat().disk.compact.output.rows - PAD2 -- 0
+sk:stat().disk.compaction.output.rows - PAD2 -- 0
---
- 0
...
@@ -985,17 +985,17 @@ box.snapshot()
- ok
...
-- Wait for compaction.
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
---
...
-pk:stat().disk.compact.count -- 1
+pk:stat().disk.compaction.count -- 1
---
- 1
...
-sk:stat().disk.compact.count -- 1
+sk:stat().disk.compaction.count -- 1
---
- 1
...
diff --git a/test/vinyl/write_iterator.test.lua b/test/vinyl/write_iterator.test.lua
index a4ba42aa..069c7f69 100644
--- a/test/vinyl/write_iterator.test.lua
+++ b/test/vinyl/write_iterator.test.lua
@@ -355,14 +355,14 @@ s:delete{8}
for i = 1001, 1000 + PAD2 do s:replace{i, i} end
box.snapshot()
-- Wait for compaction.
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
-pk:stat().disk.compact.count -- 1
-sk:stat().disk.compact.count -- 1
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
+pk:stat().disk.compaction.count -- 1
+sk:stat().disk.compaction.count -- 1
-- All INSERT+DELETE pairs should have been annihilated,
-- only padding is left.
-pk:stat().disk.compact.output.rows - PAD2 -- 0
-sk:stat().disk.compact.output.rows - PAD2 -- 0
+pk:stat().disk.compaction.output.rows - PAD2 -- 0
+sk:stat().disk.compaction.output.rows - PAD2 -- 0
pk:select(1000, {iterator = 'LE'}) -- empty
sk:select(1000, {iterator = 'LE'}) -- empty
s:drop()
@@ -408,10 +408,10 @@ s:delete{8}
for i = 1001, 1000 + PAD2 do s:replace{i, i} end
box.snapshot()
-- Wait for compaction.
-while pk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
-while sk:stat().disk.compact.count == 0 do fiber.sleep(0.001) end
-pk:stat().disk.compact.count -- 1
-sk:stat().disk.compact.count -- 1
+while pk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
+while sk:stat().disk.compaction.count == 0 do fiber.sleep(0.001) end
+pk:stat().disk.compaction.count -- 1
+sk:stat().disk.compaction.count -- 1
-- If INSERT+DELETE statements stored in the two compacted runs
-- were annihilated we would see tuples stored in the first run.
pk:select(1000, {iterator = 'LE'}) -- empty
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 05/12] vinyl: bump range version in vy_range.c
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (3 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 04/12] vinyl: rename compact to compaction Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 06/12] vinyl: don't add dropped LSM trees to the scheduler during recovery Vladimir Davydov
` (8 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
Currently, we bump range->version in vy_scheduler.c. This looks like an
encapsulation violation, and may easily result in an error (as we have
to be cautious to inc range->version whenever we modify a range). That
said, let's bump range version right in vy_range.c.
---
src/box/vy_range.c | 3 +++
src/box/vy_scheduler.c | 2 --
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/src/box/vy_range.c b/src/box/vy_range.c
index f489c95d..f649aff7 100644
--- a/src/box/vy_range.c
+++ b/src/box/vy_range.c
@@ -241,6 +241,7 @@ vy_range_add_slice(struct vy_range *range, struct vy_slice *slice)
rlist_add_entry(&range->slices, slice, in_range);
range->slice_count++;
vy_disk_stmt_counter_add(&range->count, &slice->count);
+ range->version++;
}
void
@@ -250,6 +251,7 @@ vy_range_add_slice_before(struct vy_range *range, struct vy_slice *slice,
rlist_add_tail(&next_slice->in_range, &slice->in_range);
range->slice_count++;
vy_disk_stmt_counter_add(&range->count, &slice->count);
+ range->version++;
}
void
@@ -260,6 +262,7 @@ vy_range_remove_slice(struct vy_range *range, struct vy_slice *slice)
rlist_del_entry(slice, in_range);
range->slice_count--;
vy_disk_stmt_counter_sub(&range->count, &slice->count);
+ range->version++;
}
/**
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index ba20ef53..63ac948f 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -1208,7 +1208,6 @@ vy_task_dump_complete(struct vy_task *task)
if (!vy_range_is_scheduled(range))
vy_range_heap_update(&lsm->range_heap,
&range->heap_node);
- range->version++;
}
free(new_slices);
@@ -1560,7 +1559,6 @@ vy_task_compaction_complete(struct vy_task *task)
break;
}
range->n_compactions++;
- range->version++;
vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
vy_lsm_acct_compaction(lsm, &compaction_input, &compaction_output);
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 06/12] vinyl: don't add dropped LSM trees to the scheduler during recovery
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (4 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 05/12] vinyl: bump range version in vy_range.c Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler Vladimir Davydov
` (7 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
During local recovery we may encounter an LSM tree marked as dropped.
This means that the LSM tree was dropped before restart and hence will
be deleted before recovery completion. There's no need to add such trees
to the vinyl scheduler - it looks confusing and can potentially result
in mistakes when the code gets modified.
---
src/box/vinyl.c | 42 +++++++++++++++++++++++++++---------------
src/box/vy_scheduler.c | 2 ++
2 files changed, 29 insertions(+), 15 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 8028fd2b..5965700b 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -749,18 +749,16 @@ vinyl_index_open(struct index *index)
diag_set(SystemError, "can not access vinyl data directory");
return -1;
}
- int rc;
switch (env->status) {
case VINYL_ONLINE:
/*
* The recovery is complete, simply
* create a new index.
*/
- rc = vy_lsm_create(lsm);
- if (rc == 0) {
- /* Make sure reader threads are up and running. */
- vy_run_env_enable_coio(&env->run_env);
- }
+ if (vy_lsm_create(lsm) != 0)
+ return -1;
+ /* Make sure reader threads are up and running. */
+ vy_run_env_enable_coio(&env->run_env);
break;
case VINYL_INITIAL_RECOVERY_REMOTE:
case VINYL_FINAL_RECOVERY_REMOTE:
@@ -769,7 +767,8 @@ vinyl_index_open(struct index *index)
* exist locally, and we should create the
* index directory from scratch.
*/
- rc = vy_lsm_create(lsm);
+ if (vy_lsm_create(lsm) != 0)
+ return -1;
break;
case VINYL_INITIAL_RECOVERY_LOCAL:
case VINYL_FINAL_RECOVERY_LOCAL:
@@ -779,17 +778,30 @@ vinyl_index_open(struct index *index)
* have already been created, so try to load
* the index files from it.
*/
- rc = vy_lsm_recover(lsm, env->recovery, &env->run_env,
- vclock_sum(env->recovery_vclock),
- env->status == VINYL_INITIAL_RECOVERY_LOCAL,
- env->force_recovery);
+ if (vy_lsm_recover(lsm, env->recovery, &env->run_env,
+ vclock_sum(env->recovery_vclock),
+ env->status == VINYL_INITIAL_RECOVERY_LOCAL,
+ env->force_recovery) != 0)
+ return -1;
break;
default:
unreachable();
}
- if (rc == 0)
+ /*
+ * Add the new LSM tree to the scheduler so that it can
+ * be dumped and compacted.
+ *
+ * Note, during local recovery an LSM tree may be marked
+ * as dropped, which means that it will be dropped before
+ * recovery is complete. In this case there's no need in
+ * letting the scheduler know about it.
+ */
+ if (!lsm->is_dropped)
vy_scheduler_add_lsm(&env->scheduler, lsm);
- return rc;
+ else
+ assert(env->status == VINYL_INITIAL_RECOVERY_LOCAL ||
+ env->status == VINYL_FINAL_RECOVERY_LOCAL);
+ return 0;
}
static void
@@ -911,8 +923,6 @@ vinyl_index_commit_drop(struct index *index, int64_t lsn)
struct vy_env *env = vy_env(index->engine);
struct vy_lsm *lsm = vy_lsm(index);
- vy_scheduler_remove_lsm(&env->scheduler, lsm);
-
/*
* We can't abort here, because the index drop request has
* already been written to WAL. So if we fail to write the
@@ -924,6 +934,8 @@ vinyl_index_commit_drop(struct index *index, int64_t lsn)
if (env->status == VINYL_FINAL_RECOVERY_LOCAL && lsm->is_dropped)
return;
+ vy_scheduler_remove_lsm(&env->scheduler, lsm);
+
lsm->is_dropped = true;
vy_log_tx_begin();
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index 63ac948f..f431eb24 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -503,6 +503,7 @@ vy_scheduler_destroy(struct vy_scheduler *scheduler)
void
vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
+ assert(!lsm->is_dropped);
assert(lsm->in_dump.pos == UINT32_MAX);
assert(lsm->in_compaction.pos == UINT32_MAX);
vy_dump_heap_insert(&scheduler->dump_heap, &lsm->in_dump);
@@ -513,6 +514,7 @@ vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
void
vy_scheduler_remove_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
+ assert(!lsm->is_dropped);
assert(lsm->in_dump.pos != UINT32_MAX);
assert(lsm->in_compaction.pos != UINT32_MAX);
vy_dump_heap_delete(&scheduler->dump_heap, &lsm->in_dump);
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (5 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 06/12] vinyl: don't add dropped LSM trees to the scheduler during recovery Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-16 16:36 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 08/12] vinyl: add dump count to global scheduler statistics Vladimir Davydov
` (6 subsequent siblings)
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
Although it's more convenient to maintain dump/compaction input/output
and queue metrics in vy_lsm_env, semantically it's incorrect as those
metrics characterize the scheduler not the LSM environment. Also, we
can't easily extend those stats with e.g. the number of completed dumps
or the number of tasks in progress, because those are only known to the
scheduler.
That said, let's introduce 'scheduler' section in box.stat.vinyl() and
move dump/compaction stats from 'disk' to the new section. Let's also
move the stats accounting from vy_lsm.c to vy_scheduler.c. The 'disk'
section now stores only the size of data and index on disk and no
cumulative statistics, which makes it similar to the 'memory' section.
Note, this patch flattens the stats (disk.compaction.input is moved to
scheduler.compaction_input and so forth), because all other global stats
are reported without using nested tables.
---
src/box/vinyl.c | 40 ++++++++++-----------
src/box/vy_lsm.c | 23 ++++--------
src/box/vy_lsm.h | 18 ++++++++--
src/box/vy_scheduler.c | 53 ++++++++++++++++++++++++++--
src/box/vy_scheduler.h | 9 +++++
src/box/vy_stat.h | 29 ++++++++-------
test/vinyl/errinj_stat.result | 10 +++---
test/vinyl/errinj_stat.test.lua | 10 +++---
test/vinyl/stat.result | 78 ++++++++++++++++++++---------------------
test/vinyl/stat.test.lua | 8 ++---
10 files changed, 166 insertions(+), 112 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 5965700b..b30e69a1 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -245,6 +245,20 @@ static struct trigger on_replace_vinyl_deferred_delete;
/** {{{ Introspection */
static void
+vy_info_append_scheduler(struct vy_env *env, struct info_handler *h)
+{
+ struct vy_scheduler_stat *stat = &env->scheduler.stat;
+
+ info_table_begin(h, "scheduler");
+ info_append_int(h, "dump_input", stat->dump_input);
+ info_append_int(h, "dump_output", stat->dump_output);
+ info_append_int(h, "compaction_input", stat->compaction_input);
+ info_append_int(h, "compaction_output", stat->compaction_output);
+ info_append_int(h, "compaction_queue", stat->compaction_queue);
+ info_table_end(h); /* scheduler */
+}
+
+static void
vy_info_append_regulator(struct vy_env *env, struct info_handler *h)
{
struct vy_regulator *r = &env->regulator;
@@ -295,24 +309,9 @@ vy_info_append_memory(struct vy_env *env, struct info_handler *h)
static void
vy_info_append_disk(struct vy_env *env, struct info_handler *h)
{
- struct vy_disk_stat *stat = &env->lsm_env.disk_stat;
-
info_table_begin(h, "disk");
-
- info_append_int(h, "data", stat->data);
- info_append_int(h, "index", stat->index);
-
- info_table_begin(h, "dump");
- info_append_int(h, "input", stat->dump.input);
- info_append_int(h, "output", stat->dump.output);
- info_table_end(h); /* dump */
-
- info_table_begin(h, "compaction");
- info_append_int(h, "input", stat->compaction.input);
- info_append_int(h, "output", stat->compaction.output);
- info_append_int(h, "queue", stat->compaction.queue);
- info_table_end(h); /* compaction */
-
+ info_append_int(h, "data", env->lsm_env.disk_data_size);
+ info_append_int(h, "index", env->lsm_env.disk_index_size);
info_table_end(h); /* disk */
}
@@ -325,6 +324,7 @@ vinyl_engine_stat(struct vinyl_engine *vinyl, struct info_handler *h)
vy_info_append_tx(env, h);
vy_info_append_memory(env, h);
vy_info_append_disk(env, h);
+ vy_info_append_scheduler(env, h);
vy_info_append_regulator(env, h);
info_end(h);
}
@@ -514,11 +514,7 @@ vinyl_engine_reset_stat(struct engine *engine)
struct tx_manager *xm = env->xm;
memset(&xm->stat, 0, sizeof(xm->stat));
- struct vy_disk_stat *disk_stat = &env->lsm_env.disk_stat;
- disk_stat->dump.input = 0;
- disk_stat->dump.output = 0;
- disk_stat->compaction.input = 0;
- disk_stat->compaction.output = 0;
+ vy_scheduler_reset_stat(&env->scheduler);
}
/** }}} Introspection */
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index abadab5c..b79c6e0c 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -239,9 +239,6 @@ vy_lsm_delete(struct vy_lsm *lsm)
assert(lsm->env->lsm_count > 0);
lsm->env->lsm_count--;
- lsm->env->disk_stat.compaction.queue -=
- lsm->stat.disk.compaction.queue.bytes;
-
if (lsm->pk != NULL)
vy_lsm_unref(lsm->pk);
@@ -691,11 +688,11 @@ vy_lsm_add_run(struct vy_lsm *lsm, struct vy_run *run)
/* Data size is consistent with space.bsize. */
if (lsm->index_id == 0)
- env->disk_stat.data += run->count.bytes;
+ env->disk_data_size += run->count.bytes;
/* Index size is consistent with index.bsize. */
- env->disk_stat.index += bloom_size + page_index_size;
+ env->disk_index_size += bloom_size + page_index_size;
if (lsm->index_id > 0)
- env->disk_stat.index += run->count.bytes;
+ env->disk_index_size += run->count.bytes;
}
void
@@ -720,11 +717,11 @@ vy_lsm_remove_run(struct vy_lsm *lsm, struct vy_run *run)
/* Data size is consistent with space.bsize. */
if (lsm->index_id == 0)
- env->disk_stat.data -= run->count.bytes;
+ env->disk_data_size -= run->count.bytes;
/* Index size is consistent with index.bsize. */
- env->disk_stat.index -= bloom_size + page_index_size;
+ env->disk_index_size -= bloom_size + page_index_size;
if (lsm->index_id > 0)
- env->disk_stat.index -= run->count.bytes;
+ env->disk_index_size -= run->count.bytes;
}
void
@@ -751,7 +748,6 @@ vy_lsm_acct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_collect(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
- lsm->env->disk_stat.compaction.queue += range->compaction_queue.bytes;
}
void
@@ -760,7 +756,6 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_discard(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_sub(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
- lsm->env->disk_stat.compaction.queue -= range->compaction_queue.bytes;
}
void
@@ -771,9 +766,6 @@ vy_lsm_acct_dump(struct vy_lsm *lsm,
lsm->stat.disk.dump.count++;
vy_stmt_counter_add(&lsm->stat.disk.dump.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.dump.output, output);
-
- lsm->env->disk_stat.dump.input += input->bytes;
- lsm->env->disk_stat.dump.output += output->bytes;
}
void
@@ -784,9 +776,6 @@ vy_lsm_acct_compaction(struct vy_lsm *lsm,
lsm->stat.disk.compaction.count++;
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.output, output);
-
- lsm->env->disk_stat.compaction.input += input->bytes;
- lsm->env->disk_stat.compaction.output += output->bytes;
}
int
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index e7487995..b486a662 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -91,8 +91,22 @@ struct vy_lsm_env {
size_t bloom_size;
/** Size of memory used for page index. */
size_t page_index_size;
- /** Global disk statistics. */
- struct vy_disk_stat disk_stat;
+ /**
+ * Size of disk space used for storing data of all spaces,
+ * in bytes, without taking into account disk compression.
+ * By 'data' we mean statements stored in primary indexes
+ * only, which is consistent with space.bsize().
+ */
+ int64_t disk_data_size;
+ /**
+ * Size of disk space used for indexing data in all spaces,
+ * in bytes, without taking into account disk compression.
+ * This consists of page indexes and bloom filters, which
+ * are stored in .index files, as well as the total size of
+ * statements stored in secondary index .run files, which
+ * is consistent with index.bsize().
+ */
+ int64_t disk_index_size;
/** Memory pool for vy_history_node allocations. */
struct mempool history_node_pool;
};
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index f431eb24..2d4f6bed 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -501,6 +501,40 @@ vy_scheduler_destroy(struct vy_scheduler *scheduler)
}
void
+vy_scheduler_reset_stat(struct vy_scheduler *scheduler)
+{
+ struct vy_scheduler_stat *stat = &scheduler->stat;
+ stat->dump_input = 0;
+ stat->dump_output = 0;
+ stat->compaction_input = 0;
+ stat->compaction_output = 0;
+}
+
+/**
+ * Account an LSM tree to the compaction queue stats.
+ * Called after updating LSM tree compaction priority.
+ */
+static void
+vy_scheduler_acct_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
+{
+ if (lsm->is_dropped)
+ return;
+ scheduler->stat.compaction_queue += lsm->stat.disk.compaction.queue.bytes;
+}
+
+/**
+ * Unaccount an LSM tree from the compaction queue stats.
+ * Called before updating LSM tree compaction priority.
+ */
+static void
+vy_scheduler_unacct_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
+{
+ if (lsm->is_dropped)
+ return;
+ scheduler->stat.compaction_queue -= lsm->stat.disk.compaction.queue.bytes;
+}
+
+void
vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
assert(!lsm->is_dropped);
@@ -509,6 +543,7 @@ vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
vy_dump_heap_insert(&scheduler->dump_heap, &lsm->in_dump);
vy_compaction_heap_insert(&scheduler->compaction_heap,
&lsm->in_compaction);
+ vy_scheduler_acct_lsm(scheduler, lsm);
}
void
@@ -520,6 +555,7 @@ vy_scheduler_remove_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
vy_dump_heap_delete(&scheduler->dump_heap, &lsm->in_dump);
vy_compaction_heap_delete(&scheduler->compaction_heap,
&lsm->in_compaction);
+ vy_scheduler_unacct_lsm(scheduler, lsm);
lsm->in_dump.pos = UINT32_MAX;
lsm->in_compaction.pos = UINT32_MAX;
}
@@ -613,7 +649,9 @@ void
vy_scheduler_force_compaction(struct vy_scheduler *scheduler,
struct vy_lsm *lsm)
{
+ vy_scheduler_unacct_lsm(scheduler, lsm);
vy_lsm_force_compaction(lsm);
+ vy_scheduler_acct_lsm(scheduler, lsm);
vy_scheduler_update_lsm(scheduler, lsm);
fiber_cond_signal(&scheduler->scheduler_cond);
}
@@ -1199,6 +1237,7 @@ vy_task_dump_complete(struct vy_task *task)
* LSM tree state, when the same statement is present twice,
* in memory and on disk.
*/
+ vy_scheduler_unacct_lsm(scheduler, lsm);
for (range = begin_range, i = 0; range != end_range;
range = vy_range_tree_next(lsm->tree, range), i++) {
assert(i < lsm->range_count);
@@ -1211,6 +1250,7 @@ vy_task_dump_complete(struct vy_task *task)
vy_range_heap_update(&lsm->range_heap,
&range->heap_node);
}
+ vy_scheduler_acct_lsm(scheduler, lsm);
free(new_slices);
delete_mems:
@@ -1227,6 +1267,8 @@ delete_mems:
}
lsm->dump_lsn = MAX(lsm->dump_lsn, dump_lsn);
vy_lsm_acct_dump(lsm, &dump_input, &dump_output);
+ scheduler->stat.dump_input += dump_input.bytes;
+ scheduler->stat.dump_output += dump_output.bytes;
/* The iterator has been cleaned up in a worker thread. */
task->wi->iface->close(task->wi);
@@ -1548,6 +1590,7 @@ vy_task_compaction_complete(struct vy_task *task)
* the compacted slices were.
*/
RLIST_HEAD(compacted_slices);
+ vy_scheduler_unacct_lsm(scheduler, lsm);
vy_lsm_unacct_range(lsm, range);
if (new_slice != NULL)
vy_range_add_slice_before(range, new_slice, first_slice);
@@ -1564,6 +1607,9 @@ vy_task_compaction_complete(struct vy_task *task)
vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
vy_lsm_acct_compaction(lsm, &compaction_input, &compaction_output);
+ vy_scheduler_acct_lsm(scheduler, lsm);
+ scheduler->stat.compaction_input += compaction_input.bytes;
+ scheduler->stat.compaction_output += compaction_output.bytes;
/*
* Unaccount unused runs and delete compacted slices.
@@ -1636,8 +1682,11 @@ vy_task_compaction_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
range = container_of(range_node, struct vy_range, heap_node);
assert(range->compaction_priority > 1);
- if (vy_lsm_split_range(lsm, range) ||
- vy_lsm_coalesce_range(lsm, range)) {
+ vy_scheduler_unacct_lsm(scheduler, lsm);
+ bool lsm_changed = (vy_lsm_split_range(lsm, range) ||
+ vy_lsm_coalesce_range(lsm, range));
+ vy_scheduler_acct_lsm(scheduler, lsm);
+ if (lsm_changed) {
vy_scheduler_update_lsm(scheduler, lsm);
return 0;
}
diff --git a/src/box/vy_scheduler.h b/src/box/vy_scheduler.h
index 5b09f964..2d4352d7 100644
--- a/src/box/vy_scheduler.h
+++ b/src/box/vy_scheduler.h
@@ -41,6 +41,7 @@
#define HEAP_FORWARD_DECLARATION
#include "salad/heap.h"
#include "salad/stailq.h"
+#include "vy_stat.h"
#if defined(__cplusplus)
extern "C" {
@@ -139,6 +140,8 @@ struct vy_scheduler {
double dump_start;
/** Signaled on dump round completion. */
struct fiber_cond dump_cond;
+ /** Scheduler statistics. */
+ struct vy_scheduler_stat stat;
/**
* Function called by the scheduler upon dump round
* completion. It is supposed to free memory released
@@ -184,6 +187,12 @@ void
vy_scheduler_destroy(struct vy_scheduler *scheduler);
/**
+ * Reset scheduler statistics (called by box.stat.reset).
+ */
+void
+vy_scheduler_reset_stat(struct vy_scheduler *scheduler);
+
+/**
* Add an LSM tree to scheduler dump/compaction queues.
*/
void
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index 7ed55ff5..195b09a4 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -206,23 +206,22 @@ struct vy_tx_stat {
};
/**
- * Global disk statistics.
+ * Scheduler statistics.
*
- * Fields correspond to those of per LSM tree statistics.
- * All counters are given in bytes, uncompressed.
+ * All byte counters are given without taking into account
+ * disk compression.
*/
-struct vy_disk_stat {
- int64_t data;
- int64_t index;
- struct {
- int64_t input;
- int64_t output;
- } dump;
- struct {
- int64_t input;
- int64_t output;
- int64_t queue;
- } compaction;
+struct vy_scheduler_stat {
+ /** Number of bytes read by dump tasks. */
+ int64_t dump_input;
+ /** Number of bytes written by dump tasks. */
+ int64_t dump_output;
+ /** Number of bytes read by compaction tasks. */
+ int64_t compaction_input;
+ /** Number of bytes written by compaction tasks. */
+ int64_t compaction_output;
+ /** Size of compaction queue, in bytes. */
+ int64_t compaction_queue;
};
static inline int
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
index 9d7f64e9..fe379cef 100644
--- a/test/vinyl/errinj_stat.result
+++ b/test/vinyl/errinj_stat.result
@@ -50,7 +50,7 @@ i:stat().disk.compaction.queue -- none
rows: 0
bytes: 0
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -71,7 +71,7 @@ i:stat().disk.compaction.queue -- 30 statements
rows: 30
bytes: 411
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -85,7 +85,7 @@ i:stat().disk.compaction.queue -- 40 statements
rows: 40
bytes: 548
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -99,7 +99,7 @@ i:stat().disk.compaction.queue -- 50 statements
rows: 50
bytes: 685
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -113,7 +113,7 @@ i:stat().disk.compaction.queue -- 50 statements
rows: 50
bytes: 685
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
index 6cd95cf9..3556f50a 100644
--- a/test/vinyl/errinj_stat.test.lua
+++ b/test/vinyl/errinj_stat.test.lua
@@ -22,21 +22,21 @@ i = s:create_index('pk', {run_count_per_level = 2})
function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
i:stat().disk.compaction.queue -- none
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
dump()
dump()
i:stat().disk.compaction.queue -- 30 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
dump()
i:stat().disk.compaction.queue -- 40 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
dump()
i:stat().disk.compaction.queue -- 50 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
box.stat.reset() -- doesn't affect queue size
i:stat().disk.compaction.queue -- 50 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
while i:stat().disk.compaction.count < 2 do fiber.sleep(0.01) end
i:stat().disk.compaction.queue -- none
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 68aea30b..16f01945 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -224,30 +224,29 @@ istat()
...
gstat()
---
-- disk:
- dump:
- input: 0
- output: 0
- compaction:
- input: 0
- output: 0
- queue: 0
- data: 0
- index: 0
+- tx:
+ conflict: 0
+ commit: 0
+ rollback: 0
+ statements: 0
+ transactions: 0
+ gap_locks: 0
+ read_views: 0
memory:
tuple_cache: 0
tx: 0
level0: 0
page_index: 0
bloom_filter: 0
- tx:
- conflict: 0
- commit: 0
- rollback: 0
- statements: 0
- transactions: 0
- gap_locks: 0
- read_views: 0
+ disk:
+ data: 0
+ index: 0
+ scheduler:
+ compaction_output: 0
+ compaction_queue: 0
+ dump_output: 0
+ dump_input: 0
+ compaction_input: 0
...
--
-- Index statistics.
@@ -701,19 +700,19 @@ box.rollback()
-- Global statistics.
--
-- dump and compaction totals
-gstat().disk.dump.input == istat().disk.dump.input.bytes
+gstat().scheduler.dump_input == istat().disk.dump.input.bytes
---
- true
...
-gstat().disk.dump.output == istat().disk.dump.output.bytes
+gstat().scheduler.dump_output == istat().disk.dump.output.bytes
---
- true
...
-gstat().disk.compaction.input == istat().disk.compaction.input.bytes
+gstat().scheduler.compaction_input == istat().disk.compaction.input.bytes
---
- true
...
-gstat().disk.compaction.output == istat().disk.compaction.output.bytes
+gstat().scheduler.compaction_output == istat().disk.compaction.output.bytes
---
- true
...
@@ -1074,30 +1073,29 @@ istat()
...
gstat()
---
-- disk:
- dump:
- input: 0
- output: 0
- compaction:
- input: 0
- output: 0
- queue: 0
- data: 104300
- index: 1190
+- tx:
+ conflict: 0
+ commit: 0
+ rollback: 0
+ statements: 0
+ transactions: 0
+ gap_locks: 0
+ read_views: 0
memory:
tuple_cache: 14313
tx: 0
level0: 262583
page_index: 1050
bloom_filter: 140
- tx:
- conflict: 0
- commit: 0
- rollback: 0
- statements: 0
- transactions: 0
- gap_locks: 0
- read_views: 0
+ disk:
+ data: 104300
+ index: 1190
+ scheduler:
+ compaction_output: 0
+ compaction_queue: 0
+ dump_output: 0
+ dump_input: 0
+ compaction_input: 0
...
s:drop()
---
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 0173cf89..6708fcb9 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -205,10 +205,10 @@ box.rollback()
--
-- dump and compaction totals
-gstat().disk.dump.input == istat().disk.dump.input.bytes
-gstat().disk.dump.output == istat().disk.dump.output.bytes
-gstat().disk.compaction.input == istat().disk.compaction.input.bytes
-gstat().disk.compaction.output == istat().disk.compaction.output.bytes
+gstat().scheduler.dump_input == istat().disk.dump.input.bytes
+gstat().scheduler.dump_output == istat().disk.dump.output.bytes
+gstat().scheduler.compaction_input == istat().disk.compaction.input.bytes
+gstat().scheduler.compaction_output == istat().disk.compaction.output.bytes
-- use memory
st = gstat()
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 08/12] vinyl: add dump count to global scheduler statistics
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (6 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 09/12] vinyl: don't account secondary indexes to scheduler.dump_input Vladimir Davydov
` (5 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
This patch adds scheduler.dump_count to box.stat.vinyl(), which shows
the number of memory level dumps that have happened since the instance
startup or box.stat.reset(). It's useful for estimating an average size
of a single memory dump, which in turn can be used for calculating LSM
tree fanout.
---
src/box/vinyl.c | 1 +
src/box/vy_scheduler.c | 2 ++
src/box/vy_stat.h | 2 ++
test/vinyl/stat.result | 79 ++++++++++++++++++++++++++++++++++++++++++++++--
test/vinyl/stat.test.lua | 27 +++++++++++++++++
5 files changed, 109 insertions(+), 2 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index b30e69a1..e3478679 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -250,6 +250,7 @@ vy_info_append_scheduler(struct vy_env *env, struct info_handler *h)
struct vy_scheduler_stat *stat = &env->scheduler.stat;
info_table_begin(h, "scheduler");
+ info_append_int(h, "dump_count", stat->dump_count);
info_append_int(h, "dump_input", stat->dump_input);
info_append_int(h, "dump_output", stat->dump_output);
info_append_int(h, "compaction_input", stat->compaction_input);
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index 2d4f6bed..d06da484 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -504,6 +504,7 @@ void
vy_scheduler_reset_stat(struct vy_scheduler *scheduler)
{
struct vy_scheduler_stat *stat = &scheduler->stat;
+ stat->dump_count = 0;
stat->dump_input = 0;
stat->dump_output = 0;
stat->compaction_input = 0;
@@ -696,6 +697,7 @@ vy_scheduler_complete_dump(struct vy_scheduler *scheduler)
double dump_duration = now - scheduler->dump_start;
scheduler->dump_start = now;
scheduler->dump_generation = min_generation;
+ scheduler->stat.dump_count++;
scheduler->dump_complete_cb(scheduler,
min_generation - 1, dump_duration);
fiber_cond_signal(&scheduler->dump_cond);
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index 195b09a4..b976db47 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -212,6 +212,8 @@ struct vy_tx_stat {
* disk compression.
*/
struct vy_scheduler_stat {
+ /** Number of completed memory dumps. */
+ int32_t dump_count;
/** Number of bytes read by dump tasks. */
int64_t dump_input;
/** Number of bytes written by dump tasks. */
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 16f01945..e8780e5b 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -242,8 +242,9 @@ gstat()
data: 0
index: 0
scheduler:
- compaction_output: 0
compaction_queue: 0
+ dump_count: 0
+ compaction_output: 0
dump_output: 0
dump_input: 0
compaction_input: 0
@@ -1091,8 +1092,9 @@ gstat()
data: 104300
index: 1190
scheduler:
- compaction_output: 0
compaction_queue: 0
+ dump_count: 0
+ compaction_output: 0
dump_output: 0
dump_input: 0
compaction_input: 0
@@ -1100,6 +1102,79 @@ gstat()
s:drop()
---
...
+-- sched stats
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+i1 = s:create_index('i1', {parts = {1, 'unsigned'}})
+---
+...
+i2 = s:create_index('i2', {parts = {2, 'unsigned'}})
+---
+...
+for i = 1, 100 do s:replace{i, i, string.rep('x', 1000)} end
+---
+...
+st = gstat()
+---
+...
+box.snapshot()
+---
+- ok
+...
+stat_diff(gstat(), st, 'scheduler')
+---
+- dump_count: 1
+ dump_input: 208400
+ dump_output: 103592
+...
+for i = 1, 100, 10 do s:replace{i, i, string.rep('y', 1000)} end
+---
+...
+st = gstat()
+---
+...
+box.snapshot()
+---
+- ok
+...
+stat_diff(gstat(), st, 'scheduler')
+---
+- dump_count: 1
+ dump_input: 21230
+ dump_output: 10371
+...
+st = gstat()
+---
+...
+i1:compact()
+---
+...
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+---
+...
+stat_diff(gstat(), st, 'scheduler')
+---
+- compaction_input: 112188
+ compaction_output: 101984
+...
+st = gstat()
+---
+...
+i2:compact()
+---
+...
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+---
+...
+stat_diff(gstat(), st, 'scheduler')
+---
+- compaction_input: 1775
+ compaction_output: 1608
+...
+s:drop()
+---
+...
--
-- space.bsize, index.len, index.bsize
--
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 6708fcb9..f2acf3e8 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -320,6 +320,33 @@ gstat()
s:drop()
+-- sched stats
+s = box.schema.space.create('test', {engine = 'vinyl'})
+i1 = s:create_index('i1', {parts = {1, 'unsigned'}})
+i2 = s:create_index('i2', {parts = {2, 'unsigned'}})
+
+for i = 1, 100 do s:replace{i, i, string.rep('x', 1000)} end
+st = gstat()
+box.snapshot()
+stat_diff(gstat(), st, 'scheduler')
+
+for i = 1, 100, 10 do s:replace{i, i, string.rep('y', 1000)} end
+st = gstat()
+box.snapshot()
+stat_diff(gstat(), st, 'scheduler')
+
+st = gstat()
+i1:compact()
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+stat_diff(gstat(), st, 'scheduler')
+
+st = gstat()
+i2:compact()
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+stat_diff(gstat(), st, 'scheduler')
+
+s:drop()
+
--
-- space.bsize, index.len, index.bsize
--
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 09/12] vinyl: don't account secondary indexes to scheduler.dump_input
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (7 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 08/12] vinyl: add dump count to global scheduler statistics Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 10/12] vinyl: add task accounting to global scheduler statistics Vladimir Davydov
` (4 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
Indexes of the same space share a memory level so we should account them
to box.stat.vinyl().scheduler.dump_input once per space dump, not each
time an index is dumped.
---
src/box/vy_scheduler.c | 7 ++++++-
test/vinyl/stat.result | 4 ++--
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index d06da484..e49cd211 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -1269,7 +1269,12 @@ delete_mems:
}
lsm->dump_lsn = MAX(lsm->dump_lsn, dump_lsn);
vy_lsm_acct_dump(lsm, &dump_input, &dump_output);
- scheduler->stat.dump_input += dump_input.bytes;
+ /*
+ * Indexes of the same space share a memory level so we
+ * account dump input only when the primary index is dumped.
+ */
+ if (lsm->index_id == 0)
+ scheduler->stat.dump_input += dump_input.bytes;
scheduler->stat.dump_output += dump_output.bytes;
/* The iterator has been cleaned up in a worker thread. */
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index e8780e5b..3050e43d 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -1125,7 +1125,7 @@ box.snapshot()
stat_diff(gstat(), st, 'scheduler')
---
- dump_count: 1
- dump_input: 208400
+ dump_input: 104200
dump_output: 103592
...
for i = 1, 100, 10 do s:replace{i, i, string.rep('y', 1000)} end
@@ -1141,7 +1141,7 @@ box.snapshot()
stat_diff(gstat(), st, 'scheduler')
---
- dump_count: 1
- dump_input: 21230
+ dump_input: 10420
dump_output: 10371
...
st = gstat()
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 10/12] vinyl: add task accounting to global scheduler statistics
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (8 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 09/12] vinyl: don't account secondary indexes to scheduler.dump_input Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 11/12] vinyl: add dump/compaction time to statistics Vladimir Davydov
` (3 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
This patch adds the following new fields to box.stat.vinyl():
scheduler.tasks_inprogress - number of currently running tasks
scheduler.tasks_completed - number of successfully completed tasks
scheduler.tasks_failed - number of aborted tasks
tasks_failed can be useful for monitoring disk write errors while
tasks_inprogress and tasks_completed can shed light on worker thread
pool effeciency.
---
src/box/vinyl.c | 3 +
src/box/vy_scheduler.c | 21 +++++--
src/box/vy_stat.h | 6 ++
test/vinyl/errinj_stat.result | 119 ++++++++++++++++++++++++++++++++++++++++
test/vinyl/errinj_stat.test.lua | 36 ++++++++++++
test/vinyl/stat.result | 26 ++++++---
6 files changed, 199 insertions(+), 12 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index e3478679..ea80d0ba 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -250,6 +250,9 @@ vy_info_append_scheduler(struct vy_env *env, struct info_handler *h)
struct vy_scheduler_stat *stat = &env->scheduler.stat;
info_table_begin(h, "scheduler");
+ info_append_int(h, "tasks_inprogress", stat->tasks_inprogress);
+ info_append_int(h, "tasks_completed", stat->tasks_completed);
+ info_append_int(h, "tasks_failed", stat->tasks_failed);
info_append_int(h, "dump_count", stat->dump_count);
info_append_int(h, "dump_input", stat->dump_input);
info_append_int(h, "dump_output", stat->dump_output);
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index e49cd211..d0344cdd 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -504,6 +504,8 @@ void
vy_scheduler_reset_stat(struct vy_scheduler *scheduler)
{
struct vy_scheduler_stat *stat = &scheduler->stat;
+ stat->tasks_completed = 0;
+ stat->tasks_failed = 0;
stat->dump_count = 0;
stat->dump_input = 0;
stat->dump_output = 0;
@@ -1965,15 +1967,18 @@ vy_schedule(struct vy_scheduler *scheduler, struct vy_task **ptask)
if (vy_scheduler_peek_dump(scheduler, ptask) != 0)
goto fail;
if (*ptask != NULL)
- return 0;
+ goto found;
if (vy_scheduler_peek_compaction(scheduler, ptask) != 0)
goto fail;
if (*ptask != NULL)
- return 0;
+ goto found;
/* no task to run */
return 0;
+found:
+ scheduler->stat.tasks_inprogress++;
+ return 0;
fail:
assert(!diag_is_empty(diag_get()));
diag_move(diag_get(), &scheduler->diag);
@@ -1984,10 +1989,15 @@ fail:
static int
vy_task_complete(struct vy_task *task)
{
+ struct vy_scheduler *scheduler = task->scheduler;
+
+ assert(scheduler->stat.tasks_inprogress > 0);
+ scheduler->stat.tasks_inprogress--;
+
if (task->lsm->is_dropped) {
if (task->ops->abort)
task->ops->abort(task);
- return 0;
+ goto out;
}
struct diag *diag = &task->diag;
@@ -2006,11 +2016,14 @@ vy_task_complete(struct vy_task *task)
diag_move(diag_get(), diag);
goto fail;
}
+out:
+ scheduler->stat.tasks_completed++;
return 0;
fail:
if (task->ops->abort)
task->ops->abort(task);
- diag_move(diag, &task->scheduler->diag);
+ diag_move(diag, &scheduler->diag);
+ scheduler->stat.tasks_failed++;
return -1;
}
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index b976db47..bd3b811c 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -212,6 +212,12 @@ struct vy_tx_stat {
* disk compression.
*/
struct vy_scheduler_stat {
+ /** Number of completed tasks. */
+ int32_t tasks_completed;
+ /** Number of failed tasks. */
+ int32_t tasks_failed;
+ /** Number of tasks in progress. */
+ int32_t tasks_inprogress;
/** Number of completed memory dumps. */
int32_t dump_count;
/** Number of bytes read by dump tasks. */
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
index fe379cef..9989f734 100644
--- a/test/vinyl/errinj_stat.result
+++ b/test/vinyl/errinj_stat.result
@@ -134,6 +134,125 @@ i:stat().disk.compaction.queue -- none
s:drop()
---
...
+--
+-- Check task statistics.
+--
+box.stat.reset()
+---
+...
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+_ = s:create_index('pk')
+---
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+---
+- ok
+...
+s:replace{1}
+---
+- [1]
+...
+c = fiber.channel(1)
+---
+...
+_ = fiber.create(function() box.snapshot() c:put(true) end)
+---
+...
+fiber.sleep(0.01)
+---
+...
+stat = box.stat.vinyl().scheduler
+---
+...
+stat.tasks_inprogress > 0
+---
+- true
+...
+stat.tasks_completed == 0
+---
+- true
+...
+stat.tasks_failed == 0
+---
+- true
+...
+box.stat.reset() -- doesn't affect tasks_inprogress
+---
+...
+box.stat.vinyl().scheduler.tasks_inprogress > 0
+---
+- true
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+---
+- ok
+...
+c:get()
+---
+- true
+...
+stat = box.stat.vinyl().scheduler
+---
+...
+stat.tasks_inprogress == 0
+---
+- true
+...
+stat.tasks_completed == 1
+---
+- true
+...
+stat.tasks_failed == 0
+---
+- true
+...
+errinj.set('ERRINJ_VY_RUN_WRITE', true)
+---
+- ok
+...
+errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0.01)
+---
+- ok
+...
+s:replace{2}
+---
+- [2]
+...
+box.snapshot()
+---
+- error: Error injection 'vinyl dump'
+...
+stat = box.stat.vinyl().scheduler
+---
+...
+stat.tasks_inprogress == 0
+---
+- true
+...
+stat.tasks_completed == 1
+---
+- true
+...
+stat.tasks_failed > 0
+---
+- true
+...
+errinj.set('ERRINJ_VY_RUN_WRITE', false)
+---
+- ok
+...
+errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0)
+---
+- ok
+...
+fiber.sleep(0.01)
+---
+...
+s:drop()
+---
+...
test_run:cmd("clear filter")
---
- true
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
index 3556f50a..fd80b382 100644
--- a/test/vinyl/errinj_stat.test.lua
+++ b/test/vinyl/errinj_stat.test.lua
@@ -42,6 +42,42 @@ while i:stat().disk.compaction.count < 2 do fiber.sleep(0.01) end
i:stat().disk.compaction.queue -- none
s:drop()
+--
+-- Check task statistics.
+--
+box.stat.reset()
+s = box.schema.space.create('test', {engine = 'vinyl'})
+_ = s:create_index('pk')
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+s:replace{1}
+c = fiber.channel(1)
+_ = fiber.create(function() box.snapshot() c:put(true) end)
+fiber.sleep(0.01)
+stat = box.stat.vinyl().scheduler
+stat.tasks_inprogress > 0
+stat.tasks_completed == 0
+stat.tasks_failed == 0
+box.stat.reset() -- doesn't affect tasks_inprogress
+box.stat.vinyl().scheduler.tasks_inprogress > 0
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+c:get()
+stat = box.stat.vinyl().scheduler
+stat.tasks_inprogress == 0
+stat.tasks_completed == 1
+stat.tasks_failed == 0
+errinj.set('ERRINJ_VY_RUN_WRITE', true)
+errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0.01)
+s:replace{2}
+box.snapshot()
+stat = box.stat.vinyl().scheduler
+stat.tasks_inprogress == 0
+stat.tasks_completed == 1
+stat.tasks_failed > 0
+errinj.set('ERRINJ_VY_RUN_WRITE', false)
+errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0)
+fiber.sleep(0.01)
+s:drop()
+
test_run:cmd("clear filter")
test_run:cmd('switch default')
test_run:cmd('stop server test')
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 3050e43d..6b71d4c5 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -242,10 +242,13 @@ gstat()
data: 0
index: 0
scheduler:
- compaction_queue: 0
- dump_count: 0
compaction_output: 0
+ tasks_inprogress: 0
dump_output: 0
+ tasks_completed: 0
+ dump_count: 0
+ tasks_failed: 0
+ compaction_queue: 0
dump_input: 0
compaction_input: 0
...
@@ -1092,10 +1095,13 @@ gstat()
data: 104300
index: 1190
scheduler:
- compaction_queue: 0
- dump_count: 0
compaction_output: 0
+ tasks_inprogress: 0
dump_output: 0
+ tasks_completed: 0
+ dump_count: 0
+ tasks_failed: 0
+ compaction_queue: 0
dump_input: 0
compaction_input: 0
...
@@ -1124,8 +1130,9 @@ box.snapshot()
...
stat_diff(gstat(), st, 'scheduler')
---
-- dump_count: 1
- dump_input: 104200
+- dump_input: 104200
+ dump_count: 1
+ tasks_completed: 2
dump_output: 103592
...
for i = 1, 100, 10 do s:replace{i, i, string.rep('y', 1000)} end
@@ -1140,8 +1147,9 @@ box.snapshot()
...
stat_diff(gstat(), st, 'scheduler')
---
-- dump_count: 1
- dump_input: 10420
+- dump_input: 10420
+ dump_count: 1
+ tasks_completed: 2
dump_output: 10371
...
st = gstat()
@@ -1156,6 +1164,7 @@ while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
stat_diff(gstat(), st, 'scheduler')
---
- compaction_input: 112188
+ tasks_completed: 1
compaction_output: 101984
...
st = gstat()
@@ -1170,6 +1179,7 @@ while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
stat_diff(gstat(), st, 'scheduler')
---
- compaction_input: 1775
+ tasks_completed: 1
compaction_output: 1608
...
s:drop()
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 11/12] vinyl: add dump/compaction time to statistics
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (9 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 10/12] vinyl: add task accounting to global scheduler statistics Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 12/12] vinyl: add last level size " Vladimir Davydov
` (2 subsequent siblings)
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
This patch adds dump_time and compaction_time to the scheduler section
of global vinyl statistics and disk.dump.time and disk.compaction.time
to per-index statistics. They report the total time spent doing dump and
compaction tasks, respectively and can be useful for estimating average
disk write rate, which is required for compaction-aware throttling.
---
src/box/vinyl.c | 6 ++
src/box/vy_lsm.c | 6 +-
src/box/vy_lsm.h | 4 +-
src/box/vy_scheduler.c | 14 ++++-
src/box/vy_stat.h | 8 +++
test/vinyl/errinj_stat.result | 136 ++++++++++++++++++++++++++++++++++++++++
test/vinyl/errinj_stat.test.lua | 45 +++++++++++++
test/vinyl/stat.result | 58 ++++++++++-------
test/vinyl/stat.test.lua | 10 +++
9 files changed, 257 insertions(+), 30 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index ea80d0ba..4e5903c2 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -254,8 +254,10 @@ vy_info_append_scheduler(struct vy_env *env, struct info_handler *h)
info_append_int(h, "tasks_completed", stat->tasks_completed);
info_append_int(h, "tasks_failed", stat->tasks_failed);
info_append_int(h, "dump_count", stat->dump_count);
+ info_append_double(h, "dump_time", stat->dump_time);
info_append_int(h, "dump_input", stat->dump_input);
info_append_int(h, "dump_output", stat->dump_output);
+ info_append_double(h, "compaction_time", stat->compaction_time);
info_append_int(h, "compaction_input", stat->compaction_input);
info_append_int(h, "compaction_output", stat->compaction_output);
info_append_int(h, "compaction_queue", stat->compaction_queue);
@@ -418,11 +420,13 @@ vinyl_index_stat(struct index *index, struct info_handler *h)
info_table_end(h); /* iterator */
info_table_begin(h, "dump");
info_append_int(h, "count", stat->disk.dump.count);
+ info_append_double(h, "time", stat->disk.dump.time);
vy_info_append_stmt_counter(h, "input", &stat->disk.dump.input);
vy_info_append_disk_stmt_counter(h, "output", &stat->disk.dump.output);
info_table_end(h); /* dump */
info_table_begin(h, "compaction");
info_append_int(h, "count", stat->disk.compaction.count);
+ info_append_double(h, "time", stat->disk.compaction.time);
vy_info_append_disk_stmt_counter(h, "input", &stat->disk.compaction.input);
vy_info_append_disk_stmt_counter(h, "output", &stat->disk.compaction.output);
vy_info_append_disk_stmt_counter(h, "queue", &stat->disk.compaction.queue);
@@ -480,11 +484,13 @@ vinyl_index_reset_stat(struct index *index)
/* Dump */
stat->disk.dump.count = 0;
+ stat->disk.dump.time = 0;
vy_stmt_counter_reset(&stat->disk.dump.input);
vy_disk_stmt_counter_reset(&stat->disk.dump.output);
/* Compaction */
stat->disk.compaction.count = 0;
+ stat->disk.compaction.time = 0;
vy_disk_stmt_counter_reset(&stat->disk.compaction.input);
vy_disk_stmt_counter_reset(&stat->disk.compaction.output);
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index b79c6e0c..07ddc646 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -759,21 +759,23 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
}
void
-vy_lsm_acct_dump(struct vy_lsm *lsm,
+vy_lsm_acct_dump(struct vy_lsm *lsm, double time,
const struct vy_stmt_counter *input,
const struct vy_disk_stmt_counter *output)
{
lsm->stat.disk.dump.count++;
+ lsm->stat.disk.dump.time += time;
vy_stmt_counter_add(&lsm->stat.disk.dump.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.dump.output, output);
}
void
-vy_lsm_acct_compaction(struct vy_lsm *lsm,
+vy_lsm_acct_compaction(struct vy_lsm *lsm, double time,
const struct vy_disk_stmt_counter *input,
const struct vy_disk_stmt_counter *output)
{
lsm->stat.disk.compaction.count++;
+ lsm->stat.disk.compaction.time += time;
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.output, output);
}
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index b486a662..97b567d4 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -468,7 +468,7 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range);
* Account dump in LSM tree statistics.
*/
void
-vy_lsm_acct_dump(struct vy_lsm *lsm,
+vy_lsm_acct_dump(struct vy_lsm *lsm, double time,
const struct vy_stmt_counter *input,
const struct vy_disk_stmt_counter *output);
@@ -476,7 +476,7 @@ vy_lsm_acct_dump(struct vy_lsm *lsm,
* Account compaction in LSM tree statistics.
*/
void
-vy_lsm_acct_compaction(struct vy_lsm *lsm,
+vy_lsm_acct_compaction(struct vy_lsm *lsm, double time,
const struct vy_disk_stmt_counter *input,
const struct vy_disk_stmt_counter *output);
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index d0344cdd..7a143062 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -169,6 +169,8 @@ struct vy_task {
* a worker thread.
*/
struct fiber *fiber;
+ /** Time of the task creation. */
+ double start_time;
/** Set if the task failed. */
bool is_failed;
/** In case of task failure the error is stored here. */
@@ -243,6 +245,7 @@ vy_task_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
task->ops = ops;
task->scheduler = scheduler;
task->worker = worker;
+ task->start_time = ev_monotonic_now(loop());
task->lsm = lsm;
task->cmp_def = key_def_dup(lsm->cmp_def);
if (task->cmp_def == NULL) {
@@ -507,8 +510,10 @@ vy_scheduler_reset_stat(struct vy_scheduler *scheduler)
stat->tasks_completed = 0;
stat->tasks_failed = 0;
stat->dump_count = 0;
+ stat->dump_time = 0;
stat->dump_input = 0;
stat->dump_output = 0;
+ stat->compaction_time = 0;
stat->compaction_input = 0;
stat->compaction_output = 0;
}
@@ -1138,6 +1143,7 @@ vy_task_dump_complete(struct vy_task *task)
struct vy_lsm *lsm = task->lsm;
struct vy_run *new_run = task->new_run;
int64_t dump_lsn = new_run->dump_lsn;
+ double dump_time = ev_monotonic_now(loop()) - task->start_time;
struct vy_disk_stmt_counter dump_output = new_run->count;
struct vy_stmt_counter dump_input;
struct tuple_format *key_format = lsm->env->key_format;
@@ -1270,7 +1276,7 @@ delete_mems:
vy_lsm_delete_mem(lsm, mem);
}
lsm->dump_lsn = MAX(lsm->dump_lsn, dump_lsn);
- vy_lsm_acct_dump(lsm, &dump_input, &dump_output);
+ vy_lsm_acct_dump(lsm, dump_time, &dump_input, &dump_output);
/*
* Indexes of the same space share a memory level so we
* account dump input only when the primary index is dumped.
@@ -1278,6 +1284,7 @@ delete_mems:
if (lsm->index_id == 0)
scheduler->stat.dump_input += dump_input.bytes;
scheduler->stat.dump_output += dump_output.bytes;
+ scheduler->stat.dump_time += dump_time;
/* The iterator has been cleaned up in a worker thread. */
task->wi->iface->close(task->wi);
@@ -1499,6 +1506,7 @@ vy_task_compaction_complete(struct vy_task *task)
struct vy_lsm *lsm = task->lsm;
struct vy_range *range = task->range;
struct vy_run *new_run = task->new_run;
+ double compaction_time = ev_monotonic_now(loop()) - task->start_time;
struct vy_disk_stmt_counter compaction_output = new_run->count;
struct vy_disk_stmt_counter compaction_input;
struct vy_slice *first_slice = task->first_slice;
@@ -1615,10 +1623,12 @@ vy_task_compaction_complete(struct vy_task *task)
range->n_compactions++;
vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
- vy_lsm_acct_compaction(lsm, &compaction_input, &compaction_output);
+ vy_lsm_acct_compaction(lsm, compaction_time,
+ &compaction_input, &compaction_output);
vy_scheduler_acct_lsm(scheduler, lsm);
scheduler->stat.compaction_input += compaction_input.bytes;
scheduler->stat.compaction_output += compaction_output.bytes;
+ scheduler->stat.compaction_time += compaction_time;
/*
* Unaccount unused runs and delete compacted slices.
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index bd3b811c..6f6db6ac 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -147,6 +147,8 @@ struct vy_lsm_stat {
struct {
/* Number of completed tasks. */
int32_t count;
+ /** Time spent on dump tasks, in seconds. */
+ double time;
/** Number of input statements. */
struct vy_stmt_counter input;
/** Number of output statements. */
@@ -156,6 +158,8 @@ struct vy_lsm_stat {
struct {
/* Number of completed tasks. */
int32_t count;
+ /** Time spent on compaction tasks, in seconds. */
+ double time;
/** Number of input statements. */
struct vy_disk_stmt_counter input;
/** Number of output statements. */
@@ -220,10 +224,14 @@ struct vy_scheduler_stat {
int32_t tasks_inprogress;
/** Number of completed memory dumps. */
int32_t dump_count;
+ /** Time spent on dump tasks, in seconds. */
+ double dump_time;
/** Number of bytes read by dump tasks. */
int64_t dump_input;
/** Number of bytes written by dump tasks. */
int64_t dump_output;
+ /** Time spent on compaction tasks, in seconds. */
+ double compaction_time;
/** Number of bytes read by compaction tasks. */
int64_t compaction_input;
/** Number of bytes written by compaction tasks. */
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
index 9989f734..08801dbc 100644
--- a/test/vinyl/errinj_stat.result
+++ b/test/vinyl/errinj_stat.result
@@ -253,6 +253,142 @@ fiber.sleep(0.01)
s:drop()
---
...
+--
+-- Check dump/compaction time accounting.
+--
+box.stat.reset()
+---
+...
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+i = s:create_index('pk')
+---
+...
+i:stat().disk.dump.time == 0
+---
+- true
+...
+i:stat().disk.compaction.time == 0
+---
+- true
+...
+box.stat.vinyl().scheduler.dump_time == 0
+---
+- true
+...
+box.stat.vinyl().scheduler.compaction_time == 0
+---
+- true
+...
+for i = 1, 100 do s:replace{i} end
+---
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+---
+- ok
+...
+start_time = fiber.time()
+---
+...
+c = fiber.channel(1)
+---
+...
+_ = fiber.create(function() box.snapshot() c:put(true) end)
+---
+...
+fiber.sleep(0.1)
+---
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+---
+- ok
+...
+c:get()
+---
+- true
+...
+i:stat().disk.dump.time >= 0.1
+---
+- true
+...
+i:stat().disk.dump.time <= fiber.time() - start_time
+---
+- true
+...
+i:stat().disk.compaction.time == 0
+---
+- true
+...
+box.stat.vinyl().scheduler.dump_time == i:stat().disk.dump.time
+---
+- true
+...
+box.stat.vinyl().scheduler.compaction_time == i:stat().disk.compaction.time
+---
+- true
+...
+for i = 1, 100, 10 do s:replace{i} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+---
+- ok
+...
+start_time = fiber.time()
+---
+...
+i:compact()
+---
+...
+fiber.sleep(0.1)
+---
+...
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+---
+- ok
+...
+while i:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+---
+...
+i:stat().disk.compaction.time >= 0.1
+---
+- true
+...
+i:stat().disk.compaction.time <= fiber.time() - start_time
+---
+- true
+...
+box.stat.vinyl().scheduler.compaction_time == i:stat().disk.compaction.time
+---
+- true
+...
+box.stat.reset()
+---
+...
+i:stat().disk.dump.time == 0
+---
+- true
+...
+i:stat().disk.compaction.time == 0
+---
+- true
+...
+box.stat.vinyl().scheduler.dump_time == 0
+---
+- true
+...
+box.stat.vinyl().scheduler.compaction_time == 0
+---
+- true
+...
+s:drop()
+---
+...
test_run:cmd("clear filter")
---
- true
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
index fd80b382..19f3d119 100644
--- a/test/vinyl/errinj_stat.test.lua
+++ b/test/vinyl/errinj_stat.test.lua
@@ -78,6 +78,51 @@ errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0)
fiber.sleep(0.01)
s:drop()
+--
+-- Check dump/compaction time accounting.
+--
+box.stat.reset()
+s = box.schema.space.create('test', {engine = 'vinyl'})
+i = s:create_index('pk')
+i:stat().disk.dump.time == 0
+i:stat().disk.compaction.time == 0
+box.stat.vinyl().scheduler.dump_time == 0
+box.stat.vinyl().scheduler.compaction_time == 0
+
+for i = 1, 100 do s:replace{i} end
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+start_time = fiber.time()
+c = fiber.channel(1)
+_ = fiber.create(function() box.snapshot() c:put(true) end)
+fiber.sleep(0.1)
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+c:get()
+i:stat().disk.dump.time >= 0.1
+i:stat().disk.dump.time <= fiber.time() - start_time
+i:stat().disk.compaction.time == 0
+box.stat.vinyl().scheduler.dump_time == i:stat().disk.dump.time
+box.stat.vinyl().scheduler.compaction_time == i:stat().disk.compaction.time
+
+for i = 1, 100, 10 do s:replace{i} end
+box.snapshot()
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', true)
+start_time = fiber.time()
+i:compact()
+fiber.sleep(0.1)
+errinj.set('ERRINJ_VY_RUN_WRITE_DELAY', false)
+while i:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+i:stat().disk.compaction.time >= 0.1
+i:stat().disk.compaction.time <= fiber.time() - start_time
+box.stat.vinyl().scheduler.compaction_time == i:stat().disk.compaction.time
+
+box.stat.reset()
+i:stat().disk.dump.time == 0
+i:stat().disk.compaction.time == 0
+box.stat.vinyl().scheduler.dump_time == 0
+box.stat.vinyl().scheduler.compaction_time == 0
+
+s:drop()
+
test_run:cmd("clear filter")
test_run:cmd('switch default')
test_run:cmd('stop server test')
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 6b71d4c5..0920e3da 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -86,9 +86,14 @@ end;
--
-- Note, latency measurement is beyond the scope of this test
-- so we just filter it out.
+--
+-- Filter dump/compaction time as we need error injection to
+-- test them properly.
function istat()
local st = box.space.test.index.pk:stat()
st.latency = nil
+ st.disk.dump.time = nil
+ st.disk.compaction.time = nil
return st
end;
---
@@ -97,9 +102,14 @@ end;
--
-- Note, checking correctness of the load regulator logic is beyond
-- the scope of this test so we just filter out related statistics.
+--
+-- Filter dump/compaction time as we need error injection to
+-- test them properly.
function gstat()
local st = box.stat.vinyl()
st.regulator = nil
+ st.scheduler.dump_time = nil
+ st.scheduler.compaction_time = nil
return st
end;
---
@@ -156,17 +166,17 @@ istat()
pages: 0
rows: 0
bytes: 0
- count: 0
- output:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
queue:
bytes_compressed: 0
pages: 0
rows: 0
bytes: 0
+ output:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
statement:
inserts: 0
replaces: 0
@@ -176,12 +186,12 @@ istat()
input:
rows: 0
bytes: 0
- count: 0
output:
bytes_compressed: 0
pages: 0
rows: 0
bytes: 0
+ count: 0
bloom_size: 0
iterator:
read:
@@ -242,13 +252,13 @@ gstat()
data: 0
index: 0
scheduler:
- compaction_output: 0
tasks_inprogress: 0
dump_output: 0
- tasks_completed: 0
+ compaction_queue: 0
+ compaction_output: 0
dump_count: 0
tasks_failed: 0
- compaction_queue: 0
+ tasks_completed: 0
dump_input: 0
compaction_input: 0
...
@@ -1009,17 +1019,17 @@ istat()
pages: 0
rows: 0
bytes: 0
- count: 0
- output:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
queue:
bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
+ output:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
statement:
inserts: 0
replaces: 100
@@ -1029,12 +1039,12 @@ istat()
input:
rows: 0
bytes: 0
- count: 0
output:
bytes_compressed: <bytes_compressed>
pages: 0
rows: 0
bytes: 0
+ count: 0
bloom_size: 140
iterator:
read:
@@ -1095,13 +1105,13 @@ gstat()
data: 104300
index: 1190
scheduler:
- compaction_output: 0
tasks_inprogress: 0
dump_output: 0
- tasks_completed: 0
+ compaction_queue: 0
+ compaction_output: 0
dump_count: 0
tasks_failed: 0
- compaction_queue: 0
+ tasks_completed: 0
dump_input: 0
compaction_input: 0
...
@@ -1131,9 +1141,9 @@ box.snapshot()
stat_diff(gstat(), st, 'scheduler')
---
- dump_input: 104200
- dump_count: 1
- tasks_completed: 2
dump_output: 103592
+ tasks_completed: 2
+ dump_count: 1
...
for i = 1, 100, 10 do s:replace{i, i, string.rep('y', 1000)} end
---
@@ -1148,9 +1158,9 @@ box.snapshot()
stat_diff(gstat(), st, 'scheduler')
---
- dump_input: 10420
- dump_count: 1
- tasks_completed: 2
dump_output: 10371
+ tasks_completed: 2
+ dump_count: 1
...
st = gstat()
---
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index f2acf3e8..818ec730 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -69,9 +69,14 @@ end;
--
-- Note, latency measurement is beyond the scope of this test
-- so we just filter it out.
+--
+-- Filter dump/compaction time as we need error injection to
+-- test them properly.
function istat()
local st = box.space.test.index.pk:stat()
st.latency = nil
+ st.disk.dump.time = nil
+ st.disk.compaction.time = nil
return st
end;
@@ -79,9 +84,14 @@ end;
--
-- Note, checking correctness of the load regulator logic is beyond
-- the scope of this test so we just filter out related statistics.
+--
+-- Filter dump/compaction time as we need error injection to
+-- test them properly.
function gstat()
local st = box.stat.vinyl()
st.regulator = nil
+ st.scheduler.dump_time = nil
+ st.scheduler.compaction_time = nil
return st
end;
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH 12/12] vinyl: add last level size to statistics
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (10 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 11/12] vinyl: add dump/compaction time to statistics Vladimir Davydov
@ 2019-01-15 14:17 ` Vladimir Davydov
2019-01-17 11:35 ` [tarantool-patches] " Konstantin Osipov
2019-01-17 11:32 ` [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements Konstantin Osipov
2019-01-20 21:16 ` Vladimir Davydov
13 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-15 14:17 UTC (permalink / raw)
To: tarantool-patches
In order to estimate space amplification of a vinyl database, we need to
know the size of data stored at the last LSM tree level. So this patch
adds such a counter both per index and globablly.
Per-index it is reported under disk.last_level, in rows, bytes, bytes
after compression, and pages, just like any other disk counter.
Globablly it is repoted in bytes only under disk.data_compacted. Note,
to be consistent with disk.data, it doesn't include the last level of
secondary indexes.
---
src/box/vinyl.c | 3 +
src/box/vy_lsm.c | 19 ++++
src/box/vy_lsm.h | 13 +++
src/box/vy_stat.h | 2 +
test/vinyl/stat.result | 248 +++++++++++++++++++++++++++++++++++++----------
test/vinyl/stat.test.lua | 42 ++++++++
6 files changed, 276 insertions(+), 51 deletions(-)
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 4e5903c2..01daa435 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -318,6 +318,7 @@ vy_info_append_disk(struct vy_env *env, struct info_handler *h)
info_table_begin(h, "disk");
info_append_int(h, "data", env->lsm_env.disk_data_size);
info_append_int(h, "index", env->lsm_env.disk_index_size);
+ info_append_int(h, "data_compacted", env->lsm_env.compacted_data_size);
info_table_end(h); /* disk */
}
@@ -403,6 +404,8 @@ vinyl_index_stat(struct index *index, struct info_handler *h)
info_table_begin(h, "disk");
vy_info_append_disk_stmt_counter(h, NULL, &stat->disk.count);
+ vy_info_append_disk_stmt_counter(h, "last_level",
+ &stat->disk.last_level_count);
info_table_begin(h, "statement");
info_append_int(h, "inserts", stat->disk.stmt.inserts);
info_append_int(h, "replaces", stat->disk.stmt.replaces);
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index 07ddc646..efaae377 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -239,6 +239,9 @@ vy_lsm_delete(struct vy_lsm *lsm)
assert(lsm->env->lsm_count > 0);
lsm->env->lsm_count--;
+ if (lsm->index_id == 0)
+ lsm->env->compacted_data_size -=
+ lsm->stat.disk.last_level_count.bytes;
if (lsm->pk != NULL)
vy_lsm_unref(lsm->pk);
@@ -748,6 +751,14 @@ vy_lsm_acct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_collect(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
+ if (!rlist_empty(&range->slices)) {
+ struct vy_slice *slice = rlist_last_entry(&range->slices,
+ struct vy_slice, in_range);
+ vy_disk_stmt_counter_add(&lsm->stat.disk.last_level_count,
+ &slice->count);
+ if (lsm->index_id == 0)
+ lsm->env->compacted_data_size += slice->count.bytes;
+ }
}
void
@@ -756,6 +767,14 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_discard(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_sub(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
+ if (!rlist_empty(&range->slices)) {
+ struct vy_slice *slice = rlist_last_entry(&range->slices,
+ struct vy_slice, in_range);
+ vy_disk_stmt_counter_sub(&lsm->stat.disk.last_level_count,
+ &slice->count);
+ if (lsm->index_id == 0)
+ lsm->env->compacted_data_size -= slice->count.bytes;
+ }
}
void
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index 97b567d4..2b779ca0 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -107,6 +107,17 @@ struct vy_lsm_env {
* is consistent with index.bsize().
*/
int64_t disk_index_size;
+ /**
+ * Min size of disk space required to store data of all
+ * spaces of the database. In other words, the size of
+ * disk space the database would occupy if all spaces were
+ * compacted and there were no indexes. Accounted in bytes,
+ * without taking into account disk compression. Estimated
+ * as the size of data stored in the last level of primary
+ * LSM trees. Along with disk_data_size and disk_index_size,
+ * it can be used for evaluating space amplification.
+ */
+ int64_t compacted_data_size;
/** Memory pool for vy_history_node allocations. */
struct mempool history_node_pool;
};
@@ -452,6 +463,8 @@ vy_lsm_remove_range(struct vy_lsm *lsm, struct vy_range *range);
* a range of the LSM tree.
* - vy_lsm::stat::disk::compaction::queue after compaction priority
* of a range is updated.
+ * - vy_lsm::stat::disk::last_level_count and vy_lsm_env::last_level_size
+ * after a range is compacted.
*/
void
vy_lsm_acct_range(struct vy_lsm *lsm, struct vy_range *range);
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index 6f6db6ac..1eb5a461 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -139,6 +139,8 @@ struct vy_lsm_stat {
struct {
/** Number of statements stored on disk. */
struct vy_disk_stmt_counter count;
+ /** Number of statements stored in the last LSM level. */
+ struct vy_disk_stmt_counter last_level_count;
/** Statement statistics. */
struct vy_stmt_stat stmt;
/** Run iterator statistics. */
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 0920e3da..419d3e6c 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -159,24 +159,12 @@ istat()
rows: 0
bytes: 0
disk:
- index_size: 0
- compaction:
- input:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- queue:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- output:
- bytes_compressed: 0
- pages: 0
- rows: 0
- bytes: 0
- count: 0
+ last_level:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ rows: 0
statement:
inserts: 0
replaces: 0
@@ -193,6 +181,7 @@ istat()
bytes: 0
count: 0
bloom_size: 0
+ index_size: 0
iterator:
read:
bytes_compressed: 0
@@ -206,10 +195,26 @@ istat()
get:
rows: 0
bytes: 0
- bytes: 0
+ compaction:
+ input:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ queue:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ output:
+ bytes_compressed: 0
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
pages: 0
bytes_compressed: 0
- rows: 0
+ bytes: 0
txw:
bytes: 0
rows: 0
@@ -249,6 +254,7 @@ gstat()
page_index: 0
bloom_filter: 0
disk:
+ data_compacted: 0
data: 0
index: 0
scheduler:
@@ -292,6 +298,14 @@ stat_diff(istat(), st)
run_avg: 1
run_count: 1
disk:
+ last_level:
+ bytes: 26049
+ pages: 7
+ bytes_compressed: <bytes_compressed>
+ rows: 25
+ rows: 25
+ statement:
+ replaces: 25
dump:
input:
rows: 25
@@ -302,14 +316,11 @@ stat_diff(istat(), st)
pages: 7
bytes_compressed: <bytes_compressed>
rows: 25
+ bytes: 26049
index_size: 294
- rows: 25
- bytes_compressed: <bytes_compressed>
pages: 7
+ bytes_compressed: <bytes_compressed>
bloom_size: 70
- statement:
- replaces: 25
- bytes: 26049
bytes: 26049
put:
rows: 25
@@ -332,6 +343,14 @@ wait(istat, st, 'disk.compaction.count', 1)
stat_diff(istat(), st)
---
- disk:
+ last_level:
+ bytes: 26042
+ pages: 6
+ bytes_compressed: <bytes_compressed>
+ rows: 25
+ rows: 25
+ statement:
+ replaces: 25
dump:
input:
rows: 50
@@ -342,7 +361,10 @@ stat_diff(istat(), st)
pages: 13
bytes_compressed: <bytes_compressed>
rows: 50
+ bytes: 26042
index_size: 252
+ pages: 6
+ bytes_compressed: <bytes_compressed>
compaction:
input:
bytes: 78140
@@ -355,12 +377,6 @@ stat_diff(istat(), st)
pages: 13
bytes_compressed: <bytes_compressed>
rows: 50
- rows: 25
- bytes_compressed: <bytes_compressed>
- pages: 6
- statement:
- replaces: 25
- bytes: 26042
put:
rows: 50
bytes: 53050
@@ -1012,24 +1028,12 @@ istat()
rows: 0
bytes: 0
disk:
- index_size: 1050
- compaction:
- input:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- queue:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- output:
- bytes_compressed: <bytes_compressed>
- pages: 0
- rows: 0
- bytes: 0
- count: 0
+ last_level:
+ bytes_compressed: <bytes_compressed>
+ pages: 25
+ rows: 100
+ bytes: 104300
+ rows: 100
statement:
inserts: 0
replaces: 100
@@ -1046,6 +1050,7 @@ istat()
bytes: 0
count: 0
bloom_size: 140
+ index_size: 1050
iterator:
read:
bytes_compressed: <bytes_compressed>
@@ -1059,10 +1064,26 @@ istat()
get:
rows: 0
bytes: 0
- bytes: 104300
+ compaction:
+ input:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ queue:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ output:
+ bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+ count: 0
pages: 25
bytes_compressed: <bytes_compressed>
- rows: 100
+ bytes: 104300
txw:
bytes: 0
rows: 0
@@ -1102,6 +1123,7 @@ gstat()
page_index: 1050
bloom_filter: 140
disk:
+ data_compacted: 104300
data: 104300
index: 1190
scheduler:
@@ -1557,6 +1579,9 @@ test_run:cmd('restart server test')
fiber = require('fiber')
---
...
+digest = require('digest')
+---
+...
s = box.space.test
---
...
@@ -1586,6 +1611,127 @@ i:stat().disk.statement
s:drop()
---
...
+--
+-- Last level size.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+---
+...
+i1 = s:create_index('i1', {parts = {1, 'unsigned'}})
+---
+...
+i2 = s:create_index('i2', {parts = {2, 'unsigned'}})
+---
+...
+i1:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+...
+i2:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 0
+ rows: 0
+ bytes: 0
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 0
+...
+for i = 1, 100 do s:replace{i, i, digest.urandom(100)} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+i1:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 2
+ rows: 100
+ bytes: 11815
+...
+i2:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 1
+ rows: 100
+ bytes: 1608
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 11815
+...
+for i = 1, 100, 10 do s:replace{i, i * 1000, digest.urandom(100)} end
+---
+...
+box.snapshot()
+---
+- ok
+...
+i1:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 2
+ rows: 100
+ bytes: 11815
+...
+i2:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 1
+ rows: 100
+ bytes: 1608
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 11815
+...
+i1:compact()
+---
+...
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+---
+...
+i1:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 2
+ rows: 100
+ bytes: 11841
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 11841
+...
+i2:compact()
+---
+...
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+---
+...
+i2:stat().disk.last_level
+---
+- bytes_compressed: <bytes_compressed>
+ pages: 1
+ rows: 110
+ bytes: 1794
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 11841
+...
+s:drop()
+---
+...
+box.stat.vinyl().disk.data_compacted
+---
+- 0
+...
test_run:cmd('switch default')
---
- true
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 818ec730..4a955682 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -473,6 +473,7 @@ i:stat().disk.statement
test_run:cmd('restart server test')
fiber = require('fiber')
+digest = require('digest')
s = box.space.test
i = s.index.primary
@@ -486,6 +487,47 @@ i:stat().disk.statement
s:drop()
+--
+-- Last level size.
+--
+s = box.schema.space.create('test', {engine = 'vinyl'})
+i1 = s:create_index('i1', {parts = {1, 'unsigned'}})
+i2 = s:create_index('i2', {parts = {2, 'unsigned'}})
+
+i1:stat().disk.last_level
+i2:stat().disk.last_level
+box.stat.vinyl().disk.data_compacted
+
+for i = 1, 100 do s:replace{i, i, digest.urandom(100)} end
+box.snapshot()
+
+i1:stat().disk.last_level
+i2:stat().disk.last_level
+box.stat.vinyl().disk.data_compacted
+
+for i = 1, 100, 10 do s:replace{i, i * 1000, digest.urandom(100)} end
+box.snapshot()
+
+i1:stat().disk.last_level
+i2:stat().disk.last_level
+box.stat.vinyl().disk.data_compacted
+
+i1:compact()
+while i1:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+
+i1:stat().disk.last_level
+box.stat.vinyl().disk.data_compacted
+
+i2:compact()
+while i2:stat().disk.compaction.count == 0 do fiber.sleep(0.01) end
+
+i2:stat().disk.last_level
+box.stat.vinyl().disk.data_compacted
+
+s:drop()
+
+box.stat.vinyl().disk.data_compacted
+
test_run:cmd('switch default')
test_run:cmd('stop server test')
test_run:cmd('cleanup server test')
--
2.11.0
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler
2019-01-15 14:17 ` [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler Vladimir Davydov
@ 2019-01-16 16:36 ` Vladimir Davydov
0 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-16 16:36 UTC (permalink / raw)
To: tarantool-patches
On Tue, Jan 15, 2019 at 05:17:16PM +0300, Vladimir Davydov wrote:
> +/**
> + * Account an LSM tree to the compaction queue stats.
> + * Called after updating LSM tree compaction priority.
> + */
> +static void
> +vy_scheduler_acct_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
> +{
> + if (lsm->is_dropped)
> + return;
> + scheduler->stat.compaction_queue += lsm->stat.disk.compaction.queue.bytes;
> +}
> +
> +/**
> + * Unaccount an LSM tree from the compaction queue stats.
> + * Called before updating LSM tree compaction priority.
> + */
> +static void
> +vy_scheduler_unacct_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
> +{
> + if (lsm->is_dropped)
> + return;
> + scheduler->stat.compaction_queue -= lsm->stat.disk.compaction.queue.bytes;
> +}
> @@ -1636,8 +1682,11 @@ vy_task_compaction_new(struct vy_scheduler *scheduler, struct vy_worker *worker,
> range = container_of(range_node, struct vy_range, heap_node);
> assert(range->compaction_priority > 1);
>
> - if (vy_lsm_split_range(lsm, range) ||
> - vy_lsm_coalesce_range(lsm, range)) {
> + vy_scheduler_unacct_lsm(scheduler, lsm);
> + bool lsm_changed = (vy_lsm_split_range(lsm, range) ||
> + vy_lsm_coalesce_range(lsm, range));
> + vy_scheduler_acct_lsm(scheduler, lsm);
> + if (lsm_changed) {
> vy_scheduler_update_lsm(scheduler, lsm);
> return 0;
> }
Come to think of it, moving compaction_queue from vy_lsm_env to
vy_scheduler isn't worth the complexity it introduces. Let's leave
it in vy_lsm_env along with disk_data_size, disk_index_size, and
last_level_size, but report it in the 'scheduler' section of
box.info.vinyl(). The new version of the patch goes right below.
It doesn't affect the rest of the series.
---
From 1d23a1b10ac99fa197c443aa1f621535549b1f42 Mon Sep 17 00:00:00 2001
From: Vladimir Davydov <vdavydov.dev@gmail.com>
Date: Mon, 14 Jan 2019 16:49:42 +0300
Subject: [PATCH] vinyl: move global dump/compaction statistics to scheduler
Although it's convenient to maintain dump/compaction input/output
metrics in vy_lsm_env, semantically it's incorrect as those metrics
characterize the scheduler not the LSM environment. Also, we can't
easily extend those stats with e.g. the number of completed dumps or
the number of tasks in progress, because those are only known to the
scheduler.
That said, let's introduce 'scheduler' section in box.stat.vinyl() and
move dump/compaction stats from 'disk' to the new section. Let's also
move the stats accounting from vy_lsm.c to vy_scheduler.c. The 'disk'
section now stores only the size of data and index on disk and no
cumulative statistics, which makes it similar to the 'memory' section.
Note, this patch flattens the stats (disk.compaction.input is moved to
scheduler.compaction_input and so forth), because all other global stats
are reported without using nested tables.
diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 5965700b..a89a1c27 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -245,6 +245,21 @@ static struct trigger on_replace_vinyl_deferred_delete;
/** {{{ Introspection */
static void
+vy_info_append_scheduler(struct vy_env *env, struct info_handler *h)
+{
+ struct vy_scheduler_stat *stat = &env->scheduler.stat;
+
+ info_table_begin(h, "scheduler");
+ info_append_int(h, "dump_input", stat->dump_input);
+ info_append_int(h, "dump_output", stat->dump_output);
+ info_append_int(h, "compaction_input", stat->compaction_input);
+ info_append_int(h, "compaction_output", stat->compaction_output);
+ info_append_int(h, "compaction_queue",
+ env->lsm_env.compaction_queue_size);
+ info_table_end(h); /* scheduler */
+}
+
+static void
vy_info_append_regulator(struct vy_env *env, struct info_handler *h)
{
struct vy_regulator *r = &env->regulator;
@@ -295,24 +310,9 @@ vy_info_append_memory(struct vy_env *env, struct info_handler *h)
static void
vy_info_append_disk(struct vy_env *env, struct info_handler *h)
{
- struct vy_disk_stat *stat = &env->lsm_env.disk_stat;
-
info_table_begin(h, "disk");
-
- info_append_int(h, "data", stat->data);
- info_append_int(h, "index", stat->index);
-
- info_table_begin(h, "dump");
- info_append_int(h, "input", stat->dump.input);
- info_append_int(h, "output", stat->dump.output);
- info_table_end(h); /* dump */
-
- info_table_begin(h, "compaction");
- info_append_int(h, "input", stat->compaction.input);
- info_append_int(h, "output", stat->compaction.output);
- info_append_int(h, "queue", stat->compaction.queue);
- info_table_end(h); /* compaction */
-
+ info_append_int(h, "data", env->lsm_env.disk_data_size);
+ info_append_int(h, "index", env->lsm_env.disk_index_size);
info_table_end(h); /* disk */
}
@@ -325,6 +325,7 @@ vinyl_engine_stat(struct vinyl_engine *vinyl, struct info_handler *h)
vy_info_append_tx(env, h);
vy_info_append_memory(env, h);
vy_info_append_disk(env, h);
+ vy_info_append_scheduler(env, h);
vy_info_append_regulator(env, h);
info_end(h);
}
@@ -514,11 +515,7 @@ vinyl_engine_reset_stat(struct engine *engine)
struct tx_manager *xm = env->xm;
memset(&xm->stat, 0, sizeof(xm->stat));
- struct vy_disk_stat *disk_stat = &env->lsm_env.disk_stat;
- disk_stat->dump.input = 0;
- disk_stat->dump.output = 0;
- disk_stat->compaction.input = 0;
- disk_stat->compaction.output = 0;
+ vy_scheduler_reset_stat(&env->scheduler);
}
/** }}} Introspection */
diff --git a/src/box/vy_lsm.c b/src/box/vy_lsm.c
index abadab5c..2aafe354 100644
--- a/src/box/vy_lsm.c
+++ b/src/box/vy_lsm.c
@@ -239,7 +239,7 @@ vy_lsm_delete(struct vy_lsm *lsm)
assert(lsm->env->lsm_count > 0);
lsm->env->lsm_count--;
- lsm->env->disk_stat.compaction.queue -=
+ lsm->env->compaction_queue_size -=
lsm->stat.disk.compaction.queue.bytes;
if (lsm->pk != NULL)
@@ -691,11 +691,11 @@ vy_lsm_add_run(struct vy_lsm *lsm, struct vy_run *run)
/* Data size is consistent with space.bsize. */
if (lsm->index_id == 0)
- env->disk_stat.data += run->count.bytes;
+ env->disk_data_size += run->count.bytes;
/* Index size is consistent with index.bsize. */
- env->disk_stat.index += bloom_size + page_index_size;
+ env->disk_index_size += bloom_size + page_index_size;
if (lsm->index_id > 0)
- env->disk_stat.index += run->count.bytes;
+ env->disk_index_size += run->count.bytes;
}
void
@@ -720,11 +720,11 @@ vy_lsm_remove_run(struct vy_lsm *lsm, struct vy_run *run)
/* Data size is consistent with space.bsize. */
if (lsm->index_id == 0)
- env->disk_stat.data -= run->count.bytes;
+ env->disk_data_size -= run->count.bytes;
/* Index size is consistent with index.bsize. */
- env->disk_stat.index -= bloom_size + page_index_size;
+ env->disk_index_size -= bloom_size + page_index_size;
if (lsm->index_id > 0)
- env->disk_stat.index -= run->count.bytes;
+ env->disk_index_size -= run->count.bytes;
}
void
@@ -751,7 +751,7 @@ vy_lsm_acct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_collect(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
- lsm->env->disk_stat.compaction.queue += range->compaction_queue.bytes;
+ lsm->env->compaction_queue_size += range->compaction_queue.bytes;
}
void
@@ -760,7 +760,7 @@ vy_lsm_unacct_range(struct vy_lsm *lsm, struct vy_range *range)
histogram_discard(lsm->run_hist, range->slice_count);
vy_disk_stmt_counter_sub(&lsm->stat.disk.compaction.queue,
&range->compaction_queue);
- lsm->env->disk_stat.compaction.queue -= range->compaction_queue.bytes;
+ lsm->env->compaction_queue_size -= range->compaction_queue.bytes;
}
void
@@ -771,9 +771,6 @@ vy_lsm_acct_dump(struct vy_lsm *lsm,
lsm->stat.disk.dump.count++;
vy_stmt_counter_add(&lsm->stat.disk.dump.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.dump.output, output);
-
- lsm->env->disk_stat.dump.input += input->bytes;
- lsm->env->disk_stat.dump.output += output->bytes;
}
void
@@ -784,9 +781,6 @@ vy_lsm_acct_compaction(struct vy_lsm *lsm,
lsm->stat.disk.compaction.count++;
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.input, input);
vy_disk_stmt_counter_add(&lsm->stat.disk.compaction.output, output);
-
- lsm->env->disk_stat.compaction.input += input->bytes;
- lsm->env->disk_stat.compaction.output += output->bytes;
}
int
diff --git a/src/box/vy_lsm.h b/src/box/vy_lsm.h
index e7487995..6b51c5c7 100644
--- a/src/box/vy_lsm.h
+++ b/src/box/vy_lsm.h
@@ -91,8 +91,27 @@ struct vy_lsm_env {
size_t bloom_size;
/** Size of memory used for page index. */
size_t page_index_size;
- /** Global disk statistics. */
- struct vy_disk_stat disk_stat;
+ /**
+ * Size of disk space used for storing data of all spaces,
+ * in bytes, without taking into account disk compression.
+ * By 'data' we mean statements stored in primary indexes
+ * only, which is consistent with space.bsize().
+ */
+ int64_t disk_data_size;
+ /**
+ * Size of disk space used for indexing data in all spaces,
+ * in bytes, without taking into account disk compression.
+ * This consists of page indexes and bloom filters, which
+ * are stored in .index files, as well as the total size of
+ * statements stored in secondary index .run files, which
+ * is consistent with index.bsize().
+ */
+ int64_t disk_index_size;
+ /**
+ * Size of data of all spaces that need to be compacted,
+ * in bytes, without taking into account disk compression.
+ */
+ int64_t compaction_queue_size;
/** Memory pool for vy_history_node allocations. */
struct mempool history_node_pool;
};
diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c
index f431eb24..f65f4b14 100644
--- a/src/box/vy_scheduler.c
+++ b/src/box/vy_scheduler.c
@@ -501,6 +501,16 @@ vy_scheduler_destroy(struct vy_scheduler *scheduler)
}
void
+vy_scheduler_reset_stat(struct vy_scheduler *scheduler)
+{
+ struct vy_scheduler_stat *stat = &scheduler->stat;
+ stat->dump_input = 0;
+ stat->dump_output = 0;
+ stat->compaction_input = 0;
+ stat->compaction_output = 0;
+}
+
+void
vy_scheduler_add_lsm(struct vy_scheduler *scheduler, struct vy_lsm *lsm)
{
assert(!lsm->is_dropped);
@@ -1227,6 +1237,8 @@ delete_mems:
}
lsm->dump_lsn = MAX(lsm->dump_lsn, dump_lsn);
vy_lsm_acct_dump(lsm, &dump_input, &dump_output);
+ scheduler->stat.dump_input += dump_input.bytes;
+ scheduler->stat.dump_output += dump_output.bytes;
/* The iterator has been cleaned up in a worker thread. */
task->wi->iface->close(task->wi);
@@ -1564,6 +1576,8 @@ vy_task_compaction_complete(struct vy_task *task)
vy_range_update_compaction_priority(range, &lsm->opts);
vy_lsm_acct_range(lsm, range);
vy_lsm_acct_compaction(lsm, &compaction_input, &compaction_output);
+ scheduler->stat.compaction_input += compaction_input.bytes;
+ scheduler->stat.compaction_output += compaction_output.bytes;
/*
* Unaccount unused runs and delete compacted slices.
diff --git a/src/box/vy_scheduler.h b/src/box/vy_scheduler.h
index 5b09f964..2d4352d7 100644
--- a/src/box/vy_scheduler.h
+++ b/src/box/vy_scheduler.h
@@ -41,6 +41,7 @@
#define HEAP_FORWARD_DECLARATION
#include "salad/heap.h"
#include "salad/stailq.h"
+#include "vy_stat.h"
#if defined(__cplusplus)
extern "C" {
@@ -139,6 +140,8 @@ struct vy_scheduler {
double dump_start;
/** Signaled on dump round completion. */
struct fiber_cond dump_cond;
+ /** Scheduler statistics. */
+ struct vy_scheduler_stat stat;
/**
* Function called by the scheduler upon dump round
* completion. It is supposed to free memory released
@@ -184,6 +187,12 @@ void
vy_scheduler_destroy(struct vy_scheduler *scheduler);
/**
+ * Reset scheduler statistics (called by box.stat.reset).
+ */
+void
+vy_scheduler_reset_stat(struct vy_scheduler *scheduler);
+
+/**
* Add an LSM tree to scheduler dump/compaction queues.
*/
void
diff --git a/src/box/vy_stat.h b/src/box/vy_stat.h
index 7ed55ff5..62ef2b75 100644
--- a/src/box/vy_stat.h
+++ b/src/box/vy_stat.h
@@ -206,23 +206,20 @@ struct vy_tx_stat {
};
/**
- * Global disk statistics.
+ * Scheduler statistics.
*
- * Fields correspond to those of per LSM tree statistics.
- * All counters are given in bytes, uncompressed.
+ * All byte counters are given without taking into account
+ * disk compression.
*/
-struct vy_disk_stat {
- int64_t data;
- int64_t index;
- struct {
- int64_t input;
- int64_t output;
- } dump;
- struct {
- int64_t input;
- int64_t output;
- int64_t queue;
- } compaction;
+struct vy_scheduler_stat {
+ /** Number of bytes read by dump tasks. */
+ int64_t dump_input;
+ /** Number of bytes written by dump tasks. */
+ int64_t dump_output;
+ /** Number of bytes read by compaction tasks. */
+ int64_t compaction_input;
+ /** Number of bytes written by compaction tasks. */
+ int64_t compaction_output;
};
static inline int
diff --git a/test/vinyl/errinj_stat.result b/test/vinyl/errinj_stat.result
index 9d7f64e9..fe379cef 100644
--- a/test/vinyl/errinj_stat.result
+++ b/test/vinyl/errinj_stat.result
@@ -50,7 +50,7 @@ i:stat().disk.compaction.queue -- none
rows: 0
bytes: 0
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -71,7 +71,7 @@ i:stat().disk.compaction.queue -- 30 statements
rows: 30
bytes: 411
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -85,7 +85,7 @@ i:stat().disk.compaction.queue -- 40 statements
rows: 40
bytes: 548
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -99,7 +99,7 @@ i:stat().disk.compaction.queue -- 50 statements
rows: 50
bytes: 685
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
@@ -113,7 +113,7 @@ i:stat().disk.compaction.queue -- 50 statements
rows: 50
bytes: 685
...
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
---
- true
...
diff --git a/test/vinyl/errinj_stat.test.lua b/test/vinyl/errinj_stat.test.lua
index 6cd95cf9..3556f50a 100644
--- a/test/vinyl/errinj_stat.test.lua
+++ b/test/vinyl/errinj_stat.test.lua
@@ -22,21 +22,21 @@ i = s:create_index('pk', {run_count_per_level = 2})
function dump() for i = 1, 10 do s:replace{i} end box.snapshot() end
dump()
i:stat().disk.compaction.queue -- none
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', true)
dump()
dump()
i:stat().disk.compaction.queue -- 30 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
dump()
i:stat().disk.compaction.queue -- 40 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
dump()
i:stat().disk.compaction.queue -- 50 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
box.stat.reset() -- doesn't affect queue size
i:stat().disk.compaction.queue -- 50 statements
-i:stat().disk.compaction.queue.bytes == box.stat.vinyl().disk.compaction.queue
+i:stat().disk.compaction.queue.bytes == box.stat.vinyl().scheduler.compaction_queue
errinj.set('ERRINJ_VY_COMPACTION_DELAY', false)
while i:stat().disk.compaction.count < 2 do fiber.sleep(0.01) end
i:stat().disk.compaction.queue -- none
diff --git a/test/vinyl/stat.result b/test/vinyl/stat.result
index 68aea30b..16f01945 100644
--- a/test/vinyl/stat.result
+++ b/test/vinyl/stat.result
@@ -224,30 +224,29 @@ istat()
...
gstat()
---
-- disk:
- dump:
- input: 0
- output: 0
- compaction:
- input: 0
- output: 0
- queue: 0
- data: 0
- index: 0
+- tx:
+ conflict: 0
+ commit: 0
+ rollback: 0
+ statements: 0
+ transactions: 0
+ gap_locks: 0
+ read_views: 0
memory:
tuple_cache: 0
tx: 0
level0: 0
page_index: 0
bloom_filter: 0
- tx:
- conflict: 0
- commit: 0
- rollback: 0
- statements: 0
- transactions: 0
- gap_locks: 0
- read_views: 0
+ disk:
+ data: 0
+ index: 0
+ scheduler:
+ compaction_output: 0
+ compaction_queue: 0
+ dump_output: 0
+ dump_input: 0
+ compaction_input: 0
...
--
-- Index statistics.
@@ -701,19 +700,19 @@ box.rollback()
-- Global statistics.
--
-- dump and compaction totals
-gstat().disk.dump.input == istat().disk.dump.input.bytes
+gstat().scheduler.dump_input == istat().disk.dump.input.bytes
---
- true
...
-gstat().disk.dump.output == istat().disk.dump.output.bytes
+gstat().scheduler.dump_output == istat().disk.dump.output.bytes
---
- true
...
-gstat().disk.compaction.input == istat().disk.compaction.input.bytes
+gstat().scheduler.compaction_input == istat().disk.compaction.input.bytes
---
- true
...
-gstat().disk.compaction.output == istat().disk.compaction.output.bytes
+gstat().scheduler.compaction_output == istat().disk.compaction.output.bytes
---
- true
...
@@ -1074,30 +1073,29 @@ istat()
...
gstat()
---
-- disk:
- dump:
- input: 0
- output: 0
- compaction:
- input: 0
- output: 0
- queue: 0
- data: 104300
- index: 1190
+- tx:
+ conflict: 0
+ commit: 0
+ rollback: 0
+ statements: 0
+ transactions: 0
+ gap_locks: 0
+ read_views: 0
memory:
tuple_cache: 14313
tx: 0
level0: 262583
page_index: 1050
bloom_filter: 140
- tx:
- conflict: 0
- commit: 0
- rollback: 0
- statements: 0
- transactions: 0
- gap_locks: 0
- read_views: 0
+ disk:
+ data: 104300
+ index: 1190
+ scheduler:
+ compaction_output: 0
+ compaction_queue: 0
+ dump_output: 0
+ dump_input: 0
+ compaction_input: 0
...
s:drop()
---
diff --git a/test/vinyl/stat.test.lua b/test/vinyl/stat.test.lua
index 0173cf89..6708fcb9 100644
--- a/test/vinyl/stat.test.lua
+++ b/test/vinyl/stat.test.lua
@@ -205,10 +205,10 @@ box.rollback()
--
-- dump and compaction totals
-gstat().disk.dump.input == istat().disk.dump.input.bytes
-gstat().disk.dump.output == istat().disk.dump.output.bytes
-gstat().disk.compaction.input == istat().disk.compaction.input.bytes
-gstat().disk.compaction.output == istat().disk.compaction.output.bytes
+gstat().scheduler.dump_input == istat().disk.dump.input.bytes
+gstat().scheduler.dump_output == istat().disk.dump.output.bytes
+gstat().scheduler.compaction_input == istat().disk.compaction.input.bytes
+gstat().scheduler.compaction_output == istat().disk.compaction.output.bytes
-- use memory
st = gstat()
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (11 preceding siblings ...)
2019-01-15 14:17 ` [PATCH 12/12] vinyl: add last level size " Vladimir Davydov
@ 2019-01-17 11:32 ` Konstantin Osipov
2019-01-17 12:06 ` Vladimir Davydov
2019-01-20 21:16 ` Vladimir Davydov
13 siblings, 1 reply; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:32 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> This patch set adds a few metrics necessary for implementing compaction
> randomization and transaction throttling, but it's useful on its own,
> because it makes box.stat.vinyl() a little bit more useful when it comes
> to performance analysis. Here's an example of box.stat.vinyl() output
> with this patch set applied:
Please write a documentation request which explains the meaning of
these variables. AFAIK these stats are still not described in the
manual. Please try to explain why these statistics are useful, and
how they can be used.
> ---
> - tx:
> conflict: 0
> commit: 1979052
> rollback: 0
> statements: 2
> transactions: 1
> gap_locks: 0
> read_views: 0
> regulator:
let's rename it to rate_limit or rate_limits? Regulator is not
specific enough. What does it regulate?
> dump_bandwidth: 10485760
Without comments even I forget the meaning of these.
> dump_watermark: 20023725
> write_rate: 7085581
> memory:
> tuple_cache: 0
> tx: 2388
> level0: 19394239
> page_index: 4422529
> bloom_filter: 1517177
Good.
> disk:
> data_compacted: 500330587
What's this?
> data: 762493299
> index: 41814873
> scheduler:
> dump_time: 186.61679973663
It's total dump time, the name can be confused with
last dump time.
> tasks_inprogress: 3
> dump_output: 2115930554
> compaction_queue: 213022513
> compaction_output: 4130054964
> compaction_time: 737.99443827965
> dump_count: 136
> tasks_failed: 0
> tasks_completed: 1839
> dump_input: 2061676471
> compaction_input: 5646476938
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 01/12] test: rename vinyl/info to vinyl/stat
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
@ 2019-01-17 11:32 ` Konstantin Osipov
0 siblings, 0 replies; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:32 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> The test was called 'info' in the first place, because back when it was
> introduced vinyl statistics were reported by 'info' method. Today, stats
> are reported by 'stat' so let's rename the test as well to conform.
OK to push.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 02/12] test: split vinyl/errinj
2019-01-15 14:17 ` [PATCH 02/12] test: split vinyl/errinj Vladimir Davydov
@ 2019-01-17 11:33 ` Konstantin Osipov
0 siblings, 0 replies; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:33 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> This test is huge and takes long to complete. Let's move ddl, tx, and
> stat related stuff to separate files.
oK to push.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 03/12] vinyl: rename dump/compact in/out to input/output
2019-01-15 14:17 ` [PATCH 03/12] vinyl: rename dump/compact in/out to input/output Vladimir Davydov
@ 2019-01-17 11:33 ` Konstantin Osipov
0 siblings, 0 replies; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:33 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> 'in' is a reserved keyword in Lua, so using 'in' as a map key was a bad
> decision - one has to access it with [] rather than simply with dot.
> Let's rename 'in' to 'input' and 'out' to 'output' both in the output
> and in the code.
OK to push.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 04/12] vinyl: rename compact to compaction
2019-01-15 14:17 ` [PATCH 04/12] vinyl: rename compact to compaction Vladimir Davydov
@ 2019-01-17 11:34 ` Konstantin Osipov
2019-01-17 12:08 ` Vladimir Davydov
0 siblings, 1 reply; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:34 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> compact_input sounds confusing, because 'compact' works as an adjective
> here. Saving 3 characters per variable/stat name related to compaction
> doesn't justify this. Let's rename 'compact' to 'compaction' both in
> stats and in the code.
Actually, compaction_input also can be confused with the current
compaction queue size. Perhaps we could play with word "total" to
distinguish the current/last amounts and the total amounts in variable
names.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 12/12] vinyl: add last level size to statistics
2019-01-15 14:17 ` [PATCH 12/12] vinyl: add last level size " Vladimir Davydov
@ 2019-01-17 11:35 ` Konstantin Osipov
0 siblings, 0 replies; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 11:35 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> In order to estimate space amplification of a vinyl database, we need to
> know the size of data stored at the last LSM tree level. So this patch
> adds such a counter both per index and globablly.
>
> Per-index it is reported under disk.last_level, in rows, bytes, bytes
> after compression, and pages, just like any other disk counter.
>
> Globablly it is repoted in bytes only under disk.data_compacted. Note,
> to be consistent with disk.data, it doesn't include the last level of
> secondary indexes.
All patches in these series are OK to push, all the work in review
could be done on top of the current patch set.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements
2019-01-17 11:32 ` [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements Konstantin Osipov
@ 2019-01-17 12:06 ` Vladimir Davydov
0 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-17 12:06 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: tarantool-patches
On Thu, Jan 17, 2019 at 02:32:36PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> > This patch set adds a few metrics necessary for implementing compaction
> > randomization and transaction throttling, but it's useful on its own,
> > because it makes box.stat.vinyl() a little bit more useful when it comes
> > to performance analysis. Here's an example of box.stat.vinyl() output
> > with this patch set applied:
>
> Please write a documentation request which explains the meaning of
> these variables. AFAIK these stats are still not described in the
> manual. Please try to explain why these statistics are useful, and
> how they can be used.
There are a lot of changes and they are done by spearate patches so
I'm planning to file a documentation request manually after this patch
set is pushed.
>
> > ---
> > - tx:
> > conflict: 0
> > commit: 1979052
> > rollback: 0
> > statements: 2
> > transactions: 1
> > gap_locks: 0
> > read_views: 0
> > regulator:
>
> let's rename it to rate_limit or rate_limits? Regulator is not
> specific enough. What does it regulate?
Transaction rate. I guess 'rate_limit' name would be somewhat more
straightforward, but the component is called vy_regulator in the code
and I'd like to keep the name 'regulator', because it'd be consistent
with other box.stat.vinyl() sections:
scheduler - schedules dumps and compaction tasks
regulator - regulates transaction rate basing on scheduler progress
iterator - here we will account cumulative iterator statistics
(cache hits/misses, read amplification, etc); this one hasn't been
implemented yet.
Besides, I'm planning to add 'rate_limit' member to this table and
regulator.rate_limit looks better than rate_limit.rate_limit or
rate_limit.value IMO.
>
> > dump_bandwidth: 10485760
> Without comments even I forget the meaning of these.
So we have a documentation for it.
>
> > dump_watermark: 20023725
> > write_rate: 7085581
>
> > memory:
> > tuple_cache: 0
> > tx: 2388
> > level0: 19394239
> > page_index: 4422529
> > bloom_filter: 1517177
>
> Good.
>
> > disk:
> > data_compacted: 500330587
>
> What's this?
Size of disk space (without compression) the database would take if all
spaces were compacted: (data + index) / data_compacted can be used to
estimate space amplification. It is estimated as the size of the last
LSM tree level. Wouldn't know how to name it better: data_unique may be,
or data_stripped, or simply last_level. IMO data_compacted sounds
better.
>
> > data: 762493299
> > index: 41814873
> > scheduler:
> > dump_time: 186.61679973663
>
> It's total dump time, the name can be confused with
> last dump time.
Yep, but reporting the last dump time wouldn't make any sense. I'd like
to avoid total_ prefixes as the names are already long enough.
>
> > tasks_inprogress: 3
> > dump_output: 2115930554
> > compaction_queue: 213022513
> > compaction_output: 4130054964
> > compaction_time: 737.99443827965
> > dump_count: 136
> > tasks_failed: 0
> > tasks_completed: 1839
> > dump_input: 2061676471
> > compaction_input: 5646476938
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [tarantool-patches] Re: [PATCH 04/12] vinyl: rename compact to compaction
2019-01-17 11:34 ` [tarantool-patches] " Konstantin Osipov
@ 2019-01-17 12:08 ` Vladimir Davydov
2019-01-17 13:51 ` Konstantin Osipov
0 siblings, 1 reply; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-17 12:08 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: tarantool-patches
On Thu, Jan 17, 2019 at 02:34:49PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> > compact_input sounds confusing, because 'compact' works as an adjective
> > here. Saving 3 characters per variable/stat name related to compaction
> > doesn't justify this. Let's rename 'compact' to 'compaction' both in
> > stats and in the code.
>
> Actually, compaction_input also can be confused with the current
> compaction queue size. Perhaps we could play with word "total" to
> distinguish the current/last amounts and the total amounts in variable
> names.
The names are long enough as they are so I'd avoid adding total_
prefixes. For queue, we have compaction_queue.
^ permalink raw reply [flat|nested] 24+ messages in thread
* [tarantool-patches] Re: [PATCH 04/12] vinyl: rename compact to compaction
2019-01-17 12:08 ` Vladimir Davydov
@ 2019-01-17 13:51 ` Konstantin Osipov
0 siblings, 0 replies; 24+ messages in thread
From: Konstantin Osipov @ 2019-01-17 13:51 UTC (permalink / raw)
To: tarantool-patches
* Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/17 16:08]:
> On Thu, Jan 17, 2019 at 02:34:49PM +0300, Konstantin Osipov wrote:
> > * Vladimir Davydov <vdavydov.dev@gmail.com> [19/01/15 17:20]:
> > > compact_input sounds confusing, because 'compact' works as an adjective
> > > here. Saving 3 characters per variable/stat name related to compaction
> > > doesn't justify this. Let's rename 'compact' to 'compaction' both in
> > > stats and in the code.
> >
> > Actually, compaction_input also can be confused with the current
> > compaction queue size. Perhaps we could play with word "total" to
> > distinguish the current/last amounts and the total amounts in variable
> > names.
>
> The names are long enough as they are so I'd avoid adding total_
> prefixes. For queue, we have compaction_queue.
How do you define "long enough" ? Perhaps they look long inthe
source code but they are definitely not long when displayed on a
computer screen in box.stat.vinyl() output.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: [PATCH 00/12] vinyl: statistics improvements
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
` (12 preceding siblings ...)
2019-01-17 11:32 ` [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements Konstantin Osipov
@ 2019-01-20 21:16 ` Vladimir Davydov
13 siblings, 0 replies; 24+ messages in thread
From: Vladimir Davydov @ 2019-01-20 21:16 UTC (permalink / raw)
To: tarantool-patches
On Tue, Jan 15, 2019 at 05:17:09PM +0300, Vladimir Davydov wrote:
> Vladimir Davydov (12):
> test: rename vinyl/info to vinyl/stat
> test: split vinyl/errinj
> vinyl: rename dump/compact in/out to input/output
> vinyl: rename compact to compaction
> vinyl: bump range version in vy_range.c
> vinyl: don't add dropped LSM trees to the scheduler during recovery
> vinyl: move global dump/compaction statistics to scheduler
> vinyl: add dump count to global scheduler statistics
> vinyl: don't account secondary indexes to scheduler.dump_input
> vinyl: add task accounting to global scheduler statistics
> vinyl: add dump/compaction time to statistics
> vinyl: add last level size to statistics
Pushed to 2.1 and 1.10.
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2019-01-20 21:16 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-15 14:17 [PATCH 00/12] vinyl: statistics improvements Vladimir Davydov
2019-01-15 14:17 ` [PATCH 01/12] test: rename vinyl/info to vinyl/stat Vladimir Davydov
2019-01-17 11:32 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 02/12] test: split vinyl/errinj Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 03/12] vinyl: rename dump/compact in/out to input/output Vladimir Davydov
2019-01-17 11:33 ` [tarantool-patches] " Konstantin Osipov
2019-01-15 14:17 ` [PATCH 04/12] vinyl: rename compact to compaction Vladimir Davydov
2019-01-17 11:34 ` [tarantool-patches] " Konstantin Osipov
2019-01-17 12:08 ` Vladimir Davydov
2019-01-17 13:51 ` Konstantin Osipov
2019-01-15 14:17 ` [PATCH 05/12] vinyl: bump range version in vy_range.c Vladimir Davydov
2019-01-15 14:17 ` [PATCH 06/12] vinyl: don't add dropped LSM trees to the scheduler during recovery Vladimir Davydov
2019-01-15 14:17 ` [PATCH 07/12] vinyl: move global dump/compaction statistics to scheduler Vladimir Davydov
2019-01-16 16:36 ` Vladimir Davydov
2019-01-15 14:17 ` [PATCH 08/12] vinyl: add dump count to global scheduler statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 09/12] vinyl: don't account secondary indexes to scheduler.dump_input Vladimir Davydov
2019-01-15 14:17 ` [PATCH 10/12] vinyl: add task accounting to global scheduler statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 11/12] vinyl: add dump/compaction time to statistics Vladimir Davydov
2019-01-15 14:17 ` [PATCH 12/12] vinyl: add last level size " Vladimir Davydov
2019-01-17 11:35 ` [tarantool-patches] " Konstantin Osipov
2019-01-17 11:32 ` [tarantool-patches] Re: [PATCH 00/12] vinyl: statistics improvements Konstantin Osipov
2019-01-17 12:06 ` Vladimir Davydov
2019-01-20 21:16 ` Vladimir Davydov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox