* [tarantool-patches] [PATCH v4] vshard module reload
@ 2018-07-30 8:56 AKhatskevich
2018-07-30 8:56 ` [tarantool-patches] [PATCH 1/4] Fix races related to object outdating AKhatskevich
` (4 more replies)
0 siblings, 5 replies; 32+ messages in thread
From: AKhatskevich @ 2018-07-30 8:56 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Issue1: https://github.com/tarantool/vshard/issues/112
Issue2: https://github.com/tarantool/vshard/issues/125
Branch: https://github.com/tarantool/vshard/tree/kh/gh-112-reload-mt-2
This patcheset improves vshard reload mechanism.
Changes since PATCH v2:
- Races related to object outdating are fixed.
- Reloadable fiber is refactored (allow to pass a data to a function).
AKhatskevich (4):
Fix races related to object outdating
Refactor reloadable fiber
tests: separate bootstrap routine to a lua_libs
Introduce storage reload evolution
.travis.yml | 2 +-
rpm/prebuild.sh | 2 +
test/lua_libs/bootstrap.lua | 50 +++++
test/lua_libs/git_util.lua | 51 +++++
test/lua_libs/util.lua | 20 ++
test/rebalancer/box_1_a.lua | 47 +---
test/rebalancer/errinj.result | 2 +-
test/rebalancer/errinj.test.lua | 2 +-
test/rebalancer/rebalancer.result | 4 +-
test/rebalancer/rebalancer.test.lua | 4 +-
test/rebalancer/rebalancer_lock_and_pin.result | 2 +-
test/rebalancer/rebalancer_lock_and_pin.test.lua | 2 +-
test/rebalancer/restart_during_rebalancing.result | 2 +-
.../rebalancer/restart_during_rebalancing.test.lua | 2 +-
test/rebalancer/stress_add_remove_rs.result | 2 +-
test/rebalancer/stress_add_remove_rs.test.lua | 2 +-
.../rebalancer/stress_add_remove_several_rs.result | 2 +-
.../stress_add_remove_several_rs.test.lua | 2 +-
test/rebalancer/suite.ini | 2 +-
test/reload_evolution/storage.result | 245 +++++++++++++++++++++
test/reload_evolution/storage.test.lua | 87 ++++++++
test/reload_evolution/storage_1_a.lua | 48 ++++
test/reload_evolution/storage_1_b.lua | 1 +
test/reload_evolution/storage_2_a.lua | 1 +
test/reload_evolution/storage_2_b.lua | 1 +
test/reload_evolution/suite.ini | 6 +
test/reload_evolution/test.lua | 9 +
test/router/reload.result | 4 +-
test/router/reload.test.lua | 4 +-
test/storage/reload.result | 6 +-
test/storage/reload.test.lua | 6 +-
test/unit/garbage.result | 2 +-
test/unit/garbage.test.lua | 2 +-
test/unit/reload_evolution.result | 45 ++++
test/unit/reload_evolution.test.lua | 18 ++
test/unit/util.result | 20 +-
test/unit/util.test.lua | 12 +-
vshard/replicaset.lua | 30 +--
vshard/router/init.lua | 74 ++++---
vshard/storage/init.lua | 33 ++-
vshard/storage/reload_evolution.lua | 58 +++++
vshard/util.lua | 59 ++---
42 files changed, 798 insertions(+), 175 deletions(-)
create mode 100644 test/lua_libs/bootstrap.lua
create mode 100644 test/lua_libs/git_util.lua
create mode 100644 test/reload_evolution/storage.result
create mode 100644 test/reload_evolution/storage.test.lua
create mode 100755 test/reload_evolution/storage_1_a.lua
create mode 120000 test/reload_evolution/storage_1_b.lua
create mode 120000 test/reload_evolution/storage_2_a.lua
create mode 120000 test/reload_evolution/storage_2_b.lua
create mode 100644 test/reload_evolution/suite.ini
create mode 100644 test/reload_evolution/test.lua
create mode 100644 test/unit/reload_evolution.result
create mode 100644 test/unit/reload_evolution.test.lua
create mode 100644 vshard/storage/reload_evolution.lua
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH 1/4] Fix races related to object outdating
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
@ 2018-07-30 8:56 ` AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 2/4] Refactor reloadable fiber AKhatskevich
` (3 subsequent siblings)
4 siblings, 1 reply; 32+ messages in thread
From: AKhatskevich @ 2018-07-30 8:56 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Reload/reconfigure may replace many of M fields during any yield.
Old objects should not be accessed after they are outdated.
This commit handles such cases within `vshard.router`.
---
vshard/replicaset.lua | 30 ++++++++++++++-----------
vshard/router/init.lua | 58 +++++++++++++++++++++++++++++--------------------
vshard/storage/init.lua | 1 +
3 files changed, 52 insertions(+), 37 deletions(-)
diff --git a/vshard/replicaset.lua b/vshard/replicaset.lua
index 6c8d477..87e26d3 100644
--- a/vshard/replicaset.lua
+++ b/vshard/replicaset.lua
@@ -340,16 +340,13 @@ local function replicaset_tostring(replicaset)
master)
end
-local outdate_replicasets
--
-- Copy netbox connections from old replica objects to new ones
-- and outdate old objects.
-- @param replicasets New replicasets
-- @param old_replicasets Replicasets and replicas to be outdated.
--- @param outdate_delay Number of seconds; delay to outdate
--- old objects.
--
-local function rebind_replicasets(replicasets, old_replicasets, outdate_delay)
+local function rebind_replicasets(replicasets, old_replicasets)
for replicaset_uuid, replicaset in pairs(replicasets) do
local old_replicaset = old_replicasets and
old_replicasets[replicaset_uuid]
@@ -370,9 +367,6 @@ local function rebind_replicasets(replicasets, old_replicasets, outdate_delay)
end
end
end
- if old_replicasets then
- util.async_task(outdate_delay, outdate_replicasets, old_replicasets)
- end
end
--
@@ -453,12 +447,7 @@ for fname, func in pairs(replica_mt.__index) do
outdated_replica_mt.__index[fname] = outdated_warning
end
---
--- Outdate replicaset and replica objects:
--- * Set outdated_metatables.
--- * Remove connections.
---
-outdate_replicasets = function(replicasets)
+local outdate_replicasets_internal = function(replicasets)
for _, replicaset in pairs(replicasets) do
setmetatable(replicaset, outdated_replicaset_mt)
for _, replica in pairs(replicaset.replicas) do
@@ -469,6 +458,20 @@ outdate_replicasets = function(replicasets)
log.info('Old replicaset and replica objects are outdated.')
end
+--
+-- Outdate replicaset and replica objects:
+-- * Set outdated_metatables.
+-- * Remove connections.
+-- @param replicasets Old replicasets to be outdated.
+-- @param outdate_delay Delay in seconds before the outdating.
+--
+local function outdate_replicasets(replicasets, outdate_delay)
+ if replicasets then
+ util.async_task(outdate_delay, outdate_replicasets_internal,
+ replicasets)
+ end
+end
+
--
-- Calculate for each replicaset its etalon bucket count.
-- Iterative algorithm is used to learn the best balance in a
@@ -650,4 +653,5 @@ return {
calculate_etalon_balance = cluster_calculate_etalon_balance,
wait_masters_connect = wait_masters_connect,
rebind_replicasets = rebind_replicasets,
+ outdate_replicasets = outdate_replicasets,
}
diff --git a/vshard/router/init.lua b/vshard/router/init.lua
index 142ddb6..1a0ed2f 100644
--- a/vshard/router/init.lua
+++ b/vshard/router/init.lua
@@ -52,9 +52,14 @@ if not M then
}
end
--- Set a replicaset by container of a bucket.
-local function bucket_set(bucket_id, replicaset)
- assert(replicaset)
+-- Set a bucket to a replicaset.
+local function bucket_set(bucket_id, rs_uuid)
+ local replicaset = M.replicasets[rs_uuid]
+ -- It is technically possible to delete a replicaset at the
+ -- same time when route to the bucket is discovered.
+ if not replicaset then
+ return nil, lerror.vshard(lerror.code.NO_ROUTE_TO_BUCKET, bucket_id)
+ end
local old_replicaset = M.route_map[bucket_id]
if old_replicaset ~= replicaset then
if old_replicaset then
@@ -63,6 +68,7 @@ local function bucket_set(bucket_id, replicaset)
replicaset.bucket_count = replicaset.bucket_count + 1
end
M.route_map[bucket_id] = replicaset
+ return replicaset
end
-- Remove a bucket from the cache.
@@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
log.verbose("Discovering bucket %d", bucket_id)
local last_err = nil
local unreachable_uuid = nil
- for uuid, replicaset in pairs(M.replicasets) do
- local _, err =
- replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
- if err == nil then
- bucket_set(bucket_id, replicaset)
- return replicaset
- elseif err.code ~= lerror.code.WRONG_BUCKET then
- last_err = err
- unreachable_uuid = uuid
+ for uuid, _ in pairs(M.replicasets) do
+ -- Handle reload/reconfigure.
+ replicaset = M.replicasets[uuid]
+ if replicaset then
+ local _, err =
+ replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
+ if err == nil then
+ return bucket_set(bucket_id, replicaset.uuid)
+ elseif err.code ~= lerror.code.WRONG_BUCKET then
+ last_err = err
+ unreachable_uuid = uuid
+ end
end
end
local err = nil
@@ -262,13 +271,13 @@ local function router_call(bucket_id, mode, func, args, opts)
end
end
else
- bucket_set(bucket_id, replicaset)
+ replicaset = bucket_set(bucket_id, replicaset.uuid)
lfiber.yield()
-- Protect against infinite cycle in a
-- case of broken cluster, when a bucket
-- is sent on two replicasets to each
-- other.
- if lfiber.time() <= tend then
+ if replicaset and lfiber.time() <= tend then
goto replicaset_is_found
end
end
@@ -513,27 +522,28 @@ local function router_cfg(cfg)
end
box.cfg(box_cfg)
log.info("Box has been configured")
- M.connection_outdate_delay = cfg.connection_outdate_delay
- M.total_bucket_count = total_bucket_count
- M.collect_lua_garbage = collect_lua_garbage
- M.current_cfg = new_cfg
-- Move connections from an old configuration to a new one.
-- It must be done with no yields to prevent usage both of not
-- fully moved old replicasets, and not fully built new ones.
- lreplicaset.rebind_replicasets(new_replicasets, M.replicasets,
- M.connection_outdate_delay)
- M.replicasets = new_replicasets
+ lreplicaset.rebind_replicasets(new_replicasets, M.replicasets)
-- Now the new replicasets are fully built. Can establish
-- connections and yield.
for _, replicaset in pairs(new_replicasets) do
replicaset:connect_all()
end
+ lreplicaset.wait_masters_connect(new_replicasets)
+ lreplicaset.outdate_replicasets(M.replicasets, cfg.connection_outdate_delay)
+ M.connection_outdate_delay = cfg.connection_outdate_delay
+ M.total_bucket_count = total_bucket_count
+ M.collect_lua_garbage = collect_lua_garbage
+ M.current_cfg = cfg
+ M.replicasets = new_replicasets
-- Update existing route map in-place.
- for bucket, rs in pairs(M.route_map) do
+ local old_route_map = M.route_map
+ M.route_map = {}
+ for bucket, rs in pairs(old_route_map) do
M.route_map[bucket] = M.replicasets[rs.uuid]
end
-
- lreplicaset.wait_masters_connect(new_replicasets)
if M.failover_fiber == nil then
lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
end
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index 07bd00c..c1df0e6 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -1620,6 +1620,7 @@ local function storage_cfg(cfg, this_replica_uuid)
box.once("vshard:storage:1", storage_schema_v1, uri.login, uri.password)
lreplicaset.rebind_replicasets(new_replicasets, M.replicasets)
+ lreplicaset.outdate_replicasets(M.replicasets)
M.replicasets = new_replicasets
M.this_replicaset = this_replicaset
M.this_replica = this_replica
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH 2/4] Refactor reloadable fiber
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
2018-07-30 8:56 ` [tarantool-patches] [PATCH 1/4] Fix races related to object outdating AKhatskevich
@ 2018-07-30 8:56 ` AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 3/4] tests: separate bootstrap routine to a lua_libs AKhatskevich
` (2 subsequent siblings)
4 siblings, 1 reply; 32+ messages in thread
From: AKhatskevich @ 2018-07-30 8:56 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Reloadable fiber changes:
* renamed reloadable_fiber_f -> reloadable_fiber
* reloadable_fiber creates fiber and gives name on its own
* worker name replaced with a fiber name in logs
* added extra data argument, which is passed to a function
---
test/rebalancer/rebalancer.result | 2 +-
test/rebalancer/rebalancer.test.lua | 2 +-
test/router/reload.result | 4 +--
test/router/reload.test.lua | 4 +--
test/storage/reload.result | 6 ++--
test/storage/reload.test.lua | 6 ++--
test/unit/garbage.result | 2 +-
test/unit/garbage.test.lua | 2 +-
test/unit/util.result | 20 ++++---------
test/unit/util.test.lua | 12 ++++----
vshard/router/init.lua | 16 +++++-----
vshard/storage/init.lua | 21 +++++++------
vshard/util.lua | 59 +++++++++++++++++++++----------------
13 files changed, 76 insertions(+), 80 deletions(-)
diff --git a/test/rebalancer/rebalancer.result b/test/rebalancer/rebalancer.result
index 88cbaae..bf9e63b 100644
--- a/test/rebalancer/rebalancer.result
+++ b/test/rebalancer/rebalancer.result
@@ -418,7 +418,7 @@ vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
fiber = require('fiber')
---
...
-while not test_run:grep_log('box_2_a', "Rebalancer has been started") do fiber.sleep(0.1) end
+while not test_run:grep_log('box_2_a', "vshard.rebalancer has been started") do fiber.sleep(0.1) end
---
...
while not test_run:grep_log('box_1_a', "Rebalancer location has changed") do fiber.sleep(0.1) end
diff --git a/test/rebalancer/rebalancer.test.lua b/test/rebalancer/rebalancer.test.lua
index 01f2061..24c2706 100644
--- a/test/rebalancer/rebalancer.test.lua
+++ b/test/rebalancer/rebalancer.test.lua
@@ -195,7 +195,7 @@ switch_rs1_master()
vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
fiber = require('fiber')
-while not test_run:grep_log('box_2_a', "Rebalancer has been started") do fiber.sleep(0.1) end
+while not test_run:grep_log('box_2_a', "vshard.rebalancer has been started") do fiber.sleep(0.1) end
while not test_run:grep_log('box_1_a', "Rebalancer location has changed") do fiber.sleep(0.1) end
--
diff --git a/test/router/reload.result b/test/router/reload.result
index 88122aa..28b5004 100644
--- a/test/router/reload.result
+++ b/test/router/reload.result
@@ -116,10 +116,10 @@ vshard.router.module_version()
check_reloaded()
---
...
-while test_run:grep_log('router_1', 'Failover has been started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('router_1', 'vshard.failover has been started') == nil do fiber.sleep(0.1) end
---
...
-while test_run:grep_log('router_1', 'Discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
+while test_run:grep_log('router_1', 'vshard.discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
---
...
check_reloaded()
diff --git a/test/router/reload.test.lua b/test/router/reload.test.lua
index 01b7163..d15986f 100644
--- a/test/router/reload.test.lua
+++ b/test/router/reload.test.lua
@@ -59,8 +59,8 @@ vshard.router.module_version()
check_reloaded()
-while test_run:grep_log('router_1', 'Failover has been started') == nil do fiber.sleep(0.1) end
-while test_run:grep_log('router_1', 'Discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
+while test_run:grep_log('router_1', 'vshard.failover has been started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('router_1', 'vshard.discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
check_reloaded()
diff --git a/test/storage/reload.result b/test/storage/reload.result
index b91b622..898a18f 100644
--- a/test/storage/reload.result
+++ b/test/storage/reload.result
@@ -113,13 +113,13 @@ vshard.storage.module_version()
check_reloaded()
---
...
-while test_run:grep_log('storage_2_a', 'Garbage collector has been started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('storage_2_a', 'vshard.gc has been started') == nil do fiber.sleep(0.1) end
---
...
-while test_run:grep_log('storage_2_a', 'Recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
+while test_run:grep_log('storage_2_a', 'vshard.recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
---
...
-while test_run:grep_log('storage_2_a', 'Rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
+while test_run:grep_log('storage_2_a', 'vshard.rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
---
...
check_reloaded()
diff --git a/test/storage/reload.test.lua b/test/storage/reload.test.lua
index 9140299..a2e8241 100644
--- a/test/storage/reload.test.lua
+++ b/test/storage/reload.test.lua
@@ -59,9 +59,9 @@ vshard.storage.module_version()
check_reloaded()
-while test_run:grep_log('storage_2_a', 'Garbage collector has been started') == nil do fiber.sleep(0.1) end
-while test_run:grep_log('storage_2_a', 'Recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
-while test_run:grep_log('storage_2_a', 'Rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
+while test_run:grep_log('storage_2_a', 'vshard.gc has been started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('storage_2_a', 'vshard.recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
+while test_run:grep_log('storage_2_a', 'vshard.rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
check_reloaded()
diff --git a/test/unit/garbage.result b/test/unit/garbage.result
index 0da8ee1..a352bd8 100644
--- a/test/unit/garbage.result
+++ b/test/unit/garbage.result
@@ -343,7 +343,7 @@ control.bucket_generation_collected
collect_f = vshard.storage.internal.collect_garbage_f
---
...
-f = fiber.create(collect_f, vshard.storage.module_version())
+f = fiber.create(collect_f)
---
...
fill_spaces_with_garbage()
diff --git a/test/unit/garbage.test.lua b/test/unit/garbage.test.lua
index db7821f..80d37e7 100644
--- a/test/unit/garbage.test.lua
+++ b/test/unit/garbage.test.lua
@@ -149,7 +149,7 @@ control.bucket_generation_collected
-- Test continuous garbage collection via background fiber.
--
collect_f = vshard.storage.internal.collect_garbage_f
-f = fiber.create(collect_f, vshard.storage.module_version())
+f = fiber.create(collect_f)
fill_spaces_with_garbage()
-- Wait until garbage collection is finished.
while #s2:select{} ~= 2 do fiber.sleep(0.1) end
diff --git a/test/unit/util.result b/test/unit/util.result
index 30906d1..56b863e 100644
--- a/test/unit/util.result
+++ b/test/unit/util.result
@@ -34,22 +34,18 @@ function slow_fail() fiber.sleep(0.01) error('Error happened.') end
fake_M.reloadable_function = function () fake_M.reloadable_function = slow_fail; slow_fail() end
---
...
-fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
+fib = util.reloadable_fiber('Worker_name', fake_M, 'reloadable_function')
---
...
-while not test_run:grep_log('default', 'Worker_name: reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
+while not test_run:grep_log('default', 'reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
---
...
fib:cancel()
---
...
-test_run:grep_log('default', 'Worker_name is reloaded, restarting')
+test_run:grep_log('default', 'module is reloaded, restarting')
---
-- Worker_name is reloaded, restarting
-...
-test_run:grep_log('default', 'Worker_name has been started')
----
-- Worker_name has been started
+- module is reloaded, restarting
...
log.info(string.rep('a', 1000))
---
@@ -58,16 +54,12 @@ log.info(string.rep('a', 1000))
fake_M.reloadable_function = function () fiber.sleep(0.01); return true end
---
...
-fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
+fib = util.reloadable_fiber('Worker_name', fake_M, 'reloadable_function')
---
...
-while not test_run:grep_log('default', 'Worker_name is reloaded, restarting') do fiber.sleep(0.01) end
+while not test_run:grep_log('default', 'module is reloaded, restarting') do fiber.sleep(0.01) end
---
...
fib:cancel()
---
...
-test_run:grep_log('default', 'Worker_name has been started', 1000)
----
-- Worker_name has been started
-...
diff --git a/test/unit/util.test.lua b/test/unit/util.test.lua
index 131274c..b26bb51 100644
--- a/test/unit/util.test.lua
+++ b/test/unit/util.test.lua
@@ -14,16 +14,14 @@ function slow_fail() fiber.sleep(0.01) error('Error happened.') end
-- Check autoreload on function change during failure.
fake_M.reloadable_function = function () fake_M.reloadable_function = slow_fail; slow_fail() end
-fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
-while not test_run:grep_log('default', 'Worker_name: reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
+fib = util.reloadable_fiber('Worker_name', fake_M, 'reloadable_function')
+while not test_run:grep_log('default', 'reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
fib:cancel()
-test_run:grep_log('default', 'Worker_name is reloaded, restarting')
-test_run:grep_log('default', 'Worker_name has been started')
+test_run:grep_log('default', 'module is reloaded, restarting')
log.info(string.rep('a', 1000))
-- Check reload feature.
fake_M.reloadable_function = function () fiber.sleep(0.01); return true end
-fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
-while not test_run:grep_log('default', 'Worker_name is reloaded, restarting') do fiber.sleep(0.01) end
+fib = util.reloadable_fiber('Worker_name', fake_M, 'reloadable_function')
+while not test_run:grep_log('default', 'module is reloaded, restarting') do fiber.sleep(0.01) end
fib:cancel()
-test_run:grep_log('default', 'Worker_name has been started', 1000)
diff --git a/vshard/router/init.lua b/vshard/router/init.lua
index 1a0ed2f..2df3446 100644
--- a/vshard/router/init.lua
+++ b/vshard/router/init.lua
@@ -149,9 +149,8 @@ end
-- Background fiber to perform discovery. It periodically scans
-- replicasets one by one and updates route_map.
--
-local function discovery_f(module_version)
- lfiber.name('discovery_fiber')
- M.discovery_fiber = lfiber.self()
+local function discovery_f()
+ local module_version = M.module_version
local iterations_until_lua_gc =
consts.COLLECT_LUA_GARBAGE_INTERVAL / consts.DISCOVERY_INTERVAL
while module_version == M.module_version do
@@ -459,9 +458,8 @@ end
-- tries to reconnect to the best replica. When the connection is
-- established, it replaces the original replica.
--
-local function failover_f(module_version)
- lfiber.name('vshard.failover')
- M.failover_fiber = lfiber.self()
+local function failover_f()
+ local module_version = M.module_version
local min_timeout = math.min(consts.FAILOVER_UP_TIMEOUT,
consts.FAILOVER_DOWN_TIMEOUT)
-- This flag is used to avoid logging like:
@@ -545,10 +543,12 @@ local function router_cfg(cfg)
M.route_map[bucket] = M.replicasets[rs.uuid]
end
if M.failover_fiber == nil then
- lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
+ M.failover_fiber = util.reloadable_fiber('vshard.failover', M,
+ 'failover_f')
end
if M.discovery_fiber == nil then
- lfiber.create(util.reloadable_fiber_f, M, 'discovery_f', 'Discovery')
+ M.discovery_fiber = util.reloadable_fiber('vshard.discovery', M,
+ 'discovery_f')
end
-- Destroy connections, not used in a new configuration.
collectgarbage()
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index c1df0e6..8ca81f6 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -354,8 +354,8 @@ end
-- appears to be changed, then stop recovery. It is
-- restarted in reloadable_fiber_f().
--
-local function recovery_f(module_version)
- lfiber.name('vshard.recovery')
+local function recovery_f()
+ local module_version = M.module_version
local _bucket = box.space._bucket
M.buckets_to_recovery = {}
for _, bucket in _bucket.index.status:pairs({consts.BUCKET.SENDING}) do
@@ -728,10 +728,9 @@ local function local_on_master_enable()
M._on_master_enable:run()
-- Start background process to collect garbage.
M.collect_bucket_garbage_fiber =
- lfiber.create(util.reloadable_fiber_f, M, 'collect_garbage_f',
- 'Garbage collector')
+ util.reloadable_fiber('vshard.gc', M, 'collect_garbage_f')
M.recovery_fiber =
- lfiber.create(util.reloadable_fiber_f, M, 'recovery_f', 'Recovery')
+ util.reloadable_fiber('vshard.recovery', M, 'recovery_f')
-- TODO: check current status
log.info("Took on replicaset master role")
end
@@ -1026,8 +1025,8 @@ end
-- appears to be changed, then stop GC. It is restarted
-- in reloadable_fiber_f().
--
-function collect_garbage_f(module_version)
- lfiber.name('vshard.gc')
+function collect_garbage_f()
+ local module_version = M.module_version
-- Collector controller. Changes of _bucket increments
-- bucket generation. Garbage collector has its own bucket
-- generation which is <= actual. Garbage collection is
@@ -1351,8 +1350,8 @@ end
-- Background rebalancer. Works on a storage which has the
-- smallest replicaset uuid and which is master.
--
-local function rebalancer_f(module_version)
- lfiber.name('vshard.rebalancer')
+local function rebalancer_f()
+ local module_version = M.module_version
while module_version == M.module_version do
while not M.is_rebalancer_active do
log.info('Rebalancer is disabled. Sleep')
@@ -1642,8 +1641,8 @@ local function storage_cfg(cfg, this_replica_uuid)
if min_master == this_replica then
if not M.rebalancer_fiber then
- M.rebalancer_fiber = lfiber.create(util.reloadable_fiber_f, M,
- 'rebalancer_f', 'Rebalancer')
+ M.rebalancer_fiber = util.reloadable_fiber('vshard.rebalancer', M,
+ 'rebalancer_f')
else
log.info('Wakeup rebalancer')
-- Configuration had changed. Time to rebalance.
diff --git a/vshard/util.lua b/vshard/util.lua
index fb875ce..1319acc 100644
--- a/vshard/util.lua
+++ b/vshard/util.lua
@@ -10,7 +10,7 @@ if not M then
--
M = {
-- Latest versions of functions.
- reloadable_fiber_f = nil,
+ reloadable_fiber_main_loop = nil,
}
rawset(_G, MODULE_INTERNALS, M)
end
@@ -31,6 +31,30 @@ local function tuple_extract_key(tuple, parts)
return key
end
+local function reloadable_fiber_main_loop(module, func_name, data)
+ local func = module[func_name]
+::restart_loop::
+ local ok, err = pcall(func, data)
+ -- yield serves two purposes:
+ -- * makes this fiber cancellable
+ -- * prevents 100% cpu consumption
+ fiber.yield()
+ if not ok then
+ log.error('%s has been failed: %s', func_name, err)
+ if func == module[func_name] then
+ goto restart_loop
+ end
+ -- There is a chance that error was raised during reload
+ -- (or caused by reload). Perform reload in case function
+ -- has been changed.
+ log.error('reloadable function %s has been changed', func_name)
+ end
+ log.info('module is reloaded, restarting')
+ -- luajit drops this frame if next function is called in
+ -- return statement.
+ return M.reloadable_fiber_main_loop(module, func_name, data)
+end
+
--
-- Wrapper to run a func in infinite loop and restart it on
-- errors and module reload.
@@ -44,30 +68,13 @@ end
-- For example: "Garbage Collector", "Recovery", "Discovery",
-- "Rebalancer". Used only for an activity logging.
--
-local function reloadable_fiber_f(module, func_name, worker_name)
+local function reloadable_fiber(worker_name, module, func_name, data)
+ assert(type(worker_name) == 'string')
+ local xfiber = fiber.create(reloadable_fiber_main_loop, module, func_name,
+ data)
log.info('%s has been started', worker_name)
- local func = module[func_name]
-::reload_loop::
- local ok, err = pcall(func, module.module_version)
- -- yield serves two pursoses:
- -- * makes this fiber cancellable
- -- * prevents 100% cpu consumption
- fiber.yield()
- if not ok then
- log.error('%s has been failed: %s', worker_name, err)
- if func == module[func_name] then
- goto reload_loop
- end
- -- There is a chance that error was raised during reload
- -- (or caused by reload). Perform reload in case function
- -- has been changed.
- log.error('%s: reloadable function %s has been changed',
- worker_name, func_name)
- end
- log.info('%s is reloaded, restarting', worker_name)
- -- luajit drops this frame if next function is called in
- -- return statement.
- return M.reloadable_fiber_f(module, func_name, worker_name)
+ xfiber:name(worker_name)
+ return xfiber
end
--
@@ -98,7 +105,7 @@ local function generate_self_checker(obj_name, func_name, mt, func)
end
-- Update latest versions of function
-M.reloadable_fiber_f = reloadable_fiber_f
+M.reloadable_fiber_main_loop = reloadable_fiber_main_loop
local function sync_task(delay, task, ...)
if delay then
@@ -121,7 +128,7 @@ end
return {
tuple_extract_key = tuple_extract_key,
- reloadable_fiber_f = reloadable_fiber_f,
+ reloadable_fiber = reloadable_fiber,
generate_self_checker = generate_self_checker,
async_task = async_task,
internal = M,
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH 3/4] tests: separate bootstrap routine to a lua_libs
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
2018-07-30 8:56 ` [tarantool-patches] [PATCH 1/4] Fix races related to object outdating AKhatskevich
2018-07-30 8:56 ` [tarantool-patches] [PATCH 2/4] Refactor reloadable fiber AKhatskevich
@ 2018-07-30 8:56 ` AKhatskevich
2018-08-01 12:03 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution AKhatskevich
2018-08-01 14:07 ` [tarantool-patches] [PATCH] Check self arg passed for router objects AKhatskevich
4 siblings, 1 reply; 32+ messages in thread
From: AKhatskevich @ 2018-07-30 8:56 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
What is moved to `test/lua_libs/bootstrap.lua`:
1. create schema
2. create main stored procedures
3. `wait_rebalancer_state` procedure
This code will be reused in further commits.
---
test/lua_libs/bootstrap.lua | 50 ++++++++++++++++++++++
test/rebalancer/box_1_a.lua | 47 ++------------------
test/rebalancer/errinj.result | 2 +-
test/rebalancer/errinj.test.lua | 2 +-
test/rebalancer/rebalancer.result | 2 +-
test/rebalancer/rebalancer.test.lua | 2 +-
test/rebalancer/rebalancer_lock_and_pin.result | 2 +-
test/rebalancer/rebalancer_lock_and_pin.test.lua | 2 +-
test/rebalancer/restart_during_rebalancing.result | 2 +-
.../rebalancer/restart_during_rebalancing.test.lua | 2 +-
test/rebalancer/stress_add_remove_rs.result | 2 +-
test/rebalancer/stress_add_remove_rs.test.lua | 2 +-
.../rebalancer/stress_add_remove_several_rs.result | 2 +-
.../stress_add_remove_several_rs.test.lua | 2 +-
test/rebalancer/suite.ini | 2 +-
15 files changed, 66 insertions(+), 57 deletions(-)
create mode 100644 test/lua_libs/bootstrap.lua
diff --git a/test/lua_libs/bootstrap.lua b/test/lua_libs/bootstrap.lua
new file mode 100644
index 0000000..62c2f78
--- /dev/null
+++ b/test/lua_libs/bootstrap.lua
@@ -0,0 +1,50 @@
+local log = require('log')
+
+function init_schema()
+ local format = {}
+ format[1] = {name = 'field', type = 'unsigned'}
+ format[2] = {name = 'bucket_id', type = 'unsigned'}
+ local s = box.schema.create_space('test', {format = format})
+ local pk = s:create_index('pk')
+ local bucket_id_idx =
+ s:create_index('vbucket', {parts = {'bucket_id'},
+ unique = false})
+end
+
+box.once('schema', function()
+ box.schema.func.create('do_replace')
+ box.schema.role.grant('public', 'execute', 'function', 'do_replace')
+ box.schema.func.create('do_select')
+ box.schema.role.grant('public', 'execute', 'function', 'do_select')
+ init_schema()
+end)
+
+function do_replace(...)
+ box.space.test:replace(...)
+ return true
+end
+
+function do_select(...)
+ return box.space.test:select(...)
+end
+
+function check_consistency()
+ for _, tuple in box.space.test:pairs() do
+ assert(box.space._bucket:get{tuple.bucket_id})
+ end
+ return true
+end
+
+--
+-- Wait a specified log message.
+-- Requirements:
+-- * Should be executed from a storage with a rebalancer.
+-- * NAME - global variable, name of instance should be set.
+function wait_rebalancer_state(state, test_run)
+ log.info(string.rep('a', 1000))
+ vshard.storage.rebalancer_wakeup()
+ while not test_run:grep_log(NAME, state, 1000) do
+ fiber.sleep(0.1)
+ vshard.storage.rebalancer_wakeup()
+ end
+end
diff --git a/test/rebalancer/box_1_a.lua b/test/rebalancer/box_1_a.lua
index 8fddcf0..2ca8306 100644
--- a/test/rebalancer/box_1_a.lua
+++ b/test/rebalancer/box_1_a.lua
@@ -2,7 +2,7 @@
-- Get instance name
require('strict').on()
local fio = require('fio')
-local NAME = fio.basename(arg[0], '.lua')
+NAME = fio.basename(arg[0], '.lua')
log = require('log')
require('console').listen(os.getenv('ADMIN'))
fiber = require('fiber')
@@ -23,40 +23,8 @@ if NAME == 'box_4_a' or NAME == 'box_4_b' or
end
vshard.storage.cfg(cfg, names.replica_uuid[NAME])
-function init_schema()
- local format = {}
- format[1] = {name = 'field', type = 'unsigned'}
- format[2] = {name = 'bucket_id', type = 'unsigned'}
- local s = box.schema.create_space('test', {format = format})
- local pk = s:create_index('pk')
- local bucket_id_idx =
- s:create_index('vbucket', {parts = {'bucket_id'},
- unique = false})
-end
-
-box.once('schema', function()
- box.schema.func.create('do_replace')
- box.schema.role.grant('public', 'execute', 'function', 'do_replace')
- box.schema.func.create('do_select')
- box.schema.role.grant('public', 'execute', 'function', 'do_select')
- init_schema()
-end)
-
-function do_replace(...)
- box.space.test:replace(...)
- return true
-end
-
-function do_select(...)
- return box.space.test:select(...)
-end
-
-function check_consistency()
- for _, tuple in box.space.test:pairs() do
- assert(box.space._bucket:get{tuple.bucket_id})
- end
- return true
-end
+-- Bootstrap storage.
+require('lua_libs.bootstrap')
function switch_rs1_master()
local replica_uuid = names.replica_uuid
@@ -68,12 +36,3 @@ end
function nullify_rs_weight()
cfg.sharding[names.rs_uuid[1]].weight = 0
end
-
-function wait_rebalancer_state(state, test_run)
- log.info(string.rep('a', 1000))
- vshard.storage.rebalancer_wakeup()
- while not test_run:grep_log(NAME, state, 1000) do
- fiber.sleep(0.1)
- vshard.storage.rebalancer_wakeup()
- end
-end
diff --git a/test/rebalancer/errinj.result b/test/rebalancer/errinj.result
index d09349e..826c2c6 100644
--- a/test/rebalancer/errinj.result
+++ b/test/rebalancer/errinj.result
@@ -13,7 +13,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
diff --git a/test/rebalancer/errinj.test.lua b/test/rebalancer/errinj.test.lua
index d6a2920..fc0730c 100644
--- a/test/rebalancer/errinj.test.lua
+++ b/test/rebalancer/errinj.test.lua
@@ -5,7 +5,7 @@ REPLICASET_2 = { 'box_2_a', 'box_2_b' }
test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
util.wait_master(test_run, REPLICASET_2, 'box_2_a')
diff --git a/test/rebalancer/rebalancer.result b/test/rebalancer/rebalancer.result
index bf9e63b..8d9f5e4 100644
--- a/test/rebalancer/rebalancer.result
+++ b/test/rebalancer/rebalancer.result
@@ -13,7 +13,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
diff --git a/test/rebalancer/rebalancer.test.lua b/test/rebalancer/rebalancer.test.lua
index 24c2706..669b5a1 100644
--- a/test/rebalancer/rebalancer.test.lua
+++ b/test/rebalancer/rebalancer.test.lua
@@ -5,7 +5,7 @@ REPLICASET_2 = { 'box_2_a', 'box_2_b' }
test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
util.wait_master(test_run, REPLICASET_2, 'box_2_a')
diff --git a/test/rebalancer/rebalancer_lock_and_pin.result b/test/rebalancer/rebalancer_lock_and_pin.result
index dd9fe47..0f2921c 100644
--- a/test/rebalancer/rebalancer_lock_and_pin.result
+++ b/test/rebalancer/rebalancer_lock_and_pin.result
@@ -16,7 +16,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
diff --git a/test/rebalancer/rebalancer_lock_and_pin.test.lua b/test/rebalancer/rebalancer_lock_and_pin.test.lua
index fe866c4..3a2daa0 100644
--- a/test/rebalancer/rebalancer_lock_and_pin.test.lua
+++ b/test/rebalancer/rebalancer_lock_and_pin.test.lua
@@ -6,7 +6,7 @@ REPLICASET_3 = { 'box_3_a', 'box_3_b' }
test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
util.wait_master(test_run, REPLICASET_2, 'box_2_a')
diff --git a/test/rebalancer/restart_during_rebalancing.result b/test/rebalancer/restart_during_rebalancing.result
index d2b8a12..0eb0f2e 100644
--- a/test/rebalancer/restart_during_rebalancing.result
+++ b/test/rebalancer/restart_during_rebalancing.result
@@ -25,7 +25,7 @@ test_run:create_cluster(REPLICASET_3, 'rebalancer')
test_run:create_cluster(REPLICASET_4, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'fullbox_1_a')
diff --git a/test/rebalancer/restart_during_rebalancing.test.lua b/test/rebalancer/restart_during_rebalancing.test.lua
index 5b1a8df..7b707ca 100644
--- a/test/rebalancer/restart_during_rebalancing.test.lua
+++ b/test/rebalancer/restart_during_rebalancing.test.lua
@@ -9,7 +9,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
test_run:create_cluster(REPLICASET_3, 'rebalancer')
test_run:create_cluster(REPLICASET_4, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'fullbox_1_a')
util.wait_master(test_run, REPLICASET_2, 'fullbox_2_a')
util.wait_master(test_run, REPLICASET_3, 'fullbox_3_a')
diff --git a/test/rebalancer/stress_add_remove_rs.result b/test/rebalancer/stress_add_remove_rs.result
index 8a955e2..10bcaac 100644
--- a/test/rebalancer/stress_add_remove_rs.result
+++ b/test/rebalancer/stress_add_remove_rs.result
@@ -16,7 +16,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
diff --git a/test/rebalancer/stress_add_remove_rs.test.lua b/test/rebalancer/stress_add_remove_rs.test.lua
index c80df40..b9bb027 100644
--- a/test/rebalancer/stress_add_remove_rs.test.lua
+++ b/test/rebalancer/stress_add_remove_rs.test.lua
@@ -6,7 +6,7 @@ REPLICASET_3 = { 'box_3_a', 'box_3_b' }
test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
util.wait_master(test_run, REPLICASET_2, 'box_2_a')
diff --git a/test/rebalancer/stress_add_remove_several_rs.result b/test/rebalancer/stress_add_remove_several_rs.result
index d6008b8..611362c 100644
--- a/test/rebalancer/stress_add_remove_several_rs.result
+++ b/test/rebalancer/stress_add_remove_several_rs.result
@@ -19,7 +19,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
---
...
-util = require('util')
+util = require('lua_libs.util')
---
...
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
diff --git a/test/rebalancer/stress_add_remove_several_rs.test.lua b/test/rebalancer/stress_add_remove_several_rs.test.lua
index 3cc105e..9acb8de 100644
--- a/test/rebalancer/stress_add_remove_several_rs.test.lua
+++ b/test/rebalancer/stress_add_remove_several_rs.test.lua
@@ -7,7 +7,7 @@ REPLICASET_4 = { 'box_4_a', 'box_4_b' }
test_run:create_cluster(REPLICASET_1, 'rebalancer')
test_run:create_cluster(REPLICASET_2, 'rebalancer')
-util = require('util')
+util = require('lua_libs.util')
util.wait_master(test_run, REPLICASET_1, 'box_1_a')
util.wait_master(test_run, REPLICASET_2, 'box_2_a')
diff --git a/test/rebalancer/suite.ini b/test/rebalancer/suite.ini
index afc5141..8689da5 100644
--- a/test/rebalancer/suite.ini
+++ b/test/rebalancer/suite.ini
@@ -4,6 +4,6 @@ description = Rebalancer tests
script = test.lua
is_parallel = False
release_disabled = errinj.test.lua
-lua_libs = ../lua_libs/util.lua config.lua names.lua router_1.lua
+lua_libs = ../lua_libs config.lua names.lua router_1.lua
box_1_a.lua box_1_b.lua box_2_a.lua box_2_b.lua
box_3_a.lua box_3_b.lua rebalancer_utils.lua
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
` (2 preceding siblings ...)
2018-07-30 8:56 ` [tarantool-patches] [PATCH 3/4] tests: separate bootstrap routine to a lua_libs AKhatskevich
@ 2018-07-30 8:56 ` AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-08-01 14:07 ` [tarantool-patches] [PATCH] Check self arg passed for router objects AKhatskevich
4 siblings, 1 reply; 32+ messages in thread
From: AKhatskevich @ 2018-07-30 8:56 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Changes:
1. Introduce storage reload evolution.
2. Setup cross-version reload testing.
1:
This mechanism updates Lua objects on reload in case they are
changed in a new vshard.storage version.
Since this commit, any change in vshard.storage.M has to be
reflected in vshard.storage.reload_evolution to guarantee
correct reload.
2:
The testing uses git infrastructure and is performed in the following
way:
1. Copy old version of vshard to a temp folder.
2. Run vshard on this code.
3. Checkout the latest version of the vshard sources.
4. Reload vshard storage.
5. Make sure it works (Perform simple tests).
Notes:
* this patch contains some legacy-driven decisions:
1. SOURCEDIR path retrieved differently in case of
packpack build.
2. git directory in the `reload_evolution/storage` test
is copied with respect to Centos 7 and `ro` mode of
SOURCEDIR.
Closes #112 #125
---
.travis.yml | 2 +-
rpm/prebuild.sh | 2 +
test/lua_libs/git_util.lua | 51 +++++++
test/lua_libs/util.lua | 20 +++
test/reload_evolution/storage.result | 245 +++++++++++++++++++++++++++++++++
test/reload_evolution/storage.test.lua | 87 ++++++++++++
test/reload_evolution/storage_1_a.lua | 48 +++++++
test/reload_evolution/storage_1_b.lua | 1 +
test/reload_evolution/storage_2_a.lua | 1 +
test/reload_evolution/storage_2_b.lua | 1 +
test/reload_evolution/suite.ini | 6 +
test/reload_evolution/test.lua | 9 ++
test/unit/reload_evolution.result | 45 ++++++
test/unit/reload_evolution.test.lua | 18 +++
vshard/storage/init.lua | 11 ++
vshard/storage/reload_evolution.lua | 58 ++++++++
16 files changed, 604 insertions(+), 1 deletion(-)
create mode 100644 test/lua_libs/git_util.lua
create mode 100644 test/reload_evolution/storage.result
create mode 100644 test/reload_evolution/storage.test.lua
create mode 100755 test/reload_evolution/storage_1_a.lua
create mode 120000 test/reload_evolution/storage_1_b.lua
create mode 120000 test/reload_evolution/storage_2_a.lua
create mode 120000 test/reload_evolution/storage_2_b.lua
create mode 100644 test/reload_evolution/suite.ini
create mode 100644 test/reload_evolution/test.lua
create mode 100644 test/unit/reload_evolution.result
create mode 100644 test/unit/reload_evolution.test.lua
create mode 100644 vshard/storage/reload_evolution.lua
diff --git a/.travis.yml b/.travis.yml
index 54bfe44..eff4a51 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -41,7 +41,7 @@ env:
script:
- git describe --long
- git clone https://github.com/packpack/packpack.git packpack
- - packpack/packpack
+ - packpack/packpack -e PACKPACK_GIT_SOURCEDIR=/source/
before_deploy:
- ls -l build/
diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh
index 768b22b..554032b 100755
--- a/rpm/prebuild.sh
+++ b/rpm/prebuild.sh
@@ -1 +1,3 @@
curl -s https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh | sudo bash
+sudo yum -y install python-devel python-pip
+sudo pip install tarantool msgpack
diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
new file mode 100644
index 0000000..a75bb08
--- /dev/null
+++ b/test/lua_libs/git_util.lua
@@ -0,0 +1,51 @@
+--
+-- Lua bridge for some of the git commands.
+--
+local os = require('os')
+
+local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
+
+--
+-- Exec a git command.
+-- @param params Table of parameters:
+-- * options - git options.
+-- * cmd - git command.
+-- * args - command arguments.
+-- * dir - working directory.
+-- * fout - write output to the file.
+local function exec_cmd(params)
+ local fout = params.fout
+ local shell_cmd = {'git'}
+ for _, param in pairs({'options', 'cmd', 'args'}) do
+ table.insert(shell_cmd, params[param])
+ end
+ if fout then
+ table.insert(shell_cmd, ' >' .. fout)
+ end
+ shell_cmd = table.concat(shell_cmd, ' ')
+ if params.dir then
+ shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
+ end
+ local res = os.execute(shell_cmd)
+ assert(res == 0, 'Git cmd error: ' .. res)
+end
+
+local function log_hashes(params)
+ params.args = "--format='%h' " .. params.args
+ local local_temp_file = string.format('%s/%s', os.getenv('PWD'), temp_file)
+ params.fout = local_temp_file
+ params.cmd = 'log'
+ exec_cmd(params)
+ local lines = {}
+ for line in io.lines(local_temp_file) do
+ table.insert(lines, line)
+ end
+ os.remove(local_temp_file)
+ return lines
+end
+
+
+return {
+ exec_cmd = exec_cmd,
+ log_hashes = log_hashes
+}
diff --git a/test/lua_libs/util.lua b/test/lua_libs/util.lua
index f40d3a6..935ff41 100644
--- a/test/lua_libs/util.lua
+++ b/test/lua_libs/util.lua
@@ -1,5 +1,6 @@
local fiber = require('fiber')
local log = require('log')
+local fio = require('fio')
local function check_error(func, ...)
local pstatus, status, err = pcall(func, ...)
@@ -92,10 +93,29 @@ local function has_same_fields(etalon, data)
return true
end
+-- Git directory of the project. Used in evolution tests to
+-- fetch old versions of vshard.
+local SOURCEDIR = os.getenv('PACKPACK_GIT_SOURCEDIR')
+if not SOURCEDIR then
+ SOURCEDIR = os.getenv('SOURCEDIR')
+end
+if not SOURCEDIR then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ script_path = fio.abspath(script_path)
+ SOURCEDIR = fio.abspath(script_path .. '/../../../')
+end
+
+local BUILDDIR = os.getenv('BUILDDIR')
+if not BUILDDIR then
+ BUILDDIR = SOURCEDIR
+end
+
return {
check_error = check_error,
shuffle_masters = shuffle_masters,
collect_timeouts = collect_timeouts,
wait_master = wait_master,
has_same_fields = has_same_fields,
+ SOURCEDIR = SOURCEDIR,
+ BUILDDIR = BUILDDIR,
}
diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result
new file mode 100644
index 0000000..007192c
--- /dev/null
+++ b/test/reload_evolution/storage.result
@@ -0,0 +1,245 @@
+test_run = require('test_run').new()
+---
+...
+git_util = require('lua_libs.git_util')
+---
+...
+util = require('lua_libs.util')
+---
+...
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+---
+...
+evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
+---
+...
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+---
+...
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+---
+...
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path)
+---
+...
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+---
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
+---
+...
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+---
+...
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+---
+...
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+---
+...
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+---
+...
+util = require('lua_libs.util')
+---
+...
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+---
+...
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+fiber = require('fiber')
+---
+...
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+vshard.storage.internal.reload_version
+---
+- null
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+box.space.test:insert({42, bucket_id_to_move})
+---
+- [42, 3000]
+...
+test_run:switch('default')
+---
+- true
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path})
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+package.loaded['vshard.storage'] = nil
+---
+...
+vshard.storage = require("vshard.storage")
+---
+...
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil
+---
+- true
+...
+vshard.storage.internal.reload_version
+---
+- 1
+...
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+---
+- true
+...
+vshard.storage.bucket_force_create(2000)
+---
+- true
+...
+vshard.storage.buckets_info()[2000]
+---
+- status: active
+ id: 2000
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+---
+- true
+- - [42, 3000]
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+---
+- true
+...
+vshard.storage.garbage_collector_wakeup()
+---
+...
+fiber = require('fiber')
+---
+...
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+---
+- true
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+---
+- true
+- - [42, 3000]
+...
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+---
+- true
+...
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+---
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+---
+- true
+...
+for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1400
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+vshard.storage.bucket_force_create(move_start, move_cnt)
+---
+- true
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1600
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.rebalancer_enable()
+---
+...
+wait_rebalancer_state('Rebalance routes are sent', test_run)
+---
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1500
+...
+test_run:switch('default')
+---
+- true
+...
+test_run:drop_cluster(REPLICASET_2)
+---
+...
+test_run:drop_cluster(REPLICASET_1)
+---
+...
+test_run:cmd('clear filter')
+---
+- true
+...
diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua
new file mode 100644
index 0000000..7af464b
--- /dev/null
+++ b/test/reload_evolution/storage.test.lua
@@ -0,0 +1,87 @@
+test_run = require('test_run').new()
+
+git_util = require('lua_libs.git_util')
+util = require('lua_libs.util')
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path)
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
+
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+util = require('lua_libs.util')
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+
+test_run:switch('storage_1_a')
+vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+
+test_run:switch('storage_2_a')
+fiber = require('fiber')
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+vshard.storage.internal.reload_version
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+box.space.test:insert({42, bucket_id_to_move})
+
+test_run:switch('default')
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path})
+
+test_run:switch('storage_2_a')
+package.loaded['vshard.storage'] = nil
+vshard.storage = require("vshard.storage")
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil
+vshard.storage.internal.reload_version
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+vshard.storage.bucket_force_create(2000)
+vshard.storage.buckets_info()[2000]
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+vshard.storage.garbage_collector_wakeup()
+fiber = require('fiber')
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+test_run:switch('storage_1_a')
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+test_run:switch('storage_2_a')
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_1_a')
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+vshard.storage.bucket_force_create(move_start, move_cnt)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_2_a')
+vshard.storage.rebalancer_enable()
+wait_rebalancer_state('Rebalance routes are sent', test_run)
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+
+test_run:switch('default')
+test_run:drop_cluster(REPLICASET_2)
+test_run:drop_cluster(REPLICASET_1)
+test_run:cmd('clear filter')
diff --git a/test/reload_evolution/storage_1_a.lua b/test/reload_evolution/storage_1_a.lua
new file mode 100755
index 0000000..f1a2981
--- /dev/null
+++ b/test/reload_evolution/storage_1_a.lua
@@ -0,0 +1,48 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+local log = require('log')
+local fiber = require('fiber')
+local util = require('lua_libs.util')
+local fio = require('fio')
+
+-- Get instance name
+NAME = fio.basename(arg[0], '.lua')
+
+-- test-run gate.
+test_run = require('test_run').new()
+require('console').listen(os.getenv('ADMIN'))
+
+-- Run one storage on a different vshard version.
+-- To do that, place vshard src to
+-- BUILDDIR/test/var/vshard_git_tree_copy/.
+if NAME == 'storage_2_a' then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ vshard_copy = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+ package.path = string.format(
+ '%s/?.lua;%s/?/init.lua;%s',
+ vshard_copy, vshard_copy, package.path
+ )
+end
+
+-- Call a configuration provider
+cfg = require('localcfg')
+-- Name to uuid map
+names = {
+ ['storage_1_a'] = '8a274925-a26d-47fc-9e1b-af88ce939412',
+ ['storage_1_b'] = '3de2e3e1-9ebe-4d0d-abb1-26d301b84633',
+ ['storage_2_a'] = '1e02ae8a-afc0-4e91-ba34-843a356b8ed7',
+ ['storage_2_b'] = '001688c3-66f8-4a31-8e19-036c17d489c2',
+}
+
+replicaset1_uuid = 'cbf06940-0790-498b-948d-042b62cf3d29'
+replicaset2_uuid = 'ac522f65-aa94-4134-9f64-51ee384f1a54'
+replicasets = {replicaset1_uuid, replicaset2_uuid}
+
+-- Start the database with sharding
+vshard = require('vshard')
+vshard.storage.cfg(cfg, names[NAME])
+
+-- Bootstrap storage.
+require('lua_libs.bootstrap')
diff --git a/test/reload_evolution/storage_1_b.lua b/test/reload_evolution/storage_1_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_1_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_a.lua b/test/reload_evolution/storage_2_a.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_a.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_b.lua b/test/reload_evolution/storage_2_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/suite.ini b/test/reload_evolution/suite.ini
new file mode 100644
index 0000000..5f55418
--- /dev/null
+++ b/test/reload_evolution/suite.ini
@@ -0,0 +1,6 @@
+[default]
+core = tarantool
+description = Reload evolution tests
+script = test.lua
+is_parallel = False
+lua_libs = ../lua_libs ../../example/localcfg.lua
diff --git a/test/reload_evolution/test.lua b/test/reload_evolution/test.lua
new file mode 100644
index 0000000..ad0543a
--- /dev/null
+++ b/test/reload_evolution/test.lua
@@ -0,0 +1,9 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+box.cfg{
+ listen = os.getenv("LISTEN"),
+}
+
+require('console').listen(os.getenv('ADMIN'))
diff --git a/test/unit/reload_evolution.result b/test/unit/reload_evolution.result
new file mode 100644
index 0000000..10e606d
--- /dev/null
+++ b/test/unit/reload_evolution.result
@@ -0,0 +1,45 @@
+test_run = require('test_run').new()
+---
+...
+fiber = require('fiber')
+---
+...
+log = require('log')
+---
+...
+util = require('util')
+---
+...
+reload_evolution = require('vshard.storage.reload_evolution')
+---
+...
+-- Init with the latest version.
+fake_M = { reload_version = reload_evolution.version }
+---
+...
+-- Test reload to the same version.
+reload_evolution.upgrade(fake_M)
+---
+...
+test_run:grep_log('default', 'vshard.storage.evolution') == nil
+---
+- true
+...
+-- Test downgrage version.
+log.info(string.rep('a', 1000))
+---
+...
+fake_M.reload_version = fake_M.reload_version + 1
+---
+...
+err = util.check_error(reload_evolution.upgrade, fake_M)
+---
+...
+err:match('auto%-downgrade is not implemented')
+---
+- auto-downgrade is not implemented
+...
+test_run:grep_log('default', 'vshard.storage.evolution', 1000) ~= nil
+---
+- false
+...
diff --git a/test/unit/reload_evolution.test.lua b/test/unit/reload_evolution.test.lua
new file mode 100644
index 0000000..2e99152
--- /dev/null
+++ b/test/unit/reload_evolution.test.lua
@@ -0,0 +1,18 @@
+test_run = require('test_run').new()
+fiber = require('fiber')
+log = require('log')
+util = require('util')
+reload_evolution = require('vshard.storage.reload_evolution')
+-- Init with the latest version.
+fake_M = { reload_version = reload_evolution.version }
+
+-- Test reload to the same version.
+reload_evolution.upgrade(fake_M)
+test_run:grep_log('default', 'vshard.storage.evolution') == nil
+
+-- Test downgrage version.
+log.info(string.rep('a', 1000))
+fake_M.reload_version = fake_M.reload_version + 1
+err = util.check_error(reload_evolution.upgrade, fake_M)
+err:match('auto%-downgrade is not implemented')
+test_run:grep_log('default', 'vshard.storage.evolution', 1000) ~= nil
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index 8ca81f6..6be66d2 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -10,6 +10,7 @@ if rawget(_G, MODULE_INTERNALS) then
local vshard_modules = {
'vshard.consts', 'vshard.error', 'vshard.cfg',
'vshard.replicaset', 'vshard.util',
+ 'vshard.storage.reload_evolution'
}
for _, module in pairs(vshard_modules) do
package.loaded[module] = nil
@@ -20,12 +21,16 @@ local lerror = require('vshard.error')
local lcfg = require('vshard.cfg')
local lreplicaset = require('vshard.replicaset')
local util = require('vshard.util')
+local reload_evolution = require('vshard.storage.reload_evolution')
local M = rawget(_G, MODULE_INTERNALS)
if not M then
--
-- The module is loaded for the first time.
--
+ -- !!!WARNING: any change of this table must be reflected in
+ -- `vshard.storage.reload_evolution` module to guarantee
+ -- reloadability of the module.
M = {
---------------- Common module attributes ----------------
-- The last passed configuration.
@@ -105,6 +110,11 @@ if not M then
-- a destination replicaset must drop already received
-- data.
rebalancer_sending_bucket = 0,
+
+ ------------------------- Reload -------------------------
+ -- Version of the loaded module. This number is used on
+ -- reload to determine which upgrade scripts to run.
+ reload_version = reload_evolution.version,
}
end
@@ -1863,6 +1873,7 @@ end
if not rawget(_G, MODULE_INTERNALS) then
rawset(_G, MODULE_INTERNALS, M)
else
+ reload_evolution.upgrade(M)
storage_cfg(M.current_cfg, M.this_replica.uuid)
M.module_version = M.module_version + 1
end
diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua
new file mode 100644
index 0000000..8502a33
--- /dev/null
+++ b/vshard/storage/reload_evolution.lua
@@ -0,0 +1,58 @@
+--
+-- This module is used to upgrade the vshard.storage on the fly.
+-- It updates internal Lua structures in case they are changed
+-- in a commit.
+--
+local log = require('log')
+
+--
+-- Array of upgrade functions.
+-- migrations[version] = function which upgrades module version
+-- from `version` to `version + 1`.
+--
+local migrations = {}
+
+-- Initialize reload_upgrade mechanism
+migrations[#migrations + 1] = function (M)
+ -- Code to update Lua objects.
+end
+
+--
+-- Perform an update based on a version stored in `M` (internals).
+-- @param M Old module internals which should be updated.
+--
+local function upgrade(M)
+ local start_version = M.reload_version or 1
+ if start_version > #migrations then
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'auto-downgrade is not implemented; ' ..
+ 'loaded version is %d, upgrade script version is %d',
+ start_version, #migrations
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ for i = start_version, #migrations do
+ local ok, err = pcall(migrations[i], M)
+ if ok then
+ log.info('vshard.storage.reload_evolution: upgraded to %d version',
+ i)
+ else
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'error during upgrade to %d version: %s', i, err
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ -- Update the version just after upgrade to have an
+ -- actual version in case of an error.
+ M.reload_version = i
+ end
+end
+
+return {
+ version = #migrations,
+ upgrade = upgrade,
+}
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-07-30 8:56 ` [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution AKhatskevich
@ 2018-07-30 11:55 ` Vladislav Shpilevoy
2018-07-31 11:29 ` Alex Khatskevich
2018-07-31 11:33 ` Alex Khatskevich
0 siblings, 2 replies; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-07-30 11:55 UTC (permalink / raw)
To: AKhatskevich, tarantool-patches
Thanks for the patch! See 3 comments below.
> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
> new file mode 100644
> index 0000000..a75bb08
> --- /dev/null
> +++ b/test/lua_libs/git_util.lua
> @@ -0,0 +1,51 @@
> +--
> +-- Lua bridge for some of the git commands.
> +--
> +local os = require('os')
> +
> +local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
1. C library has a ready solution: tmpfile() and tmpnam() functions
that have Lua API: io.tmpfile() to open a tmp file and automatically
delete it on close, and os.tmpname that generates really unique tmp
file name. Please, use.
> +
> +--
> +-- Exec a git command.
> +-- @param params Table of parameters:
> +-- * options - git options.
> +-- * cmd - git command.
> +-- * args - command arguments.
> +-- * dir - working directory.
> +-- * fout - write output to the file.
> +local function exec_cmd(params)
> + local fout = params.fout
> + local shell_cmd = {'git'}
> + for _, param in pairs({'options', 'cmd', 'args'}) do
> + table.insert(shell_cmd, params[param])
> + end
> + if fout then
> + table.insert(shell_cmd, ' >' .. fout)
> + end
> + shell_cmd = table.concat(shell_cmd, ' ')
> + if params.dir then
> + shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
> + end
> + local res = os.execute(shell_cmd)
> + assert(res == 0, 'Git cmd error: ' .. res)
> +end
> +
> +local function log_hashes(params)
> + params.args = "--format='%h' " .. params.args
> + local local_temp_file = string.format('%s/%s', os.getenv('PWD'), temp_file)
> + params.fout = local_temp_file
> + params.cmd = 'log'
> + exec_cmd(params)
> + local lines = {}
> + for line in io.lines(local_temp_file) do
> + table.insert(lines, line)
> + end
> + os.remove(local_temp_file)
> + return lines
> +end
> +
> +
2. Too many empty lines.
> +return {
> + exec_cmd = exec_cmd,
> + log_hashes = log_hashes
> +}
3. What about router evolution?
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-07-30 8:56 ` [tarantool-patches] [PATCH 1/4] Fix races related to object outdating AKhatskevich
@ 2018-07-30 11:55 ` Vladislav Shpilevoy
2018-07-30 16:46 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-07-30 11:55 UTC (permalink / raw)
To: tarantool-patches, AKhatskevich
Hi! Thanks for the patch! See 1 comment below and fixes on
the branch in a separate commit. Please, look and squash.
On 30/07/2018 11:56, AKhatskevich wrote:
> Reload/reconfigure may replace many of M fields during any yield.
> Old objects should not be accessed after they are outdated.
>
> This commit handles such cases within `vshard.router`.
> ---
> vshard/replicaset.lua | 30 ++++++++++++++-----------
> vshard/router/init.lua | 58 +++++++++++++++++++++++++++++--------------------
> vshard/storage/init.lua | 1 +
> 3 files changed, 52 insertions(+), 37 deletions(-)
>
> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
> index 142ddb6..1a0ed2f 100644
> --- a/vshard/router/init.lua
> +++ b/vshard/router/init.lua
> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
> log.verbose("Discovering bucket %d", bucket_id)
> local last_err = nil
> local unreachable_uuid = nil
> - for uuid, replicaset in pairs(M.replicasets) do
> - local _, err =
> - replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
> - if err == nil then
> - bucket_set(bucket_id, replicaset)
> - return replicaset
> - elseif err.code ~= lerror.code.WRONG_BUCKET then
> - last_err = err
> - unreachable_uuid = uuid
> + for uuid, _ in pairs(M.replicasets) do
> + -- Handle reload/reconfigure.
> + replicaset = M.replicasets[uuid]
> + if replicaset then
> + local _, err =
> + replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
> + if err == nil then
> + return bucket_set(bucket_id, replicaset.uuid)
Do not return error immediately. You can continue iteration in the hope
of finding the bucket out on one of next replicasets. So here you
do 'if bucket_set ~= nil then return result end'. Else continue.
> + elseif err.code ~= lerror.code.WRONG_BUCKET then
> + last_err = err
> + unreachable_uuid = uuid
> + end
> end
> end
> local err = nil
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 2/4] Refactor reloadable fiber
2018-07-30 8:56 ` [tarantool-patches] [PATCH 2/4] Refactor reloadable fiber AKhatskevich
@ 2018-07-30 11:55 ` Vladislav Shpilevoy
2018-07-31 11:24 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-07-30 11:55 UTC (permalink / raw)
To: tarantool-patches, AKhatskevich
Thanks for the patch! See 6 comments below.
On 30/07/2018 11:56, AKhatskevich wrote:
> Reloadable fiber changes:
> * renamed reloadable_fiber_f -> reloadable_fiber
1. Now the name is bad. And not only name is changed - you've
changed the semantics. Reloadable_fiber_f was a function to start,
but reloadable_fiber is a function that starts itself. And it is
not clear from its name. Please, rename it to make clear that
it starts the fiber.
> * reloadable_fiber creates fiber and gives name on its own
> * worker name replaced with a fiber name in logs
> * added extra data argument, which is passed to a function
> ---
> test/rebalancer/rebalancer.result | 2 +-
> test/rebalancer/rebalancer.test.lua | 2 +-
> test/router/reload.result | 4 +--
> test/router/reload.test.lua | 4 +--
> test/storage/reload.result | 6 ++--
> test/storage/reload.test.lua | 6 ++--
> test/unit/garbage.result | 2 +-
> test/unit/garbage.test.lua | 2 +-
> test/unit/util.result | 20 ++++---------
> test/unit/util.test.lua | 12 ++++----
> vshard/router/init.lua | 16 +++++-----
> vshard/storage/init.lua | 21 +++++++------
> vshard/util.lua | 59 +++++++++++++++++++++----------------
> 13 files changed, 76 insertions(+), 80 deletions(-)
>> diff --git a/test/unit/util.result b/test/unit/util.result
> index 30906d1..56b863e 100644
> --- a/test/unit/util.result
> +++ b/test/unit/util.result
> @@ -58,16 +54,12 @@ log.info(string.rep('a', 1000))
> fake_M.reloadable_function = function () fiber.sleep(0.01); return true end
> ---
> ...
> -fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
> +fib = util.reloadable_fiber('Worker_name', fake_M, 'reloadable_function')
> ---
> ...
> -while not test_run:grep_log('default', 'Worker_name is reloaded, restarting') do fiber.sleep(0.01) end
> +while not test_run:grep_log('default', 'module is reloaded, restarting') do fiber.sleep(0.01) end
> ---
> ...
> fib:cancel()
> ---
> ...
> -test_run:grep_log('default', 'Worker_name has been started', 1000)
2. Why did you remove 'started' message? It is not ok. Please,
return back.
> ----
> -- Worker_name has been started
> -...
> diff --git a/vshard/util.lua b/vshard/util.lua
> index fb875ce..1319acc 100644
> --- a/vshard/util.lua
> +++ b/vshard/util.lua
> @@ -31,6 +31,30 @@ local function tuple_extract_key(tuple, parts)
> return key
> end
>
> +local function reloadable_fiber_main_loop(module, func_name, data)
3. No any comment.
4. 'Data' is never used. I you need it in later patches, then add it
later.
5. Why the function almost is not changed, but her diff is 100%?
Please, fix.
> + local func = module[func_name]
> +::restart_loop::
> + local ok, err = pcall(func, data)
> + -- yield serves two purposes:
> + -- * makes this fiber cancellable
> + -- * prevents 100% cpu consumption
> + fiber.yield()
> + if not ok then
> + log.error('%s has been failed: %s', func_name, err)
> + if func == module[func_name] then
> + goto restart_loop
> + end
> + -- There is a chance that error was raised during reload
> + -- (or caused by reload). Perform reload in case function
> + -- has been changed.
> + log.error('reloadable function %s has been changed', func_name)
> + end
> + log.info('module is reloaded, restarting')
> + -- luajit drops this frame if next function is called in
> + -- return statement.
> + return M.reloadable_fiber_main_loop(module, func_name, data)
> +end
> +
> --
> -- Wrapper to run a func in infinite loop and restart it on
> -- errors and module reload.
> @@ -44,30 +68,13 @@ end
> -- For example: "Garbage Collector", "Recovery", "Discovery",
> -- "Rebalancer". Used only for an activity logging.
> --
6. Now the comment is obsolete.
> -local function reloadable_fiber_f(module, func_name, worker_name)
> +local function reloadable_fiber(worker_name, module, func_name, data)
> + assert(type(worker_name) == 'string')
> + local xfiber = fiber.create(reloadable_fiber_main_loop, module, func_name,
> + data)
> log.info('%s has been started', worker_name)
> - local func = module[func_name]
> -::reload_loop::
> - local ok, err = pcall(func, module.module_version)
> - -- yield serves two pursoses:
> - -- * makes this fiber cancellable
> - -- * prevents 100% cpu consumption
> - fiber.yield()
> - if not ok then
> - log.error('%s has been failed: %s', worker_name, err)
> - if func == module[func_name] then
> - goto reload_loop
> - end
> - -- There is a chance that error was raised during reload
> - -- (or caused by reload). Perform reload in case function
> - -- has been changed.
> - log.error('%s: reloadable function %s has been changed',
> - worker_name, func_name)
> - end
> - log.info('%s is reloaded, restarting', worker_name)
> - -- luajit drops this frame if next function is called in
> - -- return statement.
> - return M.reloadable_fiber_f(module, func_name, worker_name)
> + xfiber:name(worker_name)
> + return xfiber
> end
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
@ 2018-07-30 16:46 ` Alex Khatskevich
2018-07-30 17:50 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-30 16:46 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>> index 142ddb6..1a0ed2f 100644
>> --- a/vshard/router/init.lua
>> +++ b/vshard/router/init.lua
>> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
>> log.verbose("Discovering bucket %d", bucket_id)
>> local last_err = nil
>> local unreachable_uuid = nil
>> - for uuid, replicaset in pairs(M.replicasets) do
>> - local _, err =
>> - replicaset:callrw('vshard.storage.bucket_stat',
>> {bucket_id})
>> - if err == nil then
>> - bucket_set(bucket_id, replicaset)
>> - return replicaset
>> - elseif err.code ~= lerror.code.WRONG_BUCKET then
>> - last_err = err
>> - unreachable_uuid = uuid
>> + for uuid, _ in pairs(M.replicasets) do
>> + -- Handle reload/reconfigure.
>> + replicaset = M.replicasets[uuid]
>> + if replicaset then
>> + local _, err =
>> + replicaset:callrw('vshard.storage.bucket_stat',
>> {bucket_id})
>> + if err == nil then
>> + return bucket_set(bucket_id, replicaset.uuid)
>
> Do not return error immediately. You can continue iteration in the hope
> of finding the bucket out on one of next replicasets. So here you
> do 'if bucket_set ~= nil then return result end'. Else continue.
Please, read it again. It seems like you misunderstood something.
I did not change the behavior here.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-07-30 16:46 ` Alex Khatskevich
@ 2018-07-30 17:50 ` Vladislav Shpilevoy
2018-07-31 11:05 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-07-30 17:50 UTC (permalink / raw)
To: Alex Khatskevich, tarantool-patches
On 30/07/2018 19:46, Alex Khatskevich wrote:
>
>>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>>> index 142ddb6..1a0ed2f 100644
>>> --- a/vshard/router/init.lua
>>> +++ b/vshard/router/init.lua
>>> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
>>> log.verbose("Discovering bucket %d", bucket_id)
>>> local last_err = nil
>>> local unreachable_uuid = nil
>>> - for uuid, replicaset in pairs(M.replicasets) do
>>> - local _, err =
>>> - replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
>>> - if err == nil then
>>> - bucket_set(bucket_id, replicaset)
>>> - return replicaset
>>> - elseif err.code ~= lerror.code.WRONG_BUCKET then
>>> - last_err = err
>>> - unreachable_uuid = uuid
>>> + for uuid, _ in pairs(M.replicasets) do
>>> + -- Handle reload/reconfigure.
>>> + replicaset = M.replicasets[uuid]
>>> + if replicaset then
>>> + local _, err =
>>> + replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
>>> + if err == nil then
>>> + return bucket_set(bucket_id, replicaset.uuid)
>>
>> Do not return error immediately. You can continue iteration in the hope
>> of finding the bucket out on one of next replicasets. So here you
>> do 'if bucket_set ~= nil then return result end'. Else continue.
> Please, read it again. It seems like you misunderstood something.
> I did not change the behavior here.
You changed the behavior here. Before bucket_set never failed when
'err ~= nil' always a non-nil result was returned, even though outdated.
Now you return nil.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-07-30 17:50 ` Vladislav Shpilevoy
@ 2018-07-31 11:05 ` Alex Khatskevich
2018-08-01 12:36 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-31 11:05 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
>>>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>>>> index 142ddb6..1a0ed2f 100644
>>>> --- a/vshard/router/init.lua
>>>> +++ b/vshard/router/init.lua
>>>> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
>>>> log.verbose("Discovering bucket %d", bucket_id)
>>>> local last_err = nil
>>>> local unreachable_uuid = nil
>>>> - for uuid, replicaset in pairs(M.replicasets) do
>>>> - local _, err =
>>>> - replicaset:callrw('vshard.storage.bucket_stat',
>>>> {bucket_id})
>>>> - if err == nil then
>>>> - bucket_set(bucket_id, replicaset)
>>>> - return replicaset
>>>> - elseif err.code ~= lerror.code.WRONG_BUCKET then
>>>> - last_err = err
>>>> - unreachable_uuid = uuid
>>>> + for uuid, _ in pairs(M.replicasets) do
>>>> + -- Handle reload/reconfigure.
>>>> + replicaset = M.replicasets[uuid]
>>>> + if replicaset then
>>>> + local _, err =
>>>> + replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
>>>> + if err == nil then
>>>> + return bucket_set(bucket_id, replicaset.uuid)
>>>
>>> Do not return error immediately. You can continue iteration in the hope
>>> of finding the bucket out on one of next replicasets. So here you
>>> do 'if bucket_set ~= nil then return result end'. Else continue.
>> Please, read it again. It seems like you misunderstood something.
>> I did not change the behavior here.
>
> You changed the behavior here. Before bucket_set never failed when
> 'err ~= nil' always a non-nil result was returned, even though outdated.
> Now you return nil.
Discussed verbally.
Further iteration in case of the error do not speeds up anything.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 2/4] Refactor reloadable fiber
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
@ 2018-07-31 11:24 ` Alex Khatskevich
2018-07-31 11:30 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-31 11:24 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
On 30.07.2018 14:55, Vladislav Shpilevoy wrote:
> Thanks for the patch! See 6 comments below.
>
> On 30/07/2018 11:56, AKhatskevich wrote:
>> Reloadable fiber changes:
>> * renamed reloadable_fiber_f -> reloadable_fiber
>
> 1. Now the name is bad. And not only name is changed - you've
> changed the semantics. Reloadable_fiber_f was a function to start,
> but reloadable_fiber is a function that starts itself. And it is
> not clear from its name. Please, rename it to make clear that
> it starts the fiber.
renamed to reloadable_fiber_create.
>
>> * reloadable_fiber creates fiber and gives name on its own
>> * worker name replaced with a fiber name in logs
>> * added extra data argument, which is passed to a function
>> ---
>> test/rebalancer/rebalancer.result | 2 +-
>> test/rebalancer/rebalancer.test.lua | 2 +-
>> test/router/reload.result | 4 +--
>> test/router/reload.test.lua | 4 +--
>> test/storage/reload.result | 6 ++--
>> test/storage/reload.test.lua | 6 ++--
>> test/unit/garbage.result | 2 +-
>> test/unit/garbage.test.lua | 2 +-
>> test/unit/util.result | 20 ++++---------
>> test/unit/util.test.lua | 12 ++++----
>> vshard/router/init.lua | 16 +++++-----
>> vshard/storage/init.lua | 21 +++++++------
>> vshard/util.lua | 59
>> +++++++++++++++++++++----------------
>> 13 files changed, 76 insertions(+), 80 deletions(-)
>>> diff --git a/test/unit/util.result b/test/unit/util.result
>> index 30906d1..56b863e 100644
>> --- a/test/unit/util.result
>> +++ b/test/unit/util.result
>> @@ -58,16 +54,12 @@ log.info(string.rep('a', 1000))
>> fake_M.reloadable_function = function () fiber.sleep(0.01); return
>> true end
>> ---
>> ...
>> -fib = fiber.create(util.reloadable_fiber_f, fake_M,
>> 'reloadable_function', 'Worker_name')
>> +fib = util.reloadable_fiber('Worker_name', fake_M,
>> 'reloadable_function')
>> ---
>> ...
>> -while not test_run:grep_log('default', 'Worker_name is reloaded,
>> restarting') do fiber.sleep(0.01) end
>> +while not test_run:grep_log('default', 'module is reloaded,
>> restarting') do fiber.sleep(0.01) end
>> ---
>> ...
>> fib:cancel()
>> ---
>> ...
>> -test_run:grep_log('default', 'Worker_name has been started', 1000)
>
> 2. Why did you remove 'started' message? It is not ok. Please,
> return back.
returned
>
>> ----
>> -- Worker_name has been started
>> -...
>> diff --git a/vshard/util.lua b/vshard/util.lua
>> index fb875ce..1319acc 100644
>> --- a/vshard/util.lua
>> +++ b/vshard/util.lua
>> @@ -31,6 +31,30 @@ local function tuple_extract_key(tuple, parts)
>> return key
>> end
>> +local function reloadable_fiber_main_loop(module, func_name, data)
>
> 3. No any comment.
comment added
>
> 4. 'Data' is never used. I you need it in later patches, then add it
> later.
deleted
>
> 5. Why the function almost is not changed, but her diff is 100%?
> Please, fix.
Fixed by moving part of the comment to reloadable_fiber_main_loop.
>
>> + local func = module[func_name]
>> +::restart_loop::
>> + local ok, err = pcall(func, data)
>> + -- yield serves two purposes:
>> + -- * makes this fiber cancellable
>> + -- * prevents 100% cpu consumption
>> + fiber.yield()
>> + if not ok then
>> + log.error('%s has been failed: %s', func_name, err)
>> + if func == module[func_name] then
>> + goto restart_loop
>> + end
>> + -- There is a chance that error was raised during reload
>> + -- (or caused by reload). Perform reload in case function
>> + -- has been changed.
>> + log.error('reloadable function %s has been changed', func_name)
>> + end
>> + log.info('module is reloaded, restarting')
>> + -- luajit drops this frame if next function is called in
>> + -- return statement.
>> + return M.reloadable_fiber_main_loop(module, func_name, data)
>> +end
>> +
>> --
>> -- Wrapper to run a func in infinite loop and restart it on
>> -- errors and module reload.
>> @@ -44,30 +68,13 @@ end
>> -- For example: "Garbage Collector", "Recovery", "Discovery",
>> -- "Rebalancer". Used only for an activity logging.
>> --
>
> 6. Now the comment is obsolete.
deleted
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
@ 2018-07-31 11:29 ` Alex Khatskevich
2018-07-31 11:33 ` Alex Khatskevich
1 sibling, 0 replies; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-31 11:29 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
On 30.07.2018 14:55, Vladislav Shpilevoy wrote:
> Thanks for the patch! See 3 comments below.
>
>> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
>> new file mode 100644
>> index 0000000..a75bb08
>> --- /dev/null
>> +++ b/test/lua_libs/git_util.lua
>> @@ -0,0 +1,51 @@
>> +--
>> +-- Lua bridge for some of the git commands.
>> +--
>> +local os = require('os')
>> +
>> +local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
>
> 1. C library has a ready solution: tmpfile() and tmpnam() functions
> that have Lua API: io.tmpfile() to open a tmp file and automatically
> delete it on close, and os.tmpname that generates really unique tmp
> file name. Please, use.
io.tmpfile() is used.
>
>> +
>> +--
>> +-- Exec a git command.
>> +-- @param params Table of parameters:
>> +-- * options - git options.
>> +-- * cmd - git command.
>> +-- * args - command arguments.
>> +-- * dir - working directory.
>> +-- * fout - write output to the file.
>> +local function exec_cmd(params)
>> + local fout = params.fout
>> + local shell_cmd = {'git'}
>> + for _, param in pairs({'options', 'cmd', 'args'}) do
>> + table.insert(shell_cmd, params[param])
>> + end
>> + if fout then
>> + table.insert(shell_cmd, ' >' .. fout)
>> + end
>> + shell_cmd = table.concat(shell_cmd, ' ')
>> + if params.dir then
>> + shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
>> + end
>> + local res = os.execute(shell_cmd)
>> + assert(res == 0, 'Git cmd error: ' .. res)
>> +end
>> +
>> +local function log_hashes(params)
>> + params.args = "--format='%h' " .. params.args
>> + local local_temp_file = string.format('%s/%s', os.getenv('PWD'),
>> temp_file)
>> + params.fout = local_temp_file
>> + params.cmd = 'log'
>> + exec_cmd(params)
>> + local lines = {}
>> + for line in io.lines(local_temp_file) do
>> + table.insert(lines, line)
>> + end
>> + os.remove(local_temp_file)
>> + return lines
>> +end
>> +
>> +
>
> 2. Too many empty lines.
deleted
>
>> +return {
>> + exec_cmd = exec_cmd,
>> + log_hashes = log_hashes
>> +}
>
> 3. What about router evolution?
Reload evolution is created to make able to reload from one version to
another in
in case of noticeable internal changes.
As restarting router is much cheaper than restarting a storage and
reload evolution
adds extra complexity, let us make users to restart the server in case
of the
noticeable changes.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 2/4] Refactor reloadable fiber
2018-07-31 11:24 ` Alex Khatskevich
@ 2018-07-31 11:30 ` Alex Khatskevich
2018-08-01 11:54 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-31 11:30 UTC (permalink / raw)
To: tarantool-patches
full diff
commit 8afc94e2bb4ec880815e7f2154607d9f23a88e27
Author: AKhatskevich <avkhatskevich@tarantool.org>
Date: Thu Jul 26 19:42:14 2018 +0300
Refactor reloadable fiber
Reloadable fiber changes:
* renamed reloadable_fiber_f -> reloadable_fiber
* reloadable_fiber creates fiber and gives name on its own
* worker name replaced with a fiber name in logs
* added extra data argument, which is passed to a function
diff --git a/test/rebalancer/rebalancer.result
b/test/rebalancer/rebalancer.result
index 88cbaae..3df8d4b 100644
--- a/test/rebalancer/rebalancer.result
+++ b/test/rebalancer/rebalancer.result
@@ -418,7 +418,7 @@ vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
fiber = require('fiber')
---
...
-while not test_run:grep_log('box_2_a', "Rebalancer has been started")
do fiber.sleep(0.1) end
+while not test_run:grep_log('box_2_a', "rebalancer_f has been started")
do fiber.sleep(0.1) end
---
...
while not test_run:grep_log('box_1_a', "Rebalancer location has
changed") do fiber.sleep(0.1) end
diff --git a/test/rebalancer/rebalancer.test.lua
b/test/rebalancer/rebalancer.test.lua
index 01f2061..0cfe9a9 100644
--- a/test/rebalancer/rebalancer.test.lua
+++ b/test/rebalancer/rebalancer.test.lua
@@ -195,7 +195,7 @@ switch_rs1_master()
vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
fiber = require('fiber')
-while not test_run:grep_log('box_2_a', "Rebalancer has been started")
do fiber.sleep(0.1) end
+while not test_run:grep_log('box_2_a', "rebalancer_f has been started")
do fiber.sleep(0.1) end
while not test_run:grep_log('box_1_a', "Rebalancer location has
changed") do fiber.sleep(0.1) end
--
diff --git a/test/router/reload.result b/test/router/reload.result
index 88122aa..f0badc3 100644
--- a/test/router/reload.result
+++ b/test/router/reload.result
@@ -116,10 +116,10 @@ vshard.router.module_version()
check_reloaded()
---
...
-while test_run:grep_log('router_1', 'Failover has been started') == nil
do fiber.sleep(0.1) end
+while test_run:grep_log('router_1', 'failover_f has been started') ==
nil do fiber.sleep(0.1) end
---
...
-while test_run:grep_log('router_1', 'Discovery has been started') ==
nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
+while test_run:grep_log('router_1', 'discovery_f has been started') ==
nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
---
...
check_reloaded()
diff --git a/test/router/reload.test.lua b/test/router/reload.test.lua
index 01b7163..528222a 100644
--- a/test/router/reload.test.lua
+++ b/test/router/reload.test.lua
@@ -59,8 +59,8 @@ vshard.router.module_version()
check_reloaded()
-while test_run:grep_log('router_1', 'Failover has been started') == nil
do fiber.sleep(0.1) end
-while test_run:grep_log('router_1', 'Discovery has been started') ==
nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
+while test_run:grep_log('router_1', 'failover_f has been started') ==
nil do fiber.sleep(0.1) end
+while test_run:grep_log('router_1', 'discovery_f has been started') ==
nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
check_reloaded()
diff --git a/test/storage/reload.result b/test/storage/reload.result
index b91b622..c354dba 100644
--- a/test/storage/reload.result
+++ b/test/storage/reload.result
@@ -113,13 +113,13 @@ vshard.storage.module_version()
check_reloaded()
---
...
-while test_run:grep_log('storage_2_a', 'Garbage collector has been
started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('storage_2_a', 'collect_garbage_f has been
started') == nil do fiber.sleep(0.1) end
---
...
-while test_run:grep_log('storage_2_a', 'Recovery has been started') ==
nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
+while test_run:grep_log('storage_2_a', 'recovery_f has been started')
== nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
---
...
-while test_run:grep_log('storage_2_a', 'Rebalancer has been started')
== nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
+while test_run:grep_log('storage_2_a', 'rebalancer_f has been started')
== nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
---
...
check_reloaded()
diff --git a/test/storage/reload.test.lua b/test/storage/reload.test.lua
index 9140299..a8df1df 100644
--- a/test/storage/reload.test.lua
+++ b/test/storage/reload.test.lua
@@ -59,9 +59,9 @@ vshard.storage.module_version()
check_reloaded()
-while test_run:grep_log('storage_2_a', 'Garbage collector has been
started') == nil do fiber.sleep(0.1) end
-while test_run:grep_log('storage_2_a', 'Recovery has been started') ==
nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
-while test_run:grep_log('storage_2_a', 'Rebalancer has been started')
== nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
+while test_run:grep_log('storage_2_a', 'collect_garbage_f has been
started') == nil do fiber.sleep(0.1) end
+while test_run:grep_log('storage_2_a', 'recovery_f has been started')
== nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
+while test_run:grep_log('storage_2_a', 'rebalancer_f has been started')
== nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
check_reloaded()
diff --git a/test/unit/garbage.result b/test/unit/garbage.result
index 0da8ee1..a352bd8 100644
--- a/test/unit/garbage.result
+++ b/test/unit/garbage.result
@@ -343,7 +343,7 @@ control.bucket_generation_collected
collect_f = vshard.storage.internal.collect_garbage_f
---
...
-f = fiber.create(collect_f, vshard.storage.module_version())
+f = fiber.create(collect_f)
---
...
fill_spaces_with_garbage()
diff --git a/test/unit/garbage.test.lua b/test/unit/garbage.test.lua
index db7821f..80d37e7 100644
--- a/test/unit/garbage.test.lua
+++ b/test/unit/garbage.test.lua
@@ -149,7 +149,7 @@ control.bucket_generation_collected
-- Test continuous garbage collection via background fiber.
--
collect_f = vshard.storage.internal.collect_garbage_f
-f = fiber.create(collect_f, vshard.storage.module_version())
+f = fiber.create(collect_f)
fill_spaces_with_garbage()
-- Wait until garbage collection is finished.
while #s2:select{} ~= 2 do fiber.sleep(0.1) end
diff --git a/test/unit/util.result b/test/unit/util.result
index 30906d1..096e36f 100644
--- a/test/unit/util.result
+++ b/test/unit/util.result
@@ -34,22 +34,22 @@ function slow_fail() fiber.sleep(0.01) error('Error
happened.') end
fake_M.reloadable_function = function () fake_M.reloadable_function =
slow_fail; slow_fail() end
---
...
-fib = fiber.create(util.reloadable_fiber_f, fake_M,
'reloadable_function', 'Worker_name')
+fib = util.reloadable_fiber_create('Worker_name', fake_M,
'reloadable_function')
---
...
-while not test_run:grep_log('default', 'Worker_name: reloadable
function reloadable_function has been changed') do fiber.sleep(0.01); end
+while not test_run:grep_log('default', 'reloadable function
reloadable_function has been changed') do fiber.sleep(0.01); end
---
...
fib:cancel()
---
...
-test_run:grep_log('default', 'Worker_name is reloaded, restarting')
+test_run:grep_log('default', 'module is reloaded, restarting')
---
-- Worker_name is reloaded, restarting
+- module is reloaded, restarting
...
-test_run:grep_log('default', 'Worker_name has been started')
+test_run:grep_log('default', 'reloadable_function has been started')
---
-- Worker_name has been started
+- reloadable_function has been started
...
log.info(string.rep('a', 1000))
---
@@ -58,16 +58,16 @@ log.info(string.rep('a', 1000))
fake_M.reloadable_function = function () fiber.sleep(0.01); return
true end
---
...
-fib = fiber.create(util.reloadable_fiber_f, fake_M,
'reloadable_function', 'Worker_name')
+fib = util.reloadable_fiber_create('Worker_name', fake_M,
'reloadable_function')
---
...
-while not test_run:grep_log('default', 'Worker_name is reloaded,
restarting') do fiber.sleep(0.01) end
+while not test_run:grep_log('default', 'module is reloaded,
restarting') do fiber.sleep(0.01) end
---
...
-fib:cancel()
+test_run:grep_log('default', 'reloadable_function has been started', 1000)
---
+- reloadable_function has been started
...
-test_run:grep_log('default', 'Worker_name has been started', 1000)
+fib:cancel()
---
-- Worker_name has been started
...
diff --git a/test/unit/util.test.lua b/test/unit/util.test.lua
index 131274c..5f39e06 100644
--- a/test/unit/util.test.lua
+++ b/test/unit/util.test.lua
@@ -14,16 +14,16 @@ function slow_fail() fiber.sleep(0.01) error('Error
happened.') end
-- Check autoreload on function change during failure.
fake_M.reloadable_function = function () fake_M.reloadable_function =
slow_fail; slow_fail() end
-fib = fiber.create(util.reloadable_fiber_f, fake_M,
'reloadable_function', 'Worker_name')
-while not test_run:grep_log('default', 'Worker_name: reloadable
function reloadable_function has been changed') do fiber.sleep(0.01); end
+fib = util.reloadable_fiber_create('Worker_name', fake_M,
'reloadable_function')
+while not test_run:grep_log('default', 'reloadable function
reloadable_function has been changed') do fiber.sleep(0.01); end
fib:cancel()
-test_run:grep_log('default', 'Worker_name is reloaded, restarting')
-test_run:grep_log('default', 'Worker_name has been started')
+test_run:grep_log('default', 'module is reloaded, restarting')
+test_run:grep_log('default', 'reloadable_function has been started')
log.info(string.rep('a', 1000))
-- Check reload feature.
fake_M.reloadable_function = function () fiber.sleep(0.01); return
true end
-fib = fiber.create(util.reloadable_fiber_f, fake_M,
'reloadable_function', 'Worker_name')
-while not test_run:grep_log('default', 'Worker_name is reloaded,
restarting') do fiber.sleep(0.01) end
+fib = util.reloadable_fiber_create('Worker_name', fake_M,
'reloadable_function')
+while not test_run:grep_log('default', 'module is reloaded,
restarting') do fiber.sleep(0.01) end
+test_run:grep_log('default', 'reloadable_function has been started', 1000)
fib:cancel()
-test_run:grep_log('default', 'Worker_name has been started', 1000)
diff --git a/vshard/router/init.lua b/vshard/router/init.lua
index 1a0ed2f..4cb19fd 100644
--- a/vshard/router/init.lua
+++ b/vshard/router/init.lua
@@ -149,9 +149,8 @@ end
-- Background fiber to perform discovery. It periodically scans
-- replicasets one by one and updates route_map.
--
-local function discovery_f(module_version)
- lfiber.name('discovery_fiber')
- M.discovery_fiber = lfiber.self()
+local function discovery_f()
+ local module_version = M.module_version
local iterations_until_lua_gc =
consts.COLLECT_LUA_GARBAGE_INTERVAL / consts.DISCOVERY_INTERVAL
while module_version == M.module_version do
@@ -459,9 +458,8 @@ end
-- tries to reconnect to the best replica. When the connection is
-- established, it replaces the original replica.
--
-local function failover_f(module_version)
- lfiber.name('vshard.failover')
- M.failover_fiber = lfiber.self()
+local function failover_f()
+ local module_version = M.module_version
local min_timeout = math.min(consts.FAILOVER_UP_TIMEOUT,
consts.FAILOVER_DOWN_TIMEOUT)
-- This flag is used to avoid logging like:
@@ -545,10 +543,12 @@ local function router_cfg(cfg)
M.route_map[bucket] = M.replicasets[rs.uuid]
end
if M.failover_fiber == nil then
- lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
+ M.failover_fiber = util.reloadable_fiber_create(
+ 'vshard.failover', M, 'failover_f')
end
if M.discovery_fiber == nil then
- lfiber.create(util.reloadable_fiber_f, M, 'discovery_f',
'Discovery')
+ M.discovery_fiber = util.reloadable_fiber_create(
+ 'vshard.discovery', M, 'discovery_f')
end
-- Destroy connections, not used in a new configuration.
collectgarbage()
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index c1df0e6..5ec6898 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -352,10 +352,10 @@ end
-- @param module_version Module version, on which the current
-- function had been started. If the actual module version
-- appears to be changed, then stop recovery. It is
--- restarted in reloadable_fiber_f().
+-- restarted in reloadable_fiber.
--
-local function recovery_f(module_version)
- lfiber.name('vshard.recovery')
+local function recovery_f()
+ local module_version = M.module_version
local _bucket = box.space._bucket
M.buckets_to_recovery = {}
for _, bucket in
_bucket.index.status:pairs({consts.BUCKET.SENDING}) do
@@ -728,10 +728,9 @@ local function local_on_master_enable()
M._on_master_enable:run()
-- Start background process to collect garbage.
M.collect_bucket_garbage_fiber =
- lfiber.create(util.reloadable_fiber_f, M, 'collect_garbage_f',
- 'Garbage collector')
+ util.reloadable_fiber_create('vshard.gc', M, 'collect_garbage_f')
M.recovery_fiber =
- lfiber.create(util.reloadable_fiber_f, M, 'recovery_f', 'Recovery')
+ util.reloadable_fiber_create('vshard.recovery', M, 'recovery_f')
-- TODO: check current status
log.info("Took on replicaset master role")
end
@@ -1024,10 +1023,10 @@ end
-- @param module_version Module version, on which the current
-- function had been started. If the actual module version
-- appears to be changed, then stop GC. It is restarted
--- in reloadable_fiber_f().
+-- in reloadable_fiber.
--
-function collect_garbage_f(module_version)
- lfiber.name('vshard.gc')
+function collect_garbage_f()
+ local module_version = M.module_version
-- Collector controller. Changes of _bucket increments
-- bucket generation. Garbage collector has its own bucket
-- generation which is <= actual. Garbage collection is
@@ -1351,8 +1350,8 @@ end
-- Background rebalancer. Works on a storage which has the
-- smallest replicaset uuid and which is master.
--
-local function rebalancer_f(module_version)
- lfiber.name('vshard.rebalancer')
+local function rebalancer_f()
+ local module_version = M.module_version
while module_version == M.module_version do
while not M.is_rebalancer_active do
log.info('Rebalancer is disabled. Sleep')
@@ -1642,8 +1641,8 @@ local function storage_cfg(cfg, this_replica_uuid)
if min_master == this_replica then
if not M.rebalancer_fiber then
- M.rebalancer_fiber = lfiber.create(util.reloadable_fiber_f, M,
- 'rebalancer_f',
'Rebalancer')
+ M.rebalancer_fiber = util.reloadable_fiber_create(
+ 'vshard.rebalancer', M, 'rebalancer_f')
else
log.info('Wakeup rebalancer')
-- Configuration had changed. Time to rebalance.
diff --git a/vshard/util.lua b/vshard/util.lua
index fb875ce..ea676ff 100644
--- a/vshard/util.lua
+++ b/vshard/util.lua
@@ -10,7 +10,7 @@ if not M then
--
M = {
-- Latest versions of functions.
- reloadable_fiber_f = nil,
+ reloadable_fiber_main_loop = nil,
}
rawset(_G, MODULE_INTERNALS, M)
end
@@ -34,40 +34,52 @@ end
--
-- Wrapper to run a func in infinite loop and restart it on
-- errors and module reload.
--- To handle module reload and run new version of a function
--- in the module, the function should just return.
---
--- @param module Module which can be reloaded.
--- @param func_name Name of a function to be executed in the
--- module.
--- @param worker_name Name of the reloadable background subsystem.
--- For example: "Garbage Collector", "Recovery", "Discovery",
--- "Rebalancer". Used only for an activity logging.
+-- This loop executes the latest version of itself in case of
+-- reload of that module.
+-- See description of parameters in `reloadable_fiber_create`.
--
-local function reloadable_fiber_f(module, func_name, worker_name)
- log.info('%s has been started', worker_name)
+local function reloadable_fiber_main_loop(module, func_name)
+ log.info('%s has been started', func_name)
local func = module[func_name]
-::reload_loop::
- local ok, err = pcall(func, module.module_version)
- -- yield serves two pursoses:
+::restart_loop::
+ local ok, err = pcall(func)
+ -- yield serves two purposes:
-- * makes this fiber cancellable
-- * prevents 100% cpu consumption
fiber.yield()
if not ok then
- log.error('%s has been failed: %s', worker_name, err)
+ log.error('%s has been failed: %s', func_name, err)
if func == module[func_name] then
- goto reload_loop
+ goto restart_loop
end
-- There is a chance that error was raised during reload
-- (or caused by reload). Perform reload in case function
-- has been changed.
- log.error('%s: reloadable function %s has been changed',
- worker_name, func_name)
+ log.error('reloadable function %s has been changed', func_name)
end
- log.info('%s is reloaded, restarting', worker_name)
+ log.info('module is reloaded, restarting')
-- luajit drops this frame if next function is called in
-- return statement.
- return M.reloadable_fiber_f(module, func_name, worker_name)
+ return M.reloadable_fiber_main_loop(module, func_name)
+end
+
+--
+-- Create a new fiber which runs a function in a loop. This loop
+-- is aware of reload mechanism and it loads a new version of the
+-- function in that case.
+-- To handle module reload and run new version of a function
+-- in the module, the function should just return.
+-- @param fiber_name Name of a new fiber. E.g. "vshard.rebalancer".
+-- @param module Module which can be reloaded.
+-- @param func_name Name of a function to be executed in the
+-- module.
+-- @retval New fiber.
+--
+local function reloadable_fiber_create(fiber_name, module, func_name)
+ assert(type(fiber_name) == 'string')
+ local xfiber = fiber.create(reloadable_fiber_main_loop, module,
func_name)
+ xfiber:name(fiber_name)
+ return xfiber
end
--
@@ -98,7 +110,7 @@ local function generate_self_checker(obj_name,
func_name, mt, func)
end
-- Update latest versions of function
-M.reloadable_fiber_f = reloadable_fiber_f
+M.reloadable_fiber_main_loop = reloadable_fiber_main_loop
local function sync_task(delay, task, ...)
if delay then
@@ -121,7 +133,7 @@ end
return {
tuple_extract_key = tuple_extract_key,
- reloadable_fiber_f = reloadable_fiber_f,
+ reloadable_fiber_create = reloadable_fiber_create,
generate_self_checker = generate_self_checker,
async_task = async_task,
internal = M,
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-31 11:29 ` Alex Khatskevich
@ 2018-07-31 11:33 ` Alex Khatskevich
2018-08-01 12:36 ` Vladislav Shpilevoy
1 sibling, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-07-31 11:33 UTC (permalink / raw)
To: tarantool-patches
diff of test/lua_libs/git_util.lua
commit 199e198af97200e9a01b5b2568c8e9b2a2a9f34d
Author: AKhatskevich <avkhatskevich@tarantool.org>
Date: Fri Jun 29 20:34:26 2018 +0300
Introduce storage reload evolution
Changes:
1. Introduce storage reload evolution.
2. Setup cross-version reload testing.
1:
This mechanism updates Lua objects on reload in case they are
changed in a new vshard.storage version.
Since this commit, any change in vshard.storage.M has to be
reflected in vshard.storage.reload_evolution to guarantee
correct reload.
2:
The testing uses git infrastructure and is performed in the following
way:
1. Copy old version of vshard to a temp folder.
2. Run vshard on this code.
3. Checkout the latest version of the vshard sources.
4. Reload vshard storage.
5. Make sure it works (Perform simple tests).
Notes:
* this patch contains some legacy-driven decisions:
1. SOURCEDIR path retrieved differently in case of
packpack build.
2. git directory in the `reload_evolution/storage` test
is copied with respect to Centos 7 and `ro` mode of
SOURCEDIR.
Closes #112 #125
diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
new file mode 100644
index 0000000..8826c4c
--- /dev/null
+++ b/test/lua_libs/git_util.lua
@@ -0,0 +1,49 @@
+--
+-- Lua bridge for some of the git commands.
+--
+local os = require('os')
+
+--
+-- Exec a git command.
+-- @param params Table of parameters:
+-- * options - git options.
+-- * cmd - git command.
+-- * args - command arguments.
+-- * dir - working directory.
+-- * fout - write output to the file.
+local function exec_cmd(params)
+ local fout = params.fout
+ local shell_cmd = {'git'}
+ for _, param in pairs({'options', 'cmd', 'args'}) do
+ table.insert(shell_cmd, params[param])
+ end
+ if fout then
+ table.insert(shell_cmd, ' >' .. fout)
+ end
+ shell_cmd = table.concat(shell_cmd, ' ')
+ if params.dir then
+ shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
+ end
+ local res = os.execute(shell_cmd)
+ assert(res == 0, 'Git cmd error: ' .. res)
+end
+
+local function log_hashes(params)
+ params.args = "--format='%h' " .. params.args
+ -- Store log to the file.
+ local temp_file = os.tmpname()
+ params.fout = temp_file
+ params.cmd = 'log'
+ exec_cmd(params)
+ local lines = {}
+ for line in io.lines(temp_file) do
+ table.insert(lines, line)
+ end
+ os.remove(temp_file)
+ return lines
+end
+
+return {
+ exec_cmd = exec_cmd,
+ log_hashes = log_hashes
+}
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 2/4] Refactor reloadable fiber
2018-07-31 11:30 ` Alex Khatskevich
@ 2018-08-01 11:54 ` Vladislav Shpilevoy
0 siblings, 0 replies; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-01 11:54 UTC (permalink / raw)
To: tarantool-patches, Alex Khatskevich
Thanks for the patch! Pushed into the master.
On 31/07/2018 14:30, Alex Khatskevich wrote:
> full diff
>
> commit 8afc94e2bb4ec880815e7f2154607d9f23a88e27
> Author: AKhatskevich <avkhatskevich@tarantool.org>
> Date: Thu Jul 26 19:42:14 2018 +0300
>
> Refactor reloadable fiber
>
> Reloadable fiber changes:
> * renamed reloadable_fiber_f -> reloadable_fiber
> * reloadable_fiber creates fiber and gives name on its own
> * worker name replaced with a fiber name in logs
> * added extra data argument, which is passed to a function
>
> diff --git a/test/rebalancer/rebalancer.result b/test/rebalancer/rebalancer.result
> index 88cbaae..3df8d4b 100644
> --- a/test/rebalancer/rebalancer.result
> +++ b/test/rebalancer/rebalancer.result
> @@ -418,7 +418,7 @@ vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
> fiber = require('fiber')
> ---
> ...
> -while not test_run:grep_log('box_2_a', "Rebalancer has been started") do fiber.sleep(0.1) end
> +while not test_run:grep_log('box_2_a', "rebalancer_f has been started") do fiber.sleep(0.1) end
> ---
> ...
> while not test_run:grep_log('box_1_a', "Rebalancer location has changed") do fiber.sleep(0.1) end
> diff --git a/test/rebalancer/rebalancer.test.lua b/test/rebalancer/rebalancer.test.lua
> index 01f2061..0cfe9a9 100644
> --- a/test/rebalancer/rebalancer.test.lua
> +++ b/test/rebalancer/rebalancer.test.lua
> @@ -195,7 +195,7 @@ switch_rs1_master()
> vshard.storage.cfg(cfg, names.replica_uuid.box_2_b)
>
> fiber = require('fiber')
> -while not test_run:grep_log('box_2_a', "Rebalancer has been started") do fiber.sleep(0.1) end
> +while not test_run:grep_log('box_2_a', "rebalancer_f has been started") do fiber.sleep(0.1) end
> while not test_run:grep_log('box_1_a', "Rebalancer location has changed") do fiber.sleep(0.1) end
>
> --
> diff --git a/test/router/reload.result b/test/router/reload.result
> index 88122aa..f0badc3 100644
> --- a/test/router/reload.result
> +++ b/test/router/reload.result
> @@ -116,10 +116,10 @@ vshard.router.module_version()
> check_reloaded()
> ---
> ...
> -while test_run:grep_log('router_1', 'Failover has been started') == nil do fiber.sleep(0.1) end
> +while test_run:grep_log('router_1', 'failover_f has been started') == nil do fiber.sleep(0.1) end
> ---
> ...
> -while test_run:grep_log('router_1', 'Discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
> +while test_run:grep_log('router_1', 'discovery_f has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
> ---
> ...
> check_reloaded()
> diff --git a/test/router/reload.test.lua b/test/router/reload.test.lua
> index 01b7163..528222a 100644
> --- a/test/router/reload.test.lua
> +++ b/test/router/reload.test.lua
> @@ -59,8 +59,8 @@ vshard.router.module_version()
>
> check_reloaded()
>
> -while test_run:grep_log('router_1', 'Failover has been started') == nil do fiber.sleep(0.1) end
> -while test_run:grep_log('router_1', 'Discovery has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
> +while test_run:grep_log('router_1', 'failover_f has been started') == nil do fiber.sleep(0.1) end
> +while test_run:grep_log('router_1', 'discovery_f has been started') == nil do fiber.sleep(0.1) vshard.router.discovery_wakeup() end
>
> check_reloaded()
>
> diff --git a/test/storage/reload.result b/test/storage/reload.result
> index b91b622..c354dba 100644
> --- a/test/storage/reload.result
> +++ b/test/storage/reload.result
> @@ -113,13 +113,13 @@ vshard.storage.module_version()
> check_reloaded()
> ---
> ...
> -while test_run:grep_log('storage_2_a', 'Garbage collector has been started') == nil do fiber.sleep(0.1) end
> +while test_run:grep_log('storage_2_a', 'collect_garbage_f has been started') == nil do fiber.sleep(0.1) end
> ---
> ...
> -while test_run:grep_log('storage_2_a', 'Recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
> +while test_run:grep_log('storage_2_a', 'recovery_f has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
> ---
> ...
> -while test_run:grep_log('storage_2_a', 'Rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
> +while test_run:grep_log('storage_2_a', 'rebalancer_f has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
> ---
> ...
> check_reloaded()
> diff --git a/test/storage/reload.test.lua b/test/storage/reload.test.lua
> index 9140299..a8df1df 100644
> --- a/test/storage/reload.test.lua
> +++ b/test/storage/reload.test.lua
> @@ -59,9 +59,9 @@ vshard.storage.module_version()
>
> check_reloaded()
>
> -while test_run:grep_log('storage_2_a', 'Garbage collector has been started') == nil do fiber.sleep(0.1) end
> -while test_run:grep_log('storage_2_a', 'Recovery has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
> -while test_run:grep_log('storage_2_a', 'Rebalancer has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
> +while test_run:grep_log('storage_2_a', 'collect_garbage_f has been started') == nil do fiber.sleep(0.1) end
> +while test_run:grep_log('storage_2_a', 'recovery_f has been started') == nil do fiber.sleep(0.1) vshard.storage.recovery_wakeup() end
> +while test_run:grep_log('storage_2_a', 'rebalancer_f has been started') == nil do fiber.sleep(0.1) vshard.storage.rebalancer_wakeup() end
>
> check_reloaded()
>
> diff --git a/test/unit/garbage.result b/test/unit/garbage.result
> index 0da8ee1..a352bd8 100644
> --- a/test/unit/garbage.result
> +++ b/test/unit/garbage.result
> @@ -343,7 +343,7 @@ control.bucket_generation_collected
> collect_f = vshard.storage.internal.collect_garbage_f
> ---
> ...
> -f = fiber.create(collect_f, vshard.storage.module_version())
> +f = fiber.create(collect_f)
> ---
> ...
> fill_spaces_with_garbage()
> diff --git a/test/unit/garbage.test.lua b/test/unit/garbage.test.lua
> index db7821f..80d37e7 100644
> --- a/test/unit/garbage.test.lua
> +++ b/test/unit/garbage.test.lua
> @@ -149,7 +149,7 @@ control.bucket_generation_collected
> -- Test continuous garbage collection via background fiber.
> --
> collect_f = vshard.storage.internal.collect_garbage_f
> -f = fiber.create(collect_f, vshard.storage.module_version())
> +f = fiber.create(collect_f)
> fill_spaces_with_garbage()
> -- Wait until garbage collection is finished.
> while #s2:select{} ~= 2 do fiber.sleep(0.1) end
> diff --git a/test/unit/util.result b/test/unit/util.result
> index 30906d1..096e36f 100644
> --- a/test/unit/util.result
> +++ b/test/unit/util.result
> @@ -34,22 +34,22 @@ function slow_fail() fiber.sleep(0.01) error('Error happened.') end
> fake_M.reloadable_function = function () fake_M.reloadable_function = slow_fail; slow_fail() end
> ---
> ...
> -fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
> +fib = util.reloadable_fiber_create('Worker_name', fake_M, 'reloadable_function')
> ---
> ...
> -while not test_run:grep_log('default', 'Worker_name: reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
> +while not test_run:grep_log('default', 'reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
> ---
> ...
> fib:cancel()
> ---
> ...
> -test_run:grep_log('default', 'Worker_name is reloaded, restarting')
> +test_run:grep_log('default', 'module is reloaded, restarting')
> ---
> -- Worker_name is reloaded, restarting
> +- module is reloaded, restarting
> ...
> -test_run:grep_log('default', 'Worker_name has been started')
> +test_run:grep_log('default', 'reloadable_function has been started')
> ---
> -- Worker_name has been started
> +- reloadable_function has been started
> ...
> log.info(string.rep('a', 1000))
> ---
> @@ -58,16 +58,16 @@ log.info(string.rep('a', 1000))
> fake_M.reloadable_function = function () fiber.sleep(0.01); return true end
> ---
> ...
> -fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
> +fib = util.reloadable_fiber_create('Worker_name', fake_M, 'reloadable_function')
> ---
> ...
> -while not test_run:grep_log('default', 'Worker_name is reloaded, restarting') do fiber.sleep(0.01) end
> +while not test_run:grep_log('default', 'module is reloaded, restarting') do fiber.sleep(0.01) end
> ---
> ...
> -fib:cancel()
> +test_run:grep_log('default', 'reloadable_function has been started', 1000)
> ---
> +- reloadable_function has been started
> ...
> -test_run:grep_log('default', 'Worker_name has been started', 1000)
> +fib:cancel()
> ---
> -- Worker_name has been started
> ...
> diff --git a/test/unit/util.test.lua b/test/unit/util.test.lua
> index 131274c..5f39e06 100644
> --- a/test/unit/util.test.lua
> +++ b/test/unit/util.test.lua
> @@ -14,16 +14,16 @@ function slow_fail() fiber.sleep(0.01) error('Error happened.') end
> -- Check autoreload on function change during failure.
> fake_M.reloadable_function = function () fake_M.reloadable_function = slow_fail; slow_fail() end
>
> -fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
> -while not test_run:grep_log('default', 'Worker_name: reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
> +fib = util.reloadable_fiber_create('Worker_name', fake_M, 'reloadable_function')
> +while not test_run:grep_log('default', 'reloadable function reloadable_function has been changed') do fiber.sleep(0.01); end
> fib:cancel()
> -test_run:grep_log('default', 'Worker_name is reloaded, restarting')
> -test_run:grep_log('default', 'Worker_name has been started')
> +test_run:grep_log('default', 'module is reloaded, restarting')
> +test_run:grep_log('default', 'reloadable_function has been started')
> log.info(string.rep('a', 1000))
>
> -- Check reload feature.
> fake_M.reloadable_function = function () fiber.sleep(0.01); return true end
> -fib = fiber.create(util.reloadable_fiber_f, fake_M, 'reloadable_function', 'Worker_name')
> -while not test_run:grep_log('default', 'Worker_name is reloaded, restarting') do fiber.sleep(0.01) end
> +fib = util.reloadable_fiber_create('Worker_name', fake_M, 'reloadable_function')
> +while not test_run:grep_log('default', 'module is reloaded, restarting') do fiber.sleep(0.01) end
> +test_run:grep_log('default', 'reloadable_function has been started', 1000)
> fib:cancel()
> -test_run:grep_log('default', 'Worker_name has been started', 1000)
> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
> index 1a0ed2f..4cb19fd 100644
> --- a/vshard/router/init.lua
> +++ b/vshard/router/init.lua
> @@ -149,9 +149,8 @@ end
> -- Background fiber to perform discovery. It periodically scans
> -- replicasets one by one and updates route_map.
> --
> -local function discovery_f(module_version)
> - lfiber.name('discovery_fiber')
> - M.discovery_fiber = lfiber.self()
> +local function discovery_f()
> + local module_version = M.module_version
> local iterations_until_lua_gc =
> consts.COLLECT_LUA_GARBAGE_INTERVAL / consts.DISCOVERY_INTERVAL
> while module_version == M.module_version do
> @@ -459,9 +458,8 @@ end
> -- tries to reconnect to the best replica. When the connection is
> -- established, it replaces the original replica.
> --
> -local function failover_f(module_version)
> - lfiber.name('vshard.failover')
> - M.failover_fiber = lfiber.self()
> +local function failover_f()
> + local module_version = M.module_version
> local min_timeout = math.min(consts.FAILOVER_UP_TIMEOUT,
> consts.FAILOVER_DOWN_TIMEOUT)
> -- This flag is used to avoid logging like:
> @@ -545,10 +543,12 @@ local function router_cfg(cfg)
> M.route_map[bucket] = M.replicasets[rs.uuid]
> end
> if M.failover_fiber == nil then
> - lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
> + M.failover_fiber = util.reloadable_fiber_create(
> + 'vshard.failover', M, 'failover_f')
> end
> if M.discovery_fiber == nil then
> - lfiber.create(util.reloadable_fiber_f, M, 'discovery_f', 'Discovery')
> + M.discovery_fiber = util.reloadable_fiber_create(
> + 'vshard.discovery', M, 'discovery_f')
> end
> -- Destroy connections, not used in a new configuration.
> collectgarbage()
> diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
> index c1df0e6..5ec6898 100644
> --- a/vshard/storage/init.lua
> +++ b/vshard/storage/init.lua
> @@ -352,10 +352,10 @@ end
> -- @param module_version Module version, on which the current
> -- function had been started. If the actual module version
> -- appears to be changed, then stop recovery. It is
> --- restarted in reloadable_fiber_f().
> +-- restarted in reloadable_fiber.
> --
> -local function recovery_f(module_version)
> - lfiber.name('vshard.recovery')
> +local function recovery_f()
> + local module_version = M.module_version
> local _bucket = box.space._bucket
> M.buckets_to_recovery = {}
> for _, bucket in _bucket.index.status:pairs({consts.BUCKET.SENDING}) do
> @@ -728,10 +728,9 @@ local function local_on_master_enable()
> M._on_master_enable:run()
> -- Start background process to collect garbage.
> M.collect_bucket_garbage_fiber =
> - lfiber.create(util.reloadable_fiber_f, M, 'collect_garbage_f',
> - 'Garbage collector')
> + util.reloadable_fiber_create('vshard.gc', M, 'collect_garbage_f')
> M.recovery_fiber =
> - lfiber.create(util.reloadable_fiber_f, M, 'recovery_f', 'Recovery')
> + util.reloadable_fiber_create('vshard.recovery', M, 'recovery_f')
> -- TODO: check current status
> log.info("Took on replicaset master role")
> end
> @@ -1024,10 +1023,10 @@ end
> -- @param module_version Module version, on which the current
> -- function had been started. If the actual module version
> -- appears to be changed, then stop GC. It is restarted
> --- in reloadable_fiber_f().
> +-- in reloadable_fiber.
> --
> -function collect_garbage_f(module_version)
> - lfiber.name('vshard.gc')
> +function collect_garbage_f()
> + local module_version = M.module_version
> -- Collector controller. Changes of _bucket increments
> -- bucket generation. Garbage collector has its own bucket
> -- generation which is <= actual. Garbage collection is
> @@ -1351,8 +1350,8 @@ end
> -- Background rebalancer. Works on a storage which has the
> -- smallest replicaset uuid and which is master.
> --
> -local function rebalancer_f(module_version)
> - lfiber.name('vshard.rebalancer')
> +local function rebalancer_f()
> + local module_version = M.module_version
> while module_version == M.module_version do
> while not M.is_rebalancer_active do
> log.info('Rebalancer is disabled. Sleep')
> @@ -1642,8 +1641,8 @@ local function storage_cfg(cfg, this_replica_uuid)
>
> if min_master == this_replica then
> if not M.rebalancer_fiber then
> - M.rebalancer_fiber = lfiber.create(util.reloadable_fiber_f, M,
> - 'rebalancer_f', 'Rebalancer')
> + M.rebalancer_fiber = util.reloadable_fiber_create(
> + 'vshard.rebalancer', M, 'rebalancer_f')
> else
> log.info('Wakeup rebalancer')
> -- Configuration had changed. Time to rebalance.
> diff --git a/vshard/util.lua b/vshard/util.lua
> index fb875ce..ea676ff 100644
> --- a/vshard/util.lua
> +++ b/vshard/util.lua
> @@ -10,7 +10,7 @@ if not M then
> --
> M = {
> -- Latest versions of functions.
> - reloadable_fiber_f = nil,
> + reloadable_fiber_main_loop = nil,
> }
> rawset(_G, MODULE_INTERNALS, M)
> end
> @@ -34,40 +34,52 @@ end
> --
> -- Wrapper to run a func in infinite loop and restart it on
> -- errors and module reload.
> --- To handle module reload and run new version of a function
> --- in the module, the function should just return.
> ---
> --- @param module Module which can be reloaded.
> --- @param func_name Name of a function to be executed in the
> --- module.
> --- @param worker_name Name of the reloadable background subsystem.
> --- For example: "Garbage Collector", "Recovery", "Discovery",
> --- "Rebalancer". Used only for an activity logging.
> +-- This loop executes the latest version of itself in case of
> +-- reload of that module.
> +-- See description of parameters in `reloadable_fiber_create`.
> --
> -local function reloadable_fiber_f(module, func_name, worker_name)
> - log.info('%s has been started', worker_name)
> +local function reloadable_fiber_main_loop(module, func_name)
> + log.info('%s has been started', func_name)
> local func = module[func_name]
> -::reload_loop::
> - local ok, err = pcall(func, module.module_version)
> - -- yield serves two pursoses:
> +::restart_loop::
> + local ok, err = pcall(func)
> + -- yield serves two purposes:
> -- * makes this fiber cancellable
> -- * prevents 100% cpu consumption
> fiber.yield()
> if not ok then
> - log.error('%s has been failed: %s', worker_name, err)
> + log.error('%s has been failed: %s', func_name, err)
> if func == module[func_name] then
> - goto reload_loop
> + goto restart_loop
> end
> -- There is a chance that error was raised during reload
> -- (or caused by reload). Perform reload in case function
> -- has been changed.
> - log.error('%s: reloadable function %s has been changed',
> - worker_name, func_name)
> + log.error('reloadable function %s has been changed', func_name)
> end
> - log.info('%s is reloaded, restarting', worker_name)
> + log.info('module is reloaded, restarting')
> -- luajit drops this frame if next function is called in
> -- return statement.
> - return M.reloadable_fiber_f(module, func_name, worker_name)
> + return M.reloadable_fiber_main_loop(module, func_name)
> +end
> +
> +--
> +-- Create a new fiber which runs a function in a loop. This loop
> +-- is aware of reload mechanism and it loads a new version of the
> +-- function in that case.
> +-- To handle module reload and run new version of a function
> +-- in the module, the function should just return.
> +-- @param fiber_name Name of a new fiber. E.g. "vshard.rebalancer".
> +-- @param module Module which can be reloaded.
> +-- @param func_name Name of a function to be executed in the
> +-- module.
> +-- @retval New fiber.
> +--
> +local function reloadable_fiber_create(fiber_name, module, func_name)
> + assert(type(fiber_name) == 'string')
> + local xfiber = fiber.create(reloadable_fiber_main_loop, module, func_name)
> + xfiber:name(fiber_name)
> + return xfiber
> end
>
> --
> @@ -98,7 +110,7 @@ local function generate_self_checker(obj_name, func_name, mt, func)
> end
>
> -- Update latest versions of function
> -M.reloadable_fiber_f = reloadable_fiber_f
> +M.reloadable_fiber_main_loop = reloadable_fiber_main_loop
>
> local function sync_task(delay, task, ...)
> if delay then
> @@ -121,7 +133,7 @@ end
>
> return {
> tuple_extract_key = tuple_extract_key,
> - reloadable_fiber_f = reloadable_fiber_f,
> + reloadable_fiber_create = reloadable_fiber_create,
> generate_self_checker = generate_self_checker,
> async_task = async_task,
> internal = M,
>
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 3/4] tests: separate bootstrap routine to a lua_libs
2018-07-30 8:56 ` [tarantool-patches] [PATCH 3/4] tests: separate bootstrap routine to a lua_libs AKhatskevich
@ 2018-08-01 12:03 ` Vladislav Shpilevoy
0 siblings, 0 replies; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-01 12:03 UTC (permalink / raw)
To: tarantool-patches, AKhatskevich
Thanks for the patch! Pushed into the master.
On 30/07/2018 11:56, AKhatskevich wrote:
> What is moved to `test/lua_libs/bootstrap.lua`:
> 1. create schema
> 2. create main stored procedures
> 3. `wait_rebalancer_state` procedure
>
> This code will be reused in further commits.
> ---
> test/lua_libs/bootstrap.lua | 50 ++++++++++++++++++++++
> test/rebalancer/box_1_a.lua | 47 ++------------------
> test/rebalancer/errinj.result | 2 +-
> test/rebalancer/errinj.test.lua | 2 +-
> test/rebalancer/rebalancer.result | 2 +-
> test/rebalancer/rebalancer.test.lua | 2 +-
> test/rebalancer/rebalancer_lock_and_pin.result | 2 +-
> test/rebalancer/rebalancer_lock_and_pin.test.lua | 2 +-
> test/rebalancer/restart_during_rebalancing.result | 2 +-
> .../rebalancer/restart_during_rebalancing.test.lua | 2 +-
> test/rebalancer/stress_add_remove_rs.result | 2 +-
> test/rebalancer/stress_add_remove_rs.test.lua | 2 +-
> .../rebalancer/stress_add_remove_several_rs.result | 2 +-
> .../stress_add_remove_several_rs.test.lua | 2 +-
> test/rebalancer/suite.ini | 2 +-
> 15 files changed, 66 insertions(+), 57 deletions(-)
> create mode 100644 test/lua_libs/bootstrap.lua
>
> diff --git a/test/lua_libs/bootstrap.lua b/test/lua_libs/bootstrap.lua
> new file mode 100644
> index 0000000..62c2f78
> --- /dev/null
> +++ b/test/lua_libs/bootstrap.lua
> @@ -0,0 +1,50 @@
> +local log = require('log')
> +
> +function init_schema()
> + local format = {}
> + format[1] = {name = 'field', type = 'unsigned'}
> + format[2] = {name = 'bucket_id', type = 'unsigned'}
> + local s = box.schema.create_space('test', {format = format})
> + local pk = s:create_index('pk')
> + local bucket_id_idx =
> + s:create_index('vbucket', {parts = {'bucket_id'},
> + unique = false})
> +end
> +
> +box.once('schema', function()
> + box.schema.func.create('do_replace')
> + box.schema.role.grant('public', 'execute', 'function', 'do_replace')
> + box.schema.func.create('do_select')
> + box.schema.role.grant('public', 'execute', 'function', 'do_select')
> + init_schema()
> +end)
> +
> +function do_replace(...)
> + box.space.test:replace(...)
> + return true
> +end
> +
> +function do_select(...)
> + return box.space.test:select(...)
> +end
> +
> +function check_consistency()
> + for _, tuple in box.space.test:pairs() do
> + assert(box.space._bucket:get{tuple.bucket_id})
> + end
> + return true
> +end
> +
> +--
> +-- Wait a specified log message.
> +-- Requirements:
> +-- * Should be executed from a storage with a rebalancer.
> +-- * NAME - global variable, name of instance should be set.
> +function wait_rebalancer_state(state, test_run)
> + log.info(string.rep('a', 1000))
> + vshard.storage.rebalancer_wakeup()
> + while not test_run:grep_log(NAME, state, 1000) do
> + fiber.sleep(0.1)
> + vshard.storage.rebalancer_wakeup()
> + end
> +end
> diff --git a/test/rebalancer/box_1_a.lua b/test/rebalancer/box_1_a.lua
> index 8fddcf0..2ca8306 100644
> --- a/test/rebalancer/box_1_a.lua
> +++ b/test/rebalancer/box_1_a.lua
> @@ -2,7 +2,7 @@
> -- Get instance name
> require('strict').on()
> local fio = require('fio')
> -local NAME = fio.basename(arg[0], '.lua')
> +NAME = fio.basename(arg[0], '.lua')
> log = require('log')
> require('console').listen(os.getenv('ADMIN'))
> fiber = require('fiber')
> @@ -23,40 +23,8 @@ if NAME == 'box_4_a' or NAME == 'box_4_b' or
> end
> vshard.storage.cfg(cfg, names.replica_uuid[NAME])
>
> -function init_schema()
> - local format = {}
> - format[1] = {name = 'field', type = 'unsigned'}
> - format[2] = {name = 'bucket_id', type = 'unsigned'}
> - local s = box.schema.create_space('test', {format = format})
> - local pk = s:create_index('pk')
> - local bucket_id_idx =
> - s:create_index('vbucket', {parts = {'bucket_id'},
> - unique = false})
> -end
> -
> -box.once('schema', function()
> - box.schema.func.create('do_replace')
> - box.schema.role.grant('public', 'execute', 'function', 'do_replace')
> - box.schema.func.create('do_select')
> - box.schema.role.grant('public', 'execute', 'function', 'do_select')
> - init_schema()
> -end)
> -
> -function do_replace(...)
> - box.space.test:replace(...)
> - return true
> -end
> -
> -function do_select(...)
> - return box.space.test:select(...)
> -end
> -
> -function check_consistency()
> - for _, tuple in box.space.test:pairs() do
> - assert(box.space._bucket:get{tuple.bucket_id})
> - end
> - return true
> -end
> +-- Bootstrap storage.
> +require('lua_libs.bootstrap')
>
> function switch_rs1_master()
> local replica_uuid = names.replica_uuid
> @@ -68,12 +36,3 @@ end
> function nullify_rs_weight()
> cfg.sharding[names.rs_uuid[1]].weight = 0
> end
> -
> -function wait_rebalancer_state(state, test_run)
> - log.info(string.rep('a', 1000))
> - vshard.storage.rebalancer_wakeup()
> - while not test_run:grep_log(NAME, state, 1000) do
> - fiber.sleep(0.1)
> - vshard.storage.rebalancer_wakeup()
> - end
> -end
> diff --git a/test/rebalancer/errinj.result b/test/rebalancer/errinj.result
> index d09349e..826c2c6 100644
> --- a/test/rebalancer/errinj.result
> +++ b/test/rebalancer/errinj.result
> @@ -13,7 +13,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> diff --git a/test/rebalancer/errinj.test.lua b/test/rebalancer/errinj.test.lua
> index d6a2920..fc0730c 100644
> --- a/test/rebalancer/errinj.test.lua
> +++ b/test/rebalancer/errinj.test.lua
> @@ -5,7 +5,7 @@ REPLICASET_2 = { 'box_2_a', 'box_2_b' }
>
> test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> util.wait_master(test_run, REPLICASET_2, 'box_2_a')
>
> diff --git a/test/rebalancer/rebalancer.result b/test/rebalancer/rebalancer.result
> index bf9e63b..8d9f5e4 100644
> --- a/test/rebalancer/rebalancer.result
> +++ b/test/rebalancer/rebalancer.result
> @@ -13,7 +13,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> diff --git a/test/rebalancer/rebalancer.test.lua b/test/rebalancer/rebalancer.test.lua
> index 24c2706..669b5a1 100644
> --- a/test/rebalancer/rebalancer.test.lua
> +++ b/test/rebalancer/rebalancer.test.lua
> @@ -5,7 +5,7 @@ REPLICASET_2 = { 'box_2_a', 'box_2_b' }
>
> test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> util.wait_master(test_run, REPLICASET_2, 'box_2_a')
>
> diff --git a/test/rebalancer/rebalancer_lock_and_pin.result b/test/rebalancer/rebalancer_lock_and_pin.result
> index dd9fe47..0f2921c 100644
> --- a/test/rebalancer/rebalancer_lock_and_pin.result
> +++ b/test/rebalancer/rebalancer_lock_and_pin.result
> @@ -16,7 +16,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> diff --git a/test/rebalancer/rebalancer_lock_and_pin.test.lua b/test/rebalancer/rebalancer_lock_and_pin.test.lua
> index fe866c4..3a2daa0 100644
> --- a/test/rebalancer/rebalancer_lock_and_pin.test.lua
> +++ b/test/rebalancer/rebalancer_lock_and_pin.test.lua
> @@ -6,7 +6,7 @@ REPLICASET_3 = { 'box_3_a', 'box_3_b' }
>
> test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> util.wait_master(test_run, REPLICASET_2, 'box_2_a')
>
> diff --git a/test/rebalancer/restart_during_rebalancing.result b/test/rebalancer/restart_during_rebalancing.result
> index d2b8a12..0eb0f2e 100644
> --- a/test/rebalancer/restart_during_rebalancing.result
> +++ b/test/rebalancer/restart_during_rebalancing.result
> @@ -25,7 +25,7 @@ test_run:create_cluster(REPLICASET_3, 'rebalancer')
> test_run:create_cluster(REPLICASET_4, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'fullbox_1_a')
> diff --git a/test/rebalancer/restart_during_rebalancing.test.lua b/test/rebalancer/restart_during_rebalancing.test.lua
> index 5b1a8df..7b707ca 100644
> --- a/test/rebalancer/restart_during_rebalancing.test.lua
> +++ b/test/rebalancer/restart_during_rebalancing.test.lua
> @@ -9,7 +9,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> test_run:create_cluster(REPLICASET_3, 'rebalancer')
> test_run:create_cluster(REPLICASET_4, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'fullbox_1_a')
> util.wait_master(test_run, REPLICASET_2, 'fullbox_2_a')
> util.wait_master(test_run, REPLICASET_3, 'fullbox_3_a')
> diff --git a/test/rebalancer/stress_add_remove_rs.result b/test/rebalancer/stress_add_remove_rs.result
> index 8a955e2..10bcaac 100644
> --- a/test/rebalancer/stress_add_remove_rs.result
> +++ b/test/rebalancer/stress_add_remove_rs.result
> @@ -16,7 +16,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> diff --git a/test/rebalancer/stress_add_remove_rs.test.lua b/test/rebalancer/stress_add_remove_rs.test.lua
> index c80df40..b9bb027 100644
> --- a/test/rebalancer/stress_add_remove_rs.test.lua
> +++ b/test/rebalancer/stress_add_remove_rs.test.lua
> @@ -6,7 +6,7 @@ REPLICASET_3 = { 'box_3_a', 'box_3_b' }
>
> test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> util.wait_master(test_run, REPLICASET_2, 'box_2_a')
>
> diff --git a/test/rebalancer/stress_add_remove_several_rs.result b/test/rebalancer/stress_add_remove_several_rs.result
> index d6008b8..611362c 100644
> --- a/test/rebalancer/stress_add_remove_several_rs.result
> +++ b/test/rebalancer/stress_add_remove_several_rs.result
> @@ -19,7 +19,7 @@ test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> ---
> ...
> -util = require('util')
> +util = require('lua_libs.util')
> ---
> ...
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> diff --git a/test/rebalancer/stress_add_remove_several_rs.test.lua b/test/rebalancer/stress_add_remove_several_rs.test.lua
> index 3cc105e..9acb8de 100644
> --- a/test/rebalancer/stress_add_remove_several_rs.test.lua
> +++ b/test/rebalancer/stress_add_remove_several_rs.test.lua
> @@ -7,7 +7,7 @@ REPLICASET_4 = { 'box_4_a', 'box_4_b' }
>
> test_run:create_cluster(REPLICASET_1, 'rebalancer')
> test_run:create_cluster(REPLICASET_2, 'rebalancer')
> -util = require('util')
> +util = require('lua_libs.util')
> util.wait_master(test_run, REPLICASET_1, 'box_1_a')
> util.wait_master(test_run, REPLICASET_2, 'box_2_a')
>
> diff --git a/test/rebalancer/suite.ini b/test/rebalancer/suite.ini
> index afc5141..8689da5 100644
> --- a/test/rebalancer/suite.ini
> +++ b/test/rebalancer/suite.ini
> @@ -4,6 +4,6 @@ description = Rebalancer tests
> script = test.lua
> is_parallel = False
> release_disabled = errinj.test.lua
> -lua_libs = ../lua_libs/util.lua config.lua names.lua router_1.lua
> +lua_libs = ../lua_libs config.lua names.lua router_1.lua
> box_1_a.lua box_1_b.lua box_2_a.lua box_2_b.lua
> box_3_a.lua box_3_b.lua rebalancer_utils.lua
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-07-31 11:05 ` Alex Khatskevich
@ 2018-08-01 12:36 ` Vladislav Shpilevoy
2018-08-01 17:44 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-01 12:36 UTC (permalink / raw)
To: tarantool-patches, Alex Khatskevich
Hi! Thanks for the patch! See 2 comments below.
> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
> index 142ddb6..1a0ed2f 100644
> --- a/vshard/router/init.lua
> +++ b/vshard/router/init.lua
> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
> log.verbose("Discovering bucket %d", bucket_id)
> local last_err = nil
> local unreachable_uuid = nil
> - for uuid, replicaset in pairs(M.replicasets) do
> - local _, err =
> - replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
> - if err == nil then
> - bucket_set(bucket_id, replicaset)
> - return replicaset
> - elseif err.code ~= lerror.code.WRONG_BUCKET then
> - last_err = err
> - unreachable_uuid = uuid
> + for uuid, _ in pairs(M.replicasets) do
> + -- Handle reload/reconfigure.
> + replicaset = M.replicasets[uuid]
1. Please, explain, how is it possible, that before this line
M.replicasets[uuid] can become nil. You iterate here over
M.replicasets and on the previous line in '_' you have
stored replicaset. How can here 'replicaset' differ from '_'?
It has nothing in common with 'reload/reconfigure' case since
you always iterate over M.replicasets - the most actual
list of replicasets. Maybe you thought that pairs() stores
its first argument into a temporary variable but looks like
it is not. I checked it with a simple test:
a = {}
a.objs = {1, 2, 3, 4, 5}
for k, v in pairs(a.objs) do
print(a.objs[k])
if k == 2 then
a.objs = {6, 7, 8, 9, 10}
end
end
Output:
1
2
8
9
10
---
...
> + if replicaset then
> + local _, err =
> + replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
> + if err == nil then
> + return bucket_set(bucket_id, replicaset.uuid)
> + elseif err.code ~= lerror.code.WRONG_BUCKET then
> + last_err = err
> + unreachable_uuid = uuid
> + end
> end
> end
> local err = nil
> @@ -513,27 +522,28 @@ local function router_cfg(cfg)
> end
> box.cfg(box_cfg)
> log.info("Box has been configured")
> - M.connection_outdate_delay = cfg.connection_outdate_delay
> - M.total_bucket_count = total_bucket_count
> - M.collect_lua_garbage = collect_lua_garbage
> - M.current_cfg = new_cfg
> -- Move connections from an old configuration to a new one.
> -- It must be done with no yields to prevent usage both of not
> -- fully moved old replicasets, and not fully built new ones.
> - lreplicaset.rebind_replicasets(new_replicasets, M.replicasets,
> - M.connection_outdate_delay)
> - M.replicasets = new_replicasets
> + lreplicaset.rebind_replicasets(new_replicasets, M.replicasets)
> -- Now the new replicasets are fully built. Can establish
> -- connections and yield.
> for _, replicaset in pairs(new_replicasets) do
> replicaset:connect_all()
> end
> + lreplicaset.wait_masters_connect(new_replicasets)
> + lreplicaset.outdate_replicasets(M.replicasets, cfg.connection_outdate_delay)
> + M.connection_outdate_delay = cfg.connection_outdate_delay
> + M.total_bucket_count = total_bucket_count
> + M.collect_lua_garbage = collect_lua_garbage
> + M.current_cfg = cfg
> + M.replicasets = new_replicasets
> -- Update existing route map in-place.
> - for bucket, rs in pairs(M.route_map) do
> + local old_route_map = M.route_map
> + M.route_map = {}
> + for bucket, rs in pairs(old_route_map) do
> M.route_map[bucket] = M.replicasets[rs.uuid]
2. Why do you need to save old_route_map into a
separate variable? You can update M.route_map in place
like it was done before. It is not? You fill the new
route_map with exactly the same keys (maybe with different
values, but it does not affect 'for' iteration). Moreover,
when you create a new route_map instead of resetting the
old, you double the memory. For huge bucket count it can
be noticeable.
> end
> -
> - lreplicaset.wait_masters_connect(new_replicasets)
> if M.failover_fiber == nil then
> lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
> end
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-07-31 11:33 ` Alex Khatskevich
@ 2018-08-01 12:36 ` Vladislav Shpilevoy
2018-08-01 18:09 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-01 12:36 UTC (permalink / raw)
To: tarantool-patches, Alex Khatskevich
Thanks for the patch! See 2 comments below.
> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
> new file mode 100644
> index 0000000..8826c4c
> --- /dev/null
> +++ b/test/lua_libs/git_util.lua
> @@ -0,0 +1,49 @@
> +--
> +-- Lua bridge for some of the git commands.
> +--
> +local os = require('os')
> +
> +--
> +-- Exec a git command.
> +-- @param params Table of parameters:
> +-- * options - git options.
> +-- * cmd - git command.
> +-- * args - command arguments.
> +-- * dir - working directory.
> +-- * fout - write output to the file.
> +local function exec_cmd(params)
1. Lets rename it to just 'exec'. Obviously it executes
'cmd'. Also lets make 'cmd' be first mandatory non-named
parameter and the second is params: args, dir etc.
local function exec(cmd, params)
> + local fout = params.fout
> + local shell_cmd = {'git'}
> + for _, param in pairs({'options', 'cmd', 'args'}) do
> + table.insert(shell_cmd, params[param])
> + end
> + if fout then
> + table.insert(shell_cmd, ' >' .. fout)
> + end
> + shell_cmd = table.concat(shell_cmd, ' ')
> + if params.dir then
> + shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
> + end
> + local res = os.execute(shell_cmd)
> + assert(res == 0, 'Git cmd error: ' .. res)
> +end
> +
> diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua
> new file mode 100644
> index 0000000..8502a33
> --- /dev/null
> +++ b/vshard/storage/reload_evolution.lua
> @@ -0,0 +1,58 @@
> +--
> +-- This module is used to upgrade the vshard.storage on the fly.
> +-- It updates internal Lua structures in case they are changed
> +-- in a commit.
> +--
> +local log = require('log')
> +
> +--
> +-- Array of upgrade functions.
> +-- migrations[version] = function which upgrades module version
> +-- from `version` to `version + 1`.
> +--
> +local migrations = {}
> +
> +-- Initialize reload_upgrade mechanism
> +migrations[#migrations + 1] = function (M)
2. Redundant white space after 'function'.
> + -- Code to update Lua objects.
> +end
> +
> +--
> +-- Perform an update based on a version stored in `M` (internals).
> +-- @param M Old module internals which should be updated.
> +--
> +local function upgrade(M)
> + local start_version = M.reload_version or 1
> + if start_version > #migrations then
> + local err_msg = string.format(
> + 'vshard.storage.reload_evolution: ' ..
> + 'auto-downgrade is not implemented; ' ..
> + 'loaded version is %d, upgrade script version is %d',
> + start_version, #migrations
> + )
> + log.error(err_msg)
> + error(err_msg)
> + end
> + for i = start_version, #migrations do
> + local ok, err = pcall(migrations[i], M)
> + if ok then
> + log.info('vshard.storage.reload_evolution: upgraded to %d version',
> + i)
> + else
> + local err_msg = string.format(
> + 'vshard.storage.reload_evolution: ' ..
> + 'error during upgrade to %d version: %s', i, err
> + )
> + log.error(err_msg)
> + error(err_msg)
> + end
> + -- Update the version just after upgrade to have an
> + -- actual version in case of an error.
> + M.reload_version = i
> + end
> +end
> +
> +return {
> + version = #migrations,
> + upgrade = upgrade,
> +}
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH] Check self arg passed for router objects
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
` (3 preceding siblings ...)
2018-07-30 8:56 ` [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution AKhatskevich
@ 2018-08-01 14:07 ` AKhatskevich
4 siblings, 0 replies; 32+ messages in thread
From: AKhatskevich @ 2018-08-01 14:07 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Raise an exception in case someone calls router like
`router.info()` instead of `router:info()`.
---
test/multiple_routers/multiple_routers.result | 5 +++++
test/multiple_routers/multiple_routers.test.lua | 3 +++
vshard/router/init.lua | 9 +++++++++
3 files changed, 17 insertions(+)
diff --git a/test/multiple_routers/multiple_routers.result b/test/multiple_routers/multiple_routers.result
index 33f4034..389bf9a 100644
--- a/test/multiple_routers/multiple_routers.result
+++ b/test/multiple_routers/multiple_routers.result
@@ -201,6 +201,11 @@ routers[5]:call(1, 'read', 'do_select', {2})
---
- [[2, 2]]
...
+-- Self checker.
+util.check_error(router_2.info)
+---
+- Use router:info(...) instead of router.info(...)
+...
_ = test_run:cmd("switch default")
---
...
diff --git a/test/multiple_routers/multiple_routers.test.lua b/test/multiple_routers/multiple_routers.test.lua
index 6d470e1..2f159c7 100644
--- a/test/multiple_routers/multiple_routers.test.lua
+++ b/test/multiple_routers/multiple_routers.test.lua
@@ -76,6 +76,9 @@ vshard.router.call(1, 'read', 'do_select', {1})
router_2:call(1, 'read', 'do_select', {2})
routers[5]:call(1, 'read', 'do_select', {2})
+-- Self checker.
+util.check_error(router_2.info)
+
_ = test_run:cmd("switch default")
test_run:cmd("stop server router_1")
test_run:cmd("cleanup server router_1")
diff --git a/vshard/router/init.lua b/vshard/router/init.lua
index 128628b..e0a39b2 100644
--- a/vshard/router/init.lua
+++ b/vshard/router/init.lua
@@ -860,6 +860,15 @@ local router_mt = {
}
}
+--
+-- Wrap self methods with a sanity checker.
+--
+local mt_index = {}
+for name, func in pairs(router_mt.__index) do
+ mt_index[name] = util.generate_self_checker("router", name, router_mt, func)
+end
+router_mt.__index = mt_index
+
-- Table which represents this module.
local module = {}
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-08-01 12:36 ` Vladislav Shpilevoy
@ 2018-08-01 17:44 ` Alex Khatskevich
2018-08-02 11:51 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-08-01 17:44 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
On 01.08.2018 15:36, Vladislav Shpilevoy wrote:
> Hi! Thanks for the patch! See 2 comments below.
>
>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>> index 142ddb6..1a0ed2f 100644
>> --- a/vshard/router/init.lua
>> +++ b/vshard/router/init.lua
>> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
>> log.verbose("Discovering bucket %d", bucket_id)
>> local last_err = nil
>> local unreachable_uuid = nil
>> - for uuid, replicaset in pairs(M.replicasets) do
>> - local _, err =
>> - replicaset:callrw('vshard.storage.bucket_stat',
>> {bucket_id})
>> - if err == nil then
>> - bucket_set(bucket_id, replicaset)
>> - return replicaset
>> - elseif err.code ~= lerror.code.WRONG_BUCKET then
>> - last_err = err
>> - unreachable_uuid = uuid
>> + for uuid, _ in pairs(M.replicasets) do
>> + -- Handle reload/reconfigure.
>> + replicaset = M.replicasets[uuid]
>
> 1. Please, explain, how is it possible, that before this line
> M.replicasets[uuid] can become nil. You iterate here over
> M.replicasets and on the previous line in '_' you have
> stored replicaset. How can here 'replicaset' differ from '_'?
>
> It has nothing in common with 'reload/reconfigure' case since
> you always iterate over M.replicasets - the most actual
> list of replicasets. Maybe you thought that pairs() stores
> its first argument into a temporary variable but looks like
> it is not. I checked it with a simple test:
>
> a = {}
> a.objs = {1, 2, 3, 4, 5}
> for k, v in pairs(a.objs) do
> print(a.objs[k])
> if k == 2 then
> a.objs = {6, 7, 8, 9, 10}
> end
> end
>
> Output:
> 1
> 2
> 8
> 9
> 10
> ---
> ...
Thanks. For some reason I was sure that pairs caches the table.
Fixed.
>> + if replicaset then
>> + local _, err =
>> + replicaset:callrw('vshard.storage.bucket_stat',
>> {bucket_id})
>> + if err == nil then
>> + return bucket_set(bucket_id, replicaset.uuid)
>> + elseif err.code ~= lerror.code.WRONG_BUCKET then
>> + last_err = err
>> + unreachable_uuid = uuid
>> + end
>> end
>> end
>> local err = nil
>> @@ -513,27 +522,28 @@ local function router_cfg(cfg)
>> end
>> box.cfg(box_cfg)
>> log.info("Box has been configured")
>> - M.connection_outdate_delay = cfg.connection_outdate_delay
>> - M.total_bucket_count = total_bucket_count
>> - M.collect_lua_garbage = collect_lua_garbage
>> - M.current_cfg = new_cfg
>> -- Move connections from an old configuration to a new one.
>> -- It must be done with no yields to prevent usage both of not
>> -- fully moved old replicasets, and not fully built new ones.
>> - lreplicaset.rebind_replicasets(new_replicasets, M.replicasets,
>> - M.connection_outdate_delay)
>> - M.replicasets = new_replicasets
>> + lreplicaset.rebind_replicasets(new_replicasets, M.replicasets)
>> -- Now the new replicasets are fully built. Can establish
>> -- connections and yield.
>> for _, replicaset in pairs(new_replicasets) do
>> replicaset:connect_all()
>> end
>> + lreplicaset.wait_masters_connect(new_replicasets)
>> + lreplicaset.outdate_replicasets(M.replicasets,
>> cfg.connection_outdate_delay)
>> + M.connection_outdate_delay = cfg.connection_outdate_delay
>> + M.total_bucket_count = total_bucket_count
>> + M.collect_lua_garbage = collect_lua_garbage
>> + M.current_cfg = cfg
>> + M.replicasets = new_replicasets
>> -- Update existing route map in-place.
>> - for bucket, rs in pairs(M.route_map) do
>> + local old_route_map = M.route_map
>> + M.route_map = {}
>> + for bucket, rs in pairs(old_route_map) do
>> M.route_map[bucket] = M.replicasets[rs.uuid]
>
> 2. Why do you need to save old_route_map into a
> separate variable? You can update M.route_map in place
> like it was done before. It is not? You fill the new
> route_map with exactly the same keys (maybe with different
> values, but it does not affect 'for' iteration). Moreover,
> when you create a new route_map instead of resetting the
> old, you double the memory. For huge bucket count it can
> be noticeable.
Yes there is no point in that change.
Fixed.
>> end
>> -
>> - lreplicaset.wait_masters_connect(new_replicasets)
>> if M.failover_fiber == nil then
>> lfiber.create(util.reloadable_fiber_f, M, 'failover_f',
>> 'Failover')
>> end
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-01 12:36 ` Vladislav Shpilevoy
@ 2018-08-01 18:09 ` Alex Khatskevich
2018-08-02 11:40 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-08-01 18:09 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
I have rebased this branch over a new master.
On 01.08.2018 15:36, Vladislav Shpilevoy wrote:
> Thanks for the patch! See 2 comments below.
>
>> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
>> new file mode 100644
>> index 0000000..8826c4c
>> --- /dev/null
>> +++ b/test/lua_libs/git_util.lua
>> @@ -0,0 +1,49 @@
>> +--
>> +-- Lua bridge for some of the git commands.
>> +--
>> +local os = require('os')
>> +
>> +--
>> +-- Exec a git command.
>> +-- @param params Table of parameters:
>> +-- * options - git options.
>> +-- * cmd - git command.
>> +-- * args - command arguments.
>> +-- * dir - working directory.
>> +-- * fout - write output to the file.
>> +local function exec_cmd(params)
>
> 1. Lets rename it to just 'exec'. Obviously it executes
> 'cmd'. Also lets make 'cmd' be first mandatory non-named
> parameter and the second is params: args, dir etc.
Fixed.
>
> local function exec(cmd, params)
>
>> + local fout = params.fout
>> + local shell_cmd = {'git'}
>> + for _, param in pairs({'options', 'cmd', 'args'}) do
>> + table.insert(shell_cmd, params[param])
>> + end
>> + if fout then
>> + table.insert(shell_cmd, ' >' .. fout)
>> + end
>> + shell_cmd = table.concat(shell_cmd, ' ')
>> + if params.dir then
>> + shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
>> + end
>> + local res = os.execute(shell_cmd)
>> + assert(res == 0, 'Git cmd error: ' .. res)
>> +end
>> +
>> diff --git a/vshard/storage/reload_evolution.lua
>> b/vshard/storage/reload_evolution.lua
>> new file mode 100644
>> index 0000000..8502a33
>> --- /dev/null
>> +++ b/vshard/storage/reload_evolution.lua
>> @@ -0,0 +1,58 @@
>> +--
>> +-- This module is used to upgrade the vshard.storage on the fly.
>> +-- It updates internal Lua structures in case they are changed
>> +-- in a commit.
>> +--
>> +local log = require('log')
>> +
>> +--
>> +-- Array of upgrade functions.
>> +-- migrations[version] = function which upgrades module version
>> +-- from `version` to `version + 1`.
>> +--
>> +local migrations = {}
>> +
>> +-- Initialize reload_upgrade mechanism
>> +migrations[#migrations + 1] = function (M)
>
> 2. Redundant white space after 'function'.
Fixed.
>
>> + -- Code to update Lua objects.
>> +end
>> +
>> +--
>> +-- Perform an update based on a version stored in `M` (internals).
>> +-- @param M Old module internals which should be updated.
>> +--
>> +local function upgrade(M)
>> + local start_version = M.reload_version or 1
>> + if start_version > #migrations then
>> + local err_msg = string.format(
>> + 'vshard.storage.reload_evolution: ' ..
>> + 'auto-downgrade is not implemented; ' ..
>> + 'loaded version is %d, upgrade script version is %d',
>> + start_version, #migrations
>> + )
>> + log.error(err_msg)
>> + error(err_msg)
>> + end
>> + for i = start_version, #migrations do
>> + local ok, err = pcall(migrations[i], M)
>> + if ok then
>> + log.info('vshard.storage.reload_evolution: upgraded to
>> %d version',
>> + i)
>> + else
>> + local err_msg = string.format(
>> + 'vshard.storage.reload_evolution: ' ..
>> + 'error during upgrade to %d version: %s', i, err
>> + )
>> + log.error(err_msg)
>> + error(err_msg)
>> + end
>> + -- Update the version just after upgrade to have an
>> + -- actual version in case of an error.
>> + M.reload_version = i
>> + end
>> +end
>> +
>> +return {
>> + version = #migrations,
>> + upgrade = upgrade,
>> +}
>
Here is a full diff for git_util.lua file
commit 8fbf81cf46b579c158b2b5292a89a0b11ab75494
Author: AKhatskevich <avkhatskevich@tarantool.org>
Date: Fri Jun 29 20:34:26 2018 +0300
Introduce storage reload evolution
Changes:
1. Introduce storage reload evolution.
2. Setup cross-version reload testing.
1:
This mechanism updates Lua objects on reload in case they are
changed in a new vshard.storage version.
Since this commit, any change in vshard.storage.M has to be
reflected in vshard.storage.reload_evolution to guarantee
correct reload.
2:
The testing uses git infrastructure and is performed in the following
way:
1. Copy old version of vshard to a temp folder.
2. Run vshard on this code.
3. Checkout the latest version of the vshard sources.
4. Reload vshard storage.
5. Make sure it works (Perform simple tests).
Notes:
* this patch contains some legacy-driven decisions:
1. SOURCEDIR path retrieved differently in case of
packpack build.
2. git directory in the `reload_evolution/storage` test
is copied with respect to Centos 7 and `ro` mode of
SOURCEDIR.
Closes #112 #125
diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
new file mode 100644
index 0000000..48f9827
--- /dev/null
+++ b/test/lua_libs/git_util.lua
@@ -0,0 +1,46 @@
+--
+-- Lua bridge for some of the git commands.
+--
+local os = require('os')
+
+--
+-- Exec a git command.
+-- @param cmd Git command to run.
+-- @param params Table of parameters:
+-- * options - git options.
+-- * args - command arguments.
+-- * dir - working directory.
+-- * fout - write output to the file.
+local function exec(cmd, params)
+ params.options = params.options or ''
+ params.args = params.args or ''
+ local shell_cmd = string.format('git %s %s %s', params.options, cmd,
+ params.args)
+ if params.fout then
+ shell_cmd = string.format('%s >%s', shell_cmd, params.fout)
+ end
+ if params.dir then
+ shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
+ end
+ local res = os.execute(shell_cmd)
+ assert(res == 0, 'Git cmd error: ' .. res)
+end
+
+local function log_hashes(params)
+ params.args = "--format='%h' " .. params.args
+ -- Store log to the file.
+ local temp_file = os.tmpname()
+ params.fout = temp_file
+ exec('log', params)
+ local lines = {}
+ for line in io.lines(temp_file) do
+ table.insert(lines, line)
+ end
+ os.remove(temp_file)
+ return lines
+end
+
+return {
+ exec = exec,
+ log_hashes = log_hashes
+}
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-01 18:09 ` Alex Khatskevich
@ 2018-08-02 11:40 ` Vladislav Shpilevoy
2018-08-02 11:46 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-02 11:40 UTC (permalink / raw)
To: Alex Khatskevich, tarantool-patches
Hi! Thanks for the fixes!
Test reload_evolution/storage.test.lua fails on my computer
and detaches git head.
======================================================================================
WORKR TEST PARAMS RESULT
---------------------------------------------------------------------------------
[001] reload_evolution/storage.test.lua
[001] TarantoolInpector.handle() received the following error:
[001] Traceback (most recent call last):
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/inspector.py", line 92, in handle
[001] result = self.parser.parse_preprocessor(line)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 95, in parse_preprocessor
[001] return self.server(stype, sname, options)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 278, in server
[001] getattr(self, attr)(ctype, sname, opts)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 206, in server_create
[001] self.servers[sname].deploy(silent=True, **opts)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 507, in deploy
[001] self.install(silent)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 493, in install
[001] self.copy_files()
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 512, in copy_files
[001] shutil.copy(self.script, self.script_dst)
[001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
[001] copyfile(src, dst)
[001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile
[001] with open(src, 'rb') as fsrc:
[001] IOError: [Errno 2] No such file or directory: '/Users/v.shpilevoy/Work/Repositories/vshard/test/reload_evolution/storage_1_a.lua'
[001]
[001]
[001] Test.run() received the following error:
[001] Traceback (most recent call last):
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test.py", line 167, in run
[001] self.execute(server)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 144, in execute
[001] self.killall_servers(server, ts, crash_occured)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 107, in killall_servers
[001] crash_occured = crash_occured or server.process.returncode not in (None, 0, -signal.SIGKILL, -signal.SIGTERM)
[001] AttributeError: 'TarantoolServer' object has no attribute 'process'
[001]
[001] Worker "001_reload_evolution" received the following error; stopping...
[001] Traceback (most recent call last):
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/worker.py", line 266, in run_task
[001] task, self.server, self.inspector)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test_suite.py", line 193, in run_test
[001] short_status = test.run(server)
[001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test.py", line 219, in run
[001] shutil.copy(self.tmp_result, self.reject)
[001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
[001] copyfile(src, dst)
[001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
[001] with open(dst, 'wb') as fdst:
[001] IOError: [Errno 2] No such file or directory: 'reload_evolution/storage.reject'
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-02 11:40 ` Vladislav Shpilevoy
@ 2018-08-02 11:46 ` Vladislav Shpilevoy
2018-08-06 10:59 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-02 11:46 UTC (permalink / raw)
To: Alex Khatskevich, tarantool-patches
Maybe this can help:
------------------------- LOG ------------------------------
Starting instance test...
Run console at unix/:/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test.control
started
mkdir /Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test
2018-08-02 14:38:47.277 [82140] main/101/test C> Tarantool 1.9.1-58-g3c26b86d0
2018-08-02 14:38:47.278 [82140] main/101/test C> log level 5
2018-08-02 14:38:47.278 [82140] main/101/test I> mapping 268435456 bytes for memtx tuple arena...
2018-08-02 14:38:47.278 [82140] main/101/test I> mapping 134217728 bytes for vinyl tuple arena...
2018-08-02 14:38:47.286 [82140] iproto/101/main I> binary: bound to [::1]:38707
2018-08-02 14:38:47.286 [82140] main/101/test I> initializing an empty data directory
2018-08-02 14:38:47.311 [82140] snapshot/101/main I> saving snapshot `/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test/00000000000000000000.snap.inprogress'
2018-08-02 14:38:47.318 [82140] snapshot/101/main I> done
2018-08-02 14:38:47.318 [82140] main/101/test I> ready to accept requests
2018-08-02 14:38:47.318 [82140] main/105/checkpoint_daemon I> started
2018-08-02 14:38:47.319 [82140] main/105/checkpoint_daemon I> scheduled the next snapshot at Thu Aug 2 16:27:59 2018
2018-08-02 14:38:47.320 [82140] main/106/console/::1:38618 I> started
2018-08-02 14:38:47.321 [82140] main C> entering the event loop
ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory
Your branch is ahead of 'origin/master' by 2 commits.
(use "git push" to publish your local commits)
Note: checking out '8fbf81c~1'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at e4f8c21... Fix races related to object outdating
2018-08-02 14:38:47.591 [82140] main C> got signal 15 - Terminated: 15
------------------------ RESULT ------------------------------
test_run = require('test_run').new()
---
...
git_util = require('lua_libs.git_util')
---
...
util = require('lua_libs.util')
---
...
vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
---
...
evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
---
...
-- Cleanup the directory after a previous build.
_ = os.execute('rm -rf ' .. vshard_copy_path)
---
...
-- 1. `git worktree` cannot be used because PACKPACK mounts
-- `/source/` in `ro` mode.
-- 2. Just `cp -rf` cannot be used due to a little different
-- behavior in Centos 7.
_ = os.execute('mkdir ' .. vshard_copy_path)
---
...
_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path)
---
...
-- Checkout the first commit with a reload_evolution mechanism.
git_util.exec('checkout', {args='-f', dir=vshard_copy_path})
---
...
git_util.exec('checkout', {args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
---
...
REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
---
...
REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
---
...
test_run:create_cluster(REPLICASET_1, 'reload_evolution')
Here the test is finished.
On 02/08/2018 14:40, Vladislav Shpilevoy wrote:
> Hi! Thanks for the fixes!
>
> Test reload_evolution/storage.test.lua fails on my computer
> and detaches git head.
>
> ======================================================================================
> WORKR TEST PARAMS RESULT
> ---------------------------------------------------------------------------------
> [001] reload_evolution/storage.test.lua
> [001] TarantoolInpector.handle() received the following error:
> [001] Traceback (most recent call last):
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/inspector.py", line 92, in handle
> [001] result = self.parser.parse_preprocessor(line)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 95, in parse_preprocessor
> [001] return self.server(stype, sname, options)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 278, in server
> [001] getattr(self, attr)(ctype, sname, opts)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/preprocessor.py", line 206, in server_create
> [001] self.servers[sname].deploy(silent=True, **opts)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 507, in deploy
> [001] self.install(silent)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 493, in install
> [001] self.copy_files()
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 512, in copy_files
> [001] shutil.copy(self.script, self.script_dst)
> [001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
> [001] copyfile(src, dst)
> [001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile
> [001] with open(src, 'rb') as fsrc:
> [001] IOError: [Errno 2] No such file or directory: '/Users/v.shpilevoy/Work/Repositories/vshard/test/reload_evolution/storage_1_a.lua'
> [001]
> [001]
> [001] Test.run() received the following error:
> [001] Traceback (most recent call last):
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test.py", line 167, in run
> [001] self.execute(server)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 144, in execute
> [001] self.killall_servers(server, ts, crash_occured)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/tarantool_server.py", line 107, in killall_servers
> [001] crash_occured = crash_occured or server.process.returncode not in (None, 0, -signal.SIGKILL, -signal.SIGTERM)
> [001] AttributeError: 'TarantoolServer' object has no attribute 'process'
> [001]
> [001] Worker "001_reload_evolution" received the following error; stopping...
> [001] Traceback (most recent call last):
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/worker.py", line 266, in run_task
> [001] task, self.server, self.inspector)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test_suite.py", line 193, in run_test
> [001] short_status = test.run(server)
> [001] File "/Users/v.shpilevoy/Work/Repositories/vshard/test/test-run/lib/test.py", line 219, in run
> [001] shutil.copy(self.tmp_result, self.reject)
> [001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
> [001] copyfile(src, dst)
> [001] File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
> [001] with open(dst, 'wb') as fdst:
> [001] IOError: [Errno 2] No such file or directory: 'reload_evolution/storage.reject'
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 1/4] Fix races related to object outdating
2018-08-01 17:44 ` Alex Khatskevich
@ 2018-08-02 11:51 ` Vladislav Shpilevoy
0 siblings, 0 replies; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-02 11:51 UTC (permalink / raw)
To: tarantool-patches, Alex Khatskevich
Thanks for the patch! Pushed into the master.
On 01/08/2018 20:44, Alex Khatskevich wrote:
>
>
> On 01.08.2018 15:36, Vladislav Shpilevoy wrote:
>> Hi! Thanks for the patch! See 2 comments below.
>>
>>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>>> index 142ddb6..1a0ed2f 100644
>>> --- a/vshard/router/init.lua
>>> +++ b/vshard/router/init.lua
>>> @@ -88,15 +94,18 @@ local function bucket_discovery(bucket_id)
>>> log.verbose("Discovering bucket %d", bucket_id)
>>> local last_err = nil
>>> local unreachable_uuid = nil
>>> - for uuid, replicaset in pairs(M.replicasets) do
>>> - local _, err =
>>> - replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
>>> - if err == nil then
>>> - bucket_set(bucket_id, replicaset)
>>> - return replicaset
>>> - elseif err.code ~= lerror.code.WRONG_BUCKET then
>>> - last_err = err
>>> - unreachable_uuid = uuid
>>> + for uuid, _ in pairs(M.replicasets) do
>>> + -- Handle reload/reconfigure.
>>> + replicaset = M.replicasets[uuid]
>>
>> 1. Please, explain, how is it possible, that before this line
>> M.replicasets[uuid] can become nil. You iterate here over
>> M.replicasets and on the previous line in '_' you have
>> stored replicaset. How can here 'replicaset' differ from '_'?
>>
>> It has nothing in common with 'reload/reconfigure' case since
>> you always iterate over M.replicasets - the most actual
>> list of replicasets. Maybe you thought that pairs() stores
>> its first argument into a temporary variable but looks like
>> it is not. I checked it with a simple test:
>>
>> a = {}
>> a.objs = {1, 2, 3, 4, 5}
>> for k, v in pairs(a.objs) do
>> print(a.objs[k])
>> if k == 2 then
>> a.objs = {6, 7, 8, 9, 10}
>> end
>> end
>>
>> Output:
>> 1
>> 2
>> 8
>> 9
>> 10
>> ---
>> ...
> Thanks. For some reason I was sure that pairs caches the table.
> Fixed.
>>> + if replicaset then
>>> + local _, err =
>>> + replicaset:callrw('vshard.storage.bucket_stat', {bucket_id})
>>> + if err == nil then
>>> + return bucket_set(bucket_id, replicaset.uuid)
>>> + elseif err.code ~= lerror.code.WRONG_BUCKET then
>>> + last_err = err
>>> + unreachable_uuid = uuid
>>> + end
>>> end
>>> end
>>> local err = nil
>>> @@ -513,27 +522,28 @@ local function router_cfg(cfg)
>>> end
>>> box.cfg(box_cfg)
>>> log.info("Box has been configured")
>>> - M.connection_outdate_delay = cfg.connection_outdate_delay
>>> - M.total_bucket_count = total_bucket_count
>>> - M.collect_lua_garbage = collect_lua_garbage
>>> - M.current_cfg = new_cfg
>>> -- Move connections from an old configuration to a new one.
>>> -- It must be done with no yields to prevent usage both of not
>>> -- fully moved old replicasets, and not fully built new ones.
>>> - lreplicaset.rebind_replicasets(new_replicasets, M.replicasets,
>>> - M.connection_outdate_delay)
>>> - M.replicasets = new_replicasets
>>> + lreplicaset.rebind_replicasets(new_replicasets, M.replicasets)
>>> -- Now the new replicasets are fully built. Can establish
>>> -- connections and yield.
>>> for _, replicaset in pairs(new_replicasets) do
>>> replicaset:connect_all()
>>> end
>>> + lreplicaset.wait_masters_connect(new_replicasets)
>>> + lreplicaset.outdate_replicasets(M.replicasets, cfg.connection_outdate_delay)
>>> + M.connection_outdate_delay = cfg.connection_outdate_delay
>>> + M.total_bucket_count = total_bucket_count
>>> + M.collect_lua_garbage = collect_lua_garbage
>>> + M.current_cfg = cfg
>>> + M.replicasets = new_replicasets
>>> -- Update existing route map in-place.
>>> - for bucket, rs in pairs(M.route_map) do
>>> + local old_route_map = M.route_map
>>> + M.route_map = {}
>>> + for bucket, rs in pairs(old_route_map) do
>>> M.route_map[bucket] = M.replicasets[rs.uuid]
>>
>> 2. Why do you need to save old_route_map into a
>> separate variable? You can update M.route_map in place
>> like it was done before. It is not? You fill the new
>> route_map with exactly the same keys (maybe with different
>> values, but it does not affect 'for' iteration). Moreover,
>> when you create a new route_map instead of resetting the
>> old, you double the memory. For huge bucket count it can
>> be noticeable.
> Yes there is no point in that change.
> Fixed.
>>> end
>>> -
>>> - lreplicaset.wait_masters_connect(new_replicasets)
>>> if M.failover_fiber == nil then
>>> lfiber.create(util.reloadable_fiber_f, M, 'failover_f', 'Failover')
>>> end
>
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-02 11:46 ` Vladislav Shpilevoy
@ 2018-08-06 10:59 ` Alex Khatskevich
2018-08-06 15:36 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-08-06 10:59 UTC (permalink / raw)
To: tarantool-patches
Thanks. It helped.
It is strange, but `ls` differs mtch on mac and linuxes.
I have changed `ls --ignore` to `ls | grep -v`. Please, try again.
On 02.08.2018 14:46, Vladislav Shpilevoy wrote:
> Maybe this can help:
>
> ------------------------- LOG ------------------------------
>
> Starting instance test...
> Run console at
> unix/:/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test.control
> started
> mkdir
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test
> 2018-08-02 14:38:47.277 [82140] main/101/test C> Tarantool
> 1.9.1-58-g3c26b86d0
> 2018-08-02 14:38:47.278 [82140] main/101/test C> log level 5
> 2018-08-02 14:38:47.278 [82140] main/101/test I> mapping 268435456
> bytes for memtx tuple arena...
> 2018-08-02 14:38:47.278 [82140] main/101/test I> mapping 134217728
> bytes for vinyl tuple arena...
> 2018-08-02 14:38:47.286 [82140] iproto/101/main I> binary: bound to
> [::1]:38707
> 2018-08-02 14:38:47.286 [82140] main/101/test I> initializing an empty
> data directory
> 2018-08-02 14:38:47.311 [82140] snapshot/101/main I> saving snapshot
> `/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test/00000000000000000000.snap.inprogress'
> 2018-08-02 14:38:47.318 [82140] snapshot/101/main I> done
> 2018-08-02 14:38:47.318 [82140] main/101/test I> ready to accept requests
> 2018-08-02 14:38:47.318 [82140] main/105/checkpoint_daemon I> started
> 2018-08-02 14:38:47.319 [82140] main/105/checkpoint_daemon I>
> scheduled the next snapshot at Thu Aug 2 16:27:59 2018
> 2018-08-02 14:38:47.320 [82140] main/106/console/::1:38618 I> started
> 2018-08-02 14:38:47.321 [82140] main C> entering the event loop
> ls: illegal option -- -
> usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]
> usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
> cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ...
> target_directory
> Your branch is ahead of 'origin/master' by 2 commits.
> (use "git push" to publish your local commits)
> Note: checking out '8fbf81c~1'.
>
> You are in 'detached HEAD' state. You can look around, make experimental
> changes and commit them, and you can discard any commits you make in this
> state without impacting any branches by performing another checkout.
>
> If you want to create a new branch to retain commits you create, you may
> do so (now or later) by using -b with the checkout command again.
> Example:
>
> git checkout -b <new-branch-name>
>
> HEAD is now at e4f8c21... Fix races related to object outdating
> 2018-08-02 14:38:47.591 [82140] main C> got signal 15 - Terminated: 15
>
>
> ------------------------ RESULT ------------------------------
>
> test_run = require('test_run').new()
> ---
> ...
> git_util = require('lua_libs.git_util')
> ---
> ...
> util = require('lua_libs.util')
> ---
> ...
> vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
> ---
> ...
> evolution_log =
> git_util.log_hashes({args='vshard/storage/reload_evolution.lua',
> dir=util.SOURCEDIR})
> ---
> ...
> -- Cleanup the directory after a previous build.
> _ = os.execute('rm -rf ' .. vshard_copy_path)
> ---
> ...
> -- 1. `git worktree` cannot be used because PACKPACK mounts
> -- `/source/` in `ro` mode.
> -- 2. Just `cp -rf` cannot be used due to a little different
> -- behavior in Centos 7.
> _ = os.execute('mkdir ' .. vshard_copy_path)
> ---
> ...
> _ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A
> --ignore=build` ' .. vshard_copy_path)
> ---
> ...
> -- Checkout the first commit with a reload_evolution mechanism.
> git_util.exec('checkout', {args='-f', dir=vshard_copy_path})
> ---
> ...
> git_util.exec('checkout', {args=evolution_log[#evolution_log] .. '~1',
> dir=vshard_copy_path})
> ---
> ...
> REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
> ---
> ...
> REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
> ---
> ...
> test_run:create_cluster(REPLICASET_1, 'reload_evolution')
>
> Here the test is finished.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-06 10:59 ` Alex Khatskevich
@ 2018-08-06 15:36 ` Vladislav Shpilevoy
2018-08-06 16:08 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-06 15:36 UTC (permalink / raw)
To: tarantool-patches, Alex Khatskevich
Hi! Thanks for the patch!
Now I see this in the log:
Starting instance test...
Run console at unix/:/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test.control
started
mkdir /Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test
2018-08-06 18:31:52.251 [94220] main/101/test C> Tarantool 1.9.1-60-ga9695b98b
2018-08-06 18:31:52.252 [94220] main/101/test C> log level 5
2018-08-06 18:31:52.252 [94220] main/101/test I> mapping 268435456 bytes for memtx tuple arena...
2018-08-06 18:31:52.252 [94220] main/101/test I> mapping 134217728 bytes for vinyl tuple arena...
2018-08-06 18:31:52.260 [94220] iproto/101/main I> binary: bound to [::1]:38142
2018-08-06 18:31:52.260 [94220] main/101/test I> initializing an empty data directory
2018-08-06 18:31:52.286 [94220] snapshot/101/main I> saving snapshot `/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test/00000000000000000000.snap.inprogress'
2018-08-06 18:31:52.286 [94220] snapshot/101/main I> done
2018-08-06 18:31:52.287 [94220] main/101/test I> ready to accept requests
2018-08-06 18:31:52.287 [94220] main/105/checkpoint_daemon I> started
2018-08-06 18:31:52.287 [94220] main/105/checkpoint_daemon I> scheduled the next snapshot at Mon Aug 6 20:17:56 2018
2018-08-06 18:31:52.289 [94220] main/106/console/::1:38808 I> started
2018-08-06 18:31:52.289 [94220] main C> entering the event loop
cp: test/var/001_reload_evolution/test.control: Operation not supported on socket
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.test.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.result: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.reject: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/restart_during_rebalancing.test.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/test/00000000000000000000.snap: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/lua_libs/bootstrap.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/lua_libs/git_util.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/check_uuid_on_connect.result: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.result: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.test.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.reject: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/storage_2_a.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/storage_2_b.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/check_uuid_on_connect.test.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/router_1.lua: name too long (not copied)
cp: /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/bad_uuid_router.lua: name too long (not copied)
The log is full of such messages. But what is strange - the test passes. Please,
fix these strange things.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-06 15:36 ` Vladislav Shpilevoy
@ 2018-08-06 16:08 ` Alex Khatskevich
2018-08-06 17:18 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-08-06 16:08 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
It seems like `cp` acts differently on mac too:)
I suppose that this behavior was caused by a recurrent copy.
I did extra fix to avoid this. (copy only `.git` folder and fetch other
data by `git checkout`)
On 06.08.2018 18:36, Vladislav Shpilevoy wrote:
> Hi! Thanks for the patch!
>
> Now I see this in the log:
>
> Starting instance test...
> Run console at
> unix/:/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test.control
> started
> mkdir
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test
> 2018-08-06 18:31:52.251 [94220] main/101/test C> Tarantool
> 1.9.1-60-ga9695b98b
> 2018-08-06 18:31:52.252 [94220] main/101/test C> log level 5
> 2018-08-06 18:31:52.252 [94220] main/101/test I> mapping 268435456
> bytes for memtx tuple arena...
> 2018-08-06 18:31:52.252 [94220] main/101/test I> mapping 134217728
> bytes for vinyl tuple arena...
> 2018-08-06 18:31:52.260 [94220] iproto/101/main I> binary: bound to
> [::1]:38142
> 2018-08-06 18:31:52.260 [94220] main/101/test I> initializing an empty
> data directory
> 2018-08-06 18:31:52.286 [94220] snapshot/101/main I> saving snapshot
> `/Users/v.shpilevoy/Work/Repositories/vshard/test/var/001_reload_evolution/test/00000000000000000000.snap.inprogress'
> 2018-08-06 18:31:52.286 [94220] snapshot/101/main I> done
> 2018-08-06 18:31:52.287 [94220] main/101/test I> ready to accept requests
> 2018-08-06 18:31:52.287 [94220] main/105/checkpoint_daemon I> started
> 2018-08-06 18:31:52.287 [94220] main/105/checkpoint_daemon I>
> scheduled the next snapshot at Mon Aug 6 20:17:56 2018
> 2018-08-06 18:31:52.289 [94220] main/106/console/::1:38808 I> started
> 2018-08-06 18:31:52.289 [94220] main C> entering the event loop
> cp: test/var/001_reload_evolution/test.control: Operation not
> supported on socket
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.test.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.result:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/stress_add_remove_several_rs.reject:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/rebalancer/restart_during_rebalancing.test.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/test/00000000000000000000.snap:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/lua_libs/bootstrap.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/001_reload_evolution/lua_libs/git_util.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/check_uuid_on_connect.result:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.result:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.test.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/master_switch.reject:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/storage_2_a.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/storage_2_b.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/check_uuid_on_connect.test.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/router_1.lua:
> name too long (not copied)
> cp:
> /Users/v.shpilevoy/Work/Repositories/vshard/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/var/vshard_git_tree_copy/test/misc/bad_uuid_router.lua:
> name too long (not copied)
>
>
>
>
>
>
> The log is full of such messages. But what is strange - the test
> passes. Please,
> fix these strange things.
>
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-06 16:08 ` Alex Khatskevich
@ 2018-08-06 17:18 ` Vladislav Shpilevoy
2018-08-07 9:14 ` Alex Khatskevich
0 siblings, 1 reply; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-06 17:18 UTC (permalink / raw)
To: Alex Khatskevich, tarantool-patches
Thanks for the fixes! See 1 comment below.
> diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result
> new file mode 100644
> index 0000000..6f3857d
> --- /dev/null
> +++ b/test/reload_evolution/storage.result
> @@ -0,0 +1,245 @@
> +test_run = require('test_run').new()
> +---
> +...
> +git_util = require('lua_libs.git_util')
> +---
> +...
> +util = require('lua_libs.util')
> +---
> +...
> +vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
> +---
> +...
> +evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
> +---
> +...
> +-- Cleanup the directory after a previous build.
> +_ = os.execute('rm -rf ' .. vshard_copy_path)
> +---
> +...
> +-- 1. `git worktree` cannot be used because PACKPACK mounts
> +-- `/source/` in `ro` mode.
> +-- 2. Just `cp -rf` cannot be used due to a little different
> +-- behavior in Centos 7.
> +_ = os.execute('mkdir ' .. vshard_copy_path)
> +---
> +...
> +_ = os.execute("cd " .. util.SOURCEDIR .. " && cp -rf ./.git " .. vshard_copy_path)
> +---
> +...
> +-- Checkout the first commit with a reload_evolution mechanism.
> +git_util.exec('checkout', {args='-f', dir=vshard_copy_path})
> +---
> +...
> +git_util.exec('checkout', {args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
1. Why do you need 2 checkouts in a row? I have removed it and nothing was
changed.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-06 17:18 ` Vladislav Shpilevoy
@ 2018-08-07 9:14 ` Alex Khatskevich
2018-08-08 10:35 ` Vladislav Shpilevoy
0 siblings, 1 reply; 32+ messages in thread
From: Alex Khatskevich @ 2018-08-07 9:14 UTC (permalink / raw)
To: Vladislav Shpilevoy, tarantool-patches
>> +...
>> +-- Checkout the first commit with a reload_evolution mechanism.
>> +git_util.exec('checkout', {args='-f', dir=vshard_copy_path})
>> +---
>> +...
>> +git_util.exec('checkout', {args=evolution_log[#evolution_log] ..
>> '~1', dir=vshard_copy_path})
>
> 1. Why do you need 2 checkouts in a row? I have removed it and nothing
> was
> changed.
Yes. you are right. The first checkout is removed.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] Re: [PATCH 4/4] Introduce storage reload evolution
2018-08-07 9:14 ` Alex Khatskevich
@ 2018-08-08 10:35 ` Vladislav Shpilevoy
0 siblings, 0 replies; 32+ messages in thread
From: Vladislav Shpilevoy @ 2018-08-08 10:35 UTC (permalink / raw)
To: Alex Khatskevich, tarantool-patches
Hi! Thanks for the patch! Pushed into the master.
On 07/08/2018 12:14, Alex Khatskevich wrote:
>>> +...
>>> +-- Checkout the first commit with a reload_evolution mechanism.
>>> +git_util.exec('checkout', {args='-f', dir=vshard_copy_path})
>>> +---
>>> +...
>>> +git_util.exec('checkout', {args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
>>
>> 1. Why do you need 2 checkouts in a row? I have removed it and nothing was
>> changed.
> Yes. you are right. The first checkout is removed.
^ permalink raw reply [flat|nested] 32+ messages in thread
* [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution
2018-07-23 11:14 [tarantool-patches] [PATCH v2] vshard reload mechanism AKhatskevich
@ 2018-07-23 11:14 ` AKhatskevich
0 siblings, 0 replies; 32+ messages in thread
From: AKhatskevich @ 2018-07-23 11:14 UTC (permalink / raw)
To: v.shpilevoy, tarantool-patches
Changes:
1. Introduce storage reload evolution.
2. Setup cross-version reload testing.
1:
This mechanism updates Lua objects on reload in case they are
changed in a new vshard.storage version.
Since this commit, any change in vshard.storage.M has to be
reflected in vshard.storage.reload_evolution to guarantee
correct reload.
2:
The testing uses git infrastructure and is performed in the following
way:
1. Copy old version of vshard to a temp folder.
2. Run vshard on this code.
3. Checkout the latest version of the vshard sources.
4. Reload vshard storage.
5. Make sure it works (Perform simple tests).
Notes:
* this patch contains some legacy-driven decisions:
1. SOURCEDIR path retrieved differently in case of
packpack build.
2. git directory in the `reload_evolution/storage` test
is copied with respect to Centos 7 and `ro` mode of
SOURCEDIR.
Closes #112 #125
---
.travis.yml | 2 +-
rpm/prebuild.sh | 2 +
test/lua_libs/git_util.lua | 51 +++++++
test/lua_libs/util.lua | 20 +++
test/reload_evolution/storage.result | 248 +++++++++++++++++++++++++++++++++
test/reload_evolution/storage.test.lua | 88 ++++++++++++
test/reload_evolution/storage_1_a.lua | 48 +++++++
test/reload_evolution/storage_1_b.lua | 1 +
test/reload_evolution/storage_2_a.lua | 1 +
test/reload_evolution/storage_2_b.lua | 1 +
test/reload_evolution/suite.ini | 6 +
test/reload_evolution/test.lua | 9 ++
test/unit/reload_evolution.result | 45 ++++++
test/unit/reload_evolution.test.lua | 18 +++
vshard/storage/init.lua | 11 ++
vshard/storage/reload_evolution.lua | 58 ++++++++
16 files changed, 608 insertions(+), 1 deletion(-)
create mode 100644 test/lua_libs/git_util.lua
create mode 100644 test/reload_evolution/storage.result
create mode 100644 test/reload_evolution/storage.test.lua
create mode 100755 test/reload_evolution/storage_1_a.lua
create mode 120000 test/reload_evolution/storage_1_b.lua
create mode 120000 test/reload_evolution/storage_2_a.lua
create mode 120000 test/reload_evolution/storage_2_b.lua
create mode 100644 test/reload_evolution/suite.ini
create mode 100644 test/reload_evolution/test.lua
create mode 100644 test/unit/reload_evolution.result
create mode 100644 test/unit/reload_evolution.test.lua
create mode 100644 vshard/storage/reload_evolution.lua
diff --git a/.travis.yml b/.travis.yml
index 54bfe44..eff4a51 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -41,7 +41,7 @@ env:
script:
- git describe --long
- git clone https://github.com/packpack/packpack.git packpack
- - packpack/packpack
+ - packpack/packpack -e PACKPACK_GIT_SOURCEDIR=/source/
before_deploy:
- ls -l build/
diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh
index 768b22b..554032b 100755
--- a/rpm/prebuild.sh
+++ b/rpm/prebuild.sh
@@ -1 +1,3 @@
curl -s https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh | sudo bash
+sudo yum -y install python-devel python-pip
+sudo pip install tarantool msgpack
diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
new file mode 100644
index 0000000..a75bb08
--- /dev/null
+++ b/test/lua_libs/git_util.lua
@@ -0,0 +1,51 @@
+--
+-- Lua bridge for some of the git commands.
+--
+local os = require('os')
+
+local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
+
+--
+-- Exec a git command.
+-- @param params Table of parameters:
+-- * options - git options.
+-- * cmd - git command.
+-- * args - command arguments.
+-- * dir - working directory.
+-- * fout - write output to the file.
+local function exec_cmd(params)
+ local fout = params.fout
+ local shell_cmd = {'git'}
+ for _, param in pairs({'options', 'cmd', 'args'}) do
+ table.insert(shell_cmd, params[param])
+ end
+ if fout then
+ table.insert(shell_cmd, ' >' .. fout)
+ end
+ shell_cmd = table.concat(shell_cmd, ' ')
+ if params.dir then
+ shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
+ end
+ local res = os.execute(shell_cmd)
+ assert(res == 0, 'Git cmd error: ' .. res)
+end
+
+local function log_hashes(params)
+ params.args = "--format='%h' " .. params.args
+ local local_temp_file = string.format('%s/%s', os.getenv('PWD'), temp_file)
+ params.fout = local_temp_file
+ params.cmd = 'log'
+ exec_cmd(params)
+ local lines = {}
+ for line in io.lines(local_temp_file) do
+ table.insert(lines, line)
+ end
+ os.remove(local_temp_file)
+ return lines
+end
+
+
+return {
+ exec_cmd = exec_cmd,
+ log_hashes = log_hashes
+}
diff --git a/test/lua_libs/util.lua b/test/lua_libs/util.lua
index f40d3a6..935ff41 100644
--- a/test/lua_libs/util.lua
+++ b/test/lua_libs/util.lua
@@ -1,5 +1,6 @@
local fiber = require('fiber')
local log = require('log')
+local fio = require('fio')
local function check_error(func, ...)
local pstatus, status, err = pcall(func, ...)
@@ -92,10 +93,29 @@ local function has_same_fields(etalon, data)
return true
end
+-- Git directory of the project. Used in evolution tests to
+-- fetch old versions of vshard.
+local SOURCEDIR = os.getenv('PACKPACK_GIT_SOURCEDIR')
+if not SOURCEDIR then
+ SOURCEDIR = os.getenv('SOURCEDIR')
+end
+if not SOURCEDIR then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ script_path = fio.abspath(script_path)
+ SOURCEDIR = fio.abspath(script_path .. '/../../../')
+end
+
+local BUILDDIR = os.getenv('BUILDDIR')
+if not BUILDDIR then
+ BUILDDIR = SOURCEDIR
+end
+
return {
check_error = check_error,
shuffle_masters = shuffle_masters,
collect_timeouts = collect_timeouts,
wait_master = wait_master,
has_same_fields = has_same_fields,
+ SOURCEDIR = SOURCEDIR,
+ BUILDDIR = BUILDDIR,
}
diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result
new file mode 100644
index 0000000..54ff6b7
--- /dev/null
+++ b/test/reload_evolution/storage.result
@@ -0,0 +1,248 @@
+test_run = require('test_run').new()
+---
+...
+git_util = require('lua_libs.git_util')
+---
+...
+util = require('lua_libs.util')
+---
+...
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+---
+...
+evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
+---
+...
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+---
+...
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+---
+...
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path)
+---
+...
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+---
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
+---
+...
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+---
+...
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+---
+...
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+---
+...
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+---
+...
+util = require('lua_libs.util')
+---
+...
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+---
+...
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+fiber = require('fiber')
+---
+...
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+vshard.storage.internal.reload_evolution_version
+---
+- null
+...
+box.space.test:insert({42, bucket_id_to_move})
+---
+- [42, 3000]
+...
+while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') == nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end
+---
+...
+test_run:switch('default')
+---
+- true
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path})
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+package.loaded["vshard.storage"] = nil
+---
+...
+vshard.storage = require("vshard.storage")
+---
+...
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil
+---
+- true
+...
+vshard.storage.internal.reload_evolution_version
+---
+- 1
+...
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+---
+- true
+...
+vshard.storage.bucket_force_create(2000)
+---
+- true
+...
+vshard.storage.buckets_info()[2000]
+---
+- status: active
+ id: 2000
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+---
+- true
+- - [42, 3000]
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+---
+- true
+...
+vshard.storage.garbage_collector_wakeup()
+---
+...
+fiber = require('fiber')
+---
+...
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+---
+- true
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+---
+- true
+- - [42, 3000]
+...
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+---
+- true
+...
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+---
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+---
+- true
+...
+for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1400
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+vshard.storage.bucket_force_create(move_start, move_cnt)
+---
+- true
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1600
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.rebalancer_enable()
+---
+...
+vshard.storage.rebalancer_wakeup()
+---
+...
+wait_rebalancer_state("Rebalance routes are sent", test_run)
+---
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1500
+...
+test_run:switch('default')
+---
+- true
+...
+test_run:drop_cluster(REPLICASET_2)
+---
+...
+test_run:drop_cluster(REPLICASET_1)
+---
+...
+test_run:cmd('clear filter')
+---
+- true
+...
diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua
new file mode 100644
index 0000000..5817667
--- /dev/null
+++ b/test/reload_evolution/storage.test.lua
@@ -0,0 +1,88 @@
+test_run = require('test_run').new()
+
+git_util = require('lua_libs.git_util')
+util = require('lua_libs.util')
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR})
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path)
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path})
+
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+util = require('lua_libs.util')
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+
+test_run:switch('storage_1_a')
+vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+
+test_run:switch('storage_2_a')
+fiber = require('fiber')
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+vshard.storage.internal.reload_evolution_version
+box.space.test:insert({42, bucket_id_to_move})
+while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') == nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end
+
+test_run:switch('default')
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path})
+
+test_run:switch('storage_2_a')
+package.loaded["vshard.storage"] = nil
+vshard.storage = require("vshard.storage")
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil
+vshard.storage.internal.reload_evolution_version
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+vshard.storage.bucket_force_create(2000)
+vshard.storage.buckets_info()[2000]
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+vshard.storage.garbage_collector_wakeup()
+fiber = require('fiber')
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+test_run:switch('storage_1_a')
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+test_run:switch('storage_2_a')
+vshard.storage.call(bucket_id_to_move, 'read', 'do_select', {42})
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_1_a')
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+vshard.storage.bucket_force_create(move_start, move_cnt)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_2_a')
+vshard.storage.rebalancer_enable()
+vshard.storage.rebalancer_wakeup()
+wait_rebalancer_state("Rebalance routes are sent", test_run)
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+
+test_run:switch('default')
+test_run:drop_cluster(REPLICASET_2)
+test_run:drop_cluster(REPLICASET_1)
+test_run:cmd('clear filter')
diff --git a/test/reload_evolution/storage_1_a.lua b/test/reload_evolution/storage_1_a.lua
new file mode 100755
index 0000000..a971457
--- /dev/null
+++ b/test/reload_evolution/storage_1_a.lua
@@ -0,0 +1,48 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+local log = require('log')
+local fiber = require('fiber')
+local util = require('lua_libs.util')
+local fio = require('fio')
+
+-- Get instance name
+NAME = fio.basename(arg[0], '.lua')
+
+-- test-run gate.
+test_run = require('test_run').new()
+require('console').listen(os.getenv('ADMIN'))
+
+-- Run one storage on a different vshard version.
+-- To do that, place vshard src to
+-- BUILDDIR/test/var/vshard_git_tree_copy/.
+if NAME == 'storage_2_a' then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ vshard_copy = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+ package.path = string.format(
+ '%s/?.lua;%s/?/init.lua;%s',
+ vshard_copy, vshard_copy, package.path
+ )
+end
+
+-- Call a configuration provider
+cfg = require('localcfg')
+-- Name to uuid map
+names = {
+ ['storage_1_a'] = '8a274925-a26d-47fc-9e1b-af88ce939412',
+ ['storage_1_b'] = '3de2e3e1-9ebe-4d0d-abb1-26d301b84633',
+ ['storage_2_a'] = '1e02ae8a-afc0-4e91-ba34-843a356b8ed7',
+ ['storage_2_b'] = '001688c3-66f8-4a31-8e19-036c17d489c2',
+}
+
+replicaset1_uuid = 'cbf06940-0790-498b-948d-042b62cf3d29'
+replicaset2_uuid = 'ac522f65-aa94-4134-9f64-51ee384f1a54'
+replicasets = {replicaset1_uuid, replicaset2_uuid}
+
+-- Start the database with sharding
+vshard = require('vshard')
+vshard.storage.cfg(cfg, names[NAME])
+
+-- Bootstrap storage.
+require('lua_libs.bootstrap_test_storage')
diff --git a/test/reload_evolution/storage_1_b.lua b/test/reload_evolution/storage_1_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_1_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_a.lua b/test/reload_evolution/storage_2_a.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_a.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_b.lua b/test/reload_evolution/storage_2_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/suite.ini b/test/reload_evolution/suite.ini
new file mode 100644
index 0000000..5f55418
--- /dev/null
+++ b/test/reload_evolution/suite.ini
@@ -0,0 +1,6 @@
+[default]
+core = tarantool
+description = Reload evolution tests
+script = test.lua
+is_parallel = False
+lua_libs = ../lua_libs ../../example/localcfg.lua
diff --git a/test/reload_evolution/test.lua b/test/reload_evolution/test.lua
new file mode 100644
index 0000000..ad0543a
--- /dev/null
+++ b/test/reload_evolution/test.lua
@@ -0,0 +1,9 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+box.cfg{
+ listen = os.getenv("LISTEN"),
+}
+
+require('console').listen(os.getenv('ADMIN'))
diff --git a/test/unit/reload_evolution.result b/test/unit/reload_evolution.result
new file mode 100644
index 0000000..342ac24
--- /dev/null
+++ b/test/unit/reload_evolution.result
@@ -0,0 +1,45 @@
+test_run = require('test_run').new()
+---
+...
+fiber = require('fiber')
+---
+...
+log = require('log')
+---
+...
+util = require('util')
+---
+...
+reload_evolution = require('vshard.storage.reload_evolution')
+---
+...
+-- Init with the latest version.
+fake_M = { reload_evolution_version = reload_evolution.version }
+---
+...
+-- Test reload to the same version.
+reload_evolution.upgrade(fake_M)
+---
+...
+test_run:grep_log('default', 'vshard.storage.evolution') == nil
+---
+- true
+...
+-- Test downgrage version.
+log.info(string.rep('a', 1000))
+---
+...
+fake_M.reload_evolution_version = fake_M.reload_evolution_version + 1
+---
+...
+err = util.check_error(reload_evolution.upgrade, fake_M)
+---
+...
+err:match('auto%-downgrade is not implemented')
+---
+- auto-downgrade is not implemented
+...
+test_run:grep_log('default', 'vshard.storage.evolution', 1000) ~= nil
+---
+- false
+...
diff --git a/test/unit/reload_evolution.test.lua b/test/unit/reload_evolution.test.lua
new file mode 100644
index 0000000..c0fcdcd
--- /dev/null
+++ b/test/unit/reload_evolution.test.lua
@@ -0,0 +1,18 @@
+test_run = require('test_run').new()
+fiber = require('fiber')
+log = require('log')
+util = require('util')
+reload_evolution = require('vshard.storage.reload_evolution')
+-- Init with the latest version.
+fake_M = { reload_evolution_version = reload_evolution.version }
+
+-- Test reload to the same version.
+reload_evolution.upgrade(fake_M)
+test_run:grep_log('default', 'vshard.storage.evolution') == nil
+
+-- Test downgrage version.
+log.info(string.rep('a', 1000))
+fake_M.reload_evolution_version = fake_M.reload_evolution_version + 1
+err = util.check_error(reload_evolution.upgrade, fake_M)
+err:match('auto%-downgrade is not implemented')
+test_run:grep_log('default', 'vshard.storage.evolution', 1000) ~= nil
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index 07bd00c..3bec09f 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -10,6 +10,7 @@ if rawget(_G, MODULE_INTERNALS) then
local vshard_modules = {
'vshard.consts', 'vshard.error', 'vshard.cfg',
'vshard.replicaset', 'vshard.util',
+ 'vshard.storage.reload_evolution'
}
for _, module in pairs(vshard_modules) do
package.loaded[module] = nil
@@ -20,12 +21,16 @@ local lerror = require('vshard.error')
local lcfg = require('vshard.cfg')
local lreplicaset = require('vshard.replicaset')
local util = require('vshard.util')
+local reload_evolution = require('vshard.storage.reload_evolution')
local M = rawget(_G, MODULE_INTERNALS)
if not M then
--
-- The module is loaded for the first time.
--
+ -- !!!WARNING: any change of this table must be reflected in
+ -- `vshard.storage.reload_evolution` module to guarantee
+ -- reloadability of the module.
M = {
---------------- Common module attributes ----------------
-- The last passed configuration.
@@ -105,6 +110,11 @@ if not M then
-- a destination replicaset must drop already received
-- data.
rebalancer_sending_bucket = 0,
+
+ ------------------------- Reload -------------------------
+ -- Version of the loaded module. This number is used on
+ -- reload to determine which upgrade scripts to run.
+ reload_evolution_version = reload_evolution.version,
}
end
@@ -1863,6 +1873,7 @@ end
if not rawget(_G, MODULE_INTERNALS) then
rawset(_G, MODULE_INTERNALS, M)
else
+ reload_evolution.upgrade(M)
storage_cfg(M.current_cfg, M.this_replica.uuid)
M.module_version = M.module_version + 1
end
diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua
new file mode 100644
index 0000000..f25ad49
--- /dev/null
+++ b/vshard/storage/reload_evolution.lua
@@ -0,0 +1,58 @@
+--
+-- This module is used to upgrade the vshard.storage on the fly.
+-- It updates internal Lua structures in case they are changed
+-- in a commit.
+--
+local log = require('log')
+
+--
+-- Array of upgrade functions.
+-- migrations[version] = function which upgrades module version
+-- from `version` to `version + 1`.
+--
+local migrations = {}
+
+-- Initialize reload_upgrade mechanism
+migrations[#migrations + 1] = function (M)
+ -- Code to update Lua objects.
+end
+
+--
+-- Perform an update based on a version stored in `M` (internals).
+-- @param M Old module internals which should be updated.
+--
+local function upgrade(M)
+ local start_version = M.reload_evolution_version or 1
+ if start_version > #migrations then
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'auto-downgrade is not implemented; ' ..
+ 'loaded version is %d, upgrade script version is %d',
+ start_version, #migrations
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ for i = start_version, #migrations do
+ local ok, err = pcall(migrations[i], M)
+ if ok then
+ log.info('vshard.storage.reload_evolution: upgraded to %d version',
+ i)
+ else
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'error during upgrade to %d version: %s', i, err
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ -- Update the version just after upgrade to have an
+ -- actual version in case of an error.
+ M.reload_evolution_version = i
+ end
+end
+
+return {
+ version = #migrations,
+ upgrade = upgrade,
+}
--
2.14.1
^ permalink raw reply [flat|nested] 32+ messages in thread
end of thread, other threads:[~2018-08-08 10:35 UTC | newest]
Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-30 8:56 [tarantool-patches] [PATCH v4] vshard module reload AKhatskevich
2018-07-30 8:56 ` [tarantool-patches] [PATCH 1/4] Fix races related to object outdating AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-30 16:46 ` Alex Khatskevich
2018-07-30 17:50 ` Vladislav Shpilevoy
2018-07-31 11:05 ` Alex Khatskevich
2018-08-01 12:36 ` Vladislav Shpilevoy
2018-08-01 17:44 ` Alex Khatskevich
2018-08-02 11:51 ` Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 2/4] Refactor reloadable fiber AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-31 11:24 ` Alex Khatskevich
2018-07-31 11:30 ` Alex Khatskevich
2018-08-01 11:54 ` Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 3/4] tests: separate bootstrap routine to a lua_libs AKhatskevich
2018-08-01 12:03 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-30 8:56 ` [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution AKhatskevich
2018-07-30 11:55 ` [tarantool-patches] " Vladislav Shpilevoy
2018-07-31 11:29 ` Alex Khatskevich
2018-07-31 11:33 ` Alex Khatskevich
2018-08-01 12:36 ` Vladislav Shpilevoy
2018-08-01 18:09 ` Alex Khatskevich
2018-08-02 11:40 ` Vladislav Shpilevoy
2018-08-02 11:46 ` Vladislav Shpilevoy
2018-08-06 10:59 ` Alex Khatskevich
2018-08-06 15:36 ` Vladislav Shpilevoy
2018-08-06 16:08 ` Alex Khatskevich
2018-08-06 17:18 ` Vladislav Shpilevoy
2018-08-07 9:14 ` Alex Khatskevich
2018-08-08 10:35 ` Vladislav Shpilevoy
2018-08-01 14:07 ` [tarantool-patches] [PATCH] Check self arg passed for router objects AKhatskevich
-- strict thread matches above, loose matches on Subject: below --
2018-07-23 11:14 [tarantool-patches] [PATCH v2] vshard reload mechanism AKhatskevich
2018-07-23 11:14 ` [tarantool-patches] [PATCH 4/4] Introduce storage reload evolution AKhatskevich
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox