[tarantool-patches] Re: [PATCH 3/3] Introduce multiple routers feature
Alex Khatskevich
avkhatskevich at tarantool.org
Fri Aug 3 23:05:11 MSK 2018
On 01.08.2018 21:43, Vladislav Shpilevoy wrote:
> Thanks for the patch! See 10 comments below.
>
> On 31/07/2018 19:25, AKhatskevich wrote:
>> Key points:
>> * Old `vshard.router.some_method()` api is preserved.
>> * Add `vshard.router.new(name, cfg)` method which returns a new router.
>> * Each router has its own:
>> 1. name
>> 2. background fibers
>> 3. attributes (route_map, replicasets, outdate_delay...)
>> * Module reload reloads all configured routers.
>> * `cfg` reconfigures a single router.
>> * All routers share the same box configuration. The last passed config
>> overrides the global config.
>> * Multiple router instances can be connected to the same cluster.
>> * By now, a router cannot be destroyed.
>>
>> Extra changes:
>> * Add `data` parameter to `reloadable_fiber_create` function.
>>
>> Closes #130
>> ---> diff --git a/test/multiple_routers/multiple_routers.result
>> b/test/multiple_routers/multiple_routers.result
>> new file mode 100644
>> index 0000000..33f4034
>> --- /dev/null
>> +++ b/test/multiple_routers/multiple_routers.result
>> @@ -0,0 +1,226 @@
>> +-- Reconfigure one of routers do not affect the others.
>> +routers[3]:cfg(configs.cfg_1)
>
> 1. You did not change configs.cfg_1 so it is not reconfig
> actually. Please, change something to check that the
> parameter affects one router and does not affect others.
router[3] was configured with configs.cfg_2 before.
So, its config was changed.
>
> 2. Please, add a test on an ability to get the static router
> into a variable and use it like others. It should be possible
> to hide distinctions between static and other routers.
>
> Like this:
>
> r1 = vshard.router.static
> r2 = vshard.router.new(...)
> do_something_with_router(r1)
> do_something_with_router(r2)
>
> Here do_something_with_router() is unaware of whether the
> router is static or not.
>
Few calls are added.
>> diff --git a/vshard/router/init.lua b/vshard/router/init.lua
>> index 3e127cb..7569baf 100644
>> --- a/vshard/router/init.lua
>> +++ b/vshard/router/init.lua
>> @@ -257,13 +272,13 @@ local function router_call(bucket_id, mode,
>> func, args, opts)
>> -- but already is executed on storages.
>> while lfiber.time() <= tend do
>> lfiber.sleep(0.05)
>> - replicaset = M.replicasets[err.destination]
>> + replicaset =
>> router.replicasets[err.destination]
>> if replicaset then
>> goto replicaset_is_found
>> end
>> end
>> else
>> - replicaset = bucket_set(bucket_id,
>> replicaset.uuid)
>> + replicaset = bucket_set(router, bucket_id,
>> replicaset.uuid)
>
> 3. Out of 80 symbols.
fixed
>
>> lfiber.yield()
>> -- Protect against infinite cycle in a
>> -- case of broken cluster, when a bucket
>> @@ -488,9 +503,14 @@ end
>> -- Configuration
>> --------------------------------------------------------------------------------
>> -local function router_cfg(cfg)
>> - local vshard_cfg, box_cfg = lcfg.check(cfg, M.current_cfg)
>> - if not M.replicasets then
>> +-- Types of configuration.
>> +CFG_NEW = 'new'
>> +CFG_RELOAD = 'reload'
>> +CFG_RECONFIGURE = 'reconfigure'
>
> 4. Last two values are never used in router_cfg(). The first
> is used for logging only and can be checked as it was before
> with no explicit passing.
I have left it as it is.
Now, each of those is passed at least once.
>> +
>> +local function router_cfg(router, cfg, cfg_type)
>> + local vshard_cfg, box_cfg = lcfg.check(cfg, router.current_cfg)
>> + if cfg_type == CFG_NEW then
>> log.info('Starting router configuration')
>> else
>> log.info('Starting router reconfiguration')
>> @@ -512,44 +532,53 @@ local function router_cfg(cfg)
>> +
>> +local function updage_lua_gc_state()
>
> 5. This function is not needed actually.
>
> On router_new() the only thing that can change is start of
> the gc fiber if the new router has the flag and the gc is
> not started now. It can be checked by a simple 'if' with
> no full-scan of all routers.
>
> On reload it is not possible to change configuration, so
> the gc state can not be changed and does not need an update.
> Even if it could be changed, you already iterate over routers
> on reload to call router_cfg and can collect their flags
> along side.
>
> The next point is that it is not possible now to manage the
> gc via simple :cfg() call. You do nothing with gc when
> router_cfg is called directly. And that produces a question -
> why do your tests pass if so?
>
> The possible solution - keep a counter of set lua gc flags
> overall routers in M. On each cfg you update the counter
> if the value is changed. If it was 0 and become > 0, then
> you start gc. If it was > 0 and become 0, then you stop gc.
> No routers iteration at all.
Implemented by introducing a counter.
>
>> + local lua_gc = false
>> + for _, xrouter in pairs(M.routers) do
>> + if xrouter.collect_lua_garbage then
>> + lua_gc = true
>> + end
>> + end
>> + lua_gc.set_state(lua_gc, consts.COLLECT_LUA_GARBAGE_INTERVAL)
>> end
>> @@ -803,6 +833,93 @@ if M.errinj.ERRINJ_RELOAD then
>> error('Error injection: reload')
>> end
>> +--------------------------------------------------------------------------------
>> +-- Managing router instances
>> +--------------------------------------------------------------------------------
>>
>> +
>> +local function cfg_reconfigure(router, cfg)
>> + return router_cfg(router, cfg, CFG_RECONFIGURE)
>> +end
>> +
>> +local router_mt = {
>> + __index = {
>> + cfg = cfg_reconfigure;
>> + info = router_info;
>> + buckets_info = router_buckets_info;
>> + call = router_call;
>> + callro = router_callro;
>> + callrw = router_callrw;
>> + route = router_route;
>> + routeall = router_routeall;
>> + bucket_id = router_bucket_id;
>> + bucket_count = router_bucket_count;
>> + sync = router_sync;
>> + bootstrap = cluster_bootstrap;
>> + bucket_discovery = bucket_discovery;
>> + discovery_wakeup = discovery_wakeup;
>> + }
>> +}
>> +
>> +-- Table which represents this module.
>> +local module = {}
>> +
>> +local function export_static_router_attributes()
>> + -- This metatable bypasses calls to a module to the static_router.
>> + local module_mt = {__index = {}}
>> + for method_name, method in pairs(router_mt.__index) do
>> + module_mt.__index[method_name] = function(...)
>> + if M.static_router then
>> + return method(M.static_router, ...)
>> + else
>> + error('Static router is not configured')
>
> 6. This should not be all-time check. You should
> initialize the static router metatable with only errors.
> On the first cfg you reset the metatable to always use
> regular methods. But anyway this code is unreachable. See
> below in the comment 10 why it is so.
Yes. Fixed.
>
>> + end
>> + end
>> + end
>> + setmetatable(module, module_mt)
>> + -- Make static_router attributes accessible form
>> + -- vshard.router.internal.
>> + local M_static_router_attributes = {
>> + name = true,
>> + replicasets = true,
>> + route_map = true,
>> + total_bucket_count = true,
>> + }
>
> 7. I saw in the tests that you are using
> vshard.router.internal.static_router
> instead. Please, remove M_static_router_attributes then.
Deleted. Tests are fixed.
>
>> + setmetatable(M, {
>> + __index = function(M, key)
>> + return M.static_router[key]
>> + end
>> + })
>> +end
>> +
>> +local function router_new(name, cfg)
>> + assert(type(name) == 'string' and type(cfg) == 'table',
>> + 'Wrong argument type. Usage: vshard.router.new(name, cfg).')
>
> 8. As I said before, do not use assertions for usage checks in public
> API. Use 'if wrong_usage then error(...) end'.
Fixed.
>
>> + if M.routers[name] then
>> + return nil, string.format('Router with name %s already
>> exists', name)
>> + end
>> + local router = table.deepcopy(ROUTER_TEMPLATE)
>> + setmetatable(router, router_mt)
>> + router.name = name
>> + M.routers[name] = router
>> + if name == STATIC_ROUTER_NAME then
>> + M.static_router = router
>> + export_static_router_attributes()
>> + end
>
> 9. This check can be removed if you move
> export_static_router_attributes call into legacy_cfg.
Butbue to this if, the static router can be configured by
`vshard.box.new(static_router_name)`.
>
> 10. Looks like all your struggles in
> export_static_router_attributes() about error on non-configured
> router makes no sense since until cfg is called, vshard.router
> has no any methods except cfg and new.
>
>> + router_cfg(router, cfg, CFG_NEW)
>> + updage_lua_gc_state()
>> + return router
>> +end
>> +
Fixed.
full diff
commit f3ffb6a6a3632277f05ee4ea7d095a19dd85a42f
Author: AKhatskevich <avkhatskevich at tarantool.org>
Date: Thu Jul 26 16:17:25 2018 +0300
Introduce multiple routers feature
Key points:
* Old `vshard.router.some_method()` api is preserved.
* Add `vshard.router.new(name, cfg)` method which returns a new router.
* Each router has its own:
1. name
2. background fibers
3. attributes (route_map, replicasets, outdate_delay...)
* Module reload reloads all configured routers.
* `cfg` reconfigures a single router.
* All routers share the same box configuration. The last passed config
overrides the global box config.
* Multiple router instances can be connected to the same cluster.
* By now, a router cannot be destroyed.
Extra changes:
* Add `data` parameter to `reloadable_fiber_create` function.
Closes #130
diff --git a/test/failover/failover.result b/test/failover/failover.result
index 73a4250..50410ad 100644
--- a/test/failover/failover.result
+++ b/test/failover/failover.result
@@ -174,7 +174,7 @@ test_run:switch('router_1')
---
- true
...
-rs1 = vshard.router.internal.replicasets[rs_uuid[1]]
+rs1 = vshard.router.internal.static_router.replicasets[rs_uuid[1]]
---
...
while not rs1.replica_up_ts do fiber.sleep(0.1) end
diff --git a/test/failover/failover.test.lua
b/test/failover/failover.test.lua
index 6e06314..44c8b6d 100644
--- a/test/failover/failover.test.lua
+++ b/test/failover/failover.test.lua
@@ -74,7 +74,7 @@ echo_count
-- Ensure that replica_up_ts is updated periodically.
test_run:switch('router_1')
-rs1 = vshard.router.internal.replicasets[rs_uuid[1]]
+rs1 = vshard.router.internal.static_router.replicasets[rs_uuid[1]]
while not rs1.replica_up_ts do fiber.sleep(0.1) end
old_up_ts = rs1.replica_up_ts
while rs1.replica_up_ts == old_up_ts do fiber.sleep(0.1) end
diff --git a/test/failover/failover_errinj.result
b/test/failover/failover_errinj.result
index 3b6d986..484a1e3 100644
--- a/test/failover/failover_errinj.result
+++ b/test/failover/failover_errinj.result
@@ -49,7 +49,7 @@ vshard.router.cfg(cfg)
-- Check that already run failover step is restarted on
-- configuration change (if some replicasets are removed from
-- config).
-rs1 = vshard.router.internal.replicasets[rs_uuid[1]]
+rs1 = vshard.router.internal.static_router.replicasets[rs_uuid[1]]
---
...
while not rs1.replica or not rs1.replica.conn:is_connected() do
fiber.sleep(0.1) end
diff --git a/test/failover/failover_errinj.test.lua
b/test/failover/failover_errinj.test.lua
index b4d2d35..14228de 100644
--- a/test/failover/failover_errinj.test.lua
+++ b/test/failover/failover_errinj.test.lua
@@ -20,7 +20,7 @@ vshard.router.cfg(cfg)
-- Check that already run failover step is restarted on
-- configuration change (if some replicasets are removed from
-- config).
-rs1 = vshard.router.internal.replicasets[rs_uuid[1]]
+rs1 = vshard.router.internal.static_router.replicasets[rs_uuid[1]]
while not rs1.replica or not rs1.replica.conn:is_connected() do
fiber.sleep(0.1) end
vshard.router.internal.errinj.ERRINJ_FAILOVER_CHANGE_CFG = true
wait_state('Configuration has changed, restart ')
diff --git a/test/failover/router_1.lua b/test/failover/router_1.lua
index d71209b..664a6c6 100644
--- a/test/failover/router_1.lua
+++ b/test/failover/router_1.lua
@@ -42,7 +42,7 @@ end
function priority_order()
local ret = {}
for _, uuid in pairs(rs_uuid) do
- local rs = vshard.router.internal.replicasets[uuid]
+ local rs = vshard.router.internal.static_router.replicasets[uuid]
local sorted = {}
for _, replica in pairs(rs.priority_list) do
local z
diff --git a/test/misc/reconfigure.result b/test/misc/reconfigure.result
index c7960b3..311f749 100644
--- a/test/misc/reconfigure.result
+++ b/test/misc/reconfigure.result
@@ -250,7 +250,7 @@ test_run:switch('router_1')
-- Ensure that in a case of error router internals are not
-- changed.
--
-not vshard.router.internal.collect_lua_garbage
+not vshard.router.internal.static_router.collect_lua_garbage
---
- true
...
@@ -264,7 +264,7 @@ vshard.router.cfg(cfg)
---
- error: 'Incorrect value for option ''invalid_option'': unexpected
option'
...
-not vshard.router.internal.collect_lua_garbage
+not vshard.router.internal.static_router.collect_lua_garbage
---
- true
...
diff --git a/test/misc/reconfigure.test.lua b/test/misc/reconfigure.test.lua
index 25dc2ca..298b9b0 100644
--- a/test/misc/reconfigure.test.lua
+++ b/test/misc/reconfigure.test.lua
@@ -99,11 +99,11 @@ test_run:switch('router_1')
-- Ensure that in a case of error router internals are not
-- changed.
--
-not vshard.router.internal.collect_lua_garbage
+not vshard.router.internal.static_router.collect_lua_garbage
cfg.collect_lua_garbage = true
cfg.invalid_option = 'kek'
vshard.router.cfg(cfg)
-not vshard.router.internal.collect_lua_garbage
+not vshard.router.internal.static_router.collect_lua_garbage
cfg.invalid_option = nil
cfg.collect_lua_garbage = nil
vshard.router.cfg(cfg)
diff --git a/test/multiple_routers/configs.lua
b/test/multiple_routers/configs.lua
new file mode 100644
index 0000000..a6ce33c
--- /dev/null
+++ b/test/multiple_routers/configs.lua
@@ -0,0 +1,81 @@
+names = {
+ storage_1_1_a = '32a2d4b8-f146-44ed-9d51-2436507efdf8',
+ storage_1_1_b = 'c1c849b1-641d-40b8-9283-bcfe73d46270',
+ storage_1_2_a = '04e677ed-c7ba-47e0-a67f-b5100cfa86af',
+ storage_1_2_b = 'c7a979ee-9263-4a38-84a5-2fb6a0a32684',
+ storage_2_1_a = '88dc03f0-23fb-4f05-b462-e29186542864',
+ storage_2_1_b = '4230b711-f5c4-4131-bf98-88cd43a16901',
+ storage_2_2_a = '6b1eefbc-1e2e-410e-84ff-44c572ea9916',
+ storage_2_2_b = 'be74419a-1e56-4ba4-97e9-6b18710f63c5',
+}
+
+rs_1_1 = 'dd208fb8-8b90-49bc-8393-6b3a99da7c52'
+rs_1_2 = 'af9cfe88-2091-4613-a877-a623776c5c0e'
+rs_2_1 = '9ca8ee15-ae18-4f31-9385-4859f89ce73f'
+rs_2_2 = '007f5f58-b654-4125-8441-a71866fb62b5'
+
+local cfg_1 = {}
+cfg_1.sharding = {
+ [rs_1_1] = {
+ replicas = {
+ [names.storage_1_1_a] = {
+ uri = 'storage:storage at 127.0.0.1:3301',
+ name = 'storage_1_1_a',
+ master = true,
+ },
+ [names.storage_1_1_b] = {
+ uri = 'storage:storage at 127.0.0.1:3302',
+ name = 'storage_1_1_b',
+ },
+ }
+ },
+ [rs_1_2] = {
+ replicas = {
+ [names.storage_1_2_a] = {
+ uri = 'storage:storage at 127.0.0.1:3303',
+ name = 'storage_1_2_a',
+ master = true,
+ },
+ [names.storage_1_2_b] = {
+ uri = 'storage:storage at 127.0.0.1:3304',
+ name = 'storage_1_2_b',
+ },
+ }
+ },
+}
+
+
+local cfg_2 = {}
+cfg_2.sharding = {
+ [rs_2_1] = {
+ replicas = {
+ [names.storage_2_1_a] = {
+ uri = 'storage:storage at 127.0.0.1:3305',
+ name = 'storage_2_1_a',
+ master = true,
+ },
+ [names.storage_2_1_b] = {
+ uri = 'storage:storage at 127.0.0.1:3306',
+ name = 'storage_2_1_b',
+ },
+ }
+ },
+ [rs_2_2] = {
+ replicas = {
+ [names.storage_2_2_a] = {
+ uri = 'storage:storage at 127.0.0.1:3307',
+ name = 'storage_2_2_a',
+ master = true,
+ },
+ [names.storage_2_2_b] = {
+ uri = 'storage:storage at 127.0.0.1:3308',
+ name = 'storage_2_2_b',
+ },
+ }
+ },
+}
+
+return {
+ cfg_1 = cfg_1,
+ cfg_2 = cfg_2,
+}
diff --git a/test/multiple_routers/multiple_routers.result
b/test/multiple_routers/multiple_routers.result
new file mode 100644
index 0000000..1e309a7
--- /dev/null
+++ b/test/multiple_routers/multiple_routers.result
@@ -0,0 +1,295 @@
+test_run = require('test_run').new()
+---
+...
+REPLICASET_1_1 = { 'storage_1_1_a', 'storage_1_1_b' }
+---
+...
+REPLICASET_1_2 = { 'storage_1_2_a', 'storage_1_2_b' }
+---
+...
+REPLICASET_2_1 = { 'storage_2_1_a', 'storage_2_1_b' }
+---
+...
+REPLICASET_2_2 = { 'storage_2_2_a', 'storage_2_2_b' }
+---
+...
+test_run:create_cluster(REPLICASET_1_1, 'multiple_routers')
+---
+...
+test_run:create_cluster(REPLICASET_1_2, 'multiple_routers')
+---
+...
+test_run:create_cluster(REPLICASET_2_1, 'multiple_routers')
+---
+...
+test_run:create_cluster(REPLICASET_2_2, 'multiple_routers')
+---
+...
+util = require('lua_libs.util')
+---
+...
+util.wait_master(test_run, REPLICASET_1_1, 'storage_1_1_a')
+---
+...
+util.wait_master(test_run, REPLICASET_1_2, 'storage_1_2_a')
+---
+...
+util.wait_master(test_run, REPLICASET_2_1, 'storage_2_1_a')
+---
+...
+util.wait_master(test_run, REPLICASET_2_2, 'storage_2_2_a')
+---
+...
+test_run:cmd("create server router_1 with
script='multiple_routers/router_1.lua'")
+---
+- true
+...
+test_run:cmd("start server router_1")
+---
+- true
+...
+-- Configure default (static) router.
+_ = test_run:cmd("switch router_1")
+---
+...
+static_router = vshard.router.new('_static_router', configs.cfg_1)
+---
+...
+vshard.router.bootstrap()
+---
+- true
+...
+_ = test_run:cmd("switch storage_1_2_a")
+---
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+_ = test_run:cmd("switch router_1")
+---
+...
+vshard.router.call(1, 'write', 'do_replace', {{1, 1}})
+---
+- true
+...
+vshard.router.call(1, 'read', 'do_select', {1})
+---
+- [[1, 1]]
+...
+-- Test that static router is just a router object under the hood.
+static_router:route(1) == vshard.router.route(1)
+---
+- true
+...
+-- Configure extra router.
+router_2 = vshard.router.new('router_2', configs.cfg_2)
+---
+...
+router_2:bootstrap()
+---
+- true
+...
+_ = test_run:cmd("switch storage_2_2_a")
+---
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+_ = test_run:cmd("switch router_1")
+---
+...
+router_2:call(1, 'write', 'do_replace', {{2, 2}})
+---
+- true
+...
+router_2:call(1, 'read', 'do_select', {2})
+---
+- [[2, 2]]
+...
+-- Check that router_2 and static router serves different clusters.
+#router_2:call(1, 'read', 'do_select', {1}) == 0
+---
+- true
+...
+-- Create several routers to the same cluster.
+routers = {}
+---
+...
+for i = 3, 10 do routers[i] = vshard.router.new('router_' .. i,
configs.cfg_2) end
+---
+...
+routers[3]:call(1, 'read', 'do_select', {2})
+---
+- [[2, 2]]
+...
+-- Check that they have their own background fibers.
+fiber_names = {}
+---
+...
+for i = 2, 10 do fiber_names['vshard.failover.router_' .. i] = true;
fiber_names['vshard.discovery.router_' .. i] = true; end
+---
+...
+next(fiber_names) ~= nil
+---
+- true
+...
+fiber = require('fiber')
+---
+...
+for _, xfiber in pairs(fiber.info()) do fiber_names[xfiber.name] = nil end
+---
+...
+next(fiber_names) == nil
+---
+- true
+...
+-- Reconfigure one of routers do not affect the others.
+routers[3]:cfg(configs.cfg_1)
+---
+...
+routers[3]:call(1, 'read', 'do_select', {1})
+---
+- [[1, 1]]
+...
+#routers[3]:call(1, 'read', 'do_select', {2}) == 0
+---
+- true
+...
+#routers[4]:call(1, 'read', 'do_select', {1}) == 0
+---
+- true
+...
+routers[4]:call(1, 'read', 'do_select', {2})
+---
+- [[2, 2]]
+...
+routers[3]:cfg(configs.cfg_2)
+---
+...
+-- Try to create router with the same name.
+util = require('lua_libs.util')
+---
+...
+util.check_error(vshard.router.new, 'router_2', configs.cfg_2)
+---
+- null
+- Router with name router_2 already exists
+...
+-- Reload router module.
+_, old_rs_1 = next(vshard.router.internal.static_router.replicasets)
+---
+...
+_, old_rs_2 = next(router_2.replicasets)
+---
+...
+package.loaded['vshard.router'] = nil
+---
+...
+vshard.router = require('vshard.router')
+---
+...
+while not old_rs_1.is_outdated do fiber.sleep(0.01) end
+---
+...
+while not old_rs_2.is_outdated do fiber.sleep(0.01) end
+---
+...
+vshard.router.call(1, 'read', 'do_select', {1})
+---
+- [[1, 1]]
+...
+router_2:call(1, 'read', 'do_select', {2})
+---
+- [[2, 2]]
+...
+routers[5]:call(1, 'read', 'do_select', {2})
+---
+- [[2, 2]]
+...
+-- Check lua_gc counter.
+lua_gc = require('vshard.lua_gc')
+---
+...
+vshard.router.internal.collect_lua_garbage_cnt == 0
+---
+- true
+...
+lua_gc.internal.bg_fiber == nil
+---
+- true
+...
+configs.cfg_2.collect_lua_garbage = true
+---
+...
+routers[5]:cfg(configs.cfg_2)
+---
+...
+lua_gc.internal.bg_fiber ~= nil
+---
+- true
+...
+routers[7]:cfg(configs.cfg_2)
+---
+...
+lua_gc.internal.bg_fiber ~= nil
+---
+- true
+...
+vshard.router.internal.collect_lua_garbage_cnt == 2
+---
+- true
+...
+package.loaded['vshard.router'] = nil
+---
+...
+vshard.router = require('vshard.router')
+---
+...
+vshard.router.internal.collect_lua_garbage_cnt == 2
+---
+- true
+...
+configs.cfg_2.collect_lua_garbage = nil
+---
+...
+routers[5]:cfg(configs.cfg_2)
+---
+...
+lua_gc.internal.bg_fiber ~= nil
+---
+- true
+...
+routers[7]:cfg(configs.cfg_2)
+---
+...
+vshard.router.internal.collect_lua_garbage_cnt == 0
+---
+- true
+...
+lua_gc.internal.bg_fiber == nil
+---
+- true
+...
+_ = test_run:cmd("switch default")
+---
+...
+test_run:cmd("stop server router_1")
+---
+- true
+...
+test_run:cmd("cleanup server router_1")
+---
+- true
+...
+test_run:drop_cluster(REPLICASET_1_1)
+---
+...
+test_run:drop_cluster(REPLICASET_1_2)
+---
+...
+test_run:drop_cluster(REPLICASET_2_1)
+---
+...
+test_run:drop_cluster(REPLICASET_2_2)
+---
+...
diff --git a/test/multiple_routers/multiple_routers.test.lua
b/test/multiple_routers/multiple_routers.test.lua
new file mode 100644
index 0000000..760ad9f
--- /dev/null
+++ b/test/multiple_routers/multiple_routers.test.lua
@@ -0,0 +1,108 @@
+test_run = require('test_run').new()
+
+REPLICASET_1_1 = { 'storage_1_1_a', 'storage_1_1_b' }
+REPLICASET_1_2 = { 'storage_1_2_a', 'storage_1_2_b' }
+REPLICASET_2_1 = { 'storage_2_1_a', 'storage_2_1_b' }
+REPLICASET_2_2 = { 'storage_2_2_a', 'storage_2_2_b' }
+
+test_run:create_cluster(REPLICASET_1_1, 'multiple_routers')
+test_run:create_cluster(REPLICASET_1_2, 'multiple_routers')
+test_run:create_cluster(REPLICASET_2_1, 'multiple_routers')
+test_run:create_cluster(REPLICASET_2_2, 'multiple_routers')
+util = require('lua_libs.util')
+util.wait_master(test_run, REPLICASET_1_1, 'storage_1_1_a')
+util.wait_master(test_run, REPLICASET_1_2, 'storage_1_2_a')
+util.wait_master(test_run, REPLICASET_2_1, 'storage_2_1_a')
+util.wait_master(test_run, REPLICASET_2_2, 'storage_2_2_a')
+
+test_run:cmd("create server router_1 with
script='multiple_routers/router_1.lua'")
+test_run:cmd("start server router_1")
+
+-- Configure default (static) router.
+_ = test_run:cmd("switch router_1")
+static_router = vshard.router.new('_static_router', configs.cfg_1)
+vshard.router.bootstrap()
+_ = test_run:cmd("switch storage_1_2_a")
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+_ = test_run:cmd("switch router_1")
+
+vshard.router.call(1, 'write', 'do_replace', {{1, 1}})
+vshard.router.call(1, 'read', 'do_select', {1})
+
+-- Test that static router is just a router object under the hood.
+static_router:route(1) == vshard.router.route(1)
+
+-- Configure extra router.
+router_2 = vshard.router.new('router_2', configs.cfg_2)
+router_2:bootstrap()
+_ = test_run:cmd("switch storage_2_2_a")
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+_ = test_run:cmd("switch router_1")
+
+router_2:call(1, 'write', 'do_replace', {{2, 2}})
+router_2:call(1, 'read', 'do_select', {2})
+-- Check that router_2 and static router serves different clusters.
+#router_2:call(1, 'read', 'do_select', {1}) == 0
+
+-- Create several routers to the same cluster.
+routers = {}
+for i = 3, 10 do routers[i] = vshard.router.new('router_' .. i,
configs.cfg_2) end
+routers[3]:call(1, 'read', 'do_select', {2})
+-- Check that they have their own background fibers.
+fiber_names = {}
+for i = 2, 10 do fiber_names['vshard.failover.router_' .. i] = true;
fiber_names['vshard.discovery.router_' .. i] = true; end
+next(fiber_names) ~= nil
+fiber = require('fiber')
+for _, xfiber in pairs(fiber.info()) do fiber_names[xfiber.name] = nil end
+next(fiber_names) == nil
+
+-- Reconfigure one of routers do not affect the others.
+routers[3]:cfg(configs.cfg_1)
+routers[3]:call(1, 'read', 'do_select', {1})
+#routers[3]:call(1, 'read', 'do_select', {2}) == 0
+#routers[4]:call(1, 'read', 'do_select', {1}) == 0
+routers[4]:call(1, 'read', 'do_select', {2})
+routers[3]:cfg(configs.cfg_2)
+
+-- Try to create router with the same name.
+util = require('lua_libs.util')
+util.check_error(vshard.router.new, 'router_2', configs.cfg_2)
+
+-- Reload router module.
+_, old_rs_1 = next(vshard.router.internal.static_router.replicasets)
+_, old_rs_2 = next(router_2.replicasets)
+package.loaded['vshard.router'] = nil
+vshard.router = require('vshard.router')
+while not old_rs_1.is_outdated do fiber.sleep(0.01) end
+while not old_rs_2.is_outdated do fiber.sleep(0.01) end
+vshard.router.call(1, 'read', 'do_select', {1})
+router_2:call(1, 'read', 'do_select', {2})
+routers[5]:call(1, 'read', 'do_select', {2})
+
+-- Check lua_gc counter.
+lua_gc = require('vshard.lua_gc')
+vshard.router.internal.collect_lua_garbage_cnt == 0
+lua_gc.internal.bg_fiber == nil
+configs.cfg_2.collect_lua_garbage = true
+routers[5]:cfg(configs.cfg_2)
+lua_gc.internal.bg_fiber ~= nil
+routers[7]:cfg(configs.cfg_2)
+lua_gc.internal.bg_fiber ~= nil
+vshard.router.internal.collect_lua_garbage_cnt == 2
+package.loaded['vshard.router'] = nil
+vshard.router = require('vshard.router')
+vshard.router.internal.collect_lua_garbage_cnt == 2
+configs.cfg_2.collect_lua_garbage = nil
+routers[5]:cfg(configs.cfg_2)
+lua_gc.internal.bg_fiber ~= nil
+routers[7]:cfg(configs.cfg_2)
+vshard.router.internal.collect_lua_garbage_cnt == 0
+lua_gc.internal.bg_fiber == nil
+
+_ = test_run:cmd("switch default")
+test_run:cmd("stop server router_1")
+test_run:cmd("cleanup server router_1")
+test_run:drop_cluster(REPLICASET_1_1)
+test_run:drop_cluster(REPLICASET_1_2)
+test_run:drop_cluster(REPLICASET_2_1)
+test_run:drop_cluster(REPLICASET_2_2)
diff --git a/test/multiple_routers/router_1.lua
b/test/multiple_routers/router_1.lua
new file mode 100644
index 0000000..2e9ea91
--- /dev/null
+++ b/test/multiple_routers/router_1.lua
@@ -0,0 +1,15 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+-- Get instance name
+local fio = require('fio')
+local NAME = fio.basename(arg[0], '.lua')
+
+require('console').listen(os.getenv('ADMIN'))
+
+configs = require('configs')
+
+-- Start the database with sharding
+vshard = require('vshard')
+box.cfg{}
diff --git a/test/multiple_routers/storage_1_1_a.lua
b/test/multiple_routers/storage_1_1_a.lua
new file mode 100644
index 0000000..b44a97a
--- /dev/null
+++ b/test/multiple_routers/storage_1_1_a.lua
@@ -0,0 +1,23 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+-- Get instance name.
+local fio = require('fio')
+NAME = fio.basename(arg[0], '.lua')
+
+require('console').listen(os.getenv('ADMIN'))
+
+-- Fetch config for the cluster of the instance.
+if NAME:sub(9,9) == '1' then
+ cfg = require('configs').cfg_1
+else
+ cfg = require('configs').cfg_2
+end
+
+-- Start the database with sharding.
+vshard = require('vshard')
+vshard.storage.cfg(cfg, names[NAME])
+
+-- Bootstrap storage.
+require('lua_libs.bootstrap')
diff --git a/test/multiple_routers/storage_1_1_b.lua
b/test/multiple_routers/storage_1_1_b.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_1_1_b.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_1_2_a.lua
b/test/multiple_routers/storage_1_2_a.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_1_2_a.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_1_2_b.lua
b/test/multiple_routers/storage_1_2_b.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_1_2_b.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_2_1_a.lua
b/test/multiple_routers/storage_2_1_a.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_2_1_a.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_2_1_b.lua
b/test/multiple_routers/storage_2_1_b.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_2_1_b.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_2_2_a.lua
b/test/multiple_routers/storage_2_2_a.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_2_2_a.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/storage_2_2_b.lua
b/test/multiple_routers/storage_2_2_b.lua
new file mode 120000
index 0000000..76d196b
--- /dev/null
+++ b/test/multiple_routers/storage_2_2_b.lua
@@ -0,0 +1 @@
+storage_1_1_a.lua
\ No newline at end of file
diff --git a/test/multiple_routers/suite.ini
b/test/multiple_routers/suite.ini
new file mode 100644
index 0000000..d2d4470
--- /dev/null
+++ b/test/multiple_routers/suite.ini
@@ -0,0 +1,6 @@
+[default]
+core = tarantool
+description = Multiple routers tests
+script = test.lua
+is_parallel = False
+lua_libs = ../lua_libs configs.lua
diff --git a/test/multiple_routers/test.lua b/test/multiple_routers/test.lua
new file mode 100644
index 0000000..cb7c1ee
--- /dev/null
+++ b/test/multiple_routers/test.lua
@@ -0,0 +1,9 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+box.cfg{
+ listen = os.getenv("LISTEN"),
+}
+
+require('console').listen(os.getenv('ADMIN'))
diff --git a/test/router/exponential_timeout.result
b/test/router/exponential_timeout.result
index fb54d0f..6748b64 100644
--- a/test/router/exponential_timeout.result
+++ b/test/router/exponential_timeout.result
@@ -37,10 +37,10 @@ test_run:cmd('switch router_1')
util = require('util')
---
...
-rs1 = vshard.router.internal.replicasets[replicasets[1]]
+rs1 = vshard.router.internal.static_router.replicasets[replicasets[1]]
---
...
-rs2 = vshard.router.internal.replicasets[replicasets[2]]
+rs2 = vshard.router.internal.static_router.replicasets[replicasets[2]]
---
...
util.collect_timeouts(rs1)
diff --git a/test/router/exponential_timeout.test.lua
b/test/router/exponential_timeout.test.lua
index 3ec0b8c..75d85bf 100644
--- a/test/router/exponential_timeout.test.lua
+++ b/test/router/exponential_timeout.test.lua
@@ -13,8 +13,8 @@ test_run:cmd("start server router_1")
test_run:cmd('switch router_1')
util = require('util')
-rs1 = vshard.router.internal.replicasets[replicasets[1]]
-rs2 = vshard.router.internal.replicasets[replicasets[2]]
+rs1 = vshard.router.internal.static_router.replicasets[replicasets[1]]
+rs2 = vshard.router.internal.static_router.replicasets[replicasets[2]]
util.collect_timeouts(rs1)
util.collect_timeouts(rs2)
diff --git a/test/router/reconnect_to_master.result
b/test/router/reconnect_to_master.result
index 5e678ce..d502723 100644
--- a/test/router/reconnect_to_master.result
+++ b/test/router/reconnect_to_master.result
@@ -76,7 +76,7 @@ _ = test_run:cmd('stop server storage_1_a')
_ = test_run:switch('router_1')
---
...
-reps = vshard.router.internal.replicasets
+reps = vshard.router.internal.static_router.replicasets
---
...
test_run:cmd("setopt delimiter ';'")
@@ -95,7 +95,7 @@ end;
...
function count_known_buckets()
local known_buckets = 0
- for _, id in pairs(vshard.router.internal.route_map) do
+ for _, id in pairs(vshard.router.internal.static_router.route_map) do
known_buckets = known_buckets + 1
end
return known_buckets
@@ -127,7 +127,7 @@ is_disconnected()
fiber = require('fiber')
---
...
-while vshard.router.internal.replicasets[replicasets[1]].replica == nil
do fiber.sleep(0.1) end
+while
vshard.router.internal.static_router.replicasets[replicasets[1]].replica
== nil do fiber.sleep(0.1) end
---
...
vshard.router.info()
diff --git a/test/router/reconnect_to_master.test.lua
b/test/router/reconnect_to_master.test.lua
index 39ba90e..8820fa7 100644
--- a/test/router/reconnect_to_master.test.lua
+++ b/test/router/reconnect_to_master.test.lua
@@ -34,7 +34,7 @@ _ = test_run:cmd('stop server storage_1_a')
_ = test_run:switch('router_1')
-reps = vshard.router.internal.replicasets
+reps = vshard.router.internal.static_router.replicasets
test_run:cmd("setopt delimiter ';'")
function is_disconnected()
for i, rep in pairs(reps) do
@@ -46,7 +46,7 @@ function is_disconnected()
end;
function count_known_buckets()
local known_buckets = 0
- for _, id in pairs(vshard.router.internal.route_map) do
+ for _, id in pairs(vshard.router.internal.static_router.route_map) do
known_buckets = known_buckets + 1
end
return known_buckets
@@ -63,7 +63,7 @@ is_disconnected()
-- Wait until replica is connected to test alerts on unavailable
-- master.
fiber = require('fiber')
-while vshard.router.internal.replicasets[replicasets[1]].replica == nil
do fiber.sleep(0.1) end
+while
vshard.router.internal.static_router.replicasets[replicasets[1]].replica
== nil do fiber.sleep(0.1) end
vshard.router.info()
-- Return master.
diff --git a/test/router/reload.result b/test/router/reload.result
index f0badc3..98e8e71 100644
--- a/test/router/reload.result
+++ b/test/router/reload.result
@@ -229,7 +229,7 @@ vshard.router.cfg(cfg)
cfg.connection_outdate_delay = old_connection_delay
---
...
-vshard.router.internal.connection_outdate_delay = nil
+vshard.router.internal.static_router.connection_outdate_delay = nil
---
...
rs_new = vshard.router.route(1)
diff --git a/test/router/reload.test.lua b/test/router/reload.test.lua
index 528222a..293cb26 100644
--- a/test/router/reload.test.lua
+++ b/test/router/reload.test.lua
@@ -104,7 +104,7 @@ old_connection_delay = cfg.connection_outdate_delay
cfg.connection_outdate_delay = 0.3
vshard.router.cfg(cfg)
cfg.connection_outdate_delay = old_connection_delay
-vshard.router.internal.connection_outdate_delay = nil
+vshard.router.internal.static_router.connection_outdate_delay = nil
rs_new = vshard.router.route(1)
rs_old = rs
_, replica_old = next(rs_old.replicas)
diff --git a/test/router/reroute_wrong_bucket.result
b/test/router/reroute_wrong_bucket.result
index 7f2a494..989dc79 100644
--- a/test/router/reroute_wrong_bucket.result
+++ b/test/router/reroute_wrong_bucket.result
@@ -98,7 +98,7 @@ vshard.router.call(100, 'read', 'customer_lookup',
{1}, {timeout = 100})
---
- {'accounts': [], 'customer_id': 1, 'name': 'name'}
...
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
---
...
vshard.router.call(100, 'write', 'customer_add', {{customer_id = 2,
bucket_id = 100, name = 'name2', accounts = {}}}, {timeout = 100})
@@ -146,13 +146,13 @@ test_run:switch('router_1')
...
-- Emulate a situation, when a replicaset_2 while is unknown for
-- router, but is already known for storages.
-save_rs2 = vshard.router.internal.replicasets[replicasets[2]]
+save_rs2 = vshard.router.internal.static_router.replicasets[replicasets[2]]
---
...
-vshard.router.internal.replicasets[replicasets[2]] = nil
+vshard.router.internal.static_router.replicasets[replicasets[2]] = nil
---
...
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
---
...
fiber = require('fiber')
@@ -207,7 +207,7 @@ err
require('log').info(string.rep('a', 1000))
---
...
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
---
...
call_retval = nil
@@ -219,7 +219,7 @@ f = fiber.create(do_call, 100)
while not test_run:grep_log('router_1', 'please update configuration',
1000) do fiber.sleep(0.1) end
---
...
-vshard.router.internal.replicasets[replicasets[2]] = save_rs2
+vshard.router.internal.static_router.replicasets[replicasets[2]] = save_rs2
---
...
while not call_retval do fiber.sleep(0.1) end
diff --git a/test/router/reroute_wrong_bucket.test.lua
b/test/router/reroute_wrong_bucket.test.lua
index 03384d1..a00f941 100644
--- a/test/router/reroute_wrong_bucket.test.lua
+++ b/test/router/reroute_wrong_bucket.test.lua
@@ -35,7 +35,7 @@ customer_add({customer_id = 1, bucket_id = 100, name =
'name', accounts = {}})
test_run:switch('router_1')
vshard.router.call(100, 'read', 'customer_lookup', {1}, {timeout = 100})
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
vshard.router.call(100, 'write', 'customer_add', {{customer_id = 2,
bucket_id = 100, name = 'name2', accounts = {}}}, {timeout = 100})
-- Create cycle.
@@ -55,9 +55,9 @@ box.space._bucket:replace({100,
vshard.consts.BUCKET.SENT, replicasets[2]})
test_run:switch('router_1')
-- Emulate a situation, when a replicaset_2 while is unknown for
-- router, but is already known for storages.
-save_rs2 = vshard.router.internal.replicasets[replicasets[2]]
-vshard.router.internal.replicasets[replicasets[2]] = nil
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+save_rs2 = vshard.router.internal.static_router.replicasets[replicasets[2]]
+vshard.router.internal.static_router.replicasets[replicasets[2]] = nil
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
fiber = require('fiber')
call_retval = nil
@@ -84,11 +84,11 @@ err
-- detect it and end with ok.
--
require('log').info(string.rep('a', 1000))
-vshard.router.internal.route_map[100] =
vshard.router.internal.replicasets[replicasets[1]]
+vshard.router.internal.static_router.route_map[100] =
vshard.router.internal.static_router.replicasets[replicasets[1]]
call_retval = nil
f = fiber.create(do_call, 100)
while not test_run:grep_log('router_1', 'please update configuration',
1000) do fiber.sleep(0.1) end
-vshard.router.internal.replicasets[replicasets[2]] = save_rs2
+vshard.router.internal.static_router.replicasets[replicasets[2]] = save_rs2
while not call_retval do fiber.sleep(0.1) end
call_retval
vshard.router.call(100, 'read', 'customer_lookup', {3}, {timeout = 1})
diff --git a/test/router/retry_reads.result b/test/router/retry_reads.result
index 64b0ff3..b803ae3 100644
--- a/test/router/retry_reads.result
+++ b/test/router/retry_reads.result
@@ -37,7 +37,7 @@ test_run:cmd('switch router_1')
util = require('util')
---
...
-rs1 = vshard.router.internal.replicasets[replicasets[1]]
+rs1 = vshard.router.internal.static_router.replicasets[replicasets[1]]
---
...
min_timeout = vshard.consts.CALL_TIMEOUT_MIN
diff --git a/test/router/retry_reads.test.lua
b/test/router/retry_reads.test.lua
index 2fb2fc7..510e961 100644
--- a/test/router/retry_reads.test.lua
+++ b/test/router/retry_reads.test.lua
@@ -13,7 +13,7 @@ test_run:cmd("start server router_1")
test_run:cmd('switch router_1')
util = require('util')
-rs1 = vshard.router.internal.replicasets[replicasets[1]]
+rs1 = vshard.router.internal.static_router.replicasets[replicasets[1]]
min_timeout = vshard.consts.CALL_TIMEOUT_MIN
--
diff --git a/test/router/router.result b/test/router/router.result
More information about the Tarantool-patches
mailing list