From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B15696ECC0; Fri, 17 Dec 2021 03:26:37 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B15696ECC0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1639700797; bh=vUvDHoDYhFY2PoVz/HpIqJMPwwNLW/JMNBEbixTT5oc=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=eb1iE0EwGlwCCWygzPgvrtRJQmWsdyQUxDn8gcf/urG5kNlnt8E94p55NhUvg6NiB GW0N68eD5kCUbYgk+VNwNzNB/Nctda9B2pT4zkhoeVnBwmtp5hOAEZHBKt+fnNJFpW L/yCysS3p5tFbaNAWXyNTR5u0paPlBRYFIKYj2sY= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 2C8796ECC0 for ; Fri, 17 Dec 2021 03:25:35 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2C8796ECC0 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1my142-0007At-CV; Fri, 17 Dec 2021 03:25:34 +0300 To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org Date: Fri, 17 Dec 2021 01:25:28 +0100 Message-Id: <9d767a02cabc032ff4ad478b4a51c0a254276569.1639700518.git.v.shpilevoy@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD9B5397E24C93BDA67B3A68BB557596D46DAA29B34AF660FEF182A05F538085040D2E770D64200D5DDF3F895AAE5713EA98A69C0C3FBD10E0D8FF39B5A568007DA X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7A34C649281B21B01EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637835928C62272F24E8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8930BB8818F4677261E567CDB93E0957B117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BF1175FABE1C0F9B6A471835C12D1D977C4224003CC8364762BB6847A3DEAEFB0F43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7CD1D040B6C1ECEA3F43847C11F186F3C59DAA53EE0834AAEE X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C4C7A0BC55FA0FE5FC3163393FEE54070F2DF6D36AA2B91B3EF859FC7E368876D0B1881A6453793CE9C32612AADDFBE061C61BE10805914D3804EBA3D8E7E5B87ABF8C51168CD8EBDBD215BE4436AF2686DC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D71B56C992F8AF1961BED494CC42B6F305AE87CD2BFB3CC4A5835224C96347FC751A59BB92E1A4F21D7E09C32AA3244CBD59B0169F8B6F61D0A503F61CC5BC0CF2F5F14F68F1805B729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojieEIankJUzpCJWarh4JH/Q== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D6A0957A8465282173E341E6BEE14F47B3841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E25FEEDEB644C299C0ED14614B50AE0675 X-Mras: Ok Subject: [Tarantool-patches] [PATCH vshard 2/5] storage: auto enable/disable X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" While vshard.storage.cfg() is not done, accessing vshard functions is not safe. It will fail with low level errors like 'access denied' or 'no such function'. However there can be even worse cases. The user can have universe access rights. And vshard can be already in global namespace after require(). So vshard.storage functions are already visible. The previous patch fixed only the case when function access was restricted properly. And still fixed it just partially. New problems are: - box.cfg{} is already called, but the instance is still 'loading'. Then data is not fully recovered yet. Accessing is not safe from the data consistency perspective. - vshard.storage.cfg() is not started, or is not finished yet. In the end it might be doing something on what the public functions depend. This patch addresses these issues. Now all non-trivial vshard.storage functions are disabled until vshard.storage.cfg() is finished and the instance is fully recovered. They raise an error with a special code. Returning it via 'nil, err' pair wouldn't work. Because firstly, some functions return a boolean value and are not documented as ever failing. People would miss this new error. Second reason - vshard.storage.call() needs to signal the remote caller that the storage is disabled and it was found before the user's function was called. If it would be done via 'nil, err', then the user's function could emulate the storage being disabled. Or even worse, it could make some changes and then get that error accidentally by going to another storage remotely which would be disabled. Hence it is not allowed. Too easy to break something. It was an option to change vshard.storage.call() signature to return 'true, retvals...' when user's function was called and 'false, err' when it wasn't, but that would break backward compatibility. Supporting it only for new routers does not seem possible. The patch also drops 'memtx_memory' setting from the config because an attempt to apply it after calling box.cfg() (for example, via boot_like_vshard()) raises an error - default memory is bigger than this setting. It messed the new tests. Part of #298 Closes #123 --- example/localcfg.lua | 1 - test/storage/storage.result | 101 +++++++++++++++++++ test/storage/storage.test.lua | 43 ++++++++ test/unit/garbage.result | 17 ++-- test/unit/garbage.test.lua | 15 +-- test/unit/rebalancer.result | 2 +- test/unit/rebalancer.test.lua | 2 +- vshard/error.lua | 5 + vshard/storage/init.lua | 179 ++++++++++++++++++++++++++-------- 9 files changed, 305 insertions(+), 60 deletions(-) diff --git a/example/localcfg.lua b/example/localcfg.lua index 71f9d6f..9235d45 100644 --- a/example/localcfg.lua +++ b/example/localcfg.lua @@ -1,5 +1,4 @@ return { - memtx_memory = 100 * 1024 * 1024, sharding = { ['cbf06940-0790-498b-948d-042b62cf3d29'] = { -- replicaset #1 replicas = { diff --git a/test/storage/storage.result b/test/storage/storage.result index af48a13..e83b34f 100644 --- a/test/storage/storage.result +++ b/test/storage/storage.result @@ -951,6 +951,107 @@ vshard.storage._call('info') --- - is_master: false ... +-- +-- gh-123, gh-298: storage auto-enable/disable depending on instance state. +-- +_ = test_run:cmd('stop server storage_1_a') +--- +... +_ = test_run:cmd('start server storage_1_a with wait=False, '.. \ + 'args="boot_before_cfg"') +--- +... +_ = test_run:switch('storage_1_a') +--- +... +-- Leaving box.cfg() not called won't work because at 1.10 test-run somewhy +-- raises an error when try to start an instance without box.cfg(). It can only +-- be emulated. +old_info = box.info +--- +... +box.info = setmetatable({}, {__index = function() error('not configured') end}) +--- +... +assert(not pcall(function() return box.info.status end)) +--- +- true +... +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +--- +... +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +--- +- true +... +assert(err.message:match('box seem not to be configured') ~= nil) +--- +- true +... +box.info = old_info +--- +... +-- Disabled until box is loaded. +vshard.storage.internal.errinj.ERRINJ_CFG_DELAY = true +--- +... +f = fiber.create(vshard.storage.cfg, cfg, instance_uuid) +--- +... +f:set_joinable(true) +--- +... +old_info = box.info +--- +... +box.info = {status = 'loading'} +--- +... +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +--- +... +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +--- +- true +... +assert(err.message:match('instance status is "loading"') ~= nil) +--- +- true +... +box.info = old_info +--- +... +-- Disabled until storage is configured. +test_run:wait_cond(function() return box.info.status == 'running' end) +--- +- true +... +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +--- +... +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +--- +- true +... +assert(err.message:match('storage is not configured') ~= nil) +--- +- true +... +vshard.storage.internal.errinj.ERRINJ_CFG_DELAY = false +--- +... +f:join() +--- +- true +... +-- Enabled when all criteria are finally satisfied. +ok, res = vshard.storage.call(1, 'read', 'echo', {100}) +--- +... +assert(ok and res == 100) +--- +- true +... _ = test_run:switch("default") --- ... diff --git a/test/storage/storage.test.lua b/test/storage/storage.test.lua index 99ef2c4..ff39f2f 100644 --- a/test/storage/storage.test.lua +++ b/test/storage/storage.test.lua @@ -299,6 +299,49 @@ vshard.storage._call('info') _ = test_run:switch('storage_1_b') vshard.storage._call('info') +-- +-- gh-123, gh-298: storage auto-enable/disable depending on instance state. +-- +_ = test_run:cmd('stop server storage_1_a') +_ = test_run:cmd('start server storage_1_a with wait=False, '.. \ + 'args="boot_before_cfg"') +_ = test_run:switch('storage_1_a') +-- Leaving box.cfg() not called won't work because at 1.10 test-run somewhy +-- raises an error when try to start an instance without box.cfg(). It can only +-- be emulated. +old_info = box.info +box.info = setmetatable({}, {__index = function() error('not configured') end}) +assert(not pcall(function() return box.info.status end)) + +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +assert(err.message:match('box seem not to be configured') ~= nil) +box.info = old_info + +-- Disabled until box is loaded. +vshard.storage.internal.errinj.ERRINJ_CFG_DELAY = true +f = fiber.create(vshard.storage.cfg, cfg, instance_uuid) +f:set_joinable(true) + +old_info = box.info +box.info = {status = 'loading'} +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +assert(err.message:match('instance status is "loading"') ~= nil) +box.info = old_info + +-- Disabled until storage is configured. +test_run:wait_cond(function() return box.info.status == 'running' end) +ok, err = pcall(vshard.storage.call, 1, 'read', 'echo', {100}) +assert(not ok and err.code == vshard.error.code.STORAGE_IS_DISABLED) +assert(err.message:match('storage is not configured') ~= nil) +vshard.storage.internal.errinj.ERRINJ_CFG_DELAY = false +f:join() + +-- Enabled when all criteria are finally satisfied. +ok, res = vshard.storage.call(1, 'read', 'echo', {100}) +assert(ok and res == 100) + _ = test_run:switch("default") test_run:drop_cluster(REPLICASET_2) test_run:drop_cluster(REPLICASET_1) diff --git a/test/unit/garbage.result b/test/unit/garbage.result index a530496..c4b0fc3 100644 --- a/test/unit/garbage.result +++ b/test/unit/garbage.result @@ -16,7 +16,7 @@ test_run:cmd("setopt delimiter ';'") ... function show_sharded_spaces() local result = {} - for k, space in pairs(vshard.storage.sharded_spaces()) do + for k, space in pairs(vshard.storage._sharded_spaces()) do table.insert(result, space.name) end table.sort(result) @@ -489,7 +489,10 @@ f:cancel() util = require('util') --- ... -util.check_error(vshard.storage.bucket_delete_garbage) +delete_garbage = vshard.storage._bucket_delete_garbage +--- +... +util.check_error(delete_garbage) --- - 'Usage: bucket_delete_garbage(bucket_id, opts)' ... @@ -506,7 +509,7 @@ s:replace{6, 4} --- - [6, 4] ... -vshard.storage.bucket_delete_garbage(4) +delete_garbage(4) --- ... s:select{} @@ -526,7 +529,7 @@ s:replace{6, 4} --- - [6, 4] ... -vshard.storage.bucket_delete_garbage(4) +delete_garbage(4) --- ... s:select{} @@ -547,16 +550,16 @@ s:replace{6, 4} --- - [6, 4] ... -util.check_error(vshard.storage.bucket_delete_garbage, 4) +util.check_error(delete_garbage, 4) --- - Can not delete not garbage bucket. Use "{force=true}" to ignore this attention ... -util.check_error(vshard.storage.bucket_delete_garbage, 4, 10000) +util.check_error(delete_garbage, 4, 10000) --- - 'Usage: bucket_delete_garbage(bucket_id, opts)' ... -- 'Force' option ignores this error. -vshard.storage.bucket_delete_garbage(4, {force = true}) +delete_garbage(4, {force = true}) --- ... s:select{} diff --git a/test/unit/garbage.test.lua b/test/unit/garbage.test.lua index 250afb0..1416f58 100644 --- a/test/unit/garbage.test.lua +++ b/test/unit/garbage.test.lua @@ -6,7 +6,7 @@ engine = test_run:get_cfg('engine') test_run:cmd("setopt delimiter ';'") function show_sharded_spaces() local result = {} - for k, space in pairs(vshard.storage.sharded_spaces()) do + for k, space in pairs(vshard.storage._sharded_spaces()) do table.insert(result, space.name) end table.sort(result) @@ -197,30 +197,31 @@ f:cancel() -- util = require('util') -util.check_error(vshard.storage.bucket_delete_garbage) +delete_garbage = vshard.storage._bucket_delete_garbage +util.check_error(delete_garbage) -- Delete an existing garbage bucket. _bucket:replace{4, vshard.consts.BUCKET.SENT} s:replace{5, 4} s:replace{6, 4} -vshard.storage.bucket_delete_garbage(4) +delete_garbage(4) s:select{} -- Delete a not existing garbage bucket. _ = _bucket:delete{4} s:replace{5, 4} s:replace{6, 4} -vshard.storage.bucket_delete_garbage(4) +delete_garbage(4) s:select{} -- Fail to delete a not garbage bucket. _bucket:replace{4, vshard.consts.BUCKET.ACTIVE} s:replace{5, 4} s:replace{6, 4} -util.check_error(vshard.storage.bucket_delete_garbage, 4) -util.check_error(vshard.storage.bucket_delete_garbage, 4, 10000) +util.check_error(delete_garbage, 4) +util.check_error(delete_garbage, 4, 10000) -- 'Force' option ignores this error. -vshard.storage.bucket_delete_garbage(4, {force = true}) +delete_garbage(4, {force = true}) s:select{} -- diff --git a/test/unit/rebalancer.result b/test/unit/rebalancer.result index 19aa480..a931d63 100644 --- a/test/unit/rebalancer.result +++ b/test/unit/rebalancer.result @@ -287,7 +287,7 @@ build_routes(replicasets) -- -- Test rebalancer local state. -- -get_state = vshard.storage.rebalancer_request_state +get_state = vshard.storage._rebalancer_request_state --- ... _bucket = box.schema.create_space('_bucket') diff --git a/test/unit/rebalancer.test.lua b/test/unit/rebalancer.test.lua index 8087d42..1cdbbcb 100644 --- a/test/unit/rebalancer.test.lua +++ b/test/unit/rebalancer.test.lua @@ -75,7 +75,7 @@ build_routes(replicasets) -- -- Test rebalancer local state. -- -get_state = vshard.storage.rebalancer_request_state +get_state = vshard.storage._rebalancer_request_state _bucket = box.schema.create_space('_bucket') pk = _bucket:create_index('pk') status = _bucket:create_index('status', {parts = {{2, 'string'}}, unique = false}) diff --git a/vshard/error.lua b/vshard/error.lua index 0ce208a..2b97eae 100644 --- a/vshard/error.lua +++ b/vshard/error.lua @@ -165,6 +165,11 @@ local error_message_template = { 'Last error was %s', args = {'replicaset_uuid', 'error'} }, + [33] = { + name = 'STORAGE_IS_DISABLED', + msg = 'Storage is disabled: %s', + args = {'reason'} + }, } -- diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index 3158f3c..d3c4e2a 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -134,6 +134,16 @@ if not M then -- help routers find a new location for sent and deleted buckets without -- whole cluster scan. route_map = {}, + -- Flag whether vshard.storage.cfg() is finished. + is_configured = false, + -- Flag whether box.info.status is acceptable. For instance, 'loading' + -- is not. + is_loaded = false, + -- Reference to the function-proxy to most of the public functions. It + -- allows to avoid 'if's in each function by adding expensive + -- conditional checks in one rarely used version of the wrapper and no + -- checks into the other almost always used wrapper. + api_call_cache = nil, ------------------- Garbage collection ------------------- -- Fiber to remove garbage buckets data. @@ -670,19 +680,19 @@ local function this_is_master() M.this_replica == M.this_replicaset.master end -local function on_master_disable(...) - M._on_master_disable(...) +local function on_master_disable(new_func, old_func) + M._on_master_disable(new_func, old_func) -- If a trigger is set after storage.cfg(), then notify an -- user, that the current instance is not master. - if select('#', ...) == 1 and not this_is_master() then + if old_func == nil and not this_is_master() then M._on_master_disable:run() end end -local function on_master_enable(...) - M._on_master_enable(...) +local function on_master_enable(new_func, old_func) + M._on_master_enable(new_func, old_func) -- Same as above, but notify, that the instance is master. - if select('#', ...) == 1 and this_is_master() then + if old_func == nil and this_is_master() then M._on_master_enable:run() end end @@ -1363,6 +1373,13 @@ local function find_sharded_spaces() return spaces end +-- +-- Public wrapper for sharded spaces list getter. +-- +local function storage_sharded_spaces() + return table.deepcopy(find_sharded_spaces()) +end + if M.errinj.ERRINJ_RELOAD then error('Error injection: reload') end @@ -2764,6 +2781,7 @@ local function storage_cfg(cfg, this_replica_uuid, is_reload) M.rebalancer_fiber = nil end lua_gc.set_state(M.collect_lua_garbage, consts.COLLECT_LUA_GARBAGE_INTERVAL) + M.is_configured = true -- Destroy connections, not used in a new configuration. collectgarbage() end @@ -2915,6 +2933,58 @@ local function storage_info() return state end +-------------------------------------------------------------------------------- +-- Public API protection +-------------------------------------------------------------------------------- + +-- +-- Arguments are listed explicitly instead of '...' because the latter does not +-- jit. +-- +local function storage_api_call_safe(func, arg1, arg2, arg3, arg4) + return func(arg1, arg2, arg3, arg4) +end + +-- +-- Unsafe proxy is loaded with protections. But it is used rarely and only in +-- the beginning of instance's lifetime. +-- +local function storage_api_call_unsafe(func, arg1, arg2, arg3, arg4) + -- box.info is quite expensive. Avoid calling it again when the instance + -- is finally loaded. + if not M.is_loaded then + -- box.info raises an error until box.cfg() is started. + local ok, status = pcall(function() + return box.info.status + end) + if not ok then + local msg = 'box seem not to be configured' + return error(lerror.vshard(lerror.code.STORAGE_IS_DISABLED, msg)) + end + -- 'Orphan' is allowed because even if a replica is an orphan, it still + -- could be up to date. Just not all other replicas are connected. + if status ~= 'running' and status ~= 'orphan' then + local msg = ('instance status is "%s"'):format(status) + return error(lerror.vshard(lerror.code.STORAGE_IS_DISABLED, msg)) + end + M.is_loaded = true + end + if not M.is_configured then + local msg = 'storage is not configured' + return error(lerror.vshard(lerror.code.STORAGE_IS_DISABLED, msg)) + end + M.api_call_cache = storage_api_call_safe + return func(arg1, arg2, arg3, arg4) +end + +M.api_call_cache = storage_api_call_unsafe + +local function storage_make_api(func) + return function(arg1, arg2, arg3, arg4) + return M.api_call_cache(func, arg1, arg2, arg3, arg4) + end +end + -------------------------------------------------------------------------------- -- Module definition -------------------------------------------------------------------------------- @@ -3021,44 +3091,67 @@ M.bucket_are_all_rw = bucket_are_all_rw_public M.bucket_generation_wait = bucket_generation_wait lregistry.storage = M +-- +-- Not all methods are public here. Private methods should not be exposed if +-- possible. At least not without notable difference in naming. +-- return { - sync = sync, - bucket_force_create = bucket_force_create, - bucket_force_drop = bucket_force_drop, - bucket_collect = bucket_collect, - bucket_recv = bucket_recv, - bucket_send = bucket_send, - bucket_stat = bucket_stat, - bucket_pin = bucket_pin, - bucket_unpin = bucket_unpin, - bucket_ref = bucket_ref, - bucket_unref = bucket_unref, - bucket_refro = bucket_refro, - bucket_refrw = bucket_refrw, - bucket_unrefro = bucket_unrefro, - bucket_unrefrw = bucket_unrefrw, - bucket_delete_garbage = bucket_delete_garbage, - garbage_collector_wakeup = garbage_collector_wakeup, - rebalancer_wakeup = rebalancer_wakeup, - rebalancer_apply_routes = rebalancer_apply_routes, - rebalancer_disable = rebalancer_disable, - rebalancer_enable = rebalancer_enable, - is_locked = is_this_replicaset_locked, - rebalancing_is_in_progress = rebalancing_is_in_progress, - recovery_wakeup = recovery_wakeup, - call = storage_call, - _call = service_call, + -- + -- Bucket methods. + -- + bucket_force_create = storage_make_api(bucket_force_create), + bucket_force_drop = storage_make_api(bucket_force_drop), + bucket_collect = storage_make_api(bucket_collect), + bucket_recv = storage_make_api(bucket_recv), + bucket_send = storage_make_api(bucket_send), + bucket_stat = storage_make_api(bucket_stat), + bucket_pin = storage_make_api(bucket_pin), + bucket_unpin = storage_make_api(bucket_unpin), + bucket_ref = storage_make_api(bucket_ref), + bucket_unref = storage_make_api(bucket_unref), + bucket_refro = storage_make_api(bucket_refro), + bucket_refrw = storage_make_api(bucket_refrw), + bucket_unrefro = storage_make_api(bucket_unrefro), + bucket_unrefrw = storage_make_api(bucket_unrefrw), + bucket_delete_garbage = storage_make_api(bucket_delete_garbage), + _bucket_delete_garbage = bucket_delete_garbage, + buckets_info = storage_make_api(storage_buckets_info), + buckets_count = storage_make_api(bucket_count_public), + buckets_discovery = storage_make_api(buckets_discovery), + -- + -- Garbage collector. + -- + garbage_collector_wakeup = storage_make_api(garbage_collector_wakeup), + -- + -- Rebalancer. + -- + rebalancer_wakeup = storage_make_api(rebalancer_wakeup), + rebalancer_apply_routes = storage_make_api(rebalancer_apply_routes), + rebalancer_disable = storage_make_api(rebalancer_disable), + rebalancer_enable = storage_make_api(rebalancer_enable), + rebalancing_is_in_progress = storage_make_api(rebalancing_is_in_progress), + rebalancer_request_state = storage_make_api(rebalancer_request_state), + _rebalancer_request_state = rebalancer_request_state, + -- + -- Recovery. + -- + recovery_wakeup = storage_make_api(recovery_wakeup), + -- + -- Instance info. + -- + is_locked = storage_make_api(is_this_replicaset_locked), + info = storage_make_api(storage_info), + sharded_spaces = storage_make_api(storage_sharded_spaces), + _sharded_spaces = storage_sharded_spaces, + module_version = function() return M.module_version end, + -- + -- Miscellaneous. + -- + call = storage_make_api(storage_call), + _call = storage_make_api(service_call), + sync = storage_make_api(sync), cfg = function(cfg, uuid) return storage_cfg(cfg, uuid, false) end, - info = storage_info, - buckets_info = storage_buckets_info, - buckets_count = bucket_count_public, - buckets_discovery = buckets_discovery, - rebalancer_request_state = rebalancer_request_state, + on_master_enable = storage_make_api(on_master_enable), + on_master_disable = storage_make_api(on_master_disable), internal = M, - on_master_enable = on_master_enable, - on_master_disable = on_master_disable, - sharded_spaces = function() - return table.deepcopy(find_sharded_spaces()) - end, - module_version = function() return M.module_version end, } -- 2.24.3 (Apple Git-128)