From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 548F771814; Tue, 23 Feb 2021 03:18:25 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 548F771814 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614039505; bh=e0hqlyaQCNun6C1I7163L4VAa03Eo/shsADuQfPMz5Y=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=UuTT+qTJtbhGMMv+Q340KNuSVwG/nylTpUoihdRwPzNH4tO58z0SVFvVgU/Oc9kQD 1U9jddN1oSGCuFC423TnBfrD56aZdOZbf5XM/aiibQO3S7ISzc1h4Bp3X8JJ3yj7B2 YkjTu9lK15EoCRwuoKGrNTLIScUtAd5kHquKpCPk= Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id EDBEF71828 for ; Tue, 23 Feb 2021 03:15:53 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org EDBEF71828 Received: by smtpng2.m.smailru.net with esmtpa (envelope-from ) id 1lELMn-0003CR-91; Tue, 23 Feb 2021 03:15:53 +0300 To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org, yaroslav.dynnikov@tarantool.org Date: Tue, 23 Feb 2021 01:15:41 +0100 Message-Id: <7a52b9f05ce9c6f2b44164668f09706c7723b104.1614039039.git.v.shpilevoy@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD975C3EC174F5669221340953D8FEAAFCA26BB79E5C093AA91182A05F538085040287C623B553AAA9B8C5874CB7E754DC8C1F6BEB0D960E2BD4748B41F9C71AC65 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78EA80DE462DCD770EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637FAFFEDEAEB71C4328638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC425FEB8730D3840336C410242210D40E6E211A32D742036D389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C06030C3405640F6718941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6F532FBFD8162E58CCC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC08F9A42B2210255C75ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4930A3850AC1BE2E73579C543ECCDAE434EC4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0F347543BADC64E7283B503F486389A921A5CC5B56E945C8DA X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C2115B036FF9F6D2506CCC190338645B86BA437927F24D7E09C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF1638054B7D09EC08699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D347D10A9FCB2A62DFE9111AD307CF5BF316748B11201FA32A7F6D82AC78E2B6AC2DD1CD673E81F949C1D7E09C32AA3244CE2778DF3DD3781BCF1677D4053EFAA2233C9DC155518937FFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj2drqE2xHc+HIYuQfg8biew== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822CAC8220C3AAAC3ABFE409D6368D947763841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Bucket count calculation costs 1 FFI call in Lua, and makes a few actions and virtual calls in C. So it is not free even for memtx spaces. But it changes extremely rare, which makes reasonable to cache the value. Bucket count is not used much now, but will be used a lot in the future storage_ref() function, which is a part of map-reduce API. The idea is that a router will need to reference all the storages and ensure that all the buckets in the cluster are pinned to their storages. To check this, storage_ref() will return number of buckets successfully pinned on the storage. The router will sum counts from all storage_ref() calls and ensure it equals to total configured bucket count. This means bucket count is needed for each storage_ref() call, whose count per second can be thousands and more. The patch makes count calculation cost as much as one Lua function call and a Lua table index operation (almost always). Needed for #147 --- test/storage/storage.result | 45 +++++++++++++++++++++++++++++++++++ test/storage/storage.test.lua | 18 ++++++++++++++ vshard/storage/init.lua | 44 +++++++++++++++++++++++++++++----- 3 files changed, 101 insertions(+), 6 deletions(-) diff --git a/test/storage/storage.result b/test/storage/storage.result index 0550ad1..edb45be 100644 --- a/test/storage/storage.result +++ b/test/storage/storage.result @@ -677,6 +677,51 @@ rs:callro('echo', {'some_data'}) - null - null ... +-- +-- Bucket count is calculated properly. +-- +-- Cleanup after the previous tests. +_ = test_run:switch('storage_1_a') +--- +... +buckets = vshard.storage.buckets_info() +--- +... +for bid, _ in pairs(buckets) do vshard.storage.bucket_force_drop(bid) end +--- +... +_ = test_run:switch('storage_2_a') +--- +... +buckets = vshard.storage.buckets_info() +--- +... +for bid, _ in pairs(buckets) do vshard.storage.bucket_force_drop(bid) end +--- +... +_ = test_run:switch('storage_1_a') +--- +... +assert(vshard.storage.buckets_count() == 0) +--- +- true +... +vshard.storage.bucket_force_create(1, 5) +--- +- true +... +assert(vshard.storage.buckets_count() == 5) +--- +- true +... +vshard.storage.bucket_force_create(6, 5) +--- +- true +... +assert(vshard.storage.buckets_count() == 10) +--- +- true +... _ = test_run:switch("default") --- ... diff --git a/test/storage/storage.test.lua b/test/storage/storage.test.lua index d8fbd94..db014ef 100644 --- a/test/storage/storage.test.lua +++ b/test/storage/storage.test.lua @@ -187,6 +187,24 @@ util.has_same_fields(old_internal, vshard.storage.internal) _, rs = next(vshard.storage.internal.replicasets) rs:callro('echo', {'some_data'}) +-- +-- Bucket count is calculated properly. +-- +-- Cleanup after the previous tests. +_ = test_run:switch('storage_1_a') +buckets = vshard.storage.buckets_info() +for bid, _ in pairs(buckets) do vshard.storage.bucket_force_drop(bid) end +_ = test_run:switch('storage_2_a') +buckets = vshard.storage.buckets_info() +for bid, _ in pairs(buckets) do vshard.storage.bucket_force_drop(bid) end + +_ = test_run:switch('storage_1_a') +assert(vshard.storage.buckets_count() == 0) +vshard.storage.bucket_force_create(1, 5) +assert(vshard.storage.buckets_count() == 5) +vshard.storage.bucket_force_create(6, 5) +assert(vshard.storage.buckets_count() == 10) + _ = test_run:switch("default") test_run:drop_cluster(REPLICASET_2) test_run:drop_cluster(REPLICASET_1) diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index a3d383d..9b74bcb 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -110,6 +110,9 @@ if not M then -- replace the old function is to keep its reference. -- bucket_on_replace = nil, + -- Fast alternative to box.space._bucket:count(). But may be nil. Reset + -- on each generation change. + bucket_count_cache = nil, -- Redirects for recently sent buckets. They are kept for a while to -- help routers to find a new location for sent and deleted buckets -- without whole cluster scan. @@ -183,10 +186,44 @@ local function local_call(func_name, args) return pcall(netbox_self_call, netbox_self, func_name, args) end +-- +-- Get number of buckets stored on this storage. Regardless of their state. +-- +-- The idea is that all the code should use one function ref to get the bucket +-- count. But inside the function never branches. Instead, it points at one of 2 +-- branch-less functions. Cached one simply returns a number which is supposed +-- to be super fast. Non-cached remembers the count and changes the global +-- function to the cached one. So on the next call it is cheap. No 'if's at all. +-- +local bucket_count + +local function bucket_count_cache() + return M.bucket_count_cache +end + +local function bucket_count_not_cache() + local count = box.space._bucket:count() + M.bucket_count_cache = count + bucket_count = bucket_count_cache + return count +end + +bucket_count = bucket_count_not_cache + +-- +-- Can't expose bucket_count to the public API as is. Need this proxy-call. +-- Because the original function changes at runtime. +-- +local function bucket_count_public() + return bucket_count() +end + -- -- Trigger for on replace into _bucket to update its generation. -- local function bucket_generation_increment() + bucket_count = bucket_count_not_cache + M.bucket_count_cache = nil M.bucket_generation = M.bucket_generation + 1 M.bucket_generation_cond:broadcast() end @@ -2240,7 +2277,6 @@ local function rebalancer_request_state() if #status_index:select({consts.BUCKET.GARBAGE}, {limit = 1}) > 0 then return end - local bucket_count = _bucket:count() return { bucket_active_count = status_index:count({consts.BUCKET.ACTIVE}), bucket_pinned_count = status_index:count({consts.BUCKET.PINNED}), @@ -2501,10 +2537,6 @@ end -- Monitoring -------------------------------------------------------------------------------- -local function storage_buckets_count() - return box.space._bucket.index.pk:count() -end - local function storage_buckets_info(bucket_id) local ibuckets = setmetatable({}, { __serialize = 'mapping' }) @@ -2780,7 +2812,7 @@ return { cfg = function(cfg, uuid) return storage_cfg(cfg, uuid, false) end, info = storage_info, buckets_info = storage_buckets_info, - buckets_count = storage_buckets_count, + buckets_count = bucket_count_public, buckets_discovery = buckets_discovery, rebalancer_request_state = rebalancer_request_state, internal = M, -- 2.24.3 (Apple Git-128)