From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 44EC47030F; Thu, 25 Feb 2021 15:42:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 44EC47030F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614256967; bh=3+36cV8Ok2/W+zTf5d7U0FOkcA/orXzJjFG3+gmpNjQ=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=Qo8WQkvxNEQQx2ZSdZOw/ZyeMnlFfAivHj11ZAeVkqAgmN99lD0Pd019r8TVTrzsC YzQcTfBXvRwdSQlwytWamHs6iqVvAjZ6WMM8K9t2inILL9dnSA6my83sYoCiMpNUv2 DClPEO+u4Hd8nJrKEqNSlODk0U/7ZJTje7wRstvs= Received: from smtp38.i.mail.ru (smtp38.i.mail.ru [94.100.177.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 581F47030F for ; Thu, 25 Feb 2021 15:42:26 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 581F47030F Received: by smtp38.i.mail.ru with esmtpa (envelope-from ) id 1lFFyL-0005WT-BU; Thu, 25 Feb 2021 15:42:26 +0300 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org References: <7a52b9f05ce9c6f2b44164668f09706c7723b104.1614039039.git.v.shpilevoy@tarantool.org> <542ed6d8-9b68-7455-5a79-b192078c139e@tarantool.org> <86016630-27a0-da45-e71b-ebd573dada3b@tarantool.org> Message-ID: Date: Thu, 25 Feb 2021 15:42:25 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <86016630-27a0-da45-e71b-ebd573dada3b@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD975C3EC174F56692243410BA6471F01668A37AB103014B813182A05F538085040434A8822F712C319B60F69F3C215D88E3A365CD1D7C5ABA12935A8F900EA7568 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE751DD1FEBB966604DEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637BB46061768088B55EA1F7E6F0F101C674E70A05D1297E1BBC6CDE5D1141D2B1CF0B8B4D7FC05BBB24C0FEB1E26EA06CF7A070F57F6FC3ECB9FA2833FD35BB23D9E625A9149C048EE33AC447995A7AD182CC0D3CB04F14752D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8B3A703B70628EAD7BA471835C12D1D977C4224003CC836476EB9C4185024447017B076A6E789B0E975F5C1EE8F4F765FCA030F9CDD733202A3AA81AA40904B5D9CF19DD082D7633A078D18283394535A93AA81AA40904B5D98AA50765F7900637DDFCBB7B63D08578D81D268191BDAD3D698AB9A7B718F8C442539A7722CA490C13377AFFFEAFD26923F8577A6DFFEA7CA1D607EB49F9BFEF93EC92FD9297F6715571747095F342E857739F23D657EF2BD5E8D9A59859A8B6221D5A8161B58CB575ECD9A6C639B01B4E70A05D1297E1BBC6867C52282FAC85D9B7C4F32B44FF57285124B2A10EEC6C00306258E7E6ABB4E4A6367B16DE6309 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46DB70A6FEB20E1D98D355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A5B06074208A6050EA117CE5E383FEC978B834CCBC4D55A1D8D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7557E988E9157162368E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D345CB6DE26F165466719D867195BC1ED7D3B2257AA68D6EF3BBDB3F14D06677125EE691FB1C1031A3B1D7E09C32AA3244CAFDD9A7C8AE44B2C1CCF7C437700EACC408A6A02710B7304FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojvz1c9SWJtj/rvPsa1GcgmQ== X-Mailru-Sender: 583F1D7ACE8F49BD9317CE1922F30C7EC65C939192B5519869DD0BA39E0A16723DCE99E73093D3FC23E75C7104EB1B885DEE61814008E47C7013064206BFB89F93956FB04BA385BE9437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for your answer. You are right, let's won't overcomplicate this task. On 25.02.2021 00:47, Vladislav Shpilevoy wrote: > Thanks for the review! > > On 24.02.2021 11:27, Oleg Babin wrote: >> Thanks for your patch! LGTM. >> >> I see calls like "status_index:count({consts.BUCKET.ACTIVE})". Maybe it worth >> >> to cache whole buckets stats as well? > I thought about it a lot. But realized that I need only a few cached > metrics used for most of the requests. Count of active buckets is not > one of them, but would waste time on invaliding the cache on each > generation update. > > Talking specifically, count({consts.BUCKET.ACTIVE}) is used by > rebalancer only which happens extremely rare. So there is no win in > optimizing it for normal cluster operation. > > Even now I worry about doing too much in the generation increment > trigger. To calculate and keep the stat up to date I would need to > make it more universal. So for example store number of buckets of > each type. Then I face the issues: > > - In on_replace trigger I need to extract bucket status from the old > and new tuple, update the relevant counters. I mostly worry about > extracting the statuses (too long). > > - I need to handle the rollback to somehow revert the counters back. > > I could do something similar to the cache in this patch (I simply > calculate the counts on demand and invalidate them all on each generation > update), but it does not fix the real issue with the counts - they > can be long if bucket count is millions, and the cache will be invalidated > a lot during rebalancing. Exactly when a cache could help most. > > In the end I decided not to bother with this now in scope of map-reduce.