From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 86F5F7030F; Thu, 25 Feb 2021 00:47:31 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 86F5F7030F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614203251; bh=i/e1DL/s78zGAuk8P2S7pvvk7TiqxRsUppEb+9bIeHs=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=w6MvJbwb2/IjX5AIjwoVrWHLhcuYEstnlDOl3JWzW0Gxv/mp1dR/SdOLwOqOIn39o HZTzwPnOMEAxhIFNO7BOeTvBG619VZuejU9l4I7LWVLm2U73PRFgGophXOnNXPRdPc Hl8Nu+KCPJbnQhmE5yJFGhA1uCx0E158Qi6+61EY= Received: from smtp40.i.mail.ru (smtp40.i.mail.ru [94.100.177.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 443787030F for ; Thu, 25 Feb 2021 00:47:30 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 443787030F Received: by smtp40.i.mail.ru with esmtpa (envelope-from ) id 1lF20H-0000Cn-Hu; Thu, 25 Feb 2021 00:47:29 +0300 To: Oleg Babin , tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org References: <7a52b9f05ce9c6f2b44164668f09706c7723b104.1614039039.git.v.shpilevoy@tarantool.org> <542ed6d8-9b68-7455-5a79-b192078c139e@tarantool.org> Message-ID: <86016630-27a0-da45-e71b-ebd573dada3b@tarantool.org> Date: Wed, 24 Feb 2021 22:47:28 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <542ed6d8-9b68-7455-5a79-b192078c139e@tarantool.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD975C3EC174F56692243410BA6471F0166336C1783AA96243D182A05F5380850406F7C46C3CEA154A0A5B699B3934F1DEC26441A786DE64CA3AAD72988E1F48874 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7F8E53417176C7207EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006375121C8F3070B83748638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FCEF6CC647C0669FAF93C5944E86433E63928C24C17CA7A7B5389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0A3E989B1926288338941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B652D31B9D28593E51CC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB8D32BA5DBAC0009BE9E8FC8737B5C224966D89E201A891B4676E601842F6C81A12EF20D2F80756B5F7E9C4E3C761E06A776E601842F6C81A127C277FBC8AE2E8BB403722D27F53E293AA81AA40904B5D9DBF02ECDB25306B2B25CBF701D1BE8734AD6D5ED66289B5278DA827A17800CE7B2B7C64F398C741067F23339F89546C5A8DF7F3B2552694A6FED454B719173D6725E5C173C3A84C31C94431981024FA535872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9AE7E30DF62CE24E4E5BFE6E7EFDEDCD789D4C264860C145E X-B7AD71C0: 14C14B24D00AF5AC321EF223B8115265C69B993890792DF82CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D00A3408CC7563407355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A563A45735A3C3C96CA10494042010766AD40075DF3058DFE9D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75968C9853642EB7C3410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D343DCFC3BEDFB0242DDD21FC07B5E14D92EE417643D9AB7C9FAE0BA0B19B19C388D87CFE6BEACB4DE31D7E09C32AA3244C3AC644868762340BDBAA3D42D5C9E8DB8A6D4CC6FBFAC251FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojvz1c9SWJtj94StOr5cLdvA== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA3110699A8064728F861FCF6ABBAFDB6B01AF28F5B678EDADCBEA07784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for the review! On 24.02.2021 11:27, Oleg Babin wrote: > Thanks for your patch! LGTM. > > I see calls like "status_index:count({consts.BUCKET.ACTIVE})". Maybe it worth > > to cache whole buckets stats as well? I thought about it a lot. But realized that I need only a few cached metrics used for most of the requests. Count of active buckets is not one of them, but would waste time on invaliding the cache on each generation update. Talking specifically, count({consts.BUCKET.ACTIVE}) is used by rebalancer only which happens extremely rare. So there is no win in optimizing it for normal cluster operation. Even now I worry about doing too much in the generation increment trigger. To calculate and keep the stat up to date I would need to make it more universal. So for example store number of buckets of each type. Then I face the issues: - In on_replace trigger I need to extract bucket status from the old and new tuple, update the relevant counters. I mostly worry about extracting the statuses (too long). - I need to handle the rollback to somehow revert the counters back. I could do something similar to the cache in this patch (I simply calculate the counts on demand and invalidate them all on each generation update), but it does not fix the real issue with the counts - they can be long if bucket count is millions, and the cache will be invalidated a lot during rebalancing. Exactly when a cache could help most. In the end I decided not to bother with this now in scope of map-reduce.