From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 3528470202; Wed, 24 Feb 2021 13:30:20 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3528470202 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614162620; bh=hXhU/IcefvWy/fjH2qjOnAQD/LBfIZrpsyZ2dRer+EA=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ggLGWQLJI0IoCtNthqeNcQ5GVJud9kt7kk3ntj94aIkaHKaqyOEgY9ZvEnUmGDWHd n1K+DRetliSEVkdU4Unh7BrYvIBY0VoqryyKDInBQC628rmOn0xN7VAY/4IkSjUzms S2dASCYsuMST9+b9PiIBql0LIHv0Kv1q8hSRgt6c= Received: from smtp47.i.mail.ru (smtp47.i.mail.ru [94.100.177.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A5BCA71865 for ; Wed, 24 Feb 2021 13:27:52 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A5BCA71865 Received: by smtp47.i.mail.ru with esmtpa (envelope-from ) id 1lErOZ-00075S-G9; Wed, 24 Feb 2021 13:27:52 +0300 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org References: Message-ID: Date: Wed, 24 Feb 2021 13:27:51 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD975C3EC174F56692243410BA6471F01668A37AB103014B813182A05F538085040DF56A753AC7456C1ED4113DD526E0D5352BE1A162901947D9418CC399E841C51 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7956F10FFCC7409BAEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006371E8C7BD479B346ED8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC37516A87862161412851939DA1A836BBA9E01BDD31A396C8389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C07E7E81EEA8A9722B8941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B64854413538E1713FCC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB8D32BA5DBAC0009BE9E8FC8737B5C22497ABDCDB10AB97B1D76E601842F6C81A12EF20D2F80756B5F7E9C4E3C761E06A776E601842F6C81A127C277FBC8AE2E8B8C0002352E755DDB3AA81AA40904B5D9DBF02ECDB25306B2B25CBF701D1BE8734AD6D5ED66289B5278DA827A17800CE73349F3DFB73D9B1167F23339F89546C5A8DF7F3B2552694A6FED454B719173D6725E5C173C3A84C391A60F04C81D059235872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-C1DE0DAB: 0D63561A33F958A57AD5364069C5039217F889A48B1417694FE21741DE8603FDD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75B7BFB303F1C7DB4D8E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D344E332383F80D58BB1F3BD8AD316E66EFAF4E79B6BF355FEE304EFB529E127AD23EFB47117C2F6D2B1D7E09C32AA3244CFC5CC9DD6E2E4CAB36A9DE9E1073D6015595C85A795C7BAEFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojyK6JYJ15DtK0zTLFs8wtmg== X-Mailru-Sender: 583F1D7ACE8F49BD9317CE1922F30C7EC1080B63466CCDDB1A2DBD79099C29B25E469574750E037623E75C7104EB1B885DEE61814008E47C7013064206BFB89F93956FB04BA385BE9437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 07/11] storage: introduce bucket_generation_wait() X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for your patch. LGTM. On 23.02.2021 03:15, Vladislav Shpilevoy wrote: > In the future map-reduce code it will be needed to be able to wait > until all buckets on the storage enter writable state. If they are > not writable, the code should wait efficiently, without polling. > > The patch adds a function bucket_generation_wait() which is > registered in registry.storage. > > It helps to wait until state of any bucket is changed. The caller > code, if wants to wait for all buckets to enter writable state, > should wait on the generation and re-check the requested condition > until it matches or timeout happens. > > Part of #147 > --- > test/storage/storage.result | 86 +++++++++++++++++++++++++++++++++++ > test/storage/storage.test.lua | 36 +++++++++++++++ > vshard/storage/init.lua | 6 +++ > 3 files changed, 128 insertions(+) > > diff --git a/test/storage/storage.result b/test/storage/storage.result > index edb45be..4730e20 100644 > --- a/test/storage/storage.result > +++ b/test/storage/storage.result > @@ -722,6 +722,92 @@ assert(vshard.storage.buckets_count() == 10) > --- > - true > ... > +-- > +-- Bucket_generation_wait() registry function. > +-- > +lstorage = require('vshard.registry').storage > +--- > +... > +ok, err = lstorage.bucket_generation_wait(-1) > +--- > +... > +assert(not ok and err.message) > +--- > +- Timeout exceeded > +... > +ok, err = lstorage.bucket_generation_wait(0) > +--- > +... > +assert(not ok and err.message) > +--- > +- Timeout exceeded > +... > +small_timeout = 0.000001 > +--- > +... > +ok, err = lstorage.bucket_generation_wait(small_timeout) > +--- > +... > +assert(not ok and err.message) > +--- > +- Timeout exceeded > +... > +ok, err = nil > +--- > +... > +big_timeout = 1000000 > +--- > +... > +_ = fiber.create(function() \ > + ok, err = lstorage.bucket_generation_wait(big_timeout) \ > +end) > +--- > +... > +fiber.sleep(small_timeout) > +--- > +... > +assert(not ok and not err) > +--- > +- true > +... > +vshard.storage.bucket_force_drop(10) > +--- > +- true > +... > +test_run:wait_cond(function() return ok or err end) > +--- > +- true > +... > +assert(ok) > +--- > +- true > +... > +-- Cancel should interrupt the waiting. > +ok, err = nil > +--- > +... > +f = fiber.create(function() \ > + ok, err = lstorage.bucket_generation_wait(big_timeout) \ > +end) > +--- > +... > +fiber.sleep(small_timeout) > +--- > +... > +assert(not ok and not err) > +--- > +- true > +... > +f:cancel() > +--- > +... > +_ = test_run:wait_cond(function() return ok or err end) > +--- > +... > +assert(not ok and err.message) > +--- > +- fiber is cancelled > +... > _ = test_run:switch("default") --- ... diff --git a/test/storage/storage.test.lua > b/test/storage/storage.test.lua index db014ef..86c5e33 100644 --- > a/test/storage/storage.test.lua +++ b/test/storage/storage.test.lua @@ > -205,6 +205,42 @@ assert(vshard.storage.buckets_count() == 5) > vshard.storage.bucket_force_create(6, 5) > assert(vshard.storage.buckets_count() == 10) +-- +-- > Bucket_generation_wait() registry function. +-- +lstorage = > require('vshard.registry').storage +ok, err = > lstorage.bucket_generation_wait(-1) +assert(not ok and err.message) + > +ok, err = lstorage.bucket_generation_wait(0) +assert(not ok and > err.message) + +small_timeout = 0.000001 +ok, err = > lstorage.bucket_generation_wait(small_timeout) +assert(not ok and > err.message) + +ok, err = nil +big_timeout = 1000000 +_ = > fiber.create(function() \ + ok, err = > lstorage.bucket_generation_wait(big_timeout) \ +end) > +fiber.sleep(small_timeout) +assert(not ok and not err) > +vshard.storage.bucket_force_drop(10) +test_run:wait_cond(function() > return ok or err end) +assert(ok) + +-- Cancel should interrupt the > waiting. +ok, err = nil +f = fiber.create(function() \ + ok, err = > lstorage.bucket_generation_wait(big_timeout) \ +end) > +fiber.sleep(small_timeout) +assert(not ok and not err) +f:cancel() +_ > = test_run:wait_cond(function() return ok or err end) +assert(not ok > and err.message) + _ = test_run:switch("default") > test_run:drop_cluster(REPLICASET_2) > test_run:drop_cluster(REPLICASET_1) > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index b47665b..ffa48b6 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -31,6 +31,7 @@ local util = require('vshard.util') > local lua_gc = require('vshard.lua_gc') > local lregistry = require('vshard.registry') > local reload_evolution = require('vshard.storage.reload_evolution') > +local fiber_cond_wait = util.fiber_cond_wait > local bucket_ref_new > > local M = rawget(_G, MODULE_INTERNALS) > @@ -229,6 +230,10 @@ local function bucket_generation_increment() > M.bucket_generation_cond:broadcast() > end > > +local function bucket_generation_wait(timeout) > + return fiber_cond_wait(M.bucket_generation_cond, timeout) > +end > + > -- > -- Check if this replicaset is locked. It means be invisible for > -- the rebalancer. > @@ -2783,6 +2788,7 @@ M.schema_upgrade_handlers = schema_upgrade_handlers > M.schema_version_make = schema_version_make > M.schema_bootstrap = schema_init_0_1_15_0 > > +M.bucket_generation_wait = bucket_generation_wait > lregistry.storage = M > > return {