From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 730676ECC0; Fri, 17 Dec 2021 14:09:30 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 730676ECC0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1639739370; bh=7BEhkAOPc0AAYobxnAuEyjlzVgFFb8CNQ5w/x56GXdU=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=AsRt1avFVKDvFExpx8sLslkz8RFeWYWzCiyxvlHvEoJ4lcr/2uubNEu86iY+jlgmP uxzNf/G8ixbDcIrmMv3Q/A3IqVCVFe5iJjbvKLykH7KV43+73BPnLb3/epx+4zRoHN qA7ss+hFSJZdT98IJYWKKfIP96PGnEoyqcrmCPHA= Received: from smtp45.i.mail.ru (smtp45.i.mail.ru [94.100.177.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 4835A6ECC0 for ; Fri, 17 Dec 2021 14:09:28 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 4835A6ECC0 Received: by smtp45.i.mail.ru with esmtpa (envelope-from ) id 1myB79-000778-JG; Fri, 17 Dec 2021 14:09:27 +0300 Message-ID: Date: Fri, 17 Dec 2021 14:09:27 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.3.2 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org References: <9e5ff9cae03a55bf3cacc482449f19895360c80d.1639700518.git.v.shpilevoy@tarantool.org> In-Reply-To: <9e5ff9cae03a55bf3cacc482449f19895360c80d.1639700518.git.v.shpilevoy@tarantool.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-4EC0790: 10 X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD9B5397E24C93BDA6712610AD69032D75DE6A3E17E48EF72E3182A05F53808504028E0B0D7B1C00DD613F985BAD5F92883DE8638EF25A93F348077335FDAE59415 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7370F4F695FFFC24BEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637645995A778B1BFFAEA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F28CFE16B14521F9C26777EC66005B4BEACC7F00164DA146DAFE8445B8C89999728AA50765F79006372A3B24BF85B2E607389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC8FA486DC37A503D0BF6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA73AA81AA40904B5D9A18204E546F3947CA5ED62E35AC703CC040F9FF01DFDA4A84AD6D5ED66289B52698AB9A7B718F8C46E0066C2D8992A16725E5C173C3A84C357F6EE8AF6153984BA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE72F88032165008D51731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A522626027BF89E42F162DF847979621F8EB65C67579CED0D5D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA759D2A03B9C34326B3410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34AC223B6CC5B2DFAC23DD835692DD7782FABCDEBCBFC9CBC08D892A5BCA285B9BFED53BEC7814764F1D7E09C32AA3244C834EF0BC5F8F3A99F76A8906B9E8AC9530363D8B7DA7DD44729B2BEF169E0186 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojieEIankJUzqAYoKUl1p1aQ== X-Mailru-Sender: 583F1D7ACE8F49BD1042885CEC987B6BB22EF7815600E7B613F985BAD5F928835AB84C59671071107019711D9D5B048E1458020726E2BC9FD5ECBA0B92C0A936CDC7563AA7CEBD2872D6B4FCE48DF648AE208404248635DF X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 1/5] router: backoff on some box errors X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for your patch! LGTM. But I think tests should be extended a bit. I see that tests cover AccessDenied error but I don't see that there is simple error()/assert() error check. On 17.12.2021 03:25, Vladislav Shpilevoy wrote: > Storage configuration takes time. Firstly, box.cfg{} which can be > called before vshard.storage.cfg(). Secondly, vshard.storage.cfg() > is not immediate as well. > > During that time accessing the storage is not safe. Attempts to > call vshard.storage functions can return weird errors, or the > functions can even be not available yet. They need to be created > in _func and get access rights in _priv before becoming public. > > Routers used to forward errors like 'access denied error' and > 'no such function' to users as is, treating them as critical. > > Not only it was confusing for users, but also could make an entire > replicaset not available for requests - the connection to it is > alive, so router would send all requests into it and they all > would fail. Even if the replicaset has another instance which is > perfectly functional. > > This patch handles such specific errors inside of the router. The > faulty replicas are put into a 'backoff' state. They remain in it > for some fixed time (5 seconds for now), new requests won't be > sent to them until the time passes. Router will use other > instances. > > Backoff is activated only for vshard.* functions. If the errors > are about some user's function, it is considered a regular error. > Because the router can't tell whether any side effects were done > on the remote instance before the error happened. Hence can't > retry to another node. > > For example, if access was denied to 'vshard.storage.call', then > it is backoff. If inside of vshard.storage.call the access was > denied to 'user_test_func', then it is not backoff. > > It all works for read-only requests exclusively of course. Because > for read-write requests the instance is just one - master. Router > does not have other options so backoff here wouldn't help. > > Part of #298 > --- > test/router/router2.result | 237 +++++++++++++++++++++++++++++++++++ > test/router/router2.test.lua | 97 ++++++++++++++ > vshard/consts.lua | 1 + > vshard/error.lua | 8 +- > vshard/replicaset.lua | 100 +++++++++++++-- > vshard/router/init.lua | 3 +- > 6 files changed, 432 insertions(+), 14 deletions(-)