From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id F09C56EC6F; Fri, 26 Feb 2021 23:23:11 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org F09C56EC6F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614370992; bh=y1AXjdVNFJIkobZm2SgsGxxNVYDgHoktDTdS+8zEPB0=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=PYNf/qlpk/+3D3O+RriF9RN0L1Xq/oKZwthicCSgRW34LkfSWB9Ux+iW+XKmIclSS MADiN928Wh7E7K3eRJb0mH8DD/Q0Nqcqtk2X7yNGwkW8P3nK4HNs7pymuqTjYdZMr7 PT00nU652YqX7VCKknzvii8aj4Cfb/GZddsSepno= Received: from smtpng3.m.smailru.net (smtpng3.m.smailru.net [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 0DA436EC6F for ; Fri, 26 Feb 2021 23:23:10 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 0DA436EC6F Received: by smtpng3.m.smailru.net with esmtpa (envelope-from ) id 1lFjdl-0003O1-An; Fri, 26 Feb 2021 23:23:09 +0300 To: Konstantin Osipov , Serge Petrenko , gorcunov@gmail.com, tarantool-patches@dev.tarantool.org References: <20210224193549.70017-1-sergepetrenko@tarantool.org> <20210225130514.GB18388@starling> <20210226071829.GD18388@starling> Message-ID: <8a6c0d7e-2231-510c-788d-daa9644f5e84@tarantool.org> Date: Fri, 26 Feb 2021 21:23:08 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210226071829.GD18388@starling> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 78E4E2B564C1792B X-77F55803: 4F1203BC0FB41BD9795828B892398B728FC1D9EEA0F34691E83D75D24C38BB42182A05F53808504006F23E8020C60E10EA982049A608C9FD4ECBB22EAD8165BE68A3CE5DBEBE324C X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE75644E22E05AA81AEB287FD4696A6DC2FA8DF7F3B2552694A4E2F5AFA99E116B42401471946AA11AF7680F9384605B903880943D09B4F39CC8F08D7030A58E5ADC58D69EE07B14084C6CDE5D1141D2B1C573089F50400AA0336C89D614686FDA11A2B0464345332AD9FA2833FD35BB23D9E625A9149C048EE1E561CDFBCA1751F618001F51B5FD3F9D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8B791E6C230873D55CA471835C12D1D977C4224003CC836476EC64975D915A344093EC92FD9297F6718AA50765F7900637B8F435DEDE9E76EBA7F4EDE966BC389F395957E7521B51C24C7702A67D5C33162DBA43225CD8A89F9FFED5BD9FB4175535872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9AE7E30DF62CE24E4E5BFE6E7EFDEDCD789D4C264860C145E X-C1DE0DAB: 0D63561A33F958A5FD85BAAF2B92FB47954DB004106A597E807BF32DF2793615D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34FACF503ACC83041B94EAEA254608852C906BC8610E03AD631EEF7F4A96F7D0491451C977661E65801D7E09C32AA3244C7DE0EC0BFEB8B28A1EBE60F6C62E9E754DBEAD0ED6C55A80927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj8mqzvzJVEn1BM9Ei/PRlNQ== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822DA886FF0AC0A2B563139EED890A5FB993841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v3] wal: introduce limits on simultaneous writes X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" On 26.02.2021 08:18, Konstantin Osipov wrote: > * Vladislav Shpilevoy [21/02/26 10:15]: >>> I'd also question the place where you decided to put this gate. >>> The source of the issue is async requests, not WAL, which worked >>> fine in absence of async requests. So it's async requests that >>> should be gated, not WAL. >> >> In the commit message it is clearly stated why it is in the >> journal's code, not just in the applier: >> >> The feature is ready for `box.commit{is_async=true}`. Once it's >> implemented, it should check whether the queue is full and let the user >> decide what to do next. Either wait or roll the tx back. >> >> Async transactions will be exposed to 'userspace' to be able to reduce >> latency for network requests ending with a transaction. They won't have >> to wait for WAL write to end. > > You did not understand my comment. I tried to say that a major > part of this code is generic and should reside in lib/core as a > counting semaphore abstraction. Async transaction simply use this > counting semaphore to throttle themselves. Then neither WAL nor > any other resource used by async transactions will be overloaded. > > Otherwise, the system would be allowed to create async > transactions, and while WAL will not overflow, some other resource > (memory, transaction identifiers, whatever) may still overflow. Ok, now I understand. Yeah, I also think it is a good idea to move it libcore if nothing major will change in the patch due to any reason. Talking of the other limits - firstly we need to find if some of them really overflows. Then yes, such a semaphone-thing could be applied there too. But AFAIK, there are no other known similar bugs yet. >>> Otherwise your overflow will just spill out someplace else. >> >> On the contrary. Your proposal to do it in the applier would lead to >> queue overflow in some other place - in userspace. When the queue is >> for the entire WAL, it won't overflow. > > I did not say it should be in the applier. It was a misunderstanding.