From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B0B3D6EC5A; Sat, 27 Feb 2021 01:44:34 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B0B3D6EC5A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614379474; bh=Ei8J7S/BQz6dtv/2/n4CunLGHTThQng1Ili9Bat2G3g=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=KGLUBCbDlwQu7POtzrel2EHYWPKfqgUgvz8FDy7W9GuM4tvrtvpKfgJdyldEhSgX8 0jTHX70YUZua2ZSe30LxkxnudyIMbYDpvxQgYBRIM/I85XsSr0Iqwb3d0hafINY7fU RyjLg/tLBKSoAVxuxFERa3bryRE0hcxOnslzoFjQ= Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 840C16EC5A for ; Sat, 27 Feb 2021 01:44:33 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 840C16EC5A Received: by smtpng2.m.smailru.net with esmtpa (envelope-from ) id 1lFlqa-0005l0-Q5; Sat, 27 Feb 2021 01:44:33 +0300 To: Konstantin Osipov , Serge Petrenko , gorcunov@gmail.com, tarantool-patches@dev.tarantool.org References: <20210224193549.70017-1-sergepetrenko@tarantool.org> <20210225130514.GB18388@starling> <20210226071829.GD18388@starling> <8a6c0d7e-2231-510c-788d-daa9644f5e84@tarantool.org> <20210226212034.GE84448@starling> Message-ID: Date: Fri, 26 Feb 2021 23:44:31 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210226212034.GE84448@starling> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9795828B892398B7267196FBA0C7705D53A7B514E12B9016D182A05F5380850408557BC6143421A829FFA374C70E16077D9370C523B29BB307E6EE12F89CA4367 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE792C68BF9CD4C0E9EEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063726CA83C7ABDB938E8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC5E24787EB5006C790C0C94966C5284A9528D9F86D94640EF389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0E8801E8C4F941C018941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6B4CA5BC574AE2910CC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC08F9A42B2210255C75ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4930A3850AC1BE2E7355E1C53F199C2BB95B5C8C57E37DE458B4C7702A67D5C3316FA3894348FB808DB985633C00BAEBE4F574AF45C6390F7469DAA53EE0834AAEE X-C1DE0DAB: 0D63561A33F958A5256332A224AFF7A43F3D23A7478C5B31561A30247182C3CFD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75968C9853642EB7C3410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34B9F55CA4D2956E30EBAFE8BF07E6E47922C7C85D9861D03D7FA807857E752796A91B6A45C63545031D7E09C32AA3244C5267352282920963B835546DF5F82B9305AB220A9D022EBC927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj8mqzvzJVEn1+TN8FmWQCZA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A06382287535261902240553BAA84170DFDA29B3841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v3] wal: introduce limits on simultaneous writes X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" On 26.02.2021 22:20, Konstantin Osipov wrote: > * Vladislav Shpilevoy [21/02/26 23:24]: >> Talking of the other limits - firstly we need to find if some of them >> really overflows. Then yes, such a semaphone-thing could be applied >> there too. But AFAIK, there are no other known similar bugs yet. > > Exploring this rather theoretically, since there are no user async > transactions yet, I can imagine such transaction takes up memory > and then blocks on WAL semaphore. If there is no limit on the > number of async transactions, it can be a lot of memory. On the > other hand this can be limited by a yet another semaphore. Yes, such transactions will only occupy memory. But the plan is return an error from box.commit({is_async}) if the WAL queue is full already. Because we don't want to block the async commit in anyway. Better bail out earlier and give the user a chance to call normal box.commit() if necessary. Or introduce some kind of 'try_async' to block if the queue is full, but no block if not full, I don't know. We didn't think on the API yet. There will be a limit both on number of transactions and on their total size. The limits are introduced right in this patch, and will be used by async transactions in the future. The main usage, if I understand correctly, will be for latency-sensitive applications, when you send your transactions/calls/IPROTO commands via network, send your data to WAL and return response to the client immediately. You don't have to wait even until writev() completes. Interestingly, it won't lead to loss of any guarantees, because anyway completion of writev() does not mean anything. Still can power off before it hits the disk, and still the node can fail loosing all non-replicated data. So seems like a good feature. Especially should be good for synchrononus transactions, where commit duration can be quite big.