From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 5768D6EC56; Fri, 19 Mar 2021 18:36:09 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 5768D6EC56 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1616168169; bh=hd+Ja3ZcZ7rvyaKxA3PT75EvfNmXtKVi0F29KosgxBI=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=MZdeqKW16ygkbwJbEf3Lsn6JWri97rtF6jn3Qu5mij2ViBuPbjQHGtzOtaRpJVlBM xhszl3zlv9A5Rxdi8EHAhjDJJzKzvv6MFpwXGDzpJSnUh7m/lvT25KOXxZybkJEqZD 3lAcd5eZ1jSr7wpxNyx3BXrRbXkLaFPYz42XH9Rk= Received: from smtp44.i.mail.ru (smtp44.i.mail.ru [94.100.177.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 45F8E6EC56 for ; Fri, 19 Mar 2021 18:36:07 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 45F8E6EC56 Received: by smtp44.i.mail.ru with esmtpa (envelope-from ) id 1lNHAU-0008BN-Fm; Fri, 19 Mar 2021 18:36:06 +0300 Date: Fri, 19 Mar 2021 15:36:06 +0000 To: Serge Petrenko Message-ID: <20210319153606.ccbnnvje3eb6mzil@tarantool.org> References: <20210224193549.70017-1-sergepetrenko@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20210224193549.70017-1-sergepetrenko@tarantool.org> User-Agent: NeoMutt/20170113 (1.7.2) X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD95D6E7CC48CB1F5F1C82687294EF6886B62CC997B0B8C95C4182A05F53808504042D6771EF7E021626EB7D8E5D0BC39BF0482F5623100BAD2F1A63146C6F7D1A2 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78E8764B5BC580342EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006370758E51FBCC366D98638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C81A23F326053F827B90239574C92D595C2BE28E7AFA131AAA471835C12D1D9774AD6D5ED66289B5278DA827A17800CE709B92020B71E24959FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3FD59DAE6580CC3C3117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CF619183A7BD6BC6F13AA81AA40904B5D9CF19DD082D7633A078D18283394535A93AA81AA40904B5D98AA50765F79006379390549D7D73CB0BD81D268191BDAD3D698AB9A7B718F8C442539A7722CA490C13377AFFFEAFD26923F8577A6DFFEA7CE0F3BA37685B2B9093EC92FD9297F6715571747095F342E857739F23D657EF2BD5E8D9A59859A8B67122B551D37F75FD089D37D7C0E48F6C5571747095F342E857739F23D657EF2B6825BDBE14D8E7025EC15B47EAE72BACBD9CCCA9EDD067B1EDA766A37F9254B7 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D2FF9408C3F8AEA28355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A5E3862F20535DE0DCAFA99DBC152D7634BE4E1F89DC54611ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3435B1F14A93C4073F72A5D25CE46C6BE9887753BF7AAF12D073DEF0A14EC803964BA55A078EED081A1D7E09C32AA3244CDC93574150A72EF0474795FCEA15A7363E8609A02908F271927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojyKyJYJ15DtJEU14ZI2BuzA== X-Mailru-Sender: 05EB39F83D09414F9B5D7F52D7CDFDD2DCB79837F78C3691D3833784524468CE57CC1FBD6EC305E6CA16B95394F0DD5CE99530A0C0F27B5268329DCED823713783C0E760C018FF54112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v3] wal: introduce limits on simultaneous writes X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Kirill Yukhin via Tarantool-patches Reply-To: Kirill Yukhin Cc: tarantool-patches@dev.tarantool.org, v.shpilevoy@tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hello, On 24 фев 22:35, Serge Petrenko via Tarantool-patches wrote: > Since the introduction of asynchronous commit, which doesn't wait for a > WAL write to succeed, it's quite easy to clog WAL with huge amounts > write requests. For now, it's only possible from an applier, since it's > the only user of async commit at the moment. > > This happens when replica is syncing with master and reads new > transactions at a pace higher than it can write them to WAL (see docbot > request for detailed explanation). > > To ameliorate such behavior, we need to introduce some limit on > not-yet-finished WAL write requests. This is what this commit is trying > to do. > Two new counters are added to wal writer: queue_size (in bytes) and > queue_len (in wal messages) together with configuration settings: > `wal_queue_max_size` and `wal_queue_max_len`. > Size and length are increased on every new submitted request, and are > decreased once the tx receives a confirmation that a specific request > was written. > Actually, the limits are added to an abstract journal, but take effect > only for wal writer. > > Once size or length reach their maximum values, applier is blocked until > some of the write requests are finished. > > The size limit isn't strict, i.e. if there's at least one free byte, the > whole write request fits and no blocking is involved. > > The feature is ready for `box.commit{is_async=true}`. Once it's > implemented, it should check whether the queue is full and let the user > decide what to do next. Either wait or roll the tx back. > > Part of #5536 I've checked your patch into 2.6, 2.7 and master. -- Regards, Kirill Yukhin