From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 8CF626F3C7; Fri, 26 Mar 2021 16:42:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8CF626F3C7 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1616766123; bh=/s39e/CE5Djx55bwbg0zdvnD4+K4KKj3861MRKeTHwo=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=HusKw1CxRZhoo1j4rjI6/9hjhqKwm/7S0Trmc+BI795wWbeMmB+rSEBf3Yal5NxS3 cwwUhSf9QSmY0HBK/UzAQWVfQR9aZvsXYPb5FcQQDsA4Dta6yrEWo5K6VmuWzv3imA iMbHTzHBTfIJmyGYFwjms6O64NjF1I8xdhTpXKTE= Received: from smtp50.i.mail.ru (smtp50.i.mail.ru [94.100.177.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 65D936F3C7 for ; Fri, 26 Mar 2021 16:42:02 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 65D936F3C7 Received: by smtp50.i.mail.ru with esmtpa (envelope-from ) id 1lPmiv-0005Ra-9k; Fri, 26 Mar 2021 16:42:01 +0300 To: Cyrill Gorcunov , tml References: <20210326120605.2160131-1-gorcunov@gmail.com> <20210326120605.2160131-2-gorcunov@gmail.com> Message-ID: Date: Fri, 26 Mar 2021 16:42:00 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <20210326120605.2160131-2-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9064ADF4728AA0EE9587C800CEAD38A69F042767B24605C2B182A05F53808504052D61BCA5AB9036B5A1C421DAEF55D1687BDFE4F55CF049AB55BDDA44415F554 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE75210414551E8CD62EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006373D58C44ED3182E498638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C5DD32608FC869F5D2A8DB6E344E04699EA0B5BE7809F9387A471835C12D1D9774AD6D5ED66289B5278DA827A17800CE78A0F7C24A37A3D769FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3FB12F4B11BB5604F117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CF9239C50D91B3AEA5BA3038C0950A5D36C8A9BA7A39EFB766EC990983EF5C0329BA3038C0950A5D36D5E8D9A59859A8B6D323265F4920F39D76E601842F6C81A1F004C906525384307823802FF610243DF43C7A68FF6260569E8FC8737B5C2249B372FE9A2E580EFC725E5C173C3A84C3EB9EA62DB0D2896935872C767BF85DA2F004C90652538430E4A6367B16DE6309 X-C1DE0DAB: 0D63561A33F958A5C9FBBE829A09532CE1D9575E138324BF9CF918B44CA732EDD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D340B06C327CE4A70E8A21374FDE5EAB7FA1EF9E4991ED41847DBD87EDDBB595A0A39BD5A021ADBA7B41D7E09C32AA3244C024C6C7FE019A1CA5F0766793A7A63DE259227199D06760A927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojapPp7P/VpAi+Wd8RQ4nv0w== X-Mailru-Sender: 583F1D7ACE8F49BDD2846D59FC20E9F8C3A060058CA540FFAA9F2A23D969826393B4A7D274D4FA80424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v5 1/3] gc/xlog: delay xlog cleanup until relays are subscribed X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 26.03.2021 15:06, Cyrill Gorcunov пишет: > In case if replica managed to be far behind the master node > (so there are a number of xlog files present after the last > master's snapshot) then once master node get restarted it > may clean up the xlogs needed by the replica to subscribe > in a fast way and instead the replica will have to rejoin > reading a number of data back. > > Lets try to address this by delaying xlog files cleanup > until replicas are got subscribed and relays are up > and running. For this sake we start with cleanup fiber > spinning in nop cycle ("paused" mode) and use a delay > counter to wait until relays decrement them. > > This implies that if `_cluster` system space is not empty > upon restart and the registered replica somehow vanished > completely and won't ever come back, then the node > administrator has to drop this replica from `_cluster` > manually. > > Note that this delayed cleanup start doesn't prevent > WAL engine from removing old files if there is no > space left on a storage device. The WAL will simply > drop old data without a question. > > We need to take into account that some administrators > might not need this functionality at all, for this > sake we introduce "wal_cleanup_delay" configuration > option which allows to enable or disable the delay. > > Closes #5806 > > Signed-off-by: Cyrill Gorcunov > > @TarantoolBot document > Title: Add wal_cleanup_delay configuration parameter > > The `wal_cleanup_delay` option defines a delay in seconds > before write ahead log files (`*.xlog`) are getting started > to prune upon a node restart. > > This option is ignored in case if a node is running as > an anonymous replica (`replication_anon = true`). Similarly > if replication is unused or there is no plans to use > replication at all then this option should not be considered. > > An initial problem to solve is the case where a node is operating > so fast that its replicas do not manage to reach the node state > and in case if the node is restarted at this moment (for various > reasons, for example due to power outage) then `*.xlog` files might > be pruned during restart. In result replicas will not find these > files on the main node and have to reread all data back which > is a very expensive procedure. > > Since replicas are tracked via `_cluster` system space this we use > its content to count subscribed replicas and when all of them are > up and running the cleanup procedure is automatically enabled even > if `wal_cleanup_delay` is not expired. > > The `wal_cleanup_delay` should be set to: > > - `0` to disable the cleanup delay; > - `>= 0` to wait for specified number of seconds. > > By default it is set to `14400` seconds (ie `4` hours). > > In case if registered replica is lost forever and timeout is set to > infinity then a preferred way to enable cleanup procedure is not setting > up a small timeout value but rather to delete this replica from `_cluster` > space manually. > > Note that the option does *not* prevent WAL engine from removing > old `*.xlog` files if there is no space left on a storage device, > WAL engine can remove them in a force way. > > Current state of `*.xlog` garbage collector can be found in > `box.info.gc()` output. For example > > ``` Lua > tarantool> box.info.gc() > --- > ... > is_paused: false > ``` > > The `is_paused` shows if cleanup fiber is paused or not. > --- > .../unreleased/add-wal_cleanup_delay.md | 5 + > src/box/box.cc | 41 ++++++++ > src/box/box.h | 1 + > src/box/gc.c | 95 ++++++++++++++++++- > src/box/gc.h | 36 +++++++ > src/box/lua/cfg.cc | 9 ++ > src/box/lua/info.c | 4 + > src/box/lua/load_cfg.lua | 5 + > src/box/relay.cc | 1 + > src/box/replication.cc | 2 + > test/app-tap/init_script.result | 1 + > test/box/admin.result | 2 + > test/box/cfg.result | 4 + > test/replication/replica_rejoin.lua | 22 +++++ > test/replication/replica_rejoin.result | 18 +++- > test/replication/replica_rejoin.test.lua | 11 ++- > test/vinyl/replica_rejoin.lua | 5 +- > test/vinyl/replica_rejoin.result | 13 +++ > test/vinyl/replica_rejoin.test.lua | 8 ++ > 19 files changed, 275 insertions(+), 8 deletions(-) > create mode 100644 changelogs/unreleased/add-wal_cleanup_delay.md > create mode 100644 test/replication/replica_rejoin.lua Thanks for the  patch! LGTM. -- Serge Petrenko