Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: Konstantin Osipov <kostja.osipov@gmail.com>
Cc: tarantool-patches@freelists.org
Subject: Re: [PATCH 3/6] box: do not rotate WAL when replica subscribes
Date: Tue, 27 Nov 2018 12:57:00 +0300	[thread overview]
Message-ID: <20181127095700.hzsiirpb2pg6ncau@esperanza> (raw)
In-Reply-To: <20181126175037.GD7839@chai>

On Mon, Nov 26, 2018 at 08:50:37PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev@gmail.com> [18/11/26 10:27]:
> 
> OK to push.
> > Because this is pointless and confusing. This "feature" was silently
> > introduced by commit f2bccc18485d ("Use WAL vclock instead of TX vclock
> > in most places"). Let's revert this change. This will allow us to
> > clearly separate WAL checkpointing from WAL flushing, which will in turn
> > facilitate implementation of the checkpoint-on-WAL-threshold feature.
> > 
> > There are two problems here, however. First, not rotating the log breaks
> > expectations of replication/gc test: an xlog file doesn't get deleted in
> > time as a consequence. This happens, because we don't delete xlogs
> > relayed to a replica after join stage is complete - we only do it during
> > subscribe stage - and if we don't rotate WAL on subscribe the garbage
> > collector won't be invoked. This is actually a bug - we should advance
> > the WAL consumer associated with a replica once join stage is complete.
> > This patch fixes it, but it unveils another problem - this time in the
> > WAL garbage collection procedure.
> > 
> > Turns out, when passed a vclock, the WAL garbage collection procedure
> > removes all WAL files that were created before the vclock. Apparently,
> > this isn't quite correct - if a consumer is in the middle of a WAL file,
> > we must not delete the WAL file, but we do. This works as long as
> > consumers never track vlcocks inside WAL files - currently they are
> > advanced only when a WAL file is closed and naturally they are advanced
> > to the beginning of the next WAL file. However, if we want to advance
> > the consumer associated with a replica when join stage ends (this is
> > what the previous paragraph is about), it might occur that we will
> > advance it to the middle of a WAL file. If that happens the WAL garbage
> > collector might remove a file which is actually in use by a replica.
> > Fix this as well.

Pushed to 2.1

  reply	other threads:[~2018-11-27  9:57 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-25 13:48 [PATCH 0/6] WAL garbage collection and checkpointing fixes Vladimir Davydov
2018-11-25 13:48 ` [PATCH 1/6] vclock: allow to use const vclock as search key Vladimir Davydov
2018-11-26 17:38   ` Konstantin Osipov
2018-11-27  9:56     ` Vladimir Davydov
2018-11-25 13:48 ` [PATCH 2/6] engine: pass vclock instead of lsn to collect_garbage callback Vladimir Davydov
2018-11-26 17:41   ` Konstantin Osipov
2018-11-27  9:56     ` Vladimir Davydov
2018-11-25 13:48 ` [PATCH 3/6] box: do not rotate WAL when replica subscribes Vladimir Davydov
2018-11-26 17:50   ` Konstantin Osipov
2018-11-27  9:57     ` Vladimir Davydov [this message]
2018-11-25 13:48 ` [PATCH 4/6] box: use replicaset.vclock in replica join/subscribe Vladimir Davydov
2018-11-26 17:54   ` Konstantin Osipov
2018-11-27  9:57     ` Vladimir Davydov
2018-11-25 13:48 ` [PATCH 5/6] wal: separate checkpoint and flush paths Vladimir Davydov
2018-11-26 17:58   ` Konstantin Osipov
2018-11-26 20:19     ` Vladimir Davydov
2018-11-28 16:46       ` Konstantin Osipov
2018-11-25 13:48 ` [PATCH 6/6] wal: remove files needed for recovery from backup checkpoints on ENOSPC Vladimir Davydov
2018-11-28 16:14   ` Vladimir Davydov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181127095700.hzsiirpb2pg6ncau@esperanza \
    --to=vdavydov.dev@gmail.com \
    --cc=kostja.osipov@gmail.com \
    --cc=tarantool-patches@freelists.org \
    --subject='Re: [PATCH 3/6] box: do not rotate WAL when replica subscribes' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox