Re: [tarantool-patches] [xlog 1/1] xlog: Remove inprogress files on start

[Vladimir Davydov <vdavydov.dev@gmail.com>]

0. Please carefully read and follow our guideline on how to submit
   patches:

   https://tarantool.io/en/doc/1.9/dev_guide/developer_guidelines.html#how-to-submit-a-patch-for-review

   In particular, the subject prefix should be PATCH (default if you use
   git-format-patch), not subsystem name, and the branch name as well as
   the GitHub ticket should be given as hyperlinks.

   When you resubmit a new version of your patch, please don't forget to
   append the version to the subject prefix (e.g. PATCH v2), and write a
   brief change log, e.g. see

   https://www.freelists.org/post/tarantool-patches/PATCH-v2-box-make-gc-info-public

   I'm kinda tired of reiterating those simple rules over and over
   again. I think I'll simply ignore all patches that don't conform to
   the guideline...

   Also, before submitting a patch for review, please make sure there
   are no stray hunks, like adding/removing blank lines. Your patch has
   a few. Fixing a comment or removing a blank line is OK only if it
   goes to a hunk you need anyway.

1. engine_vtab::commit_checkpoint() must not fail, because if it does,
   it may leave the box in inconsistent state, when there's no vinyl
   checkpoint corresponding to a memtx snapshot. An attempt to recover
   from such a state will fail. That's why we panic in
   memtx_engine_commit_checkpoint().

   To test deletion of snap.inprogress files, you don't need to change
   that - it's enough to set the error injection that delays renaming of
   the snap file (it already exists), start checkpoint in a fiber, and
   then restart the server.

2. Removal of engine-specific files should be done in engine callbacks.
   For instance, snap.inprogress files should probably be removed from
   memtx_engine_end_recovery().

3. Scanning vinyl_dir is totally wrong, because information about all
   vinyl files, even .inprogress ones, is stored in vylog. Moreover,
   code that removes incomplete vinyl files already exists - see
   vinyl_engine_end_recovery() - all you need to do is implement removal
   of .inprogress files there.

On Sat, Jun 09, 2018 at 05:46:37PM +0300, Ilya Markov wrote:
> When tarantool crashes during writing to vylog, index, run or snapshot,
> inprogress files remain. But garbage collector doesn't take into account
> these files. So they remain until they are removed manually.
> 
> Fix this with adding function to box_cfg which traverses memtx_dir and
> vinyl_dir and removes inprogress files.
> 
> * Add 4 errinj which simulate the crash before renaming inprogress
>   files.
> * Change signature of engine_commit_checkpoint function in order to
> support error injection in snapshot commit.
> 
> Closes #3406
> ---
> branch: gh-3406-remove-inprogress-files
>  src/box/box.cc           |  20 +++++-
>  src/box/engine.c         |   3 +-
>  src/box/engine.h         |   2 +-
>  src/box/memtx_engine.c   |  21 +++++-
>  src/box/sysview_engine.c |   3 +-
>  src/box/vinyl.c          |   3 +-
>  src/box/vy_log.c         |   4 ++
>  src/box/vy_run.c         |  14 ++++
>  src/box/xlog.c           |  73 +++++++++++++++++++-
>  src/box/xlog.h           |  13 ++++
>  src/errinj.h             |   4 ++
>  test/box/errinj.result   | 171 ++++++++++++++++++++++++++++++++++++++++++++++-
>  test/box/errinj.test.lua |  59 ++++++++++++++++
>  13 files changed, 378 insertions(+), 12 deletions(-)