From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Mon, 18 Jun 2018 12:44:07 +0300 From: Vladimir Davydov Subject: Re: [tarantool-patches] [xlog 1/1] xlog: Remove inprogress files on start Message-ID: <20180618094407.ue6ll2yazn4dsgq5@esperanza> References: <2e9c43245524d5b760386ed78c68f7bf1bde0ed4.1528555563.git.imarkov@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2e9c43245524d5b760386ed78c68f7bf1bde0ed4.1528555563.git.imarkov@tarantool.org> To: Ilya Markov Cc: georgy@tarantool.org, tarantool-patches@freelists.org List-ID: 0. Please carefully read and follow our guideline on how to submit patches: https://tarantool.io/en/doc/1.9/dev_guide/developer_guidelines.html#how-to-submit-a-patch-for-review In particular, the subject prefix should be PATCH (default if you use git-format-patch), not subsystem name, and the branch name as well as the GitHub ticket should be given as hyperlinks. When you resubmit a new version of your patch, please don't forget to append the version to the subject prefix (e.g. PATCH v2), and write a brief change log, e.g. see https://www.freelists.org/post/tarantool-patches/PATCH-v2-box-make-gc-info-public I'm kinda tired of reiterating those simple rules over and over again. I think I'll simply ignore all patches that don't conform to the guideline... Also, before submitting a patch for review, please make sure there are no stray hunks, like adding/removing blank lines. Your patch has a few. Fixing a comment or removing a blank line is OK only if it goes to a hunk you need anyway. 1. engine_vtab::commit_checkpoint() must not fail, because if it does, it may leave the box in inconsistent state, when there's no vinyl checkpoint corresponding to a memtx snapshot. An attempt to recover from such a state will fail. That's why we panic in memtx_engine_commit_checkpoint(). To test deletion of snap.inprogress files, you don't need to change that - it's enough to set the error injection that delays renaming of the snap file (it already exists), start checkpoint in a fiber, and then restart the server. 2. Removal of engine-specific files should be done in engine callbacks. For instance, snap.inprogress files should probably be removed from memtx_engine_end_recovery(). 3. Scanning vinyl_dir is totally wrong, because information about all vinyl files, even .inprogress ones, is stored in vylog. Moreover, code that removes incomplete vinyl files already exists - see vinyl_engine_end_recovery() - all you need to do is implement removal of .inprogress files there. On Sat, Jun 09, 2018 at 05:46:37PM +0300, Ilya Markov wrote: > When tarantool crashes during writing to vylog, index, run or snapshot, > inprogress files remain. But garbage collector doesn't take into account > these files. So they remain until they are removed manually. > > Fix this with adding function to box_cfg which traverses memtx_dir and > vinyl_dir and removes inprogress files. > > * Add 4 errinj which simulate the crash before renaming inprogress > files. > * Change signature of engine_commit_checkpoint function in order to > support error injection in snapshot commit. > > Closes #3406 > --- > branch: gh-3406-remove-inprogress-files > src/box/box.cc | 20 +++++- > src/box/engine.c | 3 +- > src/box/engine.h | 2 +- > src/box/memtx_engine.c | 21 +++++- > src/box/sysview_engine.c | 3 +- > src/box/vinyl.c | 3 +- > src/box/vy_log.c | 4 ++ > src/box/vy_run.c | 14 ++++ > src/box/xlog.c | 73 +++++++++++++++++++- > src/box/xlog.h | 13 ++++ > src/errinj.h | 4 ++ > test/box/errinj.result | 171 ++++++++++++++++++++++++++++++++++++++++++++++- > test/box/errinj.test.lua | 59 ++++++++++++++++ > 13 files changed, 378 insertions(+), 12 deletions(-)