From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Mon, 19 Mar 2018 18:01:34 +0300 From: Vladimir Davydov Subject: Re: [PATCH 2/4] vinyl: do not use index lsn to identify indexes in vylog Message-ID: <20180319150133.ykm4rqjiz46prxb3@esperanza> References: <75fb076a21047e6e2d3a89ae895417b318de6d1b.1521306336.git.vdavydov.dev@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <75fb076a21047e6e2d3a89ae895417b318de6d1b.1521306336.git.vdavydov.dev@gmail.com> To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: Please ignore this patch for now. We can't use space_id/index_id for identifying indexes in vylog after all. The problem is during ALTER we have to maintain two indexes with the same space_id/index_id. It seems that we need a unique index identifier in vylog after all, and it can't be LSN. I'll rework the patch. On Sat, Mar 17, 2018 at 08:56:35PM +0300, Vladimir Davydov wrote: > An index can be dropped and then recreated with the same space/index id. > To discriminate between different incarnations of the same index during > recovery, we use LSN from the time of index creation, as it is supposed > to be unique. However, the uniqueness property won't hold once ALTER is > implemented for vinyl spaces: the problem is we have to rebuild all > indexes of a space and commit them simultaneously in case the primary > index is ALTERED. > > Actually, we don't really need a unique index ID stored both in vylog > and in WAL to recover all incarnations of vinyl indexes. Since WAL is > replayed in the chronological order, we can find the index corresponding > to index creation statement replayed from WAL by its space_id/index_id > and the number of times the index has been dropped since we started to > replay WAL. As we don't actually recover indexes that are dropped during > WAL recovery, we don't need to know their properties, we just need to > know that they existed. So we don't need to store a separate object for > each index incarnation in the recovery context, it's enough to add a > counter representing the number of incarnations the index has had since > the last checkpoint to the vy_index_recovery_info structure to be able > to recover all incarnations of the same vinyl index. Then on recovery, > vy_index_recover() would check the incarnation counter of the index and > > - recover the last incarnation if it is 1 > - emit create/drop record if it is > 1 > - do nothing if it is 0 > > and decrement the counter on success so that the next call to > vy_index_recover() with the same space and index id proceeds to > the next incarnation of this index. And this is what this patch > does in a nutshell. > > The tricky part is preserving backward compatibility. We can't just > replace index_lsn with space_id/index_id everywhere, because there > still can be records that use index_lsn to identify an index and we > need to process them. For example, VY_LOG_CREATE_INDEX is fine in > this regard, because it contains both space_id/index_id and index_lsn, > but VY_LOG_DROP_INDEX isn't, as it only has index_lsn. So we have to > maintain two index hash tables during vylog recovery instead of just > one: one for new records where index is referenced by space_id/index_id > and another for legacy records where index_lsn is used instead. > --- > src/box/vinyl.c | 39 ++----- > src/box/vy_index.c | 54 ++++----- > src/box/vy_index.h | 12 +- > src/box/vy_log.c | 286 ++++++++++++++++++++++++++++------------------- > src/box/vy_log.h | 97 +++++++++++----- > src/box/vy_scheduler.c | 12 +- > test/unit/vy_log_stub.c | 2 +- > test/vinyl/layout.result | 32 +++--- > 8 files changed, 300 insertions(+), 234 deletions(-)