[PATCH 0/3] Fix crash on rollback of mixed DML/DDL transaction

[Vladimir Davydov <vdavydov.dev@gmail.com>]

Struct txn_stmt contains a pointer to struct space, which is used by
memtx to revert the statement on transaction rollback. The pointer is
set when a statement is executed. The space a txn_stmt refers to may be
replaced in the space cache by a DDL operation (in case of memtx DDL and
DML statements may be mixed in the same transaction). This is okay as
long as the DDL operation guarantees that the old space remains valid.
All DDL operation observe this rule except MoveIndex and ModifyIndex.
The latter two swap indexes between the old and the new spaces thus
making the old space refer to a dummy place-holder index. This breaks
rollback code assumptions and may lead to a crash.

So this patch set reworks the above-mentioned DDL operations in such
a way that rather than swapping indexes, we copy the old index to
the new space, thus making sure that both spaces stay valid and can
be used for rollback. To achieve that we need to do some refactoring
at the side of the vinyl engine, which actively uses DDL callbacks.
The refactoring is done by the first two patches. The bug is fixed
by the final patch in the series.

https://github.com/tarantool/tarantool/issues/4368
https://github.com/tarantool/tarantool/commits/dv/gh-4368-fix-mixed-ddl-dml-crash

Vladimir Davydov (3):
  vinyl: use update_def index method to update vy_lsm on ddl
  vinyl: move unique check optimization setup to the space level
  alter: fix rollback in case DDL and DML are used in the same
    transaction

 src/box/alter.cc              |  85 +++++++++++++++++---------
 src/box/blackhole.c           |   2 +-
 src/box/key_def.c             |  56 +++++++++++-------
 src/box/key_def.h             |   4 +-
 src/box/memtx_space.c         |   7 ++-
 src/box/space.c               |  53 ++++++++++++++---
 src/box/space.h               |  77 +++++++++++++++---------
 src/box/sysview.c             |   2 +-
 src/box/vinyl.c               | 108 ++++++++++++++--------------------
 src/box/vy_lsm.c              |  18 +++++-
 src/box/vy_lsm.h              |  17 +++---
 test/box/transaction.result   |  27 +++++++++
 test/box/transaction.test.lua |  20 +++++++
 13 files changed, 310 insertions(+), 166 deletions(-)

-- 
2.20.1