From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp21.mail.ru (smtp21.mail.ru [94.100.179.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id DFF7A43D678 for ; Tue, 22 Oct 2019 02:00:53 +0300 (MSK) References: From: Vladislav Shpilevoy Message-ID: <5728488d-3860-8135-bff9-75806178fe5f@tarantool.org> Date: Tue, 22 Oct 2019 01:06:08 +0200 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [Tarantool-patches] [tarantool-patches] [PATCH v3 0/4] From memory wal replication List-Id: Tarantool development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tarantool-patches@freelists.org, georgy@tarantool.org, tarantool-patches@dev.tarantool.org, Konstantin Osipov Hi! Thanks for the patchset! - I see that Travis is still red on your branch: https://travis-ci.org/tarantool/tarantool/jobs/595713541. - Please, rebase your branch on the latest master. It is very outdated. Probably it also will help to fix Travis. On 09/10/2019 18:45, Georgy Kirichenko wrote: > This patchset introduces from-wal-memory-replication and consist of 4 > patches. > > The first patch purges distinct relay facilities to advanced gc consumer > state. Also it changes gc behaviour - now gc compares consumers using > lsns corresponding to local instance id instead of vclock signature. > Howevers this changes gc behavior, its look like we could restore the > gc behavior after relay would be migrated to the wal thread. The whole > patch is essential because in relay switched to memory relaying there is > no more file boundaries and on_close_log trigger invocation now.> > The second patch introduces xrow buffer structure to be used by wal > while logging. Wal encodes rows to a memory buffer (xrow_buffer > structure) and then writes encoded data to a xlog file. > > The next patch implements xrow_buf_cursor to have an ability to found > a position in the buffer by vclock and then fetch all rows one-by-one > towards the last one. > > The last patch makes a relay able to read from wal memory using > a xrow_buffer cursor. When relay went out of wal memory window it turns > back to file mode. After last known xlog file is relayed then relay > tries to return to wal memory mode. > > Changes in v3: > - A lot of commentary fixes and renaming according to review > - Slight refactoring > - On-stack heartbeat message issue fixed > > Changes in v2: > - Get rid of gc refactoring > - Rename wal_memory to xrow_buffer according to review > - Commentaries > - Bug fixes > > Branch: http://github.com/tarantool/tarantool/tree/g.kirichenko/gh-3794-memory-replication > Issue: https://github.com/tarantool/tarantool/issues/3794 > > Georgy Kirichenko (4): > relay: adjust gc state on relay status update > wal: xrow buffer structure > wal: xrow buffer cursor > replication: use wal memory buffer to fetch rows > > src/box/CMakeLists.txt | 1 + > src/box/gc.c | 7 +- > src/box/relay.cc | 286 +++++-------- > src/box/wal.c | 205 +++++++++- > src/box/wal.h | 60 +++ > src/box/xlog.c | 57 ++- > src/box/xlog.h | 16 +- > src/box/xrow_buf.c | 382 ++++++++++++++++++ > src/box/xrow_buf.h | 198 +++++++++ > src/lib/core/cbus.c | 4 + > src/lib/core/errinj.h | 1 + > test/box/errinj.result | 2 + > test/replication/force_recovery.result | 8 + > test/replication/force_recovery.test.lua | 2 + > test/replication/replica_rejoin.result | 8 + > test/replication/replica_rejoin.test.lua | 2 + > .../show_error_on_disconnect.result | 8 + > .../show_error_on_disconnect.test.lua | 2 + > test/replication/suite.ini | 2 +- > test/xlog/panic_on_wal_error.result | 12 + > test/xlog/panic_on_wal_error.test.lua | 3 + > test/xlog/suite.ini | 2 +- > 22 files changed, 1045 insertions(+), 223 deletions(-) > create mode 100644 src/box/xrow_buf.c > create mode 100644 src/box/xrow_buf.h >