From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 4CA2428878 for ; Tue, 13 Aug 2019 02:27:53 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7hGfSmmuAnpB for ; Tue, 13 Aug 2019 02:27:53 -0400 (EDT) Received: from smtp39.i.mail.ru (smtp39.i.mail.ru [94.100.177.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 69D5C28857 for ; Tue, 13 Aug 2019 02:27:52 -0400 (EDT) From: Georgy Kirichenko Subject: [tarantool-patches] [PATCH 0/7] Replication: In-memory replication Date: Tue, 13 Aug 2019 09:27:38 +0300 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: tarantool-patches@freelists.org Cc: Georgy Kirichenko This patchset introduces from-wal-memory-replication and consist of 4 preparation patches, one patch that introduces memory-replication and 2 follow ups that clear unused facility from tarantool code. First patch introduces a wal writer fiber and write queue what is needed in order to have an ability to yield during wal writes. Without this functionality there is no possibility to make a cbus call from wal writer functions. Second patch moves wal dir tracking into gc state. This is required to simplify relay state tracking as in-memory relay would not have any glue about wal files boundaries. Third patch removes wal boundaries tracking from relay Forth patch prepares wal to write all data in a memory buffer and then write the data to a file. This memory buffer contains raw encoded data and xrow_header index to support navigation thorough the data. The next patch allows a relay to attach to relay and call send callback on each row stored in a wal memory buffer. Two subsequent patches removes wal_watcher and on_close_log triggers as unused. Branch: https://github.com/tarantool/tarantool/tree/g.kirichenko/gh-3794-memory-replication Issue: https://github.com/tarantool/tarantool/issues/3794 Georgy Kirichenko (7): Refactoring: wal writer fiber and queue Refactoring: Track wal files using gc state. Replication: Relay does not rely on xlog boundaries Replication: wal memory buffer Replication: in memory replication Refactoring: remove wal_watcher routines Refactoring: get rid of on_close_log src/box/CMakeLists.txt | 1 + src/box/box.cc | 25 +- src/box/gc.c | 131 ++++- src/box/gc.h | 34 +- src/box/lua/info.c | 25 +- src/box/memtx_engine.c | 2 +- src/box/recovery.cc | 10 +- src/box/recovery.h | 5 +- src/box/relay.cc | 263 +++------ src/box/vy_log.c | 4 +- src/box/wal.c | 742 +++++++++++++++----------- src/box/wal.h | 97 +--- src/box/wal_mem.c | 273 ++++++++++ src/box/wal_mem.h | 166 ++++++ src/box/xlog.c | 87 ++- src/box/xlog.h | 11 +- test/replication/gc_no_space.result | 4 +- test/replication/gc_no_space.test.lua | 4 +- 18 files changed, 1208 insertions(+), 676 deletions(-) create mode 100644 src/box/wal_mem.c create mode 100644 src/box/wal_mem.h -- 2.22.0