From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp15.mail.ru (smtp15.mail.ru [94.100.176.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3F732469719 for ; Wed, 18 Mar 2020 22:48:05 +0300 (MSK) From: Serge Petrenko Date: Wed, 18 Mar 2020 22:47:43 +0300 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v2 0/5] replication: fix local space tracking List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: v.shpilevoy@tarantool.org, kostja.osipov@gmail.com, georgy@tarantool.org Cc: tarantool-patches@dev.tarantool.org https://github.com/tarantool/tarantool/issues/4114 https://github.com/tarantool/tarantool/tree/sp/gh-4114-local-space-replication The patchset contains 2 of Georgy's patches perorming WAL gc rework. The first patch introduces a matrix clock structure, which contains the latest known vclocks of all the cluster members and can be used to build a vclock, which is less than or equal to all the member vclocks. Such a vclock is used then in the second patch, where gc is rewritten to stop tracking needed WAL files by vclock signature. Now replica vclocks are used to define which files are still needed. These two patches allow to start using a 0 vclock component to track local space changes and ignore this component when finding WALs needed by a replica. Patches 3 and 4 are preparation for the patch 5, which performs the task itself. Now every replica has its own 0-th vclock component, which doesn't match any of the other instance components and indicates the number of local space requests executed. Georgy Kirichenko (2): box: introduce matrix clock wal: track consumer vclock and collect logs in wal thread Serge Petrenko (3): vclock: add an ability to set individual clock components replication: hide 0-th vclock components in replication responses box: start counting local space requests separately src/box/CMakeLists.txt | 4 + src/box/applier.cc | 5 +- src/box/box.cc | 70 ++-- src/box/gc.c | 216 +++------- src/box/gc.h | 95 +---- src/box/lua/info.c | 29 +- src/box/mclock.c | 394 ++++++++++++++++++ src/box/mclock.h | 151 +++++++ src/box/relay.cc | 157 ++----- src/box/replication.cc | 47 +-- src/box/replication.h | 3 - src/box/vclock.c | 15 + src/box/vclock.h | 11 + src/box/wal.c | 286 +++++++++++-- src/box/wal.h | 17 +- test/replication/anon.result | 5 + test/replication/anon.test.lua | 2 + test/replication/autobootstrap.result | 6 + test/replication/autobootstrap.test.lua | 2 + test/replication/before_replace.result | 8 +- test/replication/before_replace.test.lua | 4 +- test/replication/gc_no_space.result | 30 +- test/replication/gc_no_space.test.lua | 12 +- .../gh-4114-local-space-replication.result | 125 ++++++ .../gh-4114-local-space-replication.test.lua | 48 +++ test/replication/local_spaces.result | 4 + test/replication/local_spaces.test.lua | 3 + test/replication/misc.result | 6 + test/replication/misc.test.lua | 2 + test/replication/quorum.result | 6 + test/replication/quorum.test.lua | 2 + test/replication/replica_rejoin.result | 9 + test/replication/replica_rejoin.test.lua | 3 + test/replication/skip_conflict_row.result | 3 + test/replication/skip_conflict_row.test.lua | 1 + test/replication/suite.cfg | 1 + test/unit/CMakeLists.txt | 2 + test/unit/mclock.result | 18 + test/unit/mclock.test.c | 160 +++++++ test/vinyl/errinj.result | 5 + test/vinyl/errinj.test.lua | 4 + 41 files changed, 1430 insertions(+), 541 deletions(-) create mode 100644 src/box/mclock.c create mode 100644 src/box/mclock.h create mode 100644 test/replication/gh-4114-local-space-replication.result create mode 100644 test/replication/gh-4114-local-space-replication.test.lua create mode 100644 test/unit/mclock.result create mode 100644 test/unit/mclock.test.c -- 2.21.1 (Apple Git-122.3)