From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 8FD596EC55; Fri, 10 Sep 2021 18:29:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8FD596EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1631287757; bh=rTacDxpaizyydPZPBQiGrFher0tDfVeg3gFLncCc8FQ=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=q1YCBGjGBLAkvKAAsfhny4B+wytQordwUTm6Z1SLdz1m7FOnV+tSG14QalOX98xw1 tBsrJUBN6IvgURG8A4aQ1IldK/PO1BncRDj7NFgWX8hssVEsurjFPMlqcRi+ZcQ0lN /YWxvH4GaKIuyB9+yzpCTKT/5VZn4V/ShyqB7lKQ= Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 5278F6EC55 for ; Fri, 10 Sep 2021 18:29:15 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 5278F6EC55 Received: by mail-lf1-f46.google.com with SMTP id f18so4772675lfk.12 for ; Fri, 10 Sep 2021 08:29:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=St+/ao6qxEgrGt+7yBDUatZbDH9UpG2YTc/XoAXIXy0=; b=zI9ng5AWDWgQhINkSmvju9dfU6GiUx2gMGhpnH72f902S8ZC/V1Fu0PT6LU+IEm7zA A7NExPxETfaN+tc6gU25JlO9eJEyemdOv4b1h3nFEesJdM/TI067b99PiZ4ecDnPtyIy ZtkEXVDqL20IHeNMzQya6+nLcWDsmw1P9MoYcKdhqi/RrpCIAfY+gTE86zzGPBVc46+z 7t5gq+S6bPgdLo6ApOqsRDmKBKSY1ghMh477QOwd3bpzCRW8qUfQVQLNdWJjWX4jXhTA KyUW7BvZwNS2L31pR7OPlu4K3p7XoP4+VR/36u5jgqLUuuoBeWEQZiyzTemtbd7OfNDc xjrg== X-Gm-Message-State: AOAM533dtTznX8NubqfzNFLR5ykf5LKFy75pw1GixcS2O7VPKY7yKNXj ghYkTgCamP0G85jsDZjKLwo75TmBCX6lIw== X-Google-Smtp-Source: ABdhPJz2hrOHT5w4zQlircIV5YllEuS0UmVYqDAajSoYFeueYKv1QHTO04fzKgbuKGrMxPUWOa3+Mg== X-Received: by 2002:a05:6512:3987:: with SMTP id j7mr4476664lfu.280.1631287754075; Fri, 10 Sep 2021 08:29:14 -0700 (PDT) Received: from grain.localdomain ([5.18.253.97]) by smtp.gmail.com with ESMTPSA id q15sm394663lfg.277.2021.09.10.08.29.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Sep 2021 08:29:12 -0700 (PDT) Received: by grain.localdomain (Postfix, from userid 1000) id B96C35A001E; Fri, 10 Sep 2021 18:29:11 +0300 (MSK) To: tml Date: Fri, 10 Sep 2021 18:29:04 +0300 Message-Id: <20210910152910.607398-1-gorcunov@gmail.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v14 0/6] qsync: implement packets filtering X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Cyrill Gorcunov via Tarantool-patches Reply-To: Cyrill Gorcunov Cc: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Guys, take a look please, once time permit. The questionable moments: - use filter disabling procedure for join/recovery: we make it so since snapshot has promote record which fills initial limbo state - need more tests to cover all possible scenarios - I keep filter_confirm_rollback() as is but rereading Vlad's comment > > 9. What if rollback is for LSN > limbo's last LSN? It > also means nothing to do. The same for confirm LSN < limbo's > first LSN. > I presume I need to traverse limbo and test if incoming LSN is present inside current queue. Anyway I send this version early to gather more comments, I hope not that much left to implement to be ready for merging. previous series https://lists.tarantool.org/tarantool-patches/20210804190752.488147-1-gorcunov@gmail.com/ branch gorcunov/gh-6036-rollback-confirm-14 issue https://github.com/tarantool/tarantool/issues/6036 v6: - use txn_limbo_terms name for structure - rebase on fresh sp/gh-6034-empty-limbo-transition branch - rework filtering chains v8: - add ability to disable filtering for local recovery and join stages - update tests v9: - opencode terms tracking - fix tests to use wait function since log output might be deferred by OS v10: - rework FILTER_IN and FILTER_PROMOTE chains with more detailed packets inspection - preserve old naming for terms manipulations - require the packet's replica_id to match limbo owner_id all the time v11-13: internal v14: - use straightforward packet inspection by their type without more general type routing - tried to hide locking api inside limbo level but since journal completion is called from inside of sched fiber the lock owner get migrated which cause error thus leave explicit locking instead - added updating of limbo::confirmed_lsn since we need it for proper validation - added new error code to distinguish filter errors from anything else - use say_error instead of say_info - keep disabling of filtration inside initial join/recovery because we're filling initial limbo state Cyrill Gorcunov (6): qsync: track confirmed lsn number on reads qsync: update confirmed lsn on initial promote request latch: add latch_is_locked helper qsync: order access to the limbo terms qsync: filter incoming synchro requests test: add replication/gh-6036-rollback-confirm .../gh-6036-qsync-filter-packets.md | 9 + src/box/applier.cc | 26 +- src/box/box.cc | 30 +- src/box/errcode.h | 1 + src/box/memtx_engine.cc | 3 +- src/box/txn_limbo.c | 337 +++++++++++++++--- src/box/txn_limbo.h | 85 ++++- src/lib/core/latch.h | 11 + test/box/error.result | 1 + test/replication/gh-6036-master.lua | 1 + test/replication/gh-6036-node.lua | 33 ++ test/replication/gh-6036-replica.lua | 1 + .../gh-6036-rollback-confirm.result | 180 ++++++++++ .../gh-6036-rollback-confirm.test.lua | 92 +++++ 14 files changed, 747 insertions(+), 63 deletions(-) create mode 100644 changelogs/unreleased/gh-6036-qsync-filter-packets.md create mode 120000 test/replication/gh-6036-master.lua create mode 100644 test/replication/gh-6036-node.lua create mode 120000 test/replication/gh-6036-replica.lua create mode 100644 test/replication/gh-6036-rollback-confirm.result create mode 100644 test/replication/gh-6036-rollback-confirm.test.lua base-commit: b0431cf8f47e9d081f6a402bc18edb1d6ad49847 -- 2.31.1