From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id E9B166FF9D; Wed, 24 Mar 2021 15:24:21 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E9B166FF9D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1616588662; bh=Hztd/pjI8HiZt9wfDmYCeaKDGyOvUpyZVd+m3OrsND0=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=CNzq9l38Upalv22Dq7fq84v9Y/yZXEigYnOXziA0PHRiZgRj8YWo/Bxej8JtV6/Yr E5eG3LaiXsO2bbw5CGI3Mgc1q1fV1kuroDhM9pQTnoGRbC5h+P8NbrFo8gm6K17HNr 2KRKB/jVqivGn5YxzLvDVzqAgjSeiHCBbWCs0wF4= Received: from smtp36.i.mail.ru (smtp36.i.mail.ru [94.100.177.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 6E2116FC86 for ; Wed, 24 Mar 2021 15:24:20 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6E2116FC86 Received: by smtp36.i.mail.ru with esmtpa (envelope-from ) id 1lP2Yd-0004oK-Ce; Wed, 24 Mar 2021 15:24:19 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Wed, 24 Mar 2021 15:24:10 +0300 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD95D6E7CC48CB1F5F1C82687294EF6886B62CC997B0B8C95C4182A05F538085040EBC5AEB2784DBFF2A895363D9F23042035DBAA3C90C56262DC87EDEE492DBEC5 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7696D01AF671C4816EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637DC39B53F0A60839C8638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C5DD32608FC869F5D65ED67F92ABEF120442422ACC4A4C304A471835C12D1D9774AD6D5ED66289B5278DA827A17800CE749E2213E709ACCBA9FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C397DCCBFEAAA0BC6A117882F4460429728AD0CFFFB425014E868A13BD56FB6657D81D268191BDAD3DC09775C1D3CA48CFE21A25664D805A25BA3038C0950A5D36C8A9BA7A39EFB766EC990983EF5C0329BA3038C0950A5D36D5E8D9A59859A8B65BFCCF07C76FFC1C76E601842F6C81A1F004C906525384307823802FF610243DF43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B3BBE47FD9DD3FB59A8DF7F3B2552694A2BEBFE083D3B9BA73A03B725D353964BB11811A4A51E3B096D1867E19FE14079BA9C0B312567BB23089D37D7C0E48F6CA18204E546F3947CC0F9454058DFE53C262FEC7FBD7D1F5BC8A9BA7A39EFB7666BA297DBC24807EA089D37D7C0E48F6C8AA50765F79006377AD8C0A31FFD7ED5EFF80C71ABB335746BA297DBC24807EA27F269C8F02392CD20465B3A5AADEC6827F269C8F02392CD5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8183A4AFAF3EA6BDC446469D8A8717206BB5DD32608FC869F5D65ED67F92ABEF120442422ACC4A4C3049C2B6934AE262D3EE7EAB7254005DCED3BFC4185F4F65B9792E1F3B950BCE216D137639923C070ABBDBA34C8DF405E9A04EBA3D8E7E5B87ABF8C51168CD8EBDB55D5BE2F85BDEC5FDC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D345DE7771146E56B08428F65F252039356EF2A73A4F637011613B892D6889DE796F49029C8B850BDBC1D7E09C32AA3244CB32C46FBCC9C1CE9D3C948792AA78E55A995755A1445935E927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXjhXhGERY7yL5MdV0K+N3vA X-Mailru-Sender: 583F1D7ACE8F49BDD2846D59FC20E9F86C523D0B3819F67E43B58F4BD31422109A00A67726A1C997424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH v2 0/7] applier: handle synchronous transactions during final X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Initially this patches idea was to ignore synchronous rows on applier side and make it so that there are no rolled back transactions in final join stream on master side. Unfortunately, this easy fix didn't work. The reason for it is that once the replica receives initial data, up to `start_vclock`, the final join stream has to start right at `start_vclock` so that we do not lose any transactions. This means that once master encounters a synchro rollback and makes replica retry final join to get rid of the rollback, it still has to send it together with other data. And this rollback must be processed by applier to avoid conflicts. In order to let applier process synchro requests (CONFIRM and ROLLBACK) we need to make final join transactional, obviously. This is what this patchset does. An alternative would be to retry not only final, but also initial join every time master receives a rollback during final join stage. This would be too violent due to possibly huge data amounts being sent during initial join. Changes in v2: - Make applier transactional on final join stage - Remove guards for rollback during final join on master side - Some refactoring in preparation to #5874 https://github.com/tarantool/tarantool/issues/5566 https://github.com/tarantool/tarantool/tree/sp/gh-5566-final-join-synchro-v2 Serge Petrenko (7): replication: fix a hang on final join retry applier: extract tx boundary checks from applier_read_tx into a separate routine applier: extract plain tx application from applier_apply_tx() applier: remove excess last_row_time update from subscribe loop applier: make final join transactional replication: tolerate synchro rollback during final join replication: do not ignore replica vclock on register changelogs/unreleased/synchro-final-join.md | 4 + src/box/applier.cc | 461 +++++++++--------- src/box/box.cc | 38 +- src/box/relay.cc | 1 + .../gh-5566-final-join-synchro.result | 139 ++++++ .../gh-5566-final-join-synchro.test.lua | 61 +++ test/replication/suite.cfg | 1 + 7 files changed, 449 insertions(+), 256 deletions(-) create mode 100644 changelogs/unreleased/synchro-final-join.md create mode 100644 test/replication/gh-5566-final-join-synchro.result create mode 100644 test/replication/gh-5566-final-join-synchro.test.lua -- 2.24.3 (Apple Git-128)