From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 1F9DB469719 for ; Mon, 24 Feb 2020 13:18:56 +0300 (MSK) Received: by mail-lj1-f195.google.com with SMTP id d10so9415800ljl.9 for ; Mon, 24 Feb 2020 02:18:55 -0800 (PST) Date: Mon, 24 Feb 2020 13:18:48 +0300 From: Konstantin Osipov Message-ID: <20200224101848.GE18378@atlas> References: <2175956.ElGaqSPkdT@localhost> <20200222204930.GA23200@atlas> <9655697.nUPlyArG6x@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9655697.nUPlyArG6x@localhost> Subject: Re: [Tarantool-patches] [PATCH v3 0/4] replication: fix applying of rows originating from local instance List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Georgy Kirichenko Cc: tarantool-patches@dev.tarantool.org, v.shpilevoy@tarantool.org * Georgy Kirichenko [20/02/23 12:21]: > Please do not think you are the only person who knows about byzantine faults. > Also there is little relevance between byzantine faults and my suggestion to > enforce replica-side checking. You've been suggesting that filtering on the master is safer. I pointed out it's not, there is no way to guarantee (even in theory) correctness/safety if replica if master is malfunctioning. I merely pointed out that your safety argument has no merit. There are no other practical advantages of filtering on replica either: there is a disadvantage, more traffic and more filtering work to do inside tx thread (as opposed to relay/wal thread if done on master). It is also against the current responsibilities of IPROTO_SUBSCRIBE: the concept of a subscription is that replica specifies what it is interested in. Specifically, it specifies vclock components it's. You suggest to make the replica responsible for submitting its vclock, but the master decide what to do with it - this splits the decision making logic between the two, making the whole thing harder to understand. IPROTO_SUBSCRIBE responsibility layout today is typical for a request-response protocol: the master, being the server, executes the command as specified by the client (the replica), and the replica runs the logic to decide what command to issue. You suggest to change it because of some theoretical concerns you have. > In any case filtering on the master side is the most worst thing we could do. > In this case master has only one peer and have no chance to make a proper > decision if replica is broken. And we have no chance to know about it (except > assert which are excluded from release builds, or panic messages). For > instance if master skipped some rows then there are no any tracks of the > situation we could detect. The situation is symmetrical. Both peers do not have the whole picture. You can make either of the peers responsible for the decision, then the other peer will need to supply the missing bits. There is no way you can make it safer by changing who makes the decision, but you can certainly make it more messed up by splitting this logic or going against an established layout. If you have a specific example why things will improve if done otherwise - in the number of packets, or traffic, or some other measurable way, you should point it out. > In the opposite case a replica could connect to as many masters as they need > to filter out all invalid data or hacked masters. At least we could enforce > replication stream meta checking. I do not think the scope of this issue has ever been protecting against hacked masters. It has never been a goal of the protocol either. > Two major point I would like to mention are: > 1. Replica could consistently follow all vclock members and apply all > transactions without gaps (I already got rid of them, I hope you remember) > 2. Replica could protect itself against concurrent local writes (one was made > locally, the second one is returned from master) This was added for specific reasons. There is no known reason the master should send unnecessary data to replica or replica fast path should get slower. -- Konstantin Osipov, Moscow, Russia https://scylladb.com