Re: [Tarantool-patches] [PATCH v3 0/4] replication: fix applying of rows originating from local instance

[Konstantin Osipov <kostja.osipov@gmail.com>]

* Georgy Kirichenko <kirichenkoga@gmail.com> [20/02/23 12:21]:

> Please do not think you are the only person who knows about byzantine faults. 
> Also there is little relevance between byzantine faults and my suggestion to 
> enforce replica-side checking.

You've been suggesting that filtering on the master is safer. I
pointed out it's not, there is no way to guarantee
(even in theory) correctness/safety if replica if master is
malfunctioning.

I merely pointed out that your safety argument has no merit. 

There are no other practical advantages of filtering on replica
either: there is a disadvantage, more traffic and more filtering work to do 
inside tx thread (as opposed to relay/wal thread if done on
master).

It is also against the current responsibilities of IPROTO_SUBSCRIBE: the
concept of a subscription is that replica specifies what it is 
interested in. Specifically, it specifies vclock components it's.
You suggest to make the replica responsible for
submitting its vclock, but the master decide what to do with it -
this splits the decision making logic between the two, making the
whole thing harder to understand. 

IPROTO_SUBSCRIBE responsibility layout today is typical for a
request-response protocol: the master, being the server, executes
the command as specified by the client (the replica), and the
replica runs the logic to decide what command to issue.

You suggest to change it because of some theoretical concerns you
have. 

> In any case filtering on the master side is the most worst  thing we could do. 
> In this case master has only one peer and have no chance to make a proper 
> decision if replica is broken. And we have no chance to know about it (except 
> assert which are excluded from release builds, or panic messages). For 
> instance if master skipped some rows then there are no any tracks of the 
> situation we could detect.

The situation is symmetrical. Both peers do not have the whole
picture. You can make either of the peers responsible for the
decision, then the other peer will need to supply the missing
bits. There is no way you can make it safer by changing who makes
the decision, but you can certainly make it more messed up by
splitting this logic or going against an established layout.

If you have a specific example why things will improve if done
otherwise - in the number of packets, or traffic, or some other
measurable way, you should point it out. 

> In the opposite case a replica could connect to as many masters as they need 
> to filter out all invalid data or hacked masters. At least we could enforce 
> replication stream meta checking.

I do not think the scope of this issue has ever been protecting
against hacked masters. It has never been a goal of the protocol
either. 

> Two major point I would like to mention are:
> 1. Replica could consistently follow all vclock members and apply all 
> transactions without gaps (I already got rid of them, I hope you remember)
> 2. Replica could protect itself against concurrent local writes (one was made 
> locally, the second one is returned from master)

This was added for specific reasons. There is no known reason the
master should send unnecessary data to replica or replica fast
path should get slower.

-- 
Konstantin Osipov, Moscow, Russia
https://scylladb.com