[Tarantool-patches] [RFC] Quorum-based synchronous replication

[Vladislav Shpilevoy]

Thanks for the discussion!

On 12/05/2020 19:47, Konstantin Osipov wrote:
> * Sergey Ostanevich <sergos at tarantool.org> [20/05/12 19:43]:
> 
>>> 1) It's unclear what happens here if async tx follows a sync tx.
>>>    Does it wait for the sync tx? This reduces availability for
>>
>> Definitely yes, unless we keep the 'dirty read' as it is at the moment
>> in memtx. This is the essence of the design, and it is temporary until 
>> the MVCC similar to the vinyl machinery appears. I intentionally didn't
>> include this big task into this RFC. 
>>
>> It will provide similar capabilities, although it will keep only
>> dependent transactions in the undo log. Also, it looks like it will fit
>> well into the machinery of this RFC. 
> 
> = reduced availability for all who have at least one sync space.
> 
> If different spaces have different quorum size = quorum size of
> the biggest group is effectively used for all spaces.
> 
> Replica-local transactions, e.g. those used by vinyl compaction, 
> are rolled back if there is no quorum.
> 
> What's the value of this?

There is an example when it leaves the database in an inconsistent
state, when half of a transaction is applied. I don't know why Sergey
didn't add it. I propose to him to extend the RFC with these examples.
Since you are not the first person, who finds this strange and wrong.
So clearly the RFC still does not explain this moment diligently
enough.

>>>    async txs - so it's hardly acceptable. Besides, with
>>>    group=local spaces, one can quickly run out of memory for undo.
>>>   
>>>
>>> 3) One can quickly run out of memory for undo. Any sync
>>>    transaction should be capped with a timeout to avoid OOMs. I
>>>    don't know how many times I should repeat it. The only good
>>>    solution for load control is in-memory WAL, which will allow to
>>>    rollback all transactions as soon as network partitioning is
>>>    detected.
>>
>> How in-memry WAL can help save on _undo_ memory? 
>> To rollback whatever amount of transactions one need to store the undo. 
> 
> I wrote earlier that it works as a natural failure detector and
> throttling mechanism. If
> there is no quorum, we can see it immediately by looking at the
> number of active subscribers of the in-memory WAL, so do not
> accumulate undo.

Here we go again ...

Talking of throttling. Without in-memory WAL no need for throttling. All is
'slow' by design already, as you think.

Talking of failure detection - what??? I don't get it. This is something new.
With in-memory relay or without you anyway can see if there is a quorum.
This is a matter of API of replication and transaction modules, and their
interaction with each other, solved by txn_limbo in my branch.

But still, I don't see how knowing number of subscribers helps with the
quorum. Subscriber presence does not add to quorums by itself. Anyway every
transaction needs to be replicated before you can say that its quorum got
+1 replica ack.