[Tarantool-patches] [RFC] Quorum-based synchronous replication

[Vladislav Shpilevoy]

Hi! Thanks for the updates!

> ### Connection liveness
> 
> There is a timeout-based mechanism in Tarantool that controls the
> asynchronous replication, which uses the following config:
> ```
> * replication_connect_timeout  = 4
> * replication_sync_lag         = 10
> * replication_sync_timeout     = 300
> * replication_timeout          = 1
> ```
> For backward compatibility and to differentiate the async replication
> we should augment the configuration with the following:
> ```
> * synchro_replication_heartbeat = 4

Heartbeats are already being sent. I don't see any sense in adding a
second heartbeat option.

> * synchro_replication_quorum_timeout = 4

Since this is a replication option, it should start from replication_
prefix.

> ```
> Leader should send a heartbeat every synchro_replication_heartbeat if
> there were no messages sent. Replicas should respond to the heartbeat
> just the same way as they do it now. As soon as Leader has no response
> for another heartbeat interval, it should consider the replica is lost.

All of that is already done in the regular heartbeats, not related nor
bound to any synchronous activities. Just like failure detection should be.

> As soon as leader appears in a situation it has not enough replicas
> to achieve quorum, it should stop accepting write requests. There's an
> option for leader to rollback to the latest transaction that has quorum:
> leader issues a 'rollback' message referring to the [LEADER_ID, LSN]
> where LSN is of the first transaction in the leader's undo log.

What is that option?

> The rollback message replicated to the available cluster will put it in a
> consistent state. After that configuration of the cluster can be
> updated to a new available quorum and leader can be switched back to
> write mode.
> 
> During the quorum collection it can happen that some of replicas become
> unavailable due to some reason, so leader should wait at most for
> synchro_replication_quorum_timeout after which it issues a Rollback
> pointing to the oldest TXN in the waiting list.