* [Tarantool-patches] [RFC] Quorum-based synchronous replication
@ 2020-04-03 21:08 Sergey Ostanevich
2020-04-07 13:02 ` Aleksandr Lyapunov
` (3 more replies)
0 siblings, 4 replies; 53+ messages in thread
From: Sergey Ostanevich @ 2020-04-03 21:08 UTC (permalink / raw)
To: tarantool-patches
* **Status**: In progress
* **Start date**: 31-03-2020
* **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\>
* **Issues**:
## Summary
The aim of this RFC is to address the following list of problems
formulated at MRG planning meeting:
- protocol backward compatibility to enable cluster upgrade w/o
downtime
- consistency of data on replica and leader
- switch from leader to replica without data loss
- up to date replicas to run read-only requests
- ability to switch async replicas into sync ones
- guarantee of rollback on leader and sync replicas
- simplicity of cluster orchestration
What this RFC is not:
- high availability (HA) solution with automated failover, roles
assignments an so on
- master-master configuration support
## Background and motivation
There are number of known implemenatation of consistent data presence in
a cluster. They can be commonly named as "wait for LSN" technique. The
biggest issue with this technique is the abscence of rollback gauarantees
at replica in case of transaction failure on one master or some of the
replics in the cluster.
To provide such capabilities a new functionality should be introduced in
Tarantool core, with limitation mentioned before - backward compatilibity
and ease of cluster orchestration.
## Detailed design
### Quorum commit
The main idea behind the proposal is to reuse existent machinery as much
as possible. It will ensure the well-tested and proven functionality
across many instances in MRG and beyond is used. The transaction rollback
mechanism is in place and works for WAL write failure. If we substitute
the WAL success with a new situation which is named 'quorum' later in
this document then no changes to the machinery is needed. The same is
true for snapshot machinery that allows to create a copy of the database
in memory for the whole period of snapshot file write. Adding quorum here
also minimizes changes.
Currently replication represented by the following scheme:
```
Customer Leader WAL(L) Replica WAL(R)
|------TXN----->| | | |
| | | | |
| [TXN Rollback | | |
| created] | | |
| | | | |
| |-----TXN----->| | |
| | | | |
| |<---WAL Ok----| | |
| | | | |
| [TXN Rollback | | |
| destroyed] | | |
| | | | |
|<----TXN Ok----| | | |
| |-------Replicate TXN------->| |
| | | | |
| | | [TXN Rollback |
| | | created] |
| | | | |
| | | |-----TXN----->|
| | | | |
| | | |<---WAL Ok----|
| | | | |
| | | [TXN Rollback |
| | | destroyed] |
| | | | |
```
To introduce the 'quorum' we have to receive confirmation from replicas
to make a decision on whether the quorum is actually present. Leader
collects necessary amount of replicas confirmation plus its own WAL
success. This state is named 'quorum' and gives leader the right to
complete the customers' request. So the picture will change to:
```
Customer Leader WAL(L) Replica WAL(R)
|------TXN----->| | | |
| | | | |
| [TXN Rollback | | |
| created] | | |
| | | | |
| |-----TXN----->| | |
| | | | |
| |-------Replicate TXN------->| |
| | | | |
| | | [TXN Rollback |
| |<---WAL Ok----| created] |
| | | | |
| [Waiting | |-----TXN----->|
| of a quorum] | | |
| | | |<---WAL Ok----|
| | | | |
| |<------Replication Ok-------| |
| | | | |
| [Quorum | | |
| achieved] | | |
| | | | |
| [TXN Rollback | | |
| destroyed] | | |
| | | | |
| |----Quorum--->| | |
| | | | |
| |-----------Quorum---------->| |
| | | | |
|<---TXN Ok-----| | [TXN Rollback |
| | | destroyed] |
| | | | |
| | | |----Quorum--->|
| | | | |
```
The quorum should be collected as a table for a list of transactions
waiting for quorum. The latest transaction that collects the quorum is
considered as complete, as well as all transactions prior to it, since
all transactions should be applied in order. Leader writes a 'quorum'
message to the WAL and it is delivered to Replicas.
Replica should report a positive or a negative result of the TXN to the
Leader via the IPROTO explicitly to allow Leader to collect the quorum
or anti-quorum for the TXN. In case negative result for the TXN received
from minor number of Replicas, then Leader has to send an error message
to each Replica, which in turn has to disconnect from the replication
the same way as it is done now in case of conflict.
In case Leader receives enough error messages to do not achieve the
quorum it should write the 'rollback' message in the WAL. After that
Leader and Replicas will perform the rollback for all TXN that didn't
receive quorum.
### Recovery and failover.
Tarantool instance during reading WAL should postpone the commit until
the quorum is read. In case the WAL eof is achieved, the instance should
keep rollback for all transactions that are waiting for a quorum entry
until the role of the instance is set. In case this instance become a
Replica there are no additional actions needed, sine all info about
quorum/rollback will arrive via replication. In case this instance is
assigned a Leader role, it should write 'rollback' in its WAL and
perform rollback for all transactions waiting for a quorum.
In case of a Leader failure a Replica with the biggest LSN with former
leader's ID is elected as a new leader. The replica should record
'rollback' in its WAL which effectively means that all transactions
without quorum should be rolled back. This rollback will be delivered to
all replicas and they will perform rollbacks of all transactions waiting
for quorum.
### Snapshot generation.
We also can reuse current machinery of snapshot generation. Upon
receiving a request to create a snapshot an instance should request a
readview for the current commit operation. Although start of the
snapshot generation should be postponed until this commit operation
receives its quorum. In case operation is rolled back, the snapshot
generation should be aborted and restarted using current transaction
after rollback is complete.
After snapshot is created the WAL should start from the first operation
that follows the commit operation snapshot is generated for. That means
WAL will contain a quorum message that refers to a transaction that is
not present in the WAL. Apparently, we have to allow this for the case
quorum refers to a transaction with LSN less than the first entry in the
WAL and only once.
### Asynchronous replication.
Along with synchronous Replicas the cluster can contain asynchronous
Replicas. That means async Replica doesn't reply to the Leader with
errors since they're not contributing into quorum. Still, async
Replicas have to follow the new WAL operation, such as keep rollback
info until 'quorum' message is received. This is essential for the case
of 'rollback' message appearance in the WAL. This message assumes
Replica is able to perform all necessary rollback by itself. Cluster
information should contain explicit notification of each Replica
operation mode.
### Synchronous replication enabling.
Synchronous operation can be required for a set of spaces in the data
scheme. That means only transactions that contain data modification for
these spaces should require quorum. Such transactions named synchronous.
As soon as last operation of synchronous transaction appeared in Leader's
WAL, it will cause all following transactions - matter if they are
synchronous or not - wait for the quorum. In case quorum is not achieved
the 'rollback' operation will cause rollback of all transactions after
the synchronous one. It will ensure the consistent state of the data both
on Leader and Replicas. In case user doesn't require synchronous operation
for any space then no changes to the WAL generation and replication will
appear.
Cluster description should contain explicit attribute for each Replica
to denote it participates in synchronous activities. Also the description
should contain criterion on how many Replicas responses are needed to
achieve the quorum.
## Rationale and alternatives
There is an implementation of synchronous replication as part of gh-980
activities, still it is not in a state to get into the product. More
than that it intentionally breaks backward compatibility which is a
prerequisite for this proposal.
^ permalink raw reply [flat|nested] 53+ messages in thread* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-03 21:08 [Tarantool-patches] [RFC] Quorum-based synchronous replication Sergey Ostanevich @ 2020-04-07 13:02 ` Aleksandr Lyapunov 2020-04-08 9:18 ` Sergey Ostanevich 2020-04-14 12:58 ` Sergey Bronnikov ` (2 subsequent siblings) 3 siblings, 1 reply; 53+ messages in thread From: Aleksandr Lyapunov @ 2020-04-07 13:02 UTC (permalink / raw) To: Sergey Ostanevich, tarantool-patches On 4/4/20 12:08 AM, Sergey Ostanevich wrote: > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: > > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support > > > ## Background and motivation > > There are number of known implemenatation of consistent data presence in > a cluster. They can be commonly named as "wait for LSN" technique. The > biggest issue with this technique is the abscence of rollback gauarantees > at replica in case of transaction failure on one master or some of the > replics in the cluster. > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with limitation mentioned before - backward compatilibity > and ease of cluster orchestration. > > ## Detailed design > > ### Quorum commit > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |----Quorum--->| | | > | | | | | > | |-----------Quorum---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |----Quorum--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'quorum' > message to the WAL and it is delivered to Replicas. I think we should cal the the message something like 'confirm' (not 'quorum'), and mention here that it has its own LSN. Besides, it's very similar to phase two of two-phase-commit, we'll need it later. > > Replica should report a positive or a negative result of the TXN to the > Leader via the IPROTO explicitly to allow Leader to collect the quorum > or anti-quorum for the TXN. In case negative result for the TXN received > from minor number of Replicas, then Leader has to send an error message > to each Replica, which in turn has to disconnect from the replication > the same way as it is done now in case of conflict. I'm sure that unconfirmed transactions must not be visible both on master and on replica since the could be aborted. We need read-committed. > > In case Leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > Leader and Replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the quorum is read. In case the WAL eof is achieved, the instance should > keep rollback for all transactions that are waiting for a quorum entry > until the role of the instance is set. In case this instance become a > Replica there are no additional actions needed, sine all info about > quorum/rollback will arrive via replication. In case this instance is > assigned a Leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a Leader failure a Replica with the biggest LSN with former > leader's ID is elected as a new leader. The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its quorum. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. There is no guarantee that the replica will ever receive 'confirm' ('quorum') message, for example when the master is dead forever. That means that in some cases we are unable to make a snapshot.. But if we make unconfirmed transactions invisible, the current read view will give us exactly what we need, but I have no idea how to handle WAL rotation ('restart') in this case. > > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain a quorum message that refers to a transaction that is > not present in the WAL. Apparently, we have to allow this for the case > quorum refers to a transaction with LSN less than the first entry in the > WAL and only once. Not 'only once', there could be several unconfirmed transactions and thus several 'confirm' messages. > > ### Asynchronous replication. > > Along with synchronous Replicas the cluster can contain asynchronous > Replicas. That means async Replica doesn't reply to the Leader with > errors since they're not contributing into quorum. Still, async > Replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > Replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each Replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in Leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on Leader and Replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each Replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many Replicas responses are needed to > achieve the quorum. > > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-07 13:02 ` Aleksandr Lyapunov @ 2020-04-08 9:18 ` Sergey Ostanevich 2020-04-08 14:05 ` Konstantin Osipov 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-08 9:18 UTC (permalink / raw) To: Aleksandr Lyapunov; +Cc: tarantool-patches Hi! Thanks for review! Latest version is availabe at https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md > > The quorum should be collected as a table for a list of transactions > > waiting for quorum. The latest transaction that collects the quorum is > > considered as complete, as well as all transactions prior to it, since > > all transactions should be applied in order. Leader writes a 'quorum' > > message to the WAL and it is delivered to Replicas. > I think we should cal the the message something like 'confirm' > (not 'quorum'), and mention here that it has its own LSN. I believe it was clear from the mention that it goes to WAL. Updated. The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's LSN and it has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. > Besides, it's very similar to phase two of two-phase-commit, > we'll need it later. We already discussed this, similarity is ended as soon as one quorum means confirmation of the whole bunch of transactions before it, not the one. > > Replica should report a positive or a negative result of the TXN to the > > Leader via the IPROTO explicitly to allow Leader to collect the quorum > > or anti-quorum for the TXN. In case negative result for the TXN received > > from minor number of Replicas, then Leader has to send an error message > > to each Replica, which in turn has to disconnect from the replication > > the same way as it is done now in case of conflict. > I'm sure that unconfirmed transactions must not be visible both > on master and on replica since the could be aborted. > We need read-committed. So far I don't envision any problems with read-committed after we enable transaction manager similar to vinyl. From the standpoint of replication the rollback message will cancel all transactions that are later than confirmed one. No matter if they are visible or not. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > > receiving a request to create a snapshot an instance should request a > > readview for the current commit operation. Although start of the > > snapshot generation should be postponed until this commit operation > > receives its quorum. In case operation is rolled back, the snapshot > > generation should be aborted and restarted using current transaction > > after rollback is complete. > There is no guarantee that the replica will ever receive 'confirm' > ('quorum') message, for example when the master is dead forever. > That means that in some cases we are unable to make a snapshot.. > But if we make unconfirmed transactions invisible, the current > read view will give us exactly what we need, but I have no idea > how to handle WAL rotation ('restart') in this case. Updated. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. > > After snapshot is created the WAL should start from the first operation > > that follows the commit operation snapshot is generated for. That means > > WAL will contain a quorum message that refers to a transaction that is > > not present in the WAL. Apparently, we have to allow this for the case > > quorum refers to a transaction with LSN less than the first entry in the > > WAL and only once. > Not 'only once', there could be several unconfirmed transactions > and thus several 'confirm' messages. Updated. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-08 9:18 ` Sergey Ostanevich @ 2020-04-08 14:05 ` Konstantin Osipov 2020-04-08 15:06 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-04-08 14:05 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches * Sergey Ostanevich <sergos@tarantool.org> [20/04/08 12:23]: One thing I continue not understanding is why settle on RFC now when in-memory wal is not in yet? There is an unpleasant risk of committing to something that turns out to not work out in the best possible way. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-08 14:05 ` Konstantin Osipov @ 2020-04-08 15:06 ` Sergey Ostanevich 0 siblings, 0 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-08 15:06 UTC (permalink / raw) To: Konstantin Osipov, Aleksandr Lyapunov, tarantool-patches Hi! Thanks for review! On 08 апр 17:05, Konstantin Osipov wrote: > * Sergey Ostanevich <sergos@tarantool.org> [20/04/08 12:23]: > > One thing I continue not understanding is why settle on RFC > now when in-memory wal is not in yet? Does this RFC depend on in-memory WAL after all? The formulation of principles in RFC neither rely on nor deny any optimizations of underlying infrastructure. I believe in-memory can be introduced indepenetly. Correct me, if I'm wrong. > There is an unpleasant risk of committing to something that turns > out to not work out in the best possible way. It is maxima of current MRG management: instead of perpetually inventing 'best possible' without clear roadmap - and not finish it - identify what's needed and perform to its timely appearance. Again, if you see some conflicts between RFC and any technologies being developed - name them, let's try to resolve them. Regards, Sergos ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-03 21:08 [Tarantool-patches] [RFC] Quorum-based synchronous replication Sergey Ostanevich 2020-04-07 13:02 ` Aleksandr Lyapunov @ 2020-04-14 12:58 ` Sergey Bronnikov 2020-04-14 14:43 ` Sergey Ostanevich 2020-04-20 23:32 ` Vladislav Shpilevoy 2020-04-23 21:38 ` Vladislav Shpilevoy 3 siblings, 1 reply; 53+ messages in thread From: Sergey Bronnikov @ 2020-04-14 12:58 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches Hi, see 5 comments inline On 00:08 Sat 04 Apr , Sergey Ostanevich wrote: > > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: 1. Just for convenience, please add https://github.com/tarantool/tarantool/issues/4842 > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones 2. Ability to switch async replicas into sync ones and vice-versa? Or not? > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support > > > ## Background and motivation > > There are number of known implemenatation of consistent data presence in > a cluster. They can be commonly named as "wait for LSN" technique. The > biggest issue with this technique is the abscence of rollback gauarantees 3. typo: gauarantees -> guarantees > at replica in case of transaction failure on one master or some of the > replics in the cluster. 4. typo: replics -> replicas > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with limitation mentioned before - backward compatilibity > and ease of cluster orchestration. 5. but there is nothing mentioned before about these limitations. > ## Detailed design > > ### Quorum commit > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |----Quorum--->| | | > | | | | | > | |-----------Quorum---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |----Quorum--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'quorum' > message to the WAL and it is delivered to Replicas. > > Replica should report a positive or a negative result of the TXN to the > Leader via the IPROTO explicitly to allow Leader to collect the quorum > or anti-quorum for the TXN. In case negative result for the TXN received > from minor number of Replicas, then Leader has to send an error message > to each Replica, which in turn has to disconnect from the replication > the same way as it is done now in case of conflict. > > In case Leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > Leader and Replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the quorum is read. In case the WAL eof is achieved, the instance should > keep rollback for all transactions that are waiting for a quorum entry > until the role of the instance is set. In case this instance become a > Replica there are no additional actions needed, sine all info about > quorum/rollback will arrive via replication. In case this instance is > assigned a Leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a Leader failure a Replica with the biggest LSN with former > leader's ID is elected as a new leader. The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its quorum. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. > > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain a quorum message that refers to a transaction that is > not present in the WAL. Apparently, we have to allow this for the case > quorum refers to a transaction with LSN less than the first entry in the > WAL and only once. > > ### Asynchronous replication. > > Along with synchronous Replicas the cluster can contain asynchronous > Replicas. That means async Replica doesn't reply to the Leader with > errors since they're not contributing into quorum. Still, async > Replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > Replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each Replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in Leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on Leader and Replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each Replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many Replicas responses are needed to > achieve the quorum. > > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. -- sergeyb@ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-14 12:58 ` Sergey Bronnikov @ 2020-04-14 14:43 ` Sergey Ostanevich 2020-04-15 11:09 ` sergos 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-14 14:43 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: tarantool-patches Hi! Thanks for review! On 14 апр 15:58, Sergey Bronnikov wrote: > Hi, > > see 5 comments inline > > On 00:08 Sat 04 Apr , Sergey Ostanevich wrote: > > > > * **Status**: In progress > > * **Start date**: 31-03-2020 > > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > > * **Issues**: > > 1. Just for convenience, please add https://github.com/tarantool/tarantool/issues/4842 > Done. > > ## Summary > > > > The aim of this RFC is to address the following list of problems > > formulated at MRG planning meeting: > > - protocol backward compatibility to enable cluster upgrade w/o > > downtime > > - consistency of data on replica and leader > > - switch from leader to replica without data loss > > - up to date replicas to run read-only requests > > - ability to switch async replicas into sync ones > > 2. Ability to switch async replicas into sync ones and vice-versa? Or not? > Both ways, updated. > > - guarantee of rollback on leader and sync replicas > > - simplicity of cluster orchestration > > > > What this RFC is not: > > > > - high availability (HA) solution with automated failover, roles > > assignments an so on > > - master-master configuration support > > > > > > ## Background and motivation > > > > There are number of known implemenatation of consistent data presence in > > a cluster. They can be commonly named as "wait for LSN" technique. The > > biggest issue with this technique is the abscence of rollback gauarantees > > 3. typo: gauarantees -> guarantees > done > > at replica in case of transaction failure on one master or some of the > > replics in the cluster. > > 4. typo: replics -> replicas > > done > > To provide such capabilities a new functionality should be introduced in > > Tarantool core, with limitation mentioned before - backward compatilibity > > and ease of cluster orchestration. > > 5. but there is nothing mentioned before about these limitations. > They were named as problems to address, so I renamed them as requirements. [cut] Pushed updated version to the branch. Thanks, Sergos ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-14 14:43 ` Sergey Ostanevich @ 2020-04-15 11:09 ` sergos 2020-04-15 14:50 ` sergos 0 siblings, 1 reply; 53+ messages in thread From: sergos @ 2020-04-15 11:09 UTC (permalink / raw) To: tarantool-patches Cc: Николай Карлов, Тимур Сафин Hi! The latest version is below, also available at https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md --- * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is thecompatibility absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatilibity and ease of cluster orchestration. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN Rollback | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | [TXN Rollback | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's LSN and it has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a positive or a negative result of the TXN to the leader via the IPROTO explicitly to allow leader to collect the quorum or anti-quorum for the TXN. In case a negative result for the TXN is received from minor number of replicas, then leader has to send an error message to the replicas, which in turn have to disconnect from the replication the same way as it is done now in case of conflict. In case leader receives enough error messages to do not achieve the quorum it should write the 'rollback' message in the WAL. After that leader and replicas will perform the rollback for all TXN that didn't receive quorum. ### Recovery and failover. Tarantool instance during reading WAL should postpone the commit until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep rollback for all transactions that are waiting for a confirm entry until the role of the instance is set. In case this instance become a replica there are no additional actions needed, since all info about quorum/rollback will arrive via replication. In case this instance is assigned a leader role, it should write 'rollback' in its WAL and perform rollback for all transactions waiting for a quorum. In case of a leader failure a replica with the biggest LSN with former leader's ID is elected as a new leader. The replica should record 'rollback' in its WAL which effectively means that all transactions without quorum should be rolled back. This rollback will be delivered to all replicas and they will perform rollbacks of all transactions waiting for quorum. An interface to force apply pending transactions by issuing a confirm entry for them have to be introduced for manual recovery. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-15 11:09 ` sergos @ 2020-04-15 14:50 ` sergos 2020-04-16 7:13 ` Aleksandr Lyapunov ` (2 more replies) 0 siblings, 3 replies; 53+ messages in thread From: sergos @ 2020-04-15 14:50 UTC (permalink / raw) To: Николай Карлов, Тимур Сафин, Mons Anderson, Aleksandr Lyapunov, Sergey Bronnikov Cc: tarantool-patches Sorry for mess introduced by mail client in previous message. Here’s the correct version with 3 more misprints fixed. The version is available here https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md Please, reply all with your comments/blessings today. Regards, Sergos --- * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is the absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatibility and ease of cluster orchestration. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN Rollback | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN Rollback | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | [TXN Rollback | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | [TXN Rollback | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's LSN and it has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a positive or a negative result of the TXN to the leader via the IPROTO explicitly to allow leader to collect the quorum or anti-quorum for the TXN. In case a negative result for the TXN is received from minor number of replicas, then leader has to send an error message to the replicas, which in turn have to disconnect from the replication the same way as it is done now in case of conflict. In case leader receives enough error messages to do not achieve the quorum it should write the 'rollback' message in the WAL. After that leader and replicas will perform the rollback for all TXN that didn't receive quorum. ### Recovery and failover. Tarantool instance during reading WAL should postpone the commit until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep rollback for all transactions that are waiting for a confirm entry until the role of the instance is set. In case this instance become a replica there are no additional actions needed, since all info about quorum/rollback will arrive via replication. In case this instance is assigned a leader role, it should write 'rollback' in its WAL and perform rollback for all transactions waiting for a quorum. In case of a leader failure a replica with the biggest LSN with former leader's ID is elected as a new leader. The replica should record 'rollback' in its WAL which effectively means that all transactions without quorum should be rolled back. This rollback will be delivered to all replicas and they will perform rollbacks of all transactions waiting for quorum. An interface to force apply pending transactions by issuing a confirm entry for them have to be introduced for manual recovery. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-15 14:50 ` sergos @ 2020-04-16 7:13 ` Aleksandr Lyapunov 2020-04-17 10:10 ` Konstantin Osipov 2020-04-20 11:20 ` Serge Petrenko 2 siblings, 0 replies; 53+ messages in thread From: Aleksandr Lyapunov @ 2020-04-16 7:13 UTC (permalink / raw) To: sergos, Николай Карлов, Тимур Сафин, Mons Anderson, Sergey Bronnikov Cc: tarantool-patches lgtm On 4/15/20 5:50 PM, sergos@tarantool.org wrote: > Sorry for mess introduced by mail client in previous message. > Here’s the correct version with 3 more misprints fixed. > > The version is available here > https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md > > Please, reply all with your comments/blessings today. > > Regards, > Sergos > > --- > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: https://github.com/tarantool/tarantool/issues/4842 > > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones and vice versa > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support > > ## Background and motivation > > There are number of known implementation of consistent data presence in > a cluster. They can be commonly named as "wait for LSN" technique. The > biggest issue with this technique is the absence of rollback guarantees > at replica in case of transaction failure on one master or some of the > replicas in the cluster. > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with requirements mentioned before - backward > compatibility and ease of cluster orchestration. > > ## Detailed design > > ### Quorum commit > > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's LSN and it has its > own LSN. This confirm message is delivered to all replicas through the > existing replication mechanism. > > Replica should report a positive or a negative result of the TXN to the > leader via the IPROTO explicitly to allow leader to collect the quorum > or anti-quorum for the TXN. In case a negative result for the TXN is > received from minor number of replicas, then leader has to send an error > message to the replicas, which in turn have to disconnect from the > replication the same way as it is done now in case of conflict. > > In case leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > leader and replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the 'confirm' is read. In case the WAL eof is achieved, the instance > should keep rollback for all transactions that are waiting for a confirm > entry until the role of the instance is set. In case this instance > become a replica there are no additional actions needed, since all info > about quorum/rollback will arrive via replication. In case this instance > is assigned a leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a leader failure a replica with the biggest LSN with former > leader's ID is elected as a new leader. The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. > > An interface to force apply pending transactions by issuing a confirm > entry for them have to be introduced for manual recovery. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its confirmation. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. > > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain 'confirm' messages that refer to transactions that are > not present in the WAL. Apparently, we have to allow this for the case > 'confirm' refers to a transaction with LSN less than the first entry in > the WAL. > > In case master appears unavailable a replica still have to be able to > create a snapshot. Replica can perform rollback for all transactions that > are not confirmed and claim its LSN as the latest confirmed txn. Then it > can create a snapshot in a regular way and start with blank xlog file. > All rolled back transactions will appear through the regular replication > in case master reappears later on. > > ### Asynchronous replication. > > Along with synchronous replicas the cluster can contain asynchronous > replicas. That means async replica doesn't reply to the leader with > errors since they're not contributing into quorum. Still, async > replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on leader and replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many replicas responses are needed to > achieve the quorum. > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. > > > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-15 14:50 ` sergos 2020-04-16 7:13 ` Aleksandr Lyapunov @ 2020-04-17 10:10 ` Konstantin Osipov 2020-04-17 13:45 ` Sergey Ostanevich 2020-04-20 11:20 ` Serge Petrenko 2 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-04-17 10:10 UTC (permalink / raw) To: sergos Cc: Николай Карлов, Mons Anderson, tarantool-patches, Тимур Сафин * sergos@tarantool.org <sergos@tarantool.org> [20/04/15 17:51]: > ### Quorum commit This part looks correct. It only describes two paths out of many though: - leader is able to collect the majority - leader is not able to collect the majority What happens when a leader receives a message for a round which is complete? How does a replica which missed a round catch up? What happens if replica fails to apply txn 1 (e.g. because of a duplciate key), but confirms txn 2? What happens if txn1 gets no majority at the leader, but txn 2 gets a majority? How are the followers rolled back? > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's LSN and it has its > own LSN. This confirm message is delivered to all replicas through the > existing replication mechanism. > > Replica should report a positive or a negative result of the TXN to the > leader via the IPROTO explicitly to allow leader to collect the quorum > or anti-quorum for the TXN. In case a negative result for the TXN is > received from minor number of replicas, then leader has to send an error > message to the replicas, which in turn have to disconnect from the > replication the same way as it is done now in case of conflict. > > In case leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > leader and replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the 'confirm' is read. In case the WAL eof is achieved, the instance > should keep rollback for all transactions that are waiting for a confirm > entry until the role of the instance is set. In case this instance > become a replica there are no additional actions needed, since all info > about quorum/rollback will arrive via replication. In case this instance > is assigned a leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a leader failure a replica with the biggest LSN with former > leader's ID is elected as a new leader. As long as multi-master is not banned, there may be multiple leaders. Does this proposal suggest multi-master is banned? Then it should describe the implementation of this, and in absense of transparent query forwarding it will break all clients. > The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. > > An interface to force apply pending transactions by issuing a confirm > entry for them have to be introduced for manual recovery. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its confirmation. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. > > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain 'confirm' messages that refer to transactions that are > not present in the WAL. Apparently, we have to allow this for the case > 'confirm' refers to a transaction with LSN less than the first entry in > the WAL. > > In case master appears unavailable a replica still have to be able to > create a snapshot. Replica can perform rollback for all transactions that > are not confirmed and claim its LSN as the latest confirmed txn. Then it > can create a snapshot in a regular way and start with blank xlog file. > All rolled back transactions will appear through the regular replication > in case master reappears later on. > > ### Asynchronous replication. > > Along with synchronous replicas the cluster can contain asynchronous > replicas. That means async replica doesn't reply to the leader with > errors since they're not contributing into quorum. Still, async > replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on leader and replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many replicas responses are needed to > achieve the quorum. > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. > > -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-17 10:10 ` Konstantin Osipov @ 2020-04-17 13:45 ` Sergey Ostanevich 0 siblings, 0 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-17 13:45 UTC (permalink / raw) To: Konstantin Osipov, Николай Карлов, Тимур Сафин, Mons Anderson, Aleksandr Lyapunov, Sergey Bronnikov, tarantool-patches Hi, thanks for review! On 17 апр 13:10, Konstantin Osipov wrote: > * sergos@tarantool.org <sergos@tarantool.org> [20/04/15 17:51]: > > ### Quorum commit > > This part looks correct. It only describes two paths out of many > though: > - leader is able to collect the majority > - leader is not able to collect the majority > > What happens when a leader receives a message for a round which is > complete? It just ignores it, the reason - see next comment. > How does a replica which missed a round catch up? > What happens if replica fails to apply txn 1 (e.g. because of a > duplciate key), but confirms txn 2? This should never happen, since each replica applies txns in strict order, means failure of txn 1 will happen before the confirmation of txn 2. As soon as replica fails to apply a txn it should report an error, disconnect and roll back all txns in it's pipeline. After that the replica will ne in a consistent state with Leader's lsn before the txn 1. > > What happens if txn1 gets no majority at the leader, but txn 2 > gets a majority? How are the followers rolled back? This situation means that some of ACKs from replicas didn't arrive. Which doesn't mean they failed to apply txn 1. Althoug, success of txn 2 means the txn 1 was also applied - hence, receiveing a txn N ACK from a replica means ACK for each txn M: M < N. > > In case of a leader failure a replica with the biggest LSN with former > > leader's ID is elected as a new leader. > > As long as multi-master is not banned, there may be multiple > leaders. Does this proposal suggest multi-master is banned? Then > it should describe the implementation of this, and in absense of > transparent query forwarding it will break all clients. > It was mentioned at the top of RFC: > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support Which I tend to describe as 'do not recommend'. Similar to what we have in documentation about the cascading replication configuration. Although, I heard from some users that they successfuly use such config. Regards, Sergos ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-15 14:50 ` sergos 2020-04-16 7:13 ` Aleksandr Lyapunov 2020-04-17 10:10 ` Konstantin Osipov @ 2020-04-20 11:20 ` Serge Petrenko 2 siblings, 0 replies; 53+ messages in thread From: Serge Petrenko @ 2020-04-20 11:20 UTC (permalink / raw) To: Sergey Ostanevich Cc: Николай Карлов, Mons Anderson, tarantool-patches, Тимур Сафин LGTM. -- Serge Petrenko sergepetrenko@tarantool.org > 15 апр. 2020 г., в 17:50, sergos@tarantool.org написал(а): > > Sorry for mess introduced by mail client in previous message. > Here’s the correct version with 3 more misprints fixed. > > The version is available here > https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md > > Please, reply all with your comments/blessings today. > > Regards, > Sergos > > --- > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: https://github.com/tarantool/tarantool/issues/4842 > > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones and vice versa > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support > > ## Background and motivation > > There are number of known implementation of consistent data presence in > a cluster. They can be commonly named as "wait for LSN" technique. The > biggest issue with this technique is the absence of rollback guarantees > at replica in case of transaction failure on one master or some of the > replicas in the cluster. > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with requirements mentioned before - backward > compatibility and ease of cluster orchestration. > > ## Detailed design > > ### Quorum commit > > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's LSN and it has its > own LSN. This confirm message is delivered to all replicas through the > existing replication mechanism. > > Replica should report a positive or a negative result of the TXN to the > leader via the IPROTO explicitly to allow leader to collect the quorum > or anti-quorum for the TXN. In case a negative result for the TXN is > received from minor number of replicas, then leader has to send an error > message to the replicas, which in turn have to disconnect from the > replication the same way as it is done now in case of conflict. > > In case leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > leader and replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the 'confirm' is read. In case the WAL eof is achieved, the instance > should keep rollback for all transactions that are waiting for a confirm > entry until the role of the instance is set. In case this instance > become a replica there are no additional actions needed, since all info > about quorum/rollback will arrive via replication. In case this instance > is assigned a leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a leader failure a replica with the biggest LSN with former > leader's ID is elected as a new leader. The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. > > An interface to force apply pending transactions by issuing a confirm > entry for them have to be introduced for manual recovery. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its confirmation. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. > > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain 'confirm' messages that refer to transactions that are > not present in the WAL. Apparently, we have to allow this for the case > 'confirm' refers to a transaction with LSN less than the first entry in > the WAL. > > In case master appears unavailable a replica still have to be able to > create a snapshot. Replica can perform rollback for all transactions that > are not confirmed and claim its LSN as the latest confirmed txn. Then it > can create a snapshot in a regular way and start with blank xlog file. > All rolled back transactions will appear through the regular replication > in case master reappears later on. > > ### Asynchronous replication. > > Along with synchronous replicas the cluster can contain asynchronous > replicas. That means async replica doesn't reply to the leader with > errors since they're not contributing into quorum. Still, async > replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on leader and replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many replicas responses are needed to > achieve the quorum. > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. > > > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-03 21:08 [Tarantool-patches] [RFC] Quorum-based synchronous replication Sergey Ostanevich 2020-04-07 13:02 ` Aleksandr Lyapunov 2020-04-14 12:58 ` Sergey Bronnikov @ 2020-04-20 23:32 ` Vladislav Shpilevoy 2020-04-21 10:49 ` Sergey Ostanevich 2020-04-23 21:38 ` Vladislav Shpilevoy 3 siblings, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-04-20 23:32 UTC (permalink / raw) To: Sergey Ostanevich, tarantool-patches Hi! This is the latest version I found on the branch. I give my comments for it. Keep in mind I didn't read other reviews before writing my own, assuming that all questions were fixed, and by idea I should have understood everything after reading this now. Nonetheless see 12 comments below. > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: https://github.com/tarantool/tarantool/issues/4842 > > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones and vice versa > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on 1. So no leader election? That essentially makes single failure point for RW requests, is it correct? On the other hand I see section 'Recovery and failover.' below. And it seems to be automated, with selecting a replica with the biggest LSN. Where is the truth? > - master-master configuration support > > ## Background and motivation > > There are number of known implementation of consistent data presence in > a cluster. They can be commonly named as "wait for LSN" technique. The > biggest issue with this technique is the absence of rollback guarantees > at replica in case of transaction failure on one master or some of the > replicas in the cluster. > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with requirements mentioned before - backward > compatibility and ease of cluster orchestration. > > ## Detailed design > > ### Quorum commit > > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. 2. The problem here is that you create dependency on WAL. According to your words, replication is inside WAL, and if WAL gave ok, then all is replicated and applied. But that makes current code structure even worse than it is. Now WAL, GC, and replication code is spaghetti, basically. All depends on all. I was rather thinking, that we should fix that first. Not aggravate. WAL should provide API for writing to disk. Replication should not bother about WAL. GC should not bother about replication. All should be independent, and linked in one place by some kind of a manager, which would just use their APIs. I believe Cyrill G. would agree with me here, I remember him complaining about replication-wal-gc code inter-dependencies too. Please, request his review on this, if you didn't yet. > The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) 3. Are you saying 'leader' === 'master'? > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | 4. What is 'txn rollback', and why is it created before even a transaction is started? At least, rollback is a verb. Maybe you meant 'undo log'? > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN Rollback | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL 5. Please, define 'necessary amount'? > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN Rollback | | | > | created] | | | > | | | | | > | |-----TXN----->| | | 6. Are we going to replicate transaction after user writes commit()? Or will we replicate it while it is in progress? So called 'presumed commit'. I remember I read some papers explaining how it significantly speeds up synchronous transactions. Probably that was a paper about 2-phase commit, can't remember already. But the idea is still applicable for the replication too. > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN Rollback | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN Rollback | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | [TXN Rollback | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's LSN and it has its > own LSN. This confirm message is delivered to all replicas through the > existing replication mechanism. > > Replica should report a positive or a negative result of the TXN to the > leader via the IPROTO explicitly to allow leader to collect the quorum > or anti-quorum for the TXN. In case a negative result for the TXN is > received from minor number of replicas, then leader has to send an error > message to the replicas, which in turn have to disconnect from the > replication the same way as it is done now in case of conflict. > In case leader receives enough error messages to do not achieve the > quorum it should write the 'rollback' message in the WAL. After that > leader and replicas will perform the rollback for all TXN that didn't > receive quorum. > > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the commit until > the 'confirm' is read. In case the WAL eof is achieved, the instance > should keep rollback for all transactions that are waiting for a confirm > entry until the role of the instance is set. In case this instance > become a replica there are no additional actions needed, since all info > about quorum/rollback will arrive via replication. In case this instance > is assigned a leader role, it should write 'rollback' in its WAL and > perform rollback for all transactions waiting for a quorum. > > In case of a leader failure a replica with the biggest LSN with former > leader's ID is elected as a new leader. The replica should record > 'rollback' in its WAL which effectively means that all transactions > without quorum should be rolled back. This rollback will be delivered to > all replicas and they will perform rollbacks of all transactions waiting > for quorum. 7. Please, elaborate leader election. It is not as trivial as just 'elect'. What if the replica with the biggest LSN is temporary not available, but it knows that it has the biggest LSN? Will it become a leader without asking other nodes? What will do the other nodes? Will they wait for the new leader node to become available? Do they have a timeout on that? Basically, it would be nice to see the split-brain problem description here, and its solution for us. How leader failure is detected? Do you rely on our heartbeat messages? Are you going to adapt SWIM for this? Raft has a dedicated subsystem for election, it is not that simple. It involves voting, randomized algorithms. Am I missing something obvious in this RFC, which makes the leader election much simpler specifically for Tarantool? > An interface to force apply pending transactions by issuing a confirm > entry for them have to be introduced for manual recovery. > > ### Snapshot generation. > > We also can reuse current machinery of snapshot generation. Upon > receiving a request to create a snapshot an instance should request a > readview for the current commit operation. Although start of the > snapshot generation should be postponed until this commit operation > receives its confirmation. In case operation is rolled back, the snapshot > generation should be aborted and restarted using current transaction > after rollback is complete. 8. This section highly depends on transaction manager for memtx. If you have a transaction manager, you always have a ready-to-use read-view of the latest committed data. At least this is my understanding. After all, the manager should provide transaction isolation. And it means, that all non-committed transactions are not visible. And for that we need a read-view. Therefore, it could be used to make a snapshot. > After snapshot is created the WAL should start from the first operation > that follows the commit operation snapshot is generated for. That means > WAL will contain 'confirm' messages that refer to transactions that are > not present in the WAL. Apparently, we have to allow this for the case > 'confirm' refers to a transaction with LSN less than the first entry in > the WAL. 9. I couldn't understand that. Why confirm is in WAL for data stored in the snap? I thought you said above, that snapshot should be done for all confirmed data. Besides, having confirm out of snap means the snap is not self-sufficient anymore. > In case master appears unavailable a replica still have to be able to > create a snapshot. Replica can perform rollback for all transactions that > are not confirmed and claim its LSN as the latest confirmed txn. Then it > can create a snapshot in a regular way and start with blank xlog file. > All rolled back transactions will appear through the regular replication > in case master reappears later on. 10. You should be able to make a snapshot without rollback. Read-views are available anyway. At least it is so in Vinyl, from what I remember. And this is going to be similar in memtx. > > ### Asynchronous replication. > > Along with synchronous replicas the cluster can contain asynchronous > replicas. That means async replica doesn't reply to the leader with > errors since they're not contributing into quorum. Still, async > replicas have to follow the new WAL operation, such as keep rollback > info until 'quorum' message is received. This is essential for the case > of 'rollback' message appearance in the WAL. This message assumes > replica is able to perform all necessary rollback by itself. Cluster > information should contain explicit notification of each replica > operation mode. > > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in leader's > WAL, it will cause all following transactions - matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on leader and replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. > > Cluster description should contain explicit attribute for each replica > to denote it participates in synchronous activities. Also the description > should contain criterion on how many replicas responses are needed to > achieve the quorum. 11. Aha, I see 'necessary amount' from above is a manually set value. Ok. > > ## Rationale and alternatives > > There is an implementation of synchronous replication as part of gh-980 > activities, still it is not in a state to get into the product. More > than that it intentionally breaks backward compatibility which is a > prerequisite for this proposal. 12. How are we going to deal with fsync()? Will it be forcefully enabled on sync replicas and the leader? ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-20 23:32 ` Vladislav Shpilevoy @ 2020-04-21 10:49 ` Sergey Ostanevich 2020-04-21 22:17 ` Vladislav Shpilevoy 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-21 10:49 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! Thanks for review! > > - high availability (HA) solution with automated failover, roles > > assignments an so on > > 1. So no leader election? That essentially makes single failure point > for RW requests, is it correct? > > On the other hand I see section 'Recovery and failover.' below. And > it seems to be automated, with selecting a replica with the biggest > LSN. Where is the truth? > The failover can be manual or implemented independnetnly. By no means this means we should not explain how this should be done according to the replication schema discussed. And yes, the SPOF is the leader of the cluster. This is expected and is Ok according to all MRG planning meeting participants. > > - master-master configuration support > > > > ## Background and motivation > > > > There are number of known implementation of consistent data presence in > > a cluster. They can be commonly named as "wait for LSN" technique. The > > biggest issue with this technique is the absence of rollback guarantees > > at replica in case of transaction failure on one master or some of the > > replicas in the cluster. > > > > To provide such capabilities a new functionality should be introduced in > > Tarantool core, with requirements mentioned before - backward > > compatibility and ease of cluster orchestration. > > > > ## Detailed design > > > > ### Quorum commit > > > > The main idea behind the proposal is to reuse existent machinery as much > > as possible. It will ensure the well-tested and proven functionality > > across many instances in MRG and beyond is used. The transaction rollback > > mechanism is in place and works for WAL write failure. If we substitute > > the WAL success with a new situation which is named 'quorum' later in > > this document then no changes to the machinery is needed. > > 2. The problem here is that you create dependency on WAL. According to > your words, replication is inside WAL, and if WAL gave ok, then all is 'Replication is inside WAL' - what do you mean by that? The replication in its current state works from WAL, although it's an exaggregation to say it is 'inside WAL'. Why it means a new dependency after that? > replicated and applied. But that makes current code structure even worse > than it is. Now WAL, GC, and replication code is spaghetti, basically. > All depends on all. I was rather thinking, that we should fix that first. > Not aggravate. > > WAL should provide API for writing to disk. Replication should not bother > about WAL. GC should not bother about replication. All should be independent, > and linked in one place by some kind of a manager, which would just use their So you want to introduce a single point that will translate all messages between all participants? I believe current state was introduced exactly to avoid this situation. Each participant can be subscribed for a particular trigger inside another participant and take it into account in its activities - at the right time for itself. > APIs. I believe Cyrill G. would agree with me here, I remember him > complaining about replication-wal-gc code inter-dependencies too. Please, > request his review on this, if you didn't yet. > I personally have the same problem trying to implement a trivial test, by just figuring out the layers and dependencies of participants. This is about poor documentation im my understanding, not poor design. > > The same is > > true for snapshot machinery that allows to create a copy of the database > > in memory for the whole period of snapshot file write. Adding quorum here > > also minimizes changes. > > Currently replication represented by the following scheme: > > ``` > > Customer Leader WAL(L) Replica WAL(R) > > 3. Are you saying 'leader' === 'master'? According to international polite naming. Mark Twain nowadays goes obscene with his 'Mars Tom'. > > > |------TXN----->| | | | > > | | | | | > > | [TXN Rollback | | | > > | created] | | | > > 4. What is 'txn rollback', and why is it created before even a transaction > is started? At least, rollback is a verb. Maybe you meant 'undo log'? No, rollback has both verb and noun meaning. Nevertheless, if you tripped over this - it should be fixed. > > > | | | | | > > | |-----TXN----->| | | > > | | | | | > > | |<---WAL Ok----| | | > > | | | | | > > | [TXN Rollback | | | > > | destroyed] | | | > > | | | | | > > |<----TXN Ok----| | | | > > | |-------Replicate TXN------->| | > > | | | | | > > | | | [TXN Rollback | > > | | | created] | > > | | | | | > > | | | |-----TXN----->| > > | | | | | > > | | | |<---WAL Ok----| > > | | | | | > > | | | [TXN Rollback | > > | | | destroyed] | > > | | | | | > > ``` > > > > To introduce the 'quorum' we have to receive confirmation from replicas > > to make a decision on whether the quorum is actually present. Leader > > collects necessary amount of replicas confirmation plus its own WAL > > 5. Please, define 'necessary amount'? Apparently, resolved with comment #11 > > > success. This state is named 'quorum' and gives leader the right to > > complete the customers' request. So the picture will change to: > > ``` > > Customer Leader WAL(L) Replica WAL(R) > > |------TXN----->| | | | > > | | | | | > > | [TXN Rollback | | | > > | created] | | | > > | | | | | > > | |-----TXN----->| | | > > 6. Are we going to replicate transaction after user writes commit()? Does your 'user' means customer from the picture? In such a case do you expect to have an interactive transaction? Definitely, we do not consider it here in any form, since replication happens only for the comlpete transaction. > Or will we replicate it while it is in progress? So called 'presumed > commit'. I remember I read some papers explaining how it significantly > speeds up synchronous transactions. Probably that was a paper about > 2-phase commit, can't remember already. But the idea is still applicable > for the replication too. This can be considered only after MVCC is introduced - currently running as a separate activity. Then we can replicate transaction 'in the fly' into a separate blob/readview/better_name. By now this means we will be too much interwoven to correctly rollback afterwards, at the time of quorum is failed. > > In case of a leader failure a replica with the biggest LSN with former > > leader's ID is elected as a new leader. The replica should record > > 'rollback' in its WAL which effectively means that all transactions > > without quorum should be rolled back. This rollback will be delivered to > > all replicas and they will perform rollbacks of all transactions waiting > > for quorum. > > 7. Please, elaborate leader election. It is not as trivial as just 'elect'. > What if the replica with the biggest LSN is temporary not available, but > it knows that it has the biggest LSN? Will it become a leader without > asking other nodes? What will do the other nodes? Will they wait for the > new leader node to become available? Do they have a timeout on that? > By now I do not plan any activities on HA - including automated failover and leader re-election. In case leader sees insufficient number of replicas to achieve quorum - it stops, reporting the problem to the external orchestrator. > Basically, it would be nice to see the split-brain problem description here, > and its solution for us. > I believe the split-brain is under orchestrator control either - we should provide API to switch leader in the cluster, so that when a former leader came back it will not get quorum for any txn it has, replying to customers with failure as a result. > How leader failure is detected? Do you rely on our heartbeat messages? > Are you going to adapt SWIM for this? > > Raft has a dedicated subsystem for election, it is not that simple. It > involves voting, randomized algorithms. Am I missing something obvious in > this RFC, which makes the leader election much simpler specifically for > Tarantool? > All of these I assume as HA features, when Tarantool can automate the failover and leader re-election. Out of the scope by now. > > An interface to force apply pending transactions by issuing a confirm > > entry for them have to be introduced for manual recovery. > > > > ### Snapshot generation. > > > > We also can reuse current machinery of snapshot generation. Upon > > receiving a request to create a snapshot an instance should request a > > readview for the current commit operation. Although start of the > > snapshot generation should be postponed until this commit operation > > receives its confirmation. In case operation is rolled back, the snapshot > > generation should be aborted and restarted using current transaction > > after rollback is complete. > > 8. This section highly depends on transaction manager for memtx. If you > have a transaction manager, you always have a ready-to-use read-view > of the latest committed data. At least this is my understanding. > > After all, the manager should provide transaction isolation. And it means, > that all non-committed transactions are not visible. And for that we need > a read-view. Therefore, it could be used to make a snapshot. > Currently there's no such manager for memtx. So I proposed this workaround with minimal impact on our current machinery. Alexander Lyapunov is working on the manager in parallel, he reviewed and blessed this RFC, so apparently there's no contradiction with his plans. > > After snapshot is created the WAL should start from the first operation > > that follows the commit operation snapshot is generated for. That means > > WAL will contain 'confirm' messages that refer to transactions that are > > not present in the WAL. Apparently, we have to allow this for the case > > 'confirm' refers to a transaction with LSN less than the first entry in > > the WAL. > > 9. I couldn't understand that. Why confirm is in WAL for data stored in > the snap? I thought you said above, that snapshot should be done for all > confirmed data. Besides, having confirm out of snap means the snap is > not self-sufficient anymore. > Snap waits for confirm message to start. During this wait the WAL keep growing. At the moment confirm arrived the snap will be created - say, for txn #10. The WAL will be started with lsn #11 and commit can be somewhere lsn #30. So, starting with this snap data appears consistent for lsn #10 - it is guaranteed by the wait of commit message. Then replay of WAL will come to a confirm message lsn #30 - referring to lsn #10 - that actually ignored, since it looks beyond the WAL start. There could be confirm messages for even earlier txns if wait takes sufficient time - all of them will refer to lsn beyond the WAL. And it is Ok. > > In case master appears unavailable a replica still have to be able to > > create a snapshot. Replica can perform rollback for all transactions that > > are not confirmed and claim its LSN as the latest confirmed txn. Then it > > can create a snapshot in a regular way and start with blank xlog file. > > All rolled back transactions will appear through the regular replication > > in case master reappears later on. > > 10. You should be able to make a snapshot without rollback. Read-views are > available anyway. At least it is so in Vinyl, from what I remember. And this > is going to be similar in memtx. > You have to make a snapshot for a consistent data state. Unless we have transaction manager in memtx - this is the way to do so. And as I mentioned, this is a different activity. > > Cluster description should contain explicit attribute for each replica > > to denote it participates in synchronous activities. Also the description > > should contain criterion on how many replicas responses are needed to > > achieve the quorum. > > 11. Aha, I see 'necessary amount' from above is a manually set value. Ok. > > > > > ## Rationale and alternatives > > > > There is an implementation of synchronous replication as part of gh-980 > > activities, still it is not in a state to get into the product. More > > than that it intentionally breaks backward compatibility which is a > > prerequisite for this proposal. > > 12. How are we going to deal with fsync()? Will it be forcefully enabled > on sync replicas and the leader? To my understanding - it's up to user. I was considering a cluster that has no WAL at all - relying on sychro replication and sufficient number of replicas. Everyone who I asked about it told me I'm nuts. To my great surprise Alexander Lyapunov brought exactly the same idea to discuss. All of these is for one resolution: I would keep it for user to decide. Obviously, to speed up the processing leader can disable wal completely, but to do so we have to re-work the relay to work from memory. Replicas can use WAL in a way user wants: 2 replicas with slow HDD should'n wait for fsync(), while super-fast Intel DCPMM one can enable it. Balancing is up to user. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-21 10:49 ` Sergey Ostanevich @ 2020-04-21 22:17 ` Vladislav Shpilevoy 2020-04-22 16:50 ` Sergey Ostanevich 2020-04-23 6:58 ` Konstantin Osipov 0 siblings, 2 replies; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-04-21 22:17 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches >>> - high availability (HA) solution with automated failover, roles >>> assignments an so on >> >> 1. So no leader election? That essentially makes single failure point >> for RW requests, is it correct? >> >> On the other hand I see section 'Recovery and failover.' below. And >> it seems to be automated, with selecting a replica with the biggest >> LSN. Where is the truth? >> > > The failover can be manual or implemented independnetnly. By no means > this means we should not explain how this should be done according to > the replication schema discussed. But it is not explained. You just said 'the biggest LSN owner is chosen'. This looks very good in theory and on the paper, but it is not that simple. If you are going to explain how it works. You said 'by no means we should not explain'. Talking of the election in scope of our replication schema, I don't see where it is discussed. Is there a separate RFC I am missing? I am asking exactly about that - where are pings, healthchecks, timeouts, voting on top of our replication schema? If you don't want to make the election a part of this RFC at all, then why is there a section, which literally says that the election is present and it is 'the biggest LSN owner is chosen'? In case the election is out of this for now, did you think about how a possible future leader election algorithm could be implemented on top of this sync replication? Just to be sure we are not ruining some things which will be necessary for auto election later on. > And yes, the SPOF is the leader of the cluster. This is expected and is > Ok according to all MRG planning meeting participants. That is not about MRG only, Tarantool is not a closed-source MRG-only DB. I am not against making it non-automated for now, but I want to be sure it will be possible to implement this as an enhancement. >>> - master-master configuration support >>> >>> ## Background and motivation >>> >>> There are number of known implementation of consistent data presence in >>> a cluster. They can be commonly named as "wait for LSN" technique. The >>> biggest issue with this technique is the absence of rollback guarantees >>> at replica in case of transaction failure on one master or some of the >>> replicas in the cluster. >>> >>> To provide such capabilities a new functionality should be introduced in >>> Tarantool core, with requirements mentioned before - backward >>> compatibility and ease of cluster orchestration. >>> >>> ## Detailed design >>> >>> ### Quorum commit >>> >>> The main idea behind the proposal is to reuse existent machinery as much >>> as possible. It will ensure the well-tested and proven functionality >>> across many instances in MRG and beyond is used. The transaction rollback >>> mechanism is in place and works for WAL write failure. If we substitute >>> the WAL success with a new situation which is named 'quorum' later in >>> this document then no changes to the machinery is needed. >> >> 2. The problem here is that you create dependency on WAL. According to >> your words, replication is inside WAL, and if WAL gave ok, then all is > > 'Replication is inside WAL' - what do you mean by that? The replication in > its current state works from WAL, although it's an exaggregation to say > it is 'inside WAL'. Why it means a new dependency after that? You said 'WAL success' is substituted with a new situation 'quorum'. It means, strictly interpreting your words, that 'wal_write()' function won't return 0 until quorum is collected. This is what I mean by moving replication into WAL subsystem. >> replicated and applied. But that makes current code structure even worse >> than it is. Now WAL, GC, and replication code is spaghetti, basically. >> All depends on all. I was rather thinking, that we should fix that first. >> Not aggravate. >> >> WAL should provide API for writing to disk. Replication should not bother >> about WAL. GC should not bother about replication. All should be independent, >> and linked in one place by some kind of a manager, which would just use their > > So you want to introduce a single point that will translate all messages > between all participants? Well, this is called 'cbus', we already have it. This is a separate headache, which no one understands except Georgy. However I was not talking about it. I am wrong, it should not be a single manager. But all the subsystems should be as independent as possible still. > I believe current state was introduced exactly > to avoid this situation. Each participant can be subscribed for a > particular trigger inside another participant and take it into account > in its activities - at the right time for itself. Whatever. I don't know the code good enough, so I am probably wrong somewhere here. But every time looking at these numerous triggers depending on each other and called at arbitrary moments was enough. I tried to fix these trigger-dependencies in scope of different tasks, but cleaning this code appears to be a huge task by itself. This can be done without obscure triggers called from arbitrary threads at atribtrary moments of time. That in fact was the main blocker for the in-memory WAL, when I tried to finish it after Georgy in January. We have fibers exactly to avoid triggers. To be able to write linear and simple code. The triggers can be replaced by dedicated fibers, and fiber condition variables can be used to wait for exact moments of time of needed events where functionality is event based. >> APIs. I believe Cyrill G. would agree with me here, I remember him >> complaining about replication-wal-gc code inter-dependencies too. Please, >> request his review on this, if you didn't yet. >> > I personally have the same problem trying to implement a trivial test, > by just figuring out the layers and dependencies of participants. This > is about poor documentation im my understanding, not poor design. There is almost more documentation than the code. Just look at the number of comments and level of their details. And still it does not help. So looks like just a bad API and dependency design. Does not matter how hard it is documented, it just becomes more interdepending and harder to wrap a mind around it. >>> In case of a leader failure a replica with the biggest LSN with former >>> leader's ID is elected as a new leader. The replica should record >>> 'rollback' in its WAL which effectively means that all transactions >>> without quorum should be rolled back. This rollback will be delivered to >>> all replicas and they will perform rollbacks of all transactions waiting >>> for quorum. >> >> 7. Please, elaborate leader election. It is not as trivial as just 'elect'. >> What if the replica with the biggest LSN is temporary not available, but >> it knows that it has the biggest LSN? Will it become a leader without >> asking other nodes? What will do the other nodes? Will they wait for the >> new leader node to become available? Do they have a timeout on that? >> > By now I do not plan any activities on HA - including automated failover > and leader re-election. In case leader sees insufficient number of > replicas to achieve quorum - it stops, reporting the problem to the > external orchestrator. From the text in this section it does not look like a not planned activity, but like an already made decision. It is not even in the 'Plans' section. You just said 'is elected'. By whom? How? If the election is not a part of the RFC, I would suggest moving this out, or into a separate section 'Plans' or something. Or reformulate this by saying like 'the cluster stops serving write requests until external intervention sets a new leader'. And 'it is *advised* to use the one with the biggest LSN in the old leader's vclock component'. Something like that. >> Basically, it would be nice to see the split-brain problem description here, >> and its solution for us. >> > I believe the split-brain is under orchestrator control either - we > should provide API to switch leader in the cluster, so that when a > former leader came back it will not get quorum for any txn it has, > replying to customers with failure as a result. Exactly. We should provide something for this from inside. But are there any details? How should that work? Should all the healthy replicas reject everything from the false-leader? Should the false-leader somehow realize, that it is not considered a leader anymore, and should stop itself? If we choose the former way, how a replica defines who is the true leader? For example, some replicas still may consider the old leader as a true master. If we choose the latter way, what is the algorithm of determining that we are not a leader anymore? >>> After snapshot is created the WAL should start from the first operation >>> that follows the commit operation snapshot is generated for. That means >>> WAL will contain 'confirm' messages that refer to transactions that are >>> not present in the WAL. Apparently, we have to allow this for the case >>> 'confirm' refers to a transaction with LSN less than the first entry in >>> the WAL. >> >> 9. I couldn't understand that. Why confirm is in WAL for data stored in >> the snap? I thought you said above, that snapshot should be done for all >> confirmed data. Besides, having confirm out of snap means the snap is >> not self-sufficient anymore. >> > Snap waits for confirm message to start. During this wait the WAL keep > growing. At the moment confirm arrived the snap will be created - say, > for txn #10. The WAL will be started with lsn #11 and commit can be > somewhere lsn #30. > So, starting with this snap data appears consistent for lsn #10 - it is > guaranteed by the wait of commit message. Then replay of WAL will come > to a confirm message lsn #30 - referring to lsn #10 - that actually > ignored, since it looks beyond the WAL start. There could be confirm > messages for even earlier txns if wait takes sufficient time - all of > them will refer to lsn beyond the WAL. And it is Ok. What is 'commit message'? I don't see it on the schema above. I see only confirms. So the problem is that some data may be written to WAL after we started committing our transactions going to the snap, but before we received a quorum. And we can't truncate the WAL by the quorum, because there is already newer data, which was not included into the snap. Because WAL is not stopped, it still accepts new transactions. Now I understand. Would be good to have this example in the RFC. >>> Cluster description should contain explicit attribute for each replica >>> to denote it participates in synchronous activities. Also the description >>> should contain criterion on how many replicas responses are needed to >>> achieve the quorum. >> >> 11. Aha, I see 'necessary amount' from above is a manually set value. Ok. >> >>> >>> ## Rationale and alternatives >>> >>> There is an implementation of synchronous replication as part of gh-980 >>> activities, still it is not in a state to get into the product. More >>> than that it intentionally breaks backward compatibility which is a >>> prerequisite for this proposal. >> >> 12. How are we going to deal with fsync()? Will it be forcefully enabled >> on sync replicas and the leader? > > To my understanding - it's up to user. I was considering a cluster that > has no WAL at all - relying on sychro replication and sufficient number > of replicas. Everyone who I asked about it told me I'm nuts. To my great > surprise Alexander Lyapunov brought exactly the same idea to discuss. I didn't see an RFC on that, and this can become easily possible, when in-memory relay is implemented. If it is implemented in a clean way. We just can turn off the disk backoff, and it will work from memory-only. > All of these is for one resolution: I would keep it for user to decide. > Obviously, to speed up the processing leader can disable wal completely, > but to do so we have to re-work the relay to work from memory. Replicas > can use WAL in a way user wants: 2 replicas with slow HDD should'n wait > for fsync(), while super-fast Intel DCPMM one can enable it. Balancing > is up to user. Possibility of omitting fsync means that it is possible, that all nodes write confirm, which is reported to the client, then the nodes restart, and the data is lost. I would say it somewhere. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-21 22:17 ` Vladislav Shpilevoy @ 2020-04-22 16:50 ` Sergey Ostanevich 2020-04-22 20:28 ` Vladislav Shpilevoy 2020-04-23 6:58 ` Konstantin Osipov 1 sibling, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-22 16:50 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! On 22 апр 00:17, Vladislav Shpilevoy wrote: > >>> - high availability (HA) solution with automated failover, roles > >>> assignments an so on > >> > >> 1. So no leader election? That essentially makes single failure point > >> for RW requests, is it correct? > >> > >> On the other hand I see section 'Recovery and failover.' below. And > >> it seems to be automated, with selecting a replica with the biggest > >> LSN. Where is the truth? > >> > > > > The failover can be manual or implemented independnetnly. By no means > > this means we should not explain how this should be done according to > > the replication schema discussed. > > But it is not explained. You just said 'the biggest LSN owner is chosen'. > This looks very good in theory and on the paper, but it is not that simple. > If you are going to explain how it works. You said 'by no means we should not > explain'. > I expect this to be a reference to what is currently implemented by Tarantool users in many ways. I think I have to rephrase that one 'can keep current election apporach using biggest LSN' since proposed solution does not change current semantics of WAL generation, just 'confirm' and 'rollback' operations that are regular entries in the WAL. > Talking of the election in scope of our replication schema, I don't see > where it is discussed. Is there a separate RFC I am missing? I am asking exactly > about that - where are pings, healthchecks, timeouts, voting on top of our > replication schema? If you don't want to make the election a part of this RFC > at all, then why is there a section, which literally says that the election > is present and it is 'the biggest LSN owner is chosen'? > > In case the election is out of this for now, did you think about how a possible > future leader election algorithm could be implemented on top of this sync > replication? Just to be sure we are not ruining some things which will be > necessary for auto election later on. > My answer will be the same - no changes to WAL from this point of view, just replicas has their respective undo logs and can rollback to the consistent state. > > And yes, the SPOF is the leader of the cluster. This is expected and is > > Ok according to all MRG planning meeting participants. > > That is not about MRG only, Tarantool is not a closed-source MRG-only DB. > I am not against making it non-automated for now, but I want to be sure it > will be possible to implement this as an enhancement. > Sure we want to - using the existing SWIM module for membership and to elaborate something close to RAFT, still it is not in our immediate plans. > >>> - master-master configuration support > >>> > >>> ## Background and motivation > >>> > >>> There are number of known implementation of consistent data presence in > >>> a cluster. They can be commonly named as "wait for LSN" technique. The > >>> biggest issue with this technique is the absence of rollback guarantees > >>> at replica in case of transaction failure on one master or some of the > >>> replicas in the cluster. > >>> > >>> To provide such capabilities a new functionality should be introduced in > >>> Tarantool core, with requirements mentioned before - backward > >>> compatibility and ease of cluster orchestration. > >>> > >>> ## Detailed design > >>> > >>> ### Quorum commit > >>> > >>> The main idea behind the proposal is to reuse existent machinery as much > >>> as possible. It will ensure the well-tested and proven functionality > >>> across many instances in MRG and beyond is used. The transaction rollback > >>> mechanism is in place and works for WAL write failure. If we substitute > >>> the WAL success with a new situation which is named 'quorum' later in > >>> this document then no changes to the machinery is needed. > >> > >> 2. The problem here is that you create dependency on WAL. According to > >> your words, replication is inside WAL, and if WAL gave ok, then all is > > > > 'Replication is inside WAL' - what do you mean by that? The replication in > > its current state works from WAL, although it's an exaggregation to say > > it is 'inside WAL'. Why it means a new dependency after that? > > You said 'WAL success' is substituted with a new situation 'quorum'. It > means, strictly interpreting your words, that 'wal_write()' function > won't return 0 until quorum is collected. > > This is what I mean by moving replication into WAL subsystem. > The wal_write() should report the result of WAL operation. It should not return quorum - the WAL result should be used along with quorum messages from replicas to claim txn is complete. This shouldn't be part of WAL. > >> replicated and applied. But that makes current code structure even worse > >> than it is. Now WAL, GC, and replication code is spaghetti, basically. > >> All depends on all. I was rather thinking, that we should fix that first. > >> Not aggravate. > >> > >> WAL should provide API for writing to disk. Replication should not bother > >> about WAL. GC should not bother about replication. All should be independent, > >> and linked in one place by some kind of a manager, which would just use their > > > > So you want to introduce a single point that will translate all messages > > between all participants? > > Well, this is called 'cbus', we already have it. This is a separate headache, > which no one understands except Georgy. However I was not talking about it. > I am wrong, it should not be a single manager. But all the subsystems should > be as independent as possible still. > > > I believe current state was introduced exactly > > to avoid this situation. Each participant can be subscribed for a > > particular trigger inside another participant and take it into account > > in its activities - at the right time for itself. > > Whatever. I don't know the code good enough, so I am probably wrong > somewhere here. But every time looking at these numerous triggers depending > on each other and called at arbitrary moments was enough. I tried to fix > these trigger-dependencies in scope of different tasks, but cleaning this > code appears to be a huge task by itself. > > This can be done without obscure triggers called from arbitrary threads at > atribtrary moments of time. That in fact was the main blocker for the in-memory > WAL, when I tried to finish it after Georgy in January. We have fibers exactly > to avoid triggers. To be able to write linear and simple code. The triggers > can be replaced by dedicated fibers, and fiber condition variables can be used > to wait for exact moments of time of needed events where functionality is > event based. > I totaly agree to you that it is a big task itself. I believe we won't introduce too much extra dependencies between the parties - just tweak some of them. So far I want to start an ativity - Cyrill Gorcunov supports me - to draw a mutual dependency map of all participants: their threads, fibers, triggers and how they are connected. I believe it will help us to prepare a first step to redesign the system - or make a thoughtful decision to keep it as is. > >> APIs. I believe Cyrill G. would agree with me here, I remember him > >> complaining about replication-wal-gc code inter-dependencies too. Please, > >> request his review on this, if you didn't yet. > >> > > I personally have the same problem trying to implement a trivial test, > > by just figuring out the layers and dependencies of participants. This > > is about poor documentation im my understanding, not poor design. > > There is almost more documentation than the code. Just look at the number > of comments and level of their details. And still it does not help. So > looks like just a bad API and dependency design. Does not matter how hard > it is documented, it just becomes more interdepending and harder to wrap a > mind around it. > You can document every line of code with explanation of what it does, but there' no such line that will require to draw the 'big picture'. I believe it is the problem. Design is not the code, rather guideline for the code. To decypher it back from (perhaps not-so good sometimes) code is a big task itself. This should help to understand - and only then improve - the implementation. > >>> In case of a leader failure a replica with the biggest LSN with former > >>> leader's ID is elected as a new leader. The replica should record > >>> 'rollback' in its WAL which effectively means that all transactions > >>> without quorum should be rolled back. This rollback will be delivered to > >>> all replicas and they will perform rollbacks of all transactions waiting > >>> for quorum. > >> > >> 7. Please, elaborate leader election. It is not as trivial as just 'elect'. > >> What if the replica with the biggest LSN is temporary not available, but > >> it knows that it has the biggest LSN? Will it become a leader without > >> asking other nodes? What will do the other nodes? Will they wait for the > >> new leader node to become available? Do they have a timeout on that? > >> > > By now I do not plan any activities on HA - including automated failover > > and leader re-election. In case leader sees insufficient number of > > replicas to achieve quorum - it stops, reporting the problem to the > > external orchestrator. > > From the text in this section it does not look like a not planned activity, > but like an already made decision. It is not even in the 'Plans' section. > You just said 'is elected'. By whom? How? > Again, I want to address this to current users - I would rephrase. > If the election is not a part of the RFC, I would suggest moving this out, > or into a separate section 'Plans' or something. Or reformulate this by > saying like 'the cluster stops serving write requests until external > intervention sets a new leader'. And 'it is *advised* to use the one with > the biggest LSN in the old leader's vclock component'. Something like that. > I don't think the automated election should be even a plan for SR. It is a feature on top of it, shouldn't be a prerequisite in any form. > >> Basically, it would be nice to see the split-brain problem description here, > >> and its solution for us. > >> > > I believe the split-brain is under orchestrator control either - we > > should provide API to switch leader in the cluster, so that when a > > former leader came back it will not get quorum for any txn it has, > > replying to customers with failure as a result. > > Exactly. We should provide something for this from inside. But are there > any details? How should that work? Should all the healthy replicas reject > everything from the false-leader? Should the false-leader somehow realize, > that it is not considered a leader anymore, and should stop itself? If we > choose the former way, how a replica defines who is the true leader? For > example, some replicas still may consider the old leader as a true master. > If we choose the latter way, what is the algorithm of determining that we > are not a leader anymore? > It is all about external orchestration - if replica can't get ping from leader it stops, reporting its status to orchestrator. If leader lost number of replicas that makes quorum impossible - it stops replication, reporting to the orchestrator. Will it be sufficient to cover the question? > >>> After snapshot is created the WAL should start from the first operation > >>> that follows the commit operation snapshot is generated for. That means > >>> WAL will contain 'confirm' messages that refer to transactions that are > >>> not present in the WAL. Apparently, we have to allow this for the case > >>> 'confirm' refers to a transaction with LSN less than the first entry in > >>> the WAL. > >> > >> 9. I couldn't understand that. Why confirm is in WAL for data stored in > >> the snap? I thought you said above, that snapshot should be done for all > >> confirmed data. Besides, having confirm out of snap means the snap is > >> not self-sufficient anymore. > >> > > Snap waits for confirm message to start. During this wait the WAL keep > > growing. At the moment confirm arrived the snap will be created - say, > > for txn #10. The WAL will be started with lsn #11 and commit can be > > somewhere lsn #30. > > So, starting with this snap data appears consistent for lsn #10 - it is > > guaranteed by the wait of commit message. Then replay of WAL will come > > to a confirm message lsn #30 - referring to lsn #10 - that actually > > ignored, since it looks beyond the WAL start. There could be confirm > > messages for even earlier txns if wait takes sufficient time - all of > > them will refer to lsn beyond the WAL. And it is Ok. > > What is 'commit message'? I don't see it on the schema above. I see only > confirms. > Sorry, it is a misprint - I meant 'confirm'. > So the problem is that some data may be written to WAL after we started > committing our transactions going to the snap, but before we received a > quorum. And we can't truncate the WAL by the quorum, because there is > already newer data, which was not included into the snap. Because WAL is > not stopped, it still accepts new transactions. Now I understand. > > Would be good to have this example in the RFC. > Ok, I will try to elaborate on this. > >>> Cluster description should contain explicit attribute for each replica > >>> to denote it participates in synchronous activities. Also the description > >>> should contain criterion on how many replicas responses are needed to > >>> achieve the quorum. > >> > >> 11. Aha, I see 'necessary amount' from above is a manually set value. Ok. > >> > >>> > >>> ## Rationale and alternatives > >>> > >>> There is an implementation of synchronous replication as part of gh-980 > >>> activities, still it is not in a state to get into the product. More > >>> than that it intentionally breaks backward compatibility which is a > >>> prerequisite for this proposal. > >> > >> 12. How are we going to deal with fsync()? Will it be forcefully enabled > >> on sync replicas and the leader? > > > > To my understanding - it's up to user. I was considering a cluster that > > has no WAL at all - relying on sychro replication and sufficient number > > of replicas. Everyone who I asked about it told me I'm nuts. To my great > > surprise Alexander Lyapunov brought exactly the same idea to discuss. > > I didn't see an RFC on that, and this can become easily possible, when > in-memory relay is implemented. If it is implemented in a clean way. We > just can turn off the disk backoff, and it will work from memory-only. > It is not in RFC and we had no support from the customers in question. > > All of these is for one resolution: I would keep it for user to decide. > > Obviously, to speed up the processing leader can disable wal completely, > > but to do so we have to re-work the relay to work from memory. Replicas > > can use WAL in a way user wants: 2 replicas with slow HDD should'n wait > > for fsync(), while super-fast Intel DCPMM one can enable it. Balancing > > is up to user. > > Possibility of omitting fsync means that it is possible, that all nodes > write confirm, which is reported to the client, then the nodes restart, > and the data is lost. I would say it somewhere. The data will not be lost, unless _all_ nodes fail at the same time - including leader. Otherwise the data will be propagated from the survivor through the regular replication. No changes here to what we have currently. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-22 16:50 ` Sergey Ostanevich @ 2020-04-22 20:28 ` Vladislav Shpilevoy 0 siblings, 0 replies; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-04-22 20:28 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches >>>> Basically, it would be nice to see the split-brain problem description here, >>>> and its solution for us. >>>> >>> I believe the split-brain is under orchestrator control either - we >>> should provide API to switch leader in the cluster, so that when a >>> former leader came back it will not get quorum for any txn it has, >>> replying to customers with failure as a result. >> >> Exactly. We should provide something for this from inside. But are there >> any details? How should that work? Should all the healthy replicas reject >> everything from the false-leader? Should the false-leader somehow realize, >> that it is not considered a leader anymore, and should stop itself? If we >> choose the former way, how a replica defines who is the true leader? For >> example, some replicas still may consider the old leader as a true master. >> If we choose the latter way, what is the algorithm of determining that we >> are not a leader anymore? >> > It is all about external orchestration - if replica can't get ping from > leader it stops, reporting its status to orchestrator. > If leader lost number of replicas that makes quorum impossible - it > stops replication, reporting to the orchestrator. > Will it be sufficient to cover the question? Perhaps. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-21 22:17 ` Vladislav Shpilevoy 2020-04-22 16:50 ` Sergey Ostanevich @ 2020-04-23 6:58 ` Konstantin Osipov 2020-04-23 9:14 ` Konstantin Osipov 1 sibling, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-04-23 6:58 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/04/22 01:21]: > > To my understanding - it's up to user. I was considering a cluster that > > has no WAL at all - relying on sychro replication and sufficient number > > of replicas. Everyone who I asked about it told me I'm nuts. To my great > > surprise Alexander Lyapunov brought exactly the same idea to discuss. > > I didn't see an RFC on that, and this can become easily possible, when > in-memory relay is implemented. If it is implemented in a clean way. We > just can turn off the disk backoff, and it will work from memory-only. Sync replication must work from in-memory relay only. It works as a natural failure detector: a replica which is slow or unavailable is first removed from the subscribers of in-memory relay, and only then (possibly much much later) is marked as down. By looking at the in-memory relay you have a clear idea what peers are available and can abort a transaction if a cluster is in the downgraded state right away. You never wait for impossible events. If you do have to wait, and say your wait timeout is 1 second, you quickly run out of any fibers in the fiber pool for any work, because all of them will be waiting on the sync transactions they picked up from iproto to finish. The system will loose its throttling capability. There are other reasons, too: the protocol will eventually be quite tricky and the logic has to reside in a single place and not require inter-thread communication. Committing a transaction purely anywhere outside WAL will require inter-thread communication, which is costly and should be avoided. I am surprised I have to explain this again and again - I never assumed this spec is a precursor for a half-backed implementation, only as a high-level description of the next steps after in-memory relay is in. > > All of these is for one resolution: I would keep it for user to decide. > > Obviously, to speed up the processing leader can disable wal completely, > > but to do so we have to re-work the relay to work from memory. Replicas > > can use WAL in a way user wants: 2 replicas with slow HDD should'n wait > > for fsync(), while super-fast Intel DCPMM one can enable it. Balancing > > is up to user. > > Possibility of omitting fsync means that it is possible, that all nodes > write confirm, which is reported to the client, then the nodes restart, > and the data is lost. I would say it somewhere. Worse yet you can elect a leader "based on WAL length" and then it is no longer the leader, because it lost it long WAL after restart. fcync() is mandatory during election, in other cases it shouldn't impact durability in our case. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 6:58 ` Konstantin Osipov @ 2020-04-23 9:14 ` Konstantin Osipov 2020-04-23 11:27 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-04-23 9:14 UTC (permalink / raw) To: Vladislav Shpilevoy, Sergey Ostanevich, tarantool-patches * Konstantin Osipov <kostja.osipov@gmail.com> [20/04/23 09:58]: > > > To my understanding - it's up to user. I was considering a cluster that > > > has no WAL at all - relying on sychro replication and sufficient number > > > of replicas. Everyone who I asked about it told me I'm nuts. To my great > > > surprise Alexander Lyapunov brought exactly the same idea to discuss. > > > > I didn't see an RFC on that, and this can become easily possible, when > > in-memory relay is implemented. If it is implemented in a clean way. We > > just can turn off the disk backoff, and it will work from memory-only. > > Sync replication must work from in-memory relay only. It works as > a natural failure detector: a replica which is slow or unavailable > is first removed from the subscribers of in-memory relay, and only > then (possibly much much later) is marked as down. > > By looking at the in-memory relay you have a clear idea what peers > are available and can abort a transaction if a cluster is in the > downgraded state right away. You never wait for impossible events. > > If you do have to wait, and say your wait timeout is 1 second, you > quickly run out of any fibers in the fiber pool for any work, > because all of them will be waiting on the sync transactions they > picked up from iproto to finish. The system will loose its > throttling capability. The other issue is that if your replicas are alive but slow/lagging behind, you can't let too many undo records to pile up unacknowledged in tx thread. The in-memory relay solves this nicely too, because it kicks out replicas from memory to file mode if they are unable to keep up with the speed of change. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 9:14 ` Konstantin Osipov @ 2020-04-23 11:27 ` Sergey Ostanevich 2020-04-23 11:43 ` Konstantin Osipov 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-23 11:27 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches Hi! Thanks for review! On 23 апр 12:14, Konstantin Osipov wrote: > * Konstantin Osipov <kostja.osipov@gmail.com> [20/04/23 09:58]: > > > > To my understanding - it's up to user. I was considering a cluster that > > > > has no WAL at all - relying on sychro replication and sufficient number > > > > of replicas. Everyone who I asked about it told me I'm nuts. To my great > > > > surprise Alexander Lyapunov brought exactly the same idea to discuss. > > > > > > I didn't see an RFC on that, and this can become easily possible, when > > > in-memory relay is implemented. If it is implemented in a clean way. We > > > just can turn off the disk backoff, and it will work from memory-only. > > > > Sync replication must work from in-memory relay only. It works as > > a natural failure detector: a replica which is slow or unavailable > > is first removed from the subscribers of in-memory relay, and only > > then (possibly much much later) is marked as down. > > > > By looking at the in-memory relay you have a clear idea what peers > > are available and can abort a transaction if a cluster is in the > > downgraded state right away. You never wait for impossible events. > > > > If you do have to wait, and say your wait timeout is 1 second, you > > quickly run out of any fibers in the fiber pool for any work, > > because all of them will be waiting on the sync transactions they > > picked up from iproto to finish. The system will loose its > > throttling capability. > There's no need to explain it to customer: sync replication is not expected to be as fast as pure in-memory. By no means. We have network communication, disk operation, multiple entities quorum - all of these can't be as fast. No need to try cramp more than network can push through, obvoiusly. The quality one buys for this price: consistency of data in multiple instances distributed across different locations. > The other issue is that if your replicas are alive but > slow/lagging behind, you can't let too many undo records to > pile up unacknowledged in tx thread. > The in-memory relay solves this nicely too, because it kicks out > replicas from memory to file mode if they are unable to keep up > with the speed of change. > That is the same problem - resources of leader, so natural limit for throughput. I bet Tarantool faces similar limitations even now, although different ones. The in-memory relay supposed to keep the same interface, so we expect to hop easily to this new shiny express as soon as it appears. This will be an optimization and we're trying to implement something and then speed it up. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 11:27 ` Sergey Ostanevich @ 2020-04-23 11:43 ` Konstantin Osipov 2020-04-23 15:11 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-04-23 11:43 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/04/23 14:29]: > Hi! > > Thanks for review! > > On 23 апр 12:14, Konstantin Osipov wrote: > > * Konstantin Osipov <kostja.osipov@gmail.com> [20/04/23 09:58]: > > > > > To my understanding - it's up to user. I was considering a cluster that > > > > > has no WAL at all - relying on sychro replication and sufficient number > > > > > of replicas. Everyone who I asked about it told me I'm nuts. To my great > > > > > surprise Alexander Lyapunov brought exactly the same idea to discuss. > > > > > > > > I didn't see an RFC on that, and this can become easily possible, when > > > > in-memory relay is implemented. If it is implemented in a clean way. We > > > > just can turn off the disk backoff, and it will work from memory-only. > > > > > > Sync replication must work from in-memory relay only. It works as > > > a natural failure detector: a replica which is slow or unavailable > > > is first removed from the subscribers of in-memory relay, and only > > > then (possibly much much later) is marked as down. > > > > > > By looking at the in-memory relay you have a clear idea what peers > > > are available and can abort a transaction if a cluster is in the > > > downgraded state right away. You never wait for impossible events. > > > > > > If you do have to wait, and say your wait timeout is 1 second, you > > > quickly run out of any fibers in the fiber pool for any work, > > > because all of them will be waiting on the sync transactions they > > > picked up from iproto to finish. The system will loose its > > > throttling capability. > > > There's no need to explain it to customer: sync replication is not > expected to be as fast as pure in-memory. By no means. We have network > communication, disk operation, multiple entities quorum - all of these > can't be as fast. No need to try cramp more than network can push > through, obvoiusly. This expected performance overhead is not a grant to run out of memory or available fibers on a node failure or network partitioning. > The quality one buys for this price: consistency of data in multiple > instances distributed across different locations. The spec should demonstrate the consistency is guaranteed: right now it can easily be violated during a leader change, and this is left out of scope of the spec. My take is that any implementation which is not close enough to a TLA+ proven spec is not trustworthy, so I would not claim myself or trust any one elses claims that it is consistent. At best this RFC could achieve durability, by ensuring that no transaction is committed unless it is delivered to a majority of replicas. Consistency requires implementing RAFT spec in full and showing that leader changes preserve the write ahead log linearizability. > > The other issue is that if your replicas are alive but > > slow/lagging behind, you can't let too many undo records to > > pile up unacknowledged in tx thread. > > The in-memory relay solves this nicely too, because it kicks out > > replicas from memory to file mode if they are unable to keep up > > with the speed of change. > > > That is the same problem - resources of leader, so natural limit for > throughput. I bet Tarantool faces similar limitations even now, > although different ones. > > The in-memory relay supposed to keep the same interface, so we expect to > hop easily to this new shiny express as soon as it appears. This will be > an optimization and we're trying to implement something and then speed > it up. It is pretty clear that the implementation will be different. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 11:43 ` Konstantin Osipov @ 2020-04-23 15:11 ` Sergey Ostanevich 2020-04-23 20:39 ` Konstantin Osipov 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-23 15:11 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches On 23 апр 14:43, Konstantin Osipov wrote: > > The quality one buys for this price: consistency of data in multiple > > instances distributed across different locations. > > The spec should demonstrate the consistency is guaranteed: right > now it can easily be violated during a leader change, and this is > left out of scope of the spec. > > My take is that any implementation which is not close enough to a > TLA+ proven spec is not trustworthy, so I would not claim myself > or trust any one elses claims that it is consistent. At best this > RFC could achieve durability, by ensuring that no transaction is > committed unless it is delivered to a majority of replicas. What is exactly mentioned in RFC goals. > Consistency requires implementing RAFT spec in full and showing > that leader changes preserve the write ahead log linearizability. > So the leader should stop accepting transactions, wait for all txn in queue resolved into confirmed either issue a rollback - after a timeout as a last resort. Since no automation in leader election the cluster will appear in a consistent state after this. Now a new leader can be appointed with all circumstances taken into account - nodes availability, ping from the proxy, lsn, etc. Again, this RFC is not about any HA features, such as auto-failover. > > > The other issue is that if your replicas are alive but > > > slow/lagging behind, you can't let too many undo records to > > > pile up unacknowledged in tx thread. > > > The in-memory relay solves this nicely too, because it kicks out > > > replicas from memory to file mode if they are unable to keep up > > > with the speed of change. > > > > > That is the same problem - resources of leader, so natural limit for > > throughput. I bet Tarantool faces similar limitations even now, > > although different ones. > > > > The in-memory relay supposed to keep the same interface, so we expect to > > hop easily to this new shiny express as soon as it appears. This will be > > an optimization and we're trying to implement something and then speed > > it up. > > It is pretty clear that the implementation will be different. > Which contradicts to the interface preservance, right? > -- > Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 15:11 ` Sergey Ostanevich @ 2020-04-23 20:39 ` Konstantin Osipov 0 siblings, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-04-23 20:39 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/04/23 18:11]: > > The spec should demonstrate the consistency is guaranteed: right > > now it can easily be violated during a leader change, and this is > > left out of scope of the spec. > > > > My take is that any implementation which is not close enough to a > > TLA+ proven spec is not trustworthy, so I would not claim myself > > or trust any one elses claims that it is consistent. At best this > > RFC could achieve durability, by ensuring that no transaction is > > committed unless it is delivered to a majority of replicas. > > What is exactly mentioned in RFC goals. This is durability, though, not consistency. My point is: if consistency can not be guaranteed anyway, why assume single leader. Let's consider what happens if all replicas are allowed to collect acks, define for it the same semantics as we do today in case of async multi-master. Then add the remaining bits of RAFT. > > > Consistency requires implementing RAFT spec in full and showing > > that leader changes preserve the write ahead log linearizability. > > > So the leader should stop accepting transactions, wait for all txn in > queue resolved into confirmed either issue a rollback - after a > timeout as a last resort. > Since no automation in leader election the cluster will appear in a > consistent state after this. Now a new leader can be appointed with > all circumstances taken into account - nodes availability, ping from > the proxy, lsn, etc. > Again, this RFC is not about any HA features, such as auto-failover. > > > > > The other issue is that if your replicas are alive but > > > > slow/lagging behind, you can't let too many undo records to > > > > pile up unacknowledged in tx thread. > > > > The in-memory relay solves this nicely too, because it kicks out > > > > replicas from memory to file mode if they are unable to keep up > > > > with the speed of change. > > > > > > > That is the same problem - resources of leader, so natural limit for > > > throughput. I bet Tarantool faces similar limitations even now, > > > although different ones. > > > > > > The in-memory relay supposed to keep the same interface, so we expect to > > > hop easily to this new shiny express as soon as it appears. This will be > > > an optimization and we're trying to implement something and then speed > > > it up. > > > > It is pretty clear that the implementation will be different. > > > Which contradicts to the interface preservance, right? I don't believe internals and API can be so disconnected. I think in-memory relay is such a significant change that the implementation has to build upon it. The trigger-based implementation was contributed back in 2015 and went nowhere, in fact it was an inspiration to create a backlog of such items as parallel applier, applier in iproto, in-memory relay, and so on - all of these are "review items" for the trigger-based syncrep: https://github.com/Alexey-Ivanensky/tarantool/tree/bsync -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-03 21:08 [Tarantool-patches] [RFC] Quorum-based synchronous replication Sergey Ostanevich ` (2 preceding siblings ...) 2020-04-20 23:32 ` Vladislav Shpilevoy @ 2020-04-23 21:38 ` Vladislav Shpilevoy 2020-04-23 22:28 ` Konstantin Osipov 2020-04-30 14:50 ` Sergey Ostanevich 3 siblings, 2 replies; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-04-23 21:38 UTC (permalink / raw) To: Sergey Ostanevich, tarantool-patches, Timur Safin, Mons Anderson Hi! Here is a short summary of our late night discussion and the questions it brought up, while I was trying to design a draft plan of an implementation. Since the RFC is too far from the code, and I needed a more 'pedestrian' and detailed plan. The question is about 'confirm' message and quorum collection. Here is the schema presented in the RFC: > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN undo log | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN undo log | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN undo log | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | [TXN undo log | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | It says, that once the quorum is collected, and 'confirm' is written to local leader's WAL, it is considered committed and is reported to the client as successful. On the other hand it is said, that in case of leader change the new leader will rollback all not confirmed transactions. That leads to the following bug: Assume we have 4 instances: i1, i2, i3, i4. Leader is i1. It writes a transaction with LSN1. The LSN1 is sent to other nodes, they apply it ok, and send acks to the leader. The leader sees i2-i4 all applied the transaction (propagated their LSNs to LSN1). It writes 'confirm' to its local WAL, reports it to the client as success, the client's request is over, it is returned back to some remote node, etc. The transaction is officially synchronously committed. Then the leader's machine dies - disk is dead. The confirm was not sent to any of the other nodes. For example, it started having problems with network connection to the replicas recently before the death. Or it just didn't manage to hand the confirm out. From now on if any of the other nodes i2-i4 becomes a leader, it will rollback the officially confirmed transaction, even if it has it, and all the other nodes too. That basically means, this sync replication gives exactly the same guarantees as the async replication - 'confirm' on the leader tells nothing about replicas except that they *are able to apply the transaction*, but still may not apply it. Am I missing something? Another issue is with failure detection. Lets assume, that we wait for 'confirm' to be propagated on quorum of replicas too. Assume some replicas responded with an error. So they first said they can apply the transaction, and saved it into their WALs, and then they couldn't apply confirm. That could happen because of 2 reasons: replica has problems with WAL, or the replica becomes unreachable from the master. WAL-problematic replicas can be disconnected forcefully, since they are clearly not able to work properly anymore. But what to do with disconnected replicas? 'Confirm' can't wait for them forever - we will run out of fibers, if we have even just hundreds of RPS of sync transactions, and wait for, lets say, a few minutes. On the other hand we can't roll them back, because 'confirm' has been written to the local WAL already. Note for those who is concerned: this has nothing to do with in-memory relay. It has the same problems, which are in the protocol, not in the implementation. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 21:38 ` Vladislav Shpilevoy @ 2020-04-23 22:28 ` Konstantin Osipov 2020-04-30 14:50 ` Sergey Ostanevich 1 sibling, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-04-23 22:28 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/04/24 00:42]: > It says, that once the quorum is collected, and 'confirm' is written > to local leader's WAL, it is considered committed and is reported > to the client as successful. > > On the other hand it is said, that in case of leader change the > new leader will rollback all not confirmed transactions. That leads > to the following bug: > > Assume we have 4 instances: i1, i2, i3, i4. Leader is i1. It > writes a transaction with LSN1. The LSN1 is sent to other nodes, > they apply it ok, and send acks to the leader. The leader sees > i2-i4 all applied the transaction (propagated their LSNs to LSN1). > It writes 'confirm' to its local WAL, reports it to the client as > success, the client's request is over, it is returned back to > some remote node, etc. The transaction is officially synchronously > committed. > > Then the leader's machine dies - disk is dead. The confirm was > not sent to any of the other nodes. For example, it started having > problems with network connection to the replicas recently before > the death. Or it just didn't manage to hand the confirm out. > > >From now on if any of the other nodes i2-i4 becomes a leader, it > will rollback the officially confirmed transaction, even if it > has it, and all the other nodes too. > > That basically means, this sync replication gives exactly the same > guarantees as the async replication - 'confirm' on the leader tells > nothing about replicas except that they *are able to apply the > transaction*, but still may not apply it. > > Am I missing something? This video explains what leader has to do after it's been elected: https://www.youtube.com/watch?v=YbZ3zDzDnrw In short, the transactions in leader's wal has to be committed, not rolled back. Raft paper has https://raft.github.io/raft.pdf has answers in a concise single page summary. Why have this discussion at all, any ambiguity or discrepancy between this document and raft paper should be treated as a mistake in this document. Or do you actually think it's possible to invent a new consensus algorithm this way? > Note for those who is concerned: this has nothing to do with > in-memory relay. It has the same problems, which are in the protocol, > not in the implementation. No, the issues are distinct: 1) there may be cases where this paper doesn't follow RAFT. It should be obvious to everyone, that with the exception to external leader election and failure detection it has to if correctness is of any concern, so it's simply a matter of fixing this doc to match raft. As to the leader election, there are two alternatives: either spec out in this paper how the external election is interacting with the cluster, including finishing up old transactions and neutralizing old leaders, or allow multi-master, so forget about consistency for now. 2) an implementation based on triggers will be complicated and will have performance/stability implications. This is what I hope I was able to convey and in this case we can put the matter to rest. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-23 21:38 ` Vladislav Shpilevoy 2020-04-23 22:28 ` Konstantin Osipov @ 2020-04-30 14:50 ` Sergey Ostanevich 2020-05-06 8:52 ` Konstantin Osipov ` (2 more replies) 1 sibling, 3 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-04-30 14:50 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! Thanks for the review! After a long discussion we agreed to rework the RFC. On 23 апр 23:38, Vladislav Shpilevoy wrote: > It says, that once the quorum is collected, and 'confirm' is written > to local leader's WAL, it is considered committed and is reported > to the client as successful. > > On the other hand it is said, that in case of leader change the > new leader will rollback all not confirmed transactions. That leads This is no longer right, we decided to follow the RAFT's approach that leader rules the world, hence committing all changes in it's WAL. > > Another issue is with failure detection. Lets assume, that we wait > for 'confirm' to be propagated on quorum of replicas too. Assume > some replicas responded with an error. So they first said they can > apply the transaction, and saved it into their WALs, and then they > couldn't apply confirm. That could happen because of 2 reasons: > replica has problems with WAL, or the replica becomes unreachable > from the master. > > WAL-problematic replicas can be disconnected forcefully, since they > are clearly not able to work properly anymore. But what to do with > disconnected replicas? 'Confirm' can't wait for them forever - we > will run out of fibers, if we have even just hundreds of RPS of > sync transactions, and wait for, lets say, a few minutes. On the > other hand we can't roll them back, because 'confirm' has been > written to the local WAL already. Here we agreed that replica will be kicked out of cluster and wait for human intervention to fix the problems - probably with rejoin. In case available replics are not enough to achieve the quorum leader also reports the problem and stop the cluster operation until cluster reconfigured or number of replicas will become sufficient. Below is the new RFC, available at https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md --- * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a Tarantool cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is the absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatibility and ease of cluster orchestration. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | [TXN undo log | | | | destroyed] | | | | | | | | | |---Confirm--->| | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's [LEADER_ID, LSN] and the confirm has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a TXN application success to the leader via the IPROTO explicitly to allow leader to collect the quorum for the TXN. In case of application failure the replica has to disconnect from the replication the same way as it is done now. The replica also has to report its disconnection to the orchestrator. Further actions require human intervention, since failure means either technical problem (such as not enough space for WAL) that has to be resovled or an inconsistent state that requires rejoin. As soon as leader appears in a situation it has not enough replicas to achieve quorum, the cluster should stop accepting any requests - both write and read. The reason for this is that replication of transactions can achieve quorum on replicas not visible to the leader. On the other hand, leader can't achieve quorum with available minority. Leader has to report the state and wait for human intervention. There's an option to ask leader to rollback to the latest transaction that has quorum: leader issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN is of the first transaction in the leader's undo log. The rollback message replicated to the available cluster will put it in a consistent state. After that configuration of the cluster can be updated to available quorum and leader can be switched back to write mode. ### Leader role assignment. To assign a leader role to an instance the following should be performed: 1. among all available instances pick the one that has the biggest vclock element of the former leader ID; an arbitrary istance can be selected in case it is first time the leader is assigned 2. the leader should assure that number of available instances in the cluster is enough to achieve the quorum and proceed to step 3, otherwise the leader should report the situation of incomplete quorum, as in the last paragraph of previous section 3. the selected instance has to take the responsibility to replicate former leader entries from its WAL, obtainig quorum and commit confirm messages referring to [FORMER_LEADER_ID, LSN] in its WAL, replicating to the cluster, after that it can start adding its own entries into the WAL ### Recovery and failover. Tarantool instance during reading WAL should postpone the undo log deletion until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep undo log for all transactions that are waiting for a confirm entry until the role of the instance is set. If this instance will be assigned a leader role then all transactions that have no corresponding confirm message should be confirmed (see the leader role assignment). In case there's not enough replicas to set up a quorum the cluster can be switched into a read-only mode. Note, this can't be done by default since some of transactions can have confirmed state. It is up to human intervention to force rollback of all transactions that have no confirm and to put the cluster into a consistent state. In case the instance will be assigned a replica role, it may appear in a state that it has conflicting WAL entries, in case it recovered from a leader role and some of transactions didn't replicated to the current leader. This situation should be resolved through rejoin of the instance. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - no matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-30 14:50 ` Sergey Ostanevich @ 2020-05-06 8:52 ` Konstantin Osipov 2020-05-06 16:39 ` Sergey Ostanevich 2020-05-06 18:55 ` Konstantin Osipov 2020-05-07 23:01 ` Konstantin Osipov 2 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-05-06 8:52 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/04/30 17:51]: > Hi! > > Thanks for the review! > > After a long discussion we agreed to rework the RFC. > > On 23 апр 23:38, Vladislav Shpilevoy wrote: > > It says, that once the quorum is collected, and 'confirm' is written > > to local leader's WAL, it is considered committed and is reported > > to the client as successful. > > > > On the other hand it is said, that in case of leader change the > > new leader will rollback all not confirmed transactions. That leads > > This is no longer right, we decided to follow the RAFT's approach that > leader rules the world, hence committing all changes in it's WAL. > > > > > Another issue is with failure detection. Lets assume, that we wait > > for 'confirm' to be propagated on quorum of replicas too. Assume > > some replicas responded with an error. So they first said they can > > apply the transaction, and saved it into their WALs, and then they > > couldn't apply confirm. That could happen because of 2 reasons: > > replica has problems with WAL, or the replica becomes unreachable > > from the master. > > > > WAL-problematic replicas can be disconnected forcefully, since they > > are clearly not able to work properly anymore. But what to do with > > disconnected replicas? 'Confirm' can't wait for them forever - we > > will run out of fibers, if we have even just hundreds of RPS of > > sync transactions, and wait for, lets say, a few minutes. On the > > other hand we can't roll them back, because 'confirm' has been > > written to the local WAL already. > > Here we agreed that replica will be kicked out of cluster and wait for > human intervention to fix the problems - probably with rejoin. In case > available replics are not enough to achieve the quorum leader also > reports the problem and stop the cluster operation until cluster > reconfigured or number of replicas will become sufficient. > > Below is the new RFC, available at > https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/doc/rfc/quorum-based-synchro.md > > --- > * **Status**: In progress > * **Start date**: 31-03-2020 > * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> > * **Issues**: https://github.com/tarantool/tarantool/issues/4842 > > ## Summary > > The aim of this RFC is to address the following list of problems > formulated at MRG planning meeting: > - protocol backward compatibility to enable cluster upgrade w/o > downtime > - consistency of data on replica and leader > - switch from leader to replica without data loss > - up to date replicas to run read-only requests > - ability to switch async replicas into sync ones and vice versa > - guarantee of rollback on leader and sync replicas > - simplicity of cluster orchestration > > What this RFC is not: > > - high availability (HA) solution with automated failover, roles > assignments an so on > - master-master configuration support > > ## Background and motivation > > There are number of known implementation of consistent data presence in > a Tarantool cluster. They can be commonly named as "wait for LSN" > technique. The biggest issue with this technique is the absence of > rollback guarantees at replica in case of transaction failure on one > master or some of the replicas in the cluster. > > To provide such capabilities a new functionality should be introduced in > Tarantool core, with requirements mentioned before - backward > compatibility and ease of cluster orchestration. > > ## Detailed design > > ### Quorum commit > > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN undo log | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN undo log | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN undo log | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN undo log | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN undo log | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN undo log | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | [TXN undo log | | | > | destroyed] | | | > | | | | | > | |---Confirm--->| | | > | | | | | What happens if writing Confirm to WAL fails? TXN und log record is destroyed already. Will the server panic now on WAL failure, even if it is intermittent? > | |----------Confirm---------->| | What happens if peers receive and maybe even write Confirm to their WALs but local WAL write is lost after a restart? WAL is not synced, so we can easily lose the tail of the WAL. Tarantool will sync up with all replicas on restart, but there will be no "Replication OK" messages from them, so it wouldn't know that the transaction is committed on them. How is this handled? We may end up with some replicas confirming the transaction while the leader will roll it back on restart. Do you suggest there is a human intervention on restart as well? > | | | | | > |<---TXN Ok-----| | [TXN undo log | > | | | destroyed] | > | | | | | > | | | |---Confirm--->| > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > the confirm has its own LSN. This confirm message is delivered to all > replicas through the existing replication mechanism. > > Replica should report a TXN application success to the leader via the > IPROTO explicitly to allow leader to collect the quorum for the TXN. > In case of application failure the replica has to disconnect from the > replication the same way as it is done now. The replica also has to > report its disconnection to the orchestrator. Further actions require > human intervention, since failure means either technical problem (such > as not enough space for WAL) that has to be resovled or an inconsistent > state that requires rejoin. > As soon as leader appears in a situation it has not enough replicas > to achieve quorum, the cluster should stop accepting any requests - both > write and read. How does *the cluster* know the state of the leader and if it doesn't, how it can possibly implement this? Did you mean the leader should stop accepting transactions here? But how can the leader know if it has not enough replicas during a read transaction, if it doesn't contact any replica to serve a read? > The reason for this is that replication of transactions > can achieve quorum on replicas not visible to the leader. On the other > hand, leader can't achieve quorum with available minority. Leader has to > report the state and wait for human intervention. There's an option to > ask leader to rollback to the latest transaction that has quorum: leader > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > is of the first transaction in the leader's undo log. The rollback > message replicated to the available cluster will put it in a consistent > state. After that configuration of the cluster can be updated to > available quorum and leader can be switched back to write mode. As you should be able to conclude from restart scenario, it is possible a replica has the record in *confirmed* state but the leader has it in pending state. The replica will not be able to roll back then. Do you suggest the replica should abort if it can't rollback? This may lead to an avalanche of rejoins on leader restart, bringing performance to a halt. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 8:52 ` Konstantin Osipov @ 2020-05-06 16:39 ` Sergey Ostanevich 2020-05-06 18:44 ` Konstantin Osipov 2020-05-13 21:36 ` Vladislav Shpilevoy 0 siblings, 2 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-06 16:39 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches Hi! Thanks for review! > > | | | | | > > | [Quorum | | | > > | achieved] | | | > > | | | | | > > | [TXN undo log | | | > > | destroyed] | | | > > | | | | | > > | |---Confirm--->| | | > > | | | | | > > What happens if writing Confirm to WAL fails? TXN und log record > is destroyed already. Will the server panic now on WAL failure, > even if it is intermittent? I would like to have an example of intermittent WAL failure. Can it be other than problem with disc - be it space/availability/malfunction? For all of those it should be resolved outside the DBMS anyways. So, leader should stop and report its problems to orchestrator/admins. I would agree that undo log can be destroyed *after* the Confirm is landed to WAL - same is for replica. > > > | |----------Confirm---------->| | > > What happens if peers receive and maybe even write Confirm to their WALs > but local WAL write is lost after a restart? Did you mean WAL write on leader as a local? Then we have a replica with a bigger LSN for the leader ID. > WAL is not synced, > so we can easily lose the tail of the WAL. Tarantool will sync up > with all replicas on restart, But at this point a new leader will be appointed - the old one is restarted. Then the Confirm message will arrive to the restarted leader through a regular replication. > but there will be no "Replication > OK" messages from them, so it wouldn't know that the transaction > is committed on them. How is this handled? We may end up with some > replicas confirming the transaction while the leader will roll it > back on restart. Do you suggest there is a human intervention on > restart as well? > > > > | | | | | > > |<---TXN Ok-----| | [TXN undo log | > > | | | destroyed] | > > | | | | | > > | | | |---Confirm--->| > > | | | | | > > ``` > > > > The quorum should be collected as a table for a list of transactions > > waiting for quorum. The latest transaction that collects the quorum is > > considered as complete, as well as all transactions prior to it, since > > all transactions should be applied in order. Leader writes a 'confirm' > > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > > the confirm has its own LSN. This confirm message is delivered to all > > replicas through the existing replication mechanism. > > > > Replica should report a TXN application success to the leader via the > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > In case of application failure the replica has to disconnect from the > > replication the same way as it is done now. The replica also has to > > report its disconnection to the orchestrator. Further actions require > > human intervention, since failure means either technical problem (such > > as not enough space for WAL) that has to be resovled or an inconsistent > > state that requires rejoin. > > > As soon as leader appears in a situation it has not enough replicas > > to achieve quorum, the cluster should stop accepting any requests - both > > write and read. > > How does *the cluster* know the state of the leader and if it > doesn't, how it can possibly implement this? Did you mean > the leader should stop accepting transactions here? But how can > the leader know if it has not enough replicas during a read > transaction, if it doesn't contact any replica to serve a read? I expect to have a disconnection trigger assigned to all relays so that disconnection will cause the number of replicas decrease. The quorum size is static, so we can stop at the very moment the number dives below. > > > The reason for this is that replication of transactions > > can achieve quorum on replicas not visible to the leader. On the other > > hand, leader can't achieve quorum with available minority. Leader has to > > report the state and wait for human intervention. There's an option to > > ask leader to rollback to the latest transaction that has quorum: leader > > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > > is of the first transaction in the leader's undo log. The rollback > > message replicated to the available cluster will put it in a consistent > > state. After that configuration of the cluster can be updated to > > available quorum and leader can be switched back to write mode. > > As you should be able to conclude from restart scenario, it is > possible a replica has the record in *confirmed* state but the > leader has it in pending state. The replica will not be able to > roll back then. Do you suggest the replica should abort if it > can't rollback? This may lead to an avalanche of rejoins on leader > restart, bringing performance to a halt. No, I declare replica with biggest LSN as a new shining leader. More than that, new leader can (so far it will be by default) finalize the former leader life's work by replicating txns and appropriate confirms. Sergos. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 16:39 ` Sergey Ostanevich @ 2020-05-06 18:44 ` Konstantin Osipov 2020-05-12 15:55 ` Sergey Ostanevich 2020-05-13 21:36 ` Vladislav Shpilevoy 1 sibling, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-05-06 18:44 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/05/06 19:41]: > > > | | | | | > > > | [Quorum | | | > > > | achieved] | | | > > > | | | | | > > > | [TXN undo log | | | > > > | destroyed] | | | > > > | | | | | > > > | |---Confirm--->| | | > > > | | | | | > > > > What happens if writing Confirm to WAL fails? TXN und log record > > is destroyed already. Will the server panic now on WAL failure, > > even if it is intermittent? > > I would like to have an example of intermittent WAL failure. Can it be > other than problem with disc - be it space/availability/malfunction? For SAN disks it can simply be a networking issue. The same is true for any virtual filesystem in the cloud. For local disks it is most often out of space, but this is not an impossible event. > For all of those it should be resolved outside the DBMS anyways. So, > leader should stop and report its problems to orchestrator/admins. Sergey, I understand that RAFT spec is big and with this spec you try to split it into manageable parts. The question is how useful is this particular piece. I'm trying to point out that "the leader should stop" is not a silver bullet - especially since each such stop may mean a rejoin of some other node. The purpose of sync replication is to provide consistency without reducing availability (i.e. make progress as long as the quorum of nodes make progress). The current spec, suggesting there should be a leader stop in case of most errors, reduces availability significantly, and doesn't make external coordinator job any easier - it still has to follow to the letter the prescriptions of RAFT. > landed to WAL - same is for replica. > > > > > > | |----------Confirm---------->| | > > > > What happens if peers receive and maybe even write Confirm to their WALs > > but local WAL write is lost after a restart? > > Did you mean WAL write on leader as a local? Then we have a replica with > a bigger LSN for the leader ID. > > WAL is not synced, > > so we can easily lose the tail of the WAL. Tarantool will sync up > > with all replicas on restart, > > But at this point a new leader will be appointed - the old one is > restarted. Then the Confirm message will arrive to the restarted leader > through a regular replication. This assumes that restart is guaranteed to be noticed by the external coordinator and there is an election on every restart. > > but there will be no "Replication > > OK" messages from them, so it wouldn't know that the transaction > > is committed on them. How is this handled? We may end up with some > > replicas confirming the transaction while the leader will roll it > > back on restart. Do you suggest there is a human intervention on > > restart as well? > > > > > > > | | | | | > > > |<---TXN Ok-----| | [TXN undo log | > > > | | | destroyed] | > > > | | | | | > > > | | | |---Confirm--->| > > > | | | | | > > > ``` > > > > > > The quorum should be collected as a table for a list of transactions > > > waiting for quorum. The latest transaction that collects the quorum is > > > considered as complete, as well as all transactions prior to it, since > > > all transactions should be applied in order. Leader writes a 'confirm' > > > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > > > the confirm has its own LSN. This confirm message is delivered to all > > > replicas through the existing replication mechanism. > > > > > > Replica should report a TXN application success to the leader via the > > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > > In case of application failure the replica has to disconnect from the > > > replication the same way as it is done now. The replica also has to > > > report its disconnection to the orchestrator. Further actions require > > > human intervention, since failure means either technical problem (such > > > as not enough space for WAL) that has to be resovled or an inconsistent > > > state that requires rejoin. > > > > > As soon as leader appears in a situation it has not enough replicas > > > to achieve quorum, the cluster should stop accepting any requests - both > > > write and read. > > > > How does *the cluster* know the state of the leader and if it > > doesn't, how it can possibly implement this? Did you mean > > the leader should stop accepting transactions here? But how can > > the leader know if it has not enough replicas during a read > > transaction, if it doesn't contact any replica to serve a read? > > I expect to have a disconnection trigger assigned to all relays so that > disconnection will cause the number of replicas decrease. The quorum > size is static, so we can stop at the very moment the number dives below. What happens between the event the leader is partitioned away and a new leader is elected? The leader may be unaware of the events and serve a read just fine. So at least you can't say the leader shouldn't be serving reads without quorum - because the only way to achieve it is to collect a quorum of responses to reads as well. > > > The reason for this is that replication of transactions > > > can achieve quorum on replicas not visible to the leader. On the other > > > hand, leader can't achieve quorum with available minority. Leader has to > > > report the state and wait for human intervention. There's an option to > > > ask leader to rollback to the latest transaction that has quorum: leader > > > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > > > is of the first transaction in the leader's undo log. The rollback > > > message replicated to the available cluster will put it in a consistent > > > state. After that configuration of the cluster can be updated to > > > available quorum and leader can be switched back to write mode. > > > > As you should be able to conclude from restart scenario, it is > > possible a replica has the record in *confirmed* state but the > > leader has it in pending state. The replica will not be able to > > roll back then. Do you suggest the replica should abort if it > > can't rollback? This may lead to an avalanche of rejoins on leader > > restart, bringing performance to a halt. > > No, I declare replica with biggest LSN as a new shining leader. More > than that, new leader can (so far it will be by default) finalize the > former leader life's work by replicating txns and appropriate confirms. Right, this also assumes the restart is noticed, so it follows the same logic. -- Konstantin Osipov, Moscow, Russia https://scylladb.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 18:44 ` Konstantin Osipov @ 2020-05-12 15:55 ` Sergey Ostanevich 2020-05-12 16:42 ` Konstantin Osipov 2020-05-13 21:39 ` Vladislav Shpilevoy 0 siblings, 2 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-12 15:55 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches On 06 мая 21:44, Konstantin Osipov wrote: > * Sergey Ostanevich <sergos@tarantool.org> [20/05/06 19:41]: > > > > | | | | | > > > > | [Quorum | | | > > > > | achieved] | | | > > > > | | | | | > > > > | [TXN undo log | | | > > > > | destroyed] | | | > > > > | | | | | > > > > | |---Confirm--->| | | > > > > | | | | | > > > > > > What happens if writing Confirm to WAL fails? TXN und log record > > > is destroyed already. Will the server panic now on WAL failure, > > > even if it is intermittent? > > > > I would like to have an example of intermittent WAL failure. Can it be > > other than problem with disc - be it space/availability/malfunction? > > For SAN disks it can simply be a networking issue. The same is > true for any virtual filesystem in the cloud. For local disks it > is most often out of space, but this is not an impossible event. The SANDisk is an SSD vendor. I bet you mean NAS - network array storage, isn't it? Then I see no difference in WAL write into NAS in current schema - you will catch a timeout, WAL will report failure, replica stops. > > > For all of those it should be resolved outside the DBMS anyways. So, > > leader should stop and report its problems to orchestrator/admins. > > Sergey, I understand that RAFT spec is big and with this spec you > try to split it into manageable parts. The question is how useful > is this particular piece. I'm trying to point out that "the leader > should stop" is not a silver bullet - especially since each such > stop may mean a rejoin of some other node. The purpose of sync > replication is to provide consistency without reducing > availability (i.e. make progress as long as the quorum > of nodes make progress). I'm not sure if we're talking about the same RAFT - mine is "In Search of an Understandable Consensus Algorithm (Extended Version)" from Stanford as of May 2014. And it is 15 pages - including references, conclusions and intro. Seems not that big. Although, most of it is dedicated to the leader election itself, which we intentionally put aside from this RFC. It is written in the very beginning and I empasized this by explicit mentioning of it. > > The current spec, suggesting there should be a leader stop in case > of most errors, reduces availability significantly, and doesn't > make external coordinator job any easier - it still has to follow to > the letter the prescriptions of RAFT. So, the postponing of a commit until quorum collection is the most useful part of this RFC, also to some point I'm trying to address the WAL insconsistency. Although, it can be covered only partly: if a leader's log diverge in unconfirmed transactions only, then they can be rolled back easiy. Technically, it should be enough if leader changed for a replica from the cluster majority at the moment of failure. Otherwise it will require pre-parsing of the WAL and it can well happens that WAL is not long enough, hence ex-leader still need a complete bootstrap. > > > landed to WAL - same is for replica. > > > > > > > > > > | |----------Confirm---------->| | > > > > > > What happens if peers receive and maybe even write Confirm to their WALs > > > but local WAL write is lost after a restart? > > > > Did you mean WAL write on leader as a local? Then we have a replica with > > a bigger LSN for the leader ID. > > > > WAL is not synced, > > > so we can easily lose the tail of the WAL. Tarantool will sync up > > > with all replicas on restart, > > > > But at this point a new leader will be appointed - the old one is > > restarted. Then the Confirm message will arrive to the restarted leader > > through a regular replication. > > This assumes that restart is guaranteed to be noticed by the > external coordinator and there is an election on every restart. Sure yes, if it restarted - then connection lost can't be unnoticed by anyone, be it coordinator or cluster. > > > > but there will be no "Replication > > > OK" messages from them, so it wouldn't know that the transaction > > > is committed on them. How is this handled? We may end up with some > > > replicas confirming the transaction while the leader will roll it > > > back on restart. Do you suggest there is a human intervention on > > > restart as well? > > > > > > > > > > | | | | | > > > > |<---TXN Ok-----| | [TXN undo log | > > > > | | | destroyed] | > > > > | | | | | > > > > | | | |---Confirm--->| > > > > | | | | | > > > > ``` > > > > > > > > The quorum should be collected as a table for a list of transactions > > > > waiting for quorum. The latest transaction that collects the quorum is > > > > considered as complete, as well as all transactions prior to it, since > > > > all transactions should be applied in order. Leader writes a 'confirm' > > > > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > > > > the confirm has its own LSN. This confirm message is delivered to all > > > > replicas through the existing replication mechanism. > > > > > > > > Replica should report a TXN application success to the leader via the > > > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > > > In case of application failure the replica has to disconnect from the > > > > replication the same way as it is done now. The replica also has to > > > > report its disconnection to the orchestrator. Further actions require > > > > human intervention, since failure means either technical problem (such > > > > as not enough space for WAL) that has to be resovled or an inconsistent > > > > state that requires rejoin. > > > > > > > As soon as leader appears in a situation it has not enough replicas > > > > to achieve quorum, the cluster should stop accepting any requests - both > > > > write and read. > > > > > > How does *the cluster* know the state of the leader and if it > > > doesn't, how it can possibly implement this? Did you mean > > > the leader should stop accepting transactions here? But how can > > > the leader know if it has not enough replicas during a read > > > transaction, if it doesn't contact any replica to serve a read? > > > > I expect to have a disconnection trigger assigned to all relays so that > > disconnection will cause the number of replicas decrease. The quorum > > size is static, so we can stop at the very moment the number dives below. > > What happens between the event the leader is partitioned away and > a new leader is elected? > > The leader may be unaware of the events and serve a read just > fine. As it is stated 20 lines above: > > > > As soon as leader appears in a situation it has not enough > > > > replicas > > > > to achieve quorum, the cluster should stop accepting any > > > > requests - both > > > > write and read. So it will not serve. > > So at least you can't say the leader shouldn't be serving reads > without quorum - because the only way to achieve it is to collect > a quorum of responses to reads as well. The leader lost connection to the (N-Q)+1 repllicas out of the N in cluster with a quorum of Q == it stops serving anything. So the quorum criteria is there: no quorum - no reads. > > > > > The reason for this is that replication of transactions > > > > can achieve quorum on replicas not visible to the leader. On the other > > > > hand, leader can't achieve quorum with available minority. Leader has to > > > > report the state and wait for human intervention. There's an option to > > > > ask leader to rollback to the latest transaction that has quorum: leader > > > > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > > > > is of the first transaction in the leader's undo log. The rollback > > > > message replicated to the available cluster will put it in a consistent > > > > state. After that configuration of the cluster can be updated to > > > > available quorum and leader can be switched back to write mode. > > > > > > As you should be able to conclude from restart scenario, it is > > > possible a replica has the record in *confirmed* state but the > > > leader has it in pending state. The replica will not be able to > > > roll back then. Do you suggest the replica should abort if it > > > can't rollback? This may lead to an avalanche of rejoins on leader > > > restart, bringing performance to a halt. > > > > No, I declare replica with biggest LSN as a new shining leader. More > > than that, new leader can (so far it will be by default) finalize the > > former leader life's work by replicating txns and appropriate confirms. > > Right, this also assumes the restart is noticed, so it follows the > same logic. How a restart can be unnoticed, if it causes disconnection? > > -- > Konstantin Osipov, Moscow, Russia > https://scylladb.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-12 15:55 ` Sergey Ostanevich @ 2020-05-12 16:42 ` Konstantin Osipov 2020-05-13 21:39 ` Vladislav Shpilevoy 1 sibling, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-12 16:42 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/05/12 18:56]: > On 06 мая 21:44, Konstantin Osipov wrote: > > * Sergey Ostanevich <sergos@tarantool.org> [20/05/06 19:41]: > > > > > | | | | | > > > > > | [Quorum | | | > > > > > | achieved] | | | > > > > > | | | | | > > > > > | [TXN undo log | | | > > > > > | destroyed] | | | > > > > > | | | | | > > > > > | |---Confirm--->| | | > > > > > | | | | | > > > > > > > > What happens if writing Confirm to WAL fails? TXN und log record > > > > is destroyed already. Will the server panic now on WAL failure, > > > > even if it is intermittent? > > > > > > I would like to have an example of intermittent WAL failure. Can it be > > > other than problem with disc - be it space/availability/malfunction? > > > > For SAN disks it can simply be a networking issue. The same is > > true for any virtual filesystem in the cloud. For local disks it > > is most often out of space, but this is not an impossible event. > > The SANDisk is an SSD vendor. I bet you mean NAS - network array > storage, isn't it? Then I see no difference in WAL write into NAS in > current schema - you will catch a timeout, WAL will report failure, > replica stops. SAN stands for storage area network. There is no timeout in wal tx bus and no timeout in WAL I/O. A replica doesn't stop on an intermittent failure. Stopping a replica on an intermittent failure reduces availability of non-sync writes. It seems you have some assumptions in mind which are not in the document - e.g. that some timeouts are added. They are not in the POC either. I suppose the document is expected to explain quite accurately what has to be done, e.g. how these new timeouts work? > > > For all of those it should be resolved outside the DBMS anyways. So, > > > leader should stop and report its problems to orchestrator/admins. > > > > Sergey, I understand that RAFT spec is big and with this spec you > > try to split it into manageable parts. The question is how useful > > is this particular piece. I'm trying to point out that "the leader > > should stop" is not a silver bullet - especially since each such > > stop may mean a rejoin of some other node. The purpose of sync > > replication is to provide consistency without reducing > > availability (i.e. make progress as long as the quorum > > of nodes make progress). > > I'm not sure if we're talking about the same RAFT - mine is "In Search > of an Understandable Consensus Algorithm (Extended Version)" from > Stanford as of May 2014. And it is 15 pages - including references, > conclusions and intro. Seems not that big. > > Although, most of it is dedicated to the leader election itself, which > we intentionally put aside from this RFC. It is written in the very > beginning and I empasized this by explicit mentioning of it. I conclude that it is big from the state of this document. It provides some coverage of the normal operation. Leader election, failure detection, recovery/restart, replication configuration changes are either barely mentioned or not covered at all. I find no other reason to not cover them except to be able to come up with a MVP quicker. Do you? > > The current spec, suggesting there should be a leader stop in case > > of most errors, reduces availability significantly, and doesn't > > make external coordinator job any easier - it still has to follow to > > the letter the prescriptions of RAFT. > > So, the postponing of a commit until quorum collection is the most > useful part of this RFC, also to some point I'm trying to address the > WAL insconsistency. > Although, it can be covered only partly: if a > leader's log diverge in unconfirmed transactions only, then they can be > rolled back easiy. Technically, it should be enough if leader changed > for a replica from the cluster majority at the moment of failure. > Otherwise it will require pre-parsing of the WAL and it can well happens > that WAL is not long enough, hence ex-leader still need a complete > bootstrap. I don't understand what's pre-parsing and how what you write is relevant to the fact that reduced availability of non-raft writes is bad. > > > But at this point a new leader will be appointed - the old one is > > > restarted. Then the Confirm message will arrive to the restarted leader > > > through a regular replication. > > > > This assumes that restart is guaranteed to be noticed by the > > external coordinator and there is an election on every restart. > > Sure yes, if it restarted - then connection lost can't be unnoticed by > anyone, be it coordinator or cluster. Well, the spec doesn't say anywhere that the external coordinator has to establish a TCP connection to every participant. Could you please add a chapter where this is clarified? It seems you have a specific coordinator in mind ? > > > > > The quorum should be collected as a table for a list of transactions > > > > > waiting for quorum. The latest transaction that collects the quorum is > > > > > considered as complete, as well as all transactions prior to it, since > > > > > all transactions should be applied in order. Leader writes a 'confirm' > > > > > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > > > > > the confirm has its own LSN. This confirm message is delivered to all > > > > > replicas through the existing replication mechanism. > > > > > > > > > > Replica should report a TXN application success to the leader via the > > > > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > > > > In case of application failure the replica has to disconnect from the > > > > > replication the same way as it is done now. The replica also has to > > > > > report its disconnection to the orchestrator. Further actions require > > > > > human intervention, since failure means either technical problem (such > > > > > as not enough space for WAL) that has to be resovled or an inconsistent > > > > > state that requires rejoin. > > > > > > > > > As soon as leader appears in a situation it has not enough replicas > > > > > to achieve quorum, the cluster should stop accepting any requests - both > > > > > write and read. > > > > > > > > How does *the cluster* know the state of the leader and if it > > > > doesn't, how it can possibly implement this? Did you mean > > > > the leader should stop accepting transactions here? But how can > > > > the leader know if it has not enough replicas during a read > > > > transaction, if it doesn't contact any replica to serve a read? > > > > > > I expect to have a disconnection trigger assigned to all relays so that > > > disconnection will cause the number of replicas decrease. The quorum > > > size is static, so we can stop at the very moment the number dives below. > > > > What happens between the event the leader is partitioned away and > > a new leader is elected? > > > > The leader may be unaware of the events and serve a read just > > fine. > > As it is stated 20 lines above: > > > > > As soon as leader appears in a situation it has not enough > > > > > replicas > > > > > to achieve quorum, the cluster should stop accepting any > > > > > requests - both > > > > > write and read. > > So it will not serve. Sergey, this is recursion. I'm asking you to clarify exactly this point. Do you assume that replicas perform some kind of failure detection? What kind? Is it *in addition* to the failure detection performed by the external coordinator? Any failure detector imaginable would be asynchronous. What happens between the failure and the time it's detected? > > So at least you can't say the leader shouldn't be serving reads > > without quorum - because the only way to achieve it is to collect > > a quorum of responses to reads as well. > > The leader lost connection to the (N-Q)+1 repllicas out of the N in > cluster with a quorum of Q == it stops serving anything. So the quorum > criteria is there: no quorum - no reads. OK, so you assume that TCP connection *is* the failure detector? Failure detection in TCP is optional, asynchronous, and worst of all, unreliable. Why do think it can be used? > > > > > The reason for this is that replication of transactions > > > > > can achieve quorum on replicas not visible to the leader. On the other > > > > > hand, leader can't achieve quorum with available minority. Leader has to > > > > > report the state and wait for human intervention. There's an option to > > > > > ask leader to rollback to the latest transaction that has quorum: leader > > > > > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > > > > > is of the first transaction in the leader's undo log. The rollback > > > > > message replicated to the available cluster will put it in a consistent > > > > > state. After that configuration of the cluster can be updated to > > > > > available quorum and leader can be switched back to write mode. > > > > > > > > As you should be able to conclude from restart scenario, it is > > > > possible a replica has the record in *confirmed* state but the > > > > leader has it in pending state. The replica will not be able to > > > > roll back then. Do you suggest the replica should abort if it > > > > can't rollback? This may lead to an avalanche of rejoins on leader > > > > restart, bringing performance to a halt. > > > > > > No, I declare replica with biggest LSN as a new shining leader. More > > > than that, new leader can (so far it will be by default) finalize the > > > former leader life's work by replicating txns and appropriate confirms. > > > > Right, this also assumes the restart is noticed, so it follows the > > same logic. > > How a restart can be unnoticed, if it causes disconnection? Honestly, I'm baffled. It's like we speak different languages. I can't imagine you are unaware of the fallacies of distributed computing, but I see no other explanation to you question. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-12 15:55 ` Sergey Ostanevich 2020-05-12 16:42 ` Konstantin Osipov @ 2020-05-13 21:39 ` Vladislav Shpilevoy 2020-05-13 23:54 ` Konstantin Osipov 2020-05-14 20:38 ` Sergey Ostanevich 1 sibling, 2 replies; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-05-13 21:39 UTC (permalink / raw) To: Sergey Ostanevich, Konstantin Osipov, tarantool-patches Hi! Thanks for the discussion! On 12/05/2020 17:55, Sergey Ostanevich wrote: > On 06 мая 21:44, Konstantin Osipov wrote: >> * Sergey Ostanevich <sergos@tarantool.org> [20/05/06 19:41]: >>>>> | | | | | >>>>> | [Quorum | | | >>>>> | achieved] | | | >>>>> | | | | | >>>>> | [TXN undo log | | | >>>>> | destroyed] | | | >>>>> | | | | | >>>>> | |---Confirm--->| | | >>>>> | | | | | >>>> >>>> What happens if writing Confirm to WAL fails? TXN und log record >>>> is destroyed already. Will the server panic now on WAL failure, >>>> even if it is intermittent? >>> >>> I would like to have an example of intermittent WAL failure. Can it be >>> other than problem with disc - be it space/availability/malfunction? >> >> For SAN disks it can simply be a networking issue. The same is >> true for any virtual filesystem in the cloud. For local disks it >> is most often out of space, but this is not an impossible event. > > The SANDisk is an SSD vendor. I bet you mean NAS - network array > storage, isn't it? Then I see no difference in WAL write into NAS in > current schema - you will catch a timeout, WAL will report failure, > replica stops. > >> >>> For all of those it should be resolved outside the DBMS anyways. So, >>> leader should stop and report its problems to orchestrator/admins. >> >> Sergey, I understand that RAFT spec is big and with this spec you >> try to split it into manageable parts. The question is how useful >> is this particular piece. I'm trying to point out that "the leader >> should stop" is not a silver bullet - especially since each such >> stop may mean a rejoin of some other node. The purpose of sync >> replication is to provide consistency without reducing >> availability (i.e. make progress as long as the quorum >> of nodes make progress). > > I'm not sure if we're talking about the same RAFT - mine is "In Search > of an Understandable Consensus Algorithm (Extended Version)" from > Stanford as of May 2014. And it is 15 pages - including references, > conclusions and intro. Seems not that big. 15 pages of tightly packed theory is a big piece of data. And especially big, when it comes to application to a real project, with existing infrastructure, and all. Just my IMHO. I remember implementing SWIM - it is smaller than RAFT. Much smaller and simpler, and yet it took year to implement it, and cover all things described in the paper. This is not as simple as it looks, when it comes to edge cases. This is why the whole sync replication frustrates me more than anything else before, and why I am so reluctant to doing anything with it. The RFC mostly covers the normal operation, here I agree with Kostja. But the normal operation is not that interesting. Failures are much more important. > Although, most of it is dedicated to the leader election itself, which > we intentionally put aside from this RFC. It is written in the very > beginning and I empasized this by explicit mentioning of it. And still there will be leader election. Even though not ours for now. And Tarantool should provide API and instructions so as external applications could follow them and do the election. Usually in RFCs we describe API. With arguments, behaviour, and all. >> The current spec, suggesting there should be a leader stop in case >> of most errors, reduces availability significantly, and doesn't >> make external coordinator job any easier - it still has to follow to >> the letter the prescriptions of RAFT. >> >>> >>>> >>>>> | |----------Confirm---------->| | >>>> >>>> What happens if peers receive and maybe even write Confirm to their WALs >>>> but local WAL write is lost after a restart? >>> >>> Did you mean WAL write on leader as a local? Then we have a replica with >>> a bigger LSN for the leader ID. >> >>>> WAL is not synced, >>>> so we can easily lose the tail of the WAL. Tarantool will sync up >>>> with all replicas on restart, >>> >>> But at this point a new leader will be appointed - the old one is >>> restarted. Then the Confirm message will arrive to the restarted leader >>> through a regular replication. >> >> This assumes that restart is guaranteed to be noticed by the >> external coordinator and there is an election on every restart. > > Sure yes, if it restarted - then connection lost can't be unnoticed by > anyone, be it coordinator or cluster. Here comes another problem. Disconnect and restart have nothing to do with each other. The coordinator can loose connection without the peer leader restart. Just because it is network. Anything can happen. Moreover, while the coordinator does not have a connection, the leader can restart multiple times. We can't tell the coordinator rely on connectivity as a restart signal. >>>> but there will be no "Replication >>>> OK" messages from them, so it wouldn't know that the transaction >>>> is committed on them. How is this handled? We may end up with some >>>> replicas confirming the transaction while the leader will roll it >>>> back on restart. Do you suggest there is a human intervention on >>>> restart as well? >>>> >>>> >>>>> | | | | | >>>>> |<---TXN Ok-----| | [TXN undo log | >>>>> | | | destroyed] | >>>>> | | | | | >>>>> | | | |---Confirm--->| >>>>> | | | | | >>>>> ``` >>>>> >>>>> The quorum should be collected as a table for a list of transactions >>>>> waiting for quorum. The latest transaction that collects the quorum is >>>>> considered as complete, as well as all transactions prior to it, since >>>>> all transactions should be applied in order. Leader writes a 'confirm' >>>>> message to the WAL that refers to the transaction's [LEADER_ID, LSN] and >>>>> the confirm has its own LSN. This confirm message is delivered to all >>>>> replicas through the existing replication mechanism. >>>>> >>>>> Replica should report a TXN application success to the leader via the >>>>> IPROTO explicitly to allow leader to collect the quorum for the TXN. >>>>> In case of application failure the replica has to disconnect from the >>>>> replication the same way as it is done now. The replica also has to >>>>> report its disconnection to the orchestrator. Further actions require >>>>> human intervention, since failure means either technical problem (such >>>>> as not enough space for WAL) that has to be resovled or an inconsistent >>>>> state that requires rejoin. >>>> >>>>> As soon as leader appears in a situation it has not enough replicas >>>>> to achieve quorum, the cluster should stop accepting any requests - both >>>>> write and read. >>>> >>>> How does *the cluster* know the state of the leader and if it >>>> doesn't, how it can possibly implement this? Did you mean >>>> the leader should stop accepting transactions here? But how can >>>> the leader know if it has not enough replicas during a read >>>> transaction, if it doesn't contact any replica to serve a read? >>> >>> I expect to have a disconnection trigger assigned to all relays so that >>> disconnection will cause the number of replicas decrease. The quorum >>> size is static, so we can stop at the very moment the number dives below. >> >> What happens between the event the leader is partitioned away and >> a new leader is elected? >> >> The leader may be unaware of the events and serve a read just >> fine. > > As it is stated 20 lines above: >>>>> As soon as leader appears in a situation it has not enough >>>>> replicas >>>>> to achieve quorum, the cluster should stop accepting any >>>>> requests - both >>>>> write and read. > > So it will not serve. This breaks compatibility, since now an orphan node is perfectly able to serve reads. The cluster can't just stop doing everything, if the quorum is lost. Stop writes - yes, since the quorum is lost anyway. But reads do not need a quorum. If you say reads need a quorum, then they would need to go through WAL, collect confirmations, and all. >> So at least you can't say the leader shouldn't be serving reads >> without quorum - because the only way to achieve it is to collect >> a quorum of responses to reads as well. > > The leader lost connection to the (N-Q)+1 repllicas out of the N in > cluster with a quorum of Q == it stops serving anything. So the quorum > criteria is there: no quorum - no reads. Connection count tells nothing. Network connectivity is not a reliable source of information. Only messages and persistent data are reliable (to certain extent). >>>>> The reason for this is that replication of transactions >>>>> can achieve quorum on replicas not visible to the leader. On the other >>>>> hand, leader can't achieve quorum with available minority. Leader has to >>>>> report the state and wait for human intervention. There's an option to >>>>> ask leader to rollback to the latest transaction that has quorum: leader >>>>> issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN >>>>> is of the first transaction in the leader's undo log. The rollback >>>>> message replicated to the available cluster will put it in a consistent >>>>> state. After that configuration of the cluster can be updated to >>>>> available quorum and leader can be switched back to write mode. >>>> >>>> As you should be able to conclude from restart scenario, it is >>>> possible a replica has the record in *confirmed* state but the >>>> leader has it in pending state. The replica will not be able to >>>> roll back then. Do you suggest the replica should abort if it >>>> can't rollback? This may lead to an avalanche of rejoins on leader >>>> restart, bringing performance to a halt. >>> >>> No, I declare replica with biggest LSN as a new shining leader. More >>> than that, new leader can (so far it will be by default) finalize the >>> former leader life's work by replicating txns and appropriate confirms. >> >> Right, this also assumes the restart is noticed, so it follows the >> same logic. > > How a restart can be unnoticed, if it causes disconnection? Disconnection has nothing to do with restart. The coordinator itself may restart. Or it may loose connection to the leader temporarily. Or the leader may loose it without any restarts. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-13 21:39 ` Vladislav Shpilevoy @ 2020-05-13 23:54 ` Konstantin Osipov 2020-05-14 20:38 ` Sergey Ostanevich 1 sibling, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-13 23:54 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/05/14 00:42]: > > Sure yes, if it restarted - then connection lost can't be unnoticed by > > anyone, be it coordinator or cluster. > > Here comes another problem. Disconnect and restart have nothing to do with > each other. The coordinator can loose connection without the peer leader > restart. Just because it is network. Anything can happen. Moreover, while > the coordinator does not have a connection, the leader can restart multiple > times. yes. > We can't tell the coordinator rely on connectivity as a restart signal. Well, we could demand that the leader always demotes itself after restart. But the spec should be explicit about it and explain how the election happens in this case, because it still may have the longest WAL (but with some junk in it, thanks to lost confirms), so after restart the leader may need to reconcile its wal with the majority, fetching missing records back. Once again, RAFT is very explicit about this. By default it requires that the leader commit log is durable, i.e. wal_mode=sync. This would kill performance. Implementations exist which run in wal_mode=write (cassandra is one of them), but they know how to repair the log at the leader before proceeding with the next transaction. The reason I brought this up is that it's extremely tricky, and confusing as hell if the election is external (agree there should be an API, or better yet, abandon the idea of external election, just have no election for now at all, assume the leader never changes, and we only provide durability in multi-master config), with no consistency guarantees (but eventual one). > > How a restart can be unnoticed, if it causes disconnection? > > Disconnection has nothing to do with restart. The coordinator itself may > restart. Or it may loose connection to the leader temporarily. Or the > leader may loose it without any restarts. and yes. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-13 21:39 ` Vladislav Shpilevoy 2020-05-13 23:54 ` Konstantin Osipov @ 2020-05-14 20:38 ` Sergey Ostanevich 2020-05-20 20:59 ` Sergey Ostanevich 1 sibling, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-14 20:38 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! > >> Sergey, I understand that RAFT spec is big and with this spec you > >> try to split it into manageable parts. The question is how useful > >> is this particular piece. I'm trying to point out that "the leader > >> should stop" is not a silver bullet - especially since each such > >> stop may mean a rejoin of some other node. The purpose of sync > >> replication is to provide consistency without reducing > >> availability (i.e. make progress as long as the quorum > >> of nodes make progress). > > > > I'm not sure if we're talking about the same RAFT - mine is "In Search > > of an Understandable Consensus Algorithm (Extended Version)" from > > Stanford as of May 2014. And it is 15 pages - including references, > > conclusions and intro. Seems not that big. > > 15 pages of tightly packed theory is a big piece of data. And especially > big, when it comes to application to a real project, with existing > infrastructure, and all. Just my IMHO. I remember implementing SWIM - it > is smaller than RAFT. Much smaller and simpler, and yet it took year to > implement it, and cover all things described in the paper. That I won't object and this was the reason not to take the RAFT as is and implement it in full for next 2-3 years. That's why we had the very first part of RFC describing what it tries to address and what's not. > > This is not as simple as it looks, when it comes to edge cases. This is > why the whole sync replication frustrates me more than anything else > before, and why I am so reluctant to doing anything with it. > > The RFC mostly covers the normal operation, here I agree with Kostja. But > the normal operation is not that interesting. Failures are much more > important. Definitely and I expect to follow with more functionality on top of it. I believe it will be easier to do if the start will be as small as possible change to the existent code base, which I also try to follow. > > > Although, most of it is dedicated to the leader election itself, which > > we intentionally put aside from this RFC. It is written in the very > > beginning and I empasized this by explicit mentioning of it. > > And still there will be leader election. Even though not ours for now. > And Tarantool should provide API and instructions so as external > applications could follow them and do the election. > > Usually in RFCs we describe API. With arguments, behaviour, and all. That is something I believe should be done after we agree on the whole idea, such as confirm entry in WAL for sync transactions that appeared there earlier. Otherwise we can get very deep into the details, spending time for API definition while the idea itself can appear wrong. I believe that was a common ground to start, but we immediately went to discussion of so many details I tried to keep away before we agree on the key parts, such as WAL consistency or quorum collection. > > >> The current spec, suggesting there should be a leader stop in case > >> of most errors, reduces availability significantly, and doesn't > >> make external coordinator job any easier - it still has to follow to > >> the letter the prescriptions of RAFT. > >> > >>> > >>>> > >>>>> | |----------Confirm---------->| | > >>>> > >>>> What happens if peers receive and maybe even write Confirm to their WALs > >>>> but local WAL write is lost after a restart? > >>> > >>> Did you mean WAL write on leader as a local? Then we have a replica with > >>> a bigger LSN for the leader ID. > >> > >>>> WAL is not synced, > >>>> so we can easily lose the tail of the WAL. Tarantool will sync up > >>>> with all replicas on restart, > >>> > >>> But at this point a new leader will be appointed - the old one is > >>> restarted. Then the Confirm message will arrive to the restarted leader > >>> through a regular replication. > >> > >> This assumes that restart is guaranteed to be noticed by the > >> external coordinator and there is an election on every restart. > > > > Sure yes, if it restarted - then connection lost can't be unnoticed by > > anyone, be it coordinator or cluster. > > Here comes another problem. Disconnect and restart have nothing to do with > each other. The coordinator can loose connection without the peer leader > restart. Just because it is network. Anything can happen. Moreover, while > the coordinator does not have a connection, the leader can restart multiple > times. Definitely there should be a higher level functionality to support some sort of membership protocol, such as SWIM or RAFT itself. But introduction of it should not affect the basic priciples we have to agree upon. > > We can't tell the coordinator rely on connectivity as a restart signal. > > >>>> but there will be no "Replication > >>>> OK" messages from them, so it wouldn't know that the transaction > >>>> is committed on them. How is this handled? We may end up with some > >>>> replicas confirming the transaction while the leader will roll it > >>>> back on restart. Do you suggest there is a human intervention on > >>>> restart as well? > >>>> > >>>> > >>>>> | | | | | > >>>>> |<---TXN Ok-----| | [TXN undo log | > >>>>> | | | destroyed] | > >>>>> | | | | | > >>>>> | | | |---Confirm--->| > >>>>> | | | | | > >>>>> ``` > >>>>> > >>>>> The quorum should be collected as a table for a list of transactions > >>>>> waiting for quorum. The latest transaction that collects the quorum is > >>>>> considered as complete, as well as all transactions prior to it, since > >>>>> all transactions should be applied in order. Leader writes a 'confirm' > >>>>> message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > >>>>> the confirm has its own LSN. This confirm message is delivered to all > >>>>> replicas through the existing replication mechanism. > >>>>> > >>>>> Replica should report a TXN application success to the leader via the > >>>>> IPROTO explicitly to allow leader to collect the quorum for the TXN. > >>>>> In case of application failure the replica has to disconnect from the > >>>>> replication the same way as it is done now. The replica also has to > >>>>> report its disconnection to the orchestrator. Further actions require > >>>>> human intervention, since failure means either technical problem (such > >>>>> as not enough space for WAL) that has to be resovled or an inconsistent > >>>>> state that requires rejoin. > >>>> > >>>>> As soon as leader appears in a situation it has not enough replicas > >>>>> to achieve quorum, the cluster should stop accepting any requests - both > >>>>> write and read. > >>>> > >>>> How does *the cluster* know the state of the leader and if it > >>>> doesn't, how it can possibly implement this? Did you mean > >>>> the leader should stop accepting transactions here? But how can > >>>> the leader know if it has not enough replicas during a read > >>>> transaction, if it doesn't contact any replica to serve a read? > >>> > >>> I expect to have a disconnection trigger assigned to all relays so that > >>> disconnection will cause the number of replicas decrease. The quorum > >>> size is static, so we can stop at the very moment the number dives below. > >> > >> What happens between the event the leader is partitioned away and > >> a new leader is elected? > >> > >> The leader may be unaware of the events and serve a read just > >> fine. > > > > As it is stated 20 lines above: > >>>>> As soon as leader appears in a situation it has not enough > >>>>> replicas > >>>>> to achieve quorum, the cluster should stop accepting any > >>>>> requests - both > >>>>> write and read. > > > > So it will not serve. > > This breaks compatibility, since now an orphan node is perfectly able > to serve reads. The cluster can't just stop doing everything, if the > quorum is lost. Stop writes - yes, since the quorum is lost anyway. But > reads do not need a quorum. > > If you say reads need a quorum, then they would need to go through WAL, > collect confirmations, and all. The reads should not be inconsistent - so that cluster will keep answering A or B for the same request. And in case we lost quorum we can't say for sure that all instances will answer the same. As we discussed it before, if leader appears in minor part of the cluster it can't issue rollback for all unconfirmed txns, since the majority will re-elect leader who will collect quorum for them. Means, we will appear is a state that cluster split in two. So the minor part should stop. Am I wrong here? > > >> So at least you can't say the leader shouldn't be serving reads > >> without quorum - because the only way to achieve it is to collect > >> a quorum of responses to reads as well. > > > > The leader lost connection to the (N-Q)+1 repllicas out of the N in > > cluster with a quorum of Q == it stops serving anything. So the quorum > > criteria is there: no quorum - no reads. > > Connection count tells nothing. Network connectivity is not a reliable > source of information. Only messages and persistent data are reliable > (to certain extent). Well, persistent data can't help obtain quorum if there's no connection to the replicas who should contribute to quorum. Correct me, if I'm wrong: in case no quorum available we can't garantee that the data is stored on at least <quorum> number of servers. Means - cluster is not operable. > > >>>>> The reason for this is that replication of transactions > >>>>> can achieve quorum on replicas not visible to the leader. On the other > >>>>> hand, leader can't achieve quorum with available minority. Leader has to > >>>>> report the state and wait for human intervention. There's an option to > >>>>> ask leader to rollback to the latest transaction that has quorum: leader > >>>>> issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > >>>>> is of the first transaction in the leader's undo log. The rollback > >>>>> message replicated to the available cluster will put it in a consistent > >>>>> state. After that configuration of the cluster can be updated to > >>>>> available quorum and leader can be switched back to write mode. > >>>> > >>>> As you should be able to conclude from restart scenario, it is > >>>> possible a replica has the record in *confirmed* state but the > >>>> leader has it in pending state. The replica will not be able to > >>>> roll back then. Do you suggest the replica should abort if it > >>>> can't rollback? This may lead to an avalanche of rejoins on leader > >>>> restart, bringing performance to a halt. > >>> > >>> No, I declare replica with biggest LSN as a new shining leader. More > >>> than that, new leader can (so far it will be by default) finalize the > >>> former leader life's work by replicating txns and appropriate confirms. > >> > >> Right, this also assumes the restart is noticed, so it follows the > >> same logic. > > > > How a restart can be unnoticed, if it causes disconnection? > > Disconnection has nothing to do with restart. The coordinator itself may > restart. Or it may loose connection to the leader temporarily. Or the > leader may loose it without any restarts. But how we detect it right now in Tarantool? Is there any machinery? I suppose we can simply rely on the same at least to test the minimal - and 'normally operating' - first approach to the problem. So, thank you for all comments and please, find my updated RFC below. Sergos. --- * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a Tarantool cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is the absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatibility and ease of cluster orchestration. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | |---Confirm--->| | | | | | | [TXN undo log | [TXN undo log | | destroyed] | destroyed] | | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's [LEADER_ID, LSN] and the confirm has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a TXN application success to the leader via the IPROTO explicitly to allow leader to collect the quorum for the TXN. In case of application failure the replica has to disconnect from the replication the same way as it is done now. The replica also has to report its disconnection to the orchestrator. Further actions require human intervention, since failure means either technical problem (such as not enough space for WAL) that has to be resolved or an inconsistent state that requires rejoin. As soon as leader appears in a situation it has not enough replicas to achieve quorum, the cluster should stop accepting any requests - both write and read. The reason for this is that replication of transactions can achieve quorum on replicas not visible to the leader. On the other hand, leader can't achieve quorum with available minority. Leader has to report the state and wait for human intervention. There's an option to ask leader to rollback to the latest transaction that has quorum: leader issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN is of the first transaction in the leader's undo log. The rollback message replicated to the available cluster will put it in a consistent state. After that configuration of the cluster can be updated to available quorum and leader can be switched back to write mode. ### Leader role assignment. To assign a leader role to an instance the following should be performed: 1. among all available instances pick the one that has the biggest vclock element of the former leader ID; an arbitrary istance can be selected in case it is first time the leader is assigned 2. the leader should assure that number of available instances in the cluster is enough to achieve the quorum and proceed to step 3, otherwise the leader should report the situation of incomplete quorum, as in the last paragraph of previous section 3. the selected instance has to take the responsibility to replicate former leader entries from its WAL, obtainig quorum and commit confirm messages referring to [FORMER_LEADER_ID, LSN] in its WAL, replicating to the cluster, after that it can start adding its own entries into the WAL ### Recovery and failover. Tarantool instance during reading WAL should postpone the undo log deletion until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep undo log for all transactions that are waiting for a confirm entry until the role of the instance is set. If this instance will be assigned a leader role then all transactions that have no corresponding confirm message should be confirmed (see the leader role assignment). In case there's not enough replicas to set up a quorum the cluster can be switched into a read-only mode. Note, this can't be done by default since some of transactions can have confirmed state. It is up to human intervention to force rollback of all transactions that have no confirm and to put the cluster into a consistent state. In case the instance will be assigned a replica role, it may appear in a state that it has conflicting WAL entries, in case it recovered from a leader role and some of transactions didn't replicated to the current leader. This situation should be resolved through rejoin of the instance. Consider an example below. Originally instance with ID1 was assigned a Leader role and the cluster had 2 replicas with quorum set to 2. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Leader | Replica 1 | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | | | +---------------------+---------------------+---------------------+ | Tx6 | | | +---------------------+---------------------+---------------------+ | Tx7 | | | +---------------------+---------------------+---------------------+ ``` Suppose at this moment the ID1 instance crashes. Then the ID2 instance should be assigned a leader role since its ID1 LSN is the biggest. Then this new leader will deliver its WAL to all replicas. As soon as quorum for Tx4 and Tx5 will be obtained, it should write the corresponding Confirms to its WAL. Note that Tx are still uses ID1. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | (dead) | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | +---------------------+---------------------+---------------------+ | ID1 Tx6 | | | +---------------------+---------------------+---------------------+ | ID1 Tx7 | | | +---------------------+---------------------+---------------------+ ``` After rejoining ID1 will figure out the inconsistency of its WAL: the last WAL entry it has is corresponding to Tx7, while in Leader's log the last entry with ID1 is Tx5. In case the ID1's WAL contains corresponding entry then Replica 1 can stop reading WAL as soon as it hits the vclock[ID1] obtained from the current Leader. It will put the ID1 into a consistent state and it can obtain latest data via replication. The WAL should be rotated after a snapshot creation. The old WAL should be renamed so it will not be reused in the future and kept for postmortem. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Replica 1 | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | +---------------------+---------------------+---------------------+ | | ID2 Tx1 | ID2 Tx1 | +---------------------+---------------------+---------------------+ | | ID2 Tx2 | ID2 Tx2 | +---------------------+---------------------+---------------------+ ``` Although, there could be a situation that ID1's WAL begins with an LSN after the biggest available in the Leader's WAL. Either, for vinyl part of WAL can be referenced in .run files, hence can't be evicted by a simple WAL ignore. In such a case the ID1 needs a complete rejoin. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - no matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-14 20:38 ` Sergey Ostanevich @ 2020-05-20 20:59 ` Sergey Ostanevich 2020-05-25 23:41 ` Vladislav Shpilevoy 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-20 20:59 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! I've updated part of recovery and leader election. The latest version is at the bottom. Thanks, Sergos On 14 мая 23:38, Sergey Ostanevich wrote: > Hi! > > > >> Sergey, I understand that RAFT spec is big and with this spec you > > >> try to split it into manageable parts. The question is how useful > > >> is this particular piece. I'm trying to point out that "the leader > > >> should stop" is not a silver bullet - especially since each such > > >> stop may mean a rejoin of some other node. The purpose of sync > > >> replication is to provide consistency without reducing > > >> availability (i.e. make progress as long as the quorum > > >> of nodes make progress). > > > > > > I'm not sure if we're talking about the same RAFT - mine is "In Search > > > of an Understandable Consensus Algorithm (Extended Version)" from > > > Stanford as of May 2014. And it is 15 pages - including references, > > > conclusions and intro. Seems not that big. > > > > 15 pages of tightly packed theory is a big piece of data. And especially > > big, when it comes to application to a real project, with existing > > infrastructure, and all. Just my IMHO. I remember implementing SWIM - it > > is smaller than RAFT. Much smaller and simpler, and yet it took year to > > implement it, and cover all things described in the paper. > > That I won't object and this was the reason not to take the RAFT as is > and implement it in full for next 2-3 years. That's why we had the very > first part of RFC describing what it tries to address and what's not. > > > > > This is not as simple as it looks, when it comes to edge cases. This is > > why the whole sync replication frustrates me more than anything else > > before, and why I am so reluctant to doing anything with it. > > > > The RFC mostly covers the normal operation, here I agree with Kostja. But > > the normal operation is not that interesting. Failures are much more > > important. > > Definitely and I expect to follow with more functionality on top of it. > I believe it will be easier to do if the start will be as small as > possible change to the existent code base, which I also try to follow. > > > > > > Although, most of it is dedicated to the leader election itself, which > > > we intentionally put aside from this RFC. It is written in the very > > > beginning and I empasized this by explicit mentioning of it. > > > > And still there will be leader election. Even though not ours for now. > > And Tarantool should provide API and instructions so as external > > applications could follow them and do the election. > > > > Usually in RFCs we describe API. With arguments, behaviour, and all. > > That is something I believe should be done after we agree on the whole > idea, such as confirm entry in WAL for sync transactions that appeared > there earlier. Otherwise we can get very deep into the details, spending > time for API definition while the idea itself can appear wrong. > > I believe that was a common ground to start, but we immediately went to > discussion of so many details I tried to keep away before we agree on the > key parts, such as WAL consistency or quorum collection. > > > > > >> The current spec, suggesting there should be a leader stop in case > > >> of most errors, reduces availability significantly, and doesn't > > >> make external coordinator job any easier - it still has to follow to > > >> the letter the prescriptions of RAFT. > > >> > > >>> > > >>>> > > >>>>> | |----------Confirm---------->| | > > >>>> > > >>>> What happens if peers receive and maybe even write Confirm to their WALs > > >>>> but local WAL write is lost after a restart? > > >>> > > >>> Did you mean WAL write on leader as a local? Then we have a replica with > > >>> a bigger LSN for the leader ID. > > >> > > >>>> WAL is not synced, > > >>>> so we can easily lose the tail of the WAL. Tarantool will sync up > > >>>> with all replicas on restart, > > >>> > > >>> But at this point a new leader will be appointed - the old one is > > >>> restarted. Then the Confirm message will arrive to the restarted leader > > >>> through a regular replication. > > >> > > >> This assumes that restart is guaranteed to be noticed by the > > >> external coordinator and there is an election on every restart. > > > > > > Sure yes, if it restarted - then connection lost can't be unnoticed by > > > anyone, be it coordinator or cluster. > > > > Here comes another problem. Disconnect and restart have nothing to do with > > each other. The coordinator can loose connection without the peer leader > > restart. Just because it is network. Anything can happen. Moreover, while > > the coordinator does not have a connection, the leader can restart multiple > > times. > > Definitely there should be a higher level functionality to support some > sort of membership protocol, such as SWIM or RAFT itself. But > introduction of it should not affect the basic priciples we have to > agree upon. > > > > > We can't tell the coordinator rely on connectivity as a restart signal. > > > > >>>> but there will be no "Replication > > >>>> OK" messages from them, so it wouldn't know that the transaction > > >>>> is committed on them. How is this handled? We may end up with some > > >>>> replicas confirming the transaction while the leader will roll it > > >>>> back on restart. Do you suggest there is a human intervention on > > >>>> restart as well? > > >>>> > > >>>> > > >>>>> | | | | | > > >>>>> |<---TXN Ok-----| | [TXN undo log | > > >>>>> | | | destroyed] | > > >>>>> | | | | | > > >>>>> | | | |---Confirm--->| > > >>>>> | | | | | > > >>>>> ``` > > >>>>> > > >>>>> The quorum should be collected as a table for a list of transactions > > >>>>> waiting for quorum. The latest transaction that collects the quorum is > > >>>>> considered as complete, as well as all transactions prior to it, since > > >>>>> all transactions should be applied in order. Leader writes a 'confirm' > > >>>>> message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > > >>>>> the confirm has its own LSN. This confirm message is delivered to all > > >>>>> replicas through the existing replication mechanism. > > >>>>> > > >>>>> Replica should report a TXN application success to the leader via the > > >>>>> IPROTO explicitly to allow leader to collect the quorum for the TXN. > > >>>>> In case of application failure the replica has to disconnect from the > > >>>>> replication the same way as it is done now. The replica also has to > > >>>>> report its disconnection to the orchestrator. Further actions require > > >>>>> human intervention, since failure means either technical problem (such > > >>>>> as not enough space for WAL) that has to be resovled or an inconsistent > > >>>>> state that requires rejoin. > > >>>> > > >>>>> As soon as leader appears in a situation it has not enough replicas > > >>>>> to achieve quorum, the cluster should stop accepting any requests - both > > >>>>> write and read. > > >>>> > > >>>> How does *the cluster* know the state of the leader and if it > > >>>> doesn't, how it can possibly implement this? Did you mean > > >>>> the leader should stop accepting transactions here? But how can > > >>>> the leader know if it has not enough replicas during a read > > >>>> transaction, if it doesn't contact any replica to serve a read? > > >>> > > >>> I expect to have a disconnection trigger assigned to all relays so that > > >>> disconnection will cause the number of replicas decrease. The quorum > > >>> size is static, so we can stop at the very moment the number dives below. > > >> > > >> What happens between the event the leader is partitioned away and > > >> a new leader is elected? > > >> > > >> The leader may be unaware of the events and serve a read just > > >> fine. > > > > > > As it is stated 20 lines above: > > >>>>> As soon as leader appears in a situation it has not enough > > >>>>> replicas > > >>>>> to achieve quorum, the cluster should stop accepting any > > >>>>> requests - both > > >>>>> write and read. > > > > > > So it will not serve. > > > > This breaks compatibility, since now an orphan node is perfectly able > > to serve reads. The cluster can't just stop doing everything, if the > > quorum is lost. Stop writes - yes, since the quorum is lost anyway. But > > reads do not need a quorum. > > > > If you say reads need a quorum, then they would need to go through WAL, > > collect confirmations, and all. > > The reads should not be inconsistent - so that cluster will keep > answering A or B for the same request. And in case we lost quorum we > can't say for sure that all instances will answer the same. > > As we discussed it before, if leader appears in minor part of the > cluster it can't issue rollback for all unconfirmed txns, since the > majority will re-elect leader who will collect quorum for them. Means, > we will appear is a state that cluster split in two. So the minor part > should stop. Am I wrong here? > > > > > >> So at least you can't say the leader shouldn't be serving reads > > >> without quorum - because the only way to achieve it is to collect > > >> a quorum of responses to reads as well. > > > > > > The leader lost connection to the (N-Q)+1 repllicas out of the N in > > > cluster with a quorum of Q == it stops serving anything. So the quorum > > > criteria is there: no quorum - no reads. > > > > Connection count tells nothing. Network connectivity is not a reliable > > source of information. Only messages and persistent data are reliable > > (to certain extent). > > Well, persistent data can't help obtain quorum if there's no connection > to the replicas who should contribute to quorum. > Correct me, if I'm wrong: in case no quorum available we can't garantee > that the data is stored on at least <quorum> number of servers. Means - > cluster is not operable. > > > > > >>>>> The reason for this is that replication of transactions > > >>>>> can achieve quorum on replicas not visible to the leader. On the other > > >>>>> hand, leader can't achieve quorum with available minority. Leader has to > > >>>>> report the state and wait for human intervention. There's an option to > > >>>>> ask leader to rollback to the latest transaction that has quorum: leader > > >>>>> issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > > >>>>> is of the first transaction in the leader's undo log. The rollback > > >>>>> message replicated to the available cluster will put it in a consistent > > >>>>> state. After that configuration of the cluster can be updated to > > >>>>> available quorum and leader can be switched back to write mode. > > >>>> > > >>>> As you should be able to conclude from restart scenario, it is > > >>>> possible a replica has the record in *confirmed* state but the > > >>>> leader has it in pending state. The replica will not be able to > > >>>> roll back then. Do you suggest the replica should abort if it > > >>>> can't rollback? This may lead to an avalanche of rejoins on leader > > >>>> restart, bringing performance to a halt. > > >>> > > >>> No, I declare replica with biggest LSN as a new shining leader. More > > >>> than that, new leader can (so far it will be by default) finalize the > > >>> former leader life's work by replicating txns and appropriate confirms. > > >> > > >> Right, this also assumes the restart is noticed, so it follows the > > >> same logic. > > > > > > How a restart can be unnoticed, if it causes disconnection? > > > > Disconnection has nothing to do with restart. The coordinator itself may > > restart. Or it may loose connection to the leader temporarily. Or the > > leader may loose it without any restarts. > > But how we detect it right now in Tarantool? Is there any machinery? > I suppose we can simply rely on the same at least to test the minimal - > and 'normally operating' - first approach to the problem. > > > So, thank you for all comments and please, find my updated RFC below. > > Sergos. > > --- * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a Tarantool cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is the absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatibility and ease of cluster orchestration. The cluster operation is expected to be in a full-mesh topology, although the process of automated topology support is beyond this RFC. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | |---Confirm--->| | | | | | | [TXN undo log | [TXN undo log | | destroyed] | destroyed] | | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's [LEADER_ID, LSN] and the confirm has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a TXN application success to the leader via the IPROTO explicitly to allow leader to collect the quorum for the TXN. In case of application failure the replica has to disconnect from the replication the same way as it is done now. The replica also has to report its disconnection to the orchestrator. Further actions require human intervention, since failure means either technical problem (such as not enough space for WAL) that has to be resolved or an inconsistent state that requires rejoin. As soon as leader appears in a situation it has not enough replicas to achieve quorum, the cluster should stop accepting any requests - both write and read. The reason for this is that replication of transactions can achieve quorum on replicas not visible to the leader. On the other hand, leader can't achieve quorum with available minority. Leader has to report the state and wait for human intervention. There's an option to ask leader to rollback to the latest transaction that has quorum: leader issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN is of the first transaction in the leader's undo log. The rollback message replicated to the available cluster will put it in a consistent state. After that configuration of the cluster can be updated to available quorum and leader can be switched back to write mode. ### Leader role assignment. Be it a user-initiated assignment or an algorithmic one, it should use a common interface to assign the leader role. By now we implement a simplified machinery, still it should be feasible in the future to fit the algorithms, such as RAFT or proposed before box.ctl.promote. A system space \_voting can be used to replicate the voting among the cluster, this space should be writable even for a read-only instance. This space should contain a CURRENT_LEADER_ID at any time - means the current leader, can be a zero value at the start. This is needed to compare the appropriate vclock component below. All replicas should be subscribed to changes in the space and react as described below. promote(ID) - should be called from a replica with it's own ID. Writes an entry in the voting space about this ID is waiting for votes from cluster. The entry should also contain the current vclock[CURRENT_LEADER_ID] of the nominee. Upon changes in the space each replica should compare its appropriate vclock component with submitted one and append its vote to the space: AYE in case nominee's vclock is bigger or equal to the replica's one, NAY otherwise. As soon as nominee collects the quorum for being elected, it claims himself a Leader by switching in rw mode, writes CURRENT_LEADER_ID as a FORMER_LEADER_ID in the \_voting space and put its ID as a CURRENT_LEADER_ID. In case a NAY is appeared in the \_voting or a timeout predefined in box.cfg is reached, the nominee should remove it's entry from the space. The leader should assure that number of available instances in the cluster is enough to achieve the quorum and proceed to step 3, otherwise the leader should report the situation of incomplete quorum, as described in the last paragraph of previous section. The new Leader has to take the responsibility to replicate former Leader's entries from its WAL, obtain quorum and commit confirm messages referring to [FORMER_LEADER_ID, LSN] in its WAL, replicating to the cluster, after that it can start adding its own entries into the WAL. demote(ID) - should be called from the Leader instance. The Leader has to switch in ro mode and wait for its' undo log is empty. This effectively means all transactions are committed in the cluster and it is safe pass the leadership. Then it should write CURRENT_LEADER_ID as a FORMER_LEADER_ID and put CURRENT_LEADER_ID into 0. ### Recovery and failover. Tarantool instance during reading WAL should postpone the undo log deletion until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep undo log for all transactions that are waiting for a confirm entry until the role of the instance is set. If this instance will be assigned a leader role then all transactions that have no corresponding confirm message should be confirmed (see the leader role assignment). In case there's not enough replicas to set up a quorum the cluster can be switched into a read-only mode. Note, this can't be done by default since some of transactions can have confirmed state. It is up to human intervention to force rollback of all transactions that have no confirm and to put the cluster into a consistent state. In case the instance will be assigned a replica role, it may appear in a state that it has conflicting WAL entries, in case it recovered from a leader role and some of transactions didn't replicated to the current leader. This situation should be resolved through rejoin of the instance. Consider an example below. Originally instance with ID1 was assigned a Leader role and the cluster had 2 replicas with quorum set to 2. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Leader | Replica 1 | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | | | +---------------------+---------------------+---------------------+ | Tx6 | | | +---------------------+---------------------+---------------------+ | Tx7 | | | +---------------------+---------------------+---------------------+ ``` Suppose at this moment the ID1 instance crashes. Then the ID2 instance should be assigned a leader role since its ID1 LSN is the biggest. Then this new leader will deliver its WAL to all replicas. As soon as quorum for Tx4 and Tx5 will be obtained, it should write the corresponding Confirms to its WAL. Note that Tx are still uses ID1. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | (dead) | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | +---------------------+---------------------+---------------------+ | ID1 Tx6 | | | +---------------------+---------------------+---------------------+ | ID1 Tx7 | | | +---------------------+---------------------+---------------------+ ``` After rejoining ID1 will figure out the inconsistency of its WAL: the last WAL entry it has is corresponding to Tx7, while in Leader's log the last entry with ID1 is Tx5. Confirm for a Tx can only be issued after appearance of the Tx on the majoirty of replicas, hence there's a good chances that ID1 will have inconsistency in its WAL covered with undo log. So, by rolling back all excessive Txs (in the example they are Tx6 and Tx7) the ID1 can put its memtx and vynil in consistent state. At this point a snapshot can be created at ID1 with appropriate WAL rotation. The old WAL should be renamed so it will not be reused in the future and can be kept for postmortem. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Replica 1 | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | +---------------------+---------------------+---------------------+ | | ID2 Tx1 | ID2 Tx1 | +---------------------+---------------------+---------------------+ | | ID2 Tx2 | ID2 Tx2 | +---------------------+---------------------+---------------------+ ``` Although, in case undo log is not enough to cover the WAL inconsistence with the new leader, the ID1 needs a complete rejoin. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - no matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-20 20:59 ` Sergey Ostanevich @ 2020-05-25 23:41 ` Vladislav Shpilevoy 2020-05-27 21:17 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-05-25 23:41 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches Hi! Thanks for the changes! >>>>>>>> As soon as leader appears in a situation it has not enough >>>>>>>> replicas >>>>>>>> to achieve quorum, the cluster should stop accepting any >>>>>>>> requests - both >>>>>>>> write and read. >>>> >>>> So it will not serve. >>> >>> This breaks compatibility, since now an orphan node is perfectly able >>> to serve reads. The cluster can't just stop doing everything, if the >>> quorum is lost. Stop writes - yes, since the quorum is lost anyway. But >>> reads do not need a quorum. >>> >>> If you say reads need a quorum, then they would need to go through WAL, >>> collect confirmations, and all. >> >> The reads should not be inconsistent - so that cluster will keep >> answering A or B for the same request. And in case we lost quorum we >> can't say for sure that all instances will answer the same. >> >> As we discussed it before, if leader appears in minor part of the >> cluster it can't issue rollback for all unconfirmed txns, since the >> majority will re-elect leader who will collect quorum for them. Means, >> we will appear is a state that cluster split in two. So the minor part >> should stop. Am I wrong here? Yeah, kinda. As long as you allow reading from replicas, you *always* will have a time slot, when you will be able to read different data for the same key on different replicas. Even with reads going through quorum. Because it is physically impossible to make nodes A and B start answering the same data at the same time moment. To notify them about a confirm you will send network messages, they will have not the same delay, won't be processed in the same moment of time, and some of them probably won't be even delivered. The only correct way to read the same - read from one node only. From the leader. And since this is not our way, it means we can't beat the 'inconsistent' reads problems. And I don't think we should. Because if somebody needs to do 'consistent' reads, they should read from leader only. In other words, the concept of 'consistency' is highly application dependent here. If we provide a way to read from replicas, we give flexibility to choose: read from leader only and see always the same data, or read from all, and have a possibility, that requests may see different data on different replicas sometimes. > ## Detailed design > > ### Quorum commit > > The main idea behind the proposal is to reuse existent machinery as much > as possible. It will ensure the well-tested and proven functionality > across many instances in MRG and beyond is used. The transaction rollback > mechanism is in place and works for WAL write failure. If we substitute > the WAL success with a new situation which is named 'quorum' later in > this document then no changes to the machinery is needed. The same is > true for snapshot machinery that allows to create a copy of the database > in memory for the whole period of snapshot file write. Adding quorum here > also minimizes changes. > > Currently replication represented by the following scheme: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN undo log | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |<---WAL Ok----| | | > | | | | | > | [TXN undo log | | | > | destroyed] | | | > | | | | | > |<----TXN Ok----| | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN undo log | > | | | created] | > | | | | | > | | | |-----TXN----->| > | | | | | > | | | |<---WAL Ok----| > | | | | | > | | | [TXN undo log | > | | | destroyed] | > | | | | | > ``` > > To introduce the 'quorum' we have to receive confirmation from replicas > to make a decision on whether the quorum is actually present. Leader > collects necessary amount of replicas confirmation plus its own WAL > success. This state is named 'quorum' and gives leader the right to > complete the customers' request. So the picture will change to: > ``` > Customer Leader WAL(L) Replica WAL(R) > |------TXN----->| | | | > | | | | | > | [TXN undo log | | | > | created] | | | > | | | | | > | |-----TXN----->| | | > | | | | | > | |-------Replicate TXN------->| | > | | | | | > | | | [TXN undo log | > | |<---WAL Ok----| created] | > | | | | | > | [Waiting | |-----TXN----->| > | of a quorum] | | | > | | | |<---WAL Ok----| > | | | | | > | |<------Replication Ok-------| | > | | | | | > | [Quorum | | | > | achieved] | | | > | | | | | > | |---Confirm--->| | | > | | | | | > | |----------Confirm---------->| | > | | | | | > |<---TXN Ok-----| | |---Confirm--->| > | | | | | > | [TXN undo log | [TXN undo log | > | destroyed] | destroyed] | > | | | | | > ``` > > The quorum should be collected as a table for a list of transactions > waiting for quorum. The latest transaction that collects the quorum is > considered as complete, as well as all transactions prior to it, since > all transactions should be applied in order. Leader writes a 'confirm' > message to the WAL that refers to the transaction's [LEADER_ID, LSN] and > the confirm has its own LSN. This confirm message is delivered to all > replicas through the existing replication mechanism. > > Replica should report a TXN application success to the leader via the > IPROTO explicitly to allow leader to collect the quorum for the TXN. > In case of application failure the replica has to disconnect from the > replication the same way as it is done now. The replica also has to > report its disconnection to the orchestrator. Further actions require > human intervention, since failure means either technical problem (such > as not enough space for WAL) that has to be resolved or an inconsistent > state that requires rejoin. I don't think a replica should report disconnection. Problem of disconnection is that it leads to loosing the connection. So it may be not able to connect to the orchestrator. Also it would be strange for tarantool to depend on some external service, to which it should report. This looks like the orchestrator's business how will it determine connectivity. Replica has nothing to do with it from its side. > As soon as leader appears in a situation it has not enough replicas > to achieve quorum, the cluster should stop accepting any requests - both > write and read. The moment of not having enough replicas can't be determined properly. You may loose connection to replicas (they could be powered off), but TCP won't see that, and the node will continue working. The failure will be discovered only when a 'write' request will try to collect a quorum, or after a timeout will pass on not delivering heartbeats. During this time reads will be served. And there is no way to prevent them except collecting a quorum on that. See my first comment in this email for more details. On the summary: we can't stop accepting read requests. Btw, what to do with reads, which were *in-progress*, when the quorum was lost? Such as long vinyl reads. > The reason for this is that replication of transactions > can achieve quorum on replicas not visible to the leader. On the other > hand, leader can't achieve quorum with available minority. Leader has to > report the state and wait for human intervention. Yeah, but if the leader couldn't achieve a quorum on some transactions, they are not visible (assuming MVCC will work properly). So they can't be read anyway. And if a leader answered an error, it does not mean that the transaction wasn't replicated on the majority, as we discussed at some meeting, I don't already remember when. So here read allowance also works fine - not having some data visible and getting error at a sync transaction does not mean it is not committed. A user should be aware of that. > There's an option to > ask leader to rollback to the latest transaction that has quorum: leader > issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN > is of the first transaction in the leader's undo log. The rollback > message replicated to the available cluster will put it in a consistent > state. After that configuration of the cluster can be updated to > available quorum and leader can be switched back to write mode. > > ### Leader role assignment. > > Be it a user-initiated assignment or an algorithmic one, it should use > a common interface to assign the leader role. By now we implement a > simplified machinery, still it should be feasible in the future to fit > the algorithms, such as RAFT or proposed before box.ctl.promote. > > A system space \_voting can be used to replicate the voting among the > cluster, this space should be writable even for a read-only instance. > This space should contain a CURRENT_LEADER_ID at any time - means the > current leader, can be a zero value at the start. This is needed to > compare the appropriate vclock component below. > > All replicas should be subscribed to changes in the space and react as > described below. > > promote(ID) - should be called from a replica with it's own ID. > Writes an entry in the voting space about this ID is waiting for > votes from cluster. The entry should also contain the current > vclock[CURRENT_LEADER_ID] of the nominee. > > Upon changes in the space each replica should compare its appropriate > vclock component with submitted one and append its vote to the space: > AYE in case nominee's vclock is bigger or equal to the replica's one, > NAY otherwise. > > As soon as nominee collects the quorum for being elected, it claims > himself a Leader by switching in rw mode, writes CURRENT_LEADER_ID as > a FORMER_LEADER_ID in the \_voting space and put its ID as a > CURRENT_LEADER_ID. In case a NAY is appeared in the \_voting or a > timeout predefined in box.cfg is reached, the nominee should remove > it's entry from the space. > > The leader should assure that number of available instances in the > cluster is enough to achieve the quorum and proceed to step 3, otherwise > the leader should report the situation of incomplete quorum, as > described in the last paragraph of previous section. > > The new Leader has to take the responsibility to replicate former Leader's > entries from its WAL, obtain quorum and commit confirm messages referring > to [FORMER_LEADER_ID, LSN] in its WAL, replicating to the cluster, after > that it can start adding its own entries into the WAL. > > demote(ID) - should be called from the Leader instance. > The Leader has to switch in ro mode and wait for its' undo log is > empty. This effectively means all transactions are committed in the > cluster and it is safe pass the leadership. Then it should write > CURRENT_LEADER_ID as a FORMER_LEADER_ID and put CURRENT_LEADER_ID > into 0. This looks like box.ctl.promote() algorithm. Although I thought we decided not to implement any kind of auto election here, no? Box.ctl.promote() assumed, that it does all the steps automatically, except choosing on which node to call this function. This is what it was so complicated. It was basically raft. But yeah, as discussed verbally, this is a subject for improvement. The way I see it is that we need to give vclock based algorithm of choosing a new leader; tell how to stop replication from the old leader; allow to read vclock from replicas (basically, let the external service read box.info). Since you said you think we should not provide an API for all sync transactions rollback, it looks like no need in a special new API. But if we still want to allow to rollback all pending transactions of the old leader on a new leader (like Mons wants) then yeah, seems like we would need a new function. For example, box.ctl.sync_rollback() to rollback all pending. And box.ctl.sync_confirm() to confirm all pending. Perhaps we could add more admin-line parameters such as replica_id with which to write 'confirm/rollback' message. > ### Recovery and failover. > > Tarantool instance during reading WAL should postpone the undo log > deletion until the 'confirm' is read. In case the WAL eof is achieved, > the instance should keep undo log for all transactions that are waiting > for a confirm entry until the role of the instance is set. > > If this instance will be assigned a leader role then all transactions > that have no corresponding confirm message should be confirmed (see the > leader role assignment). > > In case there's not enough replicas to set up a quorum the cluster can > be switched into a read-only mode. Note, this can't be done by default > since some of transactions can have confirmed state. It is up to human > intervention to force rollback of all transactions that have no confirm > and to put the cluster into a consistent state. Above you said: >> As soon as leader appears in a situation it has not enough replicas >> to achieve quorum, the cluster should stop accepting any requests - both >> write and read. But here I see, that the cluster "switched into a read-only mode". So there is a contradiction. And I think it should be resolved in favor of 'read-only mode'. I explained why in the previous comments. > In case the instance will be assigned a replica role, it may appear in > a state that it has conflicting WAL entries, in case it recovered from a > leader role and some of transactions didn't replicated to the current > leader. This situation should be resolved through rejoin of the instance. > > Consider an example below. Originally instance with ID1 was assigned a > Leader role and the cluster had 2 replicas with quorum set to 2. > > ``` > +---------------------+---------------------+---------------------+ > | ID1 | ID2 | ID3 | > | Leader | Replica 1 | Replica 2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > +---------------------+---------------------+---------------------+ > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > +---------------------+---------------------+---------------------+ > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | > +---------------------+---------------------+---------------------+ > | ID1 Tx4 | ID1 Tx4 | | > +---------------------+---------------------+---------------------+ > | ID1 Tx5 | ID1 Tx5 | | > +---------------------+---------------------+---------------------+ > | ID1 Conf [ID1, Tx2] | | | > +---------------------+---------------------+---------------------+ > | Tx6 | | | > +---------------------+---------------------+---------------------+ > | Tx7 | | | > +---------------------+---------------------+---------------------+ > ``` > Suppose at this moment the ID1 instance crashes. Then the ID2 instance > should be assigned a leader role since its ID1 LSN is the biggest. > Then this new leader will deliver its WAL to all replicas. > > As soon as quorum for Tx4 and Tx5 will be obtained, it should write the > corresponding Confirms to its WAL. Note that Tx are still uses ID1. > ``` > +---------------------+---------------------+---------------------+ > | ID1 | ID2 | ID3 | > | (dead) | Leader | Replica 2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > +---------------------+---------------------+---------------------+ > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > +---------------------+---------------------+---------------------+ > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > +---------------------+---------------------+---------------------+ > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > +---------------------+---------------------+---------------------+ > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > +---------------------+---------------------+---------------------+ > | ID1 Conf [ID1, Tx2] | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | Id1 -> ID1 (typo) > +---------------------+---------------------+---------------------+ > | ID1 Tx6 | | | > +---------------------+---------------------+---------------------+ > | ID1 Tx7 | | | > +---------------------+---------------------+---------------------+ > ``` > After rejoining ID1 will figure out the inconsistency of its WAL: the > last WAL entry it has is corresponding to Tx7, while in Leader's log the > last entry with ID1 is Tx5. Confirm for a Tx can only be issued after > appearance of the Tx on the majoirty of replicas, hence there's a good > chances that ID1 will have inconsistency in its WAL covered with undo > log. So, by rolling back all excessive Txs (in the example they are Tx6 > and Tx7) the ID1 can put its memtx and vynil in consistent state. Yeah, but the problem is that the node1 has vclock[ID1] == 'Conf [ID1, Tx2]'. This row can't be rolled back. So looks like node1 needs a rejoin. > At this point a snapshot can be created at ID1 with appropriate WAL > rotation. The old WAL should be renamed so it will not be reused in the > future and can be kept for postmortem. > ``` > +---------------------+---------------------+---------------------+ > | ID1 | ID2 | ID3 | > | Replica 1 | Leader | Replica 2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > +---------------------+---------------------+---------------------+ > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > +---------------------+---------------------+---------------------+ > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > +---------------------+---------------------+---------------------+ > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > +---------------------+---------------------+---------------------+ > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > +---------------------+---------------------+---------------------+ > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > +---------------------+---------------------+---------------------+ > | | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | > +---------------------+---------------------+---------------------+ > | | ID2 Tx1 | ID2 Tx1 | > +---------------------+---------------------+---------------------+ > | | ID2 Tx2 | ID2 Tx2 | > +---------------------+---------------------+---------------------+ > ``` > Although, in case undo log is not enough to cover the WAL inconsistence > with the new leader, the ID1 needs a complete rejoin. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-25 23:41 ` Vladislav Shpilevoy @ 2020-05-27 21:17 ` Sergey Ostanevich 2020-06-09 16:19 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-27 21:17 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! Thanks for review! Some comments below. On 26 мая 01:41, Vladislav Shpilevoy wrote: > >> > >> The reads should not be inconsistent - so that cluster will keep > >> answering A or B for the same request. And in case we lost quorum we > >> can't say for sure that all instances will answer the same. > >> > >> As we discussed it before, if leader appears in minor part of the > >> cluster it can't issue rollback for all unconfirmed txns, since the > >> majority will re-elect leader who will collect quorum for them. Means, > >> we will appear is a state that cluster split in two. So the minor part > >> should stop. Am I wrong here? > > Yeah, kinda. As long as you allow reading from replicas, you *always* will > have a time slot, when you will be able to read different data for the > same key on different replicas. Even with reads going through quorum. > > Because it is physically impossible to make nodes A and B start answering > the same data at the same time moment. To notify them about a confirm you will > send network messages, they will have not the same delay, won't be processed > in the same moment of time, and some of them probably won't be even delivered. > > The only correct way to read the same - read from one node only. From the > leader. And since this is not our way, it means we can't beat the 'inconsistent' > reads problems. And I don't think we should. Because if somebody needs to do > 'consistent' reads, they should read from leader only. > > In other words, the concept of 'consistency' is highly application dependent > here. If we provide a way to read from replicas, we give flexibility to choose: > read from leader only and see always the same data, or read from all, and have > a possibility, that requests may see different data on different replicas > sometimes. So, it looks like we will follow the current approach: if quorum can't be achieved, cluster appears in r/o mode. Objections? > > > > Replica should report a TXN application success to the leader via the > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > In case of application failure the replica has to disconnect from the > > replication the same way as it is done now. The replica also has to > > report its disconnection to the orchestrator. Further actions require > > human intervention, since failure means either technical problem (such > > as not enough space for WAL) that has to be resolved or an inconsistent > > state that requires rejoin. > > I don't think a replica should report disconnection. Problem of > disconnection is that it leads to loosing the connection. So it may be > not able to connect to the orchestrator. Also it would be strange for > tarantool to depend on some external service, to which it should report. > This looks like the orchestrator's business how will it determine > connectivity. Replica has nothing to do with it from its side. External service is something I expect to be useful for the first part of implementation - the quorum part. Definitely, we will move onward to achieve some automation in leader election and failover. I just don't expect this to be part of this RFC. Anyways, orchestrator has to ask replica to figure out the connectivity between replica and leader. > > > As soon as leader appears in a situation it has not enough replicas > > to achieve quorum, the cluster should stop accepting any requests - both > > write and read. > > The moment of not having enough replicas can't be determined properly. > You may loose connection to replicas (they could be powered off), but > TCP won't see that, and the node will continue working. The failure will > be discovered only when a 'write' request will try to collect a quorum, > or after a timeout will pass on not delivering heartbeats. During this > time reads will be served. And there is no way to prevent them except > collecting a quorum on that. See my first comment in this email for more > details. > > On the summary: we can't stop accepting read requests. > > Btw, what to do with reads, which were *in-progress*, when the quorum > was lost? Such as long vinyl reads. But the quorum was in place at the start of it? Then according to transaction manager behavior only older version data will be available for read - means data that collected quorum. > > > The reason for this is that replication of transactions > > can achieve quorum on replicas not visible to the leader. On the other > > hand, leader can't achieve quorum with available minority. Leader has to > > report the state and wait for human intervention. > > Yeah, but if the leader couldn't achieve a quorum on some transactions, > they are not visible (assuming MVCC will work properly). So they can't > be read anyway. And if a leader answered an error, it does not mean that > the transaction wasn't replicated on the majority, as we discussed at some > meeting, I don't already remember when. So here read allowance also works > fine - not having some data visible and getting error at a sync transaction > does not mean it is not committed. A user should be aware of that. True, we discussed that we should guarantee only that if we answered 'Ok' then data is present in quorum number of instances. [...] > > demote(ID) - should be called from the Leader instance. > > The Leader has to switch in ro mode and wait for its' undo log is > > empty. This effectively means all transactions are committed in the > > cluster and it is safe pass the leadership. Then it should write > > CURRENT_LEADER_ID as a FORMER_LEADER_ID and put CURRENT_LEADER_ID > > into 0. > > This looks like box.ctl.promote() algorithm. Although I thought we decided > not to implement any kind of auto election here, no? Box.ctl.promote() > assumed, that it does all the steps automatically, except choosing on which > node to call this function. This is what it was so complicated. It was > basically raft. > > But yeah, as discussed verbally, this is a subject for improvement. I personally would like to postpone the algorithm should be postponed for the next stage (Q3-Q4) but now we should not mess up too much to revamp. Hence, we have to elaborate the internals - such as _voting table I mentioned. Even with introduction of terms for each leader - as in RAFT for example - we still can keep it in a replicated space, isn't it? > > The way I see it is that we need to give vclock based algorithm of choosing > a new leader; tell how to stop replication from the old leader; allow to > read vclock from replicas (basically, let the external service read box.info). That's the #1 for me by now: how a read-only replica can quit listening to a demoted leader, which can be not aware of its demotion? Still, for efficiency it should be done w/o disconnection. > > Since you said you think we should not provide an API for all sync transactions > rollback, it looks like no need in a special new API. But if we still want > to allow to rollback all pending transactions of the old leader on a new leader > (like Mons wants) then yeah, seems like we would need a new function. For example, > box.ctl.sync_rollback() to rollback all pending. And box.ctl.sync_confirm() to > confirm all pending. Perhaps we could add more admin-line parameters such as > replica_id with which to write 'confirm/rollback' message. I believe it's a good point to keep two approaches and perhaps set one of the two in configuration. This should resolve the issue with 'the rest of the cluster confirms old leader's transactions and because of it leader can't rollback'. > > > ### Recovery and failover. > > > > Tarantool instance during reading WAL should postpone the undo log > > deletion until the 'confirm' is read. In case the WAL eof is achieved, > > the instance should keep undo log for all transactions that are waiting > > for a confirm entry until the role of the instance is set. > > > > If this instance will be assigned a leader role then all transactions > > that have no corresponding confirm message should be confirmed (see the > > leader role assignment). > > > > In case there's not enough replicas to set up a quorum the cluster can > > be switched into a read-only mode. Note, this can't be done by default > > since some of transactions can have confirmed state. It is up to human > > intervention to force rollback of all transactions that have no confirm > > and to put the cluster into a consistent state. > > Above you said: > > >> As soon as leader appears in a situation it has not enough replicas > >> to achieve quorum, the cluster should stop accepting any requests - both > >> write and read. > > But here I see, that the cluster "switched into a read-only mode". So there > is a contradiction. And I think it should be resolved in favor of > 'read-only mode'. I explained why in the previous comments. My bad, I was moving around this problem already and tend to allow r/o. Will update. > > > In case the instance will be assigned a replica role, it may appear in > > a state that it has conflicting WAL entries, in case it recovered from a > > leader role and some of transactions didn't replicated to the current > > leader. This situation should be resolved through rejoin of the instance. > > > > Consider an example below. Originally instance with ID1 was assigned a > > Leader role and the cluster had 2 replicas with quorum set to 2. > > > > ``` > > +---------------------+---------------------+---------------------+ > > | ID1 | ID2 | ID3 | > > | Leader | Replica 1 | Replica 2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > +---------------------+---------------------+---------------------+ > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx4 | ID1 Tx4 | | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx5 | ID1 Tx5 | | > > +---------------------+---------------------+---------------------+ > > | ID1 Conf [ID1, Tx2] | | | > > +---------------------+---------------------+---------------------+ > > | Tx6 | | | > > +---------------------+---------------------+---------------------+ > > | Tx7 | | | > > +---------------------+---------------------+---------------------+ > > ``` > > Suppose at this moment the ID1 instance crashes. Then the ID2 instance > > should be assigned a leader role since its ID1 LSN is the biggest. > > Then this new leader will deliver its WAL to all replicas. > > > > As soon as quorum for Tx4 and Tx5 will be obtained, it should write the > > corresponding Confirms to its WAL. Note that Tx are still uses ID1. > > ``` > > +---------------------+---------------------+---------------------+ > > | ID1 | ID2 | ID3 | > > | (dead) | Leader | Replica 2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > +---------------------+---------------------+---------------------+ > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > > +---------------------+---------------------+---------------------+ > > | ID1 Conf [ID1, Tx2] | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | > > Id1 -> ID1 (typo) Thanks! > > > +---------------------+---------------------+---------------------+ > > | ID1 Tx6 | | | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx7 | | | > > +---------------------+---------------------+---------------------+ > > ``` > > After rejoining ID1 will figure out the inconsistency of its WAL: the > > last WAL entry it has is corresponding to Tx7, while in Leader's log the > > last entry with ID1 is Tx5. Confirm for a Tx can only be issued after > > appearance of the Tx on the majoirty of replicas, hence there's a good > > chances that ID1 will have inconsistency in its WAL covered with undo > > log. So, by rolling back all excessive Txs (in the example they are Tx6 > > and Tx7) the ID1 can put its memtx and vynil in consistent state. > > Yeah, but the problem is that the node1 has vclock[ID1] == 'Conf [ID1, Tx2]'. > This row can't be rolled back. So looks like node1 needs a rejoin. Confirm message is equivalent to a NOP - @sergepetrenko apparently does implementation exactly this way. So there's no need to roll it back in an engine, rather perform the xlog rotation before it. > > > At this point a snapshot can be created at ID1 with appropriate WAL > > rotation. The old WAL should be renamed so it will not be reused in the > > future and can be kept for postmortem. > > ``` > > +---------------------+---------------------+---------------------+ > > | ID1 | ID2 | ID3 | > > | Replica 1 | Leader | Replica 2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > +---------------------+---------------------+---------------------+ > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > > +---------------------+---------------------+---------------------+ > > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > > +---------------------+---------------------+---------------------+ > > | | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | > > +---------------------+---------------------+---------------------+ > > | | ID2 Tx1 | ID2 Tx1 | > > +---------------------+---------------------+---------------------+ > > | | ID2 Tx2 | ID2 Tx2 | > > +---------------------+---------------------+---------------------+ > > ``` > > Although, in case undo log is not enough to cover the WAL inconsistence > > with the new leader, the ID1 needs a complete rejoin. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-27 21:17 ` Sergey Ostanevich @ 2020-06-09 16:19 ` Sergey Ostanevich 2020-06-11 15:17 ` Vladislav Shpilevoy 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-06-09 16:19 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! Please, take a look at the latest changes, which include timeouts for quorum collection and the heartbeat for ensure the leader is alive. regards, Sergos * **Status**: In progress * **Start date**: 31-03-2020 * **Authors**: Sergey Ostanevich @sergos \<sergos@tarantool.org\> * **Issues**: https://github.com/tarantool/tarantool/issues/4842 ## Summary The aim of this RFC is to address the following list of problems formulated at MRG planning meeting: - protocol backward compatibility to enable cluster upgrade w/o downtime - consistency of data on replica and leader - switch from leader to replica without data loss - up to date replicas to run read-only requests - ability to switch async replicas into sync ones and vice versa - guarantee of rollback on leader and sync replicas - simplicity of cluster orchestration What this RFC is not: - high availability (HA) solution with automated failover, roles assignments an so on - master-master configuration support ## Background and motivation There are number of known implementation of consistent data presence in a Tarantool cluster. They can be commonly named as "wait for LSN" technique. The biggest issue with this technique is the absence of rollback guarantees at replica in case of transaction failure on one master or some of the replicas in the cluster. To provide such capabilities a new functionality should be introduced in Tarantool core, with requirements mentioned before - backward compatibility and ease of cluster orchestration. The cluster operation is expected to be in a full-mesh topology, although the process of automated topology support is beyond this RFC. ## Detailed design ### Quorum commit The main idea behind the proposal is to reuse existent machinery as much as possible. It will ensure the well-tested and proven functionality across many instances in MRG and beyond is used. The transaction rollback mechanism is in place and works for WAL write failure. If we substitute the WAL success with a new situation which is named 'quorum' later in this document then no changes to the machinery is needed. The same is true for snapshot machinery that allows to create a copy of the database in memory for the whole period of snapshot file write. Adding quorum here also minimizes changes. Currently replication represented by the following scheme: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | | | |<----TXN Ok----| | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |<---WAL Ok----| | | | | | | | | [TXN undo log | | | | destroyed] | | | | | | ``` To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader collects necessary amount of replicas confirmation plus its own WAL success. This state is named 'quorum' and gives leader the right to complete the customers' request. So the picture will change to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | | | | | | | [TXN undo log | | | | created] | | | | | | | | | |-----TXN----->| | | | | | | | | |-------Replicate TXN------->| | | | | | | | | | [TXN undo log | | |<---WAL Ok----| created] | | | | | | | [Waiting | |-----TXN----->| | of a quorum] | | | | | | |<---WAL Ok----| | | | | | | |<------Replication Ok-------| | | | | | | | [Quorum | | | | achieved] | | | | | | | | | |---Confirm--->| | | | | | | | | |----------Confirm---------->| | | | | | | |<---TXN Ok-----| | |---Confirm--->| | | | | | | [TXN undo log | [TXN undo log | | destroyed] | destroyed] | | | | | | ``` The quorum should be collected as a table for a list of transactions waiting for quorum. The latest transaction that collects the quorum is considered as complete, as well as all transactions prior to it, since all transactions should be applied in order. Leader writes a 'confirm' message to the WAL that refers to the transaction's [LEADER_ID, LSN] and the confirm has its own LSN. This confirm message is delivered to all replicas through the existing replication mechanism. Replica should report a TXN application success to the leader via the IPROTO explicitly to allow leader to collect the quorum for the TXN. In case of application failure the replica has to disconnect from the replication the same way as it is done now. The replica also has to report its disconnection to the orchestrator. Further actions require human intervention, since failure means either technical problem (such as not enough space for WAL) that has to be resolved or an inconsistent state that requires rejoin. Currently Tarantool provides no protection from dirty read from the memtx during the TXN write into the WAL. So, there is a chance of TXN can fail to be written to the WAL, while some read requests can report success of TXN. In this RFC we make no attempt to resolve the dirty read, so it should be addressed by user code. Although we plan to introduce an MVCC machinery similar to available in vinyl engnie which will resolve the dirty read problem. ### Connection liveness There is a timeout-based mechanism in Tarantool that controls the asynchronous replication, which uses the following config: ``` * replication_connect_timeout = 4 * replication_sync_lag = 10 * replication_sync_timeout = 300 * replication_timeout = 1 ``` For backward compatibility and to differentiate the async replication we should augment the configuration with the following: ``` * synchro_replication_heartbeat = 4 * synchro_replication_quorum_timeout = 4 ``` Leader should send a heartbeat every synchro_replication_heartbeat if there were no messages sent. Replicas should respond to the heartbeat just the same way as they do it now. As soon as Leader has no response for another heartbeat interval, it should consider the replica is lost. As soon as leader appears in a situation it has not enough replicas to achieve quorum, it should stop accepting write requests. There's an option for leader to rollback to the latest transaction that has quorum: leader issues a 'rollback' message referring to the [LEADER_ID, LSN] where LSN is of the first transaction in the leader's undo log. The rollback message replicated to the available cluster will put it in a consistent state. After that configuration of the cluster can be updated to a new available quorum and leader can be switched back to write mode. During the quorum collection it can happen that some of replicas become unavailable due to some reason, so leader should wait at most for synchro_replication_quorum_timeout after which it issues a Rollback pointing to the oldest TXN in the waiting list. ### Leader role assignment. Be it a user-initiated assignment or an algorithmic one, it should use a common interface to assign the leader role. By now we implement a simplified machinery, still it should be feasible in the future to fit the algorithms, such as RAFT or proposed before box.ctl.promote. A system space \_voting can be used to replicate the voting among the cluster, this space should be writable even for a read-only instance. This space should contain a CURRENT_LEADER_ID at any time - means the current leader, can be a zero value at the start. This is needed to compare the appropriate vclock component below. All replicas should be subscribed to changes in the space and react as described below. promote(ID) - should be called from a replica with it's own ID. Writes an entry in the voting space about this ID is waiting for votes from cluster. The entry should also contain the current vclock[CURRENT_LEADER_ID] of the nominee. Upon changes in the space each replica should compare its appropriate vclock component with submitted one and append its vote to the space: AYE in case nominee's vclock is bigger or equal to the replica's one, NAY otherwise. As soon as nominee collects the quorum for being elected, it claims himself a Leader by switching in rw mode, writes CURRENT_LEADER_ID as a FORMER_LEADER_ID in the \_voting space and put its ID as a CURRENT_LEADER_ID. In case a NAY is appeared in the \_voting or a timeout predefined in box.cfg is reached, the nominee should remove it's entry from the space. The leader should assure that number of available instances in the cluster is enough to achieve the quorum and proceed to step 3, otherwise the leader should report the situation of incomplete quorum, as described in the last paragraph of previous section. The new Leader has to take the responsibility to replicate former Leader's entries from its WAL, obtain quorum and commit confirm messages referring to [FORMER_LEADER_ID, LSN] in its WAL, replicating to the cluster, after that it can start adding its own entries into the WAL. demote(ID) - should be called from the Leader instance. The Leader has to switch in ro mode and wait for its' undo log is empty. This effectively means all transactions are committed in the cluster and it is safe pass the leadership. Then it should write CURRENT_LEADER_ID as a FORMER_LEADER_ID and put CURRENT_LEADER_ID into 0. ### Recovery and failover. Tarantool instance during reading WAL should postpone the undo log deletion until the 'confirm' is read. In case the WAL eof is achieved, the instance should keep undo log for all transactions that are waiting for a confirm entry until the role of the instance is set. If this instance will be assigned a leader role then all transactions that have no corresponding confirm message should be confirmed (see the leader role assignment). In case there's not enough replicas to set up a quorum the cluster can be switched into a read-only mode. Note, this can't be done by default since some of transactions can have confirmed state. It is up to human intervention to force rollback of all transactions that have no confirm and to put the cluster into a consistent state. In case the instance will be assigned a replica role, it may appear in a state that it has conflicting WAL entries, in case it recovered from a leader role and some of transactions didn't replicated to the current leader. This situation should be resolved through rejoin of the instance. Consider an example below. Originally instance with ID1 was assigned a Leader role and the cluster had 2 replicas with quorum set to 2. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Leader | Replica 1 | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | | | +---------------------+---------------------+---------------------+ | Tx6 | | | +---------------------+---------------------+---------------------+ | Tx7 | | | +---------------------+---------------------+---------------------+ ``` Suppose at this moment the ID1 instance crashes. Then the ID2 instance should be assigned a leader role since its ID1 LSN is the biggest. Then this new leader will deliver its WAL to all replicas. As soon as quorum for Tx4 and Tx5 will be obtained, it should write the corresponding Confirms to its WAL. Note that Tx are still uses ID1. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | (dead) | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | ID2 Conf [ID1, Tx5] | ID2 Conf [ID1, Tx5] | +---------------------+---------------------+---------------------+ | ID1 Tx6 | | | +---------------------+---------------------+---------------------+ | ID1 Tx7 | | | +---------------------+---------------------+---------------------+ ``` After rejoining ID1 will figure out the inconsistency of its WAL: the last WAL entry it has is corresponding to Tx7, while in Leader's log the last entry with ID1 is Tx5. Confirm for a Tx can only be issued after appearance of the Tx on the majoirty of replicas, hence there's a good chances that ID1 will have inconsistency in its WAL covered with undo log. So, by rolling back all excessive Txs (in the example they are Tx6 and Tx7) the ID1 can put its memtx and vynil in consistent state. At this point a snapshot can be created at ID1 with appropriate WAL rotation. The old WAL should be renamed so it will not be reused in the future and can be kept for postmortem. ``` +---------------------+---------------------+---------------------+ | ID1 | ID2 | ID3 | | Replica 1 | Leader | Replica 2 | +---------------------+---------------------+---------------------+ | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | +---------------------+---------------------+---------------------+ | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | +---------------------+---------------------+---------------------+ | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | +---------------------+---------------------+---------------------+ | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | +---------------------+---------------------+---------------------+ | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | +---------------------+---------------------+---------------------+ | | ID2 Conf [ID1, Tx5] | ID2 Conf [ID1, Tx5] | +---------------------+---------------------+---------------------+ | | ID2 Tx1 | ID2 Tx1 | +---------------------+---------------------+---------------------+ | | ID2 Tx2 | ID2 Tx2 | +---------------------+---------------------+---------------------+ ``` Although, in case undo log is not enough to cover the WAL inconsistence with the new leader, the ID1 needs a complete rejoin. ### Snapshot generation. We also can reuse current machinery of snapshot generation. Upon receiving a request to create a snapshot an instance should request a readview for the current commit operation. Although start of the snapshot generation should be postponed until this commit operation receives its confirmation. In case operation is rolled back, the snapshot generation should be aborted and restarted using current transaction after rollback is complete. After snapshot is created the WAL should start from the first operation that follows the commit operation snapshot is generated for. That means WAL will contain 'confirm' messages that refer to transactions that are not present in the WAL. Apparently, we have to allow this for the case 'confirm' refers to a transaction with LSN less than the first entry in the WAL. In case master appears unavailable a replica still have to be able to create a snapshot. Replica can perform rollback for all transactions that are not confirmed and claim its LSN as the latest confirmed txn. Then it can create a snapshot in a regular way and start with blank xlog file. All rolled back transactions will appear through the regular replication in case master reappears later on. ### Asynchronous replication. Along with synchronous replicas the cluster can contain asynchronous replicas. That means async replica doesn't reply to the leader with errors since they're not contributing into quorum. Still, async replicas have to follow the new WAL operation, such as keep rollback info until 'quorum' message is received. This is essential for the case of 'rollback' message appearance in the WAL. This message assumes replica is able to perform all necessary rollback by itself. Cluster information should contain explicit notification of each replica operation mode. ### Synchronous replication enabling. Synchronous operation can be required for a set of spaces in the data scheme. That means only transactions that contain data modification for these spaces should require quorum. Such transactions named synchronous. As soon as last operation of synchronous transaction appeared in leader's WAL, it will cause all following transactions - no matter if they are synchronous or not - wait for the quorum. In case quorum is not achieved the 'rollback' operation will cause rollback of all transactions after the synchronous one. It will ensure the consistent state of the data both on leader and replicas. In case user doesn't require synchronous operation for any space then no changes to the WAL generation and replication will appear. Cluster description should contain explicit attribute for each replica to denote it participates in synchronous activities. Also the description should contain criterion on how many replicas responses are needed to achieve the quorum. ## Rationale and alternatives There is an implementation of synchronous replication as part of gh-980 activities, still it is not in a state to get into the product. More than that it intentionally breaks backward compatibility which is a prerequisite for this proposal. On 28 мая 00:17, Sergey Ostanevich wrote: > Hi! > > Thanks for review! > > Some comments below. > On 26 мая 01:41, Vladislav Shpilevoy wrote: > > >> > > >> The reads should not be inconsistent - so that cluster will keep > > >> answering A or B for the same request. And in case we lost quorum we > > >> can't say for sure that all instances will answer the same. > > >> > > >> As we discussed it before, if leader appears in minor part of the > > >> cluster it can't issue rollback for all unconfirmed txns, since the > > >> majority will re-elect leader who will collect quorum for them. Means, > > >> we will appear is a state that cluster split in two. So the minor part > > >> should stop. Am I wrong here? > > > > Yeah, kinda. As long as you allow reading from replicas, you *always* will > > have a time slot, when you will be able to read different data for the > > same key on different replicas. Even with reads going through quorum. > > > > Because it is physically impossible to make nodes A and B start answering > > the same data at the same time moment. To notify them about a confirm you will > > send network messages, they will have not the same delay, won't be processed > > in the same moment of time, and some of them probably won't be even delivered. > > > > The only correct way to read the same - read from one node only. From the > > leader. And since this is not our way, it means we can't beat the 'inconsistent' > > reads problems. And I don't think we should. Because if somebody needs to do > > 'consistent' reads, they should read from leader only. > > > > In other words, the concept of 'consistency' is highly application dependent > > here. If we provide a way to read from replicas, we give flexibility to choose: > > read from leader only and see always the same data, or read from all, and have > > a possibility, that requests may see different data on different replicas > > sometimes. > > So, it looks like we will follow the current approach: if quorum can't > be achieved, cluster appears in r/o mode. Objections? > > > > > > > Replica should report a TXN application success to the leader via the > > > IPROTO explicitly to allow leader to collect the quorum for the TXN. > > > In case of application failure the replica has to disconnect from the > > > replication the same way as it is done now. The replica also has to > > > report its disconnection to the orchestrator. Further actions require > > > human intervention, since failure means either technical problem (such > > > as not enough space for WAL) that has to be resolved or an inconsistent > > > state that requires rejoin. > > > > I don't think a replica should report disconnection. Problem of > > disconnection is that it leads to loosing the connection. So it may be > > not able to connect to the orchestrator. Also it would be strange for > > tarantool to depend on some external service, to which it should report. > > This looks like the orchestrator's business how will it determine > > connectivity. Replica has nothing to do with it from its side. > > External service is something I expect to be useful for the first part > of implementation - the quorum part. Definitely, we will move onward to > achieve some automation in leader election and failover. I just don't > expect this to be part of this RFC. > > Anyways, orchestrator has to ask replica to figure out the connectivity > between replica and leader. > > > > > > As soon as leader appears in a situation it has not enough replicas > > > to achieve quorum, the cluster should stop accepting any requests - both > > > write and read. > > > > The moment of not having enough replicas can't be determined properly. > > You may loose connection to replicas (they could be powered off), but > > TCP won't see that, and the node will continue working. The failure will > > be discovered only when a 'write' request will try to collect a quorum, > > or after a timeout will pass on not delivering heartbeats. During this > > time reads will be served. And there is no way to prevent them except > > collecting a quorum on that. See my first comment in this email for more > > details. > > > > On the summary: we can't stop accepting read requests. > > > > Btw, what to do with reads, which were *in-progress*, when the quorum > > was lost? Such as long vinyl reads. > > But the quorum was in place at the start of it? Then according to > transaction manager behavior only older version data will be available > for read - means data that collected quorum. > > > > > > The reason for this is that replication of transactions > > > can achieve quorum on replicas not visible to the leader. On the other > > > hand, leader can't achieve quorum with available minority. Leader has to > > > report the state and wait for human intervention. > > > > Yeah, but if the leader couldn't achieve a quorum on some transactions, > > they are not visible (assuming MVCC will work properly). So they can't > > be read anyway. And if a leader answered an error, it does not mean that > > the transaction wasn't replicated on the majority, as we discussed at some > > meeting, I don't already remember when. So here read allowance also works > > fine - not having some data visible and getting error at a sync transaction > > does not mean it is not committed. A user should be aware of that. > > True, we discussed that we should guarantee only that if we answered > 'Ok' then data is present in quorum number of instances. > > [...] > > > > demote(ID) - should be called from the Leader instance. > > > The Leader has to switch in ro mode and wait for its' undo log is > > > empty. This effectively means all transactions are committed in the > > > cluster and it is safe pass the leadership. Then it should write > > > CURRENT_LEADER_ID as a FORMER_LEADER_ID and put CURRENT_LEADER_ID > > > into 0. > > > > This looks like box.ctl.promote() algorithm. Although I thought we decided > > not to implement any kind of auto election here, no? Box.ctl.promote() > > assumed, that it does all the steps automatically, except choosing on which > > node to call this function. This is what it was so complicated. It was > > basically raft. > > > > But yeah, as discussed verbally, this is a subject for improvement. > > I personally would like to postpone the algorithm should be postponed > for the next stage (Q3-Q4) but now we should not mess up too much to > revamp. Hence, we have to elaborate the internals - such as _voting > table I mentioned. > > Even with introduction of terms for each leader - as in RAFT for example > - we still can keep it in a replicated space, isn't it? > > > > > The way I see it is that we need to give vclock based algorithm of choosing > > a new leader; tell how to stop replication from the old leader; allow to > > read vclock from replicas (basically, let the external service read box.info). > > That's the #1 for me by now: how a read-only replica can quit listening > to a demoted leader, which can be not aware of its demotion? Still, for > efficiency it should be done w/o disconnection. > > > > > Since you said you think we should not provide an API for all sync transactions > > rollback, it looks like no need in a special new API. But if we still want > > to allow to rollback all pending transactions of the old leader on a new leader > > (like Mons wants) then yeah, seems like we would need a new function. For example, > > box.ctl.sync_rollback() to rollback all pending. And box.ctl.sync_confirm() to > > confirm all pending. Perhaps we could add more admin-line parameters such as > > replica_id with which to write 'confirm/rollback' message. > > I believe it's a good point to keep two approaches and perhaps set one > of the two in configuration. This should resolve the issue with 'the > rest of the cluster confirms old leader's transactions and because of it > leader can't rollback'. > > > > > > ### Recovery and failover. > > > > > > Tarantool instance during reading WAL should postpone the undo log > > > deletion until the 'confirm' is read. In case the WAL eof is achieved, > > > the instance should keep undo log for all transactions that are waiting > > > for a confirm entry until the role of the instance is set. > > > > > > If this instance will be assigned a leader role then all transactions > > > that have no corresponding confirm message should be confirmed (see the > > > leader role assignment). > > > > > > In case there's not enough replicas to set up a quorum the cluster can > > > be switched into a read-only mode. Note, this can't be done by default > > > since some of transactions can have confirmed state. It is up to human > > > intervention to force rollback of all transactions that have no confirm > > > and to put the cluster into a consistent state. > > > > Above you said: > > > > >> As soon as leader appears in a situation it has not enough replicas > > >> to achieve quorum, the cluster should stop accepting any requests - both > > >> write and read. > > > > But here I see, that the cluster "switched into a read-only mode". So there > > is a contradiction. And I think it should be resolved in favor of > > 'read-only mode'. I explained why in the previous comments. > > My bad, I was moving around this problem already and tend to allow r/o. > Will update. > > > > > > In case the instance will be assigned a replica role, it may appear in > > > a state that it has conflicting WAL entries, in case it recovered from a > > > leader role and some of transactions didn't replicated to the current > > > leader. This situation should be resolved through rejoin of the instance. > > > > > > Consider an example below. Originally instance with ID1 was assigned a > > > Leader role and the cluster had 2 replicas with quorum set to 2. > > > > > > ``` > > > +---------------------+---------------------+---------------------+ > > > | ID1 | ID2 | ID3 | > > > | Leader | Replica 1 | Replica 2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx4 | ID1 Tx4 | | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx5 | ID1 Tx5 | | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Conf [ID1, Tx2] | | | > > > +---------------------+---------------------+---------------------+ > > > | Tx6 | | | > > > +---------------------+---------------------+---------------------+ > > > | Tx7 | | | > > > +---------------------+---------------------+---------------------+ > > > ``` > > > Suppose at this moment the ID1 instance crashes. Then the ID2 instance > > > should be assigned a leader role since its ID1 LSN is the biggest. > > > Then this new leader will deliver its WAL to all replicas. > > > > > > As soon as quorum for Tx4 and Tx5 will be obtained, it should write the > > > corresponding Confirms to its WAL. Note that Tx are still uses ID1. > > > ``` > > > +---------------------+---------------------+---------------------+ > > > | ID1 | ID2 | ID3 | > > > | (dead) | Leader | Replica 2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Conf [ID1, Tx2] | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | > > > > Id1 -> ID1 (typo) > > Thanks! > > > > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx6 | | | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx7 | | | > > > +---------------------+---------------------+---------------------+ > > > ``` > > > After rejoining ID1 will figure out the inconsistency of its WAL: the > > > last WAL entry it has is corresponding to Tx7, while in Leader's log the > > > last entry with ID1 is Tx5. Confirm for a Tx can only be issued after > > > appearance of the Tx on the majoirty of replicas, hence there's a good > > > chances that ID1 will have inconsistency in its WAL covered with undo > > > log. So, by rolling back all excessive Txs (in the example they are Tx6 > > > and Tx7) the ID1 can put its memtx and vynil in consistent state. > > > > Yeah, but the problem is that the node1 has vclock[ID1] == 'Conf [ID1, Tx2]'. > > This row can't be rolled back. So looks like node1 needs a rejoin. > > Confirm message is equivalent to a NOP - @sergepetrenko apparently does > implementation exactly this way. So there's no need to roll it back in > an engine, rather perform the xlog rotation before it. > > > > > > At this point a snapshot can be created at ID1 with appropriate WAL > > > rotation. The old WAL should be renamed so it will not be reused in the > > > future and can be kept for postmortem. > > > ``` > > > +---------------------+---------------------+---------------------+ > > > | ID1 | ID2 | ID3 | > > > | Replica 1 | Leader | Replica 2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx1 | ID1 Tx1 | ID1 Tx1 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx2 | ID1 Tx2 | ID1 Tx2 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx3 | ID1 Tx3 | ID1 Tx3 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | ID1 Conf [ID1, Tx1] | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx4 | ID1 Tx4 | ID1 Tx4 | > > > +---------------------+---------------------+---------------------+ > > > | ID1 Tx5 | ID1 Tx5 | ID1 Tx5 | > > > +---------------------+---------------------+---------------------+ > > > | | ID2 Conf [Id1, Tx5] | ID2 Conf [Id1, Tx5] | > > > +---------------------+---------------------+---------------------+ > > > | | ID2 Tx1 | ID2 Tx1 | > > > +---------------------+---------------------+---------------------+ > > > | | ID2 Tx2 | ID2 Tx2 | > > > +---------------------+---------------------+---------------------+ > > > ``` > > > Although, in case undo log is not enough to cover the WAL inconsistence > > > with the new leader, the ID1 needs a complete rejoin. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-06-09 16:19 ` Sergey Ostanevich @ 2020-06-11 15:17 ` Vladislav Shpilevoy 2020-06-12 20:31 ` Sergey Ostanevich 0 siblings, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-06-11 15:17 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches Hi! Thanks for the updates! > ### Connection liveness > > There is a timeout-based mechanism in Tarantool that controls the > asynchronous replication, which uses the following config: > ``` > * replication_connect_timeout = 4 > * replication_sync_lag = 10 > * replication_sync_timeout = 300 > * replication_timeout = 1 > ``` > For backward compatibility and to differentiate the async replication > we should augment the configuration with the following: > ``` > * synchro_replication_heartbeat = 4 Heartbeats are already being sent. I don't see any sense in adding a second heartbeat option. > * synchro_replication_quorum_timeout = 4 Since this is a replication option, it should start from replication_ prefix. > ``` > Leader should send a heartbeat every synchro_replication_heartbeat if > there were no messages sent. Replicas should respond to the heartbeat > just the same way as they do it now. As soon as Leader has no response > for another heartbeat interval, it should consider the replica is lost. All of that is already done in the regular heartbeats, not related nor bound to any synchronous activities. Just like failure detection should be. > As soon as leader appears in a situation it has not enough replicas > to achieve quorum, it should stop accepting write requests. There's an > option for leader to rollback to the latest transaction that has quorum: > leader issues a 'rollback' message referring to the [LEADER_ID, LSN] > where LSN is of the first transaction in the leader's undo log. What is that option? > The rollback message replicated to the available cluster will put it in a > consistent state. After that configuration of the cluster can be > updated to a new available quorum and leader can be switched back to > write mode. > > During the quorum collection it can happen that some of replicas become > unavailable due to some reason, so leader should wait at most for > synchro_replication_quorum_timeout after which it issues a Rollback > pointing to the oldest TXN in the waiting list. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-06-11 15:17 ` Vladislav Shpilevoy @ 2020-06-12 20:31 ` Sergey Ostanevich 0 siblings, 0 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-06-12 20:31 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches Hi! Thanks for review, attaching a diff. Full version is available at the branch https://github.com/tarantool/tarantool/blob/sergos/quorum-based-synchro/ On 11 июн 17:17, Vladislav Shpilevoy wrote: > Hi! Thanks for the updates! > > > ### Connection liveness > > > > There is a timeout-based mechanism in Tarantool that controls the > > asynchronous replication, which uses the following config: > > ``` > > * replication_connect_timeout = 4 > > * replication_sync_lag = 10 > > * replication_sync_timeout = 300 > > * replication_timeout = 1 > > ``` > > For backward compatibility and to differentiate the async replication > > we should augment the configuration with the following: > > ``` > > * synchro_replication_heartbeat = 4 > > Heartbeats are already being sent. I don't see any sense in adding a > second heartbeat option. I had an idea that synchronous replication can co-exist with async one, so they have to have independent tuning. Now I realize that sending two types of heartbeats is too much, so I'll drop this one. > > > * synchro_replication_quorum_timeout = 4 > > Since this is a replication option, it should start from replication_ > prefix. There are number of options already exist that are very similar in naming, such as replication_sync_timeout, replication_sync_lag and even replication_connect_quorum. I expect to resolve the ambiguity with putting in a new prefix, synchro_replication. The drawback is those options reused from async mode would be not-so-clearly linked to the synch one. > > > ``` > > Leader should send a heartbeat every synchro_replication_heartbeat if > > there were no messages sent. Replicas should respond to the heartbeat > > just the same way as they do it now. As soon as Leader has no response > > for another heartbeat interval, it should consider the replica is lost. > > All of that is already done in the regular heartbeats, not related nor > bound to any synchronous activities. Just like failure detection should be. > > > As soon as leader appears in a situation it has not enough replicas > > to achieve quorum, it should stop accepting write requests. There's an > > option for leader to rollback to the latest transaction that has quorum: > > leader issues a 'rollback' message referring to the [LEADER_ID, LSN] > > where LSN is of the first transaction in the leader's undo log. > > What is that option? Good catch, thanks! This option was introduced to get to a consistent state with replicas. Although, if Leader will wait longer than timeout for quorum it will rollback anyways, so I will remove mention of this. > > > The rollback message replicated to the available cluster will put it in a > > consistent state. After that configuration of the cluster can be > > updated to a new available quorum and leader can be switched back to > > write mode. > > > > During the quorum collection it can happen that some of replicas become > > unavailable due to some reason, so leader should wait at most for > > synchro_replication_quorum_timeout after which it issues a Rollback > > pointing to the oldest TXN in the waiting list. diff --git a/doc/rfc/quorum-based-synchro.md b/doc/rfc/quorum-based-synchro.md index c7dcf56b5..0a92642fd 100644 --- a/doc/rfc/quorum-based-synchro.md +++ b/doc/rfc/quorum-based-synchro.md @@ -83,9 +83,10 @@ Customer Leader WAL(L) Replica WAL(R) To introduce the 'quorum' we have to receive confirmation from replicas to make a decision on whether the quorum is actually present. Leader -collects necessary amount of replicas confirmation plus its own WAL -success. This state is named 'quorum' and gives leader the right to -complete the customers' request. So the picture will change to: +collects replication_synchro_quorum-1 of replicas confirmation and its +own WAL success. This state is named 'quorum' and gives leader the +right to complete the customers' request. So the picture will change +to: ``` Customer Leader WAL(L) Replica WAL(R) |------TXN----->| | | | @@ -158,26 +159,21 @@ asynchronous replication, which uses the following config: For backward compatibility and to differentiate the async replication we should augment the configuration with the following: ``` -* synchro_replication_heartbeat = 4 -* synchro_replication_quorum_timeout = 4 +* replication_synchro_quorum_timeout = 4 +* replication_synchro_quorum = 4 ``` -Leader should send a heartbeat every synchro_replication_heartbeat if -there were no messages sent. Replicas should respond to the heartbeat -just the same way as they do it now. As soon as Leader has no response -for another heartbeat interval, it should consider the replica is lost. -As soon as leader appears in a situation it has not enough replicas -to achieve quorum, it should stop accepting write requests. There's an -option for leader to rollback to the latest transaction that has quorum: -leader issues a 'rollback' message referring to the [LEADER_ID, LSN] -where LSN is of the first transaction in the leader's undo log. The -rollback message replicated to the available cluster will put it in a -consistent state. After that configuration of the cluster can be -updated to a new available quorum and leader can be switched back to -write mode. +Leader should send a heartbeat every replication_timeout if there were +no messages sent. Replicas should respond to the heartbeat just the +same way as they do it now. As soon as Leader has no response for +another heartbeat interval, it should consider the replica is lost. As +soon as leader appears in a situation it has not enough replicas to +achieve quorum, it should stop accepting write requests. After that +configuration of the cluster can be updated to a new available quorum +and leader can be switched back to write mode. During the quorum collection it can happen that some of replicas become unavailable due to some reason, so leader should wait at most for -synchro_replication_quorum_timeout after which it issues a Rollback +replication_synchro_quorum_timeout after which it issues a Rollback pointing to the oldest TXN in the waiting list. ### Leader role assignment. @@ -274,9 +270,9 @@ Leader role and the cluster had 2 replicas with quorum set to 2. +---------------------+---------------------+---------------------+ | ID1 Conf [ID1, Tx2] | | | +---------------------+---------------------+---------------------+ -| Tx6 | | | +| ID1 Tx | | | +---------------------+---------------------+---------------------+ -| Tx7 | | | +| ID1 Tx | | | +---------------------+---------------------+---------------------+ ``` Suppose at this moment the ID1 instance crashes. Then the ID2 instance ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 16:39 ` Sergey Ostanevich 2020-05-06 18:44 ` Konstantin Osipov @ 2020-05-13 21:36 ` Vladislav Shpilevoy 2020-05-13 23:45 ` Konstantin Osipov 1 sibling, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-05-13 21:36 UTC (permalink / raw) To: Sergey Ostanevich, Konstantin Osipov, tarantool-patches Thanks for the discussion! On 06/05/2020 18:39, Sergey Ostanevich wrote: > Hi! > > Thanks for review! > >>> | | | | | >>> | [Quorum | | | >>> | achieved] | | | >>> | | | | | >>> | [TXN undo log | | | >>> | destroyed] | | | >>> | | | | | >>> | |---Confirm--->| | | >>> | | | | | >> >> What happens if writing Confirm to WAL fails? TXN und log record >> is destroyed already. Will the server panic now on WAL failure, >> even if it is intermittent? > > I would like to have an example of intermittent WAL failure. Can it be > other than problem with disc - be it space/availability/malfunction? > > For all of those it should be resolved outside the DBMS anyways. So, > leader should stop and report its problems to orchestrator/admins. > > I would agree that undo log can be destroyed *after* the Confirm is > landed to WAL - same is for replica. Well, in fact you can't (or can you?). Because it won't help. Once you tried to write 'Confirm', it means you got the quorum. So now in case you will fail, a new leader will write 'Confirm' for you, when will see a quorum too. So the current leader has no right to write 'Rollback' from this moment, from what I understand. Because it still can be confirmed by a new leader later, if you fail before 'Rollback' is replicated to all. However the same problem appears, if you write 'Confirm' *successfully*. Still the leader can fail, and a newer leader will write 'Rollback' if won't collect the quorum again. Don't know what to do with that really. Probably nothing. >> >>> | |----------Confirm---------->| | >> >> What happens if peers receive and maybe even write Confirm to their WALs >> but local WAL write is lost after a restart? > > Did you mean WAL write on leader as a local? Then we have a replica with > a bigger LSN for the leader ID. > >> WAL is not synced, >> so we can easily lose the tail of the WAL. Tarantool will sync up >> with all replicas on restart, > > But at this point a new leader will be appointed - the old one is > restarted. Then the Confirm message will arrive to the restarted leader > through a regular replication. > >> but there will be no "Replication >> OK" messages from them, so it wouldn't know that the transaction >> is committed on them. How is this handled? We may end up with some >> replicas confirming the transaction while the leader will roll it >> back on restart. Do you suggest there is a human intervention on >> restart as well? >> >> >>> | | | | | >>> |<---TXN Ok-----| | [TXN undo log | >>> | | | destroyed] | >>> | | | | | >>> | | | |---Confirm--->| >>> | | | | | >>> ``` >>> >>> The quorum should be collected as a table for a list of transactions >>> waiting for quorum. The latest transaction that collects the quorum is >>> considered as complete, as well as all transactions prior to it, since >>> all transactions should be applied in order. Leader writes a 'confirm' >>> message to the WAL that refers to the transaction's [LEADER_ID, LSN] and >>> the confirm has its own LSN. This confirm message is delivered to all >>> replicas through the existing replication mechanism. >>> >>> Replica should report a TXN application success to the leader via the >>> IPROTO explicitly to allow leader to collect the quorum for the TXN. >>> In case of application failure the replica has to disconnect from the >>> replication the same way as it is done now. The replica also has to >>> report its disconnection to the orchestrator. Further actions require >>> human intervention, since failure means either technical problem (such >>> as not enough space for WAL) that has to be resovled or an inconsistent >>> state that requires rejoin. >> >>> As soon as leader appears in a situation it has not enough replicas >>> to achieve quorum, the cluster should stop accepting any requests - both >>> write and read. >> >> How does *the cluster* know the state of the leader and if it >> doesn't, how it can possibly implement this? Did you mean >> the leader should stop accepting transactions here? But how can >> the leader know if it has not enough replicas during a read >> transaction, if it doesn't contact any replica to serve a read? > > I expect to have a disconnection trigger assigned to all relays so that > disconnection will cause the number of replicas decrease. The quorum > size is static, so we can stop at the very moment the number dives below. This is a very dubious statement. In TCP disconnect may be detected much later, than it happened. So to collect a quorum on something you need to literally collect this quorum, with special WAL records, via network, and all. A disconnect trigger does not help at all here. Talking of the whole 'read-quorum' idea, I don't like it. Because this really makes things unbearably harder to implement, the nodes become much slower and less available in terms of any problems. I think reads should be allowed always, and from any node (except during bootstrap, of course). After all, you have transactions for consistency. So as far as replication respects transaction boundaries, every node is in a consistent state. Maybe not all of them are in the same state, but every one is consistent. Honestly, I can't even imagine, how is it possible to implement a completely synchronous simultaneous cluster progression. It is impossible even in theory. There always will be a time period, when some nodes are further than the others. At least because of network delays. So either we allows reads from master only, or we allow reads from everywhere, and in that case nothing will save from a possibility of seeing different data on different nodes. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-13 21:36 ` Vladislav Shpilevoy @ 2020-05-13 23:45 ` Konstantin Osipov 0 siblings, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-13 23:45 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/05/14 00:37]: > > Thanks for review! > > > >>> | | | | | > >>> | [Quorum | | | > >>> | achieved] | | | > >>> | | | | | > >>> | [TXN undo log | | | > >>> | destroyed] | | | > >>> | | | | | > >>> | |---Confirm--->| | | > >>> | | | | | > >> > >> What happens if writing Confirm to WAL fails? TXN und log record > >> is destroyed already. Will the server panic now on WAL failure, > >> even if it is intermittent? > > > > I would like to have an example of intermittent WAL failure. Can it be > > other than problem with disc - be it space/availability/malfunction? > > > > For all of those it should be resolved outside the DBMS anyways. So, > > leader should stop and report its problems to orchestrator/admins. > > > > I would agree that undo log can be destroyed *after* the Confirm is > > landed to WAL - same is for replica. > > Well, in fact you can't (or can you?). Because it won't help. Once you > tried to write 'Confirm', it means you got the quorum. So now in case > you will fail, a new leader will write 'Confirm' for you, when will see > a quorum too. So the current leader has no right to write 'Rollback' > from this moment, from what I understand. Because it still can be > confirmed by a new leader later, if you fail before 'Rollback' is > replicated to all. > > However the same problem appears, if you write 'Confirm' *successfully*. > Still the leader can fail, and a newer leader will write 'Rollback' if > won't collect the quorum again. Don't know what to do with that really. > Probably nothing. Maybe consult with the raft spec? The new leader is guaranteed to see the transaction since it has reached the majority of replicas. So it will definitely write "confirm" for it. The reason I asked the question is I want the case of intermittent failures be described in the spec. For example, is "confirm" a cbus message, then if there is a cascading rollback of the batch it is part of, it can be rolled back. I would like to see all these scenarios covered in the spec. If one of them ends with panic, I would like to understand how the external coordinator is going to resolve the new election. Raft has answers for all of it. > >>> As soon as leader appears in a situation it has not enough replicas > >>> to achieve quorum, the cluster should stop accepting any requests - both > >>> write and read. > >> > >> How does *the cluster* know the state of the leader and if it > >> doesn't, how it can possibly implement this? Did you mean > >> the leader should stop accepting transactions here? But how can > >> the leader know if it has not enough replicas during a read > >> transaction, if it doesn't contact any replica to serve a read? > > > > I expect to have a disconnection trigger assigned to all relays so that > > disconnection will cause the number of replicas decrease. The quorum > > size is static, so we can stop at the very moment the number dives below. > > This is a very dubious statement. In TCP disconnect may be detected much > later, than it happened. So to collect a quorum on something you need to > literally collect this quorum, with special WAL records, via network, and > all. A disconnect trigger does not help at all here. Erhm, thanks. > Talking of the whole 'read-quorum' idea, I don't like it. Because this > really makes things unbearably harder to implement, the nodes become > much slower and less available in terms of any problems. > > I think reads should be allowed always, and from any node (except during > bootstrap, of course). After all, you have transactions for consistency. So > as far as replication respects transaction boundaries, every node is in a > consistent state. Maybe not all of them are in the same state, but every one > is consistent. In memtx, you read by default dirty, uncommitted data. It was OK for single-node transactions, since the only chance for it to be rolled back were out of space/disk failure, which were extremely rare, now you really read dirty stuff, because you can easily have it rolled back because of lack of quorum or re-election. So it's a much bigger deal. > Honestly, I can't even imagine, how is it possible to implement a completely > synchronous simultaneous cluster progression. It is impossible even in theory. > There always will be a time period, when some nodes are further than the > others. At least because of network delays. > > So either we allows reads from master only, or we allow reads from everywhere, > and in that case nothing will save from a possibility of seeing different > data on different nodes. This is why there are many consistency models out there (just google consistency models in distributed systems), and the minor details are important. It's indeed hard to implement the strictest model (serial), but it is also often unnecessary, and there is consensus in the relational databases what issues are acceptable and what are not. More specifically, I think for tarantool sync replication we should aim at read committed. The spec should say it in no uncertain terms and explain how it is achieved. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-30 14:50 ` Sergey Ostanevich 2020-05-06 8:52 ` Konstantin Osipov @ 2020-05-06 18:55 ` Konstantin Osipov 2020-05-06 19:10 ` Konstantin Osipov 2020-05-13 21:42 ` Vladislav Shpilevoy 2020-05-07 23:01 ` Konstantin Osipov 2 siblings, 2 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-06 18:55 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/04/30 17:51]: A few more issues: - the spec assumes there is a full mesh. In any other topology electing a leader based on the longest wal can easily deadlock. Yet it provides no protection against non-full-mesh setups. Currently the server can't even detect that this is not a full-mesh setup, so can't check if the precondition for this to work correctly is met. - the spec assumes that quorum is identical to the number of replicas, and the number of replicas is stable across cluster life time. Can I have quorum=2 while the number of replicas is 4? Am I allowed to increase the number of replicas online? What happens when a replica is added, how exactly and starting from which transaction is the leader required to collect a bigger quorum? - the same goes for removing a replica. How is the quorum reduced? -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 18:55 ` Konstantin Osipov @ 2020-05-06 19:10 ` Konstantin Osipov 2020-05-12 16:03 ` Sergey Ostanevich 2020-05-13 21:42 ` Vladislav Shpilevoy 1 sibling, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-05-06 19:10 UTC (permalink / raw) To: Sergey Ostanevich, Vladislav Shpilevoy, tarantool-patches * Konstantin Osipov <kostja.osipov@gmail.com> [20/05/06 21:55]: > A few more issues: > > - the spec assumes there is a full mesh. In any other > topology electing a leader based on the longest wal can easily > deadlock. Yet it provides no protection against non-full-mesh > setups. Currently the server can't even detect that this is not > a full-mesh setup, so can't check if the precondition for this > to work correctly is met. Come to think of it, it's a special case of network partitioning. A replica with the longest WAL can be reachable by the external coordinator but partitioned away from the majority, so never able to make progress. -- Konstantin Osipov, Moscow, Russia https://scylladb.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 19:10 ` Konstantin Osipov @ 2020-05-12 16:03 ` Sergey Ostanevich 0 siblings, 0 replies; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-12 16:03 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches On 06 мая 22:10, Konstantin Osipov wrote: > * Konstantin Osipov <kostja.osipov@gmail.com> [20/05/06 21:55]: > > A few more issues: > > > > - the spec assumes there is a full mesh. In any other > > topology electing a leader based on the longest wal can easily > > deadlock. Yet it provides no protection against non-full-mesh > > setups. Currently the server can't even detect that this is not > > a full-mesh setup, so can't check if the precondition for this > > to work correctly is met. > > Come to think of it, it's a special case of network partitioning. > A replica with the longest WAL can be reachable by the external > coordinator but partitioned away from the majority, so never able to > make progress. So the answer from this replica on it's appointment will be 'I have no quorum'. Hence, the orchentration should pick the next-length WAL. What's the problem? > > > -- > Konstantin Osipov, Moscow, Russia > https://scylladb.com ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-06 18:55 ` Konstantin Osipov 2020-05-06 19:10 ` Konstantin Osipov @ 2020-05-13 21:42 ` Vladislav Shpilevoy 2020-05-14 0:05 ` Konstantin Osipov 1 sibling, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-05-13 21:42 UTC (permalink / raw) To: Konstantin Osipov, Sergey Ostanevich, tarantool-patches Thanks for the discussion! On 06/05/2020 20:55, Konstantin Osipov wrote: > * Sergey Ostanevich <sergos@tarantool.org> [20/04/30 17:51]: > > A few more issues: > > - the spec assumes there is a full mesh. In any other > topology electing a leader based on the longest wal can easily > deadlock. Yet it provides no protection against non-full-mesh > setups. Currently the server can't even detect that this is not > a full-mesh setup, so can't check if the precondition for this > to work correctly is met. Yes, this is a very unstable construction. But we failed to come up with a solution right now, which would protect against accidental non-fullmesh. For example, how will it work, when I add a new node? If non-fullmesh is forbidden, the new node just can't be added ever, because this can't be done on all nodes simultaneously. > - the spec assumes that quorum is identical to the > number of replicas, and the number of replicas is stable across > cluster life time. Can I have quorum=2 while the number of > replicas is 4? Am I allowed to increase the number of replicas > online? What happens when a replica is added, > how exactly and starting from which transaction is the leader > required to collect a bigger quorum? Quorum <= number of replicas. It is a parameter, just like replication_connect_quorum. I think you are allowed to add new replicas. When a replica is added, it goes through the normal join process. > - the same goes for removing a replica. How is the quorum reduced? Node is just removed, I guess. If total number of nodes becomes less than quorum, obviously no transactions will be served. However what to do with the existing pending transactions, which already accounted the removed replica in their quorums? Should they be decremented? All what I am talking here are guesses. Which should be clarified in the RFC in the ideal world, of course. Tbh, we discussed the sync replication for may hours in voice, and this is a surprise, that all of them fit into such a small update of the RFC. Even though it didn't fit. Since we obviously still didn't clarify many things. Especially exact API look. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-13 21:42 ` Vladislav Shpilevoy @ 2020-05-14 0:05 ` Konstantin Osipov 0 siblings, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-14 0:05 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/05/14 00:47]: > > A few more issues: > > > > - the spec assumes there is a full mesh. In any other > > topology electing a leader based on the longest wal can easily > > deadlock. Yet it provides no protection against non-full-mesh > > setups. Currently the server can't even detect that this is not > > a full-mesh setup, so can't check if the precondition for this > > to work correctly is met. > > Yes, this is a very unstable construction. But we failed to come up > with a solution right now, which would protect against accidental > non-fullmesh. For example, how will it work, when I add a new node? > If non-fullmesh is forbidden, the new node just can't be added ever, > because this can't be done on all nodes simultaneously. Again the answer is present in the raft spec. The node is added in two steps, first steps commits the "add node" event to the durable state of the entire group, the second step (which is also a raft transaction) enacts the new node. This could be achieved in more or less straightforward manner if _cluster is a sync table with replication group = all members of the cluster. But as I said, I can't imagine this is possible with an external coordinator, since it may not be available during boot. Regarding detecting the full mesh, remember the task I created for using swim to discover members and bring non-full-mesh setups to full-mesh automatically? Is the reason for this task to exist clear now? Is it clear now why I asked you (multiple times) to begin working on sync replication by adding built-in swim instances on every replica and using them, instead of the current replication heartbeats, for failure detection? I believe there was a task somewhere for it, too. > > - the spec assumes that quorum is identical to the > > number of replicas, and the number of replicas is stable across > > cluster life time. Can I have quorum=2 while the number of > > replicas is 4? Am I allowed to increase the number of replicas > > online? What happens when a replica is added, > > how exactly and starting from which transaction is the leader > > required to collect a bigger quorum? > > Quorum <= number of replicas. It is a parameter, just like > replication_connect_quorum. I wrote in a comment to the task that it'd be even better if we list node uuids as group members, and assign group to space explicitly, so that it's not just ## of replicas, but specific replicas identified by their uuids. The thing is, it's vague in the spec. The spec has to be explicit about all box.schema API changes, because they will define legacy that will be hard to deal with later. > I think you are allowed to add new replicas. When a replica is added, > it goes through the normal join process. At what point is joins the group and can ACK, i.e. become part of a quorum? That's the question I wanted to be written down explicitly in this document. RAFT has an answer for it. > > - the same goes for removing a replica. How is the quorum reduced? > > Node is just removed, I guess. If total number of nodes becomes less > than quorum, obviously no transactions will be served. Other vendors support 3 different scenarios here: - it can be down for maintenance. In our turns, it means it is simply shut down, without changes to _cluster or space settings - it can be removed forever, in that case an admin may want to reduce the quorum size. - it can be replaced. with box.schema.group API all 3 cases can be translated to API calls on the group itself. e.g. it would be possible to say box.schema.group.groupname.remove(uuid) box.schema.group.groupname.replace(old_uuid, new_uuid). We don't need to implement it right away, but we must provision for these operations in the spec, and at least have a clue how they will be handled in the future. > However what to do with the existing pending transactions, which > already accounted the removed replica in their quorums? Should they be > decremented? > > All what I am talking here are guesses. Which should be clarified in the > RFC in the ideal world, of course. > > Tbh, we discussed the sync replication for may hours in voice, and this > is a surprise, that all of them fit into such a small update of the RFC. > Even though it didn't fit. Since we obviously still didn't clarify many > things. Especially exact API look. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-04-30 14:50 ` Sergey Ostanevich 2020-05-06 8:52 ` Konstantin Osipov 2020-05-06 18:55 ` Konstantin Osipov @ 2020-05-07 23:01 ` Konstantin Osipov 2020-05-12 16:40 ` Sergey Ostanevich 2 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-05-07 23:01 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy > ### Synchronous replication enabling. > > Synchronous operation can be required for a set of spaces in the data > scheme. That means only transactions that contain data modification for > these spaces should require quorum. Such transactions named synchronous. > As soon as last operation of synchronous transaction appeared in leader's > WAL, it will cause all following transactions - no matter if they are > synchronous or not - wait for the quorum. In case quorum is not achieved > the 'rollback' operation will cause rollback of all transactions after > the synchronous one. It will ensure the consistent state of the data both > on leader and replicas. In case user doesn't require synchronous operation > for any space then no changes to the WAL generation and replication will > appear. 1) It's unclear what happens here if async tx follows a sync tx. Does it wait for the sync tx? This reduces availability for async txs - so it's hardly acceptable. Besides, with group=local spaces, one can quickly run out of memory for undo. Then it should be allowed to proceed and commit. Then mixing sync and async tables in a single transaction shouldn't be allowed. Imagine t1 is sync and t2 is async. tx1 changes t1 and t2, tx2 changes t2. tx1 is not confirmed and must be rolled back. But it can not revert changes of tx2. The spec should clarify that. 2) First candidates to "sync" spaces are system spaces, especially _schema (to fix box.once()) and _cluster (to fix parallel join of multiple replicas). I can't imagine it's possible to make system spaces synchronous with an external coordinator - the coordinator may not be available during box.cfg{}. 3) One can quickly run out of memory for undo. Any sync transaction should be capped with a timeout to avoid OOMs. I don't know how many times I should repeat it. The only good solution for load control is in-memory WAL, which will allow to rollback all transactions as soon as network partitioning is detected. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-07 23:01 ` Konstantin Osipov @ 2020-05-12 16:40 ` Sergey Ostanevich 2020-05-12 17:47 ` Konstantin Osipov 0 siblings, 1 reply; 53+ messages in thread From: Sergey Ostanevich @ 2020-05-12 16:40 UTC (permalink / raw) To: Konstantin Osipov, Vladislav Shpilevoy, tarantool-patches On 08 мая 02:01, Konstantin Osipov wrote: > > > > ### Synchronous replication enabling. > > > > Synchronous operation can be required for a set of spaces in the data > > scheme. That means only transactions that contain data modification for > > these spaces should require quorum. Such transactions named synchronous. > > As soon as last operation of synchronous transaction appeared in leader's > > WAL, it will cause all following transactions - no matter if they are > > synchronous or not - wait for the quorum. In case quorum is not achieved > > the 'rollback' operation will cause rollback of all transactions after > > the synchronous one. It will ensure the consistent state of the data both > > on leader and replicas. In case user doesn't require synchronous operation > > for any space then no changes to the WAL generation and replication will > > appear. > > 1) It's unclear what happens here if async tx follows a sync tx. > Does it wait for the sync tx? This reduces availability for Definitely yes, unless we keep the 'dirty read' as it is at the moment in memtx. This is the essence of the design, and it is temporary until the MVCC similar to the vinyl machinery appears. I intentionally didn't include this big task into this RFC. It will provide similar capabilities, although it will keep only dependent transactions in the undo log. Also, it looks like it will fit well into the machinery of this RFC. > async txs - so it's hardly acceptable. Besides, with > group=local spaces, one can quickly run out of memory for undo. > > > Then it should be allowed to proceed and commit. > > Then mixing sync and async tables in a single transaction > shouldn't be allowed. > > Imagine t1 is sync and t2 is async. tx1 changes t1 and t2, tx2 > changes t2. tx1 is not confirmed and must be rolled back. But it can > not revert changes of tx2. > > The spec should clarify that. > > 2) First candidates to "sync" spaces are system spaces, especially > _schema (to fix box.once()) and _cluster (to fix parallel join > of multiple replicas). > > I can't imagine it's possible to make system spaces synchronous > with an external coordinator - the coordinator may not be > available during box.cfg{}. May not be - means no coordination, means the server can't start. Again, we're not trying to elaborate the self-driven cluster at this moment, we rely on external coonrdination. > > 3) One can quickly run out of memory for undo. Any sync > transaction should be capped with a timeout to avoid OOMs. I > don't know how many times I should repeat it. The only good > solution for load control is in-memory WAL, which will allow to > rollback all transactions as soon as network partitioning is > detected. How in-memry WAL can help save on _undo_ memory? To rollback whatever amount of transactions one need to store the undo. > > -- > Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-12 16:40 ` Sergey Ostanevich @ 2020-05-12 17:47 ` Konstantin Osipov 2020-05-13 21:34 ` Vladislav Shpilevoy 0 siblings, 1 reply; 53+ messages in thread From: Konstantin Osipov @ 2020-05-12 17:47 UTC (permalink / raw) To: Sergey Ostanevich; +Cc: tarantool-patches, Vladislav Shpilevoy * Sergey Ostanevich <sergos@tarantool.org> [20/05/12 19:43]: > > 1) It's unclear what happens here if async tx follows a sync tx. > > Does it wait for the sync tx? This reduces availability for > > Definitely yes, unless we keep the 'dirty read' as it is at the moment > in memtx. This is the essence of the design, and it is temporary until > the MVCC similar to the vinyl machinery appears. I intentionally didn't > include this big task into this RFC. > > It will provide similar capabilities, although it will keep only > dependent transactions in the undo log. Also, it looks like it will fit > well into the machinery of this RFC. = reduced availability for all who have at least one sync space. If different spaces have different quorum size = quorum size of the biggest group is effectively used for all spaces. Replica-local transactions, e.g. those used by vinyl compaction, are rolled back if there is no quorum. What's the value of this? > > > async txs - so it's hardly acceptable. Besides, with > > group=local spaces, one can quickly run out of memory for undo. > > > > > > Then it should be allowed to proceed and commit. > > > > Then mixing sync and async tables in a single transaction > > shouldn't be allowed. > > > > Imagine t1 is sync and t2 is async. tx1 changes t1 and t2, tx2 > > changes t2. tx1 is not confirmed and must be rolled back. But it can > > not revert changes of tx2. > > > > The spec should clarify that. You conveniently skip this explanation of the problem - meaning you don't intend to address it? > > > > 3) One can quickly run out of memory for undo. Any sync > > transaction should be capped with a timeout to avoid OOMs. I > > don't know how many times I should repeat it. The only good > > solution for load control is in-memory WAL, which will allow to > > rollback all transactions as soon as network partitioning is > > detected. > > How in-memry WAL can help save on _undo_ memory? > To rollback whatever amount of transactions one need to store the undo. I wrote earlier that it works as a natural failure detector and throttling mechanism. If there is no quorum, we can see it immediately by looking at the number of active subscribers of the in-memory WAL, so do not accumulate undo. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-12 17:47 ` Konstantin Osipov @ 2020-05-13 21:34 ` Vladislav Shpilevoy 2020-05-13 23:31 ` Konstantin Osipov 0 siblings, 1 reply; 53+ messages in thread From: Vladislav Shpilevoy @ 2020-05-13 21:34 UTC (permalink / raw) To: Konstantin Osipov, Sergey Ostanevich, tarantool-patches Thanks for the discussion! On 12/05/2020 19:47, Konstantin Osipov wrote: > * Sergey Ostanevich <sergos@tarantool.org> [20/05/12 19:43]: > >>> 1) It's unclear what happens here if async tx follows a sync tx. >>> Does it wait for the sync tx? This reduces availability for >> >> Definitely yes, unless we keep the 'dirty read' as it is at the moment >> in memtx. This is the essence of the design, and it is temporary until >> the MVCC similar to the vinyl machinery appears. I intentionally didn't >> include this big task into this RFC. >> >> It will provide similar capabilities, although it will keep only >> dependent transactions in the undo log. Also, it looks like it will fit >> well into the machinery of this RFC. > > = reduced availability for all who have at least one sync space. > > If different spaces have different quorum size = quorum size of > the biggest group is effectively used for all spaces. > > Replica-local transactions, e.g. those used by vinyl compaction, > are rolled back if there is no quorum. > > What's the value of this? There is an example when it leaves the database in an inconsistent state, when half of a transaction is applied. I don't know why Sergey didn't add it. I propose to him to extend the RFC with these examples. Since you are not the first person, who finds this strange and wrong. So clearly the RFC still does not explain this moment diligently enough. >>> async txs - so it's hardly acceptable. Besides, with >>> group=local spaces, one can quickly run out of memory for undo. >>> >>> >>> 3) One can quickly run out of memory for undo. Any sync >>> transaction should be capped with a timeout to avoid OOMs. I >>> don't know how many times I should repeat it. The only good >>> solution for load control is in-memory WAL, which will allow to >>> rollback all transactions as soon as network partitioning is >>> detected. >> >> How in-memry WAL can help save on _undo_ memory? >> To rollback whatever amount of transactions one need to store the undo. > > I wrote earlier that it works as a natural failure detector and > throttling mechanism. If > there is no quorum, we can see it immediately by looking at the > number of active subscribers of the in-memory WAL, so do not > accumulate undo. Here we go again ... Talking of throttling. Without in-memory WAL no need for throttling. All is 'slow' by design already, as you think. Talking of failure detection - what??? I don't get it. This is something new. With in-memory relay or without you anyway can see if there is a quorum. This is a matter of API of replication and transaction modules, and their interaction with each other, solved by txn_limbo in my branch. But still, I don't see how knowing number of subscribers helps with the quorum. Subscriber presence does not add to quorums by itself. Anyway every transaction needs to be replicated before you can say that its quorum got +1 replica ack. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [Tarantool-patches] [RFC] Quorum-based synchronous replication 2020-05-13 21:34 ` Vladislav Shpilevoy @ 2020-05-13 23:31 ` Konstantin Osipov 0 siblings, 0 replies; 53+ messages in thread From: Konstantin Osipov @ 2020-05-13 23:31 UTC (permalink / raw) To: Vladislav Shpilevoy; +Cc: tarantool-patches * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [20/05/14 00:37]: > >>> 3) One can quickly run out of memory for undo. Any sync > >>> transaction should be capped with a timeout to avoid OOMs. I > >>> don't know how many times I should repeat it. The only good > >>> solution for load control is in-memory WAL, which will allow to > >>> rollback all transactions as soon as network partitioning is > >>> detected. > >> > >> How in-memry WAL can help save on _undo_ memory? > >> To rollback whatever amount of transactions one need to store the undo. > > > > I wrote earlier that it works as a natural failure detector and > > throttling mechanism. If > > there is no quorum, we can see it immediately by looking at the > > number of active subscribers of the in-memory WAL, so do not > > accumulate undo. > > Here we go again ... > > Talking of throttling. Without in-memory WAL no need for throttling. All is > 'slow' by design already, as you think. What is the limit for transactions in txn_limbo list? How does this limit work? What about the fibers, which are pinned as long as the transaction is not committed? > > Talking of failure detection - what??? I don't get it. This is something new. > With in-memory relay or without you anyway can see if there is a quorum. How do you "see" it? You write to the WAL and wait for acks. You could add a wait timeout, and assume there is no quorum if there are no acks within the timeout. This is not the best strategy, but there is no other. The spec doesn't say even that, it simply says that somehow lack of quorum is detected, but how it is detected is not clear. With in-memory WAL you can afford to wait longer if you have space in the ring buffer, and you know immediately if you shouldn't wait because you see that the ring buffer is full and the majority of subscribers are behind the start of the buffer. > This is a matter of API of replication and transaction modules, and their > interaction with each other, solved by txn_limbo in my branch. How is it "solved"? > But still, I don't see how knowing number of subscribers helps with the > quorum. Subscriber presence does not add to quorums by itself. Anyway every > transaction needs to be replicated before you can say that its quorum got > +1 replica ack. It helps to see quickly absence of the quorum, not presence of it. -- Konstantin Osipov, Moscow, Russia ^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2020-06-12 20:31 UTC | newest] Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-03 21:08 [Tarantool-patches] [RFC] Quorum-based synchronous replication Sergey Ostanevich 2020-04-07 13:02 ` Aleksandr Lyapunov 2020-04-08 9:18 ` Sergey Ostanevich 2020-04-08 14:05 ` Konstantin Osipov 2020-04-08 15:06 ` Sergey Ostanevich 2020-04-14 12:58 ` Sergey Bronnikov 2020-04-14 14:43 ` Sergey Ostanevich 2020-04-15 11:09 ` sergos 2020-04-15 14:50 ` sergos 2020-04-16 7:13 ` Aleksandr Lyapunov 2020-04-17 10:10 ` Konstantin Osipov 2020-04-17 13:45 ` Sergey Ostanevich 2020-04-20 11:20 ` Serge Petrenko 2020-04-20 23:32 ` Vladislav Shpilevoy 2020-04-21 10:49 ` Sergey Ostanevich 2020-04-21 22:17 ` Vladislav Shpilevoy 2020-04-22 16:50 ` Sergey Ostanevich 2020-04-22 20:28 ` Vladislav Shpilevoy 2020-04-23 6:58 ` Konstantin Osipov 2020-04-23 9:14 ` Konstantin Osipov 2020-04-23 11:27 ` Sergey Ostanevich 2020-04-23 11:43 ` Konstantin Osipov 2020-04-23 15:11 ` Sergey Ostanevich 2020-04-23 20:39 ` Konstantin Osipov 2020-04-23 21:38 ` Vladislav Shpilevoy 2020-04-23 22:28 ` Konstantin Osipov 2020-04-30 14:50 ` Sergey Ostanevich 2020-05-06 8:52 ` Konstantin Osipov 2020-05-06 16:39 ` Sergey Ostanevich 2020-05-06 18:44 ` Konstantin Osipov 2020-05-12 15:55 ` Sergey Ostanevich 2020-05-12 16:42 ` Konstantin Osipov 2020-05-13 21:39 ` Vladislav Shpilevoy 2020-05-13 23:54 ` Konstantin Osipov 2020-05-14 20:38 ` Sergey Ostanevich 2020-05-20 20:59 ` Sergey Ostanevich 2020-05-25 23:41 ` Vladislav Shpilevoy 2020-05-27 21:17 ` Sergey Ostanevich 2020-06-09 16:19 ` Sergey Ostanevich 2020-06-11 15:17 ` Vladislav Shpilevoy 2020-06-12 20:31 ` Sergey Ostanevich 2020-05-13 21:36 ` Vladislav Shpilevoy 2020-05-13 23:45 ` Konstantin Osipov 2020-05-06 18:55 ` Konstantin Osipov 2020-05-06 19:10 ` Konstantin Osipov 2020-05-12 16:03 ` Sergey Ostanevich 2020-05-13 21:42 ` Vladislav Shpilevoy 2020-05-14 0:05 ` Konstantin Osipov 2020-05-07 23:01 ` Konstantin Osipov 2020-05-12 16:40 ` Sergey Ostanevich 2020-05-12 17:47 ` Konstantin Osipov 2020-05-13 21:34 ` Vladislav Shpilevoy 2020-05-13 23:31 ` Konstantin Osipov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox