Re: [Tarantool-patches] [PATCH 5/7] replication: send latest effective promote in initial join

[Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Good job on the fixes!

On 17.06.2021 23:00, Serge Petrenko via Tarantool-patches wrote:
> 
> 
> 16.06.2021 00:00, Vladislav Shpilevoy пишет:
>> Thanks for working on this!
>>
>> Hm. The patch makes me think why don't we send the Raft
>> checkpoint on join too?
>>
>> Otherwise it might happen that a replica joined, didn't
>> get the most actual Raft term yet, then the leader died,
>> and the replica would start elections with term 1. Even if
>> the latest term was 1000.
>>
>> Nothing critical, Raft will probably work, but it could
>> be an "optimization"? Also it would be consistent with the
>> libmo state - send all the snapshot data on join.
> I tried to implement such a patch, but faced the following problem:
> 
> Unfortunately, we don't have information about replica's version
> during join, so I can only send raft state based on term > 1.

But you do send the limbo state. Won't it break the old versions
like 1.10?

I tried to replicate 1.10 from the new version and got UNKNOWN_REQUEST_TYPE
error.

2021-06-19 00:50:49.781 [85883] main/105/applier/ applier.cc:345 E> ER_UNKNOWN_REQUEST_TYPE: Unknown request type 31
2021-06-19 00:50:49.781 [85883] main/101/interactive applier.cc:345 E> ER_UNKNOWN_REQUEST_TYPE: Unknown request type 31
2021-06-19 00:50:49.781 [85883] main/101/interactive F> can't initialize storage: Unknown request type 31

And that is fine. Because the replication does not need to be
compatible "forward". You don't need to be able to replicate a
new instance into an old instance.

> Also, while writing a commit message I understood that this patch
> doesn't help much. Even if a node joins, but doesn't subscribe to the
> leader, it will still subscribe to someone else and receive the latest
> Raft state.

And what if it does not? What if there are just 2 nodes?

> After all, our doc states full-mesh is required for Raft to work, so we'll
> have someone else to subscribe to and receive Raft state from for sure.

> The patch's small and simple but I think it's not needed.
> 
> I've made a tiny amendment to this commit though, please find the diff below.

The term limbo affects the filtering. Assume you sent the limbo
state without raft to 2 new replicas. They now has a big limbo term,
say 1000, and raft term 1. Now the master dies.

The replicas start new election. One of them wins with some small
raft term (let it be 10). It starts doing synchro txns. The other node's
applier will turn them all to NOPs, because txn_limbo_is_replica_outdated()
will return true. Even though it is not outdated.

Will it happen?