Re: [Tarantool-patches] [PATCH 5/7] replication: send latest effective promote in initial join

[Serge Petrenko via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

19.06.2021 01:52, Vladislav Shpilevoy пишет:
> Good job on the fixes!

Thanks for the review!

>
> On 17.06.2021 23:00, Serge Petrenko via Tarantool-patches wrote:
>>
>> 16.06.2021 00:00, Vladislav Shpilevoy пишет:
>>> Thanks for working on this!
>>>
>>> Hm. The patch makes me think why don't we send the Raft
>>> checkpoint on join too?
>>>
>>> Otherwise it might happen that a replica joined, didn't
>>> get the most actual Raft term yet, then the leader died,
>>> and the replica would start elections with term 1. Even if
>>> the latest term was 1000.
>>>
>>> Nothing critical, Raft will probably work, but it could
>>> be an "optimization"? Also it would be consistent with the
>>> libmo state - send all the snapshot data on join.
>> I tried to implement such a patch, but faced the following problem:
>>
>> Unfortunately, we don't have information about replica's version
>> during join, so I can only send raft state based on term > 1.
> But you do send the limbo state. Won't it break the old versions
> like 1.10?

I only send it if limbo is used. I.e. when limbo->owner_id != 0.

I've just realized, the problem is not only with versions like 1.10. 
It's the same for
versions like 2.8.1, 2.7.2 and so on, which will error with
UNKNOWN_REQUEST_TYPE, when receive limbo or raft state in initial join 
stream.

What can we do about that?

>
> I tried to replicate 1.10 from the new version and got UNKNOWN_REQUEST_TYPE
> error.
>
> 2021-06-19 00:50:49.781 [85883] main/105/applier/ applier.cc:345 E> ER_UNKNOWN_REQUEST_TYPE: Unknown request type 31
> 2021-06-19 00:50:49.781 [85883] main/101/interactive applier.cc:345 E> ER_UNKNOWN_REQUEST_TYPE: Unknown request type 31
> 2021-06-19 00:50:49.781 [85883] main/101/interactive F> can't initialize storage: Unknown request type 31
>
> And that is fine. Because the replication does not need to be
> compatible "forward". You don't need to be able to replicate a
> new instance into an old instance.
>
>> Also, while writing a commit message I understood that this patch
>> doesn't help much. Even if a node joins, but doesn't subscribe to the
>> leader, it will still subscribe to someone else and receive the latest
>> Raft state.
> And what if it does not? What if there are just 2 nodes?
>
>> After all, our doc states full-mesh is required for Raft to work, so we'll
>> have someone else to subscribe to and receive Raft state from for sure.
>> The patch's small and simple but I think it's not needed.
>>
>> I've made a tiny amendment to this commit though, please find the diff below.
> The term limbo affects the filtering. Assume you sent the limbo
> state without raft to 2 new replicas. They now has a big limbo term,
> say 1000, and raft term 1. Now the master dies.
>
> The replicas start new election. One of them wins with some small
> raft term (let it be 10). It starts doing synchro txns. The other node's
> applier will turn them all to NOPs, because txn_limbo_is_replica_outdated()
> will return true. Even though it is not outdated.
>
> Will it happen?

Looks like this may happen, indeed.

-- 
Serge Petrenko