Re: [Tarantool-patches] [PATCH] raft: make sure the leader stays ro till it clears the limbo

[Vladislav Shpilevoy <v.shpilevoy@tarantool.org>]

Hi! Thanks for the patch!

On 24.11.2020 14:18, Serge Petrenko wrote:
> When running a cluster with leader election, its useful to wait till the
> instance is writeable to determine that it has become a leader. However,
> sometimes the instance fails to write data right after transitioning to
> leader because its limbo still contains pending transactions from the
> old leader. Make sure the instance deals with pending transactions first
> and becomes writeable only once the limbo is empty.

I just realized one thing. We can add a function txn_limbo_is_ro(),
like we did with raft_is_ro(), account it in box_update_ro_summary(),
and call box_update_ro_summary() when we see that the limbo is emptied,
or when its ownership changes to a different instance.

Probably would be simpler, and also we could make it work with manual
election! So users could call box.ctl.wait_rw() even without using raft!

To show concrete error if somebody still tries to write, we could
patch box_check_writable() to show the reason why the instance is not
writable. We will do it anyway for raft, to tell the users the real
leader in case they are trying to write on a replica. In scope of
https://github.com/tarantool/tarantool/issues/5568.

Your version of the patch also looks good.

What do you think?