Tarantool development patches archive
 help / color / mirror / Atom feed
From: Serge Petrenko via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>,
	tarantool-patches@dev.tarantool.org, gorcunov@gmail.com
Subject: Re: [Tarantool-patches] [PATCH 7/6] raft: test join to a raft cluster
Date: Thu, 10 Jun 2021 17:17:41 +0300
Message-ID: <d430615e-1b1f-0df1-3aac-08b2c5f0677a@tarantool.org> (raw)
In-Reply-To: <3bce2555-0fb2-4edf-3373-d068d99d7309@tarantool.org>



06.06.2021 20:03, Vladislav Shpilevoy пишет:
> There was a bug that a new replica at join to a Raft cluster
> sometimes tried to register on a non-leader node which couldn't
> write to _cluster, so the join failed with ER_READONLY error.
>
> Now in scope of #5613 the algorithm of join-master selection is
> changed. A new node looks for writable members of the cluster to
> use a join-master. It will not choose a follower if there is a
> leader.
>
> Closes #6127

Thanks for working on this! LGTM.

> ---
>   .../unreleased/gh-6127-raft-join-new.md       |   4 +
>   test/replication/gh-6127-master1.lua          |  15 +++
>   test/replication/gh-6127-master2.lua          |  13 +++
>   test/replication/gh-6127-raft-join-new.result | 105 ++++++++++++++++++
>   .../gh-6127-raft-join-new.test.lua            |  41 +++++++
>   test/replication/gh-6127-replica.lua          |   9 ++
>   6 files changed, 187 insertions(+)
>   create mode 100644 changelogs/unreleased/gh-6127-raft-join-new.md
>   create mode 100644 test/replication/gh-6127-master1.lua
>   create mode 100644 test/replication/gh-6127-master2.lua
>   create mode 100644 test/replication/gh-6127-raft-join-new.result
>   create mode 100644 test/replication/gh-6127-raft-join-new.test.lua
>   create mode 100644 test/replication/gh-6127-replica.lua
>
> diff --git a/changelogs/unreleased/gh-6127-raft-join-new.md b/changelogs/unreleased/gh-6127-raft-join-new.md
> new file mode 100644
> index 000000000..a2d898df0
> --- /dev/null
> +++ b/changelogs/unreleased/gh-6127-raft-join-new.md
> @@ -0,0 +1,4 @@
> +## bugfix/raft
> +
> +* Fixed an error when a new replica in a Raft cluster could try to join from a
> +  follower instead of a leader and failed with an error `ER_READONLY` (gh-6127).
> diff --git a/test/replication/gh-6127-master1.lua b/test/replication/gh-6127-master1.lua
> new file mode 100644
> index 000000000..708574322
> --- /dev/null
> +++ b/test/replication/gh-6127-master1.lua
> @@ -0,0 +1,15 @@
> +#!/usr/bin/env tarantool
> +
> +require('console').listen(os.getenv('ADMIN'))
> +box.cfg({
> +    listen = 'unix/:./master1.sock',
> +    replication = {
> +        'unix/:./master1.sock',
> +        'unix/:./master2.sock'
> +    },
> +    election_mode = 'candidate',
> +    election_timeout = 0.1,
> +    instance_uuid = '10f9828d-b5d5-46a9-b698-ddac7cce5e27',
> +})
> +box.ctl.wait_rw()
> +box.schema.user.grant('guest', 'super')
> diff --git a/test/replication/gh-6127-master2.lua b/test/replication/gh-6127-master2.lua
> new file mode 100644
> index 000000000..1851070c7
> --- /dev/null
> +++ b/test/replication/gh-6127-master2.lua
> @@ -0,0 +1,13 @@
> +#!/usr/bin/env tarantool
> +
> +require('console').listen(os.getenv('ADMIN'))
> +box.cfg({
> +    listen = 'unix/:./master2.sock',
> +    replication = {
> +        'unix/:./master1.sock',
> +        'unix/:./master2.sock'
> +    },
> +    election_mode = 'voter',
> +    election_timeout = 0.1,
> +    instance_uuid = '20f9828d-b5d5-46a9-b698-ddac7cce5e27',
> +})
> diff --git a/test/replication/gh-6127-raft-join-new.result b/test/replication/gh-6127-raft-join-new.result
> new file mode 100644
> index 000000000..be6f8489b
> --- /dev/null
> +++ b/test/replication/gh-6127-raft-join-new.result
> @@ -0,0 +1,105 @@
> +-- test-run result file version 2
> +test_run = require('test_run').new()
> + | ---
> + | ...
> +
> +--
> +-- gh-6127: the algorithm selecting a node from which to join to a replicaset
> +-- should take into account who is the leader (is writable and can write to
> +-- _cluster) and who is a follower/candidate.
> +--
> +test_run:cmd('create server master1 with script="replication/gh-6127-master1.lua"')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('start server master1 with wait=False')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('create server master2 with script="replication/gh-6127-master2.lua"')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('start server master2')
> + | ---
> + | - true
> + | ...
> +
> +test_run:switch('master1')
> + | ---
> + | - true
> + | ...
> +box.cfg{election_mode = 'voter'}
> + | ---
> + | ...
> +test_run:switch('master2')
> + | ---
> + | - true
> + | ...
> +-- Perform manual election because it is faster - the automatic one still tries
> +-- to wait for 'death timeout' first which is several seconds.
> +box.cfg{                                                                        \
> +    election_mode = 'manual',                                                   \
> +    election_timeout = 0.1,                                                     \
> +}
> + | ---
> + | ...
> +box.ctl.promote()
> + | ---
> + | ...
> +box.ctl.wait_rw()
> + | ---
> + | ...
> +-- Make sure the other node received the promotion row. Vclocks now should be
> +-- equal so the new node would select only using read-only state and min UUID.
> +test_run:wait_lsn('master1', 'master2')
> + | ---
> + | ...
> +
> +-- Min UUID is master1, but it is not writable. Therefore must join from
> +-- master2.
> +test_run:cmd('create server replica with script="replication/gh-6127-replica.lua"')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('start server replica')
> + | ---
> + | - true
> + | ...
> +test_run:switch('replica')
> + | ---
> + | - true
> + | ...
> +assert(box.info.leader ~= 0)
> + | ---
> + | - true
> + | ...
> +
> +test_run:switch('default')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('stop server replica')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('delete server replica')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('stop server master2')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('delete server master2')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('stop server master1')
> + | ---
> + | - true
> + | ...
> +test_run:cmd('delete server master1')
> + | ---
> + | - true
> + | ...
> diff --git a/test/replication/gh-6127-raft-join-new.test.lua b/test/replication/gh-6127-raft-join-new.test.lua
> new file mode 100644
> index 000000000..3e0e9f226
> --- /dev/null
> +++ b/test/replication/gh-6127-raft-join-new.test.lua
> @@ -0,0 +1,41 @@
> +test_run = require('test_run').new()
> +
> +--
> +-- gh-6127: the algorithm selecting a node from which to join to a replicaset
> +-- should take into account who is the leader (is writable and can write to
> +-- _cluster) and who is a follower/candidate.
> +--
> +test_run:cmd('create server master1 with script="replication/gh-6127-master1.lua"')
> +test_run:cmd('start server master1 with wait=False')
> +test_run:cmd('create server master2 with script="replication/gh-6127-master2.lua"')
> +test_run:cmd('start server master2')
> +
> +test_run:switch('master1')
> +box.cfg{election_mode = 'voter'}
> +test_run:switch('master2')
> +-- Perform manual election because it is faster - the automatic one still tries
> +-- to wait for 'death timeout' first which is several seconds.
> +box.cfg{                                                                        \
> +    election_mode = 'manual',                                                   \
> +    election_timeout = 0.1,                                                     \
> +}
> +box.ctl.promote()
> +box.ctl.wait_rw()
> +-- Make sure the other node received the promotion row. Vclocks now should be
> +-- equal so the new node would select only using read-only state and min UUID.
> +test_run:wait_lsn('master1', 'master2')
> +
> +-- Min UUID is master1, but it is not writable. Therefore must join from
> +-- master2.
> +test_run:cmd('create server replica with script="replication/gh-6127-replica.lua"')
> +test_run:cmd('start server replica')
> +test_run:switch('replica')
> +assert(box.info.leader ~= 0)
> +
> +test_run:switch('default')
> +test_run:cmd('stop server replica')
> +test_run:cmd('delete server replica')
> +test_run:cmd('stop server master2')
> +test_run:cmd('delete server master2')
> +test_run:cmd('stop server master1')
> +test_run:cmd('delete server master1')
> diff --git a/test/replication/gh-6127-replica.lua b/test/replication/gh-6127-replica.lua
> new file mode 100644
> index 000000000..9f4c35ecd
> --- /dev/null
> +++ b/test/replication/gh-6127-replica.lua
> @@ -0,0 +1,9 @@
> +#!/usr/bin/env tarantool
> +
> +require('console').listen(os.getenv('ADMIN'))
> +box.cfg({
> +    replication = {
> +        'unix/:./master1.sock',
> +        'unix/:./master2.sock'
> +    },
> +})

-- 
Serge Petrenko


  parent reply	other threads:[~2021-06-10 14:17 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-04 23:37 [Tarantool-patches] [PATCH 0/6] Instance join should prefer booted instances Vladislav Shpilevoy via Tarantool-patches
2021-06-04 23:37 ` [Tarantool-patches] [PATCH 1/6] replication: refactor replicaset_leader() Vladislav Shpilevoy via Tarantool-patches
2021-06-10 13:54   ` Serge Petrenko via Tarantool-patches
2021-06-04 23:37 ` [Tarantool-patches] [PATCH 2/6] replication: ballot.is_ro -> is_ro_cfg Vladislav Shpilevoy via Tarantool-patches
2021-06-10 13:56   ` Serge Petrenko via Tarantool-patches
2021-06-04 23:37 ` [Tarantool-patches] [PATCH 3/6] replication: ballot.is_loading -> is_ro Vladislav Shpilevoy via Tarantool-patches
2021-06-10 13:58   ` Serge Petrenko via Tarantool-patches
2021-06-04 23:37 ` [Tarantool-patches] [PATCH 4/6] replication: introduce ballot.is_booted Vladislav Shpilevoy via Tarantool-patches
2021-06-10 14:02   ` Serge Petrenko via Tarantool-patches
2021-06-04 23:37 ` [Tarantool-patches] [PATCH 5/6] replication: use 'score' to find a join-master Vladislav Shpilevoy via Tarantool-patches
2021-06-10 14:06   ` Serge Petrenko via Tarantool-patches
2021-06-10 15:02   ` Cyrill Gorcunov via Tarantool-patches
2021-06-10 20:09     ` Vladislav Shpilevoy via Tarantool-patches
2021-06-10 21:28       ` Cyrill Gorcunov via Tarantool-patches
2021-06-04 23:38 ` [Tarantool-patches] [PATCH 6/6] replication: prefer to join from booted replicas Vladislav Shpilevoy via Tarantool-patches
2021-06-06 17:06   ` Vladislav Shpilevoy via Tarantool-patches
2021-06-10 14:14   ` Serge Petrenko via Tarantool-patches
2021-06-06 17:03 ` [Tarantool-patches] [PATCH 7/6] raft: test join to a raft cluster Vladislav Shpilevoy via Tarantool-patches
2021-06-06 23:01   ` Vladislav Shpilevoy via Tarantool-patches
2021-06-10 14:17   ` Serge Petrenko via Tarantool-patches [this message]
2021-06-10 15:03 ` [Tarantool-patches] [PATCH 0/6] Instance join should prefer booted instances Cyrill Gorcunov via Tarantool-patches
2021-06-11 20:56 ` Vladislav Shpilevoy via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d430615e-1b1f-0df1-3aac-08b2c5f0677a@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=gorcunov@gmail.com \
    --cc=sergepetrenko@tarantool.org \
    --cc=v.shpilevoy@tarantool.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Tarantool development patches archive

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://lists.tarantool.org/tarantool-patches/0 tarantool-patches/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tarantool-patches tarantool-patches/ https://lists.tarantool.org/tarantool-patches \
		tarantool-patches@dev.tarantool.org.
	public-inbox-index tarantool-patches

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git