From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 7FA0A6EC40; Sun, 6 Jun 2021 20:03:49 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7FA0A6EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1622999029; bh=sZyEajSgYv08Oaj6wn0/RSugcL+YT/lO/8wtt+ZplYE=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=GUiPuJuD/TRoqdbR8fn9Fj59EzPyLsdzWXcMXYFij3QYCNV2/xgu/4GzBXJabnifx saDNQwtJQU3XUp+0ehKhus+Ekdd/ukGz/U06N7FnJmR+HnVO9BHOQE9rZGirDicLtu kWN20/TPN1QN/FW22n5hAey70EHfFjR2X35YyYIo= Received: from smtpng3.m.smailru.net (smtpng3.m.smailru.net [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 7FA396EC40 for ; Sun, 6 Jun 2021 20:03:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 7FA396EC40 Received: by smtpng3.m.smailru.net with esmtpa (envelope-from ) id 1lpwBd-0000zB-TG; Sun, 06 Jun 2021 20:03:46 +0300 To: tarantool-patches@dev.tarantool.org, gorcunov@gmail.com, sergepetrenko@tarantool.org References: Message-ID: <3bce2555-0fb2-4edf-3373-d068d99d7309@tarantool.org> Date: Sun, 6 Jun 2021 19:03:44 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9D5B0DA836B685C54907A7AE9C1BA82BC67C1327DFB87C6A6182A05F538085040C8B92B7C2CF43484A07B5F39FF35C15AAFAB7579D4C1B135C7ACDFF392B46457 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7DDEA16837FFA1F5BEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637578F58D66D7052C48638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D879B75FEF002F6D2633767FE0D80B3B61117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6A45692FFBBD75A6A089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CC176FDF34A1E429F023C62C9248AEC65BA4D466E9602D7779C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EFD99FB7B2A39B4961699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3480665FBD8F49180C6C2CA5715D880640775BE54BD3F96D013E9B000880D188B1FED96579476177E81D7E09C32AA3244CE8204F9E29FFB227C8AFDFE6AF36778030363D8B7DA7DD44FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojtoSVbEA8p9DUFitimVzgIQ== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822A836EB2C0EEB111F6456C990D75557C03841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH 7/6] raft: test join to a raft cluster X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" There was a bug that a new replica at join to a Raft cluster sometimes tried to register on a non-leader node which couldn't write to _cluster, so the join failed with ER_READONLY error. Now in scope of #5613 the algorithm of join-master selection is changed. A new node looks for writable members of the cluster to use a join-master. It will not choose a follower if there is a leader. Closes #6127 --- .../unreleased/gh-6127-raft-join-new.md | 4 + test/replication/gh-6127-master1.lua | 15 +++ test/replication/gh-6127-master2.lua | 13 +++ test/replication/gh-6127-raft-join-new.result | 105 ++++++++++++++++++ .../gh-6127-raft-join-new.test.lua | 41 +++++++ test/replication/gh-6127-replica.lua | 9 ++ 6 files changed, 187 insertions(+) create mode 100644 changelogs/unreleased/gh-6127-raft-join-new.md create mode 100644 test/replication/gh-6127-master1.lua create mode 100644 test/replication/gh-6127-master2.lua create mode 100644 test/replication/gh-6127-raft-join-new.result create mode 100644 test/replication/gh-6127-raft-join-new.test.lua create mode 100644 test/replication/gh-6127-replica.lua diff --git a/changelogs/unreleased/gh-6127-raft-join-new.md b/changelogs/unreleased/gh-6127-raft-join-new.md new file mode 100644 index 000000000..a2d898df0 --- /dev/null +++ b/changelogs/unreleased/gh-6127-raft-join-new.md @@ -0,0 +1,4 @@ +## bugfix/raft + +* Fixed an error when a new replica in a Raft cluster could try to join from a + follower instead of a leader and failed with an error `ER_READONLY` (gh-6127). diff --git a/test/replication/gh-6127-master1.lua b/test/replication/gh-6127-master1.lua new file mode 100644 index 000000000..708574322 --- /dev/null +++ b/test/replication/gh-6127-master1.lua @@ -0,0 +1,15 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) +box.cfg({ + listen = 'unix/:./master1.sock', + replication = { + 'unix/:./master1.sock', + 'unix/:./master2.sock' + }, + election_mode = 'candidate', + election_timeout = 0.1, + instance_uuid = '10f9828d-b5d5-46a9-b698-ddac7cce5e27', +}) +box.ctl.wait_rw() +box.schema.user.grant('guest', 'super') diff --git a/test/replication/gh-6127-master2.lua b/test/replication/gh-6127-master2.lua new file mode 100644 index 000000000..1851070c7 --- /dev/null +++ b/test/replication/gh-6127-master2.lua @@ -0,0 +1,13 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) +box.cfg({ + listen = 'unix/:./master2.sock', + replication = { + 'unix/:./master1.sock', + 'unix/:./master2.sock' + }, + election_mode = 'voter', + election_timeout = 0.1, + instance_uuid = '20f9828d-b5d5-46a9-b698-ddac7cce5e27', +}) diff --git a/test/replication/gh-6127-raft-join-new.result b/test/replication/gh-6127-raft-join-new.result new file mode 100644 index 000000000..be6f8489b --- /dev/null +++ b/test/replication/gh-6127-raft-join-new.result @@ -0,0 +1,105 @@ +-- test-run result file version 2 +test_run = require('test_run').new() + | --- + | ... + +-- +-- gh-6127: the algorithm selecting a node from which to join to a replicaset +-- should take into account who is the leader (is writable and can write to +-- _cluster) and who is a follower/candidate. +-- +test_run:cmd('create server master1 with script="replication/gh-6127-master1.lua"') + | --- + | - true + | ... +test_run:cmd('start server master1 with wait=False') + | --- + | - true + | ... +test_run:cmd('create server master2 with script="replication/gh-6127-master2.lua"') + | --- + | - true + | ... +test_run:cmd('start server master2') + | --- + | - true + | ... + +test_run:switch('master1') + | --- + | - true + | ... +box.cfg{election_mode = 'voter'} + | --- + | ... +test_run:switch('master2') + | --- + | - true + | ... +-- Perform manual election because it is faster - the automatic one still tries +-- to wait for 'death timeout' first which is several seconds. +box.cfg{ \ + election_mode = 'manual', \ + election_timeout = 0.1, \ +} + | --- + | ... +box.ctl.promote() + | --- + | ... +box.ctl.wait_rw() + | --- + | ... +-- Make sure the other node received the promotion row. Vclocks now should be +-- equal so the new node would select only using read-only state and min UUID. +test_run:wait_lsn('master1', 'master2') + | --- + | ... + +-- Min UUID is master1, but it is not writable. Therefore must join from +-- master2. +test_run:cmd('create server replica with script="replication/gh-6127-replica.lua"') + | --- + | - true + | ... +test_run:cmd('start server replica') + | --- + | - true + | ... +test_run:switch('replica') + | --- + | - true + | ... +assert(box.info.leader ~= 0) + | --- + | - true + | ... + +test_run:switch('default') + | --- + | - true + | ... +test_run:cmd('stop server replica') + | --- + | - true + | ... +test_run:cmd('delete server replica') + | --- + | - true + | ... +test_run:cmd('stop server master2') + | --- + | - true + | ... +test_run:cmd('delete server master2') + | --- + | - true + | ... +test_run:cmd('stop server master1') + | --- + | - true + | ... +test_run:cmd('delete server master1') + | --- + | - true + | ... diff --git a/test/replication/gh-6127-raft-join-new.test.lua b/test/replication/gh-6127-raft-join-new.test.lua new file mode 100644 index 000000000..3e0e9f226 --- /dev/null +++ b/test/replication/gh-6127-raft-join-new.test.lua @@ -0,0 +1,41 @@ +test_run = require('test_run').new() + +-- +-- gh-6127: the algorithm selecting a node from which to join to a replicaset +-- should take into account who is the leader (is writable and can write to +-- _cluster) and who is a follower/candidate. +-- +test_run:cmd('create server master1 with script="replication/gh-6127-master1.lua"') +test_run:cmd('start server master1 with wait=False') +test_run:cmd('create server master2 with script="replication/gh-6127-master2.lua"') +test_run:cmd('start server master2') + +test_run:switch('master1') +box.cfg{election_mode = 'voter'} +test_run:switch('master2') +-- Perform manual election because it is faster - the automatic one still tries +-- to wait for 'death timeout' first which is several seconds. +box.cfg{ \ + election_mode = 'manual', \ + election_timeout = 0.1, \ +} +box.ctl.promote() +box.ctl.wait_rw() +-- Make sure the other node received the promotion row. Vclocks now should be +-- equal so the new node would select only using read-only state and min UUID. +test_run:wait_lsn('master1', 'master2') + +-- Min UUID is master1, but it is not writable. Therefore must join from +-- master2. +test_run:cmd('create server replica with script="replication/gh-6127-replica.lua"') +test_run:cmd('start server replica') +test_run:switch('replica') +assert(box.info.leader ~= 0) + +test_run:switch('default') +test_run:cmd('stop server replica') +test_run:cmd('delete server replica') +test_run:cmd('stop server master2') +test_run:cmd('delete server master2') +test_run:cmd('stop server master1') +test_run:cmd('delete server master1') diff --git a/test/replication/gh-6127-replica.lua b/test/replication/gh-6127-replica.lua new file mode 100644 index 000000000..9f4c35ecd --- /dev/null +++ b/test/replication/gh-6127-replica.lua @@ -0,0 +1,9 @@ +#!/usr/bin/env tarantool + +require('console').listen(os.getenv('ADMIN')) +box.cfg({ + replication = { + 'unix/:./master1.sock', + 'unix/:./master2.sock' + }, +}) -- 2.24.3 (Apple Git-128)