[PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Serge Petrenko]

One possible case when two applier errors happen one after another
wasn't handled in replica_on_applier_disconnect(), which lead to
occasional test failures and crashes. Handle this case.

Part of #3510
---
This patch fixes an assertion fail, submitted by @locker in issue comments.
I wasn't able to reproduce 2 failures reported in the issue itself, and asked
for comments, but got no answer. I also couldn't fix the latter 2
failures just by looking at code.

https://github.com/tarantool/tarantool/tree/sergepetrenko/gh-3510-replication-asserts-fail
https://github.com/tarantool/tarantool/issues/3510

 src/box/replication.cc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/src/box/replication.cc b/src/box/replication.cc
index 26bbbe32a..0efbd7c0e 100644
--- a/src/box/replication.cc
+++ b/src/box/replication.cc
@@ -350,6 +350,10 @@ replica_on_applier_disconnect(struct replica *replica)
 		assert(replicaset.applier.connected > 0);
 		replicaset.applier.connected--;
 		break;
+	case APPLIER_LOADING:
+		assert(replicaset.applier.loading > 0);
+		replicaset.applier.loading--;
+		break;
 	case APPLIER_DISCONNECTED:
 		break;
 	default:
-- 
2.15.2 (Apple Git-101.1)

Re: [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Vladimir Davydov]

Looks good.

On Fri, Aug 03, 2018 at 08:59:05AM +0300, Serge Petrenko wrote:
> One possible case when two applier errors happen one after another
> wasn't handled in replica_on_applier_disconnect(), which lead to
> occasional test failures and crashes. Handle this case.
> 
> Part of #3510
> ---
> This patch fixes an assertion fail, submitted by @locker in issue comments.
> I wasn't able to reproduce 2 failures reported in the issue itself, and asked
> for comments, but got no answer. I also couldn't fix the latter 2
> failures just by looking at code.
> 
> https://github.com/tarantool/tarantool/tree/sergepetrenko/gh-3510-replication-asserts-fail
> https://github.com/tarantool/tarantool/issues/3510
> 
>  src/box/replication.cc | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/src/box/replication.cc b/src/box/replication.cc
> index 26bbbe32a..0efbd7c0e 100644
> --- a/src/box/replication.cc
> +++ b/src/box/replication.cc
> @@ -350,6 +350,10 @@ replica_on_applier_disconnect(struct replica *replica)
>  		assert(replicaset.applier.connected > 0);
>  		replicaset.applier.connected--;
>  		break;
> +	case APPLIER_LOADING:
> +		assert(replicaset.applier.loading > 0);
> +		replicaset.applier.loading--;
> +		break;
>  	case APPLIER_DISCONNECTED:
>  		break;
>  	default:

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Kirill Yukhin]

Hello Serge,
On 03 авг 08:59, Serge Petrenko wrote:
> One possible case when two applier errors happen one after another
> wasn't handled in replica_on_applier_disconnect(), which lead to
> occasional test failures and crashes. Handle this case.
> 
> Part of #3510
> ---
> This patch fixes an assertion fail, submitted by @locker in issue comments.
> I wasn't able to reproduce 2 failures reported in the issue itself, and asked
> for comments, but got no answer. I also couldn't fix the latter 2
> failures just by looking at code.
> 
> https://github.com/tarantool/tarantool/tree/sergepetrenko/gh-3510-replication-asserts-fail
> https://github.com/tarantool/tarantool/issues/3510
Could you pls prepare a regression test as well?

--
Regards, Kirill Yukhin

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Sergey Petrenko]

Hi!

> 3 авг. 2018 г., в 18:57, Kirill Yukhin <kyukhin@tarantool.org> написал(а):
> 
> Hello Serge,
> On 03 авг 08:59, Serge Petrenko wrote:
>> One possible case when two applier errors happen one after another
>> wasn't handled in replica_on_applier_disconnect(), which lead to
>> occasional test failures and crashes. Handle this case.
>> 
>> Part of #3510
>> ---
>> This patch fixes an assertion fail, submitted by @locker in issue comments.
>> I wasn't able to reproduce 2 failures reported in the issue itself, and asked
>> for comments, but got no answer. I also couldn't fix the latter 2
>> failures just by looking at code.
>> 
>> https://github.com/tarantool/tarantool/tree/sergepetrenko/gh-3510-replication-asserts-fail
>> https://github.com/tarantool/tarantool/issues/3510
> Could you pls prepare a regression test as well?

Added a test. It fails with assertion(0) before my patch and passes with my patch.
Here’s new diff:

 src/box/replication.cc         |  4 ++++
 test/replication/er_load.lua   | 23 +++++++++++++++++++++++
 test/replication/er_load1.lua  |  1 +
 test/replication/er_load2.lua  |  1 +
 test/replication/misc.result   | 39 +++++++++++++++++++++++++++++++++++++++
 test/replication/misc.test.lua | 12 ++++++++++++
 6 files changed, 80 insertions(+)
 create mode 100644 test/replication/er_load.lua
 create mode 120000 test/replication/er_load1.lua
 create mode 120000 test/replication/er_load2.lua

diff --git a/src/box/replication.cc b/src/box/replication.cc
index 26bbbe32a..0efbd7c0e 100644
--- a/src/box/replication.cc
+++ b/src/box/replication.cc
@@ -350,6 +350,10 @@ replica_on_applier_disconnect(struct replica *replica)
 		assert(replicaset.applier.connected > 0);
 		replicaset.applier.connected--;
 		break;
+	case APPLIER_LOADING:
+		assert(replicaset.applier.loading > 0);
+		replicaset.applier.loading--;
+		break;
 	case APPLIER_DISCONNECTED:
 		break;
 	default:
diff --git a/test/replication/er_load.lua b/test/replication/er_load.lua
new file mode 100644
index 000000000..0db8c9cfa
--- /dev/null
+++ b/test/replication/er_load.lua
@@ -0,0 +1,23 @@
+#!/usr/bin/env tarantool
+
+-- get instance id from filename (er_load1.lua => 1)
+local INSTANCE_ID = string.match(arg[0], '%d')
+
+local SOCKET_DIR =  require('fio').cwd()
+local function instance_uri(instance_id)
+    return SOCKET_DIR..'/er_load'..instance_id..'.sock'
+end
+
+require('console').listen(os.getenv('ADMIN'))
+
+box.cfg{
+    listen = instance_uri(INSTANCE_ID);
+    replication = {
+	instance_uri(INSTANCE_ID),
+	'noone:pass@'..instance_uri(INSTANCE_ID % 2 + 1)
+    }
+}
+
+box.once("leader", function()
+    box.schema.user.grant('guest', 'replication')
+end)
diff --git a/test/replication/er_load1.lua b/test/replication/er_load1.lua
new file mode 120000
index 000000000..18f7ffa5a
--- /dev/null
+++ b/test/replication/er_load1.lua
@@ -0,0 +1 @@
+er_load.lua
\ No newline at end of file
diff --git a/test/replication/er_load2.lua b/test/replication/er_load2.lua
new file mode 120000
index 000000000..18f7ffa5a
--- /dev/null
+++ b/test/replication/er_load2.lua
@@ -0,0 +1 @@
+er_load.lua
\ No newline at end of file
diff --git a/test/replication/misc.result b/test/replication/misc.result
index ff0dbf549..35b51085f 100644
--- a/test/replication/misc.result
+++ b/test/replication/misc.result
@@ -208,3 +208,42 @@ test_run:drop_cluster(SERVERS)
 box.schema.user.revoke('guest', 'replication')
 ---
 ...
+-- gh-3510 assertion failure in replica_on_applier_disconnect()
+test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
+---
+- true
+...
+test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
+---
+- true
+...
+test_run:cmd('start server er_load1 with wait=False, wait_load=False')
+---
+- true
+...
+test_run:cmd('start server er_load2 with wait=False, wait_load=False')
+---
+- true
+...
+require('fiber').sleep(0.5)
+---
+...
+test_run:cmd('stop server er_load1')
+---
+- true
+...
+require('fiber').sleep(1)
+---
+...
+test_run:cmd('stop server er_load2')
+---
+- true
+...
+test_run:cmd('cleanup server er_load1')
+---
+- true
+...
+test_run:cmd('cleanup server er_load2')
+---
+- true
+...
diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
index c05e52165..27c1a4821 100644
--- a/test/replication/misc.test.lua
+++ b/test/replication/misc.test.lua
@@ -81,3 +81,15 @@ test_run:cmd("switch default")
 test_run:drop_cluster(SERVERS)
 
 box.schema.user.revoke('guest', 'replication')
+
+-- gh-3510 assertion failure in replica_on_applier_disconnect()
+test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
+test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
+test_run:cmd('start server er_load1 with wait=False, wait_load=False')
+test_run:cmd('start server er_load2 with wait=False, wait_load=False')
+require('fiber').sleep(0.5)
+test_run:cmd('stop server er_load1')
+require('fiber').sleep(1)
+test_run:cmd('stop server er_load2')
+test_run:cmd('cleanup server er_load1')
+test_run:cmd('cleanup server er_load2')
-- 
2.15.2 (Apple Git-101.1)

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Vladimir Davydov]

On Mon, Aug 06, 2018 at 05:14:05PM +0300, Sergey Petrenko wrote:
> diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
> index c05e52165..27c1a4821 100644
> --- a/test/replication/misc.test.lua
> +++ b/test/replication/misc.test.lua
> @@ -81,3 +81,15 @@ test_run:cmd("switch default")
>  test_run:drop_cluster(SERVERS)
>  
>  box.schema.user.revoke('guest', 'replication')
> +
> +-- gh-3510 assertion failure in replica_on_applier_disconnect()
> +test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
> +test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
> +test_run:cmd('start server er_load1 with wait=False, wait_load=False')
> +test_run:cmd('start server er_load2 with wait=False, wait_load=False')
> +require('fiber').sleep(0.5)
> +test_run:cmd('stop server er_load1')
> +require('fiber').sleep(1)

<kostja-mode>
  These sleep()'s are way too long. They will make our tests run forever.
  Please change to 0.00001 or smaller or rewrite the test so that you do
  not need them.
</kostja-mode>

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Sergey Petrenko]

> 7 авг. 2018 г., в 19:50, Vladimir Davydov <vdavydov.dev@gmail.com> написал(а):
> 
> On Mon, Aug 06, 2018 at 05:14:05PM +0300, Sergey Petrenko wrote:
>> diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
>> index c05e52165..27c1a4821 100644
>> --- a/test/replication/misc.test.lua
>> +++ b/test/replication/misc.test.lua
>> @@ -81,3 +81,15 @@ test_run:cmd("switch default")
>> test_run:drop_cluster(SERVERS)
>> 
>> box.schema.user.revoke('guest', 'replication')
>> +
>> +-- gh-3510 assertion failure in replica_on_applier_disconnect()
>> +test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
>> +test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
>> +test_run:cmd('start server er_load1 with wait=False, wait_load=False')
>> +test_run:cmd('start server er_load2 with wait=False, wait_load=False')
>> +require('fiber').sleep(0.5)
>> +test_run:cmd('stop server er_load1')
>> +require('fiber').sleep(1)
> 
> <kostja-mode>
>  These sleep()'s are way too long. They will make our tests run forever.
>  Please change to 0.00001 or smaller or rewrite the test so that you do
>  not need them.
> </kostja-mode>

Hi! Thank you for review.
Fixed. Here’s the new diff:

 src/box/replication.cc         |  4 ++++
 test/replication/er_load.lua   | 25 +++++++++++++++++++++++++
 test/replication/er_load1.lua  |  1 +
 test/replication/er_load2.lua  |  1 +
 test/replication/misc.result   | 32 ++++++++++++++++++++++++++++++++
 test/replication/misc.test.lua | 12 ++++++++++++
 6 files changed, 75 insertions(+)
 create mode 100644 test/replication/er_load.lua
 create mode 120000 test/replication/er_load1.lua
 create mode 120000 test/replication/er_load2.lua

diff --git a/src/box/replication.cc b/src/box/replication.cc
index 26bbbe32a..0efbd7c0e 100644
--- a/src/box/replication.cc
+++ b/src/box/replication.cc
@@ -350,6 +350,10 @@ replica_on_applier_disconnect(struct replica *replica)
 		assert(replicaset.applier.connected > 0);
 		replicaset.applier.connected--;
 		break;
+	case APPLIER_LOADING:
+		assert(replicaset.applier.loading > 0);
+		replicaset.applier.loading--;
+		break;
 	case APPLIER_DISCONNECTED:
 		break;
 	default:
diff --git a/test/replication/er_load.lua b/test/replication/er_load.lua
new file mode 100644
index 000000000..0515b3cce
--- /dev/null
+++ b/test/replication/er_load.lua
@@ -0,0 +1,25 @@
+#!/usr/bin/env tarantool
+
+-- get instance id from filename (er_load1.lua => 1)
+local INSTANCE_ID = string.match(arg[0], '%d')
+
+local SOCKET_DIR =  require('fio').cwd()
+local function instance_uri(instance_id)
+    return SOCKET_DIR..'/er_load'..instance_id..'.sock'
+end
+
+require('console').listen(os.getenv('ADMIN'))
+
+box.cfg{
+    listen = instance_uri(INSTANCE_ID);
+    replication = {
+	instance_uri(INSTANCE_ID),
+	instance_uri(INSTANCE_ID % 2 + 1)
+    },
+    replication_timeout = 0.01,
+    read_only = INSTANCE_ID == '2'
+}
+box.once('bootstrap', function()
+    box.schema.user.grant('guest', 'replication')
+    box.space._cluster:delete(2)
+end)
diff --git a/test/replication/er_load1.lua b/test/replication/er_load1.lua
new file mode 120000
index 000000000..18f7ffa5a
--- /dev/null
+++ b/test/replication/er_load1.lua
@@ -0,0 +1 @@
+er_load.lua
\ No newline at end of file
diff --git a/test/replication/er_load2.lua b/test/replication/er_load2.lua
new file mode 120000
index 000000000..18f7ffa5a
--- /dev/null
+++ b/test/replication/er_load2.lua
@@ -0,0 +1 @@
+er_load.lua
\ No newline at end of file
diff --git a/test/replication/misc.result b/test/replication/misc.result
index ff0dbf549..75316933f 100644
--- a/test/replication/misc.result
+++ b/test/replication/misc.result
@@ -208,3 +208,35 @@ test_run:drop_cluster(SERVERS)
 box.schema.user.revoke('guest', 'replication')
 ---
 ...
+-- gh-3510 assertion failure in replica_on_applier_disconnect()
+test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
+---
+- true
+...
+test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
+---
+- true
+...
+test_run:cmd('start server er_load1 with wait=False, wait_load=False')
+---
+- true
+...
+-- instance er_load2 will fail with error ER_READONLY. this is ok.
+-- We only test here that er_load1 doesn't assert.
+test_run:cmd('start server er_load2 with wait=True, wait_load=True, crash_expected = True')
+---
+- false
+...
+test_run:cmd('stop server er_load1')
+---
+- true
+...
+-- er_load2 exits automatically.
+test_run:cmd('cleanup server er_load1')
+---
+- true
+...
+test_run:cmd('cleanup server er_load2')
+---
+- true
+...
diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
index c05e52165..acdfa737e 100644
--- a/test/replication/misc.test.lua
+++ b/test/replication/misc.test.lua
@@ -81,3 +81,15 @@ test_run:cmd("switch default")
 test_run:drop_cluster(SERVERS)
 
 box.schema.user.revoke('guest', 'replication')
+
+-- gh-3510 assertion failure in replica_on_applier_disconnect()
+test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
+test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
+test_run:cmd('start server er_load1 with wait=False, wait_load=False')
+-- instance er_load2 will fail with error ER_READONLY. this is ok.
+-- We only test here that er_load1 doesn't assert.
+test_run:cmd('start server er_load2 with wait=True, wait_load=True, crash_expected = True')
+test_run:cmd('stop server er_load1')
+-- er_load2 exits automatically.
+test_run:cmd('cleanup server er_load1')
+test_run:cmd('cleanup server er_load2')
-- 
2.15.2 (Apple Git-101.1)

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Vladimir Davydov]

On Wed, Aug 08, 2018 at 01:10:30PM +0300, Sergey Petrenko wrote:
> 
> 
> > 7 авг. 2018 г., в 19:50, Vladimir Davydov <vdavydov.dev@gmail.com> написал(а):
> > 
> > On Mon, Aug 06, 2018 at 05:14:05PM +0300, Sergey Petrenko wrote:
> >> diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
> >> index c05e52165..27c1a4821 100644
> >> --- a/test/replication/misc.test.lua
> >> +++ b/test/replication/misc.test.lua
> >> @@ -81,3 +81,15 @@ test_run:cmd("switch default")
> >> test_run:drop_cluster(SERVERS)
> >> 
> >> box.schema.user.revoke('guest', 'replication')
> >> +
> >> +-- gh-3510 assertion failure in replica_on_applier_disconnect()
> >> +test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
> >> +test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
> >> +test_run:cmd('start server er_load1 with wait=False, wait_load=False')
> >> +test_run:cmd('start server er_load2 with wait=False, wait_load=False')
> >> +require('fiber').sleep(0.5)
> >> +test_run:cmd('stop server er_load1')
> >> +require('fiber').sleep(1)
> > 
> > <kostja-mode>
> >  These sleep()'s are way too long. They will make our tests run forever.
> >  Please change to 0.00001 or smaller or rewrite the test so that you do
> >  not need them.
> > </kostja-mode>
> 
> Hi! Thank you for review.
> Fixed. Here’s the new diff:
> 
>  src/box/replication.cc         |  4 ++++
>  test/replication/er_load.lua   | 25 +++++++++++++++++++++++++
>  test/replication/er_load1.lua  |  1 +
>  test/replication/er_load2.lua  |  1 +
>  test/replication/misc.result   | 32 ++++++++++++++++++++++++++++++++
>  test/replication/misc.test.lua | 12 ++++++++++++
>  6 files changed, 75 insertions(+)
>  create mode 100644 test/replication/er_load.lua
>  create mode 120000 test/replication/er_load1.lua
>  create mode 120000 test/replication/er_load2.lua
> 

With your patch replication/on_replace test hangs on Travis CI for
test/coverage targets (I tried to restart, but it doesn't help).
Actually, it hangs on my local machine too (debug build). Please fix.

Re: [tarantool-patches] [PATCH] replication: fix a failing assert in replica_on_applier_disconnect()

[Sergey Petrenko]

[-- Attachment #1: Type: text/plain, Size: 2509 bytes --]



> 8 авг. 2018 г., в 13:58, Vladimir Davydov <vdavydov.dev@gmail.com> написал(а):
> 
> On Wed, Aug 08, 2018 at 01:10:30PM +0300, Sergey Petrenko wrote:
>> 
>> 
>>> 7 авг. 2018 г., в 19:50, Vladimir Davydov <vdavydov.dev@gmail.com> написал(а):
>>> 
>>> On Mon, Aug 06, 2018 at 05:14:05PM +0300, Sergey Petrenko wrote:
>>>> diff --git a/test/replication/misc.test.lua b/test/replication/misc.test.lua
>>>> index c05e52165..27c1a4821 100644
>>>> --- a/test/replication/misc.test.lua
>>>> +++ b/test/replication/misc.test.lua
>>>> @@ -81,3 +81,15 @@ test_run:cmd("switch default")
>>>> test_run:drop_cluster(SERVERS)
>>>> 
>>>> box.schema.user.revoke('guest', 'replication')
>>>> +
>>>> +-- gh-3510 assertion failure in replica_on_applier_disconnect()
>>>> +test_run:cmd('create server er_load1 with script="replication/er_load1.lua"')
>>>> +test_run:cmd('create server er_load2 with script="replication/er_load2.lua"')
>>>> +test_run:cmd('start server er_load1 with wait=False, wait_load=False')
>>>> +test_run:cmd('start server er_load2 with wait=False, wait_load=False')
>>>> +require('fiber').sleep(0.5)
>>>> +test_run:cmd('stop server er_load1')
>>>> +require('fiber').sleep(1)
>>> 
>>> <kostja-mode>
>>> These sleep()'s are way too long. They will make our tests run forever.
>>> Please change to 0.00001 or smaller or rewrite the test so that you do
>>> not need them.
>>> </kostja-mode>
>> 
>> Hi! Thank you for review.
>> Fixed. Here’s the new diff:
>> 
>> src/box/replication.cc         |  4 ++++
>> test/replication/er_load.lua   | 25 +++++++++++++++++++++++++
>> test/replication/er_load1.lua  |  1 +
>> test/replication/er_load2.lua  |  1 +
>> test/replication/misc.result   | 32 ++++++++++++++++++++++++++++++++
>> test/replication/misc.test.lua | 12 ++++++++++++
>> 6 files changed, 75 insertions(+)
>> create mode 100644 test/replication/er_load.lua
>> create mode 120000 test/replication/er_load1.lua
>> create mode 120000 test/replication/er_load2.lua
>> 
> 
> With your patch replication/on_replace test hangs on Travis CI for
> test/coverage targets (I tried to restart, but it doesn't help).
> Actually, it hangs on my local machine too (debug build). Please fix.
Hi! This happens due to a bug in test-run.
I filed a ticket (https://github.com/tarantool/test-run/issues/109 <https://github.com/tarantool/test-run/issues/109>).
When it’s fixed, my test should pass no problem. So we have to wait for now.


[-- Attachment #2: Type: text/html, Size: 6824 bytes --]