Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
To: tarantool-patches@dev.tarantool.org, sergepetrenko@tarantool.org
Subject: [Tarantool-patches] [PATCH 1/1] raft: fix crash in worker fiber
Date: Sat,  7 Nov 2020 00:43:53 +0100	[thread overview]
Message-ID: <796ca0ba5fe99ec3c0b8370f1d955dc3c8f23bd2.1604706067.git.v.shpilevoy@tarantool.org> (raw)

Raft worker fiber does all the heavy and yielding jobs. These are
2 - disk write, and network broadcast. Disk write yields. Network
broadcast is slow, so it happens at most once per event loop
iteration.

The worker on each iteration check if any of these 2 jobs is
active, and if not, it goes to sleep until an explicit wakeup.

But there was a bug. Before going to sleep it did a yield + a
check that there is nothing to do. However during the yield new
tasks could appear, and the check failed, leading to a crash.

The patch reorganizes this part of the code so now the worker does
not yield between checking new tasks and going to sleep.

No test, because extremely hard to reproduce, and don't want to
clog this part of the code with error injections.
---
Branch: http://github.com/tarantool/tarantool/tree/gerold103/raft-crash

 src/box/raft.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/box/raft.c b/src/box/raft.c
index e1e60ce94..914b0d68f 100644
--- a/src/box/raft.c
+++ b/src/box/raft.c
@@ -682,11 +682,11 @@ raft_worker_f(va_list args)
 			raft_worker_handle_broadcast();
 			is_idle = false;
 		}
+		if (is_idle) {
+			assert(raft_is_fully_on_disk());
+			fiber_yield();
+		}
 		fiber_sleep(0);
-		if (!is_idle)
-			continue;
-		assert(raft_is_fully_on_disk());
-		fiber_yield();
 	}
 	return 0;
 }
-- 
2.21.1 (Apple Git-122.3)

             reply	other threads:[~2020-11-06 23:43 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-06 23:43 Vladislav Shpilevoy [this message]
2020-11-08 18:04 ` Vladislav Shpilevoy
2020-11-09 11:10   ` Serge Petrenko
2020-11-10 20:59   ` Alexander V. Tikhonov
2020-11-10 22:05 ` Vladislav Shpilevoy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=796ca0ba5fe99ec3c0b8370f1d955dc3c8f23bd2.1604706067.git.v.shpilevoy@tarantool.org \
    --to=v.shpilevoy@tarantool.org \
    --cc=sergepetrenko@tarantool.org \
    --cc=tarantool-patches@dev.tarantool.org \
    --subject='Re: [Tarantool-patches] [PATCH 1/1] raft: fix crash in worker fiber' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox