From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> To: tarantool-patches@freelists.org Cc: kostja@tarantool.org Subject: [tarantool-patches] [PATCH v2 2/2] swim: be suspicious when add new member Date: Sun, 30 Jun 2019 22:01:56 +0200 [thread overview] Message-ID: <be85f8851eab187e7c08c038e27ff0edf0f09576.1561924738.git.v.shpilevoy@tarantool.org> (raw) In-Reply-To: <cover.1561924738.git.v.shpilevoy@tarantool.org> The previous commit solves one important problem with too long event dissemination. Events could for too long time occupy the whole UDP packet. Now they live log() time, but 'dead' and 'left' members were bound to TTD. Such members were deleted after TTD is 0. Now they are deleted to early. Cluster nodes too early forget about dead ones, and nodes not aware of death of the latters, can accidentally resurrect them via anti-entropy. Cluster nodes need to be suspicious when someone tells them to add a new not dead member. This patch makes SWIM add a new member in two cases only: manually and if an ACK was received from it. A new member can't be added indirectly via events and anti-entropy anymore. Instead, a ping is sent to the members who are said to be new and alive. If ACK is received directly from them, then they are added. The patch does not affect updates. They are still indirect, because if something has updated in an existing member, then it is definitely alive. Part of #4253 --- src/lib/swim/swim.c | 29 ++++++++++++++++++++++++++--- test/unit/swim.c | 11 ++++------- test/unit/swim.result | 17 +++++++++++------ 3 files changed, 41 insertions(+), 16 deletions(-) diff --git a/src/lib/swim/swim.c b/src/lib/swim/swim.c index 9647094f5..6fbac00ec 100644 --- a/src/lib/swim/swim.c +++ b/src/lib/swim/swim.c @@ -1510,6 +1510,10 @@ swim_update_member(struct swim *swim, const struct swim_member_def *def, * @param swim SWIM instance to upsert into. * @param def Member definition to build a new member or update an * existing one. + * @param is_direct True, if the member definition was received + * directly from that member. Via ping or ack from him. + * Otherwise the definition can't be trusted, if it states, + * that a new member should be added. * @param[out] result A result member: a new, or an updated, or * NULL in case of nothing has changed. For example, @a def * was too old. @@ -1520,7 +1524,7 @@ swim_update_member(struct swim *swim, const struct swim_member_def *def, */ static int swim_upsert_member(struct swim *swim, const struct swim_member_def *def, - struct swim_member **result) + bool is_direct, struct swim_member **result) { struct swim_member *member = swim_find_member(swim, &def->uuid); if (member == NULL) { @@ -1538,6 +1542,25 @@ swim_upsert_member(struct swim *swim, const struct swim_member_def *def, * forever. */ goto skip; + } else if (def->status < MEMBER_DEAD && ! is_direct) { + /* + * If a member is not dead, then it can't + * be added as simple as that. In fact it + * can be already dead, but sender of that + * definition didn't know that. The only + * way to be sure - send him a ping. And + * one exception - if the sender of the + * definition is exactly that member, then + * it can be trusted of course. + */ + struct swim_task *t = swim_task_new(swim_task_delete_cb, + swim_task_delete_cb, + "probe ping"); + if (t == NULL) + return -1; + swim_send_ping(swim, t, &def->addr); + *result = NULL; + return 0; } *result = swim_new_member(swim, &def->addr, &def->uuid, def->status, &def->incarnation, @@ -1597,7 +1620,7 @@ swim_process_members(struct swim *swim, const char *prefix, struct swim_member *member; if (swim_member_def_decode(&def, pos, end, prefix) != 0) return -1; - if (swim_upsert_member(swim, &def, &member) != 0) { + if (swim_upsert_member(swim, &def, false, &member) != 0) { /* * Not a critical error. Other members * still can be updated. @@ -1639,7 +1662,7 @@ swim_process_failure_detection(struct swim *swim, const char **pos, mdef.incarnation = def.incarnation; mdef.uuid = *uuid; struct swim_member *member; - if (swim_upsert_member(swim, &mdef, &member) != 0) + if (swim_upsert_member(swim, &mdef, true, &member) != 0) return -1; /* * It can be NULL, for example, in case of too old diff --git a/test/unit/swim.c b/test/unit/swim.c index d776a8231..db9441157 100644 --- a/test/unit/swim.c +++ b/test/unit/swim.c @@ -254,7 +254,7 @@ swim_test_add_remove(void) static void swim_test_basic_failure_detection(void) { - swim_start_test(9); + swim_start_test(8); struct swim_cluster *cluster = swim_cluster_new(2); swim_cluster_set_ack_timeout(cluster, 0.5); @@ -275,11 +275,8 @@ swim_test_basic_failure_detection(void) "but it is dead after one more"); swim_run_for(1); - is(swim_cluster_member_status(cluster, 0, 1), MEMBER_DEAD, "after 2 "\ - "more unacks the member still is not deleted - dissemination TTD "\ - "keeps it"); is(swim_cluster_wait_status(cluster, 0, 1, swim_member_status_MAX, 2), - 0, "but it is dropped after 2 rounds when TTD gets 0"); + 0, "it is dropped after one more round when TTD gets 0"); /* * After IO unblock pending messages will be processed all @@ -584,12 +581,12 @@ swim_test_quit(void) swim_cluster_interconnect(cluster, 1, 2); swim_cluster_quit_node(cluster, 0); - swim_run_for(2); + swim_run_for(1); is(swim_cluster_member_status(cluster, 2, 0), MEMBER_LEFT, "S3 sees S1 as left"); is(swim_cluster_member_status(cluster, 1, 0), swim_member_status_MAX, "S2 does not see S1 at all"); - swim_run_for(2); + swim_run_for(1); is(swim_cluster_member_status(cluster, 2, 0), swim_member_status_MAX, "after more time S1 is dropped from S3"); is(swim_cluster_member_status(cluster, 1, 0), swim_member_status_MAX, diff --git a/test/unit/swim.result b/test/unit/swim.result index 04a2778e6..1fde80132 100644 --- a/test/unit/swim.result +++ b/test/unit/swim.result @@ -1,5 +1,5 @@ *** main_f *** -1..22 +1..23 *** swim_test_one_link *** 1..6 ok 1 - no rounds - no fullmesh @@ -65,16 +65,15 @@ ok 4 - subtests ok 5 - subtests *** swim_test_add_remove: done *** *** swim_test_basic_failure_detection *** - 1..9 + 1..8 ok 1 - node is added as alive ok 2 - member still is not suspected after 1 noack ok 3 - but it is suspected after one more ok 4 - it is not dead after 2 more noacks ok 5 - but it is dead after one more - ok 6 - after 2 more unacks the member still is not deleted - dissemination TTD keeps it - ok 7 - but it is dropped after 2 rounds when TTD gets 0 - ok 8 - fullmesh is restored - ok 9 - a member is added back on an ACK + ok 6 - it is dropped after one more round when TTD gets 0 + ok 7 - fullmesh is restored + ok 8 - a member is added back on an ACK ok 6 - subtests *** swim_test_basic_failure_detection: done *** *** swim_test_probe *** @@ -237,4 +236,10 @@ ok 21 - subtests ok 3 - S2 sees new generation of S1 ok 22 - subtests *** swim_test_generation: done *** + *** swim_test_dissemination_speed *** + 1..2 + ok 1 - dissemination work in log time even at the very start of a cluster + ok 2 - dissemination can withstand an event storm +ok 23 - subtests + *** swim_test_dissemination_speed: done *** *** main_f: done *** -- 2.20.1 (Apple Git-117)
next prev parent reply other threads:[~2019-06-30 20:01 UTC|newest] Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-06-30 20:01 [tarantool-patches] [PATCH v2 0/2] SWIM log TTD, and suspicious anti-entropy Vladislav Shpilevoy 2019-06-30 20:01 ` [tarantool-patches] [PATCH v2 1/2] swim: disseminate event for log(cluster_size) steps Vladislav Shpilevoy 2019-07-03 19:06 ` [tarantool-patches] " Konstantin Osipov 2019-07-03 23:30 ` Vladislav Shpilevoy 2019-07-04 8:10 ` Konstantin Osipov 2019-06-30 20:01 ` Vladislav Shpilevoy [this message] 2019-07-03 19:07 ` [tarantool-patches] Re: [PATCH v2 2/2] swim: be suspicious when add new member Konstantin Osipov 2019-07-03 23:32 ` Vladislav Shpilevoy 2019-07-04 8:10 ` Konstantin Osipov 2019-07-05 22:45 ` [tarantool-patches] Re: [PATCH v2 0/2] SWIM log TTD, and suspicious anti-entropy Vladislav Shpilevoy
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=be85f8851eab187e7c08c038e27ff0edf0f09576.1561924738.git.v.shpilevoy@tarantool.org \ --to=v.shpilevoy@tarantool.org \ --cc=kostja@tarantool.org \ --cc=tarantool-patches@freelists.org \ --subject='Re: [tarantool-patches] [PATCH v2 2/2] swim: be suspicious when add new member' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox