[tarantool-patches] [PATCH 00/10] swim Lua API

Tarantool development patches archive
 help / color / mirror / Atom feed

* [tarantool-patches] [PATCH 00/10] swim Lua API
@ 2019-05-15 19:36 Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts Vladislav Shpilevoy
                   ` (10 more replies)
  0 siblings, 11 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

This huge patchset introduces SWIM Lua module - wrapper around C SWIM. It
duplicates C API one in one, with additional internal checks related to Lua
dynamic typing.

First 3 patches are preparatory, they fix some bugs in the C module appeared
during Lua API development.

Next 4 patches are the core - 4 independent parts of Lua API. They are simple
and somewhere not efficient.

Next 3 patches are optimizations, then eliminate some performance problems of
the core patches.

Branch: http://github.com/tarantool/tarantool/tree/gerold103/swim-lua
Issue: https://github.com/tarantool/tarantool/issues/3234

Vladislav Shpilevoy (10):
  swim: fix an assertion on attempt to chage timeouts
  swim: make swim_new_round() void
  swim: validate URI in swim_probe_member()
  swim: introduce Lua interface
  swim: Lua bindings to manipulate member table
  swim: Lua bindings to access individual members
  swim: pairs() function to iterate over member table
  swim: allow to use cdata struct tt_uuid in Lua API
  swim: cache decoded payload in the Lua module
  swim: cache members in Lua member table

 extra/exports           |   26 +
 src/CMakeLists.txt      |    1 +
 src/lib/swim/swim.c     |   46 +-
 src/lib/swim/swim.h     |   12 +
 src/lib/swim/swim_ev.h  |    2 +
 src/lua/init.c          |    2 +
 src/lua/swim.lua        |  777 ++++++++++++++++++++++++++++++
 test/swim/box.lua       |   24 +
 test/swim/suite.ini     |    5 +
 test/swim/swim.result   | 1011 +++++++++++++++++++++++++++++++++++++++
 test/swim/swim.test.lua |  340 +++++++++++++
 test/unit/swim.c        |    3 +-
 test/unit/swim.result   |    7 +-
 13 files changed, 2234 insertions(+), 22 deletions(-)
 create mode 100644 src/lua/swim.lua
 create mode 100644 test/swim/box.lua
 create mode 100644 test/swim/suite.ini
 create mode 100644 test/swim/swim.result
 create mode 100644 test/swim/swim.test.lua

-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:28   ` [tarantool-patches] " Konstantin Osipov
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 10/10] swim: cache members in Lua member table Vladislav Shpilevoy
                   ` (9 subsequent siblings)
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Appeared, that libev does not allow to change ev_timer values in
flight. A timer, reset via ev_timer_set(), should be restarted,
because the function changes 'ev_timer.at', which in turn is used
internally by timer routines.

Part of #3234
---
 src/lib/swim/swim.c    | 17 ++++++++++++-----
 src/lib/swim/swim_ev.h |  2 ++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/src/lib/swim/swim.c b/src/lib/swim/swim.c
index 54c5b3250..725ebf222 100644
--- a/src/lib/swim/swim.c
+++ b/src/lib/swim/swim.c
@@ -1797,11 +1797,18 @@ swim_cfg(struct swim *swim, const char *uri, double heartbeat_rate,
 	} else {
 		addr = swim->self->addr;
 	}
-	if (swim->round_tick.repeat != heartbeat_rate && heartbeat_rate > 0)
-		swim_ev_timer_set(&swim->round_tick, 0, heartbeat_rate);
-
-	if (swim->wait_ack_tick.repeat != ack_timeout && ack_timeout > 0)
-		swim_ev_timer_set(&swim->wait_ack_tick, 0, ack_timeout);
+	struct ev_timer *t = &swim->round_tick;
+	if (t->repeat != heartbeat_rate && heartbeat_rate > 0) {
+		swim_ev_timer_set(t, 0, heartbeat_rate);
+		if (swim_ev_is_active(t))
+			swim_ev_timer_again(loop(), t);
+	}
+	t = &swim->wait_ack_tick;
+	if (t->repeat != ack_timeout && ack_timeout > 0) {
+		swim_ev_timer_set(t, 0, ack_timeout);
+		if (swim_ev_is_active(t))
+			swim_ev_timer_again(loop(), t);
+	}
 
 	if (new_self != NULL) {
 		swim->self->status = MEMBER_LEFT;
diff --git a/src/lib/swim/swim_ev.h b/src/lib/swim/swim_ev.h
index b68ed9e19..fe261ff38 100644
--- a/src/lib/swim/swim_ev.h
+++ b/src/lib/swim/swim_ev.h
@@ -52,6 +52,8 @@ swim_ev_timer_again(struct ev_loop *loop, struct ev_timer *watcher);
 void
 swim_ev_timer_stop(struct ev_loop *loop, struct ev_timer *watcher);
 
+#define swim_ev_is_active ev_is_active
+
 #define swim_ev_init ev_init
 
 #define swim_ev_timer_init ev_timer_init
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts Vladislav Shpilevoy
@ 2019-05-16  7:28   ` Konstantin Osipov
  0 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:28 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Appeared, that libev does not allow to change ev_timer values in
> flight. A timer, reset via ev_timer_set(), should be restarted,
> because the function changes 'ev_timer.at', which in turn is used
> internally by timer routines.
> 

OK to push.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 10/10] swim: cache members in Lua member table
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 02/10] swim: make swim_new_round() void Vladislav Shpilevoy
                   ` (8 subsequent siblings)
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Each time a member was returned from a SWIM instance object, it
was wrapped by a table with a special metatable, cached payload.

But next the same lookup returned a new table. It

  - created garbage as a new member wrapper;
  - lost cached decoded payload.

This commit caches in a private table all wrapped members and
returns an existing wrapper on a next lookup. A microbenchmark
showed, that cached result retrieval is 10 times faster, than
each time create a new table.

Cache table keeps week references - it means, that when a member
object looses all its references in a user's application, it is
automatically dropped from the table.

Part of #3234
---
 src/lua/swim.lua        | 34 ++++++++++++++-----
 test/swim/swim.result   | 75 +++++++++++++++++++++++++++++++++++++++++
 test/swim/swim.test.lua | 30 +++++++++++++++++
 3 files changed, 130 insertions(+), 9 deletions(-)

diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index 852192479..35ceea669 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -378,10 +378,19 @@ local swim_member_mt = {
 -- table-wrapper stores not only a pointer, but also cached
 -- decoded payload.
 --
-local function swim_member_wrap(ptr)
-    capi.swim_member_ref(ptr)
-    ffi.gc(ptr, capi.swim_member_unref)
-    return setmetatable({ptr = ptr}, swim_member_mt)
+local function swim_wrap_member(s, ptr)
+    -- Lua tables can't normally work with cdata keys. Even when
+    -- cdata is a simple number, table can't do search by it.
+    local key = tonumber(ffi.cast('unsigned long', ptr))
+    local cache = s.cache_table
+    local wrapped = cache[key]
+    if wrapped == nil then
+        capi.swim_member_ref(ptr)
+        ffi.gc(ptr, capi.swim_member_unref)
+        wrapped = setmetatable({ptr = ptr}, swim_member_mt)
+        cache[key] = wrapped
+    end
+    return wrapped
 end
 
 --
@@ -516,7 +525,8 @@ end
 -- into the member table.
 --
 local function swim_self(s)
-    return swim_member_wrap(capi.swim_self(swim_check_instance(s, 'swim:self')))
+    local ptr = swim_check_instance(s, 'swim:self')
+    return swim_wrap_member(s, capi.swim_self(ptr))
 end
 
 --
@@ -530,7 +540,7 @@ local function swim_member_by_uuid(s, uuid)
     if m == nil then
         return nil
     end
-    return swim_member_wrap(m)
+    return swim_wrap_member(s, m)
 end
 
 --
@@ -590,13 +600,14 @@ end
 -- member object as a value.
 --
 local function swim_pairs_next(ctx)
-    if ctx.swim.ptr == nil then
+    local s = ctx.swim
+    if s.ptr == nil then
         return swim_error_deleted()
     end
     local iterator = ctx.iterator
     local m = capi.swim_iterator_next(iterator)
     if m ~= nil then
-        m = swim_member_wrap(m)
+        m = swim_wrap_member(s, m)
         return m:uuid(), m
     end
     capi.swim_iterator_close(ffi.gc(iterator, nil))
@@ -732,6 +743,10 @@ local swim_not_configured_mt = {
 local swim_cfg_not_configured_mt = table.deepcopy(swim_cfg_mt)
 swim_cfg_not_configured_mt.__call = swim_cfg_first_call
 
+-- Member cache stores week references so as to do not care about
+-- removed members erasure - GC drops them automatically.
+local cache_table_mt = { __mode = 'v' }
+
 --
 -- Create a new SWIM instance, and configure if @a cfg is
 -- provided.
@@ -744,7 +759,8 @@ local function swim_new(cfg)
     ffi.gc(ptr, capi.swim_delete)
     local s = setmetatable({
         ptr = ptr,
-        cfg = setmetatable({index = {}}, swim_cfg_not_configured_mt)
+        cfg = setmetatable({index = {}}, swim_cfg_not_configured_mt),
+        cache_table = setmetatable({}, cache_table_mt)
     }, swim_not_configured_mt)
     if cfg then
         local ok, err = s:cfg(cfg)
diff --git a/test/swim/swim.result b/test/swim/swim.result
index ef7203a37..531903e7f 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -930,6 +930,81 @@ s1:delete()
 s2:delete()
 ---
 ...
+--
+-- Member table cache in Lua.
+--
+s = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+self = s:self()
+---
+...
+s:self() == self
+---
+- true
+...
+s:add_member({uuid = uuid(2), uri = 1})
+---
+- true
+...
+s2 = s:member_by_uuid(uuid(2))
+---
+...
+s2
+---
+- uri: 127.0.0.1:<port>
+  status: alive
+  incarnation: 0
+  uuid: 00000000-0000-1000-8000-000000000002
+  payload_size: 0
+...
+-- Next lookups return the same member table.
+s2 == s:member_by_uuid(uuid(2))
+---
+- true
+...
+s2_old_uri = s2:uri()
+---
+...
+-- Check, that it is impossible to take removed member from the
+-- cached table.
+s:remove_member(uuid(2))
+---
+- true
+...
+s:member_by_uuid(uuid(2))
+---
+- null
+...
+-- GC automatically removes members from the member table.
+self = nil
+---
+...
+s2 = nil
+---
+...
+collectgarbage('collect')
+---
+- 0
+...
+s.cache_table
+---
+- []
+...
+s:add_member({uuid = uuid(2), uri = 2})
+---
+- true
+...
+s2 = s:member_by_uuid(uuid(2))
+---
+...
+s2:uri() ~= s2_old_uri
+---
+- true
+...
+s:delete()
+---
+...
 test_run:cmd("clear filter")
 ---
 - true
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index c3387cf0a..3b0807f1e 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -307,4 +307,34 @@ s1_view:incarnation()
 s1:delete()
 s2:delete()
 
+--
+-- Member table cache in Lua.
+--
+s = swim.new({uuid = uuid(1), uri = uri()})
+self = s:self()
+s:self() == self
+
+s:add_member({uuid = uuid(2), uri = 1})
+s2 = s:member_by_uuid(uuid(2))
+s2
+-- Next lookups return the same member table.
+s2 == s:member_by_uuid(uuid(2))
+s2_old_uri = s2:uri()
+
+-- Check, that it is impossible to take removed member from the
+-- cached table.
+s:remove_member(uuid(2))
+s:member_by_uuid(uuid(2))
+
+-- GC automatically removes members from the member table.
+self = nil
+s2 = nil
+collectgarbage('collect')
+s.cache_table
+s:add_member({uuid = uuid(2), uri = 2})
+s2 = s:member_by_uuid(uuid(2))
+s2:uri() ~= s2_old_uri
+
+s:delete()
+
 test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 10/10] swim: cache members in Lua member table
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 10/10] swim: cache members in Lua member table Vladislav Shpilevoy
@ 2019-05-16  7:31   ` Konstantin Osipov
  0 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:31 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Each time a member was returned from a SWIM instance object, it
> was wrapped by a table with a special metatable, cached payload.

OK to push.

-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 02/10] swim: make swim_new_round() void
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 10/10] swim: cache members in Lua member table Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 03/10] swim: validate URI in swim_probe_member() Vladislav Shpilevoy
                   ` (7 subsequent siblings)
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Firstly, I thought that there is an error - swim_begin_step()
does not reschedules round timer, when new_round() fails. But
then new_round() appeared never failing. This commit makes it
void to eliminate confusion.

Probably it is a legacy since the shuffled members array was
allocated and freed in new_round().

Part of #3234
---
 src/lib/swim/swim.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

diff --git a/src/lib/swim/swim.c b/src/lib/swim/swim.c
index 725ebf222..a9a0a39b3 100644
--- a/src/lib/swim/swim.c
+++ b/src/lib/swim/swim.c
@@ -844,7 +844,7 @@ swim_shuffle_members(struct swim *swim)
  * Shuffle members, build randomly ordered queue of addressees. In
  * other words, do all round preparation work.
  */
-static int
+static void
 swim_new_round(struct swim *swim)
 {
 	int size = mh_size(swim->members);
@@ -852,7 +852,7 @@ swim_new_round(struct swim *swim)
 		assert(swim->self != NULL);
 		say_verbose("SWIM %d: skip a round - no members",
 			    swim_fd(swim));
-		return 0;
+		return;
 	}
 	/* -1 for self. */
 	say_verbose("SWIM %d: start a new round with %d members", swim_fd(swim),
@@ -866,7 +866,6 @@ swim_new_round(struct swim *swim)
 					in_round_queue);
 		}
 	}
-	return 0;
 }
 
 /**
@@ -1078,12 +1077,10 @@ swim_begin_step(struct ev_loop *loop, struct ev_timer *t, int events)
 	(void) events;
 	(void) loop;
 	struct swim *swim = (struct swim *) t->data;
-	if (! rlist_empty(&swim->round_queue)) {
+	if (! rlist_empty(&swim->round_queue))
 		say_verbose("SWIM %d: continue the round", swim_fd(swim));
-	} else if (swim_new_round(swim) != 0) {
-		diag_log();
-		return;
-	}
+	else
+		swim_new_round(swim);
 	/*
 	 * Possibly empty, if no members but self are specified.
 	 */
@@ -1994,11 +1991,7 @@ swim_quit(struct swim *swim)
 	swim_ev_timer_stop(loop(), &swim->wait_ack_tick);
 	swim_scheduler_stop_input(&swim->scheduler);
 	/* Start the last round - quiting. */
-	if (swim_new_round(swim) != 0) {
-		diag_log();
-		swim_delete(swim);
-		return;
-	}
+	swim_new_round(swim);
 	struct swim_task *task = &swim->round_step_task;
 	swim_task_destroy(task);
 	swim_task_create(task, swim_quit_step_complete, swim_task_delete_cb,
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 02/10] swim: make swim_new_round() void
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 02/10] swim: make swim_new_round() void Vladislav Shpilevoy
@ 2019-05-16  7:31   ` Konstantin Osipov
  0 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:31 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Firstly, I thought that there is an error - swim_begin_step()
> does not reschedules round timer, when new_round() fails. But
> then new_round() appeared never failing. This commit makes it
> void to eliminate confusion.
> 
> Probably it is a legacy since the shuffled members array was
> allocated and freed in new_round().

ok to push.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 03/10] swim: validate URI in swim_probe_member()
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (2 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 02/10] swim: make swim_new_round() void Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 04/10] swim: introduce Lua interface Vladislav Shpilevoy
                   ` (6 subsequent siblings)
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Similar methods validate their arguments: add_member,
remove_member. Validate here as well for consistency.

Part of #3234
---
 src/lib/swim/swim.c   | 4 ++++
 test/unit/swim.c      | 3 ++-
 test/unit/swim.result | 7 ++++---
 3 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/src/lib/swim/swim.c b/src/lib/swim/swim.c
index a9a0a39b3..fb2b1490b 100644
--- a/src/lib/swim/swim.c
+++ b/src/lib/swim/swim.c
@@ -1890,6 +1890,10 @@ int
 swim_probe_member(struct swim *swim, const char *uri)
 {
 	assert(swim_is_configured(swim));
+	if (uri == NULL) {
+		diag_set(SwimError, "swim.probe_member: URI is mandatory");
+		return -1;
+	}
 	struct sockaddr_in addr;
 	if (swim_uri_to_addr(uri, &addr, "swim.probe_member:") != 0)
 		return -1;
diff --git a/test/unit/swim.c b/test/unit/swim.c
index d9613e8e0..c6ef1eebc 100644
--- a/test/unit/swim.c
+++ b/test/unit/swim.c
@@ -359,12 +359,13 @@ swim_test_basic_gossip(void)
 static void
 swim_test_probe(void)
 {
-	swim_start_test(2);
+	swim_start_test(3);
 	struct swim_cluster *cluster = swim_cluster_new(2);
 
 	struct swim *s1 = swim_cluster_member(cluster, 0);
 	struct swim *s2 = swim_cluster_member(cluster, 1);
 	const char *s2_uri = swim_member_uri(swim_self(s2));
+	is(swim_probe_member(s1, NULL), -1, "probe validates URI");
 	is(swim_probe_member(s1, s2_uri), 0, "send probe");
 	is(swim_cluster_wait_fullmesh(cluster, 0.1), 0,
 	   "receive ACK on probe and get fullmesh")
diff --git a/test/unit/swim.result b/test/unit/swim.result
index 266d83589..587f66c7a 100644
--- a/test/unit/swim.result
+++ b/test/unit/swim.result
@@ -78,9 +78,10 @@ ok 5 - subtests
 ok 6 - subtests
 	*** swim_test_basic_failure_detection: done ***
 	*** swim_test_probe ***
-    1..2
-    ok 1 - send probe
-    ok 2 - receive ACK on probe and get fullmesh
+    1..3
+    ok 1 - probe validates URI
+    ok 2 - send probe
+    ok 3 - receive ACK on probe and get fullmesh
 ok 7 - subtests
 	*** swim_test_probe: done ***
 	*** swim_test_refute ***
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 03/10] swim: validate URI in swim_probe_member()
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 03/10] swim: validate URI in swim_probe_member() Vladislav Shpilevoy
@ 2019-05-16  7:31   ` Konstantin Osipov
  0 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:31 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Similar methods validate their arguments: add_member,
> remove_member. Validate here as well for consistency.
> 
ok to push


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 04/10] swim: introduce Lua interface
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (3 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 03/10] swim: validate URI in swim_probe_member() Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 05/10] swim: Lua bindings to manipulate member table Vladislav Shpilevoy
                   ` (5 subsequent siblings)
  10 siblings, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

SWIM as a library can be useful not only for server internals,
but for users as well. This commit exposes Lua bindings to SWIM
C API. Here only basic bindings are introduced to create, delete,
quit, check a SWIM instance. With sanity tests.

Part of #3234
---
 extra/exports           |  25 +++
 src/CMakeLists.txt      |   1 +
 src/lua/init.c          |   2 +
 src/lua/swim.lua        | 408 ++++++++++++++++++++++++++++++++++++++++
 test/swim/box.lua       |  24 +++
 test/swim/suite.ini     |   5 +
 test/swim/swim.result   | 201 ++++++++++++++++++++
 test/swim/swim.test.lua |  68 +++++++
 8 files changed, 734 insertions(+)
 create mode 100644 src/lua/swim.lua
 create mode 100644 test/swim/box.lua
 create mode 100644 test/swim/suite.ini
 create mode 100644 test/swim/swim.result
 create mode 100644 test/swim/swim.test.lua

diff --git a/extra/exports b/extra/exports
index 5375a01e4..b0e20e11f 100644
--- a/extra/exports
+++ b/extra/exports
@@ -87,6 +87,31 @@ tnt_HMAC_CTX_free
 
 lua_static_aligned_alloc
 
+swim_new
+swim_is_configured
+swim_cfg
+swim_set_payload
+swim_delete
+swim_add_member
+swim_remove_member
+swim_probe_member
+swim_broadcast
+swim_size
+swim_quit
+swim_self
+swim_member_by_uuid
+swim_member_status
+swim_iterator_open
+swim_iterator_next
+swim_iterator_close
+swim_member_uri
+swim_member_uuid
+swim_member_incarnation
+swim_member_payload
+swim_member_ref
+swim_member_unref
+swim_member_is_dropped
+
 # Module API
 
 _say
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 492b8712e..c2e0240ab 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -48,6 +48,7 @@ lua_source(lua_sources lua/table.lua)
 lua_source(lua_sources ../third_party/luafun/fun.lua)
 lua_source(lua_sources lua/httpc.lua)
 lua_source(lua_sources lua/iconv.lua)
+lua_source(lua_sources lua/swim.lua)
 # LuaJIT jit.* library
 lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/bc.lua")
 lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/bcsave.lua")
diff --git a/src/lua/init.c b/src/lua/init.c
index 303369841..5ddc5a4d8 100644
--- a/src/lua/init.c
+++ b/src/lua/init.c
@@ -115,6 +115,7 @@ extern char strict_lua[],
 	table_lua[],
 	trigger_lua[],
 	string_lua[],
+	swim_lua[],
 	p_lua[], /* LuaJIT 2.1 profiler */
 	zone_lua[] /* LuaJIT 2.1 profiler */;
 
@@ -149,6 +150,7 @@ static const char *lua_modules[] = {
 	"pwd", pwd_lua,
 	"http.client", httpc_lua,
 	"iconv", iconv_lua,
+	"swim", swim_lua,
 	/* jit.* library */
 	"jit.vmdef", vmdef_lua,
 	"jit.bc", bc_lua,
diff --git a/src/lua/swim.lua b/src/lua/swim.lua
new file mode 100644
index 000000000..bca98627c
--- /dev/null
+++ b/src/lua/swim.lua
@@ -0,0 +1,408 @@
+local ffi = require('ffi')
+local uuid = require('uuid')
+
+ffi.cdef[[
+    struct swim;
+    struct tt_uuid;
+    struct swim_iterator;
+    struct swim_member;
+
+    enum swim_gc_mode {
+        SWIM_GC_DEFAULT = -1,
+        SWIM_GC_OFF = 0,
+        SWIM_GC_ON = 1,
+    };
+
+    enum swim_member_status {
+        MEMBER_ALIVE = 0,
+        MEMBER_SUSPECTED,
+        MEMBER_DEAD,
+        MEMBER_LEFT,
+    };
+
+    struct swim *
+    swim_new(void);
+
+    bool
+    swim_is_configured(const struct swim *swim);
+
+    int
+    swim_cfg(struct swim *swim, const char *uri, double heartbeat_rate,
+             double ack_timeout, enum swim_gc_mode gc_mode,
+             const struct tt_uuid *uuid);
+
+    int
+    swim_set_payload(struct swim *swim, const char *payload, int payload_size);
+
+    void
+    swim_delete(struct swim *swim);
+
+    int
+    swim_add_member(struct swim *swim, const char *uri,
+                    const struct tt_uuid *uuid);
+
+    int
+    swim_remove_member(struct swim *swim, const struct tt_uuid *uuid);
+
+    int
+    swim_probe_member(struct swim *swim, const char *uri);
+
+    int
+    swim_broadcast(struct swim *swim, int port);
+
+    int
+    swim_size(const struct swim *swim);
+
+    void
+    swim_quit(struct swim *swim);
+
+    struct swim_member *
+    swim_self(struct swim *swim);
+
+    struct swim_member *
+    swim_member_by_uuid(struct swim *swim, const struct tt_uuid *uuid);
+
+    enum swim_member_status
+    swim_member_status(const struct swim_member *member);
+
+    struct swim_iterator *
+    swim_iterator_open(struct swim *swim);
+
+    struct swim_member *
+    swim_iterator_next(struct swim_iterator *iterator);
+
+    void
+    swim_iterator_close(struct swim_iterator *iterator);
+
+    const char *
+    swim_member_uri(const struct swim_member *member);
+
+    const struct tt_uuid *
+    swim_member_uuid(const struct swim_member *member);
+
+    uint64_t
+    swim_member_incarnation(const struct swim_member *member);
+
+    const char *
+    swim_member_payload(const struct swim_member *member, int *size);
+
+    void
+    swim_member_ref(struct swim_member *member);
+
+    void
+    swim_member_unref(struct swim_member *member);
+
+    bool
+    swim_member_is_dropped(const struct swim_member *member);
+]]
+
+-- Shortcut to avoid unnecessary lookups in 'ffi' table.
+local capi = ffi.C
+
+local swim_t = ffi.typeof('struct swim *')
+
+--
+-- Check if @a value is something that can be passed as a
+-- URI parameter. Note, it does not validate URI, because it is
+-- done in C module. Here only dynamic typing errors are checked.
+-- Throws on invalid type.
+--
+-- @param value Value to check and probably convert.
+-- @param func_name Caller function name to include into an error
+--        message.
+-- @return String that can be passed as a URI parameter.
+--
+local function swim_check_uri(value, func_name)
+    if value == nil then
+        return nil
+    end
+    if type(value) == 'string' then
+        return value
+    end
+    if type(value) == 'number' then
+        return tostring(value)
+    end
+    return error(func_name..': expected string URI or port number')
+end
+
+--
+-- Check if @a value is a number, that can be passed as a timeout.
+-- Throws on invalid type.
+--
+-- @param value Value to check.
+-- @param func_name Caller function name to include into an error
+--        message.
+-- @param param_name Timeout parameter name to include into an
+--        error message. Examples: 'heartbeat_rate',
+--        'ack_timeout'.
+-- @return Timeout value. Can be negative - SWIM treats negative
+--         value as an instruction to keep an old timeout.
+--
+local function swim_check_timeout(value, func_name, param_name)
+    if value == nil then
+        return -1
+    end
+    if type(value) ~= 'number' then
+        return error(func_name..': expected number '..param_name)
+    end
+    return value
+end
+
+--
+-- Check if @a value is a valid garbage collection mode.
+-- Throws on invalid type and unknown mode.
+--
+-- @param value Value to check and convert.
+-- @param func_name Caller function name to include into an error
+--        message.
+-- @return GC mode cdata enum value.
+--
+local function swim_check_gc_mode(value, func_name)
+    if value == nil then
+        return capi.SWIM_GC_DEFAULT
+    end
+    if value == 'on' then
+        return capi.SWIM_GC_ON
+    elseif value == 'off' then
+        return capi.SWIM_GC_OFF
+    else
+        return error(func_name..': unknown gc_mode')
+    end
+end
+
+--
+-- Check if @a value is a valid UUID. Throws on invalid type and
+-- UUID.
+--
+-- @param value Value to check.
+-- @param func_name Caller function name to include into an error
+--        message.
+-- @return Struct UUID cdata.
+--
+local function swim_check_uuid(value, func_name)
+    if value == nil then
+        return nil
+    end
+    if type(value) ~= 'string' then
+        return error(func_name..': expected string UUID')
+    end
+    value = uuid.fromstr(value)
+    if not value then
+        return error(func_name..': invalid UUID')
+    end
+    return value
+end
+
+--
+-- Check if @a s is a SWIM instance. It should be a table with
+-- cdata struct swim in 'ptr' attribute. Throws on invalid type.
+--
+-- @param value Value to check.
+-- @param func_name Caller function name to include into an error
+--        message.
+-- @return Pointer to struct swim.
+--
+local function swim_check_instance(s, func_name)
+    if type(s) == 'table' then
+        local ptr = s.ptr
+        if ffi.istype(swim_t, ptr) then
+            return ptr
+        end
+    end
+    return error(func_name..': first argument is not a SWIM instance')
+end
+
+--
+-- When a SWIM instance is deleted or has quited, it can't be used
+-- anymore. This function replaces all methods of a deleted
+-- instance to throw an error on a usage attempt.
+--
+local function swim_error_deleted()
+    return error('the swim instance is deleted')
+end
+-- This is done without 'if' in the original methods, but rather
+-- via metatable replacement after deletion has happened.
+local swim_mt_deleted = {
+    __index = swim_error_deleted
+}
+
+--
+-- Delete a SWIM instance immediately, do not notify cluster
+-- members about that.
+--
+local function swim_delete(s)
+    local ptr = swim_check_instance(s, 'swim:delete')
+    capi.swim_delete(ffi.gc(ptr, nil))
+    s.ptr = nil
+    setmetatable(s, swim_mt_deleted)
+end
+
+--
+-- Quit from a cluster gracefully, notify other members. The SWIM
+-- instance is considered deleted immediately after this function
+-- returned, and can't be used anymore.
+--
+local function swim_quit(s)
+    local ptr = swim_check_instance(s, 'swim:quit')
+    capi.swim_quit(ffi.gc(ptr, nil))
+    s.ptr = nil
+    setmetatable(s, swim_mt_deleted)
+end
+
+--
+-- Size of the local member table.
+--
+local function swim_size(s)
+    return capi.swim_size(swim_check_instance(s, 'swim:size'))
+end
+
+--
+-- Check if a SWIM instance is configured already. Not configured
+-- instance does not provide any methods.
+--
+local function swim_is_configured(s)
+    return capi.swim_is_configured(swim_check_instance(s, 'swim:is_configured'))
+end
+
+--
+-- Configuration options are printed when a SWIM instance is
+-- serialized, for example, in a console.
+--
+local function swim_serialize(s)
+    return s.cfg.index
+end
+
+--
+-- Normal metatable of a configured SWIM instance.
+--
+local swim_mt = {
+    __index = {
+        delete = swim_delete,
+        quit = swim_quit,
+        size = swim_size,
+        is_configured = swim_is_configured,
+    },
+    __serialize = swim_serialize
+}
+
+local swim_cfg_options = {
+    uri = true, heartbeat_rate = true, ack_timeout = true,
+    gc_mode = true, uuid = true
+}
+
+--
+-- SWIM 'cfg' attribute is not a trivial table nor function. It is
+-- a callable table. It allows to strongly restrict use cases of
+-- swim.cfg down to 2 applications:
+--
+-- - Cfg-table. 'swim.cfg' is a read-only table of configured
+--   parameters. A user can't write 'swim.cfg.<key> = value' - an
+--   error is thrown. But it can be indexed like
+--   'opt = swim.cfg.<key>'. Configuration is cached in Lua, so
+--   the latter example won't even call SWIM C API functions.
+--
+-- - Cfg-function. 'swim:cfg(<new config>)' reconfigures a SWIM
+--   instance and returns normal values: nil+error or true. The
+--   new configuration is cached for further indexing.
+--
+-- All the other combinations are banned. The function below
+-- implements configuration call.
+--
+local function swim_cfg_call(c, s, cfg)
+    local func_name = 'swim:cfg'
+    local ptr = swim_check_instance(s, func_name)
+    if type(cfg) ~= 'table' then
+        return error(func_name..': expected table configuration')
+    end
+    for k in pairs(cfg) do
+        if not swim_cfg_options[k] then
+            return error(func_name..': unknown option '..k)
+        end
+    end
+    local uri = swim_check_uri(cfg.uri, func_name)
+    local heartbeat_rate = swim_check_timeout(cfg.heartbeat_rate,
+                                              func_name, 'heartbeat_rate')
+    local ack_timeout = swim_check_timeout(cfg.ack_timeout, func_name,
+                                           'ack_timeout');
+    local gc_mode = swim_check_gc_mode(cfg.gc_mode, func_name)
+    local uuid = swim_check_uuid(cfg.uuid, func_name)
+    if capi.swim_cfg(ptr, uri, heartbeat_rate, ack_timeout,
+                     gc_mode, uuid) ~= 0 then
+        return nil, box.error.last()
+    end
+    local index = c.index
+    for k, v in pairs(cfg) do
+        index[k] = v
+    end
+    return true
+end
+
+local swim_cfg_mt = {
+    __call = swim_cfg_call,
+    __index = function(c, k)
+        return c.index[k]
+    end,
+    __serialize = function(c)
+        return c.index
+    end,
+    __newindex = function()
+        return error('please, use swim:cfg{key = value} instead of '..
+                     'swim.cfg.key = value')
+    end
+}
+
+--
+-- First 'swim:cfg()' call is different from others. On success it
+-- replaces swim metatable with a full one, where all the variety
+-- of methods is available.
+--
+local function swim_cfg_first_call(c, s, cfg)
+    local ok, err = swim_cfg_call(c, s, cfg)
+    if not ok then
+        return ok, err
+    end
+    -- Update 'cfg' metatable as well to never call this
+    -- function again and use ordinary swim_cfg_call() directly.
+    setmetatable(c, swim_cfg_mt)
+    setmetatable(s, swim_mt)
+    return ok
+end
+
+local swim_not_configured_mt = {
+    __index = {
+        delete = swim_delete,
+        is_configured = swim_is_configured,
+    },
+    __serialize = swim_serialize
+}
+
+local swim_cfg_not_configured_mt = table.deepcopy(swim_cfg_mt)
+swim_cfg_not_configured_mt.__call = swim_cfg_first_call
+
+--
+-- Create a new SWIM instance, and configure if @a cfg is
+-- provided.
+--
+local function swim_new(cfg)
+    local ptr = capi.swim_new()
+    if ptr == nil then
+        return nil, box.error.last()
+    end
+    ffi.gc(ptr, capi.swim_delete)
+    local s = setmetatable({
+        ptr = ptr,
+        cfg = setmetatable({index = {}}, swim_cfg_not_configured_mt)
+    }, swim_not_configured_mt)
+    if cfg then
+        local ok, err = s:cfg(cfg)
+        if not ok then
+            s:delete()
+            return ok, err
+        end
+    end
+    return s
+end
+
+return {
+    new = swim_new,
+}
diff --git a/test/swim/box.lua b/test/swim/box.lua
new file mode 100644
index 000000000..b6a39575e
--- /dev/null
+++ b/test/swim/box.lua
@@ -0,0 +1,24 @@
+#!/usr/bin/env tarantool
+
+swim = require('swim')
+fiber = require('fiber')
+listen_uri = tostring(os.getenv("LISTEN"))
+listen_port = require('uri').parse(listen_uri).service
+
+box.cfg{}
+
+function uuid(i)
+    local min_valid_prefix = '00000000-0000-1000-8000-'
+    if i < 10 then
+        return min_valid_prefix..'00000000000'..tostring(i)
+    end
+    assert(i < 100)
+    return min_valid_prefix..'0000000000'..tostring(i)
+end
+
+function uri(port)
+    port = port or 0
+    return '127.0.0.1:'..tostring(port)
+end
+
+require('console').listen(os.getenv('ADMIN'))
diff --git a/test/swim/suite.ini b/test/swim/suite.ini
new file mode 100644
index 000000000..13189c1cf
--- /dev/null
+++ b/test/swim/suite.ini
@@ -0,0 +1,5 @@
+[default]
+core = tarantool
+description = SWIM tests
+script = box.lua
+is_parallel = True
diff --git a/test/swim/swim.result b/test/swim/swim.result
new file mode 100644
index 000000000..2e5025da6
--- /dev/null
+++ b/test/swim/swim.result
@@ -0,0 +1,201 @@
+test_run = require('test_run').new()
+---
+...
+test_run:cmd("push filter '\\.lua.*:[0-9]+: ' to '.lua:<line>: '")
+---
+- true
+...
+--
+-- gh-3234: SWIM gossip protocol.
+--
+-- Invalid cfg parameters.
+swim.new(1)
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected table configuration'
+...
+swim.new({uri = true})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected string URI or port number'
+...
+swim.new({heartbeat_rate = 'rate'})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected number heartbeat_rate'
+...
+swim.new({ack_timeout = 'timeout'})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected number ack_timeout'
+...
+swim.new({gc_mode = 'not a mode'})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: unknown gc_mode'
+...
+swim.new({gc_mode = 0})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: unknown gc_mode'
+...
+swim.new({uuid = 123})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected string UUID'
+...
+swim.new({uuid = '1234'})
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: invalid UUID'
+...
+-- Valid parameters, but invalid configuration.
+swim.new({})
+---
+- null
+- 'swim.cfg: UUID and URI are mandatory in a first config'
+...
+swim.new({uuid = uuid(1)})
+---
+- null
+- 'swim.cfg: UUID and URI are mandatory in a first config'
+...
+swim.new({uri = uri()})
+---
+- null
+- 'swim.cfg: UUID and URI are mandatory in a first config'
+...
+-- Check manual deletion.
+s = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s:delete()
+---
+...
+s:cfg({})
+---
+- error: 'builtin/swim.lua:<line>: the swim instance is deleted'
+...
+s = nil
+---
+...
+_ = collectgarbage('collect')
+---
+...
+s = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s:quit()
+---
+...
+s:is_configured()
+---
+- error: '[string "return s:is_configured() "]:1: the swim instance is deleted'
+...
+s = nil
+---
+...
+_ = collectgarbage('collect')
+---
+...
+s = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s:is_configured()
+---
+- true
+...
+s:size()
+---
+- 1
+...
+s.cfg
+---
+- uuid: 00000000-0000-1000-8000-000000000001
+  uri: 127.0.0.1:0
+...
+s.cfg.gc_mode = 'off'
+---
+- error: '[string "s.cfg.gc_mode = ''off'' "]:1: please, use swim:cfg{key = value}
+    instead of swim.cfg.key = value'
+...
+s:cfg{gc_mode = 'off'}
+---
+- true
+...
+s.cfg
+---
+- gc_mode: off
+  uuid: 00000000-0000-1000-8000-000000000001
+  uri: 127.0.0.1:0
+...
+s.cfg.gc_mode
+---
+- off
+...
+s.cfg.uuid
+---
+- 00000000-0000-1000-8000-000000000001
+...
+s.cfg()
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: first argument is not a SWIM instance'
+...
+s:cfg({wrong_opt = 100})
+---
+- error: 'swim:cfg: unknown option wrong_opt'
+...
+s:delete()
+---
+...
+-- Reconfigure.
+s = swim.new()
+---
+...
+s:is_configured()
+---
+- false
+...
+-- Check that not configured instance does not provide most of
+-- methods.
+s.quit == nil
+---
+- true
+...
+s:cfg({uuid = uuid(1), uri = uri()})
+---
+- true
+...
+s.quit ~= nil
+---
+- true
+...
+s:is_configured()
+---
+- true
+...
+s:size()
+---
+- 1
+...
+s = nil
+---
+...
+_ = collectgarbage('collect')
+---
+...
+-- Invalid usage.
+s = swim.new()
+---
+...
+s.delete()
+---
+- error: 'builtin/swim.lua:<line>: swim:delete: first argument is not a SWIM instance'
+...
+s.is_configured()
+---
+- error: 'builtin/swim.lua:<line>: swim:is_configured: first argument is not a SWIM instance'
+...
+s.cfg()
+---
+- error: 'builtin/swim.lua:<line>: swim:cfg: first argument is not a SWIM instance'
+...
+s:delete()
+---
+...
+test_run:cmd("clear filter")
+---
+- true
+...
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
new file mode 100644
index 000000000..719783133
--- /dev/null
+++ b/test/swim/swim.test.lua
@@ -0,0 +1,68 @@
+test_run = require('test_run').new()
+test_run:cmd("push filter '\\.lua.*:[0-9]+: ' to '.lua:<line>: '")
+--
+-- gh-3234: SWIM gossip protocol.
+--
+
+-- Invalid cfg parameters.
+swim.new(1)
+swim.new({uri = true})
+swim.new({heartbeat_rate = 'rate'})
+swim.new({ack_timeout = 'timeout'})
+swim.new({gc_mode = 'not a mode'})
+swim.new({gc_mode = 0})
+swim.new({uuid = 123})
+swim.new({uuid = '1234'})
+
+-- Valid parameters, but invalid configuration.
+swim.new({})
+swim.new({uuid = uuid(1)})
+swim.new({uri = uri()})
+
+-- Check manual deletion.
+s = swim.new({uuid = uuid(1), uri = uri()})
+s:delete()
+s:cfg({})
+s = nil
+_ = collectgarbage('collect')
+
+s = swim.new({uuid = uuid(1), uri = uri()})
+s:quit()
+s:is_configured()
+s = nil
+_ = collectgarbage('collect')
+
+s = swim.new({uuid = uuid(1), uri = uri()})
+s:is_configured()
+s:size()
+s.cfg
+s.cfg.gc_mode = 'off'
+s:cfg{gc_mode = 'off'}
+s.cfg
+s.cfg.gc_mode
+s.cfg.uuid
+s.cfg()
+s:cfg({wrong_opt = 100})
+s:delete()
+
+-- Reconfigure.
+s = swim.new()
+s:is_configured()
+-- Check that not configured instance does not provide most of
+-- methods.
+s.quit == nil
+s:cfg({uuid = uuid(1), uri = uri()})
+s.quit ~= nil
+s:is_configured()
+s:size()
+s = nil
+_ = collectgarbage('collect')
+
+-- Invalid usage.
+s = swim.new()
+s.delete()
+s.is_configured()
+s.cfg()
+s:delete()
+
+test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 05/10] swim: Lua bindings to manipulate member table
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (4 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 04/10] swim: introduce Lua interface Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:32   ` [tarantool-patches] " Konstantin Osipov
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 06/10] swim: Lua bindings to access individual members Vladislav Shpilevoy
                   ` (4 subsequent siblings)
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Expose methods to add, remove, probe members by uri, uuid. Expose
broadcast method to probe multiple members by port.

Part of #3234
---
 src/lua/swim.lua        |  71 +++++++++++++++++
 test/swim/swim.result   | 171 ++++++++++++++++++++++++++++++++++++++++
 test/swim/swim.test.lua |  60 ++++++++++++++
 3 files changed, 302 insertions(+)

diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index bca98627c..c828faceb 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -272,6 +272,73 @@ local function swim_serialize(s)
     return s.cfg.index
 end
 
+--
+-- Ping a member probably located at @a uri.
+--
+local function swim_probe_member(s, uri)
+    local func_name = 'swim:probe_member'
+    local ptr = swim_check_instance(s, func_name)
+    uri = swim_check_uri(uri, func_name)
+    if capi.swim_probe_member(ptr, uri) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
+--
+-- Add a new member to the member table explicitly.
+--
+local function swim_add_member(s, cfg)
+    local func_name = 'swim:add_member'
+    local ptr = swim_check_instance(s, func_name)
+    if type(cfg) ~= 'table' then
+        return error(func_name..': expected table member definition')
+    end
+    local uri = swim_check_uri(cfg.uri, func_name)
+    local uuid = swim_check_uuid(cfg.uuid, func_name)
+    if capi.swim_add_member(ptr, uri, uuid) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
+--
+-- Remove a member by @a uuid immediately from the local member
+-- table.
+--
+local function swim_remove_member(s, uuid)
+    local func_name = 'swim:remove_member'
+    local ptr = swim_check_instance(s, func_name)
+    uuid = swim_check_uuid(uuid, func_name)
+    if capi.swim_remove_member(ptr, uuid) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
+--
+-- Broadcast a ping message on all interfaces with @a port
+-- destination. Port can be omitted, then currently bound is used.
+--
+local function swim_broadcast(s, port)
+    local func_name = 'swim:broadcast'
+    local ptr = swim_check_instance(s, func_name)
+    if port == nil then
+        port = -1
+    else
+        if type(port) == 'string' then
+            port = tonumber(port)
+        end
+        if type(port) ~= 'number' then
+            return error(func_name..': expected number port')
+        end
+    end
+    if capi.swim_broadcast(ptr, port) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
 --
 -- Normal metatable of a configured SWIM instance.
 --
@@ -281,6 +348,10 @@ local swim_mt = {
         quit = swim_quit,
         size = swim_size,
         is_configured = swim_is_configured,
+        probe_member = swim_probe_member,
+        add_member = swim_add_member,
+        remove_member = swim_remove_member,
+        broadcast = swim_broadcast,
     },
     __serialize = swim_serialize
 }
diff --git a/test/swim/swim.result b/test/swim/swim.result
index 2e5025da6..0ab3dafe0 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -195,6 +195,177 @@ s.cfg()
 s:delete()
 ---
 ...
+--
+-- Basic member table manipulations.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(), heartbeat_rate = 0.01})
+---
+...
+s2 = swim.new({uuid = uuid(2), uri = listen_uri, heartbeat_rate = 0.01})
+---
+...
+s1.broadcast()
+---
+- error: 'builtin/swim.lua:<line>: swim:broadcast: first argument is not a SWIM instance'
+...
+s1:broadcast('wrong port')
+---
+- error: 'swim:broadcast: expected number port'
+...
+-- Note, broadcast takes a port, not a URI.
+s1:broadcast('127.0.0.1:3333')
+---
+- error: 'swim:broadcast: expected number port'
+...
+-- Ok to broadcast on default port.
+s1:broadcast()
+---
+- true
+...
+s1:broadcast(listen_port)
+---
+- true
+...
+while s2:size() ~= 2 do fiber.sleep(0.01) end
+---
+...
+s1:size()
+---
+- 2
+...
+s2:size()
+---
+- 2
+...
+s1.remove_member()
+---
+- error: 'builtin/swim.lua:<line>: swim:remove_member: first argument is not a SWIM instance'
+...
+s1:remove_member(100)
+---
+- error: 'builtin/swim.lua:<line>: swim:remove_member: expected string UUID'
+...
+s1:remove_member('1234')
+---
+- error: 'builtin/swim.lua:<line>: swim:remove_member: invalid UUID'
+...
+s1:remove_member(uuid(2))
+---
+- true
+...
+s1:size()
+---
+- 1
+...
+s1.add_member()
+---
+- error: 'builtin/swim.lua:<line>: swim:add_member: first argument is not a SWIM instance'
+...
+s1:add_member(100)
+---
+- error: 'swim:add_member: expected table member definition'
+...
+s1:add_member({uri = true})
+---
+- error: 'builtin/swim.lua:<line>: swim:add_member: expected string URI or port number'
+...
+s1:add_member({uri = listen_uri})
+---
+- null
+- 'swim.add_member: URI and UUID are mandatory'
+...
+s1:add_member({uuid = uuid(2)})
+---
+- null
+- 'swim.add_member: URI and UUID are mandatory'
+...
+s1:add_member({uri = listen_uri, uuid = uuid(2)})
+---
+- true
+...
+s1:add_member({uri = listen_uri, uuid = uuid(2)})
+---
+- null
+- 'swim.add_member: a member with such UUID already exists'
+...
+s1:size()
+---
+- 2
+...
+s1:cfg({uuid = uuid(3)})
+---
+- true
+...
+-- Can't remove self.
+s1:remove_member(uuid(3))
+---
+- null
+- 'swim.remove_member: can not remove self'
+...
+-- Not existing.
+s1:remove_member(uuid(4))
+---
+- true
+...
+-- Old self.
+s1:remove_member(uuid(1))
+---
+- true
+...
+s1:delete()
+---
+...
+s2:delete()
+---
+...
+s1 = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s2 = swim.new({uuid = uuid(2), uri = listen_uri})
+---
+...
+s1.probe_member()
+---
+- error: 'builtin/swim.lua:<line>: swim:probe_member: first argument is not a SWIM instance'
+...
+s1:probe_member()
+---
+- null
+- 'swim.probe_member: URI is mandatory'
+...
+s1:probe_member(true)
+---
+- error: 'builtin/swim.lua:<line>: swim:probe_member: expected string URI or port number'
+...
+-- Not existing URI is ok - nothing happens.
+s1:probe_member('127.0.0.1:1')
+---
+- true
+...
+fiber.yield()
+---
+...
+s1:size()
+---
+- 1
+...
+s1:probe_member(listen_uri)
+---
+- true
+...
+while s1:size() ~= 2 do fiber.sleep(0.01) end
+---
+...
+s2:size()
+---
+- 2
+...
+s1:delete()
+---
+...
+s2:delete()
+---
+...
 test_run:cmd("clear filter")
 ---
 - true
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index 719783133..1e55a828a 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -65,4 +65,64 @@ s.is_configured()
 s.cfg()
 s:delete()
 
+--
+-- Basic member table manipulations.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(), heartbeat_rate = 0.01})
+s2 = swim.new({uuid = uuid(2), uri = listen_uri, heartbeat_rate = 0.01})
+
+s1.broadcast()
+s1:broadcast('wrong port')
+-- Note, broadcast takes a port, not a URI.
+s1:broadcast('127.0.0.1:3333')
+-- Ok to broadcast on default port.
+s1:broadcast()
+
+s1:broadcast(listen_port)
+while s2:size() ~= 2 do fiber.sleep(0.01) end
+s1:size()
+s2:size()
+
+s1.remove_member()
+s1:remove_member(100)
+s1:remove_member('1234')
+s1:remove_member(uuid(2))
+s1:size()
+
+s1.add_member()
+s1:add_member(100)
+s1:add_member({uri = true})
+s1:add_member({uri = listen_uri})
+s1:add_member({uuid = uuid(2)})
+s1:add_member({uri = listen_uri, uuid = uuid(2)})
+s1:add_member({uri = listen_uri, uuid = uuid(2)})
+s1:size()
+
+s1:cfg({uuid = uuid(3)})
+-- Can't remove self.
+s1:remove_member(uuid(3))
+-- Not existing.
+s1:remove_member(uuid(4))
+-- Old self.
+s1:remove_member(uuid(1))
+
+s1:delete()
+s2:delete()
+
+s1 = swim.new({uuid = uuid(1), uri = uri()})
+s2 = swim.new({uuid = uuid(2), uri = listen_uri})
+s1.probe_member()
+s1:probe_member()
+s1:probe_member(true)
+-- Not existing URI is ok - nothing happens.
+s1:probe_member('127.0.0.1:1')
+fiber.yield()
+s1:size()
+s1:probe_member(listen_uri)
+while s1:size() ~= 2 do fiber.sleep(0.01) end
+s2:size()
+
+s1:delete()
+s2:delete()
+
 test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 05/10] swim: Lua bindings to manipulate member table
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 05/10] swim: Lua bindings to manipulate member table Vladislav Shpilevoy
@ 2019-05-16  7:32   ` Konstantin Osipov
  0 siblings, 0 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:32 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Expose methods to add, remove, probe members by uri, uuid. Expose
> broadcast method to probe multiple members by port.

OK to push.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 06/10] swim: Lua bindings to access individual members
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (5 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 05/10] swim: Lua bindings to manipulate member table Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 07/10] swim: pairs() function to iterate over member table Vladislav Shpilevoy
                   ` (3 subsequent siblings)
  10 siblings, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Expose API to search members by UUID, to read their attributes,
to set payload.

Part of #3234
---
 src/lua/swim.lua        | 222 ++++++++++++++++++++++++
 test/swim/swim.result   | 366 +++++++++++++++++++++++++++++++++++++++-
 test/swim/swim.test.lua | 119 +++++++++++++
 3 files changed, 705 insertions(+), 2 deletions(-)

diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index c828faceb..e40edda18 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -1,5 +1,7 @@
 local ffi = require('ffi')
 local uuid = require('uuid')
+local buffer = require('buffer')
+local msgpack = require('msgpack')
 
 ffi.cdef[[
     struct swim;
@@ -100,6 +102,14 @@ ffi.cdef[[
 local capi = ffi.C
 
 local swim_t = ffi.typeof('struct swim *')
+local swim_member_t = ffi.typeof('struct swim_member *')
+
+local swim_member_status_strs = {
+    [capi.MEMBER_ALIVE] = 'alive',
+    [capi.MEMBER_SUSPECTED] = 'suspected',
+    [capi.MEMBER_DEAD] = 'dead',
+    [capi.MEMBER_LEFT] = 'left'
+}
 
 --
 -- Check if @a value is something that can be passed as a
@@ -212,6 +222,142 @@ local function swim_check_instance(s, func_name)
     return error(func_name..': first argument is not a SWIM instance')
 end
 
+--
+-- The same for SWIM member.
+--
+local function swim_check_member(m, func_name)
+    if type(m) == 'table' then
+        local ptr = m.ptr
+        if ffi.istype(swim_member_t, ptr) then
+            return ptr
+        end
+    end
+    return error(func_name..': first argument is not a SWIM member')
+end
+
+--
+-- Member methods. Most of them are one-liners, not much to
+-- comment.
+--
+
+local function swim_member_status(m)
+    local ptr = swim_check_member(m, 'member:status()')
+    return swim_member_status_strs[tonumber(capi.swim_member_status(ptr))]
+end
+
+local function swim_member_uri(m)
+    local ptr = swim_check_member(m, 'member:uri()')
+    return ffi.string(capi.swim_member_uri(ptr))
+end
+
+local function swim_member_incarnation(m)
+    local ptr = swim_check_member(m, 'member:incarnation()')
+    return capi.swim_member_incarnation(ptr)
+end
+
+local function swim_member_is_dropped(m)
+    local ptr = swim_check_member(m, 'member:is_dropped()')
+    return capi.swim_member_is_dropped(ptr)
+end
+
+local function swim_member_payload_raw(ptr)
+    local int = buffer.scalar.ai
+    local cdata = capi.swim_member_payload(ptr, int)
+    return cdata, int[0]
+end
+
+--
+-- Payload can be bigger than KB, and probably it is undesirable
+-- to copy it into a Lua string or decode MessagePack into a
+-- Lua object. This method is the cheapest way of taking payload.
+--
+local function swim_member_payload_cdata(m)
+    local ptr = swim_check_member(m, 'member:payload_cdata()')
+    return swim_member_payload_raw(ptr)
+end
+
+--
+-- Cdata requires to keep explicit size, besides not all user
+-- methods can be able to work with cdata. This is why it may be
+-- needed to take payload as a string - text or binary.
+--
+local function swim_member_payload_str(m)
+    local ptr = swim_check_member(m, 'member:payload_str()')
+    return ffi.string(swim_member_payload_raw(ptr))
+end
+
+--
+-- Since this is a Lua module, a user is likely to use Lua objects
+-- as a payload - tables, numbers, string etc. And it is natural
+-- to expect that member:payload() should return the same object
+-- which was passed into swim:set_payload() on another instance.
+-- This member method tries to interpret payload as MessagePack,
+-- and if fails, returns the payload as a string.
+--
+local function swim_member_payload(m)
+    local ptr = swim_check_member(m, 'member:payload()')
+    local cdata, size = swim_member_payload_raw(ptr)
+    if size == 0 then
+        return ''
+    end
+    local ok, res = pcall(msgpack.decode, cdata, size)
+    if not ok then
+        return ffi.string(cdata, size)
+    end
+    return res
+end
+
+--
+-- Cdata UUID. It is ok to return cdata, because struct tt_uuid
+-- type has strong support by 'uuid' Lua module with nice
+-- metatable, serialization, string conversions etc.
+--
+local function swim_member_uuid(m)
+    return capi.swim_member_uuid(swim_check_member(m, 'member:uuid()'))
+end
+
+local function swim_member_serialize(m)
+    local _, size = swim_member_payload_raw(m.ptr)
+    return {
+        status = swim_member_status(m),
+        uuid = swim_member_uuid(m),
+        uri = swim_member_uri(m),
+        incarnation = swim_member_incarnation(m),
+        -- There are many ways to interpret a payload, and it is
+        -- not a job of a serialization method. Only binary size
+        -- here is returned to allow a user to detect, whether a
+        -- payload exists.
+        payload_size = size,
+    }
+end
+
+local swim_member_mt = {
+    __index = {
+        status = swim_member_status,
+        uuid = swim_member_uuid,
+        uri = swim_member_uri,
+        incarnation = swim_member_incarnation,
+        payload_cdata = swim_member_payload_cdata,
+        payload_str = swim_member_payload_str,
+        payload = swim_member_payload,
+        is_dropped = swim_member_is_dropped,
+    },
+    __serialize = swim_member_serialize,
+    __newindex = function(m)
+        return error('swim_member is a read-only object')
+    end
+}
+
+--
+-- Wrap a SWIM member into a table with proper metamethods. Also
+-- it is going to be used to cache a decoded payload.
+--
+local function swim_member_wrap(ptr)
+    capi.swim_member_ref(ptr)
+    ffi.gc(ptr, capi.swim_member_unref)
+    return setmetatable({ptr = ptr}, swim_member_mt)
+end
+
 --
 -- When a SWIM instance is deleted or has quited, it can't be used
 -- anymore. This function replaces all methods of a deleted
@@ -339,6 +485,78 @@ local function swim_broadcast(s, port)
     return true
 end
 
+--
+-- Shortcut to get the self member in O(1) not making a lookup
+-- into the member table.
+--
+local function swim_self(s)
+    return swim_member_wrap(capi.swim_self(swim_check_instance(s, 'swim:self')))
+end
+
+--
+-- Find a member by UUID in the local member table.
+--
+local function swim_member_by_uuid(s, uuid)
+    local func_name = 'swim:member_by_uuid'
+    local ptr = swim_check_instance(s, func_name)
+    uuid = swim_check_uuid(uuid, func_name)
+    local m = capi.swim_member_by_uuid(ptr, uuid)
+    if m == nil then
+        return nil
+    end
+    return swim_member_wrap(m)
+end
+
+--
+-- Set raw payload without any preprocessing nor encoding. It can
+-- be anything, not necessary MessagePack.
+--
+local function swim_set_payload_raw(s, payload, payload_size)
+    local func_name = 'swim:set_payload_raw'
+    local ptr = swim_check_instance(s, func_name)
+    if payload_size ~= nil and type(payload_size) ~= 'number' then
+        return error(func_name..': expected number payload size')
+    end
+    if type(payload) == 'cdata' then
+        if not payload_size then
+            return error(func_name..': size is mandatory for cdata payload')
+        end
+        payload = ffi.cast('const char *', payload)
+    elseif type(payload) == 'string' then
+        if not payload_size then
+            payload_size = payload:len()
+        elseif payload_size > payload:len() then
+            return error(func_name..': explicit payload size > string length')
+        end
+    else
+        return error(func_name..': raw payload should be either string or '..
+                     'cdata')
+    end
+    if capi.swim_set_payload(ptr, payload, payload_size) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
+--
+-- Set Lua object as a payload. It is encoded into MessagePack.
+--
+local function swim_set_payload(s, payload)
+    local func_name = 'swim:set_payload'
+    local ptr = swim_check_instance(s, func_name)
+    local payload_size = 0
+    if payload ~= nil then
+        local buf = buffer.IBUF_SHARED
+        buf:reset()
+        payload_size = msgpack.encode(payload, buf)
+        payload = buf.rpos
+    end
+    if capi.swim_set_payload(ptr, payload, payload_size) ~= 0 then
+        return nil, box.error.last()
+    end
+    return true
+end
+
 --
 -- Normal metatable of a configured SWIM instance.
 --
@@ -352,6 +570,10 @@ local swim_mt = {
         add_member = swim_add_member,
         remove_member = swim_remove_member,
         broadcast = swim_broadcast,
+        self = swim_self,
+        member_by_uuid = swim_member_by_uuid,
+        set_payload_raw = swim_set_payload_raw,
+        set_payload = swim_set_payload,
     },
     __serialize = swim_serialize
 }
diff --git a/test/swim/swim.result b/test/swim/swim.result
index 0ab3dafe0..4cf5c7f90 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -5,6 +5,16 @@ test_run:cmd("push filter '\\.lua.*:[0-9]+: ' to '.lua:<line>: '")
 ---
 - true
 ...
+test_run:cmd("push filter '127.0.0.1:[0-9]+$' to '127.0.0.1:<port>'")
+---
+- true
+...
+msgpack = require('msgpack')
+---
+...
+ffi = require('ffi')
+---
+...
 --
 -- gh-3234: SWIM gossip protocol.
 --
@@ -104,7 +114,7 @@ s:size()
 s.cfg
 ---
 - uuid: 00000000-0000-1000-8000-000000000001
-  uri: 127.0.0.1:0
+  uri: 127.0.0.1:<port>
 ...
 s.cfg.gc_mode = 'off'
 ---
@@ -119,7 +129,7 @@ s.cfg
 ---
 - gc_mode: off
   uuid: 00000000-0000-1000-8000-000000000001
-  uri: 127.0.0.1:0
+  uri: 127.0.0.1:<port>
 ...
 s.cfg.gc_mode
 ---
@@ -296,6 +306,18 @@ s1:cfg({uuid = uuid(3)})
 ---
 - true
 ...
+s1:self():uuid()
+---
+- 00000000-0000-1000-8000-000000000003
+...
+s1:member_by_uuid(uuid(1))
+---
+- uri: 127.0.0.1:<port>
+  status: left
+  incarnation: 1
+  uuid: 00000000-0000-1000-8000-000000000001
+  payload_size: 0
+...
 -- Can't remove self.
 s1:remove_member(uuid(3))
 ---
@@ -366,6 +388,346 @@ s1:delete()
 s2:delete()
 ---
 ...
+--
+-- Member API.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s = s1:self()
+---
+...
+s
+---
+- uri: 127.0.0.1:<port>
+  status: alive
+  incarnation: 1
+  uuid: 00000000-0000-1000-8000-000000000001
+  payload_size: 0
+...
+s:status()
+---
+- alive
+...
+s:uuid()
+---
+- 00000000-0000-1000-8000-000000000001
+...
+s:uri()
+---
+- 127.0.0.1:<port>
+...
+s:incarnation()
+---
+- 1
+...
+s:payload_cdata()
+---
+- 'cdata<const char *>: NULL'
+- 0
+...
+s:payload_str()
+---
+- 
+...
+s:payload()
+---
+- 
+...
+s:is_dropped()
+---
+- false
+...
+s.unknown_index
+---
+- null
+...
+s.status()
+---
+- error: 'builtin/swim.lua:<line>: member:status(): first argument is not a SWIM member'
+...
+s.uuid()
+---
+- error: 'builtin/swim.lua:<line>: member:uuid(): first argument is not a SWIM member'
+...
+s.uri()
+---
+- error: 'builtin/swim.lua:<line>: member:uri(): first argument is not a SWIM member'
+...
+s.incarnation()
+---
+- error: 'builtin/swim.lua:<line>: member:incarnation(): first argument is not a SWIM
+    member'
+...
+s.payload_cdata()
+---
+- error: 'builtin/swim.lua:<line>: member:payload_cdata(): first argument is not a SWIM
+    member'
+...
+s.payload_str()
+---
+- error: 'builtin/swim.lua:<line>: member:payload_str(): first argument is not a SWIM
+    member'
+...
+s.payload()
+---
+- error: 'builtin/swim.lua:<line>: member:payload(): first argument is not a SWIM member'
+...
+s.is_dropped()
+---
+- error: 'builtin/swim.lua:<line>: member:is_dropped(): first argument is not a SWIM
+    member'
+...
+s1:member_by_uuid(uuid(1)) ~= nil
+---
+- true
+...
+s1:member_by_uuid(50)
+---
+- error: 'builtin/swim.lua:<line>: swim:member_by_uuid: expected string UUID'
+...
+s1:member_by_uuid(uuid(2))
+---
+- null
+...
+s1:quit()
+---
+...
+s:status()
+---
+- left
+...
+s:is_dropped()
+---
+- true
+...
+--
+-- Payload.
+--
+s = swim.new({uuid = uuid(1), uri = uri()})
+---
+...
+s.set_payload()
+---
+- error: 'builtin/swim.lua:<line>: swim:set_payload: first argument is not a SWIM instance'
+...
+s.set_payload_raw()
+---
+- error: 'builtin/swim.lua:<line>: swim:set_payload_raw: first argument is not a SWIM
+    instance'
+...
+self = s:self()
+---
+...
+s:set_payload()
+---
+- true
+...
+self:payload()
+---
+- 
+...
+s:set_payload({a = 100})
+---
+- true
+...
+self:payload()
+---
+- {'a': 100}
+...
+s:set_payload(100)
+---
+- true
+...
+self:payload()
+---
+- 100
+...
+s:set_payload(false)
+---
+- true
+...
+self:payload()
+---
+- false
+...
+p = self:payload_str()
+---
+...
+p
+---
+- !!binary wg==
+...
+(msgpack.decode(p))
+---
+- false
+...
+p, size = self:payload_cdata()
+---
+...
+type(p)
+---
+- cdata
+...
+size
+---
+- 1
+...
+(msgpack.decode(p, size))
+---
+- false
+...
+s:set_payload(string.rep('a', 1500))
+---
+- null
+- Payload should be <= 1200 and >= 0
+...
+self:payload()
+---
+- false
+...
+s:set_payload()
+---
+- true
+...
+self:payload()
+---
+- 
+...
+-- Raw payload setting can be used when MessagePack is not needed,
+-- or already encoded.
+s:set_payload_raw(nil, '123')
+---
+- error: 'swim:set_payload_raw: expected number payload size'
+...
+s:set_payload_raw('123', -1)
+---
+- null
+- Payload should be <= 1200 and >= 0
+...
+size = 10
+---
+...
+cdata = ffi.new('int[?]', size)
+---
+...
+for i = 0, size - 1 do cdata[i] = i end
+---
+...
+bsize = ffi.sizeof('int') * size
+---
+...
+s:set_payload_raw(cdata)
+---
+- error: 'swim:set_payload_raw: size is mandatory for cdata payload'
+...
+s:set_payload_raw('str', 4)
+---
+- error: 'swim:set_payload_raw: explicit payload size > string length'
+...
+s:set_payload_raw(true)
+---
+- error: 'swim:set_payload_raw: raw payload should be either string or cdata'
+...
+s:set_payload_raw(cdata, bsize)
+---
+- true
+...
+self:payload_str():len() == bsize
+---
+- true
+...
+self_cdata, self_bsize = self:payload_cdata()
+---
+...
+self_bsize == bsize
+---
+- true
+...
+self_cdata = ffi.cast('int *', self_cdata)
+---
+...
+for i = 0, size - 1 do assert(self_cdata[i] == cdata[i]) end
+---
+...
+s:set_payload_raw('raw str')
+---
+- true
+...
+self:payload_str()
+---
+- raw str
+...
+s:set_payload_raw('raw str', 3)
+---
+- true
+...
+self:payload_str()
+---
+- raw
+...
+s:delete()
+---
+...
+self:is_dropped()
+---
+- true
+...
+--
+-- Check payload dissemination.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(), heartbeat_rate = 0.01})
+---
+...
+s2 = swim.new({uuid = uuid(2), uri = listen_port, heartbeat_rate = 0.01})
+---
+...
+s1:add_member({uuid = uuid(2), uri = listen_port})
+---
+- true
+...
+while s2:size() ~= 2 do fiber.sleep(0.01) end
+---
+...
+s1_view = s2:member_by_uuid(uuid(1))
+---
+...
+s1_view:payload()
+---
+- 
+...
+s1_view:incarnation()
+---
+- 1
+...
+s1:set_payload('payload')
+---
+- true
+...
+while s1_view:payload() ~= 'payload' do fiber.sleep(0.01) end
+---
+...
+s1_view:incarnation()
+---
+- 2
+...
+s1:set_payload('payload2')
+---
+- true
+...
+while s1_view:payload() ~= 'payload2' do fiber.sleep(0.01) end
+---
+...
+s1_view:incarnation()
+---
+- 3
+...
+s1:delete()
+---
+...
+s2:delete()
+---
+...
 test_run:cmd("clear filter")
 ---
 - true
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index 1e55a828a..8e7b426fe 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -1,5 +1,8 @@
 test_run = require('test_run').new()
 test_run:cmd("push filter '\\.lua.*:[0-9]+: ' to '.lua:<line>: '")
+test_run:cmd("push filter '127.0.0.1:[0-9]+$' to '127.0.0.1:<port>'")
+msgpack = require('msgpack')
+ffi = require('ffi')
 --
 -- gh-3234: SWIM gossip protocol.
 --
@@ -99,6 +102,8 @@ s1:add_member({uri = listen_uri, uuid = uuid(2)})
 s1:size()
 
 s1:cfg({uuid = uuid(3)})
+s1:self():uuid()
+s1:member_by_uuid(uuid(1))
 -- Can't remove self.
 s1:remove_member(uuid(3))
 -- Not existing.
@@ -125,4 +130,118 @@ s2:size()
 s1:delete()
 s2:delete()
 
+--
+-- Member API.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri()})
+s = s1:self()
+s
+s:status()
+s:uuid()
+s:uri()
+s:incarnation()
+s:payload_cdata()
+s:payload_str()
+s:payload()
+s:is_dropped()
+s.unknown_index
+
+s.status()
+s.uuid()
+s.uri()
+s.incarnation()
+s.payload_cdata()
+s.payload_str()
+s.payload()
+s.is_dropped()
+
+s1:member_by_uuid(uuid(1)) ~= nil
+s1:member_by_uuid(50)
+s1:member_by_uuid(uuid(2))
+
+s1:quit()
+s:status()
+s:is_dropped()
+
+--
+-- Payload.
+--
+s = swim.new({uuid = uuid(1), uri = uri()})
+s.set_payload()
+s.set_payload_raw()
+
+self = s:self()
+s:set_payload()
+self:payload()
+s:set_payload({a = 100})
+self:payload()
+s:set_payload(100)
+self:payload()
+s:set_payload(false)
+self:payload()
+
+p = self:payload_str()
+p
+(msgpack.decode(p))
+
+p, size = self:payload_cdata()
+type(p)
+size
+(msgpack.decode(p, size))
+
+s:set_payload(string.rep('a', 1500))
+self:payload()
+s:set_payload()
+self:payload()
+
+-- Raw payload setting can be used when MessagePack is not needed,
+-- or already encoded.
+s:set_payload_raw(nil, '123')
+s:set_payload_raw('123', -1)
+
+size = 10
+cdata = ffi.new('int[?]', size)
+for i = 0, size - 1 do cdata[i] = i end
+bsize = ffi.sizeof('int') * size
+s:set_payload_raw(cdata)
+s:set_payload_raw('str', 4)
+s:set_payload_raw(true)
+
+s:set_payload_raw(cdata, bsize)
+self:payload_str():len() == bsize
+self_cdata, self_bsize = self:payload_cdata()
+self_bsize == bsize
+self_cdata = ffi.cast('int *', self_cdata)
+for i = 0, size - 1 do assert(self_cdata[i] == cdata[i]) end
+
+s:set_payload_raw('raw str')
+self:payload_str()
+s:set_payload_raw('raw str', 3)
+self:payload_str()
+
+s:delete()
+self:is_dropped()
+
+--
+-- Check payload dissemination.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(), heartbeat_rate = 0.01})
+s2 = swim.new({uuid = uuid(2), uri = listen_port, heartbeat_rate = 0.01})
+s1:add_member({uuid = uuid(2), uri = listen_port})
+while s2:size() ~= 2 do fiber.sleep(0.01) end
+s1_view = s2:member_by_uuid(uuid(1))
+s1_view:payload()
+s1_view:incarnation()
+
+s1:set_payload('payload')
+while s1_view:payload() ~= 'payload' do fiber.sleep(0.01) end
+s1_view:incarnation()
+
+s1:set_payload('payload2')
+while s1_view:payload() ~= 'payload2' do fiber.sleep(0.01) end
+s1_view:incarnation()
+
+s1:delete()
+s2:delete()
+
 test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 07/10] swim: pairs() function to iterate over member table
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (6 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 06/10] swim: Lua bindings to access individual members Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 08/10] swim: allow to use cdata struct tt_uuid in Lua API Vladislav Shpilevoy
                   ` (2 subsequent siblings)
  10 siblings, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

The last patch in the series of Lua bindings exposure exposes
iterators API to be able to iterate over a member table in a
'for' loop like it would just a Lua table.

Part of #3234
---
 src/lua/swim.lua        | 34 ++++++++++++++++++
 test/swim/swim.result   | 78 +++++++++++++++++++++++++++++++++++++++++
 test/swim/swim.test.lua | 15 ++++++++
 3 files changed, 127 insertions(+)

diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index e40edda18..6e55ffeba 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -557,6 +557,39 @@ local function swim_set_payload(s, payload)
     return true
 end
 
+--
+-- Lua pairs() or similar function should return 3 values:
+-- iterator function, iterator object, a key before first. This is
+-- iterator function. On each iteration it returns UUID as a key,
+-- member object as a value.
+--
+local function swim_pairs_next(ctx)
+    if ctx.swim.ptr == nil then
+        return swim_error_deleted()
+    end
+    local iterator = ctx.iterator
+    local m = capi.swim_iterator_next(iterator)
+    if m ~= nil then
+        m = swim_member_wrap(m)
+        return m:uuid(), m
+    end
+    capi.swim_iterator_close(ffi.gc(iterator, nil))
+    ctx.iterator = nil
+end
+
+--
+-- Pairs() to use in 'for' cycles.
+--
+local function swim_pairs(s)
+    local ptr = swim_check_instance(s, 'swim:pairs')
+    local iterator = capi.swim_iterator_open(ptr)
+    if iterator == nil then
+        return nil, box.error.last()
+    end
+    ffi.gc(iterator, capi.swim_iterator_close)
+    return swim_pairs_next, {swim = s, iterator = iterator}, nil
+end
+
 --
 -- Normal metatable of a configured SWIM instance.
 --
@@ -574,6 +607,7 @@ local swim_mt = {
         member_by_uuid = swim_member_by_uuid,
         set_payload_raw = swim_set_payload_raw,
         set_payload = swim_set_payload,
+        pairs = swim_pairs,
     },
     __serialize = swim_serialize
 }
diff --git a/test/swim/swim.result b/test/swim/swim.result
index 4cf5c7f90..d0838af48 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -728,6 +728,84 @@ s1:delete()
 s2:delete()
 ---
 ...
+--
+-- Iterators.
+--
+function iterate() local t = {} for k, v in s:pairs() do table.insert(t, {k, v}) end return t end
+---
+...
+s = swim.new()
+---
+...
+iterate()
+---
+- error: '[string "function iterate() local t = {} for k, v in s..."]:1: attempt to
+    call method ''pairs'' (a nil value)'
+...
+s:cfg({uuid = uuid(1), uri = uri(), gc_mode = 'off'})
+---
+- true
+...
+s.pairs()
+---
+- error: 'builtin/swim.lua:<line>: swim:pairs: first argument is not a SWIM instance'
+...
+iterate()
+---
+- - - 00000000-0000-1000-8000-000000000001
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 1
+      uuid: 00000000-0000-1000-8000-000000000001
+      payload_size: 0
+...
+s:add_member({uuid = uuid(2), uri = uri()})
+---
+- true
+...
+iterate()
+---
+- - - 00000000-0000-1000-8000-000000000002
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 0
+      uuid: 00000000-0000-1000-8000-000000000002
+      payload_size: 0
+  - - 00000000-0000-1000-8000-000000000001
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 1
+      uuid: 00000000-0000-1000-8000-000000000001
+      payload_size: 0
+...
+s:add_member({uuid = uuid(3), uri = uri()})
+---
+- true
+...
+iterate()
+---
+- - - 00000000-0000-1000-8000-000000000001
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 1
+      uuid: 00000000-0000-1000-8000-000000000001
+      payload_size: 0
+  - - 00000000-0000-1000-8000-000000000003
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 0
+      uuid: 00000000-0000-1000-8000-000000000003
+      payload_size: 0
+  - - 00000000-0000-1000-8000-000000000002
+    - uri: 127.0.0.1:<port>
+      status: alive
+      incarnation: 0
+      uuid: 00000000-0000-1000-8000-000000000002
+      payload_size: 0
+...
+s:delete()
+---
+...
 test_run:cmd("clear filter")
 ---
 - true
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index 8e7b426fe..9f237a540 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -244,4 +244,19 @@ s1_view:incarnation()
 s1:delete()
 s2:delete()
 
+--
+-- Iterators.
+--
+function iterate() local t = {} for k, v in s:pairs() do table.insert(t, {k, v}) end return t end
+s = swim.new()
+iterate()
+s:cfg({uuid = uuid(1), uri = uri(), gc_mode = 'off'})
+s.pairs()
+iterate()
+s:add_member({uuid = uuid(2), uri = uri()})
+iterate()
+s:add_member({uuid = uuid(3), uri = uri()})
+iterate()
+s:delete()
+
 test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 08/10] swim: allow to use cdata struct tt_uuid in Lua API
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (7 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 07/10] swim: pairs() function to iterate over member table Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 09/10] swim: cache decoded payload in the Lua module Vladislav Shpilevoy
  2019-05-21 16:57 ` [tarantool-patches] Re: [PATCH 00/10] swim Lua API Vladislav Shpilevoy
  10 siblings, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Sometimes, especially in tests, it is useful to make something
like this:

    s:add_member({uuid = member:uuid(), uri = member:uri()})

But member:uuid() is cdata struct tt_uuid. This commit allows
that.

Part of #3234
---
 src/lua/swim.lua        |  5 ++++-
 test/swim/swim.result   | 17 ++++++++++++++---
 test/swim/swim.test.lua |  3 +++
 3 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index 6e55ffeba..80c7375ab 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -194,7 +194,10 @@ local function swim_check_uuid(value, func_name)
         return nil
     end
     if type(value) ~= 'string' then
-        return error(func_name..': expected string UUID')
+        if ffi.istype('struct tt_uuid', value) then
+            return value
+        end
+        return error(func_name..': expected string UUID or struct tt_uuid')
     end
     value = uuid.fromstr(value)
     if not value then
diff --git a/test/swim/swim.result b/test/swim/swim.result
index d0838af48..91a8ab6e9 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -45,7 +45,7 @@ swim.new({gc_mode = 0})
 ...
 swim.new({uuid = 123})
 ---
-- error: 'builtin/swim.lua:<line>: swim:cfg: expected string UUID'
+- error: 'builtin/swim.lua:<line>: swim:cfg: expected string UUID or struct tt_uuid'
 ...
 swim.new({uuid = '1234'})
 ---
@@ -253,7 +253,8 @@ s1.remove_member()
 ...
 s1:remove_member(100)
 ---
-- error: 'builtin/swim.lua:<line>: swim:remove_member: expected string UUID'
+- error: 'builtin/swim.lua:<line>: swim:remove_member: expected string UUID or struct
+    tt_uuid'
 ...
 s1:remove_member('1234')
 ---
@@ -484,12 +485,22 @@ s1:member_by_uuid(uuid(1)) ~= nil
 ...
 s1:member_by_uuid(50)
 ---
-- error: 'builtin/swim.lua:<line>: swim:member_by_uuid: expected string UUID'
+- error: 'builtin/swim.lua:<line>: swim:member_by_uuid: expected string UUID or struct
+    tt_uuid'
 ...
 s1:member_by_uuid(uuid(2))
 ---
 - null
 ...
+-- UUID can be cdata.
+s1:member_by_uuid(s:uuid())
+---
+- uri: 127.0.0.1:<port>
+  status: alive
+  incarnation: 1
+  uuid: 00000000-0000-1000-8000-000000000001
+  payload_size: 0
+...
 s1:quit()
 ---
 ...
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index 9f237a540..a744b2a29 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -159,6 +159,9 @@ s1:member_by_uuid(uuid(1)) ~= nil
 s1:member_by_uuid(50)
 s1:member_by_uuid(uuid(2))
 
+-- UUID can be cdata.
+s1:member_by_uuid(s:uuid())
+
 s1:quit()
 s:status()
 s:is_dropped()
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] [PATCH 09/10] swim: cache decoded payload in the Lua module
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (8 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 08/10] swim: allow to use cdata struct tt_uuid in Lua API Vladislav Shpilevoy
@ 2019-05-15 19:36 ` Vladislav Shpilevoy
  2019-05-16  7:36   ` [tarantool-patches] " Konstantin Osipov
  2019-05-21 16:57 ` [tarantool-patches] Re: [PATCH 00/10] swim Lua API Vladislav Shpilevoy
  10 siblings, 1 reply; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-15 19:36 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Users of Lua SWIM module likely will use Lua objects as a
payload. Lua objects are serialized into MessagePack
automatically, and deserialized back on other instances. But
deserialization of 1.2Kb payload on each member:payload()
invocation is quite heavy operation. This commit caches decoded
payloads to return them again until change.

A microbenchmark showed, that cached payload is returned ~100
times faster, than it is decoded each time. Even though a tested
payload was quite small and simple:

    s:set_payload({a = 100, b = 200})

Even this payload is returned 100 times faster, and does not
affect GC.

Part of #3234
---
 extra/exports           |   1 +
 src/lib/swim/swim.c     |   6 +++
 src/lib/swim/swim.h     |  12 +++++
 src/lua/swim.lua        |  39 +++++++++++---
 test/swim/swim.result   | 113 ++++++++++++++++++++++++++++++++++++++++
 test/swim/swim.test.lua |  45 ++++++++++++++++
 6 files changed, 208 insertions(+), 8 deletions(-)

diff --git a/extra/exports b/extra/exports
index b0e20e11f..8a38e9ef2 100644
--- a/extra/exports
+++ b/extra/exports
@@ -111,6 +111,7 @@ swim_member_payload
 swim_member_ref
 swim_member_unref
 swim_member_is_dropped
+swim_member_is_payload_up_to_date
 
 # Module API
 
diff --git a/src/lib/swim/swim.c b/src/lib/swim/swim.c
index fb2b1490b..9d67c4cb8 100644
--- a/src/lib/swim/swim.c
+++ b/src/lib/swim/swim.c
@@ -2080,3 +2080,9 @@ swim_member_payload(const struct swim_member *member, int *size)
 	*size = member->payload_size;
 	return member->payload;
 }
+
+bool
+swim_member_is_payload_up_to_date(const struct swim_member *member)
+{
+	return member->is_payload_up_to_date;
+}
diff --git a/src/lib/swim/swim.h b/src/lib/swim/swim.h
index 8c4f8c3cf..5f3134cc4 100644
--- a/src/lib/swim/swim.h
+++ b/src/lib/swim/swim.h
@@ -220,6 +220,18 @@ swim_member_incarnation(const struct swim_member *member);
 const char *
 swim_member_payload(const struct swim_member *member, int *size);
 
+/**
+ * Check if member's payload is up to its incarnation. Sometimes
+ * it happens, that a member has changed payload, but other
+ * members learned only a new incarnation without the new payload.
+ * Then the payload is considered outdated, and is updated
+ * eventually later. The method is rather internal, and should not
+ * be used by any public API. It is exposed to implement decoded
+ * payload cache in the SWIM Lua module.
+ */
+bool
+swim_member_is_payload_up_to_date(const struct swim_member *member);
+
 /**
  * Reference a member. The member memory will be valid until unref
  * is called.
diff --git a/src/lua/swim.lua b/src/lua/swim.lua
index 80c7375ab..852192479 100644
--- a/src/lua/swim.lua
+++ b/src/lua/swim.lua
@@ -96,6 +96,9 @@ ffi.cdef[[
 
     bool
     swim_member_is_dropped(const struct swim_member *member);
+
+    bool
+    swim_member_is_payload_up_to_date(const struct swim_member *member);
 ]]
 
 -- Shortcut to avoid unnecessary lookups in 'ffi' table.
@@ -297,17 +300,36 @@ end
 -- This member method tries to interpret payload as MessagePack,
 -- and if fails, returns the payload as a string.
 --
+-- This function caches its result. It means, that only first call
+-- actually decodes cdata payload. All the next calls return
+-- pointer to the same result, until payload is changed with a new
+-- incarnation.
+--
 local function swim_member_payload(m)
     local ptr = swim_check_member(m, 'member:payload()')
+    -- Two keys are needed. Incarnation is not enough, because a
+    -- new incarnation can be disseminated earlier than a new
+    -- payload. For example, via ACK messages.
+    local key1 = capi.swim_member_incarnation(ptr)
+    local key2 = capi.swim_member_is_payload_up_to_date(ptr)
+    if key1 == m.p_key1 and key2 == m.p_key2 then
+        return m.p
+    end
     local cdata, size = swim_member_payload_raw(ptr)
+    local ok, result
     if size == 0 then
-        return ''
-    end
-    local ok, res = pcall(msgpack.decode, cdata, size)
-    if not ok then
-        return ffi.string(cdata, size)
+        result = ''
+    else
+        ok, result = pcall(msgpack.decode, cdata, size)
+        if not ok then
+            result = ffi.string(cdata, size)
+        end
     end
-    return res
+    -- Member bans new indexes. Only rawset() can be used.
+    rawset(m, 'p', result)
+    rawset(m, 'p_key1', key1)
+    rawset(m, 'p_key2', key2)
+    return result
 end
 
 --
@@ -352,8 +374,9 @@ local swim_member_mt = {
 }
 
 --
--- Wrap a SWIM member into a table with proper metamethods. Also
--- it is going to be used to cache a decoded payload.
+-- Wrap a SWIM member into a table with proper metamethods. The
+-- table-wrapper stores not only a pointer, but also cached
+-- decoded payload.
 --
 local function swim_member_wrap(ptr)
     capi.swim_member_ref(ptr)
diff --git a/test/swim/swim.result b/test/swim/swim.result
index 91a8ab6e9..ef7203a37 100644
--- a/test/swim/swim.result
+++ b/test/swim/swim.result
@@ -817,6 +817,119 @@ iterate()
 s:delete()
 ---
 ...
+--
+-- Payload caching.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(listen_port), heartbeat_rate = 0.01})
+---
+...
+s2 = swim.new({uuid = uuid(2), uri = uri(), heartbeat_rate = 0.01})
+---
+...
+s1_self = s1:self()
+---
+...
+s1:add_member({uuid = s2:self():uuid(), uri = s2:self():uri()})
+---
+- true
+...
+s2:add_member({uuid = s1_self:uuid(), uri = s1_self:uri()})
+---
+- true
+...
+s1:size()
+---
+- 2
+...
+s2:size()
+---
+- 2
+...
+s1_view = s2:member_by_uuid(s1_self:uuid())
+---
+...
+s1:set_payload({a = 100})
+---
+- true
+...
+p = s1_self:payload()
+---
+...
+s1_self:payload() == p
+---
+- true
+...
+while s1_view:payload() == '' do fiber.sleep(0.01) end
+---
+...
+p = s1_view:payload()
+---
+...
+s1_view:payload() == p
+---
+- true
+...
+-- Now a complex case. It is possible, that a new member's
+-- incarnation is learned, but new payload is not. Payload cache
+-- should correctly process that.
+s1:cfg({heartbeat_rate = 1000})
+---
+- true
+...
+s2:cfg({heartbeat_rate = 1000})
+---
+- true
+...
+s1:set_payload({a = 200})
+---
+- true
+...
+-- Via probe() S2 learns new incarnation of S1, but without new
+-- payload.
+s2:probe_member(s1_self:uri())
+---
+- true
+...
+s1_view:payload()
+---
+- {'a': 100}
+...
+s1_view:incarnation()
+---
+- 2
+...
+s1:cfg({heartbeat_rate = 0.01})
+---
+- true
+...
+s2:cfg({heartbeat_rate = 0.01})
+---
+- true
+...
+while s1_view:payload().a ~= 200 do fiber.sleep(0.01) end
+---
+...
+p = s1_view:payload()
+---
+...
+s1_view:payload() == p
+---
+- true
+...
+p
+---
+- {'a': 200}
+...
+s1_view:incarnation()
+---
+- 2
+...
+s1:delete()
+---
+...
+s2:delete()
+---
+...
 test_run:cmd("clear filter")
 ---
 - true
diff --git a/test/swim/swim.test.lua b/test/swim/swim.test.lua
index a744b2a29..c3387cf0a 100644
--- a/test/swim/swim.test.lua
+++ b/test/swim/swim.test.lua
@@ -262,4 +262,49 @@ s:add_member({uuid = uuid(3), uri = uri()})
 iterate()
 s:delete()
 
+--
+-- Payload caching.
+--
+s1 = swim.new({uuid = uuid(1), uri = uri(listen_port), heartbeat_rate = 0.01})
+s2 = swim.new({uuid = uuid(2), uri = uri(), heartbeat_rate = 0.01})
+s1_self = s1:self()
+s1:add_member({uuid = s2:self():uuid(), uri = s2:self():uri()})
+s2:add_member({uuid = s1_self:uuid(), uri = s1_self:uri()})
+s1:size()
+s2:size()
+s1_view = s2:member_by_uuid(s1_self:uuid())
+
+s1:set_payload({a = 100})
+p = s1_self:payload()
+s1_self:payload() == p
+
+while s1_view:payload() == '' do fiber.sleep(0.01) end
+p = s1_view:payload()
+s1_view:payload() == p
+
+-- Now a complex case. It is possible, that a new member's
+-- incarnation is learned, but new payload is not. Payload cache
+-- should correctly process that.
+
+s1:cfg({heartbeat_rate = 1000})
+s2:cfg({heartbeat_rate = 1000})
+
+s1:set_payload({a = 200})
+-- Via probe() S2 learns new incarnation of S1, but without new
+-- payload.
+s2:probe_member(s1_self:uri())
+s1_view:payload()
+s1_view:incarnation()
+
+s1:cfg({heartbeat_rate = 0.01})
+s2:cfg({heartbeat_rate = 0.01})
+while s1_view:payload().a ~= 200 do fiber.sleep(0.01) end
+p = s1_view:payload()
+s1_view:payload() == p
+p
+s1_view:incarnation()
+
+s1:delete()
+s2:delete()
+
 test_run:cmd("clear filter")
-- 
2.20.1 (Apple Git-117)

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 09/10] swim: cache decoded payload in the Lua module
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 09/10] swim: cache decoded payload in the Lua module Vladislav Shpilevoy
@ 2019-05-16  7:36   ` Konstantin Osipov
  2019-05-16 11:58     ` Vladislav Shpilevoy
  2019-05-16 22:46     ` Vladislav Shpilevoy
  0 siblings, 2 replies; 20+ messages in thread
From: Konstantin Osipov @ 2019-05-16  7:36 UTC (permalink / raw)
  To: Vladislav Shpilevoy; +Cc: tarantool-patches

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
> Users of Lua SWIM module likely will use Lua objects as a
> payload. Lua objects are serialized into MessagePack
> automatically, and deserialized back on other instances. But
> deserialization of 1.2Kb payload on each member:payload()
> invocation is quite heavy operation. This commit caches decoded
> payloads to return them again until change.

The entire stack is OK to push. But the stack has no docbot entry.
Please submit a documentation request with a complete description
of SWIM and the API: it's a complicated topic and one can't expect
the docs team to handle it alone.

I know we have agreed to not document it yet, but the docs should
be ready: I don't expect the api or swim nature will change
significantly in the future.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 09/10] swim: cache decoded payload in the Lua module
  2019-05-16  7:36   ` [tarantool-patches] " Konstantin Osipov
@ 2019-05-16 11:58     ` Vladislav Shpilevoy
  2019-05-16 22:46     ` Vladislav Shpilevoy
  1 sibling, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-16 11:58 UTC (permalink / raw)
  To: Konstantin Osipov; +Cc: tarantool-patches

The patch depends on gerold103/static-allocator-lua. Please,
review it first.

https://www.freelists.org/post/tarantool-patches/PATCH-11-buffer-port-static-allocator-to-Lua

On 16/05/2019 10:36, Konstantin Osipov wrote:
> * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
>> Users of Lua SWIM module likely will use Lua objects as a
>> payload. Lua objects are serialized into MessagePack
>> automatically, and deserialized back on other instances. But
>> deserialization of 1.2Kb payload on each member:payload()
>> invocation is quite heavy operation. This commit caches decoded
>> payloads to return them again until change.
> 
> The entire stack is OK to push. But the stack has no docbot entry.
> Please submit a documentation request with a complete description
> of SWIM and the API: it's a complicated topic and one can't expect
> the docs team to handle it alone.
> 
> I know we have agreed to not document it yet, but the docs should
> be ready: I don't expect the api or swim nature will change
> significantly in the future.
> 
> 

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 09/10] swim: cache decoded payload in the Lua module
  2019-05-16  7:36   ` [tarantool-patches] " Konstantin Osipov
  2019-05-16 11:58     ` Vladislav Shpilevoy
@ 2019-05-16 22:46     ` Vladislav Shpilevoy
  1 sibling, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-16 22:46 UTC (permalink / raw)
  To: tarantool-patches, Konstantin Osipov



On 16/05/2019 10:36, Konstantin Osipov wrote:
> * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/05/16 09:53]:
>> Users of Lua SWIM module likely will use Lua objects as a
>> payload. Lua objects are serialized into MessagePack
>> automatically, and deserialized back on other instances. But
>> deserialization of 1.2Kb payload on each member:payload()
>> invocation is quite heavy operation. This commit caches decoded
>> payloads to return them again until change.
> 
> The entire stack is OK to push. But the stack has no docbot entry.
> Please submit a documentation request with a complete description
> of SWIM and the API: it's a complicated topic and one can't expect
> the docs team to handle it alone.

The request appeared to be so huge, that I decided to paste it
in the ticket:
https://github.com/tarantool/tarantool/issues/3234#issuecomment-493256667

^ permalink raw reply	[flat|nested] 20+ messages in thread

* [tarantool-patches] Re: [PATCH 00/10] swim Lua API
  2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
                   ` (9 preceding siblings ...)
  2019-05-15 19:36 ` [tarantool-patches] [PATCH 09/10] swim: cache decoded payload in the Lua module Vladislav Shpilevoy
@ 2019-05-21 16:57 ` Vladislav Shpilevoy
  10 siblings, 0 replies; 20+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-21 16:57 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja

Pushed to the master.

On 15/05/2019 22:36, Vladislav Shpilevoy wrote:
> This huge patchset introduces SWIM Lua module - wrapper around C SWIM. It
> duplicates C API one in one, with additional internal checks related to Lua
> dynamic typing.
> 
> First 3 patches are preparatory, they fix some bugs in the C module appeared
> during Lua API development.
> 
> Next 4 patches are the core - 4 independent parts of Lua API. They are simple
> and somewhere not efficient.
> 
> Next 3 patches are optimizations, then eliminate some performance problems of
> the core patches.
> 
> Branch: http://github.com/tarantool/tarantool/tree/gerold103/swim-lua
> Issue: https://github.com/tarantool/tarantool/issues/3234
> 
> Vladislav Shpilevoy (10):
>   swim: fix an assertion on attempt to chage timeouts
>   swim: make swim_new_round() void
>   swim: validate URI in swim_probe_member()
>   swim: introduce Lua interface
>   swim: Lua bindings to manipulate member table
>   swim: Lua bindings to access individual members
>   swim: pairs() function to iterate over member table
>   swim: allow to use cdata struct tt_uuid in Lua API
>   swim: cache decoded payload in the Lua module
>   swim: cache members in Lua member table
> 
>  extra/exports           |   26 +
>  src/CMakeLists.txt      |    1 +
>  src/lib/swim/swim.c     |   46 +-
>  src/lib/swim/swim.h     |   12 +
>  src/lib/swim/swim_ev.h  |    2 +
>  src/lua/init.c          |    2 +
>  src/lua/swim.lua        |  777 ++++++++++++++++++++++++++++++
>  test/swim/box.lua       |   24 +
>  test/swim/suite.ini     |    5 +
>  test/swim/swim.result   | 1011 +++++++++++++++++++++++++++++++++++++++
>  test/swim/swim.test.lua |  340 +++++++++++++
>  test/unit/swim.c        |    3 +-
>  test/unit/swim.result   |    7 +-
>  13 files changed, 2234 insertions(+), 22 deletions(-)
>  create mode 100644 src/lua/swim.lua
>  create mode 100644 test/swim/box.lua
>  create mode 100644 test/swim/suite.ini
>  create mode 100644 test/swim/swim.result
>  create mode 100644 test/swim/swim.test.lua
> 

^ permalink raw reply	[flat|nested] 20+ messages in thread

end of thread, other threads:[~2019-05-21 16:57 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-15 19:36 [tarantool-patches] [PATCH 00/10] swim Lua API Vladislav Shpilevoy
2019-05-15 19:36 ` [tarantool-patches] [PATCH 01/10] swim: fix an assertion on attempt to chage timeouts Vladislav Shpilevoy
2019-05-16  7:28   ` [tarantool-patches] " Konstantin Osipov
2019-05-15 19:36 ` [tarantool-patches] [PATCH 10/10] swim: cache members in Lua member table Vladislav Shpilevoy
2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
2019-05-15 19:36 ` [tarantool-patches] [PATCH 02/10] swim: make swim_new_round() void Vladislav Shpilevoy
2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
2019-05-15 19:36 ` [tarantool-patches] [PATCH 03/10] swim: validate URI in swim_probe_member() Vladislav Shpilevoy
2019-05-16  7:31   ` [tarantool-patches] " Konstantin Osipov
2019-05-15 19:36 ` [tarantool-patches] [PATCH 04/10] swim: introduce Lua interface Vladislav Shpilevoy
2019-05-15 19:36 ` [tarantool-patches] [PATCH 05/10] swim: Lua bindings to manipulate member table Vladislav Shpilevoy
2019-05-16  7:32   ` [tarantool-patches] " Konstantin Osipov
2019-05-15 19:36 ` [tarantool-patches] [PATCH 06/10] swim: Lua bindings to access individual members Vladislav Shpilevoy
2019-05-15 19:36 ` [tarantool-patches] [PATCH 07/10] swim: pairs() function to iterate over member table Vladislav Shpilevoy
2019-05-15 19:36 ` [tarantool-patches] [PATCH 08/10] swim: allow to use cdata struct tt_uuid in Lua API Vladislav Shpilevoy
2019-05-15 19:36 ` [tarantool-patches] [PATCH 09/10] swim: cache decoded payload in the Lua module Vladislav Shpilevoy
2019-05-16  7:36   ` [tarantool-patches] " Konstantin Osipov
2019-05-16 11:58     ` Vladislav Shpilevoy
2019-05-16 22:46     ` Vladislav Shpilevoy
2019-05-21 16:57 ` [tarantool-patches] Re: [PATCH 00/10] swim Lua API Vladislav Shpilevoy

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox