From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id ADCC86EC59; Thu, 4 Feb 2021 16:28:45 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ADCC86EC59 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1612445325; bh=Zh/wqBv2b3RHeEy/2CtShRNkHNWn9+cyygXatTSWZUg=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=d35rgSTd0zHz0fIUviUU5MpbThmFFnFolOuM2FOdTQ3hdXwmi9iBh3pyR8UubYzUA kSR0Aw8wl0uVB6kHC7anjW0OvHKTOKOGkoDg6iWHwM3fvin9CcwmO1VUpRTHSg8qGv 08/0uMHVHMIrcwDV/2Z9aLBFF6PiqPpKjsAmtac4= Received: from smtp54.i.mail.ru (smtp54.i.mail.ru [217.69.128.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id DCBA56EC59 for ; Thu, 4 Feb 2021 16:28:44 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org DCBA56EC59 Received: by smtp54.i.mail.ru with esmtpa (envelope-from ) id 1l7ega-0006tk-1n; Thu, 04 Feb 2021 16:28:40 +0300 To: Cyrill Gorcunov , tml References: <20210201100037.212301-1-gorcunov@gmail.com> <20210201100037.212301-3-gorcunov@gmail.com> Message-ID: <969899ec-6200-943d-524c-a4dd02bcb601@tarantool.org> Date: Thu, 4 Feb 2021 16:28:39 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 MIME-Version: 1.0 In-Reply-To: <20210201100037.212301-3-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD953AC099BC0052A9CC9F923E808BA4A11AAD7C69F0036037C182A05F5380850401E74513B26233399F9D2A29541D82DAFDD7BBF8F829968E5F64CFFC90E82D007 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7952C4D7BD0BF3359EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637911538129B0A8D078638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC0405E24B0EFD8036B1CC980799030181A57555B83CA50DF5389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0731955DFF79023228941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6FE5EE7C534F234F5CC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC3BBA039523A4428275ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4930A3850AC1BE2E735CC4B623DB76FBBCBC4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0F05F538519369F3743B503F486389A921A5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A5C0E8319BAAC6AFBED64043F7272E535730C7E4FA827EDCD3D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3475FE4AA98865E23568660593CA70BE06445841D7DF1F42C0EAD8E25C56AC47F97D121C36A9D561F71D7E09C32AA3244CE1AED2241A3031DC2EE74CB6079D2A6D4DBEAD0ED6C55A80927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj9Cv0SAP7uQV/ElAdDM5Nag== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A446AE0A957ED05FF59F3DB1B162583A50F560A0FB66E1C815A5424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v2 2/3] box/info: report replication.X.downstream.lag value X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 01.02.2021 13:00, Cyrill Gorcunov пишет: > This is basically a reflection of replication.X.upstream.lag value. > The upstream lag can be considered as transaction RTT in direction > from master to replica, in turn downstream lag is the reverse and > represents RTT from replica to master. > > An example of output is (on replica node) > > | 2: > | id: 2 > | uuid: 8bb22366-cd21-492e-98df-693884be11bd > | lsn: 0 > | downstream: > | status: follow > | idle: 0.55381065199617 > | vclock: {1: 119} > | lag: 0.00019168853759766 > > In case if there some old replicas which are not sending > timestamp in vclock encoding we simply don't show lag > field for backward compatibility sake. > > Closes #5447 > > @TarantoolBot document > Title: Document box.info.replication[n].downstream.lag > > replication[n].downstream.lag is time difference between > local time of the replica `n` sending current vector clock > state and the time this message was received by a master > node. > > Signed-off-by: Cyrill Gorcunov > --- Hi! Thanks for the patch! Generally looks good, however, I have a comment. Actually, what you're counting here is not related to upstream lag. Upstream lag is the difference between master's WAL write time and replica's time at the moment of receiving the entry. So it may indicate how much the replica has fallen behind master. This is a meaningful value, which may be quite high when the replica is syncing with master. Downstream lag, as you count it, is simply the time it took a network packet to travel from replica to master. It doesn't show anything about replica's status as compared to master. So there's not much value in it. Sorry for noticing this so late. I had some magic representation of how it should work in my head and didn't look into the patch thoroughly enough. This is not what the ticket was about. I see two options here: 1) you may simply send applier lag in row->tm, and save it in relay->lag.    This way downstream.lag on master will be the same as upstream.lag on replica,    so it'll be just a helper, but still better than what we have now. 2) We need to measure difference between transaction WAL write time and the    time we received an ACK for this transaction. This may be too complicated for    async transactions, since you need to remember times for a bunch of already    committed transactions. So this will make sense only for synchronous transactions    and qsync statistics. I personally prefer option 1 now, as a counterpart of upstream.lag on master, and option 2 as a part of qsync statistics ticket. But maybe we don't need (2) at all. > src/box/lua/info.c | 8 ++++++++ > src/box/relay.cc | 16 ++++++++++++++++ > src/box/relay.h | 8 ++++++++ > 3 files changed, 32 insertions(+) > > diff --git a/src/box/lua/info.c b/src/box/lua/info.c > index c4c9fa0a0..1e533fe8d 100644 > --- a/src/box/lua/info.c > +++ b/src/box/lua/info.c > @@ -133,16 +133,24 @@ lbox_pushrelay(lua_State *L, struct relay *relay) > > switch(relay_get_state(relay)) { > case RELAY_FOLLOW: > + { > + double lag = relay_lag(relay); > lua_pushstring(L, "follow"); > lua_settable(L, -3); > lua_pushstring(L, "vclock"); > lbox_pushvclock(L, relay_vclock(relay)); > lua_settable(L, -3); > + if (lag != 0) { > + lua_pushstring(L, "lag"); > + lua_pushnumber(L, relay_lag(relay)); > + lua_settable(L, -3); > + } > lua_pushstring(L, "idle"); > lua_pushnumber(L, ev_monotonic_now(loop()) - > relay_last_row_time(relay)); > lua_settable(L, -3); > break; > + } > case RELAY_STOPPED: > { > lua_pushstring(L, "stopped"); > diff --git a/src/box/relay.cc b/src/box/relay.cc > index df04f8198..f23e43b30 100644 > --- a/src/box/relay.cc > +++ b/src/box/relay.cc > @@ -138,6 +138,8 @@ struct relay { > struct stailq pending_gc; > /** Time when last row was sent to peer. */ > double last_row_time; > + /** Number of seconds this master is prior the remote replica. */ > + double lag; > /** Relay sync state. */ > enum relay_state state; > > @@ -179,6 +181,12 @@ relay_last_row_time(const struct relay *relay) > return relay->last_row_time; > } > > +double > +relay_lag(const struct relay *relay) > +{ > + return relay->lag; > +} > + > static void > relay_send(struct relay *relay, struct xrow_header *packet); > static void > @@ -219,6 +227,7 @@ relay_start(struct relay *relay, int fd, uint64_t sync, > relay->sync = sync; > relay->state = RELAY_FOLLOW; > relay->last_row_time = ev_monotonic_now(loop()); > + relay->lag = 0; > } > > void > @@ -558,6 +567,13 @@ relay_reader_f(va_list ap) > /* vclock is followed while decoding, zeroing it. */ > vclock_create(&relay->recv_vclock); > xrow_decode_vclock_xc(&xrow, &relay->recv_vclock); > + /* > + * Old versions may send not a timestamp but > + * zeroified memory field. We can use +0 as > + * a sign that there is nothing encoded. > + */ > + if (xrow.tm != 0) > + relay->lag = ev_now(loop()) - xrow.tm; > fiber_cond_signal(&relay->reader_cond); > } > } catch (Exception *e) { > diff --git a/src/box/relay.h b/src/box/relay.h > index b32e2ea2a..ec9d16925 100644 > --- a/src/box/relay.h > +++ b/src/box/relay.h > @@ -93,6 +93,14 @@ relay_vclock(const struct relay *relay); > double > relay_last_row_time(const struct relay *relay); > > +/** > + * Returns relay's lag > + * @param relay relay > + * @returns relay's lag > + */ > +double > +relay_lag(const struct relay *relay); > + > /** > * Send a Raft update request to the relay channel. It is not > * guaranteed that it will be delivered. The connection may break. -- Serge Petrenko