From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id E59966EC5B; Wed, 12 May 2021 23:10:50 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E59966EC5B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1620850250; bh=x2j7EdAoXniO5dnxPMf1gIbCC//XHoCymTOV9umkctI=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=bFl53rs0bbWwEhT2IaIVOZXaItsjD2A4I7rNlG98lfrD0ckTnt0wbtvxBOHdfHqeI ijcdBeFvC0ncw541NInWidRHKQJuaBv7THIfyW7LeRYm+6BGBPtjW1Y9DauhzQWCbk Bp65mzkJxtXg5CjKZpGLGpIgO323R1E0CQJDxUEY= Received: from smtp55.i.mail.ru (smtp55.i.mail.ru [217.69.128.35]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id D2E0C6EC5B for ; Wed, 12 May 2021 23:10:49 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D2E0C6EC5B Received: by smtp55.i.mail.ru with esmtpa (envelope-from ) id 1lgvBw-0004Hb-HC; Wed, 12 May 2021 23:10:49 +0300 To: Cyrill Gorcunov , tml References: <20210506214132.533913-1-gorcunov@gmail.com> <20210506214132.533913-3-gorcunov@gmail.com> Message-ID: <1aa8f87d-bba0-2967-352b-b751837ea2cd@tarantool.org> Date: Wed, 12 May 2021 22:10:47 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: <20210506214132.533913-3-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD95978C26455E69BE0068EDCB750753685DC8F791B892E4B69182A05F538085040B42698EA02E42BEB061A38D2D8E22F25548FF20461152392A35F38740F6C6908 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7C8140302C704C25FEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063776672C316918EFDB8638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2EA709138EBADAD6FAA4D03970507C8510A05E728617FD226D2E47CDBA5A96583C09775C1D3CA48CFE97D2AE7161E217F117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE7820CF4CC0E318EFB9FA2833FD35BB23DF004C90652538430302FCEF25BFAB3454AD6D5ED66289B5278DA827A17800CE7668EB61EDC488D4AD32BA5DBAC0009BE395957E7521B51C20BC6067A898B09E4090A508E0FED6299176DF2183F8FC7C08016716D3DAB41CACD04E86FAF290E2D7E9C4E3C761E06A71DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6300D3B61E77C8D3B089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24209795067102C07E8F7B195E1C97831A705B04746A55B2CAC1B55117025DB7A X-C1DE0DAB: 0D63561A33F958A576871EE71F9166743DE6AFB26D1E9D886634DF165573EFD8D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34682FD2D2951524C52E387C3D6F6774331DD827B77E61F562286E1CD3A8A69F434C3658C4332631991D7E09C32AA3244CBCF7778FA853647314C4897A6673E28395A9E0DC41E9A4CF927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojo6y/qPNd2uyr0d0k9d+9zQ== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA31107187738568CAA60FCCD0991FC99602CD1424BECBD1EB720107784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v4 2/2] relay: provide information about downstream lag X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" I appreciate the work you did here! See 6 comments below. On 06.05.2021 23:41, Cyrill Gorcunov wrote: > We already have `box.replication.upstream.lag` entry for monitoring > sake. Same time in synchronous replication timeouts are key properties > for quorum gathering procedure. Thus we would like to know how long > it took of a transaction to traverse `initiator WAL -> network -> > remote applier -> ACK` path. > > Typical ouput is 1. ouput -> output. > .../unreleased/gh-5447-downstream-lag.md | 6 ++ > src/box/lua/info.c | 3 + > src/box/relay.cc | 18 ++++ > src/box/relay.h | 6 ++ > .../replication/gh-5447-downstream-lag.result | 93 +++++++++++++++++++ > .../gh-5447-downstream-lag.test.lua | 41 ++++++++ > 6 files changed, 167 insertions(+) > create mode 100644 changelogs/unreleased/gh-5447-downstream-lag.md > create mode 100644 test/replication/gh-5447-downstream-lag.result > create mode 100644 test/replication/gh-5447-downstream-lag.test.lua > > diff --git a/changelogs/unreleased/gh-5447-downstream-lag.md b/changelogs/unreleased/gh-5447-downstream-lag.md > new file mode 100644 > index 000000000..3b11c50ee > --- /dev/null > +++ b/changelogs/unreleased/gh-5447-downstream-lag.md > @@ -0,0 +1,6 @@ > +#feature/replication > + > + * Introduce `box.info.replication[n].downstream.lag` to monitor state 2. If was recently decided we must use Past Simple in changelogs. > + of replication which is especially important for synchronous spaces > + where malfunctioning replicas may prevent quorum from gathering votes > + to commit a transaction. 3. Please, add a reference to the ticket in the form `(gh-####)`. > diff --git a/src/box/lua/info.c b/src/box/lua/info.c > index 0eb48b823..5b7c25f28 100644 > --- a/src/box/lua/info.c > +++ b/src/box/lua/info.c > @@ -143,6 +143,9 @@ lbox_pushrelay(lua_State *L, struct relay *relay) > lua_pushnumber(L, ev_monotonic_now(loop()) - > relay_last_row_time(relay)); > lua_settable(L, -3); > + lua_pushstring(L, "lag"); > + lua_pushnumber(L, relay_lag(relay)); > + lua_settable(L, -3); > break; > case RELAY_STOPPED: > { > diff --git a/src/box/relay.cc b/src/box/relay.cc > index ff43c2fc7..1501e53c7 100644 > --- a/src/box/relay.cc > +++ b/src/box/relay.cc > @@ -158,6 +158,12 @@ struct relay { > struct stailq pending_gc; > /** Time when last row was sent to peer. */ > double last_row_time; > + /** > + * A time difference between moment when we > + * wrote a row in the local WAL and received > + * an ACK that peer has it replicated. > + */ > + double lag; 4. Lag is updated in the relay thread, therefore you can't simply read it in TX thread like you do in the diff block above. Try to deliver it to TX the same way as acked vclock is delivered. > /** Relay sync state. */ > enum relay_state state; > > @@ -621,6 +633,12 @@ relay_reader_f(va_list ap) > /* vclock is followed while decoding, zeroing it. */ > vclock_create(&relay->recv_vclock); > xrow_decode_vclock_xc(&xrow, &relay->recv_vclock); > + if (xrow.tm != 0) { > + double delta = ev_now(loop()) - xrow.tm; 5. What if the acked row didn't come from the remote peer? But from a third node. Node1 -> Node2 -> Node3. Does Node2, before sending the row to node3, update its timestamp? Otherwise you are going to send an ACK to Node2 with timestamp of Node1, which won't make much sense. I don't know. I see xlog_write_entry() updates the timestamp, but please, double-check it. In the debugger or via printfs. > + relay->lag = delta; > + } else { > + relay->lag = 0; 6. Hm. This means that if I sent a synchronous transaction, and its quorum collection takes longer than the replication timeout, I would see the lag 0. Moverover, it will constantly blink if I have many synchronous transactions working slow. Between 0 and seconds. I am not 100% it looks right. Especially it won't be applicable for any monitoring, the results won't be readable. I would rather expect the lag grow until I receive an ACK, which will bump it to an actual value. And drop to 0 only when there is no rows to wait ACK for. Or maybe never drop to 0 like applier does? Drop to 0 when no rows means we will have it blinking between 0 and seconds again if the synchronous transactions are long, but not so frequent. So there is often no rows to ACK. Did you try to ask Vlad G., who is going to actually use it probably? I tried to list the options we have https://github.com/tarantool/tarantool/issues/5447#issuecomment-840047693 and asked Vlad what he thinks. He also promised to ask more people who use Tarantool actively.