From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id C7A3B6EC5F; Fri, 30 Apr 2021 23:50:38 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org C7A3B6EC5F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1619815838; bh=cuY/8EFOGiJPLtuEG1MHnOChqSXgegDc7FBZfvhXYLg=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=pL7CxJNp95JewvNN17bsYBWgMD42UUSn7ac6bthCfGIuLWTYHZPfcNjWEz/TqhCLT ZQ3JlLBmknC8mIGSGOVNf0mWuQCwB7cSuOzd0HWspkLWrxiztLeDkGx04HClt4K/s8 CfXuJb4zF5OHum4oG+9nXdwfPSWN++bh+puQV060= Received: from smtp3.mail.ru (smtp3.mail.ru [94.100.179.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id BE0EC6EC5F for ; Fri, 30 Apr 2021 23:50:31 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BE0EC6EC5F Received: by smtp3.mail.ru with esmtpa (envelope-from ) id 1lca5n-0007j1-1K; Fri, 30 Apr 2021 23:50:31 +0300 To: Cyrill Gorcunov , tml Cc: Mons Anderson References: <20210430153940.121271-1-gorcunov@gmail.com> <20210430153940.121271-4-gorcunov@gmail.com> Message-ID: <24ead094-55e5-322a-3ab6-3229252b5e5a@tarantool.org> Date: Fri, 30 Apr 2021 22:50:29 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: <20210430153940.121271-4-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD95978C26455E69BE0B2C7F7C3B0039F139F0EF8E3717E163C182A05F538085040674AFB11F2193AD98B5BFAF50751FA381F1FA2C2FE7B2DAC478A11961B14588A X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE704FFE27C31EF363AEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AB1265E79AFCDEF58638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B27856041CF1D307EB6ED0A4DC612BC86B088CACE0DB6A842FD2E47CDBA5A96583C09775C1D3CA48CFE97D2AE7161E217F117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE749E2213E709ACCBA9FA2833FD35BB23DF004C90652538430302FCEF25BFAB3454AD6D5ED66289B5278DA827A17800CE7E655516FB2341D73D32BA5DBAC0009BE395957E7521B51C20BC6067A898B09E4090A508E0FED6299176DF2183F8FC7C0F0EB20F09E7EDBCCCD04E86FAF290E2D7E9C4E3C761E06A71DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B65D56369A3576CBA5089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2368A440D3B0F6089093C9A16E5BC824A2A04A2ABAA09D25379311020FFC8D4ADAC16E8A1BF356EE9A013A0EDDC8222DA X-C1DE0DAB: 0D63561A33F958A54E82EDF68218E6565DBBC83ABD81BDACE5902F7E6BD564ACD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D347798CB24D89904730900A005E749146093B323A184ABFDAB6610331114DF1E985EB49C90E6E58B0F1D7E09C32AA3244CB5BF2922243AF59907EFE23D0D6CDEE955E75C8D0ED9F6EEFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj9hvEon6tehPjKTCPQnXCxQ== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA31108BF547FDE9EE08A2F19370DB41AE16ED08D94F51DA2DA45907784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [RFC v3 3/3] relay: provide information about downstream lag X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" I appreciate the work you did here! See 4 comments below. On 30.04.2021 17:39, Cyrill Gorcunov via Tarantool-patches wrote: > We already have `box.replication.upstream.lag` entry for monitoring > sake. Same time in synchronous replication timeouts are key properties > for quorum gathering procedure. Thus we would like to know how long > it took of a transaction to traverse `initiator WAL -> network -> > remote applier -> ACK` path. > > In this patch we use new applier's functionality which returns us > the timestamp of first written xrow in a transaction such that we > can calculate the downstream lag. > > Closes #5447 1. There must be a docbot request and a changelog file. > diff --git a/src/box/relay.cc b/src/box/relay.cc > index ff43c2fc7..6d880932a 100644 > --- a/src/box/relay.cc > +++ b/src/box/relay.cc > @@ -158,6 +158,12 @@ struct relay { > struct stailq pending_gc; > /** Time when last row was sent to peer. */ > double last_row_time; > + /** > + * A time difference between moment when we > + * wrote the row in a local WAL and peer wrote > + * it in own WAL. > + */ > + double peer_wal_lag; 2. It is not true really. It also includes 2 network hops. I wouldn't call it 'wal'. It is simply lag. Or latency. > @@ -614,15 +626,41 @@ relay_reader_f(va_list ap) > coio_create(&io, relay->io.fd); > ibuf_create(&ibuf, &cord()->slabc, 1024); > try { > - while (!fiber_is_cancelled()) { > - struct xrow_header xrow; > + struct xrow_header xrow; > + double prev_tm; > + > + /* > + * Make a first read as a separate action because > + * we need previous timestamp from the xrow to > + * calculate delta from (to eliminate branching > + * in next reads). > + */ > + if (!fiber_is_cancelled()) { > coio_read_xrow_timeout_xc(&io, &ibuf, &xrow, > - replication_disconnect_timeout()); > + replication_disconnect_timeout()); 3. You didn't have to change this line. > + prev_tm = xrow.tm; > + } > + > + do { > /* vclock is followed while decoding, zeroing it. */ > vclock_create(&relay->recv_vclock); > xrow_decode_vclock_xc(&xrow, &relay->recv_vclock); > + /* > + * Old instances do not report the timestamp. > + * Same time in case of idle cycles the xrow.tm > + * is the same so update lag only when new data > + * been acked. > + */ > + if (prev_tm != xrow.tm) { 4. It also could be reset to zero when there are no rows and send/received vclock match. Because it means there is no rows to ack, and therefore can't be any latency. Or you can use relay heartbeats to update it even when there are no rows. > + double delta = ev_now(loop()) - xrow.tm; > + relay->peer_wal_lag = delta; > + prev_tm = xrow.tm; > + }