From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 426316EC5F; Fri, 30 Apr 2021 18:41:16 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 426316EC5F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1619797276; bh=oxZ7br94b/1Y3BLBD3/CXfhrKF/kwYlG019cO2igJzI=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=k8XsJv0aqW2aNQPhPQfRHczwliR5fp8WL7sLQh5KAnF8rYiFVeufMtM+uJYR0HYO3 tZvL+9oS6yECD1vx8wcMPuubp3gPWkV7ar3Wj91vDbPR1u/McfpXg7lcdAlWwXIZgA PjYS9A1QeU5ikFZRxggMUH+KnIw+mdDVxKbvs87U= Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id EEE0D6C7D7 for ; Fri, 30 Apr 2021 18:40:20 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org EEE0D6C7D7 Received: by mail-lj1-f175.google.com with SMTP id s9so23415254ljj.6 for ; Fri, 30 Apr 2021 08:40:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DdoOj4xfzj9ETmJnZbR3yXCOErxnUaFdmRi72VV6eWA=; b=gFMGGCl1SAkyKuLWsvqhqRbpsiYy4ToH51gGrM+K2Wq+mRNEa4LN2Awe1R5gdaeQpf wQExibOZ5yHtj30W+XUFi8KBVs7LS5Teg0WWVbZ8iWPiMia87YrAjKwW5fiL97HWHuGT WPuIiw6PWe7Dh+eVYeXSPk4Qq5HNzxJ6btG7m0h3aHYgIYjAN82L28stCO+QNmKswchI +QgyJCHG4E553hwMa7+DuW00LVaLNQAhTPNSfe53ZyBt3fjBtrKDEjyiPyDsNgY3roBU EcIgLGzeaamqCoWT8IsfeP6DfYZowccXjz4PCDMHVv7Oo3vlzx2WzYpqB8jZtMVhjRkR v4+g== X-Gm-Message-State: AOAM530BX5bdhIriP5VqCEKQzAvmKLB5TOPNenxnih4mhpthh+reoomS ZGZwcLpteem5g53lG9Je3AI= X-Google-Smtp-Source: ABdhPJwDTPdWjTKw1fYtilEvuBfKKY67ibsx3WpS/t3IseIYc0gqoDUkvCJJ0KLbQp10ET76N8o7pg== X-Received: by 2002:a2e:bf19:: with SMTP id c25mr4236805ljr.201.1619797220412; Fri, 30 Apr 2021 08:40:20 -0700 (PDT) Received: from grain.localdomain ([5.18.199.94]) by smtp.gmail.com with ESMTPSA id n15sm315768lfh.237.2021.04.30.08.40.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Apr 2021 08:40:19 -0700 (PDT) Received: by grain.localdomain (Postfix, from userid 1000) id 759B25601FC; Fri, 30 Apr 2021 18:39:41 +0300 (MSK) To: tml Date: Fri, 30 Apr 2021 18:39:40 +0300 Message-Id: <20210430153940.121271-4-gorcunov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210430153940.121271-1-gorcunov@gmail.com> References: <20210430153940.121271-1-gorcunov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [RFC v3 3/3] relay: provide information about downstream lag X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Cyrill Gorcunov via Tarantool-patches Reply-To: Cyrill Gorcunov Cc: Mons Anderson , Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" We already have `box.replication.upstream.lag` entry for monitoring sake. Same time in synchronous replication timeouts are key properties for quorum gathering procedure. Thus we would like to know how long it took of a transaction to traverse `initiator WAL -> network -> remote applier -> ACK` path. In this patch we use new applier's functionality which returns us the timestamp of first written xrow in a transaction such that we can calculate the downstream lag. Closes #5447 Signed-off-by: Cyrill Gorcunov --- src/box/lua/info.c | 3 +++ src/box/relay.cc | 46 ++++++++++++++++++++++++++++++++++++++++++---- src/box/relay.h | 3 +++ 3 files changed, 48 insertions(+), 4 deletions(-) diff --git a/src/box/lua/info.c b/src/box/lua/info.c index 0eb48b823..b72dd9915 100644 --- a/src/box/lua/info.c +++ b/src/box/lua/info.c @@ -143,6 +143,9 @@ lbox_pushrelay(lua_State *L, struct relay *relay) lua_pushnumber(L, ev_monotonic_now(loop()) - relay_last_row_time(relay)); lua_settable(L, -3); + lua_pushstring(L, "lag"); + lua_pushnumber(L, relay_peer_wal_lag(relay)); + lua_settable(L, -3); break; case RELAY_STOPPED: { diff --git a/src/box/relay.cc b/src/box/relay.cc index ff43c2fc7..6d880932a 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -158,6 +158,12 @@ struct relay { struct stailq pending_gc; /** Time when last row was sent to peer. */ double last_row_time; + /** + * A time difference between moment when we + * wrote the row in a local WAL and peer wrote + * it in own WAL. + */ + double peer_wal_lag; /** Relay sync state. */ enum relay_state state; @@ -217,6 +223,12 @@ relay_last_row_time(const struct relay *relay) return relay->last_row_time; } +double +relay_peer_wal_lag(const struct relay *relay) +{ + return relay->peer_wal_lag; +} + static void relay_send(struct relay *relay, struct xrow_header *packet); static void @@ -614,15 +626,41 @@ relay_reader_f(va_list ap) coio_create(&io, relay->io.fd); ibuf_create(&ibuf, &cord()->slabc, 1024); try { - while (!fiber_is_cancelled()) { - struct xrow_header xrow; + struct xrow_header xrow; + double prev_tm; + + /* + * Make a first read as a separate action because + * we need previous timestamp from the xrow to + * calculate delta from (to eliminate branching + * in next reads). + */ + if (!fiber_is_cancelled()) { coio_read_xrow_timeout_xc(&io, &ibuf, &xrow, - replication_disconnect_timeout()); + replication_disconnect_timeout()); + prev_tm = xrow.tm; + } + + do { /* vclock is followed while decoding, zeroing it. */ vclock_create(&relay->recv_vclock); xrow_decode_vclock_xc(&xrow, &relay->recv_vclock); + /* + * Old instances do not report the timestamp. + * Same time in case of idle cycles the xrow.tm + * is the same so update lag only when new data + * been acked. + */ + if (prev_tm != xrow.tm) { + double delta = ev_now(loop()) - xrow.tm; + relay->peer_wal_lag = delta; + prev_tm = xrow.tm; + } fiber_cond_signal(&relay->reader_cond); - } + + coio_read_xrow_timeout_xc(&io, &ibuf, &xrow, + replication_disconnect_timeout()); + } while (!fiber_is_cancelled()); } catch (Exception *e) { relay_set_error(relay, e); fiber_cancel(relay_f); diff --git a/src/box/relay.h b/src/box/relay.h index b32e2ea2a..99e2179cb 100644 --- a/src/box/relay.h +++ b/src/box/relay.h @@ -93,6 +93,9 @@ relay_vclock(const struct relay *relay); double relay_last_row_time(const struct relay *relay); +double +relay_peer_wal_lag(const struct relay *relay); + /** * Send a Raft update request to the relay channel. It is not * guaranteed that it will be delivered. The connection may break. -- 2.30.2