Tarantool development patches archive
 help / color / mirror / Atom feed
From: Cyrill Gorcunov via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: tml <tarantool-patches@dev.tarantool.org>
Cc: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>
Subject: [Tarantool-patches] [PATCH v8 0/2] relay: provide downstream lag information
Date: Mon,  7 Jun 2021 18:55:17 +0300
Message-ID: <20210607155519.109626-1-gorcunov@gmail.com> (raw)

Guys, take a look once time permit. Previous version is here

https://lists.tarantool.org/tarantool-patches/20210506214132.533913-1-gorcunov@gmail.com/

I still think we might need some general statistics engine because this
lag rather common data and we could move other "last" things like last_row_time
there but since I didn't get approve on the idea I keep the timestamp
in replica structure.

v4 (by Vlad):
 - add a test case
 - add docbot request
 - dropped off xrow_encode_vclock_timed, we use opencoded assignment
   for tm value when send ack
 - struct awstat renamed to applier_wal_stat. Vlad I think this is
   better name than "applier_lag" because this is statistics on WAL,
   we simply track remote WAL propagation here, so more general name
   is better for grep sake and for future extensions
 - instead of passing applier structure we pass replica_id
 - the real keeper of this statistics comes into "replica" structure
   thus unbound of applier itself
 - for synchro entries we pass a pointer to the applier_wal_stat instead
   of using replica_id = 0 as a sign that we don't need to update statistics
   for initial and final join cases
 - to write and read statistics we provide wal_stat_update and wal_stat_ack
   helpers to cover the case where single ACK spans several transactions

v8:
 - make one branch less in apply_synchro_row()
 - keep applier_txn_start_tm inside replica stucture
 - rename wal_stat to replica_cb_data since this is more
   logical for case where we have no general stat engine
 - make applier to send timestamp so that relay will compute
   delte upon the read, the lag is kept permanently until new
   write happens
 - extend doc and changelog a bit
 - keep reading of relay's lag from TX thread without any modifications
   because relay get deleted from TX thread and set to non-RELAY_FOLLOW
   state, thus any attempt to read it won't success. To be honest there
   is a small race window present: write doubles are not atomic operation
   thus we might read partially updated timestamp simiraly as we have with
   @idle field already. I think this should be addressed separately and better
   without heavy cmsg engine involved but with rw lock instead or plain atomics.

branch gorcunov/gh-5447-relay-lag-8
issue https://github.com/tarantool/tarantool/issues/5447

Cyrill Gorcunov (2):
  applier: send transaction's first row WAL time in the applier_writer_f
  relay: provide information about downstream lag

 .../unreleased/gh-5447-downstream-lag.md      |  6 ++
 src/box/applier.cc                            | 90 +++++++++++++++---
 src/box/lua/info.c                            |  3 +
 src/box/relay.cc                              | 50 ++++++++++
 src/box/relay.h                               |  6 ++
 src/box/replication.cc                        |  1 +
 src/box/replication.h                         |  5 +
 .../replication/gh-5447-downstream-lag.result | 93 +++++++++++++++++++
 .../gh-5447-downstream-lag.test.lua           | 41 ++++++++
 9 files changed, 280 insertions(+), 15 deletions(-)
 create mode 100644 changelogs/unreleased/gh-5447-downstream-lag.md
 create mode 100644 test/replication/gh-5447-downstream-lag.result
 create mode 100644 test/replication/gh-5447-downstream-lag.test.lua


base-commit: 2b25ca6c70908b57c011ee24d2eeb9f6155d2750
-- 
2.31.1


             reply	other threads:[~2021-06-07 15:55 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-07 15:55 Cyrill Gorcunov via Tarantool-patches [this message]
2021-06-07 15:55 ` [Tarantool-patches] [PATCH v8 1/2] applier: send transaction's first row WAL time in the applier_writer_f Cyrill Gorcunov via Tarantool-patches
2021-06-07 19:20   ` Vladislav Shpilevoy via Tarantool-patches
2021-06-15  9:36   ` Serge Petrenko via Tarantool-patches
2021-06-16 13:32     ` Cyrill Gorcunov via Tarantool-patches
2021-06-17  9:16       ` Serge Petrenko via Tarantool-patches
2021-06-07 15:55 ` [Tarantool-patches] [PATCH v8 2/2] relay: provide information about downstream lag Cyrill Gorcunov via Tarantool-patches
2021-06-07 19:21   ` Vladislav Shpilevoy via Tarantool-patches
2021-06-08  8:40     ` Cyrill Gorcunov via Tarantool-patches
2021-06-08  9:58       ` Cyrill Gorcunov via Tarantool-patches
2021-06-08 18:15       ` Vladislav Shpilevoy via Tarantool-patches
2021-06-15 10:03   ` Serge Petrenko via Tarantool-patches
2021-06-17  6:55     ` Cyrill Gorcunov via Tarantool-patches
2021-06-17  9:01       ` Serge Petrenko via Tarantool-patches
2021-06-17  9:58         ` Cyrill Gorcunov via Tarantool-patches
2021-06-07 19:20 ` [Tarantool-patches] [PATCH v8 0/2] relay: provide downstream lag information Vladislav Shpilevoy via Tarantool-patches
2021-06-07 20:00   ` Cyrill Gorcunov via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210607155519.109626-1-gorcunov@gmail.com \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=gorcunov@gmail.com \
    --cc=v.shpilevoy@tarantool.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Tarantool development patches archive

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://lists.tarantool.org/tarantool-patches/0 tarantool-patches/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tarantool-patches tarantool-patches/ https://lists.tarantool.org/tarantool-patches \
		tarantool-patches@dev.tarantool.org.
	public-inbox-index tarantool-patches

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git