Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladimir Davydov <vdavydov.dev@gmail.com>
To: kostja@tarantool.org
Cc: tarantool-patches@freelists.org
Subject: [PATCH 15/18] vinyl: improve dump start/stop logging
Date: Thu, 16 Aug 2018 19:12:09 +0300	[thread overview]
Message-ID: <8637c94dd4a9e8550b0357e22c0fa3b1f33494cb.1534432819.git.vdavydov.dev@gmail.com> (raw)
In-Reply-To: <cover.1534432819.git.vdavydov.dev@gmail.com>
In-Reply-To: <cover.1534432819.git.vdavydov.dev@gmail.com>

When initiating memory dump, print how much memory is going to be
dumped, expected dump rate, ETA, and the recent write rate. Upon dump
completion, print observed dump rate in addition to dump size and
duration.

Example:

  2018-08-16 14:11:40.802 [19044] main/2096/main I> dumping 134217746 bytes, expected rate 4.0 MB/s, ETA 32.0 s, recent write rate 16.6 MB/s
  2018-08-16 14:11:40.843 [19044] main/103/vinyl.scheduler I> 513/0: dump started
  2018-08-16 14:11:40.844 [19044] vinyl.writer.0/102/task I> writing `./513/0/00000000000000000002.run'
  2018-08-16 14:12:01.394 [19044] vinyl.writer.0/102/task I> writing `./513/0/00000000000000000002.index'
  2018-08-16 14:12:01.467 [19044] main/103/vinyl.scheduler I> 513/0: dump completed
  2018-08-16 14:12:01.467 [19044] main/103/vinyl.scheduler I> dumped 134214971 bytes in 20.7 s, rate 6.2 MB/s
---
 src/box/vinyl.c    |  2 --
 src/box/vy_quota.c | 14 ++++++++++++--
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/src/box/vinyl.c b/src/box/vinyl.c
index 25a937bc..ec7045b5 100644
--- a/src/box/vinyl.c
+++ b/src/box/vinyl.c
@@ -2414,8 +2414,6 @@ vy_env_dump_complete_cb(struct vy_scheduler *scheduler,
 	assert(mem_used_after <= mem_used_before);
 	size_t mem_dumped = mem_used_before - mem_used_after;
 	vy_quota_dump(quota, mem_dumped, dump_duration);
-
-	say_info("dumped %zu bytes in %.1f sec", mem_dumped, dump_duration);
 }
 
 static struct vy_squash_queue *
diff --git a/src/box/vy_quota.c b/src/box/vy_quota.c
index 0250b3ab..c22a8519 100644
--- a/src/box/vy_quota.c
+++ b/src/box/vy_quota.c
@@ -86,8 +86,15 @@ vy_quota_signal(struct vy_quota *q)
 static inline void
 vy_quota_check_watermark(struct vy_quota *q)
 {
-	if (!q->dump_in_progress && q->used >= q->watermark)
-		q->dump_in_progress = q->quota_exceeded_cb(q);
+	if (!q->dump_in_progress &&
+	    q->used >= q->watermark && q->quota_exceeded_cb(q)) {
+		q->dump_in_progress = true;
+		say_info("dumping %zu bytes, expected rate %.1f MB/s, "
+			 "ETA %.1f s, recent write rate %.1f MB/s", q->used,
+			 (double)q->dump_bw / 1024 / 1024,
+			 (double)q->used / (q->dump_bw + 1),
+			 (double)q->use_rate / 1024 / 1024);
+	}
 }
 
 static void
@@ -214,6 +221,9 @@ vy_quota_dump(struct vy_quota *q, size_t size, double duration)
 		q->dump_bw = histogram_percentile_lower(q->dump_bw_hist,
 							VY_DUMP_BANDWIDTH_PCT);
 	}
+
+	say_info("dumped %zu bytes in %.1f s, rate %.1f MB/s",
+		 size, duration, size / duration / 1024 / 1024);
 }
 
 int
-- 
2.11.0

  parent reply	other threads:[~2018-08-16 16:12 UTC|newest]

Thread overview: 56+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-16 16:11 [PATCH 00/18] Implement write throttling for vinyl Vladimir Davydov
2018-08-16 16:11 ` [PATCH 01/18] vinyl: rework internal quota API Vladimir Davydov
2018-08-20 11:07   ` Konstantin Osipov
2018-08-24  8:32     ` Vladimir Davydov
2018-08-27 18:29   ` Vladimir Davydov
2018-08-16 16:11 ` [PATCH 02/18] vinyl: move quota methods implementation to vy_quota.c Vladimir Davydov
2018-08-20 11:07   ` Konstantin Osipov
2018-08-27 18:30   ` Vladimir Davydov
2018-08-16 16:11 ` [PATCH 03/18] vinyl: move quota related methods and variables from vy_env to vy_quota Vladimir Davydov
2018-08-20 11:08   ` Konstantin Osipov
2018-08-27 18:33   ` Vladimir Davydov
2018-08-16 16:11 ` [PATCH 04/18] vinyl: implement vy_quota_wait using vy_quota_try_use Vladimir Davydov
2018-08-20 11:09   ` Konstantin Osipov
2018-08-27 18:36   ` Vladimir Davydov
2018-08-16 16:11 ` [PATCH 05/18] vinyl: wake up fibers waiting for quota one by one Vladimir Davydov
2018-08-20 11:11   ` Konstantin Osipov
2018-08-24  8:33     ` Vladimir Davydov
2018-08-28 13:19   ` Vladimir Davydov
2018-08-28 14:04     ` Konstantin Osipov
2018-08-28 14:39       ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 06/18] vinyl: do not wake up fibers waiting for quota if quota is unavailable Vladimir Davydov
2018-08-20 11:13   ` Konstantin Osipov
2018-08-16 16:12 ` [PATCH 07/18] vinyl: tune dump bandwidth histogram buckets Vladimir Davydov
2018-08-20 11:15   ` Konstantin Osipov
2018-08-28 15:37   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 08/18] vinyl: rename vy_quota::dump_bw to dump_bw_hist Vladimir Davydov
2018-08-20 11:15   ` Konstantin Osipov
2018-08-28 16:04   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 09/18] vinyl: cache dump bandwidth for timer invocation Vladimir Davydov
2018-08-20 11:21   ` Konstantin Osipov
2018-08-28 16:10   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 10/18] vinyl: do not add initial guess to dump bandwidth histogram Vladimir Davydov
2018-08-20 11:23   ` Konstantin Osipov
2018-08-23 20:15   ` Konstantin Osipov
2018-08-28 16:15   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 11/18] vinyl: use snap_io_rate_limit for initial dump bandwidth estimate Vladimir Davydov
2018-08-20 11:24   ` Konstantin Osipov
2018-08-24  8:31     ` Vladimir Davydov
2018-08-28 16:18   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 12/18] histogram: add function for computing lower bound percentile estimate Vladimir Davydov
2018-08-20 11:29   ` [tarantool-patches] " Konstantin Osipov
2018-08-24  8:30     ` Vladimir Davydov
2018-08-28 16:39   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 13/18] vinyl: use lower bound percentile estimate for dump bandwidth Vladimir Davydov
2018-08-28 16:51   ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 14/18] vinyl: do not try to trigger dump if it is already in progress Vladimir Davydov
2018-08-16 16:12 ` Vladimir Davydov [this message]
2018-08-23 20:18   ` [PATCH 15/18] vinyl: improve dump start/stop logging Konstantin Osipov
2018-08-16 16:12 ` [PATCH 16/18] vinyl: confine quota watermark within sane value range Vladimir Davydov
2018-08-16 16:12 ` [PATCH 17/18] vinyl: set quota timer period to 100 ms Vladimir Davydov
2018-08-23 20:49   ` Konstantin Osipov
2018-08-24  8:18     ` Vladimir Davydov
2018-08-16 16:12 ` [PATCH 18/18] vinyl: throttle tx rate if dump does not catch up Vladimir Davydov
2018-08-23 20:54   ` Konstantin Osipov
2018-08-23 20:58     ` [tarantool-patches] " Konstantin Osipov
2018-08-24  8:21     ` Vladimir Davydov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=8637c94dd4a9e8550b0357e22c0fa3b1f33494cb.1534432819.git.vdavydov.dev@gmail.com \
    --to=vdavydov.dev@gmail.com \
    --cc=kostja@tarantool.org \
    --cc=tarantool-patches@freelists.org \
    --subject='Re: [PATCH 15/18] vinyl: improve dump start/stop logging' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox