From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: [PATCH 00/18] Implement write throttling for vinyl Date: Thu, 16 Aug 2018 19:11:54 +0300 Message-Id: To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: If vinyl doesn't keep up with dumps, transactions can consume all available memory and stall until the last dump is complete, which may take quite a while (seconds, even minutes sometimes). To avoid unpredictably long stalls, this patch set introduces transaction throttling. Now, once a dump is started, vinyl will limit the rate at which transactions can write to the database so that they won't hit the limit before the dump is complete. https://github.com/tarantool/tarantool/issues/1862 https://github.com/tarantool/tarantool/commits/dv/gh-1862-vy-throttling Vladimir Davydov (18): vinyl: rework internal quota API vinyl: move quota methods implementation to vy_quota.c vinyl: move quota related methods and variables from vy_env to vy_quota vinyl: implement vy_quota_wait using vy_quota_try_use vinyl: wake up fibers waiting for quota one by one vinyl: do not wake up fibers waiting for quota if quota is unavailable vinyl: tune dump bandwidth histogram buckets vinyl: rename vy_quota::dump_bw to dump_bw_hist vinyl: cache dump bandwidth for timer invocation vinyl: do not add initial guess to dump bandwidth histogram vinyl: use snap_io_rate_limit for initial dump bandwidth estimate histogram: add function for computing lower bound percentile estimate vinyl: use lower bound percentile estimate for dump bandwidth vinyl: do not try to trigger dump if it is already in progress vinyl: improve dump start/stop logging vinyl: confine quota watermark within sane value range vinyl: set quota timer period to 100 ms vinyl: throttle tx rate if dump does not catch up src/box/CMakeLists.txt | 1 + src/box/vinyl.c | 180 +++------------------ src/box/vy_quota.c | 362 +++++++++++++++++++++++++++++++++++++++++++ src/box/vy_quota.h | 193 +++++++++++++---------- src/histogram.c | 13 ++ src/histogram.h | 7 + test/unit/histogram.c | 4 + test/vinyl/suite.ini | 2 +- test/vinyl/throttle.result | 95 ++++++++++++ test/vinyl/throttle.test.lua | 47 ++++++ 10 files changed, 664 insertions(+), 240 deletions(-) create mode 100644 src/box/vy_quota.c create mode 100644 test/vinyl/throttle.result create mode 100644 test/vinyl/throttle.test.lua -- 2.11.0