From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Vladimir Davydov Subject: Re: [PATCH 7/7] vinyl: keep track of disk idle time Date: Tue, 4 Sep 2018 20:23:43 +0300 Message-Id: In-Reply-To: <20180904115404.el6kdswgitsnopgf@esperanza> References: <20180904115404.el6kdswgitsnopgf@esperanza> To: kostja@tarantool.org Cc: tarantool-patches@freelists.org List-ID: On Tue, Sep 04, 2018 at 02:54:04PM +0300, Vladimir Davydov wrote: > Please ignore this patch, because it's incorrect: since we always > keep one worker reserved for dumps it will should that the disk is > 1/box.cfg.vinyl_writers idle even if compaction never stops, which > is misleading. I'll rework and resend it separately. Here goes the new version. The primary difference is that it reports idle time separately for dump and compaction threads. To achieve that, I had to introduce separate thread pools for dump and compaction tasks. There's still one knob though, box.cfg.vinyl_write_threads I assign 25% of threads (min 1) to dump tasks while the rest goes to compaction tasks (this is similar to what RocksDB does). Patches 1 and 2 do some trivial cleanup. Patches 3 to 7 introduce separate thread pools for different kinds of background tasks. Patch 8 adds idle ratio to global statistics. The branch is the same: https://github.com/tarantool/tarantool/commits/dv/vy-stats-for-throttling Vladimir Davydov (8): vinyl: add helper to check whether dump is in progress vinyl: don't use mempool for allocating background tasks vinyl: factor out worker pool from scheduler struct vinyl: move worker allocation closer to task creation vinyl: use separate thread pools for dump and compaction tasks vinyl: zap vy_worker_pool::idle_worker_count vinyl: don't start scheduler fiber until local recovery is complete vinyl: keep track of thread pool idle ratio src/box/vinyl.c | 44 +++--- src/box/vy_scheduler.c | 334 ++++++++++++++++++++++++++++++--------------- src/box/vy_scheduler.h | 60 ++++++-- test/vinyl/errinj.result | 82 +++++++++++ test/vinyl/errinj.test.lua | 37 +++++ test/vinyl/info.result | 14 +- test/vinyl/info.test.lua | 2 + 7 files changed, 423 insertions(+), 150 deletions(-) -- 2.11.0