From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 6 Sep 2018 12:48:33 +0300 From: Vladimir Davydov Subject: Re: [PATCH 5/8] vinyl: use separate thread pools for dump and compaction tasks Message-ID: <20180906094833.xzjy6mushae5b2lv@esperanza> References: <20180906073751.GE8205@chai> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180906073751.GE8205@chai> To: Konstantin Osipov Cc: tarantool-patches@freelists.org List-ID: On Thu, Sep 06, 2018 at 10:37:51AM +0300, Konstantin Osipov wrote: > * Vladimir Davydov [18/09/04 20:59]: > > Using the same thread pool for both dump and compaction tasks makes > > estimation of dump bandwidth unstable. For instance, if we have four > > worker threads, then the observed dump bandwidth may vary from X if > > there's high compaction demand and all worker threads tend to be busy > > with compaction tasks to 4 * X if there's no compaction demand. As a > > result, we can overestimate the dump bandwidth and trigger dump when > > it's too late, which will result in hitting the limit before dump is > > complete and hence stalling write transactions, which is unacceptable. > > > > To avoid that, let's separate thread pools used for dump and compaction > > tasks. Since LSM tree based design typically implies high levels of > > write amplification, let's allocate 1/4th of all threads for dump tasks > > and use the rest exclusively for compaction. > > Please follow this up by changing the default number of write > threads to 4 and making 4 the lower bound for this configuration > setting. I'm OK with setting vinyl_write_threads to 4 by default, but I don't think it's a good idea to increase the lower bound from 2 to 4. First, disabling parallel compaction may be useful for performance experiments. Second, it's going to break our customers who use default settings.