[PATCH 5/8] vinyl: use separate thread pools for dump and compaction tasks

[Vladimir Davydov]

On Thu, Sep 06, 2018 at 10:37:51AM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev at gmail.com> [18/09/04 20:59]:
> > Using the same thread pool for both dump and compaction tasks makes
> > estimation of dump bandwidth unstable. For instance, if we have four
> > worker threads, then the observed dump bandwidth may vary from X if
> > there's high compaction demand and all worker threads tend to be busy
> > with compaction tasks to 4 * X if there's no compaction demand. As a
> > result, we can overestimate the dump bandwidth and trigger dump when
> > it's too late, which will result in hitting the limit before dump is
> > complete and hence stalling write transactions, which is unacceptable.
> > 
> > To avoid that, let's separate thread pools used for dump and compaction
> > tasks. Since LSM tree based design typically implies high levels of
> > write amplification, let's allocate 1/4th of all threads for dump tasks
> > and use the rest exclusively for compaction.
> 
> Please follow this up by changing the default number of write
> threads to 4 and making 4 the lower bound for this configuration
> setting.

I'm OK with setting vinyl_write_threads to 4 by default, but I don't
think it's a good idea to increase the lower bound from 2 to 4. First,
disabling parallel compaction may be useful for performance experiments.
Second, it's going to break our customers who use default settings.