[PATCH 18/18] vinyl: throttle tx rate if dump does not catch up

[Vladimir Davydov]

On Thu, Aug 23, 2018 at 11:54:43PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev at gmail.com> [18/08/16 23:03]:
> > If the rate at which transactions are ready to write to the database is
> > greater than the dump bandwidth, memory will get depleted before the
> > last dump is complete and all newer transactions will have to wait until
> > the dump has been completed, which may take seconds or even minutes:
> 
> Why are you limiting the rate only during a dump? What about
> compaction throughtput? Write amplification from compaction is
> 10x+ times higher than from dump, compaction is actually the main
> consumer of disk bandwidth. Shouldn't you throttle writes when
> write bandwidth is higher than write amplification/disk bandwidth?

We use the 10th percentile of all dump/compaction write rate
observations when estimating the dump bandwidth. That is, if
a dump task is ever run in parallel with other dump tasks or
compaction, we will take this into account. This might be not
perfect, but it works well in practice. Anyways, we can tune
that later if we find such an algorithm unsatisfactory.