[tarantool-patches] Re: [PATCH 7/9] vinyl: randomize range compaction to avoid IO load spikes

[Vladimir Davydov]

On Wed, Feb 06, 2019 at 02:06:09PM +0300, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev at gmail.com> [19/02/06 13:57]:
> > On Wed, Feb 06, 2019 at 01:44:19PM +0300, Konstantin Osipov wrote:
> > > * Vladimir Davydov <vdavydov.dev at gmail.com> [19/02/06 13:31]:
> > > > Over how many dumps? What do we do after restart, when there's no
> > > > history and perhaps even no level 1?
> > > 
> > > A am thinking about something along these lines: 
> > > f(n+1) = (f(n) + x*k)(1+k) - where k is the weight used to scale
> > > the next input
> > 
> > What should be k equal to?
> 
> I don't know.

Neither do I. That's why I don't want to involve any kind of averaging.

> 
> > Also, what we should do after restart when there's no history?
> 
> Seed the function with the top run size.

> If there is no top run, seed with 0.

This way the moving average will grow very reluctantly - we'll need more
than k dumps to adapt. During that time, compaction priority calculation
will be unstable.

> > 
> > Why is using the last level size as reference bad?
> 
> Because you don't know if it's last or not?

We know which run is last. Provided the workload is stable, i.e. have
stopped growing its dataset, it will be roughly the same. Besides, the
last level run size changes only on major compaction, which is
infrequent.  After a major compaction, it's OK to use a different first
level size - the point is in order not to break LSM algorithm, we have
to maintain stable level sizing between major compactions.