From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Fri, 24 Aug 2018 11:21:54 +0300 From: Vladimir Davydov Subject: Re: [PATCH 18/18] vinyl: throttle tx rate if dump does not catch up Message-ID: <20180824082154.lq6ysmwizgyw2tfb@esperanza> References: <20180823205443.GF24711@chai> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180823205443.GF24711@chai> To: Konstantin Osipov Cc: tarantool-patches@freelists.org List-ID: On Thu, Aug 23, 2018 at 11:54:43PM +0300, Konstantin Osipov wrote: > * Vladimir Davydov [18/08/16 23:03]: > > If the rate at which transactions are ready to write to the database is > > greater than the dump bandwidth, memory will get depleted before the > > last dump is complete and all newer transactions will have to wait until > > the dump has been completed, which may take seconds or even minutes: > > Why are you limiting the rate only during a dump? What about > compaction throughtput? Write amplification from compaction is > 10x+ times higher than from dump, compaction is actually the main > consumer of disk bandwidth. Shouldn't you throttle writes when > write bandwidth is higher than write amplification/disk bandwidth? We use the 10th percentile of all dump/compaction write rate observations when estimating the dump bandwidth. That is, if a dump task is ever run in parallel with other dump tasks or compaction, we will take this into account. This might be not perfect, but it works well in practice. Anyways, we can tune that later if we find such an algorithm unsatisfactory.