[Tarantool-patches] [PATCH v3] wal: introduce limits on simultaneous writes

[Vladislav Shpilevoy]

On 26.02.2021 22:20, Konstantin Osipov wrote:
> * Vladislav Shpilevoy <v.shpilevoy at tarantool.org> [21/02/26 23:24]:
>> Talking of the other limits - firstly we need to find if some of them
>> really overflows. Then yes, such a semaphone-thing could be applied
>> there too. But AFAIK, there are no other known similar bugs yet.
> 
> Exploring this rather theoretically, since there are no user async
> transactions yet, I can imagine such transaction takes up memory
> and then blocks on WAL semaphore. If there is no limit on the
> number of async transactions, it can be a lot of memory. On the
> other hand this can be limited by a yet another semaphore.

Yes, such transactions will only occupy memory. But the plan is return
an error from box.commit({is_async}) if the WAL queue is full already.
Because we don't want to block the async commit in anyway. Better bail
out earlier and give the user a chance to call normal box.commit() if
necessary. Or introduce some kind of 'try_async' to block if the queue
is full, but no block if not full, I don't know. We didn't think on the
API yet.

There will be a limit both on number of transactions and on their
total size. The limits are introduced right in this patch, and will
be used by async transactions in the future.

The main usage, if I understand correctly, will be for latency-sensitive
applications, when you send your transactions/calls/IPROTO commands via
network, send your data to WAL and return response to the client
immediately. You don't have to wait even until writev() completes.

Interestingly, it won't lead to loss of any guarantees, because anyway
completion of writev() does not mean anything. Still can power off before
it hits the disk, and still the node can fail loosing all non-replicated
data. So seems like a good feature.

Especially should be good for synchrononus transactions, where commit
duration can be quite big.