Re: [Tarantool-patches] [PATCH v3] wal: introduce limits on simultaneous writes

[Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

On 26.02.2021 08:18, Konstantin Osipov wrote:
> * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [21/02/26 10:15]:
>>> I'd also question the place where you decided to put this gate.
>>> The source of the issue is async requests, not WAL, which worked
>>> fine in absence of async requests. So it's async requests that
>>> should be gated, not WAL.
>>
>> In the commit message it is clearly stated why it is in the
>> journal's code, not just in the applier:
>>
>> 	The feature is ready for `box.commit{is_async=true}`. Once it's
>> 	implemented, it should check whether the queue is full and let the user
>> 	decide what to do next. Either wait or roll the tx back.
>>
>> Async transactions will be exposed to 'userspace' to be able to reduce
>> latency for network requests ending with a transaction. They won't have
>> to wait for WAL write to end.
> 
> You did not understand my comment. I tried to say that a major
> part of this code is generic and should reside in lib/core as a
> counting semaphore abstraction. Async transaction simply use this
> counting semaphore to throttle themselves. Then neither WAL nor
> any other resource used by async transactions will be overloaded.
> 
> Otherwise, the system would be allowed to create async
> transactions, and while WAL will not overflow, some other resource
> (memory, transaction identifiers, whatever) may still overflow. 

Ok, now I understand. Yeah, I also think it is a good idea to move it
libcore if nothing major will change in the patch due to any reason.

Talking of the other limits - firstly we need to find if some of them
really overflows. Then yes, such a semaphone-thing could be applied
there too. But AFAIK, there are no other known similar bugs yet.

>>> Otherwise your overflow will just spill out someplace else.
>>
>> On the contrary. Your proposal to do it in the applier would lead to
>> queue overflow in some other place - in userspace. When the queue is
>> for the entire WAL, it won't overflow.
> 
> I did  not say it should be in the applier. 

It was a misunderstanding.