[tarantool-patches] Re: [PATCH v2 2/4] wal: wal memory buffer

[Konstantin Osipov]

* Georgy Kirichenko <georgy at tarantool.org> [19/09/18 12:39]:
> Introduce a wal memory buffer to store logged transactions. The patch
> changes wal writer behaviour: now wal encodes rows into the memory buffer
> first and then writes encoded data from memory buffer to disk.
> Wal memory is implemented as xrow buffer which of rotated data chunks.
> Each data chunk includes an ibuf with xrow header array and an obuf with
> encoded xrow data. Also data chunk has a vclock value as the lower boundary of
> stored rows.

This is well designed in my view. So, generally, LGTM - subject to
Vlad's comments, with which I agree.

It would be good to explain in xrow_buf comments its purpose (it
allows subscribers to keep an independent position in the single
stream of rows) and memory complexity. obuf is quite aggressive in
its memory requirements, it may up to double the total amount of
memory necessary to store data, since it has very high
fragmentation. The average should be 75%. I haven't checked your
patch, but it seems the numbers are quite accurate for your usage
scenario as well (although we may assume the average row size is
much less than obuf chunk size, so the average memory utilization
should be around 85%, not 75%). Have you measured this overhead or
at least pondered it?

-- 
Konstantin Osipov, Moscow, Russia