[tarantool-patches] Re: [PATCH v2 4/4] replication: use wal memory buffer to fetch rows

[Georgy Kirichenko]

On Friday, September 20, 2019 10:56:50 AM MSK Konstantin Osipov wrote:
> * Georgy Kirichenko <georgy at tarantool.org> [19/09/18 12:39]:
> > A relay tries to create a wal memory cursor and follow them to relay
> > data to it's replica. If a relay failed to attach to a wal memory buffer
> > or went out of the buffer then the relay recovers xlogs from files and
> > the makes a new try to attach.
> 
> A major review comment for the previous version of the series was
> that in case of in-memory replications, relays using in-memory
> buffer shouldn't bother advancing gc state independently - gc
> state should be advanced upon chunk rotation of xrow_buf.
> 
> This seems to be a pretty straightforward thing to do, all you
> need to do is have a trigger which is fired when xrow_buf rotates
> chunks. It would also save on wal gc events quite a bit and will
> allow using the old approach with on_close_log() to notify gc
> when relay is operating from files, not memory.
> 
> What was the issue with implementing these suggestions?
I already tried to do this but the approach is not working - there to many 
edge cases. For instance - what if a gc forces to delete files protected by 
memory buffer? Currently gc just 'kills' a consumer so we need to implement a 
different kind of consumer which should persist event corresponding files are 
deleted. But it breaks the gc code.
The trigger is also not good for us - because in this case we should tackle 
the race between triggers - one trigger which advances the memory buffer gc and 
others which restores a consumer. Incorrect order - and relay files would be 
cleared just before relay tried to protect the files.

Additionally, I think that use on-close_log trigger and then vclock_sum was a 
design mistake - it allowed to make fast-dirty patch of an issue but harm us a 
lot right now - I spent around 80% of time while handling this.

In earlier message you urged that there are still tons of ACK packets. You are 
absolutely right but gc also performs some kind of gc schedule request 
grouping. This is not so good as could be but I believe we could fix this just 
after the relay ACK logic was moved to wal, which allows us to put the xlog 
processing logic in one thread: currently it is spread across wal (which 
handles wal_dir), relay (which handles ACK, file boundaries) and tx (which 
decides which relay is the 'oldest').