[tarantool-patches] Re: [PATCH v3 2/4] wal: xrow buffer structure

[Konstantin Osipov]

* Georgy Kirichenko <georgy at tarantool.org> [19/10/09 19:50]:
> Introduce a xrow buffer to store encoded xrows in memory after
> transaction was finished. Wal uses an xrow buffer object in
> order to encode transactions and then writes encoded data
> to a log file so encoded data still lives in memory for
> some time after a transaction is finished and cleared by an engine.
> Xrow buffer consist of not more than XROW_BUF_CHUNK_COUNT rotating
> chunks organized in a ring. Rotation thresholds and
> XROW_BUF_CHUNK_COUNT are empiric values and were hardcoded.

Very nice comments.

> @@ -156,6 +157,14 @@ struct wal_writer
>  	 * Used for replication relays.
>  	 */
>  	struct rlist watchers;
> +	/**
> +	 * In-memory WAl write buffer used to encode transaction rows and
> +	 * write them to an xlog file. An in-memory buffer allows us to
> +	 * preserve xrows after transaction processing was finished.
> +	 * This buffer will be used by replication to fetch xrows from memory
> +	 * without xlog files access.
> +	 */
> +	struct xrow_buf xrow_buf;

And here.

> +	/* Start a wal memory buffer transaction. */
> +	xrow_buf_tx_begin(&writer->xrow_buf, &writer->vclock);
>  	stailq_foreach_entry(entry, &wal_msg->commit, fifo) {
>  		wal_assign_lsn(&vclock_diff, &writer->vclock,
>  			       entry->rows, entry->rows + entry->n_rows);
>  		entry->res = vclock_sum(&vclock_diff) +
>  			     vclock_sum(&writer->vclock);
> -		rc = xlog_write_entry(l, entry);
> -		if (rc < 0)
> +		struct iovec *iov;
> +		int iovcnt = xrow_buf_write(&writer->xrow_buf, entry->rows,
> +					    entry->rows + entry->n_rows, &iov);
> +		if (iovcnt < 0) {
> +			xrow_buf_tx_rollback(&writer->xrow_buf);
> +			goto done;
> +		}
> +		xlog_tx_begin(l);
> +		if (xlog_write_iov(l, iov, iovcnt, entry->n_rows) < 0) {
> +			xlog_tx_rollback(l);
> +			xrow_buf_tx_rollback(&writer->xrow_buf);
> +			goto done;
> +		}
> +		rc = xlog_tx_commit(l);
> +		if (rc < 0) {
> +			/* Failed write. */
> +			xrow_buf_tx_rollback(&writer->xrow_buf);
>  			goto done;
> +		}
>  		if (rc > 0) {
> +			/*
> +			 * Data flushed to disk, start a new memory
> +			 * buffer transaction
> +			 */
>  			writer->checkpoint_wal_size += rc;
>  			last_committed = &entry->fifo;
>  			vclock_merge(&writer->vclock, &vclock_diff);
> +			xrow_buf_tx_commit(&writer->xrow_buf);
> +			xrow_buf_tx_begin(&writer->xrow_buf, &writer->vclock);


I wonder why do you start the transaction here, right after you
commit it? Why not start it along with xlog tx? 

Generally I don't understand the magic relationship between xlog
transaction and xrow transaction, could you explain it?

It would also be nice to explain why xlog doesn't use xrow_buf
buffer and copies the data one more time to its own buffer - to
checksum it? To compress it? 


-- 
Konstantin Osipov, Moscow, Russia