[server-dev] [RFC] Interactive transactions in IProto

[Vladimir Davydov]

** The problem **

If a CALL request leaves a transaction open upon return, the transaction
will be forcefully aborted. This makes sense for memtx, because it
doesn't support fiber yield (yet), however in case of vinyl, the user
may want to continue execution of the same transaction in the next CALL,
but currently it isn't possible.

Another use case for this is SQL EXECUTE request, which is used for
executing SQL statements. The problem is this request can only be used
for executing a single SQL statement so without transactions in IProto
it is impossible to implement SQL transactions on a remote client (e.g.
via JDBC).

See [1] for more details.

** The solution **

Introduce the concept of streams within an IProto connection, as
suggested by Georgy Kirichenko:

 1. Introduce new request header key IPROTO_STREAM_ID.
 2. The stream id is generated by the user and passed along with all
    requests that are supposed to be executed in the same stream.
 3. All requests within the same stream are executed sequentially.
 4. Requests from different streams may be executed in parallel.
 5. If a transaction is left open by a request in a stream, the next
    request will reuse it.
 6. If IPROTO_STREAM_ID is unset (0), everything works as before, i.e.
    no transaction preservation or request serialization will occur.

The net.box API will look like this:

  c = net_box.connect(...)
  s = c:make_stream(stream_id)
  s:call(...)

A net.box stream instance will be a wrapper around the connection it was
created for. It will have all the same methods as the connection itself,
but all requests sent on its behalf will have the stream id attached.

** Benefits **

The concept of a stream should be very clear to the user. Basically, it
gives you the ability to multiplex several connections within one, even
if there's no intention to reuse transactions.

For example we could use it in JDBC to multiplex different clients via
the same connection: each client would be assigned a unique stream id,
but they would all go through the same connection thus saving system
resources.

** Technicalities **

First, we will need to make it possible to detach transactions from
fibers. This isn't as simple as it may seem, because all transaction
data is allocated on fiber->gc region. The proposal how this should be
done in my opinion is given here [2].

Second, we will have to add a hash table mapping stream ids to stream
objects in the tx thread. A stream object would basically be a queue of
request awaiting execution plus an open transaction if any. When a new
request is sent to iproto, it is submitted to the tx thread and then
either executed directly by a fiber of the fiber pool or queued to the
stream object if there's already a request from the same stream being
executed by another fiber. When a fiber finishes executing a request, it
checks if there are more request in the stream queue and continues
execution if so.

** Questions **

Should we limit the number of streams somehow? I don't think so, at
least not right now, because streams are completely user controlled,
like iproto connections.

How to close a stream so that the corresponding stream object is
destroyed on the server? Do we need to bother at all? May be dropping
all streams along with a connection would be enough?

Should we avoid a fiber_call() when queueing a request? In other words,
should streams be implemented inside fiber_pool so that we don't need to
execute a call in a fiber in case all it's going to do is just queue a
request to be executed by another fiber. This would look cleaner and
would probably be more efficient.

** Alternatives **

Instead of introducing streams, we could return transaction_id in reply
to a CALL or EXECUTE request that opened a transaction. The user could
then use this transaction_id with the next CALL to continue working with
the same transaction. However,

 - This would break compatibility. Currently, if the transaction is open
   by the time a CALL returns, it will be aborted. If we don't do that
   we can end up accumulating stale transactions on the server side in
   case there's a mistake in the client code. We could probably add a
   flag that would specify if the client intends to leave the
   transaction open though, but that would mean we'd have to add yet
   another new key to the iproto protocol.

 - Generating transaction_id on the server and returning it back to the
   client would look rather ugly when it comes to implementing net_box
   client, because we would have to add extra return parameters to
   net.box 'call' method, which isn't very convenient. Compare with
   streams where the user gets a stream object that behaves exactly like
   a connection, but guarantees sequential execution and transaction
   preservation.

 - How to deal with open connections that are kept open by the server
   for too long? Abort them on timeout?

 - Currently, there's no way to guarantee that certain requests execute
   sequentially. Streams provide the user with such a way so they can be
   useful even without transactions. Returning transaction_id can't be
   used for anything else but interactive transactions.

** References **

[1] https://github.com/tarantool/tarantool/issues/2503
[2] https://github.com/tarantool/tarantool/issues/2503#issuecomment-415480435