[server-dev] [RFC] Interactive transactions in IProto

[Vladislav Shpilevoy]

On 15/11/2018 01:11, Konstantin Osipov wrote:
> * Vladimir Davydov <vdavydov.dev at gmail.com> [18/11/14 21:26]:
> > On Wed, Nov 14, 2018 at 06:25:30PM +0300, Konstantin Osipov wrote:
> > > * Vladimir Davydov <vdavydov.dev at gmail.com> [18/11/13 20:51]:
> > > >   6. If IPROTO_STREAM_ID is unset (0), everything works as before, i.e.
> > > >      no transaction preservation or request serialization will occur.
> > > 
> > > I think this is a bit confusing. Logically they all must belong to
> > > stream id 0, but suddenly they all belong to (internally) distinct
> > > streams. What if I do begin on the client without an assigned
> > > stream id? Will I have a stream created automatically? What if I
> > > do a commit for a non-existent stream id?
> > 
> > IMO stream id 0 means no streaming and hence no interactive
> > transactions.
> 
> OK, i get it, please add to the spec. what happens if I issue
> begin in this stream? What happens if I implicitly start a
> transaction (e.g. inside a call)? Please add to the spec.
> 
> > > 
> > > Definitely, it should be a global limit, a new box.cfg setting (as
> > > much as I hate settings there is no waywe can do away without it
> > > here afaiu).
> > 
> > But why? We don't limit the number of incoming connections. Why should
> > we bother with limiting streams?
> 
> The number of incoming connections is limited implicitly by
> ulimit. A connection doesn't take database resources, it only
> consumers memory buffers and file descriptors. An open transaction
> potentially holds a lot more resources. E.g. a typical graph-based
> deadlock detector has complexity O(N^2) on the number of
> transactions.

Streams have nothing to do with deadlock resolvers or transactions.
Even without streams and even now I can create thousands of active
transactions. Streams are at a lower level that transactions. You
for unknown reason think that stream == transaction, but it is false.

Using streams I can even process not DB operations, like proxy, or
my own application server logic which do not use DB. Streams allow me
to just order my requests. Nothing more. Just. Order.

Also they can be used to order async auto-commit requests, talking
about DB.

> 
> > > > How to close a stream so that the corresponding stream object is
> > > > destroyed on the server? Do we need to bother at all? May be dropping
> > > > all streams along with a connection would be enough?
> > > 
> > > No, it's definitely not enough. We need to think about denial of
> > > service attacks and hogging all server memory by creating too many
> > > streams. This is especially important since automatic streams
> > > (those that are created if stream id is not set) will also take
> > > resources. Perhaps wee should forbid interactive transactions in
> > > stream id 0, this would at least be safe.
> > > 
> > > Please also state explicitly what happens with open transactions
> > > in a stream when a connection is closed. And yes, I presume ppl
> > 
> > When a connection is closed, all corresponding streams are destroyed and
> > all transactions are closed.
> 
> It's obvious they are closed. Are they rolled back? What happens
> with on_rollback triggers, are they fired? Closing a
> connection may thus take a long time. Please add to
> the spec.
> 
> > > will ask for ways to control the stream so we better think right
> > > now what these ways could be (even if we decide to not implement
> > > them at once).
> > 
> > We could destroy a stream in case there's no pending requests for this
> > stream id and no open transaction. That is a stream exists on the server
> > side long as there's an uncommitted transaction in it. We could also
> > rollback the transaction used by a stream and destroy the stream by
> > timeout, I guess.
> 
>  From the user perspective, is COMMIT/ROLLBACK message sent over IPROTO with
> an assigned stream id equivalent to "destruction" of a stream? What
> if I have a cursor open against an SQL result set in a stream and
> then send a COMMIT over? Does the cursor live on or get
> destroyed along? Are there other ways to destroy a stream except
> by sending a commit/rollback message? What about an implicit
> commit inside a CALL/EVAL? If streams are being created by a start
> of transaction and destroyed by end of transaction, how are they
> different from transaction ids? Essentially the only difference
> then that we allow users to assign transaction identifiers and
> call them stream identifiers for whatever reason.
> 

Please, think not only about SQL and interactive transactions. We
have application server, auto-commit transactions and async requests
as well.