On Monday, March 11, 2019 5:04:39 PM MSK Konstantin Osipov wrote:
> * Georgy Kirichenko <georgy@tarantool.org> [19/03/11 13:54]:
> 
> It seems we should be able to allow using statements with
> different server_id in the same transaction. When deciding which
> transactions to send back in multi-master mode, we should only
> look at the first statement to find out the source (origin) of the
> transaction and either send all statements in the transaction or
> skip all statements.
I don't think it is a good approach because of:
1. If replica A produces a transaction and replica B writes this transaction 
with local changes then state of replica C is unknown (we couldn't predict 
which replica A or B replicates faster). Also what should happen if C 
replicates from A and D replicates from B - they both will have different data.
2. In case of synchronous replication replica B how replica B should confirm 
its local changes? Using replica A confirmation, but replica A doesn't know 
anything about that as well as other replicaset items that replicates from A 
first. So if this local changes could not even be replicated should we allow 
such changes only for local spaces?
3. This breaks row format - now each row has full info about its transaction 
(replica id and tsn), but without separate tsn_replica_id item we should rely 
on external information (tx row number, previous rows and other). Please keep 
in mind a possibility of interleaving transactions.

> 
> > On Monday, March 11, 2019 12:59:26 PM MSK Konstantin Osipov wrote:
> > > * Georgy Kirichenko <georgy@tarantool.org> [19/03/11 09:55]:
> > > > Form a separate transaction with local changes in case of replication.
> > > > This is important because we should be able to replicate such changes
> > > > (e.g. made within an on_replace trigger) back. In the opposite case
> > > > local changes will be incorporated into originating transaction and
> > > > would be skipped by the originator replica.
> > > 
> > > I wonder will we possibly have some recovery issues, since in fact
> > > we're performing a reordering of execution here?
> > > 
> > > Imagine local and remote statements change the same set of rows.
> > > During initial execution these changes are intermixed, during
> > > recovery they are serialized.
> > 
> > If you remember we were agreed that only local spaces are allowed to
> > change in case of replication triggers.
> > 
> > > It seems we clearly have a problem here. We can either open a bug,
> > > support multiple txn ids in the same stream, support multiple
> > > server ids in the same transaction, ban triggers in
> > > multi-statement transaction?
> > 
> > You pushed me to remove txn_replica_id but it was one of the instruments I
> > planed to use in order to support distributed transactions (with multiple
> > replica ids in the same transaction) in the future. So I would prefer if
> > we
> > just disable changing of non-local spaces during replication. In such case
> > we won't have any issues with reordering.
> > 
> > > Can we attribute local changes to the same server id?
> > 
> > It is impossible because of lsn
> > 
> > > We don't have to replicate them back - this is a gray zone and we can do
> > > it
> > > in any way we want.
> > 
> > I'm afraid no because we already have this functionality and it is even
> > covered with tests.
> > 
> > So we have to make a high level decision: what is expected behavior.
> > In any case I will be agreed with your decision what we should to do:
> > disable non-local replication changes, change behavior of replication for
> > such changes or start further distributed transaction investigation.