From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id A51DB6ECE3; Wed, 20 Oct 2021 09:35:46 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A51DB6ECE3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1634711746; bh=xaVD/XxxCGVM+dC9jeAFV4afn4ARFpSQPlcUSIBfq1A=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=Uc8W0QFRxYL4LEt9d4vhwRRrrxtJpLPPQM3v+F4daNFgT+peBwCXAU15QQ73EG24x 8uly/mgoWVruhRPz/W6CmunyjUNl5wNNELclerHdmYv0uU2EIGf2oFeXVC1zZ8xDRd 29Z/kw0nwvDHckLfqoewvfg5n4Ljq8kKLyYkZeOs= Received: from smtp36.i.mail.ru (smtp36.i.mail.ru [94.100.177.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id B7B0F6ECE3 for ; Wed, 20 Oct 2021 09:35:44 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B7B0F6ECE3 Received: by smtp36.i.mail.ru with esmtpa (envelope-from ) id 1md5CS-00030d-10; Wed, 20 Oct 2021 09:35:44 +0300 To: Cyrill Gorcunov , Vladislav Shpilevoy Cc: tml References: <20211014215622.49732-1-gorcunov@gmail.com> <20211014215622.49732-4-gorcunov@gmail.com> <5ce44467-f9a9-0743-3394-c2a40cff463c@tarantool.org> Message-ID: Date: Wed, 20 Oct 2021 09:35:43 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9C7814344C8C501C8A4649CC2A06649B6A97F49A0BDC630D4182A05F538085040EFC6F7DFBDDD3EC693F480063376D1BB250CFDEE4A2FB93DFD0A8A2C09D8A986 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7EFCED9770EA041F4EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637C5BAFDCCAC60DA968638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D84AAF3ED4891B73A4174CE364A93E878B117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC1F8789D36234D406A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F4460429728776938767073520B28585415E75ADA9C26CFBAC0749D213D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B67393CE827C55B5F775ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A5DC121FA4AB7BCD37E10207049C4E836FBBBA0870CA220CA7D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7557E988E9157162368E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34F05B761BB9C2AA44D2F2802E34D4ACC11E4C55255597D3818FFC764068D73EC9FFBDE25EDBF2F3211D7E09C32AA3244C3A0DF0AD034B6E4CF25921C778047FF5B038C9161EF167A1FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojMPcpz6kU2m+H3kJby0WqNw== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A44637C15BFC850F354D2ABA80F7BECADA8726FD79FD29019084424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v23 3/3] test: add gh-6036-qsync-order test X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 20.10.2021 01:26, Cyrill Gorcunov пишет: > On Tue, Oct 19, 2021 at 06:09:50PM +0300, Serge Petrenko wrote: >>> +-- >>> +-- The election_replica1 node has no clue that there is a new leader >>> +-- and continue writing data with obsolete term. Since election_replica3 >>> +-- is delayed now the INSERT won't proceed yet but get queued. >>> +test_run:switch("election_replica1") >>> + | --- >>> + | - true >>> + | ... >>> +box.space.test:insert{3} >>> + | --- >>> + | - [3] >>> + | ... >>> + >>> +-- >>> +-- Finally enable election_replica3 back. Make sure the data from new election_replica2 >>> +-- leader get writing while old leader's data ignored. >>> +test_run:switch("election_replica3") >>> + | --- >>> + | - true >>> + | ... >> Hi and thanks for the fixes! >> >> I have only one comment left. >> >> Actually you do need to count writes here. >> The wait_cond for ERRINJ_WAL_WRITE_COUNT == write_cnt + 3 >> is needed to make sure you receive (and thus try to process) >> insert {3} **before** the replica is re-enabled. >> >> Otherwise we can't be sure that the test is correct. You may simply >> perform a select before insert{3} has reached the replica. > You know, I spent a few hours trying to pass the test waiting for > ERRINJ_WAL_WRITE_COUNT == write_cnt + 3 and finally realized that > it seems that is what happens: the replica1 is not longer a leader > and when this record reach our replica3 node we NOPify it then > we run > > apply_row > if (request.type == IPROTO_NOP) > return process_nop() > > thus this record even not reaching the journal at all and that is > why waiting for write_cnt + 3 lasts forever. If only I didn't miss > something obvious. Unfortunately, this is not the case. A NOP entry still reaches WAL. That's why we need NOP entries: they reside in WAL but do nothing. That's for vclock bump sake. Otherwise we could skip such entries completely, without nopifying them. So, even if the entry is nopified, it would enter WAL sooner or later. I just realised what the problem is: the entry is waiting on a limbo latch inside the NOPify procedure. That's why it never reaches the journal (until we re-enable replica3, at least). I don't know how to wait for this entry's arrival then. The current test version looks OK to me. Vlad, do you have any ideas here? -- Serge Petrenko