From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id E114F6C7D7; Mon, 26 Apr 2021 20:00:00 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org E114F6C7D7 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1619456401; bh=9Lt/L06lSU32ueukRgJGfFJKoklyp7yZohF0Y+IKJ7o=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=uoykgOHyntGtWFnxRr5tJVLNwTrDFT6IOf0RSWfd8d+dUy1Sjafy1lFCUjG9efHKp iV9bXWZtkg9AK6+c0HgYWAJTu2RPF5J/xVOfXthIuzJx9SpVJ9l+Qt4ykpnQL+GHgP 7CzwNucuNBu4MvImVV+NBSzEuBIkiXQ9Ke7U8FpE= Received: from smtp60.i.mail.ru (smtp60.i.mail.ru [217.69.128.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A5A206FC8E for ; Mon, 26 Apr 2021 19:59:59 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A5A206FC8E Received: by smtp60.i.mail.ru with esmtpa (envelope-from ) id 1lb4aU-0004rT-Hn; Mon, 26 Apr 2021 19:59:59 +0300 To: v.shpilevoy@tarantool.org, gorcunov@gmail.com Date: Mon, 26 Apr 2021 19:59:54 +0300 Message-Id: <20210426165954.46474-1-sergepetrenko@tarantool.org> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9ECFD8CE5F0594010E9FD8A050EF271A331F4FC8E58F6D291182A05F53808504032F34823ED1278A3227653EA68003973C0EAAC8BF70FB474A0208A0FF9012469 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7E188839ED9CF357EEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B3D52627AD81B52CEA1F7E6F0F101C67CDEEF6D7F21E0D1D9295C2E9FA3191EE1B59CA4C82EFA658EE15B92A255A7284C55402F97712B7A1F6B57BC7E64490618DEB871D839B73339E8FC8737B5C22492B7ABF3321926E07CC7F00164DA146DAFE8445B8C89999729449624AB7ADAF37F6B57BC7E64490611E7FA7ABCAF51C92176DF2183F8FC7C07752D4D38D17198D8941B15DA834481F9449624AB7ADAF37BA3038C0950A5D3613377AFFFEAFD26923F8577A6DFFEA7C6386F4032A53F1767B076A6E789B0E97A8DF7F3B2552694A1E7802607F20496D49FD398EE364050F0AC5B80A05675ACD1C9461EB66F04EBFB3661434B16C20AC78D18283394535A9E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8B9149C560DC76099D75ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8183A4AFAF3EA6BDC44C234C8B12C006B7A8F26C6362207B4A2FFE0BC0D5C5DB89C9A634D83FF85FF05B1881A6453793CE9C32612AADDFBE061D77DF2741F6D7EBCC581CE5390BD65102C6BEA0A7F9CB621BB52A3B5A10F442104EBA3D8E7E5B87ABF8C51168CD8EBDB55D5BE2F85BDEC5FDC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A5970F8A4685C8AB9EA04C7B2C15950285B135C644C520CF7BBB184F7B927FEC63156768E473EB521D7E09C32AA3244C4D52DA20CECE1947BF8684B3E700979A4DBEAD0ED6C55A80927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojmUj/IHfqgKd6XmVAgt2J2Q== X-Mailru-Sender: 583F1D7ACE8F49BDD2846D59FC20E9F89737212286DB8EB84B7100F3125F7D75720D3BCC8331086C424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: [Tarantool-patches] [PATCH] recovery: make it yield when positioning in a WAL X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" We had various places in box.cc and relay.cc which counted processed rows and yielded every now and then. These yields didn't cover cases, when recovery has to position inside a long WAL file: For example, when tarantool exits without leaving an empty WAL file which'll be used to recover instance vclock on restart. In this case the instance freezes while processing the last availabe WAL in order to recover the vclock. Another issue is with replication. If a replica connects and needs data from the end of a really long WAL, recovery will read up to the needed position without yields, making relay disconnect by timeout. In order to fix the issue, make recovery decide when a yield should happen. Introduce a new callback: schedule_yield, which is called by recovery once it processes (no matter how, either simply skips or calls xstream_write) enough rows (WAL_ROWS_PER_YIELD). schedule_yield either yields right away, in case of relay, or saves the yield for later, in case of local recovery, because it might be in the middle of a transaction. The only place with explicit row counting and manual yielding is now in relay_initial_join, since its row sources are engines rather than recovery with its WAL files. Closes #5979 --- https://github.com/tarantool/tarantool/tree/sp/gh-5979-recovery-yield https://github.com/tarantool/tarantool/issues/5979 src/box/box.cc | 32 +++++++++++++++----------------- src/box/recovery.cc | 26 +++++++++++++++++--------- src/box/recovery.h | 15 ++++++++++++++- src/box/relay.cc | 28 ++++++++++++++++++---------- 4 files changed, 64 insertions(+), 37 deletions(-) diff --git a/src/box/box.cc b/src/box/box.cc index 59925962d..69a8f87eb 100644 --- a/src/box/box.cc +++ b/src/box/box.cc @@ -335,8 +335,6 @@ struct wal_stream { * matches LSN of a first global row. */ bool has_global_row; - /** How many rows have been recovered so far. */ - size_t rows; }; /** @@ -570,12 +568,6 @@ end_diag_request: static void wal_stream_try_yield(struct wal_stream *stream) { - /* - * Save the yield. Otherwise it would happen only on rows which - * are a multiple of WAL_ROWS_PER_YIELD and are last in their - * transaction, which is probably a very rare coincidence. - */ - stream->has_yield |= (stream->rows % WAL_ROWS_PER_YIELD == 0); if (wal_stream_has_tx(stream) || !stream->has_yield) return; stream->has_yield = false; @@ -587,11 +579,6 @@ wal_stream_apply_row(struct xstream *base, struct xrow_header *row) { struct wal_stream *stream = container_of(base, struct wal_stream, base); - /* - * Account all rows, even non-DML, and even leading to an error. Because - * still need to yield sometimes. - */ - ++stream->rows; if (iproto_type_is_synchro_request(row->type)) { if (wal_stream_apply_synchro_row(stream, row) != 0) goto end_error; @@ -618,7 +605,6 @@ wal_stream_create(struct wal_stream *ctx) ctx->first_row_lsn = 0; ctx->has_yield = false; ctx->has_global_row = false; - ctx->rows = 0; } /* {{{ configuration bindings */ @@ -3101,6 +3087,19 @@ bootstrap(const struct tt_uuid *instance_uuid, } } +struct wal_stream wal_stream; + +/** + * Plan a yield in recovery stream. Wal stream will execute it as soon as it's + * ready. + */ +static void +wal_stream_schedule_yield(void) +{ + wal_stream.has_yield = true; + wal_stream_try_yield(&wal_stream); +} + /** * Recover the instance from the local directory. * Enter hot standby if the directory is locked. @@ -3124,7 +3123,6 @@ local_recovery(const struct tt_uuid *instance_uuid, say_info("instance uuid %s", tt_uuid_str(&INSTANCE_UUID)); - struct wal_stream wal_stream; wal_stream_create(&wal_stream); auto stream_guard = make_scoped_guard([&]{ wal_stream_abort(&wal_stream); @@ -3132,8 +3130,8 @@ local_recovery(const struct tt_uuid *instance_uuid, struct recovery *recovery; bool is_force_recovery = cfg_geti("force_recovery"); - recovery = recovery_new(wal_dir(), is_force_recovery, - checkpoint_vclock); + recovery = recovery_new(wal_dir(), is_force_recovery, checkpoint_vclock, + wal_stream_schedule_yield); /* * Make sure we report the actual recovery position diff --git a/src/box/recovery.cc b/src/box/recovery.cc index cd33e7635..5351d8524 100644 --- a/src/box/recovery.cc +++ b/src/box/recovery.cc @@ -81,7 +81,7 @@ */ struct recovery * recovery_new(const char *wal_dirname, bool force_recovery, - const struct vclock *vclock) + const struct vclock *vclock, schedule_yield_f schedule_yield) { struct recovery *r = (struct recovery *) calloc(1, sizeof(*r)); @@ -113,6 +113,9 @@ recovery_new(const char *wal_dirname, bool force_recovery, r->watcher = NULL; rlist_create(&r->on_close_log); + r->row_count = 0; + r->schedule_yield = schedule_yield; + guard.is_active = false; return r; } @@ -139,8 +142,12 @@ recovery_scan(struct recovery *r, struct vclock *end_vclock, if (xdir_open_cursor(&r->wal_dir, vclock_sum(end_vclock), &cursor) != 0) return; struct xrow_header row; - while (xlog_cursor_next(&cursor, &row, true) == 0) + while (xlog_cursor_next(&cursor, &row, true) == 0) { vclock_follow_xrow(end_vclock, &row); + if (++r->row_count % WAL_ROWS_PER_YIELD == 0) { + r->schedule_yield(); + } + } xlog_cursor_close(&cursor, false); } @@ -241,10 +248,16 @@ static void recover_xlog(struct recovery *r, struct xstream *stream, const struct vclock *stop_vclock) { + /* Imitate old behaviour. Rows are counted separately for each xlog. */ + r->row_count = 0; struct xrow_header row; - uint64_t row_count = 0; while (xlog_cursor_next_xc(&r->cursor, &row, r->wal_dir.force_recovery) == 0) { + if (++r->row_count % WAL_ROWS_PER_YIELD == 0) { + r->schedule_yield(); + } + if (r->row_count % 100000 == 0) + say_info("%.1fM rows processed", r->row_count / 1000000.); /* * Read the next row from xlog file. * @@ -273,12 +286,7 @@ recover_xlog(struct recovery *r, struct xstream *stream, * failed row anyway. */ vclock_follow_xrow(&r->vclock, &row); - if (xstream_write(stream, &row) == 0) { - ++row_count; - if (row_count % 100000 == 0) - say_info("%.1fM rows processed", - row_count / 1000000.); - } else { + if (xstream_write(stream, &row) != 0) { if (!r->wal_dir.force_recovery) diag_raise(); diff --git a/src/box/recovery.h b/src/box/recovery.h index c8ccaa553..4212fd192 100644 --- a/src/box/recovery.h +++ b/src/box/recovery.h @@ -43,6 +43,12 @@ extern "C" { struct xrow_header; struct xstream; +/** + * A type for a callback invoked by recovery after some batch of rows is + * processed. Is used mostly to unblock the event loop every now and then. + */ +typedef void (*schedule_yield_f)(void); + struct recovery { struct vclock vclock; /** The WAL cursor we're currently reading/writing from/to. */ @@ -56,11 +62,18 @@ struct recovery { struct fiber *watcher; /** List of triggers invoked when the current WAL is closed. */ struct rlist on_close_log; + uint64_t row_count; + /** + * A callback recovery calls eveery WAL_ROWS_PER_YIELD processed rows. + * This includes both applied rows and the rows which were skipped to + * position recovery at the right position in xlog. + */ + schedule_yield_f schedule_yield; }; struct recovery * recovery_new(const char *wal_dirname, bool force_recovery, - const struct vclock *vclock); + const struct vclock *vclock, schedule_yield_f schedule_yield); void recovery_delete(struct recovery *r); diff --git a/src/box/relay.cc b/src/box/relay.cc index ff43c2fc7..c7002bd46 100644 --- a/src/box/relay.cc +++ b/src/box/relay.cc @@ -418,6 +418,12 @@ relay_final_join_f(va_list ap) return 0; } +static void +relay_yield(void) +{ + fiber_sleep(0); +} + void relay_final_join(int fd, uint64_t sync, struct vclock *start_vclock, struct vclock *stop_vclock) @@ -438,7 +444,7 @@ relay_final_join(int fd, uint64_t sync, struct vclock *start_vclock, * always be valid. */ vclock_copy(&relay->recv_vclock, start_vclock); - relay->r = recovery_new(wal_dir(), false, start_vclock); + relay->r = recovery_new(wal_dir(), false, start_vclock, relay_yield); vclock_copy(&relay->stop_vclock, stop_vclock); int rc = cord_costart(&relay->cord, "final_join", @@ -904,7 +910,7 @@ relay_subscribe(struct replica *replica, int fd, uint64_t sync, * always be valid. */ vclock_copy(&relay->recv_vclock, replica_clock); - relay->r = recovery_new(wal_dir(), false, replica_clock); + relay->r = recovery_new(wal_dir(), false, replica_clock, relay_yield); vclock_copy(&relay->tx.vclock, replica_clock); relay->version_id = replica_version_id; @@ -928,13 +934,6 @@ relay_send(struct relay *relay, struct xrow_header *packet) coio_write_xrow(&relay->io, packet); fiber_gc(); - /* - * It may happen that the socket is always ready for write, so yield - * explicitly every now and then to not block the event loop. - */ - if (++relay->row_count % WAL_ROWS_PER_YIELD == 0) - fiber_sleep(0); - struct errinj *inj = errinj(ERRINJ_RELAY_TIMEOUT, ERRINJ_DOUBLE); if (inj != NULL && inj->dparam > 0) fiber_sleep(inj->dparam); @@ -944,6 +943,15 @@ static void relay_send_initial_join_row(struct xstream *stream, struct xrow_header *row) { struct relay *relay = container_of(stream, struct relay, stream); + /* + * It may happen that the socket is always ready for write, so yield + * explicitly every now and then to not block the event loop. + * This is not part of relay_yield(), because here the engines are the + * source of rows, and not recovery. + */ + if (++relay->row_count % WAL_ROWS_PER_YIELD == 0) + fiber_sleep(0); + /* * Ignore replica local requests as we don't need to promote * vclock while sending a snapshot. @@ -982,7 +990,7 @@ relay_restart_recovery(struct relay *relay) struct vclock restart_vclock; vclock_copy(&restart_vclock, &relay->recv_vclock); vclock_reset(&restart_vclock, 0, vclock_get(&relay->r->vclock, 0)); - struct recovery *r = recovery_new(wal_dir(), false, &restart_vclock); + struct recovery *r = recovery_new(wal_dir(), false, &restart_vclock, relay_yield); rlist_swap(&relay->r->on_close_log, &r->on_close_log); recovery_delete(relay->r); relay->r = r; -- 2.24.3 (Apple Git-128)