From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp20.mail.ru (smtp20.mail.ru [94.100.179.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 94A084696C3 for ; Tue, 31 Mar 2020 03:47:40 +0300 (MSK) From: Nikita Pettik Date: Tue, 31 Mar 2020 03:47:37 +0300 Message-Id: Subject: [Tarantool-patches] [PATCH] vinyl: bump dump_generation in case scheduler doesn't catch up with DDL List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tarantool-patches@dev.tarantool.org Cc: v.shpilevoy@tarantool.org It may turn out that dump_generation does not catch up with current generation and no other dump tasks are in progress. This may happen dump process is throttled due to errors. In this case generation is bumped but dump_generation is not (since dump is not completed). In turn, throttling opens a window for DDL operations. For instance, index dropping and creation of new one results in mentioned situation: box.snapshot() -- fails for some reason; next attempt at dumping will be -- taken in one second. s:drop() -- drop index to be dumped s = box.schema.space.create('test', {engine = 'vinyl'}) -- create new one (its mem generation is greater than scheduler's one) i = s:create_index('pk') Closes #4821 --- Note that current fix is only workaround amd doesn't pretend to be the most proper one. Alternatively, we can rollback scheduler generation in case dump is failed and index to be dumped is dropped. Branch: https://github.com/tarantool/tarantool/tree/np/gh-4821-scheduler-gen-assert Issue: https://github.com/tarantool/tarantool/issues/4821 src/box/vy_scheduler.c | 14 ++++- .../gh-4821-ddl-during-throttled-dump.result | 57 +++++++++++++++++++ ...gh-4821-ddl-during-throttled-dump.test.lua | 22 +++++++ test/vinyl/suite.ini | 2 +- 4 files changed, 92 insertions(+), 3 deletions(-) create mode 100644 test/vinyl/gh-4821-ddl-during-throttled-dump.result create mode 100644 test/vinyl/gh-4821-ddl-during-throttled-dump.test.lua diff --git a/src/box/vy_scheduler.c b/src/box/vy_scheduler.c index bf4c3fe58..880304b66 100644 --- a/src/box/vy_scheduler.c +++ b/src/box/vy_scheduler.c @@ -1846,9 +1846,19 @@ retry: /* * Dump is in progress, but all eligible LSM trees are * already being dumped. Wait until the current round - * is complete. + * is complete. But if there's no any other tasks + * in progress, it may mean that dump_generation does + * not catch up with current generation. This may happen + * due to failed dump process (i.e. generation is bumped + * but dump_generation is not). In turn, after dump is failed, + * next dump will be throttled which opens a window for DDL + * operations. For instance, index dropping and creation of + * new one results in mentioned situation. */ - assert(scheduler->dump_task_count > 0); + if (scheduler->dump_task_count == 0) { + scheduler->dump_generation = vy_lsm_generation(lsm); + goto retry; + } no_task: if (worker != NULL) vy_worker_pool_put(worker); diff --git a/test/vinyl/gh-4821-ddl-during-throttled-dump.result b/test/vinyl/gh-4821-ddl-during-throttled-dump.result new file mode 100644 index 000000000..eee031101 --- /dev/null +++ b/test/vinyl/gh-4821-ddl-during-throttled-dump.result @@ -0,0 +1,57 @@ +-- test-run result file version 2 +errinj = box.error.injection + | --- + | ... +fiber = require('fiber') + | --- + | ... + +-- During this test we verify that if dump process is throttled +-- due to error, DDL operations in window between dumps won't +-- break anything. +-- +s = box.schema.space.create('test', {engine = 'vinyl'}) + | --- + | ... +_ = s:create_index('pk') + | --- + | ... + +errinj.set('ERRINJ_VY_RUN_WRITE', true) + | --- + | - ok + | ... +s:replace{2} + | --- + | - [2] + | ... +box.snapshot() + | --- + | - error: Error injection 'vinyl dump' + | ... +errinj.set('ERRINJ_VY_RUN_WRITE', false) + | --- + | - ok + | ... +s:drop() + | --- + | ... + +s = box.schema.space.create('test1', {engine = 'vinyl'}) + | --- + | ... +i = s:create_index('pk1') + | --- + | ... +s:replace{1} + | --- + | - [1] + | ... +-- Unthrottle scheduler. +errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0) + | --- + | - ok + | ... +s:drop() + | --- + | ... diff --git a/test/vinyl/gh-4821-ddl-during-throttled-dump.test.lua b/test/vinyl/gh-4821-ddl-during-throttled-dump.test.lua new file mode 100644 index 000000000..750bed0db --- /dev/null +++ b/test/vinyl/gh-4821-ddl-during-throttled-dump.test.lua @@ -0,0 +1,22 @@ +errinj = box.error.injection +fiber = require('fiber') + +-- During this test we verify that if dump process is throttled +-- due to error, DDL operations in window between dumps won't +-- break anything. +-- +s = box.schema.space.create('test', {engine = 'vinyl'}) +_ = s:create_index('pk') + +errinj.set('ERRINJ_VY_RUN_WRITE', true) +s:replace{2} +box.snapshot() +errinj.set('ERRINJ_VY_RUN_WRITE', false) +s:drop() + +s = box.schema.space.create('test1', {engine = 'vinyl'}) +i = s:create_index('pk1') +s:replace{1} +-- Unthrottle scheduler. +errinj.set('ERRINJ_VY_SCHED_TIMEOUT', 0) +s:drop() diff --git a/test/vinyl/suite.ini b/test/vinyl/suite.ini index c8bc270f3..d1a0b0e71 100644 --- a/test/vinyl/suite.ini +++ b/test/vinyl/suite.ini @@ -2,7 +2,7 @@ core = tarantool description = vinyl integration tests script = vinyl.lua -release_disabled = errinj.test.lua errinj_ddl.test.lua errinj_gc.test.lua errinj_stat.test.lua errinj_tx.test.lua errinj_vylog.test.lua partial_dump.test.lua quota_timeout.test.lua recovery_quota.test.lua replica_rejoin.test.lua +release_disabled = errinj.test.lua errinj_ddl.test.lua errinj_gc.test.lua errinj_stat.test.lua errinj_tx.test.lua errinj_vylog.test.lua partial_dump.test.lua quota_timeout.test.lua recovery_quota.test.lua replica_rejoin.test.lua gh-4821-ddl-during-throttled-dump.test.lua config = suite.cfg lua_libs = suite.lua stress.lua large.lua txn_proxy.lua ../box/lua/utils.lua use_unix_sockets = True -- 2.17.1