From: Vladislav Shpilevoy <v.shpilevoy@tarantool.org> To: Oleg Babin <olegrok@tarantool.org>, tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org Subject: Re: [Tarantool-patches] [PATCH vshard 1/2] storage: introduce upgrade strategy Date: Mon, 23 Mar 2020 23:32:08 +0100 [thread overview] Message-ID: <955a5af8-bcfa-8b29-a808-e240e363033c@tarantool.org> (raw) In-Reply-To: <d9cd3502-b3fc-c09f-2c23-e6e9ebeb883b@tarantool.org> Hi! Thanks for the review! On 23/03/2020 07:35, Oleg Babin wrote: > On 22/03/2020 22:12, Vladislav Shpilevoy wrote: >> But for 0.1.16.0 the atomicity is possible, I added >> box.begin/commit/rollback. >> >> - for _, handler in ipairs(handlers) do >> + for _, handler in ipairs(schema_upgrade_handlers) do >> local next_version = handler.version >> if schema_compare_versions(next_version, version) > 0 then >> local next_version_str = table.concat(next_version, '.') >> log.info("Upgrade vshard to {%s}", next_version_str) >> - handler.func(username, password) >> + local ok, err = pcall(handler.func, username, password) >> + if not ok then >> + log.info("Couldn't upgrade to {%s}: %s", next_version_str, err) > > Seems that you've missed box.rollback here. And I think that it should be something like "if box.is_in_txn() then box.rollback() end" Yes, you are right. box.is_in_txn() is not needed here. box.rollback() is no-op in case there is no a transaction. ==================== diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index c0e768a..630d1ff 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -400,6 +400,7 @@ local function schema_upgrade_master(username, password) log.info("Upgrade vshard to {%s}", next_version_str) local ok, err = pcall(handler.func, username, password) if not ok then + box.rollback() log.info("Couldn't upgrade to {%s}: %s", next_version_str, err) error(err) end ==================== > I'm not sure but is it possible to drop last xlogs for user and start to use previous version of schema (and vshard version as well) as workaround in case of "fatal error"? Even if we will try to drop latest xlog files, it is not really a solution, because 1) These xlogs can contain not only upgrade changes. An xlog file could already exist at the moment when someone did reload with a newer version; 2) During xlog deletion there may appear another fatal error. Any unlink() or close() can fail; 3) Xlog may be already sent by replication; 4) It is not possible to rollback code of vshard to the previous version automatically. Someone needs to do it manually using code reload or restart. VShard does not have previous code version, so it can't checkout itself to an older version (except in tests, where I now that git is available, installed, and it is safe to checkout). In case you upgrade via restart, you should restart again with older version. In case you upgrade via reload, you should just stop reload, and put the old vshard.storage object back into _G. All should be safe in case upgrade is atomic. You can argue that we can do a snapshot before upgrade to solve the first problem, but snapshot is a disk killer if it is called automatically on several very fat instances on the same machine. I am here again trying to make things easier for the cloud team, who never restart instances unless it is unavoidable.
next prev parent reply other threads:[~2020-03-23 22:32 UTC|newest] Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-03-21 18:59 [Tarantool-patches] [PATCH vshard 0/2] vshard upgrade and _call Vladislav Shpilevoy 2020-03-21 18:59 ` [Tarantool-patches] [PATCH vshard 1/2] storage: introduce upgrade strategy Vladislav Shpilevoy 2020-03-22 5:05 ` Oleg Babin 2020-03-22 19:12 ` Vladislav Shpilevoy 2020-03-23 6:35 ` Oleg Babin 2020-03-23 22:32 ` Vladislav Shpilevoy [this message] 2020-03-24 4:32 ` Oleg Babin 2020-03-24 15:21 ` Yaroslav Dynnikov 2020-03-24 23:44 ` Vladislav Shpilevoy 2020-03-21 18:59 ` [Tarantool-patches] [PATCH vshard 2/2] storage: introduce vshard.storage._call() Vladislav Shpilevoy 2020-03-22 5:08 ` Oleg Babin 2020-03-22 19:13 ` Vladislav Shpilevoy 2020-03-23 6:42 ` Oleg Babin 2020-03-23 22:32 ` Vladislav Shpilevoy
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=955a5af8-bcfa-8b29-a808-e240e363033c@tarantool.org \ --to=v.shpilevoy@tarantool.org \ --cc=olegrok@tarantool.org \ --cc=tarantool-patches@dev.tarantool.org \ --cc=yaroslav.dynnikov@tarantool.org \ --subject='Re: [Tarantool-patches] [PATCH vshard 1/2] storage: introduce upgrade strategy' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox