From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 2A6046EC40; Mon, 16 Aug 2021 19:32:31 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2A6046EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1629131551; bh=eGWT+vKhqvkwRh5kM8Si+Ab7Rqu0uOY6xkFZWKjKDb0=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=bGTv092RalmPARvGmb/7lExCZNZDC5Kdi+Si5xjHcHjPmgPZ07np11OhGnSXpDZUx 9CG7bsoLUGqZ0nRNvkS1rjhNaQwac0isoIdkFqQeWvETJS3kkLklDmJ1Cq9i3Mb2BY H6Jq52/sTWjo1NkfsNRGeKA96t0quaQbV53+2sj0= Received: from smtp35.i.mail.ru (smtp35.i.mail.ru [94.100.177.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 06E556EC40 for ; Mon, 16 Aug 2021 19:32:30 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 06E556EC40 Received: by smtp35.i.mail.ru with esmtpa (envelope-from ) id 1mFfXI-0005qY-44; Mon, 16 Aug 2021 19:32:28 +0300 To: =?UTF-8?B?0JHQsNCx0LjQvSDQntC70LXQsw==?= References: <5d4f821905e21466ca25542f70c393c1057637fe.1628810253.git.sergepetrenko@tarantool.org> <1629119889.207698545@f480.i.mail.ru> Message-ID: Date: Mon, 16 Aug 2021 19:32:27 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <1629119889.207698545@f480.i.mail.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD9ECFD080E047A606F6525B29142351271182A05F538085040226637FAC20A00B8EBF1394CFEDA7DFD55363B1D983BEF8C0EA1E286D312F07E X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7F8E53417176C7207EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637543EE1E955FF04C98638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8434C3BC03E55FC0D02ED1BA1CFF1D946117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B6F7FD1A3A8AE6177F089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: 0D63561A33F958A5F1AA4D31BBE2DBA2238ABC9EBF562D3EB65DBAA05816C37DD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7567C209D01CC1E34B410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D343F22CE0A71FDB37E733573BE488D5EA32C30018009D7A3650A947DCD0F595A9754327D9C97ABE83C1D7E09C32AA3244CD1F94DDA32FAECD8B96EA92AC0E2D3507101BF96129E4011927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojIrFL/N5KnVEMqThuQ8s2/Q== X-Mailru-Sender: 3B9A0136629DC9125D61937A2360A446F730E2BA4F81D3A71074FD9ACA63E9FDF32926FFFE615266424AE0EB1F3D1D21E2978F233C3FAE6EE63DB1732555E4A8EE80603BA4A5B0BC112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v2 2/2] box: allow upgrading from version 1.6 X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Serge Petrenko via Tarantool-patches Reply-To: Serge Petrenko Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" 16.08.2021 16:18, Бабин Олег пишет: > Thanks for your patch. LGTM but consider several nits below. > Thanks for the review! Please, find my answers and incremental diff below. > > Пятница, 13 августа 2021, 2:35 +03:00 от Serge Petrenko via > Tarantool-patches : > Direct upgrade support from pre-1.7.5 versions was removed in commit > 7d3b80e78206c479cab75b4893629cfa1932252e > (Forbid upgrade from Tarantool < 1.7.5 and refactor upgrade.lua) > The reason for that was the mandatory space format checks introduced > back then. With these space format checks, old schema couldn't be > recovered on new Tarantool versions, because newer versions had > different system space formats. So old schema couldn't be upgraded > because it couldn't even be recovered. > > Actually this was rather inconvenient. One had to perform an extra > upgrade step when upgrading from, say, 1.6 to 2.x: instead of > performing a direct upgrade one had to do 1.6 -> 1.10 -> 2.x upgrade > which takes twice the time. > > Make it possible to boot from snapshots coming from Tarantool version > 1.6.8 and above. > > In order to do so, introduce before_replace triggers on system spaces, > which work during snapshot/xlog recovery. The triggers will set tuple > formats to the ones supported by current Tarantool (2.x). This way the > recovered data will have the correct format for a usual schema > upgrade. > > Also add upgrade_to_1_7_5() handler, which finishes transformation of > old schema to 1.7.5. The handler is fired together with other > box.schema.upgrade() handlers, so there's no user-visible behaviour > change. > > Side note: it would be great to use the same technique to allow > booting > from pre-1.6.8 snapshots. Unfortunately, this is not possible. > > Current triggers don't break the order of schema upgrades, so 1.7.1 > upgrades come before 1.7.2 and 1.7.5. This is because all the upgrades > in these versions are replacing existing tuples and not inserting new > ones, so the upgrades may be handled by the before_replace triggers. > > Upgrade to 1.6.8 requires inserting new tuples: creating sysviews, > like > _vspace, _vuser and so on. This can't be done from the before_replace > triggers, so we would have to run triggers for 1.7.x first which would > allow Tarantool to recover the snapshot, and then run an upgrade > handler for > 1.6.8. This looks really messy. > > Closes #5894 > --- >  src/box/lua/load_cfg.lua | 14 + >  src/box/lua/upgrade.lua | 276 +++++++++++- >  test/xlog/gh-5894-pre-1.7.7-upgrade.result | 400 ++++++++++++++++++ >  test/xlog/gh-5894-pre-1.7.7-upgrade.test.lua | 77 ++++ >  .../1.6.8/gh-5894-pre-1.7.7-upgrade/fill.lua | 1 + >  .../1.7.1/gh-5894-pre-1.7.7-upgrade/fill.lua | 1 + >  .../1.7.2/gh-5894-pre-1.7.7-upgrade/fill.lua | 1 + >  .../1.7.5/gh-5894-pre-1.7.7-upgrade/fill.lua | 1 + >  test/xlog/upgrade/fill.lua | 4 + >  9 files changed, 773 insertions(+), 2 deletions(-) >  create mode 100644 test/xlog/gh-5894-pre-1.7.7-upgrade.result >  create mode 100644 test/xlog/gh-5894-pre-1.7.7-upgrade.test.lua >  create mode 120000 > test/xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade/fill.lua >  create mode 120000 > test/xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade/fill.lua >  create mode 120000 > test/xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade/fill.lua >  create mode 120000 > test/xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade/fill.lua > > diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua > index 4df70c210..7a8cab3fd 100644 > --- a/src/box/lua/load_cfg.lua > +++ b/src/box/lua/load_cfg.lua > @@ -719,9 +719,23 @@ local function load_cfg(cfg) >              __call = locked(reload_cfg), >          }) > > + -- Check schema version of the snapshot we're about to recover, > if any. > + -- Some schema versions (below 1.7.5) are incompatible with > Tarantool 2.x > + -- When recovering from such an old snapshot, special recovery > triggers on > + -- system spaces are needed in order to be able to recover and > upgrade > + -- the schema then. > + local snap_dir = box.cfg.memtx_dir > + local snap_version = private.get_snapshot_version(snap_dir) > + if snap_version then > + private.set_recovery_triggers(snap_version) > + end > + >      -- This call either succeeds or calls panic() / exit(). >      private.cfg_load() > > + if snap_version then > + private.clear_recovery_triggers() > + end >      -- This block does not raise an error: all necessary checks >      -- already performed in private.cfg_check(). See >      -- comment. > diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua > index 6abce50f4..925adab18 100644 > --- a/src/box/lua/upgrade.lua > +++ b/src/box/lua/upgrade.lua > @@ -1,6 +1,8 @@ >  local log = require('log') >  local bit = require('bit') >  local json = require('json') > +local fio = require('fio') > +local xlog = require('xlog') > >  -- Guest user id - the default user >  local GUEST = 0 > @@ -86,6 +88,40 @@ local function set_system_triggers(val) >      foreach_system_space(function(s) s:run_triggers(val) end) >  end > > +-- Get schema version, stored in _schema system space, by reading > the latest > +-- snapshot file from the snap_dir. Useful to define > schema_version before > +-- recovering the snapshot, because some schema versions are too > old and cannot > +-- be recovered normally. > +local function get_snapshot_version(snap_dir) > + local snap_pattern = snap_dir..'/'..string.rep('[0-9]', 20)..'.snap' > > Probably we could use fio.pathjoin here Yep, sure. Changed. > + local snap_list = fio.glob(snap_pattern) > + table.sort(snap_list) > + local snap = snap_list[#snap_list] > + if not snap then > + return nil > + end > + local version = nil > + for _, row in xlog.pairs(snap) do > + local sid = row.BODY and row.BODY.space_id > + if sid == box.schema.SCHEMA_ID then > + local tuple = row.BODY.tuple > + if tuple and tuple[1] == 'version' then > + local major, minor, patch = tuple[2], tuple[3], tuple[4] or 0 > > Could it be replaced with tuple:unpack(2, 4)? Yep. Fixed as well. > + if major and minor and patch and type(major) == 'number' and > + type(minor) == 'number' and type(patch) == 'number' then > > Here you use type() == number checks. Could it be different? In case > of broken snap? Yep, that's a check for a broken snap. > So, we should log it I assume. The same for similar places below. Added logging. > + version = mkversion(major, minor, patch) > + break > + end > + end > + elseif sid and sid > box.schema.SCHEMA_ID then > + -- Exit early if version wasn't found in _schema space. > + -- Snapshot rows are ordered by space id. > + break > + end > + end > + return version > +end > + >  -------------------------------------------------------------------------------- >  -- Bootstrap >  -------------------------------------------------------------------------------- > @@ -131,6 +167,144 @@ local function create_sysview(source_id, > target_id) >      end >  end > > +-------------------------------------------------------------------------------- > +-- Tarantool 1.7.1 > +-------------------------------------------------------------------------------- > +local function user_trig_1_7_1(_, tuple) > + if tuple and tuple[3] == 'guest' and not tuple[5] then > > I think it’s better to use explicit check for tuple[5] value. > If it’s a boolean value could it be box.NULL? not box.NULL returns false, while not nil returns true. On the other hand, box.NULL == nil returns true. So I'd rather leave this check. Here's the full diff: ============================================ diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua index 925adab18..01e22aa7c 100644 --- a/src/box/lua/upgrade.lua +++ b/src/box/lua/upgrade.lua @@ -88,12 +88,23 @@ local function set_system_triggers(val)      foreach_system_space(function(s) s:run_triggers(val) end)  end +local function version_from_tuple(tuple) +    local major, minor, patch = tuple:unpack(2, 4) +    patch = patch or 0 +    if major and minor and type(major) == 'number' and +       type(minor) == 'number' and type(patch) == 'number' then +        return mkversion(major, minor, patch) +    end +    return nil +end +  -- Get schema version, stored in _schema system space, by reading the latest  -- snapshot file from the snap_dir. Useful to define schema_version before  -- recovering the snapshot, because some schema versions are too old and cannot  -- be recovered normally.  local function get_snapshot_version(snap_dir) -    local snap_pattern = snap_dir..'/'..string.rep('[0-9]', 20)..'.snap' +    local snap_pattern = fio.pathjoin(snap_dir, +                                      string.rep('[0-9]', 20)..'.snap')      local snap_list = fio.glob(snap_pattern)      table.sort(snap_list)      local snap = snap_list[#snap_list] @@ -106,12 +117,14 @@ local function get_snapshot_version(snap_dir)          if sid == box.schema.SCHEMA_ID then              local tuple = row.BODY.tuple              if tuple and tuple[1] == 'version' then -                local major, minor, patch = tuple[2], tuple[3], tuple[4] or 0 -                if major and minor and patch and type(major) == 'number' and -                   type(minor) == 'number' and type(patch) == 'number' then -                    version = mkversion(major, minor, patch) -                    break +                local major, minor, patch = tuple:unpack(2, 4) +                patch = patch or 0 +                version = version_from_tuple(tuple) +                if not version then +                    log.error("Corrupted version tuple in space '_schema' ".. +                              "in snapshot '%s': %s ", snap, tuple)                  end +                break              end          elseif sid and sid > box.schema.SCHEMA_ID then              -- Exit early if version wasn't found in _schema space. @@ -1284,10 +1297,8 @@ local recovery_triggers = {  -- snapshot), the triggers helping recover the old schema should be removed.  local function schema_trig_last(_, tuple)      if tuple and tuple[1] == 'version' then -        local major, minor, patch = tuple[2], tuple[3], tuple[4] or 0 -        if major and minor and patch and type(major) == 'number' and -           type(minor) == 'number' and type(patch) == 'number' then -            local version = mkversion(major, minor, patch) +        local version = version_from_tuple(tuple) +        if version then              log.info("Recovery trigger: recovered schema version %s. "..                       "Removing outdated recovery triggers.", version)              box.internal.clear_recovery_triggers(version) ============================================ > > + local auth_method_list = {} > + auth_method_list["chap-sha1"] = box.schema.user.password("") > + tuple = tuple:update{{'=', 5, auth_method_list}} > + log.info("Set empty password to user 'guest'") > + end > + return tuple > +end > + > +-------------------------------------------------------------------------------- > +-- Tarantool 1.7.2 > +-------------------------------------------------------------------------------- > +local function index_trig_1_7_2(_, tuple) > + local field_types_v16 = { > + num = 'unsigned', > + int = 'integer', > + str = 'string', > + } > + if not tuple then > + return tuple > + end > + local parts = tuple[6] > + local changed = false > + for _, part in pairs(parts) do > + local field_type = part[2]:lower() > + if field_types_v16[field_type] ~= nil then > + part[2] = field_types_v16[field_type] > + changed = true > + end > + end > + if changed then > + log.info("Update index '%s' on space '%s': set parts to %s", > tuple[3], > + box.space[tuple[1]].name, json.encode(parts)) > + tuple = tuple:update{{'=', 6, parts}} > + end > + return tuple > +end > + > +-------------------------------------------------------------------------------- > +-- Tarantool 1.7.5 > +-------------------------------------------------------------------------------- > +local function create_truncate_space() > + local _truncate = box.space[box.schema.TRUNCATE_ID] > + > + log.info("create space _truncate") > + box.space._space:insert{ > + _truncate.id, ADMIN, '_truncate', 'memtx', 0, setmap({}), > + {{name = 'id', type = 'unsigned'}, {name = 'count', type = > 'unsigned'}} > + } > + > + log.info("create index primary on _truncate") > + box.space._index:insert{ > + _truncate.id, 0, 'primary', 'tree', {unique = true}, {{0, > 'unsigned'}} > + } > + > + local _priv = box.space[box.schema.PRIV_ID] > + _priv:insert{ADMIN, PUBLIC, 'space', _truncate.id, box.priv.W} > +end > + > +local function upgrade_to_1_7_5() > + create_truncate_space() > +end > + > +local function user_trig_1_7_5(_, tuple) > + if tuple and not tuple[5] then > + tuple = tuple:update{{'=', 5, setmap({})}} > + log.info("Set empty password to %s '%s'", tuple[4], tuple[3]) > + end > + return tuple > +end > + > +local space_formats_1_7_5 = { > + _schema = { > + {name = 'key', type = 'string'}, > + }, > + _space = { > + {name = 'id', type = 'unsigned'}, > + {name = 'owner', type = 'unsigned'}, > + {name = 'name', type = 'string'}, > + {name = 'engine', type = 'string'}, > + {name = 'field_count', type = 'unsigned'}, > + {name = 'flags', type = 'map'}, > + {name = 'format', type = 'array'}, > + }, > + _index = { > + {name = 'id', type = 'unsigned'}, > + {name = 'iid', type = 'unsigned'}, > + {name = 'name', type = 'string'}, > + {name = 'type', type = 'string'}, > + {name = 'opts', type = 'map'}, > + {name = 'parts', type = 'array'}, > + }, > + _func = { > + {name = 'id', type = 'unsigned'}, > + {name = 'owner', type = 'unsigned'}, > + {name = 'name', type = 'string'}, > + {name = 'setuid', type = 'unsigned'}, > + }, > + _user = { > + {name = 'id', type = 'unsigned'}, > + {name = 'owner', type = 'unsigned'}, > + {name = 'name', type = 'string'}, > + {name = 'type', type = 'string'}, > + {name = 'auth', type = 'map'}, > + }, > + _priv = { > + {name = 'grantor', type = 'unsigned'}, > + {name = 'grantee', type = 'unsigned'}, > + {name = 'object_type', type = 'string'}, > + {name = 'object_id', type = 'unsigned'}, > + {name = 'privilege', type = 'unsigned'}, > + }, > + _cluster = { > + {name = 'id', type = 'unsigned'}, > + {name = 'uuid', type = 'string'}, > + }, > +} > + > +space_formats_1_7_5._vspace = space_formats_1_7_5._space > +space_formats_1_7_5._vindex = space_formats_1_7_5._index > +space_formats_1_7_5._vfunc = space_formats_1_7_5._func > +space_formats_1_7_5._vuser = space_formats_1_7_5._user > +space_formats_1_7_5._vpriv = space_formats_1_7_5._priv > + > +local function space_trig_1_7_5(_, tuple) > + if tuple and space_formats_1_7_5[tuple[3]] and > + not table.equals(space_formats_1_7_5[tuple[3]], tuple[7]) then > + tuple = tuple:update{{'=', 7, space_formats_1_7_5[tuple[3]]}} > + log.info("Update space '%s' format: new format %s", tuple[3], > + json.encode(tuple[7])) > + end > + return tuple > +end > + >  local function initial_1_7_5() >      -- stick to the following convention: >      -- prefer user id (owner id) in field #1 > @@ -452,6 +626,15 @@ local function upgrade_to_1_7_7() >      _priv:replace({ADMIN, SUPER, 'universe', 0, 4294967295}) >  end > > +local function priv_trig_1_7_7(_, tuple) > + if tuple and tuple[2] == ADMIN and tuple[3] == 'universe' and > + tuple[5] ~= box.priv.ALL then > + tuple = tuple:update{{'=', 5, box.priv.ALL}} > + log.info("Grant all privileges to user 'admin'") > + end > + return tuple > +end > + >  -------------------------------------------------------------------------------- >  --- Tarantool 1.10.0 >  -------------------------------------------------------------------------------- > @@ -1021,6 +1204,7 @@ end >  -------------------------------------------------------------------------------- > >  local handlers = { > + {version = mkversion(1, 7, 5), func = upgrade_to_1_7_5, auto=true}, >      {version = mkversion(1, 7, 6), func = upgrade_to_1_7_6, auto > = true}, >      {version = mkversion(1, 7, 7), func = upgrade_to_1_7_7, auto > = true}, >      {version = mkversion(1, 10, 0), func = upgrade_to_1_10_0, > auto = true}, > @@ -1061,13 +1245,98 @@ local function schema_needs_upgrade() >      return false >  end > > +local trig_oldest_version = nil > + > +-- Some schema changes before version 1.7.7 make it impossible to > recover from > +-- older snapshot. The table below consists of before_replace > triggers on system > +-- spaces, which make old snapshot schema compatible with current > Tarantool > +-- (version 2.x). The triggers replace old format tuples with new > ones > +-- in-memory, thus making it possible to recover from a rather > old snapshot > +-- (up to schema version 1.6.8). Once the snapshot is recovered, > a normal > +-- upgrade procedure may set schema version to the latest one. > +-- > +-- The triggers mostly repeat the corresponding upgrade_to_1_7_x > functions, > +-- which were used when pre-1.7.x snapshot schema was still > recoverable. > +-- > +-- When the triggers are used (i.e. when snapshot schema version > is below 1.7.5, > +-- the upgrade procedure works as follows: > +-- * first the snapshot is recovered and 1.7.5-compatible schema > is applied to > +-- it in-memory with the help of triggers. > +-- * then usual upgrade_to_X_X_X() handlers may be fired to turn > schema into the > +-- latest one. > +local recovery_triggers = { > + {version = mkversion(1, 7, 1), tbl = { > + _user = user_trig_1_7_1, > + }}, > + {version = mkversion(1, 7, 2), tbl = { > + _index = index_trig_1_7_2, > + }}, > + {version = mkversion(1, 7, 5), tbl = { > + _space = space_trig_1_7_5, > + _user = user_trig_1_7_5, > + }}, > + {version = mkversion(1, 7, 7), tbl = { > + _priv = priv_trig_1_7_7, > + }}, > +} > + > +-- Once newer schema version is recovered (say, from an xlog > following the old > +-- snapshot), the triggers helping recover the old schema should > be removed. > +local function schema_trig_last(_, tuple) > + if tuple and tuple[1] == 'version' then > + local major, minor, patch = tuple[2], tuple[3], tuple[4] or 0 > + if major and minor and patch and type(major) == 'number' and > + type(minor) == 'number' and type(patch) == 'number' then > + local version = mkversion(major, minor, patch) > + log.info("Recovery trigger: recovered schema version %s. ".. > + "Removing outdated recovery triggers.", version) > + box.internal.clear_recovery_triggers(version) > + trig_oldest_version = version > + end > + end > + return tuple > +end > + > +recovery_triggers[#recovery_triggers].tbl['_schema'] = > schema_trig_last > + > +local function on_init_set_recovery_triggers() > + log.info("Recovering snapshot with schema version %s", > trig_oldest_version) > + for _, trig_tbl in ipairs(recovery_triggers) do > + if trig_tbl.version > trig_oldest_version then > + for space, trig in pairs(trig_tbl.tbl) do > + box.space[space]:before_replace(trig) > + log.info("Set recovery trigger on space '%s' to comply with ".. > + "version %s format", space, trig_tbl.version) > + end > + end > + end > +end > + > +local function set_recovery_triggers(version) > + trig_oldest_version = version > + box.ctl.on_schema_init(on_init_set_recovery_triggers) > +end > + > +local function clear_recovery_triggers(version) > + for _, trig_tbl in ipairs(recovery_triggers) do > + if trig_tbl.version > trig_oldest_version and > + (not version or trig_tbl.version <= version) then > + for space, trig in pairs(trig_tbl.tbl) do > + box.space[space]:before_replace(nil, trig) > + log.info("Remove recovery trigger on space '%s' for version %s", > + space, trig_tbl.version) > + end > + end > + end > +end > + >  local function upgrade(options) >      options = options or {} >      setmetatable(options, {__index = {auto = false}}) > >      local version = get_version() > - if version < mkversion(1, 7, 5) then > - log.warn('can upgrade from 1.7.5 only') > + if version < mkversion(1, 6, 8) then > + log.warn('can upgrade from 1.6.8 only') >          return >      end > > @@ -1110,3 +1379,6 @@ end >  box.schema.upgrade = upgrade; >  box.internal.bootstrap = bootstrap; >  box.internal.schema_needs_upgrade = schema_needs_upgrade; > +box.internal.get_snapshot_version = get_snapshot_version; > +box.internal.set_recovery_triggers = set_recovery_triggers; > +box.internal.clear_recovery_triggers = clear_recovery_triggers; > diff --git a/test/xlog/gh-5894-pre-1.7.7-upgrade.result > b/test/xlog/gh-5894-pre-1.7.7-upgrade.result > new file mode 100644 > index 000000000..aba5a56ab > --- /dev/null > +++ b/test/xlog/gh-5894-pre-1.7.7-upgrade.result > @@ -0,0 +1,400 @@ > +-- test-run result file version 2 > +test_run = require('test_run').new() > + | --- > + | ... > + > +-- Upgrade from 1.6.8. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade"') > + | --- > + | - true > + | ... > +test_run:cmd('start server upgrade') > + | --- > + | - true > + | ... > +test_run:switch('upgrade') > + | --- > + | - true > + | ... > + > +assert(not box.internal.schema_needs_upgrade()) > + | --- > + | - true > + | ... > +box.space.distro:select{} > + | --- > + | - - ['debian', 'sarge', 31, 1118059200] > + | - ['debian', 'etch', 40, 1176033600] > + | - ['ubuntu', 'trusty', 1404, 1397736000] > + | - ['ubuntu', 'vivid', 1504, 1429790400] > + | - ['ubuntu', 'wily', 1510, 1445515200] > + | - ['debian', 'wheezy', 70, 1367668800] > + | - ['debian', 'squeeze', 60, 1296907200] > + | - ['debian', 'lenny', 50, 1234612800] > + | - ['debian', 'jessie', 80, 1430049600] > + | - ['ubuntu', 'precise', 1510, 1335441600] > + | - ['debian', 'woody', 30, 1027080000] > + | ... > +box.space._index:select{box.space.distro.id} > + | --- > + | - - [512, 0, 'primary', 'hash', {'unique': true}, [[0, > 'string'], [1, 'string'], [ > + | 2, 'unsigned']]] > + | - [512, 1, 'codename', 'hash', {'unique': true}, [[1, 'string']]] > + | - [512, 2, 'time', 'tree', {'unique': false}, [[3, 'unsigned']]] > + | ... > +box.space._space:format() > + | --- > + | - [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', > 'type': 'unsigned'}, {'name': 'name', > + | 'type': 'string'}, {'name': 'engine', 'type': 'string'}, > {'name': 'field_count', > + | 'type': 'unsigned'}, {'name': 'flags', 'type': 'map'}, > {'name': 'format', 'type': 'array'}] > + | ... > +box.schema.user.info('admin') > + | --- > + | - - - > read,write,execute,session,usage,create,drop,alter,reference,trigger,insert,update,delete > + | - universe > + | - > + | ... > +box.schema.user.info('guest') > + | --- > + | - - - execute > + | - role > + | - public > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.user.info('someuser') > + | --- > + | - - - execute > + | - function > + | - someotherfunc > + | - - execute > + | - role > + | - public > + | - - execute > + | - role > + | - somerole > + | - - read,write,drop,alter > + | - space > + | - temporary > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.role.info('somerole') > + | --- > + | - - - read,write,drop,alter > + | - space > + | - distro > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server upgrade') > + | --- > + | - true > + | ... > +test_run:cmd('delete server upgrade') > + | --- > + | - true > + | ... > + > +-- Upgrade from 1.7.1. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade"') > + | --- > + | - true > + | ... > +test_run:cmd('start server upgrade') > + | --- > + | - true > + | ... > +test_run:switch('upgrade') > + | --- > + | - true > + | ... > + > +assert(not box.internal.schema_needs_upgrade()) > + | --- > + | - true > + | ... > +box.space.distro:select{} > + | --- > + | - - ['debian', 'etch', 40, 1176033600] > + | - ['debian', 'sarge', 31, 1118059200] > + | - ['ubuntu', 'wily', 1510, 1445515200] > + | - ['ubuntu', 'trusty', 1404, 1397736000] > + | - ['ubuntu', 'vivid', 1504, 1429790400] > + | - ['debian', 'wheezy', 70, 1367668800] > + | - ['debian', 'squeeze', 60, 1296907200] > + | - ['debian', 'lenny', 50, 1234612800] > + | - ['debian', 'jessie', 80, 1430049600] > + | - ['ubuntu', 'precise', 1510, 1335441600] > + | - ['debian', 'woody', 30, 1027080000] > + | ... > +box.space._index:select{box.space.distro.id} > + | --- > + | - - [512, 0, 'primary', 'hash', {'unique': true}, [[0, > 'string'], [1, 'string'], [ > + | 2, 'unsigned']]] > + | - [512, 1, 'codename', 'hash', {'unique': true}, [[1, 'string']]] > + | - [512, 2, 'time', 'tree', {'unique': false}, [[3, 'unsigned']]] > + | ... > +box.space._space:format() > + | --- > + | - [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', > 'type': 'unsigned'}, {'name': 'name', > + | 'type': 'string'}, {'name': 'engine', 'type': 'string'}, > {'name': 'field_count', > + | 'type': 'unsigned'}, {'name': 'flags', 'type': 'map'}, > {'name': 'format', 'type': 'array'}] > + | ... > +box.schema.user.info('admin') > + | --- > + | - - - > read,write,execute,session,usage,create,drop,alter,reference,trigger,insert,update,delete > + | - universe > + | - > + | ... > +box.schema.user.info('guest') > + | --- > + | - - - execute > + | - role > + | - public > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.user.info('someuser') > + | --- > + | - - - execute > + | - function > + | - someotherfunc > + | - - execute > + | - role > + | - public > + | - - execute > + | - role > + | - somerole > + | - - read,write,drop,alter > + | - space > + | - temporary > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.role.info('somerole') > + | --- > + | - - - read,write,drop,alter > + | - space > + | - distro > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server upgrade') > + | --- > + | - true > + | ... > +test_run:cmd('delete server upgrade') > + | --- > + | - true > + | ... > + > +-- Upgrade from 1.7.2. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade"') > + | --- > + | - true > + | ... > +test_run:cmd('start server upgrade') > + | --- > + | - true > + | ... > +test_run:switch('upgrade') > + | --- > + | - true > + | ... > + > +assert(not box.internal.schema_needs_upgrade()) > + | --- > + | - true > + | ... > +box.space.distro:select{} > + | --- > + | - - ['debian', 'sarge', 31, 1118059200] > + | - ['debian', 'etch', 40, 1176033600] > + | - ['ubuntu', 'trusty', 1404, 1397736000] > + | - ['ubuntu', 'vivid', 1504, 1429790400] > + | - ['debian', 'lenny', 50, 1234612800] > + | - ['debian', 'wheezy', 70, 1367668800] > + | - ['debian', 'squeeze', 60, 1296907200] > + | - ['ubuntu', 'wily', 1510, 1445515200] > + | - ['debian', 'jessie', 80, 1430049600] > + | - ['ubuntu', 'precise', 1510, 1335441600] > + | - ['debian', 'woody', 30, 1027080000] > + | ... > +box.space._index:select{box.space.distro.id} > + | --- > + | - - [512, 0, 'primary', 'hash', {'unique': true}, [[0, > 'string'], [1, 'string'], [ > + | 2, 'unsigned']]] > + | - [512, 1, 'codename', 'hash', {'unique': true}, [[1, 'string']]] > + | - [512, 2, 'time', 'tree', {'unique': false}, [[3, 'unsigned']]] > + | ... > +box.space._space:format() > + | --- > + | - [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', > 'type': 'unsigned'}, {'name': 'name', > + | 'type': 'string'}, {'name': 'engine', 'type': 'string'}, > {'name': 'field_count', > + | 'type': 'unsigned'}, {'name': 'flags', 'type': 'map'}, > {'name': 'format', 'type': 'array'}] > + | ... > +box.schema.user.info('admin') > + | --- > + | - - - > read,write,execute,session,usage,create,drop,alter,reference,trigger,insert,update,delete > + | - universe > + | - > + | ... > +box.schema.user.info('guest') > + | --- > + | - - - execute > + | - role > + | - public > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.user.info('someuser') > + | --- > + | - - - execute > + | - function > + | - someotherfunc > + | - - execute > + | - role > + | - public > + | - - execute > + | - role > + | - somerole > + | - - read,write,drop,alter > + | - space > + | - temporary > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.role.info('somerole') > + | --- > + | - - - read,write,drop,alter > + | - space > + | - distro > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server upgrade') > + | --- > + | - true > + | ... > +test_run:cmd('delete server upgrade') > + | --- > + | - true > + | ... > + > +-- Upgrade from 1.7.5. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade"') > + | --- > + | - true > + | ... > +test_run:cmd('start server upgrade') > + | --- > + | - true > + | ... > +test_run:switch('upgrade') > + | --- > + | - true > + | ... > + > +assert(not box.internal.schema_needs_upgrade()) > + | --- > + | - true > + | ... > +box.space.distro:select{} > + | --- > + | - - ['debian', 'etch', 40, 1176033600] > + | - ['debian', 'sarge', 31, 1118059200] > + | - ['debian', 'lenny', 50, 1234612800] > + | - ['ubuntu', 'trusty', 1404, 1397736000] > + | - ['ubuntu', 'vivid', 1504, 1429790400] > + | - ['debian', 'wheezy', 70, 1367668800] > + | - ['debian', 'squeeze', 60, 1296907200] > + | - ['ubuntu', 'wily', 1510, 1445515200] > + | - ['debian', 'jessie', 80, 1430049600] > + | - ['ubuntu', 'precise', 1510, 1335441600] > + | - ['debian', 'woody', 30, 1027080000] > + | ... > +box.space._index:select{box.space.distro.id} > + | --- > + | - - [512, 0, 'primary', 'hash', {'unique': true}, [[0, > 'string'], [1, 'string'], [ > + | 2, 'unsigned']]] > + | - [512, 1, 'codename', 'hash', {'unique': true}, [[1, 'string']]] > + | - [512, 2, 'time', 'tree', {'unique': false}, [[3, 'unsigned']]] > + | ... > +box.space._space:format() > + | --- > + | - [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', > 'type': 'unsigned'}, {'name': 'name', > + | 'type': 'string'}, {'name': 'engine', 'type': 'string'}, > {'name': 'field_count', > + | 'type': 'unsigned'}, {'name': 'flags', 'type': 'map'}, > {'name': 'format', 'type': 'array'}] > + | ... > +box.schema.user.info('admin') > + | --- > + | - - - > read,write,execute,session,usage,create,drop,alter,reference,trigger,insert,update,delete > + | - universe > + | - > + | ... > +box.schema.user.info('guest') > + | --- > + | - - - execute > + | - role > + | - public > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.user.info('someuser') > + | --- > + | - - - execute > + | - function > + | - someotherfunc > + | - - execute > + | - role > + | - public > + | - - execute > + | - role > + | - somerole > + | - - read,write,drop,alter > + | - space > + | - temporary > + | - - session,usage > + | - universe > + | - > + | ... > +box.schema.role.info('somerole') > + | --- > + | - - - read,write,drop,alter > + | - space > + | - distro > + | ... > + > +test_run:switch('default') > + | --- > + | - true > + | ... > +test_run:cmd('stop server upgrade') > + | --- > + | - true > + | ... > +test_run:cmd('delete server upgrade') > + | --- > + | - true > + | ... > diff --git a/test/xlog/gh-5894-pre-1.7.7-upgrade.test.lua > b/test/xlog/gh-5894-pre-1.7.7-upgrade.test.lua > new file mode 100644 > index 000000000..9096bcb7a > --- /dev/null > +++ b/test/xlog/gh-5894-pre-1.7.7-upgrade.test.lua > @@ -0,0 +1,77 @@ > +test_run = require('test_run').new() > + > +-- Upgrade from 1.6.8. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade"') > +test_run:cmd('start server upgrade') > +test_run:switch('upgrade') > + > +assert(not box.internal.schema_needs_upgrade()) > +box.space.distro:select{} > +box.space._index:select{box.space.distro.id} > +box.space._space:format() > +box.schema.user.info('admin') > +box.schema.user.info('guest') > +box.schema.user.info('someuser') > +box.schema.role.info('somerole') > + > +test_run:switch('default') > +test_run:cmd('stop server upgrade') > +test_run:cmd('delete server upgrade') > + > +-- Upgrade from 1.7.1. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade"') > +test_run:cmd('start server upgrade') > +test_run:switch('upgrade') > + > +assert(not box.internal.schema_needs_upgrade()) > +box.space.distro:select{} > +box.space._index:select{box.space.distro.id} > +box.space._space:format() > +box.schema.user.info('admin') > +box.schema.user.info('guest') > +box.schema.user.info('someuser') > +box.schema.role.info('somerole') > + > +test_run:switch('default') > +test_run:cmd('stop server upgrade') > +test_run:cmd('delete server upgrade') > + > +-- Upgrade from 1.7.2. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade"') > +test_run:cmd('start server upgrade') > +test_run:switch('upgrade') > + > +assert(not box.internal.schema_needs_upgrade()) > +box.space.distro:select{} > +box.space._index:select{box.space.distro.id} > +box.space._space:format() > +box.schema.user.info('admin') > +box.schema.user.info('guest') > +box.schema.user.info('someuser') > +box.schema.role.info('somerole') > + > +test_run:switch('default') > +test_run:cmd('stop server upgrade') > +test_run:cmd('delete server upgrade') > + > +-- Upgrade from 1.7.5. > +test_run:cmd('create server upgrade with script="xlog/upgrade.lua", \ > + workdir="xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade"') > +test_run:cmd('start server upgrade') > +test_run:switch('upgrade') > + > +assert(not box.internal.schema_needs_upgrade()) > +box.space.distro:select{} > +box.space._index:select{box.space.distro.id} > +box.space._space:format() > +box.schema.user.info('admin') > +box.schema.user.info('guest') > +box.schema.user.info('someuser') > +box.schema.role.info('somerole') > + > +test_run:switch('default') > +test_run:cmd('stop server upgrade') > +test_run:cmd('delete server upgrade') > diff --git > a/test/xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade/fill.lua > b/test/xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade/fill.lua > new file mode 120000 > index 000000000..2f2a84962 > --- /dev/null > +++ b/test/xlog/upgrade/1.6.8/gh-5894-pre-1.7.7-upgrade/fill.lua > @@ -0,0 +1 @@ > +../../fill.lua > \ No newline at end of file > diff --git > a/test/xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade/fill.lua > b/test/xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade/fill.lua > new file mode 120000 > index 000000000..2f2a84962 > --- /dev/null > +++ b/test/xlog/upgrade/1.7.1/gh-5894-pre-1.7.7-upgrade/fill.lua > @@ -0,0 +1 @@ > +../../fill.lua > \ No newline at end of file > diff --git > a/test/xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade/fill.lua > b/test/xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade/fill.lua > new file mode 120000 > index 000000000..2f2a84962 > --- /dev/null > +++ b/test/xlog/upgrade/1.7.2/gh-5894-pre-1.7.7-upgrade/fill.lua > @@ -0,0 +1 @@ > +../../fill.lua > \ No newline at end of file > diff --git > a/test/xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade/fill.lua > b/test/xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade/fill.lua > new file mode 120000 > index 000000000..2f2a84962 > --- /dev/null > +++ b/test/xlog/upgrade/1.7.5/gh-5894-pre-1.7.7-upgrade/fill.lua > @@ -0,0 +1 @@ > +../../fill.lua > \ No newline at end of file > diff --git a/test/xlog/upgrade/fill.lua b/test/xlog/upgrade/fill.lua > index 0ef1a8bb9..310c1ca72 100644 > --- a/test/xlog/upgrade/fill.lua > +++ b/test/xlog/upgrade/fill.lua > @@ -56,4 +56,8 @@ end >  box.schema.func.create('someotherfunc') >  box.schema.user.grant('someuser', 'execute', 'function', > 'someotherfunc') >  box.schema.user.grant('someuser', 'read,write', 'space', 'temporary') > + > +box.schema.upgrade() > +box.snapshot() > + >  os.exit(0) > -- > 2.30.1 (Apple Git-130) > > -- > Oleg Babin -- Serge Petrenko