From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 421F46EC40; Tue, 10 Aug 2021 17:43:52 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 421F46EC40 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1628606632; bh=nbA2VDJg/6cMYXZ/EUKwYYRliIo6V0bqUMSdZYT+pqk=; h=In-Reply-To:Date:References:To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=rkJCjBDw8Ufd00evbdxpS1VD/G4nv4xH/8eGkIHQlnRhD3XxT7f1BSh6VnVIVgd2I 7MEliC2MdrvX9gdGV8vHXckjXUaXA2AX9AUANfS6WVb74Ty5ICl0fNXL2G0KWHc0zg jTQUiV7xT7funcL0hPCAwnVM/BWQMRPaOLRhFfpU= Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 290926EC40 for ; Tue, 10 Aug 2021 17:43:51 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 290926EC40 Received: by smtp37.i.mail.ru with esmtpa (envelope-from ) id 1mDSyr-00049E-Qr; Tue, 10 Aug 2021 17:43:50 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) In-Reply-To: <20210810112836.48775-1-sergepetrenko@tarantool.org> Date: Tue, 10 Aug 2021 17:43:48 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20210810112836.48775-1-sergepetrenko@tarantool.org> To: Serge Petrenko X-Mailer: Apple Mail (2.3654.100.0.2.22) X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD972FF4A7D76DB5E242D14FEF1BD8BF4AC182A05F53808504026F1244FC6D8CCF16B9140F90CF0CC46411D4E95E80BA805418948D07D4957DC X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE737BB76880A4CA9A4EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006373745FD4183B699148638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8AF72D20C6F858DB918C7FD2ED0E971CB117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC974A882099E279BDA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD18F04B652EEC242312D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B67393CE827C55B5F775ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A58BFD401F24634DB6293F817F971EE57D7365F462A7E1C598D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA753177526CD55AFC11410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34AC223B6CC5B2DFACBD14200049603E7411950BF7079B3075E81B9DA312FBC6F980B631CE64EE82571D7E09C32AA3244CE2E570745820CC2FFC7B321454E91D313A76366E8A9DE7CAFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj6qlzQV0oSZN6KAbdVhXc4Q== X-Mailru-Sender: 3B9A0136629DC912F4AABCEFC589C81E61C4A9B5E8C463813BB024ED4D9D51E433E57416D9AF418FAD07DD1419AC565FA614486B47F28B67C5E079CCF3B0523AED31B7EB2E253A9E112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] box: allow upgrading from version 1.6 X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Ostanevich via Tarantool-patches Reply-To: Sergey Ostanevich Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the patch! See my 7 comments below Sergos 1. I don=E2=80=99t see any doc references to=20 = https://www.tarantool.io/en/doc/latest/book/admin/upgrades/#admin-upgrades= where obviously some changes are needed: =E2=80=9CMeanwhile Tarantool = 2.x may have=20 incompatible changes when migrating from Tarantool 1.6. to 2.x = directly." > On 10 Aug 2021, at 14:28, Serge Petrenko = wrote: >=20 > Direct upgrade support from pre-1.7.5 versions was removed in commit > 7d3b80e78206c479cab75b4893629cfa1932252e > (Forbid upgrade from Tarantool < 1.7.5 and refactor upgrade.lua) > The reason for that was the mandatory space format checks introduced > back then. Later on, more incompatible changes were made, and due to = these > changes, Tarantool failed to recover from pre-1.7.5 snapshots. >=20 > Actually this is rather inconvenient. Now one has to perform an extra > upgrade step when upgrading from, say, 1.6 to 2.x: instead of = performing a > direct upgrade one has to do 1.6 -> 1.10 -> 2.x upgrade which takes > twice the time. >=20 2. I didn=E2=80=99t get the part before this line - if it part of the = commit message? I don=E2=80=99t think we need that deep explanation, partially covered = later in description how the issue is addressed.=20 > Make it possible to boot from snapshots coming from Tarantool version = 1.6.8 and > above. >=20 3. Not sure if it=E2=80=99s too long, but it looks like. > In order to do so, introduce before_replace triggers on system spaces, > which work during snapshot/xlog recovery. The triggers will set tuple > formats to the ones supported by current Tarantool (2.x). This way the > recovered data will have the correct format for a usual schema = upgrade. >=20 > Also add upgrade_to_1_7_5() handler, which finishes transformation of > old schema to 1.7.5. The handler is fired together with other > box.schema.upgrade() handlers, so there's no user-visible behaviour > change. >=20 > Side note: it would be great to use the same technique to allow = booting > from pre-1.6.8 snapshots. Unfortunately, this is not possible. >=20 > Current triggers don't break the order of schema upgrades, so 1.7.1 > upgrades come before 1.7.2 and 1.7.5. This is because all the upgrades > in these versions are replacing existing tuples and not inserting new > ones, so the upgrades may be handled by the before_replace triggers. >=20 > Upgrade to 1.6.8 requires inserting new tuples: creating sysviews, = like > _vspace, _vuser and so on. This can't be done from the before_replace > triggers, so we would have to run triggers for 1.7.x first which would > allow Tarantool to recover the snapshot, and then run an upgrade = handler for > 1.6.8. This looks really messy. >=20 > Closes #5894 > --- > https://github.com/tarantool/tarantool/issues/5894 > https://github.com/tarantool/tarantool/tree/sp/gh-5894-1.6-upgrade > src/box/lua/load_cfg.lua | 9 ++ > src/box/lua/upgrade.lua | 251 ++++++++++++++++++++++++++++++++++++++- > 2 files changed, 258 insertions(+), 2 deletions(-) >=20 > diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua > index 4df70c210..ea73337b7 100644 > --- a/src/box/lua/load_cfg.lua > +++ b/src/box/lua/load_cfg.lua > @@ -695,6 +695,12 @@ local function load_cfg(cfg) > return box.error() -- re-throw exception from check_cfg() > end >=20 > + local snap_dir =3D box.cfg.memtx_dir 4. It should be covered with comment on the matter - why is it memtx = only, what versions are expected here, what triggers will do. > + local snap_version =3D private.get_snapshot_version(snap_dir) > + if snap_version and private.need_recovery_triggers(snap_version) = then > + private.set_recovery_triggers(snap_version) > + end > + > -- NB: After this point the function should not raise an > -- error. > -- > @@ -722,6 +728,9 @@ local function load_cfg(cfg) > -- This call either succeeds or calls panic() / exit(). > private.cfg_load() >=20 > + if snap_version and private.need_recovery_triggers(snap_version) = then > + private.reset_recovery_triggers() > + end 5. So, after this point the formats will be up to 2.x. If we make a = snapshot at this moment, the new snap file will become incompatible with 1.6.8? If an 2.x instance will bootstrap from this snap, will it be able to be a replica in 1.6.8 cluster?=20 > -- This block does not raise an error: all necessary checks > -- already performed in private.cfg_check(). See > -- comment. > diff --git a/src/box/lua/upgrade.lua b/src/box/lua/upgrade.lua > index 97afc0b4d..c1f3a027a 100644 > --- a/src/box/lua/upgrade.lua > +++ b/src/box/lua/upgrade.lua > @@ -1,6 +1,8 @@ > local log =3D require('log') > local bit =3D require('bit') > local json =3D require('json') > +local fio =3D require('fio') > +local xlog =3D require('xlog') >=20 > -- Guest user id - the default user > local GUEST =3D 0 > @@ -86,6 +88,41 @@ local function set_system_triggers(val) > foreach_system_space(function(s) s:run_triggers(val) end) > end >=20 > +-- Get schema version, stored in _schema system space, by reading the = latest > +-- snapshot file from the snap_dir. Useful to define schema_version = before > +-- recovering the snapshot, because some schema versions are too old = and cannot > +-- be recovered normally. > +local function get_snapshot_version(snap_dir) > + local SCHEMA_ID =3D 272 6. Can we reuse the global definition at schema_def.h:72 ? > + local snap_pattern =3D snap_dir..'/'..string.rep('[0-9]', = 20)..'.snap' > + local snap_list =3D fio.glob(snap_pattern) > + table.sort(snap_list) > + local snap =3D snap_list[#snap_list] > + if snap =3D=3D nil then > + return nil > + end > + local version =3D nil > + for _, row in xlog.pairs(snap) do > + local sid =3D row.BODY and row.BODY.space_id > + if sid =3D=3D SCHEMA_ID then > + local tuple =3D row.BODY.tuple > + if tuple and tuple[1] =3D=3D 'version' then > + local major, minor, patch =3D tuple[2], tuple[3], = tuple[4] or 0 > + if major and minor and patch and type(major) =3D=3D = 'number' and > + type(minor) =3D=3D 'number' and type(patch) =3D=3D = 'number' then > + version =3D mkversion(major, minor, patch) > + break > + end > + end > + elseif sid and sid > SCHEMA_ID then > + -- Exit early if version wasn't found in _schema space. > + -- Snapshot rows are ordered by space id. > + break > + end > + end > + return version > +end > + > = --------------------------------------------------------------------------= ------ > -- Bootstrap > = --------------------------------------------------------------------------= ------ > @@ -131,6 +168,135 @@ local function create_sysview(source_id, = target_id) > end > end >=20 > = +-------------------------------------------------------------------------= ------- > +-- Tarantool 1.7.1 > = +-------------------------------------------------------------------------= ------- > +local function user_trig_1_7_1(_, tuple) > + if tuple[3] =3D=3D 'guest' and not tuple[5] then > + local auth_method_list =3D {} > + auth_method_list["chap-sha1"] =3D = box.schema.user.password("") > + tuple =3D tuple:update{{'=3D', 5, auth_method_list}} > + log.info("Set empty password to user 'guest'") > + end > + return tuple > +end > + > = +-------------------------------------------------------------------------= ------- > +-- Tarantool 1.7.2 > = +-------------------------------------------------------------------------= ------- > +local function index_trig_1_7_2(_, tuple) > + local field_types_v16 =3D { > + num =3D 'unsigned', > + int =3D 'integer', > + str =3D 'string', > + } > + local parts =3D tuple[6] > + local changed =3D false > + for _, part in pairs(parts) do > + local field_type =3D part[2]:lower() > + if field_types_v16[field_type] ~=3D nil then > + part[2] =3D field_types_v16[field_type] > + changed =3D true > + end > + end > + if changed then > + log.info("Updating index '%s' on space '%s': set parts to = %s", tuple[3], > + box.space[tuple[1]].name, json.encode(parts)) > + tuple =3D tuple:update{{'=3D', 6, parts}} > + end > + return tuple > +end > + > = +-------------------------------------------------------------------------= ------- > +-- Tarantool 1.7.5 > = +-------------------------------------------------------------------------= ------- > +local function create_truncate_space() > + local _truncate =3D box.space[box.schema.TRUNCATE_ID] > + > + log.info("create space _truncate") > + box.space._space:insert{_truncate.id, ADMIN, '_truncate', = 'memtx', 0, setmap({}), > + {{name =3D 'id', type =3D 'unsigned'}, = {name =3D 'count', type =3D 'unsigned'}}} > + > + log.info("create index primary on _truncate") > + box.space._index:insert{_truncate.id, 0, 'primary', 'tree', = {unique =3D true}, {{0, 'unsigned'}}} > + > + local _priv =3D box.space[box.schema.PRIV_ID] > + _priv:insert{ADMIN, PUBLIC, 'space', _truncate.id, box.priv.W} > +end > + > +local function upgrade_to_1_7_5() > + create_truncate_space() 7. Is it an intended nesting of a single-call function? > +end > + > +local function user_trig_1_7_5(_, tuple) > + if not tuple[5] then > + tuple =3D tuple:update{{'=3D', 5, setmap({})}} > + log.info("Set empty password to %s '%s'", tuple[4], tuple[3]) > + end > + return tuple > +end > +