From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 1F1F327B5D for ; Fri, 20 Jul 2018 07:32:56 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1VmK5G-g7e3p for ; Fri, 20 Jul 2018 07:32:56 -0400 (EDT) Received: from smtp34.i.mail.ru (smtp34.i.mail.ru [94.100.177.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 064DC2791C for ; Fri, 20 Jul 2018 07:32:54 -0400 (EDT) Subject: [tarantool-patches] Re: [PATCH 3/3] Introduce storage reload evolution References: From: Alex Khatskevich Message-ID: <76cb41bd-cce4-8483-9222-39176a146ce4@tarantool.org> Date: Fri, 20 Jul 2018 14:32:46 +0300 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit Content-Language: en-US Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-subscribe: List-owner: List-post: List-archive: To: Vladislav Shpilevoy , tarantool-patches@freelists.org >> * this patch contains some legacy-driven decisions: >>    1. SOURCEDIR path retrieved differentpy in case of > > 1. Typo: differentpy. fixed >>       is copied with respect to Centos 7 and `ro` mode of >>       SOURCEDIR. >> >> Closes 112 125 > > 2. Stray 'shut'. ok > >> >> diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh >> index 768b22b..554032b 100755 >> --- a/rpm/prebuild.sh >> +++ b/rpm/prebuild.sh >> @@ -1 +1,3 @@ >>   curl -s >> https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh >> | sudo bash >> +sudo yum -y install python-devel python-pip >> +sudo pip install tarantool msgpack > > 3. Why do you need it? As I understand, 'curl' above installs all > the needed things. Besides, pip install downloads not necessary > Tarantool 1.9. I cannot find those dependencies in the 'curl' above. Without this thing packpack on my laptop cannot pass tests. What is about tarantool 1.9? I cannot find this dependency in python-tarantool. > >> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua >> new file mode 100644 >> index 0000000..e2c17d0 >> --- /dev/null >> +++ b/test/lua_libs/git_util.lua >> @@ -0,0 +1,39 @@ >> +-- >> +-- Lua bridge for some of the git commands. >> +-- >> +local os = require('os') >> + >> +local temp_file = 'some_strange_rare_unique_file_name_for_git_util' >> +local function exec_cmd(options, cmd, args, files, dir, fout) > > 4. You can remove 'options' arg (it is always '' as I see). options are passed as a lua table. > >> +    files = files or '' >> +    options = options or '' >> +    args = args or '' > > 5. files, args, options are always non-nil. > 6. Why do you need to announce shell_cmd? Just do > >     local shell_cmd = ... Refactored > >> +    if fout then > > 7. 'fout' is always nil. not always (e.g. `log_hashes`) > >> +        shell_cmd = shell_cmd .. ' >' .. fout >> +    end >> +    if dir then > > 8. 'dir' is always non-nil. I would like to save this check, because this small utilite is not created especially for the test. Let it be a little more general than the current testcase. >> +        shell_cmd = string.format('cd %s && %s', dir, shell_cmd) >> +    end >> +    local res = os.execute(shell_cmd) >> +    assert(res == 0, 'Git cmd error: ' .. res) >> +end >> + >> +local function log_hashes(options, args, files, dir) >> +    args = args .. " --format='%h'" >> +    local local_temp_file = string.format('%s/%s', os.getenv('PWD'), >> temp_file) > > 9. Instead of writing output into a temporary file use > http://pgl.yoyo.org/luai/i/io.popen. Thanks for advice, I was looking for something similar. However, the `popen` has a huge disadvantage. It do not perform a join. It makes me write even more crutches than writing output to temp file. >> +    exec_cmd(options, 'log', args, files, dir, local_temp_file) >> +    local lines = {} >> +    for line in io.lines(local_temp_file) do >> +        table.insert(lines, line) >> +    end >> +    os.remove(local_temp_file) >> +    return lines >> +end >> + >> + >> +return { >> +    exec_cmd = exec_cmd, >> +    log_hashes = log_hashes >> +} >> diff --git a/test/reload_evolution/storage.result >> b/test/reload_evolution/storage.result >> new file mode 100644 >> index 0000000..2cf21fd >> --- /dev/null >> +++ b/test/reload_evolution/storage.result >> @@ -0,0 +1,184 @@ >> +test_run = require('test_run').new() >> >> +test_run:grep_log('storage_1_a', 'vshard.storage.reload_evolution: >> upgraded to') ~= nil >> +--- >> +- true >> +... >> +vshard.storage.internal.reload_evolution_version >> +--- >> +- 1 >> +... >> +-- Make sure storage operates well. >> +vshard.storage.bucket_force_drop(2) >> +--- >> +- true >> +... >> +vshard.storage.bucket_force_create(2) > > 10. This is too simple test. Force_drop/create merely do DML on > _bucket. Lets > test the rebalancer creating a dis-balance. For example, put on > storage_1 +10 > buckets, and on storage_2 -10 buckets and wait for the balance. done I also changed tests so that the tested storage is storage_2_a (rebalancer lives here), > >> diff --git a/test/reload_evolution/storage_1_a.lua >> b/test/reload_evolution/storage_1_a.lua >> new file mode 100755 >> index 0000000..3e03f8f >> --- /dev/null >> +++ b/test/reload_evolution/storage_1_a.lua >> @@ -0,0 +1,144 @@ >> +#!/usr/bin/env tarantool > > 11. Just use a symbolic link to the existing storage_1_a. Same > for other storages. > >> diff --git a/test/reload_evolution/suite.ini >> b/test/reload_evolution/suite.ini >> new file mode 100644 >> index 0000000..bb5435b >> --- /dev/null >> +++ b/test/reload_evolution/suite.ini > > 12. You do not need a new test suite for one storage test. Please, > put it into test/storage/ 11-12: I have modified storage_1_a to change luapath. It is important to change path here, because the default Vshard initialization is performed here. > >> diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua >> index bf560e6..1740c98 100644 >> --- a/vshard/storage/init.lua >> +++ b/vshard/storage/init.lua >> @@ -105,6 +110,11 @@ if not M then >>           -- a destination replicaset must drop already received >>           -- data. >>           rebalancer_sending_bucket = 0, >> + >> +        ------------------------- Reload ------------------------- >> +        -- Version of the loaded module. This number is used on >> +        -- reload to determine which upgrade scripts to run. >> +        reload_evolution_version = reload_evolution.version, > > 13. Use 'version' name. But it is not a version. Version is a git tag. We already have module_version and git tag. To make everything clear I would like to name the variable in that or similar way. > >>       } >>   end >>   diff --git a/vshard/storage/reload_evolution.lua >> b/vshard/storage/reload_evolution.lua >> new file mode 100644 >> index 0000000..cfac888 >> --- /dev/null >> +++ b/vshard/storage/reload_evolution.lua >> @@ -0,0 +1,58 @@ >> +-- >> +-- This module is used to upgrade the vshard.storage on the fly. >> +-- It updates internal Lua structures in case they are changed >> +-- in a commit. >> +-- >> +local log = require('log') >> + >> +-- >> +-- Array of upgrade functions. >> +-- magrations[version] = function which upgrades module version > > 14. Typo: magrations. fixed > >> +-- from `version` to `version + 1`. > > 15. Not +1. I think, we should use real version numbers: > 0.1.1, 0.1.2, etc similar to tarantool. Disagree. That would make us to 1. Tag commit each time M is changed 2. Maintain identical git tags and reload_evolution_version names. In the introduced all you need to do is just to append callback to the `migrations` array. One thing I worry about is out of main branch hotfixes and reload after it on main vshard branch. But this problem seems to be not solvable in a general case. > >> +-- >> +local migrations = {} >> + >> +-- Initialize reload_upgrade mechanism >> +migrations[#migrations + 1] = function (M) >> +    -- Code to update Lua objects. >> +end >> + >> +-- >> +-- Perform an update based on a version stored in `M` (internals). >> +-- @param M Old module internals which should be updated. >> +-- >> +local function upgrade(M) >> +    local start_version = M.reload_evolution_version or 1 >> +    if start_version > #migrations then >> +        local err_msg = string.format( >> +            'vshard.storage.reload_evolution: ' .. >> +            'auto-downgrade is not implemented; ' .. >> +            'loaded version is %d, upgrade script version is %d', >> +            start_version, #migrations > > 16. Did you test it? I do not see a test. It is very simple check, so, added this as a unittest. > >> +        ) >> +        log.error(err_msg) >> +        error(err_msg) >> +    end >> +    for i = start_version, #migrations  do >> +        local ok, err = pcall(migrations[i], M) >> +        if ok then >> +            log.info('vshard.storage.reload_evolution: upgraded to >> %d version', >> +                     i) >> +        else >> +            local err_msg = string.format( >> +                'vshard.storage.reload_evolution: ' .. >> +                'error during upgrade to %d version: %s', i, err >> +            ) >> +            log.error(err_msg) >> +            error(err_msg) >> +        end >> +        -- Update the version just after upgrade to have an >> +        -- actual version in case of an error. >> +        M.reload_evolution_version = i >> +    end >> +end >> + >> +return { >> +    version = #migrations, > > 17. Where do you use it? It is default value for storage.internal.M.reload_evolution_version. Here is a new diff: commit bce659db650b4a88f4285d90ec9d7726e036878f Author: AKhatskevich Date:   Fri Jun 29 20:34:26 2018 +0300     Introduce storage reload evolution     Changes:     1. Introduce storage reload evolution.     2. Setup cross-version reload testing.     1:     This mechanism updates Lua objects on reload in case they are     changed in a new vshard.storage version.     Since this commit, any change in vshard.storage.M has to be     reflected in vshard.storage.reload_evolution to guarantee     correct reload.     2:     The testing uses git infrastructure and is performed in the following     way:     1. Copy old version of vshard to a temp folder.     2. Run vshard on this code.     3. Checkout the latest version of the vshard sources.     4. Reload vshard storage.     5. Make sure it works (Perform simple tests).     Notes:     * this patch contains some legacy-driven decisions:       1. SOURCEDIR path retrieved differently in case of          packpack build.       2. git directory in the `reload_evolution/storage` test          is copied with respect to Centos 7 and `ro` mode of          SOURCEDIR.     Closes #112 #125 diff --git a/.travis.yml b/.travis.yml index 54bfe44..eff4a51 100644 --- a/.travis.yml +++ b/.travis.yml @@ -41,7 +41,7 @@ env:  script:    - git describe --long    - git clone https://github.com/packpack/packpack.git packpack -  - packpack/packpack +  - packpack/packpack -e PACKPACK_GIT_SOURCEDIR=/source/  before_deploy:    - ls -l build/ diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh index 768b22b..554032b 100755 --- a/rpm/prebuild.sh +++ b/rpm/prebuild.sh @@ -1 +1,3 @@  curl -s https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh | sudo bash +sudo yum -y install python-devel python-pip +sudo pip install tarantool msgpack diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua new file mode 100644 index 0000000..a75bb08 --- /dev/null +++ b/test/lua_libs/git_util.lua @@ -0,0 +1,51 @@ +-- +-- Lua bridge for some of the git commands. +-- +local os = require('os') + +local temp_file = 'some_strange_rare_unique_file_name_for_git_util' + +-- +-- Exec a git command. +-- @param params Table of parameters: +--        * options - git options. +--        * cmd - git command. +--        * args - command arguments. +--        * dir - working directory. +--        * fout - write output to the file. +local function exec_cmd(params) +    local fout = params.fout +    local shell_cmd = {'git'} +    for _, param in pairs({'options', 'cmd', 'args'}) do +        table.insert(shell_cmd, params[param]) +    end +    if fout then +        table.insert(shell_cmd, ' >' .. fout) +    end +    shell_cmd = table.concat(shell_cmd, ' ') +    if params.dir then +        shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd) +    end +    local res = os.execute(shell_cmd) +    assert(res == 0, 'Git cmd error: ' .. res) +end + +local function log_hashes(params) +    params.args = "--format='%h' " .. params.args +    local local_temp_file = string.format('%s/%s', os.getenv('PWD'), temp_file) +    params.fout = local_temp_file +    params.cmd = 'log' +    exec_cmd(params) +    local lines = {} +    for line in io.lines(local_temp_file) do +        table.insert(lines, line) +    end +    os.remove(local_temp_file) +    return lines +end + + +return { +    exec_cmd = exec_cmd, +    log_hashes = log_hashes +} diff --git a/test/lua_libs/util.lua b/test/lua_libs/util.lua index 2d866df..e9bbd9a 100644 --- a/test/lua_libs/util.lua +++ b/test/lua_libs/util.lua @@ -1,5 +1,6 @@  local fiber = require('fiber')  local log = require('log') +local fio = require('fio')  local function check_error(func, ...)      local pstatus, status, err = pcall(func, ...) @@ -89,10 +90,29 @@ local function has_same_fields(etalon, data)      return true  end +-- Git directory of the project. Used in evolution tests to +-- fetch old versions of vshard. +local SOURCEDIR = os.getenv('PACKPACK_GIT_SOURCEDIR') +if not SOURCEDIR then +    SOURCEDIR = os.getenv('SOURCEDIR') +end +if not SOURCEDIR then +    local script_path = debug.getinfo(1).source:match("@?(.*/)") +    script_path = fio.abspath(script_path) +    SOURCEDIR = fio.abspath(script_path .. '/../../../') +end + +local BUILDDIR = os.getenv('BUILDDIR') +if not BUILDDIR then +    BUILDDIR = SOURCEDIR +end +  return {      check_error = check_error,      shuffle_masters = shuffle_masters,      collect_timeouts = collect_timeouts,      wait_master = wait_master,      has_same_fields = has_same_fields, +    SOURCEDIR = SOURCEDIR, +    BUILDDIR = BUILDDIR,  } diff --git a/test/reload_evolution/storage.result b/test/reload_evolution/storage.result new file mode 100644 index 0000000..4ffbc26 --- /dev/null +++ b/test/reload_evolution/storage.result @@ -0,0 +1,248 @@ +test_run = require('test_run').new() +--- +... +git_util = require('git_util') +--- +... +util = require('util') +--- +... +vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy' +--- +... +evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR}) +--- +... +-- Cleanup the directory after a previous build. +_ = os.execute('rm -rf ' .. vshard_copy_path) +--- +... +-- 1. `git worktree` cannot be used because PACKPACK mounts +-- `/source/` in `ro` mode. +-- 2. Just `cp -rf` cannot be used due to a little different +-- behavior in Centos 7. +_ = os.execute('mkdir ' .. vshard_copy_path) +--- +... +_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path) +--- +... +-- Checkout the first commit with a reload_evolution mechanism. +git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path}) +--- +... +git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path}) +--- +... +REPLICASET_1 = { 'storage_1_a', 'storage_1_b' } +--- +... +REPLICASET_2 = { 'storage_2_a', 'storage_2_b' } +--- +... +test_run:create_cluster(REPLICASET_1, 'reload_evolution') +--- +... +test_run:create_cluster(REPLICASET_2, 'reload_evolution') +--- +... +util = require('util') +--- +... +util.wait_master(test_run, REPLICASET_1, 'storage_1_a') +--- +... +util.wait_master(test_run, REPLICASET_2, 'storage_2_a') +--- +... +test_run:switch('storage_1_a') +--- +- true +... +vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2) +--- +- true +... +bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT +--- +... +test_run:switch('storage_2_a') +--- +- true +... +fiber = require('fiber') +--- +... +vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2) +--- +- true +... +bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT +--- +... +vshard.storage.internal.reload_evolution_version +--- +- null +... +box.space.customer:insert({bucket_id_to_move, 1, 'customer_name'}) +--- +- [3000, 1, 'customer_name'] +... +while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') == nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end +--- +... +test_run:switch('default') +--- +- true +... +git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path}) +--- +... +test_run:switch('storage_2_a') +--- +- true +... +package.loaded["vshard.storage"] = nil +--- +... +vshard.storage = require("vshard.storage") +--- +... +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil +--- +- true +... +vshard.storage.internal.reload_evolution_version +--- +- 1 +... +-- Make sure storage operates well. +vshard.storage.bucket_force_drop(2000) +--- +- true +... +vshard.storage.bucket_force_create(2000) +--- +- true +... +vshard.storage.buckets_info()[2000] +--- +- status: active +  id: 2000 +... +vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1}) +--- +- true +- null +... +vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid) +--- +- true +... +vshard.storage.garbage_collector_wakeup() +--- +... +fiber = require('fiber') +--- +... +while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end +--- +... +test_run:switch('storage_1_a') +--- +- true +... +vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid) +--- +- true +... +test_run:switch('storage_2_a') +--- +- true +... +vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1}) +--- +- true +- null +... +-- Check info() does not fail. +vshard.storage.info() ~= nil +--- +- true +... +-- +-- Send buckets to create a disbalance. Wait until the rebalancer +-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`. +-- +vshard.storage.rebalancer_disable() +--- +... +move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1 +--- +... +move_cnt = 100 +--- +... +assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT) +--- +- true +... +for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end +--- +... +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) +--- +- 1400 +... +test_run:switch('storage_1_a') +--- +- true +... +move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1 +--- +... +move_cnt = 100 +--- +... +vshard.storage.bucket_force_create(move_start, move_cnt) +--- +- true +... +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) +--- +- 1600 +... +test_run:switch('storage_2_a') +--- +- true +... +vshard.storage.rebalancer_enable() +--- +... +vshard.storage.rebalancer_wakeup() +--- +... +wait_rebalancer_state("Rebalance routes are sent", test_run) +--- +... +wait_rebalancer_state('The cluster is balanced ok', test_run) +--- +... +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) +--- +- 1500 +... +test_run:switch('default') +--- +- true +... +test_run:drop_cluster(REPLICASET_2) +--- +... +test_run:drop_cluster(REPLICASET_1) +--- +... +test_run:cmd('clear filter') +--- +- true +... diff --git a/test/reload_evolution/storage.test.lua b/test/reload_evolution/storage.test.lua new file mode 100644 index 0000000..5deb136 --- /dev/null +++ b/test/reload_evolution/storage.test.lua @@ -0,0 +1,88 @@ +test_run = require('test_run').new() + +git_util = require('git_util') +util = require('util') +vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy' +evolution_log = git_util.log_hashes({args='vshard/storage/reload_evolution.lua', dir=util.SOURCEDIR}) +-- Cleanup the directory after a previous build. +_ = os.execute('rm -rf ' .. vshard_copy_path) +-- 1. `git worktree` cannot be used because PACKPACK mounts +-- `/source/` in `ro` mode. +-- 2. Just `cp -rf` cannot be used due to a little different +-- behavior in Centos 7. +_ = os.execute('mkdir ' .. vshard_copy_path) +_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A --ignore=build` ' .. vshard_copy_path) +-- Checkout the first commit with a reload_evolution mechanism. +git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path}) +git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log] .. '~1', dir=vshard_copy_path}) + +REPLICASET_1 = { 'storage_1_a', 'storage_1_b' } +REPLICASET_2 = { 'storage_2_a', 'storage_2_b' } +test_run:create_cluster(REPLICASET_1, 'reload_evolution') +test_run:create_cluster(REPLICASET_2, 'reload_evolution') +util = require('util') +util.wait_master(test_run, REPLICASET_1, 'storage_1_a') +util.wait_master(test_run, REPLICASET_2, 'storage_2_a') + +test_run:switch('storage_1_a') +vshard.storage.bucket_force_create(1, vshard.consts.DEFAULT_BUCKET_COUNT / 2) +bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT + +test_run:switch('storage_2_a') +fiber = require('fiber') +vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2) +bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT +vshard.storage.internal.reload_evolution_version +box.space.customer:insert({bucket_id_to_move, 1, 'customer_name'}) +while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') == nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end + +test_run:switch('default') +git_util.exec_cmd({cmd='checkout', args=evolution_log[1], dir=vshard_copy_path}) + +test_run:switch('storage_2_a') +package.loaded["vshard.storage"] = nil +vshard.storage = require("vshard.storage") +test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution: upgraded to') ~= nil +vshard.storage.internal.reload_evolution_version +-- Make sure storage operates well. +vshard.storage.bucket_force_drop(2000) +vshard.storage.bucket_force_create(2000) +vshard.storage.buckets_info()[2000] +vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1}) +vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid) +vshard.storage.garbage_collector_wakeup() +fiber = require('fiber') +while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end +test_run:switch('storage_1_a') +vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid) +test_run:switch('storage_2_a') +vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1}) +-- Check info() does not fail. +vshard.storage.info() ~= nil + +-- +-- Send buckets to create a disbalance. Wait until the rebalancer +-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`. +-- +vshard.storage.rebalancer_disable() +move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1 +move_cnt = 100 +assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT) +for i = move_start, move_start + move_cnt - 1 do box.space._bucket:delete{i} end +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) +test_run:switch('storage_1_a') +move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1 +move_cnt = 100 +vshard.storage.bucket_force_create(move_start, move_cnt) +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) +test_run:switch('storage_2_a') +vshard.storage.rebalancer_enable() +vshard.storage.rebalancer_wakeup() +wait_rebalancer_state("Rebalance routes are sent", test_run) +wait_rebalancer_state('The cluster is balanced ok', test_run) +box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE}) + +test_run:switch('default') +test_run:drop_cluster(REPLICASET_2) +test_run:drop_cluster(REPLICASET_1) +test_run:cmd('clear filter') diff --git a/test/reload_evolution/storage_1_a.lua b/test/reload_evolution/storage_1_a.lua new file mode 100755 index 0000000..d8bda60 --- /dev/null +++ b/test/reload_evolution/storage_1_a.lua @@ -0,0 +1,154 @@ +#!/usr/bin/env tarantool + +require('strict').on() + + +local fio = require('fio') + +-- Get instance name +local fio = require('fio') +local log = require('log') +local NAME = fio.basename(arg[0], '.lua') +local fiber = require('fiber') +local util = require('util') + +-- Run one storage on a different vshard version. +-- To do that, place vshard src to +-- BUILDDIR/test/var/vshard_git_tree_copy/. +if NAME == 'storage_2_a' then +    local script_path = debug.getinfo(1).source:match("@?(.*/)") +    vshard_copy = util.BUILDDIR .. '/test/var/vshard_git_tree_copy' +    package.path = string.format( +        '%s/?.lua;%s/?/init.lua;%s', +        vshard_copy, vshard_copy, package.path +    ) +end + +-- Check if we are running under test-run +if os.getenv('ADMIN') then +    test_run = require('test_run').new() +    require('console').listen(os.getenv('ADMIN')) +end + +-- Call a configuration provider +cfg = require('localcfg') +-- Name to uuid map +names = { +    ['storage_1_a'] = '8a274925-a26d-47fc-9e1b-af88ce939412', +    ['storage_1_b'] = '3de2e3e1-9ebe-4d0d-abb1-26d301b84633', +    ['storage_2_a'] = '1e02ae8a-afc0-4e91-ba34-843a356b8ed7', +    ['storage_2_b'] = '001688c3-66f8-4a31-8e19-036c17d489c2', +} + +replicaset1_uuid = 'cbf06940-0790-498b-948d-042b62cf3d29' +replicaset2_uuid = 'ac522f65-aa94-4134-9f64-51ee384f1a54' +replicasets = {replicaset1_uuid, replicaset2_uuid} + +-- Start the database with sharding +vshard = require('vshard') +vshard.storage.cfg(cfg, names[NAME]) + +box.once("testapp:schema:1", function() +    local customer = box.schema.space.create('customer') +    customer:format({ +        {'customer_id', 'unsigned'}, +        {'bucket_id', 'unsigned'}, +        {'name', 'string'}, +    }) +    customer:create_index('customer_id', {parts = {'customer_id'}}) +    customer:create_index('bucket_id', {parts = {'bucket_id'}, unique = false}) + +    local account = box.schema.space.create('account') +    account:format({ +        {'account_id', 'unsigned'}, +        {'customer_id', 'unsigned'}, +        {'bucket_id', 'unsigned'}, +        {'balance', 'unsigned'}, +        {'name', 'string'}, +    }) +    account:create_index('account_id', {parts = {'account_id'}}) +    account:create_index('customer_id', {parts = {'customer_id'}, unique = false}) +    account:create_index('bucket_id', {parts = {'bucket_id'}, unique = false}) +    box.snapshot() + +    box.schema.func.create('customer_lookup') +    box.schema.role.grant('public', 'execute', 'function', 'customer_lookup') +    box.schema.func.create('customer_add') +    box.schema.role.grant('public', 'execute', 'function', 'customer_add') +    box.schema.func.create('echo') +    box.schema.role.grant('public', 'execute', 'function', 'echo') +    box.schema.func.create('sleep') +    box.schema.role.grant('public', 'execute', 'function', 'sleep') +    box.schema.func.create('raise_luajit_error') +    box.schema.role.grant('public', 'execute', 'function', 'raise_luajit_error') +    box.schema.func.create('raise_client_error') +    box.schema.role.grant('public', 'execute', 'function', 'raise_client_error') +end) + +function customer_add(customer) +    box.begin() +    box.space.customer:insert({customer.customer_id, customer.bucket_id, +                               customer.name}) +    for _, account in ipairs(customer.accounts) do +        box.space.account:insert({ +            account.account_id, +            customer.customer_id, +            customer.bucket_id, +            0, +            account.name +        }) +    end +    box.commit() +    return true +end + +function customer_lookup(customer_id) +    if type(customer_id) ~= 'number' then +        error('Usage: customer_lookup(customer_id)') +    end + +    local customer = box.space.customer:get(customer_id) +    if customer == nil then +        return nil +    end +    customer = { +        customer_id = customer.customer_id; +        name = customer.name; +    } +    local accounts = {} +    for _, account in box.space.account.index.customer_id:pairs(customer_id) do +        table.insert(accounts, { +            account_id = account.account_id; +            name = account.name; +            balance = account.balance; +        }) +    end +    customer.accounts = accounts; +    return customer +end + +function echo(...) +    return ... +end + +function sleep(time) +    fiber.sleep(time) +    return true +end + +function raise_luajit_error() +    assert(1 == 2) +end + +function raise_client_error() +    box.error(box.error.UNKNOWN) +end + +function wait_rebalancer_state(state, test_run) +    log.info(string.rep('a', 1000)) +    vshard.storage.rebalancer_wakeup() +    while not test_run:grep_log(NAME, state, 1000) do +        fiber.sleep(0.1) +        vshard.storage.rebalancer_wakeup() +    end +end diff --git a/test/reload_evolution/storage_1_b.lua b/test/reload_evolution/storage_1_b.lua new file mode 120000 index 0000000..02572da --- /dev/null +++ b/test/reload_evolution/storage_1_b.lua @@ -0,0 +1 @@ +storage_1_a.lua \ No newline at end of file diff --git a/test/reload_evolution/storage_2_a.lua b/test/reload_evolution/storage_2_a.lua new file mode 120000 index 0000000..02572da --- /dev/null +++ b/test/reload_evolution/storage_2_a.lua @@ -0,0 +1 @@ +storage_1_a.lua \ No newline at end of file diff --git a/test/reload_evolution/storage_2_b.lua b/test/reload_evolution/storage_2_b.lua new file mode 120000 index 0000000..02572da --- /dev/null +++ b/test/reload_evolution/storage_2_b.lua @@ -0,0 +1 @@ +storage_1_a.lua \ No newline at end of file diff --git a/test/reload_evolution/suite.ini b/test/reload_evolution/suite.ini new file mode 100644 index 0000000..bb5435b --- /dev/null +++ b/test/reload_evolution/suite.ini @@ -0,0 +1,6 @@ +[default] +core = tarantool +description = Reload evolution tests +script = test.lua +is_parallel = False +lua_libs = ../lua_libs/util.lua ../lua_libs/git_util.lua ../../example/localcfg.lua diff --git a/test/reload_evolution/test.lua b/test/reload_evolution/test.lua new file mode 100644 index 0000000..ad0543a --- /dev/null +++ b/test/reload_evolution/test.lua @@ -0,0 +1,9 @@ +#!/usr/bin/env tarantool + +require('strict').on() + +box.cfg{ +    listen = os.getenv("LISTEN"), +} + +require('console').listen(os.getenv('ADMIN')) diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua index 07bd00c..3bec09f 100644 --- a/vshard/storage/init.lua +++ b/vshard/storage/init.lua @@ -10,6 +10,7 @@ if rawget(_G, MODULE_INTERNALS) then      local vshard_modules = {          'vshard.consts', 'vshard.error', 'vshard.cfg',          'vshard.replicaset', 'vshard.util', +        'vshard.storage.reload_evolution'      }      for _, module in pairs(vshard_modules) do          package.loaded[module] = nil @@ -20,12 +21,16 @@ local lerror = require('vshard.error')  local lcfg = require('vshard.cfg')  local lreplicaset = require('vshard.replicaset')  local util = require('vshard.util') +local reload_evolution = require('vshard.storage.reload_evolution')  local M = rawget(_G, MODULE_INTERNALS)  if not M then      --      -- The module is loaded for the first time.      -- +    -- !!!WARNING: any change of this table must be reflected in +    -- `vshard.storage.reload_evolution` module to guarantee +    -- reloadability of the module.      M = {          ---------------- Common module attributes ----------------          -- The last passed configuration. @@ -105,6 +110,11 @@ if not M then          -- a destination replicaset must drop already received          -- data.          rebalancer_sending_bucket = 0, + +        ------------------------- Reload ------------------------- +        -- Version of the loaded module. This number is used on +        -- reload to determine which upgrade scripts to run. +        reload_evolution_version = reload_evolution.version,      }  end @@ -1863,6 +1873,7 @@ end  if not rawget(_G, MODULE_INTERNALS) then      rawset(_G, MODULE_INTERNALS, M)  else +    reload_evolution.upgrade(M)      storage_cfg(M.current_cfg, M.this_replica.uuid)      M.module_version = M.module_version + 1  end diff --git a/vshard/storage/reload_evolution.lua b/vshard/storage/reload_evolution.lua new file mode 100644 index 0000000..f25ad49 --- /dev/null +++ b/vshard/storage/reload_evolution.lua @@ -0,0 +1,58 @@ +-- +-- This module is used to upgrade the vshard.storage on the fly. +-- It updates internal Lua structures in case they are changed +-- in a commit. +-- +local log = require('log') + +-- +-- Array of upgrade functions. +-- migrations[version] = function which upgrades module version +-- from `version` to `version + 1`. +-- +local migrations = {} + +-- Initialize reload_upgrade mechanism +migrations[#migrations + 1] = function (M) +    -- Code to update Lua objects. +end + +-- +-- Perform an update based on a version stored in `M` (internals). +-- @param M Old module internals which should be updated. +-- +local function upgrade(M) +    local start_version = M.reload_evolution_version or 1 +    if start_version > #migrations then +        local err_msg = string.format( +            'vshard.storage.reload_evolution: ' .. +            'auto-downgrade is not implemented; ' .. +            'loaded version is %d, upgrade script version is %d', +            start_version, #migrations +        ) +        log.error(err_msg) +        error(err_msg) +    end +    for i = start_version, #migrations  do +        local ok, err = pcall(migrations[i], M) +        if ok then +            log.info('vshard.storage.reload_evolution: upgraded to %d version', +                     i) +        else +            local err_msg = string.format( +                'vshard.storage.reload_evolution: ' .. +                'error during upgrade to %d version: %s', i, err +            ) +            log.error(err_msg) +            error(err_msg) +        end +        -- Update the version just after upgrade to have an +        -- actual version in case of an error. +        M.reload_evolution_version = i +    end +end + +return { +    version = #migrations, +    upgrade = upgrade, +}