[tarantool-patches] Re: [PATCH 3/3] Introduce storage reload evolution
Alex Khatskevich
avkhatskevich at tarantool.org
Fri Jul 20 14:32:46 MSK 2018
>> * this patch contains some legacy-driven decisions:
>> 1. SOURCEDIR path retrieved differentpy in case of
>
> 1. Typo: differentpy.
fixed
>> is copied with respect to Centos 7 and `ro` mode of
>> SOURCEDIR.
>>
>> Closes <shut git>112 125
>
> 2. Stray 'shut'.
ok
>
>>
>> diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh
>> index 768b22b..554032b 100755
>> --- a/rpm/prebuild.sh
>> +++ b/rpm/prebuild.sh
>> @@ -1 +1,3 @@
>> curl -s
>> https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh
>> | sudo bash
>> +sudo yum -y install python-devel python-pip
>> +sudo pip install tarantool msgpack
>
> 3. Why do you need it? As I understand, 'curl' above installs all
> the needed things. Besides, pip install downloads not necessary
> Tarantool 1.9.
I cannot find those dependencies in the 'curl' above.
Without this thing packpack on my laptop cannot pass tests.
What is about tarantool 1.9? I cannot find this dependency in
python-tarantool.
>
>> diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
>> new file mode 100644
>> index 0000000..e2c17d0
>> --- /dev/null
>> +++ b/test/lua_libs/git_util.lua
>> @@ -0,0 +1,39 @@
>> +--
>> +-- Lua bridge for some of the git commands.
>> +--
>> +local os = require('os')
>> +
>> +local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
>> +local function exec_cmd(options, cmd, args, files, dir, fout)
>
> 4. You can remove 'options' arg (it is always '' as I see).
options are passed as a lua table.
>
>> + files = files or ''
>> + options = options or ''
>> + args = args or ''
>
> 5. files, args, options are always non-nil.
> 6. Why do you need to announce shell_cmd? Just do
>
> local shell_cmd = ...
Refactored
>
>> + if fout then
>
> 7. 'fout' is always nil.
not always (e.g. `log_hashes`)
>
>> + shell_cmd = shell_cmd .. ' >' .. fout
>> + end
>> + if dir then
>
> 8. 'dir' is always non-nil.
I would like to save this check, because this small utilite is not
created especially for the test.
Let it be a little more general than the current testcase.
>> + shell_cmd = string.format('cd %s && %s', dir, shell_cmd)
>> + end
>> + local res = os.execute(shell_cmd)
>> + assert(res == 0, 'Git cmd error: ' .. res)
>> +end
>> +
>> +local function log_hashes(options, args, files, dir)
>> + args = args .. " --format='%h'"
>> + local local_temp_file = string.format('%s/%s', os.getenv('PWD'),
>> temp_file)
>
> 9. Instead of writing output into a temporary file use
> http://pgl.yoyo.org/luai/i/io.popen.
Thanks for advice, I was looking for something similar. However, the
`popen` has a huge disadvantage. It do not perform a join. It makes me
write even more crutches than writing output to temp file.
>> + exec_cmd(options, 'log', args, files, dir, local_temp_file)
>> + local lines = {}
>> + for line in io.lines(local_temp_file) do
>> + table.insert(lines, line)
>> + end
>> + os.remove(local_temp_file)
>> + return lines
>> +end
>> +
>> +
>> +return {
>> + exec_cmd = exec_cmd,
>> + log_hashes = log_hashes
>> +}
>> diff --git a/test/reload_evolution/storage.result
>> b/test/reload_evolution/storage.result
>> new file mode 100644
>> index 0000000..2cf21fd
>> --- /dev/null
>> +++ b/test/reload_evolution/storage.result
>> @@ -0,0 +1,184 @@
>> +test_run = require('test_run').new()
>>
>> +test_run:grep_log('storage_1_a', 'vshard.storage.reload_evolution:
>> upgraded to') ~= nil
>> +---
>> +- true
>> +...
>> +vshard.storage.internal.reload_evolution_version
>> +---
>> +- 1
>> +...
>> +-- Make sure storage operates well.
>> +vshard.storage.bucket_force_drop(2)
>> +---
>> +- true
>> +...
>> +vshard.storage.bucket_force_create(2)
>
> 10. This is too simple test. Force_drop/create merely do DML on
> _bucket. Lets
> test the rebalancer creating a dis-balance. For example, put on
> storage_1 +10
> buckets, and on storage_2 -10 buckets and wait for the balance.
done
I also changed tests so that the tested storage is storage_2_a
(rebalancer lives here),
>
>> diff --git a/test/reload_evolution/storage_1_a.lua
>> b/test/reload_evolution/storage_1_a.lua
>> new file mode 100755
>> index 0000000..3e03f8f
>> --- /dev/null
>> +++ b/test/reload_evolution/storage_1_a.lua
>> @@ -0,0 +1,144 @@
>> +#!/usr/bin/env tarantool
>
> 11. Just use a symbolic link to the existing storage_1_a. Same
> for other storages.
>
>> diff --git a/test/reload_evolution/suite.ini
>> b/test/reload_evolution/suite.ini
>> new file mode 100644
>> index 0000000..bb5435b
>> --- /dev/null
>> +++ b/test/reload_evolution/suite.ini
>
> 12. You do not need a new test suite for one storage test. Please,
> put it into test/storage/
11-12:
I have modified storage_1_a to change luapath.
It is important to change path here, because the default Vshard
initialization is performed here.
>
>> diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
>> index bf560e6..1740c98 100644
>> --- a/vshard/storage/init.lua
>> +++ b/vshard/storage/init.lua
>> @@ -105,6 +110,11 @@ if not M then
>> -- a destination replicaset must drop already received
>> -- data.
>> rebalancer_sending_bucket = 0,
>> +
>> + ------------------------- Reload -------------------------
>> + -- Version of the loaded module. This number is used on
>> + -- reload to determine which upgrade scripts to run.
>> + reload_evolution_version = reload_evolution.version,
>
> 13. Use 'version' name.
But it is not a version. Version is a git tag.
We already have module_version and git tag. To make everything clear I
would like to
name the variable in that or similar way.
>
>> }
>> end
>> diff --git a/vshard/storage/reload_evolution.lua
>> b/vshard/storage/reload_evolution.lua
>> new file mode 100644
>> index 0000000..cfac888
>> --- /dev/null
>> +++ b/vshard/storage/reload_evolution.lua
>> @@ -0,0 +1,58 @@
>> +--
>> +-- This module is used to upgrade the vshard.storage on the fly.
>> +-- It updates internal Lua structures in case they are changed
>> +-- in a commit.
>> +--
>> +local log = require('log')
>> +
>> +--
>> +-- Array of upgrade functions.
>> +-- magrations[version] = function which upgrades module version
>
> 14. Typo: magrations.
fixed
>
>> +-- from `version` to `version + 1`.
>
> 15. Not +1. I think, we should use real version numbers:
> 0.1.1, 0.1.2, etc similar to tarantool.
Disagree. That would make us to
1. Tag commit each time M is changed
2. Maintain identical git tags and reload_evolution_version names.
In the introduced all you need to do is just to append callback to the
`migrations` array.
One thing I worry about is out of main branch hotfixes and reload after
it on main vshard branch. But this problem seems to be not solvable in a
general case.
>
>> +--
>> +local migrations = {}
>> +
>> +-- Initialize reload_upgrade mechanism
>> +migrations[#migrations + 1] = function (M)
>> + -- Code to update Lua objects.
>> +end
>> +
>> +--
>> +-- Perform an update based on a version stored in `M` (internals).
>> +-- @param M Old module internals which should be updated.
>> +--
>> +local function upgrade(M)
>> + local start_version = M.reload_evolution_version or 1
>> + if start_version > #migrations then
>> + local err_msg = string.format(
>> + 'vshard.storage.reload_evolution: ' ..
>> + 'auto-downgrade is not implemented; ' ..
>> + 'loaded version is %d, upgrade script version is %d',
>> + start_version, #migrations
>
> 16. Did you test it? I do not see a test.
It is very simple check, so, added this as a unittest.
>
>> + )
>> + log.error(err_msg)
>> + error(err_msg)
>> + end
>> + for i = start_version, #migrations do
>> + local ok, err = pcall(migrations[i], M)
>> + if ok then
>> + log.info('vshard.storage.reload_evolution: upgraded to
>> %d version',
>> + i)
>> + else
>> + local err_msg = string.format(
>> + 'vshard.storage.reload_evolution: ' ..
>> + 'error during upgrade to %d version: %s', i, err
>> + )
>> + log.error(err_msg)
>> + error(err_msg)
>> + end
>> + -- Update the version just after upgrade to have an
>> + -- actual version in case of an error.
>> + M.reload_evolution_version = i
>> + end
>> +end
>> +
>> +return {
>> + version = #migrations,
>
> 17. Where do you use it?
It is default value for storage.internal.M.reload_evolution_version.
Here is a new diff:
commit bce659db650b4a88f4285d90ec9d7726e036878f
Author: AKhatskevich <avkhatskevich at tarantool.org>
Date: Fri Jun 29 20:34:26 2018 +0300
Introduce storage reload evolution
Changes:
1. Introduce storage reload evolution.
2. Setup cross-version reload testing.
1:
This mechanism updates Lua objects on reload in case they are
changed in a new vshard.storage version.
Since this commit, any change in vshard.storage.M has to be
reflected in vshard.storage.reload_evolution to guarantee
correct reload.
2:
The testing uses git infrastructure and is performed in the following
way:
1. Copy old version of vshard to a temp folder.
2. Run vshard on this code.
3. Checkout the latest version of the vshard sources.
4. Reload vshard storage.
5. Make sure it works (Perform simple tests).
Notes:
* this patch contains some legacy-driven decisions:
1. SOURCEDIR path retrieved differently in case of
packpack build.
2. git directory in the `reload_evolution/storage` test
is copied with respect to Centos 7 and `ro` mode of
SOURCEDIR.
Closes #112 #125
diff --git a/.travis.yml b/.travis.yml
index 54bfe44..eff4a51 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -41,7 +41,7 @@ env:
script:
- git describe --long
- git clone https://github.com/packpack/packpack.git packpack
- - packpack/packpack
+ - packpack/packpack -e PACKPACK_GIT_SOURCEDIR=/source/
before_deploy:
- ls -l build/
diff --git a/rpm/prebuild.sh b/rpm/prebuild.sh
index 768b22b..554032b 100755
--- a/rpm/prebuild.sh
+++ b/rpm/prebuild.sh
@@ -1 +1,3 @@
curl -s
https://packagecloud.io/install/repositories/tarantool/1_9/script.rpm.sh
| sudo bash
+sudo yum -y install python-devel python-pip
+sudo pip install tarantool msgpack
diff --git a/test/lua_libs/git_util.lua b/test/lua_libs/git_util.lua
new file mode 100644
index 0000000..a75bb08
--- /dev/null
+++ b/test/lua_libs/git_util.lua
@@ -0,0 +1,51 @@
+--
+-- Lua bridge for some of the git commands.
+--
+local os = require('os')
+
+local temp_file = 'some_strange_rare_unique_file_name_for_git_util'
+
+--
+-- Exec a git command.
+-- @param params Table of parameters:
+-- * options - git options.
+-- * cmd - git command.
+-- * args - command arguments.
+-- * dir - working directory.
+-- * fout - write output to the file.
+local function exec_cmd(params)
+ local fout = params.fout
+ local shell_cmd = {'git'}
+ for _, param in pairs({'options', 'cmd', 'args'}) do
+ table.insert(shell_cmd, params[param])
+ end
+ if fout then
+ table.insert(shell_cmd, ' >' .. fout)
+ end
+ shell_cmd = table.concat(shell_cmd, ' ')
+ if params.dir then
+ shell_cmd = string.format('cd %s && %s', params.dir, shell_cmd)
+ end
+ local res = os.execute(shell_cmd)
+ assert(res == 0, 'Git cmd error: ' .. res)
+end
+
+local function log_hashes(params)
+ params.args = "--format='%h' " .. params.args
+ local local_temp_file = string.format('%s/%s', os.getenv('PWD'),
temp_file)
+ params.fout = local_temp_file
+ params.cmd = 'log'
+ exec_cmd(params)
+ local lines = {}
+ for line in io.lines(local_temp_file) do
+ table.insert(lines, line)
+ end
+ os.remove(local_temp_file)
+ return lines
+end
+
+
+return {
+ exec_cmd = exec_cmd,
+ log_hashes = log_hashes
+}
diff --git a/test/lua_libs/util.lua b/test/lua_libs/util.lua
index 2d866df..e9bbd9a 100644
--- a/test/lua_libs/util.lua
+++ b/test/lua_libs/util.lua
@@ -1,5 +1,6 @@
local fiber = require('fiber')
local log = require('log')
+local fio = require('fio')
local function check_error(func, ...)
local pstatus, status, err = pcall(func, ...)
@@ -89,10 +90,29 @@ local function has_same_fields(etalon, data)
return true
end
+-- Git directory of the project. Used in evolution tests to
+-- fetch old versions of vshard.
+local SOURCEDIR = os.getenv('PACKPACK_GIT_SOURCEDIR')
+if not SOURCEDIR then
+ SOURCEDIR = os.getenv('SOURCEDIR')
+end
+if not SOURCEDIR then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ script_path = fio.abspath(script_path)
+ SOURCEDIR = fio.abspath(script_path .. '/../../../')
+end
+
+local BUILDDIR = os.getenv('BUILDDIR')
+if not BUILDDIR then
+ BUILDDIR = SOURCEDIR
+end
+
return {
check_error = check_error,
shuffle_masters = shuffle_masters,
collect_timeouts = collect_timeouts,
wait_master = wait_master,
has_same_fields = has_same_fields,
+ SOURCEDIR = SOURCEDIR,
+ BUILDDIR = BUILDDIR,
}
diff --git a/test/reload_evolution/storage.result
b/test/reload_evolution/storage.result
new file mode 100644
index 0000000..4ffbc26
--- /dev/null
+++ b/test/reload_evolution/storage.result
@@ -0,0 +1,248 @@
+test_run = require('test_run').new()
+---
+...
+git_util = require('git_util')
+---
+...
+util = require('util')
+---
+...
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+---
+...
+evolution_log =
git_util.log_hashes({args='vshard/storage/reload_evolution.lua',
dir=util.SOURCEDIR})
+---
+...
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+---
+...
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+---
+...
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A
--ignore=build` ' .. vshard_copy_path)
+---
+...
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+---
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log]
.. '~1', dir=vshard_copy_path})
+---
+...
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+---
+...
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+---
+...
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+---
+...
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+---
+...
+util = require('util')
+---
+...
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+---
+...
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_force_create(1,
vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+fiber = require('fiber')
+---
+...
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT /
2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+---
+- true
+...
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+---
+...
+vshard.storage.internal.reload_evolution_version
+---
+- null
+...
+box.space.customer:insert({bucket_id_to_move, 1, 'customer_name'})
+---
+- [3000, 1, 'customer_name']
+...
+while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') ==
nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end
+---
+...
+test_run:switch('default')
+---
+- true
+...
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1],
dir=vshard_copy_path})
+---
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+package.loaded["vshard.storage"] = nil
+---
+...
+vshard.storage = require("vshard.storage")
+---
+...
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution:
upgraded to') ~= nil
+---
+- true
+...
+vshard.storage.internal.reload_evolution_version
+---
+- 1
+...
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+---
+- true
+...
+vshard.storage.bucket_force_create(2000)
+---
+- true
+...
+vshard.storage.buckets_info()[2000]
+---
+- status: active
+ id: 2000
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1})
+---
+- true
+- null
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+---
+- true
+...
+vshard.storage.garbage_collector_wakeup()
+---
+...
+fiber = require('fiber')
+---
+...
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+---
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+---
+- true
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1})
+---
+- true
+- null
+...
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+---
+- true
+...
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+---
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+---
+- true
+...
+for i = move_start, move_start + move_cnt - 1 do
box.space._bucket:delete{i} end
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1400
+...
+test_run:switch('storage_1_a')
+---
+- true
+...
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+---
+...
+move_cnt = 100
+---
+...
+vshard.storage.bucket_force_create(move_start, move_cnt)
+---
+- true
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1600
+...
+test_run:switch('storage_2_a')
+---
+- true
+...
+vshard.storage.rebalancer_enable()
+---
+...
+vshard.storage.rebalancer_wakeup()
+---
+...
+wait_rebalancer_state("Rebalance routes are sent", test_run)
+---
+...
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+---
+...
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+---
+- 1500
+...
+test_run:switch('default')
+---
+- true
+...
+test_run:drop_cluster(REPLICASET_2)
+---
+...
+test_run:drop_cluster(REPLICASET_1)
+---
+...
+test_run:cmd('clear filter')
+---
+- true
+...
diff --git a/test/reload_evolution/storage.test.lua
b/test/reload_evolution/storage.test.lua
new file mode 100644
index 0000000..5deb136
--- /dev/null
+++ b/test/reload_evolution/storage.test.lua
@@ -0,0 +1,88 @@
+test_run = require('test_run').new()
+
+git_util = require('git_util')
+util = require('util')
+vshard_copy_path = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+evolution_log =
git_util.log_hashes({args='vshard/storage/reload_evolution.lua',
dir=util.SOURCEDIR})
+-- Cleanup the directory after a previous build.
+_ = os.execute('rm -rf ' .. vshard_copy_path)
+-- 1. `git worktree` cannot be used because PACKPACK mounts
+-- `/source/` in `ro` mode.
+-- 2. Just `cp -rf` cannot be used due to a little different
+-- behavior in Centos 7.
+_ = os.execute('mkdir ' .. vshard_copy_path)
+_ = os.execute("cd " .. util.SOURCEDIR .. ' && cp -rf `ls -A
--ignore=build` ' .. vshard_copy_path)
+-- Checkout the first commit with a reload_evolution mechanism.
+git_util.exec_cmd({cmd='checkout', args='-f', dir=vshard_copy_path})
+git_util.exec_cmd({cmd='checkout', args=evolution_log[#evolution_log]
.. '~1', dir=vshard_copy_path})
+
+REPLICASET_1 = { 'storage_1_a', 'storage_1_b' }
+REPLICASET_2 = { 'storage_2_a', 'storage_2_b' }
+test_run:create_cluster(REPLICASET_1, 'reload_evolution')
+test_run:create_cluster(REPLICASET_2, 'reload_evolution')
+util = require('util')
+util.wait_master(test_run, REPLICASET_1, 'storage_1_a')
+util.wait_master(test_run, REPLICASET_2, 'storage_2_a')
+
+test_run:switch('storage_1_a')
+vshard.storage.bucket_force_create(1,
vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+
+test_run:switch('storage_2_a')
+fiber = require('fiber')
+vshard.storage.bucket_force_create(vshard.consts.DEFAULT_BUCKET_COUNT /
2 + 1, vshard.consts.DEFAULT_BUCKET_COUNT / 2)
+bucket_id_to_move = vshard.consts.DEFAULT_BUCKET_COUNT
+vshard.storage.internal.reload_evolution_version
+box.space.customer:insert({bucket_id_to_move, 1, 'customer_name'})
+while test_run:grep_log('storage_2_a', 'The cluster is balanced ok') ==
nil do vshard.storage.rebalancer_wakeup() fiber.sleep(0.1) end
+
+test_run:switch('default')
+git_util.exec_cmd({cmd='checkout', args=evolution_log[1],
dir=vshard_copy_path})
+
+test_run:switch('storage_2_a')
+package.loaded["vshard.storage"] = nil
+vshard.storage = require("vshard.storage")
+test_run:grep_log('storage_2_a', 'vshard.storage.reload_evolution:
upgraded to') ~= nil
+vshard.storage.internal.reload_evolution_version
+-- Make sure storage operates well.
+vshard.storage.bucket_force_drop(2000)
+vshard.storage.bucket_force_create(2000)
+vshard.storage.buckets_info()[2000]
+vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1})
+vshard.storage.bucket_send(bucket_id_to_move, replicaset1_uuid)
+vshard.storage.garbage_collector_wakeup()
+fiber = require('fiber')
+while box.space._bucket:get({bucket_id_to_move}) do fiber.sleep(0.01) end
+test_run:switch('storage_1_a')
+vshard.storage.bucket_send(bucket_id_to_move, replicaset2_uuid)
+test_run:switch('storage_2_a')
+vshard.storage.call(bucket_id_to_move, 'read', 'customer_lookup', {1})
+-- Check info() does not fail.
+vshard.storage.info() ~= nil
+
+--
+-- Send buckets to create a disbalance. Wait until the rebalancer
+-- repairs it. Similar to `tests/rebalancer/rebalancer.test.lua`.
+--
+vshard.storage.rebalancer_disable()
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+assert(move_start + move_cnt < vshard.consts.DEFAULT_BUCKET_COUNT)
+for i = move_start, move_start + move_cnt - 1 do
box.space._bucket:delete{i} end
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_1_a')
+move_start = vshard.consts.DEFAULT_BUCKET_COUNT / 2 + 1
+move_cnt = 100
+vshard.storage.bucket_force_create(move_start, move_cnt)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+test_run:switch('storage_2_a')
+vshard.storage.rebalancer_enable()
+vshard.storage.rebalancer_wakeup()
+wait_rebalancer_state("Rebalance routes are sent", test_run)
+wait_rebalancer_state('The cluster is balanced ok', test_run)
+box.space._bucket.index.status:count({vshard.consts.BUCKET.ACTIVE})
+
+test_run:switch('default')
+test_run:drop_cluster(REPLICASET_2)
+test_run:drop_cluster(REPLICASET_1)
+test_run:cmd('clear filter')
diff --git a/test/reload_evolution/storage_1_a.lua
b/test/reload_evolution/storage_1_a.lua
new file mode 100755
index 0000000..d8bda60
--- /dev/null
+++ b/test/reload_evolution/storage_1_a.lua
@@ -0,0 +1,154 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+
+local fio = require('fio')
+
+-- Get instance name
+local fio = require('fio')
+local log = require('log')
+local NAME = fio.basename(arg[0], '.lua')
+local fiber = require('fiber')
+local util = require('util')
+
+-- Run one storage on a different vshard version.
+-- To do that, place vshard src to
+-- BUILDDIR/test/var/vshard_git_tree_copy/.
+if NAME == 'storage_2_a' then
+ local script_path = debug.getinfo(1).source:match("@?(.*/)")
+ vshard_copy = util.BUILDDIR .. '/test/var/vshard_git_tree_copy'
+ package.path = string.format(
+ '%s/?.lua;%s/?/init.lua;%s',
+ vshard_copy, vshard_copy, package.path
+ )
+end
+
+-- Check if we are running under test-run
+if os.getenv('ADMIN') then
+ test_run = require('test_run').new()
+ require('console').listen(os.getenv('ADMIN'))
+end
+
+-- Call a configuration provider
+cfg = require('localcfg')
+-- Name to uuid map
+names = {
+ ['storage_1_a'] = '8a274925-a26d-47fc-9e1b-af88ce939412',
+ ['storage_1_b'] = '3de2e3e1-9ebe-4d0d-abb1-26d301b84633',
+ ['storage_2_a'] = '1e02ae8a-afc0-4e91-ba34-843a356b8ed7',
+ ['storage_2_b'] = '001688c3-66f8-4a31-8e19-036c17d489c2',
+}
+
+replicaset1_uuid = 'cbf06940-0790-498b-948d-042b62cf3d29'
+replicaset2_uuid = 'ac522f65-aa94-4134-9f64-51ee384f1a54'
+replicasets = {replicaset1_uuid, replicaset2_uuid}
+
+-- Start the database with sharding
+vshard = require('vshard')
+vshard.storage.cfg(cfg, names[NAME])
+
+box.once("testapp:schema:1", function()
+ local customer = box.schema.space.create('customer')
+ customer:format({
+ {'customer_id', 'unsigned'},
+ {'bucket_id', 'unsigned'},
+ {'name', 'string'},
+ })
+ customer:create_index('customer_id', {parts = {'customer_id'}})
+ customer:create_index('bucket_id', {parts = {'bucket_id'}, unique =
false})
+
+ local account = box.schema.space.create('account')
+ account:format({
+ {'account_id', 'unsigned'},
+ {'customer_id', 'unsigned'},
+ {'bucket_id', 'unsigned'},
+ {'balance', 'unsigned'},
+ {'name', 'string'},
+ })
+ account:create_index('account_id', {parts = {'account_id'}})
+ account:create_index('customer_id', {parts = {'customer_id'},
unique = false})
+ account:create_index('bucket_id', {parts = {'bucket_id'}, unique =
false})
+ box.snapshot()
+
+ box.schema.func.create('customer_lookup')
+ box.schema.role.grant('public', 'execute', 'function',
'customer_lookup')
+ box.schema.func.create('customer_add')
+ box.schema.role.grant('public', 'execute', 'function', 'customer_add')
+ box.schema.func.create('echo')
+ box.schema.role.grant('public', 'execute', 'function', 'echo')
+ box.schema.func.create('sleep')
+ box.schema.role.grant('public', 'execute', 'function', 'sleep')
+ box.schema.func.create('raise_luajit_error')
+ box.schema.role.grant('public', 'execute', 'function',
'raise_luajit_error')
+ box.schema.func.create('raise_client_error')
+ box.schema.role.grant('public', 'execute', 'function',
'raise_client_error')
+end)
+
+function customer_add(customer)
+ box.begin()
+ box.space.customer:insert({customer.customer_id, customer.bucket_id,
+ customer.name})
+ for _, account in ipairs(customer.accounts) do
+ box.space.account:insert({
+ account.account_id,
+ customer.customer_id,
+ customer.bucket_id,
+ 0,
+ account.name
+ })
+ end
+ box.commit()
+ return true
+end
+
+function customer_lookup(customer_id)
+ if type(customer_id) ~= 'number' then
+ error('Usage: customer_lookup(customer_id)')
+ end
+
+ local customer = box.space.customer:get(customer_id)
+ if customer == nil then
+ return nil
+ end
+ customer = {
+ customer_id = customer.customer_id;
+ name = customer.name;
+ }
+ local accounts = {}
+ for _, account in
box.space.account.index.customer_id:pairs(customer_id) do
+ table.insert(accounts, {
+ account_id = account.account_id;
+ name = account.name;
+ balance = account.balance;
+ })
+ end
+ customer.accounts = accounts;
+ return customer
+end
+
+function echo(...)
+ return ...
+end
+
+function sleep(time)
+ fiber.sleep(time)
+ return true
+end
+
+function raise_luajit_error()
+ assert(1 == 2)
+end
+
+function raise_client_error()
+ box.error(box.error.UNKNOWN)
+end
+
+function wait_rebalancer_state(state, test_run)
+ log.info(string.rep('a', 1000))
+ vshard.storage.rebalancer_wakeup()
+ while not test_run:grep_log(NAME, state, 1000) do
+ fiber.sleep(0.1)
+ vshard.storage.rebalancer_wakeup()
+ end
+end
diff --git a/test/reload_evolution/storage_1_b.lua
b/test/reload_evolution/storage_1_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_1_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_a.lua
b/test/reload_evolution/storage_2_a.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_a.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/storage_2_b.lua
b/test/reload_evolution/storage_2_b.lua
new file mode 120000
index 0000000..02572da
--- /dev/null
+++ b/test/reload_evolution/storage_2_b.lua
@@ -0,0 +1 @@
+storage_1_a.lua
\ No newline at end of file
diff --git a/test/reload_evolution/suite.ini
b/test/reload_evolution/suite.ini
new file mode 100644
index 0000000..bb5435b
--- /dev/null
+++ b/test/reload_evolution/suite.ini
@@ -0,0 +1,6 @@
+[default]
+core = tarantool
+description = Reload evolution tests
+script = test.lua
+is_parallel = False
+lua_libs = ../lua_libs/util.lua ../lua_libs/git_util.lua
../../example/localcfg.lua
diff --git a/test/reload_evolution/test.lua b/test/reload_evolution/test.lua
new file mode 100644
index 0000000..ad0543a
--- /dev/null
+++ b/test/reload_evolution/test.lua
@@ -0,0 +1,9 @@
+#!/usr/bin/env tarantool
+
+require('strict').on()
+
+box.cfg{
+ listen = os.getenv("LISTEN"),
+}
+
+require('console').listen(os.getenv('ADMIN'))
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index 07bd00c..3bec09f 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -10,6 +10,7 @@ if rawget(_G, MODULE_INTERNALS) then
local vshard_modules = {
'vshard.consts', 'vshard.error', 'vshard.cfg',
'vshard.replicaset', 'vshard.util',
+ 'vshard.storage.reload_evolution'
}
for _, module in pairs(vshard_modules) do
package.loaded[module] = nil
@@ -20,12 +21,16 @@ local lerror = require('vshard.error')
local lcfg = require('vshard.cfg')
local lreplicaset = require('vshard.replicaset')
local util = require('vshard.util')
+local reload_evolution = require('vshard.storage.reload_evolution')
local M = rawget(_G, MODULE_INTERNALS)
if not M then
--
-- The module is loaded for the first time.
--
+ -- !!!WARNING: any change of this table must be reflected in
+ -- `vshard.storage.reload_evolution` module to guarantee
+ -- reloadability of the module.
M = {
---------------- Common module attributes ----------------
-- The last passed configuration.
@@ -105,6 +110,11 @@ if not M then
-- a destination replicaset must drop already received
-- data.
rebalancer_sending_bucket = 0,
+
+ ------------------------- Reload -------------------------
+ -- Version of the loaded module. This number is used on
+ -- reload to determine which upgrade scripts to run.
+ reload_evolution_version = reload_evolution.version,
}
end
@@ -1863,6 +1873,7 @@ end
if not rawget(_G, MODULE_INTERNALS) then
rawset(_G, MODULE_INTERNALS, M)
else
+ reload_evolution.upgrade(M)
storage_cfg(M.current_cfg, M.this_replica.uuid)
M.module_version = M.module_version + 1
end
diff --git a/vshard/storage/reload_evolution.lua
b/vshard/storage/reload_evolution.lua
new file mode 100644
index 0000000..f25ad49
--- /dev/null
+++ b/vshard/storage/reload_evolution.lua
@@ -0,0 +1,58 @@
+--
+-- This module is used to upgrade the vshard.storage on the fly.
+-- It updates internal Lua structures in case they are changed
+-- in a commit.
+--
+local log = require('log')
+
+--
+-- Array of upgrade functions.
+-- migrations[version] = function which upgrades module version
+-- from `version` to `version + 1`.
+--
+local migrations = {}
+
+-- Initialize reload_upgrade mechanism
+migrations[#migrations + 1] = function (M)
+ -- Code to update Lua objects.
+end
+
+--
+-- Perform an update based on a version stored in `M` (internals).
+-- @param M Old module internals which should be updated.
+--
+local function upgrade(M)
+ local start_version = M.reload_evolution_version or 1
+ if start_version > #migrations then
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'auto-downgrade is not implemented; ' ..
+ 'loaded version is %d, upgrade script version is %d',
+ start_version, #migrations
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ for i = start_version, #migrations do
+ local ok, err = pcall(migrations[i], M)
+ if ok then
+ log.info('vshard.storage.reload_evolution: upgraded to %d
version',
+ i)
+ else
+ local err_msg = string.format(
+ 'vshard.storage.reload_evolution: ' ..
+ 'error during upgrade to %d version: %s', i, err
+ )
+ log.error(err_msg)
+ error(err_msg)
+ end
+ -- Update the version just after upgrade to have an
+ -- actual version in case of an error.
+ M.reload_evolution_version = i
+ end
+end
+
+return {
+ version = #migrations,
+ upgrade = upgrade,
+}
More information about the Tarantool-patches
mailing list