From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <tarantool-patches-bounces@dev.tarantool.org>
Received: from [87.239.111.99] (localhost [127.0.0.1])
	by dev.tarantool.org (Postfix) with ESMTP id 3DF176EC5F;
	Thu, 15 Apr 2021 09:04:56 +0300 (MSK)
DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3DF176EC5F
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev;
	t=1618466696; bh=TnvtoXnm8U4/vQK1naKSNJPXP8kgu60/styVsl/hmto=;
	h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe:
	 List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc:
	 From;
	b=Sb39/HJPtr/0oh/rHa3009VGDwni/ag/VJZiGeKOH5aY3krXedt2TekrVSCncNAJ6
	 SXNCb1iaVu8mruF0CSxbPJ76IQ/eVqnh9JNXsQt1KpSF7p9EwaYN0q4eammF4Y1rW3
	 sYOWL8fT8lNqBU0Or0RvWEymfIJGUFDzAyXyT/PE=
Received: from smtp49.i.mail.ru (smtp49.i.mail.ru [94.100.177.109])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by dev.tarantool.org (Postfix) with ESMTPS id A257C6C1AC
 for <tarantool-patches@dev.tarantool.org>;
 Thu, 15 Apr 2021 09:03:57 +0300 (MSK)
DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A257C6C1AC
Received: by smtp49.i.mail.ru with esmtpa (envelope-from
 <alexander.turenko@tarantool.org>)
 id 1lWv6a-0005Jq-L5; Thu, 15 Apr 2021 09:03:57 +0300
To: Leonid Vasiliev <lvasiliev@tarantool.org>
Date: Thu, 15 Apr 2021 09:04:12 +0300
Message-Id: <6177e4e5db3c48fd9bad5166a9ce50de33c952da.1618466243.git.alexander.turenko@tarantool.org>
X-Mailer: git-send-email 2.30.0
In-Reply-To: <cover.1618466243.git.alexander.turenko@tarantool.org>
References: <cover.1618466243.git.alexander.turenko@tarantool.org>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-7564579A: B8F34718100C35BD
X-77F55803: 4F1203BC0FB41BD92FFCB8E6708E7480257C85EA0BB7A95D5E28B957962BB550182A05F53808504052A2AD6851BBC890872F09A54604CFA69A0B1F3B642C13D0F3EBF0E42787B6C1
X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE74D0D2DEF2EB846B0EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063703E3935C5A8197E98638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2B72D2C7B864615ADDFE346BDB5B99895F04B652EEC242312D2E47CDBA5A96583C09775C1D3CA48CF7EF884183F8E4D67117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE767883B903EA3BAEA9FA2833FD35BB23DF004C90652538430302FCEF25BFAB3454AD6D5ED66289B5278DA827A17800CE793C8D1582BD8047CD32BA5DBAC0009BE395957E7521B51C20BC6067A898B09E4090A508E0FED6299176DF2183F8FC7C00B97EE868906737BCD04E86FAF290E2D7E9C4E3C761E06A71DD303D21008E298D5E8D9A59859A8B6B372FE9A2E580EFC725E5C173C3A84C3ADB12FF6AD8D9AFD35872C767BF85DA2F004C90652538430E4A6367B16DE6309
X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CB69F9342289A40B37B0DB985CB63F1963AA3E7A079A0F0BD9C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF0417BEADF48D1460699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356
X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D344B4608D01B1CB59A59F40446BF34D2B676D99B3E091002CA2CFA68C5D8D048AD5CCD1EABDD1969291D7E09C32AA3244CBCBCEF57652D92BA40688DF275A8862B05AB220A9D022EBC927AC6DF5659F194
X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXi23FRaUALWyK5VnnJJpW4e
X-Mailru-Sender: FFAA8E4AEE17E37C3731A083A1A85ADE34F32CD18DBEB96635E398DECD265993B7EA9FE7735C3DBFC664A44C781FCEA7C77752E0C033A69EDF9F2CE1E9CF805D8CD356D4F938FF726C18EFA0BB12DBB0
X-Mras: Ok
Subject: [Tarantool-patches] [PATCH 2/2] box: set box.cfg options via
 environment variables
X-BeenThere: tarantool-patches@dev.tarantool.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Tarantool development patches <tarantool-patches.dev.tarantool.org>
List-Unsubscribe: <https://lists.tarantool.org/mailman/options/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=unsubscribe>
List-Archive: <https://lists.tarantool.org/pipermail/tarantool-patches/>
List-Post: <mailto:tarantool-patches@dev.tarantool.org>
List-Help: <mailto:tarantool-patches-request@dev.tarantool.org?subject=help>
List-Subscribe: <https://lists.tarantool.org/mailman/listinfo/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=subscribe>
From: Alexander Turenko via Tarantool-patches
 <tarantool-patches@dev.tarantool.org>
Reply-To: Alexander Turenko <alexander.turenko@tarantool.org>
Cc: tarantool-patches@dev.tarantool.org,
 Alexander Turenko <alexander.turenko@tarantool.org>
Errors-To: tarantool-patches-bounces@dev.tarantool.org
Sender: "Tarantool-patches" <tarantool-patches-bounces@dev.tarantool.org>

From: Roman Khabibov <roman.habibov@tarantool.org>

Add ability to set box.cfg options via environment variables. These
variables should have name `TT_<OPTION>`. When Tarantool instance is
started under tarantoolctl utility, environment variables have higher
priority than tarantoolctl configuration file.

Closes #5602

Co-authored-by: Leonid Vasiliev <lvasiliev@tarantool.org>
Co-authored-by: Alexander Turenko <alexander.turenko@tarantool.org>

@TarantoolBot document
Title: Set box.cfg options via environment variables

Now, it is possible to set box.cfg options via environment variables.
The name of variable should correspond the following pattern:
`TT_<NAME>`, where `<NAME>` is uppercase box.cfg option name. For
example: `TT_LISTEN`, `TT_READAHEAD`.

Array values are separated by comma. Example:

```sh
export TT_REPLICATION=localhost:3301,localhost:3302
```

An empty variable is the same as unset one.
---
 changelogs/unreleased/environment-cfg.md      |  7 ++
 extra/dist/tarantoolctl.in                    | 19 +++-
 src/box/lua/load_cfg.lua                      | 99 +++++++++++++++++++
 .../gh-5602-environment-cfg-test-cases.lua    | 71 +++++++++++++
 .../gh-5602-environment-vars-cfg.result       | 33 +++++++
 .../gh-5602-environment-vars-cfg.test.lua     | 57 +++++++++++
 6 files changed, 285 insertions(+), 1 deletion(-)
 create mode 100755 changelogs/unreleased/environment-cfg.md
 create mode 100755 test/box-tap/gh-5602-environment-cfg-test-cases.lua
 create mode 100644 test/box-tap/gh-5602-environment-vars-cfg.result
 create mode 100755 test/box-tap/gh-5602-environment-vars-cfg.test.lua

diff --git a/changelogs/unreleased/environment-cfg.md b/changelogs/unreleased/environment-cfg.md
new file mode 100755
index 000000000..f1d37f6e6
--- /dev/null
+++ b/changelogs/unreleased/environment-cfg.md
@@ -0,0 +1,7 @@
+## feature/core
+
+* Now, it is possible to set box.cfg options with environment variables
+  (gh-5602).
+
+  The priority of sources of configuration options is the following (from low
+  to high): default, tarantoolctl, environment, box.cfg{}.
diff --git a/extra/dist/tarantoolctl.in b/extra/dist/tarantoolctl.in
index 0726e7f46..fc0885fbe 100755
--- a/extra/dist/tarantoolctl.in
+++ b/extra/dist/tarantoolctl.in
@@ -449,10 +449,27 @@ local function wrapper_cfg(cfg)
         end
     end
 
+    -- Collect box.cfg options from environment variables if
+    -- tarantool supports this feature.
+    local ok, env_cfg = pcall(function()
+        return box.internal.cfg and box.internal.cfg.env or {}
+    end)
+    if not ok then
+        log.error(tostring(env_cfg))
+        os.exit(1)
+    end
+
     cfg = cfg or {}
     for i, v in pairs(default_cfg) do
         if cfg[i] == nil then
-            cfg[i] = v
+            -- If an option is set using an environment variable,
+            -- prefer this value. Otherwise fallback to
+            -- tarantoolctl's default value.
+            --
+            -- If we'll not do it there, the tarantoolctl's
+            -- default will rewrite the value passed via the
+            -- environment variable.
+            cfg[i] = env_cfg[i] or v
         end
     end
     -- force these startup options
diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua
index 72e889071..11fa98586 100644
--- a/src/box/lua/load_cfg.lua
+++ b/src/box/lua/load_cfg.lua
@@ -123,6 +123,10 @@ local module_cfg = {
 -- provide some API with type enumeration or
 -- similar. Currently it has use for environment
 -- processing only.
+--
+-- get_option_from_env() leans on the set of types
+-- in use: don't forget to update it when add a new
+-- type or a combination of types here.
 local module_cfg_type = {
     -- logging
     log                 = 'string',
@@ -133,6 +137,10 @@ local module_cfg_type = {
 
 -- types of available options
 -- could be comma separated lua types or 'any' if any type is allowed
+--
+-- get_option_from_env() leans on the set of types in use: don't
+-- forget to update it when add a new type or a combination of
+-- types here.
 local template_cfg = {
     listen              = 'string, number',
     memtx_memory        = 'number',
@@ -539,6 +547,18 @@ local function prepare_cfg(cfg, default_cfg, template_cfg,
     return new_cfg
 end
 
+-- Transfer options from env_cfg to cfg.
+local function apply_env_cfg(cfg, env_cfg)
+    -- Add options passed through environment variables.
+    -- Here we only add options without overloading the ones set
+    -- by the user.
+    for k, v in pairs(env_cfg) do
+        if cfg[k] == nil then
+            cfg[k] = v
+        end
+    end
+end
+
 local function apply_default_cfg(cfg, default_cfg, module_cfg)
     for k,v in pairs(default_cfg) do
         if cfg[k] == nil then
@@ -682,6 +702,10 @@ local function load_cfg(cfg)
     end
 
     cfg = upgrade_cfg(cfg, translate_cfg)
+
+    -- Set options passed through environment variables.
+    apply_env_cfg(cfg, box.internal.cfg.env)
+
     cfg = prepare_cfg(cfg, default_cfg, template_cfg,
                       module_cfg, modify_cfg)
     apply_default_cfg(cfg, default_cfg, module_cfg);
@@ -794,6 +818,81 @@ box_load_and_execute = function(...)
 end
 box.execute = box_load_and_execute
 
+--
+-- Parse TT_* environment variable that corresponds to given
+-- option.
+--
+local function get_option_from_env(option)
+    local param_type = template_cfg[option]
+    assert(type(param_type) == 'string')
+
+    if param_type == 'module' then
+        -- Parameter from module.
+        param_type = module_cfg_type[option]
+    end
+
+    local env_var_name = 'TT_' .. option:upper()
+    local raw_value = os.getenv(env_var_name)
+
+    if raw_value == nil or raw_value == '' then
+        return nil
+    end
+
+    local err_msg_fmt = 'Environment variable %s has ' ..
+        'incorrect value for option "%s": should be %s'
+
+    -- This code lean on the existing set of template_cfg and
+    -- module_cfg_type types for simplicity.
+    if param_type:find('table') and raw_value:find(',') then
+        assert(not param_type:find('boolean'))
+        local res = {}
+        for i, v in ipairs(raw_value:split(',')) do
+            res[i] = tonumber(v) or v
+        end
+        return res
+    elseif param_type:find('boolean') then
+        assert(param_type == 'boolean')
+        if raw_value:lower() == 'false' then
+            return false
+        elseif raw_value:lower() == 'true' then
+            return true
+        end
+        error(err_msg_fmt:format(env_var_name, option, '"true" or "false"'))
+    elseif param_type == 'number' then
+        local res = tonumber(raw_value)
+        if res == nil then
+            error(err_msg_fmt:format(env_var_name, option,
+                'convertible to a number'))
+        end
+        return res
+    elseif param_type:find('number') then
+        assert(not param_type:find('boolean'))
+        return tonumber(raw_value) or raw_value
+    else
+        assert(param_type == 'string')
+        return raw_value
+    end
+end
+
+--
+-- Read box configuration from environment variables.
+--
+box.internal.cfg = setmetatable({}, {
+    __index = function(self, key)
+        if key == 'env' then
+            local res = {}
+            for option, _ in pairs(template_cfg) do
+                res[option] = get_option_from_env(option)
+            end
+            return res
+        end
+        assert(false)
+    end,
+    __newindex = function(self, key, value) -- luacheck: no unused args
+        error('Attempt to modify a read-only table')
+    end,
+})
+
 -- gh-810:
 -- hack luajit default cpath
 -- commented out because we fixed luajit to build properly, see
diff --git a/test/box-tap/gh-5602-environment-cfg-test-cases.lua b/test/box-tap/gh-5602-environment-cfg-test-cases.lua
new file mode 100755
index 000000000..72031778b
--- /dev/null
+++ b/test/box-tap/gh-5602-environment-cfg-test-cases.lua
@@ -0,0 +1,71 @@
+local tap = require('tap')
+
+local test = tap.test('gh-5602')
+
+local status, err = pcall(box.cfg, {background = false, vinyl_timeout = 70.1})
+
+-- Check that environment cfg values are set correctly.
+if arg[1] == '1' then
+    test:plan(6)
+    test:ok(status, 'box.cfg is successful')
+    test:is(box.cfg['listen'], '3301', 'listen')
+    test:is(box.cfg['readahead'], 10000, 'readahead')
+    test:is(box.cfg['strip_core'], false, 'strip_core')
+    test:is(box.cfg['log_format'], 'json', 'log_format is not set')
+    test:is(box.cfg['log_nonblock'], false, 'log_nonblock')
+end
+if arg[1] == '2' then
+    test:plan(7)
+    test:ok(status, 'box.cfg is successful')
+    test:is(box.cfg['listen'], '3301', 'listen')
+    local replication = box.cfg['replication']
+    test:is(type(replication), 'table', 'replication is table')
+    test:ok(replication[1] == '0.0.0.0:12345' or
+            replication[1] == '1.1.1.1:12345', 'replication URI 1')
+    test:ok(replication[2] == '0.0.0.0:12345' or
+            replication[2] == '1.1.1.1:12345', 'replication URI 2')
+    test:is(box.cfg['replication_connect_timeout'], 0.01,
+            'replication_connect_timeout')
+    test:is(box.cfg['replication_synchro_quorum'], '4 + 1',
+            'replication_synchro_quorum')
+end
+
+-- Check that box.cfg{} values are more prioritized than
+-- environment cfg values.
+if arg[1] == '3' then
+    test:plan(3)
+    test:ok(status, 'box.cfg is successful')
+    test:is(box.cfg['background'], false,
+            'box.cfg{} background value is prioritized')
+    test:is(box.cfg['vinyl_timeout'], 70.1,
+            'box.cfg{} vinyl_timeout value is prioritized')
+end
+
+local err_msg_fmt = 'Environment variable TT_%s has incorrect value for ' ..
+    'option "%s": should be %s'
+
+-- Check bad environment cfg values.
+if arg[1] == '4' then
+    test:plan(2)
+    test:ok(not status, 'box.cfg is not successful')
+    local exp_err = err_msg_fmt:format('SQL_CACHE_SIZE', 'sql_cache_size',
+        'convertible to a number')
+    local err_msg = tostring(err)
+    while err_msg:find('^.-:.-: ') do
+        err_msg = err_msg:gsub('^.-:.-: ', '')
+    end
+    test:is(err_msg, exp_err, 'bad sql_cache_size value')
+end
+if arg[1] == '5' then
+    test:plan(2)
+    test:ok(not status, 'box.cfg is not successful')
+    local exp_err = err_msg_fmt:format('STRIP_CORE', 'strip_core',
+        '"true" or "false"')
+    local err_msg = tostring(err)
+    while err_msg:find('^.-:.-: ') do
+        err_msg = err_msg:gsub('^.-:.-: ', '')
+    end
+    test:is(err_msg, exp_err, 'bad strip_core value')
+end
+
+os.exit(test:check() and 0 or 1)
diff --git a/test/box-tap/gh-5602-environment-vars-cfg.result b/test/box-tap/gh-5602-environment-vars-cfg.result
new file mode 100644
index 000000000..3050d5747
--- /dev/null
+++ b/test/box-tap/gh-5602-environment-vars-cfg.result
@@ -0,0 +1,33 @@
+TAP version 13
+1..6
+ok - box.cfg is successful
+ok - listen
+ok - readahead
+ok - strip_core
+ok - log_format is not set
+ok - log_nonblock
+TAP version 13
+1..7
+ok - box.cfg is successful
+ok - listen
+ok - replication is table
+ok - replication URI 1
+ok - replication URI 2
+ok - replication_connect_timeout
+ok - replication_synchro_quorum
+TAP version 13
+1..3
+ok - box.cfg is successful
+ok - box.cfg{} background value is prioritized
+ok - box.cfg{} vinyl_timeout value is prioritized
+TAP version 13
+1..2
+ok - box.cfg is not successful
+ok - bad sql_cache_size value
+TAP version 13
+1..2
+ok - box.cfg is not successful
+ok - bad strip_core value
+TAP version 13
+1..1
+ok - exit status list
diff --git a/test/box-tap/gh-5602-environment-vars-cfg.test.lua b/test/box-tap/gh-5602-environment-vars-cfg.test.lua
new file mode 100755
index 000000000..be1e402ee
--- /dev/null
+++ b/test/box-tap/gh-5602-environment-vars-cfg.test.lua
@@ -0,0 +1,57 @@
+#!/usr/bin/env tarantool
+
+local os = require('os')
+local fio = require('fio')
+local tap = require('tap')
+
+local test = tap.test('gh-5602')
+
+-- gh-5602: Check that environment cfg variables working.
+local TARANTOOL_PATH = arg[-1]
+local script_name = 'gh-5602-environment-cfg-test-cases.lua'
+local path_to_script = fio.pathjoin(
+        os.getenv('PWD'),
+        'box-tap',
+        script_name)
+
+-- Generate a shell command like
+-- `FOO=x BAR=y /path/to/tarantool /path/to/script.lua 42`.
+local function shell_command(case, i)
+    return ('%s %s %s %d'):format(
+        case,
+        TARANTOOL_PATH,
+        path_to_script,
+        i)
+end
+
+local cases = {
+    ('%s %s %s %s %s'):format(
+        'TT_LISTEN=3301',
+        'TT_READAHEAD=10000',
+        'TT_STRIP_CORE=false',
+        'TT_LOG_FORMAT=json',
+        'TT_LOG_NONBLOCK=false'),
+    ('%s %s %s %s'):format(
+        'TT_LISTEN=3301',
+        'TT_REPLICATION=0.0.0.0:12345,1.1.1.1:12345',
+        'TT_REPLICATION_CONNECT_TIMEOUT=0.01',
+        'TT_REPLICATION_SYNCHRO_QUORUM=\'4 + 1\''),
+    'TT_BACKGROUND=true TT_VINYL_TIMEOUT=60.1',
+    'TT_SQL_CACHE_SIZE=a',
+    'TT_STRIP_CORE=a',
+}
+
+test:plan(1)
+local exit_status_list = {}
+local exit_status_list_exp = {}
+for i, case in ipairs(cases) do
+    local tmpdir = fio.tempdir()
+    local new_path = fio.pathjoin(tmpdir, script_name)
+    fio.copyfile(path_to_script, new_path)
+    exit_status_list[i] = os.execute(shell_command(case, i))
+    exit_status_list_exp[i] = 0
+end
+
+test:is_deeply(exit_status_list, exit_status_list_exp, 'exit status list')
+
+os.exit(test:check() and 0 or 1)
-- 
2.30.0