From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <tarantool-patches-bounces@dev.tarantool.org>
Received: from [87.239.111.99] (localhost [127.0.0.1])
	by dev.tarantool.org (Postfix) with ESMTP id BC54F6EC56;
	Sun, 14 Mar 2021 18:35:09 +0300 (MSK)
DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BC54F6EC56
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev;
	t=1615736109; bh=PnMtAqxuzp/qfAS2F97mNiT8kLM2YOb10A856XgtRJ4=;
	h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post:
	 List-Help:List-Subscribe:From:Reply-To:From;
	b=so20EXNvRu1MBRhqpoIHjTAvwAYygp1L7NS1VOs4XLKkXeVhz5qgQyucHzs12CF/Y
	 8DUPH9mnbrSvXf6iwtpZBGBhKVcSs1HhpsbbH7F/oUg1P+kJC1AzeUBby1IxKEGpYV
	 xk8jPM4+63efp7WueDDC0m2AWYboX62SFFaPNyCQ=
Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256
 bits)) (No client certificate requested)
 by dev.tarantool.org (Postfix) with ESMTPS id BE92A6EC56
 for <tarantool-patches@dev.tarantool.org>;
 Sun, 14 Mar 2021 18:35:07 +0300 (MSK)
DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BE92A6EC56
Received: by smtpng1.m.smailru.net with esmtpa (envelope-from
 <roman.habibov@tarantool.org>)
 id 1lLSlg-0001hg-6h; Sun, 14 Mar 2021 18:35:00 +0300
To: tarantool-patches@dev.tarantool.org
Date: Sun, 14 Mar 2021 18:34:57 +0300
Message-Id: <20210314153457.85909-1-roman.habibov@tarantool.org>
X-Mailer: git-send-email 2.24.3 (Apple Git-128)
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-7564579A: 646B95376F6C166E
X-77F55803: 4F1203BC0FB41BD9D3134714A9BDB69B5A6145416490033C23E814B02A2DD10F00894C459B0CD1B906B8BB30D456ABF86E20919F69C87021ACBABCD93D5C7C5A4787F701E01FBF3D
X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE743AE26858062A689EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637737629E79DA7E02D8638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C7FF294EE7CC9FD509CE85F348F54F9771469583CCF70E64EA471835C12D1D9774AD6D5ED66289B5278DA827A17800CE7ABB305BD10C6E5099FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3A12191B5F2BB8629117882F4460429728AD0CFFFB425014E868A13BD56FB6657A7F4EDE966BC389F9E8FC8737B5C224955EC1579764E9EAF089D37D7C0E48F6CCF19DD082D7633A0E7DDDDC251EA7DABAAAE862A0553A39223F8577A6DFFEA7CDC7F9DB2523BC7CDC4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0F05F538519369F3743B503F486389A921A5CC5B56E945C8DA
X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C7FF294EE7CC9FD509CE85F348F54F9771469583CCF70E64E9C2B6934AE262D3EE7EAB7254005DCED94B1027BF495DC061E0A4E2319210D9B64D260DF9561598F01A9E91200F654B02DA3D96DA0CEF5C48E8E86DC7131B365E7726E8460B7C23C
X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D344888A9AABCDD322536C3AA3F2719CF67E13AC0342AC468F1D578F5DC93E3CF5FE4553CFDA9AE07D91D7E09C32AA3244C3E6ADBA58715F1ADBEA756CB0F70A761C86C126E7119A0FE729B2BEF169E0186
X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj4I/mOTH/3lHC2N3rUKJIcg==
X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822F3D88D13C9E5236FF80987B70264F5FE0F96A0E8EF440D4CA021B31A5B48C7CFFB559BB5D741EB9609AE011F3211FEEF240331F90058701C67EA787935ED9F1B
X-Mras: Ok
Subject: [Tarantool-patches] [PATCH] box: set cfg via environment variables
X-BeenThere: tarantool-patches@dev.tarantool.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Tarantool development patches <tarantool-patches.dev.tarantool.org>
List-Unsubscribe: <https://lists.tarantool.org/mailman/options/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=unsubscribe>
List-Archive: <https://lists.tarantool.org/pipermail/tarantool-patches/>
List-Post: <mailto:tarantool-patches@dev.tarantool.org>
List-Help: <mailto:tarantool-patches-request@dev.tarantool.org?subject=help>
List-Subscribe: <https://lists.tarantool.org/mailman/listinfo/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=subscribe>
From: Roman Khabibov via Tarantool-patches
 <tarantool-patches@dev.tarantool.org>
Reply-To: Roman Khabibov <roman.habibov@tarantool.org>
Errors-To: tarantool-patches-bounces@dev.tarantool.org
Sender: "Tarantool-patches" <tarantool-patches-bounces@dev.tarantool.org>

Add ability to set box.cfg options via env variables. These
variables should have name "TT_<OPTION>". When Tarantool instance
is started under tarantoolctl utility env variables options have
more priority than instance file's cfg.

Closes #5602

@TarantoolBot document
Title: Set box.cfg options via env variables

Now, it is possible to set box.cfg options via environment
variables. The name of variable should correspond the followng
pattern: "TT_<NAME>", where <NAME> is an option name. For example:
"TT_LISTEN", "TT_READAHEAD".
If it is complex value, for example, "replication" can be set by a
table, the corresponding environment varibale value has the syntax
of lua array (without keys):

`export TT_REPLICATION="{'uri1', 'uri2'}"`

Note that every value in arary must be quoted.
---

@ChangeLog
* Add ability to set box.cfg options via environment variables (gh-5602).

Branch: https://github.com/tarantool/tarantool/tree/romanhabibov/gh-5602-environment-variables
Issue: https://github.com/tarantool/tarantool/issues/5602

 src/box/lua/load_cfg.lua                      | 74 +++++++++++++++++++
 test/app-tap/tarantoolctl.test.lua            | 33 ++++++++-
 .../gh-5602-environment-vars-cfg.result       | 42 +++++++++++
 .../gh-5602-environment-vars-cfg.test.lua     | 53 +++++++++++++
 test/box-tap/gh-5602_script.lua               | 70 ++++++++++++++++++
 5 files changed, 271 insertions(+), 1 deletion(-)
 create mode 100644 test/box-tap/gh-5602-environment-vars-cfg.result
 create mode 100755 test/box-tap/gh-5602-environment-vars-cfg.test.lua
 create mode 100755 test/box-tap/gh-5602_script.lua

diff --git a/src/box/lua/load_cfg.lua b/src/box/lua/load_cfg.lua
index 574c8bef4..eace25d46 100644
--- a/src/box/lua/load_cfg.lua
+++ b/src/box/lua/load_cfg.lua
@@ -616,6 +616,79 @@ setmetatable(box, {
      end
 })
 
+local function process_option_value(option, raw_value)
+    local param_type = template_cfg[option]
+    if param_type == nil then
+        return nil
+    end
+    local value = nil
+    if param_type == 'number' or option == 'log_level' then
+        value = tonumber(raw_value)
+        if value == nil then
+            error("Environment variable "..'TT_'..string.upper(option).." has"..
+                  " incorrect value for option '"..option.."': should be "..
+                  "converted to number")
+        end
+    elseif param_type == 'boolean' or option == 'log_nonblock' then
+        if raw_value == 'true' then
+            value = true
+        elseif raw_value == 'false' then
+            value = false
+        else
+            error("Environment variable "..'TT_'..string.upper(option).." has"..
+                  " incorrect value for option '"..option.."': should be "..
+                  "'true' or 'false'")
+        end
+    elseif param_type == 'string, number' then
+        value = tonumber(raw_value)
+        if value == nil then
+            value = raw_value
+        end
+    elseif param_type == 'string, number, table' then
+        if string.sub(raw_value, 1, 1) == '{' and
+           string.sub(raw_value, -1) == '}' then
+            value = string.sub(raw_value, 2, -2)
+            value = require('csv').load(value)[1]
+            for i, str in pairs(value) do
+                if not (string.sub(str, 1, 1) == '\'' and
+                        string.sub(str, -1) == '\'') and
+                   not (string.sub(str, 1, 1) == '"' and
+                        string.sub(str, -1) == '"') then
+                    error("Environment variable "..'TT_'..string.upper(option)..
+                          " has incorrect value for option '"..option.."': "..
+                          "elements of array should be quoted strings")
+                end
+                str = string.sub(str, 2, -2)
+                value[i] = str
+            end
+        else
+            value = tonumber(raw_value)
+            if value == nil then
+                value = raw_value
+            end
+        end
+    else
+        value = raw_value
+    end
+
+    return value
+end
+
+local function collect_env_cfg(cfg)
+    if cfg == nil then
+        cfg = {}
+    end
+    for option, _ in pairs(template_cfg) do
+        if cfg[option] == nil or os.getenv('TARANTOOLCTL') == 'true' then
+            local env_var_name = 'TT_'..string.upper(option)
+            local raw_value = os.getenv(env_var_name)
+            if raw_value ~= nil then
+                cfg[option] = process_option_value(option, raw_value)
+            end
+        end
+    end
+end
+
 -- Whether box is loaded.
 --
 -- `false` when box is not configured or when the initialization
@@ -627,6 +700,7 @@ setmetatable(box, {
 local box_is_configured = false
 
 local function load_cfg(cfg)
+    collect_env_cfg(cfg)
     -- A user may save box.cfg (this function) before box loading
     -- and call it afterwards. We should reconfigure box in the
     -- case.
diff --git a/test/app-tap/tarantoolctl.test.lua b/test/app-tap/tarantoolctl.test.lua
index 3c3023d34..1f15fe103 100755
--- a/test/app-tap/tarantoolctl.test.lua
+++ b/test/app-tap/tarantoolctl.test.lua
@@ -160,7 +160,7 @@ local function merge(...)
 end
 
 local test = tap.test('tarantoolctl')
-test:plan(8)
+test:plan(9)
 
 -- basic start/stop test
 -- must be stopped afterwards
@@ -632,4 +632,35 @@ test:test('filter_xlog', function(test)
     end
 end)
 
+-- gh-5602: Check that environment cfg variables are more
+-- prioritized than tarantoolctl cfg.
+do
+    local dir = fio.tempdir()
+    local code = [[ box.cfg ]]
+    create_script(dir, 'show_cfg.lua',  code)
+    local code = [[ box.cfg{listen=3302, readahead=10001} ]]
+    create_script(dir, 'cfg_script.lua', code)
+
+    local status, err = pcall(function()
+        test:test("check answers in case of call", function(test_i)
+            test_i:plan(4)
+            os.setenv('TT_LISTEN', '3301')
+            os.setenv('TT_READAHEAD', '10000')
+            check_ok(test_i, dir, 'start', 'cfg_script', 0)
+            tctl_wait_start(dir, 'cfg_script')
+            check_ok(test_i, dir, 'eval',  'cfg_script show_cfg.lua', 0,
+                     '---\n- 3301\n- 10000\n...', nil)
+            check_ok(test_i, dir, 'stop', 'cfg_script', 0)
+        end)
+    end)
+
+    cleanup_instance(dir, 'cfg_script')
+    recursive_rmdir(dir)
+
+    if status == false then
+        print(("Error: %s"):format(err))
+        os.exit()
+    end
+end
+
 os.exit(test:check() == true and 0 or -1)
diff --git a/test/box-tap/gh-5602-environment-vars-cfg.result b/test/box-tap/gh-5602-environment-vars-cfg.result
new file mode 100644
index 000000000..495e02a73
--- /dev/null
+++ b/test/box-tap/gh-5602-environment-vars-cfg.result
@@ -0,0 +1,42 @@
+TAP version 13
+1..5
+ok - box.cfg is successful
+ok - listen
+ok - readahead
+ok - strip_core
+ok - log_format is not set
+TAP version 13
+1..7
+ok - box.cfg is successful
+ok - listen
+ok - replication is table
+ok - replication URI 1
+ok - replication URI 2
+ok - replication_connect_timeout
+ok - replication_synchro_quorum
+TAP version 13
+1..3
+ok - box.cfg is successful
+ok - box.cfg{} background value is prioritized
+ok - box.cfg{} vinyl_timeout value is prioritized
+TAP version 13
+1..2
+ok - box.cfg is not successful
+ok - bad sql_cache_size value
+TAP version 13
+1..2
+ok - box.cfg is not successful
+builtin/box/load_cfg.lua:19: builtin/box/load_cfg.lua:638: Environment variable TT_STRIP_CORE has incorrect value for option 'strip_core': should be 'true' or 'false'
+ok - bad strip_core value
+TAP version 13
+1..2
+ok - box.cfg is not successful
+builtin/box/load_cfg.lua:19: builtin/box/load_cfg.lua:657: Environment variable TT_REPLICATION has incorrect value for option 'replication': elements of array should be quoted strings
+ok - bad replication value
+TAP version 13
+ok - set 1
+ok - set 2
+ok - set 3
+ok - set 4
+ok - set 5
+ok - set 6
diff --git a/test/box-tap/gh-5602-environment-vars-cfg.test.lua b/test/box-tap/gh-5602-environment-vars-cfg.test.lua
new file mode 100755
index 000000000..84b49b7eb
--- /dev/null
+++ b/test/box-tap/gh-5602-environment-vars-cfg.test.lua
@@ -0,0 +1,53 @@
+#!/usr/bin/env tarantool
+
+local os = require('os')
+local fio = require('fio')
+local tap = require('tap')
+local test = tap.test('gh-5602')
+
+-- gh-5602: Check that environment cfg variables working.
+local TARANTOOL_PATH = arg[-1]
+local script_name = 'gh-5602_script.lua'
+local path_to_script = fio.pathjoin(
+        os.getenv('PWD'),
+        'box-tap',
+        script_name)
+local function shell_command(set, i)
+    string.gsub(set, 'TT_', '')
+    return ('%s %s %s %d'):format(
+        set,
+        TARANTOOL_PATH,
+        path_to_script,
+        i)
+end
+
+local set_1 = 'TT_LISTEN=3301 ' ..
+              'TT_READAHEAD=10000 ' ..
+              'TT_STRIP_CORE=false ' ..
+              'TT_LOG_FORMAT=json'
+local set_2 = 'TT_LISTEN=3301 ' ..
+              'TT_REPLICATION=\"{\'0.0.0.0:12345\', ' ..
+                              '\'1.1.1.1:12345\'}\" ' ..
+              'TT_REPLICATION_CONNECT_TIMEOUT=0.01 ' ..
+              'TT_REPLICATION_SYNCHRO_QUORUM=\'4 + 1\''
+local set_3 = 'TT_BACKGROUND=true ' ..
+              'TT_VINYL_TIMEOUT=60.1'
+local set_4 = 'TT_SQL_CACHE_SIZE=a'
+local set_5 = 'TT_STRIP_CORE=a'
+local set_6 = 'TT_REPLICATION=\"{0.0.0.0:12345\', ' ..
+                              '\'1.1.1.1:12345\'}\"'
+local sets = {}
+sets[1] = set_1
+sets[2] = set_2
+sets[3] = set_3
+sets[4] = set_4
+sets[5] = set_5
+sets[6] = set_6
+
+for i, set in ipairs(sets) do
+    local tmpdir = fio.tempdir()
+    local new_path = fio.pathjoin(tmpdir, script_name)
+    fio.copyfile(path_to_script, new_path)
+    test:is(os.execute(shell_command(set, i)), 0, 'set '..i)
+end
+os.exit(0)
diff --git a/test/box-tap/gh-5602_script.lua b/test/box-tap/gh-5602_script.lua
new file mode 100755
index 000000000..8ff34deb2
--- /dev/null
+++ b/test/box-tap/gh-5602_script.lua
@@ -0,0 +1,70 @@
+local tap = require('tap')
+local test = tap.test('gh-5602')
+
+local status, err = pcall(box.cfg, {background = false, vinyl_timeout = 70.1})
+
+-- Check that environment cfg values are set correctly.
+if arg[1] == '1' then
+    test:plan(5)
+    test:ok(status ~= false, 'box.cfg is successful')
+    test:is(box.cfg['listen'], '3301', 'listen')
+    test:is(box.cfg['readahead'], 10000, 'readahead')
+    test:is(box.cfg['strip_core'], false, 'strip_core')
+    test:is(box.cfg['log_format'], 'json', 'log_format is not set')
+end
+if arg[1] == '2' then
+    test:plan(7)
+    test:ok(status ~= false, 'box.cfg is successful')
+    test:is(box.cfg['listen'], '3301', 'listen')
+    local replication = box.cfg['replication']
+    test:is(type(replication), 'table', 'replication is table')
+    test:ok(replication[1] == '0.0.0.0:12345' or
+            replication[1] == '1.1.1.1:12345', 'replication URI 1')
+    test:ok(replication[2] == '0.0.0.0:12345' or
+            replication[2] == '1.1.1.1:12345', 'replication URI 2')
+    test:is(box.cfg['replication_connect_timeout'], 0.01,
+            'replication_connect_timeout')
+    test:is(box.cfg['replication_synchro_quorum'], '4 + 1',
+            'replication_synchro_quorum')
+end
+
+-- Check that box.cfg{} values are more prioritized than
+-- environment cfg values.
+if arg[1] == '3' then
+    test:plan(3)
+    test:ok(status ~= false, 'box.cfg is successful')
+    test:is(box.cfg['background'], false,
+            'box.cfg{} background value is prioritized')
+    test:is(box.cfg['vinyl_timeout'], 70.1,
+            'box.cfg{} vinyl_timeout value is prioritized')
+end
+
+-- Check bad environment cfg values.
+if arg[1] == '4' then
+    test:plan(2)
+    test:ok(status == false, 'box.cfg is not successful')
+    local index = string.find(err, 'Environment variable TT_SQL_CACHE_SIZE '..
+        'has incorrect value for option \'sql_cache_size\': should be '..
+        'converted to number')
+    test:ok(index ~= nil, 'bad sql_cache_size value')
+end
+if arg[1] == '5' then
+    test:plan(2)
+    test:ok(status == false, 'box.cfg is not successful')
+    print(err)
+    local index = string.find(err, 'Environment variable TT_STRIP_CORE has '..
+        'incorrect value for option \'strip_core\': should be \'true\' '..
+        'or \'false\'')
+    test:ok(index ~= nil, 'bad strip_core value')
+end
+if arg[1] == '6' then
+    test:plan(2)
+    test:ok(status == false, 'box.cfg is not successful')
+    print(err)
+    local index = string.find(err, 'Environment variable TT_REPLICATION has '..
+        'incorrect value for option \'replication\': elements of array should'..
+        ' be quoted strings')
+    test:ok(index ~= nil, 'bad replication value')
+end
+
+os.exit(test:check() and 0 or 1)
\ No newline at end of file
-- 
2.24.3 (Apple Git-128)