From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com [209.85.208.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 86B504696C1 for ; Wed, 11 Dec 2019 12:29:27 +0300 (MSK) Received: by mail-lj1-f177.google.com with SMTP id j6so23263338lja.2 for ; Wed, 11 Dec 2019 01:29:27 -0800 (PST) From: Cyrill Gorcunov Date: Wed, 11 Dec 2019 12:28:32 +0300 Message-Id: <20191211092833.12212-5-gorcunov@gmail.com> In-Reply-To: <20191211092833.12212-1-gorcunov@gmail.com> References: <20191211092833.12212-1-gorcunov@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v3 4/5] popen/fio: Add ability to run external programs List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tml In the patch we add support for running external programs via fio module and two helpers fio.popen and fio.popen_posix. @TarantoolBot document Title: Document fio.popen() and fio.popen_posix() `fio.popen` and `fio.popen_posix` functions allow to spawn a new process, connect to its stdin, stdout and stderr pipes to communicate and either wait for the process to exit or terminate it. `fio.popen_posix(command, mode)` creates a process and invokes the shell inside, where: - `command` is the command to execute inside the shell; - `mode` is a string which must contain either letter `"r"` for reading output from stdout or letter `"w"` to write into stdin of a child process. To read and write data use `"rw"`. `fio.popen_posix` always close all files except std streams, resets signal handlers to default and setup a new session for a child process. Example 1 ``` popen, err = require('fio').popen_posix("date", "r") ``` Runs the "date" command as `sh -c date` to read the output, closing all file descriptors except stdout and stderr. `fio.popen(opts)` opens a process and executes a program inside depending on the `opts` options table, which should be in form of `{argv = {...}, envp= {...}, flags = {...}}`, where: - `argv` is an array of arguments to execute, the first element of which is used as name of a program to run and the rest are considered as arguments. This is the only mandatory element in `opts` table; - `envp` is an array of environment variables to be used instead of inheriting ones from a parent process. `envp` can be omitted completely then parent's environment is used, or can be set to empty array and then the child process won't get any; - `flags` is a subtable which describes additional parameters for the child process: `stdin = true | false | "devnull"` `stdout = true | false | "devnull"` `stdout = true | false | "devnull"` These keys specify which std descriptors to open (`true`), close (`false`) or set to /dev/null device (`"devnull"`). By default `stdin = false`, `stdout = false`, `stderr = false` so all streams are closed in a child process. `shell = true | false` This key specify whether a child process should be executed via `sh -c` (true) or take the binary name from `argv` array (false). By default `shell = true`. `close_fds = true | false` When set to `true` a child process will not inherit any files from a parent process (the parent process is the tarantool itself). By default `close_fds = true`. `restore_signals = true | false` When set to `true` the signal handlers are restored to defaut values from OS point of view. Since signals are shared with the process we're spawning there must be a very strong reason to set this key to `false` value. By default `restore_signals = true`. `start_new_session = true | false` Specifies whether a child process will be a session leader. By default `start_new_session = true`. Example 2 ``` popen = require('fio').popen({argv = {"date"}, flags = {stdout=true}}) ``` It runs `date` command inside shell as in example 1. Example 3 ``` popen = require('fio').popen({argv = {"/usr/bin/echo", "-n", "hello"}, flags = {stdout=true, shell=false}}) ``` It runs `/usr/bin/echo` executable directly without opening shell and keeps stdout to read output later. Once popen object is created the following methods are provided: `popen:read(buf,size,opts)` To read from stdout stream. `buf` and `size` are optional and when not specified popen will allocate an internal buffer with size of 512 bytes to keep data. The default size is managed by `popen.read_buf_size` variable. `opts` is optional as well but if provided the following keys are accepted: `timeout = microseconds` Specifies timeout in microseconds to interrupt the read procedure if there is no data provided by a writer. If set to `-1` then read waits until data appear or peer end closed (or child process exited). `stdout = true` `stderr = true` Sets which std descriptor of a child process to read. Only one of them should be provided at a time. `popen:read2(opts)` An extended version of `popen:read` which allows to specify the stream to read. `opts` is a table of options which sould have the following keys: `buf` The destination buffer. Should be an instance of `buffer` module. `size` Number of bytes to read. `flags = { stdout = true }` `flags = { stderr = true }` Specifies which stream to read. `timeout = microseconds` Read timeout. Optional, set to `-1` by default. `popen:write(str)` `popen:write(buf,size)` To write data into stdin stream of a child process. Either `str` string to write should be provided or `buf` buffer and its `size`. `popen:write2(opts)` An extended version of `popen:write`. Similar to `popen:read2` it operates with `opts` options table with keys: `buf` The source buffer. `size` Source buffer size in bytes. `flags = { stdout = true }` Specifies stream to write. `popen:info()` Returns information about a child process in form of table with the following keys: `pid` PID of a child process. Set to `-1` if child is exited or terminated. `flags` Internal representation of flags which been used to create the popen instance. `stdout = N` `stderr = N` `stdin = N` File descriptors numbers for appropriate streams. Closed state is represented by value `-1`. `state = alive` `state = exited` `state = signaled` A symbolic name for a child process state. Until process terminated we consider it as `alive`. `exited` is set when process finished execution and exit by self. `signaled` is set when a child process has been terminated by a signal. `exit_code` Return status of the child process. When `state = signaled` the `exit_code` set to signal number caused process to exit. `popen:kill()` Force termination of a child process. This is not immediate action, and might require some time before the child process get exited. To verify if the child is exited use `popen:wait` method. `popen:terminate()` Soft termination of a child process. Similar to `popen:kill()` but on soft termination the child process is allowed to do internal cleanup. Still soft termination may be completely ignored by the child. `popen:wait()` Waits for child process to exit. This function must be called only after `popen:kill()` or `popen:terminate()`. It spins in an idle cycle fetching the child state in a polling manner. Polling time is `1` second by default and controlled by `popen.wait_secs` variable. `popen:close()` Closes popen instance and release all associated resources. If there is a running child process it will be killed. Part-of #4031 Signed-off-by: Cyrill Gorcunov --- src/lua/fio.lua | 576 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 576 insertions(+) diff --git a/src/lua/fio.lua b/src/lua/fio.lua index cb224f3d0..c7be5cd02 100644 --- a/src/lua/fio.lua +++ b/src/lua/fio.lua @@ -5,6 +5,7 @@ local ffi = require('ffi') local buffer = require('buffer') local fiber = require('fiber') local errno = require('errno') +local bit = require('bit') ffi.cdef[[ int umask(int mask); @@ -519,4 +520,579 @@ fio.path.lexists = function(filename) return fio.lstat(filename) ~= nil end +local popen_methods = { } + +-- +-- Make sure the bits are matching POPEN_FLAG_x +-- flags in popen header. This is api but internal +-- one, we should not make it visible to users. +local popen_flag_bits = { + STDIN = 0, + STDOUT = 1, + STDERR = 2, + + STDIN_DEVNULL = 3, + STDOUT_DEVNULL = 4, + STDERR_DEVNULL = 5, + + STDIN_CLOSE = 6, + STDOUT_CLOSE = 7, + STDERR_CLOSE = 8, + + -- + -- bits 9,10,11 are reserved + -- for future use + + + + SHELL = 12, + SETSID = 13, + CLOSE_FDS = 14, + RESTORE_SIGNALS = 15, +} + +local popen_flags = { + NONE = 0, + STDIN = bit.lshift(1, popen_flag_bits.STDIN), + STDOUT = bit.lshift(1, popen_flag_bits.STDOUT), + STDERR = bit.lshift(1, popen_flag_bits.STDERR), + + STDIN_DEVNULL = bit.lshift(1, popen_flag_bits.STDIN_DEVNULL), + STDOUT_DEVNULL = bit.lshift(1, popen_flag_bits.STDOUT_DEVNULL), + STDERR_DEVNULL = bit.lshift(1, popen_flag_bits.STDERR_DEVNULL), + + STDIN_CLOSE = bit.lshift(1, popen_flag_bits.STDIN_CLOSE), + STDOUT_CLOSE = bit.lshift(1, popen_flag_bits.STDOUT_CLOSE), + STDERR_CLOSE = bit.lshift(1, popen_flag_bits.STDERR_CLOSE), + + SHELL = bit.lshift(1, popen_flag_bits.SHELL), + SETSID = bit.lshift(1, popen_flag_bits.SETSID), + CLOSE_FDS = bit.lshift(1, popen_flag_bits.CLOSE_FDS), + RESTORE_SIGNALS = bit.lshift(1, popen_flag_bits.RESTORE_SIGNALS), +} + +-- +-- Get default flags for popen +local function popen_default_flags() + local flags = popen_flags.NONE + + -- default flags: close everything and use shell + flags = bit.bor(flags, popen_flags.STDIN_CLOSE) + flags = bit.bor(flags, popen_flags.STDOUT_CLOSE) + flags = bit.bor(flags, popen_flags.STDERR_CLOSE) + flags = bit.bor(flags, popen_flags.SHELL) + flags = bit.bor(flags, popen_flags.SETSID) + flags = bit.bor(flags, popen_flags.CLOSE_FDS) + flags = bit.bor(flags, popen_flags.RESTORE_SIGNALS) + + return flags +end + +-- +-- Close a popen object and release all resources. +-- In case if there is a running child process +-- it will be killed. +-- +-- Returns @ret = true if popen handle closed, and +-- @ret = false, @err ~= nil on error. +popen_methods.close = function(self) + local ret, err = internal.popen_delete(self.popen_handle) + if err ~= nil then + return false, err + end + self.popen_handle = nil + return true +end + +-- +-- Kill a child process +-- +-- Returns @ret = true on success, +-- @ret = false, @err ~= nil on error. +popen_methods.kill = function(self) + return internal.popen_kill(self.popen_handle) +end + +-- +-- Terminate a child process +-- +-- Returns @ret = true on success, +-- @ret = false, @err ~= nil on error. +popen_methods.terminate = function(self) + return internal.popen_term(self.popen_handle) +end + +-- +-- Fetch a child process state +-- +-- Returns @err = nil, @state = (1 - if exited, +-- 2 - if killed by a signal) and @exit_code ~= nil, +-- otherwise @err ~= nil. +-- +-- FIXME: Should the @state be a named constant? +popen_methods.state = function(self) + return internal.popen_state(self.popen_handle) +end + +-- +-- Wait until a child process get exited. +-- +-- Returns the same as popen_methods.status. +popen_methods.wait = function(self) + local err, state, code + while true do + err, state, code = internal.popen_state(self.popen_handle) + if err or state then + break + end + fiber.sleep(self.wait_secs) + end + return err, state, code +end + +-- +-- A map for popen option keys into tfn ('true|false|nil') values +-- where bits are just set without additional manipulations. +-- +-- For example stdin=true means to open stdin end for write, +-- stdin=false to close the end, finally stdin[=nil] means to +-- provide /dev/null into a child peer. +local popen_opts_tfn = { + stdin = { + popen_flags.STDIN, + popen_flags.STDIN_CLOSE, + popen_flags.STDIN_DEVNULL, + }, + stdout = { + popen_flags.STDOUT, + popen_flags.STDOUT_CLOSE, + popen_flags.STDOUT_DEVNULL, + }, + stderr = { + popen_flags.STDERR, + popen_flags.STDERR_CLOSE, + popen_flags.STDERR_DEVNULL, + }, +} + +-- +-- A map for popen option keys into tf ('true|false') values +-- where bits are set on 'true' and clear on 'false'. +local popen_opts_tf = { + shell = { + popen_flags.SHELL, + }, + close_fds = { + popen_flags.CLOSE_FDS + }, + restore_signals = { + popen_flags.RESTORE_SIGNALS + }, + start_new_session = { + popen_flags.SETSID + }, +} + +-- +-- Parses flags options from popen_opts_tfn and +-- popen_opts_tf tables. +local function parse_popen_flags(epfx, flags, opts) + if opts == nil then + return flags + end + for k,v in pairs(opts) do + if popen_opts_tfn[k] == nil then + if popen_opts_tf[k] == nil then + error(string.format("%s: Unknown key %s", epfx, k)) + end + if v == true then + flags = bit.bor(flags, popen_opts_tf[k][1]) + elseif v == false then + flags = bit.band(flags, bit.bnot(popen_opts_tf[k][1])) + else + error(string.format("%s: Unknown value %s", epfx, v)) + end + else + if v == true then + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][2])) + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][3])) + flags = bit.bor(flags, popen_opts_tfn[k][1]) + elseif v == false then + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][1])) + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][3])) + flags = bit.bor(flags, popen_opts_tfn[k][2]) + elseif v == "devnull" then + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][1])) + flags = bit.band(flags, bit.bnot(popen_opts_tfn[k][2])) + flags = bit.bor(flags, popen_opts_tfn[k][3]) + else + error(string.format("%s: Unknown value %s", epfx, v)) + end + end + end + return flags +end + +-- +-- popen:read2 - read stream of a child with options +-- @opts: options table +-- +-- The options should have the following keys +-- +-- @flags: stdout=true or stderr=true +-- @buf: const_char_ptr_t buffer +-- @size: size of the buffer +-- @timeout: read timeout in microsecond +-- +-- Returns @res = bytes, err = nil in case if read processed +-- without errors, @res = nil, @err = nil if timeout elapsed +-- and no data appeared on the peer, @res = nil, @err ~= nil +-- otherwise. +popen_methods.read2 = function(self, opts) + -- + -- We can reuse parse_popen_flags since only + -- a few flags we're interesed in -- stdout/stderr + local flags = parse_popen_flags("fio.popen:read2", + popen_flags.NONE, + opts['flags']) + local timeout = -1 + if opts['buf'] == nil then + error("fio.popen:read2 {'buf'} key is missed") + elseif opts['size'] == nil then + error("fio.popen:read2 {'size'} key is missed") + elseif opts['timeout'] ~= nil then + timeout = tonumber(opts['timeout']) + end + + return internal.popen_read(self.popen_handle, opts['buf'], + tonumber(opts['size']), + timeout, flags) +end + +-- popen:write2 - write data to a child with options +-- @opts: options table +-- +-- The options should have the following keys +-- +-- @flags: stdin=true +-- @buf: const_char_ptr_t buffer +-- @size: size of the buffer +-- +-- Returns @res = bytes, err = nil in case if write processed +-- without errors, or @res = nil, @err ~= nil otherwise. +popen_methods.write2 = function(self, opts) + local flags = parse_popen_flags("fio.popen:write2", + popen_flags.NONE, + opts['flags']) + if opts['buf'] == nil then + error("fio.popen:write2 {'buf'} key is missed") + elseif opts['size'] == nil then + error("fio.popen:write2 {'size'} key is missed") + end + return internal.popen_write(self.popen_handle, opts['buf'], + tonumber(opts['size']), flags) +end + +-- +-- Write data to stdin of a child process +-- @buf: a buffer to write +-- @size: size of the buffer +-- +-- Both parameters are optional and some +-- string could be passed instead +-- +-- write(str) +-- write(buf, len) +-- +-- Returns @res = bytes, @err = nil in case of success, +-- @res = , @err ~= nil in case of error. +popen_methods.write = function(self, buf, size) + if not ffi.istype(const_char_ptr_t, buf) then + buf = tostring(buf) + size = #buf + end + return self:write2({ buf = buf, size = size, + flags = { stdin = true }}) +end + +-- +-- popen:read - read the output stream in blocked mode +-- @buf: a buffer to read to, optional +-- @size: size of the buffer, optional +-- @opts: options table, optional +-- +-- The options may have the following keys: +-- +-- @flags: stdout=true or stderr=true, +-- to specify which stream to read; +-- by default stdout is used +-- +-- @timeout: read timeout in microcesonds, when +-- specified the read will be interrupted +-- after specified time elapced, to make +-- nonblocking read; by default the read +-- is blocking operation +-- +-- The following variants are accepted as well +-- +-- read() -> str +-- read(buf) -> len +-- read(size) -> str +-- read(buf, size) -> len +-- +-- FIXME: Should we merged it with plain fio.write()? +popen_methods.read = function(self, buf, size, opts) + local tmpbuf + + -- read() or read(buf) + if (not ffi.istype(const_char_ptr_t, buf) and buf == nil) or + (ffi.istype(const_char_ptr_t, buf) and size == nil) then + -- read @self.read_buf_size bytes at once by default + size = self.read_buf_size + end + + -- read(size) + if not ffi.istype(const_char_ptr_t, buf) then + size = buf or size + tmpbuf = buffer.ibuf() + buf = tmpbuf:reserve(size) + end + + if opts == nil then + opts = { + timeout = -1, + flags = { stdout = true }, + } + end + + local res, err = self:read2({ buf = buf, size = size, + timeout = opts['timeout'], + flags = opts['flags'] }) + + if res == nil then + if tmpbuf ~= nil then + tmpbuf:recycle() + end + return nil, err + end + + if tmpbuf ~= nil then + tmpbuf:alloc(res) + res = ffi.string(tmpbuf.rpos, tmpbuf:size()) + tmpbuf:recycle() + end + + return res +end + +-- +-- popen:info -- obtain information about a popen object +-- +-- Returns @info table, err == nil on success, +-- @info = nil, @err ~= nil otherwise. +-- +-- The @info table contains the following keys: +-- +-- @pid: pid of a child process +-- @flags: flags associated (popen_flags bitmap) +-- @stdout: parent peer for stdout, -1 if closed +-- @stderr: parent peer for stderr, -1 if closed +-- @stdin: parent peer for stdin, -1 if closed +-- @state: alive | exited | signaled | unknown +-- @exit_code: exit code of a child process if been +-- exited or killed by a signal +-- +-- The child should never be in "unknown" state, reserved +-- for unexpected errors. +-- +popen_methods.info = function(self) + return internal.popen_info(self.popen_handle) +end + +local popen_mt = { __index = popen_methods } + +-- +-- Create a new popen object from options +local function popen_new(opts) + local handle, err = internal.popen_new(opts) + if err ~= nil then + return nil, err + end + + local popen_handle = { + -- a handle itself for future use + popen_handle = handle, + + -- sleeping period for the @wait method + wait_secs = 1, + + -- size of a read buffer to allocate + -- in case of implicit read + read_buf_size = 512, + } + + setmetatable(popen_handle, popen_mt) + return popen_handle +end + +-- +-- Parse "mode" string to flags +local function parse_popen_mode(epfx, flags, mode) + if mode == nil then + return flags + end + for i = 1, #mode do + local c = mode:sub(i, i) + if c == 'r' then + flags = bit.band(flags, bit.bnot(popen_flags.STDOUT_CLOSE)) + flags = bit.bor(flags, popen_flags.STDOUT) + flags = bit.band(flags, bit.bnot(popen_flags.STDERR_CLOSE)) + flags = bit.bor(flags, popen_flags.STDERR) + elseif c == 'w' then + flags = bit.band(flags, bit.bnot(popen_flags.STDIN_CLOSE)) + flags = bit.bor(flags, popen_flags.STDIN) + else + error(string.format("%s: Unknown mode %s", epfx, c)) + end + end + return flags +end + +-- +-- fio.popen_posix - create a new child process and execute a command inside +-- @command: a command to run +-- @mode: r - to grab child's stdout for read [*] +-- (stderr kept opened as well for 2>&1 redirection) +-- w - to obtain stdin for write +-- +-- [*] are default values +-- +-- Note: Since there are two options only the following parameters +-- are implied (see popen_default_flags): all fds except stdin/out/err +-- are closed, signals are restored to default handlers, the command +-- is executed in a new session. +-- +-- Examples: +-- +-- popen = require('fio').popen_posix("date", "r") +-- runs date as "sh -c date" to read the output, +-- closing all file descriptors except stdout/err +-- inside a child process +-- +fio.popen_posix = function(command, mode) + local flags = popen_default_flags() + + if type(command) ~= 'string' then + error("Usage: fio.popen(command[, rw])") + end + + -- Mode gives simplified flags + flags = parse_popen_mode("fio.popen", flags, mode) + + local opts = { + argv = { command }, + argc = 1, + flags = flags, + envc = -1, + } + + return popen_new(opts) +end + +-- fio.popen - execute a child program in a new process +-- @opt: table of options +-- +-- @opts carries of the following options: +-- +-- @argv: an array of a program to run with +-- command line options, mandatory +-- +-- @env: an array of environment variables to +-- be used inside a process, if not +-- set then the current environment is +-- inherited, if set to an empty array +-- then the environment will be dropped +-- +-- @flags: a dictionary to describe communication pipes +-- and other parameters of a process to run +-- +-- stdin=true to write into STDIN_FILENO of a child process +-- stdin=false to close STDIN_FILENO inside a child process [*] +-- stdin="devnull" a child will get /dev/null as STDIN_FILENO +-- +-- stdout=true to write into STDOUT_FILENO of a child process +-- stdout=false to close STDOUT_FILENO inside a child process [*] +-- stdin="devnull" a child will get /dev/null as STDOUT_FILENO +-- +-- stderr=true to read STDERR_FILENO from a child process +-- stderr=false to close STDERR_FILENO inside a child process [*] +-- stdin="devnull" a child will get /dev/null as STDERR_FILENO +-- +-- shell=true runs a child process via "sh -c" [*] +-- shell=false invokes a child process executable directly +-- +-- close_fds=true close all inherited fds from a parent [*] +-- +-- restore_signals=true +-- all signals modified by a caller reset +-- to default handler [*] +-- +-- start_new_session=true +-- start executable inside a new session [*] +-- +-- [*] are default values +-- +-- Examples: +-- +-- popen = require('fio').popen({argv = {"date"}, flags = {stdout=true}}) +-- popen:read() +-- popen:close() +-- +-- Execute 'date' command inside a shell, read the result +-- and close the popen object +-- +-- popen = require('fio').popen({argv = {"/usr/bin/echo", "-n", "hello"}, +-- flags = {stdout=true, shell=false}}) +-- popen:read() +-- popen:close() +-- +-- Execute /usr/bin/echo with arguments '-n','hello' directly +-- without using a shell, read the result from stdout and close +-- the popen object +-- +fio.popen = function(opts) + local flags = popen_default_flags() + + if opts == nil or type(opts) ~= 'table' then + error("Usage: fio.run({argv={},envp={},flags={}") + end + + -- Test for required arguments + if opts["argv"] == nil then + error("fio.run: argv key is missing") + end + + -- Process flags and save updated mask + -- to pass into engine (in case of missing + -- flags we just use defaults). + if opts["flags"] ~= nil then + flags = parse_popen_flags("fio.run", flags, opts["flags"]) + end + opts["flags"] = flags + + -- We will need a number of args for speed sake + opts["argc"] = #opts["argv"] + + -- Same applies to the environment (note though + -- that env={} is pretty valid and env=nil + -- as well. + if opts["env"] ~= nil then + opts["envc"] = #opts["env"] + else + opts["envc"] = -1 + end + + return popen_new(opts) +end + return fio -- 2.20.1