From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp44.i.mail.ru (smtp44.i.mail.ru [94.100.177.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 96B834696C3 for ; Sat, 18 Apr 2020 07:14:02 +0300 (MSK) From: Alexander Turenko Date: Sat, 18 Apr 2020 07:13:53 +0300 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH 0/2] Popen Lua module List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Cyrill Gorcunov , Igor Munkin Cc: tarantool-patches@dev.tarantool.org The patchset implements popen Lua module upward existing backend popen engine. The first patch changes popen_delete() behaviour to always free resources even when killing a process (or a process group) fails. We (Alexander, Cyrill and Igor) revisited the contract for closing the handle after observing killpg() behaviour on a group of zombie processes on Mac OS. I added those details to API documentation comments to don't surprise a user with this. The first patch continues previous preliminary popen engine series: https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/015609.html https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/015888.html The second patch implements the Lua API for popen. It is quite strightforward and should be considered as very basic implementation. We plan to enhance it further in the upcoming releases (issues are to be filed). The main goal of the module it to provide popen implementation that is integrated into our event loop (similar to fio and socket modules). Side goal is to provide ability to feed data to a child process input and read data from its output and so work with streaming programs (like `grep`) and even interactive programs (like `python -i` interpreter). Let's consider the API of module as beta: it may be change in backward-incompatible manner in future releases if it will be valuable enough. It seems the main application of the module it to write various testing code and so the API should be extended in the future with convenient shortcuts: developers usually don't very like writting tests and it should be at least more-or-less convenient. I plan to extend the API for reading with ability to read certain amount of bytes, to read line-by-line (or based on other delimiter), to read into a buffer (ibuf): like it is implemented in the socket module. I plan to share an RFC document about read streams. ph:wait() should gain ability to set a timeout and should be rewritten to use triggers / fiber conds instead of check-yield-again loop. ---- https://github.com/tarantool/tarantool/issues/4031 https://github.com/tarantool/tarantool/tree/Totktonada/gh-4031-popen-13-full-ci Changelog entry: ## Functionality added or changed ### Lua * Added `popen` built-in module (gh-4031). The module provides popen implementation that is integrated with tarantool's event loop (like built-in `fio` and `socket` modules). It support bidirectional communication with a process: the module can feed input to a process and capture its output. This way it allows to run streaming programs (like `grep`) and even work interactively with outside REPL (say, `python -i`). A key feature of the implementation is that it uses vfork() under hood and so does not copy virtual memory tables. Copying of them may be quite time consuming: os.execute() takes ~2.5 seconds when 80 GiB is allocated for memtx. Moreover, when memory overcommit is disabled (which is default) it would be not possible to fork a process when more then half of available physical memory is mapped to tarantool's process. The API should be considered as beta: it is quite basic and will be extended with convenience features. On the other hand, it may be changed in a backward-incompatible manner in the future releases if it will be valuable enough. Alexander Turenko (2): popen: always free resources in popen_delete() popen: add popen Lua module src/CMakeLists.txt | 1 + src/lib/core/popen.c | 88 +- src/lua/init.c | 2 + src/lua/popen.c | 2457 +++++++++++++++++++++++++++++++++++ src/lua/popen.h | 44 + test/app-tap/popen.test.lua | 605 +++++++++ 6 files changed, 3182 insertions(+), 15 deletions(-) create mode 100644 src/lua/popen.c create mode 100644 src/lua/popen.h create mode 100755 test/app-tap/popen.test.lua -- 2.25.0