[Tarantool-patches] [PATCH 0/2] Popen Lua module

[Alexander Turenko]

The patchset implements popen Lua module upward existing backend popen
engine.

The first patch changes popen_delete() behaviour to always free
resources even when killing a process (or a process group) fails. We
(Alexander, Cyrill and Igor) revisited the contract for closing the
handle after observing killpg() behaviour on a group of zombie processes
on Mac OS. I added those details to API documentation comments to don't
surprise a user with this.

The first patch continues previous preliminary popen engine series:

https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/015609.html
https://lists.tarantool.org/pipermail/tarantool-patches/2020-April/015888.html

The second patch implements the Lua API for popen. It is quite
strightforward and should be considered as very basic implementation. We
plan to enhance it further in the upcoming releases (issues are to be
filed).

The main goal of the module it to provide popen implementation that is
integrated into our event loop (similar to fio and socket modules). Side
goal is to provide ability to feed data to a child process input and
read data from its output and so work with streaming programs (like
`grep`) and even interactive programs (like `python -i` interpreter).

Let's consider the API of module as beta: it may be change in
backward-incompatible manner in future releases if it will be valuable
enough.

It seems the main application of the module it to write various testing
code and so the API should be extended in the future with convenient
shortcuts: developers usually don't very like writting tests and it
should be at least more-or-less convenient.

I plan to extend the API for reading with ability to read certain amount
of bytes, to read line-by-line (or based on other delimiter), to read
into a buffer (ibuf): like it is implemented in the socket module. I
plan to share an RFC document about read streams.

ph:wait() should gain ability to set a timeout and should be rewritten
to use triggers / fiber conds instead of check-yield-again loop.

----

https://github.com/tarantool/tarantool/issues/4031
https://github.com/tarantool/tarantool/tree/Totktonada/gh-4031-popen-13-full-ci

Changelog entry:

## Functionality added or changed
### Lua

* Added `popen` built-in module (gh-4031).

  The module provides popen implementation that is integrated with
  tarantool's event loop (like built-in `fio` and `socket` modules).

  It support bidirectional communication with a process: the module can
  feed input to a process and capture its output. This way it allows to
  run streaming programs (like `grep`) and even work interactively with
  outside REPL (say, `python -i`).

  A key feature of the implementation is that it uses vfork() under hood
  and so does not copy virtual memory tables. Copying of them may be
  quite time consuming: os.execute() takes ~2.5 seconds when 80 GiB is
  allocated for memtx. Moreover, when memory overcommit is disabled
  (which is default) it would be not possible to fork a process when
  more then half of available physical memory is mapped to tarantool's
  process.

  The API should be considered as beta: it is quite basic and will be
  extended with convenience features. On the other hand, it may be
  changed in a backward-incompatible manner in the future releases if it
  will be valuable enough.

Alexander Turenko (2):
  popen: always free resources in popen_delete()
  popen: add popen Lua module

 src/CMakeLists.txt          |    1 +
 src/lib/core/popen.c        |   88 +-
 src/lua/init.c              |    2 +
 src/lua/popen.c             | 2457 +++++++++++++++++++++++++++++++++++
 src/lua/popen.h             |   44 +
 test/app-tap/popen.test.lua |  605 +++++++++
 6 files changed, 3182 insertions(+), 15 deletions(-)
 create mode 100644 src/lua/popen.c
 create mode 100644 src/lua/popen.h
 create mode 100755 test/app-tap/popen.test.lua

-- 
2.25.0