[PATCH v3 0/7] Merger

[Alexander Turenko <alexander.turenko@tarantool.org>]

This patchset contains the set of auxiliary patches and two main ones:
the merger itself and the Lua wrapper for it with specific sources.

Previous version of the patchset was discussed here:
https://www.freelists.org/post/tarantool-patches/PATCH-v2-06-Merger
(also there was '[#3276] Merger API RFC' internal thread about the API).

More documentation and usage examples:
https://github.com/Totktonada/tarantool-merger-examples

Changes since v2 for 'Add functions to ease using Lua iterators from C':

- Fixed zero gen() retvals case.
- Added a unit test.
- Removed unneded cast from (void *).
- Allow to pass values w/o a table.
  - Also removed _fromtable suffix from luaL_iterator_new().
- luaL_iterator_free() -> luaL_iterator_delete().
- Handle malloc fail.
- Use luaT_call instead of lua_call.

See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,7

Changes since v2 for 'net.box: add helpers to decode msgpack headers':

- Splitted this patch to two ones:
  - lua: add non-recursive msgpack decoding functions
  - net.box: add skip_header option to use with buffer

See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,3

Changes for 'Add merger for tuple streams':

- Got rid of sources reallocation during sources parsing.
- Fixed a memleak in :ipairs().
- Set a diag in merge_next() (it will be a C part).
- Renamed opts.descending -> reverse as Vova suggests.
- Support fetch callback in iterator sources (to change it to tuple
  sources later).
- Found and fixed table source freeing bug.
- Unified fetch source API: return a new buffer from a fetch callback.
- Sources reference counting (to return it to Lua in the future).
- Use cdata wrappers for merger sources.
- Moved fetch callback to sources.
- Use global lua_State for unreferencing.
- Got rid of lua_State in arguments of virtual merger source functions.
- Replaced fetch callback with a Lua iterator.
- Postpone a first source fetch until merger_next() is called.
- Set a diag rather then push an error to a Lua stack.
- Replaced an iterator source with a tuple source.
- lbox_merger_new_*_source(): removed arguments popping.
- Removed obsolete comments re Lua stack.
- Moved source->tuple accesses from a source to a merger.
- Implemented refcounting for merger_context.
- Moved methods into a cdata<struct merger_state>
- Splitted merger.new() and :select() opts.
- Test: cache merger context for each schema.
- Handled the case with a source is used in two mergers.
- Added limit option to :select().
- Splitted to C and Lua modules.
- Deduplicated code around source creating.
- Test: decreased run time, removed from long_run.
- Rewrote the merger as a source type.
- Added cascade mergers test case.
- Splitted a source and a heap node.
- Created short API description for docbot comment.
- Factored out use cases description and code examples into a separate
  repo (see the link at the start of the email).
- Added a unit test with simple array source.
- Splitted C and Lua into separate commits.
- Test: ensure we're able to reuse a source.
- Finished with the problem with empty buffers: ensure we're call fetch
  function in the case.

issue: https://github.com/tarantool/tarantool/issues/3276
branch: https://github.com/tarantool/tarantool/tree/Totktonada/gh-3276-on-board-merger

The branch based on key_def Lua module. It is needed for merger, but is
now under review (within a separate thread). Please, don't push last two
patches before key_defs will land.

Alexander Turenko (7):
  Add luaL_iscallable with support of cdata metatype
  Add functions to ease using Lua iterators from C
  lua: optimize creation of a tuple from a tuple
  lua: add non-recursive msgpack decoding functions
  net.box: add skip_header option to use with buffer
  Add merger for tuples streams (C part)
  Add merger for tuple streams (Lua part)

 extra/exports                    |    1 +
 src/box/CMakeLists.txt           |    3 +
 src/box/lua/init.c               |    7 +-
 src/box/lua/merger.c             | 1184 ++++++++++++++++++++++++++++++
 src/box/lua/merger.h             |   47 ++
 src/box/lua/merger.lua           |   41 ++
 src/box/lua/net_box.lua          |   46 +-
 src/box/lua/tuple.c              |   65 +-
 src/box/merger.c                 |  464 ++++++++++++
 src/box/merger.h                 |  180 +++++
 src/lua/msgpack.c                |   80 ++
 src/lua/utils.c                  |  135 ++++
 src/lua/utils.h                  |   47 ++
 test/app-tap/module_api.c        |   10 +
 test/app-tap/module_api.test.lua |   85 ++-
 test/app-tap/msgpack.test.lua    |  180 ++++-
 test/box-tap/merger.test.lua     |  725 ++++++++++++++++++
 test/box/net.box.result          |  222 +++++-
 test/box/net.box.test.lua        |   86 ++-
 test/unit/CMakeLists.txt         |    7 +
 test/unit/luaL_iterator.c        |  208 ++++++
 test/unit/luaL_iterator.result   |   89 +++
 test/unit/merger.result          |   71 ++
 test/unit/merger.test.c          |  301 ++++++++
 24 files changed, 4230 insertions(+), 54 deletions(-)
 create mode 100644 src/box/lua/merger.c
 create mode 100644 src/box/lua/merger.h
 create mode 100644 src/box/lua/merger.lua
 create mode 100644 src/box/merger.c
 create mode 100644 src/box/merger.h
 create mode 100755 test/box-tap/merger.test.lua
 create mode 100644 test/unit/luaL_iterator.c
 create mode 100644 test/unit/luaL_iterator.result
 create mode 100644 test/unit/merger.result
 create mode 100644 test/unit/merger.test.c

-- 
2.20.1