[PATCH v3 0/7] Merger
Alexander Turenko
alexander.turenko at tarantool.org
Wed Apr 10 18:21:18 MSK 2019
This patchset contains the set of auxiliary patches and two main ones:
the merger itself and the Lua wrapper for it with specific sources.
Previous version of the patchset was discussed here:
https://www.freelists.org/post/tarantool-patches/PATCH-v2-06-Merger
(also there was '[#3276] Merger API RFC' internal thread about the API).
More documentation and usage examples:
https://github.com/Totktonada/tarantool-merger-examples
Changes since v2 for 'Add functions to ease using Lua iterators from C':
- Fixed zero gen() retvals case.
- Added a unit test.
- Removed unneded cast from (void *).
- Allow to pass values w/o a table.
- Also removed _fromtable suffix from luaL_iterator_new().
- luaL_iterator_free() -> luaL_iterator_delete().
- Handle malloc fail.
- Use luaT_call instead of lua_call.
See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,7
Changes since v2 for 'net.box: add helpers to decode msgpack headers':
- Splitted this patch to two ones:
- lua: add non-recursive msgpack decoding functions
- net.box: add skip_header option to use with buffer
See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,3
Changes for 'Add merger for tuple streams':
- Got rid of sources reallocation during sources parsing.
- Fixed a memleak in :ipairs().
- Set a diag in merge_next() (it will be a C part).
- Renamed opts.descending -> reverse as Vova suggests.
- Support fetch callback in iterator sources (to change it to tuple
sources later).
- Found and fixed table source freeing bug.
- Unified fetch source API: return a new buffer from a fetch callback.
- Sources reference counting (to return it to Lua in the future).
- Use cdata wrappers for merger sources.
- Moved fetch callback to sources.
- Use global lua_State for unreferencing.
- Got rid of lua_State in arguments of virtual merger source functions.
- Replaced fetch callback with a Lua iterator.
- Postpone a first source fetch until merger_next() is called.
- Set a diag rather then push an error to a Lua stack.
- Replaced an iterator source with a tuple source.
- lbox_merger_new_*_source(): removed arguments popping.
- Removed obsolete comments re Lua stack.
- Moved source->tuple accesses from a source to a merger.
- Implemented refcounting for merger_context.
- Moved methods into a cdata<struct merger_state>
- Splitted merger.new() and :select() opts.
- Test: cache merger context for each schema.
- Handled the case with a source is used in two mergers.
- Added limit option to :select().
- Splitted to C and Lua modules.
- Deduplicated code around source creating.
- Test: decreased run time, removed from long_run.
- Rewrote the merger as a source type.
- Added cascade mergers test case.
- Splitted a source and a heap node.
- Created short API description for docbot comment.
- Factored out use cases description and code examples into a separate
repo (see the link at the start of the email).
- Added a unit test with simple array source.
- Splitted C and Lua into separate commits.
- Test: ensure we're able to reuse a source.
- Finished with the problem with empty buffers: ensure we're call fetch
function in the case.
issue: https://github.com/tarantool/tarantool/issues/3276
branch: https://github.com/tarantool/tarantool/tree/Totktonada/gh-3276-on-board-merger
The branch based on key_def Lua module. It is needed for merger, but is
now under review (within a separate thread). Please, don't push last two
patches before key_defs will land.
Alexander Turenko (7):
Add luaL_iscallable with support of cdata metatype
Add functions to ease using Lua iterators from C
lua: optimize creation of a tuple from a tuple
lua: add non-recursive msgpack decoding functions
net.box: add skip_header option to use with buffer
Add merger for tuples streams (C part)
Add merger for tuple streams (Lua part)
extra/exports | 1 +
src/box/CMakeLists.txt | 3 +
src/box/lua/init.c | 7 +-
src/box/lua/merger.c | 1184 ++++++++++++++++++++++++++++++
src/box/lua/merger.h | 47 ++
src/box/lua/merger.lua | 41 ++
src/box/lua/net_box.lua | 46 +-
src/box/lua/tuple.c | 65 +-
src/box/merger.c | 464 ++++++++++++
src/box/merger.h | 180 +++++
src/lua/msgpack.c | 80 ++
src/lua/utils.c | 135 ++++
src/lua/utils.h | 47 ++
test/app-tap/module_api.c | 10 +
test/app-tap/module_api.test.lua | 85 ++-
test/app-tap/msgpack.test.lua | 180 ++++-
test/box-tap/merger.test.lua | 725 ++++++++++++++++++
test/box/net.box.result | 222 +++++-
test/box/net.box.test.lua | 86 ++-
test/unit/CMakeLists.txt | 7 +
test/unit/luaL_iterator.c | 208 ++++++
test/unit/luaL_iterator.result | 89 +++
test/unit/merger.result | 71 ++
test/unit/merger.test.c | 301 ++++++++
24 files changed, 4230 insertions(+), 54 deletions(-)
create mode 100644 src/box/lua/merger.c
create mode 100644 src/box/lua/merger.h
create mode 100644 src/box/lua/merger.lua
create mode 100644 src/box/merger.c
create mode 100644 src/box/merger.h
create mode 100755 test/box-tap/merger.test.lua
create mode 100644 test/unit/luaL_iterator.c
create mode 100644 test/unit/luaL_iterator.result
create mode 100644 test/unit/merger.result
create mode 100644 test/unit/merger.test.c
--
2.20.1
More information about the Tarantool-patches
mailing list