From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Alexander Turenko Subject: [PATCH v3 0/7] Merger Date: Wed, 10 Apr 2019 18:21:18 +0300 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit To: Vladimir Davydov Cc: Alexander Turenko , tarantool-patches@freelists.org List-ID: This patchset contains the set of auxiliary patches and two main ones: the merger itself and the Lua wrapper for it with specific sources. Previous version of the patchset was discussed here: https://www.freelists.org/post/tarantool-patches/PATCH-v2-06-Merger (also there was '[#3276] Merger API RFC' internal thread about the API). More documentation and usage examples: https://github.com/Totktonada/tarantool-merger-examples Changes since v2 for 'Add functions to ease using Lua iterators from C': - Fixed zero gen() retvals case. - Added a unit test. - Removed unneded cast from (void *). - Allow to pass values w/o a table. - Also removed _fromtable suffix from luaL_iterator_new(). - luaL_iterator_free() -> luaL_iterator_delete(). - Handle malloc fail. - Use luaT_call instead of lua_call. See https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,2 and https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,7 Changes since v2 for 'net.box: add helpers to decode msgpack headers': - Splitted this patch to two ones: - lua: add non-recursive msgpack decoding functions - net.box: add skip_header option to use with buffer See https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,2 and https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,3 Changes for 'Add merger for tuple streams': - Got rid of sources reallocation during sources parsing. - Fixed a memleak in :ipairs(). - Set a diag in merge_next() (it will be a C part). - Renamed opts.descending -> reverse as Vova suggests. - Support fetch callback in iterator sources (to change it to tuple sources later). - Found and fixed table source freeing bug. - Unified fetch source API: return a new buffer from a fetch callback. - Sources reference counting (to return it to Lua in the future). - Use cdata wrappers for merger sources. - Moved fetch callback to sources. - Use global lua_State for unreferencing. - Got rid of lua_State in arguments of virtual merger source functions. - Replaced fetch callback with a Lua iterator. - Postpone a first source fetch until merger_next() is called. - Set a diag rather then push an error to a Lua stack. - Replaced an iterator source with a tuple source. - lbox_merger_new_*_source(): removed arguments popping. - Removed obsolete comments re Lua stack. - Moved source->tuple accesses from a source to a merger. - Implemented refcounting for merger_context. - Moved methods into a cdata - Splitted merger.new() and :select() opts. - Test: cache merger context for each schema. - Handled the case with a source is used in two mergers. - Added limit option to :select(). - Splitted to C and Lua modules. - Deduplicated code around source creating. - Test: decreased run time, removed from long_run. - Rewrote the merger as a source type. - Added cascade mergers test case. - Splitted a source and a heap node. - Created short API description for docbot comment. - Factored out use cases description and code examples into a separate repo (see the link at the start of the email). - Added a unit test with simple array source. - Splitted C and Lua into separate commits. - Test: ensure we're able to reuse a source. - Finished with the problem with empty buffers: ensure we're call fetch function in the case. issue: https://github.com/tarantool/tarantool/issues/3276 branch: https://github.com/tarantool/tarantool/tree/Totktonada/gh-3276-on-board-merger The branch based on key_def Lua module. It is needed for merger, but is now under review (within a separate thread). Please, don't push last two patches before key_defs will land. Alexander Turenko (7): Add luaL_iscallable with support of cdata metatype Add functions to ease using Lua iterators from C lua: optimize creation of a tuple from a tuple lua: add non-recursive msgpack decoding functions net.box: add skip_header option to use with buffer Add merger for tuples streams (C part) Add merger for tuple streams (Lua part) extra/exports | 1 + src/box/CMakeLists.txt | 3 + src/box/lua/init.c | 7 +- src/box/lua/merger.c | 1184 ++++++++++++++++++++++++++++++ src/box/lua/merger.h | 47 ++ src/box/lua/merger.lua | 41 ++ src/box/lua/net_box.lua | 46 +- src/box/lua/tuple.c | 65 +- src/box/merger.c | 464 ++++++++++++ src/box/merger.h | 180 +++++ src/lua/msgpack.c | 80 ++ src/lua/utils.c | 135 ++++ src/lua/utils.h | 47 ++ test/app-tap/module_api.c | 10 + test/app-tap/module_api.test.lua | 85 ++- test/app-tap/msgpack.test.lua | 180 ++++- test/box-tap/merger.test.lua | 725 ++++++++++++++++++ test/box/net.box.result | 222 +++++- test/box/net.box.test.lua | 86 ++- test/unit/CMakeLists.txt | 7 + test/unit/luaL_iterator.c | 208 ++++++ test/unit/luaL_iterator.result | 89 +++ test/unit/merger.result | 71 ++ test/unit/merger.test.c | 301 ++++++++ 24 files changed, 4230 insertions(+), 54 deletions(-) create mode 100644 src/box/lua/merger.c create mode 100644 src/box/lua/merger.h create mode 100644 src/box/lua/merger.lua create mode 100644 src/box/merger.c create mode 100644 src/box/merger.h create mode 100755 test/box-tap/merger.test.lua create mode 100644 test/unit/luaL_iterator.c create mode 100644 test/unit/luaL_iterator.result create mode 100644 test/unit/merger.result create mode 100644 test/unit/merger.test.c -- 2.20.1