Tarantool development patches archive
 help / color / mirror / Atom feed
From: Alexander Turenko <alexander.turenko@tarantool.org>
To: Vladimir Davydov <vdavydov.dev@gmail.com>
Cc: Alexander Turenko <alexander.turenko@tarantool.org>,
	tarantool-patches@freelists.org
Subject: [PATCH v3 0/7] Merger
Date: Wed, 10 Apr 2019 18:21:18 +0300	[thread overview]
Message-ID: <cover.1554906327.git.alexander.turenko@tarantool.org> (raw)

This patchset contains the set of auxiliary patches and two main ones:
the merger itself and the Lua wrapper for it with specific sources.

Previous version of the patchset was discussed here:
https://www.freelists.org/post/tarantool-patches/PATCH-v2-06-Merger
(also there was '[#3276] Merger API RFC' internal thread about the API).

More documentation and usage examples:
https://github.com/Totktonada/tarantool-merger-examples

Changes since v2 for 'Add functions to ease using Lua iterators from C':

- Fixed zero gen() retvals case.
- Added a unit test.
- Removed unneded cast from (void *).
- Allow to pass values w/o a table.
  - Also removed _fromtable suffix from luaL_iterator_new().
- luaL_iterator_free() -> luaL_iterator_delete().
- Handle malloc fail.
- Use luaT_call instead of lua_call.

See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-26-Add-functions-to-ease-using-Lua-iterators-from-C,7

Changes since v2 for 'net.box: add helpers to decode msgpack headers':

- Splitted this patch to two ones:
  - lua: add non-recursive msgpack decoding functions
  - net.box: add skip_header option to use with buffer

See
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,2
and
https://www.freelists.org/post/tarantool-patches/PATCH-v2-56-netbox-add-helpers-to-decode-msgpack-headers,3

Changes for 'Add merger for tuple streams':

- Got rid of sources reallocation during sources parsing.
- Fixed a memleak in :ipairs().
- Set a diag in merge_next() (it will be a C part).
- Renamed opts.descending -> reverse as Vova suggests.
- Support fetch callback in iterator sources (to change it to tuple
  sources later).
- Found and fixed table source freeing bug.
- Unified fetch source API: return a new buffer from a fetch callback.
- Sources reference counting (to return it to Lua in the future).
- Use cdata wrappers for merger sources.
- Moved fetch callback to sources.
- Use global lua_State for unreferencing.
- Got rid of lua_State in arguments of virtual merger source functions.
- Replaced fetch callback with a Lua iterator.
- Postpone a first source fetch until merger_next() is called.
- Set a diag rather then push an error to a Lua stack.
- Replaced an iterator source with a tuple source.
- lbox_merger_new_*_source(): removed arguments popping.
- Removed obsolete comments re Lua stack.
- Moved source->tuple accesses from a source to a merger.
- Implemented refcounting for merger_context.
- Moved methods into a cdata<struct merger_state>
- Splitted merger.new() and :select() opts.
- Test: cache merger context for each schema.
- Handled the case with a source is used in two mergers.
- Added limit option to :select().
- Splitted to C and Lua modules.
- Deduplicated code around source creating.
- Test: decreased run time, removed from long_run.
- Rewrote the merger as a source type.
- Added cascade mergers test case.
- Splitted a source and a heap node.
- Created short API description for docbot comment.
- Factored out use cases description and code examples into a separate
  repo (see the link at the start of the email).
- Added a unit test with simple array source.
- Splitted C and Lua into separate commits.
- Test: ensure we're able to reuse a source.
- Finished with the problem with empty buffers: ensure we're call fetch
  function in the case.

issue: https://github.com/tarantool/tarantool/issues/3276
branch: https://github.com/tarantool/tarantool/tree/Totktonada/gh-3276-on-board-merger

The branch based on key_def Lua module. It is needed for merger, but is
now under review (within a separate thread). Please, don't push last two
patches before key_defs will land.

Alexander Turenko (7):
  Add luaL_iscallable with support of cdata metatype
  Add functions to ease using Lua iterators from C
  lua: optimize creation of a tuple from a tuple
  lua: add non-recursive msgpack decoding functions
  net.box: add skip_header option to use with buffer
  Add merger for tuples streams (C part)
  Add merger for tuple streams (Lua part)

 extra/exports                    |    1 +
 src/box/CMakeLists.txt           |    3 +
 src/box/lua/init.c               |    7 +-
 src/box/lua/merger.c             | 1184 ++++++++++++++++++++++++++++++
 src/box/lua/merger.h             |   47 ++
 src/box/lua/merger.lua           |   41 ++
 src/box/lua/net_box.lua          |   46 +-
 src/box/lua/tuple.c              |   65 +-
 src/box/merger.c                 |  464 ++++++++++++
 src/box/merger.h                 |  180 +++++
 src/lua/msgpack.c                |   80 ++
 src/lua/utils.c                  |  135 ++++
 src/lua/utils.h                  |   47 ++
 test/app-tap/module_api.c        |   10 +
 test/app-tap/module_api.test.lua |   85 ++-
 test/app-tap/msgpack.test.lua    |  180 ++++-
 test/box-tap/merger.test.lua     |  725 ++++++++++++++++++
 test/box/net.box.result          |  222 +++++-
 test/box/net.box.test.lua        |   86 ++-
 test/unit/CMakeLists.txt         |    7 +
 test/unit/luaL_iterator.c        |  208 ++++++
 test/unit/luaL_iterator.result   |   89 +++
 test/unit/merger.result          |   71 ++
 test/unit/merger.test.c          |  301 ++++++++
 24 files changed, 4230 insertions(+), 54 deletions(-)
 create mode 100644 src/box/lua/merger.c
 create mode 100644 src/box/lua/merger.h
 create mode 100644 src/box/lua/merger.lua
 create mode 100644 src/box/merger.c
 create mode 100644 src/box/merger.h
 create mode 100755 test/box-tap/merger.test.lua
 create mode 100644 test/unit/luaL_iterator.c
 create mode 100644 test/unit/luaL_iterator.result
 create mode 100644 test/unit/merger.result
 create mode 100644 test/unit/merger.test.c

-- 
2.20.1

             reply	other threads:[~2019-04-10 15:21 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-10 15:21 Alexander Turenko [this message]
2019-04-10 15:21 ` [PATCH v3 1/7] Add luaL_iscallable with support of cdata metatype Alexander Turenko
2019-04-18 17:30   ` [tarantool-patches] " Konstantin Osipov
2019-04-30 12:45   ` Vladimir Davydov
2019-04-10 15:21 ` [PATCH v3 2/7] Add functions to ease using Lua iterators from C Alexander Turenko
2019-04-18 17:31   ` [tarantool-patches] " Konstantin Osipov
2019-04-30 12:46   ` Vladimir Davydov
2019-04-10 15:21 ` [PATCH v3 3/7] lua: optimize creation of a tuple from a tuple Alexander Turenko
2019-04-18 17:32   ` [tarantool-patches] " Konstantin Osipov
2019-04-30 12:50   ` Vladimir Davydov
2019-04-30 15:07     ` Alexander Turenko
2019-04-10 15:21 ` [PATCH v3 4/7] lua: add non-recursive msgpack decoding functions Alexander Turenko
2019-04-18 17:35   ` [tarantool-patches] " Konstantin Osipov
2019-04-18 18:30     ` Alexander Turenko
2019-04-18 18:33       ` Konstantin Osipov
2019-04-18 18:44         ` Alexander Turenko
2019-04-30 13:03   ` Vladimir Davydov
2019-04-30 18:38     ` Alexander Turenko
2019-04-10 15:21 ` [PATCH v3 5/7] net.box: add skip_header option to use with buffer Alexander Turenko
2019-04-18 17:37   ` [tarantool-patches] " Konstantin Osipov
2019-04-18 18:39     ` Alexander Turenko
2019-04-30 13:16   ` Vladimir Davydov
2019-04-30 18:39     ` Alexander Turenko
2019-04-10 15:21 ` [PATCH v3 6/7] Add merger for tuples streams (C part) Alexander Turenko
2019-04-25 11:43   ` [tarantool-patches] " Konstantin Osipov
2019-04-25 13:32     ` Alexander Turenko
2019-04-25 13:45       ` Konstantin Osipov
2019-04-25 15:32         ` [tarantool-patches] " Alexander Turenko
2019-04-25 16:42           ` Konstantin Osipov
2019-04-30 15:34   ` Vladimir Davydov
2019-05-07 22:14     ` Alexander Turenko
2019-04-10 15:21 ` [PATCH v3 7/7] Add merger for tuple streams (Lua part) Alexander Turenko
2019-04-25 11:46   ` [tarantool-patches] " Konstantin Osipov
2019-04-25 12:53     ` Alexander Turenko
2019-04-25 13:30       ` Konstantin Osipov
2019-04-30 17:37   ` Vladimir Davydov
2019-04-30 21:09     ` [tarantool-patches] " Konstantin Osipov
2019-05-02  9:48       ` Vladimir Davydov
2019-05-07 22:14     ` Alexander Turenko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1554906327.git.alexander.turenko@tarantool.org \
    --to=alexander.turenko@tarantool.org \
    --cc=tarantool-patches@freelists.org \
    --cc=vdavydov.dev@gmail.com \
    --subject='Re: [PATCH v3 0/7] Merger' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox