Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org,
	yaroslav.dynnikov@tarantool.org
Subject: [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations
Date: Wed, 10 Feb 2021 00:46:06 +0100	[thread overview]
Message-ID: <cover.1612914070.git.v.shpilevoy@tarantool.org> (raw)

The patchset makes some preparations for the incoming consistent map-reduce
feature. Mostly it reworks bucket GC and recovery so as they wouldn't block
map-reduce requests for too long time.

The last commit is a first big part related directly to map-reduce. It
introduces binary heap data structure implementation, which is going to be a
core storage for map-reduce artifacts.

Some commits were done alongside while working on the code. These are rlist
extraction, and fix of a bug in bucket_recv. I did them in first version of the
patchset as necessary, and then after some reworks they were not used anymore.
But I still want to merge them. One fixes a potential bug, other improves code
readability.

Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-147-map-reduce-part1
Issue: https://github.com/tarantool/tarantool/issues/147

Vladislav Shpilevoy (9):
  rlist: move rlist to a new module
  Use fiber.clock() instead of .time() everywhere
  test: introduce a helper to wait for bucket GC
  storage: bucket_recv() should check rs lock
  util: introduce yielding table functions
  cfg: introduce 'deprecated option' feature
  gc: introduce reactive garbage collector
  recovery: introduce reactive recovery
  util: introduce binary heap data structure

 test/failover/failover.result                 |   4 +-
 test/failover/failover.test.lua               |   4 +-
 test/lua_libs/storage_template.lua            |   9 +
 test/misc/reconfigure.result                  |  10 -
 test/misc/reconfigure.test.lua                |   3 -
 test/rebalancer/bucket_ref.result             |  19 +-
 test/rebalancer/bucket_ref.test.lua           |   8 +-
 test/rebalancer/errinj.result                 |  20 +-
 test/rebalancer/errinj.test.lua               |  12 +-
 test/rebalancer/rebalancer.result             |   5 +-
 test/rebalancer/rebalancer.test.lua           |   3 +-
 .../rebalancer/rebalancer_lock_and_pin.result |  14 +
 .../rebalancer_lock_and_pin.test.lua          |   4 +
 test/rebalancer/receiving_bucket.result       |  10 +-
 test/rebalancer/receiving_bucket.test.lua     |   3 +-
 test/reload_evolution/storage.result          |   7 +-
 test/reload_evolution/storage.test.lua        |   3 +-
 test/router/reroute_wrong_bucket.result       |   8 +-
 test/router/reroute_wrong_bucket.test.lua     |   4 +-
 test/router/router.result                     |  22 +-
 test/router/router.test.lua                   |  13 +-
 test/storage/recovery.result                  |  11 +-
 test/storage/recovery.test.lua                |   5 +
 test/storage/recovery_errinj.result           |  16 +-
 test/storage/recovery_errinj.test.lua         |   9 +-
 test/storage/storage.result                   |  10 +-
 test/storage/storage.test.lua                 |   1 +
 test/unit-tap/heap.test.lua                   | 310 ++++++++++++++++
 test/unit-tap/suite.ini                       |   4 +
 test/unit/config.result                       |  35 +-
 test/unit/config.test.lua                     |  16 +-
 test/unit/garbage.result                      | 106 +++---
 test/unit/garbage.test.lua                    |  47 ++-
 test/unit/garbage_errinj.result               | 223 ------------
 test/unit/garbage_errinj.test.lua             |  73 ----
 test/unit/rebalancer.result                   |  99 -----
 test/unit/rebalancer.test.lua                 |  27 --
 test/unit/rlist.result                        | 114 ++++++
 test/unit/rlist.test.lua                      |  33 ++
 test/unit/util.result                         | 113 ++++++
 test/unit/util.test.lua                       |  49 +++
 vshard/cfg.lua                                |  10 +-
 vshard/consts.lua                             |   7 +-
 vshard/heap.lua                               | 226 ++++++++++++
 vshard/replicaset.lua                         |  13 +-
 vshard/rlist.lua                              |  53 +++
 vshard/router/init.lua                        |  16 +-
 vshard/storage/init.lua                       | 342 +++++++++---------
 vshard/storage/reload_evolution.lua           |   8 +
 vshard/util.lua                               |  40 ++
 50 files changed, 1368 insertions(+), 833 deletions(-)
 create mode 100755 test/unit-tap/heap.test.lua
 create mode 100644 test/unit-tap/suite.ini
 delete mode 100644 test/unit/garbage_errinj.result
 delete mode 100644 test/unit/garbage_errinj.test.lua
 create mode 100644 test/unit/rlist.result
 create mode 100644 test/unit/rlist.test.lua
 create mode 100644 vshard/heap.lua
 create mode 100644 vshard/rlist.lua

-- 
2.24.3 (Apple Git-128)


             reply	other threads:[~2021-02-09 23:46 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-09 23:46 Vladislav Shpilevoy via Tarantool-patches [this message]
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 1/9] rlist: move rlist to a new module Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:57   ` Oleg Babin via Tarantool-patches
2021-02-11  6:50     ` Oleg Babin via Tarantool-patches
2021-02-12  0:09       ` Vladislav Shpilevoy via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 2/9] Use fiber.clock() instead of .time() everywhere Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:57   ` Oleg Babin via Tarantool-patches
2021-02-10 22:33     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 3/9] test: introduce a helper to wait for bucket GC Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:57   ` Oleg Babin via Tarantool-patches
2021-02-10 22:33     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-11  6:50       ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 4/9] storage: bucket_recv() should check rs lock Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:59   ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 5/9] util: introduce yielding table functions Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:59   ` Oleg Babin via Tarantool-patches
2021-02-10 22:34     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-11  6:50       ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 6/9] cfg: introduce 'deprecated option' feature Vladislav Shpilevoy via Tarantool-patches
2021-02-10  8:59   ` Oleg Babin via Tarantool-patches
2021-02-10 22:34     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-11  6:50       ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 7/9] gc: introduce reactive garbage collector Vladislav Shpilevoy via Tarantool-patches
2021-02-10  9:00   ` Oleg Babin via Tarantool-patches
2021-02-10 22:35     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-11  6:50       ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 8/9] recovery: introduce reactive recovery Vladislav Shpilevoy via Tarantool-patches
2021-02-10  9:00   ` Oleg Babin via Tarantool-patches
2021-02-09 23:46 ` [Tarantool-patches] [PATCH 9/9] util: introduce binary heap data structure Vladislav Shpilevoy via Tarantool-patches
2021-02-10  9:01   ` Oleg Babin via Tarantool-patches
2021-02-10 22:36     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-11  6:51       ` Oleg Babin via Tarantool-patches
2021-02-12  0:09         ` Vladislav Shpilevoy via Tarantool-patches
2021-03-05 22:03   ` Vladislav Shpilevoy via Tarantool-patches
2021-02-09 23:51 ` [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations Vladislav Shpilevoy via Tarantool-patches
2021-02-12 11:02   ` Oleg Babin via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1612914070.git.v.shpilevoy@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=olegrok@tarantool.org \
    --cc=v.shpilevoy@tarantool.org \
    --cc=yaroslav.dynnikov@tarantool.org \
    --subject='Re: [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox