From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org> To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org, yaroslav.dynnikov@tarantool.org Subject: [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations Date: Wed, 10 Feb 2021 00:46:06 +0100 [thread overview] Message-ID: <cover.1612914070.git.v.shpilevoy@tarantool.org> (raw) The patchset makes some preparations for the incoming consistent map-reduce feature. Mostly it reworks bucket GC and recovery so as they wouldn't block map-reduce requests for too long time. The last commit is a first big part related directly to map-reduce. It introduces binary heap data structure implementation, which is going to be a core storage for map-reduce artifacts. Some commits were done alongside while working on the code. These are rlist extraction, and fix of a bug in bucket_recv. I did them in first version of the patchset as necessary, and then after some reworks they were not used anymore. But I still want to merge them. One fixes a potential bug, other improves code readability. Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-147-map-reduce-part1 Issue: https://github.com/tarantool/tarantool/issues/147 Vladislav Shpilevoy (9): rlist: move rlist to a new module Use fiber.clock() instead of .time() everywhere test: introduce a helper to wait for bucket GC storage: bucket_recv() should check rs lock util: introduce yielding table functions cfg: introduce 'deprecated option' feature gc: introduce reactive garbage collector recovery: introduce reactive recovery util: introduce binary heap data structure test/failover/failover.result | 4 +- test/failover/failover.test.lua | 4 +- test/lua_libs/storage_template.lua | 9 + test/misc/reconfigure.result | 10 - test/misc/reconfigure.test.lua | 3 - test/rebalancer/bucket_ref.result | 19 +- test/rebalancer/bucket_ref.test.lua | 8 +- test/rebalancer/errinj.result | 20 +- test/rebalancer/errinj.test.lua | 12 +- test/rebalancer/rebalancer.result | 5 +- test/rebalancer/rebalancer.test.lua | 3 +- .../rebalancer/rebalancer_lock_and_pin.result | 14 + .../rebalancer_lock_and_pin.test.lua | 4 + test/rebalancer/receiving_bucket.result | 10 +- test/rebalancer/receiving_bucket.test.lua | 3 +- test/reload_evolution/storage.result | 7 +- test/reload_evolution/storage.test.lua | 3 +- test/router/reroute_wrong_bucket.result | 8 +- test/router/reroute_wrong_bucket.test.lua | 4 +- test/router/router.result | 22 +- test/router/router.test.lua | 13 +- test/storage/recovery.result | 11 +- test/storage/recovery.test.lua | 5 + test/storage/recovery_errinj.result | 16 +- test/storage/recovery_errinj.test.lua | 9 +- test/storage/storage.result | 10 +- test/storage/storage.test.lua | 1 + test/unit-tap/heap.test.lua | 310 ++++++++++++++++ test/unit-tap/suite.ini | 4 + test/unit/config.result | 35 +- test/unit/config.test.lua | 16 +- test/unit/garbage.result | 106 +++--- test/unit/garbage.test.lua | 47 ++- test/unit/garbage_errinj.result | 223 ------------ test/unit/garbage_errinj.test.lua | 73 ---- test/unit/rebalancer.result | 99 ----- test/unit/rebalancer.test.lua | 27 -- test/unit/rlist.result | 114 ++++++ test/unit/rlist.test.lua | 33 ++ test/unit/util.result | 113 ++++++ test/unit/util.test.lua | 49 +++ vshard/cfg.lua | 10 +- vshard/consts.lua | 7 +- vshard/heap.lua | 226 ++++++++++++ vshard/replicaset.lua | 13 +- vshard/rlist.lua | 53 +++ vshard/router/init.lua | 16 +- vshard/storage/init.lua | 342 +++++++++--------- vshard/storage/reload_evolution.lua | 8 + vshard/util.lua | 40 ++ 50 files changed, 1368 insertions(+), 833 deletions(-) create mode 100755 test/unit-tap/heap.test.lua create mode 100644 test/unit-tap/suite.ini delete mode 100644 test/unit/garbage_errinj.result delete mode 100644 test/unit/garbage_errinj.test.lua create mode 100644 test/unit/rlist.result create mode 100644 test/unit/rlist.test.lua create mode 100644 vshard/heap.lua create mode 100644 vshard/rlist.lua -- 2.24.3 (Apple Git-128)
next reply other threads:[~2021-02-09 23:46 UTC|newest] Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-02-09 23:46 Vladislav Shpilevoy via Tarantool-patches [this message] 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 1/9] rlist: move rlist to a new module Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:57 ` Oleg Babin via Tarantool-patches 2021-02-11 6:50 ` Oleg Babin via Tarantool-patches 2021-02-12 0:09 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 2/9] Use fiber.clock() instead of .time() everywhere Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:57 ` Oleg Babin via Tarantool-patches 2021-02-10 22:33 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 3/9] test: introduce a helper to wait for bucket GC Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:57 ` Oleg Babin via Tarantool-patches 2021-02-10 22:33 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-11 6:50 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 4/9] storage: bucket_recv() should check rs lock Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:59 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 5/9] util: introduce yielding table functions Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:59 ` Oleg Babin via Tarantool-patches 2021-02-10 22:34 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-11 6:50 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 6/9] cfg: introduce 'deprecated option' feature Vladislav Shpilevoy via Tarantool-patches 2021-02-10 8:59 ` Oleg Babin via Tarantool-patches 2021-02-10 22:34 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-11 6:50 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 7/9] gc: introduce reactive garbage collector Vladislav Shpilevoy via Tarantool-patches 2021-02-10 9:00 ` Oleg Babin via Tarantool-patches 2021-02-10 22:35 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-11 6:50 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 8/9] recovery: introduce reactive recovery Vladislav Shpilevoy via Tarantool-patches 2021-02-10 9:00 ` Oleg Babin via Tarantool-patches 2021-02-09 23:46 ` [Tarantool-patches] [PATCH 9/9] util: introduce binary heap data structure Vladislav Shpilevoy via Tarantool-patches 2021-02-10 9:01 ` Oleg Babin via Tarantool-patches 2021-02-10 22:36 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-11 6:51 ` Oleg Babin via Tarantool-patches 2021-02-12 0:09 ` Vladislav Shpilevoy via Tarantool-patches 2021-03-05 22:03 ` Vladislav Shpilevoy via Tarantool-patches 2021-02-09 23:51 ` [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations Vladislav Shpilevoy via Tarantool-patches 2021-02-12 11:02 ` Oleg Babin via Tarantool-patches
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=cover.1612914070.git.v.shpilevoy@tarantool.org \ --to=tarantool-patches@dev.tarantool.org \ --cc=olegrok@tarantool.org \ --cc=v.shpilevoy@tarantool.org \ --cc=yaroslav.dynnikov@tarantool.org \ --subject='Re: [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox