From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 6C06870361; Wed, 10 Feb 2021 02:46:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6C06870361 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1612914378; bh=p2efFGHKuTzq5mPBStx9E8SzLx4NHV39gk0QxJZzzkg=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=BzZjUp9xi3LTx5XuoiV4u/aEdSIZM/I6AtnxMM0cnBkpVPYCNo6spYMLEclbPBw30 9EySmmV8Orgfj/0ES0NDpk5OnIQViw+B8y8+l68QXjG3y/2LpT2STr+SNkVaPTMJkH k6nmc0beSnWGW68r/mPY47euBEA/1qxngweIAOtk= Received: from smtpng3.m.smailru.net (smtpng3.m.smailru.net [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 6ECD670361 for ; Wed, 10 Feb 2021 02:46:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6ECD670361 Received: by smtpng3.m.smailru.net with esmtpa (envelope-from ) id 1l9ci0-0002gH-27; Wed, 10 Feb 2021 02:46:16 +0300 To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org, yaroslav.dynnikov@tarantool.org Date: Wed, 10 Feb 2021 00:46:06 +0100 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD953AC099BC0052A9C4647521586BE7E637AB8B70F7375365A182A05F538085040BB41C10302641824CDB5CF8B28F269557DB4C3D258BC60DC4DE7BFD6B4FCBE0E X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE71BF69A9C8C5AF260EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B323FE155BC226618638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FCA6073DB25481D6C5C06317566C392BCF481D4EDFE7EFEDF1389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0A29E2F051442AF778941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B652D31B9D28593E51CC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC9149C560DC76099D75ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4930A3850AC1BE2E735B344165809136645C4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0FECB2555BB02FD5A93B503F486389A921A5CC5B56E945C8DA X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CFE63F531D4BE5B711ED27B5F5928105DB3480521318232389C2B6934AE262D3EE7EAB7254005DCED16151DBD8DF150511E0A4E2319210D9B64D260DF9561598F01A9E91200F654B0CCA28C6D779E2CD78E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D346840168BCAD8054E51960E685274F19B4CDFB40C0190153F3DF19087B842C33ED39A6F30A91DB5A81D7E09C32AA3244C7142857641CE75A74EB3B1E73F8C4488F94338140B71B8EEFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXgXyUHXkVnLhclmvI0UCupv X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822D09B54761BA69E20B0409B1FA2AFADC83841015FED1DE5223CC9A89AB576DD93FB559BB5D741EB963CF37A108A312F5C27E8A8C3839CE0E267EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH 0/9] VShard Map-Reduce, part 1, preparations X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" The patchset makes some preparations for the incoming consistent map-reduce feature. Mostly it reworks bucket GC and recovery so as they wouldn't block map-reduce requests for too long time. The last commit is a first big part related directly to map-reduce. It introduces binary heap data structure implementation, which is going to be a core storage for map-reduce artifacts. Some commits were done alongside while working on the code. These are rlist extraction, and fix of a bug in bucket_recv. I did them in first version of the patchset as necessary, and then after some reworks they were not used anymore. But I still want to merge them. One fixes a potential bug, other improves code readability. Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-147-map-reduce-part1 Issue: https://github.com/tarantool/tarantool/issues/147 Vladislav Shpilevoy (9): rlist: move rlist to a new module Use fiber.clock() instead of .time() everywhere test: introduce a helper to wait for bucket GC storage: bucket_recv() should check rs lock util: introduce yielding table functions cfg: introduce 'deprecated option' feature gc: introduce reactive garbage collector recovery: introduce reactive recovery util: introduce binary heap data structure test/failover/failover.result | 4 +- test/failover/failover.test.lua | 4 +- test/lua_libs/storage_template.lua | 9 + test/misc/reconfigure.result | 10 - test/misc/reconfigure.test.lua | 3 - test/rebalancer/bucket_ref.result | 19 +- test/rebalancer/bucket_ref.test.lua | 8 +- test/rebalancer/errinj.result | 20 +- test/rebalancer/errinj.test.lua | 12 +- test/rebalancer/rebalancer.result | 5 +- test/rebalancer/rebalancer.test.lua | 3 +- .../rebalancer/rebalancer_lock_and_pin.result | 14 + .../rebalancer_lock_and_pin.test.lua | 4 + test/rebalancer/receiving_bucket.result | 10 +- test/rebalancer/receiving_bucket.test.lua | 3 +- test/reload_evolution/storage.result | 7 +- test/reload_evolution/storage.test.lua | 3 +- test/router/reroute_wrong_bucket.result | 8 +- test/router/reroute_wrong_bucket.test.lua | 4 +- test/router/router.result | 22 +- test/router/router.test.lua | 13 +- test/storage/recovery.result | 11 +- test/storage/recovery.test.lua | 5 + test/storage/recovery_errinj.result | 16 +- test/storage/recovery_errinj.test.lua | 9 +- test/storage/storage.result | 10 +- test/storage/storage.test.lua | 1 + test/unit-tap/heap.test.lua | 310 ++++++++++++++++ test/unit-tap/suite.ini | 4 + test/unit/config.result | 35 +- test/unit/config.test.lua | 16 +- test/unit/garbage.result | 106 +++--- test/unit/garbage.test.lua | 47 ++- test/unit/garbage_errinj.result | 223 ------------ test/unit/garbage_errinj.test.lua | 73 ---- test/unit/rebalancer.result | 99 ----- test/unit/rebalancer.test.lua | 27 -- test/unit/rlist.result | 114 ++++++ test/unit/rlist.test.lua | 33 ++ test/unit/util.result | 113 ++++++ test/unit/util.test.lua | 49 +++ vshard/cfg.lua | 10 +- vshard/consts.lua | 7 +- vshard/heap.lua | 226 ++++++++++++ vshard/replicaset.lua | 13 +- vshard/rlist.lua | 53 +++ vshard/router/init.lua | 16 +- vshard/storage/init.lua | 342 +++++++++--------- vshard/storage/reload_evolution.lua | 8 + vshard/util.lua | 40 ++ 50 files changed, 1368 insertions(+), 833 deletions(-) create mode 100755 test/unit-tap/heap.test.lua create mode 100644 test/unit-tap/suite.ini delete mode 100644 test/unit/garbage_errinj.result delete mode 100644 test/unit/garbage_errinj.test.lua create mode 100644 test/unit/rlist.result create mode 100644 test/unit/rlist.test.lua create mode 100644 vshard/heap.lua create mode 100644 vshard/rlist.lua -- 2.24.3 (Apple Git-128)