From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id DB2B22B84B for ; Tue, 27 Mar 2018 17:24:19 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vwuG_RGPNdIu for ; Tue, 27 Mar 2018 17:24:19 -0400 (EDT) Received: from smtp43.i.mail.ru (smtp43.i.mail.ru [94.100.177.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 9984C2A610 for ; Tue, 27 Mar 2018 17:24:19 -0400 (EDT) From: Vladislav Shpilevoy Subject: [tarantool-patches] [PATCH vshard 0/7] Replicaset lock and bucket pin Date: Wed, 28 Mar 2018 00:24:07 +0300 Message-Id: Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-subscribe: List-owner: List-post: List-archive: To: tarantool-patches@freelists.org Cc: georgy@tarantool.org, Vladislav Shpilevoy Branch: http://github.com/tarantool/tarantool/tree/gh-71-bucket-pin-lock Issue: https://github.com/tarantool/vshard/issues/71 Replicaset lock makes it invisible for the rebalancer - a locked replicaset can neither receive new buckets nor send its own. Bucket pin blocks this concrete bucket sending - it will stay on a replicaset to which pinned, until it is unpinned. Pinning all replicaset buckets is not the same as replicaset locking - even if you pin all buckets, the non-locked replicaset still can receive new buckets. Replicaset lock allows, for example, to separate a replicaset for testsing from production replicasets. Or to preserve some application metadata, that must not be sharded for a while. Bucket pin allows the same, but in the smaller scope. Difference between replicaset lock and all buckets pinning is motivated by ability to button-up an entire replicaset. Mostly locked and pinned buckets affect the rebalancing algorithm, which must ignore locked replicasets, and take pinned buckets into account, attempting to reach the best possible balance. It is not a trivial task, because a user can pin to a replicaset so many buckets, that a perfect balance is unreachable. Vladislav Shpilevoy (7): rebalancer: allow to lock a replicaset from rebalancing rebalancer: remember the currently sending bucket id storage: rework recovery storage: wrap bucket status checks into functions rebalancer: introduce pinned bucket concept into rebalancer algo storage: open public API to pin/unpin buckets rfc: add RFC for replicaset lock and bucket pin docs/RFC/replicaset_lock_and_bucket.md | 84 ++++ test/rebalancer/box_1_a.lua | 2 +- test/rebalancer/rebalancer_lock_and_pin.result | 503 ++++++++++++++++++++++ test/rebalancer/rebalancer_lock_and_pin.test.lua | 224 ++++++++++ test/rebalancer/restart_during_rebalancing.result | 4 + test/storage/recovery.result | 85 +++- test/storage/recovery.test.lua | 38 +- test/storage/storage.result | 7 + test/unit/rebalancer.result | 333 +++++++++++++- test/unit/rebalancer.test.lua | 79 ++++ vshard/cfg.lua | 3 +- vshard/consts.lua | 1 + vshard/error.lua | 12 +- vshard/replicaset.lua | 105 +++-- vshard/router/init.lua | 46 +- vshard/storage/init.lua | 418 +++++++++++------- 16 files changed, 1713 insertions(+), 231 deletions(-) create mode 100644 docs/RFC/replicaset_lock_and_bucket.md create mode 100644 test/rebalancer/rebalancer_lock_and_pin.result create mode 100644 test/rebalancer/rebalancer_lock_and_pin.test.lua -- 2.14.3 (Apple Git-98)