From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp41.i.mail.ru (smtp41.i.mail.ru [94.100.177.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 29E0E4696C6 for ; Fri, 1 May 2020 03:16:36 +0300 (MSK) From: Vladislav Shpilevoy Date: Fri, 1 May 2020 02:16:27 +0200 Message-Id: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH vshard 0/7] Router extended discovery List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org Goal of the patchset - solve the problem of routers downloading buckets from storages in too big chunks, making storages waste seconds of TX thread time, when they need to return a million of buckets without yields. Additionally it was asked to allow to stop discovery after all buckets are already discovered by router. For that a new configuration option is introduced and a public function. Branch: http://github.com/tarantool/vshard/tree/gerold103/gh-210-discovery-millions-of-buckets Issue: https://github.com/tarantool/vshard/issues/210 Vladislav Shpilevoy (7): test: print errors in a portable way router: introduce discovery_mode test: disable router discovery for some tests test: clear route map, respecting statistics router: keep known bucket count stat up to date router: make discovery smoother in a big cluster router: introduce discovery mode 'once' example/router.lua | 3 + test/lua_libs/util.lua | 9 + test/misc/check_uuid_on_connect.result | 20 +- test/misc/check_uuid_on_connect.test.lua | 7 +- test/rebalancer/bucket_ref.result | 8 +- test/rebalancer/bucket_ref.test.lua | 3 +- test/rebalancer/receiving_bucket.result | 18 +- test/rebalancer/receiving_bucket.test.lua | 8 +- test/router/exponential_timeout.result | 12 +- test/router/exponential_timeout.test.lua | 4 +- test/router/reload.result | 5 +- test/router/reload.test.lua | 3 +- test/router/retry_reads.result | 16 +- test/router/retry_reads.test.lua | 10 +- test/router/router.result | 56 +++--- test/router/router.test.lua | 28 ++- test/router/router2.result | 221 ++++++++++++++++++++++ test/router/router2.test.lua | 83 ++++++++ test/router/sync.result | 15 +- test/router/sync.test.lua | 4 +- test/router/wrong_config.result | 5 +- test/router/wrong_config.test.lua | 5 +- test/storage/storage.result | 9 +- test/storage/storage.test.lua | 3 +- test/unit/error.result | 18 +- test/unit/error.test.lua | 9 +- vshard/cfg.lua | 10 + vshard/consts.lua | 5 +- vshard/router/init.lua | 203 ++++++++++++++++---- vshard/storage/init.lua | 39 +++- 30 files changed, 667 insertions(+), 172 deletions(-) create mode 100644 test/router/router2.result create mode 100644 test/router/router2.test.lua -- 2.21.1 (Apple Git-122.3)