From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: Oleg Babin <olegrok@tarantool.org>,
tarantool-patches@dev.tarantool.org,
yaroslav.dynnikov@tarantool.org
Subject: Re: [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count
Date: Wed, 24 Feb 2021 22:47:28 +0100 [thread overview]
Message-ID: <86016630-27a0-da45-e71b-ebd573dada3b@tarantool.org> (raw)
In-Reply-To: <542ed6d8-9b68-7455-5a79-b192078c139e@tarantool.org>
Thanks for the review!
On 24.02.2021 11:27, Oleg Babin wrote:
> Thanks for your patch! LGTM.
>
> I see calls like "status_index:count({consts.BUCKET.ACTIVE})". Maybe it worth
>
> to cache whole buckets stats as well?
I thought about it a lot. But realized that I need only a few cached
metrics used for most of the requests. Count of active buckets is not
one of them, but would waste time on invaliding the cache on each
generation update.
Talking specifically, count({consts.BUCKET.ACTIVE}) is used by
rebalancer only which happens extremely rare. So there is no win in
optimizing it for normal cluster operation.
Even now I worry about doing too much in the generation increment
trigger. To calculate and keep the stat up to date I would need to
make it more universal. So for example store number of buckets of
each type. Then I face the issues:
- In on_replace trigger I need to extract bucket status from the old
and new tuple, update the relevant counters. I mostly worry about
extracting the statuses (too long).
- I need to handle the rollback to somehow revert the counters back.
I could do something similar to the cache in this patch (I simply
calculate the counts on demand and invalidate them all on each generation
update), but it does not fix the real issue with the counts - they
can be long if bucket count is millions, and the cache will be invalidated
a lot during rebalancing. Exactly when a cache could help most.
In the end I decided not to bother with this now in scope of map-reduce.
next prev parent reply other threads:[~2021-02-24 21:47 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-23 0:15 [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 01/11] error: introduce vshard.error.timeout() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-24 21:46 ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 10/11] sched: introduce vshard.storage.sched module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28 ` Oleg Babin via Tarantool-patches
2021-02-24 21:50 ` Vladislav Shpilevoy via Tarantool-patches
2021-03-04 21:02 ` Oleg Babin via Tarantool-patches
2021-03-05 22:06 ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09 8:03 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 11/11] router: introduce map_callrw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28 ` Oleg Babin via Tarantool-patches
2021-02-24 22:04 ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:43 ` Oleg Babin via Tarantool-patches
2021-02-26 23:58 ` Vladislav Shpilevoy via Tarantool-patches
2021-03-01 10:58 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 02/11] storage: add helper for local functions invocation Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-24 21:47 ` Vladislav Shpilevoy via Tarantool-patches [this message]
2021-02-25 12:42 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 05/11] util: introduce safe fiber_cond_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-24 21:48 ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 06/11] util: introduce fiber_is_self_canceled() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 07/11] storage: introduce bucket_generation_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 08/11] storage: introduce bucket_are_all_rw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27 ` Oleg Babin via Tarantool-patches
2021-02-24 21:48 ` Vladislav Shpilevoy via Tarantool-patches
2021-02-23 0:15 ` [Tarantool-patches] [PATCH vshard 09/11] ref: introduce vshard.storage.ref module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28 ` Oleg Babin via Tarantool-patches
2021-02-24 21:49 ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42 ` Oleg Babin via Tarantool-patches
2021-03-04 21:22 ` Oleg Babin via Tarantool-patches
2021-03-05 22:06 ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09 8:03 ` Oleg Babin via Tarantool-patches
2021-03-21 18:49 ` Vladislav Shpilevoy via Tarantool-patches
2021-03-12 23:13 ` [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-03-15 7:05 ` Oleg Babin via Tarantool-patches
2021-03-28 18:17 ` Vladislav Shpilevoy via Tarantool-patches
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=86016630-27a0-da45-e71b-ebd573dada3b@tarantool.org \
--to=tarantool-patches@dev.tarantool.org \
--cc=olegrok@tarantool.org \
--cc=v.shpilevoy@tarantool.org \
--cc=yaroslav.dynnikov@tarantool.org \
--subject='Re: [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count' \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox