Tarantool development patches archive
 help / color / mirror / Atom feed
From: Oleg Babin via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: Vladislav Shpilevoy <v.shpilevoy@tarantool.org>,
	tarantool-patches@dev.tarantool.org,
	yaroslav.dynnikov@tarantool.org
Subject: Re: [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution
Date: Wed, 24 Feb 2021 13:27:32 +0300
Message-ID: <5d24e9b3-e978-31a4-1ad0-af03e8c42da1@tarantool.org> (raw)
In-Reply-To: <92da76963b5bd26fa824ae152c7ed2dda7526320.1614039039.git.v.shpilevoy@tarantool.org>

Thanks for your patch. LGTM.

On 23.02.2021 03:15, Vladislav Shpilevoy wrote:
> Registry is a way to resolve cyclic dependencies which normally
> can exist between files of the same module/library.
>
> It is a global table hidden in _G with a long unlikely anywhere
> used name.
>
> Files, which want to expose their API to the other files, which in
> turn can't require the formers directly, should put their API to
> the registry.
>
> The files use the registry to get API of the other files. They
> don't require() and use the latter directly.
>
> At runtime, when all require() are done, the registry is full,
> and all the files see API of each other.
>
> Such circular dependency will exist between new files implementing
> map-reduce engine as a set of relatively independent submodules of
> the storage.
>
> In particular there will be storage_ref and storage_sched. Both
> require a few functions from the main storage file, and will use
> API of each other.
>
> Having the modules accessed via registry adds at lest +1 indexing
> operation at runtime when need to get a function from there. But
> sometimes it can be cached similar to how bucket count cache works
> in the main storage file.
>
> Main purpose is not to increase size of the main storage file
> again. It wouldn't fix the circular deps anyway, and would make it
> much harder to follow the code.
>
> Part of #147
> ---
>   vshard/CMakeLists.txt   |  3 +-
>   vshard/registry.lua     | 67 +++++++++++++++++++++++++++++++++++++++++
>   vshard/storage/init.lua |  5 ++-
>   3 files changed, 73 insertions(+), 2 deletions(-)
>   create mode 100644 vshard/registry.lua
>
> diff --git a/vshard/CMakeLists.txt b/vshard/CMakeLists.txt
> index 78a3f07..2a15df5 100644
> --- a/vshard/CMakeLists.txt
> +++ b/vshard/CMakeLists.txt
> @@ -7,4 +7,5 @@ add_subdirectory(router)
>   
>   # Install module
>   install(FILES cfg.lua error.lua consts.lua hash.lua init.lua replicaset.lua
> -        util.lua lua_gc.lua rlist.lua heap.lua DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard)
> +        util.lua lua_gc.lua rlist.lua heap.lua registry.lua
> +        DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard)
> diff --git a/vshard/registry.lua b/vshard/registry.lua
> new file mode 100644
> index 0000000..9583add
> --- /dev/null
> +++ b/vshard/registry.lua
> @@ -0,0 +1,67 @@
> +--
> +-- Registry is a way to resolve cyclic dependencies which normally can exist
> +-- between files of the same module/library.
> +--
> +-- Files, which want to expose their API to the other files, which in turn can't
> +-- require the formers directly, should put their API to the registry.
> +--
> +-- The files should use the registry to get API of the other files. They don't
> +-- require() and use the latter directly if there is a known loop dependency
> +-- between them.
> +--
> +-- At runtime, when all require() are done, the registry is full, and all the
> +-- files see API of each other.
> +--
> +-- Having the modules accessed via the registry adds at lest +1 indexing
> +-- operation at runtime when need to get a function from there. But sometimes it
> +-- can be cached to reduce the effect in perf-sensitive code. For example, like
> +-- this:
> +--
> +--     local lreg = require('vshard.registry')
> +--
> +--     local storage_func
> +--
> +--     local function storage_func_no_cache(...)
> +--         storage_func = lreg.storage.func
> +--         return storage_func(...)
> +--     end
> +--
> +--     storage_func = storage_func_no_cache
> +--
> +-- The code will always call storage_func(), but will load it from the registry
> +-- only on first invocation.
> +--
> +-- However in case reload is important, it is not possible - the original
> +-- function object in the registry may change. In such situation still makes
> +-- sense to cache at least 'lreg.storage' to save 1 indexing operation.
> +--
> +--     local lreg = require('vshard.registry')
> +--
> +--     local lstorage
> +--
> +--     local function storage_func_cache(...)
> +--         return lstorage.storage_func(...)
> +--     end
> +--
> +--     local function storage_func_no_cache(...)
> +--         lstorage = lref.storage
> +--         storage_func = storage_func_cache
> +--         return lstorage.storage_func(...)
> +--     end
> +--
> +--     storage_func = storage_func_no_cache
> +--
> +-- A harder way would be to use the first approach + add triggers on reload of
> +-- the cached module to update the cached function refs. If the code is
> +-- extremely perf-critical (which should not be Lua then).
> +--
> +
> +local MODULE_INTERNALS = '__module_vshard_registry'
> +
> +local M = rawget(_G, MODULE_INTERNALS)
> +if not M then
> +    M = {}
> +    rawset(_G, MODULE_INTERNALS, M)
> +end
> +
> +return M
> diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
> index 9b74bcb..b47665b 100644
> --- a/vshard/storage/init.lua
> +++ b/vshard/storage/init.lua
> @@ -16,7 +16,7 @@ if rawget(_G, MODULE_INTERNALS) then
>           'vshard.consts', 'vshard.error', 'vshard.cfg',
>           'vshard.replicaset', 'vshard.util',
>           'vshard.storage.reload_evolution',
> -        'vshard.lua_gc', 'vshard.rlist'
> +        'vshard.lua_gc', 'vshard.rlist', 'vshard.registry',
>       }
>       for _, module in pairs(vshard_modules) do
>           package.loaded[module] = nil
> @@ -29,6 +29,7 @@ local lcfg = require('vshard.cfg')
>   local lreplicaset = require('vshard.replicaset')
>   local util = require('vshard.util')
>   local lua_gc = require('vshard.lua_gc')
> +local lregistry = require('vshard.registry')
>   local reload_evolution = require('vshard.storage.reload_evolution')
>   local bucket_ref_new
>   
> @@ -2782,6 +2783,8 @@ M.schema_upgrade_handlers = schema_upgrade_handlers
>   M.schema_version_make = schema_version_make
>   M.schema_bootstrap = schema_init_0_1_15_0
>   
> +lregistry.storage = M
> +
>   return {
>       sync = sync,
>       bucket_force_create = bucket_force_create,

  reply	other threads:[~2021-02-24 10:28 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-23  0:15 [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 01/11] error: introduce vshard.error.timeout() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:46     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 10/11] sched: introduce vshard.storage.sched module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 21:50     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-04 21:02   ` Oleg Babin via Tarantool-patches
2021-03-05 22:06     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09  8:03       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 11/11] router: introduce map_callrw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 22:04     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:43       ` Oleg Babin via Tarantool-patches
2021-02-26 23:58         ` Vladislav Shpilevoy via Tarantool-patches
2021-03-01 10:58           ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 02/11] storage: add helper for local functions invocation Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:47     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches [this message]
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 05/11] util: introduce safe fiber_cond_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:48     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 06/11] util: introduce fiber_is_self_canceled() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 07/11] storage: introduce bucket_generation_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 08/11] storage: introduce bucket_are_all_rw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:48     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 09/11] ref: introduce vshard.storage.ref module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 21:49     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-03-04 21:22   ` Oleg Babin via Tarantool-patches
2021-03-05 22:06     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09  8:03       ` Oleg Babin via Tarantool-patches
2021-03-21 18:49   ` Vladislav Shpilevoy via Tarantool-patches
2021-03-12 23:13 ` [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-03-15  7:05   ` Oleg Babin via Tarantool-patches
2021-03-28 18:17 ` Vladislav Shpilevoy via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5d24e9b3-e978-31a4-1ad0-af03e8c42da1@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=olegrok@tarantool.org \
    --cc=v.shpilevoy@tarantool.org \
    --cc=yaroslav.dynnikov@tarantool.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Tarantool development patches archive

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://lists.tarantool.org/tarantool-patches/0 tarantool-patches/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tarantool-patches tarantool-patches/ https://lists.tarantool.org/tarantool-patches \
		tarantool-patches@dev.tarantool.org.
	public-inbox-index tarantool-patches

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git