Tarantool development patches archive
 help / color / mirror / Atom feed
From: Vladislav Shpilevoy via Tarantool-patches <tarantool-patches@dev.tarantool.org>
To: tarantool-patches@dev.tarantool.org, olegrok@tarantool.org,
	yaroslav.dynnikov@tarantool.org
Subject: [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution
Date: Tue, 23 Feb 2021 01:15:42 +0100
Message-ID: <92da76963b5bd26fa824ae152c7ed2dda7526320.1614039039.git.v.shpilevoy@tarantool.org> (raw)
In-Reply-To: <cover.1614039039.git.v.shpilevoy@tarantool.org>

Registry is a way to resolve cyclic dependencies which normally
can exist between files of the same module/library.

It is a global table hidden in _G with a long unlikely anywhere
used name.

Files, which want to expose their API to the other files, which in
turn can't require the formers directly, should put their API to
the registry.

The files use the registry to get API of the other files. They
don't require() and use the latter directly.

At runtime, when all require() are done, the registry is full,
and all the files see API of each other.

Such circular dependency will exist between new files implementing
map-reduce engine as a set of relatively independent submodules of
the storage.

In particular there will be storage_ref and storage_sched. Both
require a few functions from the main storage file, and will use
API of each other.

Having the modules accessed via registry adds at lest +1 indexing
operation at runtime when need to get a function from there. But
sometimes it can be cached similar to how bucket count cache works
in the main storage file.

Main purpose is not to increase size of the main storage file
again. It wouldn't fix the circular deps anyway, and would make it
much harder to follow the code.

Part of #147
---
 vshard/CMakeLists.txt   |  3 +-
 vshard/registry.lua     | 67 +++++++++++++++++++++++++++++++++++++++++
 vshard/storage/init.lua |  5 ++-
 3 files changed, 73 insertions(+), 2 deletions(-)
 create mode 100644 vshard/registry.lua

diff --git a/vshard/CMakeLists.txt b/vshard/CMakeLists.txt
index 78a3f07..2a15df5 100644
--- a/vshard/CMakeLists.txt
+++ b/vshard/CMakeLists.txt
@@ -7,4 +7,5 @@ add_subdirectory(router)
 
 # Install module
 install(FILES cfg.lua error.lua consts.lua hash.lua init.lua replicaset.lua
-        util.lua lua_gc.lua rlist.lua heap.lua DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard)
+        util.lua lua_gc.lua rlist.lua heap.lua registry.lua
+        DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard)
diff --git a/vshard/registry.lua b/vshard/registry.lua
new file mode 100644
index 0000000..9583add
--- /dev/null
+++ b/vshard/registry.lua
@@ -0,0 +1,67 @@
+--
+-- Registry is a way to resolve cyclic dependencies which normally can exist
+-- between files of the same module/library.
+--
+-- Files, which want to expose their API to the other files, which in turn can't
+-- require the formers directly, should put their API to the registry.
+--
+-- The files should use the registry to get API of the other files. They don't
+-- require() and use the latter directly if there is a known loop dependency
+-- between them.
+--
+-- At runtime, when all require() are done, the registry is full, and all the
+-- files see API of each other.
+--
+-- Having the modules accessed via the registry adds at lest +1 indexing
+-- operation at runtime when need to get a function from there. But sometimes it
+-- can be cached to reduce the effect in perf-sensitive code. For example, like
+-- this:
+--
+--     local lreg = require('vshard.registry')
+--
+--     local storage_func
+--
+--     local function storage_func_no_cache(...)
+--         storage_func = lreg.storage.func
+--         return storage_func(...)
+--     end
+--
+--     storage_func = storage_func_no_cache
+--
+-- The code will always call storage_func(), but will load it from the registry
+-- only on first invocation.
+--
+-- However in case reload is important, it is not possible - the original
+-- function object in the registry may change. In such situation still makes
+-- sense to cache at least 'lreg.storage' to save 1 indexing operation.
+--
+--     local lreg = require('vshard.registry')
+--
+--     local lstorage
+--
+--     local function storage_func_cache(...)
+--         return lstorage.storage_func(...)
+--     end
+--
+--     local function storage_func_no_cache(...)
+--         lstorage = lref.storage
+--         storage_func = storage_func_cache
+--         return lstorage.storage_func(...)
+--     end
+--
+--     storage_func = storage_func_no_cache
+--
+-- A harder way would be to use the first approach + add triggers on reload of
+-- the cached module to update the cached function refs. If the code is
+-- extremely perf-critical (which should not be Lua then).
+--
+
+local MODULE_INTERNALS = '__module_vshard_registry'
+
+local M = rawget(_G, MODULE_INTERNALS)
+if not M then
+    M = {}
+    rawset(_G, MODULE_INTERNALS, M)
+end
+
+return M
diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua
index 9b74bcb..b47665b 100644
--- a/vshard/storage/init.lua
+++ b/vshard/storage/init.lua
@@ -16,7 +16,7 @@ if rawget(_G, MODULE_INTERNALS) then
         'vshard.consts', 'vshard.error', 'vshard.cfg',
         'vshard.replicaset', 'vshard.util',
         'vshard.storage.reload_evolution',
-        'vshard.lua_gc', 'vshard.rlist'
+        'vshard.lua_gc', 'vshard.rlist', 'vshard.registry',
     }
     for _, module in pairs(vshard_modules) do
         package.loaded[module] = nil
@@ -29,6 +29,7 @@ local lcfg = require('vshard.cfg')
 local lreplicaset = require('vshard.replicaset')
 local util = require('vshard.util')
 local lua_gc = require('vshard.lua_gc')
+local lregistry = require('vshard.registry')
 local reload_evolution = require('vshard.storage.reload_evolution')
 local bucket_ref_new
 
@@ -2782,6 +2783,8 @@ M.schema_upgrade_handlers = schema_upgrade_handlers
 M.schema_version_make = schema_version_make
 M.schema_bootstrap = schema_init_0_1_15_0
 
+lregistry.storage = M
+
 return {
     sync = sync,
     bucket_force_create = bucket_force_create,
-- 
2.24.3 (Apple Git-128)


  parent reply	other threads:[~2021-02-23  0:19 UTC|newest]

Thread overview: 47+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-23  0:15 [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 01/11] error: introduce vshard.error.timeout() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:46     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 10/11] sched: introduce vshard.storage.sched module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 21:50     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-04 21:02   ` Oleg Babin via Tarantool-patches
2021-03-05 22:06     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09  8:03       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 11/11] router: introduce map_callrw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 22:04     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:43       ` Oleg Babin via Tarantool-patches
2021-02-26 23:58         ` Vladislav Shpilevoy via Tarantool-patches
2021-03-01 10:58           ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 02/11] storage: add helper for local functions invocation Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 03/11] storage: cache bucket count Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:47     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` Vladislav Shpilevoy via Tarantool-patches [this message]
2021-02-24 10:27   ` [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 05/11] util: introduce safe fiber_cond_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:48     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 06/11] util: introduce fiber_is_self_canceled() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 07/11] storage: introduce bucket_generation_wait() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 08/11] storage: introduce bucket_are_all_rw() Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:27   ` Oleg Babin via Tarantool-patches
2021-02-24 21:48     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-23  0:15 ` [Tarantool-patches] [PATCH vshard 09/11] ref: introduce vshard.storage.ref module Vladislav Shpilevoy via Tarantool-patches
2021-02-24 10:28   ` Oleg Babin via Tarantool-patches
2021-02-24 21:49     ` Vladislav Shpilevoy via Tarantool-patches
2021-02-25 12:42       ` Oleg Babin via Tarantool-patches
2021-03-04 21:22   ` Oleg Babin via Tarantool-patches
2021-03-05 22:06     ` Vladislav Shpilevoy via Tarantool-patches
2021-03-09  8:03       ` Oleg Babin via Tarantool-patches
2021-03-21 18:49   ` Vladislav Shpilevoy via Tarantool-patches
2021-03-12 23:13 ` [Tarantool-patches] [PATCH vshard 00/11] VShard Map-Reduce, part 2: Ref, Sched, Map Vladislav Shpilevoy via Tarantool-patches
2021-03-15  7:05   ` Oleg Babin via Tarantool-patches
2021-03-28 18:17 ` Vladislav Shpilevoy via Tarantool-patches

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=92da76963b5bd26fa824ae152c7ed2dda7526320.1614039039.git.v.shpilevoy@tarantool.org \
    --to=tarantool-patches@dev.tarantool.org \
    --cc=olegrok@tarantool.org \
    --cc=v.shpilevoy@tarantool.org \
    --cc=yaroslav.dynnikov@tarantool.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Tarantool development patches archive

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://lists.tarantool.org/tarantool-patches/0 tarantool-patches/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 tarantool-patches tarantool-patches/ https://lists.tarantool.org/tarantool-patches \
		tarantool-patches@dev.tarantool.org.
	public-inbox-index tarantool-patches

Example config snippet for mirrors.


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git