From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 3787070202; Wed, 24 Feb 2021 13:28:50 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3787070202 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1614162530; bh=HM51JHZoKNfWbjSBjeRQUO6RBqcwWOHmQuYVUszeZBA=; h=To:References:Date:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=BSh9BaU1a0rO8HqSlIUsnVAAf6gO+pN7ug8H9NUCjdxXQI6mvuveRjxxfD36MIfjk z0hGdpfBPZ30gYmgSmd689giSMrygDHg1tkWu5AZEnKRi4DhXZsvunXEW4OtO3SMxx 8nr5gdTMQbnrJd85qiS8Inbqk51YfSl0707V2Rsk= Received: from smtp53.i.mail.ru (smtp53.i.mail.ru [94.100.177.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id D8BDF71826 for ; Wed, 24 Feb 2021 13:27:34 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D8BDF71826 Received: by smtp53.i.mail.ru with esmtpa (envelope-from ) id 1lErOI-0007Lc-0X; Wed, 24 Feb 2021 13:27:34 +0300 To: Vladislav Shpilevoy , tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@tarantool.org References: <92da76963b5bd26fa824ae152c7ed2dda7526320.1614039039.git.v.shpilevoy@tarantool.org> Message-ID: <5d24e9b3-e978-31a4-1ad0-af03e8c42da1@tarantool.org> Date: Wed, 24 Feb 2021 13:27:32 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <92da76963b5bd26fa824ae152c7ed2dda7526320.1614039039.git.v.shpilevoy@tarantool.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB X-7564579A: 78E4E2B564C1792B X-77F55803: 4F1203BC0FB41BD975C3EC174F56692242B8E6687D03D8974314021AB65B8FCC182A05F5380850404AF0F004C361435CED0B6A0951762E99AFFB3661D195F23F76B6D1F3CF7C19E0 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7AD2F2D6F6013FF7FC2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE721B3E54BB37EA0B4EA1F7E6F0F101C674E70A05D1297E1BBC6CDE5D1141D2B1CC8CD956854CCCECADD8E6056B2AB09DB64DB118E79D6B7AC9FA2833FD35BB23D9E625A9149C048EE33AC447995A7AD18C26CFBAC0749D213D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BECADA55FE5B58BB7A471835C12D1D977C4224003CC836476EB9C4185024447017B076A6E789B0E975F5C1EE8F4F765FC02EB29AC746B6C8A3AA81AA40904B5D9CF19DD082D7633A078D18283394535A93AA81AA40904B5D98AA50765F7900637B244CABF8A135FC8D81D268191BDAD3D698AB9A7B718F8C442539A7722CA490C13377AFFFEAFD26923F8577A6DFFEA7CB59C7783CC88FA9693EC92FD9297F6715571747095F342E857739F23D657EF2BD5E8D9A59859A8B655EC1579764E9EAF089D37D7C0E48F6C5571747095F342E857739F23D657EF2B6825BDBE14D8E7028C9DFF55498CEFB0BD9CCCA9EDD067B1EDA766A37F9254B7 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D71E553C27ECEF58E355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A599E34DB7CA8029738A1BD76524DEA8028B6C4F9F6697987BD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75B7BFB303F1C7DB4D8E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D4140D813EC137C60508EC8F1EFF302AC74586946A05BBC5D22529954FCD85B95C17BC9DFB8F19231D7E09C32AA3244C26D8DBFCF096FA93A5DE206E95EB11A030452B15D76AEC14FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojyK6JYJ15DtLqIfYUhV2gug== X-Mailru-Sender: 583F1D7ACE8F49BD9317CE1922F30C7EA607BFF12E4C2A7DA8A68166142C21425FC9CED9F47C053A23E75C7104EB1B885DEE61814008E47C7013064206BFB89F93956FB04BA385BE9437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH vshard 04/11] registry: module for circular deps resolution X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Oleg Babin via Tarantool-patches Reply-To: Oleg Babin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for your patch. LGTM. On 23.02.2021 03:15, Vladislav Shpilevoy wrote: > Registry is a way to resolve cyclic dependencies which normally > can exist between files of the same module/library. > > It is a global table hidden in _G with a long unlikely anywhere > used name. > > Files, which want to expose their API to the other files, which in > turn can't require the formers directly, should put their API to > the registry. > > The files use the registry to get API of the other files. They > don't require() and use the latter directly. > > At runtime, when all require() are done, the registry is full, > and all the files see API of each other. > > Such circular dependency will exist between new files implementing > map-reduce engine as a set of relatively independent submodules of > the storage. > > In particular there will be storage_ref and storage_sched. Both > require a few functions from the main storage file, and will use > API of each other. > > Having the modules accessed via registry adds at lest +1 indexing > operation at runtime when need to get a function from there. But > sometimes it can be cached similar to how bucket count cache works > in the main storage file. > > Main purpose is not to increase size of the main storage file > again. It wouldn't fix the circular deps anyway, and would make it > much harder to follow the code. > > Part of #147 > --- > vshard/CMakeLists.txt | 3 +- > vshard/registry.lua | 67 +++++++++++++++++++++++++++++++++++++++++ > vshard/storage/init.lua | 5 ++- > 3 files changed, 73 insertions(+), 2 deletions(-) > create mode 100644 vshard/registry.lua > > diff --git a/vshard/CMakeLists.txt b/vshard/CMakeLists.txt > index 78a3f07..2a15df5 100644 > --- a/vshard/CMakeLists.txt > +++ b/vshard/CMakeLists.txt > @@ -7,4 +7,5 @@ add_subdirectory(router) > > # Install module > install(FILES cfg.lua error.lua consts.lua hash.lua init.lua replicaset.lua > - util.lua lua_gc.lua rlist.lua heap.lua DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard) > + util.lua lua_gc.lua rlist.lua heap.lua registry.lua > + DESTINATION ${TARANTOOL_INSTALL_LUADIR}/vshard) > diff --git a/vshard/registry.lua b/vshard/registry.lua > new file mode 100644 > index 0000000..9583add > --- /dev/null > +++ b/vshard/registry.lua > @@ -0,0 +1,67 @@ > +-- > +-- Registry is a way to resolve cyclic dependencies which normally can exist > +-- between files of the same module/library. > +-- > +-- Files, which want to expose their API to the other files, which in turn can't > +-- require the formers directly, should put their API to the registry. > +-- > +-- The files should use the registry to get API of the other files. They don't > +-- require() and use the latter directly if there is a known loop dependency > +-- between them. > +-- > +-- At runtime, when all require() are done, the registry is full, and all the > +-- files see API of each other. > +-- > +-- Having the modules accessed via the registry adds at lest +1 indexing > +-- operation at runtime when need to get a function from there. But sometimes it > +-- can be cached to reduce the effect in perf-sensitive code. For example, like > +-- this: > +-- > +-- local lreg = require('vshard.registry') > +-- > +-- local storage_func > +-- > +-- local function storage_func_no_cache(...) > +-- storage_func = lreg.storage.func > +-- return storage_func(...) > +-- end > +-- > +-- storage_func = storage_func_no_cache > +-- > +-- The code will always call storage_func(), but will load it from the registry > +-- only on first invocation. > +-- > +-- However in case reload is important, it is not possible - the original > +-- function object in the registry may change. In such situation still makes > +-- sense to cache at least 'lreg.storage' to save 1 indexing operation. > +-- > +-- local lreg = require('vshard.registry') > +-- > +-- local lstorage > +-- > +-- local function storage_func_cache(...) > +-- return lstorage.storage_func(...) > +-- end > +-- > +-- local function storage_func_no_cache(...) > +-- lstorage = lref.storage > +-- storage_func = storage_func_cache > +-- return lstorage.storage_func(...) > +-- end > +-- > +-- storage_func = storage_func_no_cache > +-- > +-- A harder way would be to use the first approach + add triggers on reload of > +-- the cached module to update the cached function refs. If the code is > +-- extremely perf-critical (which should not be Lua then). > +-- > + > +local MODULE_INTERNALS = '__module_vshard_registry' > + > +local M = rawget(_G, MODULE_INTERNALS) > +if not M then > + M = {} > + rawset(_G, MODULE_INTERNALS, M) > +end > + > +return M > diff --git a/vshard/storage/init.lua b/vshard/storage/init.lua > index 9b74bcb..b47665b 100644 > --- a/vshard/storage/init.lua > +++ b/vshard/storage/init.lua > @@ -16,7 +16,7 @@ if rawget(_G, MODULE_INTERNALS) then > 'vshard.consts', 'vshard.error', 'vshard.cfg', > 'vshard.replicaset', 'vshard.util', > 'vshard.storage.reload_evolution', > - 'vshard.lua_gc', 'vshard.rlist' > + 'vshard.lua_gc', 'vshard.rlist', 'vshard.registry', > } > for _, module in pairs(vshard_modules) do > package.loaded[module] = nil > @@ -29,6 +29,7 @@ local lcfg = require('vshard.cfg') > local lreplicaset = require('vshard.replicaset') > local util = require('vshard.util') > local lua_gc = require('vshard.lua_gc') > +local lregistry = require('vshard.registry') > local reload_evolution = require('vshard.storage.reload_evolution') > local bucket_ref_new > > @@ -2782,6 +2783,8 @@ M.schema_upgrade_handlers = schema_upgrade_handlers > M.schema_version_make = schema_version_make > M.schema_bootstrap = schema_init_0_1_15_0 > > +lregistry.storage = M > + > return { > sync = sync, > bucket_force_create = bucket_force_create,