From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 50C5D6465D; Sun, 7 Feb 2021 21:13:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 50C5D6465D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1612721627; bh=ypGr2wBQetSnWewfPE5ztzD9WDNTUYSLoOREEihDNpQ=; h=To:Cc:References:Date:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=pVlQxvHxOd3kBYOCv9hRWSq/SQHn5JQpZdxWo24YnTvhcXIIaDbuEIoTzb3F/q+lV Mbv8CYL9BwYxWZkY/ksJiJzorncqr1H7HbdAFX8aYcRcxaiYtZq2QLkpryngWDCZDK +ql2zojMC+MxuKFbiV628IDxNJUh8Yc9k5lURH0M= Received: from smtp30.i.mail.ru (smtp30.i.mail.ru [94.100.177.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id AD01364661 for ; Sun, 7 Feb 2021 20:20:48 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org AD01364661 Received: by smtp30.i.mail.ru with esmtpa (envelope-from ) id 1l8njq-0007z0-7p; Sun, 07 Feb 2021 20:20:46 +0300 To: Cyrill Gorcunov , tml Cc: Mons Anderson References: <20210205185436.638894-1-gorcunov@gmail.com> <20210205185436.638894-11-gorcunov@gmail.com> Message-ID: <91e9dbc5-fffd-a527-5385-242a0130d994@tarantool.org> Date: Sun, 7 Feb 2021 18:20:45 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.7.1 MIME-Version: 1.0 In-Reply-To: <20210205185436.638894-11-gorcunov@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-7564579A: 78E4E2B564C1792B X-77F55803: 4F1203BC0FB41BD953AC099BC0052A9C55152E5B36F70094641E3870F0CF5304182A05F5380850402C83BDD7926F03EC71A7857F81ED7B83A9F937FF7F3803F830A1381B58F2E3C4 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE70A10A23A3B64B805EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637C8BD49B17AF1FC7C8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FCA1425B1BFD02B95D4714691133F12F794AF27324B63865F3389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C07E7E81EEA8A9722B8941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6D52CD31C43BF465FCC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB8D32BA5DBAC0009BE9E8FC8737B5C224945744BFAA53FCC0876E601842F6C81A12EF20D2F80756B5F7E9C4E3C761E06A776E601842F6C81A127C277FBC8AE2E8BD7978A457ACEBD0A3AA81AA40904B5D9DBF02ECDB25306B2B25CBF701D1BE8734AD6D5ED66289B5278DA827A17800CE745781761687023AD67F23339F89546C5A8DF7F3B2552694A6FED454B719173D6725E5C173C3A84C307FFBFDCC3B3F35035872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C934F12F0C005D1A85E5BFE6E7EFDEDCD789D4C264860C145E X-C1DE0DAB: 0D63561A33F958A50D172FFF17C648A9A72FCEAB2D4962C0762373E60FCDA8BDD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA75448CF9D3A7B2C848410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3474B2583E51315984CAB30E85A8F058BD4AEF12E69F8414383B8D1CF6E048F08D36F576CB1F72C2BA1D7E09C32AA3244CADD70B76743D66E70546BCD60B1FD0F18894E9C85370243EFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojMxr0zTsJFiRuu2I0V5S2+g== X-Mailru-Sender: 504CC1E875BF3E7D9BC0E5172ADA3110E20A360582E3622A5E153E0927D8FF243F87A915DA423B2A07784C02288277CA03E0582D3806FB6A5317862B1921BA260ED6CFD6382C13A6112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v15 10/11] box/cmod: implement cmod Lua module X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Vladislav Shpilevoy via Tarantool-patches Reply-To: Vladislav Shpilevoy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for the patch! See 10 comments below. Overall no major issues, much better now. > module:load(name) -> obj | error` > --------------------------------- > > Loads a new function with name `name` from the previously > loaded `module` and return a callable object instance > associated with the function. On failure an error is thrown. > > Possible errors: > - IllegalParams: function name is either not supplied > or not a string. > - IllegalParams: attempt to load a function but module > has been unloaded already. > - ClientError: no such function in the module. > - ClientError: module has been updated on disk and not > reloaded. 1. We decided not to throw an errors in this case. > - OutOfMemory: unable to allocate a function. > > Executing a loaded function > =========================== > > Once function is loaded it can be executed as an ordinary Lua call. > Lets consider the following example. We have a `C` function which > takes two numbers and returns their sum. > > ``` C > int > cfunc_sum(box_function_ctx_t *ctx, const char *args, const char *args_end) > { > uint32_t arg_count = mp_decode_array(&args); > if (arg_count != 2) { > return box_error_set(__FILE__, __LINE__, ER_PROC_C, "%s", > "invalid argument count"); > } > uint64_t a = mp_decode_uint(&args); > uint64_t b = mp_decode_uint(&args); > > char res[16]; > char *end = mp_encode_uint(res, a + b); > box_return_mp(ctx, res, end); > return 0; > } > ``` > > The name of the function is `cfunc_sum` and the function is built into > `cfunc.so` shared library. > > First we should load it as > > ``` Lua > m = require('cmod').load('cfunc') > cfunc_sum = m:load('cfunc_sum') > ``` > > Once successfully loaded we can execute it. Note that unlike regular > Lua functions the context of `C` functions is different. They never > thrown an exception but return `true|nil, res` form where first value > set to `nil` in case of error condition and `res` carries an error > description. 2. In the previous review I reminded you that now we throw errors again, according to the updated Lua guidelines. > Lets call the `cfunc_sum` with wrong number of arguments > > ``` Lua > local ok, res = cfunc_sum() > if not ok then > print(res) > end > ``` > > We will the `"invalid argument count"` message in output. 3. Will what? "See"? > The error message has been set by the `box_error_set` in `C` > code above. > > Module and function caches > ========================== > > Loading a module is relatively slow procedure because operating > system needs to read the library, resolve its symbols and etc. > Thus to speedup this procedure if the module is loaded for a first > time we put it into an internal cache. If module is sitting in > the cache already and new request to load comes in we simply > reuse a previous copy immediately. Same applies to functions: > while symbol lookup is a way faster than loading module from > disk it is not completely cheap, thus we cache functions as > well. Functions entries in cache are identified by a module > path and function name. > > Still the following situation is possible: the module is loaded but > user does recompile it and overwrite on a storage device. Thus > cache content no longer matches the shared library on the disk. > > To handle this situation we use that named cache invalidation procedure: > on every attempt to load the same module again we test low level file > attributes (such as storage device number, inode, size and modification > time) and if they are differ from ones kept by the cache then old module > marked as orphan, new instance is loaded and become a valid cache > entry. Module state could be seen in module variable output > > ```Lua > m = require('cmod').load('cfunc') > m > ``` > which will output > ``` > tarantool> m > --- > - path: cfunc > state: cached > ``` 4. Lets not expose any states or other attributes until it is requested explicitly by someone. > The `state` is either `cached` if module is present in cache > and valid or `orphan` if entry in cache has been updated. > > diff --git a/src/box/lua/cmod.c b/src/box/lua/cmod.c > new file mode 100644 > index 000000000..60fd2e812 > --- /dev/null > +++ b/src/box/lua/cmod.c > + > +/** > + * Unload a symbol and free a function instance. > + */ > +static void > +func_delete(struct cmod_func *cf) 5. Methods of a struct should have the struct name as a prefix. So it should be cmod_func_delete(). The same for other cmod_func methods. For Lua the prefix 'lcmod_func' is good. > +{ > + assert(cf->refs == 0); > + module_sym_unload(&cf->mod_sym); > + TRASH(cf); > + free(cf); > +}> + > +/** > + * Load a new function. > + * > + * This function takes a function name from the caller > + * stack @a L and creates a new function object. If > + * the function is already loaded we simply return > + * a reference to existing one. > + * > + * Possible errors: > + * > + * - IllegalParams: function name is either not supplied > + * or not a string. > + * - IllegalParams: function references limit exceeded. > + * - OutOfMemory: unable to allocate a function. > + * - ClientError: no such function in the module. > + * - ClientError: module has been updated on disk and not > + * yet unloaded and loaded back. 6. It was decided not to throw an error in this case. > + * > + * @returns function object on success or throwns an error. > + */ > +static int > +lcmod_func_load(struct lua_State *L) > diff --git a/src/box/lua/cmod.h b/src/box/lua/cmod.h > new file mode 100644 > index 000000000..f0ea2d34d > --- /dev/null > +++ b/src/box/lua/cmod.h > @@ -0,0 +1,24 @@ > +/* > + * SPDX-License-Identifier: BSD-2-Clause > + * > + * Copyright 2010-2021, Tarantool AUTHORS, please see AUTHORS file. > + */ > + > +#pragma once > + > +#if defined(__cplusplus) > +extern "C" { > +#endif /* defined(__cplusplus) */ > + > +struct lua_State; > + > +/** > + * Initialize cmod Lua module. > + * > + * @param L Lua state where to register the cmod module. > + */ > +void > +box_lua_cmod_init(struct lua_State *L); 7. Please, add an empty line after the declaration. > +#if defined(__cplusplus) > +} > +#endif /* defined(__plusplus) */ > diff --git a/src/box/module_cache.c b/src/box/module_cache.c > index 22b906fd7..e96cbd1f8 100644 > --- a/src/box/module_cache.c > +++ b/src/box/module_cache.c 8. Most of changes in this file should not happen, as I described in the comments to the previous patches. You will probably need to add a few more commits for this. One to add a second hash in box/func; next to make module loading always check stat in module_cache. > @@ -514,6 +553,74 @@ module_sym_call(struct module_sym *mod_sym, struct port *args, > return rc; > } > > +struct module * > +module_load(const char *package, const char *package_end) > +{ > + char path[PATH_MAX]; > + if (module_find(package, package_end, path, sizeof(path)) != 0) > + return NULL; > + > + struct module *cached, *module; > + struct mh_strnptr_t *h = hash_tbl(false); > + cached = module_cache_find(h, package, package_end); > + if (cached == NULL) { > + module = module_new(path, h, package, package_end); > + if (module == NULL) > + return NULL; > + if (module_cache_add(module) != 0) { > + module_unref(module); > + return NULL; > + } > + return module; > + } > + > + struct stat st; > + if (stat(path, &st) != 0) { > + diag_set(SystemError, "module: stat() module %s", path); > + return NULL; > + } > + > + /* > + * When module comes from cache make sure that > + * it is not changed on the storage device. The > + * test below still can miss update if cpu data > + * been manually moved backward and device/inode > + * persisted but this is a really rare situation. > + * > + * If update is needed one can simply "touch file.so" > + * to invalidate the cache entry. > + */ > + if (cached->st.st_dev == st.st_dev && > + cached->st.st_ino == st.st_ino && > + cached->st.st_size == st.st_size && > + memcmp(&cached->st.st_mtim, &st.st_mtim, > + sizeof(st.st_mtim)) == 0) { > + module_ref(cached); > + return cached; 9. Build fails on Mac: /Users/gerold/Work/Repositories/tarantool/src/box/module_cache.c:596:25: error: no member named 'st_mtim' in 'struct stat' memcmp(&cached->st.st_mtim, &st.st_mtim, ~~~~~~~~~~ ^ /Users/gerold/Work/Repositories/tarantool/src/box/module_cache.c:596:38: error: no member named 'st_mtim' in 'struct stat' memcmp(&cached->st.st_mtim, &st.st_mtim, ~~ ^ /Users/gerold/Work/Repositories/tarantool/src/box/module_cache.c:597:16: error: no member named 'st_mtim' in 'struct stat' sizeof(st.st_mtim)) == 0) { ~~ ^ 3 errors generated. This is because here last modification time is stored as a timespec struct, and has name 'st_mtimespec'. > + } > + > + /* > + * Load a new module, update the cache > + * and orphan an old module instance. > + */ > + module = module_new(path, h, package, package_end); > + if (module == NULL) > + return NULL; > + if (module_cache_update(module) != 0) { > + module_unref(module); > + return NULL; > + } > + > + module_set_orphan(cached); > + return module; > +} > + > +void > +module_unload(struct module *module) > +{ > + module_unref(module); > +} > + > int > module_reload(const char *package, const char *package_end) > { > diff --git a/src/box/module_cache.h b/src/box/module_cache.h > index 875f2eb3c..17a2e27bb 100644 > --- a/src/box/module_cache.h > +++ b/src/box/module_cache.h > @@ -48,6 +52,10 @@ struct module { > * Count of active references to the module. > */ > int64_t refs; > + /** > + * Storage stat for identity check. > + */ > + struct stat st; 10. The structure if huge. And you use just a few fields. Better store only the need fields. > /** > * Module's package name. > */