From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B7FC36EC5D; Tue, 6 Apr 2021 21:06:12 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B7FC36EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617732372; bh=+9O+bFefc03wzQO/7dX3AeRAZc+WqUInTuFYPifkbvk=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Gr5NUx4jK/7FdmPKsKU3oB3k502+S0g94ugAfspLFyaIRDstqUV7NRdbhjlK4mcwn j8AcQPx0Zaa7yXc6oaAgzCKwqkexXlG127pjhQvw2s348dFEqayB3k5qXsG2vhIILY pCaqQnejQgC4hyOCv/OLRAq1meGlLLdNQnDlGOMo= Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id D509B6EC5D for ; Tue, 6 Apr 2021 21:06:11 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D509B6EC5D Received: by smtpng2.m.smailru.net with esmtpa (envelope-from ) id 1lTq5a-0007PK-L8; Tue, 06 Apr 2021 21:06:11 +0300 Date: Tue, 6 Apr 2021 21:05:59 +0300 To: Sergey Ostanevich Message-ID: <20210406180559.GF29703@tarantool.org> References: <6ca8010540a67a92b36327abf44b489ebddc5054.1617641697.git.imun@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.10.1 (2018-07-13) X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD912A3E3D5D4B49FC1CC94992626AE9EFE0A9192A937C653EE00894C459B0CD1B9E3E480FD7002DC13E12FB60C7184594C9C82356F7CFF9EB77583B74AD1C403A3 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78E88BD1CA827EF00C2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE7590E57235B5C00BDEA1F7E6F0F101C674E70A05D1297E1BBC6CDE5D1141D2B1C0A2590170BD2AAF5E15569F8BFF0A673C45F8002CFA446109FA2833FD35BB23D9E625A9149C048EE33AC447995A7AD18C26CFBAC0749D213D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8B2EE5AD8F952D28FBA471835C12D1D977C4224003CC8364762BB6847A3DEAEFB0F43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7CDDB9BF3B882869D543847C11F186F3C59DAA53EE0834AAEE X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A2AD77751E876CB595E8F7B195E1C97831CDCCFD8A4D33B0F260D789705920FE00 X-C1DE0DAB: 0D63561A33F958A57944B37457787D0FFFA01D8CE7AA8A6B80DC901C95CB53C1D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D343C45ADCD169245FA198237A7A40BE50D16B3BA32099A19915B5793E850E1C558BE87CAB40809CB8D1D7E09C32AA3244C832A1436156F6E350886BFEE33F40EEC3C6EB905E3A8056B927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojAX27rRizpLkhax2pRgQJcA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822E2225C050B4588C1C67611D699A3CDCBA7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3] test: fix dynamic modules loading on MacOS X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Sergos, Thanks for your review! On 06.04.21, Sergey Ostanevich wrote: > Hi! > > Thanks for the patch! > > Couple of nits, LGTM. Added your tag: | Reviewed-by: Sergey Ostanevich > > regards, > Sergos > > > > On 5 Apr 2021, at 20:11, Igor Munkin wrote: > > > > This patch resolves the issue with running the tests with auxiliary > > dynamically loaded modules in case of out-of-source build. > > > I wonder how it works for an in-source one… (just a comment) There is no magic: the paths are adjusted within to make in-source testing (e.g. while developing) easier. > > > The first issue relates to the way modules loaded at runtime are built > ^ to be Thanks, fixed. > > on MacOS. Since the auxiliary libraries are built as a dynamically > > loaded modules on MacOS instead of shared libraries as it is done on > > Linux and BSD, another environment variable should be used to guide > > while searching the extension. Hence the values collected in > > This might overlap: ffi should use ‘LUA_CPATH’, while it is the dynamic > loader and dlopen() interface of it that relies on {DY}LD_ set of variables. No, it shouldn't. Despite the fact both native shared libraries and ones implemented via Lua C API are loaded dynamically, they have different loading principles. Hence, since FFI use the native libraries, it ought to consider {DY}LD_ set of variables. LUA_CPATH on its turn provides the paths to the Lua modules loaded via and has another format and ergo requirements for the libraries (e.g. an obligatory external luaopen_ function). > > > scope of macro need to be set to DYLD_LIBRARY_PATH > > variable instead of LD_LIBRARY_PATH on MacOS. > > > > Unfortunately, this rather small change doesn't resolve the problem at > > all and the root cause lies much deeper than it seems at the beginning. > > > > Apple tries their best to "protect their users from malicious software". > > As a result SIP[1] has been designed and released. Now, Apple developers > > are *so protected*, that they can load nothing being not installed in > > the system, since some programs sanitize the environment before they > > start child processes. Specifically, environment variables starting with > > DYLD_ and LD_ are unset for child process started by system programs[2]. > > > > That which does not kill us makes us stronger: fortunately, these > > environment variables are used by FFI machinery to find the proper > > shared library, hence we can still tweak testing environment before > > calling . However, the value can't be passed via the standard > > environment variable, so we prepend TEST_ prefix to its name to get > > around SIP magic tricks. Finally, to set the variable required by FFI > > machinery the introduced routine is used. PROFIT! > > Your move, Cupertino geniuses. > > > > [1]: https://support.apple.com/en-us/HT204899 > > [2]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html > > > > Resolves tarantool/tarantool#5959 > > Follows up tarantool/tarantool#4862 > > > > Co-authored-by: Sergey Ostanevich > > Co-authored-by: Mons Anderson > > Signed-off-by: Igor Munkin > > --- > > test/tarantool-tests/CMakeLists.txt | 39 +++++++++++++++++-- > > .../gh-4427-ffi-sandwich.test.lua | 4 ++ > > .../lj-flush-on-trace.test.lua | 4 ++ > > test/tarantool-tests/utils.lua | 39 ++++++++++++++++--- > > 4 files changed, 77 insertions(+), 9 deletions(-) > > > > diff --git a/test/tarantool-tests/utils.lua b/test/tarantool-tests/utils.lua > > index d2dd71b0..9bdb71ec 100644 > > --- a/test/tarantool-tests/utils.lua > > +++ b/test/tarantool-tests/utils.lua > > @@ -1,7 +1,12 @@ > > local M = {} > > > > +local ffi = require('ffi') > > local tap = require('tap') > > > > +ffi.cdef([[ > > + int setenv(const char *name, const char *value, int overwrite); > > +]]) > > + > > Why? os.setenv() didn’t work for some reason? Comment it and I’m fine. Oh, yeah! Sorry, but I am so glad you have faced this by yourself and produce this precedent! I finally have a good argument for pushing #4577[1]. BTW, now I believe we need the similar ticket for Lua interfaces too. Back to the business: so, you are asking why I didn't use in LuaJIT tests? The answer is quite simple: because there is no such function in Lua! But those guys, who decided to spoil the standard Lua namespace, believed it is a good idea to store their new functions alongside with the ones provided by Lua standard library. This is why I am against using the Lua namespaces for platform-defined functions and we have introduced for the new extensions. So, unfortunately, this can't be done in other way without more efforts. > > tarantool> os.setenv("a","b") > --- > ... > > tarantool> os.getenv("a") > --- > - b > ... > > tarantool> os.execute('echo $a') > b > --- > > > local function luacmd(args) > > -- arg[-1] is guaranteed to be not nil. > > local idx = -2 > > @@ -13,6 +18,12 @@ local function luacmd(args) > > return table.concat(args, ' ', idx + 1, -1) > > end > > > > +local function unshiftenv(variable, value, sep) > > + local envvar = os.getenv(variable) > > + return ('%s="%s%s"'):format(variable, value, > > + envvar and ('%s%s'):format(sep, envvar) or '') > > +end > > + > > function M.selfrun(arg, checks) > > -- If TEST_SELFRUN is set, just execute the test payload below > > -- call, ... > > @@ -24,18 +35,22 @@ function M.selfrun(arg, checks) > > > > test:plan(#checks) > > > > + local libext = package.cpath:match('?.(%a+);') > > local vars = { > > LUABIN = luacmd(arg), > > SCRIPT = arg[0], > > PATH = arg[0]:gsub('%.test%.lua', ''), > > - SUFFIX = package.cpath:match('?.(%a+);'), > > + SUFFIX = libext, > > + ENV = table.concat({ > > + unshiftenv('LUA_PATH', '/?.lua', ';'), > > + unshiftenv('LUA_CPATH', '/?.', ';'), > > + unshiftenv((libext == 'dylib' and 'DYLD' or 'LD') .. '_LIBRARY_PATH', > > + '', ':'), > > + 'TEST_SELFRUN=1', > > + }, ' '), > > } > > > > - local cmd = string.gsub('LUA_PATH="/?.lua;$LUA_PATH" ' .. > > - 'LUA_CPATH="/?.;$LUA_CPATH" ' .. > > - 'LD_LIBRARY_PATH=:$LD_LIBRARY_PATH ' .. > > - 'TEST_SELFRUN=1' .. > > - ' 2>&1