From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id A61A56EC5D; Tue, 6 Apr 2021 20:02:59 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A61A56EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617728579; bh=tGDOWef9BYY2pygyVXeM6RMOmY668eUyUt718Wmp21s=; h=In-Reply-To:Date:References:To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=NISOa7uTwpUEZ+unAkr0e4KVLclN6mdkIyQTsC3M4hBiY7yHMRscqeDJYkv49Ijhp MB4GF6VIlPVWJfzS3bdHHxJMC7WThoiF8wOKSyjC2+2FEzaWMwYw1mwCv3vYKEfrCT pdeUelPbYG9GCY1YvkCS3sXXRccYW5sY+x8E4g9o= Received: from smtp34.i.mail.ru (smtp34.i.mail.ru [94.100.177.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id BA0456EC5D for ; Tue, 6 Apr 2021 20:02:58 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BA0456EC5D Received: by smtp34.i.mail.ru with esmtpa (envelope-from ) id 1lTp6P-0005EJ-NR; Tue, 06 Apr 2021 20:02:58 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) In-Reply-To: <6ca8010540a67a92b36327abf44b489ebddc5054.1617641697.git.imun@tarantool.org> Date: Tue, 6 Apr 2021 20:02:56 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <6ca8010540a67a92b36327abf44b489ebddc5054.1617641697.git.imun@tarantool.org> To: Igor Munkin X-Mailer: Apple Mail (2.3654.60.0.2.21) X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD912A3E3D5D4B49FC130D613A19A4D1F6017F5C004462A7E0D00894C459B0CD1B9CB9316AC969E177CE73D57D8408982931102AB7BB65CCEBCFB1C19767FBE8657 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE76DD23B0452F84E3CEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006375E280A1EC162AD7D8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FC864C7434B91C4612D0FCD67F7CDE65D75566BA99EB15D9D5389733CBF5DBD5E913377AFFFEAFD269A417C69337E82CC2CC7F00164DA146DAFE8445B8C89999729449624AB7ADAF37F6B57BC7E64490611E7FA7ABCAF51C92A417C69337E82CC2CC7F00164DA146DA6F5DAA56C3B73B237318B6A418E8EAB8D32BA5DBAC0009BE9E8FC8737B5C2249B8BF30860782A87376E601842F6C81A12EF20D2F80756B5F7E9C4E3C761E06A776E601842F6C81A127C277FBC8AE2E8B59892097EE87B7583AA81AA40904B5D9DBF02ECDB25306B2201CA6A4E26CD07C3BBE47FD9DD3FB595F5C1EE8F4F765FC72CEEB2601E22B093A03B725D353964B0B7D0EA88DDEDAC722CA9DD8327EE4930A3850AC1BE2E73557739F23D657EF2BB5C8C57E37DE458BEDA766A37F9254B7 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A24A6D60772A99906F8E1CD14B953EB46D771CF3F84EA6E40B355D89D7DBCDD132 X-C1DE0DAB: 0D63561A33F958A57CE355A0320EF87C233FEB19C61170DF377DD2693D21134ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D348BE83DFD8AFB1CAC69BBFBD97BAC9C9E96EB6ABA0BD824D67E7A12124B63BA2E2AED97B7437DAF241D7E09C32AA3244C5BFFD40E42BD54F2380FC4DA92EE4E7AA95CA90A1D8AC565FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojAX27rRizpLkavbENUmEJfQ== X-Mailru-Sender: 455D65AE3A139168626D8C76E86D3AC0CB9316AC969E177CE73D57D840898293DDAC019A3742A8B676D79013C85012CDC77752E0C033A69E4BBE7EBD99111A499D0AB74157175C036C18EFA0BB12DBB0 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3] test: fix dynamic modules loading on MacOS X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Ostanevich via Tarantool-patches Reply-To: Sergey Ostanevich Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the patch! Couple of nits, LGTM. regards, Sergos > On 5 Apr 2021, at 20:11, Igor Munkin wrote: >=20 > This patch resolves the issue with running the tests with auxiliary > dynamically loaded modules in case of out-of-source build. >=20 I wonder how it works for an in-source one=E2=80=A6 (just a comment) > The first issue relates to the way modules loaded at runtime are built ^ to be > on MacOS. Since the auxiliary libraries are built as a dynamically > loaded modules on MacOS instead of shared libraries as it is done on > Linux and BSD, another environment variable should be used to guide > while searching the extension. Hence the values collected = in This might overlap: ffi should use =E2=80=98LUA_CPATH=E2=80=99, while it = is the dynamic=20 loader and dlopen() interface of it that relies on {DY}LD_ set of = variables.=20 > scope of macro need to be set to DYLD_LIBRARY_PATH > variable instead of LD_LIBRARY_PATH on MacOS. >=20 > Unfortunately, this rather small change doesn't resolve the problem at > all and the root cause lies much deeper than it seems at the = beginning. >=20 > Apple tries their best to "protect their users from malicious = software". > As a result SIP[1] has been designed and released. Now, Apple = developers > are *so protected*, that they can load nothing being not installed in > the system, since some programs sanitize the environment before they > start child processes. Specifically, environment variables starting = with > DYLD_ and LD_ are unset for child process started by system = programs[2]. >=20 > That which does not kill us makes us stronger: fortunately, these > environment variables are used by FFI machinery to find the proper > shared library, hence we can still tweak testing environment before > calling . However, the value can't be passed via the = standard > environment variable, so we prepend TEST_ prefix to its name to get > around SIP magic tricks. Finally, to set the variable required by FFI > machinery the introduced routine is used. PROFIT! > Your move, Cupertino geniuses. >=20 > [1]: https://support.apple.com/en-us/HT204899 > [2]: = https://developer.apple.com/library/archive/documentation/Security/Concept= ual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtection= s.html >=20 > Resolves tarantool/tarantool#5959 > Follows up tarantool/tarantool#4862 >=20 > Co-authored-by: Sergey Ostanevich > Co-authored-by: Mons Anderson > Signed-off-by: Igor Munkin > --- > test/tarantool-tests/CMakeLists.txt | 39 +++++++++++++++++-- > .../gh-4427-ffi-sandwich.test.lua | 4 ++ > .../lj-flush-on-trace.test.lua | 4 ++ > test/tarantool-tests/utils.lua | 39 ++++++++++++++++--- > 4 files changed, 77 insertions(+), 9 deletions(-) >=20 > diff --git a/test/tarantool-tests/CMakeLists.txt = b/test/tarantool-tests/CMakeLists.txt > index 3e36ff86..c793ad60 100644 > --- a/test/tarantool-tests/CMakeLists.txt > +++ b/test/tarantool-tests/CMakeLists.txt > @@ -69,11 +69,46 @@ set(LUA_PATH > ) > set(LUA_TEST_SUFFIX .test.lua) > set(LUA_TEST_FLAGS --failures --shuffle) > +set(LUA_TEST_ENV > + "LUA_PATH=3D\"${LUA_PATH}\;\;\"" > + "LUA_CPATH=3D\"${LUA_CPATH}\;\;\"" > +) >=20 > if(CMAKE_VERBOSE_MAKEFILE) > list(APPEND LUA_TEST_FLAGS --verbose) > endif() >=20 > +# XXX: Since the auxiliary libraries are built as a dynamically > +# loaded modules on MacOS instead of shared libraries as it is > +# done on Linux and BSD, another environment variable should be > +# used to guide while searching the extension. > +if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") > + # XXX: Apple tries their best to "protect their users from > + # malware". As a result SIP (see the link[1] below) has been > + # designed and released. Now, Apple developers are so protected, > + # that they can load nothing being not installed in the system, > + # since some programs sanitize the environment before they start > + # child processes. Specifically, environment variables starting > + # with DYLD_ and LD_ are unset for child process started by > + # system programs (like /usr/bin/env used for preparing testing > + # environment). For more info, see the docs[2] below. > + # > + # That which does not kill us makes us stronger: fortunately, > + # these environment variables are used by FFI machinery to find > + # the proper shared library, hence we can still tweak testing > + # environment before calling . However, the value > + # can't be passed via the standard environment variable, so we > + # prepend TEST_ prefix to its name to get around SIP magic > + # tricks. Finally, to set the variable required by FFI machinery > + # the introduced routine is used. > + # > + # [1]: https://support.apple.com/en-us/HT204899 > + # [2]: = https://developer.apple.com/library/archive/documentation/Security/Concept= ual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtection= s.html > + list(APPEND LUA_TEST_ENV = TEST_DYLD_LIBRARY_PATH=3D"${LD_LIBRARY_PATH}") > +else() > + list(APPEND LUA_TEST_ENV LD_LIBRARY_PATH=3D"${LD_LIBRARY_PATH}") > +endif() > + > # LUA_CPATH and LD_LIBRARY_PATH variables and also TESTLIBS list > # with dependecies are set in scope of BuildTestLib macro. > add_custom_target(tarantool-tests > @@ -83,9 +118,7 @@ add_custom_command(TARGET tarantool-tests > COMMENT "Running Tarantool tests" > COMMAND > env > - LUA_PATH=3D"${LUA_PATH}\;\;" > - LUA_CPATH=3D"${LUA_CPATH}\;\;" > - LD_LIBRARY_PATH=3D"${LD_LIBRARY_PATH}" > + ${LUA_TEST_ENV} > ${PROVE} ${CMAKE_CURRENT_SOURCE_DIR} > --exec '${LUAJIT_TEST_COMMAND}' > --ext ${LUA_TEST_SUFFIX} > diff --git a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua = b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > index 64df5dbd..651dc3f4 100644 > --- a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > +++ b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > @@ -3,6 +3,10 @@ local utils =3D require('utils') > -- Disabled on *BSD due to #4819. > utils.skipcond(jit.os =3D=3D 'BSD', 'Disabled due to #4819') >=20 > +-- XXX: Tweak the process environment to get around SIP. > +-- See the comment in suite CMakeLists.txt for more info. > +utils.tweakenv(jit.os =3D=3D 'OSX', 'DYLD_LIBRARY_PATH') > + > utils.selfrun(arg, { > { > arg =3D { > diff --git a/test/tarantool-tests/lj-flush-on-trace.test.lua = b/test/tarantool-tests/lj-flush-on-trace.test.lua > index edc5cf61..1ad4f832 100644 > --- a/test/tarantool-tests/lj-flush-on-trace.test.lua > +++ b/test/tarantool-tests/lj-flush-on-trace.test.lua > @@ -3,6 +3,10 @@ local utils =3D require('utils') > -- Disabled on *BSD due to #4819. > utils.skipcond(jit.os =3D=3D 'BSD', 'Disabled due to #4819') >=20 > +-- XXX: Tweak the process environment to get around SIP. > +-- See the comment in suite CMakeLists.txt for more info. > +utils.tweakenv(jit.os =3D=3D 'OSX', 'DYLD_LIBRARY_PATH') > + > utils.selfrun(arg, { > { > arg =3D { > diff --git a/test/tarantool-tests/utils.lua = b/test/tarantool-tests/utils.lua > index d2dd71b0..9bdb71ec 100644 > --- a/test/tarantool-tests/utils.lua > +++ b/test/tarantool-tests/utils.lua > @@ -1,7 +1,12 @@ > local M =3D {} >=20 > +local ffi =3D require('ffi') > local tap =3D require('tap') >=20 > +ffi.cdef([[ > + int setenv(const char *name, const char *value, int overwrite); > +]]) > + Why? os.setenv() didn=E2=80=99t work for some reason? Comment it and = I=E2=80=99m fine. tarantool> os.setenv("a","b") --- ... tarantool> os.getenv("a") --- - b ... tarantool> os.execute('echo $a') b --- > local function luacmd(args) > -- arg[-1] is guaranteed to be not nil. > local idx =3D -2 > @@ -13,6 +18,12 @@ local function luacmd(args) > return table.concat(args, ' ', idx + 1, -1) > end >=20 > +local function unshiftenv(variable, value, sep) > + local envvar =3D os.getenv(variable) > + return ('%s=3D"%s%s"'):format(variable, value, > + envvar and ('%s%s'):format(sep, envvar) = or '') > +end > + > function M.selfrun(arg, checks) > -- If TEST_SELFRUN is set, just execute the test payload below > -- call, ... > @@ -24,18 +35,22 @@ function M.selfrun(arg, checks) >=20 > test:plan(#checks) >=20 > + local libext =3D package.cpath:match('?.(%a+);') > local vars =3D { > LUABIN =3D luacmd(arg), > SCRIPT =3D arg[0], > PATH =3D arg[0]:gsub('%.test%.lua', ''), > - SUFFIX =3D package.cpath:match('?.(%a+);'), > + SUFFIX =3D libext, > + ENV =3D table.concat({ > + unshiftenv('LUA_PATH', '/?.lua', ';'), > + unshiftenv('LUA_CPATH', '/?.', ';'), > + unshiftenv((libext =3D=3D 'dylib' and 'DYLD' or 'LD') .. = '_LIBRARY_PATH', > + '', ':'), > + 'TEST_SELFRUN=3D1', > + }, ' '), > } >=20 > - local cmd =3D string.gsub('LUA_PATH=3D"/?.lua;$LUA_PATH" ' .. > - 'LUA_CPATH=3D"/?.;$LUA_CPATH" = ' .. > - 'LD_LIBRARY_PATH=3D:$LD_LIBRARY_PATH = ' .. > - 'TEST_SELFRUN=3D1' .. > - ' 2>&1