From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 6690F6EC5D; Wed, 7 Apr 2021 20:33:23 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 6690F6EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617816803; bh=iu5lHXeXBmhOE69U0j+O4H1mCxyanVs4kA0dMPe7lh0=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=n3CLOy5cpfXBI0pj4rMmt5V/oqSPuTntgrh9RX7P95aTTAqblBvyaYQDaAImcRvin rGzODhwUQmkPgSykG4QaEy6OgFciVLvSs9NFEY+U3r+F8tJXEA8a6rmd3hk7xHEUMf eIwdz5Z/W3zCn7PVp9kttqSei4nwnli9WzIyvRAY= Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 46DBC6EC5D for ; Wed, 7 Apr 2021 20:33:22 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 46DBC6EC5D Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1lUC3N-0002Ha-04; Wed, 07 Apr 2021 20:33:21 +0300 Date: Wed, 7 Apr 2021 20:33:10 +0300 To: Sergey Kaplun Message-ID: <20210407173310.GP29703@tarantool.org> References: <6ca8010540a67a92b36327abf44b489ebddc5054.1617641697.git.imun@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.10.1 (2018-07-13) X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD912A3E3D5D4B49FC17951DFF5B22C9C640BE5EAB58CFCE0F000894C459B0CD1B90F58F04F1C5B51ECA06A018F5595ACF46378F92AAB47E6373B17CAB2765D6A15 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7BB17EE3498E810FEEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006378010A306A5B6F90A8638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B24B7DD28DD8AACC63D05F29F321906ED7618001F51B5FD3F9D2E47CDBA5A96583C09775C1D3CA48CFED8438A78DFE0A9E117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658378DA827A17800CE7328B01A8D746D8839FA2833FD35BB23DF004C906525384302BEBFE083D3B9BA71A620F70A64A45A98AA50765F79006372E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89FB26E97DCB74E625235872C767BF85DA2F004C90652538430E4A6367B16DE6309 X-C1DE0DAB: 0D63561A33F958A5E7FC82D734A608D7F9527C5BC0770561E28EDDB469513EE9D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34E318F287A436F24CEDA872E81D84D34E755E95A25A181CECF0095CFF0E3239A9D91D62663E6B42941D7E09C32AA3244C60C3FFFFE3908B120EBF2A11EE265D4A7101BF96129E4011927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioju+jaMfvANXq5nqWccD5mnA== X-Mailru-Sender: 689FA8AB762F73936BC43F508A0638221046C1E5D954FF0E5C4735F3C53731B9A7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 3/3] test: fix dynamic modules loading on MacOS X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Sergey, Thanks for your review! On 07.04.21, Sergey Kaplun wrote: > Igor, > > Thanks for the patch! > > LGTM, except a few nitpicks below. Added your tag: | Reviewed-by: Sergey Kaplun > > On 05.04.21, Igor Munkin wrote: > > This patch resolves the issue with running the tests with auxiliary > > dynamically loaded modules in case of out-of-source build. > > > > The first issue relates to the way modules loaded at runtime are built > > on MacOS. Since the auxiliary libraries are built as a dynamically > > loaded modules on MacOS instead of shared libraries as it is done on > > Linux and BSD, another environment variable should be used to guide > > while searching the extension. Hence the values collected in > > scope of macro need to be set to DYLD_LIBRARY_PATH > > variable instead of LD_LIBRARY_PATH on MacOS. > > > > Unfortunately, this rather small change doesn't resolve the problem at > > all and the root cause lies much deeper than it seems at the beginning. > > This paragraph looks uninformative, looks like you can just use > "Moreover, " for the next sentence instead. It could be "Moreover, " if it was another tiny problem, or I missed another environment variable, but I think everything below is not just "Moreover, ". BTW, considering your comment I hope this is not a big problem for you (or, so-called "Moreover, " problem), and our team has such an expert in MacOS environment! Re suggestion: Ignoring. > > > > > Apple tries their best to "protect their users from malicious software". > > As a result SIP[1] has been designed and released. Now, Apple developers > > are *so protected*, that they can load nothing being not installed in > > the system, since some programs sanitize the environment before they > > start child processes. Specifically, environment variables starting with > > DYLD_ and LD_ are unset for child process started by system programs[2]. > > Sorry, but this paragraph is too wordy. I propose to reformulate it like > the following: Does it make the commit message less clear? I hope no. At the same time, there is an important notice that some programs sanitize the environment before starting the child process. > > | Moreover, Apple tries their best to "protect their users from malicious > | software". As a result, SIP[1] has been designed and released. As a part > | of it, environment variables starting with DYLD_ and LD_ are unset > | for child process started by system programs[2]. > > Feel free to ignore. Ignoring. > > > > > That which does not kill us makes us stronger: fortunately, these > > I suppose you can just drop this preamble. Honestly, see no reason for this. Ignoring. > > | These environment variables are used by FFI machinery to find the proper > | shared library, hence we can still tweak testing environment before > | calling . > > > environment variables are used by FFI machinery to find the proper > > shared library, hence we can still tweak testing environment before > > calling . However, the value can't be passed via the standard > > environment variable, so we prepend TEST_ prefix to its name to get > > around SIP magic tricks. Finally, to set the variable required by FFI > > machinery the introduced routine is used. PROFIT! > > Your move, Cupertino geniuses. > > The last two sentences are redundant. Remove them please. This is your opinion. As for me, everything above is written just for the last one. We implicitly agreed not to use "emotional" wording in comments, but I don't like this (IMHO such wording makes grepping for pitfalls easier). There is no strong language in the commit message also. Again, see no reason for removing. Ignoring. > > > > > [1]: https://support.apple.com/en-us/HT204899 > > [2]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html > > > > Resolves tarantool/tarantool#5959 > > Follows up tarantool/tarantool#4862 > > > > Co-authored-by: Sergey Ostanevich > > Co-authored-by: Mons Anderson > > Signed-off-by: Igor Munkin > > --- > > test/tarantool-tests/CMakeLists.txt | 39 +++++++++++++++++-- > > .../gh-4427-ffi-sandwich.test.lua | 4 ++ > > .../lj-flush-on-trace.test.lua | 4 ++ > > test/tarantool-tests/utils.lua | 39 ++++++++++++++++--- > > 4 files changed, 77 insertions(+), 9 deletions(-) > > > > diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt > > index 3e36ff86..c793ad60 100644 > > --- a/test/tarantool-tests/CMakeLists.txt > > +++ b/test/tarantool-tests/CMakeLists.txt > > @@ -69,11 +69,46 @@ set(LUA_PATH > > ) > > set(LUA_TEST_SUFFIX .test.lua) > > set(LUA_TEST_FLAGS --failures --shuffle) > > +set(LUA_TEST_ENV > > + "LUA_PATH=\"${LUA_PATH}\;\;\"" > > + "LUA_CPATH=\"${LUA_CPATH}\;\;\"" > > +) > > > > if(CMAKE_VERBOSE_MAKEFILE) > > list(APPEND LUA_TEST_FLAGS --verbose) > > endif() > > > > +# XXX: Since the auxiliary libraries are built as a dynamically > > +# loaded modules on MacOS instead of shared libraries as it is > > +# done on Linux and BSD, another environment variable should be > > +# used to guide while searching the extension. > > +if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") > > + # XXX: Apple tries their best to "protect their users from > > + # malware". As a result SIP (see the link[1] below) has been > > + # designed and released. Now, Apple developers are so protected, > > + # that they can load nothing being not installed in the system, > > + # since some programs sanitize the environment before they start > > + # child processes. Specifically, environment variables starting > > + # with DYLD_ and LD_ are unset for child process started by > > + # system programs (like /usr/bin/env used for preparing testing > > + # environment). For more info, see the docs[2] below. > > + # > > + # That which does not kill us makes us stronger: fortunately, > > + # these environment variables are used by FFI machinery to find > > + # the proper shared library, hence we can still tweak testing > > + # environment before calling . However, the value > > + # can't be passed via the standard environment variable, so we > > + # prepend TEST_ prefix to its name to get around SIP magic > > + # tricks. Finally, to set the variable required by FFI machinery > > + # the introduced routine is used. > > Same comments as for the commit message. Removed the line that bothers you by its redundancy. Here is the diff: ================================================================================ diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt index c793ad60..475e2e5d 100644 --- a/test/tarantool-tests/CMakeLists.txt +++ b/test/tarantool-tests/CMakeLists.txt @@ -93,8 +93,7 @@ if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") # system programs (like /usr/bin/env used for preparing testing # environment). For more info, see the docs[2] below. # - # That which does not kill us makes us stronger: fortunately, - # these environment variables are used by FFI machinery to find + # These environment variables are used by FFI machinery to find # the proper shared library, hence we can still tweak testing # environment before calling . However, the value # can't be passed via the standard environment variable, so we ================================================================================ > > > + # > > + # [1]: https://support.apple.com/en-us/HT204899 > > + # [2]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html > > + list(APPEND LUA_TEST_ENV TEST_DYLD_LIBRARY_PATH="${LD_LIBRARY_PATH}") > > +else() > > + list(APPEND LUA_TEST_ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}") > > +endif() > > + > > # LUA_CPATH and LD_LIBRARY_PATH variables and also TESTLIBS list > > # with dependecies are set in scope of BuildTestLib macro. > > add_custom_target(tarantool-tests > > @@ -83,9 +118,7 @@ add_custom_command(TARGET tarantool-tests > > COMMENT "Running Tarantool tests" > > COMMAND > > env > > - LUA_PATH="${LUA_PATH}\;\;" > > - LUA_CPATH="${LUA_CPATH}\;\;" > > - LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" > > + ${LUA_TEST_ENV} > > ${PROVE} ${CMAKE_CURRENT_SOURCE_DIR} > > --exec '${LUAJIT_TEST_COMMAND}' > > --ext ${LUA_TEST_SUFFIX} > > diff --git a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > > index 64df5dbd..651dc3f4 100644 > > --- a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > > +++ b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua > > @@ -3,6 +3,10 @@ local utils = require('utils') > > -- Disabled on *BSD due to #4819. > > utils.skipcond(jit.os == 'BSD', 'Disabled due to #4819') > > > > +-- XXX: Tweak the process environment to get around SIP. > > +-- See the comment in suite CMakeLists.txt for more info. > > +utils.tweakenv(jit.os == 'OSX', 'DYLD_LIBRARY_PATH') > > + > > utils.selfrun(arg, { > > { > > arg = { > > diff --git a/test/tarantool-tests/lj-flush-on-trace.test.lua b/test/tarantool-tests/lj-flush-on-trace.test.lua > > index edc5cf61..1ad4f832 100644 > > --- a/test/tarantool-tests/lj-flush-on-trace.test.lua > > +++ b/test/tarantool-tests/lj-flush-on-trace.test.lua > > @@ -3,6 +3,10 @@ local utils = require('utils') > > -- Disabled on *BSD due to #4819. > > utils.skipcond(jit.os == 'BSD', 'Disabled due to #4819') > > > > +-- XXX: Tweak the process environment to get around SIP. > > +-- See the comment in suite CMakeLists.txt for more info. > > +utils.tweakenv(jit.os == 'OSX', 'DYLD_LIBRARY_PATH') > > + > > utils.selfrun(arg, { > > { > > arg = { > > diff --git a/test/tarantool-tests/utils.lua b/test/tarantool-tests/utils.lua > > index d2dd71b0..9bdb71ec 100644 > > --- a/test/tarantool-tests/utils.lua > > +++ b/test/tarantool-tests/utils.lua > > @@ -1,7 +1,12 @@ > > local M = {} > > > > +local ffi = require('ffi') > > local tap = require('tap') > > > > +ffi.cdef([[ > > + int setenv(const char *name, const char *value, int overwrite); > > +]]) > > Side note: looks like we need to move `os.setenv()` fucntion from the > Tarantool's built-ins onboard as `misc.setenv()`. I personally don't think this is a vital function to be moved to LuaJIT, but I'm open for ideas! > > > + > > > > > -- > > 2.25.0 > > > > -- > Best regards, > Sergey Kaplun -- Best regards, IM