From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 0CAF16EC5D; Mon, 5 Apr 2021 20:13:22 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 0CAF16EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617642802; bh=1laiMa0RWm1PFBuTtEhAdWrDfcrRUYLfooWfeeBsG40=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=gUaKlpJ4H/GILkZJnQtDX996dBVZ9zDRq2BX7+Q5vH+ATvRCNxDFXtpFsljV9B2jL WnCF6FM9Eys8hHv89lkFjddmfoE9H49Eikf1AIJrnD+KPpI2uQ3hfyY8KMaA6GRkEU LS2CoCde9ZbQ/niS3B00iadc375j+dWSsLsN8I9g= Received: from smtpng1.m.smailru.net (smtpng1.m.smailru.net [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 12DB26F3D8 for ; Mon, 5 Apr 2021 20:11:54 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 12DB26F3D8 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1lTSlU-0007LJ-EY; Mon, 05 Apr 2021 20:11:52 +0300 To: Sergey Ostanevich , Sergey Kaplun Date: Mon, 5 Apr 2021 20:11:40 +0300 Message-Id: <6ca8010540a67a92b36327abf44b489ebddc5054.1617641697.git.imun@tarantool.org> X-Mailer: git-send-email 2.25.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9ED7173E37F4E32947427BE79D20CABD4ABD7C98AF5DBFD37182A05F538085040EFA5A5BF330A746F8F7464B5C3357FDF741540C83B09ADB2DAF1D94CD3D169B1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE721AF84DC1D70954DEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637160171C9EBC7AFE48638F802B75D45FF914D58D5BE9E6BC131B5C99E7648C95C16EE06F5A270FE6AE4700A8739770988735656E1FA7D814BA471835C12D1D9774AD6D5ED66289B5278DA827A17800CE71AE4D56B06699BBC9FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C317B107DEF921CE79117882F4460429728AD0CFFFB425014E868A13BD56FB6657E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F79006371DFCA364D66B4E43EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C4C7A0BC55FA0FE5FC53F624E32DD534EAA185F4BDEB40BE25756E049400B79C58B1881A6453793CE9C32612AADDFBE061C61BE10805914D3804EBA3D8E7E5B87ABF8C51168CD8EBDB63AF70AF8205D7DCDC48ACC2A39D04F89CDFB48F4795C241BDAD6C7F3747799A X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3433E9BC74ABA5769FA01F30000702DC75D636719759DEA8BA025A12DBD13A4FFC1E951C4BDF591FBA1D7E09C32AA3244C60063058C33B33BF75D3C1A6A4C6CBA99CA7333006C390A0927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojM00ve/f+0okceB51Cbs0Dg== X-Mailru-Sender: 689FA8AB762F73936BC43F508A063822BD13DF232360366799A4E1638A02FAE7A7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit 3/3] test: fix dynamic modules loading on MacOS X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This patch resolves the issue with running the tests with auxiliary dynamically loaded modules in case of out-of-source build. The first issue relates to the way modules loaded at runtime are built on MacOS. Since the auxiliary libraries are built as a dynamically loaded modules on MacOS instead of shared libraries as it is done on Linux and BSD, another environment variable should be used to guide while searching the extension. Hence the values collected in scope of macro need to be set to DYLD_LIBRARY_PATH variable instead of LD_LIBRARY_PATH on MacOS. Unfortunately, this rather small change doesn't resolve the problem at all and the root cause lies much deeper than it seems at the beginning. Apple tries their best to "protect their users from malicious software". As a result SIP[1] has been designed and released. Now, Apple developers are *so protected*, that they can load nothing being not installed in the system, since some programs sanitize the environment before they start child processes. Specifically, environment variables starting with DYLD_ and LD_ are unset for child process started by system programs[2]. That which does not kill us makes us stronger: fortunately, these environment variables are used by FFI machinery to find the proper shared library, hence we can still tweak testing environment before calling . However, the value can't be passed via the standard environment variable, so we prepend TEST_ prefix to its name to get around SIP magic tricks. Finally, to set the variable required by FFI machinery the introduced routine is used. PROFIT! Your move, Cupertino geniuses. [1]: https://support.apple.com/en-us/HT204899 [2]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html Resolves tarantool/tarantool#5959 Follows up tarantool/tarantool#4862 Co-authored-by: Sergey Ostanevich Co-authored-by: Mons Anderson Signed-off-by: Igor Munkin --- test/tarantool-tests/CMakeLists.txt | 39 +++++++++++++++++-- .../gh-4427-ffi-sandwich.test.lua | 4 ++ .../lj-flush-on-trace.test.lua | 4 ++ test/tarantool-tests/utils.lua | 39 ++++++++++++++++--- 4 files changed, 77 insertions(+), 9 deletions(-) diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt index 3e36ff86..c793ad60 100644 --- a/test/tarantool-tests/CMakeLists.txt +++ b/test/tarantool-tests/CMakeLists.txt @@ -69,11 +69,46 @@ set(LUA_PATH ) set(LUA_TEST_SUFFIX .test.lua) set(LUA_TEST_FLAGS --failures --shuffle) +set(LUA_TEST_ENV + "LUA_PATH=\"${LUA_PATH}\;\;\"" + "LUA_CPATH=\"${LUA_CPATH}\;\;\"" +) if(CMAKE_VERBOSE_MAKEFILE) list(APPEND LUA_TEST_FLAGS --verbose) endif() +# XXX: Since the auxiliary libraries are built as a dynamically +# loaded modules on MacOS instead of shared libraries as it is +# done on Linux and BSD, another environment variable should be +# used to guide while searching the extension. +if(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + # XXX: Apple tries their best to "protect their users from + # malware". As a result SIP (see the link[1] below) has been + # designed and released. Now, Apple developers are so protected, + # that they can load nothing being not installed in the system, + # since some programs sanitize the environment before they start + # child processes. Specifically, environment variables starting + # with DYLD_ and LD_ are unset for child process started by + # system programs (like /usr/bin/env used for preparing testing + # environment). For more info, see the docs[2] below. + # + # That which does not kill us makes us stronger: fortunately, + # these environment variables are used by FFI machinery to find + # the proper shared library, hence we can still tweak testing + # environment before calling . However, the value + # can't be passed via the standard environment variable, so we + # prepend TEST_ prefix to its name to get around SIP magic + # tricks. Finally, to set the variable required by FFI machinery + # the introduced routine is used. + # + # [1]: https://support.apple.com/en-us/HT204899 + # [2]: https://developer.apple.com/library/archive/documentation/Security/Conceptual/System_Integrity_Protection_Guide/RuntimeProtections/RuntimeProtections.html + list(APPEND LUA_TEST_ENV TEST_DYLD_LIBRARY_PATH="${LD_LIBRARY_PATH}") +else() + list(APPEND LUA_TEST_ENV LD_LIBRARY_PATH="${LD_LIBRARY_PATH}") +endif() + # LUA_CPATH and LD_LIBRARY_PATH variables and also TESTLIBS list # with dependecies are set in scope of BuildTestLib macro. add_custom_target(tarantool-tests @@ -83,9 +118,7 @@ add_custom_command(TARGET tarantool-tests COMMENT "Running Tarantool tests" COMMAND env - LUA_PATH="${LUA_PATH}\;\;" - LUA_CPATH="${LUA_CPATH}\;\;" - LD_LIBRARY_PATH="${LD_LIBRARY_PATH}" + ${LUA_TEST_ENV} ${PROVE} ${CMAKE_CURRENT_SOURCE_DIR} --exec '${LUAJIT_TEST_COMMAND}' --ext ${LUA_TEST_SUFFIX} diff --git a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua index 64df5dbd..651dc3f4 100644 --- a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua +++ b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua @@ -3,6 +3,10 @@ local utils = require('utils') -- Disabled on *BSD due to #4819. utils.skipcond(jit.os == 'BSD', 'Disabled due to #4819') +-- XXX: Tweak the process environment to get around SIP. +-- See the comment in suite CMakeLists.txt for more info. +utils.tweakenv(jit.os == 'OSX', 'DYLD_LIBRARY_PATH') + utils.selfrun(arg, { { arg = { diff --git a/test/tarantool-tests/lj-flush-on-trace.test.lua b/test/tarantool-tests/lj-flush-on-trace.test.lua index edc5cf61..1ad4f832 100644 --- a/test/tarantool-tests/lj-flush-on-trace.test.lua +++ b/test/tarantool-tests/lj-flush-on-trace.test.lua @@ -3,6 +3,10 @@ local utils = require('utils') -- Disabled on *BSD due to #4819. utils.skipcond(jit.os == 'BSD', 'Disabled due to #4819') +-- XXX: Tweak the process environment to get around SIP. +-- See the comment in suite CMakeLists.txt for more info. +utils.tweakenv(jit.os == 'OSX', 'DYLD_LIBRARY_PATH') + utils.selfrun(arg, { { arg = { diff --git a/test/tarantool-tests/utils.lua b/test/tarantool-tests/utils.lua index d2dd71b0..9bdb71ec 100644 --- a/test/tarantool-tests/utils.lua +++ b/test/tarantool-tests/utils.lua @@ -1,7 +1,12 @@ local M = {} +local ffi = require('ffi') local tap = require('tap') +ffi.cdef([[ + int setenv(const char *name, const char *value, int overwrite); +]]) + local function luacmd(args) -- arg[-1] is guaranteed to be not nil. local idx = -2 @@ -13,6 +18,12 @@ local function luacmd(args) return table.concat(args, ' ', idx + 1, -1) end +local function unshiftenv(variable, value, sep) + local envvar = os.getenv(variable) + return ('%s="%s%s"'):format(variable, value, + envvar and ('%s%s'):format(sep, envvar) or '') +end + function M.selfrun(arg, checks) -- If TEST_SELFRUN is set, just execute the test payload below -- call, ... @@ -24,18 +35,22 @@ function M.selfrun(arg, checks) test:plan(#checks) + local libext = package.cpath:match('?.(%a+);') local vars = { LUABIN = luacmd(arg), SCRIPT = arg[0], PATH = arg[0]:gsub('%.test%.lua', ''), - SUFFIX = package.cpath:match('?.(%a+);'), + SUFFIX = libext, + ENV = table.concat({ + unshiftenv('LUA_PATH', '/?.lua', ';'), + unshiftenv('LUA_CPATH', '/?.', ';'), + unshiftenv((libext == 'dylib' and 'DYLD' or 'LD') .. '_LIBRARY_PATH', + '', ':'), + 'TEST_SELFRUN=1', + }, ' '), } - local cmd = string.gsub('LUA_PATH="/?.lua;$LUA_PATH" ' .. - 'LUA_CPATH="/?.;$LUA_CPATH" ' .. - 'LD_LIBRARY_PATH=:$LD_LIBRARY_PATH ' .. - 'TEST_SELFRUN=1' .. - ' 2>&1