From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 8728B4CC566; Tue, 27 Jun 2023 16:36:44 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8728B4CC566 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1687873004; bh=9K+EmSlj906050SIxqRj5GsLDlkLM71+qhTzlCOB6KQ=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=T0wBc2lXcMabcsJYe30glU2vf51y/5Q/xRi61B6SOWMKKkRcUmXaLAfSVw8cfKmis FTV0tNovspyybjL8amhYQszyHvcrWxEGyczzQKUFEPhYD0XlZxIhNOlFUZZOUQpOgB 2BsIhz9OaJ2WookmJYBhwkN2BNyG7IuiUpIVfXXc= Received: from smtp43.i.mail.ru (smtp43.i.mail.ru [95.163.41.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 1ACE5480007 for ; Tue, 27 Jun 2023 16:36:43 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 1ACE5480007 Received: by smtp43.i.mail.ru with esmtpa (envelope-from ) id 1qE8s5-0053M8-6d; Tue, 27 Jun 2023 16:36:42 +0300 To: Sergey Kaplun , Maxim Kokryashkin , Sergey Bronnikov Date: Tue, 27 Jun 2023 13:28:48 +0000 Message-Id: X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD91D67636EC9E920187801D96EA193266FC31611D4AF02F55D6E3621050C10BCEABD01EDE013A66790FAF9FF5CDE6886963103D39EE50699EDF8D9167951635F18 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7E2331B2371EFE129EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006377BAB68A65B44F13B8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D81BFEB7031BE1B5719F30DEE524EF951C117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC8C7ADC89C2F0B2A5A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F4460429728776938767073520CCD848CCB6FE560C6FD1C55BDD38FC3FD2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE22A4E020D69BCAC1452896749CDDA0A6D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE308214CF94FAA95E02D242C3BD2E3F4C6C4224003CC836476E2F48590F00D11D6E2021AF6380DFAD1A18204E546F3947CD2DCF9CF1F528DBC2E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89F249797B4B1AC144935872C767BF85DA2F004C90652538430E4A6367B16DE6309 X-87b9d050: 1 X-C1DE0DAB: 0D63561A33F958A57B6E4173F1682126F3A544ED32DF25DEA55295136EE8523DF87CCE6106E1FC07E67D4AC08A07B9B065B78C30F681404D9C5DF10A05D560A950611B66E3DA6D700B0A020F03D25A0997E3FB2386030E77 X-C8649E89: 1C3962B70DF3F0ADE00A9FD3E00BEEDF3FED46C3ACD6F73ED3581295AF09D3DF87807E0823442EA2ED31085941D9CD0AF7F820E7B07EA4CFBF6D99F82428424CE6518AD05B285AC06CDC5A5C13AB4FE474B1147E8080FD3BA53151558601347D00E7A357E107337478B8A9A3EB8A3FAA806D2271A7590BB2461A413F07889F2102C26D483E81D6BEECAEF3E2CCC1ED8C383653B6C8D9AE0FD16FCAA6493B703A X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXgppy5g+xWllKgkzEX66fHc X-Mailru-Sender: 2FEBA92C8E508479FE7B9A1DF348D531EB0721899D4D513A581831AD0775621422C146515D73C9EB2326FE6F2A341ACE0FB9F97486540B4CD9E8847AB8CFED4D9ABF8A61C016C2CFB0DAF586E7D11B3E67EA787935ED9F1B X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit 1/3] test: split utils.lua into several modules X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" The next patch introduces a separate JIT-related module with convenient utils for JIT engine testing. Considering this change it looks vital to make a structured utils distributed module instead of "all in one" Lua chunk. As a result the original utils.lua is split into the several modules per subsystem to be tested (e.g. GC, frontend, profilers, etc.). Lazy loading of the introduced submodules allows to use this utils in all test chunks regardless LuaJIT configuration (e.g. with JIT engine disabled, without FFI support, etc) and do not spoil utils table with the excess helpers. Signed-off-by: Igor Munkin --- Sergey, considering the changes you've made in the second patch, I propose to finally split utils.lua into submodules that loads lazily. I've pushed my commit on your branch prior to your patchset. If you have some notes regarding this, please share them. Otherwise, I'll push this into the LuaJIT long-term branches. You can also find the trivial changes made within your commits below: ================================================================================ diff --git a/test/tarantool-tests/lj-981-folding-0.test.lua b/test/tarantool-tests/lj-981-folding-0.test.lua index 64473ba3..d156f53d 100644 --- a/test/tarantool-tests/lj-981-folding-0.test.lua +++ b/test/tarantool-tests/lj-981-folding-0.test.lua @@ -8,7 +8,7 @@ local test = tap.test('lj-981-folding-0'):skipcond({ -- for -0 IR constant as table index. -- See also, https://github.com/LuaJIT/LuaJIT/issues/981. -local jparse = require('utils.jit_parse') +local jparse = require('utils').jit.parse -- XXX: Avoid any other traces compilation due to hotcount -- collisions for predictable results. diff --git a/test/tarantool-tests/unit-jit-parse.test.lua b/test/tarantool-tests/unit-jit-parse.test.lua index e9c0bb80..e4445bf4 100644 --- a/test/tarantool-tests/unit-jit-parse.test.lua +++ b/test/tarantool-tests/unit-jit-parse.test.lua @@ -4,7 +4,7 @@ local test = tap.test('unit-jit-parse'):skipcond({ ['Disabled on *BSD due to #4819'] = jit.os == 'BSD', }) -local jparse = require('utils.jit_parse') +local jparse = require('utils').jit.parse local expected_irs = { -- The different exotic builds may add different IR ================================================================================ test/tarantool-tests/CMakeLists.txt | 2 +- .../bc-jit-unpatching.test.lua | 5 +- .../fix-gc-setupvalue.test.lua | 4 +- .../gh-4427-ffi-sandwich.test.lua | 2 +- .../gh-5813-resolving-of-c-symbols.test.lua | 2 +- ...-missed-carg1-in-bctsetr-fallback.test.lua | 2 +- .../lj-351-print-tostring-number.test.lua | 2 +- .../lj-586-debug-non-string-error.test.lua | 2 +- .../lj-flush-on-trace.test.lua | 2 +- .../misclib-getmetrics-lapi.test.lua | 2 +- .../misclib-memprof-lapi.test.lua | 2 +- .../misclib-sysprof-lapi.test.lua | 2 +- test/tarantool-tests/utils.lua | 125 ------------------ test/tarantool-tests/utils/exec.lua | 52 ++++++++ test/tarantool-tests/utils/frontend.lua | 25 ++++ test/tarantool-tests/utils/gc.lua | 33 +++++ test/tarantool-tests/utils/init.lua | 7 + test/tarantool-tests/utils/jit/const.lua | 8 ++ test/tarantool-tests/utils/jit/init.lua | 7 + test/tarantool-tests/utils/tools.lua | 15 +++ 20 files changed, 162 insertions(+), 139 deletions(-) delete mode 100644 test/tarantool-tests/utils.lua create mode 100644 test/tarantool-tests/utils/exec.lua create mode 100644 test/tarantool-tests/utils/frontend.lua create mode 100644 test/tarantool-tests/utils/gc.lua create mode 100644 test/tarantool-tests/utils/init.lua create mode 100644 test/tarantool-tests/utils/jit/const.lua create mode 100644 test/tarantool-tests/utils/jit/init.lua create mode 100644 test/tarantool-tests/utils/tools.lua diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt index 527905b6..14a98cf2 100644 --- a/test/tarantool-tests/CMakeLists.txt +++ b/test/tarantool-tests/CMakeLists.txt @@ -75,7 +75,7 @@ add_subdirectory(lj-flush-on-trace) # directory), so LUA_PATH need to be updated. make_lua_path(LUA_PATH PATHS - ${CMAKE_CURRENT_SOURCE_DIR}/?.lua + ${CMAKE_CURRENT_SOURCE_DIR}/?/init.lua ${PROJECT_SOURCE_DIR}/tools/?.lua ${LUAJIT_SOURCE_DIR}/?.lua ${LUAJIT_BINARY_DIR}/?.lua diff --git a/test/tarantool-tests/bc-jit-unpatching.test.lua b/test/tarantool-tests/bc-jit-unpatching.test.lua index 2c3b7c9a..f4f148c9 100644 --- a/test/tarantool-tests/bc-jit-unpatching.test.lua +++ b/test/tarantool-tests/bc-jit-unpatching.test.lua @@ -13,12 +13,13 @@ end local ret1bc = 'RET1%s*1%s*2' -- Check that this bytecode still persists. -assert(utils.hasbc(load(string.dump(f)), ret1bc)) +assert(utils.frontend.hasbc(load(string.dump(f)), ret1bc)) jit.opt.start('hotloop=1', 'hotexit=1') -- Compile function to get JLOOP bytecode in recursion. f(5) -test:ok(utils.hasbc(load(string.dump(f)), ret1bc), 'bytecode unpatching is OK ') +test:ok(utils.frontend.hasbc(load(string.dump(f)), ret1bc), + 'bytecode unpatching is OK') os.exit(test:check() and 0 or 1) diff --git a/test/tarantool-tests/fix-gc-setupvalue.test.lua b/test/tarantool-tests/fix-gc-setupvalue.test.lua index c94d5d85..21acc9bf 100644 --- a/test/tarantool-tests/fix-gc-setupvalue.test.lua +++ b/test/tarantool-tests/fix-gc-setupvalue.test.lua @@ -1,5 +1,5 @@ local tap = require('tap') -local utils = require('utils') +local gcisblack = require('utils').gc.isblack local test = tap.test('fix-gc-setupvalue') test:plan(1) @@ -40,7 +40,7 @@ local oldstepmul = collectgarbage('setstepmul', 1) -- `parent()` function is marked before `child()`, so wait until -- it becomes black and proceed with the test. -while not utils.gcisblack(_G.parent) do +while not gcisblack(_G.parent) do collectgarbage('step') end diff --git a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua index 86544196..677a6085 100644 --- a/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua +++ b/test/tarantool-tests/gh-4427-ffi-sandwich.test.lua @@ -8,7 +8,7 @@ test:plan(2) -- runs %testname%/script.lua by -- with the given environment, launch options and CLI arguments. -local script = require('utils').makecmd(arg, { +local script = require('utils').exec.makecmd(arg, { -- XXX: Apple tries their best to "protect their users from -- malware". As a result SIP (see the link[1] below) has been -- designed and released. Now, Apple developers are so diff --git a/test/tarantool-tests/gh-5813-resolving-of-c-symbols.test.lua b/test/tarantool-tests/gh-5813-resolving-of-c-symbols.test.lua index 9f2c5f85..1209d288 100644 --- a/test/tarantool-tests/gh-5813-resolving-of-c-symbols.test.lua +++ b/test/tarantool-tests/gh-5813-resolving-of-c-symbols.test.lua @@ -17,7 +17,7 @@ local symtab = require "utils.symtab" local testboth = require "resboth" local testhash = require "reshash" local testgnuhash = require "resgnuhash" -local profilename = require("utils").profilename +local profilename = require("utils").tools.profilename local TMP_BINFILE = profilename("memprofdata.tmp.bin") diff --git a/test/tarantool-tests/gh-6084-missed-carg1-in-bctsetr-fallback.test.lua b/test/tarantool-tests/gh-6084-missed-carg1-in-bctsetr-fallback.test.lua index 86bbabe3..10db7603 100644 --- a/test/tarantool-tests/gh-6084-missed-carg1-in-bctsetr-fallback.test.lua +++ b/test/tarantool-tests/gh-6084-missed-carg1-in-bctsetr-fallback.test.lua @@ -15,7 +15,7 @@ test:plan(2) -- XXX: We need to make sure the bytecode is present in the chosen -- built-in to make sure our test is still valid. -assert(utils.hasbc(table.move, 'TSETR')) +assert(utils.frontend.hasbc(table.move, 'TSETR')) -- `t` table asize equals 1. Just copy its first element (1) -- to the field by index 2 > 1, to fallback inside TSETR. diff --git a/test/tarantool-tests/lj-351-print-tostring-number.test.lua b/test/tarantool-tests/lj-351-print-tostring-number.test.lua index 72a9ec2b..b7041f2a 100644 --- a/test/tarantool-tests/lj-351-print-tostring-number.test.lua +++ b/test/tarantool-tests/lj-351-print-tostring-number.test.lua @@ -3,7 +3,7 @@ local tap = require('tap') local test = tap.test('lj-351-print-tostring-number') test:plan(8) -local script = require('utils').makecmd(arg) +local script = require('utils').exec.makecmd(arg) local cases = { {typename = 'nil', value = 'nil'}, diff --git a/test/tarantool-tests/lj-586-debug-non-string-error.test.lua b/test/tarantool-tests/lj-586-debug-non-string-error.test.lua index dcb730a2..c00301a1 100644 --- a/test/tarantool-tests/lj-586-debug-non-string-error.test.lua +++ b/test/tarantool-tests/lj-586-debug-non-string-error.test.lua @@ -21,7 +21,7 @@ test:plan(1) -- Debugger prompt. local ldb = 'lua_debug> ' local magic = 42 -local luabin = utils.luacmd(arg) +local luabin = utils.exec.luacmd(arg) local stderr = { -- XXX: The first debugger prompt printed at the start. ldb, diff --git a/test/tarantool-tests/lj-flush-on-trace.test.lua b/test/tarantool-tests/lj-flush-on-trace.test.lua index 46db4d2a..fe740087 100644 --- a/test/tarantool-tests/lj-flush-on-trace.test.lua +++ b/test/tarantool-tests/lj-flush-on-trace.test.lua @@ -8,7 +8,7 @@ test:plan(2) -- runs %testname%/script.lua by -- with the given environment, launch options and CLI arguments. -local script = require('utils').makecmd(arg, { +local script = require('utils').exec.makecmd(arg, { -- XXX: Apple tries their best to "protect their users from -- malware". As a result SIP (see the link[1] below) has been -- designed and released. Now, Apple developers are so diff --git a/test/tarantool-tests/misclib-getmetrics-lapi.test.lua b/test/tarantool-tests/misclib-getmetrics-lapi.test.lua index 881e717b..0ee71499 100644 --- a/test/tarantool-tests/misclib-getmetrics-lapi.test.lua +++ b/test/tarantool-tests/misclib-getmetrics-lapi.test.lua @@ -10,7 +10,7 @@ local test = tap.test("lib-misc-getmetrics"):skipcond({ test:plan(10) -local MAXNINS = require('utils').const.maxnins +local MAXNINS = require('utils').jit.const.maxnins local jit_opt_default = { 3, -- level "hotloop=56", diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua index 4e413c88..eae20893 100644 --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua @@ -26,7 +26,7 @@ local bufread = require "utils.bufread" local memprof = require "memprof.parse" local process = require "memprof.process" local symtab = require "utils.symtab" -local profilename = require("utils").profilename +local profilename = require("utils").tools.profilename local TMP_BINFILE = profilename("memprofdata.tmp.bin") local BAD_PATH = profilename("memprofdata/tmp.bin") diff --git a/test/tarantool-tests/misclib-sysprof-lapi.test.lua b/test/tarantool-tests/misclib-sysprof-lapi.test.lua index 96eaaab6..2f0635db 100644 --- a/test/tarantool-tests/misclib-sysprof-lapi.test.lua +++ b/test/tarantool-tests/misclib-sysprof-lapi.test.lua @@ -15,7 +15,7 @@ pcall(jit.flush) local bufread = require("utils.bufread") local symtab = require("utils.symtab") local sysprof = require("sysprof.parse") -local profilename = require("utils").profilename +local profilename = require("utils").tools.profilename local TMP_BINFILE = profilename("sysprofdata.tmp.bin") local BAD_PATH = profilename("sysprofdata/tmp.bin") diff --git a/test/tarantool-tests/utils.lua b/test/tarantool-tests/utils.lua deleted file mode 100644 index 8c1538d6..00000000 --- a/test/tarantool-tests/utils.lua +++ /dev/null @@ -1,125 +0,0 @@ -local M = {} - -local ffi = require('ffi') -local bc = require('jit.bc') -local bit = require('bit') - -local LJ_GC_BLACK = 0x04 -local LJ_STR_HASHLEN = 8 -local GCref = ffi.abi('gc64') and 'uint64_t' or 'uint32_t' - -ffi.cdef([[ - typedef struct { -]]..GCref..[[ nextgc; - uint8_t marked; - uint8_t gct; - /* Need this fields for correct alignment and sizeof. */ - uint8_t misc1; - uint8_t misc2; - } GCHeader; -]]) - -function M.gcisblack(obj) - local objtype = type(obj) - local address = objtype == 'string' - -- XXX: get strdata first and go back to GCHeader. - and ffi.cast('char *', obj) - (ffi.sizeof('GCHeader') + LJ_STR_HASHLEN) - -- XXX: FFI ABI forbids to cast functions objects - -- to non-functional pointers, but we can get their address - -- via tostring. - or tonumber((tostring(obj):gsub(objtype .. ': ', ''))) - local marked = ffi.cast('GCHeader *', address).marked - return bit.band(marked, LJ_GC_BLACK) == LJ_GC_BLACK -end - -function M.luacmd(args) - -- arg[-1] is guaranteed to be not nil. - local idx = -2 - while args[idx] do - assert(type(args[idx]) == 'string', 'Command part have to be a string') - idx = idx - 1 - end - -- return the full command with flags. - return table.concat(args, ' ', idx + 1, -1) -end - -local function makeenv(tabenv) - if tabenv == nil then return '' end - local flatenv = {} - for var, value in pairs(tabenv) do - table.insert(flatenv, ('%s=%s'):format(var, value)) - end - return table.concat(flatenv, ' ') -end - --- creates a command that runs %testname%/script.lua by --- with the given environment, launch options --- and CLI arguments. The function yields an object (i.e. table) --- with the aforementioned parameters. To launch the command just --- call the object. -function M.makecmd(arg, opts) - return setmetatable({ - LUABIN = M.luacmd(arg), - SCRIPT = opts and opts.script or arg[0]:gsub('%.test%.lua$', '/script.lua'), - ENV = opts and makeenv(opts.env) or '', - REDIRECT = opts and opts.redirect or '', - }, { - __call = function(self, ...) - -- This line just makes the command for by the - -- following steps: - -- 1. Replace the placeholders with the corresponding values - -- given to the command constructor (e.g. script, env). - -- 2. Join all CLI arguments given to the __call metamethod. - -- 3. Concatenate the results of step 1 and step 2 to obtain - -- the resulting command. - local cmd = ('