From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 193E3BE8BB9; Thu, 11 Jul 2024 10:29:57 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 193E3BE8BB9 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1720682997; bh=1puGgFKow6nDoH+PaYTV6LXTXqhkhUx4Hi8pFzMbu5w=; h=Date:To:Cc:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=H7xL3YMcnZ4QyoSwADVKSKyefJosR22LP3/hd9f0VLhok/lIhTlwf+8irwVDlwbqm 5jHLDm5uHE6qgEH3BZYHpc+B0d5mtO+XdbImnPB+auj01vg0ZHwzfZ50yqDrJrrgVz 1YbJ4v1WcezQP51JuV52XhL4/xOo2DF0CjOPdqUM= Received: from smtp47.i.mail.ru (smtp47.i.mail.ru [95.163.41.85]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 59FF4BE8B97 for ; Thu, 11 Jul 2024 10:29:55 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 59FF4BE8B97 Received: by exim-smtp-687d8cf49b-ltjf2 with esmtpa (envelope-from ) id 1sRoFU-00000000DSK-2GgW; Thu, 11 Jul 2024 10:29:52 +0300 Date: Thu, 11 Jul 2024 10:29:43 +0300 To: Sergey Bronnikov Cc: Sergey Bronnikov , tarantool-patches@dev.tarantool.org Message-ID: References: <10ed208fcfacfa4c772f1cebe090595af3452ff3.1720182442.git.sergeyb@tarantool.org> <95ce34d1-234c-4e5c-a9a5-a62a5a5633e8@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <95ce34d1-234c-4e5c-a9a5-a62a5a5633e8@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD9985AA43F8E1EDB6ED382CF84F05CF30A907CCB1399AC09AD00894C459B0CD1B947EC7691D556EF5CAC8EDD30083ED68E94B81011C9C23C4CBA9DF8F2AC580DB6D64C754CAC7C657B X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7D4A169723F56FEDEEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637FAFFEDEAEB71C4328638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8CBDD37A1C38CC5C2E8C12765C99BDFFCEFBF3083E6BED411CC7F00164DA146DAFE8445B8C89999728AA50765F7900637D0FEED2715E18529389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC821E93C0F2A571C7BF6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA7E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8BB9CEE4F2B4A90F8475ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A5E065DC35E1635A745002B1117B3ED696352CA3EE88B147961A1B8FE1FED62FE8823CB91A9FED034534781492E4B8EEADD0953842B444AAC3BDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0ADBF74143AD284FC7177DD89D51EBB7742424CF958EAFF5D571004E42C50DC4CA955A7F0CF078B5EC49A30900B95165D34023CA4E4726A7D6C9FE34D60D507F68C8FA2BBD9E0E3EF30EA558046B2517953981325C7E08758BD1D7E09C32AA3244C8505CD43EF2F39EC77DD89D51EBB77429B32E99980C10ED4EA455F16B58544A2557BDE0DD54B3590A5AE236DF995FB59829709634694AABAED6A17656DB59BCAD427812AF56FC65B X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojYxP1dNBoSZmQ91TpDbE+VQ== X-DA7885C5: C701174CCB4FDD07F255D290C0D534F9F98CA27838DAADC054D1FC54F823E36DDA54BB359D799BCF5B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393C6D0B12EA33CAA9B86695FB4B080F7BC401B11BFC442C0CB31D694A9837F6255E49D44BB4BD9522A059A1ED8796F048DB274557F927329BE89D5A3BC2B10C37545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi, Sergey! Thanks for the fixes! I've applied some minor fixes to your branch. LGTM, after adding the comment about ncmds value. On 10.07.24, Sergey Bronnikov wrote: > Sergey, > > thanks for review. Fixes applied and force-pushed to the same branch. > > > Sergey > > On 09.07.2024 16:03, Sergey Kaplun via Tarantool-patches wrote: > > Hi, Sergey! > > Thanks for the patch! > > Please consider my comments below. > > > > May we add the test [1] to verify that there will be no regression in the > > future? > > This "test" was for a FAT binary generated by LuaJIT. > > With backported patches, FAT (aka Universal Binary) support is gone. > > mach-O header is checked by tests in proposed patch. OK, let's leave it as is. > > > > > On 05.07.24, Sergey Bronnikov wrote: > >> Reported by Sergey Bronnikov. > >> > >> (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) > >> > >> Previously, LuaJIT generated Mach-O FAT object files for ARM and > >> ARM64 on macOS. The patch removes support of 32-bit ARM and > >> FAT object files and now LuaJIT generate Mach-O object files for > >> ARM64. > > I suppose we should mention that no x86/x86_64 objects are generated > > now. > x86_64 is still there: > > $ ./build/gc64/src/luajit -b -o osx -a arm64 empty.lua empty.o > $ file empty.o > empty.o: Mach-O 64-bit arm64 object > $ ./build/gc64/src/luajit -b -o osx -a x64 empty.lua empty.o > $ file empty.o > empty.o: Mach-O 64-bit x86_64 object > $ My bad, thanks for the clarification. > > Anyway, commit message has been updated. > > >> Sergey Bronnikov: > >> * added the description and the trimmed the test for the problem > >> > >> Part of tarantool/tarantool#10199 > >> --- > >> src/jit/bcsave.lua | 155 ++------- > >> ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- > >> 2 files changed, 70 insertions(+), 379 deletions(-) > >> > >> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua > >> index 26ec29c6..61953c2d 100644 > >> --- a/src/jit/bcsave.lua > >> +++ b/src/jit/bcsave.lua > > > >> + subtest:is(mach_o.header.ncmds, 2, > > Why there are 2 commands for Mach-O format? > > Please use the named constant. > is constant name 'ncmds' self-explained? No, since I don't understand why it is 2 and not 12. Please, add a comment to make it obvious for any reader. > > > >> + 'ncmds is correct in Mach-O') > > Looks like this line may be joined with the previous one. > joined > > > >> end I've added some fixes to your branch: * Fixed the comment from --- to --. * Fixed some typos. * Moved constant declarations to the beginning of the file. * Join some lines since there is no need to add a new line. =================================================================== diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua index d30198c0..9963bf88 100644 --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -3,6 +3,12 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') local utils = require('utils') local ffi = require('ffi') +local CPU_TYPE_ARM64 = '0x100000c' +local CPU_SUBTYPE_ARM64 = 0x0 +local MH_MAGIC_64 = '0xfeedfacf' +-- Relocatable object file. +local MH_OBJECT = 0x1 + test:plan(1) -- The test creates an object file in Mach-O format with LuaJIT @@ -66,64 +72,64 @@ local function create_obj_file(name, arch) return mach_o_path end --- Parses a buffer in the Mach-O format and returns its fields --- in a table. +-- Parses a buffer in the Mach-O format and returns its fields in +-- a table. -- The test creates an object file in Mach-O format with LuaJIT -- bytecode and checks the validity of the object file fields. ---- ---- The original problem is reproduced with LuaJIT, which is built ---- with enabled AVX512F instructions. The support for AVX512F ---- could be checked in `/proc/cpuinfo` on Linux and ---- `sysctl hw.optional.avx512f` on Mac. AVX512F must be ---- implicitly enabled in a C compiler by passing a CPU codename. ---- Please take a look at the GCC Online Documentation [1] for ---- available CPU codenames. Also, see the Wikipedia for CPUs with ---- AVX-512 support [2]. ---- Execute command below to detect the CPU codename: ---- `gcc -march=native -Q --help=target | grep march`. ---- ---- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html ---- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 ---- ---- Manual steps for reproducing are the following: ---- ---- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build ---- $ cmake --build build --parallel ---- $ echo > test.lua ---- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o ---- $ file test.o ---- empty.o: DOS executable (block device driver) - ---- The format of the Mach-O is described in the document ---- "OS X ABI Mach-O File Format Reference", published by Apple ---- company. The copy of the (now removed) official documentation ---- can be found here [1]. There is a good visual representation ---- of Mach-O format in "Mac OS X Internals" book (pages 67-68) ---- [2] and in the [3]. -- ---- 0x0000000 --------------------------------------- ---- | 0xfeedface MH_MAGIC magic ---- | ------------------------------------ ---- | 0x00000012 CPU_TYPE_POWERPC cputype ---- | ------------------------------------ ---- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype ---- mach_header | ------------------------------------ ---- | 0x00000002 MH_EXECUTE filetype ---- | ------------------------------------ ---- | 0x0000000b 10 load commands ncmds ---- | ------------------------------------ ---- | 0x00000574 1396 bytes sizeofcmds ---- | ------------------------------------ ---- | 0x00000085 DYLDLINK TWOLEVEL flags ---- -------------------------------------- ---- Load commands ---- --------------------------------------- ---- Data ---- --------------------------------------- ---- ---- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference ---- 2. https://reverseengineering.stackexchange.com/a/6357/46029 ---- 3. http://formats.kaitai.io/mach_o/index.html +-- The original problem is reproduced with LuaJIT, which is built +-- with enabled AVX512F instructions. The support for AVX512F +-- could be checked in `/proc/cpuinfo` on Linux and +-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be implicitly +-- enabled in a C compiler by passing a CPU codename. +-- Please take a look at the GCC Online Documentation [1] for +-- available CPU codenames. Also, see the Wikipedia for CPUs with +-- AVX-512 support [2]. +-- Execute the command below to detect the CPU codename: +-- `gcc -march=native -Q --help=target | grep march`. +-- +-- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html +-- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 +-- +-- Manual steps for reproducing are the following: +-- +-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build +-- $ cmake --build build --parallel +-- $ echo > test.lua +-- $ ./luajit -b -o osx -a arm64 test.lua test.o +-- $ file test.o +-- test.o: Mach-O 64-bit arm64 object + +-- The format of the Mach-O is described in the document +-- "OS X ABI Mach-O File Format Reference", published by Apple +-- company. The copy of the (now removed) official documentation +-- can be found here [1]. There is a good visual representation +-- of Mach-O format in "Mac OS X Internals" book (pages 67-68) +-- [2] and in the [3]. +-- +-- 0x0000000 --------------------------------------- +-- | 0xfeedface MH_MAGIC magic +-- | ------------------------------------ +-- | 0x00000012 CPU_TYPE_POWERPC cputype +-- | ------------------------------------ +-- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype +-- mach_header | ------------------------------------ +-- | 0x00000002 MH_EXECUTE filetype +-- | ------------------------------------ +-- | 0x0000000b 10 load commands ncmds +-- | ------------------------------------ +-- | 0x00000574 1396 bytes sizeofcmds +-- | ------------------------------------ +-- | 0x00000085 DYLDLINK TWOLEVEL flags +-- -------------------------------------- +-- Load commands +-- --------------------------------------- +-- Data +-- --------------------------------------- +-- +-- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference +-- 2. https://reverseengineering.stackexchange.com/a/6357/46029 +-- 3. http://formats.kaitai.io/mach_o/index.html local function read_mach_o_hdr(buf, hw_arch) -- LuaJIT always generate 64-bit non-FAT Mach-O object files. assert(hw_arch == 'arm64') @@ -138,11 +144,11 @@ local function read_mach_o_hdr(buf, hw_arch) return mach_header end --- The function builds Mach-O object file and retrieves --- its header fields. +-- The function builds a Mach-O object file and retrieves its +-- header fields. local function build_and_check_mach_o(subtest) local hw_arch = subtest.name - -- LuaJIT always generate 64-bit non-FAT Mach-O object files. + -- LuaJIT always generates 64-bit, non-FAT Mach-O object files. assert(hw_arch == 'arm64') subtest:plan(5) @@ -159,19 +165,12 @@ local function build_and_check_mach_o(subtest) assert(os.remove(mach_o_obj_path), 'remove an object file') local magic_str = string.format('%#x', mach_o.magic) - local MH_MAGIC_64 = '0xfeedfacf' - subtest:is(magic_str, MH_MAGIC_64, - 'magic is correct in Mach-O') + subtest:is(magic_str, MH_MAGIC_64, 'magic is correct in Mach-O') local cputype_str = string.format('%#x', mach_o.cputype) - local CPU_TYPE_ARM64 = '0x100000c' - subtest:is(cputype_str, CPU_TYPE_ARM64, - 'cputype is correct in Mach-O') - local CPU_SUBTYPE_ARM64 = 0 + subtest:is(cputype_str, CPU_TYPE_ARM64, 'cputype is correct in Mach-O') subtest:is(mach_o.cpusubtype, CPU_SUBTYPE_ARM64, 'cpusubtype is correct in Mach-O') - local MH_OBJECT = 1 - subtest:is(mach_o.filetype, MH_OBJECT, - 'filetype is correct in Mach-O') + subtest:is(mach_o.filetype, MH_OBJECT, 'filetype is correct in Mach-O') subtest:is(mach_o.ncmds, 2, 'ncmds is correct in Mach-O') end =================================================================== Please let me know if you have any concerns. -- Best regards, Sergey Kaplun