From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id BEB6BBDCC19; Tue, 9 Jul 2024 16:03:40 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org BEB6BBDCC19 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1720530220; bh=CdU3VUlbqMmAstaMeWK48d7R1vViLrhcKhYuk1d1c94=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=TipKGLmV9mmHp9lRNTTso1AYh/pJDcWS8jvTr/MMW/OqXNITT4J9X+pCdthqdYRro VNZtysDhuD08hX8AEXt9N2tEeTL0RXyRriS16olaeriQLThAcfX/aW8hETdcT6nSlW yDR3z5WKiTPRgy24Zg4NRZfsYh+it7aPeGHchwV8= Received: from smtp61.i.mail.ru (smtp61.i.mail.ru [95.163.41.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 8E034BDCC19 for ; Tue, 9 Jul 2024 16:03:39 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 8E034BDCC19 Received: by exim-smtp-687d8cf49b-f4p6p with esmtpa (envelope-from ) id 1sRAVO-00000000K3F-1Ucg; Tue, 09 Jul 2024 16:03:38 +0300 Date: Tue, 9 Jul 2024 16:03:30 +0300 To: Sergey Bronnikov Message-ID: References: <10ed208fcfacfa4c772f1cebe090595af3452ff3.1720182442.git.sergeyb@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <10ed208fcfacfa4c772f1cebe090595af3452ff3.1720182442.git.sergeyb@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD979CF149FBD7EF921CC5C24F3EC0158AAA1AB214DD89E4F2100894C459B0CD1B9789F553D8E6D094115C513EAE8123D4D9070E45357F6583BE7563BFC05BA8A52AB5D286C5C76880D X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE73A0E02362971E860EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637C218CA8E848E3B7A8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D847FD23AAC4A80EAADD80BF9AACCC59903D8CF9E4C9E05DB3CC7F00164DA146DAFE8445B8C89999728AA50765F7900637F924B32C592EA89F389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC8ED96AA85C75E140D117882F4460429728AD0CFFFB425014E868A13BD56FB6657E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637149D0840703ADBE5EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: 0D63561A33F958A506ABBAE18D75CC695002B1117B3ED6962CFB3FB91A506BD3466072E6821086B3823CB91A9FED034534781492E4B8EEAD003C2D46C52F18F2BDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0ADBF74143AD284FC7177DD89D51EBB7742424CF958EAFF5D571004E42C50DC4CA955A7F0CF078B5EC49A30900B95165D34103A2CD2CFC42C6714EB59C47AED50AD1FFFD40972A115291847D5380046BC3B75B2F9BF183F61801D7E09C32AA3244C02AE54192110C97077DD89D51EBB774219DAABFEAFFBDAA2EA455F16B58544A2557BDE0DD54B3590A5AE236DF995FB59829709634694AABAED6A17656DB59BCAD427812AF56FC65B X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojfJERNbCcGXPR9TVLEwSFkw== X-DA7885C5: 86785375C3749495F255D290C0D534F9A3F9EFF815852C5346392FE2657CC48950C1CF1F02DC42475B1A4C17EAA7BC4BEF2421ABFA55128DAF83EF9164C44C7E X-Mailru-Sender: 689FA8AB762F7393C6D0B12EA33CAA9B9263E770DBFDD27FAD111CB5FDCCEDC1F19F4DEE14F697EDE49D44BB4BD9522A059A1ED8796F048DB274557F927329BE89D5A3BC2B10C37545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi, Sergey! Thanks for the patch! Please consider my comments below. May we add the test [1] to verify that there will be no regression in the future? On 05.07.24, Sergey Bronnikov wrote: > Reported by Sergey Bronnikov. > > (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) > > Previously, LuaJIT generated Mach-O FAT object files for ARM and > ARM64 on macOS. The patch removes support of 32-bit ARM and > FAT object files and now LuaJIT generate Mach-O object files for > ARM64. I suppose we should mention that no x86/x86_64 objects are generated now. > > Sergey Bronnikov: > * added the description and the trimmed the test for the problem > > Part of tarantool/tarantool#10199 > --- > src/jit/bcsave.lua | 155 ++------- > ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- > 2 files changed, 70 insertions(+), 379 deletions(-) > > diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua > index 26ec29c6..61953c2d 100644 > --- a/src/jit/bcsave.lua > +++ b/src/jit/bcsave.lua > diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > index f008f3bd..6a58de95 100644 > --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > @@ -3,109 +3,11 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') > local utils = require('utils') > local ffi = require('ffi') > > -test:plan(2) > +test:plan(1) > > -- The test creates an object file in Mach-O format with LuaJIT > -- bytecode and checks the validity of the object file fields. > --- > --- The original problem is reproduced with LuaJIT, which is built > --- with enabled AVX512F instructions. The support for AVX512F > --- could be checked in `/proc/cpuinfo` on Linux and > --- `sysctl hw.optional.avx512f` on Mac. AVX512F must be > --- implicitly enabled in a C compiler by passing a CPU codename. > --- Please take a look at the GCC Online Documentation [1] for > --- available CPU codenames. Also, see the Wikipedia for CPUs with > --- AVX-512 support [2]. > --- Execute command below to detect the CPU codename: > --- `gcc -march=native -Q --help=target | grep march`. > --- > --- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html > --- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 > --- > --- Manual steps for reproducing are the following: > --- > --- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build > --- $ cmake --build build --parallel > --- $ echo > test.lua > --- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o > --- $ file test.o > --- empty.o: DOS executable (block device driver) > > --- LuaJIT can generate so called Universal Binary with Lua > --- bytecode. The Universal Binary format is a format for > --- executable files that run natively on hardware platforms with > --- different hardware architectures. This concept is more > --- generally known as a fat binary. > --- > --- The format of the Mach-O is described in the document > --- "OS X ABI Mach-O File Format Reference", published by Apple > --- company. The copy of the (now removed) official documentation > --- can be found here [1]. Yet another source of truth is > --- XNU headers, see the definition of C-structures in: > --- [2] (`nlist_64`), [3] (`fat_arch` and `fat_header`). > --- > --- There is a good visual representation of Universal Binary > --- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6]. > --- Below is the schematic structure of Universal Binary, which > --- includes two executables for PowerPC and Intel i386 (omitted): > --- > --- 0x0000000 --------------------------------------- > --- | > --- struct | 0xcafebabe FAT_MAGIC magic > --- fat_header | ------------------------------------- > --- | 0x00000003 nfat_arch > --- --------------------------------------- > --- | 0x00000012 CPU_TYPE_POWERPC cputype > --- | ------------------------------------- > --- | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > --- struct | ------------------------------------- > --- fat_arch | 0x00001000 4096 bytes offset > --- | ------------------------------------- > --- | 0x00004224 16932 bytes size > --- | ------------------------------------- > --- | 0x0000000c 2^12 = 4096 bytes align > --- --------------------------------------- > --- --------------------------------------- > --- | 0x00000007 CPU_TYPE_I386 cputype > --- | ------------------------------------- > --- | 0x00000003 CPU_SUBTYPE_I386_ALL cpusubtype > --- struct | ------------------------------------- > --- fat_arch | 0x00006000 24576 bytes offset > --- | ------------------------------------- > --- | 0x0000292c 10540 bytes size > --- | ------------------------------------- > --- | 0x0000000c 2^12 = 4096 bytes align > --- --------------------------------------- > --- Unused > --- 0x00001000 --------------------------------------- > --- | 0xfeedface MH_MAGIC magic > --- | ------------------------------------ > --- | 0x00000012 CPU_TYPE_POWERPC cputype > --- | ------------------------------------ > --- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > --- mach_header | ------------------------------------ > --- | 0x00000002 MH_EXECUTE filetype > --- | ------------------------------------ > --- | 0x0000000b 10 load commands ncmds > --- | ------------------------------------ > --- | 0x00000574 1396 bytes sizeofcmds > --- | ------------------------------------ > --- | 0x00000085 DYLDLINK TWOLEVEL flags > --- -------------------------------------- > --- Load commands > --- --------------------------------------- > --- Data > --- --------------------------------------- > --- > --- < x86 executable > > --- > --- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference > --- 2. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/nlist.h > --- 3. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/fat.h > --- 4. https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code > --- 5. https://reverseengineering.stackexchange.com/a/6357/46029 > --- 6. http://formats.kaitai.io/mach_o/index.html I prefer to keep the non-FAT part of this comment since it is very useful. > --- > -- Using the same declarations as defined in . > ffi.cdef[[ > local function create_obj_file(name, arch) > @@ -212,108 +66,37 @@ local function create_obj_file(name, arch) > return mach_o_path > end > > --- Parses a buffer in the Mach-O format and returns the FAT magic > --- number and `nfat_arch`. > +-- Parses a buffer in the Mach-O format and returns its fields > +-- in a table. > local function read_mach_o(buf, hw_arch) I suggest renaming it to `read_mach_o_hdr()` and returning only the header without any additional wrapping in the table. > - local res = { > - header = { > - magic = 0, > - nfat_arch = 0, > - }, > - fat_arch = {}, > - } > - > local is64 = hw_arch == 'arm64' Maybe it is better to use assert here, like we do for the `build_and_check_mach_o()` routine? > > - -- Mach-O FAT object. > - local mach_fat_obj_type = ffi.typeof(is64 and > - 'mach_fat_obj_64 *' or > - 'mach_fat_obj *') > - local obj = ffi.cast(mach_fat_obj_type, buf) > + -- Mach-O object. > + local mach_obj_type = ffi.typeof(is64 and 'mach_obj_64 *') Maybe just use mach_obj_64 since there is no alternative? > + local obj = ffi.cast(mach_obj_type, buf) > > - -- Mach-O FAT object header. > - local mach_fat_header = obj.fat > - -- Mach-O FAT is BE, target arch is LE. > - local be32 = bit.bswap > - res.header.magic = be32(mach_fat_header.magic) > - res.header.nfat_arch = be32(mach_fat_header.nfat_arch) > + -- Mach-O object header. > + local mach_header = obj.hdr > > - -- Mach-O FAT object arches. > - for i = 0, res.header.nfat_arch - 1 do > - local fat_arch = obj.fat_arch[i] > - local arch = { > - cputype = be32(fat_arch.cputype), > - cpusubtype = be32(fat_arch.cpusubtype), > - } > - table.insert(res.fat_arch, arch) > - end > - > - return res > + return { > + header = { > + magic = mach_header.magic, > + cputype = mach_header.cputype, > + cpusubtype = mach_header.cpusubtype, > + filetype = mach_header.filetype, > + ncmds = mach_header.ncmds, > + }, > + } > end > > --- Universal Binary can contain executables for more than one > --- CPU architecture. For simplicity, the test compares the *sum* > --- of CPU types and CPU subtypes. > --- > --- has the definitions of the > --- numbers below. The original XNU source code may be found in > --- [1]. > --- > --- 1. https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/mach/machine.h.auto.html > --- > -local SUM_CPUTYPE = { > - -- x86 + arm. > - arm = 7 + 12, > - -- x64 + arm64. > - arm64 = 0x01000007 + 0x0100000c, > -} > -local SUM_CPUSUBTYPE = { > - -- x86 + arm. > - arm = 3 + 9, > - -- x64 + arm64. > - arm64 = 3 + 0, > -} > - > --- The function builds Mach-O FAT object file and retrieves > --- its header fields (magic and nfat_arch) and fields of each arch > --- (cputype, cpusubtype). > --- > --- The Mach-O FAT object header can be retrieved with `otool` on > --- macOS: > --- > --- $ otool -f empty.o > --- Fat headers > --- fat_magic 0xcafebabe > --- nfat_arch 2 > --- > --- > --- CPU type and subtype can be retrieved with `lipo` on macOS: > --- > --- $ luajit -b -o osx -a arm empty.lua empty.o > --- $ lipo -archs empty.o > --- i386 armv7 > --- $ luajit -b -o osx -a arm64 empty.lua empty.o > --- $ lipo -archs empty.o > --- x86_64 arm64 > +-- The function builds Mach-O object file and retrieves > +-- its header fields. > local function build_and_check_mach_o(subtest) > local hw_arch = subtest.name > - assert(hw_arch == 'arm' or hw_arch == 'arm64') > + -- LuaJIT always generate 64-bit non-FAT Mach-O object files. The fact that these files are generated only for M1 CPUs looks worth mentioning. > + assert(hw_arch == 'arm64') > > - subtest:plan(4) > - -- FAT_MAGIC is an integer containing the value 0xCAFEBABE in > - -- big-endian byte order format. On a big-endian host CPU, > - -- this can be validated using the constant FAT_MAGIC; > - -- on a little-endian host CPU, it can be validated using > - -- the constant FAT_CIGAM. > - -- > - -- FAT_NARCH is an integer specifying the number of fat_arch > - -- data structures that follow. This is the number of > - -- architectures contained in this binary. > - -- > - -- See the aforementioned "OS X ABI Mach-O File Format > - -- Reference". The similar comment about Mach-O format will be appretiated. > - local FAT_MAGIC = '0xffffffffcafebabe' > - local FAT_NARCH = 2 > + subtest:plan(5) > > local MODULE_NAME = 'lango_team' > > @@ -327,24 +110,19 @@ local function build_and_check_mach_o(subtest) > assert(os.remove(mach_o_obj_path), 'remove an object file') > > local magic_str = string.format('%#x', mach_o.header.magic) > - subtest:is(magic_str, FAT_MAGIC, > - 'fat_magic is correct in Mach-O') > - subtest:is(mach_o.header.nfat_arch, FAT_NARCH, > - 'nfat_arch is correct in Mach-O') > - > - local total_cputype = 0 > - local total_cpusubtype = 0 > - for i = 1, FAT_NARCH do > - total_cputype = total_cputype + mach_o.fat_arch[i].cputype > - total_cpusubtype = total_cpusubtype + mach_o.fat_arch[i].cpusubtype > - end > - subtest:is(total_cputype, SUM_CPUTYPE[hw_arch], > + subtest:is(magic_str, '0xfeedfacf', Please use MH_MAGIC_64 named constant for this magic string. > + 'magic is correct in Mach-O') Looks like this line may be joined with the previous one. > + local cputype_str = string.format('%#x', mach_o.header.cputype) > + subtest:is(cputype_str, '0x100000c', Please use the named constant CPU_TYPE_ARM64 for this magic string. > 'cputype is correct in Mach-O') Looks like this line may be joined with the previous one. > - subtest:is(total_cpusubtype, SUM_CPUSUBTYPE[hw_arch], > + subtest:is(mach_o.header.cpusubtype, 0, Please use the named constant CPU_SUBTYPE_ARM64 for this magic constant. > 'cpusubtype is correct in Mach-O') Looks like this line may be joined with the previous one. > + subtest:is(mach_o.header.filetype, 1, What does the 1 filetype mean? Please use the named constant. > + 'filetype is correct in Mach-O') Looks like this line may be joined with the previous one. > + subtest:is(mach_o.header.ncmds, 2, Why there are 2 commands for Mach-O format? Please use the named constant. > + 'ncmds is correct in Mach-O') Looks like this line may be joined with the previous one. > end > > -test:test('arm', build_and_check_mach_o) > test:test('arm64', build_and_check_mach_o) > > test:done(true) > -- > 2.34.1 > [1]: https://github.com/LuaJIT/LuaJIT/issues/1181#issue-2202788411 -- Best regards, Sergey Kaplun