* [Tarantool-patches] [PATCH luajit 0/2] Fixes in Mach-O generation code @ 2024-07-05 13:10 Sergey Bronnikov via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 1/2] Fix typo Sergey Bronnikov via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files Sergey Bronnikov via Tarantool-patches 0 siblings, 2 replies; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-07-05 13:10 UTC (permalink / raw) To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin From: Sergey Bronnikov <sergeyb@tarantool.org> Patch series brings two patches with fixes in <jit/bcsave.lua>. Branch: https://github.com/tarantool/luajit/tree/ligurio/gh-1181-64bit-non-FAT-Mach-O-object-files Issues: - https://github.com/tarantool/tarantool/issues/10199 - https://github.com/LuaJIT/LuaJIT/issues/1181 - https://github.com/LuaJIT/LuaJIT/issues/1223 Mike Pall (2): Fix typo. OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. src/jit/bcsave.lua | 157 +++------- ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- 2 files changed, 71 insertions(+), 380 deletions(-) -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Tarantool-patches] [PATCH luajit 1/2] Fix typo. 2024-07-05 13:10 [Tarantool-patches] [PATCH luajit 0/2] Fixes in Mach-O generation code Sergey Bronnikov via Tarantool-patches @ 2024-07-05 13:10 ` Sergey Bronnikov via Tarantool-patches 2024-07-09 12:15 ` Sergey Kaplun via Tarantool-patches 2024-07-11 9:15 ` Maxim Kokryashkin via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files Sergey Bronnikov via Tarantool-patches 1 sibling, 2 replies; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-07-05 13:10 UTC (permalink / raw) To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin Reported by Sergey Bronnikov. (cherry picked from commit 0f8b878e2f3ea280648fc04309c5527cf261bfc8) Part of tarantool/tarantool#10199 --- src/jit/bcsave.lua | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua index 069cf1a3..26ec29c6 100644 --- a/src/jit/bcsave.lua +++ b/src/jit/bcsave.lua @@ -571,7 +571,7 @@ typedef struct { o.sym_entry.strx = 1 ffi.copy(o.space+1, symname) - -- Write Macho-O object file. + -- Write Mach-O object file. local fp = savefile(output, "wb") fp:write(ffi.string(o, mach_size)) bcsave_tail(fp, output, s) -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 1/2] Fix typo. 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 1/2] Fix typo Sergey Bronnikov via Tarantool-patches @ 2024-07-09 12:15 ` Sergey Kaplun via Tarantool-patches 2024-07-11 9:15 ` Maxim Kokryashkin via Tarantool-patches 1 sibling, 0 replies; 11+ messages in thread From: Sergey Kaplun via Tarantool-patches @ 2024-07-09 12:15 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: tarantool-patches Hi, Sergey! Thanks for the patch! LGTM! -- Best regards, Sergey Kaplun ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 1/2] Fix typo. 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 1/2] Fix typo Sergey Bronnikov via Tarantool-patches 2024-07-09 12:15 ` Sergey Kaplun via Tarantool-patches @ 2024-07-11 9:15 ` Maxim Kokryashkin via Tarantool-patches 1 sibling, 0 replies; 11+ messages in thread From: Maxim Kokryashkin via Tarantool-patches @ 2024-07-11 9:15 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: tarantool-patches Hi, Sergey! Thanks for the patch! LGTM ^ permalink raw reply [flat|nested] 11+ messages in thread
* [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-05 13:10 [Tarantool-patches] [PATCH luajit 0/2] Fixes in Mach-O generation code Sergey Bronnikov via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 1/2] Fix typo Sergey Bronnikov via Tarantool-patches @ 2024-07-05 13:10 ` Sergey Bronnikov via Tarantool-patches 2024-07-09 13:03 ` Sergey Kaplun via Tarantool-patches 1 sibling, 1 reply; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-07-05 13:10 UTC (permalink / raw) To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin Reported by Sergey Bronnikov. (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) Previously, LuaJIT generated Mach-O FAT object files for ARM and ARM64 on macOS. The patch removes support of 32-bit ARM and FAT object files and now LuaJIT generate Mach-O object files for ARM64. Sergey Bronnikov: * added the description and the trimmed the test for the problem Part of tarantool/tarantool#10199 --- src/jit/bcsave.lua | 155 ++------- ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- 2 files changed, 70 insertions(+), 379 deletions(-) diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua index 26ec29c6..61953c2d 100644 --- a/src/jit/bcsave.lua +++ b/src/jit/bcsave.lua @@ -412,24 +412,12 @@ typedef struct { mach_header; uint32_t reserved; } mach_header_64; -typedef struct { - uint32_t cmd, cmdsize; - char segname[16]; - uint32_t vmaddr, vmsize, fileoff, filesize; - uint32_t maxprot, initprot, nsects, flags; -} mach_segment_command; typedef struct { uint32_t cmd, cmdsize; char segname[16]; uint64_t vmaddr, vmsize, fileoff, filesize; uint32_t maxprot, initprot, nsects, flags; } mach_segment_command_64; -typedef struct { - char sectname[16], segname[16]; - uint32_t addr, size; - uint32_t offset, align, reloff, nreloc, flags; - uint32_t reserved1, reserved2; -} mach_section; typedef struct { char sectname[16], segname[16]; uint64_t addr, size; @@ -439,133 +427,58 @@ typedef struct { typedef struct { uint32_t cmd, cmdsize, symoff, nsyms, stroff, strsize; } mach_symtab_command; -typedef struct { - int32_t strx; - uint8_t type, sect; - int16_t desc; - uint32_t value; -} mach_nlist; typedef struct { int32_t strx; uint8_t type, sect; uint16_t desc; uint64_t value; } mach_nlist_64; -typedef struct -{ - int32_t magic, nfat_arch; -} mach_fat_header; -typedef struct -{ - int32_t cputype, cpusubtype, offset, size, align; -} mach_fat_arch; -typedef struct { - struct { - mach_header hdr; - mach_segment_command seg; - mach_section sec; - mach_symtab_command sym; - } arch[1]; - mach_nlist sym_entry; - uint8_t space[4096]; -} mach_obj; typedef struct { - struct { - mach_header_64 hdr; - mach_segment_command_64 seg; - mach_section_64 sec; - mach_symtab_command sym; - } arch[1]; + mach_header_64 hdr; + mach_segment_command_64 seg; + mach_section_64 sec; + mach_symtab_command sym; mach_nlist_64 sym_entry; uint8_t space[4096]; } mach_obj_64; -typedef struct { - mach_fat_header fat; - mach_fat_arch fat_arch[2]; - struct { - mach_header hdr; - mach_segment_command seg; - mach_section sec; - mach_symtab_command sym; - } arch[2]; - mach_nlist sym_entry; - uint8_t space[4096]; -} mach_fat_obj; -typedef struct { - mach_fat_header fat; - mach_fat_arch fat_arch[2]; - struct { - mach_header_64 hdr; - mach_segment_command_64 seg; - mach_section_64 sec; - mach_symtab_command sym; - } arch[2]; - mach_nlist_64 sym_entry; - uint8_t space[4096]; -} mach_fat_obj_64; ]] local symname = '_'..LJBC_PREFIX..ctx.modname - local isfat, is64, align, mobj = false, false, 4, "mach_obj" - if ctx.arch == "x64" then - is64, align, mobj = true, 8, "mach_obj_64" - elseif ctx.arch == "arm" then - isfat, mobj = true, "mach_fat_obj" - elseif ctx.arch == "arm64" then - is64, align, isfat, mobj = true, 8, true, "mach_fat_obj_64" - else - check(ctx.arch == "x86", "unsupported architecture for OSX") + local cputype, cpusubtype = 0x01000007, 3 + if ctx.arch ~= "x64" then + check(ctx.arch == "arm64", "unsupported architecture for OSX") + cputype, cpusubtype = 0x0100000c, 0 end local function aligned(v, a) return bit.band(v+a-1, -a) end - local be32 = bit.bswap -- Mach-O FAT is BE, supported archs are LE. -- Create Mach-O object and fill in header. - local o = ffi.new(mobj) - local mach_size = aligned(ffi.offsetof(o, "space")+#symname+2, align) - local cputype = ({ x86={7}, x64={0x01000007}, arm={7,12}, arm64={0x01000007,0x0100000c} })[ctx.arch] - local cpusubtype = ({ x86={3}, x64={3}, arm={3,9}, arm64={3,0} })[ctx.arch] - if isfat then - o.fat.magic = be32(0xcafebabe) - o.fat.nfat_arch = be32(#cpusubtype) - end + local o = ffi.new("mach_obj_64") + local mach_size = aligned(ffi.offsetof(o, "space")+#symname+2, 8) -- Fill in sections and symbols. - for i=0,#cpusubtype-1 do - local ofs = 0 - if isfat then - local a = o.fat_arch[i] - a.cputype = be32(cputype[i+1]) - a.cpusubtype = be32(cpusubtype[i+1]) - -- Subsequent slices overlap each other to share data. - ofs = ffi.offsetof(o, "arch") + i*ffi.sizeof(o.arch[0]) - a.offset = be32(ofs) - a.size = be32(mach_size-ofs+#s) - end - local a = o.arch[i] - a.hdr.magic = is64 and 0xfeedfacf or 0xfeedface - a.hdr.cputype = cputype[i+1] - a.hdr.cpusubtype = cpusubtype[i+1] - a.hdr.filetype = 1 - a.hdr.ncmds = 2 - a.hdr.sizeofcmds = ffi.sizeof(a.seg)+ffi.sizeof(a.sec)+ffi.sizeof(a.sym) - a.seg.cmd = is64 and 0x19 or 0x1 - a.seg.cmdsize = ffi.sizeof(a.seg)+ffi.sizeof(a.sec) - a.seg.vmsize = #s - a.seg.fileoff = mach_size-ofs - a.seg.filesize = #s - a.seg.maxprot = 1 - a.seg.initprot = 1 - a.seg.nsects = 1 - ffi.copy(a.sec.sectname, "__data") - ffi.copy(a.sec.segname, "__DATA") - a.sec.size = #s - a.sec.offset = mach_size-ofs - a.sym.cmd = 2 - a.sym.cmdsize = ffi.sizeof(a.sym) - a.sym.symoff = ffi.offsetof(o, "sym_entry")-ofs - a.sym.nsyms = 1 - a.sym.stroff = ffi.offsetof(o, "sym_entry")+ffi.sizeof(o.sym_entry)-ofs - a.sym.strsize = aligned(#symname+2, align) - end + o.hdr.magic = 0xfeedfacf + o.hdr.cputype = cputype + o.hdr.cpusubtype = cpusubtype + o.hdr.filetype = 1 + o.hdr.ncmds = 2 + o.hdr.sizeofcmds = ffi.sizeof(o.seg)+ffi.sizeof(o.sec)+ffi.sizeof(o.sym) + o.seg.cmd = 0x19 + o.seg.cmdsize = ffi.sizeof(o.seg)+ffi.sizeof(o.sec) + o.seg.vmsize = #s + o.seg.fileoff = mach_size + o.seg.filesize = #s + o.seg.maxprot = 1 + o.seg.initprot = 1 + o.seg.nsects = 1 + ffi.copy(o.sec.sectname, "__data") + ffi.copy(o.sec.segname, "__DATA") + o.sec.size = #s + o.sec.offset = mach_size + o.sym.cmd = 2 + o.sym.cmdsize = ffi.sizeof(o.sym) + o.sym.symoff = ffi.offsetof(o, "sym_entry") + o.sym.nsyms = 1 + o.sym.stroff = ffi.offsetof(o, "sym_entry")+ffi.sizeof(o.sym_entry) + o.sym.strsize = aligned(#symname+2, 8) o.sym_entry.type = 0xf o.sym_entry.sect = 1 o.sym_entry.strx = 1 diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua index f008f3bd..6a58de95 100644 --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -3,109 +3,11 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') local utils = require('utils') local ffi = require('ffi') -test:plan(2) +test:plan(1) -- The test creates an object file in Mach-O format with LuaJIT -- bytecode and checks the validity of the object file fields. --- --- The original problem is reproduced with LuaJIT, which is built --- with enabled AVX512F instructions. The support for AVX512F --- could be checked in `/proc/cpuinfo` on Linux and --- `sysctl hw.optional.avx512f` on Mac. AVX512F must be --- implicitly enabled in a C compiler by passing a CPU codename. --- Please take a look at the GCC Online Documentation [1] for --- available CPU codenames. Also, see the Wikipedia for CPUs with --- AVX-512 support [2]. --- Execute command below to detect the CPU codename: --- `gcc -march=native -Q --help=target | grep march`. --- --- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html --- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 --- --- Manual steps for reproducing are the following: --- --- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build --- $ cmake --build build --parallel --- $ echo > test.lua --- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o --- $ file test.o --- empty.o: DOS executable (block device driver) --- LuaJIT can generate so called Universal Binary with Lua --- bytecode. The Universal Binary format is a format for --- executable files that run natively on hardware platforms with --- different hardware architectures. This concept is more --- generally known as a fat binary. --- --- The format of the Mach-O is described in the document --- "OS X ABI Mach-O File Format Reference", published by Apple --- company. The copy of the (now removed) official documentation --- can be found here [1]. Yet another source of truth is --- XNU headers, see the definition of C-structures in: --- [2] (`nlist_64`), [3] (`fat_arch` and `fat_header`). --- --- There is a good visual representation of Universal Binary --- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6]. --- Below is the schematic structure of Universal Binary, which --- includes two executables for PowerPC and Intel i386 (omitted): --- --- 0x0000000 --------------------------------------- --- | --- struct | 0xcafebabe FAT_MAGIC magic --- fat_header | ------------------------------------- --- | 0x00000003 nfat_arch --- --------------------------------------- --- | 0x00000012 CPU_TYPE_POWERPC cputype --- | ------------------------------------- --- | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype --- struct | ------------------------------------- --- fat_arch | 0x00001000 4096 bytes offset --- | ------------------------------------- --- | 0x00004224 16932 bytes size --- | ------------------------------------- --- | 0x0000000c 2^12 = 4096 bytes align --- --------------------------------------- --- --------------------------------------- --- | 0x00000007 CPU_TYPE_I386 cputype --- | ------------------------------------- --- | 0x00000003 CPU_SUBTYPE_I386_ALL cpusubtype --- struct | ------------------------------------- --- fat_arch | 0x00006000 24576 bytes offset --- | ------------------------------------- --- | 0x0000292c 10540 bytes size --- | ------------------------------------- --- | 0x0000000c 2^12 = 4096 bytes align --- --------------------------------------- --- Unused --- 0x00001000 --------------------------------------- --- | 0xfeedface MH_MAGIC magic --- | ------------------------------------ --- | 0x00000012 CPU_TYPE_POWERPC cputype --- | ------------------------------------ --- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype --- mach_header | ------------------------------------ --- | 0x00000002 MH_EXECUTE filetype --- | ------------------------------------ --- | 0x0000000b 10 load commands ncmds --- | ------------------------------------ --- | 0x00000574 1396 bytes sizeofcmds --- | ------------------------------------ --- | 0x00000085 DYLDLINK TWOLEVEL flags --- -------------------------------------- --- Load commands --- --------------------------------------- --- Data --- --------------------------------------- --- --- < x86 executable > --- --- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference --- 2. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/nlist.h --- 3. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/fat.h --- 4. https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code --- 5. https://reverseengineering.stackexchange.com/a/6357/46029 --- 6. http://formats.kaitai.io/mach_o/index.html --- -- Using the same declarations as defined in <src/jit/bcsave.lua>. ffi.cdef[[ typedef struct @@ -118,13 +20,6 @@ typedef struct mach_header; uint32_t reserved; } mach_header_64; -typedef struct { - uint32_t cmd, cmdsize; - char segname[16]; - uint32_t vmaddr, vmsize, fileoff, filesize; - uint32_t maxprot, initprot, nsects, flags; -} mach_segment_command; - typedef struct { uint32_t cmd, cmdsize; char segname[16]; @@ -132,13 +27,6 @@ typedef struct { uint32_t maxprot, initprot, nsects, flags; } mach_segment_command_64; -typedef struct { - char sectname[16], segname[16]; - uint32_t addr, size; - uint32_t offset, align, reloff, nreloc, flags; - uint32_t reserved1, reserved2; -} mach_section; - typedef struct { char sectname[16], segname[16]; uint64_t addr, size; @@ -150,13 +38,6 @@ typedef struct { uint32_t cmd, cmdsize, symoff, nsyms, stroff, strsize; } mach_symtab_command; -typedef struct { - int32_t strx; - uint8_t type, sect; - int16_t desc; - uint32_t value; -} mach_nlist; - typedef struct { int32_t strx; uint8_t type, sect; @@ -164,41 +45,14 @@ typedef struct { uint64_t value; } mach_nlist_64; -typedef struct -{ - int32_t magic, nfat_arch; -} mach_fat_header; - -typedef struct -{ - int32_t cputype, cpusubtype, offset, size, align; -} mach_fat_arch; - -typedef struct { - mach_fat_header fat; - mach_fat_arch fat_arch[2]; - struct { - mach_header hdr; - mach_segment_command seg; - mach_section sec; - mach_symtab_command sym; - } arch[2]; - mach_nlist sym_entry; - uint8_t space[4096]; -} mach_fat_obj; - typedef struct { - mach_fat_header fat; - mach_fat_arch fat_arch[2]; - struct { - mach_header_64 hdr; - mach_segment_command_64 seg; - mach_section_64 sec; - mach_symtab_command sym; - } arch[2]; + mach_header_64 hdr; + mach_segment_command_64 seg; + mach_section_64 sec; + mach_symtab_command sym; mach_nlist_64 sym_entry; uint8_t space[4096]; -} mach_fat_obj_64; +} mach_obj_64; ]] local function create_obj_file(name, arch) @@ -212,108 +66,37 @@ local function create_obj_file(name, arch) return mach_o_path end --- Parses a buffer in the Mach-O format and returns the FAT magic --- number and `nfat_arch`. +-- Parses a buffer in the Mach-O format and returns its fields +-- in a table. local function read_mach_o(buf, hw_arch) - local res = { - header = { - magic = 0, - nfat_arch = 0, - }, - fat_arch = {}, - } - local is64 = hw_arch == 'arm64' - -- Mach-O FAT object. - local mach_fat_obj_type = ffi.typeof(is64 and - 'mach_fat_obj_64 *' or - 'mach_fat_obj *') - local obj = ffi.cast(mach_fat_obj_type, buf) + -- Mach-O object. + local mach_obj_type = ffi.typeof(is64 and 'mach_obj_64 *') + local obj = ffi.cast(mach_obj_type, buf) - -- Mach-O FAT object header. - local mach_fat_header = obj.fat - -- Mach-O FAT is BE, target arch is LE. - local be32 = bit.bswap - res.header.magic = be32(mach_fat_header.magic) - res.header.nfat_arch = be32(mach_fat_header.nfat_arch) + -- Mach-O object header. + local mach_header = obj.hdr - -- Mach-O FAT object arches. - for i = 0, res.header.nfat_arch - 1 do - local fat_arch = obj.fat_arch[i] - local arch = { - cputype = be32(fat_arch.cputype), - cpusubtype = be32(fat_arch.cpusubtype), - } - table.insert(res.fat_arch, arch) - end - - return res + return { + header = { + magic = mach_header.magic, + cputype = mach_header.cputype, + cpusubtype = mach_header.cpusubtype, + filetype = mach_header.filetype, + ncmds = mach_header.ncmds, + }, + } end --- Universal Binary can contain executables for more than one --- CPU architecture. For simplicity, the test compares the *sum* --- of CPU types and CPU subtypes. --- --- <src/jit/bcsave.lua:bcsave_machobj> has the definitions of the --- numbers below. The original XNU source code may be found in --- <osfmk/mach/machine.h> [1]. --- --- 1. https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/mach/machine.h.auto.html --- -local SUM_CPUTYPE = { - -- x86 + arm. - arm = 7 + 12, - -- x64 + arm64. - arm64 = 0x01000007 + 0x0100000c, -} -local SUM_CPUSUBTYPE = { - -- x86 + arm. - arm = 3 + 9, - -- x64 + arm64. - arm64 = 3 + 0, -} - --- The function builds Mach-O FAT object file and retrieves --- its header fields (magic and nfat_arch) and fields of each arch --- (cputype, cpusubtype). --- --- The Mach-O FAT object header can be retrieved with `otool` on --- macOS: --- --- $ otool -f empty.o --- Fat headers --- fat_magic 0xcafebabe --- nfat_arch 2 --- <snipped> --- --- CPU type and subtype can be retrieved with `lipo` on macOS: --- --- $ luajit -b -o osx -a arm empty.lua empty.o --- $ lipo -archs empty.o --- i386 armv7 --- $ luajit -b -o osx -a arm64 empty.lua empty.o --- $ lipo -archs empty.o --- x86_64 arm64 +-- The function builds Mach-O object file and retrieves +-- its header fields. local function build_and_check_mach_o(subtest) local hw_arch = subtest.name - assert(hw_arch == 'arm' or hw_arch == 'arm64') + -- LuaJIT always generate 64-bit non-FAT Mach-O object files. + assert(hw_arch == 'arm64') - subtest:plan(4) - -- FAT_MAGIC is an integer containing the value 0xCAFEBABE in - -- big-endian byte order format. On a big-endian host CPU, - -- this can be validated using the constant FAT_MAGIC; - -- on a little-endian host CPU, it can be validated using - -- the constant FAT_CIGAM. - -- - -- FAT_NARCH is an integer specifying the number of fat_arch - -- data structures that follow. This is the number of - -- architectures contained in this binary. - -- - -- See the aforementioned "OS X ABI Mach-O File Format - -- Reference". - local FAT_MAGIC = '0xffffffffcafebabe' - local FAT_NARCH = 2 + subtest:plan(5) local MODULE_NAME = 'lango_team' @@ -327,24 +110,19 @@ local function build_and_check_mach_o(subtest) assert(os.remove(mach_o_obj_path), 'remove an object file') local magic_str = string.format('%#x', mach_o.header.magic) - subtest:is(magic_str, FAT_MAGIC, - 'fat_magic is correct in Mach-O') - subtest:is(mach_o.header.nfat_arch, FAT_NARCH, - 'nfat_arch is correct in Mach-O') - - local total_cputype = 0 - local total_cpusubtype = 0 - for i = 1, FAT_NARCH do - total_cputype = total_cputype + mach_o.fat_arch[i].cputype - total_cpusubtype = total_cpusubtype + mach_o.fat_arch[i].cpusubtype - end - subtest:is(total_cputype, SUM_CPUTYPE[hw_arch], + subtest:is(magic_str, '0xfeedfacf', + 'magic is correct in Mach-O') + local cputype_str = string.format('%#x', mach_o.header.cputype) + subtest:is(cputype_str, '0x100000c', 'cputype is correct in Mach-O') - subtest:is(total_cpusubtype, SUM_CPUSUBTYPE[hw_arch], + subtest:is(mach_o.header.cpusubtype, 0, 'cpusubtype is correct in Mach-O') + subtest:is(mach_o.header.filetype, 1, + 'filetype is correct in Mach-O') + subtest:is(mach_o.header.ncmds, 2, + 'ncmds is correct in Mach-O') end -test:test('arm', build_and_check_mach_o) test:test('arm64', build_and_check_mach_o) test:done(true) -- 2.34.1 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files Sergey Bronnikov via Tarantool-patches @ 2024-07-09 13:03 ` Sergey Kaplun via Tarantool-patches 2024-07-10 12:43 ` Sergey Bronnikov via Tarantool-patches 0 siblings, 1 reply; 11+ messages in thread From: Sergey Kaplun via Tarantool-patches @ 2024-07-09 13:03 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: tarantool-patches Hi, Sergey! Thanks for the patch! Please consider my comments below. May we add the test [1] to verify that there will be no regression in the future? On 05.07.24, Sergey Bronnikov wrote: > Reported by Sergey Bronnikov. > > (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) > > Previously, LuaJIT generated Mach-O FAT object files for ARM and > ARM64 on macOS. The patch removes support of 32-bit ARM and > FAT object files and now LuaJIT generate Mach-O object files for > ARM64. I suppose we should mention that no x86/x86_64 objects are generated now. > > Sergey Bronnikov: > * added the description and the trimmed the test for the problem > > Part of tarantool/tarantool#10199 > --- > src/jit/bcsave.lua | 155 ++------- > ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- > 2 files changed, 70 insertions(+), 379 deletions(-) > > diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua > index 26ec29c6..61953c2d 100644 > --- a/src/jit/bcsave.lua > +++ b/src/jit/bcsave.lua <snipped> > diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > index f008f3bd..6a58de95 100644 > --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > @@ -3,109 +3,11 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') > local utils = require('utils') > local ffi = require('ffi') > > -test:plan(2) > +test:plan(1) > > -- The test creates an object file in Mach-O format with LuaJIT > -- bytecode and checks the validity of the object file fields. > --- > --- The original problem is reproduced with LuaJIT, which is built > --- with enabled AVX512F instructions. The support for AVX512F > --- could be checked in `/proc/cpuinfo` on Linux and > --- `sysctl hw.optional.avx512f` on Mac. AVX512F must be > --- implicitly enabled in a C compiler by passing a CPU codename. > --- Please take a look at the GCC Online Documentation [1] for > --- available CPU codenames. Also, see the Wikipedia for CPUs with > --- AVX-512 support [2]. > --- Execute command below to detect the CPU codename: > --- `gcc -march=native -Q --help=target | grep march`. > --- > --- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html > --- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 > --- > --- Manual steps for reproducing are the following: > --- > --- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build > --- $ cmake --build build --parallel > --- $ echo > test.lua > --- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o > --- $ file test.o > --- empty.o: DOS executable (block device driver) > > --- LuaJIT can generate so called Universal Binary with Lua > --- bytecode. The Universal Binary format is a format for > --- executable files that run natively on hardware platforms with > --- different hardware architectures. This concept is more > --- generally known as a fat binary. > --- > --- The format of the Mach-O is described in the document > --- "OS X ABI Mach-O File Format Reference", published by Apple > --- company. The copy of the (now removed) official documentation > --- can be found here [1]. Yet another source of truth is > --- XNU headers, see the definition of C-structures in: > --- [2] (`nlist_64`), [3] (`fat_arch` and `fat_header`). > --- > --- There is a good visual representation of Universal Binary > --- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6]. > --- Below is the schematic structure of Universal Binary, which > --- includes two executables for PowerPC and Intel i386 (omitted): > --- > --- 0x0000000 --------------------------------------- > --- | > --- struct | 0xcafebabe FAT_MAGIC magic > --- fat_header | ------------------------------------- > --- | 0x00000003 nfat_arch > --- --------------------------------------- > --- | 0x00000012 CPU_TYPE_POWERPC cputype > --- | ------------------------------------- > --- | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > --- struct | ------------------------------------- > --- fat_arch | 0x00001000 4096 bytes offset > --- | ------------------------------------- > --- | 0x00004224 16932 bytes size > --- | ------------------------------------- > --- | 0x0000000c 2^12 = 4096 bytes align > --- --------------------------------------- > --- --------------------------------------- > --- | 0x00000007 CPU_TYPE_I386 cputype > --- | ------------------------------------- > --- | 0x00000003 CPU_SUBTYPE_I386_ALL cpusubtype > --- struct | ------------------------------------- > --- fat_arch | 0x00006000 24576 bytes offset > --- | ------------------------------------- > --- | 0x0000292c 10540 bytes size > --- | ------------------------------------- > --- | 0x0000000c 2^12 = 4096 bytes align > --- --------------------------------------- > --- Unused > --- 0x00001000 --------------------------------------- > --- | 0xfeedface MH_MAGIC magic > --- | ------------------------------------ > --- | 0x00000012 CPU_TYPE_POWERPC cputype > --- | ------------------------------------ > --- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > --- mach_header | ------------------------------------ > --- | 0x00000002 MH_EXECUTE filetype > --- | ------------------------------------ > --- | 0x0000000b 10 load commands ncmds > --- | ------------------------------------ > --- | 0x00000574 1396 bytes sizeofcmds > --- | ------------------------------------ > --- | 0x00000085 DYLDLINK TWOLEVEL flags > --- -------------------------------------- > --- Load commands > --- --------------------------------------- > --- Data > --- --------------------------------------- > --- > --- < x86 executable > > --- > --- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference > --- 2. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/nlist.h > --- 3. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/fat.h > --- 4. https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code > --- 5. https://reverseengineering.stackexchange.com/a/6357/46029 > --- 6. http://formats.kaitai.io/mach_o/index.html I prefer to keep the non-FAT part of this comment since it is very useful. > --- > -- Using the same declarations as defined in <src/jit/bcsave.lua>. > ffi.cdef[[ <snipped> > local function create_obj_file(name, arch) > @@ -212,108 +66,37 @@ local function create_obj_file(name, arch) > return mach_o_path > end > > --- Parses a buffer in the Mach-O format and returns the FAT magic > --- number and `nfat_arch`. > +-- Parses a buffer in the Mach-O format and returns its fields > +-- in a table. > local function read_mach_o(buf, hw_arch) I suggest renaming it to `read_mach_o_hdr()` and returning only the header without any additional wrapping in the table. > - local res = { > - header = { > - magic = 0, > - nfat_arch = 0, > - }, > - fat_arch = {}, > - } > - > local is64 = hw_arch == 'arm64' Maybe it is better to use assert here, like we do for the `build_and_check_mach_o()` routine? > > - -- Mach-O FAT object. > - local mach_fat_obj_type = ffi.typeof(is64 and > - 'mach_fat_obj_64 *' or > - 'mach_fat_obj *') > - local obj = ffi.cast(mach_fat_obj_type, buf) > + -- Mach-O object. > + local mach_obj_type = ffi.typeof(is64 and 'mach_obj_64 *') Maybe just use mach_obj_64 since there is no alternative? > + local obj = ffi.cast(mach_obj_type, buf) > > - -- Mach-O FAT object header. > - local mach_fat_header = obj.fat > - -- Mach-O FAT is BE, target arch is LE. > - local be32 = bit.bswap > - res.header.magic = be32(mach_fat_header.magic) > - res.header.nfat_arch = be32(mach_fat_header.nfat_arch) > + -- Mach-O object header. > + local mach_header = obj.hdr > > - -- Mach-O FAT object arches. > - for i = 0, res.header.nfat_arch - 1 do > - local fat_arch = obj.fat_arch[i] > - local arch = { > - cputype = be32(fat_arch.cputype), > - cpusubtype = be32(fat_arch.cpusubtype), > - } > - table.insert(res.fat_arch, arch) > - end > - > - return res > + return { > + header = { > + magic = mach_header.magic, > + cputype = mach_header.cputype, > + cpusubtype = mach_header.cpusubtype, > + filetype = mach_header.filetype, > + ncmds = mach_header.ncmds, > + }, > + } > end > > --- Universal Binary can contain executables for more than one > --- CPU architecture. For simplicity, the test compares the *sum* > --- of CPU types and CPU subtypes. > --- > --- <src/jit/bcsave.lua:bcsave_machobj> has the definitions of the > --- numbers below. The original XNU source code may be found in > --- <osfmk/mach/machine.h> [1]. > --- > --- 1. https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/mach/machine.h.auto.html > --- > -local SUM_CPUTYPE = { > - -- x86 + arm. > - arm = 7 + 12, > - -- x64 + arm64. > - arm64 = 0x01000007 + 0x0100000c, > -} > -local SUM_CPUSUBTYPE = { > - -- x86 + arm. > - arm = 3 + 9, > - -- x64 + arm64. > - arm64 = 3 + 0, > -} > - > --- The function builds Mach-O FAT object file and retrieves > --- its header fields (magic and nfat_arch) and fields of each arch > --- (cputype, cpusubtype). > --- > --- The Mach-O FAT object header can be retrieved with `otool` on > --- macOS: > --- > --- $ otool -f empty.o > --- Fat headers > --- fat_magic 0xcafebabe > --- nfat_arch 2 > --- <snipped> > --- > --- CPU type and subtype can be retrieved with `lipo` on macOS: > --- > --- $ luajit -b -o osx -a arm empty.lua empty.o > --- $ lipo -archs empty.o > --- i386 armv7 > --- $ luajit -b -o osx -a arm64 empty.lua empty.o > --- $ lipo -archs empty.o > --- x86_64 arm64 > +-- The function builds Mach-O object file and retrieves > +-- its header fields. > local function build_and_check_mach_o(subtest) > local hw_arch = subtest.name > - assert(hw_arch == 'arm' or hw_arch == 'arm64') > + -- LuaJIT always generate 64-bit non-FAT Mach-O object files. The fact that these files are generated only for M1 CPUs looks worth mentioning. > + assert(hw_arch == 'arm64') > > - subtest:plan(4) > - -- FAT_MAGIC is an integer containing the value 0xCAFEBABE in > - -- big-endian byte order format. On a big-endian host CPU, > - -- this can be validated using the constant FAT_MAGIC; > - -- on a little-endian host CPU, it can be validated using > - -- the constant FAT_CIGAM. > - -- > - -- FAT_NARCH is an integer specifying the number of fat_arch > - -- data structures that follow. This is the number of > - -- architectures contained in this binary. > - -- > - -- See the aforementioned "OS X ABI Mach-O File Format > - -- Reference". The similar comment about Mach-O format will be appretiated. > - local FAT_MAGIC = '0xffffffffcafebabe' > - local FAT_NARCH = 2 > + subtest:plan(5) > > local MODULE_NAME = 'lango_team' > > @@ -327,24 +110,19 @@ local function build_and_check_mach_o(subtest) > assert(os.remove(mach_o_obj_path), 'remove an object file') > > local magic_str = string.format('%#x', mach_o.header.magic) > - subtest:is(magic_str, FAT_MAGIC, > - 'fat_magic is correct in Mach-O') > - subtest:is(mach_o.header.nfat_arch, FAT_NARCH, > - 'nfat_arch is correct in Mach-O') > - > - local total_cputype = 0 > - local total_cpusubtype = 0 > - for i = 1, FAT_NARCH do > - total_cputype = total_cputype + mach_o.fat_arch[i].cputype > - total_cpusubtype = total_cpusubtype + mach_o.fat_arch[i].cpusubtype > - end > - subtest:is(total_cputype, SUM_CPUTYPE[hw_arch], > + subtest:is(magic_str, '0xfeedfacf', Please use MH_MAGIC_64 named constant for this magic string. > + 'magic is correct in Mach-O') Looks like this line may be joined with the previous one. > + local cputype_str = string.format('%#x', mach_o.header.cputype) > + subtest:is(cputype_str, '0x100000c', Please use the named constant CPU_TYPE_ARM64 for this magic string. > 'cputype is correct in Mach-O') Looks like this line may be joined with the previous one. > - subtest:is(total_cpusubtype, SUM_CPUSUBTYPE[hw_arch], > + subtest:is(mach_o.header.cpusubtype, 0, Please use the named constant CPU_SUBTYPE_ARM64 for this magic constant. > 'cpusubtype is correct in Mach-O') Looks like this line may be joined with the previous one. > + subtest:is(mach_o.header.filetype, 1, What does the 1 filetype mean? Please use the named constant. > + 'filetype is correct in Mach-O') Looks like this line may be joined with the previous one. > + subtest:is(mach_o.header.ncmds, 2, Why there are 2 commands for Mach-O format? Please use the named constant. > + 'ncmds is correct in Mach-O') Looks like this line may be joined with the previous one. > end > > -test:test('arm', build_and_check_mach_o) > test:test('arm64', build_and_check_mach_o) > > test:done(true) > -- > 2.34.1 > [1]: https://github.com/LuaJIT/LuaJIT/issues/1181#issue-2202788411 -- Best regards, Sergey Kaplun ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-09 13:03 ` Sergey Kaplun via Tarantool-patches @ 2024-07-10 12:43 ` Sergey Bronnikov via Tarantool-patches 2024-07-11 7:29 ` Sergey Kaplun via Tarantool-patches 0 siblings, 1 reply; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-07-10 12:43 UTC (permalink / raw) To: Sergey Kaplun, Sergey Bronnikov; +Cc: tarantool-patches [-- Attachment #1: Type: text/plain, Size: 21753 bytes --] Sergey, thanks for review. Fixes applied and force-pushed to the same branch. Sergey On 09.07.2024 16:03, Sergey Kaplun via Tarantool-patches wrote: > Hi, Sergey! > Thanks for the patch! > Please consider my comments below. > > May we add the test [1] to verify that there will be no regression in the > future? This "test" was for a FAT binary generated by LuaJIT. With backported patches, FAT (aka Universal Binary) support is gone. mach-O header is checked by tests in proposed patch. > > On 05.07.24, Sergey Bronnikov wrote: >> Reported by Sergey Bronnikov. >> >> (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) >> >> Previously, LuaJIT generated Mach-O FAT object files for ARM and >> ARM64 on macOS. The patch removes support of 32-bit ARM and >> FAT object files and now LuaJIT generate Mach-O object files for >> ARM64. > I suppose we should mention that no x86/x86_64 objects are generated > now. x86_64 is still there: $ ./build/gc64/src/luajit -b -o osx -a arm64 empty.lua empty.o $ file empty.o empty.o: Mach-O 64-bit arm64 object $ ./build/gc64/src/luajit -b -o osx -a x64 empty.lua empty.o $ file empty.o empty.o: Mach-O 64-bit x86_64 object $ Anyway, commit message has been updated. >> Sergey Bronnikov: >> * added the description and the trimmed the test for the problem >> >> Part of tarantool/tarantool#10199 >> --- >> src/jit/bcsave.lua | 155 ++------- >> ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- >> 2 files changed, 70 insertions(+), 379 deletions(-) >> >> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua >> index 26ec29c6..61953c2d 100644 >> --- a/src/jit/bcsave.lua >> +++ b/src/jit/bcsave.lua > <snipped> > >> diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua >> index f008f3bd..6a58de95 100644 >> --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua >> +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua >> @@ -3,109 +3,11 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') >> local utils = require('utils') >> local ffi = require('ffi') >> >> -test:plan(2) >> +test:plan(1) >> >> -- The test creates an object file in Mach-O format with LuaJIT >> -- bytecode and checks the validity of the object file fields. >> --- >> --- The original problem is reproduced with LuaJIT, which is built >> --- with enabled AVX512F instructions. The support for AVX512F >> --- could be checked in `/proc/cpuinfo` on Linux and >> --- `sysctl hw.optional.avx512f` on Mac. AVX512F must be >> --- implicitly enabled in a C compiler by passing a CPU codename. >> --- Please take a look at the GCC Online Documentation [1] for >> --- available CPU codenames. Also, see the Wikipedia for CPUs with >> --- AVX-512 support [2]. >> --- Execute command below to detect the CPU codename: >> --- `gcc -march=native -Q --help=target | grep march`. >> --- >> --- 1.https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html >> --- 2.https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 >> --- >> --- Manual steps for reproducing are the following: >> --- >> --- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build >> --- $ cmake --build build --parallel >> --- $ echo > test.lua >> --- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o >> --- $ file test.o >> --- empty.o: DOS executable (block device driver) >> >> --- LuaJIT can generate so called Universal Binary with Lua >> --- bytecode. The Universal Binary format is a format for >> --- executable files that run natively on hardware platforms with >> --- different hardware architectures. This concept is more >> --- generally known as a fat binary. >> --- >> --- The format of the Mach-O is described in the document >> --- "OS X ABI Mach-O File Format Reference", published by Apple >> --- company. The copy of the (now removed) official documentation >> --- can be found here [1]. Yet another source of truth is >> --- XNU headers, see the definition of C-structures in: >> --- [2] (`nlist_64`), [3] (`fat_arch` and `fat_header`). >> --- >> --- There is a good visual representation of Universal Binary >> --- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6]. >> --- Below is the schematic structure of Universal Binary, which >> --- includes two executables for PowerPC and Intel i386 (omitted): >> --- >> --- 0x0000000 --------------------------------------- >> --- | >> --- struct | 0xcafebabe FAT_MAGIC magic >> --- fat_header | ------------------------------------- >> --- | 0x00000003 nfat_arch >> --- --------------------------------------- >> --- | 0x00000012 CPU_TYPE_POWERPC cputype >> --- | ------------------------------------- >> --- | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype >> --- struct | ------------------------------------- >> --- fat_arch | 0x00001000 4096 bytes offset >> --- | ------------------------------------- >> --- | 0x00004224 16932 bytes size >> --- | ------------------------------------- >> --- | 0x0000000c 2^12 = 4096 bytes align >> --- --------------------------------------- >> --- --------------------------------------- >> --- | 0x00000007 CPU_TYPE_I386 cputype >> --- | ------------------------------------- >> --- | 0x00000003 CPU_SUBTYPE_I386_ALL cpusubtype >> --- struct | ------------------------------------- >> --- fat_arch | 0x00006000 24576 bytes offset >> --- | ------------------------------------- >> --- | 0x0000292c 10540 bytes size >> --- | ------------------------------------- >> --- | 0x0000000c 2^12 = 4096 bytes align >> --- --------------------------------------- >> --- Unused >> --- 0x00001000 --------------------------------------- >> --- | 0xfeedface MH_MAGIC magic >> --- | ------------------------------------ >> --- | 0x00000012 CPU_TYPE_POWERPC cputype >> --- | ------------------------------------ >> --- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype >> --- mach_header | ------------------------------------ >> --- | 0x00000002 MH_EXECUTE filetype >> --- | ------------------------------------ >> --- | 0x0000000b 10 load commands ncmds >> --- | ------------------------------------ >> --- | 0x00000574 1396 bytes sizeofcmds >> --- | ------------------------------------ >> --- | 0x00000085 DYLDLINK TWOLEVEL flags >> --- -------------------------------------- >> --- Load commands >> --- --------------------------------------- >> --- Data >> --- --------------------------------------- >> --- >> --- < x86 executable > >> --- >> --- 1.https://github.com/aidansteele/osx-abi-macho-file-format-reference >> --- 2.https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/nlist.h >> --- 3.https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/fat.h >> --- 4.https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code >> --- 5.https://reverseengineering.stackexchange.com/a/6357/46029 >> --- 6.http://formats.kaitai.io/mach_o/index.html > I prefer to keep the non-FAT part of this comment since it is very > useful. reverted: diff --git a/build/src/luajit b/build/src/luajit deleted file mode 100755 index bd10e09a..00000000 Binary files a/build/src/luajit and /dev/null differ diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua index 6a58de95..f9ca6e38 100644 --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -68,6 +68,62 @@ end -- Parses a buffer in the Mach-O format and returns its fields -- in a table. +-- The test creates an object file in Mach-O format with LuaJIT +-- bytecode and checks the validity of the object file fields. +--- +--- The original problem is reproduced with LuaJIT, which is built +--- with enabled AVX512F instructions. The support for AVX512F +--- could be checked in `/proc/cpuinfo` on Linux and +--- `sysctl hw.optional.avx512f` on Mac. AVX512F must be +--- implicitly enabled in a C compiler by passing a CPU codename. +--- Please take a look at the GCC Online Documentation [1] for +--- available CPU codenames. Also, see the Wikipedia for CPUs with +--- AVX-512 support [2]. +--- Execute command below to detect the CPU codename: +--- `gcc -march=native -Q --help=target | grep march`. +--- +--- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html +--- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 +--- +--- Manual steps for reproducing are the following: +--- +--- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build +--- $ cmake --build build --parallel +--- $ echo > test.lua +--- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o +--- $ file test.o +--- empty.o: DOS executable (block device driver) + +--- The format of the Mach-O is described in the document +--- "OS X ABI Mach-O File Format Reference", published by Apple +--- company. The copy of the (now removed) official documentation +--- can be found here [1]. There is a good visual representation +--- of Mach-O format in "Mac OS X Internals" book (pages 67-68) +--- [2] and in the [3]. +-- +--- 0x0000000 --------------------------------------- +--- | 0xfeedface MH_MAGIC magic +--- | ------------------------------------ +--- | 0x00000012 CPU_TYPE_POWERPC cputype +--- | ------------------------------------ +--- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype +--- mach_header | ------------------------------------ +--- | 0x00000002 MH_EXECUTE filetype +--- | ------------------------------------ +--- | 0x0000000b 10 load commands ncmds +--- | ------------------------------------ +--- | 0x00000574 1396 bytes sizeofcmds +--- | ------------------------------------ +--- | 0x00000085 DYLDLINK TWOLEVEL flags +--- -------------------------------------- +--- Load commands +--- --------------------------------------- +--- Data +--- --------------------------------------- +--- +--- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference +--- 2. https://reverseengineering.stackexchange.com/a/6357/46029 +--- 3. http://formats.kaitai.io/mach_o/index.html local function read_mach_o(buf, hw_arch) local is64 = hw_arch == 'arm64' > >> --- >> -- Using the same declarations as defined in <src/jit/bcsave.lua>. >> ffi.cdef[[ > <snipped> > >> local function create_obj_file(name, arch) >> @@ -212,108 +66,37 @@ local function create_obj_file(name, arch) >> return mach_o_path >> end >> >> --- Parses a buffer in the Mach-O format and returns the FAT magic >> --- number and `nfat_arch`. >> +-- Parses a buffer in the Mach-O format and returns its fields >> +-- in a table. >> local function read_mach_o(buf, hw_arch) > I suggest renaming it to `read_mach_o_hdr()` and returning only the > header without any additional wrapping in the table. Updated: diff --git a/build/src/luajit b/build/src/luajit deleted file mode 100755 index bd10e09a..00000000 Binary files a/build/src/luajit and /dev/null differ diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua index 6a58de95..0f269a8c 100644 --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -78,15 +134,7 @@ local function read_mach_o(buf, hw_arch) -- Mach-O object header. local mach_header = obj.hdr - return { - header = { - magic = mach_header.magic, - cputype = mach_header.cputype, - cpusubtype = mach_header.cpusubtype, - filetype = mach_header.filetype, - ncmds = mach_header.ncmds, - }, - } + return mach_header end -- The function builds Mach-O object file and retrieves @@ -104,22 +152,22 @@ local function build_and_check_mach_o(subtest) local mach_o_buf = utils.tools.read_file(mach_o_obj_path) assert(mach_o_buf ~= nil and #mach_o_buf ~= 0, 'cannot read an object file') - local mach_o = read_mach_o(mach_o_buf, hw_arch) + local mach_o = read_mach_o_hdr(mach_o_buf, hw_arch) -- Teardown. assert(os.remove(mach_o_obj_path), 'remove an object file') - local magic_str = string.format('%#x', mach_o.header.magic) + local magic_str = string.format('%#x', mach_o.magic) subtest:is(magic_str, '0xfeedfacf', 'magic is correct in Mach-O') - local cputype_str = string.format('%#x', mach_o.header.cputype) + local cputype_str = string.format('%#x', mach_o.cputype) subtest:is(cputype_str, '0x100000c', 'cputype is correct in Mach-O') - subtest:is(mach_o.header.cpusubtype, 0, + subtest:is(mach_o.cpusubtype, 0, 'cpusubtype is correct in Mach-O') - subtest:is(mach_o.header.filetype, 1, + subtest:is(mach_o.filetype, 1, 'filetype is correct in Mach-O') - subtest:is(mach_o.header.ncmds, 2, + subtest:is(mach_o.ncmds, 2, 'ncmds is correct in Mach-O') end >> - local res = { >> - header = { >> - magic = 0, >> - nfat_arch = 0, >> - }, >> - fat_arch = {}, >> - } >> - >> local is64 = hw_arch == 'arm64' > Maybe it is better to use assert here, like we do for the > `build_and_check_mach_o()` routine? Added, but read_mach_o_hdr is called by build_and_check_mach_o where assert already added. > >> >> - -- Mach-O FAT object. >> - local mach_fat_obj_type = ffi.typeof(is64 and >> - 'mach_fat_obj_64 *' or >> - 'mach_fat_obj *') >> - local obj = ffi.cast(mach_fat_obj_type, buf) >> + -- Mach-O object. >> + local mach_obj_type = ffi.typeof(is64 and 'mach_obj_64 *') > Maybe just use mach_obj_64 since there is no alternative? ok >> + local obj = ffi.cast(mach_obj_type, buf) >> >> - -- Mach-O FAT object header. >> - local mach_fat_header = obj.fat >> - -- Mach-O FAT is BE, target arch is LE. >> - local be32 = bit.bswap >> - res.header.magic = be32(mach_fat_header.magic) >> - res.header.nfat_arch = be32(mach_fat_header.nfat_arch) >> + -- Mach-O object header. >> + local mach_header = obj.hdr >> >> - -- Mach-O FAT object arches. >> - for i = 0, res.header.nfat_arch - 1 do >> - local fat_arch = obj.fat_arch[i] >> - local arch = { >> - cputype = be32(fat_arch.cputype), >> - cpusubtype = be32(fat_arch.cpusubtype), >> - } >> - table.insert(res.fat_arch, arch) >> - end >> - >> - return res >> + return { >> + header = { >> + magic = mach_header.magic, >> + cputype = mach_header.cputype, >> + cpusubtype = mach_header.cpusubtype, >> + filetype = mach_header.filetype, >> + ncmds = mach_header.ncmds, >> + }, >> + } >> end >> >> --- Universal Binary can contain executables for more than one >> --- CPU architecture. For simplicity, the test compares the *sum* >> --- of CPU types and CPU subtypes. >> --- >> --- <src/jit/bcsave.lua:bcsave_machobj> has the definitions of the >> --- numbers below. The original XNU source code may be found in >> --- <osfmk/mach/machine.h> [1]. >> --- >> --- 1.https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/mach/machine.h.auto.html >> --- >> -local SUM_CPUTYPE = { >> - -- x86 + arm. >> - arm = 7 + 12, >> - -- x64 + arm64. >> - arm64 = 0x01000007 + 0x0100000c, >> -} >> -local SUM_CPUSUBTYPE = { >> - -- x86 + arm. >> - arm = 3 + 9, >> - -- x64 + arm64. >> - arm64 = 3 + 0, >> -} >> - >> --- The function builds Mach-O FAT object file and retrieves >> --- its header fields (magic and nfat_arch) and fields of each arch >> --- (cputype, cpusubtype). >> --- >> --- The Mach-O FAT object header can be retrieved with `otool` on >> --- macOS: >> --- >> --- $ otool -f empty.o >> --- Fat headers >> --- fat_magic 0xcafebabe >> --- nfat_arch 2 >> --- <snipped> >> --- >> --- CPU type and subtype can be retrieved with `lipo` on macOS: >> --- >> --- $ luajit -b -o osx -a arm empty.lua empty.o >> --- $ lipo -archs empty.o >> --- i386 armv7 >> --- $ luajit -b -o osx -a arm64 empty.lua empty.o >> --- $ lipo -archs empty.o >> --- x86_64 arm64 >> +-- The function builds Mach-O object file and retrieves >> +-- its header fields. >> local function build_and_check_mach_o(subtest) >> local hw_arch = subtest.name >> - assert(hw_arch == 'arm' or hw_arch == 'arm64') >> + -- LuaJIT always generate 64-bit non-FAT Mach-O object files. > The fact that these files are generated only for M1 CPUs looks worth > mentioning. > >> + assert(hw_arch == 'arm64') >> >> - subtest:plan(4) >> - -- FAT_MAGIC is an integer containing the value 0xCAFEBABE in >> - -- big-endian byte order format. On a big-endian host CPU, >> - -- this can be validated using the constant FAT_MAGIC; >> - -- on a little-endian host CPU, it can be validated using >> - -- the constant FAT_CIGAM. >> - -- >> - -- FAT_NARCH is an integer specifying the number of fat_arch >> - -- data structures that follow. This is the number of >> - -- architectures contained in this binary. >> - -- >> - -- See the aforementioned "OS X ABI Mach-O File Format >> - -- Reference". > The similar comment about Mach-O format will be appretiated. mach-o header struct described in a comment above > >> - local FAT_MAGIC = '0xffffffffcafebabe' >> - local FAT_NARCH = 2 >> + subtest:plan(5) >> >> local MODULE_NAME = 'lango_team' >> >> @@ -327,24 +110,19 @@ local function build_and_check_mach_o(subtest) >> assert(os.remove(mach_o_obj_path), 'remove an object file') >> >> local magic_str = string.format('%#x', mach_o.header.magic) >> - subtest:is(magic_str, FAT_MAGIC, >> - 'fat_magic is correct in Mach-O') >> - subtest:is(mach_o.header.nfat_arch, FAT_NARCH, >> - 'nfat_arch is correct in Mach-O') >> - >> - local total_cputype = 0 >> - local total_cpusubtype = 0 >> - for i = 1, FAT_NARCH do >> - total_cputype = total_cputype + mach_o.fat_arch[i].cputype >> - total_cpusubtype = total_cpusubtype + mach_o.fat_arch[i].cpusubtype >> - end >> - subtest:is(total_cputype, SUM_CPUTYPE[hw_arch], >> + subtest:is(magic_str, '0xfeedfacf', > Please use MH_MAGIC_64 named constant for this magic string. Added > >> + 'magic is correct in Mach-O') > Looks like this line may be joined with the previous one. not enough free space for this >> + local cputype_str = string.format('%#x', mach_o.header.cputype) >> + subtest:is(cputype_str, '0x100000c', > Please use the named constant CPU_TYPE_ARM64 for this magic string. Added > >> 'cputype is correct in Mach-O') > Looks like this line may be joined with the previous one. not enough free space for this > >> - subtest:is(total_cpusubtype, SUM_CPUSUBTYPE[hw_arch], >> + subtest:is(mach_o.header.cpusubtype, 0, > Please use the named constant CPU_SUBTYPE_ARM64 for this magic constant. added > >> 'cpusubtype is correct in Mach-O') > Looks like this line may be joined with the previous one. not enough free space for this > >> + subtest:is(mach_o.header.filetype, 1, > What does the 1 filetype mean? > Please use the named constant. MH_OBJECT, added * The file type MH_OBJECT is a compact format intended as output of the * assembler and input (and possibly output) of the link editor (the .o * format). All sections are in one unnamed segment with no segment padding. * This format is used as an executable format when the file is so small the * segment padding greatly increases it's size. https://opensource.apple.com/source/xnu/xnu-344/EXTERNAL_HEADERS/mach-o/loader.h.auto.html > >> + 'filetype is correct in Mach-O') > Looks like this line may be joined with the previous one. no free space for this > >> + subtest:is(mach_o.header.ncmds, 2, > Why there are 2 commands for Mach-O format? > Please use the named constant. is constant name 'ncmds' self-explained? > >> + 'ncmds is correct in Mach-O') > Looks like this line may be joined with the previous one. joined > >> end >> >> -test:test('arm', build_and_check_mach_o) >> test:test('arm64', build_and_check_mach_o) >> >> test:done(true) >> -- >> 2.34.1 >> > [1]:https://github.com/LuaJIT/LuaJIT/issues/1181#issue-2202788411 > [-- Attachment #2: Type: text/html, Size: 30014 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-10 12:43 ` Sergey Bronnikov via Tarantool-patches @ 2024-07-11 7:29 ` Sergey Kaplun via Tarantool-patches 2024-07-23 20:07 ` Sergey Bronnikov via Tarantool-patches 0 siblings, 1 reply; 11+ messages in thread From: Sergey Kaplun via Tarantool-patches @ 2024-07-11 7:29 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: Sergey Bronnikov, tarantool-patches Hi, Sergey! Thanks for the fixes! I've applied some minor fixes to your branch. LGTM, after adding the comment about ncmds value. On 10.07.24, Sergey Bronnikov wrote: > Sergey, > > thanks for review. Fixes applied and force-pushed to the same branch. > > > Sergey > > On 09.07.2024 16:03, Sergey Kaplun via Tarantool-patches wrote: > > Hi, Sergey! > > Thanks for the patch! > > Please consider my comments below. > > > > May we add the test [1] to verify that there will be no regression in the > > future? > > This "test" was for a FAT binary generated by LuaJIT. > > With backported patches, FAT (aka Universal Binary) support is gone. > > mach-O header is checked by tests in proposed patch. OK, let's leave it as is. > > > > > On 05.07.24, Sergey Bronnikov wrote: > >> Reported by Sergey Bronnikov. > >> > >> (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) > >> > >> Previously, LuaJIT generated Mach-O FAT object files for ARM and > >> ARM64 on macOS. The patch removes support of 32-bit ARM and > >> FAT object files and now LuaJIT generate Mach-O object files for > >> ARM64. > > I suppose we should mention that no x86/x86_64 objects are generated > > now. > x86_64 is still there: > > $ ./build/gc64/src/luajit -b -o osx -a arm64 empty.lua empty.o > $ file empty.o > empty.o: Mach-O 64-bit arm64 object > $ ./build/gc64/src/luajit -b -o osx -a x64 empty.lua empty.o > $ file empty.o > empty.o: Mach-O 64-bit x86_64 object > $ My bad, thanks for the clarification. > > Anyway, commit message has been updated. > > >> Sergey Bronnikov: > >> * added the description and the trimmed the test for the problem > >> > >> Part of tarantool/tarantool#10199 > >> --- > >> src/jit/bcsave.lua | 155 ++------- > >> ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- > >> 2 files changed, 70 insertions(+), 379 deletions(-) > >> > >> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua > >> index 26ec29c6..61953c2d 100644 > >> --- a/src/jit/bcsave.lua > >> +++ b/src/jit/bcsave.lua <snipped> > > > >> + subtest:is(mach_o.header.ncmds, 2, > > Why there are 2 commands for Mach-O format? > > Please use the named constant. > is constant name 'ncmds' self-explained? No, since I don't understand why it is 2 and not 12. Please, add a comment to make it obvious for any reader. > > > >> + 'ncmds is correct in Mach-O') > > Looks like this line may be joined with the previous one. > joined > > > >> end <snipped> I've added some fixes to your branch: * Fixed the comment from --- to --. * Fixed some typos. * Moved constant declarations to the beginning of the file. * Join some lines since there is no need to add a new line. =================================================================== diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua index d30198c0..9963bf88 100644 --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -3,6 +3,12 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') local utils = require('utils') local ffi = require('ffi') +local CPU_TYPE_ARM64 = '0x100000c' +local CPU_SUBTYPE_ARM64 = 0x0 +local MH_MAGIC_64 = '0xfeedfacf' +-- Relocatable object file. +local MH_OBJECT = 0x1 + test:plan(1) -- The test creates an object file in Mach-O format with LuaJIT @@ -66,64 +72,64 @@ local function create_obj_file(name, arch) return mach_o_path end --- Parses a buffer in the Mach-O format and returns its fields --- in a table. +-- Parses a buffer in the Mach-O format and returns its fields in +-- a table. -- The test creates an object file in Mach-O format with LuaJIT -- bytecode and checks the validity of the object file fields. ---- ---- The original problem is reproduced with LuaJIT, which is built ---- with enabled AVX512F instructions. The support for AVX512F ---- could be checked in `/proc/cpuinfo` on Linux and ---- `sysctl hw.optional.avx512f` on Mac. AVX512F must be ---- implicitly enabled in a C compiler by passing a CPU codename. ---- Please take a look at the GCC Online Documentation [1] for ---- available CPU codenames. Also, see the Wikipedia for CPUs with ---- AVX-512 support [2]. ---- Execute command below to detect the CPU codename: ---- `gcc -march=native -Q --help=target | grep march`. ---- ---- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html ---- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 ---- ---- Manual steps for reproducing are the following: ---- ---- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build ---- $ cmake --build build --parallel ---- $ echo > test.lua ---- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o ---- $ file test.o ---- empty.o: DOS executable (block device driver) - ---- The format of the Mach-O is described in the document ---- "OS X ABI Mach-O File Format Reference", published by Apple ---- company. The copy of the (now removed) official documentation ---- can be found here [1]. There is a good visual representation ---- of Mach-O format in "Mac OS X Internals" book (pages 67-68) ---- [2] and in the [3]. -- ---- 0x0000000 --------------------------------------- ---- | 0xfeedface MH_MAGIC magic ---- | ------------------------------------ ---- | 0x00000012 CPU_TYPE_POWERPC cputype ---- | ------------------------------------ ---- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype ---- mach_header | ------------------------------------ ---- | 0x00000002 MH_EXECUTE filetype ---- | ------------------------------------ ---- | 0x0000000b 10 load commands ncmds ---- | ------------------------------------ ---- | 0x00000574 1396 bytes sizeofcmds ---- | ------------------------------------ ---- | 0x00000085 DYLDLINK TWOLEVEL flags ---- -------------------------------------- ---- Load commands ---- --------------------------------------- ---- Data ---- --------------------------------------- ---- ---- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference ---- 2. https://reverseengineering.stackexchange.com/a/6357/46029 ---- 3. http://formats.kaitai.io/mach_o/index.html +-- The original problem is reproduced with LuaJIT, which is built +-- with enabled AVX512F instructions. The support for AVX512F +-- could be checked in `/proc/cpuinfo` on Linux and +-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be implicitly +-- enabled in a C compiler by passing a CPU codename. +-- Please take a look at the GCC Online Documentation [1] for +-- available CPU codenames. Also, see the Wikipedia for CPUs with +-- AVX-512 support [2]. +-- Execute the command below to detect the CPU codename: +-- `gcc -march=native -Q --help=target | grep march`. +-- +-- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html +-- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 +-- +-- Manual steps for reproducing are the following: +-- +-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build +-- $ cmake --build build --parallel +-- $ echo > test.lua +-- $ ./luajit -b -o osx -a arm64 test.lua test.o +-- $ file test.o +-- test.o: Mach-O 64-bit arm64 object + +-- The format of the Mach-O is described in the document +-- "OS X ABI Mach-O File Format Reference", published by Apple +-- company. The copy of the (now removed) official documentation +-- can be found here [1]. There is a good visual representation +-- of Mach-O format in "Mac OS X Internals" book (pages 67-68) +-- [2] and in the [3]. +-- +-- 0x0000000 --------------------------------------- +-- | 0xfeedface MH_MAGIC magic +-- | ------------------------------------ +-- | 0x00000012 CPU_TYPE_POWERPC cputype +-- | ------------------------------------ +-- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype +-- mach_header | ------------------------------------ +-- | 0x00000002 MH_EXECUTE filetype +-- | ------------------------------------ +-- | 0x0000000b 10 load commands ncmds +-- | ------------------------------------ +-- | 0x00000574 1396 bytes sizeofcmds +-- | ------------------------------------ +-- | 0x00000085 DYLDLINK TWOLEVEL flags +-- -------------------------------------- +-- Load commands +-- --------------------------------------- +-- Data +-- --------------------------------------- +-- +-- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference +-- 2. https://reverseengineering.stackexchange.com/a/6357/46029 +-- 3. http://formats.kaitai.io/mach_o/index.html local function read_mach_o_hdr(buf, hw_arch) -- LuaJIT always generate 64-bit non-FAT Mach-O object files. assert(hw_arch == 'arm64') @@ -138,11 +144,11 @@ local function read_mach_o_hdr(buf, hw_arch) return mach_header end --- The function builds Mach-O object file and retrieves --- its header fields. +-- The function builds a Mach-O object file and retrieves its +-- header fields. local function build_and_check_mach_o(subtest) local hw_arch = subtest.name - -- LuaJIT always generate 64-bit non-FAT Mach-O object files. + -- LuaJIT always generates 64-bit, non-FAT Mach-O object files. assert(hw_arch == 'arm64') subtest:plan(5) @@ -159,19 +165,12 @@ local function build_and_check_mach_o(subtest) assert(os.remove(mach_o_obj_path), 'remove an object file') local magic_str = string.format('%#x', mach_o.magic) - local MH_MAGIC_64 = '0xfeedfacf' - subtest:is(magic_str, MH_MAGIC_64, - 'magic is correct in Mach-O') + subtest:is(magic_str, MH_MAGIC_64, 'magic is correct in Mach-O') local cputype_str = string.format('%#x', mach_o.cputype) - local CPU_TYPE_ARM64 = '0x100000c' - subtest:is(cputype_str, CPU_TYPE_ARM64, - 'cputype is correct in Mach-O') - local CPU_SUBTYPE_ARM64 = 0 + subtest:is(cputype_str, CPU_TYPE_ARM64, 'cputype is correct in Mach-O') subtest:is(mach_o.cpusubtype, CPU_SUBTYPE_ARM64, 'cpusubtype is correct in Mach-O') - local MH_OBJECT = 1 - subtest:is(mach_o.filetype, MH_OBJECT, - 'filetype is correct in Mach-O') + subtest:is(mach_o.filetype, MH_OBJECT, 'filetype is correct in Mach-O') subtest:is(mach_o.ncmds, 2, 'ncmds is correct in Mach-O') end =================================================================== Please let me know if you have any concerns. -- Best regards, Sergey Kaplun ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-11 7:29 ` Sergey Kaplun via Tarantool-patches @ 2024-07-23 20:07 ` Sergey Bronnikov via Tarantool-patches 2024-08-19 9:45 ` Maxim Kokryashkin via Tarantool-patches 0 siblings, 1 reply; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-07-23 20:07 UTC (permalink / raw) To: Sergey Kaplun; +Cc: Sergey Bronnikov, tarantool-patches [-- Attachment #1: Type: text/plain, Size: 12715 bytes --] Hi, Sergey thanks for the comments and fixes! But please don't squash fixes next time. Sergey On 11.07.2024 10:29, Sergey Kaplun wrote: > Hi, Sergey! > Thanks for the fixes! > I've applied some minor fixes to your branch. > LGTM, after adding the comment about ncmds value. > > On 10.07.24, Sergey Bronnikov wrote: >> Sergey, >> >> thanks for review. Fixes applied and force-pushed to the same branch. >> >> >> Sergey >> >> On 09.07.2024 16:03, Sergey Kaplun via Tarantool-patches wrote: >>> Hi, Sergey! >>> Thanks for the patch! >>> Please consider my comments below. >>> >>> May we add the test [1] to verify that there will be no regression in the >>> future? >> This "test" was for a FAT binary generated by LuaJIT. >> >> With backported patches, FAT (aka Universal Binary) support is gone. >> >> mach-O header is checked by tests in proposed patch. > OK, let's leave it as is. Thanks! > >>> On 05.07.24, Sergey Bronnikov wrote: >>>> Reported by Sergey Bronnikov. >>>> >>>> (cherry picked from commit 7110b935672489afd6ba3eef3e5139d2f3bd05b6) >>>> >>>> Previously, LuaJIT generated Mach-O FAT object files for ARM and >>>> ARM64 on macOS. The patch removes support of 32-bit ARM and >>>> FAT object files and now LuaJIT generate Mach-O object files for >>>> ARM64. >>> I suppose we should mention that no x86/x86_64 objects are generated >>> now. >> x86_64 is still there: >> >> $ ./build/gc64/src/luajit -b -o osx -a arm64 empty.lua empty.o >> $ file empty.o >> empty.o: Mach-O 64-bit arm64 object >> $ ./build/gc64/src/luajit -b -o osx -a x64 empty.lua empty.o >> $ file empty.o >> empty.o: Mach-O 64-bit x86_64 object >> $ > My bad, thanks for the clarification. > >> Anyway, commit message has been updated. >> >>>> Sergey Bronnikov: >>>> * added the description and the trimmed the test for the problem >>>> >>>> Part of tarantool/tarantool#10199 >>>> --- >>>> src/jit/bcsave.lua | 155 ++------- >>>> ...-865-cross-generation-mach-o-file.test.lua | 294 +++--------------- >>>> 2 files changed, 70 insertions(+), 379 deletions(-) >>>> >>>> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua >>>> index 26ec29c6..61953c2d 100644 >>>> --- a/src/jit/bcsave.lua >>>> +++ b/src/jit/bcsave.lua > <snipped> > >>>> + subtest:is(mach_o.header.ncmds, 2, >>> Why there are 2 commands for Mach-O format? >>> Please use the named constant. >> is constant name 'ncmds' self-explained? > No, since I don't understand why it is 2 and not 12. Please, add a > comment to make it obvious for any reader. Why it should be 12? I just used a constant from bcsave.lua [1]. I don't get an initial idea behind this number. As I got it right from Mach-O spec `ncmd` is a number of load commands following the header. The possible load commands listed in a table [2], but no one load command listed in a table is used in bcsave.lua for generation Mach-O object file. 1. https://github.com/LuaJIT/LuaJIT/blob/04dca7911ea255f37be799c18d74c305b921c1a6/src/jit/bcsave.lua#L489 2. https://github.com/aidansteele/osx-abi-macho-file-format-reference?tab=readme-ov-file#table-4-mach-o-load-commands >>>> + 'ncmds is correct in Mach-O') >>> Looks like this line may be joined with the previous one. >> joined >>>> end > <snipped> > > I've added some fixes to your branch: > * Fixed the comment from --- to --. > * Fixed some typos. > * Moved constant declarations to the beginning of the file. > * Join some lines since there is no need to add a new line. Added two more fixes: * moved test description to the top of the test. It would be better to place problem description at the beginning and Mach-O format description before constants. * sorted constants alphabetically * and added a comment: --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -64,6 +64,7 @@ test:plan(1) local CPU_SUBTYPE_ARM64 = 0x0 local CPU_TYPE_ARM64 = '0x100000c' +-- MH_MAGIC_64: mach-o, big-endian, x64. local MH_MAGIC_64 = '0xfeedfacf' -- Relocatable object file. local MH_OBJECT = 0x1 > > =================================================================== > diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > index d30198c0..9963bf88 100644 > --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua > @@ -3,6 +3,12 @@ local test = tap.test('lj-865-cross-generation-mach-o-file') > local utils = require('utils') > local ffi = require('ffi') > > +local CPU_TYPE_ARM64 = '0x100000c' > +local CPU_SUBTYPE_ARM64 = 0x0 > +local MH_MAGIC_64 = '0xfeedfacf' > +-- Relocatable object file. > +local MH_OBJECT = 0x1 > + > test:plan(1) > > -- The test creates an object file in Mach-O format with LuaJIT > @@ -66,64 +72,64 @@ local function create_obj_file(name, arch) > return mach_o_path > end > > --- Parses a buffer in the Mach-O format and returns its fields > --- in a table. > +-- Parses a buffer in the Mach-O format and returns its fields in > +-- a table. > -- The test creates an object file in Mach-O format with LuaJIT > -- bytecode and checks the validity of the object file fields. > ---- > ---- The original problem is reproduced with LuaJIT, which is built > ---- with enabled AVX512F instructions. The support for AVX512F > ---- could be checked in `/proc/cpuinfo` on Linux and > ---- `sysctl hw.optional.avx512f` on Mac. AVX512F must be > ---- implicitly enabled in a C compiler by passing a CPU codename. > ---- Please take a look at the GCC Online Documentation [1] for > ---- available CPU codenames. Also, see the Wikipedia for CPUs with > ---- AVX-512 support [2]. > ---- Execute command below to detect the CPU codename: > ---- `gcc -march=native -Q --help=target | grep march`. > ---- > ---- 1.https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html > ---- 2.https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 > ---- > ---- Manual steps for reproducing are the following: > ---- > ---- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build > ---- $ cmake --build build --parallel > ---- $ echo > test.lua > ---- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o > ---- $ file test.o > ---- empty.o: DOS executable (block device driver) > - > ---- The format of the Mach-O is described in the document > ---- "OS X ABI Mach-O File Format Reference", published by Apple > ---- company. The copy of the (now removed) official documentation > ---- can be found here [1]. There is a good visual representation > ---- of Mach-O format in "Mac OS X Internals" book (pages 67-68) > ---- [2] and in the [3]. > -- > ---- 0x0000000 --------------------------------------- > ---- | 0xfeedface MH_MAGIC magic > ---- | ------------------------------------ > ---- | 0x00000012 CPU_TYPE_POWERPC cputype > ---- | ------------------------------------ > ---- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > ---- mach_header | ------------------------------------ > ---- | 0x00000002 MH_EXECUTE filetype > ---- | ------------------------------------ > ---- | 0x0000000b 10 load commands ncmds > ---- | ------------------------------------ > ---- | 0x00000574 1396 bytes sizeofcmds > ---- | ------------------------------------ > ---- | 0x00000085 DYLDLINK TWOLEVEL flags > ---- -------------------------------------- > ---- Load commands > ---- --------------------------------------- > ---- Data > ---- --------------------------------------- > ---- > ---- 1.https://github.com/aidansteele/osx-abi-macho-file-format-reference > ---- 2.https://reverseengineering.stackexchange.com/a/6357/46029 > ---- 3.http://formats.kaitai.io/mach_o/index.html > +-- The original problem is reproduced with LuaJIT, which is built > +-- with enabled AVX512F instructions. The support for AVX512F > +-- could be checked in `/proc/cpuinfo` on Linux and > +-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be implicitly > +-- enabled in a C compiler by passing a CPU codename. > +-- Please take a look at the GCC Online Documentation [1] for > +-- available CPU codenames. Also, see the Wikipedia for CPUs with > +-- AVX-512 support [2]. > +-- Execute the command below to detect the CPU codename: > +-- `gcc -march=native -Q --help=target | grep march`. > +-- > +-- 1.https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html > +-- 2.https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512 > +-- > +-- Manual steps for reproducing are the following: > +-- > +-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build > +-- $ cmake --build build --parallel > +-- $ echo > test.lua > +-- $ ./luajit -b -o osx -a arm64 test.lua test.o > +-- $ file test.o > +-- test.o: Mach-O 64-bit arm64 object > + > +-- The format of the Mach-O is described in the document > +-- "OS X ABI Mach-O File Format Reference", published by Apple > +-- company. The copy of the (now removed) official documentation > +-- can be found here [1]. There is a good visual representation > +-- of Mach-O format in "Mac OS X Internals" book (pages 67-68) > +-- [2] and in the [3]. > +-- > +-- 0x0000000 --------------------------------------- > +-- | 0xfeedface MH_MAGIC magic > +-- | ------------------------------------ > +-- | 0x00000012 CPU_TYPE_POWERPC cputype > +-- | ------------------------------------ > +-- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype > +-- mach_header | ------------------------------------ > +-- | 0x00000002 MH_EXECUTE filetype > +-- | ------------------------------------ > +-- | 0x0000000b 10 load commands ncmds > +-- | ------------------------------------ > +-- | 0x00000574 1396 bytes sizeofcmds > +-- | ------------------------------------ > +-- | 0x00000085 DYLDLINK TWOLEVEL flags > +-- -------------------------------------- > +-- Load commands > +-- --------------------------------------- > +-- Data > +-- --------------------------------------- > +-- > +-- 1.https://github.com/aidansteele/osx-abi-macho-file-format-reference > +-- 2.https://reverseengineering.stackexchange.com/a/6357/46029 > +-- 3.http://formats.kaitai.io/mach_o/index.html > local function read_mach_o_hdr(buf, hw_arch) > -- LuaJIT always generate 64-bit non-FAT Mach-O object files. > assert(hw_arch == 'arm64') > @@ -138,11 +144,11 @@ local function read_mach_o_hdr(buf, hw_arch) > return mach_header > end > > --- The function builds Mach-O object file and retrieves > --- its header fields. > +-- The function builds a Mach-O object file and retrieves its > +-- header fields. > local function build_and_check_mach_o(subtest) > local hw_arch = subtest.name > - -- LuaJIT always generate 64-bit non-FAT Mach-O object files. > + -- LuaJIT always generates 64-bit, non-FAT Mach-O object files. > assert(hw_arch == 'arm64') > > subtest:plan(5) > @@ -159,19 +165,12 @@ local function build_and_check_mach_o(subtest) > assert(os.remove(mach_o_obj_path), 'remove an object file') > > local magic_str = string.format('%#x', mach_o.magic) > - local MH_MAGIC_64 = '0xfeedfacf' > - subtest:is(magic_str, MH_MAGIC_64, > - 'magic is correct in Mach-O') > + subtest:is(magic_str, MH_MAGIC_64, 'magic is correct in Mach-O') > local cputype_str = string.format('%#x', mach_o.cputype) > - local CPU_TYPE_ARM64 = '0x100000c' > - subtest:is(cputype_str, CPU_TYPE_ARM64, > - 'cputype is correct in Mach-O') > - local CPU_SUBTYPE_ARM64 = 0 > + subtest:is(cputype_str, CPU_TYPE_ARM64, 'cputype is correct in Mach-O') > subtest:is(mach_o.cpusubtype, CPU_SUBTYPE_ARM64, > 'cpusubtype is correct in Mach-O') > - local MH_OBJECT = 1 > - subtest:is(mach_o.filetype, MH_OBJECT, > - 'filetype is correct in Mach-O') > + subtest:is(mach_o.filetype, MH_OBJECT, 'filetype is correct in Mach-O') > subtest:is(mach_o.ncmds, 2, 'ncmds is correct in Mach-O') > end > > =================================================================== > > Please let me know if you have any concerns. > [-- Attachment #2: Type: text/html, Size: 16716 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-07-23 20:07 ` Sergey Bronnikov via Tarantool-patches @ 2024-08-19 9:45 ` Maxim Kokryashkin via Tarantool-patches 2024-08-20 8:57 ` Sergey Bronnikov via Tarantool-patches 0 siblings, 1 reply; 11+ messages in thread From: Maxim Kokryashkin via Tarantool-patches @ 2024-08-19 9:45 UTC (permalink / raw) To: Sergey Bronnikov; +Cc: tarantool-patches, Sergey Bronnikov Hi, Sergey! Thanks for the patch! As this thread replies grew way beyond comprehensible limit, I'll refer to the GitHub commit instead. > Sergey Bronnikov: > * added the description and the trimmed the test for the problem Typo: s/the trimmed/trimmed/ <lj-865-cross-generation-mach-o-file.test.lua> Lines: 149-150: > -- The function builds a Mach-O object file and retrieves its > -- header fields. I believe, "validates" fits better than "retrieves" here. Line: 153: > -- LuaJIT always generate 64-bit, non-FAT Mach-O object files. Typo: s/generate/generates/ Otherwise, LGTM. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files. 2024-08-19 9:45 ` Maxim Kokryashkin via Tarantool-patches @ 2024-08-20 8:57 ` Sergey Bronnikov via Tarantool-patches 0 siblings, 0 replies; 11+ messages in thread From: Sergey Bronnikov via Tarantool-patches @ 2024-08-20 8:57 UTC (permalink / raw) To: Maxim Kokryashkin; +Cc: tarantool-patches, Sergey Bronnikov [-- Attachment #1: Type: text/plain, Size: 1384 bytes --] Hi, Max thanks for review! On 19.08.2024 12:45, Maxim Kokryashkin wrote: > Hi, Sergey! > Thanks for the patch! > > As this thread replies grew way beyond comprehensible limit, I'll refer > to the GitHub commit instead. > >> Sergey Bronnikov: >> * added the description and the trimmed the test for the problem > Typo: s/the trimmed/trimmed/ > > <lj-865-cross-generation-mach-o-file.test.lua> > > Lines: 149-150: >> -- The function builds a Mach-O object file and retrieves its >> -- header fields. > I believe, "validates" fits better than "retrieves" here. > > Line: 153: >> -- LuaJIT always generate 64-bit, non-FAT Mach-O object files. > Typo: s/generate/generates/ Both typos were fixed: --- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua @@ -146,11 +146,11 @@ local function read_mach_o_hdr(buf, hw_arch) return mach_header end --- The function builds a Mach-O object file and retrieves its +-- The function builds a Mach-O object file and validates its -- header fields. local function build_and_check_mach_o(subtest) local hw_arch = subtest.name - -- LuaJIT always generate 64-bit, non-FAT Mach-O object files. + -- LuaJIT always generates 64-bit, non-FAT Mach-O object files. assert(hw_arch == 'arm64') subtest:plan(5) > > Otherwise, LGTM. [-- Attachment #2: Type: text/html, Size: 2612 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-08-20 8:57 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-07-05 13:10 [Tarantool-patches] [PATCH luajit 0/2] Fixes in Mach-O generation code Sergey Bronnikov via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 1/2] Fix typo Sergey Bronnikov via Tarantool-patches 2024-07-09 12:15 ` Sergey Kaplun via Tarantool-patches 2024-07-11 9:15 ` Maxim Kokryashkin via Tarantool-patches 2024-07-05 13:10 ` [Tarantool-patches] [PATCH luajit 2/2] OSX/iOS: Always generate 64 bit non-FAT Mach-O object files Sergey Bronnikov via Tarantool-patches 2024-07-09 13:03 ` Sergey Kaplun via Tarantool-patches 2024-07-10 12:43 ` Sergey Bronnikov via Tarantool-patches 2024-07-11 7:29 ` Sergey Kaplun via Tarantool-patches 2024-07-23 20:07 ` Sergey Bronnikov via Tarantool-patches 2024-08-19 9:45 ` Maxim Kokryashkin via Tarantool-patches 2024-08-20 8:57 ` Sergey Bronnikov via Tarantool-patches
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox