From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id CAF6847DF11; Mon, 29 May 2023 17:34:19 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CAF6847DF11 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1685370859; bh=vNBn/KUaX1auFmQCn1bunUPjfP8zBXJOiRu6BgUfqDA=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=ILGQDbjwJ9l4PVpMyJVaTjYRszIwcuCq47nP/btPjoQ+yVEzt22xFpD7jPTDbUu3Q sQSbQBObEAVHaAPKwiRnGtgWwKeY6dz4m7xbOzvFxilG5Ci6SeMUn0q9YpWlIHG5cq 5hWHznEQ75S+LzIjaErLhUicDrYxHdS7VL2hsmXA= Received: from smtp56.i.mail.ru (smtp56.i.mail.ru [95.163.41.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id C71CB47DF11 for ; Mon, 29 May 2023 17:34:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org C71CB47DF11 Received: by smtp56.i.mail.ru with esmtpa (envelope-from ) id 1q3dwv-004OFY-RE; Mon, 29 May 2023 17:34:18 +0300 Message-ID: <10155d67-2330-d942-59b0-e57be206d96c@tarantool.org> Date: Mon, 29 May 2023 17:34:17 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Content-Language: en-US To: Sergey Kaplun , Sergey Bronnikov References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9FA9FE7958A526B779D7BAB7BD016A2C842947415451E34B400894C459B0CD1B9E086F25EC0354692091CBDB919C985333C2EFC9ED7DB8620E49391A50CB82CE1 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7C6A43EB42C8B28B9EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637790A9327A9AFEF4F8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D88AB51E58FD17911646543A8A5CAA7832117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC6BA66BB79834B351A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F4460429728776938767073520437C869540D2AB0FC26CFBAC0749D213D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE902A1BE408319B2986750DBF4DC650F2D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE34E7D9683544204AF6136E347CC761E07C4224003CC836476E2F48590F00D11D6E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F7900637C970FD8DF19C51D2EFF80C71ABB335746BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: 0D63561A33F958A5AB055647358CC0F25F4615C89BA810D4180A554079D5EF75F87CCE6106E1FC07E67D4AC08A07B9B01DAA61796BF5227BCB5012B2E24CD356 X-C8649E89: 1C3962B70DF3F0ADE00A9FD3E00BEEDF3FED46C3ACD6F73ED3581295AF09D3DF87807E0823442EA2ED31085941D9CD0AF7F820E7B07EA4CF18A913383F6B56ED29281375B9B280796E1827A43F5D2704C01BF478E4A36AFF8DB4AB58792EC0ECA0A56A47F2C7DF64AFC2106F9B9EC16CA3E65602AA43F8F2F4E8A8FB6BF8EBF5 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojjwJ7vqadnRvA2eWTmXHJ5A== X-Mailru-Sender: 11C2EC085EDE56FAC07928AF2646A7692CC3411CEC8C1B9D088033D096CBB70452685D71546B8E9EEBA65886582A37BD66FEC6BF5C9C28D98A98C1125256619760D574B6FC815AB872D6B4FCE48DF648AE208404248635DF X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] Fix saved bytecode encapsulated in ELF objects. X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi, Sergey! Thanks for review, please see my answers (and patches with fixes) below. Changes force-pushed, and I'll send a second version to the mailing list. Sergey On 5/23/23 16:09, Sergey Kaplun wrote: > Hi, Sergey! > Thanks for the patch! > Please consider my comments below. > > On 22.05.23, Sergey Bronnikov wrote: >> From: Sergey Bronnikov >> >> Thanks to Dimitry Andric. >> >> (cherry picked from commit 7dbf0b05f1228c1c719866db5e5f3d58f87f74c8) >> >> Function `bcsave.lua:bcsave_elfobj()` produced an object file in ELF >> format with a wrong size size of `.strtab`. Wrong size causes lld to >> show an error message similar to: `ld: error: obj/bytecode.o: string >> table non-null terminated`. >> >> Sergey Bronnikov: >> * added the description and the test for the problem >> >> Signed-off-by: Sergey Bronnikov >> --- >> Branch: https://github.com/tarantool/luajit/tree/ligurio/gh-366-strtab-section-correct-size >> PR: https://github.com/tarantool/tarantool/pull/8678 >> --- >> src/jit/bcsave.lua | 2 +- >> .../lj-366-strtab-correct-size.test.lua | 148 ++++++++++++++++++ >> 2 files changed, 149 insertions(+), 1 deletion(-) >> create mode 100644 test/tarantool-tests/lj-366-strtab-correct-size.test.lua >> >> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua >> index c17c88e0..2553d97e 100644 >> --- a/src/jit/bcsave.lua >> +++ b/src/jit/bcsave.lua >> @@ -275,7 +275,7 @@ typedef struct { >> o.sect[2].size = fofs(ofs) >> o.sect[3].type = f32(3) -- .strtab >> o.sect[3].ofs = fofs(sofs + ofs) >> - o.sect[3].size = fofs(#symname+1) >> + o.sect[3].size = fofs(#symname+2) >> ffi.copy(o.space+ofs+1, symname) >> ofs = ofs + #symname + 2 >> o.sect[4].type = f32(1) -- .rodata >> diff --git a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua >> new file mode 100644 >> index 00000000..d4b51537 >> --- /dev/null >> +++ b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua >> @@ -0,0 +1,148 @@ >> +local tap = require('tap') >> +local test = tap.test('lj-366-strtab-correct-size'):skipcond({ >> + -- Test is ELF-specific and because LuaJIT exports object files in ELF format > Typo: s/Test/The test/ > Typo: s/, and/and/ Fixed! > > Nit: Comment line is wider than 66 symbols. > Here and below. Fixed! --- a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua +++ b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua @@ -1,8 +1,9 @@  local tap = require('tap')  local test = tap.test('lj-366-strtab-correct-size'):skipcond({ -  -- Test is ELF-specific and because LuaJIT exports object files in ELF format -  -- for all operating systems except macOS and Windows we skip test on -  -- these OSes. See src/jit/bcsave.lua:bcsave_obj. +  -- The test is ELF-specific, and because LuaJIT exports object +  -- files in ELF format for all operating systems except macOS +  -- and Windows we skip test on these OSes. +  -- See src/jit/bcsave.lua:bcsave_obj.    ['Disabled on Windows'] = jit.os == 'Windows',    ['Disabled on macOS'] = jit.os == 'OSX',  }) >> + -- for all operating systems except macOS and Windows we skip test on >> + -- these OSes. See src/jit/bcsave.lua:bcsave_obj. >> + ['Disabled on Windows'] = jit.os == 'Windows', > Side note: I'm not sure, that we shall mention Windows, when we still > don't support it. Still, this skipcond is necessary, so I'm OK with it. > >> + ['Disabled on macOS'] = jit.os == 'OSX', >> +}) >> + >> +local ffi = require 'ffi' >> + >> +-- Reference numbers for strtab offset and size could be obtained with >> +-- readelf(1). Note that number system of these number is hexadecimal. > Typo: s/number/numbers/ Fixed. @@ -10,7 +11,7 @@ local test = tap.test('lj-366-strtab-correct-size'):skipcond({  local ffi = require 'ffi'  -- Reference numbers for strtab offset and size could be obtained with --- readelf(1). Note that number system of these number is hexadecimal. +-- readelf(1). Note that number system of these numbers are hexadecimal.  --  -- $ luajit -b -n "module_name" -e "print()" xxx.obj  -- $ readelf --section-headers xxx.obj >> +-- >> +-- $ luajit -b -n "module_name" -e "print()" xxx.obj >> +-- $ readelf --section-headers xxx.obj > Minor: May you also provide the linker command with the failure > mentioned in the issue? Sure:     $ luajit  -b -n "module_name" -e "print()" xxx.obj     $ ld.lld xxx.obj     ld.lld: error: xxx.obj: SHT_STRTAB string table section [index 3] is non-null terminated Added to commit message. > Side note: I suppose that the starting \0 byte [1] was forgotten to > count. :) >> +-- There are 6 section headers, starting at offset 0x40: >> +-- >> +-- Section Headers: >> +-- [Nr] Name Type Address Offset >> +-- Size EntSize Flags Link Info Align >> +-- ... >> +-- >> +-- [ 3] .strtab STRTAB 0000000000000000 00000223 >> +-- 0000000000000017 0000000000000000 0 0 1 >> +-- ... >> + >> +local expected_strtab_size = 23 -- == 0x17 > Minor: Also, comment about this size (length of > `LJBC_PREFIX..ctx.modname` + 2) is desirable. Maybe we should copy this > string and calculate the size here as well. > Feel free to ignore. Added. > >> +local expected_strtab_offset = 547 -- == 0x223 > Why don't declare as 0x17, 0x223? Because I forgot that Lua allows to use hex numbers. Fixed. > >> + >> +-- Defined in elf.h. >> +local sht_symtab = 2 > Minor: Maybe it is better to use UPPER-CASE for constants? > Feel free to ignore. Switched to upper-case. > >> + >> +ffi.cdef[[ >> +typedef struct { >> + uint8_t emagic[4], eclass, eendian, eversion, eosabi, eabiversion, epad[7]; >> + uint16_t type, machine; >> + uint32_t version; >> + uint32_t entry, phofs, shofs; >> + uint32_t flags; >> + uint16_t ehsize, phentsize, phnum, shentsize, shnum, shstridx; >> +} ELF32header; > I suggest to add additional empty line between each declaration for > better readability. Added. > >> +typedef struct { >> + uint8_t emagic[4], eclass, eendian, eversion, eosabi, eabiversion, epad[7]; >> + uint16_t type, machine; >> + uint32_t version; >> + uint64_t entry, phofs, shofs; >> + uint32_t flags; >> + uint16_t ehsize, phentsize, phnum, shentsize, shnum, shstridx; >> +} ELF64header; >> +typedef struct { >> + uint32_t name, type, flags, addr, ofs, size, link, info, align, entsize; >> +} ELF32sectheader; >> +typedef struct { >> + uint32_t name, type; >> + uint64_t flags, addr, ofs, size; >> + uint32_t link, info; >> + uint64_t align, entsize; >> +} ELF64sectheader; >> +typedef struct { >> + uint32_t name, value, size; >> + uint8_t info, other; >> + uint16_t sectidx; >> +} ELF32symbol; > I have the following declaration in , maybe we should use the the > same declarations ad litteram? Just to avoid inconsistencies in naming? > The same for all other structures defined. > > | typedef struct > | { > | Elf32_Word st_name; /* Symbol name (string tbl index) */ > | Elf32_Addr st_value; /* Symbol value */ > | Elf32_Word st_size; /* Symbol size */ > | unsigned char st_info; /* Symbol type and binding */ > | unsigned char st_other; /* Symbol visibility */ > | Elf32_Section st_shndx; /* Section index */ > | } Elf32_Sym; > > Or we may use the same as defined in (as you did), > but the comment about it is desirable. C definitions were copy-pasted from bcsave.lua, added a comment about it. > >> +typedef struct { >> + uint32_t name; >> + uint8_t info, other; >> + uint16_t sectidx; >> + uint64_t value, size; >> +} ELF64symbol; >> +typedef struct { >> + ELF32header hdr; >> + ELF32sectheader sect[6]; >> + ELF32symbol sym[2]; >> + uint8_t space[4096]; >> +} ELF32obj; >> +typedef struct { >> + ELF64header hdr; >> + ELF64sectheader sect[6]; >> + ELF64symbol sym[2]; >> + uint8_t space[4096]; >> +} ELF64obj; >> +]] >> + >> +local is64_arch = { >> + ["x64"] = true, > Nit: please, use single-quotes for strings, here and below. Fixed. > >> + ["arm64"] = true, >> + ["arm64be"] = true, >> + ["ppc"] = false, >> + ["mips"] = false, > Why do we need ppc, and mips definitions here? I would left unsupported arch's here as well, it will simplify a bit work for those who will port test to these arch's in a future. > >> +} >> + >> +local is64 = is64_arch[jit.arch] or false >> +local ELFobj_type = ffi.typeof(is64 and "ELF64obj *" or "ELF32obj *") >> +local ELFsectheader_type = ffi.typeof(is64 and "ELF64sectheader *" or "ELF32sectheader") > Should it be `ELF32sectheader *`? Sure, fixed. > >> + >> +-- Reads a file located in a specified path and returns its content. >> +local function read_file(path) >> + local file = assert(io.open(path), 'cannot open an object file') >> + local content = file:read("*a") >> + file:close() >> + return content >> +end >> + >> +-- Parses a buffer in an ELF format and returns an offset and a size of strtab >> +-- section for a symtab. >> +local function read_elf_strtab(elf_content) >> + local elf = ffi.cast(ELFobj_type, elf_content) >> + local elf_header = elf.hdr >> + local sec_strtab >> + for i = 0, elf_header.shnum do >> + local sec = ffi.cast(ELFsectheader_type, elf.sect[i]) >> + if sec.type == sht_symtab then > Why don't just to find .strtab instead? > LuaJIT emits really a small obj file, (it's literally the third section). strtab is an abstract section, it could be used not for storing symbols but for other purposes too. in this test we need a strtab section for symtab. > >> + sec_strtab = ffi.cast(ELFsectheader_type, elf.sect[sec.link]) >> + break >> + end >> + end >> + return sec_strtab.size, sec_strtab.ofs >> +end >> + >> +local function lua_bin_path(arg) > Don't get why do we need this function to get luabin? > Why just `luacmd` is not enough? luacmd is not suitable for my test without fixing helper. The problem was in passing additional arguments to a final luajit cmd. It looked that easier don't use luacmd than fix it for my purposes. > >> + local args_str = require('utils').luacmd(arg) >> + local args = {} >> + for a in args_str:gmatch("%S+") do >> + table.insert(args, a); >> + end >> + return args[1] > Why do we need to parse the whole string if we return only first > occurence of %S+? Fixed. >> +end >> + >> +test:plan(5) >> + >> +local elf_filename = os.tmpname() .. '.obj' >> +local lua_path = os.getenv('LUA_PATH') >> +local lua_bin = lua_bin_path(arg) >> +local cmd = ('LUA_PATH="%s" %s -b -n "module_name" -e "print()" %s'):format(lua_path, lua_bin, elf_filename) > Nit: Code line is wider than 80 symbols. Fixed. > >> +local ret = os.execute(cmd) >> +test:ok(ret == 0, 'create an object file') > I suggest to use `assert()` for these checks for test correcness. > `test:*()` functions -- to check some bugs > `assert()` -- to verify test correctness or validity > So, we have only 2 test to plan: for size and offset. With asserts I don't like that it breaks test report. According to TAP spec there is "bail out" that should handle cases when something goes wrong and we couldn't continue testing. In our implementation there is not "bail out" support. Replaced test:ok() to asserts where checks are related to test environment and output  now looks as below: TAP version 13 1..2 ok - check .strtab size ok - check .strtab offset > >> + >> +local elf_content = read_file(elf_filename) >> +test:ok(#elf_content ~= 0, 'read an object file') >> +local strtab_size, strtab_offset = read_elf_strtab(elf_content) >> +test:ok(strtab_size == expected_strtab_size, 'check .strtab size') >> +test:ok(strtab_offset == expected_strtab_offset, 'check .strtab offset') >> + >> +ret = os.execute(("rm -f %s"):format(elf_filename)) > I suppose, that we may use `os.remove()` here. Sure, replaced to os.remove(). > >> +test:ok(ret == 0, 'remove an object file') >> + >> +os.exit(test:check() and 0 or 1) >> -- >> 2.34.1 >> > [1]: https://refspecs.linuxbase.org/elf/gabi4+/ch4.strtab.html >