From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id D19886F3E5; Tue, 21 Dec 2021 17:40:25 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org D19886F3E5 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1640097626; bh=Y0955xcJ3MTep8RRc6jAvLP/4e/71qstdSu95fGfrQQ=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=cCScZ3pbwbClSKHa5LE3p/PkGphQdgkAN/6TitxbmUwTjMnpdWR5Fp1mTwpjNWoYn sCyXJIjGBrZYhTWAYEwNr2kXUYqLAevpF2Xmrj+obJutnnTFEkBYh9VhWyUQMayUDv N0XjPpcYoM8XYsdjieAMF+DFd+3H081gFmuS4dxc= Received: from smtp57.i.mail.ru (smtp57.i.mail.ru [217.69.128.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id F25F06F3E5 for ; Tue, 21 Dec 2021 17:40:24 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org F25F06F3E5 Received: by smtp57.i.mail.ru with esmtpa (envelope-from ) id 1mzgJT-0005m0-Uq; Tue, 21 Dec 2021 17:40:24 +0300 Date: Tue, 21 Dec 2021 17:38:33 +0300 To: Maxim Kokryashkin Message-ID: References: <6759290ff3f50f8a819d940b30c8903c9cd8883c.1635600120.git.m.kokryashkin@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6759290ff3f50f8a819d940b30c8903c9cd8883c.1635600120.git.m.kokryashkin@tarantool.org> X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9B5397E24C93BDA6712610AD69032D75DE6A3E17E48EF72E3182A05F538085040DB959D0EC31ECA1879A2F125368829D1DA4F9B28566B8512DE0E82EEDE6A219D X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7F68DA2F3749BB650EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006374C960BD2D03BF0BDEA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F2C6266CE7ACE04E105EC104D311DFE319CC7F00164DA146DAFE8445B8C89999728AA50765F790063783E00425F71A4181389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC8D2DCF9CF1F528DBCF6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA7E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8B569F1129A2C6445075ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A51A51979E2D294841E408980E0CC12CAFD1FADAA826CC6987D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA759D2A03B9C34326B3410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D041FB2E16F174C46C5CB896C0358D6C0CCD1D3271975171B620164B1146A8FC123537FE6D09269F1D7E09C32AA3244C4F6FBC3BDAA940024DBB57C66AB863D2435BF7150578642F927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojj1F9ODneAx3E0PvbQSTgXg== X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB4B021A81E2E79F4B4A2CC3C70861BDBBBA9939E4A8E2ACA42F2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A84198E0F3ECE9B5443453F38A29522196 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v4 1/2] memprof: extend symtab with C-symbols X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi, Maxim! Thanks for the patch! Please consider my comments below. On 30.10.21, Maxim Kokryashkin wrote: > This commit enriches memprof's symbol table and event stream with > information about C-symbols. That information will provide demangling > capabilities to the parser. I suggest to reorganize patches: 1) One patch for C symbols dumping in symtab and their demangling in parser with corresponding tests. 2) The other patch for symtab extension on dlopen with corresponding tests. So, the first patch increases only LJS_VERSION (for dumper and parser). The second one increases only LJM_VERSION, since symtab format is the same, but the stream format from memprof is changed. I've reviewed only the first part of this patch about C symbols dumping. > > The following data is stored in symtab for each symbol: > | SYMTAB_CFUNC | symbol address | symbol name | > 1 byte 8 bytes > magic > number > > The following data is stored in event for each newly loaded symbol: > | (AEVENT_SYMTAB | ASOURCE_CFUNC) | symbol address | symbol name | > 1 byte 8 bytes > magic > number What do you think of dumping so name too for convenience? For example, dump an empty so name stands for Tarantool|LuaJIT sources, but the other one is related to some other .so library that is loaded by a user. CC-ed Igor here. > > Part of tarantool/tarantool#5813 > --- > src/lj_memprof.c | 159 ++++++++++++++++++++++++++++++++++++++++++++--- > src/lj_memprof.h | 17 +++-- > 2 files changed, 165 insertions(+), 11 deletions(-) > > diff --git a/src/lj_memprof.c b/src/lj_memprof.c > index 2c1ef3b8..30c9697c 100644 > --- a/src/lj_memprof.c > +++ b/src/lj_memprof.c > @@ -5,10 +5,16 @@ > ** Copyright (C) 2015-2019 IPONWEB Ltd. > */ > > +#define _GNU_SOURCE > +#include > + Nit: I suggest to move it down after LUA_CORE. > #define lj_memprof_c > #define LUA_CORE > > #include > +#include > +#include > +#include > > #include "lj_arch.h" > #include "lj_memprof.h" > @@ -24,7 +30,123 @@ > static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION, > 0x0, 0x0, 0x0}; Should we update LJS header due to the new format of the symtab stream? > > -static void dump_symtab(struct lj_wbuf *out, const struct global_State *g) > +struct ghashtab_header { > + uint32_t nbuckets; > + uint32_t symoffset; > + uint32_t bloom_size; > + uint32_t bloom_shift; > +}; > + > +uint32_t ghashtab_size(ElfW(Addr) ghashtab) > +{ > + /* > + * There is no easy way to get count of symbols in GNU hashtable, so the > + * only way to do this is to iterate over it. > + */ Nit: please use LuaJIT style for comments. > + uint32_t last_entry = 0; > + uint32_t *cur_bucket = NULL; > + uint32_t *entry = NULL; > + > + const void *chain_address = NULL; > + struct ghashtab_header *header = (struct ghashtab_header*)ghashtab; > + const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) + What is the size of data pointed by (void *)? What is the result of this pointer arithmetic? Also, the compiler warns about this too: | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c: In function 'ghashtab_size': | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:52:43: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith] | 52 | const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) + | | ^ | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:52:76: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith] | 52 | const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) + | | ^ | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:65:29: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith] | 65 | chain_address = buckets + sizeof(uint32_t) * header->nbuckets; | | ^ | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:67:43: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith] | 67 | entry = (uint32_t*)(chain_address + (last_entry - header->symoffset) * > + sizeof(uint64_t) * header->bloom_size; I believe that it is not uint64_t for 32bit build. > + > + cur_bucket = (uint32_t*)buckets; Also, please drop a comment about .gnu.hash. structure: Am I understanding correctly that: We take the highest possible *non empty* bucket (the first cycle) and iterate through its symbols until the last chain is over (the LSB is set to 1). > + for (uint32_t i = 0; i < header->nbuckets; ++i) { > + if (last_entry < *cur_bucket) > + last_entry = *cur_bucket; > + cur_bucket++; > + } > + > + if (last_entry < header->symoffset) > + return header->symoffset; > + > + chain_address = buckets + sizeof(uint32_t) * header->nbuckets; > + do { Can we just write that part like the follwing: | while ((chain[last_entry - header->symoffset] & 1) == 0) | last_entry++; Where `uint32_t *chain = (uint32_t *)chain_address;`? > + last_entry++; It is good to leave a comment, that chain ends with the lowest bit set to 1. > + } while (!(*entry & 1)); > + > + return last_entry; > +} > + > +struct symbol_resolver_conf { > + struct lj_wbuf *buf; > + const uint8_t header; > + > + uint32_t cur_lib; > + uint32_t lib_cnt_prev; > + uint32_t to_dump_cnt; > + uint32_t *lib_cnt; > +}; > + > +int resolve_symbolnames(struct dl_phdr_info *info, size_t info_size, void *data) > +{ > + if (strcmp(info->dlpi_name, "linux-vdso.so.1") == 0) { > + return 0; > + } For some arches it has different names (see `man 7 vdso`). I suggest to define its name with a macro. Side note: Also, why .hash and .gnu.hash seems different in format for this library? Why does parsing crash if comments those lines? Nit: {} are excess. > + > + ElfW(Dyn*) dyn = NULL; > + ElfW(Sym*) sym = NULL; > + ElfW(Word*) hashtab = NULL; > + ElfW(Word) sym_cnt = 0; > + > + char *strtab = 0; > + char *sym_name = 0; > + > + struct symbol_resolver_conf *conf = data; > + const uint8_t header = conf->header; > + struct lj_wbuf *buf = conf->buf; Nit: please move all declarations to the corresponding scopes. Also, please add `const` type qualifier, where it is possible. > + > + conf->lib_cnt_prev = *conf->lib_cnt; > + uint32_t lib_cnt_prev = conf->lib_cnt_prev; Please, fix the following warnings too: | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c: In function 'resolve_symbolnames': | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:106:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement] | 106 | uint32_t lib_cnt_prev = conf->lib_cnt_prev; | | ^~~~~~~~ | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:112:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement] | 112 | uint32_t lib_cnt = info->dlpi_adds - info->dlpi_subs; > + > + if ((conf->to_dump_cnt = info->dlpi_adds - lib_cnt_prev) == 0) Nit: AFAICS, dlpi_adds field was added in glibc 2.4. Manual recommends to check the size of the structure if we use those fields. At least we can add assert on size for glibc version. Also, you should fix the following warning: | /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:87:59: warning: unused parameter 'info_size' [-Wunused-parameter] | 87 | int resolve_symbolnames(struct dl_phdr_info *info, size_t info_size, void *data) | | > + /* No new libraries, stop resolver. */ > + return 1; > + > + uint32_t lib_cnt = info->dlpi_adds - info->dlpi_subs; > + if (conf->cur_lib < lib_cnt - conf->to_dump_cnt) { > + /* That lib is already dumped, skip it. */ > + ++conf->cur_lib; > + return 0; > + } > + > + for (size_t header_index = 0; header_index < info->dlpi_phnum; ++header_index) { > + if (info->dlpi_phdr[header_index].p_type == PT_DYNAMIC) { > + dyn = (ElfW(Dyn)*)(info->dlpi_addr + info->dlpi_phdr[header_index].p_vaddr); > + > + while(dyn->d_tag != DT_NULL) { > + if (dyn->d_tag == DT_HASH) { Only the ELF header has a fixed position in the file. The flexibility of the ELF format requires no specified order for header tables, sections or segments [1]. So, we can't rely on their order and first we need to list all program headers and safe their addresses we need and only after start working with them (like it is done in readelf, for example). Also, switch-case operator looks preferable here. > + hashtab = (ElfW(Word*))dyn->d_un.d_ptr; > + sym_cnt = hashtab[1]; OK, this part is not obvious. So please drop a comment about that. Something like that: | A hash table consists of Elf32_Word or Elf64_Word objects that provide for | symbol table access. Hash table has the following organization: | +-------------------+ | | nbucket | | +-------------------+ | | nchain | | +-------------------+ | | bucket[0] | | | ... | | | bucket[nbucket-1] | | +-------------------+ | | chain[0] | | | ... | | | chain[nchain-1] | | +-------------------+ | Chain table entries parallel the symbol table. The number of symbol | table entries should equal nchain, so symbol table indexes also select | chain table entries. Since the chain array values are indexes for not only | the chain array itself, but also for the symbol table, the chain array must | be the same size as the symbol table. This makes nchain equal to the length | of the symbol table [2]. Side note: man ld says about the type of hash (.hash or .gnu.hash): | for most Linux based systems it will be "both" But gcc by default used (behind the scenes) `-Wl,--hash=gnu`, AFAICS. > + } Please use {} everywhere in else if statements if you use it in any of them [3]. > + else if (dyn->d_tag == DT_GNU_HASH && sym_cnt == 0) > + sym_cnt = ghashtab_size(dyn->d_un.d_ptr); OK, there are several more simple (for me) options: -1) Not calculate .gnu.hash tab size by iterating the whole symtab. We can take the highest possible bucket (nbuckets - 1) and iterate through its symbols until the last chain is over. It is better because we don't need to iterate through all buckets. 0) We can dump all necessary information while we iterating through the dyn symtab. So we have no need to walk through it twice. 1) If we have access to section headers, we can simply calculate the total number of symbols from section's header: just divide the section size by an entry size. > + else if (dyn->d_tag == DT_STRTAB) > + strtab = (char*)dyn->d_un.d_ptr; > + else if (dyn->d_tag == DT_SYMTAB) { > + sym = (ElfW(Sym*))dyn->d_un.d_ptr; > + > + for (ElfW(Word) sym_index = 0; sym_index < sym_cnt; sym_index++) { Index 0 both designates the first entry in the table and serves as the undefined symbol index [4]. So we can just start from the 1st index for the symtab in the dynamic section. > + sym_name = &strtab[sym[sym_index].st_name]; Should we dump *all* symbols? Why we can't just dump symbols with type, that equals STT_FUNC? > + lj_wbuf_addbyte(buf, header); > + lj_wbuf_addu64(buf, sym[sym_index].st_value + info->dlpi_addr); > + lj_wbuf_addstring(buf, sym_name); > + } > + } > + dyn++; > + } It is good to use .dynsym symbol table to get CFUNC locations, but it doesn't provide LOCAL symbols. It is better to parse .symtab to get them. Why it is useful: Assume, we have the following code: | static __attribute__((noinline)) lib_func1(lua_State *L) | { | /* Allocations here. */ | } | | static __attribute__((noinline)) lib_func2(lua_State *L) | { | /* Allocations here. */ | } | | static lib_func3(lua_State *L) | { | /* Allocations here. */ | if (...) | lib_func1(L); | else | lib_func2(L); | /* More allocations. */ | } IINM, without .symtab information we can't determine the source of allocation for now? As I believe the most common usage of LuaC library is the following [5]: | static int l_dir (lua_State *L) { | ... /* as before */ | } | | | static const struct luaL_reg mylib [] = { | {"dir", l_dir}, | {NULL, NULL} /* sentinel */ | }; | | | int luaopen_mylib (lua_State *L) { | luaL_openlib(L, "mylib", mylib, 0); | return 1; | } For that case user can't see the source of allocation, AFAICS. So, we must dump symtab (if it is not stripped), else try to dump only dynsym hoping for the best. Or even don't try to demangle these libraries at all. Also, in this case we should to work with parsing carefully -- we can't just find the nearest symbol. First, we should check, that .symtab for the library is not stripped and dumped to memprof's symtab. If it isn't we can only report that allocation was in the libname.so with corresponding address (if we want to dump so name). > + } > + } > + > + ++conf->cur_lib; > + return 0; > +} > + > +static void dump_symtab(struct lj_wbuf *out, const struct global_State *g, uint32_t *lib_cnt) > { > const GCRef *iter = &g->gc.root; > const GCobj *o; > @@ -49,6 +171,17 @@ static void dump_symtab(struct lj_wbuf *out, const struct global_State *g) > @@ -78,6 +211,7 @@ struct memprof { > struct alloc orig_alloc; /* Original allocator. */ > struct lj_memprof_options opt; /* Profiling options. */ > int saved_errno; /* Saved errno when profiler deinstrumented. */ > + uint32_t lib_cnt; /* Number of currently loaded libs. */ I've not dived deeply, but I'm not sure in the usage of this: What happens if 2 libraries was unloaded, one load again and one new load instead the old one? > }; > > static struct memprof memprof = {0}; > @@ -105,15 +239,26 @@ static void memprof_write_lfunc(struct lj_wbuf *out, uint8_t aevent, > } > > diff --git a/src/lj_memprof.h b/src/lj_memprof.h > index 3417475d..337fa76a 100644 > --- a/src/lj_memprof.h > +++ b/src/lj_memprof.h > -- > 2.33.0 > [1]: https://docs.oracle.com/cd/E19683-01/816-1386/6m7qcoblj/index.html [2]: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-48031.html [3]: https://www.kernel.org/doc/html/v4.10/process/coding-style.html#placing-braces-and-spaces [4]: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-79797.html [5]: https://www.lua.org/pil/26.2.html -- Best regards, Sergey Kaplun