Re: [Tarantool-patches] [PATCH luajit v4 1/2] memprof: extend symtab with C-symbols

[Sergey Kaplun via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Hi, Maxim!

Thanks for the patch! Please consider my comments below.

On 30.10.21, Maxim Kokryashkin wrote:
> This commit enriches memprof's symbol table and event stream with
> information about C-symbols. That information will provide demangling
> capabilities to the parser.

I suggest to reorganize patches:
1) One patch for C symbols dumping in symtab and their demangling in
   parser with corresponding tests.
2) The other patch for symtab extension on dlopen with corresponding
   tests.

So, the first patch increases only LJS_VERSION (for dumper and parser).
The second one increases only LJM_VERSION, since symtab format is the
same, but the stream format from memprof is changed.

I've reviewed only the first part of this patch about C symbols dumping.

> 
> The following data is stored in symtab for each symbol:
> | SYMTAB_CFUNC | symbol address | symbol name |
>   1 byte            8 bytes
>   magic
>   number
> 
> The following data is stored in event for each newly loaded symbol:
> | (AEVENT_SYMTAB | ASOURCE_CFUNC) | symbol address | symbol name |
>               1 byte                   8 bytes
>               magic
>               number

What do you think of dumping so name too for convenience?
For example, dump an empty so name stands for Tarantool|LuaJIT sources,
but the other one is related to some other .so library that is loaded by
a user.
CC-ed Igor here.

> 
> Part of tarantool/tarantool#5813
> ---
>  src/lj_memprof.c | 159 ++++++++++++++++++++++++++++++++++++++++++++---
>  src/lj_memprof.h |  17 +++--
>  2 files changed, 165 insertions(+), 11 deletions(-)
> 
> diff --git a/src/lj_memprof.c b/src/lj_memprof.c
> index 2c1ef3b8..30c9697c 100644
> --- a/src/lj_memprof.c
> +++ b/src/lj_memprof.c
> @@ -5,10 +5,16 @@
>  ** Copyright (C) 2015-2019 IPONWEB Ltd.
>  */
>  
> +#define _GNU_SOURCE
> +#include <elf.h>
> +

Nit: I suggest to move it down after LUA_CORE.

>  #define lj_memprof_c
>  #define LUA_CORE
>  
>  #include <errno.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <link.h>
>  
>  #include "lj_arch.h"
>  #include "lj_memprof.h"
> @@ -24,7 +30,123 @@
>  static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION,
>  					   0x0, 0x0, 0x0};

Should we update LJS header due to the new format of the symtab stream?

>  
> -static void dump_symtab(struct lj_wbuf *out, const struct global_State *g)
> +struct ghashtab_header {
> +    uint32_t nbuckets;
> +    uint32_t symoffset;
> +    uint32_t bloom_size;
> +    uint32_t bloom_shift;
> +};
> +
> +uint32_t ghashtab_size(ElfW(Addr) ghashtab)
> +{
> +    /*
> +     * There is no easy way to get count of symbols in GNU hashtable, so the
> +     * only way to do this is to iterate over it.
> +     */

Nit: please use LuaJIT style for comments.

> +    uint32_t last_entry = 0;
> +    uint32_t *cur_bucket = NULL;
> +    uint32_t *entry = NULL;
> +
> +    const void *chain_address = NULL;
> +    struct ghashtab_header *header = (struct ghashtab_header*)ghashtab;
> +    const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) +

What is the size of data pointed by (void *)? What is the result of this
pointer arithmetic? Also, the compiler warns about this too:

| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c: In function 'ghashtab_size':
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:52:43: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith]
|    52 |     const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) +
|       |                                           ^
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:52:76: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith]
|    52 |     const void *buckets = (void*)ghashtab + sizeof(struct ghashtab_header) +
|       |                                                                            ^
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:65:29: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith]
|    65 |     chain_address = buckets + sizeof(uint32_t) * header->nbuckets;
|       |                             ^
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:67:43: warning: pointer of type 'void *' used in arithmetic [-Wpointer-arith]
|    67 |         entry = (uint32_t*)(chain_address + (last_entry - header->symoffset) *

> +                          sizeof(uint64_t) * header->bloom_size;

I believe that it is not uint64_t for 32bit build.

> +
> +    cur_bucket = (uint32_t*)buckets;

Also, please drop a comment about .gnu.hash. structure:

Am I understanding correctly that:
We take the highest possible *non empty* bucket (the first cycle) and
iterate through its symbols until the last chain is over (the LSB is set
to 1).

> +    for (uint32_t i = 0; i < header->nbuckets; ++i) {
> +        if (last_entry < *cur_bucket)
> +            last_entry = *cur_bucket;
> +        cur_bucket++;
> +    }
> +
> +    if (last_entry < header->symoffset)
> +        return header->symoffset;
> +
> +    chain_address = buckets + sizeof(uint32_t) * header->nbuckets;
> +    do {

Can we just write that part like the follwing:
| while ((chain[last_entry - header->symoffset] & 1) == 0)
|   last_entry++;
Where `uint32_t *chain = (uint32_t *)chain_address;`?

> +        last_entry++;

It is good to leave a comment, that chain ends with the lowest bit set
to 1.

> +    } while (!(*entry & 1));
> +
> +    return last_entry;
> +}
> +
> +struct symbol_resolver_conf {
> +  struct lj_wbuf *buf;
> +  const uint8_t header;
> +
> +  uint32_t cur_lib;
> +  uint32_t lib_cnt_prev;
> +  uint32_t to_dump_cnt;
> +  uint32_t *lib_cnt;
> +};
> +
> +int resolve_symbolnames(struct dl_phdr_info *info, size_t info_size, void *data)
> +{
> +  if (strcmp(info->dlpi_name, "linux-vdso.so.1") == 0) {
> +    return 0;
> +  }

For some arches it has different names (see `man 7 vdso`).
I suggest to define its name with a macro.

Side note: Also, why .hash and .gnu.hash seems different in format for this
library? Why does parsing crash if comments those lines?

Nit: {} are excess.

> +
> +  ElfW(Dyn*) dyn = NULL;
> +  ElfW(Sym*) sym = NULL;
> +  ElfW(Word*) hashtab = NULL;
> +  ElfW(Word) sym_cnt = 0;
> +
> +  char *strtab = 0;
> +  char *sym_name = 0;
> +
> +  struct symbol_resolver_conf *conf = data;
> +  const uint8_t header = conf->header;
> +  struct lj_wbuf *buf = conf->buf;

Nit: please move all declarations to the corresponding scopes.
Also, please add `const` type qualifier, where it is possible.

> +
> +  conf->lib_cnt_prev = *conf->lib_cnt;
> +  uint32_t lib_cnt_prev = conf->lib_cnt_prev;

Please, fix the following warnings too:

| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c: In function 'resolve_symbolnames':
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:106:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
|   106 |   uint32_t lib_cnt_prev = conf->lib_cnt_prev;
|       |   ^~~~~~~~
| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:112:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
|   112 |   uint32_t lib_cnt = info->dlpi_adds - info->dlpi_subs;

> +
> +  if ((conf->to_dump_cnt = info->dlpi_adds - lib_cnt_prev) == 0)

Nit: AFAICS, dlpi_adds field was added in glibc 2.4. Manual recommends
to check the size of the structure if we use those fields. At least we
can add assert on size for glibc version.

Also, you should fix the following warning:

| /home/burii/reviews/luajit/csymbols/src/lj_memprof.c:87:59: warning: unused parameter 'info_size' [-Wunused-parameter]
|    87 | int resolve_symbolnames(struct dl_phdr_info *info, size_t info_size, void *data)
|       |

> +    /* No new libraries, stop resolver. */
> +    return 1;
> +
> +  uint32_t lib_cnt = info->dlpi_adds - info->dlpi_subs;
> +  if (conf->cur_lib < lib_cnt - conf->to_dump_cnt) {
> +    /* That lib is already dumped, skip it. */
> +    ++conf->cur_lib;
> +    return 0;
> +  }
> +
> +  for (size_t header_index = 0; header_index < info->dlpi_phnum; ++header_index) {
> +    if (info->dlpi_phdr[header_index].p_type == PT_DYNAMIC) {
> +      dyn = (ElfW(Dyn)*)(info->dlpi_addr +  info->dlpi_phdr[header_index].p_vaddr);
> +
> +      while(dyn->d_tag != DT_NULL) {
> +        if (dyn->d_tag == DT_HASH) {

Only the ELF header has a fixed position in the file. The flexibility of
the ELF format requires no specified order for header tables, sections
or segments [1]. So, we can't rely on their order and first we need to
list all program headers and safe their addresses we need and only after
start working with them (like it is done in readelf, for example).

Also, switch-case operator looks preferable here.

> +          hashtab = (ElfW(Word*))dyn->d_un.d_ptr;
> +          sym_cnt = hashtab[1];

OK, this part is not obvious. So please drop a comment about that. Something
like that:

| A hash table consists of Elf32_Word or Elf64_Word objects that provide for
| symbol table access. Hash table has the following organization:
| +-------------------+
| |      nbucket      |
| +-------------------+
| |      nchain       |
| +-------------------+
| |     bucket[0]     |
| |       ...         |
| | bucket[nbucket-1] |
| +-------------------+
| |      chain[0]     |
| |       ...         |
| |  chain[nchain-1]  |
| +-------------------+
| Chain table entries parallel the symbol table. The number of symbol
| table entries should equal nchain, so symbol table indexes also select
| chain table entries. Since the chain array values are indexes for not only
| the chain array itself, but also for the symbol table, the chain array must
| be the same size as the symbol table. This makes nchain equal to the length
| of the symbol table [2].

Side note: man ld says about the type of hash (.hash or .gnu.hash):
| for most Linux based systems it will be "both"
But gcc by default used (behind the scenes) `-Wl,--hash=gnu`, AFAICS.

> +        }

Please use {} everywhere in else if statements if you use it in any of
them [3].

> +        else if (dyn->d_tag == DT_GNU_HASH && sym_cnt == 0)
> +          sym_cnt = ghashtab_size(dyn->d_un.d_ptr);

OK, there are several more simple (for me) options:

-1) Not calculate .gnu.hash tab size by iterating the whole symtab. We
    can take the highest possible bucket (nbuckets - 1) and iterate
    through its symbols until the last chain is over. It is better
    because we don't need to iterate through all buckets.
0) We can dump all necessary information while we iterating through the
   dyn symtab. So we have no need to walk through it twice.
1) If we have access to section headers, we can simply calculate the
   total number of symbols from section's header: just divide the section
   size by an entry size.

> +        else if (dyn->d_tag == DT_STRTAB)
> +          strtab = (char*)dyn->d_un.d_ptr;
> +        else if (dyn->d_tag == DT_SYMTAB) {
> +          sym = (ElfW(Sym*))dyn->d_un.d_ptr;
> +
> +          for (ElfW(Word) sym_index = 0; sym_index < sym_cnt; sym_index++) {

Index 0 both designates the first entry in the table and serves as the
undefined symbol index [4]. So we can just start from the 1st index for the
symtab in the dynamic section.

> +              sym_name = &strtab[sym[sym_index].st_name];

Should we dump *all* symbols? Why we can't just dump symbols with type, that
equals STT_FUNC?

> +              lj_wbuf_addbyte(buf, header);
> +              lj_wbuf_addu64(buf, sym[sym_index].st_value + info->dlpi_addr);
> +              lj_wbuf_addstring(buf, sym_name);
> +          }
> +        }
> +        dyn++;
> +      }

It is good to use .dynsym symbol table to get CFUNC locations, but
it doesn't provide LOCAL symbols. It is better to parse .symtab to get
them. Why it is useful:

Assume, we have the following code:

| static __attribute__((noinline)) lib_func1(lua_State *L)
| {
| 	/* Allocations here. */
| }
|
| static  __attribute__((noinline)) lib_func2(lua_State *L)
| {
| 	/* Allocations here. */
| }
|
| static lib_func3(lua_State *L)
| {
| 	/* Allocations here. */
| 	if (...)
| 		lib_func1(L);
| 	else
| 		lib_func2(L);
| 	/* More allocations. */
| }

IINM, without .symtab information we can't determine the source of
allocation for now? As I believe the most common usage of LuaC library
is the following [5]:

|  static int l_dir (lua_State *L) {
|     ...  /* as before */
|  }
|
|
|  static const struct luaL_reg mylib [] = {
|    {"dir", l_dir},
|    {NULL, NULL}  /* sentinel */
|  };
|
|
|  int luaopen_mylib (lua_State *L) {
|    luaL_openlib(L, "mylib", mylib, 0);
|    return 1;
|  }

For that case user can't see the source of allocation, AFAICS.
So, we must dump symtab (if it is not stripped), else try to dump
only dynsym hoping for the best. Or even don't try to demangle these
libraries at all.

Also, in this case we should to work with parsing carefully -- we can't
just find the nearest symbol. First, we should check, that .symtab for
the library is not stripped and dumped to memprof's symtab. If it isn't
we can only report that allocation was in the libname.so with
corresponding address (if we want to dump so name).

> +    }
> +  }
> +
> +  ++conf->cur_lib;
> +  return 0;
> +}
> +
> +static void dump_symtab(struct lj_wbuf *out, const struct global_State *g, uint32_t *lib_cnt)
>  {
>    const GCRef *iter = &g->gc.root;
>    const GCobj *o;
> @@ -49,6 +171,17 @@ static void dump_symtab(struct lj_wbuf *out, const struct global_State *g)

<snipped>

> @@ -78,6 +211,7 @@ struct memprof {
>    struct alloc orig_alloc; /* Original allocator. */
>    struct lj_memprof_options opt; /* Profiling options. */
>    int saved_errno; /* Saved errno when profiler deinstrumented. */
> +  uint32_t lib_cnt; /* Number of currently loaded libs. */

I've not dived deeply, but I'm not sure in the usage of this:
What happens if 2 libraries was unloaded, one load again and one new
load instead the old one?

>  };
>  
>  static struct memprof memprof = {0};
> @@ -105,15 +239,26 @@ static void memprof_write_lfunc(struct lj_wbuf *out, uint8_t aevent,
>  }
>  

<snipped>

> diff --git a/src/lj_memprof.h b/src/lj_memprof.h
> index 3417475d..337fa76a 100644
> --- a/src/lj_memprof.h
> +++ b/src/lj_memprof.h

<snipped>

> -- 
> 2.33.0
> 

[1]: https://docs.oracle.com/cd/E19683-01/816-1386/6m7qcoblj/index.html
[2]: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-48031.html
[3]: https://www.kernel.org/doc/html/v4.10/process/coding-style.html#placing-braces-and-spaces
[4]: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-79797.html
[5]: https://www.lua.org/pil/26.2.html

-- 
Best regards,
Sergey Kaplun