Re: [Tarantool-patches] [PATCH luajit v4 4/4] memprof: add info about trace start to symtab

[Igor Munkin via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

Misha,

Thanks for the patch! Please consider my comments below.

On 29.09.21, Mikhail Shishatskiy wrote:
> Trace allocation sources, recorded by the memory profiler,
> were reported as

Minor: As for me, it's better to say it the following way:
| Allocation events occurred on traces are recorded by the memory
| profiler the following way now

> 
> | TRACE [<trace-no>] <trace-addr>
> 
> This approach is not descriptive enough to understand, where
> exactly allocation took place, as we do not know the code
> chunk, associated with the trace.
> 
> This patch fixes the problem described above by extending the
> symbol table with <sym-trace> entries, consisting of a trace's
> mcode starting address, trace number, address of function proto,
> and line, where trace recording started:
> 
> | sym-trace  := sym-header trace-no trace-addr sym-addr sym-line
> | trace-no   := <ULEB128>
> | trace-addr := <ULEB128>
> 
> The memory profiler parser is adjusted to recognize the entries
> mentioned above. On top of that, the API of <utils/symtab.lua> changed:
> now table with symbols contains two tables: `lfunc` for Lua functions
> symbols and `trace` for trace entries.
> 
> The demangler module has not changed, but the function
> `describe_location` is added to the <memprof/humanize.lua> module,
> which allows one to get a description of the trace location in the
> format described below:
> 
> | TRACE [<trace-no>] <trace-addr> started at @<sym-chunk>:<sym-line>
> 
> Follows up tarantool/tarantool#5814
> ---
> 
> Issue: https://github.com/tarantool/tarantool/issues/5814
> Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number
> CI: https://github.com/tarantool/tarantool/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number
> 
>  src/lj_memprof.c                              | 43 +++++++++++++++++++
>  src/lj_memprof.h                              |  8 +++-
>  .../misclib-memprof-lapi.test.lua             | 15 ++++---
>  tools/memprof.lua                             |  4 +-
>  tools/memprof/humanize.lua                    | 30 ++++++++++---
>  tools/memprof/process.lua                     |  9 ++--
>  tools/utils/symtab.lua                        | 31 ++++++++++---
>  7 files changed, 118 insertions(+), 22 deletions(-)
> 
> diff --git a/src/lj_memprof.c b/src/lj_memprof.c
> index 8702557f..e8b2ebbc 100644
> --- a/src/lj_memprof.c
> +++ b/src/lj_memprof.c
> @@ -28,6 +28,45 @@
>  static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION,
>  					   0x0, 0x0, 0x0};
>  
> +#if LJ_HASJIT
> +
> +static void dump_symtab_trace(struct lj_wbuf *out, const GCtrace *trace)
> +{
> +  GCproto *pt = &gcref(trace->startpt)->pt;
> +  BCLine lineno = 0;
> +
> +  const BCIns *startpc = mref(trace->startpc, const BCIns);
> +  lua_assert(startpc >= proto_bc(pt) &&
> +             startpc < proto_bc(pt) + pt->sizebc);
> +
> +  lineno = lj_debug_line(pt, proto_bcpos(pt, startpc));
> +  lua_assert(lineno >= 0);

I doubt whether this assertion is not always true. If it is, then what
does it check?

> +

<snipped>

> diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua
> index 3f4ffea0..b9edb80d 100644
> --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua
> +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua
> @@ -87,9 +87,13 @@ local function fill_ev_type(events, symbols, event_type)
>      local addr = event.loc.addr
>      local traceno = event.loc.traceno
>  
> -    if traceno ~= 0 then
> +    if traceno ~= 0 and symbols.trace[traceno] then
> +      local trace_loc = symbols.trace[traceno].start
> +      addr = trace_loc.addr
>        ev_type.trace[traceno] = {
> -        name = string.format("TRACE [%d]", traceno),
> +        name = string.format("TRACE [%d] %s:%d",
> +          traceno, symbols.lfunc[addr].source, symbols.lfunc[addr].linedefined

<trace_loc.line> need to be used as a last argument in <string.format>
instead of <symbols.lfunc[addr].linedefined>, don't you?

> +        ),
>          num = event.num,
>        }
>      elseif addr == 0 then

<snipped>

> @@ -116,7 +120,8 @@ end
>  local function check_alloc_report(alloc, traceno, line, function_line, nevents)
>    local expected_name, event
>    if traceno ~= 0 then
> -    expected_name = string.format("TRACE [%d]", traceno)
> +    expected_name = string.format("TRACE [%d] ", traceno)..
> +                    form_source_line(function_line)

The output format differs from the one produced by memprof parser,
doesn't it?

>      event = alloc.trace[traceno]
>    else
>      expected_name = form_source_line(function_line)

<snipped>

> diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua
> index 7771005d..7d30f976 100644
> --- a/tools/memprof/humanize.lua
> +++ b/tools/memprof/humanize.lua
> @@ -7,6 +7,23 @@ local symtab = require "utils.symtab"
>  
>  local M = {}
>  
> +function M.describe_location(symbols, loc)

There is no need to export <describe_location> from humanize.lua, so it
can be just a local function within this Lua chunk.

> +  if loc.traceno == 0 then
> +    return symtab.demangle(symbols, loc)
> +  end
> +
> +  local trace = symbols.trace[loc.traceno]
> +
> +  -- If trace, which was remembered in the symtab, has not
> +  -- been flushed, assotiate it with a proto, where trace

Typo: s/assotiate/associate/.

> +  -- recording started.
> +  if trace and trace.addr == loc.addr then
> +    return symtab.demangle(symbols, loc).." started at "..
> +           symtab.demangle(symbols, trace.start)

Finally, I got the thing that bothers me the most. Why do you make
<describe_location> so complex? It looks that you can move all these
if-else branching to <symtab.demangle> and concatenation to
<demangle_trace> function, doesn't it? AFAICS, you can remove
<describe_location> as a result and trace demangling will be
encapsulated in scope of <demangle_trace> function. Feel free to correct
me if I'm wrong.

> +  end
> +  return symtab.demangle(symbols, loc)
> +end
> +

<snipped>

> diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua
> index 85945fb2..496d8480 100644
> --- a/tools/utils/symtab.lua
> +++ b/tools/utils/symtab.lua

<snipped>

> @@ -24,18 +25,38 @@ local function parse_sym_lfunc(reader, symtab)
>    local sym_chunk = reader:read_string()
>    local sym_line = reader:read_uleb128()
>  
> -  symtab[sym_addr] = {
> +  symtab.lfunc[sym_addr] = {
>      source = sym_chunk,
>      linedefined = sym_line,
>    }
>  end
>  
> +local function parse_sym_trace(reader, symtab)
> +  local traceno = reader:read_uleb128()
> +  local trace_addr = reader:read_uleb128()
> +  local sym_addr = reader:read_uleb128()
> +  local sym_line = reader:read_uleb128()
> +
> +  symtab.trace[traceno] = {
> +    addr = trace_addr,
> +    start = {
> +      addr = sym_addr,
> +      line = sym_line,
> +      traceno = 0,

Please, leave a comment regarding the fact the structure is the same as
the one yielded from <parse_location> in memprof/parse.lua.

> +    },
> +  }
> +end
> +

<snipped>

> -- 
> 2.33.0
> 

-- 
Best regards,
IM