From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id CFCD36ECE3; Mon, 1 Nov 2021 19:31:20 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CFCD36ECE3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1635784281; bh=xF2JXGeK3RQ8dRuvj7to8FtVF9WUL3JeGdJXPbOGv2w=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=wruPsrzCQN0+jzXkGJkkmk42Y72osBpXKXkZmxm0WNGrVTtmUwXmrNgPthDZfhzQg vg2Ohv9Ku5EKIYt9i1Xk93FS69qVIpTwKJasQL1cLDvPBWqNlVfiWoaWf7ruVS6pV3 3WKn962xEXjRvBWndTb8jZjaWHrZNx3oK0LHmlLI= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id EF9196ECE3 for ; Mon, 1 Nov 2021 19:31:19 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org EF9196ECE3 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1mhaDO-0008Iv-Tv; Mon, 01 Nov 2021 19:31:19 +0300 Date: Mon, 1 Nov 2021 19:31:14 +0300 To: Mikhail Shishatskiy Message-ID: <20211101163113.GE8831@tarantool.org> References: <20210929200758.149446-1-m.shishatskiy@tarantool.org> <20210929200758.149446-5-m.shishatskiy@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210929200758.149446-5-m.shishatskiy@tarantool.org> X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.10.1 (2018-07-13) X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD98893113B6235BE91D14F0A97254290008E820E7E92A2F633182A05F53808504035122702BCEAADD1CEA6381C0515825069CC8ED73A33C070419EB498B1E1B510 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7E2331B2371EFE129EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006372764AB4096FC590A8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8387E9B04353B4E6E118B2FF5BAC2FF65117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B642416645EBD45DC2089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458834459D11680B50594CAD59B14691FA146FD04A998461368 X-C1DE0DAB: 0D63561A33F958A5533D60186C842391807840D3053B7F18E85A2F6DB1B5DBC1D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA752DA3D96DA0CEF5C48E8E86DC7131B365E7726E8460B7C23C X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A503FBFE8BE8FC49B0341EA9952CB36073A69CC678F85A84A280F497DF928054F5D4E1BC29295AD81D7E09C32AA3244C882935BC003E82D368B96F9620C2D15F60759606DA2E136A927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojsm8N/O5xSHSPeYk7rerDZA== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D3CDA05E9DD310FE98A1B0AC86EEC55CEA7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v4 4/4] memprof: add info about trace start to symtab X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Misha, Thanks for the patch! Please consider my comments below. On 29.09.21, Mikhail Shishatskiy wrote: > Trace allocation sources, recorded by the memory profiler, > were reported as Minor: As for me, it's better to say it the following way: | Allocation events occurred on traces are recorded by the memory | profiler the following way now > > | TRACE [] > > This approach is not descriptive enough to understand, where > exactly allocation took place, as we do not know the code > chunk, associated with the trace. > > This patch fixes the problem described above by extending the > symbol table with entries, consisting of a trace's > mcode starting address, trace number, address of function proto, > and line, where trace recording started: > > | sym-trace := sym-header trace-no trace-addr sym-addr sym-line > | trace-no := > | trace-addr := > > The memory profiler parser is adjusted to recognize the entries > mentioned above. On top of that, the API of changed: > now table with symbols contains two tables: `lfunc` for Lua functions > symbols and `trace` for trace entries. > > The demangler module has not changed, but the function > `describe_location` is added to the module, > which allows one to get a description of the trace location in the > format described below: > > | TRACE [] started at @: > > Follows up tarantool/tarantool#5814 > --- > > Issue: https://github.com/tarantool/tarantool/issues/5814 > Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > CI: https://github.com/tarantool/tarantool/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > > src/lj_memprof.c | 43 +++++++++++++++++++ > src/lj_memprof.h | 8 +++- > .../misclib-memprof-lapi.test.lua | 15 ++++--- > tools/memprof.lua | 4 +- > tools/memprof/humanize.lua | 30 ++++++++++--- > tools/memprof/process.lua | 9 ++-- > tools/utils/symtab.lua | 31 ++++++++++--- > 7 files changed, 118 insertions(+), 22 deletions(-) > > diff --git a/src/lj_memprof.c b/src/lj_memprof.c > index 8702557f..e8b2ebbc 100644 > --- a/src/lj_memprof.c > +++ b/src/lj_memprof.c > @@ -28,6 +28,45 @@ > static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION, > 0x0, 0x0, 0x0}; > > +#if LJ_HASJIT > + > +static void dump_symtab_trace(struct lj_wbuf *out, const GCtrace *trace) > +{ > + GCproto *pt = &gcref(trace->startpt)->pt; > + BCLine lineno = 0; > + > + const BCIns *startpc = mref(trace->startpc, const BCIns); > + lua_assert(startpc >= proto_bc(pt) && > + startpc < proto_bc(pt) + pt->sizebc); > + > + lineno = lj_debug_line(pt, proto_bcpos(pt, startpc)); > + lua_assert(lineno >= 0); I doubt whether this assertion is not always true. If it is, then what does it check? > + > diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua > index 3f4ffea0..b9edb80d 100644 > --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua > +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua > @@ -87,9 +87,13 @@ local function fill_ev_type(events, symbols, event_type) > local addr = event.loc.addr > local traceno = event.loc.traceno > > - if traceno ~= 0 then > + if traceno ~= 0 and symbols.trace[traceno] then > + local trace_loc = symbols.trace[traceno].start > + addr = trace_loc.addr > ev_type.trace[traceno] = { > - name = string.format("TRACE [%d]", traceno), > + name = string.format("TRACE [%d] %s:%d", > + traceno, symbols.lfunc[addr].source, symbols.lfunc[addr].linedefined need to be used as a last argument in instead of , don't you? > + ), > num = event.num, > } > elseif addr == 0 then > @@ -116,7 +120,8 @@ end > local function check_alloc_report(alloc, traceno, line, function_line, nevents) > local expected_name, event > if traceno ~= 0 then > - expected_name = string.format("TRACE [%d]", traceno) > + expected_name = string.format("TRACE [%d] ", traceno).. > + form_source_line(function_line) The output format differs from the one produced by memprof parser, doesn't it? > event = alloc.trace[traceno] > else > expected_name = form_source_line(function_line) > diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua > index 7771005d..7d30f976 100644 > --- a/tools/memprof/humanize.lua > +++ b/tools/memprof/humanize.lua > @@ -7,6 +7,23 @@ local symtab = require "utils.symtab" > > local M = {} > > +function M.describe_location(symbols, loc) There is no need to export from humanize.lua, so it can be just a local function within this Lua chunk. > + if loc.traceno == 0 then > + return symtab.demangle(symbols, loc) > + end > + > + local trace = symbols.trace[loc.traceno] > + > + -- If trace, which was remembered in the symtab, has not > + -- been flushed, assotiate it with a proto, where trace Typo: s/assotiate/associate/. > + -- recording started. > + if trace and trace.addr == loc.addr then > + return symtab.demangle(symbols, loc).." started at ".. > + symtab.demangle(symbols, trace.start) Finally, I got the thing that bothers me the most. Why do you make so complex? It looks that you can move all these if-else branching to and concatenation to function, doesn't it? AFAICS, you can remove as a result and trace demangling will be encapsulated in scope of function. Feel free to correct me if I'm wrong. > + end > + return symtab.demangle(symbols, loc) > +end > + > diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua > index 85945fb2..496d8480 100644 > --- a/tools/utils/symtab.lua > +++ b/tools/utils/symtab.lua > @@ -24,18 +25,38 @@ local function parse_sym_lfunc(reader, symtab) > local sym_chunk = reader:read_string() > local sym_line = reader:read_uleb128() > > - symtab[sym_addr] = { > + symtab.lfunc[sym_addr] = { > source = sym_chunk, > linedefined = sym_line, > } > end > > +local function parse_sym_trace(reader, symtab) > + local traceno = reader:read_uleb128() > + local trace_addr = reader:read_uleb128() > + local sym_addr = reader:read_uleb128() > + local sym_line = reader:read_uleb128() > + > + symtab.trace[traceno] = { > + addr = trace_addr, > + start = { > + addr = sym_addr, > + line = sym_line, > + traceno = 0, Please, leave a comment regarding the fact the structure is the same as the one yielded from in memprof/parse.lua. > + }, > + } > +end > + > -- > 2.33.0 > -- Best regards, IM