From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 3FEBF6EC55; Tue, 24 Aug 2021 17:55:23 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3FEBF6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1629816923; bh=GrHuLSLpby0u3GTBFPMlDsglWVPxwREKvY7SaiUjAVQ=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=XaEwYI7LWtMNjh2EM834prpzIzda5oJqODRdhNJ54HKKJ4zDrHh16iHX3UGtpiHBC acvnbvhDTXDdj4QsBa1uYsrsTuviYbuw8MQyGyef8Z4A93eq1rUD52SmXq1HLbwlvB wnmq70dFgKoxPLCxGgT0zhWKRlJHqLcI7zQ8sIRk= Received: from smtp40.i.mail.ru (smtp40.i.mail.ru [94.100.177.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3A49C6EC55 for ; Tue, 24 Aug 2021 17:55:21 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3A49C6EC55 Received: by smtp40.i.mail.ru with esmtpa (envelope-from ) id 1mIXpg-0000WC-0z; Tue, 24 Aug 2021 17:55:20 +0300 To: tarantool-patches@dev.tarantool.org, imun@tarantool.org, skaplun@tarantool.org Date: Tue, 24 Aug 2021 21:55:09 +0700 Message-Id: <20210824145509.53838-1-m.shishatskiy@tarantool.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD972FF4A7D76DB5E242D14FEF1BD8BF4AC182A05F5380850407DDBF3826B60218E8507C1848FC8169D5E973959B980C60EEE38225FFB0D0E2A X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE76C0A440987CA342DC2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE7026187065A2CADDDEA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F227110330A137A88B28B29078F625B918CC7F00164DA146DAFE8445B8C89999728AA50765F7900637F6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F7900637F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA73AA81AA40904B5D9A18204E546F3947C2A336C651863509103F1AB874ED890284AD6D5ED66289B52698AB9A7B718F8C46E0066C2D8992A16725E5C173C3A84C33C47BDC0B0347B17BA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7F45C1E71A9DFFA2A731C566533BA786AA5CC5B56E945C8DA X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458834459D11680B505BFEEBB8C4752451DF77A9BE6B6BB4180 X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CCF4356F00A22C40282FFD936B8646306C552EFD6659D34D99C2B6934AE262D3EE7EAB7254005DCED1EE92CF13E5D5A7792E1F3B950BCE21616EB5DE968479BF5B2CF3E77A3BD9B2CBDC6A1CF3F042BAD6DF99611D93F60EF505D71D783575ABE699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34697E0FA301E28215F6782B6CBA24006061C7ABD96623E4C9380666415B13E3513AE1E68241ADDC661D7E09C32AA3244C1ADF592C5D0EFF86C4CEF533184F8EDD8580396430872480927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioj4xwK62BfefsYfGiSYqwkkw== X-Mailru-Sender: 2317F5BEA8D613097CB8A225516EF027D7F9A6A065C5F253A3142CA3419CB84D44A2D1A3EC88BB52FF6B26DEAE20951894E739346DD9ABB838AE394B99C5394F99323991B04ED3930F27244EEAA5B9A5AE208404248635DF X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit v1] memprof: enrich symtab when new trace is allocated X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mikhail Shishatskiy via Tarantool-patches Reply-To: Mikhail Shishatskiy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Since symtab can be enriched with new prototypes, it can be enriched with new traces as well. This feature can help to investigate trace-generation-heavy apps with memprof. This patch introduces the functionality described above by adding a `is_streamed` flag to GCtrace. If the profiler meets allocation from a trace, it checks if the trace was already streamed to the symtab. If not, extra information is provided with the allocation source location. | loc := loc-lua | loc-lua-sym | loc-c | loc-trace | loc-trace-sym | loc-trace := trace-no | loc-trace-sym := trace-no sym-addr line-no The profiler parser is adjusted to recognize new entries with additional information about the trace's starting point. Since this patch, traces are no longer identified by address, as all the name collisions can be resolved. To avoid traceno collisions `trace generations` are introduced: now the trace parser module adds an additional number, representing the number of traces overridden before. So, when the parser meets a new trace entry overriding the existing trace, it increments the generation counter by one. | TRACE [:] started at ... Also, the API of changed: function `parse_sym_trace` is now public in order to use it from the module. Follows up tarantool/tarantool#5815 --- Branch: https://github.com/tarantool/luajit/tree/shishqa/enrich-symtab-when-trace-is-allocated Tarantool branch: https://github.com/tarantool/tarantool/tree/shishqa/enrich-symtab-when-trace-is-allocated The results of running benchmark [1] are presented below. The time is average for 15 runs. The second parameter is the standard deviation. | -------------------------------------- | | JIT-on, memprof-on | | -------------------------------------- | | BEFORE | AFTER | | ------------------ | ----------------- | | 2.9010s (0.0481) | +0.0936s (0.0557) | | -------------------------------------- | [1]: https://gist.github.com/Shishqa/9753063a258e38520d3c2f09d18378a5 src/lj_jit.h | 4 +++ src/lj_memprof.c | 27 ++++++++++++------- src/lj_memprof.h | 5 ++-- src/lj_trace.c | 3 +++ .../misclib-memprof-lapi.test.lua | 15 ++++++----- tools/memprof/humanize.lua | 2 +- tools/memprof/parse.lua | 6 ++++- tools/utils/symtab.lua | 21 ++++++++++----- 8 files changed, 55 insertions(+), 28 deletions(-) diff --git a/src/lj_jit.h b/src/lj_jit.h index d82292f8..c456b787 100644 --- a/src/lj_jit.h +++ b/src/lj_jit.h @@ -254,7 +254,11 @@ typedef struct GCtrace { uint8_t sinktags; /* Trace has SINK tags. */ uint8_t topslot; /* Top stack slot already checked to be allocated. */ uint8_t linktype; /* Type of link. */ +#if LJ_HASMEMPROF + uint8_t is_streamed; +#else uint8_t unused1; +#endif #ifdef LUAJIT_USE_GDBJIT void *gdbjit_entry; /* GDB JIT entry. */ #endif diff --git a/src/lj_memprof.c b/src/lj_memprof.c index fb217a2c..05266acb 100644 --- a/src/lj_memprof.c +++ b/src/lj_memprof.c @@ -30,7 +30,7 @@ static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION, #if LJ_HASJIT -static void dump_symtab_trace(struct lj_wbuf *out, const GCtrace *trace) +static void dump_symtab_trace(struct lj_wbuf *out, GCtrace *trace) { const GCproto *pt = &gcref(trace->startpt)->pt; BCLine lineno = -1; @@ -42,8 +42,6 @@ static void dump_symtab_trace(struct lj_wbuf *out, const GCtrace *trace) lineno = lj_debug_line(pt, proto_bcpos(pt, startpc)); lua_assert(lineno >= 0); - lj_wbuf_addbyte(out, SYMTAB_TRACE); - lj_wbuf_addu64(out, (uint64_t)trace->mcode); lj_wbuf_addu64(out, (uint64_t)trace->traceno); /* ** All the existing prototypes have already been dumped, so we do not @@ -51,6 +49,8 @@ static void dump_symtab_trace(struct lj_wbuf *out, const GCtrace *trace) */ lj_wbuf_addu64(out, (uintptr_t)pt); lj_wbuf_addu64(out, (uint64_t)lineno); + + trace->is_streamed = 1; } #else @@ -85,6 +85,7 @@ static void dump_symtab(struct lj_wbuf *out, const struct global_State *g) break; } case (~LJ_TTRACE): { + lj_wbuf_addbyte(out, SYMTAB_TRACE); dump_symtab_trace(out, gco2trace(o)); break; } @@ -216,13 +217,19 @@ static void memprof_write_func(struct memprof *mp, uint8_t aevent) static void memprof_write_trace(struct memprof *mp, uint8_t aevent) { struct lj_wbuf *out = &mp->out; - const global_State *g = mp->g; - const jit_State *J = G2J(g); - const TraceNo traceno = g->vmstate; - const GCtrace *trace = traceref(J, traceno); - lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE); - lj_wbuf_addu64(out, (uintptr_t)trace->mcode); - lj_wbuf_addu64(out, (uint64_t)traceno); + global_State *g = mp->g; + GCtrace *trace = traceref(G2J(g), g->vmstate); + + if (LJ_UNLIKELY(!trace->is_streamed)) { + + lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE | LJM_SYMTAB); + dump_symtab_trace(out, trace); + + } else { + + lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE); + lj_wbuf_addu64(out, (uint64_t)g->vmstate); + } } #else diff --git a/src/lj_memprof.h b/src/lj_memprof.h index 0f5b4c6d..c7156fe9 100644 --- a/src/lj_memprof.h +++ b/src/lj_memprof.h @@ -73,11 +73,12 @@ ** event-realloc := event-header loc? oaddr osize naddr nsize ** event-free := event-header loc? oaddr osize ** event-header := -** loc := loc-lua | loc-lua-sym | loc-c | loc-trace +** loc := loc-lua | loc-lua-sym | loc-c | loc-trace | loc-trace-sym ** loc-lua := sym-addr line-no ** loc-lua-sym := sym-addr sym-chunk sym-line line-no ** loc-c := sym-addr -** loc-trace := trace-addr trace-no +** loc-trace := trace-no +** loc-trace-sym := trace-no sym-addr line-no ** sym-addr := ** sym-chunk := string ** sym-line := diff --git a/src/lj_trace.c b/src/lj_trace.c index 86563cdb..fd0da095 100644 --- a/src/lj_trace.c +++ b/src/lj_trace.c @@ -163,6 +163,9 @@ static void trace_save(jit_State *J, GCtrace *T) #ifdef LUAJIT_USE_PERFTOOLS perftools_addtrace(T); #endif +#if LJ_HASMEMPROF + T->is_streamed = 0; +#endif } void LJ_FASTCALL lj_trace_free(global_State *g, GCtrace *T) diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua index f4666da5..6f030efa 100644 --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua @@ -276,7 +276,7 @@ test:test("jit-output", function(subtest) return end - subtest:plan(3) + subtest:plan(4) jit.opt.start(3, "hotloop=10") jit.flush() @@ -291,14 +291,15 @@ test:test("jit-output", function(subtest) -- See also https://github.com/tarantool/tarantool/issues/5679. subtest:ok(alloc[0] == nil) - -- Run already generated traces. - symbols, events = generate_parsed_output(default_payload) - - alloc = fill_ev_type(events, symbols, "alloc") - -- We expect, that loop will be compiled into a trace. + -- 10 allocations in interpreter mode, 1 allocation for a trace + -- recording and assembling and next 9 allocations will happen + -- inside the trace. + subtest:ok(check_alloc_report( + { line = 39, function_line = 32 }, alloc, 11 + )) subtest:ok(check_alloc_report( - { traceno = 1, line = 37, function_line = 32 }, alloc, 20 + { traceno = 1, line = 37, function_line = 32 }, alloc, 9 )) -- See same checks with jit.off(). subtest:ok(check_alloc_report( diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua index 72e71080..2dc5314c 100644 --- a/tools/memprof/humanize.lua +++ b/tools/memprof/humanize.lua @@ -94,7 +94,7 @@ function M.describe_location(symbols, loc) -- If trace, which was remembered in the symtab, has not -- been flushed, assotiate it with a proto, where trace -- recording started. - if trace and trace.addr == loc.addr then + if trace then return symtab.demangle(symbols, loc).." started at ".. symtab.demangle(symbols, trace.sym_loc) end diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua index a540360b..00640fa7 100644 --- a/tools/memprof/parse.lua +++ b/tools/memprof/parse.lua @@ -84,6 +84,10 @@ local function parse_location_symbols(reader, asource, symbols) 0, symbols ) + elseif asource == ASOURCE_TRACE then + return id_location( + 0, 0, symtab.parse_sym_trace(reader, symbols), symbols + ) end error("Unknown asource "..asource) end @@ -96,7 +100,7 @@ local function parse_location_common(reader, asource, symbols) elseif asource == ASOURCE_LFUNC then return id_location(reader:read_uleb128(), reader:read_uleb128(), 0, symbols) elseif asource == ASOURCE_TRACE then - return id_location(reader:read_uleb128(), 0, reader:read_uleb128(), symbols) + return id_location(0, 0, reader:read_uleb128(), symbols) end error("Unknown asource "..asource) end diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua index 88fdb42e..c2776790 100644 --- a/tools/utils/symtab.lua +++ b/tools/utils/symtab.lua @@ -33,25 +33,31 @@ function M.parse_sym_lfunc(reader, symtab) return sym_addr end -local function parse_sym_trace(reader, symtab) - local trace_addr = reader:read_uleb128() +function M.parse_sym_trace(reader, symtab) local traceno = reader:read_uleb128() local sym_addr = reader:read_uleb128() local sym_line = reader:read_uleb128() + local prev_gen = 0 + if symtab.trace[traceno] then + prev_gen = symtab.trace[traceno].gen + end + symtab.trace[traceno] = { - addr = trace_addr, + gen = prev_gen + 1, sym_loc = { addr = sym_addr, line = sym_line, traceno = 0, }, } + + return traceno end local parsers = { [SYMTAB_LFUNC] = M.parse_sym_lfunc, - [SYMTAB_TRACE] = parse_sym_trace, + [SYMTAB_TRACE] = M.parse_sym_trace, } function M.parse(reader) @@ -119,13 +125,14 @@ local function demangle_lfunc(symtab, loc) return string_format("CFUNC %#x", addr) end -local function demangle_trace(loc) - return string_format("TRACE [%d] 0x%x", loc.traceno, loc.addr) +local function demangle_trace(symtab, loc) + local traceno = loc.traceno + return string_format("TRACE [%d:%d]", traceno, symtab.trace[traceno].gen) end function M.demangle(symtab, loc) if loc.traceno ~= 0 then - return demangle_trace(loc) + return demangle_trace(symtab, loc) end return demangle_lfunc(symtab, loc) end -- 2.32.0