From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id ADA1271224; Wed, 27 Oct 2021 16:57:13 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ADA1271224 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1635343033; bh=sixEkcGvaRKch52ce3hXy+P23cSFrizcoc0HlJT4SBw=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=R9x5T+HwaYAFmyXwqUUKFxBv/f2qPMCLnU3PM69vThDyet2dYNXc5YF9LiCsbcame 3aFwpviQJT2lhcf+Gs2a+/070c9wArki1a+PvOEZ6T2tF2+JtXM1umi/ssXFsSwpxv WVwI9QFu7ej4dLmmHEZ3u9gEYSmRwVMWWs0tkIPQ= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 5F62371224 for ; Wed, 27 Oct 2021 16:57:00 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 5F62371224 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1mfjQJ-0001KW-DO; Wed, 27 Oct 2021 16:56:59 +0300 Date: Wed, 27 Oct 2021 16:56:55 +0300 To: Mikhail Shishatskiy Message-ID: <20211027135655.GD8831@tarantool.org> References: <20210929200758.149446-1-m.shishatskiy@tarantool.org> <20210929200758.149446-4-m.shishatskiy@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210929200758.149446-4-m.shishatskiy@tarantool.org> X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.10.1 (2018-07-13) X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9D1D35DBD2D15487EB9438C392F7C4E68414DBB3B161A4FBD182A05F5380850407DBB13671791EF858A9B13B8383E7AD4467970A25D5C6B3862512EF3D0B1E209 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7E50EC9128971FD6EEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B97DA3EE4E90F98B8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D86A1629735189C6CC2B8F5F59B31347BB117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCF1175FABE1C0F9B6A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD18C26CFBAC0749D213D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B67ECBC18655D52CDF089D37D7C0E48F6C5571747095F342E88FB05168BE4CE3AF X-C1DE0DAB: 0D63561A33F958A58BF2562336A7948D4F70321FECB67B77FC8D3ED2D56FEF5ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7567C209D01CC1E34B410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D342B1F2AD168155B061E6001097814C76DCC8D649C8A26A424AB1C8DD28348473CE60852ECBB0EB5981D7E09C32AA3244C1B316973FB0CA667C264BE82C23E0E8764EE5813BBCA3A9D927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojOHwMx23X6B0VO3ggusUFWQ== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D217B54C73F790E040C320D9061CFB61AA7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v4 3/4] memprof: group allocations on traces by traceno X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Misha, Thanks for the patch! Please, consider the comments below. On 29.09.21, Mikhail Shishatskiy wrote: > When LuaJIT executes a trace, the trace number is stored in Typo: s/in/as/. > the virtual machine state. So, we can treat this number as > an allocation event source in memprof and report allocation events > from traces as well. > > Previously, all the allocations from traces were marked as INTERNAL. > > This patch introduces the functionality described above by adding > a new allocation source type named ASOURCE_TRACE. If at the moment > when allocation event occurs VM state indicates that trace executed, Minor: To make the wording a bit clearer, I propose the following: | If allocation event occurs when trace is executed, trace number... > trace number and trace's mcode starting address streamed to a binary Typo: s/streamed/is streamed/. > file: > > | loc-trace := trace-no trace-addr > | trace-no := > | trace-addr := > > Also, the memory profiler parser is adjusted to recognize entries > mentioned above. The structure is extended with field , > representing trace number. Trace locations are demangled as > > | TRACE [] > > Resolves tarantool/tarantool#5814 Does this patch "resolve" the issue or only being a "part of" it? > --- > > Issue: https://github.com/tarantool/tarantool/issues/5814 > Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > CI: https://github.com/tarantool/tarantool/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > > src/Makefile.dep.original | 3 +- > src/lj_memprof.c | 36 ++++++- > src/lj_memprof.h | 14 ++- > .../misclib-memprof-lapi.test.lua | 97 +++++++++++++++---- > tools/memprof/parse.lua | 13 ++- > tools/utils/symtab.lua | 20 ++-- > 6 files changed, 148 insertions(+), 35 deletions(-) > > diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua > index 9de4bd98..3f4ffea0 100644 > --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua > +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua > @@ -96,10 +113,21 @@ local function form_source_line(line) > return string.format("@%s:%d", arg[0], line) > end > > -local function check_alloc_report(alloc, line, function_line, nevents) > - assert(form_source_line(function_line) == alloc[line].name) > - assert(alloc[line].num == nevents, ("got=%d, expected=%d"):format( > - alloc[line].num, > +local function check_alloc_report(alloc, traceno, line, function_line, nevents) OK, here we have a function with 5 parameters. Almost half of them is used in a single particular case. Hence, I propose to change the interface to the following: | local function check_alloc_report(alloc, location, nevents) In this case location is a table with either "traceno" key or with "line" and "linedefined" keys. Such function looks better, doesn't it? Consider the following: | check_alloc_report(alloc, { traceno = 1 }, 20) | check_alloc_report(alloc, { line = 34, linedefined = 32 }, 2) > + local expected_name, event > + if traceno ~= 0 then > + expected_name = string.format("TRACE [%d]", traceno) > + event = alloc.trace[traceno] > + else > + expected_name = form_source_line(function_line) > + event = alloc.line[line] > + end > + assert(expected_name == event.name, ("got='%s', expected='%s'"):format( > + event.name, > + expected_name > + )) > + assert(event.num == nevents, ("got=%d, expected=%d"):format( > + event.num, > nevents > )) > return true > @@ -145,18 +173,18 @@ test:test("output", function(subtest) > -- one is the number of allocations. 1 event - alocation of > -- table by itself + 1 allocation of array part as far it is > -- bigger than LJ_MAX_COLOSIZE (16). > - subtest:ok(check_alloc_report(alloc, 27, 25, 2)) > - -- 100 strings allocations. > - subtest:ok(check_alloc_report(alloc, 32, 25, 100)) > + subtest:ok(check_alloc_report(alloc, 0, 34, 32, 2)) > + -- 20 strings allocations. > + subtest:ok(check_alloc_report(alloc, 0, 39, 32, 20)) As a result of the change proposed above, you need no these ugly fixes for the existing spots in the future. > > -- Collect all previous allocated objects. > - subtest:ok(free.INTERNAL.num == 102) > + subtest:ok(free.INTERNAL.num == 22) > > -- Tests for leak-only option. > -- See also https://github.com/tarantool/tarantool/issues/5812. > local heap_delta = process.form_heap_delta(events, symbols) > - local tab_alloc_stats = heap_delta[form_source_line(27)] > - local str_alloc_stats = heap_delta[form_source_line(32)] > + local tab_alloc_stats = heap_delta[form_source_line(34)] > + local str_alloc_stats = heap_delta[form_source_line(39)] > subtest:ok(tab_alloc_stats.nalloc == tab_alloc_stats.nfree) > subtest:ok(tab_alloc_stats.dbytes == 0) > subtest:ok(str_alloc_stats.nalloc == str_alloc_stats.nfree) > @@ -185,5 +213,38 @@ test:test("stack-resize", function(subtest) > misc.memprof.stop() > end) > > +-- Test profiler with enabled JIT. > jit.on() > + > +test:test("jit-output", function(subtest) > + -- Disabled on *BSD due to #4819. > + if jit.os == 'BSD' then > + subtest:plan(1) > + subtest:skip('Disabled due to #4819') > + return > + end > + > + subtest:plan(3) > + > + jit.opt.start(3, "hotloop=10") > + jit.flush() > + > + -- Pregenerate traces to fill symtab entries in the next run. > + default_payload() > + > + local symbols, events = generate_parsed_output(default_payload) > + > + local alloc = fill_ev_type(events, symbols, "alloc") > + local free = fill_ev_type(events, symbols, "free") > + > + -- We expect, that loop will be compiled into a trace. > + subtest:ok(check_alloc_report(alloc, 1, 37, 32, 20)) What are 37 and 32 in this case? I can use 0 and 0 for both parameters and the test passes, doesn't it? This is another argument for using table with different number of keys. > + -- See same checks with jit.off(). > + subtest:ok(check_alloc_report(alloc, 0, 34, 32, 2)) > + subtest:ok(free.INTERNAL.num == 22) > + > + -- Restore default JIT settings. > + jit.opt.start(unpack(jit_opt_default)) > +end) > + > os.exit(test:check() and 0 or 1) > diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua > index e01daa62..85945fb2 100644 > --- a/tools/utils/symtab.lua > +++ b/tools/utils/symtab.lua > @@ -74,21 +74,29 @@ function M.parse(reader) > end > > function M.id(loc) > - return string_format("f%#xl%d", loc.addr, loc.line) > + return string_format("f%#xl%dt%d", loc.addr, loc.line, loc.traceno) > end > > -function M.demangle(symtab, loc) > +local function demangle_lfunc(symtab, loc) > local addr = loc.addr > > if addr == 0 then > return "INTERNAL" > - end > - > - if symtab[addr] then > + elseif symtab[addr] then > return string_format("%s:%d", symtab[addr].source, loc.line) > end > - Looks like all these changes are excess, aren't they? > return string_format("CFUNC %#x", addr) > end > > +local function demangle_trace(loc) > + return string_format("TRACE [%d] %#x", loc.traceno, loc.addr) > +end > + > +function M.demangle(symtab, loc) > + if loc.traceno ~= 0 then > + return demangle_trace(loc) > + end > + return demangle_lfunc(symtab, loc) Why 3 of 4 types of ASOURCE are demangled within a separate function (with a bit strange name), but the traces have a separate one function? Looks irrationale a bit, IMHO. Then either introduce a new function per ASOURCE (too crazy as for me) or just move this block for trace into the original function (the only logic way I see here). > +end > + > return M > -- > 2.33.0 > -- Best regards, IM