From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id C741F6EC55; Wed, 21 Jul 2021 14:48:41 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org C741F6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626868121; bh=xGdTrRRAEuNzWjEKsucaSEzT4MjiEcEODmP6MHNsXbQ=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=arXKkHVOpwnV1KEaW1S6Pn5h2Z/x7WKZnm9jEDXYMAbv+CjKWouM0E6NyjtO0RYCe ohYiE0NkNlAAKhF1QPhc6TVFzmap8MrPkslG1XNqCyOfymSVSB/SYA6ISQIQcKcyRd Ky+Kmec2YEV8jCqOIRKsgvoeNoLKf1o0OEOCpbEM= Received: from smtp34.i.mail.ru (smtp34.i.mail.ru [94.100.177.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id AF86D6EC55 for ; Wed, 21 Jul 2021 14:48:40 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org AF86D6EC55 Received: by smtp34.i.mail.ru with esmtpa (envelope-from ) id 1m6AiM-00040z-N2; Wed, 21 Jul 2021 14:48:39 +0300 Date: Wed, 21 Jul 2021 14:47:28 +0300 To: Mikhail Shishatskiy Message-ID: References: <20210721094428.1382809-1-m.shishatskiy@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210721094428.1382809-1-m.shishatskiy@tarantool.org> X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C3A9514C5AE4B3B389A94BDFA06D40730D182A05F538085040EC7E649D3E7928DD5F3A09AF1BCD25FD47AEDAB0D5CBC2CD096DE74FE7527A05 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE71BDE6A359BD5B800EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637DC205F3977E1285D8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8E1AC19B35FBBFC572D18350048E0B6DF117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAA867293B0326636D2E47CDBA5A96583BD4B6F7A4D31EC0BC014FD901B82EE079FA2833FD35BB23D27C277FBC8AE2E8BAA867293B0326636D2E47CDBA5A96583BA9C0B312567BB231DD303D21008E29813377AFFFEAFD269A417C69337E82CC2E827F84554CEF50127C277FBC8AE2E8BA83251EDC214901ED5E8D9A59859A8B67393CE827C55B5F775ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-C1DE0DAB: 0D63561A33F958A594803401010FD92EDEE25BD67662CAD4755F604A5C216D94D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7501A9DF589746230F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A63B03BCD35E0C0A5966D0160955805FD7E8C8B7A05C1F33DD8C38EA0E8B5ADC3948BD25B4611B361D7E09C32AA3244C1115E00A8DA77E7563F4A1EA51783FF597FE24653F78E668FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojJX8TSRcb/Sglud5Ld0kfOg== X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB4EC4DA1F3099D8F74CF89FEFE29D16693452E44D03157C12CF2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A89437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v1] memprof: group allocations on traces by trace number X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the patch! Please consider my comments below. On 21.07.21, Mikhail Shishatskiy wrote: > When luajit executes a trace, it's id is stored in virtual machine Typo: s/luajit/LuaJIT/ Nit: suggest to change: s/it's id/trace's number/ Feel free to ignore. Typo: s/in virtual machine state/in the virtual machine state/ > state. So, we can treat this trace number as allocation event source Typo: /allocation event source/an allocation event source/ > in memprof and report allocation events from traces as well. > > Previously all the allocations from traces were marked as INTERNAL. Typo: s/Previously all/Previously, all/ > > This patch introduces functionality described above by adding new > allocation source type named ASOURCE_TRACE. If at the moment when > allocation event occur vmstate indicates that trace is executed, > trace id is streamed to binary file: Typo: s/to binary file/to a binary file/ > > | loc-trace := trace-no > | trace-no := > > Also, memory profiler parser was tuned to recognize this source type by Nit: s/tuned/adjusted/ Feel free to ignore. > extending structure: field trace, representing trace id, was > added. > > Now, for example, all the allocation events happened in trace with id 42, Nit: s/id/number/ Feel free to ignore. > will be reported at source 'TRACE [42]' and not at 'INTERNAL' source. Nit: In my opinion "reported as source" fills better here. Feel free to ignore. I suppose that we can do a little bit more, than just report trace number. We can also provide information about its start pc (i.e. its start line). It is already implemented for perftools (see `perftools_addtrace()` function in ). With it, users can get a rough estimate of what trace causes allocation, without run two tools at the moment (`jit.dump` and `memprof`). Also, it can bring some clear about the trace if `jit.flush()` is called during memprof running. > > Closes tarantool/tarantool#5814 Nit: it is better to use "Resolves" either "Closes". Rationale: Issue isn't closed by this commit. It will be closed after the LuaJIT submodule is bumped in the Tarantool. But, indeed, the issue is resolved by this commit. > --- > > Issue: https://github.com/tarantool/tarantool/issues/5814 > Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > > src/lj_memprof.c | 16 ++++++++-------- > src/lj_memprof.h | 9 ++++++--- > tools/memprof/parse.lua | 18 +++++++++++------- > tools/utils/symtab.lua | 5 +++++ I suppose, that it will be nice to add some tests for the patch to check its behaviour. Also, performance measurements are welcome! :) You can use the benchmarks from here [1] as it was done for memprof. > 4 files changed, 30 insertions(+), 18 deletions(-) > > diff --git a/src/lj_memprof.c b/src/lj_memprof.c > index 2c1ef3b8..0cbc0ed7 100644 > --- a/src/lj_memprof.c > +++ b/src/lj_memprof.c > @@ -146,6 +146,13 @@ static void memprof_write_func(struct memprof *mp, uint8_t aevent) > lua_assert(0); > } > > +static void memprof_write_trace(struct memprof *mp, uint8_t aevent) > +{ > + struct lj_wbuf *out = &mp->out; > + lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE); > + lj_wbuf_addu64(out, (uint64_t)mp->g->vmstate); /* write traceno. */ Nit: I suppose, that the comment is redundant, if we add a comment to corresponding `memprof_writers[]` entry (see below). Also, we prefer to avoid usage of inline comments (structure or array definition are exceptions). Just use the previous line: see `dump_symtab()` for example. > +} > + > static void memprof_write_hvmstate(struct memprof *mp, uint8_t aevent) > { > lj_wbuf_addbyte(&mp->out, aevent | ASOURCE_INT); > @@ -163,14 +170,7 @@ static const memprof_writer memprof_writers[] = { > memprof_write_hvmstate, /* LJ_VMST_RECORD */ > memprof_write_hvmstate, /* LJ_VMST_OPT */ > memprof_write_hvmstate, /* LJ_VMST_ASM */ > - /* > - ** XXX: In ideal world, we should report allocations from traces as well. > - ** But since traces must follow the semantics of the original code, > - ** behaviour of Lua and JITted code must match 1:1 in terms of allocations, > - ** which makes using memprof with enabled JIT virtually redundant. > - ** Hence use the stub below. > - */ I suppose that this comment is still valid -- we *should* report allocations from traces, like it is done for bytecode execution via the VM. So I suggest to adjust this comment by replacing the last sentence with the description of what `memprof_write_trace()` does instead. > - memprof_write_hvmstate /* LJ_VMST_TRACE */ > + memprof_write_trace /* LJ_VMST_TRACE */ > }; > > static void memprof_write_caller(struct memprof *mp, uint8_t aevent) > diff --git a/src/lj_memprof.h b/src/lj_memprof.h > index 3417475d..13125536 100644 > --- a/src/lj_memprof.h > +++ b/src/lj_memprof.h > @@ -69,11 +69,13 @@ > ** event-realloc := event-header loc? oaddr osize naddr nsize > ** event-free := event-header loc? oaddr osize > ** event-header := > -** loc := loc-lua | loc-c > +** loc := loc-lua | loc-c | loc-trace > ** loc-lua := sym-addr line-no > ** loc-c := sym-addr > +** loc-trace := trace-no > ** sym-addr := > ** line-no := > +** trace-no := > ** oaddr := > ** naddr := > ** osize := > @@ -90,8 +92,8 @@ > ** > ** event-header: [FUUUSSEE] Please, update this header description too. > ** * EE : 2 bits for representing allocation event type (AEVENT_*) > diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua > index 12e2758f..2bd491c8 100644 > --- a/tools/memprof/parse.lua > +++ b/tools/memprof/parse.lua > @@ -59,20 +60,23 @@ local function link_to_previous(heap_chunk, e, nsize) > end > end > > -local function id_location(addr, line) > - return string_format("f%#xl%d", addr, line), { > +local function id_location(addr, line, trace) > + return string_format("f%#xl%dxt%d", addr, line, trace), { Typo: s/"f%#xl%dxt%d"/"f%#xl%dt%d"/ - x symbol looks redundant here. Nit: MB "traceno" fills better than trace. Feel free to ignore. > addr = addr, > line = line, > + trace = trace, > } > end > > local function parse_location(reader, asource) > if asource == ASOURCE_INT then > - return id_location(0, 0) > + return id_location(0, 0, 0) > elseif asource == ASOURCE_CFUNC then > - return id_location(reader:read_uleb128(), 0) > + return id_location(reader:read_uleb128(), 0, 0) > elseif asource == ASOURCE_LFUNC then > - return id_location(reader:read_uleb128(), reader:read_uleb128()) > + return id_location(reader:read_uleb128(), reader:read_uleb128(), 0) > + elseif asource == ASOURCE_TRACE then > + return id_location(0, 0, reader:read_uleb128()) Side note: The first two arguments can be used if we will dump information about the start of a trace. > end > error("Unknown asource "..asource) > end > @@ -140,7 +144,7 @@ local parsers = { > } > > local function ev_header_is_valid(evh) > - return evh <= 0x0f or evh == LJM_EPILOGUE_HEADER > + return evh <= 0x1f or evh == LJM_EPILOGUE_HEADER 0x1f is too much: the event header maximum value is the following: | $ perl -E 'say sprintf("0x%x", (4<<2)+3)' | 0x13 4<<2 is the biggest possible allocation event type (trace). And + 3 is the reallocation event type. I suggest to create the corresponding constant for the check. It will simplify maintenance in the future. > end > > -- Splits event header into event type (aka aevent = allocation > diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua > index 3ed1dd13..6121177f 100644 > --- a/tools/utils/symtab.lua > +++ b/tools/utils/symtab.lua > @@ -75,6 +75,11 @@ end > > function M.demangle(symtab, loc) > local addr = loc.addr > + local trace = loc.trace > + > + if trace ~= 0 then Ditto, the nit about s/trace/traceno/. Feel free to ignore. > + return string_format("TRACE [%d]", trace) > + end > > if addr == 0 then > return "INTERNAL" > -- > 2.32.0 > [1]: https://github.com/LuaJIT/LuaJIT-test-cleanup/tree/master/bench -- Best regards, Sergey Kaplun