From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 19AEC6FC86; Thu, 16 Sep 2021 18:58:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 19AEC6FC86 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1631807897; bh=pyyr5uiwfwSImfguQ6LinylwZtPap44+/mwazhU0Xsg=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=Z9IQLR55N48fRITsq42Wkz/F9Q4Y9aoDNy2Wu7Yx05lDbKbNbioNRW4xAdWoHvHQd ea5nkMluLmkxKjohQmv9IsaaZGFfVpptmA6UcdhUN7OAXQdwZJdaDN8ybHnk9lTSJh Zs03X91cy0o9cVC+xsqcltXXDf/xUcc7Rrj5T+tY= Received: from smtpng1.i.mail.ru (smtpng1.i.mail.ru [94.100.181.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 2CE3A6FC86 for ; Thu, 16 Sep 2021 18:58:16 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2CE3A6FC86 Received: by smtpng1.m.smailru.net with esmtpa (envelope-from ) id 1mQtmA-0004Ek-OV; Thu, 16 Sep 2021 18:58:15 +0300 Date: Thu, 16 Sep 2021 18:32:01 +0300 To: Mikhail Shishatskiy Message-ID: <20210916153201.GC6844@tarantool.org> References: <20210820070546.115293-1-m.shishatskiy@tarantool.org> <20210820070546.115293-4-m.shishatskiy@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20210820070546.115293-4-m.shishatskiy@tarantool.org> X-Clacks-Overhead: GNU Terry Pratchett User-Agent: Mutt/1.10.1 (2018-07-13) X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD91AE02D33A9C88A2F7B0FCECE251A4B9EABD2FC9AA46919D800894C459B0CD1B986A29BC666CF10F58B58812F24D0B1262865F1F1D023BD9570620434C0A4BCCF X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE74B51810D54EC17F0C2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE788EDE44B2AC36D7BEA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F25F1A14EABF4B6F2B5B72C621F88DBF17CC7F00164DA146DAFE8445B8C89999728AA50765F7900637F6B57BC7E64490618DEB871D839B7333395957E7521B51C2DFABB839C843B9C08941B15DA834481F8AA50765F7900637F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA7E827F84554CEF5019E625A9149C048EE9ECD01F8117BC8BEE2021AF6380DFAD18AA50765F790063735872C767BF85DA227C277FBC8AE2E8BC6A536F79815AD9275ECD9A6C639B01B4E70A05D1297E1BBCB5012B2E24CD356 X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458834459D11680B5055428A3E17DA39929BF0EF70651C41038 X-C1DE0DAB: 0D63561A33F958A5D10265168950EEE134ED7FCD55354079C90D00F091B4840ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7540E9CF2C1C1CEBBA410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34431D0341F6B74DD3CE5D79D383749F272141940ECE81E7115501A31FED3782B2EBD8DC126109124B1D7E09C32AA3244CA716F63BCC5B893B2FBD811AE6A747EDF2F5F14F68F1805B927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbgLrBZmQxDQEmR+rIqfn0A== X-Mailru-Sender: 689FA8AB762F7393C37E3C1AEC41BA5D61C804B0ACCD5C834109CC5181E6E2AFA7C8D0F45F857DBFE9F1EFEE2F478337FB559BB5D741EB964C8C2C849690F8E70A04DAD6CC59E33667EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v3 3/5] memprof: dump traceno if allocate from trace X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Igor Munkin via Tarantool-patches Reply-To: Igor Munkin Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Misha, Thanks for the patch! Please consider my comments below. On 20.08.21, Mikhail Shishatskiy wrote: > When LuaJIT executes a trace, the trace number is stored in > the virtual machine state. So, we can treat this number as > an allocation event source in memprof and report allocation events > from traces as well. > > Previously, all the allocations from traces were marked as INTERNAL. > > This patch introduces the functionality described above by adding > a new allocation source type named ASOURCE_TRACE. If at the moment > when allocation event occurs VM state indicates that trace executed, > trace number streamed to a binary file: > > | loc-trace := trace-addr trace-no > | trace-addr := > | trace-no := > > Also, the memory profiler parser is adjusted to recognize this > source type by extending structure: field , > representing trace number, is added. I understand, why you've chosen this order, but I don't like it. IMHO, the binary format should not rely or depend on the particular parser implementation a lot. Please, consider more comments below. > > Part of tarantool/tarantool#5814 > --- > > Issue: https://github.com/tarantool/tarantool/issues/5814 > Luajit branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > tarantool branch: https://github.com/tarantool/tarantool/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number > > src/Makefile.dep.original | 2 +- > src/lj_memprof.c | 35 +++++++++++++++++++++++++++++++++-- > src/lj_memprof.h | 15 ++++++++++----- > tools/memprof/parse.lua | 22 ++++++++++++++-------- > 4 files changed, 58 insertions(+), 16 deletions(-) > > diff --git a/src/Makefile.dep.original b/src/Makefile.dep.original > index f3672413..ee6bafb2 100644 > --- a/src/Makefile.dep.original > +++ b/src/Makefile.dep.original > @@ -146,7 +146,7 @@ lj_mcode.o: lj_mcode.c lj_obj.h lua.h luaconf.h lj_def.h lj_arch.h \ > lj_gc.h lj_err.h lj_errmsg.h lj_jit.h lj_ir.h lj_mcode.h lj_trace.h \ > lj_dispatch.h lj_bc.h lj_traceerr.h lj_vm.h > lj_memprof.o: lj_memprof.c lj_arch.h lua.h luaconf.h lj_memprof.h \ > - lj_def.h lj_wbuf.h lj_obj.h lj_frame.h lj_bc.h lj_debug.h > + lj_def.h lj_wbuf.h lj_obj.h lj_frame.h lj_bc.h lj_debug.h lj_dispatch.h It looks some headers are missing (it's better use from Makefile.original to check yourself). > lj_meta.o: lj_meta.c lj_obj.h lua.h luaconf.h lj_def.h lj_arch.h lj_gc.h \ > lj_err.h lj_errmsg.h lj_buf.h lj_str.h lj_tab.h lj_meta.h lj_frame.h \ > lj_bc.h lj_vm.h lj_strscan.h lj_strfmt.h lj_lib.h > diff --git a/src/lj_memprof.c b/src/lj_memprof.c > index 2c1ef3b8..fb99829d 100644 > --- a/src/lj_memprof.c > +++ b/src/lj_memprof.c > @@ -168,9 +197,11 @@ static const memprof_writer memprof_writers[] = { > ** But since traces must follow the semantics of the original code, > ** behaviour of Lua and JITted code must match 1:1 in terms of allocations, > ** which makes using memprof with enabled JIT virtually redundant. > - ** Hence use the stub below. > + ** But if one wants to investigate allocations with JIT enabled, > + ** memprof_write_trace() dumps trace number to the binary output. Typo: number and mcode starting address, right? > + ** It can be useful to compare with with jit.v or jit.dump outputs. > */ > - memprof_write_hvmstate /* LJ_VMST_TRACE */ > + memprof_write_trace /* LJ_VMST_TRACE */ > }; > > static void memprof_write_caller(struct memprof *mp, uint8_t aevent) > diff --git a/src/lj_memprof.h b/src/lj_memprof.h > index 3417475d..6a35385d 100644 > --- a/src/lj_memprof.h > +++ b/src/lj_memprof.h > @@ -51,9 +51,10 @@ > */ > > #define SYMTAB_LFUNC ((uint8_t)0) > +#define SYMTAB_TRACE ((uint8_t)1) This looks like related to the next patch, doesn't it? > #define SYMTAB_FINAL ((uint8_t)0x80) > > -#define LJM_CURRENT_FORMAT_VERSION 0x01 > +#define LJM_CURRENT_FORMAT_VERSION 0x02 > > /* > ** Event stream format: > diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua > index 12e2758f..adc7c072 100644 > --- a/tools/memprof/parse.lua > +++ b/tools/memprof/parse.lua > @@ -24,8 +24,11 @@ local AEVENT_MASK = 0x3 > local ASOURCE_INT = lshift(1, 2) > local ASOURCE_LFUNC = lshift(2, 2) > local ASOURCE_CFUNC = lshift(3, 2) > +local ASOURCE_TRACE = lshift(4, 2) > > -local ASOURCE_MASK = lshift(0x3, 2) > +local ASOURCE_MASK = lshift(0x7, 2) > + > +local EV_HEADER_MAX = ASOURCE_TRACE + AEVENT_REALLOC Why so complex? I believe lshift(5, 2) is more clear and covers (i.e. is greater than) all cases of AEVENT_* and ASOURCE_*. > > local M = {} > > @@ -59,20 +62,23 @@ local function link_to_previous(heap_chunk, e, nsize) > end > end > > -local function id_location(addr, line) > - return string_format("f%#xl%d", addr, line), { > +local function id_location(addr, line, traceno) > + return string_format("f%#xl%dt%d", addr, line, traceno), { > addr = addr, > line = line, > + traceno = traceno, > } > end > > local function parse_location(reader, asource) > if asource == ASOURCE_INT then > - return id_location(0, 0) > + return id_location(0, 0, 0) > elseif asource == ASOURCE_CFUNC then > - return id_location(reader:read_uleb128(), 0) > + return id_location(reader:read_uleb128(), 0, 0) > elseif asource == ASOURCE_LFUNC then > - return id_location(reader:read_uleb128(), reader:read_uleb128()) > + return id_location(reader:read_uleb128(), reader:read_uleb128(), 0) > + elseif asource == ASOURCE_TRACE then > + return id_location(reader:read_uleb128(), 0, reader:read_uleb128()) As a result of your changes this function becomes too "cryptic". It's better to refactor this function (maybe even in a separate commit), so we have something like the function below at the final. | local function id(params) | return string_format("f%#xl%ds%d", params.addr, params.line, params.state) | end | | local function parse_location(reader, asource) | local location = { addr = 0, line = 0, traceno = 0 } | if asource == ASOURCE_INT then | -- Do nothing | elseif asource == ASOURCE_CFUNC then | location.addr = reader:read_uleb128() | elseif asource == ASOURCE_LFUNC then | location.addr = reader:read_uleb128() | location.line = reader:read_uleb128() | elseif asource == ASOURCE_TRACE then | location.trace = reader:read_uleb128() | location.addr = reader:read_uleb128() | else | error("Unknown asource "..asource) | end | return id(location), location | end You can also make this function public and move it to utils.lua module. BTW, these entries are "loaded" but not "rendered" in the final output now, aren't they? In other words, why don't you make everything in a single patch? > end > error("Unknown asource "..asource) > end > -- > 2.32.0 > -- Best regards, IM