From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 2E6286B962; Wed, 14 Apr 2021 14:33:20 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 2E6286B962 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1618400000; bh=QodXOsRuY1+FJ94lVS00GD76tLRDp5Rx23jV/FRv7SU=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=mh3Th6TEyJT9yE1SF2q/IOREo5JcMnUnie2KYKpVbgdnf/jy1eWVCcxS79DdYm32G 1eqe6SLKAqsDKgdPuHEb8nEt+RmWx1Ogzq1ei4ro/68SdT8bupeuIoq4pKfAT22tTr KWeLBDk8PrDanYg8T7hLf2AkNkFNiEGE8DMu9YdU= Received: from smtp16.mail.ru (smtp16.mail.ru [94.100.176.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CCA2F6BD23 for ; Wed, 14 Apr 2021 14:33:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CCA2F6BD23 Received: by smtp16.mail.ru with esmtpa (envelope-from ) id 1lWdll-0001tQ-US; Wed, 14 Apr 2021 14:33:18 +0300 Date: Wed, 14 Apr 2021 14:32:20 +0300 To: Sergey Ostanevich Message-ID: References: <20210331172948.10660-1-skaplun@tarantool.org> <3819D871-7EEE-4A44-AE97-8847B4928270@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3819D871-7EEE-4A44-AE97-8847B4928270@tarantool.org> X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92FFCB8E6708E74809299FB6B3996B874F289A033C44AD400182A05F5380850403B8358A26A9DB103F61C77DBF9C929642B916873ED08622E7A4C57040BC7923F X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7CE015DA2C687B387EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006376ECF4463D45BB2698638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2D60BE047C87F9784569B11EB1C393AE3B10649542A97EE81D2E47CDBA5A96583C09775C1D3CA48CFCA5A41EBD8A3A0199FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3CA5A41EBD8A3A0199FA2833FD35BB23DF004C906525384302BEBFE083D3B9BA73A03B725D353964B0B7D0EA88DDEDAC722CA9DD8327EE4930A3850AC1BE2E7358CCB3ED2A1DE2304C4224003CC83647689D4C264860C145E X-C1DE0DAB: 0D63561A33F958A55156A3126CF4B9BA411A34AB0FA6821F6E0D8BD7A3A18D5BD59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34D7D5115130EFC824292BAECDDCD6843AC1E9E63F174FDF48E2BE501A356DD3F31EAB1B671720E9F61D7E09C32AA3244C1E3064D15BB9F2739A846B85544D7DADFE8DA44ABE2443F7FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojnA7/qPBUIXFlDhJyhdN+8g== X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB457E86FC713FEB9B15B20AD3A3A62B602F426582E0103707EF2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A89437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v2 luajit] tools: introduce --leak-only memprof parser option X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for the review! On 08.04.21, Sergey Ostanevich wrote: > Hi! > > Just couple of nits, LGTM. > > Sergos > > > On 31 Mar 2021, at 20:29, Sergey Kaplun wrote: > > > > This patch indtroduces new memprof parser module to > introduces Fixed. > > post-process memory events. > > > > Memprof parser now adds postamble with the source lines of Lua chunks > > (or "INTERNAL") that allocate and do not free some amount of bytes, when > > profiler finishes. The parser also reports the number of allocation and > > deallocation events related to each line. > > > > Also, this patch adds a new --leak-only memory profiler parser option. > > When the parser runs with that option, it reports only leak > > information. > > > > Resolves tarantool/tarantool#5812 > > --- > > Changes in v2: > > * introduce new memprof's module to post-process parsed > > events > > * add tests > > > > ChangeLog entry (and postamble too Tarantool bump commit message): > ^^^^^^^ I failed to parse, typo? Yep :). > > > > =================================================================== > > ##feature/luajit > > > > * Now memory profiler parser reports heap difference occurring during > > the measurement interval. New memory profiler's option `--leak-only` > > to show only heap difference is introduced. New built-in module > shows Fixed on branch. > > `memprof.process` is introduced to perform memory events > > post-processing and aggregation. Now to launch memory profiler > > via Tarantool user should use the following command: > > `tarantool -e 'require("memprof")(arg)' - --leak-only /tmp/memprof.bin` > > > =================================================================== > > > > Branch with tests and added the corresponding built-in: > > * https://github.com/tarantool/tarantool/tree/skaplun/gh-5812-memprof-memleaks-option > > LuaJIT branch: > > * https://github.com/tarantool/luajit/tree/skaplun/gh-5812-memprof-memleaks-option > > Issue: https://github.com/tarantool/tarantool/issues/5812 > > > > .../misclib-memprof-lapi.test.lua | 21 +++++-- > > tools/memprof.lua | 33 ++++++----- > > tools/memprof/humanize.lua | 43 +++++++++++++- > > tools/memprof/parse.lua | 20 +++++-- > > tools/memprof/process.lua | 59 +++++++++++++++++++ > > 5 files changed, 151 insertions(+), 25 deletions(-) > > create mode 100644 tools/memprof/process.lua > > > > diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua > > index cb63e1b8..9affc2fe 100644 > > --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua > > +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua > > @@ -1,7 +1,7 @@ > > local tap = require("tap") > > > > local test = tap.test("misc-memprof-lapi") > > -test:plan(9) > > +test:plan(13) > > > > jit.off() > > jit.flush() > > @@ -10,6 +10,7 @@ local table_new = require "table.new" > > > > local bufread = require "utils.bufread" > > local memprof = require "memprof.parse" > > +local process = require "memprof.process" > > local symtab = require "utils.symtab" > > > > local TMP_BINFILE = arg[0]:gsub(".+/([^/]+)%.test%.lua$", "%.%1.memprofdata.tmp.bin") > > @@ -66,8 +67,12 @@ local function fill_ev_type(events, symbols, event_type) > > return ev_type > > end > > > > +local function form_source_line(line) > > + return string.format("@%s:%d", arg[0], line) > > +end > > + > > local function check_alloc_report(alloc, line, function_line, nevents) > > - assert(string.format("@%s:%d", arg[0], function_line) == alloc[line].name) > > + assert(form_source_line(function_line) == alloc[line].name) > > assert(alloc[line].num == nevents, ("got=%d, expected=%d"):format( > > alloc[line].num, > > nevents > > @@ -120,13 +125,21 @@ local free = fill_ev_type(events, symbols, "free") > > -- the number of allocations. > > -- 1 event - alocation of table by itself + 1 allocation > allocation Nice catch! Unfortunately, it is not relating to this patch, so I left it intact. > > -- of array part as far it is bigger than LJ_MAX_COLOSIZE (16). > > -test:ok(check_alloc_report(alloc, 20, 18, 2)) > > +test:ok(check_alloc_report(alloc, 21, 19, 2)) > > -- 100 strings allocations. > > -test:ok(check_alloc_report(alloc, 25, 18, 100)) > > +test:ok(check_alloc_report(alloc, 26, 19, 100)) > > > > -- Collect all previous allocated objects. > > test:ok(free.INTERNAL.num == 102) > > > > +local heap_diff = process.form_heap_diff(events, symbols) > > +local tab_alloc_source = heap_diff[form_source_line(21)] > > +local str_alloc_source = heap_diff[form_source_line(26)] > > +test:ok(tab_alloc_source.cnt_alloc == tab_alloc_source.cnt_free) > > +test:ok(tab_alloc_source.size_diff == 0) > > +test:ok(str_alloc_source.cnt_alloc == str_alloc_source.cnt_free) > > +test:ok(str_alloc_source.size_diff == 0) > > + > > -- Test for https://github.com/tarantool/tarantool/issues/5842. > > -- We are not interested in this report. > > misc.memprof.start("/dev/null") > > diff --git a/tools/memprof.lua b/tools/memprof.lua > > index 9f962085..c6c5f587 100644 > > --- a/tools/memprof.lua > > +++ b/tools/memprof.lua > > @@ -12,6 +12,7 @@ > > > > local bufread = require "utils.bufread" > > local memprof = require "memprof.parse" > > +local process = require "memprof.process" > > local symtab = require "utils.symtab" > > local view = require "memprof.humanize" > > > > @@ -33,10 +34,16 @@ luajit-parse-memprof [options] memprof.bin > > Supported options are: > > > > --help Show this help and exit > > + --leak-only Report only leaks information > > ]] > > os.exit(0) > > end > > > > +local leak_only = false > > +opt_map["leak-only"] = function() > > + leak_only = true > > +end > > + > > -- Print error and exit with error status. > > local function opterror(...) > > stderr:write("luajit-parse-memprof.lua: ERROR: ", ...) > > @@ -94,26 +101,22 @@ local function dump(inputfile) > > local reader = bufread.new(inputfile) > > local symbols = symtab.parse(reader) > > local events = memprof.parse(reader, symbols) > > - > > - stdout:write("ALLOCATIONS", "\n") > > - view.render(events.alloc, symbols) > > - stdout:write("\n") > > - > > - stdout:write("REALLOCATIONS", "\n") > > - view.render(events.realloc, symbols) > > - stdout:write("\n") > > - > > - stdout:write("DEALLOCATIONS", "\n") > > - view.render(events.free, symbols) > > - stdout:write("\n") > > - > > + if not leak_only then > > + view.profile_info(events, symbols) > > + end > > + local heap_diff = process.form_heap_diff(events, symbols) > > + view.leak_only(heap_diff) > > The name of the function is confusing: you dump whole data if _not_ only > leaks, and then without alternative the leaks data. It sounds as ‘always’ > but I propose to name it just ‘leaks’. Then the logic will be ‘dump all’ > or ‘leaks’ only. Changed to `leak_info()`, according to Igor's review; see the iterative patch below: =================================================================== diff --git a/tools/memprof.lua b/tools/memprof.lua index c6c5f587..39505020 100644 --- a/tools/memprof.lua +++ b/tools/memprof.lua @@ -105,7 +105,7 @@ local function dump(inputfile) view.profile_info(events, symbols) end local heap_diff = process.form_heap_diff(events, symbols) - view.leak_only(heap_diff) + view.leak_info(heap_diff) os.exit(0) end diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua index 6afd3ff1..3812c6c4 100644 --- a/tools/memprof/humanize.lua +++ b/tools/memprof/humanize.lua @@ -56,7 +56,7 @@ function M.profile_info(events, symbols) print("") end -function M.leak_only(heap_diff) +function M.leak_info(heap_diff) local rest_heap = {} for line, info in pairs(heap_diff) do -- Report "INTERNAL" events inconsistencies for profiling =================================================================== > > > diff --git a/tools/memprof/process.lua b/tools/memprof/process.lua > > new file mode 100644 > > index 00000000..94be187e > > --- /dev/null > > +++ b/tools/memprof/process.lua > > @@ -0,0 +1,59 @@ > > + local function process_non_alloc_events(events_by_type) > > + for _, event in pairs(events_by_type) do > > + -- Realloc and free events always have "primary” key > a > > + -- that references table with rewrited memory > the rewritten, although I’d use a different verb > and put at the end: ‘memory changed’? Fixed. See the iterative patch below. =================================================================== diff --git a/tools/memprof/process.lua b/tools/memprof/process.lua index 64acfd15..34747dec 100644 --- a/tools/memprof/process.lua +++ b/tools/memprof/process.lua @@ -33,8 +33,8 @@ function M.form_heap_delta(events, symbols) -- no collisions between different object addresses. local function process_non_alloc_events(events_by_type) for _, event in pairs(events_by_type) do - -- Realloc and free events always have "primary" key - -- that references table with rewrited memory + -- Realloc and free events always have key named "primary" + -- that references the table with memory changed -- (may be empty). for _, heap_chunk in pairs(event.primary) do local ev_line = symtab.demangle(symbols, heap_chunk.loc) =================================================================== > > + -- (may be empty). > > + for _, heap_chunk in pairs(event.primary) do > > + local ev_line = symtab.demangle(symbols, heap_chunk.loc) > > + > > + if (heap_chunk.alloced > 0) then > > + heap[ev_line].size_diff = heap[ev_line].size_diff + heap_chunk.alloced > > + heap[ev_line].cnt_alloc = heap[ev_line].cnt_alloc + heap_chunk.cnt > > + end > > + > > + if (heap_chunk.freed > 0) then > > + heap[ev_line].size_diff = heap[ev_line].size_diff - heap_chunk.freed > > + heap[ev_line].cnt_free = heap[ev_line].cnt_free + heap_chunk.cnt > > + end > > + end > > + end > > + end > > + process_non_alloc_events(events.realloc) > > + process_non_alloc_events(events.free) > > + return heap > > +end > > + > > +return M > > -- > > 2.31.0 > > > -- Best regards, Sergey Kaplun