From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 0307B6EC5D; Thu, 8 Apr 2021 15:49:56 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 0307B6EC5D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1617886196; bh=IPX0H6Bok1OFu7Kbrb5Pw3ZPTae1hcI5DT/9jfIqC9Q=; h=In-Reply-To:Date:References:To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=XgBZDdXXJmDVTC8eo0ZNmi5F1kOEi1Tpi7NwLov0XmNRkmZTJs+C6VZXFpDOspG+r Xq5z9vnL/fN+UXV1o0O6fcwsrEIn7OvpvGkt/9XMi6LSQFncFk5S6vcr79bG1VkDKo cGoyUrc1R/rzzE0L5UugVAwWCsd/D7MShUwJ/Leg= Received: from smtp30.i.mail.ru (smtp30.i.mail.ru [94.100.177.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 9C8D06EC5D for ; Thu, 8 Apr 2021 15:49:54 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 9C8D06EC5D Received: by smtp30.i.mail.ru with esmtpa (envelope-from ) id 1lUU6b-0000em-NN; Thu, 08 Apr 2021 15:49:54 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) In-Reply-To: <20210331172948.10660-1-skaplun@tarantool.org> Date: Thu, 8 Apr 2021 15:49:47 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <3819D871-7EEE-4A44-AE97-8847B4928270@tarantool.org> References: <20210331172948.10660-1-skaplun@tarantool.org> To: Sergey Kaplun X-Mailer: Apple Mail (2.3654.60.0.2.21) X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD912A3E3D5D4B49FC1FED631623C873017929456591B90F20100894C459B0CD1B97922B7BE1B3B785F6258F66F8B1006CE1A352C6F1095D814985936BB30EC2273 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7978947DCA0D4215FEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063796E7E609BFB2B1698638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B2C9562CA1160369A77B10E3E2125F0EB9CCD65611471A2689D2E47CDBA5A96583C09775C1D3CA48CFCA5A41EBD8A3A0199FA2833FD35BB23D2EF20D2F80756B5F868A13BD56FB6657A471835C12D1D977725E5C173C3A84C3CA5A41EBD8A3A0199FA2833FD35BB23DF004C906525384302BEBFE083D3B9BA71A620F70A64A45A98AA50765F79006372E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89F83C798A30B85E16B42539A7722CA490CB5C8C57E37DE458BEDA766A37F9254B7 X-C1DE0DAB: 0D63561A33F958A5CFA27A9FABBF0F9F7E1DAD35453776A558279ADA5651CE7ED59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3407FE5477D6A8AF086D2CAFA2882B92CF4B65E3BFBD9D1B9FB675352D2CE7356331990A9D868617F51D7E09C32AA3244CBCEAE6F0F4B469D9793191A62ADE120169B6CAE0477E908DFACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2bioju+jaMfvANXrFIs3PxG1u7g== X-Mailru-Sender: 3B9A0136629DC912F4AABCEFC589C81E3F3DE3243B3C3C0E80BC698C59EEF25E536FCCD46ED3D72EAD07DD1419AC565FA614486B47F28B67C5E079CCF3B0523AED31B7EB2E253A9E112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH v2 luajit] tools: introduce --leak-only memprof parser option X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Ostanevich via Tarantool-patches Reply-To: Sergey Ostanevich Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Just couple of nits, LGTM. Sergos > On 31 Mar 2021, at 20:29, Sergey Kaplun wrote: >=20 > This patch indtroduces new memprof parser module to introduces > post-process memory events. >=20 > Memprof parser now adds postamble with the source lines of Lua chunks > (or "INTERNAL") that allocate and do not free some amount of bytes, = when > profiler finishes. The parser also reports the number of allocation = and > deallocation events related to each line. >=20 > Also, this patch adds a new --leak-only memory profiler parser option. > When the parser runs with that option, it reports only leak > information. >=20 > Resolves tarantool/tarantool#5812 > --- > Changes in v2: > * introduce new memprof's module to post-process parsed > events > * add tests >=20 > ChangeLog entry (and postamble too Tarantool bump commit message): ^^^^^^^ I failed to parse, typo? >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > ##feature/luajit >=20 > * Now memory profiler parser reports heap difference occurring during > the measurement interval. New memory profiler's option `--leak-only` > to show only heap difference is introduced. New built-in module shows > `memprof.process` is introduced to perform memory events > post-processing and aggregation. Now to launch memory profiler > via Tarantool user should use the following command: > `tarantool -e 'require("memprof")(arg)' - --leak-only = /tmp/memprof.bin` > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > Branch with tests and added the corresponding built-in: > * = https://github.com/tarantool/tarantool/tree/skaplun/gh-5812-memprof-memlea= ks-option > LuaJIT branch: > * = https://github.com/tarantool/luajit/tree/skaplun/gh-5812-memprof-memleaks-= option > Issue: https://github.com/tarantool/tarantool/issues/5812 >=20 > .../misclib-memprof-lapi.test.lua | 21 +++++-- > tools/memprof.lua | 33 ++++++----- > tools/memprof/humanize.lua | 43 +++++++++++++- > tools/memprof/parse.lua | 20 +++++-- > tools/memprof/process.lua | 59 +++++++++++++++++++ > 5 files changed, 151 insertions(+), 25 deletions(-) > create mode 100644 tools/memprof/process.lua >=20 > diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua = b/test/tarantool-tests/misclib-memprof-lapi.test.lua > index cb63e1b8..9affc2fe 100644 > --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua > +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua > @@ -1,7 +1,7 @@ > local tap =3D require("tap") >=20 > local test =3D tap.test("misc-memprof-lapi") > -test:plan(9) > +test:plan(13) >=20 > jit.off() > jit.flush() > @@ -10,6 +10,7 @@ local table_new =3D require "table.new" >=20 > local bufread =3D require "utils.bufread" > local memprof =3D require "memprof.parse" > +local process =3D require "memprof.process" > local symtab =3D require "utils.symtab" >=20 > local TMP_BINFILE =3D arg[0]:gsub(".+/([^/]+)%.test%.lua$", = "%.%1.memprofdata.tmp.bin") > @@ -66,8 +67,12 @@ local function fill_ev_type(events, symbols, = event_type) > return ev_type > end >=20 > +local function form_source_line(line) > + return string.format("@%s:%d", arg[0], line) > +end > + > local function check_alloc_report(alloc, line, function_line, nevents) > - assert(string.format("@%s:%d", arg[0], function_line) =3D=3D = alloc[line].name) > + assert(form_source_line(function_line) =3D=3D alloc[line].name) > assert(alloc[line].num =3D=3D nevents, ("got=3D%d, = expected=3D%d"):format( > alloc[line].num, > nevents > @@ -120,13 +125,21 @@ local free =3D fill_ev_type(events, symbols, = "free") > -- the number of allocations. > -- 1 event - alocation of table by itself + 1 allocation allocation > -- of array part as far it is bigger than LJ_MAX_COLOSIZE (16). > -test:ok(check_alloc_report(alloc, 20, 18, 2)) > +test:ok(check_alloc_report(alloc, 21, 19, 2)) > -- 100 strings allocations. > -test:ok(check_alloc_report(alloc, 25, 18, 100)) > +test:ok(check_alloc_report(alloc, 26, 19, 100)) >=20 > -- Collect all previous allocated objects. > test:ok(free.INTERNAL.num =3D=3D 102) >=20 > +local heap_diff =3D process.form_heap_diff(events, symbols) > +local tab_alloc_source =3D heap_diff[form_source_line(21)] > +local str_alloc_source =3D heap_diff[form_source_line(26)] > +test:ok(tab_alloc_source.cnt_alloc =3D=3D tab_alloc_source.cnt_free) > +test:ok(tab_alloc_source.size_diff =3D=3D 0) > +test:ok(str_alloc_source.cnt_alloc =3D=3D str_alloc_source.cnt_free) > +test:ok(str_alloc_source.size_diff =3D=3D 0) > + > -- Test for https://github.com/tarantool/tarantool/issues/5842. > -- We are not interested in this report. > misc.memprof.start("/dev/null") > diff --git a/tools/memprof.lua b/tools/memprof.lua > index 9f962085..c6c5f587 100644 > --- a/tools/memprof.lua > +++ b/tools/memprof.lua > @@ -12,6 +12,7 @@ >=20 > local bufread =3D require "utils.bufread" > local memprof =3D require "memprof.parse" > +local process =3D require "memprof.process" > local symtab =3D require "utils.symtab" > local view =3D require "memprof.humanize" >=20 > @@ -33,10 +34,16 @@ luajit-parse-memprof [options] memprof.bin > Supported options are: >=20 > --help Show this help and exit > + --leak-only Report only leaks information > ]] > os.exit(0) > end >=20 > +local leak_only =3D false > +opt_map["leak-only"] =3D function() > + leak_only =3D true > +end > + > -- Print error and exit with error status. > local function opterror(...) > stderr:write("luajit-parse-memprof.lua: ERROR: ", ...) > @@ -94,26 +101,22 @@ local function dump(inputfile) > local reader =3D bufread.new(inputfile) > local symbols =3D symtab.parse(reader) > local events =3D memprof.parse(reader, symbols) > - > - stdout:write("ALLOCATIONS", "\n") > - view.render(events.alloc, symbols) > - stdout:write("\n") > - > - stdout:write("REALLOCATIONS", "\n") > - view.render(events.realloc, symbols) > - stdout:write("\n") > - > - stdout:write("DEALLOCATIONS", "\n") > - view.render(events.free, symbols) > - stdout:write("\n") > - > + if not leak_only then > + view.profile_info(events, symbols) > + end > + local heap_diff =3D process.form_heap_diff(events, symbols) > + view.leak_only(heap_diff) The name of the function is confusing: you dump whole data if _not_ only leaks, and then without alternative the leaks data. It sounds as = =E2=80=98always=E2=80=99 but I propose to name it just =E2=80=98leaks=E2=80=99. Then the logic = will be =E2=80=98dump all=E2=80=99 or =E2=80=98leaks=E2=80=99 only. > os.exit(0) > end >=20 > +local function dump_wrapped(...) > + return dump(parseargs(...)) > +end > + > -- FIXME: this script should be application-independent. > local args =3D {...} > if #args =3D=3D 1 and args[1] =3D=3D "memprof" then > - return dump > + return dump_wrapped > else > - dump(parseargs(args)) > + dump_wrapped(args) > end > diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua > index 2d5814c6..6afd3ff1 100644 > --- a/tools/memprof/humanize.lua > +++ b/tools/memprof/humanize.lua > @@ -28,8 +28,8 @@ function M.render(events, symbols) > )) >=20 > local prim_loc =3D {} > - for _, loc in pairs(event.primary) do > - table.insert(prim_loc, symtab.demangle(symbols, loc)) > + for _, heap_chunk in pairs(event.primary) do > + table.insert(prim_loc, symtab.demangle(symbols, = heap_chunk.loc)) > end > if #prim_loc ~=3D 0 then > table.sort(prim_loc) > @@ -42,4 +42,43 @@ function M.render(events, symbols) > end > end >=20 > +function M.profile_info(events, symbols) > + print("ALLOCATIONS") > + M.render(events.alloc, symbols) > + print("") > + > + print("REALLOCATIONS") > + M.render(events.realloc, symbols) > + print("") > + > + print("DEALLOCATIONS") > + M.render(events.free, symbols) > + print("") > +end > + > +function M.leak_only(heap_diff) > + local rest_heap =3D {} > + for line, info in pairs(heap_diff) do > + -- Report "INTERNAL" events inconsistencies for profiling > + -- with enabled jit. > + if info.size_diff > 0 then > + table.insert(rest_heap, {line =3D line, hold_bytes =3D = info.size_diff}) > + end > + end > + > + table.sort(rest_heap, function(h1, h2) > + return h1.hold_bytes > h2.hold_bytes > + end) > + > + print("HEAP SUMMARY:") > + for _, h in pairs(rest_heap) do > + print(string.format( > + "%s holds %d bytes: %d allocs, %d frees", > + h.line, h.hold_bytes, heap_diff[h.line].cnt_alloc, > + heap_diff[h.line].cnt_free > + )) > + end > + print("") > +end > + > return M > diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua > index 6dae22d5..df10a45f 100644 > --- a/tools/memprof/parse.lua > +++ b/tools/memprof/parse.lua > @@ -39,11 +39,23 @@ local function new_event(loc) > } > end >=20 > -local function link_to_previous(heap_chunk, e) > +local function link_to_previous(heap_chunk, e, nsize) > -- Memory at this chunk was allocated before we start tracking. > if heap_chunk then > -- Save Lua code location (line) by address (id). > - e.primary[heap_chunk[2]] =3D heap_chunk[3] > + if not e.primary[heap_chunk[2]] then > + e.primary[heap_chunk[2]] =3D { > + loc =3D heap_chunk[3], > + alloced =3D 0, > + freed =3D 0, > + cnt =3D 0, > + } > + end > + -- Save memory diff heap information. > + local location_data =3D e.primary[heap_chunk[2]] > + location_data.alloced =3D location_data.alloced + nsize > + location_data.freed =3D location_data.freed + heap_chunk[1] > + location_data.cnt =3D location_data.cnt + 1 > end > end >=20 > @@ -97,7 +109,7 @@ local function parse_realloc(reader, asource, = events, heap) > e.free =3D e.free + osize > e.alloc =3D e.alloc + nsize >=20 > - link_to_previous(heap[oaddr], e) > + link_to_previous(heap[oaddr], e, nsize) >=20 > heap[oaddr] =3D nil > heap[naddr] =3D {nsize, id, loc} > @@ -116,7 +128,7 @@ local function parse_free(reader, asource, events, = heap) > e.num =3D e.num + 1 > e.free =3D e.free + osize >=20 > - link_to_previous(heap[oaddr], e) > + link_to_previous(heap[oaddr], e, 0) >=20 > heap[oaddr] =3D nil > end > diff --git a/tools/memprof/process.lua b/tools/memprof/process.lua > new file mode 100644 > index 00000000..94be187e > --- /dev/null > +++ b/tools/memprof/process.lua > @@ -0,0 +1,59 @@ > +-- LuaJIT's memory profile post-processing module. > + > +local M =3D {} > + > +local symtab =3D require "utils.symtab" > + > +function M.form_heap_diff(events, symbols) > + -- Auto resurrects source event lines for counting/reporting. > + local heap =3D setmetatable({}, {__index =3D function(t, line) > + t[line] =3D { > + size_diff =3D 0, > + cnt_alloc =3D 0, > + cnt_free =3D 0, > + } > + return t[line] > + end}) > + > + for _, event in pairs(events.alloc) do > + if event.loc then > + local ev_line =3D symtab.demangle(symbols, event.loc) > + > + if (event.alloc > 0) then > + heap[ev_line].size_diff =3D heap[ev_line].size_diff + = event.alloc > + heap[ev_line].cnt_alloc =3D heap[ev_line].cnt_alloc + = event.num > + end > + end > + end > + > + -- Realloc and free events are pretty the same. > + -- We aren't interested in aggregated alloc/free sizes for > + -- the event, but only for new and old size values inside > + -- alloc-realloc-free chain. Assuming that we have > + -- no collisions between different object addresses. > + local function process_non_alloc_events(events_by_type) > + for _, event in pairs(events_by_type) do > + -- Realloc and free events always have "primary=E2=80=9D key a > + -- that references table with rewrited memory the rewritten, although I=E2=80=99d = use a different verb and put at the end: =E2=80=98memory= changed=E2=80=99? > + -- (may be empty). > + for _, heap_chunk in pairs(event.primary) do > + local ev_line =3D symtab.demangle(symbols, heap_chunk.loc) > + > + if (heap_chunk.alloced > 0) then > + heap[ev_line].size_diff =3D heap[ev_line].size_diff + = heap_chunk.alloced > + heap[ev_line].cnt_alloc =3D heap[ev_line].cnt_alloc + = heap_chunk.cnt > + end > + > + if (heap_chunk.freed > 0) then > + heap[ev_line].size_diff =3D heap[ev_line].size_diff - = heap_chunk.freed > + heap[ev_line].cnt_free =3D heap[ev_line].cnt_free + = heap_chunk.cnt > + end > + end > + end > + end > + process_non_alloc_events(events.realloc) > + process_non_alloc_events(events.free) > + return heap > +end > + > +return M > --=20 > 2.31.0 >=20