From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 5292C7030E; Mon, 25 Jan 2021 22:36:27 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 5292C7030E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1611603387; bh=mJ1FyNEPX629U32oY3p2HJ/JAp0aOzLSowok6AZrFI4=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=Nz26qvPCnLVBpC4OKTl7MutoZfbVPihLk6og2Cw3R+uaFOH33OTWBGzzLyGZYS1mA Dcu3nt2JP/3A5AR6Eqr41HQjM8J7gJHwat6rqIF/ghmhyz4EXVl0mO7hDt/U/S2dTv AXBHN3Ip0Bw+1vID0H6PZsvVIkzYHTB6O9+5AE0U= Received: from smtp52.i.mail.ru (smtp52.i.mail.ru [94.100.177.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 708927030E for ; Mon, 25 Jan 2021 22:36:23 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 708927030E Received: by smtp52.i.mail.ru with esmtpa (envelope-from ) id 1l47ew-00089a-69; Mon, 25 Jan 2021 22:36:22 +0300 To: Sergey Ostanevich , Igor Munkin Date: Mon, 25 Jan 2021 22:35:44 +0300 Message-Id: <20210125193544.18050-1-skaplun@tarantool.org> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD92BC6D7A73B5E1EC9A4696E262F57B05E134BE683453920D000894C459B0CD1B9D80CD5FF4B92A3258F503295F1AE84EED0B34205EB2C26C6E9977CD5C7BA570D X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7D114FA852BF02E08EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637B1850F7A782C2F9B8638F802B75D45FF5571747095F342E8C7A0BC55FA0FE5FCCF9CA6B51300925EF06CB84B67F5C14D121EFA3F09B1E0A9389733CBF5DBD5E913377AFFFEAFD269176DF2183F8FC7C0D9442B0B5983000E8941B15DA834481FCF19DD082D7633A0EF3E4896CB9E6436389733CBF5DBD5E9D5E8D9A59859A8B6957A4DEDD2346B42CC7F00164DA146DA6F5DAA56C3B73B23C77107234E2CFBA567F23339F89546C55F5C1EE8F4F765FC47E915A23788DB9575ECD9A6C639B01BBD4B6F7A4D31EC0BC0CAF46E325F83A522CA9DD8327EE4930A3850AC1BE2E7354DCCFF368A6260CFC4224003CC836476C0CAF46E325F83A50BF2EBBBDD9D6B0F5D41B9178041F3E72623479134186CDE6BA297DBC24807EABDAD6C7F3747799A X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C4C7A0BC55FA0FE5FCCF9CA6B51300925EF06CB84B67F5C14D8EF194C05B6D2925B1881A6453793CE9C32612AADDFBE061FA4CF60FEEF81E6A9510FB958DCE06DB6ED91DBE5ABE359ADBCB5631A0A9D21F23D4379F09C64C7393EDB24507CE13387DFF0A840B692CF8 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34B0359B88B232A0C6947E1C4CB369C5068C055D876A59AC032B4F4C0C08520D43A6A9DA02958143181D7E09C32AA3244C4CF47DF7F0C03D4B6E1E9F5EA2F28FA15595C85A795C7BAE927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXgC46Hy51GmsTfOtWPCGr3S X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB4666F23EC9F8FE26D3B9BB5400F92A0C6D732100F3E2561A8F2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A89437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit v1] tools: introduce --leak-only memprof parser option X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This patch adds a new --leak-only memory profiler parser option. When the parser runs with that option, it will report only lines (or "INTERNAL") that allocate and do not free some amount of bytes. The parser also reports the number of allocation and deallocation events related to each line. --- Branch: https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-memprof-memleaks-option Testing branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-noticket-memprof-memleaks-option Side note: I only update commit message thats why CI from the "old" commit. CI: https://gitlab.com/tarantool/tarantool/-/pipelines/246541599 This patch is a result of offline discussion of memory profiler and its usage and feedback from Mons. I don't know - is it reasonable to reference [1] issue here or create a new one? The example of output. Assuming we have file : | 1 jit.off() -- more verbose reports | 2 | 3 local function concat(a) | 4 local nstr = a.."a" | 5 return nstr | 6 end | 7 | 8 local function format(a) | 9 local nstr = string.format("%sa", a) | 10 return nstr | 11 end | 12 | 13 collectgarbage() -- cleanup | 14 | 15 local binfile = "/tmp/memprof_"..(arg[0]):match("/([^/]*).lua")..".bin" | 16 | 17 local st, err = misc.memprof.start(binfile) | 18 assert(st, err) | 19 | 20 for i = 1, 10000 do | 21 local f = format(i) | 22 local c = concat(i) | 23 end | 24 | 25 local st, err = misc.memprof.stop() | 26 assert(st, err) | 27 | 28 os.exit() Parser's output without option: | ALLOCATIONS | @/home/burii/reports/demo_memprof/format_concat.lua:8, line 9: 19998 624322 0 | | REALLOCATIONS | | DEALLOCATIONS | INTERNAL: 283 0 5602 | Overrides: | @/home/burii/reports/demo_memprof/format_concat.lua:8, line 9 | | @/home/burii/reports/demo_memprof/format_concat.lua:8, line 9: 2 0 98304 | Overrides: | @/home/burii/reports/demo_memprof/format_concat.lua:8, line 9 And with: | HEAP SUMMARY: | @/home/burii/reports/demo_memprof/format_concat.lua:8, line 9 holds 553214 bytes 19998 allocs, 283 frees Side note: The attentive reader will notice that we have 283+2 frees in the first case and 283 in the second. This is because --leak-only considers deallocation/reallocation events for which we know the source address. The default report aggregates these all deallocations into one. See `link_to_previous()` in for details. tools/memprof.lua | 23 +++++----- tools/memprof/humanize.lua | 91 +++++++++++++++++++++++++++++++++++++- tools/memprof/parse.lua | 20 +++++++-- 3 files changed, 115 insertions(+), 19 deletions(-) diff --git a/tools/memprof.lua b/tools/memprof.lua index 9f96208..5d09fbd 100644 --- a/tools/memprof.lua +++ b/tools/memprof.lua @@ -37,6 +37,11 @@ Supported options are: os.exit(0) end +local leak_only = false +opt_map["leak-only"] = function() + leak_only = true +end + -- Print error and exit with error status. local function opterror(...) stderr:write("luajit-parse-memprof.lua: ERROR: ", ...) @@ -94,19 +99,11 @@ local function dump(inputfile) local reader = bufread.new(inputfile) local symbols = symtab.parse(reader) local events = memprof.parse(reader, symbols) - - stdout:write("ALLOCATIONS", "\n") - view.render(events.alloc, symbols) - stdout:write("\n") - - stdout:write("REALLOCATIONS", "\n") - view.render(events.realloc, symbols) - stdout:write("\n") - - stdout:write("DEALLOCATIONS", "\n") - view.render(events.free, symbols) - stdout:write("\n") - + if leak_only then + view.leak_only(events, symbols) + else + view.all(events, symbols) + end os.exit(0) end diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua index 109a39d..bad0597 100644 --- a/tools/memprof/humanize.lua +++ b/tools/memprof/humanize.lua @@ -28,8 +28,8 @@ function M.render(events, symbols) )) local prim_loc = {} - for _, loc in pairs(event.primary) do - table.insert(prim_loc, symtab.demangle(symbols, loc)) + for _, heap_chunk in pairs(event.primary) do + table.insert(prim_loc, symtab.demangle(symbols, heap_chunk.loc)) end if #prim_loc ~= 0 then table.sort(prim_loc) @@ -42,4 +42,91 @@ function M.render(events, symbols) end end +function M.all(events, symbols) + print("ALLOCATIONS") + M.render(events.alloc, symbols) + print("") + + print("REALLOCATIONS") + M.render(events.realloc, symbols) + print("") + + print("DEALLOCATIONS") + M.render(events.free, symbols) + print("") +end + +function M.leak_only(events, symbols) + -- Auto resurrects source event lines for counting/reporting. + local heap = setmetatable({}, {__index = function(t, line) + t[line] = { + size = 0, + cnt_alloc = 0, + cnt_free = 0, + } + return t[line] + end}) + + for _, event in pairs(events.alloc) do + if event.loc then + local ev_line = symtab.demangle(symbols, event.loc) + + heap[ev_line].size = heap[ev_line].size + event.alloc + if (event.alloc > 0) then + heap[ev_line].cnt_alloc = heap[ev_line].cnt_alloc + event.num + end + end + end + + -- Realloc and free events are pretty the same. + -- We don't interesting in aggregated alloc/free sizes for + -- the event, but only for new and old size values inside + -- alloc-realloc-free chain. Assuming that we have + -- no collisions between different object addresses. + local function process_non_alloc_events(events_by_type) + for _, event in pairs(events_by_type) do + -- Realloc and free events always have "primary" key + -- that references table with rewrited memory + -- (may be empty). + for _, heap_chunk in pairs(event.primary) do + local ev_line = symtab.demangle(symbols, heap_chunk.loc) + + heap[ev_line].size = heap[ev_line].size + heap_chunk.alloced + if (heap_chunk.alloced > 0) then + heap[ev_line].cnt_alloc = heap[ev_line].cnt_alloc + heap_chunk.cnt + end + + heap[ev_line].size = heap[ev_line].size - heap_chunk.freed + if (heap_chunk.freed > 0) then + heap[ev_line].cnt_free = heap[ev_line].cnt_free + heap_chunk.cnt + end + end + end + end + process_non_alloc_events(events.realloc) + process_non_alloc_events(events.free) + + local rest_heap = {} + for line, info in pairs(heap) do + -- Report "INTERNAL" events inconsistencies for profiling + -- with enabled jit. + if info.size > 0 then + table.insert(rest_heap, {line = line, hold_bytes = info.size}) + end + end + + table.sort(rest_heap, function(h1, h2) + return h1.hold_bytes > h2.hold_bytes + end) + + print("HEAP SUMMARY:") + for _, h in pairs(rest_heap) do + print(string.format( + "%s holds %d bytes %d allocs, %d frees", + h.line, h.hold_bytes, heap[h.line].cnt_alloc, heap[h.line].cnt_free + )) + end + print("") +end + return M diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua index 6dae22d..374686c 100644 --- a/tools/memprof/parse.lua +++ b/tools/memprof/parse.lua @@ -39,11 +39,23 @@ local function new_event(loc) } end -local function link_to_previous(heap_chunk, e) +local function link_to_previous(heap_chunk, e, nsize) -- Memory at this chunk was allocated before we start tracking. if heap_chunk then -- Save Lua code location (line) by address (id). - e.primary[heap_chunk[2]] = heap_chunk[3] + if e.primary[heap_chunk[2]] == nil then + e.primary[heap_chunk[2]] = { + loc = heap_chunk[3], + alloced = 0, + freed = 0, + cnt = 0, + } + end + -- Save memory diff heap information. + local location_data = e.primary[heap_chunk[2]] + location_data.alloced = location_data.alloced + nsize + location_data.freed = location_data.freed + heap_chunk[1] + location_data.cnt = location_data.cnt + 1 end end @@ -97,7 +109,7 @@ local function parse_realloc(reader, asource, events, heap) e.free = e.free + osize e.alloc = e.alloc + nsize - link_to_previous(heap[oaddr], e) + link_to_previous(heap[oaddr], e, nsize) heap[oaddr] = nil heap[naddr] = {nsize, id, loc} @@ -116,7 +128,7 @@ local function parse_free(reader, asource, events, heap) e.num = e.num + 1 e.free = e.free + osize - link_to_previous(heap[oaddr], e) + link_to_previous(heap[oaddr], e, 0) heap[oaddr] = nil end -- 2.28.0 [1]: https://github.com/tarantool/tarantool/issues/5657