From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 75B7890AC21; Wed, 20 Dec 2023 15:24:18 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 75B7890AC21 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1703075058; bh=QQ8cO0PxZ6tHYW01VkT9VI8Acufi11X25fKVN9wJ0+I=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=WOqVQ35bC/jYa1dsfTZN0BtJlbtols7cq0uEW5Nn39PalTEKYPfxl2pY3pSOHLD6s 4e+eL7cvomm5WkfuwcRnu+akTKKQtkxjcAt944QvyxEiSQu7Hk2KKD016sNX0DMqKh r1zSFd9ibAQJV5u/Ahs/L3z+zTaArQIw0hgv/7vg= Received: from smtp63.i.mail.ru (smtp63.i.mail.ru [95.163.41.100]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 62F8E90AC21 for ; Wed, 20 Dec 2023 15:24:17 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 62F8E90AC21 Received: by smtp63.i.mail.ru with esmtpa (envelope-from ) id 1rFvcV-000xef-2S; Wed, 20 Dec 2023 15:24:16 +0300 Message-ID: Date: Wed, 20 Dec 2023 15:24:15 +0300 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Maxim Kokryashkin , tarantool-patches@dev.tarantool.org, skaplun@tarantool.org References: <1db6ff8c7469b5f59d1ee13b824bc223b02883a2.1701888856.git.m.kokryashkin@tarantool.org> In-Reply-To: <1db6ff8c7469b5f59d1ee13b824bc223b02883a2.1701888856.git.m.kokryashkin@tarantool.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: EEAE043A70213CC8 X-77F55803: 4F1203BC0FB41BD94F30938EFF3283ECC735D96152AC6ECEFEF6C6067019C0911313CFAB8367EF908E2BE116634AD74D3D8C4E07E3D5E8E9F83B485BC2A33D0D34AA542DAA5E6B9470001EC1C84847AC X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE77C1346FE4B18DC51EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637BB2557E27C12D3EF8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D81827A82C30C3EEF8B124242C8755DFCA117882F4460429724CE54428C33FAD305F5C1EE8F4F765FCAE9A1BBD95851C5BA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD18F04B652EEC242312D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE26055571C92BF10FB341D7040ADD27A2D8FC6C240DEA76429C9F4D5AE37F343AA9539A8B242431040A6AB1C7CE11FEE32D01283D1ACF37BA2D242C3BD2E3F4C6C4224003CC836476E2F48590F00D11D6E2021AF6380DFAD1A18204E546F3947C062BEEFFB5F8EA3E2E808ACE2090B5E1725E5C173C3A84C3C5EA940A35A165FF2DBA43225CD8A89F0A35B161A8BF67C1156CCFE7AF13BCA4B5C8C57E37DE458BEDA766A37F9254B7 X-C1DE0DAB: 0D63561A33F958A50D408FAFE8EE5DBEBECB6E4E82EE2718B53BF95D14A621FAF87CCE6106E1FC07E67D4AC08A07B9B067F1C1C3ABB44F3ACB5012B2E24CD356 X-C8649E89: 1C3962B70DF3F0ADE00A9FD3E00BEEDF3FED46C3ACD6F73ED3581295AF09D3DF87807E0823442EA2ED31085941D9CD0AF7F820E7B07EA4CF8AE1F3B23805373D2EBC9D620726561E9263E3FC9C2AEA19CB18346D97C33EBFE5D0AE1B90484215E190DC598EAEA94F368ECAB709CB203E6BCDFAB950FDB71CA74DFFEFA5DC0E7F02C26D483E81D6BE0DBAE6F56676BC7117BB6831D7356A2DEC5B5AD62611EEC62B5AFB4261A09AF0 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojYtkvzPaMmB6UFvjsLqDPpg== X-Mailru-Sender: 11C2EC085EDE56FAC07928AF2646A7694BAE919010AF8EAE0CBD693451ABD547E5E949923882BAADEBA65886582A37BD66FEC6BF5C9C28D98A98C1125256619760D574B6FC815AB872D6B4FCE48DF648AE208404248635DF X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit v2 5/6] memprof: introduce the `--human-readable` option X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Thanks for the patch! LGTM On 12/6/23 21:55, Maxim Kokryashkin wrote: > Prior to this patch, memprof could report only the raw > amount of bytes, which is hard to read. This patch adds the > `--human-readable` CLI option to the memprof parser, so the > memory is reported in KiB, MiB, or GiB, depending on what's > the biggest reasonable unit. > > This patch also refactors the options mechanism for parser, > so all of the options are passed into parser's modules as a > single config table instead of being handled individually. > > Part of tarantool/tarantool#5994 > --- > .../gh-5994-memprof-human-readable.test.lua | 73 +++++++++++++++++++ > tools/memprof.lua | 20 +++-- > tools/memprof/humanize.lua | 46 +++++++++--- > 3 files changed, 123 insertions(+), 16 deletions(-) > create mode 100644 test/tarantool-tests/gh-5994-memprof-human-readable.test.lua > > diff --git a/test/tarantool-tests/gh-5994-memprof-human-readable.test.lua b/test/tarantool-tests/gh-5994-memprof-human-readable.test.lua > new file mode 100644 > index 00000000..e34291be > --- /dev/null > +++ b/test/tarantool-tests/gh-5994-memprof-human-readable.test.lua > @@ -0,0 +1,73 @@ > +local tap = require('tap') > +local test = tap.test('gh-5994-memprof-human-readable'):skipcond({ > + ['Profile tools are implemented for x86_64 only'] = jit.arch ~= 'x86' and > + jit.arch ~= 'x64', > + ['Profile tools are implemented for Linux only'] = jit.os ~= 'Linux', > + -- XXX: Tarantool integration is required to run this test properly. > + -- luacheck: no global > + ['No profile tools CLI option integration'] = _TARANTOOL, > +}) > + > +local utils = require('utils') > +local TMP_BINFILE_MEMPROF = utils.tools.profilename('memprofdata.tmp.bin') > +local PARSE_CMD = utils.exec.luacmd(arg) .. ' -tm ' > + > +local function generate_output(bytes) > + local res, err = misc.memprof.start(TMP_BINFILE_MEMPROF) > + -- Should start successfully. > + assert(res, err) > + > + -- luacheck: no unused > + local _ = string.rep('_', bytes) > + > + res, err = misc.memprof.stop() > + -- Should stop successfully. > + assert(res, err) > +end > + > +local TEST_SET = { > + { > + bytes = 2049, > + match = '%dB', > + hr = false, > + name = 'non-human-readable mode is correct' > + }, > + { > + bytes = 100, > + match = '%dB', > + hr = true, > + name = 'human-readable mode: bytes' > + }, > + { > + bytes = 2560, > + match = '%d+%.%d%dKiB', > + hr = true, > + name = 'human-readable mode: float' > + }, > + { > + bytes = 2048, > + match = '%dKiB', > + hr = true, > + name = 'human-readable mode: KiB' > + }, > + { > + bytes = 1024 * 1024, > + match = '%dMiB', > + hr = true, > + name = 'human-readable mode: MiB' > + }, > + -- XXX: The test case for GiB is not implemented because it is > + -- OOM-prone for non-GC64 builds. > +} > + > +test:plan(#TEST_SET) > + > +for _, params in ipairs(TEST_SET) do > + generate_output(params.bytes) > + local cmd = PARSE_CMD .. (params.hr and ' --human-readable ' or '') > + local output = io.popen(cmd .. TMP_BINFILE_MEMPROF):read('*all') > + test:like(output, params.match, params.name) > +end > + > +os.remove(TMP_BINFILE_MEMPROF) > +test:done(true) > diff --git a/tools/memprof.lua b/tools/memprof.lua > index 955a1327..537cb869 100644 > --- a/tools/memprof.lua > +++ b/tools/memprof.lua > @@ -22,6 +22,12 @@ local match, gmatch = string.match, string.gmatch > -- Program options. > local opt_map = {} > > +-- Default config for the memprof parser. > +local config = { > + leak_only = false, > + human_readable = false, > +} > + > function opt_map.help() > stdout:write [[ > luajit-parse-memprof - parser of the memory usage profile collected > @@ -34,14 +40,18 @@ luajit-parse-memprof [options] memprof.bin > Supported options are: > > --help Show this help and exit > + --human-readable Use KiB/MiB/GiB notation instead of bytes > --leak-only Report only leaks information > ]] > os.exit(0) > end > > -local leak_only = false > opt_map["leak-only"] = function() > - leak_only = true > + config.leak_only = true > +end > + > +opt_map["human-readable"] = function() > + config.human_readable = true > end > > -- Print error and exit with error status. > @@ -101,11 +111,11 @@ local function dump(inputfile) > local reader = bufread.new(inputfile) > local symbols = symtab.parse(reader) > local events = memprof.parse(reader, symbols) > - if not leak_only then > - view.profile_info(events) > + if not config.leak_only then > + view.profile_info(events, config) > end > local dheap = process.form_heap_delta(events) > - view.leak_info(dheap) > + view.leak_info(dheap, config) > view.aliases(symbols) > -- XXX: The second argument is required to properly close Lua > -- universe (i.e. invoke before exiting). > diff --git a/tools/memprof/humanize.lua b/tools/memprof/humanize.lua > index 5b289ce3..a0b1693a 100644 > --- a/tools/memprof/humanize.lua > +++ b/tools/memprof/humanize.lua > @@ -5,7 +5,29 @@ > > local M = {} > > -function M.render(events) > +local function human_readable_bytes(bytes) > + local units = {"B", "KiB", "MiB", "GiB"} > + local magnitude = 1 > + > + while bytes >= 1024 and magnitude < #units do > + bytes = bytes / 1024 > + magnitude = magnitude + 1 > + end > + local is_int = math.floor(bytes) == bytes > + local fmt = is_int and "%d%s" or "%.2f%s" > + > + return string.format(fmt, bytes, units[magnitude]) > +end > + > +local function format_bytes(bytes, config) > + if config.human_readable then > + return human_readable_bytes(bytes) > + else > + return string.format('%dB', bytes) > + end > +end > + > +function M.render(events, config) > local ids = {} > > for id, _ in pairs(events) do > @@ -18,11 +40,11 @@ function M.render(events) > > for i = 1, #ids do > local event = events[ids[i]] > - print(string.format("%s: %d events\t+%d bytes\t-%d bytes", > + print(string.format("%s: %d events\t+%s\t-%s", > event.loc, > event.num, > - event.alloc, > - event.free > + format_bytes(event.alloc, config), > + format_bytes(event.free, config) > )) > > local prim_loc = {} > @@ -40,21 +62,21 @@ function M.render(events) > end > end > > -function M.profile_info(events) > +function M.profile_info(events, config) > print("ALLOCATIONS") > - M.render(events.alloc) > + M.render(events.alloc, config) > print("") > > print("REALLOCATIONS") > - M.render(events.realloc) > + M.render(events.realloc, config) > print("") > > print("DEALLOCATIONS") > - M.render(events.free) > + M.render(events.free, config) > print("") > end > > -function M.leak_info(dheap) > +function M.leak_info(dheap, config) > local leaks = {} > for line, info in pairs(dheap) do > -- Report "INTERNAL" events inconsistencies for profiling > @@ -71,8 +93,10 @@ function M.leak_info(dheap) > print("HEAP SUMMARY:") > for _, l in pairs(leaks) do > print(string.format( > - "%s holds %d bytes: %d allocs, %d frees", > - l.line, l.dbytes, dheap[l.line].nalloc, > + "%s holds %s: %d allocs, %d frees", > + l.line, > + format_bytes(l.dbytes, config), > + dheap[l.line].nalloc, > dheap[l.line].nfree > )) > end