From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 199BD6EC55; Wed, 21 Jul 2021 12:44:38 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 199BD6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626860678; bh=in33HqgMsTqmFQiUd4AB5aoY0OZJ01JJo2Khi6LmfAA=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=fGyMZ+SSNxRKb2a3bnp5iIhQk/VfqlF/KlpSB7bVCC084C9juUovhYVzsjggjSFRL /DzW7rrGy/+3gg/eKWt57jXvGMmiGZyZx58W9aMFECRn7IMjJJkwn/0KghSiitEwFa DpVTjb8oyETW7ZcQmc/9jalCQg+dFMlRwd7MFNQw= Received: from smtp31.i.mail.ru (smtp31.i.mail.ru [94.100.177.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id A49BF6EC55 for ; Wed, 21 Jul 2021 12:44:36 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org A49BF6EC55 Received: by smtp31.i.mail.ru with esmtpa (envelope-from ) id 1m68mI-0008DL-W7; Wed, 21 Jul 2021 12:44:35 +0300 To: tarantool-patches@dev.tarantool.org, imun@tarantool.org, skaplun@tarantool.org Date: Wed, 21 Jul 2021 16:44:28 +0700 Message-Id: <20210721094428.1382809-1-m.shishatskiy@tarantool.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C320FB367CC9CBEE800EEBD3117ABA9A5B182A05F538085040AA9C1674B8F3D475ED253ED87B854DF678AFE402892715F5FA4678B0226BE83E X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE70312E9A300D47E3BEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637DC205F3977E1285D8638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8AD30E47A3528B25EF872D8AA4937D67E117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC974A882099E279BDA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD18CB629EEF1311BF91D2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EEB28585415E75ADA92AE38A8E97BAFFB1D8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE386A7C529F68B8E5CBA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7588D3C263EAE74EA731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C16CF367659D795C0584160B0E76C27AEAD35265A0BD0A9109C2B6934AE262D3EE7EAB7254005DCEDEA3E392ABAD8FB996DABF04D5057A81F728CF7B057D10C70632EDEA9CD5989A3B936CB490224F2464EEA7BD89490CAC0EDDA962BC3F61961 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A541DB4061E93832BB6457DECF57D356090B88495D3DA513AA5F78836EFF0F762B6DB1E0B290F4941D7E09C32AA3244C2CF3ACD748AD6CB1632B3DDE572CE38BF94338140B71B8EE927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojbL9S8ysBdXhfeiSuiTQ1PYEuSwmd5C2A X-Mailru-Sender: 2317F5BEA8D613097CB8A225516EF0270ECB99E3D381D51EE84C1B07ECBED38F1D639FD5A6060D69FF6B26DEAE20951894E739346DD9ABB838AE394B99C5394F99323991B04ED3930F27244EEAA5B9A5AE208404248635DF X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit v1] memprof: group allocations on traces by trace number X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mikhail Shishatskiy via Tarantool-patches Reply-To: Mikhail Shishatskiy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" When luajit executes a trace, it's id is stored in virtual machine state. So, we can treat this trace number as allocation event source in memprof and report allocation events from traces as well. Previously all the allocations from traces were marked as INTERNAL. This patch introduces functionality described above by adding new allocation source type named ASOURCE_TRACE. If at the moment when allocation event occur vmstate indicates that trace is executed, trace id is streamed to binary file: | loc-trace := trace-no | trace-no := Also, memory profiler parser was tuned to recognize this source type by extending structure: field trace, representing trace id, was added. Now, for example, all the allocation events happened in trace with id 42, will be reported at source 'TRACE [42]' and not at 'INTERNAL' source. Closes tarantool/tarantool#5814 --- Issue: https://github.com/tarantool/tarantool/issues/5814 Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number src/lj_memprof.c | 16 ++++++++-------- src/lj_memprof.h | 9 ++++++--- tools/memprof/parse.lua | 18 +++++++++++------- tools/utils/symtab.lua | 5 +++++ 4 files changed, 30 insertions(+), 18 deletions(-) diff --git a/src/lj_memprof.c b/src/lj_memprof.c index 2c1ef3b8..0cbc0ed7 100644 --- a/src/lj_memprof.c +++ b/src/lj_memprof.c @@ -146,6 +146,13 @@ static void memprof_write_func(struct memprof *mp, uint8_t aevent) lua_assert(0); } +static void memprof_write_trace(struct memprof *mp, uint8_t aevent) +{ + struct lj_wbuf *out = &mp->out; + lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE); + lj_wbuf_addu64(out, (uint64_t)mp->g->vmstate); /* write traceno. */ +} + static void memprof_write_hvmstate(struct memprof *mp, uint8_t aevent) { lj_wbuf_addbyte(&mp->out, aevent | ASOURCE_INT); @@ -163,14 +170,7 @@ static const memprof_writer memprof_writers[] = { memprof_write_hvmstate, /* LJ_VMST_RECORD */ memprof_write_hvmstate, /* LJ_VMST_OPT */ memprof_write_hvmstate, /* LJ_VMST_ASM */ - /* - ** XXX: In ideal world, we should report allocations from traces as well. - ** But since traces must follow the semantics of the original code, - ** behaviour of Lua and JITted code must match 1:1 in terms of allocations, - ** which makes using memprof with enabled JIT virtually redundant. - ** Hence use the stub below. - */ - memprof_write_hvmstate /* LJ_VMST_TRACE */ + memprof_write_trace /* LJ_VMST_TRACE */ }; static void memprof_write_caller(struct memprof *mp, uint8_t aevent) diff --git a/src/lj_memprof.h b/src/lj_memprof.h index 3417475d..13125536 100644 --- a/src/lj_memprof.h +++ b/src/lj_memprof.h @@ -69,11 +69,13 @@ ** event-realloc := event-header loc? oaddr osize naddr nsize ** event-free := event-header loc? oaddr osize ** event-header := -** loc := loc-lua | loc-c +** loc := loc-lua | loc-c | loc-trace ** loc-lua := sym-addr line-no ** loc-c := sym-addr +** loc-trace := trace-no ** sym-addr := ** line-no := +** trace-no := ** oaddr := ** naddr := ** osize := @@ -90,8 +92,8 @@ ** ** event-header: [FUUUSSEE] ** * EE : 2 bits for representing allocation event type (AEVENT_*) -** * SS : 2 bits for representing allocation source type (ASOURCE_*) -** * UUU : 3 unused bits +** * SSS : 3 bits for representing allocation source type (ASOURCE_*) +** * UU : 2 unused bits ** * F : 0 for regular events, 1 for epilogue's *F*inal header ** (if F is set to 1, all other bits are currently ignored) */ @@ -105,6 +107,7 @@ #define ASOURCE_INT ((uint8_t)(1 << 2)) #define ASOURCE_LFUNC ((uint8_t)(2 << 2)) #define ASOURCE_CFUNC ((uint8_t)(3 << 2)) +#define ASOURCE_TRACE ((uint8_t)(4 << 2)) #define LJM_EPILOGUE_HEADER 0x80 diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua index 12e2758f..2bd491c8 100644 --- a/tools/memprof/parse.lua +++ b/tools/memprof/parse.lua @@ -24,8 +24,9 @@ local AEVENT_MASK = 0x3 local ASOURCE_INT = lshift(1, 2) local ASOURCE_LFUNC = lshift(2, 2) local ASOURCE_CFUNC = lshift(3, 2) +local ASOURCE_TRACE = lshift(4, 2) -local ASOURCE_MASK = lshift(0x3, 2) +local ASOURCE_MASK = lshift(0x7, 2) local M = {} @@ -59,20 +60,23 @@ local function link_to_previous(heap_chunk, e, nsize) end end -local function id_location(addr, line) - return string_format("f%#xl%d", addr, line), { +local function id_location(addr, line, trace) + return string_format("f%#xl%dxt%d", addr, line, trace), { addr = addr, line = line, + trace = trace, } end local function parse_location(reader, asource) if asource == ASOURCE_INT then - return id_location(0, 0) + return id_location(0, 0, 0) elseif asource == ASOURCE_CFUNC then - return id_location(reader:read_uleb128(), 0) + return id_location(reader:read_uleb128(), 0, 0) elseif asource == ASOURCE_LFUNC then - return id_location(reader:read_uleb128(), reader:read_uleb128()) + return id_location(reader:read_uleb128(), reader:read_uleb128(), 0) + elseif asource == ASOURCE_TRACE then + return id_location(0, 0, reader:read_uleb128()) end error("Unknown asource "..asource) end @@ -140,7 +144,7 @@ local parsers = { } local function ev_header_is_valid(evh) - return evh <= 0x0f or evh == LJM_EPILOGUE_HEADER + return evh <= 0x1f or evh == LJM_EPILOGUE_HEADER end -- Splits event header into event type (aka aevent = allocation diff --git a/tools/utils/symtab.lua b/tools/utils/symtab.lua index 3ed1dd13..6121177f 100644 --- a/tools/utils/symtab.lua +++ b/tools/utils/symtab.lua @@ -75,6 +75,11 @@ end function M.demangle(symtab, loc) local addr = loc.addr + local trace = loc.trace + + if trace ~= 0 then + return string_format("TRACE [%d]", trace) + end if addr == 0 then return "INTERNAL" -- 2.32.0