From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 84F4B6FC83; Fri, 20 Aug 2021 10:08:04 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 84F4B6FC83 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1629443284; bh=Habna1Lgd+WB+yA0hhDnKgf1Fz98iteUjFf/BJ1Jo+w=; h=To:Date:In-Reply-To:References:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=dj040NmehnqTJTv+GWAmZyVJGimWa8aJ3A7J9ObboS8DLSDh98/ktu6AK3pXQiFLm i2MroZcMeeYTbjzYMqPDWylhn4WOkpzhn6tbmTCGYe14HGTSYdtcM2Zs9VCFnahVUA KW/zOdASJsw0lAzdtmq72W2ghy7qPpbs4FcUdHjM= Received: from smtp35.i.mail.ru (smtp35.i.mail.ru [94.100.177.95]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id ADDAF6FC83 for ; Fri, 20 Aug 2021 10:07:10 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ADDAF6FC83 Received: by smtp35.i.mail.ru with esmtpa (envelope-from ) id 1mGycP-0000G3-IW; Fri, 20 Aug 2021 10:07:10 +0300 To: tarantool-patches@dev.tarantool.org, imun@tarantool.org, skaplun@tarantool.org Date: Fri, 20 Aug 2021 14:05:44 +0700 Message-Id: <20210820070546.115293-4-m.shishatskiy@tarantool.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20210820070546.115293-1-m.shishatskiy@tarantool.org> References: <20210820070546.115293-1-m.shishatskiy@tarantool.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD92087353F0EC44DD9ECFD080E047A606F6525B29142351271182A05F5380850400B62B02B220F89BBFA3683C409882C52F60D0752DD76BFF44C82D1AA62E237AC X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE711269A7C2F827F16EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637D51A55327EA515C2EA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F27FCF21037FF7D47E60C9923761912897CC7F00164DA146DAFE8445B8C89999728AA50765F7900637D0FEED2715E18529389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC82FFDA4F57982C5F4F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA73AA81AA40904B5D9A18204E546F3947CCE135D2742255B35302FCEF25BFAB3454AD6D5ED66289B52698AB9A7B718F8C46E0066C2D8992A16725E5C173C3A84C3A333A05395E4745B76E601842F6C81A1F004C906525384307823802FF610243DF43C7A68FF6260569E8FC8737B5C2249EC8D19AE6D49635B68655334FD4449CB9ECD01F8117BC8BEAAAE862A0553A39223F8577A6DFFEA7CFA80D66F452D417A43847C11F186F3C59DAA53EE0834AAEE X-B7AD71C0: AC4F5C86D027EB782CDD5689AFBDA7A213B5FB47DCBC3458834459D11680B5051D81CCDA47EB054D74164D2B98E4194D X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975CF160826E4E1956AE5D5D688262E5C40D924DFB1ADAE433F49C2B6934AE262D3EE7EAB7254005DCED7532B743992DF240BDC6A1CF3F042BAD6DF99611D93F60EF836F5ADB0B4F9314699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34C264B329661203DADE483CD31E8BB637DE436A943027F9B036CB507322E460640C05AC501A98C4031D7E09C32AA3244CB353A762450BCDFD283F8F3EE55340D0E646F07CC2D4F3D8927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojZWOt9KMsrmdUDsQNXrvvdQ== X-Mailru-Sender: 2317F5BEA8D613097CB8A225516EF02711B936E641EB0915F91980540AE82061D4674E4C918C69D8FF6B26DEAE20951894E739346DD9ABB838AE394B99C5394F99323991B04ED3930F27244EEAA5B9A5AE208404248635DF X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit v3 3/5] memprof: dump traceno if allocate from trace X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mikhail Shishatskiy via Tarantool-patches Reply-To: Mikhail Shishatskiy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" When LuaJIT executes a trace, the trace number is stored in the virtual machine state. So, we can treat this number as an allocation event source in memprof and report allocation events from traces as well. Previously, all the allocations from traces were marked as INTERNAL. This patch introduces the functionality described above by adding a new allocation source type named ASOURCE_TRACE. If at the moment when allocation event occurs VM state indicates that trace executed, trace number streamed to a binary file: | loc-trace := trace-addr trace-no | trace-addr := | trace-no := Also, the memory profiler parser is adjusted to recognize this source type by extending structure: field , representing trace number, is added. Part of tarantool/tarantool#5814 --- Issue: https://github.com/tarantool/tarantool/issues/5814 Luajit branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number tarantool branch: https://github.com/tarantool/tarantool/tree/shishqa/gh-5814-group-allocations-on-trace-by-trace-number src/Makefile.dep.original | 2 +- src/lj_memprof.c | 35 +++++++++++++++++++++++++++++++++-- src/lj_memprof.h | 15 ++++++++++----- tools/memprof/parse.lua | 22 ++++++++++++++-------- 4 files changed, 58 insertions(+), 16 deletions(-) diff --git a/src/Makefile.dep.original b/src/Makefile.dep.original index f3672413..ee6bafb2 100644 --- a/src/Makefile.dep.original +++ b/src/Makefile.dep.original @@ -146,7 +146,7 @@ lj_mcode.o: lj_mcode.c lj_obj.h lua.h luaconf.h lj_def.h lj_arch.h \ lj_gc.h lj_err.h lj_errmsg.h lj_jit.h lj_ir.h lj_mcode.h lj_trace.h \ lj_dispatch.h lj_bc.h lj_traceerr.h lj_vm.h lj_memprof.o: lj_memprof.c lj_arch.h lua.h luaconf.h lj_memprof.h \ - lj_def.h lj_wbuf.h lj_obj.h lj_frame.h lj_bc.h lj_debug.h + lj_def.h lj_wbuf.h lj_obj.h lj_frame.h lj_bc.h lj_debug.h lj_dispatch.h lj_meta.o: lj_meta.c lj_obj.h lua.h luaconf.h lj_def.h lj_arch.h lj_gc.h \ lj_err.h lj_errmsg.h lj_buf.h lj_str.h lj_tab.h lj_meta.h lj_frame.h \ lj_bc.h lj_vm.h lj_strscan.h lj_strfmt.h lj_lib.h diff --git a/src/lj_memprof.c b/src/lj_memprof.c index 2c1ef3b8..fb99829d 100644 --- a/src/lj_memprof.c +++ b/src/lj_memprof.c @@ -19,6 +19,10 @@ #include "lj_frame.h" #include "lj_debug.h" +#if LJ_HASJIT +#include "lj_dispatch.h" +#endif + /* --------------------------------- Symtab --------------------------------- */ static const unsigned char ljs_header[] = {'l', 'j', 's', LJS_CURRENT_VERSION, @@ -146,6 +150,31 @@ static void memprof_write_func(struct memprof *mp, uint8_t aevent) lua_assert(0); } +#if LJ_HASJIT + +static void memprof_write_trace(struct memprof *mp, uint8_t aevent) +{ + struct lj_wbuf *out = &mp->out; + const global_State *g = mp->g; + const jit_State *J = G2J(g); + const TraceNo traceno = g->vmstate; + const GCtrace *trace = traceref(J, traceno); + lj_wbuf_addbyte(out, aevent | ASOURCE_TRACE); + lj_wbuf_addu64(out, (uintptr_t)trace->mcode); + lj_wbuf_addu64(out, (uint64_t)traceno); +} + +#else + +static void memprof_write_trace(struct memprof *mp, uint8_t aevent) +{ + UNUSED(mp); + UNUSED(aevent); + lua_assert(0); +} + +#endif + static void memprof_write_hvmstate(struct memprof *mp, uint8_t aevent) { lj_wbuf_addbyte(&mp->out, aevent | ASOURCE_INT); @@ -168,9 +197,11 @@ static const memprof_writer memprof_writers[] = { ** But since traces must follow the semantics of the original code, ** behaviour of Lua and JITted code must match 1:1 in terms of allocations, ** which makes using memprof with enabled JIT virtually redundant. - ** Hence use the stub below. + ** But if one wants to investigate allocations with JIT enabled, + ** memprof_write_trace() dumps trace number to the binary output. + ** It can be useful to compare with with jit.v or jit.dump outputs. */ - memprof_write_hvmstate /* LJ_VMST_TRACE */ + memprof_write_trace /* LJ_VMST_TRACE */ }; static void memprof_write_caller(struct memprof *mp, uint8_t aevent) diff --git a/src/lj_memprof.h b/src/lj_memprof.h index 3417475d..6a35385d 100644 --- a/src/lj_memprof.h +++ b/src/lj_memprof.h @@ -51,9 +51,10 @@ */ #define SYMTAB_LFUNC ((uint8_t)0) +#define SYMTAB_TRACE ((uint8_t)1) #define SYMTAB_FINAL ((uint8_t)0x80) -#define LJM_CURRENT_FORMAT_VERSION 0x01 +#define LJM_CURRENT_FORMAT_VERSION 0x02 /* ** Event stream format: @@ -69,11 +70,14 @@ ** event-realloc := event-header loc? oaddr osize naddr nsize ** event-free := event-header loc? oaddr osize ** event-header := -** loc := loc-lua | loc-c +** loc := loc-lua | loc-c | loc-trace ** loc-lua := sym-addr line-no ** loc-c := sym-addr +** loc-trace := trace-addr trace-no ** sym-addr := ** line-no := +** trace-addr := +** trace-no := ** oaddr := ** naddr := ** osize := @@ -88,10 +92,10 @@ ** version: [VVVVVVVV] ** * VVVVVVVV: Byte interpreted as a plain integer version number ** -** event-header: [FUUUSSEE] +** event-header: [FUUSSSEE] ** * EE : 2 bits for representing allocation event type (AEVENT_*) -** * SS : 2 bits for representing allocation source type (ASOURCE_*) -** * UUU : 3 unused bits +** * SSS : 3 bits for representing allocation source type (ASOURCE_*) +** * UU : 2 unused bits ** * F : 0 for regular events, 1 for epilogue's *F*inal header ** (if F is set to 1, all other bits are currently ignored) */ @@ -105,6 +109,7 @@ #define ASOURCE_INT ((uint8_t)(1 << 2)) #define ASOURCE_LFUNC ((uint8_t)(2 << 2)) #define ASOURCE_CFUNC ((uint8_t)(3 << 2)) +#define ASOURCE_TRACE ((uint8_t)(4 << 2)) #define LJM_EPILOGUE_HEADER 0x80 diff --git a/tools/memprof/parse.lua b/tools/memprof/parse.lua index 12e2758f..adc7c072 100644 --- a/tools/memprof/parse.lua +++ b/tools/memprof/parse.lua @@ -11,7 +11,7 @@ local lshift = bit.lshift local string_format = string.format local LJM_MAGIC = "ljm" -local LJM_CURRENT_VERSION = 1 +local LJM_CURRENT_VERSION = 0x02 local LJM_EPILOGUE_HEADER = 0x80 @@ -24,8 +24,11 @@ local AEVENT_MASK = 0x3 local ASOURCE_INT = lshift(1, 2) local ASOURCE_LFUNC = lshift(2, 2) local ASOURCE_CFUNC = lshift(3, 2) +local ASOURCE_TRACE = lshift(4, 2) -local ASOURCE_MASK = lshift(0x3, 2) +local ASOURCE_MASK = lshift(0x7, 2) + +local EV_HEADER_MAX = ASOURCE_TRACE + AEVENT_REALLOC local M = {} @@ -59,20 +62,23 @@ local function link_to_previous(heap_chunk, e, nsize) end end -local function id_location(addr, line) - return string_format("f%#xl%d", addr, line), { +local function id_location(addr, line, traceno) + return string_format("f%#xl%dt%d", addr, line, traceno), { addr = addr, line = line, + traceno = traceno, } end local function parse_location(reader, asource) if asource == ASOURCE_INT then - return id_location(0, 0) + return id_location(0, 0, 0) elseif asource == ASOURCE_CFUNC then - return id_location(reader:read_uleb128(), 0) + return id_location(reader:read_uleb128(), 0, 0) elseif asource == ASOURCE_LFUNC then - return id_location(reader:read_uleb128(), reader:read_uleb128()) + return id_location(reader:read_uleb128(), reader:read_uleb128(), 0) + elseif asource == ASOURCE_TRACE then + return id_location(reader:read_uleb128(), 0, reader:read_uleb128()) end error("Unknown asource "..asource) end @@ -140,7 +146,7 @@ local parsers = { } local function ev_header_is_valid(evh) - return evh <= 0x0f or evh == LJM_EPILOGUE_HEADER + return evh <= EV_HEADER_MAX or evh == LJM_EPILOGUE_HEADER end -- Splits event header into event type (aka aevent = allocation -- 2.32.0