From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 62B5D6EC55; Thu, 22 Jul 2021 14:47:24 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 62B5D6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1626954444; bh=y9PBNuzC1TtBpBlZP0L2VqDpe2Nf69yOKDZ92UU8bDE=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=m3s57F3Z+XHTQOg8rKttq9UGkETSw6Ibm7+l77NpOGvBZWskgfE5jGayQTRWU3aPw 9sf1gf1U6IcT6hZCRiljjG5JHH6rmJKbnsLfMlksSm7oaVgphA9G037IbbtOzJCv0S wCgO4PgmchXwCPEBepFNAaKmB+92Npc5+RnPpE5k= Received: from smtp30.i.mail.ru (smtp30.i.mail.ru [94.100.177.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id CA7946EC55 for ; Thu, 22 Jul 2021 14:47:22 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org CA7946EC55 Received: by smtp30.i.mail.ru with esmtpa (envelope-from ) id 1m6XAf-0007oK-J1; Thu, 22 Jul 2021 14:47:22 +0300 To: tarantool-patches@dev.tarantool.org, imun@tarantool.org, skaplun@tarantool.org Date: Thu, 22 Jul 2021 18:46:17 +0700 Message-Id: <20210722114617.194747-1-m.shishatskiy@tarantool.org> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C3FDAB68B812060C77E621B90589399EB5182A05F538085040DFF1BFE69897FA7CA036C9BC83D0617EDD78EE8F100A9AC61EDF6E4FE575E568 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE745229D52CF30560CEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F790063770BCEBC9193A33448638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8D2E3EB518F0B21DD515BBFDE7DC09892117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC55B19328CBC4F849A471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F44604297287769387670735201E561CDFBCA1751F6FD1C55BDD38FC3FD2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE140C956E756FBB7A1C9461EB66F04EBFD8FC6C240DEA7642DBF02ECDB25306B2B78CF848AE20165D0A6AB1C7CE11FEE35FF72824B19451C6BA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7A3CCBC2573AEBDE1C4224003CC83647689D4C264860C145E X-C1DE0DAB: C20DE7B7AB408E4181F030C43753B8186998911F362727C414F749A5E30D975C16CF367659D795C060052791388BC672BBB26179FC5DEFEF9C2B6934AE262D3EE7EAB7254005DCEDBF2DBB2B01C011AE92E1F3B950BCE21616EB5DE968479BF5B2CF3E77A3BD9B2CBDC6A1CF3F042BAD6DF99611D93F60EF4280523C145DA091699F904B3F4130E343918A1A30D5E7FCCB5012B2E24CD356 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D3463DBE2ADA183F62FFAD673D1189E4344A65F57F76B1A784EE5BE978F46A015C8BAEE44B8021586741D7E09C32AA3244C6C97477853AB8BA510A6728C9E7570F833C9DC155518937F927AC6DF5659F194 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojaAaPr+N/4d3B1Dlmxvkvpg== X-Mailru-Sender: 2317F5BEA8D613097CB8A225516EF02783C35FD06A2311D6B3AA55EB8FDFA04B4013C8EABC40A469FF6B26DEAE20951894E739346DD9ABB838AE394B99C5394F99323991B04ED3930F27244EEAA5B9A5AE208404248635DF X-Mras: Ok Subject: [Tarantool-patches] [PATCH luajit v1] memprof: report all JIT-related allocations as internal X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mikhail Shishatskiy via Tarantool-patches Reply-To: Mikhail Shishatskiy Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" There are cases when the memory profiler attempts to attribute allocations triggered by the JIT engine recording phase with a Lua function to be recorded. In this case, lj_debug_frameline() may return BC_NOPOS (i.e. a negative value). Previously, these situations were ignored and the profiler reported, that the source line was equal to zero. This patch adjusts profiler behavior to treat allocations described above as internal by dumping ASOURCE_INT if lj_debug_frameline() returns a negative value. Resolves tarantool/tarantool#5679 --- Issue: https://github.com/tarantool/tarantool/issues/5679 Branch: https://github.com/tarantool/luajit/tree/shishqa/gh-5679-report-jit-allocations-as-internal src/lj_memprof.c | 28 +++++++---- .../misclib-memprof-lapi.test.lua | 50 ++++++++++++------- 2 files changed, 51 insertions(+), 27 deletions(-) diff --git a/src/lj_memprof.c b/src/lj_memprof.c index 2c1ef3b8..b4985b8e 100644 --- a/src/lj_memprof.c +++ b/src/lj_memprof.c @@ -89,19 +89,27 @@ static void memprof_write_lfunc(struct lj_wbuf *out, uint8_t aevent, GCfunc *fn, struct lua_State *L, cTValue *nextframe) { - const BCLine line = lj_debug_frameline(L, fn, nextframe); - lj_wbuf_addbyte(out, aevent | ASOURCE_LFUNC); - lj_wbuf_addu64(out, (uintptr_t)funcproto(fn)); /* - ** Line is >= 0 if we are inside a Lua function. - ** There are cases when the memory profiler attempts - ** to attribute allocations triggered by JIT engine recording - ** phase with a Lua function to be recorded. At this case - ** lj_debug_frameline() may return BC_NOPOS (i.e. a negative value). - ** Equals to zero when LuaJIT is built with the + ** Line equals to zero when LuaJIT is built with the ** -DLUAJIT_DISABLE_DEBUGINFO flag. */ - lj_wbuf_addu64(out, line >= 0 ? (uint64_t)line : 0); + const BCLine line = lj_debug_frameline(L, fn, nextframe); + + if (line < 0) { + /* + ** Line is >= 0 if we are inside a Lua function. + ** There are cases when the memory profiler attempts + ** to attribute allocations triggered by JIT engine recording + ** phase with a Lua function to be recorded. It this case, + ** lj_debug_frameline() may return BC_NOPOS (i.e. a negative value). + ** We report such allocations as internal in order not to confuse users. + */ + lj_wbuf_addbyte(out, aevent | ASOURCE_INT); + } else { + lj_wbuf_addbyte(out, aevent | ASOURCE_LFUNC); + lj_wbuf_addu64(out, (uintptr_t)funcproto(fn)); + lj_wbuf_addu64(out, (uint64_t)line); + } } static void memprof_write_cfunc(struct lj_wbuf *out, uint8_t aevent, diff --git a/test/tarantool-tests/misclib-memprof-lapi.test.lua b/test/tarantool-tests/misclib-memprof-lapi.test.lua index 06d96b3b..e35d8c29 100644 --- a/test/tarantool-tests/misclib-memprof-lapi.test.lua +++ b/test/tarantool-tests/misclib-memprof-lapi.test.lua @@ -7,7 +7,7 @@ require("utils").skipcond( local tap = require("tap") local test = tap.test("misc-memprof-lapi") -test:plan(13) +test:plan(14) jit.off() jit.flush() @@ -22,7 +22,7 @@ local symtab = require "utils.symtab" local TMP_BINFILE = arg[0]:gsub(".+/([^/]+)%.test%.lua$", "%.%1.memprofdata.tmp.bin") local BAD_PATH = arg[0]:gsub(".+/([^/]+)%.test%.lua$", "%1/memprofdata.tmp.bin") -local function payload() +local function default_payload() -- Preallocate table to avoid table array part reallocations. local _ = table_new(100, 0) @@ -37,7 +37,7 @@ local function payload() collectgarbage() end -local function generate_output(filename) +local function generate_output(filename, payload) -- Clean up all garbage to avoid pollution of free. collectgarbage() @@ -52,6 +52,25 @@ local function generate_output(filename) assert(res, err) end +local function generate_parsed_output(filename, payload) + local res, err = pcall(generate_output, filename, payload) + + -- Want to cleanup carefully if something went wrong. + if not res then + os.remove(filename) + error(err) + end + + local reader = bufread.new(filename) + local symbols = symtab.parse(reader) + local events = memprof.parse(reader) + + -- We don't need it any more. + os.remove(filename) + + return symbols, events +end + local function fill_ev_type(events, symbols, event_type) local ev_type = {} for _, event in pairs(events[event_type]) do @@ -107,20 +126,7 @@ test:ok(res == nil and err:match("profiler is not running")) test:ok(type(errno) == "number") -- Test profiler output and parse. -res, err = pcall(generate_output, TMP_BINFILE) - --- Want to cleanup carefully if something went wrong. -if not res then - os.remove(TMP_BINFILE) - error(err) -end - -local reader = bufread.new(TMP_BINFILE) -local symbols = symtab.parse(reader) -local events = memprof.parse(reader, symbols) - --- We don't need it any more. -os.remove(TMP_BINFILE) +local symbols, events = generate_parsed_output(TMP_BINFILE, default_payload) local alloc = fill_ev_type(events, symbols, "alloc") local free = fill_ev_type(events, symbols, "free") @@ -166,5 +172,15 @@ local co = coroutine.create(f) coroutine.resume(co) misc.memprof.stop() +-- Test for marking jit-related allocations as internal. +-- See also https://github.com/tarantool/tarantool/issues/5679. jit.on() +symbols, events = generate_parsed_output(TMP_BINFILE, function() + for _ = 1, 100 do + local _ = {_, _} + end +end) +alloc = fill_ev_type(events, symbols, "alloc") +test:ok(alloc[0] == nil) + os.exit(test:check() and 0 or 1) -- 2.32.0