[Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Sergey Kaplun]

This patch embeds a parser for binary data dumped via the memory
profiler to Tarantool binary.

It is a set of the following Lua modules:
* utils/bufread.lua: read binary data from a file.
* utils/symtab.lua: symbol table decode functions
* memprof/parse.lua: decode the memory profiler event stream
* memprof/humanize.lua: display decoded data in human readable format
* memprof.lua: Lua script and module to display data

It launch with the following command:
$ tarantool -e 'require("memprof")(arg[1])' - filename.bin

Closed #5490
---

Issue: https://github.com/tarantool/tarantool/issues/5490
Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5442-luajit-memory-profiler
CI: https://gitlab.com/tarantool/tarantool/-/commits/skaplun/gh-5442-luajit-memory-profiler

 src/CMakeLists.txt |  6 ++++++
 src/lua/init.c     | 15 ++++++++++++++-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index b95688c1a..9a712bc29 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -61,6 +61,12 @@ lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/vmdef.lua
 lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/v.lua")
 lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/p.lua")
 lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/zone.lua")
+# LuaJIT tools.* library
+lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof.lua")
+lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof/humanize.lua")
+lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof/parse.lua")
+lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/utils/bufread.lua")
+lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/utils/symtab.lua")
 
 add_custom_target(generate_lua_sources
     WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/src/box
diff --git a/src/lua/init.c b/src/lua/init.c
index a0b2fc775..82075c595 100644
--- a/src/lua/init.c
+++ b/src/lua/init.c
@@ -121,7 +121,14 @@ extern char strict_lua[],
 	string_lua[],
 	swim_lua[],
 	p_lua[], /* LuaJIT 2.1 profiler */
-	zone_lua[] /* LuaJIT 2.1 profiler */;
+	zone_lua[], /* LuaJIT 2.1 profiler */
+	/* tools.* libraries. */
+	bufread_lua[],
+	symtab_lua[],
+	parse_lua[],
+	humanize_lua[],
+	memprof_lua[]
+;
 
 static const char *lua_modules[] = {
 	/* Make it first to affect load of all other modules */
@@ -167,6 +174,12 @@ static const char *lua_modules[] = {
 	/* Profiler */
 	"jit.p", p_lua,
 	"jit.zone", zone_lua,
+	/* tools.* libraries. Order is important. */
+	"utils.bufread", bufread_lua,
+	"utils.symtab", symtab_lua,
+	"memprof.parse", parse_lua,
+	"memprof.humanize", humanize_lua,
+	"memprof", memprof_lua,
 	NULL
 };
 
-- 
2.28.0

Re: [Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Igor Munkin]

Sergey,

Thanks for the patch! LGTM except a single nit.

On 28.12.20, Sergey Kaplun wrote:
> This patch embeds a parser for binary data dumped via the memory
> profiler to Tarantool binary.
> 
> It is a set of the following Lua modules:
> * utils/bufread.lua: read binary data from a file.
> * utils/symtab.lua: symbol table decode functions
> * memprof/parse.lua: decode the memory profiler event stream
> * memprof/humanize.lua: display decoded data in human readable format
> * memprof.lua: Lua script and module to display data
> 
> It launch with the following command:
> $ tarantool -e 'require("memprof")(arg[1])' - filename.bin
> 
> Closed #5490

Typo: s/Closed/Closes/.

> ---
> 
> Issue: https://github.com/tarantool/tarantool/issues/5490
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5442-luajit-memory-profiler
> CI: https://gitlab.com/tarantool/tarantool/-/commits/skaplun/gh-5442-luajit-memory-profiler
> 
>  src/CMakeLists.txt |  6 ++++++
>  src/lua/init.c     | 15 ++++++++++++++-
>  2 files changed, 20 insertions(+), 1 deletion(-)
> 

<snipped>

> -- 
> 2.28.0
> 

-- 
Best regards,
IM

Re: [Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Sergey Kaplun]

Igor,

On 28.12.20, Igor Munkin wrote:
> Sergey,
> 
> Thanks for the patch! LGTM except a single nit.
> 
> On 28.12.20, Sergey Kaplun wrote:
> > This patch embeds a parser for binary data dumped via the memory
> > profiler to Tarantool binary.
> > 
> > It is a set of the following Lua modules:
> > * utils/bufread.lua: read binary data from a file.
> > * utils/symtab.lua: symbol table decode functions
> > * memprof/parse.lua: decode the memory profiler event stream
> > * memprof/humanize.lua: display decoded data in human readable format
> > * memprof.lua: Lua script and module to display data
> > 
> > It launch with the following command:
> > $ tarantool -e 'require("memprof")(arg[1])' - filename.bin
> > 
> > Closed #5490
> 
> Typo: s/Closed/Closes/.

Fixed and repushed branch with --force.

> 
> > ---
> > 
> > Issue: https://github.com/tarantool/tarantool/issues/5490
> > Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5442-luajit-memory-profiler
> > CI: https://gitlab.com/tarantool/tarantool/-/commits/skaplun/gh-5442-luajit-memory-profiler
> > 
> >  src/CMakeLists.txt |  6 ++++++
> >  src/lua/init.c     | 15 ++++++++++++++-
> >  2 files changed, 20 insertions(+), 1 deletion(-)
> > 
> 
> <snipped>
> 
> > -- 
> > 2.28.0
> > 
> 
> -- 
> Best regards,
> IM

-- 
Best regards,
Sergey Kaplun

Re: [Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Alexander V. Tikhonov]

Hi Sergey, thanks for the patch, as I see no new degradation found in
gitlab-ci testing commit criteria pipeline [1], patch LGTM.

[1] - https://gitlab.com/tarantool/tarantool/-/pipelines/234950657

On Mon, Dec 28, 2020 at 10:04:31AM +0300, Sergey Kaplun via Tarantool-patches wrote:
> This patch embeds a parser for binary data dumped via the memory
> profiler to Tarantool binary.
> 
> It is a set of the following Lua modules:
> * utils/bufread.lua: read binary data from a file.
> * utils/symtab.lua: symbol table decode functions
> * memprof/parse.lua: decode the memory profiler event stream
> * memprof/humanize.lua: display decoded data in human readable format
> * memprof.lua: Lua script and module to display data
> 
> It launch with the following command:
> $ tarantool -e 'require("memprof")(arg[1])' - filename.bin
> 
> Closed #5490
> ---
> 
> Issue: https://github.com/tarantool/tarantool/issues/5490
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5442-luajit-memory-profiler
> CI: https://gitlab.com/tarantool/tarantool/-/commits/skaplun/gh-5442-luajit-memory-profiler
> 
>  src/CMakeLists.txt |  6 ++++++
>  src/lua/init.c     | 15 ++++++++++++++-
>  2 files changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> index b95688c1a..9a712bc29 100644
> --- a/src/CMakeLists.txt
> +++ b/src/CMakeLists.txt
> @@ -61,6 +61,12 @@ lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/vmdef.lua
>  lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/v.lua")
>  lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/p.lua")
>  lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/src/jit/zone.lua")
> +# LuaJIT tools.* library
> +lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof.lua")
> +lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof/humanize.lua")
> +lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/memprof/parse.lua")
> +lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/utils/bufread.lua")
> +lua_source(lua_sources "${CMAKE_BINARY_DIR}/third_party/luajit/tools/utils/symtab.lua")
>  
>  add_custom_target(generate_lua_sources
>      WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/src/box
> diff --git a/src/lua/init.c b/src/lua/init.c
> index a0b2fc775..82075c595 100644
> --- a/src/lua/init.c
> +++ b/src/lua/init.c
> @@ -121,7 +121,14 @@ extern char strict_lua[],
>  	string_lua[],
>  	swim_lua[],
>  	p_lua[], /* LuaJIT 2.1 profiler */
> -	zone_lua[] /* LuaJIT 2.1 profiler */;
> +	zone_lua[], /* LuaJIT 2.1 profiler */
> +	/* tools.* libraries. */
> +	bufread_lua[],
> +	symtab_lua[],
> +	parse_lua[],
> +	humanize_lua[],
> +	memprof_lua[]
> +;
>  
>  static const char *lua_modules[] = {
>  	/* Make it first to affect load of all other modules */
> @@ -167,6 +174,12 @@ static const char *lua_modules[] = {
>  	/* Profiler */
>  	"jit.p", p_lua,
>  	"jit.zone", zone_lua,
> +	/* tools.* libraries. Order is important. */
> +	"utils.bufread", bufread_lua,
> +	"utils.symtab", symtab_lua,
> +	"memprof.parse", parse_lua,
> +	"memprof.humanize", humanize_lua,
> +	"memprof", memprof_lua,
>  	NULL
>  };
>  
> -- 
> 2.28.0
> 

Re: [Tarantool-patches] [PATCH] build: embed LuaJIT memory profiler tools

[Igor Munkin]

Sergey,

On 28.12.20, Sergey Kaplun wrote:
> This patch embeds a parser for binary data dumped via the memory
> profiler to Tarantool binary.
> 
> It is a set of the following Lua modules:
> * utils/bufread.lua: read binary data from a file.
> * utils/symtab.lua: symbol table decode functions
> * memprof/parse.lua: decode the memory profiler event stream
> * memprof/humanize.lua: display decoded data in human readable format
> * memprof.lua: Lua script and module to display data
> 
> It launch with the following command:
> $ tarantool -e 'require("memprof")(arg[1])' - filename.bin
> 
> Closed #5490
> ---
> 
> Issue: https://github.com/tarantool/tarantool/issues/5490
> Branch: https://github.com/tarantool/tarantool/tree/skaplun/gh-5442-luajit-memory-profiler
> CI: https://gitlab.com/tarantool/tarantool/-/commits/skaplun/gh-5442-luajit-memory-profiler

I've checked your patch master.

> 

<snipped>

>  
> -- 
> 2.28.0
> 

-- 
Best regards,
IM