Re: [Tarantool-patches] [PATCH luajit 2/2] profilers: purge generation mechanism

[Maxim Kokryashkin via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

[-- Attachment #1: Type: text/plain, Size: 4978 bytes --]


Hi, Sergey!
Thanks for the comments!
Here is the new message:
=======
profilers: purge generation mechanism
 
Since both of the profiler parsers are now processing
the events in a stream-like fashion, the generation
mechanism is excessive and can be purged. This results
in a significant memory consumption drop, especially
for the AVL-tree part.
 
Consider this script:
| jit.off()
| misc.sysprof.start{mode = 'C', interval=10}
| for i = 1, 1e7 do tostring(i) end
| misc.sysprof.stop()
 
After executing it with LuaJIT, you can parse it like this:
| $ time -v luajit-parse-sysprof sysprof.bin
 
So, before the patch:
| Maximum resident set size (kbytes): 224928
 
And after the patch:
| Maximum resident set size (kbytes): 32780
 
That is the 85% reduction in memory consumption.
 
Follows up tarantool/tarantool#8700
=======
 
> 
>>Hi, Max
>> 
>>On 9/12/23 13:26, Maxim Kokryashkin wrote:
>>>Hi, Sergey!
>>>Sure, consider this script:
>>>| jit.off()
>>>| misc.sysprof.start{mode = 'C', interval=10}
>>>| for i = 1, 1e7 do tostring(i) end
>>>| misc.sysprof.stop()
>>I failed with step:
>>$ cat prof.lua
>>jit.off()
>>misc.sysprof.start{mode = 'C', interval=10}
>>for i = 1, 1e7 do tostring(i) end
>>misc.sysprof.stop()
>> 
>>$ ./build/src/luajit prof.lua
>>Segmentation fault (core dumped)
>That may happen sometimes, since we still haven’t merged my patch for ffuncs.
>> 
>>119       lj_wbuf_addu64(buf, (uint64_t)pt->firstline);
>>(gdb) bt
>>#0  0x0000563092387308 in stream_lfunc (buf=0x563092449458 <sysprof+24>, func=0x7fdec2bda0a0)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:119
>>#1  0x00005630923874a9 in stream_frame_lua (buf=0x563092449458 <sysprof+24>, frame=0x7fdec48a2cd8)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:141
>>#2  0x00005630923876b1 in stream_backtrace_lua (sp=0x563092449440 <sysprof>)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:169
>>#3  0x0000563092387ab1 in stream_guest (sp=0x563092449440 <sysprof>, vmstate=2)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:237
>>#4  0x0000563092387c3f in stream_event (sp=0x563092449440 <sysprof>, vmstate=2)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:274
>>#5  0x0000563092387d26 in sysprof_record_sample (sp=0x563092449440 <sysprof>, info=0x7ffe5b59b730)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:296
>>#6  0x0000563092387ddc in sysprof_signal_handler (sig=27, info=0x7ffe5b59b730, ctx=0x7ffe5b59b600)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:312
>>#7  <signal handler called>
>>#8  0x00005630923ca3c5 in lj_fff_res1 () at buildvm_x86.dasc:1890
>>#9  0x000056309232c8e2 in lua_pcall (L=0x7fdec48a1378, nargs=0, nresults=-1, errfunc=2)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1172
>>#10 0x000056309231ec29 in docall (L=0x7fdec48a1378, narg=0, clear=0) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:133
>>#11 0x000056309231f63d in handle_script (L=0x7fdec48a1378, argx=0x7ffe5b59c0b0)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:303
>>#12 0x000056309232059b in pmain (L=0x7fdec48a1378) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:603
>>#13 0x00005630923c9029 in lj_BC_FUNCC () at buildvm_x86.dasc:852
>>#14 0x000056309232cf4a in lua_cpcall (L=0x7fdec48a1378, func=0x5630923203a3 <pmain>, ud=0x0)
>>    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1207
>>#15 0x00005630923206b7 in main (argc=2, argv=0x7ffe5b59c0a8) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:632
>>(gdb)
>> 
>> 
>>> 
>>>After running it with luajit you can parse the output
>>>like this:
>>>| $ time -v luajit-parse-sysprof sysprof.bin
>>> 
>>>So, before the patch:
>>>| Maximum resident set size (kbytes): 224928
>>> 
>>>And after the patch:
>>>| Maximum resident set size (kbytes): 32780
>>> 
>>>Which is 85% reduction in memory consumption.
>> 
>>I propose to add these numbers to commit message.
>> 
>>>--
>>>Best regards,
>>>Maxim Kokryashkin
>>> 
>>>  
>>>>Вторник, 5 сентября 2023, 14:53 +03:00 от Sergey Bronnikov  <sergeyb@tarantool.org> :
>>>> 
>>>>Hi, Max
>>>>
>>>>thanks for the patch. See my comment below.
>>>>
>>>>
>>>>On  8/28/23 18 :23, Maxim Kokryashkin wrote:
>>>>> Since both of the profiler parsers are now processing
>>>>> the events in a stream-like fashion, the generation
>>>>> mechanism is excessive and can be purged. This results
>>>>> in a significant memory consumption drop, especially
>>>>> for the AVL-tree part.
>>>>Would be interesting to see a numbers with memory consumption
>>>>
>>>>before and after the patch.
>>>>
>>>> 
>>> 
>--
>Best regards,
>Maxim Kokryashkin
> 

[-- Attachment #2: Type: text/html, Size: 7404 bytes --]