From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 171A05E0A61; Thu, 14 Sep 2023 12:47:03 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 171A05E0A61 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1694684823; bh=GLZdcWunu3vu11pIMj+2pdbZOI1Fk474tv+MhH5c3E0=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=qHzzKNxScrQKNwE50302DJTtCyQF/E/F30szPvuhpEKmBFgu1jKR5zxhQ4K08VOcC x0WuT6o4F/dWbht2fP6WELMy4uS4nNoYZuYlYMxxnA7wGOUCOZMK1N8LOI5D4UT0MY r+GAKPKK59cTWl1RzJkIjdNWu5GAfiy1LES+IV0s= Received: from smtp36.i.mail.ru (smtp36.i.mail.ru [95.163.41.77]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 31AE75E0A61 for ; Thu, 14 Sep 2023 12:47:02 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 31AE75E0A61 Received: by smtp36.i.mail.ru with esmtpa (envelope-from ) id 1qgiw9-008Jh6-1J; Thu, 14 Sep 2023 12:47:01 +0300 Content-Type: multipart/alternative; boundary="------------rJGTXGDISh0ZUZW6rboANt2I" Message-ID: <370699e3-9e7d-f24b-600d-970cd5bed080@tarantool.org> Date: Thu, 14 Sep 2023 12:47:00 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Content-Language: en-US To: Maxim Kokryashkin References: <6bb71c35-6c4e-ae08-7a8c-f501c6f935c0@tarantool.org> <1694514399.719299784@f315.i.mail.ru> In-Reply-To: <1694514399.719299784@f315.i.mail.ru> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: B8F34718100C35BD X-77F55803: 4F1203BC0FB41BD927CFE6CA1630A10CA056C88976AC17650012AE68F2FF28C300894C459B0CD1B9828B84F9FE42400B71B72D61E348958070CD404837BB71A0E23F349B1C288C77 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE789DCD78EC958A690EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637BF3E16F0C87855308638F802B75D45FF36EB9D2243A4F8B5A6FCA7DBDB1FC311F39EFFDF887939037866D6147AF826D8DC1BF16C47E7A4A00AA6A19957D8EEBF117882F4460429724CE54428C33FAD305F5C1EE8F4F765FC974A882099E279BDA471835C12D1D9774AD6D5ED66289B52BA9C0B312567BB23117882F446042972877693876707352033AC447995A7AD186FD1C55BDD38FC3FD2E47CDBA5A96583BA9C0B312567BB2376E601842F6C81A19E625A9149C048EE1E561CDFBCA1751F1DCA69B2CC609F68D8FC6C240DEA76429C9F4D5AE37F343AA9539A8B242431040A6AB1C7CE11FEE386A7C529F68B8E5C9735652A29929C6CC4224003CC836476E2F48590F00D11D6E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407978DA827A17800CE773BC869C69ECC1572DBA43225CD8A89F0A35B161A8BF67C142539A7722CA490CB5C8C57E37DE458BEDA766A37F9254B7 X-C1DE0DAB: 0D63561A33F958A5BED354BAEF2AA001399284D3EF06C765DE0984AC4959E9A9F87CCE6106E1FC07E67D4AC08A07B9B0F254576263B31EA9CB5012B2E24CD356 X-C8649E89: 1C3962B70DF3F0ADE00A9FD3E00BEEDF3FED46C3ACD6F73ED3581295AF09D3DF87807E0823442EA2ED31085941D9CD0AF7F820E7B07EA4CFF7B282CFDFB80922E5506F47B4F50D0FBE5175D1F51E57D0BFD1B2C5151637C1DDCE95BB4BF576A6C4B8F2E8CFB8F571B0125B8B637F53F6E36C3DBA696AA0ABA74DFFEFA5DC0E7F02C26D483E81D6BE0DBAE6F56676BC7117BB6831D7356A2DEC5B5AD62611EEC62B5AFB4261A09AF0 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojoMTWofjSWSQV5hy56zpzIg== X-Mailru-Sender: C4F68CFF4024C8867DFDF7C7F2588458242AA6511DB219B82EC65B258C6D793D1762187F0D029BCF282EC151BADDC1D3523A6D01B4765B2DFB59E2DDD9FE06B14FA522850F29BC30B0DAF586E7D11B3E67EA787935ED9F1B X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit 2/2] profilers: purge generation mechanism X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Cc: Maxim Kokryashkin , tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" This is a multi-part message in MIME format. --------------rJGTXGDISh0ZUZW6rboANt2I Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Hi, Max On 9/12/23 13:26, Maxim Kokryashkin wrote: > Hi, Sergey! > Sure, consider this script: > | jit.off() > | misc.sysprof.start{mode = 'C', interval=10} > | for i = 1, 1e7 do tostring(i) end > | misc.sysprof.stop() I failed with step: $ cat prof.lua jit.off() misc.sysprof.start{mode = 'C', interval=10} for i = 1, 1e7 do tostring(i) end misc.sysprof.stop() $ ./build/src/luajit prof.lua Segmentation fault (core dumped) 119       lj_wbuf_addu64(buf, (uint64_t)pt->firstline); (gdb) bt #0  0x0000563092387308 in stream_lfunc (buf=0x563092449458 , func=0x7fdec2bda0a0)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:119 #1  0x00005630923874a9 in stream_frame_lua (buf=0x563092449458 , frame=0x7fdec48a2cd8)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:141 #2  0x00005630923876b1 in stream_backtrace_lua (sp=0x563092449440 )     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:169 #3  0x0000563092387ab1 in stream_guest (sp=0x563092449440 , vmstate=2)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:237 #4  0x0000563092387c3f in stream_event (sp=0x563092449440 , vmstate=2)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:274 #5  0x0000563092387d26 in sysprof_record_sample (sp=0x563092449440 , info=0x7ffe5b59b730)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:296 #6  0x0000563092387ddc in sysprof_signal_handler (sig=27, info=0x7ffe5b59b730, ctx=0x7ffe5b59b600)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:312 #7  #8  0x00005630923ca3c5 in lj_fff_res1 () at buildvm_x86.dasc:1890 #9  0x000056309232c8e2 in lua_pcall (L=0x7fdec48a1378, nargs=0, nresults=-1, errfunc=2)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1172 #10 0x000056309231ec29 in docall (L=0x7fdec48a1378, narg=0, clear=0) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:133 #11 0x000056309231f63d in handle_script (L=0x7fdec48a1378, argx=0x7ffe5b59c0b0)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:303 #12 0x000056309232059b in pmain (L=0x7fdec48a1378) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:603 #13 0x00005630923c9029 in lj_BC_FUNCC () at buildvm_x86.dasc:852 #14 0x000056309232cf4a in lua_cpcall (L=0x7fdec48a1378, func=0x5630923203a3 , ud=0x0)     at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1207 #15 0x00005630923206b7 in main (argc=2, argv=0x7ffe5b59c0a8) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:632 (gdb) > After running it with luajit you can parse the output > like this: > | $ time -v luajit-parse-sysprof sysprof.bin > So, before the patch: > | Maximum resident set size (kbytes): 224928 > And after the patch: > | Maximum resident set size (kbytes): 32780 > Which is 85% reduction in memory consumption. I propose to add these numbers to commit message. > -- > Best regards, > Maxim Kokryashkin > > Вторник, 5 сентября 2023, 14:53 +03:00 от Sergey Bronnikov > : > Hi, Max > > thanks for the patch. See my comment below. > > > On 8/28/23 18:23, Maxim Kokryashkin wrote: > > Since both of the profiler parsers are now processing > > the events in a stream-like fashion, the generation > > mechanism is excessive and can be purged. This results > > in a significant memory consumption drop, especially > > for the AVL-tree part. > > Would be interesting to see a numbers with memory consumption > > before and after the patch. > --------------rJGTXGDISh0ZUZW6rboANt2I Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit

Hi, Max


On 9/12/23 13:26, Maxim Kokryashkin wrote:
Hi, Sergey!
Sure, consider this script:
| jit.off()
| misc.sysprof.start{mode = 'C', interval=10}
| for i = 1, 1e7 do tostring(i) end
| misc.sysprof.stop()

I failed with step:

$ cat prof.lua
jit.off()
misc.sysprof.start{mode = 'C', interval=10}
for i = 1, 1e7 do tostring(i) end
misc.sysprof.stop()

$ ./build/src/luajit prof.lua
Segmentation fault (core dumped)

119       lj_wbuf_addu64(buf, (uint64_t)pt->firstline);
(gdb) bt
#0  0x0000563092387308 in stream_lfunc (buf=0x563092449458 <sysprof+24>, func=0x7fdec2bda0a0)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:119
#1  0x00005630923874a9 in stream_frame_lua (buf=0x563092449458 <sysprof+24>, frame=0x7fdec48a2cd8)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:141
#2  0x00005630923876b1 in stream_backtrace_lua (sp=0x563092449440 <sysprof>)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:169
#3  0x0000563092387ab1 in stream_guest (sp=0x563092449440 <sysprof>, vmstate=2)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:237
#4  0x0000563092387c3f in stream_event (sp=0x563092449440 <sysprof>, vmstate=2)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:274
#5  0x0000563092387d26 in sysprof_record_sample (sp=0x563092449440 <sysprof>, info=0x7ffe5b59b730)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:296
#6  0x0000563092387ddc in sysprof_signal_handler (sig=27, info=0x7ffe5b59b730, ctx=0x7ffe5b59b600)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_sysprof.c:312
#7  <signal handler called>
#8  0x00005630923ca3c5 in lj_fff_res1 () at buildvm_x86.dasc:1890
#9  0x000056309232c8e2 in lua_pcall (L=0x7fdec48a1378, nargs=0, nresults=-1, errfunc=2)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1172
#10 0x000056309231ec29 in docall (L=0x7fdec48a1378, narg=0, clear=0) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:133
#11 0x000056309231f63d in handle_script (L=0x7fdec48a1378, argx=0x7ffe5b59c0b0)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:303
#12 0x000056309232059b in pmain (L=0x7fdec48a1378) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:603
#13 0x00005630923c9029 in lj_BC_FUNCC () at buildvm_x86.dasc:852
#14 0x000056309232cf4a in lua_cpcall (L=0x7fdec48a1378, func=0x5630923203a3 <pmain>, ud=0x0)
    at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/lj_api.c:1207
#15 0x00005630923206b7 in main (argc=2, argv=0x7ffe5b59c0a8) at /home/sergeyb/sources/MRG/tarantool/third_party/luajit/src/luajit.c:632
(gdb)


 
After running it with luajit you can parse the output
like this:
| $ time -v luajit-parse-sysprof sysprof.bin
 
So, before the patch:
| Maximum resident set size (kbytes): 224928
 
And after the patch:
| Maximum resident set size (kbytes): 32780
 
Which is 85% reduction in memory consumption.


I propose to add these numbers to commit message.


--
Best regards,
Maxim Kokryashkin
 
 
Вторник, 5 сентября 2023, 14:53 +03:00 от Sergey Bronnikov <sergeyb@tarantool.org>:
 
Hi, Max

thanks for the patch. See my comment below.


On 8/28/23 18:23, Maxim Kokryashkin wrote:
> Since both of the profiler parsers are now processing
> the events in a stream-like fashion, the generation
> mechanism is excessive and can be purged. This results
> in a significant memory consumption drop, especially
> for the AVL-tree part.

Would be interesting to see a numbers with memory consumption

before and after the patch.

 
 
--------------rJGTXGDISh0ZUZW6rboANt2I--