From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 53F0C6EC55; Sat, 31 Jul 2021 19:51:47 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 53F0C6EC55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1627750307; bh=8mYYpApJ65sIgKP7UwIahCb1Xf9yUbA6VBk3I2lJy7g=; h=To:Cc:References:In-Reply-To:Date:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=wTTDhw0F1CifVI8LqyV7jRspySZe5VjgiR8t95cvvNCcODC/phwieypFzxMgHQlos +6hTef66qVcBDUHn633B2ks0kz0PBMvQ+fV3m3szXIehu03+i7sPVx66uPOXUQFwfU qaCvjWckpKYIbcuqSj1bThejNjZe4jMJ0EAGpleI= Received: from smtp48.i.mail.ru (smtp48.i.mail.ru [94.100.177.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 288DB6EC55 for ; Sat, 31 Jul 2021 19:51:46 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 288DB6EC55 Received: by smtp48.i.mail.ru with esmtpa (envelope-from ) id 1m9sDB-0008IQ-3c; Sat, 31 Jul 2021 19:51:45 +0300 To: "'Oleg Babin'" , Cc: References: <50175cbd00c57e3bc8eed222951a551f4bb7effb.1627468002.git.tsafin@tarantool.org> <2f63968b-490f-9abf-ab36-325553bd7609@tarantool.org> <040c01d78575$2e9dcb80$8bd96280$@tarantool.org> <6278fe67-a7a2-d98d-399f-0293b413c187@tarantool.org> In-Reply-To: <6278fe67-a7a2-d98d-399f-0293b413c187@tarantool.org> Date: Sat, 31 Jul 2021 19:51:44 +0300 Message-ID: <055001d7862c$584b93b0$08e2bb10$@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQNJI0ot4YBfKbZyj2tYNprdOv6c+gGi40uBAjeWjIsCo3+40wHS9Io9qDcZQLA= Content-Language: ru X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD941C43E597735A9C33D83595CA30D6DC5179D1C9A908C47E5182A05F538085040E65FDEE3A83C222B405C92368AB9E6093E6835F12F2D91C074E555B9EF0CBE17 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7BAE5222749FC9020C2099A533E45F2D0395957E7521B51C2CFCAF695D4D8E9FCEA1F7E6F0F101C6778DA827A17800CE72407438AC6002944EA1F7E6F0F101C6723150C8DA25C47586E58E00D9D99D84E1BDDB23E98D2D38BBCA57AF85F7723F25858B5C63B98DCBA93A8A39E1ED5796CCC7F00164DA146DAFE8445B8C89999728AA50765F790063783E00425F71A4181389733CBF5DBD5E9C8A9BA7A39EFB766F5D81C698A659EA7CC7F00164DA146DA9985D098DBDEAEC81D471462564A2E19F6B57BC7E6449061A352F6E88A58FB86F5D81C698A659EA73AA81AA40904B5D9A18204E546F3947CA5ED62E35AC703CC6E0066C2D8992A164AD6D5ED66289B52698AB9A7B718F8C46E0066C2D8992A16725E5C173C3A84C34E204DB0FBBFC51DBA3038C0950A5D36B5C8C57E37DE458B0BC6067A898B09E46D1867E19FE14079C09775C1D3CA48CF3D321E7403792E342EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE7E1BCFB2C0BE3F189731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A52B8620440AB79CCFC76638666BC8E48259614718207B4838D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA751B940EDA0DFB0535410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D34A08F09726D78E07BBE1AD00881A96BD05FB04EF5BAA97B49D9D3A9E800CC4B7A35DA030B9ADD29F81D7E09C32AA3244C8A2D9790F9662CE7EE3477EE3B8050BCA8CE788DE6831205FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojNLQ8Cqcv1itCgQBDZGmT5A== X-Mailru-Sender: B5B6A6EBBD94DAD886B83A8C333864068D8DBFAAACF61CBC6A86387AA4C3AD5A522E5A9FE53339C81EC9E4A2C82A33BC8C24925A86E657CE0C70AEE3C9A96FBAB3D7EE8ED63280BE112434F685709FCF0DA7A0AF5A3A8387 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in module datetime X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Timur Safin via Tarantool-patches Reply-To: Timur Safin Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" : From: Oleg Babin : Subject: Re: [Tarantool-patches] [PATCH resend v2 02/11] lua: built-in : module datetime :=20 : Thanks for your answers. :=20 : On 30.07.2021 22:00, Timur Safin wrote: : > Ok, here is a selection list which I've seen: : > - decimal is in LuaC; : > - uuid is in ffi; : > : > It looked equally acceptable and I've selected the one which is : > more convenient - via ffi. Now, you've pointed me to the probable : > performance issues we have with ffi, which are not there with LuaC. : > I believe they have been fixed with that Mike fix for NYI in : > structure allocations (after Sergos dirty fix) and once we will land : > this patch to our local fork those performance issues for structures : > allocations will gone. : > : > In any case, 1st version of module is not yet a proper moment to = start : > to worry about performance, more rather about clarity and quality of = API. : > We need to return to benchmarking of this code soon after code = freeze : > lifted. :=20 : I don't know why uuid is implemented via FFI. Uuid as module was : introduced at least 8 yeard ago. :=20 : Probably there were supporters of the theory that FFI/LuaJIT : implementation could be fast. :=20 : But currently we can see something like : https://github.com/luafun/luafun/issues/31#issuecomment-277508076 :=20 : and rewrite netbox in C. "I feel your pain, bro!" :) [here follows seemingly irrelevant speech, but actually it's on topic] I knew you have a plenty of LuaJIT experience, and sometimes,=20 you might want to throw it out altogether. All that pain of debugging = JIT performance issues. Woodoo and black magics of LuaJIT debugging. General = unpredictability of Mike responses. And even lack of convenient tooling here and there. I do understand how all this could exhaust eventually,=20 and make to loss of believe in LuaJIT usefulness. But we should not. We have skilled LuaJIT team here, which could handle all sorts of problems. We have huge amount of Lua code, and have no any reason to start the process of getting rid of it. Yes, there are cases=20 which might hit some performance degradations, but usually there are=20 reasonable workarounds. (Whom am I teaching here? You are much more = aware of all these tricks than I am!) Having our background and grownup skillset, there are actually no much=20 reasons to try to drastically reduce amount of LuaJIT usage, because=20 we continue to be used as LuaJIT-based application server for all=20 foreseeable future. [/end of seemingly irrelevant speech] To make this story short, I believe there is no much point to start=20 rewrite of this module in LuaC before we have it published and landed to = the Tarantool code-base. We need to make it feature complete first, then = "blazingly fast". In that order. :=20 : And it's not about patches in LuaJIT about structure initialization. = You : can easily cherry-pick it into :=20 : our fork and show that FFI is really better than Lua C API. It's about : an approach. We saw benchmarks :=20 : when decimal was introduced but currently we didn't see anything. :=20 : > : > + local y, M, d, ymd : > : > + y, M, d, ymd =3D 0, 0, 0, false : > : > + : > : > + local h, m, s, frac, hms : > : > + h, m, s, frac, hms =3D 0, 0, 0, 0, false : > : > + : > : > + local dt =3D 0 : > : > + : > : > + for key, value in pairs(o) do : > : > + local handlers =3D { : > : : > : Here you recreate handlers for each iteration of the loop. I think = it : > : should be reworked. : > : > Yes, let me check how better will be if this handlers array will be : invariant : > in the loop, and not created each iteration. : > : > I still prefer this tag to function approach because of a better = clarity, : > but I need to verify in benchmark whether it provides acceptable : performance. : > : > I do not expect thought that this attribute object creation approach = would : become : > a bootleneck in any case and might be on a critical path anywhere = (I'd : rather expect : > it for datetime objects created after parsing of massive bunch of = log : text), : > but need to look into timings we would have with invariant table. : > : > : : > : Currenlty it's quite slow I think even if-elseif-else branches = will work : > : faster without : > : : > : creating redundant GC objects. : > : > Will see ... :=20 : I'll simplify your task a bit: :=20 : local clock =3D require('clock') :=20 : local mt_fun =3D function(_, key) : local mt =3D { : ['key'] =3D function() : return 'value' : end : } : return mt[key] ~=3D nil and mt[key]() : end :=20 : local mt_table =3D { : ['key'] =3D function() : return 'value' : end : } :=20 : local t1 =3D setmetatable({}, {__index =3D mt_fun}) : local t2 =3D setmetatable({}, {__index =3D mt_table}) : local _ :=20 : local start =3D clock.time() : for _ =3D 1, 1e6 do : _ =3D t1['key'] : end : print('function', clock.time() - start) :=20 : collectgarbage() : collectgarbage() :=20 : local start =3D clock.time() : for _ =3D 1, 1e6 do : _ =3D t2['key'] : end : print('table', clock.time() - start) :=20 :=20 : tarantool test.lua : function 0.14332509040833 : table 0.0002901554107666 :=20 :=20 : The difference about x500 times on my Mac. Thanks! That's convincing! Thanks for this experiment, much appreciated! Interesting observation, though, that once we remove local=20 mt table creation from inside of mt_fun to the module scope, making it invariant for the cycle. The difference is not that much huge.=20 Debug build on WSL:: - your original version: ``` 19:22 $ ../build/src/tarantool bench-index.lua=20 function 0.16711115837097 table 0.00081968307495117 ``` - modified version with invariant mt table: ``` 19:25 $ ../build/src/tarantool bench-index.lua=20 function 0.00028538703918457 table 0.00028634071350098 ``` ```lua local clock =3D require('clock') local mt =3D { ['key'] =3D function() return 'value' end } local mt_fun =3D function(_, key) -- return mt[key]() return mt[key] ~=3D nil and mt[key]() end local mt_table =3D { ['key'] =3D function() return 'value' end } local t1 =3D setmetatable({}, {__index =3D mt_fun}) local t2 =3D setmetatable({}, {__index =3D mt_table}) local _ local start =3D clock.time() for _ =3D 1, 1e6 do _ =3D t1['key'] end print('function', clock.time() - start) collectgarbage() collectgarbage() local start =3D clock.time() for _ =3D 1, 1e6 do _ =3D t2['key'] end print('table', clock.time() - start) ``` So once again we have learned that it's not a good idea to create invariant table inside of loop, and simple=20 loop invariant removal might change things dramatically. Thanks for the lesson! :=20 : > : > +end : > : > + : > : > +local function strftime(fmt, o) : > : > + assert(ffi.typeof(o) =3D=3D datetime_t) : > : > + local sz =3D 50 : > : Why 50? : > : > Good question, for ISO-8601 formats, for normal date range : > (from 0000 till 9999 years) it's more like closer to 40, but : > well, there will be various timezones and all crazy kinds of : > input format combinations, so we need to either be precautious : > with larger sizes allocations. :=20 : On the one side you are right. But if user want to format string with : length more than 50 :=20 : it will be silently truncated. No, no, no, please see my 2 pass variant suggested below. We will ask `strftime` first to calculate the length for us (passing NULL as buffer pointer), then allocate, and fill the allocated buffer with content=20 the 2nd pass.=20 :=20 :=20 : ``` :=20 : tarantool> datetime.strftime('%d' .. string.rep('content', 50), : datetime.new()) : --- : - 01contentcontentcontentcontentcontentcontentconten : ... :=20 : ``` :=20 : I'm not sure that anybody will use strftime in such way but it's = strange. :=20 : Maybe you could use preallocated buffer for short strings and allocate : huge if :=20 : it's not enough. Some kind of optimistic approach. :=20 :=20 : > Or probably rely on glibc convention (which is apparently not in : > POSIX/SUSV) with 2 passes approach: : > 1. call with NULL so it will return size of required buffer; : > 2. then, after allocation, with the adjusted allocated buffer; : > : > = https://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar- : Time.html : > = https://pubs.opengroup.org/onlinepubs/000095399/functions/strftime.html : > : > Something like that (beware of inline patch): : > : > ---------------------------------------------- : > local function strftime(fmt, o) : > check_date(o, "datetime.strftime(fmt, date)") : > - local sz =3D 50 : > - local buff =3D ffi.new('char[?]', sz) : > local p_tm =3D datetime_to_tm_ptr(o) : > - native.strftime(buff, sz, fmt, p_tm) : > + local sz =3D builtin.strftime(nil, 1024, fmt, p_tm) : > + local buff =3D ffi.new('char[?]', sz + 1) : > + builtin.strftime(buff, sz, fmt, p_tm) : > return ffi.string(buff) : > end : > ---------------------------------------------- Thanks, Timur