From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id B28F16EC5F; Wed, 21 Apr 2021 21:21:33 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org B28F16EC5F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1619029293; bh=vb0y0CcLPDU7lesz+ApAau3MFUMfDaL7bS8NEeGe4JE=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=xIlUuJM32dVg80dnJLNKWlieP5cSNvlXhJojeRc5m8z69IZHrq7iM83SXgyGqLLKl NN35tJAn5oKa3pME1Qshqdi3aQHpV9l7219jl0IWpHVvWyW/ZF2CXwFkZgQjVHXUQE eemVYJn+K9UqgIZrUmN6J/U88zNbM4/0F8dVSHwc= Received: from smtp43.i.mail.ru (smtp43.i.mail.ru [94.100.177.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 3D32B6EC5F for ; Wed, 21 Apr 2021 21:21:32 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 3D32B6EC5F Received: by smtp43.i.mail.ru with esmtpa (envelope-from ) id 1lZHTf-0006lf-3y; Wed, 21 Apr 2021 21:21:31 +0300 Date: Wed, 21 Apr 2021 21:20:31 +0300 To: Sergey Bronnikov Message-ID: References: <20210311054949.91263-1-roman.habibov@tarantool.org> <20210314124220.GA16737@root> <092bc107-40cd-2c58-7a90-236663551249@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <092bc107-40cd-2c58-7a90-236663551249@tarantool.org> X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD9DD7F0C977691F2B1BCCA6D238B99472CA08DDD1076559A57182A05F538085040C5E1556A61558EAEEC7FEFBF89F5E8BFD6992F3C0EBC5BE7D87D92EF063EFB6C X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE7CCB2BF7EAE70380AEA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F79006377BAB68A65B44F13B8638F802B75D45FF914D58D5BE9E6BC1A93B80C6DEB9DEE97C6FB206A91F05B25466FD14650BCEECEEB4453DC147AF35A033A18F845E317FD2E47CDBA5A96583C09775C1D3CA48CF17B107DEF921CE79117882F4460429724CE54428C33FAD30A8DF7F3B2552694AC26CFBAC0749D213D2E47CDBA5A9658359CC434672EE6371117882F4460429728AD0CFFFB425014E868A13BD56FB6657E2021AF6380DFAD1A18204E546F3947CB11811A4A51E3B096D1867E19FE1407959CC434672EE6371089D37D7C0E48F6C8AA50765F79006371F24DFF1B2961425731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A5DE1BAB6F090681C5343731F5546A54B2E0109AA5B7D6D621D59269BC5F550898D99A6476B3ADF6B47008B74DF8BB9EF7333BD3B22AA88B938A852937E12ACA7502E6951B79FF9A3F410CA545F18667F91A7EA1CDA0B5A7A0 X-C8649E89: 4E36BF7865823D7055A7F0CF078B5EC49A30900B95165D347341ACA13FB8BDD9670C938A1E574CF0D8A2A239BEDB9680394653A4A78CA3BD3DAF0BD0D193B7751D7E09C32AA3244C4453C2DDAF71BEAB7CEC15E58724B9CFD08D48398F32B4A6FACE5A9C96DEB163 X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu530nj6fImhcD4MUrOEAnl0W826KZ9Q+tr5ycPtXkTV4k65bRjmOUUP8cvGozZ33TWg5HZplvhhXbhDGzqmQDTd6OAevLeAnq3Ra9uf7zvY2zzsIhlcp/Y7m53TZgf2aB4JOg4gkr2biojd3ipC3Yuge2SL+hznCjF/A== X-Mailru-Sender: 3B9A0136629DC91206CBC582EFEF4CB479A8A2D3149694F0CD159E21B2B8BD0C0615CD6DD314C17DF2400F607609286E924004A7DEC283833C7120B22964430C52B393F8C72A41A89437F6177E88F7363CDA0F3B3F5B9367 X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH] serializer: serialize recursive structures X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi! Thanks for benchmarks! I like them! But I am tormented by doubts whether it is worth keeping it in a semi-working state in our repository. Shouldn't we first introduce the architecture of performance tests, and then add some new benchmarks? I'm totally for the presence of such benchmarks in our repo, but I'm afraid to make a real mess of things. Also, I suppose that it should be a collegeal descicion (not only 2 reviewers). On 21.04.21, Sergey Bronnikov wrote: > > On 21.04.2021 12:27, Sergey Kaplun via Tarantool-patches wrote: > > Hi! > > > > Thanks for the fixes and the benchmarks! > > I propose an updated version of script and propose to add benchmark as a > part of patch series. > > --[[ > -- measure serialization time > -- run: taskset -c 1 tarantool perf.lua > ]] > > local clock = require('clock') > > local function elapsed(f, n) >     local t0 = clock.monotonic() I've read discussions that we should use CLOCK_PROCESS_CPUTIME_ID instead of CLOCK_MONOTONIC. Is it one preferable than another? >     for i = 1, n do >         f() >     end >     local t1 = clock.monotonic() >     return t1 - t0 > end Shouldn't we still take into account the time to call the function `clock.monotonic()`? > > -- Get the mean of a table > function calculate_mean(t) >   local sum = 0 >   local count= 0 >   for k, v in pairs(t) do >     if type(v) == 'number' then 2 general points: 1) use `for i = 1, #t do` instead pairs (as far as we use #t later) 2) use `assert(type(v) == 'number')` instead of if statement. Here and below. >       sum = sum + v >     end >   end > >   return (sum/#t) > end > > -- Get the median of a table. > function calculate_median(t) >   local temp = {} >   -- deep copy table so that when we sort it, the original is unchanged Are we interested in unchanged original table? May be it will be better to sort values inside before and work with them? >   -- also weed out any non numbers >   for k, v in pairs(t) do >     if type(v) == 'number' then >       table.insert(temp, v) >     end >   end >   table.sort(temp) >   -- If we have an even number of table elements or odd. >   if math.fmod(#temp,2) == 0 then >     -- return mean value of middle two elements >     return (temp[#temp/2] + temp[(#temp/2)+1]) / 2 >   else >     -- return middle element >     return temp[math.ceil(#temp/2)] >   end > end > > -- Get the standard deviation of a table > function calculate_stddev(t) >   local vm Sorry, don't get it: why do you choose this scope for the variable? >   local sum = 0 >   local mean = calculate_mean(t) >   for k, v in pairs(t) do >     if type(v) == 'number' then >       vm = v - mean >       sum = sum + (vm * vm) >     end >   end > >   return math.sqrt(sum/(#t - 1)) > end > > local function timeit(f, name) >     print('======================================') >     print(name) >     print('======================================') >     local res = {} >     local iterations = 10 >     local elapsed_time = 0 >     local repetitions = 150000 It will be nice to have an ability to configure this value (via config or environment variables). >     for j = 1, iterations do >         -- warming >         for i = 1, 100 do f() end >         -- measurement >         elapsed_time = elapsed(f, repetitions) >         table.insert(res, elapsed_time) >         print(string.format("%-2d - %f sec / %d repetitions", j, > elapsed_time, repetitions)) >     end >     print(string.format("time mean   %f", calculate_mean(res))) >     print(string.format("time median %f", calculate_median(res))) >     print(string.format("time stddev %f", calculate_stddev(res))) > end > > local t = {{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}} > timeit(function() >            local console = require('console') >            console.set_default_output('yaml') >            return console.eval(tostring(t)) >        end, 'serializer console yaml') It is better to move out all this actions and create a local copy of called functions. For example: =================================================================== local tstr = tostring(t) local console = require('console') console.set_default_output('yaml') local eval = console.eval timeit(function() return eval(tstr) end, 'serializer console yaml') =================================================================== It reduces time by 5% (stderr is 0.5%). > timeit(function() >            local console = require('console') >            console.set_default_output('lua') >            return console.eval(tostring(t)) >        end, 'serializer console lua') > timeit(function() >            local serializer = require('json') >            serializer.cfg({encode_max_depth = 64}) >            return serializer.encode(t) >        end, 'serializer json') > timeit(function() >            local serializer = require('yaml') >            serializer.cfg({encode_max_depth = 64}) >            return serializer.encode(t) >        end, 'serializer yaml') > timeit(function() >            local serializer = require('msgpack') >            serializer.cfg({encode_max_depth = 64}) >            return serializer.encode(t) >        end, 'serializer msgpack') > timeit(function() >            local serializer = require('msgpackffi') >            return serializer.encode(t) >        end, 'serializer msgpackffi') > -- Best regards, Sergey Kaplun