From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id 9382D6ECED; Mon, 14 Sep 2026 11:20:41 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 9382D6ECED DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1789374041; bh=rbgYg1vU7/EW8+0+PwcISl+loHnn57ZGrl25yqJkTaA=; h=Date:To:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To:Cc: From; b=bfUpQ6xcAxkOReH9qoe3JCkoyI8tWxSU8vJ+SXZXZHhWrFD7p4cmvRmQxbLlMUCUC Xyq4KgMXSXxj8bUyvey2yE2u/aBxNHtL6siH0WzQRQnYw/mshnpqJZgLMa/Lw7ZgTs g/OCMMxcXiaBJQ7fBr8I5vwHV+woUiR2AqyXAlog= Received: from send174.i.mail.ru (send174.i.mail.ru [95.163.59.13]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 9E7526ECED for ; Mon, 14 Sep 2026 11:20:39 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 9E7526ECED Received: by exim-smtp-5d69b54bdd-b27p6 with esmtpa (envelope-from ) id 1x61va-00000000PHN-21oW; Mon, 14 Sep 2026 11:20:38 +0300 Date: Mon, 14 Sep 2026 11:20:21 +0300 To: Sergey Bronnikov Message-ID: References: <20260826143447.3761605-1-skaplun@tarantool.org> <180487a3-c956-4d5c-8a98-b460d9522111@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <180487a3-c956-4d5c-8a98-b460d9522111@tarantool.org> X-Mailru-Src: smtp X-4EC0790: 10 X-7564579A: 646B95376F6C166E X-77F55803: 4F1203BC0FB41BD969DCC1097515C590F0E608CBBC392B789CEAEAC7E4BB0CF4182A05F53808504017212E908463E53E3DE06ABAFEAF670544650200692095BA17BC280BAE685DFC5A4E072B40C18CE0 X-7FA49CB5: FF5795518A3D127A4AD6D5ED66289B5278DA827A17800CE78BAADB77C21FF6F2EA1F7E6F0F101C67BD4B6F7A4D31EC0BCC500DACC3FED6E28638F802B75D45FF8AA50765F7900637AC83A81C8FD4AD23D82A6BABE6F325AC2E85FA5F3EDFCBAA7353EFBB55337566562BCE9B9754B08EE910AB26F823679947C1F5DD7430D82CAFE186A51600EE8D389733CBF5DBD5E913377AFFFEAFD269A417C69337E82CC2CC7F00164DA146DAFE8445B8C89999729449624AB7ADAF37F6B57BC7E64490611E7FA7ABCAF51C92176DF2183F8FC7C07E7E81EEA8A9722B8941B15DA834481F9449624AB7ADAF372E808ACE2090B5E14AD6D5ED66289B5278DA827A17800CE76631511D42670FFE2EB15956EA79C166A417C69337E82CC275ECD9A6C639B01B78DA827A17800CE73BEA542F6CAF3F99731C566533BA786AA5CC5B56E945C8DA X-C1DE0DAB: 0D63561A33F958A5664C7D9F69FEE47C5002B1117B3ED696A1138FDB466AF5251A1B8FE1FED62FE8823CB91A9FED034534781492E4B8EEADF5E532225D4D775BBDAD6C7F3747799A X-C8649E89: 1C3962B70DF3F0AD73CAD6646DEDE1918E10F71CB4DF9F96AB70F9BE574AE9C625B6776AC983F447FC0B9F89525902EE6F57B2FD27647F25E66C117BDB76D659FE80E115776770F0FBB437AC67D3543D66A3CAEBFBE5B423CFB2C18E94E20A5677D7E66C31021F02B8341EE9D5BE9A0A17BFE232DE746EB8EE79EEFF2A2EAE997E4D81D5E6368C576536EB022892E5344C41F94D744909CECFA6C6B0C050A61A8CAF69B82BA93681CD72808BE417F3B9E0E7457915DAA85F X-D57D3AED: 3ZO7eAau8CL7WIMRKs4sN3D3tLDjz0dLbV79QFUyzQ2Ujvy7cMT6pYYqY16iZVKkSc3dCLJ7zSJH7+u4VD18S7Vl4ZUrpaVfd2+vE6kuoey4m4VkSEu53w8ahmwBjZKM/YPHZyZHvz5uv+WouB9+ObcCpyrx6l7KImUglyhkEat/+ysWwi0gdhEs0JGjl6ggRWTy1haxBpVdbIX1nthFXOcIETfglQORZ0zpDET4Zrk3igikrdHlWBp6vE4vxwTrAP+qXqN/L60= X-Mailru-Sender: 689FA8AB762F73937C9FA53A4753B3135A5D4D54B7BC82AFCEBCA86B038B9CB47759E16F2A0890D0E49D44BB4BD9522A059A1ED8796F048DB274557F927329BE89D5A3BC2B10C37545BD1C3CC395C826B4A721A3011E896F X-Mras: Ok Subject: Re: [Tarantool-patches] [PATCH luajit] perf: add helper for benchmark results comparison X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Kaplun via Tarantool-patches Reply-To: Sergey Kaplun Cc: tarantool-patches@dev.tarantool.org Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" Hi, Sergey! Thanks for the review! Please consider my answers below. On 31.08.26, Sergey Bronnikov wrote: > Hello, > > Thanks for the patch! Please see my comments. > > On 8/26/26 17:34, Sergey Kaplun wrote: > > This patch adds the helper, which is similar to the Google-benchmark > > compare.py script [1]. Nevertheless, it has the following semantic > > changes: > > > > * compare.py compares the absolute time of the benchmark, which may lead > > to confusion when the time is autotuned to be sure that the benchmark > > runs the minimum required time amount. Hence, this script compares the > > items_per_second metric to check the performance difference even for > > the same absolute times of the benchmarks. > > * compare.py compares only 2 files, while this script allows comparing > > directories filled with the same benchmarks (checks intersection with > > warning of unmatched benchmarks), which allows to see the full > > statistic of the patch for the suite. > > * This helper doesn't allow running the benchmarks to compare. It > > proceeds only with the given results. > > * There is no support for U test. > > * --alpha flag allows setting the relative difference threshold from > > which results are considered as changed, instead of the p-value for > > U test. > > * The geomean of the benchmarks can be hidden by the flag > > --hide_aggregates, -i. > > * The output may be filtered to contain only changed benchmarks by the > > option --changes_only, -c. > > In fact, all these reasons seem far-fetched. compere.py is just a > standalone script; > > we can bring it as is and modify it to suit your needs, as we did with > performance tests. Of course we can bring the 2k Python script with the corresponding dependencies, but for what? This is the repository about the Lua language, and it seems hypocritical to support the Lua language if even our only team doesn't want to use it. > > The commit doesn't explain why you need to write something from scratch, > specifically in Lua. It is a small patch utility patch. I don't see a problem here. > > The patch brings the script, but it's not used anywhere. Why do we need > it in the repository? It is the script for local console use. It is used to compare local benchmark runs, as I mentioned in the description. For example, during the review of the future patches with performance improvements like for the commit [1]. > > The script reinvents argparse helpers, which we already wrote in > perf/utils/bench.lua. Why? Since it is used as a script from the command line, it is done to avoid using LUA_PATH, so I prefer not to use argparse as a separate module. > > In your commit, you describe what you're missing from the script compare.py, > > but where did all these requirements come from? If you have any plans in > mind about > > how and what's missing for performance testing, please share them. > > Otherwise, it just looks like a personal need. It is common sense about what you want from the comparison script. You may share your own thoughts and suggestions. [1]: https://github.com/LuaJIT/LuaJIT/commit/14d8a7a27dc8c626ab9e7c7e9e50b6df6def4f03 -- Best regards, Sergey Kaplun