<HTML><BODY><div class="cl-n6om7bluza">Hi, Sergey! Thanks for the patch!<br>LGTM with one suggestion.<div class="mail-quote-collapse"><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px"><span>From: Sergey Kaplun <<a href="mailto:skaplun@tarantool.org">skaplun@tarantool.org</a>><br>To: Sergey Bronnikov <<a href="mailto:sergeyb@tarantool.org">sergeyb@tarantool.org</a>>, Evgeniy Temirgaleev <<a href="mailto:e.temirgaleev@tarantool.org">e.temirgaleev@tarantool.org</a>><br>Cc: tarantool-patches@dev.tarantool.org, Sergey Kaplun <<a href="mailto:skaplun@tarantool.org">skaplun@tarantool.org</a>><br>Date: Wednesday, August 26, 2026 5:35 PM +03:00</span><br> <div><div id=""><div class="cl-nqjrp597g1"><div class="js-helper_mr_css_attr js-readmsg-msg_mr_css_attr"><div id="style_17877549221272458378_mr_css_attr"><div id="style_17877549221272458378_BODY_mr_css_attr">This patch adds the helper, which is similar to the Google-benchmark<br>compare.py script [1]. Nevertheless, it has the following semantic<br>changes:<br><br>* compare.py compares the absolute time of the benchmark, which may lead<br>to confusion when the time is autotuned to be sure that the benchmark<br>runs the minimum required time amount. Hence, this script compares the<br>items_per_second metric to check the performance difference even for<br>the same absolute times of the benchmarks.<br>* compare.py compares only 2 files, while this script allows comparing<br>directories filled with the same benchmarks (checks intersection with<br>warning of unmatched benchmarks), which allows to see the full<br>statistic of the patch for the suite.<br>* This helper doesn't allow running the benchmarks to compare. It<br>proceeds only with the given results.<br>* There is no support for U test.<br>* --alpha flag allows setting the relative difference threshold from<br>which results are considered as changed, instead of the p-value for<br>U test.<br>* The geomean of the benchmarks can be hidden by the flag<br>--hide_aggregates, -i.<br>* The output may be filtered to contain only changed benchmarks by the<br>option --changes_only, -c.<br><br>For more details, see help in the script.<br><br>[1]: <a href="https://github.com/google/benchmark/blob/267a11154f14269384879dc7f6b8d25acb3684db/tools/compare.py">https://github.com/google/benchmark/blob/267a11154f14269384879dc7f6b8d25acb3684db/tools/compare.py</a><br>---<br><br>Branch: <a href="https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-perf-compare">https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-perf-compare</a><br>Side note: CI is red due to the known Tarantool metrics issue.<br><br>The example of the output (comparing master with this [2] patch<br>applied):<br><br>| $ luajit perf/helpers/compare.lua --alpha=0.05 -c ../bench/perf/output/LuaJIT-benches ../gc64-benchmarks-patched/perf/output/LuaJIT-benches<br>| Comparing ../bench/perf/output/LuaJIT-benches/chameneos.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/chameneos.json<br>| Benchmark items_per_second IPS New IPS Old<br>| -------------------------------------------------------------------------<br>| chameneos +0.14 2.87M/s 2.52M/s<br>| OVERALL_GEOMEAN +0.14 2.87M/s 2.52M/s<br>|<br>| Comparing ../bench/perf/output/LuaJIT-benches/coroutine-ring.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/coroutine-ring.json<br>| Benchmark items_per_second IPS New IPS Old<br>| -------------------------------------------------------------------------<br>| coroutine_ring +0.21 17.41M/s 14.38M/s<br>| OVERALL_GEOMEAN +0.21 17.41M/s 14.38M/s<br>|<br>| Comparing ../bench/perf/output/LuaJIT-benches/euler14-bit.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/euler14-bit.json<br>| Benchmark items_per_second IPS New IPS Old<br>| -------------------------------------------------------------------------<br>| euler14_bit +0.16 21.00M/s 18.16M/s<br>| OVERALL_GEOMEAN +0.16 21.00M/s 18.16M/s<br>|<br>| Comparing ../bench/perf/output/LuaJIT-benches/recursive-ack.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-ack.json<br>| Benchmark items_per_second IPS New IPS Old<br>| -------------------------------------------------------------------------<br>| recursive_ack +0.12 241.07M/s 214.95M/s<br>| OVERALL_GEOMEAN +0.12 241.07M/s 214.95M/s<br>|<br>| Comparing ../bench/perf/output/LuaJIT-benches/recursive-fib.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-fib.json<br>| Benchmark items_per_second IPS New IPS Old<br>| -------------------------------------------------------------------------<br>| recursive_fib +0.42 302.70M/s 212.50M/s<br>| OVERALL_GEOMEAN +0.42 302.70M/s 212.50M/s<br><br>[2]: <a href="https://github.com/LuaJIT/LuaJIT/issues/1485#issue-4900862636">https://github.com/LuaJIT/LuaJIT/issues/1485#issue-4900862636</a><br><br>perf/helpers/compare.lua | 463 +++++++++++++++++++++++++++++++++++++++<br>1 file changed, 463 insertions(+)<br>create mode 100644 perf/helpers/compare.lua<br><br>diff --git a/perf/helpers/compare.lua b/perf/helpers/compare.lua<br>new file mode 100644<br>index 00000000..2abe6256<br>--- /dev/null<br>+++ b/perf/helpers/compare.lua<br>@@ -0,0 +1,463 @@<br>+local json = require('cjson')<br>+<br>+local abs, exp, log, max = math.abs, math.exp, math.log, math.max<br>+local find, format = string.find, string.format<br>+local match, rep, sub = string.match, string.rep, string.sub<br>+local table_insert, table_remove = table.insert, table.remove<br>+local table_sort = table.sort<br>+</div></div></div></div></div></div></blockquote></div></div><div><snipped></div><div class="cl-n6om7bluza"><div class="mail-quote-collapse"><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px"><div><div><div class="cl-nqjrp597g1"><div class="js-helper_mr_css_attr js-readmsg-msg_mr_css_attr"><div><div>+------------ Comparison of the results. --------------------------<br>+<br>+-- Compare to lists and return intersection of them.<br>+-- Print warning if any element is missing.<br>+local function intersection(a, b, msga, msgb)<br>+ local intersect = {}<br>+ -- Scan tables, with sorted elements.<br>+ -- If any element missing (this element is less than the nearest<br>+ -- from another list), print warning.<br>+ table_sort(a)<br>+ table_sort(b)<br>+ local ai, bi = 1, 1<br>+ while ai <= #a or bi <= #b do<br>+ if a[ai] == b[bi] then<br>+ table_insert(intersect, a[ai])<br>+ ai = ai + 1<br>+ bi = bi + 1<br>+ elseif bi > #b or (ai < #a and a[ai] < b[bi]) then<br>+ warn(format(msga, a[ai]))<br>+ ai = ai + 1<br>+ else<br>+ assert(ai > #a or (bi < #b and a[ai] > b[bi]), 'incorrect intersection')<br>+ warn(format(msgb, b[bi]))<br>+ bi = bi + 1<br>+ end<br>+ end<br>+ return intersect<br>+end</div></div></div></div></div></div></blockquote></div></div><div>Here the multi-branch loop (Dijkstra-loop) may be used to make things more clear. In Lua it may be written with infinite loop plus multi-branch if with single break in else branch.</div><div> </div><div><div><div> local ai, bi = 1, 1</div><div> while true do</div><div> if ai <= #a and bi <= #b and a[ai] == b[bi] then</div><div> table_insert(intersect, a[ai])</div><div> ai = ai + 1</div><div> bi = bi + 1</div><div> elseif ai <= #a and (bi > #b or a[ai] < b[bi]) then</div><div> warn(format(msga, a[ai]))</div><div> ai = ai + 1</div><div> elseif bi <= #b and (ai > #a or b[bi] < a[ai]) then</div><div> warn(format(msgb, b[bi]))</div><div> bi = bi + 1</div><div> else -- ai > #a and bi > #b</div><div> break</div><div> end</div><div> end</div></div></div><div class="cl-n6om7bluza"><div class="mail-quote-collapse"><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px"><div><div><div class="cl-nqjrp597g1"><div class="js-helper_mr_css_attr js-readmsg-msg_mr_css_attr"><div><div>+<br>+-- Calculate geomean for the given bench names.<br>+local function gmean(results, list)<br>+ local n = #list<br>+ if n == 1 then<br>+ return results[list[1]]<br>+ end<br>+ local gmn = 0<br>+ -- Use Log-Transform calculation to avoid infinite values.<br>+ for i = 1, n do<br>+ gmn = gmn + log(results[list[i]])<br>+ end<br>+ gmn = gmn / n<br>+ return exp(gmn)<br>+end<br>+</div></div></div></div></div></div></blockquote></div></div><div><snipped></div><div class="cl-n6om7bluza"><div class="mail-quote-collapse"><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px"><div><div><div class="cl-nqjrp597g1"><div class="js-helper_mr_css_attr js-readmsg-msg_mr_css_attr"><div><div>+------------ Main. -----------------------------------------------<br>+<br>+argparse(arg)<br>+<br>+local baseline_path, contender_path = arg[1], arg[2]<br>+local base_isdir = isdir(baseline_path)<br>+local cont_isdir = isdir(contender_path)<br>+<br>+if base_isdir ~= cont_isdir then<br>+ fatal('Baseline and contender file types should either be file or directory.')<br>+end<br>+<br>+if base_isdir then<br>+ output_sets_results(compare_benchmarks_sets(baseline_path, contender_path),<br>+ baseline_path, contender_path<br>+ )<br>+else<br>+ output_results(compare_benchmarks(baseline_path, contender_path),<br>+ baseline_path, contender_path<br>+ )<br>+end<br>--<br>2.55.0</div></div></div></div></div></div></blockquote></div></div><div><div>Best regards,</div><div>Evgeniy Temirgaleev</div></div></BODY></HTML>