Hi, Sergey! Thanks for the patch! LGTM with one suggestion. > > From: Sergey Kaplun > To: Sergey Bronnikov , Evgeniy Temirgaleev > > Cc: tarantool-patches@dev.tarantool.org, Sergey Kaplun > > Date: Wednesday, August 26, 2026 5:35 PM +03:00 > This patch adds the helper, which is similar to the Google-benchmark > compare.py script [1]. Nevertheless, it has the following semantic > changes: > > * compare.py compares the absolute time of the benchmark, which may lead > to confusion when the time is autotuned to be sure that the benchmark > runs the minimum required time amount. Hence, this script compares the > items_per_second metric to check the performance difference even for > the same absolute times of the benchmarks. > * compare.py compares only 2 files, while this script allows comparing > directories filled with the same benchmarks (checks intersection with > warning of unmatched benchmarks), which allows to see the full > statistic of the patch for the suite. > * This helper doesn't allow running the benchmarks to compare. It > proceeds only with the given results. > * There is no support for U test. > * --alpha flag allows setting the relative difference threshold from > which results are considered as changed, instead of the p-value for > U test. > * The geomean of the benchmarks can be hidden by the flag > --hide_aggregates, -i. > * The output may be filtered to contain only changed benchmarks by the > option --changes_only, -c. > > For more details, see help in the script. > > [1]: https://github.com/google/benchmark/blob/267a11154f14269384879dc7f6b8d25acb3684db/tools/compare.py > > --- > > Branch: https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-perf-compare > > Side note: CI is red due to the known Tarantool metrics issue. > > The example of the output (comparing master with this [2] patch > applied): > > | $ luajit perf/helpers/compare.lua --alpha=0.05 -c > ../bench/perf/output/LuaJIT-benches > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches > | Comparing ../bench/perf/output/LuaJIT-benches/chameneos.json to > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/chameneos.json > | Benchmark items_per_second IPS New IPS Old > | > ------------------------------------------------------------------------- > | chameneos +0.14 2.87M/s 2.52M/s > | OVERALL_GEOMEAN +0.14 2.87M/s 2.52M/s > | > | Comparing ../bench/perf/output/LuaJIT-benches/coroutine-ring.json to > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/coroutine-ring.json > | Benchmark items_per_second IPS New IPS Old > | > ------------------------------------------------------------------------- > | coroutine_ring +0.21 17.41M/s 14.38M/s > | OVERALL_GEOMEAN +0.21 17.41M/s 14.38M/s > | > | Comparing ../bench/perf/output/LuaJIT-benches/euler14-bit.json to > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/euler14-bit.json > | Benchmark items_per_second IPS New IPS Old > | > ------------------------------------------------------------------------- > | euler14_bit +0.16 21.00M/s 18.16M/s > | OVERALL_GEOMEAN +0.16 21.00M/s 18.16M/s > | > | Comparing ../bench/perf/output/LuaJIT-benches/recursive-ack.json to > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-ack.json > | Benchmark items_per_second IPS New IPS Old > | > ------------------------------------------------------------------------- > | recursive_ack +0.12 241.07M/s 214.95M/s > | OVERALL_GEOMEAN +0.12 241.07M/s 214.95M/s > | > | Comparing ../bench/perf/output/LuaJIT-benches/recursive-fib.json to > ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-fib.json > | Benchmark items_per_second IPS New IPS Old > | > ------------------------------------------------------------------------- > | recursive_fib +0.42 302.70M/s 212.50M/s > | OVERALL_GEOMEAN +0.42 302.70M/s 212.50M/s > > [2]: https://github.com/LuaJIT/LuaJIT/issues/1485#issue-4900862636 > > perf/helpers/compare.lua | 463 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 463 insertions(+) > create mode 100644 perf/helpers/compare.lua > > diff --git a/perf/helpers/compare.lua b/perf/helpers/compare.lua > new file mode 100644 > index 00000000..2abe6256 > --- /dev/null > +++ b/perf/helpers/compare.lua > @@ -0,0 +1,463 @@ > +local json = require('cjson') > + > +local abs, exp, log, max = math.abs, math.exp, math.log, math.max > +local find, format = string.find, string.format > +local match, rep, sub = string.match, string.rep, string.sub > +local table_insert, table_remove = table.insert, table.remove > +local table_sort = table.sort > + > > > +------------ Comparison of the results. -------------------------- > + > +-- Compare to lists and return intersection of them. > +-- Print warning if any element is missing. > +local function intersection(a, b, msga, msgb) > + local intersect = {} > + -- Scan tables, with sorted elements. > + -- If any element missing (this element is less than the nearest > + -- from another list), print warning. > + table_sort(a) > + table_sort(b) > + local ai, bi = 1, 1 > + while ai <= #a or bi <= #b do > + if a[ai] == b[bi] then > + table_insert(intersect, a[ai]) > + ai = ai + 1 > + bi = bi + 1 > + elseif bi > #b or (ai < #a and a[ai] < b[bi]) then > + warn(format(msga, a[ai])) > + ai = ai + 1 > + else > + assert(ai > #a or (bi < #b and a[ai] > b[bi]), 'incorrect intersection') > > + warn(format(msgb, b[bi])) > + bi = bi + 1 > + end > + end > + return intersect > +end > Here the multi-branch loop (Dijkstra-loop) may be used to make things more clear. In Lua it may be written with infinite loop plus multi-branch if with single break in else branch. local ai, bi = 1, 1 while true do if ai <= #a and bi <= #b and a[ai] == b[bi] then table_insert(intersect, a[ai]) ai = ai + 1 bi = bi + 1 elseif ai <= #a and (bi > #b or a[ai] < b[bi]) then warn(format(msga, a[ai])) ai = ai + 1 elseif bi <= #b and (ai > #a or b[bi] < a[ai]) then warn(format(msgb, b[bi])) bi = bi + 1 else -- ai > #a and bi > #b break end end > > + > +-- Calculate geomean for the given bench names. > +local function gmean(results, list) > + local n = #list > + if n == 1 then > + return results[list[1]] > + end > + local gmn = 0 > + -- Use Log-Transform calculation to avoid infinite values. > + for i = 1, n do > + gmn = gmn + log(results[list[i]]) > + end > + gmn = gmn / n > + return exp(gmn) > +end > + > > > +------------ Main. ----------------------------------------------- > + > +argparse(arg) > + > +local baseline_path, contender_path = arg[1], arg[2] > +local base_isdir = isdir(baseline_path) > +local cont_isdir = isdir(contender_path) > + > +if base_isdir ~= cont_isdir then > + fatal('Baseline and contender file types should either be file or > directory.') > +end > + > +if base_isdir then > + output_sets_results(compare_benchmarks_sets(baseline_path, > contender_path), > + baseline_path, contender_path > + ) > +else > + output_results(compare_benchmarks(baseline_path, contender_path), > + baseline_path, contender_path > + ) > +end > -- > 2.55.0 > Best regards, Evgeniy Temirgaleev