From: Sergey Kaplun <skaplun@tarantool.org>
To: Sergey Bronnikov <sergeyb@tarantool.org>, Evgeniy Temirgaleev <e.temirgaleev@tarantool.org>
Cc: tarantool-patches@dev.tarantool.org, Sergey Kaplun <skaplun@tarantool.org>
Date: Wednesday, August 26, 2026 5:35 PM +03:00
 

This patch adds the helper, which is similar to the Google-benchmark
compare.py script [1]. Nevertheless, it has the following semantic
changes:

* compare.py compares the absolute time of the benchmark, which may lead
to confusion when the time is autotuned to be sure that the benchmark
runs the minimum required time amount. Hence, this script compares the
items_per_second metric to check the performance difference even for
the same absolute times of the benchmarks.
* compare.py compares only 2 files, while this script allows comparing
directories filled with the same benchmarks (checks intersection with
warning of unmatched benchmarks), which allows to see the full
statistic of the patch for the suite.
* This helper doesn't allow running the benchmarks to compare. It
proceeds only with the given results.
* There is no support for U test.
* --alpha flag allows setting the relative difference threshold from
which results are considered as changed, instead of the p-value for
U test.
* The geomean of the benchmarks can be hidden by the flag
--hide_aggregates, -i.
* The output may be filtered to contain only changed benchmarks by the
option --changes_only, -c.

For more details, see help in the script.

[1]: https://github.com/google/benchmark/blob/267a11154f14269384879dc7f6b8d25acb3684db/tools/compare.py
---

Branch: https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-perf-compare
Side note: CI is red due to the known Tarantool metrics issue.

The example of the output (comparing master with this [2] patch
applied):

| $ luajit perf/helpers/compare.lua --alpha=0.05 -c ../bench/perf/output/LuaJIT-benches ../gc64-benchmarks-patched/perf/output/LuaJIT-benches
| Comparing ../bench/perf/output/LuaJIT-benches/chameneos.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/chameneos.json
| Benchmark items_per_second IPS New IPS Old
| -------------------------------------------------------------------------
| chameneos +0.14 2.87M/s 2.52M/s
| OVERALL_GEOMEAN +0.14 2.87M/s 2.52M/s
|
| Comparing ../bench/perf/output/LuaJIT-benches/coroutine-ring.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/coroutine-ring.json
| Benchmark items_per_second IPS New IPS Old
| -------------------------------------------------------------------------
| coroutine_ring +0.21 17.41M/s 14.38M/s
| OVERALL_GEOMEAN +0.21 17.41M/s 14.38M/s
|
| Comparing ../bench/perf/output/LuaJIT-benches/euler14-bit.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/euler14-bit.json
| Benchmark items_per_second IPS New IPS Old
| -------------------------------------------------------------------------
| euler14_bit +0.16 21.00M/s 18.16M/s
| OVERALL_GEOMEAN +0.16 21.00M/s 18.16M/s
|
| Comparing ../bench/perf/output/LuaJIT-benches/recursive-ack.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-ack.json
| Benchmark items_per_second IPS New IPS Old
| -------------------------------------------------------------------------
| recursive_ack +0.12 241.07M/s 214.95M/s
| OVERALL_GEOMEAN +0.12 241.07M/s 214.95M/s
|
| Comparing ../bench/perf/output/LuaJIT-benches/recursive-fib.json to ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-fib.json
| Benchmark items_per_second IPS New IPS Old
| -------------------------------------------------------------------------
| recursive_fib +0.42 302.70M/s 212.50M/s
| OVERALL_GEOMEAN +0.42 302.70M/s 212.50M/s

[2]: https://github.com/LuaJIT/LuaJIT/issues/1485#issue-4900862636

perf/helpers/compare.lua | 463 +++++++++++++++++++++++++++++++++++++++
1 file changed, 463 insertions(+)
create mode 100644 perf/helpers/compare.lua

diff --git a/perf/helpers/compare.lua b/perf/helpers/compare.lua
new file mode 100644
index 00000000..2abe6256
--- /dev/null
+++ b/perf/helpers/compare.lua
@@ -0,0 +1,463 @@
+local json = require('cjson')
+
+local abs, exp, log, max = math.abs, math.exp, math.log, math.max
+local find, format = string.find, string.format
+local match, rep, sub = string.match, string.rep, string.sub
+local table_insert, table_remove = table.insert, table.remove
+local table_sort = table.sort
+

+------------ Comparison of the results. --------------------------
+
+-- Compare to lists and return intersection of them.
+-- Print warning if any element is missing.
+local function intersection(a, b, msga, msgb)
+ local intersect = {}
+ -- Scan tables, with sorted elements.
+ -- If any element missing (this element is less than the nearest
+ -- from another list), print warning.
+ table_sort(a)
+ table_sort(b)
+ local ai, bi = 1, 1
+ while ai <= #a or bi <= #b do
+ if a[ai] == b[bi] then
+ table_insert(intersect, a[ai])
+ ai = ai + 1
+ bi = bi + 1
+ elseif bi > #b or (ai < #a and a[ai] < b[bi]) then
+ warn(format(msga, a[ai]))
+ ai = ai + 1
+ else
+ assert(ai > #a or (bi < #b and a[ai] > b[bi]), 'incorrect intersection')
+ warn(format(msgb, b[bi]))
+ bi = bi + 1
+ end
+ end
+ return intersect
+end

+
+-- Calculate geomean for the given bench names.
+local function gmean(results, list)
+ local n = #list
+ if n == 1 then
+ return results[list[1]]
+ end
+ local gmn = 0
+ -- Use Log-Transform calculation to avoid infinite values.
+ for i = 1, n do
+ gmn = gmn + log(results[list[i]])
+ end
+ gmn = gmn / n
+ return exp(gmn)
+end
+

+------------ Main. -----------------------------------------------
+
+argparse(arg)
+
+local baseline_path, contender_path = arg[1], arg[2]
+local base_isdir = isdir(baseline_path)
+local cont_isdir = isdir(contender_path)
+
+if base_isdir ~= cont_isdir then
+ fatal('Baseline and contender file types should either be file or directory.')
+end
+
+if base_isdir then
+ output_sets_results(compare_benchmarks_sets(baseline_path, contender_path),
+ baseline_path, contender_path
+ )
+else
+ output_results(compare_benchmarks(baseline_path, contender_path),
+ baseline_path, contender_path
+ )
+end
--
2.55.0