Re: [Tarantool-patches] [PATCH luajit] perf: add helper for benchmark results comparison

[Evgeniy Temirgaleev via Tarantool-patches <tarantool-patches@dev.tarantool.org>]

[-- Attachment #1: Type: text/plain, Size: 7185 bytes --]

Hi, Sergey! Thanks for the patch!
LGTM with one suggestion.

> 
> From: Sergey Kaplun <skaplun@tarantool.org>
> To: Sergey Bronnikov <sergeyb@tarantool.org>, Evgeniy Temirgaleev <e.temirgaleev@tarantool.org
> >
> Cc: tarantool-patches@dev.tarantool.org, Sergey Kaplun <skaplun@tarantool.org
> >
> Date: Wednesday, August 26, 2026 5:35 PM +03:00
> This patch adds the helper, which is similar to the Google-benchmark
> compare.py script [1]. Nevertheless, it has the following semantic
> changes:
> 
> * compare.py compares the absolute time of the benchmark, which may lead
> to confusion when the time is autotuned to be sure that the benchmark
> runs the minimum required time amount. Hence, this script compares the
> items_per_second metric to check the performance difference even for
> the same absolute times of the benchmarks.
> * compare.py compares only 2 files, while this script allows comparing
> directories filled with the same benchmarks (checks intersection with
> warning of unmatched benchmarks), which allows to see the full
> statistic of the patch for the suite.
> * This helper doesn't allow running the benchmarks to compare. It
> proceeds only with the given results.
> * There is no support for U test.
> * --alpha flag allows setting the relative difference threshold from
> which results are considered as changed, instead of the p-value for
> U test.
> * The geomean of the benchmarks can be hidden by the flag
> --hide_aggregates, -i.
> * The output may be filtered to contain only changed benchmarks by the
> option --changes_only, -c.
> 
> For more details, see help in the script.
> 
> [1]: https://github.com/google/benchmark/blob/267a11154f14269384879dc7f6b8d25acb3684db/tools/compare.py
> 
> ---
> 
> Branch: https://github.com/tarantool/luajit/tree/skaplun/gh-noticket-perf-compare
> 
> Side note: CI is red due to the known Tarantool metrics issue.
> 
> The example of the output (comparing master with this [2] patch
> applied):
> 
> | $ luajit perf/helpers/compare.lua --alpha=0.05 -c
> ../bench/perf/output/LuaJIT-benches
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches
> | Comparing ../bench/perf/output/LuaJIT-benches/chameneos.json to
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/chameneos.json
> | Benchmark items_per_second IPS New IPS Old
> |
> -------------------------------------------------------------------------
> | chameneos +0.14 2.87M/s 2.52M/s
> | OVERALL_GEOMEAN +0.14 2.87M/s 2.52M/s
> |
> | Comparing ../bench/perf/output/LuaJIT-benches/coroutine-ring.json to
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/coroutine-ring.json
> | Benchmark items_per_second IPS New IPS Old
> |
> -------------------------------------------------------------------------
> | coroutine_ring +0.21 17.41M/s 14.38M/s
> | OVERALL_GEOMEAN +0.21 17.41M/s 14.38M/s
> |
> | Comparing ../bench/perf/output/LuaJIT-benches/euler14-bit.json to
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/euler14-bit.json
> | Benchmark items_per_second IPS New IPS Old
> |
> -------------------------------------------------------------------------
> | euler14_bit +0.16 21.00M/s 18.16M/s
> | OVERALL_GEOMEAN +0.16 21.00M/s 18.16M/s
> |
> | Comparing ../bench/perf/output/LuaJIT-benches/recursive-ack.json to
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-ack.json
> | Benchmark items_per_second IPS New IPS Old
> |
> -------------------------------------------------------------------------
> | recursive_ack +0.12 241.07M/s 214.95M/s
> | OVERALL_GEOMEAN +0.12 241.07M/s 214.95M/s
> |
> | Comparing ../bench/perf/output/LuaJIT-benches/recursive-fib.json to
> ../gc64-benchmarks-patched/perf/output/LuaJIT-benches/recursive-fib.json
> | Benchmark items_per_second IPS New IPS Old
> |
> -------------------------------------------------------------------------
> | recursive_fib +0.42 302.70M/s 212.50M/s
> | OVERALL_GEOMEAN +0.42 302.70M/s 212.50M/s
> 
> [2]: https://github.com/LuaJIT/LuaJIT/issues/1485#issue-4900862636
> 
> perf/helpers/compare.lua | 463 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 463 insertions(+)
> create mode 100644 perf/helpers/compare.lua
> 
> diff --git a/perf/helpers/compare.lua b/perf/helpers/compare.lua
> new file mode 100644
> index 00000000..2abe6256
> --- /dev/null
> +++ b/perf/helpers/compare.lua
> @@ -0,0 +1,463 @@
> +local json = require('cjson')
> +
> +local abs, exp, log, max = math.abs, math.exp, math.log, math.max
> +local find, format = string.find, string.format
> +local match, rep, sub = string.match, string.rep, string.sub
> +local table_insert, table_remove = table.insert, table.remove
> +local table_sort = table.sort
> +
> 

<snipped>

> 
> +------------ Comparison of the results. --------------------------
> +
> +-- Compare to lists and return intersection of them.
> +-- Print warning if any element is missing.
> +local function intersection(a, b, msga, msgb)
> + local intersect = {}
> + -- Scan tables, with sorted elements.
> + -- If any element missing (this element is less than the nearest
> + -- from another list), print warning.
> + table_sort(a)
> + table_sort(b)
> + local ai, bi = 1, 1
> + while ai <= #a or bi <= #b do
> + if a[ai] == b[bi] then
> + table_insert(intersect, a[ai])
> + ai = ai + 1
> + bi = bi + 1
> + elseif bi > #b or (ai < #a and a[ai] < b[bi]) then
> + warn(format(msga, a[ai]))
> + ai = ai + 1
> + else
> + assert(ai > #a or (bi < #b and a[ai] > b[bi]), 'incorrect intersection')
> 
> + warn(format(msgb, b[bi]))
> + bi = bi + 1
> + end
> + end
> + return intersect
> +end
> 

Here the multi-branch loop (Dijkstra-loop) may be used to make things more clear. In Lua it may be written with infinite loop plus multi-branch if with single break in else branch.

local ai, bi = 1, 1
while true do
if ai <= #a and bi <= #b and a[ai] == b[bi] then
table_insert(intersect, a[ai])
ai = ai + 1
bi = bi + 1
elseif ai <= #a and (bi > #b or a[ai] < b[bi]) then
warn(format(msga, a[ai]))
ai = ai + 1
elseif bi <= #b and (ai > #a or b[bi] < a[ai]) then
warn(format(msgb, b[bi]))
bi = bi + 1
else -- ai > #a and bi > #b
break
end
end

> 
> +
> +-- Calculate geomean for the given bench names.
> +local function gmean(results, list)
> + local n = #list
> + if n == 1 then
> + return results[list[1]]
> + end
> + local gmn = 0
> + -- Use Log-Transform calculation to avoid infinite values.
> + for i = 1, n do
> + gmn = gmn + log(results[list[i]])
> + end
> + gmn = gmn / n
> + return exp(gmn)
> +end
> +
> 

<snipped>

> 
> +------------ Main. -----------------------------------------------
> +
> +argparse(arg)
> +
> +local baseline_path, contender_path = arg[1], arg[2]
> +local base_isdir = isdir(baseline_path)
> +local cont_isdir = isdir(contender_path)
> +
> +if base_isdir ~= cont_isdir then
> + fatal('Baseline and contender file types should either be file or
> directory.')
> +end
> +
> +if base_isdir then
> + output_sets_results(compare_benchmarks_sets(baseline_path,
> contender_path),
> + baseline_path, contender_path
> + )
> +else
> + output_results(compare_benchmarks(baseline_path, contender_path),
> + baseline_path, contender_path
> + )
> +end
> --
> 2.55.0
> 

Best regards,
Evgeniy Temirgaleev

[-- Attachment #2: Type: text/html, Size: 9715 bytes --]