Hi, Sergey,
thanks for the patch! See my comments.
Sergey
bash or shell is used in the last step? (shebang in setup_env.sh is /bin/sh)This patch adds the workflow to run benchmarks from various suites, aggregate their results, and send statistics to the InfluxDB to be processed later. The workflow contains a matrix to measure GC64 and non-GC64 modes with enabled/disabled JIT for x64 architecture. --- .github/actions/setup-performance/README.md | 10 ++ .github/actions/setup-performance/action.yml | 18 +++ .github/workflows/performance.yml | 110 +++++++++++++++++++ 3 files changed, 138 insertions(+) create mode 100644 .github/actions/setup-performance/README.md create mode 100644 .github/actions/setup-performance/action.yml create mode 100644 .github/workflows/performance.yml diff --git a/.github/actions/setup-performance/README.md b/.github/actions/setup-performance/README.md new file mode 100644 index 00000000..4c4bbdab --- /dev/null +++ b/.github/actions/setup-performance/README.md @@ -0,0 +1,10 @@ +# Setup performance + +Action setups the performance on Linux runners. + +## How to use Github Action from Github workflow + +Add the following code to the running steps before LuaJIT configuration: +``` +- uses: ./.github/actions/setup-performance +``` diff --git a/.github/actions/setup-performance/action.yml b/.github/actions/setup-performance/action.yml new file mode 100644 index 00000000..24d07440 --- /dev/null +++ b/.github/actions/setup-performance/action.yml @@ -0,0 +1,18 @@ +name: Setup performance +description: The Linux machine setup for running LuaJIT benchmarks +runs: + using: composite + steps: + - name: Setup CI environment (Linux) + uses: ./.github/actions/setup-linux + - name: Install dependencies for the LuaJIT benchmarks + run: | + apt -y update + apt install -y luarocks curl + shell: bash + - name: Install Lua modules + run: luarocks install lua-cjson + shell: bash + - name: Run script to setup Linux environment + run: sh ./perf/helpers/setup_env.sh + shell: bash
it is not relevant, right?diff --git a/.github/workflows/performance.yml b/.github/workflows/performance.yml new file mode 100644 index 00000000..bfb6be97 --- /dev/null +++ b/.github/workflows/performance.yml @@ -0,0 +1,110 @@ +name: Performance + +on: + push: + branches-ignore: + - '**-noperf' + - 'tarantool/release/**' + - 'upstream-**' + tags-ignore: + - '**' + schedule: + # Once a day at 03:00 to avoid clashing with runs for the + # Tarantool benchmarks at midnight. + - cron: '0 3 * * *' + +concurrency: + # An update of a developer branch cancels the previously + # scheduled workflow run for this branch. However, the default + # branch, and long-term branch (tarantool/release/2.11, + # tarantool/release/2.10, etc) workflow runs are never canceled. + #
+ # We use a trick here: define the concurrency group as 'workflow
+ # run ID' + # 'workflow run attempt' because it is a unique
+ # combination for any run. So it effectively discards grouping.
+ #
+ # XXX: we cannot use `github.sha` as a unique identifier because
+ # pushing a tag may cancel a run that works on a branch push
+ # event.
+ group: ${{ startsWith(github.ref, 'refs/heads/tarantool/')
+ && format('{0}-{1}', github.run_id, github.run_attempt)
+ || format('{0}-{1}', github.workflow, github.ref) }}
+ cancel-in-progress: true
+
+jobs:
+ performance-luajit:
+ # The 'performance' label _must_ be set only for the single
+ # runner to guarantee that results are not dependent on the
+ # machine.
+ runs-on:
+ - self-hosted
+ - Linux
+ - x86_64
+ - 'performance'
+
+ env:
+ PERF_BRANCH: ${{ github.ref_name }}
+ PERF_COMMIT: ${{ github.sha }}
+
+ strategy:
+ fail-fast: false
+ matrix:
+ GC64: [ON, OFF]
+ JOFF: [ON, OFF]
+ # Run each job sequentially.
+ max-parallel: 1
+ name: >
+ LuaJIT
+ GC64:${{ matrix.GC64 }}
+ JOFF:${{ matrix.GC64 }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: setup performance environment
+ uses: ./.github/actions/setup-performance
+ - name: configure
+ # The taskset alone will pin all the process threads
+ # into a single (random) isolated CPU, see
+ # https://bugzilla.kernel.org/show_bug.cgi?id=116701.
+ # The workaround is using realtime scheduler for the
+ # isolated task using chrt, e. g.:
+ # sudo taskset 0xef chrt 50.
+ # But this makes the process use non-standard, real-time
+ # round-robin scheduling mechanism.
+ run: >
+ cmake -S . -B ${{ env.BUILDDIR }}
+ -DCMAKE_BUILD_TYPE=RelWithDebInfo
RelWithDebInfo is -O2 (moderate optimization), Release is -O3 (high optimization).
Do we really need RelWithDebInfo? I think it deserves a comment.
+ -DLUAJIT_ENABLE_PERF=ON
+ -DLUAJIT_BENCH_INIT="taskset 0xfe chrt 50"
+ -DLUAJIT_DISABLE_JIT=${{ matrix.JOFF }}
+ -DLUAJIT_ENABLE_GC64=${{ matrix.GC64 }}
+ - name: build
+ run: cmake --build . --parallel
+ working-directory: ${{ env.BUILDDIR }}
+ - name: perf
+ run: make LuaJIT-perf
+ working-directory: ${{ env.BUILDDIR }}
+ - name: aggregate benchmark results
+ run: make LuaJIT-perf-aggregate
+ working-directory: ${{ env.BUILDDIR }}
+ - name: send statistics to InfluxDB
+ # --silent -o /dev/null: Prevent dumping any reply part
+ # in the output in case of an error.
+ # --fail: Exit with the 22 error code is status >= 400.
+ # --write-out: See the reason for the failure, if any.
+ # --retry, --retry-delay: To avoid losing the results of
+ # running after such a long job, try to retry sending the
+ # results.
+ run: >
+ curl --request POST
+ "${{ secrets.INFLUXDB_URL }}/api/v2/write?org=tarantool&bucket=luajit-performance&precision=s"
+ --write-out "%{http_code}"
+ --retry 5
+ --retry-delay 5
+ --connect-timeout 120
+ --fail --silent -o /dev/null
+ --header "Authorization: Token ${{ secrets.INFLUXDB_TOKEN }}"
+ --data-binary @./perf/output/summary.txt
+ working-directory: ${{ env.BUILDDIR }}