[Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Alexander V. Tikhonov]

Enabled Tarantool performance testing on Gitlab-CI
for release/master branches and "*-perf" named branches.
For this purpose 'perf' and 'cleanup' stages were added
into Gitlab-CI pipeline.

Performance testing support next benchmarks:
- cbench
- linkbench
- nosqlbench (hash and tree Tarantool run modes)
- sysbench
- tpcc
- ycsb (hash and tree Tarantool run modes)

Benchmarks use scripts from repository:
  http://gitlab.com/tarantool/bench-run

Perfomance testing uses docker images, built
with docker files from bench-run repository:
- perf/ubuntu-bionic:perf_master
    parent image with benchmarks only
- perf_tmp/ubuntu-bionic:perf_<commit_SHA>
    child images used for testing Tarantool sources
---

Changes v4:
- moved the rest of Docker infrastructure for perf testing
  into the bench-run repository
- renamed calls to performance prepare/run/cleanup stages

Changes v3:
- updated commit message
- merged all make targets for perf testing into single
- set to use variables in gitlab-ci for running perf testings

Changes v2:
- moved performance variables from global setup
  to performance template
- updated commit message
- added more comments to sources

 .gitlab-ci.yml | 100 +++++++++++++++++++++++++++++++++++++++++++++++++
 .gitlab.mk     |  16 ++++++++
 2 files changed, 116 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 727b7c94e..1bc53d0fb 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,7 @@
 stages:
   - test
+  - perf
+  - cleanup
 
 variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
@@ -26,6 +28,14 @@ variables:
     - merge_requests
     - /^.*-full-ci$/
 
+.perf_only_template: &perf_only_definition
+  only:
+    - master
+    - /^.*-perf$/
+  variables: &perf_vars_definition
+    IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
+    IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
+
 .docker_test_template: &docker_test_definition
   image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
   stage: test
@@ -77,6 +87,15 @@ variables:
   after_script:
     - ${GITLAB_MAKE} vms_shutdown
 
+.perf_docker_test_template: &perf_docker_test_definition
+  <<: *perf_only_definition
+  image: ${IMAGE_PERF_BUILT}
+  stage: perf
+  tags:
+    - docker_perf
+  script:
+    - ${GITLAB_MAKE} perf_run
+
 # Tests
 
 release:
@@ -168,6 +187,87 @@ freebsd_12_release:
   script:
     - ${GITLAB_MAKE} vms_test_freebsd
 
+# ####
+# Perf
+# ####
+
+# Pre-testing part
+
+perf_bootstrap:
+  <<: *perf_only_definition
+  stage: test
+  tags:
+    - perf
+  script:
+    - ${GITLAB_MAKE} perf_prepare
+
+# Testing part
+
+perf_sysbench:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'sysbench'
+
+perf_tpcc:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'tpcc'
+
+perf_ycsb_hash:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'ycsb'
+    ARG: 'hash'
+
+perf_ycsb_tree:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'ycsb'
+    ARG: 'tree'
+
+perf_nosqlbench_hash:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'nosqlbench'
+    ARG: 'hash'
+
+perf_nosqlbench_tree:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'nosqlbench'
+    ARG: 'tree'
+
+perf_cbench:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'cbench'
+
+perf_linkbench_ssd:
+  <<: *perf_docker_test_definition
+  tags:
+    - docker_perf_ssd
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'linkbench'
+
+# Post-testing part
+
+remove_images:
+  <<: *perf_only_definition
+  stage: cleanup
+  when: always
+  tags:
+    - perf
+  script:
+    - ${GITLAB_MAKE} perf_cleanup
+
 # Packs
 
 centos_6:
diff --git a/.gitlab.mk b/.gitlab.mk
index 24b18b7f2..bc37ed7aa 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -78,6 +78,15 @@ docker_bootstrap:
 	docker push ${DEBIAN_STRETCH_IMAGE}:latest
 	docker push ${DEBIAN_BUSTER_IMAGE}:latest
 
+# Build images for performance testing
+perf_prepare:
+	git clone https://github.com/tarantool/bench-run.git
+	make -f bench-run/targets.mk prepare
+
+# Remove temporary performance image from the test host
+perf_cleanup:
+	make -f bench-run/targets.mk cleanup
+
 # #################################
 # Run tests under a virtual machine
 # #################################
@@ -126,3 +135,10 @@ deploy: package
 
 static_build:
 	docker build --network=host --build-arg RUN_TESTS="${RUN_TESTS}" -f Dockerfile.staticbuild .
+
+# ###################
+# Performance testing
+# ###################
+
+perf_run:
+	/opt/bench-run/benchs/${BENCH}/run.sh ${ARG}
-- 
2.17.1

[Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Alexander V. Tikhonov]

Enabled Tarantool performance testing on Gitlab-CI
for release/master branches and "*-perf" named branches.
For this purpose 'perf' and 'cleanup' stages were added
into Gitlab-CI pipeline.

Performance testing support next benchmarks:
- cbench
- linkbench
- nosqlbench (hash and tree Tarantool run modes)
- sysbench
- tpcc
- ycsb (hash and tree Tarantool run modes)

Benchmarks use scripts from repository:
  http://gitlab.com/tarantool/bench-run

Perfomance testing uses docker images, built
with docker files from bench-run repository:
- perf/ubuntu-bionic:perf_master
    parent image with benchmarks only
- perf_tmp/ubuntu-bionic:perf_<commit_SHA>
    child images used for testing Tarantool sources
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf

Changes v4:
- moved the rest of Docker infrastructure for perf testing
  into the bench-run repository
- renamed calls to performance prepare/run/cleanup stages

Changes v3:
- updated commit message
- merged all make targets for perf testing into single
- set to use variables in gitlab-ci for running perf testings

Changes v2:
- moved performance variables from global setup
  to performance template
- updated commit message
- added more comments to sources

 .gitlab-ci.yml | 100 +++++++++++++++++++++++++++++++++++++++++++++++++
 .gitlab.mk     |  19 ++++++++++
 2 files changed, 119 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 727b7c94e..1bc53d0fb 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,7 @@
 stages:
   - test
+  - perf
+  - cleanup
 
 variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
@@ -26,6 +28,14 @@ variables:
     - merge_requests
     - /^.*-full-ci$/
 
+.perf_only_template: &perf_only_definition
+  only:
+    - master
+    - /^.*-perf$/
+  variables: &perf_vars_definition
+    IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
+    IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
+
 .docker_test_template: &docker_test_definition
   image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
   stage: test
@@ -77,6 +87,15 @@ variables:
   after_script:
     - ${GITLAB_MAKE} vms_shutdown
 
+.perf_docker_test_template: &perf_docker_test_definition
+  <<: *perf_only_definition
+  image: ${IMAGE_PERF_BUILT}
+  stage: perf
+  tags:
+    - docker_perf
+  script:
+    - ${GITLAB_MAKE} perf_run
+
 # Tests
 
 release:
@@ -168,6 +187,87 @@ freebsd_12_release:
   script:
     - ${GITLAB_MAKE} vms_test_freebsd
 
+# ####
+# Perf
+# ####
+
+# Pre-testing part
+
+perf_bootstrap:
+  <<: *perf_only_definition
+  stage: test
+  tags:
+    - perf
+  script:
+    - ${GITLAB_MAKE} perf_prepare
+
+# Testing part
+
+perf_sysbench:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'sysbench'
+
+perf_tpcc:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'tpcc'
+
+perf_ycsb_hash:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'ycsb'
+    ARG: 'hash'
+
+perf_ycsb_tree:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'ycsb'
+    ARG: 'tree'
+
+perf_nosqlbench_hash:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'nosqlbench'
+    ARG: 'hash'
+
+perf_nosqlbench_tree:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'nosqlbench'
+    ARG: 'tree'
+
+perf_cbench:
+  <<: *perf_docker_test_definition
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'cbench'
+
+perf_linkbench_ssd:
+  <<: *perf_docker_test_definition
+  tags:
+    - docker_perf_ssd
+  variables:
+    <<: *perf_vars_definition
+    BENCH: 'linkbench'
+
+# Post-testing part
+
+remove_images:
+  <<: *perf_only_definition
+  stage: cleanup
+  when: always
+  tags:
+    - perf
+  script:
+    - ${GITLAB_MAKE} perf_cleanup
+
 # Packs
 
 centos_6:
diff --git a/.gitlab.mk b/.gitlab.mk
index 24b18b7f2..b39c5c651 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -78,6 +78,18 @@ docker_bootstrap:
 	docker push ${DEBIAN_STRETCH_IMAGE}:latest
 	docker push ${DEBIAN_BUSTER_IMAGE}:latest
 
+# Clone the benchmarks repository for performance testing
+perf_clone_benchs_repo:
+	git clone https://github.com/tarantool/bench-run.git
+
+# Build images for performance testing
+perf_prepare: perf_clone_benchs_repo
+	make -f bench-run/targets.mk prepare
+
+# Remove temporary performance image from the test host
+perf_cleanup: perf_clone_benchs_repo
+	make -f bench-run/targets.mk cleanup
+
 # #################################
 # Run tests under a virtual machine
 # #################################
@@ -126,3 +138,10 @@ deploy: package
 
 static_build:
 	docker build --network=host --build-arg RUN_TESTS="${RUN_TESTS}" -f Dockerfile.staticbuild .
+
+# ###################
+# Performance testing
+# ###################
+
+perf_run:
+	/opt/bench-run/benchs/${BENCH}/run.sh ${ARG}
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Alexander Turenko]

I still think that details might be better, but okay, I see: you need
some base now to proceed further. Don't want to block it anymore.

I commented the patch below, but didn't perform any changes except a bit
reworded commit message.

Pushed to master. CCed Kirill.

Don't sure how it should look at other release branches:

- Whether something need to be changed for 2.3/2.2?
  - perf_only_template should have "2.3" / "2.2" branch instead of
    master, that I understood.
  - Should IMAGE_PERF be tagged as "perf_2.3" / "perf_2.2" instead of
    "perf_master"? Are benchmarks should be adjusted for those versions
    and should this lead to such separation base images (or will be
    handled at runtime with bench-run scripts?).
- Whether something need to be changed for 1.10?
  - At least SQL benchmarks will not work. Should it be handled here or
    they will be skipped on bench-run side?
  - Same question re IMAGE_PERF as above.

Let's elaborate those questions. After this we can push it downward.

WBR, Alexander Turenko.

> Implement perf testing at gitlab-ci

Changed to: 'gitlab-ci: enable performance testing'.

>
> Enabled Tarantool performance testing on Gitlab-CI
> for release/master branches and "*-perf" named branches.
> For this purpose 'perf' and 'cleanup' stages were added
> into Gitlab-CI pipeline.
> 
> Performance testing support next benchmarks:
> - cbench
> - linkbench
> - nosqlbench (hash and tree Tarantool run modes)
> - sysbench
> - tpcc
> - ycsb (hash and tree Tarantool run modes)
> 
> Benchmarks use scripts from repository:
>   http://gitlab.com/tarantool/bench-run

Dead link. Changed gitlab.com to github.com.

> 
> Perfomance testing uses docker images, built

Fixed typo: 'Perfomance'.

> with docker files from bench-run repository:
> - perf/ubuntu-bionic:perf_master
>     parent image with benchmarks only
> - perf_tmp/ubuntu-bionic:perf_<commit_SHA>
>     child images used for testing Tarantool sources

Formatted a bit (to fit 72 symbols, but not much less).

> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf

> +.perf_only_template: &perf_only_definition
> +  only:
> +    - master
> +    - /^.*-perf$/
> +  variables: &perf_vars_definition
> +    IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
> +    IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
> +

The resulting bench-run API looks strage for me:

* It expects that a caller will set CI_REGISTRY, CI_REGISTRY_USER,
  CI_REGISTRY_PASSWORD environment variables, which come from GitLab-CI,
  but can be set manually.
* However it does not use CI_REGISTRY, CI_PROJECT_PATH,
  CI_COMMIT_SHORT_SHA to choose images name on its own, but expect
  IMAGE_PERF and IMAGE_PERF_BUILT from a caller.
* All those variables have prefix CI_*, not, say, BENCH_RUN_*.

> +# Pre-testing part
> +
> +perf_bootstrap:
> +  <<: *perf_only_definition
> +  stage: test
> +  tags:
> +    - perf
> +  script:
> +    - ${GITLAB_MAKE} perf_prepare

There is no reason to use two terms for the same thing: bootstrap and
prepare.

Also I don't see a reason to extract such one-two-liners into a
gitlab.mk.

> +# Post-testing part
> +
> +remove_images:
> +  <<: *perf_only_definition
> +  stage: cleanup
> +  when: always
> +  tags:
> +    - perf
> +  script:
> +    - ${GITLAB_MAKE} perf_cleanup
> +

Same as above: there is no reason to name it both 'remove_images' and
'perf_cleanup'.

Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Alexander Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]

Alexander, thanks a lot for a deep review, I've answered the questions below.


>Среда, 19 февраля 2020, 0:41 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>I still think that details might be better, but okay, I see: you need
>some base now to proceed further. Don't want to block it anymore. 
Right, thanks a lot.
>
>
>I commented the patch below, but didn't perform any changes except a bit
>reworded commit message.
>
>Pushed to master. CCed Kirill.
>
>Don't sure how it should look at other release branches:
>
>- Whether something need to be changed for 2.3/2.2?
>  - perf_only_template should have "2.3" / "2.2" branch instead of
>    master, that I understood. 
Right, that is the only change.
>
>  - Should IMAGE_PERF be tagged as "perf_2.3" / "perf_2.2" instead of
>    "perf_master"? Are benchmarks should be adjusted for those versions
>    and should this lead to such separation base images (or will be
>    handled at runtime with bench-run scripts?). 
No, there is no any need to have separate IMAGE_PERF images, due to it
has only built benchmarks w/o depends to Tarantool sources.
>
>- Whether something need to be changed for 1.10?
>  - At least SQL benchmarks will not work. Should it be handled here or
>    they will be skipped on bench-run side?
>  - Same question re IMAGE_PERF as above. 
Right, it needs special Dockerfile w/o SQL benchmarks, I already have
separate 'avtikhon/gitlab-ci_1.10-perf' branch for it, I'll update it with changes
from the current commit.
>
>
>Let's elaborate those questions. After this we can push it downward.
>
>WBR, Alexander Turenko.
>
>> Implement perf testing at gitlab-ci
>
>Changed to: 'gitlab-ci: enable performance testing'. 
Ok.
>
>
>>
>> Enabled Tarantool performance testing on Gitlab-CI
>> for release/master branches and "*-perf" named branches.
>> For this purpose 'perf' and 'cleanup' stages were added
>> into Gitlab-CI pipeline.
>> 
>> Performance testing support next benchmarks:
>> - cbench
>> - linkbench
>> - nosqlbench (hash and tree Tarantool run modes)
>> - sysbench
>> - tpcc
>> - ycsb (hash and tree Tarantool run modes)
>> 
>> Benchmarks use scripts from repository:
>>  http://gitlab.com/tarantool/bench-run
>
>Dead link. Changed gitlab.com to github.com. 
Right, thanks.
>
>
>> 
>> Perfomance testing uses docker images, built
>
>Fixed typo: 'Perfomance'.
Right, thanks.
>
>> with docker files from bench-run repository:
>> - perf/ubuntu-bionic:perf_master
>>     parent image with benchmarks only
>> - perf_tmp/ubuntu-bionic:perf_<commit_SHA>
>>     child images used for testing Tarantool sources
>
>Formatted a bit (to fit 72 symbols, but not much less). 
Ok.
>
>
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/gitlab-ci-perf
>
>> +.perf_only_template: &perf_only_definition
>> +  only:
>> +    - master
>> +    - /^.*-perf$/
>> +  variables: &perf_vars_definition
>> +    IMAGE_PERF: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf/ubuntu-bionic:perf_master"
>> +    IMAGE_PERF_BUILT: "${CI_REGISTRY}/${CI_PROJECT_PATH}/perf_tmp/ubuntu-bionic:perf_${CI_COMMIT_SHORT_SHA}"
>> +
>
>The resulting bench-run API looks strage for me:
>
>* It expects that a caller will set CI_REGISTRY, CI_REGISTRY_USER,
>  CI_REGISTRY_PASSWORD environment variables, which come from GitLab-CI,
>  but can be set manually. 
These variables are in use for docker registry login at bench-run scripts,
these variables recreating each job run by gitlab-ci.
>
>* However it does not use CI_REGISTRY, CI_PROJECT_PATH,
>  CI_COMMIT_SHORT_SHA to choose images name on its own, but expect
>  IMAGE_PERF and IMAGE_PERF_BUILT from a caller. 
Gitlab-ci initiates in .gitlab-ci.yml jobs running under docker images, so right
here the performance image IMAGE_PERF_BUILT must be set, IMAGE_PERF
is the part of the image IMAGE_PERF_BUILT and it's setup better to have at the
same place as IMAGE_PERF image.
>
>* All those variables have prefix CI_*, not, say, BENCH_RUN_*. 
CI_* are gitlab-ci variables and visible to the bench-run make targets - no need
to setup additional variables.
>
>
>> +# Pre-testing part
>> +
>> +perf_bootstrap:
>> +  <<: *perf_only_definition
>> +  stage: test
>> +  tags:
>> +    - perf
>> +  script:
>> +    - ${GITLAB_MAKE} perf_prepare
>
>There is no reason to use two terms for the same thing: bootstrap and
>prepare. 
I'm ok with your suggestion, let's do it on the next commit iteration.
>
>
>Also I don't see a reason to extract such one-two-liners into a
>gitlab.mk.
Previously we decided not to use .gitlab-ci.yml for code and to use it from
standalone make files, anyway I'm Ok with your suggestion, let's discuss
it a bit.
>
>> +# Post-testing part
>> +
>> +remove_images:
>> +  <<: *perf_only_definition
>> +  stage: cleanup
>> +  when: always
>> +  tags:
>> +    - perf
>> +  script:
>> +    - ${GITLAB_MAKE} perf_cleanup
>> +
>
>Same as above: there is no reason to name it both 'remove_images' and
>'perf_cleanup'.
Ok, the same answer as above.

-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 9052 bytes --]

Re: [Tarantool-patches] [PATCH v4] Implement perf testing at gitlab-ci

[Alexander Turenko]

> >Don't sure how it should look at other release branches:
> >
> >- Whether something need to be changed for 2.3/2.2?
> >  - perf_only_template should have "2.3" / "2.2" branch instead of
> >    master, that I understood. 
> Right, that is the only change.

Pushed to 2.3 and 2.2.

> Right, it needs special Dockerfile w/o SQL benchmarks, I already have
> separate 'avtikhon/gitlab-ci_1.10-perf' branch for it, I'll update it with changes
> from the current commit.

Pushed to 1.10.