* [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images
@ 2020-05-21 11:13 Oleg Piskunov
2020-05-21 11:43 ` Sergey Bronnikov
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Oleg Piskunov @ 2020-05-21 11:13 UTC (permalink / raw)
To: tarantool-patches; +Cc: Alexander Turenko
[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]
Each temporary docker perf image is about 5 GB
and we don’t have space to keep them localy on servers.
- add cleanup after perf docker
image preparation (perf_bootstrap job).
- cleanup perf docker images on execution hosts
after perf testing (cleanup stage).
Closes #5003
---
Github: https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
Issue: https://github.com/tarantool/tarantool/issues/5003
.gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
.gitlab.mk | 7 +++++--
2 files changed, 31 insertions(+), 9 deletions(-)
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 256b368..5f488ea 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -103,6 +103,12 @@ variables:
script:
- ${GITLAB_MAKE} perf_run
+.perf_cleanup_definition: &perf_cleanup_definition
+ <<: *perf_only_definition
+ stage: cleanup
+ script:
+ - ${GITLAB_MAKE} perf_cleanup
+
# Tests
release:
@@ -196,6 +202,8 @@ perf_bootstrap:
- perf
script:
- ${GITLAB_MAKE} perf_prepare
+ after_script:
+ - ${GITLAB_MAKE} perf_cleanup_image
# Testing part
@@ -255,14 +263,25 @@ perf_linkbench_ssd:
# Post-testing part
-remove_images:
- <<: *perf_only_definition
- stage: cleanup
- when: always
+remove_images_sh1:
+ <<: *perf_cleanup_definition
tags:
- - perf
- script:
- - ${GITLAB_MAKE} perf_cleanup
+ - sh1_shell
+
+remove_images_sh2:
+ <<: *perf_cleanup_definition
+ tags:
+ - sh2_shell
+
+remove_images_sh3:
+ <<: *perf_cleanup_definition
+ tags:
+ - sh3_shell
+
+remove_images_sh9:
+ <<: *perf_cleanup_definition
+ tags:
+ - sh9_shell
# Packages and sources
diff --git a/.gitlab.mk b/.gitlab.mk
index d1d93cd..f2a9b77 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -86,10 +86,13 @@ perf_clone_benchs_repo:
perf_prepare: perf_clone_benchs_repo
make -f bench-run/targets.mk prepare
-# Remove temporary performance image from the test host
-perf_cleanup: perf_clone_benchs_repo
+# Remove temporary performance image
+perf_cleanup_image:
make -f bench-run/targets.mk cleanup
+# Remove temporary performance image from the test host
+perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
+
# #################################
# Run tests under a virtual machine
# #################################
--
1.8.3.1
[-- Attachment #2: Type: text/html, Size: 3518 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images
2020-05-21 11:13 [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images Oleg Piskunov
@ 2020-05-21 11:43 ` Sergey Bronnikov
2020-05-21 19:35 ` Alexander V. Tikhonov
2020-05-28 7:40 ` Kirill Yukhin
2 siblings, 0 replies; 4+ messages in thread
From: Sergey Bronnikov @ 2020-05-21 11:43 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches, Alexander Turenko
Hi, Oleg
LGTM
On 14:13 Thu 21 May , Oleg Piskunov wrote:
>
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>
> - add cleanup after perf docker
> image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
> after perf testing (cleanup stage).
>
> Closes #5003
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue: https://github.com/tarantool/tarantool/issues/5003
>
> .gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
> .gitlab.mk | 7 +++++--
> 2 files changed, 31 insertions(+), 9 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 256b368..5f488ea 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -103,6 +103,12 @@ variables:
> script:
> - ${GITLAB_MAKE} perf_run
>
> +.perf_cleanup_definition: &perf_cleanup_definition
> + <<: *perf_only_definition
> + stage: cleanup
> + script:
> + - ${GITLAB_MAKE} perf_cleanup
> +
> # Tests
>
> release:
> @@ -196,6 +202,8 @@ perf_bootstrap:
> - perf
> script:
> - ${GITLAB_MAKE} perf_prepare
> + after_script:
> + - ${GITLAB_MAKE} perf_cleanup_image
>
> # Testing part
>
> @@ -255,14 +263,25 @@ perf_linkbench_ssd:
>
> # Post-testing part
>
> -remove_images:
> - <<: *perf_only_definition
> - stage: cleanup
> - when: always
> +remove_images_sh1:
> + <<: *perf_cleanup_definition
> tags:
> - - perf
> - script:
> - - ${GITLAB_MAKE} perf_cleanup
> + - sh1_shell
> +
> +remove_images_sh2:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh2_shell
> +
> +remove_images_sh3:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh3_shell
> +
> +remove_images_sh9:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh9_shell
>
> # Packages and sources
>
> diff --git a/.gitlab.mk b/.gitlab.mk
> index d1d93cd..f2a9b77 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -86,10 +86,13 @@ perf_clone_benchs_repo:
> perf_prepare: perf_clone_benchs_repo
> make -f bench-run/targets.mk prepare
>
> -# Remove temporary performance image from the test host
> -perf_cleanup: perf_clone_benchs_repo
> +# Remove temporary performance image
> +perf_cleanup_image:
> make -f bench-run/targets.mk cleanup
>
> +# Remove temporary performance image from the test host
> +perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
> +
> # #################################
> # Run tests under a virtual machine
> # #################################
> --
> 1.8.3.1
>
--
sergeyb@
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images
2020-05-21 11:13 [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images Oleg Piskunov
2020-05-21 11:43 ` Sergey Bronnikov
@ 2020-05-21 19:35 ` Alexander V. Tikhonov
2020-05-28 7:40 ` Kirill Yukhin
2 siblings, 0 replies; 4+ messages in thread
From: Alexander V. Tikhonov @ 2020-05-21 19:35 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches
Hi Oleg, thank you for the patch, LGTM.
On Thu, May 21, 2020 at 02:13:50PM +0300, Oleg Piskunov wrote:
>
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>
> - add cleanup after perf docker
> image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
> after perf testing (cleanup stage).
>
> Closes #5003
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue: https://github.com/tarantool/tarantool/issues/5003
>
> .gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
> .gitlab.mk | 7 +++++--
> 2 files changed, 31 insertions(+), 9 deletions(-)
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 256b368..5f488ea 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -103,6 +103,12 @@ variables:
> script:
> - ${GITLAB_MAKE} perf_run
>
> +.perf_cleanup_definition: &perf_cleanup_definition
> + <<: *perf_only_definition
> + stage: cleanup
> + script:
> + - ${GITLAB_MAKE} perf_cleanup
> +
> # Tests
>
> release:
> @@ -196,6 +202,8 @@ perf_bootstrap:
> - perf
> script:
> - ${GITLAB_MAKE} perf_prepare
> + after_script:
> + - ${GITLAB_MAKE} perf_cleanup_image
>
> # Testing part
>
> @@ -255,14 +263,25 @@ perf_linkbench_ssd:
>
> # Post-testing part
>
> -remove_images:
> - <<: *perf_only_definition
> - stage: cleanup
> - when: always
> +remove_images_sh1:
> + <<: *perf_cleanup_definition
> tags:
> - - perf
> - script:
> - - ${GITLAB_MAKE} perf_cleanup
> + - sh1_shell
> +
> +remove_images_sh2:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh2_shell
> +
> +remove_images_sh3:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh3_shell
> +
> +remove_images_sh9:
> + <<: *perf_cleanup_definition
> + tags:
> + - sh9_shell
>
> # Packages and sources
>
> diff --git a/.gitlab.mk b/.gitlab.mk
> index d1d93cd..f2a9b77 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -86,10 +86,13 @@ perf_clone_benchs_repo:
> perf_prepare: perf_clone_benchs_repo
> make -f bench-run/targets.mk prepare
>
> -# Remove temporary performance image from the test host
> -perf_cleanup: perf_clone_benchs_repo
> +# Remove temporary performance image
> +perf_cleanup_image:
> make -f bench-run/targets.mk cleanup
>
> +# Remove temporary performance image from the test host
> +perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
> +
> # #################################
> # Run tests under a virtual machine
> # #################################
> --
> 1.8.3.1
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images
2020-05-21 11:13 [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images Oleg Piskunov
2020-05-21 11:43 ` Sergey Bronnikov
2020-05-21 19:35 ` Alexander V. Tikhonov
@ 2020-05-28 7:40 ` Kirill Yukhin
2 siblings, 0 replies; 4+ messages in thread
From: Kirill Yukhin @ 2020-05-28 7:40 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches, Alexander Turenko
Hello,
On 21 май 14:13, Oleg Piskunov wrote:
>
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>
> - add cleanup after perf docker
> image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
> after perf testing (cleanup stage).
>
> Closes #5003
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue: https://github.com/tarantool/tarantool/issues/5003
I've checked your patch into 1.10, 2.3, 2.4 and master.
--
Regards, Kirill Yukhin
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-05-28 7:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-21 11:13 [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images Oleg Piskunov
2020-05-21 11:43 ` Sergey Bronnikov
2020-05-21 19:35 ` Alexander V. Tikhonov
2020-05-28 7:40 ` Kirill Yukhin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox