[Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images

[Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]


Each temporary docker perf image is about 5 GB
and we don’t have space to keep them localy on servers.
 
- add cleanup after perf docker
  image preparation (perf_bootstrap job).
- cleanup perf docker images on execution hosts
  after perf testing (cleanup stage).
 
Closes #5003
---
 
Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
Issue:  https://github.com/tarantool/tarantool/issues/5003
 
 .gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
 .gitlab.mk     |  7 +++++--
 2 files changed, 31 insertions(+), 9 deletions(-)
 
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 256b368..5f488ea 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -103,6 +103,12 @@ variables:
   script:
     - ${GITLAB_MAKE} perf_run
 
+.perf_cleanup_definition: &perf_cleanup_definition
+   <<: *perf_only_definition
+   stage: cleanup
+   script:
+     - ${GITLAB_MAKE} perf_cleanup
+
 # Tests
 
 release:
@@ -196,6 +202,8 @@ perf_bootstrap:
     - perf
   script:
     - ${GITLAB_MAKE} perf_prepare
+  after_script:
+    - ${GITLAB_MAKE} perf_cleanup_image
 
 # Testing part
 
@@ -255,14 +263,25 @@ perf_linkbench_ssd:
 
 # Post-testing part
 
-remove_images:
-  <<: *perf_only_definition
-  stage: cleanup
-  when: always
+remove_images_sh1:
+  <<: *perf_cleanup_definition
   tags:
-    - perf
-  script:
-    - ${GITLAB_MAKE} perf_cleanup
+    - sh1_shell
+
+remove_images_sh2:
+  <<: *perf_cleanup_definition
+  tags:
+    - sh2_shell
+
+remove_images_sh3:
+  <<: *perf_cleanup_definition
+  tags:
+    - sh3_shell
+
+remove_images_sh9:
+  <<: *perf_cleanup_definition
+  tags:
+    - sh9_shell
 
 # Packages and sources
 
diff --git a/.gitlab.mk b/.gitlab.mk
index d1d93cd..f2a9b77 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -86,10 +86,13 @@ perf_clone_benchs_repo:
 perf_prepare: perf_clone_benchs_repo
        make -f bench-run/targets.mk prepare
 
-# Remove temporary performance image from the test host
-perf_cleanup: perf_clone_benchs_repo
+# Remove temporary performance image
+perf_cleanup_image:
        make -f bench-run/targets.mk cleanup
 
+# Remove temporary performance image from the test host
+perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
+
 # #################################
 # Run tests under a virtual machine
 # #################################
--
1.8.3.1
 

[-- Attachment #2: Type: text/html, Size: 3518 bytes --]

Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images

[Sergey Bronnikov]

Hi, Oleg

LGTM

On 14:13 Thu 21 May , Oleg Piskunov wrote:
> 
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>  
> - add cleanup after perf docker
>   image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
>   after perf testing (cleanup stage).
>  
> Closes #5003
> ---
>  
> Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue:  https://github.com/tarantool/tarantool/issues/5003
>  
>  .gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
>  .gitlab.mk     |  7 +++++--
>  2 files changed, 31 insertions(+), 9 deletions(-)
>  
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 256b368..5f488ea 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -103,6 +103,12 @@ variables:
>    script:
>      - ${GITLAB_MAKE} perf_run
>  
> +.perf_cleanup_definition: &perf_cleanup_definition
> +   <<: *perf_only_definition
> +   stage: cleanup
> +   script:
> +     - ${GITLAB_MAKE} perf_cleanup
> +
>  # Tests
>  
>  release:
> @@ -196,6 +202,8 @@ perf_bootstrap:
>      - perf
>    script:
>      - ${GITLAB_MAKE} perf_prepare
> +  after_script:
> +    - ${GITLAB_MAKE} perf_cleanup_image
>  
>  # Testing part
>  
> @@ -255,14 +263,25 @@ perf_linkbench_ssd:
>  
>  # Post-testing part
>  
> -remove_images:
> -  <<: *perf_only_definition
> -  stage: cleanup
> -  when: always
> +remove_images_sh1:
> +  <<: *perf_cleanup_definition
>    tags:
> -    - perf
> -  script:
> -    - ${GITLAB_MAKE} perf_cleanup
> +    - sh1_shell
> +
> +remove_images_sh2:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh2_shell
> +
> +remove_images_sh3:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh3_shell
> +
> +remove_images_sh9:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh9_shell
>  
>  # Packages and sources
>  
> diff --git a/.gitlab.mk b/.gitlab.mk
> index d1d93cd..f2a9b77 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -86,10 +86,13 @@ perf_clone_benchs_repo:
>  perf_prepare: perf_clone_benchs_repo
>         make -f bench-run/targets.mk prepare
>  
> -# Remove temporary performance image from the test host
> -perf_cleanup: perf_clone_benchs_repo
> +# Remove temporary performance image
> +perf_cleanup_image:
>         make -f bench-run/targets.mk cleanup
>  
> +# Remove temporary performance image from the test host
> +perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
> +
>  # #################################
>  # Run tests under a virtual machine
>  # #################################
> --
> 1.8.3.1
>  

-- 
sergeyb@

Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images

[Alexander V. Tikhonov]

Hi Oleg, thank you for the patch, LGTM.

On Thu, May 21, 2020 at 02:13:50PM +0300, Oleg Piskunov wrote:
> 
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>  
> - add cleanup after perf docker
>   image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
>   after perf testing (cleanup stage).
>  
> Closes #5003
> ---
>  
> Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue:  https://github.com/tarantool/tarantool/issues/5003
>  
>  .gitlab-ci.yml | 33 ++++++++++++++++++++++++++-------
>  .gitlab.mk     |  7 +++++--
>  2 files changed, 31 insertions(+), 9 deletions(-)
>  
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 256b368..5f488ea 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -103,6 +103,12 @@ variables:
>    script:
>      - ${GITLAB_MAKE} perf_run
>  
> +.perf_cleanup_definition: &perf_cleanup_definition
> +   <<: *perf_only_definition
> +   stage: cleanup
> +   script:
> +     - ${GITLAB_MAKE} perf_cleanup
> +
>  # Tests
>  
>  release:
> @@ -196,6 +202,8 @@ perf_bootstrap:
>      - perf
>    script:
>      - ${GITLAB_MAKE} perf_prepare
> +  after_script:
> +    - ${GITLAB_MAKE} perf_cleanup_image
>  
>  # Testing part
>  
> @@ -255,14 +263,25 @@ perf_linkbench_ssd:
>  
>  # Post-testing part
>  
> -remove_images:
> -  <<: *perf_only_definition
> -  stage: cleanup
> -  when: always
> +remove_images_sh1:
> +  <<: *perf_cleanup_definition
>    tags:
> -    - perf
> -  script:
> -    - ${GITLAB_MAKE} perf_cleanup
> +    - sh1_shell
> +
> +remove_images_sh2:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh2_shell
> +
> +remove_images_sh3:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh3_shell
> +
> +remove_images_sh9:
> +  <<: *perf_cleanup_definition
> +  tags:
> +    - sh9_shell
>  
>  # Packages and sources
>  
> diff --git a/.gitlab.mk b/.gitlab.mk
> index d1d93cd..f2a9b77 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -86,10 +86,13 @@ perf_clone_benchs_repo:
>  perf_prepare: perf_clone_benchs_repo
>         make -f bench-run/targets.mk prepare
>  
> -# Remove temporary performance image from the test host
> -perf_cleanup: perf_clone_benchs_repo
> +# Remove temporary performance image
> +perf_cleanup_image:
>         make -f bench-run/targets.mk cleanup
>  
> +# Remove temporary performance image from the test host
> +perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
> +
>  # #################################
>  # Run tests under a virtual machine
>  # #################################
> --
> 1.8.3.1
>  

Re: [Tarantool-patches] [PATCH v2] gitlab-ci: cleanup temporary perf images

[Kirill Yukhin]

Hello,

On 21 май 14:13, Oleg Piskunov wrote:
> 
> Each temporary docker perf image is about 5 GB
> and we don’t have space to keep them localy on servers.
>  
> - add cleanup after perf docker
>   image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
>   after perf testing (cleanup stage).
>  
> Closes #5003
> ---
>  
> Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue:  https://github.com/tarantool/tarantool/issues/5003

I've checked your patch into 1.10, 2.3, 2.4 and master.

--
Regards, Kirill Yukhin