[Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]



- add cleanup after perf docker
  image preparation (perf_bootstrap job).
- cleanup perf docker images on execution hosts
  after perf testing (cleanup stage).
 
Closes #5003
---
 
Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
Issue:  https://github.com/tarantool/tarantool/issues/5003

 .gitlab-ci.yml | 31 ++++++++++++++++++++++++++++---
 .gitlab.mk     |  4 ++++
 2 files changed, 32 insertions(+), 3 deletions(-)
 
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 256b368..4c1fa0b 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -196,6 +196,8 @@ perf_bootstrap:
     - perf
   script:
     - ${GITLAB_MAKE} perf_prepare
+  after_script:
+    - ${GITLAB_MAKE} perf_cleanup_bootstrap
 
 # Testing part
 
@@ -255,12 +257,35 @@ perf_linkbench_ssd:
 
 # Post-testing part
 
-remove_images:
+remove_images_sh1:
   <<: *perf_only_definition
   stage: cleanup
-  when: always
   tags:
-    - perf
+    - sh1_shell
+  script:
+    - ${GITLAB_MAKE} perf_cleanup
+
+remove_images_sh2:
+  <<: *perf_only_definition
+  stage: cleanup
+  tags:
+    - sh2_shell
+  script:
+    - ${GITLAB_MAKE} perf_cleanup
+
+remove_images_sh3:
+  <<: *perf_only_definition
+  stage: cleanup
+  tags:
+    - sh3_shell
+  script:
+    - ${GITLAB_MAKE} perf_cleanup
+
+remove_images_sh9:
+  <<: *perf_only_definition
+  stage: cleanup
+  tags:
+    - sh9_shell
   script:
     - ${GITLAB_MAKE} perf_cleanup
 
diff --git a/.gitlab.mk b/.gitlab.mk
index d1d93cd..feff3f9 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -90,6 +90,10 @@ perf_prepare: perf_clone_benchs_repo
 perf_cleanup: perf_clone_benchs_repo
        make -f bench-run/targets.mk cleanup
 
+# Remove temporary performance image after bootstrap phase
+perf_cleanup_bootstrap:
+       make -f bench-run/targets.mk cleanup
+
 # #################################
 # Run tests under a virtual machine
 # #################################
--
1.8.3.1
 

[-- Attachment #2: Type: text/html, Size: 2961 bytes --]

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Sergey Bronnikov]

Hi, Oleg!

generally LGTM, but see my question regarding commit message.

On 03:36 Wed 20 May , Oleg Piskunov wrote:
> 
> 
> - add cleanup after perf docker
>   image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
>   after perf testing (cleanup stage).
>  
> Closes #5003

There is a best practice to explain 'what' and 'why' in a commit message
body. It is unclear why we should cleanup these images and cannot reuse
existed images.

<snipped>

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Alexander V. Tikhonov]

Hi Oleg, thanks for the patch, in general LGTM, just single nit comments:

1) You have 2 targets making the same:

  # Remove temporary performance image from the test host
  perf_cleanup: perf_clone_benchs_repo
          make -f bench-run/targets.mk cleanup

  # Remove temporary performance image after bootstrap phase
  perf_cleanup_bootstrap:
          make -f bench-run/targets.mk cleanup

may be its better not to duplicate the code and use the single code:

  # Remove temporary performance image after bootstrap phase
  perf_cleanup_bootstrap:
          make -f bench-run/targets.mk cleanup

  # Remove temporary performance image from the test host
  perf_cleanup: perf_clone_benchs_repo perf_cleanup_bootstrap

2) in gitlab-ci YAMl file for common parts of code better to use
templates, like:

  .perf_cleanup_definition: &perf_cleanup_definition
    <<: *perf_only_definition
    stage: cleanup
    script:
      - ${GITLAB_MAKE} perf_cleanup

in this way jobs will be like:

  remove_images_sh2:
    <<: *perf_cleanup_definition
    tags:
      - sh2_shell

On Wed, May 20, 2020 at 03:36:37AM +0300, Oleg Piskunov wrote:
> 
> 
> - add cleanup after perf docker
>   image preparation (perf_bootstrap job).
> - cleanup perf docker images on execution hosts
>   after perf testing (cleanup stage).
>  
> Closes #5003
> ---
>  
> Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
> Issue:  https://github.com/tarantool/tarantool/issues/5003
> 
>  .gitlab-ci.yml | 31 ++++++++++++++++++++++++++++---
>  .gitlab.mk     |  4 ++++
>  2 files changed, 32 insertions(+), 3 deletions(-)
>  
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 256b368..4c1fa0b 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -196,6 +196,8 @@ perf_bootstrap:
>      - perf
>    script:
>      - ${GITLAB_MAKE} perf_prepare
> +  after_script:
> +    - ${GITLAB_MAKE} perf_cleanup_bootstrap
>  
>  # Testing part
>  
> @@ -255,12 +257,35 @@ perf_linkbench_ssd:
>  
>  # Post-testing part
>  
> -remove_images:
> +remove_images_sh1:
>    <<: *perf_only_definition
>    stage: cleanup
> -  when: always
>    tags:
> -    - perf
> +    - sh1_shell
> +  script:
> +    - ${GITLAB_MAKE} perf_cleanup
> +
> +remove_images_sh2:
> +  <<: *perf_only_definition
> +  stage: cleanup
> +  tags:
> +    - sh2_shell
> +  script:
> +    - ${GITLAB_MAKE} perf_cleanup
> +
> +remove_images_sh3:
> +  <<: *perf_only_definition
> +  stage: cleanup
> +  tags:
> +    - sh3_shell
> +  script:
> +    - ${GITLAB_MAKE} perf_cleanup
> +
> +remove_images_sh9:
> +  <<: *perf_only_definition
> +  stage: cleanup
> +  tags:
> +    - sh9_shell
>    script:
>      - ${GITLAB_MAKE} perf_cleanup
>  
> diff --git a/.gitlab.mk b/.gitlab.mk
> index d1d93cd..feff3f9 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -90,6 +90,10 @@ perf_prepare: perf_clone_benchs_repo
>  perf_cleanup: perf_clone_benchs_repo
>         make -f bench-run/targets.mk cleanup
>  
> +# Remove temporary performance image after bootstrap phase
> +perf_cleanup_bootstrap:
> +       make -f bench-run/targets.mk cleanup
> +
>  # #################################
>  # Run tests under a virtual machine
>  # #################################
> --
> 1.8.3.1
>  

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]


Sergey, thanks for review. 
 
The reason is that each temporary docker image is about 5 GB and we don’t have space to keep it localy.
I will update commit message.
  
>Среда, 20 мая 2020, 11:31 +03:00 от Sergey Bronnikov <sergeyb@tarantool.org>:
> 
>Hi, Oleg!
>
>generally LGTM, but see my question regarding commit message.
>
>On 03:36 Wed 20 May , Oleg Piskunov wrote:
>>
>>
>> - add cleanup after perf docker
>>   image preparation (perf_bootstrap job).
>> - cleanup perf docker images on execution hosts
>>   after perf testing (cleanup stage).
>>  
>> Closes #5003
>There is a best practice to explain 'what' and 'why' in a commit message
>body. It is unclear why we should cleanup these images and cannot reuse
>existed images.
>
><snipped>
 
 
--
Oleg Piskunov
 

[-- Attachment #2: Type: text/html, Size: 1401 bytes --]

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup temporary perf images

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 3708 bytes --]


Sasha, thanks for review. 
 
I will modify patch as you suggested.
  
>Среда, 20 мая 2020, 19:44 +03:00 от Alexander V. Tikhonov <avtikhon@tarantool.org>:
> 
>Hi Oleg, thanks for the patch, in general LGTM, just single nit comments:
>
>1) You have 2 targets making the same:
>
>  # Remove temporary performance image from the test host
>  perf_cleanup: perf_clone_benchs_repo
>          make -f bench-run/targets.mk cleanup
>
>  # Remove temporary performance image after bootstrap phase
>  perf_cleanup_bootstrap:
>          make -f bench-run/targets.mk cleanup
>
>may be its better not to duplicate the code and use the single code:
>
>  # Remove temporary performance image after bootstrap phase
>  perf_cleanup_bootstrap:
>          make -f bench-run/targets.mk cleanup
>
>  # Remove temporary performance image from the test host
>  perf_cleanup: perf_clone_benchs_repo perf_cleanup_bootstrap
>
>2) in gitlab-ci YAMl file for common parts of code better to use
>templates, like:
>
>  .perf_cleanup_definition: &perf_cleanup_definition
>   <<: *perf_only_definition
>    stage: cleanup
>   script:
>     - ${GITLAB_MAKE} perf_cleanup
>
>in this way jobs will be like:
>
>  remove_images_sh2:
>   <<: *perf_cleanup_definition
>    tags:
>    - sh2_shell
>
>On Wed, May 20, 2020 at 03:36:37AM +0300, Oleg Piskunov wrote:
>>
>>
>> - add cleanup after perf docker
>>   image preparation (perf_bootstrap job).
>> - cleanup perf docker images on execution hosts
>>   after perf testing (cleanup stage).
>>  
>> Closes #5003
>> ---
>>  
>> Github:  https://github.com/tarantool/tarantool/tree/opiskunov/gh-5003-perf-images-cleanup
>> Issue:  https://github.com/tarantool/tarantool/issues/5003
>>
>>  .gitlab-ci.yml | 31 ++++++++++++++++++++++++++++---
>>  .gitlab.mk     |  4 ++++
>>  2 files changed, 32 insertions(+), 3 deletions(-)
>>  
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 256b368..4c1fa0b 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -196,6 +196,8 @@ perf_bootstrap:
>>      - perf
>>    script:
>>      - ${GITLAB_MAKE} perf_prepare
>> +  after_script:
>> +    - ${GITLAB_MAKE} perf_cleanup_bootstrap
>>  
>>  # Testing part
>>  
>> @@ -255,12 +257,35 @@ perf_linkbench_ssd:
>>  
>>  # Post-testing part
>>  
>> -remove_images:
>> +remove_images_sh1:
>>    <<: *perf_only_definition
>>    stage: cleanup
>> -  when: always
>>    tags:
>> -    - perf
>> +    - sh1_shell
>> +  script:
>> +    - ${GITLAB_MAKE} perf_cleanup
>> +
>> +remove_images_sh2:
>> +  <<: *perf_only_definition
>> +  stage: cleanup
>> +  tags:
>> +    - sh2_shell
>> +  script:
>> +    - ${GITLAB_MAKE} perf_cleanup
>> +
>> +remove_images_sh3:
>> +  <<: *perf_only_definition
>> +  stage: cleanup
>> +  tags:
>> +    - sh3_shell
>> +  script:
>> +    - ${GITLAB_MAKE} perf_cleanup
>> +
>> +remove_images_sh9:
>> +  <<: *perf_only_definition
>> +  stage: cleanup
>> +  tags:
>> +    - sh9_shell
>>    script:
>>      - ${GITLAB_MAKE} perf_cleanup
>>  
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index d1d93cd..feff3f9 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -90,6 +90,10 @@ perf_prepare: perf_clone_benchs_repo
>>  perf_cleanup: perf_clone_benchs_repo
>>         make -f bench-run/targets.mk cleanup
>>  
>> +# Remove temporary performance image after bootstrap phase
>> +perf_cleanup_bootstrap:
>> +       make -f bench-run/targets.mk cleanup
>> +
>>  # #################################
>>  # Run tests under a virtual machine
>>  # #################################
>> --
>> 1.8.3.1
>>   
 
 
--
Oleg Piskunov
 

[-- Attachment #2: Type: text/html, Size: 5569 bytes --]