[Tarantool-patches] [PATCH v3] gitlab-ci: save failed test results artifacts

[Tarantool-patches] [PATCH v3] gitlab-ci: save failed test results artifacts

[Alexander V. Tikhonov]

Added artifacts saver to all gitlab-ci jobs with testing.

Gitlab-ci jobs saves its results files in the following paths:

  1. base jobs for testing different features:
    - test/var/artifacts

  2. OSX jobs:
    - ${OSX_VARDIR}/artifacts

  3. pack/deploy jobs:
    - build/usr/src/*/tarantool-*/test/var/artifacts

  4. VBOX jobs (freebsd_12) on virtual host:
    - ~/tarantool/test/var/artifacts

In gitlab-ci configuration added 'after_script' section with script
which collects from different test places 'artifacts' directories
created by test-run tool. It saves 'artifacts' directories as root
path in artifacts packages. User will be able to download these
packages using gitlab-ci GUI either API.

Additionally added OSX_VARDIR environment variable to be able to
setup common path for artifacts and OSX shell scripts options.

  OSX_VARDIR: /tmp/tnt

Part of #5050
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/artifacts_rejectfiles
Issue: https://github.com/tarantool/tarantool/issues/5050

 .gitlab-ci.yml | 33 +++++++++++++++++++++++++++++++--
 .travis.mk     |  6 +++---
 2 files changed, 34 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 16528c1dc..4f4fc545c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -13,6 +13,7 @@ variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
   GIT_STRATEGY: none
   GIT_CLEAN_FLAGS: none
+  OSX_VARDIR: /tmp/tnt
 
 # 1. Git checkout strategy used from:
 #    https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout
@@ -36,6 +37,15 @@ before_script:
 
 # Jobs templates
 
+.artifacts_files_template: &artifacts_files_definition
+  artifacts:
+    expire_in: 1000 yrs
+    when: always
+    paths:
+      - "artifacts"
+  after_script:
+    - cp -r test/var/artifacts .
+
 .deploy_only_template: &deploy_only_definition
   only:
     - master
@@ -66,12 +76,19 @@ before_script:
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_files_definition
 
 .docker_test_clang8_template: &docker_test_clang8_definition
   image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-buster:latest"
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_files_definition
+
+.pack_artifacts_files_template: &pack_artifacts_files_definition
+  <<: *artifacts_files_definition
+  after_script:
+    - cp -r build/usr/src/*/tarantool-*/test/var/artifacts .
 
 .pack_template: &pack_definition
   <<: *pack_only_definition
@@ -88,6 +105,7 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} package
+  <<: *pack_artifacts_files_definition
 
 .deploy_template: &deploy_definition
   <<: *deploy_only_definition
@@ -104,16 +122,27 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} deploy
+  <<: *pack_artifacts_files_definition
 
 .osx_template: &osx_definition
   stage: test
   script:
     - ${GITLAB_MAKE} test_osx
+  after_script:
+    # Artifacts can't be used from outside the project directory, check:
+    #   https://docs.gitlab.com/ee/ci/yaml/README.html#artifactspaths
+    # "Paths are relative to the project directory ($CI_PROJECT_DIR)"
+    # "and can’t directly link outside it."
+    - cp -rf ${OSX_VARDIR}/artifacts .
+  <<: *artifacts_files_definition
 
 .vbox_template: &vbox_definition
   stage: test
   after_script:
-    - ${GITLAB_MAKE} vms_shutdown
+    - >
+      scp -r -P ${VMS_PORT} ${VMS_USER}@127.0.0.1:tarantool/test/var/artifacts . ;
+      ${GITLAB_MAKE} vms_shutdown
+  <<: *artifacts_files_definition
 
 .perf_docker_test_template: &perf_docker_test_definition
   <<: *perf_only_definition
@@ -581,7 +610,7 @@ static_build_cmake_linux:
     - ${GITLAB_MAKE} test_static_build_cmake_linux
 
 static_build_cmake_osx_15:
-  stage: test
+  <<: *osx_definition
   tags:
     - osx_15
   script:
diff --git a/.travis.mk b/.travis.mk
index d8f97ef5c..577970d8f 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -242,11 +242,11 @@ INIT_TEST_ENV_OSX=\
 		launchctl limit maxproc || : ; \
 		ulimit -u ${MAX_PROC} || : ; \
 		ulimit -u ; \
-		rm -rf /tmp/tnt
+		rm -rf ${OSX_VARDIR}
 
 test_osx_no_deps: build_osx
 	${INIT_TEST_ENV_OSX}; \
-	cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
+	cd test && ./test-run.py --vardir ${OSX_VARDIR} --force $(TEST_RUN_EXTRA_PARAMS)
 
 test_osx: deps_osx test_osx_no_deps
 
@@ -265,7 +265,7 @@ test_static_build_cmake_osx: base_deps_osx
 	cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \
 	make -j && ctest -V
 	${INIT_TEST_ENV_OSX}; \
-	cd test && ./test-run.py --vardir /tmp/tnt \
+	cd test && ./test-run.py --vardir ${OSX_VARDIR} \
 		--builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \
 		--force $(TEST_RUN_EXTRA_PARAMS)
 
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v3] gitlab-ci: save failed test results artifacts

[Alexander Turenko]

Nothing contradicts with my vision (I became a bit more far from details
now). LGTM. Several comments below (no need to re-review with me if
you'll want to make changes according to those comments).

WBR, Alexander Turenko.

>  .vbox_template: &vbox_definition
>    stage: test
>    after_script:
> -    - ${GITLAB_MAKE} vms_shutdown
> +    - >
> +      scp -r -P ${VMS_PORT} ${VMS_USER}@127.0.0.1:tarantool/test/var/artifacts . ;
> +      ${GITLAB_MAKE} vms_shutdown
> +  <<: *artifacts_files_definition

Nit: Using literal style indicator (the pipe symbol instead of the angle
bracket) would look more intuitive and does not require the semicolon.
(The folded style ('>') replaces newlines with spaces and it should be
used when this behaviour is wanted I think.)

>  static_build_cmake_osx_15:
> -  stage: test
> +  <<: *osx_definition
>    tags:
>      - osx_15
>    script:

Nit: This change is not mentioned in the commit message.

> diff --git a/.travis.mk b/.travis.mk
> index d8f97ef5c..577970d8f 100644
> --- a/.travis.mk
> +++ b/.travis.mk
> @@ -242,11 +242,11 @@ INIT_TEST_ENV_OSX=\
>  		launchctl limit maxproc || : ; \
>  		ulimit -u ${MAX_PROC} || : ; \
>  		ulimit -u ; \
> -		rm -rf /tmp/tnt
> +		rm -rf ${OSX_VARDIR}

Just in case: should not we set some default value for this variable in
case we'll run this rule outside of CI (or in CI, which is not GitLab
CI)?

Re: [Tarantool-patches] [PATCH v3] gitlab-ci: save failed test results artifacts

[Alexander V. Tikhonov]

Hi Alexander, thanks for the review, just commented your questions to
make it clear.

On Thu, Sep 24, 2020 at 06:40:31AM +0300, Alexander Turenko wrote:
> Nothing contradicts with my vision (I became a bit more far from details
> now). LGTM. Several comments below (no need to re-review with me if
> you'll want to make changes according to those comments).
> 
> WBR, Alexander Turenko.
> 
> >  .vbox_template: &vbox_definition
> >    stage: test
> >    after_script:
> > -    - ${GITLAB_MAKE} vms_shutdown
> > +    - >
> > +      scp -r -P ${VMS_PORT} ${VMS_USER}@127.0.0.1:tarantool/test/var/artifacts . ;
> > +      ${GITLAB_MAKE} vms_shutdown
> > +  <<: *artifacts_files_definition
> 
> Nit: Using literal style indicator (the pipe symbol instead of the angle
> bracket) would look more intuitive and does not require the semicolon.
> (The folded style ('>') replaces newlines with spaces and it should be
> used when this behaviour is wanted I think.)
>

I did use it after investigation and found the following, [1] - where
in the gitlab-ci issue disscussion was told to use '>' instead of '|',
and previous discussion [2], where it was described.

[1] - https://gitlab.com/gitlab-org/gitlab-runner/-/issues/166
[2] - https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-over-multiple-lines/3790497#3790497

> >  static_build_cmake_osx_15:
> > -  stage: test
> > +  <<: *osx_definition
> >    tags:
> >      - osx_15
> >    script:
> 
> Nit: This change is not mentioned in the commit message.
>

That is absolutely the same code. It was used to inherit 'after_script'
part:

.osx_template: &osx_definition
  <<: *artifacts_files_definition
  stage: test
  script:
    - ${GITLAB_MAKE} test_osx
  after_script:
    # Artifacts can't be used from outside the project directory, check:
    #   https://docs.gitlab.com/ee/ci/yaml/README.html#artifactspaths
    # "Paths are relative to the project directory ($CI_PROJECT_DIR)"
    # "and can’t directly link outside it."
    - cp -rf ${OSX_VARDIR}/artifacts .

'script' part would be overwritten from '&osx_definition':
  script:
    - ${GITLAB_MAKE} test_osx
to:
  script:
    - ${GITLAB_MAKE} test_static_build_cmake_osx


> > diff --git a/.travis.mk b/.travis.mk
> > index d8f97ef5c..577970d8f 100644
> > --- a/.travis.mk
> > +++ b/.travis.mk
> > @@ -242,11 +242,11 @@ INIT_TEST_ENV_OSX=\
> >  		launchctl limit maxproc || : ; \
> >  		ulimit -u ${MAX_PROC} || : ; \
> >  		ulimit -u ; \
> > -		rm -rf /tmp/tnt
> > +		rm -rf ${OSX_VARDIR}
> 
> Just in case: should not we set some default value for this variable in
> case we'll run this rule outside of CI (or in CI, which is not GitLab
> CI)?

Right, it' better to have, corrected .gitlab.mk:

diff --git a/.travis.mk b/.travis.mk
index 577970d8f..c09f1c4cb 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -9,6 +9,7 @@ MAX_FILES?=65534
 MAX_PROC?=2500
 OOS_SRC_PATH="/source"
 BIN_DIR=/usr/local/bin
+OSX_VARDIR?=/tmp/tnt

 CLOJURE_URL="https://download.clojure.org/install/linux-install-1.10.1.561.sh"
 LEIN_URL="https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein"

Re: [Tarantool-patches] [PATCH v3] gitlab-ci: save failed test results artifacts

[Kirill Yukhin]

Hello,

On 23 сен 09:40, Alexander V. Tikhonov wrote:
> Added artifacts saver to all gitlab-ci jobs with testing.
> 
> Gitlab-ci jobs saves its results files in the following paths:
> 
>   1. base jobs for testing different features:
>     - test/var/artifacts
> 
>   2. OSX jobs:
>     - ${OSX_VARDIR}/artifacts
> 
>   3. pack/deploy jobs:
>     - build/usr/src/*/tarantool-*/test/var/artifacts
> 
>   4. VBOX jobs (freebsd_12) on virtual host:
>     - ~/tarantool/test/var/artifacts
> 
> In gitlab-ci configuration added 'after_script' section with script
> which collects from different test places 'artifacts' directories
> created by test-run tool. It saves 'artifacts' directories as root
> path in artifacts packages. User will be able to download these
> packages using gitlab-ci GUI either API.
> 
> Additionally added OSX_VARDIR environment variable to be able to
> setup common path for artifacts and OSX shell scripts options.
> 
>   OSX_VARDIR: /tmp/tnt
> 
> Part of #5050

I've checked your patch into 1.10, 2.4, 2.5 and master.

--
Regards, Kirill Yukhin