[Tarantool-patches] [PATCH v1] gitlab-ci: save failed test results files

[Tarantool-patches] [PATCH v1] gitlab-ci: save failed test results files

[Alexander V. Tikhonov]

Added artifacts saver to all gitlab-ci jobs with testing. It
saves all found 'test/*/*.reject' files from the failed tests.
After each finished failed gitlab-ci job, user would be able to
download these files from it using gitlab-ci GUI.

Additionally added OSX_VARDIR environment variable to be able to
setup common path for artifacts and OSX shell scripts options.

Needed for #5050
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/artifacts_rejectfiles
Issue: https://github.com/tarantool/tarantool/issues/5050

 .gitlab-ci.yml | 21 ++++++++++++++++++++-
 .travis.mk     |  4 ++--
 test-run       |  2 +-
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 580825005..48f0d0098 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,7 @@ variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
   GIT_STRATEGY: none
   GIT_CLEAN_FLAGS: none
+  OSX_VARDIR: /tmp/tnt
 
 # 1. Git checkout strategy used from:
 #    https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout
@@ -35,6 +36,12 @@ before_script:
 
 # Jobs templates
 
+.artifacts_reject_files_template: &artifacts_reject_files_definition
+  artifacts:
+    when: always
+    paths:
+      - "test/*/*.reject"
+
 .deploy_only_template: &deploy_only_definition
   only:
     - master
@@ -62,12 +69,14 @@ before_script:
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_reject_files_definition
 
 .docker_test_clang8_template: &docker_test_clang8_definition
   image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-buster:latest"
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_reject_files_definition
 
 .pack_template: &pack_definition
   <<: *pack_only_definition
@@ -84,6 +93,7 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} package
+  <<: *artifacts_reject_files_definition
 
 .deploy_template: &deploy_definition
   <<: *deploy_only_definition
@@ -100,16 +110,23 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} deploy
+  <<: *artifacts_reject_files_definition
 
 .osx_template: &osx_definition
   stage: test
   script:
     - ${GITLAB_MAKE} test_osx
+  variables:
+    VARDIR: ${OSX_VARDIR}
+  after_script:
+    - rm -rf test/var ; mkdir test/var && cp -rfp ${OSX_VARDIR}/*.reject test/var/.
+  <<: *artifacts_reject_files_definition
 
 .vbox_template: &vbox_definition
   stage: test
   after_script:
     - ${GITLAB_MAKE} vms_shutdown
+  <<: *artifacts_reject_files_definition
 
 .perf_docker_test_template: &perf_docker_test_definition
   <<: *perf_only_definition
@@ -193,9 +210,10 @@ osx_15_release:
 osx_15_release_lto:
   tags:
     - osx_15
+  <<: *osx_definition
   variables:
+    VARDIR: ${OSX_VARDIR}
     EXTRA_ENV: 'export CMAKE_EXTRA_PARAMS=-DENABLE_LTO=ON ;'
-  <<: *osx_definition
 
 freebsd_12_release:
   <<: *vbox_definition
@@ -540,3 +558,4 @@ static_docker_build:
     - deploy_test
   script:
     - ${GITLAB_MAKE} test_static_docker_build
+  <<: *artifacts_reject_files_definition
diff --git a/.travis.mk b/.travis.mk
index efc05cf05..fe71a4416 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -210,8 +210,8 @@ test_osx_no_deps: build_osx
 		launchctl limit maxproc || : ; \
 		ulimit -u ${MAX_PROC} || : ; \
 		ulimit -u ; \
-		rm -rf /tmp/tnt ; \
-		cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
+		rm -rf ${VARDIR} ; \
+		cd test && ./test-run.py --vardir ${VARDIR} --force $(TEST_RUN_EXTRA_PARAMS)
 
 test_osx: deps_osx test_osx_no_deps
 
diff --git a/test-run b/test-run
index d92032233..0e8da635b 160000
--- a/test-run
+++ b/test-run
@@ -1 +1 @@
-Subproject commit d92032233e10db8934ae5fed17fe3f3ae8293645
+Subproject commit 0e8da635b76416e2ae0704c285e2bf42279f53ae
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: save failed test results files

[Alexander Turenko]

> Needed for #5050

As I stated in [2] there is nothing about CI artefacts in #5050. There
is [3] about this.

> +.artifacts_reject_files_template: &artifacts_reject_files_definition
> +  artifacts:
> +    when: always
> +    paths:
> +      - "test/*/*.reject"

Are we need it when a job passes? Maybe save artefacts only for failed
jobs?

I would set `expire_in` into some large value (it is not possible to
store an artefact infinitely, gitlab docs lie), because we not always
start to investigate a flaky fail within 4 weeks. See [1] (it suggests
`expire_in: 1000 yrs`).

I also commented [2] and it seems the proposed changes will affect this
PR.

> +  variables:
> +    VARDIR: ${OSX_VARDIR}
> +  after_script:
> +    - rm -rf test/var ; mkdir test/var && cp -rfp ${OSX_VARDIR}/*.reject test/var/.
> <...>
> -		rm -rf /tmp/tnt ; \
> -		cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
> +		rm -rf ${VARDIR} ; \
> +		cd test && ./test-run.py --vardir ${VARDIR} --force $(TEST_RUN_EXTRA_PARAMS)

I don't understand what is going on here, but I hope that we can drop it
after PR #218 (see [2]) changes.

> diff --git a/test-run b/test-run
> index d92032233..0e8da635b 160000
> --- a/test-run
> +++ b/test-run
> @@ -1 +1 @@
> -Subproject commit d92032233e10db8934ae5fed17fe3f3ae8293645
> +Subproject commit 0e8da635b76416e2ae0704c285e2bf42279f53ae

I'll update the test-run submodule separately as I always do.

[1]: https://gitlab.com/gitlab-org/gitlab/-/issues/22761#note_251722017
[2]: https://github.com/tarantool/test-run/pull/218
[3]: https://github.com/tarantool/test-run/issues/90