[Tarantool-patches] [PATCH v2 0/2] gitlab-ci: save failed test results files

[Tarantool-patches] [PATCH v2 0/2] gitlab-ci: save failed test results files

[Alexander V. Tikhonov]

Added artifacts saver to all gitlab-ci jobs with testing.
Gitlab-ci artifacts saves all found *.reject files in format:

  test/<suite>/*.reject

Also it saves logs files collected by test-run in artifacts
directory in format:

  test/var/artifacts

After each finished failed gitlab-ci job, user would be able to
download these files from it using gitlab-ci GUI.

Closes #5050

Github: https://github.com/tarantool/tarantool/tree/avtikhon/artifacts_rejectfiles
Issue: https://github.com/tarantool/tarantool/issues/5050
Test-run issue: https://github.com/tarantool/test-run/pull/218
Test-run issue: https://github.com/tarantool/test-run/issues/90

Alexander V. Tikhonov (2):
  gitlab-ci: save failed test results files
  gitlab-ci: set opensuse jobs to test group

 .gitlab-ci.yml | 56 ++++++++++++++++++++++++++++++++++++++++++++------
 .travis.mk     |  6 +++---
 2 files changed, 53 insertions(+), 9 deletions(-)

-- 
2.17.1

[Tarantool-patches] [PATCH v2 1/2] gitlab-ci: save failed test results files

[Alexander V. Tikhonov]

Added artifacts saver to all gitlab-ci jobs with testing.

Gitlab-ci jobs saves its results files in the following paths:

  1. base jobs for testing different features:
    - test/<suite>/*.reject
    - test/var/artifacts

  2. OSX jobs:
    - test/<suite>/*.reject
    - ${OSX_VARDIR}/artifacts/*

  3. pack/deploy jobs:
    - build/usr/src/*/tarantool-*/test/*/*.reject
    - build/usr/src/*/tarantool-*/test/var/artifacts

  4. VBOX jobs (freebsd_12) on virtual host:
    - ~/tarantool/test/*/*.reject
    - ~/tarantool/test/var/artifacts

Due to gitlab-ci jobs have different places of testings in patch
added 'after_script' instructions which collects it to the common
place known by gitlab-ci artifactory patterns. It will provide
gitlab-ci to have the same format of artifacts paths in its packages.

Gitlab-ci artifacts saves all found *.reject files in format:

  test/<suite>/*.reject

Also it saves logs files collected by test-run in artifacts
directory in format:

  test/var/artifacts

After each finished failed gitlab-ci job, user would be able to
download these files from it using gitlab-ci GUI.

Additionally added OSX_VARDIR environment variable to be able to
setup common path for artifacts and OSX shell scripts options.

  OSX_VARDIR: /tmp/tnt

Part of #5050
---
 .gitlab-ci.yml | 48 ++++++++++++++++++++++++++++++++++++++++++++++--
 .travis.mk     |  6 +++---
 2 files changed, 49 insertions(+), 5 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 712e248fe..5a00641d9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,7 @@ variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
   GIT_STRATEGY: none
   GIT_CLEAN_FLAGS: none
+  OSX_VARDIR: /tmp/tnt
 
 # 1. Git checkout strategy used from:
 #    https://docs.gitlab.com/ee/ci/yaml/README.html#git-checkout
@@ -35,6 +36,17 @@ before_script:
 
 # Jobs templates
 
+.artifacts_reject_files_template: &artifacts_reject_files_definition
+  artifacts:
+    expire_in: 1000 yrs
+    when: always
+    paths:
+      # default paths to reject files and logs
+      - "test/*/*.reject"
+      - "test/var/artifacts/*"
+      # OSX jobs paths to logs, reject files are allready in default locations
+      - "${OSX_VARDIR}/artifacts/*"
+
 .deploy_only_template: &deploy_only_definition
   only:
     - master
@@ -65,12 +77,26 @@ before_script:
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_reject_files_definition
 
 .docker_test_clang8_template: &docker_test_clang8_definition
   image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-buster:latest"
   stage: test
   tags:
     - docker_test
+  <<: *artifacts_reject_files_definition
+
+.pack_artifacts_reject_files_template: &pack_artifacts_reject_files_definition
+  <<: *artifacts_reject_files_definition
+  after_script:
+    - >
+      for f in `ls build/usr/src/*/tarantool-*/test/*/*.reject` ; do
+          d=`dirname $f | sed 's#^build/usr/src/.*/tarantool-.*/test/##g'` ;
+          mkdir -p test/$d 2>/dev/null ;
+          cp $f test/$d/. ;
+      done ;
+      rm -rf test/var ; mkdir -p test/var ;
+      cp -r build/usr/src/*/tarantool-*/test/var/artifacts test/var/.
 
 .pack_template: &pack_definition
   <<: *pack_only_definition
@@ -87,6 +113,7 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} package
+  <<: *pack_artifacts_reject_files_definition
 
 .deploy_template: &deploy_definition
   <<: *deploy_only_definition
@@ -103,16 +130,33 @@ before_script:
     - deploy_test
   script:
     - ${GITLAB_MAKE} deploy
+  <<: *pack_artifacts_reject_files_definition
 
 .osx_template: &osx_definition
   stage: test
   script:
     - ${GITLAB_MAKE} test_osx
+  after_script:
+    # Artifacts can't be used from outside the project directory, check:
+    #   https://docs.gitlab.com/ee/ci/yaml/README.html#artifactspaths
+    # "Paths are relative to the project directory ($CI_PROJECT_DIR)"
+    # "and can’t directly link outside it."
+    - rm -rf test/var ; mkdir test/var && cp -rf ${OSX_VARDIR}/artifacts test/var/.
+  <<: *artifacts_reject_files_definition
 
 .vbox_template: &vbox_definition
   stage: test
   after_script:
-    - ${GITLAB_MAKE} vms_shutdown
+    - >
+      for f in `ssh ${VMS_USER}@127.0.0.1 -p ${VMS_PORT} ls tarantool/test/*/*.reject` ; do
+          d=`dirname $f | sed 's#^tarantool/##g'` ;
+          mkdir -p $d 2>/dev/null ;
+          scp -P ${VMS_PORT} ${VMS_USER}@127.0.0.1:tarantool/$d/*.reject $d/.
+      done ;
+      rm -rf test/var ; mkdir -p test/var ;
+      scp -r -P ${VMS_PORT} ${VMS_USER}@127.0.0.1:tarantool/test/var/artifacts test/var/. ;
+      ${GITLAB_MAKE} vms_shutdown
+  <<: *artifacts_reject_files_definition
 
 .perf_docker_test_template: &perf_docker_test_definition
   <<: *perf_only_definition
@@ -567,7 +611,7 @@ static_build_cmake_linux:
     - ${GITLAB_MAKE} test_static_build_cmake_linux
 
 static_build_cmake_osx_15:
-  stage: test
+  <<: *osx_definition
   tags:
     - osx_15
   script:
diff --git a/.travis.mk b/.travis.mk
index 58d0c1596..db457bf56 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -217,11 +217,11 @@ INIT_TEST_ENV_OSX=\
 		launchctl limit maxproc || : ; \
 		ulimit -u ${MAX_PROC} || : ; \
 		ulimit -u ; \
-		rm -rf /tmp/tnt
+		rm -rf ${OSX_VARDIR}
 
 test_osx_no_deps: build_osx
 	${INIT_TEST_ENV_OSX}; \
-	cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
+	cd test && ./test-run.py --vardir ${OSX_VARDIR} --force $(TEST_RUN_EXTRA_PARAMS)
 
 test_osx: deps_osx test_osx_no_deps
 
@@ -240,7 +240,7 @@ test_static_build_cmake_osx: base_deps_osx
 	cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \
 	make -j && ctest -V
 	${INIT_TEST_ENV_OSX}; \
-	cd test && ./test-run.py --vardir /tmp/tnt \
+	cd test && ./test-run.py --vardir ${OSX_VARDIR} \
 		--builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \
 		--force $(TEST_RUN_EXTRA_PARAMS)
 
-- 
2.17.1

[Tarantool-patches] [PATCH v2 2/2] gitlab-ci: set opensuse jobs to test group

[Alexander V. Tikhonov]

Set opensuse jobs to test group to be sure that it will be run with
artifacts collecting and without gitlab-ci jobs extra parallization.
---
 .gitlab-ci.yml | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5a00641d9..27d65ff7a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -478,13 +478,13 @@ debian_10:
     DIST: 'buster'
 
 opensuse_15_1:
-  <<: *pack_definition
+  <<: *pack_test_definition
   variables:
     OS: 'opensuse-leap'
     DIST: '15.1'
 
 opensuse_15_2:
-  <<: *pack_definition
+  <<: *pack_test_definition
   variables:
     OS: 'opensuse-leap'
     DIST: '15.2'
@@ -587,13 +587,13 @@ debian_10_deploy:
     DIST: 'buster'
 
 opensuse_15_1_deploy:
-  <<: *deploy_definition
+  <<: *deploy_test_definition
   variables:
     OS: 'opensuse-leap'
     DIST: '15.1'
 
 opensuse_15_2_deploy:
-  <<: *deploy_definition
+  <<: *deploy_test_definition
   variables:
     OS: 'opensuse-leap'
     DIST: '15.2'
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v2 1/2] gitlab-ci: save failed test results files

[Alexander Turenko]

> +  <<: *artifacts_reject_files_definition
> +
> +.pack_artifacts_reject_files_template: &pack_artifacts_reject_files_definition
> +  <<: *artifacts_reject_files_definition
> +  after_script:
> +    - >
> +      for f in `ls build/usr/src/*/tarantool-*/test/*/*.reject` ; do

Nit: 'for f in path-with-star-globbing; do' would do almost the same and
I would suggest to use the simpler form (it also does not depend on an
external tool and does not involve forking), but this way we should
handle 'no such files' case explicitly. See example here:

https://github.com/tarantool/tarantool/commit/080beba063b05b184d7befcb15a8bcf32e7424f6

 | for file in @TARANTOOL_ENABLEDDIR@/*.lua; do
 |     instance=`basename $file .lua`
 |     [ "${instance}" = "*" ] && break  # skip empty directory
 |     <...>
 | done

When the glob is not matched, it falls into the loop with the 'as is'
value (with star symbols).

We all like bourne shell for those funny pitfalls.

> +          d=`dirname $f | sed 's#^build/usr/src/.*/tarantool-.*/test/##g'` ;
> +          mkdir -p test/$d 2>/dev/null ;
> +          cp $f test/$d/. ;
> +      done ;
> +      rm -rf test/var ; mkdir -p test/var ;
> +      cp -r build/usr/src/*/tarantool-*/test/var/artifacts test/var/.

Nit: The period at the end looks unusual, but don't change anything.

The patch would be a way simpler if test-run would collect everything
that is necessary to upload within the artifacts directory.

In fact, reject files are already there:
test/var/artifacts/<worker_name>/<test_name>.result. But if you want to
organize them in an another way, I would do this on the test-run side
(somewhere within test/var/artifacts).

Related code: [1]. See, the reject file is just copied, not moved.

[1]: https://github.com/tarantool/test-run/blob/461731d71cf27faa6295bc759941f402fedc88e4/lib/test.py#L254

Re: [Tarantool-patches] [PATCH v2 1/2] gitlab-ci: save failed test results files

[Alexander V. Tikhonov]

Hi Alexander, thanks for the review. I've removed collecting of results
files *.reject, which duplicates *.result files in artifactory path.
And set 'artifactory' path as root in saved artifacts packages.

On Mon, Sep 21, 2020 at 05:18:13AM +0300, Alexander Turenko wrote:
> > +  <<: *artifacts_reject_files_definition
> > +
> > +.pack_artifacts_reject_files_template: &pack_artifacts_reject_files_definition
> > +  <<: *artifacts_reject_files_definition
> > +  after_script:
> > +    - >
> > +      for f in `ls build/usr/src/*/tarantool-*/test/*/*.reject` ; do
> 
> Nit: 'for f in path-with-star-globbing; do' would do almost the same and
> I would suggest to use the simpler form (it also does not depend on an
> external tool and does not involve forking), but this way we should
> handle 'no such files' case explicitly. See example here:
> 
> https://github.com/tarantool/tarantool/commit/080beba063b05b184d7befcb15a8bcf32e7424f6
> 
>  | for file in @TARANTOOL_ENABLEDDIR@/*.lua; do
>  |     instance=`basename $file .lua`
>  |     [ "${instance}" = "*" ] && break  # skip empty directory
>  |     <...>
>  | done
> 
> When the glob is not matched, it falls into the loop with the 'as is'
> value (with star symbols).
> 
> We all like bourne shell for those funny pitfalls.
> 
> > +          d=`dirname $f | sed 's#^build/usr/src/.*/tarantool-.*/test/##g'` ;
> > +          mkdir -p test/$d 2>/dev/null ;
> > +          cp $f test/$d/. ;
> > +      done ;
> > +      rm -rf test/var ; mkdir -p test/var ;
> > +      cp -r build/usr/src/*/tarantool-*/test/var/artifacts test/var/.
> 
> Nit: The period at the end looks unusual, but don't change anything.
> 
> The patch would be a way simpler if test-run would collect everything
> that is necessary to upload within the artifacts directory.
> 
> In fact, reject files are already there:
> test/var/artifacts/<worker_name>/<test_name>.result. But if you want to
> organize them in an another way, I would do this on the test-run side
> (somewhere within test/var/artifacts).
> 
> Related code: [1]. See, the reject file is just copied, not moved.
> 
> [1]: https://github.com/tarantool/test-run/blob/461731d71cf27faa6295bc759941f402fedc88e4/lib/test.py#L254

Re: [Tarantool-patches] [PATCH v2 2/2] gitlab-ci: set opensuse jobs to test group

[Kirill Yukhin]

Hello,

On 17 сен 11:04, Alexander V. Tikhonov wrote:
> Set opensuse jobs to test group to be sure that it will be run with
> artifacts collecting and without gitlab-ci jobs extra parallization.
> ---
>  .gitlab-ci.yml | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

I've checked your patch into 1.10, 2.4, 2.5, master.

--
Regards, Kirill Yukhin