[Tarantool-patches] [PATCH 0/2] gitlab-ci: increase testing timeouts

[Tarantool-patches] [PATCH 0/2] gitlab-ci: increase testing timeouts

[Alexander Turenko]

The first commit allows to pass any variable from current environment to
a docker container or a ssh session with a virtualbox machine. The
second brings test-run support for setting custom timeouts via
environment variables.

I set ~300 seconds timeouts in the GitLab CI web interface.

The main reason to increase the timeouts is stalls of disk accesses when
many vinyl tests are run in parallel and amount of machine resources we
have at the moment.

See details in the commit messages.

I looked over all jobs and, where we run testing, test-run sets the
increased timeouts.

https://github.com/tarantool/tarantool/tree/Totktonada/gitlab-ci-increase-testing-timeouts
https://github.com/tarantool/test-run/issues/258

Alexander Turenko (2):
  ci: preserve certain environment variables
  test: update test-run (pass timeouts via env)

 .gitlab.mk | 14 +++++++++++---
 .travis.mk | 22 ++++++++++++++++++++--
 test-run   |  2 +-
 3 files changed, 32 insertions(+), 6 deletions(-)

-- 
2.25.0

[Tarantool-patches] [PATCH 1/2] ci: preserve certain environment variables

[Alexander Turenko]

We want to increase testing timeouts for GitLab CI, where we use our own
runners and observe stalls and high disk pressure when several vinyl
tests are run in parallel. The idea is to set variables in GitLab CI web
interface and read them from test-run (see [1]).

First, we need to pass the variables into inner environments. GitLab CI
jobs run the testing using packpack, Docker or VirtualBox.

Packpack already preserves environment variables that are listed in the
PRESERVE_ENVVARS variable (see [2]).

This commit passes the variables that are listed in the PRESERVE_ENVVARS
variable into Docker and VirtualBox environment. So, all jobs will have
given variables in the enviroment. (Also dropped unused EXTRA_ENV
variable.)

The next commit will update the test-run submodule with support of
setting timeouts using environment variables.

[1]: https://github.com/tarantool/test-run/issues/258
[2]: https://github.com/packpack/packpack/pull/135
---
 .gitlab.mk | 14 +++++++++++---
 .travis.mk | 22 ++++++++++++++++++++--
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/.gitlab.mk b/.gitlab.mk
index 5dbf66cef..f4fcc8fae 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -85,6 +85,16 @@ perf_cleanup: perf_clone_benchs_repo perf_cleanup_image
 # Run tests under a virtual machine
 # #################################
 
+# Transform the ${PRESERVE_ENVVARS} comma separated variables list
+# to the 'key="value" key="value" <...>' string.
+#
+# Add PRESERVE_ENVVARS itself to the list to allow to use this
+# make script again from the inner environment (if there will be
+# a need).
+comma := ,
+ENVVARS := PRESERVE_ENVVARS $(subst $(comma), ,$(PRESERVE_ENVVARS))
+PRESERVE_ENV := $(foreach var,$(ENVVARS),$(var)="$($(var))")
+
 vms_start:
 	VBoxManage controlvm ${VMS_NAME} poweroff || true
 	VBoxManage snapshot ${VMS_NAME} restore ${VMS_NAME}
@@ -93,9 +103,7 @@ vms_start:
 vms_test_%:
 	tar czf - ../tarantool | ssh ${VMS_USER}@127.0.0.1 -p ${VMS_PORT} tar xzf -
 	ssh ${VMS_USER}@127.0.0.1 -p ${VMS_PORT} "/bin/bash -c \
-		'${EXTRA_ENV} \
-		cd tarantool && \
-		${TRAVIS_MAKE} $(subst vms_,,$@)'"
+		'cd tarantool && ${PRESERVE_ENV} ${TRAVIS_MAKE} $(subst vms_,,$@)'"
 
 vms_shutdown:
 	VBoxManage controlvm ${VMS_NAME} poweroff
diff --git a/.travis.mk b/.travis.mk
index 8396ba9e5..b835a603b 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -20,6 +20,16 @@ LEIN_URL="https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lei
 TERRAFORM_NAME="terraform_0.13.1_linux_amd64.zip"
 TERRAFORM_URL="https://releases.hashicorp.com/terraform/0.13.1/"$(TERRAFORM_NAME)
 
+# Transform the ${PRESERVE_ENVVARS} comma separated variables list
+# to the '-e key="value" -e key="value" <...>' string.
+#
+# Add PRESERVE_ENVVARS itself to the list to allow to use this
+# make script again from the inner environment (if there will be
+# a need).
+comma := ,
+ENVVARS := PRESERVE_ENVVARS $(subst $(comma), ,$(PRESERVE_ENVVARS))
+DOCKER_ENV := $(foreach var,$(ENVVARS),-e $(var)="$($(var))")
+
 all: package
 
 package:
@@ -50,6 +60,7 @@ docker_%:
 		-e APT_EXTRA_FLAGS="${APT_EXTRA_FLAGS}" \
 		-e CC=${CC} \
 		-e CXX=${CXX} \
+		${DOCKER_ENV} \
 		${DOCKER_IMAGE} \
 		make -f .travis.mk $(subst docker_,,$@)
 
@@ -232,8 +243,14 @@ test_static_build_cmake_linux:
 # ###################
 
 test_debian_docker_luacheck:
-	docker run -w ${OOS_SRC_PATH} -v ${PWD}:${OOS_SRC_PATH} --privileged \
-		--cap-add=sys_nice --network=host -i ${DOCKER_IMAGE_TARANTOOL} \
+	docker run                       \
+		-w ${OOS_SRC_PATH}           \
+		-v ${PWD}:${OOS_SRC_PATH}    \
+		--privileged                 \
+		--cap-add=sys_nice           \
+		--network=host               \
+		${DOCKER_ENV}                \
+		-i ${DOCKER_IMAGE_TARANTOOL} \
 		make -f .travis.mk test_debian_luacheck
 
 test_debian_install_luacheck:
@@ -264,6 +281,7 @@ test_oos_build:
 	docker run --network=host -w ${OOS_SRC_PATH} \
 		--mount type=bind,source="${PWD}",target=${OOS_SRC_PATH},readonly,bind-propagation=rslave \
 		--tmpfs ${OOS_BUILD_PATH}:exec \
+		${DOCKER_ENV} \
 		-i ${DOCKER_IMAGE_TARANTOOL} \
 		make -f .travis.mk ${OOS_BUILD_RULE}
 
-- 
2.25.0

[Tarantool-patches] [PATCH 2/2] test: update test-run (pass timeouts via env)

[Alexander Turenko]

The following variables now control timeouts (if corresponding command
line options are not passed): TEST_TIMEOUT, NO_OUTPUT_TIMEOUT,
REPLICATION_SYNC_TIMEOUT. See [1] for details.

I set the following values in the GitLab CI web interface:

| Variable                 | Value                                                   |
| ------------------------ | ------------------------------------------------------- |
| REPLICATION_SYNC_TIMEOUT | 300                                                     |
| TEST_TIMEOUT             | 310                                                     |
| NO_OUTPUT_TIMEOUT        | 320                                                     |
| PRESERVE_ENVVARS         | REPLICATION_SYNC_TIMEOUT,TEST_TIMEOUT,NO_OUTPUT_TIMEOUT |

See packpack change [2] and the commit 'ci: preserve certain environment
variables' regarding the PRESERVE_ENVVARS variable.

The reason, why we need to increase timeouts, comes from the following
facts:

- We use self-hosted runners to serve GitLab CI jobs. So, the machine
  resources are limited.
- We run testing with high level of parallelism to speed it up.
- We have a bunch of vinyl tests, which intensively use disk.

Disk accesses may be quite long within this infrastructure and the
obvious way to workaround the problem is to increase timeouts.

In the long term we should scale resources depending on the testing
needs. We'll try to use GitHub hosted runners or, if we'll reach some
limits, will setup GitHub runners on the Mail.Ru Cloud Solutions
infrastructure.

[1]: https://github.com/tarantool/test-run/issues/258
[2]: https://github.com/packpack/packpack/pull/135
---
 test-run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test-run b/test-run
index 584e273b1..9e5fc6d0f 160000
--- a/test-run
+++ b/test-run
@@ -1 +1 @@
-Subproject commit 584e273b134ffa0ee7d44419f7443f752ff4fb33
+Subproject commit 9e5fc6d0fede3b1d142a44c3ca36cfff3cb32f9f
-- 
2.25.0

Re: [Tarantool-patches] [PATCH 0/2] gitlab-ci: increase testing timeouts

[Alexander V. Tikhonov]

Hi Alexander, thanks for the patch, as I see no new degradation found in
gitlab-ci testing commit criteria pipeline [1], patch LGTM.

[1] - https://gitlab.com/tarantool/tarantool/-/pipelines/234706924

On Sun, Dec 27, 2020 at 07:13:24AM +0300, Alexander Turenko wrote:
> The first commit allows to pass any variable from current environment to
> a docker container or a ssh session with a virtualbox machine. The
> second brings test-run support for setting custom timeouts via
> environment variables.
> 
> I set ~300 seconds timeouts in the GitLab CI web interface.
> 
> The main reason to increase the timeouts is stalls of disk accesses when
> many vinyl tests are run in parallel and amount of machine resources we
> have at the moment.
> 
> See details in the commit messages.
> 
> I looked over all jobs and, where we run testing, test-run sets the
> increased timeouts.
> 
> https://github.com/tarantool/tarantool/tree/Totktonada/gitlab-ci-increase-testing-timeouts
> https://github.com/tarantool/test-run/issues/258
> 
> Alexander Turenko (2):
>   ci: preserve certain environment variables
>   test: update test-run (pass timeouts via env)
> 
>  .gitlab.mk | 14 +++++++++++---
>  .travis.mk | 22 ++++++++++++++++++++--
>  test-run   |  2 +-
>  3 files changed, 32 insertions(+), 6 deletions(-)
> 
> -- 
> 2.25.0
> 

Re: [Tarantool-patches] [PATCH 0/2] gitlab-ci: increase testing timeouts

[Alexander Turenko]

On Sun, Dec 27, 2020 at 07:37:33AM +0300, Alexander V. Tikhonov wrote:
> Hi Alexander, thanks for the patch, as I see no new degradation found in
> gitlab-ci testing commit criteria pipeline [1], patch LGTM.
> 
> [1] - https://gitlab.com/tarantool/tarantool/-/pipelines/234706924

Pushed to master, 2.6, 2.5 and 1.10.

CCed Kirill.

I'll cite the note that I added to the 1.10 backport:

 | Note regarding the 1.10 backport: here we run tests during building of a
 | docker image for the static tarantool build. Here the variables from
 | PRESERVE_ENVVARS will not be passed. I would not run tests during the
 | image build, but rather do it by a separate `docker run` call. So we'll
 | able to pass -e options to increase timeouts there.

WBR, Alexander Turenko.