[Tarantool-patches] [PATCH v3] gitlab-ci: implement OSX 10.14 testing on mac mini

[Tarantool-patches] [PATCH v3] gitlab-ci: implement OSX 10.14 testing on mac mini

[Alexander V. Tikhonov]

Added mac mini hosts into testing and set to it all
OSX 10.14 test jobs with new osx_14 gitlab-runner tag.
Also made additional setup:
 - added PATHs to python2 and pip
 - set 'var' directory for test-run tool to the shorter
   path name to avoid of issues with long names.
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/osx_on_mini14-full-ci

 .gitlab-ci.yml | 20 ++++++++------------
 .gitlab.mk     |  2 +-
 .travis.mk     | 32 +++++++++++++++++++++++++-------
 3 files changed, 34 insertions(+), 20 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5177a9903..7414dd578 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -152,28 +152,24 @@ osx_13_release:
     - ${GITLAB_MAKE} vms_test_osx
 
 osx_14_release:
-  <<: *vbox_definition
+  <<: *docker_test_definition
   tags:
-    - vms_osx_14
+    - osx_14
   variables:
-    VMS_NAME: 'osx_14'
-    VMS_USER: 'tarantool'
-    VMS_PORT: '2222'
+    PYTHON2: "python"
   script:
-    - ${GITLAB_MAKE} vms_test_osx
+    - ${GITLAB_MAKE} test_osx
 
 osx_14_release_lto:
   <<: *release_only_definition
-  <<: *vbox_definition
+  <<: *docker_test_definition
   tags:
-    - vms_osx_14
+    - osx_14
   variables:
     EXTRA_ENV: "export CMAKE_EXTRA_PARAMS=-DENABLE_LTO=ON ;"
-    VMS_NAME: 'osx_14'
-    VMS_USER: 'tarantool'
-    VMS_PORT: '2222'
+    PYTHON2: "python"
   script:
-    - ${GITLAB_MAKE} vms_test_osx
+    - ${GITLAB_MAKE} test_osx
 
 freebsd_12_release:
   <<: *vbox_definition
diff --git a/.gitlab.mk b/.gitlab.mk
index b39c5c651..d804d3cf6 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -14,7 +14,7 @@ git_submodule_update:
 		git submodule update --recursive --init
 
 # Pass *_no_deps goals to .travis.mk.
-test_%_no_deps: git_submodule_update
+test_%: git_submodule_update
 	${TRAVIS_MAKE} $@
 
 # #######################################################
diff --git a/.travis.mk b/.travis.mk
index 42969ff56..9db49a6b7 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -5,6 +5,7 @@
 DOCKER_IMAGE?=packpack/packpack:debian-stretch
 TEST_RUN_EXTRA_PARAMS?=
 MAX_FILES?=65534
+MAX_PROC?=2500
 
 all: package
 
@@ -127,13 +128,24 @@ test_asan_debian: deps_debian deps_buster_clang_8 test_asan_debian_no_deps
 # OSX #
 #######
 
+# Python2 is EOF and we need to be sure that it is in use instead of Python3
+OSX_PKGS=openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake ${PYTHON2}
+
 deps_osx:
-	brew update
-	brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force
-	python2 -V || brew install python2 --force
+	# install brew using command from Homebrew repository instructions:
+	#   https://github.com/Homebrew/install
+	# NOTE: 'echo' command below is required since brew installation
+	# script obliges the one to enter a newline for confirming the
+	# installation via Ruby script.
+	brew update || echo | /usr/bin/ruby -e \
+		"$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+	# try to install the packages either upgrade it to avoid of fails
+	# if the package already exists with the previous version
+	brew install --force ${OSX_PKGS} || brew upgrade ${OSX_PKGS}
 	curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py >get-pip.py
-	python get-pip.py --user
-	pip install --user --force-reinstall -r test-run/requirements.txt
+	python get-pip.py
+	export PATH=/Users/${USER}/Library/Python/2.7/bin:${PATH} ; \
+		pip install --force-reinstall -r test-run/requirements.txt
 
 build_osx:
 	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS}
@@ -148,11 +160,17 @@ test_osx_no_deps: build_osx
 	# call as tests runs call.
 	# Tests: Temporary excluded replication/ suite with some tests
 	#        from other suites by issues #4357 and #4370
-	echo tarantool | sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
+	sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
 		launchctl limit maxfiles || : ; \
 		ulimit -n ${MAX_FILES} || : ; \
 		ulimit -n ; \
-		cd test && ./test-run.py --force $(TEST_RUN_EXTRA_PARAMS) \
+		sudo -S launchctl limit maxproc ${MAX_PROC} || : ; \
+		launchctl limit maxproc || : ; \
+		ulimit -u ${MAX_PROC} || : ; \
+		ulimit -u ; \
+		export PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH} ; \
+		rm -rf /tmp/tnt ; \
+		cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) \
 			app/ app-tap/ box/ box-py/ box-tap/ engine/ engine_long/ long_run-py/ luajit-tap/ \
 			replication-py/ small/ sql/ sql-tap/ swim/ unit/ vinyl/ wal_off/ xlog/ xlog-py/
 
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v3] gitlab-ci: implement OSX 10.14 testing on mac mini

[Oleg Piskunov]

[-- Attachment #1: Type: text/plain, Size: 5578 bytes --]


Sasha,
 
I guess it doesn’t work correctly around python and pip installation.
Here is your latest log from the gitlab-ci run ( https://gitlab.com/tarantool/tarantool/-/jobs/452400831 ):
brew install --force openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake python || brew upgrade openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake python
...
Warning: python 3.7.6_1 is already installed and up-to-date
To reinstall 3.7.6_1, run `brew reinstall python`
As you can see brew install updating python3 instead of python2.7
The reason that it still working correctly is that python2.7 preinstalled before into different path which you are using.
 
Need to investigate and fix.
 
>Четверг, 27 февраля 2020, 16:23 +03:00 от Alexander V. Tikhonov <avtikhon@tarantool.org>:
>Added mac mini hosts into testing and set to it all
>OSX 10.14 test jobs with new osx_14 gitlab-runner tag.
>Also made additional setup:
> - added PATHs to python2 and pip
> - set 'var' directory for test-run tool to the shorter
>   path name to avoid of issues with long names.
>---
>
>Github:  https://github.com/tarantool/tarantool/tree/avtikhon/osx_on_mini14-full-ci
>
> .gitlab-ci.yml | 20 ++++++++------------
> .gitlab.mk | 2 +-
> .travis.mk | 32 +++++++++++++++++++++++++-------
> 3 files changed, 34 insertions(+), 20 deletions(-)
>
>diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>index 5177a9903..7414dd578 100644
>--- a/.gitlab-ci.yml
>+++ b/.gitlab-ci.yml
>@@ -152,28 +152,24 @@ osx_13_release:
>     - ${GITLAB_MAKE} vms_test_osx
> 
> osx_14_release:
>- <<: *vbox_definition
>+ <<: *docker_test_definition
>   tags:
>- - vms_osx_14
>+ - osx_14
>   variables:
>- VMS_NAME: 'osx_14'
>- VMS_USER: 'tarantool'
>- VMS_PORT: '2222'
>+ PYTHON2: "python"
>   script:
>- - ${GITLAB_MAKE} vms_test_osx
>+ - ${GITLAB_MAKE} test_osx
> 
> osx_14_release_lto:
>   <<: *release_only_definition
>- <<: *vbox_definition
>+ <<: *docker_test_definition
>   tags:
>- - vms_osx_14
>+ - osx_14
>   variables:
>     EXTRA_ENV: "export CMAKE_EXTRA_PARAMS=-DENABLE_LTO=ON ;"
>- VMS_NAME: 'osx_14'
>- VMS_USER: 'tarantool'
>- VMS_PORT: '2222'
>+ PYTHON2: "python"
>   script:
>- - ${GITLAB_MAKE} vms_test_osx
>+ - ${GITLAB_MAKE} test_osx
> 
> freebsd_12_release:
>   <<: *vbox_definition
>diff --git a/.gitlab.mk b/.gitlab.mk
>index b39c5c651..d804d3cf6 100644
>--- a/.gitlab.mk
>+++ b/.gitlab.mk
>@@ -14,7 +14,7 @@ git_submodule_update:
>  git submodule update --recursive --init
> 
> # Pass *_no_deps goals to .travis.mk.
>-test_%_no_deps: git_submodule_update
>+test_%: git_submodule_update
>  ${TRAVIS_MAKE} $@
> 
> # #######################################################
>diff --git a/.travis.mk b/.travis.mk
>index 42969ff56..9db49a6b7 100644
>--- a/.travis.mk
>+++ b/.travis.mk
>@@ -5,6 +5,7 @@
> DOCKER_IMAGE?=packpack/packpack:debian-stretch
> TEST_RUN_EXTRA_PARAMS?=
> MAX_FILES?=65534
>+MAX_PROC?=2500
> 
> all: package
> 
>@@ -127,13 +128,24 @@ test_asan_debian: deps_debian deps_buster_clang_8 test_asan_debian_no_deps
> # OSX #
> #######
> 
>+# Python2 is EOF and we need to be sure that it is in use instead of Python3
>+OSX_PKGS=openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake ${PYTHON2}
>+
> deps_osx:
>- brew update
>- brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force
>- python2 -V || brew install python2 --force
>+ # install brew using command from Homebrew repository instructions:
>+ #  https://github.com/Homebrew/install
>+ # NOTE: 'echo' command below is required since brew installation
>+ # script obliges the one to enter a newline for confirming the
>+ # installation via Ruby script.
>+ brew update || echo | /usr/bin/ruby -e \
>+ "$(curl -fsSL  https://raw.githubusercontent.com/Homebrew/install/master/install )"
>+ # try to install the packages either upgrade it to avoid of fails
>+ # if the package already exists with the previous version
>+ brew install --force ${OSX_PKGS} || brew upgrade ${OSX_PKGS}
>  curl --silent --show-error --retry 5  https://bootstrap.pypa.io/get-pip.py >get-pip.py
>- python get-pip.py --user
>- pip install --user --force-reinstall -r test-run/requirements.txt
>+ python get-pip.py
>+ export PATH=/Users/${USER}/Library/Python/2.7/bin:${PATH} ; \
>+ pip install --force-reinstall -r test-run/requirements.txt
> 
> build_osx:
>  cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS}
>@@ -148,11 +160,17 @@ test_osx_no_deps: build_osx
>  # call as tests runs call.
>  # Tests: Temporary excluded replication/ suite with some tests
>  # from other suites by issues #4357 and #4370
>- echo tarantool | sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
>+ sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
>  launchctl limit maxfiles || : ; \
>  ulimit -n ${MAX_FILES} || : ; \
>  ulimit -n ; \
>- cd test && ./test-run.py --force $(TEST_RUN_EXTRA_PARAMS) \
>+ sudo -S launchctl limit maxproc ${MAX_PROC} || : ; \
>+ launchctl limit maxproc || : ; \
>+ ulimit -u ${MAX_PROC} || : ; \
>+ ulimit -u ; \
>+ export PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH} ; \
>+ rm -rf /tmp/tnt ; \
>+ cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) \
>  app/ app-tap/ box/ box-py/ box-tap/ engine/ engine_long/ long_run-py/ luajit-tap/ \
>  replication-py/ small/ sql/ sql-tap/ swim/ unit/ vinyl/ wal_off/ xlog/ xlog-py/
> 
>--
>2.17.1
>  
 
 
--
Oleg Piskunov
 

[-- Attachment #2: Type: text/html, Size: 10176 bytes --]

Re: [Tarantool-patches] [PATCH v3] gitlab-ci: implement OSX 10.14 testing on mac mini

[Alexander Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 6048 bytes --]

Oleg, thanks for your review, I've made the changes and commented the review.


>Пятница, 28 февраля 2020, 22:23 +03:00 от Oleg Piskunov <o.piskunov@tarantool.org>:
>
>Sasha,
>I guess it doesn’t work correctly around python and pip installation.
>Here is your latest log from the gitlab-ci run ( https://gitlab.com/tarantool/tarantool/-/jobs/452400831 ):
>brew install --force openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake python || brew upgrade openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake python
>...
>Warning: python 3.7.6_1 is already installed and up-to-date
>To reinstall 3.7.6_1, run `brew reinstall python`
>As you can see brew install updating python3 instead of python2.7
>The reason that it still working correctly is that python2.7 preinstalled before into different path which you are using.
>Need to investigate and fix. 
Found that python 2 was removed just a few days ago at all from the brew formulas. To make
it workable found the solution to use the latest Python 2 commit.
>>Четверг, 27 февраля 2020, 16:23 +03:00 от Alexander V. Tikhonov < avtikhon@tarantool.org >:
>>Added mac mini hosts into testing and set to it all
>>OSX 10.14 test jobs with new osx_14 gitlab-runner tag.
>>Also made additional setup:
>> - added PATHs to python2 and pip
>> - set 'var' directory for test-run tool to the shorter
>>   path name to avoid of issues with long names.
>>---
>>
>>Github:  https://github.com/tarantool/tarantool/tree/avtikhon/osx_on_mini14-full-ci
>>
>> .gitlab-ci.yml | 20 ++++++++------------
>> .gitlab.mk | 2 +-
>> .travis.mk | 32 +++++++++++++++++++++++++-------
>> 3 files changed, 34 insertions(+), 20 deletions(-)
>>
>>diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>index 5177a9903..7414dd578 100644
>>--- a/.gitlab-ci.yml
>>+++ b/.gitlab-ci.yml
>>@@ -152,28 +152,24 @@ osx_13_release:
>>     - ${GITLAB_MAKE} vms_test_osx
>>
>> osx_14_release:
>>- <<: *vbox_definition
>>+ <<: *docker_test_definition
>>   tags:
>>- - vms_osx_14
>>+ - osx_14
>>   variables:
>>- VMS_NAME: 'osx_14'
>>- VMS_USER: 'tarantool'
>>- VMS_PORT: '2222'
>>+ PYTHON2: "python"
>>   script:
>>- - ${GITLAB_MAKE} vms_test_osx
>>+ - ${GITLAB_MAKE} test_osx
>>
>> osx_14_release_lto:
>>   <<: *release_only_definition
>>- <<: *vbox_definition
>>+ <<: *docker_test_definition
>>   tags:
>>- - vms_osx_14
>>+ - osx_14
>>   variables:
>>     EXTRA_ENV: "export CMAKE_EXTRA_PARAMS=-DENABLE_LTO=ON ;"
>>- VMS_NAME: 'osx_14'
>>- VMS_USER: 'tarantool'
>>- VMS_PORT: '2222'
>>+ PYTHON2: "python"
>>   script:
>>- - ${GITLAB_MAKE} vms_test_osx
>>+ - ${GITLAB_MAKE} test_osx
>>
>> freebsd_12_release:
>>   <<: *vbox_definition
>>diff --git a/.gitlab.mk b/.gitlab.mk
>>index b39c5c651..d804d3cf6 100644
>>--- a/.gitlab.mk
>>+++ b/.gitlab.mk
>>@@ -14,7 +14,7 @@ git_submodule_update:
>>  git submodule update --recursive --init
>>
>> # Pass *_no_deps goals to .travis.mk.
>>-test_%_no_deps: git_submodule_update
>>+test_%: git_submodule_update
>>  ${TRAVIS_MAKE} $@
>>
>> # #######################################################
>>diff --git a/.travis.mk b/.travis.mk
>>index 42969ff56..9db49a6b7 100644
>>--- a/.travis.mk
>>+++ b/.travis.mk
>>@@ -5,6 +5,7 @@
>> DOCKER_IMAGE?=packpack/packpack:debian-stretch
>> TEST_RUN_EXTRA_PARAMS?=
>> MAX_FILES?=65534
>>+MAX_PROC?=2500
>>
>> all: package
>>
>>@@ -127,13 +128,24 @@ test_asan_debian: deps_debian deps_buster_clang_8 test_asan_debian_no_deps
>> # OSX #
>> #######
>>
>>+# Python2 is EOF and we need to be sure that it is in use instead of Python3
>>+OSX_PKGS=openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake ${PYTHON2}
>>+
>> deps_osx:
>>- brew update
>>- brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force
>>- python2 -V || brew install python2 --force
>>+ # install brew using command from Homebrew repository instructions:
>>+ #  https://github.com/Homebrew/install
>>+ # NOTE: 'echo' command below is required since brew installation
>>+ # script obliges the one to enter a newline for confirming the
>>+ # installation via Ruby script.
>>+ brew update || echo | /usr/bin/ruby -e \
>>+ "$(curl -fsSL  https://raw.githubusercontent.com/Homebrew/install/master/install )"
>>+ # try to install the packages either upgrade it to avoid of fails
>>+ # if the package already exists with the previous version
>>+ brew install --force ${OSX_PKGS} || brew upgrade ${OSX_PKGS}
>>  curl --silent --show-error --retry 5  https://bootstrap.pypa.io/get-pip.py >get-pip.py
>>- python get-pip.py --user
>>- pip install --user --force-reinstall -r test-run/requirements.txt
>>+ python get-pip.py
>>+ export PATH=/Users/${USER}/Library/Python/2.7/bin:${PATH} ; \
>>+ pip install --force-reinstall -r test-run/requirements.txt
>>
>> build_osx:
>>  cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS}
>>@@ -148,11 +160,17 @@ test_osx_no_deps: build_osx
>>  # call as tests runs call.
>>  # Tests: Temporary excluded replication/ suite with some tests
>>  # from other suites by issues #4357 and #4370
>>- echo tarantool | sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
>>+ sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
>>  launchctl limit maxfiles || : ; \
>>  ulimit -n ${MAX_FILES} || : ; \
>>  ulimit -n ; \
>>- cd test && ./test-run.py --force $(TEST_RUN_EXTRA_PARAMS) \
>>+ sudo -S launchctl limit maxproc ${MAX_PROC} || : ; \
>>+ launchctl limit maxproc || : ; \
>>+ ulimit -u ${MAX_PROC} || : ; \
>>+ ulimit -u ; \
>>+ export PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH} ; \
>>+ rm -rf /tmp/tnt ; \
>>+ cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) \
>>  app/ app-tap/ box/ box-py/ box-tap/ engine/ engine_long/ long_run-py/ luajit-tap/ \
>>  replication-py/ small/ sql/ sql-tap/ swim/ unit/ vinyl/ wal_off/ xlog/ xlog-py/
>>
>>--
>>2.17.1
>--
>Oleg Piskunov

-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 11206 bytes --]