Re: [Tarantool-patches] [PATCH v1] gitlab-ci: add Catalina OSX 10.15

["Alexander Tikhonov" <avtikhon@tarantool.org>]

[-- Attachment #1: Type: text/plain, Size: 9252 bytes --]

Alexander, thank you for the review, I've made the changes as you suggested.


>Вторник, 17 декабря 2019, 18:46 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>To be honest, I didn't get most of changes here. So below I asked for
>explanation.
>
>WBR, Alexander Turenko.
>
>On Mon, Dec 09, 2019 at 02:45:54PM +0300, Alexander V. Tikhonov wrote:
>> Added Catalina OSX 10.15 to gitlab-ci testing and removed OSX 10.13,
>> due to decided to have only 2 last major releases, for now it is
>> 10.14 and 10.15 OSX versions.
>> 
>> Also added homebrew installation routine as it was suggested in its
>> instructions. Added path in environment to reach the pip binary that
>> was additionally installed based on python2 version in addition to
>> pip3 which is the default in Catalina OSX 10.15. Added installation
>> of the cmake and tool. Added upgrade of the OSX packages to avoid of
>> fails on already existed packages, but of the previous versions.
>> ---
>> 
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/osx_15_catalina
>> 
>>  .gitlab-ci.yml |  8 ++++----
>>  .travis.mk     | 30 +++++++++++++++++++++++++-----
>>  2 files changed, 29 insertions(+), 9 deletions(-)
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 3af5a3c8a..d3cb959d0 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -90,15 +90,15 @@ release_asan_clang8:
>>    script:
>>      - ${GITLAB_MAKE} test_asan_debian_no_deps
>> 
>> -osx_13_release:
>> +osx_15_release:
>>    <<: *release_only_definition
>>    <<: *vbox_definition
>>    tags:
>> -    - vms_osx_13
>> +    - vms_test
>
>Is it separate change? Is is about load balancing between our hosts?
>
>If so, let's move this change into its own commit.
Actually no, this is the new tag for the gitlab-ci runners, because in the current
version we have different tags for each of the OSX/FreeBSD versions, so to
have the new 'osx_15' version it was needed to create the new tag and I've
created it with the name 'vms_test'. The only question which can be here is why
it was not named as 'vms_osx_15' - the answer is simple, it is just the new name
which will be changed in the nearest time to the common 'vms_test' for all the
gitlab runners jobs, in this way we'll have only the following steps
- create tag 'vms_test'
while naming as 'vms_osx_15' will have extra steps like:
- create tag 'vms_osx_15'
- create tag 'vms_test'
- move job from tag 'vms_osx_15' to 'vms_test'
- remove tag 'vms_osx_15'
So I don't see any obstacles just to name the tag as 'vms_test' from the very start,
anyway the name is new.
>
>
>>    variables:
>> -    VMS_NAME: 'osx_13'
>> +    VMS_NAME: 'osx_15'
>>      VMS_USER: 'tarantool'
>> -    VMS_PORT: '2212'
>> +    VMS_PORT: '2242'
>>    script:
>>      - ${GITLAB_MAKE} vms_test_osx
>> 
>> diff --git a/.travis.mk b/.travis.mk
>> index 42969ff56..c8cb273fa 100644
>> --- a/.travis.mk
>> +++ b/.travis.mk
>> @@ -127,17 +127,37 @@ test_asan_debian: deps_debian deps_buster_clang_8 test_asan_debian_no_deps
>>  # OSX #
>>  #######
>> 
>> +OSX_PKGS=openssl readline curl icu4c libiconv zlib autoconf automake libtool cmake
>> +
>>  deps_osx:
>> -	brew update
>> -	brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force
>> +	# install brew using command from Homebrew repository instructions:
>> +	#  https://github.com/Homebrew/install
>> +	export PATH=${PATH}:/usr/local/bin ; \
>> +		brew update || echo | /usr/bin/ruby -e \
>> +			"`curl -fsSL  https://raw.githubusercontent.com/Homebrew/install/master/install` "
>
>The only motivation I found is that it is suggested in homebrew
>instructions. What did not work before this change? Is it actually
>related to 1.0.13 -> 10.15 update? 
Additional tool homebrew is not installed at macOS by default, also it uses the
'/usr/local' path (check  https://en.wikipedia.org/wiki/Homebrew_(package_management_software) )
which is not in the PATH environment by default. Right, the fix can be separated to the
standalone patch for the other macOS version, but the current part of code doesn't
depend to any of macOS version, that is why I suggested to add this change to the
current commit, anyway I'll move it to the separate commit, as suggested.
>
>
>> +	# try to install the packages either upgrade it to avoid of fails
>> +	# if the package allready exists but of the previous version
>
>Typo: allready. 
Corrected.
>
>
>> +	export PATH=${PATH}:/usr/local/bin ; \
>> +		brew install --force ${OSX_PKGS} || brew upgrade ${OSX_PKGS}
>
>I see the following warnings on 10.14 in Travis-CI on current master:
>
> | brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force
> | Warning:  openssl@1.1 1.1.1d is already installed, it's just not linked
> | You can use `brew link openssl@1.1` to link this version.
>
>Is it about them? So, it forces brew to install last packages versions?
>It seems this is not related to 10.13 -> 10.15 update (correct me if I'm
>wrong) and deserves its own commit. 
No, the problem is that if the package already exists with the previous version
then 'brew install' fails with message about it and suggests to upgrade it. To avoid of
its issue we can try to install all needed packages and try to upgrade it on failed
installation process, if the upgrade command fails too than it means the some other
fail occurred. Ok, let's move it to separate commit.
>
>
>> +	# OSX 10.15 Catalina started to use Xcode 11 which has 'Deprications'
>> +	# part in Release Notes:
>> +	#   Use of Python 2.7 isn’t recommended. This version is included in
>> +	#   macOS for compatibility with legacy software. Future versions of
>> +	#   macOS won’t include Python 2.7. Instead, it’s recommended that
>> +	#   you run python3 in Terminal. (51097165)
>>  	python2 -V || brew install python2 --force
>>  	curl --silent --show-error --retry 5  https://bootstrap.pypa.io/get-pip.py >get-pip.py
>>  	python get-pip.py --user
>
>Is it python3 on Mac OS 10.15? If so, I didn't get what are we doing
>here. Maybe it didn't matter before, but now we should explicitly call
>python2 here?
Here we check that python2 is installed, by default macOS 10.15 use python3
instead of python2 as previous MacOS 10.14 version used. After the manual
installation of python2 the link 'python' points to the python2 binary and we don't
need to call python2 explicitly, check:
tarantool@Tarantools-iMac tarantool % ls -al `which python`
/usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
>
>
>> -	pip install --user --force-reinstall -r test-run/requirements.txt
>> +	# To make pip tools available, the PATH environment must include the
>> +	# path to its binaries
>> +	export PATH=${PATH}:/Users/tarantool/Library/Python/2.7/bin ; \
>> +		pip install --user --force-reinstall -r test-run/requirements.txt
>
>We just install pip and it is not with default PATH? This looks strange.
>Can you verify that this extra path is necessary?
Sure, please check the following:
tarantool@Tarantools-iMac tarantool % curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py >get-pip.py

tarantool@Tarantools-iMac tarantool % python get-pip.py --user
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip
Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 19.3.1
Uninstalling pip-19.3.1:
Successfully uninstalled pip-19.3.1
WARNING: The scripts pip, pip2 and pip2.7 are installed in '/Users/tarantool/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-19.3.1

tarantool@Tarantools-iMac tarantool % which pip
pip not found

tarantool@Tarantools-iMac tarantool % export PATH=${PATH}:/Users/tarantool/Library/Python/2.7/bin ; which pip
/Users/tarantool/Library/Python/2.7/bin/pip
>
>> 
>>  build_osx:
>> -	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS}
>> -	make -j
>> +	export PATH=${PATH}:/usr/local/bin ; \
>> +		cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo \
>> +			-DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS} && \
>> +		make -j
>
>Didn't get this change: what is now in /usr/local/bin that was not there
>before?
The following example shows that 'make' that was installed by default is in
the default path '/usr/bin':

tarantool@Tarantools-iMac tarantool % which make
/usr/bin/make
while 'cmake' additionally installed by the homebrew exists only at the additional
path '/usr/local/bin':
tarantool@Tarantools-iMac tarantool % which cmake
/usr/local/bin/cmake
>
>> 
>>  test_osx_no_deps: build_osx
>>  	# Limits: Increase the maximum number of open file descriptors on macOS:
>> -- 
>> 2.17.1
>> 


-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 12560 bytes --]