[tarantool-patches] Re: [PATCH v1] travis-ci: fix OSX max files limits

[tarantool-patches] Re: [PATCH v1] travis-ci: fix OSX max files limits

[Alexander Turenko]

To be honest, I know nothing about launchctl and rlimits on Mac OS. So I
cannot give 'ok' or 'not ok' here. Please, consider some common comments
below.

WBR, Alexander Turenko.

On Tue, Jul 23, 2019 at 10:51:09AM +0300, Alexander V. Tikhonov wrote:
> Increased the maximum number of open file descriptors on macOS:
>   Travis-ci needed the "ulimit -n <value>" call
>   Gitlab-ci needed the "launchctl limit maxfiles <value>" call
> Also gitlib-ci needed the password to change the limits, while
> travis-ci run under root user. Limit setup set in the same
> call as tests runs call.

Are you know what is a reason of the difference between travis-ci and
gitlab-ci? I don't insist, but if it is known, it worth to mention it.

> Also found that the travis-ci global environment values schema
> had the mistake in definition - fixed with added "global:" tag
> and added the missed travis-ci environment value to the docker
> call.

I would extract it into separate commit.

> 
> Closes #4373
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4373-travis-osx-files
> Issue: https://github.com/tarantool/tarantool/issues/4373
> 
>  .travis.mk  | 30 ++++++++++++++----------------
>  .travis.yml |  5 +++--
>  2 files changed, 17 insertions(+), 18 deletions(-)
> 
> diff --git a/.travis.mk b/.travis.mk
> index 7d349bc21..f6c5ecbd0 100644
> --- a/.travis.mk
> +++ b/.travis.mk
> @@ -34,6 +34,7 @@ docker_%:
>  		-e TRAVIS_JOB_ID=${TRAVIS_JOB_ID} \
>  		-e CMAKE_EXTRA_PARAMS=${CMAKE_EXTRA_PARAMS} \
>  		-e APT_EXTRA_FLAGS="${APT_EXTRA_FLAGS}" \
> +		-e TEST_RUN_EXTRA_PARAMS="${TEST_RUN_EXTRA_PARAMS}" \
>  		-e CC=${CC} \
>  		-e CXX=${CXX} \
>  		${DOCKER_IMAGE} \
> @@ -134,22 +135,19 @@ build_osx:
>  	make -j
>  
>  test_osx_no_deps: build_osx
> -	# Increase the maximum number of open file descriptors on macOS
> -	echo tarantool | sudo -S echo "Initialize sudo for the next commands..." :
> -	sudo sysctl -w kern.maxfiles=${MAX_FILES} || echo tarantool | sudo -S sysctl -w kern.maxfiles=${MAX_FILES} || :
> -	sysctl -w kern.maxfiles || :
> -	sudo sysctl -w kern.maxfilesperproc=${MAX_FILES} || echo tarantool | sudo -S sysctl -w kern.maxfilesperproc=${MAX_FILES} || :
> -	sysctl -w kern.maxfilesperproc || :
> -	sudo launchctl limit maxfiles ${MAX_FILES} || echo tarantool | sudo -S launchctl limit maxfiles ${MAX_FILES} || :
> -	launchctl limit maxfiles || :
> -	sudo ulimit -S -n ${MAX_FILES} || echo tarantool | sudo -S ulimit -S -n ${MAX_FILES} || :
> -	ulimit -S -n || :
> -	sudo ulimit -H -n ${MAX_FILES} || echo tarantool | sudo -S ulimit -H -n ${MAX_FILES} || :
> -	ulimit -H -n
> -	# temporary excluded replication/ suite with some tests from other suites by issues #4357 and #4370
> -	cd test && ./test-run.py --force $(TEST_RUN_EXTRA_PARAMS) \
> -		app/ app-tap/ box/ box-py/ box-tap/ engine/ engine_long/ long_run-py/ luajit-tap/ \
> -		replication-py/ small/ sql/ sql-tap/ swim/ unit/ vinyl/ wal_off/ xlog/ xlog-py/
> +	# Limits: Increase the maximum number of open file descriptors on macOS:
> +	#   Travis-ci needs the "ulimit -n <value>" call
> +	#   Gitlab-ci needs the "launchctl limit maxfiles <value>" call
> +	# Also gitlib-ci needs the password to change the limits, while
> +	# travis-ci runs under root user. Limit setup must be in the same
> +	# call as tests runs call.
> +	# Tests: Temporary excluded replication/ suite with some tests from other suites by issues #4357 and #4370
> +	sudo launchctl limit maxfiles ${MAX_FILES} || echo tarantool | sudo -S launchctl limit maxfiles ${MAX_FILES} || true ; \
> +		launchctl limit maxfiles || true ; \
> +		ulimit -n ${MAX_FILES} || echo tarantool | sudo -S ulimit ${MAX_FILES} || true ; ulimit -n ; \
> +		cd test && ./test-run.py --force $(TEST_RUN_EXTRA_PARAMS) \
> +			app/ app-tap/ box/ box-py/ box-tap/ engine/ engine_long/ long_run-py/ luajit-tap/ \
> +			replication-py/ small/ sql/ sql-tap/ swim/ unit/ vinyl/ wal_off/ xlog/ xlog-py/

rlimit (it is the thing that ulimit command get/set) is a process
property (which is inherited by its childs). When you do `sudo ulimit -n
${MAX_FILES}` it runs ulimit under childs process and it does not alter
parent's resource limits. ulimit is the bash builtin, not an external
command: it changes resource limits for processes that bash starts.

Also I dislike `echo password | sudo <...>` construction, because root
password in our images is more our internal infrastructure thing.

>  
>  test_osx: deps_osx test_osx_no_deps
>  
> diff --git a/.travis.yml b/.travis.yml
> index 61c887296..21a69d15a 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -17,8 +17,9 @@ cache:
>  git:
>      depth: 100500
>  
> -env: >
> -    TEST_RUN_EXTRA_PARAMS="-j 1"
> +env:
> +    global:
> +      - TEST_RUN_EXTRA_PARAMS="-j 1"
>  
>  jobs:
>      include:
> -- 
> 2.17.1
> 

[tarantool-patches] Re: [PATCH v1] travis-ci: fix OSX max files limits

[Alexander Turenko]

> >On Tue, Jul 23, 2019 at 10:51:09AM +0300, Alexander V. Tikhonov wrote:
> >> Increased the maximum number of open file descriptors on macOS:
> >>   Travis-ci needed the "ulimit -n <value>" call
> >>   Gitlab-ci needed the "launchctl limit maxfiles <value>" call
> >> Also gitlib-ci needed the password to change the limits, while
> >> travis-ci run under root user. Limit setup set in the same
> >> call as tests runs call.
> >
> >Are you know what is a reason of the difference between travis-ci and
> >gitlab-ci? I don't insist, but if it is known, it worth to mention it.

> Right, there is the difference that we use different images, travis-ci uses
> its own image, while gitlab-ci we provided the needed image manually.
> Also the travis-ci runs under root user, while gitlab-ci we know the root
> password and can run sudo command with the password. So I've added
> more comments into commit log about it.

I think it is better to implement non-password sudo in our images and
use it in the same way for both testing pipelines. However if it is
complex thing, okay, let it be.