Re: [Tarantool-patches] [PATCH v1] Add S3 package checker

[Sergey Bronnikov <sergeyb@tarantool.org>]

Hello,

I propose to check script tools/check_package.sh with shellcheck.
Right now it reports a number of warnings.

Sergey

On 10:20 Mon 23 Mar , Alexander V. Tikhonov wrote:
> Created Dockerfile based script to check packages available at the S3
> repository. Set this script to be run just after the package creation.
> 
> Follows up #3380
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-check-packages-s3-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/3380
> Gitlab: https://gitlab.com/tarantool/tarantool/pipelines/128701075
> 
>  .gitlab.mk             |  1 +
>  tools/Dockerfile.s3    | 23 +++++++++++++++++
>  tools/check_package.sh | 57 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 81 insertions(+)
>  create mode 100644 tools/Dockerfile.s3
>  create mode 100755 tools/check_package.sh
> 
> diff --git a/.gitlab.mk b/.gitlab.mk
> index b39c5c651..5053b914f 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -127,6 +127,7 @@ deploy: package
>  	echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
>  	./tools/update_repo.sh -o=${OS} -d=${DIST} \
>  		-b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
> +	./tools/check_package.sh "" ${OS}:${DIST} ${BUCKET}
>  	if git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null ; then \
>  		./tools/update_repo.sh -o=${OS} -d=${DIST} \
>  			-b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build ; \
> diff --git a/tools/Dockerfile.s3 b/tools/Dockerfile.s3
> new file mode 100644
> index 000000000..338cf6926
> --- /dev/null
> +++ b/tools/Dockerfile.s3
> @@ -0,0 +1,23 @@
> +ARG IMG
> +FROM ${IMG}
> +
> +ARG PKG_TOOL
> +RUN ${PKG_TOOL} update -y >1.log 2>&1 || ( cat 1.log && false )
> +RUN ${PKG_TOOL} install -y curl >2.log 2>&1 || ( cat 2.log && false )
> +
> +RUN curl -L https://tarantool.io/installer.sh >installer.sh 2>3.log || ( cat 3.log && false )
> +RUN chmod a+x installer.sh
> +RUN sed "s#/bin/bash#/bin/bash -x#g" -i installer.sh
> +
> +# ./installer.sh is the external tool which return status is not
> +# expectable and can be false because of its additional features,
> +# all that is needed for us from the script just to install Tarantool,
> +# NOTE: found that on centos:8 script fails but on rerun passes
> +ARG REL
> +RUN VER=${REL} ./installer.sh >4.log 2>&1 || ( VER=${REL} ./installer.sh >>4.log 2>&1 || ( cat 4.log && false ))
> +ARG TNT_VER
> +RUN CVER=$(tarantool -V | head -n 1 | awk '{print $2}' | sed 's#-#.#g') && \
> +    echo "Tarantool version:" && \
> +    echo "Expected: ${TNT_VER}" && \
> +    echo "Returned: ${CVER}" && \
> +    if ! echo "${CVER}" | grep -e "^${TNT_VER}" >/dev/null ; then false ; fi
> diff --git a/tools/check_package.sh b/tools/check_package.sh
> new file mode 100755
> index 000000000..b3fdcdcda
> --- /dev/null
> +++ b/tools/check_package.sh
> @@ -0,0 +1,57 @@
> +#!/bin/bash
> +set -e
> +
> +ver="$1"
> +imgs="$2"
> +rels="$3"
> +
> +if [ "$ver" == "" ]; then
> +    # some OS like CentOS/Fedora use only 6 hexadecimal
> +    # digits as the abbreviated object name
> +    ver=$(git describe --abbrev=6 | sed 's#-#.#g')
> +fi
> +
> +if [ "$imgs" == "" ]; then
> +    for dist in 14.04 16.04 18.04 19.04 19.10 ; do imgs="$imgs ubuntu:$dist" ; done
> +    for dist in jessie stretch buster ; do imgs="$imgs debian:$dist" ; done
> +    for dist in 6 7 8 ; do imgs="$imgs centos:$dist" ; done
> +    for dist in 28 29 30 31 ; do imgs="$imgs fedora:$dist" ; done
> +fi
> +
> +imgs=$(echo $imgs | sed "s#el:#centos:#g")
> +
> +if [ "$rels" == "" ]; then
> +    rels="1.10 2.1 2.2 2.3 2.4"
> +fi
> +
> +cwd=`dirname $0`
> +
> +run_docker () {
> +    VER=$1
> +    IMG=$2
> +    REL=$3
> +
> +    PKG_TOOL=yum
> +    if [[ $IMG =~ .*ubuntu.* || $IMG =~ .*debian.* ]]; then
> +        PKG_TOOL=apt
> +    fi
> +    echo "Testing $REL Tarantool on $IMG image ..."
> +    newimg="check-s3_"`echo $IMG | sed 's#:#-#g'`"_$REL"
> +    # repository update command must always be rerun to be able
> +    # to get the new available versions of the Tarantool, so the
> +    # flag "--no-cache" must be set
> +    docker build \
> +        --network=host \
> +        --no-cache \
> +        --build-arg PKG_TOOL=$PKG_TOOL \
> +        --build-arg REL=$REL \
> +        --build-arg IMG=$IMG \
> +        --build-arg TNT_VER=$VER \
> +        -f $cwd/Dockerfile.s3 -t $newimg .
> +}
> +
> +for rel in $rels ; do
> +    for img in $imgs ; do
> +        run_docker $ver $img $rel
> +    done
> +done
> -- 
> 2.17.1
> 

-- 
sergeyb@