* [Tarantool-patches] [PATCH v1] Add S3 package checker
@ 2020-03-23 7:20 Alexander V. Tikhonov
2020-03-23 9:04 ` Sergey Bronnikov
0 siblings, 1 reply; 3+ messages in thread
From: Alexander V. Tikhonov @ 2020-03-23 7:20 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches
Created Dockerfile based script to check packages available at the S3
repository. Set this script to be run just after the package creation.
Follows up #3380
---
Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-check-packages-s3-full-ci
Issue: https://github.com/tarantool/tarantool/issues/3380
Gitlab: https://gitlab.com/tarantool/tarantool/pipelines/128701075
.gitlab.mk | 1 +
tools/Dockerfile.s3 | 23 +++++++++++++++++
tools/check_package.sh | 57 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 81 insertions(+)
create mode 100644 tools/Dockerfile.s3
create mode 100755 tools/check_package.sh
diff --git a/.gitlab.mk b/.gitlab.mk
index b39c5c651..5053b914f 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -127,6 +127,7 @@ deploy: package
echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
./tools/update_repo.sh -o=${OS} -d=${DIST} \
-b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
+ ./tools/check_package.sh "" ${OS}:${DIST} ${BUCKET}
if git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null ; then \
./tools/update_repo.sh -o=${OS} -d=${DIST} \
-b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build ; \
diff --git a/tools/Dockerfile.s3 b/tools/Dockerfile.s3
new file mode 100644
index 000000000..338cf6926
--- /dev/null
+++ b/tools/Dockerfile.s3
@@ -0,0 +1,23 @@
+ARG IMG
+FROM ${IMG}
+
+ARG PKG_TOOL
+RUN ${PKG_TOOL} update -y >1.log 2>&1 || ( cat 1.log && false )
+RUN ${PKG_TOOL} install -y curl >2.log 2>&1 || ( cat 2.log && false )
+
+RUN curl -L https://tarantool.io/installer.sh >installer.sh 2>3.log || ( cat 3.log && false )
+RUN chmod a+x installer.sh
+RUN sed "s#/bin/bash#/bin/bash -x#g" -i installer.sh
+
+# ./installer.sh is the external tool which return status is not
+# expectable and can be false because of its additional features,
+# all that is needed for us from the script just to install Tarantool,
+# NOTE: found that on centos:8 script fails but on rerun passes
+ARG REL
+RUN VER=${REL} ./installer.sh >4.log 2>&1 || ( VER=${REL} ./installer.sh >>4.log 2>&1 || ( cat 4.log && false ))
+ARG TNT_VER
+RUN CVER=$(tarantool -V | head -n 1 | awk '{print $2}' | sed 's#-#.#g') && \
+ echo "Tarantool version:" && \
+ echo "Expected: ${TNT_VER}" && \
+ echo "Returned: ${CVER}" && \
+ if ! echo "${CVER}" | grep -e "^${TNT_VER}" >/dev/null ; then false ; fi
diff --git a/tools/check_package.sh b/tools/check_package.sh
new file mode 100755
index 000000000..b3fdcdcda
--- /dev/null
+++ b/tools/check_package.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+set -e
+
+ver="$1"
+imgs="$2"
+rels="$3"
+
+if [ "$ver" == "" ]; then
+ # some OS like CentOS/Fedora use only 6 hexadecimal
+ # digits as the abbreviated object name
+ ver=$(git describe --abbrev=6 | sed 's#-#.#g')
+fi
+
+if [ "$imgs" == "" ]; then
+ for dist in 14.04 16.04 18.04 19.04 19.10 ; do imgs="$imgs ubuntu:$dist" ; done
+ for dist in jessie stretch buster ; do imgs="$imgs debian:$dist" ; done
+ for dist in 6 7 8 ; do imgs="$imgs centos:$dist" ; done
+ for dist in 28 29 30 31 ; do imgs="$imgs fedora:$dist" ; done
+fi
+
+imgs=$(echo $imgs | sed "s#el:#centos:#g")
+
+if [ "$rels" == "" ]; then
+ rels="1.10 2.1 2.2 2.3 2.4"
+fi
+
+cwd=`dirname $0`
+
+run_docker () {
+ VER=$1
+ IMG=$2
+ REL=$3
+
+ PKG_TOOL=yum
+ if [[ $IMG =~ .*ubuntu.* || $IMG =~ .*debian.* ]]; then
+ PKG_TOOL=apt
+ fi
+ echo "Testing $REL Tarantool on $IMG image ..."
+ newimg="check-s3_"`echo $IMG | sed 's#:#-#g'`"_$REL"
+ # repository update command must always be rerun to be able
+ # to get the new available versions of the Tarantool, so the
+ # flag "--no-cache" must be set
+ docker build \
+ --network=host \
+ --no-cache \
+ --build-arg PKG_TOOL=$PKG_TOOL \
+ --build-arg REL=$REL \
+ --build-arg IMG=$IMG \
+ --build-arg TNT_VER=$VER \
+ -f $cwd/Dockerfile.s3 -t $newimg .
+}
+
+for rel in $rels ; do
+ for img in $imgs ; do
+ run_docker $ver $img $rel
+ done
+done
--
2.17.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Tarantool-patches] [PATCH v1] Add S3 package checker
2020-03-23 7:20 [Tarantool-patches] [PATCH v1] Add S3 package checker Alexander V. Tikhonov
@ 2020-03-23 9:04 ` Sergey Bronnikov
2020-04-15 11:53 ` Alexander Tikhonov
0 siblings, 1 reply; 3+ messages in thread
From: Sergey Bronnikov @ 2020-03-23 9:04 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
Hello,
I propose to check script tools/check_package.sh with shellcheck.
Right now it reports a number of warnings.
Sergey
On 10:20 Mon 23 Mar , Alexander V. Tikhonov wrote:
> Created Dockerfile based script to check packages available at the S3
> repository. Set this script to be run just after the package creation.
>
> Follows up #3380
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-check-packages-s3-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/3380
> Gitlab: https://gitlab.com/tarantool/tarantool/pipelines/128701075
>
> .gitlab.mk | 1 +
> tools/Dockerfile.s3 | 23 +++++++++++++++++
> tools/check_package.sh | 57 ++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 81 insertions(+)
> create mode 100644 tools/Dockerfile.s3
> create mode 100755 tools/check_package.sh
>
> diff --git a/.gitlab.mk b/.gitlab.mk
> index b39c5c651..5053b914f 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -127,6 +127,7 @@ deploy: package
> echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
> ./tools/update_repo.sh -o=${OS} -d=${DIST} \
> -b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
> + ./tools/check_package.sh "" ${OS}:${DIST} ${BUCKET}
> if git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null ; then \
> ./tools/update_repo.sh -o=${OS} -d=${DIST} \
> -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build ; \
> diff --git a/tools/Dockerfile.s3 b/tools/Dockerfile.s3
> new file mode 100644
> index 000000000..338cf6926
> --- /dev/null
> +++ b/tools/Dockerfile.s3
> @@ -0,0 +1,23 @@
> +ARG IMG
> +FROM ${IMG}
> +
> +ARG PKG_TOOL
> +RUN ${PKG_TOOL} update -y >1.log 2>&1 || ( cat 1.log && false )
> +RUN ${PKG_TOOL} install -y curl >2.log 2>&1 || ( cat 2.log && false )
> +
> +RUN curl -L https://tarantool.io/installer.sh >installer.sh 2>3.log || ( cat 3.log && false )
> +RUN chmod a+x installer.sh
> +RUN sed "s#/bin/bash#/bin/bash -x#g" -i installer.sh
> +
> +# ./installer.sh is the external tool which return status is not
> +# expectable and can be false because of its additional features,
> +# all that is needed for us from the script just to install Tarantool,
> +# NOTE: found that on centos:8 script fails but on rerun passes
> +ARG REL
> +RUN VER=${REL} ./installer.sh >4.log 2>&1 || ( VER=${REL} ./installer.sh >>4.log 2>&1 || ( cat 4.log && false ))
> +ARG TNT_VER
> +RUN CVER=$(tarantool -V | head -n 1 | awk '{print $2}' | sed 's#-#.#g') && \
> + echo "Tarantool version:" && \
> + echo "Expected: ${TNT_VER}" && \
> + echo "Returned: ${CVER}" && \
> + if ! echo "${CVER}" | grep -e "^${TNT_VER}" >/dev/null ; then false ; fi
> diff --git a/tools/check_package.sh b/tools/check_package.sh
> new file mode 100755
> index 000000000..b3fdcdcda
> --- /dev/null
> +++ b/tools/check_package.sh
> @@ -0,0 +1,57 @@
> +#!/bin/bash
> +set -e
> +
> +ver="$1"
> +imgs="$2"
> +rels="$3"
> +
> +if [ "$ver" == "" ]; then
> + # some OS like CentOS/Fedora use only 6 hexadecimal
> + # digits as the abbreviated object name
> + ver=$(git describe --abbrev=6 | sed 's#-#.#g')
> +fi
> +
> +if [ "$imgs" == "" ]; then
> + for dist in 14.04 16.04 18.04 19.04 19.10 ; do imgs="$imgs ubuntu:$dist" ; done
> + for dist in jessie stretch buster ; do imgs="$imgs debian:$dist" ; done
> + for dist in 6 7 8 ; do imgs="$imgs centos:$dist" ; done
> + for dist in 28 29 30 31 ; do imgs="$imgs fedora:$dist" ; done
> +fi
> +
> +imgs=$(echo $imgs | sed "s#el:#centos:#g")
> +
> +if [ "$rels" == "" ]; then
> + rels="1.10 2.1 2.2 2.3 2.4"
> +fi
> +
> +cwd=`dirname $0`
> +
> +run_docker () {
> + VER=$1
> + IMG=$2
> + REL=$3
> +
> + PKG_TOOL=yum
> + if [[ $IMG =~ .*ubuntu.* || $IMG =~ .*debian.* ]]; then
> + PKG_TOOL=apt
> + fi
> + echo "Testing $REL Tarantool on $IMG image ..."
> + newimg="check-s3_"`echo $IMG | sed 's#:#-#g'`"_$REL"
> + # repository update command must always be rerun to be able
> + # to get the new available versions of the Tarantool, so the
> + # flag "--no-cache" must be set
> + docker build \
> + --network=host \
> + --no-cache \
> + --build-arg PKG_TOOL=$PKG_TOOL \
> + --build-arg REL=$REL \
> + --build-arg IMG=$IMG \
> + --build-arg TNT_VER=$VER \
> + -f $cwd/Dockerfile.s3 -t $newimg .
> +}
> +
> +for rel in $rels ; do
> + for img in $imgs ; do
> + run_docker $ver $img $rel
> + done
> +done
> --
> 2.17.1
>
--
sergeyb@
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Tarantool-patches] [PATCH v1] Add S3 package checker
2020-03-23 9:04 ` Sergey Bronnikov
@ 2020-04-15 11:53 ` Alexander Tikhonov
0 siblings, 0 replies; 3+ messages in thread
From: Alexander Tikhonov @ 2020-04-15 11:53 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: Oleg Piskunov, tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 4685 bytes --]
Hi Sergey, thanks for the review, I’ve added shell-checker flag and fixed found warnings.
>Понедельник, 23 марта 2020, 12:04 +03:00 от Sergey Bronnikov <sergeyb@tarantool.org>:
>
>Hello,
>
>I propose to check script tools/check_package.sh with shellcheck.
>Right now it reports a number of warnings.
>
>Sergey
>
>On 10:20 Mon 23 Mar , Alexander V. Tikhonov wrote:
>> Created Dockerfile based script to check packages available at the S3
>> repository. Set this script to be run just after the package creation.
>>
>> Follows up #3380
>> ---
>>
>> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-check-packages-s3-full-ci
>> Issue: https://github.com/tarantool/tarantool/issues/3380
>> Gitlab: https://gitlab.com/tarantool/tarantool/pipelines/128701075
>>
>> .gitlab.mk | 1 +
>> tools/Dockerfile.s3 | 23 +++++++++++++++++
>> tools/check_package.sh | 57 ++++++++++++++++++++++++++++++++++++++++++
>> 3 files changed, 81 insertions(+)
>> create mode 100644 tools/Dockerfile.s3
>> create mode 100755 tools/check_package.sh
>>
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index b39c5c651..5053b914f 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -127,6 +127,7 @@ deploy: package
>> echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
>> ./tools/update_repo.sh -o=${OS} -d=${DIST} \
>> -b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
>> + ./tools/check_package.sh "" ${OS}:${DIST} ${BUCKET}
>> if git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null ; then \
>> ./tools/update_repo.sh -o=${OS} -d=${DIST} \
>> -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build ; \
>> diff --git a/tools/Dockerfile.s3 b/tools/Dockerfile.s3
>> new file mode 100644
>> index 000000000..338cf6926
>> --- /dev/null
>> +++ b/tools/Dockerfile.s3
>> @@ -0,0 +1,23 @@
>> +ARG IMG
>> +FROM ${IMG}
>> +
>> +ARG PKG_TOOL
>> +RUN ${PKG_TOOL} update -y >1.log 2>&1 || ( cat 1.log && false )
>> +RUN ${PKG_TOOL} install -y curl >2.log 2>&1 || ( cat 2.log && false )
>> +
>> +RUN curl -L https://tarantool.io/installer.sh >installer.sh 2>3.log || ( cat 3.log && false )
>> +RUN chmod a+x installer.sh
>> +RUN sed "s#/bin/bash#/bin/bash -x#g" -i installer.sh
>> +
>> +# ./installer.sh is the external tool which return status is not
>> +# expectable and can be false because of its additional features,
>> +# all that is needed for us from the script just to install Tarantool,
>> +# NOTE: found that on centos:8 script fails but on rerun passes
>> +ARG REL
>> +RUN VER=${REL} ./installer.sh >4.log 2>&1 || ( VER=${REL} ./installer.sh >>4.log 2>&1 || ( cat 4.log && false ))
>> +ARG TNT_VER
>> +RUN CVER=$(tarantool -V | head -n 1 | awk '{print $2}' | sed 's#-#.#g') && \
>> + echo "Tarantool version:" && \
>> + echo "Expected: ${TNT_VER}" && \
>> + echo "Returned: ${CVER}" && \
>> + if ! echo "${CVER}" | grep -e "^${TNT_VER}" >/dev/null ; then false ; fi
>> diff --git a/tools/check_package.sh b/tools/check_package.sh
>> new file mode 100755
>> index 000000000..b3fdcdcda
>> --- /dev/null
>> +++ b/tools/check_package.sh
>> @@ -0,0 +1,57 @@
>> +#!/bin/bash
>> +set -e
>> +
>> +ver="$1"
>> +imgs="$2"
>> +rels="$3"
>> +
>> +if [ "$ver" == "" ]; then
>> + # some OS like CentOS/Fedora use only 6 hexadecimal
>> + # digits as the abbreviated object name
>> + ver=$(git describe --abbrev=6 | sed 's#-#.#g')
>> +fi
>> +
>> +if [ "$imgs" == "" ]; then
>> + for dist in 14.04 16.04 18.04 19.04 19.10 ; do imgs="$imgs ubuntu:$dist" ; done
>> + for dist in jessie stretch buster ; do imgs="$imgs debian:$dist" ; done
>> + for dist in 6 7 8 ; do imgs="$imgs centos:$dist" ; done
>> + for dist in 28 29 30 31 ; do imgs="$imgs fedora:$dist" ; done
>> +fi
>> +
>> +imgs=$(echo $imgs | sed "s#el:#centos:#g")
>> +
>> +if [ "$rels" == "" ]; then
>> + rels="1.10 2.1 2.2 2.3 2.4"
>> +fi
>> +
>> +cwd=`dirname $0`
>> +
>> +run_docker () {
>> + VER=$1
>> + IMG=$2
>> + REL=$3
>> +
>> + PKG_TOOL=yum
>> + if [[ $IMG =~ .*ubuntu.* || $IMG =~ .*debian.* ]]; then
>> + PKG_TOOL=apt
>> + fi
>> + echo "Testing $REL Tarantool on $IMG image ..."
>> + newimg="check-s3_"`echo $IMG | sed 's#:#-#g'`"_$REL"
>> + # repository update command must always be rerun to be able
>> + # to get the new available versions of the Tarantool, so the
>> + # flag "--no-cache" must be set
>> + docker build \
>> + --network=host \
>> + --no-cache \
>> + --build-arg PKG_TOOL=$PKG_TOOL \
>> + --build-arg REL=$REL \
>> + --build-arg IMG=$IMG \
>> + --build-arg TNT_VER=$VER \
>> + -f $cwd/Dockerfile.s3 -t $newimg .
>> +}
>> +
>> +for rel in $rels ; do
>> + for img in $imgs ; do
>> + run_docker $ver $img $rel
>> + done
>> +done
>> --
>> 2.17.1
>>
>--
>sergeyb@
--
Alexander Tikhonov
[-- Attachment #2: Type: text/html, Size: 6258 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-04-15 11:53 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-23 7:20 [Tarantool-patches] [PATCH v1] Add S3 package checker Alexander V. Tikhonov
2020-03-23 9:04 ` Sergey Bronnikov
2020-04-15 11:53 ` Alexander Tikhonov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox