<HTML><BODY><div class="js-helper js-readmsg-msg">
<style type="text/css"></style>
<div>
<base target="_self" href="https://e.mail.ru/">
<div id="style_15762200970122529467_BODY"><div class="class_1576245818">
<br><br><br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;">
Четверг, 5 декабря 2019, 15:23 +03:00 от Igor Munkin <<a href="mailto:imun@tarantool.org">imun@tarantool.org</a>>:<br><br><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix">Sasha,<br><br>
Thanks for the patch. It looks much better now (except the fact I would<br>
prefer to see it in Perl rather in Bash), but let's polish it a bit.<br>
Consider my comments below.<br><br>
Moreover, this patch is a complex one, so please split your further work<br>
into several blocks to be committed separately.<br></div></div></div></div></blockquote>Ok, sure<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
On 12.11.19, Alexander V. Tikhonov wrote:<br>
> Added ability to store packages additionaly at MCS S3.<br>
> The target idea was to add the new way of packages creation at MCS S3,<br>
> which temporary duplicates the packaging at PackageCloud by Packpack<br>
> tool. Also it was needed to pack the binaries in the native way of the<br>
> each packing OS style. It was created standalone script for adding the<br>
> packages binaries/sources to MCS S3 at DEB either RPM repositories:<br>
> 'tools/add_pack_s3_repo.sh'<br>
> Common parts of the script are:<br>
> - create new meta files for the new binaries<br>
> - copy new binaries to MCS S3<br>
> - get previous metafiles from MCS S3 and merge the new meta data for<br>
> the new binaries<br>
> - update the meta files at MCS S3<br>
> Different parts:<br>
> - DEB script part based on reprepro external tool, also it works separately<br>
> only on OS versions level - it means that meta data it updates for all<br>
> Distritbutions together.<br>
> - RPM script part based on createrepo external tool, also it works<br>
> separately for OS/Release level - it means that meta data it updates<br>
> for all Releases separately.<br>
> <br>
> Closes #3380<br>
> ---<br>
> <br>
> Github: <a href="https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci" target="_blank" rel=" noopener noreferrer">https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci</a><br>
> Issue: <a href="https://github.com/tarantool/tarantool/issues/3380" target="_blank" rel=" noopener noreferrer">https://github.com/tarantool/tarantool/issues/3380</a><br><br>
Please add the changelog per version in further patchesets.</div></div></div></div></blockquote>Ok<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> <br>
> .gitlab-ci.yml | 5 +-<br>
> .gitlab.mk | 20 +-<br>
> .travis.mk | 41 ++--<br>
> tools/add_pack_s3_repo.sh | 493 ++++++++++++++++++++++++++++++++++++++<br>
> 4 files changed, 533 insertions(+), 26 deletions(-)<br>
> create mode 100755 tools/add_pack_s3_repo.sh<br>
> <br>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml<br>
> index cf13c382e..4dcaf9cd3 100644<br>
> --- a/.gitlab-ci.yml<br>
> +++ b/.gitlab-ci.yml<br>
> @@ -231,7 +231,10 @@ debian_10:<br>
> DIST: 'buster'<br>
> <br>
> static_build:<br>
> - <<: *deploy_test_definition<br>
> + <<: *release_only_definition<br>
> + stage: test<br>
> + tags:<br>
> + - deploy_test<br>
> variables:<br>
> RUN_TESTS: 'ON'<br>
> script:<br>
> diff --git a/.gitlab.mk b/.gitlab.mk<br>
> index 48a92e518..64664c64f 100644<br>
> --- a/.gitlab.mk<br>
> +++ b/.gitlab.mk<br>
> @@ -98,13 +98,27 @@ vms_test_%:<br>
> vms_shutdown:<br>
> VBoxManage controlvm ${VMS_NAME} poweroff<br>
> <br>
> -# ########################<br>
> -# Build RPM / Deb packages<br>
> -# ########################<br>
> +# ###########################<br>
> +# Sources tarballs & packages<br>
> +# ###########################<br>
> +<br>
> +# Push alpha and beta versions to <major>x bucket (say, 2x),<br>
> +# stable to <major>.<minor> bucket (say, 2.2).<br>
> +GIT_DESCRIBE=$(shell git describe HEAD)<br>
> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))<br>
> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))<br>
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)<br>
> +ifeq ($(MINOR_VERSION),0)<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> +endif<br>
> +ifeq ($(MINOR_VERSION),1)<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> +endif<br>
> <br>
> package: git_submodule_update<br>
> git clone <a href="https://github.com/packpack/packpack.git" target="_blank" rel=" noopener noreferrer">https://github.com/packpack/packpack.git</a> packpack<br>
> PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack<br>
> + ./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build<br>
> <br>
> # ############<br>
> # Static build<br>
> diff --git a/.travis.mk b/.travis.mk<br>
> index 42969ff56..a85f71ced 100644<br>
> --- a/.travis.mk<br>
> +++ b/.travis.mk<br>
> @@ -8,10 +8,6 @@ MAX_FILES?=65534<br>
> <br>
> all: package<br>
> <br>
> -package:<br>
> - git clone <a href="https://github.com/packpack/packpack.git" target="_blank" rel=" noopener noreferrer">https://github.com/packpack/packpack.git</a> packpack<br>
> - ./packpack/packpack<br>
> -<br>
> test: test_$(TRAVIS_OS_NAME)<br>
> <br>
> # Redirect some targets via docker<br>
> @@ -176,34 +172,35 @@ test_freebsd_no_deps: build_freebsd<br>
> <br>
> test_freebsd: deps_freebsd test_freebsd_no_deps<br>
> <br>
> -####################<br>
> -# Sources tarballs #<br>
> -####################<br>
> -<br>
> -source:<br>
> - git clone <a href="https://github.com/packpack/packpack.git" target="_blank" rel=" noopener noreferrer">https://github.com/packpack/packpack.git</a> packpack<br>
> - TARBALL_COMPRESSOR=gz packpack/packpack tarball<br>
> +###############################<br>
> +# Sources tarballs & packages #<br>
> +###############################<br>
> <br>
> # Push alpha and beta versions to <major>x bucket (say, 2x),<br>
> # stable to <major>.<minor> bucket (say, 2.2).<br>
> -ifeq ($(TRAVIS_BRANCH),master)<br>
> GIT_DESCRIBE=$(shell git describe HEAD)<br>
> MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))<br>
> MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))<br>
> -else<br>
> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))<br>
> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))<br>
> -endif<br>
> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src<br>
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)<br>
> ifeq ($(MINOR_VERSION),0)<br>
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> endif<br>
> ifeq ($(MINOR_VERSION),1)<br>
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> endif<br>
> <br>
> +packpack_prepare:<br>
> + git clone <a href="https://github.com/packpack/packpack.git" target="_blank" rel=" noopener noreferrer">https://github.com/packpack/packpack.git</a> packpack<br>
> +<br>
> +package: packpack_prepare<br>
> + ./packpack/packpack<br>
> +<br>
> +source: packpack_prepare<br>
> + TARBALL_COMPRESSOR=gz packpack/packpack tarball<br>
> +<br>
> source_deploy:<br>
> pip install awscli --user<br>
> - aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \<br>
> - cp build/*.tar.gz "s3://${BUCKET}/" \<br>
> - --acl public-read<br>
> + for tarball in `ls build/*.tar.gz 2>/dev/null` ; \<br>
> + aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \<br>
> + cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \<br>
> + --acl public-read<br>
> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh<br>
> new file mode 100755<br>
> index 000000000..2316b9015<br>
> --- /dev/null<br>
> +++ b/tools/add_pack_s3_repo.sh<br>
> @@ -0,0 +1,493 @@<br>
> +#!/bin/bash<br>
> +set -e<br>
> +<br>
> +rm_file='rm -f'<br>
> +rm_dir='rm -rf'<br>
> +mk_dir='mkdir -p'<br>
> +<br>
> +alloss='ubuntu debian centos el fedora'<br><br>
General comments for the script:<br>
* as discussed offline with you and Sasha Tu., "centos" alias is excess<br>
and it can be removed everywhere within this script.</div></div></div></div></blockquote>"centos" alias completely removed.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
* consider using $(...) instead of `...` to improve readability.</div></div></div></div></blockquote>Changed.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
* typo: adjust the tab misusage for indentation.<br></div></div></div></div></blockquote>Fixed.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> +<br>
> +get_os_dists()<br>
> +{<br>
> + os=$1<br>
> + alldists=<br>
> +<br>
> + if [ "$os" == "ubuntu" ]; then<br>
> + alldists='trusty xenial cosmic disco bionic eoan'<br>
> + elif [ "$os" == "debian" ]; then<br>
> + alldists='jessie stretch buster'<br>
> + elif [ "$os" == "centos" -o "$os" == "el" ]; then<br>
> + alldists='6 7 8'<br>
> + elif [ "$os" == "fedora" ]; then<br>
> + alldists='27 28 29 30 31'<br>
> + fi<br>
> +<br>
> + echo "$alldists"<br>
> +}<br>
> +<br>
> +ws_prefix=/tmp/tarantool_repo_s3<br><br>
Please move the prior line to the beginning of the script to organize a<br>
single place with the "constants" and "defaults".</div></div></div></div></blockquote>Ok, done, as much as possible not to make the code harder to understand.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +create_lockfile()<br>
> +{<br>
> + lockfile -l 1000 $1<br>
> +}<br>
> +<br>
> +usage()<br>
> +{<br>
> + cat <<EOF<br>
> +Usage: $0 -b <S3 bucket> -o <OS name> -d <OS distribuition> [-p <product>] <path><br>
> +Options:<br>
> + -b|--bucket<br>
> + MCS S3 bucket which will be used for storing the packages.<br>
> + Name of the package one of the appropriate Tarantool branch:<br>
> + master: 2x<br>
> + 2.3: 2_3<br>
> + 2.2: 2_2<br>
> + 1.10: 1_10<br>
> + -o|--os<br>
> + OS to be checked, one of the list (NOTE: centos == el):<br>
> + $alloss<br>
> + -d|--distribution<br>
> + Distribution appropriate to the given OS:<br>
> +EOF<br>
> + for os in $alloss ; do<br>
> + echo " $os: <"`get_os_dists $os`">"<br>
> + done<br>
> + cat <<EOF<br>
> + -p|--product<br>
> + Product name to be packed with, default name is 'tarantool'<br>
> + -h|--help<br>
> + Usage help message<br><br>
Minor: arguments description usually goes prior to options description.</div></div></div></div></blockquote>Corrected, help message improved.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + <path><br>
> + Path points to the directory with deb/prm packages to be used.<br>
> + Script can be used in one of 2 modes:<br>
> + - path with binaries packages for a single distribution<br>
> + - path with 'pool' directory with APT repository (only: debian|ubuntu)<br>
> +EOF<br>
> +}<br>
> +<br>
> +for i in "$@"<br>
> +do<br>
> +case $i in<br>
> + -b=*|--bucket=*)<br>
> + branch="${i#*=}"<br>
> + if [ "$branch" != "2x" -a "$branch" != "2_3" -a "$branch" != "2_2" -a "$branch" != "1_10" ]; then<br><br>
grep seems to be more convenient here<br>
| echo "$branch" | grep -qvP '^(1_10|2(x|_[2-4]))$'<br></div></div></div></div></blockquote>Ok, changed.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> + echo "ERROR: bucket '$branch' is not supported"<br>
> + usage<br>
> + exit 1<br>
> + fi<br>
> + shift # past argument=value<br>
> + ;;<br>
> + -o=*|--os=*)<br>
> + os="${i#*=}"<br>
> + if [ "$os" == "el" ]; then<br>
> + os=centos<br>
> + fi<br>
> + if ! echo $alloss | grep -F -q -w $os ; then<br>
> + echo "ERROR: OS '$os' is not supported"<br>
> + usage<br>
> + exit 1<br>
> + fi<br>
> + shift # past argument=value<br>
> + ;;<br>
> + -d=*|--distribution=*)<br>
> + DIST="${i#*=}"<br>
> + shift # past argument=value<br><br>
I guess DIST can be validated right here like other variables above<br>
since it is a mandatory one. I see no need in get_os_dists subroutine,<br>
you can just initialize alldists at the beginning (like, alloss) and use<br>
it instead of "qx"-ing the corresponding function.<br></div></div></div></div></blockquote>Updated help message routine to show that the DIST is not the mandatory option.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
Minor: IMHO the naming should be consistent (DIST but os).</div></div></div></div></blockquote>Renamed distribution variables<br>dist -> loop_dist to show that the variable is the loop changing<br>DIST -> option_dist to show that the variable had come from the options<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + ;;<br>
> + -p=*|--product=*)<br>
> + product="${i#*=}"<br>
> + shift # past argument=value<br>
> + ;;<br>
> + -h|--help)<br>
> + usage<br>
> + exit 0<br>
> + ;;<br>
> + *)<br>
> + repo="${i#*=}"<br>
> + pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null<br>
> + shift # past argument=value<br>
> + ;;<br>
> +esac<br>
> +done<br>
> +<br>
> +# check that all needed options were set<br>
> +if [ "$branch" == "" ]; then<br>
> + echo "ERROR: need to set -b|--bucket bucket option, check usage"<br>
> + usage<br>
> + exit 1<br>
> +fi<br>
> +if [ "$os" == "" ]; then<br>
> + echo "ERROR: need to set -o|--os OS name option, check usage"<br>
> + usage<br>
> + exit 1<br>
> +fi<br>
> +alldists=`get_os_dists $os`<br>
> +if [ -n "$DIST" ] && ! echo $alldists | grep -F -q -w $DIST ; then<br>
> + echo "ERROR: set distribution at options '$DIST' not found at supported list '$alldists'"<br>
> + usage<br>
> + exit 1<br>
> +fi<br>
> +<br>
> +# set the path with binaries<br>
> +product=$product<br><br>
I don't get this line, it seems to be an excess one, but I'm totally not<br>
a bash master.<br></div></div></div></div></blockquote>Fixed, just the obvious code line.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> +if [ "$product" == "" ]; then<br>
> + product=tarantool<br>
> +fi<br><br>
Consider moving defaults initialization prior to arguments parsing to<br>
avoid branching like the one above.</div></div></div></div></blockquote>Moved product and repo before the options parser<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +proddir=`echo $product | head -c 1`<br>
> +<br>
> +# set the path with binaries<br>
> +if [ "$repo" == "" ]; then<br>
> + repo=.<br>
> +fi<br><br>
Please consider the comment above related to defaults initialization.</div></div></div></div></blockquote>Moved product and repo before the options parser<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +<br>
> +aws='aws --endpoint-url <a href="https://hb.bizmrg.com" target="_blank" rel=" noopener noreferrer">https://hb.bizmrg.com</a>'<br><br>
Minor: I guess you can append s3 literal to the aws variable and use<br>
| $aws cp --acl public-read $deb $s3/$locdeb<br>
instead of<br>
| $aws s3 cp --acl public-read $deb $s3/$locdeb</div></div></div></div></blockquote>Ok, corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +s3="s3://tarantool_repo/$branch/$os"<br>
> +<br>
> +# The 'pack_deb' function especialy created for DEB packages. It works<br>
> +# with DEB packing OS like Ubuntu, Debian. It is based on globaly known<br>
> +# tool 'reprepro' from:<br>
> +# <a href="https://wiki.debian.org/DebianRepository/SetupWithReprepro" target="_blank" rel=" noopener noreferrer">https://wiki.debian.org/DebianRepository/SetupWithReprepro</a><br>
> +# This tool works with complete number of distributions of the given OS.<br>
> +# Result of the routine is the debian package for APT repository with<br>
> +# file structure equal to the Debian/Ubuntu:<br>
> +# <a href="http://ftp.am.debian.org/debian/pool/main/t/tarantool/" target="_blank" rel=" noopener noreferrer">http://ftp.am.debian.org/debian/pool/main/t/tarantool/</a><br>
> +# <a href="http://ftp.am.debian.org/ubuntu/pool/main/t/" target="_blank" rel=" noopener noreferrer">http://ftp.am.debian.org/ubuntu/pool/main/t/</a><br>
> +function pack_deb {<br>
> + # we need to push packages into 'main' repository only<br>
> + component=main<br>
> +<br>
> + # debian has special directory 'pool' for packages<br>
> + debdir=pool<br>
> +<br>
> + # get packages from pointed location either mirror path<br>
> + if [ "$repo" == "" ] ; then<br>
> + repo=/var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os<br>
> + fi<br><br>
Maybe I misread the logic above, but the condition above seems to be<br>
always false.</div></div></div></div></blockquote>Right, this path better to mention in Usage routine instead of default setup.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + if [ ! -d $repo/$debdir ] && ( [ "$DIST" == "" ] || ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ) ; then<br>
> + echo "ERROR: Current '$repo' path doesn't have any of the following:"<br>
> + echo "Usage with set distribuition with option '-d' and packages: $0 [path with *.deb *.dsc *.tar.*z files]"<br>
> + echo "Usage with repositories: $0 [path to repository with '$debdir' subdirectory]"<br><br>
Heredoc seems to be more convenient here.</div></div></div></div></blockquote>Removed extra message and set usage call<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + exit 1<br>
> + fi<br>
> +<br><br>
As discussed offline let's try to move the code below to a generic<br>
function with a OS-specific handler (e.g. <os>_prep) to be executed<br>
within.</div></div></div></div></blockquote>Ok, created prepare_ws routine<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + # temporary lock the publication to the repository<br>
> + ws=${ws_prefix}_${branch}_${os}<br>
> + ws_lockfile=${ws}.lock<br>
> + create_lockfile $ws_lockfile<br>
> +<br>
> + # create temporary workspace with repository copy<br>
> + $rm_dir $ws<br>
> + $mk_dir $ws<br>
> +<br>
> + # script works in one of 2 modes:<br>
> + # - path with binaries packages for a single distribution<br>
> + # - path with 'pool' directory with APT repository<br>
> + if [ "$DIST" != "" ] && ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ; then<br><br>
Minor: consider splitting the condition above into two lines.</div></div></div></div></blockquote>Corrected all long lines<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + # copy single distribution with binaries packages<br>
> + repopath=$ws/pool/${DIST}/main/$proddir/$product<br><br>
I guess you missed to s/main/$component/ here.<br></div></div></div></div></blockquote>Right - corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> + $mk_dir ${repopath}<br>
> + cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.<br>
> + elif [ -d $repo/$debdir ]; then<br>
> + # copy 'pool' directory with APT repository<br>
> + cp -rf $repo/$debdir $ws/.<br>
> + else<br>
> + echo "ERROR: neither distribution option '-d' with files $repo/*.deb $repo/*.dsc $repo/*.tar.*z set nor '$repo/$debdir' path found"<br>
> + usage<br>
> + $rm_file $wslock<br>
> + exit 1<br>
> + fi<br>
> + cd $ws<br><br>
Minor: I can't find the corresponding cd to the previous directory, so<br>
it can break a post processing.</div></div></div></div></blockquote>Actually there are only absolute paths in use, anyway I don't see any use to save the previous directories.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +<br>
> + # create the configuration file for 'reprepro' tool<br>
> + confpath=$ws/conf<br>
> + $rm_dir $confpath<br>
> + $mk_dir $confpath<br>
> +<br>
> + for dist in $alldists ; do<br>
> + cat <<EOF >>$confpath/distributions<br>
> +Origin: Tarantool<br>
> +Label: tarantool.org<br>
> +Suite: stable<br>
> +Codename: $dist<br>
> +Architectures: amd64 source<br>
> +Components: main<br><br>
I guess you missed to s/main/$component/ here.</div></div></div></div></blockquote>Right - corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +Description: Unofficial Ubuntu Packages maintained by Tarantool<br>
> +SignWith: 91B625E5<br>
> +DebIndices: Packages Release . .gz .bz2<br>
> +UDebIndices: Packages . .gz .bz2<br>
> +DscIndices: Sources Release .gz .bz2<br>
> +<br>
> +EOF<br>
> +done<br><br>
Typo: adjust the indentation.</div></div></div></div></blockquote>Corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +<br>
> + # create standalone repository with separate components<br>
> + for dist in $alldists ; do<br>
> + echo =================== DISTRIBUTION: $dist =========================<br>
> + updated_deb=0<br>
> + updated_dsc=0<br>
> +<br>
> + # 1(binaries). use reprepro tool to generate Packages file<br>
> + for deb in $ws/$debdir/$dist/$component/*/*/*.deb ; do<br>
> + [ -f $deb ] || continue<br>
> + locdeb=`echo $deb | sed "s#^$ws\/##g"`<br><br>
Minor: escaping is excess here, since you use a different char for<br>
separator. Please consider this remark for all script.</div></div></div></div></blockquote>Fixed 2 places.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + echo "DEB: $deb"<br>
> + # register DEB file to Packages file<br>
> + reprepro -Vb . includedeb $dist $deb<br>
> + # reprepro copied DEB file to local component which is not needed<br>
> + $rm_dir $debdir/$component<br>
> + # to have all packages avoid reprepro set DEB file to its own registry<br>
> + $rm_dir db<br>
> + # copy Packages file to avoid of removing by the new DEB version<br>
> + for packages in dists/$dist/$component/binary-*/Packages ; do<br>
> + if [ ! -f $packages.saved ] ; then<br>
> + # get the latest Packages file from S3<br>
> + $aws s3 ls "$s3/$packages" 2>/dev/null && \<br>
> + $aws s3 cp --acl public-read \<br>
> + "$s3/$packages" $packages.saved || \<br><br>
Minor: IMHO, splitting the previous command into two lines makes it less<br>
readable. However, feel free to ignore this note.<br></div></div></div></div></blockquote>Corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> + touch $packages.saved<br>
> + fi<br>
> + # check if the DEB file already exists in Packages from S3<br>
> + if grep "^`grep "^SHA256: " $packages`$" $packages.saved ; then<br>
> + echo "WARNING: DEB file already registered in S3!"<br>
> + continue<br>
> + fi<br>
> + # store the new DEB entry<br>
> + cat $packages >>$packages.saved<br>
> + # save the registered DEB file to S3<br>
> + $aws s3 cp --acl public-read $deb $s3/$locdeb<br>
> + updated_deb=1<br>
> + done<br>
> + done<br>
> +<br><br>
I see this part is quite similar to the corresponding one above. Please<br>
consider to move it to a separate generic function with a source/binary<br>
specific handler to be executed within.<br></div></div></div></div></blockquote>Created standalone routine for metadata Packages and Sources files update<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br>
> + # 1(sources). use reprepro tool to generate Sources file<br>
> + for dsc in $ws/$debdir/$dist/$component/*/*/*.dsc ; do<br>
> + [ -f $dsc ] || continue<br>
> + locdsc=`echo $dsc | sed "s#^$ws\/##g"`<br>
> + echo "DSC: $dsc"<br>
> + # register DSC file to Sources file<br>
> + reprepro -Vb . includedsc $dist $dsc<br>
> + # reprepro copied DSC file to component which is not needed<br>
> + $rm_dir $debdir/$component<br>
> + # to have all sources avoid reprepro set DSC file to its own registry<br>
> + $rm_dir db<br>
> + # copy Sources file to avoid of removing by the new DSC version<br>
> + sources=dists/$dist/$component/source/Sources<br>
> + if [ ! -f $sources.saved ] ; then<br>
> + # get the latest Sources file from S3<br>
> + $aws s3 ls "$s3/$sources" && \<br>
> + $aws s3 cp --acl public-read "$s3/$sources" $sources.saved || \<br>
> + touch $sources.saved<br>
> + fi<br>
> + # WORKAROUND: unknown why, but reprepro doesn`t save the Sources file<br>
> + gunzip -c $sources.gz >$sources<br>
> + # check if the DSC file already exists in Sources from S3<br>
> + hash=`grep '^Checksums-Sha256:' -A3 $sources | \<br>
> + tail -n 1 | awk '{print $1}'`<br>
> + if grep " $hash .*\.dsc$" $sources.saved ; then<br>
> + echo "WARNING: DSC file already registered in S3!"<br>
> + continue<br>
> + fi<br>
> + # store the new DSC entry<br>
> + cat $sources >>$sources.saved<br>
> + # save the registered DSC file to S3<br>
> + $aws s3 cp --acl public-read $dsc $s3/$locdsc<br>
> + tarxz=`echo $locdsc | sed 's#\.dsc$#.debian.tar.xz#g'`<br>
> + $aws s3 cp --acl public-read $ws/$tarxz "$s3/$tarxz"<br>
> + orig=`echo $locdsc | sed 's#-1\.dsc$#.orig.tar.xz#g'`<br>
> + $aws s3 cp --acl public-read $ws/$orig "$s3/$orig"<br>
> + updated_dsc=1<br>
> + done<br>
> +<br>
> + # check if any DEB/DSC files were newly registered<br>
> + [ "$updated_deb" == "0" -a "$updated_dsc" == "0" ] && \<br>
> + continue || echo "Updating dists"<br>
> +<br>
> + # finalize the Packages file<br>
> + for packages in dists/$dist/$component/binary-*/Packages ; do<br>
> + mv $packages.saved $packages<br>
> + done<br>
> +<br>
> + # 2(binaries). update Packages file archives<br>
> + for packpath in dists/$dist/$component/binary-* ; do<br>
> + pushd $packpath<br>
> + sed "s#Filename: $debdir/$component/#Filename: $debdir/$dist/$component/#g" -i Packages<br>
> + bzip2 -c Packages >Packages.bz2<br>
> + gzip -c Packages >Packages.gz<br>
> + popd<br>
> + done<br>
> +<br>
> + # 2(sources). update Sources file archives<br>
> + pushd dists/$dist/$component/source<br>
> + sed "s#Directory: $debdir/$component/#Directory: $debdir/$dist/$component/#g" -i Sources<br>
> + bzip2 -c Sources >Sources.bz2<br>
> + gzip -c Sources >Sources.gz<br>
> + popd<br>
> +<br>
> + # 3. update checksums entries of the Packages* files in *Release files<br>
> + # NOTE: it is stable structure of the *Release files when the checksum<br>
> + # entries in it in the following way:<br>
> + # MD5Sum:<br>
> + # <checksum> <size> <file orig><br>
> + # <checksum> <size> <file debian><br>
> + # SHA1:<br>
> + # <checksum> <size> <file orig><br>
> + # <checksum> <size> <file debian><br>
> + # SHA256:<br>
> + # <checksum> <size> <file orig><br>
> + # <checksum> <size> <file debian><br>
> + # The script bellow puts 'md5' value at the 1st found file entry,<br>
> + # 'sha1' - at the 2nd and 'sha256' at the 3rd<br>
> + pushd dists/$dist<br>
> + for file in `grep " $component/" Release | awk '{print $3}' | sort -u` ; do<br>
> + sz=`stat -c "%s" $file`<br>
> + md5=`md5sum $file | awk '{print $1}'`<br>
> + sha1=`sha1sum $file | awk '{print $1}'`<br>
> + sha256=`sha256sum $file | awk '{print $1}'`<br>
> + awk 'BEGIN{c = 0} ; {<br>
> + if ($3 == p) {<br>
> + c = c + 1<br>
> + if (c == 1) {print " " md " " s " " p}<br>
> + if (c == 2) {print " " sh1 " " s " " p}<br>
> + if (c == 3) {print " " sh2 " " s " " p}<br>
> + } else {print $0}<br>
> + }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \<br>
> + Release >Release.new<br><br>
Typo: adjust the indentation.</div></div></div></div></blockquote>Corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + mv Release.new Release<br>
> + done<br>
> + # resign the selfsigned InRelease file<br>
> + $rm_file InRelease<br>
> + gpg --clearsign -o InRelease Release<br>
> + # resign the Release file<br>
> + $rm_file Release.gpg<br>
> + gpg -abs -o Release.gpg Release<br>
> + popd<br>
> +<br>
> + # 4. sync the latest distribution path changes to S3<br>
> + $aws s3 sync --acl public-read dists/$dist "$s3/dists/$dist"<br>
> + done<br>
> +<br>
> + # unlock the publishing<br>
> + $rm_file $ws_lockfile<br>
> +}<br>
> +<br>
> +# The 'pack_rpm' function especialy created for RPM packages. It works<br>
> +# with RPM packing OS like Centos, Fedora. It is based on globaly known<br>
> +# tool 'createrepo' from:<br>
> +# <a href="https://linux.die.net/man/8/createrepo" target="_blank" rel=" noopener noreferrer">https://linux.die.net/man/8/createrepo</a><br>
> +# This tool works with single distribution of the given OS.<br>
> +# Result of the routine is the rpm package for YUM repository with<br>
> +# file structure equal to the Centos/Fedora:<br>
> +# <a href="http://mirror.centos.org/centos/7/os/x86_64/Packages/" target="_blank" rel=" noopener noreferrer">http://mirror.centos.org/centos/7/os/x86_64/Packages/</a><br>
> +# <a href="http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/" target="_blank" rel=" noopener noreferrer">http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/</a><br>
> +function pack_rpm {<br>
> + if ! ls $repo/*.rpm >/dev/null 2>&1 ; then<br>
> + echo "ERROR: Current '$repo' has:"<br>
> + ls -al $repo<br>
> + echo "Usage: $0 [path with *.rpm files]"<br>
> + exit 1<br>
> + fi<br>
> +<br>
> + # temporary lock the publication to the repository<br>
> + ws=${prefix_lockfile}_${branch}_${os}_${DIST}<br>
> + ws_lockfile=${ws}.lock<br>
> + create_lockfile $ws_lockfile<br>
> +<br>
> + # create temporary workspace with packages copies<br>
> + $rm_dir $ws<br>
> + $mk_dir $ws<br>
> + cp $repo/*.rpm $ws/.<br>
> + cd $ws<br>
> +<br>
> + # set the paths<br>
> + if [ "$os" == "centos" ]; then<br>
> + repopath=$DIST/os/x86_64<br>
> + rpmpath=Packages<br>
> + elif [ "$os" == "fedora" ]; then<br>
> + repopath=releases/$DIST/Everything/x86_64/os<br>
> + rpmpath=Packages/$proddir<br>
> + fi<br>
> + packpath=$repopath/$rpmpath<br>
> +<br>
> + # prepare local repository with packages<br>
> + $mk_dir $packpath<br>
> + mv *.rpm $packpath/.<br>
> + cd $repopath<br><br>
Minor: I can't find the corresponding cd to the previous directory, so<br>
it can break a post processing.</div></div></div></div></blockquote>Actually there are only absolute paths in use, anyway I don't see any use to save the previous directories.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> +<br>
> + # copy the current metadata files from S3<br>
> + mkdir repodata.base<br>
> + for file in `$aws s3 ls $s3/$repopath/repodata/ | awk '{print $NF}'` ; do<br>
> + $aws s3 ls $s3/$repopath/repodata/$file || continue<br>
> + $aws s3 cp $s3/$repopath/repodata/$file repodata.base/$file<br>
> + done<br>
> +<br>
> + # create the new repository metadata files<br>
> + createrepo --no-database --update --workers=2 --compress-type=gz --simple-md-filenames .<br>
> + mv repodata repodata.adding<br>
> +<br>
> + # merge metadata files<br>
> + mkdir repodata<br>
> + head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml<br>
> + for file in filelists.xml other.xml primary.xml ; do<br>
> + # 1. take the 1st line only - to skip the line with number of packages which is not needed<br>
> + zcat repodata.adding/$file.gz | head -n 1 >repodata/$file<br>
> + # 2. take 2nd line with metadata tag and update the packages number in it<br>
> + packsold=0<br>
> + if [ -f repodata.base/$file.gz ] ; then<br>
> + packsold=`zcat repodata.base/$file.gz | head -n 2 | tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g'`<br><br>
Typo: adjust the indentation.</div></div></div></div></blockquote>Corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + fi<br>
> + packsnew=`zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g'`<br>
> + packs=$(($packsold+$packsnew))<br>
> + zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file<br><br>
Minor: consider splitting huge pipelines into a several lines separating<br>
it by a pipe char.</div></div></div></div></blockquote>Ok, corrected.<br><blockquote style="border-left:1px solid #0857A6;margin:10px;padding:0 0 0 10px;"><div id=""><div class="js-helper_mailru_css_attribute_postfix js-readmsg-msg_mailru_css_attribute_postfix"><div><div id="style_15755486210435629679_BODY_mailru_css_attribute_postfix"><br><br>
> + # 3. take only 'package' tags from new file<br>
> + zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 >>repodata/$file<br>
> + # 4. take only 'package' tags from old file if exists<br>
> + if [ -f repodata.base/$file.gz ] ; then<br>
> + zcat repodata.base/$file.gz | tail -n +3 | head -n -1 >>repodata/$file<br>
> + fi<br>
> + # 5. take the last closing line with metadata tag<br>
> + zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file<br>
> +<br>
> + # get the new data<br>
> + chsnew=`sha256sum repodata/$file | awk '{print $1}'`<br>
> + sz=`stat --printf="%s" repodata/$file`<br>
> + gzip repodata/$file<br>
> + chsgznew=`sha256sum repodata/$file.gz | awk '{print $1}'`<br>
> + szgz=`stat --printf="%s" repodata/$file.gz`<br>
> + timestamp=`date +%s -r repodata/$file.gz`<br>
> +<br>
> + # add info to repomd.xml file<br>
> + name=`echo $file | sed 's#\.xml$##g'`<br>
> + cat <<EOF >>repodata/repomd.xml<br>
> +<data type="$name"><br>
> + <checksum type="sha256">$chsgznew</checksum><br>
> + <open-checksum type="sha256">$chsnew</open-checksum><br>
> + <location href="repodata/$file.gz"/><br>
> + <timestamp>$timestamp</timestamp><br>
> + <size>$szgz</size><br>
> + <open-size>$sz</open-size><br>
> +</data>"<br>
> +EOF<br>
> + done<br>
> + tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml<br>
> + gpg --detach-sign --armor repodata/repomd.xml<br>
> +<br>
> + # copy the packages to S3<br>
> + for file in $rpmpath/*.rpm ; do<br>
> + $aws s3 cp --acl public-read $file "$s3/$repopath/$file"<br>
> + done<br>
> +<br>
> + # update the metadata at the S3<br>
> + $aws s3 sync --acl public-read repodata "$s3/$repopath/repodata"<br>
> +<br>
> + # unlock the publishing<br>
> + $rm_file $ws_lockfile<br>
> +}<br>
> +<br>
> +if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then<br>
> + pack_deb<br>
> +elif [ "$os" == "centos" -o "$os" == "fedora" ]; then<br>
> + pack_rpm<br>
> +else<br>
> + echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"<br>
> + usage<br>
> + exit 1<br>
> +fi<br>
> -- <br>
> 2.17.1<br>
> <br><br>
-- <br>
Best regards,<br>
IM<br></div></div></div></div></blockquote>
<br>
<br>-- <br>Alexander Tikhonov<br>
</div></div>
<base target="_self" href="https://e.mail.ru/">
</div>
</div></BODY></HTML>