Re: [Tarantool-patches] [PATCH v2] gitlab-ci: implement packing into MCS S3

["Alexander Tikhonov" <avtikhon@tarantool.org>]

[-- Attachment #1: Type: text/plain, Size: 29314 bytes --]





>Четверг,  5 декабря 2019, 15:23 +03:00 от Igor Munkin < imun@tarantool.org >:
>
>Sasha,
>
>Thanks for the patch. It looks much better now (except the fact I would
>prefer to see it in Perl rather in Bash), but let's polish it a bit.
>Consider my comments below.
>
>Moreover, this patch is a complex one, so please split your further work
>into several blocks to be committed separately.
Ok, sure
>
>On 12.11.19, Alexander V. Tikhonov wrote:
>> Added ability to store packages additionaly at MCS S3.
>> The target idea was to add the new way of packages creation at MCS S3,
>> which temporary duplicates the packaging at PackageCloud by Packpack
>> tool. Also it was needed to pack the binaries in the native way of the
>> each packing OS style. It was created standalone script for adding the
>> packages binaries/sources to MCS S3 at DEB either RPM repositories:
>> 'tools/add_pack_s3_repo.sh'
>> Common parts of the script are:
>>  - create new meta files for the new binaries
>>  - copy new binaries to MCS S3
>>  - get previous metafiles from MCS S3 and merge the new meta data for
>> the new binaries
>>  - update the meta files at MCS S3
>> Different parts:
>>  - DEB script part based on reprepro external tool, also it works separately
>> only on OS versions level - it means that meta data it updates for all
>> Distritbutions together.
>>  - RPM script part based on createrepo external tool, also it works
>> separately for OS/Release level - it means that meta data it updates
>> for all Releases separately.
>> 
>> Closes #3380
>> ---
>> 
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
>> Issue:  https://github.com/tarantool/tarantool/issues/3380
>
>Please add the changelog per version in further patchesets. 
Ok
>
>> 
>>  .gitlab-ci.yml            |   5 +-
>>  .gitlab.mk                |  20 +-
>>  .travis.mk                |  41 ++--
>>  tools/add_pack_s3_repo.sh | 493 ++++++++++++++++++++++++++++++++++++++
>>  4 files changed, 533 insertions(+), 26 deletions(-)
>>  create mode 100755 tools/add_pack_s3_repo.sh
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index cf13c382e..4dcaf9cd3 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -231,7 +231,10 @@ debian_10:
>>      DIST: 'buster'
>> 
>>  static_build:
>> -  <<: *deploy_test_definition
>> +  <<: *release_only_definition
>> +  stage: test
>> +  tags:
>> +    - deploy_test
>>    variables:
>>      RUN_TESTS: 'ON'
>>    script:
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index 48a92e518..64664c64f 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -98,13 +98,27 @@ vms_test_%:
>>  vms_shutdown:
>>  	VBoxManage controlvm ${VMS_NAME} poweroff
>> 
>> -# ########################
>> -# Build RPM / Deb packages
>> -# ########################
>> +# ###########################
>> +# Sources tarballs & packages
>> +# ###########################
>> +
>> +# Push alpha and beta versions to <major>x bucket (say, 2x),
>> +# stable to <major>.<minor> bucket (say, 2.2).
>> +GIT_DESCRIBE=$(shell git describe HEAD)
>> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>> +ifeq ($(MINOR_VERSION),0)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>> +ifeq ($(MINOR_VERSION),1)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>> 
>>  package: git_submodule_update
>>  	git clone  https://github.com/packpack/packpack.git packpack
>>  	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>> +	./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build
>> 
>>  # ############
>>  # Static build
>> diff --git a/.travis.mk b/.travis.mk
>> index 42969ff56..a85f71ced 100644
>> --- a/.travis.mk
>> +++ b/.travis.mk
>> @@ -8,10 +8,6 @@ MAX_FILES?=65534
>> 
>>  all: package
>> 
>> -package:
>> -	git clone  https://github.com/packpack/packpack.git packpack
>> -	./packpack/packpack
>> -
>>  test: test_$(TRAVIS_OS_NAME)
>> 
>>  # Redirect some targets via docker
>> @@ -176,34 +172,35 @@ test_freebsd_no_deps: build_freebsd
>> 
>>  test_freebsd: deps_freebsd test_freebsd_no_deps
>> 
>> -####################
>> -# Sources tarballs #
>> -####################
>> -
>> -source:
>> -	git clone  https://github.com/packpack/packpack.git packpack
>> -	TARBALL_COMPRESSOR=gz packpack/packpack tarball
>> +###############################
>> +# Sources tarballs & packages #
>> +###############################
>> 
>>  # Push alpha and beta versions to <major>x bucket (say, 2x),
>>  # stable to <major>.<minor> bucket (say, 2.2).
>> -ifeq ($(TRAVIS_BRANCH),master)
>>  GIT_DESCRIBE=$(shell git describe HEAD)
>>  MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>>  MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> -else
>> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
>> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
>> -endif
>> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>>  ifeq ($(MINOR_VERSION),0)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>>  endif
>>  ifeq ($(MINOR_VERSION),1)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>>  endif
>> 
>> +packpack_prepare:
>> +	git clone  https://github.com/packpack/packpack.git packpack
>> +
>> +package: packpack_prepare
>> +	./packpack/packpack
>> +
>> +source: packpack_prepare
>> +	TARBALL_COMPRESSOR=gz packpack/packpack tarball
>> +
>>  source_deploy:
>>  	pip install awscli --user
>> -	aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> -		cp build/*.tar.gz "s3://${BUCKET}/" \
>> -		--acl public-read
>> +	for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
>> +		aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> +			cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
>> +			--acl public-read
>> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
>> new file mode 100755
>> index 000000000..2316b9015
>> --- /dev/null
>> +++ b/tools/add_pack_s3_repo.sh
>> @@ -0,0 +1,493 @@
>> +#!/bin/bash
>> +set -e
>> +
>> +rm_file='rm -f'
>> +rm_dir='rm -rf'
>> +mk_dir='mkdir -p'
>> +
>> +alloss='ubuntu debian centos el fedora'
>
>General comments for the script:
>* as discussed offline with you and Sasha Tu., "centos" alias is excess
>  and it can be removed everywhere within this script. 
"centos" alias completely removed.
>
>* consider using $(...) instead of `...` to improve readability. 
Changed.
>
>* typo: adjust the tab misusage for indentation.
Fixed.
>
>> +
>> +get_os_dists()
>> +{
>> +    os=$1
>> +    alldists=
>> +
>> +    if [ "$os" == "ubuntu" ]; then
>> +        alldists='trusty xenial cosmic disco bionic eoan'
>> +    elif [ "$os" == "debian" ]; then
>> +        alldists='jessie stretch buster'
>> +    elif [ "$os" == "centos" -o "$os" == "el" ]; then
>> +        alldists='6 7 8'
>> +    elif [ "$os" == "fedora" ]; then
>> +        alldists='27 28 29 30 31'
>> +    fi
>> +
>> +    echo "$alldists"
>> +}
>> +
>> +ws_prefix=/tmp/tarantool_repo_s3
>
>Please move the prior line to the beginning of the script to organize a
>single place with the "constants" and "defaults". 
Ok, done, as much as possible not to make the code harder to understand.
>
>
>> +create_lockfile()
>> +{
>> +    lockfile -l 1000 $1
>> +}
>> +
>> +usage()
>> +{
>> +    cat <<EOF
>> +Usage: $0 -b <S3 bucket> -o <OS name> -d <OS distribuition> [-p <product>] <path>
>> +Options:
>> +    -b|--bucket
>> +        MCS S3 bucket which will be used for storing the packages.
>> +        Name of the package one of the appropriate Tarantool branch:
>> +            master: 2x
>> +            2.3: 2_3
>> +            2.2: 2_2
>> +            1.10: 1_10
>> +    -o|--os
>> +        OS to be checked, one of the list (NOTE: centos == el):
>> +            $alloss
>> +    -d|--distribution
>> +        Distribution appropriate to the given OS:
>> +EOF
>> +    for os in $alloss ; do
>> +        echo "            $os: <"`get_os_dists $os`">"
>> +    done
>> +    cat <<EOF
>> +    -p|--product
>> +         Product name to be packed with, default name is 'tarantool'
>> +    -h|--help
>> +         Usage help message
>
>Minor: arguments description usually goes prior to options description. 
Corrected, help message improved.
>
>
>> +    <path>
>> +         Path points to the directory with deb/prm packages to be used.
>> +         Script can be used in one of 2 modes:
>> +          - path with binaries packages for a single distribution
>> +	  - path with 'pool' directory with APT repository (only: debian|ubuntu)
>> +EOF
>> +}
>> +
>> +for i in "$@"
>> +do
>> +case $i in
>> +    -b=*|--bucket=*)
>> +    branch="${i#*=}"
>> +    if [ "$branch" != "2x" -a "$branch" != "2_3" -a "$branch" != "2_2" -a "$branch" != "1_10" ]; then
>
>grep seems to be more convenient here
>| echo "$branch" | grep -qvP '^(1_10|2(x|_[2-4]))$'
Ok, changed.
>
>> +        echo "ERROR: bucket '$branch' is not supported"
>> +        usage
>> +        exit 1
>> +    fi
>> +    shift # past argument=value
>> +    ;;
>> +    -o=*|--os=*)
>> +    os="${i#*=}"
>> +    if [ "$os" == "el" ]; then
>> +        os=centos
>> +    fi
>> +    if ! echo $alloss | grep -F -q -w $os ; then
>> +        echo "ERROR: OS '$os' is not supported"
>> +        usage
>> +        exit 1
>> +    fi
>> +    shift # past argument=value
>> +    ;;
>> +    -d=*|--distribution=*)
>> +    DIST="${i#*=}"
>> +    shift # past argument=value
>
>I guess DIST can be validated right here like other variables above
>since it is a mandatory one. I see no need in get_os_dists subroutine,
>you can just initialize alldists at the beginning (like, alloss) and use
>it instead of "qx"-ing the corresponding function.
Updated help message routine to show that the DIST is not the mandatory option.
>
>Minor: IMHO the naming should be consistent (DIST but os). 
Renamed distribution variables
dist -> loop_dist to show that the variable is the loop changing
DIST -> option_dist to show that the variable had come from the options
>
>
>> +    ;;
>> +    -p=*|--product=*)
>> +    product="${i#*=}"
>> +    shift # past argument=value
>> +    ;;
>> +    -h|--help)
>> +    usage
>> +    exit 0
>> +    ;;
>> +    *)
>> +    repo="${i#*=}"
>> +    pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
>> +    shift # past argument=value
>> +    ;;
>> +esac
>> +done
>> +
>> +# check that all needed options were set
>> +if [ "$branch" == "" ]; then
>> +    echo "ERROR: need to set -b|--bucket bucket option, check usage"
>> +    usage
>> +    exit 1
>> +fi
>> +if [ "$os" == "" ]; then
>> +    echo "ERROR: need to set -o|--os OS name option, check usage"
>> +    usage
>> +    exit 1
>> +fi
>> +alldists=`get_os_dists $os`
>> +if [ -n "$DIST" ] && ! echo $alldists | grep -F -q -w $DIST ; then
>> +    echo "ERROR: set distribution at options '$DIST' not found at supported list '$alldists'"
>> +    usage
>> +    exit 1
>> +fi
>> +
>> +# set the path with binaries
>> +product=$product
>
>I don't get this line, it seems to be an excess one, but I'm totally not
>a bash master.
Fixed, just the obvious code line.
>
>> +if [ "$product" == "" ]; then
>> +    product=tarantool
>> +fi
>
>Consider moving defaults initialization prior to arguments parsing to
>avoid branching like the one above. 
Moved product and repo before the options parser
>
>
>> +proddir=`echo $product | head -c 1`
>> +
>> +# set the path with binaries
>> +if [ "$repo" == "" ]; then
>> +    repo=.
>> +fi
>
>Please consider the comment above related to defaults initialization. 
Moved product and repo before the options parser
>
>
>> +
>> +aws='aws --endpoint-url  https://hb.bizmrg.com '
>
>Minor: I guess you can append s3 literal to the aws variable and use
>| $aws cp --acl public-read $deb $s3/$locdeb
>instead of
>| $aws s3 cp --acl public-read $deb $s3/$locdeb 
Ok, corrected.
>
>
>> +s3="s3://tarantool_repo/$branch/$os"
>> +
>> +# The 'pack_deb' function especialy created for DEB packages. It works
>> +# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
>> +# tool 'reprepro' from:
>> +#  https://wiki.debian.org/DebianRepository/SetupWithReprepro
>> +# This tool works with complete number of distributions of the given OS.
>> +# Result of the routine is the debian package for APT repository with
>> +# file structure equal to the Debian/Ubuntu:
>> +#  http://ftp.am.debian.org/debian/pool/main/t/tarantool/
>> +#  http://ftp.am.debian.org/ubuntu/pool/main/t/
>> +function pack_deb {
>> +    # we need to push packages into 'main' repository only
>> +    component=main
>> +
>> +    # debian has special directory 'pool' for packages
>> +    debdir=pool
>> +
>> +    # get packages from pointed location either mirror path
>> +    if [ "$repo" == "" ] ; then
>> +        repo=/var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os
>> +    fi
>
>Maybe I misread the logic above, but the condition above seems to be
>always false. 
Right, this path better to mention in Usage routine instead of default setup.
>
>
>> +    if [ ! -d $repo/$debdir ] && ( [ "$DIST" == "" ] || ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ) ; then
>> +        echo "ERROR: Current '$repo' path doesn't have any of the following:"
>> +        echo "Usage with set distribuition with option '-d' and packages: $0 [path with *.deb *.dsc *.tar.*z files]"
>> +        echo "Usage with repositories: $0 [path to repository with '$debdir' subdirectory]"
>
>Heredoc seems to be more convenient here. 
Removed extra message and set usage call
>
>
>> +        exit 1
>> +    fi
>> +
>
>As discussed offline let's try to move the code below to a generic
>function with a OS-specific handler (e.g. <os>_prep) to be executed
>within. 
Ok, created prepare_ws routine
>
>
>> +    # temporary lock the publication to the repository
>> +    ws=${ws_prefix}_${branch}_${os}
>> +    ws_lockfile=${ws}.lock
>> +    create_lockfile $ws_lockfile
>> +
>> +    # create temporary workspace with repository copy
>> +    $rm_dir $ws
>> +    $mk_dir $ws
>> +
>> +    # script works in one of 2 modes:
>> +    # - path with binaries packages for a single distribution
>> +    # - path with 'pool' directory with APT repository
>> +    if [ "$DIST" != "" ] && ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ; then
>
>Minor: consider splitting the condition above into two lines. 
Corrected all long lines
>
>
>> +        # copy single distribution with binaries packages
>> +        repopath=$ws/pool/${DIST}/main/$proddir/$product
>
>I guess you missed to s/main/$component/ here.
Right - corrected.
>
>> +        $mk_dir ${repopath}
>> +        cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
>> +    elif [ -d $repo/$debdir ]; then
>> +        # copy 'pool' directory with APT repository
>> +        cp -rf $repo/$debdir $ws/.
>> +    else
>> +        echo "ERROR: neither distribution option '-d' with files $repo/*.deb $repo/*.dsc $repo/*.tar.*z set nor '$repo/$debdir' path found"
>> +	usage
>> +	$rm_file $wslock
>> +	exit 1
>> +    fi
>> +    cd $ws
>
>Minor: I can't find the corresponding cd to the previous directory, so
>it can break a post processing. 
Actually there are only absolute paths in use, anyway I don't see any use to save the previous directories.
>
>
>> +
>> +    # create the configuration file for 'reprepro' tool
>> +    confpath=$ws/conf
>> +    $rm_dir $confpath
>> +    $mk_dir $confpath
>> +
>> +    for dist in $alldists ; do
>> +        cat <<EOF >>$confpath/distributions
>> +Origin: Tarantool
>> +Label: tarantool.org
>> +Suite: stable
>> +Codename: $dist
>> +Architectures: amd64 source
>> +Components: main
>
>I guess you missed to s/main/$component/ here. 
Right - corrected.
>
>
>> +Description: Unofficial Ubuntu Packages maintained by Tarantool
>> +SignWith: 91B625E5
>> +DebIndices: Packages Release . .gz .bz2
>> +UDebIndices: Packages . .gz .bz2
>> +DscIndices: Sources Release .gz .bz2
>> +
>> +EOF
>> +done
>
>Typo: adjust the indentation. 
Corrected.
>
>
>> +
>> +    # create standalone repository with separate components
>> +    for dist in $alldists ; do
>> +        echo =================== DISTRIBUTION: $dist =========================
>> +        updated_deb=0
>> +        updated_dsc=0
>> +
>> +        # 1(binaries). use reprepro tool to generate Packages file
>> +        for deb in $ws/$debdir/$dist/$component/*/*/*.deb ; do
>> +            [ -f $deb ] || continue
>> +            locdeb=`echo $deb | sed "s#^$ws\/##g"`
>
>Minor: escaping is excess here, since you use a different char for
>separator. Please consider this remark for all script. 
Fixed 2 places.
>
>
>> +            echo "DEB: $deb"
>> +            # register DEB file to Packages file
>> +            reprepro -Vb . includedeb $dist $deb
>> +            # reprepro copied DEB file to local component which is not needed
>> +            $rm_dir $debdir/$component
>> +            # to have all packages avoid reprepro set DEB file to its own registry
>> +            $rm_dir db
>> +            # copy Packages file to avoid of removing by the new DEB version
>> +            for packages in dists/$dist/$component/binary-*/Packages ; do
>> +                if [ ! -f $packages.saved ] ; then
>> +                    # get the latest Packages file from S3
>> +                    $aws s3 ls "$s3/$packages" 2>/dev/null && \
>> +                        $aws s3 cp --acl public-read \
>> +                        "$s3/$packages" $packages.saved || \
>
>Minor: IMHO, splitting the previous command into two lines makes it less
>readable. However, feel free to ignore this note.
Corrected.
>
>> +                        touch $packages.saved
>> +                fi
>> +                # check if the DEB file already exists in Packages from S3
>> +                if grep "^`grep "^SHA256: " $packages`$" $packages.saved ; then
>> +                    echo "WARNING: DEB file already registered in S3!"
>> +                    continue
>> +                fi
>> +                # store the new DEB entry
>> +                cat $packages >>$packages.saved
>> +                # save the registered DEB file to S3
>> +                $aws s3 cp --acl public-read $deb $s3/$locdeb
>> +                updated_deb=1
>> +            done
>> +        done
>> +
>
>I see this part is quite similar to the corresponding one above. Please
>consider to move it to a separate generic function with a source/binary
>specific handler to be executed within.
Created standalone routine for metadata Packages and Sources files update
>
>> +        # 1(sources). use reprepro tool to generate Sources file
>> +        for dsc in $ws/$debdir/$dist/$component/*/*/*.dsc ; do
>> +            [ -f $dsc ] || continue
>> +            locdsc=`echo $dsc | sed "s#^$ws\/##g"`
>> +            echo "DSC: $dsc"
>> +            # register DSC file to Sources file
>> +            reprepro -Vb . includedsc $dist $dsc
>> +            # reprepro copied DSC file to component which is not needed
>> +            $rm_dir $debdir/$component
>> +            # to have all sources avoid reprepro set DSC file to its own registry
>> +            $rm_dir db
>> +            # copy Sources file to avoid of removing by the new DSC version
>> +            sources=dists/$dist/$component/source/Sources
>> +            if [ ! -f $sources.saved ] ; then
>> +                # get the latest Sources file from S3
>> +                $aws s3 ls "$s3/$sources" && \
>> +                    $aws s3 cp --acl public-read "$s3/$sources" $sources.saved || \
>> +                    touch $sources.saved
>> +            fi
>> +            # WORKAROUND: unknown why, but reprepro doesn`t save the Sources file
>> +            gunzip -c $sources.gz >$sources
>> +            # check if the DSC file already exists in Sources from S3
>> +            hash=`grep '^Checksums-Sha256:' -A3 $sources | \
>> +                tail -n 1 | awk '{print $1}'`
>> +            if grep " $hash .*\.dsc$" $sources.saved ; then
>> +                echo "WARNING: DSC file already registered in S3!"
>> +                continue
>> +            fi
>> +            # store the new DSC entry
>> +            cat $sources >>$sources.saved
>> +            # save the registered DSC file to S3
>> +            $aws s3 cp --acl public-read $dsc $s3/$locdsc
>> +            tarxz=`echo $locdsc | sed 's#\.dsc$#.debian.tar.xz#g'`
>> +            $aws s3 cp --acl public-read $ws/$tarxz "$s3/$tarxz"
>> +            orig=`echo $locdsc | sed 's#-1\.dsc$#.orig.tar.xz#g'`
>> +            $aws s3 cp --acl public-read $ws/$orig "$s3/$orig"
>> +            updated_dsc=1
>> +        done
>> +
>> +        # check if any DEB/DSC files were newly registered
>> +        [ "$updated_deb" == "0" -a "$updated_dsc" == "0" ] && \
>> +            continue || echo "Updating dists"
>> +
>> +        # finalize the Packages file
>> +        for packages in dists/$dist/$component/binary-*/Packages ; do
>> +            mv $packages.saved $packages
>> +        done
>> +
>> +        # 2(binaries). update Packages file archives
>> +        for packpath in dists/$dist/$component/binary-* ; do
>> +            pushd $packpath
>> +            sed "s#Filename: $debdir/$component/#Filename: $debdir/$dist/$component/#g" -i Packages
>> +            bzip2 -c Packages >Packages.bz2
>> +            gzip -c Packages >Packages.gz
>> +            popd
>> +        done
>> +
>> +        # 2(sources). update Sources file archives
>> +        pushd dists/$dist/$component/source
>> +        sed "s#Directory: $debdir/$component/#Directory: $debdir/$dist/$component/#g" -i Sources
>> +        bzip2 -c Sources >Sources.bz2
>> +        gzip -c Sources >Sources.gz
>> +        popd
>> +
>> +        # 3. update checksums entries of the Packages* files in *Release files
>> +	# NOTE: it is stable structure of the *Release files when the checksum
>> +	#       entries in it in the following way:
>> +	# MD5Sum:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +        # SHA1:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +        # SHA256:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +	#       The script bellow puts 'md5' value at the 1st found file entry,
>> +	#       'sha1' - at the 2nd and 'sha256' at the 3rd
>> +        pushd dists/$dist
>> +        for file in `grep " $component/" Release | awk '{print $3}' | sort -u` ; do
>> +            sz=`stat -c "%s" $file`
>> +            md5=`md5sum $file | awk '{print $1}'`
>> +            sha1=`sha1sum $file | awk '{print $1}'`
>> +            sha256=`sha256sum $file | awk '{print $1}'`
>> +            awk 'BEGIN{c = 0} ; {
>> +                if ($3 == p) {
>> +                    c = c + 1
>> +                    if (c == 1) {print " " md  " " s " " p}
>> +                    if (c == 2) {print " " sh1 " " s " " p}
>> +                    if (c == 3) {print " " sh2 " " s " " p}
>> +                } else {print $0}
>> +            }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
>> +            Release >Release.new
>
>Typo: adjust the indentation. 
Corrected.
>
>
>> +            mv Release.new Release
>> +        done
>> +        # resign the selfsigned InRelease file
>> +        $rm_file InRelease
>> +        gpg --clearsign -o InRelease Release
>> +        # resign the Release file
>> +        $rm_file Release.gpg
>> +        gpg -abs -o Release.gpg Release
>> +        popd
>> +
>> +        # 4. sync the latest distribution path changes to S3
>> +        $aws s3 sync --acl public-read dists/$dist "$s3/dists/$dist"
>> +    done
>> +
>> +    # unlock the publishing
>> +    $rm_file $ws_lockfile
>> +}
>> +
>> +# The 'pack_rpm' function especialy created for RPM packages. It works
>> +# with RPM packing OS like Centos, Fedora. It is based on globaly known
>> +# tool 'createrepo' from:
>> +#  https://linux.die.net/man/8/createrepo
>> +# This tool works with single distribution of the given OS.
>> +# Result of the routine is the rpm package for YUM repository with
>> +# file structure equal to the Centos/Fedora:
>> +#  http://mirror.centos.org/centos/7/os/x86_64/Packages/
>> +#  http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
>> +function pack_rpm {
>> +    if ! ls $repo/*.rpm >/dev/null 2>&1 ; then
>> +        echo "ERROR: Current '$repo' has:"
>> +        ls -al $repo
>> +        echo "Usage: $0 [path with *.rpm files]"
>> +        exit 1
>> +    fi
>> +
>> +    # temporary lock the publication to the repository
>> +    ws=${prefix_lockfile}_${branch}_${os}_${DIST}
>> +    ws_lockfile=${ws}.lock
>> +    create_lockfile $ws_lockfile
>> +
>> +    # create temporary workspace with packages copies
>> +    $rm_dir $ws
>> +    $mk_dir $ws
>> +    cp $repo/*.rpm $ws/.
>> +    cd $ws
>> +
>> +    # set the paths
>> +    if [ "$os" == "centos" ]; then
>> +        repopath=$DIST/os/x86_64
>> +        rpmpath=Packages
>> +    elif [ "$os" == "fedora" ]; then
>> +        repopath=releases/$DIST/Everything/x86_64/os
>> +        rpmpath=Packages/$proddir
>> +    fi
>> +    packpath=$repopath/$rpmpath
>> +
>> +    # prepare local repository with packages
>> +    $mk_dir $packpath
>> +    mv *.rpm $packpath/.
>> +    cd $repopath
>
>Minor: I can't find the corresponding cd to the previous directory, so
>it can break a post processing. 
Actually there are only absolute paths in use, anyway I don't see any use to save the previous directories.
>
>
>> +
>> +    # copy the current metadata files from S3
>> +    mkdir repodata.base
>> +    for file in `$aws s3 ls $s3/$repopath/repodata/ | awk '{print $NF}'` ; do
>> +        $aws s3 ls $s3/$repopath/repodata/$file || continue
>> +        $aws s3 cp $s3/$repopath/repodata/$file repodata.base/$file
>> +    done
>> +
>> +    # create the new repository metadata files
>> +    createrepo --no-database --update --workers=2 --compress-type=gz --simple-md-filenames .
>> +    mv repodata repodata.adding
>> +
>> +    # merge metadata files
>> +    mkdir repodata
>> +    head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
>> +    for file in filelists.xml other.xml primary.xml ; do
>> +        # 1. take the 1st line only - to skip the line with number of packages which is not needed
>> +        zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
>> +        # 2. take 2nd line with metadata tag and update the packages number in it
>> +        packsold=0
>> +        if [ -f repodata.base/$file.gz ] ; then
>> +        packsold=`zcat repodata.base/$file.gz | head -n 2 | tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g'`
>
>Typo: adjust the indentation. 
Corrected.
>
>
>> +        fi
>> +        packsnew=`zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g'`
>> +        packs=$(($packsold+$packsnew))
>> +        zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
>
>Minor: consider splitting huge pipelines into a several lines separating
>it by a pipe char. 
Ok, corrected.
>
>
>> +        # 3. take only 'package' tags from new file
>> +        zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 >>repodata/$file
>> +        # 4. take only 'package' tags from old file if exists
>> +        if [ -f repodata.base/$file.gz ] ; then
>> +            zcat repodata.base/$file.gz | tail -n +3 | head -n -1 >>repodata/$file
>> +        fi
>> +        # 5. take the last closing line with metadata tag
>> +        zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
>> +
>> +        # get the new data
>> +        chsnew=`sha256sum repodata/$file | awk '{print $1}'`
>> +        sz=`stat --printf="%s" repodata/$file`
>> +        gzip repodata/$file
>> +        chsgznew=`sha256sum repodata/$file.gz | awk '{print $1}'`
>> +        szgz=`stat --printf="%s" repodata/$file.gz`
>> +        timestamp=`date +%s -r repodata/$file.gz`
>> +
>> +        # add info to repomd.xml file
>> +        name=`echo $file | sed 's#\.xml$##g'`
>> +	cat <<EOF >>repodata/repomd.xml
>> +<data type="$name">
>> +  <checksum type="sha256">$chsgznew</checksum>
>> +  <open-checksum type="sha256">$chsnew</open-checksum>
>> +  <location href="repodata/$file.gz"/>
>> +  <timestamp>$timestamp</timestamp>
>> +  <size>$szgz</size>
>> +  <open-size>$sz</open-size>
>> +</data>"
>> +EOF
>> +    done
>> +    tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
>> +    gpg --detach-sign --armor repodata/repomd.xml
>> +
>> +    # copy the packages to S3
>> +    for file in $rpmpath/*.rpm ; do
>> +        $aws s3 cp --acl public-read $file "$s3/$repopath/$file"
>> +    done
>> +
>> +    # update the metadata at the S3
>> +    $aws s3 sync --acl public-read repodata "$s3/$repopath/repodata"
>> +
>> +    # unlock the publishing
>> +    $rm_file $ws_lockfile
>> +}
>> +
>> +if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
>> +    pack_deb
>> +elif [ "$os" == "centos" -o "$os" == "fedora" ]; then
>> +    pack_rpm
>> +else
>> +    echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
>> +    usage
>> +    exit 1
>> +fi
>> -- 
>> 2.17.1
>> 
>
>-- 
>Best regards,
>IM


-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 44996 bytes --]