Igor,

Thanks for the review, I've took the commit message that you provided and made all the changes that you suggested. 
>Четверг, 23 января 2020, 18:27 +03:00 от Igor Munkin <imun@tarantool.org>:
>
>Sasha,
>
>Thanks, I took a look on the new version of the changes since there
>was an important discussion in the previous patch[1] and significant
>changes are made within the current one. I left several comments below,
>please consider them.
>
>On 17.01.20, Alexander V. Tikhonov wrote:
>
>> Gitlab-CI rules changed to be able to create packages on branches
>> with ".*-full-ci$" pattern and additionaly deploy it to MCS S3 on
>> master branches in 'live' repository. Tagged in GIT commit will
>> be additionaly packed to 'release' repository.
>> 
>> Added ability to store packages at MCS S3 additionally to packages
>> storing at PackageCloud. The main difference that the repository
>> at MCS S3 has its native files structure as its OS distributions
>> have. Also the new repository doesn't prune the old packages and
>> stores all the saved packages for Tarantool and its modules for
>> each commit at all the release branches. Tarantool users may enable
>> the repository and fix certain version of Tarantool in dependencies
>> of an application packages.
>> 
>> For storing the DEB and RPM packages to MCS S3 was created
>> standalone script which creates/updates the packages repositories
>> with OS distribution native files structure style.
>> 
>> Common parts of the script are:
>> 
>>  - create new meta files for the new binaries
>>  - copy new binaries to MCS S3
>>  - get previous meta files from MCS S3 and merge the new meta data
>>    for the new binaries
>>  - update the meta files at MCS S3
>> 
>> Different parts:
>> 
>>  - DEB script part based on 'reprepro' external tool, also it works
>>    separately only on OS versions level - it means that meta data
>>    it updates for all distributions together.
>>  - RPM script part based on 'createrepo' external tool, also it works
>>    separately for OS/Release level - it means that meta data it
>>    updates for all releases separately.
>> 
>
>The commit message looks to be complex and unclear. I propose to reword
>it as the following:
>| The changes introduce new Gitlab-CI rules for creating packages on
>| branches with "-full-ci" suffix and their subsequent deployment to the
>| 'live' repository for master and release branches. Packages for tagged
>| commits are also delivered to the corresponding 'release' repository.
>|
>| The PackageCloud storage is replaced with the new self-hosted one
>| (based on S3 object storage) where all old packages have been synced.
>| The new builds will be pushed only to S3 based repos. Benefits of the
>| introduced approach are the following:
>| * As far as all contents of self-hosted repos are fully controlled
>| theirs layout is the same as the ones provided by the corresponding
>| distro
>| * Repo metadata rebuild is excess considering the known repo layout
>| * Old packages are not pruned since they do not affect the repo
>| metadata rebuilding time
>|
>| For these purposes the standalone script for pushing DEB and RPM
>| packages to self-hosted repositories is introduced. The script
>| implements the following flow:
>| * creates new metafiles for the new packages
>| * copies new packages to S3 storage
>| * fetches relevant metafiles from the repo
>| * merges the new metadata with the fetched one
>| * pushes the updated metadata to S3 storage
>|
>| There are distro dependent parts in the script:
>| * For RPM packages it updates metadata separately per each repo
>| considering 'createrepo' util behaviour
>| * For DEB packages it updates metadata simultaniously for all repos
>| considering 'reprepro' util behaviour
Used your comments completely.
>
>> Closes #3380
>> 
>> @TarantoolBot
>> Title: Update download instructions on the website
>> 
>> Need to update download instructions on the website, due to the new
>> repository based on MCS S3.
>> ---
>> 
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
>> Issue:  https://github.com/tarantool/tarantool/issues/3380
>> 
>> v5:  https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
>> v4:  https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
>> v3:  https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
>> v2:  https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
>> v1:  https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
>> 
>> Changes v6:
>> - implemented 2 MCS S3 repositories 'live' and 'release'
>> - added AWS and GPG keys into Gitlab-CI
>> - corrected commit message
>> - corrected return functionality code in script
>> - moved all changes for sources tarballs at the standalone patch set
>> 
>> Changes v5:
>> - code style
>> - commits squashed
>> - rebased to master
>> 
>> Changes v4:
>> - minor corrections
>> 
>> Changes v3:
>> - common code parts merged to standalone routines
>> - corrected code style, minor updates
>> - script is ready for release
>> 
>> Changes v2:
>> - made changes in script from draft to pre-release stages
>> 
>>  .gitlab-ci.yml       | 152 +++++++++++--
>>  .gitlab.mk           |  16 +-
>>  tools/update_repo.sh | 520 +++++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 668 insertions(+), 20 deletions(-)
>>  create mode 100755 tools/update_repo.sh
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 3af5a3c8a..c68594c1a 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -10,6 +10,10 @@ variables:
>>    only:
>>      refs:
>>        - master
>> +
>> +.fullci_only_template: &fullci_only_definition
>> +  only:
>> +    refs:
>>        - /^.*-full-ci$/
>> 
>>  .docker_test_template: &docker_test_definition
>> @@ -24,13 +28,29 @@ variables:
>>    tags:
>>      - docker_test
>> 
>> +.pack_template: &pack_definition
>> +  <<: *fullci_only_definition
>> +  stage: test
>> +  tags:
>> +    - deploy
>> +  script:
>> +    - ${GITLAB_MAKE} package
>> +
>> +.pack_test_template: &pack_test_definition
>> +  <<: *fullci_only_definition
>> +  stage: test
>> +  tags:
>> +    - deploy_test
>> +  script:
>> +    - ${GITLAB_MAKE} package
>> +
>>  .deploy_template: &deploy_definition
>>    <<: *release_only_definition
>>    stage: test
>>    tags:
>>      - deploy
>>    script:
>> -    - ${GITLAB_MAKE} package
>> +    - ${GITLAB_MAKE} deploy
>> 
>>  .deploy_test_template: &deploy_test_definition
>>    <<: *release_only_definition
>> @@ -38,7 +58,7 @@ variables:
>>    tags:
>>      - deploy_test
>>    script:
>> -    - ${GITLAB_MAKE} package
>> +    - ${GITLAB_MAKE} deploy
>> 
>>  .vbox_template: &vbox_definition
>>    stage: test
>> @@ -141,96 +161,194 @@ freebsd_12_release:
>>  # Packs
>> 
>>  centos_6:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'el'
>>      DIST: '6'
>> 
>>  centos_7:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'el'
>>      DIST: '7'
>> 
>>  centos_8:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'el'
>>      DIST: '8'
>> 
>>  fedora_28:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'fedora'
>>      DIST: '28'
>> 
>>  fedora_29:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'fedora'
>>      DIST: '29'
>> 
>>  fedora_30:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'fedora'
>>      DIST: '30'
>> 
>>  fedora_31:
>> -  <<: *deploy_test_definition
>> +  <<: *pack_test_definition
>>    variables:
>>      OS: 'fedora'
>>      DIST: '31'
>> 
>>  ubuntu_14_04:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'trusty'
>> 
>>  ubuntu_16_04:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'xenial'
>> 
>>  ubuntu_18_04:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'bionic'
>> 
>>  ubuntu_18_10:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'cosmic'
>> 
>>  ubuntu_19_04:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'disco'
>> 
>>  ubuntu_19_10:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'ubuntu'
>>      DIST: 'eoan'
>> 
>>  debian_8:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'debian'
>>      DIST: 'jessie'
>> 
>>  debian_9:
>> -  <<: *deploy_definition
>> +  <<: *pack_definition
>>    variables:
>>      OS: 'debian'
>>      DIST: 'stretch'
>> 
>>  debian_10:
>> +  <<: *pack_definition
>> +  variables:
>> +    OS: 'debian'
>> +    DIST: 'buster'
>> +
>> +# Deploy
>> +
>> +centos_6_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'el'
>> +    DIST: '6'
>> +
>> +centos_7_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'el'
>> +    DIST: '7'
>> +
>> +centos_8_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'el'
>> +    DIST: '8'
>> +
>> +fedora_28_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'fedora'
>> +    DIST: '28'
>> +
>> +fedora_29_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'fedora'
>> +    DIST: '29'
>> +
>> +fedora_30_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'fedora'
>> +    DIST: '30'
>> +
>> +fedora_31_deploy:
>> +  <<: *deploy_test_definition
>> +  variables:
>> +    OS: 'fedora'
>> +    DIST: '31'
>> +
>> +ubuntu_14_04_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'trusty'
>> +
>> +ubuntu_16_04_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'xenial'
>> +
>> +ubuntu_18_04_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'bionic'
>> +
>> +ubuntu_18_10_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'cosmic'
>> +
>> +ubuntu_19_04_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'disco'
>> +
>> +ubuntu_19_10_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'ubuntu'
>> +    DIST: 'eoan'
>> +
>> +debian_8_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'debian'
>> +    DIST: 'jessie'
>> +
>> +debian_9_deploy:
>> +  <<: *deploy_definition
>> +  variables:
>> +    OS: 'debian'
>> +    DIST: 'stretch'
>> +
>> +debian_10_deploy:
>>    <<: *deploy_definition
>>    variables:
>>      OS: 'debian'
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index 48a92e518..49277e176 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -98,14 +98,24 @@ vms_test_%:
>>  vms_shutdown:
>>  	VBoxManage controlvm ${VMS_NAME} poweroff
>> 
>> -# ########################
>> -# Build RPM / Deb packages
>> -# ########################
>> +# ###########################
>> +# Sources tarballs & packages
>> +# ###########################
>> 
>>  package: git_submodule_update
>>  	git clone  https://github.com/packpack/packpack.git packpack
>>  	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>> 
>> +deploy: package
>> +	for key in ${GPG_SECRET_KEY} ${GPG_PUBLIC_KEY} ; do \
>> +		echo $${key} | base64 -d | gpg --batch --import || true ; done
>> +	./tools/update_repo.sh -o=${OS} -d=${DIST} build
>> +	for tag in $$(git tag) ; do \
>> +			git describe --long $${tag} ; \
>> +		done | grep "^$$(git describe --long)$$" >/dev/null && \
>> +		./tools/update_repo.sh -o=${OS} -d=${DIST} \
>> +			-b='s3://tarantool_repo/release' build
>> +
>>  # ############
>>  # Static build
>>  # ############
>> diff --git a/tools/update_repo.sh b/tools/update_repo.sh
>> new file mode 100755
>> index 000000000..28b490fb7
>> --- /dev/null
>> +++ b/tools/update_repo.sh
>> @@ -0,0 +1,520 @@
>> +#!/bin/bash
>> +set -e
>> +
>> +rm_file='rm -f'
>> +rm_dir='rm -rf'
>> +mk_dir='mkdir -p'
>> +ws_prefix=/tmp/tarantool_repo_s3
>> +
>> +alloss='ubuntu debian el fedora'
>> +product=tarantool
>> +# the path with binaries either repository
>> +repo=.
>> +
>> +# AWS defines
>> +aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL} s3"
>
>Please add the default value for env variable AWS_S3_ENDPOINT_URL. 
Done.
>
>
>> +bucket='s3://tarantool_repo/live'
>> +aws_cp_public="$aws cp --acl public-read"
>> +aws_sync_public="$aws sync --acl public-read"
>> +
>> +function get_os_dists {
>> +    os=$1
>> +    alldists=
>> +
>> +    if [ "$os" == "ubuntu" ]; then
>> +        alldists='trusty xenial bionic cosmic disco eoan'
>> +    elif [ "$os" == "debian" ]; then
>> +        alldists='jessie stretch buster'
>> +    elif [ "$os" == "el" ]; then
>> +        alldists='6 7 8'
>> +    elif [ "$os" == "fedora" ]; then
>> +        alldists='27 28 29 30 31'
>> +    fi
>> +
>> +    echo "$alldists"
>> +}
>> +
>> +function prepare_ws {
>> +    # temporary lock the publication to the repository
>> +    ws_suffix=$1
>> +    ws=${ws_prefix}_${ws_suffix}
>> +    ws_lockfile=${ws}.lock
>> +    if [ -f $ws_lockfile ]; then
>> +        old_proc=$(cat $ws_lockfile)
>> +    fi
>> +    lockfile -l 60 $ws_lockfile
>> +    chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
>> +    if [ "$old_proc" != ""  -a "$old_proc" != "0" ]; then
>> +        kill -9 $old_proc >/dev/null || true
>> +    fi
>> +
>> +    # create temporary workspace with repository copy
>> +    $rm_dir $ws
>> +    $mk_dir $ws
>> +}
>> +
>> +function usage {
>> +    cat <<EOF
>> +Usage for store package binaries from the given path:
>> +    $0 -o=<OS name> -d=<OS distribuition> [-b=<S3 bucket>] [-p=<product>] <path to package binaries>
>> +
>> +Usage for mirroring Debian|Ubuntu OS repositories:
>> +    $0 -o=<OS name> [-b=<S3 bucket>] [-p=<product>] <path to 'pool' subdirectory with packages repository>
>> +
>> +Arguments:
>> +    <path>
>> +         Path points to the directory with deb/prm packages to be used.
>> +         Script can be used in one of 2 modes:
>> +          - path with binaries packages for a single distribution
>> +          - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
>> +                /var/spool/apt-mirror/mirror/packagecloud.io/$product/$os
>> +
>> +Options:
>
>As discussed offline, please consider introducing the --force flag
>implementing the following:
>* If flag is omitted the script fails whether checksum of local and
>  remote packages with the same name and version differs
>* If flag is specified the script updates the remote package with the
>  local one despite the checksum difference
>
>Please mention this behaviour in usage. 
Done
>
>
>> +    -b|--bucket
>> +        MCS S3 bucket already existing which will be used for storing the packages, by default bucket is:
>> +            $bucket
>> +    -o|--os
>> +        OS to be checked, one of the list:
>> +            $alloss
>> +    -d|--distribution
>> +        Distribution appropriate to the given OS:
>> +EOF
>> +    for os in $alloss ; do
>> +        echo "            $os: <"$(get_os_dists $os)">"
>> +    done
>> +    cat <<EOF
>> +    -p|--product
>> +         Product name to be packed with, default name is 'tarantool'
>> +    -h|--help
>> +         Usage help message
>> +EOF
>> +}
>> +
>> +for i in "$@"
>> +do
>> +case $i in
>> +    -b=*|--bucket=*)
>> +    bucket="${i#*=}"
>> +    shift # past argument=value
>> +    ;;
>> +    -o=*|--os=*)
>> +    os="${i#*=}"
>> +    if ! echo $alloss | grep -F -q -w $os ; then
>> +        echo "ERROR: OS '$os' is not supported"
>> +        usage
>> +        exit 1
>> +    fi
>> +    shift # past argument=value
>> +    ;;
>> +    -d=*|--distribution=*)
>> +    option_dist="${i#*=}"
>> +    shift # past argument=value
>> +    ;;
>> +    -p=*|--product=*)
>> +    product="${i#*=}"
>> +    shift # past argument=value
>> +    ;;
>> +    -h|--help)
>> +    usage
>> +    exit 0
>> +    ;;
>> +    *)
>> +    repo="${i#*=}"
>> +    pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
>> +    shift # past argument=value
>> +    ;;
>> +esac
>> +done
>> +
>> +# check that all needed options were set and correct
>> +if ! $aws ls $bucket >/dev/null ; then
>> +    echo "ERROR: bucket '$bucket' is not found"
>> +    usage
>> +    exit 1
>> +fi
>> +if [ "$os" == "" ]; then
>> +    echo "ERROR: need to set -o|--os OS name option, check usage"
>> +    usage
>> +    exit 1
>> +fi
>> +alldists=$(get_os_dists $os)
>> +if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
>> +    echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
>> +    usage
>> +    exit 1
>> +fi
>> +
>> +# set the subpath with binaries based on literal character of the product name
>> +proddir=$(echo $product | head -c 1)
>> +
>> +# set bucket path of the given OS in options
>> +bucket_path="$bucket/$os"
>> +
>> +function update_deb_packfile {
>> +    packfile=$1
>> +    packtype=$2
>> +    update_dist=$3
>> +
>> +    locpackfile=$(echo $packfile | sed "s#^$ws/##g")
>> +    # register DEB/DSC pack file to Packages/Sources file
>> +    reprepro -Vb . include$packtype $update_dist $packfile
>> +    # reprepro copied DEB/DSC file to component which is not needed
>> +    $rm_dir $debdir/$component
>> +    # to have all sources avoid reprepro set DEB/DSC file to its own registry
>> +    $rm_dir db
>> +}
>> +
>> +function update_deb_metadata {
>> +    packpath=$1
>> +    packtype=$2
>> +
>> +    if [ ! -f $packpath.saved ] ; then
>> +        # get the latest Sources file from S3
>> +        $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
>> +            $aws cp "$bucket_path/$packpath" $packpath.saved || \
>> +            touch $packpath.saved
>> +    fi
>> +
>> +    if [ "$packtype" == "dsc" ]; then
>> +        # WORKAROUND: unknown why, but reprepro doesn`t save the Sources file
>> +        gunzip -c $packpath.gz >$packpath
>> +        # check if the DSC file already exists in Sources from S3
>> +        hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
>> +            tail -n 1 | awk '{print $1}')
>> +        if grep " $hash .*$" $packpath.saved ; then
>> +            echo "WARNING: DSC file already registered in S3!"
>> +            return
>> +        fi
>> +        updated_dsc=1
>> +    elif [ "$packtype" == "deb" ]; then
>> +        # check if the DEB file already exists in Packages from S3
>> +        if grep "^$(grep '^SHA256: ' $packages)$" $packages.saved ; then
>> +            echo "WARNING: DEB file already registered in S3!"
>> +            return
>> +        fi
>> +        updated_deb=1
>> +    fi
>> +    # store the new DEB entry
>> +    cat $packpath >>$packpath.saved
>> +}
>> +
>> +# The 'pack_deb' function especialy created for DEB packages. It works
>> +# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
>> +# tool 'reprepro' from:
>> +#  https://wiki.debian.org/DebianRepository/SetupWithReprepro
>> +# This tool works with complete number of distributions of the given OS.
>> +# Result of the routine is the debian package for APT repository with
>> +# file structure equal to the Debian/Ubuntu:
>> +#  http://ftp.am.debian.org/debian/pool/main/t/tarantool/
>> +#  http://ftp.am.debian.org/ubuntu/pool/main/t/
>> +function pack_deb {
>> +    # we need to push packages into 'main' repository only
>> +    component=main
>> +
>> +    # debian has special directory 'pool' for packages
>> +    debdir=pool
>> +
>> +    # get packages from pointed location
>> +    if [ ! -d $repo/$debdir ] && \
>> +        ( [ "$option_dist" == "" ] || \
>> +            ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ) ; then
>> +        echo "ERROR: Current '$repo' path doesn't have any of the following:"
>> +        echo " - $0 run option '-d' and DEB packages in path"
>> +        echo " - 'pool' subdirectory with APT repositories"
>> +        usage
>> +        exit 1
>> +    fi
>> +
>> +    # prepare the workspace
>> +    prepare_ws ${os}
>> +
>> +    # script works in one of 2 modes:
>> +    # - path with binaries packages for a single distribution
>> +    # - path with 'pool' directory with APT repository
>> +    if [ "$option_dist" != "" ] && \
>> +            ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ; then
>> +        # copy single distribution with binaries packages
>> +        repopath=$ws/pool/${option_dist}/$component/$proddir/$product
>> +        $mk_dir ${repopath}
>> +        cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
>> +    elif [ -d $repo/$debdir ]; then
>
>As discussed offline the 'pool' flow is excess and required only for
>initial repositories sync. I guess it can be removed with all related
>entries within this script.
Removed
>
>> +        # copy 'pool' directory with APT repository
>> +        cp -rf $repo/$debdir $ws/.
>> +    else
>> +        echo "ERROR: neither distribution option '-d' with files $repo/*.deb $repo/*.dsc $repo/*.tar.*z set nor '$repo/$debdir' path found"
>> +        usage
>> +        $rm_file $wslock
>> +        exit 1
>> +    fi
>> +
>> +    pushd $ws
>> +
>> +    # create the configuration file for 'reprepro' tool
>> +    confpath=$ws/conf
>> +    $rm_dir $confpath
>> +    $mk_dir $confpath
>> +
>> +    for loop_dist in $alldists ; do
>> +        cat <<EOF >>$confpath/distributions
>> +Origin: Tarantool
>> +Label: tarantool.org
>> +Suite: stable
>> +Codename: $loop_dist
>> +Architectures: amd64 source
>> +Components: $component
>> +Description: Tarantool DBMS and Tarantool modules
>> +SignWith: 91B625E5
>> +DebIndices: Packages Release . .gz .bz2
>> +UDebIndices: Packages . .gz .bz2
>> +DscIndices: Sources Release .gz .bz2
>> +
>> +EOF
>> +    done
>> +
>> +    # create standalone repository with separate components
>> +    for loop_dist in $alldists ; do
>> +        echo ================ DISTRIBUTION: $loop_dist ====================
>> +        updated_files=0
>> +
>> +        # 1(binaries). use reprepro tool to generate Packages file
>> +        for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
>> +            [ -f $deb ] || continue
>> +            updated_deb=0
>> +            # regenerate DEB pack
>> +            update_deb_packfile $deb deb $loop_dist
>> +            echo "Regenerated DEB file: $locpackfile"
>> +            for packages in dists/$loop_dist/$component/binary-*/Packages ; do
>> +                # copy Packages file to avoid of removing by the new DEB version
>> +                # update metadata 'Packages' files
>> +                update_deb_metadata $packages deb
>> +                [ "$updated_deb" == "1" ] || continue
>> +                updated_files=1
>> +            done
>> +            # save the registered DEB file to S3
>> +            if [ "$updated_deb" == 1 ]; then
>> +                $aws_cp_public $deb $bucket_path/$locpackfile
>> +            fi
>> +        done
>> +
>> +        # 1(sources). use reprepro tool to generate Sources file
>> +        for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
>> +            [ -f $dsc ] || continue
>> +            updated_dsc=0
>> +            # regenerate DSC pack
>> +            update_deb_packfile $dsc dsc $loop_dist
>> +            echo "Regenerated DSC file: $locpackfile"
>> +            # copy Sources file to avoid of removing by the new DSC version
>> +            # update metadata 'Sources' file
>> +            update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
>> +            [ "$updated_dsc" == "1" ] || continue
>> +            updated_files=1
>> +            # save the registered DSC file to S3
>> +            $aws_cp_public $dsc $bucket_path/$locpackfile
>> +            tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
>> +            $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
>> +            orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
>> +            $aws_cp_public $ws/$orig "$bucket_path/$orig"
>> +        done
>> +
>> +        # check if any DEB/DSC files were newly registered
>> +        [ "$updated_files" == "0" ] && \
>> +            continue || echo "Updating dists"
>> +
>> +        # finalize the Packages file
>> +        for packages in dists/$loop_dist/$component/binary-*/Packages ; do
>> +            mv $packages.saved $packages
>> +        done
>> +
>> +        # 2(binaries). update Packages file archives
>> +        for packpath in dists/$loop_dist/$component/binary-* ; do
>> +            pushd $packpath
>> +            sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
>> +            bzip2 -c Packages >Packages.bz2
>> +            gzip -c Packages >Packages.gz
>> +            popd
>> +        done
>> +
>> +        # 2(sources). update Sources file archives
>> +        pushd dists/$loop_dist/$component/source
>> +        sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
>> +        bzip2 -c Sources >Sources.bz2
>> +        gzip -c Sources >Sources.gz
>> +        popd
>> +
>> +        # 3. update checksums entries of the Packages* files in *Release files
>> +        # NOTE: it is stable structure of the *Release files when the checksum
>> +        #       entries in it in the following way:
>> +        # MD5Sum:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +        # SHA1:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +        # SHA256:
>> +        #  <checksum> <size> <file orig>
>> +        #  <checksum> <size> <file debian>
>> +        #       The script bellow puts 'md5' value at the 1st found file entry,
>> +        #       'sha1' - at the 2nd and 'sha256' at the 3rd
>> +        pushd dists/$loop_dist
>> +        for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
>> +            sz=$(stat -c "%s" $file)
>> +            md5=$(md5sum $file | awk '{print $1}')
>> +            sha1=$(sha1sum $file | awk '{print $1}')
>> +            sha256=$(sha256sum $file | awk '{print $1}')
>> +            awk 'BEGIN{c = 0} ; {
>> +                if ($3 == p) {
>> +                    c = c + 1
>> +                    if (c == 1) {print " " md  " " s " " p}
>> +                    if (c == 2) {print " " sh1 " " s " " p}
>> +                    if (c == 3) {print " " sh2 " " s " " p}
>> +                } else {print $0}
>> +            }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
>> +                    Release >Release.new
>> +            mv Release.new Release
>> +        done
>> +        # resign the selfsigned InRelease file
>> +        $rm_file InRelease
>> +        gpg --clearsign -o InRelease Release
>> +        # resign the Release file
>> +        $rm_file Release.gpg
>> +        gpg -abs -o Release.gpg Release
>> +        popd
>> +
>> +        # 4. sync the latest distribution path changes to S3
>> +        $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
>> +    done
>> +
>> +    # unlock the publishing
>> +    $rm_file $ws_lockfile
>> +
>> +    popd
>> +}
>> +
>> +# The 'pack_rpm' function especialy created for RPM packages. It works
>> +# with RPM packing OS like Centos, Fedora. It is based on globaly known
>> +# tool 'createrepo' from:
>> +#  https://linux.die.net/man/8/createrepo
>> +# This tool works with single distribution of the given OS.
>> +# Result of the routine is the rpm package for YUM repository with
>> +# file structure equal to the Centos/Fedora:
>> +#  http://mirror.centos.org/centos/7/os/x86_64/Packages/
>> +#  http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
>> +function pack_rpm {
>> +    if ! ls $repo/*.rpm >/dev/null ; then
>> +        echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
>> +        usage
>> +        exit 1
>> +    fi
>> +
>> +    # prepare the workspace
>> +    prepare_ws ${os}_${option_dist}
>> +
>> +    # copy the needed package binaries to the workspace
>> +    cp $repo/*.rpm $ws/.
>> +
>> +    pushd $ws
>> +
>> +    # set the paths
>> +    if [ "$os" == "el" ]; then
>> +        repopath=$option_dist/os/x86_64
>> +        rpmpath=Packages
>> +    elif [ "$os" == "fedora" ]; then
>> +        repopath=releases/$option_dist/Everything/x86_64/os
>> +        rpmpath=Packages/$proddir
>> +    fi
>> +    packpath=$repopath/$rpmpath
>> +
>> +    # prepare local repository with packages
>> +    $mk_dir $packpath
>> +    mv *.rpm $packpath/.
>> +    cd $repopath
>> +
>> +    # copy the current metadata files from S3
>> +    mkdir repodata.base
>> +    for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
>> +        $aws ls $bucket_path/$repopath/repodata/$file || continue
>> +        $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
>> +    done
>> +
>> +    # create the new repository metadata files
>> +    createrepo --no-database --update --workers=2 \
>> +        --compress-type=gz --simple-md-filenames .
>> +    mv repodata repodata.adding
>> +
>> +    # merge metadata files
>> +    mkdir repodata
>> +    head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
>> +    for file in filelists.xml other.xml primary.xml ; do
>> +        # 1. take the 1st line only - to skip the line with
>> +        #    number of packages which is not needed
>> +        zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
>> +        # 2. take 2nd line with metadata tag and update
>> +        #    the packages number in it
>> +        packsold=0
>> +        if [ -f repodata.base/$file.gz ] ; then
>> +            packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
>> +                tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
>> +        fi
>> +        packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
>> +            tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
>> +        packs=$(($packsold+$packsnew))
>> +        zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
>> +            sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
>> +        # 3. take only 'package' tags from new file
>> +        zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
>> +            >>repodata/$file
>> +        # 4. take only 'package' tags from old file if exists
>> +        if [ -f repodata.base/$file.gz ] ; then
>> +            zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
>> +                >>repodata/$file
>> +        fi
>> +        # 5. take the last closing line with metadata tag
>> +        zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
>> +
>> +        # get the new data
>> +        chsnew=$(sha256sum repodata/$file | awk '{print $1}')
>> +        sz=$(stat --printf="%s" repodata/$file)
>> +        gzip repodata/$file
>> +        chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
>> +        szgz=$(stat --printf="%s" repodata/$file.gz)
>> +        timestamp=$(date +%s -r repodata/$file.gz)
>> +
>> +        # add info to repomd.xml file
>> +        name=$(echo $file | sed 's#\.xml$##g')
>> +        cat <<EOF >>repodata/repomd.xml
>> +<data type="$name">
>> +  <checksum type="sha256">$chsgznew</checksum>
>> +  <open-checksum type="sha256">$chsnew</open-checksum>
>> +  <location href="repodata/$file.gz"/>
>> +  <timestamp>$timestamp</timestamp>
>> +  <size>$szgz</size>
>> +  <open-size>$sz</open-size>
>> +</data>"
>> +EOF
>> +    done
>> +    tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
>> +    gpg --detach-sign --armor repodata/repomd.xml
>> +
>> +    # copy the packages to S3
>> +    for file in $rpmpath/*.rpm ; do
>> +        $aws_cp_public $file "$bucket_path/$repopath/$file"
>> +    done
>> +
>> +    # update the metadata at the S3
>> +    $aws_sync_public repodata "$bucket_path/$repopath/repodata"
>> +
>> +    # unlock the publishing
>> +    $rm_file $ws_lockfile
>> +
>> +    popd
>> +}
>> +
>> +if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
>> +    pack_deb
>> +elif [ "$os" == "el" -o "$os" == "fedora" ]; then
>> +    pack_rpm
>> +else
>> +    echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
>> +    usage
>> +    exit 1
>> +fi
>> -- 
>> 2.17.1
>> 
>
>[1]:  https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013676.html
>
>-- 
>Best regards,
>IM


-- 
Alexander Tikhonov