[Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Alexander V. Tikhonov]

Added ability to store packages additionaly at MCS S3.
The target idea was to add the new way of packages creation at MCS S3,
which temporary duplicates the packaging at PackageCloud by Packpack
tool. Also it was needed to pack the binaries in the native way of the
each packing OS style. It was created standalone script for adding the
packages binaries/sources to MCS S3 at DEB either RPM repositories:
'tools/add_pack_s3_repo.sh'
Common parts of the script are:
 - create new meta files for the new binaries
 - copy new binaries to MCS S3
 - get previous metafiles from MCS S3 and merge the new meta data for
the new binaries
 - update the meta files at MCS S3
Different parts:
 - DEB script part based on reprepro external tool, also it works separately
only on OS versions level - it means that meta data it updates for all
Distritbutions together.
 - RPM script part based on createrepo external tool, also it works
separately for OS/Release level - it means that meta data it updates
for all Releases separately.

Closes #3380
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
Issue: https://github.com/tarantool/tarantool/issues/3380

v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013588.html
v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html

Changes v5:
- code style
- commits squashed
- rebased to master

Changes v4:
- minor corrections

Changes v3:
- common code parts merged to standalone routines
- corrected code style, minor updates
- script is ready for release

Changes v2:
- made changes in script from draft to pre-release stages

 .gitlab-ci.yml            |   5 +-
 .gitlab.mk                |  20 +-
 .travis.mk                |  41 ++-
 tools/add_pack_s3_repo.sh | 522 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 562 insertions(+), 26 deletions(-)
 create mode 100755 tools/add_pack_s3_repo.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3af5a3c8a..9dc7cfe1c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -237,7 +237,10 @@ debian_10:
     DIST: 'buster'
 
 static_build:
-  <<: *deploy_test_definition
+  <<: *release_only_definition
+  stage: test
+  tags:
+    - deploy_test
   variables:
     RUN_TESTS: 'ON'
   script:
diff --git a/.gitlab.mk b/.gitlab.mk
index 48a92e518..64664c64f 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -98,13 +98,27 @@ vms_test_%:
 vms_shutdown:
 	VBoxManage controlvm ${VMS_NAME} poweroff
 
-# ########################
-# Build RPM / Deb packages
-# ########################
+# ###########################
+# Sources tarballs & packages
+# ###########################
+
+# Push alpha and beta versions to <major>x bucket (say, 2x),
+# stable to <major>.<minor> bucket (say, 2.2).
+GIT_DESCRIBE=$(shell git describe HEAD)
+MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
+MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
+BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
+ifeq ($(MINOR_VERSION),0)
+BUCKET=$(MAJOR_VERSION)x
+endif
+ifeq ($(MINOR_VERSION),1)
+BUCKET=$(MAJOR_VERSION)x
+endif
 
 package: git_submodule_update
 	git clone https://github.com/packpack/packpack.git packpack
 	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
+	./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build
 
 # ############
 # Static build
diff --git a/.travis.mk b/.travis.mk
index 42969ff56..a85f71ced 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -8,10 +8,6 @@ MAX_FILES?=65534
 
 all: package
 
-package:
-	git clone https://github.com/packpack/packpack.git packpack
-	./packpack/packpack
-
 test: test_$(TRAVIS_OS_NAME)
 
 # Redirect some targets via docker
@@ -176,34 +172,35 @@ test_freebsd_no_deps: build_freebsd
 
 test_freebsd: deps_freebsd test_freebsd_no_deps
 
-####################
-# Sources tarballs #
-####################
-
-source:
-	git clone https://github.com/packpack/packpack.git packpack
-	TARBALL_COMPRESSOR=gz packpack/packpack tarball
+###############################
+# Sources tarballs & packages #
+###############################
 
 # Push alpha and beta versions to <major>x bucket (say, 2x),
 # stable to <major>.<minor> bucket (say, 2.2).
-ifeq ($(TRAVIS_BRANCH),master)
 GIT_DESCRIBE=$(shell git describe HEAD)
 MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
 MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
-else
-MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
-MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
-endif
-BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
+BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
 ifeq ($(MINOR_VERSION),0)
-BUCKET=tarantool.$(MAJOR_VERSION)x.src
+BUCKET=$(MAJOR_VERSION)x
 endif
 ifeq ($(MINOR_VERSION),1)
-BUCKET=tarantool.$(MAJOR_VERSION)x.src
+BUCKET=$(MAJOR_VERSION)x
 endif
 
+packpack_prepare:
+	git clone https://github.com/packpack/packpack.git packpack
+
+package: packpack_prepare
+	./packpack/packpack
+
+source: packpack_prepare
+	TARBALL_COMPRESSOR=gz packpack/packpack tarball
+
 source_deploy:
 	pip install awscli --user
-	aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
-		cp build/*.tar.gz "s3://${BUCKET}/" \
-		--acl public-read
+	for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
+		aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
+			cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
+			--acl public-read
diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
new file mode 100755
index 000000000..f6e00e067
--- /dev/null
+++ b/tools/add_pack_s3_repo.sh
@@ -0,0 +1,522 @@
+#!/bin/bash
+set -e
+
+rm_file='rm -f'
+rm_dir='rm -rf'
+mk_dir='mkdir -p'
+ws_prefix=/tmp/tarantool_repo_s3
+
+alloss='ubuntu debian el fedora'
+product=tarantool
+# the path with binaries either repository
+repo=.
+
+function get_os_dists {
+    os=$1
+    alldists=
+
+    if [ "$os" == "ubuntu" ]; then
+        alldists='trusty xenial bionic cosmic disco eoan'
+    elif [ "$os" == "debian" ]; then
+        alldists='jessie stretch buster'
+    elif [ "$os" == "el" ]; then
+        alldists='6 7 8'
+    elif [ "$os" == "fedora" ]; then
+        alldists='27 28 29 30 31'
+    fi
+
+    echo "$alldists"
+}
+
+function prepare_ws {
+    # temporary lock the publication to the repository
+    ws_suffix=$1
+    ws=${ws_prefix}_${branch}_${ws_suffix}
+    ws_lockfile=${ws}.lock
+    if [ -f $ws_lockfile ]; then
+        old_proc=$(cat $ws_lockfile)
+    fi
+    lockfile -l 60 $ws_lockfile
+    chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
+    if [ "$old_proc" != ""  -a "$old_proc" != "0" ]; then
+        kill -9 $old_proc >/dev/null 2>&1 || true
+    fi
+
+    # create temporary workspace with repository copy
+    $rm_dir $ws
+    $mk_dir $ws
+}
+
+function usage {
+    cat <<EOF
+Usage for store package binaries from the given path:
+    $0 -b=<S3 bucket> -o=<OS name> -d=<OS distribuition> [-p=<product>] <path to package binaries>
+
+Usage for mirroring Debian|Ubuntu OS repositories:
+    $0 -b=<S3 bucket> -o=<OS name> [-p=<product>] <path to 'pool' subdirectory with packages repository>
+
+Arguments:
+    <path>
+         Path points to the directory with deb/prm packages to be used.
+         Script can be used in one of 2 modes:
+          - path with binaries packages for a single distribution
+          - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
+                /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os
+
+Options:
+    -b|--bucket
+        MCS S3 bucket which will be used for storing the packages.
+        Name of the package one of the appropriate Tarantool branch:
+            master: 2x
+            2.3: 2_3
+            2.2: 2_2
+            1.10: 1_10
+    -o|--os
+        OS to be checked, one of the list:
+            $alloss
+    -d|--distribution
+        Distribution appropriate to the given OS:
+EOF
+    for os in $alloss ; do
+        echo "            $os: <"$(get_os_dists $os)">"
+    done
+    cat <<EOF
+    -p|--product
+         Product name to be packed with, default name is 'tarantool'
+    -h|--help
+         Usage help message
+EOF
+}
+
+for i in "$@"
+do
+case $i in
+    -b=*|--bucket=*)
+    branch="${i#*=}"
+    if echo "$branch" | grep -qvP '^(1_10|2(x|_[2-4]))$' ; then
+        echo "ERROR: bucket '$branch' is not supported"
+        usage
+        exit 1
+    fi
+    shift # past argument=value
+    ;;
+    -o=*|--os=*)
+    os="${i#*=}"
+    if ! echo $alloss | grep -F -q -w $os ; then
+        echo "ERROR: OS '$os' is not supported"
+        usage
+        exit 1
+    fi
+    shift # past argument=value
+    ;;
+    -d=*|--distribution=*)
+    option_dist="${i#*=}"
+    shift # past argument=value
+    ;;
+    -p=*|--product=*)
+    product="${i#*=}"
+    shift # past argument=value
+    ;;
+    -h|--help)
+    usage
+    exit 0
+    ;;
+    *)
+    repo="${i#*=}"
+    pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
+    shift # past argument=value
+    ;;
+esac
+done
+
+# check that all needed options were set
+if [ "$branch" == "" ]; then
+    echo "ERROR: need to set -b|--bucket bucket option, check usage"
+    usage
+    exit 1
+fi
+if [ "$os" == "" ]; then
+    echo "ERROR: need to set -o|--os OS name option, check usage"
+    usage
+    exit 1
+fi
+alldists=$(get_os_dists $os)
+if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
+    echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
+    usage
+    exit 1
+fi
+
+# set the subpath with binaries based on literal character of the product name
+proddir=$(echo $product | head -c 1)
+
+# AWS defines
+aws='aws --endpoint-url https://hb.bizmrg.com s3'
+bucket_path="s3://tarantool_repo/$branch/$os"
+aws_cp_public="$aws cp --acl public-read"
+aws_sync_public="$aws sync --acl public-read"
+
+function update_deb_packfile {
+    packfile=$1
+    packtype=$2
+    update_dist=$3
+
+    locpackfile=$(echo $packfile | sed "s#^$ws/##g")
+    # register DEB/DSC pack file to Packages/Sources file
+    reprepro -Vb . include$packtype $update_dist $packfile
+    # reprepro copied DEB/DSC file to component which is not needed
+    $rm_dir $debdir/$component
+    # to have all sources avoid reprepro set DEB/DSC file to its own registry
+    $rm_dir db
+}
+
+function update_deb_metadata {
+    packpath=$1
+    packtype=$2
+
+    if [ ! -f $packpath.saved ] ; then
+        # get the latest Sources file from S3
+        $aws ls "$bucket_path/$packpath" 2>/dev/null && \
+            $aws cp "$bucket_path/$packpath" $packpath.saved || \
+            touch $packpath.saved
+    fi
+
+    if [ "$packtype" == "dsc" ]; then
+        # WORKAROUND: unknown why, but reprepro doesn`t save the Sources file
+        gunzip -c $packpath.gz >$packpath
+        # check if the DSC file already exists in Sources from S3
+        hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
+            tail -n 1 | awk '{print $1}')
+        if grep " $hash .*\.dsc$" $packpath.saved ; then
+            echo "WARNING: DSC file already registered in S3!"
+            return 1
+        fi
+    elif [ "$packtype" == "deb" ]; then
+        # check if the DEB file already exists in Packages from S3
+        if grep "^$(grep '^SHA256: ' $packages)$" $packages.saved ; then
+            echo "WARNING: DEB file already registered in S3!"
+            return 1
+        fi
+    fi
+    # store the new DEB entry
+    cat $packpath >>$packpath.saved
+}
+
+# The 'pack_deb' function especialy created for DEB packages. It works
+# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
+# tool 'reprepro' from:
+#     https://wiki.debian.org/DebianRepository/SetupWithReprepro
+# This tool works with complete number of distributions of the given OS.
+# Result of the routine is the debian package for APT repository with
+# file structure equal to the Debian/Ubuntu:
+#     http://ftp.am.debian.org/debian/pool/main/t/tarantool/
+#     http://ftp.am.debian.org/ubuntu/pool/main/t/
+function pack_deb {
+    # we need to push packages into 'main' repository only
+    component=main
+
+    # debian has special directory 'pool' for packages
+    debdir=pool
+
+    # get packages from pointed location
+    if [ ! -d $repo/$debdir ] && \
+        ( [ "$option_dist" == "" ] || \
+            ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ) ; then
+        echo "ERROR: Current '$repo' path doesn't have any of the following:"
+        echo " - $0 run option '-d' and DEB packages in path"
+        echo " - 'pool' subdirectory with APT repositories"
+        usage
+        exit 1
+    fi
+
+    # prepare the workspace
+    prepare_ws ${os}
+
+    # script works in one of 2 modes:
+    # - path with binaries packages for a single distribution
+    # - path with 'pool' directory with APT repository
+    if [ "$option_dist" != "" ] && \
+            ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null 2>&1 ; then
+        # copy single distribution with binaries packages
+        repopath=$ws/pool/${option_dist}/$component/$proddir/$product
+        $mk_dir ${repopath}
+        cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
+    elif [ -d $repo/$debdir ]; then
+        # copy 'pool' directory with APT repository
+        cp -rf $repo/$debdir $ws/.
+    else
+        echo "ERROR: neither distribution option '-d' with files $repo/*.deb $repo/*.dsc $repo/*.tar.*z set nor '$repo/$debdir' path found"
+        usage
+        $rm_file $wslock
+        exit 1
+    fi
+
+    pushd $ws
+
+    # create the configuration file for 'reprepro' tool
+    confpath=$ws/conf
+    $rm_dir $confpath
+    $mk_dir $confpath
+
+    for loop_dist in $alldists ; do
+        cat <<EOF >>$confpath/distributions
+Origin: Tarantool
+Label: tarantool.org
+Suite: stable
+Codename: $loop_dist
+Architectures: amd64 source
+Components: $component
+Description: Unofficial Ubuntu Packages maintained by Tarantool
+SignWith: 91B625E5
+DebIndices: Packages Release . .gz .bz2
+UDebIndices: Packages . .gz .bz2
+DscIndices: Sources Release .gz .bz2
+
+EOF
+    done
+
+    # create standalone repository with separate components
+    for loop_dist in $alldists ; do
+        echo ================ DISTRIBUTION: $loop_dist ====================
+        updated_deb=0
+        updated_dsc=0
+
+        # 1(binaries). use reprepro tool to generate Packages file
+        for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
+            [ -f $deb ] || continue
+            # regenerate DEB pack
+            update_deb_packfile $deb deb $loop_dist
+            echo "Regenerated DEB file: $locpackfile"
+            for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+                # copy Packages file to avoid of removing by the new DEB version
+                # update metadata 'Packages' files
+                ! update_deb_metadata $packages deb || continue
+                updated_deb=1
+            done
+            # save the registered DEB file to S3
+            if [ "$updated_deb" == 1 ]; then
+                $aws_cp_public $deb $bucket_path/$locpackfile
+            fi
+        done
+
+        # 1(sources). use reprepro tool to generate Sources file
+        for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
+            [ -f $dsc ] || continue
+            # regenerate DSC pack
+            update_deb_packfile $dsc dsc $loop_dist
+            echo "Regenerated DSC file: $locpackfile"
+            # copy Sources file to avoid of removing by the new DSC version
+            # update metadata 'Sources' file
+            ! update_deb_metadata dists/$loop_dist/$component/source/Sources dsc \
+                || continue
+            updated_dsc=1
+            # save the registered DSC file to S3
+            $aws_cp_public $dsc $bucket_path/$locpackfile
+            tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
+            $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
+            orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
+            $aws_cp_public $ws/$orig "$bucket_path/$orig"
+        done
+
+        # check if any DEB/DSC files were newly registered
+        [ "$updated_deb" == "0" -a "$updated_dsc" == "0" ] && \
+            continue || echo "Updating dists"
+
+        # finalize the Packages file
+        for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+            mv $packages.saved $packages
+        done
+
+        # 2(binaries). update Packages file archives
+        for packpath in dists/$loop_dist/$component/binary-* ; do
+            pushd $packpath
+            sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
+            bzip2 -c Packages >Packages.bz2
+            gzip -c Packages >Packages.gz
+            popd
+        done
+
+        # 2(sources). update Sources file archives
+        pushd dists/$loop_dist/$component/source
+        sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
+        bzip2 -c Sources >Sources.bz2
+        gzip -c Sources >Sources.gz
+        popd
+
+        # 3. update checksums entries of the Packages* files in *Release files
+        # NOTE: it is stable structure of the *Release files when the checksum
+        #       entries in it in the following way:
+        # MD5Sum:
+        #  <checksum> <size> <file orig>
+        #  <checksum> <size> <file debian>
+        # SHA1:
+        #  <checksum> <size> <file orig>
+        #  <checksum> <size> <file debian>
+        # SHA256:
+        #  <checksum> <size> <file orig>
+        #  <checksum> <size> <file debian>
+        #       The script bellow puts 'md5' value at the 1st found file entry,
+        #       'sha1' - at the 2nd and 'sha256' at the 3rd
+        pushd dists/$loop_dist
+        for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
+            sz=$(stat -c "%s" $file)
+            md5=$(md5sum $file | awk '{print $1}')
+            sha1=$(sha1sum $file | awk '{print $1}')
+            sha256=$(sha256sum $file | awk '{print $1}')
+            awk 'BEGIN{c = 0} ; {
+                if ($3 == p) {
+                    c = c + 1
+                    if (c == 1) {print " " md  " " s " " p}
+                    if (c == 2) {print " " sh1 " " s " " p}
+                    if (c == 3) {print " " sh2 " " s " " p}
+                } else {print $0}
+            }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
+                    Release >Release.new
+            mv Release.new Release
+        done
+        # resign the selfsigned InRelease file
+        $rm_file InRelease
+        gpg --clearsign -o InRelease Release
+        # resign the Release file
+        $rm_file Release.gpg
+        gpg -abs -o Release.gpg Release
+        popd
+
+        # 4. sync the latest distribution path changes to S3
+        $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
+    done
+
+    # unlock the publishing
+    $rm_file $ws_lockfile
+
+    popd
+}
+
+# The 'pack_rpm' function especialy created for RPM packages. It works
+# with RPM packing OS like Centos, Fedora. It is based on globaly known
+# tool 'createrepo' from:
+#     https://linux.die.net/man/8/createrepo
+# This tool works with single distribution of the given OS.
+# Result of the routine is the rpm package for YUM repository with
+# file structure equal to the Centos/Fedora:
+#     http://mirror.centos.org/centos/7/os/x86_64/Packages/
+#     http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
+function pack_rpm {
+    if ! ls $repo/*.rpm >/dev/null 2>&1 ; then
+        echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
+        usage
+        exit 1
+    fi
+
+    # prepare the workspace
+    prepare_ws ${os}_${option_dist}
+
+    # copy the needed package binaries to the workspace
+    cp $repo/*.rpm $ws/.
+
+    pushd $ws
+
+    # set the paths
+    if [ "$os" == "el" ]; then
+        repopath=$option_dist/os/x86_64
+        rpmpath=Packages
+    elif [ "$os" == "fedora" ]; then
+        repopath=releases/$option_dist/Everything/x86_64/os
+        rpmpath=Packages/$proddir
+    fi
+    packpath=$repopath/$rpmpath
+
+    # prepare local repository with packages
+    $mk_dir $packpath
+    mv *.rpm $packpath/.
+    cd $repopath
+
+    # copy the current metadata files from S3
+    mkdir repodata.base
+    for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
+        $aws ls $bucket_path/$repopath/repodata/$file || continue
+        $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
+    done
+
+    # create the new repository metadata files
+    createrepo --no-database --update --workers=2 \
+        --compress-type=gz --simple-md-filenames .
+    mv repodata repodata.adding
+
+    # merge metadata files
+    mkdir repodata
+    head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
+    for file in filelists.xml other.xml primary.xml ; do
+        # 1. take the 1st line only - to skip the line with
+        #    number of packages which is not needed
+        zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
+        # 2. take 2nd line with metadata tag and update
+        #    the packages number in it
+        packsold=0
+        if [ -f repodata.base/$file.gz ] ; then
+            packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
+                tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+        fi
+        packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
+            tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+        packs=$(($packsold+$packsnew))
+        zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
+            sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
+        # 3. take only 'package' tags from new file
+        zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
+            >>repodata/$file
+        # 4. take only 'package' tags from old file if exists
+        if [ -f repodata.base/$file.gz ] ; then
+            zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
+                >>repodata/$file
+        fi
+        # 5. take the last closing line with metadata tag
+        zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
+
+        # get the new data
+        chsnew=$(sha256sum repodata/$file | awk '{print $1}')
+        sz=$(stat --printf="%s" repodata/$file)
+        gzip repodata/$file
+        chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
+        szgz=$(stat --printf="%s" repodata/$file.gz)
+        timestamp=$(date +%s -r repodata/$file.gz)
+
+        # add info to repomd.xml file
+        name=$(echo $file | sed 's#\.xml$##g')
+        cat <<EOF >>repodata/repomd.xml
+<data type="$name">
+  <checksum type="sha256">$chsgznew</checksum>
+  <open-checksum type="sha256">$chsnew</open-checksum>
+  <location href="repodata/$file.gz"/>
+  <timestamp>$timestamp</timestamp>
+  <size>$szgz</size>
+  <open-size>$sz</open-size>
+</data>"
+EOF
+    done
+    tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
+    gpg --detach-sign --armor repodata/repomd.xml
+
+    # copy the packages to S3
+    for file in $rpmpath/*.rpm ; do
+        $aws_cp_public $file "$bucket_path/$repopath/$file"
+    done
+
+    # update the metadata at the S3
+    $aws_sync_public repodata "$bucket_path/$repopath/repodata"
+
+    # unlock the publishing
+    $rm_file $ws_lockfile
+
+    popd
+}
+
+if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
+    pack_deb
+elif [ "$os" == "el" -o "$os" == "fedora" ]; then
+    pack_rpm
+else
+    echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
+    usage
+    exit 1
+fi
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Alexander Turenko]

I discussed what we're expect from rpm/deb repositories with Igor,
Alexander T., Kirill Yu. I'll summarize key points below.

CCed Konstantin N., Yaroslav D. and Maxim M.

## Planned activities inside tarantool repository

* We have packagecloud repositories and we're going to keep them and
  continue update them (at least for all versions below 2.4 inclusive).
* We'll keep redirects from download.tarantool.org to packagecloud for
  those 1.10/2.1/2.2/2.3/2.4 repositories.
* We'll create a new repository (working name is 'nightly'), which will
  contain packages from all long-term branches (1.10/2.1/.../master).
  Package names will be tarantool110, tarantool21, tarantool22,
  tarantool23, tarantool24.
  - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
    Should <sepX> be the same as <sepY>? Consider that it should work
    smooothly with apt and yum repos both. (AR Alexander Tikh.)
* In addition to tarantool110 and so we'll deploy a package with name
  tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
  development.

## Necessary activities outside of tarantool repository

* Manually deploy existing packages:
  - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
    scheme. Maybe also all tagged versions. The rebuild should not
    change `git describe --tags --long` output (including a commit
    hash).
  - Copy all module / connector packages from packagecloud.
    - TBD: Propose exact list (AR Alexander Tikh.)
* Setup proper redirects from download.tarantool.org for the new
  repository.
* Update CI / CD for modules (at least for most used ones) to use this
  script: download it from the tarantool repository and use to deploy.

After that we can announce the new repository.

## Further activities

Announce the new repository and deprecation plan of old ones (planned to
fully support the old repos for 2.4 and below, but don't add 2.5 and
further repos).

Update download instructions on the website (see [1]) to use the new
repository.

Provide the similar repository, but only for tagged builds (working name
is 'release'). It seems it worth to push only tagged modules /
connectors as well. (Asked by Konstantin N.) 

We maybe (if there will be a demand) will move the script outside of the
repository and integrate with the existing server [2] that doing quite
similar job: it receives built rocks (it is like python's wheels) and
updates a repository for luarocks / tarantoolctl rocks. This can be done
w/o user-visible changes. (Proposed by Konstantin N.)

Aren't I forget anything?

[1]: https://www.tarantool.io/en/download/
[2]: https://github.com/tarantool/rocks.tarantool.org

----

Re patch itself: I don't look into the script and it seems it would be
not productive. Just noted several points around, see below.

WBR, Alexander Turenko.

> gitlab-ci: implement packing into MCS S3
>

A few words why this is needed would be valuable, I think. Our main
motivation is to provide repositories, where packages are not pruned, so
a user may enable the repository and fix certain version of tarantool in
dependencies of an application package.

> Added ability to store packages additionaly at MCS S3.

Nit: I suggest to split paragraphs with empty lines for readability.

> The target idea was to add the new way of packages creation at MCS S3,
> which temporary duplicates the packaging at PackageCloud by Packpack
> tool. Also it was needed to pack the binaries in the native way of the
> each packing OS style. It was created standalone script for adding the
> packages binaries/sources to MCS S3 at DEB either RPM repositories:
> 'tools/add_pack_s3_repo.sh'
> Common parts of the script are:
>  - create new meta files for the new binaries
>  - copy new binaries to MCS S3
>  - get previous metafiles from MCS S3 and merge the new meta data for
> the new binaries

Nit: I suggest to format those lists like so:

 | - get previous metafiles from MCS S3 and merge the new meta data for
 |   the new binaries
 |   ^^ (carried line is indented as a text above it)

I also think that when a text im separated from a list with empty lines
this is more readable:

 | Look at the following items:
 |
 | - item 1
 | - item 2
 |
 | Lorem ipsum.

>  - update the meta files at MCS S3
> Different parts:
>  - DEB script part based on reprepro external tool, also it works separately
> only on OS versions level - it means that meta data it updates for all
> Distritbutions together.

Typo: Distri*t*butions.

>  - RPM script part based on createrepo external tool, also it works
> separately for OS/Release level - it means that meta data it updates
> for all Releases separately.
> 
> Closes #3380

Let's include docbot comment to don't forget to update download
instructions on the website.

>  static_build:
> -  <<: *deploy_test_definition
> +  <<: *release_only_definition
> +  stage: test
> +  tags:
> +    - deploy_test
>    variables:
>      RUN_TESTS: 'ON'
>    script:

This change seems to be out of scope of the commit, as I see. (To be
honest, I don't understand why it is necessary.)

> diff --git a/.gitlab.mk b/.gitlab.mk
> index 48a92e518..64664c64f 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -98,13 +98,27 @@ vms_test_%:
>  vms_shutdown:
>  	VBoxManage controlvm ${VMS_NAME} poweroff
>  
> -# ########################
> -# Build RPM / Deb packages
> -# ########################
> +# ###########################
> +# Sources tarballs & packages
> +# ###########################
> +
> +# Push alpha and beta versions to <major>x bucket (say, 2x),
> +# stable to <major>.<minor> bucket (say, 2.2).
> +GIT_DESCRIBE=$(shell git describe HEAD)
> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
> +ifeq ($(MINOR_VERSION),0)
> +BUCKET=$(MAJOR_VERSION)x
> +endif
> +ifeq ($(MINOR_VERSION),1)
> +BUCKET=$(MAJOR_VERSION)x
> +endif
>  
>  package: git_submodule_update
>  	git clone https://github.com/packpack/packpack.git packpack
>  	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
> +	./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build

So, when we want to test a feature / bugfix branch with `*-full-ci`
branch name, packages will be pushed and available for a user. This
seems to be mistake.

>  # Push alpha and beta versions to <major>x bucket (say, 2x),
>  # stable to <major>.<minor> bucket (say, 2.2).
> -ifeq ($(TRAVIS_BRANCH),master)
>  GIT_DESCRIBE=$(shell git describe HEAD)
>  MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>  MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
> -else
> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
> -endif
> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>  ifeq ($(MINOR_VERSION),0)
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
> +BUCKET=$(MAJOR_VERSION)x
>  endif
>  ifeq ($(MINOR_VERSION),1)
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
> +BUCKET=$(MAJOR_VERSION)x
>  endif

AFAIU it is purely for testing: deployment should occur only from 1.10,
2.1, 2.2, ..., master branches. In any other case it is does not matter
which values those variable have. I would discard this change.

>  
> +packpack_prepare:
> +	git clone https://github.com/packpack/packpack.git packpack
> +
> +package: packpack_prepare
> +	./packpack/packpack
> +
> +source: packpack_prepare
> +	TARBALL_COMPRESSOR=gz packpack/packpack tarball
> +
>  source_deploy:
>  	pip install awscli --user
> -	aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
> -		cp build/*.tar.gz "s3://${BUCKET}/" \
> -		--acl public-read
> +	for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
> +		aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
> +			cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
> +			--acl public-read

It was not possible to have several source tarballs for one commit. Now
it is not so? This commit should not touch source tarballs, I think: it
is about rpm/deb packages.

> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
> new file mode 100755
> index 000000000..f6e00e067
> --- /dev/null
> +++ b/tools/add_pack_s3_repo.sh

'add pack s3 repo' looks as a bag of words. Just mkrepo.sh looks better for me.
sync_repo.sh or update_repo.sh are also okay. Let's choose one that is
more or less fit English grammar.

> +Arguments:
> +    <path>
> +         Path points to the directory with deb/prm packages to be used.
> +         Script can be used in one of 2 modes:
> +          - path with binaries packages for a single distribution
> +          - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
> +                /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os

I have two points about this mirroring support:

* It is only for APT (but not for YUM), so cannot be used in uniform
  way.
* It seems it is not something to use in CI / CD, so it is dead code
  from this point of view.

Why should we add it?

> +    for loop_dist in $alldists ; do
> +        cat <<EOF >>$confpath/distributions
> +Origin: Tarantool
> +Label: tarantool.org
> +Suite: stable
> +Codename: $loop_dist
> +Architectures: amd64 source
> +Components: $component
> +Description: Unofficial Ubuntu Packages maintained by Tarantool

Why unofficial?

Why Ubuntu? It is for Debian too, right?

'Tarantool DBMS and Tarantool modules' is enough, I think.

> +SignWith: 91B625E5

Shouldn't it be an argument / environment variable? How the key is
stored?

I guess the right way would be like [3] (I don't mean using Travis-CI,
it is just example I have near to hands). The key point is to store
sensitive data encrypted within the repository, but use secret
environment variables to decrypt it in CI / CD.

[3]: https://docs.travis-ci.com/user/encrypting-files

Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Alexander Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 11520 bytes --]




>Вторник, 14 января 2020, 20:42 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>I discussed what we're expect from rpm/deb repositories with Igor,
>Alexander T., Kirill Yu. I'll summarize key points below.
>
>CCed Konstantin N., Yaroslav D. and Maxim M.
>
>## Planned activities inside tarantool repository
>
>* We have packagecloud repositories and we're going to keep them and
>  continue update them (at least for all versions below 2.4 inclusive).
>* We'll keep redirects from download.tarantool.org to packagecloud for
>  those 1.10/2.1/2.2/2.3/2.4 repositories.
>* We'll create a new repository (working name is 'nightly'), which will
>  contain packages from all long-term branches (1.10/2.1/.../master).
>  Package names will be tarantool110, tarantool21, tarantool22,
>  tarantool23, tarantool24.
>  - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
>    Should <sepX> be the same as <sepY>? Consider that it should work
>    smooothly with apt and yum repos both. (AR Alexander Tikh.)
>* In addition to tarantool110 and so we'll deploy a package with name
>  tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
>  development.
>
>## Necessary activities outside of tarantool repository
>
>* Manually deploy existing packages:
>  - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
>    scheme. Maybe also all tagged versions. The rebuild should not
>    change `git describe --tags --long` output (including a commit
>    hash).
>  - Copy all module / connector packages from packagecloud.
>    - TBD: Propose exact list (AR Alexander Tikh.)
>* Setup proper redirects from download.tarantool.org for the new
>  repository.
>* Update CI / CD for modules (at least for most used ones) to use this
>  script: download it from the tarantool repository and use to deploy.
>
>After that we can announce the new repository.
>
>## Further activities
>
>Announce the new repository and deprecation plan of old ones (planned to
>fully support the old repos for 2.4 and below, but don't add 2.5 and
>further repos).
>
>Update download instructions on the website (see [1]) to use the new
>repository.
>
>Provide the similar repository, but only for tagged builds (working name
>is 'release'). It seems it worth to push only tagged modules /
>connectors as well. (Asked by Konstantin N.) 
>
>We maybe (if there will be a demand) will move the script outside of the
>repository and integrate with the existing server [2] that doing quite
>similar job: it receives built rocks (it is like python's wheels) and
>updates a repository for luarocks / tarantoolctl rocks. This can be done
>w/o user-visible changes. (Proposed by Konstantin N.)
>
>Aren't I forget anything?
>
>[1]:  https://www.tarantool.io/en/download/
>[2]:  https://github.com/tarantool/rocks.tarantool.org
>
>----
>
>Re patch itself: I don't look into the script and it seems it would be
>not productive. Just noted several points around, see below.
>
>WBR, Alexander Turenko.
>
>> gitlab-ci: implement packing into MCS S3
>>
>
>A few words why this is needed would be valuable, I think. Our main
>motivation is to provide repositories, where packages are not pruned, so
>a user may enable the repository and fix certain version of tarantool in
>dependencies of an application package.
Commit message improved - added information that the packages not pruning
and that user may use certain version of Tarantool  in dependencies of an
application package.
>
>> Added ability to store packages additionally at MCS S3.
>
>Nit: I suggest to split paragraphs with empty lines for readability. 
Done.
>
>
>> The target idea was to add the new way of packages creation at MCS S3,
>> which temporary duplicates the packaging at PackageCloud by Packpack
>> tool. Also it was needed to pack the binaries in the native way of the
>> each packing OS style. It was created standalone script for adding the
>> packages binaries/sources to MCS S3 at DEB either RPM repositories:
>> 'tools/add_pack_s3_repo.sh'
>> Common parts of the script are:
>>  - create new meta files for the new binaries
>>  - copy new binaries to MCS S3
>>  - get previous metafiles from MCS S3 and merge the new meta data for
>> the new binaries
>
>Nit: I suggest to format those lists like so:
>
> | - get previous metafiles from MCS S3 and merge the new meta data for
> |   the new binaries
> |   ^^ (carried line is indented as a text above it) 
Done.
>
>
>I also think that when a text im separated from a list with empty lines
>this is more readable:
>
> | Look at the following items:
> |
> | - item 1
> | - item 2
> |
> | Lorem ipsum.
Done.
>
>>  - update the meta files at MCS S3
>> Different parts:
>>  - DEB script part based on reprepro external tool, also it works separately
>> only on OS versions level - it means that meta data it updates for all
>> Distritbutions together.
>
>Typo: Distri*t*butions. 
Fixed.
>
>
>>  - RPM script part based on createrepo external tool, also it works
>> separately for OS/Release level - it means that meta data it updates
>> for all Releases separately.
>> 
>> Closes #3380
>
>Let's include docbot comment to don't forget to update download
>instructions on the website. 
Added short comment.
>
>
>>  static_build:
>> -  <<: *deploy_test_definition
>> +  <<: *release_only_definition
>> +  stage: test
>> +  tags:
>> +    - deploy_test
>>    variables:
>>      RUN_TESTS: 'ON'
>>    script:
>
>This change seems to be out of scope of the commit, as I see. (To be
>honest, I don't understand why it is necessary.)
Reverted as not needed here.
>
>
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index 48a92e518..64664c64f 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -98,13 +98,27 @@ vms_test_%:
>>  vms_shutdown:
>>  	VBoxManage controlvm ${VMS_NAME} poweroff
>> 
>> -# ########################
>> -# Build RPM / Deb packages
>> -# ########################
>> +# ###########################
>> +# Sources tarballs & packages
>> +# ###########################
>> +
>> +# Push alpha and beta versions to <major>x bucket (say, 2x),
>> +# stable to <major>.<minor> bucket (say, 2.2).
>> +GIT_DESCRIBE=$(shell git describe HEAD)
>> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>> +ifeq ($(MINOR_VERSION),0)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>> +ifeq ($(MINOR_VERSION),1)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>> 
>>  package: git_submodule_update
>>  	git clone  https://github.com/packpack/packpack.git packpack
>>  	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>> +	./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build
>
>So, when we want to test a feature / bugfix branch with `*-full-ci`
>branch name, packages will be pushed and available for a user. This
>seems to be mistake. 
Fixed - added standalone checks at gitlab-ci for testing branches,
also added different jobs for deploying and packing.
>
>
>>  # Push alpha and beta versions to <major>x bucket (say, 2x),
>>  # stable to <major>.<minor> bucket (say, 2.2).
>> -ifeq ($(TRAVIS_BRANCH),master)
>>  GIT_DESCRIBE=$(shell git describe HEAD)
>>  MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>>  MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> -else
>> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
>> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
>> -endif
>> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>>  ifeq ($(MINOR_VERSION),0)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>>  endif
>>  ifeq ($(MINOR_VERSION),1)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>>  endif
>
>AFAIU it is purely for testing: deployment should occur only from 1.10,
>2.1, 2.2, ..., master branches. In any other case it is does not matter
>which values those variable have. I would discard this change.
Moved changes to another standalone patch set commit.
>
>> 
>> +packpack_prepare:
>> +	git clone  https://github.com/packpack/packpack.git packpack
>> +
>> +package: packpack_prepare
>> +	./packpack/packpack
>> +
>> +source: packpack_prepare
>> +	TARBALL_COMPRESSOR=gz packpack/packpack tarball
>> +
>>  source_deploy:
>>  	pip install awscli --user
>> -	aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> -		cp build/*.tar.gz "s3://${BUCKET}/" \
>> -		--acl public-read
>> +	for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
>> +		aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> +			cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
>> +			--acl public-read
>
>It was not possible to have several source tarballs for one commit. Now
>it is not so? This commit should not touch source tarballs, I think: it
>is about rpm/deb packages. 
Moved changes to another standalone patch set commit.
>
>
>> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
>> new file mode 100755
>> index 000000000..f6e00e067
>> --- /dev/null
>> +++ b/tools/add_pack_s3_repo.sh
>
>'add pack s3 repo' looks as a bag of words. Just mkrepo.sh looks better for me.
>sync_repo.sh or update_repo.sh are also okay. Let's choose one that is
>more or less fit English grammar. 
Renamed to 'update_repo.sh'
>
>
>> +Arguments:
>> +    <path>
>> +         Path points to the directory with deb/prm packages to be used.
>> +         Script can be used in one of 2 modes:
>> +          - path with binaries packages for a single distribution
>> +          - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
>> +                /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os
>
>I have two points about this mirroring support:
>
>* It is only for APT (but not for YUM), so cannot be used in uniform
>  way.
>* It seems it is not something to use in CI / CD, so it is dead code
>  from this point of view.
>
>Why should we add it? 
We need the way for porting the packagecloud DEB and RPM packages.
For the DEB packages we use apt-mirror which creates APT repository
and for porting it to MCS S3 this additional option was enabled. For RPM
packages it is not needed, because mirroring YUM repositories results
in flat directory with packages which can be ported to MCS S3 w/o any
additional options.
>
>
>> +    for loop_dist in $alldists ; do
>> +        cat <<EOF >>$confpath/distributions
>> +Origin: Tarantool
>> +Label: tarantool.org
>> +Suite: stable
>> +Codename: $loop_dist
>> +Architectures: amd64 source
>> +Components: $component
>> +Description: Unofficial Ubuntu Packages maintained by Tarantool
>
>Why unofficial?
>
>Why Ubuntu? It is for Debian too, right?
>
>'Tarantool DBMS and Tarantool modules' is enough, I think. 
Done.
>
>
>> +SignWith: 91B625E5
>
>Shouldn't it be an argument / environment variable? How the key is
>stored? 
The key is stored at the build hosts.
>
>
>I guess the right way would be like [3] (I don't mean using Travis-CI,
>it is just example I have near to hands). The key point is to store
>sensitive data encrypted within the repository, but use secret
>environment variables to decrypt it in CI / CD.
>
>[3]:  https://docs.travis-ci.com/user/encrypting-files
I've tried a lot to make it in this way, but failed that is why I've used hosts
setup for keys. I'll reproduce the issue to show it, if I'll find the other solution
I'll provide it.

-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 16798 bytes --]

Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Igor Munkin]

Sasha,

Thanks for the discussion results you provided! Some light can be seen
at the end of the tunnel.

On 14.01.20, Alexander Turenko wrote:
> I discussed what we're expect from rpm/deb repositories with Igor,
> Alexander T., Kirill Yu. I'll summarize key points below.
> 
> CCed Konstantin N., Yaroslav D. and Maxim M.
> 
> ## Planned activities inside tarantool repository
> 
> * We have packagecloud repositories and we're going to keep them and
>   continue update them (at least for all versions below 2.4 inclusive).

Agree.

> * We'll keep redirects from download.tarantool.org to packagecloud for
>   those 1.10/2.1/2.2/2.3/2.4 repositories.

It's a tough one with both its pros and cons.
* Leaving redirects for old repos to packagecloud makes us to have
  different publishing pipelines and complicates the further
  mainteinance.
* Changing redirects to the new storage obligues users to change the
  repo key for already enabled repos.

> * We'll create a new repository (working name is 'nightly'), which will
>   contain packages from all long-term branches (1.10/2.1/.../master).
>   Package names will be tarantool110, tarantool21, tarantool22,
>   tarantool23, tarantool24.

Since the packages listed above provides the binary and all related
entities having the same path this approach leads to conflicts while
messing around with two different major versions (e.g. tarantool110 and
tarantool23) on one system. Therefore the one can't smoothly
upgrade/migrate his instance to a new version.

According to this fact I propose to make it the way similar to MySQL and
PostgreSQL do: both of them split their packages into separate repos.

### PostgreSQL

PostgreSQL installation flow is described here[1]. E.g. for Centos 7 the
maintainers provide a package with the relevant pg repos:
| $ yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
| <snipped>
| Complete!

Here you can see the package ships *all* relevant PostgreSQL repos:
| $ yum repolist enabled | grep pgdg
| pgdg10/7/x86_64       PostgreSQL 10 for RHEL/CentOS 7 - x86_64    1183
| pgdg11/7/x86_64       PostgreSQL 11 for RHEL/CentOS 7 - x86_64    1020
| pgdg12/7/x86_64       PostgreSQL 12 for RHEL/CentOS 7 - x86_64     358
| pgdg94/7/x86_64       PostgreSQL 9.4 for RHEL/CentOS 7 - x86_64   1092
| pgdg95/7/x86_64       PostgreSQL 9.5 for RHEL/CentOS 7 - x86_64   1158
| pgdg96/7/x86_64       PostgreSQL 9.6 for RHEL/CentOS 7 - x86_64   1203

There are no problems obtaining different versions on one system:
| $ yum install -y postgresql10 postgresql12
| <snipped>
| Complete!

Each package lay out its contents the way similar to Gentoo slots
approach (i.e. the paths contains parts related to the package version):
| $ rpm -ql postgresql10 | grep -P 'psql$'
| /usr/pgsql-10/bin/psql
| $ rpm -ql postgresql12 | grep -P 'psql$'
| /usr/pgsql-12/bin/psql

Furthermore every PostgreSQL version is available from its own repo:
| $ yum info postgresql10 | grep -P '^From repo'
| From repo   : pgdg10
| $ yum info postgresql12 | grep -P '^From repo'
| From repo   : pgdg12

### MySQL

MySQL installation flow is described here[2]. E.g. for Centos 7 the
maintainers provide a package with the relevant mysql repos:
| $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
| <snipped>
| Complete!

Similar to the PostgreSQL one the package ships *all* relevant MySQL
repos but the old ones are disabled (the reader can manualy check the
corresponding files in /etc/yum.repos.d directory):
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql80-community/x86_64             MySQL 8.0 Community Server    161

Whether we try to install an old one package with repos alongside we'll
see the following:
| $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
| <snipped>
| Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch

Thereby to see MySQL upgrading process let's manually remove the first
installed package and try to install the latter one again:
| $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
| <snipped>
| Complete!

Again, the package delivers the relevant MySQL repos and the reader can
manually check that all repos except the mysql57 related are disabled:
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql57-community/x86_64             MySQL 5.7 Community Server    404

As a result of the mysql-community-server installation we'll have
MySQL 5.7 software installed on the system:
| $ yum install -y mysql-community-server
| <snipped>
| Complete!
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 5.7.29
| Release     : 1.el7
| Size        : 765 M
| Repo        : installed
| From repo   : mysql57-community
| <snipped>

The layout of the package doesn't have any specific paths regarding
MySQL version:
| $ rpm -ql mysql-community-server | grep -P  mysqld$
| /usr/sbin/mysqld
| /var/run/mysqld

Attempt to upgrade the repo package fails the same way as above:
| $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
| <snipped>
| Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch

If we remove the mysql57 package the installation of the new one
succeeds:
| $ yum remove mysql57-community-release
| <snipped>
| Complete!
| $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
| <snipped>
| Complete!

When the installation finishes we can see that only the repos related to
MySQL 8.0 are enabled:
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql80-community/x86_64             MySQL 8.0 Community Server    161

However, the installed mysql-community-server is still 5.7 and 8.0
package is available now:
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 5.7.29
| Release     : 1.el7
| Size        : 765 M
| Repo        : installed
| From repo   : mysql57-community
| <snipped>
| Available Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 8.0.19
| Release     : 1.el7
| Size        : 436 M
| Repo        : mysql80-community/x86_64
| <snipped>

There are no problems with the upgrading process:
| $ yum install -y mysql-community-server
| <snipped>
| Complete!

As a result there is only a single version installed and available:
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 8.0.19
| Release     : 1.el7
| Size        : 1.9 G
| Repo        : installed
| From repo   : mysql80-community
| <snipped>

I see this way to be the most convenient approach both for maintainers
and users: a single package to be splited per versions to separated
repos. AFAICS we already follow this way, so I propose to stay with this
layout further.

>   - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
>     Should <sepX> be the same as <sepY>? Consider that it should work
>     smooothly with apt and yum repos both. (AR Alexander Tikh.)
> * In addition to tarantool110 and so we'll deploy a package with name
>   tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
>   development.

I guess we can recommend the one to use our latest repo for the bleeding
tarantool considering the name is left unchanged.

> 
> ## Necessary activities outside of tarantool repository
> 
> * Manually deploy existing packages:
>   - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
>     scheme. Maybe also all tagged versions. The rebuild should not
>     change `git describe --tags --long` output (including a commit
>     hash).

Not necessary since the naming left unchanged.

>   - Copy all module / connector packages from packagecloud.
>     - TBD: Propose exact list (AR Alexander Tikh.)

Again refer to MySQL repos layout: e.g. server, tools and connectors are
split into separate repos.

> * Setup proper redirects from download.tarantool.org for the new
>   repository.

Agree.

> * Update CI / CD for modules (at least for most used ones) to use this
>   script: download it from the tarantool repository and use to deploy.

Agree.

> 
> After that we can announce the new repository.
> 
> ## Further activities
> 
> Announce the new repository and deprecation plan of old ones (planned to
> fully support the old repos for 2.4 and below, but don't add 2.5 and
> further repos).

Agree.

> 
> Update download instructions on the website (see [1]) to use the new
> repository.

Agree though adjusted yours a bit:
| Update download instructions on the website (see [3]) to use new
| repositories and provide packages for manipulations with them (similar
| to the one shipped by MySQL maintainers).

> 
> Provide the similar repository, but only for tagged builds (working name
> is 'release'). It seems it worth to push only tagged modules /
> connectors as well. (Asked by Konstantin N.) 

Agree.

> 
> We maybe (if there will be a demand) will move the script outside of the
> repository and integrate with the existing server [2] that doing quite
> similar job: it receives built rocks (it is like python's wheels) and
> updates a repository for luarocks / tarantoolctl rocks. This can be done
> w/o user-visible changes. (Proposed by Konstantin N.)

It's a good idea, though I'm totally OK with this small script within
our repo. Generally, I'm for moving all CI/CD related parts to a
separate one, but this is definitely not related to the subj.

> 
> Aren't I forget anything?

If everyone are OK with my proposals, I'll update the key points and
corresponding ARs.

> 
> [1]: https://www.tarantool.io/en/download/
> [2]: https://github.com/tarantool/rocks.tarantool.org
> 
> ----
> 
> Re patch itself: I don't look into the script and it seems it would be
> not productive. Just noted several points around, see below.
> 
> WBR, Alexander Turenko.
> 

[1]: https://www.postgresql.org/download/linux/
[2]: https://dev.mysql.com/downloads/
[3]: https://www.tarantool.io/en/download/

-- 
Best regards,
IM

Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Alexander Turenko]

Igor, thanks a huge for the deep investigation of approaches around
other software!

Updates:

* We agreed on names 'live' (instead of 'nightly') and 'release'.
* We agreed to don't include major and minor version into package names.
* We agreed to split repos to 1.10/2.1/2.2 and so.
* We agreed on replacing current packagecloud repos with the new ones,
  because the end result looks quite similar (except that the new repos
  will be never pruned).
* We agreed to provide more recipes like 'how to update from a release
  version to one with necessary bugfix', 'how to move from this live
  version to the next release when it will be published'.
* We consider extracting modules and connectors into a separate
  repository, starting from 2.4 or 2.5.
* We consider providing tarantool-110-release and tarantool-110-live
  packages, which will install and enable corresponding repositories.
* We consider providing some tooling to manipulate tarantool
  repositories, a kind of `eselect` in Gentoo.
* We consider providing a user recipes how to stick with certain
  tarantool version or version range.

Reasoning behind those agreements is below four dashes.

'We agreed' basically means that me and Igor agreed that a proposed way
have more pros (or less cons) that an alternative, at least on typical
operations around a package.

'We consider' means that we discussed a proposal and it looks promising,
but there is no strict understanding how it should be implemented and
there is no issue tracking this task.

The discussion is open and we'll appreciate substantiated opinions: cons
and pros in light of a certain use case, examples of good and bad
experience with other software repositories, rules of thumb we didn't
notice.

----

We build packages for every push, but not on a schedule, so name
'nightly' is inaccurate. 'live package' is the term from (at least)
Gentoo that means a package that always installs a last version from a
version control system. This is very similar to what our repos offer.

We looked at PostgreSQL, MySQL and MongoDB repositories and found that
only PostgreSQL packages have major and minor version as a part of a
package name. But they can be installed simultaneously into different,
collision-free paths on a system. A user may expect that packages with
different names may be installed in parallel.

Different package names suggest a package manager that those packages
are different, but conflicting ('Provides' the same on CentOS / Fedora,
conflicts by paths on all distros). So it is not possible to update
from, say, tarantool110 to tarantool224: one need to remove tarantool110
and then install tarantool224. This leads to removing all packages that
depends on tarantool (at least on CentOS / Fedora), which unlikely what
a user expects.

Since we going to provide packages w/o a version in a name we should
give a user a stable version by default (say, last 2.2, not last 2.4).
This is usually achieved using separate repositories. For us it seems
natural to continue using <major>.<minor> names for them. (So, quick
installation guide will show a user how to enable 2.2 repository, while
more in-depth part can reveal another possibilities.) It is more
straightforward and flexible than stable/unstable/bleeding_edge
repositories.

Re stick with a version / a version range: see the following links.

* https://access.redhat.com/solutions/98873
* https://unix.stackexchange.com/a/162785/58434

WBR, Alexander Turenko.

On Sun, Jan 19, 2020 at 02:26:27AM +0300, Igor Munkin wrote:
> Sasha,
> 
> Thanks for the discussion results you provided! Some light can be seen
> at the end of the tunnel.
>
> <...>
>
> -- 
> Best regards,
> IM