* [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3
@ 2020-01-31 6:00 Alexander V. Tikhonov
2020-01-31 8:26 ` Konstantin Osipov
2020-01-31 13:28 ` Oleg Piskunov
0 siblings, 2 replies; 5+ messages in thread
From: Alexander V. Tikhonov @ 2020-01-31 6:00 UTC (permalink / raw)
To: Oleg Piskunov, Igor Munkin, Alexander Turenko; +Cc: tarantool-patches
The changes introduce new Gitlab-CI rules for creating packages on
branches with "-full-ci" suffix and their subsequent deployment to the
'live' repository for master and release branches. Packages for tagged
commits are also delivered to the corresponding 'release' repository.
Packages creation activities relocating from the PackageCloud storage
to the new self-hosted MCS S3 where all old packages have been synced.
Benefits of the introduced approach are the following:
* As far as all contents of self-hosted repos are fully controlled
theirs layout is the same as the ones provided by the corresponding
distro
* Repo metadata rebuild is excess considering the known repo layout
* Old packages are not pruned since they do not affect the repo
metadata rebuilding time
For these purposes the standalone script for pushing DEB and RPM
packages to self-hosted repositories is introduced. The script
implements the following flow:
* creates new metafiles for the new packages
* copies new packages to S3 storage
* fetches relevant metafiles from the repo
* merges the new metadata with the fetched one
* pushes the updated metadata to S3 storage
There are OS distribution dependent parts in the script:
* for RPM packages it updates metadata separately per each OS
distribution considering 'createrepo' util behavior
* for DEB packages it updates metadata simultaniously for all
distributions within single OS considering 'reprepro' util behaviour
Closes #3380
---
Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
Issue: https://github.com/tarantool/tarantool/issues/3380
v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html
v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html
v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
Changes v8:
- corrected commit message
- removed extra check for bucket naming
- changed bucket naming from underscore to dot, like: 2_2 to 2.x
- changed git tag checking routine as suggested
Changes v7:
- removed additional functionality for working with DEB repositories
using complete pool path w/o specifing packages
- implemented new flag '-f|--force' that helps to overwite the packages
at MCS S3 if it checksum changed - implemented check on the new
packages for it
- implemented check with warning on the new RPM packages with the same checksum
Changes v6:
- implemented 2 MCS S3 repositories 'live' and 'release'
- added AWS and GPG keys into Gitlab-CI
- corrected commit message
- corrected return functionality code in script
- moved all changes for sources tarballs at the standalone patch set
Changes v5:
- code style
- commits squashed
- rebased to master
Changes v4:
- minor corrections
Changes v3:
- common code parts merged to standalone routines
- corrected code style, minor updates
- script is ready for release
Changes v2:
- made changes in script from draft to pre-release stages
.gitlab-ci.yml | 152 ++++++++++--
.gitlab.mk | 20 +-
tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 723 insertions(+), 20 deletions(-)
create mode 100755 tools/update_repo.sh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3af5a3c8a..c68594c1a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -10,6 +10,10 @@ variables:
only:
refs:
- master
+
+.fullci_only_template: &fullci_only_definition
+ only:
+ refs:
- /^.*-full-ci$/
.docker_test_template: &docker_test_definition
@@ -24,13 +28,29 @@ variables:
tags:
- docker_test
+.pack_template: &pack_definition
+ <<: *fullci_only_definition
+ stage: test
+ tags:
+ - deploy
+ script:
+ - ${GITLAB_MAKE} package
+
+.pack_test_template: &pack_test_definition
+ <<: *fullci_only_definition
+ stage: test
+ tags:
+ - deploy_test
+ script:
+ - ${GITLAB_MAKE} package
+
.deploy_template: &deploy_definition
<<: *release_only_definition
stage: test
tags:
- deploy
script:
- - ${GITLAB_MAKE} package
+ - ${GITLAB_MAKE} deploy
.deploy_test_template: &deploy_test_definition
<<: *release_only_definition
@@ -38,7 +58,7 @@ variables:
tags:
- deploy_test
script:
- - ${GITLAB_MAKE} package
+ - ${GITLAB_MAKE} deploy
.vbox_template: &vbox_definition
stage: test
@@ -141,96 +161,194 @@ freebsd_12_release:
# Packs
centos_6:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'el'
DIST: '6'
centos_7:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '7'
centos_8:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '8'
fedora_28:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '28'
fedora_29:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '29'
fedora_30:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '30'
fedora_31:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '31'
ubuntu_14_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'trusty'
ubuntu_16_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'xenial'
ubuntu_18_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'bionic'
ubuntu_18_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'cosmic'
ubuntu_19_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'disco'
ubuntu_19_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'eoan'
debian_8:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'jessie'
debian_9:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'stretch'
debian_10:
+ <<: *pack_definition
+ variables:
+ OS: 'debian'
+ DIST: 'buster'
+
+# Deploy
+
+centos_6_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'el'
+ DIST: '6'
+
+centos_7_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '7'
+
+centos_8_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '8'
+
+fedora_28_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '28'
+
+fedora_29_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '29'
+
+fedora_30_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '30'
+
+fedora_31_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '31'
+
+ubuntu_14_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'trusty'
+
+ubuntu_16_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'xenial'
+
+ubuntu_18_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'bionic'
+
+ubuntu_18_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'cosmic'
+
+ubuntu_19_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'disco'
+
+ubuntu_19_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'eoan'
+
+debian_8_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'jessie'
+
+debian_9_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'stretch'
+
+debian_10_deploy:
<<: *deploy_definition
variables:
OS: 'debian'
diff --git a/.gitlab.mk b/.gitlab.mk
index 48a92e518..1f921fd6e 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -98,14 +98,28 @@ vms_test_%:
vms_shutdown:
VBoxManage controlvm ${VMS_NAME} poweroff
-# ########################
-# Build RPM / Deb packages
-# ########################
+# ########
+# Packages
+# ########
+
+GIT_DESCRIBE=$(shell git describe HEAD)
+MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
+MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
+BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)"
package: git_submodule_update
git clone https://github.com/packpack/packpack.git packpack
PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
+deploy: package
+ for key in ${GPG_SECRET_KEY} ${GPG_PUBLIC_KEY} ; do \
+ echo $${key} | base64 -d | gpg --batch --import || true ; done
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="s3://tarantool_repo/live/${BUCKET}" build
+ git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="s3://tarantool_repo/release/${BUCKET}" build
+
# ############
# Static build
# ############
diff --git a/tools/update_repo.sh b/tools/update_repo.sh
new file mode 100755
index 000000000..00e7aa0c6
--- /dev/null
+++ b/tools/update_repo.sh
@@ -0,0 +1,571 @@
+#!/bin/bash
+set -e
+
+rm_file='rm -f'
+rm_dir='rm -rf'
+mk_dir='mkdir -p'
+ws_prefix=/tmp/tarantool_repo_s3
+
+alloss='ubuntu debian el fedora'
+product=tarantool
+force=
+# the path with binaries either repository
+repo=.
+
+# AWS defines
+aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3"
+aws_cp_public="$aws cp --acl public-read"
+aws_sync_public="$aws sync --acl public-read"
+
+function get_os_dists {
+ os=$1
+ alldists=
+
+ if [ "$os" == "ubuntu" ]; then
+ alldists='trusty xenial bionic cosmic disco eoan'
+ elif [ "$os" == "debian" ]; then
+ alldists='jessie stretch buster'
+ elif [ "$os" == "el" ]; then
+ alldists='6 7 8'
+ elif [ "$os" == "fedora" ]; then
+ alldists='27 28 29 30 31'
+ fi
+
+ echo "$alldists"
+}
+
+function prepare_ws {
+ # temporary lock the publication to the repository
+ ws_suffix=$1
+ ws=${ws_prefix}_${ws_suffix}
+ ws_lockfile=${ws}.lock
+ if [ -f $ws_lockfile ]; then
+ old_proc=$(cat $ws_lockfile)
+ fi
+ lockfile -l 60 $ws_lockfile
+ chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
+ if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then
+ kill -9 $old_proc >/dev/null || true
+ fi
+
+ # create temporary workspace for the new files
+ $rm_dir $ws
+ $mk_dir $ws
+}
+
+function usage {
+ cat <<EOF
+Usage for store package binaries from the given path:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to package binaries>
+
+Usage for mirroring Debian|Ubuntu OS repositories:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to packages binaries>
+
+Arguments:
+ <path>
+ Path points to the directory with deb/prm packages to be used.
+
+Options:
+ -b|--bucket
+ MCS S3 bucket already existing which will be used for storing the packages
+ -o|--os
+ OS to be checked, one of the list:
+ $alloss
+ -d|--distribution
+ Distribution appropriate to the given OS:
+EOF
+ for os in $alloss ; do
+ echo " $os: <"$(get_os_dists $os)">"
+ done
+ cat <<EOF
+ -p|--product
+ Product name to be packed with, default name is 'tarantool'
+ -f|--force
+ Force updating the remote package with the local one despite the checksum difference
+ -h|--help
+ Usage help message
+EOF
+}
+
+for i in "$@"
+do
+case $i in
+ -b=*|--bucket=*)
+ bucket="${i#*=}"
+ shift # past argument=value
+ ;;
+ -o=*|--os=*)
+ os="${i#*=}"
+ if ! echo $alloss | grep -F -q -w $os ; then
+ echo "ERROR: OS '$os' is not supported"
+ usage
+ exit 1
+ fi
+ shift # past argument=value
+ ;;
+ -d=*|--distribution=*)
+ option_dist="${i#*=}"
+ shift # past argument=value
+ ;;
+ -p=*|--product=*)
+ product="${i#*=}"
+ shift # past argument=value
+ ;;
+ -f|--force)
+ force=1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ repo="${i#*=}"
+ pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
+ shift # past argument=value
+ ;;
+esac
+done
+
+# check that all needed options were set and correct
+if [ "$bucket" == "" ]; then
+ echo "ERROR: need to set -b|--bucket bucket option, check usage"
+ usage
+ exit 1
+fi
+if [ "$option_dist" == "" ]; then
+ echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage"
+ usage
+ exit 1
+fi
+if [ "$os" == "" ]; then
+ echo "ERROR: need to set -o|--os OS name option, check usage"
+ usage
+ exit 1
+fi
+alldists=$(get_os_dists $os)
+if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
+ echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
+ usage
+ exit 1
+fi
+
+# set the subpath with binaries based on literal character of the product name
+proddir=$(echo $product | head -c 1)
+
+# set bucket path of the given OS in options
+bucket_path="$bucket/$os"
+
+function update_deb_packfile {
+ packfile=$1
+ packtype=$2
+ update_dist=$3
+
+ locpackfile=$(echo $packfile | sed "s#^$ws/##g")
+ # register DEB/DSC pack file to Packages/Sources file
+ reprepro -Vb . include$packtype $update_dist $packfile
+ # reprepro copied DEB/DSC file to component which is not needed
+ $rm_dir $debdir/$component
+ # to have all sources avoid reprepro set DEB/DSC file to its own registry
+ $rm_dir db
+}
+
+function update_deb_metadata {
+ packpath=$1
+ packtype=$2
+
+ if [ ! -f $packpath.saved ] ; then
+ # get the latest Sources file from S3 either create empty file
+ $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
+ $aws cp "$bucket_path/$packpath" $packpath.saved || \
+ touch $packpath.saved
+ fi
+
+ if [ "$packtype" == "dsc" ]; then
+ # WORKAROUND: unknown why, but reprepro doesn`t save the Sources
+ # file, lets recreate it manualy from it's zipped version
+ gunzip -c $packpath.gz >$packpath
+ # check if the DSC hash already exists in old Sources file from S3
+ # find the hash from the new Sources file
+ hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
+ tail -n 1 | awk '{print $1}')
+ # search the new hash in the old Sources file from S3
+ if grep " $hash .* .*$" $packpath.saved ; then
+ echo "WARNING: DSC file already registered in S3!"
+ return
+ fi
+ # check if the DSC file already exists in old Sources file from S3
+ file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}')
+ if [ "$force" == "" ] && grep " .* .* $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_dsc=1
+ elif [ "$packtype" == "deb" ]; then
+ # check if the DEB file already exists in old Packages file from S3
+ # find the hash from the new Packages file
+ hash=$(grep '^SHA256: ' $packpath)
+ # search the new hash in the old Packages file from S3
+ if grep "^SHA256: $hash" $packpath.saved ; then
+ echo "WARNING: DEB file already registered in S3!"
+ return
+ fi
+ # check if the DEB file already exists in old Packages file from S3
+ file=$(grep '^Filename:' | awk '{print $2}')
+ if [ "$force" == "" ] && grep "Filename: $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_deb=1
+ fi
+ # store the new DEB entry
+ cat $packpath >>$packpath.saved
+}
+
+# The 'pack_deb' function especialy created for DEB packages. It works
+# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
+# tool 'reprepro' from:
+# https://wiki.debian.org/DebianRepository/SetupWithReprepro
+# This tool works with complete number of distributions of the given OS.
+# Result of the routine is the debian package for APT repository with
+# file structure equal to the Debian/Ubuntu:
+# http://ftp.am.debian.org/debian/pool/main/t/tarantool/
+# http://ftp.am.debian.org/ubuntu/pool/main/t/
+function pack_deb {
+ # we need to push packages into 'main' repository only
+ component=main
+
+ # debian has special directory 'pool' for packages
+ debdir=pool
+
+ # get packages from pointed location
+ if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then
+ echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}
+
+ # copy single distribution with binaries packages
+ repopath=$ws/pool/${option_dist}/$component/$proddir/$product
+ $mk_dir ${repopath}
+ cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
+ pushd $ws
+
+ # create the configuration file for 'reprepro' tool
+ confpath=$ws/conf
+ $rm_dir $confpath
+ $mk_dir $confpath
+
+ for loop_dist in $alldists ; do
+ cat <<EOF >>$confpath/distributions
+Origin: Tarantool
+Label: tarantool.org
+Suite: stable
+Codename: $loop_dist
+Architectures: amd64 source
+Components: $component
+Description: Tarantool DBMS and Tarantool modules
+SignWith: 91B625E5
+DebIndices: Packages Release . .gz .bz2
+UDebIndices: Packages . .gz .bz2
+DscIndices: Sources Release .gz .bz2
+
+EOF
+ done
+
+ # create standalone repository with separate components
+ for loop_dist in $alldists ; do
+ echo ================ DISTRIBUTION: $loop_dist ====================
+ updated_files=0
+
+ # 1(binaries). use reprepro tool to generate Packages file
+ for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
+ [ -f $deb ] || continue
+ updated_deb=0
+ # regenerate DEB pack
+ update_deb_packfile $deb deb $loop_dist
+ echo "Regenerated DEB file: $locpackfile"
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ # copy Packages file to avoid of removing by the new DEB version
+ # update metadata 'Packages' files
+ update_deb_metadata $packages deb
+ [ "$updated_deb" == "1" ] || continue
+ updated_files=1
+ done
+ # save the registered DEB file to S3
+ if [ "$updated_deb" == 1 ]; then
+ $aws_cp_public $deb $bucket_path/$locpackfile
+ fi
+ done
+
+ # 1(sources). use reprepro tool to generate Sources file
+ for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
+ [ -f $dsc ] || continue
+ updated_dsc=0
+ # regenerate DSC pack
+ update_deb_packfile $dsc dsc $loop_dist
+ echo "Regenerated DSC file: $locpackfile"
+ # copy Sources file to avoid of removing by the new DSC version
+ # update metadata 'Sources' file
+ update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
+ [ "$updated_dsc" == "1" ] || continue
+ updated_files=1
+ # save the registered DSC file to S3
+ $aws_cp_public $dsc $bucket_path/$locpackfile
+ tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
+ $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
+ orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
+ $aws_cp_public $ws/$orig "$bucket_path/$orig"
+ done
+
+ # check if any DEB/DSC files were newly registered
+ [ "$updated_files" == "0" ] && \
+ continue || echo "Updating dists"
+
+ # finalize the Packages file
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ mv $packages.saved $packages
+ done
+
+ # finalize the Sources file
+ sources=dists/$loop_dist/$component/source/Sources
+ mv $sources.saved $sources
+
+ # 2(binaries). update Packages file archives
+ for packpath in dists/$loop_dist/$component/binary-* ; do
+ pushd $packpath
+ sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
+ bzip2 -c Packages >Packages.bz2
+ gzip -c Packages >Packages.gz
+ popd
+ done
+
+ # 2(sources). update Sources file archives
+ pushd dists/$loop_dist/$component/source
+ sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
+ bzip2 -c Sources >Sources.bz2
+ gzip -c Sources >Sources.gz
+ popd
+
+ # 3. update checksums entries of the Packages* files in *Release files
+ # NOTE: it is stable structure of the *Release files when the checksum
+ # entries in it in the following way:
+ # MD5Sum:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA1:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA256:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # The script bellow puts 'md5' value at the 1st found file entry,
+ # 'sha1' - at the 2nd and 'sha256' at the 3rd
+ pushd dists/$loop_dist
+ for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
+ sz=$(stat -c "%s" $file)
+ md5=$(md5sum $file | awk '{print $1}')
+ sha1=$(sha1sum $file | awk '{print $1}')
+ sha256=$(sha256sum $file | awk '{print $1}')
+ awk 'BEGIN{c = 0} ; {
+ if ($3 == p) {
+ c = c + 1
+ if (c == 1) {print " " md " " s " " p}
+ if (c == 2) {print " " sh1 " " s " " p}
+ if (c == 3) {print " " sh2 " " s " " p}
+ } else {print $0}
+ }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
+ Release >Release.new
+ mv Release.new Release
+ done
+ # resign the selfsigned InRelease file
+ $rm_file InRelease
+ gpg --clearsign -o InRelease Release
+ # resign the Release file
+ $rm_file Release.gpg
+ gpg -abs -o Release.gpg Release
+ popd
+
+ # 4. sync the latest distribution path changes to S3
+ $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
+ done
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+# The 'pack_rpm' function especialy created for RPM packages. It works
+# with RPM packing OS like Centos, Fedora. It is based on globaly known
+# tool 'createrepo' from:
+# https://linux.die.net/man/8/createrepo
+# This tool works with single distribution of the given OS.
+# Result of the routine is the rpm package for YUM repository with
+# file structure equal to the Centos/Fedora:
+# http://mirror.centos.org/centos/7/os/x86_64/Packages/
+# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
+function pack_rpm {
+ if ! ls $repo/*.rpm >/dev/null ; then
+ echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}_${option_dist}
+
+ # copy the needed package binaries to the workspace
+ cp $repo/*.rpm $ws/.
+
+ pushd $ws
+
+ # set the paths
+ if [ "$os" == "el" ]; then
+ repopath=$option_dist/os/x86_64
+ rpmpath=Packages
+ elif [ "$os" == "fedora" ]; then
+ repopath=releases/$option_dist/Everything/x86_64/os
+ rpmpath=Packages/$proddir
+ fi
+ packpath=$repopath/$rpmpath
+
+ # prepare local repository with packages
+ $mk_dir $packpath
+ mv *.rpm $packpath/.
+ cd $repopath
+
+ # copy the current metadata files from S3
+ mkdir repodata.base
+ for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
+ $aws ls $bucket_path/$repopath/repodata/$file || continue
+ $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
+ done
+
+ # create the new repository metadata files
+ createrepo --no-database --update --workers=2 \
+ --compress-type=gz --simple-md-filenames .
+
+ updated_rpms=0
+ # loop by the new hashes from the new meta file
+ for hash in $(zcat repodata/other.xml.gz | grep "<package pkgid=" | \
+ awk -F'"' '{print $2}') ; do
+ updated_rpm=0
+ name=$(zcat repodata/other.xml.gz | grep "<package pkgid=\"$hash\"" | \
+ awk -F'"' '{print $4}')
+ # search the new hash in the old meta file from S3
+ if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \
+ grep "name=\"$name\"" ; then
+ echo "WARNING: $name file already registered in S3!"
+ echo "File hash: $hash"
+ continue
+ fi
+ updated_rpms=1
+ # check if the hashed file already exists in old meta file from S3
+ file=$(zcat repodata/primary.xml.gz | \
+ grep -e "<checksum type=" -e "<location href=" | \
+ grep "$hash" -A1 | grep "<location href=" | \
+ awk -F'"' '{print $2}')
+ # check if the file already exists in S3
+ if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \
+ grep "<location href=\"$file\"" ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ done
+
+ # check if any RPM files were newly registered
+ [ "$updated_rpms" == "0" ] && \
+ return || echo "Updating dists"
+
+ # move the repodata files to the standalone location
+ mv repodata repodata.adding
+
+ # merge metadata files
+ mkdir repodata
+ head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
+ for file in filelists.xml other.xml primary.xml ; do
+ # 1. take the 1st line only - to skip the line with
+ # number of packages which is not needed
+ zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
+ # 2. take 2nd line with metadata tag and update
+ # the packages number in it
+ packsold=0
+ if [ -f repodata.base/$file.gz ] ; then
+ packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ fi
+ packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ packs=$(($packsold+$packsnew))
+ zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
+ sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
+ # 3. take only 'package' tags from new file
+ zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ # 4. take only 'package' tags from old file if exists
+ if [ -f repodata.base/$file.gz ] ; then
+ zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ fi
+ # 5. take the last closing line with metadata tag
+ zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
+
+ # get the new data
+ chsnew=$(sha256sum repodata/$file | awk '{print $1}')
+ sz=$(stat --printf="%s" repodata/$file)
+ gzip repodata/$file
+ chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
+ szgz=$(stat --printf="%s" repodata/$file.gz)
+ timestamp=$(date +%s -r repodata/$file.gz)
+
+ # add info to repomd.xml file
+ name=$(echo $file | sed 's#\.xml$##g')
+ cat <<EOF >>repodata/repomd.xml
+<data type="$name">
+ <checksum type="sha256">$chsgznew</checksum>
+ <open-checksum type="sha256">$chsnew</open-checksum>
+ <location href="repodata/$file.gz"/>
+ <timestamp>$timestamp</timestamp>
+ <size>$szgz</size>
+ <open-size>$sz</open-size>
+</data>"
+EOF
+ done
+ tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
+ gpg --detach-sign --armor repodata/repomd.xml
+
+ # copy the packages to S3
+ for file in $rpmpath/*.rpm ; do
+ $aws_cp_public $file "$bucket_path/$repopath/$file"
+ done
+
+ # update the metadata at the S3
+ $aws_sync_public repodata "$bucket_path/$repopath/repodata"
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
+ pack_deb
+elif [ "$os" == "el" -o "$os" == "fedora" ]; then
+ pack_rpm
+else
+ echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
+ usage
+ exit 1
+fi
--
2.17.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3
2020-01-31 6:00 [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3 Alexander V. Tikhonov
@ 2020-01-31 8:26 ` Konstantin Osipov
2020-01-31 18:01 ` Alexander Turenko
2020-01-31 13:28 ` Oleg Piskunov
1 sibling, 1 reply; 5+ messages in thread
From: Konstantin Osipov @ 2020-01-31 8:26 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
* Alexander V. Tikhonov <avtikhon@tarantool.org> [20/01/31 10:43]:
I understand it's a v8 and I am jumping on this late, but I'm
really curious why:
- you use shell in 2020.
- you create a bunch of essentially proprietary scripts (noone is
going to look at this) instead of making it a feature of packpack.
Many people need to create repositories, and many people use
packpack already. It's best if this is just added there.
> The changes introduce new Gitlab-CI rules for creating packages on
> branches with "-full-ci" suffix and their subsequent deployment to the
> 'live' repository for master and release branches. Packages for tagged
> commits are also delivered to the corresponding 'release' repository.
>
> Packages creation activities relocating from the PackageCloud storage
> to the new self-hosted MCS S3 where all old packages have been synced.
> Benefits of the introduced approach are the following:
> * As far as all contents of self-hosted repos are fully controlled
> theirs layout is the same as the ones provided by the corresponding
> distro
> * Repo metadata rebuild is excess considering the known repo layout
> * Old packages are not pruned since they do not affect the repo
> metadata rebuilding time
>
> For these purposes the standalone script for pushing DEB and RPM
> packages to self-hosted repositories is introduced. The script
> implements the following flow:
> * creates new metafiles for the new packages
> * copies new packages to S3 storage
> * fetches relevant metafiles from the repo
> * merges the new metadata with the fetched one
> * pushes the updated metadata to S3 storage
>
> There are OS distribution dependent parts in the script:
> * for RPM packages it updates metadata separately per each OS
> distribution considering 'createrepo' util behavior
> * for DEB packages it updates metadata simultaniously for all
> distributions within single OS considering 'reprepro' util behaviour
>
> Closes #3380
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/3380
>
> v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html
> v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html
> v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
> v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
> v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
> v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
> v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
>
> Changes v8:
> - corrected commit message
> - removed extra check for bucket naming
> - changed bucket naming from underscore to dot, like: 2_2 to 2.x
> - changed git tag checking routine as suggested
>
> Changes v7:
> - removed additional functionality for working with DEB repositories
> using complete pool path w/o specifing packages
> - implemented new flag '-f|--force' that helps to overwite the packages
> at MCS S3 if it checksum changed - implemented check on the new
> packages for it
> - implemented check with warning on the new RPM packages with the same checksum
>
> Changes v6:
> - implemented 2 MCS S3 repositories 'live' and 'release'
> - added AWS and GPG keys into Gitlab-CI
> - corrected commit message
> - corrected return functionality code in script
> - moved all changes for sources tarballs at the standalone patch set
>
> Changes v5:
> - code style
> - commits squashed
> - rebased to master
>
> Changes v4:
> - minor corrections
>
> Changes v3:
> - common code parts merged to standalone routines
> - corrected code style, minor updates
> - script is ready for release
>
> Changes v2:
> - made changes in script from draft to pre-release stages
>
> .gitlab-ci.yml | 152 ++++++++++--
> .gitlab.mk | 20 +-
> tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 723 insertions(+), 20 deletions(-)
> create mode 100755 tools/update_repo.sh
>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 3af5a3c8a..c68594c1a 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -10,6 +10,10 @@ variables:
> only:
> refs:
> - master
> +
> +.fullci_only_template: &fullci_only_definition
> + only:
> + refs:
> - /^.*-full-ci$/
>
> .docker_test_template: &docker_test_definition
> @@ -24,13 +28,29 @@ variables:
> tags:
> - docker_test
>
> +.pack_template: &pack_definition
> + <<: *fullci_only_definition
> + stage: test
> + tags:
> + - deploy
> + script:
> + - ${GITLAB_MAKE} package
> +
> +.pack_test_template: &pack_test_definition
> + <<: *fullci_only_definition
> + stage: test
> + tags:
> + - deploy_test
> + script:
> + - ${GITLAB_MAKE} package
> +
> .deploy_template: &deploy_definition
> <<: *release_only_definition
> stage: test
> tags:
> - deploy
> script:
> - - ${GITLAB_MAKE} package
> + - ${GITLAB_MAKE} deploy
>
> .deploy_test_template: &deploy_test_definition
> <<: *release_only_definition
> @@ -38,7 +58,7 @@ variables:
> tags:
> - deploy_test
> script:
> - - ${GITLAB_MAKE} package
> + - ${GITLAB_MAKE} deploy
>
> .vbox_template: &vbox_definition
> stage: test
> @@ -141,96 +161,194 @@ freebsd_12_release:
> # Packs
>
> centos_6:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'el'
> DIST: '6'
>
> centos_7:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'el'
> DIST: '7'
>
> centos_8:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'el'
> DIST: '8'
>
> fedora_28:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '28'
>
> fedora_29:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '29'
>
> fedora_30:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '30'
>
> fedora_31:
> - <<: *deploy_test_definition
> + <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '31'
>
> ubuntu_14_04:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'trusty'
>
> ubuntu_16_04:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'xenial'
>
> ubuntu_18_04:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'bionic'
>
> ubuntu_18_10:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'cosmic'
>
> ubuntu_19_04:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'disco'
>
> ubuntu_19_10:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'eoan'
>
> debian_8:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'debian'
> DIST: 'jessie'
>
> debian_9:
> - <<: *deploy_definition
> + <<: *pack_definition
> variables:
> OS: 'debian'
> DIST: 'stretch'
>
> debian_10:
> + <<: *pack_definition
> + variables:
> + OS: 'debian'
> + DIST: 'buster'
> +
> +# Deploy
> +
> +centos_6_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'el'
> + DIST: '6'
> +
> +centos_7_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'el'
> + DIST: '7'
> +
> +centos_8_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'el'
> + DIST: '8'
> +
> +fedora_28_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'fedora'
> + DIST: '28'
> +
> +fedora_29_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'fedora'
> + DIST: '29'
> +
> +fedora_30_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'fedora'
> + DIST: '30'
> +
> +fedora_31_deploy:
> + <<: *deploy_test_definition
> + variables:
> + OS: 'fedora'
> + DIST: '31'
> +
> +ubuntu_14_04_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'trusty'
> +
> +ubuntu_16_04_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'xenial'
> +
> +ubuntu_18_04_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'bionic'
> +
> +ubuntu_18_10_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'cosmic'
> +
> +ubuntu_19_04_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'disco'
> +
> +ubuntu_19_10_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'ubuntu'
> + DIST: 'eoan'
> +
> +debian_8_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'debian'
> + DIST: 'jessie'
> +
> +debian_9_deploy:
> + <<: *deploy_definition
> + variables:
> + OS: 'debian'
> + DIST: 'stretch'
> +
> +debian_10_deploy:
> <<: *deploy_definition
> variables:
> OS: 'debian'
> diff --git a/.gitlab.mk b/.gitlab.mk
> index 48a92e518..1f921fd6e 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -98,14 +98,28 @@ vms_test_%:
> vms_shutdown:
> VBoxManage controlvm ${VMS_NAME} poweroff
>
> -# ########################
> -# Build RPM / Deb packages
> -# ########################
> +# ########
> +# Packages
> +# ########
> +
> +GIT_DESCRIBE=$(shell git describe HEAD)
> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
> +BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)"
>
> package: git_submodule_update
> git clone https://github.com/packpack/packpack.git packpack
> PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>
> +deploy: package
> + for key in ${GPG_SECRET_KEY} ${GPG_PUBLIC_KEY} ; do \
> + echo $${key} | base64 -d | gpg --batch --import || true ; done
> + ./tools/update_repo.sh -o=${OS} -d=${DIST} \
> + -b="s3://tarantool_repo/live/${BUCKET}" build
> + git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
> + ./tools/update_repo.sh -o=${OS} -d=${DIST} \
> + -b="s3://tarantool_repo/release/${BUCKET}" build
> +
> # ############
> # Static build
> # ############
> diff --git a/tools/update_repo.sh b/tools/update_repo.sh
> new file mode 100755
> index 000000000..00e7aa0c6
> --- /dev/null
> +++ b/tools/update_repo.sh
> @@ -0,0 +1,571 @@
> +#!/bin/bash
> +set -e
> +
> +rm_file='rm -f'
> +rm_dir='rm -rf'
> +mk_dir='mkdir -p'
> +ws_prefix=/tmp/tarantool_repo_s3
> +
> +alloss='ubuntu debian el fedora'
> +product=tarantool
> +force=
> +# the path with binaries either repository
> +repo=.
> +
> +# AWS defines
> +aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3"
> +aws_cp_public="$aws cp --acl public-read"
> +aws_sync_public="$aws sync --acl public-read"
> +
> +function get_os_dists {
> + os=$1
> + alldists=
> +
> + if [ "$os" == "ubuntu" ]; then
> + alldists='trusty xenial bionic cosmic disco eoan'
> + elif [ "$os" == "debian" ]; then
> + alldists='jessie stretch buster'
> + elif [ "$os" == "el" ]; then
> + alldists='6 7 8'
> + elif [ "$os" == "fedora" ]; then
> + alldists='27 28 29 30 31'
> + fi
> +
> + echo "$alldists"
> +}
> +
> +function prepare_ws {
> + # temporary lock the publication to the repository
> + ws_suffix=$1
> + ws=${ws_prefix}_${ws_suffix}
> + ws_lockfile=${ws}.lock
> + if [ -f $ws_lockfile ]; then
> + old_proc=$(cat $ws_lockfile)
> + fi
> + lockfile -l 60 $ws_lockfile
> + chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
> + if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then
> + kill -9 $old_proc >/dev/null || true
> + fi
> +
> + # create temporary workspace for the new files
> + $rm_dir $ws
> + $mk_dir $ws
> +}
> +
> +function usage {
> + cat <<EOF
> +Usage for store package binaries from the given path:
> + $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to package binaries>
> +
> +Usage for mirroring Debian|Ubuntu OS repositories:
> + $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to packages binaries>
> +
> +Arguments:
> + <path>
> + Path points to the directory with deb/prm packages to be used.
> +
> +Options:
> + -b|--bucket
> + MCS S3 bucket already existing which will be used for storing the packages
> + -o|--os
> + OS to be checked, one of the list:
> + $alloss
> + -d|--distribution
> + Distribution appropriate to the given OS:
> +EOF
> + for os in $alloss ; do
> + echo " $os: <"$(get_os_dists $os)">"
> + done
> + cat <<EOF
> + -p|--product
> + Product name to be packed with, default name is 'tarantool'
> + -f|--force
> + Force updating the remote package with the local one despite the checksum difference
> + -h|--help
> + Usage help message
> +EOF
> +}
> +
> +for i in "$@"
> +do
> +case $i in
> + -b=*|--bucket=*)
> + bucket="${i#*=}"
> + shift # past argument=value
> + ;;
> + -o=*|--os=*)
> + os="${i#*=}"
> + if ! echo $alloss | grep -F -q -w $os ; then
> + echo "ERROR: OS '$os' is not supported"
> + usage
> + exit 1
> + fi
> + shift # past argument=value
> + ;;
> + -d=*|--distribution=*)
> + option_dist="${i#*=}"
> + shift # past argument=value
> + ;;
> + -p=*|--product=*)
> + product="${i#*=}"
> + shift # past argument=value
> + ;;
> + -f|--force)
> + force=1
> + ;;
> + -h|--help)
> + usage
> + exit 0
> + ;;
> + *)
> + repo="${i#*=}"
> + pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
> + shift # past argument=value
> + ;;
> +esac
> +done
> +
> +# check that all needed options were set and correct
> +if [ "$bucket" == "" ]; then
> + echo "ERROR: need to set -b|--bucket bucket option, check usage"
> + usage
> + exit 1
> +fi
> +if [ "$option_dist" == "" ]; then
> + echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage"
> + usage
> + exit 1
> +fi
> +if [ "$os" == "" ]; then
> + echo "ERROR: need to set -o|--os OS name option, check usage"
> + usage
> + exit 1
> +fi
> +alldists=$(get_os_dists $os)
> +if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
> + echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
> + usage
> + exit 1
> +fi
> +
> +# set the subpath with binaries based on literal character of the product name
> +proddir=$(echo $product | head -c 1)
> +
> +# set bucket path of the given OS in options
> +bucket_path="$bucket/$os"
> +
> +function update_deb_packfile {
> + packfile=$1
> + packtype=$2
> + update_dist=$3
> +
> + locpackfile=$(echo $packfile | sed "s#^$ws/##g")
> + # register DEB/DSC pack file to Packages/Sources file
> + reprepro -Vb . include$packtype $update_dist $packfile
> + # reprepro copied DEB/DSC file to component which is not needed
> + $rm_dir $debdir/$component
> + # to have all sources avoid reprepro set DEB/DSC file to its own registry
> + $rm_dir db
> +}
> +
> +function update_deb_metadata {
> + packpath=$1
> + packtype=$2
> +
> + if [ ! -f $packpath.saved ] ; then
> + # get the latest Sources file from S3 either create empty file
> + $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
> + $aws cp "$bucket_path/$packpath" $packpath.saved || \
> + touch $packpath.saved
> + fi
> +
> + if [ "$packtype" == "dsc" ]; then
> + # WORKAROUND: unknown why, but reprepro doesn`t save the Sources
> + # file, lets recreate it manualy from it's zipped version
> + gunzip -c $packpath.gz >$packpath
> + # check if the DSC hash already exists in old Sources file from S3
> + # find the hash from the new Sources file
> + hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
> + tail -n 1 | awk '{print $1}')
> + # search the new hash in the old Sources file from S3
> + if grep " $hash .* .*$" $packpath.saved ; then
> + echo "WARNING: DSC file already registered in S3!"
> + return
> + fi
> + # check if the DSC file already exists in old Sources file from S3
> + file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}')
> + if [ "$force" == "" ] && grep " .* .* $file$" $packpath.saved ; then
> + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
> + echo "New hash: $hash"
> + # unlock the publishing
> + $rm_file $ws_lockfile
> + exit 1
> + fi
> + updated_dsc=1
> + elif [ "$packtype" == "deb" ]; then
> + # check if the DEB file already exists in old Packages file from S3
> + # find the hash from the new Packages file
> + hash=$(grep '^SHA256: ' $packpath)
> + # search the new hash in the old Packages file from S3
> + if grep "^SHA256: $hash" $packpath.saved ; then
> + echo "WARNING: DEB file already registered in S3!"
> + return
> + fi
> + # check if the DEB file already exists in old Packages file from S3
> + file=$(grep '^Filename:' | awk '{print $2}')
> + if [ "$force" == "" ] && grep "Filename: $file$" $packpath.saved ; then
> + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
> + echo "New hash: $hash"
> + # unlock the publishing
> + $rm_file $ws_lockfile
> + exit 1
> + fi
> + updated_deb=1
> + fi
> + # store the new DEB entry
> + cat $packpath >>$packpath.saved
> +}
> +
> +# The 'pack_deb' function especialy created for DEB packages. It works
> +# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
> +# tool 'reprepro' from:
> +# https://wiki.debian.org/DebianRepository/SetupWithReprepro
> +# This tool works with complete number of distributions of the given OS.
> +# Result of the routine is the debian package for APT repository with
> +# file structure equal to the Debian/Ubuntu:
> +# http://ftp.am.debian.org/debian/pool/main/t/tarantool/
> +# http://ftp.am.debian.org/ubuntu/pool/main/t/
> +function pack_deb {
> + # we need to push packages into 'main' repository only
> + component=main
> +
> + # debian has special directory 'pool' for packages
> + debdir=pool
> +
> + # get packages from pointed location
> + if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then
> + echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found"
> + usage
> + exit 1
> + fi
> +
> + # prepare the workspace
> + prepare_ws ${os}
> +
> + # copy single distribution with binaries packages
> + repopath=$ws/pool/${option_dist}/$component/$proddir/$product
> + $mk_dir ${repopath}
> + cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
> + pushd $ws
> +
> + # create the configuration file for 'reprepro' tool
> + confpath=$ws/conf
> + $rm_dir $confpath
> + $mk_dir $confpath
> +
> + for loop_dist in $alldists ; do
> + cat <<EOF >>$confpath/distributions
> +Origin: Tarantool
> +Label: tarantool.org
> +Suite: stable
> +Codename: $loop_dist
> +Architectures: amd64 source
> +Components: $component
> +Description: Tarantool DBMS and Tarantool modules
> +SignWith: 91B625E5
> +DebIndices: Packages Release . .gz .bz2
> +UDebIndices: Packages . .gz .bz2
> +DscIndices: Sources Release .gz .bz2
> +
> +EOF
> + done
> +
> + # create standalone repository with separate components
> + for loop_dist in $alldists ; do
> + echo ================ DISTRIBUTION: $loop_dist ====================
> + updated_files=0
> +
> + # 1(binaries). use reprepro tool to generate Packages file
> + for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
> + [ -f $deb ] || continue
> + updated_deb=0
> + # regenerate DEB pack
> + update_deb_packfile $deb deb $loop_dist
> + echo "Regenerated DEB file: $locpackfile"
> + for packages in dists/$loop_dist/$component/binary-*/Packages ; do
> + # copy Packages file to avoid of removing by the new DEB version
> + # update metadata 'Packages' files
> + update_deb_metadata $packages deb
> + [ "$updated_deb" == "1" ] || continue
> + updated_files=1
> + done
> + # save the registered DEB file to S3
> + if [ "$updated_deb" == 1 ]; then
> + $aws_cp_public $deb $bucket_path/$locpackfile
> + fi
> + done
> +
> + # 1(sources). use reprepro tool to generate Sources file
> + for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
> + [ -f $dsc ] || continue
> + updated_dsc=0
> + # regenerate DSC pack
> + update_deb_packfile $dsc dsc $loop_dist
> + echo "Regenerated DSC file: $locpackfile"
> + # copy Sources file to avoid of removing by the new DSC version
> + # update metadata 'Sources' file
> + update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
> + [ "$updated_dsc" == "1" ] || continue
> + updated_files=1
> + # save the registered DSC file to S3
> + $aws_cp_public $dsc $bucket_path/$locpackfile
> + tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
> + $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
> + orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
> + $aws_cp_public $ws/$orig "$bucket_path/$orig"
> + done
> +
> + # check if any DEB/DSC files were newly registered
> + [ "$updated_files" == "0" ] && \
> + continue || echo "Updating dists"
> +
> + # finalize the Packages file
> + for packages in dists/$loop_dist/$component/binary-*/Packages ; do
> + mv $packages.saved $packages
> + done
> +
> + # finalize the Sources file
> + sources=dists/$loop_dist/$component/source/Sources
> + mv $sources.saved $sources
> +
> + # 2(binaries). update Packages file archives
> + for packpath in dists/$loop_dist/$component/binary-* ; do
> + pushd $packpath
> + sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
> + bzip2 -c Packages >Packages.bz2
> + gzip -c Packages >Packages.gz
> + popd
> + done
> +
> + # 2(sources). update Sources file archives
> + pushd dists/$loop_dist/$component/source
> + sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
> + bzip2 -c Sources >Sources.bz2
> + gzip -c Sources >Sources.gz
> + popd
> +
> + # 3. update checksums entries of the Packages* files in *Release files
> + # NOTE: it is stable structure of the *Release files when the checksum
> + # entries in it in the following way:
> + # MD5Sum:
> + # <checksum> <size> <file orig>
> + # <checksum> <size> <file debian>
> + # SHA1:
> + # <checksum> <size> <file orig>
> + # <checksum> <size> <file debian>
> + # SHA256:
> + # <checksum> <size> <file orig>
> + # <checksum> <size> <file debian>
> + # The script bellow puts 'md5' value at the 1st found file entry,
> + # 'sha1' - at the 2nd and 'sha256' at the 3rd
> + pushd dists/$loop_dist
> + for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
> + sz=$(stat -c "%s" $file)
> + md5=$(md5sum $file | awk '{print $1}')
> + sha1=$(sha1sum $file | awk '{print $1}')
> + sha256=$(sha256sum $file | awk '{print $1}')
> + awk 'BEGIN{c = 0} ; {
> + if ($3 == p) {
> + c = c + 1
> + if (c == 1) {print " " md " " s " " p}
> + if (c == 2) {print " " sh1 " " s " " p}
> + if (c == 3) {print " " sh2 " " s " " p}
> + } else {print $0}
> + }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
> + Release >Release.new
> + mv Release.new Release
> + done
> + # resign the selfsigned InRelease file
> + $rm_file InRelease
> + gpg --clearsign -o InRelease Release
> + # resign the Release file
> + $rm_file Release.gpg
> + gpg -abs -o Release.gpg Release
> + popd
> +
> + # 4. sync the latest distribution path changes to S3
> + $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
> + done
> +
> + # unlock the publishing
> + $rm_file $ws_lockfile
> +
> + popd
> +}
> +
> +# The 'pack_rpm' function especialy created for RPM packages. It works
> +# with RPM packing OS like Centos, Fedora. It is based on globaly known
> +# tool 'createrepo' from:
> +# https://linux.die.net/man/8/createrepo
> +# This tool works with single distribution of the given OS.
> +# Result of the routine is the rpm package for YUM repository with
> +# file structure equal to the Centos/Fedora:
> +# http://mirror.centos.org/centos/7/os/x86_64/Packages/
> +# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
> +function pack_rpm {
> + if ! ls $repo/*.rpm >/dev/null ; then
> + echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
> + usage
> + exit 1
> + fi
> +
> + # prepare the workspace
> + prepare_ws ${os}_${option_dist}
> +
> + # copy the needed package binaries to the workspace
> + cp $repo/*.rpm $ws/.
> +
> + pushd $ws
> +
> + # set the paths
> + if [ "$os" == "el" ]; then
> + repopath=$option_dist/os/x86_64
> + rpmpath=Packages
> + elif [ "$os" == "fedora" ]; then
> + repopath=releases/$option_dist/Everything/x86_64/os
> + rpmpath=Packages/$proddir
> + fi
> + packpath=$repopath/$rpmpath
> +
> + # prepare local repository with packages
> + $mk_dir $packpath
> + mv *.rpm $packpath/.
> + cd $repopath
> +
> + # copy the current metadata files from S3
> + mkdir repodata.base
> + for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
> + $aws ls $bucket_path/$repopath/repodata/$file || continue
> + $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
> + done
> +
> + # create the new repository metadata files
> + createrepo --no-database --update --workers=2 \
> + --compress-type=gz --simple-md-filenames .
> +
> + updated_rpms=0
> + # loop by the new hashes from the new meta file
> + for hash in $(zcat repodata/other.xml.gz | grep "<package pkgid=" | \
> + awk -F'"' '{print $2}') ; do
> + updated_rpm=0
> + name=$(zcat repodata/other.xml.gz | grep "<package pkgid=\"$hash\"" | \
> + awk -F'"' '{print $4}')
> + # search the new hash in the old meta file from S3
> + if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \
> + grep "name=\"$name\"" ; then
> + echo "WARNING: $name file already registered in S3!"
> + echo "File hash: $hash"
> + continue
> + fi
> + updated_rpms=1
> + # check if the hashed file already exists in old meta file from S3
> + file=$(zcat repodata/primary.xml.gz | \
> + grep -e "<checksum type=" -e "<location href=" | \
> + grep "$hash" -A1 | grep "<location href=" | \
> + awk -F'"' '{print $2}')
> + # check if the file already exists in S3
> + if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \
> + grep "<location href=\"$file\"" ; then
> + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
> + echo "New hash: $hash"
> + # unlock the publishing
> + $rm_file $ws_lockfile
> + exit 1
> + fi
> + done
> +
> + # check if any RPM files were newly registered
> + [ "$updated_rpms" == "0" ] && \
> + return || echo "Updating dists"
> +
> + # move the repodata files to the standalone location
> + mv repodata repodata.adding
> +
> + # merge metadata files
> + mkdir repodata
> + head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
> + for file in filelists.xml other.xml primary.xml ; do
> + # 1. take the 1st line only - to skip the line with
> + # number of packages which is not needed
> + zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
> + # 2. take 2nd line with metadata tag and update
> + # the packages number in it
> + packsold=0
> + if [ -f repodata.base/$file.gz ] ; then
> + packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
> + tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
> + fi
> + packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
> + tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
> + packs=$(($packsold+$packsnew))
> + zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
> + sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
> + # 3. take only 'package' tags from new file
> + zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
> + >>repodata/$file
> + # 4. take only 'package' tags from old file if exists
> + if [ -f repodata.base/$file.gz ] ; then
> + zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
> + >>repodata/$file
> + fi
> + # 5. take the last closing line with metadata tag
> + zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
> +
> + # get the new data
> + chsnew=$(sha256sum repodata/$file | awk '{print $1}')
> + sz=$(stat --printf="%s" repodata/$file)
> + gzip repodata/$file
> + chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
> + szgz=$(stat --printf="%s" repodata/$file.gz)
> + timestamp=$(date +%s -r repodata/$file.gz)
> +
> + # add info to repomd.xml file
> + name=$(echo $file | sed 's#\.xml$##g')
> + cat <<EOF >>repodata/repomd.xml
> +<data type="$name">
> + <checksum type="sha256">$chsgznew</checksum>
> + <open-checksum type="sha256">$chsnew</open-checksum>
> + <location href="repodata/$file.gz"/>
> + <timestamp>$timestamp</timestamp>
> + <size>$szgz</size>
> + <open-size>$sz</open-size>
> +</data>"
> +EOF
> + done
> + tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
> + gpg --detach-sign --armor repodata/repomd.xml
> +
> + # copy the packages to S3
> + for file in $rpmpath/*.rpm ; do
> + $aws_cp_public $file "$bucket_path/$repopath/$file"
> + done
> +
> + # update the metadata at the S3
> + $aws_sync_public repodata "$bucket_path/$repopath/repodata"
> +
> + # unlock the publishing
> + $rm_file $ws_lockfile
> +
> + popd
> +}
> +
> +if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
> + pack_deb
> +elif [ "$os" == "el" -o "$os" == "fedora" ]; then
> + pack_rpm
> +else
> + echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
> + usage
> + exit 1
> +fi
> --
> 2.17.1
--
Konstantin Osipov, Moscow, Russia
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3
2020-01-31 6:00 [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3 Alexander V. Tikhonov
2020-01-31 8:26 ` Konstantin Osipov
@ 2020-01-31 13:28 ` Oleg Piskunov
2020-01-31 22:42 ` Alexander Turenko
1 sibling, 1 reply; 5+ messages in thread
From: Oleg Piskunov @ 2020-01-31 13:28 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 29319 bytes --]
Patch introduce preparation for changing packages deploying process:
1. Instead of PackageCloud storage will be used S3 compatible storage.
The main reason for such changes is that PackageCloud repository is
limited with space and not enough for keeping all packages.
2. Introduce two new root repositories:
* "live" for packages from master and release branches.
* "release" for packages from tagged commits
3. Developer branches with "-full-ci" building packages but don’t deploy.
For these purposes the standalone script for pushing DEB and RPM
packages to the new S3 storage is used. For creating RPM packages
it uses 'createrepo' utility and for DEB packages - 'reprepro' utility.
The script implements the following flow:
* creates new metafiles for the new packages
* fetches relevant metafiles from the storage
* copies new packages to S3 storage
* merges and pushes the updated metadata to S3 storage
>Пятница, 31 января 2020, 9:00 +03:00 от Alexander V. Tikhonov <avtikhon@tarantool.org>:
>
>The changes introduce new Gitlab-CI rules for creating packages on
>branches with "-full-ci" suffix and their subsequent deployment to the
>'live' repository for master and release branches. Packages for tagged
>commits are also delivered to the corresponding 'release' repository.
>Packages creation activities relocating from the PackageCloud storage
>to the new self-hosted MCS S3 where all old packages have been synced.
>Benefits of the introduced approach are the following:
>* As far as all contents of self-hosted repos are fully controlled
>theirs layout is the same as the ones provided by the corresponding
>distro
>* Repo metadata rebuild is excess considering the known repo layout
>* Old packages are not pruned since they do not affect the repo
>metadata rebuilding time
>
>For these purposes the standalone script for pushing DEB and RPM
>packages to self-hosted repositories is introduced. The script
>implements the following flow:
>* creates new metafiles for the new packages
>* copies new packages to S3 storage
>* fetches relevant metafiles from the repo
>* merges the new metadata with the fetched one
>* pushes the updated metadata to S3 storage
>
>There are OS distribution dependent parts in the script:
>* for RPM packages it updates metadata separately per each OS
>distribution considering 'createrepo' util behavior
>* for DEB packages it updates metadata simultaniously for all
>distributions within single OS considering 'reprepro' util behaviour
>
>Closes #3380
>---
>
>Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
>Issue: https://github.com/tarantool/tarantool/issues/3380
>
>v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html
>v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html
>v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
>v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
>v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
>v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
>v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
>
>Changes v8:
>- corrected commit message
>- removed extra check for bucket naming
>- changed bucket naming from underscore to dot, like: 2_2 to 2.x
>- changed git tag checking routine as suggested
>
>Changes v7:
>- removed additional functionality for working with DEB repositories
> using complete pool path w/o specifing packages
>- implemented new flag '-f|--force' that helps to overwite the packages
> at MCS S3 if it checksum changed - implemented check on the new
> packages for it
>- implemented check with warning on the new RPM packages with the same checksum
>
>Changes v6:
>- implemented 2 MCS S3 repositories 'live' and 'release'
>- added AWS and GPG keys into Gitlab-CI
>- corrected commit message
>- corrected return functionality code in script
>- moved all changes for sources tarballs at the standalone patch set
>
>Changes v5:
>- code style
>- commits squashed
>- rebased to master
>
>Changes v4:
>- minor corrections
>
>Changes v3:
>- common code parts merged to standalone routines
>- corrected code style, minor updates
>- script is ready for release
>
>Changes v2:
>- made changes in script from draft to pre-release stages
>
> .gitlab-ci.yml | 152 ++++++++++--
> .gitlab.mk | 20 +-
> tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 723 insertions(+), 20 deletions(-)
> create mode 100755 tools/update_repo.sh
>
>diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>index 3af5a3c8a..c68594c1a 100644
>--- a/.gitlab-ci.yml
>+++ b/.gitlab-ci.yml
>@@ -10,6 +10,10 @@ variables:
> only:
> refs:
> - master
>+
>+.fullci_only_template: &fullci_only_definition
>+ only:
>+ refs:
> - /^.*-full-ci$/
>
> .docker_test_template: &docker_test_definition
>@@ -24,13 +28,29 @@ variables:
> tags:
> - docker_test
>
>+.pack_template: &pack_definition
>+ <<: *fullci_only_definition
>+ stage: test
>+ tags:
>+ - deploy
>+ script:
>+ - ${GITLAB_MAKE} package
>+
>+.pack_test_template: &pack_test_definition
>+ <<: *fullci_only_definition
>+ stage: test
>+ tags:
>+ - deploy_test
>+ script:
>+ - ${GITLAB_MAKE} package
>+
> .deploy_template: &deploy_definition
> <<: *release_only_definition
> stage: test
> tags:
> - deploy
> script:
>- - ${GITLAB_MAKE} package
>+ - ${GITLAB_MAKE} deploy
>
> .deploy_test_template: &deploy_test_definition
> <<: *release_only_definition
>@@ -38,7 +58,7 @@ variables:
> tags:
> - deploy_test
> script:
>- - ${GITLAB_MAKE} package
>+ - ${GITLAB_MAKE} deploy
>
> .vbox_template: &vbox_definition
> stage: test
>@@ -141,96 +161,194 @@ freebsd_12_release:
> # Packs
>
> centos_6:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'el'
> DIST: '6'
>
> centos_7:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'el'
> DIST: '7'
>
> centos_8:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'el'
> DIST: '8'
>
> fedora_28:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '28'
>
> fedora_29:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '29'
>
> fedora_30:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '30'
>
> fedora_31:
>- <<: *deploy_test_definition
>+ <<: *pack_test_definition
> variables:
> OS: 'fedora'
> DIST: '31'
>
> ubuntu_14_04:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'trusty'
>
> ubuntu_16_04:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'xenial'
>
> ubuntu_18_04:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'bionic'
>
> ubuntu_18_10:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'cosmic'
>
> ubuntu_19_04:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'disco'
>
> ubuntu_19_10:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'ubuntu'
> DIST: 'eoan'
>
> debian_8:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'debian'
> DIST: 'jessie'
>
> debian_9:
>- <<: *deploy_definition
>+ <<: *pack_definition
> variables:
> OS: 'debian'
> DIST: 'stretch'
>
> debian_10:
>+ <<: *pack_definition
>+ variables:
>+ OS: 'debian'
>+ DIST: 'buster'
>+
>+# Deploy
>+
>+centos_6_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'el'
>+ DIST: '6'
>+
>+centos_7_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'el'
>+ DIST: '7'
>+
>+centos_8_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'el'
>+ DIST: '8'
>+
>+fedora_28_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'fedora'
>+ DIST: '28'
>+
>+fedora_29_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'fedora'
>+ DIST: '29'
>+
>+fedora_30_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'fedora'
>+ DIST: '30'
>+
>+fedora_31_deploy:
>+ <<: *deploy_test_definition
>+ variables:
>+ OS: 'fedora'
>+ DIST: '31'
>+
>+ubuntu_14_04_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'trusty'
>+
>+ubuntu_16_04_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'xenial'
>+
>+ubuntu_18_04_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'bionic'
>+
>+ubuntu_18_10_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'cosmic'
>+
>+ubuntu_19_04_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'disco'
>+
>+ubuntu_19_10_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'ubuntu'
>+ DIST: 'eoan'
>+
>+debian_8_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'debian'
>+ DIST: 'jessie'
>+
>+debian_9_deploy:
>+ <<: *deploy_definition
>+ variables:
>+ OS: 'debian'
>+ DIST: 'stretch'
>+
>+debian_10_deploy:
> <<: *deploy_definition
> variables:
> OS: 'debian'
>diff --git a/.gitlab.mk b/.gitlab.mk
>index 48a92e518..1f921fd6e 100644
>--- a/.gitlab.mk
>+++ b/.gitlab.mk
>@@ -98,14 +98,28 @@ vms_test_%:
> vms_shutdown:
> VBoxManage controlvm ${VMS_NAME} poweroff
>
>-# ########################
>-# Build RPM / Deb packages
>-# ########################
>+# ########
>+# Packages
>+# ########
>+
>+GIT_DESCRIBE=$(shell git describe HEAD)
>+MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>+MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>+BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)"
>
> package: git_submodule_update
> git clone https://github.com/packpack/packpack.git packpack
> PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>
>+deploy: package
>+ for key in ${GPG_SECRET_KEY} ${GPG_PUBLIC_KEY} ; do \
>+ echo ${key} | base64 -d | gpg --batch --import || true ; done
>+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
>+ -b="s3://tarantool_repo/live/${BUCKET}" build
>+ git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
>+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
>+ -b="s3://tarantool_repo/release/${BUCKET}" build
>+
> # ############
> # Static build
> # ############
>diff --git a/tools/update_repo.sh b/tools/update_repo.sh
>new file mode 100755
>index 000000000..00e7aa0c6
>--- /dev/null
>+++ b/tools/update_repo.sh
>@@ -0,0 +1,571 @@
>+#!/bin/bash
>+set -e
>+
>+rm_file='rm -f'
>+rm_dir='rm -rf'
>+mk_dir='mkdir -p'
>+ws_prefix=/tmp/tarantool_repo_s3
>+
>+alloss='ubuntu debian el fedora'
>+product=tarantool
>+force=
>+# the path with binaries either repository
>+repo=.
>+
>+# AWS defines
>+aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3"
>+aws_cp_public="$aws cp --acl public-read"
>+aws_sync_public="$aws sync --acl public-read"
>+
>+function get_os_dists {
>+ os=$1
>+ alldists=
>+
>+ if [ "$os" == "ubuntu" ]; then
>+ alldists='trusty xenial bionic cosmic disco eoan'
>+ elif [ "$os" == "debian" ]; then
>+ alldists='jessie stretch buster'
>+ elif [ "$os" == "el" ]; then
>+ alldists='6 7 8'
>+ elif [ "$os" == "fedora" ]; then
>+ alldists='27 28 29 30 31'
>+ fi
>+
>+ echo "$alldists"
>+}
>+
>+function prepare_ws {
>+ # temporary lock the publication to the repository
>+ ws_suffix=$1
>+ ws=${ws_prefix}_${ws_suffix}
>+ ws_lockfile=${ws}.lock
>+ if [ -f $ws_lockfile ]; then
>+ old_proc=$(cat $ws_lockfile)
>+ fi
>+ lockfile -l 60 $ws_lockfile
>+ chmod u+w $ws_lockfile && echo $ >$ws_lockfile && chmod u-w $ws_lockfile
>+ if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then
>+ kill -9 $old_proc >/dev/null || true
>+ fi
>+
>+ # create temporary workspace for the new files
>+ $rm_dir $ws
>+ $mk_dir $ws
>+}
>+
>+function usage {
>+ cat <<EOF
>+Usage for store package binaries from the given path:
>+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to package binaries>
>+
>+Usage for mirroring Debian|Ubuntu OS repositories:
>+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to packages binaries>
>+
>+Arguments:
>+ <path>
>+ Path points to the directory with deb/prm packages to be used.
>+
>+Options:
>+ -b|--bucket
>+ MCS S3 bucket already existing which will be used for storing the packages
>+ -o|--os
>+ OS to be checked, one of the list:
>+ $alloss
>+ -d|--distribution
>+ Distribution appropriate to the given OS:
>+EOF
>+ for os in $alloss ; do
>+ echo " $os: <"$(get_os_dists $os)">"
>+ done
>+ cat <<EOF
>+ -p|--product
>+ Product name to be packed with, default name is 'tarantool'
>+ -f|--force
>+ Force updating the remote package with the local one despite the checksum difference
>+ -h|--help
>+ Usage help message
>+EOF
>+}
>+
>+for i in "$@"
>+do
>+case $i in
>+ -b=*|--bucket=*)
>+ bucket="${i#*=}"
>+ shift # past argument=value
>+ ;;
>+ -o=*|--os=*)
>+ os="${i#*=}"
>+ if ! echo $alloss | grep -F -q -w $os ; then
>+ echo "ERROR: OS '$os' is not supported"
>+ usage
>+ exit 1
>+ fi
>+ shift # past argument=value
>+ ;;
>+ -d=*|--distribution=*)
>+ option_dist="${i#*=}"
>+ shift # past argument=value
>+ ;;
>+ -p=*|--product=*)
>+ product="${i#*=}"
>+ shift # past argument=value
>+ ;;
>+ -f|--force)
>+ force=1
>+ ;;
>+ -h|--help)
>+ usage
>+ exit 0
>+ ;;
>+ *)
>+ repo="${i#*=}"
>+ pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
>+ shift # past argument=value
>+ ;;
>+esac
>+done
>+
>+# check that all needed options were set and correct
>+if [ "$bucket" == "" ]; then
>+ echo "ERROR: need to set -b|--bucket bucket option, check usage"
>+ usage
>+ exit 1
>+fi
>+if [ "$option_dist" == "" ]; then
>+ echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage"
>+ usage
>+ exit 1
>+fi
>+if [ "$os" == "" ]; then
>+ echo "ERROR: need to set -o|--os OS name option, check usage"
>+ usage
>+ exit 1
>+fi
>+alldists=$(get_os_dists $os)
>+if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
>+ echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
>+ usage
>+ exit 1
>+fi
>+
>+# set the subpath with binaries based on literal character of the product name
>+proddir=$(echo $product | head -c 1)
>+
>+# set bucket path of the given OS in options
>+bucket_path="$bucket/$os"
>+
>+function update_deb_packfile {
>+ packfile=$1
>+ packtype=$2
>+ update_dist=$3
>+
>+ locpackfile=$(echo $packfile | sed "s#^$ws/##g")
>+ # register DEB/DSC pack file to Packages/Sources file
>+ reprepro -Vb . include$packtype $update_dist $packfile
>+ # reprepro copied DEB/DSC file to component which is not needed
>+ $rm_dir $debdir/$component
>+ # to have all sources avoid reprepro set DEB/DSC file to its own registry
>+ $rm_dir db
>+}
>+
>+function update_deb_metadata {
>+ packpath=$1
>+ packtype=$2
>+
>+ if [ ! -f $packpath.saved ] ; then
>+ # get the latest Sources file from S3 either create empty file
>+ $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
>+ $aws cp "$bucket_path/$packpath" $packpath.saved || \
>+ touch $packpath.saved
>+ fi
>+
>+ if [ "$packtype" == "dsc" ]; then
>+ # WORKAROUND: unknown why, but reprepro doesn`t save the Sources
>+ # file, lets recreate it manualy from it's zipped version
>+ gunzip -c $packpath.gz >$packpath
>+ # check if the DSC hash already exists in old Sources file from S3
>+ # find the hash from the new Sources file
>+ hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
>+ tail -n 1 | awk '{print $1}')
>+ # search the new hash in the old Sources file from S3
>+ if grep " $hash .* .*__BODY_HTML_PLACEHOLDER__quot; $packpath.saved ; then
>+ echo "WARNING: DSC file already registered in S3!"
>+ return
>+ fi
>+ # check if the DSC file already exists in old Sources file from S3
>+ file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}')
>+ if [ "$force" == "" ] && grep " .* .* $file__BODY_HTML_PLACEHOLDER__quot; $packpath.saved ; then
>+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
>+ echo "New hash: $hash"
>+ # unlock the publishing
>+ $rm_file $ws_lockfile
>+ exit 1
>+ fi
>+ updated_dsc=1
>+ elif [ "$packtype" == "deb" ]; then
>+ # check if the DEB file already exists in old Packages file from S3
>+ # find the hash from the new Packages file
>+ hash=$(grep '^SHA256: ' $packpath)
>+ # search the new hash in the old Packages file from S3
>+ if grep "^SHA256: $hash" $packpath.saved ; then
>+ echo "WARNING: DEB file already registered in S3!"
>+ return
>+ fi
>+ # check if the DEB file already exists in old Packages file from S3
>+ file=$(grep '^Filename:' | awk '{print $2}')
>+ if [ "$force" == "" ] && grep "Filename: $file__BODY_HTML_PLACEHOLDER__quot; $packpath.saved ; then
>+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
>+ echo "New hash: $hash"
>+ # unlock the publishing
>+ $rm_file $ws_lockfile
>+ exit 1
>+ fi
>+ updated_deb=1
>+ fi
>+ # store the new DEB entry
>+ cat $packpath >>$packpath.saved
>+}
>+
>+# The 'pack_deb' function especialy created for DEB packages. It works
>+# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
>+# tool 'reprepro' from:
>+# https://wiki.debian.org/DebianRepository/SetupWithReprepro
>+# This tool works with complete number of distributions of the given OS.
>+# Result of the routine is the debian package for APT repository with
>+# file structure equal to the Debian/Ubuntu:
>+# http://ftp.am.debian.org/debian/pool/main/t/tarantool/
>+# http://ftp.am.debian.org/ubuntu/pool/main/t/
>+function pack_deb {
>+ # we need to push packages into 'main' repository only
>+ component=main
>+
>+ # debian has special directory 'pool' for packages
>+ debdir=pool
>+
>+ # get packages from pointed location
>+ if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then
>+ echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found"
>+ usage
>+ exit 1
>+ fi
>+
>+ # prepare the workspace
>+ prepare_ws ${os}
>+
>+ # copy single distribution with binaries packages
>+ repopath=$ws/pool/${option_dist}/$component/$proddir/$product
>+ $mk_dir ${repopath}
>+ cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
>+ pushd $ws
>+
>+ # create the configuration file for 'reprepro' tool
>+ confpath=$ws/conf
>+ $rm_dir $confpath
>+ $mk_dir $confpath
>+
>+ for loop_dist in $alldists ; do
>+ cat <<EOF >>$confpath/distributions
>+Origin: Tarantool
>+Label: tarantool.org
>+Suite: stable
>+Codename: $loop_dist
>+Architectures: amd64 source
>+Components: $component
>+Description: Tarantool DBMS and Tarantool modules
>+SignWith: 91B625E5
>+DebIndices: Packages Release . .gz .bz2
>+UDebIndices: Packages . .gz .bz2
>+DscIndices: Sources Release .gz .bz2
>+
>+EOF
>+ done
>+
>+ # create standalone repository with separate components
>+ for loop_dist in $alldists ; do
>+ echo ================ DISTRIBUTION: $loop_dist ====================
>+ updated_files=0
>+
>+ # 1(binaries). use reprepro tool to generate Packages file
>+ for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
>+ [ -f $deb ] || continue
>+ updated_deb=0
>+ # regenerate DEB pack
>+ update_deb_packfile $deb deb $loop_dist
>+ echo "Regenerated DEB file: $locpackfile"
>+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
>+ # copy Packages file to avoid of removing by the new DEB version
>+ # update metadata 'Packages' files
>+ update_deb_metadata $packages deb
>+ [ "$updated_deb" == "1" ] || continue
>+ updated_files=1
>+ done
>+ # save the registered DEB file to S3
>+ if [ "$updated_deb" == 1 ]; then
>+ $aws_cp_public $deb $bucket_path/$locpackfile
>+ fi
>+ done
>+
>+ # 1(sources). use reprepro tool to generate Sources file
>+ for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
>+ [ -f $dsc ] || continue
>+ updated_dsc=0
>+ # regenerate DSC pack
>+ update_deb_packfile $dsc dsc $loop_dist
>+ echo "Regenerated DSC file: $locpackfile"
>+ # copy Sources file to avoid of removing by the new DSC version
>+ # update metadata 'Sources' file
>+ update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
>+ [ "$updated_dsc" == "1" ] || continue
>+ updated_files=1
>+ # save the registered DSC file to S3
>+ $aws_cp_public $dsc $bucket_path/$locpackfile
>+ tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
>+ $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
>+ orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
>+ $aws_cp_public $ws/$orig "$bucket_path/$orig"
>+ done
>+
>+ # check if any DEB/DSC files were newly registered
>+ [ "$updated_files" == "0" ] && \
>+ continue || echo "Updating dists"
>+
>+ # finalize the Packages file
>+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
>+ mv $packages.saved $packages
>+ done
>+
>+ # finalize the Sources file
>+ sources=dists/$loop_dist/$component/source/Sources
>+ mv $sources.saved $sources
>+
>+ # 2(binaries). update Packages file archives
>+ for packpath in dists/$loop_dist/$component/binary-* ; do
>+ pushd $packpath
>+ sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
>+ bzip2 -c Packages >Packages.bz2
>+ gzip -c Packages >Packages.gz
>+ popd
>+ done
>+
>+ # 2(sources). update Sources file archives
>+ pushd dists/$loop_dist/$component/source
>+ sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
>+ bzip2 -c Sources >Sources.bz2
>+ gzip -c Sources >Sources.gz
>+ popd
>+
>+ # 3. update checksums entries of the Packages* files in *Release files
>+ # NOTE: it is stable structure of the *Release files when the checksum
>+ # entries in it in the following way:
>+ # MD5Sum:
>+ # <checksum> <size> <file orig>
>+ # <checksum> <size> <file debian>
>+ # SHA1:
>+ # <checksum> <size> <file orig>
>+ # <checksum> <size> <file debian>
>+ # SHA256:
>+ # <checksum> <size> <file orig>
>+ # <checksum> <size> <file debian>
>+ # The script bellow puts 'md5' value at the 1st found file entry,
>+ # 'sha1' - at the 2nd and 'sha256' at the 3rd
>+ pushd dists/$loop_dist
>+ for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
>+ sz=$(stat -c "%s" $file)
>+ md5=$(md5sum $file | awk '{print $1}')
>+ sha1=$(sha1sum $file | awk '{print $1}')
>+ sha256=$(sha256sum $file | awk '{print $1}')
>+ awk 'BEGIN{c = 0} ; {
>+ if ($3 == p) {
>+ c = c + 1
>+ if (c == 1) {print " " md " " s " " p}
>+ if (c == 2) {print " " sh1 " " s " " p}
>+ if (c == 3) {print " " sh2 " " s " " p}
>+ } else {print $0}
>+ }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
>+ Release >Release.new
>+ mv Release.new Release
>+ done
>+ # resign the selfsigned InRelease file
>+ $rm_file InRelease
>+ gpg --clearsign -o InRelease Release
>+ # resign the Release file
>+ $rm_file Release.gpg
>+ gpg -abs -o Release.gpg Release
>+ popd
>+
>+ # 4. sync the latest distribution path changes to S3
>+ $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
>+ done
>+
>+ # unlock the publishing
>+ $rm_file $ws_lockfile
>+
>+ popd
>+}
>+
>+# The 'pack_rpm' function especialy created for RPM packages. It works
>+# with RPM packing OS like Centos, Fedora. It is based on globaly known
>+# tool 'createrepo' from:
>+# https://linux.die.net/man/8/createrepo
>+# This tool works with single distribution of the given OS.
>+# Result of the routine is the rpm package for YUM repository with
>+# file structure equal to the Centos/Fedora:
>+# http://mirror.centos.org/centos/7/os/x86_64/Packages/
>+# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
>+function pack_rpm {
>+ if ! ls $repo/*.rpm >/dev/null ; then
>+ echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
>+ usage
>+ exit 1
>+ fi
>+
>+ # prepare the workspace
>+ prepare_ws ${os}_${option_dist}
>+
>+ # copy the needed package binaries to the workspace
>+ cp $repo/*.rpm $ws/.
>+
>+ pushd $ws
>+
>+ # set the paths
>+ if [ "$os" == "el" ]; then
>+ repopath=$option_dist/os/x86_64
>+ rpmpath=Packages
>+ elif [ "$os" == "fedora" ]; then
>+ repopath=releases/$option_dist/Everything/x86_64/os
>+ rpmpath=Packages/$proddir
>+ fi
>+ packpath=$repopath/$rpmpath
>+
>+ # prepare local repository with packages
>+ $mk_dir $packpath
>+ mv *.rpm $packpath/.
>+ cd $repopath
>+
>+ # copy the current metadata files from S3
>+ mkdir repodata.base
>+ for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
>+ $aws ls $bucket_path/$repopath/repodata/$file || continue
>+ $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
>+ done
>+
>+ # create the new repository metadata files
>+ createrepo --no-database --update --workers=2 \
>+ --compress-type=gz --simple-md-filenames .
>+
>+ updated_rpms=0
>+ # loop by the new hashes from the new meta file
>+ for hash in $(zcat repodata/other.xml.gz | grep "<package pkgid=" | \
>+ awk -F'"' '{print $2}') ; do
>+ updated_rpm=0
>+ name=$(zcat repodata/other.xml.gz | grep "<package pkgid=\"$hash\"" | \
>+ awk -F'"' '{print $4}')
>+ # search the new hash in the old meta file from S3
>+ if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \
>+ grep "name=\"$name\"" ; then
>+ echo "WARNING: $name file already registered in S3!"
>+ echo "File hash: $hash"
>+ continue
>+ fi
>+ updated_rpms=1
>+ # check if the hashed file already exists in old meta file from S3
>+ file=$(zcat repodata/primary.xml.gz | \
>+ grep -e "<checksum type=" -e "<location href=" | \
>+ grep "$hash" -A1 | grep "<location href=" | \
>+ awk -F'"' '{print $2}')
>+ # check if the file already exists in S3
>+ if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \
>+ grep "<location href=\"$file\"" ; then
>+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
>+ echo "New hash: $hash"
>+ # unlock the publishing
>+ $rm_file $ws_lockfile
>+ exit 1
>+ fi
>+ done
>+
>+ # check if any RPM files were newly registered
>+ [ "$updated_rpms" == "0" ] && \
>+ return || echo "Updating dists"
>+
>+ # move the repodata files to the standalone location
>+ mv repodata repodata.adding
>+
>+ # merge metadata files
>+ mkdir repodata
>+ head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
>+ for file in filelists.xml other.xml primary.xml ; do
>+ # 1. take the 1st line only - to skip the line with
>+ # number of packages which is not needed
>+ zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
>+ # 2. take 2nd line with metadata tag and update
>+ # the packages number in it
>+ packsold=0
>+ if [ -f repodata.base/$file.gz ] ; then
>+ packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
>+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
>+ fi
>+ packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
>+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
>+ packs=$(($packsold+$packsnew))
>+ zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
>+ sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
>+ # 3. take only 'package' tags from new file
>+ zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
>+ >>repodata/$file
>+ # 4. take only 'package' tags from old file if exists
>+ if [ -f repodata.base/$file.gz ] ; then
>+ zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
>+ >>repodata/$file
>+ fi
>+ # 5. take the last closing line with metadata tag
>+ zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
>+
>+ # get the new data
>+ chsnew=$(sha256sum repodata/$file | awk '{print $1}')
>+ sz=$(stat --printf="%s" repodata/$file)
>+ gzip repodata/$file
>+ chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
>+ szgz=$(stat --printf="%s" repodata/$file.gz)
>+ timestamp=$(date +%s -r repodata/$file.gz)
>+
>+ # add info to repomd.xml file
>+ name=$(echo $file | sed 's#\.xml$##g')
>+ cat <<EOF >>repodata/repomd.xml
>+<data type="$name">
>+ <checksum type="sha256">$chsgznew</checksum>
>+ <open-checksum type="sha256">$chsnew</open-checksum>
>+ <location href="repodata/$file.gz"/>
>+ <timestamp>$timestamp</timestamp>
>+ <size>$szgz</size>
>+ <open-size>$sz</open-size>
>+</data>"
>+EOF
>+ done
>+ tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
>+ gpg --detach-sign --armor repodata/repomd.xml
>+
>+ # copy the packages to S3
>+ for file in $rpmpath/*.rpm ; do
>+ $aws_cp_public $file "$bucket_path/$repopath/$file"
>+ done
>+
>+ # update the metadata at the S3
>+ $aws_sync_public repodata "$bucket_path/$repopath/repodata"
>+
>+ # unlock the publishing
>+ $rm_file $ws_lockfile
>+
>+ popd
>+}
>+
>+if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
>+ pack_deb
>+elif [ "$os" == "el" -o "$os" == "fedora" ]; then
>+ pack_rpm
>+else
>+ echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
>+ usage
>+ exit 1
>+fi
>--
>2.17.1
>
--
Oleg Piskunov
[-- Attachment #2: Type: text/html, Size: 36158 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3
2020-01-31 8:26 ` Konstantin Osipov
@ 2020-01-31 18:01 ` Alexander Turenko
0 siblings, 0 replies; 5+ messages in thread
From: Alexander Turenko @ 2020-01-31 18:01 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: Oleg Piskunov, tarantool-patches
On Fri, Jan 31, 2020 at 11:26:39AM +0300, Konstantin Osipov wrote:
> * Alexander V. Tikhonov <avtikhon@tarantool.org> [20/01/31 10:43]:
>
> I understand it's a v8 and I am jumping on this late, but I'm
> really curious why:
>
> - you use shell in 2020.
> - you create a bunch of essentially proprietary scripts (noone is
> going to look at this) instead of making it a feature of packpack.
>
> Many people need to create repositories, and many people use
> packpack already. It's best if this is just added there.
Sure, a standalone tool like [1] would be better. Here I totally agree.
I would try to reuse mkrepo (my points are in [2]). But several my
co-workers think that the script is more maintainable, so I didn't
blocked this way.
[1]: https://github.com/packpack/packpack/blob/34180facf8a9f0c793b09220e2cfd2d68e1faf59/tools/packagecloud
[2]: https://github.com/tarantool/tarantool/issues/3380#issuecomment-527497014
WBR, Alexander Turenko.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3
2020-01-31 13:28 ` Oleg Piskunov
@ 2020-01-31 22:42 ` Alexander Turenko
0 siblings, 0 replies; 5+ messages in thread
From: Alexander Turenko @ 2020-01-31 22:42 UTC (permalink / raw)
To: Oleg Piskunov; +Cc: tarantool-patches
On Fri, Jan 31, 2020 at 04:28:41PM +0300, Oleg Piskunov wrote:
>
> Patch introduce preparation for changing packages deploying process:
> 1. Instead of PackageCloud storage will be used S3 compatible storage.
> The main reason for such changes is that PackageCloud repository is
> limited with space and not enough for keeping all packages.
> 2. Introduce two new root repositories:
> * "live" for packages from master and release branches.
> * "release" for packages from tagged commits
> 3. Developer branches with "-full-ci" building packages but don’t deploy.
>
> For these purposes the standalone script for pushing DEB and RPM
> packages to the new S3 storage is used. For creating RPM packages
> it uses 'createrepo' utility and for DEB packages - 'reprepro' utility.
>
> The script implements the following flow:
> * creates new metafiles for the new packages
> * fetches relevant metafiles from the storage
> * copies new packages to S3 storage
> * merges and pushes the updated metadata to S3 storage
Oleg, thanks for the new description!
I looked over it, tweaked here and there to adjust style or give better
explanation and ends with the variant below the email.
Alexander, please, consider both update your one.
WBR, Alexander Turenko.
----
gitlab-ci: push Deb/RPM packages to S3 based repos
We're going to use S3 compatible storage for Deb and RPM repositories
instead of packagecloud.io service. The main reason is that
packagecloud.io provides a limited amount of storage, which is not
enough for keeping all packages (w/o regular pruning of old versions).
Note: At the moment packages are still pushed to packagecloud.io from
Travis-CI. Disabling this is out of scope of this patch.
This patch implements saving of packages on an S3 compatible storage and
regeneration of a repository metadata.
The layout is a bit different from one we have on packagecloud.io.
packagecloud.io:
| - 1.10
| - 2.1
| - 2.2
| - ...
S3 compatible storage:
| - live
| - 1.10
| - 2.1
| - 2.2
| - ...
| - release
| - 1.10
| - 2.1
| - 2.2
| - ...
Both 'live' and 'release' repositories track release branches (named as
<major>.<minor>) and master branch. The difference is that 'live' is
updated on every push, but 'release' is only for tagged versions
(<major>.<minor>.<patch>.0).
Packages are also built on '*-full-ci' branches, but only for testing
purposes: they don't pushed anywhere.
The core logic is in the tools/update_repo.sh script, which implements
the following flow:
- create metadata for new packages
- fetch relevant metadata from the S3 storage
- push new packages to the S3 storage
- merge and push the updated metadata to the S3 storage
The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
repositories.
Closes #3380
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-01-31 22:42 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-31 6:00 [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3 Alexander V. Tikhonov
2020-01-31 8:26 ` Konstantin Osipov
2020-01-31 18:01 ` Alexander Turenko
2020-01-31 13:28 ` Oleg Piskunov
2020-01-31 22:42 ` Alexander Turenko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox