* [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos
@ 2020-02-04 5:37 Alexander V. Tikhonov
2020-02-04 7:52 ` Alexander Turenko
0 siblings, 1 reply; 4+ messages in thread
From: Alexander V. Tikhonov @ 2020-02-04 5:37 UTC (permalink / raw)
To: Oleg Piskunov, Igor Munkin, Alexander Turenko; +Cc: tarantool-patches
We're going to use S3 compatible storage for Deb and RPM repositories
instead of packagecloud.io service. The main reason is that
packagecloud.io provides a limited amount of storage, which is not
enough for keeping all packages (w/o regular pruning of old versions).
Note: At the moment packages are still pushed to packagecloud.io from
Travis-CI. Disabling this is out of scope of this patch.
This patch implements saving of packages on an S3 compatible storage and
regeneration of a repository metadata.
The layout is a bit different from one we have on packagecloud.io.
packagecloud.io:
| - 1.10
| - 2.1
| - 2.2
| - ...
S3 compatible storage:
| - live
| - 1.10
| - 2.1
| - 2.2
| - ...
| - release
| - 1.10
| - 2.1
| - 2.2
| - ...
Both 'live' and 'release' repositories track release branches (named as
<major>.<minor>) and master branch. The difference is that 'live' is
updated on every push, but 'release' is only for tagged versions
(<major>.<minor>.<patch>.0).
Packages are also built on '*-full-ci' branches, but only for testing
purposes: they don't pushed anywhere.
The core logic is in the tools/update_repo.sh script, which implements
the following flow:
- create metadata for new packages
- fetch relevant metadata from the S3 storage
- push new packages to the S3 storage
- merge and push the updated metadata to the S3 storage
The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
repositories.
Closes #3380
---
Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
Issue: https://github.com/tarantool/tarantool/issues/3380
v9: https://lists.tarantool.org/pipermail/tarantool-patches/2020-February/013908.html
v8: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013873.html
v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html
v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html
v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
Changes v10:
- added filter schedules to avoid of deploing on it
- removed dead code
Changes v9:
- changed commit message as suggested
- removed public key extra installation
- moved buckets naming to the gitlab-ci environment
- added filters on gitlab-ci jobs to run packing on "external_pull_requests"
and "merge_requests" flags and avoid of deploing on the same flags
Changes v8:
- corrected commit message
- removed extra check for bucket naming
- changed bucket naming from underscore to dot, like: 2_2 to 2.x
- changed git tag checking routine as suggested
Changes v7:
- removed additional functionality for working with DEB repositories
using complete pool path w/o specifing packages
- implemented new flag '-f|--force' that helps to overwite the packages
at MCS S3 if it checksum changed - implemented check on the new
packages for it
- implemented check with warning on the new RPM packages with the same checksum
Changes v6:
- implemented 2 MCS S3 repositories 'live' and 'release'
- added AWS and GPG keys into Gitlab-CI
- corrected commit message
- corrected return functionality code in script
- moved all changes for sources tarballs at the standalone patch set
Changes v5:
- code style
- commits squashed
- rebased to master
Changes v4:
- minor corrections
Changes v3:
- common code parts merged to standalone routines
- corrected code style, minor updates
- script is ready for release
Changes v2:
- made changes in script from draft to pre-release stages
.gitlab-ci.yml | 172 +++++++++++--
.gitlab.mk | 19 +-
tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 737 insertions(+), 25 deletions(-)
create mode 100755 tools/update_repo.sh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3af5a3c8a..727b7c94e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -8,9 +8,23 @@ variables:
.release_only_template: &release_only_definition
only:
- refs:
- - master
- - /^.*-full-ci$/
+ - master
+ - /^.*-full-ci$/
+
+.deploy_only_template: &deploy_only_definition
+ only:
+ - master
+ except:
+ - schedules
+ - external_pull_requests
+ - merge_requests
+
+.pack_only_template: &pack_only_definition
+ only:
+ - schedules
+ - external_pull_requests
+ - merge_requests
+ - /^.*-full-ci$/
.docker_test_template: &docker_test_definition
image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
@@ -24,22 +38,38 @@ variables:
tags:
- docker_test
-.deploy_template: &deploy_definition
- <<: *release_only_definition
+.pack_template: &pack_definition
+ <<: *pack_only_definition
stage: test
tags:
- deploy
script:
- ${GITLAB_MAKE} package
-.deploy_test_template: &deploy_test_definition
- <<: *release_only_definition
+.pack_test_template: &pack_test_definition
+ <<: *pack_only_definition
stage: test
tags:
- deploy_test
script:
- ${GITLAB_MAKE} package
+.deploy_template: &deploy_definition
+ <<: *deploy_only_definition
+ stage: test
+ tags:
+ - deploy
+ script:
+ - ${GITLAB_MAKE} deploy
+
+.deploy_test_template: &deploy_test_definition
+ <<: *deploy_only_definition
+ stage: test
+ tags:
+ - deploy_test
+ script:
+ - ${GITLAB_MAKE} deploy
+
.vbox_template: &vbox_definition
stage: test
before_script:
@@ -141,96 +171,194 @@ freebsd_12_release:
# Packs
centos_6:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'el'
DIST: '6'
centos_7:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '7'
centos_8:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '8'
fedora_28:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '28'
fedora_29:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '29'
fedora_30:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '30'
fedora_31:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '31'
ubuntu_14_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'trusty'
ubuntu_16_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'xenial'
ubuntu_18_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'bionic'
ubuntu_18_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'cosmic'
ubuntu_19_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'disco'
ubuntu_19_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'eoan'
debian_8:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'jessie'
debian_9:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'stretch'
debian_10:
+ <<: *pack_definition
+ variables:
+ OS: 'debian'
+ DIST: 'buster'
+
+# Deploy
+
+centos_6_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'el'
+ DIST: '6'
+
+centos_7_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '7'
+
+centos_8_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '8'
+
+fedora_28_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '28'
+
+fedora_29_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '29'
+
+fedora_30_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '30'
+
+fedora_31_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '31'
+
+ubuntu_14_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'trusty'
+
+ubuntu_16_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'xenial'
+
+ubuntu_18_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'bionic'
+
+ubuntu_18_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'cosmic'
+
+ubuntu_19_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'disco'
+
+ubuntu_19_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'eoan'
+
+debian_8_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'jessie'
+
+debian_9_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'stretch'
+
+debian_10_deploy:
<<: *deploy_definition
variables:
OS: 'debian'
diff --git a/.gitlab.mk b/.gitlab.mk
index 48a92e518..7cdcd6ae7 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -98,14 +98,27 @@ vms_test_%:
vms_shutdown:
VBoxManage controlvm ${VMS_NAME} poweroff
-# ########################
-# Build RPM / Deb packages
-# ########################
+# ########
+# Packages
+# ########
+
+GIT_DESCRIBE=$(shell git describe HEAD)
+MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
+MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
+BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)"
package: git_submodule_update
git clone https://github.com/packpack/packpack.git packpack
PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
+deploy: package
+ echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
+ git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build
+
# ############
# Static build
# ############
diff --git a/tools/update_repo.sh b/tools/update_repo.sh
new file mode 100755
index 000000000..65a977187
--- /dev/null
+++ b/tools/update_repo.sh
@@ -0,0 +1,571 @@
+#!/bin/bash
+set -e
+
+rm_file='rm -f'
+rm_dir='rm -rf'
+mk_dir='mkdir -p'
+ws_prefix=/tmp/tarantool_repo_s3
+
+alloss='ubuntu debian el fedora'
+product=tarantool
+force=
+# the path with binaries either repository
+repo=.
+
+# AWS defines
+aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3"
+aws_cp_public="$aws cp --acl public-read"
+aws_sync_public="$aws sync --acl public-read"
+
+function get_os_dists {
+ os=$1
+ alldists=
+
+ if [ "$os" == "ubuntu" ]; then
+ alldists='trusty xenial bionic cosmic disco eoan'
+ elif [ "$os" == "debian" ]; then
+ alldists='jessie stretch buster'
+ elif [ "$os" == "el" ]; then
+ alldists='6 7 8'
+ elif [ "$os" == "fedora" ]; then
+ alldists='27 28 29 30 31'
+ fi
+
+ echo "$alldists"
+}
+
+function prepare_ws {
+ # temporary lock the publication to the repository
+ ws_suffix=$1
+ ws=${ws_prefix}_${ws_suffix}
+ ws_lockfile=${ws}.lock
+ if [ -f $ws_lockfile ]; then
+ old_proc=$(cat $ws_lockfile)
+ fi
+ lockfile -l 60 $ws_lockfile
+ chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
+ if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then
+ kill -9 $old_proc >/dev/null || true
+ fi
+
+ # create temporary workspace for the new files
+ $rm_dir $ws
+ $mk_dir $ws
+}
+
+function usage {
+ cat <<EOF
+Usage for store package binaries from the given path:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to package binaries>
+
+Usage for mirroring Debian|Ubuntu OS repositories:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to packages binaries>
+
+Arguments:
+ <path>
+ Path points to the directory with deb/prm packages to be used.
+
+Options:
+ -b|--bucket
+ MCS S3 bucket already existing which will be used for storing the packages
+ -o|--os
+ OS to be checked, one of the list:
+ $alloss
+ -d|--distribution
+ Distribution appropriate to the given OS:
+EOF
+ for os in $alloss ; do
+ echo " $os: <"$(get_os_dists $os)">"
+ done
+ cat <<EOF
+ -p|--product
+ Product name to be packed with, default name is 'tarantool'
+ -f|--force
+ Force updating the remote package with the local one despite the checksum difference
+ -h|--help
+ Usage help message
+EOF
+}
+
+for i in "$@"
+do
+case $i in
+ -b=*|--bucket=*)
+ bucket="${i#*=}"
+ shift # past argument=value
+ ;;
+ -o=*|--os=*)
+ os="${i#*=}"
+ if ! echo $alloss | grep -F -q -w $os ; then
+ echo "ERROR: OS '$os' is not supported"
+ usage
+ exit 1
+ fi
+ shift # past argument=value
+ ;;
+ -d=*|--distribution=*)
+ option_dist="${i#*=}"
+ shift # past argument=value
+ ;;
+ -p=*|--product=*)
+ product="${i#*=}"
+ shift # past argument=value
+ ;;
+ -f|--force)
+ force=1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ repo="${i#*=}"
+ pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
+ shift # past argument=value
+ ;;
+esac
+done
+
+# check that all needed options were set and correct
+if [ "$bucket" == "" ]; then
+ echo "ERROR: need to set -b|--bucket bucket option, check usage"
+ usage
+ exit 1
+fi
+if [ "$option_dist" == "" ]; then
+ echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage"
+ usage
+ exit 1
+fi
+if [ "$os" == "" ]; then
+ echo "ERROR: need to set -o|--os OS name option, check usage"
+ usage
+ exit 1
+fi
+alldists=$(get_os_dists $os)
+if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
+ echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
+ usage
+ exit 1
+fi
+
+# set the subpath with binaries based on literal character of the product name
+proddir=$(echo $product | head -c 1)
+
+# set bucket path of the given OS in options
+bucket_path="$bucket/$os"
+
+function update_deb_packfile {
+ packfile=$1
+ packtype=$2
+ update_dist=$3
+
+ locpackfile=$(echo $packfile | sed "s#^$ws/##g")
+ # register DEB/DSC pack file to Packages/Sources file
+ reprepro -Vb . include$packtype $update_dist $packfile
+ # reprepro copied DEB/DSC file to component which is not needed
+ $rm_dir $debdir/$component
+ # to have all sources avoid reprepro set DEB/DSC file to its own registry
+ $rm_dir db
+}
+
+function update_deb_metadata {
+ packpath=$1
+ packtype=$2
+
+ if [ ! -f $packpath.saved ] ; then
+ # get the latest Sources file from S3 either create empty file
+ $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
+ $aws cp "$bucket_path/$packpath" $packpath.saved || \
+ touch $packpath.saved
+ fi
+
+ if [ "$packtype" == "dsc" ]; then
+ # WORKAROUND: unknown why, but reprepro doesn`t save the Sources
+ # file, lets recreate it manualy from it's zipped version
+ gunzip -c $packpath.gz >$packpath
+ # check if the DSC hash already exists in old Sources file from S3
+ # find the hash from the new Sources file
+ hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
+ tail -n 1 | awk '{print $1}')
+ # search the new hash in the old Sources file from S3
+ if grep " $hash .* .*$" $packpath.saved ; then
+ echo "WARNING: DSC file already registered in S3!"
+ return
+ fi
+ # check if the DSC file already exists in old Sources file from S3
+ file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}')
+ if [ "$force" == "" ] && grep " .* .* $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_dsc=1
+ elif [ "$packtype" == "deb" ]; then
+ # check if the DEB file already exists in old Packages file from S3
+ # find the hash from the new Packages file
+ hash=$(grep '^SHA256: ' $packpath)
+ # search the new hash in the old Packages file from S3
+ if grep "^SHA256: $hash" $packpath.saved ; then
+ echo "WARNING: DEB file already registered in S3!"
+ return
+ fi
+ # check if the DEB file already exists in old Packages file from S3
+ file=$(grep '^Filename:' $packpath | awk '{print $2}')
+ if [ "$force" == "" ] && grep "Filename: $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_deb=1
+ fi
+ # store the new DEB entry
+ cat $packpath >>$packpath.saved
+}
+
+# The 'pack_deb' function especialy created for DEB packages. It works
+# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
+# tool 'reprepro' from:
+# https://wiki.debian.org/DebianRepository/SetupWithReprepro
+# This tool works with complete number of distributions of the given OS.
+# Result of the routine is the debian package for APT repository with
+# file structure equal to the Debian/Ubuntu:
+# http://ftp.am.debian.org/debian/pool/main/t/tarantool/
+# http://ftp.am.debian.org/ubuntu/pool/main/t/
+function pack_deb {
+ # we need to push packages into 'main' repository only
+ component=main
+
+ # debian has special directory 'pool' for packages
+ debdir=pool
+
+ # get packages from pointed location
+ if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then
+ echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}
+
+ # copy single distribution with binaries packages
+ repopath=$ws/pool/${option_dist}/$component/$proddir/$product
+ $mk_dir ${repopath}
+ cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
+ pushd $ws
+
+ # create the configuration file for 'reprepro' tool
+ confpath=$ws/conf
+ $rm_dir $confpath
+ $mk_dir $confpath
+
+ for loop_dist in $alldists ; do
+ cat <<EOF >>$confpath/distributions
+Origin: Tarantool
+Label: tarantool.org
+Suite: stable
+Codename: $loop_dist
+Architectures: amd64 source
+Components: $component
+Description: Tarantool DBMS and Tarantool modules
+SignWith: 91B625E5
+DebIndices: Packages Release . .gz .bz2
+UDebIndices: Packages . .gz .bz2
+DscIndices: Sources Release .gz .bz2
+
+EOF
+ done
+
+ # create standalone repository with separate components
+ for loop_dist in $alldists ; do
+ echo ================ DISTRIBUTION: $loop_dist ====================
+ updated_files=0
+
+ # 1(binaries). use reprepro tool to generate Packages file
+ for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
+ [ -f $deb ] || continue
+ updated_deb=0
+ # regenerate DEB pack
+ update_deb_packfile $deb deb $loop_dist
+ echo "Regenerated DEB file: $locpackfile"
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ # copy Packages file to avoid of removing by the new DEB version
+ # update metadata 'Packages' files
+ update_deb_metadata $packages deb
+ [ "$updated_deb" == "1" ] || continue
+ updated_files=1
+ done
+ # save the registered DEB file to S3
+ if [ "$updated_deb" == 1 ]; then
+ $aws_cp_public $deb $bucket_path/$locpackfile
+ fi
+ done
+
+ # 1(sources). use reprepro tool to generate Sources file
+ for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
+ [ -f $dsc ] || continue
+ updated_dsc=0
+ # regenerate DSC pack
+ update_deb_packfile $dsc dsc $loop_dist
+ echo "Regenerated DSC file: $locpackfile"
+ # copy Sources file to avoid of removing by the new DSC version
+ # update metadata 'Sources' file
+ update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
+ [ "$updated_dsc" == "1" ] || continue
+ updated_files=1
+ # save the registered DSC file to S3
+ $aws_cp_public $dsc $bucket_path/$locpackfile
+ tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
+ $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
+ orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
+ $aws_cp_public $ws/$orig "$bucket_path/$orig"
+ done
+
+ # check if any DEB/DSC files were newly registered
+ [ "$updated_files" == "0" ] && \
+ continue || echo "Updating dists"
+
+ # finalize the Packages file
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ mv $packages.saved $packages
+ done
+
+ # finalize the Sources file
+ sources=dists/$loop_dist/$component/source/Sources
+ mv $sources.saved $sources
+
+ # 2(binaries). update Packages file archives
+ for packpath in dists/$loop_dist/$component/binary-* ; do
+ pushd $packpath
+ sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
+ bzip2 -c Packages >Packages.bz2
+ gzip -c Packages >Packages.gz
+ popd
+ done
+
+ # 2(sources). update Sources file archives
+ pushd dists/$loop_dist/$component/source
+ sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
+ bzip2 -c Sources >Sources.bz2
+ gzip -c Sources >Sources.gz
+ popd
+
+ # 3. update checksums entries of the Packages* files in *Release files
+ # NOTE: it is stable structure of the *Release files when the checksum
+ # entries in it in the following way:
+ # MD5Sum:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA1:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA256:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # The script bellow puts 'md5' value at the 1st found file entry,
+ # 'sha1' - at the 2nd and 'sha256' at the 3rd
+ pushd dists/$loop_dist
+ for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
+ sz=$(stat -c "%s" $file)
+ md5=$(md5sum $file | awk '{print $1}')
+ sha1=$(sha1sum $file | awk '{print $1}')
+ sha256=$(sha256sum $file | awk '{print $1}')
+ awk 'BEGIN{c = 0} ; {
+ if ($3 == p) {
+ c = c + 1
+ if (c == 1) {print " " md " " s " " p}
+ if (c == 2) {print " " sh1 " " s " " p}
+ if (c == 3) {print " " sh2 " " s " " p}
+ } else {print $0}
+ }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
+ Release >Release.new
+ mv Release.new Release
+ done
+ # resign the selfsigned InRelease file
+ $rm_file InRelease
+ gpg --clearsign -o InRelease Release
+ # resign the Release file
+ $rm_file Release.gpg
+ gpg -abs -o Release.gpg Release
+ popd
+
+ # 4. sync the latest distribution path changes to S3
+ $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
+ done
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+# The 'pack_rpm' function especialy created for RPM packages. It works
+# with RPM packing OS like Centos, Fedora. It is based on globaly known
+# tool 'createrepo' from:
+# https://linux.die.net/man/8/createrepo
+# This tool works with single distribution of the given OS.
+# Result of the routine is the rpm package for YUM repository with
+# file structure equal to the Centos/Fedora:
+# http://mirror.centos.org/centos/7/os/x86_64/Packages/
+# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
+function pack_rpm {
+ if ! ls $repo/*.rpm >/dev/null ; then
+ echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}_${option_dist}
+
+ # copy the needed package binaries to the workspace
+ cp $repo/*.rpm $ws/.
+
+ pushd $ws
+
+ # set the paths
+ if [ "$os" == "el" ]; then
+ repopath=$option_dist/os/x86_64
+ rpmpath=Packages
+ elif [ "$os" == "fedora" ]; then
+ repopath=releases/$option_dist/Everything/x86_64/os
+ rpmpath=Packages/$proddir
+ fi
+ packpath=$repopath/$rpmpath
+
+ # prepare local repository with packages
+ $mk_dir $packpath
+ mv *.rpm $packpath/.
+ cd $repopath
+
+ # copy the current metadata files from S3
+ mkdir repodata.base
+ for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
+ $aws ls $bucket_path/$repopath/repodata/$file || continue
+ $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
+ done
+
+ # create the new repository metadata files
+ createrepo --no-database --update --workers=2 \
+ --compress-type=gz --simple-md-filenames .
+
+ updated_rpms=0
+ # loop by the new hashes from the new meta file
+ for hash in $(zcat repodata/other.xml.gz | grep "<package pkgid=" | \
+ awk -F'"' '{print $2}') ; do
+ updated_rpm=0
+ name=$(zcat repodata/other.xml.gz | grep "<package pkgid=\"$hash\"" | \
+ awk -F'"' '{print $4}')
+ # search the new hash in the old meta file from S3
+ if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \
+ grep "name=\"$name\"" ; then
+ echo "WARNING: $name file already registered in S3!"
+ echo "File hash: $hash"
+ continue
+ fi
+ updated_rpms=1
+ # check if the hashed file already exists in old meta file from S3
+ file=$(zcat repodata/primary.xml.gz | \
+ grep -e "<checksum type=" -e "<location href=" | \
+ grep "$hash" -A1 | grep "<location href=" | \
+ awk -F'"' '{print $2}')
+ # check if the file already exists in S3
+ if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \
+ grep "<location href=\"$file\"" ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ done
+
+ # check if any RPM files were newly registered
+ [ "$updated_rpms" == "0" ] && \
+ return || echo "Updating dists"
+
+ # move the repodata files to the standalone location
+ mv repodata repodata.adding
+
+ # merge metadata files
+ mkdir repodata
+ head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
+ for file in filelists.xml other.xml primary.xml ; do
+ # 1. take the 1st line only - to skip the line with
+ # number of packages which is not needed
+ zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
+ # 2. take 2nd line with metadata tag and update
+ # the packages number in it
+ packsold=0
+ if [ -f repodata.base/$file.gz ] ; then
+ packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ fi
+ packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ packs=$(($packsold+$packsnew))
+ zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
+ sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
+ # 3. take only 'package' tags from new file
+ zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ # 4. take only 'package' tags from old file if exists
+ if [ -f repodata.base/$file.gz ] ; then
+ zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ fi
+ # 5. take the last closing line with metadata tag
+ zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
+
+ # get the new data
+ chsnew=$(sha256sum repodata/$file | awk '{print $1}')
+ sz=$(stat --printf="%s" repodata/$file)
+ gzip repodata/$file
+ chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
+ szgz=$(stat --printf="%s" repodata/$file.gz)
+ timestamp=$(date +%s -r repodata/$file.gz)
+
+ # add info to repomd.xml file
+ name=$(echo $file | sed 's#\.xml$##g')
+ cat <<EOF >>repodata/repomd.xml
+<data type="$name">
+ <checksum type="sha256">$chsgznew</checksum>
+ <open-checksum type="sha256">$chsnew</open-checksum>
+ <location href="repodata/$file.gz"/>
+ <timestamp>$timestamp</timestamp>
+ <size>$szgz</size>
+ <open-size>$sz</open-size>
+</data>"
+EOF
+ done
+ tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
+ gpg --detach-sign --armor repodata/repomd.xml
+
+ # copy the packages to S3
+ for file in $rpmpath/*.rpm ; do
+ $aws_cp_public $file "$bucket_path/$repopath/$file"
+ done
+
+ # update the metadata at the S3
+ $aws_sync_public repodata "$bucket_path/$repopath/repodata"
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
+ pack_deb
+elif [ "$os" == "el" -o "$os" == "fedora" ]; then
+ pack_rpm
+else
+ echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
+ usage
+ exit 1
+fi
--
2.17.1
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos
2020-02-04 5:37 [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos Alexander V. Tikhonov
@ 2020-02-04 7:52 ` Alexander Turenko
2020-02-04 8:38 ` Alexander Turenko
0 siblings, 1 reply; 4+ messages in thread
From: Alexander Turenko @ 2020-02-04 7:52 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
Pushed to master, 2.3, 2.2, 2.1 (w/o Fedora 31), 1.10.
CCed Kirill.
I'll file an issue for follow up activities.
WBR, Alexander Turenko.
On Tue, Feb 04, 2020 at 08:37:50AM +0300, Alexander V. Tikhonov wrote:
> We're going to use S3 compatible storage for Deb and RPM repositories
> instead of packagecloud.io service. The main reason is that
> packagecloud.io provides a limited amount of storage, which is not
> enough for keeping all packages (w/o regular pruning of old versions).
>
> Note: At the moment packages are still pushed to packagecloud.io from
> Travis-CI. Disabling this is out of scope of this patch.
>
> This patch implements saving of packages on an S3 compatible storage and
> regeneration of a repository metadata.
>
> The layout is a bit different from one we have on packagecloud.io.
>
> packagecloud.io:
>
> | - 1.10
> | - 2.1
> | - 2.2
> | - ...
>
> S3 compatible storage:
>
> | - live
> | - 1.10
> | - 2.1
> | - 2.2
> | - ...
> | - release
> | - 1.10
> | - 2.1
> | - 2.2
> | - ...
>
> Both 'live' and 'release' repositories track release branches (named as
> <major>.<minor>) and master branch. The difference is that 'live' is
> updated on every push, but 'release' is only for tagged versions
> (<major>.<minor>.<patch>.0).
>
> Packages are also built on '*-full-ci' branches, but only for testing
> purposes: they don't pushed anywhere.
>
> The core logic is in the tools/update_repo.sh script, which implements
> the following flow:
>
> - create metadata for new packages
> - fetch relevant metadata from the S3 storage
> - push new packages to the S3 storage
> - merge and push the updated metadata to the S3 storage
>
> The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
> repositories.
>
> Closes #3380
> ---
>
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/3380
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos
2020-02-04 7:52 ` Alexander Turenko
@ 2020-02-04 8:38 ` Alexander Turenko
0 siblings, 0 replies; 4+ messages in thread
From: Alexander Turenko @ 2020-02-04 8:38 UTC (permalink / raw)
To: Alexander V. Tikhonov; +Cc: Oleg Piskunov, tarantool-patches
On Tue, Feb 04, 2020 at 10:52:04AM +0300, Alexander Turenko wrote:
> Pushed to master, 2.3, 2.2, 2.1 (w/o Fedora 31), 1.10.
>
> CCed Kirill.
>
> I'll file an issue for follow up activities.
>
> WBR, Alexander Turenko.
Force pushed the following change:
diff --git a/.gitlab.mk b/.gitlab.mk
index 7cdcd6ae7..24b18b7f2 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -115,9 +115,10 @@ deploy: package
echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
./tools/update_repo.sh -o=${OS} -d=${DIST} \
-b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
- git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
+ if git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null ; then \
./tools/update_repo.sh -o=${OS} -d=${DIST} \
- -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build
+ -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build ; \
+ fi
# ############
# Static build
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos
@ 2020-02-03 14:44 Alexander V. Tikhonov
0 siblings, 0 replies; 4+ messages in thread
From: Alexander V. Tikhonov @ 2020-02-03 14:44 UTC (permalink / raw)
To: Oleg Piskunov, Igor Munkin, Alexander Turenko; +Cc: tarantool-patches
We're going to use S3 compatible storage for Deb and RPM repositories
instead of packagecloud.io service. The main reason is that
packagecloud.io provides a limited amount of storage, which is not
enough for keeping all packages (w/o regular pruning of old versions).
Note: At the moment packages are still pushed to packagecloud.io from
Travis-CI. Disabling this is out of scope of this patch.
This patch implements saving of packages on an S3 compatible storage and
regeneration of a repository metadata.
The layout is a bit different from one we have on packagecloud.io.
packagecloud.io:
| - 1.10
| - 2.1
| - 2.2
| - ...
S3 compatible storage:
| - live
| - 1.10
| - 2.1
| - 2.2
| - ...
| - release
| - 1.10
| - 2.1
| - 2.2
| - ...
Both 'live' and 'release' repositories track release branches (named as
<major>.<minor>) and master branch. The difference is that 'live' is
updated on every push, but 'release' is only for tagged versions
(<major>.<minor>.<patch>.0).
Packages are also built on '*-full-ci' branches, but only for testing
purposes: they don't pushed anywhere.
The core logic is in the tools/update_repo.sh script, which implements
the following flow:
- create metadata for new packages
- fetch relevant metadata from the S3 storage
- push new packages to the S3 storage
- merge and push the updated metadata to the S3 storage
The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
repositories.
Closes #3380
---
Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
Issue: https://github.com/tarantool/tarantool/issues/3380
v8: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013873.html
v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html
v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html
v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html
v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html
v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html
v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html
v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html
Changes v10:
- added filter schedules to avoid of deploing on it
- removed dead code
Changes v9:
- changed commit message as suggested
- removed public key extra installation
- moved buckets naming to the gitlab-ci environment
- added filters on gitlab-ci jobs to run packing on "external_pull_requests"
and "merge_requests" flags and avoid of deploing on the same flags
Changes v8:
- corrected commit message
- removed extra check for bucket naming
- changed bucket naming from underscore to dot, like: 2_2 to 2.x
- changed git tag checking routine as suggested
Changes v7:
- removed additional functionality for working with DEB repositories
using complete pool path w/o specifing packages
- implemented new flag '-f|--force' that helps to overwite the packages
at MCS S3 if it checksum changed - implemented check on the new
packages for it
- implemented check with warning on the new RPM packages with the same checksum
Changes v6:
- implemented 2 MCS S3 repositories 'live' and 'release'
- added AWS and GPG keys into Gitlab-CI
- corrected commit message
- corrected return functionality code in script
- moved all changes for sources tarballs at the standalone patch set
Changes v5:
- code style
- commits squashed
- rebased to master
Changes v4:
- minor corrections
Changes v3:
- common code parts merged to standalone routines
- corrected code style, minor updates
- script is ready for release
Changes v2:
- made changes in script from draft to pre-release stages
.gitlab-ci.yml | 172 +++++++++++--
.gitlab.mk | 19 +-
tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 737 insertions(+), 25 deletions(-)
create mode 100755 tools/update_repo.sh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 3af5a3c8a..727b7c94e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -8,9 +8,23 @@ variables:
.release_only_template: &release_only_definition
only:
- refs:
- - master
- - /^.*-full-ci$/
+ - master
+ - /^.*-full-ci$/
+
+.deploy_only_template: &deploy_only_definition
+ only:
+ - master
+ except:
+ - schedules
+ - external_pull_requests
+ - merge_requests
+
+.pack_only_template: &pack_only_definition
+ only:
+ - schedules
+ - external_pull_requests
+ - merge_requests
+ - /^.*-full-ci$/
.docker_test_template: &docker_test_definition
image: "${CI_REGISTRY}/${CI_PROJECT_PATH}/testing/debian-stretch:latest"
@@ -24,22 +38,38 @@ variables:
tags:
- docker_test
-.deploy_template: &deploy_definition
- <<: *release_only_definition
+.pack_template: &pack_definition
+ <<: *pack_only_definition
stage: test
tags:
- deploy
script:
- ${GITLAB_MAKE} package
-.deploy_test_template: &deploy_test_definition
- <<: *release_only_definition
+.pack_test_template: &pack_test_definition
+ <<: *pack_only_definition
stage: test
tags:
- deploy_test
script:
- ${GITLAB_MAKE} package
+.deploy_template: &deploy_definition
+ <<: *deploy_only_definition
+ stage: test
+ tags:
+ - deploy
+ script:
+ - ${GITLAB_MAKE} deploy
+
+.deploy_test_template: &deploy_test_definition
+ <<: *deploy_only_definition
+ stage: test
+ tags:
+ - deploy_test
+ script:
+ - ${GITLAB_MAKE} deploy
+
.vbox_template: &vbox_definition
stage: test
before_script:
@@ -141,96 +171,194 @@ freebsd_12_release:
# Packs
centos_6:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'el'
DIST: '6'
centos_7:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '7'
centos_8:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'el'
DIST: '8'
fedora_28:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '28'
fedora_29:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '29'
fedora_30:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '30'
fedora_31:
- <<: *deploy_test_definition
+ <<: *pack_test_definition
variables:
OS: 'fedora'
DIST: '31'
ubuntu_14_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'trusty'
ubuntu_16_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'xenial'
ubuntu_18_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'bionic'
ubuntu_18_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'cosmic'
ubuntu_19_04:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'disco'
ubuntu_19_10:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'ubuntu'
DIST: 'eoan'
debian_8:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'jessie'
debian_9:
- <<: *deploy_definition
+ <<: *pack_definition
variables:
OS: 'debian'
DIST: 'stretch'
debian_10:
+ <<: *pack_definition
+ variables:
+ OS: 'debian'
+ DIST: 'buster'
+
+# Deploy
+
+centos_6_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'el'
+ DIST: '6'
+
+centos_7_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '7'
+
+centos_8_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'el'
+ DIST: '8'
+
+fedora_28_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '28'
+
+fedora_29_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '29'
+
+fedora_30_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '30'
+
+fedora_31_deploy:
+ <<: *deploy_test_definition
+ variables:
+ OS: 'fedora'
+ DIST: '31'
+
+ubuntu_14_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'trusty'
+
+ubuntu_16_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'xenial'
+
+ubuntu_18_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'bionic'
+
+ubuntu_18_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'cosmic'
+
+ubuntu_19_04_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'disco'
+
+ubuntu_19_10_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'ubuntu'
+ DIST: 'eoan'
+
+debian_8_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'jessie'
+
+debian_9_deploy:
+ <<: *deploy_definition
+ variables:
+ OS: 'debian'
+ DIST: 'stretch'
+
+debian_10_deploy:
<<: *deploy_definition
variables:
OS: 'debian'
diff --git a/.gitlab.mk b/.gitlab.mk
index 48a92e518..7cdcd6ae7 100644
--- a/.gitlab.mk
+++ b/.gitlab.mk
@@ -98,14 +98,27 @@ vms_test_%:
vms_shutdown:
VBoxManage controlvm ${VMS_NAME} poweroff
-# ########################
-# Build RPM / Deb packages
-# ########################
+# ########
+# Packages
+# ########
+
+GIT_DESCRIBE=$(shell git describe HEAD)
+MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
+MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
+BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)"
package: git_submodule_update
git clone https://github.com/packpack/packpack.git packpack
PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
+deploy: package
+ echo ${GPG_SECRET_KEY} | base64 -d | gpg --batch --import || true
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="${LIVE_REPO_S3_DIR}/${BUCKET}" build
+ git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \
+ ./tools/update_repo.sh -o=${OS} -d=${DIST} \
+ -b="${RELEASE_REPO_S3_DIR}/${BUCKET}" build
+
# ############
# Static build
# ############
diff --git a/tools/update_repo.sh b/tools/update_repo.sh
new file mode 100755
index 000000000..00e7aa0c6
--- /dev/null
+++ b/tools/update_repo.sh
@@ -0,0 +1,571 @@
+#!/bin/bash
+set -e
+
+rm_file='rm -f'
+rm_dir='rm -rf'
+mk_dir='mkdir -p'
+ws_prefix=/tmp/tarantool_repo_s3
+
+alloss='ubuntu debian el fedora'
+product=tarantool
+force=
+# the path with binaries either repository
+repo=.
+
+# AWS defines
+aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3"
+aws_cp_public="$aws cp --acl public-read"
+aws_sync_public="$aws sync --acl public-read"
+
+function get_os_dists {
+ os=$1
+ alldists=
+
+ if [ "$os" == "ubuntu" ]; then
+ alldists='trusty xenial bionic cosmic disco eoan'
+ elif [ "$os" == "debian" ]; then
+ alldists='jessie stretch buster'
+ elif [ "$os" == "el" ]; then
+ alldists='6 7 8'
+ elif [ "$os" == "fedora" ]; then
+ alldists='27 28 29 30 31'
+ fi
+
+ echo "$alldists"
+}
+
+function prepare_ws {
+ # temporary lock the publication to the repository
+ ws_suffix=$1
+ ws=${ws_prefix}_${ws_suffix}
+ ws_lockfile=${ws}.lock
+ if [ -f $ws_lockfile ]; then
+ old_proc=$(cat $ws_lockfile)
+ fi
+ lockfile -l 60 $ws_lockfile
+ chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile
+ if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then
+ kill -9 $old_proc >/dev/null || true
+ fi
+
+ # create temporary workspace for the new files
+ $rm_dir $ws
+ $mk_dir $ws
+}
+
+function usage {
+ cat <<EOF
+Usage for store package binaries from the given path:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to package binaries>
+
+Usage for mirroring Debian|Ubuntu OS repositories:
+ $0 -o=<OS name> -d=<OS distribuition> -b=<S3 bucket> [-p=<product>] <path to packages binaries>
+
+Arguments:
+ <path>
+ Path points to the directory with deb/prm packages to be used.
+
+Options:
+ -b|--bucket
+ MCS S3 bucket already existing which will be used for storing the packages
+ -o|--os
+ OS to be checked, one of the list:
+ $alloss
+ -d|--distribution
+ Distribution appropriate to the given OS:
+EOF
+ for os in $alloss ; do
+ echo " $os: <"$(get_os_dists $os)">"
+ done
+ cat <<EOF
+ -p|--product
+ Product name to be packed with, default name is 'tarantool'
+ -f|--force
+ Force updating the remote package with the local one despite the checksum difference
+ -h|--help
+ Usage help message
+EOF
+}
+
+for i in "$@"
+do
+case $i in
+ -b=*|--bucket=*)
+ bucket="${i#*=}"
+ shift # past argument=value
+ ;;
+ -o=*|--os=*)
+ os="${i#*=}"
+ if ! echo $alloss | grep -F -q -w $os ; then
+ echo "ERROR: OS '$os' is not supported"
+ usage
+ exit 1
+ fi
+ shift # past argument=value
+ ;;
+ -d=*|--distribution=*)
+ option_dist="${i#*=}"
+ shift # past argument=value
+ ;;
+ -p=*|--product=*)
+ product="${i#*=}"
+ shift # past argument=value
+ ;;
+ -f|--force)
+ force=1
+ ;;
+ -h|--help)
+ usage
+ exit 0
+ ;;
+ *)
+ repo="${i#*=}"
+ pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null
+ shift # past argument=value
+ ;;
+esac
+done
+
+# check that all needed options were set and correct
+if [ "$bucket" == "" ]; then
+ echo "ERROR: need to set -b|--bucket bucket option, check usage"
+ usage
+ exit 1
+fi
+if [ "$option_dist" == "" ]; then
+ echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage"
+ usage
+ exit 1
+fi
+if [ "$os" == "" ]; then
+ echo "ERROR: need to set -o|--os OS name option, check usage"
+ usage
+ exit 1
+fi
+alldists=$(get_os_dists $os)
+if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then
+ echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'"
+ usage
+ exit 1
+fi
+
+# set the subpath with binaries based on literal character of the product name
+proddir=$(echo $product | head -c 1)
+
+# set bucket path of the given OS in options
+bucket_path="$bucket/$os"
+
+function update_deb_packfile {
+ packfile=$1
+ packtype=$2
+ update_dist=$3
+
+ locpackfile=$(echo $packfile | sed "s#^$ws/##g")
+ # register DEB/DSC pack file to Packages/Sources file
+ reprepro -Vb . include$packtype $update_dist $packfile
+ # reprepro copied DEB/DSC file to component which is not needed
+ $rm_dir $debdir/$component
+ # to have all sources avoid reprepro set DEB/DSC file to its own registry
+ $rm_dir db
+}
+
+function update_deb_metadata {
+ packpath=$1
+ packtype=$2
+
+ if [ ! -f $packpath.saved ] ; then
+ # get the latest Sources file from S3 either create empty file
+ $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \
+ $aws cp "$bucket_path/$packpath" $packpath.saved || \
+ touch $packpath.saved
+ fi
+
+ if [ "$packtype" == "dsc" ]; then
+ # WORKAROUND: unknown why, but reprepro doesn`t save the Sources
+ # file, lets recreate it manualy from it's zipped version
+ gunzip -c $packpath.gz >$packpath
+ # check if the DSC hash already exists in old Sources file from S3
+ # find the hash from the new Sources file
+ hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \
+ tail -n 1 | awk '{print $1}')
+ # search the new hash in the old Sources file from S3
+ if grep " $hash .* .*$" $packpath.saved ; then
+ echo "WARNING: DSC file already registered in S3!"
+ return
+ fi
+ # check if the DSC file already exists in old Sources file from S3
+ file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}')
+ if [ "$force" == "" ] && grep " .* .* $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_dsc=1
+ elif [ "$packtype" == "deb" ]; then
+ # check if the DEB file already exists in old Packages file from S3
+ # find the hash from the new Packages file
+ hash=$(grep '^SHA256: ' $packpath)
+ # search the new hash in the old Packages file from S3
+ if grep "^SHA256: $hash" $packpath.saved ; then
+ echo "WARNING: DEB file already registered in S3!"
+ return
+ fi
+ # check if the DEB file already exists in old Packages file from S3
+ file=$(grep '^Filename:' | awk '{print $2}')
+ if [ "$force" == "" ] && grep "Filename: $file$" $packpath.saved ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ updated_deb=1
+ fi
+ # store the new DEB entry
+ cat $packpath >>$packpath.saved
+}
+
+# The 'pack_deb' function especialy created for DEB packages. It works
+# with DEB packing OS like Ubuntu, Debian. It is based on globaly known
+# tool 'reprepro' from:
+# https://wiki.debian.org/DebianRepository/SetupWithReprepro
+# This tool works with complete number of distributions of the given OS.
+# Result of the routine is the debian package for APT repository with
+# file structure equal to the Debian/Ubuntu:
+# http://ftp.am.debian.org/debian/pool/main/t/tarantool/
+# http://ftp.am.debian.org/ubuntu/pool/main/t/
+function pack_deb {
+ # we need to push packages into 'main' repository only
+ component=main
+
+ # debian has special directory 'pool' for packages
+ debdir=pool
+
+ # get packages from pointed location
+ if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then
+ echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}
+
+ # copy single distribution with binaries packages
+ repopath=$ws/pool/${option_dist}/$component/$proddir/$product
+ $mk_dir ${repopath}
+ cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/.
+ pushd $ws
+
+ # create the configuration file for 'reprepro' tool
+ confpath=$ws/conf
+ $rm_dir $confpath
+ $mk_dir $confpath
+
+ for loop_dist in $alldists ; do
+ cat <<EOF >>$confpath/distributions
+Origin: Tarantool
+Label: tarantool.org
+Suite: stable
+Codename: $loop_dist
+Architectures: amd64 source
+Components: $component
+Description: Tarantool DBMS and Tarantool modules
+SignWith: 91B625E5
+DebIndices: Packages Release . .gz .bz2
+UDebIndices: Packages . .gz .bz2
+DscIndices: Sources Release .gz .bz2
+
+EOF
+ done
+
+ # create standalone repository with separate components
+ for loop_dist in $alldists ; do
+ echo ================ DISTRIBUTION: $loop_dist ====================
+ updated_files=0
+
+ # 1(binaries). use reprepro tool to generate Packages file
+ for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do
+ [ -f $deb ] || continue
+ updated_deb=0
+ # regenerate DEB pack
+ update_deb_packfile $deb deb $loop_dist
+ echo "Regenerated DEB file: $locpackfile"
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ # copy Packages file to avoid of removing by the new DEB version
+ # update metadata 'Packages' files
+ update_deb_metadata $packages deb
+ [ "$updated_deb" == "1" ] || continue
+ updated_files=1
+ done
+ # save the registered DEB file to S3
+ if [ "$updated_deb" == 1 ]; then
+ $aws_cp_public $deb $bucket_path/$locpackfile
+ fi
+ done
+
+ # 1(sources). use reprepro tool to generate Sources file
+ for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do
+ [ -f $dsc ] || continue
+ updated_dsc=0
+ # regenerate DSC pack
+ update_deb_packfile $dsc dsc $loop_dist
+ echo "Regenerated DSC file: $locpackfile"
+ # copy Sources file to avoid of removing by the new DSC version
+ # update metadata 'Sources' file
+ update_deb_metadata dists/$loop_dist/$component/source/Sources dsc
+ [ "$updated_dsc" == "1" ] || continue
+ updated_files=1
+ # save the registered DSC file to S3
+ $aws_cp_public $dsc $bucket_path/$locpackfile
+ tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g')
+ $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz"
+ orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g')
+ $aws_cp_public $ws/$orig "$bucket_path/$orig"
+ done
+
+ # check if any DEB/DSC files were newly registered
+ [ "$updated_files" == "0" ] && \
+ continue || echo "Updating dists"
+
+ # finalize the Packages file
+ for packages in dists/$loop_dist/$component/binary-*/Packages ; do
+ mv $packages.saved $packages
+ done
+
+ # finalize the Sources file
+ sources=dists/$loop_dist/$component/source/Sources
+ mv $sources.saved $sources
+
+ # 2(binaries). update Packages file archives
+ for packpath in dists/$loop_dist/$component/binary-* ; do
+ pushd $packpath
+ sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages
+ bzip2 -c Packages >Packages.bz2
+ gzip -c Packages >Packages.gz
+ popd
+ done
+
+ # 2(sources). update Sources file archives
+ pushd dists/$loop_dist/$component/source
+ sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources
+ bzip2 -c Sources >Sources.bz2
+ gzip -c Sources >Sources.gz
+ popd
+
+ # 3. update checksums entries of the Packages* files in *Release files
+ # NOTE: it is stable structure of the *Release files when the checksum
+ # entries in it in the following way:
+ # MD5Sum:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA1:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # SHA256:
+ # <checksum> <size> <file orig>
+ # <checksum> <size> <file debian>
+ # The script bellow puts 'md5' value at the 1st found file entry,
+ # 'sha1' - at the 2nd and 'sha256' at the 3rd
+ pushd dists/$loop_dist
+ for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do
+ sz=$(stat -c "%s" $file)
+ md5=$(md5sum $file | awk '{print $1}')
+ sha1=$(sha1sum $file | awk '{print $1}')
+ sha256=$(sha256sum $file | awk '{print $1}')
+ awk 'BEGIN{c = 0} ; {
+ if ($3 == p) {
+ c = c + 1
+ if (c == 1) {print " " md " " s " " p}
+ if (c == 2) {print " " sh1 " " s " " p}
+ if (c == 3) {print " " sh2 " " s " " p}
+ } else {print $0}
+ }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \
+ Release >Release.new
+ mv Release.new Release
+ done
+ # resign the selfsigned InRelease file
+ $rm_file InRelease
+ gpg --clearsign -o InRelease Release
+ # resign the Release file
+ $rm_file Release.gpg
+ gpg -abs -o Release.gpg Release
+ popd
+
+ # 4. sync the latest distribution path changes to S3
+ $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist"
+ done
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+# The 'pack_rpm' function especialy created for RPM packages. It works
+# with RPM packing OS like Centos, Fedora. It is based on globaly known
+# tool 'createrepo' from:
+# https://linux.die.net/man/8/createrepo
+# This tool works with single distribution of the given OS.
+# Result of the routine is the rpm package for YUM repository with
+# file structure equal to the Centos/Fedora:
+# http://mirror.centos.org/centos/7/os/x86_64/Packages/
+# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/
+function pack_rpm {
+ if ! ls $repo/*.rpm >/dev/null ; then
+ echo "ERROR: Current '$repo' path doesn't have RPM packages in path"
+ usage
+ exit 1
+ fi
+
+ # prepare the workspace
+ prepare_ws ${os}_${option_dist}
+
+ # copy the needed package binaries to the workspace
+ cp $repo/*.rpm $ws/.
+
+ pushd $ws
+
+ # set the paths
+ if [ "$os" == "el" ]; then
+ repopath=$option_dist/os/x86_64
+ rpmpath=Packages
+ elif [ "$os" == "fedora" ]; then
+ repopath=releases/$option_dist/Everything/x86_64/os
+ rpmpath=Packages/$proddir
+ fi
+ packpath=$repopath/$rpmpath
+
+ # prepare local repository with packages
+ $mk_dir $packpath
+ mv *.rpm $packpath/.
+ cd $repopath
+
+ # copy the current metadata files from S3
+ mkdir repodata.base
+ for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do
+ $aws ls $bucket_path/$repopath/repodata/$file || continue
+ $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file
+ done
+
+ # create the new repository metadata files
+ createrepo --no-database --update --workers=2 \
+ --compress-type=gz --simple-md-filenames .
+
+ updated_rpms=0
+ # loop by the new hashes from the new meta file
+ for hash in $(zcat repodata/other.xml.gz | grep "<package pkgid=" | \
+ awk -F'"' '{print $2}') ; do
+ updated_rpm=0
+ name=$(zcat repodata/other.xml.gz | grep "<package pkgid=\"$hash\"" | \
+ awk -F'"' '{print $4}')
+ # search the new hash in the old meta file from S3
+ if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \
+ grep "name=\"$name\"" ; then
+ echo "WARNING: $name file already registered in S3!"
+ echo "File hash: $hash"
+ continue
+ fi
+ updated_rpms=1
+ # check if the hashed file already exists in old meta file from S3
+ file=$(zcat repodata/primary.xml.gz | \
+ grep -e "<checksum type=" -e "<location href=" | \
+ grep "$hash" -A1 | grep "<location href=" | \
+ awk -F'"' '{print $2}')
+ # check if the file already exists in S3
+ if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \
+ grep "<location href=\"$file\"" ; then
+ echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file"
+ echo "New hash: $hash"
+ # unlock the publishing
+ $rm_file $ws_lockfile
+ exit 1
+ fi
+ done
+
+ # check if any RPM files were newly registered
+ [ "$updated_rpms" == "0" ] && \
+ return || echo "Updating dists"
+
+ # move the repodata files to the standalone location
+ mv repodata repodata.adding
+
+ # merge metadata files
+ mkdir repodata
+ head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml
+ for file in filelists.xml other.xml primary.xml ; do
+ # 1. take the 1st line only - to skip the line with
+ # number of packages which is not needed
+ zcat repodata.adding/$file.gz | head -n 1 >repodata/$file
+ # 2. take 2nd line with metadata tag and update
+ # the packages number in it
+ packsold=0
+ if [ -f repodata.base/$file.gz ] ; then
+ packsold=$(zcat repodata.base/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ fi
+ packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \
+ tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g')
+ packs=$(($packsold+$packsnew))
+ zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \
+ sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file
+ # 3. take only 'package' tags from new file
+ zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ # 4. take only 'package' tags from old file if exists
+ if [ -f repodata.base/$file.gz ] ; then
+ zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \
+ >>repodata/$file
+ fi
+ # 5. take the last closing line with metadata tag
+ zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file
+
+ # get the new data
+ chsnew=$(sha256sum repodata/$file | awk '{print $1}')
+ sz=$(stat --printf="%s" repodata/$file)
+ gzip repodata/$file
+ chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}')
+ szgz=$(stat --printf="%s" repodata/$file.gz)
+ timestamp=$(date +%s -r repodata/$file.gz)
+
+ # add info to repomd.xml file
+ name=$(echo $file | sed 's#\.xml$##g')
+ cat <<EOF >>repodata/repomd.xml
+<data type="$name">
+ <checksum type="sha256">$chsgznew</checksum>
+ <open-checksum type="sha256">$chsnew</open-checksum>
+ <location href="repodata/$file.gz"/>
+ <timestamp>$timestamp</timestamp>
+ <size>$szgz</size>
+ <open-size>$sz</open-size>
+</data>"
+EOF
+ done
+ tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml
+ gpg --detach-sign --armor repodata/repomd.xml
+
+ # copy the packages to S3
+ for file in $rpmpath/*.rpm ; do
+ $aws_cp_public $file "$bucket_path/$repopath/$file"
+ done
+
+ # update the metadata at the S3
+ $aws_sync_public repodata "$bucket_path/$repopath/repodata"
+
+ # unlock the publishing
+ $rm_file $ws_lockfile
+
+ popd
+}
+
+if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then
+ pack_deb
+elif [ "$os" == "el" -o "$os" == "fedora" ]; then
+ pack_rpm
+else
+ echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss"
+ usage
+ exit 1
+fi
--
2.17.1
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-02-04 8:37 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-04 5:37 [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM packages to S3 based repos Alexander V. Tikhonov
2020-02-04 7:52 ` Alexander Turenko
2020-02-04 8:38 ` Alexander Turenko
-- strict thread matches above, loose matches on Subject: below --
2020-02-03 14:44 Alexander V. Tikhonov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox