From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-f196.google.com (mail-lj1-f196.google.com [209.85.208.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 7668046970E for ; Fri, 31 Jan 2020 11:26:41 +0300 (MSK) Received: by mail-lj1-f196.google.com with SMTP id q8so6175979ljj.11 for ; Fri, 31 Jan 2020 00:26:41 -0800 (PST) Date: Fri, 31 Jan 2020 11:26:39 +0300 From: Konstantin Osipov Message-ID: <20200131082639.GD10740@atlas> References: <3840321b3c8368d4da86951c0a3e3bcfb3ded063.1580450098.git.avtikhon@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3840321b3c8368d4da86951c0a3e3bcfb3ded063.1580450098.git.avtikhon@tarantool.org> Subject: Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3 List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Alexander V. Tikhonov" Cc: Oleg Piskunov , tarantool-patches@dev.tarantool.org * Alexander V. Tikhonov [20/01/31 10:43]: I understand it's a v8 and I am jumping on this late, but I'm really curious why: - you use shell in 2020. - you create a bunch of essentially proprietary scripts (noone is going to look at this) instead of making it a feature of packpack. Many people need to create repositories, and many people use packpack already. It's best if this is just added there. > The changes introduce new Gitlab-CI rules for creating packages on > branches with "-full-ci" suffix and their subsequent deployment to the > 'live' repository for master and release branches. Packages for tagged > commits are also delivered to the corresponding 'release' repository. > > Packages creation activities relocating from the PackageCloud storage > to the new self-hosted MCS S3 where all old packages have been synced. > Benefits of the introduced approach are the following: > * As far as all contents of self-hosted repos are fully controlled > theirs layout is the same as the ones provided by the corresponding > distro > * Repo metadata rebuild is excess considering the known repo layout > * Old packages are not pruned since they do not affect the repo > metadata rebuilding time > > For these purposes the standalone script for pushing DEB and RPM > packages to self-hosted repositories is introduced. The script > implements the following flow: > * creates new metafiles for the new packages > * copies new packages to S3 storage > * fetches relevant metafiles from the repo > * merges the new metadata with the fetched one > * pushes the updated metadata to S3 storage > > There are OS distribution dependent parts in the script: > * for RPM packages it updates metadata separately per each OS > distribution considering 'createrepo' util behavior > * for DEB packages it updates metadata simultaniously for all > distributions within single OS considering 'reprepro' util behaviour > > Closes #3380 > --- > > Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci > Issue: https://github.com/tarantool/tarantool/issues/3380 > > v7: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013872.html > v6: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013763.html > v5: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013636.html > v4: https://lists.tarantool.org/pipermail/tarantool-patches/2020-January/013568.html > v3: https://lists.tarantool.org/pipermail/tarantool-patches/2019-December/013060.html > v2: https://lists.tarantool.org/pipermail/tarantool-patches/2019-November/012352.html > v1: https://lists.tarantool.org/pipermail/tarantool-patches/2019-October/012021.html > > Changes v8: > - corrected commit message > - removed extra check for bucket naming > - changed bucket naming from underscore to dot, like: 2_2 to 2.x > - changed git tag checking routine as suggested > > Changes v7: > - removed additional functionality for working with DEB repositories > using complete pool path w/o specifing packages > - implemented new flag '-f|--force' that helps to overwite the packages > at MCS S3 if it checksum changed - implemented check on the new > packages for it > - implemented check with warning on the new RPM packages with the same checksum > > Changes v6: > - implemented 2 MCS S3 repositories 'live' and 'release' > - added AWS and GPG keys into Gitlab-CI > - corrected commit message > - corrected return functionality code in script > - moved all changes for sources tarballs at the standalone patch set > > Changes v5: > - code style > - commits squashed > - rebased to master > > Changes v4: > - minor corrections > > Changes v3: > - common code parts merged to standalone routines > - corrected code style, minor updates > - script is ready for release > > Changes v2: > - made changes in script from draft to pre-release stages > > .gitlab-ci.yml | 152 ++++++++++-- > .gitlab.mk | 20 +- > tools/update_repo.sh | 571 +++++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 723 insertions(+), 20 deletions(-) > create mode 100755 tools/update_repo.sh > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 3af5a3c8a..c68594c1a 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -10,6 +10,10 @@ variables: > only: > refs: > - master > + > +.fullci_only_template: &fullci_only_definition > + only: > + refs: > - /^.*-full-ci$/ > > .docker_test_template: &docker_test_definition > @@ -24,13 +28,29 @@ variables: > tags: > - docker_test > > +.pack_template: &pack_definition > + <<: *fullci_only_definition > + stage: test > + tags: > + - deploy > + script: > + - ${GITLAB_MAKE} package > + > +.pack_test_template: &pack_test_definition > + <<: *fullci_only_definition > + stage: test > + tags: > + - deploy_test > + script: > + - ${GITLAB_MAKE} package > + > .deploy_template: &deploy_definition > <<: *release_only_definition > stage: test > tags: > - deploy > script: > - - ${GITLAB_MAKE} package > + - ${GITLAB_MAKE} deploy > > .deploy_test_template: &deploy_test_definition > <<: *release_only_definition > @@ -38,7 +58,7 @@ variables: > tags: > - deploy_test > script: > - - ${GITLAB_MAKE} package > + - ${GITLAB_MAKE} deploy > > .vbox_template: &vbox_definition > stage: test > @@ -141,96 +161,194 @@ freebsd_12_release: > # Packs > > centos_6: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'el' > DIST: '6' > > centos_7: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'el' > DIST: '7' > > centos_8: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'el' > DIST: '8' > > fedora_28: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'fedora' > DIST: '28' > > fedora_29: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'fedora' > DIST: '29' > > fedora_30: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'fedora' > DIST: '30' > > fedora_31: > - <<: *deploy_test_definition > + <<: *pack_test_definition > variables: > OS: 'fedora' > DIST: '31' > > ubuntu_14_04: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'trusty' > > ubuntu_16_04: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'xenial' > > ubuntu_18_04: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'bionic' > > ubuntu_18_10: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'cosmic' > > ubuntu_19_04: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'disco' > > ubuntu_19_10: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'ubuntu' > DIST: 'eoan' > > debian_8: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'debian' > DIST: 'jessie' > > debian_9: > - <<: *deploy_definition > + <<: *pack_definition > variables: > OS: 'debian' > DIST: 'stretch' > > debian_10: > + <<: *pack_definition > + variables: > + OS: 'debian' > + DIST: 'buster' > + > +# Deploy > + > +centos_6_deploy: > + <<: *deploy_definition > + variables: > + OS: 'el' > + DIST: '6' > + > +centos_7_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'el' > + DIST: '7' > + > +centos_8_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'el' > + DIST: '8' > + > +fedora_28_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'fedora' > + DIST: '28' > + > +fedora_29_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'fedora' > + DIST: '29' > + > +fedora_30_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'fedora' > + DIST: '30' > + > +fedora_31_deploy: > + <<: *deploy_test_definition > + variables: > + OS: 'fedora' > + DIST: '31' > + > +ubuntu_14_04_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'trusty' > + > +ubuntu_16_04_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'xenial' > + > +ubuntu_18_04_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'bionic' > + > +ubuntu_18_10_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'cosmic' > + > +ubuntu_19_04_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'disco' > + > +ubuntu_19_10_deploy: > + <<: *deploy_definition > + variables: > + OS: 'ubuntu' > + DIST: 'eoan' > + > +debian_8_deploy: > + <<: *deploy_definition > + variables: > + OS: 'debian' > + DIST: 'jessie' > + > +debian_9_deploy: > + <<: *deploy_definition > + variables: > + OS: 'debian' > + DIST: 'stretch' > + > +debian_10_deploy: > <<: *deploy_definition > variables: > OS: 'debian' > diff --git a/.gitlab.mk b/.gitlab.mk > index 48a92e518..1f921fd6e 100644 > --- a/.gitlab.mk > +++ b/.gitlab.mk > @@ -98,14 +98,28 @@ vms_test_%: > vms_shutdown: > VBoxManage controlvm ${VMS_NAME} poweroff > > -# ######################## > -# Build RPM / Deb packages > -# ######################## > +# ######## > +# Packages > +# ######## > + > +GIT_DESCRIBE=$(shell git describe HEAD) > +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE))) > +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE))) > +BUCKET="$(MAJOR_VERSION).$(MINOR_VERSION)" > > package: git_submodule_update > git clone https://github.com/packpack/packpack.git packpack > PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack > > +deploy: package > + for key in ${GPG_SECRET_KEY} ${GPG_PUBLIC_KEY} ; do \ > + echo $${key} | base64 -d | gpg --batch --import || true ; done > + ./tools/update_repo.sh -o=${OS} -d=${DIST} \ > + -b="s3://tarantool_repo/live/${BUCKET}" build > + git name-rev --name-only --tags --no-undefined HEAD 2>/dev/null && \ > + ./tools/update_repo.sh -o=${OS} -d=${DIST} \ > + -b="s3://tarantool_repo/release/${BUCKET}" build > + > # ############ > # Static build > # ############ > diff --git a/tools/update_repo.sh b/tools/update_repo.sh > new file mode 100755 > index 000000000..00e7aa0c6 > --- /dev/null > +++ b/tools/update_repo.sh > @@ -0,0 +1,571 @@ > +#!/bin/bash > +set -e > + > +rm_file='rm -f' > +rm_dir='rm -rf' > +mk_dir='mkdir -p' > +ws_prefix=/tmp/tarantool_repo_s3 > + > +alloss='ubuntu debian el fedora' > +product=tarantool > +force= > +# the path with binaries either repository > +repo=. > + > +# AWS defines > +aws="aws --endpoint-url ${AWS_S3_ENDPOINT_URL:-https://hb.bizmrg.com} s3" > +aws_cp_public="$aws cp --acl public-read" > +aws_sync_public="$aws sync --acl public-read" > + > +function get_os_dists { > + os=$1 > + alldists= > + > + if [ "$os" == "ubuntu" ]; then > + alldists='trusty xenial bionic cosmic disco eoan' > + elif [ "$os" == "debian" ]; then > + alldists='jessie stretch buster' > + elif [ "$os" == "el" ]; then > + alldists='6 7 8' > + elif [ "$os" == "fedora" ]; then > + alldists='27 28 29 30 31' > + fi > + > + echo "$alldists" > +} > + > +function prepare_ws { > + # temporary lock the publication to the repository > + ws_suffix=$1 > + ws=${ws_prefix}_${ws_suffix} > + ws_lockfile=${ws}.lock > + if [ -f $ws_lockfile ]; then > + old_proc=$(cat $ws_lockfile) > + fi > + lockfile -l 60 $ws_lockfile > + chmod u+w $ws_lockfile && echo $$ >$ws_lockfile && chmod u-w $ws_lockfile > + if [ "$old_proc" != "" -a "$old_proc" != "0" ]; then > + kill -9 $old_proc >/dev/null || true > + fi > + > + # create temporary workspace for the new files > + $rm_dir $ws > + $mk_dir $ws > +} > + > +function usage { > + cat < +Usage for store package binaries from the given path: > + $0 -o= -d= -b= [-p=] > + > +Usage for mirroring Debian|Ubuntu OS repositories: > + $0 -o= -d= -b= [-p=] > + > +Arguments: > + > + Path points to the directory with deb/prm packages to be used. > + > +Options: > + -b|--bucket > + MCS S3 bucket already existing which will be used for storing the packages > + -o|--os > + OS to be checked, one of the list: > + $alloss > + -d|--distribution > + Distribution appropriate to the given OS: > +EOF > + for os in $alloss ; do > + echo " $os: <"$(get_os_dists $os)">" > + done > + cat < + -p|--product > + Product name to be packed with, default name is 'tarantool' > + -f|--force > + Force updating the remote package with the local one despite the checksum difference > + -h|--help > + Usage help message > +EOF > +} > + > +for i in "$@" > +do > +case $i in > + -b=*|--bucket=*) > + bucket="${i#*=}" > + shift # past argument=value > + ;; > + -o=*|--os=*) > + os="${i#*=}" > + if ! echo $alloss | grep -F -q -w $os ; then > + echo "ERROR: OS '$os' is not supported" > + usage > + exit 1 > + fi > + shift # past argument=value > + ;; > + -d=*|--distribution=*) > + option_dist="${i#*=}" > + shift # past argument=value > + ;; > + -p=*|--product=*) > + product="${i#*=}" > + shift # past argument=value > + ;; > + -f|--force) > + force=1 > + ;; > + -h|--help) > + usage > + exit 0 > + ;; > + *) > + repo="${i#*=}" > + pushd $repo >/dev/null ; repo=$PWD ; popd >/dev/null > + shift # past argument=value > + ;; > +esac > +done > + > +# check that all needed options were set and correct > +if [ "$bucket" == "" ]; then > + echo "ERROR: need to set -b|--bucket bucket option, check usage" > + usage > + exit 1 > +fi > +if [ "$option_dist" == "" ]; then > + echo "ERROR: need to set -d|--option_dist OS distribuition name option, check usage" > + usage > + exit 1 > +fi > +if [ "$os" == "" ]; then > + echo "ERROR: need to set -o|--os OS name option, check usage" > + usage > + exit 1 > +fi > +alldists=$(get_os_dists $os) > +if [ -n "$option_dist" ] && ! echo $alldists | grep -F -q -w $option_dist ; then > + echo "ERROR: set distribution at options '$option_dist' not found at supported list '$alldists'" > + usage > + exit 1 > +fi > + > +# set the subpath with binaries based on literal character of the product name > +proddir=$(echo $product | head -c 1) > + > +# set bucket path of the given OS in options > +bucket_path="$bucket/$os" > + > +function update_deb_packfile { > + packfile=$1 > + packtype=$2 > + update_dist=$3 > + > + locpackfile=$(echo $packfile | sed "s#^$ws/##g") > + # register DEB/DSC pack file to Packages/Sources file > + reprepro -Vb . include$packtype $update_dist $packfile > + # reprepro copied DEB/DSC file to component which is not needed > + $rm_dir $debdir/$component > + # to have all sources avoid reprepro set DEB/DSC file to its own registry > + $rm_dir db > +} > + > +function update_deb_metadata { > + packpath=$1 > + packtype=$2 > + > + if [ ! -f $packpath.saved ] ; then > + # get the latest Sources file from S3 either create empty file > + $aws ls "$bucket_path/$packpath" >/dev/null 2>&1 && \ > + $aws cp "$bucket_path/$packpath" $packpath.saved || \ > + touch $packpath.saved > + fi > + > + if [ "$packtype" == "dsc" ]; then > + # WORKAROUND: unknown why, but reprepro doesn`t save the Sources > + # file, lets recreate it manualy from it's zipped version > + gunzip -c $packpath.gz >$packpath > + # check if the DSC hash already exists in old Sources file from S3 > + # find the hash from the new Sources file > + hash=$(grep '^Checksums-Sha256:' -A3 $packpath | \ > + tail -n 1 | awk '{print $1}') > + # search the new hash in the old Sources file from S3 > + if grep " $hash .* .*$" $packpath.saved ; then > + echo "WARNING: DSC file already registered in S3!" > + return > + fi > + # check if the DSC file already exists in old Sources file from S3 > + file=$(grep '^Files:' -A3 $packpath | tail -n 1 | awk '{print $3}') > + if [ "$force" == "" ] && grep " .* .* $file$" $packpath.saved ; then > + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file" > + echo "New hash: $hash" > + # unlock the publishing > + $rm_file $ws_lockfile > + exit 1 > + fi > + updated_dsc=1 > + elif [ "$packtype" == "deb" ]; then > + # check if the DEB file already exists in old Packages file from S3 > + # find the hash from the new Packages file > + hash=$(grep '^SHA256: ' $packpath) > + # search the new hash in the old Packages file from S3 > + if grep "^SHA256: $hash" $packpath.saved ; then > + echo "WARNING: DEB file already registered in S3!" > + return > + fi > + # check if the DEB file already exists in old Packages file from S3 > + file=$(grep '^Filename:' | awk '{print $2}') > + if [ "$force" == "" ] && grep "Filename: $file$" $packpath.saved ; then > + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file" > + echo "New hash: $hash" > + # unlock the publishing > + $rm_file $ws_lockfile > + exit 1 > + fi > + updated_deb=1 > + fi > + # store the new DEB entry > + cat $packpath >>$packpath.saved > +} > + > +# The 'pack_deb' function especialy created for DEB packages. It works > +# with DEB packing OS like Ubuntu, Debian. It is based on globaly known > +# tool 'reprepro' from: > +# https://wiki.debian.org/DebianRepository/SetupWithReprepro > +# This tool works with complete number of distributions of the given OS. > +# Result of the routine is the debian package for APT repository with > +# file structure equal to the Debian/Ubuntu: > +# http://ftp.am.debian.org/debian/pool/main/t/tarantool/ > +# http://ftp.am.debian.org/ubuntu/pool/main/t/ > +function pack_deb { > + # we need to push packages into 'main' repository only > + component=main > + > + # debian has special directory 'pool' for packages > + debdir=pool > + > + # get packages from pointed location > + if ! ls $repo/*.deb $repo/*.dsc $repo/*.tar.*z >/dev/null ; then > + echo "ERROR: files $repo/*.deb $repo/*.dsc $repo/*.tar.*z not found" > + usage > + exit 1 > + fi > + > + # prepare the workspace > + prepare_ws ${os} > + > + # copy single distribution with binaries packages > + repopath=$ws/pool/${option_dist}/$component/$proddir/$product > + $mk_dir ${repopath} > + cp $repo/*.deb $repo/*.dsc $repo/*.tar.*z $repopath/. > + pushd $ws > + > + # create the configuration file for 'reprepro' tool > + confpath=$ws/conf > + $rm_dir $confpath > + $mk_dir $confpath > + > + for loop_dist in $alldists ; do > + cat <>$confpath/distributions > +Origin: Tarantool > +Label: tarantool.org > +Suite: stable > +Codename: $loop_dist > +Architectures: amd64 source > +Components: $component > +Description: Tarantool DBMS and Tarantool modules > +SignWith: 91B625E5 > +DebIndices: Packages Release . .gz .bz2 > +UDebIndices: Packages . .gz .bz2 > +DscIndices: Sources Release .gz .bz2 > + > +EOF > + done > + > + # create standalone repository with separate components > + for loop_dist in $alldists ; do > + echo ================ DISTRIBUTION: $loop_dist ==================== > + updated_files=0 > + > + # 1(binaries). use reprepro tool to generate Packages file > + for deb in $ws/$debdir/$loop_dist/$component/*/*/*.deb ; do > + [ -f $deb ] || continue > + updated_deb=0 > + # regenerate DEB pack > + update_deb_packfile $deb deb $loop_dist > + echo "Regenerated DEB file: $locpackfile" > + for packages in dists/$loop_dist/$component/binary-*/Packages ; do > + # copy Packages file to avoid of removing by the new DEB version > + # update metadata 'Packages' files > + update_deb_metadata $packages deb > + [ "$updated_deb" == "1" ] || continue > + updated_files=1 > + done > + # save the registered DEB file to S3 > + if [ "$updated_deb" == 1 ]; then > + $aws_cp_public $deb $bucket_path/$locpackfile > + fi > + done > + > + # 1(sources). use reprepro tool to generate Sources file > + for dsc in $ws/$debdir/$loop_dist/$component/*/*/*.dsc ; do > + [ -f $dsc ] || continue > + updated_dsc=0 > + # regenerate DSC pack > + update_deb_packfile $dsc dsc $loop_dist > + echo "Regenerated DSC file: $locpackfile" > + # copy Sources file to avoid of removing by the new DSC version > + # update metadata 'Sources' file > + update_deb_metadata dists/$loop_dist/$component/source/Sources dsc > + [ "$updated_dsc" == "1" ] || continue > + updated_files=1 > + # save the registered DSC file to S3 > + $aws_cp_public $dsc $bucket_path/$locpackfile > + tarxz=$(echo $locpackfile | sed 's#\.dsc$#.debian.tar.xz#g') > + $aws_cp_public $ws/$tarxz "$bucket_path/$tarxz" > + orig=$(echo $locpackfile | sed 's#-1\.dsc$#.orig.tar.xz#g') > + $aws_cp_public $ws/$orig "$bucket_path/$orig" > + done > + > + # check if any DEB/DSC files were newly registered > + [ "$updated_files" == "0" ] && \ > + continue || echo "Updating dists" > + > + # finalize the Packages file > + for packages in dists/$loop_dist/$component/binary-*/Packages ; do > + mv $packages.saved $packages > + done > + > + # finalize the Sources file > + sources=dists/$loop_dist/$component/source/Sources > + mv $sources.saved $sources > + > + # 2(binaries). update Packages file archives > + for packpath in dists/$loop_dist/$component/binary-* ; do > + pushd $packpath > + sed "s#Filename: $debdir/$component/#Filename: $debdir/$loop_dist/$component/#g" -i Packages > + bzip2 -c Packages >Packages.bz2 > + gzip -c Packages >Packages.gz > + popd > + done > + > + # 2(sources). update Sources file archives > + pushd dists/$loop_dist/$component/source > + sed "s#Directory: $debdir/$component/#Directory: $debdir/$loop_dist/$component/#g" -i Sources > + bzip2 -c Sources >Sources.bz2 > + gzip -c Sources >Sources.gz > + popd > + > + # 3. update checksums entries of the Packages* files in *Release files > + # NOTE: it is stable structure of the *Release files when the checksum > + # entries in it in the following way: > + # MD5Sum: > + # > + # > + # SHA1: > + # > + # > + # SHA256: > + # > + # > + # The script bellow puts 'md5' value at the 1st found file entry, > + # 'sha1' - at the 2nd and 'sha256' at the 3rd > + pushd dists/$loop_dist > + for file in $(grep " $component/" Release | awk '{print $3}' | sort -u) ; do > + sz=$(stat -c "%s" $file) > + md5=$(md5sum $file | awk '{print $1}') > + sha1=$(sha1sum $file | awk '{print $1}') > + sha256=$(sha256sum $file | awk '{print $1}') > + awk 'BEGIN{c = 0} ; { > + if ($3 == p) { > + c = c + 1 > + if (c == 1) {print " " md " " s " " p} > + if (c == 2) {print " " sh1 " " s " " p} > + if (c == 3) {print " " sh2 " " s " " p} > + } else {print $0} > + }' p="$file" s="$sz" md="$md5" sh1="$sha1" sh2="$sha256" \ > + Release >Release.new > + mv Release.new Release > + done > + # resign the selfsigned InRelease file > + $rm_file InRelease > + gpg --clearsign -o InRelease Release > + # resign the Release file > + $rm_file Release.gpg > + gpg -abs -o Release.gpg Release > + popd > + > + # 4. sync the latest distribution path changes to S3 > + $aws_sync_public dists/$loop_dist "$bucket_path/dists/$loop_dist" > + done > + > + # unlock the publishing > + $rm_file $ws_lockfile > + > + popd > +} > + > +# The 'pack_rpm' function especialy created for RPM packages. It works > +# with RPM packing OS like Centos, Fedora. It is based on globaly known > +# tool 'createrepo' from: > +# https://linux.die.net/man/8/createrepo > +# This tool works with single distribution of the given OS. > +# Result of the routine is the rpm package for YUM repository with > +# file structure equal to the Centos/Fedora: > +# http://mirror.centos.org/centos/7/os/x86_64/Packages/ > +# http://mirrors.kernel.org/fedora/releases/30/Everything/x86_64/os/Packages/t/ > +function pack_rpm { > + if ! ls $repo/*.rpm >/dev/null ; then > + echo "ERROR: Current '$repo' path doesn't have RPM packages in path" > + usage > + exit 1 > + fi > + > + # prepare the workspace > + prepare_ws ${os}_${option_dist} > + > + # copy the needed package binaries to the workspace > + cp $repo/*.rpm $ws/. > + > + pushd $ws > + > + # set the paths > + if [ "$os" == "el" ]; then > + repopath=$option_dist/os/x86_64 > + rpmpath=Packages > + elif [ "$os" == "fedora" ]; then > + repopath=releases/$option_dist/Everything/x86_64/os > + rpmpath=Packages/$proddir > + fi > + packpath=$repopath/$rpmpath > + > + # prepare local repository with packages > + $mk_dir $packpath > + mv *.rpm $packpath/. > + cd $repopath > + > + # copy the current metadata files from S3 > + mkdir repodata.base > + for file in $($aws ls $bucket_path/$repopath/repodata/ | awk '{print $NF}') ; do > + $aws ls $bucket_path/$repopath/repodata/$file || continue > + $aws cp $bucket_path/$repopath/repodata/$file repodata.base/$file > + done > + > + # create the new repository metadata files > + createrepo --no-database --update --workers=2 \ > + --compress-type=gz --simple-md-filenames . > + > + updated_rpms=0 > + # loop by the new hashes from the new meta file > + for hash in $(zcat repodata/other.xml.gz | grep " + updated_rpm=0 > + name=$(zcat repodata/other.xml.gz | grep " + awk -F'"' '{print $4}') > + # search the new hash in the old meta file from S3 > + if zcat repodata.base/filelists.xml.gz | grep "pkgid=\"$hash\"" | \ > + grep "name=\"$name\"" ; then > + echo "WARNING: $name file already registered in S3!" > + echo "File hash: $hash" > + continue > + fi > + updated_rpms=1 > + # check if the hashed file already exists in old meta file from S3 > + file=$(zcat repodata/primary.xml.gz | \ > + grep -e " + # check if the file already exists in S3 > + if [ "$force" == "" ] && zcat repodata.base/primary.xml.gz | \ > + grep " + echo "ERROR: the file already exists, but changed, set '-f' to overwrite it: $file" > + echo "New hash: $hash" > + # unlock the publishing > + $rm_file $ws_lockfile > + exit 1 > + fi > + done > + > + # check if any RPM files were newly registered > + [ "$updated_rpms" == "0" ] && \ > + return || echo "Updating dists" > + > + # move the repodata files to the standalone location > + mv repodata repodata.adding > + > + # merge metadata files > + mkdir repodata > + head -n 2 repodata.adding/repomd.xml >repodata/repomd.xml > + for file in filelists.xml other.xml primary.xml ; do > + # 1. take the 1st line only - to skip the line with > + # number of packages which is not needed > + zcat repodata.adding/$file.gz | head -n 1 >repodata/$file > + # 2. take 2nd line with metadata tag and update > + # the packages number in it > + packsold=0 > + if [ -f repodata.base/$file.gz ] ; then > + packsold=$(zcat repodata.base/$file.gz | head -n 2 | \ > + tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g') > + fi > + packsnew=$(zcat repodata.adding/$file.gz | head -n 2 | \ > + tail -n 1 | sed 's#.*packages="\(.*\)".*#\1#g') > + packs=$(($packsold+$packsnew)) > + zcat repodata.adding/$file.gz | head -n 2 | tail -n 1 | \ > + sed "s#packages=\".*\"#packages=\"$packs\"#g" >>repodata/$file > + # 3. take only 'package' tags from new file > + zcat repodata.adding/$file.gz | tail -n +3 | head -n -1 \ > + >>repodata/$file > + # 4. take only 'package' tags from old file if exists > + if [ -f repodata.base/$file.gz ] ; then > + zcat repodata.base/$file.gz | tail -n +3 | head -n -1 \ > + >>repodata/$file > + fi > + # 5. take the last closing line with metadata tag > + zcat repodata.adding/$file.gz | tail -n 1 >>repodata/$file > + > + # get the new data > + chsnew=$(sha256sum repodata/$file | awk '{print $1}') > + sz=$(stat --printf="%s" repodata/$file) > + gzip repodata/$file > + chsgznew=$(sha256sum repodata/$file.gz | awk '{print $1}') > + szgz=$(stat --printf="%s" repodata/$file.gz) > + timestamp=$(date +%s -r repodata/$file.gz) > + > + # add info to repomd.xml file > + name=$(echo $file | sed 's#\.xml$##g') > + cat <>repodata/repomd.xml > + > + $chsgznew > + $chsnew > + > + $timestamp > + $szgz > + $sz > +" > +EOF > + done > + tail -n 1 repodata.adding/repomd.xml >>repodata/repomd.xml > + gpg --detach-sign --armor repodata/repomd.xml > + > + # copy the packages to S3 > + for file in $rpmpath/*.rpm ; do > + $aws_cp_public $file "$bucket_path/$repopath/$file" > + done > + > + # update the metadata at the S3 > + $aws_sync_public repodata "$bucket_path/$repopath/repodata" > + > + # unlock the publishing > + $rm_file $ws_lockfile > + > + popd > +} > + > +if [ "$os" == "ubuntu" -o "$os" == "debian" ]; then > + pack_deb > +elif [ "$os" == "el" -o "$os" == "fedora" ]; then > + pack_rpm > +else > + echo "USAGE: given OS '$os' is not supported, use any single from the list: $alloss" > + usage > + exit 1 > +fi > -- > 2.17.1 -- Konstantin Osipov, Moscow, Russia