Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3

[Alexander Turenko <alexander.turenko@tarantool.org>]

On Fri, Jan 31, 2020 at 04:28:41PM +0300, Oleg Piskunov wrote:
>
> Patch introduce preparation for changing packages deploying process:
> 1. Instead of PackageCloud storage will be used S3 compatible storage. 
> The main reason for such changes is that PackageCloud repository is
> limited with space and not enough for keeping all packages.
> 2. Introduce two new root repositories:
> * "live" for packages from master and release branches.
> * "release" for packages from tagged commits
> 3. Developer branches with "-full-ci" building packages but don’t deploy.
>  
> For these purposes the standalone script for pushing DEB and RPM 
> packages to the new S3 storage is used. For creating RPM packages 
> it uses 'createrepo' utility and for DEB packages - 'reprepro' utility.
>  
> The script implements the following flow:
> * creates new metafiles for the new packages
> * fetches relevant metafiles from the storage
> * copies new packages to S3 storage
> * merges and pushes the updated metadata to S3 storage

Oleg, thanks for the new description!

I looked over it, tweaked here and there to adjust style or give better
explanation and ends with the variant below the email.

Alexander, please, consider both update your one.

WBR, Alexander Turenko.

----

gitlab-ci: push Deb/RPM packages to S3 based repos

We're going to use S3 compatible storage for Deb and RPM repositories
instead of packagecloud.io service. The main reason is that
packagecloud.io provides a limited amount of storage, which is not
enough for keeping all packages (w/o regular pruning of old versions).

Note: At the moment packages are still pushed to packagecloud.io from
Travis-CI. Disabling this is out of scope of this patch.

This patch implements saving of packages on an S3 compatible storage and
regeneration of a repository metadata.

The layout is a bit different from one we have on packagecloud.io.

packagecloud.io:

 | - 1.10
 | - 2.1
 | - 2.2
 | - ...

S3 compatible storage:

 | - live
 |   - 1.10
 |   - 2.1
 |   - 2.2
 |   - ...
 | - release
 |   - 1.10
 |   - 2.1
 |   - 2.2
 |   - ...

Both 'live' and 'release' repositories track release branches (named as
<major>.<minor>) and master branch. The difference is that 'live' is
updated on every push, but 'release' is only for tagged versions
(<major>.<minor>.<patch>.0).

Packages are also built on '*-full-ci' branches, but only for testing
purposes: they don't pushed anywhere.

The core logic is in the tools/update_repo.sh script, which implements
the following flow:

- create metadata for new packages
- fetch relevant metadata from the S3 storage
- push new packages to the S3 storage
- merge and push the updated metadata to the S3 storage

The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
repositories.

Closes #3380