From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp61.i.mail.ru (smtp61.i.mail.ru [217.69.128.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id F378E46970E for ; Sat, 1 Feb 2020 01:42:24 +0300 (MSK) Date: Sat, 1 Feb 2020 01:42:34 +0300 From: Alexander Turenko Message-ID: <20200131224234.sfvv6g3if7vsjeda@tkn_work_nb> References: <3840321b3c8368d4da86951c0a3e3bcfb3ded063.1580450098.git.avtikhon@tarantool.org> <1580477321.270149775@f416.i.mail.ru> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1580477321.270149775@f416.i.mail.ru> Subject: Re: [Tarantool-patches] [PATCH v8] gitlab-ci: implement packing into MCS S3 List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Oleg Piskunov Cc: tarantool-patches@dev.tarantool.org On Fri, Jan 31, 2020 at 04:28:41PM +0300, Oleg Piskunov wrote: > > Patch introduce preparation for changing packages deploying process: > 1. Instead of PackageCloud storage will be used S3 compatible storage.  > The main reason for such changes is that PackageCloud repository is > limited with space and not enough for keeping all packages. > 2. Introduce two new root repositories: > * "live" for packages from master and release branches. > * "release" for packages from tagged commits > 3. Developer branches with "-full-ci" building packages but don’t deploy. >   > For these purposes the standalone script for pushing DEB and RPM  > packages to the new S3 storage is used. For creating RPM packages  > it uses 'createrepo' utility and for DEB packages - 'reprepro' utility. >   > The script implements the following flow: > * creates new metafiles for the new packages > * fetches relevant metafiles from the storage > * copies new packages to S3 storage > * merges and pushes the updated metadata to S3 storage Oleg, thanks for the new description! I looked over it, tweaked here and there to adjust style or give better explanation and ends with the variant below the email. Alexander, please, consider both update your one. WBR, Alexander Turenko. ---- gitlab-ci: push Deb/RPM packages to S3 based repos We're going to use S3 compatible storage for Deb and RPM repositories instead of packagecloud.io service. The main reason is that packagecloud.io provides a limited amount of storage, which is not enough for keeping all packages (w/o regular pruning of old versions). Note: At the moment packages are still pushed to packagecloud.io from Travis-CI. Disabling this is out of scope of this patch. This patch implements saving of packages on an S3 compatible storage and regeneration of a repository metadata. The layout is a bit different from one we have on packagecloud.io. packagecloud.io: | - 1.10 | - 2.1 | - 2.2 | - ... S3 compatible storage: | - live | - 1.10 | - 2.1 | - 2.2 | - ... | - release | - 1.10 | - 2.1 | - 2.2 | - ... Both 'live' and 'release' repositories track release branches (named as .) and master branch. The difference is that 'live' is updated on every push, but 'release' is only for tagged versions (...0). Packages are also built on '*-full-ci' branches, but only for testing purposes: they don't pushed anywhere. The core logic is in the tools/update_repo.sh script, which implements the following flow: - create metadata for new packages - fetch relevant metadata from the S3 storage - push new packages to the S3 storage - merge and push the updated metadata to the S3 storage The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb repositories. Closes #3380