From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <alexander.turenko@tarantool.org>
Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by dev.tarantool.org (Postfix) with ESMTPS id A5E1C46970E
 for <tarantool-patches@dev.tarantool.org>;
 Tue,  4 Feb 2020 10:51:53 +0300 (MSK)
Date: Tue, 4 Feb 2020 10:52:05 +0300
From: Alexander Turenko <alexander.turenko@tarantool.org>
Message-ID: <20200204075204.fc7hvu6gi7qco37m@tkn_work_nb>
References: <6053f76b5c8fd80df91392f7a02b61c47006e341.1580793835.git.avtikhon@tarantool.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <6053f76b5c8fd80df91392f7a02b61c47006e341.1580793835.git.avtikhon@tarantool.org>
Subject: Re: [Tarantool-patches] [PATCH v10] gitlab-ci: push Deb/RPM
	packages to S3 based repos
List-Id: Tarantool development patches <tarantool-patches.dev.tarantool.org>
List-Unsubscribe: <https://lists.tarantool.org/mailman/options/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=unsubscribe>
List-Archive: <https://lists.tarantool.org/pipermail/tarantool-patches/>
List-Post: <mailto:tarantool-patches@dev.tarantool.org>
List-Help: <mailto:tarantool-patches-request@dev.tarantool.org?subject=help>
List-Subscribe: <https://lists.tarantool.org/mailman/listinfo/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=subscribe>
To: "Alexander V. Tikhonov" <avtikhon@tarantool.org>
Cc: Oleg Piskunov <o.piskunov@tarantool.org>, tarantool-patches@dev.tarantool.org

Pushed to master, 2.3, 2.2, 2.1 (w/o Fedora 31), 1.10.

CCed Kirill.

I'll file an issue for follow up activities.

WBR, Alexander Turenko.

On Tue, Feb 04, 2020 at 08:37:50AM +0300, Alexander V. Tikhonov wrote:
> We're going to use S3 compatible storage for Deb and RPM repositories
> instead of packagecloud.io service. The main reason is that
> packagecloud.io provides a limited amount of storage, which is not
> enough for keeping all packages (w/o regular pruning of old versions).
> 
> Note: At the moment packages are still pushed to packagecloud.io from
> Travis-CI. Disabling this is out of scope of this patch.
> 
> This patch implements saving of packages on an S3 compatible storage and
> regeneration of a repository metadata.
> 
> The layout is a bit different from one we have on packagecloud.io.
> 
> packagecloud.io:
> 
>  | - 1.10
>  | - 2.1
>  | - 2.2
>  | - ...
> 
> S3 compatible storage:
> 
>  | - live
>  |   - 1.10
>  |   - 2.1
>  |   - 2.2
>  |   - ...
>  | - release
>  |   - 1.10
>  |   - 2.1
>  |   - 2.2
>  |   - ...
> 
> Both 'live' and 'release' repositories track release branches (named as
> <major>.<minor>) and master branch. The difference is that 'live' is
> updated on every push, but 'release' is only for tagged versions
> (<major>.<minor>.<patch>.0).
> 
> Packages are also built on '*-full-ci' branches, but only for testing
> purposes: they don't pushed anywhere.
> 
> The core logic is in the tools/update_repo.sh script, which implements
> the following flow:
> 
> - create metadata for new packages
> - fetch relevant metadata from the S3 storage
> - push new packages to the S3 storage
> - merge and push the updated metadata to the S3 storage
> 
> The script uses 'createrepo' for RPM repositories and 'reprepro' for Deb
> repositories.
> 
> Closes #3380
> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-3380-push-packages-s3-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/3380