[Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

Sun Jan 19 02:26:27 MSK 2020

Sasha,

Thanks for the discussion results you provided! Some light can be seen
at the end of the tunnel.

On 14.01.20, Alexander Turenko wrote:
> I discussed what we're expect from rpm/deb repositories with Igor,
> Alexander T., Kirill Yu. I'll summarize key points below.
> 
> CCed Konstantin N., Yaroslav D. and Maxim M.
> 
> ## Planned activities inside tarantool repository
> 
> * We have packagecloud repositories and we're going to keep them and
>   continue update them (at least for all versions below 2.4 inclusive).

Agree.

> * We'll keep redirects from download.tarantool.org to packagecloud for
>   those 1.10/2.1/2.2/2.3/2.4 repositories.

It's a tough one with both its pros and cons.
* Leaving redirects for old repos to packagecloud makes us to have
  different publishing pipelines and complicates the further
  mainteinance.
* Changing redirects to the new storage obligues users to change the
  repo key for already enabled repos.

> * We'll create a new repository (working name is 'nightly'), which will
>   contain packages from all long-term branches (1.10/2.1/.../master).
>   Package names will be tarantool110, tarantool21, tarantool22,
>   tarantool23, tarantool24.

Since the packages listed above provides the binary and all related
entities having the same path this approach leads to conflicts while
messing around with two different major versions (e.g. tarantool110 and
tarantool23) on one system. Therefore the one can't smoothly
upgrade/migrate his instance to a new version.

According to this fact I propose to make it the way similar to MySQL and
PostgreSQL do: both of them split their packages into separate repos.

### PostgreSQL

PostgreSQL installation flow is described here[1]. E.g. for Centos 7 the
maintainers provide a package with the relevant pg repos:
| $ yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
| <snipped>
| Complete!

Here you can see the package ships *all* relevant PostgreSQL repos:
| $ yum repolist enabled | grep pgdg
| pgdg10/7/x86_64       PostgreSQL 10 for RHEL/CentOS 7 - x86_64    1183
| pgdg11/7/x86_64       PostgreSQL 11 for RHEL/CentOS 7 - x86_64    1020
| pgdg12/7/x86_64       PostgreSQL 12 for RHEL/CentOS 7 - x86_64     358
| pgdg94/7/x86_64       PostgreSQL 9.4 for RHEL/CentOS 7 - x86_64   1092
| pgdg95/7/x86_64       PostgreSQL 9.5 for RHEL/CentOS 7 - x86_64   1158
| pgdg96/7/x86_64       PostgreSQL 9.6 for RHEL/CentOS 7 - x86_64   1203

There are no problems obtaining different versions on one system:
| $ yum install -y postgresql10 postgresql12
| <snipped>
| Complete!

Furthermore every PostgreSQL version is available from its own repo:
| $ yum info postgresql10 | grep -P '^From repo'
| From repo   : pgdg10
| $ yum info postgresql12 | grep -P '^From repo'
| From repo   : pgdg12

### MySQL

MySQL installation flow is described here[2]. E.g. for Centos 7 the
maintainers provide a package with the relevant mysql repos:
| $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
| <snipped>
| Complete!

Similar to the PostgreSQL one the package ships *all* relevant MySQL
repos but the old ones are disabled (the reader can manualy check the
corresponding files in /etc/yum.repos.d directory):
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql80-community/x86_64             MySQL 8.0 Community Server    161

Whether we try to install an old one package with repos alongside we'll
see the following:
| $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
| <snipped>
| Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch

Thereby to see MySQL upgrading process let's manually remove the first
installed package and try to install the latter one again:
| $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm
| <snipped>
| Complete!

Again, the package delivers the relevant MySQL repos and the reader can
manually check that all repos except the mysql57 related are disabled:
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql57-community/x86_64             MySQL 5.7 Community Server    404

As a result of the mysql-community-server installation we'll have
MySQL 5.7 software installed on the system:
| $ yum install -y mysql-community-server
| <snipped>
| Complete!
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 5.7.29
| Release     : 1.el7
| Size        : 765 M
| Repo        : installed
| From repo   : mysql57-community
| <snipped>

The layout of the package doesn't have any specific paths regarding
MySQL version:
| $ rpm -ql mysql-community-server | grep -P  mysqld$
| /usr/sbin/mysqld
| /var/run/mysqld

Attempt to upgrade the repo package fails the same way as above:
| $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
| <snipped>
| Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch

When the installation finishes we can see that only the repos related to
MySQL 8.0 are enabled:
| $ yum repolist enabled | grep mysql
| mysql-connectors-community/x86_64    MySQL Connectors Community    141
| mysql-tools-community/x86_64         MySQL Tools Community         105
| mysql80-community/x86_64             MySQL 8.0 Community Server    161

However, the installed mysql-community-server is still 5.7 and 8.0
package is available now:
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 5.7.29
| Release     : 1.el7
| Size        : 765 M
| Repo        : installed
| From repo   : mysql57-community
| <snipped>
| Available Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 8.0.19
| Release     : 1.el7
| Size        : 436 M
| Repo        : mysql80-community/x86_64
| <snipped>

There are no problems with the upgrading process:
| $ yum install -y mysql-community-server
| <snipped>
| Complete!

As a result there is only a single version installed and available:
| $ yum info mysql-community-server
| <snipped>
| Installed Packages
| Name        : mysql-community-server
| Arch        : x86_64
| Version     : 8.0.19
| Release     : 1.el7
| Size        : 1.9 G
| Repo        : installed
| From repo   : mysql80-community
| <snipped>

I see this way to be the most convenient approach both for maintainers
and users: a single package to be splited per versions to separated
repos. AFAICS we already follow this way, so I propose to stay with this
layout further.

>   - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
>     Should <sepX> be the same as <sepY>? Consider that it should work
>     smooothly with apt and yum repos both. (AR Alexander Tikh.)
> * In addition to tarantool110 and so we'll deploy a package with name
>   tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
>   development.

I guess we can recommend the one to use our latest repo for the bleeding
tarantool considering the name is left unchanged.

> 
> ## Necessary activities outside of tarantool repository
> 
> * Manually deploy existing packages:
>   - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
>     scheme. Maybe also all tagged versions. The rebuild should not
>     change `git describe --tags --long` output (including a commit
>     hash).

Not necessary since the naming left unchanged.

>   - Copy all module / connector packages from packagecloud.
>     - TBD: Propose exact list (AR Alexander Tikh.)

Again refer to MySQL repos layout: e.g. server, tools and connectors are
split into separate repos.

> * Setup proper redirects from download.tarantool.org for the new
>   repository.

Agree.

> * Update CI / CD for modules (at least for most used ones) to use this
>   script: download it from the tarantool repository and use to deploy.

Agree.

> 
> After that we can announce the new repository.
> 
> ## Further activities
> 
> Announce the new repository and deprecation plan of old ones (planned to
> fully support the old repos for 2.4 and below, but don't add 2.5 and
> further repos).

Agree.

> 
> Update download instructions on the website (see [1]) to use the new
> repository.

Agree though adjusted yours a bit:
| Update download instructions on the website (see [3]) to use new
| repositories and provide packages for manipulations with them (similar
| to the one shipped by MySQL maintainers).

> 
> Provide the similar repository, but only for tagged builds (working name
> is 'release'). It seems it worth to push only tagged modules /
> connectors as well. (Asked by Konstantin N.) 

Agree.

> 
> We maybe (if there will be a demand) will move the script outside of the
> repository and integrate with the existing server [2] that doing quite
> similar job: it receives built rocks (it is like python's wheels) and
> updates a repository for luarocks / tarantoolctl rocks. This can be done
> w/o user-visible changes. (Proposed by Konstantin N.)

It's a good idea, though I'm totally OK with this small script within
our repo. Generally, I'm for moving all CI/CD related parts to a
separate one, but this is definitely not related to the subj.

> 
> Aren't I forget anything?

If everyone are OK with my proposals, I'll update the key points and
corresponding ARs.

> 
> [1]: https://www.tarantool.io/en/download/
> [2]: https://github.com/tarantool/rocks.tarantool.org
> 
> ----
> 
> Re patch itself: I don't look into the script and it seems it would be
> not productive. Just noted several points around, see below.
> 
> WBR, Alexander Turenko.
> 

[1]: https://www.postgresql.org/download/linux/
[2]: https://dev.mysql.com/downloads/
[3]: https://www.tarantool.io/en/download/

-- 
Best regards,
IM