From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtpng3.m.smailru.net (smtpng3.m.smailru.net [94.100.177.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 516E146970E for ; Sun, 19 Jan 2020 02:28:44 +0300 (MSK) Date: Sun, 19 Jan 2020 02:26:27 +0300 From: Igor Munkin Message-ID: <20200118232627.GO31304@tarantool.org> References: <20200114174229.vnf6ubpgw265arsb@tkn_work_nb> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20200114174229.vnf6ubpgw265arsb@tkn_work_nb> Subject: Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3 List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexander Turenko Cc: Yaroslav Dynnikov , tarantool-patches@dev.tarantool.org, Konstantin Nazarov Sasha, Thanks for the discussion results you provided! Some light can be seen at the end of the tunnel. On 14.01.20, Alexander Turenko wrote: > I discussed what we're expect from rpm/deb repositories with Igor, > Alexander T., Kirill Yu. I'll summarize key points below. > > CCed Konstantin N., Yaroslav D. and Maxim M. > > ## Planned activities inside tarantool repository > > * We have packagecloud repositories and we're going to keep them and > continue update them (at least for all versions below 2.4 inclusive). Agree. > * We'll keep redirects from download.tarantool.org to packagecloud for > those 1.10/2.1/2.2/2.3/2.4 repositories. It's a tough one with both its pros and cons. * Leaving redirects for old repos to packagecloud makes us to have different publishing pipelines and complicates the further mainteinance. * Changing redirects to the new storage obligues users to change the repo key for already enabled repos. > * We'll create a new repository (working name is 'nightly'), which will > contain packages from all long-term branches (1.10/2.1/.../master). > Package names will be tarantool110, tarantool21, tarantool22, > tarantool23, tarantool24. Since the packages listed above provides the binary and all related entities having the same path this approach leads to conflicts while messing around with two different major versions (e.g. tarantool110 and tarantool23) on one system. Therefore the one can't smoothly upgrade/migrate his instance to a new version. According to this fact I propose to make it the way similar to MySQL and PostgreSQL do: both of them split their packages into separate repos. ### PostgreSQL PostgreSQL installation flow is described here[1]. E.g. for Centos 7 the maintainers provide a package with the relevant pg repos: | $ yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm | | Complete! Here you can see the package ships *all* relevant PostgreSQL repos: | $ yum repolist enabled | grep pgdg | pgdg10/7/x86_64 PostgreSQL 10 for RHEL/CentOS 7 - x86_64 1183 | pgdg11/7/x86_64 PostgreSQL 11 for RHEL/CentOS 7 - x86_64 1020 | pgdg12/7/x86_64 PostgreSQL 12 for RHEL/CentOS 7 - x86_64 358 | pgdg94/7/x86_64 PostgreSQL 9.4 for RHEL/CentOS 7 - x86_64 1092 | pgdg95/7/x86_64 PostgreSQL 9.5 for RHEL/CentOS 7 - x86_64 1158 | pgdg96/7/x86_64 PostgreSQL 9.6 for RHEL/CentOS 7 - x86_64 1203 There are no problems obtaining different versions on one system: | $ yum install -y postgresql10 postgresql12 | | Complete! Each package lay out its contents the way similar to Gentoo slots approach (i.e. the paths contains parts related to the package version): | $ rpm -ql postgresql10 | grep -P 'psql$' | /usr/pgsql-10/bin/psql | $ rpm -ql postgresql12 | grep -P 'psql$' | /usr/pgsql-12/bin/psql Furthermore every PostgreSQL version is available from its own repo: | $ yum info postgresql10 | grep -P '^From repo' | From repo : pgdg10 | $ yum info postgresql12 | grep -P '^From repo' | From repo : pgdg12 ### MySQL MySQL installation flow is described here[2]. E.g. for Centos 7 the maintainers provide a package with the relevant mysql repos: | $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm | | Complete! Similar to the PostgreSQL one the package ships *all* relevant MySQL repos but the old ones are disabled (the reader can manualy check the corresponding files in /etc/yum.repos.d directory): | $ yum repolist enabled | grep mysql | mysql-connectors-community/x86_64 MySQL Connectors Community 141 | mysql-tools-community/x86_64 MySQL Tools Community 105 | mysql80-community/x86_64 MySQL 8.0 Community Server 161 Whether we try to install an old one package with repos alongside we'll see the following: | $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm | | Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch Thereby to see MySQL upgrading process let's manually remove the first installed package and try to install the latter one again: | $ yum install https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm | | Complete! Again, the package delivers the relevant MySQL repos and the reader can manually check that all repos except the mysql57 related are disabled: | $ yum repolist enabled | grep mysql | mysql-connectors-community/x86_64 MySQL Connectors Community 141 | mysql-tools-community/x86_64 MySQL Tools Community 105 | mysql57-community/x86_64 MySQL 5.7 Community Server 404 As a result of the mysql-community-server installation we'll have MySQL 5.7 software installed on the system: | $ yum install -y mysql-community-server | | Complete! | $ yum info mysql-community-server | | Installed Packages | Name : mysql-community-server | Arch : x86_64 | Version : 5.7.29 | Release : 1.el7 | Size : 765 M | Repo : installed | From repo : mysql57-community | The layout of the package doesn't have any specific paths regarding MySQL version: | $ rpm -ql mysql-community-server | grep -P mysqld$ | /usr/sbin/mysqld | /var/run/mysqld Attempt to upgrade the repo package fails the same way as above: | $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm | | Error: mysql80-community-release conflicts with mysql57-community-release-el7-9.noarch If we remove the mysql57 package the installation of the new one succeeds: | $ yum remove mysql57-community-release | | Complete! | $ yum install https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm | | Complete! When the installation finishes we can see that only the repos related to MySQL 8.0 are enabled: | $ yum repolist enabled | grep mysql | mysql-connectors-community/x86_64 MySQL Connectors Community 141 | mysql-tools-community/x86_64 MySQL Tools Community 105 | mysql80-community/x86_64 MySQL 8.0 Community Server 161 However, the installed mysql-community-server is still 5.7 and 8.0 package is available now: | $ yum info mysql-community-server | | Installed Packages | Name : mysql-community-server | Arch : x86_64 | Version : 5.7.29 | Release : 1.el7 | Size : 765 M | Repo : installed | From repo : mysql57-community | | Available Packages | Name : mysql-community-server | Arch : x86_64 | Version : 8.0.19 | Release : 1.el7 | Size : 436 M | Repo : mysql80-community/x86_64 | There are no problems with the upgrading process: | $ yum install -y mysql-community-server | | Complete! As a result there is only a single version installed and available: | $ yum info mysql-community-server | | Installed Packages | Name : mysql-community-server | Arch : x86_64 | Version : 8.0.19 | Release : 1.el7 | Size : 1.9 G | Repo : installed | From repo : mysql80-community | I see this way to be the most convenient approach both for maintainers and users: a single package to be splited per versions to separated repos. AFAICS we already follow this way, so I propose to stay with this layout further. > - TBD: Can we add some separator: tarantool110? Should we? > Should be the same as ? Consider that it should work > smooothly with apt and yum repos both. (AR Alexander Tikh.) > * In addition to tarantool110 and so we'll deploy a package with name > tarantool (or, maybe, tarantool-live?) from master as bleeding edge of > development. I guess we can recommend the one to use our latest repo for the bleeding tarantool considering the name is left unchanged. > > ## Necessary activities outside of tarantool repository > > * Manually deploy existing packages: > - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming > scheme. Maybe also all tagged versions. The rebuild should not > change `git describe --tags --long` output (including a commit > hash). Not necessary since the naming left unchanged. > - Copy all module / connector packages from packagecloud. > - TBD: Propose exact list (AR Alexander Tikh.) Again refer to MySQL repos layout: e.g. server, tools and connectors are split into separate repos. > * Setup proper redirects from download.tarantool.org for the new > repository. Agree. > * Update CI / CD for modules (at least for most used ones) to use this > script: download it from the tarantool repository and use to deploy. Agree. > > After that we can announce the new repository. > > ## Further activities > > Announce the new repository and deprecation plan of old ones (planned to > fully support the old repos for 2.4 and below, but don't add 2.5 and > further repos). Agree. > > Update download instructions on the website (see [1]) to use the new > repository. Agree though adjusted yours a bit: | Update download instructions on the website (see [3]) to use new | repositories and provide packages for manipulations with them (similar | to the one shipped by MySQL maintainers). > > Provide the similar repository, but only for tagged builds (working name > is 'release'). It seems it worth to push only tagged modules / > connectors as well. (Asked by Konstantin N.) Agree. > > We maybe (if there will be a demand) will move the script outside of the > repository and integrate with the existing server [2] that doing quite > similar job: it receives built rocks (it is like python's wheels) and > updates a repository for luarocks / tarantoolctl rocks. This can be done > w/o user-visible changes. (Proposed by Konstantin N.) It's a good idea, though I'm totally OK with this small script within our repo. Generally, I'm for moving all CI/CD related parts to a separate one, but this is definitely not related to the subj. > > Aren't I forget anything? If everyone are OK with my proposals, I'll update the key points and corresponding ARs. > > [1]: https://www.tarantool.io/en/download/ > [2]: https://github.com/tarantool/rocks.tarantool.org > > ---- > > Re patch itself: I don't look into the script and it seems it would be > not productive. Just noted several points around, see below. > > WBR, Alexander Turenko. > [1]: https://www.postgresql.org/download/linux/ [2]: https://dev.mysql.com/downloads/ [3]: https://www.tarantool.io/en/download/ -- Best regards, IM