Re: [Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3

[Alexander Turenko <alexander.turenko@tarantool.org>]

I discussed what we're expect from rpm/deb repositories with Igor,
Alexander T., Kirill Yu. I'll summarize key points below.

CCed Konstantin N., Yaroslav D. and Maxim M.

## Planned activities inside tarantool repository

* We have packagecloud repositories and we're going to keep them and
  continue update them (at least for all versions below 2.4 inclusive).
* We'll keep redirects from download.tarantool.org to packagecloud for
  those 1.10/2.1/2.2/2.3/2.4 repositories.
* We'll create a new repository (working name is 'nightly'), which will
  contain packages from all long-term branches (1.10/2.1/.../master).
  Package names will be tarantool110, tarantool21, tarantool22,
  tarantool23, tarantool24.
  - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
    Should <sepX> be the same as <sepY>? Consider that it should work
    smooothly with apt and yum repos both. (AR Alexander Tikh.)
* In addition to tarantool110 and so we'll deploy a package with name
  tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
  development.

## Necessary activities outside of tarantool repository

* Manually deploy existing packages:
  - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
    scheme. Maybe also all tagged versions. The rebuild should not
    change `git describe --tags --long` output (including a commit
    hash).
  - Copy all module / connector packages from packagecloud.
    - TBD: Propose exact list (AR Alexander Tikh.)
* Setup proper redirects from download.tarantool.org for the new
  repository.
* Update CI / CD for modules (at least for most used ones) to use this
  script: download it from the tarantool repository and use to deploy.

After that we can announce the new repository.

## Further activities

Announce the new repository and deprecation plan of old ones (planned to
fully support the old repos for 2.4 and below, but don't add 2.5 and
further repos).

Update download instructions on the website (see [1]) to use the new
repository.

Provide the similar repository, but only for tagged builds (working name
is 'release'). It seems it worth to push only tagged modules /
connectors as well. (Asked by Konstantin N.) 

We maybe (if there will be a demand) will move the script outside of the
repository and integrate with the existing server [2] that doing quite
similar job: it receives built rocks (it is like python's wheels) and
updates a repository for luarocks / tarantoolctl rocks. This can be done
w/o user-visible changes. (Proposed by Konstantin N.)

Aren't I forget anything?

[1]: https://www.tarantool.io/en/download/
[2]: https://github.com/tarantool/rocks.tarantool.org

----

Re patch itself: I don't look into the script and it seems it would be
not productive. Just noted several points around, see below.

WBR, Alexander Turenko.

> gitlab-ci: implement packing into MCS S3
>

A few words why this is needed would be valuable, I think. Our main
motivation is to provide repositories, where packages are not pruned, so
a user may enable the repository and fix certain version of tarantool in
dependencies of an application package.

> Added ability to store packages additionaly at MCS S3.

Nit: I suggest to split paragraphs with empty lines for readability.

> The target idea was to add the new way of packages creation at MCS S3,
> which temporary duplicates the packaging at PackageCloud by Packpack
> tool. Also it was needed to pack the binaries in the native way of the
> each packing OS style. It was created standalone script for adding the
> packages binaries/sources to MCS S3 at DEB either RPM repositories:
> 'tools/add_pack_s3_repo.sh'
> Common parts of the script are:
>  - create new meta files for the new binaries
>  - copy new binaries to MCS S3
>  - get previous metafiles from MCS S3 and merge the new meta data for
> the new binaries

Nit: I suggest to format those lists like so:

 | - get previous metafiles from MCS S3 and merge the new meta data for
 |   the new binaries
 |   ^^ (carried line is indented as a text above it)

I also think that when a text im separated from a list with empty lines
this is more readable:

 | Look at the following items:
 |
 | - item 1
 | - item 2
 |
 | Lorem ipsum.

>  - update the meta files at MCS S3
> Different parts:
>  - DEB script part based on reprepro external tool, also it works separately
> only on OS versions level - it means that meta data it updates for all
> Distritbutions together.

Typo: Distri*t*butions.

>  - RPM script part based on createrepo external tool, also it works
> separately for OS/Release level - it means that meta data it updates
> for all Releases separately.
> 
> Closes #3380

Let's include docbot comment to don't forget to update download
instructions on the website.

>  static_build:
> -  <<: *deploy_test_definition
> +  <<: *release_only_definition
> +  stage: test
> +  tags:
> +    - deploy_test
>    variables:
>      RUN_TESTS: 'ON'
>    script:

This change seems to be out of scope of the commit, as I see. (To be
honest, I don't understand why it is necessary.)

> diff --git a/.gitlab.mk b/.gitlab.mk
> index 48a92e518..64664c64f 100644
> --- a/.gitlab.mk
> +++ b/.gitlab.mk
> @@ -98,13 +98,27 @@ vms_test_%:
>  vms_shutdown:
>  	VBoxManage controlvm ${VMS_NAME} poweroff
>  
> -# ########################
> -# Build RPM / Deb packages
> -# ########################
> +# ###########################
> +# Sources tarballs & packages
> +# ###########################
> +
> +# Push alpha and beta versions to <major>x bucket (say, 2x),
> +# stable to <major>.<minor> bucket (say, 2.2).
> +GIT_DESCRIBE=$(shell git describe HEAD)
> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
> +ifeq ($(MINOR_VERSION),0)
> +BUCKET=$(MAJOR_VERSION)x
> +endif
> +ifeq ($(MINOR_VERSION),1)
> +BUCKET=$(MAJOR_VERSION)x
> +endif
>  
>  package: git_submodule_update
>  	git clone https://github.com/packpack/packpack.git packpack
>  	PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
> +	./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build

So, when we want to test a feature / bugfix branch with `*-full-ci`
branch name, packages will be pushed and available for a user. This
seems to be mistake.

>  # Push alpha and beta versions to <major>x bucket (say, 2x),
>  # stable to <major>.<minor> bucket (say, 2.2).
> -ifeq ($(TRAVIS_BRANCH),master)
>  GIT_DESCRIBE=$(shell git describe HEAD)
>  MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>  MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
> -else
> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
> -endif
> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>  ifeq ($(MINOR_VERSION),0)
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
> +BUCKET=$(MAJOR_VERSION)x
>  endif
>  ifeq ($(MINOR_VERSION),1)
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
> +BUCKET=$(MAJOR_VERSION)x
>  endif

AFAIU it is purely for testing: deployment should occur only from 1.10,
2.1, 2.2, ..., master branches. In any other case it is does not matter
which values those variable have. I would discard this change.

>  
> +packpack_prepare:
> +	git clone https://github.com/packpack/packpack.git packpack
> +
> +package: packpack_prepare
> +	./packpack/packpack
> +
> +source: packpack_prepare
> +	TARBALL_COMPRESSOR=gz packpack/packpack tarball
> +
>  source_deploy:
>  	pip install awscli --user
> -	aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
> -		cp build/*.tar.gz "s3://${BUCKET}/" \
> -		--acl public-read
> +	for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
> +		aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
> +			cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
> +			--acl public-read

It was not possible to have several source tarballs for one commit. Now
it is not so? This commit should not touch source tarballs, I think: it
is about rpm/deb packages.

> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
> new file mode 100755
> index 000000000..f6e00e067
> --- /dev/null
> +++ b/tools/add_pack_s3_repo.sh

'add pack s3 repo' looks as a bag of words. Just mkrepo.sh looks better for me.
sync_repo.sh or update_repo.sh are also okay. Let's choose one that is
more or less fit English grammar.

> +Arguments:
> +    <path>
> +         Path points to the directory with deb/prm packages to be used.
> +         Script can be used in one of 2 modes:
> +          - path with binaries packages for a single distribution
> +          - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
> +                /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os

I have two points about this mirroring support:

* It is only for APT (but not for YUM), so cannot be used in uniform
  way.
* It seems it is not something to use in CI / CD, so it is dead code
  from this point of view.

Why should we add it?

> +    for loop_dist in $alldists ; do
> +        cat <<EOF >>$confpath/distributions
> +Origin: Tarantool
> +Label: tarantool.org
> +Suite: stable
> +Codename: $loop_dist
> +Architectures: amd64 source
> +Components: $component
> +Description: Unofficial Ubuntu Packages maintained by Tarantool

Why unofficial?

Why Ubuntu? It is for Debian too, right?

'Tarantool DBMS and Tarantool modules' is enough, I think.

> +SignWith: 91B625E5

Shouldn't it be an argument / environment variable? How the key is
stored?

I guess the right way would be like [3] (I don't mean using Travis-CI,
it is just example I have near to hands). The key point is to store
sensitive data encrypted within the repository, but use secret
environment variables to decrypt it in CI / CD.

[3]: https://docs.travis-ci.com/user/encrypting-files