[Tarantool-patches] [PATCH v5] gitlab-ci: implement packing into MCS S3
Alexander Tikhonov
avtikhon at tarantool.org
Wed Jan 15 20:32:59 MSK 2020
>Вторник, 14 января 2020, 20:42 +03:00 от Alexander Turenko <alexander.turenko at tarantool.org>:
>
>I discussed what we're expect from rpm/deb repositories with Igor,
>Alexander T., Kirill Yu. I'll summarize key points below.
>
>CCed Konstantin N., Yaroslav D. and Maxim M.
>
>## Planned activities inside tarantool repository
>
>* We have packagecloud repositories and we're going to keep them and
> continue update them (at least for all versions below 2.4 inclusive).
>* We'll keep redirects from download.tarantool.org to packagecloud for
> those 1.10/2.1/2.2/2.3/2.4 repositories.
>* We'll create a new repository (working name is 'nightly'), which will
> contain packages from all long-term branches (1.10/2.1/.../master).
> Package names will be tarantool110, tarantool21, tarantool22,
> tarantool23, tarantool24.
> - TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?
> Should <sepX> be the same as <sepY>? Consider that it should work
> smooothly with apt and yum repos both. (AR Alexander Tikh.)
>* In addition to tarantool110 and so we'll deploy a package with name
> tarantool (or, maybe, tarantool-live?) from master as bleeding edge of
> development.
>
>## Necessary activities outside of tarantool repository
>
>* Manually deploy existing packages:
> - Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming
> scheme. Maybe also all tagged versions. The rebuild should not
> change `git describe --tags --long` output (including a commit
> hash).
> - Copy all module / connector packages from packagecloud.
> - TBD: Propose exact list (AR Alexander Tikh.)
>* Setup proper redirects from download.tarantool.org for the new
> repository.
>* Update CI / CD for modules (at least for most used ones) to use this
> script: download it from the tarantool repository and use to deploy.
>
>After that we can announce the new repository.
>
>## Further activities
>
>Announce the new repository and deprecation plan of old ones (planned to
>fully support the old repos for 2.4 and below, but don't add 2.5 and
>further repos).
>
>Update download instructions on the website (see [1]) to use the new
>repository.
>
>Provide the similar repository, but only for tagged builds (working name
>is 'release'). It seems it worth to push only tagged modules /
>connectors as well. (Asked by Konstantin N.)
>
>We maybe (if there will be a demand) will move the script outside of the
>repository and integrate with the existing server [2] that doing quite
>similar job: it receives built rocks (it is like python's wheels) and
>updates a repository for luarocks / tarantoolctl rocks. This can be done
>w/o user-visible changes. (Proposed by Konstantin N.)
>
>Aren't I forget anything?
>
>[1]: https://www.tarantool.io/en/download/
>[2]: https://github.com/tarantool/rocks.tarantool.org
>
>----
>
>Re patch itself: I don't look into the script and it seems it would be
>not productive. Just noted several points around, see below.
>
>WBR, Alexander Turenko.
>
>> gitlab-ci: implement packing into MCS S3
>>
>
>A few words why this is needed would be valuable, I think. Our main
>motivation is to provide repositories, where packages are not pruned, so
>a user may enable the repository and fix certain version of tarantool in
>dependencies of an application package.
Commit message improved - added information that the packages not pruning
and that user may use certain version of Tarantool in dependencies of an
application package.
>
>> Added ability to store packages additionally at MCS S3.
>
>Nit: I suggest to split paragraphs with empty lines for readability.
Done.
>
>
>> The target idea was to add the new way of packages creation at MCS S3,
>> which temporary duplicates the packaging at PackageCloud by Packpack
>> tool. Also it was needed to pack the binaries in the native way of the
>> each packing OS style. It was created standalone script for adding the
>> packages binaries/sources to MCS S3 at DEB either RPM repositories:
>> 'tools/add_pack_s3_repo.sh'
>> Common parts of the script are:
>> - create new meta files for the new binaries
>> - copy new binaries to MCS S3
>> - get previous metafiles from MCS S3 and merge the new meta data for
>> the new binaries
>
>Nit: I suggest to format those lists like so:
>
> | - get previous metafiles from MCS S3 and merge the new meta data for
> | the new binaries
> | ^^ (carried line is indented as a text above it)
Done.
>
>
>I also think that when a text im separated from a list with empty lines
>this is more readable:
>
> | Look at the following items:
> |
> | - item 1
> | - item 2
> |
> | Lorem ipsum.
Done.
>
>> - update the meta files at MCS S3
>> Different parts:
>> - DEB script part based on reprepro external tool, also it works separately
>> only on OS versions level - it means that meta data it updates for all
>> Distritbutions together.
>
>Typo: Distri*t*butions.
Fixed.
>
>
>> - RPM script part based on createrepo external tool, also it works
>> separately for OS/Release level - it means that meta data it updates
>> for all Releases separately.
>>
>> Closes #3380
>
>Let's include docbot comment to don't forget to update download
>instructions on the website.
Added short comment.
>
>
>> static_build:
>> - <<: *deploy_test_definition
>> + <<: *release_only_definition
>> + stage: test
>> + tags:
>> + - deploy_test
>> variables:
>> RUN_TESTS: 'ON'
>> script:
>
>This change seems to be out of scope of the commit, as I see. (To be
>honest, I don't understand why it is necessary.)
Reverted as not needed here.
>
>
>> diff --git a/.gitlab.mk b/.gitlab.mk
>> index 48a92e518..64664c64f 100644
>> --- a/.gitlab.mk
>> +++ b/.gitlab.mk
>> @@ -98,13 +98,27 @@ vms_test_%:
>> vms_shutdown:
>> VBoxManage controlvm ${VMS_NAME} poweroff
>>
>> -# ########################
>> -# Build RPM / Deb packages
>> -# ########################
>> +# ###########################
>> +# Sources tarballs & packages
>> +# ###########################
>> +
>> +# Push alpha and beta versions to <major>x bucket (say, 2x),
>> +# stable to <major>.<minor> bucket (say, 2.2).
>> +GIT_DESCRIBE=$(shell git describe HEAD)
>> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>> +ifeq ($(MINOR_VERSION),0)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>> +ifeq ($(MINOR_VERSION),1)
>> +BUCKET=$(MAJOR_VERSION)x
>> +endif
>>
>> package: git_submodule_update
>> git clone https://github.com/packpack/packpack.git packpack
>> PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack
>> + ./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build
>
>So, when we want to test a feature / bugfix branch with `*-full-ci`
>branch name, packages will be pushed and available for a user. This
>seems to be mistake.
Fixed - added standalone checks at gitlab-ci for testing branches,
also added different jobs for deploying and packing.
>
>
>> # Push alpha and beta versions to <major>x bucket (say, 2x),
>> # stable to <major>.<minor> bucket (say, 2.2).
>> -ifeq ($(TRAVIS_BRANCH),master)
>> GIT_DESCRIBE=$(shell git describe HEAD)
>> MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))
>> MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))
>> -else
>> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))
>> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))
>> -endif
>> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src
>> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)
>> ifeq ($(MINOR_VERSION),0)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>> endif
>> ifeq ($(MINOR_VERSION),1)
>> -BUCKET=tarantool.$(MAJOR_VERSION)x.src
>> +BUCKET=$(MAJOR_VERSION)x
>> endif
>
>AFAIU it is purely for testing: deployment should occur only from 1.10,
>2.1, 2.2, ..., master branches. In any other case it is does not matter
>which values those variable have. I would discard this change.
Moved changes to another standalone patch set commit.
>
>>
>> +packpack_prepare:
>> + git clone https://github.com/packpack/packpack.git packpack
>> +
>> +package: packpack_prepare
>> + ./packpack/packpack
>> +
>> +source: packpack_prepare
>> + TARBALL_COMPRESSOR=gz packpack/packpack tarball
>> +
>> source_deploy:
>> pip install awscli --user
>> - aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> - cp build/*.tar.gz "s3://${BUCKET}/" \
>> - --acl public-read
>> + for tarball in `ls build/*.tar.gz 2>/dev/null` ; \
>> + aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \
>> + cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \
>> + --acl public-read
>
>It was not possible to have several source tarballs for one commit. Now
>it is not so? This commit should not touch source tarballs, I think: it
>is about rpm/deb packages.
Moved changes to another standalone patch set commit.
>
>
>> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh
>> new file mode 100755
>> index 000000000..f6e00e067
>> --- /dev/null
>> +++ b/tools/add_pack_s3_repo.sh
>
>'add pack s3 repo' looks as a bag of words. Just mkrepo.sh looks better for me.
>sync_repo.sh or update_repo.sh are also okay. Let's choose one that is
>more or less fit English grammar.
Renamed to 'update_repo.sh'
>
>
>> +Arguments:
>> + <path>
>> + Path points to the directory with deb/prm packages to be used.
>> + Script can be used in one of 2 modes:
>> + - path with binaries packages for a single distribution
>> + - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:
>> + /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os
>
>I have two points about this mirroring support:
>
>* It is only for APT (but not for YUM), so cannot be used in uniform
> way.
>* It seems it is not something to use in CI / CD, so it is dead code
> from this point of view.
>
>Why should we add it?
We need the way for porting the packagecloud DEB and RPM packages.
For the DEB packages we use apt-mirror which creates APT repository
and for porting it to MCS S3 this additional option was enabled. For RPM
packages it is not needed, because mirroring YUM repositories results
in flat directory with packages which can be ported to MCS S3 w/o any
additional options.
>
>
>> + for loop_dist in $alldists ; do
>> + cat <<EOF >>$confpath/distributions
>> +Origin: Tarantool
>> +Label: tarantool.org
>> +Suite: stable
>> +Codename: $loop_dist
>> +Architectures: amd64 source
>> +Components: $component
>> +Description: Unofficial Ubuntu Packages maintained by Tarantool
>
>Why unofficial?
>
>Why Ubuntu? It is for Debian too, right?
>
>'Tarantool DBMS and Tarantool modules' is enough, I think.
Done.
>
>
>> +SignWith: 91B625E5
>
>Shouldn't it be an argument / environment variable? How the key is
>stored?
The key is stored at the build hosts.
>
>
>I guess the right way would be like [3] (I don't mean using Travis-CI,
>it is just example I have near to hands). The key point is to store
>sensitive data encrypted within the repository, but use secret
>environment variables to decrypt it in CI / CD.
>
>[3]: https://docs.travis-ci.com/user/encrypting-files
I've tried a lot to make it in this way, but failed that is why I've used hosts
setup for keys. I'll reproduce the issue to show it, if I'll find the other solution
I'll provide it.
--
Alexander Tikhonov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.tarantool.org/pipermail/tarantool-patches/attachments/20200115/29b5f6df/attachment.html>
More information about the Tarantool-patches
mailing list