<HTML><BODY><br><br><br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;">
Вторник, 14 января 2020, 20:42 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:<br><br><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY">I discussed what we're expect from rpm/deb repositories with Igor,<br>
Alexander T., Kirill Yu. I'll summarize key points below.<br><br>
CCed Konstantin N., Yaroslav D. and Maxim M.<br><br>
## Planned activities inside tarantool repository<br><br>
* We have packagecloud repositories and we're going to keep them and<br>
continue update them (at least for all versions below 2.4 inclusive).<br>
* We'll keep redirects from download.tarantool.org to packagecloud for<br>
those 1.10/2.1/2.2/2.3/2.4 repositories.<br>
* We'll create a new repository (working name is 'nightly'), which will<br>
contain packages from all long-term branches (1.10/2.1/.../master).<br>
Package names will be tarantool110, tarantool21, tarantool22,<br>
tarantool23, tarantool24.<br>
- TBD: Can we add some separator: tarantool<sepX>1<sepY>10? Should we?<br>
Should <sepX> be the same as <sepY>? Consider that it should work<br>
smooothly with apt and yum repos both. (AR Alexander Tikh.)<br>
* In addition to tarantool110 and so we'll deploy a package with name<br>
tarantool (or, maybe, tarantool-live?) from master as bleeding edge of<br>
development.<br><br>
## Necessary activities outside of tarantool repository<br><br>
* Manually deploy existing packages:<br>
- Rebuild last versions for 1.10/2.1/2.2/2.3/2.4 using the new naming<br>
scheme. Maybe also all tagged versions. The rebuild should not<br>
change `git describe --tags --long` output (including a commit<br>
hash).<br>
- Copy all module / connector packages from packagecloud.<br>
- TBD: Propose exact list (AR Alexander Tikh.)<br>
* Setup proper redirects from download.tarantool.org for the new<br>
repository.<br>
* Update CI / CD for modules (at least for most used ones) to use this<br>
script: download it from the tarantool repository and use to deploy.<br><br>
After that we can announce the new repository.<br><br>
## Further activities<br><br>
Announce the new repository and deprecation plan of old ones (planned to<br>
fully support the old repos for 2.4 and below, but don't add 2.5 and<br>
further repos).<br><br>
Update download instructions on the website (see [1]) to use the new<br>
repository.<br><br>
Provide the similar repository, but only for tagged builds (working name<br>
is 'release'). It seems it worth to push only tagged modules /<br>
connectors as well. (Asked by Konstantin N.) <br><br>
We maybe (if there will be a demand) will move the script outside of the<br>
repository and integrate with the existing server [2] that doing quite<br>
similar job: it receives built rocks (it is like python's wheels) and<br>
updates a repository for luarocks / tarantoolctl rocks. This can be done<br>
w/o user-visible changes. (Proposed by Konstantin N.)<br><br>
Aren't I forget anything?<br><br>
[1]: <a href="https://www.tarantool.io/en/download/" target="_blank">https://www.tarantool.io/en/download/</a><br>
[2]: <a href="https://github.com/tarantool/rocks.tarantool.org" target="_blank">https://github.com/tarantool/rocks.tarantool.org</a><br><br>
----<br><br>
Re patch itself: I don't look into the script and it seems it would be<br>
not productive. Just noted several points around, see below.<br><br>
WBR, Alexander Turenko.<br><br>
> gitlab-ci: implement packing into MCS S3<br>
><br><br>
A few words why this is needed would be valuable, I think. Our main<br>
motivation is to provide repositories, where packages are not pruned, so<br>
a user may enable the repository and fix certain version of tarantool in<br>
dependencies of an application package.<br></div></div></div></div></blockquote><p>Commit message improved - added information that the packages not pruning<br>and that user may use certain version of Tarantool in dependencies of an<br>application package.<br></p><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br>
> Added ability to store packages additionally at MCS S3.<br><br>
Nit: I suggest to split paragraphs with empty lines for readability.</div></div></div></div></blockquote>Done.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> The target idea was to add the new way of packages creation at MCS S3,<br>
> which temporary duplicates the packaging at PackageCloud by Packpack<br>
> tool. Also it was needed to pack the binaries in the native way of the<br>
> each packing OS style. It was created standalone script for adding the<br>
> packages binaries/sources to MCS S3 at DEB either RPM repositories:<br>
> 'tools/add_pack_s3_repo.sh'<br>
> Common parts of the script are:<br>
> - create new meta files for the new binaries<br>
> - copy new binaries to MCS S3<br>
> - get previous metafiles from MCS S3 and merge the new meta data for<br>
> the new binaries<br><br>
Nit: I suggest to format those lists like so:<br><br>
| - get previous metafiles from MCS S3 and merge the new meta data for<br>
| the new binaries<br>
| ^^ (carried line is indented as a text above it)</div></div></div></div></blockquote>Done.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
I also think that when a text im separated from a list with empty lines<br>
this is more readable:<br><br>
| Look at the following items:<br>
|<br>
| - item 1<br>
| - item 2<br>
|<br>
| Lorem ipsum.<br></div></div></div></div></blockquote>Done.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br>
> - update the meta files at MCS S3<br>
> Different parts:<br>
> - DEB script part based on reprepro external tool, also it works separately<br>
> only on OS versions level - it means that meta data it updates for all<br>
> Distritbutions together.<br><br>
Typo: Distri*t*butions.</div></div></div></div></blockquote>Fixed.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> - RPM script part based on createrepo external tool, also it works<br>
> separately for OS/Release level - it means that meta data it updates<br>
> for all Releases separately.<br>
> <br>
> Closes #3380<br><br>
Let's include docbot comment to don't forget to update download<br>
instructions on the website.</div></div></div></div></blockquote>Added short comment.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> static_build:<br>
> - <<: *deploy_test_definition<br>
> + <<: *release_only_definition<br>
> + stage: test<br>
> + tags:<br>
> + - deploy_test<br>
> variables:<br>
> RUN_TESTS: 'ON'<br>
> script:<br><br>
This change seems to be out of scope of the commit, as I see. (To be<br>
honest, I don't understand why it is necessary.)</div></div></div></div></blockquote><p>Reverted as not needed here.</p><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> diff --git a/.gitlab.mk b/.gitlab.mk<br>
> index 48a92e518..64664c64f 100644<br>
> --- a/.gitlab.mk<br>
> +++ b/.gitlab.mk<br>
> @@ -98,13 +98,27 @@ vms_test_%:<br>
> vms_shutdown:<br>
> VBoxManage controlvm ${VMS_NAME} poweroff<br>
> <br>
> -# ########################<br>
> -# Build RPM / Deb packages<br>
> -# ########################<br>
> +# ###########################<br>
> +# Sources tarballs & packages<br>
> +# ###########################<br>
> +<br>
> +# Push alpha and beta versions to <major>x bucket (say, 2x),<br>
> +# stable to <major>.<minor> bucket (say, 2.2).<br>
> +GIT_DESCRIBE=$(shell git describe HEAD)<br>
> +MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))<br>
> +MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))<br>
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)<br>
> +ifeq ($(MINOR_VERSION),0)<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> +endif<br>
> +ifeq ($(MINOR_VERSION),1)<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> +endif<br>
> <br>
> package: git_submodule_update<br>
> git clone <a href="https://github.com/packpack/packpack.git" target="_blank">https://github.com/packpack/packpack.git</a> packpack<br>
> PACKPACK_EXTRA_DOCKER_RUN_PARAMS='--network=host' ./packpack/packpack<br>
> + ./tools/add_pack_s3_repo.sh -b=${BUCKET} -o=${OS} -d=${DIST} build<br><br>
So, when we want to test a feature / bugfix branch with `*-full-ci`<br>
branch name, packages will be pushed and available for a user. This<br>
seems to be mistake.</div></div></div></div></blockquote>Fixed - added standalone checks at gitlab-ci for testing branches,<br>also added different jobs for deploying and packing.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> # Push alpha and beta versions to <major>x bucket (say, 2x),<br>
> # stable to <major>.<minor> bucket (say, 2.2).<br>
> -ifeq ($(TRAVIS_BRANCH),master)<br>
> GIT_DESCRIBE=$(shell git describe HEAD)<br>
> MAJOR_VERSION=$(word 1,$(subst ., ,$(GIT_DESCRIBE)))<br>
> MINOR_VERSION=$(word 2,$(subst ., ,$(GIT_DESCRIBE)))<br>
> -else<br>
> -MAJOR_VERSION=$(word 1,$(subst ., ,$(TRAVIS_BRANCH)))<br>
> -MINOR_VERSION=$(word 2,$(subst ., ,$(TRAVIS_BRANCH)))<br>
> -endif<br>
> -BUCKET=tarantool.$(MAJOR_VERSION).$(MINOR_VERSION).src<br>
> +BUCKET=$(MAJOR_VERSION)_$(MINOR_VERSION)<br>
> ifeq ($(MINOR_VERSION),0)<br>
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> endif<br>
> ifeq ($(MINOR_VERSION),1)<br>
> -BUCKET=tarantool.$(MAJOR_VERSION)x.src<br>
> +BUCKET=$(MAJOR_VERSION)x<br>
> endif<br><br>
AFAIU it is purely for testing: deployment should occur only from 1.10,<br>
2.1, 2.2, ..., master branches. In any other case it is does not matter<br>
which values those variable have. I would discard this change.<br></div></div></div></div></blockquote>Moved changes to another standalone patch set commit.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br>
> <br>
> +packpack_prepare:<br>
> + git clone <a href="https://github.com/packpack/packpack.git" target="_blank">https://github.com/packpack/packpack.git</a> packpack<br>
> +<br>
> +package: packpack_prepare<br>
> + ./packpack/packpack<br>
> +<br>
> +source: packpack_prepare<br>
> + TARBALL_COMPRESSOR=gz packpack/packpack tarball<br>
> +<br>
> source_deploy:<br>
> pip install awscli --user<br>
> - aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \<br>
> - cp build/*.tar.gz "s3://${BUCKET}/" \<br>
> - --acl public-read<br>
> + for tarball in `ls build/*.tar.gz 2>/dev/null` ; \<br>
> + aws --endpoint-url "${AWS_S3_ENDPOINT_URL}" s3 \<br>
> + cp ${tarball} "s3://tarantool_repo/${BUCKET}/sources/" \<br>
> + --acl public-read<br><br>
It was not possible to have several source tarballs for one commit. Now<br>
it is not so? This commit should not touch source tarballs, I think: it<br>
is about rpm/deb packages.</div></div></div></div></blockquote>Moved changes to another standalone patch set commit.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> diff --git a/tools/add_pack_s3_repo.sh b/tools/add_pack_s3_repo.sh<br>
> new file mode 100755<br>
> index 000000000..f6e00e067<br>
> --- /dev/null<br>
> +++ b/tools/add_pack_s3_repo.sh<br><br>
'add pack s3 repo' looks as a bag of words. Just mkrepo.sh looks better for me.<br>
sync_repo.sh or update_repo.sh are also okay. Let's choose one that is<br>
more or less fit English grammar.</div></div></div></div></blockquote>Renamed to 'update_repo.sh'<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> +Arguments:<br>
> + <path><br>
> + Path points to the directory with deb/prm packages to be used.<br>
> + Script can be used in one of 2 modes:<br>
> + - path with binaries packages for a single distribution<br>
> + - path with 'pool' subdirectory with APT repository (only: debian|ubuntu), like:<br>
> + /var/spool/apt-mirror/mirror/packagecloud.io/tarantool/$branch/$os<br><br>
I have two points about this mirroring support:<br><br>
* It is only for APT (but not for YUM), so cannot be used in uniform<br>
way.<br>
* It seems it is not something to use in CI / CD, so it is dead code<br>
from this point of view.<br><br>
Why should we add it?</div></div></div></div></blockquote>We need the way for porting the packagecloud DEB and RPM packages.<br>For the DEB packages we use apt-mirror which creates APT repository<br>and for porting it to MCS S3 this additional option was enabled. For RPM<br>packages it is not needed, because mirroring YUM repositories results<br>in flat directory with packages which can be ported to MCS S3 w/o any<br>additional options.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> + for loop_dist in $alldists ; do<br>
> + cat <<EOF >>$confpath/distributions<br>
> +Origin: Tarantool<br>
> +Label: tarantool.org<br>
> +Suite: stable<br>
> +Codename: $loop_dist<br>
> +Architectures: amd64 source<br>
> +Components: $component<br>
> +Description: Unofficial Ubuntu Packages maintained by Tarantool<br><br>
Why unofficial?<br><br>
Why Ubuntu? It is for Debian too, right?<br><br>
'Tarantool DBMS and Tarantool modules' is enough, I think.</div></div></div></div></blockquote>Done.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
> +SignWith: 91B625E5<br><br>
Shouldn't it be an argument / environment variable? How the key is<br>
stored?</div></div></div></div></blockquote>The key is stored at the build hosts.<br><blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"><div id=""><div class="js-helper js-readmsg-msg"><div><div id="style_15790237451977764898_BODY"><br><br>
I guess the right way would be like [3] (I don't mean using Travis-CI,<br>
it is just example I have near to hands). The key point is to store<br>
sensitive data encrypted within the repository, but use secret<br>
environment variables to decrypt it in CI / CD.<br><br>
[3]: <a href="https://docs.travis-ci.com/user/encrypting-files" target="_blank">https://docs.travis-ci.com/user/encrypting-files</a><br></div></div></div></div></blockquote>I've tried a lot to make it in this way, but failed that is why I've used hosts<br>setup for keys. I'll reproduce the issue to show it, if I'll find the other solution<br>I'll provide it.<br>
<br>-- <br>Alexander Tikhonov<br></BODY></HTML>