[Tarantool-patches] [PATCH v1] gitlab-ci: cleanup submodules

[Tarantool-patches] [PATCH v1] gitlab-ci: cleanup submodules

[Alexander V. Tikhonov]

Found that durring runs on Gitlab-CI workspaces
some extra files from tests from the submodules
like src/lib/small/* leaves in subdirectories,
which produce warning messages at the Gitlab-CI
run log, to avoid of it the submodules cleanup
it needed before each Gitlab-CI job.

Closes #4258
---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4258-clean-ws-full-ci
Issue: https://github.com/tarantool/tarantool/issues/4258

 .gitlab-ci.yml | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 04bb4b1cc..91326b218 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -4,6 +4,9 @@ stages:
 variables:
   GITLAB_MAKE: "make -f .gitlab.mk"
 
+before_script:
+  - git submodule foreach git clean -xfd
+
 # Jobs templates
 
 .release_only_template: &release_only_definition
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup submodules

[Alexander Turenko]

All jobs are run in a docker or in a virtual machine. How can we get a
dirty directory for a run?

I don't mind if it solves some problems, but now I don't understand what
the problems are.

WBR, Alexander Turenko.

On Thu, Nov 07, 2019 at 02:13:46PM +0300, Alexander V. Tikhonov wrote:
> Found that durring runs on Gitlab-CI workspaces

durring -> during

> some extra files from tests from the submodules
> like src/lib/small/* leaves in subdirectories,
> which produce warning messages at the Gitlab-CI
> run log, to avoid of it the submodules cleanup

Which kind of warnings?

> it needed before each Gitlab-CI job.

it -> is?

> 
> Closes #4258

I would treat the issue as a more wide problem, then CI's one (Kirill
met it locally). I propose to left the issue open, but mention it like:

 | Follows on #4258.

> ---
> 
> Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4258-clean-ws-full-ci
> Issue: https://github.com/tarantool/tarantool/issues/4258
> 
>  .gitlab-ci.yml | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 04bb4b1cc..91326b218 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -4,6 +4,9 @@ stages:
>  variables:
>    GITLAB_MAKE: "make -f .gitlab.mk"
>  
> +before_script:
> +  - git submodule foreach git clean -xfd
> +

1. Run `git clean -xfd` too (not only submodules).
2. Add --recursive after 'foreach' just in case.
3. Enclose 'git clean -xfd' for submodules into quotes: otherwise it
   does not work on some git versions (I found that after some update).

For the reference, my one-liner that always work (at least for me):

$ git clean -xfd; git submodule foreach --recursive 'git clean -xfd'

>  # Jobs templates
>  
>  .release_only_template: &release_only_definition
> -- 
> 2.17.1
> 

Re: [Tarantool-patches] [PATCH v1] gitlab-ci: cleanup submodules

[Alexander Tikhonov]

[-- Attachment #1: Type: text/plain, Size: 3343 bytes --]


Alexander,

Thanks for a review, please check my comments 
>Понедельник,  9 декабря 2019, 6:12 +03:00 от Alexander Turenko <alexander.turenko@tarantool.org>:
>
>All jobs are run in a docker or in a virtual machine. How can we get a
>dirty directory for a run?
That is not quiet right, yes - we use the docker for running the testing jobs,
but the source code comes into the docker using shared path from the real host:
/home/gitlab-runner/builds/<uniq name per each job runner>/<number of the parallel job of the runner>/tarantool/tarantool
Also we use the following setting:
https://gitlab.com/tarantool/tarantool/-/settings/ci_cd
git fetch  
Faster as it re-uses the project workspace (falling back to clone if it doesn't exist)
It means that the source code is saving between each gitlab-runner jobs, and only fetching.
In this way the submodule may have some files left from the previous checked tag, like we
have files left after switching from 2.x releases to 1.10 - at this switching some tests files
are left in the sources and it makes the test-run tool tries to recreate the tests result files
automatically, like here:
https://gitlab.com/tarantool/tarantool/-/jobs/371542808
3367   [385] TAP13 parse failed: Missing plan in the TAP source
3368   [385] [ new ]
3369   [385] small/slab_arena.test [ pass ]
3370   [385] small/slab_cache.test [ pass ]
3371   [385] small/small_alloc.test [ pass ]
3372   [385] small/static.test 
3373   [385] TAP13 parse failed: Missing plan in the TAP source
3374   [385] [ new ]
>
>
>I don't mind if it solves some problems, but now I don't understand what
>the problems are.
>
>WBR, Alexander Turenko.
>
>On Thu, Nov 07, 2019 at 02:13:46PM +0300, Alexander V. Tikhonov wrote:
>> Found that durring runs on Gitlab-CI workspaces
>
>durring -> during
>
>> some extra files from tests from the submodules
>> like src/lib/small/* leaves in subdirectories,
>> which produce warning messages at the Gitlab-CI
>> run log, to avoid of it the submodules cleanup
>
>Which kind of warnings?
>
>> it needed before each Gitlab-CI job.
>
>it -> is?
>
>> 
>> Closes #4258
>
>I would treat the issue as a more wide problem, then CI's one (Kirill
>met it locally). I propose to left the issue open, but mention it like:
>
> | Follows on #4258.
>
>> ---
>> 
>> Github:  https://github.com/tarantool/tarantool/tree/avtikhon/gh-4258-clean-ws-full-ci
>> Issue:  https://github.com/tarantool/tarantool/issues/4258
>> 
>>  .gitlab-ci.yml | 3 +++
>>  1 file changed, 3 insertions(+)
>> 
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 04bb4b1cc..91326b218 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -4,6 +4,9 @@ stages:
>>  variables:
>>    GITLAB_MAKE: "make -f .gitlab.mk"
>> 
>> +before_script:
>> +  - git submodule foreach git clean -xfd
>> +
>
>1. Run `git clean -xfd` too (not only submodules).
>2. Add --recursive after 'foreach' just in case.
>3. Enclose 'git clean -xfd' for submodules into quotes: otherwise it
>   does not work on some git versions (I found that after some update).
>
>For the reference, my one-liner that always work (at least for me):
>
>$ git clean -xfd; git submodule foreach --recursive 'git clean -xfd'
>
>>  # Jobs templates
>> 
>>  .release_only_template: &release_only_definition
>> -- 
>> 2.17.1
>> 


-- 
Alexander Tikhonov

[-- Attachment #2: Type: text/html, Size: 22823 bytes --]