[tarantool-patches] Re: [PATCH] Set format for spaces with sysview engine

[Vladislav Shpilevoy <v.shpilevoy@tarantool.org>]

On 21/04/2019 01:36, Alexander Turenko wrote:
> On Thu, Apr 18, 2019 at 05:18:04PM +0300, Konstantin Osipov wrote:
>> * Kirill Yukhin <kyukhin@tarantool.org> [19/04/18 17:12]:
>>> How ofter do you scan list of regression tests? I bet - never.
>>> But you very often try to extract 2-3 lines of failed case.
>>
>> If you work with the test suite as an active maintainer you work
>> with the files all the time. Otherwise your tests just rot.
> 
> It does not matter much whether test cases spread around its own files
> or grouped into categories within one file. The key point here is to
> make cases independent. Even when you have independent test cases you
> still can extract common preparation code that will be run before each
> test case (or before a group of test cases).
> 
> Look how test cases are organized in our projects with pytest or with
> failsafe maven plugin. They are grouped into files, but I still able to
> do something like `pytest test/test_foo.py -k test_bar` or `mvn verify
> -Dit.test=FooIT#testBar` and concentrate on one case.
> 
> When test cases depend on each other you need to manually find and
> extract all needed parts of a test file to run a case. When you do it
> several times per day, hey, it becomes annoying.
> 
> Now even if developers highly self-disciplined and write independent
> test cases, an extra work is still necessary to run a case if a harness
> is unable to run just what you need.
> 
> And while we are here there is other problem with console-input-like
> tests: often you don't sure which statements results are checked
> intentionally and which ones are run only for its side effects.
> Sometimes you need to git-blame over several attempts to rewrite a test
> to understand what the idea was.
> 
> Maybe we are near to the point when such simple approach gives more
> problems then solves.
> 
> Yep, I stated problems here, but don't propose anything. I don't know
> where is a right balance between pushing developers to write structured
> tests and the freedom developers have now.
> 
> I see that some developers like ability to test something with injecting
> just one line for one case into another one. I think that it will make
> tests more and more embrassing. So, I don't know.

My point is that many issues are because of typos or another extra simple
mistakes, and what is more - they are linked somehow to a swarm of other
similar issues, already tested somewhere.

For example, the test you've fixed for netbox about 'unique' index
flag. It was just a typo, and we have a place, where other space/index
properties are tested. It is logically to test one another index property
together with others. Just imagine how many tests you would have had if we
had dedicated one test file to each space and index option. In such a
situation parallel test run will not help you, when number of test files
will be thousands. You just do not have so many cores so as to swiftly
start and shutdown new processes and threads.

Even SQLite does not dedicate new test file per each single tiny typo-like
issue. They have dedicated file per issue, but only for really hard ones.

> 
> WBR, Alexander Turenko.
>