[tarantool-patches] Re: [PATCH] Set format for spaces with sysview engine

[Alexander Turenko <alexander.turenko@tarantool.org>]

On Thu, Apr 18, 2019 at 05:18:04PM +0300, Konstantin Osipov wrote:
> * Kirill Yukhin <kyukhin@tarantool.org> [19/04/18 17:12]:
> > How ofter do you scan list of regression tests? I bet - never.
> > But you very often try to extract 2-3 lines of failed case.
> 
> If you work with the test suite as an active maintainer you work
> with the files all the time. Otherwise your tests just rot.

It does not matter much whether test cases spread around its own files
or grouped into categories within one file. The key point here is to
make cases independent. Even when you have independent test cases you
still can extract common preparation code that will be run before each
test case (or before a group of test cases).

Look how test cases are organized in our projects with pytest or with
failsafe maven plugin. They are grouped into files, but I still able to
do something like `pytest test/test_foo.py -k test_bar` or `mvn verify
-Dit.test=FooIT#testBar` and concentrate on one case.

When test cases depend on each other you need to manually find and
extract all needed parts of a test file to run a case. When you do it
several times per day, hey, it becomes annoying.

Now even if developers highly self-disciplined and write independent
test cases, an extra work is still necessary to run a case if a harness
is unable to run just what you need.

And while we are here there is other problem with console-input-like
tests: often you don't sure which statements results are checked
intentionally and which ones are run only for its side effects.
Sometimes you need to git-blame over several attempts to rewrite a test
to understand what the idea was.

Maybe we are near to the point when such simple approach gives more
problems then solves.

Yep, I stated problems here, but don't propose anything. I don't know
where is a right balance between pushing developers to write structured
tests and the freedom developers have now.

I see that some developers like ability to test something with injecting
just one line for one case into another one. I think that it will make
tests more and more embrassing. So, I don't know.

WBR, Alexander Turenko.