[tarantool-patches] Re: [PATCH] Set format for spaces with sysview engine

[Konstantin Osipov <kostja@tarantool.org>]

* Alexander Turenko <alexander.turenko@tarantool.org> [19/04/21 01:38]:

Adding tap-like commands to tell the harness about start/end of an
individual test case to our .test files is a great
idea. In fact, if you look at most of the .test files, they
already contain well-written individual cases. 
I think we should just do it.

> On Thu, Apr 18, 2019 at 05:18:04PM +0300, Konstantin Osipov wrote:
> > * Kirill Yukhin <kyukhin@tarantool.org> [19/04/18 17:12]:
> > > How ofter do you scan list of regression tests? I bet - never.
> > > But you very often try to extract 2-3 lines of failed case.
> > 
> > If you work with the test suite as an active maintainer you work
> > with the files all the time. Otherwise your tests just rot.
> 
> It does not matter much whether test cases spread around its own files
> or grouped into categories within one file. The key point here is to
> make cases independent. Even when you have independent test cases you
> still can extract common preparation code that will be run before each
> test case (or before a group of test cases).
> 
> Look how test cases are organized in our projects with pytest or with
> failsafe maven plugin. They are grouped into files, but I still able to
> do something like `pytest test/test_foo.py -k test_bar` or `mvn verify
> -Dit.test=FooIT#testBar` and concentrate on one case.
> 
> When test cases depend on each other you need to manually find and
> extract all needed parts of a test file to run a case. When you do it
> several times per day, hey, it becomes annoying.
> 
> Now even if developers highly self-disciplined and write independent
> test cases, an extra work is still necessary to run a case if a harness
> is unable to run just what you need.
> 
> And while we are here there is other problem with console-input-like
> tests: often you don't sure which statements results are checked
> intentionally and which ones are run only for its side effects.
> Sometimes you need to git-blame over several attempts to rewrite a test
> to understand what the idea was.
> 
> Maybe we are near to the point when such simple approach gives more
> problems then solves.
> 
> Yep, I stated problems here, but don't propose anything. I don't know
> where is a right balance between pushing developers to write structured
> tests and the freedom developers have now.
> 
> I see that some developers like ability to test something with injecting
> just one line for one case into another one. I think that it will make
> tests more and more embrassing. So, I don't know.
> 
> WBR, Alexander Turenko.

-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov