[Tarantool-patches] [PATCH v2] Add exclude option as suite/test pattern

[Tarantool-patches] [PATCH v2] Add exclude option as suite/test pattern

[Alexander V. Tikhonov]

Added exclude option to be able to exclude the tests
by patterns suite/test from testing, it means that
exclude option even excludes the suites given with
option suite either test pattern.
Also added ability to set the list of exclude patterns
by the environment variable TEST_RUN_EXCLUDE_TESTS.

Close #54
---

Github: https://github.com/tarantool/test-run/tree/avtikhon/gh-54-exclude-option
Issue: https://github.com/tarantool/test-run/issues/54

 lib/app_server.py       | 10 +++++++++-
 lib/options.py          | 11 +++++++++++
 lib/tarantool_server.py | 21 +++++++++++++++++++--
 lib/unittest_server.py  | 10 +++++++++-
 4 files changed, 48 insertions(+), 4 deletions(-)

diff --git a/lib/app_server.py b/lib/app_server.py
index 705af18..5a729bf 100644
--- a/lib/app_server.py
+++ b/lib/app_server.py
@@ -127,10 +127,18 @@ class AppServer(Server):
 
     @staticmethod
     def find_tests(test_suite, suite_path):
+        def is_test_excluded(test_suite, test_name):
+            # loop the list of test patterns need to exclude
+            for name in filter(None, test_suite.args.exclude):
+                # check if the test matches the exclude pattern
+                if test_name.find(name) != -1:
+                    return True
+            return False
+
         def patterned(test_name, patterns):
             answer = []
             for i in patterns:
-                if test_name.find(i) != -1:
+                if test_name.find(i) != -1 and not is_test_excluded(test_suite, test_name):
                     answer.append(test_name)
             return answer
 
diff --git a/lib/options.py b/lib/options.py
index 94dbd17..d97ec9e 100644
--- a/lib/options.py
+++ b/lib/options.py
@@ -49,6 +49,17 @@ class Options:
                 tests starting with "show" in "box" suite. Default: run all
                 tests in all specified suites.""")
 
+        parser.add_argument(
+                "--exclude",
+                action='append',
+                default = env_list('TEST_RUN_EXCLUDE_TESTS', ['']),
+                help="""Can be empty. List of excluding test names, to look for in
+                suites. Each name is used as a substring to look for in the
+                path to test file, e.g. "show" will exclude all tests that have
+                "show" in their name in all suites, "box/show" will only disable
+                tests starting with "show" in "box" suite. Default: "" -
+                means no tests to exclude""")
+
         parser.add_argument(
                 "--suite",
                 dest='suites',
diff --git a/lib/tarantool_server.py b/lib/tarantool_server.py
index 0340ada..96a158c 100644
--- a/lib/tarantool_server.py
+++ b/lib/tarantool_server.py
@@ -1077,6 +1077,14 @@ class TarantoolServer(Server):
                 res.extend(sorted(glob.glob(path_pattern)))
             return res
 
+        def is_test_excluded(test_suite, test):
+            # loop the list of test patterns need to exclude
+            for name in filter(None, test_suite.args.exclude):
+                # check if the test matches the exclude pattern
+                if test.name.find(name) != -1:
+                    return True
+            return False
+
         # Add Python tests.
         tests = [PythonTest(k, test_suite.args, test_suite.ini)
                  for k in get_tests("*.test.py")]
@@ -1102,13 +1110,22 @@ class TarantoolServer(Server):
             else:
                 tests.append(LuaTest(k, test_suite.args, test_suite.ini))
 
-        test_suite.tests = []
+        found_tests = []
         # don't sort, command line arguments must be run in
         # the specified order
+        # loop the list of test patterns need to run
         for name in test_suite.args.tests:
+            # loop the list of found tests at the sources
             for test in tests:
+                # check if the test matches the needed pattern
                 if test.name.find(name) != -1:
-                    test_suite.tests.append(test)
+                    found_tests.append(test)
+
+        test_suite.tests = []
+        # loop the list of found tests need to run
+        for test in found_tests:
+            if not is_test_excluded(test_suite, test):
+                test_suite.tests.append(test)
 
     def get_param(self, param=None):
         if param is not None:
diff --git a/lib/unittest_server.py b/lib/unittest_server.py
index a89a807..3aa33ab 100644
--- a/lib/unittest_server.py
+++ b/lib/unittest_server.py
@@ -61,10 +61,18 @@ class UnittestServer(Server):
 
     @staticmethod
     def find_tests(test_suite, suite_path):
+        def is_test_excluded(test_suite, test):
+            # loop the list of test patterns need to exclude
+            for name in filter(None, test_suite.args.exclude):
+                # check if the test matches the exclude pattern
+                if test.name.find(name) != -1:
+                    return True
+            return False
+
         def patterned(test, patterns):
             answer = []
             for i in patterns:
-                if test.name.find(i) != -1:
+                if test.name.find(i) != -1 and not is_test_excluded(test_suite, test):
                     answer.append(test)
             return answer
 
-- 
2.17.1

Re: [Tarantool-patches] [PATCH v2] Add exclude option as suite/test pattern

[Alexander Turenko]

I started to simplify the code, reduce code duplication, clarification
of texts and found that I completely rewrote the whole patch. So I sent
it as the separate PR: https://github.com/tarantool/test-run/pull/191

So now I asked you for review :)

WBR, Alexander Turenko.

On Thu, Oct 24, 2019 at 01:58:28PM +0300, Alexander V. Tikhonov wrote:
> Added exclude option to be able to exclude the tests
> by patterns suite/test from testing, it means that
> exclude option even excludes the suites given with
> option suite either test pattern.
> Also added ability to set the list of exclude patterns
> by the environment variable TEST_RUN_EXCLUDE_TESTS.
> 
> Close #54
> ---
> 
> Github: https://github.com/tarantool/test-run/tree/avtikhon/gh-54-exclude-option
> Issue: https://github.com/tarantool/test-run/issues/54

Re: [Tarantool-patches] [tarantool-patches] Re: [PATCH v2] Add exclude option as suite/test pattern

[Kirill Yukhin]

Hello,

On 27 окт 22:25, Alexander Turenko wrote:
> I started to simplify the code, reduce code duplication, clarification
> of texts and found that I completely rewrote the whole patch. So I sent
> it as the separate PR: https://github.com/tarantool/test-run/pull/191

Could you please post (and conduct review) the patch here?

--
Regards, Kirill Yukhin