From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <alexander.turenko@tarantool.org>
Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by dev.tarantool.org (Postfix) with ESMTPS id 25449469710
 for <tarantool-patches@dev.tarantool.org>;
 Tue,  9 Jun 2020 17:36:59 +0300 (MSK)
Date: Tue, 9 Jun 2020 17:36:44 +0300
From: Alexander Turenko <alexander.turenko@tarantool.org>
Message-ID: <20200609143644.tyeh277rb5cwmv5t@tkn_work_nb>
References: <cover.1591629414.git.sergeyb@tarantool.org>
 <37fb3b360327ef0f93d5074153adda9e61eda697.1591629414.git.sergeyb@tarantool.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <37fb3b360327ef0f93d5074153adda9e61eda697.1591629414.git.sergeyb@tarantool.org>
Subject: Re: [Tarantool-patches] [PATCH v3 2/2] Add option to update file
	with reference output
List-Id: Tarantool development patches <tarantool-patches.dev.tarantool.org>
List-Unsubscribe: <https://lists.tarantool.org/mailman/options/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=unsubscribe>
List-Archive: <https://lists.tarantool.org/pipermail/tarantool-patches/>
List-Post: <mailto:tarantool-patches@dev.tarantool.org>
List-Help: <mailto:tarantool-patches-request@dev.tarantool.org?subject=help>
List-Subscribe: <https://lists.tarantool.org/mailman/listinfo/tarantool-patches>, 
 <mailto:tarantool-patches-request@dev.tarantool.org?subject=subscribe>
To: sergeyb@tarantool.org
Cc: tarantool-patches@dev.tarantool.org

Made several changes, see below.

Pushed the patchset to test-run's master.

Updated the submodule in tarantool and pushed the update to master, 2.4,
2.3 and 1.10 branches.

WBR, Alexander Turenko.

> -        elif (self.is_executed_ok and not
> -              self.is_equal_result and not
> -              os.path.isfile(self.result)) and not is_tap:
> +        elif (self.is_executed_ok and
> +              not self.is_equal_result and
> +              not os.path.isfile(self.result) and not is_tap):
>              shutil.copy(self.tmp_result, self.result)
>              short_status = 'new'
>              color_stdout("[ new ]\n", schema='test_new')

We should write new result files only under --update-result. Added the check
for the option:

 |                  os.remove(self.tmp_result)
 |          elif (self.is_executed_ok and
 |                not self.is_equal_result and
 | -              not os.path.isfile(self.result) and not is_tap):
 | +              not os.path.isfile(self.result) and
 | +              not is_tap and
 | +              lib.Options().args.update_result):
 |              shutil.copy(self.tmp_result, self.result)
 |              short_status = 'new'
 |              color_stdout("[ new ]\n", schema='test_new')

> +        elif (self.is_executed_ok and
> +              not self.is_equal_result and
> +              lib.Options().args.update_result):
> +            shutil.copy(self.tmp_result, self.result)
> +            short_status = 'updated'
> +            color_stdout("[ updated ]\n", schema='test_new')

It will write a new result file for a failed TAP13 test under the
option. I guess it is not the expected behaviour in most of cases. Say,
I changed something in the net.box implementation and want to update
result files for all related tests in a batch (`./test-run.py
--update-result net.box`), but it should not write a result file (and
report successful testing) if some TAP13 test fails. Added the check:

 |              color_stdout("[ new ]\n", schema='test_new')
 |          elif (self.is_executed_ok and
 |                not self.is_equal_result and
 | +              os.path.isfile(self.result) and
 | +              not is_tap and
 |                lib.Options().args.update_result):
 |              shutil.copy(self.tmp_result, self.result)
 |              short_status = 'updated'