[tarantool-patches] [PATCH 1/1] Allow to ignore space formats via command line option

[tarantool-patches] [PATCH 1/1] Allow to ignore space formats via command line option

[Vladislav Shpilevoy]

It is a common case that an instance is running on a version
1.6.*, then is upgraded to 1.9 with box.schema.upgrade(). But
some of users has malformed space formats, and some of them got
and ignored errors on box.schema.upgrade(). Such half-upgraded
data can not be used to start a new version due to format
violations, and can not be rolled back because some of new system
spaces managed to be created. And we can not repair raw xlogs and
snapshots because we have no such tools and they are zipped.

So it would be useful to have a command line option
--ignore-space-formats. It would allow to start an instance on
the spaces with malformed formats, fix them, create a snapshot
and then start with the formats turned on.

Closes #3605

@TarantoolBot document
Title: Command line option '--ignore-space-formats'
The option allows to turn off space formats validation before
the instance is started. Usefull to fix malformed formats after
an upgrade from version < 1.7.5 to >= 1.7.5.
---
Issue: https://github.com/tarantool/tarantool/issues/3605
Branch: https://github.com/tarantool/tarantool/tree/gerold103/gh-3605-ignore-space-formats-cmd

 src/box/alter.cc                     | 14 +++++++++++++-
 src/box/box.h                        |  1 +
 src/main.cc                          |  6 ++++++
 test/app-tap/ignore_formats.test.lua | 19 +++++++++++++++++++
 4 files changed, 39 insertions(+), 1 deletion(-)
 create mode 100755 test/app-tap/ignore_formats.test.lua

diff --git a/src/box/alter.cc b/src/box/alter.cc
index ec3b91bff..b379dadfa 100644
--- a/src/box/alter.cc
+++ b/src/box/alter.cc
@@ -53,6 +53,18 @@
 #include "version.h"
 #include "sequence.h"
 
+/**
+ * A flag to ignore space formats and do not validate tuples by
+ * them.
+ */
+static bool ignore_space_formats = false;
+
+void
+box_set_ignore_space_formats(bool value)
+{
+	ignore_space_formats = value;
+}
+
 /**
  * chap-sha1 of empty string, i.e.
  * base64_encode(sha1(sha1(""), 0)
@@ -533,7 +545,7 @@ space_def_new_from_tuple(struct tuple *tuple, uint32_t errcode,
 	const char *space_opts;
 	struct field_def *fields;
 	uint32_t field_count;
-	if (dd_version_id >= version_id(1, 7, 6)) {
+	if (dd_version_id >= version_id(1, 7, 6) && !ignore_space_formats) {
 		/* Check space opts. */
 		space_opts =
 			tuple_field_with_type_xc(tuple, BOX_SPACE_FIELD_OPTS,
diff --git a/src/box/box.h b/src/box/box.h
index 9dfb3fd2a..d4043db1f 100644
--- a/src/box/box.h
+++ b/src/box/box.h
@@ -176,6 +176,7 @@ void box_set_vinyl_timeout(void);
 void box_set_replication_timeout(void);
 void box_set_replication_connect_timeout(void);
 void box_set_replication_connect_quorum(void);
+void box_set_ignore_space_formats(bool value);
 
 extern "C" {
 #endif /* defined(__cplusplus) */
diff --git a/src/main.cc b/src/main.cc
index 1682baea0..e5d7d8af2 100644
--- a/src/main.cc
+++ b/src/main.cc
@@ -637,6 +637,7 @@ main(int argc, char **argv)
 	static struct option longopts[] = {
 		{"help", no_argument, 0, 'h'},
 		{"version", no_argument, 0, 'v'},
+		{"ignore-space-formats", no_argument, 0, 'f'},
 		{NULL, 0, 0, 0},
 	};
 	static const char *opts = "+hVvie:l:";
@@ -670,6 +671,11 @@ main(int argc, char **argv)
 			optv[optc++] = argv[optind - 2];
 			optv[optc++] = argv[optind - 1];
 			break;
+		case 'f':
+			/* Ignore space formats. */
+			box_set_ignore_space_formats(true);
+			say_info("Space formats validation is turned off");
+			break;
 		default:
 			/* "invalid option" is printed by getopt */
 			return EX_USAGE;
diff --git a/test/app-tap/ignore_formats.test.lua b/test/app-tap/ignore_formats.test.lua
new file mode 100755
index 000000000..a9d1bab0c
--- /dev/null
+++ b/test/app-tap/ignore_formats.test.lua
@@ -0,0 +1,19 @@
+#!/usr/bin/env tarantool --ignore-space-formats
+--
+-- gh-3605: allow to ignore space formats via command line
+-- arguments.
+--
+local tap = require('tap')
+local test = tap.test("ignore_formats")
+test:plan(1)
+
+box.cfg{}
+
+local format = {}
+format[1] = {'field1', 'unsigned'}
+format[2] = {'field2', 'unsigned'}
+local s = box.schema.create_space('test', {format = format})
+local pk = s:create_index('pk')
+test:ok(pcall(s.replace, s, {1, 'string'}), 'can violate format')
+s:drop()
+os.exit(test:check() == true and 0 or 1)
-- 
2.15.2 (Apple Git-101.1)

Re: [tarantool-patches] [PATCH 1/1] Allow to ignore space formats via command line option

[Vladimir Davydov]

On Fri, Aug 03, 2018 at 03:46:26PM +0300, Vladislav Shpilevoy wrote:
> It is a common case that an instance is running on a version
> 1.6.*, then is upgraded to 1.9 with box.schema.upgrade(). But
> some of users has malformed space formats, and some of them got
> and ignored errors on box.schema.upgrade(). Such half-upgraded
> data can not be used to start a new version due to format
> violations, and can not be rolled back because some of new system
> spaces managed to be created. And we can not repair raw xlogs and
> snapshots because we have no such tools and they are zipped.
> 
> So it would be useful to have a command line option
> --ignore-space-formats. It would allow to start an instance on
> the spaces with malformed formats, fix them, create a snapshot
> and then start with the formats turned on.
> 
> Closes #3605
> 
> @TarantoolBot document
> Title: Command line option '--ignore-space-formats'
> The option allows to turn off space formats validation before
> the instance is started. Usefull to fix malformed formats after
> an upgrade from version < 1.7.5 to >= 1.7.5.

Why a command line option? Why not a box.cfg parameter?
Or, even better, box.cfg.force_recovery?

Re: [tarantool-patches] Re: [PATCH 1/1] Allow to ignore space formats via command line option

[Vladislav Shpilevoy]

On 03/08/2018 16:47, Vladimir Davydov wrote:
> On Fri, Aug 03, 2018 at 03:46:26PM +0300, Vladislav Shpilevoy wrote:
>> It is a common case that an instance is running on a version
>> 1.6.*, then is upgraded to 1.9 with box.schema.upgrade(). But
>> some of users has malformed space formats, and some of them got
>> and ignored errors on box.schema.upgrade(). Such half-upgraded
>> data can not be used to start a new version due to format
>> violations, and can not be rolled back because some of new system
>> spaces managed to be created. And we can not repair raw xlogs and
>> snapshots because we have no such tools and they are zipped.
>>
>> So it would be useful to have a command line option
>> --ignore-space-formats. It would allow to start an instance on
>> the spaces with malformed formats, fix them, create a snapshot
>> and then start with the formats turned on.
>>
>> Closes #3605
>>
>> @TarantoolBot document
>> Title: Command line option '--ignore-space-formats'
>> The option allows to turn off space formats validation before
>> the instance is started. Usefull to fix malformed formats after
>> an upgrade from version < 1.7.5 to >= 1.7.5.
> 
> Why a command line option? Why not a box.cfg parameter?
> Or, even better, box.cfg.force_recovery?
> 

I just do not want to clog box.cfg options with this
pure debug one. Force_recovery should skip violating
records instead of ignoring the format. But I don't
mind, for me all the 3 ways are ok and helps me to
repair customers with broken schemes. So if you want,
I can add a box.cfg option or reuse force_recovery
to always ignore space formats. What should I do?

Re: [tarantool-patches] Re: [PATCH 1/1] Allow to ignore space formats via command line option

[Kirill Yukhin]

Hello Vlad,
On 03 авг 16:58, Vladislav Shpilevoy wrote:
> On 03/08/2018 16:47, Vladimir Davydov wrote:
> > On Fri, Aug 03, 2018 at 03:46:26PM +0300, Vladislav Shpilevoy wrote:
> > > It is a common case that an instance is running on a version
> > > 1.6.*, then is upgraded to 1.9 with box.schema.upgrade(). But
> > > some of users has malformed space formats, and some of them got
> > > and ignored errors on box.schema.upgrade(). Such half-upgraded
> > > data can not be used to start a new version due to format
> > > violations, and can not be rolled back because some of new system
> > > spaces managed to be created. And we can not repair raw xlogs and
> > > snapshots because we have no such tools and they are zipped.
> > > 
> > > So it would be useful to have a command line option
> > > --ignore-space-formats. It would allow to start an instance on
> > > the spaces with malformed formats, fix them, create a snapshot
> > > and then start with the formats turned on.
> > > 
> > > Closes #3605
> > > 
> > > @TarantoolBot document
> > > Title: Command line option '--ignore-space-formats'
> > > The option allows to turn off space formats validation before
> > > the instance is started. Usefull to fix malformed formats after
> > > an upgrade from version < 1.7.5 to >= 1.7.5.
> > 
> > Why a command line option? Why not a box.cfg parameter?
> > Or, even better, box.cfg.force_recovery?
> > 
> 
> I just do not want to clog box.cfg options with this
> pure debug one. Force_recovery should skip violating
> records instead of ignoring the format. But I don't
> mind, for me all the 3 ways are ok and helps me to
> repair customers with broken schemes. So if you want,
> I can add a box.cfg option or reuse force_recovery
> to always ignore space formats. What should I do?
I think adding box.cfg{} parameter is more flexible approach.
You can start w/ such option turned ON, and the turn it OFF
w/o restart of whole instance.

--
Regards, Kirill Yukhin