[Tarantool-patches] [PATCH 0/2] box.cfg() check memtx/vinyl_memory

[Tarantool-patches] [PATCH 0/2] box.cfg() check memtx/vinyl_memory

[Vladislav Shpilevoy]

The patchset fixes
1) forgotten diag_raise() for sql cache size check;
2) assertion when memtx_memory is too big.

Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-4705-memtx-memory-too-big
Issue: https://github.com/tarantool/tarantool/issues/4705

@ChangeLog
- It is not possible to set memtx_memory and vinyl_memory more
  than 4398046510080 bytes now (gh-4705).

Vladislav Shpilevoy (2):
  box: fail in box_check_config() on bad sql cache size
  box: on cfg properly check memory quota

 src/box/box.cc        | 49 +++++++++++++++++++++----------------------
 test/box/cfg.result   | 19 +++++++++++++++--
 test/box/cfg.test.lua |  5 +++++
 3 files changed, 46 insertions(+), 27 deletions(-)

-- 
2.21.1 (Apple Git-122.3)

[Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Vladislav Shpilevoy]

It was calling box_check_sql_cache_size() assuming that it
throws. But it returns 0/-1.
---
 src/box/box.cc | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/box/box.cc b/src/box/box.cc
index 09dd67ab4..0212f34ad 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -669,7 +669,8 @@ box_check_config()
 	box_check_memtx_memory(cfg_geti64("memtx_memory"));
 	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
 	box_check_vinyl_options();
-	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
+	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
+		diag_raise();
 }
 
 /*
-- 
2.21.1 (Apple Git-122.3)

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Nikita Pettik]

On 05 Mar 01:14, Vladislav Shpilevoy wrote:
> It was calling box_check_sql_cache_size() assuming that it
> throws. But it returns 0/-1.
> ---
>  src/box/box.cc | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/src/box/box.cc b/src/box/box.cc
> index 09dd67ab4..0212f34ad 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -669,7 +669,8 @@ box_check_config()
>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>  	box_check_vinyl_options();
> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
> +		diag_raise();
>  }

Is it possible to test this scenario: box.sql_cache_size is set to
incorrect value but box.internal:cfg_check() doesn't verify it?
I mean supply this patch with test case. If it turns out to be
too complicated, then nevermind. LGTM otherwise.
 
>  /*
> -- 
> 2.21.1 (Apple Git-122.3)
> 

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Vladislav Shpilevoy]

Hi! Thanks for the review!

On 05/03/2020 12:35, Nikita Pettik wrote:
> On 05 Mar 01:14, Vladislav Shpilevoy wrote:
>> It was calling box_check_sql_cache_size() assuming that it
>> throws. But it returns 0/-1.
>> ---
>>  src/box/box.cc | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/box/box.cc b/src/box/box.cc
>> index 09dd67ab4..0212f34ad 100644
>> --- a/src/box/box.cc
>> +++ b/src/box/box.cc
>> @@ -669,7 +669,8 @@ box_check_config()
>>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>>  	box_check_vinyl_options();
>> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
>> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
>> +		diag_raise();
>>  }
> 
> Is it possible to test this scenario: box.sql_cache_size is set to
> incorrect value but box.internal:cfg_check() doesn't verify it?
> I mean supply this patch with test case. If it turns out to be
> too complicated, then nevermind. LGTM otherwise.

It is not testable. Sql_cache_size is always set from
box_set_prepared_stmt_cache_size(), on first and all
next configurations, and is always checked for returned
value.

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Nikita Pettik]

On 05 Mar 21:53, Vladislav Shpilevoy wrote:
> Hi! Thanks for the review!
> 
> On 05/03/2020 12:35, Nikita Pettik wrote:
> > On 05 Mar 01:14, Vladislav Shpilevoy wrote:
> >> It was calling box_check_sql_cache_size() assuming that it
> >> throws. But it returns 0/-1.
> >> ---
> >>  src/box/box.cc | 3 ++-
> >>  1 file changed, 2 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/src/box/box.cc b/src/box/box.cc
> >> index 09dd67ab4..0212f34ad 100644
> >> --- a/src/box/box.cc
> >> +++ b/src/box/box.cc
> >> @@ -669,7 +669,8 @@ box_check_config()
> >>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
> >>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
> >>  	box_check_vinyl_options();
> >> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
> >> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
> >> +		diag_raise();
> >>  }
> > 
> > Is it possible to test this scenario: box.sql_cache_size is set to
> > incorrect value but box.internal:cfg_check() doesn't verify it?
> > I mean supply this patch with test case. If it turns out to be
> > too complicated, then nevermind. LGTM otherwise.
> 
> It is not testable. Sql_cache_size is always set from
> box_set_prepared_stmt_cache_size(), on first and all
> next configurations, and is always checked for returned
> value.

Ok, then mb it would be better to check rc in assertion
instead of diag_raise().. (since it can't fail here anyway)?

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Vladislav Shpilevoy]

On 05/03/2020 22:00, Nikita Pettik wrote:
> On 05 Mar 21:53, Vladislav Shpilevoy wrote:
>> Hi! Thanks for the review!
>>
>> On 05/03/2020 12:35, Nikita Pettik wrote:
>>> On 05 Mar 01:14, Vladislav Shpilevoy wrote:
>>>> It was calling box_check_sql_cache_size() assuming that it
>>>> throws. But it returns 0/-1.
>>>> ---
>>>>  src/box/box.cc | 3 ++-
>>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/src/box/box.cc b/src/box/box.cc
>>>> index 09dd67ab4..0212f34ad 100644
>>>> --- a/src/box/box.cc
>>>> +++ b/src/box/box.cc
>>>> @@ -669,7 +669,8 @@ box_check_config()
>>>>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>>>>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>>>>  	box_check_vinyl_options();
>>>> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
>>>> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
>>>> +		diag_raise();
>>>>  }
>>>
>>> Is it possible to test this scenario: box.sql_cache_size is set to
>>> incorrect value but box.internal:cfg_check() doesn't verify it?
>>> I mean supply this patch with test case. If it turns out to be
>>> too complicated, then nevermind. LGTM otherwise.
>>
>> It is not testable. Sql_cache_size is always set from
>> box_set_prepared_stmt_cache_size(), on first and all
>> next configurations, and is always checked for returned
>> value.
> 
> Ok, then mb it would be better to check rc in assertion
> instead of diag_raise().. (since it can't fail here anyway)?
> 

It can fail in box_check_config(). But the error won't be different
from checking it later during set.

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Igor Munkin]

Vlad,

Thanks for the patch, LGTM.

On 05.03.20, Vladislav Shpilevoy wrote:
> It was calling box_check_sql_cache_size() assuming that it
> throws. But it returns 0/-1.
> ---
>  src/box/box.cc | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/src/box/box.cc b/src/box/box.cc
> index 09dd67ab4..0212f34ad 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -669,7 +669,8 @@ box_check_config()
>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>  	box_check_vinyl_options();
> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)

Side note: I'm just curious what are the rationale to use '!= 0' here
and '< 0' nearby in the following patch.

> +		diag_raise();
>  }
>  
>  /*
> -- 
> 2.21.1 (Apple Git-122.3)
> 

-- 
Best regards,
IM

Re: [Tarantool-patches] [PATCH 1/2] box: fail in box_check_config() on bad sql cache size

[Vladislav Shpilevoy]

Thanks for the review!

On 19/03/2020 16:27, Igor Munkin wrote:
> Vlad,
> 
> Thanks for the patch, LGTM.
> 
> On 05.03.20, Vladislav Shpilevoy wrote:
>> It was calling box_check_sql_cache_size() assuming that it
>> throws. But it returns 0/-1.
>> ---
>>  src/box/box.cc | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/box/box.cc b/src/box/box.cc
>> index 09dd67ab4..0212f34ad 100644
>> --- a/src/box/box.cc
>> +++ b/src/box/box.cc
>> @@ -669,7 +669,8 @@ box_check_config()
>>  	box_check_memtx_memory(cfg_geti64("memtx_memory"));
>>  	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
>>  	box_check_vinyl_options();
>> -	box_check_sql_cache_size(cfg_geti("sql_cache_size"));
>> +	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
> 
> Side note: I'm just curious what are the rationale to use '!= 0' here
> and '< 0' nearby in the following patch.

box_check_sql_cache_size() returns 0/-1.

box_check_memory_quota() returns -1/size.

So for 0/-1 I used a more or less commonly used way of
checking for an error when a function returns these 2 values.

For box_check_memory_quota() I used < 0, because != 0
does not mean an error here.

> 
>> +		diag_raise();
>>  }
>>  
>>  /*
>> -- 
>> 2.21.1 (Apple Git-122.3)
>>
> 

[Tarantool-patches] [PATCH 2/2] box: on cfg properly check memory quota

[Vladislav Shpilevoy]

box_check_config() didn't check memtx_memory and vinyl_memory
upper bound. As a result, it was possible to set memory size
higher than what the quota allows as maximum.

That worked only when box.cfg() was called first time, because
quota_init() does not check its value. Subsequent box.cfg() calls
use quota_set(), which aborts the program if a size is too big.
Only in debug mode. In release quota_set() also worked with any
sizes.

Closes #4705
---
 src/box/box.cc        | 46 +++++++++++++++++++++----------------------
 test/box/cfg.result   | 19 ++++++++++++++++--
 test/box/cfg.test.lua |  5 +++++
 3 files changed, 44 insertions(+), 26 deletions(-)

diff --git a/src/box/box.cc b/src/box/box.cc
index 0212f34ad..9045cefe4 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -577,24 +577,16 @@ box_check_wal_max_size(int64_t wal_max_size)
 	return wal_max_size;
 }
 
-static int64_t
-box_check_memtx_memory(int64_t memory)
-{
-	if (memory < 0) {
-		tnt_raise(ClientError, ER_CFG, "memtx_memory",
-			  "must not be less than 0");
-	}
-	return memory;
-}
-
-static int64_t
-box_check_vinyl_memory(int64_t memory)
-{
-	if (memory < 0) {
-		tnt_raise(ClientError, ER_CFG, "vinyl_memory",
-			  "must not be less than 0");
-	}
-	return memory;
+static ssize_t
+box_check_memory_quota(const char *quota_name)
+{
+	int64_t size = cfg_geti64(quota_name);
+	if (size >= 0 && (size_t) size <= QUOTA_MAX)
+		return size;
+	diag_set(ClientError, ER_CFG, quota_name,
+		 tt_sprintf("must be >= 0 and <= %zu, but it is %lld",
+		 QUOTA_MAX, size));
+	return -1;
 }
 
 static void
@@ -608,7 +600,8 @@ box_check_vinyl_options(void)
 	double run_size_ratio = cfg_getd("vinyl_run_size_ratio");
 	double bloom_fpr = cfg_getd("vinyl_bloom_fpr");
 
-	box_check_vinyl_memory(cfg_geti64("vinyl_memory"));
+	if (box_check_memory_quota("vinyl_memory") < 0)
+		diag_raise();
 
 	if (read_threads < 1) {
 		tnt_raise(ClientError, ER_CFG, "vinyl_read_threads",
@@ -666,7 +659,8 @@ box_check_config()
 	box_check_checkpoint_count(cfg_geti("checkpoint_count"));
 	box_check_wal_max_size(cfg_geti64("wal_max_size"));
 	box_check_wal_mode(cfg_gets("wal_mode"));
-	box_check_memtx_memory(cfg_geti64("memtx_memory"));
+	if (box_check_memory_quota("memtx_memory") < 0)
+		diag_raise();
 	box_check_memtx_min_tuple_size(cfg_geti64("memtx_min_tuple_size"));
 	box_check_vinyl_options();
 	if (box_check_sql_cache_size(cfg_geti("sql_cache_size")) != 0)
@@ -895,8 +889,10 @@ box_set_memtx_memory(void)
 	struct memtx_engine *memtx;
 	memtx = (struct memtx_engine *)engine_by_name("memtx");
 	assert(memtx != NULL);
-	memtx_engine_set_memory_xc(memtx,
-		box_check_memtx_memory(cfg_geti64("memtx_memory")));
+	ssize_t size = box_check_memory_quota("memtx_memory");
+	if (size < 0)
+		diag_raise();
+	memtx_engine_set_memory_xc(memtx, size);
 }
 
 void
@@ -954,8 +950,10 @@ box_set_vinyl_memory(void)
 {
 	struct engine *vinyl = engine_by_name("vinyl");
 	assert(vinyl != NULL);
-	vinyl_engine_set_memory_xc(vinyl,
-		box_check_vinyl_memory(cfg_geti64("vinyl_memory")));
+	ssize_t size = box_check_memory_quota("vinyl_memory");
+	if (size < 0)
+		diag_raise();
+	vinyl_engine_set_memory_xc(vinyl, size);
 }
 
 void
diff --git a/test/box/cfg.result b/test/box/cfg.result
index 8024b5516..9dd417c4f 100644
--- a/test/box/cfg.result
+++ b/test/box/cfg.result
@@ -254,11 +254,13 @@ box.cfg{memtx_memory = "100500"}
  | ...
 box.cfg{memtx_memory = -1}
  | ---
- | - error: 'Incorrect value for option ''memtx_memory'': must not be less than 0'
+ | - error: 'Incorrect value for option ''memtx_memory'': must be >= 0 and <= 4398046510080,
+ |     but it is -1'
  | ...
 box.cfg{vinyl_memory = -1}
  | ---
- | - error: 'Incorrect value for option ''vinyl_memory'': must not be less than 0'
+ | - error: 'Incorrect value for option ''vinyl_memory'': must be >= 0 and <= 4398046510080,
+ |     but it is -1'
  | ...
 box.cfg{vinyl = "vinyl"}
  | ---
@@ -268,6 +270,19 @@ box.cfg{vinyl_write_threads = "threads"}
  | ---
  | - error: 'Incorrect value for option ''vinyl_write_threads'': should be of type number'
  | ...
+--
+-- gh-4705: too big memory size led to an assertion.
+--
+box.cfg{memtx_memory = 5000000000000}
+ | ---
+ | - error: 'Incorrect value for option ''memtx_memory'': must be >= 0 and <= 4398046510080,
+ |     but it is 5000000000000'
+ | ...
+box.cfg{vinyl_memory = 5000000000000}
+ | ---
+ | - error: 'Incorrect value for option ''vinyl_memory'': must be >= 0 and <= 4398046510080,
+ |     but it is 5000000000000'
+ | ...
 
 --------------------------------------------------------------------------------
 -- Dynamic configuration check
diff --git a/test/box/cfg.test.lua b/test/box/cfg.test.lua
index e6a90d770..875466a25 100644
--- a/test/box/cfg.test.lua
+++ b/test/box/cfg.test.lua
@@ -28,6 +28,11 @@ box.cfg{memtx_memory = -1}
 box.cfg{vinyl_memory = -1}
 box.cfg{vinyl = "vinyl"}
 box.cfg{vinyl_write_threads = "threads"}
+--
+-- gh-4705: too big memory size led to an assertion.
+--
+box.cfg{memtx_memory = 5000000000000}
+box.cfg{vinyl_memory = 5000000000000}
 
 --------------------------------------------------------------------------------
 -- Dynamic configuration check
-- 
2.21.1 (Apple Git-122.3)

Re: [Tarantool-patches] [PATCH 2/2] box: on cfg properly check memory quota

[Nikita Pettik]

On 05 Mar 01:14, Vladislav Shpilevoy wrote:
> box_check_config() didn't check memtx_memory and vinyl_memory
> upper bound. As a result, it was possible to set memory size
> higher than what the quota allows as maximum.
> 
> That worked only when box.cfg() was called first time, because
> quota_init() does not check its value. Subsequent box.cfg() calls
> use quota_set(), which aborts the program if a size is too big.
> Only in debug mode. In release quota_set() also worked with any
> sizes.
> 
> Closes #4705
> ---

LGTM.

Re: [Tarantool-patches] [PATCH 0/2] box.cfg() check memtx/vinyl_memory

[Nikita Pettik]

On 05 Mar 01:14, Vladislav Shpilevoy wrote:
> The patchset fixes
> 1) forgotten diag_raise() for sql cache size check;
> 2) assertion when memtx_memory is too big.
> 
> Branch: http://github.com/tarantool/tarantool/tree/gerold103/gh-4705-memtx-memory-too-big
> Issue: https://github.com/tarantool/tarantool/issues/4705
> 
> @ChangeLog
> - It is not possible to set memtx_memory and vinyl_memory more
>   than 4398046510080 bytes now (gh-4705).

Pushed first patch to master and 2.3, second one to master, 2.3 and 2.2.
Updated changelogs correspondingly. Branch is dropped.
 
> Vladislav Shpilevoy (2):
>   box: fail in box_check_config() on bad sql cache size
>   box: on cfg properly check memory quota
> 
>  src/box/box.cc        | 49 +++++++++++++++++++++----------------------
>  test/box/cfg.result   | 19 +++++++++++++++--
>  test/box/cfg.test.lua |  5 +++++
>  3 files changed, 46 insertions(+), 27 deletions(-)
> 
> -- 
> 2.21.1 (Apple Git-122.3)
>