[tarantool-patches] [PATCH v1 1/1] box: restore region after box.snapshot()

[tarantool-patches] [PATCH v1 1/1] box: restore region after box.snapshot()

[imeevma]

Before this patch region wasn't restored after box.snapshot() so
some tests fail due to "memory leak". After this patch region will
be restored using region_truncate(). It could be done by cleaning
memory in case there wasn't active transaction but it seems like
region_truncate() is more universal.

Closes #3732
---
Issue: https://github.com/tarantool/tarantool/issues/3732
Branch: https://github.com/tarantool/tarantool/tree/imeevma/gh-3732-restore_region_after_box_snapshot

 src/box/box.cc                |  2 ++
 test/engine/snapshot.result   | 35 +++++++++++++++++++++++++++++++++++
 test/engine/snapshot.test.lua | 17 +++++++++++++++++
 3 files changed, 54 insertions(+)

diff --git a/src/box/box.cc b/src/box/box.cc
index c4b4fb4..0d27aa2 100644
--- a/src/box/box.cc
+++ b/src/box/box.cc
@@ -2157,6 +2157,7 @@ box_checkpoint()
 	if (! is_box_configured)
 		return 0;
 	int rc = 0;
+	size_t used = region_used(&fiber()->gc);
 	if (box_checkpoint_is_in_progress) {
 		diag_set(ClientError, ER_CHECKPOINT_IN_PROGRESS);
 		return -1;
@@ -2181,6 +2182,7 @@ end:
 
 	latch_unlock(&schema_lock);
 	box_checkpoint_is_in_progress = false;
+	region_truncate(&fiber()->gc, used);
 	return rc;
 }
 
diff --git a/test/engine/snapshot.result b/test/engine/snapshot.result
index a5aa08d..42009c2 100644
--- a/test/engine/snapshot.result
+++ b/test/engine/snapshot.result
@@ -238,3 +238,38 @@ box.space.test.index.secondary:select{}
 box.space.test:drop()
 ---
 ...
+--
+-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
+-- One of the reason why this test failed - box.snapshot didn't
+-- free region.
+--
+fiber = require('fiber')
+---
+...
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
+---
+- 0
+...
+box.snapshot()
+---
+- ok
+...
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
+---
+- 0
+...
+box.snapshot()
+---
+- ok
+...
+box.snapshot()
+---
+- ok
+...
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
+---
+- 0
+...
diff --git a/test/engine/snapshot.test.lua b/test/engine/snapshot.test.lua
index 4bbbe40..2a32280 100644
--- a/test/engine/snapshot.test.lua
+++ b/test/engine/snapshot.test.lua
@@ -41,3 +41,20 @@ box.space.test:select{}
 box.space.test.index.primary:select{}
 box.space.test.index.secondary:select{}
 box.space.test:drop()
+
+--
+-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
+-- One of the reason why this test failed - box.snapshot didn't
+-- free region.
+--
+
+fiber = require('fiber')
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
+box.snapshot()
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
+box.snapshot()
+box.snapshot()
+-- Should be 0.
+fiber.info()[fiber.self().id()].memory.used
-- 
2.7.4

[tarantool-patches] Re: [PATCH v1 1/1] box: restore region after box.snapshot()

[Vladislav Shpilevoy]

Hi! Thanks for the patch!

On 03/11/2018 17:11, imeevma@tarantool.org wrote:
> Before this patch region wasn't restored after box.snapshot() so
> some tests fail due to "memory leak". After this patch region will
> be restored using region_truncate(). It could be done by cleaning
> memory in case there wasn't active transaction but it seems like
> region_truncate() is more universal.
> 
> Closes #3732

LGTM.

> ---
> Issue: https://github.com/tarantool/tarantool/issues/3732
> Branch: https://github.com/tarantool/tarantool/tree/imeevma/gh-3732-restore_region_after_box_snapshot
> 
>   src/box/box.cc                |  2 ++
>   test/engine/snapshot.result   | 35 +++++++++++++++++++++++++++++++++++
>   test/engine/snapshot.test.lua | 17 +++++++++++++++++
>   3 files changed, 54 insertions(+)
> 
> diff --git a/src/box/box.cc b/src/box/box.cc
> index c4b4fb4..0d27aa2 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -2157,6 +2157,7 @@ box_checkpoint()
>   	if (! is_box_configured)
>   		return 0;
>   	int rc = 0;
> +	size_t used = region_used(&fiber()->gc);
>   	if (box_checkpoint_is_in_progress) {
>   		diag_set(ClientError, ER_CHECKPOINT_IN_PROGRESS);
>   		return -1;
> @@ -2181,6 +2182,7 @@ end:
>   
>   	latch_unlock(&schema_lock);
>   	box_checkpoint_is_in_progress = false;
> +	region_truncate(&fiber()->gc, used);
>   	return rc;
>   }
>   
> diff --git a/test/engine/snapshot.result b/test/engine/snapshot.result
> index a5aa08d..42009c2 100644
> --- a/test/engine/snapshot.result
> +++ b/test/engine/snapshot.result
> @@ -238,3 +238,38 @@ box.space.test.index.secondary:select{}
>   box.space.test:drop()
>   ---
>   ...
> +--
> +-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
> +-- One of the reason why this test failed - box.snapshot didn't
> +-- free region.
> +--
> +fiber = require('fiber')
> +---
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> diff --git a/test/engine/snapshot.test.lua b/test/engine/snapshot.test.lua
> index 4bbbe40..2a32280 100644
> --- a/test/engine/snapshot.test.lua
> +++ b/test/engine/snapshot.test.lua
> @@ -41,3 +41,20 @@ box.space.test:select{}
>   box.space.test.index.primary:select{}
>   box.space.test.index.secondary:select{}
>   box.space.test:drop()
> +
> +--
> +-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
> +-- One of the reason why this test failed - box.snapshot didn't
> +-- free region.
> +--
> +
> +fiber = require('fiber')
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +box.snapshot()
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +box.snapshot()
> +box.snapshot()
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> 

Re: [tarantool-patches] [PATCH v1 1/1] box: restore region after box.snapshot()

[Vladislav Shpilevoy]

Vova, please, review.

On 03/11/2018 17:11, imeevma@tarantool.org wrote:
> Before this patch region wasn't restored after box.snapshot() so
> some tests fail due to "memory leak". After this patch region will
> be restored using region_truncate(). It could be done by cleaning
> memory in case there wasn't active transaction but it seems like
> region_truncate() is more universal.
> 
> Closes #3732
> ---
> Issue: https://github.com/tarantool/tarantool/issues/3732
> Branch: https://github.com/tarantool/tarantool/tree/imeevma/gh-3732-restore_region_after_box_snapshot
> 
>   src/box/box.cc                |  2 ++
>   test/engine/snapshot.result   | 35 +++++++++++++++++++++++++++++++++++
>   test/engine/snapshot.test.lua | 17 +++++++++++++++++
>   3 files changed, 54 insertions(+)
> 
> diff --git a/src/box/box.cc b/src/box/box.cc
> index c4b4fb4..0d27aa2 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -2157,6 +2157,7 @@ box_checkpoint()
>   	if (! is_box_configured)
>   		return 0;
>   	int rc = 0;
> +	size_t used = region_used(&fiber()->gc);
>   	if (box_checkpoint_is_in_progress) {
>   		diag_set(ClientError, ER_CHECKPOINT_IN_PROGRESS);
>   		return -1;
> @@ -2181,6 +2182,7 @@ end:
>   
>   	latch_unlock(&schema_lock);
>   	box_checkpoint_is_in_progress = false;
> +	region_truncate(&fiber()->gc, used);
>   	return rc;
>   }
>   
> diff --git a/test/engine/snapshot.result b/test/engine/snapshot.result
> index a5aa08d..42009c2 100644
> --- a/test/engine/snapshot.result
> +++ b/test/engine/snapshot.result
> @@ -238,3 +238,38 @@ box.space.test.index.secondary:select{}
>   box.space.test:drop()
>   ---
>   ...
> +--
> +-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
> +-- One of the reason why this test failed - box.snapshot didn't
> +-- free region.
> +--
> +fiber = require('fiber')
> +---
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +box.snapshot()
> +---
> +- ok
> +...
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +---
> +- 0
> +...
> diff --git a/test/engine/snapshot.test.lua b/test/engine/snapshot.test.lua
> index 4bbbe40..2a32280 100644
> --- a/test/engine/snapshot.test.lua
> +++ b/test/engine/snapshot.test.lua
> @@ -41,3 +41,20 @@ box.space.test:select{}
>   box.space.test.index.primary:select{}
>   box.space.test.index.secondary:select{}
>   box.space.test:drop()
> +
> +--
> +-- sql: fix sql/gh-3199-no-mem-leaks.test.lua
> +-- One of the reason why this test failed - box.snapshot didn't
> +-- free region.
> +--
> +
> +fiber = require('fiber')
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +box.snapshot()
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> +box.snapshot()
> +box.snapshot()
> +-- Should be 0.
> +fiber.info()[fiber.self().id()].memory.used
> 

Re: [tarantool-patches] [PATCH v1 1/1] box: restore region after box.snapshot()

[Vladimir Davydov]

On Sat, Nov 03, 2018 at 05:11:42PM +0300, imeevma@tarantool.org wrote:
> Before this patch region wasn't restored after box.snapshot() so
> some tests fail due to "memory leak". After this patch region will
> be restored using region_truncate(). It could be done by cleaning
> memory in case there wasn't active transaction but it seems like
> region_truncate() is more universal.
> 
> Closes #3732
> ---
> Issue: https://github.com/tarantool/tarantool/issues/3732
> Branch: https://github.com/tarantool/tarantool/tree/imeevma/gh-3732-restore_region_after_box_snapshot
> 
>  src/box/box.cc                |  2 ++
>  test/engine/snapshot.result   | 35 +++++++++++++++++++++++++++++++++++
>  test/engine/snapshot.test.lua | 17 +++++++++++++++++
>  3 files changed, 54 insertions(+)
> 
> diff --git a/src/box/box.cc b/src/box/box.cc
> index c4b4fb4..0d27aa2 100644
> --- a/src/box/box.cc
> +++ b/src/box/box.cc
> @@ -2157,6 +2157,7 @@ box_checkpoint()
>  	if (! is_box_configured)
>  		return 0;
>  	int rc = 0;
> +	size_t used = region_used(&fiber()->gc);
>  	if (box_checkpoint_is_in_progress) {
>  		diag_set(ClientError, ER_CHECKPOINT_IN_PROGRESS);
>  		return -1;
> @@ -2181,6 +2182,7 @@ end:
>  
>  	latch_unlock(&schema_lock);
>  	box_checkpoint_is_in_progress = false;
> +	region_truncate(&fiber()->gc, used);
>  	return rc;
>  }

IMHO this looks confusing, because we typically do region_truncate only
in functions/modules that actually allocate something on the region.
Here one has to go much deeper to understand what's going on (I assume
it's memtx_engine_begin_checkpoint) so chances are high somebody will
try to delete this useless code sooner or later. We could probably add a
comment here to clear things up, but if memtx_engine_begin_checkpoint is
changed one day, we will definitely forget to remove the comment that
will become obsolete.

I'd prefer if you did region_truncate where objects are actually
allocated or not use the region allocator at all - it doesn't seem to be
a hot path after all.