[PATCH v4 0/2] lib/core/fiber: Increase default stack size

[PATCH v4 0/2] lib/core/fiber: Increase default stack size

[Cyrill Gorcunov]

v4:
  We split the stack management task into two: first one to
  increase stack size asap, to not block users and testers,
  and second one to rework stack management to relax rss
  pressue.

  This patchset covers the first task only. The second part
  is still under rfc and hopefully won't take too long.

Cyrill Gorcunov (2):
  lib/core/fiber: Increase default stack size
  lib/core/fiber: Mark stack as unneeded on creation

 src/lib/core/fiber.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

-- 
2.20.1

[PATCH 1/2] lib/core/fiber: Increase default stack size

[Cyrill Gorcunov]

The default 64K stack size used for years become too small
for modern distors (Fedora 29 and etc) where third party libraries
(such as ncurses) started to use 64K for own buffers and we get
SIGSGV early without reaching interactive console phase.

Thus we increase default size up to 512K which should fit
for common case. Later we will make this value configurable
to address arbitrary stack sizes without a need to rebuild
the whole code.

Note the values are switched to 4K page granularity for sake
of future modifications -- we gonna manipulate pages to
relax rss usage if OS allows.

Closes #3418
---
 src/lib/core/fiber.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
index abd6c6b11..f16ac873f 100644
--- a/src/lib/core/fiber.c
+++ b/src/lib/core/fiber.c
@@ -93,9 +93,9 @@ static int stack_direction;
 
 enum {
 	/* The minimum allowable fiber stack size in bytes */
-	FIBER_STACK_SIZE_MINIMAL = 16384,
+	FIBER_STACK_SIZE_MINIMAL = 4 << 12,
 	/* Default fiber stack size in bytes */
-	FIBER_STACK_SIZE_DEFAULT = 65536
+	FIBER_STACK_SIZE_DEFAULT = 128 << 12
 };
 
 /** Default fiber attributes */
-- 
2.20.1

[PATCH 2/2] lib/core/fiber: Mark stack as unneeded on creation

[Cyrill Gorcunov]

Since we've increased the default stack size we hope
the whole 512K won't be used for regular loads thus
we mark the stack area as unneeded to minimize rss
pressure.

Note we do this on fiber creation at the moment, more
detailed stack usage analisys will be implemented in
another patch series later.

Closes #3418
---
 src/lib/core/fiber.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
index f16ac873f..a1b261ad4 100644
--- a/src/lib/core/fiber.c
+++ b/src/lib/core/fiber.c
@@ -749,6 +749,14 @@ fiber_stack_create(struct fiber *fiber, size_t stack_size)
 	fiber->stack_id = VALGRIND_STACK_REGISTER(fiber->stack,
 						  (char *)fiber->stack +
 						  fiber->stack_size);
+#ifndef TARGET_OS_DARWIN
+	/*
+	 * We don't expect the whole stack usage in regular
+	 * loads, lets try to minimize rss pressure if OS
+	 * allows us.
+	 */
+	madvise(fiber->stack, fiber->stack_size, MADV_DONTNEED);
+#endif
 
 	mprotect(guard, page_size, PROT_NONE);
 	return 0;
-- 
2.20.1

Re: [PATCH 2/2] lib/core/fiber: Mark stack as unneeded on creation

[Vladimir Davydov]

On Fri, Mar 08, 2019 at 12:31:46AM +0300, Cyrill Gorcunov wrote:
> Since we've increased the default stack size we hope
> the whole 512K won't be used for regular loads thus
> we mark the stack area as unneeded to minimize rss
> pressure.
> 
> Note we do this on fiber creation at the moment, more
> detailed stack usage analisys will be implemented in
> another patch series later.
> 
> Closes #3418
> ---
>  src/lib/core/fiber.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
> index f16ac873f..a1b261ad4 100644
> --- a/src/lib/core/fiber.c
> +++ b/src/lib/core/fiber.c
> @@ -749,6 +749,14 @@ fiber_stack_create(struct fiber *fiber, size_t stack_size)
>  	fiber->stack_id = VALGRIND_STACK_REGISTER(fiber->stack,
>  						  (char *)fiber->stack +
>  						  fiber->stack_size);
> +#ifndef TARGET_OS_DARWIN
> +	/*
> +	 * We don't expect the whole stack usage in regular
> +	 * loads, lets try to minimize rss pressure if OS
> +	 * allows us.
> +	 */
> +	madvise(fiber->stack, fiber->stack_size, MADV_DONTNEED);
> +#endif

madvise on fiber creation doesn't make much sense without madvise on
fiber recycling, because fat tasks will quickly fault-in that memory.
I thought you were going to implement stack shrinking to some predefined
size (say, allocate 1 MB max, shrink to 64 KB on recycle if watermark
was overwritten), leaving only dynamic stack sizing for the future (i.e.
estimating the optimal size to do madvise for basing on average fiber
stack usage).

Re: [PATCH 2/2] lib/core/fiber: Mark stack as unneeded on creation

[Cyrill Gorcunov]

On Mon, Mar 11, 2019 at 11:18:14AM +0300, Vladimir Davydov wrote:
> > diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
> > index f16ac873f..a1b261ad4 100644
> > --- a/src/lib/core/fiber.c
> > +++ b/src/lib/core/fiber.c
> > @@ -749,6 +749,14 @@ fiber_stack_create(struct fiber *fiber, size_t stack_size)
> >  	fiber->stack_id = VALGRIND_STACK_REGISTER(fiber->stack,
> >  						  (char *)fiber->stack +
> >  						  fiber->stack_size);
> > +#ifndef TARGET_OS_DARWIN
> > +	/*
> > +	 * We don't expect the whole stack usage in regular
> > +	 * loads, lets try to minimize rss pressure if OS
> > +	 * allows us.
> > +	 */
> > +	madvise(fiber->stack, fiber->stack_size, MADV_DONTNEED);
> > +#endif
> 
> madvise on fiber creation doesn't make much sense without madvise on
> fiber recycling, because fat tasks will quickly fault-in that memory.
> I thought you were going to implement stack shrinking to some predefined
> size (say, allocate 1 MB max, shrink to 64 KB on recycle if watermark
> was overwritten), leaving only dynamic stack sizing for the future (i.e.
> estimating the optimal size to do madvise for basing on average fiber
> stack usage).

Ah. OK, will rework this.

[PATCH v6 0/2] lib/core/fiber: Increase default stack size

[Cyrill Gorcunov]

v4:
  We split the stack management task into two: first one to
  increase stack size asap, to not block users and testers,
  and second one to rework stack management to relax rss
  pressue.

  This patchset covers the first task only. The second part
  is still under rfc and hopefully won't take too long.

v5:
  Use random offsets for each fiber instead of global one.
  Shrink stack to 64K on recycle.

v6:
  Drop overflow mark competely since it doesn't seem to
  be very useful.
  Simplify code, and put #ifdefs where needed

Take a look please, once time permit.

Cyrill Gorcunov (2):
  lib/core/fiber: Increase default stack size
  lib/core/fiber: Relax stack memory usage on recycle

 src/lib/core/fiber.c | 184 ++++++++++++++++++++++++++++++++++++++++++-
 src/lib/core/fiber.h |  12 +++
 test-run             |   2 +-
 3 files changed, 196 insertions(+), 2 deletions(-)

-- 
2.20.1

[PATCH 1/2] lib/core/fiber: Increase default stack size

[Cyrill Gorcunov]

The default 64K stack size used for years become too small
for modern distors (Fedora 29 and etc) where third party libraries
(such as ncurses) started to use 64K for own buffers and we get
SIGSGV early without reaching interactive console phase.

Thus we increase default size up to 512K which should fit
for common case. Later we will make this value configurable
to address arbitrary stack sizes without a need to rebuild
the whole code.

Part-of #3418
---
 src/lib/core/fiber.c | 2 +-
 test-run             | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/lib/core/fiber.c b/src/lib/core/fiber.c
index abd6c6b11..bf2a22bed 100644
--- a/src/lib/core/fiber.c
+++ b/src/lib/core/fiber.c
@@ -95,7 +95,7 @@ enum {
 	/* The minimum allowable fiber stack size in bytes */
 	FIBER_STACK_SIZE_MINIMAL = 16384,
 	/* Default fiber stack size in bytes */
-	FIBER_STACK_SIZE_DEFAULT = 65536
+	FIBER_STACK_SIZE_DEFAULT = 524288
 };
 
 /** Default fiber attributes */
-- 
2.20.1