From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 1AE9F27C9B for ; Fri, 22 Feb 2019 02:36:32 -0500 (EST) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FXfNdOIvBFWv for ; Fri, 22 Feb 2019 02:36:31 -0500 (EST) Received: from smtp47.i.mail.ru (smtp47.i.mail.ru [94.100.177.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id B354227C92 for ; Fri, 22 Feb 2019 02:36:31 -0500 (EST) Received: by smtp47.i.mail.ru with esmtpa (envelope-from ) id 1gx5Nl-0003m0-OR for tarantool-patches@freelists.org; Fri, 22 Feb 2019 10:36:30 +0300 From: =?utf-8?B?0JPQtdC+0YDQs9C40Lkg0JrQuNGA0LjRh9C10L3QutC+?= Subject: [tarantool-patches] Re: [RFC] fiber: Increase default stack size Date: Fri, 22 Feb 2019 10:38:42 +0300 Message-ID: <7936473.6eviP72C8M@home.lan> In-Reply-To: <20190221212642.GT7198@uranus> References: <20190221212642.GT7198@uranus> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2363217.1O1kWk0Wpq"; micalg="pgp-sha256"; protocol="application/pgp-signature" Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-subscribe: List-owner: List-post: List-archive: To: tarantool-patches@freelists.org --nextPart2363217.1O1kWk0Wpq Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Hi Cyrill! Thanks for the patch, please see some comments bellow. I like your approach but I think we should do it in a different manner: we could poison 64k page of a fiber stack on start and then check the poison mark when fiber finished. If watermark was overwritten by fiber activity we could use madvise in order to decrease RSS usage. On Friday, February 22, 2019 12:26:42 AM MSK Cyrill Gorcunov wrote: > The default 64K stack size used for years become too > small for modern distors (Fedora 29 and etc) where third > party libraries (such as ncurses) started to use 64K for > own buffers and we get SIGSGV early without reaching > interactive console phase. > > To address this problem and hopefully eliminate such > problems in future we increase default size up to 1M. > Because this value may be too big for old distros or > other libraries, which would never use such deep stack, > we do a trick: put watermark at 64K offset of the stack > and once fiber get scheduled out we test if the mark is > still present. If we're lucky and noone touched the memory > we use madvise() syscall to reduce RSS usage. > > https://github.com/tarantool/tarantool/issues/3418 > > Signed-off-by: Cyrill Gorcunov > --- > Please review the patch very very carefully since I'm > not that familiar with the codebase. > > src/fiber.c | 110 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- src/fiber.h | > 6 +++ > 2 files changed, 115 insertions(+), 1 deletion(-) > > Index: tarantool.git/src/fiber.c > =================================================================== > --- tarantool.git.orig/src/fiber.c > +++ tarantool.git/src/fiber.c > @@ -91,11 +91,26 @@ pthread_t main_thread_id; > static size_t page_size; > static int stack_direction; > > +/* > + * A random unique value with help of uuidgen. > + * > + * 39ee5420-13f7-417b-9610-ea100c591ab6 > + */ > +static const char stack_wmark[] = { > + 0x39, 0xee, 0x54, 0x20, 0x13, 0xf7, 0x41, 0x7b, > + 0x96, 0x10, 0xea, 0x10, 0x0c, 0x59, 0x1a, 0xb6 > +}; > + > +static void > +stack_sched_out(struct fiber *fiber); > + > enum { > /* The minimum allowable fiber stack size in bytes */ > FIBER_STACK_SIZE_MINIMAL = 16384, > + /* Stack size for stack relaxed tasks */ > + FIBER_STACK_MADVISE_LIMIT = 64536, > /* Default fiber stack size in bytes */ > - FIBER_STACK_SIZE_DEFAULT = 65536 > + FIBER_STACK_SIZE_DEFAULT = 1048576 > }; > > /** Default fiber attributes */ > @@ -188,6 +203,7 @@ fiber_call_impl(struct fiber *callee) > ASAN_START_SWITCH_FIBER(asan_state, 1, > callee->stack, > callee->stack_size); > + stack_sched_out(callee); > coro_transfer(&caller->ctx, &callee->ctx); > ASAN_FINISH_SWITCH_FIBER(asan_state); > } > @@ -441,6 +457,7 @@ fiber_yield(void) > (caller->flags & FIBER_IS_DEAD) == 0, > callee->stack, > callee->stack_size); > + stack_sched_out(callee); > coro_transfer(&caller->ctx, &callee->ctx); > ASAN_FINISH_SWITCH_FIBER(asan_state); > } > @@ -710,6 +727,96 @@ page_align_up(void *ptr) > return page_align_down(ptr + page_size - 1); > } > > +static inline void * > +stack_wmark_pos(struct fiber *fiber) > +{ > + void *pos; > + > + assert(fiber->stack); > + assert(fiber->stack_size); > + > + if (stack_direction < 0) { > + pos = fiber->stack + fiber->stack_size; > + pos -= FIBER_STACK_MADVISE_LIMIT; > + return page_align_up(pos); > + } else { > + pos = fiber->stack - fiber->stack_size; > + pos += FIBER_STACK_MADVISE_LIMIT; > + return page_align_down(pos); > + } > +} > + > +/* > + * Set watermark to the predefined place thus on > + * fiber sched-out procedure we may detect if > + * a task was too eager for stack usage. > + */ > +static inline void > +stack_set_wmark(struct fiber *fiber) > +{ > + void *pos = stack_wmark_pos(fiber); > + memcpy(pos, stack_wmark, sizeof(stack_wmark)); > +} > + > +static inline bool > +stack_has_wmark(struct fiber *fiber) > +{ > + void *pos = stack_wmark_pos(fiber); > + return !!!memcmp(pos, stack_wmark, sizeof(stack_wmark)); > +} > + > +/* > + * Process stack switching: we try to eliminate unneeded > + * physical page usage for stack. If fiber never touched > + * area after/before the watermark, the OS get ralaxed in > + * RSS usage. > + */ > +static void > +stack_sched_out(struct fiber *fiber) > +{ > + if (!fiber->stack || (fiber->flags & FIBER_CUSTOM_STACK)) > + return; > + > + /* > + * If fiber ecxeed a watermark, just clear the > + * flag and don't waste time on it in future. > + */ > + if (!stack_has_wmark(fiber)) { > + fiber->flags &= ~FIBER_MADVISED_STACK; > + return; > + } > + > + /* > + * This is a good one, we simply notify > + * OS about unused stack tail, so associated > + * pages would be put back into a page pool. > + * > + * Note though the fiber still can use > + * remaining space, simply won't be handled > + * that fast on _first_ #pf. > + */ > + if (!(fiber->flags & FIBER_MADVISED_STACK)) { > + size_t size; > + void *tail; > + > + if (stack_direction < 0) { > + tail = stack_wmark_pos(fiber) - page_size; > + size = tail - fiber->stack; > + } else { > + tail = stack_wmark_pos(fiber) + page_size; > + size = fiber->stack - tail; > + } > + > + /* > + * Set the flag iif've successed, > + * otherwise will try on the next > + * round. > + */ > + if (!madvise(fiber->stack, size, MADV_DONTNEED)) > + fiber->flags |= FIBER_MADVISED_STACK; > + } > +} > + > static int > fiber_stack_create(struct fiber *fiber, size_t stack_size) > { > @@ -751,6 +858,7 @@ fiber_stack_create(struct fiber *fiber, > fiber->stack_size); > > mprotect(guard, page_size, PROT_NONE); > + stack_set_wmark(fiber); > return 0; > } > > Index: tarantool.git/src/fiber.h > =================================================================== > --- tarantool.git.orig/src/fiber.h > +++ tarantool.git/src/fiber.h > @@ -89,6 +89,12 @@ enum { > * This flag is set when fiber uses custom stack size. > */ > FIBER_CUSTOM_STACK = 1 << 5, > + /** > + * This flag is set when fiber didn't exceed stack's > + * size watermark and considered being small enough. > + */ > + FIBER_MADVISED_STACK = 1 << 6, > + > FIBER_DEFAULT_FLAGS = FIBER_IS_CANCELLABLE > }; --nextPart2363217.1O1kWk0Wpq Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEFB+nbqWGnp59Rk9ZFSyY70x8X3sFAlxvpwIACgkQFSyY70x8 X3uLKQgAjw2zqmMlZZMyXFXzzQnVSTz+iHOM5UhhWWMxTHicRWe5nPtGLhRhTn8V 5z1tQf1tHwQOQrYZcoyA3Ib8NX27AkMMh6IGT/tWbri8zQvm7e1FyUv1Eypen4us 11ajXbdijd6Eadf5+rKNuOvrtsJfwW4dh50zEKRL285lIC3a5eSECb6bpOLqULLP c83pQRToqVKBURDjfHjkXV4C8kxu3212zhCWaa6CI43ARo8h5d2eAq0Wi9EvDGrZ 4E6NbcSDi8k0GTKDhZ9uS82ON7Ut2H3Ub9n3xCvSx5WPMje0RNB3POU2LL6Vf1nO IlNW1HBKnpy4Whxz3IEhu0tht4bTeg== =M2Yw -----END PGP SIGNATURE----- --nextPart2363217.1O1kWk0Wpq--