From: Vladimir Davydov <vdavydov.dev@gmail.com> To: kostja@tarantool.org Cc: tarantool-patches@freelists.org Subject: [PATCH 8/8] memtx: run garbage collection on demand Date: Tue, 22 May 2018 14:46:16 +0300 [thread overview] Message-ID: <fad9a9fcc1e8eb97da03d61bebd6f0b14384460d.1526987033.git.vdavydov.dev@gmail.com> (raw) In-Reply-To: <cover.1526987033.git.vdavydov.dev@gmail.com> In-Reply-To: <cover.1526987033.git.vdavydov.dev@gmail.com> When a memtx space is dropped or truncated, we delegate freeing tuples stored in it to a background fiber so as not to block the caller (and tx thread) for too long. Turns out it doesn't work out well for ephemeral spaces, which share the destruction code with normal spaces: the problem is the user might issue a lot of complex SQL SELECT statements that create a lot of ephemeral spaces and do not yield and hence don't give the garbage collection fiber a chance to clean up. There's a test that emulates this, 2.0:test/sql-tap/gh-3083-ephemeral-unref-tuples.test.lua. For this test to pass, let's run garbage collection procedure on demand, i.e. when any of memtx allocation functions fails to allocate memory. Follow-up #3408 --- src/box/memtx_engine.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c index 3b21bcaa..afb453f9 100644 --- a/src/box/memtx_engine.c +++ b/src/box/memtx_engine.c @@ -1069,7 +1069,11 @@ memtx_tuple_new(struct tuple_format *format, const char *data, const char *end) return NULL; } - struct memtx_tuple *memtx_tuple = smalloc(&memtx->alloc, total); + struct memtx_tuple *memtx_tuple; + while ((memtx_tuple = smalloc(&memtx->alloc, total)) == NULL) { + if (!memtx_engine_run_gc(memtx)) + break; + } if (memtx_tuple == NULL) { diag_set(OutOfMemory, total, "slab allocator", "memtx_tuple"); return NULL; @@ -1150,7 +1154,11 @@ memtx_index_extent_alloc(void *ctx) "mempool", "new slab"); return NULL; }); - void *ret = mempool_alloc(&memtx->index_extent_pool); + void *ret; + while ((ret = mempool_alloc(&memtx->index_extent_pool)) == NULL) { + if (!memtx_engine_run_gc(memtx)) + break; + } if (ret == NULL) diag_set(OutOfMemory, MEMTX_EXTENT_SIZE, "mempool", "new slab"); @@ -1183,6 +1191,8 @@ memtx_index_extent_reserve(struct memtx_engine *memtx, int num) while (memtx->num_reserved_extents < num) { void *ext = mempool_alloc(&memtx->index_extent_pool); if (ext == NULL) { + if (memtx_engine_run_gc(memtx)) + continue; diag_set(OutOfMemory, MEMTX_EXTENT_SIZE, "mempool", "new slab"); return -1; -- 2.11.0
next prev parent reply other threads:[~2018-05-22 11:46 UTC|newest] Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-05-22 11:46 [PATCH 0/8] Follow-up on async memtx index cleanup Vladimir Davydov 2018-05-22 11:46 ` [PATCH 1/8] memtx: init index extent allocator in engine constructor Vladimir Davydov 2018-05-22 13:43 ` Konstantin Osipov 2018-05-22 11:46 ` [PATCH 2/8] memtx: fold memtx_tuple.cc into memtx_engine.c Vladimir Davydov 2018-05-22 13:45 ` Konstantin Osipov 2018-05-22 11:46 ` [PATCH 3/8] memtx: pass engine to memory allocation functions Vladimir Davydov 2018-05-22 13:47 ` Konstantin Osipov 2018-05-22 14:39 ` Vladimir Davydov 2018-05-22 11:46 ` [PATCH 4/8] memtx: move all global variables to engine Vladimir Davydov 2018-05-22 13:48 ` Konstantin Osipov 2018-05-22 11:46 ` [PATCH 5/8] memtx: destroy slab arena on engine shutdown Vladimir Davydov 2018-05-22 13:50 ` Konstantin Osipov 2018-05-22 16:26 ` Vladimir Davydov 2018-05-22 11:46 ` [PATCH 6/8] memtx: embed light hash into memtx_hash_index Vladimir Davydov 2018-05-22 13:51 ` Konstantin Osipov 2018-05-22 11:46 ` [PATCH 7/8] memtx: rework background garbage collection procedure Vladimir Davydov 2018-05-22 13:56 ` Konstantin Osipov 2018-05-22 14:49 ` Vladimir Davydov 2018-05-22 16:42 ` Konstantin Osipov 2018-05-22 11:46 ` Vladimir Davydov [this message] 2018-05-22 14:00 ` [PATCH 8/8] memtx: run garbage collection on demand Konstantin Osipov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=fad9a9fcc1e8eb97da03d61bebd6f0b14384460d.1526987033.git.vdavydov.dev@gmail.com \ --to=vdavydov.dev@gmail.com \ --cc=kostja@tarantool.org \ --cc=tarantool-patches@freelists.org \ --subject='Re: [PATCH 8/8] memtx: run garbage collection on demand' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox