From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 8FCC929F41 for ; Wed, 19 Jun 2019 04:05:01 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JF7IN17t_mt7 for ; Wed, 19 Jun 2019 04:05:01 -0400 (EDT) Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 00B0925F93 for ; Wed, 19 Jun 2019 04:05:00 -0400 (EDT) Date: Wed, 19 Jun 2019 11:04:57 +0300 From: Mergen Imeev Subject: [tarantool-patches] Re: [PATCH v1 26/28] sql: cleanup of legacy memory management system Message-ID: <20190619080457.GB18464@tarantool.org> References: <83a31a15dfe41460a572961ae55540ebb41e8c3e.1560174553.git.imeevma@gmail.com> <87ff4cf8-d816-8e97-be34-e4272e9a3218@tarantool.org> <20190615100444.GI32365@tarantool.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: Vladislav Shpilevoy Cc: tarantool-patches@freelists.org On Tue, Jun 18, 2019 at 10:40:09PM +0200, Vladislav Shpilevoy wrote: > Thanks for the squash. I've found one another part of > SQL memory subsystem that can be removed. Please, consider > my review fixes here and on the branch in a separate commit. > Thank you! I squashed this commit. New patch: >From 20be4d79979c0869ceaab4c465f31b00ff1fa025 Mon Sep 17 00:00:00 2001 From: Nikita Pettik Date: Sat, 25 May 2019 18:50:33 +0300 Subject: [PATCH] sql: cleanup of legacy memory management system Follow-up for #4074 diff --git a/src/box/sql/CMakeLists.txt b/src/box/sql/CMakeLists.txt index 7059b57..1f2a664 100644 --- a/src/box/sql/CMakeLists.txt +++ b/src/box/sql/CMakeLists.txt @@ -31,7 +31,6 @@ add_library(sql STATIC date.c delete.c expr.c - fault.c fk_constraint.c func.c global.c @@ -49,7 +48,6 @@ add_library(sql STATIC random.c resolve.c select.c - status.c tokenize.c treeview.c trigger.c diff --git a/src/box/sql/date.c b/src/box/sql/date.c index 5cb7c85..2e2a71a 100644 --- a/src/box/sql/date.c +++ b/src/box/sql/date.c @@ -536,9 +536,6 @@ clearYMD_HMS_TZ(DateTime * p) * is available. This routine returns 0 on success and * non-zero on any kind of error. * - * If the sqlGlobalConfig.bLocaltimeFault variable is true then this - * routine will always fail. - * * EVIDENCE-OF: R-62172-00036 In this implementation, the standard C * library function localtime_r() is used to assist in the calculation of * local time. @@ -550,18 +547,10 @@ osLocaltime(time_t * t, struct tm *pTm) #if !HAVE_LOCALTIME_R && !HAVE_LOCALTIME_S struct tm *pX; pX = localtime(t); - - if (sqlGlobalConfig.bLocaltimeFault) - pX = 0; - if (pX) *pTm = *pX; rc = pX == 0; #else - - if (sqlGlobalConfig.bLocaltimeFault) - return 1; - #if HAVE_LOCALTIME_R rc = localtime_r(t, pTm) == 0; #else diff --git a/src/box/sql/fault.c b/src/box/sql/fault.c deleted file mode 100644 index 62e9924..0000000 --- a/src/box/sql/fault.c +++ /dev/null @@ -1,104 +0,0 @@ -/* - * Copyright 2010-2017, Tarantool AUTHORS, please see AUTHORS file. - * - * Redistribution and use in source and binary forms, with or - * without modification, are permitted provided that the following - * conditions are met: - * - * 1. Redistributions of source code must retain the above - * copyright notice, this list of conditions and the - * following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following - * disclaimer in the documentation and/or other materials - * provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL - * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -/* - * - * This file contains code to support the concept of "benign" - * malloc failures (when the xMalloc() or xRealloc() method of the - * sql_mem_methods structure fails to allocate a block of memory - * and returns 0). - * - * Most malloc failures are non-benign. After they occur, sql - * abandons the current operation and returns an error - * to the user. However, sometimes a fault is not necessarily - * fatal. For example, if a malloc fails while resizing a hash table, this - * is completely recoverable simply by not carrying out the resize. The - * hash table will continue to function normally. So a malloc failure - * during a hash table resize is a benign fault. - */ - -#include "sqlInt.h" - -/* - * Global variables. - */ -typedef struct BenignMallocHooks BenignMallocHooks; -static SQL_WSD struct BenignMallocHooks { - void (*xBenignBegin) (void); - void (*xBenignEnd) (void); -} sqlHooks = { -0, 0}; - -/* The "wsdHooks" macro will resolve to the appropriate BenignMallocHooks - * structure. If writable static data is unsupported on the target, - * we have to locate the state vector at run-time. In the more common - * case where writable static data is supported, wsdHooks can refer directly - * to the "sqlHooks" state vector declared above. - */ -#define wsdHooksInit -#define wsdHooks sqlHooks - -/* - * Register hooks to call when sqlBeginBenignMalloc() and - * sqlEndBenignMalloc() are called, respectively. - */ -void -sqlBenignMallocHooks(void (*xBenignBegin) (void), void (*xBenignEnd) (void) - ) -{ - wsdHooksInit; - wsdHooks.xBenignBegin = xBenignBegin; - wsdHooks.xBenignEnd = xBenignEnd; -} - -/* - * This (sqlEndBenignMalloc()) is called by sql code to indicate that - * subsequent malloc failures are benign. A call to sqlEndBenignMalloc() - * indicates that subsequent malloc failures are non-benign. - */ -void -sqlBeginBenignMalloc(void) -{ - wsdHooksInit; - if (wsdHooks.xBenignBegin) { - wsdHooks.xBenignBegin(); - } -} - -void -sqlEndBenignMalloc(void) -{ - wsdHooksInit; - if (wsdHooks.xBenignEnd) { - wsdHooks.xBenignEnd(); - } -} - diff --git a/src/box/sql/global.c b/src/box/sql/global.c index 2d937d7..6cadef8 100644 --- a/src/box/sql/global.c +++ b/src/box/sql/global.c @@ -137,29 +137,6 @@ const unsigned char sqlCtypeMap[256] = { 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40, 0x40 /* f8..ff ........ */ }; -/* EVIDENCE-OF: R-02982-34736 In order to maintain full backwards - * compatibility for legacy applications, the URI filename capability is - * disabled by default. - * - * EVIDENCE-OF: R-38799-08373 URI filenames can be enabled or disabled - * using the SQL_USE_URI=1 or SQL_USE_URI=0 compile-time options. - * - * EVIDENCE-OF: R-43642-56306 By default, URI handling is globally - * disabled. The default value may be changed by compiling with the - * SQL_USE_URI symbol defined. - */ -#ifndef SQL_USE_URI -#define SQL_USE_URI 0 -#endif - -/* EVIDENCE-OF: R-38720-18127 The default setting is determined by the - * SQL_ALLOW_COVERING_INDEX_SCAN compile-time option, or is "on" if - * that compile-time option is omitted. - */ -#ifndef SQL_ALLOW_COVERING_INDEX_SCAN -#define SQL_ALLOW_COVERING_INDEX_SCAN 1 -#endif - /* The minimum PMA size is set to this value multiplied by the database * page size in bytes. */ @@ -172,35 +149,16 @@ const unsigned char sqlCtypeMap[256] = { * the sql library. */ SQL_WSD struct sqlConfig sqlConfig = { - SQL_DEFAULT_MEMSTATUS, /* bMemstat */ - SQL_USE_URI, /* bOpenUri */ - SQL_ALLOW_COVERING_INDEX_SCAN, /* bUseCis */ - 0x7ffffffe, /* mxStrlen */ - 0, /* neverCorrupt */ - 0, /* nStmtSpill */ - (void *)0, /* pHeap */ - 0, /* nHeap */ - 0, 0, /* mnHeap, mxHeap */ SQL_DEFAULT_MMAP_SIZE, /* szMmap */ SQL_MAX_MMAP_SIZE, /* mxMmap */ - (void *)0, /* pScratch */ - 0, /* szScratch */ - 0, /* nScratch */ - (void *)0, /* pPage */ - 0, /* szPage */ - SQL_DEFAULT_PCACHE_INITSZ, /* nPage */ - 0, /* mxParserStack */ - 0, /* sharedCacheEnabled */ SQL_SORTER_PMASZ, /* szPma */ /* All the rest should always be initialized to zero */ 0, /* isInit */ 0, /* inProgress */ - 0, /* isMallocInit */ #ifdef SQL_VDBE_COVERAGE 0, /* xVdbeBranch */ 0, /* pVbeBranchArg */ #endif - 0, /* bLocaltimeFault */ 0x7ffffffe /* iOnceResetThreshold */ }; diff --git a/src/box/sql/hash.c b/src/box/sql/hash.c index 5bc8109..36f965b 100644 --- a/src/box/sql/hash.c +++ b/src/box/sql/hash.c @@ -149,18 +149,7 @@ rehash(Hash * pH, unsigned int new_size) if (new_size == pH->htsize) return 0; #endif - - /* The inability to allocates space for a larger hash table is - * a performance hit but it is not a fatal error. So mark the - * allocation as a benign. Use sqlMalloc()/memset(0) instead of - * sqlMallocZero() to make the allocation, as sqlMallocZero() - * only zeroes the requested number of bytes whereas this module will - * use the actual amount of space allocated for the hash table (which - * may be larger than the requested amount). - */ - sqlBeginBenignMalloc(); new_ht = (struct _ht *)sqlMalloc(new_size * sizeof(struct _ht)); - sqlEndBenignMalloc(); if (new_ht == 0) return 0; diff --git a/src/box/sql/main.c b/src/box/sql/main.c index d2688e7..59c8315 100644 --- a/src/box/sql/main.c +++ b/src/box/sql/main.c @@ -105,11 +105,6 @@ sql_initialize(void) if (sqlGlobalConfig.isInit) return 0; - if (!sqlGlobalConfig.isMallocInit) - sqlMallocInit(); - if (rc == 0) - sqlGlobalConfig.isMallocInit = 1; - /* If rc is not 0 at this point, then the malloc * subsystem could not be initialized. */ diff --git a/src/box/sql/malloc.c b/src/box/sql/malloc.c index 62184f4..d92709b 100644 --- a/src/box/sql/malloc.c +++ b/src/box/sql/malloc.c @@ -62,23 +62,6 @@ sql_sized_malloc(int nByte) } /* - * Like free() but works for allocations obtained from sql_sized_malloc() - * or sql_sized_realloc(). - * - * For this low-level routine, we already know that pPrior!=0 since - * cases where pPrior==0 will have been intecepted and dealt with - * by higher-level routines. - */ -static void -sql_sized_free(void *pPrior) -{ - sql_int64 *p = (sql_int64 *) pPrior; - assert(pPrior != 0); - p--; - free(p); -} - -/* * Report the allocated size of a prior return from sql_sized_malloc() * or sql_sized_realloc(). */ @@ -121,145 +104,6 @@ sql_sized_realloc(void *pPrior, int nByte) } /* - * Attempt to release up to n bytes of non-essential memory currently - * held by sql. An example of non-essential memory is memory used to - * cache database pages that are not currently in use. - */ -int -sql_release_memory(int n) -{ - /* IMPLEMENTATION-OF: R-34391-24921 The sql_release_memory() routine - * is a no-op returning zero if sql is not compiled with - * SQL_ENABLE_MEMORY_MANAGEMENT. - */ - UNUSED_PARAMETER(n); - return 0; -} - -/* - * An instance of the following object records the location of - * each unused scratch buffer. - */ -typedef struct ScratchFreeslot { - struct ScratchFreeslot *pNext; /* Next unused scratch buffer */ -} ScratchFreeslot; - -/* - * State information local to the memory allocation subsystem. - */ -static SQL_WSD struct Mem0Global { - sql_int64 alarmThreshold; /* The soft heap limit */ - - /* - * Pointers to the end of sqlGlobalConfig.pScratch memory - * (so that a range test can be used to determine if an allocation - * being freed came from pScratch) and a pointer to the list of - * unused scratch allocations. - */ - void *pScratchEnd; - ScratchFreeslot *pScratchFree; - u32 nScratchFree; - - /* - * True if heap is nearly "full" where "full" is defined by the - * sql_soft_heap_limit() setting. - */ - int nearlyFull; -} mem0 = { -0, 0, 0, 0, 0}; - -#define mem0 GLOBAL(struct Mem0Global, mem0) - -/* - * Initialize the memory allocation subsystem. - */ -void -sqlMallocInit(void) -{ - memset(&mem0, 0, sizeof(mem0)); - if (sqlGlobalConfig.pScratch && sqlGlobalConfig.szScratch >= 100 - && sqlGlobalConfig.nScratch > 0) { - int i, n, sz; - ScratchFreeslot *pSlot; - sz = ROUNDDOWN8(sqlGlobalConfig.szScratch); - sqlGlobalConfig.szScratch = sz; - pSlot = (ScratchFreeslot *) sqlGlobalConfig.pScratch; - n = sqlGlobalConfig.nScratch; - mem0.pScratchFree = pSlot; - mem0.nScratchFree = n; - for (i = 0; i < n - 1; i++) { - pSlot->pNext = (ScratchFreeslot *) (sz + (char *)pSlot); - pSlot = pSlot->pNext; - } - pSlot->pNext = 0; - mem0.pScratchEnd = (void *)&pSlot[1]; - } else { - mem0.pScratchEnd = 0; - sqlGlobalConfig.pScratch = 0; - sqlGlobalConfig.szScratch = 0; - sqlGlobalConfig.nScratch = 0; - } - if (sqlGlobalConfig.pPage == 0 || sqlGlobalConfig.szPage < 512 - || sqlGlobalConfig.nPage <= 0) { - sqlGlobalConfig.pPage = 0; - sqlGlobalConfig.szPage = 0; - } -} - -/* - * Return true if the heap is currently under memory pressure - in other - * words if the amount of heap used is close to the limit set by - * sql_soft_heap_limit(). - */ -int -sqlHeapNearlyFull(void) -{ - return mem0.nearlyFull; -} - -/* - * Trigger the alarm - */ -static void -sqlMallocAlarm(int nByte) -{ - if (mem0.alarmThreshold <= 0) - return; - sql_release_memory(nByte); -} - -/* - * Do a memory allocation with statistics and alarms. Assume the - * lock is already held. - */ -static int -mallocWithAlarm(int n, void **pp) -{ - int nFull; - void *p; - nFull = ROUND8(n); - sqlStatusHighwater(SQL_STATUS_MALLOC_SIZE, n); - if (mem0.alarmThreshold > 0) { - sql_int64 nUsed = - sqlStatusValue(SQL_STATUS_MEMORY_USED); - if (nUsed >= mem0.alarmThreshold - nFull) { - mem0.nearlyFull = 1; - sqlMallocAlarm(nFull); - } else { - mem0.nearlyFull = 0; - } - } - p = sql_sized_malloc(nFull); - if (p) { - nFull = sqlMallocSize(p); - sqlStatusUp(SQL_STATUS_MEMORY_USED, nFull); - sqlStatusUp(SQL_STATUS_MALLOC_COUNT, 1); - } - *pp = p; - return nFull; -} - -/* * Allocate memory. This routine is like sql_malloc() except that it * assumes the memory subsystem has already been initialized. */ @@ -275,8 +119,6 @@ sqlMalloc(u64 n) * this amount. The only way to reach the limit is with sql_malloc() */ p = 0; - } else if (sqlGlobalConfig.bMemstat) { - mallocWithAlarm((int)n, &p); } else { p = sql_sized_malloc((int)n); } @@ -311,37 +153,17 @@ sqlMallocSize(void *p) return sql_sized_sizeof(p); } -sql_uint64 -sql_msize(void *p) -{ - return p ? sql_sized_sizeof(p) : 0; -} - /* * Free memory previously obtained from sqlMalloc(). */ void sql_free(void *p) { - if (p == 0) - return; /* IMP: R-49053-54554 */ - if (sqlGlobalConfig.bMemstat) { - sqlStatusDown(SQL_STATUS_MEMORY_USED, - sqlMallocSize(p)); - sqlStatusDown(SQL_STATUS_MALLOC_COUNT, 1); - sql_sized_free(p); - } else - sql_sized_free(p); -} - -/* - * Add the size of memory allocation "p" to the count in - * *db->pnBytesFreed. - */ -static SQL_NOINLINE void -measureAllocationSize(sql * db, void *p) -{ - *db->pnBytesFreed += sqlMallocSize(p); + if (p == NULL) + return; + sql_int64 *raw_p = (sql_int64 *) p; + raw_p--; + free(raw_p); } /* @@ -351,14 +173,7 @@ measureAllocationSize(sql * db, void *p) void sqlDbFree(sql * db, void *p) { - if (p == 0) - return; - if (db) { - if (db->pnBytesFreed) { - measureAllocationSize(db, p); - return; - } - } + (void) db; sql_free(p); } @@ -368,7 +183,7 @@ sqlDbFree(sql * db, void *p) void * sqlRealloc(void *pOld, u64 nBytes) { - int nOld, nNew, nDiff; + int nOld, nNew; void *pNew; if (pOld == 0) { return sqlMalloc(nBytes); /* IMP: R-04300-56712 */ @@ -383,36 +198,14 @@ sqlRealloc(void *pOld, u64 nBytes) } nOld = sqlMallocSize(pOld); nNew = ROUND8((int)nBytes); - if (nOld == nNew) { + if (nOld == nNew) pNew = pOld; - } else if (sqlGlobalConfig.bMemstat) { - sqlStatusHighwater(SQL_STATUS_MALLOC_SIZE, (int)nBytes); - nDiff = nNew - nOld; - if (nDiff > 0 - && sqlStatusValue(SQL_STATUS_MEMORY_USED) >= - mem0.alarmThreshold - nDiff) { - sqlMallocAlarm(nDiff); - } + else pNew = sql_sized_realloc(pOld, nNew); - if (pNew == 0 && mem0.alarmThreshold > 0) { - sqlMallocAlarm((int)nBytes); - pNew = sql_sized_realloc(pOld, nNew); - } - if (pNew) { - nNew = sqlMallocSize(pNew); - sqlStatusUp(SQL_STATUS_MEMORY_USED, nNew - nOld); - } - } else { - pNew = sql_sized_realloc(pOld, nNew); - } assert(EIGHT_BYTE_ALIGNMENT(pNew)); /* IMP: R-11148-40995 */ return pNew; } -/* - * The public interface to sqlRealloc. Make sure that the memory - * subsystem is initialized prior to invoking sqlRealloc. - */ void * sql_realloc64(void *pOld, sql_uint64 n) { @@ -481,7 +274,7 @@ sqlDbMallocRaw(sql * db, u64 n) void * sqlDbMallocRawNN(sql * db, u64 n) { - assert(db != NULL && db->pnBytesFreed == NULL); + assert(db != NULL); if (db->mallocFailed) return NULL; void *p = sqlMalloc(n); @@ -576,17 +369,6 @@ sqlDbStrNDup(sql * db, const char *z, u64 n) } /* - * Call this routine to record the fact that an OOM (out-of-memory) error - * has happened. This routine will set db->mallocFailed. - */ -void -sqlOomFault(sql * db) -{ - if (db->mallocFailed == 0 && db->bBenignMalloc == 0) - db->mallocFailed = 1; -} - -/* * This routine reactivates the memory allocator and clears the * db->mallocFailed flag as necessary. * @@ -601,16 +383,6 @@ sqlOomClear(sql * db) } /* - * Take actions at the end of an API call to indicate an OOM error - */ -static SQL_NOINLINE int -apiOomError(sql * db) -{ - sqlOomClear(db); - return -1; -} - -/* * This function must be called before exiting any API function (i.e. * returning control to the user) that has called sql_malloc or * sql_realloc. @@ -623,7 +395,8 @@ sqlApiExit(sql * db, int rc) { assert(db != 0); if (db->mallocFailed) { - return apiOomError(db); + sqlOomClear(db); + return -1; } return rc; } diff --git a/src/box/sql/sqlInt.h b/src/box/sql/sqlInt.h index 434a5bd..e4c5843 100644 --- a/src/box/sql/sqlInt.h +++ b/src/box/sql/sqlInt.h @@ -168,15 +168,6 @@ #define SQL_NOINLINE #endif -/* - * EVIDENCE-OF: R-25715-37072 Memory allocation statistics are enabled by - * default unless sql is compiled with sql_DEFAULT_MEMSTATUS=0 in - * which case memory allocation statistics are disabled by default. - */ -#if !defined(SQL_DEFAULT_MEMSTATUS) -#define SQL_DEFAULT_MEMSTATUS 1 -#endif - #if defined(SQL_SYSTEM_MALLOC) \ + defined(SQL_ZERO_MALLOC) > 1 #error "Two or more of the following compile-time configuration options\ @@ -578,13 +569,6 @@ sql_initialize(void); #define SQL_DETERMINISTIC 0x800 -#define SQL_STATUS_MEMORY_USED 0 -#define SQL_STATUS_MALLOC_SIZE 5 -#define SQL_STATUS_PARSER_STACK 6 -#define SQL_STATUS_PAGECACHE_SIZE 7 -#define SQL_STATUS_SCRATCH_SIZE 8 -#define SQL_STATUS_MALLOC_COUNT 9 - int sql_create_function_v2(sql * db, const char *zFunctionName, @@ -782,16 +766,6 @@ sql_bind_parameter_lindex(sql_stmt * pStmt, const char *zName, #endif /* - * The default initial allocation for the pagecache when using separate - * pagecaches for each database connection. A positive number is the - * number of pages. A negative number N translations means that a buffer - * of -1024*N bytes is allocated and used for as many pages as it will hold. - */ -#ifndef SQL_DEFAULT_PCACHE_INITSZ -#define SQL_DEFAULT_PCACHE_INITSZ 100 -#endif - -/* * GCC does not define the offsetof() macro so we'll have to do it * ourselves. */ @@ -1133,7 +1107,6 @@ struct sql { u16 dbOptFlags; /* Flags to enable/disable optimizations */ u8 enc; /* Text encoding */ u8 mallocFailed; /* True if we have seen a malloc failure */ - u8 bBenignMalloc; /* Do not require OOMs if true */ u8 dfltLockMode; /* Default locking-mode for attached dbs */ u8 mTrace; /* zero or more sql_TRACE flags */ u32 magic; /* Magic number for detect library misuse */ @@ -1162,7 +1135,6 @@ struct sql { void (*xUpdateCallback) (void *, int, const char *, const char *, sql_int64); Hash aFunc; /* Hash table of connection functions */ - int *pnBytesFreed; /* If not NULL, increment this in DbFree() */ }; /* @@ -1223,11 +1195,7 @@ struct sql { * than being distinct from one another. */ #define SQL_MAGIC_OPEN 0xa029a697 /* Database is open */ -#define SQL_MAGIC_CLOSED 0x9f3c2d33 /* Database is closed */ -#define SQL_MAGIC_SICK 0x4b771290 /* Error and awaiting close */ #define SQL_MAGIC_BUSY 0xf03b7906 /* Database currently in use */ -#define SQL_MAGIC_ERROR 0xb5357930 /* An sql_MISUSE error occurred */ -#define SQL_MAGIC_ZOMBIE 0x64cffc7f /* Close with last statement close */ /** * SQL type definition. Now it is an alias to type, but in @@ -2563,32 +2531,14 @@ struct StrAccum { * This structure also contains some state information. */ struct sqlConfig { - int bMemstat; /* True to enable memory status */ - int bOpenUri; /* True to interpret filenames as URIs */ - int bUseCis; /* Use covering indices for full-scans */ - int mxStrlen; /* Maximum string length */ - int neverCorrupt; /* Database is always well-formed */ - int nStmtSpill; /* Stmt-journal spill-to-disk threshold */ - void *pHeap; /* Heap storage space */ - int nHeap; /* Size of pHeap[] */ - int mnReq, mxReq; /* Min and max heap requests sizes */ sql_int64 szMmap; /* mmap() space per open file */ sql_int64 mxMmap; /* Maximum value for szMmap */ - void *pScratch; /* Scratch memory */ - int szScratch; /* Size of each scratch buffer */ - int nScratch; /* Number of scratch buffers */ - void *pPage; /* Page cache memory */ - int szPage; /* Size of each page in pPage[] */ - int nPage; /* Number of pages in pPage[] */ - int mxParserStack; /* maximum depth of the parser stack */ - int sharedCacheEnabled; /* true if shared-cache mode enabled */ u32 szPma; /* Maximum Sorter PMA size */ /* The above might be initialized to non-zero. The following need to always * initially be zero, however. */ int isInit; /* True after initialization has finished */ int inProgress; /* True while initialization in progress */ - int isMallocInit; /* True after malloc is initialized */ #ifdef SQL_VDBE_COVERAGE /* The following callback (if not NULL) is invoked on every VDBE branch * operation. Set the callback using sql_TESTCTRL_VDBE_COVERAGE. @@ -2596,7 +2546,6 @@ struct sqlConfig { void (*xVdbeBranch) (void *, int iSrcLine, u8 eThis, u8 eMx); /* Callback */ void *pVdbeBranchArg; /* 1st argument */ #endif - int bLocaltimeFault; /* True to fail localtime() calls */ int iOnceResetThreshold; /* When to reset OP_Once counters */ }; @@ -2685,7 +2634,6 @@ int sqlStrICmp(const char *, const char *); unsigned sqlStrlen30(const char *); #define sqlStrNICmp sql_strnicmp -void sqlMallocInit(void); void *sqlMalloc(u64); void *sqlMallocZero(u64); void *sqlDbMallocZero(sql *, u64); @@ -2698,8 +2646,6 @@ void *sqlDbReallocOrFree(sql *, void *, u64); void *sqlDbRealloc(sql *, void *, u64); void sqlDbFree(sql *, void *); int sqlMallocSize(void *); -void sqlBenignMallocHooks(void (*)(void), void (*)(void)); -int sqlHeapNearlyFull(void); /* * On systems with ample stack space and that support alloca(), make @@ -2719,11 +2665,6 @@ int sqlHeapNearlyFull(void); #define sqlStackFree(D,P) sqlDbFree(D,P) #endif -sql_int64 sqlStatusValue(int); -void sqlStatusUp(int, int); -void sqlStatusDown(int, int); -void sqlStatusHighwater(int, int); - int sqlIsNaN(double); /* @@ -4348,7 +4289,14 @@ int sqlCreateFunc(sql *, const char *, enum field_type, void (*)(sql_context *, int, sql_value **), void (*)(sql_context *), FuncDestructor * pDestructor); -void sqlOomFault(sql *); + +/** Set OOM error flag. */ +static inline void +sqlOomFault(struct sql *db) +{ + db->mallocFailed = 1; +} + void sqlOomClear(sql *); int sqlApiExit(sql * db, int); @@ -4531,13 +4479,6 @@ bool fk_constraint_is_required(struct space *space, const int *changes); /* - * The interface to the code in fault.c used for identifying "benign" - * malloc failures. - */ -void sqlBeginBenignMalloc(void); -void sqlEndBenignMalloc(void); - -/* * Allowed return values from sqlFindInIndex() */ #define IN_INDEX_EPH 2 /* Search an ephemeral b-tree */ diff --git a/src/box/sql/status.c b/src/box/sql/status.c deleted file mode 100644 index 515274a..0000000 --- a/src/box/sql/status.c +++ /dev/null @@ -1,127 +0,0 @@ -/* - * Copyright 2010-2017, Tarantool AUTHORS, please see AUTHORS file. - * - * Redistribution and use in source and binary forms, with or - * without modification, are permitted provided that the following - * conditions are met: - * - * 1. Redistributions of source code must retain the above - * copyright notice, this list of conditions and the - * following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above - * copyright notice, this list of conditions and the following - * disclaimer in the documentation and/or other materials - * provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL - * OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR - * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF - * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -/* - * - * This module implements the sql_status() interface and related - * functionality. - */ -#include "sqlInt.h" -#include "vdbeInt.h" -/* - * Variables in which to record status information. - */ -#if SQL_PTRSIZE>4 -typedef sql_int64 sqlStatValueType; -#else -typedef u32 sqlStatValueType; -#endif -typedef struct sqlStatType sqlStatType; -static SQL_WSD struct sqlStatType { - sqlStatValueType nowValue[10]; /* Current value */ - sqlStatValueType mxValue[10]; /* Maximum value */ -} sqlStat = { { -0,}, { -0,}}; - - -/* The "wsdStat" macro will resolve to the status information - * state vector. In the common case where writable static data is - * supported, wsdStat can refer directly to the "sqlStat" state - * vector declared above. - */ -#define wsdStatInit -#define wsdStat sqlStat - -/* - * Return the current value of a status parameter. - */ -sql_int64 -sqlStatusValue(int op) -{ - wsdStatInit; - assert(op >= 0 && op < ArraySize(wsdStat.nowValue)); - - return wsdStat.nowValue[op]; -} - -/* - * Add N to the value of a status record. - * - * The StatusUp() routine can accept positive or negative values for N. - * The value of N is added to the current status value and the high-water - * mark is adjusted if necessary. - * - * The StatusDown() routine lowers the current value by N. The highwater - * mark is unchanged. N must be non-negative for StatusDown(). - */ -void -sqlStatusUp(int op, int N) -{ - wsdStatInit; - assert(op >= 0 && op < ArraySize(wsdStat.nowValue)); - - wsdStat.nowValue[op] += N; - if (wsdStat.nowValue[op] > wsdStat.mxValue[op]) { - wsdStat.mxValue[op] = wsdStat.nowValue[op]; - } -} - -void -sqlStatusDown(int op, int N) -{ - wsdStatInit; - assert(N >= 0); - - assert(op >= 0 && op < ArraySize(wsdStat.nowValue)); - wsdStat.nowValue[op] -= N; -} - -/* - * Adjust the highwater mark if necessary. - */ -void -sqlStatusHighwater(int op, int X) -{ - sqlStatValueType newValue; - wsdStatInit; - assert(X >= 0); - newValue = (sqlStatValueType) X; - assert(op >= 0 && op < ArraySize(wsdStat.nowValue)); - - assert(op == SQL_STATUS_MALLOC_SIZE - || op == SQL_STATUS_PAGECACHE_SIZE - || op == SQL_STATUS_SCRATCH_SIZE - || op == SQL_STATUS_PARSER_STACK); - if (newValue > wsdStat.mxValue[op]) { - wsdStat.mxValue[op] = newValue; - } -} diff --git a/src/box/sql/tokenize.c b/src/box/sql/tokenize.c index ea364de..9fa069d 100644 --- a/src/box/sql/tokenize.c +++ b/src/box/sql/tokenize.c @@ -430,7 +430,7 @@ sql_token(const char *z, int *type, bool *is_reserved) static void parser_space_delete(struct sql *db, struct space *space) { - if (space == NULL || db == NULL || db->pnBytesFreed == 0) + if (space == NULL || db == NULL) return; assert(space->def->opts.is_ephemeral); for (uint32_t i = 0; i < space->index_count; ++i) @@ -514,11 +514,6 @@ sqlRunParser(Parse * pParse, const char *zSql) } } pParse->zTail = &zSql[i]; -#ifdef YYTRACKMAXSTACKDEPTH - sqlStatusHighwater(SQL_STATUS_PARSER_STACK, - sqlParserStackPeak(pEngine) - ); -#endif /* YYDEBUG */ sqlParserFree(pEngine, sql_free); if (db->mallocFailed) pParse->is_aborted = true; diff --git a/src/box/sql/vdbe.c b/src/box/sql/vdbe.c index 8d512d3..c8887f9 100644 --- a/src/box/sql/vdbe.c +++ b/src/box/sql/vdbe.c @@ -226,10 +226,6 @@ allocateCursor( * different sized allocations. Memory cells provide growable * allocations. * - * * When using ENABLE_MEMORY_MANAGEMENT, memory cell buffers can - * be freed lazily via the sql_release_memory() API. This - * minimizes the number of malloc calls made by the system. - * * The memory cell for cursor 0 is aMem[0]. The rest are allocated from * the top of the register space. Cursor 1 is at Mem[p->nMem-1]. * Cursor 2 is at Mem[p->nMem-2]. And so forth. @@ -751,7 +747,6 @@ int sqlVdbeExec(Vdbe *p) assert(p->explain==0); p->pResultSet = 0; #ifdef SQL_DEBUG - sqlBeginBenignMalloc(); if (p->pc == 0 && (p->sql_flags & (SQL_VdbeListing|SQL_VdbeEQP|SQL_VdbeTrace)) != 0) { int i; @@ -775,7 +770,6 @@ int sqlVdbeExec(Vdbe *p) if ((p->sql_flags & SQL_VdbeTrace) != 0) printf("VDBE Trace:\n"); } - sqlEndBenignMalloc(); #endif for(pOp=&aOp[p->pc]; 1; pOp++) { /* Errors are detected by individual opcodes, with an immediate diff --git a/src/box/sql/vdbeaux.c b/src/box/sql/vdbeaux.c index bc5a14a..364f7ed 100644 --- a/src/box/sql/vdbeaux.c +++ b/src/box/sql/vdbeaux.c @@ -664,17 +664,6 @@ freeEphemeralFunction(sql * db, FuncDef * pDef) static void vdbeFreeOpArray(sql *, Op *, int); -/* - * Delete a P4 value if necessary. - */ -static SQL_NOINLINE void -freeP4Mem(sql * db, Mem * p) -{ - if (p->szMalloc) - sqlDbFree(db, p->zMalloc); - sqlDbFree(db, p); -} - static SQL_NOINLINE void freeP4FuncCtx(sql * db, sql_context * p) { @@ -705,14 +694,9 @@ freeP4(sql * db, int p4type, void *p4) freeEphemeralFunction(db, (FuncDef *) p4); break; } - case P4_MEM:{ - if (db->pnBytesFreed == 0) { - sqlValueFree((sql_value *) p4); - } else { - freeP4Mem(db, (Mem *) p4); - } - break; - } + case P4_MEM: + sqlValueFree((sql_value *) p4); + break; } } @@ -1305,13 +1289,6 @@ releaseMemArray(Mem * p, int N) if (p && N) { Mem *pEnd = &p[N]; sql *db = p->db; - if (db->pnBytesFreed) { - do { - if (p->szMalloc) - sqlDbFree(db, p->zMalloc); - } while ((++p) < pEnd); - return; - } do { assert((&p[1]) == pEnd || p[0].db == p[1].db); assert(sqlVdbeCheckMemInvariants(p)); diff --git a/src/box/sql/vdbesort.c b/src/box/sql/vdbesort.c index 5873fab..7091a99 100644 --- a/src/box/sql/vdbesort.c +++ b/src/box/sql/vdbesort.c @@ -819,43 +819,32 @@ sqlVdbeSorterInit(sql * db, /* Database connection (for malloc()) */ pSorter = (VdbeSorter *) sqlDbMallocZero(db, sizeof(VdbeSorter)); pCsr->uc.pSorter = pSorter; - if (pSorter == 0) { + if (pSorter == 0) + return -1; + + pSorter->key_def = pCsr->key_def; + pSorter->pgsz = pgsz = 1024; + pSorter->db = db; + pSorter->aTask.pSorter = pSorter; + + /* Cache size in bytes */ + i64 mxCache; + u32 szPma = sqlGlobalConfig.szPma; + pSorter->mnPmaSize = szPma * pgsz; + + mxCache = SQL_DEFAULT_CACHE_SIZE; + mxCache = mxCache * -1024; + mxCache = MIN(mxCache, SQL_MAX_PMASZ); + pSorter->mxPmaSize = MAX(pSorter->mnPmaSize, (int)mxCache); + assert(pSorter->iMemory == 0); + pSorter->nMemory = pgsz; + pSorter->list.aMemory = (u8 *) sqlMalloc(pgsz); + if (!pSorter->list.aMemory) rc = -1; - } else { - pSorter->key_def = pCsr->key_def; - pSorter->pgsz = pgsz = 1024; - pSorter->db = db; - pSorter->aTask.pSorter = pSorter; - - i64 mxCache; /* Cache size in bytes */ - u32 szPma = sqlGlobalConfig.szPma; - pSorter->mnPmaSize = szPma * pgsz; - - mxCache = SQL_DEFAULT_CACHE_SIZE; - mxCache = mxCache * -1024; - mxCache = MIN(mxCache, SQL_MAX_PMASZ); - pSorter->mxPmaSize = - MAX(pSorter->mnPmaSize, (int)mxCache); - - /* EVIDENCE-OF: R-26747-61719 When the application provides any amount of - * scratch memory using SQL_CONFIG_SCRATCH, sql avoids unnecessary - * large heap allocations. - */ - if (sqlGlobalConfig.pScratch == 0) { - assert(pSorter->iMemory == 0); - pSorter->nMemory = pgsz; - pSorter->list.aMemory = - (u8 *) sqlMalloc(pgsz); - if (!pSorter->list.aMemory) - rc = -1; - } - if (pCsr->key_def->part_count < 13 - && (pCsr->key_def->parts[0].coll == NULL)) { - pSorter->typeMask = - SORTER_TYPE_INTEGER | SORTER_TYPE_TEXT; - } - } + if (pCsr->key_def->part_count < 13 && + pCsr->key_def->parts[0].coll == NULL) + pSorter->typeMask = SORTER_TYPE_INTEGER | SORTER_TYPE_TEXT; return rc; } @@ -1486,9 +1475,6 @@ sqlVdbeSorterWrite(const VdbeCursor * pCsr, /* Sorter cursor */ * * * The total memory allocated for the in-memory list is greater * than (page-size * cache-size), or - * - * * The total memory allocated for the in-memory list is greater - * than (page-size * 10) and sqlHeapNearlyFull() returns true. */ nReq = pVal->n + sizeof(SorterRecord); nPMA = pVal->n + sqlVarintLen(pVal->n); @@ -1497,10 +1483,7 @@ sqlVdbeSorterWrite(const VdbeCursor * pCsr, /* Sorter cursor */ bFlush = pSorter->iMemory && (pSorter->iMemory + nReq) > pSorter->mxPmaSize; } else { - bFlush = ((pSorter->list.szPMA > pSorter->mxPmaSize) - || (pSorter->list.szPMA > pSorter->mnPmaSize - && sqlHeapNearlyFull()) - ); + bFlush = ((pSorter->list.szPMA > pSorter->mxPmaSize)); } if (bFlush) { rc = vdbeSorterFlushPMA(pSorter);