* [tarantool-patches] [PATCH 0/3] introduce static allocator
@ 2019-04-28 16:56 Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 1/3] small: introduce small/static Vladislav Shpilevoy
` (4 more replies)
0 siblings, 5 replies; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-04-28 16:56 UTC (permalink / raw)
To: tarantool-patches; +Cc: kostja
The patchset 1) fixes a bug about static buffers duplication in each C-file
using them, 2) introduces static memory allocator to avoid temporary on-stack
and BSS allocations.
Branch: http://github.com/tarantool/tarantool/tree/gerold103/small-static-alloc
Vladislav Shpilevoy (3):
small: introduce small/static
Use static_alloc() instead of 'static char[]' where possible
sio: optimize sio_strfaddr() for the most common case
src/box/call.c | 1 +
src/box/identifier.c | 2 +-
src/box/identifier.h | 1 -
src/box/iproto.cc | 3 +-
src/box/memtx_engine.c | 8 +--
src/box/opt_def.c | 1 +
src/box/relay.cc | 2 +-
src/box/session.cc | 1 +
src/box/space_def.c | 1 +
src/box/space_def.h | 2 +-
src/box/sql/sqlLimit.h | 2 +-
src/box/tuple.h | 1 +
src/box/tuple_format.c | 1 +
src/box/user.cc | 1 +
src/box/vclock.c | 2 +-
src/box/vy_log.c | 2 +-
src/box/xlog.c | 24 +++----
src/box/xlog.h | 2 +-
src/box/xrow.c | 2 +-
src/cfg.c | 15 ++---
src/httpc.c | 7 +-
src/lib/coll/coll.c | 2 +-
src/lib/core/fio.c | 2 +-
src/lib/core/say.c | 1 +
src/lib/core/sio.c | 24 ++++---
src/lib/core/tt_static.h | 118 ++++++++++++++++++++++++++++++++++
src/lib/small | 2 +-
src/lib/swim/swim_io.h | 2 +-
src/lib/swim/swim_proto.h | 2 +-
src/lib/swim/swim_transport.h | 1 -
src/lib/uuid/tt_uuid.c | 2 +-
src/lua/utf8.c | 1 +
src/trivia/util.h | 53 ---------------
33 files changed, 182 insertions(+), 109 deletions(-)
create mode 100644 src/lib/core/tt_static.h
--
2.20.1 (Apple Git-117)
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] [PATCH 1/3] small: introduce small/static
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
@ 2019-04-28 16:56 ` Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 2/3] Use static_alloc() instead of 'static char[]' where possible Vladislav Shpilevoy
` (3 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-04-28 16:56 UTC (permalink / raw)
To: tarantool-patches; +Cc: kostja
Before the patch Tarantool had a thread- and C-file- local array
of 4 static buffers, each 1028 bytes. It provided an API
tt_static_buf() allowing to return them one by one in a cycle.
Firstly, it consumed totally 200Kb of BSS memory in summary over
all C-files using these buffers. Obviously, it was a bug and was
not made intentionally. The buffers were supposed to be a one
process-global array.
Secondly, even if the bug above had been fixed somehow, sometimes
it would have been needed to obtain a bit bigger buffer. For
example, to store a UDP packet - ~1.5Kb.
This commit replaces these 4 buffers with small/ static allocator
which does basically the same, but in more granulated and
manoeuvrable way. This commit frees ~188Kb of BSS section.
A main motivation for this commit is a wish to use a single
global out-of-stack buffer to read UDP packets into it in the
SWIM library, and on the other hand do not pad out BSS section
with a new SWIM-special static buffer. Now SWIM uses stack for
this and in the incoming cryptography SWIM component it will need
more.
---
src/box/call.c | 1 +
src/box/identifier.c | 2 +-
src/box/identifier.h | 1 -
src/box/iproto.cc | 1 +
src/box/opt_def.c | 1 +
src/box/relay.cc | 2 +-
src/box/session.cc | 1 +
src/box/space_def.c | 1 +
src/box/space_def.h | 2 +-
src/box/sql/sqlLimit.h | 2 +-
src/box/tuple.h | 1 +
src/box/tuple_format.c | 1 +
src/box/user.cc | 1 +
src/box/vclock.c | 2 +-
src/box/vy_log.c | 2 +-
src/box/xrow.c | 2 +-
src/lib/coll/coll.c | 2 +-
src/lib/core/fio.c | 2 +-
src/lib/core/say.c | 1 +
src/lib/core/sio.c | 9 +--
src/lib/core/tt_static.h | 118 ++++++++++++++++++++++++++++++++++
src/lib/small | 2 +-
src/lib/swim/swim_io.h | 2 +-
src/lib/swim/swim_proto.h | 2 +-
src/lib/swim/swim_transport.h | 1 -
src/lib/uuid/tt_uuid.c | 2 +-
src/lua/utf8.c | 1 +
src/trivia/util.h | 53 ---------------
28 files changed, 146 insertions(+), 72 deletions(-)
create mode 100644 src/lib/core/tt_static.h
diff --git a/src/box/call.c b/src/box/call.c
index b9750c5f3..56da53fb3 100644
--- a/src/box/call.c
+++ b/src/box/call.c
@@ -41,6 +41,7 @@
#include "iproto_constants.h"
#include "rmean.h"
#include "small/obuf.h"
+#include "tt_static.h"
/**
* Find a function by name and check "EXECUTE" permissions.
diff --git a/src/box/identifier.c b/src/box/identifier.c
index 252d53618..b1c56bdc3 100644
--- a/src/box/identifier.c
+++ b/src/box/identifier.c
@@ -32,7 +32,7 @@
#include "say.h"
#include "diag.h"
-
+#include "tt_static.h"
#include <unicode/utf8.h>
#include <unicode/uchar.h>
diff --git a/src/box/identifier.h b/src/box/identifier.h
index f34380d62..a0ed6c10e 100644
--- a/src/box/identifier.h
+++ b/src/box/identifier.h
@@ -30,7 +30,6 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
-#include "trivia/util.h"
#include <stdbool.h>
#include "error.h"
diff --git a/src/box/iproto.cc b/src/box/iproto.cc
index 10b6a6792..b9d2004f0 100644
--- a/src/box/iproto.cc
+++ b/src/box/iproto.cc
@@ -62,6 +62,7 @@
#include "rmean.h"
#include "execute.h"
#include "errinj.h"
+#include "tt_static.h"
enum {
IPROTO_SALT_SIZE = 32,
diff --git a/src/box/opt_def.c b/src/box/opt_def.c
index c78021440..e2820853b 100644
--- a/src/box/opt_def.c
+++ b/src/box/opt_def.c
@@ -35,6 +35,7 @@
#include "small/region.h"
#include "error.h"
#include "diag.h"
+#include "tt_static.h"
const char *opt_type_strs[] = {
/* [OPT_BOOL] = */ "boolean",
diff --git a/src/box/relay.cc b/src/box/relay.cc
index e3d7f0ed1..906bf8efd 100644
--- a/src/box/relay.cc
+++ b/src/box/relay.cc
@@ -31,7 +31,7 @@
#include "relay.h"
#include "trivia/config.h"
-#include "trivia/util.h"
+#include "tt_static.h"
#include "scoped_guard.h"
#include "cbus.h"
#include "cfg.h"
diff --git a/src/box/session.cc b/src/box/session.cc
index 70822e8fd..4bb13d031 100644
--- a/src/box/session.cc
+++ b/src/box/session.cc
@@ -35,6 +35,7 @@
#include "trigger.h"
#include "user.h"
#include "error.h"
+#include "tt_static.h"
const char *session_type_strs[] = {
"background",
diff --git a/src/box/space_def.c b/src/box/space_def.c
index d0864cc72..d825def75 100644
--- a/src/box/space_def.c
+++ b/src/box/space_def.c
@@ -34,6 +34,7 @@
#include "error.h"
#include "sql.h"
#include "msgpuck.h"
+#include "tt_static.h"
/**
* Make checks from msgpack.
diff --git a/src/box/space_def.h b/src/box/space_def.h
index 52ff56764..c6639a8d8 100644
--- a/src/box/space_def.h
+++ b/src/box/space_def.h
@@ -30,9 +30,9 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
-#include "trivia/util.h"
#include "tuple_dictionary.h"
#include "schema_def.h"
+#include <stdint.h>
#include <stdlib.h>
#include <stdbool.h>
diff --git a/src/box/sql/sqlLimit.h b/src/box/sql/sqlLimit.h
index 76e0d9c53..53dbe1505 100644
--- a/src/box/sql/sqlLimit.h
+++ b/src/box/sql/sqlLimit.h
@@ -35,7 +35,7 @@
*
* This file defines various limits of what sql can process.
*/
-#include "trivia/util.h"
+#include "tt_static.h"
enum {
/*
diff --git a/src/box/tuple.h b/src/box/tuple.h
index eed8e1a34..a496bf6f0 100644
--- a/src/box/tuple.h
+++ b/src/box/tuple.h
@@ -35,6 +35,7 @@
#include "diag.h"
#include "error.h"
#include "uuid/tt_uuid.h" /* tuple_field_uuid */
+#include "tt_static.h"
#include "tuple_format.h"
#if defined(__cplusplus)
diff --git a/src/box/tuple_format.c b/src/box/tuple_format.c
index 804a678a1..9f742d900 100644
--- a/src/box/tuple_format.c
+++ b/src/box/tuple_format.c
@@ -33,6 +33,7 @@
#include "json/json.h"
#include "tuple_format.h"
#include "coll_id_cache.h"
+#include "tt_static.h"
#include "third_party/PMurHash.h"
diff --git a/src/box/user.cc b/src/box/user.cc
index 403d6d6c4..48bdf18ee 100644
--- a/src/box/user.cc
+++ b/src/box/user.cc
@@ -38,6 +38,7 @@
#include "session.h"
#include "scoped_guard.h"
#include "sequence.h"
+#include "tt_static.h"
struct universe universe;
static struct user users[BOX_USER_MAX];
diff --git a/src/box/vclock.c b/src/box/vclock.c
index cea67017c..90ae27591 100644
--- a/src/box/vclock.c
+++ b/src/box/vclock.c
@@ -35,7 +35,7 @@
#include <ctype.h>
#include "diag.h"
-#include "trivia/util.h"
+#include "tt_static.h"
int64_t
vclock_follow(struct vclock *vclock, uint32_t replica_id, int64_t lsn)
diff --git a/src/box/vy_log.c b/src/box/vy_log.c
index be97cdbbe..de39b746f 100644
--- a/src/box/vy_log.c
+++ b/src/box/vy_log.c
@@ -57,7 +57,7 @@
#include "replication.h" /* INSTANCE_UUID */
#include "salad/stailq.h"
#include "say.h"
-#include "trivia/util.h"
+#include "tt_static.h"
#include "wal.h"
#include "vclock.h"
#include "xlog.h"
diff --git a/src/box/xrow.c b/src/box/xrow.c
index aaed84e38..fef4f2dff 100644
--- a/src/box/xrow.c
+++ b/src/box/xrow.c
@@ -37,7 +37,7 @@
#include "fiber.h"
#include "version.h"
-
+#include "tt_static.h"
#include "error.h"
#include "vclock.h"
#include "scramble.h"
diff --git a/src/lib/coll/coll.c b/src/lib/coll/coll.c
index 495eb6d64..98e4d0e64 100644
--- a/src/lib/coll/coll.c
+++ b/src/lib/coll/coll.c
@@ -35,7 +35,7 @@
#include "assoc.h"
#include <unicode/ucol.h>
#include <unicode/ucasemap.h>
-#include <trivia/config.h>
+#include "tt_static.h"
struct UCaseMap *icu_ucase_default_map = NULL;
diff --git a/src/lib/core/fio.c b/src/lib/core/fio.c
index b79d3d058..4388b9a53 100644
--- a/src/lib/core/fio.c
+++ b/src/lib/core/fio.c
@@ -41,7 +41,7 @@
#include <lib/bit/bit.h>
#include <say.h>
-#include <trivia/util.h>
+#include "tt_static.h"
const char *
fio_filename(int fd)
diff --git a/src/lib/core/say.c b/src/lib/core/say.c
index 3b13b766a..24963cf82 100644
--- a/src/lib/core/say.c
+++ b/src/lib/core/say.c
@@ -31,6 +31,7 @@
#include "say.h"
#include "fiber.h"
#include "errinj.h"
+#include "tt_static.h"
#include <errno.h>
#include <stdarg.h>
diff --git a/src/lib/core/sio.c b/src/lib/core/sio.c
index 3e1b7e784..64aae97bc 100644
--- a/src/lib/core/sio.c
+++ b/src/lib/core/sio.c
@@ -38,12 +38,12 @@
#include <netinet/tcp.h> /* TCP_NODELAY */
#include <arpa/inet.h>
#include "say.h"
-#include "trivia/util.h"
+#include "tt_static.h"
#include "exception.h"
#include "uri/uri.h"
#include "errinj.h"
-static_assert(TT_STATIC_BUF_LEN > NI_MAXHOST,
+static_assert(SMALL_STATIC_SIZE > NI_MAXHOST + NI_MAXSERV,
"static buffer should fit host name");
const char *
@@ -330,8 +330,9 @@ sio_strfaddr(const struct sockaddr *addr, socklen_t addrlen)
NI_NUMERICHOST | NI_NUMERICSERV) != 0)
return tt_sprintf("(host):(port)");
- return tt_sprintf(addr->sa_family == AF_INET ?
- "%s:%s" : "[%s]:%s", host, serv);
+ return tt_snprintf(NI_MAXHOST + NI_MAXSERV,
+ addr->sa_family == AF_INET ?
+ "%s:%s" : "[%s]:%s", host, serv);
}
}
unreachable();
diff --git a/src/lib/core/tt_static.h b/src/lib/core/tt_static.h
new file mode 100644
index 000000000..5621c640c
--- /dev/null
+++ b/src/lib/core/tt_static.h
@@ -0,0 +1,118 @@
+#ifndef TARANTOOL_CORE_STATIC_H_INCLUDED
+#define TARANTOOL_CORE_STATIC_H_INCLUDED
+/*
+ * Copyright 2010-2019, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+#include "small/static.h"
+#include <string.h>
+#include <stdio.h>
+#include <stdarg.h>
+#include <assert.h>
+
+#ifndef MIN
+#define UNDEF_MIN
+#define MIN(a, b) ((a) < (b) ? (a) : (b))
+#endif
+
+/** Can't be enum - used by preprocessor for static assertions. */
+#define TT_STATIC_BUF_LEN 1024
+
+/**
+ * Return a thread-local statically allocated temporary buffer of
+ * size @a TT_STATIC_BUF_LEN.
+ */
+static inline char *
+tt_static_buf(void)
+{
+ return static_alloc(TT_STATIC_BUF_LEN);
+}
+
+/**
+ * Return a null-terminated string for @a str of length @a len.
+ */
+static inline const char *
+tt_cstr(const char *str, size_t len)
+{
+ len = MIN(len, SMALL_STATIC_SIZE - 1);
+ char *buf = static_alloc(len + 1);
+ memcpy(buf, str, len);
+ buf[len] = '\0';
+ return buf;
+}
+
+/**
+ * Wrapper around vsnprintf() that prints the result to
+ * the static buffer.
+ */
+static inline const char *
+tt_vsnprintf(size_t size, const char *format, va_list ap)
+{
+ char *buf = static_reserve(size);
+ assert(buf != NULL);
+ int rc = vsnprintf(buf, size, format, ap);
+ if (rc >= 0) {
+ /* +1 for terminating zero. */
+ rc = MIN((int) size, rc + 1);
+ static_alloc(rc);
+ }
+ return buf;
+}
+
+/** @copydoc tt_vsnprintf() */
+static inline const char *
+tt_sprintf(const char *format, ...)
+{
+ va_list ap;
+ va_start(ap, format);
+ const char *result = tt_vsnprintf(TT_STATIC_BUF_LEN, format, ap);
+ va_end(ap);
+ return result;
+}
+
+/**
+ * The same as tt_sprintf() but allows to specify more precise
+ * string limits.
+ */
+static inline const char *
+tt_snprintf(size_t size, const char *format, ...)
+{
+ va_list ap;
+ va_start(ap, format);
+ const char *result = tt_vsnprintf(MIN(size, (size_t) SMALL_STATIC_SIZE),
+ format, ap);
+ va_end(ap);
+ return result;
+}
+
+#ifdef UNDEF_MIN
+#undef MIN
+#endif
+
+#endif /* TARANTOOL_CORE_STATIC_H_INCLUDED */
diff --git a/src/lib/small b/src/lib/small
index cdf7d4a8e..50ba09107 160000
--- a/src/lib/small
+++ b/src/lib/small
@@ -1 +1 @@
-Subproject commit cdf7d4a8e24ae465298d0ddacaf1aaed1085281e
+Subproject commit 50ba091077c074c4eecc1ee63f53d2f4f245bd51
diff --git a/src/lib/swim/swim_io.h b/src/lib/swim/swim_io.h
index b17629ba9..42d94fc92 100644
--- a/src/lib/swim/swim_io.h
+++ b/src/lib/swim/swim_io.h
@@ -30,7 +30,7 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
-#include "trivia/util.h"
+#include "tt_static.h"
#include "small/rlist.h"
#include "salad/stailq.h"
#include "swim_transport.h"
diff --git a/src/lib/swim/swim_proto.h b/src/lib/swim/swim_proto.h
index 81770c01c..beb9bb1fe 100644
--- a/src/lib/swim/swim_proto.h
+++ b/src/lib/swim/swim_proto.h
@@ -30,7 +30,7 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
-#include "trivia/util.h"
+#include "tt_static.h"
#include "uuid/tt_uuid.h"
#include <arpa/inet.h>
#include <netinet/in.h>
diff --git a/src/lib/swim/swim_transport.h b/src/lib/swim/swim_transport.h
index 6f4370087..c4e48cdeb 100644
--- a/src/lib/swim/swim_transport.h
+++ b/src/lib/swim/swim_transport.h
@@ -30,7 +30,6 @@
* THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
-#include "trivia/util.h"
#include <netinet/in.h>
#include <arpa/inet.h>
diff --git a/src/lib/uuid/tt_uuid.c b/src/lib/uuid/tt_uuid.c
index 530aa31a1..1bd2e2cfe 100644
--- a/src/lib/uuid/tt_uuid.c
+++ b/src/lib/uuid/tt_uuid.c
@@ -33,7 +33,7 @@
#include <msgpuck.h>
#include <random.h>
#include <trivia/config.h>
-#include <trivia/util.h>
+#include "tt_static.h"
/* Zeroed by the linker. */
const struct tt_uuid uuid_nil;
diff --git a/src/lua/utf8.c b/src/lua/utf8.c
index d829b57b4..9c678cad4 100644
--- a/src/lua/utf8.c
+++ b/src/lua/utf8.c
@@ -35,6 +35,7 @@
#include "lua/utf8.h"
#include "diag.h"
#include "small/ibuf.h"
+#include "tt_static.h"
extern struct ibuf *tarantool_lua_ibuf;
diff --git a/src/trivia/util.h b/src/trivia/util.h
index a3e84e6d5..1e01013db 100644
--- a/src/trivia/util.h
+++ b/src/trivia/util.h
@@ -479,59 +479,6 @@ int clock_gettime(uint32_t clock_id, struct timespec *tp);
#define CLOCK_THREAD_CPUTIME_ID 3
#endif
-#define TT_STATIC_BUF_LEN 1028
-
-/**
- * Return a thread-local statically allocated temporary buffer of size
- * \a TT_STATIC_BUF_LEN
- */
-static inline char *
-tt_static_buf(void)
-{
- enum { TT_STATIC_BUFS = 4 };
- static __thread char bufs[TT_STATIC_BUFS][TT_STATIC_BUF_LEN];
- static __thread int bufno = TT_STATIC_BUFS - 1;
-
- bufno = (bufno + 1) % TT_STATIC_BUFS;
- return bufs[bufno];
-}
-
-/**
- * Return a null-terminated string for \a str of length \a len
- */
-static inline const char *
-tt_cstr(const char *str, size_t len)
-{
- char *buf = tt_static_buf();
- len = MIN(len, TT_STATIC_BUF_LEN - 1);
- memcpy(buf, str, len);
- buf[len] = '\0';
- return buf;
-}
-
-/**
- * Wrapper around sprintf() that prints the result to
- * the static buffer returned by tt_static_buf().
- */
-static inline const char *
-tt_vsprintf(const char *format, va_list ap)
-{
- char *buf = tt_static_buf();
- vsnprintf(buf, TT_STATIC_BUF_LEN, format, ap);
- return buf;
-}
-
-/** @copydoc tt_vsprintf() */
-static inline const char *
-tt_sprintf(const char *format, ...)
-{
- va_list ap;
- va_start(ap, format);
- const char *result = tt_vsprintf(format, ap);
- va_end(ap);
- return result;
-}
-
/**
* Escape special characters in @a data to @a buf
*/
--
2.20.1 (Apple Git-117)
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] [PATCH 2/3] Use static_alloc() instead of 'static char[]' where possible
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 1/3] small: introduce small/static Vladislav Shpilevoy
@ 2019-04-28 16:56 ` Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case Vladislav Shpilevoy
` (2 subsequent siblings)
4 siblings, 0 replies; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-04-28 16:56 UTC (permalink / raw)
To: tarantool-patches; +Cc: kostja
This patch harnesses freshly introduced static memory allocator
to eliminate wasteful usage of BSS memory. This commit frees
11Kb per each thread.
---
src/box/iproto.cc | 2 +-
src/box/memtx_engine.c | 8 ++++----
src/box/xlog.c | 24 ++++++++++--------------
src/box/xlog.h | 2 +-
src/cfg.c | 15 +++++----------
src/httpc.c | 7 +++++--
src/lib/core/sio.c | 10 +++++-----
7 files changed, 31 insertions(+), 37 deletions(-)
diff --git a/src/box/iproto.cc b/src/box/iproto.cc
index b9d2004f0..873548e09 100644
--- a/src/box/iproto.cc
+++ b/src/box/iproto.cc
@@ -1810,7 +1810,7 @@ tx_process_connect(struct cmsg *m)
diag_raise();
con->session->meta.connection = con;
tx_fiber_init(con->session, 0);
- static __thread char greeting[IPROTO_GREETING_SIZE];
+ char *greeting = static_alloc(IPROTO_GREETING_SIZE);
/* TODO: dirty read from tx thread */
struct tt_uuid uuid = INSTANCE_UUID;
random_bytes(con->salt, IPROTO_SALT_SIZE);
diff --git a/src/box/memtx_engine.c b/src/box/memtx_engine.c
index 4d99910cb..64a311b64 100644
--- a/src/box/memtx_engine.c
+++ b/src/box/memtx_engine.c
@@ -771,7 +771,7 @@ memtx_engine_commit_checkpoint(struct engine *engine,
char to[PATH_MAX];
snprintf(to, sizeof(to), "%s",
xdir_format_filename(dir, lsn, NONE));
- char *from = xdir_format_filename(dir, lsn, INPROGRESS);
+ const char *from = xdir_format_filename(dir, lsn, INPROGRESS);
#ifndef NDEBUG
struct errinj *delay = errinj(ERRINJ_SNAP_COMMIT_DELAY,
ERRINJ_BOOL);
@@ -814,7 +814,7 @@ memtx_engine_abort_checkpoint(struct engine *engine)
small_alloc_setopt(&memtx->alloc, SMALL_DELAYED_FREE_MODE, false);
/** Remove garbage .inprogress file. */
- char *filename =
+ const char *filename =
xdir_format_filename(&memtx->checkpoint->dir,
vclock_sum(&memtx->checkpoint->vclock),
INPROGRESS);
@@ -837,8 +837,8 @@ memtx_engine_backup(struct engine *engine, const struct vclock *vclock,
engine_backup_cb cb, void *cb_arg)
{
struct memtx_engine *memtx = (struct memtx_engine *)engine;
- char *filename = xdir_format_filename(&memtx->snap_dir,
- vclock_sum(vclock), NONE);
+ const char *filename = xdir_format_filename(&memtx->snap_dir,
+ vclock_sum(vclock), NONE);
return cb(filename, cb_arg);
}
diff --git a/src/box/xlog.c b/src/box/xlog.c
index b70e8e25f..8254cce20 100644
--- a/src/box/xlog.c
+++ b/src/box/xlog.c
@@ -41,7 +41,7 @@
#include <msgpuck.h>
#include "coio_file.h"
-
+#include "tt_static.h"
#include "error.h"
#include "xrow.h"
#include "iproto_constants.h"
@@ -646,17 +646,14 @@ xdir_check(struct xdir *dir)
return 0;
}
-char *
+const char *
xdir_format_filename(struct xdir *dir, int64_t signature,
enum log_suffix suffix)
{
- static __thread char filename[PATH_MAX + 1];
- const char *suffix_str = (suffix == INPROGRESS ?
- inprogress_suffix : "");
- snprintf(filename, PATH_MAX, "%s/%020lld%s%s",
- dir->dirname, (long long) signature,
- dir->filename_ext, suffix_str);
- return filename;
+ return tt_snprintf(PATH_MAX + 1, "%s/%020lld%s%s",
+ dir->dirname, (long long) signature,
+ dir->filename_ext, suffix == INPROGRESS ?
+ inprogress_suffix : "");
}
static void
@@ -683,8 +680,8 @@ xdir_collect_garbage(struct xdir *dir, int64_t signature, unsigned flags)
struct vclock *vclock;
while ((vclock = vclockset_first(&dir->index)) != NULL &&
vclock_sum(vclock) < signature) {
- char *filename = xdir_format_filename(dir, vclock_sum(vclock),
- NONE);
+ const char *filename =
+ xdir_format_filename(dir, vclock_sum(vclock), NONE);
if (flags & XDIR_GC_ASYNC)
eio_unlink(filename, 0, xdir_complete_gc, NULL);
else
@@ -946,9 +943,8 @@ err:
int
xdir_touch_xlog(struct xdir *dir, const struct vclock *vclock)
{
- char *filename;
int64_t signature = vclock_sum(vclock);
- filename = xdir_format_filename(dir, signature, NONE);
+ const char *filename = xdir_format_filename(dir, signature, NONE);
if (dir->type != SNAP) {
assert(false);
@@ -987,7 +983,7 @@ xdir_create_xlog(struct xdir *dir, struct xlog *xlog,
xlog_meta_create(&meta, dir->filetype, dir->instance_uuid,
vclock, prev_vclock);
- char *filename = xdir_format_filename(dir, signature, NONE);
+ const char *filename = xdir_format_filename(dir, signature, NONE);
if (xlog_create(xlog, filename, dir->open_wflags, &meta,
&dir->opts) != 0)
return -1;
diff --git a/src/box/xlog.h b/src/box/xlog.h
index 6539d1b0b..a48b05fc4 100644
--- a/src/box/xlog.h
+++ b/src/box/xlog.h
@@ -199,7 +199,7 @@ xdir_check(struct xdir *dir);
* Return a file name based on directory type, vector clock
* sum, and a suffix (.inprogress or not).
*/
-char *
+const char *
xdir_format_filename(struct xdir *dir, int64_t signature,
enum log_suffix suffix);
diff --git a/src/cfg.c b/src/cfg.c
index 78654be62..46cff1999 100644
--- a/src/cfg.c
+++ b/src/cfg.c
@@ -31,14 +31,15 @@
#include "say.h"
#include "cfg.h"
#include "lua/utils.h"
+#include "tt_static.h"
-enum { MAX_OPT_NAME_LEN = 256, MAX_OPT_VAL_LEN = 256, MAX_STR_OPTS = 8 };
+enum { MAX_OPT_NAME_LEN = 256, MAX_OPT_VAL_LEN = 256 };
static void
cfg_get(const char *param)
{
- char buf[MAX_OPT_NAME_LEN];
- snprintf(buf, sizeof(buf), "return box.cfg.%s", param);
+ const char *buf =
+ tt_snprintf(MAX_OPT_NAME_LEN, "return box.cfg.%s", param);
if (luaL_dostring(tarantool_L, buf) != 0)
panic("cfg_get('%s')", param);
}
@@ -90,15 +91,9 @@ cfg_geti64(const char *param)
static const char *
cfg_tostring(struct lua_State *L)
{
- static __thread char values[MAX_STR_OPTS][MAX_OPT_VAL_LEN];
- static __thread int i = 0;
if (lua_isnil(L, -1))
return NULL;
- else {
- snprintf(values[i % MAX_STR_OPTS], MAX_OPT_VAL_LEN,
- "%s", lua_tostring(L, -1));
- return values[i++ % MAX_STR_OPTS];
- }
+ return tt_snprintf(MAX_OPT_VAL_LEN, "%s", lua_tostring(L, -1));
}
const char *
diff --git a/src/httpc.c b/src/httpc.c
index 65eeaa743..80dabd59c 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -33,12 +33,15 @@
#include <assert.h>
#include <curl/curl.h>
-
+#include "tt_static.h"
#include "fiber.h"
#include "errinj.h"
#define MAX_HEADER_LEN 8192
+static_assert(MAX_HEADER_LEN < SMALL_STATIC_SIZE,
+ "HTTP header fits into the static buffer");
+
/** The HTTP headers that may be set automatically. */
#define HTTP_ACCEPT_HEADER "Accept:"
#define HTTP_CONNECTION_HEADER "Connection:"
@@ -178,7 +181,7 @@ httpc_request_delete(struct httpc_request *req)
int
httpc_set_header(struct httpc_request *req, const char *fmt, ...)
{
- static __thread char header[MAX_HEADER_LEN + 1];
+ char *header = static_alloc(MAX_HEADER_LEN + 1);
va_list ap;
va_start(ap, fmt);
int rc = vsnprintf(header, MAX_HEADER_LEN + 1, fmt, ap);
diff --git a/src/lib/core/sio.c b/src/lib/core/sio.c
index 64aae97bc..996b7faad 100644
--- a/src/lib/core/sio.c
+++ b/src/lib/core/sio.c
@@ -51,22 +51,22 @@ sio_socketname(int fd)
{
/* Preserve errno */
int save_errno = errno;
- static __thread char name[2 * SERVICE_NAME_MAXLEN];
- int n = snprintf(name, sizeof(name), "fd %d", fd);
+ int name_size = 2 * SERVICE_NAME_MAXLEN;
+ char *name = static_alloc(name_size);
+ int n = snprintf(name, name_size, "fd %d", fd);
if (fd >= 0) {
struct sockaddr_storage addr;
socklen_t addrlen = sizeof(addr);
int rc = getsockname(fd, (struct sockaddr *) &addr, &addrlen);
if (rc == 0) {
- n += snprintf(name + n,
- sizeof(name) - n, ", aka %s",
+ n += snprintf(name + n, name_size - n, ", aka %s",
sio_strfaddr((struct sockaddr *)&addr,
addrlen));
}
addrlen = sizeof(addr);
rc = getpeername(fd, (struct sockaddr *) &addr, &addrlen);
if (rc == 0) {
- n += snprintf(name + n, sizeof(name) - n,
+ n += snprintf(name + n, name_size - n,
", peer of %s",
sio_strfaddr((struct sockaddr *)&addr,
addrlen));
--
2.20.1 (Apple Git-117)
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 1/3] small: introduce small/static Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 2/3] Use static_alloc() instead of 'static char[]' where possible Vladislav Shpilevoy
@ 2019-04-28 16:56 ` Vladislav Shpilevoy
2019-05-14 8:09 ` Vladimir Davydov
2019-04-29 4:14 ` [tarantool-patches] Re: [PATCH 0/3] introduce static allocator Konstantin Osipov
2019-05-13 22:57 ` Vladislav Shpilevoy
4 siblings, 1 reply; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-04-28 16:56 UTC (permalink / raw)
To: tarantool-patches; +Cc: kostja
SIO library provides a wrapper for getnameinfo able to stringify
Unix socket addresses. But it does not care about limited
Tarantool stack and allocates buffers for getnameinfo() right on
it - ~1Kb. Besides, after successful getnameinfo() the result is
copied onto another static buffer.
This patch optimizes sio_strfaddr() for the most common case -
AF_INET, when 32 bytes is more than enough for any IP:Port pair,
and writes the result into the target buffer directly.
The main motivation behind this commit is that SWIM makes active
use of sio_strfaddr() for logging - for each received/sent
message it writes a couple of addresses into a log. It does it in
verbose mode, but the say() function arguments are still
calculated even when the active mode is lower than verbose.
---
src/lib/core/sio.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/src/lib/core/sio.c b/src/lib/core/sio.c
index 996b7faad..28aef1d6c 100644
--- a/src/lib/core/sio.c
+++ b/src/lib/core/sio.c
@@ -323,6 +323,12 @@ sio_strfaddr(const struct sockaddr *addr, socklen_t addrlen)
return tt_sprintf("unix/:(socket)");
break;
}
+ case AF_INET: {
+ struct sockaddr_in *in = (struct sockaddr_in *) addr;
+ return tt_snprintf(SERVICE_NAME_MAXLEN, "%s:%d",
+ inet_ntoa(in->sin_addr),
+ ntohs(in->sin_port));
+ }
default: {
char host[NI_MAXHOST], serv[NI_MAXSERV];
if (getnameinfo(addr, addrlen, host, sizeof(host),
@@ -330,9 +336,8 @@ sio_strfaddr(const struct sockaddr *addr, socklen_t addrlen)
NI_NUMERICHOST | NI_NUMERICSERV) != 0)
return tt_sprintf("(host):(port)");
- return tt_snprintf(NI_MAXHOST + NI_MAXSERV,
- addr->sa_family == AF_INET ?
- "%s:%s" : "[%s]:%s", host, serv);
+ return tt_snprintf(NI_MAXHOST + NI_MAXSERV, "[%s]:%s",
+ host, serv);
}
}
unreachable();
--
2.20.1 (Apple Git-117)
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] Re: [PATCH 0/3] introduce static allocator
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
` (2 preceding siblings ...)
2019-04-28 16:56 ` [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case Vladislav Shpilevoy
@ 2019-04-29 4:14 ` Konstantin Osipov
2019-04-29 8:35 ` Vladislav Shpilevoy
2019-05-13 22:57 ` Vladislav Shpilevoy
4 siblings, 1 reply; 8+ messages in thread
From: Konstantin Osipov @ 2019-04-29 4:14 UTC (permalink / raw)
To: Vladislav Shpilevoy; +Cc: tarantool-patches
* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/04/28 20:01]:
I don't have other comments for this patch set, except that I
think you should ditch tt_static.h, but I'd like to take a look at
the diff for small/ (can't find it) before okaying it.
--
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] Re: [PATCH 0/3] introduce static allocator
2019-04-29 4:14 ` [tarantool-patches] Re: [PATCH 0/3] introduce static allocator Konstantin Osipov
@ 2019-04-29 8:35 ` Vladislav Shpilevoy
0 siblings, 0 replies; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-04-29 8:35 UTC (permalink / raw)
To: Konstantin Osipov; +Cc: tarantool-patches
On 29/04/2019 07:14, Konstantin Osipov wrote:
> * Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [19/04/28 20:01]:
>
> I don't have other comments for this patch set, except that I
> think you should ditch tt_static.h, but I'd like to take a look at
> the diff for small/ (can't find it) before okaying it.
The patch is sent in a separate thread, because tarantool/small is
a separate repository.
>
>
> --
> Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* [tarantool-patches] Re: [PATCH 0/3] introduce static allocator
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
` (3 preceding siblings ...)
2019-04-29 4:14 ` [tarantool-patches] Re: [PATCH 0/3] introduce static allocator Konstantin Osipov
@ 2019-05-13 22:57 ` Vladislav Shpilevoy
4 siblings, 0 replies; 8+ messages in thread
From: Vladislav Shpilevoy @ 2019-05-13 22:57 UTC (permalink / raw)
To: tarantool-patches; +Cc: kostja
Pushed into the master, but with a change: tt_static_buf()
now uses static_aligned_alloc with alignment by 'intptr_t'.
So it can be used to allocate temporary arrays of any types.
Note, that I kept tt_static.h header, because small/static is
an allocator, not a string formatter. And I decided, that it
should not include string-related utils, stdio.h header etc.
On 28/04/2019 19:56, Vladislav Shpilevoy wrote:
> The patchset 1) fixes a bug about static buffers duplication in each C-file
> using them, 2) introduces static memory allocator to avoid temporary on-stack
> and BSS allocations.
>
> Branch: http://github.com/tarantool/tarantool/tree/gerold103/small-static-alloc
>
> Vladislav Shpilevoy (3):
> small: introduce small/static
> Use static_alloc() instead of 'static char[]' where possible
> sio: optimize sio_strfaddr() for the most common case
>
> src/box/call.c | 1 +
> src/box/identifier.c | 2 +-
> src/box/identifier.h | 1 -
> src/box/iproto.cc | 3 +-
> src/box/memtx_engine.c | 8 +--
> src/box/opt_def.c | 1 +
> src/box/relay.cc | 2 +-
> src/box/session.cc | 1 +
> src/box/space_def.c | 1 +
> src/box/space_def.h | 2 +-
> src/box/sql/sqlLimit.h | 2 +-
> src/box/tuple.h | 1 +
> src/box/tuple_format.c | 1 +
> src/box/user.cc | 1 +
> src/box/vclock.c | 2 +-
> src/box/vy_log.c | 2 +-
> src/box/xlog.c | 24 +++----
> src/box/xlog.h | 2 +-
> src/box/xrow.c | 2 +-
> src/cfg.c | 15 ++---
> src/httpc.c | 7 +-
> src/lib/coll/coll.c | 2 +-
> src/lib/core/fio.c | 2 +-
> src/lib/core/say.c | 1 +
> src/lib/core/sio.c | 24 ++++---
> src/lib/core/tt_static.h | 118 ++++++++++++++++++++++++++++++++++
> src/lib/small | 2 +-
> src/lib/swim/swim_io.h | 2 +-
> src/lib/swim/swim_proto.h | 2 +-
> src/lib/swim/swim_transport.h | 1 -
> src/lib/uuid/tt_uuid.c | 2 +-
> src/lua/utf8.c | 1 +
> src/trivia/util.h | 53 ---------------
> 33 files changed, 182 insertions(+), 109 deletions(-)
> create mode 100644 src/lib/core/tt_static.h
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case
2019-04-28 16:56 ` [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case Vladislav Shpilevoy
@ 2019-05-14 8:09 ` Vladimir Davydov
0 siblings, 0 replies; 8+ messages in thread
From: Vladimir Davydov @ 2019-05-14 8:09 UTC (permalink / raw)
To: Vladislav Shpilevoy; +Cc: tarantool-patches, kostja
On Sun, Apr 28, 2019 at 07:56:27PM +0300, Vladislav Shpilevoy wrote:
> The main motivation behind this commit is that SWIM makes active
> use of sio_strfaddr() for logging - for each received/sent
> message it writes a couple of addresses into a log. It does it in
> verbose mode, but the say() function arguments are still
> calculated even when the active mode is lower than verbose.
They are not as say() is a macro.
At compile time
say(level, error, format, ...)
is expanded into
if (say_log_level_is_enabled(level))
_say(level, file, line, error, format, ...);
That is, neither format nor its arguments are evaluated at runtime
unless say_log_level_is_enabled(level) returns true.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-05-14 8:09 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-28 16:56 [tarantool-patches] [PATCH 0/3] introduce static allocator Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 1/3] small: introduce small/static Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 2/3] Use static_alloc() instead of 'static char[]' where possible Vladislav Shpilevoy
2019-04-28 16:56 ` [tarantool-patches] [PATCH 3/3] sio: optimize sio_strfaddr() for the most common case Vladislav Shpilevoy
2019-05-14 8:09 ` Vladimir Davydov
2019-04-29 4:14 ` [tarantool-patches] Re: [PATCH 0/3] introduce static allocator Konstantin Osipov
2019-04-29 8:35 ` Vladislav Shpilevoy
2019-05-13 22:57 ` Vladislav Shpilevoy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox