[PATCH 1/1] iproto: protect from false-correct size in msg header

[PATCH 1/1] iproto: protect from false-correct size in msg header

[Vladislav Shpilevoy]

Consider this packet:

    msgpack = require('msgpack')
    data = msgpack.encode(18400000000000000000)..'aaaaaaa'

Tarantool interprets 18400000000000000000 as size of a coming
iproto request, and tries with no any checks to allocate buffer
of such size. It calculates needed capacity like this:

    capacity = start_value;
    while (capacity < size)
        capacity *= 2;

Here it is possible that on i-th iteration 'capacity' < 'size',
but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
so this loop never ends and occupies 100% CPU.

Strictly speaking overflow has undefined behavior. On the
original system it led to nullifying 'capacity'.

Such size is improbable as a real packet gabarits, but can appear
as a result of parsing of some invalid packet, first bytes of
which accidentally appears to be valid MessagePack uint. This is
how the bug emerged on the real system.

Lets restrict the maximal packet size to 2GB.

Closes #3464
---
Branch: https://github.com/tarantool/tarantool/tree/gerold103/gh-3464-iproto-100-cpu
Issue: https://github.com/tarantool/tarantool/issues/3464

 src/box/iproto.cc         | 15 +++++++++++++--
 test/box/net.box.result   | 22 ++++++++++++++++++++++
 test/box/net.box.test.lua | 10 ++++++++++
 3 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/src/box/iproto.cc b/src/box/iproto.cc
index 0b92c316e..d3765eb1d 100644
--- a/src/box/iproto.cc
+++ b/src/box/iproto.cc
@@ -64,7 +64,10 @@
 #include "cfg.h"
 
 /* The number of iproto messages in flight */
-enum { IPROTO_MSG_MAX = 768 };
+enum {
+	IPROTO_MSG_MAX = 768,
+	IPROTO_PACKET_SIZE_MAX = 2UL * 1024 * 1024 * 1024,
+};
 
 /**
  * Network readahead. A signed integer to avoid
@@ -524,8 +527,16 @@ iproto_connection_input_buffer(struct iproto_connection *con)
 	/* The type code is checked in iproto_enqueue_batch() */
 	if (con->parse_size) {
 		const char *pos = old_ibuf->wpos - con->parse_size;
-		if (mp_check_uint(pos, old_ibuf->wpos) <= 0)
+		if (mp_check_uint(pos, old_ibuf->wpos) <= 0) {
 			to_read = mp_decode_uint(&pos);
+			if (to_read > IPROTO_PACKET_SIZE_MAX) {
+				const char *err =
+					tt_sprintf("too big packet size in "\
+						   "the header: %llu",
+						   (unsigned long long)to_read);
+				tnt_raise(ClientError, ER_INVALID_MSGPACK, err);
+			}
+		}
 	}
 
 	if (ibuf_unused(old_ibuf) >= to_read)
diff --git a/test/box/net.box.result b/test/box/net.box.result
index aa418b87d..7180f7f66 100644
--- a/test/box/net.box.result
+++ b/test/box/net.box.result
@@ -2360,6 +2360,28 @@ box.internal.collation.drop('test')
 space:drop()
 ---
 ...
+--
+-- gh-3464: iproto hangs in 100% CPU when too big packet size
+-- is received due to size_t overflow.
+--
+c = net:connect(box.cfg.listen)
+---
+...
+data = msgpack.encode(18400000000000000000)..'aaaaaaa'
+---
+...
+c._transport.perform_request(nil, nil, 'inject', nil, data)
+---
+- 77
+- Peer closed
+...
+c:close()
+---
+...
+test_run:grep_log('default', 'too big packet size in the header') ~= nil
+---
+- true
+...
 box.schema.user.revoke('guest', 'read,write,execute', 'universe')
 ---
 ...
diff --git a/test/box/net.box.test.lua b/test/box/net.box.test.lua
index 4cadd3b29..1e716ca89 100644
--- a/test/box/net.box.test.lua
+++ b/test/box/net.box.test.lua
@@ -961,4 +961,14 @@ c:close()
 box.internal.collation.drop('test')
 space:drop()
 
+--
+-- gh-3464: iproto hangs in 100% CPU when too big packet size
+-- is received due to size_t overflow.
+--
+c = net:connect(box.cfg.listen)
+data = msgpack.encode(18400000000000000000)..'aaaaaaa'
+c._transport.perform_request(nil, nil, 'inject', nil, data)
+c:close()
+test_run:grep_log('default', 'too big packet size in the header') ~= nil
+
 box.schema.user.revoke('guest', 'read,write,execute', 'universe')
-- 
2.15.1 (Apple Git-101)

Re: [PATCH 1/1] iproto: protect from false-correct size in msg header

[Vladimir Davydov]

Looks OK to me.

On Mon, Jun 18, 2018 at 10:53:01PM +0300, Vladislav Shpilevoy wrote:
> Consider this packet:
> 
>     msgpack = require('msgpack')
>     data = msgpack.encode(18400000000000000000)..'aaaaaaa'
> 
> Tarantool interprets 18400000000000000000 as size of a coming
> iproto request, and tries with no any checks to allocate buffer
> of such size. It calculates needed capacity like this:
> 
>     capacity = start_value;
>     while (capacity < size)
>         capacity *= 2;
> 
> Here it is possible that on i-th iteration 'capacity' < 'size',
> but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
> so this loop never ends and occupies 100% CPU.
> 
> Strictly speaking overflow has undefined behavior. On the
> original system it led to nullifying 'capacity'.
> 
> Such size is improbable as a real packet gabarits, but can appear
> as a result of parsing of some invalid packet, first bytes of
> which accidentally appears to be valid MessagePack uint. This is
> how the bug emerged on the real system.
> 
> Lets restrict the maximal packet size to 2GB.
> 
> Closes #3464
> ---
> Branch: https://github.com/tarantool/tarantool/tree/gerold103/gh-3464-iproto-100-cpu
> Issue: https://github.com/tarantool/tarantool/issues/3464
> 
>  src/box/iproto.cc         | 15 +++++++++++++--
>  test/box/net.box.result   | 22 ++++++++++++++++++++++
>  test/box/net.box.test.lua | 10 ++++++++++
>  3 files changed, 45 insertions(+), 2 deletions(-)

Re: [tarantool-patches] [PATCH 1/1] iproto: protect from false-correct size in msg header

[Konstantin Osipov]

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [18/06/18 22:55]:

Please move the check for buffer overflow to
iproto_enqueue_batch(). We should not pass invalid packet lengths
to iproto_connection_input_buffer().

> Consider this packet:
> 
>     msgpack = require('msgpack')
>     data = msgpack.encode(18400000000000000000)..'aaaaaaa'
> 
> Tarantool interprets 18400000000000000000 as size of a coming
> iproto request, and tries with no any checks to allocate buffer
> of such size. It calculates needed capacity like this:
> 
>     capacity = start_value;
>     while (capacity < size)
>         capacity *= 2;
> 
> Here it is possible that on i-th iteration 'capacity' < 'size',
> but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
> so this loop never ends and occupies 100% CPU.
> 
> Strictly speaking overflow has undefined behavior. On the
> original system it led to nullifying 'capacity'.
> 
> Such size is improbable as a real packet gabarits, but can appear
> as a result of parsing of some invalid packet, first bytes of
> which accidentally appears to be valid MessagePack uint. This is
> how the bug emerged on the real system.
> 
> Lets restrict the maximal packet size to 2GB.
> 
> Closes #3464
> ---
> Branch: https://github.com/tarantool/tarantool/tree/gerold103/gh-3464-iproto-100-cpu
> Issue: https://github.com/tarantool/tarantool/issues/3464
> 
>  src/box/iproto.cc         | 15 +++++++++++++--
>  test/box/net.box.result   | 22 ++++++++++++++++++++++
>  test/box/net.box.test.lua | 10 ++++++++++
>  3 files changed, 45 insertions(+), 2 deletions(-)

-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov

[tarantool-patches] [PATCH v2 1/1] iproto: protect from false-correct size in msg header

[Vladislav Shpilevoy]

Consider this packet:

    msgpack = require('msgpack')
    data = msgpack.encode(18400000000000000000)..'aaaaaaa'

Tarantool interprets 18400000000000000000 as size of a coming
iproto request, and tries with no any checks to allocate buffer
of such size. It calculates needed capacity like this:

    capacity = start_value;
    while (capacity < size)
        capacity *= 2;

Here it is possible that on i-th iteration 'capacity' < 'size',
but 'capacity * 2' overflows 64 bits and becomes < 'size' again,
so this loop never ends and occupies 100% CPU.

Strictly speaking overflow has undefined behavior. On the
original system it led to nullifying 'capacity'.

Such size is improbable as a real packet gabarits, but can appear
as a result of parsing of some invalid packet, first bytes of
which accidentally appears to be valid MessagePack uint. This is
how the bug emerged on the real system.

Lets restrict the maximal packet size to 2GB.

Closes #3464
---
Issue: https://github.com/tarantool/tarantool/issues/3464
Branch: https://github.com/tarantool/tarantool/tree/gerold103/gh-3464-iproto-100-cpu

Changelog.
- v2: move packet size check in enqueue_batch() function.

 src/box/iproto.cc         | 19 +++++++++++++++----
 test/box/net.box.result   | 22 ++++++++++++++++++++++
 test/box/net.box.test.lua | 10 ++++++++++
 3 files changed, 47 insertions(+), 4 deletions(-)

diff --git a/src/box/iproto.cc b/src/box/iproto.cc
index 0b92c316e..d69958962 100644
--- a/src/box/iproto.cc
+++ b/src/box/iproto.cc
@@ -64,7 +64,10 @@
 #include "cfg.h"
 
 /* The number of iproto messages in flight */
-enum { IPROTO_MSG_MAX = 768 };
+enum {
+	IPROTO_MSG_MAX = 768,
+	IPROTO_PACKET_SIZE_MAX = 2UL * 1024 * 1024 * 1024,
+};
 
 /**
  * Network readahead. A signed integer to avoid
@@ -581,18 +584,26 @@ iproto_enqueue_batch(struct iproto_connection *con, struct ibuf *in)
 {
 	int n_requests = 0;
 	bool stop_input = false;
+	const char *errmsg;
 	while (con->parse_size && stop_input == false) {
 		const char *reqstart = in->wpos - con->parse_size;
 		const char *pos = reqstart;
 		/* Read request length. */
 		if (mp_typeof(*pos) != MP_UINT) {
+			errmsg = "packet length";
+err_msgpack:
 			cpipe_flush_input(&tx_pipe);
-			tnt_raise(ClientError, ER_INVALID_MSGPACK,
-				  "packet length");
+			tnt_raise(ClientError, ER_INVALID_MSGPACK, errmsg);
 		}
 		if (mp_check_uint(pos, in->wpos) >= 0)
 			break;
-		uint32_t len = mp_decode_uint(&pos);
+		uint64_t len = mp_decode_uint(&pos);
+		if (len > IPROTO_PACKET_SIZE_MAX) {
+			errmsg = tt_sprintf("too big packet size in the "\
+					    "header: %llu",
+					    (unsigned long long) len);
+			goto err_msgpack;
+		}
 		const char *reqend = pos + len;
 		if (reqend > in->wpos)
 			break;
diff --git a/test/box/net.box.result b/test/box/net.box.result
index aa418b87d..7180f7f66 100644
--- a/test/box/net.box.result
+++ b/test/box/net.box.result
@@ -2360,6 +2360,28 @@ box.internal.collation.drop('test')
 space:drop()
 ---
 ...
+--
+-- gh-3464: iproto hangs in 100% CPU when too big packet size
+-- is received due to size_t overflow.
+--
+c = net:connect(box.cfg.listen)
+---
+...
+data = msgpack.encode(18400000000000000000)..'aaaaaaa'
+---
+...
+c._transport.perform_request(nil, nil, 'inject', nil, data)
+---
+- 77
+- Peer closed
+...
+c:close()
+---
+...
+test_run:grep_log('default', 'too big packet size in the header') ~= nil
+---
+- true
+...
 box.schema.user.revoke('guest', 'read,write,execute', 'universe')
 ---
 ...
diff --git a/test/box/net.box.test.lua b/test/box/net.box.test.lua
index 4cadd3b29..1e716ca89 100644
--- a/test/box/net.box.test.lua
+++ b/test/box/net.box.test.lua
@@ -961,4 +961,14 @@ c:close()
 box.internal.collation.drop('test')
 space:drop()
 
+--
+-- gh-3464: iproto hangs in 100% CPU when too big packet size
+-- is received due to size_t overflow.
+--
+c = net:connect(box.cfg.listen)
+data = msgpack.encode(18400000000000000000)..'aaaaaaa'
+c._transport.perform_request(nil, nil, 'inject', nil, data)
+c:close()
+test_run:grep_log('default', 'too big packet size in the header') ~= nil
+
 box.schema.user.revoke('guest', 'read,write,execute', 'universe')
-- 
2.15.1 (Apple Git-101)

[tarantool-patches] Re: [PATCH v2 1/1] iproto: protect from false-correct size in msg header

[Konstantin Osipov]

* Vladislav Shpilevoy <v.shpilevoy@tarantool.org> [18/06/25 17:04]:

Pushed.


-- 
Konstantin Osipov, Moscow, Russia, +7 903 626 22 32
http://tarantool.io - www.twitter.com/kostja_osipov