Tarantool development patches archive
 help / color / mirror / Atom feed
* [tarantool-patches] [PATCH 1/1] rfc: describe a Tarantool wire protocol
@ 2018-04-04 21:22 Vladislav Shpilevoy
  2018-04-05  8:25 ` [tarantool-patches] " Konstantin Osipov
       [not found] ` <CAFoyxqh0QqNBVr7tuFF_uoUw-CvBKOVA3FCyWvixikberOxP9w@mail.gmail.com>
  0 siblings, 2 replies; 7+ messages in thread
From: Vladislav Shpilevoy @ 2018-04-04 21:22 UTC (permalink / raw)
  To: tarantool-patches; +Cc: kostja, alg1973, Vladislav Shpilevoy

---
Original: https://github.com/tarantool/tarantool/blob/sql-proto-rfc/doc/RFC/wire_protocol.md

 doc/RFC/wire_protocol.md       | 214 +++++++++++++++++++++++++++++++++++++++++
 doc/RFC/wire_protocol_img1.png | Bin 0 -> 48970 bytes
 2 files changed, 214 insertions(+)
 create mode 100644 doc/RFC/wire_protocol.md
 create mode 100644 doc/RFC/wire_protocol_img1.png

diff --git a/doc/RFC/wire_protocol.md b/doc/RFC/wire_protocol.md
new file mode 100644
index 000000000..d1fc1329c
--- /dev/null
+++ b/doc/RFC/wire_protocol.md
@@ -0,0 +1,214 @@
+# Tarantool Wire protocol
+
+* **Status**: In progress
+* **Start date**: 04-04-2018
+* **Authors**: Vladislav Shpilevoy @Gerold103 v.shpilevoy@tarantool.org, Konstantin Osipov @kostja kostja@tarantool.org, Alexey Gadzhiev @alg1973 alg1973@gmail.com
+* **Issues**: [#2677](https://github.com/tarantool/tarantool/issues/2677), [#2620](https://github.com/tarantool/tarantool/issues/2620), [#2618](https://github.com/tarantool/tarantool/issues/2618)
+
+## Summary
+
+Tarantool wire protocol is a convention how to encode and send results of execution of SQL, Lua and C stored functions, DML (Data Manipulation Language), DDL (Data Definition Language), DQL (Data Query Language) requests to remote clients via network. The protocol is unified for all request types. For a single request multiple responses of different types can be sent.
+
+## Background and motivation
+
+Tarantool wire protocol is called **IProto**, and is used by database connectors written on different languages and working on remote clients. The protocol describes how to distinguish different message types and what data can be stored in each message. Tarantool has the following response types:
+* A response, that finalizes a request, and has just data - tuples array, or scalar values, or mixed. It has no any metadata. This response type incorporates results of any pure Lua and C calls including stored procedures, space and index methods. Such response is single per request;
+* A response with just data, but with no request finalization - it is so called push-message. During single request execution multiple pushes can be sent, and they do not finalize the request - a client must be ready to receive more responses;
+* A formatted response, that is sent on SQL DQL and does not finalize a request. Such response contains metadata with result set column names, types, flags etc;
+* A response with metadata only, that is sent on SQL DDL/DML requests, and contains affected row count, last autoincrement column value, flags;
+* A response with error message and code, that finalizes a request.
+
+In supporting this responses set 2 main challenges appear:
+1. How to unify responses;
+2. How to support multiple messages inside a single request.
+
+To understand how a single request can produce multiple responses, consider the stored procedure (do not pay attention to the syntax - it does not matter here):
+```SQL
+FUNCTION my_sql_func(a1, a2, a3, a4) BEGIN
+    SELECT my_lua_func(a1);
+    SELECT * FROM table1;
+    SELECT my_c_func(a2);
+    INSERT INTO table1 VALUES (1, 2, 3);
+    RETURN a4;
+END
+```
+, where `my_lua_func()` is the function, written in Lua and sending its own push-messages:
+```Lua
+function my_lua_func(arg)
+    box.session.push(arg)
+    return arg
+end
+```
+and `my_c_func()` is the function, written in C and returning some raw data:
+```C
+int
+my_c_func(box_function_ctx_t *ctx) {
+    box_tuple_t *tuple;
+    /* Fill a tuple with any data. */
+    return box_return_tuple(ctx, tuple);
+}
+```
+Consider each statement:
+* `SELECT FROM` can split a big result set in multiple messages;
+* `SELECT my_lua_func()` produces 2 messages: one is the push-message generated in `my_lua_func` and another is the result of `SELECT` itself;
+* `INSERT` creates 1 message with metadata;
+* `RETURN` creates a final response message.
+
+Of course, some of messages, or even all of them can be batched and send as a single TCP packet, but it does not matter for the wire protocol.
+
+In the next section it is described, how the Tarantool wire protocol deals with this mess.
+
+For the protocol details - code values, all header and body keys - see Tarantool [website](tarantool.io).
+
+## Detailed design
+
+Tarantool response consists of a body and a header. Header is used to store response code and some internal metainfo such as schema version, request id (called **sync** in Tarantool). Body is used to store result data and request-dependent metainfo.
+
+### Header
+
+There are 3 response codes in header:
+* `IPROTO_OK` - the last response in a request, that is finished successfully;
+* `IPROTO_CHUNK` - non-final response. One request can generate multuple chunk messages;
+* `IPROTO_ERROR | error code` - the last response in a request, that is finished with an error.
+
+`IPROTO_ERROR` response is trivial, and consists just of code and message. It is no considered further.
+`IPROTO_OK` and `IPROTO_CHUNK` have the same body format. The only exception between them is that `IPROTO_OK` finalizes the request. In the next subsection the body format is presented.
+
+### Body
+
+The common body structure:
+```
++----------------------------------------------+
+| IPROTO_BODY: {                               |
+|     IPROTO_METADATA: [                       |
+|         {                                    |
+|             IPROTO_FIELD_NAME: string,       |
+|             IPROTO_FIELD_TYPE: number,       |
+|             IPROTO_FIELD_FLAGS: number,      |
+|         },                                   |
+|         ...                                  |
+|     ],                                       |
+|                                              |
+|     IPROTO_SQL_INFO: {                       |
+|         SQL_INFO_ROW_COUNT: number,          |
+|         SQL_INFO_LAST_ID: number,            |
+|         SQL_INFO_FLAGS: number,              |
+|         ...                                  |
+|     },                                       |
+|                                              |
+|     IPROTO_DATA: [                           |
+|         tuple/scalar,                        |
+|         ...                                  |
+|     ]                                        |
+| }                                            |
++----------------------------------------------+
+```
+For a while the single `SQL_INFO_FLAGS` value is available: `SQL_INFO_HAS_NEXT_CHUNK` - a response having this flag means, that the current result set is not fully read - more responses are available. For example, it could be big `SELECT FROM` sent in multiple chunks.
+
+Consider, how different responses use the body, and how they can be distinguished.
+
+_A non formatted response_ has only `IPROTO_DATA` key in a body. It is the result of Lua and C DML, DDL, DQL, stored procedures calls, push messages. Such response is never linked with next or previous messages of the same request.
+
+_A non formatted response with metadata_ has only `IPROTO_SQL_INFO` and it is always result of DDL/DML executed via SQL. As well as the previous type, this response is all-independent.
+
+_A formatted response_ always has `IPROTO_DATA`, and can have both `IPROTO_SQL_INFO` and `IPROTO_METADATA`. It is always result of SQL DQL (`SELECT`). The response can be part of a continuous sequence of responses. A first message of the sequence always contains `IPROTO_METADATA`, while all non-last ones always contain `IPROTO_SQL_INFO` with `SQL_INFO_HAS_NEXT_CHUNK` flag. The last message has only `IPROTO_DATA` or nothing.
+
+On the picture the state machine of the protocol is showed:
+![alt text](https://raw.githubusercontent.com/tarantool/tarantool/sql-proto-rfc/doc/RFC/wire_protocol_img1.png)
+
+For the `FUNCTION my_sql_func` call the following responses are sent:
+```
+/* Push from my_lua_func(a1). */
++----------------------------------------------+
+| HEADER: IPROTO_CHUNK                         |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|     IPROTO_DATA: [ a1 ]                      |
+| }                                            |
++----------------------------------------------+
+
+/* Result of SELECT my_lua_func(a1). */
++----------------------------------------------+
+| HEADER: IPROTO_CHUNK                         |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|     IPROTO_DATA: [ [ a1 ] ],                 |
+|     IPROTO_METADATA: [                       |
+|         { /* field name, type ... */ }       |
+|     ]                                        |
+| }                                            |
++----------------------------------------------+
+
+/* First chunk of SELECT * FROM table1. */
++----------------------------------------------+
+| HEADER: IPROTO_CHUNK                         |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|    IPROTO_DATA: [ tuple1, tuple2, ... ]      |
+|    IPROTO_METADATA: [                        |
+|        { /* field1 name, type ... */ },      |
+|        { /* field2 name, type ... */ },      |
+|        ...                                   |
+|    ],                                        |
+|    IPROTO_SQL_INFO: {                        |
+|        SQL_INFO_FLAGS:                       |
+|            SQL_INFO_HAS_NEXT_CHUNK           |
+|    }                                         |
+| }                                            |
++----------------------------------------------+
+
+        /* From second to next to last chunk. */
+	+----------------------------------------------+
+	| HEADER: IPROTO_CHUNK                         |
+	+- - - - - - - - - - - - - - - - - - - - - - - +
+	| BODY: {                                      |
+	|    IPROTO_DATA: [ tuple1, tuple2, ... ],     |
+	|    IPROTO_SQL_INFO: {                        |
+	|        SQL_INFO_FLAGS:                       |
+	|            SQL_INFO_HAS_NEXT_CHUNK           |
+	|    }                                         |
+	| }                                            |
+	+----------------------------------------------+
+
+	/* Last chunk. */
+	+----------------------------------------------+
+	| HEADER: IPROTO_CHUNK                         |
+	+- - - - - - - - - - - - - - - - - - - - - - - +
+	| BODY: {                                      |
+	|    IPROTO_DATA: [ tuple1, tuple2, ... ]      |
+	| }                                            |
+	+----------------------------------------------+
+
+/* Result of SELECT my_c_func(a2). */
++----------------------------------------------+
+| HEADER: IPROTO_CHUNK                         |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|     IPROTO_DATA: [ [ tuple ] ],              |
+|     IPROTO_METADATA: [                       |
+|         { /* field name, type ... */ }       |
+|     ]                                        |
+| }                                            |
++----------------------------------------------+
+
+/* Result of INSERT INTO table1 VALUES (1, 2, 3). */
++----------------------------------------------+
+| HEADER: IPROTO_CHUNK                         |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|     IPROTO_SQL_INFO: {                       |
+|         SQL_INFO_ROW_COUNT: number,          |
+|         SQL_INFO_LAST_ID: number,            |
+|     }                                        |
+| }                                            |
++----------------------------------------------+
+
+/* Result of RETURN a4 */
++----------------------------------------------+
+| HEADER: IPROTO_OK                            |
++- - - - - - - - - - - - - - - - - - - - - - - +
+| BODY: {                                      |
+|     IPROTO_DATA: [ a4 ]                      |
+| }                                            |
++----------------------------------------------+
+```
-- 
2.14.3 (Apple Git-98)

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-06-28 11:45 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-04 21:22 [tarantool-patches] [PATCH 1/1] rfc: describe a Tarantool wire protocol Vladislav Shpilevoy
2018-04-05  8:25 ` [tarantool-patches] " Konstantin Osipov
2018-04-09 15:31   ` [tarantool-patches] " Vladislav Shpilevoy
2018-06-28 11:45     ` [tarantool-patches] " Konstantin Osipov
     [not found] ` <CAFoyxqh0QqNBVr7tuFF_uoUw-CvBKOVA3FCyWvixikberOxP9w@mail.gmail.com>
2018-04-05  9:01   ` [tarantool-patches] Re: Fwd: " Алексей Гаджиев
2018-04-05  9:37     ` Vladislav Shpilevoy
2018-04-05 10:03     ` Konstantin Osipov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox