[Tarantool-patches] [PATCH v3 0/4] http: add CURLOPT_ACCEPT_ENCODING option

[Tarantool-patches] [PATCH v3 0/4] http: add CURLOPT_ACCEPT_ENCODING option

[Ilya Kosarev]

This patchset introduces CURLOPT_ACCEPT_ENCODING option. It brought up
fix for CURLE_WRITE_ERROR processing and addition of
CURLE_BAD_CONTENT_ENCODING in curl request code processing, as well as
enhancement of provided error info for curl request.

Changes in v2:
- added docbot request
- fixed comments
- enhanced httpc_set_accept_encoding description
- fixed error handling for unsupported encodings

Changes in v3:
- moved error handling for unsupported encodings fix to separate commit
- added error message obtained using CURLOPT_ERRORBUFFER option
- added CURLE_BAD_CONTENT_ENCODING for curl request code processing

Branch: https://github.com/tarantool/tarantool/tree/i.kosarev/gh-4232-curlopt-accept-encoding
Issues: https://github.com/tarantool/tarantool/issues/4232
        https://github.com/tarantool/tarantool/issues/4578
        https://github.com/tarantool/tarantool/issues/4579

Ilya Kosarev (4):
  http: add CURLOPT_ACCEPT_ENCODING option
  http: remove redundant & incorrect case for curl_request code
  http: enrich httpc_request with curl error message buffer
  http: add CURLE_BAD_CONTENT_ENCODING case for curl_request code

 src/httpc.c       | 24 +++++++++++++++++++-----
 src/httpc.h       | 29 +++++++++++++++++++++++++++++
 src/lua/httpc.c   |  9 +++++++++
 src/lua/httpc.lua |  2 ++
 4 files changed, 59 insertions(+), 5 deletions(-)

-- 
2.17.1

[Tarantool-patches] [PATCH v3 1/4] http: add CURLOPT_ACCEPT_ENCODING option

[Ilya Kosarev]

CURLOPT_ACCEPT_ENCODING option is now supported.
This option enables automatic decompression of HTTP downloads.
See https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html

Closes #4232

@TarantoolBot document
Title: http: CURLOPT_ACCEPT_ENCODING option
Update the documentation for curl options to reflect new
CURLOPT_ACCEPT_ENCODING option. It enables automatic
decompression of HTTP downloads by setting the contents of the
Accept-Encoding header sent in a HTTP request and enabling
decoding of a response when a Content-Encoding header is received.
This is a request, not an order; the server may or may not do it.
Servers might respond with Content-Encoding even without getting
an Accept-Encoding in the request. Servers might respond with a
different Content-Encoding than what was asked for in the request.
@param encoding specifies what encoding you'd like. This param
can be an empty string which means Accept-Encoding header will
contain all built-in supported encodings. This param can be
comma-separated list of accepted encodings, like:
"br, gzip, deflate". Bundled libcurl supports "identity",
meaning non-compressed, "deflate" which requests the server to
compress its response using the zlib algorithm and "gzip" which
requests the gzip algorithm. System libcurl also possibly
supports "br" which is brotli.
See https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
---
 src/httpc.c       | 16 ++++++++++++++++
 src/httpc.h       | 24 ++++++++++++++++++++++++
 src/lua/httpc.c   |  5 +++++
 src/lua/httpc.lua |  2 ++
 4 files changed, 47 insertions(+)

diff --git a/src/httpc.c b/src/httpc.c
index 8d18b9966..146a6f067 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -361,6 +361,22 @@ httpc_set_follow_location(struct httpc_request *req, long follow)
 			 follow);
 }
 
+void
+httpc_set_accept_encoding(struct httpc_request *req, const char *encoding)
+{
+/*
+* CURLOPT_ACCEPT_ENCODING was called CURLOPT_ENCODING before
+* libcurl 7.21.6.
+*/
+#if LIBCURL_VERSION_NUM >= 0x071506
+	curl_easy_setopt(req->curl_request.easy, CURLOPT_ACCEPT_ENCODING,
+			 encoding);
+#else
+	curl_easy_setopt(req->curl_request.easy, CURLOPT_ENCODING,
+			 encoding);
+#endif
+}
+
 int
 httpc_execute(struct httpc_request *req, double timeout)
 {
diff --git a/src/httpc.h b/src/httpc.h
index 99fd8fbd4..f710b3d13 100644
--- a/src/httpc.h
+++ b/src/httpc.h
@@ -372,6 +372,30 @@ httpc_set_interface(struct httpc_request *req, const char *interface);
 void
 httpc_set_follow_location(struct httpc_request *req, long follow);
 
+/**
+ * Enable automatic decompression of HTTP downloads: set the
+ * contents of the Accept-Encoding header sent in a HTTP request
+ * and enable decoding of a response when a Content-Encoding
+ * header is received. This is a request, not an order; the
+ * server may or may not do it. Servers might respond with
+ * Content-Encoding even without getting an Accept-Encoding in the
+ * request. Servers might respond with a different
+ * Content-Encoding than what was asked for in the request.
+ * @param req request
+ * @param encoding - specify what encoding you'd like. This param
+ * can be an empty string which means Accept-Encoding header will
+ * contain all built-in supported encodings. This param can be
+ * comma-separated list of accepted encodings, like:
+ * "br, gzip, deflate". Bundled libcurl supports "identity",
+ * meaning non-compressed, "deflate" which requests the server to
+ * compress its response using the zlib algorithm and "gzip" which
+ * requests the gzip algorithm. System libcurl also possibly
+ * supports "br" which is brotli.
+ * @see https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
+ */
+void
+httpc_set_accept_encoding(struct httpc_request *req, const char *encoding);
+
 /**
  * This function does async HTTP request
  * @param request - reference to request object with filled fields
diff --git a/src/lua/httpc.c b/src/lua/httpc.c
index a8e3e2525..6ed4eb788 100644
--- a/src/lua/httpc.c
+++ b/src/lua/httpc.c
@@ -315,6 +315,11 @@ luaT_httpc_request(lua_State *L)
 		httpc_set_follow_location(req, lua_toboolean(L, -1));
 	lua_pop(L, 1);
 
+	lua_getfield(L, 5, "accept_encoding");
+	if (!lua_isnil(L, -1))
+		httpc_set_accept_encoding(req, lua_tostring(L, -1));
+	lua_pop(L, 1);
+
 	if (httpc_execute(req, timeout) != 0) {
 		httpc_request_delete(req);
 		return luaT_error(L);
diff --git a/src/lua/httpc.lua b/src/lua/httpc.lua
index ce9bb9771..3224d8c10 100644
--- a/src/lua/httpc.lua
+++ b/src/lua/httpc.lua
@@ -296,6 +296,8 @@ end
 --          'Location' header that a server sends as part of an
 --          3xx response;
 --
+--      accept_encoding - enables automatic decompression of HTTP downloads;
+--
 --  Returns:
 --      {
 --          status=NUMBER,
-- 
2.17.1

[Tarantool-patches] [PATCH v3 2/4] http: remove redundant & incorrect case for curl_request code

[Ilya Kosarev]

After executing curl request we need to process curl_request code. It
might be CURL_WRITE_ERROR. We had special case for it, which assumed
diagnostic being set and contained corresponding assert, though it is
incorrect. Better way is to handle it as any other non-standard event.

Part of #4232
---
 src/httpc.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/src/httpc.c b/src/httpc.c
index 146a6f067..22b54d16a 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -445,16 +445,12 @@ httpc_execute(struct httpc_request *req, double timeout)
 	case CURLE_COULDNT_RESOLVE_PROXY:
 	case CURLE_COULDNT_RESOLVE_HOST:
 	case CURLE_COULDNT_CONNECT:
+	case CURLE_WRITE_ERROR:
 		/* 595 Connection Problem (AnyEvent non-standard) */
 		req->status = 595;
 		req->reason = curl_easy_strerror(req->curl_request.code);
 		++env->stat.failed_requests;
 		break;
-	case CURLE_WRITE_ERROR:
-		/* Diag is already set by curl_write_cb() */
-		assert(!diag_is_empty(&fiber()->diag));
-		++env->stat.failed_requests;
-		return -1;
 	case CURLE_OUT_OF_MEMORY:
 		diag_set(OutOfMemory, 0, "curl", "internal");
 		++env->stat.failed_requests;
-- 
2.17.1

[Tarantool-patches] [PATCH v3 3/4] http: enrich httpc_request with curl error message buffer

[Ilya Kosarev]

When processing curl request error code, we are collecting error
message using curl_easy_strerror. Now we are providing more info
in errmsg, which is obtained using curl_easy_setopt with
CURLOPT_ERRORBUFFER option.

Closes #4578

@TarantoolBot document
Title: http: new field in client_object:request return table
Update the documentation for client_object:request method to reflect
new field errmsg in return table. It contains extended curl request
error message obtained using CURLOPT_ERRORBUFFER option.
See https://curl.haxx.se/libcurl/c/CURLOPT_ERRORBUFFER.html
---
 src/httpc.c     | 1 +
 src/httpc.h     | 5 +++++
 src/lua/httpc.c | 4 ++++
 3 files changed, 10 insertions(+)

diff --git a/src/httpc.c b/src/httpc.c
index 22b54d16a..b493a8498 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -398,6 +398,7 @@ httpc_execute(struct httpc_request *req, double timeout)
 	curl_easy_setopt(req->curl_request.easy, CURLOPT_HEADERDATA, (void *) req);
 	curl_easy_setopt(req->curl_request.easy, CURLOPT_PRIVATE, (void *) &req->curl_request);
 	curl_easy_setopt(req->curl_request.easy, CURLOPT_HTTPHEADER, req->headers);
+	curl_easy_setopt(req->curl_request.easy, CURLOPT_ERRORBUFFER, req->errmsg);
 
 	++env->stat.total_requests;
 
diff --git a/src/httpc.h b/src/httpc.h
index f710b3d13..145b59929 100644
--- a/src/httpc.h
+++ b/src/httpc.h
@@ -105,6 +105,11 @@ struct httpc_request {
 	int status;
 	/** Error message */
 	const char *reason;
+	/**
+	 * Buffer for error message obtained using
+	 * CURLOPT_ERRORBUFFER option.
+	 */
+        const char errmsg[CURL_ERROR_SIZE];
 	/** buffer of headers */
 	struct region resp_headers;
 	/** buffer of body */
diff --git a/src/lua/httpc.c b/src/lua/httpc.c
index 6ed4eb788..9fb8e4801 100644
--- a/src/lua/httpc.c
+++ b/src/lua/httpc.c
@@ -335,6 +335,10 @@ luaT_httpc_request(lua_State *L)
 	lua_pushstring(L, req->reason);
 	lua_settable(L, -3);
 
+	lua_pushstring(L, "errmsg");
+	lua_pushstring(L, req->errmsg);
+	lua_settable(L, -3);
+
 	size_t headers_len = region_used(&req->resp_headers);
 	if (headers_len > 0) {
 		char *headers = region_join(&req->resp_headers, headers_len);
-- 
2.17.1

[Tarantool-patches] [PATCH v3 4/4] http: add CURLE_BAD_CONTENT_ENCODING case for curl_request code

[Ilya Kosarev]

Currently in case of unknown encoding curl returns CURLE_WRITE_ERROR
request code. Since curl/curl#4449 CURLE_BAD_CONTENT_ENCODING request code
will be returned in this case. Therefore mentioned case has to be
added into switch clause processing curl request code.

Closes #4579
---
 src/httpc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/httpc.c b/src/httpc.c
index b493a8498..d3516049e 100644
--- a/src/httpc.c
+++ b/src/httpc.c
@@ -447,6 +447,7 @@ httpc_execute(struct httpc_request *req, double timeout)
 	case CURLE_COULDNT_RESOLVE_HOST:
 	case CURLE_COULDNT_CONNECT:
 	case CURLE_WRITE_ERROR:
+	case CURLE_BAD_CONTENT_ENCODING:
 		/* 595 Connection Problem (AnyEvent non-standard) */
 		req->status = 595;
 		req->reason = curl_easy_strerror(req->curl_request.code);
-- 
2.17.1