From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 56CFB20A3D for ; Mon, 9 Sep 2019 15:45:03 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E7nlB5fjWVug for ; Mon, 9 Sep 2019 15:45:03 -0400 (EDT) Received: from smtpng2.m.smailru.net (smtpng2.m.smailru.net [94.100.179.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 13745208CF for ; Mon, 9 Sep 2019 15:45:02 -0400 (EDT) From: Ilya Kosarev Subject: [tarantool-patches] [PATCH v2] http: add CURLOPT_ACCEPT_ENCODING option Date: Mon, 9 Sep 2019 22:44:54 +0300 Message-Id: <20190909194454.15845-1-i.kosarev@tarantool.org> Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: tarantool-patches@freelists.org Cc: georgy@tarantool.org, alexander.turenko@tarantool.org, Ilya Kosarev CURLOPT_ACCEPT_ENCODING option is now supported. This option enables automatic decompression of HTTP downloads. See https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html Closes #4232 @TarantoolBot document Title: http: CURLOPT_ACCEPT_ENCODING option Update the documentation for curl options to reflect new CURLOPT_ACCEPT_ENCODING option. It enables automatic decompression of HTTP downloads by setting the contents of the Accept-Encoding header sent in a HTTP request and enabling decoding of a response when a Content-Encoding header is received. This is a request, not an order; the server may or may not do it. Servers might respond with Content-Encoding even without getting an Accept-Encoding in the request. Servers might respond with a different Content-Encoding than what was asked for in the request. @param encoding specifies what encoding you'd like. This param can be an empty string which means Accept-Encoding header will contain all built-in supported encodings. This param can be comma-separated list of accepted encodings, like: "br, gzip, deflate". Bundled libcurl supports "identity", meaning non-compressed, "deflate" which requests the server to compress its response using the zlib algorithm and "gzip" which requests the gzip algorithm. System libcurl also possibly supports "br" which is brotli. See https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html --- https://github.com/tarantool/tarantool/tree/i.kosarev/gh-4232-curlopt-accept-encoding https://github.com/tarantool/tarantool/issues/4232 Changes in v2: - added docbot request - fixed comments - enhanced httpc_set_accept_encoding description - fixed error handling for unsupported encodings src/httpc.c | 22 +++++++++++++++++----- src/httpc.h | 24 ++++++++++++++++++++++++ src/lua/httpc.c | 5 +++++ src/lua/httpc.lua | 2 ++ 4 files changed, 48 insertions(+), 5 deletions(-) diff --git a/src/httpc.c b/src/httpc.c index 8d18b9966..22b54d16a 100644 --- a/src/httpc.c +++ b/src/httpc.c @@ -361,6 +361,22 @@ httpc_set_follow_location(struct httpc_request *req, long follow) follow); } +void +httpc_set_accept_encoding(struct httpc_request *req, const char *encoding) +{ +/* +* CURLOPT_ACCEPT_ENCODING was called CURLOPT_ENCODING before +* libcurl 7.21.6. +*/ +#if LIBCURL_VERSION_NUM >= 0x071506 + curl_easy_setopt(req->curl_request.easy, CURLOPT_ACCEPT_ENCODING, + encoding); +#else + curl_easy_setopt(req->curl_request.easy, CURLOPT_ENCODING, + encoding); +#endif +} + int httpc_execute(struct httpc_request *req, double timeout) { @@ -429,16 +445,12 @@ httpc_execute(struct httpc_request *req, double timeout) case CURLE_COULDNT_RESOLVE_PROXY: case CURLE_COULDNT_RESOLVE_HOST: case CURLE_COULDNT_CONNECT: + case CURLE_WRITE_ERROR: /* 595 Connection Problem (AnyEvent non-standard) */ req->status = 595; req->reason = curl_easy_strerror(req->curl_request.code); ++env->stat.failed_requests; break; - case CURLE_WRITE_ERROR: - /* Diag is already set by curl_write_cb() */ - assert(!diag_is_empty(&fiber()->diag)); - ++env->stat.failed_requests; - return -1; case CURLE_OUT_OF_MEMORY: diag_set(OutOfMemory, 0, "curl", "internal"); ++env->stat.failed_requests; diff --git a/src/httpc.h b/src/httpc.h index 99fd8fbd4..f710b3d13 100644 --- a/src/httpc.h +++ b/src/httpc.h @@ -372,6 +372,30 @@ httpc_set_interface(struct httpc_request *req, const char *interface); void httpc_set_follow_location(struct httpc_request *req, long follow); +/** + * Enable automatic decompression of HTTP downloads: set the + * contents of the Accept-Encoding header sent in a HTTP request + * and enable decoding of a response when a Content-Encoding + * header is received. This is a request, not an order; the + * server may or may not do it. Servers might respond with + * Content-Encoding even without getting an Accept-Encoding in the + * request. Servers might respond with a different + * Content-Encoding than what was asked for in the request. + * @param req request + * @param encoding - specify what encoding you'd like. This param + * can be an empty string which means Accept-Encoding header will + * contain all built-in supported encodings. This param can be + * comma-separated list of accepted encodings, like: + * "br, gzip, deflate". Bundled libcurl supports "identity", + * meaning non-compressed, "deflate" which requests the server to + * compress its response using the zlib algorithm and "gzip" which + * requests the gzip algorithm. System libcurl also possibly + * supports "br" which is brotli. + * @see https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html + */ +void +httpc_set_accept_encoding(struct httpc_request *req, const char *encoding); + /** * This function does async HTTP request * @param request - reference to request object with filled fields diff --git a/src/lua/httpc.c b/src/lua/httpc.c index a8e3e2525..6ed4eb788 100644 --- a/src/lua/httpc.c +++ b/src/lua/httpc.c @@ -315,6 +315,11 @@ luaT_httpc_request(lua_State *L) httpc_set_follow_location(req, lua_toboolean(L, -1)); lua_pop(L, 1); + lua_getfield(L, 5, "accept_encoding"); + if (!lua_isnil(L, -1)) + httpc_set_accept_encoding(req, lua_tostring(L, -1)); + lua_pop(L, 1); + if (httpc_execute(req, timeout) != 0) { httpc_request_delete(req); return luaT_error(L); diff --git a/src/lua/httpc.lua b/src/lua/httpc.lua index ce9bb9771..3224d8c10 100644 --- a/src/lua/httpc.lua +++ b/src/lua/httpc.lua @@ -296,6 +296,8 @@ end -- 'Location' header that a server sends as part of an -- 3xx response; -- +-- accept_encoding - enables automatic decompression of HTTP downloads; +-- -- Returns: -- { -- status=NUMBER, -- 2.17.1