From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 809142E3B3 for ; Tue, 14 May 2019 10:24:53 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DZhVVnRhzFS0 for ; Tue, 14 May 2019 10:24:53 -0400 (EDT) Received: from smtp37.i.mail.ru (smtp37.i.mail.ru [94.100.177.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id 3E75A2E39B for ; Tue, 14 May 2019 10:24:53 -0400 (EDT) Date: Tue, 14 May 2019 17:24:46 +0300 From: Kirill Yukhin Subject: [tarantool-patches] Re: [PATCH] Pass max_total_connections parameter to httpclient Message-ID: <20190514142446.gbe4uulwshdwt2p3@tarantool.org> References: <20190417145139.59554-1-runsfor@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20190417145139.59554-1-runsfor@gmail.com> Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: tarantool-patches@freelists.org Cc: totktonada.ru@gmail.com, Ilya Konyukhov Hello, On 17 Apr 17:51, Ilya Konyukhov wrote: > There are some usecases when current http client may leak sockets (due > TIME_WAIT timeout). The reason is that the `max_connection` option of > tarantool's http client (actually is a binding for CURLMOPT_MAXCONNECTS > option in libcurl) may not be enough for some situations. > > This option sets up a size of connection cache, which curl maintains > during its lifetime. When this cache is full (i.e. all connections are > waiting for a response from a server), newly created request would > create a new connection to handle it. In that case right after old > connection got response, curl has to close that connection to keep up an > appropriate number of connections in cache. That results in a new socket > in TIME_WAIT state (for 60 seconds by default). > > This is actually easy to calculate. Imagine we have http client with > default 8 maximum connections in cache. Also lets say we have 16384 > sockets available in the system. We need to make requests to external > system which responses in 100ms in average. So, if we start making > request every 11ms (100/9), every finished request will be followed by > curl closing that socket (switches to TIME_WAIT). Result is about 90 > wasted sockets per second and additional overhead for socket creation > for each new request. > > Also if we do more than 274 requests per second, we will likely to be > out of available sockets in 1 minute. > > The solution is to add another binding for max_total_connections option, > which will let curl hold new requests, before old ones have finished. > It also adds more control over sockets resources. I've checked your patch into 1.10, 2.1 and master. -- Regards, Kirill Yukhin