From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from localhost (localhost [127.0.0.1]) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTP id 1A78425158 for ; Thu, 15 Aug 2019 22:33:02 -0400 (EDT) Received: from turing.freelists.org ([127.0.0.1]) by localhost (turing.freelists.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G29oJNcfKHzJ for ; Thu, 15 Aug 2019 22:33:02 -0400 (EDT) Received: from smtp3.mail.ru (smtp3.mail.ru [94.100.179.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by turing.freelists.org (Avenir Technologies Mail Multiplex) with ESMTPS id A4918250FE for ; Thu, 15 Aug 2019 22:33:01 -0400 (EDT) From: Alexander Turenko Subject: [tarantool-patches] [PATCH 2/2] build: link libcurl statically from a submodule Date: Fri, 16 Aug 2019 05:32:36 +0300 Message-Id: <42fea157aba4d4938d9023804b516eabc478f19c.1565920130.git.alexander.turenko@tarantool.org> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: tarantool-patches-bounce@freelists.org Errors-to: tarantool-patches-bounce@freelists.org Reply-To: tarantool-patches@freelists.org List-Help: List-Unsubscribe: List-software: Ecartis version 1.0.0 List-Id: tarantool-patches List-Subscribe: List-Owner: List-post: List-Archive: To: Georgy Kirichenko Cc: Mergen Imeev , tarantool-patches@freelists.org, "Alexander V . Tikhonov" , Mergen Imeev From: Mergen Imeev Hold libcurl-7.65.3. This version is not affected by the following issues: * #4180 ('httpc: redirects are broken with libcurl-7.30 and older'); * #4389 ('libcurl memory leak'); * #4397 ('HTTPS seem to be unstable'). After this patch libcurl will be statically linked when ENABLE_BUNDLED_LIBCURL option is set. This option is set by default. Closes #4318 @TarantoolBot document Title: Tarantool dependency list was changed * Added build dependencies: autoconf, automake, libtool, zlib-devel (zlib1g-dev on Debian). * Added runtime dependencies: zlib (zlib1g on Debian). * Removed build dependencies: libcurl-devel (libcurl4-openssl-dev on Debian). * Removed runtime dependencies: curl. The reason is that now we use compiled-in libcurl: so we don't depend on a system libcurl, but inherit its dependencies. --- .gitmodules | 3 + .travis.mk | 8 +-- CMakeLists.txt | 12 +++- Dockerfile.staticbuild | 14 +---- cmake/BuildLibCURL.cmake | 118 +++++++++++++++++++++++++++++++++++++++ debian/control | 9 ++- rpm/tarantool.spec | 9 ++- src/CMakeLists.txt | 3 + test/unit/CMakeLists.txt | 1 + third_party/curl | 1 + 10 files changed, 156 insertions(+), 22 deletions(-) create mode 100644 cmake/BuildLibCURL.cmake create mode 160000 third_party/curl diff --git a/.gitmodules b/.gitmodules index 1062f7379..ac13f1c3f 100644 --- a/.gitmodules +++ b/.gitmodules @@ -37,3 +37,6 @@ [submodule "third_party/serpent"] path = third_party/serpent url = https://github.com/tarantool/serpent.git +[submodule "third_party/curl"] + path = third_party/curl + url = https://github.com/curl/curl.git diff --git a/.travis.mk b/.travis.mk index c0c23b6d7..4e43ad6f3 100644 --- a/.travis.mk +++ b/.travis.mk @@ -52,7 +52,7 @@ deps_debian: libcurl4-openssl-dev libunwind-dev libicu-dev \ python python-pip python-setuptools python-dev \ python-msgpack python-yaml python-argparse python-six python-gevent \ - lcov ruby clang llvm llvm-dev + lcov ruby clang llvm llvm-dev zlib1g-dev autoconf automake libtool deps_buster_clang_8: deps_debian echo "deb http://apt.llvm.org/buster/ llvm-toolchain-buster-8 main" > /etc/apt/sources.list.d/clang_8.list @@ -117,7 +117,7 @@ test_asan_debian: deps_debian deps_buster_clang_8 test_asan_debian_no_deps deps_osx: brew update - brew install openssl readline curl icu4c libiconv --force + brew install openssl readline curl icu4c libiconv zlib autoconf automake libtool --force python2 -V || brew install python2 --force curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py >get-pip.py python get-pip.py --user @@ -152,10 +152,10 @@ test_osx: deps_osx test_osx_no_deps deps_freebsd: sudo pkg install -y git cmake gmake gcc coreutils \ - readline ncurses libyaml openssl curl libunwind icu \ + readline ncurses libyaml openssl libunwind icu \ python27 py27-pip py27-setuptools py27-daemon \ py27-yaml py27-argparse py27-six py27-gevent \ - gdb bash + gdb bash autoconf automake libtool build_freebsd: cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS} diff --git a/CMakeLists.txt b/CMakeLists.txt index bfb15effb..9b3950bdf 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -334,8 +334,15 @@ endif() # # Curl # -set(CURL_FIND_REQUIRED ON) -find_package(CURL) +option(ENABLE_BUNDLED_LIBCURL "Enable building of the bundled libcurl" ON) +if (ENABLE_BUNDLED_LIBCURL) + include(BuildLibCURL) + curl_build() + add_dependencies(build_bundled_libs bundled-libcurl) +else() + set(CURL_FIND_REQUIRED ON) + find_package(CURL) +endif() # # ReadLine @@ -525,6 +532,7 @@ set(options PACKAGE VERSION BUILD C_COMPILER CXX_COMPILER C_FLAGS CXX_FLAGS ENABLE_BACKTRACE ENABLE_DOC ENABLE_DIST + ENABLE_BUNDLED_LIBCURL ENABLE_BUNDLED_LIBYAML ENABLE_BUNDLED_MSGPUCK) foreach(option IN LISTS options) diff --git a/Dockerfile.staticbuild b/Dockerfile.staticbuild index 6a784990b..9f97aa910 100644 --- a/Dockerfile.staticbuild +++ b/Dockerfile.staticbuild @@ -19,7 +19,7 @@ RUN set -x \ zip \ unzip \ libunwind \ - libcurl \ + zlib \ && yum -y install \ perl \ gcc-c++ \ @@ -43,18 +43,9 @@ RUN set -x && \ curl -O -L https://www.openssl.org/source/openssl-1.1.0h.tar.gz && \ tar -xvf openssl-1.1.0h.tar.gz && \ cd openssl-1.1.0h && \ - ./config && \ + ./config --libdir=lib && \ make && make install -RUN set -x && \ - cd / && \ - git clone https://github.com/curl/curl.git && \ - cd curl && \ - git checkout curl-7_59_0 && \ - ./buildconf && \ - LD_LIBRARY_PATH=/usr/local/lib64 LIBS=" -lssl -lcrypto -ldl" ./configure --enable-static --enable-shared --with-ssl && \ - make -j && make install - RUN set -x && \ cd / && \ wget http://download.icu-project.org/files/icu4c/62.1/icu4c-62_1-src.tgz && \ @@ -86,6 +77,7 @@ RUN set -x \ -DENABLE_DIST:BOOL=ON \ -DBUILD_STATIC=ON \ -DOPENSSL_USE_STATIC_LIBS=ON \ + -DOPENSSL_ROOT_DIR=/usr/local \ .) \ && make -C /tarantool -j diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake new file mode 100644 index 000000000..866b3c49e --- /dev/null +++ b/cmake/BuildLibCURL.cmake @@ -0,0 +1,118 @@ +# A macro to build the bundled libcurl +macro(curl_build) + set(LIBCURL_SOURCE_DIR ${PROJECT_SOURCE_DIR}/third_party/curl) + set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work) + set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest) + + if (BUILD_STATIC) + set(LIBZ_LIB_NAME libz.a) + else() + set(LIBZ_LIB_NAME z) + endif() + find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) + if ("${LIBZ_LIBRARY}" STREQUAL "LIBZ_LIBRARY-NOTFOUND") + message(FATAL_ERROR "Unable to find zlib") + endif() + + # Set curl option to find OpenSSL library. + if ("${OPENSSL_ROOT_DIR}" STREQUAL "") + # Linux / FreeBSD. + set(LIBCURL_OPENSSL_OPT "--with-ssl") + else() + # Mac OS. + set(LIBCURL_OPENSSL_OPT "--with-ssl=${OPENSSL_ROOT_DIR}") + endif() + + include(ExternalProject) + ExternalProject_Add( + bundled-libcurl-project + SOURCE_DIR ${LIBCURL_SOURCE_DIR} + PREFIX ${LIBCURL_INSTALL_DIR} + DOWNLOAD_DIR ${LIBCURL_BINARY_DIR} + TMP_DIR ${LIBCURL_BINARY_DIR}/tmp + STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp + BINARY_DIR ${LIBCURL_BINARY_DIR} + CONFIGURE_COMMAND + cd && ./buildconf && + cd && /configure + --prefix + --enable-static + --enable-shared + + --with-zlib + ${LIBCURL_OPENSSL_OPT} + --with-ca-fallback + + --without-brotli + --without-gnutls + --without-mbedtls + --without-cyassl + --without-wolfssl + --without-mesalink + --without-nss + --without-ca-bundle + --without-ca-path + --without-libpsl + --without-libmetalink + --without-librtmp + --without-winidn + --without-libidn2 + --without-nghttp2 + --without-ngtcp2 + --without-nghttp3 + --without-quiche + --without-zsh-functions-dir + --without-fish-functions-dir + + --enable-http + --enable-proxy + --enable-ipv6 + --enable-threaded-resolver + --enable-unix-sockets + --enable-cookies + --enable-http-auth + --enable-mime + --enable-dateparse + + --disable-ares + --disable-ftp + --disable-file + --disable-ldap + --disable-ldaps + --disable-rtsp + --disable-dict + --disable-telnet + --disable-tftp + --disable-pop3 + --disable-imap + --disable-smb + --disable-smtp + --disable-gopher + --disable-manual + --disable-sspi + --disable-crypto-auth + --disable-ntlm-wb + --disable-tls-srp + --disable-doh + --disable-netrc + --disable-progress-meter + --disable-dnsshuffle + --disable-alt-svc + BUILD_COMMAND cd && $(MAKE) + INSTALL_COMMAND cd && $(MAKE) install) + + add_library(bundled-libcurl STATIC IMPORTED GLOBAL) + set_target_properties(bundled-libcurl PROPERTIES IMPORTED_LOCATION + ${LIBCURL_INSTALL_DIR}/lib/libcurl.a) + add_dependencies(bundled-libcurl bundled-libcurl-project) + + set(CURL_INCLUDE_DIRS ${LIBCURL_INSTALL_DIR}/include) + set(CURL_LIBRARIES bundled-libcurl ${LIBZ_LIBRARY}) + if (TARGET_OS_LINUX OR TARGET_OS_FREEBSD) + set(CURL_LIBRARIES ${CURL_LIBRARIES} rt) + endif() + + unset(LIBCURL_INSTALL_DIR) + unset(LIBCURL_BINARY_DIR) + unset(LIBCURL_SOURCE_DIR) +endmacro(curl_build) diff --git a/debian/control b/debian/control index b9d9e1862..91d0b0e44 100644 --- a/debian/control +++ b/debian/control @@ -10,9 +10,10 @@ Build-Depends: cdbs (>= 0.4.100), debhelper (>= 9), dpkg-dev (>= 1.16.1~), libncurses5-dev, libyaml-dev, libssl-dev, - libcurl4-gnutls-dev | libcurl4-openssl-dev | libcurl4-nss-dev, libunwind-dev | libunwind8-dev | libunwind7-dev, libicu-dev, +# libcurl build dependencies + autoconf, automake, libtool, zlib1g-dev, Section: database Standards-Version: 3.9.8 Homepage: http://tarantool.org/ @@ -42,7 +43,7 @@ Replaces: tarantool-common (<< 1.5.3), tarantool-lts-common Depends: ${misc:Depends}, adduser, lsb-base, # Deps for built-in package manager # https://github.com/tarantool/tarantool/issues/2612 - openssl, curl + openssl Recommends: tarantool-dev, git, build-essential, cmake Description: Tarantool in-memory database - common files Tarantool is an in-memory database and Lua application server. @@ -53,7 +54,9 @@ Package: tarantool Architecture: i386 amd64 armhf arm64 Priority: optional Depends: ${shlibs:Depends}, ${misc:Depends}, netbase, binutils, libgomp1, - libyaml-0-2, openssl, tarantool-common (>= 1.7.5.46) + libyaml-0-2, openssl, tarantool-common (>= 1.7.5.46), +# libcurl dependencies (except ones we have already) + zlib1g Replaces: tarantool-lts Conflicts: tarantool-lts-modules, tarantool-lts-postgresql-module, diff --git a/rpm/tarantool.spec b/rpm/tarantool.spec index 002df26b1..10daf1458 100644 --- a/rpm/tarantool.spec +++ b/rpm/tarantool.spec @@ -17,7 +17,6 @@ BuildRequires: sed BuildRequires: readline-devel BuildRequires: libyaml-devel BuildRequires: openssl-devel -BuildRequires: libcurl-devel BuildRequires: libicu-devel #BuildRequires: msgpuck-devel %if 0%{?fedora} > 0 @@ -27,6 +26,13 @@ BuildRequires: perl-podlators Requires(pre): %{_sbindir}/useradd Requires(pre): %{_sbindir}/groupadd +# libcurl dependencies (except ones we have already). +BuildRequires: autoconf +BuildRequires: automake +BuildRequires: libtool +BuildRequires: zlib-devel +Requires: zlib + %if %{with systemd} Requires(post): systemd Requires(preun): systemd @@ -84,7 +90,6 @@ Requires: /etc/services # Deps for built-in package manager # https://github.com/tarantool/tarantool/issues/2612 Requires: openssl -Requires: curl %if (0%{?fedora} >= 22 || 0%{?rhel} >= 8) # RHEL <= 7 doesn't support Recommends: Recommends: tarantool-devel diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index acd719e9b..e12de6005 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -168,6 +168,7 @@ endif() set_source_files_compile_flags(${server_sources}) add_library(server STATIC ${server_sources}) +add_dependencies(server build_bundled_libs) target_link_libraries(server core coll http_parser bit uri uuid swim swim_udp swim_ev crypto) @@ -226,6 +227,8 @@ if(BUILD_STATIC) list(APPEND EXPORT_LIST ${SYMBOLS_LIB}) # set variable to allow rescan (CMake depended) set(SYMBOLS_LIB "SYMBOLS_LIB-NOTFOUND") + elseif (${libstatic} STREQUAL bundled-libcurl) + message("We don't need to export symbols from statically linked libcurl, skipped") else() message(WARNING "${libstatic} should be a static") endif() diff --git a/test/unit/CMakeLists.txt b/test/unit/CMakeLists.txt index 73ee0a974..4a57597e9 100644 --- a/test/unit/CMakeLists.txt +++ b/test/unit/CMakeLists.txt @@ -10,6 +10,7 @@ include_directories(${PROJECT_BINARY_DIR}/src) include_directories(${PROJECT_SOURCE_DIR}/src/box) include_directories(${CMAKE_SOURCE_DIR}/third_party) include_directories(${ICU_INCLUDE_DIRS}) +include_directories(${CURL_INCLUDE_DIRS}) add_library(unit STATIC unit.c) diff --git a/third_party/curl b/third_party/curl new file mode 160000 index 000000000..aa73eb47b --- /dev/null +++ b/third_party/curl @@ -0,0 +1 @@ +Subproject commit aa73eb47bc8583070734696b25b34ad54c2c1f5e -- 2.22.0