From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp18.mail.ru (smtp18.mail.ru [94.100.176.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 8E1FB469719 for ; Wed, 14 Oct 2020 12:25:15 +0300 (MSK) From: "Alexander V. Tikhonov" Date: Wed, 14 Oct 2020 12:25:13 +0300 Message-Id: In-Reply-To: <0a48783b74bcb4fbdb3ff6f3decc79bded564bec.1602663758.git.avtikhon@tarantool.org> References: <0a48783b74bcb4fbdb3ff6f3decc79bded564bec.1602663758.git.avtikhon@tarantool.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v4 4/5] build: enable cmake in curl build List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kirill Yukhin , Igor Munkin Cc: tarantool-patches@dev.tarantool.org Initially tried to change autoconf tools in Curl build to cmake and found the following build issue on: CentOS 6,7 Debian 8 Ubuntu 14.04 Issue found: CMake Error at CMakeLists.txt:41 (cmake_minimum_required): CMake 3.0 or higher is required. You are running version 2.8.12.2 To fix the issue check is removed of the version from curl sources in Tarantool's third party '/third_party/curl': 12af024bc85606b14ffc415413a7e86e6bbee7eb ('Enable curl build with old cmake') After this fix completely changed autoconf to cmake in curl build. Autoconf part was completely removed and code cleaned up for cmake. For curl cmake build all autoconf options were ported to cmake configuration call, check the accurate list of the change in [1]. Also the following issues resolved: 1. Found that CURL cmake configuration file: third_party/curl/lib/CMakeLists.txt has installation part for built libcurl.a library: install(TARGETS ${LIB_NAME} EXPORT ${TARGETS_EXPORT_NAME} ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} where it changes CMAKE_INSTALL_LIBDIR to appropriate name with suffix of the architecture, like: lib lib64 x86_64 Found that find_library routine from the file: cmake/FindLibCURL.cmake returns only 'lib' value and it breaks the building of the depends binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to cmake call: -DCMAKE_INSTALL_LIBDIR=lib 2. Found issue with building on CentOS 6: Linking C executable curl build/ares/dest/lib/libcares.a(ares__timeval.c.o): In function `ares__tvnow': ares__timeval.c:(.text+0x15): undefined reference to `clock_gettime' collect2: error: ld returned 1 exit status It was fixed with added "-lrt" flag to CMAKE_C_FLAGS and CMAKE_CXX_FLAGS build flags, when cmake version is lower than 3.0 and RT library had needed function. 3. Found issues with building Tarantool statically on Debian 9 and its package build on CentOS 8. Building statically got the issues with openssl linking, like [2]: static-build/openssl-prefix/lib/libcrypto.a(threads_pthread.o): In function `CRYPTO_THREAD_lock_new': threads_pthread.c:(.text+0x45): undefined reference to `pthread_rwlock_init' It happened because in openssl radically changed how threads-related things were handled before 1.1.0 version. It required the application to provide us with the locking functionality in form of callbacks. Since 1.1.0, these matters are entirely internal, so libcrypto requires the presence of the platform specific thread implementation of our choosing, which is pthread on everything. After '-pthread' added to C compile flags package build on CentOS 8 failed with the issue [3]: /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:101:4: error: #error "SIZEOF_CURL_OFF_T not defined" # error "SIZEOF_CURL_OFF_T not defined" ^~~~~ /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c: In function 'curlx_uztoso': /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:192:40: error: 'CURL_MASK_SCOFFT' undeclared (first use in this function); did you mean 'CURL_MASK_SINT'? return (curl_off_t)(uznum & (size_t) CURL_MASK_SCOFFT); ^~~~~~~~~~~~~~~~ CURL_MASK_SINT To avoid of the issue decided to use '-pthread' flag only for openssl when it had not this flag in openssl compilation: # check '-pthread' in openssl compile options execute_process(COMMAND openssl version -f OUTPUT_VARIABLE OPENSSL_COMPILE_OPTIONS) # add pthread library for openssl static library linking if(NOT OPENSSL_COMPILE_OPTIONS MATCHES ".* -pthread .*") list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-pthread") endif() 4. Found issue with static build using CentOS 7, where SSL cmake rule failed. Building the image got the issue: [ 1%] Performing configure step for 'bundled-libcurl-project' CMake Warning at CMakeLists.txt:50 (message): the curl cmake build system is poorly maintained. Be aware -- curl version=[7.66.0-DEV] -- Found c-ares: /tarantool/build/ares/dest/lib/libcares.a Found *nroff option: -- -man CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:278 (list): list GET given empty list Call Stack (most recent call first): CMakeLists.txt:347 (find_package) Root cause of the issue that Dockerfile uses globaly installed openSSL with: cmake ... -DOPENSSL_ROOT_DIR=/usr/local ... Its cmake build file: /usr/share/cmake/Modules/FindOpenSSL.cmake fails on parsing the SSL version: it has: REGEX "^#define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*") but it should to use: REGEX "^#[\t ]*define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*") Anyway we want to use the same OpenSSL library for libcurl, as is used for Tarantool itself. So the path to it set for cURL build: list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake") 5. Found that on Gentoo need to switch off use of libSSH2: list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_LIBSSH2=OFF") to avoid of linking issues, like: ld: libssh2.c:(.text+0x4d8): undefined reference to `libssh2_*... this issue exists in curl issues [4]. Closes #4968 Closes #5019 Closes #5396 [1] - https://github.com/tarantool/tarantool/issues/4968#issue-617183031 [2] - https://gitlab.com/tarantool/tarantool/-/jobs/779176133#L6021 [3] - https://gitlab.com/tarantool/tarantool/-/jobs/778309145#L3060 [4] - https://github.com/curl/curl/issues/1146 --- Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4874-out-of-source-build-full-ci Issue: https://github.com/tarantool/tarantool/issues/4968 V4: Changes: - Disabled libSSH2. - Added disable symbols hiding flag. cmake/BuildLibCURL.cmake | 250 +++++++++++++++++++-------------------- 1 file changed, 121 insertions(+), 129 deletions(-) diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake index 86fec39e9..0c3b2fda4 100644 --- a/cmake/BuildLibCURL.cmake +++ b/cmake/BuildLibCURL.cmake @@ -3,6 +3,7 @@ macro(curl_build) set(LIBCURL_SOURCE_DIR ${PROJECT_SOURCE_DIR}/third_party/curl) set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work) set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest) + set(LIBCURL_CMAKE_FLAGS "") message(STATUS "Looking for zlib") find_path(ZLIB_INCLUDE_DIR zlib.h) @@ -19,40 +20,126 @@ macro(curl_build) message(FATAL_ERROR "Unable to find zlib") endif() get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR} DIRECTORY) + list(APPEND LIBCURL_CMAKE_FLAGS "-DZLIB_ROOT=${FOUND_ZLIB_ROOT_DIR}") + + # check '-pthread' in openssl compile options + execute_process(COMMAND openssl version -f + OUTPUT_VARIABLE OPENSSL_COMPILE_OPTIONS) + # add pthread library for openssl static library linking + if(NOT OPENSSL_COMPILE_OPTIONS MATCHES ".* -pthread .*") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-pthread") + endif() + + # add librt for clock_gettime function definition + if(${CMAKE_MAJOR_VERSION} VERSION_LESS "3") + CHECK_LIBRARY_EXISTS (rt clock_gettime "" HAVE_LIBRT) + if (HAVE_LIBRT) + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-lrt") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_CXX_FLAGS=-lrt") + endif() + endif() + + # switch on the static build + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_STATICLIB=ON") + + # switch off the shared build + list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_SHARED_LIBS=OFF") - # Use the same OpenSSL library for libcurl as is used for - # tarantool itself. + # curl uses visibility hiding mode for its symbols by default and + # Tarantool tests can't use it. To fix it symbols hiding disabled + # for gcc and clang. + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_HIDDEN_SYMBOLS=OFF") + + # let's disable testing for curl to save build time + list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_TESTING=OFF") + + # Setup use of openssl, use the same OpenSSL library + # for libcurl as is used for tarantool itself. get_filename_component(FOUND_OPENSSL_ROOT_DIR ${OPENSSL_INCLUDE_DIR} DIRECTORY) - set(LIBCURL_OPENSSL_OPT "--with-ssl=${FOUND_OPENSSL_ROOT_DIR}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_OPENSSL=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DOPENSSL_ROOT_DIR=${FOUND_OPENSSL_ROOT_DIR}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake") - # Use either c-ares bundled with tarantool or - # libcurl-default threaded resolver. + # Setup ARES and its library path, use either c-ares bundled + # with tarantool or libcurl-default threaded resolver. if(BUNDLED_LIBCURL_USE_ARES) - set(ASYN_DNS_USED "ares") - set(ASYN_DNS_UNUSED "threaded-resolver") - set(ASYN_DNS_PATH "=${ARES_INSTALL_DIR}") + set(ENABLE_ARES "ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_FIND_ROOT_PATH=${ARES_INSTALL_DIR}") else() - set(ASYN_DNS_USED "threaded-resolver") - set(ASYN_DNS_UNUSED "ares") - set(ASYN_DNS_PATH "") - endif() - - set(ENABLED_DNS_OPT "--enable-${ASYN_DNS_USED}${ASYN_DNS_PATH}") - set(DISABLED_DNS_OPT "--disable-${ASYN_DSN_UNUSED}") - - # Pass -isysroot= option on Mac OS to a preprocessor - # and a C compiler to find header files installed with an SDK. - # - # The idea here is to don't pass all compiler/linker options - # that is set for tarantool, but only a subset that is - # necessary for choosen toolchain, and let curl's configure - # script set options that are appropriate for libcurl. - set(LIBCURL_CPPFLAGS "") - set(LIBCURL_CFLAGS "") - if (TARGET_OS_DARWIN AND NOT "${CMAKE_OSX_SYSROOT}" STREQUAL "") - set(LIBCURL_CPPFLAGS "${LIBCURL_CPPFLAGS} ${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}") - set(LIBCURL_CFLAGS "${LIBCURL_CFLAGS} ${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}") + set(ENABLE_ARES "OFF") endif() + list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_ARES=${ENABLE_ARES}") + + # Found that on Gentoo need to switch off use of libSSH2 to avoid of issues, like: + # ld: libssh2.c:(.text+0x4d8): undefined reference to `libssh2_*... + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_LIBSSH2=OFF") + + # Switch off the group of protocols with special flag HTTP_ONLY: + # ftp, file, ldap, ldaps, rtsp, dict, telnet, tftp, pop3, imap, smtp + list(APPEND LIBCURL_CMAKE_FLAGS "-DHTTP_ONLY=ON") + + # additionaly disable some more protocols + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_SMB=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_GOPHER=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_CRYPTO_AUTH=ON") + + # switch on ca-fallback feature + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_FALLBACK=ON") + + # Even though we set the external project's install dir + # below, we still need to pass the corresponding install + # prefix via cmake arguments. + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_PREFIX=${LIBCURL_INSTALL_DIR}") + + # The default values for the options below are not always + # "./lib", "./bin" and "./include", while curl expects them + # to be. + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_LIBDIR=lib") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_INCLUDEDIR=include") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_BINDIR=bin") + + # Pass the same toolchain as is used to build tarantool itself, + # because they can be incompatible. + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_LINKER=${CMAKE_LINKER}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_AR=${CMAKE_AR}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_RANLIB=${CMAKE_RANLIB}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_NM=${CMAKE_NM}") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_STRIP=${CMAKE_STRIP}") + + # Need to set values explicitly everything that is default, because + # we don't know how defaults will be changed in a future and we don't + # want to verify them each time we'll bump new libcurl version. + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_BROTLI=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_GNUTLS=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_MBEDTLS=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_WOLFSSL=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_NSS=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_BUNDLE=none") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_PATH=none") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_LIBRTMP=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DHAVE_LIBIDN2=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGHTTP2=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGTCP2=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGHTTP3=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_QUICHE=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_HTTP=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_PROXY=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_IPV6=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_COOKIES=OFF") + list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_UNIX_SOCKETS=ON") + # should be already set by "-DHTTP_ONLY=ON" above + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_FTP=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_FILE=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_LDAP=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_LDAPS=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_RTSP=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_DICT=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_TELNET=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_TFTP=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_POP3=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_IMAP=ON") + list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_SMTP=ON") include(ExternalProject) ExternalProject_Add( @@ -62,101 +149,11 @@ macro(curl_build) DOWNLOAD_DIR ${LIBCURL_BINARY_DIR} TMP_DIR ${LIBCURL_BINARY_DIR}/tmp STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp - BINARY_DIR ${LIBCURL_BINARY_DIR} + BINARY_DIR ${LIBCURL_BINARY_DIR}/curl CONFIGURE_COMMAND - cd && ./buildconf && - cd && /configure - # Pass the same toolchain as is used to build - # tarantool itself, because they can be - # incompatible. - CC=${CMAKE_C_COMPILER} - LD=${CMAKE_LINKER} - AR=${CMAKE_AR} - RANLIB=${CMAKE_RANLIB} - NM=${CMAKE_NM} - STRIP=${CMAKE_STRIP} - - # Pass -isysroot= option on Mac OS, see - # above. - # Note: Passing of CPPFLAGS / CFLAGS explicitly - # discards using of corresponsing environment - # variables. - CPPFLAGS=${LIBCURL_CPPFLAGS} - CFLAGS=${LIBCURL_CFLAGS} - - # Pass empty LDFLAGS to discard using of - # corresponding environment variable. - # It is possible that a linker flag assumes that - # some compilation flag is set. We don't pass - # CFLAGS from environment, so we should not do it - # for LDFLAGS too. - LDFLAGS= - - --prefix - --enable-static - --disable-shared - --disable-symbol-hiding - - --with-zlib=${FOUND_ZLIB_ROOT_DIR} - ${LIBCURL_OPENSSL_OPT} - --with-ca-fallback - - --without-brotli - --without-gnutls - --without-mbedtls - --without-cyassl - --without-wolfssl - --without-mesalink - --without-nss - --without-ca-bundle - --without-ca-path - --without-libpsl - --without-libmetalink - --without-librtmp - --without-winidn - --without-libidn2 - --without-nghttp2 - --without-ngtcp2 - --without-nghttp3 - --without-quiche - --without-zsh-functions-dir - --without-fish-functions-dir - - ${ENABLED_DNS_OPT} - --enable-http - --enable-proxy - --enable-ipv6 - --enable-unix-sockets - --enable-cookies - --enable-http-auth - --enable-mime - --enable-dateparse - - ${DISABLED_DNS_OPT} - --disable-ftp - --disable-file - --disable-ldap - --disable-ldaps - --disable-rtsp - --disable-dict - --disable-telnet - --disable-tftp - --disable-pop3 - --disable-imap - --disable-smb - --disable-smtp - --disable-gopher - --disable-manual - --disable-sspi - --disable-crypto-auth - --disable-ntlm-wb - --disable-tls-srp - --disable-doh - --disable-netrc - --disable-progress-meter - --disable-dnsshuffle - --disable-alt-svc - BUILD_COMMAND cd && $(MAKE) + cd && cmake + ${LIBCURL_CMAKE_FLAGS} + BUILD_COMMAND cd && $(MAKE) -j INSTALL_COMMAND cd && $(MAKE) install) add_library(bundled-libcurl STATIC IMPORTED GLOBAL) @@ -168,6 +165,7 @@ macro(curl_build) endif() add_dependencies(bundled-libcurl bundled-libcurl-project) + # setup CURL_INCLUDE_DIRS & CURL_LIBRARIES for global use set(CURL_INCLUDE_DIRS ${LIBCURL_INSTALL_DIR}/include) set(CURL_LIBRARIES bundled-libcurl ${LIBZ_LIBRARY}) if (BUNDLED_LIBCURL_USE_ARES) @@ -177,13 +175,7 @@ macro(curl_build) set(CURL_LIBRARIES ${CURL_LIBRARIES} rt) endif() - unset(ASYN_DNS_USED) - unset(ASYN_DNS_UNUSED) - unset(ASYN_DNS_PATH) - unset(ENABLED_DNS_OPT) - unset(DISABLED_DNS_OPT) - unset(LIBCURL_CPPFLAGS) - unset(LIBCURL_CFLAGS) + unset(FOUND_ZLIB_ROOT_DIR) unset(FOUND_OPENSSL_ROOT_DIR) unset(LIBCURL_INSTALL_DIR) unset(LIBCURL_BINARY_DIR) -- 2.25.1