[Tarantool-patches] [PATCH v3] build: enable cmake in curl build

["Alexander V. Tikhonov" <avtikhon@tarantool.org>]

Initially tryed to change autoconf tools in Curl build to cmake and
found the following build issue on:
  CentOS 6,7
  Debian 8
  Ubuntu 14.04

  Issue found:
    CMake Error at CMakeLists.txt:41 (cmake_minimum_required):
      CMake 3.0 or higher is required.  You are running version 2.8.12.2

To fix the issue removed the check of the version from curl sources
in Tarantool's third party '<root Tarantool sources>/third_party/curl':

  12af024bc85606b14ffc415413a7e86e6bbee7eb "Enable curl build with old cmake"

After this fix completely changed autoconf to cmake in curl build.
Autoconf part completely removed and code cleaned up for cmake.
For curl cmake build all autoconf options were ported to cmake
configuration call, check the accurate list of the change in [1].

Also the following issues resolved:

1. Found that CURL cmake configuration file:
  third_party/curl/lib/CMakeLists.txt

has installation part for built libcurl.a library:
  install(TARGETS ${LIB_NAME}
    EXPORT ${TARGETS_EXPORT_NAME}
    ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}

where it changes CMAKE_INSTALL_LIBDIR to appropriate name with
suffix of the architecture, like:
  lib
  lib64
  x86_64

Found that find_library routine from the file:
  cmake/FindLibCURL.cmake
returns only 'lib' value and it breaks the building of the depends
binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to
cmake call:
  -DCMAKE_INSTALL_LIBDIR=lib

2. Found issue with building on CentOS 6:

     Linking C executable curl
     build/ares/dest/lib/libcares.a(ares__timeval.c.o): In function `ares__tvnow':
     ares__timeval.c:(.text+0x15): undefined reference to `clock_gettime'
     collect2: error: ld returned 1 exit status

   It was fixed with added "-lrt" flag to CMAKE_C_FLAGS and
   CMAKE_CXX_FLAGS build flags, when cmake version is lower
   than 3.0 and RT library had needed function.

3. Found issues with building Tarantool statically on Debian 9 and
   its package build on CentOS 8.

   Building statically got the issues with openssl linking, like [2]:

     static-build/openssl-prefix/lib/libcrypto.a(threads_pthread.o): In function `CRYPTO_THREAD_lock_new':
     threads_pthread.c:(.text+0x45): undefined reference to `pthread_rwlock_init'

   It happened because in openssl radically changed how threads-related
   things were handled before 1.1.0 version. It required the application
   to provide us with the locking functionality in form of callbacks.
   Since 1.1.0, these matters are entirely internal, so libcrypto
   requires the presence of the platform specific thread implementation
   of our choosing, which is pthread on everything.

   After '-pthread' added to C compile flags package build on CentOS 8
   failed with the issue [3]:

     /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:101:4: error: #error "SIZEOF_CURL_OFF_T not defined"
      #  error "SIZEOF_CURL_OFF_T not defined"
         ^~~~~
     /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c: In function 'curlx_uztoso':
     /build/usr/src/debug/tarantool-2.6.0.141/third_party/curl/lib/warnless.c:192:40: error: 'CURL_MASK_SCOFFT' undeclared (first use in this function); did you mean 'CURL_MASK_SINT'?
      return (curl_off_t)(uznum & (size_t) CURL_MASK_SCOFFT);
                                           ^~~~~~~~~~~~~~~~
                                           CURL_MASK_SINT

   To avoid of the issue decided to use '-pthread' flag only for openssl
   when it had not this flag in openssl compilation:

     # check '-pthread' in openssl compile options
     execute_process(COMMAND openssl version -f
                     OUTPUT_VARIABLE OPENSSL_COMPILE_OPTIONS)
     # add pthread library for openssl static library linking
     if(NOT OPENSSL_COMPILE_OPTIONS MATCHES ".* -pthread .*")
         list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-pthread")
     endif()

4. Found issue with static build using CentOS 7, where SSL cmake rule
   failed. Building the image got the issue:

      [  1%] Performing configure step for 'bundled-libcurl-project'
      CMake Warning at CMakeLists.txt:50 (message):
        the curl cmake build system is poorly maintained.  Be aware

      -- curl version=[7.66.0-DEV]
      -- Found c-ares: /tarantool/build/ares/dest/lib/libcares.a
      Found *nroff option: -- -man
      CMake Error at /usr/share/cmake/Modules/FindOpenSSL.cmake:278 (list):
        list GET given empty list
      Call Stack (most recent call first):
        CMakeLists.txt:347 (find_package)

    Root cause of the issue that Dockerfile uses globaly
    installed openSSL with:

      cmake ... -DOPENSSL_ROOT_DIR=/usr/local ...

    Its cmake build file:

      /usr/share/cmake/Modules/FindOpenSSL.cmake

    fails on parsing the SSL version:

    it has:
           REGEX "^#define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")

    but it should to use:
           REGEX "^#[\t ]*define[\t ]+OPENSSL_VERSION_NUMBER[\t ]+0x([0-9a-fA-F])+.*")

    Anyway we want to use the same OpenSSL library for libcurl, as is used
    for Tarantool itself. So the path to it set for cURL build:

      list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake")

Closes #4968
Closes #5396
Closes #5019

[1] - https://github.com/tarantool/tarantool/issues/4968#issue-617183031
[2] - https://gitlab.com/tarantool/tarantool/-/jobs/779176133#L6021
[3] - https://gitlab.com/tarantool/tarantool/-/jobs/778309145#L3060
---


Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4874-out-of-source-build-full-ci
Issue: https://github.com/tarantool/tarantool/issues/4968
Issue: https://github.com/tarantool/tarantool/issues/5396
Issue: https://github.com/tarantool/tarantool/issues/5019

V3: Added explicitly all disabled flags.
    Fixed #5396.
    Removed fix for #5020 as not needed now.

 cmake/BuildLibCURL.cmake | 249 +++++++++++++++++++--------------------
 1 file changed, 118 insertions(+), 131 deletions(-)

diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake
index 86fec39e9..911e49f11 100644
--- a/cmake/BuildLibCURL.cmake
+++ b/cmake/BuildLibCURL.cmake
@@ -3,11 +3,12 @@ macro(curl_build)
     set(LIBCURL_SOURCE_DIR ${PROJECT_SOURCE_DIR}/third_party/curl)
     set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work)
     set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest)
+    set(LIBCURL_CMAKE_FLAGS "")
 
     message(STATUS "Looking for zlib")
     find_path(ZLIB_INCLUDE_DIR zlib.h)
     message(STATUS "Looking for zlib.h - ${ZLIB_INCLUDE_DIR}")
-    if (BUILD_STATIC)
+    if(BUILD_STATIC)
         set(LIBZ_LIB_NAME libz.a)
     else()
         set(LIBZ_LIB_NAME z)
@@ -15,44 +16,125 @@ macro(curl_build)
     find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME})
     message(STATUS "Looking for libz - ${LIBZ_LIBRARY}")
 
-    if (NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY)
+    if(NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY)
         message(FATAL_ERROR "Unable to find zlib")
     endif()
     get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR} DIRECTORY)
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DZLIB_ROOT=${FOUND_ZLIB_ROOT_DIR}")
+
+    # check '-pthread' in openssl compile options
+    execute_process(COMMAND openssl version -f
+                    OUTPUT_VARIABLE OPENSSL_COMPILE_OPTIONS)
+    # add pthread library for openssl static library linking
+    if(NOT OPENSSL_COMPILE_OPTIONS MATCHES ".* -pthread .*")
+        list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-pthread")
+    endif()
+
+    # add librt for clock_gettime function definition
+    if(${CMAKE_MAJOR_VERSION} VERSION_LESS "3")
+        CHECK_LIBRARY_EXISTS (rt clock_gettime "" HAVE_LIBRT)
+        if (HAVE_LIBRT)
+            list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_FLAGS=-lrt")
+            list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_CXX_FLAGS=-lrt")
+        endif()
+    endif()
+
+    # switch on the static build
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_STATICLIB=ON")
 
-    # Use the same OpenSSL library for libcurl as is used for
-    # tarantool itself.
+    # switch off the shared build
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_SHARED_LIBS=OFF")
+
+    # let's disable testing for curl to save build time
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DBUILD_TESTING=OFF")
+
+    # Setup use of openssl, use the same OpenSSL library
+    # for libcurl as is used for tarantool itself.
     get_filename_component(FOUND_OPENSSL_ROOT_DIR ${OPENSSL_INCLUDE_DIR} DIRECTORY)
-    set(LIBCURL_OPENSSL_OPT "--with-ssl=${FOUND_OPENSSL_ROOT_DIR}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_OPENSSL=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DOPENSSL_ROOT_DIR=${FOUND_OPENSSL_ROOT_DIR}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_MODULE_PATH=${PROJECT_SOURCE_DIR}/cmake")
 
-    # Use either c-ares bundled with tarantool or
-    # libcurl-default threaded resolver.
+    # Setup ARES and its library path, use either c-ares bundled
+    # with tarantool or libcurl-default threaded resolver.
     if(BUNDLED_LIBCURL_USE_ARES)
-        set(ASYN_DNS_USED "ares")
-        set(ASYN_DNS_UNUSED "threaded-resolver")
-        set(ASYN_DNS_PATH "=${ARES_INSTALL_DIR}")
+        set(ENABLE_ARES "ON")
+        list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_FIND_ROOT_PATH=${ARES_INSTALL_DIR}")
     else()
-        set(ASYN_DNS_USED "threaded-resolver")
-        set(ASYN_DNS_UNUSED "ares")
-        set(ASYN_DNS_PATH "")
-    endif()
-
-    set(ENABLED_DNS_OPT "--enable-${ASYN_DNS_USED}${ASYN_DNS_PATH}")
-    set(DISABLED_DNS_OPT "--disable-${ASYN_DSN_UNUSED}")
-
-    # Pass -isysroot=<SDK_PATH> option on Mac OS to a preprocessor
-    # and a C compiler to find header files installed with an SDK.
-    #
-    # The idea here is to don't pass all compiler/linker options
-    # that is set for tarantool, but only a subset that is
-    # necessary for choosen toolchain, and let curl's configure
-    # script set options that are appropriate for libcurl.
-    set(LIBCURL_CPPFLAGS "")
-    set(LIBCURL_CFLAGS "")
-    if (TARGET_OS_DARWIN AND NOT "${CMAKE_OSX_SYSROOT}" STREQUAL "")
-        set(LIBCURL_CPPFLAGS "${LIBCURL_CPPFLAGS} ${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}")
-        set(LIBCURL_CFLAGS "${LIBCURL_CFLAGS} ${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}")
+        set(ENABLE_ARES "OFF")
     endif()
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_ARES=${ENABLE_ARES}")
+
+    # switch off the group of protocols with special flag HTTP_ONLY:
+    #   ftp, file, ldap, ldaps, rtsp, dict, telnet, tftp, pop3, imap, smtp
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DHTTP_ONLY=ON")
+
+    # additionaly disable some more protocols
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_SMB=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_GOPHER=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_CRYPTO_AUTH=ON")
+
+    # switch on ca-fallback feature
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_FALLBACK=ON")
+
+    # Even though we set the external project's install dir
+    # below, we still need to pass the corresponding install
+    # prefix via cmake arguments.
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_PREFIX=${LIBCURL_INSTALL_DIR}")
+
+    # The default values for the options below are not always
+    # "./lib", "./bin"  and "./include", while curl expects them
+    # to be.
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_LIBDIR=lib")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_INCLUDEDIR=include")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_INSTALL_BINDIR=bin")
+
+    # Pass the same toolchain as is used to build tarantool itself,
+    # because they can be incompatible.
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_LINKER=${CMAKE_LINKER}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_AR=${CMAKE_AR}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_RANLIB=${CMAKE_RANLIB}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_NM=${CMAKE_NM}")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_STRIP=${CMAKE_STRIP}")
+
+    # In hardened mode, which enables -fPIE by default,
+    # the cmake checks don't work without -fPIC.
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_REQUIRED_FLAGS=-fPIC")
+
+    # Need to set values explicitly everything that is default, because
+    # we don't know how defaults will be changed in a future and we don't
+    # want to verify them each time we'll bump new libcurl version.
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_BROTLI=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_GNUTLS=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_MBEDTLS=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_WOLFSSL=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCMAKE_USE_NSS=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_BUNDLE=none")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_CA_PATH=none")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_LIBRTMP=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DHAVE_LIBIDN2=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGHTTP2=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGTCP2=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_NGHTTP3=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DUSE_QUICHE=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_HTTP=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_PROXY=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_IPV6=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_COOKIES=OFF")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DENABLE_UNIX_SOCKETS=ON")
+    # should be already set by "-DHTTP_ONLY=ON" above
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_FTP=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_FILE=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_LDAP=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_LDAPS=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_RTSP=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_DICT=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_TELNET=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_TFTP=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_POP3=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_IMAP=ON")
+    list(APPEND LIBCURL_CMAKE_FLAGS "-DCURL_DISABLE_SMTP=ON")
 
     include(ExternalProject)
     ExternalProject_Add(
@@ -62,101 +144,11 @@ macro(curl_build)
         DOWNLOAD_DIR ${LIBCURL_BINARY_DIR}
         TMP_DIR ${LIBCURL_BINARY_DIR}/tmp
         STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp
-        BINARY_DIR ${LIBCURL_BINARY_DIR}
+        BINARY_DIR ${LIBCURL_BINARY_DIR}/curl
         CONFIGURE_COMMAND
-            cd <SOURCE_DIR> && ./buildconf &&
-            cd <BINARY_DIR> && <SOURCE_DIR>/configure
-                # Pass the same toolchain as is used to build
-                # tarantool itself, because they can be
-                # incompatible.
-                CC=${CMAKE_C_COMPILER}
-                LD=${CMAKE_LINKER}
-                AR=${CMAKE_AR}
-                RANLIB=${CMAKE_RANLIB}
-                NM=${CMAKE_NM}
-                STRIP=${CMAKE_STRIP}
-
-                # Pass -isysroot=<SDK_PATH> option on Mac OS, see
-                # above.
-                # Note: Passing of CPPFLAGS / CFLAGS explicitly
-                # discards using of corresponsing environment
-                # variables.
-                CPPFLAGS=${LIBCURL_CPPFLAGS}
-                CFLAGS=${LIBCURL_CFLAGS}
-
-                # Pass empty LDFLAGS to discard using of
-                # corresponding environment variable.
-                # It is possible that a linker flag assumes that
-                # some compilation flag is set. We don't pass
-                # CFLAGS from environment, so we should not do it
-                # for LDFLAGS too.
-                LDFLAGS=
-
-                --prefix <INSTALL_DIR>
-                --enable-static
-                --disable-shared
-                --disable-symbol-hiding
-
-                --with-zlib=${FOUND_ZLIB_ROOT_DIR}
-                ${LIBCURL_OPENSSL_OPT}
-                --with-ca-fallback
-
-                --without-brotli
-                --without-gnutls
-                --without-mbedtls
-                --without-cyassl
-                --without-wolfssl
-                --without-mesalink
-                --without-nss
-                --without-ca-bundle
-                --without-ca-path
-                --without-libpsl
-                --without-libmetalink
-                --without-librtmp
-                --without-winidn
-                --without-libidn2
-                --without-nghttp2
-                --without-ngtcp2
-                --without-nghttp3
-                --without-quiche
-                --without-zsh-functions-dir
-                --without-fish-functions-dir
-
-                ${ENABLED_DNS_OPT}
-                --enable-http
-                --enable-proxy
-                --enable-ipv6
-                --enable-unix-sockets
-                --enable-cookies
-                --enable-http-auth
-                --enable-mime
-                --enable-dateparse
-
-                ${DISABLED_DNS_OPT}
-                --disable-ftp
-                --disable-file
-                --disable-ldap
-                --disable-ldaps
-                --disable-rtsp
-                --disable-dict
-                --disable-telnet
-                --disable-tftp
-                --disable-pop3
-                --disable-imap
-                --disable-smb
-                --disable-smtp
-                --disable-gopher
-                --disable-manual
-                --disable-sspi
-                --disable-crypto-auth
-                --disable-ntlm-wb
-                --disable-tls-srp
-                --disable-doh
-                --disable-netrc
-                --disable-progress-meter
-                --disable-dnsshuffle
-                --disable-alt-svc
-        BUILD_COMMAND cd <BINARY_DIR> && $(MAKE)
+            cd <BINARY_DIR> && cmake <SOURCE_DIR>
+                ${LIBCURL_CMAKE_FLAGS}
+        BUILD_COMMAND cd <BINARY_DIR> && $(MAKE) -j
         INSTALL_COMMAND cd <BINARY_DIR> && $(MAKE) install)
 
     add_library(bundled-libcurl STATIC IMPORTED GLOBAL)
@@ -168,6 +160,7 @@ macro(curl_build)
     endif()
     add_dependencies(bundled-libcurl bundled-libcurl-project)
 
+    # setup CURL_INCLUDE_DIRS & CURL_LIBRARIES for global use
     set(CURL_INCLUDE_DIRS ${LIBCURL_INSTALL_DIR}/include)
     set(CURL_LIBRARIES bundled-libcurl ${LIBZ_LIBRARY})
     if (BUNDLED_LIBCURL_USE_ARES)
@@ -177,13 +170,7 @@ macro(curl_build)
         set(CURL_LIBRARIES ${CURL_LIBRARIES} rt)
     endif()
 
-    unset(ASYN_DNS_USED)
-    unset(ASYN_DNS_UNUSED)
-    unset(ASYN_DNS_PATH)
-    unset(ENABLED_DNS_OPT)
-    unset(DISABLED_DNS_OPT)
-    unset(LIBCURL_CPPFLAGS)
-    unset(LIBCURL_CFLAGS)
+    unset(FOUND_ZLIB_ROOT_DIR)
     unset(FOUND_OPENSSL_ROOT_DIR)
     unset(LIBCURL_INSTALL_DIR)
     unset(LIBCURL_BINARY_DIR)
-- 
2.25.1