[Tarantool-patches] [PATCH v1 0/4] implement OOS build

[Tarantool-patches] [PATCH v1 0/4] implement OOS build

[Alexander V. Tikhonov]

1) gitlab-ci: add out-of-source build
    
Implemented out-of-source build at cmake files.
Added out of source build make targets and added
test job to gitlab-ci.
    
Skipped failing test app-tap/http_client.test.lua
from out-of-source build. Left #4258 issue opened
till it will be fixed.
    
Temporary added branch in curl repository
with fix for out-of-source build.
    
2) build: port autoconf to cmake in curl build
    
Changed autoconf tools in Curl build to cmake use for builds where
cmake major starts from 3. It was made so because cmake in curl
build works only with 3.0 version and above, the following issue
found on:
  CentOS 6,7
  Ubuntu 14.04
  
Issue with curl cmake based build with cmake before 3.0 version:
  CMake Error at CMakeLists.txt:41 (cmake_minimum_required):
    CMake 3.0 or higher is required.  You are running version 2.8.12.2

For curl cmake buld all autoconf options were ported to cmake
configuration call.
    
Also found that CURL cmake configuration file:
  third_party/curl/lib/CMakeLists.txt
    
has installation part for built libcurl.a library:
  install(TARGETS ${LIB_NAME}
    EXPORT ${TARGETS_EXPORT_NAME}
      ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
      LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}
    
where it changes CMAKE_INSTALL_LIBDIR to appropriate name with
  suffix of the architecture, like:
    lib
    lib64
    x86_64
    
Found that find_library routine from the file:
  cmake/FindLibCURL.cmake
    
returns only 'lib' value and it breaks the building of the depends
binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to
cmake call:
  -DCMAKE_INSTALL_LIBDIR=lib
    
Closes #4874
Closes #4968

---

Github: https://github.com/tarantool/tarantool/tree/avtikhon/gh-4874-out-of-source-build-full-ci
Issue: https://github.com/tarantool/tarantool/issues/4968
Issue: https://github.com/tarantool/tarantool/issues/4874

Alexander V. Tikhonov (4):
  build: port autoconf to cmake in curl build
  Fix curl repository for out-of-source build
  Skip failing test from out-of-source build
  gitlab-ci: add out-of-source build

 .gitlab-ci.yml                    |   7 +
 .travis.mk                        |  26 ++++
 cmake/BuildLibCURL.cmake          | 240 +++++++++++++++++-------------
 cmake/utils.cmake                 |   4 +-
 src/box/CMakeLists.txt            |   1 +
 test/app-tap/http_client.skipcond |   5 +
 third_party/curl                  |   2 +-
 7 files changed, 180 insertions(+), 105 deletions(-)

-- 
2.17.1

[Tarantool-patches] [PATCH v1 1/4] build: port autoconf to cmake in curl build

[Alexander V. Tikhonov]

Changed autoconf tools in Curl build to cmake use for builds where
cmake major starts from 3. It was made so because cmake in curl
build works only with 3.0 version and above, the following issue
found on:
  CentOS 6,7
  Ubuntu 14.04

Issue with curl cmake based build with cmake before 3.0 version:
  CMake Error at CMakeLists.txt:41 (cmake_minimum_required):
    CMake 3.0 or higher is required.  You are running version 2.8.12.2

For curl cmake buld all autoconf options were ported to cmake
configuration call.

Also found that CURL cmake configuration file:
  third_party/curl/lib/CMakeLists.txt

has installation part for built libcurl.a library:
  install(TARGETS ${LIB_NAME}
    EXPORT ${TARGETS_EXPORT_NAME}
    ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR}
    LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}

where it changes CMAKE_INSTALL_LIBDIR to appropriate name with
suffix of the architecture, like:
  lib
  lib64
  x86_64

Found that find_library routine from the file:
  cmake/FindLibCURL.cmake

returns only 'lib' value and it breaks the building of the depends
binaries. To avoid of it the CMAKE_INSTALL_LIBDIR option was set to
cmake call:
  -DCMAKE_INSTALL_LIBDIR=lib

Closes #4968
---
 cmake/BuildLibCURL.cmake | 240 ++++++++++++++++++++++-----------------
 1 file changed, 138 insertions(+), 102 deletions(-)

diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake
index 5f8b15a63..39566145e 100644
--- a/cmake/BuildLibCURL.cmake
+++ b/cmake/BuildLibCURL.cmake
@@ -25,10 +25,14 @@ macro(curl_build)
         set(ASYN_DNS_USED "ares")
         set(ASYN_DNS_UNUSED "threaded-resolver")
         set(ASYN_DNS_PATH "=${ARES_INSTALL_DIR}")
+        set(ENABLE_ARES "ON")
+        set(CMAKE_FIND_ROOT_PATH "${ARES_INSTALL_DIR}")
     else()
         set(ASYN_DNS_USED "threaded-resolver")
         set(ASYN_DNS_UNUSED "ares")
         set(ASYN_DNS_PATH "")
+        set(ENABLE_ARES "OFF")
+        set(CMAKE_FIND_ROOT_PATH "")
     endif()
 
     set(ENABLED_DNS_OPT "--enable-${ASYN_DNS_USED}${ASYN_DNS_PATH}")
@@ -49,108 +53,140 @@ macro(curl_build)
     endif()
 
     include(ExternalProject)
-    ExternalProject_Add(
-        bundled-libcurl-project
-        SOURCE_DIR ${LIBCURL_SOURCE_DIR}
-        PREFIX ${LIBCURL_INSTALL_DIR}
-        DOWNLOAD_DIR ${LIBCURL_BINARY_DIR}
-        TMP_DIR ${LIBCURL_BINARY_DIR}/tmp
-        STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp
-        BINARY_DIR ${LIBCURL_BINARY_DIR}
-        CONFIGURE_COMMAND
-            cd <SOURCE_DIR> && ./buildconf &&
-            cd <BINARY_DIR> && <SOURCE_DIR>/configure
-                # Pass the same toolchain as is used to build
-                # tarantool itself, because they can be
-                # incompatible.
-                CC=${CMAKE_C_COMPILER}
-                LD=${CMAKE_LINKER}
-                AR=${CMAKE_AR}
-                RANLIB=${CMAKE_RANLIB}
-                NM=${CMAKE_NM}
-                STRIP=${CMAKE_STRIP}
-
-                # Pass -isysroot=<SDK_PATH> option on Mac OS, see
-                # above.
-                # Note: Passing of CPPFLAGS / CFLAGS explicitly
-                # discards using of corresponsing environment
-                # variables.
-                CPPFLAGS=${LIBCURL_CPPFLAGS}
-                CFLAGS=${LIBCURL_CFLAGS}
-
-                # Pass empty LDFLAGS to discard using of
-                # corresponding environment variable.
-                # It is possible that a linker flag assumes that
-                # some compilation flag is set. We don't pass
-                # CFLAGS from environment, so we should not do it
-                # for LDFLAGS too.
-                LDFLAGS=
-
-                --prefix <INSTALL_DIR>
-                --enable-static
-                --enable-shared
-
-                --with-zlib
-                ${LIBCURL_OPENSSL_OPT}
-                --with-ca-fallback
-
-                --without-brotli
-                --without-gnutls
-                --without-mbedtls
-                --without-cyassl
-                --without-wolfssl
-                --without-mesalink
-                --without-nss
-                --without-ca-bundle
-                --without-ca-path
-                --without-libpsl
-                --without-libmetalink
-                --without-librtmp
-                --without-winidn
-                --without-libidn2
-                --without-nghttp2
-                --without-ngtcp2
-                --without-nghttp3
-                --without-quiche
-                --without-zsh-functions-dir
-                --without-fish-functions-dir
-
-                ${ENABLED_DNS_OPT}
-                --enable-http
-                --enable-proxy
-                --enable-ipv6
-                --enable-unix-sockets
-                --enable-cookies
-                --enable-http-auth
-                --enable-mime
-                --enable-dateparse
-
-                ${DISABLED_DNS_OPT}
-                --disable-ftp
-                --disable-file
-                --disable-ldap
-                --disable-ldaps
-                --disable-rtsp
-                --disable-dict
-                --disable-telnet
-                --disable-tftp
-                --disable-pop3
-                --disable-imap
-                --disable-smb
-                --disable-smtp
-                --disable-gopher
-                --disable-manual
-                --disable-sspi
-                --disable-crypto-auth
-                --disable-ntlm-wb
-                --disable-tls-srp
-                --disable-doh
-                --disable-netrc
-                --disable-progress-meter
-                --disable-dnsshuffle
-                --disable-alt-svc
-        BUILD_COMMAND cd <BINARY_DIR> && $(MAKE)
-        INSTALL_COMMAND cd <BINARY_DIR> && $(MAKE) install)
+    if(${CMAKE_MAJOR_VERSION} VERSION_LESS "3")
+        ExternalProject_Add(
+            bundled-libcurl-project
+            SOURCE_DIR ${LIBCURL_SOURCE_DIR}
+            PREFIX ${LIBCURL_INSTALL_DIR}
+            DOWNLOAD_DIR ${LIBCURL_BINARY_DIR}
+            TMP_DIR ${LIBCURL_BINARY_DIR}/tmp
+            STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp
+            BINARY_DIR ${LIBCURL_BINARY_DIR}
+            CONFIGURE_COMMAND
+                cd <SOURCE_DIR> && ./buildconf &&
+                cd <BINARY_DIR> && <SOURCE_DIR>/configure
+                    # Pass the same toolchain as is used to build
+                    # tarantool itself, because they can be
+                    # incompatible.
+                    CC=${CMAKE_C_COMPILER}
+                    LD=${CMAKE_LINKER}
+                    AR=${CMAKE_AR}
+                    RANLIB=${CMAKE_RANLIB}
+                    NM=${CMAKE_NM}
+                    STRIP=${CMAKE_STRIP}
+
+                    # Pass -isysroot=<SDK_PATH> option on Mac OS, see
+                    # above.
+                    # Note: Passing of CPPFLAGS / CFLAGS explicitly
+                    # discards using of corresponsing environment
+                    # variables.
+                    CPPFLAGS=${LIBCURL_CPPFLAGS}
+                    CFLAGS=${LIBCURL_CFLAGS}
+
+                    # Pass empty LDFLAGS to discard using of
+                    # corresponding environment variable.
+                    # It is possible that a linker flag assumes that
+                    # some compilation flag is set. We don't pass
+                    # CFLAGS from environment, so we should not do it
+                    # for LDFLAGS too.
+                    LDFLAGS=
+
+                    --prefix <INSTALL_DIR>
+                    --enable-static
+                    --enable-shared
+
+                    --with-zlib
+                    ${LIBCURL_OPENSSL_OPT}
+                    --with-ca-fallback
+
+                    --without-brotli
+                    --without-gnutls
+                    --without-mbedtls
+                    --without-cyassl
+                    --without-wolfssl
+                    --without-mesalink
+                    --without-nss
+                    --without-ca-bundle
+                    --without-ca-path
+                    --without-libpsl
+                    --without-libmetalink
+                    --without-librtmp
+                    --without-winidn
+                    --without-libidn2
+                    --without-nghttp2
+                    --without-ngtcp2
+                    --without-nghttp3
+                    --without-quiche
+                    --without-zsh-functions-dir
+                    --without-fish-functions-dir
+
+                    ${ENABLED_DNS_OPT}
+                    --enable-http
+                    --enable-proxy
+                    --enable-ipv6
+                    --enable-unix-sockets
+                    --enable-cookies
+                    --enable-http-auth
+                    --enable-mime
+                    --enable-dateparse
+
+                    ${DISABLED_DNS_OPT}
+                    --disable-ftp
+                    --disable-file
+                    --disable-ldap
+                    --disable-ldaps
+                    --disable-rtsp
+                    --disable-dict
+                    --disable-telnet
+                    --disable-tftp
+                    --disable-pop3
+                    --disable-imap
+                    --disable-smb
+                    --disable-smtp
+                    --disable-gopher
+                    --disable-manual
+                    --disable-sspi
+                    --disable-crypto-auth
+                    --disable-ntlm-wb
+                    --disable-tls-srp
+                    --disable-doh
+                    --disable-netrc
+                    --disable-progress-meter
+                    --disable-dnsshuffle
+                    --disable-alt-svc
+            BUILD_COMMAND cd <BINARY_DIR> && $(MAKE)
+            INSTALL_COMMAND cd <BINARY_DIR> && $(MAKE) install)
+    else()
+        ExternalProject_Add(
+            bundled-libcurl-project
+            SOURCE_DIR ${LIBCURL_SOURCE_DIR}
+            PREFIX ${LIBCURL_INSTALL_DIR}
+            DOWNLOAD_DIR ${LIBCURL_BINARY_DIR}
+            TMP_DIR ${LIBCURL_BINARY_DIR}/tmp
+            STAMP_DIR ${LIBCURL_BINARY_DIR}/stamp
+            BINARY_DIR ${LIBCURL_BINARY_DIR}/curl
+            CONFIGURE_COMMAND
+	        cd <BINARY_DIR> && cmake <SOURCE_DIR>
+                    -DCPPFLAGS=${LIBCURL_CPPFLAGS}
+                    -DCFLAGS=${LIBCURL_CFLAGS}
+                    -DLDFLAGS=
+                    -DCMAKE_INSTALL_LIBDIR=lib
+                    -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
+                    -DBUILD_TESTING=OFF
+                    -DCURL_STATICLIB=ON
+                    -DBUILD_SHARED_LIBS=OFF
+                    -DCMAKE_USE_OPENSSL=ON
+                    -DOPENSSL_ROOT_DIR=${FOUND_OPENSSL_ROOT_DIR}
+                    -DCURL_CA_FALLBACK=ON
+                    -DCMAKE_FIND_ROOT_PATH=${CMAKE_FIND_ROOT_PATH}
+                    -DENABLE_ARES=${ENABLE_ARES}
+                    -DHTTP_ONLY=ON
+                    -DCURL_DISABLE_SMB=ON
+                    -DCURL_DISABLE_GOPHER=ON
+                    -DCURL_DISABLE_CRYPTO_AUTH=ON
+            BUILD_COMMAND cd <BINARY_DIR> && $(MAKE) -j
+            INSTALL_COMMAND cd <BINARY_DIR> && $(MAKE) install)
+    endif()
 
     add_library(bundled-libcurl STATIC IMPORTED GLOBAL)
     set_target_properties(bundled-libcurl PROPERTIES IMPORTED_LOCATION
-- 
2.17.1

[Tarantool-patches] [PATCH v1 2/4] Fix curl repository for out-of-source build

[Alexander V. Tikhonov]

Temporary added branch in curl repository
with fix for out-of-source build.

Part of #4874
---
 third_party/curl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/third_party/curl b/third_party/curl
index 9cd755e1d..bf6df48c8 160000
--- a/third_party/curl
+++ b/third_party/curl
@@ -1 +1 @@
-Subproject commit 9cd755e1d768bbf228e7c9faf223b7459f7e0105
+Subproject commit bf6df48c8c9b8377f71c2a09ff545d828c7ed2fb
-- 
2.17.1

[Tarantool-patches] [PATCH v1 3/4] Skip failing test from out-of-source build

[Alexander V. Tikhonov]

Skipped failing test app-tap/http_client.test.lua
from out-of-source build. Left #4258 issue opened
till it will be fixed.

Part of #4874
---
 test/app-tap/http_client.skipcond | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/test/app-tap/http_client.skipcond b/test/app-tap/http_client.skipcond
index 108e776c4..df811dcc1 100644
--- a/test/app-tap/http_client.skipcond
+++ b/test/app-tap/http_client.skipcond
@@ -1,5 +1,10 @@
+import os
 import platform
 
+# Disabled at out-of-source build due to issue #4258.
+if os.getenv("OUTOFSRC_BUILD") == 'ON':
+    self.skip = 1
+
 # Disabled on FreeBSD due to flaky fail #4271.
 if platform.system() == 'FreeBSD':
     self.skip = 1
-- 
2.17.1

[Tarantool-patches] [PATCH v1 4/4] gitlab-ci: add out-of-source build

[Alexander V. Tikhonov]

Implemented out-of-source build at cmake files.
Added out of source build make targets and added
test job to gitlab-ci.

Part of #4874
---
 .gitlab-ci.yml         |  7 +++++++
 .travis.mk             | 26 ++++++++++++++++++++++++++
 cmake/utils.cmake      |  4 ++--
 src/box/CMakeLists.txt |  1 +
 4 files changed, 36 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 256b368c4..d05c80692 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -105,6 +105,13 @@ variables:
 
 # Tests
 
+out_of_source:
+  stage: test
+  tags:
+    - deploy_test
+  script:
+    - ${GITLAB_MAKE} test_oos_build
+
 release:
   <<: *docker_test_definition
   script:
diff --git a/.travis.mk b/.travis.mk
index 063537f25..a7717c92a 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -3,9 +3,13 @@
 #
 
 DOCKER_IMAGE?=packpack/packpack:debian-stretch
+DOCKER_TEST_IMAGE?=registry.gitlab.com/tarantool/tarantool/testing/debian-stretch:latest
 TEST_RUN_EXTRA_PARAMS?=
 MAX_FILES?=65534
 MAX_PROC?=2500
+OOS_SRC_PATH?=/source
+OOS_BUILD_PATH?=/rw_bins
+OOS_BUILD_RULE?=test_oos_no_deps
 
 all: package
 
@@ -146,6 +150,28 @@ test_static_build: deps_debian_static
 test_static_docker_build:
 	docker build --no-cache --network=host --build-arg RUN_TESTS=ON -f Dockerfile.staticbuild .
 
+# Out-Of-Source build
+
+build_oos:
+	mkdir ${OOS_BUILD_PATH} 2>/dev/null || : ; \
+		cd ${OOS_BUILD_PATH} && \
+		cmake ${OOS_SRC_PATH} ${CMAKE_EXTRA_PARAMS} && \
+		make -j
+
+test_oos_no_deps: build_oos
+	cd ${OOS_BUILD_PATH}/test && \
+		OUTOFSRC_BUILD=ON ${OOS_SRC_PATH}/test/test-run.py \
+			--builddir ${OOS_BUILD_PATH} \
+			--vardir ${OOS_BUILD_PATH}/test/var --force
+
+test_oos: deps_debian test_oos_no_deps
+
+test_oos_build:
+	docker run --network=host -w ${OOS_SRC_PATH} \
+		--mount type=bind,source="${PWD}",target=${OOS_SRC_PATH},readonly,bind-propagation=rslave \
+		-i ${DOCKER_TEST_IMAGE} \
+		make -f .travis.mk ${OOS_BUILD_RULE}
+
 #######
 # OSX #
 #######
diff --git a/cmake/utils.cmake b/cmake/utils.cmake
index 3ab2d3ff2..eaec821b3 100644
--- a/cmake/utils.cmake
+++ b/cmake/utils.cmake
@@ -70,9 +70,9 @@ endfunction()
 
 function(bin_source varname srcfile dstfile)
     set(var ${${varname}})
-    set(${varname} ${var} ${dstfile} PARENT_SCOPE)
     set (srcfile "${CMAKE_CURRENT_SOURCE_DIR}/${srcfile}")
-    set (dstfile "${CMAKE_CURRENT_SOURCE_DIR}/${dstfile}")
+    set (dstfile "${CMAKE_CURRENT_BINARY_DIR}/${dstfile}")
+    set(${varname} ${var} "${dstfile}" PARENT_SCOPE)
     set (tmpfile "${dstfile}.tmp")
     get_filename_component(module ${dstfile} NAME_WE)
 
diff --git a/src/box/CMakeLists.txt b/src/box/CMakeLists.txt
index 7dc94d893..feec2de01 100644
--- a/src/box/CMakeLists.txt
+++ b/src/box/CMakeLists.txt
@@ -32,6 +32,7 @@ set_property(DIRECTORY PROPERTY ADDITIONAL_MAKE_CLEAN_FILES ${lua_sources})
 
 include_directories(${ZSTD_INCLUDE_DIRS})
 include_directories(${CMAKE_BINARY_DIR}/src/box/sql)
+include_directories(${CMAKE_BINARY_DIR}/src/box)
 
 add_library(vclock STATIC vclock.c)
 target_link_libraries(vclock core)
-- 
2.17.1