Tarantool development patches archive
 help / color / mirror / Atom feed
* [Tarantool-patches] [PATCH v3] build: refactor static build process
@ 2020-09-09 15:45 HustonMmmavr
  2020-09-10 10:00 ` Alexandr Barulev
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: HustonMmmavr @ 2020-09-09 15:45 UTC (permalink / raw)
  To: tarantool-patches, yaroslav.dynnikov, alexander.turenko, imun

From: Yaroslav Dynnikov <yaroslav.dynnikov@gmail.com>

Refactored static build process to use static-build/CMakeLists.txt
instead of Dockerfile.staticbuild (this allows to support static
build on macOS). Following third-party dependencies for static build
are installed via cmake `ExternalProject_Add`:
  - OpenSSL
  - Zlib
  - Ncurses
  - Readline
  - Unwind
  - ICU

* Added support static build for macOS
* Fixed `CONFIGURE_COMMAND` while building bundled libcurl for static
  build at file cmake/BuildLibCURL.cmake:
    - disable building shared libcurl libraries (by setting
      `--disable-shared` option)
    - disable hiding libcurl symbols (by setting
      `--disable-symbol-hiding` option)
    - prevent linking libcurl with system libz (by setting
      `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option)
* Removed Dockerfile.staticbuild
* Added new gitlab.ci jobs to test new style static build:
  - static_build_cmake_linux
  - static_build_cmake_osx_15
* Removed static_docker_build gitlab.ci job

Closes #5095
---

Branch: https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build
Issue: https://github.com/tarantool/tarantool/issues/5095

Changes in v3:
1. Fixed typos at commit message
2. Fixed commentaries at .travis.mk
3. Fixed finding libz library at cmake/BuildLibCurl.cmake as at master
   branch
4. Instead of using static-build/.gitignore use the root .gitignore file
5. Refactored static-build/CMakeLists.txt:
   * Project name is set to tarantool-static
   * Instead of explicitly finding of c/c++ compilers current version uses
     cmake `project(tarantool-static C CXX)` command. This change requires
     to manually set CFLAGS/CPPFLAGS/LDFLAGS for all static-build
     dependencies (because building dependencies at macOS requires path
     to macOS SDK) the idea was taken from cmake/BuildLibCurl.cmake
   * Added commentaries about problem with libicu
   * Removed unused TEST_COMMAND at ExternalProject_Add(zlib ...) also
     removed unnecessary # STEP_TARGETS at ExternalProject_Add(readline ...)
   * Deleted doubled whitespaces
6. Fixed static-build/README.md:
   * Added list of required tools
   * Added example for ubuntu/debian
   * Fixed indentation and typos
7. Refactored tests:
   * Got rid of test-run
   * Deleted doubled whitespaces at exports.test.lua
   * Fixed defining of RTLD_DEFAULT


 .gitignore                                    |   8 +
 .gitlab-ci.yml                                |  11 +-
 .travis.mk                                    |  56 +++-
 Dockerfile.staticbuild                        |  98 ------
 cmake/BuildLibCURL.cmake                      |  13 +-
 cmake/compiler.cmake                          |  24 +-
 cmake/os.cmake                                |   5 +-
 static-build/CMakeLists.txt                   | 311 ++++++++++++++++++
 static-build/README.md                        |  90 +++++
 static-build/test/CheckDependencies.cmake     |  43 +++
 .../test/static-build/exports.test.lua        | 142 ++++++++
 .../test/static-build/traceback.test.lua      |  15 +
 12 files changed, 692 insertions(+), 124 deletions(-)
 delete mode 100644 Dockerfile.staticbuild
 create mode 100644 static-build/CMakeLists.txt
 create mode 100644 static-build/README.md
 create mode 100644 static-build/test/CheckDependencies.cmake
 create mode 100755 static-build/test/static-build/exports.test.lua
 create mode 100755 static-build/test/static-build/traceback.test.lua

diff --git a/.gitignore b/.gitignore
index a42c7db35..c6e261e18 100644
--- a/.gitignore
+++ b/.gitignore
@@ -151,3 +151,11 @@ src/box/sql/parse.h
 src/box/sql/parse.c
 src/box/sql/opcodes.h
 src/box/sql/opcodes.c
+
+# static-build
+static-build/*-prefix
+static-build/Makefile
+static-build/build
+
+# macOS files
+.DS_Store
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0ead08711..c9aef3dc7 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -534,9 +534,14 @@ static_build:
   script:
     - ${GITLAB_MAKE} test_static_build
 
-static_docker_build:
+static_build_cmake_linux:
+  <<: *docker_test_definition
+  script:
+    - ${GITLAB_MAKE} test_static_build_cmake_linux
+
+static_build_cmake_osx_15:
   stage: test
   tags:
-    - deploy_test
+    - osx_15
   script:
-    - ${GITLAB_MAKE} test_static_docker_build
+    - ${GITLAB_MAKE} test_static_build_cmake_osx
diff --git a/.travis.mk b/.travis.mk
index efc05cf05..58d0c1596 100644
--- a/.travis.mk
+++ b/.travis.mk
@@ -148,8 +148,14 @@ deps_debian_static:
 test_static_build: deps_debian_static
 	CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk test_debian_no_deps
 
-test_static_docker_build:
-	docker build --no-cache --network=host --build-arg RUN_TESTS=ON -f Dockerfile.staticbuild .
+# New static build
+# builddir used in this target - is a default build path from cmake
+# ExternalProject_Add()
+test_static_build_cmake_linux:
+	cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \
+	make -j && ctest -V
+	cd test && /usr/bin/python test-run.py --force \
+		--builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build $(TEST_RUN_EXTRA_PARAMS)
 
 # ###################
 # Static Analysis
@@ -193,15 +199,16 @@ build_osx:
 	cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS}
 	make -j
 
-test_osx_no_deps: build_osx
-	# Limits: Increase the maximum number of open file descriptors on macOS:
-	#   Travis-ci needs the "ulimit -n <value>" call
-	#   Gitlab-ci needs the "launchctl limit maxfiles <value>" call
-	# Also gitlib-ci needs the password to change the limits, while
-	# travis-ci runs under root user. Limit setup must be in the same
-	# call as tests runs call.
-	# Tests: Temporary excluded replication/ suite with some tests
-	#        from other suites by issues #4357 and #4370
+
+# Limits: Increase the maximum number of open file descriptors on macOS:
+#   Travis-ci needs the "ulimit -n <value>" call
+#   Gitlab-ci needs the "launchctl limit maxfiles <value>" call
+# Also gitlib-ci needs the password to change the limits, while
+# travis-ci runs under root user. Limit setup must be in the same
+# call as tests runs call.
+# Tests: Temporary excluded replication/ suite with some tests
+#        from other suites by issues #4357 and #4370
+INIT_TEST_ENV_OSX=\
 	sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
 		launchctl limit maxfiles || : ; \
 		ulimit -n ${MAX_FILES} || : ; \
@@ -210,11 +217,34 @@ test_osx_no_deps: build_osx
 		launchctl limit maxproc || : ; \
 		ulimit -u ${MAX_PROC} || : ; \
 		ulimit -u ; \
-		rm -rf /tmp/tnt ; \
-		cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
+		rm -rf /tmp/tnt
+
+test_osx_no_deps: build_osx
+	${INIT_TEST_ENV_OSX}; \
+	cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS)
 
 test_osx: deps_osx test_osx_no_deps
 
+# Static macOS build
+
+STATIC_OSX_PKGS=autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rb
+base_deps_osx:
+	brew update || echo | /usr/bin/ruby -e \
+		"$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+	brew install --force ${STATIC_OSX_PKGS} || brew upgrade ${STATIC_OSX_PKGS}
+	pip install --force-reinstall -r test-run/requirements.txt
+
+# builddir used in this target - is a default build path from cmake
+# ExternalProject_Add()
+test_static_build_cmake_osx: base_deps_osx
+	cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \
+	make -j && ctest -V
+	${INIT_TEST_ENV_OSX}; \
+	cd test && ./test-run.py --vardir /tmp/tnt \
+		--builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \
+		--force $(TEST_RUN_EXTRA_PARAMS)
+
+
 ###########
 # FreeBSD #
 ###########
diff --git a/Dockerfile.staticbuild b/Dockerfile.staticbuild
deleted file mode 100644
index f67f46f5e..000000000
--- a/Dockerfile.staticbuild
+++ /dev/null
@@ -1,98 +0,0 @@
-FROM centos:7
-
-RUN yum install -y epel-release
-RUN yum install -y yum install https://centos7.iuscommunity.org/ius-release.rpm
-
-RUN set -x \
-    && yum -y install \
-        libstdc++ \
-        libstdc++-static \
-        readline \
-        openssl \
-        lz4 \
-        binutils \
-        ncurses \
-        libgomp \
-        lua \
-        curl \
-        tar \
-        zip \
-        unzip \
-        libunwind \
-        zlib \
-    && yum -y install \
-        perl \
-        gcc-c++ \
-        cmake \
-        lz4-devel \
-        binutils-devel \
-        lua-devel \
-        make \
-        git \
-        autoconf \
-        automake \
-        libtool \
-        wget
-
-RUN yum -y install ncurses-static readline-static zlib-static pcre-static glibc-static
-
-RUN yum -y install python-devel python-pip
-
-RUN set -x && \
-    cd / && \
-    curl -O -L https://www.openssl.org/source/openssl-1.1.1f.tar.gz && \
-    tar -xvf openssl-1.1.1f.tar.gz && \
-    cd openssl-1.1.1f && \
-    ./config --libdir=lib && \
-    make -j && make install
-
-RUN set -x && \
-    cd / && \
-    curl -O -L https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz && \
-    tar -xvf icu4c-62_1-src.tgz && \
-    cd icu/source && \
-    ./configure --with-data-packaging=static --enable-static --enable-shared && \
-    make -j && make install
-
-RUN set -x && \
-    cd / && \
-    curl -O -L http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.3-rc1.tar.gz && \
-    tar -xvf libunwind-1.3-rc1.tar.gz && \
-    cd libunwind-1.3-rc1 && \
-    ./configure --enable-static --enable-shared && \
-    make -j && make install
-
-COPY . /tarantool
-
-WORKDIR /tarantool
-
-RUN set -x && \
-    git submodule init && \
-    git submodule update
-
-# Cleanup for 'build' directory added, because it purges all artefacts
-# produced for curl build, including the old configuration in build/curl
-RUN set -x && \
-    find . -name 'CMakeFiles' -type d -exec rm -rf {} + && \
-    find . -name 'CMakeCache.txt' -type f -delete && \
-    rm -rf build test/small test/luajit-tap
-
-RUN pip install -r /tarantool/test-run/requirements.txt
-
-RUN set -x && \
-    cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-         -DENABLE_DIST:BOOL=ON \
-         -DBUILD_STATIC=ON \
-         -DOPENSSL_USE_STATIC_LIBS=ON \
-         -DOPENSSL_ROOT_DIR=/usr/local \
-         . && \
-    make -j && make install
-
-ARG RUN_TESTS
-RUN if [ -n "${RUN_TESTS}" ]; then \
-        set -x && \
-        cd test && \
-        /usr/bin/python test-run.py --force; \
-    fi
-
-ENTRYPOINT ["/bin/bash"]
diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake
index 5f8b15a63..86fec39e9 100644
--- a/cmake/BuildLibCURL.cmake
+++ b/cmake/BuildLibCURL.cmake
@@ -4,15 +4,21 @@ macro(curl_build)
     set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work)
     set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest)
 
+    message(STATUS "Looking for zlib")
+    find_path(ZLIB_INCLUDE_DIR zlib.h)
+    message(STATUS "Looking for zlib.h - ${ZLIB_INCLUDE_DIR}")
     if (BUILD_STATIC)
         set(LIBZ_LIB_NAME libz.a)
     else()
         set(LIBZ_LIB_NAME z)
     endif()
     find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME})
-    if ("${LIBZ_LIBRARY}" STREQUAL "LIBZ_LIBRARY-NOTFOUND")
+    message(STATUS "Looking for libz - ${LIBZ_LIBRARY}")
+
+    if (NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY)
         message(FATAL_ERROR "Unable to find zlib")
     endif()
+    get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR} DIRECTORY)
 
     # Use the same OpenSSL library for libcurl as is used for
     # tarantool itself.
@@ -88,9 +94,10 @@ macro(curl_build)
 
                 --prefix <INSTALL_DIR>
                 --enable-static
-                --enable-shared
+                --disable-shared
+                --disable-symbol-hiding
 
-                --with-zlib
+                --with-zlib=${FOUND_ZLIB_ROOT_DIR}
                 ${LIBCURL_OPENSSL_OPT}
                 --with-ca-fallback
 
diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake
index 5a1141ebd..db2ae6227 100644
--- a/cmake/compiler.cmake
+++ b/cmake/compiler.cmake
@@ -33,10 +33,6 @@ if (CMAKE_COMPILER_IS_GNUCC)
            Your GCC version is ${CMAKE_CXX_COMPILER_VERSION}, please update
                    ")
        endif()
-else()
-     if (BUILD_STATIC)
-           message(FATAL_ERROR "Static build is supported for GCC only")
-     endif()
 endif()
 
 #
@@ -120,10 +116,23 @@ set (CMAKE_CXX_FLAGS_RELWITHDEBINFO
 
 unset(CC_DEBUG_OPT)
 
+message(STATUS "Looking for libunwind.h")
+find_path(UNWIND_INCLUDE_DIR libunwind.h)
+message(STATUS "Looking for libunwind.h - ${UNWIND_INCLUDE_DIR}")
+
+if (UNWIND_INCLUDE_DIR)
+    include_directories(${UNWIND_INCLUDE_DIR})
+endif()
+
+set(CMAKE_REQUIRED_INCLUDES ${UNWIND_INCLUDE_DIR})
 check_include_file(libunwind.h HAVE_LIBUNWIND_H)
-if(BUILD_STATIC)
+set(CMAKE_REQUIRED_INCLUDES "")
+
+if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
     set(UNWIND_LIB_NAME libunwind.a)
 else()
+    # libunwind can't be compiled on macOS.
+    # But there exists libunwind.dylib as a part of MacOSSDK
     set(UNWIND_LIB_NAME unwind)
 endif()
 find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME})
@@ -185,7 +194,10 @@ if (ENABLE_BACKTRACE)
     find_package_message(UNWIND_LIBRARIES "Found unwind" "${UNWIND_LIBRARIES}")
 endif()
 
-if(BUILD_STATIC)
+# On macOS there is no '-static-libstdc++' flag and it's use will
+# raise following error:
+# error: argument unused during compilation: '-static-libstdc++'
+if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
     # Static linking for c++ routines
     add_compile_flags("C;CXX" "-static-libstdc++")
 endif()
diff --git a/cmake/os.cmake b/cmake/os.cmake
index 905be61df..276a79b42 100644
--- a/cmake/os.cmake
+++ b/cmake/os.cmake
@@ -107,7 +107,10 @@ elseif (${CMAKE_SYSTEM_NAME} STREQUAL "Darwin")
 
     # Latest versions of Homebrew wont 'link --force' for libraries, that were
     # preinstalled in system. So we'll use this dirty hack
-    find_program(HOMEBREW_EXECUTABLE brew)
+
+    if (NOT BUILD_STATIC)
+        find_program(HOMEBREW_EXECUTABLE brew)
+    endif()
     if(EXISTS ${HOMEBREW_EXECUTABLE})
         execute_process(COMMAND ${HOMEBREW_EXECUTABLE} --prefix
                         OUTPUT_VARIABLE HOMEBREW_PREFIX
diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt
new file mode 100644
index 000000000..90029fdd8
--- /dev/null
+++ b/static-build/CMakeLists.txt
@@ -0,0 +1,311 @@
+cmake_minimum_required(VERSION 2.8)
+
+# Detect system compilers for further configuring dependencies to be
+# builded with these compilers. This is used to build tarantool and
+# it's dependencies by usign one compiler system (for example libicu
+# by default uses clang if it exists when others uses gcc/g++ on
+# linux machine).
+project(tarantool-static C CXX)
+
+include(ExternalProject)
+set(OPENSSL_VERSION 1.1.1f)
+set(ZLIB_VERSION 1.2.11)
+set(NCURSES_VERSION 6.2)
+set(READLINE_VERSION 8.0)
+set(UNWIND_VERSION 1.3-rc1)
+
+# Pass -isysroot=<SDK_PATH> option on Mac OS to a preprocessor and a C
+# compiler to find header files installed with an SDK.
+#
+# The idea is to set these (DEPENDENCY_*) variables to corresponding
+# environment variables at each depenency configure script.
+#
+# Note: Passing of CPPFLAGS / CFLAGS explicitly discards using of
+# corresponsing environment variables. So pass empty LDFLAGS to discard
+# using of corresponding environment variable. It is possible that a
+# linker flag assumes that some compilation flag is set. We don't pass
+# CFLAGS from environment, so we should not do it for LDFLAGS too.
+set(DEPENDENCY_CFLAGS "")
+set(DEPENDENCY_CPPFLAGS "")
+set(DEPENDENCY_LDFLAGS)
+if (APPLE)
+    set(DEPENDENCY_CFLAGS   "${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}")
+    set(DEPENDENCY_CPPFLAGS "${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}")
+endif()
+
+# Install all libraries required by tarantool at current build dir
+
+#
+# OpenSSL
+#
+ExternalProject_Add(openssl
+    URL https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz
+    CONFIGURE_COMMAND <SOURCE_DIR>/config
+        CC=${CMAKE_C_COMPILER}
+        CXX=${CMAKE_CXX_COMPILER}
+        CFLAGS=${DEPENDENCY_CFLAGS}
+        CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+        LDFLAGS=${DEPENDENCY_LDFLAGS}
+
+        --prefix=<INSTALL_DIR>
+        --libdir=lib
+        no-shared
+    INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install_sw
+)
+
+#
+# ICU
+#
+ExternalProject_Add(icu
+    URL https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz
+    # By default libicu is built by using clang/clang++ compiler (if it
+    # exists). Here is a link for detecting compilers at libicu configure
+    # script: https://github.com/unicode-org/icu/blob/7c7b8bd5702310b972f888299169bc3cc88bf0a6/icu4c/source/configure.ac#L135
+    # This will cause the problem on linux machine: tarantool is built
+    # with gcc/g++ and libicu is built with clang/clang++ (if it exists)
+    # so at linking stage `rellocation` errors will occur. To solve this,
+    # we can set CC/CXX to CMAKE_C_COMPILER/CMAKE_CXX_COMPILER variables
+    # manually which are detected above (by cmake `project()` command)
+    CONFIGURE_COMMAND <SOURCE_DIR>/source/configure
+        CC=${CMAKE_C_COMPILER}
+        CXX=${CMAKE_CXX_COMPILER}
+        CFLAGS=${DEPENDENCY_CFLAGS}
+        CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+        LDFLAGS=${DEPENDENCY_LDFLAGS}
+
+        --with-data-packaging=static
+        --prefix=<INSTALL_DIR>
+        --disable-shared
+        --enable-static
+)
+
+#
+# ZLIB
+#
+ExternalProject_Add(zlib
+    URL https://zlib.net/zlib-${ZLIB_VERSION}.tar.gz
+    CONFIGURE_COMMAND env
+        CC=${CMAKE_C_COMPILER}
+        CFLAGS=${DEPENDENCY_CFLAGS}
+        CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+        LDFLAGS=${DEPENDENCY_LDFLAGS}
+        <SOURCE_DIR>/configure
+        --prefix=<INSTALL_DIR>
+        --static
+)
+
+#
+# Ncurses
+#
+ExternalProject_Add(ncurses
+    URL https://ftp.gnu.org/gnu/ncurses/ncurses-${NCURSES_VERSION}.tar.gz
+    CONFIGURE_COMMAND <SOURCE_DIR>/configure
+        CC=${CMAKE_C_COMPILER}
+        CXX=${CMAKE_CXX_COMPILER}
+        CFLAGS=${DEPENDENCY_CFLAGS}
+        CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+        LDFLAGS=${DEPENDENCY_LDFLAGS}
+
+        --prefix=<INSTALL_DIR>
+
+        # This flag enables creation of libcurses.a as a symlink to libncurses.a
+        # and disables subdir creation `ncurses` at <install_dir>/include. It is
+        # necessary for correct work of FindCurses.cmake module (this module is
+        # builtin at cmake package) which used in cmake/FindReadline.cmake
+        --enable-overwrite
+
+        # enable building libtinfo to prevent linking with libtinfo from system
+        # directories
+        --with-termlib
+
+        # set search paths for terminfo db
+        --with-terminfo-dirs=/lib/terminfo:/usr/share/terminfo:/etc/terminfo
+
+        # disable install created terminfo db, use db from system
+        --disable-db-install
+        --without-progs
+        --without-manpages
+)
+
+#
+# ReadLine
+#
+ExternalProject_Add(readline
+    URL https://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
+    CONFIGURE_COMMAND <SOURCE_DIR>/configure
+        CC=${CMAKE_C_COMPILER}
+        CFLAGS=${DEPENDENCY_CFLAGS}
+        CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+        LDFLAGS=${DEPENDENCY_LDFLAGS}
+
+        --prefix=<INSTALL_DIR>
+        --disable-shared
+)
+
+#
+# ICONV
+#
+if (APPLE)
+    ExternalProject_Add(iconv
+        URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
+        CONFIGURE_COMMAND <SOURCE_DIR>/configure
+            CC=${CMAKE_C_COMPILER}
+            CFLAGS=${DEPENDENCY_CFLAGS}
+            CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+            LDFLAGS=${DEPENDENCY_LDFLAGS}
+
+            --prefix=<INSTALL_DIR>
+            --disable-shared
+            --enable-static
+            --with-gnu-ld
+        STEP_TARGETS download
+    )
+else()
+    # In linux iconv is embedded into glibc
+    # So we find system header and copy it locally
+    find_path(ICONV_INCLUDE_DIR iconv.h)
+    if(NOT ICONV_INCLUDE_DIR)
+        message(FATAL_ERROR "iconv include header not found")
+    endif()
+
+    set(ICONV_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix")
+
+    add_custom_command(
+        OUTPUT "${ICONV_INSTALL_PREFIX}/include/iconv.h"
+        COMMAND ${CMAKE_COMMAND} -E make_directory
+            "${ICONV_INSTALL_PREFIX}/include"
+        COMMAND ${CMAKE_COMMAND} -E copy
+            "${ICONV_INCLUDE_DIR}/iconv.h"
+            "${ICONV_INSTALL_PREFIX}/include/iconv.h"
+    )
+    add_custom_target(iconv
+        DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h"
+    )
+    # This is a hack for further getting install directory of library
+    # by ExternalProject_Get_Property
+    set_target_properties(iconv
+        PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX}
+    )
+endif()
+
+#
+# Unwind
+#
+if (APPLE)
+    # On macOS libunwind is a part of MacOSX.sdk
+    # So we need to find library and header and
+    # copy it locally
+    find_path(UNWIND_INCLUDE_DIR libunwind.h)
+    find_library(UNWIND_LIBRARY libunwind.dylib
+        PATH_SUFFIXES system
+    )
+
+    set(UNWIND_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/unwind-prefix")
+
+    set(UNWIND_DEPENDENCIES)
+
+    if (UNWIND_INCLUDE_DIR AND UNWIND_LIBRARY)
+        add_custom_command(
+            OUTPUT "${UNWIND_INSTALL_PREFIX}/include/unwind.h"
+            OUTPUT "${UNWIND_INSTALL_PREFIX}/include/libunwind.h"
+            COMMAND ${CMAKE_COMMAND} -E make_directory
+                "${UNWIND_INSTALL_PREFIX}/include"
+            COMMAND ${CMAKE_COMMAND} -E copy
+                "${UNWIND_INCLUDE_DIR}/libunwind.h"
+                "${UNWIND_INCLUDE_DIR}/unwind.h"
+                "${UNWIND_INSTALL_PREFIX}/include/"
+        )
+        add_custom_command(
+            OUTPUT "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib"
+            COMMAND ${CMAKE_COMMAND} -E make_directory
+                "${UNWIND_INSTALL_PREFIX}/lib"
+            COMMAND ${CMAKE_COMMAND} -E copy
+                "${UNWIND_LIBRARY}"
+                "${UNWIND_INSTALL_PREFIX}/lib/"
+        )
+        set(UNWIND_DEPENDENCIES
+            ${UNWIND_DEPENDENCIES}
+            "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib"
+            "${UNWIND_INSTALL_PREFIX}/include/libunwind.h"
+        )
+    else()
+        message(STATUS "Unwind not found")
+    endif()
+
+    add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES})
+    # This is a hack for further getting install directory of library
+    # by ExternalProject_Get_Property
+    set_target_properties(unwind
+        PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX}
+    )
+else()
+    ExternalProject_Add(unwind
+        URL https://download.savannah.nongnu.org/releases/libunwind/libunwind-${UNWIND_VERSION}.tar.gz
+        CONFIGURE_COMMAND <SOURCE_DIR>/configure
+            CC=${CMAKE_C_COMPILER}
+            CXX=${CMAKE_CXX_COMPILER}
+            CFLAGS=${DEPENDENCY_CFLAGS}
+            CPPFLAGS=${DEPENDENCY_CPPFLAGS}
+            LDFLAGS=${DEPENDENCY_LDFLAGS}
+            --prefix=<INSTALL_DIR>
+            --disable-shared
+            --enable-static
+            --disable-minidebuginfo # to prevent linking with liblzma
+        STEP_TARGETS download
+    )
+endif()
+
+# Get install directories of builded libraries for building
+# tarantool with custon CMAKE_PREFIX_PATH
+foreach(PROJ openssl icu zlib ncurses readline iconv unwind)
+    ExternalProject_Get_Property(${PROJ} install_dir)
+    set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir})
+    set(TARANTOOL_DEPENDS ${PROJ} ${TARANTOOL_DEPENDS})
+    message(STATUS "Add external project ${PROJ} in ${install_dir}")
+endforeach()
+
+ExternalProject_Add(tarantool
+    DEPENDS ${TARANTOOL_DEPENDS}
+    SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..
+    LIST_SEPARATOR :
+    CMAKE_ARGS
+        # Override LOCALSTATEDIR to avoid cmake "special" cases:
+        # https://cmake.org/cmake/help/v3.4/module/GNUInstallDirs.html#special-cases
+        -DCMAKE_INSTALL_LOCALSTATEDIR=<INSTALL_DIR>/var
+        -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
+        -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
+        -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE
+        -DOPENSSL_USE_STATIC_LIBS=TRUE
+        -DBUILD_STATIC=TRUE
+        -DENABLE_DIST=TRUE
+        -DENABLE_BACKTRACE=TRUE
+        -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
+        -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
+        ${CMAKE_TARANTOOL_ARGS}
+    STEP_TARGETS build
+    BUILD_COMMAND $(MAKE)
+)
+
+enable_testing()
+ExternalProject_Get_Property(tarantool binary_dir)
+SET(TARANTOOL_BINARY_DIR ${binary_dir})
+
+add_test(
+    NAME check-dependencies
+    COMMAND ${CMAKE_COMMAND}
+        -D FILE=${TARANTOOL_BINARY_DIR}/src/tarantool
+        -P CheckDependencies.cmake
+    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
+)
+
+add_test(
+    NAME check-exports
+    COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool
+            ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/exports.test.lua
+)
+
+add_test(
+    NAME check-traceback
+    COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool
+            ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/traceback.test.lua
+)
diff --git a/static-build/README.md b/static-build/README.md
new file mode 100644
index 000000000..2219b5026
--- /dev/null
+++ b/static-build/README.md
@@ -0,0 +1,90 @@
+# Tarantool static build tooling
+
+These files help to prepare environment for building Tarantool
+statically. And builds it.
+
+## Prerequisites
+
+Please install following tools and libraries that will
+be necessary for building and testing:
+* git
+* A C/C++ compiler.
+
+  Ordinarily, this is gcc and g++ version 4.6 or later. On Mac OS X, this
+  is Clang version 3.2+.
+* cmake
+* autoconf automake libtool
+* make
+* Python and modules.
+
+  Python interpreter is not necessary for building Tarantool itself, unless you
+  intend to use the “Run the test suite". For all platforms, this is python
+  version 2.7+ (but not 3.x). You need the following Python modules:
+  * pyyaml
+  * argparse
+  * msgpack-python
+  * gevent
+  * six
+
+### Here is an examples for your OS:
+
+CentOS:
+
+```bash
+yum install -y \
+    git perl gcc cmake make gcc-c++ libstdc++-static autoconf automake libtool \
+    python-msgpack python-yaml python-argparse python-six python-gevent
+```
+
+Ubuntu/Debian:
+
+```bash
+apt-get install -y \
+    build-essential cmake make coreutils autoconf automake libtool sed \
+    python python-pip python-setuptools python-dev \
+    python-msgpack python-yaml python-argparse python-six python-gevent
+```
+
+MacOS:
+
+Before you start please install default Xcode Tools by Apple:
+
+```bash
+sudo xcode-select --install
+sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
+```
+
+Install brew using command from
+[Homebrew repository instructions](https://github.com/Homebrew/inst)
+
+After that run next script:
+
+```bash
+brew install autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rbs
+pip install --force-reinstall -r test-run/requirements.txt
+```
+
+## Usage
+
+```bash
+cmake .
+make -j
+ctest -V
+```
+
+## Customize your build
+
+If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` variable
+
+### Usage
+
+There are three types of `CMAKE_BUILD_TYPE`:
+* Debug - default
+* Release
+* RelWithDebInfo
+
+And you want to build tarantool with RelWithDebInfo:
+
+```bash
+cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" .
+```
diff --git a/static-build/test/CheckDependencies.cmake b/static-build/test/CheckDependencies.cmake
new file mode 100644
index 000000000..49e91e7fa
--- /dev/null
+++ b/static-build/test/CheckDependencies.cmake
@@ -0,0 +1,43 @@
+## This is a cmake-based test, it checks that tarantool static binary
+# has no dependencies except allowed ones.
+
+include(GetPrerequisites)
+if(NOT FILE)
+    message(FATAL_ERROR "Usage: "
+        "${CMAKE_COMMAND} -DFILE=<FILENAME> -P CheckDependencies.cmake")
+elseif(NOT EXISTS ${FILE})
+    message(FATAL_ERROR "${FILE}: No such file")
+endif()
+
+get_prerequisites(${FILE} DEPENDENCIES 0 0 "" "")
+
+if (APPLE)
+    set(ALLOWLIST
+        libSystem
+        CoreFoundation
+        libc++
+    )
+elseif(UNIX)
+    set(ALLOWLIST
+        libdl
+        librt
+        libc
+        libm
+        libgcc_s
+        libpthread
+    )
+else()
+    message(FATAL_ERROR "Unknown platform")
+endif()
+
+foreach(DEPENDENCY_FILE ${DEPENDENCIES})
+    message("Dependency: ${DEPENDENCY_FILE}")
+endforeach()
+
+foreach(DEPENDENCY_FILE ${DEPENDENCIES})
+    get_filename_component(libname ${DEPENDENCY_FILE} NAME_WE)
+    list (FIND ALLOWLIST ${libname} _index)
+    if (_index EQUAL -1)
+        message(FATAL_ERROR "Blocklisted dependency: ${DEPENDENCY_FILE}")
+    endif()
+endforeach()
diff --git a/static-build/test/static-build/exports.test.lua b/static-build/test/static-build/exports.test.lua
new file mode 100755
index 000000000..de54973d8
--- /dev/null
+++ b/static-build/test/static-build/exports.test.lua
@@ -0,0 +1,142 @@
+#!/usr/bin/env tarantool
+
+local tap = require('tap')
+local ffi = require('ffi')
+ffi.cdef([[
+    void *dlsym(void *handle, const char *symbol);
+]])
+
+local test = tap.test('exports')
+
+-- See `man 3 dlsym`:
+-- RTLD_DEFAULT
+--   Find the first occurrence of the desired symbol using the default
+--   shared object search order. The search will include global symbols
+--   in the executable and its dependencies, as well as symbols in shared
+--   objects that were dynamically loaded with the RTLD_GLOBAL flag.
+local RTLD_DEFAULT = ffi.cast("void *", jit.os == "OSX" and -2LL or 0LL)
+
+local function check_symbol(sym)
+    test:ok(ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, ('Symbol %q found'):format(sym))
+end
+
+local check_symbols = {
+    -- FFI
+
+    'guava',
+    'base64_decode',
+    'base64_encode',
+    'SHA1internal',
+    'random_bytes',
+    'fiber_time',
+    'ibuf_create',
+    'ibuf_destroy',
+    'port_destroy',
+    'csv_create',
+    'csv_destroy',
+    'title_get',
+    'title_update',
+    'tnt_iconv',
+    'tnt_iconv_open',
+    'tnt_iconv_close',
+    'exception_get_int',
+    'exception_get_string',
+
+    'tarantool_lua_ibuf',
+    'uuid_nil',
+    'tt_uuid_create',
+    'tt_uuid_str',
+    'tt_uuid_is_equal',
+    'tt_uuid_is_nil',
+    'tt_uuid_bswap',
+    'tt_uuid_from_string',
+    'log_level',
+    'log_format',
+    'uri_parse',
+    'uri_format',
+    'PMurHash32',
+    'PMurHash32_Process',
+    'PMurHash32_Result',
+    'crc32_calc',
+    'mp_encode_double',
+    'mp_encode_float',
+    'mp_encode_decimal',
+    'mp_decode_double',
+    'mp_decode_float',
+    'mp_decode_extl',
+    'mp_sizeof_decimal',
+    'decimal_unpack',
+
+    'log_type',
+    'say_set_log_level',
+    'say_logrotate',
+    'say_set_log_format',
+    'tarantool_uptime',
+    'tarantool_exit',
+    'log_pid',
+    'space_by_id',
+    'space_run_triggers',
+    'space_bsize',
+    'box_schema_version',
+
+    'crypto_EVP_MD_CTX_new',
+    'crypto_EVP_MD_CTX_free',
+    'crypto_HMAC_CTX_new',
+    'crypto_HMAC_CTX_free',
+    'crypto_stream_new',
+    'crypto_stream_begin',
+    'crypto_stream_append',
+    'crypto_stream_commit',
+    'crypto_stream_delete',
+
+    -- Module API
+
+    '_say',
+    'swim_cfg',
+    'swim_quit',
+    'fiber_new',
+    'fiber_cancel',
+    'coio_wait',
+    'coio_close',
+    'coio_call',
+    'coio_getaddrinfo',
+    'luaT_call',
+    'box_txn',
+    'box_select',
+    'clock_realtime',
+    'string_strip_helper',
+
+    -- Lua / LuaJIT
+
+    'lua_newstate',
+    'lua_close',
+    'luaL_loadstring',
+    'luaJIT_profile_start',
+    'luaJIT_profile_stop',
+    'luaJIT_profile_dumpstack',
+
+    'ERR_error_string',
+    'ERR_get_error',
+
+    'EVP_get_digestbyname',
+    'EVP_get_cipherbyname',
+    'EVP_CIPHER_CTX_new',
+    'EVP_CIPHER_CTX_free',
+    'EVP_CIPHER_block_size',
+    'HMAC_Init_ex',
+    'HMAC_Update',
+    'HMAC_Final',
+
+    'ZSTD_compress',
+    'ZSTD_decompress',
+    'ZSTD_free',
+    'ZSTD_malloc',
+    'ZSTD_versionString',
+}
+
+test:plan(#check_symbols)
+for _, sym in ipairs(check_symbols) do
+    check_symbol(sym)
+end
+
+os.exit(test:check() and 0 or 1)
diff --git a/static-build/test/static-build/traceback.test.lua b/static-build/test/static-build/traceback.test.lua
new file mode 100755
index 000000000..32e1ed8b5
--- /dev/null
+++ b/static-build/test/static-build/traceback.test.lua
@@ -0,0 +1,15 @@
+#!/usr/bin/env tarantool
+
+local tap = require('tap')
+local fiber = require('fiber')
+
+local test = tap.test('traceback')
+test:plan(1)
+
+local info = fiber.info()[fiber.id()]
+test:ok(info.backtrace ~= nil, 'fiber.info() has backtrace')
+for _, l in pairs(info.backtrace or {}) do
+    test:diag('%s: %s', next(l))
+end
+
+os.exit(test:check() and 0 or 1)
-- 
2.26.2

^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-09-15 14:25 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-09 15:45 [Tarantool-patches] [PATCH v3] build: refactor static build process HustonMmmavr
2020-09-10 10:00 ` Alexandr Barulev
2020-09-14 16:32 ` Igor Munkin
2020-09-14 18:31   ` Alexandr Barulev
2020-09-14 18:42     ` Igor Munkin
2020-09-15 13:53 ` Kirill Yukhin
2020-09-15 14:14   ` Igor Munkin

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox