[Tarantool-patches] [PATCH v3] build: refactor static build process
Alexandr Barulev
huston.mavr at gmail.com
Thu Sep 10 13:00:05 MSK 2020
Hello!
Here is a new commit message(as discussed at telegram chat):
Author: HustonMmmavr <huston.mavr at gmail.com>
Date: Thu Sep 10 12:43:20 2020 +0300
build: refactor static build process
Refactored static build process to use static-build/CMakeLists.txt
instead of Dockerfile.staticbuild (this allows to support static
build on macOS). Following third-party dependencies for static build
are installed via cmake `ExternalProject_Add`:
- OpenSSL
- Zlib
- Ncurses
- Readline
- Unwind
- ICU
* Added support static build for macOS
* Fixed `CONFIGURE_COMMAND` while building bundled libcurl for static
build at file cmake/BuildLibCURL.cmake:
- disable building shared libcurl libraries (by setting
`--disable-shared` option)
- disable hiding libcurl symbols (by setting
`--disable-symbol-hiding` option)
- prevent linking libcurl with system libz (by setting
`--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option)
* Removed Dockerfile.staticbuild
* Added new gitlab.ci jobs to test new style static build:
- static_build_cmake_linux
- static_build_cmake_osx_15
* Removed static_docker_build gitlab.ci job
Closes #5095
Co-authored-by: Yaroslav Dynnikov <yaroslav.dynnikov at gmail.com>
ср, 9 сент. 2020 г. в 18:45, HustonMmmavr <huston.mavr at gmail.com>:
> From: Yaroslav Dynnikov <yaroslav.dynnikov at gmail.com>
>
> Refactored static build process to use static-build/CMakeLists.txt
> instead of Dockerfile.staticbuild (this allows to support static
> build on macOS). Following third-party dependencies for static build
> are installed via cmake `ExternalProject_Add`:
> - OpenSSL
> - Zlib
> - Ncurses
> - Readline
> - Unwind
> - ICU
>
> * Added support static build for macOS
> * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for static
> build at file cmake/BuildLibCURL.cmake:
> - disable building shared libcurl libraries (by setting
> `--disable-shared` option)
> - disable hiding libcurl symbols (by setting
> `--disable-symbol-hiding` option)
> - prevent linking libcurl with system libz (by setting
> `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option)
> * Removed Dockerfile.staticbuild
> * Added new gitlab.ci jobs to test new style static build:
> - static_build_cmake_linux
> - static_build_cmake_osx_15
> * Removed static_docker_build gitlab.ci job
>
> Closes #5095
> ---
>
> Branch:
> https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build
> Issue: https://github.com/tarantool/tarantool/issues/5095
>
> Changes in v3:
> 1. Fixed typos at commit message
> 2. Fixed commentaries at .travis.mk
> 3. Fixed finding libz library at cmake/BuildLibCurl.cmake as at master
> branch
> 4. Instead of using static-build/.gitignore use the root .gitignore file
> 5. Refactored static-build/CMakeLists.txt:
> * Project name is set to tarantool-static
> * Instead of explicitly finding of c/c++ compilers current version uses
> cmake `project(tarantool-static C CXX)` command. This change requires
> to manually set CFLAGS/CPPFLAGS/LDFLAGS for all static-build
> dependencies (because building dependencies at macOS requires path
> to macOS SDK) the idea was taken from cmake/BuildLibCurl.cmake
> * Added commentaries about problem with libicu
> * Removed unused TEST_COMMAND at ExternalProject_Add(zlib ...) also
> removed unnecessary # STEP_TARGETS at ExternalProject_Add(readline
> ...)
> * Deleted doubled whitespaces
> 6. Fixed static-build/README.md:
> * Added list of required tools
> * Added example for ubuntu/debian
> * Fixed indentation and typos
> 7. Refactored tests:
> * Got rid of test-run
> * Deleted doubled whitespaces at exports.test.lua
> * Fixed defining of RTLD_DEFAULT
>
>
> .gitignore | 8 +
> .gitlab-ci.yml | 11 +-
> .travis.mk | 56 +++-
> Dockerfile.staticbuild | 98 ------
> cmake/BuildLibCURL.cmake | 13 +-
> cmake/compiler.cmake | 24 +-
> cmake/os.cmake | 5 +-
> static-build/CMakeLists.txt | 311 ++++++++++++++++++
> static-build/README.md | 90 +++++
> static-build/test/CheckDependencies.cmake | 43 +++
> .../test/static-build/exports.test.lua | 142 ++++++++
> .../test/static-build/traceback.test.lua | 15 +
> 12 files changed, 692 insertions(+), 124 deletions(-)
> delete mode 100644 Dockerfile.staticbuild
> create mode 100644 static-build/CMakeLists.txt
> create mode 100644 static-build/README.md
> create mode 100644 static-build/test/CheckDependencies.cmake
> create mode 100755 static-build/test/static-build/exports.test.lua
> create mode 100755 static-build/test/static-build/traceback.test.lua
>
> diff --git a/.gitignore b/.gitignore
> index a42c7db35..c6e261e18 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -151,3 +151,11 @@ src/box/sql/parse.h
> src/box/sql/parse.c
> src/box/sql/opcodes.h
> src/box/sql/opcodes.c
> +
> +# static-build
> +static-build/*-prefix
> +static-build/Makefile
> +static-build/build
> +
> +# macOS files
> +.DS_Store
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 0ead08711..c9aef3dc7 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -534,9 +534,14 @@ static_build:
> script:
> - ${GITLAB_MAKE} test_static_build
>
> -static_docker_build:
> +static_build_cmake_linux:
> + <<: *docker_test_definition
> + script:
> + - ${GITLAB_MAKE} test_static_build_cmake_linux
> +
> +static_build_cmake_osx_15:
> stage: test
> tags:
> - - deploy_test
> + - osx_15
> script:
> - - ${GITLAB_MAKE} test_static_docker_build
> + - ${GITLAB_MAKE} test_static_build_cmake_osx
> diff --git a/.travis.mk b/.travis.mk
> index efc05cf05..58d0c1596 100644
> --- a/.travis.mk
> +++ b/.travis.mk
> @@ -148,8 +148,14 @@ deps_debian_static:
> test_static_build: deps_debian_static
> CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk
> test_debian_no_deps
>
> -test_static_docker_build:
> - docker build --no-cache --network=host --build-arg RUN_TESTS=ON -f
> Dockerfile.staticbuild .
> +# New static build
> +# builddir used in this target - is a default build path from cmake
> +# ExternalProject_Add()
> +test_static_build_cmake_linux:
> + cd static-build && cmake
> -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON"
> . && \
> + make -j && ctest -V
> + cd test && /usr/bin/python test-run.py --force \
> + --builddir
> ${PWD}/static-build/tarantool-prefix/src/tarantool-build
> $(TEST_RUN_EXTRA_PARAMS)
>
> # ###################
> # Static Analysis
> @@ -193,15 +199,16 @@ build_osx:
> cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON
> ${CMAKE_EXTRA_PARAMS}
> make -j
>
> -test_osx_no_deps: build_osx
> - # Limits: Increase the maximum number of open file descriptors on
> macOS:
> - # Travis-ci needs the "ulimit -n <value>" call
> - # Gitlab-ci needs the "launchctl limit maxfiles <value>" call
> - # Also gitlib-ci needs the password to change the limits, while
> - # travis-ci runs under root user. Limit setup must be in the same
> - # call as tests runs call.
> - # Tests: Temporary excluded replication/ suite with some tests
> - # from other suites by issues #4357 and #4370
> +
> +# Limits: Increase the maximum number of open file descriptors on macOS:
> +# Travis-ci needs the "ulimit -n <value>" call
> +# Gitlab-ci needs the "launchctl limit maxfiles <value>" call
> +# Also gitlib-ci needs the password to change the limits, while
> +# travis-ci runs under root user. Limit setup must be in the same
> +# call as tests runs call.
> +# Tests: Temporary excluded replication/ suite with some tests
> +# from other suites by issues #4357 and #4370
> +INIT_TEST_ENV_OSX=\
> sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \
> launchctl limit maxfiles || : ; \
> ulimit -n ${MAX_FILES} || : ; \
> @@ -210,11 +217,34 @@ test_osx_no_deps: build_osx
> launchctl limit maxproc || : ; \
> ulimit -u ${MAX_PROC} || : ; \
> ulimit -u ; \
> - rm -rf /tmp/tnt ; \
> - cd test && ./test-run.py --vardir /tmp/tnt --force
> $(TEST_RUN_EXTRA_PARAMS)
> + rm -rf /tmp/tnt
> +
> +test_osx_no_deps: build_osx
> + ${INIT_TEST_ENV_OSX}; \
> + cd test && ./test-run.py --vardir /tmp/tnt --force
> $(TEST_RUN_EXTRA_PARAMS)
>
> test_osx: deps_osx test_osx_no_deps
>
> +# Static macOS build
> +
> +STATIC_OSX_PKGS=autoconf automake libtool cmake
> file://$${PWD}/tools/brew_taps/tntpython2.rb
> +base_deps_osx:
> + brew update || echo | /usr/bin/ruby -e \
> + "$$(curl -fsSL
> https://raw.githubusercontent.com/Homebrew/install/master/install)"
> + brew install --force ${STATIC_OSX_PKGS} || brew upgrade
> ${STATIC_OSX_PKGS}
> + pip install --force-reinstall -r test-run/requirements.txt
> +
> +# builddir used in this target - is a default build path from cmake
> +# ExternalProject_Add()
> +test_static_build_cmake_osx: base_deps_osx
> + cd static-build && cmake
> -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON"
> . && \
> + make -j && ctest -V
> + ${INIT_TEST_ENV_OSX}; \
> + cd test && ./test-run.py --vardir /tmp/tnt \
> + --builddir
> ${PWD}/static-build/tarantool-prefix/src/tarantool-build \
> + --force $(TEST_RUN_EXTRA_PARAMS)
> +
> +
> ###########
> # FreeBSD #
> ###########
> diff --git a/Dockerfile.staticbuild b/Dockerfile.staticbuild
> deleted file mode 100644
> index f67f46f5e..000000000
> --- a/Dockerfile.staticbuild
> +++ /dev/null
> @@ -1,98 +0,0 @@
> -FROM centos:7
> -
> -RUN yum install -y epel-release
> -RUN yum install -y yum install
> https://centos7.iuscommunity.org/ius-release.rpm
> -
> -RUN set -x \
> - && yum -y install \
> - libstdc++ \
> - libstdc++-static \
> - readline \
> - openssl \
> - lz4 \
> - binutils \
> - ncurses \
> - libgomp \
> - lua \
> - curl \
> - tar \
> - zip \
> - unzip \
> - libunwind \
> - zlib \
> - && yum -y install \
> - perl \
> - gcc-c++ \
> - cmake \
> - lz4-devel \
> - binutils-devel \
> - lua-devel \
> - make \
> - git \
> - autoconf \
> - automake \
> - libtool \
> - wget
> -
> -RUN yum -y install ncurses-static readline-static zlib-static pcre-static
> glibc-static
> -
> -RUN yum -y install python-devel python-pip
> -
> -RUN set -x && \
> - cd / && \
> - curl -O -L https://www.openssl.org/source/openssl-1.1.1f.tar.gz && \
> - tar -xvf openssl-1.1.1f.tar.gz && \
> - cd openssl-1.1.1f && \
> - ./config --libdir=lib && \
> - make -j && make install
> -
> -RUN set -x && \
> - cd / && \
> - curl -O -L
> https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz
> && \
> - tar -xvf icu4c-62_1-src.tgz && \
> - cd icu/source && \
> - ./configure --with-data-packaging=static --enable-static
> --enable-shared && \
> - make -j && make install
> -
> -RUN set -x && \
> - cd / && \
> - curl -O -L
> http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.3-rc1.tar.gz
> && \
> - tar -xvf libunwind-1.3-rc1.tar.gz && \
> - cd libunwind-1.3-rc1 && \
> - ./configure --enable-static --enable-shared && \
> - make -j && make install
> -
> -COPY . /tarantool
> -
> -WORKDIR /tarantool
> -
> -RUN set -x && \
> - git submodule init && \
> - git submodule update
> -
> -# Cleanup for 'build' directory added, because it purges all artefacts
> -# produced for curl build, including the old configuration in build/curl
> -RUN set -x && \
> - find . -name 'CMakeFiles' -type d -exec rm -rf {} + && \
> - find . -name 'CMakeCache.txt' -type f -delete && \
> - rm -rf build test/small test/luajit-tap
> -
> -RUN pip install -r /tarantool/test-run/requirements.txt
> -
> -RUN set -x && \
> - cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \
> - -DENABLE_DIST:BOOL=ON \
> - -DBUILD_STATIC=ON \
> - -DOPENSSL_USE_STATIC_LIBS=ON \
> - -DOPENSSL_ROOT_DIR=/usr/local \
> - . && \
> - make -j && make install
> -
> -ARG RUN_TESTS
> -RUN if [ -n "${RUN_TESTS}" ]; then \
> - set -x && \
> - cd test && \
> - /usr/bin/python test-run.py --force; \
> - fi
> -
> -ENTRYPOINT ["/bin/bash"]
> diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake
> index 5f8b15a63..86fec39e9 100644
> --- a/cmake/BuildLibCURL.cmake
> +++ b/cmake/BuildLibCURL.cmake
> @@ -4,15 +4,21 @@ macro(curl_build)
> set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work)
> set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest)
>
> + message(STATUS "Looking for zlib")
> + find_path(ZLIB_INCLUDE_DIR zlib.h)
> + message(STATUS "Looking for zlib.h - ${ZLIB_INCLUDE_DIR}")
> if (BUILD_STATIC)
> set(LIBZ_LIB_NAME libz.a)
> else()
> set(LIBZ_LIB_NAME z)
> endif()
> find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME})
> - if ("${LIBZ_LIBRARY}" STREQUAL "LIBZ_LIBRARY-NOTFOUND")
> + message(STATUS "Looking for libz - ${LIBZ_LIBRARY}")
> +
> + if (NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY)
> message(FATAL_ERROR "Unable to find zlib")
> endif()
> + get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR}
> DIRECTORY)
>
> # Use the same OpenSSL library for libcurl as is used for
> # tarantool itself.
> @@ -88,9 +94,10 @@ macro(curl_build)
>
> --prefix <INSTALL_DIR>
> --enable-static
> - --enable-shared
> + --disable-shared
> + --disable-symbol-hiding
>
> - --with-zlib
> + --with-zlib=${FOUND_ZLIB_ROOT_DIR}
> ${LIBCURL_OPENSSL_OPT}
> --with-ca-fallback
>
> diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake
> index 5a1141ebd..db2ae6227 100644
> --- a/cmake/compiler.cmake
> +++ b/cmake/compiler.cmake
> @@ -33,10 +33,6 @@ if (CMAKE_COMPILER_IS_GNUCC)
> Your GCC version is ${CMAKE_CXX_COMPILER_VERSION}, please
> update
> ")
> endif()
> -else()
> - if (BUILD_STATIC)
> - message(FATAL_ERROR "Static build is supported for GCC only")
> - endif()
> endif()
>
> #
> @@ -120,10 +116,23 @@ set (CMAKE_CXX_FLAGS_RELWITHDEBINFO
>
> unset(CC_DEBUG_OPT)
>
> +message(STATUS "Looking for libunwind.h")
> +find_path(UNWIND_INCLUDE_DIR libunwind.h)
> +message(STATUS "Looking for libunwind.h - ${UNWIND_INCLUDE_DIR}")
> +
> +if (UNWIND_INCLUDE_DIR)
> + include_directories(${UNWIND_INCLUDE_DIR})
> +endif()
> +
> +set(CMAKE_REQUIRED_INCLUDES ${UNWIND_INCLUDE_DIR})
> check_include_file(libunwind.h HAVE_LIBUNWIND_H)
> -if(BUILD_STATIC)
> +set(CMAKE_REQUIRED_INCLUDES "")
> +
> +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
> set(UNWIND_LIB_NAME libunwind.a)
> else()
> + # libunwind can't be compiled on macOS.
> + # But there exists libunwind.dylib as a part of MacOSSDK
> set(UNWIND_LIB_NAME unwind)
> endif()
> find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME})
> @@ -185,7 +194,10 @@ if (ENABLE_BACKTRACE)
> find_package_message(UNWIND_LIBRARIES "Found unwind"
> "${UNWIND_LIBRARIES}")
> endif()
>
> -if(BUILD_STATIC)
> +# On macOS there is no '-static-libstdc++' flag and it's use will
> +# raise following error:
> +# error: argument unused during compilation: '-static-libstdc++'
> +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
> # Static linking for c++ routines
> add_compile_flags("C;CXX" "-static-libstdc++")
> endif()
> diff --git a/cmake/os.cmake b/cmake/os.cmake
> index 905be61df..276a79b42 100644
> --- a/cmake/os.cmake
> +++ b/cmake/os.cmake
> @@ -107,7 +107,10 @@ elseif (${CMAKE_SYSTEM_NAME} STREQUAL "Darwin")
>
> # Latest versions of Homebrew wont 'link --force' for libraries, that
> were
> # preinstalled in system. So we'll use this dirty hack
> - find_program(HOMEBREW_EXECUTABLE brew)
> +
> + if (NOT BUILD_STATIC)
> + find_program(HOMEBREW_EXECUTABLE brew)
> + endif()
> if(EXISTS ${HOMEBREW_EXECUTABLE})
> execute_process(COMMAND ${HOMEBREW_EXECUTABLE} --prefix
> OUTPUT_VARIABLE HOMEBREW_PREFIX
> diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt
> new file mode 100644
> index 000000000..90029fdd8
> --- /dev/null
> +++ b/static-build/CMakeLists.txt
> @@ -0,0 +1,311 @@
> +cmake_minimum_required(VERSION 2.8)
> +
> +# Detect system compilers for further configuring dependencies to be
> +# builded with these compilers. This is used to build tarantool and
> +# it's dependencies by usign one compiler system (for example libicu
> +# by default uses clang if it exists when others uses gcc/g++ on
> +# linux machine).
> +project(tarantool-static C CXX)
> +
> +include(ExternalProject)
> +set(OPENSSL_VERSION 1.1.1f)
> +set(ZLIB_VERSION 1.2.11)
> +set(NCURSES_VERSION 6.2)
> +set(READLINE_VERSION 8.0)
> +set(UNWIND_VERSION 1.3-rc1)
> +
> +# Pass -isysroot=<SDK_PATH> option on Mac OS to a preprocessor and a C
> +# compiler to find header files installed with an SDK.
> +#
> +# The idea is to set these (DEPENDENCY_*) variables to corresponding
> +# environment variables at each depenency configure script.
> +#
> +# Note: Passing of CPPFLAGS / CFLAGS explicitly discards using of
> +# corresponsing environment variables. So pass empty LDFLAGS to discard
> +# using of corresponding environment variable. It is possible that a
> +# linker flag assumes that some compilation flag is set. We don't pass
> +# CFLAGS from environment, so we should not do it for LDFLAGS too.
> +set(DEPENDENCY_CFLAGS "")
> +set(DEPENDENCY_CPPFLAGS "")
> +set(DEPENDENCY_LDFLAGS)
> +if (APPLE)
> + set(DEPENDENCY_CFLAGS "${CMAKE_C_SYSROOT_FLAG}
> ${CMAKE_OSX_SYSROOT}")
> + set(DEPENDENCY_CPPFLAGS "${CMAKE_C_SYSROOT_FLAG}
> ${CMAKE_OSX_SYSROOT}")
> +endif()
> +
> +# Install all libraries required by tarantool at current build dir
> +
> +#
> +# OpenSSL
> +#
> +ExternalProject_Add(openssl
> + URL https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz
> + CONFIGURE_COMMAND <SOURCE_DIR>/config
> + CC=${CMAKE_C_COMPILER}
> + CXX=${CMAKE_CXX_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> +
> + --prefix=<INSTALL_DIR>
> + --libdir=lib
> + no-shared
> + INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install_sw
> +)
> +
> +#
> +# ICU
> +#
> +ExternalProject_Add(icu
> + URL
> https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz
> + # By default libicu is built by using clang/clang++ compiler (if it
> + # exists). Here is a link for detecting compilers at libicu configure
> + # script:
> https://github.com/unicode-org/icu/blob/7c7b8bd5702310b972f888299169bc3cc88bf0a6/icu4c/source/configure.ac#L135
> + # This will cause the problem on linux machine: tarantool is built
> + # with gcc/g++ and libicu is built with clang/clang++ (if it exists)
> + # so at linking stage `rellocation` errors will occur. To solve this,
> + # we can set CC/CXX to CMAKE_C_COMPILER/CMAKE_CXX_COMPILER variables
> + # manually which are detected above (by cmake `project()` command)
> + CONFIGURE_COMMAND <SOURCE_DIR>/source/configure
> + CC=${CMAKE_C_COMPILER}
> + CXX=${CMAKE_CXX_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> +
> + --with-data-packaging=static
> + --prefix=<INSTALL_DIR>
> + --disable-shared
> + --enable-static
> +)
> +
> +#
> +# ZLIB
> +#
> +ExternalProject_Add(zlib
> + URL https://zlib.net/zlib-${ZLIB_VERSION}.tar.gz
> + CONFIGURE_COMMAND env
> + CC=${CMAKE_C_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> + <SOURCE_DIR>/configure
> + --prefix=<INSTALL_DIR>
> + --static
> +)
> +
> +#
> +# Ncurses
> +#
> +ExternalProject_Add(ncurses
> + URL https://ftp.gnu.org/gnu/ncurses/ncurses-${NCURSES_VERSION}.tar.gz
> + CONFIGURE_COMMAND <SOURCE_DIR>/configure
> + CC=${CMAKE_C_COMPILER}
> + CXX=${CMAKE_CXX_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> +
> + --prefix=<INSTALL_DIR>
> +
> + # This flag enables creation of libcurses.a as a symlink to
> libncurses.a
> + # and disables subdir creation `ncurses` at
> <install_dir>/include. It is
> + # necessary for correct work of FindCurses.cmake module (this
> module is
> + # builtin at cmake package) which used in cmake/FindReadline.cmake
> + --enable-overwrite
> +
> + # enable building libtinfo to prevent linking with libtinfo from
> system
> + # directories
> + --with-termlib
> +
> + # set search paths for terminfo db
> +
> --with-terminfo-dirs=/lib/terminfo:/usr/share/terminfo:/etc/terminfo
> +
> + # disable install created terminfo db, use db from system
> + --disable-db-install
> + --without-progs
> + --without-manpages
> +)
> +
> +#
> +# ReadLine
> +#
> +ExternalProject_Add(readline
> + URL
> https://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz
> + CONFIGURE_COMMAND <SOURCE_DIR>/configure
> + CC=${CMAKE_C_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> +
> + --prefix=<INSTALL_DIR>
> + --disable-shared
> +)
> +
> +#
> +# ICONV
> +#
> +if (APPLE)
> + ExternalProject_Add(iconv
> + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
> + CONFIGURE_COMMAND <SOURCE_DIR>/configure
> + CC=${CMAKE_C_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> +
> + --prefix=<INSTALL_DIR>
> + --disable-shared
> + --enable-static
> + --with-gnu-ld
> + STEP_TARGETS download
> + )
> +else()
> + # In linux iconv is embedded into glibc
> + # So we find system header and copy it locally
> + find_path(ICONV_INCLUDE_DIR iconv.h)
> + if(NOT ICONV_INCLUDE_DIR)
> + message(FATAL_ERROR "iconv include header not found")
> + endif()
> +
> + set(ICONV_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix")
> +
> + add_custom_command(
> + OUTPUT "${ICONV_INSTALL_PREFIX}/include/iconv.h"
> + COMMAND ${CMAKE_COMMAND} -E make_directory
> + "${ICONV_INSTALL_PREFIX}/include"
> + COMMAND ${CMAKE_COMMAND} -E copy
> + "${ICONV_INCLUDE_DIR}/iconv.h"
> + "${ICONV_INSTALL_PREFIX}/include/iconv.h"
> + )
> + add_custom_target(iconv
> + DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h"
> + )
> + # This is a hack for further getting install directory of library
> + # by ExternalProject_Get_Property
> + set_target_properties(iconv
> + PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX}
> + )
> +endif()
> +
> +#
> +# Unwind
> +#
> +if (APPLE)
> + # On macOS libunwind is a part of MacOSX.sdk
> + # So we need to find library and header and
> + # copy it locally
> + find_path(UNWIND_INCLUDE_DIR libunwind.h)
> + find_library(UNWIND_LIBRARY libunwind.dylib
> + PATH_SUFFIXES system
> + )
> +
> + set(UNWIND_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/unwind-prefix")
> +
> + set(UNWIND_DEPENDENCIES)
> +
> + if (UNWIND_INCLUDE_DIR AND UNWIND_LIBRARY)
> + add_custom_command(
> + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/unwind.h"
> + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/libunwind.h"
> + COMMAND ${CMAKE_COMMAND} -E make_directory
> + "${UNWIND_INSTALL_PREFIX}/include"
> + COMMAND ${CMAKE_COMMAND} -E copy
> + "${UNWIND_INCLUDE_DIR}/libunwind.h"
> + "${UNWIND_INCLUDE_DIR}/unwind.h"
> + "${UNWIND_INSTALL_PREFIX}/include/"
> + )
> + add_custom_command(
> + OUTPUT "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib"
> + COMMAND ${CMAKE_COMMAND} -E make_directory
> + "${UNWIND_INSTALL_PREFIX}/lib"
> + COMMAND ${CMAKE_COMMAND} -E copy
> + "${UNWIND_LIBRARY}"
> + "${UNWIND_INSTALL_PREFIX}/lib/"
> + )
> + set(UNWIND_DEPENDENCIES
> + ${UNWIND_DEPENDENCIES}
> + "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib"
> + "${UNWIND_INSTALL_PREFIX}/include/libunwind.h"
> + )
> + else()
> + message(STATUS "Unwind not found")
> + endif()
> +
> + add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES})
> + # This is a hack for further getting install directory of library
> + # by ExternalProject_Get_Property
> + set_target_properties(unwind
> + PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX}
> + )
> +else()
> + ExternalProject_Add(unwind
> + URL
> https://download.savannah.nongnu.org/releases/libunwind/libunwind-${UNWIND_VERSION}.tar.gz
> + CONFIGURE_COMMAND <SOURCE_DIR>/configure
> + CC=${CMAKE_C_COMPILER}
> + CXX=${CMAKE_CXX_COMPILER}
> + CFLAGS=${DEPENDENCY_CFLAGS}
> + CPPFLAGS=${DEPENDENCY_CPPFLAGS}
> + LDFLAGS=${DEPENDENCY_LDFLAGS}
> + --prefix=<INSTALL_DIR>
> + --disable-shared
> + --enable-static
> + --disable-minidebuginfo # to prevent linking with liblzma
> + STEP_TARGETS download
> + )
> +endif()
> +
> +# Get install directories of builded libraries for building
> +# tarantool with custon CMAKE_PREFIX_PATH
> +foreach(PROJ openssl icu zlib ncurses readline iconv unwind)
> + ExternalProject_Get_Property(${PROJ} install_dir)
> + set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir})
> + set(TARANTOOL_DEPENDS ${PROJ} ${TARANTOOL_DEPENDS})
> + message(STATUS "Add external project ${PROJ} in ${install_dir}")
> +endforeach()
> +
> +ExternalProject_Add(tarantool
> + DEPENDS ${TARANTOOL_DEPENDS}
> + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/..
> + LIST_SEPARATOR :
> + CMAKE_ARGS
> + # Override LOCALSTATEDIR to avoid cmake "special" cases:
> + #
> https://cmake.org/cmake/help/v3.4/module/GNUInstallDirs.html#special-cases
> + -DCMAKE_INSTALL_LOCALSTATEDIR=<INSTALL_DIR>/var
> + -DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
> + -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
> + -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE
> + -DOPENSSL_USE_STATIC_LIBS=TRUE
> + -DBUILD_STATIC=TRUE
> + -DENABLE_DIST=TRUE
> + -DENABLE_BACKTRACE=TRUE
> + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
> + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
> + ${CMAKE_TARANTOOL_ARGS}
> + STEP_TARGETS build
> + BUILD_COMMAND $(MAKE)
> +)
> +
> +enable_testing()
> +ExternalProject_Get_Property(tarantool binary_dir)
> +SET(TARANTOOL_BINARY_DIR ${binary_dir})
> +
> +add_test(
> + NAME check-dependencies
> + COMMAND ${CMAKE_COMMAND}
> + -D FILE=${TARANTOOL_BINARY_DIR}/src/tarantool
> + -P CheckDependencies.cmake
> + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
> +)
> +
> +add_test(
> + NAME check-exports
> + COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool
> + ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/exports.test.lua
> +)
> +
> +add_test(
> + NAME check-traceback
> + COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool
> +
> ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/traceback.test.lua
> +)
> diff --git a/static-build/README.md b/static-build/README.md
> new file mode 100644
> index 000000000..2219b5026
> --- /dev/null
> +++ b/static-build/README.md
> @@ -0,0 +1,90 @@
> +# Tarantool static build tooling
> +
> +These files help to prepare environment for building Tarantool
> +statically. And builds it.
> +
> +## Prerequisites
> +
> +Please install following tools and libraries that will
> +be necessary for building and testing:
> +* git
> +* A C/C++ compiler.
> +
> + Ordinarily, this is gcc and g++ version 4.6 or later. On Mac OS X, this
> + is Clang version 3.2+.
> +* cmake
> +* autoconf automake libtool
> +* make
> +* Python and modules.
> +
> + Python interpreter is not necessary for building Tarantool itself,
> unless you
> + intend to use the “Run the test suite". For all platforms, this is
> python
> + version 2.7+ (but not 3.x). You need the following Python modules:
> + * pyyaml
> + * argparse
> + * msgpack-python
> + * gevent
> + * six
> +
> +### Here is an examples for your OS:
> +
> +CentOS:
> +
> +```bash
> +yum install -y \
> + git perl gcc cmake make gcc-c++ libstdc++-static autoconf automake
> libtool \
> + python-msgpack python-yaml python-argparse python-six python-gevent
> +```
> +
> +Ubuntu/Debian:
> +
> +```bash
> +apt-get install -y \
> + build-essential cmake make coreutils autoconf automake libtool sed \
> + python python-pip python-setuptools python-dev \
> + python-msgpack python-yaml python-argparse python-six python-gevent
> +```
> +
> +MacOS:
> +
> +Before you start please install default Xcode Tools by Apple:
> +
> +```bash
> +sudo xcode-select --install
> +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
> +```
> +
> +Install brew using command from
> +[Homebrew repository instructions](https://github.com/Homebrew/inst)
> +
> +After that run next script:
> +
> +```bash
> +brew install autoconf automake libtool cmake
> file://$${PWD}/tools/brew_taps/tntpython2.rbs
> +pip install --force-reinstall -r test-run/requirements.txt
> +```
> +
> +## Usage
> +
> +```bash
> +cmake .
> +make -j
> +ctest -V
> +```
> +
> +## Customize your build
> +
> +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS`
> variable
> +
> +### Usage
> +
> +There are three types of `CMAKE_BUILD_TYPE`:
> +* Debug - default
> +* Release
> +* RelWithDebInfo
> +
> +And you want to build tarantool with RelWithDebInfo:
> +
> +```bash
> +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" .
> +```
> diff --git a/static-build/test/CheckDependencies.cmake
> b/static-build/test/CheckDependencies.cmake
> new file mode 100644
> index 000000000..49e91e7fa
> --- /dev/null
> +++ b/static-build/test/CheckDependencies.cmake
> @@ -0,0 +1,43 @@
> +## This is a cmake-based test, it checks that tarantool static binary
> +# has no dependencies except allowed ones.
> +
> +include(GetPrerequisites)
> +if(NOT FILE)
> + message(FATAL_ERROR "Usage: "
> + "${CMAKE_COMMAND} -DFILE=<FILENAME> -P CheckDependencies.cmake")
> +elseif(NOT EXISTS ${FILE})
> + message(FATAL_ERROR "${FILE}: No such file")
> +endif()
> +
> +get_prerequisites(${FILE} DEPENDENCIES 0 0 "" "")
> +
> +if (APPLE)
> + set(ALLOWLIST
> + libSystem
> + CoreFoundation
> + libc++
> + )
> +elseif(UNIX)
> + set(ALLOWLIST
> + libdl
> + librt
> + libc
> + libm
> + libgcc_s
> + libpthread
> + )
> +else()
> + message(FATAL_ERROR "Unknown platform")
> +endif()
> +
> +foreach(DEPENDENCY_FILE ${DEPENDENCIES})
> + message("Dependency: ${DEPENDENCY_FILE}")
> +endforeach()
> +
> +foreach(DEPENDENCY_FILE ${DEPENDENCIES})
> + get_filename_component(libname ${DEPENDENCY_FILE} NAME_WE)
> + list (FIND ALLOWLIST ${libname} _index)
> + if (_index EQUAL -1)
> + message(FATAL_ERROR "Blocklisted dependency: ${DEPENDENCY_FILE}")
> + endif()
> +endforeach()
> diff --git a/static-build/test/static-build/exports.test.lua
> b/static-build/test/static-build/exports.test.lua
> new file mode 100755
> index 000000000..de54973d8
> --- /dev/null
> +++ b/static-build/test/static-build/exports.test.lua
> @@ -0,0 +1,142 @@
> +#!/usr/bin/env tarantool
> +
> +local tap = require('tap')
> +local ffi = require('ffi')
> +ffi.cdef([[
> + void *dlsym(void *handle, const char *symbol);
> +]])
> +
> +local test = tap.test('exports')
> +
> +-- See `man 3 dlsym`:
> +-- RTLD_DEFAULT
> +-- Find the first occurrence of the desired symbol using the default
> +-- shared object search order. The search will include global symbols
> +-- in the executable and its dependencies, as well as symbols in shared
> +-- objects that were dynamically loaded with the RTLD_GLOBAL flag.
> +local RTLD_DEFAULT = ffi.cast("void *", jit.os == "OSX" and -2LL or 0LL)
> +
> +local function check_symbol(sym)
> + test:ok(ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, ('Symbol %q
> found'):format(sym))
> +end
> +
> +local check_symbols = {
> + -- FFI
> +
> + 'guava',
> + 'base64_decode',
> + 'base64_encode',
> + 'SHA1internal',
> + 'random_bytes',
> + 'fiber_time',
> + 'ibuf_create',
> + 'ibuf_destroy',
> + 'port_destroy',
> + 'csv_create',
> + 'csv_destroy',
> + 'title_get',
> + 'title_update',
> + 'tnt_iconv',
> + 'tnt_iconv_open',
> + 'tnt_iconv_close',
> + 'exception_get_int',
> + 'exception_get_string',
> +
> + 'tarantool_lua_ibuf',
> + 'uuid_nil',
> + 'tt_uuid_create',
> + 'tt_uuid_str',
> + 'tt_uuid_is_equal',
> + 'tt_uuid_is_nil',
> + 'tt_uuid_bswap',
> + 'tt_uuid_from_string',
> + 'log_level',
> + 'log_format',
> + 'uri_parse',
> + 'uri_format',
> + 'PMurHash32',
> + 'PMurHash32_Process',
> + 'PMurHash32_Result',
> + 'crc32_calc',
> + 'mp_encode_double',
> + 'mp_encode_float',
> + 'mp_encode_decimal',
> + 'mp_decode_double',
> + 'mp_decode_float',
> + 'mp_decode_extl',
> + 'mp_sizeof_decimal',
> + 'decimal_unpack',
> +
> + 'log_type',
> + 'say_set_log_level',
> + 'say_logrotate',
> + 'say_set_log_format',
> + 'tarantool_uptime',
> + 'tarantool_exit',
> + 'log_pid',
> + 'space_by_id',
> + 'space_run_triggers',
> + 'space_bsize',
> + 'box_schema_version',
> +
> + 'crypto_EVP_MD_CTX_new',
> + 'crypto_EVP_MD_CTX_free',
> + 'crypto_HMAC_CTX_new',
> + 'crypto_HMAC_CTX_free',
> + 'crypto_stream_new',
> + 'crypto_stream_begin',
> + 'crypto_stream_append',
> + 'crypto_stream_commit',
> + 'crypto_stream_delete',
> +
> + -- Module API
> +
> + '_say',
> + 'swim_cfg',
> + 'swim_quit',
> + 'fiber_new',
> + 'fiber_cancel',
> + 'coio_wait',
> + 'coio_close',
> + 'coio_call',
> + 'coio_getaddrinfo',
> + 'luaT_call',
> + 'box_txn',
> + 'box_select',
> + 'clock_realtime',
> + 'string_strip_helper',
> +
> + -- Lua / LuaJIT
> +
> + 'lua_newstate',
> + 'lua_close',
> + 'luaL_loadstring',
> + 'luaJIT_profile_start',
> + 'luaJIT_profile_stop',
> + 'luaJIT_profile_dumpstack',
> +
> + 'ERR_error_string',
> + 'ERR_get_error',
> +
> + 'EVP_get_digestbyname',
> + 'EVP_get_cipherbyname',
> + 'EVP_CIPHER_CTX_new',
> + 'EVP_CIPHER_CTX_free',
> + 'EVP_CIPHER_block_size',
> + 'HMAC_Init_ex',
> + 'HMAC_Update',
> + 'HMAC_Final',
> +
> + 'ZSTD_compress',
> + 'ZSTD_decompress',
> + 'ZSTD_free',
> + 'ZSTD_malloc',
> + 'ZSTD_versionString',
> +}
> +
> +test:plan(#check_symbols)
> +for _, sym in ipairs(check_symbols) do
> + check_symbol(sym)
> +end
> +
> +os.exit(test:check() and 0 or 1)
> diff --git a/static-build/test/static-build/traceback.test.lua
> b/static-build/test/static-build/traceback.test.lua
> new file mode 100755
> index 000000000..32e1ed8b5
> --- /dev/null
> +++ b/static-build/test/static-build/traceback.test.lua
> @@ -0,0 +1,15 @@
> +#!/usr/bin/env tarantool
> +
> +local tap = require('tap')
> +local fiber = require('fiber')
> +
> +local test = tap.test('traceback')
> +test:plan(1)
> +
> +local info = fiber.info()[fiber.id()]
> +test:ok(info.backtrace ~= nil, 'fiber.info() has backtrace')
> +for _, l in pairs(info.backtrace or {}) do
> + test:diag('%s: %s', next(l))
> +end
> +
> +os.exit(test:check() and 0 or 1)
> --
> 2.26.2
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.tarantool.org/pipermail/tarantool-patches/attachments/20200910/fc6b330a/attachment.html>
More information about the Tarantool-patches
mailing list