From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-f193.google.com (mail-lj1-f193.google.com [209.85.208.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id C7C98469719 for ; Wed, 9 Sep 2020 18:45:51 +0300 (MSK) Received: by mail-lj1-f193.google.com with SMTP id a22so4137976ljp.13 for ; Wed, 09 Sep 2020 08:45:51 -0700 (PDT) From: HustonMmmavr Date: Wed, 9 Sep 2020 18:45:46 +0300 Message-Id: <20200909154546.1850-1-huston.mavr@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH v3] build: refactor static build process List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: tarantool-patches@dev.tarantool.org, yaroslav.dynnikov@gmail.com, alexander.turenko@tarantool.org, imun@tarantool.org From: Yaroslav Dynnikov Refactored static build process to use static-build/CMakeLists.txt instead of Dockerfile.staticbuild (this allows to support static build on macOS). Following third-party dependencies for static build are installed via cmake `ExternalProject_Add`: - OpenSSL - Zlib - Ncurses - Readline - Unwind - ICU * Added support static build for macOS * Fixed `CONFIGURE_COMMAND` while building bundled libcurl for static build at file cmake/BuildLibCURL.cmake: - disable building shared libcurl libraries (by setting `--disable-shared` option) - disable hiding libcurl symbols (by setting `--disable-symbol-hiding` option) - prevent linking libcurl with system libz (by setting `--with-zlib=${FOUND_ZLIB_ROOT_DIR}` option) * Removed Dockerfile.staticbuild * Added new gitlab.ci jobs to test new style static build: - static_build_cmake_linux - static_build_cmake_osx_15 * Removed static_docker_build gitlab.ci job Closes #5095 --- Branch: https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build Issue: https://github.com/tarantool/tarantool/issues/5095 Changes in v3: 1. Fixed typos at commit message 2. Fixed commentaries at .travis.mk 3. Fixed finding libz library at cmake/BuildLibCurl.cmake as at master branch 4. Instead of using static-build/.gitignore use the root .gitignore file 5. Refactored static-build/CMakeLists.txt: * Project name is set to tarantool-static * Instead of explicitly finding of c/c++ compilers current version uses cmake `project(tarantool-static C CXX)` command. This change requires to manually set CFLAGS/CPPFLAGS/LDFLAGS for all static-build dependencies (because building dependencies at macOS requires path to macOS SDK) the idea was taken from cmake/BuildLibCurl.cmake * Added commentaries about problem with libicu * Removed unused TEST_COMMAND at ExternalProject_Add(zlib ...) also removed unnecessary # STEP_TARGETS at ExternalProject_Add(readline ...) * Deleted doubled whitespaces 6. Fixed static-build/README.md: * Added list of required tools * Added example for ubuntu/debian * Fixed indentation and typos 7. Refactored tests: * Got rid of test-run * Deleted doubled whitespaces at exports.test.lua * Fixed defining of RTLD_DEFAULT .gitignore | 8 + .gitlab-ci.yml | 11 +- .travis.mk | 56 +++- Dockerfile.staticbuild | 98 ------ cmake/BuildLibCURL.cmake | 13 +- cmake/compiler.cmake | 24 +- cmake/os.cmake | 5 +- static-build/CMakeLists.txt | 311 ++++++++++++++++++ static-build/README.md | 90 +++++ static-build/test/CheckDependencies.cmake | 43 +++ .../test/static-build/exports.test.lua | 142 ++++++++ .../test/static-build/traceback.test.lua | 15 + 12 files changed, 692 insertions(+), 124 deletions(-) delete mode 100644 Dockerfile.staticbuild create mode 100644 static-build/CMakeLists.txt create mode 100644 static-build/README.md create mode 100644 static-build/test/CheckDependencies.cmake create mode 100755 static-build/test/static-build/exports.test.lua create mode 100755 static-build/test/static-build/traceback.test.lua diff --git a/.gitignore b/.gitignore index a42c7db35..c6e261e18 100644 --- a/.gitignore +++ b/.gitignore @@ -151,3 +151,11 @@ src/box/sql/parse.h src/box/sql/parse.c src/box/sql/opcodes.h src/box/sql/opcodes.c + +# static-build +static-build/*-prefix +static-build/Makefile +static-build/build + +# macOS files +.DS_Store diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 0ead08711..c9aef3dc7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -534,9 +534,14 @@ static_build: script: - ${GITLAB_MAKE} test_static_build -static_docker_build: +static_build_cmake_linux: + <<: *docker_test_definition + script: + - ${GITLAB_MAKE} test_static_build_cmake_linux + +static_build_cmake_osx_15: stage: test tags: - - deploy_test + - osx_15 script: - - ${GITLAB_MAKE} test_static_docker_build + - ${GITLAB_MAKE} test_static_build_cmake_osx diff --git a/.travis.mk b/.travis.mk index efc05cf05..58d0c1596 100644 --- a/.travis.mk +++ b/.travis.mk @@ -148,8 +148,14 @@ deps_debian_static: test_static_build: deps_debian_static CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk test_debian_no_deps -test_static_docker_build: - docker build --no-cache --network=host --build-arg RUN_TESTS=ON -f Dockerfile.staticbuild . +# New static build +# builddir used in this target - is a default build path from cmake +# ExternalProject_Add() +test_static_build_cmake_linux: + cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \ + make -j && ctest -V + cd test && /usr/bin/python test-run.py --force \ + --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build $(TEST_RUN_EXTRA_PARAMS) # ################### # Static Analysis @@ -193,15 +199,16 @@ build_osx: cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_WERROR=ON ${CMAKE_EXTRA_PARAMS} make -j -test_osx_no_deps: build_osx - # Limits: Increase the maximum number of open file descriptors on macOS: - # Travis-ci needs the "ulimit -n " call - # Gitlab-ci needs the "launchctl limit maxfiles " call - # Also gitlib-ci needs the password to change the limits, while - # travis-ci runs under root user. Limit setup must be in the same - # call as tests runs call. - # Tests: Temporary excluded replication/ suite with some tests - # from other suites by issues #4357 and #4370 + +# Limits: Increase the maximum number of open file descriptors on macOS: +# Travis-ci needs the "ulimit -n " call +# Gitlab-ci needs the "launchctl limit maxfiles " call +# Also gitlib-ci needs the password to change the limits, while +# travis-ci runs under root user. Limit setup must be in the same +# call as tests runs call. +# Tests: Temporary excluded replication/ suite with some tests +# from other suites by issues #4357 and #4370 +INIT_TEST_ENV_OSX=\ sudo -S launchctl limit maxfiles ${MAX_FILES} || : ; \ launchctl limit maxfiles || : ; \ ulimit -n ${MAX_FILES} || : ; \ @@ -210,11 +217,34 @@ test_osx_no_deps: build_osx launchctl limit maxproc || : ; \ ulimit -u ${MAX_PROC} || : ; \ ulimit -u ; \ - rm -rf /tmp/tnt ; \ - cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) + rm -rf /tmp/tnt + +test_osx_no_deps: build_osx + ${INIT_TEST_ENV_OSX}; \ + cd test && ./test-run.py --vardir /tmp/tnt --force $(TEST_RUN_EXTRA_PARAMS) test_osx: deps_osx test_osx_no_deps +# Static macOS build + +STATIC_OSX_PKGS=autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rb +base_deps_osx: + brew update || echo | /usr/bin/ruby -e \ + "$$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" + brew install --force ${STATIC_OSX_PKGS} || brew upgrade ${STATIC_OSX_PKGS} + pip install --force-reinstall -r test-run/requirements.txt + +# builddir used in this target - is a default build path from cmake +# ExternalProject_Add() +test_static_build_cmake_osx: base_deps_osx + cd static-build && cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DENABLE_WERROR=ON" . && \ + make -j && ctest -V + ${INIT_TEST_ENV_OSX}; \ + cd test && ./test-run.py --vardir /tmp/tnt \ + --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \ + --force $(TEST_RUN_EXTRA_PARAMS) + + ########### # FreeBSD # ########### diff --git a/Dockerfile.staticbuild b/Dockerfile.staticbuild deleted file mode 100644 index f67f46f5e..000000000 --- a/Dockerfile.staticbuild +++ /dev/null @@ -1,98 +0,0 @@ -FROM centos:7 - -RUN yum install -y epel-release -RUN yum install -y yum install https://centos7.iuscommunity.org/ius-release.rpm - -RUN set -x \ - && yum -y install \ - libstdc++ \ - libstdc++-static \ - readline \ - openssl \ - lz4 \ - binutils \ - ncurses \ - libgomp \ - lua \ - curl \ - tar \ - zip \ - unzip \ - libunwind \ - zlib \ - && yum -y install \ - perl \ - gcc-c++ \ - cmake \ - lz4-devel \ - binutils-devel \ - lua-devel \ - make \ - git \ - autoconf \ - automake \ - libtool \ - wget - -RUN yum -y install ncurses-static readline-static zlib-static pcre-static glibc-static - -RUN yum -y install python-devel python-pip - -RUN set -x && \ - cd / && \ - curl -O -L https://www.openssl.org/source/openssl-1.1.1f.tar.gz && \ - tar -xvf openssl-1.1.1f.tar.gz && \ - cd openssl-1.1.1f && \ - ./config --libdir=lib && \ - make -j && make install - -RUN set -x && \ - cd / && \ - curl -O -L https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz && \ - tar -xvf icu4c-62_1-src.tgz && \ - cd icu/source && \ - ./configure --with-data-packaging=static --enable-static --enable-shared && \ - make -j && make install - -RUN set -x && \ - cd / && \ - curl -O -L http://download.savannah.nongnu.org/releases/libunwind/libunwind-1.3-rc1.tar.gz && \ - tar -xvf libunwind-1.3-rc1.tar.gz && \ - cd libunwind-1.3-rc1 && \ - ./configure --enable-static --enable-shared && \ - make -j && make install - -COPY . /tarantool - -WORKDIR /tarantool - -RUN set -x && \ - git submodule init && \ - git submodule update - -# Cleanup for 'build' directory added, because it purges all artefacts -# produced for curl build, including the old configuration in build/curl -RUN set -x && \ - find . -name 'CMakeFiles' -type d -exec rm -rf {} + && \ - find . -name 'CMakeCache.txt' -type f -delete && \ - rm -rf build test/small test/luajit-tap - -RUN pip install -r /tarantool/test-run/requirements.txt - -RUN set -x && \ - cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo \ - -DENABLE_DIST:BOOL=ON \ - -DBUILD_STATIC=ON \ - -DOPENSSL_USE_STATIC_LIBS=ON \ - -DOPENSSL_ROOT_DIR=/usr/local \ - . && \ - make -j && make install - -ARG RUN_TESTS -RUN if [ -n "${RUN_TESTS}" ]; then \ - set -x && \ - cd test && \ - /usr/bin/python test-run.py --force; \ - fi - -ENTRYPOINT ["/bin/bash"] diff --git a/cmake/BuildLibCURL.cmake b/cmake/BuildLibCURL.cmake index 5f8b15a63..86fec39e9 100644 --- a/cmake/BuildLibCURL.cmake +++ b/cmake/BuildLibCURL.cmake @@ -4,15 +4,21 @@ macro(curl_build) set(LIBCURL_BINARY_DIR ${PROJECT_BINARY_DIR}/build/curl/work) set(LIBCURL_INSTALL_DIR ${PROJECT_BINARY_DIR}/build/curl/dest) + message(STATUS "Looking for zlib") + find_path(ZLIB_INCLUDE_DIR zlib.h) + message(STATUS "Looking for zlib.h - ${ZLIB_INCLUDE_DIR}") if (BUILD_STATIC) set(LIBZ_LIB_NAME libz.a) else() set(LIBZ_LIB_NAME z) endif() find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME}) - if ("${LIBZ_LIBRARY}" STREQUAL "LIBZ_LIBRARY-NOTFOUND") + message(STATUS "Looking for libz - ${LIBZ_LIBRARY}") + + if (NOT ZLIB_INCLUDE_DIR OR NOT LIBZ_LIBRARY) message(FATAL_ERROR "Unable to find zlib") endif() + get_filename_component(FOUND_ZLIB_ROOT_DIR ${ZLIB_INCLUDE_DIR} DIRECTORY) # Use the same OpenSSL library for libcurl as is used for # tarantool itself. @@ -88,9 +94,10 @@ macro(curl_build) --prefix --enable-static - --enable-shared + --disable-shared + --disable-symbol-hiding - --with-zlib + --with-zlib=${FOUND_ZLIB_ROOT_DIR} ${LIBCURL_OPENSSL_OPT} --with-ca-fallback diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake index 5a1141ebd..db2ae6227 100644 --- a/cmake/compiler.cmake +++ b/cmake/compiler.cmake @@ -33,10 +33,6 @@ if (CMAKE_COMPILER_IS_GNUCC) Your GCC version is ${CMAKE_CXX_COMPILER_VERSION}, please update ") endif() -else() - if (BUILD_STATIC) - message(FATAL_ERROR "Static build is supported for GCC only") - endif() endif() # @@ -120,10 +116,23 @@ set (CMAKE_CXX_FLAGS_RELWITHDEBINFO unset(CC_DEBUG_OPT) +message(STATUS "Looking for libunwind.h") +find_path(UNWIND_INCLUDE_DIR libunwind.h) +message(STATUS "Looking for libunwind.h - ${UNWIND_INCLUDE_DIR}") + +if (UNWIND_INCLUDE_DIR) + include_directories(${UNWIND_INCLUDE_DIR}) +endif() + +set(CMAKE_REQUIRED_INCLUDES ${UNWIND_INCLUDE_DIR}) check_include_file(libunwind.h HAVE_LIBUNWIND_H) -if(BUILD_STATIC) +set(CMAKE_REQUIRED_INCLUDES "") + +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) set(UNWIND_LIB_NAME libunwind.a) else() + # libunwind can't be compiled on macOS. + # But there exists libunwind.dylib as a part of MacOSSDK set(UNWIND_LIB_NAME unwind) endif() find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES ${UNWIND_LIB_NAME}) @@ -185,7 +194,10 @@ if (ENABLE_BACKTRACE) find_package_message(UNWIND_LIBRARIES "Found unwind" "${UNWIND_LIBRARIES}") endif() -if(BUILD_STATIC) +# On macOS there is no '-static-libstdc++' flag and it's use will +# raise following error: +# error: argument unused during compilation: '-static-libstdc++' +if(BUILD_STATIC AND NOT TARGET_OS_DARWIN) # Static linking for c++ routines add_compile_flags("C;CXX" "-static-libstdc++") endif() diff --git a/cmake/os.cmake b/cmake/os.cmake index 905be61df..276a79b42 100644 --- a/cmake/os.cmake +++ b/cmake/os.cmake @@ -107,7 +107,10 @@ elseif (${CMAKE_SYSTEM_NAME} STREQUAL "Darwin") # Latest versions of Homebrew wont 'link --force' for libraries, that were # preinstalled in system. So we'll use this dirty hack - find_program(HOMEBREW_EXECUTABLE brew) + + if (NOT BUILD_STATIC) + find_program(HOMEBREW_EXECUTABLE brew) + endif() if(EXISTS ${HOMEBREW_EXECUTABLE}) execute_process(COMMAND ${HOMEBREW_EXECUTABLE} --prefix OUTPUT_VARIABLE HOMEBREW_PREFIX diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt new file mode 100644 index 000000000..90029fdd8 --- /dev/null +++ b/static-build/CMakeLists.txt @@ -0,0 +1,311 @@ +cmake_minimum_required(VERSION 2.8) + +# Detect system compilers for further configuring dependencies to be +# builded with these compilers. This is used to build tarantool and +# it's dependencies by usign one compiler system (for example libicu +# by default uses clang if it exists when others uses gcc/g++ on +# linux machine). +project(tarantool-static C CXX) + +include(ExternalProject) +set(OPENSSL_VERSION 1.1.1f) +set(ZLIB_VERSION 1.2.11) +set(NCURSES_VERSION 6.2) +set(READLINE_VERSION 8.0) +set(UNWIND_VERSION 1.3-rc1) + +# Pass -isysroot= option on Mac OS to a preprocessor and a C +# compiler to find header files installed with an SDK. +# +# The idea is to set these (DEPENDENCY_*) variables to corresponding +# environment variables at each depenency configure script. +# +# Note: Passing of CPPFLAGS / CFLAGS explicitly discards using of +# corresponsing environment variables. So pass empty LDFLAGS to discard +# using of corresponding environment variable. It is possible that a +# linker flag assumes that some compilation flag is set. We don't pass +# CFLAGS from environment, so we should not do it for LDFLAGS too. +set(DEPENDENCY_CFLAGS "") +set(DEPENDENCY_CPPFLAGS "") +set(DEPENDENCY_LDFLAGS) +if (APPLE) + set(DEPENDENCY_CFLAGS "${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}") + set(DEPENDENCY_CPPFLAGS "${CMAKE_C_SYSROOT_FLAG} ${CMAKE_OSX_SYSROOT}") +endif() + +# Install all libraries required by tarantool at current build dir + +# +# OpenSSL +# +ExternalProject_Add(openssl + URL https://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz + CONFIGURE_COMMAND /config + CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + + --prefix= + --libdir=lib + no-shared + INSTALL_COMMAND ${CMAKE_MAKE_PROGRAM} install_sw +) + +# +# ICU +# +ExternalProject_Add(icu + URL https://github.com/unicode-org/icu/releases/download/release-62-1/icu4c-62_1-src.tgz + # By default libicu is built by using clang/clang++ compiler (if it + # exists). Here is a link for detecting compilers at libicu configure + # script: https://github.com/unicode-org/icu/blob/7c7b8bd5702310b972f888299169bc3cc88bf0a6/icu4c/source/configure.ac#L135 + # This will cause the problem on linux machine: tarantool is built + # with gcc/g++ and libicu is built with clang/clang++ (if it exists) + # so at linking stage `rellocation` errors will occur. To solve this, + # we can set CC/CXX to CMAKE_C_COMPILER/CMAKE_CXX_COMPILER variables + # manually which are detected above (by cmake `project()` command) + CONFIGURE_COMMAND /source/configure + CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + + --with-data-packaging=static + --prefix= + --disable-shared + --enable-static +) + +# +# ZLIB +# +ExternalProject_Add(zlib + URL https://zlib.net/zlib-${ZLIB_VERSION}.tar.gz + CONFIGURE_COMMAND env + CC=${CMAKE_C_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + /configure + --prefix= + --static +) + +# +# Ncurses +# +ExternalProject_Add(ncurses + URL https://ftp.gnu.org/gnu/ncurses/ncurses-${NCURSES_VERSION}.tar.gz + CONFIGURE_COMMAND /configure + CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + + --prefix= + + # This flag enables creation of libcurses.a as a symlink to libncurses.a + # and disables subdir creation `ncurses` at /include. It is + # necessary for correct work of FindCurses.cmake module (this module is + # builtin at cmake package) which used in cmake/FindReadline.cmake + --enable-overwrite + + # enable building libtinfo to prevent linking with libtinfo from system + # directories + --with-termlib + + # set search paths for terminfo db + --with-terminfo-dirs=/lib/terminfo:/usr/share/terminfo:/etc/terminfo + + # disable install created terminfo db, use db from system + --disable-db-install + --without-progs + --without-manpages +) + +# +# ReadLine +# +ExternalProject_Add(readline + URL https://ftp.gnu.org/gnu/readline/readline-${READLINE_VERSION}.tar.gz + CONFIGURE_COMMAND /configure + CC=${CMAKE_C_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + + --prefix= + --disable-shared +) + +# +# ICONV +# +if (APPLE) + ExternalProject_Add(iconv + URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz + CONFIGURE_COMMAND /configure + CC=${CMAKE_C_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + + --prefix= + --disable-shared + --enable-static + --with-gnu-ld + STEP_TARGETS download + ) +else() + # In linux iconv is embedded into glibc + # So we find system header and copy it locally + find_path(ICONV_INCLUDE_DIR iconv.h) + if(NOT ICONV_INCLUDE_DIR) + message(FATAL_ERROR "iconv include header not found") + endif() + + set(ICONV_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix") + + add_custom_command( + OUTPUT "${ICONV_INSTALL_PREFIX}/include/iconv.h" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${ICONV_INSTALL_PREFIX}/include" + COMMAND ${CMAKE_COMMAND} -E copy + "${ICONV_INCLUDE_DIR}/iconv.h" + "${ICONV_INSTALL_PREFIX}/include/iconv.h" + ) + add_custom_target(iconv + DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h" + ) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property + set_target_properties(iconv + PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX} + ) +endif() + +# +# Unwind +# +if (APPLE) + # On macOS libunwind is a part of MacOSX.sdk + # So we need to find library and header and + # copy it locally + find_path(UNWIND_INCLUDE_DIR libunwind.h) + find_library(UNWIND_LIBRARY libunwind.dylib + PATH_SUFFIXES system + ) + + set(UNWIND_INSTALL_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/unwind-prefix") + + set(UNWIND_DEPENDENCIES) + + if (UNWIND_INCLUDE_DIR AND UNWIND_LIBRARY) + add_custom_command( + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/unwind.h" + OUTPUT "${UNWIND_INSTALL_PREFIX}/include/libunwind.h" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${UNWIND_INSTALL_PREFIX}/include" + COMMAND ${CMAKE_COMMAND} -E copy + "${UNWIND_INCLUDE_DIR}/libunwind.h" + "${UNWIND_INCLUDE_DIR}/unwind.h" + "${UNWIND_INSTALL_PREFIX}/include/" + ) + add_custom_command( + OUTPUT "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib" + COMMAND ${CMAKE_COMMAND} -E make_directory + "${UNWIND_INSTALL_PREFIX}/lib" + COMMAND ${CMAKE_COMMAND} -E copy + "${UNWIND_LIBRARY}" + "${UNWIND_INSTALL_PREFIX}/lib/" + ) + set(UNWIND_DEPENDENCIES + ${UNWIND_DEPENDENCIES} + "${UNWIND_INSTALL_PREFIX}/lib/libunwind.dylib" + "${UNWIND_INSTALL_PREFIX}/include/libunwind.h" + ) + else() + message(STATUS "Unwind not found") + endif() + + add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES}) + # This is a hack for further getting install directory of library + # by ExternalProject_Get_Property + set_target_properties(unwind + PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX} + ) +else() + ExternalProject_Add(unwind + URL https://download.savannah.nongnu.org/releases/libunwind/libunwind-${UNWIND_VERSION}.tar.gz + CONFIGURE_COMMAND /configure + CC=${CMAKE_C_COMPILER} + CXX=${CMAKE_CXX_COMPILER} + CFLAGS=${DEPENDENCY_CFLAGS} + CPPFLAGS=${DEPENDENCY_CPPFLAGS} + LDFLAGS=${DEPENDENCY_LDFLAGS} + --prefix= + --disable-shared + --enable-static + --disable-minidebuginfo # to prevent linking with liblzma + STEP_TARGETS download + ) +endif() + +# Get install directories of builded libraries for building +# tarantool with custon CMAKE_PREFIX_PATH +foreach(PROJ openssl icu zlib ncurses readline iconv unwind) + ExternalProject_Get_Property(${PROJ} install_dir) + set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir}) + set(TARANTOOL_DEPENDS ${PROJ} ${TARANTOOL_DEPENDS}) + message(STATUS "Add external project ${PROJ} in ${install_dir}") +endforeach() + +ExternalProject_Add(tarantool + DEPENDS ${TARANTOOL_DEPENDS} + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/.. + LIST_SEPARATOR : + CMAKE_ARGS + # Override LOCALSTATEDIR to avoid cmake "special" cases: + # https://cmake.org/cmake/help/v3.4/module/GNUInstallDirs.html#special-cases + -DCMAKE_INSTALL_LOCALSTATEDIR=/var + -DCMAKE_INSTALL_PREFIX= + -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH} + -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE + -DOPENSSL_USE_STATIC_LIBS=TRUE + -DBUILD_STATIC=TRUE + -DENABLE_DIST=TRUE + -DENABLE_BACKTRACE=TRUE + -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER} + -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER} + ${CMAKE_TARANTOOL_ARGS} + STEP_TARGETS build + BUILD_COMMAND $(MAKE) +) + +enable_testing() +ExternalProject_Get_Property(tarantool binary_dir) +SET(TARANTOOL_BINARY_DIR ${binary_dir}) + +add_test( + NAME check-dependencies + COMMAND ${CMAKE_COMMAND} + -D FILE=${TARANTOOL_BINARY_DIR}/src/tarantool + -P CheckDependencies.cmake + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test +) + +add_test( + NAME check-exports + COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool + ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/exports.test.lua +) + +add_test( + NAME check-traceback + COMMAND ${TARANTOOL_BINARY_DIR}/src/tarantool + ${CMAKE_CURRENT_SOURCE_DIR}/test/static-build/traceback.test.lua +) diff --git a/static-build/README.md b/static-build/README.md new file mode 100644 index 000000000..2219b5026 --- /dev/null +++ b/static-build/README.md @@ -0,0 +1,90 @@ +# Tarantool static build tooling + +These files help to prepare environment for building Tarantool +statically. And builds it. + +## Prerequisites + +Please install following tools and libraries that will +be necessary for building and testing: +* git +* A C/C++ compiler. + + Ordinarily, this is gcc and g++ version 4.6 or later. On Mac OS X, this + is Clang version 3.2+. +* cmake +* autoconf automake libtool +* make +* Python and modules. + + Python interpreter is not necessary for building Tarantool itself, unless you + intend to use the “Run the test suite". For all platforms, this is python + version 2.7+ (but not 3.x). You need the following Python modules: + * pyyaml + * argparse + * msgpack-python + * gevent + * six + +### Here is an examples for your OS: + +CentOS: + +```bash +yum install -y \ + git perl gcc cmake make gcc-c++ libstdc++-static autoconf automake libtool \ + python-msgpack python-yaml python-argparse python-six python-gevent +``` + +Ubuntu/Debian: + +```bash +apt-get install -y \ + build-essential cmake make coreutils autoconf automake libtool sed \ + python python-pip python-setuptools python-dev \ + python-msgpack python-yaml python-argparse python-six python-gevent +``` + +MacOS: + +Before you start please install default Xcode Tools by Apple: + +```bash +sudo xcode-select --install +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer +``` + +Install brew using command from +[Homebrew repository instructions](https://github.com/Homebrew/inst) + +After that run next script: + +```bash +brew install autoconf automake libtool cmake file://$${PWD}/tools/brew_taps/tntpython2.rbs +pip install --force-reinstall -r test-run/requirements.txt +``` + +## Usage + +```bash +cmake . +make -j +ctest -V +``` + +## Customize your build + +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS` variable + +### Usage + +There are three types of `CMAKE_BUILD_TYPE`: +* Debug - default +* Release +* RelWithDebInfo + +And you want to build tarantool with RelWithDebInfo: + +```bash +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" . +``` diff --git a/static-build/test/CheckDependencies.cmake b/static-build/test/CheckDependencies.cmake new file mode 100644 index 000000000..49e91e7fa --- /dev/null +++ b/static-build/test/CheckDependencies.cmake @@ -0,0 +1,43 @@ +## This is a cmake-based test, it checks that tarantool static binary +# has no dependencies except allowed ones. + +include(GetPrerequisites) +if(NOT FILE) + message(FATAL_ERROR "Usage: " + "${CMAKE_COMMAND} -DFILE= -P CheckDependencies.cmake") +elseif(NOT EXISTS ${FILE}) + message(FATAL_ERROR "${FILE}: No such file") +endif() + +get_prerequisites(${FILE} DEPENDENCIES 0 0 "" "") + +if (APPLE) + set(ALLOWLIST + libSystem + CoreFoundation + libc++ + ) +elseif(UNIX) + set(ALLOWLIST + libdl + librt + libc + libm + libgcc_s + libpthread + ) +else() + message(FATAL_ERROR "Unknown platform") +endif() + +foreach(DEPENDENCY_FILE ${DEPENDENCIES}) + message("Dependency: ${DEPENDENCY_FILE}") +endforeach() + +foreach(DEPENDENCY_FILE ${DEPENDENCIES}) + get_filename_component(libname ${DEPENDENCY_FILE} NAME_WE) + list (FIND ALLOWLIST ${libname} _index) + if (_index EQUAL -1) + message(FATAL_ERROR "Blocklisted dependency: ${DEPENDENCY_FILE}") + endif() +endforeach() diff --git a/static-build/test/static-build/exports.test.lua b/static-build/test/static-build/exports.test.lua new file mode 100755 index 000000000..de54973d8 --- /dev/null +++ b/static-build/test/static-build/exports.test.lua @@ -0,0 +1,142 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local ffi = require('ffi') +ffi.cdef([[ + void *dlsym(void *handle, const char *symbol); +]]) + +local test = tap.test('exports') + +-- See `man 3 dlsym`: +-- RTLD_DEFAULT +-- Find the first occurrence of the desired symbol using the default +-- shared object search order. The search will include global symbols +-- in the executable and its dependencies, as well as symbols in shared +-- objects that were dynamically loaded with the RTLD_GLOBAL flag. +local RTLD_DEFAULT = ffi.cast("void *", jit.os == "OSX" and -2LL or 0LL) + +local function check_symbol(sym) + test:ok(ffi.C.dlsym(RTLD_DEFAULT, sym) ~= nil, ('Symbol %q found'):format(sym)) +end + +local check_symbols = { + -- FFI + + 'guava', + 'base64_decode', + 'base64_encode', + 'SHA1internal', + 'random_bytes', + 'fiber_time', + 'ibuf_create', + 'ibuf_destroy', + 'port_destroy', + 'csv_create', + 'csv_destroy', + 'title_get', + 'title_update', + 'tnt_iconv', + 'tnt_iconv_open', + 'tnt_iconv_close', + 'exception_get_int', + 'exception_get_string', + + 'tarantool_lua_ibuf', + 'uuid_nil', + 'tt_uuid_create', + 'tt_uuid_str', + 'tt_uuid_is_equal', + 'tt_uuid_is_nil', + 'tt_uuid_bswap', + 'tt_uuid_from_string', + 'log_level', + 'log_format', + 'uri_parse', + 'uri_format', + 'PMurHash32', + 'PMurHash32_Process', + 'PMurHash32_Result', + 'crc32_calc', + 'mp_encode_double', + 'mp_encode_float', + 'mp_encode_decimal', + 'mp_decode_double', + 'mp_decode_float', + 'mp_decode_extl', + 'mp_sizeof_decimal', + 'decimal_unpack', + + 'log_type', + 'say_set_log_level', + 'say_logrotate', + 'say_set_log_format', + 'tarantool_uptime', + 'tarantool_exit', + 'log_pid', + 'space_by_id', + 'space_run_triggers', + 'space_bsize', + 'box_schema_version', + + 'crypto_EVP_MD_CTX_new', + 'crypto_EVP_MD_CTX_free', + 'crypto_HMAC_CTX_new', + 'crypto_HMAC_CTX_free', + 'crypto_stream_new', + 'crypto_stream_begin', + 'crypto_stream_append', + 'crypto_stream_commit', + 'crypto_stream_delete', + + -- Module API + + '_say', + 'swim_cfg', + 'swim_quit', + 'fiber_new', + 'fiber_cancel', + 'coio_wait', + 'coio_close', + 'coio_call', + 'coio_getaddrinfo', + 'luaT_call', + 'box_txn', + 'box_select', + 'clock_realtime', + 'string_strip_helper', + + -- Lua / LuaJIT + + 'lua_newstate', + 'lua_close', + 'luaL_loadstring', + 'luaJIT_profile_start', + 'luaJIT_profile_stop', + 'luaJIT_profile_dumpstack', + + 'ERR_error_string', + 'ERR_get_error', + + 'EVP_get_digestbyname', + 'EVP_get_cipherbyname', + 'EVP_CIPHER_CTX_new', + 'EVP_CIPHER_CTX_free', + 'EVP_CIPHER_block_size', + 'HMAC_Init_ex', + 'HMAC_Update', + 'HMAC_Final', + + 'ZSTD_compress', + 'ZSTD_decompress', + 'ZSTD_free', + 'ZSTD_malloc', + 'ZSTD_versionString', +} + +test:plan(#check_symbols) +for _, sym in ipairs(check_symbols) do + check_symbol(sym) +end + +os.exit(test:check() and 0 or 1) diff --git a/static-build/test/static-build/traceback.test.lua b/static-build/test/static-build/traceback.test.lua new file mode 100755 index 000000000..32e1ed8b5 --- /dev/null +++ b/static-build/test/static-build/traceback.test.lua @@ -0,0 +1,15 @@ +#!/usr/bin/env tarantool + +local tap = require('tap') +local fiber = require('fiber') + +local test = tap.test('traceback') +test:plan(1) + +local info = fiber.info()[fiber.id()] +test:ok(info.backtrace ~= nil, 'fiber.info() has backtrace') +for _, l in pairs(info.backtrace or {}) do + test:diag('%s: %s', next(l)) +end + +os.exit(test:check() and 0 or 1) -- 2.26.2