Re: [Tarantool-patches] [PATCH] build: refactor static build process

[Alexandr Barulev <huston.mavr@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 16497 bytes --]

I've modified this patch: disabled building libunwind with minidebuginfo
to prevent linking libunwind with liblzma.
Here is a diff:

diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt
index 53ceb609c..d90a642e6 100644
--- a/static-build/CMakeLists.txt
+++ b/static-build/CMakeLists.txt
@@ -185,6 +185,7 @@ else()
             --prefix=<INSTALL_DIR>
             --disable-shared
             --enable-static
+            --disable-minidebuginfo # to prevent linking with liblzma
         STEP_TARGETS download
     )
 endif()

чт, 6 авг. 2020 г. в 16:32, Alexandr Barulev <huston.mavr@gmail.com>:

> I’ve squashed commit and changed it’s message;
> Also I’ve sended diff at previous answer
>
> https://github.com/tarantool/tarantool/tree/rosik/refactor-static-build
>
> ср, 5 авг. 2020 г. в 20:08, Mavr Huston <huston.mavr@gmail.com>:
>
>> Hi, thanks for the review!
>>
>> libicu installs as ExternalProject_Add too, its missed in commit message;
>>
>> Problem with curses and ncurses was on macOS and linux, because libcurses
>> is an entire copy of libncurses, and tarantool links with system
>> libcurses instead of libncurses installed as tarantool dependency, but
>> module FindCurses.cmkae provides workaround for this problem -
>> CURSES_NEED_NCURSES flag.- to use ncurses instead of curses. (i will fix
>> this part at commit message)
>>
>> About disable-shred flag, used at libcurl building - we want to link only
>> with
>> static libraries, so we prevent creating unused .so.
>>
>> I've renamed static_build_no_deps_* jobs after review to
>> static_build_cmake_*
>>
>> Also about such path tarantool-prefix/* - it's a cmake
>> ExternalProject_Add()
>> default path (i've also added comment at .travis.mk)
>>
>> Useless comments "Init macOS test env" deleted.
>>
>> >      if (BUILD_STATIC)
>> > -        set(LIBZ_LIB_NAME libz.a)
>> > +        find_library(LIBZ_LIBRARY NAMES libz.a)
>> >      else()
>> > -        set(LIBZ_LIB_NAME z)
>> > +        find_library(LIBZ_LIBRARY NAMES z)
>> >      endif()
>> > -    find_library(LIBZ_LIBRARY NAMES ${LIBZ_LIB_NAME})
>> Here we simplified code, by deleting useless variable.
>>
>> I've added commentaries to cmake/compiler.cmake about libunwind on macOS
>> and about ignoring flag -static-libstdc++ on macOS
>>
>> I've fixed static-build for using system compiler:  gcc/g++ on linux
>> and clang/clang++ on macOS
>>
>> I've refactored IF (NOT APPLE) condition to IF (APPLE) at
>> static-build/CMakeLists.txt
>>
>> I've mentioned macOS dependencies at static-build/README.md xcode-tools
>> and
>> others, also I've added example with CMAKE_TARANTOOL_ARGS.
>>
>> Added commentaries about _EP_INSTALL_DIR at static-build/CMakeLists.txt
>>
>> Also deleted unused use_unix_sockets_iproto = True
>>
>> Also deleted curl-features.test.lua, because after rebase on master it
>> fails,
>> due to missing curl_version_info symbol at tarantool binary. This symbol
>> lost
>> after #807c7fa584f21ee955b2a14623d70f7510a3650d (build: update curl
>> submodule
>> to 7.71.1 version )
>>
>>
>> After pass the review I'll squash this changes to base commit and update
>> commit
>> message.
>>
>> Here is a diff of changes:
>> =====================================
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 5ec5dd9b9..c9aef3dc7 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -534,14 +534,14 @@ static_build:
>>    script:
>>      - ${GITLAB_MAKE} test_static_build
>>
>> -static_build_no_deps_linux:
>> +static_build_cmake_linux:
>>    <<: *docker_test_definition
>>    script:
>> -    - ${GITLAB_MAKE} test_static_build_no_deps_linux
>> +    - ${GITLAB_MAKE} test_static_build_cmake_linux
>>
>> -static_build_no_deps_osx_15:
>> +static_build_cmake_osx_15:
>>    stage: test
>>    tags:
>>      - osx_15
>>    script:
>> -    - ${GITLAB_MAKE} test_static_build_no_deps_osx
>> +    - ${GITLAB_MAKE} test_static_build_cmake_osx
>> diff --git a/.travis.mk b/.travis.mk
>> index 64862348f..482672429 100644
>> --- a/.travis.mk
>> +++ b/.travis.mk
>> @@ -149,8 +149,8 @@ test_static_build: deps_debian_static
>>   CMAKE_EXTRA_PARAMS=-DBUILD_STATIC=ON make -f .travis.mk
>> test_debian_no_deps
>>
>>  # New static build
>> -
>> -test_static_build_no_deps_linux:
>> +# builddir used in this target - is a default build path from cmake
>> ExternalProject_Add()
>> +test_static_build_cmake_linux:
>>   cd static-build && cmake . && make -j && ctest -V
>>   cd test && /usr/bin/python test-run.py --force \
>>   --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build
>> $(TEST_RUN_EXTRA_PARAMS)
>> @@ -218,7 +218,6 @@ INIT_TEST_ENV_OSX=\
>>   rm -rf /tmp/tnt
>>
>>  test_osx_no_deps: build_osx
>> - # Init macOS test env
>>   ${INIT_TEST_ENV_OSX}; \
>>   cd test && ./test-run.py --vardir /tmp/tnt --force
>> $(TEST_RUN_EXTRA_PARAMS)
>>
>> @@ -233,9 +232,9 @@ base_deps_osx:
>>   brew install --force ${STATIC_OSX_PKGS} || brew upgrade
>> ${STATIC_OSX_PKGS}
>>   pip install --force-reinstall -r test-run/requirements.txt
>>
>> -test_static_build_no_deps_osx: base_deps_osx
>> +# builddir used in this target - is a default build path from cmake
>> ExternalProject_Add()
>> +test_static_build_cmake_osx: base_deps_osx
>>   cd static-build && cmake . && make -j && ctest -V
>> - # Init macOS test env
>>   ${INIT_TEST_ENV_OSX}; \
>>   cd test && ./test-run.py --vardir /tmp/tnt \
>>   --builddir ${PWD}/static-build/tarantool-prefix/src/tarantool-build \
>> diff --git a/cmake/FindReadline.cmake b/cmake/FindReadline.cmake
>> index 8422181d6..afe480679 100644
>> --- a/cmake/FindReadline.cmake
>> +++ b/cmake/FindReadline.cmake
>> @@ -14,6 +14,14 @@ if(BUILD_STATIC)
>>      if (NOT CURSES_INFO_LIBRARY)
>>          set(CURSES_INFO_LIBRARY "")
>>      endif()
>> +
>> +    # From Modules/FindCurses.cmake:
>> +    # Set ``CURSES_NEED_NCURSES`` to ``TRUE`` before the
>> +    # ``find_package(Curses)`` call if NCurses functionality is required.
>> +    # This flag is set for linking with required library (installed
>> +    # via static-build/CMakeLists.txt). If this variable won't be set
>> +    # then tarantool binary links with system library curses which is an
>> +    # entire copy of ncurses
>>      set(CURSES_NEED_NCURSES TRUE)
>>  endif()
>>
>> diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake
>> index 14f1e1186..db2ae6227 100644
>> --- a/cmake/compiler.cmake
>> +++ b/cmake/compiler.cmake
>> @@ -131,6 +131,8 @@ set(CMAKE_REQUIRED_INCLUDES "")
>>  if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
>>      set(UNWIND_LIB_NAME libunwind.a)
>>  else()
>> +    # libunwind can't be compiled on macOS.
>> +    # But there exists libunwind.dylib as a part of MacOSSDK
>>      set(UNWIND_LIB_NAME unwind)
>>  endif()
>>  find_library(UNWIND_LIBRARY PATH_SUFFIXES system NAMES
>> ${UNWIND_LIB_NAME})
>> @@ -192,6 +194,9 @@ if (ENABLE_BACKTRACE)
>>      find_package_message(UNWIND_LIBRARIES "Found unwind"
>> "${UNWIND_LIBRARIES}")
>>  endif()
>>
>> +# On macOS there is no '-static-libstdc++' flag and it's use will
>> +# raise following error:
>> +# error: argument unused during compilation: '-static-libstdc++'
>>  if(BUILD_STATIC AND NOT TARGET_OS_DARWIN)
>>      # Static linking for c++ routines
>>      add_compile_flags("C;CXX" "-static-libstdc++")
>> diff --git a/static-build/CMakeLists.txt b/static-build/CMakeLists.txt
>> index 86582af0a..53ceb609c 100644
>> --- a/static-build/CMakeLists.txt
>> +++ b/static-build/CMakeLists.txt
>> @@ -9,11 +9,18 @@ set(NCURSES_VERSION 6.2)
>>  set(READLINE_VERSION 8.0)
>>  set(UNWIND_VERSION 1.3-rc1)
>>
>> -find_program(C_COMPILER gcc)
>> -find_program(CXX_COMPILER g++)
>> +if (APPLE)
>> +    find_program(C_COMPILER clang)
>> +    find_program(CXX_COMPILER clang++)
>> +else()
>> +    find_program(C_COMPILER gcc)
>> +    find_program(CXX_COMPILER g++)
>> +endif()
>>  set(CMAKE_C_COMPILER ${C_COMPILER})
>>  set(CMAKE_CXX_COMPILER ${CXX_COMPILER})
>>
>> +# Install all libraries required by tarantool at current build dir
>> +
>>  #
>>  # OpenSSL
>>  #
>> @@ -80,7 +87,18 @@ ExternalProject_Add(readline
>>  #
>>  # ICONV
>>  #
>> -if (NOT APPLE)
>> +if (APPLE)
>> +    ExternalProject_Add(iconv
>> +        URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
>> +        CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER}
>> +            <SOURCE_DIR>/configure
>> +            --prefix=<INSTALL_DIR>
>> +            --disable-shared
>> +            --enable-static
>> +            --with-gnu-ld
>> +        STEP_TARGETS download
>> +    )
>> +else()
>>      # In linux iconv is embedded into glibc
>>      # So we find system header and copy it locally
>>      find_path(ICONV_INCLUDE_DIR iconv.h)
>> @@ -101,20 +119,11 @@ if (NOT APPLE)
>>      add_custom_target(iconv
>>          DEPENDS
>> "${CMAKE_CURRENT_BINARY_DIR}/iconv-prefix/include/iconv.h"
>>      )
>> +    # This is a hack for further getting install directory of library
>> +    # by ExternalProject_Get_Property
>>      set_target_properties(iconv
>>          PROPERTIES _EP_INSTALL_DIR ${ICONV_INSTALL_PREFIX}
>>      )
>> -else()
>> -    ExternalProject_Add(iconv
>> -        URL https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
>> -        CONFIGURE_COMMAND CC=${CMAKE_C_COMPILER}
>> -            <SOURCE_DIR>/configure
>> -            --prefix=<INSTALL_DIR>
>> -            --disable-shared
>> -            --enable-static
>> -            --with-gnu-ld
>> -        STEP_TARGETS download
>> -    )
>>  endif()
>>
>>  #
>> @@ -162,6 +171,8 @@ if (APPLE)
>>      endif()
>>
>>      add_custom_target(unwind DEPENDS ${UNWIND_DEPENDENCIES})
>> +    # This is a hack for further getting install directory of library
>> +    # by ExternalProject_Get_Property
>>      set_target_properties(unwind
>>          PROPERTIES _EP_INSTALL_DIR ${UNWIND_INSTALL_PREFIX}
>>      )
>> @@ -178,6 +189,8 @@ else()
>>      )
>>  endif()
>>
>> +# Get install directories of builded libraries for building
>> +# tarantool with custon CMAKE_PREFIX_PATH
>>  foreach(PROJ openssl icu zlib ncurses readline iconv unwind)
>>      ExternalProject_Get_Property(${PROJ} install_dir)
>>      set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH}:${install_dir})
>> @@ -197,16 +210,14 @@ ExternalProject_Add(tarantool
>>          -DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}
>>          -DCMAKE_FIND_USE_CMAKE_SYSTEM_PATH=FALSE
>>          -DOPENSSL_USE_STATIC_LIBS=TRUE
>> -        -DCMAKE_BUILD_TYPE=Debug
>>          -DBUILD_STATIC=TRUE
>>          -DENABLE_DIST=TRUE
>>          -DENABLE_BACKTRACE=TRUE
>> -        -DPACKAGE:STRING=${PACKAGE_NAME}
>>          -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
>>          -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
>>          ${CMAKE_TARANTOOL_ARGS}
>> -    BUILD_COMMAND ${CMAKE_MAKE_PROGRAM} -j
>>      STEP_TARGETS build
>> +    BUILD_COMMAND $(MAKE)
>>  )
>>
>>  enable_testing()
>> diff --git a/static-build/README.md b/static-build/README.md
>> index 29fe085c3..0019e963f 100644
>> --- a/static-build/README.md
>> +++ b/static-build/README.md
>> @@ -13,6 +13,24 @@ yum install -y \
>>      python-msgpack python-yaml python-argparse python-six python-gevent
>>  ```
>>
>> +MacOS:
>> +
>> +Before you start please install default Xcode Tools by Apple:
>> +
>> +```
>> +sudo xcode-select --install
>> +sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer
>> +```
>> +
>> +Install brew using command from
>> +[Homebrew repository instructions](https://github.com/Homebrew/inst)
>> +
>> +After that run next script:
>> +
>> +```bash
>> +  brew install autoconf automake libtool cmake
>> file://$${PWD}/tools/brew_taps/tntpython2.rbs
>> +  pip install --force-reinstall -r test-run/requirements.txt
>> +```
>>
>>  ### Usage
>>
>> @@ -21,3 +39,20 @@ cmake .
>>  make -j
>>  ctest -V
>>  ```
>> +
>> +## Customize your build
>> +
>> +If you want to customise build, you need to set `CMAKE_TARANTOOL_ARGS`
>> variable
>> +
>> +### Usage
>> +
>> +There is three types of `CMAKE_BUILD_TYPE`:
>> +* Debug - default
>> +* Release
>> +* RelWithDebInfo
>> +
>> +And you want to build tarantool with RelWithDebInfo:
>> +
>> +```bash
>> +cmake -DCMAKE_TARANTOOL_ARGS="-DCMAKE_BUILD_TYPE=RelWithDebInfo" .
>> +```
>> diff --git a/static-build/test/static-build/curl-features.test.lua
>> b/static-build/test/static-build/curl-features.test.lua
>> deleted file mode 100755
>> index 57b1c4306..000000000
>> --- a/static-build/test/static-build/curl-features.test.lua
>> +++ /dev/null
>> @@ -1,67 +0,0 @@
>> -#!/usr/bin/env tarantool
>> -
>> -local tap = require('tap')
>> -local ffi = require('ffi')
>> -ffi.cdef([[
>> -    struct curl_version_info_data {
>> -        int age;                  /* see description below */
>> -        const char *version;      /* human readable string */
>> -        unsigned int version_num; /* numeric representation */
>> -        const char *host;         /* human readable string */
>> -        int features;             /* bitmask, see below */
>> -        char *ssl_version;        /* human readable string */
>> -        long ssl_version_num;     /* not used, always zero */
>> -        const char *libz_version; /* human readable string */
>> -        const char * const *protocols; /* protocols */
>> -
>> -        /* when 'age' is CURLVERSION_SECOND or higher, the members below
>> exist */
>> -        const char *ares;         /* human readable string */
>> -        int ares_num;             /* number */
>> -
>> -        /* when 'age' is CURLVERSION_THIRD or higher, the members below
>> exist */
>> -        const char *libidn;       /* human readable string */
>> -
>> -        /* when 'age' is CURLVERSION_FOURTH or higher (>= 7.16.1), the
>> members
>> -           below exist */
>> -        int iconv_ver_num;       /* '_libiconv_version' if iconv support
>> enabled */
>> -
>> -        const char *libssh_version; /* human readable string */
>> -
>> -        /* when 'age' is CURLVERSION_FIFTH or higher (>= 7.57.0), the
>> members
>> -           below exist */
>> -        unsigned int brotli_ver_num; /* Numeric Brotli version
>> -                                        (MAJOR << 24) | (MINOR << 12) |
>> PATCH */
>> -        const char *brotli_version; /* human readable string. */
>> -
>> -        /* when 'age' is CURLVERSION_SIXTH or higher (>= 7.66.0), the
>> members
>> -           below exist */
>> -        unsigned int nghttp2_ver_num; /* Numeric nghttp2 version
>> -                                         (MAJOR << 16) | (MINOR << 8) |
>> PATCH */
>> -        const char *nghttp2_version; /* human readable string. */
>> -
>> -        const char *quic_version;    /* human readable quic (+ HTTP/3)
>> library +
>> -                                        version or NULL */
>> -
>> -        /* when 'age' is CURLVERSION_SEVENTH or higher (>= 7.70.0), the
>> members
>> -           below exist */
>> -        const char *cainfo;          /* the built-in default
>> CURLOPT_CAINFO, might
>> -                                        be NULL */
>> -        const char *capath;          /* the built-in default
>> CURLOPT_CAPATH, might
>> -                                        be NULL */
>> -    };
>> -
>> -    struct curl_version_info_data *curl_version_info(int age);
>> -]])
>> -
>> -local info = ffi.C.curl_version_info(7)
>> -local test = tap.test('curl-features')
>> -test:plan(2)
>> -
>> -if test:ok(info.ssl_version ~= nil, 'Curl built with SSL support') then
>> -    test:diag('ssl_version: ' .. ffi.string(info.ssl_version))
>> -end
>> -if test:ok(info.libz_version ~= nil, 'Curl built with LIBZ') then
>> -    test:diag('libz_version: ' .. ffi.string(info.libz_version))
>> -end
>> -
>> -os.exit(test:check() and 0 or 1)
>> diff --git a/static-build/test/static-build/suite.ini
>> b/static-build/test/static-build/suite.ini
>> index 4da3d5d2f..92e349466 100644
>> --- a/static-build/test/static-build/suite.ini
>> +++ b/static-build/test/static-build/suite.ini
>> @@ -3,4 +3,3 @@ core = app
>>  description = Static build tests
>>  script = box.lua
>>  is_parallel = True
>> -use_unix_sockets_iproto = True
>>
>

[-- Attachment #2: Type: text/html, Size: 18442 bytes --]