[Tarantool-patches] [PATCH v2 2/2] cmake: remove dynamic-list linker option
Vladislav Shpilevoy
v.shpilevoy at tarantool.org
Sat Apr 18 19:51:10 MSK 2020
Thanks for the review and the patch!
> : diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
> : index 6c756f00f..ca978aa89 100644
> : --- a/src/CMakeLists.txt
> : +++ b/src/CMakeLists.txt
> : @@ -245,56 +245,12 @@ if(BUILD_STATIC)
> : endif()
> : endif()
> :
> : -# Exports syntax is toolchain-dependent, preprocessing is necessary
> : -set(exports_file ${CMAKE_BINARY_DIR}/extra/exports.${CMAKE_SYSTEM_NAME})
> : -add_custom_target(preprocess_exports
> : - DEPENDS ${exports_file})
> : -add_custom_command(
> : - OUTPUT ${exports_file}
> : - DEPENDS ${CMAKE_SOURCE_DIR}/extra/exports
> : - COMMAND ${CMAKE_SOURCE_DIR}/extra/mkexports
>
> Should we delete extra/mkexports as well? Which is apparently not needed anymore.
Indeed. Removed:
====================
diff --git a/extra/mkexports b/extra/mkexports
deleted file mode 100755
index 145e5b8ce..000000000
--- a/extra/mkexports
+++ /dev/null
@@ -1,27 +0,0 @@
-#! /bin/sh
-# $1 - in file
-# $2 - out file
-# $3 - os
-# $4 - export templates
-if [ "x$3x" = xDarwinx ]; then
- # _func1
- # _func2
- sed -e 's/#.*//; /^[[:space:]]*$/d; s/^/_/;' $1 > $2
-else
- # {
- # func1;
- # func2;
- # };
- echo "$4"
- ( echo "{" && {
- # combine static defined list of functions
- cat $1 ;
- # with list of exported functions of bundled libraries
- for so in $4 ; do {
- # exported names from shared object
- nm -D $so ||
- # or follow patch from shared object script
- nm -D `cat $so | grep GROUP | awk '{print $3}'` ;
- } | awk '{print $3}' ; done ;
- } | sed '/^\s*$/d;s/$/;/;' && echo "};" ) > $2
-fi
====================
> : diff --git a/src/exports.c b/src/exports.c
> : new file mode 100644
> : index 000000000..243fdde28
> : --- /dev/null
> : +++ b/src/exports.c
> : @@ -0,0 +1,511 @@
> : +/**
> : + * Symbol is just an address. No need to know its definition or
> : + * even type to get that address. Even an integer global variable
> : + * can be referenced as extern void(*)(void).
>
> I'd emphasize here the C-symbol aspect, i.e.
>
> "Non-decorated C symbol is just an address for the linker purposes...
But it is decorated. The decoration is fake, but it is still decorated.
Besides, I don't know about a way to reference a symbol without any
decoration. Nor what it has to do with C. From linker's point of view
there is no C anyway. We can even reference a symbol, which is
implemented not in C, but linked from some static or dynamic library.
It can be implemented in assembly. Or can be a symbol created in a linker
script, without having anything under it.
> : + */
> : +#define EXPORT(symbol) do { \
> : + extern void *symbol(void); \
> : + syms[__COUNTER__ - index_base] = symbol; \
> : +} while (0)
> : +
> : +void
> : +export_syms(void)
> : +{
> : + /*
> : + * Compiler should think the exported symbols are
> : + * reachable. Volatile condition prevents their removal
> : + * during optimizations appliance.
> : + */
> : + volatile int never_true = 0;
> : + if (!never_true)
> : + return;
> : + const int index_base = __COUNTER__ + 1;
> : + void **syms = (void **)&never_true;
> : + /*
> : + * Keep the symbols sorted by name for search and addition
> : + * simplicity, to avoid duplicates. Makes no much sense to
> : + * split them into any sections. Anyway some symbols will
> : + * end-up belonging to several of them at once, and it
> : + * would require moving lines here when a symbol goes from
> : + * privately exported to the public C API.
> : + */
> : + EXPORT(base64_bufsize);
> ...
> : + EXPORT(tt_uuid_create);
> : + EXPORT(tt_uuid_from_string);
> : + EXPORT(tt_uuid_is_equal);
> : + EXPORT(tt_uuid_is_nil);
> : + EXPORT(tt_uuid_str);
> : + EXPORT(uri_format);
> : + EXPORT(uri_parse);
> : + EXPORT(uuid_nil);
> : + EXPORT(uuid_unpack);
> : +
> : + const int func_count = __COUNTER__ - index_base;
> : + for (int i = 0; i < func_count; ++i)
> : + ((void **(*)(void))syms[i])();
> : +}
> : +
> : +#undef EXPORT
>
> This is very cool! But I'm still confused though (and, eventually, any static checker would be even more confused) - why there is no simple static array or even blatant original ffi_syms[] array as in https://github.com/tarantool/tarantool/commit/4c4c1ccbe422e1420ce0c0b36194fc61bb857ed4#diff-0d537fbabe94394d24abfc403247c75eL67 which would be initialized with these declared symbols?
>
> Using volatile integer as and array looks very, very much confusing.
I didn't want the symbol in a global array. It would occupy memory in .data,
and a global array looks like it is really going to be accessed, even if it
is not.
Volatile variable is to avoid actual usage of the functions, and yet not
remove them. That was an attempt to avoid doing a fake condition of kind
'time(NULL) == 0', like it was done in the old exporting machinery.
> Update - I've just looked into object code generated by compiler, and it's apparently just blatantly ignored all the tricks - leaving no references to all functions mentioned here:
Yeah, I saw that in Travis. Looks like gcc on Linux is 'too smart' in
Release mode. All was fine on Mac, and with clang on Linux.
> (please see U type symbols for undefined, external functions)
>
> tsafin at M1BOOK6319:~/tarantool1/build-symbols$ nm -a ./src/CMakeFiles/tarantool.dir/exports.c.o
> 0000000000000000 b .bss
> 0000000000000000 n .comment
> 0000000000000000 d .data
> 0000000000000000 N .debug_abbrev
> 0000000000000000 N .debug_aranges
> 0000000000000000 N .debug_info
> 0000000000000000 N .debug_line
> 0000000000000000 N .debug_loc
> 0000000000000000 N .debug_str
> 0000000000000000 r .eh_frame
> 0000000000000000 a exports.c
> 0000000000000000 T export_syms
> 0000000000000000 n .note.GNU-stack
> 0000000000000000 t .text
>
> There is no undefined symbol to link them against :(
>
> And here is the disassembly - please see all EXPORT(x) lines thrown out.
>
> ```
> objdump -dtS -a ./src/CMakeFiles/tarantool.dir/exports.c.o
> ...
> Disassembly of section .text:
>
> 0000000000000000 <export_syms>:
> syms[__COUNTER__ - index_base] = symbol; \
> } while (0)
>
> void
> export_syms(void)
> {
> 0: 55 push %rbp
> 1: 48 89 e5 mov %rsp,%rbp
> 4: 41 54 push %r12
> 6: 53 push %rbx
> 7: 48 83 ec 10 sub $0x10,%rsp
> /*
> * Compiler should think the exported symbols are
> * reachable. Volatile condition prevents their removal
> * during optimizations appliance.
> */
> volatile int never_true = 0;
> b: c7 45 ec 00 00 00 00 movl $0x0,-0x14(%rbp)
> if (!never_true)
> 12: 8b 45 ec mov -0x14(%rbp),%eax
> 15: 85 c0 test %eax,%eax
> 17: 74 18 je 31 <export_syms+0x31>
> 19: 31 db xor %ebx,%ebx
> 1b: 4c 8d 65 ec lea -0x14(%rbp),%r12
> 1f: 90 nop
> EXPORT(uuid_nil);
> EXPORT(uuid_unpack);
>
> const int func_count = __COUNTER__ - index_base;
> for (int i = 0; i < func_count; ++i)
> ((void **(*)(void))syms[i])();
> 20: 41 ff 14 dc callq *(%r12,%rbx,8)
> 24: 48 83 c3 01 add $0x1,%rbx
> for (int i = 0; i < func_count; ++i)
> 28: 48 81 fb 80 01 00 00 cmp $0x180,%rbx
> 2f: 75 ef jne 20 <export_syms+0x20>
> }
> 31: 48 83 c4 10 add $0x10,%rsp
> 35: 5b pop %rbx
> 36: 41 5c pop %r12
> 38: 5d pop %rbp
> 39: c3 retq
> ```
I managed to fix it like this:
====================
diff --git a/src/exports.c b/src/exports.c
index 243fdde28..7c6b84ef0 100644
--- a/src/exports.c
+++ b/src/exports.c
@@ -93,8 +93,8 @@
* can be referenced as extern void(*)(void).
*/
#define EXPORT(symbol) do { \
- extern void *symbol(void); \
- syms[__COUNTER__ - index_base] = symbol; \
+ extern void symbol(void); \
+ symbol(); \
} while (0)
void
@@ -108,8 +108,6 @@ export_syms(void)
volatile int never_true = 0;
if (!never_true)
return;
- const int index_base = __COUNTER__ + 1;
- void **syms = (void **)&never_true;
/*
* Keep the symbols sorted by name for search and addition
* simplicity, to avoid duplicates. Makes no much sense to
@@ -502,10 +500,6 @@ export_syms(void)
EXPORT(uri_parse);
EXPORT(uuid_nil);
EXPORT(uuid_unpack);
-
- const int func_count = __COUNTER__ - index_base;
- for (int i = 0; i < func_count; ++i)
- ((void **(*)(void))syms[i])();
}
#undef EXPORT
====================
But still don't understand why gcc decided that all the exports could
be removed. In the assembly I see that the cycle, calling all the
functions, is not removed. There are still 384 calls. But what does it
call, if all the exports were dropped?
> diff --git a/src/exports.c b/src/exports.c
> index 243fdde28..54a4d9946 100644
> --- a/src/exports.c
> +++ b/src/exports.c
> @@ -88,424 +88,25 @@
> */
>
> /**
> - * Symbol is just an address. No need to know its definition or
> - * even type to get that address. Even an integer global variable
> - * can be referenced as extern void(*)(void).
> + * Non-decorated C symbol is just an address for the linker purposes.
> + * No need to know its definition or even type to get that address.
> + * Even an integer global variable can be referenced as simple
> + * extern void(*)(void).
> */
> -#define EXPORT(symbol) do { \
> - extern void *symbol(void); \
> - syms[__COUNTER__ - index_base] = symbol; \
> -} while (0)
> +#define EXPORT_SYMBOL(symbol) extern void symbol();
> +#include "export-symbols.h"
>
> -void
> +void*
> export_syms(void)
The problem here is that the function's result is never used. I am
afraid some smart ass compiler may notice that, since exports.c and
main.cc are built together, and may optimize this call out.
Also return of a value assumes a caller needs to do something with it.
I didn't like that in the Nikolay's solution. I wanted to make a function,
which does all the work by itself.
However I like that all exports are in a separate file, not surrounded
by hacks of exports.c. So I applied your solution with some code
style problems fixed. Thanks for the help.
The patch diff is close to 100%, so below is the whole new patch,
without incremental diff.
====================
cmake: remove dynamic-list linker option
dynamic-list (exported_symbols_list on Mac) was used to forbid
export of all symbols of the tarantool executable except a given
list. Motivation of that was to avoid hacking the linker with
false usage of symbols needed to be exported. As a consequence,
symbols not listed in these options became invisible.
Before these options, when a symbol was defined, but not used in
the final executable, the linker could throw it away, even though
many symbols were used by Lua FFI, or should be visible for user's
dynamic modules. Where the linker, obviously, can't see if they
are needed.
To make the linker believe the symbols are actually needed there
was a hack with getting pointers at these functions and doing
something with them.
For example, assume we have 'test()' function in 'box' static
library:
int
test(void);
It is not used anywhere in the final executable. So to trick the
linker there is a function 'export_syms()' declared, which takes a
pointer at 'test()' and seemingly does something with it (or
actually does - it does not matter):
void
export_syms()
{
void *syms[] = {test};
if (time(NULL) == 0) {
syms[0]();
syms[1]();
...
}
}
Some users want to use not documented but visible symbols, so the
patch removes the dynamic-list option, and returns the linker
hack back. But with 0 dependencies in the export file.
Closes #2971
diff --git a/.gitignore b/.gitignore
index cda28d79f..a42c7db35 100644
--- a/.gitignore
+++ b/.gitignore
@@ -57,7 +57,6 @@ extra/dist/tarantool.logrotate
extra/dist/tarantool at .service
extra/dist/tarantool.tmpfiles.conf
extra/dist/tarantool-generator
-extra/exports.*
cmake_install.cmake
config.mk
config.guess
diff --git a/extra/exports b/extra/exports
deleted file mode 100644
index a9add2cc1..000000000
--- a/extra/exports
+++ /dev/null
@@ -1,404 +0,0 @@
-# Symbols exported by the main Tarantool executable
-
-# FFI
-
-password_prepare
-lbox_socket_local_resolve
-lbox_socket_nonblock
-base64_decode
-base64_encode
-base64_bufsize
-SHA1internal
-guava
-random_bytes
-fiber_time
-fiber_time64
-fiber_clock
-fiber_clock64
-tarantool_lua_slab_cache
-ibuf_create
-ibuf_reinit
-ibuf_destroy
-ibuf_reserve_slow
-port_destroy
-csv_create
-csv_destroy
-csv_setopt
-csv_iterator_create
-csv_next
-csv_feed
-csv_escape_field
-title_update
-title_get
-title_set_interpretor_name
-title_get_interpretor_name
-title_set_script_name
-title_get_script_name
-title_set_custom
-title_get_custom
-title_set_status
-title_get_status
-tnt_iconv_open
-tnt_iconv_close
-tnt_iconv
-exception_get_string
-exception_get_int
-console_get_output_format
-console_set_output_format
-
-tarantool_lua_ibuf
-uuid_nil
-tt_uuid_create
-tt_uuid_str
-tt_uuid_is_equal
-tt_uuid_is_nil
-tt_uuid_bswap
-tt_uuid_from_string
-log_level
-log_format
-uri_parse
-uri_format
-PMurHash32
-PMurHash32_Process
-PMurHash32_Result
-crc32_calc
-mp_encode_double
-mp_encode_float
-mp_encode_decimal
-mp_encode_uuid
-mp_decode_double
-mp_decode_float
-mp_decode_extl
-mp_sizeof_decimal
-mp_sizeof_uuid
-decimal_unpack
-uuid_unpack
-
-log_type
-say_set_log_level
-say_logrotate
-say_set_log_format
-tarantool_uptime
-tarantool_exit
-log_pid
-space_by_id
-space_run_triggers
-space_bsize
-box_schema_version
-
-crypto_EVP_MD_CTX_new
-crypto_EVP_MD_CTX_free
-crypto_HMAC_CTX_new
-crypto_HMAC_CTX_free
-crypto_stream_new
-crypto_stream_begin
-crypto_stream_append
-crypto_stream_commit
-crypto_stream_delete
-
-lua_static_aligned_alloc
-
-swim_is_configured
-swim_cfg
-swim_set_payload
-swim_set_codec
-swim_add_member
-swim_remove_member
-swim_probe_member
-swim_broadcast
-swim_size
-swim_quit
-swim_self
-swim_member_by_uuid
-swim_member_status
-swim_iterator_open
-swim_iterator_next
-swim_iterator_close
-swim_member_uri
-swim_member_uuid
-swim_member_incarnation
-swim_member_payload
-swim_member_ref
-swim_member_unref
-swim_member_is_dropped
-swim_member_is_payload_up_to_date
-
-# Module API
-
-_say
-fiber_attr_new
-fiber_attr_delete
-fiber_attr_setstacksize
-fiber_attr_getstacksize
-fiber_self
-fiber_new
-fiber_new_ex
-fiber_yield
-fiber_start
-fiber_wakeup
-fiber_cancel
-fiber_set_cancellable
-fiber_set_joinable
-fiber_join
-fiber_sleep
-fiber_is_cancelled
-fiber_time
-fiber_time64
-fiber_reschedule
-fiber_cond_new
-fiber_cond_delete
-fiber_cond_signal
-fiber_cond_broadcast
-fiber_cond_wait_timeout
-fiber_cond_wait
-cord_slab_cache
-coio_wait
-coio_close
-coio_call
-coio_getaddrinfo
-luaL_pushcdata
-luaL_checkcdata
-luaL_setcdatagc
-luaL_ctypeid
-luaL_cdef
-luaL_pushuint64
-luaL_pushint64
-luaL_checkuint64
-luaL_checkint64
-luaL_touint64
-luaL_toint64
-luaT_checktuple
-luaT_pushtuple
-luaT_istuple
-luaT_error
-luaT_call
-luaT_cpcall
-luaT_state
-luaT_tolstring
-luaL_iscallable
-box_txn
-box_txn_begin
-box_txn_commit
-box_txn_savepoint
-box_txn_rollback
-box_txn_rollback_to_savepoint
-box_txn_alloc
-box_txn_id
-box_key_def_new
-box_key_def_delete
-box_tuple_format_default
-box_tuple_new
-box_tuple_ref
-box_tuple_unref
-box_tuple_field_count
-box_tuple_bsize
-box_tuple_to_buf
-box_tuple_format
-box_tuple_format_new
-box_tuple_format_ref
-box_tuple_format_unref
-box_tuple_field
-box_tuple_iterator
-box_tuple_iterator_free
-box_tuple_position
-box_tuple_rewind
-box_tuple_seek
-box_tuple_next
-box_tuple_update
-box_tuple_upsert
-box_tuple_extract_key
-box_tuple_compare
-box_tuple_compare_with_key
-box_return_tuple
-box_space_id_by_name
-box_index_id_by_name
-box_select
-box_insert
-box_replace
-box_delete
-box_update
-box_upsert
-box_truncate
-box_sequence_next
-box_sequence_current
-box_sequence_set
-box_sequence_reset
-box_session_push
-box_index_iterator
-box_iterator_next
-box_iterator_free
-box_index_len
-box_index_bsize
-box_index_random
-box_index_get
-box_index_min
-box_index_max
-box_index_count
-box_error_type
-box_error_code
-box_error_message
-box_error_last
-box_error_clear
-box_error_set
-error_set_prev
-box_latch_new
-box_latch_delete
-box_latch_lock
-box_latch_trylock
-box_latch_unlock
-clock_realtime
-clock_monotonic
-clock_process
-clock_thread
-clock_realtime64
-clock_monotonic64
-clock_process64
-clock_thread64
-string_strip_helper
-
-# Lua / LuaJIT
-
-lua_newstate
-lua_close
-lua_newthread
-lua_atpanic
-lua_gettop
-lua_settop
-lua_pushvalue
-lua_remove
-lua_insert
-lua_replace
-lua_checkstack
-lua_xmove
-lua_isnumber
-lua_isstring
-lua_iscfunction
-lua_isuserdata
-lua_type
-lua_typename
-lua_equal
-lua_rawequal
-lua_lessthan
-lua_tonumber
-lua_tointeger
-lua_toboolean
-lua_tolstring
-lua_objlen
-lua_tocfunction
-lua_touserdata
-lua_tothread
-lua_topointer
-lua_pushnil
-lua_pushnumber
-lua_pushinteger
-lua_pushlstring
-lua_pushstring
-lua_pushvfstring
-lua_pushfstring
-lua_pushcclosure
-lua_pushboolean
-lua_pushlightuserdata
-lua_pushthread
-lua_gettable
-lua_getfield
-lua_rawget
-lua_rawgeti
-lua_createtable
-lua_newuserdata
-lua_getmetatable
-lua_getfenv
-lua_settable
-lua_setfield
-lua_rawset
-lua_rawseti
-lua_setmetatable
-lua_setfenv
-lua_call
-lua_pcall
-lua_cpcall
-lua_load
-lua_dump
-lua_yield
-lua_resume
-lua_status
-lua_gc
-lua_error
-lua_next
-lua_concat
-lua_getallocf
-lua_setallocf
-lua_getstack
-lua_getinfo
-lua_getlocal
-lua_setlocal
-lua_getupvalue
-lua_setupvalue
-lua_sethook
-lua_gethook
-lua_gethookmask
-lua_gethookcount
-lua_upvalueid
-lua_upvaluejoin
-lua_loadx
-
-luaopen_base
-luaopen_math
-luaopen_string
-luaopen_table
-luaopen_io
-luaopen_os
-luaopen_package
-luaopen_debug
-luaopen_bit
-luaopen_jit
-luaopen_ffi
-
-luaL_openlibs
-luaL_openlib
-luaL_register
-luaL_getmetafield
-luaL_callmeta
-luaL_typerror
-luaL_argerror
-luaL_checklstring
-luaL_optlstring
-luaL_checknumber
-luaL_optnumber
-luaL_checkinteger
-luaL_optinteger
-luaL_checkstack
-luaL_checktype
-luaL_checkany
-luaL_newmetatable
-luaL_checkudata
-luaL_where
-luaL_error
-luaL_checkoption
-luaL_ref
-luaL_unref
-luaL_loadfile
-luaL_loadbuffer
-luaL_loadstring
-luaL_newstate
-luaL_gsub
-luaL_findtable
-luaL_fileresult
-luaL_execresult
-luaL_loadfilex
-luaL_loadbufferx
-luaL_traceback
-luaL_setfuncs
-luaL_pushmodule
-luaL_testudata
-luaL_setmetatable
-luaL_buffinit
-luaL_prepbuffer
-luaL_addlstring
-luaL_addstring
-luaL_addvalue
-luaL_pushresult
-
-luaJIT_setmode
-luaJIT_profile_start
-luaJIT_profile_stop
-luaJIT_profile_dumpstack
diff --git a/extra/mkexports b/extra/mkexports
deleted file mode 100755
index 145e5b8ce..000000000
--- a/extra/mkexports
+++ /dev/null
@@ -1,27 +0,0 @@
-#! /bin/sh
-# $1 - in file
-# $2 - out file
-# $3 - os
-# $4 - export templates
-if [ "x$3x" = xDarwinx ]; then
- # _func1
- # _func2
- sed -e 's/#.*//; /^[[:space:]]*$/d; s/^/_/;' $1 > $2
-else
- # {
- # func1;
- # func2;
- # };
- echo "$4"
- ( echo "{" && {
- # combine static defined list of functions
- cat $1 ;
- # with list of exported functions of bundled libraries
- for so in $4 ; do {
- # exported names from shared object
- nm -D $so ||
- # or follow patch from shared object script
- nm -D `cat $so | grep GROUP | awk '{print $3}'` ;
- } | awk '{print $3}' ; done ;
- } | sed '/^\s*$/d;s/$/;/;' && echo "};" ) > $2
-fi
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 6c756f00f..ca978aa89 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -245,56 +245,12 @@ if(BUILD_STATIC)
endif()
endif()
-# Exports syntax is toolchain-dependent, preprocessing is necessary
-set(exports_file ${CMAKE_BINARY_DIR}/extra/exports.${CMAKE_SYSTEM_NAME})
-add_custom_target(preprocess_exports
- DEPENDS ${exports_file})
-add_custom_command(
- OUTPUT ${exports_file}
- DEPENDS ${CMAKE_SOURCE_DIR}/extra/exports
- COMMAND ${CMAKE_SOURCE_DIR}/extra/mkexports
- ${CMAKE_SOURCE_DIR}/extra/exports
- ${exports_file} ${CMAKE_SYSTEM_NAME}
- ${EXPORT_LIST}
-)
-
add_executable(
- tarantool main.cc
+ tarantool main.cc exports.c
${LIBUTIL_FREEBSD_SRC}/flopen.c
${LIBUTIL_FREEBSD_SRC}/pidfile.c)
-add_dependencies(tarantool build_bundled_libs preprocess_exports sql)
-
-# Re-link if exports changed
-set_target_properties(tarantool PROPERTIES LINK_DEPENDS ${exports_file})
-
-# A note about linkers:
-# [GNU linker] When linking an *executable* visibility is ignored, and
-# either nothing is exported (default), or any non-static
-# symbol is exported (-rdynamic), or explicitly listed
-# symbols are exported (--dynamic-list).
-#
-# However, if a symbol listed lives in a static library it
-# won't be automatically pulled, hence --whole-archive
-# option.
-#
-# [Apple linker] One can provide an explicit export list; pulls symbols
-# from static libraries.
-#
-if (TARGET_OS_DARWIN)
- target_link_libraries(tarantool box ${common_libraries})
- set_target_properties(tarantool PROPERTIES
- LINK_FLAGS "-Wl,-exported_symbols_list,${exports_file}")
-else ()
- target_link_libraries(tarantool
- -Wl,--whole-archive box ${reexport_libraries}
- salad -Wl,--no-whole-archive
- ${common_libraries}
- ${generic_libraries})
- set_target_properties(tarantool PROPERTIES
- LINK_FLAGS "-Wl,--dynamic-list,${exports_file}")
- # get rid of -rdynamic
- set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "")
-endif()
+add_dependencies(tarantool build_bundled_libs sql)
+target_link_libraries(tarantool box ${common_libraries})
install (TARGETS tarantool DESTINATION bin)
diff --git a/src/exports.c b/src/exports.c
new file mode 100644
index 000000000..a3c27143e
--- /dev/null
+++ b/src/exports.c
@@ -0,0 +1,114 @@
+/*
+ * Copyright 2010-2020, Tarantool AUTHORS, please see AUTHORS file.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the
+ * following disclaimer.
+ *
+ * 2. Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+ * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
+ * <COPYRIGHT HOLDER> OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+ * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+ * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+ * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+ * THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/**
+ * The file is a hack to force the linker keep the needed symbols
+ * in the result tarantool executable file.
+ *
+ * Problem is that if a symbol is defined inside a static library,
+ * but never used in the final executable, the linker may throw it
+ * away. But many symbols are needed for Lua FFI and for the
+ * public C API used by dynamic modules.
+ *
+ * This file creates a 'false usage' of needed symbols. It stores
+ * pointers at them into a big array, and returns it like if the
+ * caller will use them somehow. Call, write by their address, or
+ * anything else.
+ *
+ * In reality the symbols are never touched after export from
+ * here, but the compiler and the linker can't tell that.
+ *
+ * Below are some alternatives, which may replace the current
+ * solution in future. Both are better in being able to declare a
+ * symbol as exported right where it is defined. Or sometimes in
+ * another place if necessary. For example, when the needed
+ * symbol is defined somewhere in a third_party library.
+ *
+ * ** Solution 1 - user-defined ELF sections. **
+ *
+ * That way is similar to what is used by the Linux kernel. To
+ * implement it there is a macros, lets call it EXPORT_SYMBOL. The
+ * macros takes one parameter - symbol name. In its implementation
+ * the macros defines a global struct keeping pointer at that
+ * symbol, and stored in a special section. For example, .tntexp
+ * section. Later when all is complied into the final executable,
+ * there is a linker script, which takes all the symbols defined
+ * in that section, and creates a reference at them, which is then
+ * somehow used in the code.
+ *
+ * A pseudocode example of how can it look in theory:
+ *
+ * struct tnt_exported_symbol {
+ * void *sym;
+ * };
+ *
+ * #define EXPORT_SYMBOL(symbol) \
+ * __attribute__((section(".tntexp")))
+ * struct tnt_exported_symbol tnt_exported_##sym = { \
+ * .sym = (void *) symbol \
+ * };
+ *
+ * For more info see EXPORT_SYMBOL() macros in
+ * include/linux/export.h file in the kernel sources.
+ *
+ * ** Solution 2 - precompile script which would find all exported
+ * functions and generate this file automatically. **
+ *
+ * Not much to explain. Introduce a macros EXPORT_SYMBOL, and walk
+ * all the source code, looking for it. When see a symbol marked
+ * so, remember it. Then generate the exports.c like it is defined
+ * below. But automatically.
+ */
+
+/**
+ * Symbol is just an address. No need to know its definition or
+ * even type to get that address. Even an integer global variable
+ * can be referenced as extern void(*)(void).
+ */
+#define EXPORT(symbol) extern void symbol(void);
+#include "exports.h"
+#undef EXPORT
+
+void **
+export_syms(void)
+{
+ /*
+ * Compiler should think the exported symbols are
+ * reachable. When they are returned as an array, the
+ * compiler can't assume anything, and can't remove them.
+ */
+ #define EXPORT(symbol) ((void *)symbol),
+ static void *symbols[] = {
+ #include "exports.h"
+ };
+ #undef EXPORT
+ return symbols;
+}
diff --git a/src/exports.h b/src/exports.h
new file mode 100644
index 000000000..580ec19d2
--- /dev/null
+++ b/src/exports.h
@@ -0,0 +1,392 @@
+/*
+ * Keep the symbols sorted by name for search and addition
+ * simplicity, to avoid duplicates. Makes no much sense to
+ * split them into any sections. Anyway some symbols will
+ * end-up belonging to several of them at once, and it
+ * would require moving lines here when a symbol goes from
+ * privately exported to the public C API.
+ */
+EXPORT(base64_bufsize)
+EXPORT(base64_decode)
+EXPORT(base64_encode)
+EXPORT(box_delete)
+EXPORT(box_error_clear)
+EXPORT(box_error_code)
+EXPORT(box_error_last)
+EXPORT(box_error_message)
+EXPORT(box_error_set)
+EXPORT(box_error_type)
+EXPORT(box_index_bsize)
+EXPORT(box_index_count)
+EXPORT(box_index_get)
+EXPORT(box_index_id_by_name)
+EXPORT(box_index_iterator)
+EXPORT(box_index_len)
+EXPORT(box_index_max)
+EXPORT(box_index_min)
+EXPORT(box_index_random)
+EXPORT(box_insert)
+EXPORT(box_iterator_free)
+EXPORT(box_iterator_next)
+EXPORT(box_key_def_delete)
+EXPORT(box_key_def_new)
+EXPORT(box_latch_delete)
+EXPORT(box_latch_lock)
+EXPORT(box_latch_new)
+EXPORT(box_latch_trylock)
+EXPORT(box_latch_unlock)
+EXPORT(box_replace)
+EXPORT(box_return_tuple)
+EXPORT(box_schema_version)
+EXPORT(box_select)
+EXPORT(box_sequence_current)
+EXPORT(box_sequence_next)
+EXPORT(box_sequence_reset)
+EXPORT(box_sequence_set)
+EXPORT(box_session_push)
+EXPORT(box_space_id_by_name)
+EXPORT(box_truncate)
+EXPORT(box_tuple_bsize)
+EXPORT(box_tuple_compare)
+EXPORT(box_tuple_compare_with_key)
+EXPORT(box_tuple_extract_key)
+EXPORT(box_tuple_field)
+EXPORT(box_tuple_field_count)
+EXPORT(box_tuple_format)
+EXPORT(box_tuple_format_default)
+EXPORT(box_tuple_format_new)
+EXPORT(box_tuple_format_ref)
+EXPORT(box_tuple_format_unref)
+EXPORT(box_tuple_iterator)
+EXPORT(box_tuple_iterator_free)
+EXPORT(box_tuple_new)
+EXPORT(box_tuple_next)
+EXPORT(box_tuple_position)
+EXPORT(box_tuple_ref)
+EXPORT(box_tuple_rewind)
+EXPORT(box_tuple_seek)
+EXPORT(box_tuple_to_buf)
+EXPORT(box_tuple_unref)
+EXPORT(box_tuple_update)
+EXPORT(box_tuple_upsert)
+EXPORT(box_txn)
+EXPORT(box_txn_alloc)
+EXPORT(box_txn_begin)
+EXPORT(box_txn_commit)
+EXPORT(box_txn_id)
+EXPORT(box_txn_rollback)
+EXPORT(box_txn_rollback_to_savepoint)
+EXPORT(box_txn_savepoint)
+EXPORT(box_update)
+EXPORT(box_upsert)
+EXPORT(clock_monotonic64)
+EXPORT(clock_monotonic)
+EXPORT(clock_process64)
+EXPORT(clock_process)
+EXPORT(clock_realtime64)
+EXPORT(clock_realtime)
+EXPORT(clock_thread64)
+EXPORT(clock_thread)
+EXPORT(coio_call)
+EXPORT(coio_close)
+EXPORT(coio_getaddrinfo)
+EXPORT(coio_wait)
+EXPORT(console_get_output_format)
+EXPORT(console_set_output_format)
+EXPORT(cord_slab_cache)
+EXPORT(crc32_calc)
+EXPORT(crypto_EVP_MD_CTX_free)
+EXPORT(crypto_EVP_MD_CTX_new)
+EXPORT(crypto_HMAC_CTX_free)
+EXPORT(crypto_HMAC_CTX_new)
+EXPORT(crypto_stream_append)
+EXPORT(crypto_stream_begin)
+EXPORT(crypto_stream_commit)
+EXPORT(crypto_stream_delete)
+EXPORT(crypto_stream_new)
+EXPORT(csv_create)
+EXPORT(csv_destroy)
+EXPORT(csv_escape_field)
+EXPORT(csv_feed)
+EXPORT(csv_iterator_create)
+EXPORT(csv_next)
+EXPORT(csv_setopt)
+EXPORT(decimal_unpack)
+EXPORT(error_set_prev)
+EXPORT(exception_get_int)
+EXPORT(exception_get_string)
+EXPORT(fiber_attr_delete)
+EXPORT(fiber_attr_getstacksize)
+EXPORT(fiber_attr_new)
+EXPORT(fiber_attr_setstacksize)
+EXPORT(fiber_cancel)
+EXPORT(fiber_clock64)
+EXPORT(fiber_clock)
+EXPORT(fiber_cond_broadcast)
+EXPORT(fiber_cond_delete)
+EXPORT(fiber_cond_new)
+EXPORT(fiber_cond_signal)
+EXPORT(fiber_cond_wait)
+EXPORT(fiber_cond_wait_timeout)
+EXPORT(fiber_is_cancelled)
+EXPORT(fiber_join)
+EXPORT(fiber_new)
+EXPORT(fiber_new_ex)
+EXPORT(fiber_reschedule)
+EXPORT(fiber_self)
+EXPORT(fiber_set_cancellable)
+EXPORT(fiber_set_joinable)
+EXPORT(fiber_sleep)
+EXPORT(fiber_start)
+EXPORT(fiber_time64)
+EXPORT(fiber_time)
+EXPORT(fiber_wakeup)
+EXPORT(fiber_yield)
+EXPORT(guava)
+EXPORT(ibuf_create)
+EXPORT(ibuf_destroy)
+EXPORT(ibuf_reinit)
+EXPORT(ibuf_reserve_slow)
+EXPORT(lbox_socket_local_resolve)
+EXPORT(lbox_socket_nonblock)
+EXPORT(log_format)
+EXPORT(log_level)
+EXPORT(log_pid)
+EXPORT(log_type)
+EXPORT(lua_atpanic)
+EXPORT(lua_call)
+EXPORT(lua_checkstack)
+EXPORT(lua_close)
+EXPORT(lua_concat)
+EXPORT(lua_cpcall)
+EXPORT(lua_createtable)
+EXPORT(lua_dump)
+EXPORT(lua_equal)
+EXPORT(lua_error)
+EXPORT(lua_gc)
+EXPORT(lua_getallocf)
+EXPORT(lua_getfenv)
+EXPORT(lua_getfield)
+EXPORT(lua_gethook)
+EXPORT(lua_gethookcount)
+EXPORT(lua_gethookmask)
+EXPORT(lua_getinfo)
+EXPORT(lua_getlocal)
+EXPORT(lua_getmetatable)
+EXPORT(lua_getstack)
+EXPORT(lua_gettable)
+EXPORT(lua_gettop)
+EXPORT(lua_getupvalue)
+EXPORT(lua_insert)
+EXPORT(lua_iscfunction)
+EXPORT(lua_isnumber)
+EXPORT(lua_isstring)
+EXPORT(lua_isuserdata)
+EXPORT(lua_lessthan)
+EXPORT(lua_load)
+EXPORT(lua_loadx)
+EXPORT(lua_newstate)
+EXPORT(lua_newthread)
+EXPORT(lua_newuserdata)
+EXPORT(lua_next)
+EXPORT(lua_objlen)
+EXPORT(lua_pcall)
+EXPORT(lua_pushboolean)
+EXPORT(lua_pushcclosure)
+EXPORT(lua_pushfstring)
+EXPORT(lua_pushinteger)
+EXPORT(lua_pushlightuserdata)
+EXPORT(lua_pushlstring)
+EXPORT(lua_pushnil)
+EXPORT(lua_pushnumber)
+EXPORT(lua_pushstring)
+EXPORT(lua_pushthread)
+EXPORT(lua_pushvalue)
+EXPORT(lua_pushvfstring)
+EXPORT(lua_rawequal)
+EXPORT(lua_rawget)
+EXPORT(lua_rawgeti)
+EXPORT(lua_rawset)
+EXPORT(lua_rawseti)
+EXPORT(lua_remove)
+EXPORT(lua_replace)
+EXPORT(lua_resume)
+EXPORT(lua_setallocf)
+EXPORT(lua_setfenv)
+EXPORT(lua_setfield)
+EXPORT(lua_sethook)
+EXPORT(lua_setlocal)
+EXPORT(lua_setmetatable)
+EXPORT(lua_settable)
+EXPORT(lua_settop)
+EXPORT(lua_setupvalue)
+EXPORT(lua_static_aligned_alloc)
+EXPORT(lua_status)
+EXPORT(lua_toboolean)
+EXPORT(lua_tocfunction)
+EXPORT(lua_tointeger)
+EXPORT(lua_tolstring)
+EXPORT(lua_tonumber)
+EXPORT(lua_topointer)
+EXPORT(lua_tothread)
+EXPORT(lua_touserdata)
+EXPORT(lua_type)
+EXPORT(lua_typename)
+EXPORT(lua_upvalueid)
+EXPORT(lua_upvaluejoin)
+EXPORT(lua_xmove)
+EXPORT(lua_yield)
+EXPORT(luaJIT_profile_dumpstack)
+EXPORT(luaJIT_profile_start)
+EXPORT(luaJIT_profile_stop)
+EXPORT(luaJIT_setmode)
+EXPORT(luaL_addlstring)
+EXPORT(luaL_addstring)
+EXPORT(luaL_addvalue)
+EXPORT(luaL_argerror)
+EXPORT(luaL_buffinit)
+EXPORT(luaL_callmeta)
+EXPORT(luaL_cdef)
+EXPORT(luaL_checkany)
+EXPORT(luaL_checkcdata)
+EXPORT(luaL_checkint64)
+EXPORT(luaL_checkinteger)
+EXPORT(luaL_checklstring)
+EXPORT(luaL_checknumber)
+EXPORT(luaL_checkoption)
+EXPORT(luaL_checkstack)
+EXPORT(luaL_checktype)
+EXPORT(luaL_checkudata)
+EXPORT(luaL_checkuint64)
+EXPORT(luaL_ctypeid)
+EXPORT(luaL_error)
+EXPORT(luaL_execresult)
+EXPORT(luaL_fileresult)
+EXPORT(luaL_findtable)
+EXPORT(luaL_getmetafield)
+EXPORT(luaL_gsub)
+EXPORT(luaL_iscallable)
+EXPORT(luaL_loadbuffer)
+EXPORT(luaL_loadbufferx)
+EXPORT(luaL_loadfile)
+EXPORT(luaL_loadfilex)
+EXPORT(luaL_loadstring)
+EXPORT(luaL_newmetatable)
+EXPORT(luaL_newstate)
+EXPORT(luaL_openlib)
+EXPORT(luaL_openlibs)
+EXPORT(luaL_optinteger)
+EXPORT(luaL_optlstring)
+EXPORT(luaL_optnumber)
+EXPORT(luaL_prepbuffer)
+EXPORT(luaL_pushcdata)
+EXPORT(luaL_pushint64)
+EXPORT(luaL_pushmodule)
+EXPORT(luaL_pushresult)
+EXPORT(luaL_pushuint64)
+EXPORT(luaL_ref)
+EXPORT(luaL_register)
+EXPORT(luaL_setcdatagc)
+EXPORT(luaL_setfuncs)
+EXPORT(luaL_setmetatable)
+EXPORT(luaL_testudata)
+EXPORT(luaL_toint64)
+EXPORT(luaL_touint64)
+EXPORT(luaL_traceback)
+EXPORT(luaL_typerror)
+EXPORT(luaL_unref)
+EXPORT(luaL_where)
+EXPORT(luaopen_base)
+EXPORT(luaopen_bit)
+EXPORT(luaopen_debug)
+EXPORT(luaopen_ffi)
+EXPORT(luaopen_io)
+EXPORT(luaopen_jit)
+EXPORT(luaopen_math)
+EXPORT(luaopen_os)
+EXPORT(luaopen_package)
+EXPORT(luaopen_string)
+EXPORT(luaopen_table)
+EXPORT(luaT_call)
+EXPORT(luaT_checktuple)
+EXPORT(luaT_cpcall)
+EXPORT(luaT_error)
+EXPORT(luaT_istuple)
+EXPORT(luaT_pushtuple)
+EXPORT(luaT_state)
+EXPORT(luaT_tolstring)
+EXPORT(mp_decode_double)
+EXPORT(mp_decode_extl)
+EXPORT(mp_decode_float)
+EXPORT(mp_encode_decimal)
+EXPORT(mp_encode_double)
+EXPORT(mp_encode_float)
+EXPORT(mp_encode_uuid)
+EXPORT(mp_sizeof_decimal)
+EXPORT(mp_sizeof_uuid)
+EXPORT(password_prepare)
+EXPORT(PMurHash32)
+EXPORT(PMurHash32_Process)
+EXPORT(PMurHash32_Result)
+EXPORT(port_destroy)
+EXPORT(random_bytes)
+EXPORT(_say)
+EXPORT(say_logrotate)
+EXPORT(say_set_log_format)
+EXPORT(say_set_log_level)
+EXPORT(SHA1internal)
+EXPORT(space_bsize)
+EXPORT(space_by_id)
+EXPORT(space_run_triggers)
+EXPORT(string_strip_helper)
+EXPORT(swim_add_member)
+EXPORT(swim_broadcast)
+EXPORT(swim_cfg)
+EXPORT(swim_is_configured)
+EXPORT(swim_iterator_close)
+EXPORT(swim_iterator_next)
+EXPORT(swim_iterator_open)
+EXPORT(swim_member_by_uuid)
+EXPORT(swim_member_incarnation)
+EXPORT(swim_member_is_dropped)
+EXPORT(swim_member_is_payload_up_to_date)
+EXPORT(swim_member_payload)
+EXPORT(swim_member_ref)
+EXPORT(swim_member_status)
+EXPORT(swim_member_unref)
+EXPORT(swim_member_uri)
+EXPORT(swim_member_uuid)
+EXPORT(swim_probe_member)
+EXPORT(swim_quit)
+EXPORT(swim_remove_member)
+EXPORT(swim_self)
+EXPORT(swim_set_codec)
+EXPORT(swim_set_payload)
+EXPORT(swim_size)
+EXPORT(tarantool_exit)
+EXPORT(tarantool_lua_ibuf)
+EXPORT(tarantool_lua_slab_cache)
+EXPORT(tarantool_uptime)
+EXPORT(title_get)
+EXPORT(title_get_custom)
+EXPORT(title_get_interpretor_name)
+EXPORT(title_get_script_name)
+EXPORT(title_get_status)
+EXPORT(title_set_custom)
+EXPORT(title_set_interpretor_name)
+EXPORT(title_set_script_name)
+EXPORT(title_set_status)
+EXPORT(title_update)
+EXPORT(tnt_iconv)
+EXPORT(tnt_iconv_close)
+EXPORT(tnt_iconv_open)
+EXPORT(tt_uuid_bswap)
+EXPORT(tt_uuid_create)
+EXPORT(tt_uuid_from_string)
+EXPORT(tt_uuid_is_equal)
+EXPORT(tt_uuid_is_nil)
+EXPORT(tt_uuid_str)
+EXPORT(uri_format)
+EXPORT(uri_parse)
+EXPORT(uuid_nil)
+EXPORT(uuid_unpack)
diff --git a/src/main.cc b/src/main.cc
index bb0794dfe..8321acc02 100644
--- a/src/main.cc
+++ b/src/main.cc
@@ -711,6 +711,9 @@ break_loop(struct trigger *, void *)
return 0;
}
+extern "C" void **
+export_syms(void);
+
int
main(int argc, char **argv)
{
@@ -808,6 +811,8 @@ main(int argc, char **argv)
title_set_script_name(argv[0]);
}
+ export_syms();
+
random_init();
crc32_init();
diff --git a/test/box/gh-2971-symbol-visibility.result b/test/box/gh-2971-symbol-visibility.result
new file mode 100644
index 000000000..9f91a4edf
--- /dev/null
+++ b/test/box/gh-2971-symbol-visibility.result
@@ -0,0 +1,22 @@
+-- test-run result file version 2
+ffi = require('ffi')
+ | ---
+ | ...
+
+--
+-- gh-2971: Tarantool should not hide symbols. Even those which
+-- are not a part of the public API.
+--
+
+-- This symbol is not public, but should be defined.
+ffi.cdef[[ \
+bool \
+box_is_configured(void); \
+]]
+ | ---
+ | ...
+
+ffi.C.box_is_configured()
+ | ---
+ | - true
+ | ...
diff --git a/test/box/gh-2971-symbol-visibility.test.lua b/test/box/gh-2971-symbol-visibility.test.lua
new file mode 100644
index 000000000..f19effeba
--- /dev/null
+++ b/test/box/gh-2971-symbol-visibility.test.lua
@@ -0,0 +1,14 @@
+ffi = require('ffi')
+
+--
+-- gh-2971: Tarantool should not hide symbols. Even those which
+-- are not a part of the public API.
+--
+
+-- This symbol is not public, but should be defined.
+ffi.cdef[[ \
+bool \
+box_is_configured(void); \
+]]
+
+ffi.C.box_is_configured()
More information about the Tarantool-patches
mailing list