[Tarantool-patches] [PATCH 01/11] cmake: enable misc types of UB detection in clang

[Vladislav Shpilevoy <v.shpilevoy@tarantool.org>]

Option ENABLE_UB_SANITIZER enables clang undefined behaviour
sanitizer. So far the only UB to detect was alignment violation.
This was the biggest problem found by the sanitizer. Now when it
is fixed, most of the other types of UB are also turned on to fix
them as well.

There is a few of exceptions - pointer type overflow, vptr check,
and all types of integer overflow and truncation.

Pointer type overflow detection is disabled because it is abused
in the source code a lot, by stailq data structure.

Vptr sanitation is a runtime check ensuring that a pointer at a
non-POD type really points at an object of this type, using RTTI.
The check false-positively fails in alter.cc when AlterSpaceOp
class objects are stored in an rlist, and the list is iterated
using rlist_foreach_entry(). In the cycle there is a condition:

    &item->member != head

In the end the 'item' points not at an AlterSpaceOp, but at the
rlist head - offsetof(typeof(item), member), at an rlist
structure. Despite 'item' is never dereferenced, clang anyway
generates vptr check here, which of course fails. Note,
'&item->member' does not dereference item. It is
item + offsetof(typeof(item), member). Just another address a few
bytes after item.

Integer overflow and truncation are disabled because SQL uses
int64_t variables as a container of values of range [INT64_MIN,
UINT64_MAX]. This works because there is a flag 'is_neg' near
each such value which tells how to interpret it - as negative
int64_t, or as positive uint64_t. As a result, some operations
lead to a false-positive overflow. For example, consider
expr_code_int() function. It essentially can do this:

    int64_t value;
    ((uint64_t *)&value) = 9223372036854775808;
    value = -value;

9223372036854775808 is -INT64_MIN. It can't be stored in int64_t.
But the thing is that (uint64_t)9223372036854775808 is stored
exactly like (int64_t)INT64_MIN, in binary form. So the expression
"value = -value" looks perfectly valid:
"value = -9223372036854775808", But in fact it is interpreted as
"value = -(-9223372036854775808)".

These integer overflow/truncation problems are going to be fixed
in a separate commit due to big amount of changes needed for that.

Part of #4609
---
 cmake/compiler.cmake | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/cmake/compiler.cmake b/cmake/compiler.cmake
index 4b23d6631..6c0fa635c 100644
--- a/cmake/compiler.cmake
+++ b/cmake/compiler.cmake
@@ -269,7 +269,21 @@ macro(enable_tnt_compile_flags)
         if (NOT CMAKE_COMPILER_IS_CLANG)
             message(FATAL_ERROR "Undefined behaviour sanitizer only available for clang")
         else()
-            add_compile_flags("C;CXX" "-fsanitize=alignment -fno-sanitize-recover=alignment")
+            set(SANITIZE_FLAGS "-fsanitize=undefined -fno-sanitize-recover=undefined")
+            # Stailq data structure subtracts a positive value from NULL.
+            set(SANITIZE_FLAGS ${SANITIZE_FLAGS} -fno-sanitize=pointer-overflow)
+            # Intrusive data structures may abuse '&obj->member' on pointer
+            # 'obj' which is not really a pointer at an object of its type.
+            # For example, rlist uses '&item->member' expression in macro cycles
+            # to check end of cycle, but on the last iteration 'item' points at
+            # the list metadata head, not at an object of type stored in this
+            # list.
+            set(SANITIZE_FLAGS ${SANITIZE_FLAGS} -fno-sanitize=vptr)
+            # Integer overflow and truncation are disabled due to extensive
+            # usage of this UB in SQL code to 'implement' some kind of int65_t.
+            set(SANITIZE_FLAGS ${SANITIZE_FLAGS} -fno-sanitize=implicit-signed-integer-truncation -fno-sanitize=implicit-integer-sign-change -fno-sanitize=signed-integer-overflow)
+
+            add_compile_flags("C;CXX" "${SANITIZE_FLAGS}")
         endif()
     endif()
 
-- 
2.21.1 (Apple Git-122.3)