Tarantool development patches archive
 help / color / mirror / Atom feed
* [Tarantool-patches] [PATCH luajit v2 0/5] Add ASan support in LuaJIT
@ 2023-07-21  8:12 Igor Munkin via Tarantool-patches
  2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 1/5] ci: clean up workflow for exotic builds Igor Munkin via Tarantool-patches
                   ` (5 more replies)
  0 siblings, 6 replies; 25+ messages in thread
From: Igor Munkin via Tarantool-patches @ 2023-07-21  8:12 UTC (permalink / raw)
  To: Sergey Kaplun, Sergey Bronnikov; +Cc: tarantool-patches

This series implements the second attempt to enable ASan support back
for LuaJIT repository.

The only thing that was missing the whole time was the option per se
(and the corresponding compile flags, obviously). Anyway, when the
option has been finally added, the dedicated CI workflow has been added
too in scope of the second patch in the series. All other info can be
found in the corresponding patches.

Besided, internal LuaJIT memory allocator is not instrumented yet
unfortunately, so to find any memory faults it's better to build LuaJIT
with system provided memory allocator (i.e. run CMake configuration
phase with -DLUAJIT_USE_SYSMALLOC=ON). However, LUAJIT_USE_SYSMALLOC
cannot be enabled on x64 without GC64, since realloc usually doesn't
return addresses in the right address range. For more info, see root
CMakeLists.txt.

Surprisingly, some hidden bugs were found while testing the
aforementioned configuration.

1. The assertions in memprof initialization, checking the state of the
allocator against NULL, can fail if this allocator requires no internal
state (e.g. glibc functions for allocating dynamic memory). In fact,
when building LuaJIT with LUAJIT_USE_SYSMALLOC option enabled, NULL is
given as the second parameter to <lua_newstate> and these assertions
fail as a result. Hence, they are simply removed.

2. Before the patch all tests in tarantool-tests suite (except
<lj-603-err-snap-restore.test.lua>) terminate their execution via
<os.exit> with the status depending on the test results. However, the
second argument of <os.exit> was omitted and Lua universe was not
properly finalized as a result. This behaviour becomes a problem, when
LuaJIT is build with LUAJIT_USE_SYSMALLOC option and AddressSanitizer
support enabled, since the sanitizer starts reporting false positive
errors about the memory allocations without the corresponding memory
releases. To resolve these errors, the second parameter to each
<os.exit> call terminating the test has to be added. To avoid loss of
the aforementioned parameter in future, <test:done> helper has been
added to the TAP module. Depending on the single parameter, the new
helper either properly finalize the test being run, or simply checks all
the test assertions and raises an error if any of them fail. The latter
case is added especially to handle <lj-603-err-snap-restore.test.lua>
specifics and still check that everything works fine.


Last but not least: for all ARM64 jobs in exotic builds testing pipeline
non-GC64 configurations were disabled, since LUAJIT_ENABLE_GC64 takes no
effect for this arch (GC64 is the only option).

Issue: https://github.com/tarantool/tarantool/issues/5878
Branch: https://github.com/tarantool/luajit/tree/hackaton/gh-5878-enable-ASAN
Tarantool related changes and CI can be found in #8846[1].

v1: https://lists.tarantool.org/tarantool-patches/cover.1689195028.git.imun@tarantool.org/T/#t

Changes in v2:
  * Fixed comments as per review by Sergey B. and Sergey K
  * Enabled LUAJIT_USE_SYSMALLOC option in sanitizers-testing.yml
  * Removed two invalid assertions in memprof sources (found via
    enabling LUAJIT_USE_SYSMALLOC option)
  * Introduced test:done helper for proper test finalization (found via
    enabling both LUAJIT_USE_ASAN and LUAJIT_USE_SYSMALLOC)
  * Little maintenance of exotic builds workflow

Igor Munkin (5):
  ci: clean up workflow for exotic builds
  memprof: remove invalid assertions
  test: introduce test:done TAP helper
  build: introduce LUAJIT_USE_ASAN option
  ci: introduce testing workflow with sanitizers

 .github/actions/setup-sanitizers/README.md    | 12 +++
 .github/actions/setup-sanitizers/action.yml   | 24 +++++
 .github/workflows/exotic-builds-testing.yml   |  4 +-
 .github/workflows/sanitizers-testing.yml      | 89 +++++++++++++++++++
 CMakeLists.txt                                | 33 +++++--
 src/lj_memprof.c                              |  2 -
 .../arm64-ccall-fp-convention.test.lua        |  2 +-
 .../bc-jit-unpatching.test.lua                |  2 +-
 .../c-library-path-length.test.lua            |  2 +-
 .../fix-bit-shift-generation.test.lua         |  2 +-
 .../tarantool-tests/fix-cdata-concat.test.lua |  2 +-
 test/tarantool-tests/fix-emit-rma.test.lua    |  2 +-
 .../fix-fold-simplify-conv-sext.test.lua      |  2 +-
 .../fix-gc-setupvalue.test.lua                |  2 +-
 .../fix-slot-check-for-mm-record.test.lua     |  2 +-
 .../gh-3196-incorrect-string-length.test.lua  |  2 +-
 .../gh-4199-gc64-fuse.test.lua                |  2 +-
 .../gh-4427-ffi-sandwich.test.lua             |  2 +-
 ...gh-4476-fix-string-find-recording.test.lua |  2 +-
 ...gh-4773-tonumber-fail-on-NUL-char.test.lua |  2 +-
 .../gh-5813-resolving-of-c-symbols.test.lua   |  2 +-
 .../gh-6065-jit-library-smoke-tests.test.lua  |  2 +-
 ...-missed-carg1-in-bctsetr-fallback.test.lua |  2 +-
 ...-6096-external-unwinding-on-arm64.test.lua |  2 +-
 ...8-fix-side-exit-patching-on-arm64.test.lua |  2 +-
 test/tarantool-tests/gh-6163-min-max.test.lua |  2 +-
 test/tarantool-tests/gh-6189-cur_L.test.lua   |  2 +-
 ...ytecode-allocator-for-comparisons.test.lua |  2 +-
 .../gh-6371-string-char-no-arg.test.lua       |  2 +-
 ...6782-stitching-in-vmevent-handler.test.lua |  2 +-
 .../gh-6976-narrowing-of-unary-minus.test.lua |  2 +-
 ...4-add-proto-trace-sysprof-default.test.lua |  2 +-
 .../gh-7745-oom-on-trace.test.lua             |  2 +-
 .../lj-350-sload-typecheck.test.lua           |  2 +-
 .../lj-351-print-tostring-number.test.lua     |  2 +-
 .../lj-356-ir-khash-non-string-obj.test.lua   |  2 +-
 .../lj-357-arm64-hrefk.test.lua               |  2 +-
 .../lj-366-strtab-correct-size.test.lua       |  2 +-
 .../lj-375-ir-bufput-signed-char.test.lua     |  2 +-
 .../lj-378-string-format-c-null-char.test.lua |  2 +-
 .../lj-408-tonumber-cdata-record.test.lua     |  2 +-
 .../lj-416-xor-before-jcc.test.lua            |  2 +-
 .../lj-418-assert-any-type.test.lua           |  2 +-
 ...6-arm64-incorrect-check-closed-uv.test.lua |  2 +-
 .../lj-430-maxirconst.test.lua                |  2 +-
 ...-arm64-constant-rematerialization.test.lua |  2 +-
 ...-445-fix-memory-probing-allocator.test.lua |  2 +-
 .../lj-463-os-date-oom.test.lua               |  2 +-
 .../lj-494-table-chain-infinite-loop.test.lua |  2 +-
 ...lj-505-fold-no-strref-for-ptrdiff.test.lua |  2 +-
 .../lj-512-profiler-hook-finalizers.test.lua  |  2 +-
 .../lj-524-fold-conv-respect-src-irt.test.lua |  2 +-
 .../lj-528-tonumber-0.test.lua                |  2 +-
 .../lj-556-fix-loop-realignment.test.lua      |  2 +-
 .../lj-574-overflow-unpack.test.lua           |  2 +-
 ...j-584-bad-renames-for-sunk-values.test.lua |  2 +-
 .../lj-586-debug-non-string-error.test.lua    |  2 +-
 .../lj-601-fix-gc-finderrfunc.test.lua        |  2 +-
 .../lj-603-err-snap-restore.test.lua          | 15 ++--
 .../lj-671-arm64-assert-after-mremap.test.lua |  2 +-
 ...lj-672-cdata-allocation-recording.test.lua |  2 +-
 .../lj-688-snap-ir-rename.test.lua            |  2 +-
 .../lj-695-ffi-vararg-call.test.lua           |  2 +-
 .../lj-698-arm-pcall-panic.test.lua           |  2 +-
 .../lj-704-bc-varg-use-def.test.lua           |  2 +-
 .../lj-726-profile-flush-close.test.lua       |  2 +-
 .../lj-727-lightuserdata-itern.test.lua       |  2 +-
 .../lj-735-io-close-on-closed-file.test.lua   |  2 +-
 .../lj-762-pcall-no-arg.test.lua              |  2 +-
 .../lj-819-fix-missing-uclo.test.lua          |  2 +-
 .../lj-864-varg-rec-base-offset.test.lua      |  2 +-
 .../lj-906-fix-err-mem.test.lua               |  2 +-
 .../tarantool-tests/lj-981-folding-0.test.lua |  2 +-
 ...instable-types-during-loop-unroll.test.lua |  2 +-
 .../lj-flush-on-trace.test.lua                |  2 +-
 test/tarantool-tests/math-modf.test.lua       |  2 +
 .../misclib-getmetrics-lapi.test.lua          |  2 +-
 .../misclib-memprof-lapi.test.lua             |  2 +-
 .../misclib-sysprof-lapi.test.lua             |  2 +-
 .../or-144-gc64-asmref-l.test.lua             |  2 +-
 .../or-232-unsink-64-kptr.test.lua            |  2 +-
 .../or-94-arm64-ir-ahuvload-bool.test.lua     |  2 +-
 test/tarantool-tests/tap.lua                  | 14 ++-
 ...onumber-negative-non-decimal-base.test.lua |  2 +-
 test/tarantool-tests/tools-utils-avl.test.lua |  2 +-
 test/tarantool-tests/unit-jit-parse.test.lua  |  2 +-
 86 files changed, 252 insertions(+), 97 deletions(-)
 create mode 100644 .github/actions/setup-sanitizers/README.md
 create mode 100644 .github/actions/setup-sanitizers/action.yml
 create mode 100644 .github/workflows/sanitizers-testing.yml

[1]: https://github.com/tarantool/tarantool/pull/8846

-- 
2.30.2


^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2023-08-03  7:51 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-21  8:12 [Tarantool-patches] [PATCH luajit v2 0/5] Add ASan support in LuaJIT Igor Munkin via Tarantool-patches
2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 1/5] ci: clean up workflow for exotic builds Igor Munkin via Tarantool-patches
2023-07-24 10:36   ` Sergey Bronnikov via Tarantool-patches
2023-07-26  9:37   ` Sergey Kaplun via Tarantool-patches
2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 2/5] memprof: remove invalid assertions Igor Munkin via Tarantool-patches
2023-07-24 10:46   ` Sergey Bronnikov via Tarantool-patches
2023-07-26  9:41   ` Sergey Kaplun via Tarantool-patches
2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 3/5] test: introduce test:done TAP helper Igor Munkin via Tarantool-patches
2023-07-24 10:53   ` Sergey Bronnikov via Tarantool-patches
2023-07-26 10:43   ` Sergey Kaplun via Tarantool-patches
2023-07-26 12:42     ` Igor Munkin via Tarantool-patches
2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 4/5] build: introduce LUAJIT_USE_ASAN option Igor Munkin via Tarantool-patches
2023-07-24 11:41   ` Sergey Bronnikov via Tarantool-patches
2023-07-26 13:06     ` Igor Munkin via Tarantool-patches
2023-07-25 12:26   ` Sergey Bronnikov via Tarantool-patches
2023-07-26 12:54     ` Igor Munkin via Tarantool-patches
2023-07-27 11:06       ` Sergey Bronnikov via Tarantool-patches
2023-07-26 11:03   ` Sergey Kaplun via Tarantool-patches
2023-07-21  8:12 ` [Tarantool-patches] [PATCH luajit v2 5/5] ci: introduce testing workflow with sanitizers Igor Munkin via Tarantool-patches
2023-07-24 11:54   ` Sergey Bronnikov via Tarantool-patches
2023-07-26 14:53     ` Igor Munkin via Tarantool-patches
2023-07-27 11:13       ` Sergey Bronnikov via Tarantool-patches
2023-07-26 11:29   ` Sergey Kaplun via Tarantool-patches
2023-07-26 16:35     ` Igor Munkin via Tarantool-patches
2023-08-03  7:31 ` [Tarantool-patches] [PATCH luajit v2 0/5] Add ASan support in LuaJIT Igor Munkin via Tarantool-patches

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox