From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from [87.239.111.99] (localhost [127.0.0.1]) by dev.tarantool.org (Postfix) with ESMTP id ADCFC66C07F; Mon, 23 Oct 2023 12:49:09 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org ADCFC66C07F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarantool.org; s=dev; t=1698054549; bh=VUUj1uMKuINaHhtQ/Uj1gno1pZ1m5fTXpqKl3T5He4E=; h=To:Date:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=xP7GYTjWIhp8Q/awmNNBFO5ffhH4fdr9M4TFh5sltjwSaD6+ywuvyuGnI52OJjznZ c8CeFnVod9b4W85wnaylgslrwrxATRDE0Sokw8kehiqdIj69pvQVrmfN1yJqg5HfY+ RK8gDEMRHsSTNXpS03RnW/eGsmteZEwQsx8eeMhw= Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dev.tarantool.org (Postfix) with ESMTPS id 738BD66C07F for ; Mon, 23 Oct 2023 12:49:08 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 dev.tarantool.org 738BD66C07F Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-53e2308198eso4704785a12.1 for ; Mon, 23 Oct 2023 02:49:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698054547; x=1698659347; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wBBSANs4W69GchyPz4+E5ulu6hXmv2qDo74VFRwae0Q=; b=wGXwekfqw6PkDiukgT7eFHjtdnrj23qAVUd+Zi8XifrlqDnEHnhVUfOJX+Wca7E7qe eGMgQniOrYAp22D0f+JJ+HFDMcOOWd+jL3HwMQjNByBHDueYJ61RBz1FpDbsh0HMKT79 gXU1W4JDi/Oe1AgdHjwyrkc/u84LpzZ3qUABTIm+K3mLcO5AM68aLS96V5uBqUpC+9tf C3fgwfE3tbldIMpy4gCdjzcZsY0J2YlNRUze2PywfNiT8k0KVt9WSSHVSIVE7vBOkB5X MquE1MYYMa18zQS7SSukakiQ6KSVfN4fLhZgt3fkwjEDJVwiJPld43RG740V4Wof70uk geAw== X-Gm-Message-State: AOJu0Yywpo/59UWHENXRaR40ZVi9/yt75f7qANK4FP995pSfMRv12nQp +Wz9sduwde0Qogd7ewhhju3VLvigTht+qg== X-Google-Smtp-Source: AGHT+IFQA3D9BT+6CKbIk6VLm0/2cc/EC6lXuREx2gxdOVIcl++cVBdDq3CCx4GxCgXPo73vvDGO4Q== X-Received: by 2002:a50:9fe3:0:b0:53e:158b:586d with SMTP id c90-20020a509fe3000000b0053e158b586dmr5067444edf.21.1698054547032; Mon, 23 Oct 2023 02:49:07 -0700 (PDT) Received: from pony.. ([185.6.247.97]) by smtp.gmail.com with ESMTPSA id e13-20020a50ec8d000000b0053e775e428csm6008052edr.83.2023.10.23.02.49.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 02:49:06 -0700 (PDT) To: tarantool-patches@dev.tarantool.org, Sergey Kaplun , max.kokryashkin@gmail.com Date: Mon, 23 Oct 2023 12:48:56 +0300 Message-Id: <9b4befc22674bcc924ad74313d968d1748b85bd9.1698053812.git.sergeyb@tarantool.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Tarantool-patches] [PATCH luajit][v2] cmake: replace prove with CTest X-BeenThere: tarantool-patches@dev.tarantool.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Tarantool development patches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Sergey Bronnikov via Tarantool-patches Reply-To: Sergey Bronnikov Errors-To: tarantool-patches-bounces@dev.tarantool.org Sender: "Tarantool-patches" From: Sergey Bronnikov The patch replaces the main test runner prove(1) with CTest, see [1] and [2]. The benefits are: - less external dependencies, prove(1) is gone - running tests by test title - extended test running options in comparison to prove(1) - unified test output (finally!) Note, it is not possible to attach targets to the automatically generated `test` target. It means that target `test` now is executing `LuaJIT-test`, but not `LuaJIT-lint`. Note, the testsuites in `test/LuaJIT-tests` and in `test/PUC-Rio-Lua-5.1` directories have their own test runners and currently CTest runs these testsuites as a single tests. In a future we could change these test runners to specify test from outside and after this we could run tests separately by CTest in these test suites. Note, it is not possible to add dependencies to `add_test()` in CMake, see [3]. It means that all object files and executables must be build before running `ctest(1)`. Examples of using CTest in a build directory: $ ctest # Running all tests. $ ctest -L tarantool # Running tests in tarantool-tests dir. $ ctest -L tarantool-c # Running tests in tarantool-c-tests dir. $ ctest -L lua-Harness # Running tests in lua-Harness dir. $ ctest -L luajit # Running tests in LuaJIT dir. $ ctest -L lua # Running tests in PUC-Rio-Lua-5.1 dir. $ ctest --rerun-fail $ ctest --output-on-failure 1. https://cmake.org/cmake/help/latest/manual/ctest.1.html 2. https://cmake.org/cmake/help/book/mastering-cmake/chapter/Testing%20With%20CMake%20and%20CTest.html 3. https://gitlab.kitware.com/cmake/cmake/-/issues/8774 --- Changes v2: - rebased to master - fixed tests that require loading object files - fixed backward compatibility with CMake targets PR: https://github.com/tarantool/tarantool/pull/9286 Branch: https://github.com/tarantool/luajit/tree/ligurio/enable_test_target Patch v1 was sent as RFC without list address in CC. CMakeLists.txt | 1 + test/CMakeLists.txt | 28 ++++-------- test/LuaJIT-tests/CMakeLists.txt | 17 +++++-- test/PUC-Rio-Lua-5.1-tests/CMakeLists.txt | 23 ++++++---- test/lua-Harness-tests/CMakeLists.txt | 42 +++++++++--------- test/tarantool-c-tests/CMakeLists.txt | 49 ++++++++++---------- test/tarantool-tests/CMakeLists.txt | 54 ++++++++--------------- 7 files changed, 100 insertions(+), 114 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index eebf3d6f..1943d894 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -356,6 +356,7 @@ set(LUAJIT_TEST_INIT "${PROJECT_SOURCE_DIR}/test/luajit-test-init.lua" CACHE STR "Lua code need to be run before tests are started." ) +enable_testing() add_subdirectory(test) # --- Misc rules --------------------------------------------------------------- diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt index 58cba5ba..bc0c8dd8 100644 --- a/test/CMakeLists.txt +++ b/test/CMakeLists.txt @@ -73,6 +73,15 @@ add_custom_target(${PROJECT_NAME}-lint DEPENDS set(LUAJIT_TEST_COMMAND "${LUAJIT_TEST_BINARY} -e dofile[[${LUAJIT_TEST_INIT}]]") separate_arguments(LUAJIT_TEST_COMMAND) +set(TEST_FLAGS + --output-on-failure + --schedule-random + --parallel ${CMAKE_BUILD_PARALLEL_LEVEL} +) +if(CMAKE_VERBOSE_MAKEFILE) + list(APPEND TEST_FLAGS --verbose) +endif() + add_subdirectory(LuaJIT-tests) add_subdirectory(PUC-Rio-Lua-5.1-tests) add_subdirectory(lua-Harness-tests) @@ -86,22 +95,3 @@ add_custom_target(${PROJECT_NAME}-test DEPENDS tarantool-c-tests tarantool-tests ) - -if(LUAJIT_USE_TEST) - if(POLICY CMP0037) - if(CMAKE_VERSION VERSION_LESS 3.11) - # CMake below 3.11 reserves the name 'test'. Use old policy. - # https://cmake.org/cmake/help/v3.11/release/3.11.html#other-changes - cmake_policy(SET CMP0037 OLD) - else() - # Starting from CMake 3.11 the name 'test' is reserved in - # special cases and can be used as a target name. - cmake_policy(SET CMP0037 NEW) - endif() - endif(POLICY CMP0037) - - add_custom_target(test DEPENDS - ${PROJECT_NAME}-test - ${PROJECT_NAME}-lint - ) -endif() diff --git a/test/LuaJIT-tests/CMakeLists.txt b/test/LuaJIT-tests/CMakeLists.txt index 9cd76ee9..aa36097a 100644 --- a/test/LuaJIT-tests/CMakeLists.txt +++ b/test/LuaJIT-tests/CMakeLists.txt @@ -1,12 +1,21 @@ # See the rationale in the root CMakeLists.txt cmake_minimum_required(VERSION 3.1 FATAL_ERROR) -add_custom_target(LuaJIT-tests DEPENDS ${LUAJIT_TEST_BINARY}) - -add_custom_command(TARGET LuaJIT-tests - COMMENT "Running LuaJIT-tests" +# The test suite has its own test runner +# (test/LuaJIT-tests/test.lua), it is not possible +# to run these tests separately by CTest. +message(STATUS "Add test test/LuaJIT-tests") +add_test(NAME "test/LuaJIT-tests" COMMAND ${LUAJIT_TEST_COMMAND} ${CMAKE_CURRENT_SOURCE_DIR}/test.lua +slow +ffi +bit +jit WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} ) +set_tests_properties("test/LuaJIT-tests" PROPERTIES + LABELS luajit +) + +add_custom_target(LuaJIT-tests + COMMAND ${CMAKE_CTEST_COMMAND} -L luajit ${TEST_FLAGS} + DEPENDS luajit-main +) diff --git a/test/PUC-Rio-Lua-5.1-tests/CMakeLists.txt b/test/PUC-Rio-Lua-5.1-tests/CMakeLists.txt index 98277f9a..af180c0a 100644 --- a/test/PUC-Rio-Lua-5.1-tests/CMakeLists.txt +++ b/test/PUC-Rio-Lua-5.1-tests/CMakeLists.txt @@ -34,17 +34,22 @@ add_subdirectory(libs) # But, unfortunately, depends on specific PUC-Rio # Lua 5.1 internal headers and should be adapted for LuaJIT. -add_custom_target(PUC-Rio-Lua-5.1-tests - DEPENDS ${LUAJIT_TEST_BINARY} PUC-Rio-Lua-5.1-tests-prepare +# The test suite has its own test runner +# (test/PUC-Rio-Lua-5.1-tests/all.lua), it is not possible +# to run these tests separately by CTest. +message(STATUS "Add test test/PUC-Rio-Lua-5.1") +add_test(NAME "test/PUC-Rio-Lua-5.1" + COMMAND ${LUAJIT_TEST_COMMAND} ${CMAKE_CURRENT_SOURCE_DIR}/all.lua + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} +) +set_tests_properties("test/PUC-Rio-Lua-5.1" PROPERTIES + ENVIRONMENT "LUA_PATH=${LUA_PATH}\;\;" + LABELS lua ) -add_custom_command(TARGET PUC-Rio-Lua-5.1-tests - COMMENT "Running PUC-Rio Lua 5.1 tests" - COMMAND - env - LUA_PATH="${LUA_PATH}" - ${LUAJIT_TEST_COMMAND} ${CMAKE_CURRENT_SOURCE_DIR}/all.lua - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} +add_custom_target(PUC-Rio-Lua-5.1-tests + COMMAND ${CMAKE_CTEST_COMMAND} -L lua ${TEST_FLAGS} + DEPENDS luajit-main PUC-Rio-Lua-5.1-tests-prepare ) # vim: expandtab tabstop=2 shiftwidth=2 diff --git a/test/lua-Harness-tests/CMakeLists.txt b/test/lua-Harness-tests/CMakeLists.txt index f748a8fd..70489061 100644 --- a/test/lua-Harness-tests/CMakeLists.txt +++ b/test/lua-Harness-tests/CMakeLists.txt @@ -4,12 +4,6 @@ # See the rationale in the root CMakeLists.txt cmake_minimum_required(VERSION 3.1 FATAL_ERROR) -find_program(PROVE prove) -if(NOT PROVE) - message(WARNING "`prove' is not found, so lua-Harness-tests target is not generated") - return() -endif() - # Tests create temporary files (see 303-package.t and 411-luajit.t # for example) to be required. Also, they require some files from # the original test source directory. @@ -20,23 +14,29 @@ make_lua_path(LUA_PATH ${LUAJIT_SOURCE_DIR}/?.lua ${LUAJIT_BINARY_DIR}/?.lua ) -set(LUA_TEST_FLAGS --failures --shuffle) -if(CMAKE_VERBOSE_MAKEFILE) - list(APPEND LUA_TEST_FLAGS --verbose) -endif() +set(SUITE_NAME "lua-Harness-tests") +file(GLOB tests ${CMAKE_CURRENT_SOURCE_DIR} "*.t") +foreach(test_path ${tests}) + get_filename_component(test_name ${test_path} NAME) + if (${test_name} STREQUAL ${SUITE_NAME}) + continue() + endif (${test_name} STREQUAL ${SUITE_NAME}) + set(test_title "test/${SUITE_NAME}/${test_name}") + message(STATUS "Add test ${test_title}") + add_test(NAME ${test_title} + COMMAND ${LUAJIT_TEST_COMMAND} -l profile_luajit21 ${test_path} + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + ) + set_tests_properties(${test_title} PROPERTIES + ENVIRONMENT "LUA_PATH=${LUA_PATH}\;" + LABELS lua-Harness + ) +endforeach() -add_custom_target(lua-Harness-tests DEPENDS ${LUAJIT_TEST_BINARY}) -add_custom_command(TARGET lua-Harness-tests - COMMENT "Running lua-Harness tests" - COMMAND - env - LUA_PATH="${LUA_PATH}" - ${PROVE} ${CMAKE_CURRENT_SOURCE_DIR} - --exec '${LUAJIT_TEST_COMMAND} -l profile_luajit21' - --jobs ${CMAKE_BUILD_PARALLEL_LEVEL} - ${LUA_TEST_FLAGS} - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} +add_custom_target(lua-Harness-tests + COMMAND ${CMAKE_CTEST_COMMAND} -L lua-Harness ${TEST_FLAGS} + DEPENDS luajit-main ) # vim: expandtab tabstop=2 shiftwidth=2 diff --git a/test/tarantool-c-tests/CMakeLists.txt b/test/tarantool-c-tests/CMakeLists.txt index 17255345..902828bf 100644 --- a/test/tarantool-c-tests/CMakeLists.txt +++ b/test/tarantool-c-tests/CMakeLists.txt @@ -1,15 +1,4 @@ -find_program(PROVE prove) -if(NOT PROVE) - message(WARNING "`prove' is not found, so tarantool-c-tests target is not generated") - return() -endif() - set(C_TEST_SUFFIX .c_test) -set(C_TEST_FLAGS --failures --shuffle) - -if(CMAKE_VERBOSE_MAKEFILE) - list(APPEND C_TEST_FLAGS --verbose) -endif() # Build libtest. @@ -49,21 +38,29 @@ foreach(test_source ${tests}) LIST(APPEND TESTS_COMPILED ${exe}) endforeach() +# Note, we cannot globbing generated files in CMake, see +# https://stackoverflow.com/questions/44076307/cmake-globbing-generated-files +# To workaround this we globbing source files and generated tests +# with path to executable binaries. +file(GLOB tests ${CMAKE_CURRENT_SOURCE_DIR} "*.test.c") +set(SUITE_NAME "tarantool-c-tests") +foreach(test_path ${tests}) + get_filename_component(test_name ${test_path} NAME_WE) + if (${test_name} STREQUAL ${SUITE_NAME}) + continue() + endif (${test_name} STREQUAL ${SUITE_NAME}) + set(test_title "test/${SUITE_NAME}/${test_name}${C_TEST_SUFFIX}") + message(STATUS "Add test ${test_title}") + add_test(NAME ${test_title} + COMMAND ${CMAKE_CURRENT_BINARY_DIR}/${test_name}${C_TEST_SUFFIX} + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + ) + set_tests_properties(${test_title} PROPERTIES + LABELS tarantool-c + ) +endforeach() + add_custom_target(tarantool-c-tests + COMMAND ${CMAKE_CTEST_COMMAND} -L tarantool-c ${TEST_FLAGS} DEPENDS libluajit libtest ${TESTS_COMPILED} ) - -add_custom_command(TARGET tarantool-c-tests - COMMENT "Running Tarantool C tests" - COMMAND - ${PROVE} - ${CMAKE_CURRENT_BINARY_DIR} - --ext ${C_TEST_SUFFIX} - --jobs ${CMAKE_BUILD_PARALLEL_LEVEL} - # Report any TAP parse errors, if any, since test module is - # maintained by us. - --parse - ${C_TEST_FLAGS} - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} -) - diff --git a/test/tarantool-tests/CMakeLists.txt b/test/tarantool-tests/CMakeLists.txt index c15d6037..e9d89366 100644 --- a/test/tarantool-tests/CMakeLists.txt +++ b/test/tarantool-tests/CMakeLists.txt @@ -4,12 +4,6 @@ # See the rationale in the root CMakeLists.txt. cmake_minimum_required(VERSION 3.1 FATAL_ERROR) -find_program(PROVE prove) -if(NOT PROVE) - message(WARNING "`prove' is not found, so tarantool-tests target is not generated") - return() -endif() - macro(BuildTestCLib lib sources) add_library(${lib} SHARED EXCLUDE_FROM_ALL ${sources}) target_include_directories(${lib} PRIVATE @@ -83,21 +77,14 @@ make_lua_path(LUA_PATH ${PROJECT_SOURCE_DIR}/tools/?.lua ${LUAJIT_SOURCE_DIR}/?.lua ${LUAJIT_BINARY_DIR}/?.lua + ${PROJECT_BINARY_DIR}/src/?.lua ) + # Update LUA_CPATH with the library paths collected within # macro. make_lua_path(LUA_CPATH PATHS ${LUA_CPATHS}) set(LUA_TEST_SUFFIX .test.lua) -set(LUA_TEST_FLAGS --failures --shuffle) -set(LUA_TEST_ENV - "LUA_PATH=\"${LUA_PATH}\"" - "LUA_CPATH=\"${LUA_CPATH}\"" -) - -if(CMAKE_VERBOSE_MAKEFILE) - list(APPEND LUA_TEST_FLAGS --verbose) -endif() # XXX: Since the auxiliary libraries are built as a dynamically # loaded modules on MacOS instead of shared libraries as it is @@ -133,25 +120,22 @@ else() list(APPEND LUA_TEST_ENV_MORE LD_LIBRARY_PATH=${LD_LIBRARY_PATH}) endif() -# LUA_CPATH and LD_LIBRARY_PATH variables and also TESTLIBS list -# with dependecies are set in scope of BuildTestLib macro. +file(GLOB_RECURSE tests ${CMAKE_CURRENT_SOURCE_DIR} "*${LUA_TEST_SUFFIX}") +foreach(test_path ${tests}) + get_filename_component(test_name ${test_path} NAME) + set(test_title "test/tarantool-tests/${test_name}") + message(STATUS "Add test ${test_title}") + add_test(NAME ${test_title} + COMMAND ${LUAJIT_TEST_COMMAND} ${test_path} + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} + ) + set_tests_properties(${test_title} PROPERTIES + ENVIRONMENT "LUA_PATH=${LUA_PATH};\;\;;LUA_CPATH=${LUA_CPATH}\;\;;${LUA_TEST_ENV_MORE}" + LABELS tarantool + ) +endforeach() + add_custom_target(tarantool-tests - DEPENDS ${LUAJIT_TEST_BINARY} ${TESTLIBS} + COMMAND ${CMAKE_CTEST_COMMAND} -L tarantool ${TEST_FLAGS} + DEPENDS luajit-main ${TESTLIBS} ) -add_custom_command(TARGET tarantool-tests - COMMENT "Running Tarantool tests" - COMMAND - # XXX: We can't move everything to the "inner" env, since there - # are some issues with escaping ';' for different shells. As - # a result LUA_PATH/LUA_CPATH variables are set via the "outer" - # env, since they are not stripped by SIP like LD_*/DYLD_* are. - env - ${LUA_TEST_ENV} - ${PROVE} ${CMAKE_CURRENT_SOURCE_DIR} - --exec 'env ${LUA_TEST_ENV_MORE} ${LUAJIT_TEST_COMMAND}' - --ext ${LUA_TEST_SUFFIX} - --jobs ${CMAKE_BUILD_PARALLEL_LEVEL} - ${LUA_TEST_FLAGS} - WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} -) - -- 2.34.1