SKIP test without behavoir of failing if no tests are executed

[SimeonEhrig]

Description

We use Catch2 to test an abstraction library for CPU/GPU programming named alpaka. In some parts of the test we need to get a device (CPU/GPU). But sometimes the device is not available. This is expected behavior of our library, because you can compile an application with support for Nvidia, AMD and Intel GPUs, ship the application on different systems and check at runtime if the GPU is available. Therefore the tests needs to work in the same way.

At the moment, we use this code snippet:

TEMPLATE_LIST_TEST_CASE("atomicOperationsWorking", "[atomic]", TestApis)
{
    auto cfg = TestType::makeDict();

    auto deviceSpec = cfg[object::deviceSpec];
    auto exec = cfg[object::exec];

    auto devSelector = onHost::makeDeviceSelector(deviceSpec);
    if(!devSelector.isAvailable())
    {
        SUCCEED("No device available for " << deviceSpec.getName());
        return;
    }

    auto device = devSelector.makeDevice(0);
    // ...
}

Since getting a device is a function we use very frequently, I would like to move it into a helper function as follows:

    auto getDevice(auto cfg)
    {
        auto deviceSpec = cfg[object::deviceSpec];
        auto devSelector = onHost::makeDeviceSelector(deviceSpec);
        UNSCOPED_INFO("DeviceSpec: " << getName(deviceSpec));
        UNSCOPED_INFO("API: " << deviceSpec.getApi().getName());

        if(!devSelector.isAvailable())
        {
            SKIP("No device available for " << deviceSpec.getName());
        }

        onHost::Device device = devSelector.makeDevice(0);
        UNSCOPED_INFO("Device: " << device.getName());
        return device;
    }

The problem is that we have test files that contain only tests that rely on a device, and we have test files that also contain tests that do not require a device and are intended to test something device-independent. In the first cases, ctest fails if no device is available. If I set the argument --allow-running-no-tests for all test, it is possible that in the second case tests passes, which does nothing.

I could set the argument --allow-running-no-tests for specific test executables but to be honest, I think this really error prone and we have some developers and contributors, which are not familiar with such Catch2 details.

Is is possible to implement an SKIP macro which does not trigger the no-running-test check? It does not need to be part of Catch2. I'm also fine to implement it as test helper for our library.

[horenmar]

If you are using plain ctest, set SKIP_RETURN_CODE to 4. The catch_discover_tests helper already does this by default.

[SimeonEhrig]

If you are using plain ctest, set SKIP_RETURN_CODE to 4. The catch_discover_tests helper already does this by default.

I think this will be the same, like set the test argument --allow-running-no-tests for specific tests. Instead setting this argument, I set the test property.

[horenmar]

The difference is in how running ctest will report the skipped tests.

$ ctest -j 16
...
1145/1145 Test  #803: This test is fine ....................................................   Passed  181.78 sec

98% tests passed, 23 tests failed out of 1145

Total Test time (real) = 231.35 sec

The following tests did not run:
	560 - hardware dependent test 1 (Skipped)
	1009 - hardware dependent test 2 (Skipped)
	1010 - another skipped test (Skipped)

The following tests FAILED:
	483 - blah blah blah (Failed)
	484 - bleh bleh bleh (Failed)
	485 - xxxxxxxxxxxxxx (Failed)
...

With --allow-running-no-tests, the skipped tests will instead be reported as Passed.

[SimeonEhrig]

Okay that's improves the output but it does not solve my problem.

In the meantime, I've looked into how SKIP() works. Ultimately, it throws a catch2 exception, which is caught. The exception handler notes that the test was skipped, and the test is terminated. Therefore, the best solution would be to throw a custom exception and add a custom exception handling to the exception handler that simply terminates the test. Unfortunately it looks like that the exception handler cannot be extended.

[horenmar]

In that case I don't understand your problem. Based on the original post, I thought that you wanted to

1. SKIP from a shared helper function based on HW available
2. But not have it trigger a failure if all tests in that run end up skipped.
3. Ideally without having to use --allow-running-no-tests as that would mask failures where all tests would be filtered out.

[SimeonEhrig]

In that case I don't understand your problem. Based on the original post, I thought that you wanted to

1. `SKIP` from a shared helper function based on HW available

2. But not have it trigger a failure if all tests in that run end up skipped.

3. Ideally without having to use `--allow-running-no-tests` as that would mask failures where all tests would be filtered out.

That's correct. Maybe I misunderstand your solution.

Example

Let's try it with a example. I have two test files, each will be a own test executable.

test_with_get_device.cpp

TEST_CASE("use optional device", ""){
  auto device = test::get_device();
  REQUIRE(does_work(device) == true);
}

test_without_device.cpp

TEST_CASE("use optional device", ""){
  std::optinal<Foo> foo = get_foo();
  if(foo){
    REQUIRE((*foo).has_bar());
  }
}

In test_with_get_device.cpp depending if a device is available there is no REQUIRE performed because test::get_device() does a SKIP() if no device is available. In this case the test should not fail.

In test_without_device.cpp, the test does not run a REQUIRE if foo is not valid. In this case I want to get an error, that no check was performed.

I would assume I have to write the following CMake code to compile both test executeables:

add_executable(test_with_get_device test_with_get_device.cpp)
target_link_libraries(test_with_get_device  PRIVATE Catch2::Catch2WithMain)
catch_discover_tests(test_with_get_device)
set_tests_properties(test_with_get_device PROPERTIES SKIP_RETURN_CODE 4)

add_executable(test_without_device test_without_device.cpp)
target_link_libraries(test_without_device  PRIVATE Catch2::Catch2WithMain)
catch_discover_tests(test_without_device)

Does a modification later

Now I extent the test_with_get_device.cpp with a second test case, which can miss the REQUIRE.

test_with_get_device.cpp

TEST_CASE("use optional device", ""){
  auto device = test::get_device();
  REQUIRE(does_work(device) == true);
}

TEST_CASE("other test without device", ""){
  Bar bar = get_bar();
  if(bar){
    REQUIRE(bar.is_valid());
  }
}

I would assume that ctest passes if no device is available and if(bar) is false until I remove the CMake line set_tests_properties(test_with_get_device PROPERTIES SKIP_RETURN_CODE 4). Does I understand it correctly?

[horenmar]

I see what you are asking about, and the answer is a bit more complex.

Given this:

TEST_CASE( "blah blah 1" ) { SKIP(); }
TEST_CASE( "blah blah 2" ) { /* empty */ }

a test run that contains both will pass:

$ ./tests/SelfTest "blah blah*" -r compact
Filters: "blah blah*"
RNG seed: 2536557436
/mnt/c/ubuntu/Catch2/tests/SelfTest/UsageTests/Misc.tests.cpp:578: skipped:
test cases: 2 | 1 passed | 1 skipped
assertions: - none -


$ echo $?
0

In fact, just running the second test will pass as well:

$ ./tests/SelfTest "blah blah 2" -r compact
Filters: "blah blah 2"
RNG seed: 1618592201
test cases: 1 | 1 passed
assertions: - none -


$ echo $?
0

This is because the automatic failure is for cases where no tests were ran, rather than no assertions were executed (skipped tests for this count as not being run).

If you want the second test to fail, you need to use the NoAssertions "warning":

$ ./tests/SelfTest "blah blah 2" -r compact -w NoAssertions
Filters: "blah blah 2"
RNG seed: 480993569
test cases: 1 | 1 failed
assertions: 1 | 1 failed


xarn@LAPTOP-HFAGAACA:~/builds/Catch2/debug-v3-all$ echo $?
42

This will then also cause failure if you run both, whether you use --allow-running-no-tests, or SKIP_RETURN_CODE, or not, because now you have an explicitly failing test.

[horenmar]

To sum it up:

	nothing	SKIP_RETURN_CODE	-w NoAss...	both
1 test, SKIP	PASSED	SKIPPED (not failure)	PASSED	SKIPPED
1 test, empty	PASSED	PASSED	FAILED	FAILED
both tests ^^	PASSED	PASSED	FAILED	FAILED

As I understand your last post, I think you want both options.

[SimeonEhrig]

To sum it up:
nothing SKIP_RETURN_CODE -w NoAss... both
1 test, SKIP PASSED SKIPPED (not failure) PASSED SKIPPED
1 test, empty PASSED PASSED FAILED FAILED
both tests ^^ PASSED PASSED FAILED FAILED

As I understand your last post, I think you want both options.

@horenmar Thanks for the example. I tested in our project and it works like you written which means different to my last observation. In the mean time we changed some stuff in our code including updating to Catch 3.15. I have to reevaluate my old solution with SKIP(). Unfortunately this takes some time because of the slow CI. I will report my observations soon as possible.