ParameterizedTest does not work accurately

[felix-seifert]

I want to assert that the exit code will be 1 for several input files. If I test this by writing several tests, the test reports display truthfully what happens if I start the program with these input file conventionally. As the tests have the same structure, I combined them in a @ParameterizedTest with different file names as the parameter. This parameterized test, however, seems to report a test success for every iteration if at least one iteration is successful which does not give me the correct test result.

When I run the two tests separately, one test fails and one test succeeds.

  @Test
  @ExpectSystemExitWithStatus(1)
  void parseErrorTest_fail() throws IOException {
    final String pathToError = pathPrefix + "error/comments-error-1.cpp";
    CompilerMain.main(new String[]{pathToError});
  }

  @Test
  @ExpectSystemExitWithStatus(1)
  void parseErrorTest_succeed() throws IOException {
    final String pathToError = pathPrefix + "error/parse-error.cpp";
    CompilerMain.main(new String[]{pathToError});
  }

If I combine these two tests in a @ParameterizedTest, each iteration seems to be succeeding.

  @ParameterizedTest
  @ValueSource(strings = {"parse-error.cpp", "comments-error-1.cpp"})
  @ExpectSystemExitWithStatus(1)
  void parseErrorTest(final String fileName) throws IOException {
    final String pathToError = pathPrefix + "error/" + fileName;
    CompilerMain.main(new String[]{pathToError});
  }

I am using JUnit version 5.8.1 for junit-jupiter-api and junit-jupiter-params. The version of junit5-system-exit is 1.1.1. I am running on Java 17 (OpenJDK Runtime Environment GraalVM CE 21.3.0 (build 17.0.1+12-jvmci-21.3-b05)).

[felix-seifert]

If both test iterations of the parameterized test fail, the parameterized test shows a fail.

[tginsberg]

Hey @felix-seifert thanks for the detailed report!

I was able to reproduce this and I think the issue is fixable.

Here's why this happens. When the system exit extension starts, it sets some state before the test is run and interprets it after the test has concluded (either naturally or via an exception). Because I didn't have parameterized tests in mind when I wrote this, when the end-of-test method concluded, I didn't reset the state. This works fine during non-parameterized tests because the extension doesn't have any more work to do and is thrown away. However, during parameterized tests, the same extension instance is used for each set of parameters. This means when a parameterized test runs, it is already seeded with state information from past runs.

The fix to this is to clear the state at the end of a test, in anticipation of this. Or find a way to mark the extension as needing reinstantiation for each parameterized run. I'll do some research on that, but I do have the first option (clearing state) already working.

Thanks again for the report, I'll have a fix for this soon.

[tginsberg]

This has been fixed in release 1.1.2.

[felix-seifert]

Thanks for the fix. I already enjoy using this. 😄