Random test timeouts bringing down PR build & test iterations 2023

[bartlettroscoe]

CC: @ccober6, @sebrowne, @trilinos/framework

Description

Anecdotal evidence seems to suggest that random test failures, including random timeouts, are bringing down PR testing iterations fairly regularly. When this happens, all of the builds need to be run again from scratch, wasting testing computing resources, blocking PR testing iterations for other PRs, and delaying the merge of PRs.

For example, this query over the last two months suggest that random test timeouts took out PR testing iterations for the following PRs:

• Teuchos: Fix LARND with Kokkos::complex<double> #12388
• Revert "Framework: Enable CMAKE_LINK_LIBRARIES_ONLY_TARGETS" #12372
• Framework: Disable randomly failing ROL test #12367
• Belos: add Tpetra pseudo_cg_indefinite test #12281
• Tpetra: Adding labels to Kokkos::fence(), fixing bug #12259
• Tpetra: Adding Kokkos::fence tracking support to timer injection #12223
• Tpetra: use kokkos kernels BsrMatrix spmv #12103

Note that the test timeouts for the PRs #12050 and #12297 shown in that query don't appear to be random. Filtering out those PRs yields this reduced query over the last two months shows the 7 randomly failing tests:

Site	Build Name	Test Name	Status	Time	Proc Time	Details	Build Time	Processors
ascic164	PR-12388-test-rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-off_no-package-enables-1204	ROL_example_PDE-OPT_ginzburg-landau_example_01_MPI_4	Failed	10m 130ms	40m 520ms	Completed (Timeout)	2023-10-10T12:08:21 MDT	4
ascicgpu036	PR-12372-test-rhel7_sems-cuda-11.4.2-sems-gnu-10.1.0-sems-openmpi-4.0.5_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables-2552	Adelus_vector_random_npr3_rhs4_MPI_3	Failed	10m 60ms	30m 180ms	Completed (Timeout)	2023-10-06T10:29:53 MDT	3
ascic114	PR-12367-test-rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-off_no-package-enables-1181	MueLu_ParameterListInterpreterTpetra_MPI_4	Failed	10m 90ms	40m 360ms	Completed (Timeout)	2023-10-05T09:36:07 MDT	4
ascic166	PR-12281-test-rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables-1452	Tempus_IMEX_RK_Partitioned_Staggered_FSA_Partitioned_IMEX_RK_1st_Order_MPI_1	Failed	10m 40ms	10m 40ms	Completed (Timeout)	2023-09-25T11:50:41 MDT	1
ascic166	PR-12259-test-rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables-1347	Tempus_BackwardEuler_MPI_1	Failed	10m 30ms	10m 30ms	Completed (Timeout)	2023-09-14T00:58:50 MDT	1
ascic166	PR-12223-test-rhel7_sems-gnu-8.3.0-openmpi-1.10.1-serial_debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables-1317	Tempus_BackwardEuler_MPI_1	Failed	10m 40ms	10m 40ms	Completed (Timeout)	2023-09-11T14:11:26 MDT	1
ascic164	PR-12103-test-rhel7_sems-intel-2021.3-sems-openmpi-4.0.5_release-debug_shared_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-off_no-package-enables-784	ROL_test_algorithm_TypeE_StabilizedLCL_MPI_1	Failed	10m 120ms	10m 120ms	Completed (Timeout)	2023-08-10T11:19:22 MDT	1

NOTE: Further analysis would be needed to confirm that all of these tests were random timeouts. But I believe that a tool could be written to automatically determine if a timeout (or any test failure) was random. It would actually not be that hard to do.

Suggested solution

The simple solution would seem for the ctest -S driver to just rerun the failing tests again, in serial, to avoid the timeouts. For example, CTest directly supports this with the --repeat after-timeout:<n> argument and the ctest_test() argument REPEAT after-timeout:<n>.

[bartlettroscoe]

FYI: I created the following internal Trilinos HelpDesk issue asking about rerunning timing out tests:

• https://sems-atlassian-son.sandia.gov/jira/servicedesk/customer/portal/7/TRILINOSHD-261

[cgcgcg]

@bartlettroscoe What was the random failure for "MueLu_ParameterListInterpreterTpetra_MPI_4"? How often does that happen?

[sebrowne]

I proposed a --repeat-until pass to the operational leaders about a month ago and we decided not to pursue it. I'm personally much more open to the --repeat after-timeout approach, and will bring this to the next operational leaders meeting.

[ccober6]

@bartlettroscoe , #12316 went through about 3 weeks ago and should have taken care of Tempus_BackwardEuler timeouts. It is believed that these timeouts where due to debug builds and machine loads. Have you seen timeouts from this in the past ~three weeks?

[ccober6]

Ok, looking closer at the table above, those timeouts occurred 9/14 and 9/11 and #12316 went through on 9/23. Hopefully this fixed it, but let me know if they pop up again.

[bartlettroscoe]

I proposed a --repeat-until pass to the operational leaders about a month ago and we decided not to pursue it.

@sebrowne, I don't think that would be effective because there may be persistently failing tests that will never pass. I think that if the Trilinos Framework CTest -S driver reran the failing tests just once, that would likely resolve 95% of random test failures (including random test timeouts) and avoid having to run all of the builds from scratch again.

In addition, The Trilinos PR testing system desperately needs to have some automated monitoring set up to detect random failures so they can be addressed. Right now Trilinos is flying blind and only when someone does manual CDash queries like I have done above do you see the issues. Simply rerunning randomly failing tests is a good step forward but does not address the underlying problems with the Trilinos test suite.

NOTE: Random timeouts are a much easier problem because you can just rerun them in serial and they will pass. That is why I scoped this issue to focus on just random timeouts. (Also note that some timeouts are not random but are due to changes on the PR topic branch.)

Can we discuss this topic at the next Trilinos Developers meeting or during the TUG developer day?

[ccober6]

I will look at Tempus_IMEX_RK_Partitioned_Staggered_FSA_Partitioned_IMEX_RK_1st_Order and see if I can break it down into smaller jobs. Otherwise we may need to disable it for debug builds.

[bartlettroscoe]

What was the random failure for "MueLu_ParameterListInterpreterTpetra_MPI_4"? How often does that happen?

Have you seen timeouts from this in the past ~three weeks?

@cgcgcg and @ccober6, those are all questions that can be answered by running CDash queries, like I did above. But note that there is insufficient data on CDash to fully analyzing what is happening. That is, you don't know the exact version of the 'develop' branch and the PR topic branches that are being tested by just looking at CDash. You have to also look at the Trilinos PR comments. Not clear how hard it would be to write a tool that could do that (since you would have to parse the HTML pages).

[bartlettroscoe]

I'm personally much more open to the --repeat after-timeout approach, and will bring this to the next operational leaders meeting.

@sebrowne, I would be curious what the argument could be against running a small number of timing out tests again in serial at the end of a PR build.

[csiefer2]

For the one time MueLu_ParameterListInterpreterTpetra_MPI_4 timed out in the last month, we were getting this:

[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)
[ascic114][[11936,1],1][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)
[ascic114][[11936,1],1][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],1][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],1][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(18) failed: Connection reset by peer (104)
[ascic114][[11936,1],2][btl_tcp.c:559:mca_btl_tcp_recv_blocking] recv(17) failed: Connection reset by peer (104)

This looks like a system-level MPI problem to me. I don't think this is something we, as Trilinos developers need to worry about.

[rppawlo]

@sebrowne, I would be curious what the argument could be against running a small number of timing out tests again in serial at the end of a PR build.

Race conditions. A lot of race conditions show up only when we put a load on the cuda cards with multiple tests. I've got many examples from the atdm apps where the tests pass fine when run by themselves, but when run with other tests it triggers the race condition. Rerunning failed tests by themselves would allow a much higher chance of getting a race condition into the code base. Personally I'd rather hit retest once in a while than try to figure out which commit might be causing a race condition. That was the thinking.

[bartlettroscoe]

This looks like a system-level MPI problem to me. I don't think this is something we, as Trilinos developers need to worry about.

@csiefer2, that is not the point of this issue #12391. The point of this issues is that we should not have the Trilinos PR testing system blow away all of the builds and start from scratch for one single test timeout due to a system issue (like overloading the machine, or some silly MPI communication problems).

[sebrowne]

The ability to restart single jobs to rerun ONLY those jobs is one of the considerations for the GitHub Actions based CI that we are working towards. That may help address some of the concern at least with respect to the overhead of re-running a more-appropriate set of jobs.

[bartlettroscoe]

The ability to restart single jobs to rerun ONLY those jobs is one of the considerations for the GitHub Actions based CI that we are working towards. That may help address some of the concern at least with respect to the overhead of re-running a more-appropriate set of jobs.

@sebrowne, you should be able to do that from within the same CTest -S invocation with another call to ctest_test() and just submit the results again (and determine pass/fail based on the 2nd ctest_test() invocation that reruns all of the failing tests). You will need put in some logic to ensure you don't rerun like 2000 failing tests in serial. (For example, you would only rerun failing tests if say less than 5 tests or something failed. A safe and logic trigger can be discussed)