[CI] Tests timing out occasionally

[IAlibay]

For some reason some of the tests are every so often timing out. It's been mostly the ubuntu full deps py3.10 gh actions tests, but we've seen it in the macos and azure runners too.

It's rather hard to debug and I'm currenttly cycling CI with pytest-timeout in #4197

[IAlibay]

From a look at the top 50 slowest, there seems to be some kind of systematic increase in test run times in the failing runners.

Note: test_streamplot_2D is > 20s even on a good day!

[IAlibay]

@MDAnalysis/coredevs this is actually a pretty big issue, if anyone has any ideas here that'd be appreciated.

[IAlibay]

My initial suspicion is that this might be happening: pytest-dev/pytest#11174

Need to gather a bit more evidence though (which annoyingly means re-running the tests until we can gather more random failures).

A potential short term solution is to just stick on pytest-timeout, but it really only shifts the problem from a job not finishing to one failing (so really it counts as a failure both ways and CI remains red).

[IAlibay]

Unfortunately I've been unable to narrow down where this is happening. From runs on #4197 my current conclusions are:

1. The frequency at which these stalling runs occus is somewhat low. It affects nearly every PR at some point, but we're looking at maybe a ~ 1 / 20 chance?
2. There are no specific tests which time out (once it was a multiprocessing one, another was nsgrid..)
3. It is not isolated to github actions (we saw it happen on azure pipelines).

My current completely random guess is that it's something to do with the latest 7.4.0 release and maybe coverage? (I haven't seen it happen on runner not trying to use pytest-cov functionality).

I've ran out of ideas for now and I don't think I have the time to debug any furtther. So I'm going to propose the following:

1. We add pytest-timeout to our CI runs. At least runners will complete even though they will fail. That will save a few trees and allow us to just ignore timeout issues if we don't think they are relevant.
2. We keep this issue ope and re-asses over time.

[hmacdope]

Sounds good to me @IAlibay, thanks for putting in a great effort trying to figure it out.

[RMeli]

Sounds very sensible. Thanks for all the work!

[drew-parsons]

I'm seeing a consistent timeout (after 2.5 hours) in mdanalysis 2.5.0 on i386 after updating gsd to v3, see https://ci.debian.net/packages/m/mdanalysis/testing/i386/

The timeout occurs in testsuite/MDAnalysisTests/parallelism/test_multiprocessing.py
So I suspect it might be a race condition in multiprocessing, hence might be a separate issue from the pytest-timeout issue discussed here.

[IAlibay]

Interesting, I completely forgot that the latest version of GSD had a very strange memory buffer thing going on.

We did see tests time out in other places though, so I'm not fully convinced it's the sole issue here, but it is definitely worth investigating.

[IAlibay]

Although @drew-parsons, MDAnalysis 2.5.0 doesn't support GSD 3.0+. The GSD release was after the 2.5.0 release so we didn't get to retrospectively pin this in our setup.py but it is reflected in our conda recipe.

We're pretty swamped with just the two package indexes (pypi and conda-forge) so this kind of stuff tends to completely fall off our radar. Is there any way we can better synch up / report on this?

[drew-parsons]

True, I patched MDAnalysis 2.5.0 with PR #4153 and PR #4174 .