Motivation

Our tests are incredibly frustrating. If I'm adding a new feature and writing tests for it in the fashion we currently do it, adding stuff to the test_data.json and using that in the tests I'm writing, entirely different tests fail – tests that assume that there are exactly $n$ pages for a region, but I just made it $n+1$, or that there are exactly $m$ db queries performed to render a view, but because of the added page there now were a few more. While keeping an eye on the amount of queries is definitely a good idea, both of these situations are incredibly confusing to understand.

Ideally, tests should

• be isolated into easily understandable test cases (ideally easily memorable user stories), with a descriptive name that makes it easy to infer the thing being tested (like test_redirect_to_login_form_when_not_logged_in or test_redirect_back_to_original_page_after_login_redirect)
• provide a human readable (or at least easily comprehensible) "description" of what is being tested – meaning that as a first time reader (e.g. after the test suddenly failed for me), using my first assumption from the test name, I can easily follow which exact steps are performed in order to test the thing, and in doing so hopefully understand how my changes made this test fail and form a strategy for adjusting my changes, other parts of the system or, if I'm sure I'm doing a meaningful shift in intention of the systems behaviour, modify the failing test itself. Here, "description" doesn't mean comments or docstrings, but the code itself. Test code might need to be more verbose and violate some code style rules we would otherwise want to enforce throughout our code base (like minimal code duplication, newline or formatting policies), but for tests the highest priority (besides being valid and effective) is to be readable, since they are the tool that help us verify that the system does what it's supposed to, and we need to be able to tell whether the tests make sense or are wrong themselves with high certainty. Comments can provide additional insight, but the code itself must already make sense to the reader.
• the "description" has to contain all necessary information for the test, without relying on magic text files that make it harder to understand what exactly is being done. Helper functions should only be used if the reduced code lines and outsourced complexity are worth the possibly added confusion by a reader new to the topic. When helper functions are used in tests, they should always have meaningful and possibly slightly more verbose names to reduce the potential for wrong assumptions.

Currently, tests like test_api_result or test_view_status_code_1.py through test_view_status_code_16.py are far from fulfilling these requirements for me, as I'm sure you heard me complain about already. They are not isolated but essentially perform batch checks whether all available values match their expected ones. They are not very understandable at all, as they immediately delegate values to a helper function which is generalized and far from any memorable use case or user story, making it even more difficult to understand which part exactly is different than intended and why. Obviously, the names have no chance of being helpful either. The parametrization is difficult to look up and understand, and is ultimately reduction of redundancy that completely hinders traceability and hinders adjusting of the parametrized data, in case that it really has to be changed, as it is far too easy to loose overview and adjust the wrong spot.

Proposed Solution

Alternatives

• Write new tests only containing "one logical thought" (user story etc.) and with as little reliance on the test_data fixture as possible. Instead, create pages etc. as part of the test, with exactly the data you need it to have. Consider very obviously named helper functions if this takes up too many lines that seem uninteresting/distracting. This way, reading the test function should make it immediately obvious what the intention of the test is, and easy to prove that the test correctly follows through checking that intention.
• Gradually convert old tests to this ideal, starting at those that attract attention for being difficult to understand while being small enough to convert in a PR.
  • As for the huge, unwieldy test suites… maybe they're fine for now? Until someone gets too fed up.
  • The idea of performance tests is fine (keep amount of queries in check), but maybe we can come up with a better concept

User Story

As a developer I want to run tests so that I can verify that the rest of the system still works as expected after my changes.
As a developer I want to write tests so that I can also ensure this for newly added features or existing parts of the system that previously lacked tests.
As a developer I want to quickly understand what certain tests do and why so that I can understand the current expectation of the systems behaviour as well as how my changes possibly unintentionally changed that behaviour, so that I can fix it.
As a developer I want to not spend hours on chasing errors in tests I didn't write and don't understand so that I can go to sleep quickly and not feel shit about not having gotten anything done.

Additional Context

Please note that these "requirements" are mostly my gut feeling, even though I do feel strongly about it. Everything is open to discussion, if you have a better strategy it should absolutely stand here instead of mine, I just want to start reforms into something easy to work with already.