Motivation
Our tests are sometimes flaky and can influence each other. This is last but not least fostered by the different approaches the django framework and pytest have to testing, and the way pytest-django tries to frankenstein the two together.
One effect is that test data is loaded into the database once and then every test is just run one after the other. While this works for tests that do not rely on being able to complete transactions, since then the test suite can wrap the whole test in a transaction and roll back any changes it made afterwards, tests that do just empty the database afterwards, since django test class expects to be under the django system where whatever test data is needed is loaded into the database at the start of each test anyways – which doesn't happen under in the pytest-django environment, dooming all non-transactional tests running after the first transactional one. While we try to patch that up by marking all transactional tests to run last, there seem to be still other effects where objects expected to be in the database suddenly cannot be found at some point.
Proposed Solution
While an alternative for transactional=True, serialized_rollback=True (that only half-works with pytest-django) has been proposed on the django project along with a proof-of-concept that also showed greatly improved performance, we will not want to wait for this to be released and available to us, since we are still quite behind the newest django version. Instead, and possibly also easier given our altered situation under pytest-django, we should use the concept of cloning database instances used in the proof-of-concept and implement it in a form best suited for our use.
I propose a set of (pytest) fixtures to create cloned instances of the current database as some sort of stack, with which data can be prepared for use in multiple tests as a template. Each tests will finally create a last clone to create an environment that can be safely discarded after the test finishes, keeping the parent state unchanged for any subsequent tests to use as basis.
Alternatives
- Wait for the solution to land upstream.
This seems infeasible as it might take a while for this to be finished and land in a django release, at which point pytest-django will need to adapt to expose any new flags, while the possibility exists that, much like currently the purging of data after a test is and the loading of data before the next test isn't handled by django or the pytest-django glue code, the new functionality will not be exposed in an entirely usable capacity. Furthermore, we will still need to navigate the dependency jungle to get our project to the newest django version when it arrives
- Deal with the current state of affairs.
With a slowly growing test suite, we are getting more and more instances where pipelines fail because of side effects and race conditions that should not occur. If we want to be able to rely on and in any reasonable capacity work with our test suite in the future, we need to address this
User Story
As a developer I want to be able to quickly check whether my changes impacted some unexpected aspect of the system so that I don't code up medusa.git but continually improve the quality of the project.
Additional Context
Design Requirements
Related Issues
#765 #2119 #2711 #2978 #3343 #3370
Summary of discussion and updates to the description
Motivation
Our tests are sometimes flaky and can influence each other. This is last but not least fostered by the different approaches the django framework and pytest have to testing, and the way pytest-django tries to frankenstein the two together.
One effect is that test data is loaded into the database once and then every test is just run one after the other. While this works for tests that do not rely on being able to complete transactions, since then the test suite can wrap the whole test in a transaction and roll back any changes it made afterwards, tests that do just empty the database afterwards, since django test class expects to be under the django system where whatever test data is needed is loaded into the database at the start of each test anyways – which doesn't happen under in the pytest-django environment, dooming all non-transactional tests running after the first transactional one. While we try to patch that up by marking all transactional tests to run last, there seem to be still other effects where objects expected to be in the database suddenly cannot be found at some point.
Proposed Solution
While an alternative for
transactional=True, serialized_rollback=True(that only half-works with pytest-django) has been proposed on the django project along with a proof-of-concept that also showed greatly improved performance, we will not want to wait for this to be released and available to us, since we are still quite behind the newest django version. Instead, and possibly also easier given our altered situation under pytest-django, we should use the concept of cloning database instances used in the proof-of-concept and implement it in a form best suited for our use.I propose a set of (pytest) fixtures to create cloned instances of the current database as some sort of stack, with which data can be prepared for use in multiple tests as a template. Each tests will finally create a last clone to create an environment that can be safely discarded after the test finishes, keeping the parent state unchanged for any subsequent tests to use as basis.
Alternatives
This seems infeasible as it might take a while for this to be finished and land in a django release, at which point pytest-django will need to adapt to expose any new flags, while the possibility exists that, much like currently the purging of data after a test is and the loading of data before the next test isn't handled by django or the pytest-django glue code, the new functionality will not be exposed in an entirely usable capacity. Furthermore, we will still need to navigate the dependency jungle to get our project to the newest django version when it arrives
With a slowly growing test suite, we are getting more and more instances where pipelines fail because of side effects and race conditions that should not occur. If we want to be able to rely on and in any reasonable capacity work with our test suite in the future, we need to address this
User Story
As a developer I want to be able to quickly check whether my changes impacted some unexpected aspect of the system so that I don't code up medusa.git but continually improve the quality of the project.
Additional Context
Design Requirements
Related Issues
#765 #2119 #2711 #2978 #3343 #3370
Summary of discussion and updates to the description