Skip to content

Fixed flaky tests in test_context.py#555

Open
yashsaboo wants to merge 2 commits intocanonical-ols:masterfrom
yashsaboo:master
Open

Fixed flaky tests in test_context.py#555
yashsaboo wants to merge 2 commits intocanonical-ols:masterfrom
yashsaboo:master

Conversation

@yashsaboo
Copy link
Copy Markdown

Test test_context_api and test_null_context are flaky in nature.

TestCase passes consistently in the actual order with command:

pytest tests/test_context.py

TestCase fails when one randomises the order in test_context.py module by using the plugin pytest-random-order with command:

pytest tests/test_context.py --random-order-seed=<Insert-Value-Here> -v

The victims, test_context_api and test_null_context, fails in a failing test order because there is at least one test that runs before the victim and “pollutes” the state (e.g., global state: here, it is Context()) on which the victim depends. (Paper)

After investigation, we found the polluter test to be test_context_gevent and test_context_eventlet
For test_context_gevent: --random-order-seed=795276
For test_context_eventlet: --random-order-seed=570036

Potential Fix: After the polluters are run, reset the Context() to its initial state so that it does not pollute the victims.

Please let us know if you require the pass and fail logs.

Contributors:
Yash Saboo
Nirupam K N

Saboo and others added 2 commits November 26, 2020 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant