Story: we want to automate post-deployment integration testing, but it requires DAG-style manual testing and long waits (for slow application tasks, waiting on events, like email, etc.)
- We can model Tests as DAG where they require earlier steps to complete, but some can be done in parallel
- The framework should allow us to write Python (or TypeScript)
- Ideally we can use Playwright for any browser automation (or similar) because the primary purpose will be to interact with a web page, then wait for events or poll for readiness
- There must be a way to write assertions (such as with Playwright), but also separately in the case of evaluating downloaded files, etc.
- The framework will need some way to register event handlers, such as waiting for an email to one or more configured email accounts (e.g. polling the Gmail API)
- The framework needs to have good Dx - specific steps can be run (calculates shortest path in DAG to requested tests), ways to see test progress in real time, review test results and performance when finished, option to end early or not, able to review Playwright interactions, good experience in CLI like pytest, etc.
- Unlike more sophisticated tools like https://hatchet.run, this framework should be runnable as a standalone instance (e.g. on a single ECS instance, but primarily run locally without the need to scale horizontal for load testing)
The framework should ideally be flexible enough for this specific use case, but also viable for other types of browser automation tasks