[EPIC] Test coverage and quality gates

[kristenwomack]

Problem statement

azd's code coverage currently sits at 58%, and the team is investing in pushing it higher. As azd ships more capabilities — parallel provisioning, extensions, AI commands — comprehensive test coverage becomes increasingly valuable for catching regressions early and keeping the CLI reliable. We're building toward a workflow where every PR gets fast, automated feedback on test quality: coverage is visible, tracked, and trending upward, giving both the core team and community contributors confidence that their changes are solid.

Vision

Every PR to azd gets automated, fast feedback on test quality. Coverage is visible, tracked, and trending up — not as a vanity metric, but as a real signal that the CLI developers rely on is getting more reliable with every commit.

Who this helps

• Experienced Azure developers: Benefit from fewer regressions in azd releases, meaning fewer surprises when they update. Reliability is a feature.
• azd contributors (team members): Get immediate feedback on whether their PR maintains or improves quality, with clear signals right in the PR workflow.
• azd contributors (community): Coverage tooling on PRs makes contributing easier — they can see exactly what needs testing and whether their tests are sufficient.
• Platform engineers: Higher test coverage provides more confidence when recommending azd as a team-wide standard, with a concrete answer to "is it well-tested?"

Goals (in scope)

• Expand code coverage from 58% to 70%+ across the azd codebase
• Ship playback tests as PR preflight gates: PRs that break recorded scenarios are blocked before merge
• Implement coverage diff tooling on PRs: every PR shows whether it increases or decreases coverage
• Continue adding major error conditions to validation logic, with a path toward automation
• Establish PR governance: coverage requirements, test expectations, and quality signals visible to every contributor

Non-goals (out of scope)

• Owning or building an eval platform; that's a separate epic (see Evals for azd Core)
• Full E2E integration test suite rewrite; incremental improvement, not a from-scratch rebuild
• Achieving 100% coverage; the target is meaningful coverage of critical paths, not checkbox coverage
• Testing infrastructure-side behavior (ARM, Bicep); azd tests azd

Success criteria

• Code coverage reaches 70%+ (measured via CI reporting)
• Coverage diff tooling active on PRs, so contributors see coverage impact before merge
• Playback tests run as PR preflight gates in CI pipeline
• Major error conditions in validation logic are covered with automated tests
• PR governance documented and enforced: what's required to merge

Dependencies

• CI infrastructure: Coverage tooling and playback test integration require CI pipeline changes
• Boundary with Evals: If "Test Pipelines" and "Evals for azd core" are determined to be the same initiative, this epic's scope will expand. If separate, the boundary is: this epic = code quality and reliability; evals = AI scenario quality measurement.
• Validation workstream: Validation improvement work feeds into this epic but ownership and bucketing need clarification