Description
We regularly observe the effects of changes to red-knot in the diagnostics emitted when running our (performance) benchmark against tomllib, and sometimes catch issues by seeing new false positives. But a) tomllib is one small codebase, not necessarily representative, and b) the ergonomics of catching these issues in the performance benchmark is poor.
We should select some larger real-world codebases and implement tooling to run red-knot on them and snapshot the diagnostics emitted, such that we can see (and easily update) this snapshot when we make changes to red-knot.
We should run this in CI to ensure our snapshot stays up to date.
(This is very similar to the ruff ecosystem check, and we can likely reuse some of that infrastructure?)
Bonus: if we implement astral-sh/ty#213, we can also track changes to the prevalence of Todo types over time.
Description
We regularly observe the effects of changes to red-knot in the diagnostics emitted when running our (performance) benchmark against tomllib, and sometimes catch issues by seeing new false positives. But a) tomllib is one small codebase, not necessarily representative, and b) the ergonomics of catching these issues in the performance benchmark is poor.
We should select some larger real-world codebases and implement tooling to run red-knot on them and snapshot the diagnostics emitted, such that we can see (and easily update) this snapshot when we make changes to red-knot.
We should run this in CI to ensure our snapshot stays up to date.
(This is very similar to the ruff ecosystem check, and we can likely reuse some of that infrastructure?)
Bonus: if we implement astral-sh/ty#213, we can also track changes to the prevalence of Todo types over time.