Skip to content

Commit ab37a05

Browse files
authored
Improve instructions to agents for running tests (#23804)
## Summary Several changes to our instructions that should, I think, make it easier and faster for agents to run our tests: - Suggest setting `CARGO_PROFILE_DEV_OPT_LEVEL=1` rather than `--cargo-profile=fast-test`. `--cargo-profile` is only a valid command if you're invoking the tests using `cargo nextest run`; if you're invoking the tests using `cargo test`, you have to use `--profile=fast-test` instead. That means that it sometimes takes three tries for Claude to run our tests successfully if it's being invoked from Claude Code for Web: 1. Try `cargo nextest run --cargo-profile=fast-test` <-- oh no, `cargo nextest` isn't installed 2. Try `cargo test --cargo-profile=fast-test` <-- oh no, `--cargo-profile` isn't a valid CLI argument for `cargo test` 3. Try `cargo test --profile=fast-test` <-- hooray, it finally worked! - Set two environment variables for `cargo-insta` so that snapshot tests are easier for agents to handle. Together these are equivalent to running `cargo insta test --accept`, but they should work even if `cargo-insta` isn't installed (which it won't be in a Claude Code for Web environment): - `INSTA_FORCE_PASS` means that the test runner will keep going even if it encounters a failing snapshot in an mdtest - `INSTA_UPDATE=always` means that the snapshots are automatically accepted, so the agent doesn't need to do a separate `cargo insta review` step; it can just review the changes that have been applied to the existing snapshot files. ## Test Plan Merge it and see?
1 parent be5db3c commit ab37a05

1 file changed

Lines changed: 7 additions & 17 deletions

File tree

AGENTS.md

Lines changed: 7 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,41 +4,31 @@ This repository contains both Ruff (a Python linter and formatter) and ty (a Pyt
44

55
## Running Tests
66

7-
Run all tests (using `nextest` for faster execution):
7+
Run all tests (using `nextest` for faster execution, setting `CARGO_PROFILE_DEV_OPT_LEVEL=1` to enable optimizations while retaining debug info, and setting `INSTA_FORCE_PASS=1 INSTA_UPDATE=always` to ensure all snapshots are updated):
88

99
```sh
10-
cargo nextest run
11-
```
12-
13-
For faster test execution, use the `fast-test` profile which enables optimizations while retaining debug info:
14-
15-
```sh
16-
cargo nextest run --cargo-profile fast-test
10+
CARGO_PROFILE_DEV_OPT_LEVEL=1 INSTA_FORCE_PASS=1 INSTA_UPDATE=always cargo nextest run
1711
```
1812

1913
Run tests for a specific crate:
2014

2115
```sh
22-
cargo nextest run -p ty_python_semantic
16+
CARGO_PROFILE_DEV_OPT_LEVEL=1 INSTA_FORCE_PASS=1 INSTA_UPDATE=always cargo nextest run -p ty_python_semantic
2317
```
2418

25-
Run a single mdtest file:
19+
Run a single mdtest file. The path to the mdtest file should be relative to the `crates/ty_python_semantic/resources/mdtest` folder:
2620

2721
```sh
28-
cargo nextest run -p ty_python_semantic --test mdtest -- mdtest::<path/to/mdtest_file.md>
22+
CARGO_PROFILE_DEV_OPT_LEVEL=1 INSTA_FORCE_PASS=1 INSTA_UPDATE=always cargo nextest run -p ty_python_semantic -- mdtest::<path/to/mdtest_file.md>
2923
```
3024

3125
To run a specific mdtest within a file, use a substring of the Markdown header text as `MDTEST_TEST_FILTER`. Only use this if it's necessary to isolate a single test case:
3226

3327
```sh
34-
MDTEST_TEST_FILTER="<filter>" cargo nextest run -p ty_python_semantic --test mdtest -- mdtest::<path/to/mdtest_file.md>
28+
MDTEST_TEST_FILTER="<filter>" CARGO_PROFILE_DEV_OPT_LEVEL=1 INSTA_FORCE_PASS=1 INSTA_UPDATE=always cargo nextest run -p ty_python_semantic -- mdtest::<path/to/mdtest_file.md>
3529
```
3630

37-
Update snapshots after running tests:
38-
39-
```sh
40-
cargo insta accept
41-
```
31+
After running the tests, always review the contents of any snapshots that have been added or updated.
4232

4333
## Running Clippy
4434

0 commit comments

Comments
 (0)