Allow only running a specific set of tests

[camelid]

Describe the problem you are trying to solve
It's currently not possible (to my understanding) to filter out multiple tests with Cargo. It would be nice to be able to rerun all the tests in a CI build that failed, without running all the passing tests too.

Describe the solution you'd like
A --tests option that lets you pass a list of tests to run, excluding all other tests. Although I would like it if it were not limited to just integration tests (as --test is), and have it work on unit tests too. If I understand correctly, this would be the equivalent of passing --test to the underlying test runner, except that the test runner does not allow multiple --test options:

error: Option 'test' given more than once

Also it would be nice if at the end of a test run where there were failed tests, Cargo printed out the command that you can run to only re-run the failed tests.

Notes
I find the --test option confusing as it seems to only run integration tests. Could this be changed to allow you to use it with unit tests as well?

See https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/.60cargo.20test.20--test.20foo.60.20for.20multiple.20tests for more context.

Cc @jyn514

[jyn514]

Minor correction: cargo test test_name passes test_name to the test runner, not --test test_name.

[jyn514]

Maybe this should be a feature request for libtest? The bit about --test only working for integration tests is a good one though.

[ehuss]

There is an important distinction about --test vs a test name filter, and they can't really be unified (cargo has no insight into where tests are defined). One way to think of it is like an onion, such as:

workspace > package > target crate > modules > #[test] function

-p flag determines the workspace and package (defaults to the package in the current directory, or the entire workspace in the root of a virtual workspace), the target selection flags select the target (--lib, --bin, --example, --test, etc.), and then the actual test name can be passed as an argument like cargo test my_fancy_test. If you don't pass a target flag, Cargo has no choice but to build all targets and pass the filter to all of them.

As for passing multiple filters, that is part of the libtest harness which lives in the rust-lang/rust repository. The issue tracking that is rust-lang/rust#30422. I personally would like to see support for multiple filters added, and I think it should be relatively easy. Unfortunately there isn't any active work or maintenance happening with libtest, but if you're interested you could maybe see if the libs team would accept/review such a change. dtolnay (a libs team member) mentioned here that they would be ok with it, although that was 3 years ago, so things might have changed.

[heisen-li]

It may be better if it supports cargo test --test TESTNAME1, TESTNAME2, FOO::TESTNAME3.

From this, I feel like should there be a parameter to support re-running test cases that failed in the previous round of testing?
like：cargo test --rerun-failed-tests

[epage]

#7864 was our issue for --test and <TEST_NAME> being confused which was fixed in #12915.

rust-lang/rust#30422 was closed.

So it seems like there is nothing left for this issue

It may be better if it supports cargo test --test TESTNAME1, TESTNAME2, FOO::TESTNAME3.

Having --test accept test names, in place of test targets, is potentially breaking could be quite confusing.

From this, I feel like should there be a parameter to support re-running test cases that failed in the previous round of testing?
like：cargo test --rerun-failed-tests

Huh, not seeing an issue for this. There'd be a lot of design issues to work out but its useful.

[ehuss]

Huh, not seeing an issue for this. There'd be a lot of design issues to work out but its useful.

#10673?