chore: Parallelize mutating projects and running the initial test#3415
chore: Parallelize mutating projects and running the initial test#3415richardwerkman wants to merge 16 commits intomasterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR aims to improve overall runtime on large solutions by overlapping the “initial test run” phase with the project mutation phase.
Changes:
- Renamed orchestration entrypoint to
MutateAndTestProjectsAsyncand wired it throughStrykerRunner. - Split initialization into “create inputs” vs “run initial tests”, enabling initial tests and mutation to execute in parallel.
- Adjusted mutator behavior to enrich test-project metadata after initial tests complete, plus updated unit tests for the new flow.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.Core/Stryker.Core/StrykerRunner.cs | Uses the new orchestrator method that performs parallel initial testing + mutation. |
| src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs | Implements the parallel execution of initial tests and mutation, and applies initial-test results after both complete. |
| src/Stryker.Core/Stryker.Core/Initialisation/ProjectMutator.cs | Moves initial-test enrichment out of MutateProject into a dedicated post-initial-test method. |
| src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs | Splits input creation from running initial tests (now returns a per-input result map). |
| src/Stryker.Core/Stryker.Core.UnitTest/StrykerRunnerTests.cs | Updates mocks/verifications for the renamed orchestrator API. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectOrchestratorTests.cs | Updates tests to call the new orchestration method and stubs the enrichment call. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectMutatorTests.cs | Ensures enrichment is invoked explicitly after mutation. |
| src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/InitialisationProcessTests.cs | Updates tests to use the new “inputs then initial tests” API split. |
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
|
src/Stryker.Core/Stryker.Core/ProjectComponents/TestProjects/TestProjectsInfo.cs
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/InitialisationProcessTests.cs
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs
Outdated
Show resolved
Hide resolved
…cess.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…sationProcessTests.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs
Outdated
Show resolved
Hide resolved
src/Stryker.Core/Stryker.Core/MutationTest/CsharpMutationProcess.cs
Outdated
Show resolved
Hide resolved
Change method signature to async for proper handling of asynchronous operations.
| TestDescriptions = executedTests.IsEveryTest | ||
| ? vsTestDescriptions.ToList() | ||
| : vsTestDescriptions.Where(p => executedTests.GetIdentifiers().Contains(p.Id)).ToList(); |
There was a problem hiding this comment.
The filtering branch calls executedTests.GetIdentifiers() inside the Where predicate, which can cause repeated enumeration and quadratic behavior for ITestIdentifiers implementations backed by non-materialized IEnumerable (e.g., WrappedIdentifierEnumeration). Materialize identifiers once (e.g., to a HashSet) before the Where to avoid repeated enumeration and reduce overhead when many tests are present.
| TestDescriptions = executedTests.IsEveryTest | |
| ? vsTestDescriptions.ToList() | |
| : vsTestDescriptions.Where(p => executedTests.GetIdentifiers().Contains(p.Id)).ToList(); | |
| var executedTestIds = executedTests.GetIdentifiers().ToHashSet(); | |
| TestDescriptions = executedTests.IsEveryTest | |
| ? vsTestDescriptions.ToList() | |
| : vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList(); |
| foreach (var project in projects) | ||
| { | ||
| DiscoverTests(project, runner); | ||
| } |
There was a problem hiding this comment.
GetMutationTestInputs now runs DiscoverTests in a simple foreach, which makes test discovery strictly sequential across projects. Previously (via the async-per-project input creation) discovery and initial test preparation could run concurrently, which can matter in solution mode with many test projects. Consider parallelizing discovery per project (or integrating it back into RunInitialTestsAsync per-input) so this refactor doesn’t introduce a new bottleneck before the parallel initial-test/mutation phase starts.
| foreach (var project in projects) | |
| { | |
| DiscoverTests(project, runner); | |
| } | |
| Parallel.ForEach(projects, project => | |
| { | |
| DiscoverTests(project, runner); | |
| }); |




I'm expecting a modest performance improvement from this on large projects. After this change we:
This will use up more threads, but reduces the runtime of stryker on multi core systems.
Test results:
I got the initial testing and mutating duration on the integration test project down from 4500ms to 1600ms on a 10 core system.
todo:
closes #3414