Skip to content

chore: Parallelize mutating projects and running the initial test#3415

Open
richardwerkman wants to merge 16 commits intomasterfrom
3414_parallel_mutating_testing
Open

chore: Parallelize mutating projects and running the initial test#3415
richardwerkman wants to merge 16 commits intomasterfrom
3414_parallel_mutating_testing

Conversation

@richardwerkman
Copy link
Copy Markdown
Member

@richardwerkman richardwerkman commented Jan 31, 2026

I'm expecting a modest performance improvement from this on large projects. After this change we:

  • Run the initial test while mutating the syntax trees
  • Mutate syntax trees in parallel, instead of mutate projects in parallel

This will use up more threads, but reduces the runtime of stryker on multi core systems.

Test results:
I got the initial testing and mutating duration on the integration test project down from 4500ms to 1600ms on a 10 core system.

todo:

  • Test on a real world project to measure performance boost
  • Delay writing mutated assembly to after inital test is finished
  • Fix NUnit and MSTest integration tests

closes #3414

Copilot AI review requested due to automatic review settings January 31, 2026 15:39
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to improve overall runtime on large solutions by overlapping the “initial test run” phase with the project mutation phase.

Changes:

  • Renamed orchestration entrypoint to MutateAndTestProjectsAsync and wired it through StrykerRunner.
  • Split initialization into “create inputs” vs “run initial tests”, enabling initial tests and mutation to execute in parallel.
  • Adjusted mutator behavior to enrich test-project metadata after initial tests complete, plus updated unit tests for the new flow.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/Stryker.Core/Stryker.Core/StrykerRunner.cs Uses the new orchestrator method that performs parallel initial testing + mutation.
src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs Implements the parallel execution of initial tests and mutation, and applies initial-test results after both complete.
src/Stryker.Core/Stryker.Core/Initialisation/ProjectMutator.cs Moves initial-test enrichment out of MutateProject into a dedicated post-initial-test method.
src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs Splits input creation from running initial tests (now returns a per-input result map).
src/Stryker.Core/Stryker.Core.UnitTest/StrykerRunnerTests.cs Updates mocks/verifications for the renamed orchestrator API.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectOrchestratorTests.cs Updates tests to call the new orchestration method and stubs the enrichment call.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/ProjectMutatorTests.cs Ensures enrichment is invoked explicitly after mutation.
src/Stryker.Core/Stryker.Core.UnitTest/Initialisation/InitialisationProcessTests.cs Updates tests to use the new “inputs then initial tests” API split.

Copilot AI review requested due to automatic review settings February 3, 2026 20:08
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Copilot AI review requested due to automatic review settings February 5, 2026 19:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Feb 5, 2026

Quality Gate Failed Quality Gate failed

Failed conditions
1 New Minor Issues (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copilot AI review requested due to automatic review settings February 5, 2026 20:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings February 5, 2026 20:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

…cess.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 5, 2026 20:39
…sationProcessTests.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Copilot AI review requested due to automatic review settings February 7, 2026 16:15
@richardwerkman richardwerkman enabled auto-merge (squash) February 7, 2026 16:17
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Change method signature to async for proper handling of asynchronous operations.
Copilot AI review requested due to automatic review settings February 9, 2026 12:37
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Comment on lines +30 to +32
TestDescriptions = executedTests.IsEveryTest
? vsTestDescriptions.ToList()
: vsTestDescriptions.Where(p => executedTests.GetIdentifiers().Contains(p.Id)).ToList();
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filtering branch calls executedTests.GetIdentifiers() inside the Where predicate, which can cause repeated enumeration and quadratic behavior for ITestIdentifiers implementations backed by non-materialized IEnumerable (e.g., WrappedIdentifierEnumeration). Materialize identifiers once (e.g., to a HashSet) before the Where to avoid repeated enumeration and reduce overhead when many tests are present.

Suggested change
TestDescriptions = executedTests.IsEveryTest
? vsTestDescriptions.ToList()
: vsTestDescriptions.Where(p => executedTests.GetIdentifiers().Contains(p.Id)).ToList();
var executedTestIds = executedTests.GetIdentifiers().ToHashSet();
TestDescriptions = executedTests.IsEveryTest
? vsTestDescriptions.ToList()
: vsTestDescriptions.Where(p => executedTestIds.Contains(p.Id)).ToList();

Copilot uses AI. Check for mistakes.
Comment on lines +130 to +133
foreach (var project in projects)
{
DiscoverTests(project, runner);
}
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetMutationTestInputs now runs DiscoverTests in a simple foreach, which makes test discovery strictly sequential across projects. Previously (via the async-per-project input creation) discovery and initial test preparation could run concurrently, which can matter in solution mode with many test projects. Consider parallelizing discovery per project (or integrating it back into RunInitialTestsAsync per-input) so this refactor doesn’t introduce a new bottleneck before the parallel initial-test/mutation phase starts.

Suggested change
foreach (var project in projects)
{
DiscoverTests(project, runner);
}
Parallel.ForEach(projects, project =>
{
DiscoverTests(project, runner);
});

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve performance by running initial test and mutating in parallel

2 participants