Flaky coverage result on the same code

[lg2de]

Two builds/tests on the same code base can create different coverage status.
Here, I have the two builds on the same commit. One build shows missing overage on a single branch, while the other build has full coverage on that line.
https://coveralls.io/builds/78514666/source?filename=Src%2FAwesomeAssertions%2FCollections%2FGenericCollectionAssertions.cs#L278
https://coveralls.io/builds/78515357/source?filename=Src%2FAwesomeAssertions%2FCollections%2FGenericCollectionAssertions.cs#L278

I'm quite sure that it is not caused by flaky test execution according to my code analysis.
Can you help to investigate the root cause?
How to identify which part of the line is missing coverage?

[afinetooth]

Hi @lg2de

When you're comparing two coverage reports of the same type of entity (build, job or source_file), you'll want to forego any comparison of percentages, and focus on actual coverage stats, since percentages can obfuscate what's actually changed.

RCA Approach

You can analyze coverage stats in a couple of ways:

---

1. Side-by-Side Comparison:

First, you can get a quick, visual comparison, with a side-by-side of the same Source File Report across two different builds, which may help you catch obvious differences quickly:

Example: Your example source_file report, with the first build on the LEFT and the second build on the RIGHT:

Changes shown: In your case, a change from 0 hits to 1 hit on one branch (branch 283) causes a yellow line to turn green in the subsequent build. Any changes between 0 hits and 1 hits will "pop" visually, because they cause a color change.

These two screenshots show the exact changes:

From Build 1:

To Build 2:

---

2. Exact Comparison:

A side-by-side comparison of the full JSON version of each report—which is the only thing stored at Coveralls, BTW, the coverage stats for each source_file at each commit; no code—will give you the exact differences.

For very large files, performing a git diff on the two JSON files, or, easier, asking an LLM to identify all differences, will reveal what's going on:

---

2.a) Visual Comparison:

Even a quick visual comparison of the full JSON for:

Build 1 (JSON): https://coveralls.io/builds/78514666/source.json?filename=Src%2FAwesomeAssertions%2FCollections%2FGenericCollectionAssertions.cs

To:

Build 2 (JSON): https://coveralls.io/builds/78515357/source.json?filename=Src%2FAwesomeAssertions%2FCollections%2FGenericCollectionAssertions.cs

Reveals that you have a doubling of the stats on the RIGHT side, suggesting two identical reports were uploaded for the same file (perhaps in two different formats, and perhaps for the entire test suite):

---

2.b) Machine-Assisted Comparison:

But certainly, a direct machine comparison will reveal all:

LLM (ChatGPT) Assessment:

TL;DR:
At a high-level, coverage shape is identical, but execution intensity is roughly doubled.

✅ Same lines covered vs. uncovered
✅ No meaningful structural changes (no added/removed code paths)
🔼 Nearly all execution counts ≈ 2× higher in the second build

One exception:
The difference between the builds, at line 278, is not just execution amplification. At least at line 278, there is a real branch-coverage improvement.

Therefore:
Coveralls is rendering line 278 from both:

• a line hit count (44 vs 88), and
• separate branch coverage metadata for that line.

---

And that is true:

If branch coverage analysis is enabled for your project—which it is, in your repo settings:

And branch coverage is present in your original reports—which it is, I can see it in your original report for that file:

    {
      "name": "Src/AwesomeAssertions/Collections/GenericCollectionAssertions.cs",
      "coverage": [
        null,
        null,
        null,
        [...]
        null,
        null,
        124
      ],
      "branches": [
        81,
        81,
        246,
        [...]
        3680,
        761,
        8
      ]
    },

Then Coveralls is considering both line coverage and branch coverage for each source_file's aggregate coverage report.

Analysis

So in this case, you have:

1. A doubling of your line coverage stats; and
2. Some—apparently minimal—improvement of your branch coverage stats.

Deeper Branch Coverage Analysis

If you really want to learn more about what those differences are—especially the branch coverage differences, you can use the Source File Directories and Paths subsection of the JSON-Format Web Data section of our API Docs, which shows you how to get full coverage reports for individual source files and source directories, or groups of the same.

This URL, using the paths= URL param, will show you both the line coverage and branch coverage stored at Coveralls for your source_file—so let's get a report for both builds:

1. Build 1: https://coveralls.io/builds/78514666.json?paths=/Src/AwesomeAssertions/Collections/GenericCollectionAssertions.cs

{
id: 78514666,
repo_name: "AwesomeAssertions/AwesomeAssertions",
branch: "because-format",
url: "https://github.com/AwesomeAssertions/AwesomeAssertions/actions/runs/23712618138",
badge_url: "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_98.svg",
calculated_at: "2026-03-29T15:48:50Z",
created_at: "2026-03-29T15:48:39Z",
commit_message: "Merge 9933856be9e7c51c1ddbc33248f419f7580e79b6 into cbfbea385712ca2ca9664c56e5a35a7aefb9d996",
committer_name: "GitHub",
committer_email: "noreply@github.com",
commit_sha: "798ba5129e67bc80a38051093e8ebc43f44555d0",
updated_at: "2026-03-29T15:50:10Z",
coverage_change: -0.006,
covered_percent: 97.87937858113297,
covered_lines: 12742,
missed_lines: 155,
relevant_lines: 12897,
covered_branches: 3828,
missed_branches: 204,
relevant_branches: 4032,
paths: "/Src/AwesomeAssertions/Collections/GenericCollectionAssertions.cs",
selected_source_files_count: 0,
paths_covered_percent: null,
paths_previous_covered_percent: null,
paths_covered_percent_change: null
}

2. Build 2: https://coveralls.io/builds/78515357.json?paths=/Src/AwesomeAssertions/Collections/GenericCollectionAssertions.cs

{
id: 78515357,
repo_name: "AwesomeAssertions/AwesomeAssertions",
branch: "because-format",
url: "https://github.com/AwesomeAssertions/AwesomeAssertions/actions/runs/23714694881",
badge_url: "https://s3.amazonaws.com/assets.coveralls.io/badges/coveralls_98.svg",
calculated_at: "2026-03-29T19:18:56Z",
created_at: "2026-03-29T17:34:12Z",
commit_message: "Merge 5aabb71f8eb6ec7e2fa49d32226d03ca496647ee into cbfbea385712ca2ca9664c56e5a35a7aefb9d996",
committer_name: "GitHub",
committer_email: "noreply@github.com",
commit_sha: "3275fcd292452bc63be298367fce17a91541bca3",
updated_at: "2026-03-29T19:20:10Z",
coverage_change: 0,
covered_percent: 97.88528560458386,
covered_lines: 12742,
missed_lines: 155,
relevant_lines: 12897,
covered_branches: 3829,
missed_branches: 203,
relevant_branches: 4032,
paths: "/Src/AwesomeAssertions/Collections/GenericCollectionAssertions.cs",
selected_source_files_count: 0,
paths_covered_percent: null,
paths_previous_covered_percent: null,
paths_covered_percent_change: null
}

Details on Branch Coverage Changes:

Exactly 1 additional branch became covered in Build 2.

The Exact Difference:

• Build 1
  • Covered branches: 3828
  • Missed branches: 204
  • Total branches: 4032
• Build 2
  • Covered branches: 3829
  • Missed branches: 203
  • Total branches: 4032

What changed:

• +1 covered branch
• −1 missed branch
• Total branches unchanged

This is a net improvement of exactly one branch

Interpretation:

This aligns perfectly with what we observed visually at line 278:

• In Build 1: one branch (branch 283) was not executed
• In Build 2: that same branch was executed at least once

That single change:

• Flips the line from partial (yellow) → fully covered (green)
• Accounts for the entire delta in branch coverage across the file

Summary

The branch coverage difference between the two builds is very small but precise: exactly one additional branch is covered in Build 2.

• Build 1: 3828 / 4032 branches covered
• Build 2: 3829 / 4032 branches covered

This corresponds to a single branch (visible at line 278 in the screenshots) that was not executed in Build 1 but was executed in Build 2.

Because the total number of branches is large (4032), this results in only a very small increase in overall coverage percentage.